• deutsch | english

Knowledge Spaces in Virtual Reality: Intuitive Interfacing with a Multiperspective Hypermedia Environment

Project description

Interfaces between our real and a digital world have the potential to facilitate knowledge work in educational and scientific contexts. Multiperspective Hypermedia Environments (MHEs) can be accessed by means of such interfaces. MHE are essentially datasets that enable the generation of multiple perspectives onto the data, such as medical data, data about the contents of a museum, or even data about a large group of animals or plants. Each perspective corresponds to a data display in a panel within which the data – or parts of it – are arranged with respect to particular feature axis, in clusters, or in a graph structure. Indeed, it was shown that MHEs equipped with elaborated 2D interfaces can support multiperspective reasoning skills (MPRSs), developing the ability to reason about the shown context in an elaborate, relational manner by taking on and combining different perspectives on the knowledge content. However, only learners with sufficient cognitive capacity at their disposal profited from MHEs. 


Our analysis suggests that the provided 2D interfaces onto the MHEs are highly demanding in terms of the necessary cognitive resources to (i) interact with the different knowledge displays (panels) representing perspectives on a topic, (ii) orient oneself within each panel, and (iii) inter-relate contents in different panels. Based on design principles derived from theories of event segmentation, predictive processing, and embodied cognition, we propose to create and study interactive, highly immersive 3D-interfaces to MHEs using virtual reality (VR) technology. The goal is to facilitate the development of MPRSs further, especially for learners with lower cognitive capacities, by (i) implementing intuitive embodied interactions with panels arranged in the virtual knowledge space by means of a sensorimotor user interface, (ii) supporting the fast orientation within panels by optimal semantic-spatial mappings and the provision of gist-like information about content arrangements, (iii) fostering mental inter-relations of content on different panels according to spatial arrangement and visual cues, making content inter-relations directly accessible. To evaluate the developed 3D cognitive interface, we will contrast learners’ MPRSs when working with the available 2D and the targeted 3D interface in two diverse knowledge domains, namely marine biodiversity and art history.


  2. Project team
Publications
Butz, M. V., Bilkey, D., Humaidan, D., Knott, A., & Otte, S. (2019). Learning, planning, and control in amonolithic neural event inference architecture. Neural Networks, 117, 135-144.

Butz, M.V., Menge, T., Humaidan, D., & Otte, S. (2019). Inferring event-predictive goal-directedobject manipulations in REPRISE. International Conference on Artificial Neural Networks, 2019, 639-653

Gumbsch, C., Butz, M.V., & Martius, G. (2019). Autonomous identification and goal-directedinvocation of event-predictive behavioral primitives. IEEE Transactions on Cognitive andDevelopmental Systems. https://ieeexplore.ieee.org/document/8753716 

Lohmann, J. & Butz, M.V. (2019). Unflinching predictions: Anticipatory crossmodal interactions areunaffected by the current hand posture. Proceedings of the 41st Annual Meeting of the CognitiveScience Society. 692-698.

Lohmann, J., Belardinelli, A., Butz, M. V. (2019). Hands ahead in mind and motion: Active inference in Peripersonal Hand Space. Vision, 3(2), 15; https://doi.org/10.3390/vision3020015

Lohmann, J., Weigert, P., & Butz, M. V. (2019). Grasping Uncertainty: A free Energy Approach to Anticipatory Behavior Control. In Proceeding of the European Conference for Cognitive Science 2019 (EuroCogSci). Bochum.


Conference Contributions
Gerjets, P., Lachmair, M., Butz, M. V., & Lohmann, J. (2018). Knowledge Spaces in VR: Intuitive Interfacing with a Multiperspective Hypermedia Environment. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE. Reutlingen. https://doi.org/10.1109/vr.2018.8446137 [Poster]