Hapticity in Hybrid Space from an Enactive Perspective

Visuality has been intricately linked to sight and its dominance in sensory perception, while hapticity, traditionally associated with touch, offers a significant way to comprehend the interaction between our bodies and the world. Although historically grounded in the body’s physiological capacities, contemporary perspectives have expanded the understanding of hapticity. Garrington acknowledges hapticity as a “broad sense modality,” encompassing kinaesthesis, proprioception, and the vestibular sense, thereby suggesting additional somatic sensations beyond touch [2]. J. J. Gibson explained that the haptic system is an apparatus by which the individual gets information about both his environment and his body. It is the perceptual system by which animals and men are literally in touch with the environment [3]. Architect Juhani Pallasmaa defines hapticity and tactile experience as “multi-sensory,” wherein the qualities of space, matter, and scale are equally measured by the eye, ear, nose, skin, tongue, skeleton, and muscle [4]. Mallgrave posits hapticity as a “synonym for the emotive and multisensory experience of architecture, which includes the visual dimension” [5]. Paterson argues that touch is not solely a physiological sense but also a form of communication [6]. Building on these interpretations, this study views hapticity as a form of multi-sensory perceptual integration, exploring the relationship between vision and the tactile sense to redefine hapticity.

Vision is not solely confined to optical perception or “retinal vision,” but is intricately linked to the tactile sense. Touch can be understood as an augmentation of vision, as opposed to any articulation of the specificity of tactility or the haptic system in itself. Pallasmaa proposes the existence of an “unconscious tactile ingredient in vision,” essential in architectural appreciation. [7]. Laura Marks’s theory of “haptic visuality” offers an alternative model that recognizes the significance of haptic perception - integrating senses such as movement, touch, and embodied memory - in shaping the temporal experience of architecture [8]. Similarly, Deleuze and Guattari’s re-conceptualization of space through Riegl’s “haptic”/“optic” distinction presents a “haptic space,” where the eye performs a “non-optical function” [9]. These interpretations of haptic vision are closely intertwined with bodily movement and embodied experience. To understand haptic vision and the complex relationship between sense modalities, it is necessary to refer to the theory of embodied cognition, which draws attention to the non-visual senses and challenges simplistic relationships between sense modalities.

Embodied cognition re-focuses cognitive science research to include the body as a critical component of cognition. This theory arose out of phenomenology. Merleau-Ponty coined the term “flesh-of-the-world” to describe the shared fabric of material embodiment [10]. The idea that the body and the world are essentially one and the same is becoming influential in embodied cognitive science. It suggests that cognitive processes are grounded in bodily interaction in real space and real-time.

Further evidence of how the body perceives and copes with the environment and things comes from the study of the mirror neuron system. It reveals significant overlap in the neural mechanisms active during both perception and action [11]. The neural circuits involved in performing a particular action are also activated when observing someone else carrying out the same movements. This suggests that the motor system not only controls movement but also plays a cognitive role. Motor neurons respond to visual, tactile, and auditory stimuli. By activating the mirror neurons, we engage in an “embodied simulation”, a basic functional mechanism of the brain [12].

The response of empathic mirror-neuron system is not limited to the physical space. Researchers Nai Li and colleagues claimed that the meaningful integration of physical environments with mobile AR-supported activities can shape cognitive process in an embodied way [13]. Moreover, Jason Farman suggested that the body is conceived out of sensory engagement across material and digital landscapes. He argues that bodies and spaces exist through their use and movement, through person-to-person and person-to-object relationships. Once embodiment is enacted, it is not necessarily constrained to a physical space [14]. These studies highlight that these spaces, whether digital, material, or a hybrid space, never function as disembodied zones. When the virtual space is integrated into physical space, the embodied experience of virtual space might in a way impacts the cognitive experience of physical space. The paper examines recent practices that involve new media technologies, with a focus on their potential to create more interactive environments. The question arises: might virtual affordances trigger a more profound haptic experience?

At the Louvre-Lens Museum, a virtual representation of the exhibition space is integrated in the real space through a mobile guiding device interface. Unlike traditional museum guides that typically rely on audio information and focus on the physical exhibition space, this exhibition space incorporates a direct visual interface in the guide. The device features a buttonless touch screen interface showcasing a 3D exhibition hall (Fig. 1).

At first glance, it may appear that the viewpoint provided by the guide engages the user as a disembodied voyeur rather than a situated subject. However, it could be argued that the body’s interaction with the virtual space is an integral part of the embodied experience within the exhibition space. At the same time, movement in the physical space and engagement with the art can also lead to interactions with the virtual space (Fig. 2).

In this space, two interaction processes are constantly at play. The first involves continuous contact with the screen and the virtual space. Using the guide may evoke a sensation of effortless “gliding” through the virtual environment, prompting movement and exploration in the physical space. This “gliding” introduces an additional layer of interaction, anticipating real movements in the physical space. Furthermore, the guide not only assists visitors in selecting and locating artworks, but also provides detailed information for each piece through simple clicks on the virtual objects, thereby presenting the virtual space as a field of opportunities for action.

The other important interaction occurs during the cognitive transitional reading between the virtual and real space, with the mirror neurons playing a pivotal role in mediating these two. When the finger touches the screen and the virtual body “flies” in the virtual space, motor neurons are activated. This activation doesn’t just confine the exploration to the virtual body; it also prompts physical movements in the real space as a result of the overlapping reading of the two spaces. Visitors’ attention constantly flows between the real and virtual spaces. The actions of engaging with the space result from the response of mirror neurons and motor neurons to both physical and virtual spaces. The familiarity with touch screen interfaces, stemming from the widespread use of smartphones, renders the guide’s interface second nature. Additionally, the similarity in materiality between the two spaces reduces the cognitive effort required; the real space itself resembles a virtual space due to SANAA’s characteristic expression of materiality - pure whiteness in this case. It can be inferred that the introduction of the virtual space has transformed how individuals experience the physical exhibition space: they perceive within the hybrid space and act within the physical space.

At the Switzerland National Museum, a large terrain model of Switzerland takes centre stage in a spacious exhibition room near the entrance. Different from conventional way of annotating physical model, a set of mobile devices with touch screen is placed in front of the model. Visitors can pick up a device, point it at the model, and instantly access augmented geographical information. The touch screen interface allows users to select different geographical details, seamlessly integrating a virtual layer with the physical space, offering additional information. The images on the screen represent a real-time fusion of the physical and virtual worlds, enabling users to simultaneously absorb information from both the screen and the physical model. This haptic interaction with the touch screen, combined with bodily movement around the model, introduces a temporal dimension to the learning process. Visitors have the freedom to sit, stand, or move around the space, engaging with the terrain in a three-dimensional manner, rather than merely viewing a 2D image. These devices entice passive observers to become active participants, involving their attention, bodies, and senses in a deeper engagement with the physical exhibition space.

Beyond individual interaction, these devices serve as effective tools for fostering social engagement. Visitors observe others using the devices, waiting for their own turn to play with them. They share their experiences with the other visitors, passing the device between them. In this way, the mobile interfaces become an integral part of their being-with-others, facilitating social interaction and shared engagement with the exhibition space (Fig. 3).

In contrast to hand-held touch screens, the digital layer at the Qingdao Beer Museum offers visitors a more immersive experience. Housed within the old factory, the museum’s exhibition narrative mirrors the beer production process. In a particular workshop, life-sized figure projections are strategically placed next to the beer pool, vividly illustrating the area’s historical use. They give faces and voices to the people who once worked there, which is a perfect alignment of digital imagery with the authentic 3D environment. As shown in Fig. 4, a life-sized projection of a German expert tasting the beer is repeatedly animated, intertwining the stories of people and places into a captivating storytelling process. Accompanied by background music, the once neutral space becomes infused with memories and emotions, adding an auditory dimension to the overall experience. This haptic perception is further heightened by the simulated beer (water) that is “spat” from the virtual character to the visitors at the end of each play. Usually confined to optical readings, the museum has been remapped in the realm of haptic encounters, consistently delivering engaging, meaningful and memorable experiences to its visitors. This seamless experience and atmosphere are a result of a careful synchronization of the figure’s image, sound, and tactile sensation of the water, weaving together the haptic, optic, and aural dimensions.

Neither the actual site nor the projected animation alone can evoke a strong sense of presence as effectively as mixed reality can. It provides visitors with an intimate experience to evoke a sense of presence. Visitors can engage in a multi-sensory journey, experiencing the interaction between the present and the past. The surfaces of the screens express a new materiality as they convey the virtual transformation while themselves keep almost invisible in the scene to create a seamless integration between the virtual and the real.

In the Department of Architecture at Xi’an Jiaotong-Liverpool University, the design studio presentations have been transformed through the adoption of a 3D mapping digital performance format. A permanent projector has been installed in the atrium of the Design Building, the usual venue for students’ work exhibitions. The entire floor area of the atrium serves as the canvas for the projection, with physical site models positioned in the center to be augmented by digital information during the performance. Additionally, a large vertical screen is placed adjacent to the horizontal projection, creating a seamless and easily accessible display for the audience on the ground floor. As students present, the digital performance unfolds simultaneously, allowing them to freely interact with the large-scale site and building plans projected on the floor, engaging with the graphic information using their entire bodies. This integration of the digital layer with the physical space creates a haptic experience, encouraging more physical movement, particularly walking, as a response to the animation of the larger-scale graphic information. It is this literality of contact that motivates anthropologist Tim Ingold, in his article on perceiving the world through the feet, to consider a more literally grounded approach to perception [15] (Fig. 5).

Beyond individual haptic experiences, this digital mapping also fosters collective participation and shared experiences. It introduces a new mode of co-presence, transforming the configurations of public spaces within the Design Building. The presenter’s movements and the digital mapping captivate the audience’s attention as the haptic images unfold, drawing them deeper into the content of the presentation. The audience becomes more actively involved, responding to the presenter’s actions and the movement of the large-scale graphic information projected on the floor, which is a participatory sense-making process. This shared display of information serves as a new platform for users to interact in this public space.