Ultrasound Mid-Air Haptics for Touchless Interfaces

In this chapter, we discuss the basic physical principles of ultrasonic mid-air haptics. Our aim is to provide a holistic introduction to the technology, facilitate a better understanding of these principles, and help newcomers to join this exciting field of research in following the rest of this book. To that end, we have assumed a simplified and idealized situation and divide our discussion into four sub-topics: acoustic radiation pressure, phased array focusing, vibrotactile stimulation, and by-product audible sounds.

Mid-air haptic feedback presents exciting new opportunities for useful and delightful interactive systems. However, with these opportunities come several design challenges that vary greatly depending on the application at hand. In this chapter, we reveal these challenges from a user experience perspective. To that end, we first provide a comprehensive literature review covering many of the different applications of the technology. Then, we present 12 design guidelines and make recommendations for effective mid-air haptic interaction designs and implementations. Finally, we suggest an iterative haptic design framework that can be followed to create a quality mid-air haptic experience.

Touchless user interfaces enable people to interact with digital services and information without physically touching an input device. There are numerous benefits to touchless interaction (including convenience, hygiene and the potential for more expressive input), and sensing technologies have advanced significantly in recent years. As a result, touchless user interfaces have been adopted on a wider scale across a variety of application areas, e.g. automotive, digital signage and gaming. However, usability remains a key concern; touchless gesture input poses several interaction challenges, many related to uncertainty and the inherent loss of tactile cues. Ultrasound haptic feedback has shown promise in helping users overcome such interaction challenges, restoring the missing sense of touch and closing the feedback loop for effective haptic interaction. This chapter explores how ultrasound haptic feedback has been used in touchless user interface design and presents design patterns used by industry and academia alike.

Mid-air technology is not well studied in the context of multisensory experience. Despite increasing advances in mid-air interaction and mid-air haptics, we still lack a good understanding of how such technologies might influence human behaviour and experience. Compare this with the understanding, we currently have about physical touch, which highlights the need for more knowledge in this area. In this chapter, I describe three areas of development that consider human multisensory perception and relate these to the study and use of mid-air haptics. I focus on three main challenges of developing multisensory mid-air interactions. First, I describe how crossmodal correspondence could improve the experience of mid-air touch. Then, I outline some opportunities to introduce mid-air touch to the study of multisensory integration. Finally, I discuss how this multisensory approach can benefit applications that encourage and support a sense of agency in interaction with autonomous systems. Considering these three contributions, when developing mid-air technologies can provide a new multisensory perspective, resulting in the design of more meaningful and emotionally-loaded mid-air interactions.

A growing body of work is demonstrating the potential benefits of haptic interfaces for the automotive domain, including the use of ultrasound haptic interfaces. In this chapter, we present our work into the development of an in-vehicle mid-air gesture interface for drivers, utilising ultrasound haptic feedback. Our interface uses carefully designed “ultrahapticons” to give feedback and present information to users. We discuss the design and evaluation of our ultrahapticons, giving insight into the design of ultrasound haptic feedback for a gesture interface and contributing a set of effective haptic patterns that can be applied in new application areas.

Ultrasound mid-air haptic (UMH) devices are promising for tactile feedback in virtual reality (VR), as they do not require users to be tethered to, hold, or wear any device. This approach is less cumbersome, easy to set up, can simplify tracking, and leaves the hands free for concurrent interactions. This chapter explores work conducted at CNRS-IRISA dealing with the challenges arising from the integration of UMH interfaces in immersive VR through three main axes. These are discussed in the wider context of the state of the art on UMH for augmented and virtual reality, and illustrated through several VR use-cases. A first axis deals with device integration into the VR ecosystem. Interaction in immersive VR is based on the synergy between complex input devices allowing real-time tracking of the user and multimodal feedback devices delivering a coherent visual, auditory and haptic picture of a simulated virtual environment (VE). Using UMH in immersive VR therefore hinges on integrating UMH devices such that their operation does not interfere with other input and feedback devices. It is also critical to ensure that UMH feedback is adequately synchronized and co-located with respect to other stimuli, and delivered within a workspace that is compatible with that of VR interaction. Regarding this final point, we propose PUMAH, a robotic solution for increasing the usable workspace of UMH devices. The second and third axes, respectively, focus on stimulus perception and rendering of VE properties. Virtual object properties can be rendered in a variety of ways, through, e.g., amplitude modulation (AM) or spatiotemporal modulation (STM), with many parameters (modulation frequency, spatial sampling, etc.) coming into play, raising questions about the limits of the design space. To tackle this challenge, we begin by conducting psychophysical experimentation to understand the usable ranges for stimulus parameters and understand the perceptual implications of stimulus design choices. We propose an open-source software framework intended to facilitate UMH stimulus design and perceptual evaluation. These results in turn serve as the basis for the design and evaluation of rendering schemes for VR. Using amplitude variations along a focal point path in STM, we investigate the possibility of rendering geometric details and in a second step, sensations of stiffness in VR.

Multimodal human–computer interfaces utilize sensory information from different modalities to create richer and more immersive user experiences. The use of ultrasound haptics in a multimodal interface is compelling: It requires no physical contact, has high spatial and temporal resolution, and can even be physically co-located with other sensory information (e.g., visual content in mid-air). In this chapter, we explore examples of multimodal interface that use ultrasound haptics, and reflect on the technical and design challenges of creating a high-quality multimodal ultrasound haptic interface.

Unconventional displays, such as 3D displays, projection screens formed of flowing light-scattering particles (fogscreens), and virtual reality (VR) headsets, can create illusions of images floating in mid-air. Paired with hand tracking, gestural interaction with floating user interfaces (UI) is possible on this permeable imagery, thus creating reach-through touchscreens that react and recover instantly from intersecting fingers and objects. The user can explore virtual environments and control floating UIs with hand gestures which could help, for example, in simulated training and in creating an improved feeling of immersion. However, hand-based gestural interaction with such UIs can be difficult without haptic sensations typical in daily activities. Without haptics, the level of immersion and smoothness of interaction suffers if the hands can pass through virtual objects without triggering tactile sensations. Ultrasound haptics is a method to produce a focused airborne acoustic air pressure on a user’s skin, thus creating an unobtrusive, mid-air sensation of touch. Fogscreens, VR headsets, or some other unconventional displays together with ultrasound haptics enable tactile interaction with “touchless touchscreens”. These tactile, floating UIs open new opportunities, e.g., for immersive interaction, advertisement, and entertainment. It can bring back the missing haptic feedback for these displays.

This chapter describes techniques for midair vibrotactile stimulation based on ultrasound foci with temporally varied intensities or positions. This variation of the focal property is called modulations and can be categorized into three fundamental types of methods according to the modulating fashions and obtained vibrotactile effects: amplitude modulation (AM), lateral modulation (LM), and spatiotemporal modulation (STM). Appropriate modulation is useful in designing vibrotactile textures, enhancing the subjective strength of aroused sensation, and transmitting geometrical information about spatial vibrotactile patterns. The aim of this chapter is to provide a brief and simple overview of the main rendering techniques and to discuss their pros and cons, thus providing sufficient engineering insight to the haptics and human–computer interaction (HCI) readers.

Although smaller and thinner ultrasonic mid-air haptic phased arrays are important to business-to-consumer applications, large-scale phased arrays are exploring the possibility of powerful mid-air haptics. For research applications to ultimately “hack” the entire ultrasonic field of a room or public environment, large array operations are inevitable. In this chapter, after the advantages and disadvantages of employing large-scale arrays are summarized, we present an implementation of a scalable phased array unit that can be installed in a distributed fashion and some of its potential applications.

In ultrasound haptics, a tactile sensation is evoked on human skin that touches the high sound pressure area generated due to the interference of ultrasonic waves. Therefore, by controlling the distribution of sound pressure amplitude using a transducer array, it is possible to create tactile sensations at multiple points simultaneously. In this chapter, we first describe the relationship between the complex gains of the transducer array and the generated sound field. Then, we provide various algorithms to control the amplitude pattern based on this relationship.

Mid-air haptic stimulation will be accompanied by visual information in many future applications. Vision is a fast and precise means of obtaining spatial information and is how we consume most forms of digital information. The superimposition of visual and haptic feedback at the same position would provide an intuitive interface that requires minimal learning and could lead to a more engaging and immersive user experience. In this chapter, we introduce typical strategies of visual–tactile superimposition when using an ultrasound haptics device, including: an aerial computer interface with ultrasound haptic feedback, visuo-tactile projection on the skin, holographic transmission of physical entities, and applications with head-mounted displays. We also discuss the problem of positional matching between vision and haptics which is crucial for a high-quality user experience.

Ultrasound haptic feedback is typically used to augment the multi-sensory experience with spatiotemporal patterns projected on the users’ hands. Many studies have considered the usability of such techniques on the users’ palms as it is more sensitive to ultrasound stimuli. Studies exploring the ultrasound feedback on the users’ fingertips have utilized large ultrasound phased arrays to project perceptible haptic stimuli. Spatiotemporal patterns at the fingertips using smaller phased arrays have been largely unexplored due to their weaker sensations. In this chapter, we first present a survey of ultrasound stimuli patterns that have considered the users’ fingers for haptic feedback. Then, a set of spatiotemporal stimuli for ultrasound feedback on the finger is presented along with results from a user study and associated examples of mid-air gestures. In the end, the prospect of ultrasound haptic sensations at the fingertip is summarized from a survey.

Arrays of emitters and receivers are seen in a wide range of applications, from the square kilometre array used in radio astronomy to those used for medical ultrasound imaging. Here, we explore the use of arrays to steer and focus ultrasound for the purposes of mid-air haptics, but many of the basic principles are shared with these other applications. To achieve a mid-air haptic effect, we must use the array to focus ultrasound to a point and thereby create a high-intensity local region. The force then occurs when an object, such as a human hand, is positioned at the focus of the ultrasound beam. Here, the momentum of the sound wave is transferred directly to the object, and the haptic force is proportional to the ultrasonic intensity. High-intensity ultrasound also creates a flow called acoustic streaming, as some of the wave momentum is absorbed by the air causing it to move. These forces and flows interact with the skin where users perceive the presence of a physical object. This chapter will introduce and bring together these ideas to provide an understanding of how mid-air ultrasonic haptics works and how such systems can be designed.

Focused ultrasound is the base mechanism for mid-air tactile feedback generation, acoustic levitation, wireless power transfer, directional audio and other emerging applications. The basic required set-up is an ultrasonic emitter with the capability of focusing its acoustic power at a target point. Ideally, a multi-emitter phased array is used since it is capable of steering and shaping the sound field with millimetre accuracy and a time response in the order of milliseconds. There are compelling commercial products and open designs for this kind of ultrasonic arrays. Here, we review the different elements that compose an ultrasonic array: from the emitters and the driving electronics to the signal generators or algorithms. We review some techniques to simulate the output of ultrasonic arrays or to determine the emission phases for target fields. Also, we provide some suggestions for future challenges related to cost, power and heat reduction.

In this chapter, we review the effects of exposure of the human body to ultrasonic waves and discuss the tolerance sound pressure levels reported in the literature. We then consider the theory of nonlinear absorption and discuss its implications for the optimal safety distance that should be maintained from ultrasonic devices during operation. The aims of this chapter are to provide insight into what is currently known about the safety of mid-air ultrasound and to highlight areas where more research is needed.

Ultrasound mid-air haptic technology has advanced in many ways over the past decade and has found meaningful application in a plethora of use cases. As the technology matures further and progresses from lab to market, in this chapter, we take a step back and discuss three specific directions that we think could result in the greatest impact. Namely, we highlight challenges and opportunities in improving (1) the hardware platforms used, (2) the rendering algorithms employed to create rich haptic sensations, and (3) the resulting user experience and added value the technology can instill to different end-user applications. We hope that this “wish-list” inspires the mid-air haptics and human computer interaction (HCI) community and others to join our efforts toward a deeper technology understanding, integration, and readiness.