Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Introduction to Ultrasonic Mid-Air Haptic Effects Read chapter Read first chapter

2022 | OriginalPaper | Chapter

Ultrasound Mid-Air Tactile Feedback for Immersive Virtual Reality Interaction

Authors : Thomas Howard, Maud Marchal, Claudio Pacchierotti

Published in: Ultrasound Mid-Air Haptics for Touchless Interfaces

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Ultrasound mid-air haptic (UMH) devices are promising for tactile feedback in virtual reality (VR), as they do not require users to be tethered to, hold, or wear any device. This approach is less cumbersome, easy to set up, can simplify tracking, and leaves the hands free for concurrent interactions. This chapter explores work conducted at CNRS-IRISA dealing with the challenges arising from the integration of UMH interfaces in immersive VR through three main axes. These are discussed in the wider context of the state of the art on UMH for augmented and virtual reality, and illustrated through several VR use-cases. A first axis deals with device integration into the VR ecosystem. Interaction in immersive VR is based on the synergy between complex input devices allowing real-time tracking of the user and multimodal feedback devices delivering a coherent visual, auditory and haptic picture of a simulated virtual environment (VE). Using UMH in immersive VR therefore hinges on integrating UMH devices such that their operation does not interfere with other input and feedback devices. It is also critical to ensure that UMH feedback is adequately synchronized and co-located with respect to other stimuli, and delivered within a workspace that is compatible with that of VR interaction. Regarding this final point, we propose PUMAH, a robotic solution for increasing the usable workspace of UMH devices. The second and third axes, respectively, focus on stimulus perception and rendering of VE properties. Virtual object properties can be rendered in a variety of ways, through, e.g., amplitude modulation (AM) or spatiotemporal modulation (STM), with many parameters (modulation frequency, spatial sampling, etc.) coming into play, raising questions about the limits of the design space. To tackle this challenge, we begin by conducting psychophysical experimentation to understand the usable ranges for stimulus parameters and understand the perceptual implications of stimulus design choices. We propose an open-source software framework intended to facilitate UMH stimulus design and perceptual evaluation. These results in turn serve as the basis for the design and evaluation of rendering schemes for VR. Using amplitude variations along a focal point path in STM, we investigate the possibility of rendering geometric details and in a second step, sensations of stiffness in VR.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
previous chapter Augmenting Automotive Gesture Infotainment Interfaces Through Mid-Air Haptic Icon Design
next chapter Multimodal Interaction with Mid-Air Haptics
Footnotes
Literature
Arafsha F, Zhang L, Dong H, El Saddik A (2015) Contactless haptic feedback: state of the art. In: 2015 IEEE International symposium on haptic, audio and visual environments and games (HAVE). IEEE, pp 1–6
Ariga K, Fujiwara M, Makino Y, Shinoda H (2020) Midair haptic presentation using concave reflector. In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 307–315
Balint P, Althoefer K (2018) Medical virtual reality palpation training using ultrasound based haptics and image processing. In: Proc. Jt. Work. New Technol. Comput. Assist. Surg
Barreiro H, Sinclair S, Otaduy MA (2020) Path routing optimization for STM ultrasound rendering. IEEE Trans Haptics 13(1):45–51CrossRef
Beattie D, Frier W, Georgiou O, Long B, Ablart D (2020) Incorporating the perception of visual roughness into the design of mid-air haptic textures. In: ACM Symposium on applied perception, pp 1–10
Brice D, McRoberts T, Rafferty K (2019) A proof of concept integrated multi-systems approach for large scale tactile feedback in VR. In: International conference on augmented reality, virtual reality and computer graphics. Springer, Berlin, pp 120–137
Carter T, Seah SA, Long B, Drinkwater B, Subramanian S (2013) Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces. In: Proceedings of the 26th annual ACM symposium on user interface software and technology, pp 505–514
Carvalheiro C, Nóbrega R, da Silva H, Rodrigues R (2016) User redirection and direct haptics in virtual environments. In: Proceedings of the 24th ACM international conference on multimedia, pp 1146–1155
Chatzopoulos D, Bermejo C, Huang Z, Hui P (2017) Mobile augmented reality survey: from where we are to where we go. IEEE Access 5:6917–6950CrossRef
Cirio G, Marchal M, Otaduy MA, Lécuyer A (2013) Six-DoF haptic interaction with fluids, solids, and their transitions. In: 2013 World haptics conference (WHC). IEEE, pp 157–162
Corenthy L, Giordano M, Hayden R, Griffiths D, Jeffrey C, Limerick H, Georgiou O, Carter T, Müller J, Subramanian S (2018) Touchless tactile displays for digital signage: mid-air haptics meets large screens. In: Extended abstracts of the 2018 CHI conference on human factors in computing systems, pp 1–4
Cornelio Martinez PI, De Pirro S, Vi CT, Subramanian S (2017) Agency in mid-air interfaces. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 2426–243
de Tinguy X, Pacchierotti C, Marchal M, Lécuyer A (2019) Toward universal tangible objects: optimizing haptic pinching sensations in 3d interaction. In: 2019 IEEE Conference on virtual reality and 3D user interfaces (VR). IEEE, pp 321–330
de Tinguy X, Howard T, Pacchierotti C, Marchal M, Lécuyer A (2020) Weatavix: wearable actuated tangibles for virtual reality experiences. In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 262–270
Di Luca M, Mahnan A (2019) Perceptual limits of visual-haptic simultaneity in virtual reality interactions. In: 2019 IEEE world haptics conference (WHC). IEEE, pp 67–72
Dzidek B, Frier W, Harwood A, Hayden R (2018) Design and evaluation of mid-air haptic interactions in an augmented reality environment. In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 489–499
Eid MA, Al Osman H (2015) Affective haptics: current research and future directions. IEEE Access 4:26–40CrossRef
Evangelou G, Limerick H, Moore JW (2021) I feel it in my fingers! Sense of agency with mid-air haptics. IEEE World Haptics
Freeman E, Anderson R, Williamson J, Wilson G, Brewster SA (2017) Textured surfaces for ultrasound haptic displays. In: Proceedings of the 19th ACM international conference on multimodal interaction, pp 491–492
Frier W, Ablart D, Chilles J, Long B, Giordano M, Obrist M, Subramanian S (2018) Using spatiotemporal modulation to draw tactile patterns in mid-air. In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 270–281
Frier W, Pittera D, Ablart D, Obrist M, Subramanian S (2019) Sampling strategy for ultrasonic mid-air haptics. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–11
Frutos-Pascual M, Harrison JM, Creed C, Williams I (2019) Evaluation of ultrasound haptics as a supplementary feedback cue for grasping in virtual environments. In: 2019 International conference on multimodal interaction, pp 310–318
Georgiou O, Biscione V, Harwood A, Griffiths D, Giordano M, Long B, Carter T (2017) Haptic in-vehicle gesture controls. In: Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications adjunct, pp 233–238
Georgiou O, Jeffrey C, Chen Z, Tong BX, Chan SH, Yang B, Harwood A, Carter T (2018) Touchless haptic feedback for VR rhythm games. In: 2018 IEEE Conference on virtual reality and 3D user interfaces (VR). IEEE, pp 553–554
Gil H, Son H, Kim JR, Oakley I (2018) Whiskers: exploring the use of ultrasonic haptic cues on the face. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–13
Hajas D, Pittera D, Nasce A, Georgiou O, Obrist M (2020) Mid-air haptic rendering of 2d geometric shapes with a dynamic tactile pointer. IEEE Trans Haptics 13(4):806–817CrossRef
Hasegawa K, Qiu L, Noda A, Inoue S, Shinoda H (2017) Electronically steerable ultrasound-driven long narrow air stream. Appl Phys Lett 111(6):064104
Hoshi T (2016) Gradual phase shift to suppress noise from airborne ultrasound tactile display. In: Proceedings of the ACM CHI workshop mid-air haptics displays: systems for un-instrumented mid-air interactions. Session 2: Provide vis. haptic feedback
Hoshi T, Abe D, Shinoda H (2009a) Adding tactile reaction to hologram. In: RO-MAN 2009—The 18th IEEE international symposium on robot and human interactive communication. IEEE, pp 7–11
Hoshi T, Iwamoto T, Shinoda H (2009b) Non-contact tactile sensation synthesized by ultrasound transducers. In: World haptics 2009—Third joint EuroHaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems. IEEE, pp 256–260
Hoshi T, Takahashi M, Iwamoto T, Shinoda H (2010) Noncontact tactile display based on radiation pressure of airborne ultrasound. IEEE Trans Haptics 3(3):155–165CrossRef
Howard T, Gallagher G, Lécuyer A, Pacchierotti C, Marchal M (2019a) Investigating the recognition of local shapes using mid-air ultrasound haptics. In: 2019 IEEE World haptics conference (WHC). IEEE, pp 503–508
Howard T, Marchal M, Lécuyer A, Pacchierotti C (2019b) PUMAH: pan-tilt ultrasound mid-air haptics for larger interaction workspace in virtual reality. IEEE Trans Haptics 13(1):38–44
Howard T, Gicquel G, Marchal M, Lécuyer A, Pacchierotti C (2021) PUMAH: pan-tilt ultrasound mid-air haptics. In: WHC 2021-IEEE World haptics conference
Hwang I, Son H, Kim JR (2017) AirPiano: enhancing music playing experience in virtual reality with mid-air haptic feedback. In: 2017 IEEE World haptics conference (WHC). IEEE, pp 213–218
Inoue S, Kobayashi-Kirschvink KJ, Monnai Y, Hasegawa K, Makino Y, Shinoda H (2014) Horn: the hapt-optic reconstruction. In: ACM SIGGRAPH 2014 emerging technologies, pp 1–1
Inoue S, Makino Y, Shinoda H (2015) Active touch perception produced by airborne ultrasonic haptic hologram. In: 2015 IEEE World haptics conference (WHC). IEEE, pp 362–367
Ito M, Kokumai Y, Shinoda H (2019) Midair click of dual-layer haptic button. In: 2019 IEEE World haptics conference (WHC). IEEE, pp 349–352
Jang J, Park J (2020) SPH fluid tactile rendering for ultrasonic mid-air haptics. IEEE Trans Haptics 13(1):116–122CrossRef
Kamigaki T, Suzuki S, Shinoda H (2020) Noncontact thermal and vibrotactile display using focused airborne ultrasound. In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 271–278
Kappus B, Long B (2018) Spatiotemporal modulation for mid-air haptic feedback from an ultrasonic phased array. J Acoust Soc Am 143(3):1836–1836CrossRef
Karafotias G, Korres G, Teranishi A, Park W, Eid M (2017) Mid-air tactile stimulation for pain distraction. IEEE Trans Haptics 11(2):185–191CrossRef
Kervegant C, Raymond F, Graeff D, Castet J (2017) Touch hologram in mid-air. In: ACM SIGGRAPH 2017 emerging technologies, pp 1–2 (2017)
Kimura Y, Makino Y, Shinoda H (2016) Computer-created interactive 3d image with midair haptic feedback. In: International AsiaHaptics conference. Springer, Berlin, pp 491–494
Korres G, Eid M (2016) Haptogram: ultrasonic point-cloud tactile stimulation. IEEE Access 4:7758–7769CrossRef
Korres G, Aujeszky T, Eid M (2017) Characterizing tactile rendering parameters for ultrasound based stimulation. In: 2017 IEEE World haptics conference (WHC). IEEE, pp 293–298
Kreimeier J, Hammer S, Friedmann D, Karg P, Bühner C, Bankel L, Götzelmann T (2019) Evaluation of different types of haptic feedback influencing the task-based presence and performance in virtual reality. In: Proceedings of the 12th ACM international conference on pervasive technologies related to assistive environments, pp 289–298
Long B, Seah SA, Carter T, Subramanian S (2014) Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans Graph (TOG) 33(6):1–10CrossRef
Lugo-Villeda LI, Frisoli A, Pabon S, Padilla MA, Sotgiu E, Bergamasco M (2009) Light-exoskeleton and data-glove integration for enhancing virtual reality applications. In: 2009 International conference on advanced robotics. IEEE, pp 1–6
Makino Y, Furuyama Y, Inoue S, Shinoda H (2016) Haptoclone (haptic-optical clone) for mutual tele-environment by real-time 3d image transfer with midair force feedback. In: CHI, pp 1980–1990
Marchal M, Gallagher G, Lécuyer A, Pacchierotti C (2020) Can stiffness sensations be rendered in virtual reality using mid-air ultrasound haptic technologies? In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 297–306
Martinez J, Griffiths D, Biscione V, Georgiou O, Carter T (2018) Touchless haptic feedback for supernatural VR experiences. In: 2018 IEEE Conference on virtual reality and 3D user interfaces (VR). IEEE, pp 629–630
Martinez J, Harwood A, Limerick H, Clark R, Georgiou O (2019) Mid-air haptic algorithms for rendering 3d shapes. In: 2019 IEEE International symposium on haptic, audio and visual environments and games (HAVE). IEEE, pp 1–6
Matsubayashi A, Makino Y, Shinoda H (2019) Direct finger manipulation of 3d object image with ultrasound haptic feedback. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–11
Meli L, Scheggi S, Pacchierotti C, Prattichizzo D (2014) Wearable haptics and hand tracking via an RGB-D camera for immersive tactile experiences. In: ACM SIGGRAPH 2014 posters, pp 1–1
Meli L, Pacchierotti C, Salvietti G, Chinello F, Maisto M, De Luca A, Prattichizzo D (2018) Combining wearable finger haptics and augmented reality: user evaluation using an external camera and the microsoft hololens. IEEE Rob Autom Lett 3(4):4297–4304CrossRef
Mercado V, Howard T, Si-Mohammed H, Argelaguet F, Lécuyer A (2021) Alfred: the haptic butler—on-demand tangibles for object manipulation in virtual reality using an ETHD. IEEE World Haptics
Mestre D, Fuchs P, Berthoz A, Vercher J (2006) Immersion et présence. Le traité de la réalité virtuelle. Ecole des Mines de Paris, Paris, pp 309–38
Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda H (2014) HaptoMime: mid-air haptic interaction with a floating virtual screen. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, pp 663–667
Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda H (2015) Adding texture to aerial images using ultrasounds. In: Haptic interaction. Springer, Berlin, pp 59–61
Mulot L, Gicquel G, Frier W, Marchal M, Pacchierotti C, Howard T (2021a) Curvature discrimination for dynamic ultrasound mid-air haptic stimuli. In: WHC 2021-IEEE World haptics conference
Mulot L, Gicquel G, Zanini Q, Frier W, Marchal M, Pacchierotti C, Howard T (2021b) DOLPHIN: a framework for the design and perceptual evaluation of ultrasound mid-air haptic stimuli. In: ACM Symposium on applied perception
Obrist M, Seah SA, Subramanian S (2013) Talking about tactile experiences. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1659–1668
Obrist M, Subramanian S, Gatti E, Long B, Carter T (2015) Emotions mediated through mid-air haptics. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp 2053–2062
Ochiai Y, Hoshi T, Suzuki I (2017) Holographic whisper: rendering audible sound spots in three-dimensional space by focusing ultrasonic waves. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 4314–4325
Pittera D, Ablart D, Obrist M (2019a) Creating an illusion of movement between the hands using mid-air touch. IEEE Trans Haptics 12(4):615–623
Pittera D, Gatti E, Obrist M (2019b) I’m sensing in the rain: spatial incongruity in visual-tactile mid-air stimulation can elicit ownership in VR users. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–15
Popescu GV, Burdea GC, Trefftz H (2002) Multimodal interaction modeling. In: Handbook of virtual environments. CRC Press, pp 475–494
Price A, Long B (2018) Fibonacci spiral arranged ultrasound phased array for mid-air haptics. In: 2018 IEEE International ultrasonics symposium (IUS). IEEE, pp 1–4
Rakkolainen I, Freeman E, Sand A, Raisamo R, Brewster S (2020) A survey of mid-air ultrasound haptics and its applications. IEEE Trans Haptics 14(1):2–19CrossRef
Ramsamy P, Haffegee A, Jamieson R, Alexandrov V (2006) Using haptics to improve immersion in virtual environments. In: International conference on computational science. Springer, Berlin, pp 603–609
Romanus T, Frish S, Maksymenko M, Frier W, Corenthy L, Georgiou O (2019) Mid-air haptic bio-holograms in mixed reality. In: 2019 IEEE International symposium on mixed and augmented reality adjunct (ISMAR-adjunct). IEEE, pp 348–352
Rümelin S, Gabler T, Bellenbaum J (2017) Clicks are in the air: how to support the interaction with floating objects through ultrasonic feedback. In: Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications, pp 103–108
Rutten I, Frier W, Van den Bogaert L, Geerts D (2019) Invisible touch: how identifiable are mid-air haptic shapes? In: Extended abstracts of the 2019 CHI conference on human factors in computing systems, pp 1–6
Rutten E, Van Den Bogaert L, Geerts D (2020) From initial encounter with mid-air haptic feedback to repeated use: the role of the novelty effect in user experience. IEEE Trans Haptics
Saint-Aubert J, Regnier S, Haliyo S (2018) Cable driven haptic interface for co-localized desktop VR. In: 2018 IEEE Haptics symposium (HAPTICS). IEEE, pp 351–356
Salazar SV, Pacchierotti C, de Tinguy X, Maciel A, Marchal M (2020) Altering the stiffness, friction, and shape perception of tangible objects in virtual reality using wearable haptics. IEEE Trans Haptics 13(1):167–174CrossRef
Sand A, Rakkolainen I, Isokoski P, Kangas J, Raisamo R, Palovuori K (2015) Head-mounted display with mid-air tactile feedback. In: Proceedings of the 21st ACM symposium on virtual reality software and technology, pp 51–58
Shakeri G, Freeman E, Frier W, Iodice M, Long B, Georgiou O, Andersson C (2019) Three-in-one: levitation, parametric audio, and mid-air haptic feedback. In: Extended abstracts of the 2019 CHI conference on human factors in computing systems, pp 1–4
Suzuki S, Takahashi R, Nakajima M, Hasegawa K, Makino Y, Shinoda H (2018) Midair haptic display to human upper body. In: 2018 57th Annual conference of the society of instrument and control engineers of Japan (SICE). IEEE, pp 848–853
Van den Bogaert L, Geerts D (2020) User-defined mid-air haptic sensations for interacting with an AR menu environment. In: International conference on human haptic sensing and touch enabled computer applications. Springer, Berlin, pp 25–32
Vi CT, Ablart D, Gatti E, Velasco C, Obrist M (2017) Not just seeing, but also feeling art: mid-air haptic experiences integrated in a multisensory art exhibition. Int J Hum-Comput Stud 108:1–14CrossRef
Waltemate T, Senna I, Hülsmann F, Rohde M, Kopp S, Ernst M, Botsch M (2016) The impact of latency on perceptual judgments and motor performance in closed-loop interaction in virtual reality. In: Proceedings of the 22nd ACM conference on virtual reality software and technology, pp 27–35
Wilson G, Carter T, Subramanian S, Brewster SA (2014) Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1133–1142
Yoshida K, Horiuchi Y, Inoue S, Makino Y, Shinoda H (2017) HaptoCloneAR: mutual haptic-optic interactive system with 2d image superimpose. In: ACM SIGGRAPH 2017 emerging technologies, pp 1–2
Yoshino K, Shinoda H (2013) Visio-acoustic screen for contactless touch interface with tactile sensation. In: 2013 World haptics conference (WHC). IEEE, pp 419–423
Metadata
Title
Ultrasound Mid-Air Tactile Feedback for Immersive Virtual Reality Interaction
Authors
Thomas Howard
Maud Marchal
Claudio Pacchierotti
Copyright Year
2022
Book
Ultrasound Mid-Air Haptics for Touchless Interfaces

Print ISBN: 978-3-031-04042-9

Electronic ISBN: 978-3-031-04043-6

Copyright Year: 2022

DOI
https://doi.org/10.1007/978-3-031-04043-6_6