Top

Published in:

2022 | OriginalPaper | Chapter

Superimposing Visual Images on Mid-Air Ultrasonic Haptic Stimulation

Authors : Hiroyuki Shinoda, Yasutoshi Makino

Published in: Ultrasound Mid-Air Haptics for Touchless Interfaces

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Mid-air haptic stimulation will be accompanied by visual information in many future applications. Vision is a fast and precise means of obtaining spatial information and is how we consume most forms of digital information. The superimposition of visual and haptic feedback at the same position would provide an intuitive interface that requires minimal learning and could lead to a more engaging and immersive user experience. In this chapter, we introduce typical strategies of visual–tactile superimposition when using an ultrasound haptics device, including: an aerial computer interface with ultrasound haptic feedback, visuo-tactile projection on the skin, holographic transmission of physical entities, and applications with head-mounted displays. We also discuss the problem of positional matching between vision and haptics which is crucial for a high-quality user experience.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Sound-Field Creation for Haptic Reproduction

next chapter Ultrasound Mid-Air Haptic Feedback at the Fingertip

MicroMirror Array Plate. A glass-plate-like device including strip mirrors perpendicular to the plate surface. The plate was produced by Asukanet Co., Ltd. as the product name AI plate. A similar device principle was also independently presented by Dr. Satoshi Maekawa, NICT.

https://www.youtube.com/watch?v=1B-7uQ5RD2A “BMW HoloActive TOUCH at CES 2017”

https://www.youtube.com/watch?v=Y-P1zZAcPuw “Touchable Holography” video produced by Takayuki Hoshi in Shinoda laboratory

https://www.youtube.com/watch?v=Bb0hNMxxewg “Visuo-Tactile Projector” video produced by Keisuke Hasegawa in Shinoda laboratory

https://materialized-graphics.hapislab.org/ JST CREST, materialized graphics project since 2018

Brice D, McRoberts T, Rafferty K (2019) A proof of concept integrated multi-systems approach for large scale tactile feedback in VR. In: De Paolis L., Bourdot P. (eds) augmented reality, virtual reality, and computer graphics, lecture notes in computer science, vol 11613

Frier W, Ablart D, Chilles J, Long B, Giordano M, Obrist M, Subramanian S (2018) Using spatiotemporal modulation to draw tactile patterns in mid-air. In: 2018 proceedings of Euro haptics, Part I, pp 270–281

Hasegawa K, Qiu L, Noda A, Inoue S, Shinoda H (2017) Electronically steerable ultrasound-driven long narrow air stream. Appl Phys Lett 111:064104CrossRef

Hasegawa K, Qiu L, Shinoda H (2018) Midair ultrasound fragrance rendering. IEEE Trans vis Comput Graphics 24(4):1477–1485CrossRef

Hasegawa K, Shinoda H (2013) Aerial display of vibrotactile sensation with high spatial-temporal resolution using large-aperture airborne ultrasound phased array. In: 2013 Proceedings of the IEEE world haptics conference, pp 31–36

Hirayama R, Martinez Plasencia D, Masuda N, Subramanian S (2019) A volumetric display for visual, tactile and audio presentation using acoustic trapping. Nature 575:320–323CrossRef

Hoshi T, Takahashi M, Iwamoto T, Shinoda H (2010) Noncontact tactile display based on radiation pressure of airborne ultrasound. IEEE Trans Haptics 3(3):155–165CrossRef

Hoshi T, Takahashi M, Nakatsuma K, Shinoda H (2009) Touchable holography. In: 2009 proceedings of ACM SIGGRAPH emerging technologies. ACM, New York, NY, USA, Article no. 23

Howard T, Marchal M, Lécuyer A, Pacchierotti C (2020) PUMAH: pan-tilt ultrasound mid-air haptics for larger interaction workspace in virtual reality. IEEE Trans Haptics 13(1):38–44CrossRef

Howard T, Gallagher G, Lécuyer A, Pacchierotti C, Marchal M (2019) Investigating the recognition of local shapes using mid-air ultrasound haptics. In: 2019 Proceeding IEEE world haptics conference, pp 503–508

Inoue S, Kobayashi-Kirschvink KJ, Monnai Y, Hasegawa K, Makino Y, Shinoda H (2014) HORN: the hapt-optic reconstruction. In: 2014 ACM SIGGRAPH emerging technologies

Inoue S, Makino Y, Shinoda H (2015) Active touch perception produced by airborne ultrasonic haptic hologram. In: 2015 Proceedings of the IEEE world haptics conference (WHC), pp 362–367

Ito M, Kokumai Y, Shinoda H (2019) Midair click of dual-layer haptic button. In: 2019 proceedings of the IEEE world haptics conference, pp 349–352, Tokyo, Japan

Iwamoto T, Tatezono M, Shinoda H (2008) Non-contact method for producing tactile sensation using airborne ultrasound. Proc Euro Haptics 2008:504–513

Kuchenbecker KJ, Fiene J, Niemeyer G (2006) Improving contact realism through event-based haptic feedback. IEEE Trans Visual Comput Graphics 12(2):219–230CrossRef

Loomis JM (1981) Tactile pattern perception. Perception 10:5–27CrossRef

Makino Y, Furuyama Y, Inoue S, Shinoda H (2016) HaptoClone (Haptic-Optical Clone) for mutual tele-environment by real-time 3D image transfer with midair force feedback. In: 2016 Proceedings of the chi conference on human factors in computing systems, pp 1980–1990

Marchal M, Gallagher G, Lécuyer A, Pacchierotti C (2020) Can stiffness sensations be rendered in virtual reality using mid-air ultrasound haptic technologies? Proc Euro Haptics 2020:297–306

Matsubayashi A, Makino Y, Shinoda H (2019a) Direct finger manipulation of 3d object image with ultrasound haptic feedback. In: 2019a proceedings of the 2019a CHI conference on human factors in computing systems, Paper no 87, pp 1–11

Matsubayashi A, Oikawa H, Mizutani S, Makino Y, Shinoda H (2019b) Display of haptic shape using ultrasound pressure distribution forming cross-sectional shape.In: 2019b Proceedings of the 2019b IEEE world haptics conference, pp 419–424, Tokyo, Japan

Matsubayashi A, Yamaguchi T, Makino Y, Shinoda H (2021) Rendering softness using airborne ultrasound. In: 2021 proceedings of the IEEE world haptics conference, pp 355–360

Monnai Y, Hasegawa K, Fujiwara M, Yoshino K, Inoue S, Shinoda H (2014) HaptoMime: Mid-air haptic interaction with a floating virtual screen. In: 2014 proceedings of the 27th annual ACM symposium on user interface software and technology (UIST’14). ACM, New York, NY, USA, pp 663–667

Morisaki T, Fujiwara M, Makino Y, Shinoda H (2021) Non-vibratory pressure sensation produced by ultrasound focus moving laterally and repetitively with fine spatial step width. IEEE trans haptics (early access)

Nakajima M, Hasegawa K, Makino Y, Shinoda H (2021) Spatiotemporal pinpoint cooling sensation produced by ultrasound-driven mist vaporization on skin. 2021 IEEE trans haptics (Early access)

Norasikin MA, Martinez-Plasencia D, Memoli G, Subramanian S (2019) SonicSpray: a technique to reconfigure permeable mid-air displays. In: 2019 proceedings of the 2019 ACM international conference on interactive surfaces and spaces (ISS ‘19), pp 113–122

Ochiai Y, Hoshi T, Suzuki I (2017) Holographic whisper: rendering audible sound spots in three-dimensional space by focusing ultrasonic waves. In: 2017 Proceedings of the 2017 CHI conference on human factors in computing systems, pp 4314–4325

Pacchierotti C, Sinclair S, Solazzi M, Frisoli A, Hayward V, Prattichizzo D (2017) Wearable haptic systems for the fingertip and the hand: taxonomy, review, and perspectives. IEEE Trans Haptics 10(4):580–600CrossRef

Pittera D, Gatti E, Obrist M (2019) I’m sensing in the rain: spatial incongruity in visual-tactile mid-air stimulation can elicit ownership in VR users. In: 2019 Proceedings of the CHI conference on human factors in computing systems. Association for computing machinery, New York, NY, USA, Paper 132, pp 1–15

Rakkolainen I, Sand A, Palovuori K (2015) Midair user interfaces employing particle screens. IEEE Comput Graph Appl 35(2):96–102CrossRef

Rim S, Suzuki S, Toide Y, Fujiwara M, Makino Y, Shinoda H (2020) Sound-image icon with aerial haptic feedback. Proc Euro Haptics 2020:497–505

Satoshi S (2019) Thermal-radiation-based haptic display using laser-emission-based radiation control. In: 2019 Proceedings of IEEE world haptics 2019, WP2P.10 (Work-in-Progress Papers)

Sand A, Rakkolainen I, Isokoski P, Kangas J, Raisamo R, Palovuori K (2015) Head-mounted display with mid-air tactile feedback. In: 2015 proceedings of the 21st ACM symposium on virtual reality software and technology (VRST ‘15), pp 51–58

Serizawa K, Morisaki T, Delfosse C, Fujiwara M, Makino Y, Shinoda H (2020) Super haptoclone: upper-body mutual telexistence system with haptic feedback. SIGGRAPH ’20, emerging technologies, Washington, D.C, USA (moved to a virtual conference)

Shinoda H (2010) Tactile interaction with 3D images. In: 2010 the 17th international display workshops (IDW’10), INP4: 3D interactive systems, pp 1743–1746

Suzuki S, Fujiwara M, Makino Y, Shinoda H (2018) Midair ultrasound haptic display with large workspace. In: 2018 AsiaHaptics 2018, Incheon, Korea

Takahashi R, Hasegawa K, Shinoda H (2020) Tactile stimulation by repetitive lateral movement of midair ultrasound focus. IEEE Trans Haptics 13(2):334–342CrossRef

Takahashi R, Hasegawa K, Shinoda H (2018) Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli. In: 2018 proceedings of Euro haptics, Part II, pp 276–288

Vega-Bermudez F, Johnson KO, Hsiao SS (1991) Human tactile pattern recognition: active versus passive touch, velocity effects, and patterns of confusion. J Neurophysiol 65:531–546CrossRef

Wilson G, Carter T, Subramanian S, Brewster SA (2014) Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion. In: 2014 Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’14), pp 1133–1142

Yoshida K, Horiuchi Y, Inoue S, Makino Y, Shinoda H (2017) HaptoCloneAR: mutual haptic-optic interactive system with 2D image superimpose. In: 2017 Proceedings of SIGGRAPH emerging technologies, Los Angeles, California, USA

Yoshino K, Shinoda H (2013) Visio-acoustic screen for contactless touch interface with tactile sensation. Proc IEEE World Haptics Conf 2013:419–423

Yoshino K, Hasegawa K, Shinoda H (2012) Measuring visio-tactile threshold for visio-tactile projector. Proc SICE Annu Conf 2012:1996–2000

Title: Superimposing Visual Images on Mid-Air Ultrasonic Haptic Stimulation
Authors: Hiroyuki Shinoda
Yasutoshi Makino
Publisher: Springer International Publishing
Book: Ultrasound Mid-Air Haptics for Touchless Interfaces
Print ISBN: 978-3-031-04042-9

Electronic ISBN: 978-3-031-04043-6

Copyright Year: 2022
DOI: https://doi.org/10.1007/978-3-031-04043-6_12