Hemant Bhaskar Surale
Aakar Gupta
Mark Hancock
Daniel Vogel
ABSTRACT
Complex virtual reality (VR) tasks, like 3D solid modelling, are challenging with standard input controllers. We propose exploiting the affordances and input capabilities when using a 3D-tracked multi-touch tablet in an immersive VR environment. Observations gained during semi-structured interviews with general users, and those experienced with 3D software, are used to define a set of design dimensions and guidelines. These are used to develop a vocabulary of interaction techniques to demonstrate how a tablet's precise touch input capability, physical shape, metaphorical associations, and natural compatibility with barehand mid-air input can be used in VR. For example, transforming objects with touch input, "cutting" objects by using the tablet as a physical "knife", navigating in 3D by using the tablet as a viewport, and triggering commands by interleaving bare-hand input around the tablet. Key aspects of the vocabulary are evaluated with users, with results validating the approach.
Supplemental Material
- Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 185. Google Scholar
Digital Library
- Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George W Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the SIGCHI conference on Human factors in computing systems. 5643--5654. Google ScholarDigital Library
- Ilhan Aslan, Ida Buchwald, Philipp Koytek, and Elisabeth André. 2016. Pen+ Mid-Air: An Exploration of Mid-Air Gestures to Complement Pen Input on Tablets. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. ACM, 1. Google ScholarDigital Library
- Rob Aspin and Kien Hoang Le. 2007. Augmenting the CAVE: An initial study into close focused, inward looking, exploration in IPT systems. In Distributed Simulation and Real-Time Applications, 2007. DS-RT 2007. 11th IEEE International Symposium. IEEE, 217--224. Google ScholarDigital Library
- Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek. 2016. Normaltouch and texturetouch: High-fidelity 3d haptic shape rendering on handheld virtual reality controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 717--728. Google ScholarDigital Library
- Hrvoje Benko, Edward W Ishak, and Steven Feiner. 2004. Collaborative mixed reality visualization of an archaeological excavation. In Proceedings of the 3rd IEEE/ACM international Symposium on Mixed and Augmented Reality. IEEE Computer Society, 132--140. Google ScholarDigital Library
- Hrvoje Benko, Edward W Ishak, and Steven Feiner. 2005. Crossdimensional gestural interaction techniques for hybrid immersive environments. In Virtual Reality, 2005. Proceedings. VR 2005. IEEE. IEEE, 209--216. Google ScholarDigital Library
- Mark Billinghurst, Sisinio Baldis, Lydia Matheson, and Mark Philips. 1997. 3D palette: a virtual reality content creation tool. In Proceedings of the ACM symposium on Virtual reality software and technology. ACM, 155--156. Google ScholarDigital Library
- Mark Billinghurst, Tham Piumsomboon, and Huidong Bai. 2014. Hands in space: Gesture interaction with augmented-reality interfaces. IEEE computer graphics and applications 34, 1 (2014), 77--80.Google Scholar
- Alexander Bornik, Reinhard Beichel, Ernst Kruijff, Bernhard Reitinger, and Dieter Schmalstieg. 2006. A hybrid user interface for manipulation of volumetric medical data. In 3D User Interfaces, 2006. 3DUI 2006. IEEE Symposium on. IEEE, 29--36. Google Scholar
- Doug A Bowman, Mehdi Setareh, Márcio Serolli Pinho, Ndiwalana Ali, Alex Kalita, Yunha Lee, John Lucas, Matthew Gracey, Malini Kothapalli, Qinwei Zhu, et al. 2003. Virtual-SAP: an immersive tool for visualizing the response of building structures to environmental conditions. In Virtual Reality, 2003. Proceedings. IEEE. IEEE, 243--250. Google ScholarDigital Library
- Jeff Butterworth, Andrew Davidson, Stephen Hench, and Marc T Olano. 1992. 3DM: A three dimensional modeler using a head-mounted display. In Proceedings of the 1992 symposium on Interactive 3D graphics. ACM, 135--138. Google ScholarDigital Library
- William Buxton and Brad Myers. 1986. A study in two-handed input. In ACM SIGCHI Bulletin, Vol. 17. ACM, 321--326. Google ScholarDigital Library
- Li-Chieh Chen, Yun-Maw Cheng, Po-Ying Chu, and Frode Eika Sandnes. 2016. The Common Characteristics of User-Defined and MidAir Gestures for Rotating 3D Digital Contents. In International Conference on Universal Access in Human-Computer Interaction. Springer, 15--22.Google Scholar
- Michael Chen, S Joy Mountford, and Abigail Sellen. 1988. A study in interactive 3-D rotation using 2-D control devices. ACM SIGGRAPH Computer Graphics 22, 4 (1988), 121--129. Google ScholarDigital Library
- Inrak Choi, Eyal Ofek, Hrvoje Benko, Mike Sinclair, and Christian Holz. 2018. CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 654. Google ScholarDigital Library
- Bruno R De Araùjo, Géry Casiez, and Joaquim A Jorge. 2012. Mockup builder: direct 3d modeling on and above the surface in a continuous interaction space. In Proceedings of Graphics Interface 2012. Canadian Information Processing Society, 173--180. Google ScholarDigital Library
- Barrett Ens, Juan David Hincapié-Ramos, and Pourang Irani. 2014. Ethereal planes: a design framework for 2D information space in 3D mixed reality environments. In Proceedings of the 2nd ACM symposium on Spatial user interaction. ACM, 2--12. Google ScholarDigital Library
- Ajoy S Fernandes and Steven K Feiner. 2016. Combating VR sickness through subtle dynamic field-of-view modification. In 3D User Interfaces (3DUI), 2016 IEEE Symposium on. IEEE, 201--210.Google Scholar
- James D Foley, Victor L Wallace, and Peggy Chan. 1984. The human factors of computer graphics interaction techniques. IEEE computer Graphics and Applications 4, 11 (1984), 13--48. Google ScholarDigital Library
- Cédric Foucault, Manfred Micaux, David Bonnet, and Michel Beaudouin-Lafon. 2014. SPad: a bimanual interaction technique for productivity applications on multi-touch tablets. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM, 1879--1884. Google ScholarDigital Library
- Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and Per Ola Kristensson. 2018. Text Entry in Immersive Head-Mounted Display-Based Virtual Reality Using Standard Keyboards. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (2018), 159-- 166.Google Scholar
- Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of motor behavior 19, 4 (1987), 486--517.Google ScholarCross Ref
- Genesis Virtual Guide. 2018. Genesis Virtual Guide. (2018). https: //www.youtube.com/watch?v=hDxOX8LYKHs.Google Scholar
- Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 627--636. Google ScholarDigital Library
- Otmar Hilliges, Shahram Izadi, Andrew D Wilson, Steve Hodges, Armando Garcia-Mendoza, and Andreas Butz. 2009. Interactions in the air: adding further depth to interactive tabletops. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. ACM, 139--148. Google ScholarDigital Library
- Ken Hinckley, Michel Pahud, Hrvoje Benko, Pourang Irani, Francois Guimbretiere, Marcel Gavriliu, Xiang'Anthony' Chen, Fabrice Matulic, William Buxton, and Andrew Wilson. 2014. Sensing techniques for tablet+ stylus interaction. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 605--614. Google ScholarDigital Library
- Ken Hinckley, Randy Pausch, Dennis Proffitt, and Neal F Kassell. 1998. Two-handed virtual manipulation. ACM Transactions on ComputerHuman Interaction (TOCHI) 5, 3 (1998), 260--302. Google ScholarDigital Library
- Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen+ touch= new tools. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 27--36. Google ScholarDigital Library
- HTC. 2017. "Introducing the Logitech BRIDGE SDK". (2 November 2017). Retrieved Mar 31, 2018 from https://blog.vive.com/us/2017/11/ 02/introducing-the-logitech-bridge-sdk/.Google Scholar
- Hessam Jahani and Manolya Kavakli. 2017. Exploring a user-defined gesture vocabulary for descriptive mid-air interactions. Cognition, Technology & Work 20 (2017), 11--22. Google ScholarDigital Library
- Daniel F Keefe, Robert C Zeleznik, and David H Laidlaw. 2007. Drawing on air: Input techniques for controlled 3D line illustration. IEEE Transactions on Visualization and Computer Graphics 13, 5 (2007), 1067-- 1081. Google ScholarDigital Library
- Ji-Sun Kim, Denis Gracanin, Kresimir Matkovic, and Francis Quek. 2008. Finger Walking in Place (FWIP): A traveling technique in virtual environments. In International Symposium on Smart Graphics. Springer, 58--69. Google ScholarDigital Library
- Robert W Lindeman, John L Sibert, and James K Hahn. 1999. Handheld windows: towards effective 2D interaction in immersive virtual environments. In Virtual Reality, 1999. Proceedings., IEEE. IEEE, 205-- 212. Google ScholarDigital Library
- Robert W Lindeman, John L Sibert, and James K Hahn. 1999. Towards usable VR: an empirical study of user interfaces for immersive virtual environments. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 64--71. Google ScholarDigital Library
- Nicolai Marquardt, Ricardo Jota, Saul Greenberg, and Joaquim A Jorge. 2011. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In IFIP Conference on Human-Computer Interaction. Springer, 461--476. Google ScholarDigital Library
- John C McClelland, Robert J Teather, and Audrey Girouard. 2017. HaptoBend: shape-changing passive haptic feedback in virtual reality. In Proceedings of the 5th Symposium on Spatial User Interaction. ACM, 82--90. Google ScholarDigital Library
- Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A dose of reality: Overcoming usability challenges in vr head-mounted displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2143--2152. Google ScholarDigital Library
- Daniel Mendes, Filipe Relvas, Alfredo Ferreira, and Joaquim Jorge. 2016. The benefits of DOF separation in mid-air 3D object manipulation. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. ACM, 261--268. Google ScholarDigital Library
- Minecraft. 2018. Minecraft Wiki. (2018). https://minecraft.gamepedia. com/Block.Google Scholar
- Oculus. 2018. Oculus Locomotion. (2018). https://developer.oculus. com/design/latest/concepts/bp-locomotion.Google Scholar
- Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy Cockburn. 2013. User-defined gestures for augmented reality. In IFIP Conference on Human-Computer Interaction. Springer, 282--299. Google ScholarDigital Library
- Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The go-go interaction technique: non-linear mapping for direct manipulation in VR. In Proceedings of the 9th annual ACM symposium on User interface software and technology. ACM, 79--80. Google ScholarDigital Library
- Ivan Poupyrev, Numada Tomokazu, and Suzanne Weghorst. 1998. Virtual Notepad: handwriting in immersive VR. In Virtual Reality Annual International Symposium, 1998. Proceedings., IEEE 1998. IEEE, 126--132. Google ScholarDigital Library
- Gerhard Reitmayr and Dieter Schmalstieg. 2001. Mobile collaborative augmented reality. In Augmented Reality, 2001. Proceedings. IEEE and ACM International Symposium on. IEEE, 114--123.Google ScholarCross Ref
- Élisabeth Rousset, François Bérard, and Michaël Ortega. 2014. Twofinger 3D rotations for novice users: surjective and integral interactions. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces. ACM, 217--224. Google ScholarDigital Library
- Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-defined motion gestures for mobile interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 197--206. Google ScholarDigital Library
- Markus Sareika and Dieter Schmalstieg. 2010. Bimanual handheld mixed reality interfaces for urban planning. In Proceedings of the International Conference on Advanced Visual Interfaces. ACM, 189--196. Google ScholarDigital Library
- Peng Song, Wooi Boon Goh, William Hutama, Chi-Wing Fu, and Xiaopei Liu. 2012. A handle bar metaphor for virtual object manipulation with mid-air interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1297--1306. Google ScholarDigital Library
- Peng Song, Wooi Boon Goh, William Hutama, Chi-Wing Fu, and Xiaopei Liu. 2012. A handle bar metaphor for virtual object manipulation with mid-air interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1297--1306. Google ScholarDigital Library
- Richard Stoakley, Matthew J Conway, and Randy Pausch. 1995. Virtual reality on a WIM: interactive worlds in miniature. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 265--272. Google ScholarDigital Library
- Evan Strasnick, Christian Holz, Eyal Ofek, Mike Sinclair, and Hrvoje Benko. 2018. Haptic Links: Bimanual Haptics for Virtual Reality Using Variable Stiffness Actuation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 644. Google ScholarDigital Library
- Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2017. Experimental Analysis of Mode Switching Techniques in Touch-based User Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3267--3280. Google ScholarDigital Library
- Ivan E Sutherland. 1965. The ultimate display. Multimedia: From Wagner to virtual reality (1965), 506--508.Google Scholar
- Zsolt Szalavári and Michael Gervautz. 1997. The Personal Interaction Panel - a Two-Handed Interface for Augmented Reality. Comput. Graph. Forum 16 (1997), 335--346.Google ScholarCross Ref
- Anthony Talvas. 2014. Bimanual haptic interaction with virtual environments. Ph.D. Dissertation. INSA de Rennes.Google Scholar
- HTC VIVE. 2018. HTC Custom Tracker. (2018). https: //dl.vive.com/Tracker/Guideline/HTC_Vive_Tracker_Developer_ Guidelines_v1.3.pdf.Google Scholar
- Julie Wagner, Stéphane Huot, and Wendy Mackay. 2012. BiTouch and BiPad: designing bimanual interaction for hand-held tablets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2317--2326. Google ScholarDigital Library
- Jia Wang and Robert Lindeman. 2014. Coordinated 3D interaction in tablet-and HMD-based hybrid virtual environments. In Proceedings of the 2nd ACM symposium on Spatial user interaction. ACM, 70--79. Google ScholarDigital Library
- Eric Whitmire, Hrvoje Benko, Christian Holz, Eyal Ofek, and Mike Sinclair. 2018. Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 86. Google ScholarDigital Library
- Jacob O Wobbrock, Meredith Ringel Morris, and Andrew D Wilson. 2009. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1083--1092. Google ScholarDigital Library
- Siju Wu, Amine Chellali, Samir Otmane, and Guillaume Moreau. 2015. TouchSketch: a touch-based interface for 3D object manipulation and editing. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology. ACM, 59--68. Google ScholarDigital Library
- Yukang Yan, Chun Yu, Xiaojuan Ma, Shuai Huang, Hasan Iqbal, and Yuanchun Shi. 2018. Eyes-Free Target Acquisition in Interaction Space around the Body for Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 42. Google ScholarDigital Library
- Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, and Shumin Zhai. 2018. M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 249. Google ScholarDigital Library
Index Terms
- TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality
Recommendations
A study on touch & hover based interaction for zooming
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing SystemsProximity is a useful medium for interaction with high interactive digital contents. It can be used in different contexts such as for navigation through depth in 3D space in zoomable interfaces. In this paper, we propose hover-based zoom interaction as ...
Bezel swipe: conflict-free scrolling and multiple selection on mobile touch screen devices
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsZooming user interfaces are increasingly popular on mobile devices with touch screens. Swiping and pinching finger gestures anywhere on the screen manipulate the displayed portion of a page, and taps open objects within the page. This makes navigation ...
Preliminary Investigation of Text Entry Method with Haptic Feedback from Real Object Surfaces Estimated Using Hand Tracking on HMD
Asian CHI '21: Proceedings of the Asian CHI Symposium 2021In virtual reality systems, users enter text by selecting virtual keys with the fingers. A virtual keyboard is displayed in mid-air and thus does not provide haptic feedback. To address this problem, we present a text entry method that uses the surfaces ...
Comments