Jun Gong
Da-Yuan Huang
ABSTRACT
Interacting with media and games is a challenging user experience on smartwatches due to their small screens. We propose using lateral force feedback to enhance these experiences. When virtual objects on the smartwatch display visually collide or push the edge of the screen, we add haptic feedback so that the user also feels the impact. This addition creates the illusion of a virtual object that is physically hitting or pushing the smartwatch, from within the device itself. Using this approach, we extend virtual space and scenes into a 2D physical space. To create realistic lateral force feedback, we first examined the minimum change in force magnitude that is detectable by users in different directions and weight levels, finding an average JND of 49% across all tested conditions, with no significant effect of weight and force direction. We then developed a proof-of-concept hardware prototype called Jetto and demonstrated its unique capabilities through a set of impact-enhanced videos and games. Our preliminary user evaluations indicated the concept was welcomed and is regarded as a worthwhile addition to smartwatch output and media experiences.
Supplemental Material
- Apple Watch survey finds most customers plan to buy more bands, third-party interest increasing. 2015. Retrieved August 20, 2017 from https://9to5mac.com/2015/11/03/apple-watch-bandsurvey/Google Scholar
- Flexible SmartWatch display by SmartSound Case Inc. 2013. Retrieved August 20, 2017 from http://www.smartsoundcase.com/portfolioview/smartwatch-flexible-display-panel-for-apple-bysmartsound-case-inc/Google Scholar
- Lenovo flexible Phone. 2016. Retrieved August 15, 2017 from https://www.cnet.com/news/lenovo-showsoff-superflexible-phone-you-wear-like-a-watch/Google Scholar
- Nintendo Switch. 2017. Retrieved August 15, 2017 from https://www.nintendo.com/switch/presentation2017/Google Scholar
- Jessalyn Alvina, Shengdong Zhao, Simon T. Perrault, Maryam Azh, Thijs Roumen and Morten Fjeld. 2015. OmniVib: Towards Cross-body Spatiotemporal Vibrotactile Notifications for Mobile Phones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), 24872496. Google ScholarDigital Library
- Tomohiro Amemiya, Hideyuki Ando and Taro Maeda. 2008. Lead-me interface for a pulling sensation from hand-held devices. ACM Transactions on Applied Perception (TAP), 1--17. Google ScholarDigital Library
- Akash Badshah, Sidhant Gupta, Daniel Morris, Shwetak Patel and Desney Tan. 2012. GyroTab: a handheld device that provides reactive torque feedback. In Proceedings of the 30th Annual ACM Conference on Human Factors in Computing Systems (CHI'12), 31533156. Google ScholarDigital Library
- Karlin Bark, Jason W. Wheeler, Sunthar Premakumar and Mark R. Cutkosky. 2008. Comparison of Skin Stretch and Vibrotactile Stimulation for Feedback of Proprioceptive Information. In Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS'08), 71--78. Google ScholarDigital Library
- Olivier Bau, Ivan Poupyrev, Ali Israr and Chris Harrison. 2010. TeslaTouch: electrovibration for touch surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST'10), 283--292. Google ScholarDigital Library
- Mohamed Benali-khoudja, Moustapha Hafez, Jean-marc Alex and Abderrahmane Kheddar. 2003. Tactile interfaces: a state-of-the-art survey. Int Symposium on Robotics.Google Scholar
- Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White and Samual White. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST'10), 333--342. Google ScholarDigital Library
- Francesco Chinello, Claudio Pacchierotti, Nikos G. Tsagarakis and Domenico Prattichizzo. 2016. Design of a wearable skin stretch cutaneous device for the upper limb. In IEEE Haptics Symposium (HAPTICS'16), 1420.Google Scholar
- Artem Dementyev, Hsin-Liu Kao, Inrak Choi, Deborah Ajilo, Maggie Xu, Joseph A. Paradiso, Chris Schmandt and Sean Follmer. 2016. Rovables: Miniature On-Body Robots as Mobile Wearables. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST'16), 111--120. Google ScholarDigital Library
- Panteleimon Dimitriadis and Jason Alexander. 2014. Evaluating the effectiveness of physical shape-change for in-pocket mobile device notifications. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (CHI'14), 25892592. Google ScholarDigital Library
- Alex Dobie. 2005. First look: Lenovo's crazy Magic View smartwatch concept. Retrieved August 10, 2017 from https://www.androidcentral.com/first-looklenovos-crazy-magic-view-smartwatch-conceptGoogle Scholar
- Jun Gong, Lan Li, Daniel Vogel and Xing-Dong Yang. 2017. Cito: An Actuated Smartwatch for Extended Interactions. In Proceedings of the 35th annual ACM Conference on Human Factors in Computing Systems (CHI'17), 5331--5345. Google ScholarDigital Library
- Ashley L. Guinan, Nathaniel A. Caswell, Frank A. Drews and William R. Provancher. 2013. A video game controller with skin stretch haptic feedback. In IEEE International Conference on Consumer Electronics (ICCE'13), 456--457.Google Scholar
- Ashley L. Guinan, Markus N. Montandon, Andrew J. Doxon and William R. Provancher. 2014. {D03} kinesthetic physical interaction with a multi-handed tactile display. In IEEE Haptics Symposium (HAPTICS'14), 1--1.Google Scholar
- Liang He, Cheng Xu, Ding Xu and Ryan Brill. 2015. PneuHaptic: delivering haptic cues with a pneumatic armband. In Proceedings of the 2015 ACM International Symposium on Wearable Computers (ISWC'15), 47--48. Google ScholarDigital Library
- Fabian Hemmert. 2008. Ambient Life: Permanent Tactile Life-like Actuation as a Status Display in Mobile Phones. In Adjunct Proc. of the 21st annual ACM symposium on User Interface Software and Technology (UIST'08).Google Scholar
- Fabian Hemmert, Susann Hamann, Matthias Löwe, Anne Wohlauf and Gesche Joost. 2010. Shape-changing mobiles: tapering in one-dimensional deformational displays in mobile phones. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (TEI'10), 249--252. Google ScholarDigital Library
- Fabian Hemmert, Matthias Löwe, Anne Wohlauf and Gesche Joost. 2013. Animate mobiles: proxemically reactive posture actuation as a means of relational interaction with mobile phones. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (TEI'13), 267--270. Google ScholarDigital Library
- Da-Yuan Huang, Ruizhen Guo, Jun Gong, Jingxian Wang, John Graham, De-Nian Yang and Xing-Dong Yang. 2017. RetroShape: Leveraging Rear-Surface Shape Displays for 2.5D Interaction on Smartwatches. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST'17), 539--551. Google ScholarDigital Library
- Francis Iannacci, Erik Turnquist, Daniel Avrahami, and Shwetak N. Patel. 2011. The haptic laser: multisensation tactile feedback for at-a-distance physical space perception and interaction. In Proceedings of 29th Annual ACM Conference on Human Factors in Computing Systems (CHI '11), 2047--2050. Google ScholarDigital Library
- Alexandra Ion, Edward Jay Wang and Patrick Baudisch. 2015. Skin Drag Displays: Dragging a Physical Tactor across the User's Skin Produces a Stronger Tactile Stimulus than Vibrotactile. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), 2501--2504. Google ScholarDigital Library
- Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, 234--241. Google ScholarDigital Library
- Alexander Kron and Gunther Schmidt. 2003. Multifingered tactile feedback from virtual and remote environments. In 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS'03), 16--23. Google ScholarDigital Library
- Shinobu Kuroki, Hiroyuki Kajimoto, Hideaki Nii, Naoki Kawakami and Susumu Tachi. 2007. Proposal for tactile sense presentation that combines electrical and mechanical stimulus. In Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07), 121--126. Google ScholarDigital Library
- Jaeyeon Lee, Jaehyun Han and Geehyuk Lee. 2015. Investigating the Information Transfer Efficiency of a 3x3 Watch-back Tactile Display. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), 1229--1232. Google ScholarDigital Library
- Seungyon "Claire" Lee and Thad Starner. 2010. BuzzWear: alert perception in wearable tactile displays on the wrist. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'10), 433--442. Google ScholarDigital Library
- Seungyon Claire Lee and Thad Starner. 2009. Mobile gesture interaction using wearable tactile displays. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA'09), 3437--3442. Google ScholarDigital Library
- Joanne Leong, Patrick Parzer, Florian Perteneder, Teo Babic, Christian Rendl, Anita Vogl, Hubert Egger, Alex Olwal and Michael Haller. 2016. proCover: Sensory Augmentation of Prosthetic Limbs Using Smart Textile Covers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST'16), 335--346. Google ScholarDigital Library
- Yi-Chi Liao, Yi-Ling Chen, Jo-Yu Lo, Rong-Hao Liang, Liwei Chan and Bing-Yu Chen. 2016. EdgeVib: Effective Alphanumeric Character Output Using a Wrist-Worn Tactile Display. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST'16), 595--601. Google ScholarDigital Library
- Yi-Chi Liao, Yi-Ling Chen, Jo-Yu Lo, Rong-Hao Liang, Liwei Chan and Bing-Yu Chen. 2016. EdgeVib: Effective Alphanumeric Character Output Using a Wrist-Worn Tactile Display. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST'16), 595--601. Google ScholarDigital Library
- Yi-Chi Liao, Shun-Yao Yang, Rong-Hao Liang, Liwei Chan and Bing-Yu Chen. 2015. ThirdHand: wearing a robotic arm to experience rich force feedback. In SIGGRAPH Asia 2015 Emerging Technologies (SA'15), 1--1. Google ScholarDigital Library
- Jeff Lieberman and Cynthia Breazeal. 2007. TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning. In IEEE Transactions on Robotics, 919--926. Google ScholarDigital Library
- Pedro Lopes and Patrick Baudisch. 2013. Musclepropelled force feedback: bringing force feedback to mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'13), 2577--2580. Google ScholarDigital Library
- Pedro Lopes, Alexandra Ion and Patrick Baudisch. 2015. Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST'15), 11--19. Google ScholarDigital Library
- Kent Lyons, David Nguyen, Daniel Ashbrook and Sean White. 2012. Facet: a multi-segment wrist worn system In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST'12), 123130. Google ScholarDigital Library
- Michael Matscheko, Alois Ferscha, Andreas Riener and Manuel Lehner. 2010. Tactor placement in wrist worn wearables. In International Symposium on Wearable Computers (ISWC'10), 1--8.Google ScholarCross Ref
- Troy McDaniel, Morris Goldberg, Daniel Villanueva, Lakshmie Narayan Viswanathan and Sethuraman Panchanathan. 2011. Motor learning using a kinematicvibrotactile mapping targeting fundamental movements. In Proceedings of the 19th ACM international conference on Multimedia (MM'11), 543--552. Google ScholarDigital Library
- Troy McDaniel, Sreekar Krishna, Vineeth Balasubramanian, Dirk Colbry and Sethuraman Panchanathan. 2008. Using a haptic belt to convey nonverbal communication cues during social interactions to individuals who are blind. In Haptic Audio visual Environments and Games (HAVE'08), 13--18.Google Scholar
- Ken Nakagaki, Sean Follmer and Hiroshi Ishii. 2015. LineFORM: Actuated Curve Interfaces for Display, Interaction, and Constraint. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST'15), 333--339. Google ScholarDigital Library
- Akio Nakamura, Sou Tabata, Tomoya Ueda, Shinichiro Kiyofuji and Yoshinori Kuno. 2005. Multimodal presentation method for a dance training system. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA'05), 1685--1688. Google ScholarDigital Library
- Simon Olberding, Kian Peen Yeo, Suranga Nanayakkara and Jurgen Steimle. 2013. AugmentedForearm: exploring the design space of a display-enhanced forearm. In Proceedings of the 4th Augmented Human International Conference (AH'13), 9--12. Google ScholarDigital Library
- Stefano Papetti, Federico Fontana, Marco Civolani, Amir Berrezag and Vincent Hayward. 2010. Audiotactile display of ground properties using interactive shoes. In Haptic and Audio Interaction Design (HAID'10). 117--128. Google ScholarDigital Library
- Jerome Pasquero, Scott J. Stobbe and Noel Stonehouse. 2011. A haptic wristwatch for eyes-free interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'11), 3257--3266. Google ScholarDigital Library
- Fabrizio Pece, Juan Zarate, Velko Vechev, Nadine Besse, Olexandr Gudozhnik, Herbert Shea and Otmar Hilliges. 2017. MagTics: Flexible and Thin Form Factor Magnetic Actuators for Dynamic and Wearable Haptic Feedback. In Proceedings of the 30th annual ACM symposium on User interface software and technology (UIST'17). Google ScholarDigital Library
- Esben W Pedersen, Sriram Subramanian and Kasper Hornbæk. 2014. Is my phone alive?: a large-scale study of shape change in handheld devices using videos. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (CHI'14), 25792588. Google ScholarDigital Library
- Ioannis Politis, Stephen A Brewster and Frank Pollick. 2014. Evaluating multimodal driver displays under varying situational urgency. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (CHI'14), 4067--4076. Google ScholarDigital Library
- Ioannis Politis, Stephen Brewster and Frank Pollick. 2015. To Beep or Not to Beep?: Comparing Abstract versus Language-Based Multimodal Driver Displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), 39713980. Google ScholarDigital Library
- Helena Pongrac. 2008. Vibrotactile perception: examining the coding of vibrations and the just noticeable difference under various conditions. In Multimedia Systems, 297--307. Google ScholarDigital Library
- Wang Qi and Vincent Hayward. 2009. Biomechanically Optimized Distributed Tactile Transducer Based on Lateral Skin Deformation. In International Journal of Robotics Research, 323--335. Google ScholarDigital Library
- Zhan F. Quek, Samuel B. Schorr, Ilana Nisky, William R. Provancher and Allison M. Okamura. 2014. Sensory substitution using 3-degree-of-freedom tangential and normal skin deformation feedback. In IEEE Haptics Symposium (HAPTICS'14), 27--33.Google Scholar
- Jun Rekimoto. 2013. Traxion: a tactile interaction device with virtual force sensation. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13), 427--432. Google ScholarDigital Library
- Joseph M. Romano and Katherine J. Kuchenbecker. 2009. The AirWand: Design and characterization of a large-workspace haptic device. In IEEE International Conference on Robotics and Automation (ICRA'09), 1461--1466. Google ScholarDigital Library
- Anne Roudaut, Abhijit Karnik, Markus Löchtefeld and Sriram Subramanian. 2013. Morphees: toward high shape resolution in self-actuated flexible mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'13), 593602. Google ScholarDigital Library
- Thijs Roumen, Simon T Perrault and Shengdong Zhao. 2015. Notiring: A comparative study of notification channels for wearable interactive rings. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), 2497--2500. Google ScholarDigital Library
- Christian Schonauer, Kenichiro Fukushi, Alex Olwal, Hannes Kaufmann and Ramesh Raskar. 2012. Multimodal motion guidance: techniques for adaptive and dynamic feedback. In Proceedings of the 14th ACM international conference on Multimodal interaction (ICMI'12), 133--140. Google ScholarDigital Library
- Teddy Seyed, Xing-Dong Yang and Daniel Vogel. 2016. Doppio: A Reconfigurable Dual-Face Smartwatch for Tangible Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16), 4675--4686. Google ScholarDigital Library
- Alejandro Jarillo Silva, Omar A Domínguez Ramirez, Vicente Parra Vega and Jesus P Ordaz Oliver. 2009. Phantom omni haptic device: Kinematic and manipulability. In Electronics, Robotics and Automotive Mechanics Conference (CERMA'09), 193--198. Google ScholarDigital Library
- Daniel Spelmezan, Mareike Jacobs, Anke Hilgers and Jan Borchers. 2009. Tactile motion instructions for physical activities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'09), 2243--2252. Google ScholarDigital Library
- H. Christiaan Stronks, Daniel J. Parker, Janine Walker, Paulette Lieby and Nick Barnes. 2015. The feasibility of coin motors for use in a vibrotactile display for the blind. Artificial organs, 480--491.Google Scholar
- Koji Tsukada and Michiaki Yasumura. 2004. Activebelt: Belt-type wearable tactile display for directional navigation. In Ubiquitous Computing (UbiComp'04), 384--399.Google Scholar
- SAMSUNG GEAR VR. 2017. Retrieved August 15, 2017 from www.samsung.com/us/explore/gear-vr/.Google Scholar
- Ernst Weber. 1978. De pulsu, resorptione, audita et tactu-annotationes anatomicae et physiologicae (HE Ross, Academic Press.Google Scholar
- Cary Williams, Xing Dong Yang, Grant Partridge, Joshua Millarusiskin, Arkady Major and Pourang Irani. 2011. TZee: exploiting the lighting properties of multitouch tabletops for tangible 3d interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 1363--1372. Google ScholarDigital Library
- John Williamson, Roderick Murray-Smith and Stephen Hughes. 2007. Shoogle: excitatory multimodal interaction on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'07), 121--124. Google ScholarDigital Library
- Lining Yao, Ryuma Niiyama, Jifei Ou, Sean Follmer, Clark Della Silva and Hiroshi Ishii. 2013. PneUI: Pneumatically actuated soft composite materials for shape changing interfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST'13), 13--22. Google ScholarDigital Library
- Vibol Yem, Ryuta Okazaki, and Hiroyuki Kajimoto. 2016. FinGAR: combination of electrical and mechanical stimulation for high-fidelity tactile presentation. In ACM SIGGRAPH 2016 Emerging Technologies (SIGGRAPH '16). Google ScholarDigital Library
Index Terms
- Jetto: Using Lateral Force Feedback for Smartwatch Interactions
Recommendations
Improving Transparency in Teleoperation by Means of Cutaneous Tactile Force Feedback
A study on the role of cutaneous and kinesthetic force feedback in teleoperation is presented. Cutaneous cues provide less transparency than kinesthetic force, but they do not affect the stability of the teleoperation system. On the other hand, ...
FlexTorque, FlexTensor, and HapticEye: exoskeleton haptic interfaces for augmented interaction
AH '11: Proceedings of the 2nd Augmented Human International ConferenceIn order to realize haptic interaction (e.g., holding, pushing, and contacting the object) in virtual environment and mediated haptic communication with human beings (e.g., handshaking), the force feedback is required. Recently there has been a ...
ExoInterfaces: novel exosceleton haptic interfaces for virtual reality, augmented sport and rehabilitation
AH '10: Proceedings of the 1st Augmented Human International ConferenceWe developed novel haptic interfaces, FlexTorque and FlexTensor that enable realistic physical interaction with real and Virtual Environments. The idea behind FlexTorque is to reproduce human muscle structure, which allows us to perform dexterous ...
Comments