Wei Wang
Ke Sun
ABSTRACT
Device-free gesture tracking is an enabling HCI mechanism for small wearable devices because fingers are too big to control the GUI elements on such small screens, and it is also an important HCI mechanism for medium-to-large size mobile devices because it allows users to provide input without blocking screen view. In this paper, we propose LLAP, a device-free gesture tracking scheme that can be deployed on existing mobile devices as software, without any hardware modification. We use speakers and microphones that already exist on most mobile devices to perform device-free tracking of a hand/finger. The key idea is to use acoustic phase to get fine-grained movement direction and movement distance measurements. LLAP first extracts the sound signal reflected by the moving hand/finger after removing the background sound signals that are relatively consistent over time. LLAP then measures the phase changes of the sound signals caused by hand/finger movements and then converts the phase changes into the distance of the movement. We implemented and evaluated LLAP using commercial-off-the-shelf mobile phones. For 1-D hand movement and 2-D drawing in the air, LLAP has a tracking accuracy of 3.5 mm and 4.6 mm, respectively. Using gesture traces tracked by LLAP, we can recognize the characters and short words drawn in the air with an accuracy of 92.3% and 91.2%, respectively.
- Google project soli. https://www.google.com/atap/project-soli/.Google Scholar
- Teng Wei and Xinyu Zhang. mTrack: High-precision passive tracking using millimeter wave radios. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Chi Zhang, Josh Tabor, Jialiang Zhang, and Xinyu Zhang. Extending mobile interaction through near-field visible light sensing. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Leap Motion. https://www.leapmotion.com/.Google Scholar
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. FingerIO: Using active sonar for fine-grained finger tracking. In Proc. ACM CHI, 2016. Google ScholarDigital Library
- A Rodrìguez Valiente, A Trinidad, JR García Berrocal, C Górriz, and R Ramírez Camacho. Extended high-frequency (9--20 kHz) audiometry reference thresholds in 645 healthy subjects. International journal of audiology, 53(8):531--545, 2014.Google Scholar
- Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. Beepbeep: a high accuracy acoustic ranging system using COTS mobile devices. In Proc. ACM SenSys, 2007. Google ScholarDigital Library
- Zengbin Zhang, David Chu, Xiaomeng Chen, and Thomas Moscibroda. Swordfight: Enabling a new class of phone-to-phone action games on commodity phones. In Proc. ACM MobiSys, 2012. Google ScholarDigital Library
- Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. Soundwave: using the doppler effect to sense gestures. In Proc. ACM CHI, 2012. Google ScholarDigital Library
- Zheng Sun, Aveek Purohit, Raja Bose, and Pei Zhang. Spartacus: spatially-aware interaction for mobile devices through energy-efficient audio sensing. In Proc. ACM MobiSys, 2013. Google ScholarDigital Library
- Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. Doplink: Using the doppler effect for multi-device interaction. In Proc. ACM UbiComp, 2013. Google ScholarDigital Library
- Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. Airlink: sharing files between multiple devices using in-air gestures. In Proc. ACM UbiComp, 2014. Google ScholarDigital Library
- Sangki Yun, Yi-Chao Chen, and Lili Qiu. Turning a mobile device into a mouse in the air. In Proc. ACM MobiSys, 2015. Google ScholarDigital Library
- Nissanka B Priyantha, Anit Chakraborty, and Hari Balakrishnan. The cricket location-support system. In Proc. ACM MobiCom, 2000. Google ScholarDigital Library
- Jie Yang, Simon Sidhom, Gayathri Chandrasekaran, Tam Vu, Hongbo Liu, Nicolae Cecan, Yingying Chen, Marco Gruteser, and Richard P. Martin. Detecting driver phone use leveraging car speakers. In Proc. ACM MobiCom, 2011. Google ScholarDigital Library
- Yu-Chih Tung and Kang G Shin. Echotag: Accurate infrastructure-free indoor location tagging with smartphones. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Wenchao Huang, Yan Xiong, Xiang-Yang Li, Hao Lin, Xufei Mao, Panlong Yang, and Yunhao Liu. Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones. In Proc. IEEE INFOCOM, 2014.Google ScholarCross Ref
- Rajalakshmi Nandakumar, Shyamnath Gollakota, and Nathaniel Watson. Contactless sleep apnea detection on smartphones. In Proc. ACM MobiSys, 2015. Google ScholarDigital Library
- Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization. In Proc. ACM MobiSys, 2014. Google ScholarDigital Library
- Tong Zhu, Qiang Ma, Shanfeng Zhang, and Yunhao Liu. Context-free attacks using keyboard acoustic emanations. In Proc. ACM CCS, 2014. Google ScholarDigital Library
- Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. Snooping keystrokes with mm-level audio ranging on a single phone. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Maotian Zhang, Panlong Yang, Chang Tian, Lei Shi, Shaojie Tang, and Fu Xiao. Soundwrite: Text input on surfaces through mobile acoustic sensing. In Proc. ACM SmartObjects, 2015. Google ScholarDigital Library
- Wei Wang, Alex X. Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. Understanding and modeling of WiFi signal based human activity recognition. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Kamran Ali, Alex X. Liu, Wei Wang, and Muhammad Shahzad. Keystroke recognition using WiFi signals. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. Whole-home gesture recognition using wireless signals. In Proc. ACM MobiCom, 2013. Google ScholarDigital Library
- Fadel Adib, Zachary Kabelac, and Dina Katabi. Multi-person motion tracking via RF body reflections. In Proc. Usenix NSDI, 2015. Google ScholarDigital Library
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. Bringing gesture recognition to all devices. In Proc. Usenix NSDI, 2014. Google ScholarDigital Library
- Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. E-eyes: In-home device-free activity identification using fine-grained WiFi signatures. In Proc. ACM MobiCom, 2014. Google ScholarDigital Library
- Heba Abdelnasser, Moustafa Youssef, and Khaled A Harras. WiGest: A ubiquitous WiFi-based gesture recognition system. In Proc. IEEE INFOCOM, 2015.Google ScholarCross Ref
- Pedro Melgarejo, Xinyu Zhang, Parameswaran Ramanathan, and David Chu. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proc. ACM UbiComp, 2014. Google ScholarDigital Library
- Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. WiDraw: Enabling hands-free drawing in the air on commodity wifi devices. In Proc. ACM MobiCom, 2015. Google ScholarDigital Library
- Jue Wang, Deepak Vasisht, and Dina Katabi. RF-IDraw: virtual touch screen in the air using RF signals. In Proc. ACM SIGCOMM, 2014. Google ScholarDigital Library
- Microsoft Kinect. http://www.microsoft.com/en-us/kinectforwindows/.Google Scholar
- Robert Xiao, Chris Harrison, Karl DD Willis, Ivan Poupyrev, and Scott E Hudson. Lumitrack: low cost, high precision, high speed tracking with projected m-sequences. In Proc. ACM UIST, 2013. Google ScholarDigital Library
- Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. In-air gestures around unmodified mobile devices. In Proc. ACM UIST, 2014. Google ScholarDigital Library
- Leon Cohen. Time-frequency analysis. Prentice hall, 1995.Google Scholar
- David Tse and Pramod Viswanath. Fundamentals of wireless communication. Cambridge university press, 2005. Google ScholarDigital Library
- Norden E Huang, Zheng Shen, Steven R Long, Manli C Wu, Hsing H Shih, Quanan Zheng, Nai-Chyuan Yen, Chi Chao Tung, and Henry H Liu. The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. In Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, volume 454, pages 903--995. The Royal Society, 1998.Google ScholarCross Ref
- AK Agnihotri, B Purwar, N Jeebun, and S Agnihotri. Determination of sex by hand dimensions. The Internet Journal of Forensic Science, 1(2):12--24, 2006.Google Scholar
- MyScript. http://myscript.com/.Google Scholar
- Knowles Electronics. SPH0641LU4H-1: Digital zero-height SiSonic™micophone with multi-mode and ultrasonic support, 2014.Google Scholar
Index Terms
- Device-free gesture tracking using acoustic signals
Recommendations
Device-free gesture tracking using acoustic signals: demo
MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and NetworkingIn this demo, we present LLAP, a hand tracking system that uses ultrasound to localize the hand of the user to enable device-free gesture inputs. LLAP utilizes speakers and microphones on Commercial-Off-The-Shelf (COTS) mobile devices to play and record ...
Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback
OzCHI '15: Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human InteractionIn this paper we present the results of a user experience and preference study into the combination of gaze and gesture in a lounge-style remote-interaction, using a novel system that tracks gaze and gesture using only the Kinect device at a distance of ...
A Parallel Algorithm for Dynamic Gesture Tracking
RATFG-RTS '99: Proceedings of the International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time SystemsWe present a parallel algorithm for the extraction of human hand gestures from a video sequence that is able to exploit spatial and momentum coherence and color constraints using a fuzzy image integration approach. The dynamics of hand movement are ...
Comments