Rajalakshmi Nandakumar
Vikram Iyer
Desney Tan
Shyamnath Gollakota
ABSTRACT
We present fingerIO, a novel fine-grained finger tracking solution for around-device interaction. FingerIO does not require instrumenting the finger with sensors and works even in the presence of occlusions between the finger and the device. We achieve this by transforming the device into an active sonar system that transmits inaudible sound signals and tracks the echoes of the finger at its microphones. To achieve sub-centimeter level tracking accuracies, we present an innovative approach that use a modulation technique commonly used in wireless communication called Orthogonal Frequency Division Multiplexing (OFDM). Our evaluation shows that fingerIO can achieve 2-D finger tracking with an average accuracy of 8 mm using the in-built microphones and speaker of a Samsung Galaxy S4. It also tracks subtle finger motion around the device, even when the phone is in the pocket. Finally, we prototype a smart watch form-factor fingerIO device and show that it can extend the interaction space to a 0.5×0.25 m2 region on either side of the device and work even when it is fully occluded from the finger.
Supplemental Material
- Adafruit. https://www.adafruit.com/products/1063.Google Scholar
- Apple Watch - Guided Tour: Phone Calls. https://www.youtube.com/watch?v=Zj5KisMVv8.Google Scholar
- Chirp Microsystems. http://www.chirpmicro.com/technology.html.Google Scholar
- A MimioTeach Interaction Whiteboard. http://www.mimio.com/en-NA/Products/MimioTeach-Interactive-Whiteboard.aspx.Google Scholar
- Adib, F., Kabelac, Z., Katabi, D., and Miller, R. C. 3D Tracking via Body Radio Reflections. NSDI 2014, 317--329. Google ScholarDigital Library
- Aumi, M. T. I., Gupta, S., Goel, M., Larson, E., and Patel, S. DopLink: Using the Doppler Effect for Multi-device Interaction. UbiComp 2013, 583--586. Google ScholarDigital Library
- Boleskei, H. Principles of MIMO-OFDM wireless systems. 2004.Google Scholar
- Braun, A., Krepp, S., and Kuijper, A. Acoustic Tracking of Hand Activities on Surfaces. WOAR 2015, 1--5. Google ScholarDigital Library
- Butler, A., Izadi, S., and Hodges, S. SideSight: Multi-"Touch" Interaction Around Small Devices. UIST 2008, 201--204. Google ScholarDigital Library
- Chan, L., Liang, R.-H., Tsai, M.-C., Cheng, K.-Y., Su, C.-H., Chen, M. Y., Cheng, W.-H., and Chen, B.-Y. FingerPad: Private and Subtle Interaction Using Fingertips. UIST 2013, 255--260. Google ScholarDigital Library
- Chen, K.-Y., Ashbrook, D., Goel, M., Lee, S.-H., and Patel, S. AirLink: Sharing Files Between Multiple Devices Using In-air Gestures. UbiComp 2014, 565--569. Google ScholarDigital Library
- Chen, K.-Y., Lyons, K., White, S., and Patel, S. uTrack: 3D Input Using Two Magnetic Sensors. UIST 2013, 237--244. Google ScholarDigital Library
- Goel, M., Lee, B., Islam Aumi, M. T., Patel, S., Borriello, G., Hibino, S., and Begole, B. Surface Link: Using Inertial and Acoustic Sensing to Enable Multi-device Interaction on a Surface. CHI 2014, 1387--1396. Google ScholarDigital Library
- Google. Project Soli. https://www.youtube.com/watch?v=_Zj5KisMVv8.Google Scholar
- Gupta, S., Morris, D., Patel, S., and Tan, D. SoundWave: Using the Doppler Effect to Sense Gestures. CHI 2012, 1911--1914. Google ScholarDigital Library
- Heiskala, J., and Terry, J. OFDM Wireless LANs: A Theoretical and Practical Guide. Sams publishing, 2001. Google ScholarDigital Library
- Huang, W., Xiong, Y., Li, X.-Y., Lin, H., Mao, X., Yang, P., and Liu, Y. Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones. INFOCOM 2014, 370--278.Google ScholarCross Ref
- Kellogg, B., Talla, V., and Gollakota, S. Bringing Gesture Recognition to All Devices. NSDI 2014, 303--316. Google ScholarDigital Library
- Khyam, M., Alam, M., Lambert, A., Benson, C., and Pickering, M. High precision multiple ultrasonic transducer positioning using a robust optimization approach. ISSPIT 2013, 192--197.Google ScholarCross Ref
- Khyam, M., Alam, M., and Pickering, M. OFDM based low-complexity time of arrival estimation in active sonar. OCEANS 2014, 1--5.Google Scholar
- Kienzle, W., and Hinckley, K. LightRing: Always-available 2D Input on Any Surface. UIST 2014, 157--160. Google ScholarDigital Library
- Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. UIST 2012, 167--176. Google ScholarDigital Library
- Kratz, S., and Rohs, M. HoverFlow: Expanding the Design Space of Around-device Interaction. MobileHCI 2009, 1--8. Google ScholarDigital Library
- Liu, J., Wang, Y., Kar, G., Chen, Y., Yang, J., and Gruteser, M. Snooping Keystrokes with Mm-level Audio Ranging on a Single Phone. MobiCom 2015, 142--154. Google ScholarDigital Library
- MacNeish. The Intersections of Two Conic Sections with a Common Focus. The American Mathematical Monthly 28, 6/7, 260--262.Google Scholar
- Nandakumar, R., Chinatalapudi, K., Padmanaban, V., and Venkatesan, R. Dhwani: Secure Peer-to-Peer Acoustic NFC. Sigcomm 2013 2013. Google ScholarDigital Library
- Nandakumar, R., Gollakota, S., and Watson, N. Contactless Sleep Apnea Detection on Smartphones. Mobisys 2015, 45--57. Google ScholarDigital Library
- Ogata, M., Sugiura, Y., Osawa, H., and Imai, M. iRing: Intelligent Ring Using Infrared Reflection. UIST 2012, 131--136. Google ScholarDigital Library
- Priyantha, N. B., Chakraborty, A., and Balakrishnan, H. The Cricket Location-support System. Mobicom 2000, 32--43. Google ScholarDigital Library
- Proakis, J., and Salehi, M. Digital Communications. McGraw-hill, 2007.Google Scholar
- Przybyla, R., Tang, H.-Y., Guedes, A., Shelton, S., Horsley, D., and Boser, B. 3D Ultrasonic Rangefinder on a Chip. IEEE Journal of Solid-State Circuits 2015, 320--334.Google Scholar
- Pu, Q., Gupta, S., Gollakota, S., and Patel, S. Whole-home Gesture Recognition Using Wireless Signals. Mobicom 2013, 27--38. Google ScholarDigital Library
- Reju, V., Khong, A., and Sulaiman, A. Localization of Taps on Solid Surfaces for Human-Computer Touch Interfaces. IEEE Trans. on Multimedia 2013, 1365--1376. Google ScholarDigital Library
- Saponas, T. S., Harrison, C., and Benko, H. PocketTouch: Through-fabric Capacitive Touch Input. UIST 2011, 303--308. Google ScholarDigital Library
- Song, J., Soros, G., Pece, F., Fanello, S. R., Izadi, S., Keskin, C., and Hilliges, O. In-air Gestures Around Unmodified Mobile Devices. UIST 2014, 319--329. Google ScholarDigital Library
- Sun, L., Sen, S., Koutsonikolas, D., and Kim, K.-H. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. Mobicom 2015, 77--89. Google ScholarDigital Library
- Sun, Z., Purohit, A., Bose, R., and Zhang, P. Spartacus: Spatially-aware Interaction for Mobile Devices Through Energy-efficient Audio Sensing. MobiSys 2013, 263--276. Google ScholarDigital Library
- Wang, J., Zhao, K., Zhang, X., and Peng, C. Ubiquitous Keyboard for Small Mobile Devices: Harnessing Multipath Fading for Fine-grained Keystroke Localization. MobiSys 2014, 14--27. Google ScholarDigital Library
- Xiao, R., Lew, G., Marsanico, J., Hariharan, D., Hudson, S., and Harrison, C. Toffee: Enabling Ad Hoc, Around-device Interaction with Acoustic Time-of-arrival Correlation. MobileHCI 2014, 67--76. Google ScholarDigital Library
- Yang, X.-D., Grossman, T., Wigdor, D., and Fitzmaurice, G. Magic Finger: Always-available Input Through Finger Instrumentation. UIST 2012, 147--156. Google ScholarDigital Library
- Yang, X.-D., Hasan, K., Bruce, N., and Irani, P. Surround-see: Enabling Peripheral Vision on Smartphones During Active Use. UIST 2013, 291--300. Google ScholarDigital Library
- Yun, S., Chen, Y.-C., and Qiu, L. Turning a Mobile Device into a Mouse in the Air. Mobisys 2015, 15--29. Google ScholarDigital Library
- Zhao, C., Chen, K.-Y., Aumi, M. T. I., Patel, S., and Reynolds, M. S. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using Actual GSM Signal. UIST 2014, 527--534. Google ScholarDigital Library
Index Terms
- FingerIO: Using Active Sonar for Fine-Grained Finger Tracking
Recommendations
Finger identification and hand gesture recognition techniques for natural user interface
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human InteractionThe natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...
MagiMusic: using embedded compass (magnetic) sensor for touch-less gesture based interaction with digital music instruments in mobile devices
TEI '11: Proceedings of the fifth international conference on Tangible, embedded, and embodied interactionPlaying musical instruments such as chordophones, percussions and keyboard types accompany with harmonic interaction of player's hand with the instruments. In this work, we present a novel approach that enables the user to imitate the music playing ...
MagiTact: interaction with mobile devices based on compass (magnetic) sensor
IUI '10: Proceedings of the 15th international conference on Intelligent user interfacesIn this work, we present a new technique for efficient use of 3D space around a mobile device for interaction with the device. Around Device Interaction (ADI) enables extending interaction space of small mobile and tangible devices beyond their physical ...
Comments