Jaime Lien
Skip Abstract Section
Skip Supplemental Material SectionAbstract
This paper presents Soli, a new, robust, high-resolution, low-power, miniature gesture sensing technology for human-computer interaction based on millimeter-wave radar. We describe a new approach to developing a radar-based sensor optimized for human-computer interaction, building the sensor architecture from the ground up with the inclusion of radar design principles, high temporal resolution gesture tracking, a hardware abstraction layer (HAL), a solid-state radar chip and system architecture, interaction models and gesture vocabularies, and gesture recognition. We demonstrate that Soli can be used for robust gesture recognition and can track gestures with sub-millimeter accuracy, running at over 10,000 frames per second on embedded hardware.
Supplemental Material
- Axelsson, S. R. 2004. Noise radar using random phase and frequency modulation. IEEE Transactions on Geoscience and Remote Sensing 42, 11, 2370--2384.Google Scholar
Cross Ref
- Azevedo, S., and McEwan, T. 1996. Micropower impulse radar. Science and Technology Review, 17--29.Google Scholar
- Barnes, S. B. 1997. Douglas Carl Engelbart: Developing the underlying concepts for contemporary computing. IEEE Annals of the History of Computing 19, 3, 16--26. Google ScholarDigital Library
- Baum, C. E., Rothwell, E. J., Chen, K.-M., and Nyquist, D. P. 1991. The singularity expansion method and its application to target identification. Proceedings of the IEEE 79, 10, 1481--1492.Google ScholarCross Ref
- Benezeth, Y., Jodoin, P. M., Emile, B., Laurent, H., and Rosenberger, C. 2008. Review and evaluation of commonly-implemented background subtraction algorithms. In ICPR 2008.Google Scholar
- Bishop, C. M. 2006. Pattern recognition. Machine Learning. Google ScholarDigital Library
- Bolt, R. a. 1980. Put-that-there. SIGGRAPH '80, 262--270. Google ScholarDigital Library
- Breiman, L. 2001. Random forests. Machine Learning 45, 1, 5--32. Google ScholarDigital Library
- Brookner, E. 1985. Phased-array radars. Scientific American 252, 2, 94--102.Google ScholarCross Ref
- Brown, L. 1999. A Radar History of World War II: Technical and Military Imperatives. Institute of Physics Publishing.Google Scholar
- Brunnbauer, M., Meyer, T., Ofner, G., Mueller, K., and Hagen, R. 2008. Embedded wafer level ball grid array (eWLB). In IEMT 2008, IEEE, 1--6.Google Scholar
- Card, S., Moran, T., and Newell, A. 1983. The Psychology of Human-Computer Interaction. L. Erlbaum Associates. Google ScholarDigital Library
- Chan, L., Chen, C.-h. H. Y.-l., and Yang, S. 2015. Cyclops: Wearable and single-piece full-body gesture input devices. In CHI 2015, ACM, 3001--3009. Google ScholarDigital Library
- Comscore Inc. 2014. The US mobile app report. Tech. rep.Google Scholar
- Cooperstock, J. R., Fels, S. S., Buxton, W., and Smith, K. C. 1997. Reactive environments. Communications of the ACM 40, 9 (Sep), 65--73. Google ScholarDigital Library
- Dardas, N. H., and Georganas, N. D. 2011. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on Instrumentation and Measurement 60, 11, 3592--3607.Google ScholarCross Ref
- Dietz, P., and Leigh, D. 2001. Diamond Touch. In UIST '01, 219.Google Scholar
- Dorfmuller-Ulhaas, K., and Schmalstieg, D. 2001. Finger tracking for interaction in augmented environments. ISMAR 2001, 55--64. Google ScholarDigital Library
- Duffner, S., Berlemont, S., Lefebvre, G., and Garcia, C. 2014. 3D gesture classification with convolutional neural networks. In ICASSP 2014, IEEE, 5432--5436.Google Scholar
- FCC, 2016. FCC online table of frequency allocations.Google Scholar
- Geng, S., Kivinen, J., Zhao, X., and Vainikainen, P. 2009. Millimeter-wave propagation channel characterization for short-range wireless communications. IEEE Transactions on Vehicular Technology 58, 1 (Jan), 3--13.Google Scholar
- Gillian, N., Knapp, R. B., and O'Modhrain, S. 2011. Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping. In Proceedings of the 2011 International Conference on New Interfaces for Musical Expression (NIME 11), Oslo, Norway.Google Scholar
- Gillian, N. 2011. Gesture Recognition for Musician Computer Interaction. PhD thesis, School of Music and Sonic Arts, Queen's University Belfast.Google Scholar
- Gustafson, S., Holz, C., and Baudisch, P. 2011. Imaginary phone: Learning imaginary interfaces by transferring spatial memory from a familiar device. UIST '11, 283--292. Google ScholarDigital Library
- Guyon, I., and Elisseeff, A. 2003. An introduction to variable and feature selection. The Journal of Machine Learning Research 3. Google ScholarDigital Library
- Hansen, C. J. 2011. WiGiG: Multi-gigabit wireless communications in the 60 GHz band. IEEE Wireless Communications 18, 6, 6--7.Google ScholarCross Ref
- Harrison, C., Tan, D., and Morris, D. 2010. Skinput: Appropriating the body as an input surface. CHI 2010, 453--462. Google ScholarDigital Library
- Holleis, P., Schmidt, A., Paasovaara, S., Puikkonen, A., and Häkkilä, J. 2008. Evaluating capacitive touch input on clothes. In MobileHCI '08, ACM Press, New York, New York, USA, 81--90. Google ScholarDigital Library
- Holz, C., and Wilson, A. 2011. Data miming: Inferring spatial object descriptions from human gesture. CHI 2011, 811--820. Google ScholarDigital Library
- Hussain, M. G. 1998. Ultra-wideband impulse radar -- an overview of the principles. IEEE Aerospace and Electronic Systems Magazine 13, 9, 9--14.Google ScholarCross Ref
- Ji, S., Xu, W., Yang, M., and Yu, K. 2013. 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1, 221--231. Google ScholarDigital Library
- Junker, H., Amft, O., Lukowicz, P., and Tröster, G. 2008. Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition 41, 6, 2010--2024. Google ScholarDigital Library
- Keller, J. B. 1962. Geometrical theory of diffraction. JOSA 52, 2, 116--130.Google ScholarCross Ref
- Kellogg, B., Talla, V., and Gollakota, S. 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), 303--316. Google ScholarDigital Library
- Keskin, C., Kiraç, F., Kara, Y. E., and Akarun, L. 2013. Real time hand pose estimation using depth sensors. In Consumer Depth Cameras for Computer Vision. Springer, 119--137.Google Scholar
- Khoshelham, K., and Elberink, S. O. 2012. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors 12, 2, 1437--1454.Google ScholarCross Ref
- Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. 2012. Digits. UIST '12, 167--176. Google ScholarDigital Library
- Knott, E. F. 2012. Radar Cross Section Measurements. Springer Science & Business Media.Google Scholar
- Kurakin, A., Zhang, Z., and Liu, Z. 2012. A real time system for dynamic hand gesture recognition with a depth sensor. In 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), 1975--1979.Google Scholar
- Lee, S., and Buxton, W. 1985. A multi-touch three dimensional touch-sensitive tablet. In CHI'85, 21--25. Google ScholarDigital Library
- Levanon, N. 2000. Multifrequency complementary phase-coded radar signal. IEE Proceedings - Radar, Sonar and Navigation 147, 6, 276--284.Google ScholarCross Ref
- LoPresti, L., LaCascia, M., Sclaroff, S., and Camps, O. 2015. Gesture modeling by Hanklet-based hidden Markov model. In Computer Vision -- ACCV 2014, D. Cremers, I. Reid, H. Saito, and M.-H. Yang, Eds., vol. 9005 of Lecture Notes in Computer Science. Springer International Publishing, 529--546.Google Scholar
- Melgarejo, P., Zhang, X., Ramanathan, P., and Chu, D. 2014. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 541--551. Google ScholarDigital Library
- Molchanov, P., Gupta, S., Kim, K., and Kautz, J. 2015. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1--7.Google Scholar
- Molchanov, P., Gupta, S., Kim, K., and Pulli, K. 2015. Multi-sensor system for driver's hand-gesture recognition. In IEEE Conference on Automatic Face and Gesture Recognition.Google Scholar
- Molchanov, P., Gupta, S., Kim, K., and Pulli, K. 2015. Short-range FMCW monopulse radar for hand-gesture sensing. In IEEE Radar Conference (RadarCon 2015), IEEE, 1491--1496.Google Scholar
- Nasr, I., Karagozler, E., Poupyrev, I., and Trotta, S. 2015. A highly integrated 60-GHz 6-channel transceiver chip in 0.35 um SiGe technology for smart sensing and short-range communications. In IEEE CSICS 2015, IEEE, 1--4.Google Scholar
- Nasr, I., Jungmaier, R., Baheti, A., Noppeney, D., Bal, J. S., Wojnowski, M., Karagozler, E., Raja, H., Lien, J., Poupyrev, I., and Trotta, S. 2016. A highly integrated 60 GHz 6-channel transceiver with antenna in package for smart sensing and short range communications. submitted to IEEE Journal of Solid State Circuits.Google Scholar
- Nathanson, F. E., Reilly, J. P., and Cohen, M. N. 1991. Radar Design Principles -- Signal Processing and the Environment, vol. 91.Google Scholar
- Novelda. Novelda Xethru NVA620x IC.Google Scholar
- Otero, M. 2005. Application of a continuous wave radar for human gait recognition. Proceedings of SPIE 5809, 538--548.Google ScholarCross Ref
- Paradiso, J., Abler, C., Hsiao, K.-y., and Reynolds, M. 1997. The Magic Carpet: Physical sensing for immersive environments. In CHI'97 Extended Abstracts on Human Factors in Computing Systems, ACM, 277--278. Google ScholarDigital Library
- Paradiso, J. A. 1999. The Brain Opera technology: New instruments and gestural sensors for musical interaction and performance. Journal of New Music Research 28, 2, 130--149.Google Scholar
- Park, J.-I., and Kim, K.-T. 2010. A comparative study on ISAR imaging algorithms for radar target identification. Progress In Electromagnetics Research 108, 155--175.Google ScholarCross Ref
- Potter, L. C., Chiang, D.-M., Carriere, R., and Gerry, M. J. 1995. A GTD-based parametric model for radar scattering. IEEE Transactions on Antennas and Propagation 43, 10, 1058--1067.Google ScholarCross Ref
- Pu, Q., Gupta, S., Gollakota, S., and Patel, S. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, ACM, 27--38. Google ScholarDigital Library
- Rahman, T., Adams, A. T., Zhang, M., and Choudhury, T. 2015. DoppleSleep: A contactless unobtrusive sleep sensing system using short-range doppler radar. In Ubicomp '15, ACM. Google ScholarDigital Library
- Ralston, T. S., Charvat, G. L., and Peabody, J. E. 2010. Real-time through-wall imaging using an ultrawideband MIMO phased array radar system. In IEEE International Symposium on Phased Array Systems and Technology (ARRAY), IEEE.Google Scholar
- Rekimoto, J. 2001. GestureWrist and GesturePad: Unobtrusive wearable interaction devices. ISWC 2001, 21--27. Google ScholarDigital Library
- Richards, M. A., Scheer, J., Holm, W. A., et al. 2010. Principles of modern radar: basic principles. SciTech Pub.Google Scholar
- Romero, J., Kjellström, H., Ek, C. H., and Kragic, D. 2013. Non-parametric hand pose estimation with object context. Image and Vision Computing 31, 8, 555--564. Google ScholarDigital Library
- Russell, D., Streitz, N., and Winograd, T. 2005. Building disappearing computers. Communications of the ACM 48, 3, 42--48. Google ScholarDigital Library
- Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R., Turner, J., and Landay, J. A. 2009. Enabling always-available input with muscle-computer interfaces. UIST 2009, 167--176. Google ScholarDigital Library
- Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., Freedman, D., Kohli, P., Krupka, E., Fitzgibbon, A., and Izadi, S. 2015. Accurate, robust, and flexible real-time hand tracking. CHI 2015, 3633--3642. Google ScholarDigital Library
- Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., and Moore, R. 2013. Real-time human pose recognition in parts from single depth images. Communications of the ACM 56, 1, 116--124. Google ScholarDigital Library
- Shuey, D., Bailey, D., and Morrissey, T. P. 1986. PHIGS: A standard, dynamic, interactive graphics interface. IEEE Computer Graphics and Applications 6, 8, 50--57. Google ScholarDigital Library
- Skolnik, M. I. 1962. Introduction to radar. Radar Handbook 2.Google Scholar
- Smith, C., and Goggans, P. 1993. Radar target identification. IEEE Antennas and Propagation Magazine 35, 2 (April), 27--38.Google ScholarCross Ref
- Smith, J., White, T., Dodge, C., Paradiso, J., Gershenfeld, N., and Allport, D. 1998. Electric field sensing for graphical interfaces. IEEE Computer Graphics and Applications 18, June, 54--59. Google ScholarDigital Library
- Smulders, P. 2002. Exploiting the 60 GHz band for local wireless multimedia access: Prospects and future directions. IEEE Communications Magazine 40, 1, 140--147. Google ScholarDigital Library
- Song, J., Sörös, G., Pece, F., Fanello, S. R., Izadi, S., Keskin, C., and Hilliges, O. 2014. In-air gestures around unmodified mobile devices. UIST 2014, 319--329. Google ScholarDigital Library
- Storcheus, D., Rostamizadeh, A., and Kumar, S. 2015. A survey of modern questions and challenges in feature extraction. In Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges, NIPS, 1--18.Google Scholar
- Stove, A. G. 1992. Linear FMCW radar techniques. In IEEE Proceedings on Radar and Signal Processing, vol. 139, IET, 343--350.Google ScholarCross Ref
- Strickon, J., and Paradiso, J. 1998. Tracking hands above large interactive surfaces with a low-cost scanning laser rangefinder. CHI'98, 231--232. Google ScholarDigital Library
- Sutherland, I. E., Blackwell, A., and Rodden, K. 1963. Sketchpad: A man-machine graphical communication system. Tech. Rep. TK296, Lincoln Laboratory, MIT, Lexington, MA.Google ScholarDigital Library
- Wan, Q., Li, Y., Li, C., and Pal, R. 2014. Gesture recognition for smart home applications using portable radar sensors. In 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), IEEE, 6414--6417.Google Scholar
- Wang, Y., and Fathy, A. E. 2011. Micro-doppler signatures for intelligent human gait recognition using a UWB impulse radar. In IEEE Antennas and Propagation Society, AP-S International Symposium (Digest), 2103--2106.Google Scholar
- Wang, R. Y., and Popović, J. 2009. Real-time hand-tracking with a color glove. In SIGGRAPH 2009, ACM Press, New York, New York, USA, 1. Google ScholarDigital Library
- Watson-Watt, R. 1945. Radar in war and in peace. Nature 155, 3935, 319--324.Google Scholar
- Wei, T., and Zhang, X. 2015. mTrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, ACM, 117--129. Google ScholarDigital Library
- Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D. 2013. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 5, 6380--6393.Google ScholarCross Ref
- Weigel, M., Lu, T., Bailly, G., Oulasvirta, A., Majidi, C., and Steimle, J. 2015. iSkin: Flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In CHI 2015, ACM, 1--10. Google ScholarDigital Library
- Wu, J., Konrad, J., and Ishwar, P. 2013. Dynamic time warping for gesture-based user identification and authentication with Kinect. In ICASSP 2013, 2371--2375.Google Scholar
- Yatani, K., and Truong, K. N. 2012. BodyScope: A wearable acoustic sensor for activity recognition. In Ubicomp '12, ACM, 341--350. Google ScholarDigital Library
- Zhai, S., Milgram, P., and Buxton, W. 1996. The influence of muscle groups on performance of multiple degree-of-freedom input. CHI '96, 308--315. Google ScholarDigital Library
- Zhuang, Y., Song, C., Wang, A., Lin, F., Li, Y., Gu, C., Li, C., and Xu, W. 2015. SleepSense: Non-invasive sleep event recognition using an electromagnetic probe. In IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN 2015), 1--6.Google ScholarCross Ref
Index Terms
- Soli: ubiquitous gesture sensing with millimeter wave radar
Recommendations
LeapLook: a free-hand gestural travel technique using the leap motion finger tracker
SUI '14: Proceedings of the 2nd ACM symposium on Spatial user interactionContactless motion sensing devices enable a new form of input that does not encumber the user with wearable tracking equipment. We present a novel travel technique using the Leap Motion finger tracker which adopts a 2DOF steering metaphor used in ...
MicroRolls: expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThe input vocabulary for touch-screen interaction on handhelds is dramatically limited, especially when the thumb must be used. To enrich that vocabulary we propose to discriminate, among thumb gestures, those we call MicroRolls, characterized by zero ...
Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit
ITS '11: Proceedings of the ACM International Conference on Interactive Tabletops and SurfacesRecent work in multi-touch tabletop interaction introduced many novel techniques that let people manipulate digital content through touch. Yet most only detect touch blobs. This ignores richer interactions that would be possible if we could identify (1) ...
Comments