ABSTRACT
Expressiveness, which we define as the extent to which rich and complex intent can be conveyed through action, is a vital aspect of many human interactions. For instance, paint on canvas is said to be an expressive medium, because it affords the artist the ability to convey multifaceted emotional intent through intricate manipulations of a brush. To date, touch devices have failed to offer users a level of expressiveness in their interactions that rivals that experienced by the painter and those completing other skilled physical tasks. We investigate how data about hand movement -- provided by a motion sensor, similar to those found in many smart watches or fitness trackers -- can be used to expand the expressiveness of touch interactions. We begin by introducing a conceptual model that formalizes a design space of possible expressive touch interactions. We then describe and evaluate Expressy, an approach that uses a wrist-worn inertial measurement unit to detect and classify qualities of touch interaction that extend beyond those offered by today's typical sensing hardware. We conclude by describing a number of sample applications, which demonstrate the enhanced, expressive interaction capabilities made possible by Expressy.
Supplemental Material
- Apple, (2015). iOS 9 - What's New - Apple (UK). {online} Available at: http://www.apple.com/uk/ios/whats-new/ {Accessed 23 Sep. 2015}.Google Scholar
- Baglioni, M., Malacria, S., Lecolinet, E. & Guiard, Y. 2011. Flick-and-brake: finger control over inertial/sustained scroll motion. In CHI '11 Extended Abstracts on Human Factors in Computing Systems (CHI EA '11), 2281--2286. http://doi.acm.org/10.1145/1979742.1979853 Google ScholarDigital Library
- Benko, H., Saponas, T. S., Morris, D., & Tan, D. 2009. Enhancing input on and above the interactive surface with muscle sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09), 93--100. http://doi.acm.org/10.1145/1731903.1731924 Google ScholarDigital Library
- Bi, X., Moscovich, T., Ramos, G., Balakrishnan, R., & Hinckley, K. 2008. An exploration of pen rolling for pen-based interaction. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08). ACM, NY, NY, USA, 191200. http://dx.doi.org/10.1145/1449715.1449745 Google ScholarDigital Library
- Bonnet, D., Appert, C., & Beaudouin-Lafon, M. 2013. Extending the vocabulary of touch events with ThumbRock. In Proceedings of Graphics Interface 2013 (GI '13), 221--228. Google ScholarDigital Library
- Boring, S., Ledo, D., Chen, X. A., Marquardt, N., Tang, A., & Greenberg, S. 2012. The fat thumb: using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services companion (MobileHCI '12), 207208. http://doi.acm.org/10.1145/2371664.2371711 Google ScholarDigital Library
- Butler, A., Izadi, S. & Hodges, S. 2008. SideSight: multi-touch interaction around small devices. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08), 201--204. http://doi.acm.org/10.1145/1449715.1449746 Google ScholarDigital Library
- Buxton, W. 1995. Chunking and phrasing and the design of human-computer dialogues. In Human-computer interaction, Baecker, R., Grudin, J., Buxton, W., & Greenberg, S. (Eds.). Morgan Kaufmann Publishers Inc., 494--499. Google ScholarDigital Library
- Buxton, W. 1990. A three-state model of graphical input. Human-computer interaction-INTERACT. Vol. 90. Google ScholarDigital Library
- Cao, X., Wilson, A. D., Balakrishnan, R., Hinckley, K., & Hudson, S.E. 2008. Shapetouch: Leveraging contact shape on interactive surfaces. In Proceedings of the 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems (TABLETOP '08), 129--136. http://dx.doi.org/10.1109/TABLETOP.2008.4660195Google Scholar
- Chen, X. A., Grossman, T., Wigdor, D. J., & Fitzmaurice, G. 2014. Duet: exploring joint interactions on a smart phone and a smart watch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14), 159--168. http://doi.acm.org/10.1145/2556288.2556955 Google ScholarDigital Library
- Chen, X. A., Schwarz, J., Harrison, C., Mankoff, J., & Hudson, S. E. 2014. Air+touch: interweaving touch & in-air gestures. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14), 519--525. http://doi.acm.org/10.1145/2642918.2647392 Google ScholarDigital Library
- Davidson, P. L. & Han, J. Y. 2008. Extending 2D object arrangement with pressure-sensitive layering cues. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08), 87--90. http://doi.acm.org/10.1145/1449715.1449730 Google ScholarDigital Library
- Deyle, T., Palinko, S., Poole, E. S., & Starner, T. 2007. Hambone: A Bio-Acoustic Gesture Interface. In Proceedings of the 11th IEEE International Symposium on Wearable Computers (ISWC '07), 1--8. http://dx.doi.org/10.1109/ISWC.2007.4373768 Google ScholarDigital Library
- Essl, G., Rohs, M. & Kratz, S. 2010. Use the force (or something)-pressure and pressure-like input for mobile music performance. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME '10).Google Scholar
- Fitbit Surge. Available at: http://www.fitbit.com/uk/surge {Accessed March 2015}Google Scholar
- Goel, M., Wobbrock, J., & Patel, S. 2012. GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), 545--554. http://doi.acm.org/10.1145/2380116.2380184 Google ScholarDigital Library
- Hancock, M., Ten Cate, T., Carpendale, S., & Isenberg, T. 2010. Supporting sandtray therapy on an interactive tabletop. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 21332142. http://doi.acm.org/10.1145/1753326.1753651 Google ScholarDigital Library
- Harrison, B. L., Fishkin, K. P., Gujar, A., Mochon, C., & Want, R. 1998. Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98), 17--24. http://dx.doi.org/10.1145/274644.274647 Google ScholarDigital Library
- Harrison, C., Schwarz, J., & Hudson, S. E. 2011. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11), 627--636. http://doi.acm.org/10.1145/2047196.2047279 Google ScholarDigital Library
- Heo, S. & Lee, G. 2011. Force gestures: augmenting touch screen gestures with normal and tangential forces. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11), 621626. http://doi.acm.org/10.1145/2047196.2047278 Google ScholarDigital Library
- Hinckley, K. & Song, H. 2011. Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 801--810. http://doi.acm.org/10.1145/1978942.1979059 Google ScholarDigital Library
- Hinckley, K., Chen, X. & Benko, H. 2013. Motion and context sensing techniques for pen computing. In Proceedings of Graphics Interface 2013. Canadian Information Processing Society, 71--78. Google ScholarDigital Library
- Hinckley, K. et al. 2014. Sensing techniques for tablet+stylus interaction. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14. NY, NY, USA: ACM Press, 605--614. http://dx.doi.org/10.1145/2642918.2647379 Google ScholarDigital Library
- Hoggan, E., Williamson, J., Oulasvirta, A., Nacenta, M., Kristensson, P.O., & Lehtiö, A. 2013. Multi-touch rotation gestures: Performance and ergonomics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 3047--3050. http://doi.acm.org/10.1145/2470654.2481423 Google ScholarDigital Library
- Hwang, S., Bianchi, A., & Wohn, K. 2013. VibPress: estimating pressure input using vibration absorption on mobile devices. In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services (MobileHCI '13), 31--34. http://doi.acm.org/10.1145/2493190.2493193 Google ScholarDigital Library
- Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., & Olivier, P. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), 167--176. http://doi.acm.org/10.1145/2380116.2380139 Google ScholarDigital Library
- Kharrufa, A., Nicholson, J., Dunphy, P., Hodges, S., Briggs, P., & Olivier, P. 2015. Using IMUs to Identify Supervisors on Touch Devices. In Proceedings of the IFIP TC.13 International Conference on HumanComputer Interaction (INTERACT '15), 565--583. http://dx.doi.org/10.1007/978--3--319--22668--2_44Google ScholarCross Ref
- Lopes, P., Jota, R., & Jorge, J.A. 2011. Augmenting touch interaction through acoustic sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '11), 53--56. http://doi.acm.org/10.1145/2076354.2076364 Google ScholarDigital Library
- Madgwick, S. O. H., Harrison, A. J. L., & Vaidyanathan, R. 2011. Estimation of IMU and MARG orientation using a gradient descent algorithm. In Proceedings of the IEEE International Conference on Rehabilitation Robotics (ICORR '11), 1--7. http://dx.doi.org/10.1109/ICORR.2011.5975346Google ScholarCross Ref
- Malle, B. F., & Knobe, J. The folk concept of intentionality. Journal of Experimental Social Psychology (1997), 33(2), 101--121.Google Scholar
- Marquardt, N., Jota, R., Greenberg, S., & Jorge, J. A. 2011. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In Proceedings of the 13th IFIP TC 13 international conference on Human-computer interaction (INTERACT '11), 461--476. Google ScholarDigital Library
- Matejka, J., Grossman, T., & Fitzmaurice, G. 2013. Swifter: improved online video scrubbing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 1159--1168. http://doi.acm.org/10.1145/2470654.2466149 Google ScholarDigital Library
- Microsoft Band. Available at: http://www.microsoft.com/Microsoft-Band/en-us {Accessed March 2015}Google Scholar
- Murugappan, S., Vinayak, Elmqvist, N., & Ramani, K. 2012. Extended multitouch: recovering touch posture and differentiating users using a depth camera. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12), 487--496. http://doi.acm.org/10.1145/2380116.2380177 Google ScholarDigital Library
- Nguyen, Q., & Kipp, M. 2014. Orientation matters: efficiency of translation-rotation multitouch tasks. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (CHI '14), 2013--2016. http://doi.acm.org/10.1145/2556288.2557399 Google ScholarDigital Library
- Open Movement WAX9. Available at: http://github.com/digitalinteraction/openmovement/wiki /WAX9 {Accessed January 2015}Google Scholar
- Ramos, G., Boulos, M., & Balakrishnan, R. 2004. Pressure widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04), 487--494. http://doi.acm.org/10.1145/985692.985754 Google ScholarDigital Library
- Ramos, G. and Balakrishnan, R. 2005. Zliding: Fluid Zooming and Sliding for High Precision Parameter Manipulation. In Proceedings of the 18th annual ACM symposium on User interface software and technology UIST '05. NY, NY, USA: ACM Press, 143. http://dx.doi.org/10.1145/1095034.1095059 Google ScholarDigital Library
- Ramos, G. and Balakrishnan, R. 2007. Pressure marks. In Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '07. NY, NY, USA: ACM Press, 1375. http://dx.doi.org/10.1145/1240624.1240834 Google ScholarDigital Library
- Xiao, R., Schwarz, J. & Harrison, C. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS '15). ACM, NY, NY, USA, 47--50. http://dx.doi.org/10.1145/2817721.2817737 Google ScholarDigital Library
- Rogers, S., Williamson, J., Stewart, C., & MurraySmith, R. 2011. AnglePose: robust, precise capacitive touch tracking via 3d orientation estimation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 2575--2584. http://doi.acm.org/10.1145/1978942.1979318 Google ScholarDigital Library
- Sprang, S. (2015). Brushes. {online} Brushesapp.com. Available at: http://www.brushesapp.com/ {Accessed 22 Sep. 2015}.Google Scholar
- Sturman, D. J. & Zeltzer, D. A. 1994. Survey of Glovebased Input. IEEE Comput. Graph. Appl. 14 (1), 30--39. http://dx.doi.org/10.1109/38.250916 Google ScholarDigital Library
- Wang, F., & Ren, X. 2009. Empirical evaluation for finger input properties in multi-touch interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 1063--1072. http://doi.acm.org/10.1145/1518701.1518864 Google ScholarDigital Library
- Wang, F., Cao, X., Ren, X., & Irani, P. Detecting and leveraging finger orientation for interaction with directtouch surfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09), 23--32. http://doi.acm.org/10.1145/1622176.1622182 Google ScholarDigital Library
- Wilson, A. D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., and Kirk, D. 2008. Bringing physics to the surface. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08), 6776. http://doi.acm.org/10.1145/1449715.1449728 Google ScholarDigital Library
- Xiao, R., Schwarz, J. and Harrison, C. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces - ITS '15. NY, NY, USA: ACM Press, 47--50. http://dx.doi.org/10.1145/2817721.2817737 Google ScholarDigital Library
Index Terms
- Expressy: Using a Wrist-worn Inertial Measurement Unit to Add Expressiveness to Touch-based Interactions
Recommendations
Estimating 3D Finger Angle on Commodity Touchscreens
ITS '15: Proceedings of the 2015 International Conference on Interactive Tabletops & SurfacesWe describe a novel approach for estimating the pitch and yaw of fingers relative to a touchscreen's surface, offering two additional, analog degrees of freedom for interactive functions. Further, we show that our approach can be achieved on off-the-...
Pre-Touch Sensing for Mobile Interaction
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsTouchscreens continue to advance including progress towards sensing fingers proximal to the display. We explore this emerging pre-touch modality via a self-capacitance touchscreen that can sense multiple fingers above a mobile device, as well as grip ...
Ready, Steady, Touch!: Sensing Physical Contact with a Finger-Mounted IMU
A finger held in the air exhibits microvibrations, which are reduced when it touches a static object. When a finger moves along a surface, the friction between them produces vibrations, which can not be produced with a free-moving finger in the air. With ...
Comments