ABSTRACT
Over the last decades, there have been numerous efforts in wearable computing research to enable interactive textiles. Most work focus, however, on integrating sensors for planar touch gestures, and thus do not fully take advantage of the flexible, deformable and tangible material properties of textile. In this work, we introduce SmartSleeve, a deformable textile sensor, which can sense both surface and deformation gestures in real-time. It expands the gesture vocabulary with a range of expressive interaction techniques, and we explore new opportunities using advanced deformation gestures, such as, Twirl, Twist, Fold, Push and Stretch. We describe our sensor design, hardware implementation and its novel non-rigid connector architecture. We provide a detailed description of our hybrid gesture detection pipeline that uses learning-based algorithms and heuristics to enable real-time gesture detection and tracking. Its modular architecture allows us to derive new gestures through the combination with continuous properties like pressure, location, and direction. Finally, we report on the promising results from our evaluations which demonstrate real-time classification.
Supplemental Material
- Daniel Ashbrook. 2010. Enabling Mobile Microinteractions. Dissertation. Georgia Institute of Technology. http: //smartech.gatech.edu/handle/1853/33986 Google ScholarDigital Library
- Patrick Baudisch and Gerry Chu. 2009. Back-of-device interaction allows creating very small touch devices. In CHI. ACM Press, NY, NY, USA, 1923. Google ScholarDigital Library
- Alan Bränzel, Christian Holz, Daniel Hoffmann, Dominik Schmidt, Marius Knaust, Patrick Lühne, René Meusel, Stephan Richter, and Patrick Baudisch. 2013. GravitySpace: tracking users and their poses in a smart room using a pressure-sensing floor. In CHI. ACM Press, NY, NY, USA, 725. Google ScholarDigital Library
- Leah Buechley. 2008. SensorMania - Notes from e-Fashion day. (2008). http: //www.mediamatic.net/36637/en/sensormaniaGoogle Scholar
- Leah Buechley and Michael Eisenberg. 2009. Fabric PCBs, electronic sequins, and socket buttons: Techniques for e-textile craft. Personal and Ubiquitous Computing 13, 2 (feb 2009), 133--150. Google ScholarDigital Library
- Leah. Buechley, Kylie A. Peppler, Michael Eisenberg, and Yasmin B. Kafai. 2013. Textile messages : dispatches from the world of e-textiles and education (2 ed.). Peter Lang. 246 pages.Google Scholar
- Lina M Castano and Alison B Flatau. 2014. Smart fabric sensors and e-textile technologies: a review. Smart Materials and Structures 23, 5 (may 2014), 053001.Google ScholarCross Ref
- Jared Cechanowicz, Pourang Irani, and Sriram Subramanian. 2007. Augmenting the mouse with pressure sensitive input. In CHI. ACM Press, NY, NY, USA, 1385. Google ScholarDigital Library
- Fu Chang and Chun Jen Chen. 2003. A component-labeling algorithm using contour tracing technique. In ICDAR, Vol. 2003. IEEE Comput. Soc, 741--745. Google ScholarDigital Library
- Tommaso D'Alessio. 1999. Measurement errors in the scanning of piezoresistive sensors arrays. Sensors and Actuators A: Physical 72, 1 (1999), 71--76.Google ScholarCross Ref
- Lucy E. Dunne, Kaila Bibeau, Lucie Mulligan, Ashton Frith, and Cory Simon. 2012. Multi-layer e-textile circuits. In UbiComp. 649. Google ScholarDigital Library
- Sean Follmer, Micah Johnson, Edward Adelson, and Hiroshi Ishii. 2011. deForm: an interactive malleable surface for capturing 2.5D arbitrary objects, tools and touch. In UIST. ACM Press, NY, NY, USA, 527. Google ScholarDigital Library
- Markus Funk, Alireza Sahami, Niels Henze, and Albrecht Schmidt. 2014. Using a touch-sensitive wristband for text entry on smart watches. In CHI EA. ACM Press, NY, NY, USA, 2305--2310. Google ScholarDigital Library
- Guido Gioberto, James Coughlin, Kaila Bibeau, and Lucy E Dunne. 2013. Detecting Bends and Fabric Folds using Stitched Sensors. In ISWC. NY, NY, USA, 53--56. Google ScholarDigital Library
- Antonio Gomes, Andrea Nesbitt, and Roel Vertegaal. 2013. MorePhone: an actuated shape changing flexible smartphone. In CHI. ACM Press, NY, NY, USA, 583. Google ScholarDigital Library
- Nan-Wei Gong, Jürgen Steimle, Simon Olberding, Steve Hodges, Nicholas Edward Gillian, Yoshihiro Kawahara, and Joseph A. Paradiso. 2014. PrintSense: a versatile sensing technique to support multimodal flexible surface interaction. In CHI. ACM Press, NY, NY, USA, 1407--1410. Google ScholarDigital Library
- Nur Al-huda Hamdan, Jeffrey R Blum, Florian Heller, Ravi Kanth Kosuru, and Jan Borchers. 2016. Grabbing at an angle: menu selection for fabric interfaces. In ISWC. ACM Press, NY, NY, USA, 1--7. Google ScholarDigital Library
- Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In UIST. ACM Press, NY, NY, USA, 441. Google ScholarDigital Library
- Chris Harrison and Haakon Faste. 2014. Implications of location and touch for on-body projected interfaces. In DIS. ACM Press, NY, NY, USA, 543--552. Google ScholarDigital Library
- Chris Harrison, Brian Y. Lim, Aubrey Shick, and Scott E. Hudson. 2009. Where to locate wearable displays?. In CHI. ACM Press, NY, NY, USA, 941. Google ScholarDigital Library
- Chris Harrison, Shilpa Ramamurthy, and Scott E. Hudson. 2012. On-body interaction: armed and dangerous. In TEI. ACM Press, NY, NY, USA, 69. Google ScholarDigital Library
- Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the skin as an interactive canvas. In CHI. ACM Press, NY, NY, USA, 453. Google ScholarDigital Library
- Florian Heller, Stefan Ivanov, Chat Wacharamanotham, and Jan Borchers. 2014. FabriTouch: exploring flexible touch input on textiles. In ISWC. ACM Press, NY, NY, USA, 59--62. Google ScholarDigital Library
- Christopher F Herot and Guy Weinzapfel. 1978. One-point touch input of vector information for computer displays. ACM SIGGRAPH Computer Graphics 12, 3 (1978), 210--216. Google ScholarDigital Library
- David Holman and Roel Vertegaal. 2008. Organic user interfaces: designing computers in any way, shape, or form. Commun. ACM 51, 6 (2008), 48. Google ScholarDigital Library
- Christian Holz, Tovi Grossman, George Fitzmaurice, and Anne Agur. 2012. Implanted user interfaces. In CHI. ACM Press, NY, NY, USA, 503. Google ScholarDigital Library
- Jonathan Hook, Stuart Taylor, Alex Butler, Nicolas Villar, and Shahram Izadi. 2009. A reconfigurable ferromagnetic input device. In UIST. ACM Press, NY, NY, USA, 51--54. Google ScholarDigital Library
- Sungjune Jang, Lawrence H Kim, Kesler Tanner, Hiroshi Ishii, and Sean Follmer. 2016. Haptic Edge Display for Mobile Tactile Interaction. In CHI. 3706--3716. Google ScholarDigital Library
- Thorsten Karrer, Moritz Wittenhagen, Leonhard Lichtschlag, Florian Heller, and Jan Borchers. 2011. Pinstripe: eyes-free continuous input anywhere on interactive clothing. In CHI. ACM Press, NY, NY, USA, 1313. Google ScholarDigital Library
- Byron Lahey, Audrey Girouard, Winslow Burleson, and Roel Vertegaal. 2011. PaperPhone: understanding the use of bend gestures in mobile devices with ?exible electronic paper displays. In CHI. ACM Press, NY, NY, USA, 1303. Google ScholarDigital Library
- Sang-Su Lee, Sohyun Kim, Bipil Jin, Eunji Choi, Boa Kim, Xu Jia, Daeeop Kim, and Kun-pyo Lee. 2010. How users manipulate deformable displays as input devices. In CHI. ACM Press, NY, NY, USA, 1647. Google ScholarDigital Library
- Joanne Leong, Patrick Parzer, Florian Perteneder, Teo Babic, Christian Rendl, Anita Vogl, Hubert Egger, Alex Olwal, and Michael Haller. 2016. proCover: Sensory Augmentation of Prosthetic Limbs Using Smart Textile Covers. In UIST. 335--346. Google ScholarDigital Library
- Dinesh Mandalapu and Sriram Subramanian. 2011. Exploring Pressure as an Alternative to Multi-Touch Based Interaction. IndiaHCI (2011), 88. Google ScholarDigital Library
- David C. McCallum, Edward Mak, Pourang Irani, and Sriram Subramanian. 2009. PressureText: pressure input for mobile phone text entry. CHI (2009), 4519. Google ScholarDigital Library
- Sachi Mizobuchi, Shinya Terasaki, Turo Keski-Jaskari, Jari Nousiainen, Matti Ryynanen, and Miika Silfverberg. 2005. Making an impression: force-controlled pen input for handheld devices. CHI EA (2005), 1661. Google ScholarDigital Library
- Simon Olberding, Kian Peen Yeo, Suranga Nanayakkara, and Jurgen Steimle. 2013. AugmentedForearm: exploring the design space of a display-enhanced forearm. In AH. ACM Press, NY, NY, USA, 9--12. Google ScholarDigital Library
- Patrick Parzer, Kathrin Probst, Teo Babic, Christian Rendl, Anita Vogl, Alex Olwal, and Michael Haller. 2016. FlexTiles: A Flexible, Stretchable, Formable, Pressure-Sensitive, Tactile Input Sensor. In CHI EA. ACM Press, NY, NY, USA, 3754--3757. Google ScholarDigital Library
- Jerome Pasquero, Scott J. Stobbe, and Noel Stonehouse. 2011. A haptic wristwatch for eyes-free interactions. In CHI. ACM Press, NY, NY, USA, 3257. Google ScholarDigital Library
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830. Google ScholarDigital Library
- Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E. Robinson. 2016. Project Jacquard: Interactive Digital Textiles at Scale. In CHI. ACM Press, NY, NY, USA, 4216--4227. Google ScholarDigital Library
- Philip Quinn and Andy Cockburn. 2009. Zoofing!: faster list selections with pressure-zoom-flick-scrolling. In OzCHI. 185. Google ScholarDigital Library
- Gonzalo Ramos and Ravin Balakrishnan. 2005. Zliding: fluid zooming and sliding for high precision parameter manipulation. In UIST. 143. Google ScholarDigital Library
- Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure widgets. In CHI, Vol. 6. 487--494. Google ScholarDigital Library
- Christian Rendl, Patrick Greindl, Kathrin Probst, Martin Behrens, and Michael Haller. 2014a. Presstures: exploring pressure-sensitive multi-touch gestures on trackpads. In CHI. 431--434. Google ScholarDigital Library
- Christian Rendl, David Kim, Sean Fanello, Patrick Parzer, Christoph Rhemann, Jonathan Taylor, Martin Zirkl, Gregor Scheipl, Thomas Rothl¨ander, Michael Haller, and Shahram Izadi. 2014b. FlexSense: a transparent self-sensing deformable surface. In UIST. 129--138. Google ScholarDigital Library
- Ilya Rosenberg and Ken Perlin. 2009. The UnMousePad: the future of touch sensing. In ACM Transactions on Graphics, Vol. 28. ACM Press, NY, NY, USA, 1.Google Scholar
- T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: through-fabric capacitive touch input. In UIST. ACM Press, NY, NY, USA, 303. Google ScholarDigital Library
- T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. 2009. Enabling always-available input with muscle-computer interfaces. In UIST. ACM Press, NY, NY, USA, 167. Google ScholarDigital Library
- Toshiki Sato, Haruko Mamiya, Hideki Koike, and Kentaro Fukuchi. 2009. PhotoelasticTouch: transparent rubbery tangible interface using an LCD and photoelasticity. In UIST. ACM Press, NY, NY, USA, 43. Google ScholarDigital Library
- Mika Satomi and Hannah Perner-Wilson. 2016. How To Get What You Want. (2016). http://www.kobakant.at/DIY/?cat=24Google Scholar
- R. S. Saxena, R. K. Bhan, and Anita Aggrawal. 2009. A new discrete circuit for readout of resistive sensor arrays. Sensors and Actuators, A: Physical 149, 1 (2009), 93--99.Google Scholar
- Raghvendra Sahai Saxena, R. K. Bhan, Navneet Kaur Saini, and R. Muralidharan. 2011a. Virtual ground technique for crosstalk suppression in networked resistive sensors. IEEE Sensors Journal 11, 2 (feb 2011), 432--433.Google Scholar
- Raghvendra Sahai Saxena, Navneet Kaur Saini, and R. K. Bhan. 2011b. Analysis of crosstalk in networked arrays of resistive sensors. IEEE Sensors Journal 11, 4 (2011), 920--924.Google ScholarCross Ref
- Stefan Schneegass and Alexandra Voit. 2016. GestureSleeve: using touch sensitive fabrics for gestural input on the forearm for controlling smartwatches. In ISWC. ACM Press, NY, NY, USA, 108--115. Google ScholarDigital Library
- Carsten Schwesig, Ivan Poupyrev, and Eijiro Mori. 2003. Gummi: user interface for deformable computers. In CHI EA. 954. Google ScholarDigital Library
- M. Shimojo, M. Ishikawa, and K. Kanaya. 1991. A flexible high resolution tactile imager with video signal output. In ICRA. 384--391.Google Scholar
- Lin Shu, Xiaoming Tao, and David Dagan Feng. 2015. A New Approach for Readout of Resistive Sensor Arrays for Wearable Electronic Applications. IEEE Sensors Journal 15, 1 (2015), 442--452.Google ScholarCross Ref
- Craig Stewart, Michael Rohs, Sven Kratz, and Georg Essl. 2010. Characteristics of Pressure-Based Input for Mobile Devices. In CHI. ACM Press, NY, NY, USA, 801--810. Google ScholarDigital Library
- Mathias Sundholm, Jingyuan Cheng, Bo Zhou, Akash Sethi, and Paul Lukowicz. 2014. Smart-mat: recognizing and counting gym exercises with low-cost resistive pressure sensing matrix. In UbiComp. ACM Press, NY, NY, USA, 373--382. Google ScholarDigital Library
- Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-hover-swipe in 96 bytes: a motion sensing mechanical keyboard. CHI (2014), 1695--1704. Google ScholarDigital Library
- Martijn ten Bhömer and Pauline van Dongen. 2014. Vigour. Interactions 21, 5 (Sep 2014), 12--13.Google Scholar
- Giovanni Maria Troiano, Esben Warming Pedersen, and Kasper Hornbæk. 2014. User-defined gestures for elastic, deformable displays. AVI (2014), 1--8. Google ScholarDigital Library
- Nirzaree Vadgama and Jürgen Steimle. 2017. Flexy: Shape-Customizable, Single-Layer, Inkjet Printable Patterns for 1D and 2D Flex Sensing. In TEI. ACM Press, NY, NY, USA, 153--162. Google ScholarDigital Library
- Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In CHI. ACM Press, NY, NY, USA, 2991--3000. Google ScholarDigital Library
- Martin Weigel, Vikram Mehta, and Jürgen Steimle. 2014. More than touch: understanding how people use skin as an input surface for mobile computing. In CHI. ACM Press, NY, NY, USA, 179--188. Google ScholarDigital Library
- Mark Weiser. 1995. Human-computer Interaction. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, Chapter The Computer for the 21st Century, 933--940. http: //dl.acm.org/citation.cfm?id=212925.213017 Google ScholarDigital Library
- G Wilson. 2013. Using Pressure Input and Thermal Feedback to Broaden Haptic Interaction with Mobile Devices. In PhD Thesis. 1--273. http://encore.lib. gla.ac.uk/iii/encore/record/C__Rb2982123Google Scholar
- Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined gestures for surface computing. In CHI. ACM Press, NY, NY, USA, 1083. Google ScholarDigital Library
- Bo Zhou, Harald Koerger, Markus Wirth, Constantin Zwick, Christine Martindale, Heber Cruz, Bjoern Eskofier, and Paul Lukowicz. 2016a. Smart soccer shoe: monitoring foot-ball interaction with shoe integrated textile pressure sensor matrix. In ISWC. ACM Press, NY, NY, USA, 64--71. Google ScholarDigital Library
- Bo Zhou, Mathias Sundholm, Jingyuan Cheng, Heber Cruz, and Paul Lukowicz. 2016b. Never skip leg day: A novel wearable approach to monitoring gym leg exercises. In PerCom.Google Scholar
Recommendations
Project Jacquard: Interactive Digital Textiles at Scale
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsProject Jacquard presents manufacturing technologies that enable deploying invisible ubiquitous interactivity at scale. We propose novel interactive textile materials that can be manufactured inexpensively using existing textile weaving technology and ...
RESi: A Highly Flexible, Pressure-Sensitive, Imperceptible Textile Interface Based on Resistive Yarns
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyWe present RESi (Resistive tExtile Sensor Interfaces), a novel sensing approach enabling a new kind of yarn-based, resistive pressure sensing. The core of RESi builds on a newly designed yarn, which features conductive and resistive properties. We run a ...
E-Textile Microinteractions: Augmenting Twist with Flick, Slide and Grasp Gestures for Soft Electronics
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsE-textile microinteractions advance cord-based interfaces by enabling the simultaneous use of precise continuous control and casual discrete gestures. We leverage the recently introduced I/O Braid sensing architecture to enable a series of user studies ...
Comments