ABSTRACT
We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures produced by the nonpreferred hand, from touch gestures produced by the hand holding the pen, which necessarily impart a detectable motion signal to the stylus. We can sense which hand grips the tablet, and determine the screen's relative orientation to the pen. By selectively combining these signals and using them to complement one another, we can tailor interaction to the context, such as by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the pen tucked between his fingers. These and other techniques can be used to impart new, previously unanticipated subtleties to pen + touch interaction on tablets.
Supplemental Material
- Annett, M., et al. The Pen is Mightier: Understanding Stylus Behaviour While Inking on Tablets. Graphics Interface (GI 2014). Google ScholarDigital Library
- Annett, M., et al. Is it Intended or Unintended? Palm Rejection during Direct Pen Interaction. (Currently submitted for review), 2014.Google Scholar
- Becchio, C., et al. Grasping intentions: from thought experiments to emprical evidence. Frontiers in Human Neuroscience, 2012. 6(117).Google ScholarCross Ref
- Bi, X., et al. An Exploration of Pen Rolling. UIST'08.Google Scholar
- Brandl, P., et al. Occlusion-aware menu design for digital tabletops. CHI'09 Extended Abstracts. Google ScholarDigital Library
- Buxton, W. Integrating the Periphery and Context: A New Taxonomy of Telematics. Proc. Graphics Interface '95.Google Scholar
- Cheng, L.-P., et al. iRotate grasp: automatic screen rotation based on grasp of mobile devices. UIST Adjunct Proc. '12. Google ScholarDigital Library
- Cheng, L.-P., et al. iGrasp: grasp-based adaptive keyboard for mobile devices. CHI '13 Extended Abstracts. Google ScholarDigital Library
- Chun Yat Li, F., et al. The 1Line Keyboard: A QWERTY Layout in a Single Line. UIST'11.Google Scholar
- FiftyThree, I. A closer look at zoom. http://tinyurl.com/mxqdshz.Google Scholar
- FiftyThree, I. Pencil (for Paper). http://www.fiftythree.com/pencil.Google Scholar
- Goel, M., et al. GripSense: Using Built-In Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones. UIST'12. Google ScholarDigital Library
- Guiard, Y., Millerat, F., Writing Postures in Left-Handers: Inverters are Hand-Crossers. Neuropsychologia, 1984. 22(5): p. 535--538.Google Scholar
- Harrison, B., et al. Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces. CHI'98. Google ScholarDigital Library
- Hasan, K., et al. A-coord input: coordinating auxiliary input streams for augmenting contextual pen-based interactions. CHI '12. Google ScholarDigital Library
- Hinckley, K. Synchronous Gestures for Multiple Users and Computers. UIST'03. Google ScholarDigital Library
- Hinckley, K., et al. Motion and Context Sensing Techniques for Pen Computing. Proc. Graphics Interface 2013 (GI'13). Google ScholarDigital Library
- Hinckley, K., et al. Codex: A Dual-Screen Tablet Computer. CHI'09. Google ScholarDigital Library
- Hinckley, K., Pierce, J., Sinclair, M., Horvitz, E. Sensing techniques for mobile interaction. UIST 2000. Google ScholarDigital Library
- Hinckley, K., Song, H. Sensor Synaesthesia: Touch in Motion, and Motion in Touch. CHI 2011. Google ScholarDigital Library
- Hinckley, K., et al. Pen + Touch = New Tools. UIST 2010. Google ScholarDigital Library
- Holmquist, L., et al. Smart-Its Friends: A Technique for Users to Easily Establish Connections between Smart Artifacts. Ubicomp'01. Google ScholarDigital Library
- Hudson, S., et al. Whack Gestures: Inexact and Inattentive Interaction with Mobile Devices. TEI 2010. Google ScholarDigital Library
- Kim, K.-E., et al. Hand Grip Pattern Recognition for Mobile User Interfaces. Proceedings of AAAI/IAAI-2006: Innovative Appl. Google ScholarDigital Library
- Kurtenbach, G., et al. The design of a GUI paradigm based on tablets, two-hands, and transparency. CHI'97. Google ScholarDigital Library
- Lee, D., et al. PhantomPen: virtualization of pen head for digital drawing free from pen occlusion & visual parallax. UIST'12. Google ScholarDigital Library
- Lester, J., et al. Are You With Me? Using Accelerometers to Determine if Two Devices are Carried. Pervasive'04.Google Scholar
- Liu, S., Guimbretiere, F. FlexAura: A Flexible Near-Surface Range Sensor. UIST '12. Google ScholarDigital Library
- Luff, P., Heath, C. Mobility in collaboration. CSCW '98. Google ScholarDigital Library
- Mackenzie, C., Iberall, T., The Grasping Hand. Advances in Psychology 104. 1994, Amsterdam: North Holland.Google Scholar
- Mahoney, R., Nonlinear Complementary Filters on the Special Orthogonal Group. IEEE Trans. on Automatic Control, 2008. 53(5).Google Scholar
- Marquardt, N., et al. Cross-Device Interaction via Micro-mobility and F-formations. UIST '12. 2012. Cambridge, MA. Google ScholarDigital Library
- Mason, A., MacKenzie, C., Grip forces when passing an object to a partner. Exp. Brain Res, 2005(163): p. 173--187.Google Scholar
- Matulic, F., Norrie, M. Supporting active reading on pen and touchoperated tabletops. Proc. Int'l Working Conf. on Advanced Visual Interfaces (AVI'12). 2012. Capri Island, Italy. Google ScholarDigital Library
- Moleskine Journal (iPad app). 2014. http://tinyurl.com/kqhtomz.Google Scholar
- Penultimate by Evernote (iPad app). 2014. http://tinyurl.com/kx82c8a.Google Scholar
- Premerlani, W., Bizard, P. Direction Cosine Matrix IMU: Theory. 2009. http://gentlenav.googlecode.com/files/DCMDraft2.pdf.Google Scholar
- Rofouei, M., et al. Your Phone or Mine? Fusing Body, Touch and Device Sensing for MultiUser Device-Display Interaction. CHI 2012. Google ScholarDigital Library
- Roudaut, A., et al. TimeTilt: Using Sensor-Based Gestures to Travel through Multiple Applications on a Mobile Device. Interact '09. Google ScholarDigital Library
- Schmidt, A., et al. Advanced interaction in context. Handheld and Ubiquitous Computing (HUC'99). Google ScholarDigital Library
- Schmidt, D., et al. PhoneTouch: A Technique for Direct Phone Interaction on Surfaces. UIST'10. Google ScholarDigital Library
- Schwarz, J., et al. Probabilistic Palm Rejection Using Spatiotemporal Touch Features and Iterative Classification. CHI'14. Google ScholarDigital Library
- Selin, A., Pencil Grip: A descriptive Model and Four Empirical Studies. 2003: Akademi University Press.Google Scholar
- Sellen, A., Shaw., K. How Knowledge Workers Use the Web. CHI'02. Google ScholarDigital Library
- Siio, I., Tsujita, H. Mobile interaction using paperweight metaphor. UIST '06. Google ScholarDigital Library
- Song, H., et al. Grips and Gestures on a Multi-Touch Pen. CHI'11. Google ScholarDigital Library
- Sun, M., et al. Enhancing Naturalness of Pen-and-Tablet Drawing through Context Sensing. Interactive Tabletops and Surfaces (ITS '11). Google ScholarDigital Library
- Suzuki, Y., et al. Stylus Enhancement to Enrich Interaction with Computers. HCII 2007, Part II. 2007: Springer-Verlag. Google ScholarDigital Library
- Taylor, B., Bove Jr., V. Graspables: Grasp-Recognition as a User Interface. CHI'09. Google ScholarDigital Library
- Tian, F., et al. Tilt Menu: Using the 3D Orientation Information of Pen Devices to Extend the Selection Capability. CHI'08. Google ScholarDigital Library
- Vogel, D., Casiez, G. Conte: Multimodal Input Inspired by an Artist's Crayon. UIST 2011. Google ScholarDigital Library
- Wagner, J., et al. BiTouch and BiPad: Designing Bimanual Interaction for Hand-held Tablets. CHI'12. Google ScholarDigital Library
- Wimmer, R., Boring, S. HandSense Discriminating Different Ways of Grasping and Holding a Tangible User Interface. TEI '09. Google ScholarDigital Library
- Xin, Y., et al. Natural Use Profiles for the Pen: An Empirical Exploration of Pressure, Tilt, and Azimuth. CHI'12. Google ScholarDigital Library
- Yoon, D., Chen, N., Guimbretiere, F. TextTearing: Expanding Whitespace for Digital Ink Annotation. UIST'13. Google ScholarDigital Library
Index Terms
- Sensing techniques for tablet+stylus interaction
Recommendations
Thumb + Pen Interaction on Tablets
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsModern tablets support simultaneous pen and touch input, but it remains unclear how to best leverage this capability for bimanual input when the nonpreferred hand holds the tablet. We explore Thumb + Pen interactions that support simultaneous pen and ...
Wearables as Context for Guiard-abiding Bimanual Touch
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and TechnologyWe explore the contextual details afforded by wearable devices to support multi-user, direct-touch interaction on electronic whiteboards in a way that-unlike previous work-can be fully consistent with natural bimanual-asymmetric interaction as set forth ...
Sensing Posture-Aware Pen+Touch Interaction on Tablets
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsMany status-quo interfaces for tablets with pen + touch input capabilities force users to reach for device-centric UI widgets at fixed locations, rather than sensing and adapting to the user-centric posture. To address this problem, we propose sensing ...
Comments