Ken Hinckley
Koji Yatani
Hrvoje Benko
Bill Buxton
ABSTRACT
We describe techniques for direct pen+touch input. We observe people's manual behaviors with physical paper and notebooks. These serve as the foundation for a prototype Microsoft Surface application, centered on note-taking and scrapbooking of materials. Based on our explorations we advocate a division of labor between pen and touch: the pen writes, touch manipulates, and the combination of pen + touch yields new tools. This articulates how our system interprets unimodal pen, unimodal touch, and multimodal pen+touch inputs, respectively. For example, the user can hold a photo and drag off with the pen to create and place a copy; hold a photo and cross it in a freeform path with the pen to slice it in two; or hold selected photos and tap one with the pen to staple them all together. Touch thus unifies object selection with mode switching of the pen, while the muscular tension of holding touch serves as the "glue" that phrases together all the inputs into a unitary multimodal gesture. This helps the UI designer to avoid encumbrances such as physical buttons, persistent modes, or widgets that detract from the user's focus on the workspace.
Supplemental Material
- }}Adler, A., Gujar, A., Harrison, B., O'Hara, K. Sellen, A. A diary study of work-related reading design implications for digital reading devices. CHI'98. p. 241--48. Google Scholar
Digital Library
- }}Agarawala, A. Balakrishnan, R. Keepin' It Real: Pushing the Desktop Metaphor with Physics, Piles and the Pen. CHI '06. Google ScholarDigital Library
- }}Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., Buxton, W. Digital Tape Drawing. UIST'99. p. 161--70. Google ScholarDigital Library
- }}Bondarenko, O., Janssen, R. Documents at Hand: Learning from Paper to Improve Digital Technologies. CHI 2005. Google ScholarDigital Library
- }}Bragdon, A., Zeleznik, R., Williamson, B., Miller, T., LaViola, J. GestureBar: improving the approachability of gesture-based interfaces. CHI'09. p. 2269--78. Google ScholarDigital Library
- }}Brandl, P., Forlines, C., Wigdor, D., Haller, M., Shen, C. Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces. AVI 2008 Conf. on Advanced Visual interfaces. p. 154--61. Google ScholarDigital Library
- }}Brandl, P., Leitner, J., Seifried, T., Haller, M., Doray, B., To, P. Occlusion-aware menu design for digital tabletops. CHI 2009 Extended Abstracts. p. 3223--28. Google ScholarDigital Library
- }}Buxton, B., Sketching User Experiences: Getting the Design Right and the Right Design. 2007, San Francisco: Morgan Kaufman. Google ScholarDigital Library
- }}Buxton, W. Chunking and Phrasing and the Design of Human-Computer Dialogues. IFIP '86, p. 475--80.Google Scholar
- }}Buxton, W., Myers, B. A Study in Two-Handed Input. CHI'86. Google ScholarDigital Library
- }}Chen, N., Guimbretiere, F., Dixon, M., Lewis, C., Agrawala, M. Navigation Techniques for Dual-Display E-Book Readers. CHI '08. Google ScholarDigital Library
- }}Cohen, P. The role of natural language in a multimodal interface. UIST'92. p. 143--49. Google ScholarDigital Library
- }}Cohen, P., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J. QuickSet: Multimodal Interaction for Distributed Applications. ACM Multimedial 97. Google ScholarDigital Library
- }}Davidson, P. L. and Han, J. Y. Extending 2D object arrangement with pressure-sensitive layering cues. UIST '08. Google ScholarDigital Library
- }}Engelhard, L. Native Dual Mode Digitizers: Supporting Pen, Touch and Multi-Touch Inputs in One Device on any LCD. Society for Information Display SID 08 Digest. p. 1306--09.Google Scholar
- }}Frisch, M., Heydekorn, J., Dachselt, R. Investigating Multi-Touch and Pen Gestures for Diagram Editing on Interactive Surfaces ITS '09 Conf. on Interactive Tabletops and Surfaces. Google ScholarDigital Library
- }}Greenberg, S., Buxton, B. Usability evaluation considered harmful (some of the time). CHI '08. p. 111--20. Google ScholarDigital Library
- }}Guiard, Y., Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model. The Journal of Motor Behavior, 1987. 19(4): p. 486--517.Google Scholar
- }}Hartmann, B. Pictionaire: Supporting Collaborative Design Work by Integrating Physical and Digital. CSCW '10. Google ScholarDigital Library
- }}Hinckley, K., et al. Codex: a dual screen tablet computer. CHI '09. Google ScholarDigital Library
- }}Hinckley, K., Pahud, M. and Buxton, B. Direct Display Interaction via Simultaneous Pen + Multi-touch Input. Society for Information Display SID 2010 Digest. p. 537--40.Google Scholar
- }}Hinckley, K., Pierce, J., Sinclair, M. and Horvitz, E. Sensing Techniques for Mobile Interaction. UIST 2000, p. 91--100. Google ScholarDigital Library
- }}Hinckley, K., Zhao, S., Sarin, R., Baudisch, P., Cutrell, E., Shilman, M., Tan, D. InkSeine: In Situ Search for Active Note Taking. CHI'07. Google ScholarDigital Library
- }}Kidd, A. The marks are on the knowledge worker. CHI '94. 186--91. Google ScholarDigital Library
- }}Kurtenbach, G. and Buxton, W. Issues in Combining Marking and Direct Manipulation Techniques. UIST'91. Google ScholarDigital Library
- }}Kurtenbach, G., Fitzmaurice, G., Baudel, T. and Buxton, B. The Design of a GUI Paradigm based on Tablets, Two-hands, and Transparency. CHI'97, p. 35--42. Google ScholarDigital Library
- }}Li, Y., Hinckley, K., Guan, Z., Landay, J. Experimental Analysis of Mode Switching Techniques in Pen-based User Interfaces. CHI '05. Google ScholarDigital Library
- }}Marshall, C., Bly, S. Saving and Using Encountered Information: Implications for Electronic Periodicals. CHI '05. Google ScholarDigital Library
- }}http://en.wikipedia.org/wiki/Microsoft_Courier. 2009.Google Scholar
- }}Roth, V., Turner, T. Bezel swipe: conflict-free scrolling and multiple selection on mobile touch screen devices. CHI 2009. Google ScholarDigital Library
- }}Ruiz, J., Bunt, A., Lank, E. A model of non-preferred hand mode switching. Graphics interface 2008 p. 49--56. Google ScholarDigital Library
- }}Ruiz, J., Lank, E. A study on the scalability of non-preferred hand mode manipulation. ICMI '07, p. 170--77. Google ScholarDigital Library
- }}Schilit, B., Golovchinsky, G., Price, M. Beyond paper: supporting active reading with free form digital ink annotations. CHI'98. Google ScholarDigital Library
- }}Sellen, A., Kurtenbach, G., Buxton, W., The Prevention of Mode Errors through Sensory Feedback. J. Human Computer Interaction, 1992. 7(2): p. 141--64. Google ScholarDigital Library
- }}Terrenghi, L., Kirk, D., Sellen, A., Izadi, S. Affordances for manipulation of physical versus digital media on interactive surfaces. CHI '07. p. 1157--66. Google ScholarDigital Library
- }}Wilson, A. D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., Kirk, D. Bringing physics to the surface. UIST 2008. Google ScholarDigital Library
- }}Wu, M., Balakrishnan, R. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. UIST 2003. Google ScholarDigital Library
- }}Wu, M., Shen, C., Ryall, K., Forlines, C., Balakrishnan, R., Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces, in IEEE Intl. Workshop on Horizontal Interactive Human-Computer Systems. 2006. Google ScholarDigital Library
- }}Yee, K.-P. Two-Handed Interaction on a Tablet Display. CHI 2004. Google ScholarDigital Library
Index Terms
- Pen + touch = new tools
Recommendations
Manual deskterity: an exploration of simultaneous pen + touch direct input
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing SystemsManual Deskterity is a prototype digital drafting table that supports both pen and touch input. We explore a division of labor between pen and touch that flows from natural human skill and differentiation of roles of the hands. We also explore the ...
Tablets, tabletops, and smartphones: cross-platform comparisons of children’s touchscreen interactions
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal InteractionThe proliferation of smartphones and tablets has increased children’s access to and usage of touchscreen devices. Prior work on smartphones has shown that children’s touch interactions differ from adults’. However, larger screen devices like tablets ...
Video Interaction on Tablet Computers: Browsing with Pinch Gesture and Pen Tilt
MUM '22: Proceedings of the 21st International Conference on Mobile and Ubiquitous MultimediaDigital video technologies have been changing dramatically in the field of media production but reduced the tangibility of working with film. One interesting approach to get back such tangibility in digital systems is using pen and touch interactions. ...
Comments