Andrew M. Webb
Michel Pahud
ABSTRACT
We explore the contextual details afforded by wearable devices to support multi-user, direct-touch interaction on electronic whiteboards in a way that-unlike previous work-can be fully consistent with natural bimanual-asymmetric interaction as set forth by Guiard.
Our work offers the following key observation. While Guiard's framework has been widely applied in HCI, for bimanual interfaces where each hand interacts via direct touch, subtle limitations of multi-touch technologies as well as limitations in conception and design-mean that the resulting interfaces often cannot fully adhere to Guiard's principles even if they want to. The interactions are fundamentally ambiguous because the system does not know which hand, left or right, contributes each touch. But by integrating additional context from wearable devices, our system can identify which user is touching, as well as distinguish what hand they use to do so. This enables our prototypes to respect lateral preference the assignment of natural roles to each hand as advocated by Guiard in a way that has not been articulated before.
Supplemental Material
- Adobe Illustrator. 2016. http://www.adobe.com/products/illustratorGoogle Scholar
- Annett, M., Grossman, T., Wigdor, D. and Fitzmaurice, G. Medusa: a proximity-aware multi-touch tabletop. In Proceedings of the 24th annual ACM symposium on User interface software and technology (2011). ACM, New York, NY, USA, 337--346. DOI=10.1145/2047196.2047240 http://doi.acm.org/10.1145/2047196.2047240 Google ScholarDigital Library
- Apple Watch. 2015. http://www.apple.com/watchGoogle Scholar
- Ardito, C., Buono, P., Costabile, M.F. and Desolda, G. Interaction with Large Displays: A Survey. ACM Computing Survey. 47, 3, Article 46 (February 2015), 38 pages. DOI=http://doi.acm.org/10.1145/2682623 Google ScholarDigital Library
- Azad, A., Ruiz, J., Vogel, D., Hancock, M., and Lank, E. Territoriality and behaviour on and around large vertical publicly-shared displays. In Proceedings of the Designing Interactive Systems Conference (2012). ACM, New York, NY, USA, 468--477. DOI=10.1145/2317956.2318025 http://doi.acm.org/10.1145/2317956.2318025 Google ScholarDigital Library
- Balakrishnan, R. and Hinckley, K. Symmetric bimanual interaction. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (2000). ACM, New York, NY, USA, 33--40. DOI=http://dx.doi.org/10.1145/332040.332404 Google ScholarDigital Library
- Bau, O. and Mackay. W.E. OctoPocus: a dynamic guide for learning gesture-based command sets. In Proceedings of the 21st annual ACM symposium on User interface software and technology (2008). ACM, New York, NY, USA, 37--46. DOI=http://dx.doi.org/10.1145/1449715.1449724 Google ScholarDigital Library
- Beyer, H. and Holtzblatt, K. Contextual design. Interactions 6, 1 (January 1999), 32--42. DOI=10.1145/291224.291229 http://doi.acm.org/10.1145/291224.291229 Google ScholarDigital Library
- Bier, E.A., Stone, M.C., Pier, K., Buxton, W., and DeRose, T.D. Toolglass and magic lenses: the seethrough interface. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques (1993). ACM, New York, NY, USA, 73--80. DOI=10.1145/166117.166126 http://doi.acm.org/10.1145/166117.166126 Google ScholarDigital Library
- Blažica, B., Vladušič, D., and Mladenić, D. MTi: A method for user identification for multitouch displays. International Journal of Human-Computer Studies. 71, 6 (June 2013), 691--702. DOI=10.1016/j.ijhcs.2013.03.002 http://dx.doi.org/10.1016/j.ijhcs.2013.03.002 Google ScholarDigital Library
- Brandl, P., Forlines, C., Wigdor, D., Haller, M., and Shen, C. Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces. In Proceedings of the working conference on Advanced visual interfaces (2008). ACM, New York, NY, USA, 154--161. DOI=http://dx.doi.org/10.1145/1385569.1385595 Google ScholarDigital Library
- Buxton, B. Sketching User Experiences: getting the design right and the right design. 2007. Morgan Kaufmann. Google ScholarDigital Library
- Buxton, W. Chunking and phrasing and the design of human-computer dialogues. In IFIP Congress, 1986, 475--480.Google Scholar
- Buxton, W., Fitzmaurice, G., Balakrishnan, R., and Kurtenbach, G. Large Displays in Automotive Design. IEEE Computer Graphics and Applications 20, 4 (July 2000), 68--75. DOI=10.1109/38.851753 http://dx.doi.org/10.1109/38.851753 Google ScholarDigital Library
- Buxton, W. and Myers, B. A study in two-handed input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (1986), Marilyn Mantei and Peter Orbeton (Eds.). ACM, New York, NY, USA, 321--326. DOI=10.1145/22627.22390 http://doi.acm.org/10.1145/22627.22390 Google ScholarDigital Library
- Chatty, S. Extending a graphical toolkit for two-handed interaction. In Proceedings of the 7th annual ACM symposium on User interface software and technology (1994). ACM, New York, NY, USA, 195--204. DOI=http://dx.doi.org/10.1145/192426.192500 Google ScholarDigital Library
- Chen, X.A., Grossman, T., Wigdor, D.J., and Fitzmaurice, G. Duet: exploring joint interactions on a smart phone and a smart watch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2014). ACM, New York, NY, USA, 159--168. DOI=http://dx.doi.org/10.1145/2556288.2556955 Google ScholarDigital Library
- Davidson, P.L. and Han, J.Y. Extending 2D object arrangement with pressure-sensitive layering cues. In Proceedings of the 21st annual ACM symposium on User interface software and technology (2008). ACM, New York, NY, USA, 87--90. DOI=10.1145/1449715.1449730 http://doi.acm.org/10.1145/1449715.1449730 Google ScholarDigital Library
- Dietz, P. and Leigh, D. DiamondTouch: a multi-user touch technology. In Proceedings of the 14th annual ACM symposium on User interface software and technology (2001). ACM, New York, NY, USA, 219226. DOI=10.1145/502348.502389 http://doi.acm.org/10.1145/502348.502389 Google ScholarDigital Library
- Ewerling, P., Kulik, A., and Froehlich, B. Finger and hand detection for multi-touch interfaces based on maximally stable extremal regions. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (2012). ACM, New York, NY, USA, 173--182. DOI=http://dx.doi.org/10.1145/2396636.2396663 Google ScholarDigital Library
- Fass, A., Forlizzi, J., and Pausch, R. MessyDesk and MessyBoard: two designs inspired by the goal of improving human memory. In Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques (2002). ACM, New York, NY, USA, 303--311. DOI=10.1145/778712.778754 http://doi.acm.org/10.1145/778712.778754 Google ScholarDigital Library
- Fitbit. 2016. https://www.fitbit.comGoogle Scholar
- Freeman, D., Benko, H., Morris, M.R., and Wigdor, D. ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (2009). ACM, New York, NY, USA, 165--172. DOI=http://dx.doi.org/10.1145/1731903.1731935 Google ScholarDigital Library
- Garner, S., and McDonagh-Philp, D. Problem Interpretation and Resolution via Visual Stimuli: The Use of "Mood Boards" in Design Education. Journal of Art & Design Education, 2001, 20: 57--64. DOI=10.1111/1468--5949.00250Google ScholarCross Ref
- Ghomi, E., Huot, S., Bau, O., Beaudouin-Lafon, M., and Mackay, W.E. Arpège: learning multitouch chord gestures vocabularies. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces (2013). ACM, New York, NY, USA, 209--218. DOI=http://dx.doi.org/10.1145/2512349.2512795 Google ScholarDigital Library
- Greenberg, S., Marquardt, N., Ballendat, T., DiazMarino, R., and Wang, M. Proxemic interactions: the new ubicomp'. interactions 18, 1 (January 2011), 4250. DOI=http://dx.doi.org/10.1145/1897239.1897250 Google ScholarDigital Library
- Guiard, Y. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. In Journal of motor behavior, 1987, 19(4), 486--517. Google ScholarCross Ref
- Guimbretière, F., Stone, M., and Winograd, T. Fluid interaction with high-resolution wall-size displays. In Proceedings of the 14th annual ACM symposium on User interface software and technology (2001). ACM, New York, NY, USA, 21--30. DOI=10.1145/502348.502353 http://doi.acm.org/10.1145/502348.502353 Google ScholarDigital Library
- Guzdial, M. Software-realized scaffolding to facilitate programming for science learning. Interactive Learning Environments, 1994, 4(1), 001--044.Google ScholarCross Ref
- Hamilton, W., Kerne, A., and Robbins, T. Highperformance pen + touch modality interactions: a realtime strategy game eSports context. In Proceedings of the 25th annual ACM symposium on User interface software and technology (2012). ACM, New York, NY, USA, 309--318. DOI=10.1145/2380116.2380156 http://doi.acm.org/10.1145/2380116.2380156 Google ScholarDigital Library
- Harrison, C., Schwarz, J., and Hudson, S.E. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology (2011). ACM, New York, NY, USA, 627--636. DOI=10.1145/2047196.2047279 http://doi.acm.org/10.1145/2047196.2047279 Google ScholarDigital Library
- Hinckley, K., Pahud, M., Benko, H., Irani, P., Guimbretière, F., Gavriliu, M., Chen, X.A., Matulic, F., Buxton, W., and Wilson, A. 2014. Sensing techniques for tablet+stylus interaction. In Proceedings of the 27th annual ACM symposium on User interface software and technology (2014). ACM, New York, NY, USA, 605--614. DOI=10.1145/2642918.2647379 http://doi.acm.org/10.1145/2642918.2647379 Google ScholarDigital Library
- Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, J., Wilson, A., Benko, H., and Buxton, B. 2010. Pen + touch = new tools. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (2010). ACM, New York, NY, USA, 27--36. DOI=10.1145/1866029.1866036 http://doi.acm.org/10.1145/1866029.1866036 Google ScholarDigital Library
- Hinrichs, U. and Carpendale, S. Gestures in the wild: studying multi-touch gesture sequences on interactive tabletop exhibits. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2011). ACM, New York, NY, USA, 30233032. DOI=10.1145/1978942.1979391 http://dx.doi.org/10.1145/1978942.1979391 Google ScholarDigital Library
- Holz, C. and Baudisch, P. Fiberio: a touchscreen that senses fingerprints. InProceedings of the 26th annual ACM symposium on User interface software and technology (2013). ACM, New York, NY, USA, 41--50. DOI=http://dx.doi.org/10.1145/2501988.2502021 Google ScholarDigital Library
- Ju, W., Lee, B.A., and Klemmer, S.R. Range: exploring implicit interaction through electronic whiteboard design. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (2008). ACM, New York, NY, USA, 17--26. DOI=10.1145/1460563.1460569 http://doi.acm.org/10.1145/1460563.1460569 Google ScholarDigital Library
- Kabbash, P., Buxton, W., and Sellen, A. Two-handed input in a compound task. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (1994), Beth Adelson, Susan Dumais, and Judith Olson (Eds.). ACM, New York, NY, USA, 417--423. DOI=10.1145/191666.191808 http://doi.acm.org/10.1145/191666.191808 Google ScholarDigital Library
- Kienzle, W. and Hinckley, K. LightRing: alwaysavailable 2D input on any surface. In Proceedings of the 27th annual ACM symposium on User interface software and technology (2014). ACM, New York, NY, USA, 157--160. DOI=http://dx.doi.org/10.1145/2642918.2647376 Google ScholarDigital Library
- Kharrufa, A., Nicholson, J., Dunphy, P., Hodges, S., Briggs, P., and Olivier, P. Using IMUs to Identify Supervisors on Touch Devices. In INTERACT, 2015, Springer, 565--583. Google ScholarCross Ref
- Kurtenbach, G., Fitzmaurice, G., Baudel, T., and Buxton, B. The design of a GUI paradigm based on tablets, two-hands, and transparency. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (1997). ACM, New York, NY, USA, 35--42. DOI=http://dx.doi.org/10.1145/258549.258574 Google ScholarDigital Library
- Kurtenbach, G., Moran, T. P., & Buxton, W. Contextual animation of gestural commands. In Computer Graphics Forum, December 1994, 13(5), Blackwell Science Ltd. Google ScholarCross Ref
- Latulipe, C., Mann, S., Kaplan, C.S., and Clarke, C.L.A. symSpline: symmetric two-handed spline manipulation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2006), Rebecca Grinter, Thomas Rodden, Paul Aoki, Ed Cutrell, Robin Jeffries, and Gary Olson (Eds.). ACM, New York, NY, USA, 349--358. DOI=http://dx.doi.org/10.1145/1124772.1124825 Google ScholarDigital Library
- Leganchuk, A., Zhai, S., and Buxton, W. Manual and cognitive benefits of two-handed input: an experimental study. ACM Transactions on ComputerHuman Interaction. 5, 4 (December 1998), 326--359. DOI=10.1145/300520.300522 http://doi.acm.org/10.1145/300520.300522 Google ScholarDigital Library
- Li, F.C.Y., Guy, R.T., Yatani, K., and Truong, K.N. The 1line keyboard: a QWERTY layout in a single line. In Proceedings of the 24th annual ACM symposium on User interface software and technology (2011). ACM, New York, NY, USA, 461--470. DOI=10.1145/2047196.2047257 http://doi.acm.org/10.1145/2047196.2047257 Google ScholarDigital Library
- Lucero, A. Framing, aligning, paradoxing, abstracting, and directing: how design mood boards work. In Proceedings of the Designing Interactive Systems Conference (2012). ACM, New York, NY, USA, 438447. DOI=10.1145/2317956.2318021 http://doi.acm.org/10.1145/2317956.2318021 Google ScholarDigital Library
- Lucero, A., Aliakseyeu, D., and Martens, J.B. Funky wall: presenting mood boards using gesture, speech and visuals. In Proceedings of the working conference on Advanced visual interfaces (2008). ACM, New York, NY, USA, 425--428. DOI=10.1145/1385569.1385650 http://doi.acm.org/10.1145/1385569.1385650 Google ScholarDigital Library
- Marquardt, N., Kiemer, J., and Greenberg, S. What caused that touch': expressive interaction with a surface through fiduciary-tagged gloves. In ACM International Conference on Interactive Tabletops and Surfaces (2010). ACM, New York, NY, USA, 139142. DOI=10.1145/1936652.1936680 http://doi.acm.org/10.1145/1936652.1936680 Google ScholarDigital Library
- Mare, S., Markham, A.M., Cornelius, C., Peterson, R., and Kotz, D. ZEBRA: zero-effort bilateral recurring authentication. In IEEE Symposium on Security and Privacy (SP), 2014, IEEE 705--720. DOI=10.1109/SP.2014.51 Google ScholarDigital Library
- Matulic, F. and Norrie, M.C. Pen and touch gestural environment for document editing on interactive tabletops. In Proceedings of the ACM international conference on Interactive tabletops and surfaces (2013). ACM, New York, NY, USA, 41--50. DOI=10.1145/2512349.2512802 http://doi.acm.org/10.1145/2512349.2512802 Google ScholarDigital Library
- Microsoft Band. 2016. http://www.microsoft.com/microsoft-bandGoogle Scholar
- Morris, M.R., Huang, A., Paepcke, A., and Winograd, T. Cooperative gestures: multi-user gestural interactions for co-located groupware. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2006), Rebecca Grinter, Thomas Rodden, Paul Aoki, Ed Cutrell, Robin Jeffries, and Gary Olson (Eds.). ACM, New York, NY, USA, 1201--1210. DOI=http://dx.doi.org/10.1145/1124772.1124952 Google ScholarDigital Library
- Mynatt, E.D., Igarashi, T., Edwards, W.K., and LaMarca, A. Flatland: new dimensions in office whiteboards. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (1999). ACM, New York, NY, USA, 346--353. DOI=10.1145/302979.303108 http://doi.acm.org/10.1145/302979.303108 Google ScholarDigital Library
- Ogata, M., Sugiura, Y., Osawa, H., and Imai, M. iRing: intelligent ring using infrared reflection. In Proceedings of the 25th annual ACM symposium on User interface software and technology (2012). ACM, New York, NY, USA, 131--136. DOI=http://dx.doi.org/10.1145/2380116.2380135 Google ScholarDigital Library
- Pedersen, E.R., McCall, K., Moran, T.P., and Halasz, F.G. Tivoli: an electronic whiteboard for informal workgroup meetings. In Proceedings of the Conference on Human Factors in Computing Systems (1993). ACM, New York, NY, USA, 391--398. DOI=10.1145/169059.169309 http://doi.acm.org/10.1145/169059.169309 Google ScholarDigital Library
- Ramakers, R., Vanacken, D., Luyten, K., Coninx, K., and Schöning, J. Carpus: a non-intrusive user identification technique for interactive surfaces. In Proceedings of the 25th annual ACM symposium on User interface software and technology (2012). ACM, New York, NY, USA, 35--44. DOI=10.1145/2380116.2380123 http://doi.acm.org/10.1145/2380116.2380123 Google ScholarDigital Library
- Ramos, G., Robertson, G., Czerwinski, M., Tan, D., Baudisch, P., Hinckley, K., and Agrawala, M. Tumble! Splat! helping users access and manipulate occluded content in 2D drawings. In Proceedings of the working conference on Advanced visual interfaces (2006). ACM, New York, NY, USA, 428--435. DOI=10.1145/1133265.1133351 http://doi.acm.org/10.1145/1133265.1133351 Google ScholarDigital Library
- Rofouei, M., Wilson, A., Brush, A.J., and Tansley. S. Your phone or mine': fusing body, touch and device sensing for multi-user device-display interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012). ACM, New York, NY, USA, 1915--1918. DOI=http://dx.doi.org/10.1145/2207676.2208332 Google ScholarDigital Library
- Strange, A. Wear this Bluetooth Ring to Control All Your Devices With Your Finger, 2014, http://mashable.com/2014/04/29/nod-bluetooth-ring/Google Scholar
- Vogel, D. and Balakrishnan, R. Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In Proceedings of the 17th annual ACM symposium on User interface software and technology (2004). ACM, New York, NY, USA, 137--146. DOI=10.1145/1029632.1029656 http://doi.acm.org/10.1145/1029632.1029656 Google ScholarDigital Library
- Vogel, D. and Casiez, G. Conté: multimodal input inspired by an artist's crayon. In Proceedings of the 24th annual ACM symposium on User interface software and technology (2011). ACM, New York, NY, USA, 357--366. DOI=http://dx.doi.org/10.1145/2047196.2047242 Google ScholarDigital Library
- Weiser, M. The computer for the 21st century. Scientific American, 1991, 265(3), 94--104. Google ScholarCross Ref
- Wigdor, D. and Wixon, D. Design Guidelines: SelfRevealing Multi-Touch Gestures. Brave NUI world: designing natural user interfaces for touch and gesture, 2011, Elsevier: 150--154.Google Scholar
- Wilkinson, G., Kharrufa, A., Hook, J., Pursglove, B., Wood, G., Haeuser, H., Hammerla, N.Y., Hodges, S., and Olivier, P. Expressy: Using a Wrist-worn Inertial Measurement Unit to Add Expressiveness to Touchbased Interactions. In Proceedings of the CHI Conference on Human Factors in Computing Systems (2016). ACM, New York, NY, USA, 2832--2844. DOI: http://dx.doi.org/10.1145/2858036.2858223 Google ScholarDigital Library
- Wu, M., and Balakrishnan, R. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proceedings of the 16th annual ACM symposium on User interface software and technology (2003). ACM, New York, NY, USA, 193--202. DOI=10.1145/964696.964718 http://doi.acm.org/10.1145/964696.964718 Google ScholarDigital Library
- Yang, X.D., Grossman, T., Wigdor, D., and Fitzmaurice, G. Magic finger: always-available input through finger instrumentation. In Proceedings of the 25th annual ACM symposium on User interface software and technology (2012). ACM, New York, NY, USA, 147--156. DOI=10.1145/2380116.2380137 http://doi.acm.org/10.1145/2380116.2380137 Google ScholarDigital Library
- Zhang, H., Yang, X.D., Ens, B., Liang, H.N., Boulanger, P., and Irani, P. See me, see you: a lightweight method for discriminating user touches on tabletop displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012). ACM, New York, NY, USA, 2327--2336. DOI=10.1145/2207676.2208392 http://doi.acm.org/10.1145/2207676.2208392 Google ScholarDigital Library
Index Terms
- Wearables as Context for Guiard-abiding Bimanual Touch
Recommendations
Thumb + Pen Interaction on Tablets
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsModern tablets support simultaneous pen and touch input, but it remains unclear how to best leverage this capability for bimanual input when the nonpreferred hand holds the tablet. We explore Thumb + Pen interactions that support simultaneous pen and ...
Lucid touch: a see-through mobile device
UIST '07: Proceedings of the 20th annual ACM symposium on User interface software and technologyTouch is a compelling input modality for interactive devices; however, touch input on the small screen of a mobile device is problematic because a user's fingers occlude the graphical elements he wishes to work with. In this paper, we present LucidTouch,...
Sensing techniques for tablet+stylus interaction
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technologyWe explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his ...
Comments