Abstract
We present insights from a gesture elicitation study conducted for TV control, during which 18 participants contributed gesture commands and rated the execution difficulty and recall likeliness of free-hand gestures for 21 television control tasks. Our study complements previous work on gesture interaction design for the TV set with the first exploration of fine-grained resolution 3-D finger movements and hand gestures. We report lower agreement rates than previous gesture studies (\({\mathcal {AR}}=.158\)) with 72.8 % recall rate and 15.8 % false positives, results that are explained by the complexity and variability of unconstrained finger and hand gestures. However, our observations also confirm previous findings, such as people preferring related gestures for dichotomous tasks and more disagreement occurring for abstract tasks, such as “open browser” or “show the list of channels” for our specific TV scenario. To reach a better understanding of our participants’ preferences for articulating finger and hand gestures, we defined five measures for Leap Motion gestures, such as gesture volume and finger-to-palm distance, which we employed to evaluate gestures performed by our participants. We also contribute a set of guidelines for practitioners interested in designing free-hand gestures for interactive TV scenarios involving similar gesture acquisition technology. We release our dataset consisting in 378 Leap Motion gestures described by fingertips position, direction, and velocity coordinates to foster further studies in the community. This first exploration of viewers’ preferences for fine-grained resolution free-hand gestures for TV control represents one more step toward designing low-effort gesture interfaces for lean-back interaction with the TV set.
Similar content being viewed by others
References
Aslan I, Uhl A, Meschtscherjakov A, Tscheligi M (2014) Mid-air authentication gestures: an exploration of authentication based on palm and finger motions. In: Proceedings of the 16th international conference on multimodal interaction (ICMI ’14). ACM, New York, NY, USA, pp 311–318
Barkhuus L (2009) Television on the internet: new practices, new viewers. In: Proceedings of CHI ’09 extended abstracts on human factors in computing systems (CHI EA ’09). ACM, New York, NY, USA, pp 2479–2488
Bernhaupt R, Obrist M, Weiss A, Beck E, Tscheligi M (2008) Trends in the living room and beyond: results from ethnographic studies using creative and playful probing. Comput Entertain 6(1), Article 5 (May 2008), 23 pages
Benford S, Greenhalgh C, Craven M (2000) Inhabited television: broadcasting interaction from within collaborative virtual environments. ACM Trans Comput Human Interact 7(4):510–547
Bobeth J, Schmehl S, Kruijff E, Deutsch S, Tscheligi M (2012) Evaluating performance and acceptance of older adults using freehand gestures for TV menu control. In: Proceedings of the 10th European conference on interactive TV and video (EuroITV’12). ACM, New York, NY, USA, pp 35–44
Cesar P, Chorianopoulos K (2009) The evolution of TV systems, content, and users toward interactivity. Found Trends Human Comput Interact 2(4):279–373
Cesar P, Bulterman, DC, Jansen AJ (2008) Usages of the secondary screen in an interactive television environment: control, enrich, share, and transfer television content. In: Tscheligi M, Obrist M, Lugmayr A (eds) Proceedings of the 6th European conference on changing television environments (EuroITV ’08). Springer, Berlin, pp 168–177
Christian S, Alves J, Ferreira A, Jesus D, Freitas R, Vieira N (2014) Volcano salvation: interaction through gesture and head tracking. In: Proceedings of CHI ’14 extended abstracts on human factors in computing systems (CHI EA ’14). ACM, New York, NY, USA, pp 297–300
Codd-Downey R, Stuerzlinger W (2014) LeapLook: a free-hand gestural travel technique using the Leap Motion finger tracker. In: Proceedings of the 2nd ACM symposium on spatial user interaction (SUI ’14). ACM, New York, NY, USA, pp 153–153
Cooke NJ (1994) Varieties of knowledge elicitation techniques. Int J Hum Comput Stud 41(6):801–849
Dezfuli N, Khalilbeigi M, Huber J, Mller F, Mhlhuser M, Palm RC (2012) Imaginary palm-based remote control for eyes-free television interaction. In: Proceedings of the 10th European conference on interactive TV and video (EuroITV ’12). ACM, New York, NY, USA, pp 27–34
Dezfuli N, Khalilbeigi M, Mhlhuser M, Geerts D (2011) A study on interpersonal relationships for social interactive television. In: Proceddings of the 9th international conference on interactive television (EuroITV ’11). ACM, New York, NY, USA, pp 21–24
Dias T, Variz M, Jorge P, Jesus R (2013) Gesture interaction system for social web applications on smart TVs. In: Proceedings of the 10th conference on open research areas in information retrieval (OAIR ’13), pp 225–226
Elsweiler D, Mandl S, Lunn BK (2010) Understanding casual-leisure information needs: a diary study in the context of television viewing. In: Proceedings of the 3rd symposium on information interaction in context (IIiX ’10). ACM, New York, NY, USA, pp 25–34
Fanciulli M (2008) Principles of entertainment in inhabited television. In: Proceedings of the working conference on advanced visual interfaces (AVI ’08). ACM, New York, NY, USA, pp 5–12
Freeman D, Vennelakanti R, Madhvanath S. Freehand pose-based gestural interaction: studies and implications for interface design. In: Proceedings of the 4th international conference on intelligent human computer interaction (IHCI ’12), pp 1–6
Freeman WT, Weissman CD (1995) Television control by hand gestures. In: Proceedings of the IEEE international workshop on automatic face and gesture recognition
Hantrakul L Kaczmarek K (2014) Implementations of the Leap Motion device in sound synthesis and interactive live performance. In: Proceedings of the 2014 international workshop on movement and computing (MOCO ’14). ACM, New York, NY, USA, 4 pages
Jeong S, Song T, Kwon K, Jeon JW (2012) TV remote control using human hand motion based on optical flow system. In: Proceedings of the 12th international conference on computational science and its applications (ICCSA’12). Springer, Berlin, Heidelberg, pp 311–323
Jones LA, Lederman SJ (2006) Human hand function. Oxford University Press Inc, New York
Juhlin O, Önnevall E (2013) On the relation of ordinary gestures to TV screens: General lessons for the design of collaborative interactive techniques. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’13). ACM, New York, NY, USA, pp 919–930
Kane SK, Wobbrock JO, Ladner RE (2011) Usable gestures for blind people: Understanding preference and performance. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’11). ACM, New York, NY, USA, pp 413–422
Kendall MG, Babington Smith B (1939) The problem of m rankings. Ann Math Stat 10(3):275287
Kray C, Nesbitt D, Dawson J, Rohs M (2010) User-defined gestures for connecting mobile phones, public displays, and tabletops. In: Proceedings of the 12th international conference on human–computer interaction with mobile devices and services (MobileHCI ’10). ACM, New York, NY, USA, pp 239–248
Kühnel C, Westermann T, Hemmert F, Kratz S, Müller A, Möller S (2011) I’m home: defining and evaluating a gesture set for smart-home control. Int J Hum Comput Stud 69(11):693–704
Kurtenbach G, Buxton W (1991) Issues in combining marking and direct manipulation techniques. In: Proceedings of the 4th annual ACM symposium on user interface software and technology (UIST ’91). ACM, New York, NY, USA, pp 137–144
Leap Motion. https://www.leapmotion.com/
Lim JH, Jo C, Kim D-H (2012) Analysis on user variability in gesture interaction. Convergence and hybrid information technology. Springer LNCS 7425, pp 295–302
Magic Remote. http://www.lg.com/global/magicremote/
Macek T, Labský M, Vystrčil J, Luksch D, Kašparová T, Kunc L, Kleindienst J (2014) Interactive car owner’s manual user study. In: Proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications (AutomotiveUI ’14). ACM, New York, NY, USA, Article 29, 4 pages
Malizia A, Bellucci A (2012) The artificiality of natural user interfaces. Commun ACM 55(3):36–38
Matos N, Santos A, Vasconcelos A (2014) Kinteract: a multi-sensor physical rehabilitation solution based on interactive games. In: Proceedings of the 8th international conference on pervasive computing technologies for healthcare (PervasiveHealth ’14). ICST, Brussels, Belgium, pp 350–353
Mauney D, Howarth J, Wirtanen A, Capra M (2010) Cultural similarities and differences in user-defined gestures for touchscreen user interfaces. In: Proceedings of CHI ’10 extended abstracts on human factors in computing systems (CHI EA ’10). ACM, New York, NY, USA, pp 4015–4020
Morris MR, Wobbrock JO, Wilson AD (2010) Understanding users’ preferences for surface gestures. In: Proceedings of graphics interface 2010 (GI ’10). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, pp 261–268
Morris MR (2012) Web on the wall: insights from a multimodal interaction elicitation study. In: Proceedings of the 2012 ACM international conference on interactive tabletops and surfaces (ITS ’12). ACM, New York, NY, USA, pp 95–104
Nadamoto A, Tanaka K (2005) Complementing your TV-viewing by web content automatically-transformed into TV-program-type content. In: Proceedings of the 13th annual ACM international conference on multimedia (MULTIMEDIA ’05). ACM, New York, NY, USA, pp 41–50
Norman DA (2010) Natural user interfaces are not natural. Interactions 17(3):6–10
Olsen DR, Partridge B, Lynn S (2010) Time warp sports for Internet television. ACM transactions on computer–human interaction, vol 17, no 4, Article 16, (December 2010), 37 pages
Phillips Ambilight. http://www.philips.co.uk/c/televisions/33092/cat/
Philips uWand. http://www.uwand.com/
Ren G, O’Neill E (2013) Freehand gestural text entry for interactive TV. In: Proceedings of the 11th European conference on interactive TV and Video (EuroITV ’13). ACM, New York, NY, USA, pp 121–130
Roy Q, Malacria S, Guiard Y, Lecolinet E, Eagan J (2013) Augmented letters: mnemonic gesture-based shortcuts. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’13). ACM, New York, NY, USA, pp 2325–2328
Ruiz J, Li Y, Lank E (2011) User-defined motion gestures for mobile interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’11). ACM, New York, NY, USA, pp 197–206
Samsung Smart TV: TV Gesture Book. http://www.samsung.com/global/microsite/tv/common/guide_book_5p_vi/waving.html
Sen TK, Sinko MW, Wilson AT, Hoque ME (2014) M.I.D.A.S. touch: magnetic interactive device for alternative sight through touch. In: Proceedings of the 16th international ACM SIGACCESS conference on computers and accessibility (ASSETS ’14). ACM, New York, NY, USA, pp 307–308
Torok KS, Baker NA, Lucas M, Domsic RT, Boudreau R, Medsger TA Jr. (2010) Reliability and validity of the delta finger-to-palm (FTP), a new measure of finger range of motion in systemic sclerosis. Clin Exp Rheumatol 28(2 Suppl. 58):S28–S36
Vatavu RD, Pentiuc SG (2008) Interactive coffee tables: interfacing TV within an intuitive, fun and shared experience. In: Proceedings of the 6th European interactive TV conference (EuroITV ’08). LNCS 5066, Springer, Berlin Heidelberg, pp 183–187
Vatavu RD, Pentiuc SG (2008) Multi-level representation of gesture as command for human–computer interaction. Comput Inform 27(6):837–851
Vatavu RD (2012) User-defined gestures for free-hand TV control. In: Proceedings of the 10th European conference on interactive TV and Video (EuroITV ’12). ACM, New York, NY, USA, pp 45–48
Vatavu RD (2012) Point & click mediated interactions for large home entertainment displays. Multimed Tools Appl 59(1):113–128
Vatavu RD (2013) A comparative study of user-defined handheld vs. freehand gestures for home entertainment environments. J Ambient Intell Smart Environ 5(2):187–211
Vatavu RD (2013) There’s a world outside your TV: Exploring interactions beyond the physical TV screen. In: Proceedings of the 11th European conference on interactive TV and video (EuroITV ’13). ACM, New York, NY, USA, pp 143–152
Vatavu RD, Zaiti IA (2014) Leap gestures for TV: insights from an elicitation study. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video (TVX ’14). ACM, New York, NY, USA, pp 131–138
Vatavu RD, Mancas M (2014) Visual attention measures for multi-screen TV. In: Proceedings of the 2014 ACM international conference on interactive experiences for TV and online video (TVX ’14). ACM, New York, NY, USA, pp 111–118
Vatavu RD, Wobbrock JO (2015) Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit. In: Proceedings of CHI ’15, the 33rd ACM SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA
Vatavu RD (2015) Audience silhouettes: peripheral awareness of synchronous audience kinesics for social television. In: Proceedings of the 2015 ACM international conference on interactive experiences for TV and online video (TVX ’15). ACM, New York
Vatavu RD, Mancas M (2015) Evaluating visual attention for multi-screen TV: measures, toolkit, and experimental findings. Personal and Ubiquitous Computing. Springer, New York
Wobbrock JO, Morris MR, Wilson AD (2009) User-defined gestures for surface computing. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’09). ACM, New York, NY, USA, pp 1083–1092
Wobbrock JO, Aung HH, Rothrock B, Myers BA (2005) Maximizing the guessability of symbolic input. In: Proceedings of CHI ’05 extended abstracts on human factors in computing systems (CHI EA ’05). ACM, New York, NY, USA, pp 1869–1872
Wu H, Wang J (2012) User-defined body gestures for TV-based applications. In: Proceedings of the 2012 4th international conference on digital home (ICDH ’12). IEEE Computer Society, Washington, DC, USA, pp 415–420
Acknowledgments
This research was supported by the liFe-StaGE project, “Multimodal Feedback for Supporting Gestural Interaction in Smart Environments”, No. 740/2014, co-funded by UEFISCDI, Romania and OeAD, Austria. The infrastructure was provided by the University Stefan cel Mare of Suceava and was partially supported from the project “Integrated center for research, development and innovation in Advanced Materials, Nanotechnologies, and Distributed Systems for fabrication and control”, No. 671/09.04.2015, Sectoral Operational Program for Increase of the Economic Competitiveness co-funded from the European Regional Development Fund.
Author information
Authors and Affiliations
Corresponding author
Appendix: Complete set of gestures collected during the elicitation experiment
Appendix: Complete set of gestures collected during the elicitation experiment
No. | Referent | Participants’ gesture proposals |
---|---|---|
1 | Open
| Open palm, wave hand, move hand up, move fist up followed by wrist rotation, move hand down, move palm away from the body, thumbs-up |
2 | Close
| Close palm, wave hand, close hand into pinch, move hand down, fist moves up and down twice, move fist up followed by wrist rotation, hand performs the “go away” cultural gesture, draw “X”, perform click in mid-air with the index finger |
3 | Next
| Move hand from right to left and Move hand from left to right, thumbs-up while move hand to the right, “OK” sign with the thumb, moving the index finger from left to right and then click in mid-air. Please note the two options in terms of movement direction (left-to-right and right-to-left) that correspond to two distinct metaphors: moving the viewing window (as it happens with scrolling actions and traditional GUI) and moving the items themselves [37] |
4 | Previous
| Move hand from left to right and Move hand from right to left, thumbs-up while move hand to the left, perform double click in mid-air with the index finger, draw circle clockwise. Please note the two options in terms of movement direction (left-to-right and right-to-left) that correspond to moving the viewing window (as it happens with the scrolling action and traditional GUI) or the items themselves [37] |
5 | Volume up
| Move hand up, hand in pinch pose expanding fingers, thumbs-up with hand moving to the right, thumbs-up with hand moving up twice, rotate imaginary button to the right, draw the “+” symbol in mid-air, open hand from thumb-index pinch, draw triangle, open palm, draw circle clockwise, hand performs the “go away” cultural gesture |
6 | Volume down
| Move hand down, hand changes pose from open palm to index-thumb pinch, thumbs-up with hand moving to the left, thumbs-up with hand moving down twice, rotate imaginary button to the left, move hand from left to right, draw triangle, hand closing fingers into pinch, draw circle counterclockwise, hand performs the “come here” cultural gesture |
7 | Volume mute
| Close fingers into pinch, fist followed by extending the little finger, open palm, thumbs-down to thumbs-up, draw letter “X”, close fist, open palm to index-thumb pinch, pinch with the thumb and little finger, open palm facing down while hand moves from left to right, draw crossed zero, draw circle counterclockwise, move hand down, move hand from left to right, move hand from right to left, move palm away from the body |
8 | Open menu
| Draw letter “M”, draw small “o”, small “m”, rotate wrist, move hand up, open fingers from the thumb-index pinch, perform click in mid-air with the index finger, hand in pinch pose opens fingers, wave hand, move hand down |
9 | Hide menu
| Hand wave, closed fist opens index and little finger, close fist, open hand rotates, hand changes from open palm to pinch with the thumb and index finger, move hand up, hand performs the “go away” cultural gesture, rotate palm from left to right, hand performs the “come here” cultural gesture |
10 | Help
| Draw symbol “?” or letter “H” in mid-air, hand performs the “come here” cultural gesture, hand performs the “peace” sign, wave fingers, finger snapping, move hand toward the body, wave hand, pinch and wave |
11 | Yes
| Thumbs-up, draw the “check” sign, three fingers down, draw letter “Y” in mid-air, thumbs-up rotated 90 degrees to the right, perform a click in mid-air with the index finger, open palm, pinch followed by thumbs-up |
12 | No
| Hand wave, close fist, three fingers down, move hand up, thumbs-up rotate from right to up, draw letter “X” in mid-air, move hand down |
13 | Go to favorite channel
| Show index finger, thumbs-up, perform the “come here” cultural gesture, pinch between the thumb and little finger, move index finger down and up, draw the “check” sign, open palm, fingers snapping, draw star, thumbs-up rotated to right, move hand down and back then and forward and down |
14 | Go to 2nd favorite channel
| Show two fingers (index and middle), thumbs-up rotated 90 degrees to the right, pinch with the thumb and the ring finger, two fingers up, “peace” sign left to right, fingers snapping twice, draw digit “2” in mid-air, two fingers up followed by thumbs-up, move hand forward two times |
15 | Go to random channel
| Palm facing down rotates to palm up, draw circle clockwise, draw circle anticlockwise, draw circle anticlockwise three times, close fingers, open palm rotates down to up, wave with fingers, perform a click in mid-air with the index finger, open palm, draw letter “R”, wave hand, draw the universal quantifier symbol “\(\forall\)” |
16 | Go to channel #7
| Draw digit “7” in mid-air, various finger configurations to indicate number “7” |
17 | Go to channel #27
| Draw number “27” in mid-air, various finger configurations to indicate number “27” |
18 | TV Guide
| Draw letter “G” in mid-air, move hand up, thumbs-up, draw letter “M”, move palm in front of the body, move hand down, move hand from left to right, show the index finger, pinch followed by wave, open palm |
19 | Show channels list
| Draw square in mid-air, four fingers move down, open palm, draw letters “C” and “L”, move hand down, hand performs the “come here” cultural gesture, wave hand, move hand up, pinch followed by wave, perform double click in mid-air with the index finger, move hand toward the body, draw letter “L” in mid-ar, draw circle anticlockwise |
20 | Last channel
| Draw circle anti-clockwise, index and middle fingers move to the left, fingers snapping, two fingers up, thumbs-up moving back, hand rotates an imaginary button to the left, wave hand |
21 | Open browser
| Draw symbol “@”, move palm down twice, close fist, draw letter “W” three times as in “WWW”, hand performs the “go away” cultural gesture, pinch twice, pinch followed by thumbs-up, move index finger from left to right, move hand from right to left, move hand down, move palm in front of the body, show three fingers |
Rights and permissions
About this article
Cite this article
Zaiţi, IA., Pentiuc, ŞG. & Vatavu, RD. On free-hand TV control: experimental results on user-elicited gestures with Leap Motion. Pers Ubiquit Comput 19, 821–838 (2015). https://doi.org/10.1007/s00779-015-0863-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-015-0863-y