Thomas Kosch
Mariam Hassib
Paweł W. Woźniak
Daniel Buschek
Florian Alt
ABSTRACT
A common objective for context-aware computing systems is to predict how user interfaces impact user performance regarding their cognitive capabilities. Existing approaches such as questionnaires or pupil dilation measurements either only allow for subjective assessments or are susceptible to environmental influences and user physiology. We address these challenges by exploiting the fact that cognitive workload influences smooth pursuit eye movements. We compared three trajectories and two speeds under different levels of cognitive workload within a user study (N=20). We found higher deviations of gaze points during smooth pursuit eye movements for specific trajectory types at higher cognitive workload levels. Using an SVM classifier, we predict cognitive workload through smooth pursuit with an accuracy of 99.5% for distinguishing between low and high workload as well as an accuracy of 88.1% for estimating workload between three levels of difficulty. We discuss implications and present use cases of how cognition-aware systems benefit from inferring cognitive workload in real-time by smooth pursuit eye movements.
Supplemental Material
- Yomna Abdelrahman, Eduardo Velloso, Tilman Dingler, Albrecht Schmidt, and Frank Vetere. 2017. Cognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 33 (Sept. 2017), 20 pages. Google Scholar
Digital Library
- Sylvia Ahern and Jackson Beatty. 1979. Pupillary responses during information processing vary with Scholastic Aptitude Test scores. Science 205, 4412 (1979), 1289--1292.Google Scholar
- Ulf Ahlstrom and Ferne J Friedman-Berg. 2006. Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics 36, 7 (2006), 623--636.Google ScholarCross Ref
- Alan D. Baddeley and Graham Hitch. 1974. Working Memory. Psychology of Learning and Motivation, Vol. 8. Academic Press, 47 -- 89.Google Scholar
- Graham R Barnes. 2008. Cognitive processes involved in smooth pursuit eye movements. Brain and cognition 68, 3 (2008), 309--326.Google Scholar
- Nikolaus Bee and Elisabeth André. 2008. Writing with your eye: A dwell time free writing system adapted to the nature of human eye gaze. In International Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems. Springer, 111--122. Google ScholarDigital Library
- Simone Benedetto, Marco Pedrotti, Luca Minin, Thierry Baccino, Alessandra Re, and Roberto Montanari. 2011. Driver workload and eye blink duration. Transportation research part F: traffic psychology and behaviour 14, 3 (2011), 199--208. www.hcilab.org/your_eyes_tell_data_set - last access 2018-0108Google Scholar
- Chris Berka, Daniel J Levendowski, Michelle N Lumicao, Alan Yau, Gene Davis, Vladimir T Zivkovic, Richard E Olmstead, Patrice D Tremoulet, and Patrick L Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine 78, Supplement 1 (2007), B231--B244.Google Scholar
- Anne-Marie Brouwer, Maarten A Hogervorst, Jan BF Van Erp, Tobias Heffelaar, Patrick H Zimmerman, and Robert Oostenveld. 2012. Estimating workload using EEG spectral power and ERPs in the n-back task. Journal of neural engineering 9, 4 (2012), 045008.Google ScholarCross Ref
- Andreas Bulling and Thorsten O Zander. 2014. Cognition-aware computing. IEEE Pervasive Computing 13, 3 (2014), 80--83.Google ScholarCross Ref
- Han Collewijn and Ernst P Tamminga. 1984. Human smooth and saccadic eye movements during voluntary pursuit of different target motions on different backgrounds. The Journal of physiology 351 (1984), 217.Google ScholarCross Ref
- R Contreras, J Ghajar, S Bahar, and M Suh. 2011. Effect of cognitive load on eye-target synchronization during smooth pursuit eye movement. Brain research 1398 (2011), 55--63.Google Scholar
- Alexander De Luca, Roman Weiss, and Heiko Drewes. 2007. Evaluation of eye-gaze interaction methods for security enhanced PIN-entry. In Proceedings of the 19th australasian conference on computer-human interaction: Entertaining user interfaces. ACM, 199--202. Google ScholarDigital Library
- Tilman Dingler. 2016. Cognition-aware Systems As Mobile Personal Assistants. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 1035--1040. Google ScholarDigital Library
- Andrew Duchowski. 2017. Eye tracking methodology: Theory and practice. Vol. 373. Springer Science & Business Media. Google ScholarDigital Library
- Andrew T Duchowski. 2002. A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers 34, 4 (2002), 455--470.Google ScholarCross Ref
- Augusto Esteves, Eduardo Velloso, Andreas Bulling, and Hans Gellersen. 2015. Orbits: gaze interaction for smart watches using smooth pursuit eye movements. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 457--466. Google ScholarDigital Library
- DMA Gronwall. 1977. Paced auditory serial-addition task: a measure of recovery from concussion. Perceptual and motor skills 44, 2 (1977), 367--373.Google Scholar
- Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (2006), 904--908.Google ScholarCross Ref
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183.Google Scholar
- Eckhard H Hess and James M Polt. 1964. Pupil size in relation to mental activity during simple problem-solving. Science 143, 3611 (1964), 1190--1192.Google Scholar
- Thomas E Hutchinson, K Preston White, Worthy N Martin, Kelly C Reichert, and Lisa A Frey. 1989. Human-computer interaction using eye-gaze input. IEEE Transactions on systems, man, and cybernetics 19, 6 (1989), 1527--1534.Google ScholarCross Ref
- Poika Isokoski. 2000. Text input methods for eye trackers using off-screen targets. In Proceedings of the 2000 symposium on Eye tracking research & applications. ACM, 15--21. Google ScholarDigital Library
- RJ Jacob and Keith S Karn. 2003. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind 2, 3 (2003), 4.Google Scholar
- Robert J. K. Jacob. 1990. What You Look at is What You Get: Eye Movement-based Interaction Techniques. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '90). ACM, New York, NY, USA, 11--18. Google ScholarDigital Library
- Marcel Adam Just and Patricia A Carpenter. 1976. Eye fixations and cognitive processes. Cognitive Psychology 8, 4 (1976), 441 -- 480.Google ScholarCross Ref
- Michael J Kane, Andrew RA Conway, Timothy K Miura, and Gregory JH Colflesh. 2007. Working memory, attention control, and the N-back task: a question of construct validity. Journal of Experimental Psychology: Learning, Memory, and Cognition 33, 3 (2007), 615.Google ScholarCross Ref
- Mohamed Khamis, Florian Alt, and Andreas Bulling. 2015. A field study on spontaneous gaze-based interaction with a public display using pursuits. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers. ACM, 863--872. Google ScholarDigital Library
- Mohamed Khamis, Ozan Saltuk, Alina Hang, Katharina Stolz, Andreas Bulling, and Florian Alt. 2016. TextPursuits: Using Text for Pursuits-Based Interaction and Calibration with Public Displays. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM. Google ScholarDigital Library
- Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. 2008. Measuring the task-evoked pupillary response with a remote eye tracker. In Proceedings of the 2008 symposium on Eye tracking research & applications. ACM, 69--72. Google ScholarDigital Library
- J. Kranjec, S. Beguš, G. Geršak, and J. Drnovšek. 2014. Non-contact heart rate and heart rate variability measurements: A review. Biomedical Signal Processing and Control 13, Supplement C (2014), 102 -- 112.Google ScholarCross Ref
- Jan-Louis Kruger, Esté Hefer, and Gordon Matthew. 2013. Measuring the impact of subtitles on cognitive load: Eye tracking and dynamic audiovisual texts. In Proceedings of the 2013 Conference on Eye Tracking South Africa. ACM, 62--66. Google ScholarDigital Library
- R John Leigh and David S Zee. 2015. The neurology of eye movements. Vol. 90. Oxford University Press, USA.Google Scholar
- Yulan Liang and John D Lee. 2008. Driver cognitive distraction detection using eye movements. In Passive Eye Monitoring. Springer, 285--300.Google Scholar
- Dachuan Liu, Bo Dong, Xing Gao, and Haining Wang. 2015. Exploiting Eye Tracking for Smartphone Authentication. Springer International Publishing, Cham, 457--477.Google Scholar
- Dillon James Lohr and Oleg V. Komogortsev. 2017. A Comparison of Smooth Pursuit- and Dwell-based Selection at Multiple Levels of Spatial Accuracy. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2760--2766. Google ScholarDigital Library
- Päivi Majaranta and Kari-Jouko Räihä. 2002. Twenty years of eye typing: systems and design issues. In Proceedings of the 2002 symposium on Eye tracking research & applications. ACM, 15--22. Google ScholarDigital Library
- Stefan Mattes and Anders Hallén. 2009. Surrogate distraction measurement techniques: The lane change test. Driver distraction: Theory, effects, and mitigation (2009), 107--121.Google Scholar
- Bruce Mehler, Bryan Reimer, and JA Dusek. 2011. MIT AgeLab delayed digit recall task (n-back). Cambridge, MA: Massachusetts Institute of Technology (2011).Google Scholar
- Takehiko Ohno. 1998. Features of eye gaze interface for selection tasks. In Computer Human Interaction, 1998. Proceedings. 3rd Asia Pacific. IEEE, 176--181. Google ScholarDigital Library
- Annie Pauzié. 2008. A method to assess the driver mental workload: The driving activity load index (DALI). IET Intelligent Transport Systems 2, 4 (2008), 315--322.Google ScholarCross Ref
- Peter Peltonen, Esko Kurvinen, Antti Salovaara, Giulio Jacucci, Tommi Ilmonen, John Evans, Antti Oulasvirta, and Petri Saarikko. 2008. It's Mine, Don'T Touch!: Interactions at a Large Multi-touch Display in a City Centre. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 1285--1294. Google ScholarDigital Library
- Vsevolod Peysakhovich. 2016. Study of pupil diameter and eye movements to enhance flight safety. Etude de diamètre pupillaire et de mouvements oculaires pour la sécurité aérienne. Ph.D. Dissertation. Université de Toulouse, Université Toulouse III-Paul Sabatier.Google Scholar
- Ken Pfeuffer, Melodie Vidal, Jayson Turner, Andreas Bulling, and Hans Gellersen. 2013. Pursuit calibration: Making gaze calibration less tedious and more flexible. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 261--270. Google ScholarDigital Library
- Bastian Pfleging, Drea K Fekety, Albrecht Schmidt, and Andrew L Kun. 2016. A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5776--5788. Google ScholarDigital Library
- Alex Poole and Linden J Ball. 2006. Eye tracking in HCI and usability research. Encyclopedia of human computer interaction 1 (2006), 211--219.Google Scholar
- Dale Purves, George J Augustine, David Fitzpatrick, Lawrence C Katz, Anthony-Samuel LaMantia, James O McNamara, and S Mark Williams. 2001. Types of eye movements and their functions. (2001).Google Scholar
- Carlos H Schenck, Scott R Bundlie, Andrea L Patterson, and Mark W Mahowald. 1987. Rapid eye movement sleep behavior disorder: a treatable parasomnia affecting older adults. Jama 257, 13 (1987), 1786--1789.Google ScholarCross Ref
- Simon Schenk, Marc Dreiser, Gerhard Rigoll, and Michael Dorr. 2017. GazeEverywhere: Enabling Gaze-only User Interaction on an Unmodified Desktop PC in Everyday Scenarios. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3034--3044. Google ScholarDigital Library
- Linda E Sibert and Robert JK Jacob. 2000. Evaluation of eye gaze interaction. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 281--288. Google ScholarDigital Library
- Eva Siegenthaler, Francisco M Costela, Michael B McCamy, Leandro L Di Stasi, Jorge Otero-Millan, Andreas Sonderegger, Rudolf Groner, Stephen Macknik, and Susana Martinez-Conde. 2014. Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes. European Journal of Neuroscience 39, 2 (2014), 287--294.Google ScholarCross Ref
- David L Sparks. 2002. The brainstem control of saccadic eye movements. Nature Reviews Neuroscience 3, 12 (2002), 952--964.Google ScholarCross Ref
- John Stemberger, Robert S Allison, and Thomas Schnell. 2010. Thermal imaging as a way to classify cognitive workload. In Computer and Robot Vision (CRV), 2010 Canadian Conference on. IEEE, 231--238. Google ScholarDigital Library
- Els Stuyven, Koen Van der Goten, André Vandierendonck, Kristl Claeys, and Luc Crevits. 2000. The effect of cognitive load on saccadic eye movements. Acta psychologica 104, 1 (2000), 69--85.Google Scholar
- Yi-Fang Tsai, Erik Viirre, Christopher Strychacz, Bradley Chase, and Tzyy-Ping Jung. 2007. Task performance and eye activity: predicting behavior relating to cognitive workload. Aviation, space, and environmental medicine 78, Supplement 1 (2007), B176--B185.Google Scholar
- Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. 2004. Support vector machine learning for interdependent and structured output spaces. In Proceedings of the twenty-first international conference on Machine learning. ACM, 104. Google ScholarDigital Library
- Outi Tuisku, Päivi Majaranta, Poika Isokoski, and Kari-Jouko Räihä. 2008. Now Dasher! Dash away!: longitudinal study of fast text entry by Eye Gaze. In Proceedings of the 2008 symposium on Eye tracking research & applications. ACM, 19--26. Google ScholarDigital Library
- Marilyn L Turner and Randall W Engle. 1989. Is working memory capacity task dependent? Journal of memory and language 28, 2 (1989), 127--154.Google ScholarCross Ref
- Eduardo Velloso, Markus Wirth, Christian Weichel, Augusto Esteves, and Hans Gellersen. 2016. AmbiGaze: Direct Control of Ambient Devices by Gaze. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 812--817. Google ScholarDigital Library
- Trent W. Victor, Joanne L. Harbluk, and Johan A. Engström. 2005. Sensitivity of eye-movement measures to in-vehicle task difficulty. Transportation Research Part F: Traffic Psychology and Behaviour 8, 2 (2005), 167 -- 190. The relationship between distraction and driving performance: towards a test regime for in-vehicle information systemsIn-vehicle information systems.Google ScholarCross Ref
- Mélodie Vidal, Andreas Bulling, and Hans Gellersen. 2013. Pursuits: spontaneous interaction with displays based on smooth pursuit eye movement and moving targets. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 439--448. Google ScholarDigital Library
- Daniel Vogel and Ravin Balakrishnan. 2004. Interactive Public Ambient Displays: Transitioning from Implicit to Explicit, Public to Personal, Interaction with Multiple Users. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, New York, NY, USA, 137--146. Google ScholarDigital Library
- Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. 2016. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann. Google ScholarDigital Library
- Johannes Zagermann, Ulrike Pfeil, Harald Reiterer, Yunlong Wang, Ulrike Pfeil, Harald Reiterer, Johannes Zagermann, Ulrike Pfeil, Roman Rädle, Hans-Christian Jetter, and others. 2015. Measuring Cognitive Load using Eye Tracking Technology in Visual Computing. Proceedings of BELIV'16 (2015), 259--260. Google ScholarDigital Library
- Yanxia Zhang, Andreas Bulling, and Hans Gellersen. 2013. SideWays: a gaze interface for spontaneous interaction with situated displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 851--860. Google ScholarDigital Library
- Josef Zihl, D Von Cramon, and Norbert Mai. 1983. Selective disturbance of movement vision after bilateral brain damage. Brain 106, 2 (1983), 313--340.Google ScholarCross Ref
Index Terms
- Your Eyes Tell: Leveraging Smooth Pursuit for Assessing Cognitive Workload
Recommendations
Look into my Eyes: Using Pupil Dilation to Estimate Mental Workload for Task Complexity Adaptation
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsCognition-aware systems acquire physiological data to derive implications about physical and mental states. Pupil dilation has recently attracted attention in the HCI community as an indicator for mental workload. The impact of mental workload on ...
Just blink your eyes: a head-free gaze tracking system
CHI EA '03: CHI '03 Extended Abstracts on Human Factors in Computing SystemsWe propose a head-free, easy-setup gaze tracking system designed for a gaze-based Human-Computer Interaction. Our system enables the user to interact with the computer soon after catching the user's eye blinks. The user can move his/her head freely ...
Comparing Dwell time, Pursuits and Gaze Gestures for Gaze Interaction on Handheld Mobile Devices
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsGaze is promising for hands-free interaction on mobile devices. However, it is not clear how gaze interaction methods compare to each other in mobile settings. This paper presents the first experiment in a mobile setting that compares three of the most ...
Comments