Bastian Pfleging
Albrecht Schmidt
Andrew L. Kun
ABSTRACT
In this paper, we present a proof-of-concept approach to estimating mental workload by measuring the user's pupil diameter under various controlled lighting conditions. Knowing the user's mental workload is desirable for many application scenarios, ranging from driving a car, to adaptive workplace setups. Typically, physiological sensors allow inferring mental workload, but these sensors might be rather uncomfortable to wear. Measuring pupil diameter through remote eye-tracking instead is an unobtrusive method. However, a practical eye-tracking-based system must also account for pupil changes due to variable lighting conditions. Based on the results of a study with tasks of varying mental demand and six different lighting conditions, we built a simple model that is able to infer the workload independently of the lighting condition in 75% of the tested conditions.
Supplemental Material
- Ulf Ahlstrom and Ferne J. Friedman-Berg. 2006. Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics 36, 7 (Jul. 2006), 623--636. DOI: http://dx.doi.org/10.1016/j.ergon.2006.04.002Google Scholar
Cross Ref
- Amanda N. Baraldi and Craig K. Enders. 2010. An introduction to modern missing data analyses. Journal of School Psychology 48, 1 (Feb. 2010), 5--37. DOI: http://dx.doi.org/10.1016/j.jsp.2009.10.001Google ScholarCross Ref
- Jackson Beatty. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin 91, 2 (Mar. 1982), 276--292. DOI: http://dx.doi.org/10.1037/0033--2909.91.2.276Google ScholarCross Ref
- Simone Benedetto, Marco Pedrotti, Luca Minin, Thierry Baccino, Alessandra Re, and Roberto Montanari. 2011. Driver workload and eye blink duration. Transportation Research Part F: Traffic Psychology and Behavior 14, 3 (May 2011), 199--208. DOI: http://dx.doi.org/10.1016/j.trf.2010.12.001Google ScholarCross Ref
- Anne-Marie Brouwer, Maarten A. Hogervorst1, Jan B. F. van Erp, Tobias Heffelaar, Patrick H. Zimmerman, and Robert Oostenveld. 2012. Estimating workload using EEG spectral power and ERPs in the n-back task. Journal of Neural Engineering 9, 4, Article 045008 (Jul. 2012), 045008, 14 pages. DOI: http://dx.doi.org/10.1088/1741--2560/9/4/045008Google ScholarCross Ref
- Joseph F. Coughlin, Bryan Reimer, and Bruce Mehler. 2009. Driver Wellness, Safety & the Development of an AwareCar. MIT AgeLab White Paper. Massachusetts Institute of Technology, Cambridge, MA. http://web.mit.edu/reimer/www/pdfs/coughlin_wellness_2009.pdfGoogle Scholar
- Andrew T. Duchowski. 2002. A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers 34, 4 (Nov. 2002), 455--470. DOI: http://dx.doi.org/10.3758/BF03195475Google ScholarCross Ref
- Christine Fogarty, and John A. Stern, 1989. Eye movements and blinks: Their relationship to higher cognitive processes. International Journal of Psychophysiology 8, 1 (Sep. 1983), 35--42. http://dx.doi.org/10.1016/0167--8760(89)90017--2Google ScholarCross Ref
- Thomas M. Gable, Andrew L. Kun, Bruce N. Walker, and Riley J. Winton. 2015. Comparing heart rate and pupil size as objective measures of workload in the driving context: initial look. In Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '15). ACM, NY, NY, USA, 20--25. DOI: http://doi.acm.org/10.1145/2809730.2809745 Google ScholarDigital Library
- Alan Gevins, and Michael E. Smith. 2003. Neurophysiological measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomics Science 4, 1--2 (2003), 113--131. DOI: http://dx.doi.org/1080/14639220210159717Google ScholarCross Ref
- Audrey Girouard and Erin Treacy Solovey and Robert J.K. Jacob. 2013. Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. International Journal of Autonomous and Adaptive Communication Systems 6,1 (2013), 26--44. DOI: http://dx.doi.org/10.1504/IJAACS.2013.050689 Google ScholarDigital Library
- D. M. A. Gronwall. 1977. Paced Auditory Serial-Addition Task: A Measure of Recovery From Concussion. Perceptual and Motor Skills 44 (1977), 367--373. DOI: http://dx.doi.org/10.2466/pms.1977.44.2.367Google ScholarCross Ref
- Jennifer A. Healey, and Rosalind W. Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intelligent Transportation Systems 6, 2 (June 2005), 156--166. DOI: http://dx.doi.org/10.1109/TITS.2005.848368 Google ScholarDigital Library
- Peter A. Heeman, Tomer Meshorer, Andrew L. Kun, Oskar Palinko, and Zeljko Medenica. 2013. Estimating cognitive load using pupil diameter during a spoken dialogue task. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '13). ACM, NY, NY, USA, 242--245. DOI: http://dx.doi.org/10.1145/2516540.2516570 Google ScholarDigital Library
- Daniel Kahneman, D., and Jackson Beatty. 1966. Pupil Diameter and Load on Memory. Science 154, 3756 (Dec. 1966), 1583--1585. DOI: http://dx.doi.org/10.1126/science.154.3756.1583Google ScholarCross Ref
- Daniel Kahneman, Linda Onuska, L., and Ruth E. Wolman. 1968. Effects of pupillary response in a short-term memory task. The Quarterly Journal of Experimental Psychology 20, 3 (1968), 309--311. DOI: http://dx.doi.org/10.1080/14640746808400168Google ScholarCross Ref
- W. K. Kirchner. Age differences in short-term retention of rapidly changing information. Journal of experimental psychology 55, 4 (Apr. 1958), 352--358.Google Scholar
- Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. 2008. Measuring the task-evoked pupillary response with a remote eye tracker. In Proceedings of the 2008 symposium on Eye tracking research & applications (ETRA '08). ACM, NY, NY, USA, 69--72. DOI: http://dx.doi.org/10.1145/1344471.1344489 Google ScholarDigital Library
- Jeff Klingner, Barbara Tversky, and Pat Hanrahan. 2011. Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology 48, 3 (Mar. 2011), 323--332. DOI: http://dx.doi.org/10.1111/j.1469--8986.2010.01069.xGoogle ScholarCross Ref
- Andrew L. Kun, Susanne Boll, and Albrecht Schmidt. 2016. Shirting gears: User interfaces in the age of autonomous driving. IEEE Pervasive Computing 15, 1 (January-March 2016), 32--37. DOI: http://dx.doi.org/10.1109/MPRV.2016.14 Google ScholarDigital Library
- Andrew L. Kun, and Zeljko Medenica. 2012. Video call or not, that is the question. In CHI '12 Extended Abstracts on Human Factors in Computing Systems. ACM, NY, NY, USA, 1631--1636. DOI: http://dx.doi.org/10.1145/2212776.2223684 Google ScholarDigital Library
- Andrew L. Kun, Oskar Palinko, Zeljko Medenica, and Peter A. Heeman. 2013. On the Feasibility of Using Pupil Diameter to Estimate Cognitive Load Changes for In-Vehicle Spoken Dialogues. In INTERSPEECH, (2013), 3766--3770.Google Scholar
- Andrew L. Kun, Oskar Palinko, and Razumeniø, I. Exploring the effects of size and luminance of visual targets on the pupillary light reflex. In Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '12). ACM, NY, NY, USA, 183--186. DOI: http://doi.acm.org/10.1145/2390256.2390287 Google ScholarDigital Library
- Sandra P. Marshall, 2002. The Index of Cognitive Activity: measuring cognitive workload. In Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants. IEEE, 7--5 - 7--9. DOI: http://dx.doi.org/10.1109/HFPP.2002.1042860Google ScholarCross Ref
- Bruce Mehler, Bryan Reimer, Joseph F. Coughlin, and Jeffery A. Dusek. 2009. Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transportation Research Record: Journal of the Transportation Research Board 2138, 1 (2009), 6--12. DOI: http://dx.doi.org/10.3141/2138-02Google ScholarCross Ref
- Bruce Mehler, Bryan Reimer, and Joseph F. Coughlin. 2012. Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task an on-road study across three age groups. Human Factors 54, 3 (2010), 396--412. DOI: http://dx.doi.org/10.1177/0018720812442086Google ScholarCross Ref
- Bruce Mehler, Bryan Reimer and Jeffery A. Dusek. 2011. MIT AgeLab Delayed Digit Recall Task (n-back) Working Paper 2011--3B. Massachusetts Institute of Technology, Cambridge, MA. http://agelab.mit.edu/system/files/Mehler_et_al_n-back-white-paper_2011_B.pdfGoogle Scholar
- W. Thomas Miller, and Andrew L. Kun. 2013. Using speech, GUIs and buttons in police vehicles: field data on user preferences for the Project54 system. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '13). ACM, NY, NY, USA, 108--113. DOI: http://dx.doi.org/10.1145/2516540.2516564 Google ScholarDigital Library
- William F. Moroney, David W. Biers, Thomas Eggemeier, and Jennifer A. Mitchell. 1992. A comparison of two scoring procedures with the NASA Task Load Index in a simulated flight task. In Proceedings of the IEEE 1992 National Aerospace and Electronics Conference, NAECON 1992. IEEE, 734--740. DOI: http://dx.doi.org/10.1109/NAECON.1992.220513Google ScholarCross Ref
- Nesrine Mezhoudi, Iyad Khaddam, and Jean Vanderdonckt. 2015. Toward Usable Intelligent User Interface. In: Proceedings of the 17th International Conference on Human-Computer Interaction (HCII '15), Part II, LNCS 9170, Springer International Publishing, 459--471. DOI: http://dx.doi.org/10.1007/978--3--319--20916--6_43Google ScholarCross Ref
- Michael Myrtek, Doris Weber, Georg Brügner, and Wolfgang Müller. 1996. Occupational stress and strain of female students: results of physiological, behavioral, and psychological monitoring. Biological Psychology 42, 3 (Feb. 1996), 379--391. DOI: http://dx.doi.org/10.1016/0301-0511(95)05168--6Google ScholarCross Ref
- Luis Nunes and Miguel A. Rescarte 2002. Cognitive demands of hands-free-phone conversation while driving. Transportation Research Part F 5, 2 (Jun. 2002), 133--144. DOI: http://dx.doi.org/10.1016/S1369--8478(02)00012--8Google ScholarCross Ref
- Julie Onton, Arnaud Delorme, and Scott Makeig. 2005. Frontal midline EEG dynamics during working memory. NeuroImage 27, 2 (Aug. 2005), 341--356. DOI: http://dx.doi.org/10.1016/j.neuroimage.2005.04.014Google ScholarCross Ref
- Oskar Palinko and Andrew Kun. 2011. Exploring the influence of light and cognitive load on pupil diameter in driving simulator studies. In Proceedings of the Sixth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design (Driving Assessment 2011), University of Iowa, Iowa City, IA, 329--336. http://drivingassessment.uiowa.edu/sites/default/files/DA2011/Papers/048_PalinkoKun.pdfGoogle ScholarCross Ref
- Oskar Palinko and Andrew Kun. 2012. Exploring the effects of visual cognitive load and illumination on pupil diameter in driving simulators. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12), ACM, NY, NY, 413--416. DOI: http://dx.doi.org/10.1145/2168556.2168650 Google ScholarDigital Library
- Oskar Palinko, Andrew L. Kun, Alexander Shyrokov, and Peter A. Heeman. 2010. Estimating cognitive load using remote eye tracking in a driving simulator. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '10), ACM, NY, NY, 141--144. DOI: http://dx.doi.org/10.1145/1743666.1743701 Google ScholarDigital Library
- Bastian Pfleging and Albrecht Schmidt. 2015. (Non-) Driving-Related Activities in the Car: Defining Driver Activitgies for Manual and Automated Driving. In Workshop on Experiencing Autonomous Vehicles: Crossing the Boundaries between a Drive and a Ride at CHI '15. http://www.hcilab.org/wp-content/uploads/pfleging-2015-drivingrelatedactivities.pdf Google ScholarDigital Library
- Bastian Pfleging, Stefan Schneegass, and Albrecht Schmidt. 2013. Exploring user expectations for context and road video sharing while calling and driving. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '13). ACM, NY, NY, USA, 132--139. DOI: http://dx.doi.org/10.1145/2516540.2516547 Google ScholarDigital Library
- Bryan Reimer, Bruce Mehler, Joseph F. Coughlin, Kathryn M. Godfrey, and Chauanzhong Tan. 2009. An on-road assessment of the impact of cognitive workload on physiological arousal in young adult drivers. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Application (AutomotiveUI '09), ACM, NY, NY, 115--118. DOI: http://dx.doi.org/10.1145/1620509.1620531 Google ScholarDigital Library
- A. H. Roscoe. 1992. Assessing pilot workload. Why measure heart rate, HRV and respiration? Biological Psychology 34, 2 (Nov. 1992), 259--287. DOI: http://dx.doi.org/10.1016/0301-0511(92)90018-PGoogle ScholarCross Ref
- Dennis W. Rowe, John Sibert, and Don Irwin. 1998. Heart rate variability: Indicator of user state as an aid to human-computer interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98), ACM, NY, NY, 480--487. DOI: http://dx.doi.org/10.1145/274644.274709 Google ScholarDigital Library
- Kilseop Ryu and Rohae Myung. 2005. Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics 35, 11 (Nov. 2005), 991--1009. DOI: http://dx.doi.org/10.1016/j.ergon.2005.04.005Google ScholarCross Ref
- Stefan Schneegass, Bastian Pfleging, Nora Broy, Frederik Heinrich, and Albrecht Schmidt. A data set of real world driving to assess driver workload. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI 2013). ACM, NY, NY, 150--157. DOI: http://dx.doi.org/10.1145/2516540.2516561 Google ScholarDigital Library
- Erin T. Solovey, Paul Schermerhorn, Matthias Scheutz, Angelo Sassaroli, Sergio Fantini, and Robert J.K. Jacob. 2012. Brainput: Enhancing Interactive Systems with Streaming fNIRS Brain Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 2193--2202. DOI: http://dx.doi.org/10.1145/2207676.2208372 Google ScholarDigital Library
- Yi-Fang Tsai, Erik Viirre, Christopher Strychacz, Bradley Chase, and Tzyy-Ping Jung. 2007. Task performance and eye activity: Predicting behavior relating to cognitive workload. Aviation, Space, and Environmental Medicine 78, 5 (May 2007), B176-B185.Google Scholar
- Karl F. Van Orden, Wendy Limbert, Scott Makeig, and Tzyy-Ping Jung. 2001. Eye activity correlates of workload during visuospatial memory task. Human Factors: The Journal of the Human Factors and Ergonomics Society 43, 1 (Spring 2001), 111--121. DOI: http://dx.doi.org/10.1518/001872001775992570Google ScholarCross Ref
- Ying Wang, Bryan Reimer, Bruce Mehler, Jun Zhang, Alea Mehler, and Joseph F. Coughlin. 2010. The Impact of Repeated Cognitive Tasks on Driving Performance and Visual Attention. In Proceedings of the 3rd International Conference on Applied Human Factors and Ergonomics (AHFE '10), Miami, FL, USA.Google Scholar
- Yan Yang, Bryan Reimer, Bruce Mehler,and Jonathan Dobres. 2013. A field study assessing driving performance, visual attention, heart rate and subjective ratings in response to two types of cognitive workload. In Proceedings of the 7th International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design (Driving Assessment 2013). University of Iowa, Iowa City, IA, 397--403.Google ScholarCross Ref
- Yabo Yang, Keith Thompson, and Stephen A. Burns. 2002. Pupil location under mesopic, photopic, and pharmacologically dilated conditions. Investigative Ophthalmology & Visual Science 43, 7 (Jul. 2002), 2508--2512.Google Scholar
- Zhiwei Zhu, Kikuo Fujimura, and Qiang Ji. 2002. Real-time eye detection and tracking under various light conditions. In Proceedings of the 2002 Symposium on Eye Tracking Research & Applications (ETRA 2002). ACM, NY, NY, 139--144. DOI: http://dx.doi.org/10.1145/507072.507100 Google ScholarDigital Library
Index Terms
- A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions
Recommendations
Measuring the task-evoked pupillary response with a remote eye tracker
ETRA '08: Proceedings of the 2008 symposium on Eye tracking research & applicationsThe pupil-measuring capability of video eye trackers can detect the task-evoked pupillary response: subtle changes in pupil size which indicate cognitive load. We performed several experiments to measure cognitive load using a remote video eye tracker, ...
Pupil center as a function of pupil diameter
ETRA '16: Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & ApplicationsWe investigate the gaze estimation error induced by pupil size changes using simulated data. We investigate the influence of pupil diameter changes on estimated gaze point error obtained by two gaze estimation models. Simulation data show that at wider ...
Pupillometry to assess Air Traffic Controller workload through the Mental Workload Model
ATACCS '15: Proceedings of the 5th International Conference on Application and Theory of Automation in Command and Control SystemsThe introduction of automation in Air Traffic Management (ATM) implies a change in the Air Traffic Controller (ATCo) mental workload. In order to assess this impact, CRIDA is developing a mental workload model based on Wickens's and McCarley's Theories [...
Comments