Yomna Abdelrahman
Eduardo Velloso
Tilman Dingler
Albrecht Schmidt
Frank Vetere
Skip Abstract Section
Abstract
Current digital systems are largely blind to users’ cognitive states. Systems that adapt to users’ states show great potential for augmenting cognition and for creating novel user experiences. However, most approaches for sensing cognitive states, and cognitive load specifically, involve obtrusive technologies, such as physiological sensors attached to users’ bodies. This paper present an unobtrusive indicator of the users’ cognitive load based on thermal imaging that is applicable in real-world. We use a commercial thermal camera to monitor a person’s forehead and nose temperature changes to estimate their cognitive load. To assess the effect of different levels of cognitive load on facial temperature we conducted a user study with 12 participants. The study showed that different levels of the Stroop test and the complexity of reading texts affect facial temperature patterns, thereby giving a measure of cognitive load. To validate the feasibility for real-time assessments of cognitive load, we conducted a second study with 24 participants, we analyzed the temporal latency of temperature changes. Our system detected temperature changes with an average latency of 0.7 seconds after users were exposed to a stimulus, outperforming latency in related work that used other thermal imaging techniques. We provide empirical evidence showing how to unobtrusively detect changes in cognitive load in real-time. Our exploration of exposing users to different content types gives rise to thermal-based activity tracking, which facilitates new applications in the field of cognition-aware computing.
- Yomna Abdelrahman. 2013. Thermal imaging for interactive surfaces. Master’s thesis.Google Scholar
- Yomna Abdelrahman, Mohamed Khamis, Stefan Schneegass, and Florian Alt. 2017. Stay cool! understanding thermal attacks on mobile-based user authentication. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3751--3763. Google ScholarDigital Library
- Yomna Abdelrahman, Pascal Knierim, and Albrecht Schmidt. 2017. Snake View: Exploring Thermal Imaging as a Vision Extender in Mountains. In (UbiMount) ubiquitous computing in the mountains.Google ScholarDigital Library
- Yomna Abdelrahman, Pascal Knierim, Pawel Wozniak, Niels Henze, and Albrecht Schmidt. 2017. See through the Fire: Evaluating the Augmentation of Visual Perception of Firefighters Using Depth and Thermal Cameras. In (WAHM) Workshop on Ubiquitous Technologies for Augmenting the Human Mind.Google ScholarDigital Library
- Yomna Abdelrahman, Alireza Sahami Shirazi, Niels Henze, and Albrecht Schmidt. 2015. Investigation of Material Properties for Thermal Imaging-Based Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 15--18. Google ScholarDigital Library
- Jackson Beatty. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological bulletin 91, 2 (1982), 276.Google Scholar
- Kirsch C. (Producer) Moxham J. B. (Producer) Alusi G. (Producer) Berkovitz, B. (Producer) and T. (Producer) Cheesman. 2013. 3D Head and Neck Anatomy with Special Senses and Basic Neuroanatomy. {DVD-ROM}. United Kingdom: Primal Pictures.Google Scholar
- Katie Crowley, Aidan Sliney, Ian Pitt, and Dave Murphy. 2010. Evaluating a Brain-Computer Interface to Categorise Human Emotional Response.. In ICALT. 276--278. Google ScholarDigital Library
- Mai ElKomy, Yomna Abdelrahman, Markus Funk, Tilman Dingler, Albrecht Schmidt, and Slim Abdennadher. 2017. ABBAS: An Adaptive Bio-sensors Based Assistive System. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2543--2550. Google ScholarDigital Library
- Douglas C Engelbart. 1962. Augmenting human intellect: a conceptual framework (1962). Technical Report for the Air Force Office of Scientific Research (1962).Google Scholar
- Veronika Engert, Arcangelo Merla, Joshua A Grant, Daniela Cardone, Anita Tusche, and Tania Singer. 2014. Exploring the use of thermal infrared imaging in human stress research. PloS one 9, 3 (2014), e90782.Google ScholarCross Ref
- Stephen H. Fairclough. 2009. Fundamentals of Physiological Computing. Interact. Comput. 21, 1- 2 (Jan. 2009), 133--145. Google ScholarDigital Library
- Ismael Fernández-Cuevas, Joao Carlos Bouzas Marins, Javier Arnáiz Lastras, Pedro María Gómez Carmona, Sergio Piñonosa Cano, Miguel Ángel García-Concepción, and Manuel Sillero-Quintana. 2015. Classification of factors influencing the use of infrared thermography in humans: A review. Infrared Physics 8 Technology 71 (2015), 28--55.Google Scholar
- Markus Funk, Tilman Dingler, Jennifer Cooper, and Albrecht Schmidt. 2015. Stop helping me-I’m bored!: why assembly assistance needs to be adaptive. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers. ACM, 1269--1273. Google ScholarDigital Library
- Rikke Gade and Thomas B Moeslund. 2014. Thermal cameras and applications: a survey. Machine vision and applications 25, 1 (2014), 245--262. Google ScholarDigital Library
- Marc Germain, Michel Jobin, and Michel Cabanac. 1987. The effect of face fanning during recovery from exercise hyperthermia. Canadian journal of physiology and pharmacology 65, 1 (1987), 87--91.Google Scholar
- Nitesh Goyal and Susan R. Fussell. 2017. Intelligent Interruption Management using Electro Dermal Activity based Physiological Sensor on Collaborative Sensemaking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 16. Google ScholarDigital Library
- M Haak, S Bos, S Panic, and LJM Rothkrantz. 2009. Detecting stress using eye blinks and brain activity from EEG signals. Proceeding of the 1st driver car interaction and interface (DCII 2008) (2009), 35--60.Google Scholar
- Jennifer A Healey and Rosalind W Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. Intelligent Transportation Systems, IEEE Transactions on 6, 2 (2005), 156--166. Google ScholarDigital Library
- Javier Hernandez, Pablo Paredes, Asta Roseway, and Mary Czerwinski. 2014. Under pressure: sensing stress of computer users. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 51--60. Google ScholarDigital Library
- Sarah M Hosni, Mahmoud E Gadallah, Sayed F Bahgat, and Mohamed S AbdelWahab. 2007. Classification of EEG signals using different feature extraction techniques for mental-task BCI. In Computer Engineering 8 Systems, 2007. ICCES’07. International Conference on. IEEE, 220--226.Google Scholar
- Curtis S Ikehara and Martha E Crosby. 2005. Assessing cognitive load with physiological sensors. In Proceedings of the 38th annual hawaii international conference on system sciences. IEEE, 295a--295a. Google ScholarDigital Library
- Stephanos Ioannou, Sjoerd Ebisch, Tiziana Aureli, Daniela Bafunno, Helene Alexi Ioannides, Daniela Cardone, Barbara Manini, Gian Luca Romani, Vittorio Gallese, and Arcangelo Merla. 2013. The autonomic signature of guilt in children: a thermal infrared imaging study. PloS one 8, 11 (2013), e79440.Google ScholarCross Ref
- Stephanos Ioannou, Vittorio Gallese, and Arcangelo Merla. 2014. Thermal infrared imaging in psychophysiology: potentialities and limits. Psychophysiology 51, 10 (2014), 951--963.Google ScholarCross Ref
- Giulio Jacucci, Anna Spagnolli, Jonathan Freeman, and Luciano Gamberini. 2014. Symbiotic Interaction: A Critical Definition and Comparison to other Human-Computer Paradigms. Springer International Publishing, Cham, 3--20.Google Scholar
- SD Jenkins and RDH Brown. 2014. A correlational analysis of human cognitive activity using Infrared Thermography of the supraorbital region, frontal EEG and self-report of core affective state. QIRT.Google Scholar
- Jihun Kang and Kari Babski-Reeves. 2008. Detecting Mental Workload Fluctuation during Learning of a Novel Task Using Thermography. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 52, 19 (2008), 1527--1531.Google ScholarCross Ref
- J Kang, JA McGinley, G McFadyen, and K Babski-Reeves. 2006. Determining learning level and effective training times using thermography. In Proceedings of Army Science Conference, Orlando, Florida, USA.Google Scholar
- Hisanori Kataoka, Hiroshi Kano, Hiroaki Yoshida, Atsuo Saijo, Masashi Yasuda, and Masato Osumi. 1998. Development of a skin temperature measuring system for non-contact stress evaluation. In Engineering in Medicine and Biology Society, 1998. Proceedings of the 20th Annual International Conference of the IEEE, Vol. 2. IEEE, 940--943.Google ScholarCross Ref
- Koji Kuraoka and Katsuki Nakamura. 2011. The use of nasal skin temperature measurements in studying emotion in macaque monkeys. Physiology 8 behavior 102, 3 (2011), 347--355.Google Scholar
- Hindra Kurniawan, Alexey V Maslov, and Mykola Pechenizkiy. 2013. Stress detection from speech and galvanic skin response signals. In Computer-Based Medical Systems (CBMS), 2013 IEEE 26th International Symposium on. IEEE, 209--214.Google ScholarCross Ref
- Elise Labbé, Nicholas Schmidt, Jonathan Babin, and Martha Pharr. 2007. Coping with stress: the effectiveness of different types of music. Applied psychophysiology and biofeedback 32, 3-4 (2007), 163--168.Google Scholar
- Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: Thermal Imaging for Surface User Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 2565--2574. Google ScholarDigital Library
- Robert W Levenson. 1988. Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity. (1988).Google Scholar
- Joseph CR Licklider. 1960. Man-computer symbiosis. IRE transactions on human factors in electronics 1 (1960), 4--11.Google Scholar
- Barbara Manini, Daniela Cardone, Sjoerd JH Ebisch, Daniela Bafunno, Tiziana Aureli, and Arcangelo Merla. 2013. Mom feels what her child feels: thermal signatures of vicarious autonomic response while watching children in a stressful situation. Front. Hum. Neurosci 7 (2013).Google Scholar
- TV McCaffrey, RD McCook, and RD Wurster. 1975. Effect of head skin temperature on tympanic and oral temperature in man. Journal of Applied Physiology 39, 1 (1975), 114--118.Google ScholarCross Ref
- Arcangelo Merla and Gian Luca Romani. 2007. Thermal signatures of emotional arousal: a functional infrared imaging study. In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 247--249.Google ScholarCross Ref
- Takakazu Oka, Kae Oka, and Tetsuro Hori. 2001. Mechanisms and mediators of psychological stress-induced rise in core temperature. Psychosomatic medicine 63, 3 (2001), 476--486.Google Scholar
- Calvin KL Or and Vincent G Duffy. 2007. Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occupational Ergonomics 7, 2 (2007), 83--94.Google ScholarCross Ref
- Avinash Parnandi, Beena Ahmed, Eva Shipp, and Ricardo Gutierrez-Osuna. 2013. Chill-Out: Relaxation training through respiratory biofeedback in a mobile casual game. In Mobile Computing, Applications, and Services. Springer, 252--260.Google Scholar
- Avinash Parnandi, Youngpyo Son, and Ricardo Gutierrez-Osuna. 2013. A control-theoretic approach to adaptive physiological games. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, 7--12. Google ScholarDigital Library
- Ioannis Pavlidis and James Levine. 2002. Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine 21, 6 (2002), 56--64.Google ScholarCross Ref
- Ioannis Pavlidis, Panagiotis Tsiamyrtzis, Dvijesh Shastri, Avinash Wesley, Yan Zhou, Peggy Lindner, Pradeep Buddharaju, Rohan Joseph, Anitha Mandapati, Brian Dunkin, et al. 2012. Fast by nature-how stress patterns define human experience and performance in dexterous tasks. Scientific Reports 2 (2012).Google Scholar
- Michael Kai Petersen, Carsten Stahlhut, Arkadiusz Stopczynski, Jakob Eg Larsen, and Lars Kai Hansen. 2011. Smartphones get emotional: mind reading images and reconstructing the neural sources. In International Conference on Affective Computing and Intelligent Interaction. Springer, 578--587. Google ScholarDigital Library
- Bastian Pfleging, Drea K. Fekety, Albrecht Schmidt, and Andrew L. Kun. 2016. A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA. Google ScholarDigital Library
- Rosalind W. Picard. 1997. Affective Computing. MIT Press, Cambridge, MA, USA. Google ScholarDigital Library
- Colin Puri, Leslie Olson, Ioannis Pavlidis, James Levine, and Justin Starren. 2005. StressCam: non-contact measurement of users’ emotional states through thermal imaging. In CHI’05 extended abstracts on Human factors in computing systems. ACM, 1725--1728. Google ScholarDigital Library
- B. A. Rajoub and R. Zwiggelaar. 2014. Thermal Facial Analysis for Deception Detection. IEEE Transactions on Information Forensics and Security 9, 6 (June 2014), 1015--1023. Google ScholarDigital Library
- Alireza Sahami Shirazi, Yomna Abdelrahman, Niels Henze, Stefan Schneegass, Mohammadreza Khalilbeigi, and Albrecht Schmidt. 2014. Exploiting Thermal Reflection for Interactive Systems. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 3483--3492. Google ScholarDigital Library
- Albrecht Schmidt. 2017. Augmenting Human Intellect and Amplifying Perception and Cognition. IEEE Pervasive Computing 16, 1 (2017), 6--10. Google ScholarDigital Library
- Stefan Schneegass, Bastian Pfleging, Nora Broy, Frederik Heinrich, and Albrecht Schmidt. 2013. A data set of real world driving to assess driver workload. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 150--157. Google ScholarDigital Library
- Nandita Sharma and Tom Gedeon. 2012. Objective measures, sensors and computational techniques for stress recognition and classification: A survey. Computer methods and programs in biomedicine 108, 3 (2012), 1287--1301. Google ScholarDigital Library
- Dvijesh Shastri, Arcangelo Merla, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2009. Imaging facial signs of neurophysiological responses. IEEE Transactions on Biomedical Engineering 56, 2 (2009), 477--484.Google ScholarCross Ref
- Dvijesh Shastri, Manos Papadakis, Panagiotis Tsiamyrtzis, Barbara Bass, and Ioannis Pavlidis. 2012. Perinasal imaging of physiological stress and its affective potential. IEEE Transactions on Affective Computing 3, 3 (2012), 366--378. Google ScholarDigital Library
- Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and Fang Chen. 2007. Galvanic skin response (GSR) as an index of cognitive load. In CHI’07 extended abstracts on Human factors in computing systems. ACM, 2651--2656. Google ScholarDigital Library
- Alireza Sahami Shirazi, Markus Funk, Florian Pfleiderer, Hendrik Glück, and Albrecht Schmidt. 2012. MediaBrain: Annotating Videos based on Brain-Computer Interaction.. In Mensch 8 Computer. 263--272.Google Scholar
- Alireza Sahami Shirazi, Mariam Hassib, Niels Henze, Albrecht Schmidt, and Kai Kunze. 2014. What’s on your mind?: mental task awareness using single electrode brain computer interfaces. In Proceedings of the 5th Augmented Human International Conference. ACM, 45. Google ScholarDigital Library
- Rajita Sinha, William R Lovallo, and Oscar A Parsons. 1992. Cardiovascular differentiation of emotions. Psychosomatic Medicine 54, 4 (1992), 422--435.Google ScholarCross Ref
- John Stemberger, Robert S Allison, and Thomas Schnell. 2010. Thermal imaging as a way to classify cognitive workload. In Computer and Robot Vision (CRV), 2010 Canadian Conference on. IEEE, 231--238. Google ScholarDigital Library
- J Ridley Stroop. 1935. Studies of interference in serial verbal reactions. Journal of Experimental Psychology: General 121, 1 (1935), 15.Google ScholarCross Ref
- John Sweller, Paul Ayres, and Slava Kalyuga. 2011. Measuring Cognitive Load. Springer New York, New York, NY, 71--85.Google Scholar
- Boris M Velichkovsky and John Paulin Hansen. 1996. New technological windows into mind: There is more in eyes and brains for human-computer interaction. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 496--503. Google ScholarDigital Library
- Paul Viola and Michael J Jones. 2004. Robust real-time face detection. International journal of computer vision 57, 2 (2004), 137--154. Google ScholarDigital Library
- Zixuan Wang, Jinyun Yan, and Hamid Aghajan. 2012. A framework of personal assistant for computer users by analyzing video stream. In Proceedings of the 4th Workshop on Eye Gaze in Intelligent Human Machine Interaction. ACM, 14. Google ScholarDigital Library
- Mark Weiser. 1991. The computer for the 21st century. Scientific american 265, 3 (1991), 94--104.Google Scholar
- Robert B Zajonc, Sheila T Murphy, and Marita Inglehart. 1989. Feeling and facial efference: implications of the vascular theory of emotion. Psychological review 96, 3 (1989), 395.Google Scholar
- Zhen Zhu, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2007. Forehead thermal signature extraction in lie detection. In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 243--246.Google ScholarCross Ref
Index Terms
- Cognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load
Recommendations
Using stroop task to assess cognitive load
ECCE '10: Proceedings of the 28th Annual European Conference on Cognitive ErgonomicsMotivation -- Assessment of cognitive load on user tasks is useful for characterizing user interfaces and tasks with respect to their demands on the user's mental effort.
Research approach -- We conducted a controlled experiment with 48 subjects. The ...
The Cognitive Load Affects the Interaction Pattern of Emotion and Working Memory
Emotion might selectively affect spatial and verbal cognitive activities and the selective interaction patter could be modulated by cognitive load. To test the hypotheses, the authors used event-related potentials ERPs technique to investigate the ...
Analysis of Influencing Factors of Heat Transfer Performance of Heat Pipe Heat Exchanger
ICEET '09: Proceedings of the 2009 International Conference on Energy and Environment Technology - Volume 01Heat transfer performance of heat pipe heat exchanger affected by many factors, such as the opening temperature of the heat pipe , the rate of fluid-filled, the physical nature of liquid refrigerant 、the work temperature of tube , angle, tube spacing, ...
Comments