Michael Xuelin Huang
Grace Ngai
Andreas Bulling
ABSTRACT
Internal thought refers to the process of directing attention away from a primary visual task to internal cognitive processing. It is pervasive and closely related to primary task performance. As such, automatic detection of internal thought has significant potential for user modeling in human-computer interaction and multimedia applications. Despite the close link between the eyes and the human mind, only few studies have investigated vergence behavior during internal thought and none has studied moment-to-moment detection of internal thought from gaze. While prior studies relied on long-term data analysis and required a large number of gaze characteristics, we describe a novel method that is user-independent, computationally light-weight and only requires eye vergence information readily available from binocular eye trackers. We further propose a novel paradigm to obtain ground truth internal thought annotations by exploiting human blur perception. We evaluated our method during natural viewing of lecture videos and achieved a 12.1% improvement over the state of the art. These results demonstrate the effectiveness and robustness of vergence-based detection of internal thought and, as such, open new research directions for attention-aware interfaces.
- Benedek, M. et al. 2017. Eye Behavior Associated with Internally versus Externally Directed Cognition. Frontiers in Psychology. 8, June (Jun. 2017), 1--9. DOI:https://doi.org/10.3389/fpsyg.2017.01092.Google Scholar
- Bixler, R. and D'Mello, S. 2016. Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Modeling and User-Adapted Interaction. 26, 1 (Mar. 2016), 33--68. DOI:https://doi.org/10.1007/s11257-015--9167--1.Google ScholarDigital Library
- Bulling, A. and Zander, T.O. 2014. Cognition-Aware Computing. IEEE Pervasive Computing. 13, 3 (Jul. 2014), 80--83. DOI:https://doi.org/10.1109/MPRV.2014.42.Google ScholarCross Ref
- Casiez, G. et al. 2012. 1 filter: a simple speed-based low-pass filter for noisy input in interactive systems. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12 (New York, New York, USA, 2012), 2527.Google Scholar
- Christoff, K. et al. 2016. Mind-wandering as spontaneous thought: a dynamic framework. Nature Reviews Neuroscience. 17, 11 (Nov. 2016), 718--731. DOI:https://doi.org/10.1038/nrn.2016.113.Google ScholarCross Ref
- Conati, C. et al. 2013. Eye-Tracking for Student Modelling in Intelligent Tutoring Systems. Design Recommendations for Intelligent Tutoring Systems. 227--236.Google Scholar
- D'Mello, S.K. 2016. Giving Eyesight to the Blind: Towards Attention-Aware AIED. International Journal of Artificial Intelligence in Education. 26, 2 (Jun. 2016), 645--659. DOI:https://doi.org/10.1007/s40593-016-0104-1.Google Scholar
- Dingler, T. 2016. Cognition-Aware systems as mobile personal assistants. UbiComp 2016 Adjunct - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. (2016), 1035--1040. DOI:https://doi.org/10.1145/2968219.2968565.Google ScholarDigital Library
- Dixon, M.L. et al. 2014. A framework for understanding the relationship between externally and internally directed cognition. Neuropsychologia. 62, (Sep. 2014), 321--330. DOI:https://doi.org/10.1016/j.neuropsychologia.2014.05.024.Google Scholar
- Engbert, R. and Kliegl, R. 2003. Microsaccades uncover the orientation of covert attention. Vision Research. 43, 9 (2003), 1035--1045. DOI:https://doi.org/10.1016/S0042-6989(03)00084-1.Google ScholarCross Ref
- Faber, M. et al. 2017. An automated behavioral measure of mind wandering during computerized reading. Behavior Research Methods. (2017). DOI:https://doi.org/10.3758/s13428-017-0857-y.Google Scholar
- Feit, A.M. et al. 2017. Toward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17 (New York, New York, USA, 2017), 1118--1130.Google ScholarDigital Library
- Franklin, M.S. et al. 2011. Catching the mind in flight: Using behavioral indices to detect mindless reading in real time. Psychonomic Bulletin and Review. 18, 5 (2011), 992--997. DOI:https://doi.org/10.3758/s13423-011-0109-6.Google ScholarCross Ref
- Franklin, M.S. et al. 2013. Window to the wandering mind: Pupillometry of spontaneous thought while reading. The Quarterly Journal of Experimental Psychology. 66, 12 (Dec. 2013), 2289--2294. DOI:https://doi.org/10.1080/17470218.2013.858170.Google ScholarCross Ref
- Hansen, D.W. and Ji, Q. 2010. In the eye of the beholder: a survey of models for eyes and gaze. IEEE transactions on pattern analysis and machine intelligence. 32, 3 (2010), 478--500. DOI:https://doi.org/10.1109/TPAMI.2009.30.Google Scholar
- http://developer.tobii.com/:2018..Google Scholar
- Hutt, S. et al. 2017. Gaze-based Detection of Mind Wandering during Lecture Viewing. 10th International Conference on Educational Data Mining (2017).Google Scholar
- Hutt, S. et al. 2016. The Eyes Have It: Gaze-based Detection of Mind Wandering during Learning with an Intelligent Tutoring System. Proceedings of the 9th International Conference on Educational Data Mining, International Educational Data Mining Society (2016), 86--93.Google Scholar
- Killingsworth, M.A. and Gilbert, D.T. 2010. A Wandering Mind Is an Unhappy Mind. Science. 330, 6006 (Nov. 2010), 932--932. DOI:https://doi.org/10.1126/science.1192439.Google ScholarCross Ref
- Kudo, S. et al. 2013. Input method using divergence eye movement. CHI '13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA '13 (New York, New York, USA, 2013), 1335.Google ScholarDigital Library
- Li, J. et al. 2016. Your Eye Tells How Well You Comprehend. IEEE 40th Annual Computer Software and Applications Conference (COMPSAC) (2016), 503--508.Google Scholar
- Mijovic, P. et al. 2017. Towards continuous and real-time attention monitoring at work: reaction time versus brain response. Ergonomics. 60, 2 (Feb. 2017), 241--254. DOI:https://doi.org/10.1080/00140139.2016.1142121.Google ScholarCross Ref
- Mills, C. et al. 2016. Automatic Gaze-Based Detection of Mind Wandering during Narrative Film Comprehension. Proceedings of the 9th International Conference on Educational Data Mining (EDM 2016) (2016), 30--37.Google Scholar
- Niehorster, D.C. et al. 2017. What to expect from your remote eye-tracker when participants are unrestrained. Behavior Research Methods. (Feb. 2017). DOI:https://doi.org/10.3758/s13428-017-0863-0.Google Scholar
- Olney, A.M. et al. 2015. Attention in Educational Contexts: The Role of the Learning Task in Guiding Attention. The Handbook of Attention. MIT Press. 623--642.Google Scholar
- Pham, P. and Wang, J. 2015. AttentiveLearner: Improving Mobile MOOC Learning via Implicit Heart Rate Tracking. International Conference on Artificial Intelligence in Education (2015), 367--376.Google ScholarCross Ref
- Reichle, E.D. et al. 2010. Eye Movements During Mindless Reading. Psychological Science. 21, (2010), 1300--1310. DOI:https://doi.org/10.1177/0956797610378686.Google Scholar
- Salvucci, D.D. and Goldberg, J.H. 2000. Identifying fixations and saccades in eye-tracking protocols. Proceedings of the symposium on Eye tracking research & applications - ETRA '00 (New York, New York, USA, New York, USA, 2000), 71--78.Google ScholarDigital Library
- Sattar, H. et al. 2017. Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) (Oct. 2017), 2740--2748.Google Scholar
- Sattar, H. et al. 2015. Prediction of search targets from fixations in open-world settings. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Jun. 2015), 981--990.Google ScholarCross Ref
- Schooler, J.W. et al. 2011. Meta-awareness, perceptual decoupling and the wandering mind. Trends in Cognitive Sciences. (Jun. 2011). DOI:https://doi.org/10.1016/j.tics.2011.05.006.Google Scholar
- Smallwood, J. et al. 2004. Subjective experience and the attentional lapse: Task engagement and disengagement during sustained attention. Consciousness and Cognition. 13, 4 (Dec. 2004), 657--690. DOI:https://doi.org/10.1016/j.concog.2004.06.003.Google ScholarCross Ref
- Smallwood, J. et al. 2004. The consequences of encoding information on the maintenance of internally generated images and thoughts: The role of meaning complexes. Consciousness and Cognition. 13, 4 (Dec. 2004), 789--820. DOI:https://doi.org/10.1016/j.concog.2004.07.004.Google ScholarCross Ref
- Smallwood, J. and Schooler, J.W. 2006. The restless mind. Psychological Bulletin. 132, 6 (2006), 946--958. DOI:https://doi.org/10.1037/0033--2909.132.6.946.Google ScholarCross Ref
- Smallwood, J. and Schooler, J.W. 2015. The Science of Mind Wandering: Empirically Navigating the Stream of Consciousness. Annual Review of Psychology. 66, 1 (2015), 487--518. DOI:https://doi.org/10.1146/annurev-psych-010814-015331.Google ScholarCross Ref
- Solé Puig, M. et al. 2013. A Role of Eye Vergence in Covert Attention. PLoS ONE. 8, 1 (Jan. 2013), e52955. DOI:https://doi.org/10.1371/journal.pone.0052955.Google ScholarCross Ref
- Solé Puig, M. et al. 2015. Attention-Related Eye Vergence Measured in Children with Attention Deficit Hyperactivity Disorder. PLOS ONE. 10, 12 (Dec. 2015), e0145281. DOI:https://doi.org/10.1371/journal.pone.0145281.Google ScholarCross Ref
- Steil, J. and Bulling, A. 2015. Discovery of everyday human activities from long-term visual behaviour using topic models. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp '15 (New York, New York, USA, 2015), 75--85.Google ScholarDigital Library
- Toates, F.M. 1974. Vergence eye movements. Documenta Ophthalmologica. 37, 1 (1974), 153--214. DOI:https://doi.org/10.1007/BF00149678.Google ScholarCross Ref
- Unsworth, N. and Robison, M.K. 2016. Pupillary correlates of lapses of sustained attention. Cognitive, Affective, & Behavioral Neuroscience. April (2016), 601--615. DOI:https://doi.org/10.3758/s13415-016-0417-4.Google Scholar
- Walcher, S. et al. 2017. Looking for ideas: Eye behavior during goal-directed internally focused cognition. Consciousness and Cognition. 53, (Aug. 2017), 165--175. DOI:https://doi.org/10.1016/j.concog.2017.06.009.Google Scholar
- Watson, A.B. and Ahumada, A.J. 2011. Blur clarified: A review and synthesis of blur discrimination. Journal of Vision. 11, 5 (Sep. 2011), 10--10. DOI:https://doi.org/10.1167/11.5.10.Google ScholarCross Ref
- Xiao, X. and Wang, J. 2017. Undertanding and Detecting Divided Attention in Mobile MOOC Learning. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17. (2017), 2411--2415. DOI:https://doi.org/10.1145/3025453.3025552.Google ScholarDigital Library
Index Terms
- Moment-to-Moment Detection of Internal Thought during Video Viewing from Eye Vergence Behavior
Recommendations
User Eye Fatigue Detection via Eye Movement Behavior
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing SystemsIn this study we propose and evaluate a novel approach that allows detection of physical eye fatigue. The proposed approach is based on the analysis of the recorded eye movements via what is called behavioral scores. These easy-to-compute scores can be ...
Eye detection and coarse localization of pupil for video-based eye tracking systems
AbstractA video-based eye tracking system generally captures NIR images, each of which contains one or two eyes of a subject. The subject’s point of gaze is then determined using 3D eye model and pupil centre corneal reflection technique. Eye detection ...
Evaluating Calibration-free Webcam-based Eye Tracking for Gaze-based User Modeling
ICMI '22: Proceedings of the 2022 International Conference on Multimodal InteractionEye tracking has been a research tool for decades, providing insights into interactions, usability, and, more recently, gaze-enabled interfaces. Recent work has utilized consumer-grade and webcam-based eye tracking, but is limited by the need to ...
Comments