Jiajia Li
Grace Ngai
Skip Abstract Section
Abstract
Affective computing has recently become an important area in human-computer interaction research. Techniques have been developed to enable computers to understand human affects or emotions, in order to predict human intention more precisely and provide better service to user to enhance user experience. In this paper, we investigate into the detection of human attention level as a useful from of human affect, which could be influential in intelligent e-learning applications. We adopt ubiquitous hardware available in most computer systems, namely, webcam and mouse. Information from multiple input modalities is fused together for effective human attention detection. We invite human subjects to carry out experiments in reading articles when being imposed upon different kinds of distraction to induce them into different levels of attention. Machine-learning techniques are applied to identify useful features to recognize human attention level by building up user-independent models. Our result indicate performance improvement with multimodal inputs from webcam and mouse over that of a single device. We believe that our work has revealed and intersting affective computing direction with potential application in e-learning.
- Adamczyk, P.D. and Bailey, B.P. If not now, when? Proceedings of Conference on Human fators in Computing Systems - CHI '04, ACM Press (2004), 271--278. Google Scholar
Digital Library
- Ambadar, Z., Cohn, J.F., and Reed, L.I. All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous. Journal of Nonverbal Behavior 33, 1 (2009), 17--34.Google ScholarCross Ref
- Anderson, A. R., Christenson, S.L., Sinclair, M.F., and Lehr, C.A. Check & Connect: The importance of relationships for promoting engagement with school. Journal of School Psychology 42, 2 (2004), 95--113.Google ScholarCross Ref
- Ashraf, A.B., Lucey, S., Cohn, J.F., et al. The Painful Face - Pain Expression Recognition Using Active Appearance Models. Image and Vision Computing 27, 12(2009), 1788--1796. Google ScholarDigital Library
- Avila, C., Furnham, A., and McClelland, A. The influence of distracting familiar vocal music on cognitive performance of introverts and extraverts. Psychology of Music 40, 1 (2012), 84--93.Google ScholarCross Ref
- Bixler, R. and D'Mello, S. Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. Proceedings of International Conference on Intelligent User Interfaces - IUI '13, ACM Press (2013), 225--234. Google ScholarDigital Library
- Bolt, R.A. "Put-that-there." Proceedings of Conference on Computer Graphics and Interactive Techniques - SIGGRAPH '80, ACM Press (1980), 262--270. Google ScholarDigital Library
- Calvo, R.A. and D'Mello, S. Affect Detection: An Interdiciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Appective Computing 1, 1 (2010), 18--37. Google ScholarDigital Library
- Carrino, S., Péclat, A., Mugellini, E., Abou Khaled, O., and Ingold, R. Humans and smart environments. Proceedings of International Conference on Multimodal Interfaces - ICMI '11, ACM Press (2011), 105--112. Google ScholarDigital Library
- Conningham, D.W., Kleiner, M., Bülthoff, H.H., and Wallraven, C. The components of conversational facial expressions. Proceedings of Symposium on Applied perception in Graphics and Visualization - APGV '04, ACM Press (2004), 143--150. Google ScholarDigital Library
- Divjak, M. and Bischof, H. Eye blink based fatigue detection for prevention of computer vision syndrome. Proceedings of IAPR Conference on Machine Vision Applications, (2009), 350--353.Google Scholar
- Ekman, P. Universals and Cultural Differences in Facial Expression of Emotion. Nebraska Symposium on Motivation, (1972), 207--283.Google Scholar
- Fox, A.B., Rosen, J., and Crawford, M. Distractions, Distractions: Does Instant Messaging Affect College Students' Performance on a Concurrent Reading Comprehension Task? CyberPsychology & Behavior 12, (2009), 51--53.Google ScholarCross Ref
- Fu, Y., Leong, H.V., Ngai, G., Huang, M.X., and Chan, S.C.F. Physiological Mouse: Towards an Emotion-Aware Mouse. Proceedings of International Conference Workshops on Computer Software and Applications, IEEE (2014), 258--263. Google ScholarDigital Library
- Gross, R., Matthews, I., Cohn, J., Kanade, T., and Baker, S. Multi-PIE. Proceedings of International Conference on Automatic Face & Gesture Recognition, IEEE (2008), 1--8.Google Scholar
- Henderson, J.M. and Hollingworth, A. Eye movements during scene viewing: an overview. In G. Underwood, ed., Eye Guidance in Reading and Scene Perception. Elsevier Science, 1998, 269--293.Google Scholar
- Huang, M., Ngai, G., Hua, K., Chan, S., and Leong, H.V. Identifying User-Specific Facial Affects from Spontaneous Expressions with Minimal Annotation. IEEE Transactions on Affective Computing PP, 99 (2015), 1--14.Google Scholar
- Huang, M.X., Kwok, T.C.K., Ngai, G., Leong H.V., and Chan, S.CV.F. Building a Self-Learning Eye Gaze Model from User Interaction Data. Proceedings of International Conference on Multimedia - MM '14, ACM Press (2014), 1017--1020. Google ScholarDigital Library
- Huang, X.M., Li, J., Ngai, G., and Leong, H.V. StressClick: Sensing Stress from Gaze-Click Patterns. To appear in Proceedings of International Conference on Multimedia, (2016). Google ScholarDigital Library
- Inhoff, A.W. and Rayner, K. Parafoveal word processing during eye fixations in reading: Effects of word frequency. Preception & Psychophysics 40, 6 (1986), 431--439.Google Scholar
- Ji, Q., Lan, P., and Looney, C.A probabilistic framework for modeling ad real-time monitoring human fatigue. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 36, 5 (2006), 862--875. Google ScholarDigital Library
- Liversedge, S.P. and Findlay, J.M. Saccadic eye movements and cognition. Trends in Congnitive Sciences 4, 1 (2000), 6--14.Google ScholarCross Ref
- Oviatt, S.L. Multimodal interfaces. In Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emergine Applications. L. Erlbaum Associates Inc, Hillsdale, NJ, USA, 2007, 286--304. Google ScholarDigital Library
- Picard, R. W. Affective computing. MIT Press, Cambridge, MA, USA, 1997. Google ScholarDigital Library
- Radach, R. and Kennedy, A. Theoretical Perspectives on eye movements in reading: Past controversies, current issues, and an agenda for future research. European Journal of Cognitive Psychology 16, 1--2 (2004), 3--26.Google ScholarCross Ref
- Rayner, K. Eye Movements in reading and information processing: 20 years of research. Psychological bulletin 124, 3 (1998), 372--422.Google Scholar
- Reichle, E.D., Reineberg, A.E., and Schooler, J.W. Eye Movements During Mindless Reading. Psychological Science 21, 9 (2010), 1300--1310.Google Scholar
- Rodrigue, M., Son, J., Giesbrecht, B., Truk, M., and Höllerer, T. Spatio-Temporal Detection of Divided Attention in reading Applications Using EEG and Eye Tracking Proceedings of International Conference on Intelligent User Interfaces - IUI '15, ACM Press (2015), 121--125. Google ScholarDigital Library
- Salvucci, D.D. and Goldberg, J.H. Identifying fixations and saccades in eye-tracking protocols. Proceedings of Symposium on Eye Tracking Research & Applications - ETRA '00, ACM Press (2000), 71--78. Google ScholarDigital Library
- Saragih, J.M., Lucey, S., and Cohn, J.F. Deformable Model Fitting by Regularized Landmark Mean-Shift. International Journal of Computer Vision 91, 2 (2010), 200--215 Google ScholarDigital Library
- Schleicher, R., Galley, N., Briest, S., and Galley, L. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (2008), 982--1010.Google ScholarCross Ref
- Sen, T. and Megaw, T. The Effects of Task Variables and Prolonged Perfomance on Saccadic Eye Movement Parameters. In A.G. Gale and F. Johnson, eds., Theoretical and Applied Aspects of Eye Movement Research, Elsevier, 1984, 103--111.Google Scholar
- Talavera, L. An Evaluation of Filter and Wrapper Methods for Feature Selection in Categorical Clustering. In A. Famili, J. Kok, J. Pena, A. Siebes and A. Feelders, eds., Advances in Intelligent Data Analysis VI. Springer Berlin Heidlberg, 2005, 440--451. Google ScholarDigital Library
- Tsalakanidou, F. and Malassiotis, S. Real-time 2D+3D facial action and expression recognition. Pattern Recognition 43, 5 (2010), 1763--1775. Google ScholarDigital Library
- Tsoulouhas, G., Georgiou, D., and Karkos, A. Detection of Learners' Affective State Based on Mouse Movement. Journal of Computing 3, 11 (2011), 9--18.Google Scholar
- Vizer, L.M., Zhou, L., and Sears, A. Automated stress detection using keystroke and linguistic features: An exploratory study International Journal of Human-Computer Studies 67, 10 (2009), 870--886. Google ScholarDigital Library
- Weigle, C. and Banks, D.C. Analysis of eye-tracking experiments performed on a Tobii T6. Presented at the Conference on Visualization and Data Analysis, (2008), San José California, USA.Google Scholar
- Whitehill, J., Serpell, Z., Foster, A., and Movellan, J.R. The Faces of Engagement: Automatic Recognition of Student Engagementform Facial Expressions. IEEE Transactions on Affective Computing 5, 1 (2004), 86--98.Google Scholar
- Xiong, X. and De la Torre, F. Supervised Descent Method and Its Applications to Face Alignment. Proceedings of Conference on Computer Vision and Pattern Recognition, IEEE (2013), 532--539. Google ScholarDigital Library
- Yamauchi, T. Mouse Trajectories and State Anxiety: Feature Selection with Random Forest. Proceedings of Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE (2013), 399--404. Google ScholarDigital Library
- Zeng, Z., Hu, Y., Roisman, G.I., Wen, Z., Fu, Y., and Huang, T.S. Audio-Visual Spontaneous Emotion Recognition. Artifical Intelligence for Human Computing 4451, (2007), 72--90. Google ScholarDigital Library
- Zeng, Z., Pantic, M., Roisman, G.I., and Huang, T.S. A Survey of Affect Recognition Methods: Audio, visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 1 (2009), 39--58. Google ScholarDigital Library
Index Terms
- Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics
Recommendations
Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze
ICMI '19: 2019 International Conference on Multimodal InteractionContinuous emotion recognition is of great significance in affective computing and human-computer interaction. Most of existing methods for video based continuous emotion recognition utilize facial expression. However, besides facial expression, other ...
Eye, Head and Torso Coordination During Gaze Shifts in Virtual Reality
Humans perform gaze shifts naturally through a combination of eye, head and body movements. Although gaze has been long studied as input modality for interaction, this has previously ignored the coordination of the eyes, head and body. This article ...
Facial and Eye Gaze Detection
BMCV '02: Proceedings of the Second International Workshop on Biologically Motivated Computer VisionGaze detection is to locate the position on a monitor screen where a user is looking. In our work, we implement it with a computer vision system setting a IR-LED based single camera. To detect the gaze position, we locate facial features, which is ...
Comments