Andreas Bulling
Jamie A. Ward
ABSTRACT
In this work we investigate eye movement analysis as a new modality for recognising human activity. We devise 90 different features based on the main eye movement characteristics: saccades, fixations and blinks. The features are derived from eye movement data recorded using a wearable electrooculographic (EOG) system. We describe a recognition methodology that combines minimum redundancy maximum relevance feature selection (mRMR) with a support vector machine (SVM) classifier. We validate the method in an eight participant study in an office environment using five activity classes: copying a text, reading a printed paper, taking hand-written notes, watching a video and browsing the web. In addition, we include periods with no specific activity. Using a person-independent (leave-one-out) training scheme, we obtain an average precision of 76.1% and recall of 70.5% over all classes and participants. We discuss the most relevant features and show that eye movement analysis is a rich and thus promising modality for activity recognition.
- D. Bannach, P. Lukowicz, and O. Amft. Rapid Prototyping of Activity Recognition Applications. IEEE Pervasive Computing, 7(2):22--31, 2008. Google Scholar
Digital Library
- L. Bao and S.S. Intille. Activity Recognition from User-Annotated Acceleration Data. In Proc. Pervasive 2004, pages 1--17. Springer, 2004.Google ScholarCross Ref
- A. Bulling, D. Roggen, and G. Tröster. Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments. Journal of Ambient Intelligence and Smart Environments (JAISE), 1(2):157--171, 2009. Google ScholarDigital Library
- A. Bulling, J.A. Ward, H. Gellersen, and G. Tröster. Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography. In Proc. Pervasive 2008, pages 19--37. Springer, 2008. Google ScholarDigital Library
- P.P. Caffier, U. Erdmann, and P. Ullsperger. Experimental evaluation of eye-blink parameters as a drowsiness measure. European Journal of Applied Physiology, 89(3-4):319--325, 2003.Google ScholarCross Ref
- R.L. Canosa. Real-world vision: Selective perception and task. ACM Transactions on Applied Perception, 6(2):1--34, 2009. Google ScholarDigital Library
- K. Crammer and Y. Singer. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951--991, 2003. Google ScholarDigital Library
- L. Dempere-Marco, X. Hu, S.L.S. MacDonald, S.M. Ellis, D.M. Hansell, and G.-Z. Yang. The use of visual search for knowledge gathering in image decision support. IEEE Transactions on Medical Imaging, 21(7):741--754, 2002.Google ScholarCross Ref
- A.T. Duchowski. Eye Tracking Methodology: Theory and Practice. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2007. Google ScholarDigital Library
- H. Ehrlichman, D. Micic, A. Sousa, and J. Zhu. Looking for answers: Eye movements in non-visual cognitive tasks. Brain and Cognition, 64(1):7--20, 2007.Google ScholarCross Ref
- M. Elhelw, M. Nicolaou, A. Chung, G.-Z. Yang, and M.S. Atkins. A gaze-based study for investigating the perception of visual realism in simulated scenes. ACM Transactions on Applied Perception, 5(1):1--20, 2008. Google ScholarDigital Library
- J. Fogarty, C. Au, and S.E. Hudson. Sensing from the basement: a feasibility study of unobtrusive and low-cost home activity recognition. In Proc. UIST 2006, pages 91--100. ACM Press, 2006. Google ScholarDigital Library
- J.J. Gu, M. Meng, A. Cook, and G. Faulkner. A study of natural eye movement detection and ocular implant movement control using processed EOG signals. In Proc. ICRA 2001, volume 2, pages 1555--1560, 2001.Google ScholarCross Ref
- S.S. Hacisalihzade, L.W. Stark, and J.S. Allen. Visual perception and sequences of eye movement fixations: a stochastic modeling approach. IEEE Transactions on Systems, Man and Cybernetics, 22(3):474--481, 1992.Google ScholarCross Ref
- J.M. Henderson. Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11):498--504, 2003.Google ScholarCross Ref
- T. Huynh, M. Fritz, and B. Schiele. Discovery of Activity Patterns using Topic Models. In Proc. UbiComp 2008, pages 10--19. ACM Press, 2008. Google ScholarDigital Library
- C.N. Karson, K.F. Berman, E.F. Donnelly, W.B. Mendelson, J.E. Kleinman, and R.J. Wyatt. Speaking, thinking, and blinking. Psychiatry Research, 5(3):243--246, 1981.Google ScholarCross Ref
- N. Kern, B. Schiele, and A. Schmidt. Recognizing context for annotating a live life recording. Personal and Ubiquitous Computing, 11(4):251--263, 2007. Google ScholarDigital Library
- C.-J. Lin. LIBLINEAR -- a library for large linear classification, 2008. http://www.csie.ntu.edu.tw/Ücjlin/liblinear/.Google Scholar
- S.P. Liversedge and J.M. Findlay. Saccadic eye movements and cognition. Trends in Cognitive Sciences, 4:6--14, 2000.Google ScholarCross Ref
- B. Logan, J. Healey, M. Philipose, E. Tapia, and S. S. Intille. A Long-Term Evaluation of Sensing Modalities for Activity Recognition. In Proc. UbiComp 2007, pages 483--500. ACM Press, 2007. Google ScholarDigital Library
- D. Palomba, M. Sarlo, A. Angrilli, A. Mini, and L. Stegagno. Cardiac responses associated with affective processing of unpleasant film stimuli. International Journal of Psychophysiology, 36(1):45--57, 2000.Google ScholarCross Ref
- H. Peng. mRMR Feature Selection Toolbox for MATLAB, 2007. http://research.janelia.org/peng/proj/mRMR/.Google Scholar
- H. Peng, F. Long, and C. Ding. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1226--1238, 2005. Google ScholarDigital Library
- D.D. Salvucci and J.R. Anderson. Automated eye-movement protocol analysis. Human-Computer Interaction, 16(1):39--86, 2001. Google ScholarDigital Library
- R. Schleicher, N. Galley, S. Briest, and L. Galley. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics, 51(7):982--1010, 2008.Google ScholarCross Ref
- M.A. Tinati and B. Mozaffary. A wavelet packets approach to electrocardiograph baseline drift cancellation. International Journal of Biomedical Imaging, Article ID 97157, 2006.Google Scholar
- J.A. Ward, P. Lukowicz, and G. Tröster. Evaluating performance in continuous context recognition using event-driven error characterisation. In Proc. LoCA 2006, pages 239--255, 2006. Google ScholarDigital Library
- H. Widdel. Theoretical and Applied Aspects of Eye Movement Research, chapter Operational problems in analysing eye movements, pages 21--29. Elsevier, 1984.Google Scholar
Index Terms
- Eye movement analysis for activity recognition
Recommendations
Recognition of visual memory recall processes using eye movement analysis
UbiComp '11: Proceedings of the 13th international conference on Ubiquitous computingPhysical activity, location, as well as a person's psychophysiological and affective state are common dimensions for developing context-aware systems in ubiquitous computing. An important yet missing contextual dimension is the cognitive context that ...
It's in your eyes: towards context-awareness and mobile HCI using wearable EOG goggles
UbiComp '08: Proceedings of the 10th international conference on Ubiquitous computingIn this work we describe the design, implementation and evaluation of a novel eye tracker for context-awareness and mobile HCI applications. In contrast to common systems using video cameras, this compact device relies on Electrooculography (EOG). It ...
Exploring Eye Expressions for Enhancing EOG-Based Interaction
Human-Computer Interaction – INTERACT 2023AbstractThis paper explores the classification of eye expressions for EOG-based interaction using JINS MEME, an off-the-shelf eye-tracking device. Previous studies have demonstrated the potential for using electrooculography (EOG) for hands-free human-...
Comments