ABSTRACT
Human-robot collaboration systems benefit from recognizing people's intentions. This capability is especially useful for collaborative manipulation applications, in which users operate robot arms to manipulate objects. For collaborative manipulation, systems can determine users' intentions by tracking eye gaze and identifying gaze fixations on particular objects in the scene (i.e., semantic gaze labeling). Translating 2D fixation locations (from eye trackers) into 3D fixation locations (in the real world) is a technical challenge. One approach is to assign each fixation to the object closest to it. However, calibration drift, head motion, and the extra dimension required for real-world interactions make this position matching approach inaccurate. In this work, we introduce velocity features that compare the relative motion between subsequent gaze fixations and a finite set of known points and assign fixation position to one of those known points. We validate our approach on synthetic data to demonstrate that classifying using velocity features is more robust than a position matching approach. In addition, we show that a classifier using velocity features improves semantic labeling on a real-world dataset of human-robot assistive manipulation interactions.
- Reuben M. Aronson and Henny Admoni. 2018. Gaze for Error Detection During Human-Robot Shared Manipulation. In Fundamentals of Joint Action workshop, Robotics: Science and Systems.Google Scholar
- Reuben M. Aronson, Thiago Santini, Thomas. C. Kübler, Enkelejda Kasneci, Siddhartha Srinivasa, and Henny Admoni. 2018. Eye-Hand Behavior in Human-Robot Shared Manipulation. In ACM/IEEE International Conference on Human-Robot Interaction. Google ScholarDigital Library
- Rowel Atienza and Alexander Zelinsky. 2005. Intuitive Human-Robot Interaction Through Active 3D Gaze Tracking. In Robotics Research. The Eleventh International Symposium, Paolo Dario and Raja Chatila (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 172--181.Google ScholarCross Ref
- Chaiyawan Auepanwiriyakul, Alex Harston, Pavel Orlov, Ali Shafti, and A. Aldo Faisal. 2018. Semantic Fovea: Real-time Annotation of Ego-centric Videos with Gaze Context. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA '18). ACM, New York, NY, USA, Article 87, 3 pages. Google ScholarDigital Library
- Matthias Bernhard, Efstathios Stavrakis, Michael Hecher, and Michael Wimmer. 2014. Gaze-to-Object Mapping During Visual Search in 3D Virtual Environments. ACM Trans. Appl. Percept. 11, 3, Article 14 (Aug. 2014), 17 pages. Google ScholarDigital Library
- C. Braunagel, E. Kasneci, W. Stolzmann, and W. Rosenstiel. 2015. Driver-activity recognition in the context of conditionally autonomous driving. In IEEE 18th International Conference on Intelligent Transportation Systems. IEEE, 1652--1657. Google ScholarDigital Library
- Anne-Marie Brouwer and David C. Knill. 2007. The role of memory in visually guided reaching. Journal of Vision 7, 5 (06 2007), 6--6.Google ScholarCross Ref
- S. Garrido-Jurado, R. Muñoz Salinas, F.J. Madrid-Cuevas, and M.J. Marín-Jiménez. 2014. Automatic Generation and Detection of Highly Reliable Fiducial Markers Under Occlusion. Pattern Recogn. 47, 6 (June 2014), 2280--2292. Google ScholarDigital Library
- Kakeru Hagihara, Keiichiro Taniguchi, Irshad Abibouraguimane, Yuta Itoh, Keita Higuchi, Jiu Otsuka, Maki Sugimoto, and Yoichi Sato. 2018. Object-wise 3D Gaze Mapping in Physical Workspace. In Proceedings of the 9th Augmented Human International Conference (AH '18). ACM, New York, NY, USA, Article 25, 5 pages. Google ScholarDigital Library
- Kevin A Hallgren. 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology 8, 1 (2012), 23--34. http://www.ncbi.nlm.nih.gov/pubmed/22833776Google Scholar
- Mary Hayhoe and Dana Ballard. 2005. Eye movements in natural behavior. Trends in Cognitive Sciences 9, 4 (2005), 188--194.Google ScholarCross Ref
- Mary M. Hayhoe, Anurag Shrivastava, Ryan Mruczek, and Jeff B. Pelz. 2003. Visual memory and motor planning in a natural task. Journal of Vision 3, 1 (02 2003), 6--6.Google ScholarCross Ref
- Laura Herlant, Rachel Holladay, and Siddhartha Srinivasa. 2016. Assistive Teleoperation of Robot Arms via Automatic Time-Optimal Mode Switching. In ACM/IEEE International Conference on Human-Robot Interaction. Google ScholarDigital Library
- Roy S. Hessels, Diederick C. Niehorster, Marcus Nyström, Richard Andersson, and Ignace T. C. Hooge. 2018. Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers. Royal Society Open Science 5, 8 (aug 2018), 180502.Google ScholarCross Ref
- Roland S Johansson, Gö Ran Westling, Anders Bäckström, and J Randall Flanagan. 2001. Eye-Hand Coordination in Object Manipulation. The Journal of Neuroscience 21, 17 (2001), 6917--6932.Google ScholarCross Ref
- Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. (1980), 329--354.Google Scholar
- Enkelejda Kasneci, Gjergji Kasneci, Thomas C. Kübler, and Wolfgang Rosenstiel. 2014. The applicability of probabilistic methods to the online recognition of fixations and saccades in dynamic scenes. In Proceedings of the Symposium on Eye Tracking Research and Applications - ETRA '14. ACM Press, New York, New York, USA, 323--326. Google ScholarDigital Library
- Michael Land, Neil Mennie, and Jennifer Rusted. 1999. The Roles of Vision and Eye Movements in the Control of Activities of Daily Living. Perception 28, 11 (1999), 1311--1328. PMID: 10755142.Google ScholarCross Ref
- Michael F. Land and Mary Hayhoe. 2001. In what ways do eye movements contribute to everyday activities? Vision Research 41, 25 (2001), 3559--3565.Google ScholarCross Ref
- S. Li, X. Zhang, and J. D. Webb. 2017. 3-D-Gaze-Based Robotic Grasping Through Mimicking Human Visuomotor Function for People With Motion Impairments. IEEE Transactions on Biomedical Engineering 64, 12 (Dec 2017), 2824--2835.Google ScholarCross Ref
- R. Mantiuk, B. Bazyluk, and R. K. Mantiuk. 2013. Gaze-driven Object Tracking for Real Time Rendering. Computer Graphics Forum 32, 2pt2 (2013), 163--173.Google Scholar
- Neil Mennie, Mary Hayhoe, and Brian Sullivan. 2007. Look-ahead fixations: anticipatory eye movements in natural tasks. Experimental Brain Research 179, 3 (01 May 2007), 427--442.Google Scholar
- T. Moore and D. Stouch. 2014. A Generalized Extended Kalman Filter Implementation for the Robot Operating System. In Proceedings of the 13th International Conference on Intelligent Autonomous Systems (IAS-13). Springer.Google Scholar
- Benjamin A. Newman, Reuben M. Aronson, Siddhartha S. Srinivasa, Kris Kitani, and Henny Admoni. 2018. HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaboration. ArXiv e-prints (July 2018). arXiv:cs.RO/1807.11154Google Scholar
- Pavel Orlov, Ali Shafti, Chaiyawan Auepanwiriyakul, Noyan Songur, and A. Aldo Faisal. 2018. A Gaze-contingent Intention Decoding Engine for Human Augmentation. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA '18). ACM, New York, NY, USA, Article 91, 3 pages. Google ScholarDigital Library
- Lucas Paletta, Katrin Santner, Gerald Fritz, Albert Hofmann, Gerald Lodron, Georg Thallinger, and Heinz Mayer. 2013a. FACTS - A Computer Vision System for 3D Recovery and Semantic Mapping of Human Factors. In Computer Vision Systems, Mei Chen, Bastian Leibe, and Bernd Neumann (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 62--72. Google ScholarDigital Library
- Lucas Paletta, Katrin Santner, Gerald Fritz, Heinz Mayer, and Johann Schrammel. 2013b. 3D Attention: Measurement of Visual Saliency Using Eye Tracking Glasses. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHIEA '13). ACM, New York, NY, USA, 199--204. Google ScholarDigital Library
- Thies Pfeiffer. 2012. Measuring and Visualizing Attention in Space with 3D Attention Volumes. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, NY, USA, 29--36. Google ScholarDigital Library
- Thies Pfeiffer and Patrick Renner. 2014. EyeSee3D: A Low-cost Approach for Analyzing Mobile 3D Eye Tracking Data Using Computer Vision and Augmented Reality Technology. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '14). ACM, New York, NY, USA, 369--376. Google ScholarDigital Library
- Thies Pfeiffer, Patrick Renner, and Nadine Pfeiffer-Leßmann. 2016. EyeSee3D 2.0: Model-based Real-time Analysis of Mobile Eye-tracking in Static and Dynamic Three-dimensional Scenes. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA '16). ACM, New York, NY, USA, 189--196. Google ScholarDigital Library
- Pupil Labs, Inc. 2017. Pupil Labs - Pupil. Retrieved Jan 5, 2018 from https://pupil-labs.com/pupil/Google Scholar
- Karishma Singh, Mahmoud Kalash, and Neil Bruce. 2018. Capturing Real-world Gaze Behaviour: Live and Unplugged. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA '18). ACM, New York, NY, USA, Article 20, 9 pages. Google ScholarDigital Library
- Enkelejda Tafaj, Gjergji Kasneci, Wolfgang Rosenstiel, and Martin Bogdan. 2012. Bayesian Online Clustering of Eye Movement Data. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, NY, USA, 285--288. Google ScholarDigital Library
- Mélodie Vidal, Andreas Bulling, and Hans Gellersen. 2013. Pursuits: Spontaneous Interaction with Displays Based on Smooth Pursuit Eye Movement and Moving Targets. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 439--448. Google ScholarDigital Library
- Haofei Wang, Jimin Pi, Tong Qin, Shaojie Shen, and Bertram E. Shi. 2018. SLAM-based Localization of 3D Gaze Using a Mobile Eye Tracker. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA '18). ACM, New York, NY, USA, Article 65, 5 pages. Google ScholarDigital Library
- L. Wu and H. Ren. 2017. Finding the Kinematic Base Frame of a Robot by Hand-Eye Calibration Using 3D Position Data. IEEE Transactions on Automation Science and Engineering 14, 1 (Jan 2017), 314--324.Google ScholarCross Ref
Index Terms
- Semantic gaze labeling for human-robot shared manipulation
Recommendations
Eye-Hand Behavior in Human-Robot Shared Manipulation
HRI '18: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot InteractionShared autonomy systems enhance people's abilities to perform activities of daily living using robotic manipulators. Recent systems succeed by first identifying their operators' intentions, typically by analyzing the user's joystick input. To enhance ...
Beyond gaze: preliminary analysis of pupil dilation and blink rates in an fMRI study of program comprehension
EMIP '18: Proceedings of the Workshop on Eye Movements in ProgrammingResearchers have been employing psycho-physiological measures to better understand program comprehension, for example simultaneous fMRI and eye tracking to validate top-down comprehension models. In this paper, we argue that there is additional value in ...
A non-contact device for tracking gaze in a human computer interface
Special issue on eye detection and trackingThis paper presents a novel design for a non-contact eye detection and gaze tracking device. It uses two cameras to maintain real-time tracking of a person's eye in the presence of head motion. Image analysis techniques are used to obtain accurate ...
Comments