Spatial and temporal visual attention prediction in videos using eye movement data

Abstract

Visual attention detection in static images has achieved outstanding progress in recent years whereas much less effort has been devoted to learning visual attention in video sequences. In this paper, we propose a novel method to model spatial and temporal visual attention for videos respectively through learning from human gaze data. The spatial visual attention mainly predicts where viewers look in each video frame while the temporal visual attention measures which video frame is more likely to attract viewers׳ interest. Our underlying premise is that objects as well as their movements, instead of conventional contrast-related information, are major factors in dynamic scenes to drive visual attention. Firstly, the proposed models extract two types of bottom-up features derived from multi-scale object filter responses and spatiotemporal motion energy, respectively. Then, spatiotemporal gaze density and inter-observer gaze congruency are generated using a large collection of human-eye gaze data to form two training sets. Finally, prediction models of temporal visual attention and spatial visual attention are learned based on those two training sets and bottom-up features, respectively. Extensive evaluations on publicly available video benchmarks and applications in interestingness prediction of movie trailers demonstrate the effectiveness of the proposed work.

Introduction

The human visual attention mechanism plays an important role in selecting the most informative contents from the massive input for further processing. In the engineering field, computational visual attention models have been developed to mimic the human visual attention mechanism, which automatically estimates which locations are more likely to attract viewers׳ attention in a scene based on analyzing visual features. These models can serve as a foundation for many multimedia applications such as image/video retrieval [1], image segmentation [2], [37], [41], video summarization [3], video retargeting [4], and video surveillance [5].

In recent years, most efforts have been devoted to studying computational models for static images whereas limited work has been performed to explore visual attention of users when they watch dynamic videos. As videos are fundamentally different from static images, those computational models developed for static images cannot be applied to dynamic videos directly. For video understanding, two aspects of visual attention need to be addressed: spatial visual attention (SVA) and temporal visual attention (TVA). The computation of SVA is to predict where viewers look in each video frame whereas the TVA mainly aims to study when (at which frames) viewers are more likely to be attracted by the video contents. SVA has gained relatively more studies in the literature. In contrast, as far as we know, very limited work has been done to model TVA systematically.

Motivated by the findings that visual attention and eye movement are highly correlated [6], a recent promising idea is to implement eye trackers to investigate the relation and further build the mapping between eye fixations and visual features. In addition, recent research [8] suggests that the allocation of attention while image/video viewing is primarily driven by a number of the higher-level cognitive factors, e.g., meaningful objects. This differs from most traditional methods, in which only global or local contrast information is considered. The concept [8] sounds interesting, however, there lack fully automatic computational models of visual attention via modeling “meaningful” objects.

In this paper, we attempt to develop a unified framework to model both SVA and TVA for videos by learning from human gaze data. Our underlying assumption is that objects and their movements instead of traditional contrast-related information are major factors in dynamic scenes to drive human visual attention. The proposed framework consists of three major components. First, we extract two types of bottom-up features derived from multi-scale object filter responses and oriented spatiotemporal motion energy, respectively. Second, spatiotemporal gaze density and inter-observer gaze congruency are generated using a large collection of human-eye gaze data, forming two training sets. Then, prediction models of temporal visual attention and spatial visual attention are learned based on those two training sets and the bottom-up features, respectively.

The rest in this paper is organized as follows. Section 2 reviews related works. Section 3 describes how to model SVA and Section 4 presents the computation of TVA. Experimental results are shown in Section 5. One possible application of TVA is described in Section 6. Finally, conclusions are drawn in Section 7.