An Image Statistics–Based Model for Fixation Prediction

In our approach, we model local colour contrast statistics with the Weibull distribution. After that, we estimate the joint distribution of the Weibull parameters at the fixated and non-fixated regions.

Given the Weibull parameters describing an image region, we want to decide whether or not this region is salient. We base our classifier on the log-likelihood ratio [32] of the probability of the Weibull parameters occurrence on the fixated region with respect to the probability of the Weibull parameters occurrence on the non-fixated region: where P(β, γ|fix) and P(β, γ|nonFix) are class-conditional probability density functions of the Weibull parameters β and γ. These probability density functions are estimated using a two-dimensional histogram of the Weibull parameters occurrence on fixated (salient) and non-fixated (non-salient) regions. We estimate P(β, γ|fix) and P(β, γ|nonFix) using images from the training data set.

It is important to investigate the peaks of the saliency map as it is expected that new observers will focus attention there. Therefore, to asses the performance of the proposed Weibull method, we follow [21] and report area under the adapted receiver operating characteristics (ROC) curve. The adapted ROC curve depicts the trade-off between hit rate and the percentage of salient area. Particularly, the hit rate is the ratio of ground truth fixated locations classified as fixated, and we threshold the saliency map such that a given percentage of the most salient image pixels is predicted as fixated and the rest of the image is predicted as non-fixated. Thus, when the whole image is predicted as fixated, the hit rate reaches its maximum. When we lower the threshold, only peaks of the saliency map are predicted as fixated and the hit rate is changing. The aim of accurate fixation prediction is to achieve a high hit rate with a low percentage of salient area. The adapted ROC curve summarizes the performance of a classifier across all possible percentages of salient area. The area under the adapted ROC curve (AUC) is regarded as an indication of the classification power. For the perfect classifier, the AUC equals to 1, and for the random classifier, the AUC is 0.5.

We consider two eye-fixation data sets. The first is the standard eye-fixation data set collected by Bruce and Tsotsos [4]. It contains 120 natural colour images of size 680 × 510 pixels with eye-fixations collected from 20 subjects. All images depict indoor and outdoor urban scenes. In addition, we recorded eye-tracking data for artistic professional photos from National Geographic wallpapers ¹. Eye-fixations of 17 subjects were collected in a free-viewing setting. All participants were naive to the purpose of the study and had normal or corrected to normal vision. Subjects viewed 49 images of size 800 × 540 pixels. These were selected from three categories of National Geographic wallpapers: animals, landscapes and people. Typical pictures are shown in Fig. 3. All procedures conformed with National and Institutional Guidelines for Experiments with human subjects and with the Declaration of Helsinki. Eye movements were recorded using an eye tracker (EyeLink II, SR Research Ltd.), sampling pupil position at 1000 Hz. Subjects were seated in a darkened room at 85 cm from a computer monitor and used a chin-rest so that head position was stable. To calibrate eye position and to validate the calibration, subjects made saccades to the 12 fixation spots on the screen, which appeared one by one in random order. During the experiment, images were presented on a 17 inch screen (FlexScan L568) for 5 s. After each stimulus presentation, a fixation spot appeared at a random position of the screen in order to distribute first fixations uniformly over the experiment. These fixations were excluded from the analysis. Fixation locations and durations were calculated online by the eye tracker. The MATLAB psychophysics toolbox was used for stimulus presentation [3]. In addition, the Eyelinktoolbox was utilized to provide communication with the eyetracker [6].

In our experiments, we first investigate the variability of eye-fixations across subjects in order to construct a stable ground truth. Then, we evaluate the performance of the proposed Weibull method for each single data set. Finally, we investigate the generalization of the Weibull method by a cross data set analysis. We compare the proposed method with the classical saliency map by Itti et al. [19], and with the state-of-the-art method by Bruce and Tsotsos [4]. Both implementations are unaltered code from the original authors. We assume humans to be an ideal saliency detector. Hence, performance of saliency methods is upper-bounded by the behaviour of an inter-subject model. In this model, the saliency map is generated from the fixations of training subjects, and the result is compared with the same ground truth as used in the cross-validation experiments. To construct inter-subject saliency maps, we convolve fixation locations with a fovea-sized two-dimensional Gaussian kernel (σ = 1°, i.e. 30 pixels).

As there is a high inter-subject variability in eye movements, fixation locations are subject dependent. Hence, it is important to consider a sufficient amount of subjects in order to learn consistent patterns in eye-fixation data and to construct a stable ground truth. As we show in the next Sect. 3.4 (Table 1, Fig. 5), subjects have less consistent eye-scanning behaviour on the National Geographic data set in comparison with the Bruce&Tsotsos data set: for the inter-subject model in Table 1, we obtain an AUC of 0.7931 versus 0.8722, respectively. Therefore, we use the National Geographic eye-tracking data set to estimate inter-subject variability. We assess the variation in fixation locations for an increasing number of randomly drawn subjects. Particularly, for each image, we construct a sequence of fixation maps by convolving subject fixations with a fovea-sized two-dimensional Gaussian kernel (σ = 1°, i.e. 30 pixels). An example of fixation maps of the same image using an increasing number of subjects is given in the top row of Fig. 4. As can be observed from the example, the fixation map becomes stable when based on more subjects. The quantitative evaluation is summarized in the bottom of Fig. 4. The graph depicts three distance measures between fixation maps: symmetrical Kullback-Leibler divergence, (one minus) histogram intersection and (one minus) area under the ROC curve (AUC). Clearly, distance decreases as the number of subjects increases. Results show that for all measures, it is hardly possible to distinguish between fixation maps based on more than nine subjects. Hence, nine subjects are enough to construct a stable ground truth regardless of the type of considered distance measure. Given the similar behaviour of the considered distance measures, we will only report further results for the well-established AUC [4, 38, 41].

We start with the evaluation of how well the proposed method predicts human fixations when the Bruce&Tsotsos and the National Geographic data sets are analysed separately. Based on the results of inter-subject variability, as analysed in Sect. 3.3,, we use nine subjects to train the classifier and the remaining subjects to test the results. We consider 5-fold cross-validation and repeat 30 times random drawing of train and test subjects in our experiments. The mean and the standard deviation of the areas under the curves are summarized in Table 1. In more detail, Fig. 5a, b show that the considered saliency methods behave similarly for both the Bruce&Tsotsos and the National Geographic data sets. The proposed Weibull method performs similar to the state-of-the-art Bruce&Tsotsos algorithm. Both outperform the traditional Itti et al. saliency map. Subject performance is higher for the Bruce&Tsotsos data set than for the National Geographic data set, implying more consistent eye-scanning behaviour for the former data set. One possible explanation is that the artistic images from National Geographic wallpapers have more diverse content in comparison with urban images from the Bruce&Tsotsos data set. Therefore, subjects might use more diverse strategies when viewing images from the National Geographic data set. Hence, the National Geographic data set can be seen as a more difficult data set. Note there is still a large gap between human and algorithm performance, which suggests room for improvement. However, not all fixations can be explained by bottom-up features alone [38].

To investigate the generalization of the proposed method, we perform a cross data set experiment. Here, we train the parameters of our model on one data set and evaluate its performance on the other data set. Here, we again use the National Geographic and the Bruce&Tsotsos data sets. In order to minimize the differences between these data sets caused by the way they were recorded, we do not use the whole data sets in the training phase. Instead, we restrict each data set to only 17 subjects, 49 images and eye-fixations acquired within the first 4-seconds viewing time (minus the first fixation). We start with training the parameters of the Weibull method using the National Geographic data set, while evaluating the performance on the Bruce&Tsotsos test set as used in Sect. 3.4. This ensures comparability with Fig. 5. Next, we switch data sets and train from the Bruce&Tsotsos data set while evaluating the performance on the National Geographic test set. Note that the methods by Itti et al. and Bruce&Tsotsos do not involve a training phase. Therefore, their performance in these settings is the same as for the single data set analysis, see Table 1 and Fig. 5. The results are summarized in Table 2. In more detail, Fig. 6a illustrates that the parameters of the Weibull model learned from the National Geographic data set can be used to predict saliency for the Bruce&Tsotsos data set without any performance loss. Hence, the cross-validation used in Sect. 3.4 does not introduce a positive bias in the results (Table 1, Fig. 5). However, as can be seen in Fig. 6b, when the parameters of the Weibull method are trained on the Bruce&Tsotsos data set, our method does not show the same performance as when the parameters are trained on the National Geographic data set. Again, we attribute this to the higher variation in the content of images from the National Geographic data set. As some feature patterns do not occur or occur infrequently in the Bruce&Tsotsos data set, our classifier cannot learn if these are salient or not. Hence, we conclude that our Weibull model has good generalization power when the variation in the training data set is representative for the test data set.

A number of studies investigate how natural image statistics influence the locations of human fixations [28‐30]. Despite the variety of the considered low-level image features, most researchers agree that contrast distribution plays a significant role in guidance of eye movements. Usually, the local contrast is defined as the standard deviation of the image intensities within some small region, divided by the mean intensity within that region, i.e., the local root mean square RMS contrast. However, as the distribution of natural images is non-Gaussian [25, 37], in this paper, we follow [14, 17] and model image contrast with the Weibull distribution. Figure 2 illustrates that the two-parameter Weibull distribution fits the local contrast statistics adequately well. Baddeley and Tatler [1] argue that high-frequency edges turn out to have most impact on fixation prediction, whereas contrast is highly correlated. The next most important feature in their analysis is low-frequency edges. Geusebroek and Smeulders [14] show both contrast and edge frequency to be simultaneously captured by the Weibull distribution. It allows to combine these two image regularities in an elegant way taking into account the strong correlation between them. In our analysis, we investigate a joint distribution of the local contrast and the edge frequency and, thereby, combine low-level image features that are known to be the most powerful in fixation prediction. Moreover, we do not separate high- and low-frequency edges. Instead, we use the minimal reliable scale selection principle [30] as discussed in Sect. 2.1.2 and implicitly consider edges over the available frequency range all together. Inspired by the centre-surround receptive field design of neurons in the retina [18], several successful saliency models are based on comparison of centre-surround regions at each image location [4, 5, 10, 15, 19, 23, 36]. Intuitively, image locations which deviate from their surrounding should be salient. Itti et al. [19] consider visual features salient if they have different brightness, colour or orientation than the surrounding features. Overall, their model combines a total of 42 feature maps into a single saliency map. In contrast, we do not make any assumption about patterns in the spatial structure of feature responses and base our model on comparison of local image statistics with statistics learned from fixation and non-fixation regions. Table 1 and Fig. 5 show that the proposed Weibull method outperforms the method by Itti et al. It might indicate the advantage of direct training of the model parameters from an eye movement data set. Moreover, the higher performance of our method might be due to the explicit modelling of the correlation between image features. Bruce and Tsotsos [4] follow an information-theoretic approach and use information maximization sampling to discriminate centre-surround regions. They calculate Shannon’s self-information based on the likelihood of the local image content in the centre region given the image content of the surround. Regions with unexpected content in comparison with their surrounding are more informative, and thus salient. As shown in Table 1 and Fig. 5, our model achieves a performance comparable with the elaborate approach by Bruce and Tsotsos, while we use as few as two parameters learned from a set of images with associated eye movements. We have explored the generalization of the proposed method by considering the two eye movements data sets: a standard data set from [4] with urban images, and an artistic photo collection with diverse context from National Geographic wallpapers. Examples of images from both data sets are shown in Fig. 3. Table 2 and Fig. 6 show that training the parameters of our Weibull method on the National Geographic data set and testing it on the Bruce&Tsotsos data gives the same results as both training and testing on the Bruce&Tsotsos data. However, for the National Geographic data set, there is a small drop in performance when the parameters of the Weibull model are trained on the Bruce&Tsotsos data instead of the National Geographic. We attribute this to the higher variation in image content from the National Geographic data. We conclude that the proposed model has good generalization power when the variation in the training data set is sufficiently diverse.