Deep Filter Banks for Texture Recognition, Description, and Segmentation

Psychological experiments suggest that, while there are a few hundred words that people commonly use to describe textures, this vocabulary is redundant and can be reduced to a much smaller number of representative words. Our starting point is the list of 98 words identified by Bhushan et al. (1997). Their seminal work aimed to achieve for texture recognition the same that color words have achieved for describing color spaces (Berlin and Kay 1991). However, their work mainly focuses on the cognitive aspects of texture perception, including perceptual similarity and the identification of directions of perceptual texture variability. Since our interest is in the visual aspects of texture, words such as “corrugated” that are more related to surface shape or haptic properties were ignored. Other words such as “messy” that are highly subjective and do not necessarily correspond to well defined visual features were also ignored. After this screening phase we analyzed the remaining words and merged similar ones such as “coiled”, “spiraled” and “corkscrewed” into a single term. This resulted in a set of 47 words, illustrated in Fig. 2.

So far only the key attribute of each image is known while any of the remaining 46 attributes may apply as well. Exhaustively collecting annotations for 46 attributes and 5640 texture images is fairly expensive. To reduce this cost we propose to exploit the correlation and sparsity of the attribute occurrences (Fig. 3). For each attribute q, twelve key images are annotated exhaustively and used to estimate the probability \(p(q'|q)\) that another attribute \(q'\) could co-exist with q. Then for the remaining key images of attribute q, only annotations for attributes \(q'\) with non negligible probability are collected, assuming that the remaining attributes would not apply. In practice, this requires annotating around 10 attributes per texture instance, instead of 47. This procedure occasionally misses attribute annotations; Fig. 3 evaluates attribute recall by 12-fold cross-validation on the 12 exhaustive annotations for a fixed budget of collecting 10 annotations per image.

A further refinement is to suggest which attributes \(q'\) to annotate not just based on the prior \(p(q'|q)\), but also based on the appearance of an image \(\mathbf {x}_i\). This was done by using the attribute classifier learned in Sect. 6; after Platt’s calibration (Platt 2000) on a held-out test set, the classifier score \(c_{q'}(\mathbf {x}_i) \in \mathbb {R}\) is transformed in a probability \(p(q'|\mathbf {x}_i) = \sigma (c_{q'}(\mathbf {x}))\) where \(\sigma (z) = 1 / (1 + e^{-z})\) is the sigmoid function. By construction, Platt’s calibration reflects the prior probability \(p(q') \approx p_0 = 1/47\) of \(q'\) on the validation set. To reflect the probability \(p(q'|q)\) instead, the score is adjusted asand used to find which attributes should be annotated for each image. As shown in Fig. 3, for a fixed annotation budget this method increases attribute recall.

Overall, with roughly 10 annotations per image it was possible to recover all of the attributes for at least 75 % of the images, and miss one out of four (on average) for another 20 %, while keeping the annotation cost to a reasonable level. To put this in perspective, directly annotating the 5640 images for 46 attributes and collecting five annotations per attributed would have required 1.2M binary annotations, i.e. roughly 12K USD at the very low rate of 1¢ per annotation. Using the proposed method, the cost would have been 546 USD. In practice, we spent around 2.5K USD in order to pay annotators better as well as to account for occasional errors in setting up experiments and the fact that, as explained above, bootstrapping still relies on exhaustive annotations for a subset of the data.

The study of perceptual properties of textures originated in computer vision as well as in cognitive sciences. Some of the earliest work on texture perception conducted by Julesz (1981) focussed on pre-attentive aspects of perception. It led to the concept of “textons,” primitives such as line-terminators, crossings, intersections, etc., that are responsible for pre-attentive discrimination of textures. In computer vision, Tamura et al. (1978) identified six common directions of variability of images in the Broadatz dataset; coarse versus fine; high-contrast versus low-contrast; directional versus non-directional; linelike versus bloblike; regular versus irregular; and rough versus smooth. Similar perceptual attributes of texture (Amadasun and King 1989; Bajcsy 1973) have been found by other researchers.

Our work is motivated by that of Rao and Lohse (1996) and Bhushan et al. (1997). Their experiments suggest that there is a strong correlation between the structure of the lexical space and perceptual properties of texture. While they studied the psychological aspects of texture perception, the focus of this paper is the challenge of estimating such properties from images automatically. Their work Bhushan et al. (1997), in particular, identified a set of words sufficient to describe a wide variety of texture patterns; the same set of words was used to bootstrap DTD.

While recent work in computer vision has been focussed on texture identification and material recognition, notable contributions to the recognition of perceptual properties exist. Most of this work is part of the general research on visual attributes (Farhadi et al. 2009; Parikh and Grauman 2011; Patterson and Hays 2012; Bourdev et al. 2011; Kumar et al. 2011). Texture attributes have an important role in describing objects, particularly for those that are best characterized by a pattern, such as items of clothing and parts of animals such as birds. Notably, the first work on modern visual attributes by Ferrari and Zisserman (2007) focused on the recognition of a few perceptual properties of textures. Later work, such as Berg et al. (2010) that mined visual attributes from images on the Internet, also contain some attributes that describe textures. Nevertheless, so far the attributes of textures have been investigated only tangentially. DTD address the question of whether there exists a “universal” set of attributes that can describe a wide range of texture patterns, whether these can be reliably estimated from images, and for what tasks they are useful.

Datasets that focus on the recognition of subjective properties of textures are less common. One example is Pertex (Clarke et al. 2011), containing 300 texture images taken in a controlled setting (Lambertian renderings of 3D reconstructions of real materials) as well as a semantic similarity matrix obtained form human similarity judgments. The work most related to ours is probably the one of Matthews et al. (2013) that analyzed images in the Outex dataset (Ojala et al. 2002) using a subset of the texture attributes that we consider. DTD differs in scope (containing more attributes) and, especially, in the nature of the data (controlled vs uncontrolled conditions). In particular, working in uncontrolled conditions allows us to transfer the texture attributes to real-world applications, including material recognition in the wild and in clutter, as shown in the experiments.

Most of the recent work in texture recognition focuses on the recognition of texture instances and material categories, as reflected by the development of corresponding benchmarks (Fig. 4). The Brodatz (1966) catalogue was used in early works on textures to study the problem of identifying texture instances (e.g. matching half of the texture image given the other half). Others including CUReT (Dana et al. 1999), UIUC (Lazebnik et al. 2005), KTH-TIPS  (Caputo et al. 2005; Hayman et al. 2004), Outex (Ojala et al. 2002), Drexel Texture Database (Oxholm et al. 2012), and ALOT  (Burghouts and Geusebroek 2009) address the recognition of specific instances of one or more materials. UMD (Xu et al. 2009) is similar, but the imaged objects are not necessarily composed of a single material. As textures are imaged under variable truncation, viewpoint, and illumination, these datasets have stimulated the creation of texture representations that are invariant to viewpoint and illumination changes (Varma and Zisserman 2005; Ojala et al. 2002; Varma and Zisserman 2003; Leung and Malik 2001). Frequently, texture understanding is formulated as the problem of recognizing the material of an object rather than a particular texture instance (in this case any two slabs of marble would be considered equal). KTH-T2b (Mallikarjuna et al. 2006) is one of the first datasets to address this problem by grouping textures not only by the instance, but also by the type of materials (e.g. “wood”).

However, these datasets make the simplifying assumption that textures fill images, and often, there is limited intra-class variability, due to a single or limited number of instances, captured under controlled scale, view-angle and illumination. Thus, they are not representative of the problem of recognizing materials in natural images, where textures appear under poor viewing conditions, low resolution, and in clutter. Addressing this limitation is the main goal of the Flickr Material Database (FMD) (Sharan et al. 2009). FMD samples just one viewpoint and illumination per object, but contains many different object instances grouped in several different material classes. Sect. 3 will introduce datasets addressing the problem of clutter as well.

The performance of recognition algorithms on most of this data is close to perfect, with classification accuracies well above 95 %; KTH-T2b and FMD are an exception due to their increased complexity. A review of these datasets and classification methodologies is presented in Timofte and Van Gool (2012), who also propose a training-free framework to classify textures, significantly improving on other methods. Table 1 and Fig. 4 provides a summary of the nature and size of various texture datasets that are used in our experiments.

Some of the earliest local image descriptors were developed as linear filter banks in texture recognition. As an evolution of earlier texture filters (Bovik et al. 1990; Malik and Perona 1990), the filter bank of Leung Malik (LM) Leung and Malik (2001) includes 48 filters matching bars, edges and spots, at various scales and orientations. These filters are first and second derivatives of Gaussians at 6 orientations and 3 scales (36), 8 Laplacian of Gaussian (LOG) filters, and 4 Gaussians. Combinations of the filter responses, identified by vector quantisation (Sect. 4.2.1), were used as the computational basis of the “textons” proposed by Julesz and Bergen (1983). The filter bank MR8 of Varma and Zisserman (2003) and Geusebroek et al. (2003) consists instead of 38 filters, similar to LM. For two of the oriented filters, only the maximum response across the scales is recorded, reducing the number of responses to 8 (3 scales for two oriented filters, and two isotropic – Gaussian and Laplacian of Gaussian).

The importance of using linear filters as local features was later questioned by Varma and Zisserman (2003). The VZ descriptors are in fact small image patches which, remarkably, were shown to outperform LM and MR8 on earlier texture benchmarks such as CuRET. However, as will be demonstrated in the experiments, trivial local descriptors are not competitive in harder tasks.

Another early local image descriptor are the Local Binary Patterns (LBP) of Ojala et al. (1996) and Ojala et al. (2002), a special case of the texture units of Wang and He (1990). A LBP \(\mathbf {d}_i = (b_1,\dots ,b_m)\) computed a pixel \(\mathbf {p}_0\) is the sequence of bits \(b_j = [\mathbf {x}(\mathbf {p}_i) > \mathbf {x}(\mathbf {p}_j)]\) comparing the intensity \(\mathbf {x}(\mathbf {p}_i)\) of the central pixel to the one of m neighbors \(\mathbf {p}_j\) (usually 8 in a circle). LBPs have specialized quantization schemes; the most common one maps the bit string \(\mathbf {d}_i\) to one of a number of uniform patterns (Ojala et al. 2002). The quantized LBPs can be averaged over the image to build a histogram; alternatively, such histograms can be computed for small image patches and used in turn as local image descriptors.

In the context of object recognition, the best known local descriptor is undoubtedly Lowe’s SIFT (Lowe 1999). SIFT is the histogram of the occurrences of image gradients quantized with respect to their location within a patch as well to their orientation. While SIFT was originally introduced to match object instances, it was later applied to an impressive diversity of tasks, from object categorization to semantic segmentation and face recognition.

Handcrafted image descriptors are nowadays outperformed by features learned using the latest generation of deep CNNs (Krizhevsky et al. 2012). A CNN can be seen as a composition \(\phi _K \circ \dots \circ \phi _2 \circ \phi _1\) of K functions or layers. The output of each layer \(\mathbf {x}_k = (\phi _k \circ \dots \circ \phi _2 \circ \phi _1)(\mathbf {x})\) is a descriptor field \(\mathbf {x}_k \in \mathbb {R}^{W_k \times H_k \times D_k}\), where \(W_k\) and \(H_k\) are the width and height of the field and \(D_k\) is the number of feature channels. By collecting the \(D_k\) responses at a certain spatial location, one obtains a \(D_k\) dimensional descriptor vector. The network is called convolutional if all the layers are implemented as (non-linear) filters, in the sense that they act locally and uniformly on their input. If this is the case, since compositions of filters are filters, the feature field \(\mathbf {x}_k\) is the result of applying a non-linear filter bank to the image \(\mathbf {x}\).

As computation progresses, the resolution of the descriptor fields decreases whereas the number of feature channels increases. Often, the last several layers \(\phi _k\) of a CNN are called “fully connected” because, if seen as filters, their support is the same as the size of the input field \(\mathbf {x}_{k-1}\) and therefore lack locality. By contrast, earlier layers that act locally will be referred to as “convolutional”. If there are C convolutional layers, the CNN \(\phi =\phi _e \circ \phi _f\) can be decomposed into a filter bank (local descriptors) \(\phi _f = \phi _C \circ \dots \circ \phi _1\) followed by a pooling encoder \(\phi _e = \phi _K \circ \dots \circ \phi _{C+1}\).

An orderless pooling encoder \(\phi _e\) maps a sequence \(\mathcal {F} = (\mathbf {f}_1,\dots ,\mathbf {f}_n), \mathbf {f}_i\in \mathbb {R}^D\) of local image descriptors to a feature vector \(\phi _e(\mathcal {F}) \in \mathbb {R}^d\). The encoder is orderless in the sense that the function \(\phi _e\) is invariant to permutations of the input \(\mathcal {F}\).⁴ Furthermore, the encoder can be applied to any number of features; for example, the encoder can be applied to the sub-sequence \(\mathcal {F}' \subset \mathcal {F}\) of local descriptors contained in a target image region without recomputing the local descriptors themselves.

All common orderless encoders are obtained by applying a non-linear descriptor encoder \(\eta (\mathbf {f}_i)\in \mathbb {R}^d\) to individual local descriptors and then aggregating the result by using a commutative operator such as average or max. For example, average-pooling yields \(\bar{\phi }_e(\mathcal {F}) = \frac{1}{n}\sum _{i=1}^n \eta (\mathbf {f}_i)\). The pooled vector \(\bar{\phi }_e(\mathcal {F})\) is post-processed to obtain the final representation \(\phi _e(\mathcal {F})\) as discussed later.

The best-known orderless encoder is the Bag of Visual Words (BoVW). This encoder starts by vector-quantizing (VQ) the local features \(\mathbf {f}_i\in \mathbb {R}^D\) by assigning them to their closest visual word in a dictionary \(C = \begin{bmatrix} \mathbf {c}_1&\dots&\mathbf {c}_d\end{bmatrix} \in \mathbb {R}^{D \times d}\) of d elements. Visual words can be thought of as “prototype features” and are obtained during training by clustering example local features. The descriptor encoder \(\eta _1(\mathbf {f}_i)\) is the one-hot vector indicating the visual word corresponding to \(\mathbf {f}_i\) and average-pooling these one-hot vectors yields the histogram of visual words occurrences. BoVW was introduced in the work of Leung and Malik (2001) to characterize the distribution of textons, defined as configuration of local filter responses, and then ported to object instance and category understanding by Sivic and Zisserman (2003) and Csurka et al. (2004) respectively. It was then extended in several ways as described below.

The kernel codebook encoder (Philbin et al. 2008) assigns each local feature to several visual words, weighted by a degree of membership: \([\eta _{\text {KC}}(\mathbf {f}_i)]_j \propto \exp \left( - \lambda \Vert \mathbf {f}_i-\mathbf {c}_j \Vert ^2 \right) \), where \(\lambda \) is a parameter controlling the locality of the assignment. The descriptor code \(\eta _{\text {KC}}(\mathbf {f}_i)\) is \(L^1\) normalized before aggregation, such that \(\Vert \eta _{\text {KC}}(\mathbf {f}_i)\Vert _1 = 1\). Several related methods used concepts from sparse coding to define the local descriptor encoder (Zhou et al. 2010; Liu et al. 2011). Locality constrained linear coding (LLC) (Wang et al. 2010), in particular, extends soft assignment by making the assignments reconstructive, local, and sparse: the descriptor encoder \(\eta _\text {LLC}(\mathbf {f}_i) \in \mathbb {R}^d_+\), \(\Vert \eta _\text {LLC}(\mathbf {f}_i)\Vert _1=1\), \(\Vert \eta _\text {LLC}(\mathbf {f}_i)\Vert _0\le r\) is computed such that \(\mathbf {f}_i \approx C\eta _\text {LLC} (\mathbf {f}_i)\) while allowing only the \(r \ll d\) visual words closer to \(\mathbf {f}_i\) to have a non-zero coeffcient.

In the Vector of locally-aggregated descriptors (VLAD) (Jégou et al. 2010) the descriptor encoder is richer. Local image descriptors are first assigned to their nearest neighbor visual word in a dictionary of K elements like in BoVW; then the descriptor encoder is given by \(\eta _{\text {VLAD}}(\mathbf {f}_i) = (\mathbf {f}_i - C \eta _1(\mathbf {f}_i)) \otimes \eta _1(\mathbf {f}_i)\), where \(\otimes \) is the Kronecker product. Intuitively, this subtracts from \(\mathbf {f}_i\) the corresponding visual word \(C \eta _1(\mathbf {f}_i)\) and then copies the difference into one of K possible subvectors, one for each visual word. Hence average-pooling \(\eta _{\text {VLAD}}(\mathbf {f}_i)\) accumulates first-order descriptor statistics instead of simple occurrences as in BoVW.

VLAD can be seen as a variant of the Fisher vector (FV) (Perronnin and Dance 2007). The FV differs from VLAD as follows. First, the quantizer is not K-means but a Gaussian Mixture Model (GMM) with components \((\pi _k,\mu _k,\Sigma _k),\) \(k=1,\dots ,K\), where \(\pi _k \in \mathbb {R}\) is the prior probability of the component, \(\mu _k\in \mathbb {R}^D\) the Gaussian mean and \(\Sigma _k \in \mathbb {R}^{D\times D}\) the Gaussian covariance (assumed diagonal). Second, hard-assignments \(\eta _1(\mathbf {f}_i)\) are replaced by soft-assignments \(\eta _{\text {GMM}}(\mathbf {f}_i)\) given by the posterior probability of each GMM component. Third, the FV descriptor encoder \(\eta _{\text {FV}}(\mathbf {f}_i)\) includes both first \(\Sigma _k^{-\frac{1}{2}}(\mathbf {f}_i - \mu _k)\) and second order \(\Sigma _k^{-1} (\mathbf {f}_i - \mu _k) \odot (\mathbf {f}_i - \mu _k) - \mathbf {1}\) statistics, weighted by \(\eta _\text {GMM}(\mathbf {f}_i)\) [see Perronnin and Dance (2007), Perronnin et al. (2010) and Chatfield et al. (2011) for details]. Hence, average pooling \(\eta _{\text {FV}}(\mathbf {f}_i)\) accumulates both first and second order statistics of the local image descriptors.

All the encoders discussed above use average pooling, except LLC that uses max pooling.

An order-sensitive encoder differs from an orderless encoder in that the map \(\phi _e(\mathcal {F})\) is not invariant to permutation of the input \(\mathcal {F}\). Such an encoder can therefore reflect the layout of the local image desctiptors, which may be ineffective or even counter-productive in texture recognition, but is usually helpful in the recognition of objects, scenes, and others.

The most common order-sensitive encoder method is the spatial pyramid pooling (SPP) of Lazebnik et al. (2006). SSP transforms any orderless encoder into one with (weak) spatial sensitivity by dividing the image in subregions, computing any encoder for each subregion, and stacking the results. This encoder is only be sensitive to reassignments of the local descriptors to different subregions.

The fully-connected layers (FC) in a CNN also form an order-sensitive encoder. Compared to the encoders seen above, FC are pre-trained discriminatively, which can be either an advantage or disadvantage, depending on whether the information that they captured can be transferred to the domain of interest. FC poolers are much less flexible than the encoders seen above as they work only with a particular type of local descriptors, namely the corresponding CNN convolutional layers. Furthermore, a standard FC pooler can only operate on a well defined layout of local descriptors (e.g. a \(6 \times 6\)), which in turn means that the image needs to be resized to a standard size before the FC encoder can be evaluated. This is particularly expensive when, as in object detection or image segmentation, many image subregions must be considered.

The vector \({\mathbf {y}}= {\phi }_e({\mathcal {F}})\) obtained by pooling local image descriptors is usually post-processed before being used in a classifier. In the simplest case, this amounts to performing \(L^2\) normalization \(\phi _e(\mathcal {F}) = \mathbf {y}/ \Vert \mathbf {y}\Vert _2\). However, this is usually preceded by a non-linear transformation \(\phi _K(\mathbf {y})\) which is best understood in term of kernels. A kernel \(K(\mathbf {y}',\mathbf {y}'')\) specifies a notion of similarity between data points \(\mathbf {y}'\) and \(\mathbf {y}''\). If K is a positive semidefinite function, then it can always be rewritten as the inner product \(\langle \phi _K(\mathbf {y}'),\phi _K(\mathbf {y}'')\rangle \) where \(\phi _K\) is a suitable pre-processing function called a kernel embedding (Maji et al. 2008; Vedaldi and Zisserman 2010). Typical kernels include the linear, Hellinger’s, additive-\(\chi ^2\), and exponential-\(\chi ^2\) ones, given respectively by:In practice, the kernel embedding \(\phi _K\) can be computed easily only in a few cases, including the linear kernel (\(\phi _K\) is the identity) and Hellinger’s kernel (for each scalar component, \(\phi _\text {Hell.}(y) = \sqrt{y}\)). In the latter case, if y can take negative values, then the embedding is extended to the so called signed square rooting ⁵ \(\phi _\text {Hell.}(y) = {\text {sign}} (y) \sqrt{|y|}\).

Even if \(\phi _K\) is not explicitly computed, any kernel can be used to learn a classifier such as an SVM (kernel trick). In this case, \(L^2\) normalizing the kernel embedding \(\phi _K(\mathbf {y})\) amounts to normalizing the kernel asAll the pooling encoders discussed above are usually followed by post-processing. In particular, the Improved Fisher Vector (IFV) (Perronnin et al. 2010) prescribes the use of the signed-square root embedding followed by \(L^2\) normalization. VLAD has several standard variants that differ in the post-processing; here we use the one that \(L^2\) normalizes the individual VLAD subvectors (one for each visual word) before \(L^2\) normalizing the whole vector (Arandjelovic and Zisserman 2012).

The experiments are centered around two types of local descriptors. The first type are SIFT descriptors extracted densely from the image (denoted DSIFT). SIFT descriptors are sampled with a step of two pixels and the support of the descriptor is scaled such that a SIFT spatial bin has size \(8\times 8\) pixels. Since there are \(4 \times 4\) spatial bins, the support or “receptive field” of each DSIFT descriptor is \(40\times 40\) pixels, (including a border of half a bin due to bilinear interpolation). Descriptors are 128-dimensional (Lowe 1999), but their dimensionality is further reduced to 80 using PCA, in all experiments. Besides improving the classification accuracy, this significantly reduces the size of the Fisher Vector and VLAD encodings.

The second type of local image descriptors are deep convolutional features (denoted CNN) extracted from the convolutional layers of CNNs pre-trained on ImageNet ILSVRC data. Most experiments build on the VGG-M model of Chatfield et al. (2014) as this network performs better than standard networks such as the Caffe reference model (Jia 2013) and AlexNet (Krizhevsky et al. 2012) while having a similar computational cost. The VGG-M convolutional features are extracted as the output of the last convolutional layer, directly from the linear filters excluding ReLU and max pooling, which yields a field of 512-dimensional descriptor vectors. In addition to VGG-M, experiments consider the recent VGG-VD (very deep with 19 layers) model of Simonyan and Zisserman (2014). The receptive field of CNN descriptors is much larger compared to SIFT: \(139\times 139\) pixels for VGG-M and \(252 \times 252\) for VGG-VD.

When combined with a pooling encoder, local descriptors are extracted at multiple scales, obtained by rescaling the image by factors \(2^s, s=-3,-2.5,\dots ,1.5\) (but, for efficiency, discarding scales that would make the image larger than \(1024^2\) pixels).

The dimensionality of the final representation strongly depends on the encoder type and parameters. For K visual words, BoVW and LLC have K dimensions, VLAD has KD and FV 2KD, where D is the dimension of the local descriptors. For the FC encoder, the dimensionality is fixed by the CNN architecture; here the representation is extracted from the penultimate FC layer (before the final classification layer) of the CNNs and happens to have 4096 dimensions for all the CNNs considered. In practice, dimensions vary widely, with BoVW, LLC, and FC having a comparable dimensionality, and VLAD and FV a much higher one. For example, FV-CNN has \(\simeq 64 \cdot 10^3\) dimensions with \(K=64\) Gaussian mixture components, versus the 4096 of FC, BoVW, and LLC (when used with \(K=4096\) visual words). In practice, however, dimensions are hardly comparable as VLAD and FV vectors are usually highly compressible (Parkhi et al. 2014). We verified that by using PCA to reduce FV to 4096 dimensions and observing only a marginal reduction in classification performance in the PASCAL VOC object recognition task, as described below.

Unless otherwise specified, learning uses a standard non-linear SVM solver. Initially, cross-validation was used to select the parameter C of the SVM in the range \(\{0.1, 1, 10, 100\}\); however, after noting that performance was nearly identical in this range (probably due to the data normalization), C was simply set to the constant 1. Instead, it was found that recalibrating the SVM scores for each class improves classification accuracy (but of course not mAP). Recalibration is obtained by changing the SVM bias and rescaling the SVM weight vector in such a way that the median scores of the negative and positive training samples for each class are mapped respectively to the values \(-1\) and 1.

All the experiments in the paper use the VLFeat library (Vedaldi and Fulkerson 2010) for the computation of SIFT features and the pooling embedding (BoVW, VLAD, FV). The MatConvNet (Vedaldi and Lenc 2014) library is used instead for all the experiments involving CNNs. Further details specific to the setup of each experiment are given below as needed.

The evaluation is performed on a diversity of tasks: the new describable attribute and material recognition benchmarks in DTD and OpenSurfaces, existing ones in FMD and KTH-T2b, object recognition in PASCAL VOC 2007, and scene recognition in MIT Indoor. All experiments follow standard evaluation protocols for each dataset, as detailed below.

DTD (Sect. 2) contains 47 texture classes, one per visual attribute, containing 120 images each. Images are equally spilt into train, test and validation, and include experiments on the prediction of “key attributes” as well as “joint attributes”, as as defined in Sect. 2.1, and reports accuracy averaged over the 10 default splits provided with the datasets.

OpenSurfaces (Bell et al. 2013) is used in the setup described in Sect. 3 and contains 25,357 images, out of which we selected 10,422 images, spanning across 21 categories. When segments are provided, the dataset is referred to as OS\(+\)R, and recognition accuracy is reported on a per-segment basis. We also annotated the segments with the attributes from DTD, and called this subset OSA (and OSA\(+\)R for the setup when segments are provided). For the recognition task on OSA\(+\)R we report mean average precision, as this is a multi-label dataset.

FMD (Sharan et al. 2009) consists of 1,000 images with 100 for each of ten material categories. The standard evaluation protocol of Sharan et al. (2009) uses 50 images per class for training and the remaining 50 for testing, and reports classification accuracy averaged over 14 splits. KTH-T2b [65] contains 4,752 images, grouped into 11 material categories. For each material category, images of four samples were captured under various conditions, resulting in 108 images per sample. Following the standard procedure (Caputo et al. 2005; Timofte and Van Gool 2012), images of one material sample are used to train the model, and the other three samples for evaluating it, resulting in four possible splits of the data, for which average per-class classification accuracy is reported. MIT Indoor Scenes (Quattoni and Torralba 2009) contains 6,700 images divided in 67 scene categories. There is one split of the data into train (80 %) and test (20 %), provided with the dataset, and the evaluation metric is average per-class classification accuracy. PASCAL VOC 2007 (Everingham et al. 2007) contains 9963 images split across 20 object categories. The dataset provides a standard split in training, validation and test data. Performance is reported in term of mean average precision (mAP) computed using the TRECVID 11-point interpolation scheme (Everingham et al. 2007).⁶

The goal of this section is to establish which local image descriptors work best in a texture representation. The question is relevant because: (i) while SIFT is the de-facto standard handcrafted-feature in object and scene recognition, most authors use specialized descriptors for texture recognition and (ii) learned convolutional features in CNNs have not yet been compared when used as local descriptors (instead, they have been compared to classical image representations when used in combination with their FC layers).

The experiments are carried on the the task of recognizing describable texture attributes in DTD (Sect. 2) using the BoVW encoder. As a byproduct, the experiments determine the relative difficulty of recognizing the different 47 perceptual attributes in DTD.

6.1.3.1 Experimental Setup The following local image descriptors are compared: the linear filter banks of Leung and Malik (LM) (Leung and Malik 2001) (48D descriptors) and MR8 (8D descriptors) (Varma and Zisserman 2005; Geusebroek et al. 2003), the \(3 \times 3\) and \(7\times 7\) raw image patches of Varma and Zisserman (2003) (respectively 9D and 49D), the local binary patterns (LBP) of Ojala et al. (2002) (58D), SIFT (128D), and CNN features extracted from VGG-M and VGG-VD (512D).

After the BoVW representation is extracted, it is used to train a 1-vs-all SVM using the different kernels discussed in Sect. 4.2.3: linear, Hellinger, additive-\(\chi ^2\), and exponential-\(\chi ^2\). Kernels are normalized as described before. The exponential-\(\chi ^2\) kernel requires choosing the parameter \(\lambda \); this is set as the reciprocal of the mean of the \(\chi ^2\) distance matrix of the training BoVW vectors. Before computing the exponential-\(\chi ^2\) kernel, furthermore, BoVW vectors are \(L^1\) normalized. An important parameter in BoVW is the number of visual words selected. K was varied in the range of 256, 512, 1024, 2048, 4096 and performance evaluated on a validation set. Regardless of the local feature and embedding, performance was found to increase with K and to saturate around \(K=4096\) (although the relative benefit of increasing K was larger for features such as SIFT and CNNs). Therefore K was set to this value in all experiments.

6.1.3.2 Analysis Table 2 reports the classification accuracy for 47 1-vs-all SVM attribute classifiers, computed as (1). As often found in the literature, the best kernel was found to be exponential-\(\chi ^2\), followed by additive-\(\chi ^2\), Hellinger’s, and linear kernels. Among the hand-crafted descriptors, dense SIFT significantly outperforms the best specialized texture descriptor on the DTD data (52.3 % for BoVW-exp-\(\chi ^2\)-SIFT vs 44 % for BoVW-exp-\(\chi ^2\)-LM). CNN local descriptors handily outperform handcrafted features by a 10–15 % recognition accuracy margin. It is also interesting to note that the choice of kernel function has a much stronger effect for image patches and linear filters (e.g. accuracy nearly doubles moving from BoVW-linear-patches to BoVW-exp-\(\chi ^2\)-patches) and an almost negligible effect for the much stronger CNN features.

Figure 5 reports the classification accuracy for each attribute in DTD for the BoVW-SIFT, BoVW-VGG-M, and BoVW-VGG-VD descriptors and the additive-\(\chi ^2\) kernel. As it may be expected, concepts such as chequered, waffled, knitted, paisley achieve nearly perfect classification, while others such as blotchy, smeared or stained are far harder.

The previous section established the primacy of SIFT and CNN local image descriptors on alternatives. The goal of this section is to determine which pooling encoders (Sect. 4.2) work best with these descriptors, comparing the orderless BoVW, LLC, VLAD, FV encoders and the order-sensitive FC encoder. The latter, in particular, reproduces the CNN transfer learning setting commonly found in the literature where CNN features are extracted in correspondence to the FC layers of a network.

6.1.4.1 Experimental Setup The experimental setup is similar to the previous experiment: the same SIFT and CNN VGG-M descriptors are used; BoVW is used in combination with the Hellinger kernel (the exponential variant is slightly better, but much more expensive); the same \(K=4096\) codebook size is used with LLC. VLAD and FV use a much smaller codebook as these representations multiply the dimensionality of the descriptors (Sect. 6.1.1). Since SIFT and CNN features are respectively 128 and 512-dimensional, K is set to 256 and 64 respectively. The impact of varying the number of visual words in the FV representation is further analyzed in Sect. 6.1.5.

Before pooling local descriptors with a FV, these are usually de-correlated by using PCA whitening. Here PCA is applied to SIFT, additionally reducing its dimension to 80, as this was empirically shown to improve recognition performance. The effect of PCA-reduction to the convolutional features is studied in Sect. 6.1.7. The improved version of the FV is used in all the experiments (Sect. 3), and, similarly, for VLAD, we applied signed square root to the resulting encoding, which is then normalized component-wise (Sect. 4.2.3).

6.1.4.2 Analysis Results are reported in Table 3. In term of orderless encoders, BoVW and LLC result in similar performance for SIFT, while the difference is slightly larger and in favor of LLC for CNN features. Note that BoVW is used with the Hellinger kernel, which contributes to reducing the gap between BoVW and LLC. IFV and VLAD significantly outperform BoVW and LLC in almost all tasks; FV is definitely better than VALD with SIFT features and about the same with CNN features. CNN features maintain a healthy lead on SIFT features regardless of the encoder used. Importantly, VLAD and FV (and to some extent BoVW and LLC) perform either substantially better or as well as the original FC encoders. Some of these observations can are confirmed by other experiments such as Table 4.

Next, we compare using CNN features with an orderless encoder (FV-CNN) as opposed to the standard FC layer (FC-CNN). As seen in Tables 3 and 4, in PASCAL VOC and MIT Indoor the FC-CNN descriptor performs very well but in line with previous results for this class of methods (Chatfield et al. 2014). FV-CNN performs similarly to FC-CNN in PASCAL VOC, KTH-T2b and FMD, but substantially better for DTD, OS\(+\)R, and MIT Indoor (e.g. for the latter \(+5\,\%\) for VGG-M and \(+13\,\%\) for VGG-VD).

As a sanity check, results are within 1 % of the ones reported in Chatfield et al. (2011) and Chatfield et al. (2014) for matching experiments on FV-SIFT and FC-VGG-M. The differences in case of SIFT LLC and BoVW are easily explained by the fact that, differently from Chatfield et al. (2011), our present experiments do not use SPP and image augmentation.

This section conducts additional experiments on CNN local descriptors to find the best variants.

6.1.5.1 Experimental Setup The same setup of the previous section is used. We compare the performance of FC-CNN and FV-CNN local descriptors obtained from VGG-M, VGG-VD as well as the simpler AlexNet (Krizhevsky et al. 2012) CNN which is widely adopted in the literature.

6.1.5.2 Analysis Results are presented in detail in Table 4. Within that table, the analysis here focuses mainly on texture and material datasets, but conclusions are similar for the other datasets. In general, VGG-M is better than AlexNet and VGG-VD is substantially better than VGG-M (e.g. on FMD, FC-AlexNet obtains 64.8 %, FC-VGG-M obtains 70.3 % (\(+\)5.5 %), FC-VGG-VD obtains 77.4 % (\(+\)7.1 %)). However, switching from FC to FV pooling improves the performance more than switching to a better CNN (e.g. on DTD going from FC-VGG-M to FC-VGG-VD yields a 7.1 % improvement, while going from FC-VGG-M to FV-VGG-M yields a 11.3 % improvement). Combining FV-CNN and FC-CNN (by stacking the corresponding image representations) improves the accuracy by 1–2 % for VGG-VD, and up to 3–5 % for VGG-M. There is no significant benefit from adding FV-SIFT as well, as the improvement is at most 1 %, and in some cases (MIT, FMD) it degrades the performance.

Next, we analyze in detail the effect of depth on the convolutional features. Figure 6 reports the accuracy of VGG-M and VGG-VD on several datasets for features extracted at increasing depths. The pooling method is fixed to FV and the number of Gaussian centers K is set such that the overall dimensionality of the descriptor \(2KD_k\) is constant. For both VGG-M and VGG-VD, the improvement with increasing depth is substantial and the best performance is obtained by the deepest features (up to 32 % absolute accuracy improvement in VGG-M and up to 48 % in VGG-VD). Performance increases at a faster rate up to the third convolutional layer (conv3) and then the rate tapers off somewhat. The performance of the earlier levels in VGG-VD is much worse than the corresponding layers in VGG-M. In fact, the performance of VGG-VD matches the performance of the deepest (fifth) layer in VGG-M in correspondence of conv5_1, which has depth 13.

Finally, we look at the effect of the number of Gaussian components (visual words) in the FV-CNN representation, testing possible values in the range 1 to 128 in small (1-16) increments. Results are presented in Fig. 7. While there is a substantial improvement in moving from one Gaussian component to about 64 (up to \(+\)15 % on DTD and up to 6 % on OS), there is little if any advantage at increasing the number of components further.

In the previous section, we have seen that switching from FC to FV pooling may have a substantial impact in certain problems. We could find three reasons that can explain this difference.

The second reason is that FV pooling may reduce overfitting in domain transfer. Pre-trained FC layers could be too specialized for the source domain (e.g. ImageNet ILSVRC) and there may not be enough training data in the target domain to retrain them properly. On the contrary, a linear classifier built on FV pooling is less prone to overfitting as it encodes a simpler, smoother classification function than a sequence of FC layers in a CNN. This is further investigated in Sect. 6.2.3.

In order to investigate this hypothesis, we evaluated FV-CNN by pooling CNN descriptors at a single scale instead of multiple ones, for both VGG-M and VGG-VD models. For datasets like FMD, DTD and MIT Indoor, FV-CNN at a single scale still generally outperforms FC-CNN (columns FC (SS) and FV (SS) in Table 5), by up to 5.6 % for VGG-M, and by up to 9.1 % for VGG-VD; however, the difference is less marked as using a single scale in FV-CNN looses up to 3.8 % accuracy points and and in some cases the representations is overtaken by FC-CNN.

The complementary experiment, namely using multiple scales in FC pooling, is less obvious as, by construction, FC-CNN resizes the input image to a fixed resolution. However, we can relax this restriction by computing multiple FC representations in a sliding-window manner (also know as a “fully-convolutional” network). Then individual representations computed at multiple locations and, after resizing the image, at multiple scales can be averaged in a single representation vector. We refer to this as multi-scale FC pooling. Multi-scale FC codes perform slightly better than single-scale FC in most (but not all) cases; however, the benefit of using multiple scales is not as large as for multi-scale FV pooling, which is still significantly better than multi-scale FC.

This section explores the effect of applying dimensionality reduction to the CNN local descriptors before FV pooling.

This experiment investigates the effect of two parameters, the number of Gaussians in the mixture model used by the FV encoder, and the dimensionality of the convolutional features, which we reduce using PCA. Various local descriptor dimensions are evaluated, from 512 (no PCA) to 32, reporting mAP on PASCAL VOC 2007, as a function of the pooled descriptor dimension. The latter is equal to 2KD, where K is the number of Gaussian centers, and D the dimensionality of the local descriptor after PCA reduction.

Results are presented in Fig. 8 for VGG-M and VGG-VD. It can be noted that, for similar values of the total representation dimensionality 2KD, the performance of PCA-reduced descriptors is a little better than not using PCA, provided that this is compensated by a large number of GMM components. In particular, similar to what was observed for SIFT in Perronnin et al. (2010), using PCA does improve the performance by 1–2 % mAP point; furthermore, reducing descriptors to 64 or 80 dimensions appears to result in the best performance.

In this experiment we are interested in understanding which GMM components in the FV-CNN representation code for a particular concept, as well as in determining which areas of the input image contribute the most to the classification score.

In order to do so, let \(\mathbf {w}\) be the weight vector learned by a SVM classifier for a target class using the FB-CNN representation as input. We partition \(\mathbf {w}\) in subvectors \(\mathbf {w}_k\), one for each GMM component k, and rank components by decreasing value \(\Vert \mathbf {w}_k\Vert \), matching the intuition that the GMM component that is predictive of the target class will result in larger weights. Having identified the top components for a target concept, the CNN local descriptors are then extracted from a test image, the descriptors that are assigned to a top component are selected, and their location is marked on the image. To simplify the visualization, features are extracted at a single scale.

As can be noted in Fig. 9 for some indicative texture types in DTD, the strongest GMM components do tend to fire in correspondence to the characteristic features of each texture. Hence, we conclude that GMM components, while trained in an unsupervised manner, contain clusters that can consistently localize features that capture distinctive characteristics of different texture types.

Experiments on textures are divided in recognition in controlled conditions (Sect. 6.2.1.3), where the main sources of variability are viewpoint and illumination, recognition in the wild (Sect. 6.2.1.4), characterized by larger intra-class variations, and recognition in the wild and clutter (Sect. 6.2.1.5), where textures are a small portion of a larger scene.

6.2.1.1 Datasets and Evaluation Measures In addition to the datasets evaluated in Sect. 6.1, DTD, OS\(+\)R, FMD and KTH-T2b, we consider here also the standard benchmarks for texture recognition. CUReT (Dana et al. 1999) (5612 images, 61 classes), UIUC (Lazebnik et al. 2005) (1000 images, 25 classes), KTH-TIPS  (Burghouts and Geusebroek 2009) (810 images, 10 classes) are collected in controlled conditions, by photographing the same instance of a material, under varying scale, viewing angle and illumination. UMD (Xu et al. 2009) consists of 1000 images, spread across 25 classes, but was collected in uncontrolled conditions. For these datasets, we follow the standard evaluation procedures, that is, we are using half of the images for training, and the remaining half for testing, and we are reporting accuracy, averaged over 10 random splits. The ALOT dataset (Burghouts and Geusebroek 2009) is similar to the existing texture datasets, but significantly larger, having 250 categories. For our experiments we used the protocol of Sulc and Matas (2014), using 20 images per class for training and the rest for testing.

This paragraph evaluates texture representations on datasets which are collected under controlled condition (Table 4, Sect. a).

For instance recognition, CUReT, UIMD, UIUC are saturated by modern techniques such as Sifre and Mallat (2013); Sharma et al. 2012) and Sulc and Matas (2014), with accuracies above \(\ge 99\,\%\). There is little difference between methods, and FV-SIFT, FV-CNN, and FC-CNN behave similarly. KT is also saturated, although FC-CNN looses about (3 %) accuracy compared to FV-CNN.

In material recognition, KTH-T2b and ALOT offer a somewhat more interesting challenge. First, there is a significant difference between FC-CNN and FV-CNN (3–6 % absolute difference in KTH-T2b and 8–10 % in ALOT), consistent across all CNN evaluated. Second, CNN descriptors are significantly better than SIFT on KTH-T2b and ALOT with absolute accuracy gains of up to 11 %.

Compared to the state of the art, FV-SIFT is generally very competitive. In KTH-T2b, FV-SIFT outperforms all recent methods (Cimpoi et al. 2014) with the exception of Sulc and Matas (2014) which is based on a variant of LBP. The latter is very strong in ALOT too, but in this case FV-SIFT is virtually as good. In the case of KTH-T2b, (Sulc and Matas 2014) is better than most of the deep descriptors as well, but it is still significantly bested by FV-VGG-VD (\(+\)5.5 %). Nevertheless, this is an example in which a specialized texture descriptor can be competitive with deep features, although of course deep features apply unchanged to several other problems.

On ALOT, FV-CNN with VGG-VD is on par with the result obtained by Badri et al. (2014)—98.45 %—but their model was trained with 30 images per class instead of 20. The same paper reports even better results, but when training with 50 images per class or by integrating additional synthetic training data.

Texture recognition in the wild is more comparable, in term of the type of intra-class variations, to object recognition than to texture recognition in controlled conditions. Hence, one can expect larger gains in moving from texture-specific descriptors to general-purpose descriptors. This is confirmed by the results. SIFT is competitive with AlexNet and VGG-M features in FMD (within 3 % accuracy), but it is significantly worse in DTD (\(+\)4.3 % for FV-AlexNet and \(+\)8.2 % for FV-VGG-M). FV-CNN is a little better than FC-CNN (\(\sim \)3 %) on FMD and substantially better in DTD (\(\sim \)8 %). Different CNN architectures exhibit very different performance; moving from AlexNet to VGG-VD, the accuracy absolute improvement is more than 11 % across the board.

Compared to the state of the art, FV-SIFT is generally very competitive, outperforming the specialized texture descriptors developed by Qi et al. (2014) and Sharan et al. (2013) in FMD (and this without using ground-truth texture segmentations as used by Sharan et al. (2013)). Yet FV-VGG-VD is significantly better than all these descriptors (\(+\)24.7 %).

In term of complementarity of the features, the combination of FC-CNN and FV-CNN improves performance by about 3 % across the board, but including FV-SIFT (labelled FV-SIFT/FC\(+\)FV-VD in the table) as well does not seem to improve performance further. This is in contrast with the fact that SIFT was found to be fairly complementary to FC-CNN on a variant of AlexNet in Cimpoi et al. (2014).

This section evaluates texture representations on recognizing texture materials and describable attributes in clutter. Since there is no standard benchmark for this setting, we introduce here the first analysis of this kind using the the OS\(+\)R and OSA\(+\)R datasets of Sect. 3.1. Recall that the \(+\)R suffix indicates that, while textures are imaged in clutter, the classifier is given the ground-truth region segmentation; therefore, the goal of this experiment is to evaluate the effect of realistic viewing conditions on texture recognition, but the problem of segmenting the textures is evaluated later, in Sect. 7.3.

Results are reported in Table 4 in sections b and c. As before, performance improves with the depth of CNNs. For example, in material recognition (OS\(+\)R) accuracy starts at about \(39.1\,\%\) for FV-SIFT, is about the same for FC-VGG-M (\(41.3\,\%\)) and a little better for FC-VGG-VD (\(43.4\,\%\)). However, the benefit of switching from FC encoding to FV encoding is now even more dramatic. For example, on OS\(+\)R FV-VGG-M has accuracy \(52.5\,\%\) (\(+11.2\,\%\)) while FV-VGG-VD \(59.5\,\%\) (\(+16.1\,\%\)). This clearly demonstrates the advantage of orderless pooling of CNN local descriptors on FC pooling when regions of different sizes and shapes must be evaluated. There is also a significant computational advantage (evaluated further in Sect. 6.2.3) if, as it is typical, several regions must be classified: in that case, CNN features need not to be recomputed for each region. Results on OSA\(+\)R are entirely analogous.

This section evaluates texture descriptors on tasks other than texture recognition, namely coarse and fine-grained object categorization, scene recognition, and semantic region recognition.

6.2.2.1 Datasets and Evaluation Measures In addition to the datasets seen before, here we experiment with fine grained recognition in the CUB (Wah et al. 2011) dataset. This dataset contains 11788 images, representing 200 species of birds. The images are split approximately into half for training and half for testing, according to the list that accompanies the dataset. Image representations are either applied to the whole image (denoted CUB) or on the region counting the target bird using ground-truth bounding boxes (CUB\(+\)R). Performance in CUB and CUB\(+\)R is reported as per-image classification accuracy. For this dataset, the local descriptors are again extracted at multiple scales, but now only for the smaller range \(\{0.5, 0.75, 1\}\) which was found to work better for this task.

Performance is also evaluated on the MSRC dataset, designed to benchmark semantic segmentation algorithms. The dataset contains 591 images, for which some pixels are labelled with one of the 23 classes. In order to be consistent with the results reported in the literature, performance is reported in term of per-pixel classification accuracy, similar to the measure used for the OS task as defined in Sect. 3.1. However, this measure is further modified such that it is not normalized per class: 6.2.2.2 Analysis Results are reported in Table 4 section d. On PASCAL VOC, MIT Indoor, CUB, and CUB\(+\)R the relative performance of the different descriptors is similar to what has been observed above for textures. Compared to the state-of-the-art results in each dataset, FC-CNN and particularly the FV-CNN descriptors are very competitive. The best result obtained in PASCAL VOC is comparable to the current state-of-the-art set by the deep learning method of Wei et al. (2014) (85.2 vs \(84.9\,\%\) mAP), but using a much more straightforward pipeline. In MIT Places the best performance is also substantially superior (\(+10\,\%\)) to the current state-of-the-art using deep convolutional networks learned on the MIT Place dataset (Zhou et al. 2014) (this is discussed further below). In the CUB dataset, the best performance is short (\(\sim \)6 %) of the state-of-the-art results of Zhang et al. (2014). However, Zhang et al. (2014) uses a category-specific part detector and corresponding part descriptor as well as a CNN fine-tuned on the CUB data; by contrast, FV-CNN and FC-CNN are used here as global image descriptors which, furthermore, are the same for all the datasets considered. Compared to the results of Zhang et al. (2014) without part-based descriptors (but still using a part-based object detector), the best of our global image descriptors perform substantially better (62.1 vs \(67.3\,\%\)).

Results on MSRC\(+\)R for semantic segmentation are entirely analogous; it is worth noting that, although ground-truth segments are used in this experiment and hence this number is not comparable with other reported in the literature, the best model achieves an outstanding \(99.1\,\%\) per-pixel classification rate in this dataset.

This section investigates in more detail the problem of domain transfer in CNN-based features. So far, the same underlying CNN features, trained on the ImageNet’s ILSVCR data, were used in all cases. To investigate the effect of the source domain on performance, this section consider, in addition to these networks, new ones trained on the PLACES dataset (Zhou et al. 2014) to recognize scenes on a dataset of about 2.5 million labeled images. Zhou et al. (2014) showed that, applied to the task of scene recognition in MIT Indoor, these features outperform similar ones trained on ILSVCR [denoted CAFFE (Jia 2013) below]—a fact explained by the similarity of domains. We repeat this experiment using FC- and FV-CNN descriptors on top of VGG-M, VGG-VD, PLACES, and CAFFE.

Results are shown in Table 6. The FC-CNN performance is in line with those reported in Zhou et al. (2014)—in scene recognition with FC-CNN the same CNN architecture performs better if trained on the Places dataset instead of the ImageNet data (58.6 vs \(65.0\,\%\) accuracy⁷). Nevertheless, stronger CNN architectures such as VGG-M and VGG-VD can approach and outperform PLACES even if trained on ImageNet data (65.0 vs \(62.5/67.6\,\%\)).

However, when it comes to using the filter banks with FV-CNN, conclusions are very different. First, FV-CNN outperforms FC-CNN in all cases, with substantial gains up to \(\sim \)11–12 % in correspondence of a domain transfer from ImageNet to MIT Indoor. The gap between FC-CNN and FV-CNN is the highest for VGG-VD models (67.6 vs 81.0 %, nearly 14 % difference), a trend also exhibited by other datasets as seen in Table 4. Second, the advantage of using domain-specific CNNs disappears. In fact, the same CAFFE model that is 6.4 % worse than PLACES with FC-CNN, is actually 2.1 % better when used in FV-CNN. The conclusion is that FV-CNN appears to be immune, or at least substantially less sensitive, to domain shifts.

Our explanation of this phenomenon is that the convolutional features are substantially less committed to a specific dataset than the fully connected layers. Hence, by using those, FV-CNN tends to be a lot more general than FC-CNN. A second explanation is that PLACES CNN may learn filters that tend to capture the overall spatial structure of the image, whereas CNNs trained on ImageNet tend to focus on localized attributes which may work well with orderless pooling.

Finally, we compare FV-CNN to alternative CNN pooling techniques in the literature. The closest method is the one of Gong et al. (2014), which uses a similar underlying CNN to extract local image descriptors and VLAD instead of FV for pooling. Notably, however, FV-CNN results on MIT Indoor are markedly better than theirs for both VGG-M and VGG-VD (68.8 vs 74.2  %/ 81.0 % resp.) and marginally better (69.7 %—Tables 4 and 6) when the same CAFFE CNN is used. Also, when using VLAD instead of FV for pooling the convolutional layer descriptors, the performance of our method is still better (68.8 % vs 71.2 %), as seen in Table 3. The key difference is that FV-CNN pools convolutional features, whereas (Gong et al. 2014) pools fully connected descriptors extracted from square image patches. Thus, even without spatial information as used by Gong et al. (2014), FV-CNN is not only substantially faster—8.5\(\times \) speedup when using the same network and three scales, but at least as accurate.