Person re-identification with mutual re-ranking

With the popularity of surveillance cameras, security observation systems are applied ubiquitously, especially in public places such as supermarkets, airports, and hospitals. Such a system includes multiple cameras connected to an operation center. Operators who are displayed images recorded from cameras have to observe and perform various tasks: detecting, recognizing, and keeping track of characters. Among these tasks, tracking people crossing multiple cameras plays an important role.

This task becomes much more challenging when the number of cameras increases and there are more people appearing in camera’s view. Automatic systems which can automatically recognize people across multiple cameras are needed. The essential problem of such system has recently been studied and named person re-identification. In other words, it is defined as the problem of matching human images recorded from multiple cameras distributed over non-overlapped areas, in the context of the persons crossing through many cameras.

Formally, the person re-identification problem can be formulated as follows: Given n persons crossing camera 1, some of them appear later in camera 2. For each image (or person) recorded from the camera 2 (called probe image), determine a list of images (or persons) recorded from camera 1 (called gallery images). Gallery images in the list are ranked by their likelihood of being the same person of the currently considered as probe image (see Fig. 1).

A person re-identification system receives images or videos from multiple cameras as input and output the matching of images of people appearing in those images or videos [1].

Due to the low resolution of surveillance cameras, traditional recognition methods such as biometric cues or face and iris recognition could not be applied. In addition, variation of viewpoints and illumination across different cameras, which cause appearance changes, is among the most challenging problems leading to mismatching. Other challenging issues relating to person re-identification are occlusion and background clutter.

A typical person re-identification pipeline consists of two components: feature extraction and image matching. State-of-the-art methods usually employ multiple features. SDALF [6] integrates three kinds of low-level features: weighted color histogram, maximally stable color regions (MSCR), and recurrent high-structured patches (RHSP). Meanwhile, semantic features are used in [13] together with other low-level features. Another approach is to learn appropriate metrics for specific data [25].

With existing person re-identification systems, probe images of individual persons are treated independently. Given a probe image, they compute the distances from gallery images to the probe image. Using the computed distances, a ranked list of gallery images is then generated.

However, in reality, there are cases in which multiple persons appear concurrently in a camera. Human beings could assess and utilize such information to give a more accurate prediction. Namely, if a gallery image of one person is ranked very high in a list, that image should be ranked very low in other lists, given the ranked lists of different probe images. In this paper, we propose a method using such constraint in post-processing to improve identification accuracy. Information of co-occurrence persons is employed to mutually re-rank the returned lists. Specifically, each highly ranked gallery image in a list is assigned a penalty. The penalties are then used to update scores of the gallery images in other lists. We compute the penalties based on similarities of gallery images to probe images.

Compared to existing work [23], we provide two main extensions:The remainder of this paper is organized as follows: Sect. 2 is an overview of related works. Section 3 presents our proposed re-ranking method. Experimental results are shown in Sect. 4. Finally, Sect. 5 is the conclusion of the paper.

There are two main parts in a typical person re-identification system: feature extraction and similarity estimation. Re-ranking is usually applied in the post-processing process to improve identification accuracy.

Low-level features are widely used in feature extraction. In [6], weighted histogram and blobs are used. Specifically, color histograms are computed with weights. The weights are based on distances of pixels to the asymmetric axis of the person image. Besides, the authors extracted blobs using the method called maximally stable colour regions (MSCR) in [7]. In [24], the authors detected blobs on person images. Then, they extracted color histogram and histogram of oriented gradient (HOG) for visual features. Ma et al. [21] introduced a new feature, called biologically inspired feature (BIF). It is extracted by convolving images with Gabor filters. Then, MAX pooling is applied for two convolved images with consecutive bands. Prosser et al. [25] and Gray et al. [11] focused on color and texture features. Specifically, 8 color channels from RGB, HS, and YCbCr color systems are used. The authors used Gabor and Schmid filters on the luminance channel for texture features. In [33], the authors used local features and proposed an unsupervised method for determining feature weight for fusion. Local descriptors of pixels are transferred into Fisher Vectors to represent images in [22]. Unlike other image retrieval problems, local features are not commonly used in person re-identification [9, 12].

Mid-level features built from low-level features are used in recent study due to their high-level abstraction and efficiency. In [32], selected discriminative and representative local patches are used for learning mid-level feature filters. In [16], the authors used a deep learning framework to learn pairs of mid-level filters which encode the transformation of mid-level appearance between the two cameras. Inspired by the recognition ability of human being, the authors in [31] proposed an unsupervised method for detecting salient and distinctive local patches and used them for matching images.

Semantic features, human understandable mid-level features, are applied in [13, 20] for person re-identification. In [13], semantic features are first detected by applying SVM with texture features (introduced in [25]). The detected mid-level features are then used for re-identification. Liu et al. proposed to use topic models [20] to represent the attributes (i.e. semantic mid-level features) of images.

Feature selection and weighting were also addressed in recent related works. In [19], the authors used an unsupervised approach to adaptively identify the confidence of features in different circumstances. Or in [11], the authors defined a feature space. Then, they proposed a learning approach to search for the optimal representation. Zhao et al. [30] focused on extracting discriminative image patches and learning human salient descriptor from them.

To accurately match person images, body part localization is required. The method, named SDALF in [6], employed a simple body part detector. The aim was to determine upper and lower parts of a person image by a horizontal line. The line is set so that the separated parts have minimum difference in area and maximum difference in color. More complicatedly, Gheissari et al. [9] used a technique, called decomposable triangulated graph, for localizing human body parts. Triangulated graphs are fitted to human images by minimizing the energy function. Given the fitted graph, body parts can be localized for matching. Or in [2], pictorial structures were applied for detecting body configuration.

Given extracted features, similarity estimation is also important to a person re-identification system. Besides traditional distances like L1 distance, L2 distance, or Bhattacharyya distance, recent works also focus on learning a new type of distance [25, 26]. In Prosser et al. [25] reformulated person re-identification as a ranking problem in which they learn a ranking function. The function ranks relevant pairs higher than irrelevant pairs. In Schwartz and Davis [26], the authors use partial least square to learn the weights of different features by considering their discrimination. The other direction is to learn the transformation between two cameras. In Zhao et al. [29], a function is defined to present the transformation from a fixed camera to another fixed camera. Zheng et al. [34] considered person re-identification as a relative distance comparison learning problem which aims at learning appropriate distances for each pair of images. Different from other works, Zhen et al. [17] proposed to simultaneously distance metrics and the decision threshold instead of the distance metrics only.

In general, re-ranking approaches for image retrieval can be applied to person re-identification. In [28], users’ intentions expressed in their feedback are used to re-rank the output lists. In [4], the authors proposed a method for query expansion by using images selected from the initial ranked list. Similarly, top images from an output ranked list are used for re-querying [3]. By doing this, more relevant images are returned. Assuming relevant images are highly similar to the nearest neighbors of a query, the authors in [27] introduced a method to accurately localize interest instances in the retrieved images. The features extracted from localized instances in top ranked images are then used to refine the retrieval results.

There are several recently proposed re-ranking methods dedicated to person re-identification. The authors in [14] claimed that true matched pairs of images are supposed to have many common visually similar neighbors, called context similarity, in addition to mutual visually similar appearance, called content similarity. They suggested reversely querying each gallery image with newly formed set including the probe image and other gallery images. Then, the initial result is revised using the bidirectional ranking lists. Inspired by [14], Garcia et al. [8] proposed to eliminate ambiguous cases in the first ranks of the lists by assuming that the ground truths appear in the first ranks of the lists as well. Unlike the above two works which utilize internal information of ranked lists for optimization, the authors in [18] designed an interactive method which allows users to pick strong and weak negative samples from the returned list. The selected negative samples will then be used to refine the list. This approach is very important in practical applications when users need acceptably accurate results.

In this paper, we introduce a method to improve person re-identification accuracy by utilizing information of the co-occurrence of people for re-ranking. To the best of our knowledge, such information has not been explicitly employed in existing person re-identification approaches.

In this section, we introduce our proposed post-processing method. Given initial ranked lists returned by a person re-identification system, our method is then used to re-rank the lists by taking co-occurrence constraints into account.

In this paper, we proposed a re-ranking method which refines person re-identification results in the context of multiple people appearing concurrently in a camera. The experimental results with different state-of-the-art person re-identification methods on different datasets showed remarkable improvement when applying our method, especially when there are more people appearing at the same time. As a post-processing procedure, our proposed method can be applied to any state-of-the-art re-identification systems to boost their performance. For more accurate re-ranking, considering reliability of ranking lists would be a promising future study.