Gait recognition for person re-identification

Across multiple cameras, person recognition and identification are important targets for many computer vision applications, especially monitoring systems [1]. The operation of recognition of a person from a set of images captured by several cameras is called person re-identification. The similarity measures can be the key to compute the matching between two or a set of images. However, the re-identification using video clips can be a problem for many applications [2], for example people tracking across multiple cameras. The video sequences captured by different cameras should be analyzed to re-identify the person and keep tracking him across all cameras in the surveilled areas.

The sequential methods that use a list of features are not efficient for the re-identification of persons due to several limitations such as differences between the analyzed objects in terms of shape, colors, scales, and others [3]. which, in turn, implies that the use of a limited number of features cannot be enough for proper identification. On the other hand, with deep learning techniques, the use of different and non-limited features for learning becomes a good alternative for solving person re-identifications problems. For that, to train these methods, a big scale of data is required from multiple camera views [4]. In addition, using preprocessing techniques the performance can help the learning model to better learning.

Owing to the differences between multiple images of the same person captured by different cameras, the re-identification can be difficult even with deep learning models. The difference can be in the shape, the color of the cloths, as well as the scale of the images. Contrarily, the gait is a feature that cannot be changed and is repetitively performed by the person; thus, it could be considered as an identification feature [5, 6]. The gait of each person can be used for re-identifying a person across different cameras in Fig. 1.

Human gait is generally observed to be a uniquely human characteristic that is difficult to replicate or hide. Hence, it represents a critical biometric identification feature for human identification [7]. Therefore, gait identification has been recognized as an essential identification technology for different applications in crime control and detection systems in high security, civilian, and public areas such as airports, stations, banks, and military bases [8, 9]. Recently, gait recognition has also found significant application in gender, age, and ethnicity prediction systems and in cyber-physical healthcare systems enabled through connected wearable devices [10‐13].

Person re-identification using human gait can is an efficient identification technique that can overcome the person re-identification problems including shape, colors, and scales. The gait analysis can be a good solution due to its unicity for each person; nonetheless, some limitations can affect the performance of such a gait recognition algorithm. One of these limitations is the view angle of each gait because there is a remarkable difference between the gait of a person. To handle the gait recognition under different view angles, we propose a multitask-based method using convolutional neural networks (CNNs) on gait energy image (GEI) features. The proposed method starts by extracting the gait energy image (GEI) of each person in a scene using the motion detection and segmentation of each person based on a proposed background subtraction method. Before recognizing the gait, the view angle is estimated using a proposed CNN model. Then, the verification of the estimated angle is performed using another proposed CNN model on GEIs. This technique aims to improve gait recognition accuracy. We train and test our proposed gait recognition method on three publicly available datasets. The proposed method achieves better results compared with some existing methods.

The rest of the paper is organized as follows: The literature overview related to our work is presented in Sect. 2. The proposed system is presented in Sect. 3. Experimental analysis is provided in Sect. 4. The conclusion and future works are given in Sect. 5.

Gait recognition involves four main steps: image acquisition, preprocessing to segment out the binarized silhouettes from the background, training and/or feature extraction from silhouettes, and, lastly, classification or recognition of gait sequences by matching the testing and training feature space. Assuming the silhouettes to be obtained from fixed cameras observing static scenes, the silhouettes can be preprocessed by simple and less computationally expensive techniques like background segmentation [14]. After the preprocessing of the acquired images, features are extracted from the foreground silhouettes and several recognition techniques are used to match the feature space of the training sequences or “gallery” and test sequences or “probe.” Since the size of the feature vector involved in this scheme becomes very large, the dimensionality of the feature vector may be reduced before classification/recognition. For this purpose, principal component analysis (PCA) and multiple discriminant analysis (MDA) are used to achieve good reduced data representation by discarding the features which show low variance.

Since human gait is a human behavior that involves movement of various parts of a human’s body, the early research on gait recognition started with attempts to model the human body as a whole. Hence, this approach was given the name of the model-based approach to gait recognition. In the model-based approach, after acquiring the images, silhouettes are obtained by noise removal and binarization of the walking person’s 2D image. From the silhouettes, specific parameters of the human model are extracted. The parameters typically pertain to various body parts—for example, lengths of body parts (typically limbs); widths of body parts like head, torso, knees, and arms; the position of head and shoulders; and trajectory defined by hip joint angles. In fact, the hip rotation pattern from the sequence of images carries essential information for gait recognition. The authors in [15] defined 22 such parameters of human body parts involved in the gait and used these to form a layered deformable model of the human body. The parameters of the human body model can be updated over time for the individual silhouettes obtained from the sequential motion frames and hence used to represent the gait. Although the model-based approach has shown the ability to obtain good gait recognition, yet, the recognition has faced some challenges. For instance, the images have many occlusions and shadows, and locating the body segments from the binarized image silhouettes is difficult. For increased recognition accuracy, modeling the human body often calls for the conversion of excellent quality 2D images to 3D computer models, which is a complicated and compute-intensive task [7]. Moreover, the quality of images captured by surveillance cameras is quite weak which adversely affects the quality of gait recognition. Therefore, the focus of further research shifted more toward the model-free approach.

Model-free, also called motion-based approach, can again be categorized into two types: sequential motion-based and spatiotemporal motion-based approaches. In sequential motion approaches, gait is represented as a time sequence of human poses, while the spatiotemporal approach represents gait by mapping the distribution of motion through space and time [14]. The sequential motion-based approach proposed in [34] involves representing the motion through temporal templates that identify where the motion has occurred and also recording the history of these motions. The spatiotemporal approaches proposed in the literature differ primarily in the preprocessing, feature extraction, and classification techniques used for the silhouette-based gait sequences. The authors in [35] proposed a feature selection mechanism called gait energy image (GEI) through which a history of gait movements is recorded in a single 2D template instead of storing it as a sequence of templates. The spatiotemporal GEI feature is obtained by averaging the pixels of the silhouette in different frames of a gait cycle. Recognition involved statistical gait feature fusion of real and synthesized (distorted) gait templates. This approach does not only preserve space but also report high recognition performance. The authors in [9] proposed gait entropy image to be used as an automatic feature selection mechanism for the gallery (ground truth) and probe (testing) images. This feature selection scheme is proven to mitigate the effects of covariate walking conditions. The proposed recognition approach, which is called the adaptive component and discriminant analysis (ACDA), is a fast recognition approach to gait recognition.

In the same context, many researchers have conducted studies and proposed different approaches that aim to handle recognition under different angles of view. The authors in [16] and [17] proposed a walking speed-invariant gait recognition method based on RBF neural networks. Other authors in [18] proposed a gait recognition method by extracting a couple of joint angles from two signature and then using a baseline 1-NN classifier to classify the gait. In the same context, Rida et al. [19] proposed a gait recognition approach based on the phase-only correlation. In [20], using convolutional neural networks, the authors proposed a gait recognition approach. For cross-view gait recognition, authors in [23] proposed an approach based on a model named video transformation model (VTM). For that same purpose, Wu et al. [21] proposed a method using the CNN model. Using tensor representation, the authors in [36] proposed an approach for cross-view gait recognition. Spatiotemporal HOG features are used also for cross-view gait recognition in [25].

Recently, many methods have been proposed for handling the angle variation exploiting the silhouette sequences [26, 28] or proposing some other features [22, 32], or using GEI images for training their deep learning models [27]. Authors in [26] proposed a gait recognition method named localized Grassmann mean representatives with partial least squares regression (LoGPLS), whereas authors in [22] proposed new features called autocorrelation feature where the image at lag time zero is similar to GEI. Another new feature is proposed in [32] and called period energy image (PEI) which is a multi-channel gait template. The proposed feature is used for gait recognition. In addition, the authors proposed a gait recognition method based on the local GEI (LGEI) feature with a self-adaptive hidden Markov model (SAHMM) [29]. Instead of using GEI or other similar features, some authors trained their models on silhouette sequences, like in [28]. In [30], the authors used the silhouette sequence to train a deep learning method based on ResNet and LSTM. In order to recognize the gaits on a very large scale of data, generative adversarial networks (GAN) are used recently in many works [22, 27, 37]. The recognition in [27] is performed on 10000 subjects, whereas in [37] two-stream GAN model is used to learn gait features. Authors in [32] have used two gait templates: GEI and PEI with a GAN model for recognizing gaits under different angles of view. In the same context, authors in [31] used RGB image sequences as an input of the proposed architecture based on autoencoder networks and LSTM. Using silhouette-based features captured by specified cameras, the authors in [33] proposed a CNN model for predicting the angle and also used it for recognizing the gait.

The obtained results by different methods for gait recognition are convincing, but there is still a chance for improved efficiency. Table 1 represents a summarization of the cited gait recognition methods . The deep learning methods using CNNs or GAN improve the performance of the recognition, but the complexity of each new dataset disables existing models to handle the new challenges like the variation of the angle of view, such as the dataset in [38] representing 14 angles of view where [39] represented just 11. So for each new dataset, the existing method cannot be suitable to recognize the gait on it.

Gait recognition technology is an efficient technique to re-identify a person during a pass across multiple cameras. This is due to that gait represents an effective measure for person identification from distance owing that it is a unique characteristic of each person, and unlike the other recognition measures, gait cannot be different in multiple images of the same person as the case for shape, clothes colors, and scale that might vary from an image to another. However, the change in view angles during visual surveillance scenes is a common challenge that often faces the gait recognition. Therefore, in this work, we attempt to deal with this challenge. First, each person’s silhouette, after the background subtraction-based methods for motion detection, extraction method is proposed.

After the extraction of the binary sequence of each target, the GEI of each sequence is extracted as illustrated in Fig. 2. To recognize the person’s gait, the proposed approach combines the estimation of the view angle before starting the recognition using a multitask CNN model. As illustrated in Fig. 2, the view angle falls within a range of [0–270] similar to the example presented in Fig. 3 obtained from the OU-MVLP dataset [38]. Here, the angle is estimated from the GEI image using a CNN model. Then, the recognition is performed using the second CNN model. A detailed description of each step is represented in this section. For the training, a set of datasets is used including CASIA-(B) [24], OU-ISIR [39], and OU-MVLP [38] datasets.

In this work, we present a gait recognition method for person re-identification. The proposed method contains two phases: first the detection of moving objects based on background subtraction and then the gait recognition which is performed by estimating the angle of view and then recognizing the gait. The proposed method for background modeling has been tested on SBI dataset and then compared with some existing methods. Our approach involves the recognition of gait images based on viewing angle estimation of the GEIs of input probe images. GEI is the temporal average of the gait silhouettes in a gait cycle. Angle estimation is performed using our proposed CNN-based angle learning model. From the estimated angle, we recognize the gait of the subject by using the CNN classifier on the gallery GEIs for the corresponding angle. We evaluate our proposed method using three publicly available datasets, namely CASIA-(B) gait database; OU-ISIR large population gait database, and OU-MVLP multi-view large population database.

For the OU-ISIR and CASIA-(B) datasets, performance of our method was compared against several benchmark methods including [21, 23, 31‐33, 36].

Since our method involves gait angle estimation prior to gait recognition, we needed datasets with gait images acquired from different viewing angles to evaluate the performance of our approach. We chose the databases of cross-view gait images because all these three datasets provide gait images of subjects from different viewing angles. Below, the gait datasets used to test our method are briefly described.

Person re-identification is a challenging task for computer vision applications due to the variation of the appearances of the same person from different camera views. Cross-view of the gait is also posing a problem because different capturing angles limit the recognition of the gait. This paper presents a discriminant method to overcome this problem. Multitask method for gait recognition starts with the detection of people using the background subtraction method and then extraction of the GEIs for each person. After that, the proposed CNN-based model is used to estimate the angle before recognizing the gait. Experimental results exploited CASIA-(B), OU-ISIR, and OU-MVLP gait datasets which demonstrate that our multitask method is effective and on average more robust than other state-of-the-art methods.