Gaitdlf: global and local fusion for skeleton-based gait recognition in the wild

Appearance-based gait recognition methods focus on directly feeding appearance video sequences into the network for gait recognition. This method also performs very well in low resolution conditions, so more and more researchers are studying this method [30, 31]. The research of appearance-based methods has mainly focused on time series modeling and spatial feature extraction due to the rapid development in the fields of video understanding and deep learning.

Appearance-based gait recognition methods capitalize on the processing of video sequences to identify individuals based on their gait. Researchers have paid considerable attention to these methods, as they proved robust under low-resolution conditions [30, 31]. Advances in video analysis and deep learning have spurred focused investigations into spatial feature extraction and temporal modeling within this domain.

Among other things, GaitSet [17] believes that given a collection of gait sequences of one cycle, it is possible to discriminate between different people. They processed each frame of the gait sequence using statistical functions and viewed it as a set of frames. This simple but effective method is known as one of the important contributions to the research of gait recognition this year. GaitPart [18] considers the dependency between local information of input contours and short-range time, and it designs a micro-action pattern builder, which investigates the relationship between the local detail information of the input contours and the short-range time and combines it with different window sizes to extract the complex local spatial-temporal features. In contrast, GaitGL [19] achieves global and local feature extraction by using multiple convolutional layers. The approach is able to capture subtle local motion information, which overcomes the limitations of ignoring detailed information based on global representations and the difficulty of effectively capturing the relationship between neighboring regions based on local region descriptors. CSTL [32] takes inspiration from the observation that humans can discriminate between gaits by adapting their focus to different time scales. It proposes a context-sensitive temporal feature learning (CSTL) network to assess the importance of features by modeling relationships between multi-scale features. By enhancing scales that are more important and suppressing scales that are less important, the network can adapt. Another technique to solve the misalignment problem caused by temporal operations (e.g., temporal convolution) is SSFL. This method extracts the most discriminatory parts of a sequence in order to reassemble the frame of salient spatial features. Additionally, 3DLocal [20] asserts that current methods of providing localized parts do not produce accurate results because they average feature maps. They propose to solve this problem by supporting 3D localization operations that extract body parts in sequences with adaptive spatio-temporal scales, positions and lengths. In this way, a 3D local neighborhood with specific scales, positions, frequencies, and lengths can be used to learn the spatio-temporal patterns of body parts. Meanwhile, GaitEdge [33] argues that end-to-end approaches are inevitably affected by gait-related noise (i.e., low-level texture and color information). An end-to-end framework is presented which effectively masks gait-independent information and unlocks the potential for end-to-end training, synthesizing pedestrian segmentation network output and then feeding it to a recognition network. In order to limit the amount of information available to the recognition network, the synthesized silhouette is comprised of trainable edges and a fixed interior. In addition, a large-scale self-supervised gait recognition benchmark using contrast learning is also proposed by GaitSSB [34], as well as the collection of a large-scale unlabeled gait data set, GaitLU-1 M, consisting of 1.02 million walking sequences. A conceptually simple model of gait recognition, the GaitSSB, is proposed to provide a robust empirical basis for the model.

It is worth noting that in recent years, appearance-based approaches have typically achieved higher recognition accuracies compared to model-based approaches.

Skeleton-based methods are among the most widely used for model-based gait recognition. In traditional skeleton-based gait recognition methodologies, a great deal of effort is put into extracting discriminative parameters from raw skeleton data to determine the gait and then applying those to the gait recognition algorithms [27, 35, 36]. For example, several spatiotemporal and dynamic parameters for gait recognition were computed by Deng et al. [37]. However, these methods rely on manually designed features, making the whole process complex and undesirable.

To address real-world application demands, researchers collected the GREW [38] and Gait3D [39] datasets in outdoor environments in 2021 and 2022, respectively, marking a shift from indoor to outdoor research. Model-based approaches to gait recognition have gained new vigor with the gradual shift in research direction from indoor to outdoor and the improved robustness of lightweight pose estimators [21, 22]. Model-based methods enable gait recognition to perform more robustly in complex environments in the face of unpredictable factors such as noise interference, mixed backgrounds, unconstrained walking paths, and occlusions.

Specifically, the pose-based spatio-temporal network PTSN [23] uses CNN for spatial modeling and LSTM for temporal modeling for spatio-temporal gait feature extraction. It is the first to propose a gait recognition method using pose estimation and utilizing pose keypoints. PoseGait [27] computes joint angles, bone lengths, and joint motions by means of 3D keypoints in Euclidean space. The method utilizes 3D human pose and a priority knowledge of the human body and overcomes the problem of clothing variations by using hand-crafted 3D pose estimation features. These hand-crafted features are then input into a convolutional neural network to learn advanced spatio-temporal features. The method was evaluated in the cross-view setting of CASIA-B [36], with highly competitive results compared to appearance-based methods.

In order to represent gait more accurately, GaitGraph [28] uses RGB images directly to determine robust skeleton postures. Based on their arguments, these methods misrepresent fine-grained spatial information while silhouette images retain several recognizable visual cues as well as gait features as they lose the fine-grained spatial information. Therefore, they proposed GaitGraph to achieve a number of modern gait recognition methods by combining skeleton poses with graph convolutional networks (GCNs) to create a method based on modern model-based gait recognition. Gaitgraph2 [40] believes that silhouette images contain a wide range of visual cues that aren’t actually gait features but can be used for recognizing gait patterns, but they can also provide context for deceiving the system in the process. Hence, they propose an algorithm for gait recognition that combines higher-order inputs with residual networks that use graph convolutional networks (GCNs) to produce an architecture that is efficient and effective for the recognition of gait and its variations. HMRGait [41] propose a model-based approach for end-to-end gait recognition. The study utilizes a skinned multi-person linear (SMPL) model in order to model humans and parameters of this model can then be estimated through the use of a pre-trained human mesh recovery (HMR) network. The reconstruction loss that is introduced between the contour masks of the gait dataset and the contours of the renders of the estimated SMPL model that is generated by the differentiable renderer allow them to compensate for the discrepancy between the gait dataset and the dataset used for pre-training HMR. A study conducted by SMPLGait [29] has found that most existing gait recognition methods learn features from either contours or skeletons, but that the combination of these two data sources may offer more than just one advantage. In the past, multimodal gait recognition methods have mainly utilized the skeleton as an aid in extracting local features, instead of taking advantage of the inherent discriminative properties of skeleton data. In concert with bimodal fusion (BiFusion) techniques, they propose that to learn rich recognition features from gait patterns, the skeleton should be mined for discriminative gait patterns, which can then be combined with contour representations.

We found that many studies in the field of appearance-based approaches emphasize modeling local limb motion details and extracting local motion features through corresponding modules. Drawing inspiration from these appearance-based approaches, our proposed skeleton-based gait recognition network focuses on modeling the relationships between different limb motions to achieve richer feature representations.

The whole process of GaitDLF is shown in the diagram Fig. 3. To be more specific, considering \(X_i\) as the representation of the input skeleton sequence, the entire process can be mathematically formalized as follows:where \(F\left( \cdot \right)\) denotes the Feature Extractor responsible for extracting key motion patterns and features \(Z_i\) from the skeleton sequence. \(F_\textrm{G}\left( \cdot \right)\) represents GAS, \(F_\textrm{L}\left( \cdot \right)\) represents LDS, \(concat\left( \cdot \right)\) signifies the channel-wise concatenation operation, and \(Y_i\) denotes the final gait features used for loss computation.

In comparison with earlier research, GaitDLF not only emphasizes global motion representation but also underscores local motion representation, thus achieving superior recognition capability. The details of the Feature Extractor are discussed in Sect. 3.3, while the Local Detail Stream is elaborated in Sect. 3.4 and the Global Awareness Stream is detailed in Sect. 3.5.

In our GaitDLF, the first step is to pass the input skeleton sequence through a feature extractor to extract temporal and spatial features.

Our approach mainly addresses how to enable the network to focus on the unique information of different limbs and how to adaptively fuse limb-level features, for which we refer to the structure proposed by GaitGraph2. As shown in Fig. 4, this extractor computes the input skeleton sequence into three features, position, velocity, and skeleton. Then these three features through three branches of ST-GCNs, respectively. To reduce computational complexity, channel splicing is performed in the middle of the network to fuse the features. Subsequently, the fused features continue to extract spatio-temporal features through multi-layer ST-GCNs. The ST-GCN blocks utilize the ability of graph convolutional networks (GCNs) to model skeleton data in the spatial domain to capture the connectivity between human joints through the graph structure. Each ST-GCN block consists of a graph convolutional layer for learning feature representations on the skeleton graph and a temporal convolutional layer for capturing the evolution of movements over time. This combination of modeling in the temporal and spatial domains allows our network to effectively understand and characterize the complexity of human actions. With this in-depth feature extraction, the ST-GCN block significantly improves the accuracy and efficiency of gait recognition [40]. Among them, the improved ST-GCN adds a bottleneck and residuals to the original ST-GCN [42, 43] as shown in Fig. 5.

With this extraction method, we can effectively capture the basic motion patterns and features in the skeleton sequence, thus laying the foundation for the subsequent gait recognition task. It should be emphasized that this feature extractor has the same architecture and the same feature computation method as GaitGraph2 [40].

Subsequently, the extracted features are separately fed into the Global Awareness Stream (GAS) and Local Detail Stream (LDS). This facilitates the extraction of both global and local joint motion information. While GAS focuses on encompassing the overall skeletal motion representation, LDS intricately captures local limb motion information, thus enhancing the network’s recognition capacity.

Prior methods commonly employed graph convolutional networks and temporal convolutional networks (TCN) to extract spatial and temporal features from skeleton sequences, culminating in significant recognition accuracy. However, they pooled the extracted high-level features directly across the joint dimensions to obtain the final gait representation. This approach is simple and effective, but since most people’s gait movements are very similar and more accurate recognition often requires subtle limb movements, we further model the high-level features based on different limbs, and the proposed Local Detail Stream achieves more accurate recognition accuracy.

The key of LDS is to effectively partition the high-level features \(Z_i\) into different limb parts and map them separately to obtain limb-level motion features that carry information unique to different limb parts. This dynamic fusion allows the network to adaptively focus on more significant information and extract finer features. LDS consists of the following two modules.

Beyond the Local Detail Stream, perceiving the global skeletal structure is also paramount. Effectively utilizing both the global skeletal framework and local limb information can offer rich features for gait recognition. The GAS module in this paper aims to extract global motion features from skeleton sequences, thus giving the network the ability to recognize global motion. Through global average pooling, we reduce the dimensions of the high-level features from output \(Z_i\), yielding an overall representation of global motion. Average pooling computes the average of multiple features, thus yielding information about the global skeleton that complements the local information extracted from the local detail stream. Subsequently, we employ a fully connected layer to map these features into the discriminant space to enhance the discriminative and unique nature of the features. Through GAS, we capture overall motion patterns in the skeleton sequences, providing information support from a global perspective for recognition tasks. This process can be expressed as:where \(\delta \left( \cdot \right)\) denotes the activation function, \(\mathrm{{FC}}\left( \cdot \right)\) signifies fully connected layers, \(\mathrm{{GP}}\left( \cdot \right)\) represents global average pooling, \(f_\textrm{g}\) denotes global perception features, and \(F_\textrm{G}\left( \cdot \right)\) represents GAS.

A trio of excellent gait datasets are employed, including CASIA-B [36], which is widely used indoors, Gait3D [39], which is famous for its diversity in real-life settings, and GREW [38], which is commonly used in wild environments. As shown in Table 1, the relevant statistics pertaining to the number of sequences and identities are provided. Subsequent sections present a comprehensive outline of the data collection procedures for each dataset, underscoring the notable distinctions between indoor and outdoor datasets.

The aim of this study is to compare GaitDLF with state-of-the-art skeleton-based methods for gait recognition. A systematic and comprehensive experiment was conducted on three different datasets to validate GaitDLF’s effectiveness. These datasets are CASIA-B, Gait3D, and GREW. Our first comparison is between GaitDLF and representative skeleton-based methods, including PoseGait, GaitGraph, and GaitGraph2. This choice was made because of the consistency between these three methods and our own evaluation protocol.

As shown in Table 3 in the CASIA-B indoor dataset, the performance of GaitDLF is only \(84.86\%\) in the NM condition, which is not as good as GaitGraph’s \(87.7\%\), and only \(70.74\%\) in the BG condition, which is also not as good as GaitGraph’s \(74.8\%\). But comparing these several skeleton-based methods, our method was able to obtain higher results in the coat-loaded (CL) condition, which suggests that methods focusing on localized limb movements can show higher robustness when the overall skeleton is more occluded by clothing. However, the CASIA-B indoor dataset was collected in the laboratory in order to bridge the gap between laboratory studies and real-world applications. We need methods that are more applicable to wild environments.

On both the GREW and Gait3D datasets, our method achieves the highest recognition accuracy and has the best recognition rate compared to the best methods on both datasets, as shown in Table 4. Importantly, on the GREW dataset, our accuracy is an impressive \(30\%\) higher than that of GaitGraph2. This result highlights the ability of our method to efficiently learn complex and unique gait features from large-scale datasets, which makes it more suitable for real-world gait recognition tasks. It further shows that compared to other methods that pool over the whole skeleton to obtain global features, our proposed modeling and dynamically adaptive fusion of local limbs is clearly more advantageous.

Given the unique characteristics of these three datasets, it is possible to see that our method performs well primarily on larger, more influential datasets (e.g., Gait3D and the GREW dataset in the wild). For the smaller CASIA-B dataset, our method does not achieve the best results and performs second only to GaitGraph. This is because GaitDLF possesses a more complex design than other skeleton-based networks to cope with the challenges of complex scenarios, and thus, on the small dataset, this may have led to overfitting, making GaitDLF difficult to achieve the best performance. However, on large datasets, GaitDLF achieves very high performance.

Since appearance-based methods have been predominantly used in gait recognition and the silhouette images used include gait information and other recognizable visual cues (e.g., clothing, handbags), the skeleton-based methods only use dynamic skeletal sequences of the subject to extract gait features. Skeletal-based methods, on the other hand, use only the dynamic skeletal sequence of the subject to extract gait features. The GaitDLF is then compared with various state-of-the-art appearance-based methods, such as GaitSet, GaitPart, GaitGL, and GaitBase, in order to assess its performance.

As shown in Table 5, the appearance-based state-of-the-art method outperforms the accuracy of our GaitDLF on the CASIA-B dataset. However, it is worth noting that the CASIA-B dataset was collected under the 2006 constraints and contains only 124 individuals and approximately 13K video sequences, as well as artificially formulated tilt angles, and occlusions. It is shown that in indoor environments, the appearance-based approach can exhibit the highest gait recognition accuracy due to the lack of many complicating factors.

It can be concluded from Tables 3 and 5 that the performance of all the methods is susceptible to the change in walking conditions, based on the performance of skeleton-based and silhouette-based methods in the indoor dataset. Moreover, the recognition performance of the appearance-based methods exhibits large fluctuations when the tilt angle changes, and it can be expected that such fluctuations will be even larger in complex environments. Therefore, in wild environments, skeleton-based methods, due to their greater robustness, are bound to be more advantageous.

One of the most notable features of the GREW dataset is that it is available in 2021 in a highly complex and free environment with 26K individuals and approximately 128K video sequences. The dataset is completely unconstrained, has a large number of undefined variations, and has diverse and useful perspectives. GREW also includes a variety of challenging factors such as complex backgrounds, occlusion, carrying objects, and wearing jewelry. As shown in Table 6, GaitDLF achieves a Rank-1 accuracy of \(68.77\%\), and the Rank-1 accuracy of the state-of-the-art silhouette-based method is \(60.1\%\). This indicates that GaitDLF outperforms the existing state-of-the-art appearance-based methods in an unconstrained wild environment.

Skeletal-based gait recognition can prioritize pose, angle, and directly relevant gait information, whereas appearance-based networks tend to emphasize subjective features such as color and texture. Therefore, for such a large dataset collected in unconstrained and complex environments, a skeleton-based approach that extracts gait features only from skeleton sequences is more appropriate than an appearance-based approach.

The Gait3D dataset contains only 4K individuals and 25K video sequences, which are collected in supermarkets, and the amount of data is much smaller than that of the GREW dataset, as shown in Table 6, the Rank-1 accuracy of GaitDLF is only \(27.2\%\), and that of the state-of-the-art silhouette-based method is \(64.4\%\). It shows that the silhouette-based and skeleton-based methods have their own advantages and usage scenarios.

The above results show that for gait recognition under strict constraints, good results can be achieved using an appearance-based approach, but for unconstrained wild environments with a large number of samples, a skeleton-based approach would be more appropriate. We believe this is caused by the fact that the skeleton sequence contains much less information than the silhouette sequence. Skeleton sequences simply include only the coordinates of each joint and the confidence level, whereas silhouette maps contain a large number of visual cues. However, large datasets have more data volume and diversity, which can help skeleton-based models better capture different variations and characteristics of gait. Therefore, on smaller datasets, the diversity of data may not be sufficient to support the skeleton based model to learn a wider range of gait variations, leading to performance degradation.

In conclusion, a larger capacity and more diverse dataset can help the skeleton-based model better capture a variety of gait variations and features. Compared with appearance-based methods, model-based methods focus only on human gait information and do not focus on rich appearance information, although it performs poorly on small datasets, in datasets with many unpredictable influences, and due to the large amount of data, skeleton-based methods are precisely able to ignore very sensitive appearance features and learn gait features. Therefore, based on this fact, this approach can be more effective in realizing its potential when applying gait recognition tasks in datasets of complexity and large data volumes or even in the real world.

Our objective in this section is to determine the effectiveness of the modules we have designed by performing an extensive ablation analysis of each module in GaitDLF for each module. Using Gaitgraph2 as a baseline, this study is able to improve the effectiveness of the method by modifying its loss function, data augmentation, and test time augmentation (TTA) strategies in order to improve its performance. During the testing of Gaitgraph2, the left-right flip sample and the time reversal sample are used as two supplementary samples, and the three embedded data obtained are connected to facilitate the subsequent distance calculation. And the distance metric between its gallery samples and probe samples uses the cosine similarity function. On the other hand, our approach avoids the use of supplementary samples during the testing process and instead computes the distance between gallery samples and probe samples by using the Euclidean distance between the respective feature vectors of gallery samples and probe samples. Additionally, auxiliary samples are not utilized, and all frames are input to the network for testing. It is shown in the table that in comparison with the original Gaitgraph2, the adjusted baseline achieves a recognition accuracy of approximately \(58\%\), which shows an increase of approximately \(25\%\) when compared to the original Gaitgraph2.

GaitDLF will be the subject of more experiments in this section. Besides testing the two modules above, we will also evaluate the role these two modules play in improving the feature extraction capability of GaitDLF by comparing the baseline model without the integrated SLM and DFF modules with the full GaitDLF model. Moreover, we will analyze GaitDLF’s complexity and compare it with currently available skeleton-based gait recognition methods.

We perform a complexity analysis of GaitDLF with the state-of-the-art skeleton-based methods, as shown in Table 8, the parameter of GaitDLF is 0.94 M, which is just 0.18 M more than the 0.76 M of GaitGraph2, while their FLOPs are comparable, indicating that GaitDLF optimizes the computational burden while maintaining a similar model design while maintaining a similar computational burden. More importantly, GaitDLF shows excellent performance on two important outdoor datasets, Gait3D and GREW, with the top-ranked accuracy reaching \(27.2\%\) on Gait3D and \(68.77\%\) on GREW, which clearly outperforms the other comparison methods. In contrast, Gaitgraph, despite being more lightweight in terms of model size, with a parameter count of only 0.35 M, does not perform as well in terms of actual performance.