Human Pose Estimation Based on a Spatial Temporal Graph Convolutional Network

Abstract

To address the problem of poor detection and under-utilization of the spatial relationship between nodes in human pose estimation, a method based on an improved spatial temporal graph convolutional network (ST-GCN) model is proposed. Firstly, upsampling and segmented random sampling strategies are used to effectively solve the problems of class imbalance and the large sequence length of the dataset. Secondly, an improved detection transformer (DETR) structure is added to effectively suppress the generation of non-maximal suppression (NMS) and anchor points, a multi-head attention (M-ATT) module is introduced into each ST-GCN cell to capture richer feature information, and a residual module is introduced into the 9th ST-GCN cell to avoid possible network degradation in deep networks. In addition, strategies such as warmup, regularization, loss functions, and optimizers are configured to improve the model’s performance. The experimental results show that the average percentage of correct keypoints (PCK) of this method are 93.2% and 92.7% for the FSD and MPII datasets, respectively, which is 1.9% and 1.7% higher than the average PCK of the original ST-GCN method. Moreover, the confusion matrix corresponding to this method also indicated that the model has high recognition accuracy. In addition, comparison experiments with ST-GCN and other methods show that the computation of the model corresponding to this method is about 1.7 GFLOPs and the corresponding MACs are about 6.4 GMACs, which is a good performance.

1. Introduction

In recent years, computer vision has become increasingly mature, with the successful development of artificial intelligence technology and a dramatic increase in the computing power of hardware devices. A more fundamental task in this field, human pose estimation, is gradually becoming a hot topic. Thanks to the development of video capture devices and networks, it is essential to analyze and understand human posture from video data or image data [1] and then prepare it for tasks such as motion recognition, anomaly detection, autonomous driving, and human–computer interactions.

The essence of human pose estimation is to detect the essential parts and major joints of the human body in the video or picture, such as the eyes, nose, hands, shoulders, knees, feet, etc. It generally includes single-person scene detection and multi-person scene detection, the former of which is relatively simple and only requires the algorithm to extract the feature points of body parts and create a connected pose. In contrast, multi-person scene detection is more complex, mainly consisting of “top-down” and “bottom-up” methods. “Top-down” refers to identifying all the bodies in the video or picture first, followed by identifying the feature points of each person. The “bottom-up” approach first identifies all the feature points in the video or image [2] and then corresponds all the feature points to the relative human body through an algorithm.

In this paper, we take the application scenario of human pose estimation, i.e., an anomaly detection task, as there is no fixed basis for determining “abnormal human behavior”. For example, actions occurring at different positions in a fixed scene, actions occurring at different moments in a fixed scene, and different actions occurring at the same position and at the same moment in a specific scene can be judged as “abnormal behavior”. This is because whether human or an animal, behavior often depends on posture, movement, and the environment. For example, drawing on a blackboard is normal behavior, while drawing on a monument is judged as “abnormal”.

In addition, the lower probability of abnormal behavior compared to normal behavior makes data acquisition more difficult, which leads to an imbalance of positive and negative samples and makes it difficult for the model to learn enough features of abnormal behavior. In addition, the model performance is limited by the lighting, angle, and object movement speed [3]. Most detection methods also face the problems of complex behavior recognition, a multi-parameter search space, and high computational cost [4]. This is due to the differences in the appearance of the human body and the need to ensure multiple targets associated with time sequence are not skipped or missed when detecting different behaviors in video or image data, which leads to a sharp increase in the algorithm parameters and computation, making the training process increasingly complicated and ultimately reducing the performance of the model.

For example, the supervised convolutional neural network proposed by Newell et al. [5] to accomplish detection and grouping tasks can perform pose estimation in multi-person scenes. The AlphaPose [6] algorithm proposed by Shanghai Jiao Tong University can perform self-body estimation tasks while ensuring a balanced real-time performance and pose estimation accuracy. Park et al. [7] used convolutional neural networks to connect 2D pose results with image features for end-to-end learning to solve 3D human pose estimation tasks.

In addition, the human behavior detection algorithm proposed by Satybaldina [8] to detect the front of ATMs based on gesture features has poor model performance due to the influence of the surrounding environment, such as pedestrians, fallen leaves, and lighting. Although the algorithm proposed by Chengfei [9] to estimate the human posture of students in schools has improved significantly in model performance, it has many parameters and the model training process is complicated. The DNN model proposed by Toshev et al. [10] uses the AlexNet network to capture different joint features; however, the weight parameters obtained by this method are more coupled with the training data distribution.

In contrast, graph convolutional networks are more adept at handling typical non-Euclidean structured data such as human skeleton information. For example, the ST-GCN algorithm based on human skeleton joints proposed by Yan [11] combines a graph convolutional network (GCN) [12] and a temporal convolutional network (TCN) [13]. It extends to the spatial-temporal graph model, which forms a hierarchical representation of the skeleton sequence through a spatial-temporal graph. The GCN module is used to learn the local features of adjacent joints in space, and the TCN module is used to learn the local features of joint changes with time.

The space-time-separable graph convolutional network (STS-GCN) proposed by Sofianos et al. [14] includes the temporal evolution and the spatial joint interaction within a single-graph framework, which favors the cross-talk of space and time, while bottlenecking the space-time interaction allows to better learn the fully-trainable joint and time interactions. A novel graph-based method was proposed by Y. Cai et al. [15] to tackle the problem of 3D human body and 3D hand pose estimation from a short sequence of 2D joint detections. The method first uses a pre-trained cascaded pyramid network [16] for 2D pose prediction and then feeds it into a spatial temporal graph convolution network. The spatial temporal graph convolution connects each joint to its corresponding joint in adjacent frames in the time dimension and connects the joint points in each frame that have direct and indirect kinematic dependencies (adjacent and symmetric) in the spatial dimension. The joints are also classified into six classes, and different convolution kernels are used for training for different node classes.

The manuscript structure is shown in Figure 1 and the rest of this paper is organized as follows. In Section 2, we introduce the work related to data preprocessing and model structure design. In Section 3, we introduce the improved algorithm in detail including the model structure, the model configuration strategy, and the model training strategy. Section 4 is the experimental part, including an introduction to the dataset used in the improved algorithm, model evaluation indicators, comparison experiments, ablation experiments, and experimental results analysis. Furthermore, finally, we conclude the paper in Section 5.

2. Related Work

2.1. Data Preprocessing

In this paper, we will test two datasets, MPII [17] and FSD [18], as shown in Figure 2, which also shows the proportion of different categories of data in the FSD dataset. An up-sampling strategy is used in the data preprocessing stage to address the problem of unbalanced data categories in the FSD training set. It will randomly repeat a small number of sample categories so that the data in the processed training set have the same percentage in each category.

For a sample x in the minority class, the distance from the sample to all samples in the minority class sample set is calculated using the Euclidean distance as the criterion to obtain its k-nearest neighbors. Then, a sampling ratio is set according to the sample imbalance ratio to determine the sampling multiplier N (N should be less than the number of minority samples x minus 1). For each minority class sample x, some samples are randomly selected from its k-nearest neighbors. Assuming that the selected nearest neighbors are $\tilde{x}$, each randomly selected nearest neighbor $\tilde{x}$ can be constructed with samples x according to Equation (1) ($x_{new}$):

$$x_{new}=x+rand(0,1)\ast (\tilde{x}-x)$$

(1)

The model’s accuracy on the test set with or without the upsampling strategy is shown in

Table 1. Effects of up-sampling on model accuracy.

Up-Sampling Strategy	Test Set Accuracy
Yes	98.35%
No	97.23%

, and it can be seen that the accuracy of the model on the test set is improved after processing with the upsampling strategy.

In addition, since the critical point sequence length in the FSD training set data varies greatly, as shown in Figure 3, it is necessary to select a suitable sequence length to input into the model. However, the maximum value of sequence length in the training set data is 2037 frames, and the minimum value is 42 frames, in which fewer than 500 frames account for 27% of the samples. Therefore, the sample data can be divided into 350 intervals according to the effective frame length. A random frame is sampled from each interval to form a sample frame with a frame length of 350 during training, and it is used as the input of the model, which can effectively improve the model performance, as shown in

Table 2. Effects of frame length on model accuracy.

Frame Length	Test Set Accuracy
250	95.74%
400	94.39%
800	94.77%

.

It is worth noting that since the original training data only contains human joint coordinates and the confidence degree, this paper draws on the angular feature extraction and human activity model design methods proposed by Zhenyue Qin et al. [19] for AngNet regarding joint angle and He Jian et al. [20] for a human activity model based on the Cartesian coordinate system, respectively.

2.2. Design Model Structure

A technical view of the improved ST-GCN network is shown in Figure 4, which contains the improved algorithm structure, the algorithm application, and the algorithm model deployment. The algorithm structure combines OpenPose, DETR [21], and M-ATT [22]. An improved algorithm structure diagram, that redesigns both the model configuration and the model training strategy to address the problems of a high computational complexity, the difficulty in modeling multi-level semantic information, and the overlooked relationship between non-physically connected nodes (which leads to lower model accuracy), and more details of this algorithm structure are described in detail in Section 3. The algorithm application section shows the scenario tasks to which the improved algorithm proposed in this paper can be applied, including anomalous behavior detection [23], motion tracking [24], intelligent security [25], and so on. The algorithm model deployment section shows the tool platforms on which the improved algorithm can be deployed, including OpenVino [26], TensorRT [27], MediaPipe [28], and so on.

3. Proposed Methods

The flowchart of this algorithm is shown in Figure 5, the general flow is as follows. Firstly, the skeletal points are extracted frame by frame from the dataset used in this paper by the OpenPose algorithm, and the skeletal points are used as the pre-input of the improved ST-GCN network. Then, the temporal and spatial dimensions are transformed by the ST-GCN unit, and the TCN and GCN modules are used alternately to transform the joint features. It is worth noting that this algorithm also introduces the multi-headed attention mechanism and residual network and redesigns the transformer DETR structure. Finally, after processing the ST-GCN unit, the output results go through the average pooling layer, the fully connected layer and finally the softmax to complete the output.

3.1. Overview of Model Structure

The network structure of the improved ST-GCN is shown in Figure 6, and its core idea contains the following three aspects. Firstly, in terms of the network model structure, the improved network model still maintains the original ST-GCN nine-layer structure. The TCN module is used alternately with the GCN module with the following differences: Firstly, an improved DETR structure is added, i.e., the OpenPose algorithm module is used to complete skeletal point extraction frame by frame and replace the previous CNN module, which can detect all objects in parallel and avoid the generation of NMS and anchor points. Secondly, the improved network changes the attention model before the graph convolution operation to M-ATT and adds batch normalization (BN) [29] and rectified linear units (ReLU) [30] between the GCN module and the TCN module. The structure of the first eight ST-GCN units remains the same. Thirdly, a 9th ST-GCN unit is introduced into the residual network. From the training strategies, the expressiveness, detection performance, and generalization ability of the model are significantly improved by using warmup [31], stepwise-cosine, mix-up [32], and regularization strategies.

3.1.1. End-to-End Detection with an Improved DETR

Most detection methods perform detection based on detectors such as an anchor or Dense Box. However, since these methods perform repeated detection of the same target body, i.e., redundant detection frames are predicted, this problem must be addressed by means such as NMS. In contrast, the DETR structure proposed by Carion et al. does not require prediction of a candidate anchor and directly uses the model for violent regression.

Therefore, this method combines DETR and improves it. The whole DETR architecture consists of extracting skeletal points frame by frame by the OpenPose algorithm and encoder-decoder architecture, which does not require any custom network structure and simplifies the training process. It only requires a fixed set of object queries (query object); the DETR will predict the results in parallel based on the relationship between the target object and the global image context. The structure of the improved DETR is shown in Figure 7.

The input data will first be extracted by the OpenPose algorithm frame by frame and then combined with positional encoding as the input data of the encoder. The decoder will use a small number of fixed query objects and the encoder output as the input of the decoder at the same time. The decoder-processed data will flow into the improved ST-GCN-1 unit and be used as the input of the improved ST-GCN network. In the original ST-GCN network, which contains nine layers, the GCN and TCN modules are used alternately to transform the temporal and spatial dimensions. In the GCN module, a random division is used to combine the graph convolution operations of three subgraphs into one, which effectively improves the computational performance of the algorithm.

The DETR uses a decoder to predict a fixed number of detection frames; it does not need to use post-processing means such as NMS. It uses the set loss function as a supervised signal for end-to-end training, where the set loss function uses the bipartite matching algorithm to match the prediction object with the ground truth object, as shown in Equation (2):

$$\widehat{\sigma }=argmin{\sum }_i^N{L}_{match}\left(y_i,{\widehat{y}}_{{\sigma }_i}\right)$$

(2)

where y denotes the ground truth target set, ${\widehat{y}}_{{\sigma }_i}$ denotes the N elements in the prediction set, and $L_{match}\left(y_i,{\widehat{y}}_{{\sigma }_i}\right)$ denotes the loss value of the prediction and ground truth matching about element i of ${\sigma }_i$. The matching effect is shown in Figure 8.

Figure 8 contains the correspondence between two sets of prediction and ground truth, each set contains N target bodies, and each target body contains two elements, which are the confidence level of the class to which the target body belongs ($prob{s}_i$, where $i\in [1,N]$) and the location and size of the target body ($bbo{x}_i$, where $i\in [1,N]$, containing the center point of location coordinates and the width and height of the detection box). After matching the elements in the two sets, for example, ($prob{s}_1,bbo{x}_1$) in the prediction sets corresponds to ($prob{s}_3,bbo{x}_3$) in the ground truth, etc. (where ⌀ indicates that the class is empty), the Hungarian algorithm can be used to calculate its loss value, which abstracts the target detection task as a set prediction problem and then predicts all targets simultaneously and trains them end-to-end by set loss, thus simplifying the model training process and effectively avoiding problems such as anchor and NMS.

3.1.2. Stabilizing the Training Process Using Multi-Head Attention

The original ST-GCN network is designed with a layer of attention model to weigh the human torso before the graph convolution operation on the data. In contrast, M-ATT is introduced in the improved ST-GCN proposed in this paper, which can ensure that the DETR notices the information of different subspaces, which is beneficial for the network to capture richer feature information. It uses GCN and TCN alternately to transform the temporal and spatial dimensions, up-dimensionalize the human joint feature dimension, and down-dimensionalize the key frame dimension. Finally, the output result of the improved ST-GCN unit is output after the average pooling layer, the fully connected layer, and SoftMax, as shown in Figure 9, which is the M-ATT structure diagram.

Each head in M-ATT contains its parameters, and the integration of each M-ATT output is as follows: ST-GCN-1 to ST-GCN-8 are integrated by “splicing”, and ST-GCN-9 is integrated by “averaging”. When given $q\in R^{d_q}$, $k\in R^{d_k}$, and $v\in R^{d_v}$, each attention $hea{d}_i\left(i=1,2\dots h\right)$ is calculated as shown in Equation (3):

$$hea{d}_i=f\left(W_i^{\left(q\right)}q,W_i^{\left(k\right)}k,W_i^{\left(vs.\right)}v\right)\in R^{P_v}$$

(3)

where f denotes a function of convergent attention and $W_i^{\left(q\right)}$, $W_i^{\left(k\right)}$, and $W_i^{\left(vs.\right)}$ are learnable parameters.

Next, the results of each attention are stitched together and subjected to a linear transformation to obtain the final result, as shown in Equation (4).

$$MultiHead\left(q,k,v\right)=Concat\left(hea{d}_1,hea{d}_2\dots hea{d}_h\right)W^o$$

(4)

The application of M-ATT in improved ST-GCN is shown in Figure 10 (Figure 10 shows the ST-GCN-1 unit).

3.1.3. Using the Residual Network

Each cell of the original ST-GCN network will have a residual module and use dropouts for feature processing to enhance the spatial temporal information. This method is based on the structure of the original ST-GCN network. It adds a residual network in the ninth cell, which is used to solve the phenomenon of network degradation in deep networks and improve the model’s accuracy, as shown in Equation (5), which is the primary representation of a residual block.

$$x_{l+1}=h\left(x_l\right)+F\left(x_l,W_l\right)$$

(5)

$$h\left(x_l\right)=W_l^{{}^{\prime }}x$$

(6)

where $h\left(x_l\right)$ denotes the direct mapping, which is defined in Equation (6). $W_l^{{}^{\prime }}$ in Equation (6) denotes the 1*1 convolution operation and $F\left(x_l,W_l\right)$ is the residual part, as shown in Figure 11 (a structure diagram of the residual network designed in this method).

3.2. Model Configuration Strategies

3.2.1. Loss Function

In the selection of the loss function, this paper combines the characteristics of the pose estimation task, assuming that the set of real poses of a human body is $\left\{g_1,g_2\dots g_n\right\}$, where the parameter n is the batch size. The anchor pose is denoted as $\left\{a_1,a_2\dots a_m\right\}$, where the parameter m indicates the number of anchor poses. The set of anchor pose labels is $\left\{l_1,l_2\dots l_n\right\}$, the set of output results corresponding to the regression branch is $\left\{y_1,y_2\dots y_n\right\}$, the set of output results corresponding to the classification branch is $\left\{p_1^i,p_2^i,\dots p_m^i\mid i\in \left[1,n\right]\right\}$, and the value corresponding to the regression amount is the difference between the true pose and the corresponding anchor pose $\left\{g_i^{{}^{\prime }}=g_i-a_{l_i}\mid i\in \left[1,n\right]\right\}$. The loss function of the regression branch adopts the mean square error loss $L_{regression}=\frac{1}{n}{\sum }_{i=1}^n{\Vert y_i-g_i^{{}^{\prime }}\Vert }_2^2$, and the classification loss adopts the cross entropy loss $L_{crossentropy}=-\frac{1}{n}{\sum }_{i=1}^n{\sum }_{j=0}^m{l}_{i}ln\left(p_j^i\right)$, where $y_i$ denotes the output value corresponding to the ith data regression branch, $g_i^{{}^{\prime }}$ denotes the regression volume of the ith data, $l_i$ indicates the label value of the i th data anchor pose, and $p_i^j$ indicates the probability that the output of the classification branch of the ith data belongs to the jth anchor pose, as shown in Equation (7), which is the loss function used in this model.

$$L=L_{regression}\times {\theta }_{loc}+L_{crossentropy}$$

(7)

Among these metrics, the likelihood function of the regression branch obeys a Gaussian distribution with a mean value of the regression branch output and variance of ${\theta }_{loc}$, where ${\theta }_{loc}$ is a learnable parameter. In the process of model inference, the output result of the regression branch is used as the final output result of $x_i=y_i+a_{l_i}$, where $y_i$ denotes the output result of the regression branch and $a_{l_i}$ denotes the anchor pose corresponding to the input data.

3.2.2. Optimizer

When choosing an optimizer, the momentum-based SGD [33] algorithm is used in this paper instead of the original SGD algorithm because the update direction of the original SGD algorithm entirely depends on the gradient calculated by the current batch, which in turn leads to a less stable model. In contrast, the momentum-based SGD algorithm simulates the inertia of an object in motion by borrowing the concept of momentum from physics, preserves the direction of the previous update to some extent during the update, and fine-tunes the final update direction using the gradient of the current batch, i.e., replaces the real gradient with the momentum before accumulation.

The SGD algorithm based on momentum observes the historical gradient. It reduces the gradient in the current direction if the current gradient is not consistent with the historical gradient direction; conversely, if the current gradient is consistent with the historical gradient direction, the gradient in the current direction is increased. In general, at the beginning of model iteration, the current gradient of the model is consistent with the direction of the historical gradient, so momentum will play an important role in helping the model reach the optimal point more quickly. In contrast, at the later stage of model iterations, the model’s current gradient is inconsistent with the direction of the historical gradient. It oscillates around the convergence value, so momentum will play a decelerating role in increasing the stability of the model and thus stop the model from falling into a local optimum.

Assuming that SGD depends on the gradient of the current batch, the current gradient, $m_t$, is calculated as shown in Equation (8), where $g_t$ denotes the gradient and $\eta$ denotes the learning rate.

$$m_t=\mu \ast m_{t-1}+g_t$$

(8)

where $\mu$ is the momentum factor. In addition to this, the warmup strategy is also used as the method of learning the rate decay. The variation in learning rate with epoch value is shown in Figure 12. It can be observed that the learning rate gradually increases at the early stage of network model training, i.e., when $epoch<10$, when the model will correct the data distribution, which helps to slow down the overfitting phenomenon that may occur in the initial stage of the model. When $epoch>10$, the learning rate gradually decreases, when the model will converge better, which helps to maintain the stability of the model at a deeper level.

3.3. Model Training Strategies

In the model training strategy, the values of the model-related parameters are as follows: the $MaxEpoch$ value is 300, the $WarmupEpoch$ value is 10, the $WarmupStartlr$ value is 0.005, the $CosineBaselr$ value is 0.05, the $alpha$ value is 0.2, and the $lsEps$ value is 0.1. For the epoch and $BatchSize$ values, in training with the validation set data, the model accuracy increases with the increase in epoch value and increases with the decrease in $BatchSize$ value. After comparison, the optimal value of epoch is 300 and the optimal value of $BatchSize$ is 8.

In addition, the graphics card used for model training is an NVIDIA GeForce RTX 3070 Ti with the following software configuration: Python version 3.8, torch version 1.7.1, CUDA version 11.0, cuDNN version 8.0.5.39, torchvision version 0.82, ptflops version 0.6.9, pytorchModelSummary version 0.1.2, numpy version 1.18.5, matplotlib version 3.3.2, and PyCharm version 2020.2.

4. Experiments

4.1. Dataset Details

This paper will test the MPII and FSD datasets to enhance the model’s generalization performance. The MPII dataset was collected from YouTube videos covering 410 human activities (such as rock climbing, ice skating, fishing, etc.), containing 25,000 images with annotations. The annotation points contain ankle, knee, hip, shoulder, elbow, wrist, and 16 other types of annotations. The FSD10 dataset was constructed from the 2017 to 2018 World Figure Skating Championships, and the video frame rate was normalized to 30 frames per second with 10 action categories, including ChComboSpin4, 3Axel, FlyCameSpin4, 3Flip ChoreoSequence1, 3Loop, StepSequence3, 3Lutz, 2Axel, and 3Lutz-3Toeloop, with a resolution of 1080 × 720.

4.2. Evaluation Indicators

For the algorithmic models mentioned in this paper, the following four evaluation metrics will be used to evaluate them: $PCK$, Confusion Matrix, model computation, and $MACs$.

The $PCK$ indicator is used to measure the proportion of correctly estimated key points. Taking the total number of key points to be evaluated as $k\left(k>0\right)$, the $PCK$ indicator for the $i\left(i\in \left(0,k\right)\right)$th key point is calculated as shown in Equation (9):

$$PC{K}_i^k=\frac{{\sum }_p^{}\delta \left(\frac{d_{pi}}{d_p^{def}}\le T_k\right)}{{\sum }_p^{}1}$$

(9)

where the parameter p denotes the pth individual, the parameter $T_k$ denotes the manually set threshold ($T_k\in \left[0:0.01:0.1\right]$), the parameter k denotes the kth threshold, the parameter $d_{pi}$ denotes the Euclidean distance between the predicted value of the ith key point of the pth individual and the manually labeled value, and the parameter $d_p^{def}$ denotes the scale factor of the pth individual. In the MPII dataset, this parameter denotes the Euclidean distance between the upper left point of the head and the right Euclidean distance of the lower left point of the head, and the parameter $\delta$ takes a value of 1 if the condition holds and 0 otherwise.

The confusion matrix indicator is used to more clearly and intuitively portray the error indicators in the specific classification results. In a binary classification problem, if the classifier judges a positive case as a positive case, a true positive case (TP) is generated; if the classifier judges a negative case as a negative one, it is considered a true negative case (TN). The other two cases are called False Negative (FN) and False Positive (FP). The corresponding confusion matrices are shown in

Table 3. Confusion Matrix.

Confusion Matrix		True Results
		Positive	Negative
Prediction Results	Positive	TP	FP
	Negative	FN	TN

.

The amount of model computation indicates the number of computations of the model on the hardware unit, generally expressed as $FLOPs$ (floating point operations per second), which for a convolutional layer is shown in Equation (10):

$$FLOP{s}_{conv}=k_H\times k_W\times C_{in}\times C_{out}\times H_{out}\times W_{out}$$

(10)

where $k_W$ and $k_H$ denote the width and height of the convolution kernel, respectively, $C_{in}$ and $C_{out}$ denote the number of channels of the input feature map and the output feature map, respectively, and $W_{out}$ and $H_{out}$ denote the width and height of the output feature map, respectively. For a fully connected layer, the $FLOPs$ calculation is shown in Equation (11).

$$FLOP{s}_{FC}=C_{in}\times C_{out}$$

(11)

MACs (multiply–accumulate computations) represent the cumulative number of multiplications and additions of the model. $1MACs$ contains one multiplication and one addition, which are calculated as shown in Equation (12).

$$MACs=C_{out}\times H_{out}\times W_{out}\times C_{in}\times K_H\times K_W$$

(12)

In this paper, the ptflops and $pytorch\_model\_summary$ modules provided by the torch deep learning framework will be used to calculate the computation and MACs of the model.

4.3. Experimental Results Analysis

4.3.1. Comparison Experiments

As shown in

Table 4. PCK of different methods on the FSD dataset.

Method	Ankle*	Knee*	Hip*	Shoulder*	Wrist*	Elbow*	Thorax	Head Top	Mean
AGCN	90.2%	95.3%	90.2%	98.1%	88.6%	87.5%	89.1%	95.6%	91.8%
CTR-GCN	98.1%	96.1%	89.5%	90.7%	87.4%	89.8%	90.0%	91.8%	91.7%
ST-GCN	94.3%	93.0%	87.9%	87.6%	86.2%	90.9%	96.7%	94.0%	91.3%
Proposed	99.7%	99.1%	99.8%	87.6%	89.0%	91.2%	83.2%	95.7%	93.2%

and

Table 5. PCK of different methods on the MPII dataset.

Method	Ankle*	Knee*	Hip*	Shoulder*	Wrist*	Elbow*	Thorax	Head Top	Mean
AGCN	98.7%	95.4%	87.6%	89.7%	97.3%	89.2%	83.0%	91.2%	91.5%
CTR-GCN	93.4%	94.1%	89.4%	90.3%	90.9%	91.3%	85.2%	93.9%	91.1%
ST-GCN	90.0%	89.4%	93.7%	97.5%	89.7%	90.3%	84.0%	94.2%	91.0%
Proposed	99.1%	96.4%	99.8%	83.8%	90.3%	91.9%	85.4%	95.0%	92.7%

, the PCK values of the four network models of AGCN [34], CTR-GCN [35], ST-GCN, and the improved ST-GCN (proposed model) on the FSD and MPII test set data, respectively, are shown, where the values in the columns (Ankle*, Knee*, etc.) indicate the average values of the corresponding left and right nodes for that node. It can be seen that the average PCK of the AGCN network is 91.8% and 91.5% on both FSD and MPII datasets, respectively, the average PCK of the CTR-GCN network is 91.7% and 91.1% on both FSD and MPII datasets, respectively, and the average PCK of the ST-GCN network is 91.3% and 91.0% on both FSD and MPII datasets, respectively. In addition to the targeted preprocessing of the original dataset to improve as much as possible the model performance due to the irregularity of the dataset, the method mentioned in this paper redesigned the network model structure, network model configuration, model training, etc., so that the performance of the improved network model was improved to a certain extent on the FSD and MPII datasets. The average PCKs on the FSD and MPII datasets are 93.2% and 92.7%, respectively, as shown in Figure 13, and it can be seen that the method has a high recognition accuracy.

As shown in Figure 14 and Figure 15, the confusion matrices of the method on the FSD and MPII datasets show that the method has the highest probability of misclassifying the 3Loop action as FlyCamelSpin4 action among the 10 classifications of the FSD dataset (16%) and the highest probability of misclassifying the R-wrist node as L-knee action among the 16 classifications of the MPII dataset (14%). Among the 16 classifications of the MPII dataset, this method has the highest probability of misclassifying R-wrist nodes as L-knee actions, at 14%. In addition, the confusion matrix produced by this method on these two datasets has the largest element values on the diagonal, so it can be considered that this method has very good classification results.

The performance of the ST-GCN, AGCN, and CTR-GCN networks and the proposed method in this paper, which contains two indexes, model computation, and MACs, is shown in

Table 6. Performance of the different methods.

Methods	FLOPs (G)	MACs (G)
AGCN	~2.1	~7.0
CTR-GCN	~1.7	~6.8
ST-GCN	~1.8	~6.7
Proposed	~1.7	~6.4

. It can be seen that the model computation of the proposed method is about 1.7 G, which is similar to the model computation of the proposed method in the CTG-GCN network. The MACs index of the proposed method is about 6.4 G, which is smaller than that of the CTG-GCN network; that is, the proposed method in this paper has a better performance.

4.3.2. Ablation Experiments

Before conducting the ablation experiment, this paper will compare the average PCK values of different numbers of attention heads on both FSD and MPII datasets, as shown in

Table 7. Mean PCK of different attention head counts in FSD and MPII datasets.

h (Multi-Head)	Mean PCK of FSD	Mean PCK of MPII
2	90.6%	91.2%
4	91.7%	91.8%
6	92.1%	92.0%
8	91.0%	90.7%

. It can be seen that when the number of attention heads is six, the average PCK values are the largest in both datasets, which are 92.1% and 92.0%, respectively.

In order to evaluate the experimental effectiveness of the proposed method more accurately, the average PCK values of the proposed method on both FSD and MPII datasets with different experimental settings are shown in

Table 8. Mean PCK of our methods with different settings in the MPII dataset.

Methods	Mean PCK of FSD	Mean PCK of MPII
I${}_{original}$	91.3%	91.0%
I${}_{original}$ + I${}_{DETR}$	92.8%	92.3%
I${}_{original}$ + I${}_{MATT}$ (h = 6)	92.1%	92.0%
I${}_{original}$ + I${}_{Res}$	91.7%	91.4%
Proposed (I${}_{original}$ + I${}_{DETR}$ + I${}_{MATT}$ + I${}_{Res}$)	93.2%	92.7%

.

$I_{original}$ denotes the original ST-GCN network model, $I_{original}+I_{DETR}$ denotes the network model after adding the improved DETR structure, $I_{original}+I_{MATT}\left(h=6\right)$ denotes the network model after adding the MATT module, $I_{original}+I_{Res}$ denotes the network model after adding the residual network, and $I_{original}+I_{Res}$ denotes the network model after adding the DETR, MATT $(h=6)$, and residual modules, i.e., the method proposed in this paper. It can be seen that although adding $I_{original}$, $I_{MATT}\left(h=6\right)$, or $I_{Res}$ can all improve the average PCK values of the corresponding network models on both FSD and MPII datasets, $I_{DETR}$ is more effective compared to $I_{MATT}$ and $I_{Res}$, while the corresponding average PCK values are the largest when the methods mentioned in this paper are used simultaneously.

5. Conclusions

This paper proposes a human pose estimation algorithm based on an improved ST-GCN network. From the network model structure, an improved DETR structure is introduced. The OpenPose algorithm module is used to complete the skeletal point extraction frame by frame and replace the previous CNN module, which effectively suppresses the generation of NMS and anchor points. M-ATT is introduced to ensure that the transformer DETR can capture richer feature information. Introducing a residual network in the 9th ST-GCN unit in the improved ST-GCN avoids possible network degradation during training, and thus improves the model performance. From the training strategy, the problem of unbalanced FSD data categories is solved by the up-sampling strategy in the data processing stage, and the problem of inconsistent FSD data sequences is solved by the segmented random sampling. In addition, the loss function of the regression branch adopts the mean square error loss, the classification loss adopts the cross-entropy loss, and the momentum-based SGD algorithm is used as the optimizer and configured with warmup, mixup, and regularization strategies. The experimental results show that although $I_{DETR}$, $I_{MATT}(h=6)$, and $I_{Res}$ all help to improve the average PCK of the model on the datasets, our proposed method corresponds to the largest average PCK values, with 93.2% and 92.7% on the FSD and MPII datasets, respectively, which is higher than the average PCK of the original ST-GCN method on these two datasets by 1.9% and 1.7%, respectively. The values of the elements on the diagonal of the confusion matrix obtained for these two datasets are the largest, so this method has a very good recognition effect. In addition, the corresponding model computation of this method is about 1.7 GFLOPs and the corresponding MACs is about 6.4 GMACs, which has a better performance compared with the other three methods analyzed in the paper.