End-to-End Joint Multi-Object Detection and Tracking for Intelligent Transportation Systems

With the advancements in deep learning, object detection technology has developed rapidly. Existing object detection algorithms can be divided into two categories: Anchor-based [7‐9] and anchor-free [10‐12]. Anchor-based algorithms set a series of anchor boxes and regress offsets between the anchor boxes and ground truth using local features. The methods based on region convolution neural network (R-CNN) utilizes heuristic algorithm [13] and region proposal network (RPN) [7, 14, 15] to generate region proposals as anchor. Most anchor-free algorithms use full convolution networks to estimate the key points of targets, and further obtain the bounding boxes through the key points. These algorithms consider local features for bounding box regression, such that they only obtain the offsets between the anchor boxes or key points and the ground truth, rather than absolve bounding box coordination. Detection transformer (DETR) [12] adopted an encoder-decoder architecture based on transformers to achieve object detection. A transformer can integrate global information into the features at each position; however, the self-attention mechanism of the transformer requires a considerable amount of computation and GPU memory, which is difficult to apply to high-resolution feature maps. In the proposed joint detection and tracking framework, the network detects objects in a single image, and tracks objects in different images. However, the offsets for the same object in different images are hard to define. Hence, this paper introduces a global correlation layer to embed global information into the features at each position for absolute coordinate regression, which can be applied to higher-resolution feature maps, rather than the transformer.

With the improvement in detection accuracy, tracking-by-detection methods [16‐18] have become mainstream in the field of MOT. Tracking is considered as a data association problem in tracking-by-detection frameworks. Features, such as motion [19], shape [20], and appearance [21, 22], are used to describe the correlation between detections and tracks, and thus, a correlation matrix is established. Algorithms including the Hungary algorithm [23], JPDA [16] and MHT [24], input the correlation matrix to complete data association. Although these algorithms have made significant progress, there are certain drawbacks. First, they do not combine the detector and tracker efficiently, and a majority of them need to perform feature extraction separately, which involves unnecessary computation. Second, they often rely on complicated tracking rules for lifecycle management, resulting in numerous hyperparameters and difficult tuning. In the proposed approach, detection and tracking are performed in the same manner, such that they are well combined and the computation of feature extraction is reduced. Additionally, the proposed approach eliminates the complex tracking rules.

In the field of MOT, it is an important research direction to combine detection and tracking. With the quick maturity of multi-task learning in deep learning, many methods using a single network to complete detection and tracking tasks by adding ReID feature extraction to existing object detection networks [25‐27]. Wang et al. [28] proposed the joint detection and embedding (JDE) method that allows target detection and appearance embedding to be learned in a shared model. Bergmann et al. [29] proposed a JDT method that adopts Faster-RCNN framework, and accomplishes tracking by region of interest (RoI) pooling and bounding box regression without data association. Zhou et al. [10] considered current and previous frames as well as a heatmap, rendered from tracked object centers, as inputs, and produces an offset map, which simplifies data association considerably. Peng et al. [30] converted the MOT problem into a pair-wise object detection problem, and proposed chained-tracker method realizing end-to-end joint object detection and tracking. Similarly, this study also provides a new idea for joint detection and tracking. Compared with trackformer [31], which formulate the MOT task as a frame-to-frame set prediction problem and propose a tracking-by-attention network based on DETR [12], the network structure of GCNet is simpler and can reach a higher inference speed.

In this part, the global correlation layer and its application principle in end-to-end joint detection and tracking framework are introduced. Furthermore, the specific implementation of detection module and tracking module in the proposed GCNet are described.

Global correlation layer: The global correlation layer in GCNet encodes global information to generate the correlation vectors, which can be utilized in detection module and tracking module. Using feature map \({\varvec{F}}\in {{\varvec{R}}}^{h\times w\times c}\), two feature maps \({\varvec{Q}}\) and \({\varvec{K}}\) are obtained from the following two linear transformations:where \({{\varvec{F}}}_{ij}\in {{\varvec{R}}}^{c}\) denotes the feature vector at the \(i\) th row and \(j\) th column of \({\varvec{F}}\). Further, for each feature vector \({{\varvec{Q}}}_{ij}\), the cosine distance between it and all \({{\varvec{K}}}_{ij}\) is calculated. Following another linear transformation \(\dot{W}\), the correlation vectors \({{\varvec{C}}}_{ij}\in {R}^{{c}{\prime}}\) is obtained:

These correlation vectors \({{\varvec{C}}}_{ij}\) encode the correlation between the local feature vectors \({{\varvec{Q}}}_{ij}\) with the global feature map \({\varvec{K}}\), such that it can be used to regress the absolute bounding boxes for the objects at the corresponding positions in the image. All of the correlation vectors \({{\varvec{C}}}_{ij}\) can form a correlation map \({\varvec{C}}\in {R}^{h\times w\times {c}{\prime}}\), allowing us to obtain bounding boxes \({\varvec{B}}\in {R}^{h\times w\times 4}\) using a convolution layer with \(1\times 1\) kernel size. \({\varvec{K}}\) and \({\varvec{Q}}\) from the image in the same frame are used while performing object detection; conversely, \({\varvec{Q}}\) from the image in the previous frame and \({\varvec{K}}\) from the image in the current frame are used while performing object tracking. In this manner, detection and tracking are unified under the same framework.

Compared with the traditional self-attention layer, the global correlation layer has advantage in computation. The computation of traditional self-attention layer includes three parts: Computing attention weight, \(c\times \left(h\times w\right)\times {\left(h\times w\right)}^{\mathrm{T}}=c{h}^{2}{w}^{2}\); SoftMax, \(chw\); weighted summation, \(c\times \left(h\times w\right)\times {\left(h\times w\right)}^{\mathrm{T}}=c{h}^{2}{w}^{2}.\) As shown in Eq. (2), the computation of global correlation layer is \(c\times \left(h\times w\right)\times {\left(h\times w\right)}^{\mathrm{T}}=c{h}^{2}{w}^{2}\), which is significantly less than that of the total computation of self-attention layer \((2c+1){h}^{2}{w}^{2}\).

In terms of object classification brunch, this study uses the same network structure and training strategy as CenterNet. When infers, a detection heatmap \({{\varvec{Y}}}_{d}\) and tracking heatmap \({{\varvec{Y}}}_{t}\) are obtained in each frame. The detection heatmap \({{\varvec{Y}}}_{d}\) denotes the detection confidence of the object centers in the current frame, while the tracking heatmap \({{\varvec{Y}}}_{t}\) denotes the tracking confidence between the current and next frame. The peaks in the heatmaps correspond to the detection and tracking key points, and max-pooling is used to obtain the final bounding boxes, without applying box non-maximum suppression (NMS).where \(\mathrm{maxpool}\left({\varvec{H}},a,b\right)\) represents a max-pooling layer with kernel size \(a\) and stride \(b\). Hence, the GCNet can realize joint multi-object detection (MOD) and MOT, without complicated post-processes, such as NMS and data association, which have a concise pipeline.

Detection module: The detection module architecture is depicted as Figure 2, which contains three parts: Backbone, classification branch, and regression branch. The backbone is for high-level feature extraction. Because the classification is identical to CenterNet, each location of the feature map corresponds to an object center point, while the resolution of the feature map crucially affects the network performance. To obtain high resolution and retain a large receptive field, the same skip connection structure is acquired as a feature pyramid network (FPN); however, it only outputted the finest level feature map \({\varvec{F}}\). The size of the feature map \({\varvec{F}}\) is \({h}^{\prime}\times {w}^{\prime}\times c\), which is equivalent to \(\frac{h}{8}\times \frac{w}{8}\times c\); here, \(h\) and \(w\) are the height and width of the original image, respectively. This resolution is 4 times that of DETR. The classification branch is a full convolution network, and outputs a confidence map \({{\varvec{Y}}}_{d}\in {R}^{{h}^{\prime}\times {w}^{\prime}\times n}\) with values between 0 and 1. The peaks of the \(i\)th channel of \({{\varvec{Y}}}_{d}\) correspond to the centers of the objects belonging to the \(i\)th category. The regression branch is used to calculate bounding boxes \(\left\{{\left[x,y,h,w\right]}_{i}|1\le i\le N\right\}\). First, this paper considers \({\varvec{F}}\) and \({{\varvec{Y}}}_{d}\) as inputs, and generates three feature maps \({\varvec{K}}\), \({\varvec{Q}}\), and \({\varvec{V}}\).where \(Conv\left({\varvec{F}},a,b\right)\) denotes a convolution layer with kernel size \(a\), strides \(b\) and kernel number \(c\), and \(BN\) denotes batch normalization layer. \(Gate\left({\varvec{X}},{\varvec{Y}}\right)\) is depicted in Figure 3, which is a form of spatial attention. \({\varvec{P}}\) is the position embedding with the same shape as \({\varvec{F}}\), and is expressed as:

The two embedding vectors that are close in the position have a large cosine similarity, while the two that are farther away have a smaller cosine similarity. This attribute reduces the negative influence of similar objects while tracking. Further, the correlation vectors \({{\varvec{C}}}_{ij}\) between \({{\varvec{Q}}}_{ij}\) and \({\varvec{K}}\) are calculated using Eq. (2). The final bounding boxes \({{\varvec{B}}}_{d,ij}=\left[{x}_{ij},{y}_{ij},{h}_{ij},{w}_{ij}\right]\) can be obtained using Eq. (6). Here, the absolute coordinates and size of the bounding box are directly regressed, which differs from most existing methods, especially anchor-based methods.

Tracking module: Tracking is the process of assigning objects in the current frame to historical tracks, or generating new tracks. The architecture of the tracking module is depicted in Figure 4. The inputs of the tracking module are: (1) Feature map \({\varvec{K}}\) of the current frame, (2) detection confidence map of the current frame, and (3) feature vectors of historical tracks. Additionally, the tracking module outputs a tracking confidence and bounding box for each historical track. It can be observed, this architecture is almost identical to that of the detection module. Most of its network parameters are shared with the detection module, except for the fully connected layer used for calculating tracking confidence (the green block in Figure 4). The tracked bounding boxes are consistent with the detected target boxes in terms of expression, which is \({{\varvec{B}}}_{i}=\left[{x}_{i},{y}_{i},{h}_{i},{w}_{i}\right]\), with absolute coordinates and size. The tracking confidences indicate whether the objects are still present in the image of the current frame. The tracking module functions in an object-wise manner, such that it can naturally pass the ID of each object to the next frame, which is similar to parallel single-object tracking.

Although the proposed model can be trained end-to-end, the GCNet is trained in two stages in this study. First, the detection module is trained, and then, the entire network is fine-tuned. The training strategy of the classification branch is consistent with CornerNet. A heatmap \({{\varvec{Y}}}_{gt}\in {R}^{h\mathrm{^{\prime}}\times w\mathrm{^{\prime}}\times n}\) with 2D Gaussian kernel is defined as follows:where \({N}_{k}\) is the number of objects of class \(k\), \(\left[{x}_{n},{y}_{n}\right]\) is the center of object \(n\), and variance \({\sigma }^{2}\) is relative to the object size. \({\sigma }_{x}\) and \({\sigma }_{y}\) are expressed as shown in Eq. (8), and \({\eta }_{\text{IoU}}\) is set to \(0.3\).

The classification loss is a penalty-reduced pixel-wise focal loss.

The regression branch is trained using CIoU loss, as follows:where \(\left[ij\right]=1\) indicates the corresponding \({B}_{d,ij}\), and is assigned to a ground truth. A bounding box \({B}_{d,ij}\) is assigned to a ground truth if \({G}_{ijn}>0.3\) and \(\sum_{n}{G}_{ijn}-{\mathrm{max}}_{n}{G}_{ijn}<0.3\).

Furthermore, for \({B}_{ij}\) with \({\mathrm{max}}_{n}{G}_{ijn}=1\), the weight of their regression loss \({w}_{ij}\) is set to \(2\), and the other weights to \(1\). This is done to enhance the precision of the bounding boxes at the center points.

The entire network is fine-tuned using a pretrained detection module. At this training step, two images \({{\varvec{I}}}_{t-i}\) and \({{\varvec{I}}}_{t}\) are treated as inputs simultaneously, where \(i\) lies between \(1\) and \(5\). The loss contains two parts, i.e., detection loss of \({{\varvec{I}}}_{t-i}\) and tracking loss between the two images. The tracking loss also comprises two terms, i.e., regression loss and classification loss. The tracking ground truth is determined by object ID. \({B}_{t,ij}\) and \({Y}_{t,ij}\) are positive if \(\left[ij\right]\) in \({{\varvec{I}}}_{t-i}\) is equal to \(1\), and the corresponding objects exist in \({{\varvec{I}}}_{t}\). The total train loss is expressed as:

The inference pipeline for joint MOD and MOT is described in Algorithm 1. The inputs of the algorithm are consecutive frames of images \({{\varvec{I}}}_{1}-{{\varvec{I}}}_{t}\). Trajectorie \({{\varvec{T}}}_{i}\), confidence \({{\varvec{Y}}}_{i}\), and vector \(\left[{{\varvec{V}}}_{i},{{\varvec{Q}}}_{i}\right]\) of all tracks and candidates are recorded in four collections: \(\varvec{\mathcal{T}}\), \(\varvec{\mathcal{O}}\), \(\varvec{\mathcal{Y}}\), and \(\varvec{\mathcal{C}}\). At each time step, object detection is performed on the current frame of image \({\varvec{I}}\), and tracked the existing track \(\varvec{\mathcal{T}}\) and candidate \(\varvec{\mathcal{C}}\). Tracking confidences are used to update all confidences in sets \(\mathcal{Y}\) and \(\varvec{\mathcal{C}}\), and obtained \({\varvec{Y}}_{i}=\mathrm{min}\left(2\times {\varvec{Y}}_{i}\times {\varvec{Y}}_{t,i},1.5\right)\). The tracks and candidates with a confidence lower than \({p}_{2}\) are deleted, and other trajectories, candidates, and corresponding features are updated. This update strategy, \({\varvec{Y}}_{i}=\mathrm{min}\left(2\times {\varvec{Y}}_{i}\times {\varvec{Y}}_{t,i},1.5\right)\), provides these tracks with a higher tracking confidence, certain trust margin, and confidence possibly greater than \(1\). The detections with an IoU greater than \({p}_{3}\), or confidence less than \({p}_{2}\), are ignored. For the remaining detections, those with a detection confidence greater than \({p}_{1}\) are used to generate new tracks, and the rest are added to the candidate set \(\varvec{\mathcal{C}}\). As observed, the entire detection and tracking process can be performed in sparse mode, such that the overall computational complexity of the algorithm is extremely low.

Experiments of this study are conducted using the vehicle detection and tracking dataset, UA-DETRAC, which is captured from a roadside view, and can be seen as a typical application of environment perception in ITS. This dataset contains \(100\) sequences; \(60\) were used for training, and the remaining 40 were used for testing. The data in the training and test sets, which are derived from different traffic scenarios, make the test more difficult. The UA-DETRAC benchmark employs AP to rank the performance of the detectors as well as PR-MOTA, PR-MOTP, PR-MT, PR-ML, PR-IDS, PR-FM, PR-FP, and PR-FN scores for tracking evaluation. This paper refers to Ref. [32] for further details on the metrics.

All the experiments are performed using TensorFlow 2.0. The proposed model is trained with Adam on the complete training dataset of UA-DETRAC. The size of the input images is \(512\times 896\). Three commonly used data augmentation methods are employed: Random horizontal flip, random brightness adjustment, and scale adjustment. Hyperparameters \({p}_{1}\), \({p}_{2}\), and \({p}_{3}\) for the inference are set to \(0.5\), \(0.3\), and \(0.5\) respectively.

In the proposed joint detection and tracking framework, three main components influence the performance: 1) Gate by confidence map \({{\varvec{Y}}}_{d}\); 2) concatenated feature vector in \({\varvec{V}}\) for bounding box regression; and 3) specially designed position embedding \({\varvec{P}}\). The detection effects of the three models are compared with the GCNet to demonstrate the effectiveness of these components. Table 1 shows the results of the comparison. The full version of the GCNet exhibited the best performance, with \(74.04\mathrm{\%}\) AP on UA-DETRAC. The gate and feature vector of \({\varvec{V}}\) both yielded \(2\mathrm{\%}\) AP. The gate step explicitly merges the classification result into the regression branch, which plays a role of spatial attention and is conducive to the training of the regression branch. The concatenated feature vectors of \({\varvec{V}}\) for regression introduce more texture and local information, which is not included in the correlation vectors. This information is beneficial for inferring the size of the objects. To demonstrate the role of the position embedding, it is replaced with a normal explicit position embedding, where \({P}_{ijk}\) equals \(i\) when \(0\le k<c/2\), and equals \(j\) when \(c/2\le k<c\). Notably, the self-designed position embedding attains a 5.80% increase in AP.

The ablation study is conducted only on the detection benchmark. This is because the tracking module shares most of its parameters with the detection module, and the tracking performance is highly correlated with the detection performance. The results of the ablation study can thus be extended to the tracking module.

Table 2 shows the results obtained using the UA-DETRAC detection benchmark. The GCNet demonstrates promising performance, and outperforms most detection algorithms on this benchmark. It attains a high AP on full and medium difficulty as well as on night and rainy images of the test set. Figure 5 shows the PR curves of the GCNet and other algorithms, exposed by the UA-DETRAC dataset. It can be observed that the proposed model is far more effective than the baselines in each scenario. Notably, the proposed model does not employ any other components for better precision, and the backbone network is the original version of ResNet50. Compared with other methods, the performance improvement of GCNet benefits from the global correlation mechanism in the model. In the complex traffic scenarios, there are many non-critical areas such as trees and buildings, as well as many traffic participants with similar appearances. When using correlation convolution for object detection, the correlation between different objects will decrease with the increase of the distance, which can effectively reduce the false and missed detection. When only the detection module of the GCNet is used, it can run at \(36\) frame/s on a single Nvidia 2080Ti.

The aim of designing GCNet considers both MOD and MOT. This is the real purpose of introducing the global correlation layer to regress the absolute coordinates. The tracking results are shown in Table 3. The MOT metrics with “PR-” can evaluate the overall effect of detection and tracking. EB and KIoU are the UA-DETRAC challenge winners. In the process of multi-objects tracking, the pixel coordinate distance of the same target among continuous frame images is generally close. Benefiting from the position embedding and global correlation, our method can encode spatiotemporal motion of tracking target implicitly, which can improve the matching accuracy between trajectory in the precious frames and detection results in the current frame. Additionally, a significant PR-MOTA score and an excellent PR-MOTP score are obtained, approximately twice as high as that of EB and KIoU combined. Moreover, the leading scores are obtained in PR-ML and PR-FN on the UA-DETRAC tracking benchmark. Because the detection and tracking modules share most of the features, calculating the entire joint detection and tracking pipeline is approximately the same as calculating detection alone, and it can achieve a speed of approximately 34 frame/s.