Learning sequence-to-sequence affinity metric for near-online multi-object tracking

In this paper, we propose a sequence-to-sequence affinity metric for the data association of near-online multi-object tracking. The proposed metric learns the affinity between track sequence consisting of the already associated detections and hypothesis sequence consisting of detections in the near future. With the potential hypothesis sequences, we leverage the idea that if a track sequence has a high affinity for a hypothesis sequence, and the hypothesis sequence also shares a close affinity for a current detection, then the affinity between the track sequence and the detection is high. By using the short hypothesis sequence as a “bridge”, the proposed sequence-to-sequence affinity metric enhances the conventional track sequence to detection affinity metric and improves its robustness to object occlusion and missing. Besides, in order to eliminate the negative effects of false alarms, we propose a false alarm model using both appearance and scale features of detection. The robustness of the proposed affinity metric allows us to use a simple greedy data association algorithm. Experimental results on the challenging MOT16 and MOT17 benchmarks demonstrate the effectiveness of our method.