Survey of pedestrian detection with occlusion

Papageorgiou and Poggio proposed Haar in 2000. It can reflect the change of gray image scale, including four categories: edge feature, line feature, center-surround feature, and special diagonal line feature. Haar is the foundation of pedestrian detection technology.

The traditional detection methods always followed the structure of the “artificial feature + classifier”. First, the picture's features should be extracted, including grayscale, edge, color, gradient histogram, and other information for the object. Then, the classifier determines which features belong to pedestrians. Such as, SVM, Adaboost, etc. The traditional method's frame is shown in Fig. 3.

There are two main approaches to deal with occlusion in traditional detection methods: (1) The object is divided into different parts, and the visual part can infer the location of pedestrians. (2) A specific classifier is trained for the common occlusion in daily life to reduce the influence of occlusion and correctly judge the pedestrian position.

The component-based method is the most common and effective method to deal with the occlusion problem. The idea of this method is simple: though part of the pedestrian to be detected is occluded, the other parts can be used to locate the position of the pedestrian.

Leibe and Seemann [19] proposed a pedestrian detection algorithm in crowded scenes, which is equivalent to the prototype of pedestrian detection under occlusion. This kind of occlusion is an intra-class occlusion. The core part of their method is the combination of local and global cues via a probabilistic top-down segmentation. Mohan [20] found that if pedestrians are divided into four parts: head and shoulder, leg, left arm, and right arms, it is more effective to deal with occlusion. Mikolajczyk [21] further divided people into seven parts based on Mohan’s method. Inspired by this, Bo and Nevatia [22] modeled humans as a collection of natural body parts, prompting Edgelet features. An Edgelet is a short segment of line or curve that denote the positions of normal vectors points in an Edgelet of \(\left\{ {u_{i} } \right\}_{i = 1}^{k}\) _and \({\text{\{ }}n_{i}^{E} \}_{i = 1}^{k}\), where k is the length of the Edgelet. Given an input image I, denote by MI(P) and NI(P) is the edge intensity and normal at position p of I. The affinity between the Edgelet and the image I at position w is calculated by the equator (1):

Xiaoyu takes HOG (Histograms of Oriented Gradients) and LBP (Local Binary Pattern) [23] as the feature set and proposed a new human body detection method capable of handling local occlusion based on the component detector. Although part-based detectors perform better than other detectors, the sliding-window approach handles partial occlusions poorly. Two detectors are used to integrate the advantage of part-based detectors in occlusion handling to the sliding-window detectors: a global detector that scans the entire window and a partial detector in a local area. The response of the HOG-LBP feature of each block to the detector is used to construct an occlusion likelihood map. Once the occlusion is detected, part of the detector will be triggered to detect the visual part. Enzweiler and Eigenstetter [24] present a multi-cue component-based mixture-of-experts framework. Figure 4 shows the frame. The framework involves a set of component-based expert classifiers trained on features derived from intensity, depth and motion. This method, unlike Wu and Nevatia's approach. Wu requires specific camera settings, which need the camera to be positioned from top to bottom with the assumption that the heads of pedestrians in the scene were always visible of semantic segmentation. Flores-Calero [25] uses logic inference, HOG, and SVM are proposed to deal with occlusion. The input image is divided into twelve regions, and the feature vector is extracted for each region, and a classifier based on SVM has been built. These classifiers are used to build the final classifier. With this design, it is possible to capture the specific detail of each part of the human body, such as the head, legs, arms, and body.

Training a set of special classifiers is another way to deal with occlusion. Each classifier is designed for a certain type of occlusion. Training special occluded classifier requires the prior knowledge of the occlusion types.

M. Isard found that adding the background appearance model into a pedestrian tracking algorithm is more robust and could effectively deal with deformation and occlusion. Wojek and Walk [26] apply the idea that not only individual pedestrians, but also surroundings need to be detected. They combined 3D scene tracking with detectors that perform occlusion handling by explicitly leveraging 3D scene information. The disadvantage of this approach, however, is that it is too costly. To solve this problem, Mathias and Benenson proposed Franken-classifiers [27]. It is less expensive to train a set of occlusion-specific classifiers. Sixteen occlusion-specific classifiers can be trained at only one-tenth of the cost of one full training. Felzenszwalb [28] proposed deformable part models (DPM). The algorithm adopts the improved HOG feature and uses SVM classifier and sliding-window detection, which is robust to the deformation of the target. Based on DPM(Deformable Parts Model), the model includes a linear filter incorporating a dense feature graph. A filter is a rectangular template defined by an array of d-dimensional weight vectors. The response, or score, of a filter F at a position (x, y) in a feature map G is the “dot product” of the filter and a subwindow of the feature map with a top-left corner at (x, y):

Andriluka and Schiele proposed a new two-person detector based on the DPM method to deal with occlusion. Instead of regarding the occlusion between people as interference, they think it is a peculiarity. This detector can predict the boundary boxes of two people with good results even under severe occlusion. The performance of this special occluded classifier is better than a single detector. However, the algorithm based on special classifier is time-consuming, and its robustness is not good. The algorithm does not work very well with a complex background.

There are three mainframes of pedestrian detection algorithms based on deep learning. (1) Based on depth belief network (DBN) [29]; (2) based on a convolutional neural network [30] (CNN); and (3) based on recurrent neural network (RNN). The Convolutional Neural Network is used widely in the pedestrian detection algorithm. There are two ways to deal with occlusion in deep learning algorithm: One approach is to introduce the idea of part into a specific layer of the neural network; the other one is the optimization of neural network's judgment mechanism.

A deep belief network (DBN) proposed by Geoffrey Hinton in 2006 is an extremely efficient learning algorithm, which is a generic model. By training the weights among its neurons, we can let the whole neural network generate training data according to the maximum probability. In other words, pre-training + Fine-tuning. This idea has become the main framework of deep learning algorithms. The components of DBN are Restricted Boltzmann Machines (RBM). The process of training DBN is carried out layer by layer. In each layer, data vectors are used to infer the hidden layer, which is then treated as the data vector of the next layer (the higher layer).

Wanli and Xiaogang [31] combined the component model with DBN. They formulate feature extraction deformation handling, occlusion handling, and classification into a joint deep learning framework and propose a new deep network architecture. When part detection map and part scores are obtained, the joint framework can take full advantage of them. However, when there is an occlusion or large deformation, to integrate the fraction of partial detectors is a key problem to be solved urgently. In order to solve the defects of part detectors, they proposed a probability model based on improved RBM [32]. The hierarchical structure of the DBN model matches the multi-layers of the parts model well. This can achieve more reliable visibility estimation, and it is better to eliminate the influence of occlusion. The framework is shown in Fig. 5. It works well with both single-detector and multi-pedestrians systems.

Convolutional Neural Networks (CNN) are a class of Feedforward Neural Networks that contain convolutional computation. Figure 6 shows the framework. Pedestrian detection algorithm based on Convolutional Neural Networks is mainly divided into two categories. First, it is the two-stage detector algorithm, which divides target recognition and target location into two parts. Region-Convolutional Neural Networks(R-CNN) series algorithm has high accuracy at a slower speed. Second, it is one-stage detector algorithm that includes Single-Shot MultiBox Detector (SSD) [33, 34] and You Only Look Once (YOLO) [35, 36]. YoLo is fast, but it has erratic effects with inherent advantages in detecting small targets and dense targets. SSD has high accuracy while maintaining fast speed.

At present literature, most of the pedestrian detection algorithms are based on a two-stage detector framework. Wanli [37] proposed deformable deep convolutional neural networks for generic object detection. The proposed algorithm has a new pre-training strategy to learn feature representations more suitable for the object detection task, which significantly improves the effectiveness of model averaging. Furthermore, jointly learning deep features [38], deformable parts, occlusion, and classification are proposed to established automatic mutual interaction among components. Yonglong Tian and Ping Luo [39] proposed the Deep-Parts, which is inspired by Franken-classifiers. Deep-Part introduces the idea of constructing a part pool that covers all the scales of different body parts and automatically chooses important parts for occlusion handling. These methods' occlusion handling strategy is to learn a set of detectors and integrate the output of these ensemble models. But it is complicated and time-consuming. Shanshan Zhang combines Faster R-CNN with an attention mechanism [40]. This method is easy to train and has low overhead. The attention mechanism has been widely used in CNN for object detection. The additional attention mechanism guides the detector to pay more attention to visible body parts, as it is shown in Fig. 7.

Zou [41] proposed an attention guided neural network model (AGNN), which uses a fixed-size window slides on a still image without overlapping to generate a set of sub-images. The attention network performs local feature weighting by selecting the features of the pedestrian's body parts. Zhou and Yuan [42] propose a reduced computational complexity of a multi-label learning approach that jointly learn part detectors to capture partial occlusion patterns. The part detectors share a set of decision trees via boosting to exploit part correlations.

The introduction of part detectors to optimize loss function is a good strategy to deal with occlusion. Xinlong Wang and Tete Xiao [43] set repulsion loss function on the Faster R-CNN. This loss is driven by two motivations: the attraction by target, and the repulsion by other surrounding objects. The crowd occlusion makes the detector sensitive to the threshold of non-maximum suppression (NMS): a higher threshold brings in more false positives, while a lower threshold leads to more missed detections. The repulsion loss consists of two parts: the attraction term to narrow the gap between a proposal and its designated target, and the repulsion term to distance it from the surrounding non-target objects. Then, Shifeng and Longyin [44] propose a new aggregation loss function. The function enforces proposals to be close and locate compactly to the corresponding objects. At the same time, a new part occlusion-aware region of interest (PORoI) is proposed to replace the original RoI. PORoI can integrate the prior structure information of the human body with visibility prediction into the network to handle occlusion. Then, Cao, JL proposed location bootstrap and semantic transition, which is used to reweight regression loss and adds more contextual information and relieves semantic inconsistency of the skip-layer fusion. Sumi [45] proposed Frame-Level Difference (FLD) features, which will extract the features by finding the difference between the adjacent frame and retaining the noticeable differences. Using a combination of proposed features with other existing algorithms can improve the occluded pedestrian detection accuracy. Wei [46] proposed an occluded pedestrian detection method based on binocular vision. The Binocular introduced visual salience prior information, which solves the problem of occlusion.

Recurrent Neural Network (RNN) takes sequence data as input and recursively in the evolutional direction of the sequence with all nodes are linked by a chain. Bidirectional RNN (Bi-RNN) and Long Short-Term Memory networks (LSTM) are common Recurrent Neural Network.

Stewart and Andriluka propose a model that is based on decoding an image into a set of people detections in crowded scenes, as it is shown in Fig. 8 [47]. A recurrent LSTM layer is used for sequence generation to train the model end-to-end with a new loss function that operates on sets of detections.

The MIT pedestrian database (MIT-CBCL Pedestrian Database) was created by the Massachusetts Institute of Technology. It contains 924 Pedestrian images (in PPM format with a width and height of 64 × 128). The images in the database contain both front and back perspectives. The images of USC Pedestrian Set are mostly from surveillance video, including three sets of data sets USC-A, USC-B, and USC-C. Daimler Pedestrian Detection Benchmark include grayscale images. The database contains many images of occluded pedestrians. Caltech pedestrian database is a large-scale pedestrian database that has a relatively consistent pedestrian occlusion image with the actual situation in life. INRIA pedestrian database is the most widely used static pedestrian detection database having a clear picture. It has corresponding labeling files that are divided into a training set, test set, positive and negative samples. CUHK Occlusion Dataset, published by The Chinese University of Hong Kong, contains 1063 images of people. It has a large number of occluded pedestrian images. In addition, CUHK can release the “Person Re-identification Datasets” and “Square Dataset.” CUHK-PRe-D recorded 971 pedestrian samples from different perspectives. The CVC pedestrian database contains three subsets of cvc-01, cvc-02, and cvc-virtual, with each subset serving different tasks. The NICTA pedestrian database is a large static image pedestrian database, which is divided into a test set and a training set, containing 25,551 single person images and 5207 high-resolution non-pedestrian images. The images provided by the TUD pedestrian database are mainly convenient for calculating optical flow information. These databases are shown in Table 2, which are often used in pedestrian detection and tracking research.