Recent progress in semantic image segmentation

At present, there are many general datasets related to image segmentation, such as, PASCAL VOC (Everingham et al. 2010), MS COCO (Lin et al. 2014), ADE20K (Zhou et al. 2017), especially in autonomous driving area Cityscapes (Cordts et al. 2016), and KITTI (Fritsch et al. 2013; Menze and Geiger 2015).

The PASCAL Visual Object Classes (VOC) Challenge (Everingham et al. 2010) consists of two components: (1) dataset of images and annotation made publicly available; (2) an annual workshop and competition. The main challenges have run each year since 2005. Until 2012, the challenge contains 20 classes. The train/val data has 11,530 images containing 27,450 ROI annotated objects and 6929 segmentations. In addition, the dataset has been widely used in image segmentations.

Microsoft COCO dataset (Lin et al. 2014) contains photos of 91 objects types which would be recognized easily by a 4-year-old person with a total of 2.5 million labeled instances in 328k images. They also present a detailed statistical analysis of the dataset in comparison to PASCAL (Everingham et al. 2010), ImageNet (Deng et al. 2009), and SUN (Xiao et al. 2010).

ADE20K (Zhou et al. 2017) is another scene parsing benchmark with 150 objects and stuff classes. Unlike other datasets, ADE20K includes object segmentation mask and parts segmentation mask. Also, there are a few images with segmentation showing parts of the heads (e.g. mouth, eyes, and nose). There are exactly 20,210 images in the training set, 2000 images in the validation set, and 3000 images in the testing set (Zhou et al. 2017). A group of images are shown in Fig. 1.

The Cityscapes Dataset (Cordts et al. 2016) is a benchmark which focuses on semantic understanding of urban street scenes. It consists of 30 classes in 5000 fine annotated images that are collected from 50 cities. Besides, the collection time spans over several months, which covers season of spring, summer, and fall. A fine-annotated image is shown in Fig. 2.

KITTI dataset (Fritsch et al. 2013; Menze and Geiger 2015), as another dataset for autonomous driving, captured by driving around mid-size city of Karlsruhe, on highways, and in rural areas. Averagely, in every image, up to 15 cars and 30 pedestrians are visible. The main tasks of this dataset are road detection, stereo reconstruction, optical flow, visual odometry, 3D object detection, and 3D tracking (http://​www.​cvlibs.​net/​datasets/​kitti/​).

In addition to the above datasets, there are also many others, such as SUN (Xiao et al. 2010), Shadow detection/Texture segmentation vision dataset (https://​zenodo.​org/​record/​59019#.​WWHm3oSGNeM), Berkeley segmentation dataset (Martin and Fowlkes 2017), and LabelMe images database (Russell et al. 2008). More details about the dataset can refer to http://​homepages.​inf.​ed.​ac.​uk/​rbf/​CVonline/​Imagedbase.​htm.

Regular performance evaluation metrics for image segmentation and scene parsing include: pixel accuracy \(P_{acc}\), mean accuracy \(M_{acc}\), region intersection upon union (IU) \(M_{IU}\), and frequency weighted IU \(FW_{IU}\). Let \(n_{ij}\) indicates the number of pixels of class i predicted correctly to belong to class j, where there are \(n_{cl}\) different classes, and let \(t_i = \sum _j n_{ij}\) indicates the number of pixels of class i. All of the four metrics are described as below (Long et al. 2014):

The paper (Long et al. 2014) is the first work that introduces ANNFCN to image segmentation area. The main insight is the replacement of fully connected layer by fully convolutional layer. With the use of the interpolation layer, it realizes that the size of output is the same as the input, which is essential in segmentation. To enhance the segmentation evidence, skips is adopted. More importantly, the network is trained end to end, takes arbitrary size, and produces correspondingly-sized output with efficient inference and learning.

FCN is implemented in VGG-Net and achieves the state of art on segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU in 2012) at that time, while the inference takes less than one fifth of a second for a typical image. The main architecture is shown in Fig. 5.

In addition to the FCN architecture, deconvolution layer is also adopted in semantic segmentation. The deconvolution network used in Noh et al. (2015) consists of deconvolution and un-pooling layers, which identify pixel-wise class labels and predict segmentation masks. Unlike FCN in paper (Noh et al. 2015), the network is applied to individual object proposals so as to obtain instance-wise segmentations combined for the final semantic segmentation.

Up-sample stage adopts bi-linear interpolation, which can refer to Long et al. (2014). Due to its computation efficiency and good recovery of the original image, the up-sample stage adopts bi-linear interpolation broadly. Deconvolution is the reverse calculation of convolution operation, which can also recover the input size. Thus, it can be applied into segmentation to recover the feature map size to original input size. The architecture implemented in Noh et al. (2015) is illustrated in Fig. 6. Also, other researchers implement semantic segmentation by deconvolution layer in different versions, which can refer to Mohan (2014), Monvel et al. (2003), Saito et al. (2016).

According to the research of Deeplab, the responses at the final layer of Deep Convolutional Neural Networks (DCNNs) are not sufficiently localized for accurate object segmentation (Chen et al. 2016b). They overcome this poor localization property by combining a fully connected Conditional Random Field (CRF) at the final DCNN layer. Their method reaches 71.6% IOU accuracy in the test set at the PASCAL VOC-2012 image semantic segmentation task. After this work, they carry out another segmentation architecture by combining domain transform (DT) with DCNN (Chen et al. 2016a) because dense CRF inference is computationally expensive. DT refers to a modern edge-preserving filtering method, in which the amount of smoothing is controlled by a reference edge map. Domain transform filtering is several times faster than dense CRF inference. Lastly, through experiments, it not only yields comparable semantic segmentation results but also accurately captures the object boundaries. Researchers also exploit segmentation by using super-pixels (Mostajabi et al. 2015; Sharma et al. 2015).

Paper (Liu et al. 2015) addresses image semantic segmentation by combining rich information into Markov Random Field (MRF), including mixture of label contexts and high-order relations (Figs. 7, 8, 9).

Most semantic segmentations are based on the adaptations of Convolutional Neural Networks (CNNs) that had originally been devised for image classification task. However, dense prediction, such as image semantic segmentation tasks, is structurally different from classification.

Paper (Chen et al. 2016b) has already applied this strategy in their work. It is called ‘Atrous Convolution’ or ‘Hole Convolution (Chen et al. 2016b)’ or ‘dilated convolution (Yu and Koltun 2015)’ . Atrous convolution is originally developed for the efficient computation of the undecimated wavelet transform in the “algorithme à trous” scheme of paper (Holschneider et al. 1989). In Yu and Koltun (2015), they have presented a module using dilated convolutions to aggregate multi-scale contextual information systematically. The architecture is based on dilated convolutions that support exponential receptive field expansion without loss of resolution or coverage. Since the dilated convolution has griding artifacts, paper (Yu et al. 2017) develops an approach named dilated residual networks (DRN) to remove these artifacts and further increase the performance of the network.

The backbone network refers to the main structure of the network. As is known to all, the backbone used in semantic segmentation is derived from image classification tasks. The FCN (Long et al. 2014) adopts VGG-16 net (Simonyan and Zisserman 2014) which did exceptionally well in ILSVRC14. Also, they consider AlexNet architecture (Krizhevsky et al. 2012) that won ILSVRC12 as well as GoogLeNet (Szegedy et al. 2015) that also did well in ILSVRC14. VGG net is adopted in many literatures, such as in Chen et al. (2016b) Liu et al. (2015).

After the release of ResNet (Deep residual network) (He et al. 2016) which Deeplab implement their work on which won the first place on the ILSVRC 2015 classification task, the semantic segmentation has made a new breakthrough. To find out the best configuration, paper (Wu et al. 2016a) evaluates different variations of a fully convolutional residual network, including the resolution of feature maps, the number of layers, and the size of field-of-view. Furthermore, paper (Wu et al. 2016b) studies the deep residual networks and explains some behaviors that have been observed experimentally. As a result, they derive a shallower architecture of residual network which significantly outperforms much deeper models on the ImageNet classification dataset.

Recently, ResNeXt (Xie et al. 2016) have been brought up as the next generation of ResNet. It is the foundation of our entry to the ILSVRC 2016 classification task in which we secured the 2nd place. GoogleNet also obtains development as Inception-v2, Inception-v3 (Szegedy et al. 2016), Incetion-v4 and Inception-ResNet (Szegedy et al. 2017), which has already been adopted in the paper (Li et al. 2017b).

Apart from adopting stronger backbone networks, researchers also attempt to combine pyramid strategy to CNN. The typical one is pyramid method.

1. Image pyramid

An image pyramid (Adelson et al. 1984) is a collection of images which are successively downsampled until some desired stopping criteria are reached. There are two common kinds of image pyramids: Gaussian pyramid which is used to downsample images and Laplacian pyramid which is used to reconstruct an upsampled image from an image lower in the pyramid (with less resolution).

In semantic image segmentation area, paper (Lin et al. 2016a) devises a network with traditional multi-scale image input and sliding pyramid pooling that can effectively improve the performance. This architecture captures the patch-background context. Similarly, Deeplab implements an image pyramid structure (Chen et al. 2016c) which extracts multi-scale features by feeding multiple resized input images to a shared deep network. At the end of each deep network, the resulting features are merged for pixel-wise classification.

Laplacian pyramid is also utilized in semantic image segmentation which can refer to paper (Ghiasi and Fowlkes 2016). They bring out a multi-resolution reconstruction architecture based on a Laplacian pyramid, which uses skip connections from higher-resolution feature maps and multiplicative gating to progressively refine boundaries reconstructed from lower-resolution feature maps. Paper (Farabet et al. 2013) presents a scene parsing system. The raw input image is transformed through a Laplacian pyramid. Meanwhile, each scale is fed to a two-stage CNN that produces a set of feature maps.

2. Atrous spatial pyramid pooling

Inspired by image pyramid strategy, (Chen et al. 2016b) proposes Atrous Spatial Pyramid Pooling (ASPP) to segment objects robustly at multiple scales. ASPP probes effective fields-of-views (FOV) and convolutional feature layer with filters at multiple sampling rates, and then captures objects image context at multiple scales. The architecture is shown in Fig. 10.

3. Pooling pyramid

Through pyramid pooling module illustrated in Fig. 11, paper (Zhao et al. 2016) exploits the capability of global context information by different-region based context aggregation and names their pyramid scene parsing network (PSPNet). Through experiments they report their outstanding results: with pyramid pooling, a single PSPNet yields new record of mIoU score as 85.4% on PASCAL VOC 2012 and 80.2% on Cityscapes.

The pyramid pooling adopts different scales of pooling size, then does up-sample process on the outputs to the original size, and finally concatenates the results to form a mixed feature representation. In Fig. 11, different scales of pooling sizes are marked with different colors. Generally speaking, the pyramid pooling can be applied to any feature map. For example, the application in Zhao et al. (2016) applies pyramid pooling in pool5 layer.

4. Feature pyramid

As pointed out by literature (Lin et al. 2016b), feature pyramid is a basic component in image tasks for detecting objects at different scales. In fact, recent deep learning object detectors have avoided pyramid representation because it is compute and memory intensive. In Lin et al. (2016b), they exploit the multi-scale, pyramidal hierarchy of CNN to construct feature pyramids with marginal extra cost. Also, Feature Pyramid Network (FPN) is developed for building high-level semantic feature maps at all scales.

CNN can be treated as a feature extractor (Hariharan et al. 2015). Typically speaking, recognition algorithms based on convolutional networks (CNNs) use the output of the last layer as a feature representation. However, the information in this layer is too coarse for dense prediction. On the contrary, earlier layers may be precise in localization, but they will not capture semantics. To get the best of both advantages, they define the hypercolumns as the vector of activations of all CNN units above that pixel.

Indeed, skips have already been adopted in FCN (Long et al. 2014) which is depicted in Fig. 5. It seems that the multi-level method has been used in their work.

Multi-model is an ensemble way to deal with image tasks (Li et al. 2015; Viola and Jones 2001). Apart from multi-level strategy, a multi-stage method is used in semantic segmentation (Li et al. 2017b). They propose deep layer cascade (LC) method to improve the accuracy and speed of semantic segmentation. Unlike the conventional model cascade (MC) (Li et al. 2015; Viola and Jones 2001) that consists of multiple independent models. LC treats a single deep model as a cascade of several sub-models and classifies most of the easy regions in the shallow stage and makes deeper stage focus on a few hard regions. It not only improves the segmentation performance but also accelerates both training and testing of deep network (Fig. 12).

Most of the progress in semantic image segmentation are done under supervised scheme. However, researchers also dedicate to semi-supervised or non-supervised learning. More details can refer to Papandreou et al. (2015), Xia et al. (2013), Zhu et al. (2014), Xu et al. (2015).