Efficient contour-based annotation by iterative deep learning for organ segmentation from volumetric medical images

Nowadays, deep learning (DL) demonstrated promising performance for medical image analysis, and deep neural networks became the mainstream method for organ segmentation from medical images. So far, the training of an effective deep segmentation network still requires the annotation of large datasets. It is well known that manual annotation of volumetric medical images is extremely tedious and prone to subjective variabilities. Although some recent efforts have been made on few-shot or unsupervised learning, these methods are still not generalizable enough for different imaging modalities or target organs. Human supervision is still indispensable for most cases of deep segmentation network training. Therefore, efficient network training and human annotation methodologies are needed to relieve the burden of data annotation.

To tackle this problem, the strategy of annotation by iterative deep learning (AID) [1] becomes popular in the research community. In a typical AID workflow, a segmentation network is preliminarily trained via a small number of manually annotated data. Then, the preliminarily trained network is used for automatically annotating more training data. Since the preliminary trained network is not accurate enough, human supervisors are involved to proofread the automatic annotation results, and the human-corrected annotations are added to the training set to retrain a more accurate network. As this procedure is iteratively repeated, the network performance is gradually improved, and thus less-and-less user proofreading is needed.

The efficiency of an AID workflow mainly depends on two factors, i.e. the learning speed of the network model and the convenience of the proofreading method. On the one hand, a network with good learning ability can quickly approach satisfactory performance with a small number of training data. On the other hand, a convenient human-intervention method is essential for saving the interaction time and reducing the inter-operator variability. By far, many efforts have been made to improve the network learning ability. Methods based on few-shot [2‐4] or semi-supervised learning [5‐8] were proposed to train a segmentation network with limited training data. To alleviate the high annotation cost, weakly supervision approaches [9‐11] were developed to complement the pixel-level full supervision. As for human-proofreading, there are already several publicly available medical image processing software with manual annotation main tools, such as ITK-SNAP [12], MITK [13], 3D Slicer [14], TurgleSeg [15] and Seg3D [16]. Most of them provided interaction functions including pixel painting, contour interpolation [13, 15, 17, 18], interactive level sets [19], surface adjustment [20], super-pixel [21] and super-voxel [22] modification. However, these tools were developed for general segmentation purposes; none of them was dedicatedly designed for efficient correction of neural network outputs. Recently, some specialized proofreading methods based on convolutional neural networks (CNNs) were proposed to involve human intervention into DL-based segmentation pipelines, such as DeepIGeoS [23], IFSeg [24] and BIFSeg [25]. These methods receive the user’s proofreading along with the medical image as the network inputs. For 3D segmentation networks, the speed of network inference cannot reach real-time output, limiting their applicability for efficient proofreading.

In this study, we further accelerate of AID workflow for organ segmentation from volumetric medical images. Our key idea is to use contour-based representation instead of voxel-label representation to improve the network learning ability and the user-interaction efficiency. By using contour representations, shape prior of the target organ is incorporated into the learning process, achieving anatomically plausible segmentation with only a few training images. Our user-interaction method also takes advantage of the contour-representation to facilitate real-time expert proofreading. As shown in Fig. 1, expert-proofreading of organ contours can be achieved via simple mouse interaction (e.g. dragging), while conventional voxel-label-based representation requires multiple manual editing tools such as pens, brushes and erasers. Our method is dedicated to the segmentation of human organs with strong shape characteristics. The source code and the proposed AID model are integrated into our open-source software AnatomySketch for the research community to use (https://​github.​com/​DlutMedimgGroup/​AnatomySketch-Software/​tree/​master/​AID).

Our contour-based AID method is composed of an organ contour prediction network and an organ boundary adjustment module (as illustrated in Fig. 2). The learning loop starts from only a few expert-labelled preliminary training images. We convert the voxel labels into boundary contours to better represent the organ shape. Although the number of the preliminary training image is small (usually less than ten), our network can learn strong organ shape knowledge from the contour representation. The preliminarily learned shape knowledge helps the network to yield anatomically plausible segmentation of more training images, and the human expert quickly proofread the segmentation results using our boundary adjustment method. Then, the proofreading results are added to the training set and the network is retrained to achieve better prediction accuracy. The key idea of our method is to use contour representation instead of conventional label representation to facilitate better shape learning and faster expert proofreading. Details of the contour prediction network and the boundary adjustment method are described in the following sections.

For validation, our method is compared with two approaches widely adopted by the research community, i.e. manual annotation assisted by inter-slice interpolation and the AID method based on voxel-label representation.

The first method is mostly used for segmentation network training and is implemented in the MITK software [13]. With this method, the user selectively annotated the organ contours in a few parallel slices and then performed contour interpolation to obtain the contours in all slices. This method is similar to the selective contour annotation step of our method except that the contours are manually created. We compare our method with this method to evaluate the influence of the contour-learning network on inter-operator consistency and proofreading efficiency. For both methods, the number of selected slices was determined by the human operator to ensure accurate annotation.

The second method follows the similar AID workflow of our method. The main difference is that it uses the conventional voxel-label prediction model for automated segmentation. The prediction model is a Dense V-net [35], and the loss function is the weighted sum of an L2 regularization loss with label smoothed probabilistic Dice scores for each organ. We choose this method for comparison to prove the advantage of contour-based representation on shape learning and human proofreading. For a fair comparison, we used the same human-proofreading scheme (i.e. our contour correction method) for both methods.

The comparison was made from the aspects of annotation accuracy, network learning ability, inter-rater stability and proofreading efficiency based on a multi-institute abdominal CT image dataset. The dataset includes 20 CT volumes with pixel sizes between 0.72 and 1.37 mm and slice spacing between 1.60 and 3.00 mm, consisting of four images from the online 3D-irCadb database¹ and 16 retrospective images from four of our collaborated hospitals across the country. The images were acquired with 28–298 mA tube current and 100–140 kV tube potential. All the images had corresponding label images of the liver, spleen and kidneys annotated by an experienced human expert as the ground truth.

Two well-trained human operators were invited to independently use the three methods to annotate the 20 volumetric images slice by slice. For each operator, the Dice scores and the averaged symmetric surface distances (ASDs) were computed using the organ labels of the experienced expert as the ground truth. When comparing different methods, we averaged the Dice scores and ASDs of the two operators for each image and used the averaged value for method-wise comparison. When evaluating inter-operator variabilities, we ignored the labels of the experienced expert and computed the mutual Dice and ASD between the two operators to measure their discrepancies.

For the validation of the two AID-type methods (i.e. our method and the voxel-label-based AID), a fourfold cross-validation strategy was used. The 20 CT volumes were randomly divided into fourfolds. Each time onefold was used as the test images and the other threefolds were used for network training. From the threefolds of training data, onefold was randomly selected to train a preliminary network which was then used to automatically annotate the other twofolds of training data. From each training volume image, all the 2D slices were used for training the contour-based network. The human operators proofread the annotations of the preliminary network and retrained the network using the threefolds of training data. Our method was trained end-to-end for 300 epochs. The learning rate started from 1e−4 and decayed by half at 80, 120, 150 and 170 epochs. The Dense V-net was trained with a learning rate of 0.01 and a batch size of 6 for 200 epochs.

For visual inspection, Fig. 5a shows the automated annotation results of the two AID-based methods in two representative slices. For both methods, the retrained network yielded more accurate results than the preliminary network. It is encouraging to see that the results of our preliminary network maintain plausible organ shape although the contours do not perfectly fit the true organ boundary. In contrast, the preliminarily trained voxel-label-based network produces erroneous contour shapes due to insufficient training data. Figure 5b displays the segmentation results in 2D slices of different vertical levels, including the middle, transition and periphery slices of the organs.

As a quantitative comparison of the annotation accuracy, Fig. 6 shows the box plots of Dice and ASD of the three methods. For the traditional method, Dice and ASD were calculated for the contour interpolation results. For the two AID-based methods, Dice and ASD were calculated for the automatic annotation results of the retrained network. The traditional contour interpolation achieved the highest median Dice (> 0.88) and lowest median ASD (< 2.0 mm) for all the organs. This is because the annotations of the traditional method were almost generated manually. In contrast, the annotations of the two AID-based methods were generated by the network model; therefore, were less accurate than the traditional method. Nevertheless, we can see that our method obtained a comparable accuracy (median Dice > 0.85 and median ASD < 2.5 mm) to the traditional methods with only two iterations. The voxel-label-based AID method was the least accurate. The boxes of this method were also the widest among the three, implying imperfect robustness against different test images.

The learning ability comparison was made between our method and the voxel-label-based AID method. Figure 7 shows the mean and standard deviation values of Dice and ASD before and after the retraining. After the retraining, accuracy improvements (i.e. higher mean Dice with smaller standard deviations) are observed for both methods. Our method was more accurate than the voxel-label-based AID method either before or after the retraining. For the kidneys and the spleen, our preliminary network is even more accurate than the retrained Dense V-net, thanks to the shape knowledge learned from the very small preliminary training dataset.

To compare the inter-rater variability, the mutual Dice and ASD between the two operators were calculated and are plotted in Fig. 8. Our method demonstrates the highest median Dice (> 0.93) among the three methods, meaning the best inter-operator agreement for the annotated organ regions. The inter-operator ASDs of our method was not the lowest but almost approached the most accurate one (the traditional method). The traditional method is the most accurate in terms of the contour distance (ASD) because its contours were manually sketched by the human operators. Nevertheless, the boxes of our method are much narrower than the other two methods, meaning that our method is superior in inter-rater consistency since our contour-correction method reduces the amount of subjective interaction.

To evaluate the annotation efficiency, the operators recorded the interaction time required for proofreading the automatic annotation results of the preliminary network. The proofreading time per image per iteration was 16.19 ± 2.58 min for our method and 29.79 ± 3.50 min for the voxel-label-based AID method. Our method requires much less interaction time because our preliminary network is more accurate and reasonable than Dense V-net. Figure 9 shows the multi-iteration Dice increment curves with increasing proofreading time. As the proofreading time increases, the performance of the network continues to get better. For most organs, the Dice score reaches 0.9 within 20 min.

To further validate the few-sample learning ability, we compared our contour prediction network with the state-of-the-art (SOTA) nnU-Net [36] based on three additional datasets. The first dataset is the open-source Chest X-ray Masks and Labels [37, 38], containing 704 chest X-ray images with lung area annotations. The second dataset includes 88 cardiac ultrasound images from a local hospital with myocardium annotations by experienced physicians. The third dataset is the Lung Tumours dataset of the medical segmentation decathlon [39] and contains 64 CT volumes with tumour area annotations.

For the first two 2D datasets, we randomly selected a small number of 2D training slices (21 and 66 images for the X-ray and ultrasound datasets, respectively) and validated the trained model on the rest of the datasets as testing images. Due to the limited number of training data, both methods produced minor segmentation errors in some test images, as shown in Fig. 10. However, nnU-net tends to generate topological errors (pointed by red arrows) which needs tedious manual correction. In contrast, our approach consistently yields anatomically plausible shapes which are easy to correct with our contour dragging tool.

As a quantitative comparison, box plots of Dice scores of both networks are shown in Fig. 11. We also added another compared network which used the same architecture of ours but replaced the multi-scale feature (MSF) extraction backbone with the conventional CenterNet backbone (named ‘ours w/o MSF’). For both datasets, our method achieved a median Dice close to that of nnU-Net, with the guarantee of correct topology. The boxes of our method were also narrower than that of nnU-Net, implying better robustness against different test images. Compared to ‘our network w/o multi-scale feature’, our network also achieved more accurate and more robust results, proving the advantage of the multi-scale feature extraction backbone. Besides, the inference time of our method is much shorter than nnU-Net (both using one NVIDIA RTX 3090 GPU). On the chest X-ray dataset, the inference time per image was 26.32 ± 6.22 ms for our method and 4698.51 ± 1124.65 ms for nnU-Net. On the cardiac ultrasound dataset, the inference time per image for our method and nnU-Net was 19.93 ± 1.26 ms and 181.63 ± 11.62 ms, respectively. The reasons for the significant time advantage of our network are attributed to the simpler network structure and smaller parameter size, as well as the avoidance of using sliding convolution kernel and multiple mirror inference. Faster inference helps to reduce device dependency and is more beneficial for real-time applications such as cardiac ultrasound detection.

We further validated the 3D segmentation performance of the networks with the lung tumour segmentation dataset [39]. Compared to 2D segmentation networks, the 3D segmentation network faces more severe challenges of larger network parameter size and less training data. Because our method is based on contour representation, it is more light-weight than the voxel-based CNNs; thus, it suffers less from the problem of large parameter vs. small training set. Moreover, our method is able to learn plausible contour shape from small training set. Without the shape prior, nnU-Net may yield unsatisfactory segmentation with insufficient training dataset. In addition, our method is also compared with the nnU-Net with boundary loss [40] for segmenting small objects. Different from our contour-based learning method, this boundary loss approximates the differential boundary variation using an integral approach, which avoids completely local differential computations involving contour points and represents boundary change as a regional integral. In this way, the boundary loss is easily combined with the standard regional losses, such as Dice or cross-entropy. The boundary loss improves the performance for highly unbalanced segmentation tasks in which the voxel number differs by several orders of magnitude across classes. In the experiment, a fourfold cross-validation strategy was used. The 64 volumetric test images were evenly divided into fourfolds. Each time onefold (16 images) was used as the training images, and the other threefolds (48 images) were used for testing to simulate the small training data situation. The nnU-Net with boundary loss is trained following the training method in [40].

Figure 12 shows two representative results of the comparison experiment. With insufficient training data, nnU-Net yielded unsmooth tumour boundary for example 1 and false positive segmentation of the sporadic vessels for example 2. With the boundary loss, the boundary smoothness was improved, but the false positives still exist. Our method generates smooth segmentation and avoids the false positives of implausible tumour shape, thanks to the ability of shape learning from limited training data.

To evaluate the efficiency of the contour-based boundary adjustment, we compared it with two deep-learning-based interactive segmentation mothed, i.e. MIDeepSeg [41] and DeepGrow [24]. Both methods learn the way to react to user-input seed points, while MIDeepSeg is trained with images of multiple modalities and target objects for generic segmentation tasks, and DeepGrow needs to be specifically trained for the target modality and object. For MIDeepSeg, we used the implementation provided by authors. For the DeepGrow model, we used a model trained for abdominal CT images.² We chose a difficult task of segmenting tumour-contaminated liver. As shown in Fig. 13, although the two compared models were well trained for the interaction scenario, they need many user inputs at the difficult regions (e.g. big tumour region and fuzzy liver boundary) and finally did not converge perfectly. This is mainly because that tumour appearance is highly variable between patients, thus these methods perform unsatisfactorily for unseen tumours. In contrast, our method only needs two simple drags because the contour network learned strong shape prior of the liver. Of course, our method is mainly developed for the objects with shape priors, while the two compared methods are not restricted to the segmentation of regular shape.

As analysed in the Introduction section, the efficiency of an AID workflow is directly affected by the learning ability of the DL model. We focus on developing a DL model which only needs a few training images and a few AID iterations to achieve satisfactory annotation accuracy. Experimental results show that our contour prediction network achieved a mean Dice over 0.86 for the liver and kidneys using only five preliminary training images. Such a level of accuracy is even better than the retrained Dense V-net. After retraining our network with 10 more images, the median Dice scores of the liver and kidneys were above 0.92, and the median ASDs of the kidneys and spleen drop below 1.5 mm. Thanks to the good learning ability from limited training data, the human operators only need to make very few corrections in the proofreading step; therefore, our method requires much less proofreading time than the voxel-label-based AID method. A human operator using our method only took ~ 16 min for each training image, i.e. ~ 4 min per organ.

The good learning ability of our method is attributed to the use of contour-based shape representation instead of the voxel-label representation. It should be mentioned that boundary information has been used in recent segmentation networks, such as the boundary loss for improving small object segmentation [40]. Different from the loss-based methods, our method represents boundaries with contour points, facilitating strong shape memory and efficient boundary proofreading. The combination of contour point coordinates into the vertex feature vector implants the memory of contour shape into the network, helping the network to produce plausible contour shape even when the training images are not abundant. The short proofreading time of our method is attributed to the contour-based adjustment method. Since the automatically generated contours are already close to the true boundaries, the efforts required for contour adjustment are much reduced. Moreover, our interactive proofreading method facilitates convenient contour correction via simple mouse dragging, saving considerable interaction time as compared to the conventional voxel-label correction.

Another advantage of our method is low inter-operator variability. As reflected from the mutual Dice and ASD between the two human operators, our method demonstrates good agreements between the two operators. This is because the annotation was generated by the retrained network rather than pure manual sketching. Because our network already had reasonable accuracy after preliminary training, the human operators need not perform too much proofreading; thus, the subjective influence on proofreading was reduced.

We noticed that both our method and the voxel-label-based AID method have different levels of annotation accuracy for different organs. The spleen usually has lower Dice than the other organs, and the liver tends to have higher ASD. This is because both the liver and spleen have complex shapes and fuzzy boundaries with the adjacent soft tissues. However, the traditional contour interpolation method does not have such obvious inter-organ differences, because the human operators have full control of the annotation accuracy. Therefore, we remind the readers that although the AID-based methods save the annotation time, they may have variable accuracy for different organs.

The experimental results show that the voxel-label-based AID method needs more training data than our method to achieve satisfactory accuracy. However, voxel-label and boundary contour are like two sides of the coin. Although contour representation benefits the segmentation of regularly shaped objects (e.g. human organs), voxel-label may be more suitable for segmenting objects with complex or irregular shapes such as tumours. It should also be noted that the accuracy of our contour-based method may be constrained by the shape variability of the training set, yielding unsatisfactory segmentation of the organ shapes inexistent in the training data (e.g. lesion-distorted organ shape). Nevertheless, thanks to the FFD-based contour adjustment method, such inaccurate segmentation can be conveniently corrected with minor user interaction. For future work, we will make efforts on combining the advantages of contour-based on voxel-label-based method, as well as on improving the shape learning generalizability based on limited training samples.

This paper introduces an iterative annotation and proofreading workflow to reduce the annotation workload for organ segmentation network training. By integrating a contour generation network and a convenient contour adjustment approach, organ annotation using our framework becomes more efficient, more accurate and less prone to inter-operator variability than using the voxel-label-based AID method and the traditional contour interpolation method. We have implemented this method as an extension model to our open-source software AnatomySketch.³ Our future work will explore better shape representation to improve the learning ability and generalizability for more complex objects such as tumours and vessels.