Segmentation of magnetic resonance images using a combination of neural networks and active contour models
Introduction
Segmentation has been defined [1:p. 347] as the process of: “dividing the image into regions that … correspond to structural units in the scene or distinguish objects of interest”. It is a necessary first step in the visualisation and interpretation of many complex images, such as those typically encountered in medical imaging. In this area, fully automatic and robust segmentation techniques would have an enormous beneficial impact on clinical practice and research, by decreasing dramatically the manual effort which must otherwise be devoted to this task. Not only are medical images themselves inherently complex, but acquisition must also recognise practical needs to limit radiation dose, scan time, etc., so that image quality is often compromised. Given this, deployment of conventional image-processing techniques has not so far led to a robust fully automatic solution usable in a range of clinical settings, although semi-automatic systems do exist.
Semi-automatic segmentation has been used extensively in nuclear medicine based on thresholding or gradient techniques: both two- and three-dimensional techniques have been described [2]. Medical image-processing systems such as ANALYZE (from the Mayo Foundation, Rochester, MN) also include tools for partially automating manual segmentation. Fully automatic segmentation is possible in specific instances, such as thresholding to identify bone in computer tomography (CT) images [3]. Various new approaches look to have considerable potential for automatic segmentation in more general applications (e.g. [4], [5]) although their use in clinical practice has yet to be proven. Thus, segmentation remains the “image-processing bottleneck” [6].
The method proposed here is developed and illustrated on the practical problem of segmenting lung outlines from magnetic resonance (MR) images of the thorax. It consists of two stages. First, a neural network (multilayer perceptron, MLP) trained in supervised fashion is used to classify each pixel of the MR image into boundary and non-boundary classes, so producing a binary, edge-point image. Second, to compensate for classification errors, the edge-point images are then post-processed using an active contour model, or ‘snake’ [7], [8]. In this way, the edge-point image acts as the external energy function for the snake. A similar combination of MLP classifier and active contour model has previously been used to locate the interior contour of the brain from MR images of the head [9]. However, the initial classification achieved by the neural network in that work was relatively poor and required a rather complex model-based active contour technique (using a stochastic decision mechanism based on a Gibbs sampler) to extract the final boundary. In this work, we aim to produce a sufficiently good initial classification to be able to use a simple and standard snake as the post-processor.
Early results for the classification stage, using data from a single subject and a restricted number of slices, were reported by Middleton and Damper [10], and showed good segmentation of the lung boundaries in a given MR image of the torso. Unfortunately, however, generalisation to other (unseen) slices and subjects was very much worse. We have since shown that an elastic net [11] modified to give robustness against initial classification errors, can be used to extract the region of the lungs very effectively from some of these classifications [12]. However, many of the results from the classification stage were too poor for the lungs to be accurately identified in this way. In the present work, several modifications and improvements have been made to our earlier work, which allow a much more accurate classification to be achieved from unseen MR data from different slices and different subjects. In particular, a novel cost function is proposed that simplifies the process of selecting the training data. Further, automatic exclusion of some pixels from the training set leads to dramatic improvement in classification. Consequently, the lungs can now be successfully segmented from the vast majority of available images using a standard active contour model, for which purpose we use the Cohen snake [13].
The remainder of this paper is structured as follows. The next two sections describe the images used (Section 2) and review alternative segmentation techniques (Section 3). The purpose of the latter section is to illustrate the difficulties in segmenting MR images using conventional image-processing techniques and to motivate the use of a neural network. The initial approach to classification using an MLP is then described in Section 4. We then detail our method of quantifying the quality of the segmentation (Section 5). Section 6 presents preliminary results of MLP classification using the standard squared-error cost function in training the network, and details the steps that were found necessary to achieve reasonable results. In Section 7, we define a new cost function for training based on the measure used to quantify segmentation performance. Section 8 describes MLP training using the new cost function, and Section 9 presents classification results using this new function. Post-processing using the snake and the results of the final segmentation are given in Section 10, and Section 11 concludes.
Section snippets
Image data and labelling
MR imaging is a non-invasive method of acquiring precise anatomical information in the form of three-dimensional data sets (see [14], [15] for good introductory treatments). The data used here consist of transverse slices of the thorax obtained from a 0.5 T MR machine from 13 subjects. All were healthy volunteers, identified hereafter by a two letter abbreviation of the subject’s name. Fig. 1 shows two examples of slices from subject AC. The lungs are clearly visible in these images as two
Alternative segmentation techniques
Difficulties such as those described above mean that standard image-processing techniques are often unable to segment MR images satisfactorily (unlike some other imaging modalities). For example, it is well documented that (unlike CT images) MR images cannot be segmented using histogram-based thresholding because of the non-uniform nature of the data [18], [25]. To justify the approach taken here, various other standard image-processing techniques have been investigated for MR image
Initial classification using a neural network
Classification of each pixel of the image as either a boundary or non-boundary edge-point uses a multi-layer perceptron (MLP) trained on error back-propagation [30], [31]. Since back-propagation is a supervised, gradient-descent technique, it requires labelled training data and some cost function which is differentiable, to give gradient information used in the search for a minimum-cost configuration of network connection weights. Initially, we have used the standard squared-error cost
Quantifying segmentation performance
To assess results, a method of measuring the accuracy of the segmentation techniques is required. This is a common problem in medical image segmentation [18]. Visual inspection is sometimes used to evaluate performance “since ‘perfect’ segmentations cannot be defined” [36:p. 341]. For instance, Brown et al. [37] assess the quality of chest CT segmentations through visual inspection by an experienced thoracic radiologist. Also, in the work of Chiou and Hwang [9], results are assessed
Results using squared-error cost function
Initially, the MLP was trained to identify the lung boundaries on a single slice (S=1) of an MR image of the torso using the squared-error cost function (1). This showed that the method of selecting the training data as described in Section 4.2, using all lung-boundary examples plus an equal number of randomly selected non-boundary examples, led to a very poor classification [10]. In classifying the interior contour of the brain with a similar MLP to that used here, Chiou and Hwang [9] also
Defining a new cost function
A potential advantage of the MLP classifier trained on squared error (Eq. (1)) or cross-entropy cost functions is that its output can be interpreted as an estimate of posterior probability [32:pp. 245–247]. However, the discussion in Section 4.3 indicated that this potential advantage is of limited value here, because of the imbalance of positive and negative examples in the test data. This suggests that there might be advantage to minimising during training a cost function more directly
MLP training with the new cost function
In theory, the precision and recall should be recalculated each time the network is modified during training. For incremental learning, this would impose a severe computational burden, since weights are updated for each training example. If batch training was used, precision and recall would only have to be recalculated once per epoch. However, rather poor results were obtained using this batch method. Better results were obtained using incremental learning and approximate values of precision
Classification results with the new cost function
Table 4 shows typical segmentation performance of a network trained using the E cost function with the (non-iterative) method of training set selection just described. To allow a fair comparison with earlier results, the network here was trained (with spatial inputs) on the same slices from subjects AC, CB and LP as used in producing the training data for the squared-error cost function (see Section 6.1).
These new results indicate that performance is comparable to that obtained using the
Post-processing with an active contour model
The initial segmentation by an MLP classifier produces an edge-point image of candidate boundary points which can never realistically give an acceptable closed contour. False negatives will lead to gaps in the contour (especially where the great blood vessels join the lungs and image evidence for a boundary is low or absent) and false positives will arise and need to be eliminated. Therefore, post-processing is required to close these gaps and to distinguish false positives from true positives.
Conclusions
MR image segmentation is an important but inherently difficult problem in medical image processing. In general, it cannot be solved using straightforward, conventional image-processing techniques. The solution proposed here is to use a multilayer perceptron to form the external energy function for an active contour model (‘snake’). Initial work used the conventional squared-error cost function for training the MLP. This showed that the MLP could classify the lung boundaries in MR images of the
Acknowledgements
We are grateful to Dr. Liz Moore and Prof. John Fleming for supplying the MR images used here and for valuable advice in connection with this work. Liz Moore provided the semi-automatic labellings of the lung outlines.
References (49)
On active contour models and balloons
Comput Vision Graphics Image Process
(1991)- et al.
MRI segmentation: methods and applications
Magn Reson Imaging
(1995) - et al.
A technique for single-channel MR brain tissue segmentation: application to a pediatric sample
Magn Reson Imaging
(1996) Generalizing the Hough transform to detect arbitrary shapes
Pattern Recog
(1981)- et al.
A survey of the Hough transform
Comput Vision Graphics Image Process
(1988) Which Hough transform?
Comput Vision Graphics Image Process
(1993)- et al.
A comparative study of efficient generalised Hough transform techniques
Image Vision Comput
(1999) - et al.
Kohonen networks for multiscale image segmentation
Image Vision Comput
(1994) - et al.
Deformable models in medical image analysis: a survey
Med Image Anal
(1996) - et al.
T-snakes: topology adaptive snakes
Med Image Anal
(2000)
A snake for CT image segmentation integrating region and edge information
Image Vision Comput
Using deformable surfaces to segment 3-D images and infer differential structures
Comput Vision Graphics Image Process
The image processing handbook
Quantitative measurements for gamma camera images
Craniosynotosis: diagnostic value of 3D CT reconstructions
Radiology
An adaptive Bayesian approach to three-dimensional MR brain segmentation
The use of active shape models for locating structures in medical images
3D image understanding in radiology
IEEE Eng Med Biol Mag
Snakes: active contour models
Int J Comput Vision
Active contours: the application of techniques from graphics, control theory and statistics to visual tracking of shapes in motion
A neural network-based stochastic active model (NNS-SNAKE) for contour finding of distinct features
IEEE Trans Image Process
Segmentation of magnetic resonance images of the thorax by back-propagation
An analogue approach to the travelling salesman problem using an elastic net method
Nature
A semi-localized elastic net for surface reconstruction of objects from multislice images
Int J Neural Syst
Cited by (85)
Deep learning in biomedical informatics
2022, Intelligent Nanotechnology: Merging Nanoscience and Artificial IntelligenceEvaluation of clinical applicability of automated liver parenchyma segmentation of multi-center magnetic resonance images
2022, European Journal of Radiology OpenModel-based segmentation using neural network-based boundary detectors: Application to prostate and heart segmentation in MR images
2021, Machine Learning with ApplicationsFractal and multifractal analysis of atherosclerotic plaque in ultrasound images of the carotid artery
2019, Chaos, Solitons and FractalsCitation Excerpt :The options for treating atherosclerosis are medical therapy and carotid endarterectomy or surgery [5,6]. The important factors that are taken into account when taking the decision for surgery are the thickness of the vessel wall, that is, the intima-media thickness as shown in Fig. 1, the degree of stenosis of the carotid artery and the existence or non-existence of symptoms [7-11]. The characteristics of plaque have diverse impacts on human life.
A gentle introduction to deep learning in medical image processing
2019, Zeitschrift fur Medizinische PhysikAssessment of despeckle filtering algorithms for segmentation of breast tumours from ultrasound images
2019, Biocybernetics and Biomedical EngineeringCitation Excerpt :Several algorithms have been developed by many researchers for the segmentation of medical images [52–57]. Out of all the segmentation techniques, active contour method of segmentation has been widely used in case of medical images [57–74]. The active contour models can be classified to be either edge-based or region-based.