Morphological Separation of Clustered Nuclei in Histological Images

Automated nuclear segmentation plays an important role in computer-assisted analysis of histopathological images. Improving the quality and accuracy of nuclear segmentation has become an increasingly important topic, for which numerous analytical procedures have been proposed. One main challenge is the separation of clusters of overlapping nuclei. These often appear as irregular shapes resulting from an overlap of the 3D extent of nuclei on 2D image projections. An additional complication is that the overlapping nucleus boundaries are often indistinct and make the algorithmic separation a non-trivial challenge. This is especially relevant for the diagnosis of many diseases, including cancer, where identifying and characterising cellular abnormalities play an important role. There has been extensive, well-focused research by various groups on the splitting of nuclear clusters; a review paper [1] addresses recent advances and current challenges with respect to this problem. The well-known watershed algorithm [2, 3] has been used to address region separation by creating unique ‘basins’ where nuclei are defined by the ridges bounding the basins. Although this can result in successful separation in some images, there are well-documented problems with over- and under-segmentation in cells/nuclei that overlap, particularly when the boundary gradients at the overlaps are weak. Other methods use measures of concavity of the cluster contour to find the nuclei [4‐6] or use ellipse-fitting to infer the overlapping nuclei, one drawback being that the performance tends to be sensitive to fluctuations of the contours. Also, most methods detect but do not split clusters into individual nuclei segments in the 2D image. While it might not be possible to resolve this in all cases, such splittings are often necessary for extracting additional features (object counting, computing spatial relations) which are operationally useful even when derived from segmentations that include some mis-assigned pixels. Other methods [7‐9] employ iterative split-line algorithms to generate inner edges, and use concavity points to construct edge-path graphs. From these, the possible combinations of lines linking every pair of concavity points are computed and the shortest set of lines satisfying certain conditions is generated at the final splitting step. The split-lines are generated subject to the conditions that they yield sub-contours with acceptable nuclear sizes and do not intersect with other lines. While this can produce reasonable results in simple cases, it often fails in complex configurations, for a number of reasons: (a) from all the possible combinations of split-lines it is difficult to identify which point pairs to link; (b) the number of split combinations increases dramatically with the number of concavity points, becoming computationally costly and slow; (c) object separation uses straight lines along the inferred boundaries, while real nuclear boundaries are curved; (d) iterative separation usually leads to over-segmentation, as illustrated later.

Here we propose a new method for the detection and separation of individual nuclei in clusters based on the geometrical characteristics of the cluster boundary, particularly contour curvature; this approach overcomes several limitations mentioned earlier. The expected positions and shapes of individual candidate nuclei are estimated and followed by a series of morphological operations that separate the cluster into individual nuclear regions. The validity and effectiveness of the proposed framework was assessed through a series of experiments on images of clumped nuclei.

We investigated cluster separation on images of monolayers of H400 cells (an oral cancer cell line) grown on glass and captured at \(\times \)20 magnification (inter-pixel distance is 0.34 \(\upmu \)m) stained with Haematoxylin. These are typical conditions used in a variety of gene expression analyses. The haematoxylin (blue/violet) dye is primarily taken up by nucleic acids (therefore highlighting nuclei). Often eosin (pink dye) is used as a counter-stain, staining proteins in the intra- and extra-cellular compartments. A typical analysis of these cultures starts with a standard image pre-processing step such as colour deconvolution to unmix the dyes (if more than one is used) in order to facilitate extraction of the objects of interest. Nucleus segmentation is best performed on the Haematoxylin channel image owing to its aforementioned affinity to nucleic acids. Following this, a global thresholding method is applied to obtain a binary image of the nuclei where any clustered groups of nuclei would require separation. In the experiments described later, however, we work on a manually segmented ‘gold standard’ set of nuclear images. All imaging procedures were implemented on the popular ImageJ platform [10].

Our proposed method was tested using four large monolayer images of H400 cells. A total of 2610 nuclei were hand-drawn by one of us to produce a gold-standard set, which was used to obtain another set of binary images.³ Among these, a total of 497 nuclei formed 203 clusters with various degrees of complexity in their fused boundaries. Potential clusters were processed when their area was larger than 1600 pixels\(^{2}\) and contained concavity points. The optimal range of parameter values were initially chosen before applying to the tested images. The optimal values of gap and \(\check{O}\) depended on the geometry and size of the cluster, so they were tested at values of {5, 10, 15} and {2, 3}, respectively. Those were the values that generated regions with the highest circularity and within the optimal nuclear area. The thresholds \(\theta _1\) and \(\theta _2\) were constrained to be larger than 20 and less than 20 pixels, respectively. The qualitative results shown in Fig. 2 (upper row) demonstrate the ability of our algorithm to resolve complex clusters (with four or more overlapping nuclei) while avoiding over- and under-segmentation. The procedure can generate contours close to actual nucleus boundaries. The lower rows of Fig. 2 show the superiority of our approach over the watershed separation, which generates spurious edges in some simple configurations.

Quantitative results (summarized in Table 1) were obtained using three different measures. Our segmentation result was compared visually with the gold-standard to estimate the True Positive Rate \(TPR=\frac{TP}{TP+FN}\) and the Positive Predictive Value \(PPV=\frac{TP}{TP+FP}\), where the True Positives (TP), False Positives (FP), and False Negatives (FN) are the numbers of correctly detected nuclei, incorrectly detected nuclei, and undetected nuclei, respectively. The third measure was given by the Jaccard Index (JI), which is defined as the ratio \(JI = \frac{|\check{G} \cap I|}{|\check{G} \cup I|}\) between the pixel-counts of the intersection and union of the gold-standard segmented image \(\check{G}\) and the test segmentation I. The JI ranges from 0 (no overlap between the images) to 1 (complete congruence). Note that the line-of-sight 2D projections of the gold-standard images show inferred overlapping nuclear boundaries which cannot be separated and represented in a single binary image. The JI match of \(\check{G}\) and I is measured as follows. Using standard morphological operations we (i) extract the symmetric difference of the filled overlapping regions with respect to their filled intersections (‘lenses’) to generate a set of ‘lunes’; (ii) separate the lunes into their component parts by a morphological erosion, (iii) apply a binary dilation without merging operation to the separated lune parts within a mask image (the merged filled nuclear profiles). The split-line generated in each of the lens-like regions approximates their medial-axis transform. Overall, the proposed method outperforms the classical watershed in terms of TPR, PPV and JI. During the experiments, we observed that the algorithm preserves the ellipticity of the reconstructed nuclear regions, and in particular it outperforms watershed segmentation in clusters that lack prominent ‘necks’; it is, however, sensitive to the gap and \(\check{O}\) parameters.

We presented a novel mathematical morphology-based algorithm for separating clustered binary nuclear profiles. Concavity features of the cluster boundary are extracted and guide the subsequent region-splitting steps. Optimal split boundaries are computed using a series of morphological operations. Unlike in iterative split-line models, our non-iterative algorithm provides separation while avoiding over-segmentation. Qualitative and quantitative results tested on hand-segmented datasets of images of H400 cells verify that the segmentation accuracy of the proposed method outperforms the watershed separation approach.