Nuclei Detection in Cytological Images Using Convolutional Neural Network and Ellipse Fitting Algorithm

Morphometric analysis of nuclei play an essential role in cytological diagnostics. Cytological samples contain hundreds or thousands of nuclei that need to be examined for cancer. The process is tedious and time-consuming but can be automated. Unfortunately, segmentation of cytological samples is very challenging due to the complexity of cellular structures. To deal with this problem, we are proposing an approach, which combines convolutional neural network and ellipse fitting algorithm to segment nuclei in cytological images of breast cancer. Images are preprocessed by the colour deconvolution procedure to extract hematoxylin-stained objects (nuclei). Next, convolutional neural network is performing semantic segmentation of preprocessed image to extract nuclei silhouettes. To find the exact location of nuclei and to separate touching and overlapping nuclei, we approximate them using ellipses of various sizes and orientations. They are fitted using the Bayesian object recognition approach. The accuracy of the proposed approach is evaluated with the help of reference nuclei segmented manually. Tests carried out on breast cancer images have shown that the proposed method can accurately segment elliptic-shaped objects.