Nuclei Recognition Using Convolutional Neural Network and Hough Transform

The paper presents method of nuclei segmentation on cytological images based on the Convolutional Neural Network (CNN) and modified Hough Transform method. It approximates nuclei by ellipses fitted to nuclei regions segmented by CNN. As study data set 50 cytological RGB images were used, divided into training set (50 images) and test set (10 images). The first step is to create a CNN model for pixel-wise classification of cytological images. As training set for CNN, patches of size 28\(\,\times \,\)28 pixels were created based on images from training set and corresponding ground-truth labels. Using trained model, nuclei regions classification and segmentation from test set images was conducted. The reason of choosing the CNN for segmentation it’s better accuracy in separated overlapping nuclei than conventional methods such as for example Otsu thresholding etc. Subsequently, using Canny algorithm and Euclidean Distance Transform (EDT), edges and centers of segmented regions were extracted. Edges and centers of nuclei were extracted for reduce time computation for next step. Finally, finding nuclei using the modified Hough Transform by fitted ellipses was carried out.