Deep learning for thyroid nodule examination: a technical review

Most nodules are asymptomatic and can be identified during a normal medical check. There is no correlation between the size, morphology and function of the nodule and the severity of symptoms (Hegedus et al. 2003). A large nodule can cause important organs in the neck or upper thoracic cavity to become compressed. Compression of the trachea may cause shortness of breath, coughing, choking feelings, and other symptoms. In patients with large intrathoracic nodules, esophageal compression causes difficulty swallowing. Infrequent signs of malignancy include vocal cord paralysis caused by stretching and compression of the laryngeal nerve, phrenic nerve paralysis, and Horner syndrome induced by sympathetic nerves (Hegedus et al. 2003). In patients with benign nodules, family history, radiation history, and other comorbidities help define future risk. A family history of thyroid cancer (Nagy and Ringel 2015) and early radiation exposure (Kang et al. 2009) increase the probability of developing a malignant thyroid nodule.

Thyroid nodules are part of a complex group of diseases. Complex diseases are prevalent, but their severity varies considerably. Genetic and environmental variables are involved in the genesis of thyroid nodules (Brix and Hegedüs 2000). Iodine deficiency is one of the most prevalent environmental causes of thyroid nodules (Lo et al. 2000). Smoking, naturally occurring goitrogens, some types of medications, and infection are additional environmental influences. The interaction between genetic predisposition and environmental variables could result in the development of thyroid nodules. Age and sex are other risk factors for thyroid nodules. According to research, women are more likely than men to have thyroid nodules. Similarly, children and older individuals are more likely to receive a diagnosis of a thyroid nodule (Hegedus et al. 2003).

Ultrasound imaging is a noninvasive diagnostic technique that uses high-frequency sound waves to produce images of the body's interior. It is commonly used to identify thyroid nodules by examining the thyroid gland and its surrounding tissues. Ultrasound is a common imaging technique for thyroid nodules because it is safe, painless, and relatively inexpensive. Additionally, it is widely accessible, making it a practical option for patients with suspected nodules. Ultrasound imaging can provide valuable information regarding the size, shape, and location of thyroid nodules, as well as their characteristics such as echogenicity and the presence of suspicious characteristics. This information can help physicians determine the best course of treatment for patients with thyroid nodules. For instance, the need for fine needle aspiration (FNA) depends on the level of risk associated with a detected nodule. For instance, a hypogenic nodule is more likely to be malignant if it has irregular boundaries, microcalcifications, and a form that is taller than wide. Preoperative CT imaging can also be performed if an aggressive form of thyroid cancer is detected.

FNA is a minimally invasive diagnostic procedure. During FNA, samples are obtained from a location with aberrant tissues or body fluids using a small needle to determine the type of nodule (i.e., benign, or malignant) and extract other important information. The American Thyroid Association in 2015 advised FNA for nodules 1 cm or larger with a high or intermediate suspicious pattern on US images, 1.5 cm or larger with a low suspicious pattern, and 2 cm or larger with an extremely low suspicious pattern (Haugen et al. 2016).

The most common and longest-lasting treatment for larger nodules that cause organ compression and aesthetic problems is surgery. Hemithyroidectomy involves the removal of a portion of the thyroid gland. This is considered a low-risk surgery. After surgery, nerve injury to the recurrent laryngeal nerve has been observed in a small percentage of patients (0 to 5.8%) (Lo et al. 2000). Total thyroidectomy refers to the removal of the thyroid gland in its entirety. Multiple hyperfunctioning nodules require a total thyroidectomy to remove them. This will result in hypothyroidism and lifelong thyroid replacement therapy (Trimboli 2022).

Pathology reports are widely used to determine specific information about tumor type. When a nodule satisfies certain criteria, FNA samples are obtained during the standard clinical workflow. The cytologic results obtained are reviewed by a pathologist to determine the benignity or malignancy of the nodules (Buda et al. 2019). In certain studies, in which nodules do not meet the criteria for FNA but are highly suspected of malignancy by ultrasound imaging, FNA is recommended prior to further treatment (Wang et al. 2019). In some cases, additional screening steps are added during the preprocessing phase (Zhao et al. 2022). The radiologist uses pathology data to confirm the labels of the images. Pathology data are often absent for benign nodules that are less suspicious. Pathology data can aid in the development of an architecture that takes advantage of domain-specific knowledge.

Technical and medical researchers frequently work together to investigate various aspects of ultrasound image analysis, including the development and evaluation of new algorithms for image processing or diagnosis. Collaboration between technical and medical researchers can be advantageous because it allows the integration of technical and medical knowledge. These collaborations also lead to the collection of patient data, which is often not made publicly available due to patient privacy restrictions. Some of the datasets are only available upon request for specific research purposes. This strategy helps to ensure that data is used ethically and responsibly, and that patient privacy is protected. However, with recent advances in DL and calls for reproducible and open science, many institutes have begun releasing deidentified versions of their datasets (Pedraza et al. 2015; “Thyroid for pretraining” 2022). In most studies, thyroid ultrasound imaging involves a transverse or sagittal view of B-mode ultrasound imaging (see Figs. 2 and 3). Zhang (Zhang et al. 2021) uses B-mode as well as Doppler ultrasound images (see Fig. 4) for the analysis.

Objects in an image can be annotated with bounding boxes or polygons, depending on the task. In medical image segmentation and detection, polygon annotation is preferred. In contrast, classification requires annotation at the image-level, i.e., annotation of the entire image. The objects in the US images are manually annotated by a skilled sonographer and radiologist with the aid of pathology data. Various tools, such as the VGG Image Annotator (VIA) (Dutta et al. 2016), are used to effectively annotate image data. Frequently, annotated results are saved in XML or JSON format. Sometimes the experience of experts exceeds 10 years (Zhang et al. 2021). TUN-Det is trained on a dataset manually annotated and validated by five sonographers and three radiologists (Shahroudnejad et al. 2021). Annotation varies based on the radiologist's experience and is time consuming; therefore, one study (Song et al. 2022) recommends the implementation of pseudo-labeling techniques. Typically, to locate and classify thyroid nodules, training images that contain at least one nodule are chosen (Peng et al. 2021; Wang et al. 2019). In some instances, the architectures are trained on images with nodules, but tested on images with or without nodules (Li et al. 2018).

Historically, ML-based classifiers required the designer to define the key features used (Chai et al. 2020). Song et al. (2015) applies six statistical techniques, such as a support vector machine (SVM), to the output of the gray co-occurrence matrix (Singh et al. 2017) to classify nodules as benign or malignant. A recent multicenter study of thyroid ultrasound images demonstrated that the combined architectures of three networks trained with vast amounts of data can outperform a radiologist (Peng et al. 2021).

In addition to using DL algorithms as standalone classifiers, researchers have examined the use of image processing techniques and convolutional neural networks (CNNs) in tandem, resulting in an increase in the accuracy and performance of DL models for thyroid US image analysis. One such example is Feature Fusion ResNet (Zhao et al. 2022). Due to the similarity of texture information associated with lung and thyroid nodules, ResNet is initially trained on a Kaggle dataset for lung nodule recognition and then refined for thyroid nodule classification. Feature extraction techniques include the gray-level size zone matrix, the gray-level co-occurrence matrix, and the gray-level run length matrix. These are used to extract features that are then sent to a fully linked layer in a neural network. Principal component analysis to is used to eliminate redundant characteristics. Feature Fusion ResNet outperformed VGG16, ResNet, and VGG16.

Thyroid nodule segmentation is an integral part of ultrasound image diagnosis of thyroid diseases (Russ et al. 2017). In recent years, the segmentation of additional tissues related to the thyroid gland (Ma et al. 2022) has been the subject of investigation. A comprehensive review (Chen et al. 2020) focusing on a comparative analysis of 28 studies on CAD-based systems and DL demonstrates the evolution of thyroid gland and nodule segmentation until 2019. That review highlights the similarities and contrasts between the approaches and discusses the benefits and drawbacks of each system.

U-Net (Chu et al. 2021) is a DL architecture that was specifically designed for image segmentation tasks. It consists of a CNN with an encoder-decoder structure, where the encoder extracts feature from the input image and the decoder reconstructs a precise segmentation of the input image. U-Net has been shown to be highly effective and precise for ultrasound (US) image segmentation tasks involving thyroid nodules. This is because it can automatically learn and extract relevant features from the data (such as the shape and location of thyroid nodules) without requiring prior knowledge or manual feature engineering (Chu et al. 2021).

The Semantic Guided U-Net (SG-UNET) (Pan et al. 2021) was recently proposed in response to its success. The use of average pooling and leaky ReLU reduces noise and attenuates the unfavorable filter response. To reduce noise interference caused by the mirror structure of the U-Net, a side network accepts high-dimensional features and transforms them into one-dimensional semantic features. The suggested architecture outperforms U-Net and U-Net++ in terms of performance. In addition to fully automated systems, there are semi-automated systems, such as mark-guided U-Net-based segmentation systems (Lu et al. 2022). The network proposed by Chu (Lu et al. 2022) showed segmentation precision of 0.9785.

Sun's TNSnet (Sun et al. 2022) is a dual network comprising two sub-networks: a form network and a regional network. The form network oversees figuring out an object's general shape or form, while the regional network is in charge of figuring out the object's unique details and traits. Compared to conventional single-network architectures, this dual network architecture permits more precise and accurate object detection.

Recently, researchers used a two-stage network to detect a specific form of thyroid cancer, medullary thyroid carcinoma, which is the second most prevalent (yet rare) form of thyroid cancer (Pan et al. 2022). The segmentation map is generated using a coarse-to-fine segmentation network (C2F-SegNet), which is a combination of CoarseNet and FineNet. A classifier based on past knowledge is then used to classify the nodules. The backbone structure of the knowledge-based classifier network is Resnet-34, which has been pre-trained on ImageNet (Deng et al. 2009). The suggested network not only accurately differentiates the malignant nodule but also the papillary and medullary forms of thyroid cancer. The suggested network's segmentation architecture has a greater IoU and DSC performance than U-Net and U-Net++, while U-Net++ has a higher recall and precision. In contrast, the classification architecture surpasses ResNest34 and ResNest50 in all aspects of evaluation.

A weakly supervised model can help with the management of over-or under-segmentation of thyroid nodules (Yu et al. 2022). Semantic features are extracted using image-level classification labels. The nodule site is activated using a dual branch soft erase module (DBSM) and a scale feature adaptation module (SFAM). The edge self-attention module (ESAM) is implemented to handle blurred edges.

For further details regarding thyroid gland segmentation and thyroid nodule segmentation methods for medical ultrasound images, see the existing comprehensive review (Chen et al. 2020).

The detection task determines the location and identification of anomalies (e.g., lesions and tumors) and other anatomical objects (e.g., fetal standard plans, organs, tissues, etc.) in US image analysis. A new era in thyroid nodule detection has begun with the introduction of DL architectures such as region-based convolution neural networks (R-CNN) (Girshick et al. 2014), Faster R-CNN (Ren et al. 2015), and you only look once (YOLO) (Redmon et al. 2016).

Li et al. (2018) proposed an enhanced version of Faster R-CNN for the detection of thyroid nodules. A spatial constraint layer is implemented to extract the characteristics of the surrounding region. Combining the shallow and deep layers of the networks allows identification of small hazy nodules. Standard, Faster R-CNN with a layer concatenation technique was unable to detect solid nodules with uneven borders. Buda et al. (2019) suggests a three-part model. The first component is a Faster R-CNN trained to recognize capillaries using Resnet101 as its backbone. The recovered square image of the nodules is then sent to a second network. The second network is a multitask deep convolutional neural network that is utilized to detect whether a nodule is malignant. The third section focuses on risk classification using the ACR TI-RAD output. In comparison to the consensus of three radiologists, the proposed network generates similar results. In addition, the risk classification network improves the specificity of the referral for thyroid nodule biopsy.

Xie et al. (2019) proposed three completely convolutional networks based on single-shot multi-box detectors (SSD) (Liu et al. 2016), namely SSD300, SSD300 cov3, and SSD512. The base model is a typical image classification model with a fully linked layer. A convolutional layer stack is applied to extract additional features. These layers downsample the photos to generate a diverse range of large and small features. At the end of the network, a convolutional predictor is added to produce a class score for certain feature maps. The suggested architecture's training loss function combines smooth L1 loss and class-weighted entropy loss. The SSD300 was unable to identify the tiny nodules. Sometimes, SSD300 cov3 and SSD512 identified false positives among smaller nodules. The SSD300 was unable to identify certain dense nodules. The application of SSD showcases their suitability for capturing both global and local features present in thyroid ultrasound images.

Song et al. (2018) suggested a CNN with several cascaded architecture to detect thyroid nodules, based on SSD and the multi-box framework. The detection layer of the SSD is reconstructed by adding several convolution layers and anchor-generated layers to extract local and global features. The two-stage network designed to localize and classify nodules in a pyramidal structure. After the first localization, the potential region of interest is input into a spatial pyramid supplemented by CNNs to achieve adequate recognition of the thyroid. The architecture was constructed using a huge dataset. The architecture was unable to detect excessively small or large nodules. To combat the lack of high-quality data, the authors proposed investigating the use of transfer learning as future work. Multitasking within deep learning models involves training a single architecture to perform multiple related tasks concurrently. This approach offers advantages such as efficient resource utilization, shared information across tasks, and regularization against overfitting. However, it introduces trade-offs that necessitate careful consideration. Efficient resource allocation is crucial to prevent compromising task-specific performance, and increased model complexity can hinder interpretability and optimization. Task interference, stemming from dissimilarities between tasks or conflicting patterns, must be balanced. Striking a balance between resource efficiency and maintaining task-specific performance is pivotal.

Zaho et al. (2021) recently presented a two-part model. SSD with ResNet50 as backbone is used for thyroid nodule detection. The last few layers of the ResNet50 architecture are replaced with a residual block and a few more blocks. To account for the various aspect ratios of the thyroid nodule, anchor boxes between 0.3 and 3 are produced. Nodules are clipped with a 50-pixel margin, and a 256 × 256 image is sent to the classification network. For classification, A-ResNet50-F, a modified version of ResNet50, was used. A-ResNet50-F is made up of an additional attention block and a fire block. The decision to clip nodules with a 50-pixel margin and process them as 256 × 256 images is a reasonable preprocessing step, likely aiding in standardizing the input size for the classification network. The introduction of A-ResNet50-F, which includes an attention block and a fire block, is intriguing as it implies a focus on capturing crucial nodule-specific features and optimizing the network for the classification task. The suggested architecture demonstrates a higher average precision than SSD (backbone vgg16), Faster R-CNN (backbone ResNet50), and YOLO v3 (with Darknet53). For semi-solid nodules with irregular edges, the identification system detected a somewhat larger bounding box. However, the classification network has beaten well-known networks such as VGG16 and Inception v3. In terms of classification, the proposed technique also beats expert radiologists.Mask R-CNN (He et al. 2017) is a two-stage network repeatedly used for object detection and segmentation. The increasing popularity of Mask R-CNN has paved the way for a new direction in thyroid nodule detection (Abdolali et al. 2020). The R-CNN mask combines classification, segmentation, and localization loss, which has been shown to be advantageous for the detection of thyroid nodules. The combined loss function favors detection over-segmentation. Resnet50 beat all other pre-trained backbone networks, including Resnet, U-Net, Mobilenet, and Inception V2, due to the smaller data size. Comparing the performance of mask R-CNN to R-CNN with the new loss function, which is faster, demonstrates a focus on both accuracy and efficiency. However, the noted limitation of the network's suboptimal performance in primarily solid nodules reveals an area that requires attention. The absence of thyroid parenchyma and microcystic nodules in the dataset used is also a relevant consideration for the generalizability of the results. The future direction outlined by the authors, which involves reducing the complexity of mask R-CNN and addressing overfitting, provides a proactive approach to refining the model. Simplifying the architecture and enhancing its generalization capabilities can contribute to the model's robustness and usability in diverse clinical scenarios. As a solution to inaccurate localization, Zheng et al. (2022b) proposes a more efficient cascade mask R-CNN. The structure is composed of two stages. In the first stage, an ROI is determined using FPN (Lin et al. 2017), the backbone, and RPN. The subsequent stage is the multi-cascading detector network, which provides a more specific location for the nodule. Using a modified version of L1 loss, the gradient of the easy samples is increased, providing a balance between the easy and difficult samples during the training phase. Experimental results demonstrate that using more than three cascading networks affects performance. A "soft NMS" is used to reduce the likelihood of an object being missed due to non-maximal suppression (NMS). The baseline with soft NMS did not improve localization on its own. Using a novel detector with five convolutional layers and one fully linked layer, together with L1 regularization, results in a considerable improvement in localization. The authors claim that the proposed network is more precise than Mask R-CNN, Faster R-CNN, and Libra R-CNN (Pang et al. 2019) for medium and small nodules. The proposed network has a lower true negative rate than Mask R-CNN.

Wang et al. (2019) proposed a modified version of YOLO to identify thyroid nodules with greater precision. YOLO is a one-stage model that decreases temporal complexity and enhances precision. Detecting thyroid nodules requires more precision than quickness. YOLO v2 and Resnet v2-50 are combined to create this model. The feature maps from the deep layers are blended with those from the shallow layers to generate more accurate feature maps. This strategy capitalizes on the diverse information captured across different layers, aiming to generate more accurate and comprehensive feature maps. The suggested network is quick and claims to reach the same sensitivity, positive predictive value, and accuracy as the radiologist. The dataset is divided into two classes: nodules and without nodules. The nodule classes are also classified as benign or malignant. The dataset's division into nodules and without-nodules classes, along with the classification of nodule classes into benign and malignant, highlights a structured and comprehensive dataset setup. This approach likely contributes to the model's ability to differentiate between various nodule types effectively. The network is effective in detecting images without nodules; in only two instances the network does not detect the absence of a nodule. The network is effective in localizing cancerous nodules. Most malignant thyroid cases in the dataset are papillary thyroid carcinoma. The work does not explore the improved performance of the radiologist using the suggested network.

Redmon and Farhadi developed YOLO v3 (Redmon and Farhadi 2018), which is renowned for having greater precision than its predecessors. Thus, the researchers developed YOLO-HRNet by using YOLO v3 as the base model (Zhang et al. 2021). It consists of five distinct layers: input, downsampling, feature extraction, multi-scale detection, and prediction. The downsampling layer is used to scale the network's parameters. The incorporation of HRNet (Wang et al. 2020), known for extracting high-level semantic features, is a strategic choice to enhance the network's ability to understand and capture complex visual characteristics. This choice aligns with the objective of achieving superior performance over previous architectures. In all respects, YOLO-HRNet outperforms the Yolo V3 baseline. However, SSD beats YOLO-HRNet in terms of speed, while faster R-CNN with Resnet50 as its backbone outperforms the network in terms of precision. The network generates larger bounding boxes for small hypogenic nodules with fuzzy edges. Redmon and Farhadi anticipate more tagged medical data and the use of thyroid antibodies for detection.

Song et al. (2022) proposed a feature-enhanced dual branch network (FDnet) to detect thyroid nodules. The detection network includes a semantic segmentation network and a feature enhancement method. A more precise localization is achieved by using an iterative training technique that blends the ground truth with the branch result. Resnet and FPN serve as the backbone structures. An RPN is utilized to target the nodule region. Using a pseudo-labeling system, the pseudo-labels generated in an epoch are used as the ground truth in the subsequent epoch. Given their proven effectiveness in feature extraction tasks, using ResNet and FPN as backbone structures is a prudent choice. Additionally, incorporating an RPN (Region Proposal Network) to focus on the nodule region suggests a strategic approach to narrowing the areas of interest. The introduction of a pseudo-labeling system is an intriguing aspect of the approach. This strategy, where pseudo-labels generated in one epoch are used as ground truth in the next epoch, demonstrates an innovative approach to augmenting the training data. It's particularly impressive that the proposed network exhibits strong performance even when trained on a small dataset, potentially making it more accessible and practical for real-world medical applications. It outperforms the most widely used R-CNN designs. CornerNet (Law and Deng 2018) and DETR (Zhu et al. 2020) surpass FDNet in mAP and F1 score, despite requiring 240 and 150 training epochs, respectively. In addition to saving processing time, Song utilized a pseudo-labeling scheme to construct further unsupervised schemes.

Shahroudnejad et al. (2021) developed a one-stage FPN model, known as TUD-Net, to gather multiscale characteristics from multiple resolution feature maps. The suggested architecture consists of three parallel layers of RSU designed to replace a CNN for classification and regression tasks. The model recognizes heterogeneous, large, and hypoechoic nodules successfully. In terms of average precision, the model outperforms models like Retinanet and Faster R-CNN. TUD-Net performs well under different IoU thresholds.

Lu et al. (2022) recently applied a GAN-guided CAM-based technique to the diagnosis of thyroid nodules. The architecture includes class activation mapping (CAM) (Zhou et al. 2016) to identify discriminatory features in thyroid nodules, and a GAN (Yi et al. 2019) guided deformable module to capture finer-grained distinctions between benign and malignant nodules. CAM supplies the saliency map to the outer layers of the deformable network also known as deformable convolution layers. The CAM and the deformable module together successfully detect the subtle differences between the nodules. Real samples are generated by augmenting the ground truth mask using prior knowledge. The number of augmentation techniques used are limited and can be easily captured by the GAN. Thus, a simple augmentation technique such as zooming at the nodule can deceive the discriminator. If for a particular application the nodule shape is not relevant for diagnosis, or the boundaries are regular, the GAN may prevent the deformable module from capturing the meaningless features from the images.

Integrating the CAM mechanism to identify discriminatory features in thyroid nodules is a strategic step. The CAM's ability to generate saliency maps contributes to a focused understanding of relevant areas within the nodule images. A notable advancement is the subsequent collaboration with a GAN-guided deformable module to capture nuanced distinctions between benign and malignant nodules. By incorporating the deformable convolution layers with the saliency information from the CAM, the model seems well-equipped to discern subtle differences between nodules. The inclusion of real sample generation through augmented ground truth masks, based on prior knowledge, highlights a data augmentation strategy. This approach likely contributes to a more diverse training dataset, which can be particularly useful in addressing potential overfitting and enhancing the model's generalization capabilities.

Based on the research work cited in this review, it is evident that there is a need for standardized evaluation matrices. DL networks using the same base model have been evaluated inconsistent performance matrices; for example, see (Xie et al. 2019; Song et al. 2018). It is very difficult to draw any kind of comparison between the networks by just looking at the results. Apart from that, there is a need for a standardized data set for testing all the networks.