Improving ductal carcinoma in situ classification by convolutional neural network with exponential linear unit and rank-based weighted pooling

BT is a sub-science field within IR imaging sciences. IR cameras detect radiation in the long IR range (9–14 µm), with the thermal images generated being dubbed thermograms. Physically, Planck’s law stated the spectral of a body for frequency \(\omega\) at absolute temperature T is given aswhere B stands for the spectral radiance, \(o\) the Planck constant, k_B the Boltzmann constant, and \(l_{s}\) the light speed,. If replacing frequency \(\omega\) by wavelength λ using \(l_{s} = \lambda \omega\), above equation can be written as:

Both charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS) sensors in optical cameras detect visible light, and even near-infra-red (NIR) by utilizing parts of the IR spectrum. Basically, they could produce true thermograms with temperatures beyond 280 °C.

In our breast thermogram cases, the thermal imaging cameras have a range of 15–45 °C, and a sensitivity around 0.05 °C. Furthermore, three emitted components (ECs) help generate the following breast thermogram images: (1) EC of the breast, (2) EC of the surrounding medium, and (3) EC in the neighboring tissue.

In healthy tissue, the major regulation and control of dermal circulation is neurovascular, i.e. through the sympathetic nervous system. Its sympathetic response includes both adrenergic and cholinergic. The former causes vasoconstriction (VC, narrowing of blood vessels); conversely, the latter leads to vasodilation (VD, widening of blood vessels). The difference between VC and VD is presented in Fig. 1.

In the early stages of cancer growth, cancer cells produce nitric oxide (NO), resulting in VD. Tumor cells then initiate angiogenesis, which is necessary to sustain breast tumor growth. Both VD and angiogenesis lead to increased blood flow; therefore, the increased heat released as a result of increased blood flow to the tumor results in hotter areas than healthy skin.

The thermogram of a healthy person is symmetrical across the midline. Asymmetry in the thermogram might signify an abnormality, or even a tumor. Therefore, the thermogram illustrates the status of the breast and presence of breast diseases by identifying asymmetric temperature distribution.

Despite this, previous studies [7, 8, 14] have not measured asymmetry directly. As an alternative, those papers employed texture or statistical measures. As a result, this study did not use asymmetry information, and treated each side image (left breast or right breast) as individual images.

240 DCIS breast images and 240 healthy breast (HB) images were obtained from 5 sources: (1) our previous study [12] and further collections after its publication. (2) Ann Arbor thermography [15]; (3) The Breast Thermography Image dataset [16]; (4) The Database for Mastology Research with Infrared Image [17] and (5) online resources using search engines including Google, Yahoo, etc.

Since our dataset is multi-source, we normalized all the collected images using preprocessing techniques. These included: (I) crop: remove background contents and only preserve the breast tissue and (II) resize: all images were re-sampled to the size of \(\left[ {128 \times 128 \times 3} \right]\). Suppose original image is \(x_{1} \left( t \right),t \in \left[ {1,480} \right]\). After Step I, we havewhere \(\left( {l_{t} ,r_{t} ,t_{t} ,b_{t} } \right)\) are four parameters denotes left, right, top, and bottom margins of t-th image to be cropped.

Finally, after Step II, we have all the images \(x_{3} \left( t \right) \in X_{3}\)

Note some BT images used different pseudo colormaps (PCMs). For example, some used yellow to denote high temperature while some used red; conversely, some used blue to denote low temperature while some used green. We did not apply the same PCM to all BT images within our dataset for four reasons: (1) we expected our AI model would learn to determine a diagnosis based on color difference, not the color itself; (2) humans can make a diagnosis regardless of the PCM configuration, so we believed AI can do the same; (3) we expected our AI model can be universal, i.e., PCM-independent and (4) mixing of PCM color schemes in the training set can help make our AI model more robust when analyzing the test set, i.e., it does not require a particular PCM scheme.

Figure 2 shows a DCIS case, where we can clearly see the temperature difference of the lesion and the surrounding healthy tissues. All the images included in our dataset were checked by agreement of two professional radiologists \(\left( {R_{1} ,R_{2} } \right)\) with more than 10 years of experience. If their decisions \(\left[ {H\left( {R_{1} } \right),H\left( {R_{2} } \right)} \right]\) agreed, then the images were labelled correspondingly, otherwise, a senior radiologist \(\left( {R_{3} } \right)\) was consulted to achieve a consensus:

Here \(H\) is the labelling result, \(M\) denotes the majority voting, \(H\left[ {x_{3} \left( t \right),\left( {R_{1} ,R_{2} ,R_{3} } \right)} \right]\) denotes the labelling results by all three radiologists.

The activation function mimics the influence of an extra-cellular field on a brain axon/neuron. The real activation function for an axon is quite complicated, and can be written aswhere n means the index of axon’s compartment model, c the membrane capacity, \(R_{n}\) the axonal resistance of compartment n, \(V_{n}^{e}\) the extra-cellular voltage outside compartment n relative to the ground [18]. This is difficult to determine in an “artificial neural network”, and thus AI scientists designed some simplistic and ideal activation functions (AFs), which have no direct connection with the axon’s activating function, but those AFs work well for ANNs [19].

An important property of AF is nonlinearity. The reason is stacks of linear function will also be linear, and those kinds of linear AFs can only solve trivial problems and cannot make decisions. Only nonlinear AF can allow neural networks to solve non-trivial problems, such as decision-making. Similar ideas were mentioned as “even our mind is governed by the nonlinear dynamics of complex systems” by Mainzer [20].

Suppose the input is t, traditional rectified linear unit (ReLU) [21] \(f_{{{\text{ReLU}}}}\) is defined aswith its derivative as

When \(t < 0\), the activation of \(f_{{{\text{ReLU}}}}\) values are set to zero, so ReLU cannot train the networks via gradient-based learning. Clevert et al. [22] proposed the exponential linear unit (ELU)

ELU’s derivative is

The default value of \(\gamma = 1\). Figure 3 represents the shapes of five different but common AFs. Each subplot has the same range on the x-axis and y-axis for easy comparison. Information regarding the three AFs (Sigmoid, HT, and LReLU) can be found in “Appendix B”.

The activation maps (AMs) after conv layer are usually too large, i.e., the size of their width, length, and channels are too large to handle, which will cause (1) overfitting of the training set and (2) large computational costs. Instead pooling layer (PL) is a form of nonlinear downsampling (NLDS) used to solve the above issue. Further, PL can provide invariance-to-translation properties to the AMs.

For a \(2 \times 2\) region, suppose the pixels within the region \(\Phi = \left\{ {\varphi_{ij} } \right\},\left( {i = 1,2,j = 1,2} \right)\) are

Strided convolution (SC) can be regarded as a convolution followed by a special pooling. If the stride is set to 2, the output of SC is:

The shortcoming of SC is that it will miss stronger activations if \(\varphi_{1,1}\) is not the strongest activation. The advantage of SC is the convolution layer only needs to calculate 1/4 of all outputs in this case, so it can save computation.

L2P calculates the \(l_{2}\) norm [23] of a given region \(\Phi\). Assume the output value after NLDS is y, L2P output \(y_{\Phi }^{{{\text{L}}2{\text{P}}}}\) is defined as \(y_{\Phi }^{{{\text{L}}2{\text{P}}}} = {\text{sqrt}}\left( {\mathop \sum \nolimits_{i,j = 1}^{2} \phi_{ij}^{2} } \right)\). In this study, we add a constant \(1/\left| \Phi \right|\), where \(\left| \Phi \right|\) means the number of elements of region \(\Phi\). Here \(\left| \Phi \right| = 4\) if we use a \(2 \times 2{ }\) NLDS pooling. This added new constant 1/4 does not influence training and inference.

The average pooling (AP) calculates the mean value in the region \(\Phi\) as

The max pooling (MP) operates on the region \(\Phi\) and selects the max value. Note that L2P, AP and MP work on every slice separately.

Rank-based weighted pooling (RWP) was introduced to overcome the down-weight (DW), overfitting, and lack of generation (LG) caused by the above pooling methods (L2P, AP, and MP). Instead of computing the \(l_{2}\) norm, average, or the max, the output of the RWP \(y_{\Phi }^{{{\text{RWP}}}}\) is calculated based on the rank matrix.

First, rank matrix (RM) \(R = \left\{ {r_{m} } \right\}\) is calculated based on the values of each element \(\varphi_{m} \in \Phi\), usually lower ranks \(r \in R\) are assigned to higher values (\(\varphi\)) as

In case of tied values (\(\varphi_{m1} = \varphi_{m2}\)), a constraint is added as

Second, (ER) map \(E = \left\{ {e_{m} } \right\}\) is defined aswhere α is a hyper-parameter. \(\alpha = 0.5\) for all RWP layers, so we do not need to tune \(\alpha\) in this study. Equation (18) can be updated as

Third, RWP [24] is defined as the summation of \(\varphi_{ij}\) and \(e_{ij}\) as below

Figure 7 in “Appendix C” gives a schematic comparison of L2P, AP, MP, and RWP.

For better understanding, a pseudocode of RWP is presented in Table 1. We suppose there is an activation map \(X_{{{\text{AM}}}}\) with size of \(\left[ {R,C} \right]\), where \(R\) means the number of rows, and \(C\) means the number of columns. Note row index is set to \(r\) and column index \(c\). The RWP output of \(X_{{{\text{AM}}}}\) is symbolized as \(y^{{{\text{RWP}}}}\) with size of \(\left[ {\frac{R}{2},\frac{C}{2}} \right]\). Table 2 itemizes the equations of every pooling methods.

Traditional data augmentation is a strategy that enables AI practitioners to radically increase the diversity of training data, without collecting new data actually. In this study, we proposed a L-way data augmentation (LDA) technology to further increase the diversity of the training data. The whole preprocessed image set \(X_{3}\), from Eq. (4), will separate into \(K\) folds:where \(k\) represents the fold index.

At k-th trial, fold k will be used as the test set \(D_{k}\), and other folds will be used as the training set \(C_{k}\):

If we do not consider the index k, and just simplify the situations as \(X_{3} \to \left\{ {C,D} \right\}\), for each training image \(c\left( k \right) \in C,k = 1, \ldots ,\left| C \right|\), we will do the following eight DA techniques. Here we suppose each DA technique will generate \(W\) new images.

The randomness effect of each run reduced performance reliability, so we used \(K\)-fold cross validation to analyze unbiased performances. The size of each fold is \(\left| {X_{3} } \right|/K\). Due to there being two balanced classes (DCIS and HB), each class will have \(\left| {X_{3} } \right|/\left( {2 \times K} \right)\) images. The split setting of one trial is shown in Table 6. Within each trial, \(\left( {K - 1} \right)\) folds were used as training, and the rest fold were used as test. After combining all \(K\) trials, the test image grew to \(\left| {X_{3} } \right|\). If above \(K\)-fold cross validation repeats \(Z\) runs, the performance will be reported on \(\left| {X_{3} } \right| \times Z\) images.

Suppose the ideal confusion matrix \(E^{{{\text{ideal}}}}\) over the test set at k-th trial and z-th run iswhere the constant 2 is because our dataset is a balanced, i.e., DCIS class has the same size of HB. After combining \(K\) trials, the ideal confusion matrix is at \(z\)-th run is

In realistic inference, we cannot get the perfect diagonal matrix as shown in Eq. (36), suppose the z-th run real confusion matrix iswhere \(0 \le a,b,c,d \le \left| {X_{3} } \right|/2\). The four variables \(\left( {a,b,c,d} \right)\) represent TP, FN, FP, and TN, respectively. Here P means DCIS and N means healthy breast (HB).

Four simple measures \(\left( {\nu^{1} ,\nu^{2} ,\nu^{3} ,\nu^{4} } \right)\) can be defined aswhere \(\left[ {\nu^{1} \left( z \right),\nu^{2} \left( z \right),\nu^{3} \left( z \right),\nu^{4} \left( z \right)} \right]\) means sensitivity, specificity, precision, and accuracy at z-th run, respectively. Besides, F1 score \(\nu^{5} \left( z \right)\), Matthews correlation coefficient (MCC) \(\nu^{6} \left( z \right)\), and Fowlkes–Mallows index (FMI) \(\nu^{7} \left( z \right)\) can be defined as:

After averaging \(Z\) runs, we can calculate the mean \(\left( M \right)\) and standard deviation (SD) of all k-th \(\left( {\forall k \in \left[ {1,7} \right]} \right)\) measures as

The result is reported in the format of \(M \pm {\text{SD}}\). For ease of typing, we write it in short as MSD.

Table 4 shows the parameter setting of variables in this study. The values were obtained using trial-and-error. The total size of our dataset was 480, and thus the size of the preprocessed image set is \(\left| {X_{3} } \right| = 480\). The number of folds and runs were all set to 10, i.e., \(K = 10, Z = 10\). Then, each fold contained 48 images, that is 24 DCIS and 24 HB images. The training set contained \(\left| C \right| = 432\) images, and the test set contained \(\left| D \right| = 48\) images. The number of DA ways was \(L = 16\), the number of new images for each DA technique was \(W = 30\). Thus, we created \(L \times W = 480\) new images for every training image. The number of conv layers/blocks was \(N_{{{\text{CL}}}} = 4\), and the number of fully connected layers/blocks was \(N_{{{\text{FCL}}}} = 2\).

Table 5 itemizes the LDA parameter settings. The GC factors \(\eta^{{{\text{GC}}}}\) varied from 0.4 to 1.6 with an increase of 0.04, skipping the value of 1. The rotation vector \(\eta^{{{\text{RO}}}}\) was in the value from \(- W\) to \(W\) an increase of 2°, skipping \(\eta^{{{\text{RO}}}} = 0\). Scaling factor \(\eta^{{{\text{SC}}}}\) varied from 0.7 to 1.3 with an increase of 0.02, skipping \(\eta^{{{\text{SC}}}} = 1\). HS factors \(\eta^{{{\text{HS}}}}\) varied from − 0.15 to 0.15 with an increase of 0.01, skipping the value of \(\eta^{{{\text{HS}}}} = 0\). The maximum shift factor \(\Delta = 15\). The maximum color shift value was \(\varpi = 50\).

Table 6 shows the K-fold cross validation setting, which was used in the experiment to report unbiased performances [28]. For each trial, the training image set contained 216 DCIS and 216 HB images. Then after L-way data augmentation, the LDA training set contained 103,896 images for each class, and thus together \(\left| {{\text{DA}}\left( C \right)} \right| = 207,792\) images. The size of the test set during each trial was only 48 images. Combining 10 trials, the final combined test set is the same as the original dataset of 480 images.

The ten runs of our Model-4 results are shown in Table 7. Here it shows using our Model-4 CNN-BDER yielded \(\nu^{1} = 94.08 \pm { }1.22\), \(\nu^{2} = 93.58 \pm { }1.49\), \(\nu^{3} = 93.63 \pm { }1.37\), \(\nu^{4} = 93.83 \pm { }0.96\), \(\nu^{5} = 93.85 \pm { }0.94\), \(\nu^{6} = 87.68 \pm { }1.91\), \(\nu^{7} = 93.85 \pm { }0.94\). In summary, our model-4 showed high accuracy, potentially aiding radiologists to make fast and accurate decisions.

We next compared the Model-4 CNN-BDER result with other four models (Model-0 BCNN, Model-1 CNN-BD, Model-2 CNN-BDE, and Model-3 CNN-BDR). The comparison results are shown in Table 8. Here, Model-4 CNN-BDER yielded the best results among all five models. Note that \(\nu^{2}\) and \(\nu^{3}\) of Model-3 CNN-BDR are quite close to those of Model-4 CNN-BDER, but considering the results were obtained using an average of ten runs, we can still conclude that Model-4 CNN-BDER has higher accuracy than Model-3 CNN-BDR in terms of all seven indicators.

Kruskal–Wallis test was preformed based on Model-4 against Model-(m), where \(m = 0,1,2,3\). The p value result matrix \(P\) is listed in Table 9. The null hypothesis is the indicator vector \(\nu^{n} \left( {n = 1, \ldots ,7} \right)\) of \(Z\) runs of Model-(m) and that of Model-4 come from the same distribution, and the alternative hypothesis that not all samples are obtained from the same distribution. Then we recorded the corresponding p value as \(p\left( {m,n} \right)\). The final matrix \(P = \left[ {p\left( {m,n} \right)} \right],m = 0, \ldots ,3,n = 1, \ldots ,7\). Note here we chose \(Z = 30\). The reason is our data are not normally distributed (see Table 7), so it is important to obtain a larger sample set.

The first row and second row of Table 9 show that all p values are < 0.05. So, the test rejects the null hypothesis at the 5% significance level, indicating that Model-4 is significantly better than Model-0 and Model-1 for all seven indicators. For the third row, the p values show that Model-4 is significantly better than Model-2 for all indicators other than sensitivity \(\nu^{1}\). For the last row, the p values show that Model-4 is significantly better than Model-3 for all indicators other than specificity \(\nu^{2}\) and precision \(\nu^{3}\).

Table 10 presents the results of not using LDA, showing decreased accuracy compared to those using LDA and highlights the effectiveness of our proposed LDA. The future research direction is to explore more types of DA techniques and increase the diversity of LDA, hence, improving the generalization ability of our AI models. Note that Model-0 BCNN and Model-1 CNN-BD without LDA obtain performances lower than 90%, which are worse than traditional AI methods that do not utilize deep learning. This means deep learning with big data can improve performance, if we do not have big data (not using data augmentation means our training set is only 432 images as shown in Table 6), then deep models may not compete with traditional shallow models.

Figure 5 summarizes and compares all ten models, where LDA and NLDA represent use and non-use of LDA, respectively. From Fig. 5 we can clearly observe that our Model-4 CNN-BDER using LDA can obtain the best performance among all six models.

Here we do not run hypothesis test, since all the models without LDA show reduced performance than the models with LDA. We have already proven that the statement “Model-4 is better than Models-(0–3)” is statistically significant, so we can conclude that Model-4 is better than Models without LDA.

Our proposed Model-4 CNN-BDER was compared with state-of-the-art approaches. First, we used the 40-image dataset in Ref. [12]. The comparison results are presented in Table 11. Note here the performance of our Model-4 differs from previous experiments, because we analyzed a smaller dataset (40-images). The reason why our method is better than SMFD [12] is because SMFD, i.e., statistical measure and fractal dimension, can help extract statistical and global texture information, but it is inefficient in extracting local information.

Next, we compared our Model-4 with recent state-of-the-art algorithms on the entire 480-image dataset using 10 runs of tenfold cross validation. The comparison algorithms include NBC [8], CRNBC [7], CRSVM [7], WEE [9], SMMP [10], HMI [11], SMFD [12], WESVM [13]. The comparative results are shown in Table 12. Here Ref. [7] provided two methods, one using naive Bayesian classifier, and the other using support vector machine.

The results in Table 12 showed that our Model-4 CNN-BDER method performed better than eight state-of-the-art approaches. Except MCC \(\nu^{6}\), the other six indicators of our method are greater than 93%. While, the second best method is SMFD [12], whose seven indicator values are all less than 91%. SMFD [12] can help extract statistical and global texture information, but it is inefficient when extracting local information. WEE [9] has a similar problem, since wavelet energy entropy uses wavelet to extract multi-resolution information, and a higher decomposition level of wavelet can extract finer-resolution. But it is difficult to run high-level decomposition in practice. Hence, the information from WEE [9] is mostly at a coarse level. CRNBC [7] and CRSVM [7] used co-occurrence matrix (COM) and run length matrix (RLM) as the feature extraction method, and employed naive Bayesian classifier (NBC) and support vector machine (SVM) as classifiers. COM computes the distribution of co-occurring pixel values at given offsets, while RLM computes the size of homogeneous runs for each grey level. Both features are easy to implement for computer scientists, but their capability of distinguishing tumors from surrounding healthy issues needs to be verified. Also, NBC and SVM are traditional classifiers, whose performances are not as high compared to recent deep learning approaches. SMMP [10] combined self-organizing map (SOM) and multilayer perceptron (MLP) methods. SOM used unsupervised learning to generate a low-dimensional discretized representation of the input space from the training image samples, while MLP has only one hidden layer that may limit its expressivity power. WESVM [13] used wavelet energy support vector machine as the classifier. However, wavelet energy is not a popular feature descriptor, whose improvements and modifications on wavelet energy are still in progress. The two worst methods are NBC [8] and HMI [11]. The former assumes the presence/absence of a feature of a class is unrelated to the presence/absence of any other features; however, this assumption is difficult to fulfil in practice. The latter employed seven Hu moment invariants as feature descriptors, which may be insufficient to capture information regarding breast cancer masses. The performance can be improved by combining with other feature descriptors. In all, Table 12 shows the improved performance of our Model-4 CNN-BDER method.

Three experienced radiologists \(\left( {P_{1} ,P_{2} ,P_{3} } \right)\) were invited to independently inspect our dataset of 480 thermogram images. None of the radiologists had observed any of the images in advance.

The results of three radiologists are itemized in Table 13. The first radiologist \(\left( {P_{1} } \right)\) obtained a sensitivity of 71.67%, a specificity of 74.17%, a precision of 73.50%, and an accuracy of 72.92%. The second radiologist \(\left( {P_{2} } \right)\) obtained the four indicators as 81.25%, 73.75%, 75.58%, and 77.50%, respectively. The third radiologist \(P_{3}\) obtained the four measures as 75.42%, 82.50%, 81.17%, and 78.96%, respectively. Comparing Table 13 with our method Model-4, which is also illustrated in Fig. 6, from which we can see that our proposed CNN-BDER method can give higher performance than manual diagnosis. The reason may be DCIS is Stage 0 of breast cancer, so some lesions are difficult to discern by radiologists while AI can potentially capture those slight and minor lesions.