3.1.

Deep Learning Model for Binary Segmentation

We designed a deep learning model, which aims to take advantage of all available voxelwise and image-level annotations. We propose to extend a segmentation CNN with an additional subnetwork performing image-level classification and to train the model for the two tasks jointly. Most of the layers are shared between the classification and segmentation subnetworks to transfer the information between the two subnetworks. In this paper, we present the 2-D version of our model, which can be used on different types of medical images, such as slices of a CT scan or a multisequence MRI.

The proposed network takes as input a multimodal image of dimensions $300\times 300$ and extends U-Net,¹¹ which is currently one of the most used architectures for segmentation tasks in medical imaging. The different image modalities (e.g., sequences of an MRI) correspond to channels of the data layer and are the input of the first convolutional layer of the network (as in most of the currently used CNNs for image segmentation). U-Net is composed of an encoder part and a decoder part, which are connected by concatenations between layers at the same scale, to combine low-level and local features with high-level and global features. This design is well suited for the tumor segmentation task since the classification of a voxel as tumor requires comparison of its value with its close neighborhood but also taking into account a large spatial context. The last convolutional layer of U-Net produces pixelwise classification scores, which are normalized by softmax function during the training phase. We apply batch normalization³⁷ in all convolutional layers except the final layer.

We propose to add an additional branch to the network, performing image-level classification (Fig. 2) to exploit the information contained in weakly annotated images during the training. This classification branch takes as input the second to last convolutional layer of U-Net (representing rich information extracted from a local and a long-range spatial context) and is composed of one mean-pooling, one convolutional layer, and seven fully connected layers.

Fig. 2

Architecture of our model for binary segmentation. The numbers of outputs are specified below boxes representing layers. The height of rectangles represents the scale (increasing with pooling operations). The dashed lines represent concatenation operations. The proposed architecture is an extended version of U-Net, with a subnetwork performing image-level classification. Training of the model corresponds to a joint minimization of two loss functions related, respectively, to segmentation and image-level classification tasks.

The goals of taking a layer from the final part of U-Net as input of the classification branch are to guide the image-level classification task and to force the major part of the segmentation network to take into account weakly annotated images. This also helps the optimization process by taking advantage of the connectivity of layers in U-Net, helping the flow of gradients of the loss function during the training (in particular, note the connection between the first part and the last part of U-Net).

The second to last layer of the segmentation network outputs 64 feature maps of size $101\times 101$ from which the classification branch has to output two global (image-level) classification scores (tumor absent/tumor present). We first reduce the size of these feature maps by applying a mean-pooling with kernels of size $8\times 8$ and the stride of $8\times 8$. We use the mean pooling rather than max-pooling to avoid information loss and optimization problems. One convolutional layer, with ReLU activation and kernels of size $3\times 3$, is then added to reduce the number of feature maps from 64 to 32. The resulting 32 feature maps of size $11\times 11$ are the input of the first fully connected layer of the classification branch.

According to our experiments, a relatively deep architecture of the classification branch with a limited number of parameters and a skip-connection between layers yields the best performance. This observation is in agreement with current common designs of neural networks. Deep networks have the capacity to learn more complex features due to applied nonlinearities. The connectivity between layers at different depths helps the optimization process (e.g., Res-Net³⁸). In our case, we use seven fully connected layers with ReLU activations (except the final layer) and we concatenate the outputs of the first and the fifth fully connected layer. The role of this concatenation is similar to the one connecting the first and the last sequence of convolutional layers in U-Net. The concatenation is used before the second to last layer in order to have one layer to process the mixed information (concatenation of two layers) before the final decision in the seventh fully connected layer. We use only one concatenation as the subnetwork is composed of only a few layers while concatenations increase the number of parameters in the network. The last fully connected layer outputs image-level classification scores (tumor tissue absent or present).

The model is trained on both fully annotated and weakly annotated images for the two tasks jointly (segmentation and classification). We can distinguish between three types of training images. First, images containing a tumor and with provided ground truth segmentation are the most costly ones. The second type corresponds to images that do not contain tumor, which implies that none of their pixels corresponds to a tumor. In this case, the ground truth segmentation is simply the zero matrix. The only problematic case is the third one, when the image is labeled as containing a tumor but without provided segmentation.

To train our model, we propose to form training batches containing the three mentioned types of images: $k$ positive cases (containing a tumor) with provided segmentation, $m$ negative cases, and $n$ positive cases without provided segmentation.

Given a training batch $b$ and the network parameters $\theta$, we use a weighted pixelwise cross-entropy loss on images of types 1 and 2: ${\text{Loss}}_s^b(\theta )=-\sum _{i=1}^{k+m}\sum _{(x,y)}{w}_{(x,y)}^i\log [p_{i,(x,y)}^l(\theta )]$, where $p_{i,(x,y)}^l$ is the classification score given by the network to the ground truth label for pixel $(x,y)$ of the $i$’th image of the batch and $w_{(x,y)}^i$ is the weight given to this pixel. The weights are used to limit the effect of class imbalance since tumor pixels represent a small portion of the image. Weights of pixels are set automatically according to the composition of the training batch (number of pixels of each class) so that pixels associated with healthy tissues have a total weight of $t_0$ in the loss function and the pixels of the tumor class have a total weight of $t_1$, where $t_0$ and $t_1$ are target weights fixed manually. It means that if the training batch contains $N_t$ pixels labeled as tumor, then each tumor pixel has a weight of $t_1/N_1$ (the pixelwise weight is high when the number of tumor pixels is low). This type of loss function was used in our previous work.³⁹

The classification loss is a standard cross-entropy loss on all images of the training batch: ${\text{Loss}}_c=-\frac{1}{k+m+n}\sum _{i=1}^{k+m+n}\log [p_i^l(\theta )]$, where $p_i^l$ is the global classification score given by the network to the ground truth global label for the $i$’th image of the batch. In particular, fully annotated images are also used for training of the classification branch in order to transfer the knowledge from the segmentation task to the image-level classification. We do not apply weights on the classification loss as image-level labels are balanced through the sampling of training batches (having a fixed number of nontumor images).

Since both segmentation and classification losses are normalized, we define the total loss as a convex combination of the classification and segmentation losses: $\text{Loss}=a*{\text{Loss}}_s+(1-a)*{\text{Loss}}_c$.

We train our model with a variant of stochastic gradient descent (SGD) with momentum,⁴⁰ used also in our previous work.³⁹ The main differences with the standard SGD are to divide the gradient by its norm and to compute gradients on several training batches in each iteration to take into account many training examples while bypassing GPU memory constraints.

3.2.

Extension to the Multiclass Problem

We extend our model to the multiclass case, where each pixel has to be labeled with one of the $K$ classes, such as the four ones considered in the BRATS Challenge (nontumor, contrast-enhancing core, edema, nonenhancing core). We now assume that image-level labels are provided for each class (absent/present in the image).

Extension of the segmentation subnetwork to the multiclass problem is straightforward by changing the number of final feature maps to match the number of classes. However, image-level labels are not exclusive, i.e., an image may contain several tumor subclasses. For this reason, we propose to consider one image-level classification output per tumor subclass, indicating the absence or presence of the given subclass.

According to our experiments, better performances are obtained when each subclass has its dedicated entire classification branch (Fig. 3). A possible reason is that the image-level classification of tumor subclasses is a challenging task requiring a sufficient number of dedicated parameters.

Fig. 3

Extension of our model to the multiclass problem. The number of final feature maps of the segmentation subnetwork is equal to the number of classes (four in our case). As image-level labels (class present/absent) are not exclusive, we consider one classification branch per tumor subclass.

Training batches are sampled similarly to the binary case, however, each tumor subclass has to be present at least once in each training batch. In our implementation, we store lists of paths of images containing tumor subclasses to sample from these lists during the training of the model.

In the segmentation loss, we empirically fix the following target weights for the four classes (nontumor, nonenhancing tumor core, edema, enhancing-core): $t_0=0.7$, $t_1=0.1$, $t_2=0.1$, and $t_3=0.1$ (all tumor subclasses have equal weight in the loss function). The loss associated with each classification branch is the same as in the binary case and the total classification loss is the average across all classification branches.

4.1.

Data

We evaluate our method on the challenging task of brain tumor segmentation in multisequence MR scans, using the “training” dataset of the BRATS 2018 Challenge. It contains 285 multisequence MRI of patients diagnosed with low-grade gliomas or high-grade gliomas. For each patient, manual ground truth segmentation is provided. In each case, four MR sequences are available (Fig. 4): T1, T1 + gadolinium, T2, and fluid-attenuated inversion recovery (FLAIR). Preprocessing performed by the organizers includes skull-stripping, resampling to $1{\mathrm{mm}}^3$ resolution, and registration of images to a common brain atlas. The resulting volumes are of size $240\times 240\times 155$. The images were acquired in 19 different imaging centers. In order to normalize image intensities, each image is divided by the median of nonzero voxels (which is supposed to be less affected by the tumor zone than the mean) and multiplied the image by a fixed constant.

Fig. 4

Examples of multisequence MRI from the BRATS 2018 database. While T2 and T2-FLAIR highlight the edema induced by the tumor, T1 is suitable for determining the tumor core. In particular, T1 acquired after injection of a contrast product (T1c) highlights the tumor angiogenesis, indicating the presence of highly proliferative cancer cells.

Each voxel is labeled with one of the following classes: nontumor (class 0), contrast-enhancing core (class 3), nonenhancing core (class 1), and edema (class 2). The benchmark of the challenge groups classes in three regions: “whole tumor” (formed by all tumor subclasses), “tumor core” (classes 1 and 3, corresponding to the visible tumor mass), and “enhancing core” (class 3).

Given that all 3-D images of the database contain tumors (no negative cases to train a 3-D classification network), we consider the 2-D problem of tumor segmentation in axial slices of the brain.

4.2.

Test Setting

The goal of our experiments is to compare our approach with the standard supervised learning. In each of the performed tests, our model is trained on fully annotated and weakly annotated images and is compared with the standard U-Net trained on fully annotated images only. The goal is to compare our model with a commonly used segmentation model on a publicly available database.

We consider three different training scenarios, with a varying number of patients for which we assume a provided manual tumor segmentation. In each scenario, we perform a fivefold cross-validation. In each fold, 57 patients are used for the test and 228 patients are used for training. Among the 228 training images, few cases are assumed to be fully annotated and the remaining ones are considered to be weakly annotated with slice-level labels. The fully annotated images are different in each fold. If the 3-D volumes are numbered from 0 to 284, then in $k$’th fold, the test images correspond to the interval $[(k-1)\times 57,k\times 57-1]$, the next few images correspond to fully annotated images and the remaining ones are considered as weakly annotated (the folds are generated in a circular way). In the following, FA denotes the number of fully annotated cases and WA denotes the number of weakly annotated cases (with slice-level labels). In particular, note that the split training/test is on 3-D MRIs, i.e., the different slices of the same patient are always in the same set (training or test).

In the first training scenario, five patients are assumed to be provided with manual segmentation and 223 patients have slice-level labels. In the second and third scenarios, the numbers of fully annotated cases are, respectively, 15 and 30 and the numbers of weakly annotated images are, therefore, respectively, 213 and 198. The three training scenarios are independent, i.e., folds are regenerated randomly (the list of all images is permuted randomly and the folds are generated). In fact, results are likely to depend not only on the number of fully annotated images but also on qualitative factors (for example, the few fully annotated images may correspond to atypical cases), and the goal is to test the method in various settings. Overall, our approach is compared to the standard supervised learning on 60 tests (fivefold cross-validation, three independent training scenarios, three binary problems, and one multiclass problem).

We evaluate our method on both binary segmentation problems (separately for each of three tumor regions considered in the challenge) and the end-to-end multiclass segmentation problem. In each binary case, the model is trained for segmentation and classification of one tumor region (whole tumor, tumor core, or enhancing core).

Segmentation performance is expressed in terms of Dice score quantifying the overlap between the ground truth ($Y$) and the output of a model ($\tilde{Y}$):

In order to measure the statistical significance of the obtained results, we perform two-tailed and paired $t$-tests. Pairs of observations correspond to segmentation scores obtained with the standard supervised learning (U-Net trained on fully annotated images) and with our approach. Dice scores for all patients from fivefolds are concatenated to form a set of 285 pairs of observations. The statistical test is performed for each training scenario and for each segmentation task (three binary problems and one multiclass problem). We consider the significance level of 5%.

4.3.

Model Hyperparameters

4.3.1.

Loss function and training of the model

The main introduced training hyperparameter is the parameter $a$ corresponding to the trade-off between classification and segmentation losses. We report mean Dice scores obtained with a varying value of the parameter $a$ on a validation set of 57 patients (20% of the database used for testing and 80% used for training) in the case with 5 fully annotated cases and 223 weakly annotated cases. Segmentation accuracy obtained for the whole tumor in the binary case is reported in Fig. 5. The peak of performance is observed for $a=0.7$ (improvement of approximately 12 points of Dice over the standard supervised learning on this validation set), i.e., for the configuration, where the segmentation loss accounts for 70% of the total loss. With high values of $a$, the improvement over the standard supervised learning is limited: around 2.5 points of Dice for $a=0.9$. In fact, setting a high value of $a$ corresponds to giving less importance to the image-level classification task and therefore ignoring weakly annotated images. For too low values of $a$, segmentation accuracy decreases too, probably because the model focuses on the secondary task of image-level classification. In the end-to-end multiclass case (Fig. 6), lower values of $a$ seem more suitable, possibly because of an increased complexity of the image-level classification task. In all subsequent tests, we fix $a=0.7$ for binary segmentations problems and $a=0.3$ for the end-to-end multiclass segmentation.

Fig. 5

Mean Dice scores for the “whole tumor” region obtained with a varying value of the parameter “$a$,” corresponding to the trade-off between segmentation and image-level classification losses. Segmentation scores are evaluated on a validation set of 57 MRI in the training scenario, where 5 fully annotated MRI and 223 weakly annotated MRI are available for training. The case $a=1.0$ corresponds to ignoring the classification loss and therefore ignoring weakly annotated images.

Fig. 6

Mean Dice scores for whole tumor and “tumor core” regions obtained with a varying value of the parameter $a$ in the multiclass case. Segmentation scores are evaluated on a validation set of 57 MRI in the training scenario, where 5 fully annotated MRI and 223 weakly annotated MRI are available for training. The case $a=1.0$ corresponds to ignoring the classification loss and weakly annotated images.

Training batches in our experiments contain 10 images, including 8 images with tumors (4 images with provided tumor segmentation and 4 without provided segmentation) and 2 images without tumors. The number of images was fixed to fit in the memory of the used GPUs (Nvidia GeForce GTX 1080 Ti), i.e., to form training batches for which backpropagation can be performed using the memory of the GPU. In each training batch, there are only two images without tumors because most of the pixels of tumor images correspond to nontumor zones.

The parameters $t_c$, corresponding to target weights of classes in the segmentation loss, were fixed manually. In both binary and multiclass cases, we chose $t_0=0.7$, which corresponds to giving a target weight of 70% to nontumor voxels. In fact, tumor pixels represent approximately 1% of pixels of the training batch and, therefore, nontumor pixels account approximately for 99% of nonweighted cross-entropy segmentation loss. With $t_0=0.7$, the relative weight of nontumor pixels is therefore decreased compared to the standard, nonweighted cross-entropy while still giving the nontumor class a high weight to avoid oversegmentation. In the multiclass setting, we fixed the same target weight to all three tumor subclasses, i.e., $t_1=0.1$, $t_2=0.1$, and $t_3=0.1$. As a good convergence of the training was obtained in terms of Dice scores of tumor subclasses, we did not further need to optimize these hyperparameters. Moreover, U-Net trained with these weights and using 228 fully annotated images obtained a mean Dice score of almost 0.87 for whole tumor (last row of Table 1), which is a satisfactory performance for a model independently processing axial slices without any postprocessing.

4.3.2.

Model architecture

One of the most important attributes of our method is the architecture of classification branches extending segmentation networks. We perform experiments to compare our model with alternative types of architectures of classification subnetworks. We report the segmentation accuracy obtained on the previously defined validation set of 57 patients.

In the binary case, we consider two alternative architectures of classification subnetworks. The first one is composed of four fully connected layers having, respectively, 2000, 500, 100, and 2 neurons. It corresponds, therefore, to a shallow variant of the classification subnetwork with a relatively high number of parameters. We name this architecture “shallow” model. The second variant has the same architecture as our model (seven fully connected layers) but with removed concatenation between the first and the fifth fully connected layer. We name this architecture “deep-sequential.” The comparison of segmentation accuracy for the whole tumor obtained by these two variants and by our model is reported in Fig. 7. All three models using a mixed level of supervision obtain a better segmentation accuracy than the standard U-Net using five fully annotated images (64.48). Among the three architectures, the shallow variant yields the lowest accuracy (72.29). Our model obtains the highest accuracy (76.56) and performs slightly better than its counterpart with removed concatenation, deep-sequential model (75.78). The improvements over the standard model and the shallow model were found statistically significant (two-tailed and paired $t$-tests).

Fig. 7

Mean Dice scores for the whole tumor region obtained by the standard U-Net and by different models using a mixed level of supervision. Standard deviations are represented by error bars. The segmentation scores are evaluated on a validation set of 57 MRI in the training scenario, where 5 fully annotated MRI and 223 weakly annotated MRI are available for training. Our model corresponds to U-Net extended with a classification branch composed of seven fully connected layers and containing one skip-connection.

We also report results obtained with an alternative architecture of the multiclass model. In our model, we considered separate classification branches for all tumor subclasses. We consider an alternative architecture, having only one classification branch (with the same architecture as our model for binary segmentation and classification) shared between the three final fully connected layers performing image-level classification. In this configuration, the classification layer of each tumor subclass takes as input the sixth fully connected layer of the shared classification branch. We name this architecture “shared classification.” The comparison with our multiclass model (separate classification branches for all tumor subclasses) on the same validation set as previously is reported on Fig. 8. Our model obtains the highest accuracy for the three tumor subregions while the alternative model (shared classification) obtains higher accuracy than the standard multiclass U-Net for whole tumor and tumor core. The improvements of our model over the standard model were found statistically significant for the whole tumor and tumor core regions. The improvements over the alternative model with mixed supervision (shared classification) were not found statistically significant ($p$-values $>0.05$).

Fig. 8

Mean Dice scores for the whole tumor region obtained by the standard multiclass U-Net and by different multiclass models using mixed level of supervision. The error bars represent standard deviations. The segmentation scores are evaluated on a validation set of 57 MRI in the training scenario, where 5 fully annotated MRI and 223 weakly annotated MRI are available for training. Our model is multiclass U-Net extended with three separate classification branches (for each tumor subclass), each branch having the same architecture as in the binary segmentation/classification problem.

4.4.

Results

The main observation is that our model with mixed supervision provides a significant improvement over the standard supervised approach (U-Net trained on fully annotated images) when the number of fully annotated images is limited. In the two first training scenarios (5 FA and 15 FA), our model outperformed the supervised approach on the three binary segmentation problems (Table 1) and in the multiclass setting (Table 2). The largest improvements are in the first scenario (5 FA) for the whole tumor region, where the improvement is of eight points of the mean Dice score in the binary setting and of nine points of Dice in the multiclass setting. Results on different folds of the second scenario (intermediate case, 15 FA) are displayed in Table 3 for the binary problems and in Table 4 for the multiclass problem. Our approach provided an improvement in all folds of the second scenario and for all tumor regions, except one fold for enhancing core in the binary setting. In the third scenario (30 FA + 198 WA), our approach and the standard supervised approach obtained similar performances. Furthermore, we observe that standard deviations are consistently lower with our approach in all training scenarios and for all tumor subregions. The results obtained with mixed supervision are therefore more stable than the ones obtained with the standard supervised learning.

Table 1

Mean Dice scores (fivefold cross-validation, 57 test cases in each fold) in the three binary segmentation problems obtained by the standard supervised approach and by our model trained with mixed supervision. The numbers in brackets denote standard deviations computed on the distribution of Dice scores for all patients of the fivefolds.

Table 2

Mean Dice scores (fivefold cross-validation, 57 test cases in each fold) obtained by the standard supervised approach and by our model in the multiclass setting. The numbers in brackets denote standard deviations computed on the distribution of Dice scores for all patients of the fivefolds.

Table 3

Results obtained for the three binary problems (whole tumor, tumor core, enhancing core) on different folds in the case with 15 fully annotated images and 213 weakly annotated images. The numbers in brackets denote standard deviations computed on the distribution of Dice scores for all patients.

Table 4

Results obtained in the multiclass setting on different folds in the case with 15 fully annotated images and 213 weakly annotated images. The numbers in brackets denote standard deviations computed on the distribution of Dice scores for all patients.

All improvements were found statistically significant for binary segmentation problems. In the multiclass case, all improvements were found statistically significant except for enhancing core in the first training scenario and for whole tumor in the third training scenario.

Qualitative results are displayed in Figs. 9Fig. 10–11. Each figure shows segmentations of one tumor region (whole tumor, tumor core, and enhancing core) produced by models trained with a varying number of fully annotated and weakly annotated images available for training.

Fig. 9

Comparison of our approach with the standard supervised learning for binary segmentation of the whole tumor region. Each row represents the same test example (first image of Fig. 4) from a different training scenario (5, 15, or 30 fully annotated scans available for training). FA and WA refer, respectively, to the number of fully annotated MRI and weakly annotated MRI (with slice-level labels). The results are displayed on MRI T2-FLAIR sequence. The performance of both models improves with the number of manual segmentations available for training.

Fig. 10

Comparison of our approach with the standard supervised learning for binary segmentation of the tumor core region (test example corresponding to the bottom image of Fig. 4). Each row corresponds to a different training scenario (5, 15, or 30 fully annotated scans available for training). FA and WA refer to the numbers of fully annotated and weakly annotated scans. The results are displayed on MRI T1+gadolinium. The observations are similar to the problem of binary segmentation of the whole tumor region. In particular, in the first training scenario, the standard supervised approach does not detect the tumor core zone in contrast to our method.

Fig. 11

Comparison of our approach with the standard supervised learning for binary segmentation of the “enhancing core” region. Each row corresponds to a different training scenario (5, 15, or 30 fully annotated scans available for training). FA and WA refer to the numbers of fully annotated and weakly annotated scans. The results are displayed on MRI T1+gadolinium. The example shows false positives obtained by the model trained with standard supervision. The number of false positives decreases with the number of fully annotated images available for training. No false positives are observed for our model trained with mixed supervision, in any of the three training scenarios.

Segmentation performance increases quickly with the first fully annotated cases, both for the standard supervised learning and the learning with mixed supervision. For instance, mean Dice score obtained by the supervised approach for whole tumor increases from 70.39, in the case with 5 fully annotated images, to 77.9 in the case with 15 fully annotated images. Our approach using 5 fully annotated images and 223 weakly annotated images obtained a slightly better performance (78.3) than the supervised approach using 15 fully annotated cases (77.9). This result is represented on Fig. 12.

Fig. 12

Illustration of the improvement provided by the mixed supervision for binary segmentation of the whole tumor region (mean Dice scores and their standard deviations). Mixed supervision using 5 fully annotated MRI and 223 weakly annotated MRI obtains a slightly better performance than the standard supervised approach using 15 fully annotated MRI. The improvement provided by the weakly annotated images decreases with the number of available ground truth segmentations.

From Fig. 13, we report cross-validated results obtained with a varying number of weakly annotated while images keeping a fixed number of fully annotated images. This complementary experiment is performed for segmentation of whole tumor in the first training scenario (five fully annotated images). We observe that the improvement slows down with the number of added weakly annotated scans. Inclusion of the first 100 weakly annotated MRIs yields an improvement of approximately five points of the cross-validated mean Dice score (from 70.39 to 75.28), whereas the addition of the remaining 123 weakly annotated images improves this score by three points (from 75.28 to 78.34).

Fig. 13

Mean Dice scores (fivefold cross-validation, 57 test cases in each fold) obtained for binary segmentation of the whole tumor with training on five fully annotated scans and a varying number of weakly annotated scans. The error bars represent standard deviations.

Note that each fully annotated case corresponds to a large 3-D volume with voxelwise annotations. Each manually segmented axial slice of size $240\times 240$ corresponds to 57,600 labels, which represents indeed a huge amount of information compared to one global label simply indicating the presence or absence of a tumor tissue within the slice.

In terms of the annotation cost, manual delineation of tumor tissues in one MRI may take about 45 min for an experienced oncologist using a dedicated segmentation tool. Determining the range of axial slices containing tumor tissues may take 1 to 2 min but can be done without specialized software. More importantly, determining global labels may require less medical expertise than performing an exact tumor delineation and can, therefore, be performed by a larger community.