Towards Accurate Segmentation of Retinal Vessels and the Optic Disc in Fundoscopic Images with Generative Adversarial Networks

Abstract

Automatic segmentation of the retinal vasculature and the optic disc is a crucial task for accurate geometric analysis and reliable automated diagnosis. In recent years, Convolutional Neural Networks (CNN) have shown outstanding performance compared to the conventional approaches in the segmentation tasks. In this paper, we experimentally measure the performance gain for Generative Adversarial Networks (GAN) framework when applied to the segmentation tasks. We show that GAN achieves statistically significant improvement in area under the receiver operating characteristic (AU-ROC) and area under the precision and recall curve (AU-PR) on two public datasets (DRIVE, STARE) by segmenting fine vessels. Also, we found a model that surpassed the current state-of-the-art method by 0.2 − 1.0% in AU-ROC and 0.8 − 1.2% in AU-PR and 0.5 − 0.7% in dice coefficient. In contrast, significant improvements were not observed in the optic disc segmentation task on DRIONS-DB, RIM-ONE (r3) and Drishti-GS datasets in AU-ROC and AU-PR.

Appendices

Appendix A: Network of a Generator

For the generator, we follow the main structure of U-Net [27] where initial convolutional feature maps are depth-concatenated to layers that are upsampled from the bottleneck layer. U-Net is a fully convolutional network [28] which consists only of convolutional layers and depth concatenation is known to be crucial to segmentation tasks as the initial feature maps maintain low-level features such as edges and blobs that can be properly exploited for accurate segmentation. We modified U-Net to maintain width and height of feature maps during convolution operations by padding zeros at the edges of each feature map (zero-padding). Though concerns could be cast on the boundary effect with zero-padding, we observed that our U-Net style network does not suffer from false positives at the boundary. Also, the number of feature maps for each layer is shrunk to avoid overfitting and accelerate training and inference. Details of the generator network are shown in Table 5. We used ReLU for the activation function [48] and batch-normalization [49] before the activation.

Table 5 Details of the generator network

Appendix B: Network of Discriminators

There is variability in the choice of discriminators by the size of pixels on which the decision is made. We explored several models for the discriminators with different output size as done in the previous work [50]. In the atomic level, a discriminator can determine the authenticity pixel-wise (Pixel GAN) while the judgment can also be made in the image level (Image GAN). Between the extremes, it is also possible to set the receptive field to a K × K patch where the decision can be given in the patch level (Patch GAN). We investigated Pixel GAN, Image GAN, and Patch GAN with two intermediary patch sizes (8 × 8, 64 × 64). Details of the network configuration for discriminators are given in Table 6.

Table 6 Details of the discriminator networks with difference size of receptive fields

Appendix C: Objective Function

Let the generator G be a mapping from a fundoscopic image x to a probability map y, or G : x↦y. Then, the discriminator D maps a pair of {x, y} to binary classification {0, 1}^N where 0 represents machine-generated y and 1 denotes human-annotated y and N is the number of decisions. Note that N = 1 for Image GAN and N = W × H for Pixel GAN with image size of W × H.

Then, the objective function of GAN for the segmentation problem can be formulated as

$$\begin{array}{@{}rcl@{}} L_{GAN}(G,D)&=&\mathbb{E}_{x,y\sim p_{data}(x,y)}[\log D(x,y)]\\ &&+~\mathbb{E}_{x\sim p_{data}(x)}[\log (1-D(x,G(x)))]. \end{array} $$

(2)

Note that G takes the input of an image, thus, analogous to conditional GAN [34], but there is no randomness involved in G. Then, the GAN framework solves the optimization problem of

$$\begin{array}{@{}rcl@{}} G^{*}&=&\arg\min_{G} \left[\max_{D}\mathbb{E}_{x,y\sim p_{data}(x,y)}[\log D(x,y)]\right.\\ &&\left.+~\mathbb{E}_{x\sim p_{data}(x)}[\log (1-D(x,G(x)))]\right]. \end{array} $$

(3)

For training the discriminator D to make correct judgment, D(x, y) needs to be maximized while D(x, G(x)) should be minimized. On the other hand, the generator should prevent the discriminator from making the correct judgment by producing outputs that are indiscernible to real data. Since the ultimate goal is to obtain realistic outputs from the generator, the objective function is defined as the minimax of the objective.

In fact, the segmentation task can also utilizes gold standard images by adding a loss function that penalizes distance from the gold standard such as binary cross entropy

$$\begin{array}{@{}rcl@{}} L_{SEG}(G)&=&\mathbb{E}_{x,y\sim p_{data}(x,y)} \left[-y\cdot \log G(x) \right.\\ &&\left.-~(1-y)\cdot\log (1-G(x))\right]. \end{array} $$

(4)

By summing up both the GAN objective and the segmentation loss, we can formulate the objective function as

$$ G^{*}=\underset{G}{\arg\min} \left[\max_{D} L_{GAN}(G,D)\right] + \lambda L_{SEG}(G) $$

(5)

where λ balances the two objective functions.

Appendix D: Experimental Details

D.1 Image Preprocessing

In the public datasets that are used in our experiments, the number of data is not sufficient for successful training. To overcome the limitation, we augmented the data to retrieve new images that are similar but slightly different to the originals. First, each image is rotated with interval of 3 degrees and flipped horizontally yielding additional 239 augmented images from one original image. We empirically found that 3 degree was sufficient for the rotational augmentation. Finally, photographic information is perturbed by the following formula,

$$\begin{array}{@{}rcl@{}} I_{xyd}^{\prime}&=&(1-\alpha)I_{xy}^{gray} +\alpha I_{xyd},\quad I_{xy}^{gray}\\ &=&0.299I_{xyr}+ 0.587I_{xyg}+ 0.114I_{xyb} \end{array} $$

(6)

where I_xyd is a pixel value in channel d ∈{r, g, b} at coordinate (x, y) in an image I and α is a coefficient that balances a color-deprived image (gray image) and the original image. In our experiment, α is randomly sampled from [0.8, 1.2]. When α is above 1, color in each channel intensifies while subtracting intensity in gray scale. From multiple experiments, we found that these augmentation methods can result in a reliable segmentation for both vessels and the optic discs.

When feeding fundoscopic input images to the network, we computed z-score by subtracting the mean and dividing by standard deviation per channel for each individual image. Formally, the following equation is applied to each pixel,

$$\begin{array}{@{}rcl@{}} z_{xyd}&=&\frac{I_{xyd}-\mu_{d}}{\sigma_{d}},\quad \mu_{d}=\frac{1}{WH}\sum\limits_{x,y}I_{xyd}\quad \sigma_{d}\\ &=&\frac{1}{WH}\sum\limits_{x,y}(I_{xyd}-\mu_{d})^{2} \end{array} $$

(7)

where I_xyd denotes a pixel value in channel d ∈{r, g, b} at (x, y) in an image $I\in \mathbb {R}^{W\times H}$. Since the mean and standard deviation in every image are set to 0 and 1 respectively, the sheer difference in brightness and contrast between the two images is eliminated. Therefore, any meaningful features will not be extracted based upon brightness or contrast during training.

D.2 Validation and Training

After the augmentation of original images, 5% of images are reserved for validation set and the remaining 95% of images are used for training. The models with the least generator loss on the validation set are chosen for comparison. We ran 10 rounds of training in which the discriminator and the generator are trained for 240 epochs alternatively. Total of 2400 epochs during training was enough to observe convergence in the generator loss. We emphasize that training both networks alternatively with sufficient iterations can lead to stable learning in which the discriminator perfectly classifies at first and the generator completely fakes the discriminator by letting the discriminator become gradually weaker.

D.3 Hyper-parameters

As an optimizer during training, we used Adam [51] with fixed learning rate of 2e− 4 and first moment coefficient (β₁) of 0.5, and second moment coefficient (β₂) of 0.999. We fixed the trade-off coefficient in Eq. 5 to 10 (λ = 10).