GANFuse: a novel multi-exposure image fusion method based on generative adversarial networks

In recent years, since the deep learning has aroused extensive concern, deep learning has been applied in image fusion, due to its outstanding ability of feature extraction and universality. The main research theories are divided into the following three categories. (1) Methods combine traditional methods with deep learning. In these methods, deep learning framework functions as a tool to extract image features. Representatively, Liu et al. [20] decomposes the source images into detail layer and base layer, and then utilizes convolutional sparse representation (CSR) to merge these layers. Finally, the fused image is reconstructed by the fused base layer and detail layer. In IFCNN [15], Liu et al. proposed a universal network for image fusion and designed different fusion rule according to different type of source images. (2) These methods regard the convolutional network function as a way to generate weight map which shows the importance of each pixel from source images. For instance, Li et al. [21] uses the VGGNet to extract image features and construct a robust weight map for fusion. (3) Other methods present an end-to-end learning framework for image fusion. Prabhakar et al. [13] proposes an unsupervised deep learning framework for multi-exposure fusion. They utilize a novel CNN architecture and designed a no-reference quality metric as the loss function. In FusionGAN and its variants [16, 22, 23], a generative adversarial network is applied to fuse infrared and visible images. The fused image generated by the generator is forced to have more details existing in the visible image by applying the discriminator to distinguish differences between them.

Although these works have achieved promising progress, there are still some drawbacks. (a) Many existing methods use neural network to extract features and reconstruct these features. However, fusion rules are designed manually. Thus, these methods still have limitations of traditional fusion methods. (b) Unsupervised deep learning methods are implemented by designing a suitable loss function. However, finding an effective loss function is still a challenge. (c) Existing GAN-based fusion methods applied discriminator to force fused image to contain more details in one of the source image, leading to the loss of information from the other source image.

To address these drawbacks, we research an approach to MEF that can preserve more effective information from source images under the framework of GAN. Motivated by the success of the FusionGAN on infrared and visible image fusion, we aim to develop the structure of FusionGAN and make our structure suitable for MEF. In general, there are three improvements in our method.

Generative adversarial net was initially proposed by IanJ Goodfellow et al. in 2014 [26]. Different from conventional neural networks, network training requires a generator (G) and a discriminator (D) to work simultaneously. This framework corresponds to a minimax two-player game. Game players are generator and discriminator network. During training step, the ability of G and D are gradually improved until the two sides to achieve equilibrium. Given the input variable (x), the generator G is used to generate output \(y = G(x)\). Through training process, G can learn a training distribution \(P_{G}(x)\) which approximate to real data distribution \(P_{\mathrm{Data}}(x)\). Then, the discriminator D is trained to determine whether the input is from \(P_{\mathrm{Data}}(x)\) or \(P_{G}(x)\). The purpose of G is to generate a fake data which can fool the D. However, D aims at differentiating between real data and fake data. Through this adversarial relationship, the distribution generated by G will gradually approximate the real data.

The optimization formulation of G is formulated as:where Div denotes the divergence between \(P_{\mathrm{Data}}(x)\) and \(P_{G}(x)\). The function of D can be expressed as:where V(G, D) is defined as follows:

Thus, the optimization formulation of generative adversarial network can be expressed as:

G and D are alternately trained. With the advance of the adversarial process, the data generated by G will be gradually similar to the real data.

GAN is a novel network which can generate more real-like data. However, GAN suffers from unstable training. Since the year of 2014, several works have attempted to solve this problem. For example, deep convolutional GAN [27] defines a set of constraints on the architecture of GAN that makes their model stable to train. For optimizing the unreasonable divergence measurement in original GAN, WGAN [28] introduces the Wasserstein distance to improve the stability of training. To overcome the vanishing gradients problem caused by loss function, least squares GAN (LSGAN) [29] adopts the least squares loss function for the discriminator. StyleGAN [30] embeds the input latent code into an intermediate latent space and proposes two new distribution quality metrics for generator architecture that makes their model more linear, less entangled representation of variation. Conditional GAN (cGAN) extends GAN to a conditional model by feeding auxiliary information such as class labels or data from other modalities into the discriminator and generator [31]. For translating clothing images between two specific clothing categories, Liu et al. [32] proposes category-attribute GAN (CA-GAN) framework, including three discriminators. Overall, in the future, GANs have the potential to apply in many fields.

Learning ability of GAN is depending on the structure of network and the loss function. There are three differences between the GANFuse and FusionGAN. Firstly, we train two discriminators to optimize the generator that makes the fused image to contain more details in extreme exposure image pairs. Secondly, we design a new input mode to improve the discretion ability of discriminators. Thirdly, according to the purpose of MEF, we set pixel intensity loss and gradient loss of source images as generator’s loss function. The proposed network overall architecture is shown in Fig. 2. The Y channel of under-exposure image and the Y channel of over-exposure image are fed to generator (G), and the output of the generator is the Y channel of the fused image.

As mentioned in Sect. 2.1, we input \(|F-S_{2}|\) and \(S_{1}\) into discriminator 1 \((D_{1})\) to distinguish between contribution of \(S_{1}\) in fused image and \(S_{1}\). In the meantime, we input \(|F-S_{1}|\) and \(S_{2}\) into discriminator 2 \((D_{2})\) to distinguish between contribution of \(S_{2}\) in fused image and \(S_{2}\). This input model enhances the differentiating capacity of discriminators, which force generator to generate real-like fused image. In the training phase, the two discriminators are trained simultaneously by testing them against the G. After training procedure, \(D_{1}\) cannot differentiate between the contribution of \(S_{1}\) in fused image and \(S_{1}\), and \(D_{2}\) cannot differentiate between the contribution of \(S_{2}\) in fused image and \(S_{2}\).

The loss function contains two parts, losses of generator and discriminators. The details are presented as follows. The loss function of generator (G) consists of over-exposure image’s loss \(L_{I_{\mathrm{o}}}\) and under-exposure image’s loss \(L_{I_{\mathrm{u}}}\).where \(\gamma \) is used to control the trade-off between over-exposure image and under-exposure image loss. \(L_{I_{\mathrm{o}}}\) is defined as follows:where \(\alpha \) is a weight controlling the trade-off between two terms. \(L_{I_{\mathrm{o}}}^{\mathrm{con}}\) denotes the content loss of over-exposure image, which aims to save the over-exposure image information in the fused image. As mentioned in Sect. 2, we aim to reserve the gradient information and pixel intensities information in fused image. Therefore, \(L_{I_{\mathrm{o}}}^{\mathrm{con}}\) is defined as follows:where the weight \(\sigma \) is used to control the trade-off. h and w presents the height and width of the source image. sum represents element summation of the input. \(\left\| \cdot \right\| _{F}\) is the matrix Frobenius norm, \(\varGamma \) denotes the gradient operation. \(L_{I_{\mathrm{o}}}^{\mathrm{adv}}\) conveys the adversarial loss between G and \(D_{1}\), which is defined as follows:

In order to establish an adversarial relationship between discriminators and generator, we set a negative sign in front of \(D_{1}\).

The second term of \(L_{G}\) presents the loss of under-exposure image, which is defined as follows:where the weight \(\beta \) is used to control the trade-off. Similarly, \(L_{I_{\mathrm{u}}}^{\mathrm{con}}\) is the content loss of \(I_{\mathrm{u}}\) and \(I_{\mathrm{f}}\), which is defined as follows:

\(L_{I_{\mathrm{u}}}^{\mathrm{adv}}\) conveys the adversarial loss between G and \(D_{2}\), which is defined as follows:

Discriminator shortens the difference between fused image and source images. The adversarial loss of \(D_{1}\) and \(D_{2}\) judge the similarity of source images and fused image. The loss function of \(D_{1}\) and \(D_{2}\) are shown as follows:

We regard \(D_{1}(| I_{\mathrm{f}}-I_{\mathrm{u}}|)\) and \(D_{2}(| I_{\mathrm{f}}-I_{\mathrm{o}}|)\) as fake data which is decreased by discriminator, and regard \(D_{1}(I_{\mathrm{o}})\) and \(D_{2}(I_{\mathrm{u}})\) as real data which is increased by discriminator. Thus, there was a negative sign in front of real data.

From Fig. 2, we can see that the whole network structure consists of two discriminators \((D_{1}\) and \(D_{2})\) and one generator (G). In this section, the structure of \((D_{1}\), \(D_{2})\) and G will be introduced.

As for the training data set, we collect 30 pairs of exposure stacks which are available publicly from the Internet [36]. It contains all kinds of conditions. Due to the huge amount of source data, we down-sample the source images and crop them into 7552 patch pairs with the size of \(84\times 84\). We set the learning rate to \(10^{-4}\) and train the network for 5 epochs with all the training patches being processed in each epoch.

After training phase, we can get the fused image in the Y channel. The chromaticity channels of fused image are got by weighting sum of input chromaticity channel values. The main information is presented in the Y channel. Thus, different fusion strategies are applied in literature for Y, Cb and Cr fusion [13, 37]. We can choose different methods to merge RGB channels. However, there is usually a substantial correlation between the RGB channels. Therefore, fusing source image in RGB channels will ignore this correlation and cause obvious color difference. We merge the chromaticity channels of the source image by following the strategy of Prabhakar [37], which is shown as follows:where the \(x_{1}\) and \(x_{2}\) denote the pixel intensities of image pairs. The fused chrominance value is obtained by weighing two chrominance values with \(\tau \) subtracted value from itself. In our work, the value of \(\tau \) is set to 128. The final step is converting \(\mathrm{Y}_{\mathrm{fuse}}\), \(\mathrm{Cb}_{\mathrm{fuse}}\) and \(\mathrm{Cr}_{\mathrm{fuse}}\) channels into RGB image.

The method we proposed is compared against with five representative methods, including GFF [38], DSIFT [39], FLER [40], the gradient-based method (GBM) [36], DeepFuse [13]. GFF is a novel guided filtering-based fusion method for creating a highly informative fused image [38]. In DSIFT, the dense SIFT descriptor is applied as the activity level measurement to extract information from source images [39]. FLER proposed a strategy which brightens the high-light regions in the dark image and darkens the darkest regions in the bright image and finally generates virtual image via intensity mapping functions [40]. In the GBM, two different fusion strategies are applied for chrominance and luminance channels separately [36]. DeepFuse is the landmark multi-exposure fusion method based on deep learning [13].

We firstly perform qualitative comparison experiments on three typical image sequences. Fused results of our method and five comparison methods are shown in Figs. 7, 8 and 9. In this paper, we evaluate the effect of image fusion from two aspects, the overall image visual effect and the detail effect of the image. From the aspect of the overall visual effect, the method we proposed is well proportioned in light distribution and closer to the actual scene. There are local dark areas in the image of compared methods, which will lead to the loss of detail features. As for the detail effect, our method can provide additional texture information in some regions.

From Figs. 7, 8 and 9, we can see the results of the three methods of GFF, DSIFT and FLER, these methods have obvious black regions in the fused image. The fusion results of GBM and DeepFuse are more consistent with human visual perception. However, as we have shown in the red box in the ground truth, they also have some loss of detail textures. To be specific, as can be seen in Fig. 7, the details of the tree in our method are of abundant texture information. The same phenomenon can be found in Fig. 8. In the red box, our method shows that the texture on the wall is more clear and the outline of texture is closer to ground truth. As for the window we marked in Fig. 9, there is a bird pattern in the center part of the mark land. The bird pattern of ours shows the most colorful and clear result.

In the multi-exposure image fusion community, MEF-SSIM [36] is a commonly used metric for quantitative evaluation. In addition, we select SD, PSNR, CC and SCD as metrics. These methods are commonly used in MEF evaluation. We apply these metric to evaluate the source images and fused results of five comparisons methods. These five metrics are introduced as follows.

To prove the effect of creations in our framework, we perform an ablation study on the components of GANFuse. The comparative experiment 1 shows the result of GANFuse without discriminators. In the comparative experiment 2, removing the theory of SCD, we directly feed F and \(S_{1}\) into discriminator 1 and F and \(S_{2}\) into discriminator 2. The qualitative result is displayed in Fig. 11. It is obvious that the result of Fig. 11b is most similar to ground truth, including color and textural detail. Particularly, as presented in red box, tableware of Fig. 11b has better visual effect. And the quantity result is given in Table 2 which also testify to the effect of creations of our method.

Moreover, we perform a parametric study on those important parameters in our GANFuse. In our GANFuse, we mainly set the weight \(\sigma \) to trade off the gradient loss and pixel intensities loss. In the following part, we will show results of different \(\sigma \). In our model, we set \(\sigma \) as 0.1. And the weight \(\sigma \) is 0.3 and 0.5 in the comparative experiment 3 and the comparative experiment 4, respectively. The quantity result is presented in Table 3. In Table 3, we can find that GANFuse owns the best quantity result when the value of \(\sigma \) is 0.1. Due to the fact that the values of the result is approximate, the quality results are almost same. Therefore, the quality results will not show anymore.