Methods based on the physical Retinex model are effective in enhancing low-light images, adeptly handling the challenges posed by low signal-to-noise ratios and high noise in images captured under weak lighting conditions. However, traditional models based on manually designed Retinex priors do not adapt well to complex and varying degradation environments. DEANet (Jiang et al., Tsinghua Sci Technol. 2023;28(4):743–53 2023) combines frequency and Retinex to address the interference of high-frequency noise in low-light image restoration. Nonetheless, low-frequency noise still significantly impacts the restoration of low-light images. To overcome this issue, this paper integrates the physical Retinex model with deep learning to propose a joint network model, DEANet++, for enhancing low-light images. The model is divided into three modules: decomposition, enhancement, and adjustment. The decomposition module employs a data-driven approach based on Retinex theory to split the image; the enhancement module restores degradation and adjusts brightness in the decomposed images; and the adjustment module restores details and adjusts complex features in the enhanced images. Trained on the publicly available LOL dataset, DEANet++ not only surpasses the control group in both visual and quantitative aspects but also achieves superior results compared to other Retinex-based enhancement methods. Ablation studies and additional experiments highlight the importance of each component in this method.
Notes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Introduction
In everyday scenarios, as photographic equipment becomes increasingly compact, individuals frequently capture and share images of interest. Due to the diminutive aperture sizes in portable photography devices, limited light transmission often results in images that are dark, lack detail, and offer poor visual quality. While professional cameras can mitigate these issues by employing higher sensitivity settings (ISO), longer exposure times, and flash usage, these adjustments are not without drawbacks. High ISO settings tend to amplify noise, particularly in dark and solid color areas of an image, significantly reducing the peak signal-to-noise ratio (PSNR) and degrading image quality. Extended exposure times allow more light to enter, enhancing brightness but introducing Johnson noise, which can disrupt image clarity. Although using a flash can improve local contrast, it often produces results that are visually displeasing. Traditional methods for enhancing image quality exhibit clear limitations. However, the advent of deep learning has spurred the development of innovative low-light image enhancement techniques in recent years. Despite these advancements, achieving enhanced images that maintain rich detail and high color accuracy remains a challenging endeavor.
Fig. 1
Top column: three natural images captured under different light conditions. Bottom column: our enhanced results
×
Figure 1 illustrates this point with three natural images captured under challenging lighting conditions. The first image was taken indoors with low light during the day, the second captures a backlit scene at sunrise with ample ambient light, and the third was shot in dim outdoor conditions under cloudy skies. The subsequent line displays the enhanced results achieved using the model proposed in this paper, demonstrating notable improvements in image clarity and detail.
Advertisement
In recent years, significant progress had been made in low-light image enhancement. Traditional methods such as histogram equalization [2, 3] and their developments [4, 5] had merely simplified brightness enhancement, resulting in modest improvements in image quality. The Retinex-based methods [6], such as SSR [7] and MSRCR [8], had been widely adopted. These approaches, based on the Retinex decomposition concept, assumed that the scene perceived by the human eye was a product of reflectance and illumination layers. While these methods used manually designed prior filters for image decomposition and enhancement, and the results were visually pleasing in terms of light distribution, they introduced noticeable noise, significantly reducing visual quality. Subsequently, BM3D [9] was employed to remove noise from the illumination and reflection components, but issues with residual noise or overly smoothed details persisted. With the advancement of deep learning, data-driven approaches had enhanced the effectiveness and applicability of low-light image enhancement. R2RNet [10] utilized spatial information of images to improve contrast while preserving details through frequency information. LR3M [11] infused a low-rank prior into the Retinex decomposition process, aiming to effectively enhance low-light images and suppress intense noise. SCI [12] had established a cascaded illumination learning process with weight sharing to achieve low-light image enhancement. URetinex-Net [13] formulated the decomposition problem as an implicit prior regularization model, enhancing low-light images through learned modules. These Retinex-based low-light image enhancement schemes focus on decomposing images with minimal noise while maintaining more detail. However, they overlook the complexity of feature fusion after enhancement. During the image enhancement process, although the feature distribution is preliminarily restored, it still deviates from the true image feature distribution. In DEANet, not only do we combine frequency with Retinex to address the interference of high-frequency noise during the image restoration process, but we also incorporate an adjustment module to resolve the issues of feature fusion in enhanced images. Nevertheless, the performance in reducing the impact of low-frequency noise on low-light image restoration is not optimal.
Based on the above analysis, the challenges in low-light image enhancement are summarized as follows:
How can Retinex theory be leveraged to devise a decomposition network within a neural framework, ensuring the effective separation of lighting and reflective components in both normal and low-light images following neural processing?
How can the degradation in the reflective components of low-light images, which is typically exacerbated by noise interference, be effectively counteracted?
How can the loss of intricate image details in the output of neural networks be averted, thereby guaranteeing the production of high-quality images?
A new deep learning network is proposed in this paper which addresses the problem of low-light image enhancement. The contribution of this paper can be summarized as follows:
1.
The proposed DEANet++ network integrates the Retinex physical model with deep learning techniques to enhance low-light images under non-ideal conditions.
2.
The DEANet++ network comprises three modules: a decomposition module, an enhancement module, and an adjustment module. The decomposition and adjustment modules utilize a novel network structure that combines channel attention, U-Net, and residual network (ResNet) [14] to minimize information loss and enhance the representation of image features. In the enhancement module, a new structure employing channel attention, dense convolutional network (DenseNet) [15], and U-Net [16] is implemented to deeply extract and refine image features and details. The adjustment module is specifically designed to align and integrate complex feature images from the enhancement module, thus more effectively restoring image detail and structure.
3.
New loss functions have been introduced in the enhancement and adjustment modules of the network to more efficiently guide the image enhancement process.
In addition, the decomposition network, the enhancement network, and the adjustment network are trained separately. This approach not only reduces the model’s memory requirements but also allows for flexible adjustments during network training. Numerous experiments have been conducted to verify the effectiveness of the design and demonstrate its superiority over existing technical solutions.
Related Works
In recent years, many low-light enhancement approaches had been proposed. This section briefly introduced the related classic and contemporary technical solutions.
Traditional Methods
For images with overall low brightness, when the distribution of each grayscale value was highly uniform, the amount of information contained in the image was substantial. Conversely, if there was only a single grayscale value, the information content was considerably limited. Techniques such as histogram equalization (HE) [2, 3, 17] and its further developments [4, 5] mapped the grayscale range of an image to [0,1]. This reassigned pixel values across different grayscale ranges, aiming to equalize the number of pixels in each range. Such redistribution balanced the output histogram, prevented excessive concentration of grayscale values, and thus enhanced image contrast. Another approach involved applying nonlinear gamma correction (GC) to each pixel, effectively improving brightness but overlooking the relationships among pixels. Traditional methods primarily focused on enhancing brightness, disregarding real illumination factors, which led to significant discrepancies between the enhanced image and the actual scene. Furthermore, some traditional algorithms started with image decomposition and reconstruction to achieve image restoration. Liang et al. [18] proposed a multi-scale tone mapping scheme based on a layer decomposition model, addressing issues of halo artifacts and over-enhancement. Shibata et al. [19] introduced a gradient-domain image reconstruction framework with intensity range and structural constraints. Similar methodologies were developed by [20‐23].
Advertisement
Lighting-Based Methods
Unlike traditional methods, ligh-ting-based methods were based on the Retinex theory [6]. A lighting-based approach separated normal-light images and low-light images into light components and object reflection components. At the same time, it also ensured that the object reflection components were as close as possible between the normal-light image and the low-light image. Early attempts that adopted this method included single-scale Retinex (SSR) [7], which obtained the illumination component through a single-scale Gaussian filter, and multi-scale Retinex (MSR), which enhanced SSR by performing a weighted average of the results obtained by multi-scale Gaussian filters. MSR maintained high image fidelity and dynamic compression. Based on MSR, Multi-Scale Retinex with Color Restoration (MSRCR) was proposed [8] that added a color restoration factor C to adjust the problem of color distortion caused by the contrast enhancement of the local area of the image. BIMEF [24] used a multi-exposure fusion framework for image enhancement. LIME [25] estimated the lighting through presuppositions, obtained the estimated lighting through a weighted model, and then used block-matching and 3D filtering (BM3D) [9] for post-processing. Wang et al. proposed a method called NPE [26], which enhanced the contrast while maintaining the naturalness of the image lighting. Fu et al. [27] proposed a method that enhanced low-light images by fusing multiple images, but this method sacrificed the realism of some detailed areas of the image. In SRIE [28], it was shown that the reflection and illumination components could be better obtained by using a weighted variational model.
Deep Learning Methods
With the popularity of deep learning, many low-level visual tasks had been improved by the application of deep learning models. For example, different methods were used to perform tasks such as rain removal [29], super-resolution [30] and [31], artifact removal [32], and impurity removal [33].
The task of low-light image enhancement was mainly divided into two categories: supervised and unsupervised.
In supervised learning, LLNet [34] employed a deep learning network, using a stacked sparse denoising autoencoder (SSDA) for enhancing and denoising low-light noisy images. Wei et al. designed RetinexNet [35] that combined the Retinex theory with a convolutional neural network (CNN) to estimate and adjust the illumination map for image contrast enhancement, and then used BM3D [9] for post-processing to achieve denoising. MBLLEN [36] first assigned image feature extraction to different branches through the convolutional layers of a CNN. Second, multiple subnets were used for simultaneous enhancement. Finally, the multi-branch output results were fused into the final enhanced image. KinD [37] decomposed the image into two parts. One component (illumination) was responsible for light conditioning, while the other (reflectance) was responsible for degradation removal. Lim et al. [38] proposed a deep stacked Laplacian restorer (DSLR) that learned the feature mapping from a cell phone to a DSLR camera. KinD++ [39] used a two-stage approach to first decompose the image into reflectance and illumination components and then adjusted the illumination component to brighten the image while preserving its naturalness. Li et al. [40] proposed a progressive-recursive image enhancement network to enhance low-light images. In [41], a noise-suppressing low-light image enhancement approach based on the extent of exposedness at each image pixel was proposed. Lv et al. [42] combined the attention mechanism with a multi-branch convolutional neural network for low-light enhancement on synthetic datasets. Li et al. [43] proposed a neural network—a progressive-recursive image enhancement network (PRIEN)—to enhance low-light images. Recursive units consisting of recurrent layers and residual blocks were used to extract features from the input image. Xu et al. [44] exploited the structure and texture features of low-light images to improve perceptual quality. Lv et al. [45] first employed a sharpening-smoothing image filter (SSIF) for multi-scale decomposition of the image, subsequently applying contrast-limited adaptive histogram equalization (CLAHE) to the decomposed image segments to enhance low-light images effectively. Qian et al. [46] and Zhu et al. [47] were likewise enhanced by supervised methods.
In unsupervised learning, GLAD [48] calculated the global illumination estimate of the low-light input, adjusted the illumination under the guidance of the illumination estimate, and finally supplemented the details by cascading with the original input. Guo et al. [49] proposed a lightweight network, ZeroDCE, for low-light image enhancement. This is a new deep learning method—zero-reference depth curve estimation. Jiang et al. [50] proposed a GAN-based network for low-light image enhancement. Liu et al. [51] proposed a new network called RUAS. This discovered low-light prior architectures from a compact search space by designing a collaborative reference-free learning strategy and used the prior architecture for low-light enhancement. Zhou et al. [52] is also enhanced by unsupervised methods.
In addition, there are some methods that are different from those listed above. MSR-net [53] was a model of multi-scale Retinex combined with a CNN. Ying et al. [54] first obtained a camera response model by analyzing the relationship between images with different exposures. Then, the exposure ratio map of the image was obtained by using the estimation method of the luminance component of the image. Finally, the low-light images were enhanced using the corresponding camera model and exposure ratio map. Yu et al. [55] proposed a physical illumination model which described the degradation of low illumination images to enhance images. CRENet [56] introduced a new re-enhancement module which allows the network to iteratively refine the enhanced image. MAXIM [57] was a novel approach to optimizing MLPs for image processing tasks by introducing convolutional layers and spatial fusion. Similar methods are [58] and [59].
However, the existing models still cannot perfectly address the problem of region degradation; they still suffer from many problems.
In supervised models, some methods used synthetic low-light images for training. However, these synthetic images could not accurately represent real-world illumination conditions, such as spatially varying lighting and degree of noise. Some methods also used pre-defined loss functions to simulate Retinex decomposition, which led to reduced generalization performance of the trained decomposition network. Additionally, some methods idealized the relationship between reflection and illumination components. When generating images, simply multiplying the reflection and illumination components using the Retinex formula led to poor fusion results of complex features.
Although unsupervised approaches demonstrated competitive performance, they continued to face several limitations. First, achieving stable training, avoiding color bias, and establishing relationships across domain-specific information remained challenging for current methodologies. Second, the design of non-reference loss functions became complex when considerations such as color preservation, artifact removal, and gradient backpropagation were integrated. As a result, the visually enhanced images often failed to meet satisfactory quality standards.
This article addresses several challenges associated with supervised low-light enhancement methods. Specifically, it utilizes the LOL dataset to tackle the issue of training with non-realistic images. A data-driven decomposition approach is employed to enhance the transferability of decomposition networks. Furthermore, an adjustment network is implemented to improve the fusion between the illumination and reflection components.
Fig. 2
The proposed DEANet++ architecture consists of several key modules. The decomposition module applies Retinex theory to split the low-light image into lighting and object reflectance components. The enhancement module is designed for the recovery of reflection and the enhancement of lighting. Following this, the enhanced lighting and object reflectance components are processed by the adjustment module, which aims to improve image contrast and detail reconstruction
×
Methodology
An excellent low-light image enhancement model should be able to restore the details of the image, adjust its brightness, and solve the problem of the degradation of reflection components hidden in the dark. A deep convolutional neural network architecture is proposed to achieve this goal. As shown in Fig. 2, the network can be functionally decomposed into three modules: the decomposition module, the enhancement module, and the adjustment module. The decomposition module is used to extract the lighting and reflection components of the image. It uses a data-driven decomposition method which addresses the problem of poor transferability that is caused when using preset constraints to decompose images. The enhancement module is used to reduce the degradation of object reflection and lighting components caused by noise. The adjustment module is used for complex feature fusion, ensuring the restoration of image detail and color. More information about this network is given below.
Further Thoughts and Hypothesis
Analysis of the Retinex Theory
The Retinex theory posits that the color of an object is primarily determined by its reflectivity, an intrinsic property that remains constant regardless of the light intensity or illumination inhomogeneities to which it is exposed. Thus, an object’s image results from the incident light reflecting off its surface. This reflectivity depends solely on the object’s own characteristics and is unaffected by the characteristics of the incident light. According to Retinex theory, an image I comprises a light component L and a reflection component R, mathematically represented as \( I = R \circ L \).
Although designed for image enhancement under ideal conditions, the traditional decomposition model \( I=R\circ L \) does not account for the influence of noise, which can significantly affect both the reflectance and illumination components after decomposition. This paper introduces the DEANet++ network, which integrates the Retinex model with deep learning techniques to effectively address noise interference. Building on the physical model, DEANet++ offers enhanced interpretability over other deep learning networks and improves the performance of the physical model under non-ideal conditions.
Improvements to DEANet
In our previous research, we introduced DEANet [1] to mitigate the impact of noise on the reflective components during low-light image enhancement. The model was initially based on the premise that image noise predominantly affects high-frequency information with minimal impact on low frequencies. However, subsequent empirical studies have shown that even slight noise within the low-frequency domain can adversely affect the reflective components, leading to decreased image quality and blurriness under dim lighting conditions. To address the interference of low-frequency noise and further improve the quality of restored images, DEANet++ has been redesigned. The enhanced module now integrates a channel attention mechanism to effectively extract features and suppress noise. In the final adjustment module, the inclusion of attention mechanisms ensures thorough integration of image features. Ultimately, this paper presents improvements in the network structure and loss functions, significantly enhancing the quality of image restoration under low-light conditions.
Image Decomposition
Traditional methods often decompose images into reflection and lighting components using well-designed, yet rigid constraints that may not be suitable for all applications. In contrast, a data-driven approach to image decomposition has been implemented, effectively overcoming these limitations. During the training phase, pairs of low-light and normal-light images are simultaneously fed into the decomposition network module. This module then processes and separates the images into distinct lighting and reflection components. Trained on a substantial dataset, the module demonstrates enhanced performance in component separation compared to traditional methods restricted by pre-defined rules.
Image Enhancement
In practical scenarios, the degradation observed in low-light images is typically more severe than in normal-light images. This issue extends to the reflection and lighting components derived post-decomposition of these images. Specifically, in the reflection component of objects, the intensity of the lighting component significantly influences the extent of degradation. To address this, an enhancement module is proposed. This module initially learns the mapping relationship between the reflection components of low-light and normal-light objects. Subsequently, it applies this mapping to restore the degraded reflected images.
Image Adjustment
Guided by the principle of \(I = R \circ L\), the adjustment module integrates feature fusion of the reflection and illumination components processed by the enhancement module. This integration facilitates precise tuning of image detail restoration and illumination enhancement. Considering the potential loss of image details in earlier stages within the decomposition and enhancement networks, the adjustment network overlays the enhanced reflection and illumination components onto the original input. This step ensures a comprehensive restoration of image details.
DEANet++
Building on the analysis and assumptions discussed in the previous section, we propose the deep neural network, DEANet++. This network not only ensures the restoration of image details but also enhances the brightness of low-light images. The architecture of DEANet++ is illustrated in Fig. 2.
DEANet++ comprises three subnets: a decomposition module, an enhancement module, and an adjustment module. These modules are responsible for decomposing low-light images, enhancing contrast, and adjusting details, respectively. Specifically, the decomposition module separates the low-light image into lighting and object reflectance components based on Retinex theory. The enhancement module focuses on the recovery of reflection and the enhancement of lighting. The processed components are then forwarded to the adjustment module, which aims to further enhance image contrast and detail reconstruction.
Subsequent sections will provide a detailed description of these modules, examining their functionality and contribution to the overall performance of the network.
Fig. 3
DecomposeNet. This proposed new hybrid network can utilize the skip connections of ResNet [14] and the cross-layer connections of U-Net [16] to minimize the information loss in the convolution process
×
Decomposition Module
Firstly, guided by Retinex theory, the image is decomposed into components R and L through the decomposition network, where R retains the color and details of the image, and L captures the light intensity. Due to the absence of actual lighting guidance, it is challenging to design a constraint a priori that effectively separates the lighting and reflection components from the image. Fortunately, the dataset cited in [35], which contains paired images [\(I_{low}\), \(I_{high}\)] configured at different exposures, facilitates this process. Here, \(I_{low}\) represents a low-light image, and \(I_{high}\), a normal-light image.
As depicted in Fig. 3, the decomposition network leverages the extensive data within the dataset to drive learning and refine an optimal decomposition method. This method enables the network to accurately decompose image pairs under various illumination conditions, resulting in paired object reflection components [\(R_{low}\), \(R_{high}\)] and lighting components [\(L_{low}\), \(L_{high}\)]. In Retinex theory, the object reflection component is expected to remain constant across different illumination intensities, suggesting that \(R_{low}\) and \(R_{high}\) should be identical. However, in practice, the object reflection component exhibits varying degrees of degradation under different illumination intensities, with degradation intensifying as illumination decreases. Consequently, adjustments to the object reflection components aim to align them as closely as possible.
Fig. 4
DenseNet_R. This proposed network combines the powerful information extraction ability of DenseNet [15] and the cross-layer connection of U-Net [16] to capture image features for recovery
×
To enhance the training of the network, several loss functions are employed to ensure accurate image separation. In LIME [25], the light change constraint is derived by weighting the initial light map, determined by the maximum pixel values in the RGB channels. However, this method of weighting the light map does not perfectly adapt to changes in image brightness. Therefore, inspired by LIME, the maximum pixel values across the RGB channels are used as the light map. This single-channel light map is then combined with the low-light image along the channel dimension and fed into the decomposition network. The loss functions specified in Eqs. 1 and 2 ensure the structural correctness of the decomposition for both low-light and normal-light images according to Retinex theory, where \(\left\| \cdot \right\| \) signifies the mean absolute loss (L1 loss). Additionally, the loss functions in Eqs. 3, 4, and 5 are designed to ensure consistency in the object reflections.
Total variation minimization (TVM) is incorporated to impose smoothness constraints on the decomposition network. TVM is commonly utilized in image restoration tasks to minimize the overall image gradient. However, if applied directly as a loss function, it fails in areas with significant gradient variations, due to its assumption of uniform image gradients. To enhance the smoothness of the image post-decomposition, the TVM is modulated by the gradient of the object reflection component, as delineated in Eq. 6. Here, \(\nabla \) represents the image gradient, with horizontal and vertical components denoted by \(\nabla _h\) and \(\nabla _v\), respectively. The smoothing constraint coefficient, \(\lambda _g\), and the exponential weight \(\exp \left( -\lambda _{g} \nabla R_{\textrm{i}}\right) \) adjust the sensitivity of the smooth loss function in regions of abrupt gradient changes. The total loss function of the decomposition module is thus formulated.
The decomposition module employs a novel hybrid network structure that integrates elements of ResNet and U-Net, as depicted in Fig. 3. This hybrid architecture leverages the skip connections from ResNet and the cross-layer connections from U-Net to minimize information loss during the convolution process. The network processes pairs of images, \(I_{low}\) and \(I_{high}\), as input. Initially, under the guidance of the loss function in Eq. 7, the network decomposes \(I_{low}\) to obtain \(R_{low}\) and \(L_{low}\). Subsequently, using weight sharing, it decomposes \(I_{high}\), yielding \(R_{high}\) and \(L_{high}\). Backpropagation is then performed by calculating the loss between the newly generated components (\(R_{low}\), \(L_{low}\), \(R_{high}\), \(L_{high}\)) and the original image pair (\(I_{low}\), \(I_{high}\)). To address feature loss during the image upsampling and subsampling processes, the decomposition network utilizes skip-layer connections within its architecture. Additionally, a channel attention mechanism is incorporated into the skip-layer connections between ResNet and U-Net, enhancing the network’s capacity to emphasize key features by weighting the channels of the convolutional outputs.
Fig. 5
ResNet_L. The light enhancement network learns the mapping between the low-light components and the normal-light components. This mapping can adjust the brightness of the light and adjust the low-light image to the normal-light level
Fig. 6
AdjustNet. The adjustment network is a typical combined network of the five-layer U-Net [15] and ResNet [14]. This adds a skip-layer connection method with a channel attention mechanism and combines the upsampling of nearest-neighbor interpolation and convolution
×
×
Enhancement Module
In the enhancement module, we use a DenseNet_R is used with excellent performance and a simple ResNet_L. These are employed to recover the degraded object reflection component and enhance the lighting component, respectively.
In DenseNet_R, the degradation intensity of the object reflection component varies with different illumination intensities. Consequently, the lighting component \(L_{low}\) and the object reflection component \(R_{low}\) from the low-light image are concatenated along the channel dimension and fed into the enhancement network. Governed by the L1 loss function as specified in Eq. 8, the enhancement network effectively learns the mapping between the object reflection components of the low-light and normal-light images.
The reflection component, initially obtained in the decomposition module, is input into the enhancement module to mitigate degradation. The enhancement module generates a new reflection component, denoted as \(R_{new}\). This process not only restores the degraded reflection but also reduces noise compared to the original low-light reflection component. This mapping effectively eliminates the degradation caused by noise between the reflection component of the low-light object and that of the normal-light object. Furthermore, Eq. 9 ensures that the gradients’ changes in the reflected components of both objects are identical, implying that the boundary contours of the reflection components are aligned. Ultimately, the restoration network employs \(\mathcal {L}{R-S S I M}={\text {SSIM}}\left( R{\text{ new }}, R_{\text{ high }}\right) \), where SSIM (\(\cdot \), \(\cdot \)) measures structural similarity. This metric confirms that the reflected components of both objects are structurally and perceptually similar. The loss associated with the recovery of the object reflection component is expressed in Eq. 10.
$$\begin{aligned} \mathcal {L}_{R-\text{ res }}= & \left\| R_{\text{ new } }-R_{high}\right\| _{1} \end{aligned}$$
(8)
$$\begin{aligned} \mathcal {L}_{R-\text{ smooth }}= & \left\| \nabla R_{\text{ new } }-\nabla R_{\text{ high } }\right\| _{2}^{2} \end{aligned}$$
(9)
$$\begin{aligned} \mathcal {L}_{R}= & \mathcal {L}_{R-\text{ res }}+\mathcal {L}_{R-\text{ smooth }}+\mathcal {L}_{R-S S I M} \end{aligned}$$
(10)
The network for object reflection component recovery is based on the combination of DenseNet and U-Net, as shown in Fig. 4. This proposed network combines the powerful information extraction ability of DenseNet and the cross-layer connection of U-Net to capture image features for recovery. In the DenseNet_R network, DenseNet is combined with UNet in the following way. The first layer is a Resnet_block that subsamples the input feature map. The second to the fifth layers are Dense_block blocks, which contains {2, 4, 12, 8} Conv_blocks. This network utilizes the powerful feature extraction capability of DenseNet and can obtain sufficient features in the subsampled process. At the same time, U-Net’s skip-layer connection is used to minimize the loss of features in the convolution process. Like the decomposition module, the channel attention mechanism is also added to the U-Net skip-layer connection. This can efficiently transfer effective information to the upsampling layers to ensure that the final generated image has excellent detail information and better visual perception.
Fig. 7
Visual comparison with low-light image enhancement methods on the LOL [35] Dataset
Fig. 8
Visual comparison with low-light image enhancement methods on the LOL [35] Dataset
×
×
As shown in Fig. 5. The ResNet_L network is a five-layer simple convolutional neural network. In order to effectively enhance the lighting component, the \(L_{low}\) obtained by the decomposition module is enhanced under the constraint of the L1 loss function in Eq. 11. In other words, the light enhancement network learns the mapping between the low-light components and the normal-light components. This mapping can adjust the brightness of the light and adjust the low-light image to the normal-light level. \(L_{new}\) represents the newly generated lighting component. Finally, the loss function in Eq. 12 is also used in this network to ensure that the gradient change of \(L_{new}\) equals that of \(L_{high}\). The loss of the light enhancement network can be expressed as Eq. 13.
$$\begin{aligned} \mathcal {L}_{L-r e s}= & \left\| L_{\text{ new } }-L_{\text{ high } }\right\| _{1} \end{aligned}$$
(11)
$$\begin{aligned} \mathcal {L}_{L-s m o o t h}= & \left\| \nabla L_{\text{ new } }-\nabla L_{\text{ high } }\right\| _{2}^{2} \end{aligned}$$
Note that RetinexNet, KinD, KinD++, STANet, and DEANet are Retinex-based methods
Bold indicates the best results
Adjustment Module
In the adjustment module, a network structurally similar to the decomposition module is proposed, which is primarily designed for the complex fusion of image features and the fine-tuning of image details and illumination intensity. Relying solely on \(I=R\circ L\) to merge the newly generated object reflection component \(R_{new}\) and the new lighting component \(L_{new}\) from the enhancement network is insufficient for achieving intricate feature fusion. Consequently, this paper introduces an adjustment network to address this shortfall. Moreover, the original input \(I_{low}\) is reintegrated into the adjustment network to mitigate the feature loss encountered in the earlier decomposition and enhancement stages. This loss often results in diminished image details and color inaccuracies. Thus, reintegrating the original input into the final adjustment network helps refine details and preserve color fidelity. Additionally, a channel attention mechanism is incorporated into the U-Net’s skip connections within the adjustment network. This mechanism effectively reduces noise and mitigates low-light interference introduced by the original input, thereby enhancing the network’s capability to represent significant features.
Fig. 9
Visual comparison with other low-light image enhancement methods on LIME [25] and NPE [26] datasets
×
To ensure that the adjustment network accurately restores image features, a content loss function is incorporated, utilizing the pre-trained Visual Geometry Group (VGG) 19 network [60]. This network is employed to simultaneously extract feature pairs [\(Feature_{new}\), \(Feature_{real}\)] from the generated image and the normal-light image. The L1 loss, as defined in Eq. 14, is applied to these feature pairs to assure feature similarity. Here, \(Feature_{new}\) denotes the features extracted from the image \(I_{new}\) by the adjustment network, while \(Feature_{real}\) refers to the features extracted from the normal-light image \(I_{high}\).Additionally, the adjustment network implements the L1 loss constraint outlined in Eq. 15 between the generated image and the normal-light image pair [\(I_{new}\), \(I_{high}\)]. This constraint is crucial for ensuring the consistency of color reproduction in the generated image. The overall loss function of the adjustment network is defined in Eq. 16.
$$\begin{aligned} \mathcal {L}_{A-\text{ content }}= & \Vert \text{ Feature}_{\text{ new }}-\text{ Feature}_{\text{ real }} \Vert _{1} \end{aligned}$$
(14)
$$\begin{aligned} \mathcal {L}_{A-\text{ res } }= & \left\| I_{\text{ new } }-I_{\text{ high } }\right\| _{1} \end{aligned}$$
As depicted in Fig. 6, the adjustment network integrates a typical configuration of the five-layer U-Net and ResNet. This architecture incorporates a skip-layer connection strategy enhanced with a channel attention mechanism and employs both nearest-neighbor interpolation and convolution for upsampling. These techniques are strategically utilized to establish mappings that are critical for the restoration of image details and color accuracy.
Fig. 10
Detailed zoom-in comparison of different low-light enhancement methods on the LOL [35] dataset
×
Experiment
Dataset Details
The LOL dataset [35], meticulously curated through adjustments in camera exposure times and ISO settings, consists of 500 authentic pairs of low/normal illumination images. This dataset, which includes 485 pairs designated for training and 15 pairs earmarked for testing, is a quintessential resource for low-light image enhancement. Unlike datasets composed of artificially synthesized images, the LOL dataset offers a more faithful representation of real-world low-light conditions and serves as a pivotal resource for research in authentic low-light image processing scenarios.
In contrast, LIME, NPE, and MEF are often utilized as benchmark datasets to evaluate low-light image enhancement methods. These smaller-scale datasets lack reference images.
Experimental Details
DEANet++ is implemented using the PyTorch framework, and experiments are conducted with the LOL dataset. To enhance the model’s accuracy and convergence speed, an initial max-min normalization is applied to each image. Subsequently, each image undergoes random cropping and is resized to \(128\times 128\) pixels to bolster the model’s generalization capabilities. Additionally, the cropped images are subjected to random vertical and/or horizontal flips, as well as random rotations at right angles, to further augment the dataset variability. The entire model is trained on an NVIDIA RTX 3060 GPU. The Adam optimizer is utilized with an initial learning rate of 0.0002, and after 1000 epochs, the model reaches its optimal state. Further details will be available in the forthcoming code release.
Comparison with Existing Methods on Real Datasets
The model proposed in this study is benchmarked against a comprehensive control group, including MSRCR [8], BIMEF [24], LIME [25], RetinexNet [35], MBLLE [36], KinD [37], DSLR [38], KinD++ [39], STANet [44], GLAD (unsupervised) [48], ZeroDCE (unsupervised) [49], EnlightenGAN (unsupervised) [50], R2RNet (unsupervised) [10], and DEANet [1], across four public datasets: LOL [35], LIME [25], NPE [26], and MEF [61].
To evaluate image quality comprehensively, five widely used metrics are employed: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) [62], feature similarity index (FSIM) [63], mean absolute error (MAE), and gradient magnitude similarity deviation (GMSD) [64]. PSNR assesses overall image quality, while SSIM evaluates image distortion and similarity between images. FSIM focuses on feature-based quality assessment, MAE quantifies the mean of absolute errors between predicted and observed values, and GMSD measures local image quality via local gradient magnitude similarity and calculates its standard deviation for a global quality assessment. The proposed model demonstrates superior performance, securing the highest ranks in four metrics and the second-highest in one. Although a leading evaluation index does not always correlate with better visual perception, the visual quality produced by our model significantly surpasses that of the control group..This superior visual effect is illustrated in Figs. 7 and 8. Specific evaluation index values are detailed in Table 1, and a set of control images is presented to delineate the experimental nuances. Notably, the blue-boxed section is magnified in Figs. 9 and 10, showcasing the enhanced detail and visual perception achieved by our results.
Table 2
Quantitative comparison on LIME, NPE, and MEF datasets in terms of NIQE [65]
As demonstrated in the graphs, the proposed model, DEANet++, significantly outperforms the control group methods in both peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) on the LOL dataset. DEANet++ achieves the best performance with scores of 22.542 dB in PSNR and 0.850 in SSIM. This represents an improvement over the second-best method, KinD++, by 1.242 dB in PSNR and 0.038 in SSIM. It surpasses the third-best method, STANet, by 1.192 dB in PSNR and 0.045 in SSIM, the fourth-best method, KinD, by 1.816 dB in PSNR and 0.050 in SSIM, the fifth-best method, DEANet, by 1.282 dB in PSNR and 0.062 in SSIM, and the sixth-best method, MBLLEN, by 3.645 dB in PSNR and 0.015 in SSIM. Notably, methods like MSRCR, LIME, RetinexNet, and ZeroDCE are significantly less effective. It is observed that some methods based on Retinex theory, such as LIME and RetinexNet, tend to blur details and/or amplify noise. The enhanced results produced by DEANet++ not only improve both local and global contrast but also exhibit sharper detail and higher color accuracy while effectively mitigating noise in images. These improvements are clearly reflected in the experimental outcomes.
Since LIME, NPE, and MEF consist solely of low-light images, conventional quantitative evaluations using PSNR and SSIM are not applicable. Therefore, the performance on these datasets is assessed using a no-reference image quality evaluation method, the Natural Image Quality Evaluator (NIQE) [65]. NIQE measures the deviation in the multivariate distribution of the test images. The results are tabulated in Table 2. Visual comparisons for other datasets are illustrated in Fig. 9.
Application Experiment
The experimental results demonstrate the application of enhanced low-light images to an advanced visual task: object detection. For robust validation, YOLOv3 [66], a widely utilized object detection algorithm known for its substantial recognition rate and broad applicability, is employed. Both the original low-light images and the enhanced low-light images serve as inputs for YOLOv3. As depicted in Fig. 11, it is evident that the object detector identifies more objects in the enhanced low-light images compared to the original low-light images.
Fig. 11
Application of enhanced low-light images for advanced visual tasks. The first line shows the result of Yolov3 [66] detecting objects in the low-light images. The second line shows the result of Yolov3 detecting objects in the enhanced low-light images, where it can be seen that more objects are detected compared to the first line. The third line shows the result of Yolov4 detecting objects in the normal lighting images
Fig. 12
a Input image. b, c Denote images learned by decomposing network. d, e Denote images learned by decomposing network and enhancing network. f The result generated by the adjustment network
×
×
Ablation Experiment
We conduct ablation studies on the composition of the network modules, network structure, and loss functions to demonstrate the rationality and effectiveness of the DEANet++ architecture.
Initially, ablation experiments are performed on the three network modules using the LOL dataset, with the results depicted in Fig. 12. To evaluate the significance of each module, both the enhancement and adjustment modules are removed sequentially to observe the impact on model performance. The results, as evidenced in Table 3, indicate that removing either the enhancement or adjustment modules significantly compromises the model’s effectiveness, underscoring their importance in the DEANet++ structure.
Second, ablation experiments are conducted on the network structure.
1.
In the decomposition, enhancement, and adjustment networks, the ResNet and DenseNet architectures are sequentially removed, compelling the model to rely solely on the U-Net structure for experiments. As U-Net serves as the foundational network, it is retained throughout. The results of these modifications are summarized in Table 4. The removal of the ResNet structure yields experimental results with a PSNR of 20.360 and an SSIM of 0.781. When the DenseNet structure is eliminated, the results show a PSNR of 18.412 and an SSIM of 0.776. These outcomes validate the efficacy and rational design of the newly proposed network structure.
2.
Furthermore, removing the attention mechanisms from individual modules did not result in significant changes; therefore, the channel attention mechanisms were subsequently removed from all three modules. The results post-removal are recorded as PSNR = 21.324 dB and SSIM = 0.799.
3.
Additionally, the process of concatenating with the original input was omitted in the adjustment module to assess its impact. The findings indicate that such concatenation indeed contributes to enhancing the quality of the final generated image, with experimental results documenting a PSNR of 20.720 dB and an SSIM of 0.801.
Table 3
Comparison of module ablation experiments
Operation
SSIM
PSNR
W/o the adjustment module
0.780
19.700
W/o the enhancement module
0.692
16.431
Normal
0.850
22.542
Note that the best results are highlighted in bold
Table 4
Comparison of network structure ablative experiments
Operation
SSIM
PSNR
W/o the ResNet
0.781
20.360
W/o the DenseNet
0.776
18.412
W/o the channel attention
0.799
21.324
W/o the original input
0.801
20.720
Normal
0.850
22.542
Note that the best results are highlighted in bold
Finally, ablation experiments are performed on the loss function. The loss functions are divided into three categories: content loss, MAE loss, and smooth loss. Content loss is \(\mathcal {L}_{A-\text{ content }}\). MAE loss includes \(\mathcal {L}_{R-\text{ res }}\), \(\mathcal {L}_{L-r e s}\), and \(\mathcal {L}_{A-\text{ res } }\). Smoothing loss includes \(\mathcal {L}_{\text{ smooth-loss } }\), \(\mathcal {L}_{R-\text{ smooth }}\), and \(\mathcal {L}_{L-s m o o t h}\). A combination of several sets of loss functions is set up; the results are shown in Table 5. For each module, different loss functions were ablated to assess their individual contributions. Due to the numerous combinations, we selected the most significant ones for our ablation study. Moreover, for methods based on the Retinex model, the decomposition loss function is a prerequisite, and therefore, ablation on this was not feasible. The results indicate that loss functions play a crucial role in guiding the model to learn image enhancement effectively.
Conclusions
This paper introduces an end-to-end low-light image en-hancement network model grounded in Retinex theory. The network comprises three primary modules: a decomposition module, an enhancement module, and an adjustment module. Within the decomposition module, the network adeptly decomposes images under various lighting conditions. The enhancement module includes an object degradation recovery network that effectively restores the degradation induced by poor illumination in low-light images and a lighting adjustment network that significantly enhances image contrast. The adjustment module conducts complex feature alignment, not only perfectly restoring the details and color accuracy with the support of the original image but also generating high-quality images.
DEANet++ has demonstrated exceptional performance on the LOL dataset, achieving the highest PSNR and SSIM scores. This underscores the effectiveness of the proposed method, which utilizes illumination guidance to mitigate object reflection degradation and noise, integrates the recovered object degradation components with illumination enhancement, and subsequently applies these improvements to enhance low-light images. The versatility of the proposed model extends beyond its current application; it has shown impressive results in dehazing images, restoring clarity to seabed images, and eliminating image artifacts. Furthermore, when combined with algorithms for object detection and face recognition, DEANet++ significantly broadens its utility in low-light environments, thereby expanding its applicability in the field of computer vision.
Table 5
Comparison of loss function ablation experiments
Operation
SSIM
PSNR
MAE loss
0.792
23.771
MAE loss (Enhancement Module) + content loss (Adjustment Module)
0.819
21.242
Smoothing loss (Enhancement Module) + MAE loss (Adjustment Module)
0.823
21.962
MAE loss + content loss + smoothing loss
0.850
22.542
Note that the best results are highlighted in bold
Declarations
We hereby declare that there are no conflicts of interest in the submission of this manuscript and that the content is an original work created by the authors. All co-authors have taken responsibility for critically reviewing and revising its content.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
