Two-Exposure Image Fusion Based on Optimized Adaptive Gamma Correction

Abstract

In contrast to conventional digital images, high-dynamic-range (HDR) images have a broader range of intensity between the darkest and brightest regions to capture more details in a scene. Such images are produced by fusing images with different exposure values (EVs) for the same scene. Most existing multi-scale exposure fusion (MEF) algorithms assume that the input images are multi-exposed with small EV intervals. However, thanks to emerging spatially multiplexed exposure technology that can capture an image pair of short and long exposure simultaneously, it is essential to deal with two-exposure image fusion. To bring out more well-exposed contents, we generate a more helpful intermediate virtual image for fusion using the proposed Optimized Adaptive Gamma Correction (OAGC) to have better contrast, saturation, and well-exposedness. Fusing the input images with the enhanced virtual image works well even though both inputs are underexposed or overexposed, which other state-of-the-art fusion methods could not handle. The experimental results show that our method performs favorably against other state-of-the-art image fusion methods in generating high-quality fusion results.

1. Introduction

Image fusion has been a crucial low-level image processing task for various applications, such as multi-spectrum image fusion [1,2], multi-focus image fusion [3], multi-modal image fusion [4], and multi-exposure image fusion [5]. Among these applications, thanks to smartphones’ prevalence with their built-in cameras, multi-exposure image fusion is one of the most common applications. Since most natural scenes have a larger ratio of light to dark than what a single camera shot can capture, a single-shot image usually cannot present details of high dynamic ranges, thus having under- or overexposed parts for the scene. When a camera captures an image, its sensors can only catch a limited luminance range during a specific exposure time, resulting in a so-called low-dynamic-range image. An image taken for short exposure tends to be dark, while it is bright for long exposure, as shown in Figure 1a. Fusing differently exposed low-dynamic-range (LDR) images to obtain a high-dynamic-range (HDR) image requires extracting well-exposed (highlighted) regions from each LDR image to generate an excellent fused image, which has been very challenging.

Several research works have been performed for Multi-scale Exposure Fusion (MEF) [6,7,8]. In general, it is common to fuse LDR images using a weighted sum, where the weight associated with each input LDR is determined in a pixel-wise fashion [6,7,8]. Mertens et al. [6] proposed the fusion of images in a multi-scale manner based on pixel contrast, saturation, and well-exposedness to ease content inconsistency issues in the fused results. However, this often yields halo artifacts in its fusion results. In [7,8], the authors addressed the artifacts by applying modified guided image filtering to weight maps to eliminate halos around edges.

The abovementioned methods produce good results using a sequence of images exposed in a small interval of different exposure values (EV). Thanks to advanced sensor technology, a camera with Binned Multiplexed Exposure High-Dynamic-Range (BME-HDR) or Spatially Multiplexed Exposure High-Dynamic-Range (SME-HDR) technology can simultaneously capture an image pair with short- and long-exposure image sensors. The captured pair has only a negligible difference, possibly caused by local motion blur between them. The existing MEF methods may not work well with two exposure images, since none of the inputs may have well-exposed contents. In addition, weighted-sum fusion based on well-exposedness may not be able to deal with highlighted regions of a short-exposure image that are darker than dark parts in a long-exposure image, resulting in the method ignoring contents in the short-exposure image. Yang et al. [9] proposed the production of an intermediate virtual image with a medium exposure based on an image pair with two exposures to help generate better fusion results. Nevertheless, it does not work in situations where highlighted regions of both input LDR images are not well exposed.

In recent years, deep convolutional neural networks (CNNs) have gained tremendous success in low-level image processing works. In MEF, CNN-based methods [10,11] can better learn features from input multiple-exposure images and fuse them into a nice image. However, the fused images often lack image details [12], since spatial information may be lost when features pass through deep layers. Xu et al. [13] proposed a unified unsupervised image fusion network trained based on the importance and information carried by the two input images to generate fusion results. However, these learning-based methods can only produce a fused image based on the two input images’ interpolation. They cannot deal with cases where both of the input images do not have highlighted regions/contents.

This paper presents a two-exposure fusion framework that generates a more helpful intermediate virtual image for fusion using the proposed Optimized Adaptive Gamma Correction (OAGC). The virtual image has better contrast, saturation, and well-exposedness, and it is not restricted to being an interpolated version of the two input images. Fusing the input images with their virtual image processed by OAGC works well even though both inputs have no well-exposed contents or regions. Figure 1b shows an example where the proposed framework can still generate a good fusion result for when both of the input images lack highlighted regions (Figure 1a). Our primary contributions are three-fold:

• Our image fusion framework adopting the proposed OAGC can produce better fusion results for two input images with various exposure ratios, even when both of the input images lack well-exposed regions.
• The proposed framework with OAGC can also adapt to single-image enhancement.
• We conduct an extensive experiment using a public multi-exposure dataset [14] to demonstrate that the proposed fusion framework performs favorably against the state-of-the-art image fusion methods.

2. Related Work

MEF-based methods produce fusion results using a weighted combination of the input images based on each pixel’s “well-exposedness”. In [15], fusion weight maps were calculated based on the correlation-based match and salience measures of the input images. With the weight maps, one can fuse the input images into one by using the gradient pyramid.

Mertens et al. [6] constructed fusion weight maps based on contrast, saturation, and exposedness of the input images. Differently from [15], the fusion was performed with the Gaussian and Laplacian pyramids. The problem was that using the smoothed weight maps in fusion often causes halo artifacts, especially around the edges. The method proposed in [7] addressed this issue by applying an edge-preserving filter (weighted guided image filtering [16]) to fusion weight maps. Kou et al. [8] further proposed an edge-preserving gradient-domain guided image filter (GGIF) to avoid generating halo artifacts in the fused image. To extract image details, Li et al. [7] proposed a weighted structure tensor to manipulate details presented in a fused image. In general, MEF-based methods can generate decent fusion results.

General MEF algorithms [6,8] that require a sequence of images with different exposure ratios as the inputs may not work with only two input images. Yang et al. [9] proposed the use of the MEF algorithm for two-exposure-ratio image fusion, where an intermediate virtual image with a medium exposure is generated to help produce a better fusion result. However, the virtual image’s intensity and exposedness are bounded by the two input images, which often fails to work for cases where two images are both underexposed and overexposed. Yang’s method [9] can only generate both the intermediate and fusion results with approximate medium exposure between its two input images. The problem is that medium exposure between the inputs may still be under- or overexposure. Image fusion will not improve visual quality. We will discuss this issue more in the next section.

In the following paragraphs, we introduce the techniques adopted in the work of Yang et al., including the generation of the virtual image and fusion weights and the multi-scale image fusion. Before continuing, we define several notations that are used here. Let $\mathbf{I}\in R^{M\times N\times 3}$ be a color image. We denote ${\mathbf{I}}^{\left(c\right)}$ as the color channel c, where $c\in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$ stand for the red, green, and blue channels. $I\left(m,n\right)$ represents the pixel located at $\left(m,n\right)$, where $0\le m<M$ and $0\le n<N$. M and N are the image width and height. Let $\mathbf{Y}$ be the luminance component or the grayscale version of $\mathbf{I}$. Note that the values of images in this paper are normalized to $\left[0,1\right]$.

2.1. Quality Measures and Fusion Weight Maps

In HDR imaging, an image taken at a certain exposure may contain underexposed or overexposed regions, which are less informative and should be assigned fewer weights in multi-exposure fusion. The input’s contrast, saturation, and well-exposedness determine a pixel’s weight at $\left(m,n\right)$ [6]. The contrast of a pixel, denoted by $C\left(m,n\right)$, is obtained by applying a $3\times 3$ Laplacian filter to a grayscale version of the image:

$$C\left(m,n\right) = \left|4Y\left(m,n\right) - Y\left(m-1,n\right)\right. - \left.Y\left(m,n-1\right) - Y\left(m+1,n\right) - Y\left(m,n+1\right)\right|.$$

(1)

Let $\mathbf{C}= \left(\begin{array}{c}C\left(m,n\right)\end{array}\right)$ be the map of the contrast of $\mathbf{I}$; therefore,

$$\mathbf{C}= \left|4\mathbf{Y}-{\mathbf{Y}}_l-{\mathbf{Y}}_r-{\mathbf{Y}}_u-{\mathbf{Y}}_d\right|,$$

where ${\mathbf{Y}}_l$, ${\mathbf{Y}}_r$, ${\mathbf{Y}}_u$, and ${\mathbf{Y}}_d$ are obtained from ${\mathbf{I}}_l$, ${\mathbf{I}}_r$, ${\mathbf{I}}_u$, and ${\mathbf{I}}_d$; i.e., shifting $\mathbf{I}$ one pixel left, right, up, and down, respectively. The saturation of the pixel, denoted by $S\left(m,n\right)$, is obtained by computing the standard deviation across the red, green, and blue channels:

$$S\left(m,n\right) =\sqrt{\frac{1}{3}{\sum }_{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}{\left[I^{\left(c\right)}\left(m,n\right) - \overline{I}\left(m,n\right)\right]}^2},$$

(2)

where

$$\overline{I}\left(m,n\right) =\frac{1}{3}{\sum }_{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}{I}^{\left(c\right)}\left(m,n\right).$$

The well-exposedness of the pixel, $E\left(m,n\right)$, is defined as:

$$E\left(m,n\right) =exp\left[-\frac{1}{2{\sigma }^2}{\sum }_{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}{\left(I^{\left(c\right)}\left(m,n\right) - \xi \right)}^2\right],$$

(3)

where $\sigma =0.2$ and $\xi =0.5$. Essentially, E is a normal distribution centered at $0.5$ with a standard deviation of $0.2$. The maps of saturation and well-exposedness of $\mathbf{I}$ can, respectively, be represented as $\mathbf{S}= \left(S\left(m,n\right)\right)$ and $\mathbf{E}= \left(E\left(m,n\right)\right)$. Next, the weight of the pixel for fusion is computed using:

$$\tilde{W}\left(m,n\right) =C^{{\omega }_c}\left(m,n\right)S^{{\omega }_s}\left(m,n\right)E^{{\omega }_e}\left(m,n\right),$$

(4)

where ${\omega }_c$, ${\omega }_s$, and ${\omega }_e$ can be adjusted to emphasize or ignore one or more measures. Considering a set of P images ${\mathbf{I}}_1,\dots ,{\mathbf{I}}_P$ for image fusion, the weight of this pixel in the $p_{th}$ image is normalized by the sum of the weights across all the images at the same pixel:

$$W_p\left(m,n\right) =\frac{{\tilde{W}}_p\left(m,n\right)}{{\sum }_{p^{\prime }=1}^P{\tilde{W}}_{p^{\prime }}\left(m,n\right)}.$$

(5)

The weight map of the image ${\mathbf{I}}_p$ is represented as ${\mathbf{W}}_p= \left(W_p\left(m,n\right)\right)$.

2.2. Multi-Scale Fusion

In the MEF algorithm [6], a fusion image, $\widehat{\mathbf{I}}$, is obtained through multi-scale image fusion based on the standard Gaussian and Laplacian pyramids. For each input image ${\mathbf{I}}_p$ in the set of ${\left\{{\mathbf{I}}_p\right\}}_{p=1}^P$, the Laplacian pyramid, ${\mathcal{L}}^{\left(l\right)}\left\{{\mathbf{I}}_p\right\}$, and the Gaussian pyramid of its weight map, ${\mathcal{G}}^{\left(l\right)}\left\{{\mathbf{W}}_p\right\}$, in the $l_{th}$ level are constructed by applying the Gaussian pyramid generation [17]. In this level, the overall Laplacian pyramid is collapsed by performing weighted averaging on the Laplacian pyramids from all of the input images in the set:

$${\mathcal{L}}^{\left(l\right)}\{\widehat{\mathbf{I}}\}=\sum _{p=1}^P\left({\mathcal{G}}^{\left(l\right)}\left\{{\mathbf{W}}_p\right\}\odot {\mathcal{L}}^{\left(l\right)}\left\{{\mathbf{I}}_p\right\}\right),$$

(6)

where ⊙ denotes element-wise multiplication. Finally, the fusion image, $\widehat{\mathbf{I}}$, is reconstructed by collapsing the Laplacian pyramids ${\mathcal{L}}^{\left(l\right)}\{\widehat{\mathbf{I}}\}$.

Applying edge-preserving filtering to preserve edges in the weight maps before averaging the Laplacian pyramids in Equation (6) can reduce halo artifacts in fused images. In [9], the GGIF [18] was adopted to smooth the weight maps ${\mathbf{W}}_p$ and to preserve the significant change as well. Let ${\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)$ be the square local patch with a radius of $\rho$ centered at $\left(m_0,n_0\right)$, and let $\left(m,n\right)$ be a pixel in the patch. In ${\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)$, the weight map in the $l_{th}$ level of the $p_{th}$ image, $W_p^{\left(l\right)}\left(m,n\right)$, is the linear transform of the luminance component, $Y_p^{\left(l\right)}\left(m,n\right)$:

$$Y_p^{\left(l\right)}\left(m,n\right) =a_{p,\left(m_0,n_0\right)}^{\left(l\right)}{W}_p^{\left(l\right)}\left(m,n\right) + b_{p,\left(m_0,n_0\right)},$$

(7)

where $a_{p,\left(m_0,n_0\right)}^{\left(l\right)}$ and $b_{p,\left(m_0,n_0\right)}^{\left(l\right)}$ are the coefficients and are assumed to be constant in ${\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)$. $a_{p,\left(m_0,n_0\right)}^{\left(l\right)}$ and $b_{p,\left(m_0,n_0\right)}^{\left(l\right)}$ can be obtained by minimizing the objective function:

$${\mathsf{\Lambda }}_{GIF}=\sum _{\begin{array}{c}m,n\in \\ {\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)\end{array}}\left[\left(a_{p,\left(m_0,n_0\right)}^{\left(l\right)}{W}_p^{\left(l\right)}\left(m,n\right) + b_{p,\left(m_0,n_0\right)}^{\left(l\right)} - \right.\right.{\left.Y_p^{\left(l\right)}\left(m,n\right)\right)}^2-\left.ϵ{\left(a_{p,\left(m_0,n_0\right)}^{\left(l\right)}\right)}^2\right],$$

(8)

where $ϵ$ is a constant for regularization. The variance of the intensities within this local patch, ${\sigma }_{Y_p^{\left(l\right)},\mathsf{\Omega }}^2$, is computed when solving for the coefficients in Equation (8).

In GGIF, a $3\times 3$ local window, $\mathsf{\Psi }$, is applied to the pixels within ${\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)$ for capturing the structure within ${\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)$ by computing the variance within $\mathsf{\Psi }$, ${\sigma }_{Y_p^{\left(l\right)},\mathsf{\Psi }}^2$ [18]. This local window makes GGIF a content-adaptive filter; thus, GGIF produces fewer halos and better preserves the edge than the GIF. In GGIF, the regularization term is designed to yield:

$${\mathsf{\Lambda }}_{GG}=\sum _{\begin{array}{c}m,n\in \\ {\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)\end{array}}\left[\left(a_{p,\left(m_0,n_0\right)}^{\left(l\right)}{W}_p^{\left(l\right)}\left(m,n\right) + b_{p,\left(m_0,n_0\right)}^{\left(l\right)}\right.\right.{\left. - Y_p^{\left(l\right)}\left(m,n\right)\right)}^2-\left.\lambda \frac{{\left(a_{p,\left(m_0,n_0\right)}^{\left(l\right)}-{\zeta }_{\left(m_0,n_0\right)}\right)}^2}{{\mathsf{\Gamma }}_{Y_k}\left(m_0,n_0\right)}\right],$$

(9)

where ${\mathsf{\Gamma }}_{Y_k}\left(m_0,n_0\right)$ and ${\zeta }_{\left(m_0,n_0\right)}$ are computed according to the product of ${\sigma }_{Y_p^{\left(l\right)},\mathsf{\Omega }}$ and ${\sigma }_{Y_p^{\left(l\right)},\mathsf{\Psi }}$ (the standard deviations of the pixels within ${\mathsf{\Omega }}_{\rho }\left(m_0,n_0\right)$ and $\mathsf{\Psi }$), and $\lambda$ is a constant for regularization. The filter coefficients $a_{p,\left(m_0,n_0\right)}^{\left(l\right)}$ and $b_{p,\left(m_0,n_0\right)}^{\left(l\right)}$ can solved by minimizing ${\mathsf{\Lambda }}_{GG}$ in Equation (9).

The fused image $\widehat{\mathbf{I}}$ can be obtained by fusing the Laplacian pyramids of the input images taken at different exposures using the weight maps retrieved from the Gaussian pyramids, ${\mathcal{G}}^{\left(l\right)}\left\{{\mathbf{W}}_p\right\}$. Note that the weight maps are filtered using GGIF, as described in Equation (9), to preserve edges.

2.3. Virtual Image Generation

In [9], Yang et al. proposed the modification of two differently exposed images to have the same medium exposure using the intensity mapping function based on the cross-histogram between two images, called the comparagram (Ref. [19]), and fused them to produce an intermediate virtual image. Let $I_1$ and $I_2$ be the two input images and let ${\mathcal{F}}_{12}$ and ${\mathcal{F}}_{21}$ be the intensity mapping functions (IMFs) that map ${\mathbf{I}}_1$ to ${\mathbf{I}}_2$ and ${\mathbf{I}}_2$ to ${\mathbf{I}}_1$. Based on [19], the IMFs that map the two images to the same exposure, denoted as ${\mathcal{F}}_{13}$ and ${\mathcal{F}}_{23}$, are computed as

$${\mathcal{F}}_{13}\left(z\right)\left(I_i\right)={\left(z{\mathcal{F}}_{12}\left(z\right)\right)}^{0.5},{\mathcal{F}}_{23}\left(z\right)={\left(z{\mathcal{F}}_{21}\left(z\right)\right)}^{0.5},$$

(10)

where z is a pixel intensity. The two modified images with the same exposure are $\widetilde{{\mathbf{I}}_i}={\mathcal{F}}_{i3}\left({\mathbf{I}}_i\right)$, $i=1,2$. The desired virtual image ${\mathbf{I}}_v$ is computed by fusing $\widetilde{{\mathbf{I}}_1}$ and $\widetilde{{\mathbf{I}}_2}$ using the weighting functions adopted in [9]. The two-exposure-fusion image in [9] is obtained by fusing ${\mathbf{I}}_1$, ${\mathbf{I}}_2$, and ${\mathbf{I}}_v$ based on the MEF algorithm [8].

As described previously, Yang’s method often fails to produce a satisfying fusion result when the medium exposure between inputs is still under- or overexposure. The proposed method addresses this issue by improving the contrast, saturation, and well-exposedness for the intermediate virtual image to generate better fusion results under different input conditions.

3. Proposed Method

The algorithm in [9] can work for two images with a large difference between their exposure ratios. In this case, the intermediate virtual image with medium exposure helps bridge the dynamic range gap between the two inputs. Thus, it can improve the quality of the fusion result. However, if the two inputs’ exposure is under- or overexposure, the generated virtual image would not help fusion. Thus, the quality of the fused image is not improved much.

For example, to fuse Figure 2a,b, both of which look overexposed, the virtual image ${\mathbf{I}}_v$ (Figure 2c) generated by [9] with medium exposure between the inputs is still overexposed and, thus, not helpful for the fusion result (Figure 2e). We propose Optimized Adaptive Gamma Correction (OAGC) to enhance the intermediate virtual image to have better contrast, saturation, and well-exposedness (Figure 2d) so that it can improve the fusion quality and produce a better result (Figure 2f).

In OAGC, we derive an optimal $\gamma$ based on the input’s contrast, saturation, and well-exposedness by formulating an objective function based on these image quality metrics and apply it to the input image using gamma correction. Let $Y\left(m,n\right)$ be the luminance of a pixel. One can gamma-correct the image $\mathbf{Y}$ to alter its luminance through the power function as follows:

$$Y_{\gamma }\left(m,n\right) =\eta Y{\left(m,n\right)}^{\gamma },$$

(11)

where ${\mathbf{Y}}_{\gamma }$ is the corrected image, $\eta$ and $\gamma$ are positive scalars, and $\eta$ is usually set to 1 [20]. Here, the notation ${\mathbf{Y}}_{\gamma }$ in bold represents the entire image, while $Y_{\gamma }\left(m,n\right)$ stands for the pixel located at $\left(m,n\right)$. If $\gamma <1$, it stretches the contrast of shadow regions (pixel intensities less than the mid-tone of $0.5$), and features in these regions become discernible, whereas if $\gamma >1$, it stretches the contrast of bright regions (intensities larger than $0.5$), and features in the regions become perceptible. For $\gamma =1$, it is linear mapping.

To derive the optimal gamma, we design an objective function as follows:

$${\displaystyle f\left(\gamma \right) =q_1{∥\widehat{\mathbf{c}}\left(\gamma \right)∥}^2+q_2{∥\widehat{\mathbf{s}}\left(\gamma \right)∥}^2+q_3{∥\widehat{\mathbf{e}}\left(\gamma \right)∥}^2+\delta {∥\widehat{\mathbf{r}}\left(\gamma \right)∥}^2,}$$

(12)

where $\widehat{\mathbf{c}}\left(\gamma \right) =k_c\mathbf{1}-\mathrm{vec}\left({\mathbf{C}}_{\gamma }\right)$, $\widehat{\mathbf{s}}\left(\gamma \right) =k_s\mathbf{1}-\mathrm{vec}\left({\mathbf{S}}_{\gamma }\right)$, $\widehat{\mathbf{e}}\left(\gamma \right) =k_e\mathbf{1}-\mathrm{vec}\left({\mathbf{E}}_{\gamma }\right)$, and $\widehat{\mathbf{r}}\left(\gamma \right) =\mathrm{vec}\left({\mathbf{I}}_v-{\mathbf{I}}_{\gamma }\right)$, where ${\mathbf{C}}_{\gamma }$, ${\mathbf{S}}_{\gamma }$, and ${\mathbf{E}}_{\gamma }$ are the maps of quality measures computed based on the gamma-corrected version of the input image, denoted as ${\mathbf{I}}_{\gamma }$. Here, the virtual image $I_v$ is used as the input, which is ${\mathbf{I}}_{\gamma }:={\mathbf{I}}_v^{\gamma }$. We set $k_c$, $k_s$, and $k_e$ to 4, $0.5$, and 1 according to the upper bounds of the corresponding quality measures (contrast, saturation, and well-exposednesse; refer to the Appendix A for the derivation). The term with $\widehat{\mathbf{r}}\left(\gamma \right)$ in the objective function prevents the corrected image from deviating the input too much. Hence, minimizing the objective function $f\left(\gamma \right)$ is to maximize all three quality measures: the contrast, saturation, and well-exposedness. $q_1$, $q_2$, and $q_3$ are the weighting factors for the contributions from different quality measures (independent from ${\omega }_c$, ${\omega }_s$, and ${\omega }_e$ in Equation (4) and are all set to $\frac{1}{3}$. $\delta$ is a small, fixed scalar and is set to $0.1$ in the present study. $\mathbf{1}$ is the vector of 1s, $\mathrm{vec}\left(·\right)$ is the vectorization of a matrix, and $∥·∥$ represents the 2-norm of a vector. The regularization term is added to avoid possible color distortion caused by gamma correction.

The optimal gamma, ${\gamma }^{\ast }$, which aims to increase contrast, saturation, and well-exposedness simultaneously, can be obtained by minimizing the optimization function $f\left(\gamma \right)$:

$${\gamma }^{\ast }=arg\underset{\gamma }{min}f\left(\gamma \right).$$

(13)

Since there is no closed-form solution for Equation (13), we apply the gradient descent to iteratively approximate it:

$$\begin{array}{cc}\hfill {\gamma }^{(k+1)}={\gamma }^{\left(k\right)} +& {\alpha }^{\left(k\right)}\frac{d}{d\gamma }f\left(\gamma \right)\hfill \\ \hfill ={\gamma }^{\left(k\right)} +& {\alpha }^{\left(k\right)}\left[q_1{\left(\frac{d}{d\gamma }\widehat{\mathbf{c}}\left(\gamma \right)\right)}^{\mathrm{T}}\widehat{\mathbf{c}}\left(\gamma \right)\right. + q_2{\left(\frac{d}{d\gamma }\widehat{\mathbf{s}}\left(\gamma \right)\right)}^{\mathrm{T}}\widehat{\mathbf{s}}\left(\gamma \right)\hfill \\ \hfill +& q_3{\left(\frac{d}{d\gamma }\widehat{\mathbf{e}}\left(\gamma \right)\right)}^{\mathrm{T}}\widehat{\mathbf{e}}\left(\gamma \right) + \left.\delta {\left(\frac{d}{d\gamma }\widehat{\mathbf{r}}\left(\gamma \right)\right)}^{\mathrm{T}}\widehat{\mathbf{r}}\left(\gamma \right)\right],\hfill \end{array}$$

(14)

where

$$\begin{array}{cc}\hfill \frac{d}{d\gamma }\widehat{\mathbf{c}}\left(\gamma \right)& =-\mathrm{sgn}\left[\sum _{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}{t}^{\left(c\right)}\left(4{\mathbf{i}}_v^{{\left(c\right)}^{\gamma }}\right.\right.-\left.\left.{\mathbf{i}}_{v,l}^{{\left(c\right)}^{\gamma }}-{\mathbf{i}}_{v,r}^{{\left(c\right)}^{\gamma }}-{\mathbf{i}}_{v,u}^{{\left(c\right)}^{\gamma }}-{\mathbf{i}}_{v,d}^{{\left(c\right)}^{\gamma }}\right)\right]\hfill \\ \hfill & \times \left[\sum _{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}{t}^{\left(c\right)}\left(4{\mathbf{i}}_v^{{\left(c\right)}^{\gamma }}\odot log{\mathbf{i}}_v\right.\right.-{\mathbf{i}}_{v,l}^{{\left(c\right)}^{\gamma }}\odot log{\mathbf{i}}_{v,l}-{\mathbf{i}}_{v,r}^{{\left(c\right)}^{\gamma }}\odot log{\mathbf{i}}_{v,r}\hfill \\ \hfill & \left.\left.-{\mathbf{i}}_{v,u}^{{\left(c\right)}^{\gamma }}\odot log{\mathbf{i}}_{v,u}-{\mathbf{i}}_{v,d}^{{\left(c\right)}^{\gamma }}\odot log{\mathbf{i}}_{v,d}\right)\right],\hfill \end{array}$$

(15)

with ${\mathbf{i}}_{v,l}$, ${\mathbf{i}}_{v,r}$, ${\mathbf{i}}_{v,u}$, and ${\mathbf{i}}_{v,d}$ being the vectorization of ${\mathbf{I}}_{v,l}$, ${\mathbf{I}}_{v,r}$, ${\mathbf{I}}_{v,u}$, and ${\mathbf{I}}_{v,d}$, as well as $t^{\left(\mathrm{R}\right)}$, $t^{\left(\mathrm{G}\right)}$, and $t^{\left(\mathrm{B}\right)}$ being $0.299$, $0.587$, and $0.114$ respectively.

$$\begin{array}{cc}\hfill \frac{d}{d\gamma }\widehat{\mathbf{s}}\left(\gamma \right)& =-\frac{1}{9}\left[2\sum _{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}\left({\mathbf{i}}_v^{{\left(c\right)}^{\left(2\gamma \right)}}\odot log{\mathbf{i}}_v^{\left(c\right)}\right)\right.\hfill \\ & - {\mathbf{i}}_v^{{\left(R\right)}^{\gamma }}\odot {\mathbf{i}}_v^{{\left(G\right)}^{\gamma }}\odot log\left({\mathbf{i}}_v^{{\left(R\right)}^{\gamma }}+{\mathbf{i}}_v^{{\left(G\right)}^{\gamma }}\right)\hfill \\ & - {\mathbf{i}}_v^{{\left(G\right)}^{\gamma }}\odot {\mathbf{i}}_v^{{\left(B\right)}^{\gamma }}\odot log\left({\mathbf{i}}_v^{{\left(G\right)}^{\gamma }}+{\mathbf{i}}_v^{{\left(B\right)}^{\gamma }}\right)\hfill \\ & - \left.{\mathbf{i}}_v^{{\left(B\right)}^{\gamma }}\odot {\mathbf{i}}_v^{{\left(R\right)}^{\gamma }}\odot log\left({\mathbf{i}}_v^{{\left(B\right)}^{\gamma }}+{\mathbf{i}}_v^{{\left(R\right)}^{\gamma }}\right)\right]\oslash {\mathbf{s}}_{\gamma },\hfill \end{array}$$

(16)

with ⊘ being the element-wise division,

$$\frac{d}{d\gamma }\widehat{\mathbf{e}}\left(\gamma \right)=\frac{{\mathbf{e}}_{\gamma }}{2{\sigma }^2}\odot \sum _{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}\left[\left(2{\mathbf{i}}_v^{{\left(c\right)}^{2\gamma }}-{\mathbf{i}}_v^{{\left(c\right)}^{\gamma }}\right)\odot log{\mathbf{i}}_v^{\left(c\right)}\right],$$

(17)

$$\frac{d}{d\gamma }\widehat{\mathbf{r}}\left(\gamma \right)=\sum _{c\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}}\left[\left({\mathbf{i}}_v^{{\left(c\right)}^{\left(2\gamma \right)}}-{\mathbf{i}}_v^{{\left(c\right)}^{\left(\gamma +1\right)}}\right)\odot log{\mathbf{i}}_v^{\left(c\right)}\right],$$

(18)

and ${\alpha }^{\left(k\right)}$ is the adjustable learning rate.

Figure 3 shows the flowchart of the presented two-exposure image fusion framework, where the two inputs are taken in the same scene at different exposure ratios. The virtual image is first generated using the intensity mapping function [9]. Next, we solve Equation (12) to find the optimal gamma value ${\gamma }^{\ast }$ for the virtual image, which enhances the contrast, saturation, and well-exposedness of ${\mathbf{I}}_v$. The final fused image, $\widehat{\mathbf{I}}$, is obtained by applying the MEF algorithm [8,9] to the fusion of two input images and ${\mathbf{I}}_{\gamma }$.

4. Experimental Results

In the experiment, we compared the proposed method against state-of-the-art image fusion methods, which included Kou’s method [8], DeepFuse [10], Yang’s method [9], and U2Fusion [13]. We adopted the SICE datasest [14] and collected 116 image pairs that consisted of various scenes to evaluate the performance. The presented algorithm was implemented using MATLAB R2019b on a MacBook Pro with an Intel i5 dual-core processor at 2.7 GHz and 8 GB 1867 MHz DDR3 RAM. We present a performance evaluation with a qualitative visual comparison and a quantitative objective assessment in the following.

4.1. Qualitative Assessment

We compared different fusion results under various input conditions. First, Figure 4 shows the fusion results of using the compared image fusion algorithms [8,9,10,13] and our presented framework. As can be seen, one input image is underexposed, and the other is overexposed in the two cases, where the fusion results should have middle exposure between the two inputs. All of the compared methods worked fine in such cases, although U2fusion’s [13] fusion results were a little darker than the others’ results.

Figure 5 shows image fusion cases where the difference between the two input images’ EVs was not large. Thus, fusion methods that can only produce results with medium exposure between the inputs do not work. As shown, all of the compared methods except for the proposed framework output fusion results similar to the input images, and were thus unable to reveal more details than the inputs. In contrast, the proposed framework produced an intermediate virtual image enhanced by OAGC with additional well-exposed highlighted contents and generated better fusion results. Therefore, we can further improve the overall image visibility by revealing details in regions that are too dark or bright.

4.2. Quantitative Assessment

Objectively, we compare the performance of our presented framework against other image fusion methods using five benchmark metrics: the Naturalness Image Quality Evaluator (NIQE) [21], Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [22], No-Reference Image Quality Assessment (NR-IQA) [23], and discrete entropy (DE) [24]. The NIQE [21] is a no-reference image quality metric that is trained on pristine images without subjective scores from humans. Therefore, it can measure image quality degradation if any distortions exist, but is correlated little with human perceptuality. A smaller value means a better quality. BRISQUE [22] is a natural scene statistics-based distortion-generic no-reference image quality assessment model that is trained on images with known distortions and subjective quality scores. It can evaluate losses of naturalness of an image caused by possible distortions. A BRISQUE value ranges between 0 and 100. A smaller value means worse visual quality. NR-IQA [23] is another no-reference image quality metric for HDR images that is constructed using deep CNNs while considering image saliency, and it constructs deep CNNs to extract quality features across the HDR and LDR domains. DE [24] can represent the information contained in an information source, i.e., if an image has higher entropy, it contains more information. Consequently, it is often used to measure the richness of image details. It is defined as:

$$\mathrm{DE}\left(I\right)=-\sum _{l=0}^{L-1}{p}_I\left(l\right)log\left(p_I\left(l\right)\right),$$

(19)

where I is a grayscale image, L represents the largest pixel intensity value, and $p_I\left(l\right)$ is the probability density function of a given grayscale intensity l.

Table 1. Quantitative comparisons of different fusion methods. The scores are averaged over all of the test images. The values in bold represent the best scores.

	NIQE ↓	BRISQE ↓	NR-IQA ↑	DE ↑
Kou [8]	2.7866	22.7140	43.6202	6.9171
DeepFuse [10]	2.9555	31.5365	32.9696	6.9564
Yang [9]	2.7895	23.0660	43.5312	6.8617
U2Fusion [13]	3.421	35.5939	34.5449	6.4195
Proposed	2.7339	21.7120	43.8968	6.9814

shows the quantitative performance of different fusion methods, where the scores are averaged over all of the test images. As can be seen, the results demonstrate that the presented framework achieved the best scores in all four categories, meaning that our fusion results looked natural with the fewest distortions (having the lowest NIQE [21] and lowest BRISQUE values [22]). In assessing the HDR image quality (NR-IQA [23]), our method performed favorably against other fusion methods. Our method could also preserve the most image details in the fusion results (largest DE value [24]).

4.3. Extension to Single Image Enhancement

As stated previously, the proposed framework works well for two-exposure image fusion in cases where the difference between the two input images’ EVs varies. In recent years, fusion-based single image enhancement methods have attracted much attention [14]. We can also extend our framework to single image enhancement by applying OAGC to the input image, $\mathbf{I}$, to yield a quality-improved image ${\mathbf{I}}_{\gamma }$. Then, both $\mathbf{I}$ and ${\mathbf{I}}_{\gamma }$ are fused to obtain an enhancement result.

Figure 6 compares the results obtained using various single image enhancement methods, including global histogram equalization (HE) [25], CVC [26], AGCWD [27], EPMP [28], SICE [14], and the proposed method. As shown, the conventional HE tended to over-enhance/introduce noise to the processed images (Figure 6b), since the input images had over- and underexposed regions. SICE [14] only performed well for the second row of Figure 6, where the input image was underexposed. For the other cases, it tended to overexpose the input images. The other methods [26,27,28] could only enhance the contrast of the input images, while the proposed framework not only did that, but also revealed unseen details from the input and increased the color vividness (Figure 6g).

4.4. Analysis of OAGC

Convergence of Gradient Descent: To further analyze the process of attaining the target gamma value ${\gamma }^{\ast }$ in OAGC, we take the case in the top row of Figure 5 as an example to show the iterative steps of finding ${\gamma }^{\ast }$ for the intermediate virtual image ${\mathbf{I}}_v$. Figure 7 shows that it takes about 66 steps for the objective function $f\left(\gamma \right)$ to converge with gradient descent, and it attains ${\gamma }^{\ast }=0.434$. The value of the objective function changes from the initial $2.690$ to $2.688$. As $\gamma ={\gamma }^{\ast }$, $\widehat{\mathbf{e}}\left(\gamma \right)$ reaches its minimum, and $\widehat{\mathbf{c}}\left(\gamma \right)$ is close to its minimum while $\widehat{\mathbf{s}}\left(\gamma \right)$ is at its maximum, indicating that contrast, saturation, and well-exposedness are all maximized. This also shows that solving the objective function strikes a balance among these three measures.

To further attest to the effectiveness of OAGC, Figure 8 shows the trend of values of the objective function and its quality measure terms using the grid-search method on $\gamma$. As seen, the minimum of the objective function is $2.688$ when $\gamma =0.434$, consistently with the ${\gamma }^{\ast }$ obtained using gradient descent.

Limitation of OAGC: Using OAGC, we can attain a gamma coefficient ${\gamma }^{\ast }$ from the input image by optimizing the objective function in Equation (12), and we can then apply gamma correction to the input to generate a corrected image whose contrast, saturation, and well-exposedness are improved. However, if both of the input images have no content at all for the same regions due to extremely low or high exposure, even OAGC cannot help generate or restore those regions from nothing. Figure 9 shows a failure case of OAGC, where both of the input images are very underexposed and bear little content. Figure 9c shows the intermediate virtual images obtained using the intensity mapping algorithm described in [9], which is similar to the interpolation of the inputs, and the result is still very dark and lacks content. After applying OAGC to it (${\gamma }^{\ast }=0.2015$), the processed virtual image (Figure 9d) presents more details than before. Still, it inevitably has noise in some regions (such as the door and the banner on the façade).

5. Conclusions

This paper presented a two-exposure image fusion framework that utilizes the proposed OAGC to bring out additional well-exposed contents from an intermediate virtual image derived from the two inputs. It can work better for the input images with various combinations of exposure ratios and can produce more well-exposed fusion results. In addition, the proposed framework with OAGC can easily adapt to single image enhancement. The experimental results have demonstrated that the proposed method performs favorably against the state-of-the-art image fusion methods.