Multi-Task Learning Approach Using Dynamic Hyperparameter for Multi-Exposure Fusion

Abstract

High-dynamic-range (HDR) image synthesis is a technology developed to accurately reproduce the actual scene of an image on a display by extending the dynamic range of an image. Multi-exposure fusion (MEF) technology, which synthesizes multiple low-dynamic-range (LDR) images to create an HDR image, has been developed in various ways including pixel-based, patch-based, and deep learning-based methods. Recently, methods to improve the synthesis quality of images using deep-learning-based algorithms have mainly been studied in the field of MEF. Despite the various advantages of deep learning, deep-learning-based methods have a problem in that numerous multi-exposed and ground-truth images are required for training. In this study, we propose a self-supervised learning method that generates and learns reference images based on input images during the training process. In addition, we propose a method to train a deep learning model for an MEF with multiple tasks using dynamic hyperparameters on the loss functions. It enables effective network optimization across multiple tasks and high-quality image synthesis while preserving a simple network architecture. Our learning method applied to the deep learning model shows superior synthesis results compared to other existing deep-learning-based image synthesis algorithms.

1. Introduction

In general, the dynamic range of the human visual system is approximately 10,000 nits, whereas the dynamic range of existing display devices has a range of approximately 100 nits; therefore, there is a limit to realizing realistic images on a display. Moreover, low-dynamic-range (LDR) images cannot adequately depict a real scene because of the limited response to the high-dynamic-range (HDR) of natural light. To display HDR images on standard dynamic range (SDR) devices, such as computer monitors and televisions, two common techniques are used: tone-mapping operators (TMOs) and multi-exposure fusion (MEF).

TMOs compress the dynamic range of an HDR image into several visible LDR images, allowing HDR scenes to be rendered on an LDR display. There are two types of TMOs: global and local. Global operators use identical nonlinear functions to compress the entire image, making operators fast to compute. Reinhard et al. [1] developed an automated method for mapping HDR world images to LDR images based on a zone system. They used a local averaging logarithmic operator to map the entire image and implemented an automated dodging and burning algorithm to correct bright and dark regions. Duan et al. [2] proposed a histogram adjustment method for displaying HDR images, based on a global TMO, called histogram-adjustment-based linear to equalized quantizer (HALEQ).

However, TMOs can cause a loss of local contrast and detail because they treat the entire image as a whole without considering differences in brightness and contrast within the image [3]. Local operators can reproduce a better quality of the tone-processed image; however, these algorithms are more complicated than global operators, and the results can be unrealistic because of artifacts.

The MEF technique was developed to create HDR images from multiple LDR images with varying exposures and to realize digital images that resemble real eyes [4]. The MEF technique can implement an image with a wider visual brightness range by appropriately synthesizing several LDR images, and thus, realistic images can be implemented on the display device [5,6]. As shown in Figure 1, unlike under- or over-exposed images that may have saturated areas, the MEF image can depict finer details by combining information from multiple source images.

MEF techniques can be divided into traditional and deep-learning-based methods. Traditional methods include the pixel-based MEF method and the patch-based method. While researchers have developed various traditional methods, with the rise of deep learning, deep-learning-based methods have become mainstream. Deep-learning-based methods have several advantages, such as flexibility and scalability. However, convolutional neural networks (CNNs) have inherent problems in that it is difficult to preserve extracted features due to the defection in establishing long-range dependencies [8], and large amounts of datasets are required to train networks. Moreover, owing to the lack of sufficient multi-exposure training data and ground-truth images for HDR, many unsupervised MEF methods have been proposed.

In this study, we present a method for training MEF networks by setting up multiple tasks based on our image features. Although it is difficult to train due to the difficulty of preserving extracted intricate features with CNNs architectures, it can be overcome by learning several reference images with well-revealed features. These reference images are composed of features that provide useful information for fusion by reflecting the characteristics of the multi-exposure images. Moreover, by applying dynamic hyperparameters to each multitask loss function, the MEF networks can be trained to effectively reproduce HDR images. The contributions of our study are summarized as follows:

• To train MEF networks that require learning, we present a method for setting up a customized dataset;
• To generate multiple tasks based on source features, we perform a procedure that filters unnecessary regions from the source images using multi-exposure image characteristics;
• To reflect the information between multiple tasks, we set dynamic hyperparameters on the loss functions. This helps the network reproduce better-contrast images;
• To produce a high-quality image with a simple network design, we prove that it is possible to utilize multiple tasks and dynamic hyperparameters for the loss function.

The remainder of this paper is organized as follows: Section 2 introduces the existing methods used in our work. Section 3 describes the implementation of the proposed method. Section 4 provides the quantitative and qualitative results of the proposed model as well as other deep-learning-based image fusion algorithms. Finally, we present our conclusions in Section 5.

2. Related Works

2.1. Multi-Exposure Image Fusion

Among the traditional image synthesis methods, there are two major techniques: pixel- and patch-based methods. The pixel-based method synthesizes multi-exposed images according to specific pixel-wise fusion rules. This method selects the best pixels from each source image based on certain criteria and then generates weight maps to combine these pixels. The acquisition of pixel weight maps from source images is typically based on measures of image quality such as sharpness, contrast, and color fidelity. In addition, the weight maps of source images can be obtained in various ways.

Bruce [9] calculated entropy in a circular area centered on each pixel with a radius from the source image. They assigned a weight to each pixel based on its information entropy and merged source images based on these weights. Song et al. [10] proposed an HDR fusion method using a probabilistic model that preserved the calculated image luminance levels and suppressed reversals in the image luminance gradients. Lee et al. [11] calculated the weights for each pixel in the source images by analyzing the relative pixel intensity and global gradient. The relative pixel intensity determines the brightness of each pixel in the image, whereas the global gradient measures the contrast between the adjacent pixels. While these methods have the advantage of obtaining fused images by calculating accurate pixel weight maps, the main drawback of pixel-based methods is the tendency to introduce artifacts in the final fused image, such as noise and halos [12]. Moreover, they cannot properly represent the edge information of images [13].

In the patch-based method, the source images are first divided into small patches or sub-images. Each patch is then processed independently to produce a fused image. This approach allows for better handling of the local differences in brightness and contrast.

Huang et al. [14] introduced a patch-based MEF method that divides an image patch into three independent parts: contrast extraction, structural preservation, and intensity adjustment. They fused these parts separately and reconstructed the desired patch, which was then fed back into the fused image. Wang et al. [15] proposed a method that uses a super-pixel segmentation approach to divide the input images into non-overlapping patches composed of pixels with similar visual properties. This method has the advantages of avoiding the blocking effect and preserving the color attributes of source images. The main advantage of these methods is that the weight map has less noise than pixel-based MEF methods because it combines the neighborhood information of the pixels. However, the performance of patch-based methods is sensitive to the choice of patch size and overlap as well as the characteristics of the source images, making it challenging to achieve consistent and reliable results across different images and applications.

2.2. Deep-Learning-Based Multi-Exposure Image Fusion

As shown in Figure 2, deep-learning-based image fusion networks typically consist of three characteristic stages: feature extraction, feature fusion, and feature reconstruction. Feature extraction involves the extraction of high-level features from source images using deep neural networks. These features represent the source images in high-dimensional space and are used to capture the complex and abstract features of the images. The feature-fusion stage involves combining the extracted features from each source image to obtain a fused feature map. This can be performed in different ways such as using element-wise summation, $l_1$-norm, or other fusion strategies [7]. Finally, in the feature-reconstruction stage, the fused feature maps are fed into a decoder network to reconstruct the final fused image.

The main advantage of deep-learning-based image fusion networks is that they can learn the optimal feature representations for image fusion directly from the data without the need for handcrafted features or explicit rules. Additionally, deep-learning-based approaches can be trained end-to-end, which enables the joint optimization of all stages of the image fusion process. However, the performance of deep-learning-based image fusion networks depends on the training dataset’s quality and size and the network architecture’s complexity.

In the MEF field, deep-learning-based image fusion methods are divided into two categories: supervised learning and unsupervised learning. In supervised learning, a large number of ground truth images are required for training even though the MEF ground truth is insufficient. To overcome the lack of dataset images, the generation of the ground truth was studied together. Kalantari et al. [16] combined three exposed LDR images to generate a ground-truth HDR image. Subsequently, they trained a deep CNN model to reproduce the HDR image from a set of images aligned with the optical flow. However, the generated ground truth images are not real, and their use may reduce the fusion performance. Therefore, many researchers have attempted to train networks in an unsupervised manner.

Unsupervised learning MEF methods usually modify network structures to extract informative features from source images. For example, Xu et al. [17] proposed U2Fusion, which extracts features with a pre-trained VGGNet-16 and utilizes DenseNet for fusion. The authors obtained various shapes of feature maps using a convolutional layer before the max-pooling layer. This could preserve deep-level features such as spatial structures as well as shallow features such as textures and details.

Qu et al. [8] introduced a transformer-based MEF framework called the TransMEF. They designed a transformer module in addition to a CNN module for training to address the inherent limitations of CNNs. By combining the CNN and transformer modules, the network can extract both local and global information from a pair of source images. Although these models have demonstrated good fusion performance and the ability to produce high-quality fused images, training their complex network architecture still requires numerous datasets. This can lead to slow training and increased memory use. Hence, finding the correct balance between network complexity and performance is crucial for developing effective fusion networks.

3. Proposed Methods

3.1. Framework Overview

As shown in Figure 3a, our network is based on the DenseFuse [7] architecture. We focused on optimizing the network by setting up multiple informative target images.

During the training phase, the classified multi-exposed images were resized to 256 × 256 pixels, and only the luminance images of the LDR images entered each channel in the encoder. At the same time, new target images that contain useful information of the input images are generated using the MaskBuilder block. In addition, ParamTuner creates weight values that are multiplied by $Los{s}_{un}$ and $Los{s}_{ov}$ to optimize the network effectively.

As observed in Figure 3a, the encoder consists of two channels. The DenseBlock contains three ConvLayer blocks, in which the output of each layer is connected to the other layers, and each ConvLayer of the network contains one reflection padding layer, namely a convolutional layer with a kernel size of 3 × 3, and a ReLU activation function layer, as shown in Figure 3a,b. This structure preserves the information of the source images as much as possible and allows the network to be trained easily. The features of the under-/over-exposed images are extracted from the encoder block through the above convolution layers, and they are simply added. The fused feature maps enter the decoder, and the fused image is finally reconstructed using four ConvLayer blocks. The reconstructed image learns the correct-exposed image of the pair. In addition, we applied masks generated from MaskBuilder to the reconstructed image to perform multi-task self-supervised learning.

3.2. Dataset Acquisition

Networks that synthesize multi-exposure images can be trained in a supervised manner using ground truth images or in an unsupervised manner using similarity metric-based loss functions to retain the features of the source images [18]. Recently, owing to the lack of sufficient ground truth images in the MEF field, many researchers have investigated the unsupervised reconstruction of high-quality images [8]. To reconstruct higher-quality fusion images without reference images, it is important to learn the various features of each multi-exposure image.

We categorized multi-exposure images into three groups, i.e., under-exposure, over-exposure, and correct exposure, and paired these three types of exposure images. As shown in Figure 4, to acquire large amounts of exposure pairs, we first fixed the appropriate correct-exposed image from the correct exposure group; second, we set the image that was captured with less exposure time than the correct-exposed image as an under-exposed image, and the others that were captured with a higher exposure time than the correct-exposed image as an over-exposed image. Because the correct-exposed image contains some information about the under-exposed and over-exposed images as well as its own information, we initially aimed to converge a reconstructed image of LDR images into a correct-exposed image.

3.3. MaskBuilder

MaskBuilder is designed to create multi-task images because only the correct-exposed image is not sufficient to learn sufficient multi-exposure image features. The under-exposed image contains meaningful information in a relatively bright area, as the dark portion of the image is saturated; conversely, the over-exposed image contains useful information on a relatively dark area, as the bright portion of the image is saturated, as shown in Figure 5. Based on this characteristic, we devised masks that filter unnecessary information from the input LDR images.

To determine whether the regions of the inputs are instructive, we blurred the images of each input image as follows:

$$G\left(x, y\right)=\frac{1}{2\pi {\sigma }^2}exp\left(-\frac{x^2+y^2}{2{\sigma }^2}\right)$$

(1)

$$I_g\left(i, j\right)={\displaystyle \sum }_{y=-m}^m{\displaystyle \sum }_{x=-m}^{m}I\left(i+x, j+y\right)G\left(x, y\right)$$

(2)

where $G\left(·\right)$ represents a two-dimensional Gaussian function, $I\left(i,j\right)$ represents the pixel value of the input image at $\left(i, j\right)$, $\sigma$ is the standard deviation, $m$ represents the kernel size of the Gaussian filter, and $I_g$ is the blurred image. The higher the $\sigma$ values that blur over a wider radius, the greater the amount of regions of the input images that can be removed. Thus, it is required to find an appropriate $\sigma$ value. In Section 4, we arbitrarily set the $\sigma$ value to 10, 20, and 50 and compared the results to determine the most suitable $\sigma$ value. Lastly, the kernel size was conventionally set using Equation (3),

$$m=6\sigma +1$$

(3)

Since the final mask should remove saturated areas from incorrect-exposed images, it should have a value of 0 for saturated areas and a value of 1 for unsaturated areas. Thus, min-max normalization was utilized to generate the final mask ${\widehat{I}}_g$ using Equation (4),

$${\widehat{I}}_g\left(i, j\right)=\frac{I_g\left(i, j\right)-mi{n}_{I_g}}{ma{x}_{I_g}-mi{n}_{I_g}}$$

(4)

where $ma{x}_{I_g}$ and $mi{n}_{I_g}$ are the maximum value of $I_g$ and the minimum values of $I_g$, respectively.

The generated mask eliminates dark regions of the inputs or outputs through multiplication. However, in the case of over-exposed images, the mask eliminates bright regions of the images because there is useful information in the dark region. Thus, we obtained the other filter mask ${\widehat{I}}_g^{\prime }$ for the over-exposed images using Equation (5).

$${\widehat{I}}_g^{\prime }\left(i, j\right)=1-\frac{I_g^{\prime }\left(i, j\right)-mi{n}_{I_g^{\prime }}}{ma{x}_{I_g^{\prime }}-mi{n}_{I_g^{\prime }}}$$

(5)

where $I_g^{\prime }$ indicates the blurred over-exposed image in Equation (5), and $ma{x}_{I_g^{\prime }}$ and $mi{n}_{I_g^{\prime }}$ indicate the maximum value of $I_g^{\prime }$ and the minimum values of $I_g^{\prime }$, respectively. A diagram of the mask building is shown in Figure 6.

For the generated masks ${\widehat{I}}_g$ and ${\widehat{I}}_g^{\prime }$ in Figure 6, grayscale-normalized images were used because it is difficult to visually examine the original images.

3.4. Loss Function

The proposed model has three different types of target images. To learn each feature of the targets, multi-loss functions were designed, and $Los{s}_{total}$ is defined as follows:

$$Los{s}_{total}={\lambda }_{1}Los{s}_{un}+{\lambda }_{2}Los{s}_{ov}+Los{s}_{co}$$

(6)

where $Los{s}_{un}$, $Los{s}_{ov}$, and $Los{s}_{co}$ are the losses between the different multiexposed images and outputs, and ${\lambda }_1$ and ${\lambda }_2$ are the hyperparameters that control the ratio of $Los{s}_{un}$ and $Los{s}_{ov}$, respectively.

Each loss consists of two loss functions that can preserve the salient information of the target image as follows:

$$Loss=Los{s}_{mse}+\alpha Los{s}_{ms–ssim}$$

(7)

$$Los{s}_{mse}=\Vert I_{out}-I_{target}{\Vert }_2$$

(8)

$$Los{s}_{ssim}=1-MS\text{-}SSIM\left(I_{out}, I_{target}\right)$$

(9)

where $Los{s}_{mse}$ denotes the mean square error (MSE) loss, $Los{s}_{ms–ssim}$ denotes the multi-scale structural similarity (MS-SSIM) loss [19], $MS$-$SSIM\left(·\right)$ denotes the MS-SSIM operator between the output and target, and $\alpha$ is a hyperparameter used to match the magnitude of the MSE and MS-SSIM losses. The $\alpha$ was experimentally set to 1000 according to [7].

3.5. ParamTuner

Although we classified multi-exposure datasets into under-exposed, over-exposed, and correct-exposed images, we realized that the exposure value (EV) difference between the under-exposed and correct-exposed images and the difference between the over-exposed and correct-exposed images are not consistent. This is because LDR images have different exposure times even though they belong to the same category. Thus, we assumed that it is necessary to set the dynamic ${\lambda }_1$ and ${\lambda }_2$ values for each multi-exposed image pair. For example, let us assume that five multi-exposed images, which consist of 0 EV, ±1 EV, and ±2 EV, can be paired, as shown in Figure 7.

There is an identical EV difference between the correct-exposed and incorrect-exposed images in Figure 7a. However, in Figure 7b,c, the difference between the correct-exposed and under-exposed images is larger than that between the correct-exposed and over-exposed images or vice versa. We expected that the larger EV difference between the correct-exposed and incorrect-exposed images means that the incorrect-exposed image has more significant information than the other incorrect-exposed image; thus, the network can be trained more effectively by setting a higher lambda value on the incorrect-exposed image loss.

However, because of the challenge of measuring the relative exposure value difference between the correct-exposed image and the under-exposed or over-exposed image, we employed an adaptive gamma value inspired by adaptive gamma correction (AGC) [20]. The AGC method can adequately evaluate the gamma value of an image by classifying the image’s contrast as follows:

$$g\left(I\right)=\left\{\begin{array}{c}{\varrho }_1, D\le 1/\tau \\ {\varrho }_2, otherwise \end{array}\right.$$

(10)

where $I$ is an input image, and ${\varrho }_1$ and ${\varrho }_2$ are low-contrast class and high- (or moderate-) contrast class, respectively. $D$ is defined as $diff\left(\left(\mu +2\sigma \right),\left(\mu -2\sigma \right)\right)$, $\tau$ is a parameter used to define the contrast of an image, and $\sigma$ and $\mu$ are the standard deviation and mean of the image intensity, respectively. The input image can be categorized differently based on the $\tau$, and we determined $\tau$ = 3 to be the appropriate value by referring to [20].

After classifying the constant class, the gamma value of the image was calculated according to its group. Equations (11) and (12) show $\gamma$ of the image belonging to ${\varrho }_1$ group and belonging to ${\varrho }_2$, respectively.

$$\gamma =-{\log }_2(\sigma )$$

(11)

$$\gamma =exp[\frac{1-\left(\mu +\sigma \right)}{2}]$$

(12)

In ParamTuner, each adaptive gamma value of the multi-exposed images was calculated using the AGC algorithm. Next, the difference in gamma values between the correct-exposed image and the under-exposed or over-exposed image was computed separately. Finally, ${\lambda }_1$ and ${\lambda }_2$ were determined using the normalized ratio function. A diagram of this strategy is shown in Figure 8.

In Figure 8, $I_{un}$, $I_{co}$, and, $I_{ov}$ are under-exposed, correct-exposed, and over-exposed images, respectively. ${\gamma }_{ov}$, ${\gamma }_{co}$, and ${\gamma }_{ov}$ are the gamma values of the multi-exposed images, and $D_1$ and $D_2$ are defined as $diff\left({\gamma }_{un},{\gamma }_{co}\right)$ and $diff\left({\gamma }_{ov},{\gamma }_{co}\right)$, respectively. The final value of ${\lambda }_i\left(i=1, 2\right)$ was calculated using Equation (13).

$${\lambda }_i=\beta \frac{D_i}{{{\displaystyle \sum }}_{j=1}^k{D}_j}$$

(13)

where $\beta$ is a hyperparameter that controls the ratio of each incorrect-exposed image, and $k=2$ indicates the number of lambda. The value of $\beta$ was set to two to account for the entire loss ratio.

3.6. Image Fusion Process

When two images with different exposures are captured with a single sensor camera, the boundaries of the two images may not match because of changes in the shooting conditions, such as movement of the subject, which may cause blending problems (e.g., ghost, double boundary, blurring, etc.). Therefore, we aligned two LDR images using scale-invariant feature transform (SIFT) [21] and homography [22] before image fusion. The SIFT algorithm is a widely adopted technique for feature extraction, and it can extract distinctive invariant features between two LDR images, which can be aligned by homography based on the extracted features.

In image processing, homography matching involves matching two images with different viewpoints to obtain the same viewpoint through homography conversion. Homography transformation refers to a series of processes for obtaining a transformation matrix that maps points on a two-dimensional plane to another plane in a three-dimensional space and transforms the points.

$$\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right)=\left(\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right)\times \left(\begin{array}{c}x\\ y\\ 1\end{array}\right)$$

(14)

Equation (14) shows the equation for homographic transformation that maps a point $\left(x, y, 1\right)$ in a three-dimensional space to $\left(x^{\prime }, y^{\prime }, 1\right)$. The $h_{11}$ through $h_{33}$ 3$\times$ 3 matrices denote a homography matrix. To homographically match two images from different viewpoints, a process of finding keypoints of each image and matching these keypoints through a descriptor is required, which can be performed using the SIFT algorithm [21]. The SIFT algorithm mainly consists of four parts: scale-space extrema detection, keypoint localization, orientation assignment, and keypoints and descriptors [23]. After calculating each keypoint and descriptor in two different images, keypoint pairs with the highest similarity were obtained by feature mapping. The final aligned image was computed by a homography matrix using these keypoint pairs.

However, it is sometimes difficult to extract features between under-exposed and over-exposed images because of the dissimilarity in the exposure values. Thus, we applied AGC [20] to correct the image dissimilarity. A diagram of the proposed image-fusion scheme is shown in Figure 9.

After aligning the LDR images, the color space of the images was converted from RGB to CIELAB. Because CIELAB has excellent color separation based on human vision, it preserves the color components of each input image. In Figure 9, $l_{un}$ and $l_{ov}$ denote the luminance channels of the under-exposed and over-exposed images, respectively. C11, C12, C2, C3, C4, and C5 are the trained ConvLayers. DB11 and DB12 are the trained DenseBlocks. $l_{fused}$ is determined from $l_{un}$ and $l_{ov}$ by trained neural networks; $a_{fused}$ and $b_{fused}$ are simply calculated by adding half of $a_{un}$ and $a_{ov}$ or $b_{un}$ and $b_{ov}$, which are the color channels of the under-exposed and over-exposed images. Finally, the fused image was obtained by converting the color space into an RGB space.

4. Experimental Results

In this section, we analyze our approach and compare it with other deep-learning-based image fusion methods. A total of 460 image pairs (under-/over-/correct-exposed images) from the open dataset [24,25] and our acquired dataset were used as the training datasets. All training images were resized to 256 × 256. The network was trained for 40 epochs with a batch size of 1 and a learning rate of 1 × 10^-4. The training networks and all experiments were implemented with an NVIDIA RTX 2060 GPU and Intel Core i5-6500 CPU.

To confirm the effect of the sigma value of the Gaussian filter in MaskBuilder, we compared each $Los{s}_{total}$ as shown in Figure 10. As observed, a larger value of sigma leads to a relatively unstable convergence loss, and the best loss is also the lowest when the sigma value is 10. Therefore, our method set the sigma value to 10.

We compared the fusion results of our method with those of three deep-learning-based image fusion methods [7,8,17]. Fourteen under-/over-exposed image pairs were selected from the previous available dataset [26,27] for the experiment, and the fused images of each method were evaluated using 10 image-quality metrics: average gradient (AG), edge intensity (EI), feature mutual information (FMI), LPC-SI, S3, spatial frequency (SF), perceptual image quality evaluator (PIQE), MS SSIM, Qabf, and visual information fidelity for fusion (VIFF). The AG metric computes the gradient, which contains details and texture information in the image [28]. EI is a metric that represents the image quality and sharpness based on the edge intensity value [29]. FMI determines the mutual feature information between the source and fused images [30]. LPC-SI can detect the sharpness of visual images in the complex wavelet transform domain [31]. S3 shows how images have sharper areas considering human cognitive characteristics [32]. SF reflects the distribution of gradients to compute the amount of detail and texture in the fused image [33]. PIQE calculates image quality using a spatial mask, with a smaller score indicating better perceptual quality [34]. The MS SSIM is a structural similarity index measure combined with multiscale information of the source image [19]. The Qabf metric reflects the quality of visual information from source images, and a larger score indicates a better fusion effect [35]. The VIFF uses VIF models to assemble visual information from the source and fused images and measure the effective visual information of fusion [36].

As shown in

Table 1. Comparison of image-quality metrics score with existing image fusion methods: HyperP_MB denotes a deep learning model that applies our training methods.

Method	AG	EI	FMI	LPC-SI	S3	SF	PIQE	MS-SSIM	Qabf	VIFF
DenseFuse	4.1796	41.3714	0.3366	0.9369	0.1830	13.7378	26.0896	0.8781	0.4208	0.4285
U2Fusion	4.7719	48.9770	0.2891	0.9438	0.1798	15.3116	35.4318	0.8468	0.3898	0.4181
TransMEF	3.9983	40.3294	0.2910	0.9360	0.1627	13.1360	37.8373	0.8636	0.3520	0.4084
HyperP_MB	6.1319	61.6277	0.3447	0.9543	0.2369	19.7974	25.8651	0.8916	0.5059	0.5477

, our method achieved the best performance and showed excellent results, particularly in terms of image sharpness and human perception-based quality metrics. The proposed method has an average improvement of about 32.8% over the existing synthesis methods in the image sharpness matrix (AG, EI, LPC-SI, S3, SF) and an average improvement of about 27.35% in the image-quality evaluation metrics (PIQE, Qabf, VIFF).

Figure 11, Figure 12, Figure 13, Figure 14, Figure 15 and Figure 16 show the results of each fusion method. Overall, the proposed method has better contrast and clarity than other methods. Whereas our method has detailed features from source images in the highlighted region, the other fusion methods have low distinctness. Because our method preserves useful information of under-/over-exposed images, the boundaries of the object and texture are distinguishable.

The proposed model was trained using a correct-exposed image from incorrectly exposed inputs. However, it does not reflect useful information from under-or over-exposed images. Thus, we added the MaskBuilder and ParamTuner to robustly learn the information from the inputs. To verify the effectiveness of the MaskBuilder and ParamTuner, we trained and evaluated the networks by ablating our blocks, as listed in

Table 2. Comparison of image-quality metrics score between the proposed methods: Co-target denotes correct-exposed image target, and MB and PT denote the MaskBuilder and the ParamTuner, respectively.

Co-Target	MB	PT	AG	EI	FMI	LPC-SI	S3	SF	PIQE	MS-SSIM	Qabf	VIFF
✓			4.8802	49.2164	0.3491	0.9526	0.2021	16.0696	27.4676	0.8809	0.4622	0.4604
✓	✓		5.9602	60.0486	0.3459	0.9549	0.2294	19.3997	27.9720	0.8962	0.5184	0.5421
✓	✓	✓	6.1319	61.6277	0.3447	0.9543	0.2369	19.7974	25.8651	0.8916	0.5059	0.5477

.

As we observed, the network with MaskBuilder and ParamTuner had superior scores, which indicate image sharpness and human perceptual quality. However, in terms of the quality of fusion with the source images, it showed slightly lower quality scores than networks without ParamTuner. We speculate that this is because the features of the source images are not synthesized equally, allowing the dynamic hyperparameter to learn more information from a more useful source image. As a result, the outcome of the network with ParamTuner can be considered effective because the objective of our approach is to synthesize a better quality HDR image using useful details from the LDR images.

5. Conclusions

In this paper, we propose a novel training method for MEF using multi-task learning. First, we classified the multi-exposure images into three categories: under-exposure, over-exposure, and correct exposure. The images that belong to the under-exposure or over-exposure category were used as source images and as reference images. This allowed the MEF networks to compensate for the saturated regions of the LDR images. Moreover, we suggest MaskBuilder to reproduce advanced reference images that contain useful information from source images. Thus, the ground truth of the LDR images is not necessary. Finally, our ParamTuner has the effect of fusing high-quality images by applying dynamic hyperparameters to the incorrect-exposed image losses. The quantitative and qualitative experimental results demonstrate that the proposed method can produce images with sharper and enhanced quality as perceived by humans. However, since this study only tested the learning method of the proposed approach on one CNNs model, it is necessary to evaluate the validity of the proposed method on various networks in future studies. In addition, future research should explore how to optimize a deep-learning-based fusion model by applying our proposed learning method so that it can be used in practical applications such as surveillance systems or autonomous driving, which require diverse types of image information.