CBFM: Contrast Balance Infrared and Visible Image Fusion Based on Contrast-Preserving Guided Filter

Abstract

Infrared (IR) and visible image fusion is an important data fusion and image processing technique that can accurately and comprehensively integrate the thermal radiation and texture details of source images. However, existing methods neglect the high-contrast fusion problem, leading to suboptimal fusion performance when thermal radiation target information in IR images is replaced by high-contrast information in visible images. To address this limitation, we propose a contrast-balanced framework for IR and visible image fusion. Specifically, a novel contrast balance strategy is proposed to process visible images and reduce energy while allowing for detailed compensation of overexposed areas. Moreover, a contrast-preserving guided filter is proposed to decompose the image into energy-detail layers to reduce high contrast and filter information. To effectively extract the active information in the detail layer and the brightness information in the energy layer, we proposed a new weighted energy-of-Laplacian operator and a Gaussian distribution of the image entropy scheme to fuse the detail and energy layers, respectively. The fused result was obtained by adding the detail and energy layers. Extensive experimental results demonstrate that the proposed method can effectively reduce the high contrast and highlighted target information in an image while simultaneously preserving details. In addition, the proposed method exhibited superior performance compared to the state-of-the-art methods in both qualitative and quantitative assessments.

1. Introduction

Numerous sensors are used for data collection in real-life scenarios. Unlike a single sensor, the use of multiple sensors to collect data from the same scene facilitates a comprehensive and accurate interpretation [1,2,3]. However, redundant data may be present when multiple sensors are used. Therefore, specific algorithms should be developed to extract critical pixel information. In infrared and visible image fusion (IVIF) [4,5,6], the target information of the thermal radiation is extracted from the infrared (IR) image, and the details and contrast information are obtained from the visible image. In IR imaging sensors, the physical properties of IR radiation are used to measure objects. Therefore, these sensors are not easily affected by complex external environments (i.e., haze and high-exposure areas). However, although IR images can approximately locate the area of a target based on the temperature or radiation difference, they cannot capture the detail and color information of the target scene, which results in low contrast. Therefore, using only a single IR image cannot provide comprehensive information about the scene. Visible imaging sensors capture the reflection of the target scene and can effectively preserve contrast and detail information in the scene. However, these sensors are easily affected by external environmental factors. Integrating useful detail information from a visible image and target information from an IR image into a single image can effectively compensate for the innate defects of various sensors and facilitate human observation without the influence of complex environments. Therefore, IVIF technology has been developed rapidly and is applied in military reconnaissance [7] and target tracking [8], among other fields. Figure 1 shows the application of IVIF in the military domain, demonstrating its effectiveness in enhancing military reconnaissance tasks. The fusion results demonstrate the ability of the technology to detect and identify military facilities, such as tanks and armored vehicles, within a scene.

IVIF methods can be categorized into deep learning (DL)-based and conventional methods. DL-based methods have been widely used in image fusion and achieve satisfactory results. For example, Xu et al. [9] proposed an unsupervised network for image fusion tasks that uses the same model and parameters to process various fusion problems. Zhang et al. [10] proposed a CNN-based image fusion framework that processes images in an end-to-end manner without postprocessing. To address the dependence of DL on the training dataset, Liu et al. [11] proposed a deep architecture for IVIF and achieved excellent fusion performance. However, DL-based algorithms rely on computer hardware and require intensive training. Furthermore, inadequate real-world datasets have limited the development of DL-based methods.

Multi-scale transform (MST)-based methods are commonly used in conventional IVIF methods, in which the source image is typically decomposed into multiple sub-bands and appropriate fusion rules are designed based on the features represented by the sub-bands [1]. For example, Chen et al. [12] proposed a target-enhanced MST decomposition model to extract details and energy information at various scales. However, multilevel and multidirectional decomposition may result in significant computational complexity of the algorithm. For IVIF tasks, Li et al. [13] proposed a total variation decomposition strategy to decompose an image into structure and texture layers, and construct appropriate weights for fusion in various layers. Nie et al. [14] proposed a total variation-based IVIF strategy to obtain fusion results using a variation model without designing fusion rules for various decomposition components such as in multi-scale decomposition methods. Furthermore, various filters are used in IVIF tasks. For example, to achieve the fine classification of IR and visible images, Mo et al. [15] proposed an attribute filter for IVIF that fully considers the attributes of objects in the source image and can effectively extract the target information in the IR image. In addition to the use of filters as a decomposition method, LatLRR [16] is widely used for IVIF. In this decomposition strategy, the source image is a combination of a low-rank component that represents the global structure and a prominent component that represents the local structure. Liu et al. [17] proposed a multi-decomposition LatLRR-based IVIF method that combined the advantages of MST and LatLRR-based methods to extract details from the source image. However, the inefficiency of this decomposition process limits its practical application.

In IVIF, target information extracted from the IR image and texture information obtained from the visible image are integrated into a single image. Typically, pixels at the same location in different source images do not affect each other, and the measurement values of the feature area pixels are higher than those in flat areas. However, because of the diversity and complexity of the shooting environments, many images can be affected to varying degrees. For example, when capturing outdoor photographs during the day, images may be overexposed and exhibit high contrast because of intense sunlight. When capturing photographs outdoors at night, light from passing cars and streetlamps may cause unreasonable contrast and scattering. Under hazy weather, visible images may enhance gray color information, which may result in the loss of scene details and colors. These problems increase the difficulty of developing IVIF methods. However, in most IVIF algorithms, only the ideal situation for IVIF is considered, and the aforementioned problems are ignored. In complex scenes, the high contrast and haze grayscale information provided by visible image may cause the scene to lose useful details and textures. If these adverse visual phenomena are not suppressed, thermal radiation information or high-contrast features in the fusion results may be lost, resulting in unclear targets. Figure 2 displays the fusion results for the proposed algorithm and the comparison method, namely, relative total variation decomposition (RTVD)] [13], for the two groups of IR and visible source images. When the source image exhibits overexposure and light-scattering effects, the RTVD method loses target pixel information and produces suboptimal fusion results. Although many image processing techniques, such as enhancement [18], dehazing [19,20], and denoising [21], have been proposed, if the image is preprocessed separately before fusion, this increases the complexity of the fusion task and is not conducive to practical applications.

Therefore, we propose a contrast balance approach based on the IVIF method, CBFM, to adjust the overly high-contrast areas of visible source images through a contrast balance strategy. Thus, the pixels in these areas can be prevented from affecting the distribution of target information in the IR image. First, we propose a multilayer decomposition scheme based on guided curvature filtering to obtain the most distinctive energy layer in the visible image. The high-brightness pixel information in the energy layer causes an overly high contrast in the image; therefore, we reduce the weight of the energy layer when reconstructing the source image. To avoid the detail loss caused by pixel loss in the energy layer, we perform detail compensation in the detail layer; i.e., we increase the weight of the useful detail information in the detail layer. Using the contrast balance strategy, we obtain a preprocessed visible image. We then propose a novel weighted guided filter that constructs weights using a combination of contrast fidelity and sparse constraints to effectively blur overexposed areas and extract important detail information. Using this weighted guided filter, various source images can be decomposed into energy and detail layers. For energy layer fusion, we propose a weighted fusion rule, based on entropy and a Gaussian distribution, which adjusts the contrast in the energy layer via weighted averaging. For detail layer fusion, we propose a weighted energy-of-Laplacian (EOL) operator that can effectively detect pixels with high activity and extract clear structural and detail information. The source code of this work is publicly available at https://github.com/ixilai/CBFM.

The contributions of this study are as follows:

• A novel IVIF algorithm based on contrast balance is proposed to effectively address the fusion challenge in complex environments, including maintaining reasonable contrast and detail fusion tasks affected by adverse phenomena such as overexposure, haze, and light diffusion.
• A novel contrast balance strategy is proposed to reduce the adverse effects of overexposure in the visible light image by decreasing the weight of the energy layers in the source image and supplementing the details.
• A contrast-preserving guided filter (CPGF) that constructs weights specifically for IVIF tasks is proposed. The IVIF outperforms the guided filter (GIF) and weighted GIF (WGF).

The remainder of this paper is structured as follows: Section 2 provides a detailed presentation of the proposed algorithm, Section 3 describes the experimental setup, and Section 4 presents the main conclusions of the study.

2. Materials and Methods

In this section, the specific steps of the proposed method are described in detail (Figure 3). First, to prevent high-contrast pixel information in visible images from affecting the distribution of thermal radiation target information, we proposed a contrast balance strategy for processing visible images. Additionally, we designed a new CPGF filter specifically for the IVIF task to effectively extract feature information of different dimensions from IR and visible images. This filter can decompose the source image into an energy layer and a detail layer. The energy layer contains most of the energy of the image and low-frequency information, which determines the overall structure and texture of the image. To extract pixels with the most information from the energy layer, we used a new weighted average fusion rule based on entropy Gaussian distribution. The detail layer represents image details or local features with high pixel activity. The weighted EOL operator can effectively extract significant edge details, making it suitable for the design of detail layer fusion rules. Finally, we added the fusion energy layer to the detail layer to obtain the final fusion result.

2.1. Contrast Balance Strategy

Highly exposed visible images typically result in high contrast and loss of target thermal radiation information. Therefore, we proposed a strategy to balance the contrast in overexposed regions of visible images. We considered the visible image to be a combination of energy and detail layers. The energy layer generally carries considerable luminance and energy information in the source image. Therefore, the weights in the energy layer were reduced to adjust the high-contrast regions in the source image. However, detailed texture information may be lost when the weight of the energy layer is reduced. Therefore, we performed a detail enhancement process; i.e., the weight of the detail layer was increased in the source image so that the obtained preprocessed visible source image can effectively balance the contrast and enhance the distribution of useful details in the image.

To effectively extract detail information at various scales using the energy in the source image subject to [22], we proposed a novel guided curvature filtering (GDCF)-based image decomposition strategy. GDCF is based on GIF [23] and curvature filtering [24] and can decompose the source image into energy, fine structure (FS), and coarse structure (CS) layers.

2.1.1. GDCF-based Multilayer Decomposition Strategy

GIF is used as a filter for image smoothing to effectively smooth out detail features of the image and preserve strong edges. Suppose the guide image is G, the input image is $f$ and the output image is O. The local linear model can be expressed as follows:

$$O\left(p\right)={\alpha }_{p^{\prime }}G\left(p\right)+{\beta }_{p^{\prime }},\forall p\in {\Omega }_{\psi 1}\left(p^{\prime }\right)$$

(1)

where ${\Omega }_{\psi 1}\left(p^{\prime }\right)$ represents a square window with radius ${\psi }_1$ centered on pixel $p^{\prime }$, and the values of ${\alpha }_{p^{\prime }}$ and ${\beta }_{p^{\prime }}$ are fixed in this window; ${\psi }_1$ is set to 15.

The parameters of $\alpha$, $\beta$ are determined by minimizing the cost function, as follows:

$$E\left({\alpha }_{p^{\prime }},{\beta }_{p^{\prime }}\right)={{\displaystyle \sum }}_{p\in {\Omega }_{{\psi }_1}\left(p^{\prime }\right)}\left({\left({\alpha }_{p^{\prime }}G\left(p\right)+{\beta }_{p^{\prime }}-f\left(p\right)\right)}^2+\widehat{\lambda }{\alpha }_{p^{\prime }}^2\right)$$

(2)

where $\widehat{\lambda }$ is a regularization parameter that penalizes a large ${\alpha }_{p^{\prime }}$, which was set to 0.3 in this study. Subsequently, the GIF process is represented as $GIF\left(\cdot \right)$.

Regarding the GDCF-based multilayer decomposition strategy, given that $f_g^{\left(i\right)}\left(i=1,2,\dots ,n\right)$ is the filtered result of the ith GIF; $f_c^{\left(i\right)}\left(i=1,2,\dots ,n\right)$ is the filtered result of the ith CF; and $n$ is a constant that represents the number of decomposition layers (and is set to 3 in this study). Furthermore, $f$ can be expressed as follows:

$$f={{\displaystyle \sum }}_{i=1}^n\left(F{S}^{\left(i\right)}+C{S}^{\left(i\right)}\right)+EG$$

(3)

where $EG$, $F{S}^{\left(i\right)}$, and $C{S}^{\left(i\right)}$ represent the base layer, ith fine structure, and ith coarse structure, respectively, which can be expressed as follows:

$$EG=f_g^{\left(n\right)}$$

(4)

$$F{S}^{\left(i\right)}=\left\{\begin{array}{c}f-f_c^{\left(1\right)},i=1\\ f_g^{\left(i-1\right)}-f_c^{\left(i\right)},i=2,3,\dots ,n\end{array}\right.$$

(5)

$$C{S}^{\left(i\right)}=f_c^{\left(i\right)}-f_g^{\left(i\right)}, \left(i=1,2,\dots ,n\right)$$

(6)

where $f_g^{\left(i\right)}$ and $f_c^{\left(i\right)}$ are computed as follows:

$$f_g^{\left(i\right)}=\left\{\begin{array}{c}GIF\left(f,f,\widehat{\lambda }\right),i=1\\ GIF\left(f_g^{\left(i-1\right)},f_g^{\left(i-1\right)},\widehat{\lambda }\right),i=2,3,\dots ,n\end{array}\right.$$

(7)

$$f_c^{\left(i\right)}=\left\{\begin{array}{c}CF\left(f,\upsilon \right),i=1\\ CF\left(f_c^{\left(i-1\right)},\upsilon \right),i=2,3,\dots ,n\end{array}\right.$$

(8)

where $CF\left(\cdot \right)$ represents the CF operation [22] and $\upsilon$ represents the iteration number, which was set to five in this study.

2.1.2. Contrast Balance

After obtaining the fine structure, rough structure, and base layers, we adjusted the contrast in the source image using the following weighted fusion rules to enhance the detail information in the figure:

$$\begin{gathered} P=0.8\times EG+K\times F{S}^{\left(1\right)}+\left(K-1\right)\times F{S}^{\left(2\right)}+\left(K-1.5\right)\times F{S}^{\left(3\right)} \\ +K\times C{S}^{\left(1\right)}+\left(K-0.5\right)C{S}^{\left(2\right)}+\left(K-1\right)\times C{S}^{\left(3\right)} \end{gathered}$$

(9)

where $K$ was set to 1.8. In Equation (9), the weights assigned to the first 3 layers of $FS$ are $K, K-1$, and $K-1.5$, whereas the weights for the first 3 layers of $CS$ are $K$, $K-0.5$, and $K-1$. This weighting scheme is based on the characteristics of $FS$ and $CS$ in representing different types of information in the source image. FS captures fine structure details, whereas CS represents large-scale edge structure information. The weight values gradually decrease as we move from the first layer to subsequent layers, because the amount of detailed information tends to diminish. This weighting scheme ensures a balanced contribution from each layer. In regions with excessive contrast, almost all the detailed information is lost, and only a small amount of large-scale edge information exists. Hence, a larger weight must be assigned to the CS overall during detailed compensation. In addition, because the FS layer contains more detailed information, if its weight is considerably large, it may lead to information redundancy in the preprocessed image, which can affect the visual result. In addition, because undesirable visual phenomena, such as overexposure, generally occur only in visible images, the contrast-balancing strategy applies only to the visible images. In the subsequent fusion process, we denoted the IR source image as $f_1$ and preprocessed the visible image as $f_2$.

2.2. Image Decomposition by CPGF

2.2.1. Proposed CPGF

A previous study [25] revealed that because the parameter λ is fixed in conventional GIF, blurring is concentrated near the edges and halos are introduced. To solve this problem, Li et al. [25] added edge-aware weights to GIF to form the WGF, as follows:

$$E\left({\alpha }_{p^{\prime }},{\beta }_{p^{\prime }}\right)={{\displaystyle \sum }}_{p\in {\Omega }_{{\psi }_1}\left(p^{\prime }\right)}\left({\left({\alpha }_{p^{\prime }}G\left(p\right)+{\beta }_{p^{\prime }}-f\left(p\right)\right)}^2+\frac{\lambda }{{\Gamma }_G\left(p^{\prime }\right)}{\alpha }_{p^{\prime }}^2\right)$$

(10)

where ${\Gamma }_G\left(p^{\prime }\right)$ denotes the edge perception weight and $\lambda$ is a regularization parameter.

Compared with GIF, WGF can avoid halo artifacts associated with the filtering process and produce a higher-quality filtered image. However, for IR and visible image fusion tasks, image filtering is not performed for edge-preserving smoothing, but to efficiently extract target and detail information in the scene without the interference of high-contrast areas and to smooth overly bright pixel information. Therefore, edge-aware weights in [25] are not necessary for IVIF; thus, the assignment of appropriate weighting factors for specific fusion tasks is critical. Subject to [26], we introduced a weight that can improve the excessive contrast in the source image by constructing new weights using the specification of contrast fidelity and sparsity constraints, as follows:

Contrast fidelity ${\zeta }_1$ is defined by the following expression:

$${\zeta }_1=\Vert \omega -f{\Vert }_2^2$$

(11)

where $\omega$ denotes the weight map, and Equation (11) balances the contrast in the input image by using the L2 norm to maintain a reasonable light-to-dark ratio of the target to background information, without requiring the exact pixel intensity as the input image.

Sparse constraint ${\zeta }_2$ is defined as follows:

$${\zeta }_2=\Vert \nabla \omega -\nabla f{\Vert }_1$$

(12)

where $\nabla \omega$ and $\nabla f$ represent the gradients of $\omega$ and $f$, respectively.

The weighting problem can be formulated by minimizing the following objective function according to the contrast fidelity and sparsity norms:

$$\xi \left(\omega \right)={\zeta }_1+\gamma {\zeta }_2=\Vert \omega -f{\Vert }_2^2+\gamma \Vert \nabla \omega -\nabla f{\Vert }_1$$

(13)

where $\gamma$ is the balance parameter control between ${\zeta }_1$ and ${\zeta }_2$, which was set to 15 in this study. In addition, ${\Vert \cdot \Vert }_2^2$ refers to the squared L2 norm and ${\Vert \cdot \Vert }_1$ represents the L1 norm, which calculates the sum of the absolute values of the elements of a vector or matrix. According to [26], the above Equation (13) can be solved to obtain the contrast weights $\omega$. $\omega$ is embedded in Equation (10) to form a new CPGF, as follows:

$$E\left({\alpha }_{p^{\prime }},{\beta }_{p^{\prime }}\right)={\displaystyle \sum }_{p\in {\Omega }_{\psi 1}\left(p^{\prime }\right)}\left({\left({\alpha }_{p^{\prime }}G\left(p\right)+{\beta }_{p^{\prime }}-f\left(p\right)\right)}^2+\frac{\lambda }{\omega \left(p^{\prime }\right)}{\alpha }_{p^{\prime }}^2\right)$$

(14)

where ${\alpha }_{p^{\prime }}$ and ${\beta }_{p^{\prime }}$ are obtained as:

$${\alpha }_{p^{\prime }}=\frac{{\mu }_{G\odot f,{\psi }_1}\left(p^{\prime }\right)-{\mu }_{G,{\psi }_1}\left(p^{\prime }\right){\mu }_{f,{\psi }_1}\left(p^{\prime }\right)}{{\sigma }_{G,{\psi }_1}^2\left(p^{\prime }\right)+\frac{\lambda }{\omega \left(p^{\prime }\right)}}$$

(15)

$$b_{p^{\prime }}={\mu }_{f,{\psi }_1}\left(p^{\prime }\right)-{\alpha }_{p^{\prime }{\mu }_G,{\psi }_1}\left(p^{\prime }\right)$$

(16)

where $\odot$ represents the element-by-element product of two matrices; ${\mu }_{G\odot f,{\psi }_1}\left(p^{\prime }\right)$, ${\mu }_{G,{\psi }_1}\left(p^{\prime }\right)$, and ${\mu }_{f,{\psi }_1}\left(p^{\prime }\right)$ are the mean values of $G\odot f$, $G$, and $f$, respectively. Furthermore, ${\sigma }_{G,{\psi }_1}^2\left(p^{\prime }\right)$ is the variance in a local window with a radius of $3\times 3$ centered on pixel $p^{\prime }$.

2.2.2. Image Decomposition

First, we filtered the IR and visible source images using the proposed WGF and obtained the energy layer $E_t$ of the source image $f_t$, where $t$ denotes the number of source images, which was set to two in this study. Next, we obtained the detail layer $D_t$ using the following equation:

$$D_t=f_t-E_t$$

(17)

Figure 4 displays an example of image decomposition using CPGF. The effect of excessive contrast on detail extraction can be effectively avoided using the proposed model, and the performance of the algorithm is improved.

2.3. Energy Layer Fusion

Although excessive contrast information in the energy layer can be effectively reduced using the proposed WGF, the energy layer still contains the most luminance and target information in the source image. Entropy represents the aggregation of the luminance distribution in the source image, and the extreme probability density function of entropy can be used to determine the degree of image blurring [27]. In the energy layer of IR and visible source images, clear pixels typically contain more energy information, which indicates that they have a higher entropy value compared to blurred pixels. Therefore, in this study, we used the Gaussian distribution of image entropy [28] to construct fusion weights to effectively retain the most representative energy information in various source images. The energy layer $E_t$ and the entropy of $e_t$ can be expressed as follows:

$$e_t=-{\displaystyle \sum }_j{{\rm P}}_j\left(f_t\right){\log }_2{{\rm P}}_j\left(f_t\right)$$

(18)

where ${{\rm P}}_j\left(f_t\right)$ is the probability of intensity value $j$ in the image $f_t$. Therefore, the probability can be calculated as follows:

$${\rm P}\left(e_t\right)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(\frac{-{\left(e_t-\epsilon \right)}^2}{2{\sigma }^2}\right)$$

(19)

As in [28], $\epsilon$ and $\sigma$ were set to 7.4600 and 0.8732, respectively. According to ${\rm P}\left(e_t\right)$, we can construct the fusion weights to obtain the fusion results of the energy layer as follows:

$$FE=\frac{P\left(e_1\right)}{P\left(e_1\right)+P\left(e_2\right)}\times E_1+\frac{P\left(e_2\right)}{P\left(e_1\right)+P\left(e_2\right)}\times E_2$$

(20)

where $E_1$ and $E_2$ represent the energy layers of the source images $f_1$ and $f_2$, respectively.

2.4. Detail Layer Fusion

The detail layer represents rich texture information in the source image, and the ideal fusion result should contain clear details in both the visible and IR images. The energy of Laplacian of the image (EOL) [29] can be used by the Laplace operator to analyze the high spatial frequencies associated with the sharpness of the image boundaries, and the model can effectively detect the information of pixels with greater sharpness in the image, i.e., the detail corresponding to the source image. Therefore, we proposed a novel weighted EOL (WEOL) method to construct a weight map of the detail layer. The input image is $D_t$, and EOL is expressed as follows:

$$EOL\left(f_t\right)={\displaystyle \sum }_x{\displaystyle \sum }_y{\left(D_{xx}+D_{yy}\right)}^2$$

(21)

according to [29], $D_{xx}+D_{yy}$ can be defined as:

$$\begin{gathered} D_{xx}+D_{yy}=-D\left(x-1,y-1\right)-4D\left(x-1,y\right)-D\left(x-1,y+1\right)-4D\left(x,y-1\right)+ \\ 20D\left(x,y\right)-4D\left(x,y+1\right)-D\left(x+1,y-1\right)-4D\left(x+1,y\right)-D\left(x+1,y+1\right) \end{gathered}$$

(22)

In the proposed algorithm, the WEOL is denoted as follows:

$$WEOL\left(D_t\right)={\displaystyle \sum }_x{\displaystyle \sum }_y{W}^{\prime }\cdot {\left(D_{xx}+D_{yy}\right)}^2$$

(23)

where the weight matrix $W^{\prime }$ is defined as follows:

$$W^{\prime }=\frac{1}{15}\left|\begin{array}{ccc}1& 2& 1\\ 2& 3& 2\\ 1& 2& 1\end{array}\right|$$

(24)

The detail layer fusion result is obtained based on $WEOL\left(f_t\right)$ to construct the detail layer fusion weights to obtain the detail layer fusion results.

$$FD=\frac{WEOL\left(D_1\right)}{WEOL\left(D_1\right)+WEOL\left(D_2\right)}\times D_1+\frac{WEOL\left(D_2\right)}{WEOL\left(D_1\right)+WEOL\left(D_2\right)}\times D_2$$

(25)

where $D_1$ and $D_2$ represent the detail layers of source images $f_1$ and $f_2$, respectively.

2.5. Fusion Result Construction

Finally, the fused energy layer and detail layer are added to develop fusion result F as follows:

F = FE + FD

(26)

3. Experiments

3.1. Experimental Setup

We compared the proposed algorithm with nine state-of-the-art algorithms and conducted extensive experiments on two publicly available IR and visible image datasets. The first dataset was a road dataset [9] that contained 221 sets of IR and visible images captured under complex road conditions. The primary challenge was to effectively suppress the impacts of high contrast from strong light sources during the day and night. The second dataset was from [30] and contained 48 sets of IR and visible images captured in outdoor scenes. The proposed algorithm is abbreviated as CBFM in our experiments. LatLRR [16], RTVD [13], TEMF [12], MFEIF [11], U2Fusion [9], DIVFusion [31], GANMcC [32], SDDGAN [33], and UMFusion [34] are the nine state-of-the-art algorithms that were compared with our proposed algorithm. The source code with default parameters provided by the original authors was used. In addition, CBFM and three traditional methods were tested using Matlab 2021b on a PC with an AMD Ryzen 5 4600H with a Radeon Graphics processor and NVIDIA GeForce GTX 1650 graphics card. The results of the fusion of six deep learning-based comparison methods were obtained using Python 3.8 and a discrete PC with an Intel Core i7-7700HQ processor and NVIDIA GeForce GTX-1070 graphics card.

Six objective evaluation metrics, namely, gradient-based fusion performance (Q_G) [35], image fusion metric based on a multi-scale scheme (Q_M) [36], Piella’s metric (Q_S) [36], entropy (EN) [37], average gradient (AG) [38], and spatial frequency (SF) [39], were used to evaluate the fusion performance of all the methods. Specifically, Q_G evaluates the amount of edge information; Q_M is a fusion image quality metric based on a multi-scale scheme; AG reflects the gradient information in the fusion result (i.e., the richer the gradient information, the higher the image quality); EN is based on information theory and measures the amount of information contained in the image; and SF reflects the sharpness of the fused image, wherein a higher value represents a higher-quality fused image and better algorithm performance. The six metrics can be used to comprehensively evaluate the fusion performance of various methods, with higher values representing better performance.

3.2. Parameter Analysis

For the regularization parameter in Equation (10), $\lambda$ considerably affects the performance of the proposed algorithm. Therefore, we performed parametric analysis to determine the optimal parameter values to ensure excellent fusion performance. First, we fixed the values of other parameters of the model and set $\lambda$ ∈ {0.05, 0.07, …, 0.4}. We then randomly selected 30 sets of images from the road dataset to add to the parameter analysis experiment. The scores for each index when different values were obtained are presented in

Table 1. Quantitatively comparing the effect of different values of $\lambda$.

$\mathit{\lambda }$	QG	QM	QS	EN	AG	SF
0.05	0.5238	0.6027	0.7914	7.0539	4.9628	13.1062
0.07	0.5240	0.6123	0.7901	7.0659	5.0875	13.4770
0.1	0.5240	0.6203	0.7886	7.0770	5.1983	13.8106
0.2	0.5234	0.6296	0.7862	7.0930	5.3534	14.2851
0.3	0.5229	0.6338	0.7851	7.0994	5.4133	14.4711
0.4	0.5227	0.6345	0.7846	7.1028	5.4450	14.5696

; the maximum values are in bold. When $\lambda$ increases, Q_M, EN, AG, and SF gradually increase, indicating that the parameter $\lambda$ can affect the ability of the model to extract features. However, the trend in the values of the Q_G and Q_S indicators does not conform to the aforementioned rule. When the value of $\lambda$ was set to 0.4, both the Q_G and Q_S metrics achieved the lowest scores among all the comparison experiments. This outcome suggests that an excessively large $\lambda$ value can result in redundant detail information and structural distortion in the fusion results. Such distortions are unfavorable for producing visually useful images that are consistent with the human visual system. Therefore, to achieve visual balance and optimal fusion performance, we set the value of $\lambda$ to 0.3. This value met the requirements for the proposed algorithm, resulting in excellent scores for all the metrics.

3.3. Ablation Analysis

3.3.1. Ablation Analysis of Contrast Balance Strategy

To effectively reduce the effects of high contrast and diffusion from overexposed regions in visible images, a novel contrast-balancing strategy was proposed. An ablation analysis was performed to determine whether this strategy could improve the fusion performance of the proposed model. The algorithm without contrast-balancing treatment was labeled N-CBFM, as shown in Figure 5; the three sets of fusion results for CBFM and N-CBFM are also provided. The contrast-balancing strategy effectively improved the fusion performance of the model to better retain reasonable contrast and present the detail information in source images.

3.3.2. Ablation Analysis of the Proposed Filter

For the specificity of the IVIF task, we designed a novel weighted guided filter whose weights consisted of contrast fidelity with sparsity constraints. To evaluate the effectiveness of the strategy, we performed an ablation analysis. First, we selected 20 sets of source images from the road dataset as the data source. Second, we compared GIF and WGF to the proposed method as the filters for image decomposition in the proposed algorithm. For comparison, we named the method of image decomposition by GIF as A-CBFM and the method of image decomposition by WGF as B-CBFM. The same parameters were used for all three filters to ensure an unbiased comparison. The experimental results for the methods are shown in

Table 2. Quantitative comparison of different comparison algorithms.

Methods	QG	QM	QS	EN	AG	SF
A-CBFM	0.5333	0.5789	0.7680	7.1411	5.9602	15.6960
B-CBFM	0.5375	0.4986	0.7811	6.9772	4.7292	11.8979
CBFM	0.5327	0.5915	0.7665	7.1500	6.0415	15.9530

. The maximum values are in bold. Table 2 shows that CBFM obtained the highest scores for the four metrics, and although the Q_G and Q_S metrics did not score as well as B-CBFM, they were generally better than the compared algorithms. Thus, the decomposition of images using the proposed CPGF yields better fusion performance than GIF and WGF methods.

3.4. Subjective Evaluation

Figure 6 displays the fusion results for the proposed algorithm with nine state-of-the-art comparison methods for two classical image pairs from the road dataset. For an intuitive comparison, we zoomed in on a local region of the fusion results located in the high-contrast region of the visible image. The IR image provides rich target information and provides a clear view of the distribution of cars on the road, whereas the visible image contains texture information. RTVD and TEMF are easily affected by the high-contrast region in the visible image and cannot identify the target information in the IR image. In contrast, the four methods LatLRR, MFEIF, U2Fusion, and UMFusion can retain some target information such as power lines in the sky and leaves on trees. However, they cannot maintain reasonable contrast to some extent, which can reduce the visual effect of the fusion results, and do not provide a comprehensive interpretation of the scene. Three methods, DIVFusion, GANMcC, and SDDGAN, appear to produce blurred images and lose the target and detail information in the scene. The CBFM exhibits superior ability to extract target information from the IR image and the texture information from the visible image, and can maintain a reasonable contrast in the fusion result. This result could be attributed to the filtering of most of the high-contrast pixel information in the proposed WGF to ensure that the luminance target information in the IR image can be effectively extracted. If contrast balance is not applied to the highlighted region, this phenomenon may lead to a situation similar to that of the RTVD and TEMF methods, in which the overly bright pixel information in the visible image can mask the target luminance information in the IR image. The proposed algorithm can achieve better fusion performance than the five state-of-the-art contrast methods.

Figure 7 displays the fusion results of the proposed algorithm for nine state-of-the-art comparison methods for two classical image pairs from the road dataset. In this study, we selected images captured at night, and both headlights and streetlights on the road had light-diffusion effects, which could produce undesirable visual effects in the fusion results if they are not suppressed. All five algorithms—RTVD, TEMF, MFEIF, DIVFusion, and UMFusion—were affected by the light-diffusion effect, which resulted in the loss of some of the target information in the IR images. Although LatLRR and U2Fusion effectively mitigated this situation, they still lost part of the image contrast and exhibited insufficient feature extraction capability. Three methods, namely, DIVFusion, GANMcC, and SDDGAN, appeared to blur the target information and generate suboptimal fusion results. The proposed algorithm can maintain excellent fusion performance despite the light diffusion effect at night, effectively balancing the contrast between various source images and providing excellent visual effects for fusion results. The proposed algorithm handled the images of various situations and exhibited a superior fusion performance than the five state-of-the-art algorithms.

Figure 8 shows the fusion results of the two sets of IR and visible images captured from outdoor scenes. As displayed in the figure, in the first fusion task, the main challenge arises from the sky region of the visible image, which has high contrast and energy information because of the intense light source of the sun, which may lead to the loss of target information in the IR image if it not suppressed. Moreover, the visible image is rich in detail information; therefore, the preservation of details should be considered when balancing the contrast. In the second fusion task, the primary challenge originates from the foggy sky in the visible image, which may lead to the loss of texture information and inconspicuousness of the target information in the fusion result. Therefore, a comparison of these two fusion tasks can effectively reflect the performance of the various fusion algorithms. Both the TEMF and MFEIF methods were disturbed by high-contrast information, which led to inconspicuous target information of clouds in the sky. The DIVFusion method contained incorrect target information, resulting in poor visual quality fusion results. Under the influence of a foggy sky, all four methods, LatLRR, TEMF, SDDGAN, and UMFusion, failed to retain the texture information of tree leaves in visible images and produced suboptimal fusion results. As observed in the red magnified area, all five methods lost some useful pixel information during the fusion process, which resulted in a lower contrast with a loss of texture. Unlike the comparison methods, the proposed algorithm, CBFM, exhibited excellent performance in both sets of fusion tasks, proving that the contrast-balancing strategy can effectively solve the fusion problem in complex situations without losing detail information while achieving reasonable contrast.

3.5. Objective Evaluation

In addition to subjective visual comparisons, objective evaluations based on the entire road dataset were performed in the comparison experiments.

Table 3. Quantitative comparison of the proposed algorithm and different comparison methods on the road scene dataset. Bold is the best, red is second, and blue is third.

Methods	QG	QM	QS	EN	AG	SF
LatLRR	0.3990	0.4393	0.7786	6.9161	3.7787	10.1186
RTVD	0.4555	0.5538	0.7415	7.0207	4.1072	10.9419
TEMF	0.3657	0.4118	0.7335	6.9794	3.6752	9.8647
MFEIF	0.4312	0.4558	0.7832	7.0488	3.7644	9.5561
U2Fusion	0.4884	0.4514	0.8133	6.8021	4.6377	11.4237
DIVFusion	0.2883	0.3309	0.6213	7.5318	4.8050	11.6477
GANMcC	0.3606	0.4034	0.7126	7.2366	3.7788	9.0192
SDDGAN	0.3655	0.3946	0.7377	7.5261	4.3825	10.4413
UMFusion	0.4785	0.4771	0.8133	7.0474	4.0954	10.5501
CBFM	0.5489	0.7217	0.8186	7.1383	5.8674	15.2249

presents the scores of the proposed algorithm and the comparison methods on the road dataset for various metrics. The maximum values are highlighted in bold, the second-highest scores are shown in red, and the third-highest scores are shown in blue. Table 3 shows that the proposed algorithm obtained the best scores for all five metrics, proving that it performed well in extracting features and retaining information in the source images. The U2Fusion method obtained the second highest score for the two metrics, similar to the subjective evaluation, and achieved the best visual results among all the compared methods. However, U2Fusion obtained the lowest score in the EN metric because it lost most of the contrast information during fusion.

Table 4. Quantitative comparison of the proposed algorithm and different comparison methods on IVIF dataset [30]. Bold is the best, red is second, and blue is third.

Methods	QG	QM	QS	EN	AG	SF
LatLRR	0.4176	0.3169	0.8251	7.1010	4.8726	14.0734
RTVD	0.4914	0.5246	0.7986	7.1284	5.1918	14.3747
TEMF	0.4268	0.3709	0.8085	6.8659	4.9588	14.0858
MFEIF	0.4642	0.3622	0.8330	7.1174	4.8914	12.8592
U2Fusion	0.4830	0.3106	0.8442	7.0989	6.3451	16.0479
DIVFusion	0.3029	0.2252	0.7218	7.5289	4.9692	12.0900
GANMcC	0.3847	0.2971	0.7791	7.1071	4.7454	12.6711
SDDGAN	0.3573	0.2654	0.7675	7.4608	4.3714	11.1505
UMFusion	0.4582	0.3566	0.8438	7.1387	5.1984	14.4103
CBFM	0.5246	0.4097	0.8277	7.1164	7.0740	20.2707

shows the performance of the methods on the dataset [9]. The maximum values are highlighted in bold, the second-highest scores are shown in red, and the third-highest scores are shown in blue. The results of the quantitative comparison show that the proposed algorithm achieved the best scores in three metrics: Q_G, AG, and SF. The proposed algorithm achieved fusion with high-resolution and rich feature information. Given that the purpose of this algorithm was to maintain a reasonable contrast of the image, some of the pixel energy information may be missing. Thus, the data does not facilitate comparison with some of the methods for the EN metrics. RTVD and U2Fusion achieved excellent performance on this dataset. However, a qualitative evaluation revealed that RTVD exhibited a partial loss of texture information during fusion. Therefore, its score on the SF index was lower compared to the scores for U2Fusion and CBFM. Thus, consistent with the conclusions obtained for the subjective evaluation, the proposed algorithm outperformed the other nine methods in fusion. This result proved the superiority of the contrast-balancing strategy.

3.6. Computational Time

Table 5. Comparison of the average computational time of 10 methods on the road scene dataset.

Methods	Time	Methods	Time
LatLRR	29.1462	DIVFusion	2.64
RTVD	0.5451	GANMcC	1.103
TEMF	0.01	SDDGAN	0.166
MFEIF	0.093	UMFusion	0.7692
U2Fusion	0.861	CBFM	8.5802

presents the computational time comparison results for all 10 methods. Notably, the TEMF method exhibits the lowest computational cost among the compared methods. However, this can be attributed to its decomposition of fewer layers, potentially leading to a compromised performance in handling overexposed regions. In contrast, the LatLRR method, which relies on learning the projection matrix, demonstrates relatively higher time consumption. Although the proposed algorithm may not achieve optimal computational efficiency owing to the longer runtime of CPGF, we acknowledge this limitation and plan to explore the impact of different parameters on the computational load in future research. A direct comparison of the six DL-based methods may seem unfair owing to the various platforms used to run the proposed algorithm. Nonetheless, DL-based algorithms facilitate higher computational efficiency, which is advantageous.

4. Conclusions

In this study, a novel IVIF algorithm based on a contrast-balancing strategy was proposed to effectively mitigate the impact of high-contrast regions, such as highly exposed areas, and address the issue of light diffusion effects present in the source image. Furthermore, we proposed the CPGF to categorize the source image into detail and energy layers. To fuse the detail layers, we proposed a weighted EOL operator to obtain the highest fusion weights. To fuse the energy layers, we designed a weighted fusion rule based on the Gaussian distribution of entropy. The final fusion results were obtained by summing the fused energy and detail layers. Experimental results revealed that the fusion performance of the proposed algorithm was superior compared to other state-of-the-art comparison methods and effectively suppressed undesirable visual effects in the source image. In addition, excellent fusion results that provide a comprehensive interpretation of the scene were obtained. The proposed contrast balance strategy may not achieve significant advantages in terms of computational efficiency because a multi-scale decomposition approach is used to extract the sub-bands of various features. Therefore, in the future, we will explore methods to optimize the strategy to reduce computational complexity.