Infusion-Net: Inter- and Intra-Weighted Cross-Fusion Network for Multispectral Object Detection

Abstract

Object recognition is conducted using red, green, and blue (RGB) images in object recognition studies. However, RGB images in low-light environments or environments where other objects occlude the target objects cause poor object recognition performance. In contrast, infrared (IR) images provide acceptable object recognition performance in these environments because they detect IR waves rather than visible illumination. In this paper, we propose an inter- and intra-weighted cross-fusion network (Infusion-Net), which improves object recognition performance by combining the strengths of the RGB-IR image pairs. Infusion-Net connects dual object detection models using a high-frequency (HF) assistant (HFA) to combine the advantages of RGB-IR images. To extract HF components, the HFA transforms input images into a discrete cosine transform domain. The extracted HF components are weighted via pretrained inter- and intra-weights for feature-domain cross-fusion. The inter-weighted fused features are transmitted to each other’s networks to complement the limitations of each modality. The intra-weighted features are also used to enhance any insufficient HF components of the target objects. Thus, the experimental results present the superiority of the proposed network and present improved performance of the multispectral object recognition task.

1. Introduction

Among various developments in computer vision, object detection is an essential funciton. Multimodality object detection can be employed in surveillance cameras, drones, autonomous driving, license plate recognition [1,2], crack detection [3], and other applications, as shown in Figure 1. The current object detection models display remarkable speed and accuracy. Among them, YOLOv7 [4] recently surpassed all known object detection models in standard datasets, such as the Microsoft Common Objects in Context dataset [5], which included areas such as proper illumination and clear boundaries between objects and the background.

However, in the real world, some object detection degradation components are unsuitable for object detection, such as rain, fog, shadow, low light, and low resolution. Hence, under these conditions, object detection models suffer severe performance degradation. Infrared (IR) images with clearer object edges than RGB images can be used as input images to prevent performance degradation. Nevertheless, object detection models still exhibit unacceptable performance due to the lack of information about the IR image (object color, texture, etc.).

In addition, object detection models cannot purify object detection-irrelevant features, such as dark illumination, occluded objects in RGB images, and IR image colors. As unpurified object detection-irrelevant features interrupt the model training stability, the accuracy of object detection models diminishes, as illustrated in Figure 2. For example, Figure 3 presents examples of the advantages of RGB and IR images under certain conditions. On a bright day, the RGB image in Figure 3a provides more information, including color, edges, and textures, than the IR image in Figure 3a. In contrast, at night, the IR image in Figure 3b displays a more precise outline of pedestrians under dark illumination than the RGB image in Figure 3b. In addition, even if a branch occludes the object, the IR image presents a clearer boundary of the item. Hence, an object detection model that can exploit the advantages of each image is necessary to achieve the best recognition performance in various environments.

Due to the development of convolution neural networks (CNNs), dual-stream CNN-based object detectors have been proposed. The dual-stream CNN-based object detectors consist of two streams: an RGB and IR stream. Using the two streams as features, these models improve the performance of object detection in a multispectral environment. In addition, multispectral datasets, such as FLIR [6] and LLVIP [7], that match the resolution of RGB images and IR images, have also led to continuous development in this field.

Recent dual-stream CNN-based object detectors fuse the features of RGB and IR images in the detection model. Various studies have been conducted regarding object recognition in a multispectral environment based on these fusion approaches. However, these approaches cannot properly exploit the advantages of RGB and IR streams. As these models add or multiply the features of each stream, models cannot adaptively fuse information on features according to each feature level. In addition, object detection-irrelevant features (occluded objects in RGB images, the color of IR images, etc.) may be fused, causing instability in model training and performance degradation.

To address this issue, in this paper, we propose an inter- and intra-weighted cross-fusion network (Infusion-Net) using a high-frequency (HF) assistant (HFA). The Infusion-Net exchanges the features of RGB and IR with four HFA blocks, depending on the feature level. In this process, the Infusion-Net adaptively exploits the object detection-relevant features of each stream. The HFA block consists of a HF extraction and an enhancing process based on the discrete cosine transform (DCT). The HF extraction in the HFA block purifies the object detection-irrelevant features. In enhancing the process in the HFA, a residual channel attention block (RCAB) [8] reinforces the purified features. The Infusion-Net adaptably adjusts information utilization and feature enhancement for each stream according to each fusion phase via learnable inter- and intra-weight parameters. Moreover, Infusion-Net surpasses other multistream approaches on multispectral datasets, such as FLIR and LLVIP. In addition, the extensive experiments present the effectiveness of these approaches.

The contributions of this paper are summarized as follows:

• We propose an Infusion-Net that gradually fuses the features of RGB and IR streams according to feature level to exploit the advantages of each stream.
• We propose an HFA block that interchanges, purifies, and reinforces the HF information based on DCT. In the HFA block, the HF information is extracted and reinforced by the proposed extraction method and RCAB.
• We propose the learnable inter- and intra-weight parameters for HF interchange and reinforcement according to the fusion phases. The features of each stream can be adaptably enhanced and fused in each phase using learnable parameters.

2. Related Work

2.1. Object Detection Models

Owing to the development of the CNN, object detectors demonstrate remarkable performance. Several outstanding works have been proposed for object detection, including CenterNet [9], Faster R-CNN [10], and the YOLO series. The YOLO series has shown remarkable accuracy and inference speed, advancing the one-stage object detection design.

In addition, YOLOv4 [11] employs cross stage partial (CSP) darknet, which matches almost all optimal architecture features obtained by the network architecture search technique as a backbone. The scaled YOLOv4 was proposed based on the CSP approach by scaling the features, and it applies to both small and large networks. Moreover, YOLOv5 [12] has four models: YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. Generally, YOLOv5 uses the architecture of CSPDarknet with a spatial pyramid pooling layer as a backbone. Further, YOLOv6 [13] was proposed using a self-distillation strategy performed on the classification and regression tasks. Furthermore, YOLOv6 dynamically adjusts the information from the teacher and labels it to help the student model learn knowledge more efficiently during all training phases.

Next, YOLOv7 [4] is the latest work in the YOLO series. This network further improves the detection speed and accuracy based on the previous work. Specifically, in terms of the overall architecture, this paper proposes extended-ELAN (E-ELAN). Specifically, the ELAN uses expand, shuffle, and merge cardinality to continuously enhance the network learning ability without destroying the original gradient path by considering the following design strategy [14]. In addition, E-ELAN can guide various groups of computational blocks to learn diverse features. Further, YOLOv7 also proposes a compound model scaling method to maintain the model properties from the initial design and the optimal structure. Regarding network optimization strategy, YOLOv7 introduces model reparameterization and dynamic label assignment, analyzes the existing problems, and reduces them.

Specifically, YOLOv7 was proposed with direct access to the cascade of ResNet [15] or DenseNet [16], providing more gradient diversity for various characteristic graphs. However, these structures destroy the network structure because RepConv [17] has an identity connection. Therefore, the YOLOv7 was proposed by removing the identity connection in RepConv and designing the planned reparameterized convolution, realizing the efficient combination of the reparameterized convolution and various networks. In addition, YOLOv7 uses the idea of deep supervision and adds an additional auxiliary head structure in the middle network layer as an auxiliary loss to guide the weight of the shallow network. These mono-modality object detection methods provide real-time inference and achieve object detection performance.

However, these object detection models use only one stream. Hence, these models cannot exploit the advantages of each stream, such as the object color and detail information in RGB images and the precise edges and proper illumination in IR images. The proper feature utilization is required across each stream to acquire better object detection performance.

2.2. Fusion Mechanism of the Multispectral Object Detection

Despite numerous attempts at object detection, problems with improving performance by fusing features from other input modalities still exist. The conventional fusion mechanism-based approaches [18,19,20] focus on preprocessing the input image to obtain a fusion of RGB and IR information. Moreover, these fusion mechanism-based models can be categorized by the position of the feature fusion, such as early, late, and halfway fusion. Moreover, these approaches focus on fusing the features, such as through element-wise adding, multiplication, max-pooling, average-pooling, element-wise product, and other methods. Among the previous studies, Wagner et al. [21] analyzed two fusion methods, early and late fusion, and their performance on multispectral datasets. Based on the analysis, Wagner et al. proposed the cyclic fuse-and-refine (CFR) method to improve performance by using cyclical fuses and refines for each spectral feature. Liu et al. [22] designed two other fusion methods, halfway and score fusion, using two convolutional networks. The guided attentive feature fusion (GAFF) [23] was proposed to guide efficient and effective multispectral feature fusion by using attention modules. A gated fuse unit [24] was proposed to learn the combination of feature maps generated between RGB and IR streams to determine an optimal fusion mechanism. Recently, a cross-modality fusion transformer (CFT) [25] was proposed to combine the features of RGB and IR streams with state-of-the-art (SOTA) performance in a multispectral object detection task by using a transformer [26]. In particular, the CFT learns long-range dependencies and integrates global contextual information at the feature level. The network can robustly capture the latent interactions between RGB and IR images. These multimodality object detection methods present higher accuracy than mono-modality methods by exploiting the features of each stream. Moreover, these methods efficiently fuse the features of each stream in the feature domain. Hence, these models contain more powerful feature information than other approaches.

However, these fusion-based models [21,22,23,24,25] utilized the unpurified object detection-irrelevant features (occluded objects in RGB images, the color of IR images, etc.). As result, these object detection-irrelevant features may be fused, leading to instability in model training and performance degradation. Moreover, even though these fusion-based models fuse the features of each stream, they do not fully adaptably fuse the features of each stream according to the feature level. Therefore, the fusion information must be adaptably fused according to the feature level and fusion phase. Thus, we propose the Infusion-Net, which gradually fuses the features of each stream according to the feature level using learnable weights. It provides optimal feature utilization and selectively enhances the features according to the fusion phase. When fusing the features in the Infusion-Net, the object detection-irrelevant features are eliminated through HF extraction. Only object detection-relevant features are interchanged and reinforced by HFA.

3. Proposed Method

This section describes the Infusion-Net for multispectral object detection. Section 3.1 provides an overview of DCT. Section 3.2 describes the HFA based on the DCT. Finally, Section 3.3 presents the overall architecture of Infusion-Net.

3.1. Discrete Cosine Transform

This section describes the HFA block, which interchanges, purifies, and reinforces the HF information based on DCT. Before describing the HFA, we explain why we use the DCT in the HFA blocks and the principle behind the DCT [27].

The image of the spatial domain can be transformed into the frequency of the spectral domain. Inversely, the frequency can be transformed into an image without loss of image quality. The discrete Fourier transform is commonly employed to transform the image or frequency. However, even if the input value consists of integer values, the transformed output includes complex numbers. Calculating complex numbers causes memory overhead and high computational costs in CNN-based models. To address this issue, we apply the DCT, transforming the input values into integer values using the cosine function. We convert the two-dimensional (2D) images to a frequency; thus, we employ the 2D DCT and 2D inverse DCT. The equations for the 2D DCT and 2D inverse DCT (IDCT) are represented as follows:

$$2D\_DCT\left(u,v\right)=\alpha \left(u\right)\beta \left(v\right){\displaystyle \sum }_{\mathrm{i}=0}^N{\displaystyle \sum }_{j=0}^{M}f\left(i,j\right)\cos (\frac{\pi \left(2i+1\right)u}{2N})\cos \left(\frac{\pi \left(2j+1\right)v}{2M}\right),$$

(1)

$$2D\_IDCT\left(i,j\right)={\displaystyle \sum }_{\mathrm{u}=0}^N{\displaystyle \sum }_{j=0}^M\alpha \left(u\right)\beta \left(\mathrm{v}\right)F\left(u,v\right)\cos (\frac{\pi \left(2u+1\right)i}{2N})\cos \left(\frac{\pi \left(2v+1\right)j}{2M}\right),$$

(2)

$$\alpha \left(u\right)=\left\{\begin{array}{c}\sqrt{\frac{1}{N}}, u=0\\ \sqrt{\frac{2}{N}}, u\ne 0\end{array}\right.,$$

(3)

$$\beta \left(v\right)=\left\{\begin{array}{c}\sqrt{\frac{1}{M}}, v=0\\ \sqrt{\frac{2}{M}}, v\ne 0\end{array}\right. ,$$

(4)

where $F\left(u,v\right)$ denotes the value of the transformed frequency, where the pixel value $f\left(i,j\right)$ of the $\left(i,j\right)$ position of the image is transformed, and $f\left(i,j\right)$ denotes the value of the transformed image. Equations (1) and (2) represent the 2D DCT and 2D IDCT, respectively. Equations (3) and (4) display the cosine basis function and regularization constant.

In the DCT, the high frequency is concentrated on the bottom-right side of the frequency. In contrast, the low frequency is concentrated on the top-left side. With this principle, we can adaptably extract the desired HF information via the binary mask $\mathcal{M}$. The mask has zero values at the top of the diagonal and a value of one at the bottom of the diagonal. Hence, we can extract the desired HF information through hyperparameters $\tau$ that adjust diagonal positions, as presented in Figure 4a. Black pixels indicate a value of zero, and white pixels indicate a value of one. The binary mask function can be formulated as follows:

$$\mathcal{M}\left(i,j\right)=\left\{\begin{array}{c}0, y<-i+2\tau w\\ 1, \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e} \end{array}\right. ,$$

(5)

where $w$ denotes the image width, and $i \mathrm{and} j$ denote the horizontal and vertical coordinates of $\mathcal{M}$, respectively. The hyperparameter $\tau$ ranges from 0 to 0.5. The generated $\mathcal{M}$ is used to extract HF components $\mathit{H}\mathit{f}$ by multiplying the binary mask $\mathcal{M}$ and the input, transformed by the 2D DCT. The equation for the HF extraction $HE$ is formulated as follows:

$$\mathit{H}{\mathit{f}}_{DCT}=2D\_DCT\left(\mathit{I}\right) ⨂ \mathcal{M},$$

(6)

$$\mathit{H}\mathit{f}=2D\_IDCT\left(\mathit{H}{\mathit{f}}_{DCT}\right),$$

(7)

where $\mathit{I}$ denotes the input and $⨂$ denotes the element-wise product. In addition, $2D\_DCT$ and $2D\_IDCT$ denote the 2D DCT and 2D IDCT functions, respectively. Figure 4b depicts the results of the HF extraction, according to hyperparameter $\tau$. As illustrated in Figure 4b, when hyperparameter $\tau$ is too small, object detection-irrelevant information, such as an unwanted color, exists, and when the hyperparameter $\tau$ is too large, object detection-relevant information, such as an edge of an object, is removed. Hence, the proper hyperparameter $\tau$ is needed to improve object detection performance.

In object detection tasks, the edges and texture information for objects are essential to improve object detection performance. This information helps detect the object position and classify the object class. This paper demonstrates that edges and texture information are included in HF information. Figure 5 presents the DCT-based HF extraction results according to the HF extraction order. The RGB and IR images in Figure 5a,b are transformed into the DCT frequency domain. Afterward, HF information is extracted using a predefined binary mask. The extracted HF information is presented in Figure 5c,d. As illustrated in Figure 5e,f, the HF information contains edge and texture information that significantly affects the detection performance. Figure 5e displays more noticeable edges of objects than the original RGB image, making it difficult to recognize the object position with the naked eye. These results demonstrate that HF extraction can help improve object detection performance.

3.2. High-Frequency Assistant Based on the Discrete Cosine Transform

Based on the HF observation, we devised the HFA block, which interchanges, purifies, and reinforces HF features, as depicted in Figure 6. We applied the HF extraction to the feature level, not the image level. The HFA block transforms the features of the RGB and IR streams to the DCT frequency. Then, only the necessary HF features are extracted using a predefined binary mask $\mathcal{M}$. The extracted HF features are transformed by the IDCT. With this process, the object detection-irrelevant features are purified. In addition, we reinforced the HF information using an RCAB [8]. As the RCAB focuses on the correlation around the features, the HF information is more precise and sharper. The reinforced HF information is added to the original feature. The enhanced features of the RGB and IR streams are interchanged with each stream. Hence, the proposed model has powerful representation compared with other approaches. The equation for the HFA is formulated as follows:

$$\left({\mathit{f}}_{\mathit{r}\mathit{g}\mathit{b}}+\mathcal{F}\left({\mathit{f}}_{\mathit{i}\mathit{r}}\right), {\mathit{f}}_{\mathit{i}\mathit{r}}+\mathcal{F}\left({\mathit{f}}_{\mathit{r}\mathit{g}\mathit{b}}\right)\right)=HFA\left({\mathit{f}}_{\mathit{r}\mathit{g}\mathit{b}},{\mathit{f}}_{\mathit{i}\mathit{r}}\right),$$

(8)

$$\mathcal{F}\left(\mathit{f}\right)=\mathit{f}+\mathcal{R}\left(HE\left(\mathit{f}\right)\right),$$

(9)

where ${\mathit{f}}_{\mathit{r}\mathit{g}\mathit{b}}$ and ${\mathit{f}}_{\mathit{i}\mathit{r}}$ denote features of the RGB and IR streams, respectively, $\mathcal{R}$ represents the process of the RCAB, $\mathcal{F}$ denotes the function for reinforcing the HF information in the HFA, and $HE$ is the process of HF extraction, as in Equations (6) and (7).

3.3. Overall Architecture of Inter- and Intra-Weighted Cross-Fusion Network (Infusion-Net)

This section describes the Infusion-Net for multispectral object detection. As presented in Figure 7, the Infusion-Net primarily consists of RGB and IR streams. Each stream receives an RGB image and an IR image as input. Then, four HFA blocks are employed to interchange the features of each stream, purifying only the necessary information. By dividing the interchange part according to the phase, we adaptably fuse the RGB and IR features by feature level. When fusing the features of each stream, the learnable intra- and inter-weight parameters are employed for the components of each stream. The intra-weight parameter $\alpha$ controls the degree of feature enhancement. As revealed in Figure 7, the extracted HF features of each stream are multiplied by each intra-weight parameter $\alpha$ according to each phase. The features are adaptably enhanced, according to each phase; thus, the Infusion-Net can exploit the optimal feature information depending on the fusion phase. The inter-weight parameter $\beta$ controls the degree of utilization of the other stream feature, as depicted in Figure 7. Each stream adaptably receives features of another stream, according to feature level; thus, the Infusion-Net comprises the advantages of each stream. The features of each stream according to the phase in the Infusion-Net can be formulated as follows:

$${\mathit{f}}_{rgb}^{i+1}={\mathit{f}}_{rgb}^i+\left(HE\left({\mathit{f}}_{rgb}^i\right)\times {\alpha }_{rgb}^{pi}\right)+\left(\mathcal{F}\left({\mathit{f}}_{ir}^i\times {\beta }_{ir}^{pi}\right)\right),$$

(10)

$${\mathit{f}}_{ir}^{i+1}={\mathit{f}}_{ir}^i+\left(HE\left({\mathit{f}}_{ir}^i\right)\times {\alpha }_{ir}^{pi}\right)+\left(\mathcal{F}\left({\mathit{f}}_{rgb}^i\times {\beta }_{rgb}^{pi}\right)\right),$$

(11)

where $i$ denotes the phase of the Infusion-Net, ${\mathit{f}}_{rgb}^i$ and ${\mathit{f}}_{ir}^i$ represent the features of the RGB and IR streams when the phase is $i$, respectively, ${\alpha }_{ir}^{pi}$ and ${\beta }_{ir}^{pi}$ are the intra- and inter-weight parameters of the IR stream, respectively, and ${\alpha }_{rgb}^{pi}$ and ${\beta }_{rgb}^{pi}$ denote the intra- and inter-weight weight parameters of the RGB stream, respectively.

The fused features of each phase are concatenated with the detection module. When concatenating the features of each phase, the stream scaler weights are applied to emphasize the features of the stream according to phase selectively. Therefore, the Infusion-Net can filter the object-irrelevant features and emphasize more precise object-relevant features. A concatenated feature $C$ is formulated as follows:

$${\mathit{f}}_{\mathit{m}}^{\mathit{i}}=\left({\mathit{f}}_{rgb}^i\times w_{rgb}^{pi}\right)+\left({\mathit{f}}_{ir}^i\times w_{ir}^{pi}\right)$$

(12)

$$\mathit{C}=Concat\left({\mathit{f}}_{\mathbf{m}}^1, {\mathit{f}}_{\mathit{m}}^{\mathbf{2}}, {\mathit{f}}_{\mathit{m}}^3\right),$$

(13)

where $w_{rgb}^{pi}$ and $w_{ir}^{pi}$ denote the stream scaler weights of each stream in phase $i$, ${\mathit{f}}_{\mathit{m}}^{\mathit{i}}$ denotes fused features by using stream scaler, and $Concat$ denotes the feature concatenation function.

As depicted in Figure 7, the concatenated features are used to predict the position and class of the object using the detection module. As in other approaches, we designed the detection module by applying YOLO v7 [4].

The structures of each phase are described in Figure 8. In Phase 0, the features are extracted by four convolutional layers from each image. The extracted features are extracted to another size of features to exploit the different scale features to improve the performance. In another phase, the max-pooling layer and more convolutional layers are employed to reduce the model parameters and inference time while maintaining the model representability. The extracted features are concatenated in the concatenation layer and aggregated by the convolutional layer.

The step-by-step block diagram of an application of Infusion-Net, as shown in Figure 9. The RGB and IR images are obtained by each measurement, such as from the RGB and IR camera. In general, the resolution of an RGB image is higher than that of an IR image. Hence, the images are aligned to match the position of the objects; then, aligned images are inputted into Infusion-Net. In Infusion-Net, the features of each stream are gradually fused with inter- and intra- weights and HFA. The features of each phase are utilized in the detection module. The detection module predicts the class and position of objects in the input image.

4. Experiments and Analysis

4.1. Experimental Setup

4.1.1. Dataset

In the real world, the resolution of an RGB image is commonly higher than that of an IR image. As the positions of objects are mismatched, object detection models suffer performance degradation in multispectral object detection tasks. Hence, we used the FLIR [6] and LLVIP [7] datasets, which have RGB-IR pairs with the object position exactly matched. The FLIR dataset [6] is a multispectral object detection dataset that includes bright day and night scenes. The FLIR dataset consists of 5142 RGB-IR image pairs with 640 $\times$ 512 resolution. The classes of the FLIR dataset comprise people, cars, and motorcycles. In this experiment, 4129 RGB-IR image pairs were used for model training and 1013 for evaluation. The LLVIP [7] is a pedestrian detection dataset in a low-light night environment. The LLVIP dataset consists of 15,487 RGB-IR image pairs, and the object positions are accurately aligned. The resolution of the LLVIP dataset is 1280 $\times$ 1024. We split LLVIPR image pairs into 12,025 RGB-IR pairs for model training and 3,462 RGB-IR image pairs for evaluation.

4.1.2. Metric

We used the mean average precision (mAP) to analyze the detection results, an accuracy metric generally used to evaluate object detection models. We employed the mAP50, mAP75, and mAP50:95. The mAP50 indicates the mAP at the intersection over union (IoU) threshold of 0.5. The mAP75 is the mAP at the IoU threshold of 0.75, and mAP50:95 represents the average mAP at the IoU threshold of 0.5 to 0.95, with intervals of 0.05.

4.1.3. Environment

We implemented the proposed framework in PyTorch 1.8.0 and used Python 3.8.3, CUDA 11.2, and cuDNN 8.2.0. The experiment was performed using an AMD Ryzen 5 5600X 6-Core Processor CPU with 32 GB of memory and an NVIDIA RTX 3090 GPU. The hyperparameters of the framework were as follows: the initial learning rate was 10⁻², the momentum was 0.937, and the weight attenuation was 0.0005. The training batch size was set to 10, and the number of epochs was set to 100.

4.2. Comparison of Conventional and Proposed Multispectral Object Detection Methods

We set YOLOv5 [12] and YOLOv7 [4] as the baselines in this experiment. Specifically, YOLOv7 provides SOTA accuracy and speed in the object detection task. In addition, we conducted experiments on the mono-modality object detection models to demonstrate the advantage of multimodality. Furthermore, we set the YOLO series-based multimodality models as baselines; they consist of two back-bone structures, similar to the multimodality-based models. The multimodality-based models, such as CFR [21], GAFF [23], and CFT [25], are used to verify the Infusion-Net. Each object detection model is trained from scratch according to their losses and training methods on FLIR and LLVIP datasets.

Table 1. Comparison of the Infusion-Net and other models on the FLIR dataset [6].

Model	Data	Backbone	mAP50 (%)	mAP75 (%)	mAP50:95 (%)
Mono-modality model
YOLOv5 [12]	RGB	CSPDarknet	67.8	25.9	31.8
YOLOv5 [12]	IR	CSPDarknet	73.9	35.7	39.5
YOLOv7 [4]	RGB	E-ELAN	70.2	32.7	31.6
YOLOv7 [4]	IR	E-ELAN	75.6	32.2	38.2
Multimodality model
halfway fusion [21]	RGB + IR	VGG16	71.2	-	-
CFR [21]	RGB + IR	VGG16	72.4	-	-
GAFF [23]	RGB + IR	ResNet18	72.9	32.9	37.5
GAFF [23]	RGB + IR	VGG16	72.7	30.9	37.3
YOLOv7 [4]	RGB + IR	E-ELAN	77.5	34.0	39.0
CFT [25]	RGB + IR	CFB	77.7	34.8	40.0
Infusion-Net (Ours)	RGB + IR	Infusion-Net	79.1	35.2	40.3

and

Table 2. Comparison of the Infusion-Net and other models on the LLVIP dataset [7].

Model	Data	Backbone	mAP50 (%)	mAP75 (%)	mAP50:95 (%)
Mono-modality model
YOLOv5 [12]	RGB	CSPDarknet	90.8	51.9	50.0
YOLOv5 [12]	IR	CSPDarknet	94.6	72.2	61.9
YOLOv7 [4]	RGB	E-ELAN	91.9	52.9	51.2
YOLOv7 [4]	IR	E-ELAN	96.0	72.9	63.9
Multimodality model
YOLOv5 [12]	RGB + IR	CSPDarknet	95.8	71.4	62.3
YOLOv7 [4]	RGB + IR	E-ELAN	96.9	73.1	64.4
CFT [25]	RGB + IR	CFB	97.5	72.9	63.6
Infusion-Net (Ours)	RGB + IR	Infusion-Net	98.6	73.3	64.6

compare the Infusion-Net and other models on the FLIR and LLVIP datasets. The Infusion-Net has 3.5% and 7.8% higher mAP50 values than the YOLOv7 mono-modality model on the FLIR and LLVIP datasets, respectively, demonstrating the superiority of the dual-stream approach. Among the multimodality models, the Infusion-Net performs best for mAP50, mAP75, and mAP50:95 on the FLIR and LLVIP datasets, as listed in Table 1 and Table 2. In particular, the Infusion-Net scores 1.4% higher on the FLIR dataset and 1.1% higher on the LLVIP dataset than CFT, which has the second-best score. The performance of the Infusion-Net proves the superiority of this approach.

In this work, we propose an Infusion-Net that gradually fuses the features of RGB and IR streams, depending on each fusion phase, using four HFA blocks to exploit the object detection-relevant features of each stream. The HFA extracts HF information, which has clear boundaries and textures of the object, and the extracted HF information is strengthened by the RCAB. Enhanced features are exchanged between each stream with learnable inter- and intra-weight parameters and stream scaler weights. Using learnable weights, the Infusion-Net enables effective feature utilization and enhancement with the fusion phase, leading to improved object detection accuracy.

To qualitatively evaluate the Infusion-Net, we compared it with CFT [25], which has the second-best mAP50 on the FLIR and LLVIP datasets. In Figure 10, the Infusion-Net detects a car, whereas CFT provides the missing object. As the Infusion-Net gradually fuses the features of RGB and IR streams according to the feature level, the Infusion-Net is better at detecting small objects than the CFT model, improving the object detection performance. As depicted in Figure 11, the Infusion-Net detects the person, whereas CFT provides the missing person. The visual outcomes present that Infusion-Net exploits HF information using HFA, which shows a precise edge of the object and texture, even if the input image exhibits dark illumination. Therefore, the Infusion-Net is robust in low-light environments.

4.3. Ablation Study

4.3.1. Learnable Weights

This section investigates the effects of learnable weights, such as inter- and intra-weight parameters and stream scaler weights.

Table 3. Quantitative results for evaluating the effects of learnable weights on the FLIR dataset.

Intra-Weights $\mathit{\alpha }$	Inter-Weights $\mathit{\beta }$	Scaler Weights $\mathit{w}$	mAP50 (%)
✗	✗	✗	77.9
✗	✗	$\mathsf{O}$	78.6
✗	$\mathsf{O}$	✗	78.5
$\mathsf{O}$	✗	✗	78.0
$\mathsf{O}$	$\mathsf{O}$	$\mathsf{O}$	79.1

reveals that the learnable weights applied to the Infusion-Net improve object detection performance. In particular, the Infusion-Net exhibits the best mAP50 when all learnable weights are employed, indicating that the proposed learnable weights provide optimal feature utilization and selectively emphasize the features. Thus, the learnable weight approach is suitable for the multimodality model.

4.3.2. Hyperparameters for High-Frequency Extraction

This section specifies how we determined the hyperparameters for HF extraction. As listed in

Table 4. Quantitative results for evaluating the effects according to hyperparameter $\tau$ for the high-frequency extraction on the FLIR dataset.

Hyperparameter $\mathit{\tau }$	mAP50 (%)
0	77.1
0.1	78.3
0.2	79.1
0.3	77.4
0.4	76.8
0.5	75.5

, the Infusion-Net exhibits the best mAP50 when hyperparameter $\tau$ is 0.2. In contrast, when the hyperparameter $\tau$ is 0, the Infusion-Net does not provide the best mAP50 score because the object detection-irrelevant features are not purified. In addition, when the hyperparameter is greater than 0.2, the object detection-relevant features are eliminated. Thus, the Infusion-Net cannot exhibit the optimal mAP50 score, demonstrating that the detection-relevant features, such as the edges and textures, are included in properly extracted HF information. Hence, the proper hyperparameter is needed to improve accuracy. In this paper, we empirically set the hyperparameter $\tau$ to 0.2.

4.3.3. Computational Complexity

The proposed Infusion-Net consists of two backbone structures for multimodality; hence, the proposed model requires more inference time than the mono-modality models. Moreover, the resolution of an RGB image is higher than that of an IR image. Hence, Infusion-Net requires the image alignment process between an RGB and IR image, unlike other mono-modality models. Hence, Infusion-Net requires more computational complexity than mono-modality models. Nevertheless, the inference time for the Infusion-Net is 30 ms, whereas the inference time for the multimodality model CFT is 78 ms, when the resolution of the input images is 640 $\times$ 512. Even if the proposed model requires more inference time than the mono-modality models, the Infusion-Net performs an average of 5.6% higher on mAP50 than the mono-modality models on the FLIR and LLVIP datasets.

5. Conclusions

In this paper, to obtain the benefits of each stream, we suggest the Infusion-Net, which gradually fuses the features of RGB and IR streams, depending on the feature level. To this end, we devised an HFA that interchanges, purifies, and reinforces the object detection-relevant information based on the DCT and RCAB. In addition, the learnable inter- and intra-weight parameters and stream scaler weights provide optimal feature utilization across streams and selectively emphasize the features of the stream according to the fusion phase. The experimental results of the proposed model reveal the best performance in the mAP50, mAP70, and mAP50:95, with the fastest inference time, demonstrating the effectiveness of the Infusion-Net. The visual outcomes show more exact results than other SOTA models, demonstrating the superiority of Infusion-Net. In addition, the Infusion-Net can be considered for various object detection applications, such as fault diagnosis [28,29], autonomous driving [30], face attribute recognition [31,32], and smart parking in low-light environments. In particular, our Infusion-Net can be applied to prevent accidents such as car and pedestrian accidents at night because Infusion-Net presents a robust performance in low-light environments. In future work, the Infusion-Net will be extended to multi-input-based computer vision fields, such as RGB-LiDAR, RGB-Depth, and 3D images. In addition, the super-resolution methods can be employed to improve the object detection performance when input images are small.