Attention Network with Outdoor Illumination Variation Prior for Spectral Reconstruction from RGB Images

Abstract

Hyperspectral images (HSIs) are widely used to identify and characterize objects in scenes of interest, but they are associated with high acquisition costs and low spatial resolutions. With the development of deep learning, HSI reconstruction from low-cost and high-spatial-resolution RGB images has attracted widespread attention. It is an inexpensive way to obtain HSIs via the spectral reconstruction (SR) of RGB data. However, due to a lack of consideration of outdoor solar illumination variation in existing reconstruction methods, the accuracy of outdoor SR remains limited. In this paper, we present an attention neural network based on an adaptive weighted attention network (AWAN), which considers outdoor solar illumination variation by prior illumination information being introduced into the network through a basic 2D block. To verify our network, we conduct experiments on our Variational Illumination Hyperspectral (VIHS) dataset, which is composed of natural HSIs and corresponding RGB and illumination data. The raw HSIs are taken on a portable HS camera, and RGB images are resampled directly from the corresponding HSIs, which are not affected by illumination under CIE-1964 Standard Illuminant. Illumination data are acquired with an outdoor illumination measuring device (IMD). Compared to other methods and the reconstructed results not considering solar illumination variation, our reconstruction results have higher accuracy and perform well in similarity evaluations and classifications using supervised and unsupervised methods.

1. Introduction

Compared with RGB images, HSIs are characterized by the combination of spatial and spectral information. Due to numerous spectral bands, HSIs can capture the spectral distribution, which is helpful to describe the detailed spectral characteristics of the scene. The abundant spectral information of HSIs is beneficial to numerous applications such as those for object detection [1,2,3], classification [4,5,6], segmentation [7], face recognition [8] and other aspects.

Although hyperspectral devices have many obvious advantages, the following factors limit their further application: (1) due to the complexity of their optical systems, most hyperspectral devices are too heavy to be mounted on portable platforms, such as drones and hand-held cameras [9]; (2) hyperspectral devices are expensive and not easily affordable [10]; and (3) the long-term acquisition of HSIs results in spatial and spectral degradation, which is unsuitable for rapid acquisition or scenes with moving objects [11].

Compared to hyperspectral devices, RGB cameras have the advantages of smaller sizes, lighter weights, shorter imaging times, higher signal-to-noise ratios, and minor geometric deformations [12]. The reconstruction of HSIs using RGB images can overcome many of the limitations of existing hyperspectral instruments and combine the advantages of HSIs and RGB images. In this way, high-precision and high-quality HSIs can be obtained using low-cost equipment, and the spectral resolutions of existing RGB cameras can be improved [11].

The essence of reconstructing HSIs from RGB images is to learn the mapping relationship of the spectral dimension between corresponding pixels. This is a sick problem [13]. Prior to the rise of deep learning, there was a lot of research on learning the mapping of RGB images to the the corresponding HSIs. Some early methods for RGB spectral recovery images relied on linear models that adopted statistical analysis strategies, which assumed that 3–10 basis functions can usually represent HSIs. Then, most methods were based on PCA [14,15] or the matrix R approach [16]. However, sparse coding approaches outperformed them quickly [17,18]. In recent years, with the development of computer hardware and the abundance of datasets, deep learning methods have caused wide concern. Deep learning has shown outstanding ability in image generation [19,20,21], and Goodfellow et al. [19] introduced generative adversarial networks (GANs), a framework that significantly advanced image generation. GANs employ adversarial training between a generator and a discriminator, leading to the synthesis of realistic images by mapping random noise to the target data distribution. Isola et al. [20] make a significant contribution to image generation through conditional adversarial networks. It introduces an effective approach for image-to-image translation, enabling the synthesis of realistic and high-quality images in various visual domains. Dong et al. [21] propose an accelerated super-resolution convolutional neural network (SRCNN). Their work focuses on enhancing the efficiency of super-resolution tasks, providing a valuable advancement in generating high-quality and detailed images. A series of CNN-based SR models is also presented to learn a mapping function from a single RGB image to its corresponding HSI [22,23,24,25,26,27]. Xiong et al. [24] present an innovative contribution with an HSCNN, a convolutional neural network (CNN) designed for the recovery of hyperspectral images from spectral projections. This work addresses the challenge of reconstructing high-quality hyperspectral imagery from limited spectral information, offering advancements in the field of image recovery. Kaya et al. [22] focus on spectral estimation from a single RGB image in real-world scenarios, addressing the challenges posed by diverse environmental conditions. Arad et al. [23] emphasize spectral reconstruction from RGB images and foster benchmarking in the field. Fu et al. [25] introduce a novel approach for coded hyperspectral image reconstruction, leveraging deep external and internal learning methods. Their work enhances the efficiency of spectral recovery processes. Shi et al. [26] present an HSCNN+, an advanced CNN-based method for hyperspectral recovery from RGB images. Their contribution lies in the development of an effective model for reconstructing hyperspectral information, demonstrating superior performance in comparison to existing methods. Li et al. [27] propose a progressive spatial–spectral joint network for hyperspectral image reconstruction, introducing a novel approach that integrates spatial and spectral information progressively.

Illumination is an important source of error in SR. It mainly causes the variation in the intensity and angle of incident light. Nguyen et al. show the reconstruction result without considering the illumination is not robust, so the white balance method is used to improve the accuracy of the result and make the result more robust [28]. This variation will affect the radiance of objects. Fu et al. prove that illumination has an impact on the accuracy of reconstructed images. They improve the model to reconstruct high-precision hyperspectral images under different illumination conditions [13]. In order to improve the accuracy of reconstruction, some methods have considered the effect of illumination on SR. Chi et al. develop an approach by leveraging optimized wide-band filtered illumination [29]. Nguyen et al. present a strategy to model the mapping relationship between RGB values which are acquired by camera and scene spectral by using training images [28].

However, due to a lack of consideration of outdoor solar illumination variation in existing reconstruction methods, the accuracy of outdoor SR remains limited, especially for long-term HS imaging. To address this weakness, we develop an attention network based on prior inter-frame illumination variation information. Our network is based on AWAN [10], which performs well in SR. AWAN consists of a dual residual attention block (DRAB) and a patch-level second-order non-local (PSNL) module. It can not only allow abundant low-frequency information to be bypassed to enhance feature correlation learning but also capture long-range spatial contextual information. The outdoor solar illumination is fed into the network via a basic 2D block. As a weighted attention mechanism, the weights can guide the network to generate HSIs in which the solar illumination variation is corrected. Finally, to make full use of the low-frequency features of RGB images, we sum the output of AWAN and RGB features after a layer of convolution.

The main contributions of this paper are summarized as follows.

(1) The consideration of the impact of solar illumination variation on UAV hyperspectral imaging. To minimize the influence of illumination variation on SR, we conduct relative illumination correction on the push-broom line of each hyperspectral image to generate HSI and RGB data, which are not affected by solar illumination variation. This method of reconstruction closely represents the outdoor situations of imaging.

(2) The presentation of an attention network framework based on prior inter-frame outdoor solar illumination variation information. The backbone of the proposed network is based on an AWAN. The illumination information is introduced into the network as prior information via the basic 2D block to reconstruct HSIs from RGB images more accurately.

(3) The collection of hyperspectral dataset with a spectral range of 400–1000 nm using an unmanned aerial vehicle (UAV). The dataset contains 120 images and labels of illumination information. The illumination information precisely corresponds to GPS time. Compared to all existing datasets, our dataset is for UAV reconstruction that considers outdoor solar illumination variation. It can provide support for research into outdoor spectral imaging, processing, and analysis in the future.

2. Related Works

In order to intuitively present the changes in characteristics of the outdoor observed object or region, it is necessary to consider the effects of solar illumination and atmospheric factors during imaging to improve the accuracy of SR.

2.1. Influence of Outdoor Illumination Variation

Scholars consider the influence of illumination, and the experimental results show that illumination is an important factor in spectral reconstruction. A stable light source is generally used for illumination in laboratories, so there are no influences from atmospheric variation. The intensity and incident angle of outdoor illumination change over time, which causes variation in the radiance received by sensors and multiple scattering between objects [30,31,32]. Ximena et al. [33] studied the influence of incident illumination on spectral data acquisition at different times and dates. J. Pablo et al. [34] evaluated the effects of push-broom hyperspectral cameras on imaging under various outdoor illumination conditions.

Some methods have considered the effect of illumination on spectral reconstruction. Nguyen et al. [28] present a radial basis function network that uses white balancing to reconstruct spectral reflectance images from single RGB images. This white-balance step helps to make the approach robust to input images captured under illumination conditions that are not represented in the training data. To obtain multispectral reflectance information, Park et al. propose a multiplexed illumination technique to recover the continuous spectral information for each scene point using a linear model for spectral reflectance [35]. All above methods try to remove or reduce the illumination effect and conduct SR. However, variation in outdoor solar illumination is ignored.

2.2. Influence of Time Measurements Mismatch

Jong-Min et al. [36] composed a dual ASD FieldSpec^® 3 system with two mutually calibrated ASD spectrometers. This system simultaneously used two radiometers for mutual calibration and wireless synchronization and also simultaneously collected the reference spectra of standard boards and the spectra of sample targets. The main purpose of the system is to eliminate the errors caused by the variation in illumination conditions when the measurement time is not strictly synchronized. Burkart et al. [37] use two marine optical spectrometers: one mounted on a multirotor UAV to measure the spectra of ground objects and the other to measure standard plates on the ground. This method calculates reflection coefficients through the continuous measurement of target and Lambert reference panels. All of the above scholars take into account the influence of outdoor solar illumination variation on HS measurement calculations and aim to reduce the errors caused by solar illumination variation and target measurement time asynchronicity. It only takes a short time to collect RGB images of scenes. However, HSIs are acquired by spatial push-broom or band-wise models; this way takes a long time. Due to the longer acquisition times, outdoor solar illumination variation becomes a source of errors in SR. In real imaging scenes, solar illumination variation cannot be ignored in spectral reconstruction.

To conduct high-precision SR, it is necessary to consider changes in the environment during the acquisition period. If these variations are not considered, the reconstructed results can only represent the virtual HS information at the moments when the RGB images are taken rather than being closely representative of the real results collected by HS instruments under real conditions. During the process of SR, differences in acquisition times are mainly caused by the frame frequency limits of spectroscopic systems and detectors. The imaging times of HS cameras are usually longer than those of RGB cameras. Generally, the instruments used for spectral acquisition extend acquisition times to improve the signal-to-noise ratio (SNR). It is necessary to consider the real imaging environment during HSI reconstruction from RGB images. The purpose of SR is to obtain hyperspectral curves via high-cost acquisition methods and analyze the composition properties of substances, which is very important to ensure the accuracy of SR [13,18,38,39,40,41].

CNN-based reconstruction algorithms do not depend too much on specific instrument acquisition systems. Nie et al. [42] advocate a network which based on U-net to learn the filter response functions in the infinite space of nonnegative and smooth curves. Li et al. [10] develop a deep adaptive weighted attention network for SR. This model is proposed to adaptively recalibrate channel-wise feature responses by exploiting adaptive weighted feature statistics. Fu et al. [13] present an efficient network that can jointly select the optimal camera spectral response (CSR) and learn mapping functions to recover HSIs from single RGB images captured with an algorithmically selected camera. Xiong et al. [24] propose a general deep-learning network framework for spectral reconstruction and achieve good results for coded aperture snapshot spectral imaging (CASSI). Zhang et al. [43] present a pixel-aware deep function-mixture network for super-resolution spectral information that can flexibly handle pixels from different categories or spatial positions within HSIs. Han et al. [44] obtained super-resolution spectral information using a spectral reconstruction CNN, which can use the available RGB images to predict the high-frequency content of fine spectral wavelengths in narrowband intervals. Yan et al. [39] introduce a novel framework based on U-Net (C2H-Net), which can reconstruct HSIs using prior information about the classification and location of objects, especially for targets with similar RGB features.

3. Data and Method

3.1. Data

We evaluate the proposed network on our VIHS dataset. The VIHS dataset comprises natural HSIs and the corresponding illumination information. The dataset includes 16 scenes, 100 images for training, 10 images for testing, and 10 images for validation. Several test images are presented in Figure 1.

3.1.1. Illumination Data

The outdoor illumination measuring device (IMD) which is used in this study consists of an ASD, optical fibers, and a cosine corrector (Figure 2). In addition, the solar irradiance at each moment is recorded using GPS time, which is convenient for time matching each frame of data collected by the hyperspectral camera. The cosine corrector can increase the accuracy of solar downward radiation measurement [45]. The spectral resolution obtained by the ASD is 1 nm. To ensure that the solar illumination can be measured by the ASD and the HSIs have the same spectral resolution, the spectral resampling method is adopted to preprocess the illumination data. Figure 3 shows the outdoor solar illumination variation over time.

3.1.2. HSI Data

The raw HSIs in the VIHS dataset are taken on a portable HS camera (Corning^® microHSI™ 410 SHARK [46]) with a spatial resolution of 1364 pixels, a spectral resolution of 4 nm, and 150 spectral bands from 400 to 1000 nm. When we collect the images, the camera is mounted on a UAV. The flight height is 100 m. The solar irradiance observation equipment is placed in a wide-view area. We use the calibration coefficient of the instrument to convert the image digital number (DN) values into radiance and obtain the hyperspectral radiance images. The GPS time is recorded when each frame is acquired by the camera, which can be used to match the time recorded by the IMD.

The processing of each scene of hyperspectral image consists of three parts. Firstly, the first frame of the hyperspectral radiance image is taken as the standard frame, and the IMD time corresponding to this frame is taken as the standard time. The illumination value of standard time is used as the reference, and the illumination correction coefficients are obtained by dividing the illumination value of IMD corresponding to other frame time. Secondly, we take the standard frame as the benchmark, the remaining frames of the hyperspectral radiance image are corrected by the coefficients to obtain the hyperspectral radiance image which is not affected by illumination. The purpose of this step is to obtain RGB image that is not affected by solar illumination variation. All kinds of images have been cropped to a size of 512 × 482 to be consistent with the existing reconstructed dataset.

3.1.3. RGB Data

RGB images of the VIHS dataset are resampled directly from the hyperspectral radiance image that is not affected by illumination under CIE-1964 Standard Illuminant. Since the hyperspectral radiance image not affected by illumination is corrected with illumination, the generated RGB is actually illumination independent. This is in line with the characteristics of general RGB cameras, i.e., no push-broom, less time, and near-instantaneous imaging.

3.2. Method

In this section, we first formulate the problem. Then, the SR is presented. Finally, we provide the learning details.

3.2.1. Formulation

When acquiring an image using a RGB or monochromatic camera, the radiance Y at the location $(i,j)$ of RGB is:

$$Y(i,j)={\int }_{\lambda }\mu \left(\lambda \right)L(i,j,\lambda )d\lambda$$

(1)

where $\mu \left(\lambda \right)$ denotes the CSR function. $L(i,j,\lambda )$ represents that spectral radiance of HSI at the location $(i,j)$. However, when HS data are collected outdoors, $Y(i,j)$ changes with push-broom time. Then, we calculate the illumination correction coefficients $k(\lambda ,t)$ as below:

$$k(\lambda ,t)=S_t\left(\lambda \right)/S_0\left(\lambda \right)$$

(2)

$S_t$ denotes the illumination value of imaging moment t. $S_0$ represents the illumination value of standard time, which is treated as the reference. In this process, the illumination I needs to be resampled to remain consistent with the spectral resolution of HSI.

We can represent the radiance $Y^{*}$ with the illumination correction as follows:

$$Y^{*}(i,j)={\int }_{\lambda }k(\lambda ,t)\mu \left(\lambda \right)L_t(i,j,\lambda )d\lambda$$

(3)

t is the imaging time with the frame of each HSI. In reality, the discrete representation is the formula below:

$$Y^{*}(i,j)=\sum _{n=1}^{N}k({\lambda }_n,t)\mu \left({\lambda }_n\right)L\left(i,j,{\lambda }_n\right)$$

(4)

where ${\lambda }_n(n=1,2,\cdots ,N)$ represents the wavelength $\lambda$ at the n-th spectral band, usually $N=31$. According to the above formula, illumination is an important factor for SR.

3.2.2. Architecture of Network

The entire network structure is shown in Figure 4. The network consists of two convolutions, a basic 2D block (as shown in Figure 5), and a backbone AWAN (as shown in Figure 6), which was proposed by Li et al. [10] and performs well for spectral reconstruction.

We add the illumination data to our network via the basic 2D block, as shown in Figure 5. The basic 2D block consists of two convolutions and a ReLU. The convolutions are used to extract illumination features and the ReLU is used for the nonlinear activation function.

The AWAN consists of 12 dual residual attention blocks (DRABs) (as shown in Figure 7) and a PSNL module (as shown in Figure 8). Each DRAB consists of a residual module, convolutional operations with different kernel sizes, and adaptive weighted channel attention (AWCA), as shown in Figure 9. Long and short skip connections are contained in the block. This module can make full use of the abundant low-frequency information in the input images and enhance feature correlation learning. The AWCA module is adopted for stronger feature correlation.

Given a set of intermediate feature maps $\mathit{F}$, the size of the feature map is $H\times W$. The number of feature maps is C. $\mathit{F}$ is reshaped to $\mathit{Y}\in F_{\mathrm{reshape}}^{1\times H\times W}$, and we reshape $\mathit{Y}\in F_{\mathrm{reshape}}^{1\times H\times W}$ to $\mathit{Y}\in F_{\mathrm{reshape}}^{(H\times W)\times 1}$. Then, softmax is adopted to normalize $\mathit{Y}$, and we multiply $\mathit{F}$ with $\mathit{Y}$, and adaptive weighted pooling $f_{AWP}(·)$ can be represented as below:

$$\mathit{D}=f_{AWP}(·)\left(\mathit{F}\right)$$

(5)

where $\mathit{D}\in F_{\mathrm{reshape}}^{C\times 1}$ is the channel-wise descriptors. In order to exploit the aggregated information of adaptive weighted pooling, AWAC employs sigmoid and two convolutional layers. The final channel map is computed as:

$$\mathit{G}=\delta \left({Conv}_2\left(\sigma \left({Conv}_1\left(\mathit{D}\right)\right)\right)\right)$$

(6)

where ${Conv}_1$ and ${Conv}_2$ denote two convolutional layers. $\delta (·)$ and $\sigma (·)$ represent sigmoid and ReLU, respectively. Then, we assign the channel feature map $\mathit{G}$ to rescale the input $\mathit{F}$. The output feature map $\mathit{E}$ of the AWAC module is obtained as follows:

$$\mathit{E}=\mathit{G}·\mathit{F}$$

(7)

The AWCA module can adjust channel-wise features to adaptively recalibrate, thereby enhancing the network’s representation capabilities.

The PSNL is used to model distant region relationships simultaneously. The inputs for the PSNL $\mathit{O}\in {\mathbb{R}}^{H\times W\times N}$ are divided into four sub-feature maps (${\mathit{O}}_a$, where a = 1, 2, 3, 4). When the sub-feature map $\mathit{O}$ is fed into two convolutions with 1 × 1 kernel size, we obtain the new sub-feature maps ${\mathit{C}}_a$ and ${\mathit{D}}_a$, which we can then reshape and transpose. The spatial attention map $\mathit{M}$ can be represented as ${\mathit{M}}_{\mathit{a}}={\mathit{C}}_{\mathit{a}}\overline{\mathit{I}}{\mathit{C}}_{\mathit{a}}^T$, where $\mathit{I}$ and $\overline{\mathit{I}}$ denote the identity matrix and the all-ones matrix, respectively. Then, we compute ${\mathit{U}}_{\mathit{a}}$ as below:

$${\mathit{U}}_{\mathrm{a}}=softmax\left({\mathit{M}}_{\mathit{a}}\right){\mathit{D}}_{\mathit{a}}$$

(8)

Then, the ${\mathit{U}}_{\mathrm{a}}$ is fed to a $1\times 1$ convolution $\phi$. We obtain the final output ${\mathit{S}}_{\mathit{a}}=\phi \left({\mathit{U}}_{\mathrm{a}}\right)+{\mathit{O}}_{\mathit{a}}$.

The purple line in Figure 4 shows that the illumination correction is performed before resampling HSIs into RGB images, which aims to greatly restore the situation of RGB imaging. To simplify the network, this operation is finished in the preprocessing stage. The green line in the figure represents resampling HSIs into RGB images using the CIE-1964 standard illuminant.

3.2.3. Loss Function

Generally, we adopt the mean relative absolute error (MRAE) as a loss function instead of the mean square error (MSE). As it is affected by illumination and atmospheric factors, the radiance of ground objects in different bands varies greatly and MSE has limitations for this kind of problem. Our HSI reconstruction network can be optimized by minimizing the MRAE between a reconstructed HSI ${\mathbf{P}}_{REC}$ and the corresponding ground truth (GT) image ${\mathbf{P}}_{HSI}$. The formula for MRAE is as below:

$${\mathcal{L}}_{\mathrm{MRAE}}=\frac{1}{WHN}\sum _{n=1}^N\sum _{i=1}^W\sum _{j=1}^H\frac{\left|{\mathbf{P}}_{HSI}^{ijn}-{\mathbf{P}}_{REC}^{ijn}\right|}{{\mathbf{P}}_{HSI}^{ijn}}$$

(9)

i, j represent the row and column of feature maps, respectively, and n denotes the channel.

Li et al. [10] suggest the CSR prior is important for more accurate reconstruction. The CSR function can represent the discrepancies of RGB images resampled from GT HSI and reconstruct HSI more finely; the CSR function is shown as below:

$${\mathcal{L}}_{\mathrm{CSR}}=\frac{1}{WHN}\sum _{n=1}^N\sum _{i=1}^W\sum _{j=1}^H\left|\left(\mu \left({\mathbf{P}}_{HSI}^{ijn}\right)-\mu \left({\mathbf{P}}_{REC}^{ijn}\right)\right)\right|$$

(10)

Then, we combine these two loss functions.

$${\mathcal{L}}_{\mathrm{Total}}={\mathcal{L}}_{\mathrm{MRAE}}+\eta {\mathcal{L}}_{\mathrm{CSR}}$$

(11)

where $\mu$ denotes CSR function, which is the CIE-1964 color-matching function. $\eta$ is hyper-parameter. The value of $\eta$ is set to 10 empirically.

4. Experiment

4.1. Experimental Setup

Our approach is implemented in the PyTorch framework with a NVIDIA RTX 3080 GPU. The backbone of our network adopts a pre-trained model which is trained on the NTIRE2020 dataset [23]. The whole model is also trained on our training set. In the training phase, we set the mini-batch size to 16 and the parameters of the Adam optimization algorithm as ${\beta }_1=0.9$, ${\beta }_2=0.99$, and $ϵ=1\times {10}^{-8}$ to minimize our loss. During the training, we crop sample pairs of RGB images and HSIs to $64\times 64$. The learning rate is initialized as $1\times {10}^{-4}$ and the polynomial function is set to decay by the power of 1.5. We stop the network training after 100 epochs. The training and validation loss curves are shown in Figure 10. The RMSE is used to evaluate the validation results more intuitively.

4.2. Evaluation Metrics

We adopt qualitative and quantitative evaluation metrics to compare the differences between our reconstructed results and the GT results.

The overall accuracy is evaluated by MRAE and RMSE.

$$RMSE=\sqrt{\frac{1}{WHN}\sum _{n=1}^N\sum _{i=1}^W\sum _{j=1}^H{\left({\mathbf{P}}_{HSI}^{ijn}-{\mathbf{P}}_{REC}^{ijn}\right)}^2}$$

(12)

In order to analyze the similarity of spectral and spatial features between the reconstructed result and GT, Pearson correlation coefficient (CC) [47], and peak signal-to-noise ratio (PSNR), and the absolute error (AE) between the GT and reconstructed result are used to calculate pixel-wise [48].

$$CC=\frac{\left({\mathbf{P}}_{HSI}-\overline{{\mathbf{P}}_{HSI}}\right)\left({\mathbf{P}}_{REC}-\overline{{\mathbf{P}}_{REC}}\right)}{\sqrt{{\left({\mathbf{P}}_{HSI}-\overline{{\mathbf{P}}_{HSI}}\right)}^2}{\left({\mathbf{P}}_{REC}-\overline{{\mathbf{P}}_{REC}}\right)}^2}$$

(13)

$$AE=|{\mathbf{P}}_{HSI}-{\mathbf{P}}_{REC}|$$

(14)

$$PSNR=10\times {log}_{10}\left(\frac{{\left(2^{\mathrm{bit}}-1\right)}^2}{MSE}\right)$$

(15)

$$MSE=\frac{1}{WHN}\sum _{n=1}^N\sum _{i=1}^W\sum _{j=1}^H{\left({\mathbf{P}}_{HSI}^{ijn}-{\mathbf{P}}_{REC}^{ijn}\right)}^2$$

(16)

4.3. Experimental Results

Figure 11 shows the false color reconstruction results of HSCNN+, HRNet [49], our proposed method (with/without considering illumination variation), and the GT results, which are used to demonstrate the reconstruction effect and visual evaluation of image quality. Our method without considering illumination variation uses the AWAN architecture. In terms of visual effects, our reconstructed images are similar to the GT results in color, texture, and shape.

We present an example to analyze the spatial characteristics of the errors. In Figure 12, rows 3, 4, and 5 show the reconstructed images with/without solar illumination variation and the GT images, respectively. It can be seen from the figure that the reconstructed and GT images of the four selected bands are similar. The corresponding AE maps are shown in rows 1 and 2 in Figure 12 and the color distributions of the error graphs present that the reconstruction errors are very small in all four bands. As the value bars of the AE maps show, the reconstructed images without consideration of solar illumination variation have wider error value ranges than the reconstructed results with solar illumination variation.

Table 1. Quantitative evelation of reconstruction results.

Metrics	RMSE	MRAE	CC	PSNR
HSCNN+ w/o illumination	0.2698	0.3665	0.9174	64.5812
HRNet w/o illumination	0.2355	0.3862	0.9191	65.6605
AWAN w/o illumination	0.1307	0.3791	0.9253	67.8038
HSCNN with illumination	0.0783	0.3022	0.9315	72.7564
HRNet with illumination	0.0565	0.2748	0.9482	74.1125
Ours	0.0399	0.2298	0.9496	77.7594

shows the average accuracy of reconstructed HSIs on the test set with different methods with and without considering illumination variation. The MRAE and RMSE values for our reconstructed results with solar illumination variation on our dataset are 0.2298 and 0.0399; the values are lower than those for the reconstructed results without considering solar illumination variation. The RMSE value is reduced by a certain order of magnitude and the MARE is reduced by 0.1493. This demonstrates that the absolute accuracy of the reconstructed HSIs is high and have a better spectral reconstruction effect. The errors between our reconstructed results considering solar illumination variation and the GT results are smaller than those between the reconstructed results without considering solar illumination variation and the GT results. The PSNR and CC values of our reconstructed results considering solar illumination variation are 77.7594 and 0.9496. They are higher than the results without considering solar illumination variation. It can indicate that our reconstructed results considering solar illumination variation are more similar to the GT results. Outdoor solar illumination variation is random and irregular. When the conditions are close to real imaging environments and with real imaging instruments, this random phenomenon affects network learning when the solar illumination variation factor is not considered during reconstruction. Our reconstruction results are better than the listed method (considering or not considering illumination variation) in both qualitative and quantitative aspects. Therefore, we collect illumination data and input them into the model as constraints in the form of vectors, so our model can more accurately map RGB images to HSIs.

Figure 13 shows the spectral curves of roads, shrubs, and grass. On the whole, the curves of the reconstructed HSIs considering illumination variation are more similar to the GT curves than those of the reconstructed HSIs without considering illumination variation for all selected objects from our dataset. By comparing the images of the road in Figure 13b, it can be seen that the curve of the reconstructed image considering illumination variation is closer to that of the GT image and that with the increase in wavelength, the accuracy tends to improve gradually. The overall trend is roughly similar for the reconstructed images without considering illumination variation, but the accuracy is significantly lower when the band is nearer to the blue and near-infrared bands. The atmospheric scattering of sunlight in the blue band is relatively strong. With outdoor illumination variation, the intensity of the scattering and the energy received by ground objects in this band also change. These variations affect the reconstruction accuracy. In our study, when illumination variation is considered, the accuracy curves tend to be stable with the increase in wavelength. This is consistent with the law that atmospheric scattering decreases with increasing wavelength. Figure 13c,d compare the spectral reconstruction results for different plant types. By comparing the reconstructed results in Figure 13c, it can be seen that the reconstructed results for grass in different scenes are more stable when illumination is considered. The results without considering illumination variation are quite different from the GT results in terms of both overall trend and value. The reconstructed results obtained with our model are more stable. In Figure 13d, the reconstructed results of shrubs in different scenes are compared. The error curves of the top and bottom images are consistent, and the accuracy is better than when illumination is not considered. By comparing Figure 13c,d, it can be concluded that the reconstruction accuracy and stability within the plant category are greatly improved by considering illumination variation. The above analysis shows that the proposed network has better spatial and spectral adaptability.

4.4. Image Classification and Accuracy Comparison

To further validate the quality of our HSI reconstruction, classification experiments are conducted on the reconstructed HSIs and the GT images. Qualitative visual effects and quantitative accuracy evaluation metrics are employed to evaluate the spatial and spectral consistency between the reconstructed images and the GT images.

We use the support vector machine (SVM) supervised classifier and the iterative self-organizing data analysis algorithm (IsoData) unsupervised classifier. The SVM and IsoData are conducted in ENVI 5.3. We extract the specific targets, namely road, grass, and shrub, from the GT samples using the SVM classifier. The detailed parameters of the IsoData ENVI settings are shown in

Table 2. The detailed parameters of IsoData ENVI setting.

Paremeter	Value
Maximum number of iteration	5
Change threshold	5.0
Minimum pixels for class	1
Maximum class stdv	1.0
Class distance minimum for class	5.0
Maximum merge pairs	2

.

We adopt the classification GT results as the reference for the unsupervised IsoData classification method and quantitatively calculate the similarity index (SI) between the classification results of our reconstructed HSIs and those of the GT images:

$$SI=Q/K\times 100\%$$

(17)

where Q is the number of pixels in the same classified category of reconstructed HSI and GT, K denotes the total number of pixels in the whole image. For the SVM classifier, we not only calculate SI between the classification results of our reconstructed HSI and GT but also evaluate the classification accuracy with overall accuracy (OA) and the kappa coefficient (Kappa).

The classification results of unsupervised method IsoData are shown in Figure 14. The classification results of most objects are similar with the classification results of GT from the visual effect.

Figure 15 shows the classification results from the SVM. Due to the indistinguishability of shrubs and grass, there are some errors in the unsupervised classification.

Then, we calculate the average SI between the classification results of the reconstructed HSIs and those of the GT images, as shown in Figure 14. The IsoData classification SI values for the reconstructed HSIs considering and without considering illumination variation are 81.80% and 80.26%, respectively. The average SI value of the reconstructed HSIs considering illumination variation using the IsoData classifier is 1.54% higher than that of the reconstructed HSIs without considering illumination variation.

Table 3. Quantitative classification results of SVM (GT, reconstructed HSI with IV (illumination variation), reconstructed HSI without IV).

GT		Reconstructed HSI (with IV)			Reconstructed HSI (w/o IV)
OA (%)	Kappa (%)	OA (%)	Kappa (%)	SI (%)	OA (%)	Kappa (%)	SI (%)
91.92	76.66	91.96	82.69	87.76	89.17	78.82	86.12

shows the average classification accuracy and the SI, OA, and Kappa values from the SVM classifier. Using the SVM classification method, the SI value of the reconstructed HSIs considering and without considering illumination variation are 87.76% and 86.12%, respectively. The similarity, OA, and Kappa values of the reconstructed HSIs considering illumination variation are higher than those of the reconstructed HSIs without considering illumination variation. On the whole, the results demonstrate that reconstructed HSIs can be used for classification and that the reconstructed classification results considering illumination variation are better than those without considering illumination variation.

In addition, in our method, a softmax layer is used to normalize the feature. To make use of the aggregated information by adaptive weighted pooling, we adopt a simple gating mechanism with sigmoid function. ReLU is essentially a function that takes the maximum value. If the input is negative, it will output 0, and the neuron will not be activated. In future research, we will adopt the adaptive activation functions to improve our network performance [49,50].

5. Conclusions

In this paper, due to the lack of consideration of outdoor solar illumination variation in existing reconstruction methods, we collect a UAV hyperspectral dataset (VIHS) that can alleviate the problem. The dataset provides illumination information labels that correspond precisely to GPS times. Furthermore, our dataset can provide a benchmark for future research into outdoor spectral imaging, processing, and analysis. Additionally, we present an attention network consisting of a basic 2D block and an AWAN comprising 12 DRABs and a PSNL module. Prior illumination information is introduced into our network via the basic 2D block to generate hyperspectral images from RGB images more accurately. We compare our method with other reconstruction methods with/without considering illumination variation, evaluate the effectiveness of our reconstructed results with/without considering illumination variation on real data, and verify the effects of the reconstructed results using supervised and unsupervised classification methods. In future work, we will further explore the physical factors that affect hyperspectral reconstruction under outdoor conditions, such as atmosphere and terrain, to further improve our reconstruction accuracy and allow reconstructed hyperspectral images to be used in more applications.