CRNN: Collaborative Representation Neural Networks for Hyperspectral Anomaly Detection

Abstract

Hyperspectral anomaly detection aims to separate anomalies and backgrounds without prior knowledge. The collaborative representation (CR)-based hyperspectral anomaly detection methods have gained significant interest and development because of their interpretability and high detection rate. However, the traditional CR presents a low utilization rate for deep latent features in hyperspectral images, making the dictionary construction and the optimization of weight matrix sub-optimal. Due to the excellent capacity of neural networks for generation, we formulate the deep learning-based method into CR optimization in both global and local streams, and propose a novel hyperspectral anomaly detection method based on collaborative representation neural networks (CRNN) in this paper. In order to gain a complete background dictionary and avoid the pollution of anomalies, the global dictionary is collected in the global stream by optimizing the dictionary atom loss, while the local background dictionary is obtained by using a sliding dual window. Based on the two dictionaries, our two-stream networks are trained to learn the global and local representation of hyperspectral data by optimizing the objective function of CR. The detection result is calculated by the fusion of residual maps of original and represented data in the two streams. In addition, an autoencoder is introduced to obtain the hidden feature considered as the dense expression of the original hyperspectral image, and a feature extraction network is concerned to further learn the comprehensive features. Compared with the shallow learning CR, the proposed CRNN learns the dictionary and the representation weight matrix in neural networks to increase the detection performance, and the fixed network parameters instead of the complex matrix operations in traditional CR bring a high inference efficiency. The experiments on six public hyperspectral datasets prove that our proposed CRNN presents the state-of-the-art performance.

1. Introduction

Hyperspectral remote sensing integrates two sensing technologies of panchromatic imaging and spectrometry to fully exploit the spatial and spectral features [1]. Hyperspectral images record nearly continuous spectral bands and better reflect the physical properties of the objects, which helps to identify subtle differences between various materials. Therefore, they are widely employed in both civil and military fields, such as image classification [2,3,4] and target detection [5,6,7]. As one of the vital research branches of target detection, hyperspectral anomaly detection distinguishes anomalies from background in unsupervised way for lack of prior knowledge [8]. Due to the challenges in establishing a comprehensive and accurate spectrum database for targets, hyperspectral anomaly detection has received a lot of interest and has great practical prospects [9].

In hyperspectral anomaly detection, the anomaly is typically considered as the target with a notably different spectral features from the surrounding background pixels [1,10]. In real hyperspectral images, it is difficult to consider all background clusters since the background occupies most of the pixels and has a complex composition [11]. Moreover, hyperspectral images are highly unbalanced, and the occurrence probability of anomalies is low, which makes it impossible to extract the intrinsic features of anomalies. Therefore, the main challenge in hyperspectral anomaly detection is to effectively suppress the background while distinguishing the anomalies.

The existing anomaly detection methods exploit the property of anomalies with low probability and high deviation from the surrounding background cluster to distinguish anomalies from background [10,12]. The most widely known benchmark algorithm is the Reed–Xiaoli (RX) detector [13]. RX utilizes the whole image to build background model under the presumption that the background is distributed according to a multivariate Gaussian distribution. The probability of anomalies is measured by the Mahalanobis distance between the test pixel and the background. The statistical theory-based hyperspectral anomaly detection methods use a simple mathematical model and have low computational consumption. Subsequently, researchers have proposed many improved algorithms based on RX, such as Kernel RX [14], Local RX [15], Dual-Window RX [16], and Weighted RX [17]. In addition to the generalized likelihood ratio test-based RX, Chang [9] developed dummy variable trick (DVT) to convert hyperspectral target detection to anomaly detection, and also introduced effective anomaly space (EAS) [18] to remove the background while retain the anomaly. Cluster-based hyperspectral anomaly detectors [19,20], the support vector data description (SVDD) [21], and graph-based methods [22,23] take the distance to measure the anomaly score. Ref. [24] introduced a fractional Fourier entropy (FrFE)-based hyperspectral anomaly detection method to distinguish anomalies from background. To utilize the spatial features in the hyperspectral data, an attribute and edge-preserving filtering detector (AED) [25] and other filtering-based methods [26,27] were proposed. Tensor-based methods treat the hyperspectral cube as a third-order tensor and utilize the spectral and spatial information [28]. A prior-based tensor approximation (PTA) method for hyperspectral anomaly detection is proposed in [29], which decomposed the hyperspectral data into a background tensor and an anomaly tensor under a low-rank and a piecewise-smooth prior. Iterative spectral–spatial hyperspectral anomaly detector (ISSHAD) [30] extracted spectral–spatial information via anomaly detection maps and gave feedback to create a new expanded data cube in iteration.

The representation-based methods, including sparse representation (SR) [31,32,33], collaborative representation (CR) [34,35,36,37,38,39,40], and low-rank representation (LRR) [41,42,43,44,45], try to use a linear combination of atoms in the constructed background dictionary to represent the hyperspectral images, and assume that the background pixels will be represented better than anomalous pixels since the clusters of anomalies are far different from the background clusters. Therefore, representation-based methods usually use the residual map of the represented image and the original image to detect the anomaly.

The SR-based approaches assume that only a small subset of atoms in the given over-complete dictionary contribute to the representation under the sparsity constraint. Li et al. [32] adopted principal component analysis (PCA) [46] to learn a robust background dictionary and proposed a SR model based on the re-weighted Laplace prior. The CR-based methods utilize all of the background dictionary atoms to represent each pixel with the $l_2$-norm [47] to constrain the representation coefficients for better background modeling capability [34]. The CR-based methods have received many developments due to their computational simplicity and good performance, but the pollution of anomalous atom caused by the sliding dual window limits the performance of the method. To solve this issue, researchers have devised lots of methods. Vafadar and Ghassemian [35] removed anomalous pixels that deviate noticeably from the distribution of most pixels when constructing the background dictionary. Other methods were also tried in [36,37,38,39] to exclude potential anomalous pixels and obtain a pure background. In contrast to the popular dual window strategy, Wu et al. [40] introduced a practical non-global dictionary and proposed a new relaxed CR hyperspectral anomaly detector. The LRR-based approaches divide the hyperspectral images into background and anomalies with the assumption that the background pixels are low rank because they are widely dispersed and homogeneous, while the anomalies are few and sparse. The LRR theory determines the lowest rank representation of the background based on the background dictionary, and constrains the anomaly with sparsity [41]. Xu et al. [42] combined low-rank and sparse representation (LRASR) and constructed a stable and discriminative dictionary. Zhang et al. [43] implemented low-rank and sparse decomposition (LRaSMD) through the Go Decomposition (GoDec) [48] approach and adopted the RX detector to detect anomalies. Cheng et al. [44] proposed a graph and total variation regularized LRR-based model (GTVLRR) to retain the local geometrical and spatial structure. LRR-based methods are still widely used due to their high detection performance [11,45,49].

With the development of artificial intelligence, researchers have introduced deep learning into hyperspectral anomaly detection [50]. Due to the small proportion of anomalies in hyperspectral data that limits their contribution to model training, and the absence of prior spectral knowledge, most current deep learning-based hyperspectral anomaly detection methods chose unsupervised learning [49,51,52,53,54,55,56,57].

The deep neural networks are introduced to learn the intrinsic expression in the hyperspectral data to take advantage of their capacity on complex data modeling and feature extraction. The traditional method is then performed on the extracted features to detect anomalies. Lei et al. [51] utilized a deep brief network (DBN) [58] to automatically learn discriminative features and high-level representations in the spectral domain and fed them into an RX detector. Based on the generative models of autoencoder (AE) [59] and Generative Adversarial Networks (GAN) [60], the original hyperspectral data are reconstructed and the residuals of the original and the reconstructed data are used for anomaly detection. The AE-based methods have received a lot of attention recently. In [52], manifold learning is introduced to an AE network for hyperspectral anomaly detection. Jiang et al. [49] proposed a low-rank embedded network (LREN) to learn spectral features via an AE, and the LRR-based anomaly detector is used in the deep feature space. Xiang et al. [61] proposed a guided AE (GAE) which introduced a guided image to suppress the feature learning of anomalies and used the reconstruction error as the metric. GAN-based methods introduce adversarial learning [53] into background reconstruction. Jiang et al. [54] proposed GAN for hyperspectral anomaly detection (HADGAN) and reconstructed the hyperspectral data via an adversarial autoencoder (AAE) [62]. The morphological filter and RX were implemented on the residual image. Xie et al. [55] also used the AAE for reconstruction, but chose to use the morphological method on the hidden layer features.

Since the above methods usually detect anomalies in two stages, and deep features are not extracted specially for the hyperspectral anomaly detection tasks, the separate training of the feature extractor and the anomaly detector hinders deep neural networks from attaining their full potential. Therefore, an autonomous hyperspectral anomaly detection network (Auto-AD) is proposed in [56] to avoid the pre-processing and post-processing in other two-stage networks. Xie et al. [57] proposed an end-to-end Spectral Distribution-Aware Estimation network (SDEN) to directly estimate anomalies, which jointly learned the two parts of the feature extraction network and the anomaly detector. Meanwhile, in addition to the above unsupervised methods, weakly supervised methods are gradually becoming an attractive option. Most of the current weakly supervised methods firstly filtered out background pseudo-samples for training based on the estimated background distribution [10,11,63,64].

The traditional hyperspectral anomaly detection methods have a reasonable mathematical foundation and interpretability, but the shallow learning method somehow limits their detection performance. While the deep learning-based methods have a high rate in feature utilization and strong generalization ability, their interpretability is weaker than that of traditional algorithms. Therefore, a new hyperspectral anomaly detection method based on Collaborative Representation Neural Networks (CRNN) is proposed in this paper. Our motivation is to implement traditional representation theory of hyperspectral anomaly detection into the framework of deep neural networks.

To be specific, the fairly complete background dictionaries of global and local streams are constructed, and the representation weight matrices are learned by optimizing the objective function of CR in the two-stream networks. The global background dictionary is a generalization of the global background, which is constructed based on all pixels of the entire image. To avoid the pollution of the anomalous pixels, the global CR networks is trained to obtain the most representative dictionary atoms under the constraint of the dictionary atom loss function. The local background dictionary focuses on the local information surrounding the test pixel, which is obtained by a sliding dual window. The global and local representation images of the test pixel are calculated through the linear combination of atoms in the two dictionaries, respectively. In traditional CR-based methods, the optimal solution of weight coefficients usually involves matrix inversion and other complex operations, which would bring large computational load and time consumption. While, in our proposed method, the two-stream networks learn the weight matrix via iterative optimization during training and the fixed parameters in inference, which leads to a high calculation efficiency. Finally, the anomaly score of each pixel can be measured by the residual input data and its representation through networks based on the jointly considered global and local background dictionary.

In addition, CRNN firstly introduces an AE to reconstruct the original input hyperspectral data to obtain the dimensionality reduction and dense expression of the original image. The global and local CR is then operated on the hidden feature map of the AE. Moreover, we design a feature extraction network to further learn the comprehensive spectral–spatial features in the hidden feature map.

The main contributions of this paper are as follows:

• We propose a new collaborative representation neural network for hyperspectral anomaly detection that combines the mathematical foundation of CR theory with the effective feature learning capacity of deep neural networks. CRNN optimizes the objective function of CR through iterative training, and improves the detection performance with faster inference speed.
• We perform both global and local streams of CR, and fuse the detection maps to obtain the final result comprehensively. The global dictionary is collected in the networks by optimizing the dictionary atom loss to refine the background without the pollution of anomalies. While the local dictionary by a sliding dual window reflects the background neighboring the test pixel.
• We utilize a feature extraction network to generate the comprehensive feature map, including spectral–spatial features from both global and local perspectives. The deep latent feature space helps to learn the representation weigh matrices more effectively.

The remainder of our article is organized as follows. Section 2 reviews some related works. Section 3 introduces the proposed method. Section 4 provides the experiments in detail. Section 5 makes the discussion. Finally, the conclusion is presented in Section 6.

2. Related Works

We briefly review major works on CR-based detector [34] and AE [59] in this section.

2.1. Collaborative Representation-Based Detector

For a given hyperspectral dataset, $\mathit{X}=\left[{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_N\right]$, where ${\mathit{x}}_i\in {\mathbb{R}}^d$ is the i-th pixel in the data, the CR theory believes that the approximate representation $\widehat{\mathit{y}}$ can be calculated via the linear combination of background pixels in the surrounding area of the test pixel $\mathit{y}$,

$$\widehat{\mathit{y}}={\mathit{X}}_s\alpha$$

(1)

where $\widehat{\mathit{y}}$ is the collaborative representation of the test pixel. ${\mathit{X}}_s$ is the collection of background pixels around the test pixel through a dual window. $\alpha \in {\mathbb{R}}^{s\times 1}$ is the weight vector.

The Euclidean distance r between $\mathit{y}$ and $\widehat{\mathit{y}}$ is usually introduced to measure the probability of anomaly

$$r=\parallel \mathit{y}-\widehat{\mathit{y}}{\parallel }_2={∥\mathit{y}-{\mathit{X}}_s\alpha ∥}_2$$

(2)

where ${\parallel ·\parallel }_2$ is the $l_2$-norm.

Collaboration means that the constraint on the weight vector $\alpha$ is that ${∥\mathit{y}-{\mathit{X}}_s\alpha ∥}_2^2$ is the smallest, while ${\parallel \alpha \parallel }_2^2$ is the smallest, so the objective function is

$$\underset{\alpha }{min}{∥\mathit{y}-{\mathit{X}}_s\alpha ∥}_2^2+\lambda {\parallel \alpha \parallel }_2^2$$

(3)

where $\lambda$ is the Lagrange multiplier. $l_2$-norm provides each element in $\alpha$ a tiny value and guarantees that all atoms take part in the representation.

The formula (3) is equal to:

$$\underset{\alpha }{min}\left[{\alpha }^T\left({\mathit{X}}_s^T{\mathit{X}}_s+\lambda I\right)\alpha -2{\alpha }^T{\mathit{X}}_s^T\mathit{y}\right]$$

(4)

The above formula (4) belongs to the convex function optimization problem and exists as a closed-form solution for $\alpha$. Setting the derivative to 0 gains the optimal solution for $\alpha$,

$$\alpha ={\left({\mathit{X}}_s^T{\mathit{X}}_s+\lambda I\right)}^{-1}{\mathit{X}}_s^T\mathit{y}$$

(5)

2.2. Autoencoder

AE is an unsupervised deep learning-based neural network model and consists of the encoder and the decoder. The encoder firstly maps the input vector $\mathit{x}$ from input layer to hidden layer, while the decoder is a structure where the vector $\mathit{z}$ in hidden layer is mapped to the output vector $\widehat{\mathit{x}}$,

$$\begin{array}{c}\hfill \mathit{z}=f(\mathit{x};\theta )\\ \hfill \widehat{\mathit{x}}=\widehat{f}(\mathit{z};\widehat{\theta })\end{array}$$

(6)

where $f(·;\theta )$ and $\widehat{f}(·;\widehat{\theta })$ are the mapping functions of the encoder and the decoder.

In order to accurately reconstruct the input vector $\mathit{x}$, the divergence between $\mathit{x}$ and $\widehat{\mathit{x}}$ is as small as possible during training, so that the hidden $\mathit{z}$ can well reflect the feature of $\mathit{x}$. The loss function $L_R$ is as follows:

$$L_R={\parallel \mathit{x}-\widehat{\mathit{x}}\parallel }_2$$

(7)

3. Proposed Method

In this section, a hyperspectral anomaly detection method named CRNN is proposed. The flow chart is presented in Figure 1. Firstly, an AE reconstructs the original input hyperspectral data $\mathit{X}$. The hidden feature $\mathit{Z}$ during the reconstruction process serves as the basis for subsequent anomaly detection. Secondly, the comprehensive feature extraction network further learns the global and local spectral–spatial feature in $\mathit{Z}$ to obtain comprehensive feature map ${\mathit{Z}}^c$. The CR learning network includes global stream and local stream. The global background dictionary $D^g$ and the local background dictionary $D^l$ are both obtained from $\mathit{Z}$, while the global representation weight ${\alpha }^g$ and local representation weight ${\alpha }^l$ are learned based on ${\mathit{Z}}^c$. Finally, the anomaly score map $\mathit{A}$ is calculated in the detection fusion module.

3.1. Hidden Feature Map Generation

High spectral resolution and continuous spectral bands in hyperspectral data produce the issue of highly correlated and redundant bands. Therefore, the original hyperspectral data are densely expressed before anomaly detection to improve computational efficiency, specifically through the encoding process in an AE [59].

Supposing the original hyperspectral data as $\mathit{X}\in {\mathbb{R}}^{H\times W\times K}$, where K is the number of bands of the original data, and N is the total amount of pixels in the image that $N=H\times W$ (H and W are the height and width, respectively). As shown in Figure 1, the reshaped $\mathit{X}=\left[{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_N\right]\in {\mathbb{R}}^{K\times N}$ is firstly input into an AE, and the output reconstructed image is $\widehat{\mathit{X}}=\left[{\widehat{\mathit{x}}}_1,{\widehat{\mathit{x}}}_2,\cdots ,{\widehat{\mathit{x}}}_N\right]\in {\mathbb{R}}^{K\times N}$. In the encoder E, the original hyperspectral data $\mathit{X}$ are firstly encoded into a lower-dimensional and denser hidden variable $\mathit{Z}=\left[{\mathit{z}}_1,{\mathit{z}}_2,\cdots ,{\mathit{z}}_N\right]\in {\mathbb{R}}^{k\times N}$, where k is the number of bands of the hidden feature map. This hidden variable $\mathit{Z}$ is mapped back to the original hyperspectral space in the decoder De for the reconstruction $\widehat{\mathit{X}}$. Huber loss $L_{\delta }$ [65] is chosen as the loss function,

$$L_{\delta }(\mathit{x},\widehat{\mathit{x}})=\left\{\begin{array}{cc}\frac{1}{2}{(\mathit{x}-\widehat{\mathit{x}})}^2& \mathrm{for}|\mathit{x}-\widehat{\mathit{x}}|\le \delta \\ \delta |\mathit{x}-\widehat{\mathit{x}}|-\frac{1}{2}{\delta }^2& \mathrm{otherwise}\end{array}\right.$$

(8)

where $\delta$ is the hyperparameters.

The structures of E and De are similar as both are composed of $l_a$ numbers of 1 × 1 two-dimensional convolutional layers. The hidden feature map $\mathit{Z}$ is obtained from the output of the $l_a$-th layer in E,

$$\begin{array}{cc}\hfill {\mathit{X}}_{i+1}& =\sigma \left(GN\left(f_i^E\left({\mathit{X}}_i;{\Theta }_i^E\right)\right)\right),i=1,\dots ,l_a-1\hfill \\ \hfill \mathit{Z}& =f_{l_a}^E\left({\mathit{X}}_{l_a};{\Theta }_{l_a}^E\right)\hfill \end{array}$$

(9)

where ${\mathit{X}}_i$ is the input datum of the i-th layer, $f_i^E\left(·;{\Theta }_i^E\right)$ denotes the two-dimensional convolution with parameters ${\Theta }_i^E$ of the i-th layer in encoder E, $GN(·)$ is the group normalization operation [66], and $\sigma$ is a leakly rectified linear unit (Leakly ReLU) activation function [67].

3.2. Comprehensive Feature Extraction

Since the AE only considers the spectral dimension of each pixel during the reconstruction without the global and local structural information, it is necessary to further extract the features of $\mathit{Z}$ to mine its deep potential features.

3.2.1. Non-Local Module

A non-local network [68] is introduced to learn global correlation features. $\mathit{Z}=\left[{\mathit{z}}_1,{\mathit{z}}_2,\cdots ,{\mathit{z}}_N\right]\in {\mathbb{R}}^{k\times N}$ is the input of the module, and ${\mathit{Y}}^g={\left[{\mathit{y}}_1^g,{\mathit{y}}_2^g,\cdots ,{\mathit{y}}_N^g\right]}^T\in {\mathbb{R}}^{N\times k}$ is the global feature calculated by the module,

$${\mathit{y}}_i^g=\frac{1}{{\mathcal{C}}_i}\sum _{\forall j}f\left({\mathit{z}}_i,{\mathit{z}}_j\right)g\left({\mathit{z}}_j\right)$$

(10)

where i and j are the indices of the pixel position; the function $f(·)$ calculates the similarity between the two pixels, and Gaussian function is chosen in our model, $f\left({\mathit{z}}_i,{\mathit{z}}_j\right)=e^{z_i^T{z}_j}$; the function $g(·)$ denotes the feature representation, which is implemented by a 1 × 1 convolutional layer; ${\mathcal{C}}_i$ is the normalization factor, $C_i={\sum }_{\forall j}f\left({\mathit{z}}_i,{\mathit{z}}_j\right)$.

Therefore, the non-local module is equivalent to the self-attention mechanism [69], normalized in the form of softmax [70]. The global correlation feature ${\mathit{Y}}^g$ can be expressed as:

$${\mathit{Y}}^g=softmax\left({\mathit{Z}}^T\mathit{Z}\right)g\left(\mathit{Z}\right)$$

(11)

Finally, the output feature map ${\mathit{Z}}^{nl}\in {\mathbb{R}}^{N\times k}$ of the non-local module with the residual connection is

$${\mathit{Z}}^{nl}=\mathit{W}{\mathit{Y}}^g+{\mathit{Z}}^T$$

(12)

where $\mathit{W}$ denotes the weight matrix learned by the network.

3.2.2. Local Feature Learning Module

The local feature learning module consists of two convolutional layers, a two-dimensional one and a three-dimensional one, as shown in Figure 1. In the 1 × 1 two-dimensional convolution layer, the spectral features for each pixel are learned as $\mathit{V}\in {\mathbb{R}}^{H\times W\times k}$. In the 3 × 3 × 3 three-dimensional convolutional layer, the local spectral–spatial features are extracted on the expanded data ${\mathit{V}}^e\in {\mathbb{R}}^{H\times W\times 1\times k}$ with an additional dimension. The three-dimensional convolution not only fuses the spatial neighboring pixels to learn the local spatial features, like the 3 × 3 two-dimensional convolution, but also extracts the local spectral features via the convolution on the adjacent spectral bands, since some anomalies typically occur in the local spectral bands.

For the reshaped input feature ${\mathit{Z}}^{nl}\in {\mathbb{R}}^{H\times W\times k}$, the comprehensive feature map ${\mathit{Z}}^c\in {\mathbb{R}}^{H\times W\times nk}$ is obtained as followed

$$\begin{array}{cc}\hfill \mathit{V}& =\sigma \left(GN\left(f_{\mathrm{conv}2}\left({\mathit{Z}}^{nl};{\Theta }_{\mathrm{conv}2}\right)\right)\right)\hfill \\ \hfill {\mathit{Z}}^c& =\sigma \left(GN\left(f_{\mathrm{conv}3}\left({\mathit{V}}^e;{\Theta }_{\mathrm{conv}3}\right)\right)\right)\hfill \end{array}$$

(13)

where $f_{\mathrm{conv}2}\left(·;{\Theta }_{\mathrm{conv}2}\right)$ denotes the two-dimensional convolution and parameters ${\Theta }_{\mathrm{conv}2}$ and $f_{\mathrm{conv}3}\left(·;{\Theta }_{\mathrm{conv}3}\right)$ denote the three-dimensional convolution with parameters ${\Theta }_{\mathrm{conv}3}$.

3.3. Two-Stream Collaborative Representation Learning Networks

Based on the hidden layer features $\mathit{Z}\in {\mathbb{R}}^{k\times N}$, we design a two-stream CR learning network for global and local streams.

3.3.1. Global Collaborative Representation Learning

The calculation of the global collaborative representation image ${\widehat{\mathit{Z}}}^g\in {\mathbb{R}}^{k\times N}$ is

$${\widehat{\mathit{Z}}}^g=D^g{\alpha }^g$$

(14)

where $D^g\in {\mathbb{R}}^{k\times C}$ is the global background dictionary; ${\alpha }^g\in {\mathbb{R}}^{C\times N}$ is the global representation weight matrix; and C is the amount of dictionary atoms.

The global representation weight matrix ${\alpha }^g$ is obtained through the global CR learning networks based on the comprehensive feature map ${\mathit{Z}}^c$, by $l_b$ layers of 1 × 1 convolutional layer,

$$\begin{array}{cc}\hfill {\mathit{Z}}_{i+1}^g& =\sigma \left(GN\left(f_i^{GC}\left({\mathit{Z}}_i^g;{\Theta }_i^{GC}\right)\right)\right),i=1,\dots ,l_b-1\hfill \\ \hfill {\alpha }^g& =f_{l_b}^{GC}\left({\mathit{Z}}_{l_b}^g;{\Theta }_{l_b}^{GC}\right)\hfill \end{array}$$

(15)

where ${\mathit{Z}}_i^g$ is the input datum of the i-th layer in the global CR learning networks, and ${\mathit{Z}}_1^g={\mathit{Z}}^c$.

In order to make the error between the global representation ${\widehat{\mathit{Z}}}^g$ of the network and the hidden feature map $\mathit{Z}$ as small as possible, the loss function $L_{Global}$ of the global CR learning network is designed according to the CR objective function in formula (3)

$$L_{Global}=\sum _{i=1}^N{∥{\mathit{z}}_i-{\widehat{\mathit{z}}}_i^g∥}_2^2+\lambda {∥{\alpha }^g∥}_2^2$$

(16)

where ${\mathit{z}}_i$ is the i-th pixel in $\mathit{Z}$, and ${\widehat{\mathit{z}}}_i^g$ is the i-th pixel in ${\widehat{\mathit{Z}}}^g$.

Moreover, we design a loss function $L_{Dic}$ to constrain the dictionary atoms and make them more representative in $D^g$

$$L_{Dic}=\sum _{j=1}^C\sum _{i=1}^N{∥d_j^g-{\mathit{z}}_i∥}_2^2$$

(17)

where $d_j^g$ is the j-th atom in $D^g$. The purpose of this constraint is to minimize the distance from each dictionary atom to the hidden feature $\mathit{Z}$.

3.3.2. Local Collaborative Representation Learning

Different from the global CR learning networks, the local background dictionary is obtained via a sliding dual window on the hidden feature $\mathit{Z}$, with the sizes of inner and outer windows as ${win}_{\mathrm{in}}$ and ${win}_{\mathrm{out}}$, respectively. For the i-th pixel in $\mathit{Z}$, the local collaborative representation ${\widehat{\mathit{Z}}}^l=\left\{{\widehat{\mathit{z}}}_1^l,{\widehat{\mathit{z}}}_2^l,\dots ,{\widehat{\mathit{z}}}_N^l\right\}\in {\mathbb{R}}^{k\times N}$ is calculated as

$${\widehat{\mathit{z}}}_i^l=D_i^l{\alpha }_i^{l}i=1,2,3\cdots N$$

(18)

where ${\widehat{\mathit{z}}}_i^l$ is the i-th pixel of ${\widehat{\mathit{Z}}}^l$; $D_i^l\in {\mathbb{R}}^{k\times S}$ denotes the local background dictionary of the i-th pixel, which is a collection of the feature vectors between the inner and outer windows centered on the i-th pixel in $\mathit{Z}$; ${\alpha }_i^l\in {\mathbb{R}}^{S\times 1}$ is the weight coefficient of the i-th pixel in matrix ${\alpha }^l\in {\mathbb{R}}^{S\times N}$; and S represents the number of background pixels between the inner and outer windows that $S={win}_{\mathrm{out}}\times {win}_{\mathrm{out}}-{win}_{\mathrm{in}}\times {win}_{\mathrm{in}}$.

Since the weight matrix ${\alpha }^l$ depends on the background pixels in the dual window, the convolutional kernel size in local CR learning networks is set to ${win}_{\mathrm{out}}\times {win}_{\mathrm{out}}$. The number of layers is still $l_b$ the same as the global CR learning networks, and the calculation of ${\alpha }_l$ is similar with ${\alpha }_g$ in Equation (15).

The loss function of local collaborative representation ${\widehat{\mathit{Z}}}^l$ is also designed according to the objective function (3):

$$L_{Local}=\sum _{i=1}^N{∥{\mathit{z}}_i-{\widehat{\mathit{z}}}_i^l∥}_2^2+\lambda {∥{\alpha }^l∥}_2^2$$

(19)

Therefore, combining the two streams of global and local, the objective function of CR learning networks is obtained as follows

$$L_{CR}=L_{Dic}+{\lambda }_1{L}_{Global}+{\lambda }_2{L}_{Local}$$

(20)

where ${\lambda }_1$ and ${\lambda }_2$ are the factors of the trade-off between the global and local streams.

Due to the introduction of learning layers in the neural network, the convex optimization problem in Formula (3) is transformed into a non-convex loss function in (20). The deep learning-based algorithm seeks multiple possible local optimal solutions during network iteration, instead of being restricted to a unique closed-form solution. In this way, the generalization performance of the detector would be enhanced and noise interference would be more difficult.

3.4. Detection Fusion

The Euclidean distance between the hidden feature $\mathit{Z}$ and the global representation ${\widehat{\mathit{Z}}}^g$ or the local representation ${\widehat{\mathit{Z}}}^l$ is calculated, respectively, to obtain the global detection result ${\mathit{A}}^g\in {\mathbb{R}}^{H\times W}$ and the local detection result ${\mathit{A}}^l\in {\mathbb{R}}^{H\times W}$

$${\mathit{A}}_i^g={∥{\mathit{z}}_i-{\widehat{\mathit{z}}}_i^g∥}_{2}i=1,2,3\cdots N$$

(21)

$${\mathit{A}}_i^l={∥{\mathit{z}}_i-{\widehat{\mathit{z}}}_i^l∥}_{2}i=1,2,3\cdots N$$

(22)

where ${\mathit{A}}_i^g$ is the i-th pixel in ${\mathit{A}}^g$, and ${\mathit{A}}_i^l$ is the i-th pixel in ${\mathit{A}}^l$.

The final anomaly score map $\mathit{A}\in {\mathbb{R}}^{H\times W}$ will be obtained by element-wise multiplying of ${\mathit{A}}^g$ and ${\mathit{A}}^l$ to achieve a higher detection accuracy:

$$\mathit{A}={\mathit{A}}^g\odot {\mathit{A}}^l$$

(23)

The proposed CRNN method is described in Algorithm 1. CFL-Net stands for the comprehensive feature extraction networks and CRL-Net stands for the two-stream CR learning networks. AE_pre_num is epochs that the AE is trained individually during the pre-training phase to strengthen the stability of network training. AE_pre_num = 10.

Algorithm 1: Algorithm flow diagram of CRNN

Input: original Hyperspectral data $\mathit{X}=\left[{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_{H\times W}\right]\in {\mathbb{R}}^{H\times W\times K}$.   Output: Anomaly score map $\mathit{A}=\left[{\mathit{a}}_1,{\mathit{a}}_2,\cdots ,{\mathit{a}}_{H\times W}\right]\in {\mathbb{R}}^{H\times W}$.

4. Experiments

To validate the detection performance of our proposed CRNN, we perform experiments on six real hyperspectral images and compare with eight previous algorithms.

4.1. Experimental Settings

4.1.1. Datasets Description

Six datasets are employed in the experiment, including Beach [25], Urban [25], XiongAn [71], HYDICE [72], Gulfport [25], and SanDiego [42].

Table 1. Details of the six hyperspectral datasets in the experiment.

Dataset	Sensor	Resolution	Spatial Size	Bands	Anomaly Type	Anomaly Proportion	Anomaly Size
Beach	AVIRIS 1	4.4 m	100 × 100	188	boat	0.61%	11
Urban	AVIRIS 1	17.2 m	100 × 100	204	buildings	0.67%	2–14
HYDICE	HYDICE 2	1 m	80 × 100	162	vehicles	0.24%	1–4
XiongAn	Gaofeng 3	0.5 m	100 × 100	250	vehicle	0.10%	10
Gulfport	AVIRIS 1	3.4 m	100 × 100	188	3 airplanes	0.60%	10–40
SanDiego	AVIRIS 1	3.5 m	100 × 100	188	3 airplanes	0.34%	30–50

¹ Airborne Visible Infrared Imaging Spectrometer. ² Hyperspectral Digital Imagery Collection Experiment. ³ Gaofeng Full Spectrum Multi-modal Imaging Spectrometer.

lists the details of each dataset. The targets in the images of Beach, Urban, XiongAn, and HYDICE are small. While the anomalies in the datasets of Gulfport and SanDiego are relatively large and have specific shape features, which are planar targets.

4.1.2. Evaluation Criteria

To quantitatively evaluate the detection performance, the three-dimensional receiver operating characteristic (3D ROC) curves and the area under the ROC curve (AUC) [73] are introduced. The 3D ROC curves depict the trade-off among the three parameters of the true positive rate $P_d$, the false positive rate $P_f$, and the detection threshold $\tau$. Three types of two-dimensional ROC (2D ROC) curves decomposed from the 3D ROC curve are also applied in the experiment, including $(P_d,P_f)$, $(P_d,\tau )$, and $(P_f,\tau )$. Correspondingly, three types of AUC values are applied, and their values range in [0, 1]. The ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ (AUC of curve $(P_d,P_f)$) value approaching to 1 indicates a perfect effectiveness, the ${\mathrm{AUC}}_{(\mathrm{D},\tau )}$ (AUC of curve $(P_d,\tau )$) value closer to 1 evaluates the better detection probability, and the ${\mathrm{AUC}}_{(\mathrm{F},\tau )}$ (AUC of curve $(P_f,\tau )$) value approaching 0 indicates an outstanding performance on background suppression. In addition, another five AUC values [74] are introduced to further evaluate the detector of joint anomaly detection (JAD), joint background suppression (JBS), anomaly detection with background suppression (ADBS), overall anomaly detection probability (OADP), and signal-to-noise probability ratio (SNPR). They are denoted as ${\mathrm{AUC}}_{\mathrm{JAD}}$, ${\mathrm{AUC}}_{\mathrm{JBS}}$, ${\mathrm{AUC}}_{\mathrm{ADBS}}$, ${\mathrm{AUC}}_{\mathrm{OADP}}$, and ${\mathrm{AUC}}_{\mathrm{SNPR}}$, respectively. The five AUC values are calculated in the following equation, and larger AUC values indicate a better performance. The values of ${\mathrm{AUC}}_{\mathrm{JAD}}$, ${\mathrm{AUC}}_{\mathrm{JBS}}$, and ${\mathrm{AUC}}_{\mathrm{ADBS}}$ range in [0, 2], while ${\mathrm{AUC}}_{\mathrm{OADP}}$ value is between [0, 3].

$$\begin{array}{cc}\hfill {\mathrm{AUC}}_{\mathrm{JAD}}& ={\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}+{\mathrm{AUC}}_{(\mathrm{D},\tau )}\hfill \\ \hfill {\mathrm{AUC}}_{\mathrm{JBS}}& ={\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}+\left(1-{\mathrm{AUC}}_{(\mathrm{F},\tau )}\right)\hfill \\ \hfill {\mathrm{AUC}}_{\mathrm{ADBS}}& ={\mathrm{AUC}}_{(\mathrm{D},\tau )}+\left(1-{\mathrm{AUC}}_{(\mathrm{F},\tau )}\right)\hfill \\ \hfill {\mathrm{AUC}}_{\mathrm{OADP}}& ={\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}+{\mathrm{AUC}}_{(\mathrm{D},\tau )}+\left(1-{\mathrm{AUC}}_{(\mathrm{F},\tau )}\right)\hfill \\ \hfill {\mathrm{AUC}}_{\mathrm{SNPR}}& =\frac{{\mathrm{AUC}}_{(\mathrm{D},\tau )}}{{\mathrm{AUC}}_{(\mathrm{F},\tau )}}\hfill \end{array}$$

(24)

4.1.3. Compared Method

The comparison methods include traditional anomaly detection methods of the benchmark algorithm such as global RX (GRX) [13], and four representation-based methods, such as the original CRD [34], CRD in EAS (EAS_CR) [18], relaxed CRD (RCRD) [40], and the local summation unsupervised nearest regularized subspace anomaly detector (LSUNRS) [38]. We also experimented with recent deep learning-based anomaly detection methods, such as LREN [49], HADGAN [54], and GAE [61]. The comparison algorithms used in this paper are all verified based on the source code released by the original author. In order to assure the equitable results for the comparison experiment, the parameters of the CRD, EAS_CR, RCRD, LSUNRS, LREN, HADGAN, and GAE in the experiment were all optimal.

4.1.4. Implementation Details

Our approach is implemented in Pytorch 1.8, and all the experiments were tested on a workstation with an Nvidia GeForce GTX 1080-Ti GPU and an Intel i9 CPU. According to the network structure described in Section 3, the specific experiment settings are as follows. Both the encoder and the decoder use the structure of a 4-layer convolutional network, i.e., $l_a=4$. The channel dimension of each layer is configured as E{K, 100, 50, 20, k} and De{k, 20, 50, 100, K}, where K is the number of input hyperspectral channels, and k is the feature dimension of $\mathit{Z}$ as $k=10$. The comprehensive feature ${\mathit{Z}}^c\in {\mathbb{R}}^{H\times W\times nk}$ is obtained in the feature extraction networks, where $n=5$. In CR learning networks, both the global and local streams adopt the 3-layer convolutional network, i.e., $l_b=3$. The channel number configuration is GC{$nk$, k, k, C} and LC{$nk$, k, k, S}. $\delta$ in the Huber loss $L_{\delta }$ in Equation (8) is set as 1. ${\lambda }_1$ and ${\lambda }_2$ in the loss function $L_{CR}$ in Equation (20) are set to 0.1 and 0.1. During training, the number of epochs is set as 500 to balance the detection accuracy and the calculation load, and the Adam algorithm is chosen to optimize the parameters of networks. The network learning rate is initially set to ${10}^{-4}$ and an exponential decay would be performed after 100 epochs. The above parameters are the default in the experiment and can be adjusted in other versions of this experiment.

4.2. Detection Performance

The anomaly detection maps of the proposed CRNN and the other eight algorithms of the six datasets are displayed in Figure 2. For a further and more intuitive analysis of the quantitative performance of each algorithm, the 3D ROC curves and the three types of 2D ROC curves, including $(P_d,P_f)$, $(P_d,\tau )$, and $(P_f,\tau )$, are shown in Figure 3. The 2D ROC curves of $(P_d,P_f)$ are set in a logarithmic scale for better visualization. The eight AUC performance comparisons for each dataset are shown in

Table 2. AUC values of the proposed CRNN and compared algorithms on six datasets. (The best of each dataset is in bold).

Dataset	Method	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{F})}$ ↑	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{\tau })}$ ↑	${\mathbf{AUC}}_{(\mathbf{F},\mathbf{\tau })}$ ↓	${\mathbf{AUC}}_{\mathbf{JAD}}$ ↑	${\mathbf{AUC}}_{\mathbf{JBS}}$ ↑	${\mathbf{AUC}}_{\mathbf{ADBS}}$ ↑	${\mathbf{AUC}}_{\mathbf{OADP}}$ ↑	${\mathbf{AUC}}_{\mathbf{SNPR}}$ ↑
Beach	CRNN	1.0000	0.2803	$\mathbf{7.75}\times {\mathbf{10}}^{-\mathbf{9}}$	1.2803	2.0000	1.2803	2.2803	$\mathbf{3.62}\times {\mathbf{10}}^{\mathbf{7}}$
	CRNN_t 1	1.0000	0.7747	0.0014	1.7747	1.9986	1.7733	2.7733	553.6985
	GAE [61]	0.9773	0.3542	0.0176	1.3315	1.9596	1.3366	2.3139	20.0874
	HADGAN [54]	1.0000	0.6598	0.0448	1.6597	1.9552	1.6150	2.6150	14.7397
	LREN [49]	0.8451	0.4071	0.1662	1.2521	1.6788	1.2408	2.0859	2.4489
	LSUNRS [38]	0.9998	0.4414	0.0092	1.4413	1.9907	1.4323	2.4321	48.1575
	RCRD [40]	0.9996	0.4036	0.0114	1.4032	1.9882	1.3922	2.3918	35.3348
	EAS_CR [18]	0.9986	0.5020	0.0149	1.5005	1.9837	1.4871	2.4857	33.7063
	CRD [34]	0.9944	0.5351	0.0469	1.5295	1.9474	1.4882	2.4825	11.4035
	GRX [13]	0.9998	0.5314	0.0259	1.5313	1.9739	1.5055	2.5053	20.4912
Urban	CRNN	0.9973	0.0899	$\mathbf{2.45}\times {\mathbf{10}}^{\mathbf{5}}$	1.0872	1.9973	1.0899	2.0872	$\mathbf{3.66}\times {\mathbf{10}}^{\mathbf{3}}$
	CRNN_t 1	0.9973	0.6896	0.0277	1.6869	1.9696	1.6619	2.6592	24.8633
	GAE [61]	0.9622	0.3695	0.0666	1.3317	1.8956	1.3029	2.2651	5.5496
	HADGAN [54]	0.9584	0.2483	0.0763	1.2067	1.8822	1.1720	2.1304	3.2555
	LREN [49]	0.9399	0.4838	0.1406	1.4237	1.7993	1.3432	2.2831	3.4411
	LSUNRS [38]	0.9963	0.1352	0.0105	1.1315	1.9858	1.1247	2.1210	12.8765
	RCRD [40]	0.9937	0.3162	0.0160	1.3099	1.9777	1.3003	2.2939	19.8251
	EAS_CR [18]	0.9911	0.3082	0.0496	1.2993	1.9414	1.2586	2.2496	6.2098
	CRD [34]	0.9655	0.0848	0.0127	1.0503	1.9528	1.0721	2.0376	6.6565
	GRX [13]	0.9907	0.3143	0.0556	1.3049	1.9351	1.2587	2.2494	5.6570
XiongAn	CRNN	1.0000	0.2718	$\mathbf{7.48}\times {\mathbf{10}}^{-\mathbf{6}}$	1.2718	2.0000	1.2718	2.2718	$\mathbf{3.63}\times {\mathbf{10}}^{\mathbf{4}}$
	CRNN_t 1	1.0000	0.8708	0.2460	1.8708	1.7540	1.6247	2.6247	3.5392
	GAE [61]	0.9999	0.4014	0.0079	1.4013	1.9920	1.3935	2.3934	50.6621
	HADGAN [54]	1.0000	0.4634	0.0718	1.4634	1.9282	1.3917	2.3916	6.4562
	LREN [49]	1.0000	0.7751	0.1641	1.7751	1.8359	1.6109	2.6109	4.7220
	LSUNRS [38]	0.9996	0.4296	0.0499	1.4292	1.9497	1.3797	2.3793	8.6172
	RCRD [40]	0.9999	0.5710	0.0209	1.5709	1.9791	1.5501	2.5501	27.3467
	EAS_CR [18]	0.9881	0.1909	0.0105	1.1790	1.9776	1.1804	2.1685	18.1455
	CRD [34]	0.9783	0.4413	0.1315	1.4196	1.8469	1.3098	2.2881	3.3569
	GRX [13]	0.9765	0.4222	0.1000	1.3986	1.8765	1.3222	2.2987	4.2235
HYDICE	CRNN	0.9999	0.1570	$\mathbf{9.75}\times {\mathbf{10}}^{-\mathbf{7}}$	1.1570	1.9999	1.1570	2.1570	$\mathbf{1.61}\times {\mathbf{10}}^{\mathbf{5}}$
	CRNN_t 1	0.9999	0.8132	0.0210	1.8131	1.9790	1.7923	2.7922	38.8089
	GAE [61]	0.9982	0.3973	0.0099	1.3955	1.9883	1.3874	2.3856	40.1662
	HADGAN [54]	0.9903	0.5024	0.0571	1.4927	1.9332	1.4453	2.4356	8.7944
	LREN [49]	0.9784	0.4271	0.0093	1.4055	1.9691	1.4179	2.3963	46.1746
	LSUNRS [38]	0.9929	0.3449	0.0166	1.3378	1.9763	1.3283	2.3212	20.7202
	RCRD [40]	0.9962	0.3380	0.0059	1.3342	1.9903	1.3322	2.3284	57.7771
	EAS_CR [18]	0.9233	0.0554	0.0037	0.9787	1.9196	1.0517	1.9751	15.0686
	CRD [34]	0.9868	0.4117	0.0351	1.3985	1.9517	1.3765	2.3634	11.7149
	GRX [13]	0.9763	0.2184	0.0380	1.1947	1.9382	1.1803	2.1566	5.7406
Gulfport	CRNN	0.9955	0.0843	$\mathbf{3.32}\times {\mathbf{10}}^{\mathbf{9}}$	1.0799	1.9955	1.0843	2.0799	$\mathbf{2.54}\times {\mathbf{10}}^{\mathbf{7}}$
	CRNN_t 1	0.9955	0.4825	0.0105	1.4780	1.9851	1.4720	2.4676	46.1582
	GAE [61]	0.9690	0.2675	0.0314	1.2364	1.9376	1.2361	2.2051	8.5239
	HADGAN [54]	0.9602	0.3380	0.1496	1.2982	1.8106	1.1884	2.1486	2.2588
	LREN [49]	0.7521	0.3104	0.1739	1.0626	1.5782	1.1365	1.8887	1.7851
	LSUNRS [38]	0.9589	0.0455	0.0139	1.0044	1.9450	1.0316	1.9905	3.2769
	RCRD [40]	0.9899	0.2552	0.0160	1.2451	1.9739	1.2392	2.2291	15.9448
	EAS_CR [18]	0.5714	0.0059	0.0015	0.5773	1.5698	1.0044	1.5758	3.8782
	CRD [34]	0.7762	0.0112	0.0069	0.7874	1.7693	1.0043	1.7805	1.6246
	GRX [13]	0.9526	0.0736	0.0248	1.0262	1.9278	1.0489	2.0015	2.9743
SanDiego	CRNN	0.9965	0.0368	$\mathbf{1.18}\times {\mathbf{10}}^{-\mathbf{5}}$	1.0333	1.9965	1.0368	2.0333	$\mathbf{3.11}\times {\mathbf{10}}^{\mathbf{3}}$
	CRNN_t 1	0.9965	0.5481	0.0178	1.5446	1.9787	1.5303	2.5268	30.7612
	GAE [61]	0.9910	0.2380	0.0079	1.2290	1.9831	1.2301	2.2211	30.1127
	HADGAN [54]	0.9644	0.2898	0.0653	1.2542	1.8991	1.2244	2.1888	4.4353
	LREN [49]	0.4387	0.1041	0.0346	0.5428	1.4041	1.0695	1.5082	3.0098
	LSUNRS [38]	0.9796	0.2756	0.0789	1.2552	1.9007	1.1967	2.1763	3.4914
	RCRD [40]	0.9825	0.1575	0.0084	1.1399	1.9740	1.1490	2.1315	18.6983
	EAS_CR [18]	0.7515	0.0464	0.0149	0.7979	1.7366	1.0316	1.7831	3.1202
	CRD [34]	0.9071	0.1993	0.1272	1.1064	1.7799	1.0721	1.9792	1.5671
	GRX [13]	0.9403	0.1778	0.0589	1.1181	1.8814	1.1189	2.0592	3.0176

¹ A logarithmic transformation was performed on the result maps of CRNN, and this method is only used as a supplementary illustration and does not participate in the comparison of other methods.

, and the average values of the AUC metrics over six datasets are listed in

Table 3. Average AUC values of the proposed CRNN and compared algorithms over six datasets. (The best is in bold).

	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{F})}$ ↑	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{\tau })}$ ↑	${\mathbf{AUC}}_{(\mathbf{F},\mathbf{\tau })}$ ↓	${\mathbf{AUC}}_{\mathbf{JAD}}$ ↑	${\mathbf{AUC}}_{\mathbf{JBS}}$ ↑	${\mathbf{AUC}}_{\mathbf{ADBS}}$ ↑	${\mathbf{AUC}}_{\mathbf{OADP}}$ ↑	${\mathbf{AUC}}_{\mathbf{SNPR}}$ ↑
CRNN	0.9982	0.1534	$\mathbf{7.47}\times {\mathbf{10}}^{-\mathbf{6}}$	1.1516	1.9982	1.1534	2.1516	$\mathbf{1.03}\times {\mathbf{10}}^{\mathbf{7}}$
CRNN_t 1	0.9982	0.6965	0.0541	1.6947	1.9441	1.6424	2.6406	116.3049
GAE [61]	0.9829	0.3380	0.0236	1.3209	1.9594	1.3144	2.2974	25.8503
HADGAN [54]	0.9789	0.4169	0.0775	1.3958	1.9014	1.3395	2.3183	6.6567
LREN [49]	0.8257	0.4179	0.1148	1.2436	1.7109	1.3032	2.1289	10.2636
LSUNRS [38]	0.9879	0.2787	0.0298	1.2666	1.9580	1.2489	2.2367	16.1900
RCRD [40]	0.9936	0.3403	0.0131	1.3339	1.9805	1.3272	2.3208	29.1545
EAS_CR [18]	0.8706	0.1848	0.0159	1.0555	1.8548	1.1690	2.0396	13.3548
CRD [34]	0.9347	0.2806	0.0601	1.2153	1.8747	1.2205	2.1552	6.0539
GRX [13]	0.9727	0.2896	0.0505	1.2623	1.9222	1.2391	2.2118	7.0174

¹ A logarithmic transformation was performed on the result maps of CRNN, and this method is only used as a supplementary illustration and does not participate in the comparison of other methods.

.

4.2.1. Detection Performance for Small Targets

The detection maps of the Beach dataset are shown in Figure 2a; the target in the detection map of CRNN is prominent, and the suppression of the background is the best since CRNN can effectively suppress the false alarms in the background of this dataset. In the detection maps of other methods, the beach at the bottom left is a more obvious false alarm, especially in LREN. Figure 2b shows the detection maps of the Urban dataset and the result is that CRNN presents outstanding targets and better background suppression. The results of GAE, HADGAN, LREN, RCRD, EAS_CR, and GRX present obvious background texture and banding noise; in particular, the result of LREN presents obvious false detection. The background and targets are both at a low brightness level in the result maps of CRD and LSUNRS. However, the ROC curves $(P_d,P_f)$ of CRNN and LSUNRS are quite close when $P_d$ is reaching 1. It may be due to the disparity in the brightness of different targets in the detection result of CRNN, which means only some anomalous pixels are visually obvious. The detection maps of the XiongAn dataset are shown in Figure 2c; the background suppression is not so effective in the results of the eight compared methods, especially in LREN, and the detection maps of HADGAN, LSUNRS, CRD, and GRX present evident banding noise. Compared with CRD, the two CR-based methods of RCRD and EAS_CR improve the background suppression, but still present some false alarms in detection maps. While the result of CRNN not only presents a perfect suppression on the background but also has a strong anti-interference ability against noise. The detection maps of the HYDICE image are shown in Figure 2d. The detection results of CRNN and LREN are best when dealing with prominent targets and good background suppression, while LREN has some instances of missed detection and the detection score of some anomalous pixels is low. The background suppression in other methods is poor; in addition, the background texture is clear in HADGAN, CRD, and GRX, while some points of false alarm present higher brightness in RCRD and EAS_CR.

For small anomalous targets, the proposed CRNN produces visual results closest to the reference compared with other seven state-of-the-art methods. In Figure 3, the ROC curves of $(P_d,P_f)$ are all the first to reach 1, and the ROC curves of $(P_f,\tau )$ are all the closest to the lower left corner. It is clear from the study above that the proposed CRNN can clearly highlight the small anomalous targets in the detection maps while effectively suppressing the background, since it utilizes neural networks to learn features that are more conducive to anomaly detection.

4.2.2. Detection Performance for Planar Targets

As the detection maps of the Gulfport dataset show in Figure 2e, it can be seen that only the larger aircraft is significant in the detection results of CRNN, and the other two smaller ones are difficult to observe directly but still present higher values than the background. However, in the result maps of traditional detectors of GRX, CRD, EAS_CR, and LSUNRS, the saliency of targets is poor, and obvious false alarm points occur. However, in GAE, HADGAN, and LREN, the networks only learn the texture features of the hyperspectral image, causing a poor detection result. Targets in the detection maps of RCRD are obvious, and the ROC curves $(P_d,P_f)$ of CRNN and RCRD are close, but CRNN is better at background suppression. The detection results of the SanDiego dataset are shown in Figure 2f. The detection map of CRNN presents clear targets and perfect background suppression, but the anomaly scores vary greatly between the three airplanes. In the detection results of GRX, CRD, EAS_CR, LSUNRS, and HADGAN, the targets are lost in the background and obvious noise, causing some false alarms. Although some pixels of the three anomalies have higher brightness in the result of GAE, LREN, and RCRD, the background suppression is not so good that the roof on the lower right turns to be a false alarm target.

For planar targets on the Gulfport and SanDiego datasets, the ROC curves of $(P_d,P_f)$, and $(P_f,\tau )$ are all the closest to the upper left corner and the lower left corner, respectively. It can be concluded that CRNN also presents a perfect detection accuracy with the lowest false alarm rate for planar targets such as airplanes. Although the target texture is not obvious in the detection maps of CRNN, the outline of the aircraft can be clearly seen after performing the logarithmic transformation on the detection maps as show in Figure 4, which proves that the anomalies and the background are still separable. The logarithmic transformation is as follows

$$R={log}_{v+1}(1+v·I)$$

(25)

where $(v+1)$ is the base of the logarithm. I and R denote the input and output images, respectively, and the values are all in the range of [0, 1].

4.2.3. AUC Performance

According to the comparison of the average AUC values in Table 3, CRNN has the highest ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ value of 0.9982, which is 6.79% higher than that of the traditional CRD, indicating that CRNN has perfect detection efficiency and finds the balance between improving the detection rate and reducing the false alarm rate. At the same time, CRNN also shows the smallest ${\mathrm{AUC}}_{(\mathrm{F},\tau )}$ score, and the largest ${\mathrm{AUC}}_{\mathrm{JBS}}$ and ${\mathrm{AUC}}_{\mathrm{SNPR}}$, which are $7.47\times {10}^{-6}$, 1.9982, $1.03\times {10}^7$, respectively. It indicates that CRNN has a strong ability to suppress the background. However, the performance of CRNN in ${\mathrm{AUC}}_{(\mathrm{D},\tau )}$, ${\mathrm{AUC}}_{\mathrm{JAD}}$, ${\mathrm{AUC}}_{\mathrm{ADBS}}$, and ${\mathrm{AUC}}_{\mathrm{OADP}}$ needs to be improved, and the values of the latter three are mainly related to the former. This may be because the high efficiency of background suppression in CRNN also has an impact on some anomalous pixels, and the detection rate needs to rely on a lower threshold, causing the poor performance on $(P_d,\tau )$ curves and ${\mathrm{AUC}}_{(\mathrm{D},\tau )}$. However, it does not mean that the background and anomalies are inseparable. The average ${\mathrm{AUC}}_{\mathrm{SNPR}}$ of the detection maps reaches seven orders of magnitude, and the anomaly scores of background pixels is concentrated around ${10}^{-11}$ to ${10}^{-7}$. The order of magnitude of the potential anomalous pixels spans a large range that the anomaly scores of some pixels are quite close to 1, while other pixels may reach about ${10}^{-4}$. As a supplementary illustration, since the orders of magnitude of the background and partial anomalies are small, we tried logarithmic transformation on the result maps of CRNN as the image enhancement strategy to re-evaluate the AUC performance. The logarithmic transformation is in shown in Equation (25) and v is set to ${10}^2$ in all datasets. This version is not included in the comparison of other methods in Table 3 because it is only used as the post-processing method. According to the results of AUC, logarithmic transformation searches a balance between ${\mathrm{AUC}}_{(\mathrm{D},\tau )}$ and ${\mathrm{AUC}}_{(\mathrm{F},\tau )}$, and improves the target detectability.

In addition, we repeated 10 experiments for each dataset to measure the uncertainty of network training. The mean and variance of the three AUC values of ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$, ${\mathrm{AUC}}_{(\mathrm{D},\tau )}$, and ${\mathrm{AUC}}_{(\mathrm{F},\tau )}$ are shown in

Table 4. ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ ${\mathrm{AUC}}_{(\mathrm{D},\tau )}$ and ${\mathrm{AUC}}_{(\mathrm{F},\tau )}$ values of CRNN in repeated experiments for each dataset.

	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{F})}$		${\mathbf{AUC}}_{(\mathbf{D},\mathbf{\tau })}$		${\mathbf{AUC}}_{(\mathbf{F},\mathbf{\tau })}$
	Mean	Variance(%)	Mean	Variance(%)	Mean	Variance(%)
Beach	0.99950	0.04849	0.22151	4.75515	$4.25\times {10}^{-5}$	0.00432
Urban	0.99642	0.10841	0.07129	1.24275	$1.51\times {10}^{-5}$	0.00110
XiongAn	0.99994	0.00611	0.2561	4.02692	$1.90\times {10}^{-5}$	0.00242
HYDICE	0.99988	0.00953	0.18799	3.54861	$4.80\times {10}^{-6}$	0.00070
Gulfport	0.98564	1.36994	0.08248	2.59059	$2.52\times {10}^{-4}$	0.02073
SanDiego	0.98954	0.43000	0.02573	1.11803	$2.70\times {10}^{-4}$	0.01517

, which verifies the credibility of the experimental results for our proposed model.

4.3. Computing Time

In addition to the accuracy of anomaly detection, computing time is also an important part of algorithm efficiency. The computing time of the proposed CRNN and the compared eight algorithms is listed

Table 5. Computing time in second of the proposed CRNN and compared algorithms. (The content in brackets in columns 2–4 represents the inference time of the network at test).

	CRNN	GAE [61]	HADGAN [54]	LREN [49]	LSUNRS [38]	RCRD [40]	EAS_CR [18]	CRD [34]	GRX [13]
Beach	49.8535 (0.0279)	47.7064 (6.4304)	674.0821 (1.8471)	100.2248	18.7829	38.2593	2.5918	1.5395	0.0854
Urban	60.6503 (0.0259)	48.6605 (6.4776)	677.5559 (2.0005)	109.0844	19.3547	40.2936	3.1943	1.6119	0.0896
XiongAn	21.0506 (0.0558)	56.7479 (7.9990)	673.0219 (2.6153)	102.5597	30.9906	51.7006	3.3039	8.8312	0.1826
HYDICE	19.2016 (0.0189)	26.0721 (5.2584)	693.5843 (1.6805)	73.0596	14.5599	25.5156	1.6684	1.2025	0.0669
Gulfport	70.5033 (0.0488)	47.8685 (6.4286)	676.432 (1.8710)	100.8034	18.8575	39.4005	2.6521	1.5377	0.0807
SanDiego	31.5995 (0.0229)	48.6626 (6.7521)	676.4849 (1.8861)	101.3657	19.0317	36.6709	2.1232	1.5573	0.0840

. For deep learning-based methods, the training epochs of CRNN, GAE, and HADGAN are all set to 500. We counted their training time on GPU and inference time tested on CPU. While LREN has no module for testing, so we only recorded the entire computing time including network training on GPU. The training epochs of LREN are 1000 according to [49]. For non-deep learning methods, LSUNRS, RCRD, EAS_CR, CRD, and RX, were implemented on MATLAB R2019b.

The comparison of computing times in Table 5 are just for reference due to the differences in the implementation of these methods, but it still shows the high efficiency of our proposed CRNN. Compared with the three deep learning-based method, the inference time of CRNN is the shortest, mainly because of the simple linear computation of matrices in a few network layers. GAE calculates the guided images through a spectral similarity method that takes most of the computation time. HADGAN directly used the traditional detector on the residual image and still retains the computational cost of traditional algorithms. LREN needs to consume part of computing time in solving the optimization of LRR.

5. Discussion

In this section, discussions of the ablation study and parameter analysis are made.

5.1. Ablation Study

To verify the effectiveness of the two-stream CR learning networks, including the fusion strategy and the joint training strategy in the proposed CRNN, we set a series of ablation experiments for research. In this experiment, the following scenarios were compared.

• CRNN: the proposed CRNN model, whose detection result is $\mathit{A}={\mathit{A}}^g\odot {\mathit{A}}^l$;
• CRNN+: the CRNN model in additive fusion version, whose detection result is ${\mathit{A}}^{+}={\mathit{A}}^g+{\mathit{A}}^l$;
• LCRNN: the local stream of CR networks in CRNN, whose detection result is ${\mathit{A}}^l$;
• GCRNN: the global stream of CR networks in CRNN, whose detection result is ${\mathit{A}}^g$;
• CRNN_se: the local stream and global stream are separately trained and fused to obtain the detection result. It is a separate training version for CRNN;
• LCRNN_se: the local stream of CR networks in CRNN_se;
• GCRNN_se: the global stream of CR networks in CRNN_se.

Table 6. Average AUC values over six datasets for the ablation study. (The best is in bold).

	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{F})}$ ↑	${\mathbf{AUC}}_{(\mathbf{D},\mathbf{\tau })}$ ↑	${\mathbf{AUC}}_{(\mathbf{F},\mathbf{\tau })}$ ↓	${\mathbf{AUC}}_{\mathbf{JAD}}$ ↑	${\mathbf{AUC}}_{\mathbf{JBS}}$ ↑	${\mathbf{AUC}}_{\mathbf{ADBS}}$ ↑	${\mathbf{AUC}}_{\mathbf{OADP}}$ ↑	${\mathbf{AUC}}_{\mathbf{SNPR}}$ ↑
CRNN	0.9982	0.1534	$\mathbf{7.47}\times {\mathbf{10}}^{-\mathbf{6}}$	1.1516	1.9982	1.1534	2.1516	$\mathbf{1.03}\times {\mathbf{10}}^{\mathbf{7}}$
CRNN+	0.9976	0.2880	$1.29\times {10}^{-3}$	1.2856	1.9963	1.2867	2.2843	$1.01\times {10}^4$
LCRNN	0.9973	0.2503	$1.61\times {10}^{-3}$	1.2475	1.9956	1.2487	2.2459	$1.04\times {10}^4$
GCRNN	0.9968	0.1921	$2.51\times {10}^{-4}$	1.1889	1.9965	1.1919	2.1887	$1.03\times {10}^4$
CRNN_se	0.9978	0.1400	$3.77\times {10}^{-5}$	1.1377	1.9977	1.1399	2.1377	$1.24\times {10}^6$
LCRNN_se	0.9942	0.2770	$3.61\times {10}^{-3}$	1.2712	1.9906	1.2734	2.2676	$1.25\times {10}^2$
GCRNN_se	0.9972	0.1912	$6.45\times {10}^{-4}$	1.1884	1.9966	1.1906	2.1878	$2.13\times {10}^4$

lists the average of the eight AUC values for the ablation study. In addition, the detailed eight AUC values on each dataset are presented in the Appendix A. It is suggested that either the local CR networks or the global CR networks have outstanding detection performance. Their average ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ values are all above 0.9940, which are obviously better than the other eight comparison algorithms. Compared with other six scenarios of CRNN+, LCRNN, GCRNN, CRNN_se, LCRNN_se, and GCRNN_se, the average ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ scores of CRNN are, respectively, improved by 0.06%, 0.08%, 0.13%, 0.03%, 0.40%, and 0.10%.

(1)

Global CR and Local CR

The anomaly detection maps for six datasets of LCRNN and GCRNN are shown in Figure 5. Compared with the global CR networks, the detection results of the local CR networks present more false alarm targets and worse suppression on the background. This may be, mainly, because the local CR networks are composed of the local background dictionary, which is more likely to highlight specific areas in the background, such as the white area in HYDICE, the runway in Gulfport, and the roof in SanDiego. In addition, the anomaly scores of different targets in the results of local CR networks are relatively close, while the anomaly scores of different targets vary greatly in the results of global CR networks.

(2)

Detection Fusion of Global CR and Local CR

From the quantitative perspective, CRNN obtained the best ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ values when compared to LCRNN and GCRNN. The results of separate training versions also presents the similar findings that the average ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ values of CRNN_se have, respectively, improved by 0.36% and 0.06% compared with LCRNN_se and GCRNN_se. This demonstrates that the detection results fused with the local and global ones is better than that of the two alone. Similar to the random forest theory [75], it is possible to create a stronger and better detector by comprehensively analyzing the results of several detectors with good detection performance. In addition, the multiplicative fusion strategy used in proposed CRNN is better than additive fusion in CRNN+. As shown in Figure 5, the background suppression in the result maps of LCRNN is not as good as that of GCRNN, so multiplicative fusion can offer the advantages of global result maps and better background suppression compared with additive fusion.

(3)

Joint Training and Separate Training

According to the average ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ value, GCRNN is 0.04% smaller compared with GCRNNse, while the score of LCRNN is 0.3% higher than LCRNNse. For the final fusion result, the ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ score of CRNN is improved by 0.03% compared with the separate training version. Therefore, the joint training of the local and global streams of the networks is better than separate training in general. In our opinion, this may be because of the comprehensive usage of the global and local streams with a better balance in joint training. It optimizes each part in the networks through the objective function of $L_{CR}$, and extracts features more conducive to detection performance.

5.2. Parameter Analysis

Under the hyperparameters configured in Section 4.1.4, the performance of CRNN is mainly related to two key parameters, C as the amount of the global background dictionary atoms and S as the amount of the local background dictionary atoms, where S is mainly determined by the scale of the dual windows.

Since the size of C affects the ability of the global dictionary to generalize the background of the entire image to a certain extent, it is vital to set C at the appropriate value to ensure the comprehensiveness of the background and avoid anomalous pollution. The parameter C is analyzed in the range of {5, 10, 15, 20}. In Figure 6, it is shown by multiple experiments that the anomaly detection performance reaches the optimum when C is 15. In addition, the analysis of parameter C on the eight types of AUC values for the six datasets are listed in the Appendix B.

The detection performance of the local CR networks is mainly affected by the scale of inner and outer windows, which is generally correlated with the targets size. The larger the size of the windows presents a higher computational cost and a slower inference speed for the networks. In the experiments, the range of the inner window size is set in [3, 9], and the range of the outer window size is in [2 + $wi{n}_{in}$, 8 + $wi{n}_{in}$] relatively. As shown in Figure 7, for the Beach, Urban, XiongAn and HYDICE datasets that only contain small targets, the detection results are not sensitive to the scale of the inner and outer windows, and the ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ values are all above 0.99. For the Gulfport and SanDiego datasets with planar targets, the ${\mathrm{AUC}}_{(\mathrm{D},\mathrm{F})}$ values remain at a suboptimal level when the scale of inner windows is much smaller than the size of anomalies, and they are significantly reduced when the inner window size is excessively wider than the anomaly. However, the selection of the optimal window size still involves a certain dynamic range. According to the experimental results, it is suggested that the range of the inner window is set in [5, 7], and the range of outer window is [9, 11] for the planar target.

6. Conclusions

In this paper, we proposed a deep learning-based hyperspectral anomaly detection method named CRNN, which implements CR theory into neural networks. CRNN mines the deep features that are helpful for the background construction and representation weight optimization in CR, which enhances the detection performance and efficiency. Firstly, an autoencoder is introduced to reconstruct the original hyperspectral data and obtain a dense expression. Based on the hidden feature map in the AE, the feature extraction network is used to further extract its comprehensive feature. Then, the two-stream global and local CR learning networks are ideally designed to learn the representation data of the hidden features under the objective function of CR. Finally, the anomaly detection result is contributed by fusing the residual maps of original and representation data in the two streams. Compared with state-of-the-art methods, experiments on six real hyperspectral images show that CRNN presents a perfect balance on detection accuracy and false alarm rate with a faster inference speed. In the future, we may try a more reasonable structure for CR learning networks instead of only applying convolutional networks to learn features, and also study a better fusion strategy to balance the global and local detection results.