Semantic-Guided Attention Refinement Network for Salient Object Detection in Optical Remote Sensing Images

Abstract

Although remarkable progress has been made in salient object detection (SOD) in natural scene images (NSI), the SOD of optical remote sensing images (RSI) still faces significant challenges due to various spatial resolutions, cluttered backgrounds, and complex imaging conditions, mainly for two reasons: (1) accurate location of salient objects; and (2) subtle boundaries of salient objects. This paper explores the inherent properties of multi-level features to develop a novel semantic-guided attention refinement network (SARNet) for SOD of NSI. Specifically, the proposed semantic guided decoder (SGD) roughly but accurately locates the multi-scale object by aggregating multiple high-level features, and then this global semantic information guides the integration of subsequent features in a step-by-step feedback manner to make full use of deep multi-level features. Simultaneously, the proposed parallel attention fusion (PAF) module combines cross-level features and semantic-guided information to refine the object’s boundary and highlight the entire object area gradually. Finally, the proposed network architecture is trained through an end-to-end fully supervised model. Quantitative and qualitative evaluations on two public RSI datasets and additional NSI datasets across five metrics show that our SARNet is superior to 14 state-of-the-art (SOTA) methods without any post-processing.

1. Introduction

In recent years, with the continuous improvement of aerial remote sensing and sensor technology, it becomes more and more convenient to obtain very high resolution (VHR) optical remote sensing images (RSI), which, to a certain extent, meets the urgent needs of scene analysis and object detection in airborne earth observation tasks. Naturally, various applications of RSI in the military and civilian fields have received a high degree of attention from all walks of life, such as scene monitoring [1], ship detection [2], oil tank detection [3], and military object discovery [4]. However, how to effectively improve the efficiency and accuracy of scene analysis and rapid object detection of massive optical remote sensing data with cluttered backgrounds is crucial for further exploration and application of RSI.

The goal of object-level salient object detection (SOD) is to locate and separate the most attractive regions from the scene, which is a simulated representation of visual attention mechanism [5]. Unlike visual fixation prediction, SOD focuses on segmenting images to generate pixel-wise saliency maps [6]. Because of its low computational cost and excellent scalability, SOD has aroused interest in many fields, including image retrieval [7], object tracking [8], semantic segmentation [9], medical image segmentation [10], camouflage object detection [11], etc. In general, in the large-scale optical RSI with cluttered background and intricate noise, only a small number of regions with great color, shape, or texture differences can attract people’s attention. Therefore, the SOD for RSI aims to segment these regions or objects of interest. As a fast and beneficial tool for massive information processing, the SOD method has been widely applied to various visual tasks of RSI analysis, such as human-made object detection [3,12], change detection [13,14] and ROI extraction [15,16]. Unlike NSI photographed on the ground, optical RSI is usually directly captured by satellites, aircraft, or drones equipped with sensors, so the difference in data acquisition methods makes it a big challenge for SOD from NSI to RSI: (1) RSI coverage is broader, which leads to large changes in the spatial resolution and number of salient objects in the RSI (or scenes without salient objects, such as the ocean, snow, and forest). (2) The shooting angle of the overlooking makes the salient object in RSI have a considerable difference in appearance compared with NSI, and the object also has various directions. (3) Affected by different imaging conditions, RSI usually contains interference information such as shadows, clouds, and fog, making the object’s background area more cluttered and complicated.

To alleviate the above situation, in the previous SOD methods for RSI [17,18], a bottom-up dense link method is usually used to integrate multi-level depth features to locate the object area and filter the background noise. However, this non-discrimination treatment of different features may introduce local noise so that the object area’s edge details can not be restored. For example, the saliency prediction maps of LVNet [18] and MINet [19] in the first and second rows of Figure 1 lose the edge information of the target (cars and buildings). Besides, with the continuous downsampling of the input features by the backbone network in feature extraction, the depth feature patterns of different levels will change, ignoring the relationship between different attention maps and only splices from multi-level feature maps is considered suboptimal.

Further, compared with the NSI scene, the salient areas in RSI have more scale changes, similar ambiguous appearance, and tedious topology information [20]. As shown in Figure 1, in the prediction results of various SOD methods, most SOD methods will have encounter unsatisfactory conditions, such as missed detection, error detection, and overall inconsistency of the object. On the one hand, the result is that the saliency feature of the local area (or attention activation patterns) suppresses the natural global saliency feature. On the other hand, it is due to the difference of representation features in the same salient area due to spatial distribution. Previous work has shown that the convolution operation components that make up the network inevitably cause a local receptive field [23]. In response to this limitation, feature pyramid [24,25], intermediate feature integration [22,26], and atrous convolution [27] are the mainstream strategies. However, these methods usually do not consider the semantic features of long-distance, which may lead to salient objects’ incompleteness.

Inspired by the above challenges, we propose a semantic-guided attention refinement network (SARNet) for SOD of optical RSI. The motivation of the method comes from the empirical fact that for the object search of large scene RSI, we usually scan the whole image and locate the ROI roughly and quickly through the visual attention mechanism, and then accurately infer and identify the boundary details according to the location guidance and combined with the local information around the region. Therefore, we regard the object’s accurate positioning and the boundary details as the two keys of SOD in the RSI scene. The proposed method first uses the semantic guidance decoder (SGD) to integrate multiple high-level side-out features to locate the object and guide low-level information refinement. The parallel attention fusion (PAF) module combines cross-level and global semantic guidance features to refine the object boundary gradually. Overall, our main contributions are summarized as follows:

(1)

We design a novel semantic-guided attention refinement network (SARNet) for SOD in optical RSI. The network has better robustness and generalization through the high-level semantic information guidance and top-down boundary refinement strategy to improve scale-varying objects’ saliency detection performance;

(2)

The proposed semantic guided decoding (SGD) module combines several high-level feature representations to improve the semantic feature differences in long-distance space. Simultaneously, the accurate salient area location information is used to guide the subsequent multi-level feature fusion;

(3)

The proposed parallel perception fusion (PAF) module models global semantic information and cross-level features to fill the differences between different visual representation levels and gradually restore salient objects’ edge details;

(4)

We compare the proposed methods with 14 SOTA approaches on two challenging optical RSI datasets and additional NSI datasets. Without bells and whistles, our method achieves the best performance under five evaluation metrics. Besides, the model has a real-time inference speed of 47.3 FPS on a single GPU. The code will be available at https://github.com/laoyezi/SARNet (accessed on 29 May 2021).

The rest of this article is organized as follows. Section 2 discusses the work related to the saliency detection of NSI and RSI, as well as the attention mechanism in SOD. Section 3 describes in detail our proposed network architecture, including SGD and PAF modules. Section 4 introduces the experimental settings, including datasets, evaluation metrics, and implementation details. The proposed method was compared with the 14 SOTA method qualitatively and quantitatively, and then the ablation of the key components was studied. Finally, Section 5 summarizes the research work and points out our future research direction.

2. Related Works

In this section, we first introduce some representative SOD models designed for NSI in Section 2.1, then examine the SOD model specifically for optical RSI in Section 2.2, and, finally, describe some related attention mechanisms for SOD in Section 2.3.

2.1. Saliency Detection for NSI

In the past two decades, we have witnessed the diversified development of the theoretical system of SOD and the rapid improvement of detection performance under the heatwave of deep learning. Early works were mainly devoted to studying hand-made features, such as color transform-based model [28], sparse representation [29,30], low-rank decomposition [31], and graph-based model [32] and so on, their effectiveness and efficiency limit these methods. In the past five years, the SOD method based on convolution neural network (CNN) has been widely and deeply explored [5,33]. Initially, Li et al. [34] used the multi-level context features of CNN to infer the saliency of image segments. Zhao et al. [35] combined local and global context information to rank the superpixels. Compared with traditional models, these methods significantly improve performance, but are still limited by low-resolution prediction results. Consequently, to overcome the above deficiency, most current methods use full convolution network (FCN) to predict pixel-level saliency. Deng et al. [36] proposed a recursive residual refinement network (R3Net), which uses the cross-level features of the integrated FCN with alternating residual refinement blocks. In order to highlight the complementarity of object features and edge features, Zhao et al. [22] proposed an edge guidance network for SOD. Pang et al. [19] proposed to use aggregation interaction module in the decoder to integrate adjacent-level features to avoid a large amount of noise caused by sampling operations.

2.2. Saliency Detection for RSI

Compared with many SOD methods for NSI, only a few works are devoted to the SOD of optical RSI. Usually, SOD is used as an auxiliary tool for RSI image analysis, such as airport detection [12,37], oil tank detection [3], region change detection [13], and ROI extraction [16]. With the in-depth research on SOD, some SOD works on optical RSI have appeared in recent years. Considering the internal relationship of multiple saliency cues, zhang et al. [16] developed an adaptive multi-feature fusion method for saliency detection of RSI. Huang et al. [29] proposed a novel SOD method by exploring sparse representation based on contrast weighted atoms. In the CNN-based method, Li et al. [18] used the SOD of optical RSI by constructing a tow-stream pyramid module and a nested structure with encoding-decoding. In another related work, Li et al. [17] designed a parallel processing structure network for optical RSI by using intra-path, cross-path information, and multi-scale features. Recently, Zhang et al. [20] merged low-level attention cues into high-level attention maps and combined the global upper and lower attention mechanism to propose a SOD framework for optical RSI. Although these methods effectively improve optical RSI’s saliency detection performance, they do not treat different levels of feature information separately, ignore the complementarity between cross-level features, and lack filtering and attention to practical features.

2.3. Attention Mechanism in SOD

In recent years, attention mechanism (AM) has gradually become an essential factor in network structure design and has been deeply studied in many fields [38]. AM simulates the human visual system, which only pays attention to a part of the scene’s prominent area rather than the whole region. This mechanism improves the efficiency of data processing and the pertinence of the target. In other words, AM is a resource allocation mechanism that reallocates fixed resources according to the importance of the object of concern. In the network, AM needs to allocate the resources that can be understood as the weight scores of different dimensional features, such as channel domain attention [39], spatial domain attention [40], mixed domain attention [41], and position-wise attention [42].

This suitable mechanism is also widely used in the field of SOD. Kuen et al. [43] proposed a recurrent attentional convolution-deconvolution network for SOD, in which a spatial converter based on sub-differentiable sampling is used to transform the input features to achieve spatial attention spatially. Considering that most of the previous SOD methods are fine-tuned from image classification networks that only respond to small and sparse differentiated objects, Chen et al. [44] proposed a residual learning method based on reverse attention, which is used to expand the object area gradually. Wang et al. [45] proposed a pyramid attention module with an enlarged receptive field that can effectively enhance the corresponding network layer’s expression ability. Zhang et al. explored a global context-aware AM that captures long-term semantic dependencies between all spatial locations in an attentive manner. Some works directly embed the existing attention module into the network architecture to focus on the salient region’s features and reduce the feature redundancy [26,46].

3. Approach

This section begins with an overview of the proposed semantic-directed attention refinement of the network’s entire architecture in Section 3.1. Then, the proposed semantic guided decoding (SGD) module is introduced in detail in Section 3.2. The proposed parallel attention fusion (PAF) module is described in Section 3.3. Finally, the loss function for SARNet supervision training is given in Section 3.4.

3.1. Overall Network Architecture

As shown in Figure 2, in order to specifically solve the challenges of SOD in RSI, the proposed SARNet is mainly composed of a backbone network (such as VGG [47] and ResNet series [48] and Res2Net [49], etc.), an SGD module for integrating high-level semantic information and guiding top-down feature refinement, and a PAF module for merging cross-level features and global semantic information in a parallel manner. Specifically, take ResNet-50 as an example, the backbone network extracts the features of the input RSI at five different resolutions, which can be expressed as $\left\{F^i|i=1,2,\cdots ,5\right\}$. First of all, we compare the side-out features of the last three layers of the network, i.e., $\left\{F^i|i=3,4,5\right\}$. As the input of the GSD module to obtain the global semantic features that can roughly locate the object. Then high-level features and global semantic features are input into the PAF module as supplements and guides of low-level features to enhance the object’s edge details. The entire SARNet adopts a coarse-to-fine feedback strategy to integrate multiple features and refine salient objects’ details gradually. All side-outputs and global semantic pseudo saliency maps are supervised, and the final saliency map is obtained by mapping output after $F^1$ feature integration.

3.2. Semantic Guided Decoder (SGD)

The current popular SOD deep network model usually aggregates all side-out features without discrimination [17,18,24,36,50], but this strategy will lead to confusion and redundancy of cross-level feature fusion. On the other hand, considering that the backbone network obtains multi-scale representation by continuous downsampling, the feature resolution after the first three feature extraction stages (i.e., three $\times 2$ downsampling) is low enough. We presume that the features extracted in the later stages are high-level representations with rich semantic information. Therefore, we propose an SGD that aggregates the last three layers’ output features $\left\{F^i|i=3,4,5\right\}$ to obtain more accurate contextual semantic information from the global scope.

Specifically, as shown in Figure 3, in order to provide sufficient semantic information required for the location of salient objects with scale-changeable RSI, when a plurality of high-level features are given, $F^4$ and $F^5$ are first upsampled to the same size as $F^3$ and then concatenated to obtain the initial global semantic feature $F_1^g$, can be expressed as:

$$F_1^g=Cat\left(F^3,U{p}_{\times 2}\left(F^4\right),U{p}_{\times 4}\left(F^5\right)\right),$$

(1)

where $Cat$ is concatenating operation, and then the salient object’s position is roughly located, and the probability of existence is calculated by the discriminator $Dis$ and Sigmoid function $\mathit{S}$, respectively. Meanwhile, after feature channel compression and vectorization processing, the initial $F_1^g$ is combined with the position probability of salient objects by matrix multiplication $\mathit{M}$, so as to weight each layer of feature map in space and aggregate global information, and then the weight $C_1^g$ of salient objects on each channel is obtained. The process is defined as:

$$F_2^g=\mathit{T}\left(CRB\left(F_1^g\right)\right),C_1^g=\mathit{M}\left\{\mathit{S}\left[\mathit{T}\left(Dis\left(CRB\left(F_1^g\right)\right)\right)\right],F_2^g\right\},$$

(2)

where $Dis$ represents the discriminant operation of mapping high-dimensional features to 1-dimensional features with a kernel size of $1\times 1$. Further, we perform weighted aggregation on the $F_2^g$ of the previous stage in the channel dimension to obtain the channel feature representation $F_3^g$ of each pixel. This process is expressed as:

$$C_2^g=\mathit{T}\left(C_1^g\right),F_3^g=\mathit{M}\left(CRB\left(F_2^g\right),C_2^g\right).$$

(3)

Finally, the channel feature $F_3^g$ of each pixel is normalized and matrix multiplied with the weight $C_2^g$ of the feature map on each channel to reconstruct the feature representation of each pixel. The process is defined as:

$$F_m^g=\mathit{T}\left(\mathit{M}\left(\mathit{S}\left(F_3^g\right),C_2^g\right)\right).$$

(4)

It is worth noting that $F_m^g$ is a global guide feature with rich semantic information. After the above series of operations, we comprehensively integrate multiple high-level features with semantic information. As shown in Figure 4, compared with the output feature visualization results of the last layer of the backbone (here taking ResNet50 as an example), after the above transformation and calculation in the feature channel and space of the SGD module are adopted, the network enhances the feature representation of pixels and the perception of the object region, and can locate salient objects more accurately.

3.3. Parallel Attention Fusion (PAF) Module

Although the output of SGD can approximately locate salient objects, it seriously lacks detailed features (mostly tiny objects). Therefore, to make full use of this improved semantic feature, we use the output of SGD as a global guiding feature to guide the aggregation of low-level information. At the same time, as a supplement to low-level features, we use the previous layer’s side-out feature as an additional feature to participate in the recovery of salient object details. This strategy is widely used in SOD to reconstruct the object’s edge [20,24,25,26]. On the other hand, the features usually obtained by the encoder are redundant for the SOD task [51], and indistinguishable integrated multi-level features may activate non-salient areas. Therefore, it is necessary to filter and retain these features stream information.

Taken together, we propose the PAF module, as shown in Figure 5. First, the reverse attention weight [44] is applied to the global semantic guide feature $F_i^g$ to explore the details of complementary regions and boundaries by erasing existing salient object regions. For high-level auxiliary features $F^h$ and low-level refinement features $F^l$, channel attention mechanism (CAM) is used to filter to obtain more representative and essential features. Then, to further enhance the discrimination of features, we feed the concatenated features into a feature weighted structure with skip connections. Next, the output weighted feature and the reverse attention weight are combined and fed into the discriminator. Finally, the output of the discriminator and $F_m^g$ are combined in the manner of residual connection to obtain the global semantic guidance feature $F_{m-1}^g$ of the next stage.

Precisely, the reverse attention weight $w^g$ for $F_m^g$ can be calculated as:

$$w^g=E-\mathit{S}\left(F_m^g\right),$$

(5)

where E represents a unit matrix with the same size as $F_m^g$. Simultaneously, the $F_{cam}^l$ and $F_{cam}^h$ features processed by CAM are concatenated and then fed into the feature weighted structure with skip connection to obtain the enhanced feature representation $F_e$ of the object. This process is defined as:

$$F_e=F_{cam}^l\otimes \mathit{S}\left(Dis\left(Cat\left(F_{cam}^l,F_{cam}^h\right)\right)\right).$$

(6)

Further, element-wise multiplication is used to reduce $F_e$ and $w^g$’s gap and then fed into the discriminator. Finally, the residual connection method is combined to obtain the global semantic guidance feature $F_{m-1}^g$ of the next stage. The process can be expressed as:

$$F_{m-1}^g=F_m^g\oplus Dis\left(CRB\left(w^g\otimes F_e\right)\right).$$

(7)

The entire PAF integrates distinctive feature representations in parallel. The PAF module’s output visualization in Figure 6 shows that the step-by-step feedback strategy generates more recognizable and precise object discriminating features in the decoding network.

3.4. Loss Function

In the supervision phase, to avoid the loss function treating all pixels equally, and to guide it to pay more attention to the details of hard pixels and object boundaries, our loss function consists of weighted IoU and binary cross-entropy loss (BCE), i.e., $\mathcal{L}={\mathcal{L}}_{wIoU}+{\mathcal{L}}_{wBCE}$. The loss here is the same as in [50], and its validity has been verified in SOD. Therefore, our total loss is expressed as:

$${\mathcal{L}}_{total}=\mathcal{L}\left(G,S_1\right)+\sum _{i=2}^6\frac{1}{2^{i-2}}\mathcal{L}\left(G,S_i\right),$$

(8)

where G is GT map and $S_i$ represents the side-output map at stage i.

4. Experiments

In Section 4.1, we introduce in detail the RSI datasets and the extended NSI datasets, the evaluation metrics of the experimental results, and the implementation details of the network model. In Section 4.2, we compare the model’s performance in multiple scenarios from both quantitative and quantitative aspects. In Section 4.3, we conducted a series of ablation experiments to demonstrate the compatibility of the model and the necessity of model components. In Section 4.4 and Section 4.5, we analyze the complexity and failure cases of the proposed method, respectively.

4.1. Experimental Settings

4.1.1. Datasets

Experiments were performed on two optical RSI datasets dedicated to SOD, namely ORSSD [18] and EORSSD [20]. The ORSSD dataset contains most of the 800 images collected and pixel-wise annotated from Google Earth and some conventional RSI datasets for classification or object detection (such as NWPU VHR-10 [52], AID [53], Levir [54], etc.), of which 600 are for training, and 200 are for testing. To get closer to the actual scene, EORSSD expands the ORSSD to 2000 images, including 1400 for training and 600 for testing. These images summarize more complex real-world scene types and more challenging object attributes. In these two RSI datasets, accurate and robust SOD is very challenging because of the cluttered and complex background, multiple spatial resolutions, type, size, number of salient objects, etc.

Besides, in order to further demonstrate the robustness and stability of the SARNet. We tested the proposed model on three popular natural scene image (NSI) datasets for SOD, including DUTS [55], DUT-OMRON [56], and HKU-IS [34]. DUTS is a large SOD dataset with two subsets, of which 10,553 images in DUT-TR are used for training, and 5019 images in DUT-TE are used for testing. DUT-OMRON consists of 5168 images, of which objects are usually structurally complex. The HKU-IS includes 4447 images, which contain a plurality of foreground objects. Like other SOD methods [22,57,58,59], we use DUT-TR to retrain our SARNet, and the experimental results are as shown in Section 4.2.3.

4.1.2. Evaluation Metrics

We adopt five widely used evaluation metrics in SOD to comprehensively demonstrate the proposed model’s effectiveness, including mean absolute error $\left(MAE,\mathcal{M}\right)$, mean F-measure $\left(m{F}_{\beta }\right)$, weighted F-measure $\left(w{F}_{\beta }\right)$ [60], mean E-measure $\left(m{E}_{\varphi }\right)$ [61], and S-measure $\left(S_{\xi }\right)$ [62]. Besides, the precision-recall (PR) curves and F-measure curves are also used to compare with the SOTA models. The details of these evaluation metrics are as follows:

(1)

MAE $\left(\mathcal{M}\right)$ evaluates the average difference between all the corresponding pixels of the predicted saliency map $\left(P\right)$ and GT map $\left(G\right)$ after normalization processing. We compute the $\mathcal{M}$ score by:

$$\mathcal{M}=\frac{1}{W\times H}\sum _{i=1}^W\sum _{i=1}^H\left|S\left(i,j\right)-G\left(i,j\right)\right|,$$

(9)

where W and H are the width and height of the evaluate map.

(2)

Mean F-measure $\left(m{F}_{\beta }\right)$ and weighted F-measure $\left(w{F}_{\beta }\right)$ can improve interpolation, dependency, and equality problems, leading to inaccurate estimates of $MAE$ and original F-measure values [60]. This metric is calculated as follows:

$$F_{\beta }=\frac{\left(1+{\beta }^2\right)Precision\times Recall}{{\beta }^2\times Precision+Recall},$$

(10)

where different weight ${\beta }^2$ values are set to emphasize the importance of recall or precision. It is customary to set 0.5 (i.e., $m{F}_{\beta }$) to treat equally or 0.3 (i.e., $w{F}_{\beta }$) to emphasize precision over recall in previous works [29,46,51,59].

(3)

Mean E-measure $\left(m{E}_{\varphi }\right)$ combines the image-level average’s local value with the image-level average to obtain global statistical information and local pixel matching information, an evaluation metric based on cognitive vision. It is computed as:

$$E_{\varphi }=\frac{1}{W\times H}\sum _{i=1}^W\sum _{i=1}^H\theta \left(\varphi \right),$$

(11)

where $\varphi$ is the alignment matrix and $\theta \left(\varphi \right)$ denotes the enhanced alignment matrix [61]. To make a fair comparison, we took the mean score of the evaluation index in the experiment.

(4)

S-measure $\left(S_{\xi }\right)$ takes the structural similarity of region-aware $\left(S_r\right)$ and object-aware $\left(S_o\right)$ as the evaluation of structural information to consider the structural information of the image. $S_{\xi }$ is calculated as follows:

$$S_{\alpha }=\alpha \times S_o+\left(1-\alpha \right)\times S_r,$$

(12)

where $\alpha \in \left[0,1\right]$ is a trade-off parameter, usually set to 0.5.

4.1.3. Implementation Details

As in [20], we use the divided data in EORSSD for training and testing, respectively, and combine rotation, flipping and random cropping strategies to augment all training data. The model is implemented using PyTorch and deployed on a NVIDIA GeForce RTX 3090. For model training, the Adam algorithm is used to optimize model parameters with a learning rate of 1$\times {10}^{-4}$. When the mini-batch size is 16, it takes about 4.5 h to train the model for 80 epochs. In the inference stage, the average processing speed is about 47.3 FPS.

4.2. Comparison with SOTAs

Our results were evaluated with 14 SOTA SOD competitors, including three unsupervised methods (i.e., HDCT [28], SMD [31] and DSG [32]), seven deep learning-based methods (i.e., R3Net [36], DSS [21], PFA [24], EGNet [22], MINet [19], GateNet [59], and F3Net [50]), and four methods developed for RSI (i.e., SMFF [16], CMC [3], LVNet [18], and DAFNet [20]). To make a fair comparison, the results of all non-deep learning methods are provided by the authors or calculated directly by their released codes. The deep learning-based methods use the same training data as the proposed model to retrain and infer the test sets under the default parameter setting (if there are multiple backbone results, the best one is taken).

4.2.1. Quantitative Comparison

Due to the use of different backbone networks, feature extraction performance is affected in varying degrees. To make a comprehensive comparison, we use VGG16 [47], ResNet-50 [48], and Res2Net-50 [49] as feature extractors at the same time.

Table 1. Comparison with the SOTAs. The top three of our one and the other methods are highlighted in red, blue, and green. ↑ and ↓ denote larger and smaller scores are better, respectively. † and ‡ denote CNN-based and RSI-based SOD methods, respectively.

Models	ORSSD [18]					EORSSD [20]
	Sξ ↑	mEϕ ↑	mFβ ↑	wFβ ↑	$\mathcal{M}\downarrow$	Sξ ↑	mEϕ ↑	mFβ ↑	wFβ ↑	$\mathcal{M}\downarrow$
HDCT [28]	0.620	0.650	0.424	0.372	0.131	0.597	0.639	0.403	0.266	0.109
DSG [32]	0.719	0.734	0.575	0.566	0.104	0.643	0.661	0.462	0.402	0.125
SMD [31]	0.764	0.775	0.621	0.557	0.072	0.711	0.731	0.550	0.409	0.077
R3Net † [36]	0.814	0.868	0.738	0.738	0.040	0.819	0.831	0.632	0.418	0.017
DSS † [21]	0.826	0.836	0.696	0.621	0.036	0.787	0.764	0.582	0.461	0.019
PFA † [24]	0.861	0.855	0.731	0.672	0.024	0.836	0.866	0.679	0.549	0.016
EGNet † [22]	0.872	0.901	0.750	0.645	0.022	0.860	0.877	0.697	0.538	0.011
MINet † [19]	0.849	0.894	0.779	0.709	0.028	0.858	0.915	0.772	0.694	0.013
GateNet † [59]	0.893	0.927	0.827	0.763	0.015	0.880	0.904	0.770	0.643	0.011
F3Net † [50]	0.908	0.947	0.827	0.763	0.015	0.907	0.944	0.810	0.769	0.010
SMFF ‡ [16]	0.531	0.568	0.268	0.250	0.185	0.540	0.521	0.301	0.209	0.143
CMC ‡ [3]	0.603	0.642	0.345	0.311	0.127	0.580	0.590	0.270	0.201	0.106
LVNet †‡ [18]	0.882	0.926	0.800	0.751	0.021	0.864	0.883	0.736	0.631	0.015
DAFNet †‡ [20]	0.919	0.954	0.844	0.756	0.011	0.918	0.938	0.798	0.652	0.005
SARNet-VGG16	0.913	0.948	0.862	0.851	0.019	0.924	0.955	0.854	0.830	0.010
SARNet-ResNet50	0.921	0.952	0.867	0.853	0.014	0.926	0.955	0.849	0.832	0.008
SARNet-Res2Net50	0.935	0.966	0.887	0.872	0.010	0.929	0.961	0.857	0.824	0.008

summarizes the evaluation scores across five metrics on two RSI datasets. It can be seen that the results of our method on different backbones are almost better than those of other methods (especially with Res2Net), which verifies the robustness of the proposed method. Figure 7 shows the PR curves and the F-measure curves on two datasets (our result is solid lines marked in red), further demonstrating the proposed model’s superiority.

Compared with other unsupervised learning algorithms for SOD, SMD [31] achieves the best performance under all the two RSI datasets’ evaluation metrics. On the other hand, among the deep learning-based SOD algorithms for NSI retrained with RSI data, F3Net achieves the best performance, with $S_{\xi }$ reaching 0.908 and 0.907 on the ORSSD and the RORSSD datasets, respectively. LVNet [18] and DAFNet [20], as deep learning-based algorithms for RSI saliency detection, performance is significantly better than other algorithms, especially DAFNet. This further demonstrates the necessity of specially designing the detection model for the SOD of optical RSI. For the first two best methods (LVNet [18] and DAPNet [20]) dedicated to optical RSI, our performance gains on the four metrics ($S_{\xi }$, $w{F}_{\beta }$, $m{E}_{\varphi }$, $m{F}_{\beta }$) are 1.7%∼7.5%, 1.2%∼8.8%, 5.0%∼16.4% and 15.3%∼30.5%, respectively.

4.2.2. Visual Comparison

The visual comparison of some representative scenes is shown in Figure 1 and Figure 8, where the results are obtained by training or retraining on the EROSOD dataset through SARNet with ResNet50 backbone and other deep learning-based models. In Figure 1, although other methods are disturbed by object resolution and scene contrast, our method can accurately identify the whole object and obtain a clear boundary. A variety of challenging scenarios are covered in Figure 8, including small objects (a), large objects (b), low-contrast environments (c), and cluttered backgrounds (d). We observe that the proposed method can consistently generate more complete salient maps with sharp boundaries and meticulous details, as reflected in the following aspects:

(1)

Accurately locate the object. The proposed model perceives and accurately locates salient objects with varied scales and shapes in various scenes and has excellent background suppression ability. In detecting the scene with small objects in Figure 8a, most of the methods will miss aircraft and ships’ detection. For the river area (i.e., the second row of (b) and the first row of (d) in Figure 8), LVNet [18], EGNet [22], and SMD [31] can only roughly discover the potential location of the object.

(2)

The sharp edge of the object. How to get a clear object edge has always been a hot issue in the field of SOD. For all the specific challenging scenarios in Figure 8, the competitors can hardly get a saliency map with sharp edges. On the contrary, the proposed method can obtain precise and reliable object edges, especially in small objects and low contrast scenes.

(3)

The internal integrity of the object. From the second image of (b) and two images of (d) in Figure 8, it can be seen that most models cannot maintain the integrity of the object for the saliency detection of scenes containing slender and large targets, such as LVNet [18], F3Net [50], GateNet [59], and MINet [19]. In comparison, our SARNet can obtain internally consistent saliency maps.

4.2.3. Extension Experiment on NSI Datasets

To further discuss the proposed model’s compatibility and scalability, we compare it with nine SOTA saliency detection models, including BMPM [63], PiCA [64], RAS [44], PAGE [45], AFNet [65], BASNet [66], F3Net [50], GateNet [59], and MINet [19], on three NSI datasets that are widely used in SOD. Judging from the four evaluation metrics’ scores in

Table 2. The extended experimental results on three NSI datasets for saliency detection. The best three results are highlighted in red, blue and green. ↑ and ↓ denote larger and smaller is better, respectively.

Models		DUT-OMRON [56]				DUTS-TE [55]				HKU-IS [34]
		Sξ ↑	mEϕ ↑	wFβ ↑	$\mathcal{M}\downarrow$	Sξ ↑	mEϕ ↑	wFβ ↑	$\mathcal{M}\downarrow$	Sξ ↑	mEϕ ↑	wFβ ↑	$\mathcal{M}\downarrow$
2018	BMPM [63]	0.809	0.837	0.681	0.064	0.862	0.860	0.761	0.049	0.907	0.937	0.859	0.039
	PiCA [64]	0.832	0.841	0.695	0.065	0.869	0.862	0.754	0.043	0.904	0.936	0.840	0.042
	RAS [44]	0.814	0.846	0.695	0.062	0.839	0.861	0.740	0.059	0.887	0.929	0.843	0.045
2019	PAGE [45]	0.824	0.853	0.722	0.062	0.854	0.869	0.769	0.052	0.903	0.940	0.866	0.037
	AFNet [65]	0.826	0.853	0.717	0.057	0.867	0.879	0.785	0.046	0.905	0.942	0.869	0.036
	BASNet [66]	0.836	0.869	0.751	0.056	0.866	0.884	0.803	0.048	0.909	0.946	0.889	0.032
2020	F3Net [50]	0.838	0.870	0.747	0.053	0.888	0.902	0.835	0.035	0.917	0.953	0.900	0.028
	GateNet [59]	0.838	0.862	0.729	0.055	0.885	0.889	0.809	0.040	0.915	0.949	0.880	0.033
	MINet [19]	0.833	0.865	0.738	0.055	0.884	0.898	0.825	0.037	0.919	0.953	0.897	0.029
SARNet		0.843	0.873	0.773	0.058	0.890	0.904	0.827	0.037	0.920	0.956	0.914	0.028

(marked in red is the best), our SARNet can be highly competitive with these SOTA models, even better than most SOD models. In addition, some visual comparison results in NSI are shown in Figure 9, our results have sharp boundaries while maintaining the integrity of salient objects.

4.3. Ablation Study

Section 3 describes and explains the details of the proposed architecture in detail, from which we can see that our SARNet is composed of three key components, i.e., the backbone network for feature extraction, the integration strategy of side-out features, and the proposed semantic guidance decoding (SGD) module and parallel attention fusion (PAF) module. Therefore, this section conducts ablation experiments in the following three aspects to evaluate the necessity and contribution of each key component:

• Scalability. Table 1 shows that the performance of SARNet can be effectively improved by using better backbones, and it also demonstrates the scalability of the proposed architecture. As shown in Table 1, the benchmark results on two RSI datasets show that the performance of SARNet can be effectively improved through a better backbone, which demonstrates the scalability of the proposed network architecture. As shown by the expansion experiments on NSI datasets in Table 2, the proposed model has exceptional competitive detection performance on multiple natural scene datasets, which further shows the compatibility and robustness of our SARNet.
• Aggregation strategy.

  Table 3. Ablation study of different aggregation strategies on the ORSSD and EORSSD datasets.

  Settings	ORSSD [18]					EORSSD [20]
  	Sξ ↑	mEϕ ↑	mFβ ↑	wFβ ↑	$\mathcal{M}\downarrow$	Sξ ↑	mEϕ ↑	mFβ ↑	wFβ ↑	$\mathcal{M}\downarrow$
  $L{F}^3$	0.875	0.831	0.742	0.682	0.032	0.843	0.825	0.732	0.688	0.031
  $H{F}^3$	0.889	0.862	0.778	0.691	0.022	0.881	0.846	0.755	0.681	0.026
  $SG{D}^1$	0.904	0.918	0.833	0.792	0.019	0.894	0.901	0.812	0.773	0.023
  $SG{D}^2$	0.918	0.931	0.866	0.849	0.014	0.902	0.928	0.838	0.792	0.017
  $PA{F}^g$	0.924	0.938	0.873	0.864	0.012	0.920	0.946	0.845	0.816	0.011
  $PA{F}^h$	0.917	0.927	0.868	0.853	0.014	0.916	0.937	0.841	0.810	0.013
  SARNet	0.935	0.966	0.887	0.872	0.010	0.929	0.961	0.857	0.824	0.008

  quantitatively shows the interaction and contribution of the proposed semantic guidance and cascade refinement mechanism on two RSI datasets. “$L{F}^3$” and “$H{F}^3$” respectively involve only three low-level features ($F^1\sim F^3$) and three high-level features ($F^3\sim F^5$). “$SG{D}^1$” and “$SG{D}^2$”, respectively, refer to only the combination of $F^5$ and $F^5+F^4$ in the semantic guidance stage. “$PA{F}^g$” and “$PA{F}^h$”, respectively, use the combination of $F^l+F^g$ and $F^l+F^h$ in the parallel feature fusion stage. It can be seen from the metric scores in Table 3 that the proposed model benefits from the additional global semantic features and the feature aggregation strategy adopted.
• Module. We conducted some evaluations on the effectiveness of the proposed modules. The baseline model (BM) used is a network with FPN structure. We assembled the SGD and PAF modules on the BM during the experiment, and the scores are shown in

  Table 4. Ablation study with different components combinations on the ORSSD and EORSSD datasets.

  Settings			ORSSD [18]					EORSSD [20]
  BM	SGD	PAF	${\mathit{S}}_{\mathit{\xi }}\uparrow$	${\mathit{mE}}_{\mathit{\varphi }}\uparrow$	${\mathit{mF}}_{\mathit{\beta }}\uparrow$	${\mathit{wF}}_{\mathit{\beta }}\uparrow$	$\mathcal{M}\downarrow$	${\mathit{S}}_{\mathit{\xi }}\uparrow$	${\mathit{mE}}_{\mathit{\varphi }}\uparrow$	${\mathit{mF}}_{\mathit{\beta }}\uparrow$	${\mathit{wF}}_{\mathit{\beta }}\uparrow$	$\mathcal{M}\downarrow$
  ✔			0.807	0.855	0.727	0.742	0.047	0.796	0.832	0.711	0.724	0.045
  ✔	✔		0.868	0.896	0.812	0.800	0.028	0.857	0.894	0.811	0.793	0.024
  ✔		✔	0.876	0.923	0.834	0.815	0.018	0.867	0.927	0.835	0.807	0.017
  ✔	✔	✔	0.935	0.966	0.887	0.872	0.010	0.929	0.961	0.857	0.824	0.008

  . Note that multi-level features are integrated by simple concatenate or addition operations to replace the proposed feature aggregation modules in the experiment. From the experimental results of the two datasets, we can see that both SGD and PAF modules can improve the model’s performance in varying degrees. The PAF module contributes more to the network than the SGD module. With the combination of the two modules, the proposed model can achieve the best performance.

4.4. Complexity Analysis

We provide some comparisons of the complexity of CNN-based SOD algorithms, including model parameters (#Param), GPU memory usage, and the number of floating-point operations (FLOP), as shown in

Table 5. Model complexity comparison of some CNN-based methods.

Complexity	DSS	EGNet	PiCA	AFNet	BASNet	F3Net	GateNet	MINet	SARNet
	[21]	[22]	[64]	[65]	[66]	[50]	[59]	[19]	Ours
#Param(M)	62	108	33	37	87	26	100	270	40
Memory(M)	3209	1177	1541	1123	1103	652	2058	2355	1126
FLOPs(G)	115	271	37	36	127	15	84	93	38

. For SOD detectors, memory usage and FLOPs tested using a 336 × 336 input image except that a method specifies its input dimensions. Here, #Param and GPU memory usage are measured in millions (M), and the number of FLOPs is measured in Giga (G). From these criteria for evaluating the complexity of the model, we can see that our method is at a lower-middle level.

4.5. Failure Case

Although our model is more advanced than SOTAs in qualitative and quantitative experiments, a few cases in which the detection results are not satisfactory are shown in Figure 10. As shown in the first three columns in Figure 10, when the scene contains salient objects other than GT objects (that is, oil tanks, roofs, and water in the first to third columns), the detector can find all potential objects, which may require further contextual constraints to mitigate the situation. Our model detects small objects (cars and ships) in the fourth and fifth columns rather than bridges in the scene, which may be due to the lack of image data with salient objects on bridges in the training data (only four in EORSSD). In the sixth and seventh columns, we show examples of incomplete target detection, which can be improved by fine-tuning or training optimization.

5. Conclusions

This paper explores salient object detection in complex optical remote sensing scenes and tries to solve the challenging problems of inaccurate location and the unclear edge of salient objects. We propose a novel semantic-guided attention refinement network for SOD of optical RSI, which is an end-to-end encoding-decoding network architecture. The proposed SGD module focuses on the aggregation of high-level features to roughly but accurately locate the objects in the scene and guides the aggregation of low-level features through top-down feedback to refine the boundaries. The PAF module further integrates high-level and low-level side-out features and semantic guidance features through corresponding attention mechanisms. The comprehensive comparison between two RSI datasets and three extended NSI datasets and various ablation experimental results show that our SARNet shows the most advanced performance and strong robustness and compatibility on multi-scene datasets.

In future works, we will further study the following two directions: (1) to make more and larger optical RSI data sets for SOD. At present, the largest optical RSI dataset, i.e., EORSSD [20], contains 2000 images, which is higher than the number of the ORSSD [18] images. However, compared with the number of NSI for SOD or other datasets for object detection and semantic segmentation, the number of images is insufficient to support large-scale deep learning. Besides, remote sensing data usually cover a wide range of land, so it is necessary to expand the scale of the image to cover more. (2) Explore the multi-modal SOD method for RSI. In the last two years, the SOD method for RGB-D has been widely studied in NSI [33,46,67]. A variety of sensors can capture remote sensing images; naturally, the idea of multi-modal SOD can be extended to the SOD of multi-source remote sensing scenes.