An Effective Multi-Layer Attention Network for SAR Ship Detection

Abstract

The use of deep learning-based techniques has improved the performance of synthetic aperture radar (SAR) image-based applications, such as ship detection. However, all existing methods have limited object detection performance under the conditions of varying ship sizes and complex background noise, to the best of our knowledge. In this paper, to solve both the multi-scale problem and the noisy background issues, we propose a multi-layer attention approach based on the thorough analysis of both location and semantic information. The solution works by exploring the richness of spatial information of the low-level feature maps generated by a backbone and the richness of semantic information of the high-level feature maps created by the same method. Additionally, we integrate an attention mechanism into the network to exclusively extract useful features from the input maps. Tests involving multiple SAR datasets show that our proposed solution enables significant improvements to the accuracy of ship detection regardless of vessel size and background complexity. Particularly for the widely-adopted High-Resolution SAR Images Dataset (HRSID), the new method provides a 1.3% improvement in the average precision for detection. The proposed new method can be potentially used in other feature-extraction-based classification, detection, and segmentation.

1. Introduction

Synthetic aperture radar (SAR) is widely employed in marine exploration (itself an important field for multiple areas, ranging from biological to chemical research [1]) for being able to overcome interference caused by adverse weather or insufficient light during observations [2,3]. Not only has SAR become more widely used, but the volume of data produced through SAR usage has also increased steadily due to the proliferation of high-resolution satellite imaging. Among the interesting applications, SAR-based ship detection has garnered special attention in international research in the field of maritime exploration due to its applications in trajectory prediction, congestion avoidance, and activity monitoring [4,5].

The conventional methods of ship detection in SAR images have a low level of generalization, which is an issue as SAR is used in very different scenarios, such as varying sunlight and changing weather conditions, and for different types of ship [6,7]. Another problem, illustrated in Figure 1a, is that the images obtained are often full of unnecessary and interfering information that pertains to the environment, such as the bank edges and piers, but does not include the ship, which is the object that needs to be detected. Finally, as shown in Figure 1b, ships come in very different sizes, so an efficient detection solution needs to be able to handle objects of various scales and still properly detect them, which is an issue since flexibility and adequate response is essential [8]. There are many works in the literature on SAR-based ship detection, as will be elaborated in Section 2, but none handles all the problems listed here.

In this paper, we will propose a solution that addresses all of these issues. Our proposal is based on the construction of a precise feature map that identifies key regions in the figure where the ship probably is. This is performed by careful analysis of the semantic and location information of the image through a multi-layer attention module (MAM) in our novel multi-layer attention network (MANet). The processing of both semantic and spatial data is performed on feature maps generated on the original image from ResNet50, a widely adopted standard neural network, which trains deep neural networks and extracts features at different resolutions. This richness in information not only allows our proposal to exclude noise of unimportant parts of the image more efficiently but also allows us to lower the chance of ship misidentification due to the scale of the vessel. The resulting enhanced feature map produced by our MANet can then be used as an input for the Feature Pyramid Network (FPN) [10], a standard deep learning solution for multi-scale object detection, for improved performance at ship detection from SAR images.

The main contributions of this paper are as follows. (i) We propose a cross-level fusion of multi-layer feature maps that can integrate semantic and location data, thus providing a better solution to the multi-scale problem. (ii) We design an attention mechanism that ensures that the feature maps of each layer are efficiently built so that the interference of inshore objects is reduced and more focus is given to the ship object. (iii) We carefully evaluate the performance of our proposed approach and show that our method is significantly better in detection precision than conventional solutions.

The remainder of this paper is organized into the following sections. Section 2 discusses in depth the state-of-the-art methods regarding ship detection. Section 3 explains what our proposed approach is. Section 4 contains the performance evaluation on our proposed solution through images obtained by SAR. Section 5 concludes the paper.

2. Related Works

Ship detection in SAR images can be divided into traditional SAR image processing and SAR image processing based on deep learning.

2.1. Traditional SAR Ship Detection

The traditional solutions for ship detection are based on a statistical clutter modeling called constant false alarm rate (CFAR) [6,7,11,12,13,14,15]. The foundation of such methods is the exploring of the statistical distribution of different marine clutters and using adaptive thresholding techniques. For example, Ai et al. [11] utilized the strong correlation in gray intensity and 2D joint log-normal distribution of pixels in their surrounding area to identify ships. Wang et al. [6] used a quick block detector to remove sea clutter and then identified ships based on kernel density estimate and aspect ratio of the remaining pixels. Leng et al. [7] focused on minimizing the impact of ambiguities and sea clutter in SAR images. Wang et al. [13] introduced an intensity space domain-based detector that takes advantage of the intensity and correlation between pixels. Pappas et al. [14] used super-pixels to define the guard bands and backdrops used in CFAR. Similarly, Li et al. [15] differentiated objects from clutter through weighted information entropy, and used a two-stage CFAR detection system that offered global and local detection. However, in the above CFAR-based solutions, key parameters related to multiple scenario conditions have to be manually pre-determined, significantly decreasing the capacity of generalization in traditional solutions and worsening the overall performance for unexpected situations. They are also particularly sensitive to objects in the background. This deteriorates the performance of the detector when the backdrop objects have complex features.

2.2. Deep Learning-Based SAR Ship Detection

With the popularization of deep learning [16,17,18,19], the application of convolutional neural network (CNN) in SAR-based ship detection is becoming a research hotspot [20]. For example, Li et al. [20] evaluated hard negative mining, transfer learning, feature fusion, and other implementation techniques, alongside a novel dataset and CNNs to achieve faster detection [16]. Moreover, Zhou et al. [21] evaluated the lightweight algorithm of You Only Look Once (YOLO) v4 [19] applied to ship detection to reduce the number of needed features by depth-wise separable convolution and MobileNet v2 as the network architecture. Chang et al. [22] simplified the classic YOLOv2 architecture for CNN to reduce calculation time when detection ships in SAR images. Zhang et al. [23,24] proposed a variant of YOLO-FA to incorporate the frequency-domain information. Sun et al. [25] used fully convolutional one-stage object identification as the foundation of their detector to improve the network’s position regression branch features, and bounding box regression detection. Similarly, Mao et al. [26] proposed an anchor-free SAR-based ship detection system for bounding box regression, and a different network for score map regression. Zhou et al. [27] added to this by simplifying the CNN operation through a two-way dense connection module. Bai et al. [28] adapted the anchor-free detector and established a boundary frame detection strategy to detect objects. However, these works fail to address the substantial interference caused by near-shore background and the vast diversity of scale (see Figure 1 for an example). Because of that, the accuracy of detecting multi-scale ships using SAR still needs improvement, even when using deep learning.

Existing research shows that feature pyramid networks (FPN) [10] are one of the solutions that can address multi-scale object detection and significantly improve small object identification. Additionally, an attention mechanism object detection can detect and focus on specific areas of interest in a big image [29,30], which is a possibility with useful application in SAR-based ship detection [31,32,33]. Cui et al. [31] introduced a dense attention pyramid network that used a convolutional block attention module (CBAM) densely connected to each concatenated feature map from top to bottom. Zhao et al. [32] also used CBAM with an attention-receptive pyramid scheme. Yang et al. [33] used a coordinate attention module and a receptive field expanded module that works better with varying sizes of vessels. Bai et al. [34] proposed a feature enhanced pyramid and shallow feature reconstruction network to improve the detection for small and medium-sized ships. However, these studies did not fully integrate the feature maps of low and high levels, resulting in limited identification performance.

Nonetheless, none of the mentioned solutions integrate multi-layer features, which is essential for the success of deep learning-based object recognition. For example, for identifying ships, high-level feature maps are better due to their wider receptive field, more semantic information, and less spatial information. However, low-level feature maps are more suited for recognizing smaller ships due to their narrower receptive fields, greater emphasis on spatial information, and less reliance on semantic information. Multi-scale ship detection relies heavily on this kind of seamless combination of semantic and geographical data. Such a technique allows the accuracy to be improved by extracting relevant data from bigger feature maps. Attention-based approaches can even be used to draw focus to relevant details and ignore backdrop noise to further increase ship detection accuracy. Our proposed MAM is based on these ideas, integrating semantic and spatial information to highlight the salient features of the desired object and using an attention-based technique alongside a multi-layer feature map to highlight features of the object. These characteristics make the feature map output by our proposal more useful for the underlying network.

3. Methods

In this section, we will explain the details of our proposed MANet-based architecture for ship detection in SAR images.

3.1. Overall Design

Figure 2 shows how the CNN-based solution for SAR-based ship detection works and where we propose to integrate our novel MANet, composed of a Faster R-CNN [16] and a new multi-layer attention module (MAM), into such a system. In Figure 3, we show, in detail, how our proposed MAM is designed. We divide it into two sub-modules, multi-layer channel attention module (MCAM) and multi-layer spatial attention module (MSAM).

Our MAM works as follows. First, the MCAM performs channel attention processing on each layer’s feature maps generated by the backbone network, i.e., ResNet50 in Figure 3, so that the significant features on the channel axis are identified. Each channel’s focus is then combined and applied to the initial feature map to enable the adaptive acquisition of a feature map with significant scale information. Then, the MSAM will perform spatial attention processing on the outputs of each layer of the MCAM. These will be fused to obtain weights for each position. The final optimum feature map is obtained by broadcasting these weights to MCAM’s feature maps for each layer’s output. Through this process, the prominent aspects of the multi-scale feature map can be employed by the MAM when searching for ships. While FPN is capable of fusing feature maps from top to bottom, our proposed MAM uses an attention-based approach to reduce the amount of interference from the environment, filter out undesirable noise, highlight the needed information, and produce more relevant feature maps through a complete integration of semantic and geographic information contained in the data of each layer. The use of deep learning is paramount in identifying the relevant patterns in the data [35].

3.2. MAM

As mentioned earlier, the MAM is divided into two sub-modules, MCAM and MSAM, that are, themselves, responsible for the functions of multi-layer channel attention (MCA) enhancement and multi-layer spatial attention enhancement, respectively. The ResNet50 structure is chosen as the feature extraction network because it produces four layers of feature maps at varying resolutions. The set of generated features is denoted by ${\{C}_1{,C}_2{,C}_3{,C}_4\}$. Accordingly, the MCAM is composed of the processing unit set ${\{\mathrm{MCA}}_1{,\mathrm{MCA}}_2{,\mathrm{MCA}}_3{,\mathrm{MCA}}_4\}$. Each processing unit implements a channel attention enhancement focused on its corresponding feature map, and the resulting enhanced feature maps are defined as ${\{X}_1{,X}_2{,X}_3{,X}_4\}$. Similarly, the MSAM consists of the processing unit set ${\{\mathrm{MSA}}_1{,\mathrm{MSA}}_2{,\mathrm{MSA}}_3{,\mathrm{MSA}}_4\}$, which implements spatial attention enhancement for ${\{X}_1{,X}_2{,X}_3{,X}_4\}$, respectively. The final output feature graph, which will be the input of the FPN network, is denoted as $\left\{P_1{,P}_2{,P}_3{,P}_4\right\}$. Therefore, the whole MAM process can be summarized as follows:

$$X_i{=\mathrm{MCA}}_i\left(C_i\right)\hspace{1em}i=1,2,3,4;$$

(1)

$$P_i{=\mathrm{MSA}}_i\left(X_i\right)\hspace{1em}i=1,2,3,4.$$

(2)

3.3. MCAM

The MCAM is made of four MCA processing units denominated as ${\{\mathrm{MCA}}_1{,\mathrm{MCA}}_2{,\mathrm{MCA}}_3{,\mathrm{MCA}}_4\}$, as is illustrated in Figure 3a. Because the process is identical for each unit, the following explanations will consider a single MCA only. The specific operation steps of MCA are divided into the generation of the single-layer channel attention (SCAM), the generation of the map weights, and the generation of the feature map. The three steps are thoroughly explained as follows.

3.3.1. Generation of Single-Layer Channel Attention

To capture rich semantic and location information, all four feature maps of the backbone network module, ${\{C}_1{,C}_2{,C}_3{,C}_4\}$, are used as input of the MCA processing unit. First, each feature map ${\{C}_i,i=1,2,3,4\}$ goes through the SCAM (see Figure 4 for an illustration). For convenience of description, the subscript of $C_i$ is removed to obtain $C\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$, where c, h, and w represent the number of channels, height, and width of the feature map, respectively. The detailed procedures that make up SCAM’s operation are presenting in the following.

Global spatial average pooling (GSAP). The operation of GSAP for feature map $C\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$ is represented as follows

$${GS}_{\mathrm{avg}}^k=\frac{1}{\mathrm{h}\times \mathrm{w}}{\displaystyle \sum }_{\begin{array}{c}m=1,2,3,\dots ,\mathrm{h}\\ n=1,2,3,\dots ,\mathrm{w}\end{array}}{C}_{m,n,k}$$

(3)

where $C_{m,n,k}$ denotes the feature map of the m-th row and n-th column on the k-th channel, and ${GS}_{\mathrm{avg}}^k$ represents the feature map of the k-th channel after GSAP. In other words, the set of all ${GS}_{\mathrm{avg}}^k$ make up the bigger feature map ${GS}_{\mathrm{avg}}=\left\{{GS}_{\mathrm{avg}}^k|k=1,2,3,\dots ,c\right\}\in {\mathrm{R}}^{c\times 1\times 1}$ after GSAP is concluded.

Global spatial max pooling (GSMP). The operation of GSMP for feature map $C\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$ is represented as follows

$$\begin{array}{ll}{GS}_{\max }^k=\max \left(C_{m,n,k}\right)& \hspace{1em}m=1,2,3,\dots ,\mathrm{h}\\ & \hspace{1em}n=1,2,3,\dots ,\mathrm{w},\end{array}$$

(4)

where max indicates an operator that obtains the maximum value in the channel, and ${GS}_{\max }^k$ denotes the feature map of the k-th channel after GSMP. In other words, the set of all ${GS}_{\max }^k$ make up the bigger feature map ${GS}_{\max }=\left\{{GS}_{\max }^k|k=1,2,3,\dots ,c\right\}\in {\mathrm{R}}^{c\times 1\times 1}$ after GSMP is concluded.

Then, the channel attention measures $B_{\mathrm{avg}}$ and $B_{\max }$ are generated by using a multi-layer perceptron (MLP) that receives, respectively, $G{S}_{\mathrm{avg}}$ and ${GS}_{\max }$ as input. Then, an element-wise summation is performed to obtain the single-layer channel attention B, i.e., channel attention B is computed as:

$$\begin{array}{cc}\hfill B& =B_{\mathrm{avg}}+B_{\max }\hfill \\ & =\mathrm{MLP}\left({GS}_{\mathrm{avg}}\right)+\mathrm{MLP}\left({GS}_{\max }\right).\hfill \end{array}$$

(5)

To minimize the number of parameters in the MLP model, the hidden activation size is changed to ${\mathrm{R}}^{\frac{\mathrm{C}}{\mathrm{r}}\times 1\times 1}$, where r represents the reduction ratio.

Using this method, $B_1\in {\mathrm{R}}^{{\mathrm{c}}_1\times 1\times 1}$, $B_2\in {\mathrm{R}}^{{\mathrm{c}}_2\times 1\times 1}$, $B_3\in {\mathrm{R}}^{{\mathrm{c}}_3\times 1\times 1}$, and $B_4\in {\mathrm{R}}^{{\mathrm{c}}_4\times 1\times 1}$ are obtained through SCAM, where ${\mathrm{c}}_i(i=1,2,3,4)$ is the channel index, and, without loss of generality, the condition ${\mathrm{c}}_1{<\mathrm{c}}_2{<\mathrm{c}}_3{<\mathrm{c}}_4$ is assumed.

3.3.2. Generation of MCA

Considering that the dimensionality of the attention maps $B_i(i=1,2,3,4)$ are not necessarily the same, it is necessary to first convert them into the same dimension. Taking the ${\mathrm{MCA}}_2$ processing module as an example, $B_1$, $B_3$, and $B_4$ should be converted to the exact dimensions as $B_2$. The conversion process is indicated as follows:

$$D_i=FC\left(B_i\right)\hspace{1em}i=1,3,4,$$

$$D_2{=B}_2,$$

(6)

where $FC$ represents the convolution operation.

After the conversion, the outputs will satisfy $D_1\in {\mathrm{R}}^{{\mathrm{c}}_2\times 1\times 1}$, $D_3\in {\mathrm{R}}^{{\mathrm{c}}_2\times 1\times 1}$, and $D_4\in {\mathrm{R}}^{{\mathrm{c}}_2\times 1\times 1}$. By doing this, all four layers of the channel attention map can be added through an element-wise summation. The sigmoid procedure is then used to determine the MCA map’s weight. This process is represented by

$$E_2=\sigma \left(D_1{+D}_2{+D}_3{+D}_4\right),$$

(7)

where $\sigma$ denotes the sigmoid function.

Similarly, $E_1\in {\mathrm{R}}^{{\mathrm{c}}_1\times 1\times 1}$, $E_3\in {\mathrm{R}}^{{\mathrm{c}}_3\times 1\times 1}$, and $E_4\in {\mathrm{R}}^{{\mathrm{c}}_4\times 1\times 1}$ can be obtained following the same process.

3.3.3. Generation of the MCA Feature Map

Finally, the MCA feature map of each layer is obtained through Equation (8):

$$X_i{=C}_i\times \left({1+E}_i\right)\hspace{1em}i=1,2,3,4.$$

(8)

3.4. MSAM

The MSAM, which is illustrated in Figure 3a, is made up of four MSA processing units represented by ${\{\mathrm{MSA}}_1{,\mathrm{MSA}}_2{,\mathrm{MSA}}_3{,\mathrm{MSA}}_4\}$. The multi-channel attention feature ${\{X}_1{,X}_2{,X}_3{,X}_4\}$ obtained according to Equation (8) is used as input for the MSA units. As with the MCAM, the four MSA processing units have the same flow, thus our description will focus on a single module. Additionally, similar to MCA, the specific operation steps of the MSA are divided into the generation of the single-layer spatial attention, generation of the multi-layer spatial attention, and generation of the multi-layer spatial attention feature map.

3.4.1. Generation of Single-Layer Spatial Attention

First, each feature map ${\{X}_i,i=1,2,3,4\}$ goes through the single-layer spatial attention module (shown in Figure 5). To simplify the description, the subscript of the output characteristics of the MCA is removed to obtain $X\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$.

Global channel average pooling (GCAP). The operation GCAP is completed for each feature map $X\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$, with process being computed as follows

$${GC}_{m,n,avg}=\frac{1}{c}{\displaystyle \sum }_{k=1}^c{X}_{m,n,k},$$

(9)

where $X_{m,n,k}$ denotes the feature mapping value at the position of row m and column n on channel k, and ${GC}_{m,n,avg}$ represents the feature map of the corresponding spatial position after GCAP. The set of all ${GC}_{m,n,avg}$ makes up the feature map ${GC}_{\mathrm{avg}}{=\{GC}_{m,n,\mathrm{avg}}|m=1,2,3,\dots ,\mathrm{h},n=1,2,3,\dots ,\mathrm{w}\}$ after the GCAP operation is completed, where ${GC}_{\mathrm{avg}}\in {\mathrm{R}}^{1\times \mathrm{h}\times \mathrm{w}}$.

Global channel max pooling (GCMP). The operation of GCMP is completed for each feature map $X\in {\mathrm{R}}^{c\times \mathrm{h}\times \mathrm{w}}$, with the operation being summarized by Equation (10).

$${GC}_{m,n,max}=Max\left(X_{m,n,k}\right),$$

(10)

where Max obtains the maximum value in row m and column n among all channels. Moreover, ${GC}_{m,n,\max }$ indicates the feature map of the corresponding spatial position after GCMP is completed. The set of all ${GC}_{m,n,\max }$ forms the feature map ${GC}_{\max }{=\{GC}_{m,n,\max }|m=1,2,3,\dots ,\mathrm{h},n=1,2,3,\dots ,\mathrm{w}\}$ after GCMP is completed, where ${GC}_{\max }\in {\mathrm{R}}^{1\times \mathrm{h}\times \mathrm{w}}$.

${GC}_{\mathrm{avg}}$ and ${GC}_{\max }$ are then concatenated, and a convolution with a standard layer is performed, culminating in the creation of the spatial attention map. The operation is summarized as

$$H=FC\left(Concat\left({GC}_{avg}{,GC}_{max}\right)\right).$$

(11)

where Concat and $\mathrm{FC}$ stand for concatenation and convolution with a $7\times 7$ filter size, respectively. Next, we obtain single-layer spatial attention feature map $H\in {\mathrm{R}}^{1\times \mathrm{h}\times \mathrm{w}}$.

Following the described process, a four-layer spatial attention map is obtained composed of $H_1\in {\mathrm{R}}^{{1\times \mathrm{h}}_1{\times \mathrm{w}}_1}$, $H_2\in {\mathrm{R}}^{{1\times \mathrm{h}}_2{\times \mathrm{w}}_2}$, $H_3\in {\mathrm{R}}^{{1\times \mathrm{h}}_3{\times \mathrm{w}}_3}$, and $H_4\in {\mathrm{R}}^{{1\times \mathrm{h}}_4{\times \mathrm{w}}_4}$, where ${\mathrm{h}}_1$, ${\mathrm{w}}_1$, ${\mathrm{h}}_2$, ${\mathrm{w}}_2$, ${\mathrm{h}}_3$, ${\mathrm{w}}_3$, ${\mathrm{h}}_4$, and ${\mathrm{w}}_4$ are the height and width of the four-layer feature map and, without loss of generality, we assume that ${\mathrm{h}}_1{>\mathrm{h}}_2{>\mathrm{h}}_3{>\mathrm{h}}_4$ and ${\mathrm{w}}_1{>\mathrm{w}}_2{>\mathrm{w}}_3{>\mathrm{w}}_4$.

3.4.2. Generation of Multi-Layer Spatial Attention

Given that the dimensions of $H_1$, $H_2$, $H_3$, and $H_4$ are different, we need to modify them so that they have same dimensionality. For example, considering the ${\mathrm{MSA}}_2$ processing module, $H_1$, $H_2$, and $H_4$ should be converted into the same dimension as $H_2$. The conversion process is calculated by Equation (12).

$$I_i=Sample\left(H_i\right)\hspace{1em}i=1,3,4,$$

$$I_2{=H}_2,$$

(12)

where Sample denotes an up-sampling/down-sampling transformation.

After transformation, $I_1\in {\mathrm{R}}^{{1\times \mathrm{h}}_2{\times \mathrm{w}}_2}$,$I_3\in {\mathrm{R}}^{{1\times \mathrm{h}}_2{\times \mathrm{w}}_2}$, and $I_4\in {\mathrm{R}}^{1\times {\mathrm{h}}_2\times {\mathrm{w}}_2}$ are obtained, and then concatenated, convolved, and activated by a sigmoid function. The whole process is as follows

$$G_2=\sigma \left(FC\left(Concat\left(I_1{,I}_2{,I}_3{,I}_4\right)\right)\right)$$

(13)

where $G_2\in {\mathrm{R}}^{{1\times \mathrm{h}}_2{\times \mathrm{w}}_2}$.

Similarly, $G_1\in {\mathrm{R}}^{{1\times \mathrm{h}}_1{\times \mathrm{w}}_1}$, $G_3\in {\mathrm{R}}^{{1\times \mathrm{h}}_3{\times \mathrm{w}}_3}$, and $G_4\in {\mathrm{R}}^{{1\times \mathrm{h}}_4{\times \mathrm{w}}_4}$ can be obtained, giving us the global spatial attention feature map weights of each layer.

3.4.3. Generation of the Multi-Layer Spatial Attention Feature Map

The final refined output can be obtained by Equation (14).

$$P_i{=X}_i\times \left({1+G}_i\right)\hspace{1em}i=1,2,3,4.$$

(14)

4. Experiments

In this section, we will detail the experiments conducted to evaluate how well the suggested approach detects the objects of interest. First, the dataset and parameters used in the experiments are presented. Then, the influence of our proposed MANet is shown by comparing its performance with other techniques in the existing datasets.

4.1. Datasets

Three standard SAR-based ship detection datasets were used for the experiments in this paper. The SAR Ship Detection Dataset (SSDD) [9] generated by the RadarSat-2, TerraSAR-X, and Sentinel-1 [36] satellites has 1160 pictures of ships in various situations, which cover places including Yantai, China, and Visakhapatnam, India. Following the original paper, the dataset was divided in a ratio of 8:2 between training and testing. The SAR-Ship-Dataset [37] was created using 102 Chinese Gaofen-3 pictures and 108 Sentinel-1 pictures. These images cover several near-shore areas in China, South Korea, Japan, etc. A sliding window of 256 × 256 pixels with a sliding step of 128 pixels was used on these pictures, resulting in 43,819 images that include 59,535 ships. Out of these images, a ratio of 7:2:1 was used to randomly select samples for training, verification, and testing. Finally, the high-resolution SAR images dataset (HRSID) [38] has 5604 cropped images and 16,951 ships placed at Houston, Sao Paulo, Aswan Dam, Shanghai, etc. These images come from 136 panoramic SAR pictures, with a 25% overlap rate. The original pictures have a relatively high resolution of greater than 800 × 800 pixels. Overall, 65% of the images from the HRSID were used for training, while 35% were used for testing.

4.2. Parameters and Metrics

The MMDetection framework [39] and machines powered by AMD Ryzen 7 3700X processors and NVIDIA GeForce GTX 1080Ti GPU cards manufactured by AMD and NVIDIA respectively were used for all tests. The machine was running Ubuntu 18.04.

The parameters in the experiment are set in

Table 1. Test settings.

Batch Size	Initial Leaning Rate	Momentum Decay	Weight Decay	Number of Epochs
1	1.25 × 10−4	1 × 10−4	0.9	30

. Such values were obtained after careful tests to find the best hyperparameters for our desired application [40]. Other hyperparameters are the same as the default values in MMDetection.

Four assessment indicators, precision, recall, F1 score, and AP (short for “average precision”, a quality assessment index used in the literature), as given below [41], were used to gauge how well our proposed strategy performed.

$$Precision=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

$$Recall=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

$$F1score=2\times \frac{Precision\times Recall}{Precision+Recall}$$

where false-positive (FP) represents the number of false detections if the object is not present, false-negative (FN) represents the number of no-detections when the object exists, true-negative (TN) is the number of the correct no-detections, and true-positive (TP) represents the number of correct detections.

AP is defined as the integral of the precision as a function of the recall, as given by

$$AP={{\displaystyle \int }}_0^{1}Precision\left(Recall\right)\mathrm{d}Recall$$

The most often used indices for precision and recall are the ratios of properly predicted ships in all forecasts, for precision, and in all ground truth ships, for recall. Precision and recall are then combined to provide the full assessment measure known as the F1 score. Furthermore, AP is another widely used metric that also considers precision and recall. ${\mathrm{AP}}_{50}$, ${\mathrm{AP}}_{75}$, and ${\mathrm{AP}}_{\mathrm{s}}$ are the metrics calculated here, with calculations following the same process seen in the literature [42]. Specifically, AP₅₀ is the value of AP when the ratio of the intersection area of the detection bounding box and the real bounding box to the union area of the two boxes is greater than 50%. Similarly, ${\mathrm{AP}}_{75}$ is the value of AP when this ratio is greater than 75%. ${\mathrm{AP}}_{\mathrm{s}}$ is the AP when dealing with small-sized ship objects with the area of the bounding box below 32 × 32 pixels.

4.3. Results and Analysis

Table 2. Comparison of different methods on SSDD.

Method	Precision	Recall	F1	AP50
RetinaNet [43]	0.910	0.915	0.912	0.896
Cascade R-CNN [44]	0.940	0.899	0.919	0.893
SSD512 [18]	0.929	0.896	0.912	0.937
CenterNet [45]	0.943	0.945	0.944	0.935
CenterNet++ [46]	0.932	0.945	0.938	0.927
Baseline [16]	0.946	0.935	0.940	0.950
Proposal	0.953	0.949	0.951	0.957

,

Table 3. Comparison of different methods on SAR-Ship-Dataset.

Method	Precision	Recall	F1	AP50
RetinaNet [43]	0.881	0.938	0.909	0.938
Cascade R-CNN [44]	0.910	0.926	0.918	0.920
SSD512 [18]	0.901	0.905	0.903	0.942
CenterNet [45]	0.927	0.935	0.931	0.950
CenterNet++ [46]	0.925	0.934	0.929	0.949
Baseline [16]	0.941	0.949	0.945	0.948
Proposal	0.952	0.950	0.951	0.954

and

Table 4. Comparison of different methods on HRSID.

Method	Precision	Recall	F1	AP50
RetinaNet [43]	0.701	0.838	0.763	0.826
Cascade R-CNN [44]	0.899	0.794	0.843	0.793
SSD512 [18]	0.909	0.855	0.881	0.888
CenterNet [45]	0.818	0.874	0.845	0.863
CenterNet++ [46]	0.822	0.873	0.847	0.863
Baseline [16]	0.926	0.891	0.908	0.923
Proposal	0.931	0.902	0.916	0.936

show the performance of our proposed solution and other existing CNN methods on the datasets SSDD [9], SAR-Ship-Dataset [36], and HRSID [38], respectively.

SSDD. Our proposal can achieve the best results in all metrics, with a precision of 0.953, recall of 0.949, F1 of 0.951, and AP of 0.957. Compared to the single-stage approach RetinaNet [43], our proposal shows an improvement of approximately 6.1%. Even when compared with two-stage algorithms, such as Cascade R-CNN [44], the proposed method is approximately 6.4% better.

SAR-Ship-DataSet. As was the case with SSDD, our proposal shows the best performance, being, for example, approximately 3.4% better than Cascade R-CNN.

HRSID. Compared to the other datasets, HRSID has more complex backdrops and many pictures with tiny boats. These characteristics significantly degrade the performance of the other methods. However, the proposed solution is still showing the best performance, with an improvement of up to 14.3% compared with RetinaNet and 1.3% compared with Faster R-CNN, the second-best solution.

In summary, the mechanisms in our proposal to analyze semantic and spatial data gives it an edge that increases its accuracy and precision. The index fluctuation is also extremely small, highlighting the strong generalization of the proposal. The results in all datasets confirm that our method is highly competitive and consistent. The results on HRSID, in particular, show that the proposal is good at both multi-scale situations and the suppression of undesired background data.

To further understand what is behind the efficacy of our proposal, ablation experiments were performed where portions of our framework were removed to measure their impact. First, we compared the performance of our full proposal (MANet) and versions of it with just the MCAM and with just the MSAM. Baseline performance in the form of pure Faster R-CNN is also provided for reference. Results are shown in

Table 5. MCAM and MSAM ablation experiments on SSDD.

Method	MCAM	MSAM	AP50	AP75	APS
Baseline			0.950	0.823	0.683
MCAM	√		0.953	0.830	0.684
MSAM		√	0.956	0.848	0.693
Ours (MANet)	√	√	0.957	0.852	0.697

Note: MCAM represents multi-layer channel attention module. MSAM represents multi-layer spatial attention module. MANet stands for multi-layer attention network.

. First, it is noteworthy that full MANet is better than its counterparts with just MCAM and with just MSAM. For example, on SSDD, our full proposal sees improvements of greater than approximately 0.7%. Comparing the baseline and MCAM, the ${\mathrm{AP}}_{50}$ improves from 0.950 to 0.953, showing that incorporating MCAM and producing more accurate feature maps provides a small advantage. Comparing the baseline and MSAM, the ${\mathrm{AP}}_{50}$ increased to 0.956 and the ${\mathrm{AP}}_{\mathrm{s}}$ increased from 0.683 to 0.693. This indicates that the module can improve the identification of the object location, reduce the influence of complex backgrounds on the detection performance, and improve the detection performance of small ships.

Next, tests were performed on different architectures of the backbone network, with results being shown with and without the use of the MAM. The performance is shown in Figure 6 and proves that adding MAM increases the method’s overall recognition rate, regardless of the underlying network.

Figure 7 provides a visual illustration of the detection effects between the baseline (pure Faster R-CNN) and our proposed MANet. The real ship position (ground truth) is given by Figure 7a, highlighted by the green boxes. The detection outcome of the baseline is seen in Figure 7b. The ships highlighted in purple were successfully detected while the ships highlighted in red were missed. In Figure 7c, the ships identified by our proposal, MANet, are highlighted in yellow. This works as an example of how our proposal is capable of detecting small vessels and ships among complex backgrounds even when the conventional methods encounter difficulty and fail.

4.4. Limitations and Possible Improvements

As for the densely arranged and parallel-placed ships, we have found that the new method has some limitation in identifying the vessels. The detection of densely side-by-side ships in the nearshore scene is the focus of future research. It is also the focus of future research to detect the difference between ships and objects, such as small islands. The result is not included to save space. We conject the reason for this limitation is that the horizontal and vertical bounding boxes we use may not be close enough to the ships’ edges, especially when the ships do not appear horizontal or vertical in the image.

In addition, the vertical bounding box we use in this work would limit the detection performance, especially for the inshore scenario, which may contain buildings and other types of ships. Using an oriented bounding box can be a possible improvement of the proposed method. Along with the position information of the detected ships, the oriented bounding box can also provide the aspect ratio of the ship and the heading angle information, which greatly help the trajectory prediction and attitude determination.

Although our proposed method works well on the datasets, it is still an open problem to improve the detection precision of the small ships. The small ships in the images have fewer features than the large objects by nature. The multi-layer attention model we use in the work to extract the features of the object to be detected is one of the possible methods that can be adopted for small object detection. A modified multi-layer attention model or more advanced models will possibly appear and become feasible to enhance the detection performance for small ships in the future.

From the perspective of real-time application of the proposed method, we still have room for improvement. As of now, our main focus is to improve the detection precision. To realize real-time detection, we need to reduce the complexity of the network and/or implement the method on more powerful computational platforms. Considering the size, weight, and power constraint systems, such as miniature drones, small satellites, and unmanned aerial vehicles, developing low-complexity methods and small-scale models seem to be a more feasible way to realize real-time object detection.

5. Conclusions

In this paper, we proposed a multi-layer attention network (MANet) method as a solution to both the multi-scale and complex inshore background issues in SAR-based ship detection. The proposed new approach can simultaneously explore the rich semantic information of high-level features and the accurate location data of low-level features. Different from the conventional methods, we integrate an attention mechanism into the feature map of each layer to adaptively improve the weight of essential details depending on the importance of different scales and positions for that particular image. Finally, we conduct extensive performance evaluation based on multiple widely-accepted SAR datasets. Result shows that the proposed multi-layer attention mechanism network significantly increases the accuracy of ship detection compared to the existing conventional solutions.

In addition to ship detection from SAR images discussed in the paper, the proposed method can be utilized in many other applications. Based on the structures and procedures of the new method, we can extend its application to any network based on feature extraction for classification, detection, and segmentation. In the future, we will investigate the detection of densely side-by-side ships in the nearshore scene, explore these applications of our new method and conduct more experiments with various datasets. Other focuses of future research include detecting the difference between ships and objects, such as small islands, and improving the computational efficiency.