Fusing Context Features and Spatial Attention to Improve Object Detection

Abstract

Context features are mostly used to determine the boundary of a target, which allows one to better locate an object. In this paper, we propose the fusion of the spatial attention mechanism and contextual features to simulate the recognition of objects based on the human eye, thereby improving the detection accuracy of detectors. We chose the PASCAL VOC2007+2012 general dataset to test the generality of our method and examined the improved accuracy of our proposed detector on various targets. Our method showed improved accuracy for small targets and partially overlapping targets. Our proposed model improved the detector’s accuracy by 3.34%.

1. Introduction

With the development of object detection, the existing object detection algorithms have been classified into two-stage and single-stage algorithms. The representative model for two-stage algorithms is Faster RCNN [1], which generates a number of candidate regions and sends them into the network for object detection. The representative model for one-stage algorithms is YOLO. YOLO-v1 [2] partitions the input image into $S\times S$ grids and predicts N bounding boxes for each grid. YOLO-v2 to YOLO-v4 [3,4,5] generate the anchor box through the K-means clustering method and predict the offset of the bounding box relative to the prior box. YOLO-v5 preprocesses the input image using mosaic data enhancement, adaptive image zoom, and other mechanisms. YOLO-v5 can compute the best anchor box value for adaptation during training. YOLO-v6 [6] can adapt models of different sizes due to the design of the reparameterizable EfficientRep Backbone and RepPAN Neck. YOLO-v7 [7] proposed a high-aggregation network, heavily parameterized convolution, auxiliary head detection, and dynamic label allocation to achieve an optimal speed and accuracy for the detector. Two-stage object detection aims to enhance the ability to detect targets by selecting a candidate box with a high coincidence degree with the real box after adjusting the candidate box twice. Therefore, the detection accuracy of two-stage object detection is high, but the detection speed is low, and does not meet the real-time requirements. Single-stage target detection algorithms omit the process of generating candidate boxes and directly extract the image features to predict the classification and location of objects through the network, which saves time, but has a low detection accuracy. In recent years, one-stage object detection has been continually improved, surpassing two-stage object detection in terms of accuracy and speed, and has been widely used in industry. Currently, there are two main approaches to improving the accuracy of object detection algorithms: enhancing general-purpose algorithms and specifically targeting complex objects. For instance, [8,9,10,11,12] proposes more optimized algorithms for all objects, while [13,14] is tailored to design algorithms for low-precision complex objects. Most two-stage and one-stage object detection algorithms are based on anchor frames. A anchor frame is an a priori frame with different aspect ratios predefined in the object detection algorithm. In object detection, multiple anchor frames are first generated, and then the category and offset are predicted for each anchor frame. Next, the anchor frame is adjusted according to the predicted offset to obtain the predicted bounding box. Finally, the predicted bounding box needed by the output is screened. The size, aspect ratio, and number of anchor frames have an important influence on the performance of target detectors based on anchor boxes. When selecting the size of the anchor box, the aim is that the anchor box perfectly fits the size of the object itself. Therefore, YOLO-9000 [1] was proposed, which uses K-means [4] instead of the manual selection method to determine the average size of the object, so that the size of the anchor box fits the object more perfectly; however, as the anchor box is closer to the object, the loss of surrounding environment information is greater. Context features can better help us determine the boundaries of objects and better localize objects. For example, birds and airplanes are small colored or white objects in a blue sky. Comparing the blue background information with the white object information can better help determine the boundaries and object positions. At the same time, the context features can also help determine the category of the object. If the background is a blue sky, the target may be a bird or an airplane. If the background is a beach, the object may be a human or a vehicle. Context features can help us further narrow the range of information to ensure feature extraction. Adding context features [15] to target detectors can help improve the object detection accuracy. Meanwhile, context features cannot be used alone. Just like human eyes, if the object is as important as the background, then most of the attention may be paid to the relationship between the background and the object rather than the properties of the object itself, which is also an attention mechanism [16] source. Therefore, we ignore most of the background information and only fuse a small part of the background information and object information for object detection.

This paper proposes the following three contributions:

• Improving the YOLO-v3 model to better extract the features of small objects;
• Adding context features to improve the detection accuracy of overlapping objects;
• Integrating an attention mechanism to better detect difficult targets.

2. Related Works

Object detection algorithms based on deep learning are categorized into anchor and anchorless detection algorithms. An anchorless detection algorithm automatically learns to generate anchor boxes during network training according to the features of the dataset. CornerNet [10] outputs the prediction box by combining two sets of heat maps, where CNN predicts top left and bottom right points of the heat map. The works [10,17] located anchor boxes based on key points. FCOS predicts the upper, lower, left, and right offset of the center point to the object boundary and the category to which the center point belongs. YOLOx [18] predicts object locations by calculating the offset of the coordinates in the upper-left corner of the grid and the height and width of the prediction box. FoveaBox [19] predicts a category-sensitive language intention for each possible target object and generates bounding boxes for unknown categories at each possible location containing the target. The anchorage detection algorithm also needs to pair the key points, resulting in a decrease in the detection speed. If two targets overlap and share the same central point, anchor-free detection algorithms may detect only one of the targets, so there are missed and false tests.

The scale of the anchor frame is manually defined. By adjusting the setting of the anchor frame, it can cover as many target objects as possible, which improves the accuracy of target prediction. For target detection, the selection method of the anchor is an important factor that affects the accuracy and speed of the target detector. The target detector RCNN [1] uses deep learning for target detection, applying the “Select Search” method to select the anchor box, which relies on violent search, and sending about 2000 candidate boxes into the detector for detection; this improves the accuracy at the cost of considerably increasing the detection time. YOLO considers an image as a whole for detection, and it was the first method to design the anchor box selection process based on the idea of anchor-free detection, which greatly improved the detection speed. SSD [20] uses a fixed anchor box size for target detection and adds a feature pyramid to greatly improve the detection accuracy of the target, while YOLO-v2 [3] also uses the feature pyramid method and employs K-means to design the anchor box and achieve a higher detection accuracy. GA-RPN [21] positions the possible target center point and then generates the optimal anchor box according to the semantic features guided by the center point, which greatly improves the detection performance of the target detector.

However, YOLO-v2 still has its shortcomings, and the selected k may not be suitable if its value is uncertain regarding the clusters based on object partitions and clusters based on object size when the mean value is scattered; thus, the obtained anchor box may not be suitable for the corresponding detector. Therefore, we manually adjusted the anchor box based on K-means in order to improve the accuracy of the detector.

3. Attention Mechanism

The proposal of the attention mechanism itself was inspired by humans’ visual attention mechanisms for objects. We apply an attention mechanism to target detection so that certain important features of an object more significantly affect our recognition of that object. For example, the features of the long trunk and large ears of an elephant dominate our perception of the visual information of the elephant, and an attention mechanism is adopted to increase the weight of these features. Humans tend to focus more on recognized objects; while they try to ignore the background, they cannot completely dismiss it. For example, when we observe birds flying in the sky, we neither pay attention to how blue the sky is, nor recognize how many clouds there are, but the visual information we receive still tells us that the birds are flying in the sky, and thus the background information of the sky also has a certain impact on our recognition of the birds. We integrated such contextual information into the feature information of objects with a low weight in the form of spatial attention to improve the model’s detection of objects. For example, [22,23] incorporates attention mechanisms to enhance detection accuracy. This study adopted the attention mechanism SAM [16]. We let the detector imitate a human to detect objects. We fused the target features and context features extracted in the feature layer and then applied a spatial attention mechanism and sent them to the enhanced feature extraction network for enhanced feature extraction. The addition of a BiFPN module instead of an FPN (feature pyramid network) module could improve feature extraction, strengthening in particular the extraction of small target features, and the BiFPN module could also increase the fusion of the context features with the originally extracted features and better utilize the advantages of the context features.

4. Context Feature

The pixels in an image do not exist independently. There are relationships between neighboring pixels. A large number of pixels are related to each other to produce objects in an image. Therefore, context features refer to the relationships between pixels and surrounding pixels. Inspired by [24,25], we believe that context features contain context information that can fully utilize the constraints and dependencies between objects and their background, as well as between objects themselves, in contrast to the general features extracted by convolution. This improves the detection performance of the target detector.

The fusion of context features and an attention mechanism was used to detect small targets, and FA-SSD was proposed. In order to solve the problem of the low detection accuracy for small objects, low-level features and high-level features were fused in the attention mechanism. YOLO-v3 [4] uses FPN as the target extraction network, which introduces fewer features in the shallow layer, while it employs ResNet convolution to better extract feature information. We used YOLO-v3 as the foundation upon which to add context and spatial attention mechanisms, eliminating complicated information processing steps for the feature fusion of different layers. Moreover, we chose the same layer for context extraction to process less information and improve the detection speed of the detector.

5. Proposed Methods

5.1. Context Features

We introduced a multi-scale dilated convolution module to extract context features in the original network model, increased the receptive field while maintaining the resolution to make full use of context information, and applied a spatial attention mechanism after fusing the extracted context features and object features.

$$\begin{array}{cc}\hfill AM& =\sum _{i=1}^M\omega {}_{i}p{x}_i+\sum _{j=1}^N\omega {}_{j}p{x}_j(p{x}_i\in CF,p{x}_j\in FM)\hfill \\ & =\sum _{k=1}^N\omega {}_{k}p{x}_k,\hfill \end{array}$$

(1)

where $AM\in R^{H\times W\times C}$ represents the extracted attention map, $CF\in R^{H\times W\times C}$ the context features, $FM\in R^{H\times W\times C}$ the feature map, $p{x}_i$ the value of the i-th pixel, and N represents the number of pixels in the image. Each attention map we extracted was concatenated from the context features and the original feature map with the addition of a weight $\omega {}_k$.

First, we used the k-means method to determine the k average sizes of the objects and then divided each of the k average sizes into n average sizes. Here, we took n and k, respectively, to be 3 and 4. That is to say, we divided the sizes of the objects into 4 large average sizes and then divided each large average size into 3 small average sizes. We used 4 feature layers for feature extraction, and small objects were extracted in the shallow layer, while large objects were extracted in the deep layer; anchor boxes of 3 sizes were used to locate the objects on each grid point of each feature layer. The adoption of the feature-enhanced extraction FPN (feature pyramid network) could help avoid the problem of insufficient semantic information in shallow networks and their inability to detect small objects. At the same time, it could avoid the decrease in detection speed caused by fusing too many feature layers together. FPN could better integrate deeper semantic information and shallow information to increase the amount of feature extraction information and thereby improve detection accuracy. We extracted the feature information of the objects through ordinary convolution and then extracted the context feature information through multi-scale dilated convolution. The structure of the multi-scale dilated convolution process is shown in Figure 1. Multi-scale features were extracted using dilated convolutions with a convolution kernel size of $3\times 3$ and expansion rates of 1, 3, and 5, respectively, and multi-scale features were fused to obtain features containing context information. We sent the context feature information and the object feature information into the enhanced feature extraction network for further feature extraction. In Figure 2, the blue part represents the object information features, and the red parts represents the context features. The two sets of features were spliced according to the channel, and the spliced features were sent to the enhanced feature extraction network to obtain information-rich features.

5.2. Spatial Attention Mechanism

We used a spatial attention mechanism to extract target and context features from the network input for better target recognition. This mechanism assigned different weights to the features of the target itself and its surrounding context, thereby achieving the more precise extraction of the target.

$$\begin{array}{cc}\hfill MF& =\sigma (f^{3\times 3}\left(\right|AvgPool\left(F\right);MaxPool\left(F\right)\left|\right))\hfill \\ & =\sigma (f^{3\times 3}\left(\right|F_{avg};F_{max}\left|\right)),\hfill \end{array}$$

(2)

where $\sigma$ is the sigmoid function used to obtain the weight coefficient of 0 1; $f^{3\times 3}$ indicates that the convolution operation is performed with a $3\times 3$ convolution kernel; and $\left|AvgPool\right(F);MaxPool(F\left)\right|$ means that the average and maximum pooling operations are carried out on feature F, and the obtained features are spliced according to the channel dimension.

The SAM [26] module was used to implement our spatial attention mechanism. We sent the context features extracted from the feature extraction network to the SAM module. The structure of the SAM module is shown in Figure 3. Firstly, the maximum pooling operation and the average pooling operation were performed on the input features $F^{\prime }\in R^{C\times H\times W}$ to obtain two feature maps with a channel number of 1. The two feature maps were spliced to obtain a feature map with a channel number of 2. Then, convolution with a convolution kernel of $3\times 3$ was used to extract features, and a sigmoid function was used to calculate the spatial weight of the extracted features. We generated a spatial attention feature map $M\in R^{C\times H\times W}$. Finally, the spatial attention features M and features $F^{\prime }$ were multiplied pixel by pixel to highlight the target features of $F^{\prime }$. In order to gather as much semantic information as possible, the spatial attention mechanism was added to the blue module in Figure 4, the deep semantic information was sent to the SAM module and then fused with the shallow information, and the spatial attention mechanism was applied again to the fused features to better extract the feature information of the object.

5.3. Improved YOLO-v3

We took YOLO-v3 as the basis network and made certain modifications to it; then, we added context features and attention mechanism modules to the modified network to improve the detection accuracy. We added a feature extraction layer with dimensions of $104\times 104$ to improve the feature extraction for small target objects and attached this to the enhanced feature extraction network and the feature layer that had been compressed several times for splicing. YOLO-v3 uses the enhanced feature extraction FPN to extract the features of objects, that is to say, the width and height were compressed to $13\times 13$ after the continuous convolution from top to bottom through the backbone extraction network. Then, we convolved the feature layer with a width and height of $13\times 13$ from bottom to top, expanded the width and height, and performed the convolution with other feature maps after splicing. An SAM module was applied between the convolutional layer where deep convolution and shallow convolution were used to performed information fusion and the previous convolutional layer, and the target features and contextual features were extracted in the last convolutional layer.

$$\begin{array}{cc}\hfill p_7^{out}=& SAM\left(p_7^{in}\right)\hfill \\ \hfill p_6^{out}=& SAM(upsample\left(p_7^{out}\right)+p_6^{\mathrm{in}})\hfill \\ \hfill p_5^{out}=& SAM(upsample\left(p_6^{out}\right)+p_5^{\mathrm{in}})\hfill \\ \hfill p_4^{out}=& SAM(upsample\left(p_5^{out}\right)+p_4^{\mathrm{in}}),\hfill \end{array}$$

(3)

where $p_7^{out}$ is the input of the 7th convolution layer of the backbone network that is added to the SAM for output; $p_6^{out}$ upsamples the output of $p_7$ and then fuses it with the input of $p_6$ and adds it to the SAM for output; $p_5^{out}$ upsamples the output of $p_6$ and then fuses it with the input of $p_5$ and adds it to the SAM for output; and, similarly, $p_4^{out}$ upsamples the output of $p_5$ and then fuses it with the input of $p_4$ and adds it to the SAM for the output.

YOLO-v3 uses ResNet-53 as the backbone feature extraction network. We modified the residual block module as the basis for the inverted residual block, which reduced the loss of information since the residual network was compressed and then expanded, and a linear activation function was used with low dimensions to prevent features from being destroyed. Depthwise separable convolution was used as the convolution method to reduce the operation parameters as much as possible and improve the detection speed of the model. We used (5) as the loss function for our detection target as a whole, where the confidence loss was calculated using the softmax function, andthe loss for predicted box l and ground truth box g was calculated using a smooth function. Additionally, we used the offset of the regression center coordinates ($c_x$, $c_y$) to the height (h) and width (w) of the border for positioning.

$$\begin{array}{cc}\hfill L\left(x,c,\mathrm{l},g\right)=& \frac{1}{N}\left(L_{conf}(x,c)\right)+\alpha L_{loc}(x,l,g)\hfill \end{array}$$

(4)

where

$$\begin{array}{cc}\hfill L_{conf}(x,c)=& -\sum _{i\in Pos}^N{x}_{ij}^p{log}_{}\left(\widehat{c_j^p}\right)-\sum _{i\in Neg}^N{x}_{ij}^p{log}_{}\left(\widehat{c_j^p}\right)\hfill \\ \hfill \widehat{c_j^p}=& \frac{exp\left(c_j^p\right)}{{\displaystyle \sum _p}exp\left(c_i^p\right)}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill L_{loc}(x,l,g)=& \sum _{i\in Pos,m\in \{cx,cy,w,h\}}^N\sum x_{ij}^{k}smoot{h}_{L1}(l_i^m-\widehat{{\mathrm{g}}_j^m})\hfill \end{array}$$

(6)

6. Performance

6.1. Evaluation Metrics

Generally speaking, precision (P) and recall (R) are two contradictory measurement standards. If one wants to improve P, R decreases, and vice versa. The relationship between the P and R of the same algorithm can be represented by the p(r) curve, shown in Figure 5. Ideally, one hopes that both P and R approach 1 infinitely and, therefore, that the area under the curve approaches 1 infinitely, which is referred to as the average precision (AP) of the detection algorithm. In object detection, the mainstream evaluation metric is mean average precision (mAP), as it simultaneously considers the precision and recall of a model and evaluates the performance of the model more comprehensively.

Evaluation metrics are utilized to assess the performance of a model. In

Table 1. A brief introduction to TP, FP, TN, and FN.

Detection Result	Abbreviation	Definition
true positive	TP	correctly predicted positive samples
false positive	FP	incorrectly predicted positive samples
true negative	TN	correctly predicted negative samples
false negative	FN	incorrectly predicted negative samples

, we provide a brief introduction to the four types of detection results (TP, FP, TN and FN). and the commonly used metrics are presented in

Table 2. Commonly used model evaluation metrics.

Evaluation Metric	Abbreviation	Computing Method	Definition
Accuracy	A	$A=\frac{TP+TN}{TP+TN+FP+FN}$	The proportion of correct predictions made by the model among all predictions.
Precision	P	$P=\frac{TP}{TP+FP}$	The proportion of true-positive samples among all samples predicted as positive by the model.
Recall	R	$R=\frac{TP}{TP+FN}$	The proportion of samples correctly predicted by the model among all positive samples.
Average precision	AP	$AP={\int }_0^{1}P\left(r\right)\mathrm{d}r$	Area under the precision–recall curve.
Mean average precision	mAP	$mAP=\frac{1}{k}{\sum }_1^{k}A{P}_i$	The average of AP for all classes.
Frames per second	FPS	-	The number of pictures that can be processed per second.

. In this paper, we chose P, R, mAP, and FPS as the evaluation metrics.

6.2. Experimental Settings

Dataset: We tested our model on PASCAL VOC2007 and VOC2012, and we combined the two datasets as voc2007+2012 to test our model. The datasets were partitioned into a training set, validation set, and test set with a ratio of 7:1.5:1.5. The VOC dataset comprised a total of 20 categories, with VOC2007 containing a total of 9963 pictures and VOC2012 11,540 pictures. We combined these two datasets to reach a total of 21,503 pictures as our final dataset, and, according to the proportions mentioned above, we separated the validation and test sets to train and test the model.

Implementation details: We chose python to implement our network model, employed the pytorch framework to build the model, and used RTX2080 on the server for testing. In order to speed up training, that is, to speed up the convergence of the model, we partitioned the training period into a freezing period and a thawing period. We trained our model for 50 freeze epochs and 30 thaw epochs depending on how well the model converged. According to the computing power of the server, we set the batch size of the freeze period to eight, the batch size of the thaw period to four, and the IOU to 0.5 based on the general standard.

6.3. Experiments

We combined VOC 2007 with VOC 2012 data to test our model. As observed from

Table 3. The experimental results of all models.

Method	Backbone	Input Size	Precision	Recall	mAP	FPS
Faster RCNN	ResNet-101	600 × 600	77.10	64.63	73.41	10.31
FCOS	Normal Convolution	600 × 600	76.38	67.06	74.53	14.53
SSD512	VGG16	512 × 512	84.15	67.58	76.80	17.21
CenterNet	Hourglass-104 Convolution	600 × 600	85.88	70.11	80.23	12.80
YOLO-v3	ResNet53	412 × 412	90.26	73.83	82.34	16.95
Our model	ResNet53+	412 × 412	91.67	76.97	85.68	19.89

, compared with the traditional two-stage detector, our model had a faster detection speed and higher accuracy, as we used a one-stage detector based on YOLO-v3 that eliminated the stage of generating anchor boxes and improved the detection speed. The accuracy of our proposed model outperformed that of one-stage detectors such as SSD and YOLO-v3. Moreover, because depthwise separable convolution was used, the number of unnecessary parameters was reduced, and the detection speed was improved. We also compared the model with the CenterNet model, which adopts an anchor-free design, and our proposed model again had a higher detection accuracy.Compared with other models, we also achieved a higher R and P. The R and P values of our model for each target class are shown in

Table 4. The R and P values for all classes.

Class	Airplane	Bicycle	Bird	Boat	Bottle	Bus	Car	Cat	Chair	Cow
R	83.33	85.92	73.17	61.47	54.09	83.06	82.17	83.61	59.37	53.16
P	99.01	96.83	96.77	95.71	85.00	100	93.81	95.03	79.74	58.33
Class	Dining Table	Dog	Horse	Motorbike	Person	Potted Plant	Sheep	Sofa	Train	TV Monitor
R	63.89	88.70	86.47	86.05	82.98	40.44	28.42	76.47	83.47	80.45
P	80.23	87.80	95.83	97.37	93.86	88.10	100	85.71	91.82	91.45

, which indicates that with sufficient training data, our model produced good predictive results.

6.4. Ablation Study

In

Table 5. Results of ablation study.

ResNet53+	Depthwise Separable Convolution	Context feature	SAM	mAP	FPS
✓	×	×	×	82.34	16.95
✓	✓	×	×	82.13	22.01
✓	✓	✓	×	83.73	20.60
✓	✓	✓	✓	85.68	19.89

where ✓ represents the addition of the corresponding module, while × represents the absence of the corresponding module.

, we present the results of our ablation experiments which were conducted to determine the impact of our modifications on each model. We combined the two datasets VOC2007 and VOC2012 to test our model. With ResNet53 as the backbone, we replaced the ordinary convolution step with depthwise separable convolution, which reduced the number of parameters and improved the detection speed of the model. The detection accuracy of the model only decreased slightly, and then the context features were added. It can be seen that our model presented only slight improvements, though its accuracy was greatly improved after adding the attention mechanism. Furthermore, only the combination of the attention mechanism and context features achieved better detection results.

6.5. Example Visualization of Experimental Results

We compared YOLO-v3 with our proposed target detection algorithm. The comparison results are shown in Figure 6. We found that the network with the attention mechanism and context feature extraction module better detected the targets. Our algorithm improved the accuracy of detecting small target objects. For certain large targets and small targets, some overlaps were also better detected.

7. Conclusions

In order to further improve the accuracy conferred by the context detection mechanism, we proposed the combination of the context mechanism with a spatial attention mechanism to take advantage of the former. We tested our context feature module on the basis of the YOLO-v3 model. After adding context features, the detection accuracy of our model improved to a certain extent, increasing from 82.34% to 83.73%. We assigned the detection target to not only the target itself, but also to the background surrounding the target, which produced a better target detector and improved our model’s detection accuracy (mAP) to 85.68%. It can be seen that the proposed addition of context and spatial attention mechanism modules improved the accuracy of the detector.