A YOLO-Based Target Detection Model for Offshore Unmanned Aerial Vehicle Data

Abstract

Target detection in offshore unmanned aerial vehicle data is still a challenge due to the complex characteristics of targets, such as multi-sizes, alterable orientation, and complex backgrounds. Herein, a YOLO-based detection model (YOLO-D) was proposed for target detection in offshore unmanned aerial vehicle data. Based on the YOLOv3 network, the residual module was improved by establishing dense connections and adding a dual-attention mechanism (CBAM) to enhance the use of features and global information. Then, the loss function of the YOLO-D model was added to the weight coefficients to increase detection accuracy for small-size targets. Finally, the feature pyramid network (FPN) was replaced by the secondary recursive feature pyramid network to reduce the impacts of a complicated environment. Taking the car, boat, and deposit near the coastline as the targets, the proposed YOLO-D model was compared against other models, including the faster R-CNN, SSD, YOLOv3, and YOLOv5, to evaluate its detection performance. The results showed that the evaluation metrics of the YOLO-D model, including precision (Pr), recall (Re), average precision (AP), and the mean of average precision (mAP), had the highest values. The mAP of the YOLO-D model increased by 37.95%, 39.44%, 28.46%, and 5.08% compared to the faster R-CNN, SSD, YOLOv3, and YOLOv5, respectively. The AP of the car, boat, and deposit reached 96.24%, 93.70%, and 96.79% respectively. Moreover, the YOLO-D model had a higher detection accuracy than other models, especially in the detection of small-size targets. Collectively, the proposed YOLO-D model is a suitable model for target detection in offshore unmanned aerial vehicle data.

1. Introduction

Unmanned aerial vehicle (UAV) imagery shows great potential for offshore monitoring due to real-time collection of temporal/spatial data [1]. Different targets can be identified by the detection models in UAV data. However, it is still unavailable for an automatic target detection model due to the multi-sizes, alterable orientation, and complex backgrounds of the target objects. Currently, deep learning has been widely used for extracting features and detecting targets. Deep learning has great potential for improving the accuracy and efficiency of target detection in offshore unmanned aerial vehicle data.

In general, target detection based on deep learning can be divided into two major types, a two-stage detection model and a one-stage detection model [2]. In a two-stage target detection model, different targets are detected based on the series of candidate boxes [3]. The model based on a region with a CNN feature (RCNN) is a typical two-stage target detection model, showing great advantage in detection accuracy and positioning accuracy [4,5,6,7]. In a one-stage target detection model, different targets are directly detected, where target detection is abstracted as a regression problem [8]. Models based on a you-only-look-once (YOLO) or a single-shot multibox detector (SSD) are typical one-stage target detection models, showing great advantage in detection efficiency [9,10,11,12,13]. In addition, there are some detection methods deigned based on machine learning, such as sparse target detection [14,15,16], sub-pixel target detection [17,18], and visual saliency target detection [19,20]. In recent years, there also have been efforts to apply and improve these target detection models in UAVs for offshore monitoring [21,22,23,24,25,26]. However, these models are still not the optimal option for target detection in offshore unmanned aerial vehicle data, due to the complex characteristics of targets, such as multi-sizes, alterable orientation, and complex backgrounds.

In this study, we proposed a target detection model for target detection in offshore unmanned aerial vehicle data based on the improved YOLOv3 (YOLO-D) network.

The residual module was improved to enhance the use of features and global information. The loss function of YOLO-D was then improved to enhance the detection accuracy for small targets. The feature pyramid network (FPN) was finally replaced by the secondary recursive feature pyramid network to reduce the impacts of a complicated environment.

2. Dataset and Model

2.1. Dataset

UAV data were acquired from 2019 to 2020, which were collected over Jinshan District, Fengxian District, and Pudong New District in Shanghai (China), and 504 images with a size of 1920 × 1080 pixels were extracted. The labeling software LabelImg v1.8.5 was used to label the offshore monitoring targets, including a car, a boat, and a deposit (Figure 1).

The images were preprocessed to balance the ratio of positive and negative samples and strengthen the learning efficiency of small-size targets. The images were then augmented by rotating, trimming, horizontal flipping, and splicing. The expanded dataset contained 1010 images, included 12,747 marked cars, 1247 marked boats, and 1431 marked deposits. According to the standard of the COCO dataset, a target with pixels smaller than 32 × 32 was classified as a small-size target. A target with pixels greater than 96 × 96 was classified as a large-size target [27]. Here, the number of small-size targets exceeded half of the total number of targets.

2.2. Model

Figure 2 shows the flowchart of the YOLO-based detection model (YOLO-D), including the improved backbone network, the improved feature pyramid network (FPN), and improved CCR modules.

2.2.1. Backbone Network of the YOLO-D Model

Based on Darknet-53, the backbone network of the YOLO-D model was improved by establishing dense connections and adding a dual-attention mechanism (CBAM). Figure 3 shows the mechanism of dual attention, including channel and spatial attention, which not only considered the importance of different feature channels but also considered the importance of different positions of the same feature channel [28].

In the YOLO-D model, dual attention (CBAM) was added into the residual module as the RC module (the main components of the CCR module) to enhance feature extraction (Figure 4).

Dense connections between CCR modules were established to strengthen the transmission and use of features and improve the use of features (Figure 5).

X_i represents the output of the i-th layer. X_n = RC(CBL(CBL([X₀, X₁, X₂, …, X_n−1]))), and [X₀, X₁, X₂, …, X_n−1] represents the splicing of the output features from layer 0 to layer n−1. CBL(x) represents the passing x through the CBL module. RC(x) represents the passing x through the RC module. The CBL module (Figure 2) includes a convolutional layer, a regularization layer, and an activation layer. The RC module (Figure 4) is a residual module that includes two CBL modules and a CBAM module [29].

2.2.2. Feature Pyramid Network of the YOLO-D Model

A feature pyramid network (FPN) can make use of the feature information about the bottom layer and the high layer at the same time and construct multi-size feature images [30]. Here, the FPN of YOLOv3 was improved by building a secondary recursive feature pyramid. The first-output features ($f_n^{out}$) of the FPN were concatenated with the first-input features ($f_n^{in}$) of the backbone network. Then, the first-output features passed through the atrous spatial pyramid pooling (ASPP) [31] as the second-input features ($f_n^{}$) of the backbone network, ($f_n=ASPP(concat(f_n^{in},f_n^{out}))$). Finally, the second-output features of the feature pyramid network were used as the final features and were outputted to the detection layer. Figure 6 shows the network structure of the YOLO-D model, including the structure of ASPP (Figure 6A), the second recursive model of the YOLO-D model (Figure 6B), and the network structure of the YOLO-D model (Figure 6C).

2.2.3. Loss Function of the YOLO-D Model

The loss function of YOLOv3 is the cross-entropy loss function, which is composed of center coordinate loss, confidence loss, and classification loss. In the YOLO-D model, the center coordinate loss function was replaced by the GIOU loss function [32] and the confidence and classification loss functions were improved by adding weight coefficients [33], which enhanced the learning efficiency of confusing samples and difficult samples.

The YOLO-D loss function is defined as:

$$Loss =L_{GIOU}+L_{Conf\_fl}+L_{Class\_fl}$$

(1)

$L_{GIOU}$ is the GIOU loss function, defined as:

$$L_{GIOU}=1-GIOU$$

(2)

$$GIOU=IOU-\frac{|C-(A\cup B)|}{\left|C\right|}$$

(3)

$$IOU=\frac{|A\cap B|}{|A\cup B|}$$

(4)

A is the true frame. B is the predicted frame.$A\cup B$ is the area of the union of A and B. $A\cap B$ is the area of the intersection of A and B. C is the area of the smallest bounding box including A and B.

The focal loss function $L_{focal}$ is defined as [27]:

$$L_{focal}=\left\{\begin{array}{c}-\alpha {(1-y^{\prime })}^{\gamma }log{y}^{\prime },\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}y=1\\ -(1-\alpha ){y^{\prime }}^{\gamma }log(1-y^{\prime }),\hspace{1em}\hspace{1em}y=0\end{array}\right.$$

(5)

$\alpha \in (0,1]$ and $\gamma \in (0,2]$ are self-defined constants. $y^{\prime }$ is the predicted output.$y$ is the label of the real sample. When $\gamma >1$, the network focuses on samples with learning difficulties. Weight coefficients are added to the original confidence loss. The new confidence loss function $L_{Conf\_fl}$ is defined as:

$$\begin{gathered} L_{Conf\_fl}={(|y-y^{\prime }|)}^{\gamma }\times ({\displaystyle \sum _{i=0}^{s^2}{\displaystyle \sum _{j=0}^B{I}_{ij}^{obj}}}[{\widehat{C}}_i^{j}log(C_i^j)+(1-{\widehat{C}}_i^j)log(1-C_i^j)] \\ +{\lambda }_{noobj}{\displaystyle \sum _{i=0}^{s^2}{\displaystyle \sum _{j=0}^B{I}_{ij}^{noobj}}}[{\widehat{C}}_i^{j}log(C_i^j)+(1-{\widehat{C}}_i^j)log(1-C_i^j)]) \end{gathered}$$

(6)

If the predicted box contains an object, $I_{ij}^{obj}$ = 1, $I_{ij}^{noobj}$ = 0, otherwise $I_{ij}^{obj}$ = 0, $I_{ij}^{noobj}$ = 1. $C_i^j$ is the predicted value. ${\widehat{C}}_i^j$ is the true value. If the predicted box is responsible for predicting an object, ${\widehat{C}}_i^j$ = 1, otherwise ${\widehat{C}}_i^j$ = 0. $y^{\prime }$ is the predicted output. $y$ is the label of the real sample. We set $\gamma =2$. If one sample is difficult to detect, $y^{\prime }$ will tend to 0 and the confidence loss value will increase.

$L_{Class\_fl}$ is the classification loss function, which is defined as:

$$L_{Class\_fl}=|a+y-1|\times ({\displaystyle \sum _{i=0}^{s^2}{I}_{ij}^{obj}{\displaystyle \sum _{cÎclass}^{}[{\widehat{p}}_i(c)\log (p_i(c))+(1-{\widehat{p}}_i(c))\log (1-p_i(c))]}})$$

(7)

${\widehat{p}}_i(c)$ is the probability that the anchor is predicted to be class c. $p_i(c)$ is the true value. If the predicted box contains an object and is difficult to be classified, the correct classification loss will be smaller and the wrong classification loss will be greater.

3. Experiment

To evaluate the detection performance of the YOLO-D model, two comparison experiments were conducted. In the ablative experiments, the improved efficiency of the backbone network, feature pyramid network, and loss function was analyzed. The detection performance of the YOLO-D model was evaluated by comparing it with other end-to-end models, including the faster R-CNN [7], SSD [12], YOLOv3 [11], and YOLOv5.

3.1. Experiment Metrics

Five different metrics, that is, average Precision (AP), the mean of average precision (mAP), precision (Pr), recall (Re), and frame per second (FPS), were calculated to estimate the target detection performance [34]:

$$AP={\displaystyle {\int }_0^{1}P(r)}dr$$

(8)

$$mAP=\frac{1}{N}{\displaystyle \sum _{i=1}^{N}A{P}_i}$$

(9)

$$Pr=\frac{TP}{TP+FP}$$

(10)

$$Re=\frac{TP}{TP+FN}$$

(11)

$$FPS=\frac{1}{t}$$

(12)

where the true positive (TP) represents the number of positive samples that are predicted to be positive, the false positive (FP) represents the number of samples that are predicted to be positive but are actually negative, and the false negative (FN) represents the number of samples predicted to be negative but actually positive. FPS represents the number of pictures that can be processed per second, and t represents the time required to process a picture.

The experiment was based on the tensor flow deep learning framework, and we built a virtual environment of Python 3.6 and TensorFlow-gpu 2.0 on Anaconda. The training and accuracy tests were carried out on a Ubuntu 16.04.4 system, NVIDIA Tesla P100 16 GB graphics card, and CUDA 10.0. The intersection over union (IOU) threshold was set as 0.3 and the score threshold as 0.45.

3.2. Results

Figure 7 showed the images and detection results obtained by the YOLO-D model. Taking Re and Pr as the abscissa and the ordinate axis, respectively, the P-R curve of each class is shown in Figure 8.

3.2.1. Ablation Experiment

Table 1. Comparison of evaluation metrics in ablation studies.

Improved Backbone Network	Improved FPN	Improved Loss Function	AP_Boat	AP_Car	AP_Deposit	mAP@0.5	FPS
√	√	—	83.39	91.75	91.34	88.83	3
√	—	√	91.98	94.10	92.84	92.97	5
—	√	√	80.02	94.99	94.52	89.84	4.5
√	√	√	93.70	96.24	96.79	95.58	3

shows the performance comparison between ablation studies, including YOLOv3 with an improved backbone, YOLOv3 with an improved FPN, and YOLOv3 with an improved loss function. Based on Table 1, we can see that all designs of the YOLO-D model could prominently enhance the AP and mAP of each class target. In the YOLO-D model, the AP of the boat, car, and deposit reached 93.7%, 96.24%, and 96.79%, respectively. Notably, the mAP reached 95.58%, although the speed increase of the YOLO-D model was not obvious.

3.2.2. Comparison with Other Models

The detection performance of the YOLO-D model was further compared with other models, including the faster R-CNN, SSD, YOLOv3, and YOLOv5. As shown in Figure 9 and

Table 2. Comparison of evaluation metrics using different models.

Model	Pr	Re	AP_Boat	AP_Car	AP_Deposit	mAP@0.5	FPS
Faster R-CNN	79.97	39.61	60.81	55.34	56.74	57.63	0.35
SSD	91.44	16.32	68.14	52.11	47.25	55.84	0.38
YOLOv3	90.70	55.08	60.34	78.67	63.48	67.50	9
YOLOv5	92.20	87.40	93.10	88.70	89.90	90.50	14
YOLO-D	92.70	92.06	93.70	96.24	96.79	95.58	3

, the YOLO-D model had the highest values of Pr, Re, AP, and mAP. The mAP value of the YOLO-D model increased by 37.95%, 39.44%, 28.46%, and 5.08% compared to the faster R-CNN, SSD, YOLOv3, and YOLOv5, respectively. Importantly, the AP value of the car reached 96.24%. Notably, most of car the targets were small-size targets. The detection speed of YOLO-D was 10 times faster than that of the faster R-CNN and SSD but was slightly slower than that of YOLOv3 and YOLOv5. Collectively, the YOLO-D model reduced the rate of false detection and showed great potential for accurate detection, especially for the detection of small-size targets.

4. Discussion

Unmanned aerial vehicle (UAV) obtains increased real-time datasets for offshore monitoring. However, the automatic detection of UAV data is still a tricky problem due to the multi-sizes, alterable orientation, and complex backgrounds of the target objects. It is particularly difficult to detect small-size targets. In this study, a YOLO-based target detection model (YOLO-D) was proposed for offshore unmanned aerial vehicle data.

Compared with other detection models, such as the faster R-CNN, SSD, YOLOv3, and YOLOv5, the proposed YOLO-D model can significantly enhance detection accuracy. The evaluation metrics of the YOLO-D model, including precision (Pr), recall (Re), average precision (AP), and the mean of average precision (mAP), had the highest score. The mAP value of detection targets increased by 37.95%, 39.44%, 28.46%, and 5.08% compared to the faster R-CNN, SSD, YOLOv3, and YOLOv5, respectively. The result suggested that the YOLO-D model has great potential for accurate detection of offshore UAV data.

In addition, the YOLO-D model can efficiently and accurately detect targets in offshore UAV data. However, there are still some limitations. The YOLO-D model is designed based on the spatial information in offshore UAV data. However, it ignores the contextual information, which is also important for target detection. The YOLO-D model shows great potential for accurate detection, but the speed is slightly slower than YOLOv3 and YOLOv5. In the future, we will further improve the YOLO-D model to enhance the accuracy and efficiency of target detection in offshore UAV data.

Taken together, this study proposed a YOLO-D model for the detection of offshore UAV data. In this model, the residual module is improved by establishing dense connections and adding a dual-attention mechanism (CBAM), which can enhance the use of features and global information. The loss function of YOLO-D is improved by adding weight coefficients, which can enhance the detection accuracy for small targets. The feature pyramid network (FPN) is replaced by a secondary recursive feature pyramid network to reduce the impacts of a complicated environment. The YOLO-D model shows great potential for accurate detection of offshore UAV data, especially for the detection of small-size targets.