A Novel Crop Pest Detection Model Based on YOLOv5

Abstract

The damage caused by pests to crops results in reduced crop yield and compromised quality. Accurate and timely pest detection plays a crucial role in helping farmers to defend against and control pests. In this paper, a novel crop pest detection model named YOLOv5s-pest is proposed. Firstly, we design a hybrid spatial pyramid pooling fast (HSPPF) module, which enhances the model’s capability to capture multi-scale receptive field information. Secondly, we design a new convolutional block attention module (NCBAM) that highlights key features, suppresses redundant features, and improves detection precision. Thirdly, the recursive gated convolution ($g^{3}Conv$) is introduced into the neck, which extends the potential of self-attention mechanism to explore feature representation to arbitrary-order space, enhances model capacity and detection capability. Finally, we replace the non-maximum suppression (NMS) in the post-processing part with Soft-NMS, which improves the missed problem of detection in crowded and dense scenes. The experimental results show that the mAP@0.5 (mean average precision at intersection over union (IoU) threshold of 0.5) of YOLOv5s-pest achieves 92.5% and the mAP@0.5:0.95 (mean average precision from IoU 0.5 to 0.95) achieves 72.6% on the IP16. Furthermore, we also validate our proposed method on other datasets, and the outcomes indicate that YOLOv5s-pest is also effective in other detection tasks.

1. Introduction

The frequent infestations of pests pose a substantial threat to agricultural productivity, leading to considerable economic losses [1]. Consequently, identifying effective strategies to combat pest and disease issues, enhance crop yields, and minimize economic losses in agriculture has emerged as a paramount concern [2，3，4]. In the past, pest detection tasks required agricultural experts to work in the field. However, the large agricultural planting area and diverse pest species led to problems such as labor-intensive, time-consuming, and insufficient agricultural experts [5,6]. As a result, there was an urgent need to research pest detection technologies to meet the growing demands of the market. However, pest detection based on machine learning (ML) required feature design, which was not only inefficient, cumbersome, and error-prone, but also extremely susceptible to background and light [7]. Therefore, the use of ML techniques to achieve accurate and robust pest detection continued to face significant challenges.

In recent years, convolutional neural networks (CNNs) of deep learning (DL), have brought new solutions to these challenges and have attracted significant attention [8]. Several excellent classical object detection models based on CNN have been proposed by researchers, including two-stage models: RCNN [9], Fast-RCNN [10], Faster-RCNN [11], Mask-RCNN [12] and one-stage models: you only look once (YOLO) series [13,14,15,16,17], single shot multibox detector (SSD) [18] etc. These models greatly improve feature extraction ability, detection speed, and precision. However, due to the unique characteristics of samples and different practical application requirements, existing models may perform poorly in certain scenarios, making the inspection task still challenging.

In order to enhance the overall performance of detection models, domestic and foreign scholars have improved existing models or proposed new models. To achieve real-time monitoring of orchard pests, Zhang et al. [19] compared the placement of ghost modules in different locations. The mean average precision (mAP) of their proposed method was 1.5% higher than YOLOv5 and required fewer parameters and GFLOPs. To realize the rapid lightweight of the passion fruit pest detection model, Li et al. [20] added a convolutional block attention module (CBAM) to the YOLOv5 model’s neck to enhance target object detection and used a point-line distance intersection over union (PLDIoU) loss function to reduce calculation and detection time. To expand the training set and prevent overfitting, they applied a hybrid online data augmentation algorithm. The mAP of their model achieved 1.59% higher than YOLOv5 while retaining its lightweight and real-time capabilities. To achieve precise detection of unopened bolls, Zhang et al. [21] incorporated the squeeze and excitation (SE) attention model into the end of the cross-stage partial (CSP) module and modified the residual block on CSP to a dense block structure inspired by DenseNet. They replaced the path aggregation network (PAN) module with a bidirectional feature pyramid network (BiFPN) to achieve advanced feature fusion performance. Their experiments showed that their proposed method is superior in terms of detection precision, speed, model size, and computational cost compared with YOLOv3, SSD, Faster-RCNN, and YOLOv5. To address the issues of low efficiency and unreliable cotton pest detection, Zhang et al. [22] used efficient channel attention (ECA) to mitigate the impact of complex backgrounds, adopted the focal loss function in YOLOX to address precision loss caused by sample imbalance, and utilized the hard swish activation function to achieve better performance. The experimental findings demonstrated that their model’s mAP increased by 11.5%, 21.17%, 9.34%, 10.22%, and 8.33% compared to Faster R-CNN, SSD, YOLOv3, YOLOv4, and YOLOv5, respectively. Xiang et al. [23] addressed the issue of feature extraction in small pest learning by introducing the CAC3 module into the backbone network. They enhanced feature extraction in the Neck with the ConvNext module. The mAP@0.5 of their proposal achieved 91.9% on the teddy cup pest dataset, representing an almost 8% improvement over the original model. Yang et al. [24] replaced part of the backbone network with CSPResNeXt and the ELAN-W module with the VoVGSCSP module. This modification reduced model parameters and computational workload while maintaining the original precision and improving detection speed on a maize pest dataset. Chen et al. [25] introduced a refinement method for pest detection based on a two-stage detection framework. They proposed the context feature enhancement module, the region of interest (RoI) feature fusion module, and the non-task separation module to optimize the two-stage algorithm. This approach obtained features at various scales, adaptively weighted and fused pest features, and decoupled the features of the classification and localization networks, further enhancing pest detection precision. The mAP of their proposed model achieved 72.7% on the SimilarPest5 dataset, outperforming other state-of-the-art object detection methods. Dai et al. [26] incorporated the Swin transformer (SWinTR) and transformer (C3TR) mechanisms into the backbone network of YOLOv5m to increase the receptive field and capture more global features. They replaced spatial pyramid pooling (SPP) with ResSPP to extract more features. Finally, they replaced the three C3 outputs in the neck with SWinTR and concatenate (Concat) operation with WConcat to enhance the network’s feature fusion capabilities. Experimental results showed that the mAP of the proposed model achieved 96.4% on the pest dataset. Moreover, it outperformed the original YOLOv3, YOLOv4, and YOLOv5m models.

From the above literature, object detection plays a crucial role in the prevention and control of plant diseases and pests. It enables the identification and localization of pests on crops, facilitating prompt implementation of appropriate control measures. However, while classic models have been rigorously trained, validated, and tested on datasets like Coco and VOC, their efficacy might be suboptimal when applied to agricultural pest scenarios. To address this issue, we create the IP16 dataset from the IP102 dataset specifically for agricultural pest detection. This dataset encompasses a series of challenges, including variations in target sizes and complex background noise interference. In order to help farmers achieve accurate and efficient pest detection, we propose a high-performance YOLOv5s-pest model. Our method focuses on the enhancement of the model’s feature extraction capabilities, thereby improving the model’s performance in pest detection tasks. The contributions of this paper are as follows:

We design the hybrid spatial pyramid pooling fast (HSPPF) module, replacing the SPPF module within the end of YOLOv5 backbone, enhancing the backbone’s ability to extract pest information.

Our innovation, the C3NCBAM module, replaces the conventional C3 module, amplifying the model’s feature extraction capability without incurring a significant surge in parameters or computational demands.

Leveraging the $g^{3}Conv$ module, we optimize certain Conv modules in the model’s neck, augmenting the model’s modeling capability and refining its pest detection precision.

We employ the soft non-maximum suppression (Soft-NMS) to replace the NMS to minimize the missed detection rate of pests.

Furthermore, we also create the IP16 dataset to validating our proposed method, a resourceful repository tailored for training and validating pest detection algorithms. From the experimental results, it can be concluded that our approach, YOLOv5s-pest, achieves the best balance between precision and computational complexity.

For the convenience of the readers, the subsequent sections of this paper unfold as follows: Section 2 provides a comprehensive exposition of the materials and methods employed in our study. Section 3 is results and discussion, where we present the findings from our experimental analyses and engage in a detailed discourse. Section 4 is our conclusion.

2. Materials and Methods

2.1. Data Acquisition

Currently, there is a limited availability of publicly accessible datasets for pest detection. In this paper, we created a pest dataset named IP16, we select 14 classes of pests from the IP102 [27] dataset. These pests have complex backgrounds and diverse characteristics, and are mainly found in eight common crop categories: rice, wheat, beet, alfalfa, corn, citrus, mango, and vine. These pests represent typical pests in common crops. We divide wireworm and rice leaf roller into larvae and adults, resulting in 16 classes, as shown in Figure 1. Then, we use LabelImg (version 1.8.6, accessed on 12 January 2023) software to annotate the pests in the images. After selecting and labeling images of 16 pest classes, we obtain 2696 raw images containing various sizes of pests, occluded targets, and complex backgrounds.

2.2. Data Augmentation

To avoid problems of the validation set containing training set images, we partition the raw images into the raw training and validation set at a ratio of 8:2 before data augmentation, as illustrated in Figure 2a. To address the data class imbalance and improve the model’s generalization ability, we apply data augmentation methods to the raw training set and raw validation set. Data augmentation methods are divided into simple augmentation and image fusion, where simple augmentation methods involve adding Gaussian noise, salt and pepper noise, rotating images, and scaling images.

Image fusion is used for the combination of different images, which can combine different raw images or images enhanced through simple augmentation. We select two or four images from the same class to fuse into one multi-objective image, or select images from two or four classes to fuse into one image with multiple classes of pests.

After the above operation (simple augmentation and image fusion), we obtain the training set and the validation set, which are 8960 and 2240 images, respectively, for a total of 11,200 images, which is the required dataset IP16. Each class of IP16 has at least 2000 instances, as shown in

Table 1. The statistics of the number of instances per class.

Class	Instances	Class	Instances
alfalfa plant bug	2000	rice leaf roller (larvae)	2216
aphids	2474	rice leaf roller (adult)	2187
locustoidea	2333	rice leafhopper	2123
mole cricket	2029	salurnis marginella	2016
oides decempunctata	2597	Serica orientalis	2370
papilio xuthus	2235	thrips	2136
penthaleus major	2375	wireworm (larvae)	2592
potosia brevitarsis	2330	wireworm (adult)	2107

. Taking the training set as an example, the augmentation process of the two images in training set is shown in Figure 2b.

2.3. YOLOv5 Model

We choose version 7.0 of YOLOv5 for experimentation in this paper. The network structure of YOLOv5 is divided into Backbone, Neck, and Prediction. Backbone is responsible for extracting image features. The main task of the Neck is to fuse the features provided by Backbone. Prediction obtains three feature maps of sizes 20 × 20, 40 × 40, and 80 × 80 generated by Neck, which are used to determine the positions of objects in the image and classify them. The YOLOv5 model is composed of modules including Conv, CBS, C3, Upsample, SPPF, and Concat, where CBS is Conv + batch normalization (BN) + sigmoid linear unit (SiLU). SiLU is a commonly activation function. SPPF consists of CBS and three maximum pooling (MaxPool), where the kernels size (k) of MaxPool is 5. The function of Concat is to concatenate channels along a specific dimension. The purpose of Upsample is to increase the resolution of the feature maps. The structure of the YOLOv5 model and its modules is depicted in Figure 3.

2.4. HSPPF Module

The primary purpose of SPP [28] is to generate fixed-size output, effectively capturing information across various receptive fields through MaxPool operations with k = 1, 5, 9, and 13. The parallel pooling structure enhances the flexibility and performance of the model. The detailed structure of the SPP module is illustrated in Figure 4a.

Ultralytics proposes a variant called SPPF based on SPP. Compared to SPP, SPPF also has different large receptive fields and fixed-size output, but it significantly reduces computation, thereby enhancing the speed of the model. The detailed structure of the SPPF module is illustrated in Figure 4b.

We propose the HSPPF to replace SPPF in this paper. This module integrates both average pooling (AvgPool) and MaxPool across various receptive fields. AvgPool retains more background information about the image, emphasizes the downsampling of the overall feature information. MaxPool retains more texture information, selects features with better recognition and handles irrelevant information. The HSPPF module enhances the precision of model detection by fusing features of different pooling receptive fields. Therefore, we utilize the HSPPF module to further enhance the performance of the model. The detailed structure of the HSPPF module is depicted in Figure 4c.

Given an input $x\in {\mathbb{R}}^{H\times W\times C}$, the HSPPF module works as follows: It first acquires $F_c=CBS\left(x\right)\in {\mathbb{R}}^{H\ast W\ast 0.5\ast C}$ through first CBS. Subsequently, after traversing three MaxPool, it obtains $F_m\left(0\right)$, $F_m\left(1\right)$ and $F_m\left(2\right)$. Likewise, after passing through three AvgPool, it obtains $F_a\left(0\right)$, $F_a\left(1\right)$ and $F_a\left(2\right)$. Finally, it connects $F_c$, $F_m\left(0\right)$, $F_m\left(1\right)$, $F_m\left(2\right)$, $F_a\left(0\right)$, $F_a\left(1\right)$ and $F_a\left(2\right)$, resulting in $F_{ConCat}\in {\mathbb{R}}^{H\times W\times 3.5\times C}$, as shown in Equations (1)–(3). This process aggregates feature maps through MaxPool and AvgPool of various receptive fields and connects them to obtain abundant features that combine the advantages of both pooling techniques.

$$F_{ConCat}=[F_c,{\sum }_{0\le i\le 2}{F}_m\left(i\right),{\sum }_{0\le i\le 2}{F}_a\left(i\right)]$$

(1)

$$F_m(i)=\left\{\begin{array}{cc}{F}_m\left(F_c\right),\hfill & i=0,\\ {F_m(F}_m(i-1)),\hfill & 1\le i\le 2.\end{array}\right.$$

(2)

$$F_a(i)=\left\{\begin{array}{cc}{F}_a\left(F_c\right),\hfill & i=0,\\ {F_a(F}_a(i-1)),\hfill & 1\le i\le 2.\end{array}\right.$$

(3)

where $F_m\left(i\right)$ is the output through MaxPool and $F_a\left(i\right)$ is the output through AvgPool. [·,·,·] is the channel concatenation operation.

2.5. NCBAM

Attention mechanisms commonly used today include SE [29], CBAM [30], ECA [31], and coordinate attention (CA) [32]. SE focuses solely on channel importance, it overlooks spatial information. CBAM adds a spatial attention module (SAM) after the channel attention module (CAM), thereby achieving a dual mechanism that includes both channel information and spatial information. Nonetheless, it still falls short of capturing long-range dependencies within the feature maps. Both ECA and CA build upon SE’s emphasis on channel relationships.

In this paper, we propose a new attention mechanism, called NCBAM. This module comprises two components: an improved coordinate attention module (ICAM) and a SAM, which makes up for the shortcoming of spatial information in CA and the inability of CBAM to capture long-range correlations in feature maps by emphasizing important information in both channel and spatial dimensions while suppressing non-essential information. The combination of NCBAM and C3 enhances the model’s complexity, improves its feature extraction capability, and enhances the precision while maintaining detection speed. Assuming the input feature is $x=[x_1,x_2\dots ,x_C]\in {\mathbb{R}}^{C\times H\times W}$, where $C$, $H$ and $W$ are the channel, height, and width of the input feature, respectively. $y=[y_1,y_2\dots ,y_C]\in {\mathbb{R}}^{C\times H\times W}$ is the output of ICAM, while $M_s$ is the output of the gray dotted box of SAM. The detailed structure of the new convolutional block attention module (NCBAM) is depicted in Figure 5.

2.5.1. ICAM

CA, as a novel and effective attention mechanism, captures cross-channel information, and directional and location-sensitive information, thereby improving the model’s ability to detect pests accurately. The encoding process of CA is divided into two steps: coordinate information embedding and coordinate attention generation. To enhance the robustness of the model and obtain rich features, we add channel shuffle [33] in coordinate attention generation stages to rewash the channels to reduce the influence of channel order. The improved attention mechanism is called ICAM, the detailed structure of ICAM is shown in Figure 6.

Coordinate information embedding: In order to solve the problem of global pooling easy to lose position information, the input feature x is decomposed into two one-dimensional features (X and Y directions) encoding process. This decomposition method generates a pair of direction-aware feature maps along two spatial directions so that CA has long-term dependencies and saves the precise location information. Assuming that the c-th channel of $x$ is represented by $x_c\left(i,j\right)$. Each channel is encoded along the horizontal and vertical coordinates using the pooling kernel of size (H, 1) and (1, W), as shown in Equations (4) and (5).

$$z_c^h\left(h\right)=\frac{1}{W}{\sum }_{0\le j<W}{x}_c\left(h,j\right)$$

(4)

$$z_c^w\left(w\right)=\frac{1}{H}{\sum }_{0\le i<H}{x}_c\left(i,w\right)$$

(5)

where $z_c^h\left(h\right)$ is the output of the cth channel with height $h$, and $z_c^w\left(w\right)$ is the output of the cth channel with width $w$.

Coordinate attention generation: This step aims to effectively leverage precise position information and the global receptive fields obtained in the previous step. To bolster the model’s robustness and derive richer feature maps, we add channel shuffle to rearrange the channels. The process of coordinated attention generation is as follows:

① it first connects $z^h$, $z^w$ from the previous step. Then, it applies the convolution function $f^{1\times 1}$ of k = 1 to reduce the channels, as depicted in Equation (6).

$$f=s\left(\delta \left(f^{1\times 1}\left(\left[z^h,z^w\right]\right)\right)\right),f\in {\mathbb{R}}^{\raisebox{1ex}{$C$}\!\left/ \!\raisebox{-1ex}{$r$}\right.\times 1\times (H+W)}$$

(6)

where $\delta$ is the non-linear activation function; r is the reduction ratio; $s$ is the channel shuffle operation. [·,·] is the concatenation operation along certain dimension. f is the output through Concat, $f^{1\times 1}$, $\delta$, and $s$ operation.

② Then, it splits f along the horizontal direction and the vertical direction into two separate tensors $f^h$ and $f^w$. Subsequently, two convolution functions, $F_h$ and $F_w$ of k = 1, are applied to transform $f^h$ and $f^w$ into tensors with the same number of channels to the input $x$, as shown in Equations (7) and (8).

$$g^h=\sigma \left(F_h\left(f^h\right)\right)$$

(7)

$$g^w=\sigma \left(F_w\left(f^w\right)\right)$$

(8)

where σ is the sigmoid function.

③ The c-th channel of $g^h$ is $g_c^h\left(i\right)$, and $g^w$ is $g_c^w\left(j\right)$. Finally, the output of c-th channel of ICAM as shown in Equation (9).

$$y_c(i,j)=x_c(i,j)\times g_c^h\left(i\right)\times g_c^w\left(j\right)$$

(9)

2.5.2. SAM

To exploit the spatial relationships among features, we add the SAM following the ICAM. The SAM module works as follows: it first uses MaxPool and AvgPool to pool feature maps $y$ from ICAM. Then, it concatenates channels from MaxPool and AvgPool, applies convolution functions, $f^{7\times 7}$ of k = $7$, and activates results after $f^{7\times 7}$ using the sigmoid function, resulting in spatial attention feature maps $M_s$. These feature maps effectively capture spatial information. The detailed structure of the SAM is illustrated in Figure 7. The $M_s$ is shown in Equation (10).

$$M_s\left(y\right)=\sigma \left(f^{7\times 7}\left(\left[F_a\left(y\right),F_m\left(y\right)\right]\right)\right)$$

(10)

where $\sigma$ refers to the sigmoid function.

2.5.3. C3NCBAM

C3 utilizes a residual structure for learning, which facilitates the training of deeper networks while maintaining good performance. In our study, we incorporate the NCBAM module into C3, resulting in the creation of a new module called C3NCBAM. This module combines the advantages of C3 and NCBAM, reduces the loss of feature information, and improves the feature extraction ability and detection precision of the model. The detailed structure of C3NCBAM is depicted in Figure 8.

2.6. $g^{3}Conv$ Module

The Conv operator of YOLOv5 has a limitation: the receptive field of small convolution kernels is restricted, and captures global information by stacking multiple layers. However, increasing the number of layers can lead to loss of fine-grained features. The transformer’s self-attention mechanism enables model to effectively capture global information, thereby overcoming the limitations of convolutional operations of small convolution kernels and enhancing the model’s expressive power.

With the great success of the new model of vision transformer [34,35,36,37,38], Rao et al. [39] introduced the input adaptive of the vision transformer, long-range, and high-order spatial characteristics into CNNs and proposed the recursive gated convolution ($g^{n}Conv$) to enable high-order spatial interaction through gated convolution and recursive design. The $g^{n}Conv$ exhibits high flexibility and customizability, allowing for the extension of second-order interactions in self-attention to any order without introducing excessive computations. As a result, the $g^{n}Conv$ effectively integrates the strengths of vision transformers and CNNs.

In pest detection tasks, the YOLOv5 model lacks global informations and the capability to effectively model long-range dependencies, which limits the effectiveness of pest detection to some extent. To address these limitations, we integrate the $g^{3}Conv$ (When n = 3, $g^{n}Conv$ is $g^{3}Conv$) into YOLOv5, combining the advantages of CNNs and transformers to improve the effectiveness of pest detection.

The $g^{n}Conv$ module works as follows: Assuming the input is $x\in {\mathbb{R}}^{H\times W\times C}$, the output channels of the first Conv (k = 3, stride = 2, p = 1) are $2\times C$. The feature size is ($\frac{H+2p-k}{stride}+1$, $\frac{W+2p-k}{stride}+1$). For convenience, this size is denoted as ($H_1$, $W_1$). If the result is not an integer, it can be rounded down. Then, the output channels of the last Conv are C. To ensure that higher-order interactions do not introduce significant computational overhead, this module sets the number of channels in each order to be an exponential multiple of 2. The $g^{n}Conv$ achieves n-order spatial interactions. $C_i$ is channel dimension in each order, is shown in Equation (11):

$$C_i=\frac{C}{2^{n-i-1}},0\le i\le n-1$$

(11)

When $n=1$, this structure of $g^{1}Conv$ is illustrated in Figure 9. Assuming ${\varphi }_{in}$ is a set of projected features composed of $p_0$ and ${\left\{q_i\right\}}_{i=0}^{n-1}$, and ${\varphi }_{out}$ is a set of projected features composed of ${\left\{p_i\right\}}_{i=1}^n$ to perform channel mixing, where $p_0$ is feature from the frist split and ${\left\{q_i\right\}}_{i=0}^{n-1}$ stores recursive branches features $q_0,q_1\dots ..q_{n-1}$ from the split of depth-wise convolution (DWConv). ${\left\{p_i\right\}}_{i=1}^n$ stores the result of multiplying two features. $i$ will be increased by 1 after each order. $y$ is the result of multiplying two features of last recursion step of the recursive body. The output of recursive body of $g^{1}Conv\left(x\right)$ is shown in Equation (12).

$$\begin{gathered} \left[p_0^{H_1{\times W}_1\times C},q_0^{H_1{\times W}_1\times C}\right]={\varphi }_{in}\left(x\right)\in {\mathbb{R}}^{H_1{\times W}_1\times 2C}, \\ {p}_1=d\left(q_0\right)⨀p_0\in {\mathbb{R}}^{H_1{\times W}_1\times C},y={\varphi }_{out}\left(p_1\right)\in {\mathbb{R}}^{H_1{\times W}_1\times C}, \end{gathered}$$

(12)

When $n>1$, the $g^{n}Conv$ further enhances model capacity by introducing higher-order interactions. The operation of the $g^{n}Conv$ module performs the gated convolution recursively, as shown in Equation (13). The structure of the $g^{3}Conv$ is depicted in Figure 10.

$$\begin{gathered} \left[p_0^{H_1{\times W}_1\times C_0},q_0^{H_1{\times W}_1\times C_0},\cdots ,q_n^{H_1{\times W}_1\times C_{n-1}}\right]={\varphi }_{in}\left(x\right)\in {\mathbb{R}}^{H_1{\times W}_1\times \left(C_0+{\sum }_{0\le i\le n-1}{C}_i\right)}, \\ {g}_i=\left\{\begin{array}{cc}Identity,\hfill & n=0,\\ Linear(C_{i-1},C_i),\hfill & 1\le n\le n-1.\end{array}\right. \\ {p}_{i+1}=d_i\left(q_i\right)\odot g_i\left(p_i\right)/\alpha \in {\mathbb{R}}^{H_1{\times W}_1\times C_i},i=\mathrm{0,1},\cdots ,n-1. \end{gathered}$$

(13)

where ${\{d}_i\}$ is a set of depth-wise convolutions, ${\{g}_i\}$ is matched the dimensions in different orders, and 1$/\alpha$ is scale factor to stabilize the training. Finally, the output of recursive body of $g^{n}Conv$ is ${\varphi }_{out}\left(q_n\right)$.

2.7. Soft-NMS

In the post-processing stage of object detection tasks, NMS is a critical step, which aims to eliminate redundant bounding boxes that represent the same object [40], thereby improving the model’s precision and efficiency. The NMS algorithm arranges the candidate boxes in descending order based on their scores, selects the candidate box with the highest score, and deletes other candidate boxes with an overlap greater than a threshold. This process is repeated for the remaining candidate boxes. The major issue of greedy algorithm is that the overoverlapping targets can lead to the deletion of candidate boxes for other targets. In addition, the threshold of NMS is not easy to determine. If a low threshold is applied, it may result in an increase in the miss-rate and a drop in average precision. If a high threshold is applied, the increase in false positives (FP) will surpass the increase in true positives (TP) by a considerable margin. In real life, pest distribution is dense, so the traditional NMS algorithm is used to process the detection results, which is easy to cause adjacent pests to be missed. Aiming at the shortcomings of traditional NMS algorithm, Soft-NMS algorithm [41] optimizes the NMS algorithm by decaying the detection score. The Soft-NMS algorithm has the following advantages: ① It addresses the issue of missed detection that can occur when objects are in close proximity to each other. ② It does not have additional hyperparameters. ③ Relative to NMS, it has the same computational complexity ${O(N}^2)$, where $N$ is the number of detection boxes.

Assuming that the current detection boxes with the highest score is $M$, $N_t$ is the NMS threshold, ${B=\{b}_i,1\le i\le N\}$ is a set of the detection boxes, $D$ is a set of final detection boxes and ${S=\{s}_i,1\le i\le N\}$ is a set of the scores of ${\{b}_i,1\le i\le N\}$. The Soft-NMS algorithm has two forms, linear and Gaussian penalty function, as shown in Equations (14) and (15).

$$s_i=\left\{\begin{array}{cc}{s}_i,\hfill & iou\left(M,b_i\right)<N_t\\ s_i\left(1-iou\left(M,b_i\right)\right),\hfill & iou\left(M,b_i\right)\ge N_t\end{array}\right.$$

(14)

$$s_i=s_i{e}^{-\frac{{iou\left(M,b_i\right)}^2}{\sigma }},\forall b_i\notin D.$$

(15)

where $iou\left(M,b_i\right)$ is the IoU value of $M$ and $b_i$, $\sigma$ is sigma function.

In real life, pest detection is mostly crowded and dense, with multiple overlapping targets. Therefore, the use of Soft-NMS algorithm can reduce missed detection, and improve the detection effect of pest detection.

2.8. YOLOv5-Pest Model

To improve the performance of the model in detecting pests, we propose the YOLOv5-pest model by implementing several improvements to the YOLOv5 model. These improvements include designing the HSPPF and C3NCBAM modules, introducing the $g^{3}Conv$ module in the neck, and replacing the NMS module in the post-processing stage with Soft-NMS. The structure of the YOLOv5-pest model is depicted in Figure 11.

2.9. Evaluation Indicators

To verify the performance of model, we use precision (P), recall (R), and mAP for evaluation indicators, as shown in Equations (16)–(19). In addition, we also use parameters, layers and GFLOPs (giga floating-point operations per second) to measure model size, model depth, and computational power.

$$P=\frac{TP}{TP+FP}$$

(16)

$$R=\frac{TP}{TP+FN}$$

(17)

$$AP={\int }_0^{1}P\left(R\right)dR$$

(18)

$$mAP=\frac{\sum _{i=1}^N{AP}_i}{N}$$

(19)

where $TP$ represents the number of pests correctly identified as belonging to their corresponding classes, $FP$ signifies the count of pests not belonging to the class but identified as such, and False negatives ($FN$) indicates the number of pests belonging to the class but erroneously identified as not belonging to the class. $N$ is the number of classes of pests. In Equation (18), $AP$ is the average precision, which is the area under the $P-R$ curve. In Equation (19), ${AP}_i$ is the $AP$ of the i-th class, $mAP$ is the mean average precision of all classes. In this study, mAP contained mAP@0.5 and mAP@0.5:0.95, where mAP@0.5 represents IoU = 0.5, the mean average precision of all classes, mAP@0.5:0.95 represents the mean average precision of all classes of IoUs increasing from 0.5 to 0.95 at intervals of 0.05.

3. Results and Discussion

3.1. Training Environment

The hardware and software configuration parameters of experiments are shown in

Table 2. Training environment.

Configuration	Parameter
Operating System	64-bit Windows 10
CPU	Intel(R) Core(TM) i9-9900K CPU @ 3.60 GHz
GPU	NVIDIA GeForce RTX 2080 Ti
Cuda	Cuda 10.1
Cudnn	Cudnn 7.6.5
Deep Learning Frame	Pytorch 1.8.1

.

3.2. Hyperparameter Configuration

Given our constraints on computational resources, we set the model’s hyperparameters as follows: the image resolution is set to 640 × 640. During training, we set the maximum number of epochs to 300 with a batch size of 16. Additionally, we configure the momentum at 0.937, initialize the learning rate to 0.01, and set the weight decay to 0.0005.

3.3. Selection of Baseline Model

YOLOv5 is divided into five versions according to different depth coefficient (the number of residual blocks) and width coefficient (the number of channels) of the model: YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. To select the best version, each of the five versions is trained on the IP16 dataset.

As shown in

Table 3. Comparison results of different versions of YOLOv5.

Model	Layers	P	R	mAP@0.5	mAP@0.5:0.95	Parameters	GFLOPs
YOLOv5x	445	0.966	0.909	0.939	0.721	86,318,749	204.9
YOLOv5l	368	0.939	0.89	0.928	0.705	46,219,069	108.5
YOLOv5m	291	0.931	0.885	0.92	0.694	20,931,933	48.4
YOLOv5s	214	0.92	0.856	0.902	0.664	7,062,781	16.1
YOLOv5n	214	0.913	0.834	0.885	0.618	1,785,565	4.3

, the experimental results indicate that the larger the depth and width coefficients, the stronger the feature extraction and fusion capabilities of the model, and the higher the precision, but the deeper the layers, the larger the parameters, and the larger the GFLOPs. Although YOLOv5n has the smallest Parameters and GFLOPs, the mAP@0.5:0.95 of YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x is 4.6%, 7.6%, 8.7%, and 10.3% higher than YOLOv5n, respectively. The P, R, and mAP of YOLOv5n are significantly lower than other models. Compared with YOLOv5m, YOLOv5l, YOLOv5x, the P, R, mAP@0.5 and mAP@0.5:0.95 of YOLOv5s are close, but the parameters and GFLOPs of the model are also relatively slightly. Therefore, after considering the requirements for detection precision and real-time performance, we select YOLOv5s as the baseline model.

3.4. Ablation Experiments

To validate the effectiveness of the HSPPF, C3NCBAM, $g^{3}Conv$, and Soft-NMS modules, we integrate these modules into the baseline model, as delineated in

Table 4. Results of ablation experiment after integrating HSPPF, C3NCBAM, $g^{3}Conv$, Soft-NMS.

Model	P	R	mAP@0.5	mAP@0.5:0.95	Parameters	GFLOPs
YOLOv5s	0.92	0.856	0.902	0.664	7,062,781	16.1
YOLOv5s + HSPPF	0.937	0.857	0.91	0.665	7,455,997	16.4
YOLOv5s + C3NCBAM	0.941	0.878	0.92	0.671	7,091,213	16.2
$\mathrm{YOLOv}5\mathrm{s}+g^{3}Conv$	0.934	0.87	0.909	0.671	7,967,741	17.2
YOLOv5s + Soft-NMS	0.93	0.864	0.919	0.719	7,062,781	16.1

.

These experiments demonstrate that after adding of the HSPPF module, the mAP@0.5 is 0.8% higher than YOLOv5s. After adding the C3NCBAM module, the P, R, mAP@0.5 is 2.1%, 2.2%, 1.8% and 0.7% higher than YOLOv5s, respectively. After adding $g^{3}Conv$, the mAP@0.5:0.95 is 0.7% higher than YOLOv5s. After the introduction of Soft-NMS, the mAP@0.5:0.95 is 5.5% higher than YOLOv5s. It can be concluded that the HSPPF module gets more information effectively by the operation of combining AvgPool and MaxPool. The NCBAM module emphases important information while suppressing non-essential details. The structure of C3NCBAM is more conducive to extracting features than C3. Compared to Conv, the $g^{3}Conv$ module enhances model’s ability to model the complex spatial interactions and is easy to capture long-term dependencies. The Soft-NMS reduces missed detection and false positives improving overall detection precision. These improvement methods are very effective in improving precision, recall, and mean average precision.

3.5. Comparison Experiments

3.5.1. Comparison of Different Attention Mechanisms

To verify the effectiveness of the NCBAM attention mechanism, we compare NCBAM with different attention mechanisms. To ensure fairness of the comparison, each attention is added to the same location in C3 as the C3NCBAM module. The comparison results are shown in

Table 5. Comparison results before and after integrating different attention mechanisms.

Model	P	R	mAP@0.5	mAP@0.5:0.95	Parameters	GFLOPs
YOLOv5s	0.92	0.856	0.902	0.664	7,062,781	16.1
YOLOv5s + C3SE	0.928	0.869	0.909	0.664	7,086,461	16.1
YOLOv5s + C3CBAM	0.934	0.87	0.91	0.664	7,087,245	16.1
YOLOv5s + C3ECA	0.937	0.866	0.907	0.668	7,062,805	16.1
YOLOv5s + C3CA	0.939	0.873	0.914	0.668	7,090,429	16.1
YOLOv5s + C3NCBAM	0.941	0.878	0.92	0.671	7,091,213	16.2

.

The experimental results demonstrate that our proposed module NCBAM outperforms SE, CBAM, ECA, and CA modules. Specifically, the mAP@0.5 is 1.1%, 1%, 1.3%, and 0.6% higher than SE, CBAM, ECA, and CA, respectively. The mAP@0.5:0.95 is 0.7% higher than SE, CBAM. It can be concluded that compared with SE, CBAM, ECA, and CA, the NCBAM is more effective in extracting important characteristics of pests.

3.5.2. Comparison of Different Methods

To validate the detection performance of YOLOv5s-pest, we compare it with several methods, such as YOLOv5s-transfromer, YOLOv5s-bifpn, YOLOv5m-ghost from the hub of YOLOv5, YOLOv5-Lite-g from version 1.4 of YOLOv5-Lite [42], SE-YOLOv5s [43], MG-YOLO [44], YOLOv7-tiny [45], YOLOXs [46], and YOLOv8s [47]. Ensuring a fair comparison, we ensure that all models have similar computational complexity. The comparison results are shown in

Table 6. Comparison results of different methods.

Method	P	R	mAP@0.5	mAP@0.5:0.95	Parameters	GFLOPs
YOLOv5s	0.92	0.856	0.902	0.664	7,062,781	16.1
YOLOv5s-transformer	0.931	0.854	0.91	0.653	7,063,037	15.9
YOLOv5s-bifpn	0.937	0.867	0.912	0.661	7,128,317	16.3
YOLOv5m-ghost	0.908	0.807	0.855	0.609	8,586,789	18.7
YOLOv5-lite-g	0.931	0.859	0.907	0.66	5,500,541	15.9
SE-YOLOv5s	0.9	0.804	0.85	0.593	7,282,621	16.5
MG-YOLO	0.933	0.871	0.908	0.651	6,523,702	14.6
YOLOv7-tiny	0.927	0.875	0.904	0.664	6,055,578	13.3
YOLOX-s	-	-	0.909	0.659	8,943,487	26.8
YOLOv8s	0.918	0.858	0.905	0.701	11,166,560	28.8
YOLOv5s-pest	0.944	0.871	0.925	0.726	8,389,389	17.6

.

The experimental results demonstrate that the mAP@0.5 of YOLOv5s-pest is 92.5% and the mAP@0.5:0.95 of is 72.6%. Compared to YOLOv5s, YOLOv5s-transformer, YOLOv5s-bifpn, YOLOv5m-ghost, YOLOv5-lite-g, SE-YOLOv5s, MG-YOLO, YOLOv7-tiny, YOLOX-s and YOLOv8s, the mAP@0.5 of the YOLOv5s-pest is 2.3%, 1.5%, 1.3%, 7%, 1.8%, 7.5%, 1.7%, 2.1%, 1.6% and 2% higher than these models, respectively. The mAP@0.5:0.95 is 6.2%, 7.3%, 6.5%, 11.7%, 6.6%, 13.3%, 7.5%, 6.2%, 6.7% and 2.5% higher than these models, respectively. In terms of the comparison results of the above methods, our proposed method, YOLOv5s-pest, achieves the outstanding performance. After improving the backbone, neck and postprocessing of the baseline model, P, R, and mAP have increased greatly while parameters and GFLOPs increase only a little. Hence, YOLOv5s-pest can be chosen as the preferred option for pest detection.

3.5.3. Comparison of YOLOv5s and YOLOv5s-Pest on Different Datasets

To further verify the effectiveness of YOLOv5s-pest, we perform a comparative experiment on the insect [48], pest24 [49], and pascal visual object classes (VOC) 2007 + 2012 dataset [50], the results are shown in

Table 7. Comparison results of YOLOv5s and YOLOv5s-pest on different datasets.

Dataset	Train	Val	Class	Method	P	R	mAP@0.5	mAP@0.5:0.95
Insect	1693	245	7	YOLOv5s	0.753	0.853	0.875	0.627
				YOLOv5s-pest	0.802	0.83	0.869	0.641
Pest24	12,701	5077	24	YOLOv5s	0.69	0.62	0.636	0.395
				YOLOv5s-pest	0.724	0.59	0.664	0.429
VOC	16,551	4952	20	YOLOv5s	0.815	0.775	0.831	0.596
				YOLOv5s-pest	0.819	0.785	0.843	0.665

.

As can be seen from Table 7, the mAP@0.5:0.95 of the YOLOv5s-pest on the insect, pest24, and pascal VOC dataset, is 64.1%, 42.9%, 66.5%, which is 1.4%, 3.4%, 6.9% higher than the YOLOv5s, respectively. This experiment proves that our model also has a certain degree of improvement effect on other datasets. In the future, we can try to apply our proposed method to more detection tasks.

3.6. Training Curve Analysis

The training curves change of mAP of different methods on the IP16 are shown in Figure 12. During training, we train models using pretrained weights provided by the official YOLOv5, which are trained on large public datasets. These pretrained weights include yolov5n.pt, yolov5s.pt, yolov5m.pt, yolov5l.pt, and yolov5x.pt. The training curve of YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x is depicted as shown in Figure 12a. All methods achieve fast convergence and the best performance of in 300 epochs.

The training curves change of mAP of our custom methods, which add HSPPF, C3NCBAM, $g^{3}Conv$, Soft-NMS, C3SE, C3CBAM, C3ECA, and C3CA, respectively, are shown in Figure 12b,c. The mAP of our custom methods are relatively low in the first few epochs. With the increase in epochs, the detection performance of models is improved by learning. The mAP of models gradually increase, eventually surpass the performance of YOLOv5s.

The training curves change of mAP of different methods of comparison experiments, are shown in Figure 12d. We use different pretrained weights to train these methods. These pretrained weights include yolov5s.pt (YOLOv5s-transformer, YOLOv5s-bifpn, SE-YOLOv5s, and MG-YOLO), yolov5m.pt (YOLOv5m-ghost), v5lite-g.pt (YOLOv5-Lite-g), yolov7-tiny.pt (YOLOv7-tiny), yolox_s.pth (YOLOs), and yolov8s.pt (YOLOv8s). The results clearly demonstrate that the training curve of mAP of YOLOv5s-pest is higher than other methods and the epochs of the end of training curve of YOLOv5s-pest are less than other methods.

3.7. Visualization of Attention Mechanisms

To visualize the reasoning process, we employ gradient-weighted class activation mapping (Grad-CAM) [51] to generate heat maps. The heat maps after applying different attention mechanisms are shown in Figure 13. It can be observed that, different columns add different attention mechanisms. After adding NCBAM, the red region in the target is larger than SE, CBAM, ECA, and CA. Therefore, from the visual results, it can be concluded that NCBAM has a stronger ability to filter redundant features and extract key features than SE, CBAM, ECA, and CA.

3.8. Discussion

In agricultural production, pests are a continuous concern as they pose potential threats to crop yield and quality. DL has garnered significant attention and application in the domain of pest monitoring [52,53]. Recent researches [54,55,56,57,58,59,60] exhibit the YOLO series’ efficacy in addressing diverse crop-related challenges. However, CNNs base on DL approaches, have limitations in practical pest detection scenarios [61], restricting their real-world applicability. Under complex backgrounds and the presence of different pest species, We propose YOLOv5s-pest (HSPPF + C3NCBAM + $g^{3}Conv$ + Soft-NMS) to improve detection precision, where HSPPF module is strategically crafted to fortify the model’s capacity to extract multi-scale receptive field information within feature maps. Additionally, the incorporation of the NCBAM serves to emphasize pertinent features while concurrently suppressing redundant information, thereby enhancing detection precision without compromising computational speed. This model not only demonstrates significant improvement in detecting multiple pest species, but also has scalability in other object detection tasks. However, a limitation of this approach is that it slightly increases the model’s parameters and GFLOPs.

Future research should focus on developing more sustainable, efficient, and environmentally friendly pest detection strategies to ensure the sustainability and stability of agricultural production. Lightweight networks provide efficient and resource-saving solutions for pest detection.

4. Conclusions

Crop pest detection plays a critical role in modern agriculture [62,63]. With the advancement of technology, advanced techniques such as ML and DL have been extensively utilized for crop monitoring and management. Traditional manual methods for pest detection have been laborious and time-consuming [64]. Pest detection base on ML [65,66,67] approachesrely heavily on handcrafted feature extractors and intricate rule-based systems. DL techniques, particularly the YOLO series, facilitate rapid feature extraction from vast datasets, ensuring timely and accurate pest detection. The complexities exist in real-world agricultural settings [68], such as intricate backgrounds, overlapping, and occlusion, present considerable challenges for effective pest detection. To address these, we propose YOLOv5s-pest. To validate the effectiveness of our modules and model, we create a dataset called IP16, which has complex backgrounds and scenarios where pests overlap and occlude. Upon individually integrating HSPPF, C3NCBAM, $g^{3}Conv,$ and Soft-NMS into the YOLOv5, distinct improvements are observed in the P, R, and mAP on the IP16 dataset. This proves the effectiveness of each of these modules in enhancing the model’s capabilities. The P, R, mAP@0.5, and mAP@0.5:0.95 of YOLOv5s-pest, which integrates innovative modules (HSPPF, C3NCBAM, and $g^{3}Conv$) and strategies (Soft-NMS), achieves 94.4%, 87.1%, 92.5%, and 72.6%, respectively. These novel additions endow the model with distinct advantages, setting it apart in terms of efficiency and performance within the realm of pest detection, demonstrating exceptional prowess in navigating challenges such as intricate agricultural backgrounds, overlapping, and occlusion scenarios. Consequently, the YOLOv5s-pest can serve as a technical reference and tool for pest control, assist farmers in taking timely preventive measures, provide innovative and effective solutions for pest control in agriculture and contribute to sustainable agricultural development.

Given the pivotal role of on-site cameras or agricultural robots in pest detection, we can deploy our model across various platforms, such as cloud servers, local servers, and embedded devices. Such deployment not only contributes to the enhancement of agricultural productivity and food safety but also facilitates environmental conservation efforts. By leveraging our model across diverse platforms, we can achieve broader coverage in pest monitoring and provide more effective solutions for the agricultural sector, thereby contributing to the realization of sustainable agricultural development goals.

As DL methodologies in pest detection continue to advance, future trajectories emphasize the development of lightweight, real-time monitoring systems. Anticipating this evolution, we aim to more intelligent, scalable, and easily deployable pest detection systems, further driving innovation and sustainable development in agriculture.