Enhanced Non-Maximum Suppression for the Detection of Steel Surface Defects

Abstract

Quality control in manufacturing equipment relies heavily on the detection of steel surface defects. Recently, there have been an increasing number of efforts in which object detection techniques have been utilized to achieve promising results in the detection of steel surface defects since the defect patterns can be considered objects. To enhance the detection performance in the object detection problem, the non-maximum suppression (NMS) step, which eliminates redundant boxes overlapped with a box having the greatest detection score, is essential. In this work, we propose a novel NMS to improve the detection method of steel surface defects. The proposed NMS approach is composed of three novel techniques: IoU regularization, threshold adjustment, and comparison rule modification to enhance the detection performance. To evaluate the performance of the proposed NMS, we carry out extensive numerical experiments using the YOLOv7 and EfficientDet models on the steel surface defect datasets, NEU-DET and GC10-DET. The experimental results demonstrate that the proposed NMS outperforms the conventional NMS methods in both quantitative and qualitative manners.

1. Introduction

In the manufacturing industry, the quality of products is highly dependent on the quality of the product surface [1], which is easily damaged during the production process, and thus, the detection of surface defects is crucial [2]. Recently, defect inspection on steel surfaces has gained increased attention as steel plates are a common material in numerous products. Metallurgical and mechanical flaws can result in a variety of defects on steel surfaces, including stains, cracks, pits, inclusions, and scratches [3]. These surface defects degrade the appearance and harm the fatigue strength of steel plates, resulting in poor-quality steel products. Moreover, quality issues affect not only the reputation and economy of manufacturers but also the safety and reliability of products for end-users [4].

Traditional defect detection methods mainly relied on manual inspection, which requires workers to work long hours at high intensity. Thus, traditional methods suffer from drawbacks such as low effectiveness, unreliable detection, high costs, and potential safety risks. To address these issues, recently, the inspection of surface defects using computer vision techniques has been widely adopted to replace manual inspection [3,4,5]. In particular, due to the recent advances in convolutional neural networks (CNNs) [6], CNNs have been actively adopted to detect various types of surface defects by regarding the surface defects as objects [7].

The CNN-based defect (object) detection approaches, in general, fall into two categories: the segmentation-based approach [8,9,10] and the detection-based approach [11,12]. The segmentation-based approaches have the following drawbacks. First, in the supervised segmentation approaches [8,9], pixel-level classification is conducted, which in turn requires the pixel-level ground truth in the training process, which is usually very expensive to acquire. Second, in the unsupervised segmentation approaches, a very sensitive thresholding process should often be involved in the segmentation algorithms [10]. On the other hand, the detection-based methods [11,12] only require annotating defect areas with bounding boxes (BBoxes) for training since they are threshold-free. Combined with Fast R-CNN [13], YOLO [14,15,16], or SSD [17] as the backbone network, these detection-based approaches have been extensively studied in the field of defect (object) detection [4,5,18,19,20,21].

Specifically, in [5], an end-to-end steel surface defect identification system was proposed based on fusing multiple hierarchical features. The authors adopted a convolution neural network (CNN) [6] to achieve a strong classification ability and proposed a multi-level feature fusion network to effectively combine multiple hierarchical (lower-level and higher-level) features into one feature. Additionally, a region proposal network (RPN) [22] is adopted based on the multi-level features to generate regions of interest (ROIs), and a detector consisting of a classifier and a bounding box regressor is adopted for each ROI to produce final results. In [18], a fast and accurate one-stage detector, called SSDNet, was proposed to detect steel surface defects where a feature retaining block (FRB) was first proposed to preserve surface defects’ texture information systematically. Specifically, the FRB is applied after the pooling layer to compensate for downsampling-induced texture loss, and a skip-densely connected module (SDCM) is designed to carry the fine-grained information of small surface defects to the high-level layers [18] by skipping connections between early and later prediction layers.

In [19], a steel surface defect detection system based on a fused-attention network (FANet) was proposed, which employs an attention mechanism on a single balanced feature map rather than on multiple feature maps. In particular, an adaptively balanced feature fusion (ABFF) and a fused-attention module (FAM) were respectively proposed in [19] to handle defects with multiple scales and to locate/classify defects more accurately. Additionally, a modified YoloV4 algorithm in [20] for defect identification on steel surfaces proposed the adoption of a convolutional block attrition module (CBAM) [23] and a custom receptive field block (RFB) [24] to enhance classification and feature reconstruction.

In [4], DCC-CenterNet, a steel surface defect detector was developed that uses keypoint estimation to specify center points and regresses other defect properties. Specifically, to enlarge the detector’s receptive field, a dilated feature enhancement model is proposed, and a novel center-weighting function is proposed to make keypoint estimation more accurate. Moreover, in [21], a lightweight feature fusion network, LFF-YOLO, was proposed for identifying steel surface defects with improved detection accuracy and speed. In LFF-YOLO, to accommodate defects at different scales, an adaptive receptive field feature extraction (ARFFE) method was designed to obtain features for adaptive sensory fields. In addition, a lightweight feature pyramid network (LFPN) was proposed, introducing fewer parameters to improve the fusion efficiency of multiscale features [21].

Object detection techniques generally consist of the following three-step procedure: (i) generating BBoxes for specified object categories, (ii) refining or scoring the BBoxes with a regressor or classifier, and (iii) merging BBoxes belonging to the same object together. The final step in the object detection procedure is commonly referred to as non-maximum suppression (NMS) [13,14,15]. The NMS procedure can improve the detection accuracy by eliminating the redundant boxes, and thus, it has been integrated as a post-processing method in many object detection algorithms in computer vision, including edge detection [25], feature point detection [26,27], face detection [28], object detection [13,14,15], etc.

The greedy NMS [29], in which the bounding box with the highest detection score is selected and the boxes surrounding it are suppressed by applying a pre-defined threshold, has demonstrated improved performance over the approach used in [28]. Since then, the greedy NMS and its extensions have been widely adopted as a post-processing step in object detection [13,14,15,30]. However, the existing NMS approaches leave many false positive BBoxes, i.e., normal (flawless) products would be misclassified as defective and discarded. This means that the production costs of the manufacturing procedure are likely to rise due to misclassification (false positives) in the NMS procedure. In this work, therefore, we develop an enhanced NMS approach that consists of three novel and innovative techniques, i.e., IoU regularization, threshold adjustment, and comparison rule modification. Specifically, in the IoU regularization process, we introduce a regularizing parameter that can regularize the size of bounding boxes. It will be shown in the experimental results that by using a fine-tuned regularization parameter, the proposed NMS not only successfully suppresses more false positives in the detection of steel surface defects but also outperforms the comparable NMS approaches in both quantitative and qualitative manners.

In the following, we first briefly summarize the related works on NMS, and then we discuss the motivation and contributions of the proposed NMS in detail.

1.1. Related Works

The greedy NMS in [29] is a well-known NMS technique in which the BBox with the highest detection score is selected, and the BBoxes surrounding or surrounded by it are suppressed in accordance with a predefined thresholding algorithm. Specifically, the greedy NMS starts with a list of detection boxes $\mathcal{B}=\{b_1,b_2,\dots ,b_K\}$ with corresponding confidence scores $\mathcal{S}=\{s_1,s_2,\dots ,s_K\}$ where K is the number of initially detected BBoxes. The first step of the greedy NMS is to remove the box with the highest confidence score among $\mathcal{B}$ (resp., the highest confidence score among $\mathbf{s}$), i.e., $b_m$ (resp., $s_m$) for $m={arg\; max}_i{s}_i$, from $\mathcal{B}$ (resp., $\mathbf{s}$), and to include $b_m$ in the set of the finally detected BBoxes, $\mathcal{D}$. The second step is to calculate the values of the intersection over union (IoU) between the boxes in the set $\mathcal{B}$ and the box $b_m$, and to suppress the boxes in $\mathcal{B}$ with IoU exceeding the predefined threshold, which is hereafter called the IoU threshold. These two steps are repeated for the remaining boxes until $\mathcal{B}$ becomes empty.

In [31], Soft NMS was proposed as a single-line modification to the greedy NMS approach. In Soft NMS, instead of immediately suppressing the detection boxes whose IoU exceeds an IoU threshold, the confidence score is reduced using a Gaussian penalty function. Then, the bounding boxes are suppressed if the confidence score drops below a confidence score threshold. In [32], DIoU (distance IoU) NMS was proposed, which extends DIoU Loss to NMS. DIoU NMS uses the Euclidean distance of the center coordinates between two boxes and the diagonal distance of two boxes as a penalty term in the IoU calculation process. As a result, the farther the center coordinates between the boxes and/or the shorter the diagonal distance, the smaller the IoU. That is, by considering the overlap area as well as the distance between central points of two bounding boxes for suppressing redundancy, DIoU MNS is more robust to the cases with occlusions. In [33], Exponential NMS was proposed, which adjusts the sizes of the bounding boxes using an exponential operation in the IoU calculation process. By adjusting this, the problem of suppression failure, which occurs when the difference between the sizes of two BBoxes is large, can be effectively resolved. Meanwhile, a novel method for fusing predicted BBoxes, called weighted boxes fusion (WBF), was proposed in [30]. It was shown in [30] that by ensembling BBoxes with the corresponding confidence score instead of suppressing them, the performance of object detection could be improved. Finally, in [34], Confluence NMS was proposed, which utilizes a normalized Manhattan distance-inspired proximity metric instead of the confidence score for clustering BBoxes.

1.2. Motivations and Contributions

The conventional NMS approaches discussed above, however, have critical shortcomings as follows. First, in the greedy NMS, the confidence scores of boxes overlapping with the box that have the greatest confidence score are set to zero. That is, only the boxes overlapping with the most confident box can be suppressed. This would result in a suppression failure when some false positive boxes have IoUs that fall behind the IoU threshold. Furthermore, the Soft NMS and DIoU NMS will likely eliminate fewer boxes. As a result, many false positive BBoxes could be obtained in the qualitative analysis.

In addition, the conventional NMS schemes are validated only on the MS-COCO [35] dataset. Though the MS-COCO dataset is one of the most frequently used datasets for object detection, there are distinct differences between the MS-COCO dataset and the steel surface defect dataset, e.g., the NEU-DET dataset as shown in Figure 1. Thus, the performances of the NMS algorithms for steel surface defect detection should be evaluated and compared on a practical steel surface defect dataset such as the NEU-DET dataset [36] or GC10-DET [37].

Motivated by these drawbacks, in this paper, we propose a novel NMS approach that effectively improves the performance of the object detection algorithm, especially for detecting steel surface defects. The main contributions of this paper are listed as follows:

• We propose an enhanced NMS approach that consists of three novel and innovative techniques, IoU regularization, threshold adjustment, and comparison rule modification, to enhance the detection performance by strictly suppressing false positives.
• We carry out extensive experiments on the popular steel surface defect datasets NEU-DET and GC10-DET to evaluate the performance of the proposed NMS method for the YOLOv7 and EfficientDet models and compare it to the conventional NMS approaches. The experimental results demonstrate that the proposed method outperforms the comparable NMS approaches in quantitative and qualitative manners.

1.3. Organization

The remainder of the paper is organized as follows. Section 2 introduces the framework of the proposed NMS algorithm. Section 3 presents the experimental results and discussions, and finally, Section 4 concludes the paper with future research directions.

2. Proposed Scheme

In this section, the description of the proposed NMS approach is presented. First, to point out the difference between the traditional NMS and the proposed NMS algorithms, we briefly introduce the procedure of the traditional NMS (the greedy NMS). The greedy NMS algorithm can be written as follows:

$$s_i=\left\{\begin{array}{cc}{s}_i,\hfill & \mathrm{if} IoU(M,b_i)<N_t,\hfill \\ 0,\hfill & \mathrm{if} IoU(M,b_i)\ge N_t,\hfill \end{array}\right.$$

(1)

for

$$IoU(M,b_i)=\frac{M\cap b_i}{M\cup b_i}$$

(2)

where $b_i$ and $s_i$ are the ith BBox and its corresponding confidence score, and $N_t$ is the IoU threshold. In addition, M denotes the BBox having the highest confidence score among $\mathcal{B}$, i.e., $M=b_m$ where $m={arg\; max}_i{s}_i$. As shown in Equation (1), if the ratio of overlap between the M and $b_i$ is higher than a specified threshold, $N_t$, then the BBox $b_i$ is suppressed, or its confidence score is set to zero.

Next, we present the proposed NMS algorithm. In this work, to enhance the performance of steel surface defect detection, we focus on eliminating false positives in the predicted BBoxes. To this end, we modify the traditional NMS to strictly suppress false positives. The proposed modification is three-fold: (1) IoU regularization, (2) threshold adjustment, and (3) comparison rule modification. The key components of the proposed NMS are described as follows.

2.1. IoU Regularization

Firstly, in the proposed NMS method, a novel regularized IoU is developed instead of the original IoU. The regularized IoU between two BBoxes M and $b_i$ can be characterized by a regularizing parameter $\lambda$ as follows:

$$rIoU(M,b_i;\lambda )=\frac{\frac{b_L\cap b_S}{2}}{\lambda b_L+(1-\lambda )b_S-\frac{b_L\cap b_S}{2}},$$

(3)

for $0<\lambda <\frac{A_S}{A_L+A_S}$ where $b_L$ (resp., $b_S$) means the larger (resp., smaller) box of M and $b_i$, and $A_L$ (resp., $A_S$) means the area of the larger (resp., smaller) box. The rationale behind this is as follows: suppose that the original IoU is adopted and the IoU threshold is 0.5; if the sizes of M and $b_i$ are the same, then the two boxes must overlap by at least two-thirds to suppress the less confident box, $b_i$; however, if the sizes of M and $b_i$ are twice the difference, the two boxes must overlap completely to remove the less confident box, $b_i$. That is, by regularizing the sizes of two compared boxes, more false positives can be suppressed.

As depicted in Equation (3), the parameter $\lambda$ helps to regularize the sizes of $b_i$ and $b_j$. The parameter $\lambda$ is set to a number less than $\frac{A_S}{A_L+A_S}$ to keep the order in the size of the box. Further, as defined in Equation (3), the intersection area is divided into half to match the size, which was refined by extensive experiments. By dividing this, it is possible to suppress more false positives even when there is a large difference in the sizes of the compared BBoxes.

2.2. Threshold Adjustment

Secondly, the proposed NMS method exploits the confidence scores of BBoxes to further suppress false positives by intelligently adjusting the IoU threshold. As shown in Equation (1), the greedy NMS adopts a constant IoU threshold regardless of the confidence scores of the compared boxes. Otherwise, in the proposed NMS method, we have adjusted the IoU threshold according to the compared boxes with the corresponding confidence scores.

The proposed IoU threshold for the boxes M and $b_i$ is given by

$$N_t(M,b_i)=N_t\times [1-(s_m-s_{b_i})]$$

(4)

where $N_t$ is the base IoU threshold. From the definition of index m, i.e., $m={arg\; max}_i{s}_i$, it is always satisfied that $s_m\ge s_{b_i}$. Thus, the adjusted IoU threshold decreases by the difference between the confidence scores. Consequently, boxes with low confidence scores could be suppressed even if the IoU values between the most confident box and themselves are slightly less than the base IoU threshold.

2.3. Suppression Rule Modification

In the greedy NMS method, when the IoU of M and $b_i$ exceeds the IoU threshold $N_t$, the box $b_i$ is suppressed from $\mathcal{B}$. Once $b_i$ is suppressed, it can no longer be compared to the other BBoxes in $\mathcal{B}$. However, the fact that the IoU between M and $b_i$ exceeds the IoU threshold indicates that $b_i$ is also a most likely correct prediction. In the proposed NMS, therefore, instead of immediately suppressing $b_i$, it is proposed to temporarily retain the suppressed box $b_i$ and to further suppress any other box with an IoU (between $b_i$ and itself) exceeding the IoU threshold. The detailed comparison rule is summarized in Algorithm 1 shown below.

Algorithm 1: Modified comparison rule

First, initialize the temporary sets ${\mathcal{T}}_b$ and ${\mathcal{T}}_s$ as empty sets, and set the list of final detection boxes, $\mathcal{D}=\mathcal{B}$ (lines 7 and 8). Next, find the most confident box M, and remove it from $\mathcal{B}$ (lines 10–12). Then, check whether the IoU between M and any box $b_i$ in $\mathcal{B}$ exceeds the IoU threshold or not. If so, instead of deleting $b_i$ from $\mathcal{B}$ as in the original NMS, $b_i$ and $s_i$ are copied into the temporary sets ${\mathcal{T}}_b$ and ${\mathcal{T}}_s$, respectively. Repeat lines 10–18 until set $\mathcal{B}$ becomes empty. Finally, remove the elements in ${\mathcal{T}}_b$ (resp., ${\mathcal{T}}_s$) from $\mathcal{D}$ (resp., $\mathcal{S}$). Then, sets $\mathcal{D}$ and $\mathcal{S}$, respectively, become the set of final detection boxes and the set of the corresponding confidence scores (Lines 20 and 21).

It is noteworthy that by replacing $IoU(M,b_i)$ and $N_t$ with $rIoU(M,b_i;\lambda )$ in (3) and $N_t(M,b_i)$ in (4), respectively, one can readily extend Algorithm 1 to the entire procedure of the proposed NMS scheme.

3. Experimental Results and Discussions

In this section, we examine the performance of the proposed NMS and compare it with the conventional NMS approaches, i.e., the greedy NMS [29], DIoU NMS [32], Soft NMS [31], Exponential NMS [33], WBF [30], and Confluence NMS [34]. To this end, we evaluate these NMS methods on a dataset of steel surface defects using the well-known object detection models YOLOv7 and EfficientDet. The details of the datasets and the hyperparameters of the models are described in the later sections.

3.1. Dataset Description

In this paper, we adopt two popular steel surface defect datasets for the purpose of cross-validation of the performance of the proposed NMS. The first dataset is the NEU-DET [36] dataset, which is the most popular steel surface defect dataset from Northeastern University. The NEU-DET dataset has 1800 gray-scale images in six classes (i.e., each class has 300 images with $200\times 200$ resolution). Figure 2 illustrates the examples for the six classes of the NEU-DET dataset, including crazing, inclusion, patches, pitted surface, rolled-in scale, and scratches. Among the 1800 images, 1260, 180, and 360 images are used for training, validation, and test images, respectively.

The second dataset is the GC10-DET [37] steel surface defect dataset from Tianjin University. The GC10-DET dataset has 2300 gray-scale images in ten classes. Unlike the NEU-DET dataset, the GC10-DET dataset is unbalanced, i.e., the defect classes have different numbers of images. Figure 3 illustrates the examples for the ten classes of the GC10-DET dataset, including punching, weld line, crescent gap, water spot, oil spot, silk spot, inclusion, rolled pit, crease, and waist folding. Among the 2300 images, 1610, 230, and 460 images are used for training, validation, and test images, respectively.

3.2. Hyperparameters for Models and NMS Schemes

In this work, we conduct experiments on two well-known real-time object detection models, YOLOv7 [16] and EfficientDet [38], with several BoF (Bag of Freebies) and BoS (Bag of Specials) techniques. Specifically, for the YOLOv7 model, the mosaic data augmentation technique is adopted while the EfficientDet mode utilizes the cosine annealing scheduler [39]. Furthermore, in the case of the EfficientDet model, the d0 model is employed with consideration for the dataset size. The details of models including the number of parameters, fine-tuned hyperparameters, augmentation methods, BoF, and BoS for each model are summarized in

Table 1. The hyperparameters of each object detection model.

	YOLOv7	EfficientDet
Data augmentation	Random horizontal flip Random translation	Random horizontal flip
Number of parameters	36.6 M	3.9 M
Learning rate	0.005	0.0001
Batch size	8	8
Optimizer	SGD	Adam
Epochs	130	120
Warm-up epochs	5	5
BoF/BoS	Mosaic data augmentation	Cosine annealing scheduler

.

The IoU threshold and the confidence score threshold for all NMS schemes are set to 0.5 and 0.25, respectively. In addition, the Gaussian penalty parameter $\sigma$ in Soft NMS and Confluence NMS, the exponent parameter n in Exponential NMS, the confluence threshold $C_t$ in Confluence NMS, and the regularizing parameter $\lambda$ in the proposed NMS for each pair of object detection model and dataset are summarized in

Table 2. The hyperparameters of each NMS method.

Model (Dataset)	Soft NMS	Exponential NMS	Confluence NMS	Proposed NMS
YOLOv7 (NEU-DET)	$\sigma =0.7$	$n=1.1$	$\sigma =0.4$, $C_t=0.5$	$\lambda =0.4$
EfficientDet (NEU-DET)	$\sigma =0.8$	$n=1.3$	$\sigma =0.2$, $C_t=0.5$	$\lambda =0.4$
YOLOv7 (GC10-DET)	$\sigma =0.7$	$n=1.1$	$\sigma =0.5$, $C_t=0.5$	$\lambda =0.4$

.

3.3. Performance Evaluation Metric

In this paper, we utilize recall, precision, and F1-score as evaluation metrics to verify the performance of the proposed NMS approach. In addition, for a more thorough comparison, the three metrics are further divided into micro-average and macro-average. The mathematical formulation for the metrics of micro-average recall, micro-average precision, and micro-average F1-score are presented in Equations (5)–(7).

• Micro-Average Recall

  $$Micr{o}_R=\frac{{\sum }_{i=1}^{N}T{P}_i}{{\sum }_{i=1}^{N}T{P}_i+{\sum }_{i=1}^{N}F{N}_i},$$

  (5)
• Micro-Average Precision

  $$Micr{o}_P=\frac{{\sum }_{i=1}^{N}T{P}_i}{{\sum }_{i=1}^{N}T{P}_i+{\sum }_{i=1}^{N}F{P}_i},$$

  (6)
• Micro-Average F1-score

  $$Micr{o}_{F_1}=2\times \frac{Micr{o}_R\times Micr{o}_P}{Micr{o}_R+Micr{o}_P},$$

  (7)

where N is the number of classes in the dataset (e.g., $N=6$ for the NEU-DET dataset and $N=10$ for the GC10-DET dataset) and $T{P}_i$ (resp., $F{P}_i$) denotes true positives (false positives, resp.) in class i, i.e., boxes predicted as class i and with an IoU (with the ground truth) exceeding (resp., falling behind) 0.5. In addition, $F{N}_i$ represents false negatives in class i, meaning the objects in class i that the model fails to predict.

Similarly, the macro-average recall, macro-average precision, and macro-average F1-score metrics can be mathematically formulated as in Equations (8)–(10) shown below.

• Macro-Average Recall

  $$Macr{o}_R=\frac{1}{N}\sum _{i=1}^N\frac{T{P}_i}{T{P}_i+F{N}_i},$$

  (8)
• Macro-Average Precision

  $$Macr{o}_P=\frac{1}{N}\sum _{i=1}^N\frac{T{P}_i}{T{P}_i+F{P}_i},$$

  (9)
• Macro-Average F1-score

  $$Macr{o}_{F_1}=2\times \frac{Macr{o}_R\times Macr{o}_P}{Macr{o}_R+Macr{o}_P},$$

  (10)

3.4. Results and Discussions

3.4.1. Quantitative Result

In this section, we have presented the results of evaluation metrics obtained on the NEU-DET and GC1-DET datasets by incorporating the NMS approaches (the greedy NMS, DIoU NMS, Soft NMS, Exponential NMS, WBF, Confluence NMS, and the proposed NMS) into the YOLOv7 and EfficientDet models.

The micro- and macro-average F1-score metric results on the NEU-DET dataset using the YOLOv7 model are presented in

Table 3. Results using YOLOv7 and EfficientDet with various NMS methods.

Models (Datasets)	Metrics	Greedy NMS	DIoU NMS	Soft NMS	Exponential NMS	WBF	Confluence NMS	Proposed
YOLOv7 (NEU-DET)	${\mathit{Micro}}_{F_1}$	0.6642	0.6618	0.6680	0.6697	0.6711	0.6494	0.6791
	${\mathit{Macro}}_{F_1}$	0.6686	0.6663	0.6768	0.6751	0.6753	0.6518	0.6860
EfficientDet (NEU-DET)	${\mathit{Micro}}_{F_1}$	0.5947	0.5883	0.6488	0.6566	0.5955	0.5825	0.6604
	${\mathit{Macro}}_{F_1}$	0.6009	0.5952	0.6491	0.6541	0.6016	0.5908	0.6631
YOLOv7 (GC10-DET)	${\mathit{Micro}}_{F_1}$	0.6373	0.6368	0.6346	0.6386	0.6364	0.6032	0.6396
	${\mathit{Macro}}_{F_1}$	0.6215	0.6214	0.6141	0.6226	0.6198	0.5938	0.6232

Bold in table means the highest score.

. It is observed in Table 3 that the proposed NMS approach achieves performance improvements of 2.24%, 2.61%, 1.66%, 1.40%, 1.19%, and 4.57% when compared to the greedy NMS, DIoU NMS, Soft NMS, Exponential NMS, WBF, and Confluence NMS, respectively, in terms of the micro-average F1-score. In addition, regarding the macro-average F1-score, the proposed NMS approach achieves performance improvements of 2.60%, 2.96%, 1.36%, 1.61%, 1.58%, and 5.25% when compared, respectively, to the greedy NMS, DIoU NMS, Soft NMS, Exponential NMS, WBF, and Confluence NMS.

Similar results with the EfficientDet model for the NEU-DET dataset and the YOLOv7 model for GC10-DET can also be found in Table 3. Specifically, Table 3 illustrates that the proposed NMS approach with the EfficientDet model achieves up to approximately a 13% and 12% performance improvement in terms of the micro-average F1-score metric and micro-average F1-score metric, respectively, for the NEU-DET dataset. In addition, the YOLOv7 model with the proposed NMS achieves up to approximately a 6% and 5% performance improvement in terms of the micro-average F1-score metric and micro-average F1-score metric, respectively, for the NEU-DET dataset. Thus, one can conclude that the proposed NMS considerably and consistently outperforms the existing NMS techniques in terms of the micro- and macro-average F1-score metrics when integrated with both the YOLOv7 and EfficientDet object detectors.

3.4.2. Qualitative Result

In this section, we have analyzed the qualitative performance of the proposed NMS against the existing NMS approaches. The detection images acquired by the YOLOv7 and EfficientDet models with each NMS approach are shown in Figure 4, Figure 5, Figure 6 and Figure 7. The results regarding the Scratches class, obtained by YOLOv7, are presented in Figure 4. It is observed in Figure 4 that the results from the proposed NMS prediction are the most similar to the ground truth. However, the predictions given by the greedy NMS, DIoU NMS, Soft NMS, and Exponential NMS approaches look inaccurate. The detection images of the Patches class obtained by YOLOv7, presented in Figure 5, also depict that the results from the proposed NMS method are again the most similar to the ground truth when compared to other NMS approaches.

Similar observations can be found in Figure 6 and Figure 7 in which we have presented the detection images obtained by EfficientDet with various NMS approaches for predicting the Pitted_surface and Scratches classes. As shown in Figure 6, the greedy NMS and DIoU NMS leave more false positives (four BBoxes) than the other NMS approaches. Further, the Soft NMS and Exponential NMS approaches also failed to predict the images accurately with respect to the ground truth. In contrast, the proposed NMS successfully predicts the ground truth by suppressing all false positives. The detection results of Scratches presented in Figure 7 also depict that the greedy NMS, DIoU NMS, Soft NMS, and Exponential NMS failed to obtain predictions similar to the ground truth. The proposed method suppresses false positives well and shows similar results to the ground truth.

Through the qualitative evaluation, we can observe that the proposed method predicts the most similar results to the ground truth by suppressing false positives well, regardless of the object detection model.

3.5. Ablation Study

The proposed NMS approach consists of three strategies: IoU regularization (IR), threshold adjustment (TA), and comparison rule modification (CRM). Thus, to evaluate the efficacy of each strategy, we conducted an ablation study using the YOLOv7 and EfficientDet models on the NEU-DET dataset. The experiments are conducted in different setups to confirm the effectiveness of the proposed NMS algorithm. First, we have implemented each strategy of the proposed NMS alone and identified the detection performance of the model. Next, we incorporated a combination of two strategies from the proposed NMS and assessed the performance. The results in

Table 4. Results for the ablation study.

Method			YOLOv7						EfficientDet
IR	TA	SRM	${\mathit{Micro}}_{\mathit{P}}$	${\mathit{Micro}}_{\mathit{R}}$	${\mathit{Micro}}_{{\mathit{F}}_{\mathbf{1}}}$	${\mathit{Macro}}_{\mathit{P}}$	${\mathit{Macro}}_{\mathit{R}}$	${\mathit{Macro}}_{{\mathit{F}}_{\mathbf{1}}}$	${\mathit{Micro}}_{\mathit{P}}$	${\mathit{Micro}}_{\mathit{R}}$	${\mathit{Micro}}_{{\mathit{F}}_{\mathbf{1}}}$	${\mathit{Macro}}_{\mathit{P}}$	${\mathit{Macro}}_{\mathit{R}}$	${\mathit{Macro}}_{{\mathit{F}}_{\mathbf{1}}}$
-	-	-	0.5928	0.7552	0.6642	0.6048	0.7568	0.6686	0.4730	0.8009	0.5947	0.4921	0.7987	0.6009
✓	-	-	0.6031	0.7529	0.6697	0.6162	0.7539	0.6743	0.5233	0.7728	0.6241	0.5405	0.7715	0.6280
-	✓	-	0.6192	0.7447	0.6762	0.6385	0.7457	0.6837	0.5741	0.7482	0.6497	0.5910	0.7455	0.6536
-	-	✓	0.6060	0.7529	0.6715	0.6215	0.7545	0.6777	0.4750	0.8009	0.5963	0.4934	0.7987	0.6021
✓	✓	-	0.6257	0.7400	0.6781	0.6428	0.7399	0.6841	0.6024	0.7400	0.6642	0.6152	0.7374	0.6664
✓	-	✓	0.6085	0.7517	0.6726	0.6232	0.7526	0.6780	0.5166	0.7810	0.6219	0.5342	0.7799	0.6268
-	✓	✓	0.6203	0.7423	0.6759	0.6400	0.7425	0.6831	0.5806	0.7459	0.6529	0.5956	0.7431	0.6562
✓	✓	✓	0.6258	0.7423	0.6791	0.6445	0.7426	0.6860	0.5954	0.7412	0.6600	0.6106	0.7384	0.6631

Bold in table means the highest score.

show that using each of the three proposed strategies improves micro- and macro-average precision but degrades micro- and macro-average recall. This is because the proposed strategies strictly suppress BBoxes, and this may cause a decrease in true positives. However, since the performance improvement achieved in precision is greater than the performance degradation in recall, the micro- and macro-average F1-score for each strategy is substantially increased. Similar trends are observed in the cases of combinations of two and all three of the proposed strategies. The best results in the micro- and macro-average F1-scores with YOLOv7 are obtained by the proposed NMS scheme, but, in the case of the micro- and macro-average F1-scores with EfficientDet, the best result is obtained with the proposed NMS without suppression rule modification.

4. Conclusions and Discussions

This paper proposed a novel NMS approach to enhancing the object detection method for identifying steel surface defects. The proposed NMS approach presented three modifications: IoU regularization, threshold adjustment, and comparison rule modification. The experiments were carried out by incorporating the NMS approaches into the YOLOv7 and EfficientDet models on the NEU-DET and GC10-DET datasets, i.e., two well-known steel surface defect detection datasets. The proposed NMS outperformed the existing NMS approaches in terms of the micro- and macro-average F1-score with the YOLOv7 and EfficientDet models. Further, it was also observed that the proposed NMS technique dominates the other NMS approaches qualitatively.

The main reason behind the enhanced performance of the proposed NMS is suppressing more false positive BBoxes. However, this may lead to a reduction in the true positives. Therefore, as future research topics, we will improve the performance of the proposed NMS to suppress false positive BBoxes more while simultaneously preserving true positive BBoxes and to consider the theoretical analysis of the proposed NMS.