Determination of Decarburization Depth Base on Deep Learning Methods

Abstract

In the heat treatment of steel, decarburization is a serious issue that leads to poor wear resistance and low fatigue life. At present, the decarburization depth was determined using a visual estimation by the human eye, and the software estimation was determined through traditional image analysis. Therefore, decarburization depth analysis remains limited in experts and traditional algorithms. Artificial intelligence is a general-purpose technology that has a multitude of applications. This paper uses the concept of deep learning to propose a decarburization layer detector (DLD) that can determine the depth of decarburized layers. This DLD system boasts high performance, real-time, low learning, and computation costs. In addition, we used several kinds of decarburized layers images to compare the proposed method with other deep learning network architectures. The experimental results show that the proposed method yields a detection accuracy of 92.97%, which is higher than existing methods and boasts computational demands which are far lower than other network architectures. Therefore, we propose a novel system for automatic decarburization depth determination as an application for metallographic analysis.

1. Introduction

Steel is often categorized according to its carbon content. In general, steel can be divided into three categories based on carbon levels, including low-carbon, medium-carbon, and high-carbon. Low-carbon steel consists of less than 0.30% carbon. Medium-carbon steel consists of 0.30–0.60% carbon. Additionally, high-carbon steel contains more than 0.60% carbon. The strength and hardness of the steel decrease as the amount of carbon content decreases. Therefore, all steel contains at least some amount of carbon and is defined as an alloy of iron and carbon. Excessive decarburization can result in unstable steel, which leads to reduced product performance. In addition, it can also lead to various problems with machinery created from steel. Thus, determining the depth of decarburization is important in various fields.

Decarburization is the process in which steel is heated during heat treatment, leading to oxidation and the loss of carbon. Generally speaking, decarburization occurs in an improperly controlled heat treatment process. Decarburization is a serious issue as the near-surface carbon can be lost from steel at high temperatures, and the surface of the steel has a lower hardness and tensile strength. Industry standards provide guidelines for the measurement of decarburization depth which are set in ASTM E1077 (standard test methods for estimating the depth of decarburization of steel specimens) [1] and ISO 3887 [2]. The two industry standards describe several methods for determining the decarburization depth. There are four basic procedures for determining decarburization depth: screening method, microscopical method, microindentation hardness method, and chemical analytical method. These methods can be employed with any cross-sectional shape. Through the availability of equipment, the microscopical method generally reveals a lesser depth of decarburization than chemical analytical methods in certain simple shapes. The microindentation hardness method measures the hardness of steel on a microscopic scale. In addition, the International Standard ISO 3887 used two main methods for measuring the decarburizing depth: micro-structural observation and the plotting hardness profile. Both of them are time-consuming and expensive: those methods cannot examine all materials.

In addition to ASTM E1077 and ISO 3887, some researchers have begun to explore the utilization of machine vision and image processing in microstructure science. Over the past few decades, optical decarburization measurements have been preferred for quality tests, modeling, or indirect methods over carbon steel products. The measurements are commonly performed by metallographs and can be supported by image processing. Image processing allows for the collection of multiple measurements from digital micrographs. Huang et al. [3] proposed a digital image analysis approach by comparing the variation in the image gray gradient for determining the decarburization depth. Chávez-Campos et al. [4] used the image processing method to examine the impact of the segmentation method parameters on the precision of decarburization depth measurements. However, the performance of the image processing method is often time-consuming and inaccurate to determine decarburization depth. With the increasing amount of data and computing resources, traditional image processing methods have been replaced by artificial intelligence (AI). Nowadays, deep learning has had success in many fields, including pattern recognition, object recognition, and audio processing. The main function of deep learning methods is that they enable real-time analysis and precise measurements. Because deep learning is totally excellent in terms of accuracy when trained with massive amounts of data, its popularity has increased more and more in different fields.

In general, deep learning methods for target detection are dominated by CNN-based algorithms, which can be mainly based on YOLO (you-only-look-once) [5] series. After the YOLO model was developed, Redmon et al. also proposed YOLOv2 [6] and YOLOv3 [7] object detection models successively. YOLOv3 is different from other existing methods, as detection at different scales makes it easier to detect various objects. After YOLOv3, a series of methods were proposed, including YOLOv4 [8] and YOLOv6 [9]. They effectively integrate all kinds of advantages of deep learning to create an optimal neural network for object detection. Until now, the YOLO series were already developed to YOLOv7 [10]. YOLOv7 proposes several bag-of-freebies methods and effectively integrated deep learning tricks to greatly promote performance without increasing the inference cost. To sum up, the YOLO series has become a mainstream trend in the object detection of natural images. For metallographic analysis, Lee et al. [11] proposed a fast image classifier (FIC) to classify grain size based on the concept of YOLO. Compared with the classical network of deep learning, the performance of the FIC algorithm was higher than the existing methods. However, to the best of our knowledge, no paper discusses deep learning models that can determine decarburization depth. To address the above problem, we propose the decarburization layer detector (DLD) to determine the decarburization depth based on CNN-based. This detector provides an automatic determination of the decarburization layer depth by providing metallic microstructure images. In this work, the proposed method is to verify the determination of the decarburization depth based on the deep learning method. The research work and contributions of this paper include the following aspects:

• Based on a convolutional neural network (CNN) model, we proposed the use of a decarburization layer detector (DLD) to determine the depth of decarburized layers.
• For real-time detection, we used a neural network with only 48 layers to replace the existing methods for the determination of decarburization depth.
• The proposed method decreases the computing cost while improving the performance of the DLD system over existing methods.
• To determine the decarburization depth, we also proposed a re-training strategy to make the DLD system more robust.

After the introduction in Section 1, Section 2 will describe in detail the proposed strategy for determining decarburization depth. The experimental results and comparisons with other representative object detection methods are discussed in Section 3, and Section 4 gives the conclusion and proposes future work.

2. Proposed Method

YOLO series-based object detectors are clearly faster and more accurate than other existing methods for the 80 classes in the MS COCO dataset. However, for determining decarburization depth, it can only to detect one target category. To find out the proper decarburization depth detection, we reduce the architecture of YOLOv4 and modify the original three output layers of YOLOv4 two output layers. This section describes the DLD framework for determining the decarburization layer depth with metallic microstructure images. The solutions and steps are described in detail below.

2.1. Architecture of Proposed Method

In this paper, we combine several famous CNN networks to extract features from decarburization layer images. The residual networks (ResNet) [12] and cross-stage partial networks (CSPNet) [8] are mainly adopted to extract local descriptors from each image. ResNet, one of the outstanding architectures in object detection, proposed shortcut connections to bypass a layer and move to the next layer in the sequence, which makes it possible to solve the problem of the vanishing gradient and still achieve a compelling performance. The CSPNet divides the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The system framework for determining the decarburization layer depth is shown in Figure 1.

There are no small targets of decarburization depth in the decarburization layer images. Therefore, the YOLO series model still did not extract the proper features for determining the decarburization layer depth. Unlike the YOLO series model, DLD predicts boxes in two different scales to replace the three scales (small, medium, and large) of the original model. In other words, DLD divides the input image and uses two different grid sizes to detect medium and large objects, respectively. While reducing the neural network architecture, it also reduces the computing time. In addition, in contrast to other deep learning network methods, the DLD model is not down-sampled to reduce more information at the first level. On the contrary, this network starts with a convolutional layer with 32 filters to reserve more information. In addition, the DLD model uses three ResNets and CSPNets to extract more information from the backbone networks. The DLD model used various skills of deep learning to extract different features such as edges, corners, and color patterns from the decarburization layer images. The framework of DLD is defined in more detail in

Table 1. The framework of the decarburization layer detector model.

Layer	Operation Type	Input	Filter	Size/Stride	Output	Layer
0	Convolution	800 × 544 × 3	32	3 × 3/1	800 × 544 × 32
1	Convolution	800 × 544 × 32	64	3 × 3/2	400 × 272 × 64
2	Convolution	400 × 272 × 64	32	1 × 1/1	400 × 272 × 32	ResNet
3	Convolution	400 × 272 × 32	64	3 × 3/1	400 × 272 × 64
4	Shortcut	400 × 272 × 64	-	-	400 × 272 × 64
5	Convolution	400 × 272 × 64	128	3 × 3/2	200 × 136 × 128
6	Convolution	200 × 136 × 128	64	1 × 1/1	200 × 136 × 64	ResNet
7	Convolution	200 × 136 × 64	128	3 × 3/1	200 × 136 × 128
8	Shortcut	200 × 136 × 128	-	-	200 × 136 × 128
9	Convolution	200 × 136 × 128	64	1 × 1/1	200 × 136 × 64	ResNet
10	Convolution	200 × 136 × 64	128	3 × 3/1	200 × 136 × 128
11	Shortcut	200 × 136 × 128	-	-	200 × 136 × 128
12	Max pooling	200 × 136 × 128	-	2 × 2/2	100 × 68 × 128
13	Convolution	100 × 68 × 128	128	3 × 3/1	100 × 68 × 128	CSPNet
14	Route	13	-	-	100 × 68 × 64
15	Convolution	100 × 68 × 64	64	3 × 3/1	100 × 68 × 64
16	Convolution	100 × 68 × 64	64	3 × 3/1	100 × 68 × 64
17	Concatenation	15,16	-	-	100 × 68 × 128
18	Convolution	100 × 68 × 128	128	1 × 1/1	100 × 68 × 128
19	Concatenation	13,18	-	-	100 × 68 × 256
20	Max pooling	100 × 68 × 256	-	2 × 2/2	50 × 34 × 256
21	Convolution	50 × 34 × 256	256	3 × 3/1	50 × 34 × 256	CSPNet
22	Route	21	-	-	50 × 34 × 128
23	Convolution	50 × 34 × 128	128	3 × 3/1	50 × 34 × 128
24	Convolution	50 × 34 × 128	128	3 × 3/1	50 × 34 × 128
25	Concatenation	23,24	-	-	50 × 34 × 256
26	Convolution	50 × 34 × 256	256	1 × 1/1	50 × 34 × 256
27	Concatenation	21,26	-	-	50 × 34 × 512
28	Max pooling	50 × 34 × 512	-	2 × 2/2	25 × 17 × 512
29	Convolution	25 × 17 × 512	512	3 × 3/1	25 × 17 × 512	CSPNe
30	Route	29	-	-	25 × 17 × 256
31	Convolution	25 × 17 × 256	256	3 × 3/1	25 × 17 × 256
32	Convolution	25 × 17 × 256	256	3 × 3/1	25 × 17 × 256
33	Concatenation	31,32	-	-	25 × 17 × 512
34	Convolution	25 × 17 × 512	512	1 × 1/1	25 × 17 × 512
35	Concatenation	29,34	-	-	25 × 17 × 1024
36	Convolution	25 × 17 × 1024	512	1 × 1/1	25 × 17 × 512
37	Convolution	25 × 17 × 512	512	3 × 3/1	25 × 17 × 512
38	Convolution	25 × 17 × 512	256	1 × 1/1	25 × 17 × 256
39	Convolution	25 × 17 × 256	512	3 × 3/1	25 × 17 × 512
40	Convolution	25 × 17 × 512	18	1 × 1/1	25 × 17 × 18
41	Large object detection
42	Route	38	-	-	25 × 17 × 256
43	Convolution	25 × 17 × 256	128	1 × 1/1	25 × 17 × 128
44	2 × upsample	25 × 17 × 128	-	-	50 × 34 × 128
45	Route	44,26	-	-	50 × 34 × 384
46	Convolution	50 × 34 × 384	256	3 × 3/1	50 × 34 × 256
47	Convolution	50 × 34 × 256	18	1 × 1/1	50 × 34 × 18
48	Medium object detection

. To improve the accuracy of decarburization layer detection, DLD removes the unnecessary network architecture to detect decarburization depth. In addition, the decarburization layer can detect more accurately and reduce the computation cost. Therefore, we only used a 48-layer network architecture to replace the traditional assessment methods for the determination of decarburization depth.

2.2. Re-Training Strategy

Modern deep learning models have millions of parameters. At the same time, training the DLD network architecture requires many labeled data and much computing time. Transfer learning is a machine learning technique where we can reuse a pre-trained model as the starting point for a model on a new task. In this paper, we propose a retraining strategy to improve efficiency in the DLD model. As a matter of fact, there are not enough decarburization layer samples to train the DLD models. Therefore, after each testing phase, we could use the testing results to make the DLD model more robust, as shown in Figure 2. The poor results mean that the recognition results of the decarburization depth are inaccurate for experienced specialists. In Figure 3, the red boxes show poorer recognition results by applying the DLD method. On the other hand, the yellow boxes present a better result with the manually confirmed correction. For poor results, we revise the error images to implement retraining. To save much manual labeling time, automated pre-labeling employs the re-training models to label the test images automatically. The DLD models achieve better performance for determining decarburization depth through the use of re-training strategies that benefit the final application.

3. Experimental Results and Analysis

To evaluate the DLD model, we describe the used metrics and datasets in this section. We collected the decarburization layer dataset from the metallographic analysis laboratory of the Metal Industries Research and Development Centre (MIRDC) [13]. These experiments were conducted in CUDA C++ API on a computer with NVIDIA 3060 GPUs. The Zeiss Axiovert 200 Mat optical microscope was employed to collect the decarburization layer images, as shown in Figure 4. The average precision (AP) [14] was a metric used to evaluate the DLD model. Moreover, we compared the results with the YOLO series models in terms of AP and inference speed. Finally, we divided the decarburization layer dataset into training and testing sets by randomly splitting the dataset. In our experiments, the training and testing data were selected at a rate of 80:20 for training and testing, and five-fold cross-validation was conducted.

3.1. Decarburization Layer Dataset

Up to this time, there have been no appropriate decarburization layer datasets for deep learning. Thus, we collected the decarburization layer images from MIRDC [13]. The decarburization layer datasets contain eight different classes of decarburization under the optical microscopy 100× magnification, including SAE 9254 (518), S45C (497), SAE 8620H (730), SCM418H (420), S53C (554), S55C (503), 5120H (519), and 4320H (468) steels, as shown in Figure 5. The number of decarburization layer images in each class is shown in parentheses. Each steel class varies from 420 to 730 decarburization layer images. In general, more data resulted in better performance of the training model [15]. Therefore, data augmentation was used to expand the number of our training dataset. In our dataset, we only used the flipping policies of data augmentation techniques for decarburization layer images. Thus, the total images of the decarburization layer dataset were composed of 8418 images in the 24-bit JPEG format.

3.2. Validation of Proposed Method

The DLD system has been completed based on the concept of the graphical user interface (GUI), as shown in Figure 6. The GUI system of the DLD model consists of four parts which are called the main window and three control blocks, respectively. Initially, the decarburization layer images are read and displayed on the main window. Next, the user can explore the detection result of the video or image in the main window with the two control blocks of video and image recognition. However, the up and down adjustment buttons were used to correct the image labeling of error detection in the re-training strategy.

In this work, the size of decarburization layer images was resized into three different sizes-416 × 288, 608 × 416, and 800 × 544.

Table 2. The detection results of eight different decarburization classes.

Resolution	416 × 288	608 × 416	800 × 544
SAE 9254	85.61%	88.04%	91.78%
S45C	84.79%	90.39%	92.17%
SAE 8620H	95.53%	99.08%	99.45%
SCM418H	93.55%	95.48%	98.12%
S53C	90.78%	93.72%	96.88%
S55C	84.35%	88.19%	91.31%
5120H	78.54%	82.78%	86.94%
4320H	81.47%	85.61%	87.12%

shows the detection results on our decarburization layer dataset for various image resolutions with a mean average precision (mAP). In this experiment, the better performance of the decarburization layer detection could be achieved through the use of an 800 × 544 size image. However, the decarburization layer images with 416 × 288 were smaller than other size images to detect effectively. Overall, using the 800 × 544 size image, our proposed method achieved better performance than the 416 × 288 size image, improving the mAP from 86.82 to 92.97%. In addition, the results also showed that most of the SAE 9254, S45C, SAE 8620H, SCM418H, and S53C steels successfully detected the decarburization layer depth by the proposed method. The lower performance occurred on S55C, 5120H, and 4320H steels that were difficult to distinguish with the decarburization layer depth, even with the human eye, as shown in Figure 7. In Table 2, we also show the results for the different steel categories to demonstrate the strong performance of decarburization layer detection.

3.3. Comparison with Other Existing Method

Not only the performance but also the computation cost must be considered for neural network architectures. Hence, we employed our decarburization layer dataset to compare our method with state-of-the-art YOLO series in terms of recognition accuracy and FPS (frames per second), as shown in

Table 3. The detection result of different methods.

Method	mAP	Precision	Recall	F1-Score	FPS
YOLOv7	95.05%	0.96	0.93	0.95	31
YOLOv7-tiny	88.46%	0.84	0.89	0.87	85
YOLOv6	92.97%	0.91	0.93	0.92	55
YOLOv6-tiny	88.14%	0.85	0.88	0.87	62
YOLOv4	91.51%	0.92	0.92	0.91	22
YOLOv4-tiny	86.12%	0.87	0.84	0.85	67
Scaled-YOLOv4	93.64%	0.96	0.91	0.93	37
Proposed method	92.78%	0.97	0.98	0.97	78

. A similar result could be obtained by the use of YOLOv7, YOLOv6, Scaled-YOLOv4, and the proposed method, which resulted in better accuracy. The other evaluation values (the precision, recall, and F1 score) are also listed in Table 3. All detection methods were evaluated under the same training and test sets, and the metric results were performed under exactly the same standard. These detection methods included YOLOv4, YOLOv4-tiny [8], YOLOv6, YOLOv6-tiny [9], YOLOv7, YOLOv7-tiny [10], and the proposed method. From Table 3, our proposed method was better than the YOLO-tiny series methods and similar to the YOLO series methods. In conjunction with the results in Table 3, it effectively proves that the DLD algorithm in this paper had good performance in decarburization layer detection. The YOLO-tiny series methods experienced the loss of more information because of down-sampling to reduce the computation cost at the first level of the neural network. The experimental results show that the DLD model is the faster detector, while the YOLO series methods are the slower, mostly because YOLO series methods have several million parameters, leading to greater computational costs. In addition, although parts of the YOLO series methods have higher accuracy, their FPS is lower than our proposed method for determining the decarburization depth. Thus, the performance of the DLD method is not the best, but its accuracy approximates that of the YOLO series methods, making it still useful for real-time image detection.

In these experiments, some ambiguities concerning the decarburization layer images were still not accurately detected by the DLD model. An analysis of the final results shows that some decarburization layer images were difficult to distinguish even manually (as shown in Figure 7) and that some images were not part of the training data. In future work, solving these problems will be quite a challenging job. Generally speaking, in order to improve the object detection performance and accuracy of the neural network model, it is necessary to increase the number of training images first. Therefore, we will continue to extend the dataset with a re-training strategy and integrate more advanced neural network architectures to further improve the DLD system.

4. Conclusions

In this work, we proposed the decarburization layer detector module, which could effectively fuse features from different stages of the network to extract features that were crucial to determining the decarburization layer for carbon steel. The DLD algorithm was experimentally analyzed on MIRDC datasets, and the mAP of this paper’s algorithm was 92.97%. The results show that the DLD algorithm can better utilize the feature information in the shallow feature layer in the large and medium target detection process and has better detection performance for the determination of decarburization depth. In addition, we collected the decarburization layer dataset to evaluate the effectiveness of DLD and used a neural network with only 48 layers to replace the existing methods. The experimental results also show that the DLD model provides higher performance and even outperforms state-of-the-art algorithms. To promote the steel products with more accurate determination for decarburization layer depth and relieve the operators’ burden, an intelligent de-carburization layer detection system is essential. How to reduce redundancy and optimize the network structure while ensuring accuracy will be the main direction of future research.