Deep-Learning-Based Surface Texture Feature Simulation for Surface Defect Inspection

Abstract

In this research, a simulation system based on a physical model and its lighting feature is developed to perform three-dimensional model creation, and graphics software is used to randomly generate a simulated surface with defects, which also cooperates with the virtual environment to reproduce the original environment. Furthermore, the use of a generative adversarial network to optimize the virtual dataset created symmetrically by the system is studied in order to reduce the effect of the difference between the real and virtual images. This system compensates for the condition of data imbalance occurring between qualified products and defective products in the production line, and a large amount of random data with and without defects can be created. In addition, the process of the database creation is classified and marked, such that complicated and time-consuming preliminary steps can be reduced; therefore, the data collection cost can be significantly reduced and the uncertainly associated with manual operation is also reduced. When a simulated textured surface generated from this system is used to perform training, the inspection background accuracy reaches 98%, and the accuracy also reaches 78% in real defect inspection process; therefore, the location of the defect can be determined completely.

1. Introduction

The quality of manual inspection is often subjective and unstable. The main objective of Automatic Optical Inspection (AOI) is to replace manual inspection. According to the application method, AOI can also be divided into traditional optical inspection and deep learning auxiliary optical inspection. The former inspection method merely performs the actions of adjustment and segmentation according to the image threshold value, and when it encounters the situation of target defect and background devoid of threshold difference or sample color quality non-uniformity, such an inspection method will fail. For the present operating production lines that require the implementation of defect inspection in the industry, since a stable inspection environment cannot be provided and the condition of color difference in samples often occurs, other inspection methods have to be used and implemented. For supervised learning inspection, a large database is often required. Nevertheless, most of the production lines are subject to the conditions of environmental variation and data imbalance, such that data cannot be properly collected and processed, leading to failure in the inspection training. In this paper, a virtual texture simulation and generation system with the application of deep learning method is developed for the aforementioned condition; the target object for inspection refers to scratches on plastic housings. The deep learning auxiliary optical inspection is able to utilize a multi-layer complicated network design such that a large number of images can be inputted to analyze the pixel intensity and arrangement method of images, thereby allowing the training model to determine the features of a specific area. Such a method is equipped with the fine feature classification that cannot be achieved by the traditional optical inspection method. In addition, the deep learning auxiliary optical inspection method can also increase the color distribution acceptance level such that it is able to overcome the deficiency of the aforementioned traditional image determination. The datasets and training results of the training associated with the deep learning method are closely related to each other. If the training data amount involves any deviation or incomplete marking, then the entire set of training will fail. Most production lines, typically, are often subject to the condition of data imbalance, and when the number of datasets is small, the training may also lose its accuracy. In general, it is extremely difficult and time consuming to obtain a large number of accurate samples. In addition, when the manual marking is unclear and the type of defects and location of defects cannot be clearly defined, it is not possible to properly train the classifier. Consequently, there is a great challenge in the process of collection and preparation of datasets.

In the past, there have been numerous papers discussing the use of virtual or synthesized images to replace real images in order to increase the number of samples. The training methods used in defect detection are mostly unsupervised training or supervised training, and the inspection network uses, such as autoencoder and residual network, have been discussed in various applications. However, there has been little research on simulation images. Accordingly, for relevant applications where virtual or synthesized images are used to perform training, the sources of datasets mostly come from the application of currently existing open-source databases. If the target for inspection is not available in the existing datasets, then it is necessary to re-collect sample data; otherwise, initial training cannot be performed. If any synthesized data or generative adversarial network is to be used to perform conversion, then it still requires the step of pre-collection of data. From the above, it can be understood that the process of collection and preparation of datasets is complicated but necessary. Accordingly, in this paper, a virtual rendering system is designed that is able to reduce the time, labor and subsequent hassles associated with the environmental control required for the settling and installation of the light source environment and machines. In the system, fine and real-mimicking images can be generated by simply selecting the environmental parameters at the beginning such that a large number of datasets equipped with randomness and quality control can then be created easily. For virtual model texture mapping, in 2003, Lifeng Wang [1] and others presented a view-related displacement mapping paper, which mentioned that the microstructure of the surface will produce significant visual effects. This paper applied a method called viewpoint-related displacement, a technique of view-dependent displacement mapping (VDM), which models the displacement of a surface along the viewing direction. Unlike traditional displacement maps, VDM can efficiently render self-shadows and outlines. To use the substitute image, it is necessary to make up for the difference between the distribution of the substitute image and the predicted image. There are many cases where generative adversarial networks are used for image conversion. For example, in 2017, scholars Ashish Shrivastava [2] and others proposed to use adversarial Learning in Simulation and Unsupervised Images. To narrow the gap between simulated and real images, this proposes Simulated + Unsupervised (S + U) learning to add realism to the simulator without affecting the synthetic image annotations. In 2018, Jun-Yan Zhu [3] proposed to use interactive adversarial networks (cycle-consistent adversarial networks, Cycle-GAN) for unpaired image-to-image conversion, and proposed a method without paired patterns in the case of learning methods to transform images from the source domain to the target domain. This network architecture applies two complete adversarial networks and is trained with two transformation iterations to achieve robustness of two-way transformation. In addition, without applying adversarial network conversion, in 2018, Jonathan Tremblay [4] proposed to supplement the reality gap data by randomly generating patterns. This paper presented a system for training deep neural networks for object detection using synthetic images. To deal with variability in real-world data, the system relied on a source-domain randomization technique in which parameters of the simulator (such as lighting, pose, object texture, etc.) are randomized in a non-realistic way to force the neural network to learn the basic characteristics of the object of interest, and to explore the respective importance of each parameter and how the network is trained.

There are many ways to apply deep learning classification and detection. For example, in 2018, Abdul Mujeeb [5] proposed an unsupervised neural network learning method. This method does not need to rely on defective samples for training, and only generates a similarity score after training normal pictures. The score will indicate whether a defect was found. However, the training images in this method need to be sufficiently different. If the data set has fine textures, more images and pre-processing are required. In 2019, Chen Yifan [6], the senior member of our laboratory, proposed an automatic defect detection system based on deep convolutional neural network. Since it is not easy to observe scratches on textured surfaces with high roughness, and the features to be tested are too narrow, it is difficult to detect defects. The special network design method of his paper will be able to detect defects that cannot be detected by traditional machine vision algorithms. In 2019, scholar Fei Zhou [7] proposed a general automatic surface defect detection based on a bilinear model (double-visual geometry group16, D-VGG16). Through fully extracted global and local features from the bilinear model output to the soft-max function, after processing the heatmap with a threshold segmentation algorithm, it can automatically locate defects in the input image to achieve automatic classification of surface defects. Since most defect prediction methods require a large amount of data from the item under test for training, there are usually not enough training data. In 2019, scholars Jinyin Chen [8] and others proposed a network architecture to implement collective transfer learning for defect prediction networks, so data from other related projects were used to build the model. This proposes a collective training mechanism for defect prediction (CTDP), which makes the feature distributions of source and target items similar to each other through transfer learning, and comprehensively considers multiple source items to predict the target. Recently, deep-learning-based AOI inspection system have been emerging for manufacturers, e.g., eyeglass [9], mobile phone back glass [10], fabric [11], TFT-LCD [12], and wafer classification [13]. In 2022, our previous work [14] showed an AOI solution with a deep residual neural network for defect detection over complex backgrounds. Nevertheless, the data collection cost is a significant factor to be considered for model training.

In this paper, the subject of study is a plastic housing with a textured surface. The outer surface has a texture of irregular grain shape. If the object has a scratch defect, it would occur between the texture, and such a scratch defect often has an elongated shape, such that it cannot be detected easily via the manual inspection method, and the use of conventional machine vision algorithms still cannot detect such defects easily. Furthermore, as most of the production lines nowadays specify a certain requirement on the yield rate, it means that the defective product quantity accounts for an absolute minority. Accordingly, during the preparation of datasets, the unfavorable situation of failure of training due to quantity imbalance would occur. It is necessary to find ones with defects from a large quantity of products first, followed by performing defect marking, which is a time-consuming non-automatic process, and is also considered to be a major challenge faced during the current implementation of deep learning inspection.

2. Materials and Methods

The content of this paper can be mainly divided into two parts, i.e., surface description and the application of deep learning after the creation of the simulation images to perform inspection and conversion. The simulated surface creation uses OpenGL to perform rendering and Figure 1 shows the framework of this research.

2.1. Surface Information

The target object for inspection is shown in Figure 2. Its surface has a fine textured pattern and a scratch defect is shown in Figure 3. From Figure 3, it can be observed that the surface scratch is a defect with an elongated and continuous feature. Since the textured background is complicated, during the marking of feature on this product, it is likely to cause confusion between the defect and the background. In this research, a 3D microscope is used to perform surface undulation measurements on the defect first, in order to determine the defect surface feature. As shown in Figure 4a, the scratch in the image was made manually by hand, and the red color in the altitude chart represents an area of higher surface height, and the blue color represents an area of lower height. Figure 4c refers to the red line area in Figure 4b and it is a surface height graph.

From a number of 3D microscope measurement maps, it can be observed that the texture height interval is random and broad, and it is distributed approximately between 60 µm and 100 µm. In addition, the defect area also has the same depth as the surface of the textured surface; therefore, the height feature is extremely small, and the use of one single transverse sectional image is insufficient to clearly observe the defect.

2.2. Texture Mapping and Rendering

Texture mapping is where computer graphics are applied to a 3D model; then, 2D color information is added to define the surface detail, such that a graphic surface under the rendering mode gains the textured features. Computer graphics are used to cover a bitmap onto a 3D geometric surface under rendering in order to provide a rich appearance on an object. There are numerous mapping methods for texture mapping, such as bump mapping and displacement mapping. Most mapping methods are used to map an image onto the curved surface of an object in order to allow the object to present a pattern estimated by the computer in different views. In this research, the displacement mapping technique is selected, and such a technique utilizes the height displacement mapping by converting 2D information into 3D information. During the conversion process, the height information of the mapping image is considered in order to generate a real third axis displacement. Figure 5a is an example of the texture mapping application effect. Figure 5b represents bump mapping, and it can be seen that its transverse section shows no height undulation. Figure 5c shows displacement mapping, and it can be seen that the surface actually shows the height feature; accordingly, such undulation mode design is used in conjunction with rendering in order to generate in the simulator. Originally, most of the physically based rendering methods used complicated and unintuitive parameters, and Physically Based Rendering (PBR) could not be used to create an optical model easily [15].

3. Lighting Simulation

In view of the parameter interlacing impact during the virtual model creation process, after the selection of the mapping method, it is necessary to select and adjust the parameter criteria. In this section, the model generation will be discussed, and various influential parameters are introduced. Figure 6 illustrates the steps required for the model simulation generation process.

3.1. Virtual Model Creation

If a random image is used directly for the texture, it is not sufficient to present the feature of the surface. The target surface and simulated surface are shown in Figure 7. Figure 7a is an actual image, and despite that the surface looks random, it, substantially, has the undulated texture feature after lighting. Through human eye observation, the surface has undulation and shadows, and a random array generated image is unable to present such features, as shown in Figure 7b. In addition, the integration of the random array into the simulated surface generated in this system is as shown in Figure 7c. The simulation image shows the surface undulation feature, and due to the random generation of texture, its arrangement combination is highly random and the situation of repeated background is impossible to occur; therefore, it is able to ensure the randomness and achieve the texture feature for the surface image. The conversion process from Figure 7b to Figure 7c uses the NURBS [16] curve surface for mapping, and control point third axis setting is performed to allow the surface to show z-axis displacement in order to control the simulated surface undulation interval (Hz). The displacement calculation uses the random number map distribution control and the expected distribution interval to perform the design.

For the overall conversion process, random numbers are used to generate a rectangular gray matrix with a brightness distribution interval within 40, and its image size corresponds to the actual size. The size of the corresponding real imaging photo is mapped onto the graphic software. Accordingly, the image size is multiplied by the pixel size (PS) of 0.075 mentioned in Ref. [6], and a surface of the same size is drawn in the virtual world. Next, the control point distribution and displacement calculation are performed on such a surface. Subsequently, after the surface is generated, control point and other parameter designs are then performed. To obtain the height distribution mentioned in Figure 4c, the converted random number map brightness distribution is divided by the maximum brightness (256) in order to be used as a fixed ratio, which is then multiplied by the design undulation interval in order to obtain the height distribution parameter. Finally, such values are fed into the simulation system to perform z-axis definition. The displaced texture generation flow chart is as shown in Figure 8. All of the parameters necessary for setting in the process are the ones shown in the blue background in the figure in order to achieve a large amount of automatic generation of random surface. The generated simulation image is shown in Figure 9, and the transverse section is shown in Figure 10. The simulation image generated by this system is able to obtain the height distribution identical to the actual sample and is also equipped with randomness.

3.2. Parameter Setting

3.2.1. Material Setting

We used the graphic software of OpenGL to perform the four criteria of the color, gloss, reflection and transparency for the parameter setting of a material. In addition, for this system, a program was used to control the material setting in order to assist in the creation of a simulation image. The material color is preferably close to the color presented by the actual photo as much as possible, which can be achieved with the advantage of a simulation system; then, the target color and brightness for simulation generation can be selected, and the image can be designed and create in order to math all intervals. Accordingly, for this system, the impacts of different parameter combinations on the image are collected in advance before the batch generation, following which setting is then performed according to the corresponding parameters during the subsequent image creation. For example, the impact and change of the reflection parameter on the brightness of the image is as shown in Figure 11. The reflection color is set to black and the reflectivity relative to the image at each stage is shown in Figure 12a–e.

3.2.2. Control Point Number

The control point number is the key to the presentation of the fineness level of surface undulation, which is also one of the costs for the surface reconstruction control speed. As the number of control points increases, the curvature calculation complexity of the simulator also increases, and the surface creation speed is reduced. Accordingly, the selection of an appropriate control point number is also a cost requirement consideration. In addition, when the image becomes larger, if the same number of control points for small images are used, then the surface presentation will become poorer, similar to the reduction of resolution. Accordingly, in this system, the control points are multiplied by a certain factor according to the size of the image. Figure 13a–f illustrate the presentations of the textured surface relative to different power of resolutions. In the simulation software, the image is multiplied by the pixel size of the original image-taking camera, and the pixel size is converted into the actual length in order to create an image size corresponding to the real image. Consequently, a corresponding relationship between the textured surface undulation and the image size is established. Through observation and experiment, it is found that when such a simulation ratio is set to 20 times, it is able to perform surface matching; therefore, in the subsequent experiment, the ratio of 20 times is used for the image creation.

3.2.3. Light Source Setting

In this research, the experimental control group refers to Ref. [6]. Accordingly, in addition to the selection of identical objects for inspection, the environment also adopts the lighting method with reference to the original article in advance. In the virtual world, the optical architecture of the simulator is restored and improved in order to achieve close resemblance with the finished product. In addition to the lighting method simulating the real image taking, in this research, the virtual light source difference in the simulation environment is also used in conjunction in order to perform virtual environment design, thereby achieving the expected effect. Figure 14 shows a real image-taking architecture, which uses a set of white high-angle back light and white low-angle back light in combination. The image taken under such a lighting architecture is shown in Figure 15. It can be observed that such optical architecture tends to cause the central area of the image to become brighter while the surrounding area is relatively darker. The area brightness mean value can be used to describe such conditions. When the overall image brightness mean value is 73, ROI areas are marked on Figure 15, and the ROI areas are all 64 × 64 pixels. In addition, the brightness mean values are marked at the side of the frame, and they are 43, 58, 72, 85, 93 and 100 from the outer toward the inner, respectively. It can be seen that the brightness plus and minus difference at least exceeds 57. Such condition of non-uniformity of color is taken into consideration for this system simulation creation.

Since the graphic software has a built-in ideal light source, such a light source would not be lost due to manual operation or hardware environment changes, and it is an ideal and uniform parallel light projected from top to bottom, which can also be treated as the high angle light mentioned in the previous section for use. In the graphic software, we arranged four rectangular lights for obliquely projecting onto the target plane at low angle direction, as shown in Figure 16. The simulated surface will be created at the origin location (intersection of red line and green line) in the coordinate system. The real image resolution is 2592 × 1944 pixels. Figure 15 illustrates that the surrounding and the central area of the image show different brightness. If simulation is performed based on the mean value of the entire image directly, the simulation image only presents a portion of the brightness value such that not all areas can be matched. Such a situation can cause the areas not being simulated to have insufficient datasets, which will then result in training failure. Additionally, during the generation of simulation images of this system, a smaller area is used for the simulation. Through the symmetrical distribution control, all brightness can be presented, and datasets are created at the same time in order to completely cover all of the brightness distribution of a real image.

3.2.4. Defect Generation and Marking

Figure 17 shows the real scratch patterns. If such scratch patterns are to be simulated and generated, their physical features are added to the random number images of the system previously mentioned in order to generate such physical features on the simulation image. In view of the above, before this system generates the simulated surface, a dark color segment is created on the random number image. Accordingly, during the displacement mapping computation, the target area shows an indented groove, and the defect of Figure 18 can be simulated and generated. Its transverse section graph is shown in Figure 19, and the defect is located at the area indicated by the blue arrow in the graph.

The defect of this system is generated autonomously, and is able to achieve the automatic defect marking function at the same time. The automatic marking process is shown in Figure 20. During the same time when the system generates the defect, it also generates a corresponding location mask image. When the system captures the simulation image, a marking image of corresponding size is also generated simultaneously; therefore, it is able to achieve the objective of 100% of automatic marking completely.

4. Deep Neural Network Inspection

In this research, the improved ResNet is used to perform the training inspection and prediction. The network architecture is obtained from Ref. [6] and is adjusted for the fine surface defect. The improved network architecture is shown in

Table 1. Modified ResNet network table.

Layer Name	Output Size	Our Proposed Network 50-Layer
inputs	64 × 64 × 3
conv1	64 × 64 × 64	3 × 3, 64, stride 1
conv2.x	32 × 32 × 256	$\left[\begin{array}{c}1\times 1, 64\\ 3\times 3, 64\\ 1\times 1, 256\end{array}\right]$ × 3, stride 2
conv3.x	16 × 16 × 512	$\left[\begin{array}{c}1\times 1, 128\\ 3\times 3, 128\\ 1\times 1, 512\end{array}\right]$ × 4, stride 2
conv4.x	8 × 8 × 1024	$\left[\begin{array}{c}1\times 1, 256\\ 3\times 3, 256\\ 1\times 1, 1024\end{array}\right]$ × 6, stride 2
conv5.x	8 × 8 × 2048	$\left[\begin{array}{c}1\times 1, 512\\ 3\times 3, 512\\ 1\times 1, 2048\end{array}\right]$ × 3, stride 1
concat1	8 × 8 × 3904	(conv1, conv2.x, conv3.x, conv4.x, conv5.x)
conv6	8 × 8 × i	1 × 1, i, stride 1
pool5	1 × i	reduce mean

. From experimental results from the literature, it can be understood that when the input image is 64 × 64, the training result is better than the input image of larger size. This is mainly due to the defect feature of this target being small; if the input image is too large, then the defect would disappear during the dimension reduction process. In response to the reduction of the input image size, network architecture is performed on this sample. The convolution layer of conv1 is changed to 3 × 3 in order to prevent excessive parameters that may cause overfitting, and the stride length is 1. At the rear end of the network, a concatenation layer concat1 is added after the last convolution layer, and the former features from conv2.x to conv5.x are overlaid, such that the concatenation of the feature of each year can increase the accuracy. Finally, the pool5 is switched with conv6, in order to prevent the disappearing of features after the concatenation of layers. The network training parameter settings are shown in

Table 2. Inspection training parameter table.

	Pre-Training	Training
Image size	64 × 64	64 × 64
Number of iteration (Epoch)	4	8
Learning rate	0.01	0.005
Batch size	64	64

. The inspection training is divided into pre-training and training, for the purpose of accelerating the network convergence.

5. Experimental Results and Discussion

5.1. Dataset

In this research, the size of the originally taken image of the target data is 2592 × 1944, and the image taken shows background other than the housing. In addition, due to the aforementioned real image photo obtained and the significant difference between the section brightness and the image brightness, if the original image is used, it could cause the simulation parameter range to expand, and the simulation uncertainty factor would also increase. Accordingly, in this experiment, sampling is made on the cutting range of the area of 1100 × 900 at the central fixed location of the original image, as shown in the red bracket area in Figure 21, thereby preventing the occurrence of background other than the housing and reducing the color difference. In addition, only images of interest with the area brightness mean value between 90 and 100 are selected as the simulation subject in this research. The simulation datasets generated by this system are created by using a random number image of 300 × 300, and the output image size ranges from 280 to 350. The purpose of the change of the output size is to increase the image variability as a method to enhance the simulation data. To allow the simulation image to have a distribution range covering the distribution interval of a real image, the simulation image brightness mean value range is set to be between 80 and 110; with the control of the internal parameters and light source changes via this system, simulated surfaces having minor differences are created. The surface types use (1) simulated low-angle back light, (2) built-in light + simulated low-angle back light, (3) built-in light and without use of gloss parameter, (4) built-in light with the use of gloss parameter. The three defect types refer to flat defect, micro-texture defect and gradient defect. The simulation image database is as shown in

Table 3. Simulation images with different background types and defect type combination pattern table.

Surface Type	Flat Defect	Micro-Texture Defect	Gradient Defect
(1)
(2)
(3)
(4)

.

5.2. Inspection Network Training Result

The simulation images are used to perform training, following which the weights are used to inspect the real images in order to observe whether the simulation images are able to replace the real images, thereby achieving the transfer learning effect. In the future, the simulation images can be used to replace real images to perform training. The training uses approximately 350,000 simulation images of the size 300 × 300. The training process uses the learning curve and the loss function for presentation. Figure 22a,b are learning curves and Figure 23 is the loss function curve.

Table 4. Simulation image test confusion matrix table.

		Real
		Defect Type (Positive)	Background Type (Negative)
Prediction	Positive type	23,281 (99.74%)	16 (0.07%)
	Negative type	60 (0.26%)	23,337 (99.93%)

refers to the verification confusion matrix; accuracy reaches 99.83%, recall is 99.74% and precision is 99.93%.

Table 5. Real image test confusion matrix table.

		Real
		Defect Type (Positive)	Background Type (Negative)
Prediction	Positive type	1704 (78.7%)	35 (1.95%)
	Negative type	461 (21.3%)	1769 (98.05%)

refers to the real image test confusion matrix table, indicating differences in the score due to the difference between the simulation images and the real image, and accuracy only reaches 87.5%, whereas recall is 78.7% and precision is 97.98%. For the observation of the scope, it can be found that the background type training is relatively more successful, and it is 19% higher than the defect type. The predicted real image result observed is presented in Figure 24. The red locations indicated in the prediction results refer to the network predicted defect locations. It can be seen that the main defects can be successfully identified, and the erroneous determination locations are at the edges or areas with relative scarce defects. By performing noise filtering on the result of this experiment, it shows that all of the defect areas on the entire image can be detected, indicating that the use of the simulation images of this system to perform transfer learning demonstrates a remarkable outcome.

5.3. Image Inspection Result Discussion

During the implementation process of this research, it was found that if only one type of simulation image is used to perform training, the pattern of the simulation images would be too simple, such that the real images cannot be inspected properly. On the other hand, when a great number of types of simulation images are used to perform training inspection, the inspection outcome can be significantly improved and the real images can be identified successfully. Such an experimental result is consistent with the effect achieved by using numerous types of defects for comprehensive training in the transfer learning in order to predict unknown defects. The experiment result indicates a high precision and few background type prediction errors, demonstrating that the simulated textured surface in this research is equipped with an adequate texture feature for the network learning determination.

In addition, regarding the search of the cause for low recall, from the observation and verification of the images, it was found that the areas with erroneous determination are mostly located at the areas with relatively scarce real defect, ROI edges and areas with unclear definition of defect. For the first type of error, areas where the defects are relatively scarce, the errors at such areas are the areas where the real image training weights would determine incorrectly. The cause of such types of error is speculated to be that since the area and ratio of the defect feature in the image are relatively small, consequently, the feature disappears during the network dimension reduction process. The second type of error areas are at the ROI edges. The cause for the occurrence of such types of error is speculated to be that since the real defect quantity is small, the moving steps need to be adjusted at a smaller scale during the cutting of images; consequently, the real image defect data tend to occur at the edges more often. On the other hand, since the simulation image defect data can be obtained easily, the cutting step is relatively larger, and the defect locations are not likely to distribute at all locations of the area map with interest. As a result, when the defect is located at the edge, in this experiment, no relevant feature is learned, which leads to determination errors. The third type of error, due to unclear definition of defect, is the key difference between the simulated images and the real images. The simulation images have clearly defined defects and backgrounds, which can be understood from the observation of all of the types and patterns of simulation images in Table 3. Under this inspection network structure, such a defect pattern cannot be detected successfully; therefore, it is expected to use the generative adversarial network to perform training in order to increase the reality of the simulation images, thereby compensating for the difference between the real and simulation images.

Since there are differences between the simulation images and the real images, such as the clear boundary of defects in the simulation images, whereas the boundary line between the defect and the background is unclear in the real images, it is expected that we use the generative adversarial network to enhance such visual effects and to include it in the simulation image. In recent years, there have been some publications on improved generative adversarial network architectures. Among such publications, the unsupervised network [3] published in 2017 and used to perform image-to-image conversion without the need of paired images is considered to be the best network in terms of the overall conversion effect. Its network uses two sets of generators and two sets of discriminators to perform the adversarial process, such that two sets of types in different fields can learn the features from each other. Accordingly, in this experiment, the Cycle-GAN network is used to perform the style conversion. During the defect conversion, the following several types of conditions may be encountered, causing training failure:

• The difference between the simulated defect color and real defect color is too great.
• The simulated defect has obvious edges and shadows, causing color shift at the defect area.
• Areas of black spots occur during conversion.

Figure 25 shows the aforementioned types of conversion failure conditions. The weights of the conversion failure exist during the entire training process; therefore, if such a conversion method needs to be used to perform the image creation, it would be impossible to avoid the human screening of weights. In comparison to the direct generation module of the system in this research, the use of an adversarial network for generation is relatively more complicated and time consuming and is also less reliable.

When it is applied to the background type, the condition of areas with broken texture caused by significant defect differences in the defect type conversion rarely occurs; however, training and image colors read during the network weight training affecting the color shift in subsequent image creation still occur. Figure 26 refers to an original simulation image, and after conversion with different weights. The image after conversion is as shown in Figure 27. The conversion result indicates that the resolution is reduced and the edge still shows abnormality. If the reference color difference during the testing of input image and training is too large, the conversion would also fail. The output image of the generative adversarial network is unstable, and its reason is speculated to be that despite the Cycle-GAN network architecture being successful when it is applied to the landscape conversion field, its network architecture is insufficient to resolve the fine texture, and as the resolution decreases, its conversion result still requires manual screening and conformation. This is mainly due to a need to provide images having mean values similar to the images for training in order to achieve the effect of conversion to the desired brightness. Since the images of this system have already been equipped with a broad distribution during the image generation, the use of such types of generative adversarial networks to perform conversion is not applicable. The converted images are then used to perform the inspection training at the end; however, their weights cannot predict the real defect.

6. Conclusions

In this research, a simulation system based on surface shadow physical change is designed. Displacement undulation is performed on the surface based on the image brightness change, which is then projected onto a three-dimensional space. Through such conversion, it is able to simulate complicated textured surfaces and defects, in order to overcome the situation of relatively few real sample defects, thereby preventing the damage of a good product surface for data collection. In addition, since this sample surface is complicated without order, after the automatic design with randomness generation, a large number of random surfaces can be obtained in a short time; therefore, it is not required to prepare and taking images of real samples in great quantity. Furthermore, since the defects are generated by the system during the generation of simulation images, the defect areas can be clearly defined, and marking action can be performed directly and simultaneously, thereby overcoming the problem of massive preliminary processes required for all supervised learning. Consequently, the cost can be reduced and the marking location precision can be improved at the same time. Moreover, a virtual dataset created symmetrically using a generative adversarial network optimization system is investigated to reduce the effect of differences between real and virtual images. However, the output image of the generative adversarial network is unstable. From the experiment, it is verified that the simulated samples generated by this system are able to successfully detect the real samples, and the accuracy reaches 87.5%, recall is 78.7%, and precision is 97.98%, which is sufficient to intuitively determine the defect location. In addition, after subsequent background treatment, the defect areas can then be correctly screened out.