Deconvolutional Neural Network for Generating Spray Trajectory of Shoe Soles

Abstract

The footwear industry is moving towards automation and intellectualization. To overcome the drawbacks of the high-cost and low-efficiency traditional manual shoe sole gluing process, automatic methods were utilized for generating spray trajectories. Currently, most of the reported automatic methods for generating spray trajectories mainly rely on the outer contour bias method. However, the glue is only applied to the area near the edge/contour of shoe soles and the fixed offset distance in the outer contour bias method cannot adapt to the immense amount of shoe styles with high precision and achieve applicability for irregular and unique sole designs. An intuitive yet logical approach to fulfill the requirements is to utilize the deconvolutional neural network for generating shoe sole spray trajectories. In this work, we treated the glue trajectory prediction as an image-to-image prediction and established a novel deconvolutional neural network to generate shoe sole spray trajectories. The as-proposed deconvolutional neural network for generating spray trajectory offered significant advantages over the existing bias-based methods, including: (1) based on the novel deconvolutional neural network, the proposed method for generating shoe sole spray trajectories exhibits greater applicability to irregular shoe soles, which improves the spray accuracy without compromising the spray efficiency; (2) we discard all the pooling layers, which only consist of convolutional and deconvolutional layers, to preserve more spatial information and achieve higher spray accuracy through end-to-end mapping from shoe sole images to shoe sole spray trajectories, resulting in an improved spray accuracy without sacrificing spray efficiency. The Dice similarity coefficient and Hausdorff distance were used as the evaluation metrics to assess the performance of our approach. Our proposed method showed an ultra-high accuracy and precision with a Dice similarity coefficient over 99.25% and a Hausdorff distance less than 1.2 mm, which are ~10% higher than the spray accuracy of other reported traditional methods. Our findings would bring significant improvements to the field of automatic shoe sole spray trajectory generation, which has the potential to promote the utilization of intelligent technologies in the footwear industry.

1. Introduction

According to the World Footwear Yearbook, there were 24.3 billion pairs of shoes produced globally in 2019, with China accounting for 55.5% of the total production, producing a total of 13.48 billion pairs of shoes, which was significantly higher than any other country. As the footwear industry continues to develop, the footwear industry has been undergoing a significant transformation towards intelligent and digitalized production methods due to the high cost and low efficiency of traditional manual shoe sole production [1]. Therefore, it is crucial for enterprises to adopt innovative production methods to cope with changes in resources and markets. Combining the contour curve of shoe soles with robot spraying technology can significantly enhance the productivity and efficiency of the footwear industry [2]. As part of this transformation, automatic shoe sole spraying has gained prominence for its potential to improve production efficiency and accuracy.

However, most of the existing methods for generating spray trajectories during auto-mated shoe sole spraying suffer from several significant drawbacks and limitations [3,4]. For example, J. Wu et al. put forward a block maximum algorithm to accurately obtain the sole profile. In their groundbreaking research, they constructed an algorithm to precisely capture the intricate details of the shoe sole [3]. G. Xu et al. demonstrated an automatic interpolation algorithm for the NURBS (Non-Uniform Rational B-Splines) trajectory of shoe sole spraying based on a 7-DOF (degrees of freedom) robot. The as-proposed algorithm solved the problem of sparse and incomplete trajectory points of the shoe sole when extracting robot motion trajectories [4]. The above-mentioned methods assumed that the shoes had regular shapes and patterns, which made their methods unsuitable for generating accurate spray trajectories for irregular or unique soles and failed to handle the complexities and variations in sole designs. In this regard, the corresponding works have non-negligible limitations for the effective and practical applications in the designing of irregular and unique soles, which hampers their ability to meet the diverse style demands of modern footwear. Additionally, the existing algorithms for generating sole spray trajectories merely rely on simplistic reconstruction techniques, such as linear interpolation or simple geometric transformations, leading to an inadequate reconstruction accuracy. The resulting inaccurate representations of the spray trajectories would cause suboptimal spraying results, which could compromise the quality and durability of the adhesive bonding between the sole and upper material.

Currently, the advanced intelligent technologies, e.g., deep learning [5], computer vision [6], and big data analytics [7], have shown great potential to be labor-saving technologies in a variety of industrial conditions. In particular, deep learning, a subfield of machine learning that involves building and training multi-layered neural networks, is used to simulate learning and decision-making processes of human brain. The core idea behind deep learning is to automatically learn feature representations from raw data without relying on traditional handcrafted feature engineering. The main advantage of deep learning lies in its ability to automatically learn meaningful representations from the data, making it suitable for tasks such as image and speech recognition, natural language processing, and recommendation systems. For example, L. Bi et al. constructed deep learning algorithms to help doctors diagnose [8]. W. Wang et al. applied the deep learning technique to remote sensing image segmentation [9]. Hussain A et al. have applied deep learning to human activity detection [10]. Computer vision, a science and technology that explores how to enable computers to “see” and “understand” images and videos, allows computers to perceive, comprehend, and interpret visual information through the developed algorithms and models, thereby achieving functionalities similar to the human visual system; for example, W. Fang et al. used computer vision for construction risk monitoring [11]. B. G. Weinstein has applied computer vision to detect animal species in the field of zooecology [12]. Remarkably, in the field of computer vision, deep learning has achieved remarkable success in tasks such as image classification [13], object detection [14], and image generation [15]. By employing deep convolutional neural networks, computers can accurately identify and classify objects in images, such as faces, vehicles, animals, and more. Thus, integrating the intelligent method of deep learning with computer vision is considered to be an intuitive yet logical approach for automatically generating high-precision spraying trajectories that are suitable for the various types of shoe soles, which possibly can significantly enhance the accuracy, efficiency, and adaptability of spray trajectory generation methods.

In this work, we established a deconvolutional neural network for generating shoe sole spray trajectories during the automotive process of shoe sole spraying. The shoe sole image segmentation algorithm was based on a deconvolutional neural network and a three-dimensional (3D) computer vision method. The 3D point cloud data of the sole were captured by utilizing a 3D profile scanner, while the deconvolutional neural networks were applied to create the glue trajectory using an image-to-image prediction method. By leveraging deconvolutional neural networks, the 3D point cloud data of the shoe soles were fed into a specially designed convolutional neural network to extract highly distinctive features in the images. Then, a deconvolution strategy was employed to expand the feature map size to match the input and output predicted image size of the network, while the resulting predicted image contour served as the spray trajectory during automated shoe sole spraying. To evidence that the proposed method could meet the demand for diverse shoe sole styles with improved production efficiency and accuracy, the performance of our proposed method was evaluated by using the Dice similarity coefficient and Hausdorff distance as evaluation metrics. The results demonstrated that our proposed method outperformed most other reported traditional approaches with a remarkable Dice similarity coefficient of 99.2% and a distance of 1.2 mm, surpassing the 90% Dice similarity coefficient and 2 mm distance achieved by conventional algorithms. Moreover, the processing time of our method is less than two seconds, which is an acceptable range for industrial production requirements. By deploying our method on high-performance GPUs, the processing time could be further optimized. Summarily, we provided a groundbreaking approach to automated shoe sole spraying, which could promote the applications of neural networks to improve the labor productivity of the footwear industry. Our proposed method could achieve significant improvement in the accuracy and precision of generating the spray trajectory of shoe soles, and was adapted to the various characteristics of different shoe soles for personalized and irregular sole designs. We believe that our work could open a new avenue for the potential applications of artificial intelligence in the field of industry.

2. Related Work

The applications of automotive and intelligent methods to the shoe gluing process will effectively improve the product quality and the labor productivity of the footwear industry. In this chapter, we introduce the current algorithm for generating the spray trajectory of shoe soles.

2.1. Adaptive Gluing Trajectory Extraction Method

D. S. Kwon et al. developed a method to automatically extract gluing trajectories for different shoe shapes and sizes [16]. In their approach, they transferred the initial shoe sole gluing trajectory to a database and utilized it during the actual production process. NURBS interpolation was used to fill in the missing points in the trajectory, and the interpolated curve was compared to the ideal curve to obtain an interpolation function. Then, the position points were sent to the robot controller based on the position of the error control points until the error of each segment was consistent. However, their method was primarily proposed for flat shoe soles and cannot provide accurate spray trajectories for irregular or uniquely designed soles. And, they used error control points to determine positions which could not meet the precise requirements of personalized shoe soles. G. Xu et al. tried to improve the accuracy of the gluing trajectory by introducing velocity constraints based on the local maximum curvature of the trajectory for curve segmentation [4]. They also utilized S-shaped acceleration/deceleration planning to address the issue of sparse and incomplete trajectory points at the toe and heel of the sole. Their optimization strategy has partially improved the accuracy of the method, but the adaptability of this method is very limited, particularly in the production of highly customized shoe soles. To address these limitations, more advanced and flexible intelligent methods are necessary to cater to the diverse needs of various shoe soles with the ensured precision and accuracy of gluing trajectories.

2.2. CAD Software-Based Automatic Acquisition System for Shoe Sole Coating Trajectory

J. Y. Kim et al. developed a CAD (computer-aided design) software-based automatic acquisition system for shoe sole gluing trajectories [17]. The system consisted of a CAD module, motion calculation module, trajectory analysis module, and simulation module. The sequential steps of this system were to plan the gluing trajectory, simulate the process, and convert it into executable instructions for the robot. However, this method has several notable limitations, including: (1) the complexity of the trajectory planning process had challenges in achieving accurate and efficient results, especially for complex shoe shapes; (2) the deviations between the planned and actual gluing processes might lead to low accuracy in the generated trajectories; (3) the usability and adaptability of the system would be limited by its dependence on CAD software (2002 R15.6) and specialized knowledge; (4) the system could not effectively meet the requirements of personalized and unique sole designs. Based on the above CAD software-based automatic acquisition system, C. Wu et al. reduced its complexity and developed an interface for automatically generating robot spray trajectories from different types of 3D CAD shoe sole models [18]. However, this optimization scheme does not address the issue of insufficient accuracy. Overall, these drawbacks hinder the practicality and effectiveness of the system in meeting the diverse needs of the footwear industry.

2.3. Structured Light-Based Shoe Sole Adhesive Trajectory Extraction

C. Y. Wu et al. constructed light scanning to capture the shoe sole and extracted the endpoints of the scan lines in order to create a point set for the edge contour curve [19]. They derived the spray trajectory by interpolating the contour curve with an applied offset value. But their method was constrained by the shape of the shoe sole and it was incapable of handling irregular sole shapes, leading to inaccuracies in the spray trajectory alignment with the desired specifications. Moreover, the use of simplistic interpolation techniques might fail to fully capture the intricacies of the contour curve, resulting in inevitable deviations between the generated spray trajectory and actual gluing position. In view of the above drawbacks, T. Wang et al. introduced a “Block Maximum” algorithm to improve the applicability of the method to certain personalized shoe soles [20]. But their method still had limited capability to handle personalized shoe soles and failed to adequately address the requirements for diversification and customized product offerings.

3. Methods

Convolutional neural networks (CNNs) can effectively express the essential features of target data by simulating the neural structure of the human brain and extracting layer-by-layer features of input data through “pre-training + fine-tuning” [21].

In recent years, CNNs have been widely applied in various computer vision tasks such as image classification, face recognition, object detection, and image segmentation [22], and impressive results have been achieved. However, to date, there has rarely been research on the application of deep learning to shoe sole spraying. Therefore, a fully convolutional neural network model for generating adhesive trajectories in shoe sole spraying processes is proposed based on the current deconvolution techniques. The deep deconvolutional network used in this study is a supervised learning approach, and its basic workflow is illustrated in Figure 1.

During the training phase, the sole point cloud data were collected with a 3D contour scanner and then the processed data were input into the established deconvolutional neural network. The training samples consisted of manually segmented images of shoe soles with the actual adhesive trajectory fed into the neural network, and the weights of the feature extractor and image generator were iteratively updated through forward and backward propagation. After training, a well-trained feature extractor and image generator was obtained, which can generate binary shoe sole images with adhesive trajectory outlines based on different shoe sole inputs.

In the application phase, the shoe sole images were preprocessed and input into the trained network model consisting of a feature extractor and image generator. Through a single forward pass, the model predicts a binary shoe sole image with spray trajectory contours. After obtaining the binary shoe sole image, the Laplacian operator was applied to extract the edges, and a smooth adhesive trajectory line could be gained. To achieve better spraying effects, the spray trajectory points were allocated based on the curvature of the spray trajectory, where more spray points were allocated for areas with higher curvature. Thus, a set of spray trajectory points for the shoe sole are obtained, and the robotic arm can successfully complete the shoe sole spraying task based on these points.

3.1. Point Cloud Data Filtering

Due to the presence of external light interference in the actual measurement environment, the acquired 3D point cloud data often contain a significant amount of noise, as shown in Figure 2. This noise can originate from variations in ambient lighting, sensor errors, or other interfering factors. In order to improve the quality and accuracy of the data, it is necessary to filter the 3D point cloud data.

In this study, median filtering was utilized for denoising because it can effectively remove outliers and noise while preserving edge details, and it is suitable for processing 3D point cloud data. Figure 3 shows the filtered image. By calculating the median value of the points within the neighborhood, median filtering effectively eliminates noisy points, improving the quality and accuracy of the data.

3.2. Architecture

Figure 4 illustrates the detailed configuration of the deep network. The trained network is composed of two parts—convolution and deconvolution networks. The convolution network corresponds to a feature extractor that transforms the input image into a multidimensional feature representation, whereas the deconvolution network is a shape generator that produces object segmentation from the feature extracted from the convolution network [23,24]. The final output of the network is a binary shoe sole image that has same size as the input image, and its contour represents the glue spraying trajectory line that we need.

To facilitate the network input, the input image is uniformly cropped to a size of 1 × 256 × 512 pixels. In the convolutional network part, a structure similar to FCN was used, where the final fully connected layer was replaced by convolutional layers and the pooling layers were removed. All pooling layers were discarded to preserve more spatial information and enhance the model’s sensitivity to position information, which can improve the spray accuracy without sacrificing spray efficiency. The convolutional network model contains 5 convolutional layers and the step size of the convolutional kernel in the network is 2. The mirror structure of the convolutional network is used in the deconvolutional network part, which aims to reconstruct the shape of the input target. Moreover, the multi-layer deconvolutional structure is also able to capture the shape details at different levels like the convolutional network. In the convolutional network model, the low-level features can describe the whole target coarse information, such as target location, general shape, etc., while the more complex high-level features have classification properties and also contain more target detail information [25].

3.2.1. Convolutional Layer

On the convolution layer, multiple convolution kernels are used to convolve with the input image, and a series of feature maps can be obtained by an activation function after adding bias [26], and the convolution process is expressed as follows:

$$h^l=f({\displaystyle \sum }_{j=1}^{M_x}{\displaystyle \sum }_{k=1}^{M_y}{w}_{jk}\ast h^{l-1}\left(j,k\right)+b^l)$$

(1)

where $M_x$ and $M_y$ represent the length and width of the convolutional filter M, $w_{jk}$ represents the learned weights in the convolutional kernel, $h^{l-1}$ represents the input to convolutional layer, $b^l$ represents the bias of the $l^{th}$ layer filters, and Equation (1) denotes the activation function.

Nowadays, the ReLU activation function [27] instead of the traditional sigmoid function is often used in popular deep neural networks to accelerate network convergence. The mathematical expression of the ReLU function is:

$$f\left(x\right)=max\left(0,x\right)$$

(2)

where x is the input to the activation function.

Subsequently, a convolutional neural network with five layers was built to extract the feature information from input images. In each of these convolutional layers, we used the ReLU activation function, and the stride of the convolutional kernel was set to 2.

3.2.2. Deconvolution Layers

The deconvolution layer is also called a transposed convolution layer, and its purpose is to perform an up-sampling operation [28]. The deconvolution operation can be viewed as the opposite of the convolution operation. Therefore, the deconvolution layer takes an input feature map and produces an output feature map with a larger spatial resolution. In this work, we defined the deconvolution operation as follows:

$$y\left(i,j\right)=sum\left(W\left(k,l\right)\ast x\left(i+k,j+l\right)\right)$$

(3)

where y is the output feature map, x is the input feature map, W is the learnable deconvolution kernel, and k and l are the spatial offsets of the kernel.

Furthermore, we utilized five deconvolution layers to perform deconvolution on the feature maps outputted by the convolutional layers, thereby achieving the consistency of the resolution between the output segmentation image and the input image.

3.3. Post-Processing

After using the established model to process the original data, a binary shoe sole image with a spray trajectory line was generated. We converted the image into the actual spray trajectory of the sole through post-processing.

3.3.1. Segmented Image Filtering

Due to the possible presence of noise in the images generated by the model, these segmented images are directly used to extract the spray trajectory lines, which may affect the quality of the extracted curves. Therefore, a filtering operation is needed to generate images, and median filtering is adopted as the filtering method. Median filtering can remove noise while preserving the image’s edges and details [29,30], which has a positive impact on the quality of extracting coating trajectory lines. Thus, we can obtain cleaner and more accurate images, thereby improving the extraction of coating trajectory lines. Its expression is:

$$F_{i,j}\left(x,y\right)={}_A{}^{Med}\left\{x_{i,j}\right\}$$

(4)

where A represents the filter window and Med represents the median part of the filter template.

The raw and denoised images are compared in Figure 5. It can be observed that median filtering can eliminate noise while preserving the edges and details of the image, which is more suitable for extracting adhesive tracks.

3.3.2. Pickup Point

The glue trajectory line obtained by this model has plenty of points, while in the actual production process, only 44 points on one circle of the closed curve on the sole glue surface are needed to complete a continuous and good glue application. However, it is obviously unreasonable to spread these 44 points evenly on the closed curve. More points are assigned to the toe and tail with the larger curvature, and fewer points are assigned to the middle of the sole with the smaller curvature, which can better fit the spraying trajectory line of the sole. Meanwhile, the robot arm can accurately spray the sole without overflowing or leakage when spraying the toe and tail. The proposed trajectory point extraction method is described as follows.

Firstly, calculate the curvature of each point on the trajectory line to measure the curvature degree. For the ith point on the trajectory line, its curvature, denoted as k_i, is calculated using the following formula:

$$k_i=\left|\left(x_i^{\prime }\ast y_i^{″}-y_i^{\prime }\ast x_i^{″}\right)/{\left(x_i^{\prime }{}^2+y_i^{\prime }{}^2\right)}^{\frac{3}{2}}\right|$$

(5)

where $x_i^{\prime }$ and $y_i^{\prime }$ represent the first derivatives of the point coordinates ($x_i$, $y_i$) with respect to index i, while $x_i^{″}$ and $y_i^{″}$ represent the second derivatives.

Secondly, to reduce noise and ensure smoothness, we apply a smoothing filter to the curvature curve. This can be achieved by using methods such as the Savitzky–Golay filter. After smoothing, we obtain the smoothed curvature curve, denoted as $k_i$.

Next, we need to determine the number of points to extract on the trajectory line, considering the curvature. We normalize the smoothed curvature $k_i$ to obtain the normalized curvature, denoted as $n_i$. Based on the desired total number of points, $num\_points$, we calculate the number of points to extract in each curvature segment, denoted as $num\_points\_segmen{t}_i$. This can be calculated using the following formula:

$$num\_points\_segmen{t}_i=ceil(n_i\ast \left(num\_points-1\right)$$

(6)

where $ceil\left(x\right)$ denotes rounding $x$ up to the nearest integer.

Finally, the corresponding number of points are extracted from the trajectory line based on the number of points in each curvature segment. For each curvature segment, we uniformly select interpolation points on the trajectory line using linear interpolation. Assuming the starting point coordinates of a curvature segment are ($x_{start}$, $y_{start}$), the ending point coordinates are ($x_{end}$, $y_{end}$), and the extracted points are $num\_points\_segmen{t}_i$, the coordinates of the interpolation points ($x\_inter{p}_i$, $y\_inter{p}_i$) can be calculated using the following linear interpolation formula:

$$x\_inter{p}_i=\left(1-t_i\right)\ast x_{start}+t_i\ast x_{end}$$

(7)

$$y\_inter{p}_i=\left(1-t_i\right)\ast y_{start}+t_i\ast y_{end}$$

(8)

where $t_i$ varies from 0 to 1 and represents the interpolation parameter for selecting points along the curvature segment.

The trajectory after point selection is shown in Figure 6, where X, Y, Z are the coordinate points when applying the adhesive. In the data sheet, the A, B, C represent the Euler angles of rotation around the x, y, and z axes, respectively.

4. Experiments and Results

This section describes the implementation details, the experiment setup, and the experimental results.

4.1. Experimental Platform

The experiments were carried out on a computer with an 8 GB RAM, an Intel Core i5-8300H CPU, and a Windows 10 operating system. To train a deep fully convolutional neural network (DFCN) model, we employed the TensorFlow framework, along with an Nvidia GeForce GTX 1050 GPU. The program was written in Python 3.6.7. The “Qingbo” 3D contour scanner has a scanning frequency of up to 2000 Hz, enabling the accurate and high-resolution capture of the shoe sole data, ensuring the quality of our network model’s training data.

4.2. Make Dataset

During the process of deep learning, the quantity and quality of training data have a significant impact on the network model. In this study, the used dataset consisted of real shoe sole images captured using the “Qingbo” 3D contour scanner LH model in a factory environment. These images included different types, sizes, and shapes of shoe soles. By collecting real shoe sole image datasets, we could provide more representative and diverse training samples to enhance the generalization ability and performance of the network model. In the process of creating the dataset, we used various image processing methods, such as resizing, rotating, and flipping, to increase the diversity and quantity of the dataset. A total of 70 images were collected, of which 45 were used as the training set and 25 as the test set. Figure 7 shows the platform we built to collect the data, and

Table 1. Table of parameters for 3D line laser contour scanner.

Serial Number	Name	Parameters
1	Installation status	Diffuse reflection
2	Laser source	405 nm (Blue light)
3	X-direction field of view	200–300 mm
4	Z-axis measurement range	350–400 mm
5	Installation distance	425 mm
6	Sampling frequency	≤1000 Hz
7	Weight	About 1.8 kg
8	Enclosure material	Aluminum alloy

shows the parameters of the 3D contour scanner used in the platform.

For data annotation, we used a manual method to mark the unique region of the image. Specifically, annotation software was utilized to trace the contour of the shoe sole binary image, where the contour was the spray trajectory line. The shoe sole binary image was labeled as a positive sample, while the remaining regions were labeled as negative samples. The annotation process was independently completed by two experienced annotators to ensure the accuracy of the labels. Finally, the annotated data were saved in JSON format, recording the coordinates of the unique region and the corresponding label information.

4.3. Training Setting

In our research, the process of selecting the optimal hyperparameters for the proposed model involved a systematic approach, combining recommendations from the literature and empirical experiments. Specifically, we adopted the Stochastic Gradient Descent (SGD) optimizer, in which the initial learning rate, momentum, and weight decay were set to 0.01, 0.9, and 0.0005, respectively [31]. To initialize the weights in the convolutional network, we utilized a zero-mean normal distribution. This choice of weight initialization has been proven effective in various deep learning applications and is commonly used to ensure stable and efficient model training. During the training process, we continuously monitored the validation accuracy and dynamically adjusted the learning rate based on its performance. If no improvement in validation accuracy was observed, we reduced the learning rate by an order of magnitude. This adaptive learning rate adjustment strategy helped us avoid convergence issues and achieve better optimization during training. Throughout the training phase, we conducted a total of 150 epochs with a batch size of four. Additionally, we incorporated batch normalization after each convolutional and deconvolutional layer to address internal covariate shift and stabilize the training process. The batch normalization was used to optimize the network and improve the overall convergence performance. To validate the effectiveness of the chosen hyperparameters, we performed extensive experiments using various combinations of hyperparameter settings. We systematically compared the performances of the models based on the metrics, including accuracy, convergence speed, and generalization ability. The results demonstrated that the selected hyperparameters, in conjunction with the proposed weight initialization and batch normalization strategy, could improve the training effectiveness and the performance of our model.

Figure 8a illustrates the changes in the loss function (Mean Absolute Error) and accuracy during the training process in our study. It can be observed that the loss value of the loss function steadily decreases in the initial region and fluctuates slightly in the following region. Correspondingly, the plotted curve of the accuracy as a function of the Epoch is shown in Figure 8b. The estimated accuracy rapidly increased with the Epoch before reaching a plateau of around 25 Epoch. The results demonstrate that our established DFCN model has a strong learning ability, which is consistent with the current training strategies of neural networks.

4.4. Comparison of Experimental Results

In this part, the performance of our proposed DFCN model is compared with other reported sole glue application trajectory algorithms.

4.4.1. Evaluation and Comparison of the Glue Tracks

In the evaluation experiment, two parameters were used to quantitatively evaluate the performance of different segmentation algorithms—the Dice similarity coefficient (DSC, calculates the similarity between two contour regions) and the Hausdorff distance (HD, reflects the maximum difference between two contour point sets).

Given A and B as the point sets of the two contour regions, the DSC and HD are defined as below:

$$DSC\left(A,B\right)=\frac{2\left|A\cap B\right|}{\left|A\right|+\left|B\right|},$$

(9)

$$HD\left(A,B\right)=max\left(h\left(A,B\right),h\left(B,A\right)\right),$$

(10)

where h(A, B) represents the maximum distance between each point in set A to the closest point in set B, i.e.,

$$h\left(A,B\right)=\max (\min \left(\Vert a-b\Vert \right)$$

(11)

where a is a point in point set A, b is a point in point set B, and ‖a − b‖ denotes the Euclidean distance between point a and point b. A smaller value of HD indicates a higher segmentation accuracy.

We compared the performance of our DFCN (Deep Fully Convolutional Network) model with other reported algorithms for sole glue application trajectory based on key performance metrics, namely DSC (Dice Similarity Coefficient) and HD (Hausdorff Distance). A benchmark dataset was collected, including various sole glue images along with their corresponding ground truth trajectories. This dataset was used to evaluate the performance of our model and existing algorithms. To ensure a fair comparison, we collected state-of-the-art algorithms and applied them to the same dataset, comparing their results with our DFCN model.

The quantitative metrics DSC and HD were used to evaluate the accuracy of trajectory extraction and the smoothness of generated trajectories.

Table 2. Quantitative comparison of segmentation results.

Method	Dice Similarity Coefficient (%)	Hausdorff Distance (mm)	Time (s)
Adaptive gluing trajectory extraction method	90.7	2.3	0.4~0.6
CAD software-based automatic acquisition system for shoe sole coating trajectory	91.2	2.2	0.5~0.7
Structured light-based shoe sole adhesive trajectory extraction	92.6	2.0	0.4~0.6
Our work	99.2	1.2	1.0~1.5

presents the quantitative evaluation values for the segmentation results of different methods. It can be observed in the table that our proposed method outperforms traditional shoe sole glue track generation methods in terms of segmentation accuracy measured by both DSC and HD metrics, which evidences that our method is able to accurately capture the desired glue application trajectories while generating smoother and visually appealing trajectories. In the comparison, the process time of our method is ~1.5 s, which is longer than other works. But the processing time of our model is still less than the time required for a robotic arm to spray one shoe. Thus, our processing time is able to meet the requirements for industrial production. Furthermore, it is worth noting that our inference was performed on an industrial computer with a low-end CPU (i5-8300H). We believe that the inference time of our method will be significantly improved by using a high-performance GPU. Overall, the comparison results demonstrate the effectiveness and superiority of our proposed DFCN model in extracting sole glue application trajectories. The combination of high accuracy, smoothness, and moderate computational efficiency makes it a promising choice for automated sole glue application tasks.

To visually demonstrate the accuracy of our model in generating sole spraying trajectories, we employed visualization tools to present the results. The degree of similarity between the model-generated trajectories and the ground truth trajectories can be visually observed. In the visualization, we use the ground truth trajectories as references and display the model-generated trajectories in different colors or line styles. This allows us to clearly see if the model successfully captures the shape and details of the trajectories. And, we can quickly evaluate the differences between the model-generated trajectories and the ground truth trajectories. If they closely align, it can be concluded that the model has a high level of accuracy. Conversely, if there are noticeable differences or deviations, we can further adjust the model or improve the algorithm to enhance the accuracy.

We randomly selected a set of shoe sole point cloud images to test our model. The generated spray trajectories are shown as yellow lines (Figure 9), while the standard spray trajectories for these shoe soles are indicated by the red lines. By comparing the two trajectories placed on the same graph, we can observe that the two lines are almost perfectly aligned. These results demonstrate the reliability of our model in generating spray trajectories for shoe soles. The ability of our model to accurately predict the trajectory enables precise and consistent spray application, leading to improved product quality. Moreover, the excellent alignment between the predicted spray trajectories and the standard ones is a testament to the robustness and generalizability of our model. It successfully captures the intricate details and shapes of various shoe sole designs, allowing for precise reproduction of the desired spray patterns.

4.4.2. Extract Spray Point

In addition to generating a high-precision spray trajectory, it is also crucial to extract 44 trajectory points in a reasonable manner, particularly in areas with significant curvature. Therefore, we need to include more points on the trajectory line to ensure fine control and accurate adhesive application. Figure 10 displays the post-processed adhesive trajectory points by using our method. It can be observed that these points closely match the actual adhesive path, indicating that our method has precise localization of adhesive points for the spray trajectory generation. We can achieve precise glue application on the shoe sole, ensuring precise positioning and paths for each adhesive point. This not only improves the efficiency of adhesive application but also ensures uniform coverage and quality of the adhesive. Compared to traditional manual adhesive methods, our approach exhibits significant advantages in terms of the accuracy and consistency of adhesive trajectories. This provides reliable technical support for automation and digitization in shoe sole production, thereby enhancing efficiency and quality assurance in the footwear industry.

4.4.3. Special Shoe Path Comparison

To validate the robustness of our proposed model, we compared it with the current mainstream methods by extracting spray trajectories of the same unique sole. The quality of the extracted spray trajectories from both methods was compared to assess their performance.

Most existing methods rely on unique edge contours and offsets to obtain the spray trajectories [20]. The discrete coordinate points of the sole’s edge contour are interpolated to obtain a smooth contour, and then the offset is applied to derive the spray trajectories. Following the method described in reference [32], the contour curve is initially used as a ridge line, and the intersection points between the pipe surface and the contour curve are calculated with a radius r to determine the trajectory points using the following formula:

$$\left\{\begin{array}{c}{\left[B_x\left(u\right)-C_x\left(t\right)\right]}^2+{\left[B_y\left(u\right)-C_y\left(t\right)\right]}^2+{\left[B_z\left(u\right)-C_z\left(t\right)\right]}^2=r^2\\ a\left[B_x\left(u\right)-C_x\left(t\right)\left]+b\right[B_y\left(u\right)-C_y\left(t\right)\left]+c\right[B_z\left(u\right)-C_z\left(t\right)\right]=0\end{array}\right.$$

(12)

where $C_x\left(t\right), C_y\left(t\right), C_z\left(t\right)$ are the corresponding point on the contour curve, $B_x\left(u\right), B_y\left(u\right), B_z\left(u\right)$ represent the point on the spray trajectory, and a, b, and c are the tangent vector of the point on the contour curve.

We compared our proposed method with the aforementioned approach and evaluated its performance in extracting spray trajectories. The obtained spray trajectories are shown in Figure 11. In the figure, the red point is the discrete trajectory obtained using the bias method. It can be observed that due to the inconsistency between the external contour of the shoe and the spray trajectory, the spray trajectory appears to be bent and deviates significantly from the actual spray trajectory. Furthermore, the missing information in the front part of the sole results in an additional small loop in the generated spray trajectory, which is a significant error. It can be seen that the proposed model in this study exhibits better robustness and accuracy in extracting spray trajectories of the same unique sole. It can capture subtle curve variations and edge shapes more precisely, resulting in smooth and accurate trajectories. In comparison, the method based on edge contours and offsets is more sensitive to complex sole shapes and curve variations, often leading to errors and inaccurate trajectories. Therefore, the proposed method in this study offers better robustness and reliability in practical applications, making it suitable for various types of sole spraying tasks.

5. Conclusions

In this paper, we constructed a deep deconvolutional neural network to provide a more effective solution for generating spray trajectories which exhibits greater applicability to irregular shoe soles and improves spray accuracy without compromising spray efficiency. The advantages of the proposed method lie in its ability to preserve more spatial information and achieve higher spray accuracy through end-to-end mapping from shoe sole images to shoe sole spray trajectories, resulting in an improved spray accuracy without sacrificing spray efficiency. Furthermore, according to the evaluation results, including a Dice similarity coefficient of 99.2% and a distance of 1.2 mm, this method surpasses the performance of traditional algorithms, which typically achieve around 90% Dice similarity coefficient and 2 mm distance. Moreover, the proposed method demonstrates practical feasibility, with a processing time limited to within 2 s, making it suitable for real-world implementation in manufacturing environments. Therefore, the algorithm in this paper has a much lower computing time than the time for the sole to be transported on the conveyor belt, and thus is more suitable for industrialization. This significant improvement highlights the enhanced accuracy and efficiency achieved by the proposed method. These unique features of our proposed method will contribute to boosting the production of spray trajectories that better align with the actual requirements of shoe sole gluing. Overall, the proposed method offers a valuable contribution to the footwear industry, addressing the limitations of existing approaches and providing a more effective solution for generating spray trajectories during automated shoe sole gluing. Future research will explore how to use different gluing processes according to the style of the sole and generate the corresponding gluing trajectory for the process to achieve a higher degree of intelligent production.