Intelligent Alignment Monitoring Method for Tortilla Processing Based on Machine Vision

Abstract

As people pay more and more attention to a healthy diet, it has become a consensus to eat more coarse grains. The development of its edible value is of great significance for a healthy human diet and has attracted the attention of many scholars and food processing companies. However, due to the differences in protein composition and structure between corn flour and wheat protein, it is difficult to form a network structure during processing, and the viscoelasticity and flexibility are poor. Based on this, this paper proposes a machine vision-based noodle positioning monitoring method to achieve noodle alignment monitoring in the noodle processing process. First, the images are captured by binocular cameras and preprocessed. Further, feature detection and matching algorithms are used to recover the pose information between binocular cameras, and then the recognition targets are matched. Finally, noodle alignment monitoring during noodle processing is achieved. Experiments show that the detection accuracy of the method proposed in this paper is much higher than the traditional manual detection, which can improve the noodle quality and reduce labor costs.

1. Introduction

Noodles are a traditional staple of Chinese food. They are easy to make and eat. Traditional noodles are mainly made from wheat flour [1]. However, in recent years, the increased precision of wheat flour processing has resulted in more nutritional losses. Long-term consumption can lead to a nutritional imbalance in the body and is harmful to health. As people pay more attention to a healthy diet, it has become a consensus to eat more coarse grains. Corn is one of the most productive and nutritious coarse grains. The development of its edible value is of great significance for a healthy human diet and has attracted the attention of many scholars and food processing companies. At present, corn flour has become one of the research hotspots in corn staple food processing. However, due to the differences in protein composition and structure between corn flour and wheat protein, it is difficult to form a network structure during processing, and the viscoelasticity and flexibility are poor. As a result, the dough quality is poor, and the texture is coarse and unsuitable for processing and making noodles, which greatly limits the application of corn flour in tortilla processing.

Among research advances in corn flour pretreatment to overcome the shortcomings of corn flour without gluten protein and to improve the suitability of corn flour for noodle products, corn flour is often pretreated. Commonly used treatments include raw material ultrafine, gelatinization, bio enzyme modification, and raw material puffing. Bai et al. [2] performed ultrafine grinding of corn to study the effect of corn flour fineness on the quality of corn noodles. The results showed that the fineness of corn flour had a great influence on the quality of corn noodles. As the fineness of cornmeal increased, it became more delicate, its crushing rate decreased significantly, and its sensory score showed a sharp increase. Particle size affected the physicochemical properties of the cornmeal. As the average particle size of maize flour decreased, the straight-chain starch content of maize flour first decreased slightly and then stabilized [3], while the noodles made from maize flour with less straight-chain starch were of better quality [4]. Li et al. [5] investigated the application of the gelatinization process in the development of maize noodles by pregelatinizing the maize flour. The results showed that the gelatinization process could improve the quality of corn noodles. Hou et al. [6] used yeast to ferment a corn-wheat flour mixture with the appropriate amount of gluten to produce corn-fermented noodles with good palatability, good elasticity, good toughness, moderately strong fermented flavor, and strong corn flavor.

After the improvement in raw materials, but in the processing process, if the noodles are not aligned, there will be noodle breakage, serious cases appear to block the machine, burn the machine, or even start a factory fire, which greatly reduces the efficiency of noodle processing. However, few studies have been carried out on such problems as noodles. Therefore, we use a machine vision to monitor whether the noodles processing process is aligned. The structure of the monitoring system is shown in Figure 1.

As shown in Figure 1, our detection system uses a dual camera. The camera is placed on top of the noodle processing; by collecting the position of each noodle, we go through the data and processing, and we have to reconstruct the noodle in 3D, after reconstruction we want to match it with the aligned noodle, here we use our alignment monitoring algorithm to see if each 3D angle is the same, and finally, if there is no alignment, we issue a warning.

Our contributions are as follows:

• We designed a binocular vision-based vision monitoring system that can easily monitor the alignment of noodles during the noodle processing;
• The article uses binocular vision cameras and inserts an SVM classifier in the subsequent algorithm, a stable and inexpensive architecture that is well-suited for noodle production.

The experiments prove that our method is much more efficient than traditional human efficiency in terms of accuracy rate and time efficiency.

2. Related Work

2.1. Theoretical Basis of Teaching Quality Evaluation Related Research

With the improvement of people’s lives in urban and rural areas, their dietary habits are also changing. Balanced nutrition has become a way of life. Coarse grain noodles with nutritional and health benefits are becoming more and more popular. Grains and cereals are rich in essential vitamins, minerals, and other nutrients and have a comprehensive amino acid composition. They have health benefits, such as enhancing human physiological functions and promoting metabolism. Consumed together with wheat flour, they are beneficial for balancing nutrition and delaying aging [7].

Maize is one of the three major food crops in the world and is mainly used as a food ingredient, feed ingredient, and raw material for industrial production. Since it is a dryland crop, it plays an important role in agricultural production because of its high acreage yield and a high potential for yield increase [8]. Currently, the United States ranks first in the world in corn production, with an annual output of more than 300 million tons, of which a total of 50 million tons of corn is deeply processed, accounting for more than 15% of the total corn production, and most of the rest is processed into feed. Maize is widely distributed in China, with an annual production of more than 150 million tons. It is one of the main foods for people in the mountainous regions of northern and southwestern China and other arid river valleys, with the second highest production in the world. However, the utilization of maize in China is low [9]. About 70% of maize is used for feed processing, and maize is poorly utilized in industry. Compared with developed countries such as the United States, China’s corn deep processing industry still has problems such as a small processing scale, scattered processing enterprises, and a short deep processing industry chain [10]. Therefore, it is important to promote the return of maize to the staple food, improve the utilization rate and economic value of maize, and accelerate the transformation rate of maize industrialization [11].

2.2. Target Recognition Methods

In the application of target recognition under complex background, it is difficult for the traditional threshold segmentation method to segment the target from the background because the difference between the target to be recognized and the surrounding environment is small [12]. The commonly used recognition methods are mainly divided into two categories: one is to establish a target template and match the target directly through template matching; the other is to preprocess the image to obtain the target features to be recognized and then perform secondary analysis within the featured neighborhood to extract the target. Albertop [13] et al. proposed a target recognition algorithm based on hough voting. For the problem that traditional edge detection results are sensitive to noise, a gradient direction-based edge detection method is proposed. Based on the CAD model features of the target, voting detection is performed on selected edge points to identify targets with high edge integrity. Jong [14] et al. used template matching to accomplish the identification and localization of connecting rods and other workpieces. By identifying distinct geometric features in connecting rod parts and defining the relationships between them, they built an offline template library that enables real-time identification and localization of simple stacked workpieces. Buchholz [15] et al. used the Ransam method to identify workpieces. Point cloud images of stacked workpieces were obtained by combining structured light scanning with a monocular camera. The RANSAM algorithm was used to match the point cloud with a 3D model of the workpiece. The stacked artifacts are segmented and localized and are highly robust to noise.

In recent years, machine learning has achieved good results in the field of target recognition. Szegedyc [16] et al. used deep neural networks for target recognition and localization. Through multi-scale analysis and recognition of the image, the target is selected from the rectangular frame with the smallest area, thus locating the target accurately in the image. Lee [17] et al. used hog feature descriptions to identify the vehicle in the image by sliding window detection traversing the whole image and machine learning to identify the vehicle. Lin [18] et al. proposed a morphology-based edge extraction operator that better solves the conflict between noise suppression and image detail retention. The method uses multiple morphological structure elements with different orientations and binarizes the target edges by using the grayscale weighted average as the threshold. At the same time, the final target is recognized using a fuzzy integrated judging technique. Li [19] et al. completed the recognition of occluded artifacts using a binary feature-matching method. The method combines the advantages of the free model and the simple model. The initial set of feature points is first obtained by fixed sampling, and then the final target matching is completed by random sampling in its neighborhood.

2.3. SVM Classification Methods

A support vector machine (SVM) is a supervised machine learning algorithm. It is a statistical learning method based on structural risk minimization and VC dimensional theory. It accomplishes classification in the sample space by creating an optimal hyperplane with a maximum interval in the sample space. The goal of support vector machine optimization is to minimize the structural risk, so it requires less data size and distribution and has better classification results than other methods on small sample sets. It is a well-researched classification algorithm that has been applied in many fields. Danieleb [20] et al. applied support vector machines to cell phone environment recognition by analyzing the sound signals captured by cell phones to determine the environment. The collected data are clustered and analyzed to reduce the dimensionality to solve the problems of real-time classification and space occupation, thus reducing the number of support vectors required to form the optimal hyperplane. Support vector machines have more accurate recognition and consume less memory than the commonly used Hidden Markov Models. Christians [21] et al. used SVMs to recognize human behavior in videos and used local spatial features and histogram features to characterize specific recognition behaviors. By classifying and learning from 2300 scenes, they obtained more accurate results than the nearest neighbor algorithm.

KD Renuka [22] and Ke Xu [23] applied support vector machines to email classification, built a database of spam samples, and analyzed their semantics. The combination of latent semantic indexing and support vector machine can determine whether emails are spam or not. Although support vector machines are effective in many fields, suitable classifier parameters require user experience and a lot of experiments. Subasia [24] et al. used a particle swarm optimization algorithm to optimize the input parameters to solve the parameter selection problem of support vector machines in myoelectric signal classification, which can save time and avoid getting into local minima compared with the traditional interpolation methods. The classification by the PSO-SVM classifier can accurately diagnose patients’ diseases. Dong Wenfei [25] et al. proposed an image retrieval technique based on multi-feature fusion, using DS theory to complete the weighted fusion of multiple features and combined with an SVM-based semantic classification technique. The results showed that semantic classification is better than traditional content classification in image retrieval. Yufeng Li [26] et al. studied the problem of completing multi-class image classification using SVM. The weight coefficients of individual features were obtained by normalizing the classification effects of different features in an image, thus combining individual classifiers into a complex classifier with certain adaptive capability and robustness.

3. Method

In this section, we first introduce the mathematical model of binocular vision 3D coordinate measurement, and further, the 3D deformation measurement algorithm is introduced to monitor and identify the noodles.

3.1. Mathematical Analysis of Binocular Stereo Vision

The 3D coordinate measurement principle and mathematical model binocular vision 3D coordinate measurement is based on the principle of parallax [27], as shown in Figure 2, which assumes that the two cameras’ imaging plane is in the same plane, i.e., the two cameras optical axis is parallel; the two cameras simultaneously image the spatial object point P, and the coordinates in the left and right images are $p_{left}=(X_{left},Y_{left}),p_{right}=(X_{right},Y_{right}),P(x_c,y_c,z_c)$. The spatial coordinates of the object point in the left camera coordinate system.

From trigonometric geometry, we have

$$\left\{\begin{array}{l}{x}_c=\frac{L\cdot X_{left}}{X_{left}-X_{right}}\\ y_c=\frac{L\cdot Y}{X_{left}-X_{right}}\\ z_c=\frac{L\cdot f}{X_{left}-X_{right}}\end{array}\right.$$

(1)

where the baseline distance L is the distance between the projection center lines of the two cameras, and, since the two camera imaging planes are in the same plane, $Y_{left}=Y_{left}=Y$.

On the basis of the principle of flat binocular stereo vision 3D measurement, the more general case is considered, i.e., there is no requirement for the position of the two cameras. As shown in Figure 3, let the left camera coordinate system $O_l-x_l{y}_l{z}_l$ coincide with the world coordinate system $O-xyz$, the coordinates of the space point P are $(x_p,y_p,z_p)$, and the image coordinate system is $O_1-x_1{y}_1$; the right camera coordinate system is $O_r-x_r{y}_r{z}_r$, and the image coordinate system is $O_r-x_r{y}_r$.

From the projective transformation, we have

$$\left\{\begin{array}{l}{s}_l{p}_l=M_l{X}_w\\ s_r{p}_r=M_r{X}_w\end{array}\right.$$

(2)

where $p_l$ and $p_r$ are the image coordinates of the spatial object point in the left and right cameras, respectively, $M_l$ and $M_r$ are the projection matrices of the left and right cameras, respectively, and $X_w$ is the 3D coordinate of the spatial object point in the world coordinate system $O_l-x_l{y}_l{z}_l$ The relationship between the coordinate system and the $O_r-x_r{y}_r{z}_r$ coordinate system is represented by the transformation matrix as

$$\left[\begin{array}{l}{x}_r\\ y_r\\ z_r\end{array}\right]=M_{lr}\left[\begin{array}{l}{x}_l\\ y_l\\ z_l\\ 1\end{array}\right]=\left[\begin{array}{l}\begin{array}{cccc}{r}_1& r_2& r_3& r_x\end{array}\\ \begin{array}{cccc}{r}_4& r_5& r_6& r_y\end{array}\\ \begin{array}{cccc}{r}_7& r_8& r_9& r_z\end{array}\end{array}\right]\left[\begin{array}{l}{x}_l\\ y_l\\ z_l\\ 1\end{array}\right]$$

(3)

The 3D coordinates of the spatial object point P can be obtained by associating Equations (2) and (3): where f_l and f_r are the effective focal lengths of the left and right cameras, respectively.

$$\left\{\begin{array}{l}{x}_p=z_p{X}_l/f_l\\ y_p=z_p{Y}_l/f_l\\ z_p=\frac{f_l(f_r{t}_x-X_r{t}_z)}{X_r(r_7{X}_l+r_8{Y}_l+f_l{r}_9)-f_r(r_1{X}_l+r_2{Y}_l+f_l{r}_3)}\end{array}\right.$$

(4)

3.2. Three-Dimensional Deformation Measurement Algorithm

For the 3D deformation at the structure measurement point, this paper uses the coordinate change of the measurement point in the world coordinate system at each moment to obtain the deformation information of the structure measurement point.

3.2.1. Displacement Measurement Algorithm

For the displacement measurement of the structure measurement point position, in order to make the measurement results easier to understand and intuitively represent the displacement information of the structure, this paper uses the 2D checkerboard grid target to establish the $xoy$ plane of the world coordinate system $o-xyz$ on the structure plane, and the specific establishment method is as follows.

1. Based on the known internal and external parameters of the camera of the binocular stereo vision structure deformation measurement system, the 2D checkerboard target is placed at the desired plane, and a single frame is acquired by the left camera of the measurement system.

2. Extract the image coordinates of each corner point of the checkerboard grid in the checkerboard grid image by the Hough transformation algorithm, as shown in Figure 4.

3. Assign the corresponding real 3D coordinates to each corner point, where the Z-axis coordinates of each corner point are assumed to be zero, and the X-axis and Y-axis coordinates are the real coordinates of the checkerboard grid relative to the set coordinate origin O, respectively.

4. Solve Equation (5) to obtain the stiffness transformation matrix [R, T] of the left camera coordinate system $O_l-x_l{y}_l{z}_l$ of the measurement system with respect to the established world coordinate system $o-xyz$.

$$s\left[\begin{array}{l}\begin{array}{ccc}{f}_x& \lambda & u\end{array}\\ \begin{array}{ccc}0& f_y& v\end{array}\\ \begin{array}{ccc}0& 0& 1\end{array}\end{array}\right]\left[\begin{array}{l}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{l}\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_x\end{array}\\ \begin{array}{cccc}{r}_{21}& r_{22}& r_{23}& t_y\end{array}\\ \begin{array}{cccc}{r}_{31}& r_{32}& r_{33}& t_z\end{array}\end{array}\right]\left[\begin{array}{l}x\\ y\\ z\\ 1\end{array}\right]=[R,T]\left[\begin{array}{l}x\\ y\\ z\\ 1\end{array}\right]$$

(5)

where $s\left[\begin{array}{l}\begin{array}{ccc}{f}_x& \lambda & u\end{array}\\ \begin{array}{ccc}0& f_y& v\end{array}\\ \begin{array}{ccc}0& 0& 1\end{array}\end{array}\right]$ is a known internal parameter of the left camera, s is a scale factor, and ${\left[u,v,1\right]}^T$ is the chi-square form of the angular image coordinates.

5. Using the stiffness conversion matrix obtained in the previous step, the 3D coordinates in the coordinate system of the left camera calculated by the measurement point Equation (4) are converted to the world coordinate system o, as shown in Equation (6).

$$\left[\begin{array}{l}x\\ y\\ z\end{array}\right]=R^{-1}(\left[\begin{array}{l}{x}_l\\ y_l\\ z_l\end{array}\right]-T)$$

(6)

The in-plane and out-of-plane displacements of the structure are calculated from the 3D coordinates of the measurement points in the established world coordinate system. Assume that the 3D coordinates of the measurement point P at time t are $(x_0,y_0,z_0)$. Due to the deformation of the structure, the 3D coordinates of the measurement point P at time t are $(x_i,y_i,z_i)$, as shown in Figure 5.

Then:

$$\left\{\begin{array}{l}{d}_i=\sqrt{{(x_i-x_0)}^2+{(y_i-y_0)}^2}\times sign(\frac{x_i-x_0}{\sqrt{{(x_i-x_0)}^2+{(y_i-y_0)}^2}})\\ d_i=z_i-z_0\end{array}\right.$$

(7)

3.2.2. Strain Measurement Algorithm

In this paper, the strain information in the area of the structural measurement point is determined by calculating the relative change in the distance between the measurement points. As shown in Figure 6, suppose the coordinates of structural measurement points at $t_0$ a time are $A(x_{A0},y_{A0},z_{A0})$ and $B(x_{B0},y_{B0},z_{B0})$, the distance between AB two measurement points is $l_0$, due to structural deformation, the coordinates of measurement point A and B at their time change to $A(x_{Ai},y_{Ai},z_{Ai})$ and $B(x_{Bi},y_{Bi},z_{Bi})$, the distance between AB two measurement points are $l_i$.

Then the strain between the measurement points AB at $t_i$ the time is:

$${\epsilon }_i=\frac{l_i-l_0}{l_0}$$

where

$$\begin{array}{l}{l}_0=\sqrt{{(x_{A0}-x_{B0})}^2+{(y_{A0}-y_{B0})}^2+{(z_{A0}-z_{B0})}^2}\\ l_i=\sqrt{{(x_{Ai}-x_{Bi})}^2+{(y_{Ai}-y_{Bi})}^2+{(z_{Ai}-z_{Bi})}^2}\end{array}$$

The change of 3D coordinates at the structure measurement point reflects the deformation information of the structure. Binocular stereo vision recovers the 3D deformation information of the measurement point based on the parallax of the same measurement point in different scenes. Therefore, the key problem of structural deformation measurement is how to determine the images of the same object point in different fields of view. One way to solve this problem is to select suitable image features to match the images of the measurement points in different fields of view. Therefore, image feature extraction and matching are key elements in binocular stereo-vision structural deformation measurement algorithms.

4. Experimental and Analysis

4.1. Datasets Collection and Preprocessing

The raw image captured by the binocular camera and transmitted to the computer includes the entire section of the tool and unnecessary noodle processing environments such as conveyor belts, spindles, machine workpieces, etc. After the acquired images are acquired, to further improve the detection accuracy, some noises in the images are first eliminated and attenuated, and then image preprocessing is performed. The specific steps of this link include smooth filtering, image enhancement, erosion, dilation, and other operations. Smoothing filtering is to filter out small noises in the image to achieve the effect of smoothing the overall image; image enhancement is to make up for the image loss caused by the filtering operation and enhance details; erosion and expansion are part of the selective filtering of non-interest points and the enhancement of interest points. Also part of noise filtering. After preliminary preprocessing, the image features are extracted. The demonstration of data preprocessing is shown in Figure 7.

During the structural deformation monitoring process, the binocular stereo vision deformation measurement system should not be disturbed to ensure that the relative positions between the two cameras in the measurement system and the position of the measurement system relative to the structural measurement points do not change. Before measurement, adjust the field of view so that the structural measurement point will not move out of the image of the two cameras due to displacement to ensure the continuous monitoring of the structural measurement point.

The brightness and lighting conditions of the environment often create noise disturbances in the image. For image processing methods based on gray pixel value, noise has a great influence on subsequent image processing results, so it is necessary to denoise the image. Median filtering is an effective noise reduction algorithm that minimizes the effects of noise while preserving image details [12]. Use a 3 × 3 median filter algorithm: Statistically sort the gray values of all pixel points, and set the sorting median as the gray size of the processed points. As shown in Figure 8, it is the histogram display of the image.

4.2. Train

For training, all composition images are randomly cropped to 320 × 320, 640 × 640, and 800 × 800. Then they are resized to a resolution of 320 × 320. The structure of the neural network used in this paper is 128 × 128 × 1, as shown in Figure 9. For loss optimization, we use ADAM. The learning rate is initialized to 0.01. The classification module is trained first. As shown in Figure 10, this is the graph of the loss based on the above parameter settings.

The actual grasping results are used to determine the classifier samples. Depending on the position of the noodle, the camera captures the pose and attempts to grasp the artifact. Since the failure rate of grasping in the initial case is high, grasping is performed by micro-motioning the camera to the grasping pose. First, the initial sample is captured 20 times, and the initial classifier is trained. Then incremental learning method is used. The acquired samples are included in the training sample set every three times, and the classifier is updated. A total of 50 subsequent attempts can be made to obtain the variation in the crawling success rate, as shown in Figure 11. The experiments show that as the number of crawls increases, the number of samples also increases, and the crawl success rate also increases. This demonstrates the effectiveness of incremental learning in this discipline.

4.3. Monitoring Accuracy Results

Further, we compare the method proposed in this paper with the traditional manual monitoring method. In this paper, the classification accuracy is selected as the evaluation index, and the index is calculated by dividing the number of correct classifications by the total number of noodles.

Table 1. Accuracy experimental results.

Number of Instances	Classification Accuracy (Ours)	Classification Accuracy (Traditional Manual Monitoring)
100	87.49	85.14
300	87.50	84.99
500	87.65	85.02
1000	88.51	84.94
3000	90.92	84.92

shows the results of the classification accuracy comparison between our method and traditional manual monitoring [4] when the number of instances (noodles) in the test set is 100, 300, 500, 1000, and 3000, respectively.

As can be seen in Table 1, the classification accuracy of our algorithm is generally higher than that of the traditional algorithm under different test sets. As the number of noodles increases, the accuracy of traditional manual monitoring decreases. In particular, as the number of instances in the test set increases, the accuracy is much higher than that of the traditional manual monitoring, which proves that our method is very effective in the forward monitoring of the noodle processing function.

5. Discussion

The accuracy of the noodle alignment monitoring method based on machine vision proposed in this paper is significantly better than the traditional manual efficiency, and the time efficiency can be greatly improved by using the noodle alignment monitoring method. Compared with other detection methods in other fields, the method proposed in this paper can also be applied to the detection of items such as tool wear, only changing the input.

6. Conclusions

In this paper, we propose a noodle alignment monitoring method based on machine vision to solve problems such as it being difficult for the noodle machine to align the noodles; the noodles are easy to break and block the machine. We first use the binocular camera to capture the image, and then after the image and processing, we carry out a 3D reconstruction of noodles, then use our alignment monitoring algorithm, and finally give the monitoring results of whether to align. Experiments show that our method is more effective than traditional artificial efficiency in terms of accuracy and time efficiency. In the future, we will be committed to researching and extending the method in this paper to other fields of agriculture, such as cutting machine recognition.