Research of Hand–Eye System with 3D Vision towards Flexible Assembly Application

Abstract

In order to improve industrial production efficiency, a hand–eye system based on 3D vision is proposed and the proposed system is applied to the assembly task of workpieces. First, a hand–eye calibration optimization algorithm based on data filtering is proposed in this paper. This method ensures the accuracy required for hand–eye calibration by filtering out part of the improper data. Furthermore, the improved U-net is adopted for image segmentation and SAC-IA coarse registration ICP fine registration method is adopted for point cloud registration. This method ensures that the 6D pose estimation of the object is more accurate. Through the hand–eye calibration method based on data filtering, the average error of hand–eye calibration is reduced by 0.42 mm to 0.08 mm. Compared with other models, the improved U-net proposed in this paper has higher accuracy for depth image segmentation, and the A_cc coefficient and D_ice coefficient achieve 0.961 and 0.876, respectively. The average translation error, average rotation error and average time-consuming of the object recognition and pose estimation methods proposed in this paper are 1.19 mm, 1.27°, and 7.5 s, respectively. The experimental results show that the proposed system in this paper can complete high-precision assembly tasks.

1. Introduction

The field of automatic robotic assembly has attracted much attention. In recent years, automatic robotic assembly technology has been gradually applied to various fields such as automobiles, aerospace, and electronics manufacturing. The application of automatic robotic assembly technology has greatly improved the production efficiency of enterprises. In automatic robotic assembly tasks, the robot is guided by vision sensors or force/torque (F/T) sensors to complete the assembly work.

The automatic robotic assembly system based on force/torque sensors senses the force of the workpiece in the assembly process, and guides the robot to complete the assembly task by analyzing the force model and adjusting feedback system. Peng et al. [1] designed a novel three-layer pose adjustment mechanism consisting of two parallel mechanisms as a force sensor to assist robots in completing automatic assembly tasks. Wang et al. used an elastic displacement device to sense errors in the assembly process and assist the robot in automatic assembly through closed-loop feedback [2]. Zeng et al. proposed an external force/torque calculation algorithm based on dynamic model identification to realize the flexible assembly of robots [3]. Gai et al. proposed a compliance control method to solve the insertion assembly problem [4]. Park et al. proposed a compliant nail hole assembly method based on blind search using spiral force trajectory (SFT) [5].

Though the automatic assembly problem can be solved with the assistance of a force/torque sensor, it still does not appear to be “flexible.” Therefore, researchers usually equip the system with vision sensors to assist the robot in more flexible automatic assembly. Ma et al. built an assembly system consisting of a robot, three cameras, a micro-force sensor and a specific gripper [6]. Li et al. proposed a three-dimensional visual method for object pose estimation coupled with admittance control to promote robotic shaft-in-hole assembly [7]. Liu et al. solved the peg-in-hole precise assembly problem by combining microscopic vision and force information. [8]. Qin et al. proposed a precision assembly method based on multi-camera micro-vision and three-dimensional force feedback [9]. Song et al. proposed robotic assembly skill learning with deep Q-learning using visual perspectives and force sensing to learn an assembly policy [10]. Wang et al. developed a high-precision assembly system combining robotic vision servo technology and robot force feedback control technology [11].

While the combination of a force/torque sensor and a vision sensor can solve the problem of automatic assembly by a robot, the force/torque sensor is expensive, which will increase the cost of the entire system. In order to improve assembly efficiency and reduce system cost, researchers are committed to the research of automatic robotic assembly system using only a visual sensor as the auxiliary. For example, an automatic assembly system based on stereo vision was researched by Chang et al. [12], which completes the precise assembly of mobile phone cases. Jiang et al. proposed a calibration method for the large-scale cabin assembly system (LCSS) with visual guidance [13]. Dong et al. realized robot assembly pose estimation through point cloud registration [14]. Yan et al. used a structured light 3D camera to build a high-precision robot assembly system to achieve high-precision assembly of two workpieces [15]. Litvak et al. proposed a high-precision two-stage attitude estimation method based on deep learning to realize automated assembly of workpieces [16]. Li et al. proposed an automatic vision positioning for precise grasping of workpieces in the assembly process [17].

For robot flexible automatic assembly tasks, this paper designs a hand–eye system based on the 3D vision, which includes two modules: hand–eye calibration and automatic assembly. The working flow chart of the system designed in this paper is shown in Figure 1.

The main contributions of this paper are as follows:

• A hand–eye calibration optimization method based on hand–eye data filtering is proposed to improve the accuracy of hand–eye calibration;
• An improved U-net segmentation method is proposed to accurately segment the depth image and achieve fast and accurate segmentation of point cloud;
• The point cloud registration strategy of “SAC-IA coarse registration-ICP fine registration” is adopted to achieve the target pose acquisition.

The rest of the paper is arranged as the following: the Section 2 introduces the principle of hand–eye calibration and the method flow based on hand–eye calibration data filtering; the Section 3 describes point cloud segmentation and object 6D pose estimation based on deep learning; the Section 4 verifies the advantages and feasibility of the system; finally, the Section 5 summarizes the work of the paper and prospects future research issues.

2. Optimization of Hand–Eye Calibration Based on Data Filtering

2.1. Mathematical Model and Error Analysis of Hand–Eye Calibration

The hand–eye calibration problem is usually defined as the problem of solving the equation AX = XB. The Figure 2 shows the schematic diagram of hand–eye calibration. The hand–eye calibration matrix solution satisfies Formula (1).

$$\begin{array}{c}{A}_{1}X{B}_1=A_{2}X{B}_2\\ {A_2}^{-1}{A}_{1}X=X{B}_2{B_1}^{-1}\\ AX=XB\end{array}$$

(1)

In the case of eye-to-hand, the positional relationship between the calibration object and the robot end remains unchanged. Therefore, according to multiple sets of hand–eye calibration data and the obtained X matrix, the transformation matrix $P^b$ from the point p in the calibration object coordinate system to the robot terminal coordinate system is estimated multiple times. The accuracy of matrix X is judged by calculating the standard deviation of multiple sets of transformation matrices $P^b$. The smaller the standard deviation, the more accurate the matrix obtained. This error evaluation method is called the reprojection error analysis method. Formulas (2) and (3) respectively represent the solution formulas for matrix $P^b$ and reprojection error E.

$$\left[\begin{array}{c}{P}^b\\ 1\end{array}\right]={{\overline{A}}_i}^{-1}\cdot X\cdot {\overline{B}}_i$$

(2)

where $P^b$ represents the transformation matrix from point p on the calibration object to the coordinate system of the robot end, ${\overline{A}}_i$ represents the transformation matrix of the i-th robot base coordinate system to the robot end coordinate system, ${\overline{B}}_i$ represents the transformation matrix from the i-th camera coordinate system to the calibration object coordinate system, X represents the hand–eye calibration matrix.

$$E=\sqrt{\frac{{\displaystyle {\sum }_{i=1}^n{(p_i-\overline{p})}^2}}{n}}$$

(3)

where, $p_i$ represents the estimated value of the pose of the point p on the end of the robot on the i-th set of hand–eye calibration data, $\overline{p}$ represents the average of the pose estimation values of point p on the end of the robot on the n sets of calibration objects.

2.2. Hand–Eye Calibration Optimization Based on Data Filtering

Researchers have proposed various theories to solve the problem of hand–eye calibration AX = XB [18,19,20]. Due to the calibration error of the camera’s external parameters and the robot’s own motion error, there will be errors in the hand–eye calibration solution, which is inevitable. The basic process of hand–eye calibration is shown in the Figure 3.

In the practical application of hand–eye calibration, improper selection of one or several sets of data will seriously affect the final calibration results. Aiming at the problem of excessive hand–eye calibration error caused by “bad” data, this paper proposes an optimization method. The method flow proposed in this paper is shown in Figure 4. Firstly, calculate the hand–eye calibration matrix X and its corresponding reprojection error E according to the M sets of initial data. Then, remove one set of data in turn and calculate the corresponding M sets of hand–eye calibration matrix $X_i$ and the corresponding reprojection error $E_i$ (i = 1, 2, …, M). Hand–eye calibration matrix $X_i$ and corresponding reprojection error $E_i$ are stored in array $X[ ]$ and $E[ ]$. The smallest element $E_{\min }$ in $E[ ]$ is compared with E. If $E_{\min }\le E$, then $E=E_{\min }$ and X is equal to the hand–eye calibration matrix $X\left[E_{\min }\right]$ corresponding to $E_{\min }$. This process only realizes the filtering of a single set of data. If you wish to realize the filtering of multiple sets of data, you can carry out this process several times. (Note: Data and matrix need to be updated.)

3. Object Recognition and Pose Estimation

3.1. Object Segmentation and Recognition Based on Improved U-Net

Point cloud segmentation refers to the segmentation of data points with the same attributes and different attributes according to the relevant functional definition of the original point cloud data to obtain the classification of each point. In automatic assembly tasks based on 3D vision, accurate segmentation of point cloud is helpful to improve the efficiency and accuracy of high point cloud registration. Traditional methods use the shape, color, curvature and other features of a point cloud to classify them, but these methods are slightly inadequate in accuracy and robustness. Aiming at the difficulty of segmentation of 3D point cloud data, an improved U-net [21] is proposed to segment a 2D depth image to achieve segmentation of a target 3D point cloud. Compared with the existing 3D point cloud segmentation algorithm, this method performs segmentation on a 2D image without the complex preprocessing of point clouds and has a higher segmentation efficiency. As a classical deep learning model, U-net has the advantages of fewer training samples, high segmentation accuracy, and being a lightweight model. In order to improve the segmentation accuracy of U-net, an attention module [22] is introduced in this paper to achieve accurate segmentation of object depth image.

The improved U-net structure in this paper is shown in Figure 5. The network is divided into two parts: a contracting path and an expansive path. The first layer uses a standard convolution module composed of two sets of 3 × 3 convolutions (string 1 and padding 1), batch normalization (BN), and ReLU activation to extract features. After down-sampling using max pooling, the second layer performs the aforementioned standard convolution operation. On the third and fifth layers of the network, in order to strengthen the feature extraction ability of the network, two sets of standard convolution operations are used after down-sampling. In the contracting path, the size of feature map is halved and the number of channels is doubled with each additional layer (the number of channels in the fifth layer remains unchanged). In the expansion path, in order to prevent over-fitting, the fifth layer is put into Dropout operation with a probability of 0.5. After up-sampling by bilinear interpolation, the feature maps from the fourth layer in the down-sampling stage are sent into the attention module together with the feature maps from the fourth layer in the down-sampling stage to output feature maps that enhance semantic information. The above operations are used on the sixth to the ninth layer of the network. With each additional floor, the size of the feature map is doubled and the number of channels is halved. For the feature map of the ninth layer, 1 × 1 convolution is used to reduce the channel number of the feature map to 1, and then sigmoid operation is performed to obtain the prediction probability map. In the prediction, the output probability map of layer nine is used to generate a binarized segmentation image with 0.5 as the threshold.

As shown in Figure 6, the attention feature fusion module (AFFM) constructed in this paper consists of two branches, which are the shallow feature map L from the contracting path and the deep feature map H from the expanding path. Shallow feature map L and deep feature map H focus and retain key features and spatial information through channel attention module (CAM) and spatial attention module (SAM). After completing residual calculation, the shallow feature map L is sent into the CAM. After deep feature map H is multiplied by shallow feature map L, residual calculation is completed and output feature map O is obtained.

AFFM mainly includes CAM and SAM, whose structures are shown in Figure 7 and Figure 8 respectively.

The image segmentation of assembly work is essentially a pixel-level binary classification task, and its loss function usually adopts the binary cross entropy loss (binary cross entropy loss, BCE Loss). In depth images, the number of target pixels is far less than the number of non-target pixels. If BCE is used as the loss function, the predicted results will be dominated by non-target pixels and the recognition ability will decrease. In addition, the target area in the image is often more difficult to identify than the non-target area. Therefore, in order to overcome these problems, this paper adopts the focus loss function, and its calculation formula is as follows:

$$\begin{array}{c}{F}_L(p_t)=-{\alpha }_t{(1-p_t)}^{\gamma }\log (p_t)\\ p_t=\{\begin{array}{c}p\\ 1-p\end{array}\begin{array}{c}y=1\\ y=0\end{array}\end{array}$$

(4)

where $p$ ($p\in \left[0,1\right]$) is the class probability output by the model, ${\alpha }_t$ is the loss weight of the t-th class sample, the sum of the loss weights of all classes is 1, and $\gamma$ ($\gamma \ge 0$) controls the size of the loss of the difficult and easy samples. When $\gamma$ increases, the model will pay more attention to samples that are difficult to distinguish; when $\gamma =0$, the focus loss function degenerates into a normal cross entropy function with ${\alpha }_t$.

3.2. Object Pose Estimation Based on “SAC-IA Coarse Registration–ICP Precise Registration”

Common point cloud registration algorithms include normal distribution transformation (NDT) [23], singular value decomposition (SVD) [24], iterative closest point (ICP) [25], and many improved algorithms. Among them, the principle of ICP algorithm is simple and easy to understand, and the registration effect is remarkable. But the ICP algorithm is very sensitive to the object’s initial pose; a bad initial pose may lead ICP to converge in a wrong pose. To solve the problems of ICP, the point cloud registration strategy of “SAC-IA coarse registration–ICP precise registration” is adopted in this paper. As shown in Figure 9, it is the flow chart of the point cloud registration strategy used in this paper. Firstly, the Fast Point Feature Histogram Description (FPFH) [26] is used as the point cloud feature description, and the sampling consistent initial algorithm (SAC-IA) is used to coarsely register the point cloud. Through SAC-IA coarse registration, the ICP point cloud registration can obtain a good initial value and avoid falling into the local optimum. The k-d tree data structure is used to improve the query speed of the nearest neighbors of the ICP algorithm.

In the preprocessing phase, it is necessary to use a filtering algorithm to remove outliers and down-sample the point cloud. The statistical outlier removal method was used to remove the noise and outliers. The point cloud $P_q^{\ast }$ after removing outliers and noise can be represented by Formula (5).

$$P_q^{\ast }=\left\{P_q^{\ast }\in P|({\mu }_k-{\sigma }_k)\le \overline{d}\le ({\mu }_k+{\sigma }_k)\right\}$$

(5)

where $\overline{d}$ represents the mean distance from a point $p_q$ in the point cloud P to the k nearest neighbors, ${\mu }_k$ and ${\sigma }_k$ represents the mean and standard deviation of the Gaussian distribution of the average distance of the point cloud, respectively.

After the removal of outliers and noises, we used Voxel-Grid algorithm to down-sample the point cloud. The Voxel-Grid down-sampling algorithm builds multiple voxels based on the input size (each voxel is a set containing a different number of points). Then, the centroid of each voxel is calculated. Finally, the other points of the corresponding voxels are represented by the centroids. The calculation formula of the centroid is shown in Formula (6).

$${\mu }_p=\frac{1}{m}{\displaystyle \sum _{i=1}^m{p}_i}$$

(6)

where $p_i$ represents the points contained in a voxel, and m represents the number of points in the voxel.

The FPFH feature was first proposed in [26], which reduces the computational complexity of the algorithm to O(nk) while still retaining most of the discriminative power of PFH. The FPFH feature diagram is shown in Figure 10. The expression of the FPFH feature is as follows:

$$FPFH(p)=SPF(p)+\frac{1}{k}{\displaystyle \sum _{i=1}^k\frac{1}{w_k}\cdot SPF(p_k)}$$

(7)

where the weight $w_k$ represents the distance between query point p and a neighbor point $p_k$ in a given metric space.

RANSAC is a common method to find the best match in cases where outliers are included. SAC-IA is a RANSAC-based algorithm for finding the best 3D rigid transformation matrix for 3D model registration. The algorithm flow is shown in Figure 11.

The ICP algorithm minimizes the distance between the point on the source point cloud and the corresponding point on the target point cloud through parameter update iteration. For the points in the source point cloud P, the corresponding closest point in the target point cloud Q is calculated by Euclidean distance. According to its corresponding relationship, solve the optimal transformation matrix (transition matrix R and displacement vector T). A new source point cloud P is obtained according to the transformation matrix, and the above process is iteratively executed until the convergence condition is satisfied. The objective function expression of the optimal transformation matrix is shown in Formula (8).

$$f(R,T)=\frac{1}{k}{\displaystyle \sum _{i=1}^k{\Vert q_i-(R{p}_i+T)\Vert }^2}$$

(8)

where the source cloud collection is $P=\left\{p_i|i=1,2,3,...\right\}$, $Q=\left\{q_i|i=1,2,3,...\right\}$.

4. Experiment and Discussion

The automatic robotic assembly system is shown in Figure 12. When the hand–eye calibration matrix is correct, the error of the rotation matrix has a negligible effect on the assembly, so the error is not discussed during the experiment.

4.1. Hand–Eye Calibration Experiment

In order to quantitatively evaluate the feasibility of the hand–eye calibration data filtering algorithm, we conducted 10 sets of repetitive experiments. The experimental platform is shown in Figure 13. In each experiment, 20 sets of hand–eye calibration data were collected and filtered out for 5 iterations. The error was calculated by Formula (3), and the result is shown in Figure 14. It can be seen from the experimental results that the hand–eye data filtering algorithm proposed in this paper reduces the reprojection error of the hand–eye calibration by 0.42 mm to 0.08 mm compared to the original data.

As shown in Figure 15, in order to further verify the feasibility of the proposed hand–eye calibration optimization algorithm, a point-to-point verification experiment was carried out. The error analysis between the transformation matrix and the actual thimble arrival pose was also carried out. The results of the hand–eye calibration error are shown in Figure 16. From the experimental results, the actual hand–eye calibration error was larger than the reprojection error. Through the hand–eye calibration optimization algorithm proposed in this paper, the hand–eye calibration error was reduced by 0.65 mm to 0.06 mm compared with the original data.

From the two verification experiments, the hand–eye calibration optimization method based on hand–eye data filtering proposed in this paper can improve the accuracy of hand–eye result. The method in this paper ensures the reliability of hand–eye calibration accuracy when non-professionals perform hand–eye calibration, which is very meaningful for industrial applications.

4.2. Pose Estimation and Assembly Experiment

In this paper, the Neutrik plug was used as the experimental object, and the binocular structured light 3D camera mentioned in this paper was used for data collection. During the data collection process, we set the aperture and exposure time of the camera and the projection light intensity of the projector to ensure that the collected data were under the same lighting environment (the experimental environment was indoors, so the influence of ambient light can be ignored). We collected 500 sets of Neutrik plug point cloud data with different poses and converted them into depth images. The image and depth image of the Neutrik plug are shown in Figure 17. After finishing the labeling of all image data, the data set was divided into a training set (400 sheets), a validation set (50 sheets), and a test set (50 sheets).

All models were trained and tested in the environment of Windows10 + Python3.6 + Pytorch1.1, and are accelerated by an RTX 2060 graphics card with 16 GB memory. In this paper, an adaptive moment estimation optimizer (Adam) was used to update the network parameters iteratively. The batch size was 4 and the training was 50 epochs. The initial value of the learning rate was set to 0.001, and the ReduceLROnPlateau dynamic learning rate adjustment strategy was used to complete the automatic decay of the learning rate.

In order to quantitatively evaluate the performance of the improved U-net image segmentation in this paper, accuracy (A_cc) and Dice coefficient (D_ice) were used as evaluation indicators. The calculation formula is as follows:

$$A_{cc}=\frac{T_P+T_N}{T_P+T_N+F_N+F_P}$$

(9)

$$D_{ice}=\frac{2T_P}{2T_P+F_P+F_N}$$

(10)

where $T_P$ is the number of pixels that are actually the target area and accurately recognized as the target area. $F_N$ is the number of pixels that are actually the target area but are recognized as non-target areas. $T_N$ is the actual non-target area and accurately recognized as non-target areas. $F_P$ is the number of pixels that are actually not the target area but are recognized as the target areas. The value range of the above indicators is between 0 and 1. The larger the value, the better the model performance.

As shown in

Table 1. Performance comparison of image segmentation of different deep learning models.

Methods	Acc	Dice
U-net	0.921	0.813
Attention U-Net [27]	0.933	0.825
R2U-Net [28]	0.937	0.831
DeepLab V3+ [29]	0.944	0.843
Our Method	0.961	0.876

,comparing the improved U-net in this paper with U-net, Attention U-net [27], R2U-net [28] and DeepLab V3+ [29], the results show that the A_cc coefficient and D_ice coefficient of the method in this paper reached 0.961 and 0.876, respectively, which is better than the other four methods.

As shown in Figure 18, it is the Neutrik plug depth image and the segmented image using the method in this paper. From the segmentation results, the improved U-net proposed in this paper can achieve point cloud segmentation for specific regions.

We used the point cloud registration method proposed in this paper to perform point cloud registration on the segmented point cloud data, and compare the registration results with other methods. The comparison results are shown in

Table 2. 6D Pose Estimation Results.

Methods	Rotation Mean Error (°)	Translation Mean Error (mm)	Runtime (s)
Seget-ICP [30]	1.55	1.27	6.5
BiLuNetICP [31]	1.32	1.25	6.8
PoseCNN + ICP [32]	1.29	1.21	11.5
GO-ICP [33]	1.35	1.33	16.7
Our Method	1.27	1.19	7.5

. It can be seen from the table that although the method proposed in this paper takes more time overall than Seget-ICP [30] and BiLuNetICP [31], the registration accuracy is improved. For assembly tasks, the increase in registration accuracy is conducive to the stability of the assembly process, so the method in this paper is valuable and meaningful.

The effect of point cloud registration using the method proposed in this paper is shown in Figure 19. As shown in the Figure 20, in order to verify the feasibility of the automatic assembly system proposed in this paper, a verification platform is built. The female Neutrik plug (Workpiece B) was fixed under the 3D camera, and the male Neutrik plug (Workpiece A) was clamped by the robot in a designated posture.

We compared the posture calculated by the automatic assembly system with the real posture and analyzed the error. The error of the experiments is shown in Figure 21. The experimental results show that the assembly error of the automatic assembly system in this paper is between 0.7~1.5 mm, which meets the requirements of Neutrik plug assembly.

5. Conclusions

The experiment results show that the general automatic assembly system based on 3D vision proposed in this paper is feasible. Through the hand–eye calibration method based on data filtering, the coordinates of the vision sensor and the coordinates of the robot are more accurately correlated. The improved U-net is used for image segmentation, which solves the issue of separating the target point cloud from the background. Compared with the traditional point cloud segmentation method, this method is more efficient. “SAC-IA coarse registration--ICP fine registration” is adopted to ensure the accuracy and efficiency of point cloud registration.

Of course, for the research of automatic assembly technology, the research in this paper still has some shortcomings. In the hand–eye calibration work, we directly obtained the motion parameters of the robot without compensating for the error disturbance of the robot motion parameters. Using inertial measurement units (IMU) to obtain the position of the robot is a common and effective means [34,35]. In future work, we will try to use IMU to obtain the motion parameters of the robot, which may help to further improve the accuracy of hand–eye calibration. In this paper, we used a deep-learning algorithm to improve the efficiency and accuracy of point cloud registration, but it needs to collect a large number of data samples and perform sample processing in advance. We need to further study the method of training deep learning based on small data samples to reduce the time cost.