MPCR-Net: Multiple Partial Point Clouds Registration Network Using a Global Template

Abstract

With advancements in photoelectric technology and computer image processing technology, the visual measurement method based on point clouds is gradually being applied to the 3D measurement of large workpieces. Point cloud registration is a key step in 3D measurement, and its registration accuracy directly affects the accuracy of 3D measurements. In this study, we designed a novel MPCR-Net for multiple partial point cloud registration networks. First, an ideal point cloud was extracted from the CAD model of the workpiece and used as the global template. Next, a deep neural network was used to search for the corresponding point groups between each partial point cloud and the global template point cloud. Then, the rigid body transformation matrix was learned according to these correspondence point groups to realize the registration of each partial point cloud. Finally, the iterative closest point algorithm was used to optimize the registration results to obtain the final point cloud model of the workpiece. We conducted point cloud registration experiments on untrained models and actual workpieces, and by comparing them with existing point cloud registration methods, we verified that the MPCR-Net could improve the accuracy and robustness of the 3D point cloud registration.

1. Introduction

Technical advances and market competition have pushed manufacturing companies to offer larger and more precise machine parts. This trend calls for 3D measurement systems with higher measuring efficiency and accuracy [1,2]. The visual measurement method based on point cloud data has the advantages of possessing high measurement speed and accuracy, and no contact with the workpiece; thus, it has gradually been applied to the 3D measurement of large workpieces.

3D point cloud reconstruction involves obtaining a series of partial point clouds of a workpiece under multiple poses using a 3D measuring device and then fusing these point clouds to generate a complete 3D point cloud of the workpiece [3,4]. The accuracy and reliability of 3D reconstruction directly affect the accuracy of the measurement.

A key step in point cloud 3D reconstruction is establishing a partial or global reconstruction model of the workpiece through a series of processing steps such as point cloud denoising, point cloud registration, and surface reconstruction. The purpose of point cloud registration is to move point clouds with different poses to the same posture through rigid body transformation to eliminate the misalignment between pairwise point clouds [5].

In the field of 3D reconstruction, the vast majority of point cloud registration methods concern multiple partial point cloud registration [6], the basic principle of which is to reduce or eliminate the cumulative error of the 3D reconstruction by minimizing the registration error between partial point clouds under multiple poses, thereby improving the reconstruction accuracy. However, traditional point cloud registration methods, such as the turntable method [7] and the labeling method [8], are handcrafted methods with disadvantages such as low efficiency, high requirements for equipment accuracy, and missing point clouds in labeled areas.

With the development of deep learning technology and a greater number of publicly available datasets for point cloud models, many scholars have attempted to use deep learning methods to achieve point cloud registration and verify its effectiveness through experiments. However, to reduce computational complexity and improve the efficiency of 3D reconstruction, the existing deep learning-based point cloud registration methods often degenerate the multiple partial point cloud registration problem into a pairwise registration problem [9]. Then, by optimizing the relative spatial pose between the pairwise partial point clouds, the final reconstruction model of the workpiece is constructed based on the pairwise registration results.

However, due to the camera pose and environmental limitations, it is often impossible to ensure a sufficient overlap area between the pairwise point clouds. If the relationship between the partial point clouds and the overall structure of the workpiece is ignored, only the registration of partial point clouds is performed. This will increase the difficulty of point cloud registration, reduce the registration accuracy, and increase the final 3D reconstruction error.

The following are the remaining gaps in point cloud registration algorithms that need to be addressed:

• Some algorithms require the structures of pairwise point clouds to be the same. If the geometric structures of the pairwise point clouds are quite different, the registration accuracy will decrease;
• Some algorithms can complete the registration of two partially overlapping point clouds through partial-to-partial point-cloud registration methods. However, these methods rely on the individual training of specific partial data of the point cloud to establish the correspondence point relationship between the pairwise point clouds. Moreover, the registration accuracy is very sensitive to changes in the points.

From the above analysis, it can be concluded that existing deep learning-based point cloud registration methods can only be used for scene reconstruction and other occasions with low accuracy requirements. They are not suitable for the 3D reconstruction of large workpieces with high accuracy requirements. To solve this problem, we propose a multiple partial point cloud registration network using a global template named MPCR-Net.

MPCR-Net was inspired by PointNet [10]. It uses deep neural networks to extract and fuse the learnable features of partial point clouds and the global template point cloud. It then trains the rigid body transformation matrix for partial point clouds to register the correspondence partial point cloud to the global template point cloud and finally forms a complete point cloud of the workpiece. In MPCR-Net, the partial point cloud has the characteristics of a local geometric structure of the workpiece and is sampled by the measuring device; the global template point cloud has the complete geometric structure of the workpiece and is converted from the CAD model of the workpiece.

The key contributions of our work are listed as follows:

• A multiple partial point cloud registration method based on a global template is proposed. Each partial point cloud is gradually registered to the global template in patches, which can effectively improve the accuracy of the point cloud registration.
• A clipping network for the global template point cloud, TPCC-Net (clipping network for template point cloud), was designed. In TPCC-Net, the features of partial point clouds and the global template point cloud are extracted and fused through a neural network, and the correspondence points of each partial point cloud are cut out from the global template point cloud. Compared to the existing registration algorithm based on deep learning, this method can reduce the correspondence point estimation error and improve registration efficiency.
• A parameter estimation network for rigid body transformation, TMPE-Net (parameter estimating network for transformation matrix), was designed. The learnable features of a partial point cloud and its correspondence points generated through the TPCC-NET were extracted through a neural network, and the parameters of the rigid body transformation matrix were estimated to minimize the learnable feature gap between the partial point cloud and the global template point cloud.

2. Related Work

2.1. Classic Registration Algorithms

Classic registration algorithms for point clouds mainly include the iterative closest point (ICP) algorithm [11,12], variants of ICP [13,14,15,16,17,18,19], and geometry-based registration algorithms [20,21,22,23,24].

The ICP algorithm [11,12] represents a major milestone in point cloud registration and is extensively applied in various ways. The essence of ICP is to minimize the sum of the distances between correspondence points of pairwise point clouds using iterative calculations, thereby optimizing the relative pose of the pairwise point clouds. When the relative pose deviation of the pairwise point clouds is small, the algorithm is guaranteed convergence and can obtain an excellent registration result. Scholars have made various improvements to the ICP algorithm to enhance registration efficiency [16] and accuracy [17,18,19]. However, all ICP-style algorithms still rely on the direct calculation of the closest point correspondences; moreover, they cannot dynamically adjust according to the number of points and easily fall into local minima.

The workflow of geometry-based registration algorithms is as follows: calculate the geometric feature descriptors between pairwise point clouds [25], determine the correspondence relationship of the pairwise point clouds according to the similarity of the descriptors, and calculate the optimal matrix for rigid body transformation between the pairwise point clouds. Random-sample-consistency-based registration algorithms [20], such as the sample consensus initial alignment (SAC-IA) [24], are the most widely used geometry-based registration algorithms. In SAC-IA, the correspondence points are established by calculating the local fast point feature histograms (FPFH) of the pairwise point clouds, and registration is subsequently accomplished by minimizing the distance between correspondence points. The algorithm can achieve invariance to the initial pose, and a satisfactory registration result can still be obtained when the overlap in the pairwise point clouds is low.

Unfortunately, the registration results of the geometry-based registration algorithms mainly depend on the calculation accuracy of the geometric feature descriptors between pairwise point clouds. It is necessary to manually adjust the calculation parameters involved in the geometric feature descriptor, such as the neighborhood radius of the FPFH descriptor, to minimize calculation errors and improve registration accuracy. This method consumes a significant amount of time, and it is not easy to determine its optimal parameters.

2.2. Deep Learning-Based Registration Algorithms

Recent studies have shown that point cloud registration algorithms based on deep learning have higher registration accuracy than classic point cloud registration algorithms [26,27,28,29,30].

Charles et al. [10] proposed an end-to-end deep neural network (PointNet) that could directly take point clouds as a network input. PointNet overcomes the shortcomings of general deep learning methods that cannot effectively extract features from unstructured point clouds, and it establishes a learnable structured representation method for unstructured point clouds. PointNet and its variants have been successfully applied in point cloud classification, object detection [31], and point cloud completion tasks [32].

In point cloud registration tasks, some deep learning algorithms use the PointNet architecture to obtain the learnable structural features of the unstructured point clouds, train rigid body transformation matrices for point cloud registration based on these features, and obtain satisfactory point cloud registration results.

PointNetLK [33] is the first deep learning-based algorithm to use a learnable structured representation method for point cloud registration. PointNetLK modifies the classical Lucas and Kanade (LK) algorithm [34] to circumvent the inherent inability of the PointNet representation to accommodate gradient estimates through convolution. This modified LK framework is then unrolled as a recurrent neural network in which PointNet is integrated to construct the PointNetLK architecture. However, PointNetLK and similar algorithms, such as the DCP [35], DeepGMR [36], and PCRNet [37], work on the assumption that all the points in the point clouds are inliers by default. Naturally, they perform poorly when one of the point clouds has missing points, as in the case of partial point clouds.

Unfortunately, in actual point cloud 3D reconstruction tasks, it is rare for the geometric structures of pairwise point clouds to be the same, especially for large workpieces. Each raw point cloud collected by the measuring device can only map the local structure of the workpiece.

A class of point cloud registration algorithms, PRNet [38] and RPM-Net [39], which also contain the PointNet architecture, can handle partial-to-partial point cloud registration. Their application range is wider than that of PointNetLK and other algorithms that can only perform global point cloud registration. Unfortunately, these algorithms do not scale well when the number of points increases. If the number of points of the pairwise point clouds differs significantly, the estimation result of the rigid body transformation matrix will fluctuate with the number of points, resulting in a decrease in registration accuracy.

Other deep learning-based algorithms, such as the DGR [40] and multi-view registration network [41], can use neural networks to filter out some outliers from the correspondence points of pairwise point clouds, but these algorithms require a clear correspondence between the pairwise point clouds.

3. MPCR-Net

3.1. Overview

Most existing deep learning-based partial-to-partial point cloud registration algorithms iteratively perform registration between partial point clouds to realize the registration between multiple partial point clouds, before finally building a complete reconstruction model of the workpiece. However, when the overlapping area between the pairwise partial point clouds is small, the registration accuracy cannot be guaranteed, and a large 3D reconstruction error accumulates after the iterative registration of multiple partial point clouds.

We propose MPCR-Net, a multiple partial point cloud registration network, which uses a global template to improve the registration accuracy of a large workpiece. In this network, the template point cloud extracted from a CAD model is used as the global registration template, and multiple partial point clouds are gradually pasted onto the global template; the relative poses of partial point clouds and the template point clouds are subsequently optimized to obtain a fully registered point cloud of all the partial point clouds. Compared with existing registration algorithms, the MPCR-Net can guarantee the overlap rate between the pairwise point clouds (the partial point cloud can be approximately regarded as a subset of the template point cloud), thus reducing the registration difficulty and error.

MPCR-Net mainly comprises TPCC-Net and TMPE-Net. TPCC-Net uses a deep neural network to extract and merge the features of a partial point cloud and the global template point cloud; it then “cuts out” a correspondence partial template point cloud in the global template point cloud. TMPE-Net merges the features of partial point clouds and the correspondence partial template point clouds, then iteratively learns the optimal rigid body transformation matrix.

As shown in Figure 1, the overall workflow of the MPCR-Net is as follows:

• Suppose there are n partial point clouds $S_1,\dots ,S_n$ and a global template point cloud $T$, $S_1,\dots ,S_n$ and $T$ are used as the inputs of TPCC-Net. In the TPCC-Net, the feature matrices $F\left(S_1\right),\dots ,F\left(S_n\right)$ of $S_1,\dots ,S_n$ and $F\left(T\right)$ of $T$ are obtained through the point cloud feature perceptron, respectively;
• Global feature vectors $Ø\left(S_1\right),\dots ,Ø\left(S_n\right)$ are obtained by pooling $F\left(S_1\right),\dots ,F\left(S_n\right)$, then the fusion features $F_1,\dots ,F_n$ are obtained by splicing $Ø\left(S_1\right),\dots ,Ø\left(S_n\right)$ and $F\left(T\right)$ [42];
• Index features $M_1,\dots ,M_n$ of $F_1,\dots ,F_n$ are obtained through the index feature perceptron, and the indexes $M_1^{\prime },\dots ,M_n^{\prime }$ are obtained by normalizing, filtering, and addressing $M_1,\dots ,M_n$. According to $M_1^{\prime },\dots ,M_n^{\prime }$, partial template clouds $T_1^{\prime },\dots ,T_n^{\prime }$ corresponding to $S_1,\dots ,S_n$ respectively are cut out from $T$;
• Partial template clouds and partial point clouds are input into the TMPE-Net in the form of correspondence point groups $\left\{\left(S_1,T_1^{\prime }\right),\left(S_2,T_2^{\prime }\right),\dots ,\left(S_n,T_n^{\prime }\right)\right\}$. In the TMPE -Net, the global feature vector $\left\{\left(Ø\left(S_1\right),Ø\left(T_1^{\prime }\right)\right),\left(Ø\left(S_2\right),Ø\left(T_2^{\prime }\right)\right),\dots ,\left(Ø\left(S_n\right),Ø\left(T_n^{\prime }\right)\right)\right\}$ of $\left\{\left(S_1,T_1^{\prime }\right),\left(S_2,T_2^{\prime }\right),\dots ,\left(S_n,T_n^{\prime }\right)\right\}$ is obtained from the point cloud feature perceptron and the pooling layer, and each group of global feature vectors is spliced to obtain the global fusion feature $F_1^{\prime },\dots ,F_n^{\prime }$;
• The dimension of $F_1^{\prime },\dots ,F_n^{\prime }$ is reduced through the transformation parameter perceptron and output the transformation parameter vectors $Z_1,\dots ,Z_n$, and the rigid body transformation matrixes $G_1,\dots ,G_n$ are constructed according to $Z_1,\dots ,Z_n$;
• Rigid body transformations are performed on $S_1,\dots ,S_n$ according to transformation matrices $G_1,\dots ,G_n$, and the above steps are repeated iteratively to calculate $G_1,\dots ,G_n$ until $G_1,\dots ,G_n$ meet the stop condition C; then, all the iteration results are combined to construct the optimal rigid body transformation matrix $G_1^{\prime },\dots ,G_n^{\prime }$;
• $G_1,\dots ,G_n$ is used to register $S_1,\dots ,S_n$ to $T$, and the ICP algorithm optimizes the registration results to obtain point clouds $W_1,\dots ,W_n$. $W_1,\dots ,W_n$ are adjusted to the same coordinate system and obtain a fully registered point cloud $W^{\prime }$ spliced by $W_1,\dots ,W_n$. Subsequent 3D reconstruction tasks, such as surface reconstruction, can be performed based on $W^{\prime }$.

3.2. TPCC-Net

3.2.1. Mathematical Formulation

Suppose that the index of a partial template point cloud $T_i^{\prime }$ corresponding to the partial point cloud $S_i$ in the global template point cloud $T$ is $M_i^{\prime }$; then

$$T_i^{\prime }=M_i^{\prime }\otimes Ti\in \left[1,n\right]$$

(1)

where the operator $\otimes$ is used to represent the process of estimating $T_i^{\prime }$ in $T$.

The symbol $Ø:{ℝ}^{N\times 3}\to {ℝ}^{N\times K}$ represents the extraction operation of the global features of a point cloud. For example, $Ø\left(P\right)$ means to transform point cloud $P$ into a K-dimensional global feature vector.

Assuming that $S_i$ can be registered to $T$ completely, then $S_i$ and $T$ have similar global features of the point cloud, that is:

$$Ø\left(S_i\right)\approx Ø\left(R{T}_i^{\prime }+t\right)$$

(2)

In Equation (2), $R\in {ℝ}^{3\times 3}$ and $t\in {ℝ}^{N_y\times 3}$ are the rotation and translation matrices used in the registration, respectively. The pose relationship between $S_i$ and $T$ is uncertain before registration, that is, the values of $R$ and $t$ are unknown; therefore, we temporarily ignore the influence of $R$ and $t$ and use Equation (3) to establish a weaker condition to relate $S_i$ to $T_i^{\prime }$:

$$Ø\left(S_i\right)\approx Ø\left(T_i^{\prime }\right)$$

(3)

that is,

$$Ø\left(S_i\right)\approx Ø\left(M_i^{\prime }\otimes T\right)$$

(4)

Equation (4) indicates that the point clouds $T$ and $S_i$ input to the TPCC-Net are linked together through the index $M_i^{\prime }$, and all correspondence points that are similar to the features of $S_i$ can be determined from $T$. We assume that $M_i^{\prime }$ can be calculated by operation $f(\cdot )$; then:

$$M_i^{\prime }=f\left(Ø\left(S_i\right),Ø\left(T\right)\right)$$

(5)

The search for correspondence points is to learn the index $M_i^{\prime }$ by training the TPCC-Net and then cutting out the correspondence point set $T_i^{\prime }$ of $S_i$ from the global template point cloud $T$ according to $M_i^{\prime }$. We consider estimating $M_i^{\prime }$ by fusing the point cloud features of $T$ and $S_i$.

3.2.2. Network Architecture

TPCC-Net mainly includes a point cloud feature perceptron, pooling layer, and index feature perceptron. The architecture of TPCC-Net is shown in Figure 2.

The point cloud feature perceptron comprises of five convolutional layers of sizes 3–64, 64–128, 128–128, 128–512, and 512–1024. The main purpose of the point cloud feature perceptron and pooling layer is to generate the feature matrix and global feature vector of two input point clouds. The index feature perceptron consists of three convolutional layers of sizes 2048–1024, 1024–512, and 512–256 and two fully connected layers of sizes 256–128 and 128–1. Its main purpose is to estimate the index vector for correspondence point searching through dimensional transformation.

3.2.3. Working Process

TPCC-Net is divided into three main functional blocks. Taking the correspondence point estimation process of $S_i$ as an example, the working process of TPCC-Net is as follows.

• Extract and fuse point cloud features
  • Suppose point clouds $S_i$ and $T$ contain $N_x$ and $N_y$ data points, respectively, and $N_x<N_y$. Input $S_i$ and $T$ to the point cloud feature perceptron; it consists of five multi-layered perceptrons (MLPs), similar to PointNet. The dimensions of $S_i$ and $T$ are both increased to 1024 after the convolution processing of the point cloud feature perceptron. Afterward, generate the feature matrices $F\left(S_i\right)\in {ℝ}^{N_y\times 1024}$ and $F\left(T\right)\in {ℝ}^{N_x\times 1024}$ of $S_i$ and $T$. Weights are shared between the MLPs used for $S_i$ and $T$.
  • Use the max-pooling function to downsample $F\left(S_i\right)$ to generate a global feature vector $Ø\left(S_i\right)$ corresponding to $S_i$.
  • Join $Ø\left(S_i\right)$ and $F\left(T\right)$ to build a point cloud fusion feature $F_i\in {ℝ}^{N_x\times 2048}$.
• Construct the index vector
  • Input $F_i$ to the indexed feature perceptron; the dimension of $F_i$ is reduced to one, and the index feature $M\in {ℝ}^{N_x\times 1}$ is then output.
  • Use the Tanh activation function to normalize $M$ to construct the index vector $M^{\prime }\in {ℝ}^{N_x\times 1}$.
• Predict the correspondence points
  • Encode all data points in $T$ to construct the index $M_T\in {ℝ}^{N_x\times 2}$.
  • Filter out the first $N_y$ elements close to zero from the index vector $M^{\prime }\in {ℝ}^{N_x\times 1}$ to form the index element vector $M^{″}\in {ℝ}^{N_y\times 1}$.
  • Find the address of the above $N_y$ elements in $M^{\prime }$ to construct the index $M^{‴}\in {ℝ}^{N_y\times 2}$.
  • $M^{‴}\in M_T$, according to each index in $M^{‴}$; the elements corresponding to the index in $M^{‴}$ are extracted from the global template point cloud $T$, and all the extracted elements in $T$ are combined to construct the estimated correspondence point set $T_i^{\prime }$ of $S_i$ in $T$. The correspondence point estimation process is shown in Figure 3, where the purple part is the correspondence point set $T_i^{\prime }$.

3.3. TMPE-Net

3.3.1. Mathematical Formulation

Suppose that $S_i$ can be fully registered to $T$ through a rigid body transformation; then the global features of the point clouds of $S_i$ and $T_i^{\prime }$ are similar, where $T_i^{\prime }$ is the corresponding point set of $T$ that is generated through TPCC-Net. Assuming that the rotation matrix and the translation vector used in the rigid body transformation are $R\in {ℝ}^{3\times 3}$ and $t\in {ℝ}^{N_y\times 3}$, respectively, then:

$$Ø\left(S_i\right)\approx Ø\left(R{T}_i^{\prime }+t\right)$$

(6)

Assuming that $R$ and $t$ can be calculated by operation $\mathrm{H}(\cdot )$, then

$$\left\{R,t\right\}=\mathrm{H}\left(Ø\left(S_i\right),Ø\left(T_i^{\prime }\right)\right)$$

(7)

Similar to PointNetLK [33], in the TMPE-Net, we directly estimate $R$ and $t$ by fusing the point cloud features of $T_i^{\prime }$ and $S_i$, and further optimize $R$ and $t$ by iteration $S_i$.

3.3.2. Network Architecture

As shown in Figure 4, TMPE-Net mainly includes a point cloud feature perceptron, pooling layer, and transformation parameter perceptron. The architecture of the point cloud feature perceptron is the same as that of TPCC-Net. The architecture of transformation parameter perceptron is also similar to that of TPCC-Net, which consists of three convolutional layers of sizes 2048–1024, 1024–512, and 512–256, and two fully connected layers of sizes 256–128 and 128–7. Its main function is to estimate $R$ and $t$.

3.3.3. Working Process

Taking the rigid body transformation of $S_i$ as an example, TPCC-Net works as follows:

• Extract and fuse the global feature vector of point clouds
  • Input $S_i\in {ℝ}^{N_y\times 3}$ and its corresponding point set $T_i^{\prime }\in {ℝ}^{N_y\times 3}$ into point cloud feature perceptron; then, the dimensions of $S_i$ and $T_i^{\prime }$ are both increased to 1024, and the feature matrices $F\left(S_i\right)\in {ℝ}^{N_y\times 1024}$ and $F\left({T^{\prime }}_i\right)\in {ℝ}^{N_y\times 1024}$ of $S_i$ and $T_i^{\prime }$ are generated. The weights of all convolutional layers in the point cloud feature perceptron are shared for $S_i$ and $T_i^{\prime }$.
  • Use the max-pooling function to downsample $F\left(S_i\right)$ and $F\left(T_i^{\prime }\right)$ to construct the global feature vectors $Ø\left(S_i\right)\in {ℝ}^{1\times 1024}$ and $Ø\left(T_i^{\prime }\right)\in {ℝ}^{1\times 1024}$ that correspond to $S_i$ and $T_i^{\prime }$, respectively.
  • Join $Ø\left(S_i\right)$ and $F\left(T_i^{\prime }\right)$ to build a global fusion feature $F_i\in {ℝ}^{N_x\times 2048}$ of the point clouds.
• Construct the parameter vector

Input $F_i^{\prime }$ to the transformation parameter perceptron and the dimension of $F_i^{\prime }$ is reduced to seven; then, output the transformation parameter vector $Z\in {ℝ}^{1\times 7}$.

3.

Estimate the rigid body transformation

In TMPE-Net, we estimate the rigid body transformation matrix through iterative training, and the process is as follows:

Suppose that the rigid body transformation matrix has been iteratively calculated $m$ times, and parameter vector $Z_j$ can be obtained in the j-th ($j\in \left[1,m\right]$) iteration. Assuming that $Z_j=\left[\begin{array}{ccccccc}{a}_0& a_1& a_2& a_3& a_4& a_5& a_6\end{array}\right]$, we use the first four elements ($a_0,\dots ,a_3$) in $Z_j$ to construct the rotation matrix $R_j$ [43]:

$$R_j=\left[\begin{array}{ccc}{a}_0{}^2+a_1{}^2-a_2{}^2-a_3{}^2& 2\left(a_1{a}_2-a_0{a}_3\right)& 2\left(a_1{a}_3+a_0{a}_2\right)\\ 2\left(a_1{a}_2+a_0{a}_3\right)& a_0{}^2+a_2{}^2-a_1{}^2-a_3{}^2& 2\left(a_2{a}_3-a_0{a}_1\right)\\ 2\left(a_1{a}_3-a_0{a}_2\right)& 2\left(a_2{a}_3+a_0{a}_1\right)& a_0{}^2+a_3{}^2-a_1{}^2-a_1{}^2\end{array}\right]$$

(8)

We then use the last three elements ($a_4,\dots ,a_6$) in $Z_j$ to construct the translation vector $t_j$:

$$t_j=\left[\begin{array}{ccc}{a}_4& a_5& a_6\end{array}\right]$$

(9)

The rigid body transformation matrix estimated in one iteration is $G_j=\left\{R_j,t_j\right\}$.

Suppose that the input partial point cloud of the j-th iteration is ${\left(\left[S_i\right]\right)}_j$, and $G_j$ is the corresponding estimated rigid body transformation matrix for ${\left(\left[S_i\right]\right)}_j$. Use $G_j$ to perform a rigid body transformation on ${\left(\left[S_i\right]\right)}_j$ to form a new partial point cloud ${\left(\left[S_i\right]\right)}_{j+1}$, and use $\odot$ to represent the rigid body transformation operation.

$${\left(\left[S_i\right]\right)}_{j+1}=G_j\odot {\left(\left[S_i\right]\right)}_j$$

(10)

Input ${\left(\left[S_i\right]\right)}_{j+1}$ to TMPE-Net for the (j + 1)-th iteration and output the rigid body transformation matrix $G_{j+1}$. Then, the difference $\Delta G$ between the results of two consecutive iterations is calculated.

$$\Delta G=\Vert {\overline{G}}_j{\left({\overline{G}}_{j+1}\right)}^{-1}-I\Vert$$

(11)

In Equation (11), ${\overline{G}}_j\in {ℝ}^{4\times 4}$ and ${\overline{G}}_{j+1}\in {ℝ}^{4\times 4}$ are the exponential mapping of $G_j$ and $G_{j+1}$ respectively, which refer to the rigid body transformation matrix of two consecutive iterations. $\Vert \cdot \Vert$ represents the sum of squares of all elements of the matrix.

Set the minimum rigid body transformation threshold $H_G$ and the maximum number of iterations $H_N$, and terminate the iterative training when $\Delta G\le H_G$ or the current iteration number $N\ge H_N$. We assume that TMPE-Net performs $m$ iteration calculations according to these conditions. Combine the rigid body transformation matrices from each iteration to obtain the final trained rigid body transformation matrix $G^{\prime }$.

If the total number of iterations is $m$, the final estimate $G^{\prime }$ can be computed during the iterative loop:

$$G^{\prime }=G_1\otimes G_2\otimes \dots \otimes G_m=\left\{R_1\times R_2\times ,\dots \times R_n,t_1+t_2+,\dots +t_m\right\}$$

(12)

In Equation (12), $\otimes$ represents the combined operation of all the iterated rigid body transformation matrices. The iterative estimation process of the rigid body transformation matrix is shown in Figure 5.

3.4. Loss Function

TPCC-Net uses the loss function $Los{s}_1$ to maximize the registration performance (i.e., the total number of correspondence points between the partial point clouds in the global template point cloud). In TMPE-Net, the loss function $Los{s}_2$ is the summation of two terms $Los{s}_a$ and $Los{s}_b$. The objective of $Los{s}_a$ is to minimize the difference between the real rigid body transformation matrix $g$ and the estimated rigid body transformation matrix $G^{\prime }$ between the partial point clouds and corresponding partial template point clouds. The target of $Los{s}_b$ is to minimize the difference between the global feature vectors of the partial template point clouds $T_1^{\prime },\dots ,T_n^{\prime }$ and partial point clouds $S_1,\dots ,S_n$.

The mean square error (MSE) [44] is used to express the above loss function as follows:

$$\{\begin{array}{l}Los{s}_1={\Vert N_{TPCC}-N_y\Vert }_F\\ Los{s}_2=Los{s}_a+Los{s}_b={\Vert G^{\prime }-g\Vert }_F+{\Vert Ø\left(T^{\prime }\right)-Ø\left(S\right)\Vert }_F\end{array}$$

(13)

where ${\Vert \cdot \Vert }_F$ represents the MSE between the internal elements calculated; $Ø\left(T^{\prime }\right)$ and $Ø\left(S\right)$ represent the global feature vectors of the point cloud $T^{\prime }$ and $S$, respectively; $N_{TPCC}$ is the number of correspondence points correctly estimated by the TPCC-Net in the global template point cloud, and $N_y$ is the number of points in the partial point cloud.

3.5. Training

3.5.1. Preprocessing of Training Data

The original data used for training was the ModelNet40 [45] dataset, which contains 12468 CAD models in 40 categories. We randomly designated 20 categories of models as the training set and the other 20 categories of models as the test set. The creation process of a global template point cloud and a partial point cloud is illustrated in Figure 6.

For each original point cloud in ModelNet40, the global template point cloud was obtained through voxel grid downsampling [46,47], consisting of 1024 data points.

The process of creating a partial point cloud was as follows:

• Each initial partial point cloud contained 568 data points, which were “cut” from a global template point cloud using the farthest point sampling (FPS) [42] algorithm.
• Gaussian noise of 0.0075 level to the initial partial point cloud was added to simulate the deviation of the coordinate value between the scanned data point and the data point in the global template point cloud under noisy conditions.
• 284 outlier noise points were randomly added to the initial partial point cloud to simulate the disturbance of the scanned point cloud structure by environmental noise and sensor error. This increased the structural difference between the initial partial point cloud and the template point cloud.
• A random rigid body transformation matrix was created, through which the initial partial point cloud was subjected to random rotation transformation (±45° around each Cartesian coordinate axis) around the origin of the coordinate and a random translation transformation (±0.5 unit along each Cartesian coordinate axis) to obtain the partial point cloud.

3.5.2. Training Method

TPCC-Net and TMPE-Net were trained using transfer learning [43]. First, TPCC-Net was trained separately to obtain the optimal network model parameter $P_a$, which means that TPCC-Net using parameter $P_a$ would perform the best on the test set. Then, using the model parameter $P_a$ as the pre-training model of TMPE-Net, the rigid body transformation matrix between the correspondence point set and the partial point cloud was iteratively trained and evaluated in the test set. Finally, the optimal network model parameter $P_b$ of TMPE-Net was obtained.

4. Experiments

4.1. Experimental Environment

The software and hardware specifications used in the experiment are shown in

Table 1. Experimental environment.

Environment	Configuration
Software	Operating system	Windows 10
	Deep learning framework	Pytorch 1.8.1 + CUDA 11.0 + cuDNN
	Programming language	Python 3.8.3
	Point cloud processing library	Open3D
Hardware	CPU Memory	Intel(R) Core(TM) i5-9400F 16 GB
	Graphics card	Nvidia GeForce GTX 1070 8 GB

.

4.2. Experiments Based on Untrained Models

We used untrained models in ModelNet40 to carry out correspondence point estimation and registration experiments, and explored the influence of correspondence points estimation accuracy, point cloud registration accuracy, and registration efficiency.

4.2.1. Evaluation Criteria for Experiments

1.

Estimation accuracy

The estimation accuracy of correspondence points refers to the proportion of correspondence points correctly estimated by the algorithm during the experiment to the actual number of correspondence points. Taking MPCR-Net as an example, the estimation process of TPCC-Net is as follows:

Suppose the point clouds input into the network are partial point cloud A and global template point cloud B, and the point number of B is $N_B$. Encode data points in B and extract the index addresses $M_{True}\in {ℝ}^{N_x\times 2}$ of all correspondence points of A and B. Input A and B to TPCC-Net to estimate the index address $M_{ES}\in {ℝ}^{N_x\times 2}$. Count the number of elements that are the same in $M_{True}$ and $M_{ES}$, and denote it as $N_{ES}$. The actual number of correspondence points is equal to $N_B$; thus, the estimation accuracy of correspondence points of TPCC-Net is:

$$P_1=\frac{N_{ES}}{N_B}$$

(14)

2.

Registration error

Rigid body transformation is composed of translation and rotation transformations. The registration error is subdivided into rotation and translation transformation errors. The registration error is calculated as follows:

Suppose the actual rotation angle of the partial point cloud relative to the template point cloud in the three Cartesian coordinate axis directions is $R=\left\{\alpha ,\beta ,\gamma \right\}$, and the actual translation distance is $t=\left\{d_0,d_1,d_2\right\}$. Input the global template point cloud and the corresponding point set of the partial point cloud in the global template point cloud into TMPE-Net and estimate the parameter vector $Z=\left[\begin{array}{ccccccc}{a}_0& a_1& a_2& a_3& a_4& a_5& a_6\end{array}\right]$.

Elements $a_0,\dots ,a_3$ in $Z$ represent the four parameters for the quaternion rotation matrix. The rotation Euler angles in the three directions corresponding to the parameter vector can be obtained according to the relationship between the quaternion rotation matrix and the Euler angle rotation matrix:

$$R^{\prime }=\left\{\begin{array}{l}{\alpha }^{\prime }\\ {\beta }^{\prime }\\ {\gamma }^{\prime }\end{array}\right\}=\left\{\begin{array}{l}\arctan \frac{2(a_0{a}_1+a_2{a}_3)}{1-2\left(a_1{}^2-a_2{}^2\right)}\\ 2\arcsin \left(a_0{a}_2-a_3{a}_1\right)\\ \arctan \frac{2(a_0{a}_3+a_1{a}_2)}{1-2\left(a_2{}^2-a_3{}^2\right)}\end{array}\right\}$$

(15)

Calculate the mean absolute error $\Delta R$ between $R^{\prime }$ and $R$:

$$\Delta R=\frac{\left|{\alpha }^{\prime }-\alpha \right|+\left|{\beta }^{\prime }-\beta \right|+\left|\gamma -{\gamma }^{\prime }\right|}{3}$$

(16)

$\Delta R$ is the rotation transformation error of TMPE-Net.

Elements $a_4,a_5,a_6$ in $Z$ represents the translation distance $t^{\prime }=\left\{a_4,a_5,a_6\right\}$ of the partial point cloud relative to the template point cloud in the three directions estimated by TMPE-Net, and the average absolute difference $\Delta t$ between $t^{\prime }$ and $t$ is calculated as follows:

$$\Delta t=\frac{\left|a_4-d_0\right|+\left|a_5-d_1\right|+\left|a_6-d_3\right|}{3}$$

(17)

$\Delta t$ is the translation transformation error of TMPE-Net.

3.

Evaluation of work efficiency

We consider the correspondence point search between partial point clouds and the template point cloud as the preparation stage for the point cloud registration, and the total time of the correspondence point estimation and rigid body transformation matrix calculation (registration) is used to measure the efficiency of the point cloud registration.

4.2.2. Correspondence Point Estimation

We used deep learning-based algorithms such as MPCR-Net, PRNet, and RPM-Net, which have correspondence point estimation and point cloud local registration functions, to carry out the correspondence point estimation experiments.

In the experiment, each global template point cloud contained 1024 data points and was sampled from each original point cloud model in the ModelNet40 dataset through the voxel grid downsampling [44,45,46] algorithm. The process of creating a partial point cloud was as follows:

• Using the FPS algorithm, the initial partial point cloud was sampled from the global template point cloud according to a sampling ratio of 0.05 to 0.95; the sampling ratio refers to the ratio of the data volume of the initial partial point cloud to the global template point cloud.
• The initial local point cloud was rotated by 20° around the three Cartesian coordinate axes with the coordinate origin as the center and translated 0.5 units along the three coordinate axes to obtain the local point cloud.

Taking the chair model in the ModelNet40 dataset as an example, its creation process of the global template point cloud, the initial partial point cloud, and the partial point cloud is shown in Figure 7, where the correspondence point estimation experiment was carried out under the condition that the proportion of correspondence points was 0.5. The green part is a global template point cloud that contained 1024 data points, the orange part is the initial partial point cloud that contained 512 data points sampled from the template point cloud at a sampling ratio of 0.5, and the blue part is the partial point cloud obtained after the initial partial point cloud was rotated and translated.

Taking the chair model as an example, when the proportion of correspondence points was 0.5, the accuracy of estimation of MPCR-Net, PRNet, and RPM-Net algorithms are shown in Figure 8, where green parts are the global template point clouds, and purple parts are the correspondence points estimated by three algorithms, respectively.

The distribution of these estimated correspondence points and the initial partial point cloud (Figure 8c) in the global template point cloud were compared. The closer two distributions are, the more correspondence points are correctly estimated, and the higher the estimation accuracy p of the correspondence point estimation.

Under the condition that the proportion of correspondence points was 0.05 to 0.95, estimation experiments were performed on all models in the test set. Then, the average estimation accuracy of the correspondence point of the three algorithms was calculated.

As shown in Figure 9, the average estimation accuracy of the corresponding points of the three algorithms increased as the proportion of correspondence points increased. However, as the proportion of correspondence points decreased, the accuracy gap between PRNet, RPM-Net, and MPCR-Net gradually widened. When the proportion of correspondence points was 0.95, the estimation accuracies of the correspondence points of the PRNet and RPM-Net were 26.0% and 8.0% lower than that of MPCR-Net, respectively, and when the proportion of correspondence points was 0.05, the estimation accuracies of correspondence points of PRNet and RPM-Net were 86.9% and 91.2% lower than that of MPCR-Net, respectively. This indicates that compared to the PRNet and RPM-Net algorithms, the MPCR-Net algorithm proposed in this paper can effectively improve the estimation accuracy of the correspondence point, especially when the proportion of correspondence points is low.

4.2.3. Point Cloud Registration

To explore the influence of the proportion of correspondence points on the accuracy of point cloud registration, we used the MPCR-Net, PRNet, and RPM-Net algorithms to perform point cloud registration experiments.

The point cloud registration results of three algorithms on the chair model in the ModelNet40 dataset when the proportion of correspondence points is 0.5 are shown in Figure 10. The green parts are the template point cloud, and the red parts are the positions of the partial point cloud after registration by the three algorithms. $R$ and $t$ represent the calculation error of the rotation angle and the translation distance of each algorithm, respectively; the smaller the values of $\Delta R$ and $\Delta t$ are, the higher is the accuracy of point cloud registration.

Under the condition that the proportion of correspondence points was 0.05 to 0.95, the registration experiment was performed on all models in the test set, and the average registration errors of the three algorithms were calculated. Figure 11a, b show the average calculation error of the rotation angle, and the average calculation error of the translation distance, respectively. The average calculation error of the rotation angle of the MPCR-Net is 23.8–72.8% smaller than that of the PRNet and 27.1–77.6% smaller than that of the RPM-Net. The average calculation error of the translation distance of the MPCR-Net is 27.0–90.9% smaller than that of the PRNet and 30.2–94.3% smaller than that of the RPM-Net. This indicates that the registration accuracy of the MPCR-Net is higher than that of the other two algorithms, especially when the proportion of correspondence points is low.

Figure 11 shows that average calculation errors of the rotation angle and the translation distance are negatively correlated with the proportion of correspondence points, indicating that the point cloud registration accuracy decreases as the proportion of correspondence points decreases. This is because the coordinate information of the correspondence points directly participates in the calculation of the rigid body transformation matrix in the point cloud registration. Since the average estimation accuracy of the correspondence points of PRNet and RPM-Net is significantly lower than that of MPCR-Net, the average registration accuracy is also lower than that of MPCR-Net.

When the proportion of correspondence points changes from 0.95 to 0.05, the average calculation errors of the rotation angle of the MPCR-Net increase by 13.6°, whereas the calculation errors of PRNet and RPM-Net increase by 29.7° and 31.6°, respectively. Simultaneously, the average calculation error of the translation distance also changes similarly. This indicates that, compared to the PRNet and RPM-Net algorithms, the MPCR-Net algorithm is more robust to changes in the number of correspondence points. When the correspondence point is low, the MPCR-Net algorithm can still maintain a high registration accuracy and effectively register two point clouds with a large difference in the amount of data.

4.2.4. Work Efficiency

As shown in Figure 12, under the condition that the proportion of correspondence points was 0.05 to 0.95, the average values of the total time for estimating the correspondence points and calculating the rigid body transformation matrix using the three algorithms for all models in the test set was recorded. Then, the values were used to measure the efficiency of the point cloud registration of the three algorithms.

Figure 12 shows that as the proportion of correspondence points increases, the average time of each algorithm rises; this is because the number of point cloud data to be processed increases. Furthermore, as the proportion of correspondence points increases, the time-consuming growth rate of the MPCR-Net is slightly lower than that of the PRNet and RPM-Net. This is because the MPCR-Net dynamically adjusts the iterations based on the rigid body transformation matrix difference calculated by two consecutive iterations. As the proportion of correspondence points increases, the iterations required to obtain the optimal rigid body transformation matrix gradually decrease, thereby reducing the time consumption.

4.3. Experiments with Actual Workpieces

We used the MPCR-Net to perform a point cloud registration experiment on actual workpieces and then generated the surface reconstruction models. Then, we evaluated the point cloud registration accuracy by detecting the deviations between surface reconstruction models and actual digital models. Finally, we compared with other point cloud registration algorithms, such as PR-Net and RPM-Net, to verify the effectiveness and advancement of MPCR-Net.

4.3.1. Data Sampling and Processing

Actual workpieces used in the reconstruction experiment were the egg-shaped pressure hulls [47], which were tailor-welded using multiple stainless steel plates with a thickness of 2 mm. All the egg-shaped pressure hulls were numbered a–i sequentially, as shown in Figure 13. The overall dimensions of the egg-shaped pressure hull ‘a’ are: long-axis L = 256 mm and short-axis B = 180 mm; the egg-shaped coefficient S = 0.69. The CAD model and the global template point cloud of the shell ‘a’ are shown in Figure 14.

As shown in Figure 15, the process of obtaining partial point clouds of the hull ‘a’ is as follows:

• Paint the surface of the hull; place the painted hull on the rotating table and ensure that the 3D scanner is aligned with the geometric center of the hull.
• Control the rotating table to rotate the hull to a certain angle.
• Use the 3D scanner to scan the hull and obtain its partial point cloud under the initial angle.
• Repeat steps b and c to obtain partial point clouds $S_2–S_7$ of the hull at certain angles. The partial point clouds obtained are shown in Figure 16.

Compared to the CAD model, the actual digital model includes machining errors. The generation process of the actual digital model of the hull ‘a’ is shown in Figure 17, and the details are as follows:

• Paste labels on the surface of the painted hull ‘a’.
• Using scan steps similar to the process of obtaining partial point clouds of the egg-shaped pressure hull, obtain partial point clouds $V_1–V_7$ of the hull at angles $A_1–A_7$, respectively.
• In the measurement software Optical RevEng 2.4, which is provided by the 3D scanner manufacturer, use the turntable method and the label method to register point clouds $V_1–V_7$, and use the ICP algorithm to fine-register the registration point clouds.
• Continuously adjust the fine registration point clouds manually according to the measurement results to make the registration point clouds closer to the hull.
• Perform surface reconstruction on the manually adjusted point clouds to obtain the actual digital model (Figure 17b) of hull ‘a’.

4.3.2. Analysis of Registration Accuracy of Multiple Partial Point Clouds

Taking the egg-shaped pressure hull ‘a’ as an example, use MPCR-Net to register all the partial point clouds to the global template point cloud and the ICP algorithm to optimize the registration results. Then generate the full registered point cloud. Figure 18 shows the full registration point cloud (Figure 18a) and its surface reconstruction model (Figure 18b) of hull ‘a’.

The above method is used to generate surface reconstruction models of all egg-shaped pressure hulls a–i, and the registration accuracies can be indicated by detecting the surface contour deviation between surface reconstruction models and actual digital models. The contour deviation cloud maps of hulls a–i are shown in Figure 19; the smaller the deviation, the higher the registration accuracy.

The maximum positive deviation, maximum negative deviation, and the area ratio exceeding the distance tolerance $\pm 0.5mm$ of the surface contour between the surface reconstruction model and the actual digital model are denoted as indicators A, B, and C, respectively.

MPCR-Net, PRNet, and RPM-Net algorithms were used to perform surface reconstruction experiments on all egg-shaped pressure hulls a–i, and indicators A, B, and C were used to evaluate the reconstruction accuracy of three algorithms. The results are presented in Figure 20.

As shown in Figure 20, the smaller the values corresponding to indicators A, B, and C are, the higher the registration accuracy is. Each bar represents the average registration accuracy of each algorithm under different indicators for hulls a–i, and each error bar indicates the distribution range of the indicator on hulls a–i.

For indicator A, the average maximum positive deviation of the MPCR-Net was 17.6% smaller than that of PRNet and 27.1% smaller than that of RPM-Net. For indicator B, the average maximum negative deviation of MPCR-Net is 18.2% smaller than that of PRNet and 28.0% smaller than that of RPM-Net. For indicator C, the area ratio exceeding the distance tolerance of MPCR-Net is 32.1% lower than that of PRNet and 42.4% lower than that of RPM-Net. In summary, the performance of MPCR-Net for the accuracy indicators A, B, and C is better than that of PRNet and RPM-Net, indicating that MPCR-Net can provide higher registration accuracy for actual workpieces.

5. Conclusions

We designed a multiple partial point cloud registration network based on deep learning, called the MPCR-Net. MPCR-Net uses the global template point cloud converted from the CAD model of the workpiece to guide the registration of partial point clouds. All partial point clouds are registered to the global template point cloud through TPCC-Net and TMPE-Net in MPCR-Net, forming a fully registered point cloud of a workpiece. This can effectively reduce errors in multiple partial point cloud reconstruction.

Experiment results demonstrated that MPCR-Net has the following advantages:

• Using a global-template-based multiple partial point cloud registration method can fully guarantee the overlap rate between each partial point cloud and its corresponding partial template point cloud, thereby reducing the registration error and improving the point cloud reconstruction accuracy.
• Searching for correspondence points between partial point clouds and the global template point cloud through TPCC-Net does not require separate training for specific local data of point clouds, thereby effectively reducing the correspondence point estimation error.
• The rigid body transformation matrix parameters in the registration are estimated through TMPE-Net, and estimation results are robust to changes in data points. It eliminates the shortcomings of other algorithms that cannot effectively register two point clouds with significant differences in the amount of data.