Path Planning of Autonomous 3-D Scanning and Reconstruction for Robotic Multi-Model Perception System

Abstract

Applying a three-dimensional (3-D) reconstruction from mapping-oriented offline modeling to intelligent agent-oriented environment understanding and real-world environment construction oriented to agent autonomous behavior has important research and application value. Using a scanner to scan objects is a common way to obtain a 3-D model. However, the existing scanning methods rely heavily on manual work, fail to meet efficiency requirements, and are not sufficiently compatible with scanning objects of different sizes. In this article, we propose a creative visual coverage path planning approach for the robotic multi-model perception system (RMMP) in a 3-D environment under photogrammetric constraints. To realize the 3-D scanning of real scenes automatically, we designed a new robotic multi-model perception system. To reduce the influence of image distortion and resolution loss in 3-D reconstruction, we set scanner-to-scene projective geometric constraints. To optimize the scanning efficiency, we proposed a novel path planning method under photogrammetric and kinematics constraints. Under the RMMP system, a constraints-satisfied coverage path could be generated, and the 3-D reconstruction from the images collected along the way was carried out. In this way, the autonomous planning of the pose of the end scanner in scanning tasks was effectively solved. Experimental results show that the RMMP-based 3-D visual coverage method can improve the efficiency and quality in 3-D reconstruction.

1. Introduction

1.1. Background and Significance

With the revolutionary development of manufacturing, computer vision, virtual reality, human–computer interaction, and other fields, the demand for 3-D models is increasing. They are widely used in industrial manufacturing [1,2], construction engineering [3], film and television entertainment [4], driverless [5], cultural relics restoration [6,7], medical and health care [8], etc. There is an increasing demand [9,10] for 3-D reconstruction for objects or scenes of different sizes, such as the reconstruction of desktop decorations at the scale of cm [11,12,13], reconstruction of car and airplane surfaces that are as large as meters [14,15,16], and reconstruction of outdoor natural scenes [17]. In addition, the requirements for the degree of automation and efficiency of the reconstruction process [18,19] are getting higher and higher. It is of great significance for industrial manufacturing and production life to realize the 3-D reconstruction of different-sized objects or scenes and 3-D scanning with a high automation and efficiency.

Currently, 3-D scanning solutions for different-sized objects have been proposed in succession. In order to scan small-sized objects, Hall-Holt O et al. [20] used a combined approach of a camera and projector to achieve scanning within an operating range of 10 cm, which guaranteed an image acquisition frequency of 60 Hz and a 100 μm accuracy. Percoco, G et al. [21] used the digital close-range photogrammetry (DCRP) method for scanning millimeter-scale objects. In order to scan larger-than-millimeter-scale objects, Li W et al. [22] designed a rotating binocular vision measurement system (RBVMS), which uses two cameras and two warp meters as a two-dimensional rotating platform. It measures 300 mm and 600 mm targets with an accuracy of approximately 0.014 mm and 0.027 mm. Straub J et al. [23] built a 3-D scan space with eight prismatic regions and used 50 cameras for data acquisition. Zeraatkar M et al. [24] used 100 cameras to build a cylindrical scanning space for a 3-D scanning system targeting person. Yin S et al. [25] and Du H et al. [26] designed a method for an industrial robotic arm to hold the scanner, and the method was combined with the use of a laser range finder for end positioning. In order to scan large-sized objects, Zhen W et al. [27,28] used a binocular plus LiDAR acquisition scheme for the 3-D scanning of large civil engineering designs, such as bridges and buildings.

The above scanners all use fixed schemes or handheld methods, which do not have autonomous motion exploration capabilities and are not suitable for large-range and long-time 3-D scanning tasks. With the development of robotics, an increasing amount of research has been conducted to increase the scanner motion capability in order to improve the efficiency and automation of the scanning process. Shi J et al. [29] designed multiple-vision systems mounted on an L-shaped robotic arm. The systems slide on a rail and use a laser rangefinder to acquire the distance between one end of the rail and robotic arm to facilitate 3-D reconstruction. Since the camera only acquires image data in an overhead manner, the reconstruction effect is rational for the top of the object but poor for the side of the object. Wang J et al. [30,31] proposed a measurement solution based on a mobile robot and a 3-D reconstruction method based on multi-view point cloud alignment. The mobile optical scanning robot system for large and complex components was also investigated for positioning using ground position coded markers. Although the method has a good error accuracy, it places new requirements on the ground coding and positioning process, which increases the task difficulty. Zhou Z et al. [32,33] proposed a better measurement, combining a global laser tracker, local scanning system, and industrial robot with a mobile platform. The measurement based on a priori knowledge establishes the correlation constraints between local and global calibrations, improving the measurement accuracy.

To improve the automation of 3-D scanning, it is necessary to plan the path for the robot, making it maximize the information gained about the objects or scenes. Path planning is widely used in robotic systems with mobility functions. For example, using distributed smart cameras in low-cost autonomous transport systems for adapting to future requirements of a smart factory [34]. Solutions toward path planning problems include sampling-based methods, frontier-based methods, or a combination of both. In sampling-based methods, ref. [35] presented a new RRT*-inspired online informative path planning algorithm by expanding a single tree of candidate trajectories and rewiring nodes to maintain the tree and refine intermediate paths. Furthermore, ref. [36] proposed a next-best-view planner that can perform full exploration and user-oriented exploration with an inspection of the regions of interest using a mobile manipulator robot. In frontier-based methods, ref. [37] proposed an exploration planning framework that is a reformulation of information gain as a differentiable function. This allows us to simultaneously optimize information gain with other differentiable quality measures. In the method of combination, ref. [38] achieved a fast and efficient exploration performance with tight integration between the octree-based occupancy mapping approach, frontier extraction, and motion planning.

To address the shortcomings of generic path planning methods in 3-D scanning tasks, a new multi-model perception robotic system was designed in the manuscript. Furthermore, a novel path planning method based on the system is proposed in this paper. The robotic system includes four submodules: a mechanical module, driving module, control module, and self-made sensing module. Therefore, it can scan objects of different sizes automatically. In addition, the path planning method using photogrammetric and kinematic constraints improves the level of 3-D scanning automation. The method proposed in this paper is applicable to 3-D scanning and reconstruction tasks and can satisfy 3-D scanning and reconstruction for outdoor natural scenes, indoor environments, or objects; for example, the 3-D scanning and reconstruction of workpieces in industrial manufacturing. The research content of this paper is shown in Figure 1.

1.2. Aims and Contributions

This paper mainly focuses on the path planning of autonomous 3-D scanning and reconstruction for the robotic multi-model perception system. The main contributions of this work are as follows:

(1)

This paper proposes a novel path planning approach for 3-D visual coverage scanning. By defining a set of practical photogrammetric constraints, the quality of the images collected along the way of the robotic system is effectively improved. Different from the existing path planning algorithms—either frontier-based methods, sampling-based methods, or a combination of both—we planned the path based on the shape of the objects. The proposed strategy can obtain dynamic feasible paths based on the true shapes of the scanning objects;

(2)

This paper defines two new photogrammetric constraints for image acquisition based on the shape of scenes or objects. In order to obtain an accurate model of the ROI, the images obtained from the cameras integrated into the scanner should satisfy the equidistant and frontal constraints. In this way, the problems of image deformation and resolution loss caused by the shape change of the scenes or objects to be reconstructed can be resolved;

(3)

This paper proposes a novel design of a scanner and robotic system for coverage scanning tasks. Firstly, we designed a scanner equipped with LiDAR, Realsense, and multiple cameras. Secondly, we used a mobile platform and robotic arm to ensure the robotic system’s ability to move in 3-D space. Finally, the designed scanner was mounted at the end of the robotic arm to form the scanning robotic system, which can significantly improve both the quality of photography and the efficiency of scanning.

2. Problem Formulation

2.1. Assumptions

Consider a scenario where a 3-D ROI needs to be fully covered by images obtained from cameras mounted on the scanner. The moving of the scanner is subjected to photogrammetric and kinematics constraints based on the preliminary work of our subject group [39]. We make the following assumptions:

(1)

The ground in the three-dimensional space where the robotic system is located is required to be flat without potholes or bulges, so as to achieve smooth robotic system movement;

(2)

The LiDAR, Realsense, and multiple cameras on the scanner should have been pre-calibrated. In addition, the parameters of the cameras and images, such as the camera field of view (FOV) and image resolution, should have been known in advance. In addition, the scanner and the end of the robotic arm should have been pre-calibrated.

(3)

Individual images are used or multiple images are seen as a composite individual image for further theoretical analysis, for the purpose of path planning.

2.2. Definition of Photogrammetric Constraints and Problem Statement

We aimed to capture images of $\mathcal{D}$ with high quality, and to further obtain an accurate 3-D model. $\mathcal{D}$ is the side of scanned objects. Therefore, some constraints should be considered when the shutter of the camera is triggered.

Let $\mathcal{X}$ represent the discrete scanner waypoints: $\mathcal{X}={\left[xyz\gamma \beta \alpha \right]}^{\mathrm{T}}\in \mathbf{X}\subset {\mathbb{R}}^6$. $g_1\left(\mathbf{X}\right)$ is a set of constraint functions that satisfy the following photogrammetric conditions at a specific waypoint, as shown in Figure 2.

(1)

Equidistant: In order to maintain a consistent image resolution, the position of each waypoint $\mathcal{X}$ should keep a constant offset distance h from the fitted plane with a bounded position error ${ϵ}_1$;

(2)

Frontal: For each waypoint $\mathcal{X}$, the orientation of the camera $\left(\right[\gamma \beta \alpha \left]\right)$ is perpendicular to the fitted plane to capture an orthophotograph with a bounded angle error ${ϵ}_2$;

(3)

Overlap: In the 3-D reconstruction application, the images should be acquired with a horizontal overlapping rate ${\sigma }_0$ and vertical overlapping rate ${\sigma }_1$.

In this paper, the equidistant h was set by evaluating the expected resolution when given the distance and the intrinsic parameters of the camera, which is consistent with [40,41]. The normal vector of the fitted plane was used to set the frontal constraint. According to [42], the overlapping rates ${\sigma }_0$ and ${\sigma }_1$ were determined.

To cover the 3-D space $\mathcal{D}$, a robotic system was involved, which was equipped with a scanner for multi-model perception. As shown in Figure 2, the robotic system follows a path that contains all of the waypoints generated under photogrammetric and kinematic constraints, and ultimately achieves the coverage scanning task.

For the robot path planning, we parameterized the obtained path as $\mathbf{X}=[{\mathcal{X}}_1{\mathcal{X}}_2\cdots {\mathcal{X}}_n]$, where $\mathbf{X}$ was set to fully cover $\mathcal{D}$. The objective of the mission was to minimize the path length $\mathcal{L}\left(\mathbf{X}\right)$ under the constraints $g\left(\mathbf{X}\right)$. The exact formulas of the objectives and constraints will be given in the following sections.

2.3. Next View Planner

The next view planner is one of the robotic system functions, and determines the ideal scanning position and attitude of the 3-D scanner at the next moment according to the shape change of the scenes or objects. The planner needs to input LiDAR or Realsense point cloud data. LiDAR covers medium and long-range 3-D scanning, and Realsense covers 0.2 m∼10 m 3-D scene scanning. Since the laser point cloud acquires 360-degree point cloud data, the next view planner only needs a certain angle in front of the scanner, so it is considered to intercept the laser point cloud with a slightly larger angle than the image resolution. After data preprocessing, plane fitting, photogrammetric constraints, and other processing, the ideal scanning pose of the 3-D scanner in the robotic system at the next moment can be calculated. The whole calculation and derivation process of the next view planner is as follows:

(1) Rotation: As shown in Figure 3, SRC_Data represents the obtained LiDAR point cloud data matrix. Define the rotation matrix as K obtained by Realsense imu data. The LiDAR coordinate system changes from L to $L^{\prime }$ by $L^{\prime }=\mathbf{K}·L$, and ROT_Data is obtained by Rot_Data = K· SRC_Data;

(2) Cutting: Select Rot_Cut_Data from Rot_Data, covering positive and negative 45 degrees in front of the scanner;

(3) Denoising: Filter the Rot_Cut_Data by comparing it with the mean value of the spatial points’ coordinates, named the denoised data as Rot_Cut_Mean_Data;

(4) Fitting line: Project the Rot_Cut_Mean_Data under the $L^{\prime }$ coordinate system onto x-0-y plane, fit a straight line by the least square method, and name the line as ${\mathcal{F}}_i$. The straight line is used to solve the ideal shooting plane of the next view. Process (1) to process (4) are shown in Figure 3;

(5) Frontal: Establish a perpendicular line (frontal constraint) from the coordinate origin to the fitted line. Generate a vector from the perpendicular foot named ${\mathcal{A}}_i$ to the origin, which is the normal vector of both the ideal shooting direction and the ideal shooting plane in the next view;

(6) Fitting plane: Fit a plane by the plane normal vector in the process (5) and the straight line on the plane in the process (4). Use the plane and the plane’s normal vector to calculate the ideal shooting pose for the next view;

(7) Equidistant: Calculate a point by x-y coordinate mean of Rot_Cut_Mean_Data. Use the point and plane’s normal vector to find the ideal shooting pose $\mathcal{H}\left({\mathcal{X}}_i\right)$ at a distance h from the plane. The $\mathcal{H}\left({\mathcal{X}}_i\right)$ is defined by Formula (1);

$$\begin{array}{cc}\hfill \mathcal{H}\left({\mathcal{X}}_i\right)& =\left[\begin{array}{cc}\mathbf{R}& \mathbf{C}\\ \mathbf{0}& 1\end{array}\right]\hfill \\ \hfill \mathrm{let}:\mathbf{R}& =R_Z\left(\alpha \right)R_Y\left(\beta \right)R_X\left(\gamma \right)\hfill \\ \hfill & =\left[\begin{array}{ccc}c\alpha & -s\alpha & 0\\ s\alpha & c\alpha & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}c\beta & 0& s\beta \\ 0& 1& 0\\ -s\beta & 0& c\beta \end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& c\gamma & -s\gamma \\ 0& s\gamma & c\gamma \end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}c\alpha c\beta & c\alpha s\beta s\gamma -s\alpha c\gamma & c\alpha s\beta c\gamma +s\alpha s\gamma \\ s\alpha c\beta & s\alpha s\beta s\gamma +c\alpha c\gamma & s\alpha s\beta c\gamma -c\alpha s\gamma \\ -s\beta & c\beta s\gamma & c\beta c\gamma \end{array}\right]\hfill \\ \hfill \mathbf{C}& ={\left[xyz\right]}^{\mathrm{T}}\subset {\mathbb{R}}^3\hfill \end{array}$$

(1)

(8) Recovering the coordinate system: Use the inverse matrix K′ of K to recover coordinate system form $L^{\prime }$ to L. Meanwhile, the ${\mathcal{H}}^{\prime }\left({\mathcal{X}}_i\right)$ is obtained by Formula (2). Process (5) to process (8) are shown in Figure 4;

$${\mathcal{H}}^{\prime }\left({\mathcal{X}}_i\right)={\mathbf{K}}^{\prime }·\mathcal{H}\left({\mathcal{X}}_i\right)$$

(2)

(9) Calculating n: Define n as the number of shots. For every scan horizon line, the number of shots is $n_l$. For every scan vertical line, the number of shots is $n_v$. n is the sum of $\lceil n_l\rceil$ plus $\lceil n_v\rceil$ as Formula (3), where $L\times W$ represents the size of area $\mathcal{D}$, and $L_c\times W_c$ represents the camera resolution.

$$\begin{array}{cc}\hfill & n_l=\frac{L-{\sigma }_1{L}_c}{L_c-{\sigma }_1{L}_c}\hfill \\ \hfill & n_v=\frac{W-{\sigma }_0{W}_c}{W_c-{\sigma }_0{W}_c}\hfill \\ \hfill & n=\lceil n_l\rceil \times \lceil n_v\rceil \hfill \end{array}$$

(3)

Define $\mathbf{NVP}$ as a constraint function that represents the above process (1) to (9). Define ${\mathbf{P}}_1$, which is the set of all points ${\mathcal{X}}_i$ that satisfies the $\mathbf{NVP}$.

2.4. Inverse Kinematics Constraints

Describe the mobile platform position and attitude with three parameters ${\mathbf{C}}_m=[x_m,y_m,{\theta }_0]$, and the flexible robot arm has six joint degrees of freedom described by ${\mathbf{C}}_r=[{\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5,{\theta }_6]$. Therefore, the control towards the scanner pose located at the end of the mobile manipulator has redundant parameters. Name $P_i$ as the scanner ideal pose. The relationship between $P_i$ and ${\mathcal{H}}^{\prime }\left({\mathcal{X}}_i\right)$ is:

$${\mathcal{P}}_i={\mathcal{H}}^{\prime }\left({\mathcal{X}}_i\right)·M=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& p_{x_i}\\ r_{21}& r_{22}& r_{23}& p_{y_i}\\ r_{31}& r_{32}& r_{33}& p_{z_i}\\ 0& 0& 0& 1\end{array}\right]$$

(4)

where M represents the inverse matrix of the transformation matrix from the end of the manipulator to the laser radar. It is pre-calibrated.

Define ${\mathcal{D}}_i$ as the euclidean distance between ${\mathcal{P}}_{d_i}=\left[p_{x_i}{p}_{y_i}{p}_{z_i}\right]$ and ${\mathcal{P}}_{d_{i-1}}$:

$${\mathcal{D}}_i=\parallel {\mathcal{P}}_{d_i}-{\mathcal{P}}_{d_{i-1}}\parallel$$

(5)

Describe the space coordinates $(x_m,y_m)$ and direction angle ${\theta }_0$ of the mobile platform and six joint angles of the manipulator $({\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5,{\theta }_6)$ as the system state quantity $({\mathbf{C}}_m,{\mathbf{C}}_r)$. Since the control to the end of the mobile manipulator has redundant parameters, the solution to the joint angle of the manipulator and the pose of the mobile platform requires setting reasonable constraints. Therefore, the state quantity of the solved system can reach the specified pose by the ends scanner. A reasonable constraint not only needs to consider the mechanical characteristics, required to ensure the flexibility of the ends scanner and the stability of the robot, but also the difficulty of the solving process. The relationship between the system state quantity and the desired pose can be obtained from Formula (6):

$$\begin{array}{c}\hfill {}_E^{W}T({\mathbf{C}}_m,{\mathbf{C}}_r)={\mathcal{P}}_i\end{array}$$

(6)

where ${}_E^{W}T$ represents the ends scanner pose matrix in the world coordinate system.

The pose of the scanner at the end of the manipulator has six degrees of freedom (DOFs), which are three rotational DOFs and three translational DOFs, respectively. The three rotational DOFs are controlled by the rotational parameters $({\theta }_1,{\theta }_4,{\theta }_5,{\theta }_6)$ of the manipulator and directional parameter ${\theta }_0$ of the mobile platform. The three translational DOFs are controlled by the rotational parameters $({\theta }_2,{\theta }_3)$ of the manipulator and position parameters $(x_m,y_m)$ of the mobile platform.

From the perspective of mechanical characteristics and ease-solved parameters. Since the manipulator ${\theta }_1$ and mobile platform ${\theta }_0$ both rotate around the same Z-axis, the two motions can be equivalent. Therefore, constraint 1—${\theta }_1=0^{\circ }$—was designed and the shortage was made up by ${\theta }_0$. To ensure the stability of the robot, constraint 2—${\theta }_3=2\times ({\theta }_2+{90}^{\circ })$—was designed to keep the robot’s center of gravity within a certain range. In addition, considering the situation that if ${\theta }_4,{\theta }_5$ are constrained, the dexterity space of the robot will be reduced, ${\theta }_4,{\theta }_5$ should be released. However, considering that even if the flexibility of the sixth axis is guaranteed, its effect on the scanner at the end is not significant, a constraint 3 of ${\theta }_6=0^{\circ }$ was set.

Use Equation (8) to figure out the six joint angles of the manipulator, and then use Equation (9) to find the position coordinate parameter $(x_m,y_m)$ and the direction angle parameter ${\theta }_0$ of the mobile platform.

$$\begin{array}{c}\hfill {}_E^{B}T\left({\mathbf{C}}_r\right){=}_B^W{T}^{-1}·{\mathcal{P}}_i=\left[\begin{array}{cccc}{r}_{11}^{{}^{\prime }}& r_{12}^{{}^{\prime }}& r_{13}^{{}^{\prime }}& p_{x_i}^{{}^{\prime }}\\ r_{21}^{{}^{\prime }}& r_{22}^{{}^{\prime }}& r_{23}^{{}^{\prime }}& p_{y_i}^{{}^{\prime }}\\ r_{31}^{{}^{\prime }}& r_{32}^{{}^{\prime }}& r_{33}^{{}^{\prime }}& p_{z_i}^{{}^{\prime }}\\ 0& 0& 0& 1\end{array}\right]\end{array}$$

(7)

$$\begin{array}{c}\hfill {}_B^{W}T\left({\mathbf{C}}_m\right)={\mathcal{P}}_i{·}_E^B{T}^{-1}=\left[\begin{array}{cccc}{r}_{11}^{{}^{″}}& r_{12}^{{}^{″}}& r_{13}^{{}^{″}}& p_{x_i}^{{}^{″}}\\ r_{21}^{{}^{″}}& r_{22}^{{}^{″}}& r_{23}^{{}^{″}}& p_{y_i}^{{}^{″}}\\ r_{31}^{{}^{″}}& r_{32}^{{}^{″}}& r_{33}^{{}^{″}}& p_{z_i}^{{}^{″}}\\ 0& 0& 0& 1\end{array}\right]\end{array}$$

(8)

where ${}_E^{B}T$ denotes the ends scanner pose matrix in the base coordinate system, and ${}_B^{W}T$ denotes the base pose matrix in the world coordinate system, which is a known matrix by navigation and positioning. Therefore, the homogeneous matrix on the right side of Formula (7) is a known parameter. According to constraint 1, constraint 3, and Formula (7), we can obtain:

$$\left\{\begin{array}{c}{\theta }_2=asin\left(\frac{(t-d_5\times r^{{}^{\prime }}32-p_{z_i}^{{}^{\prime }})}{a_2+a_3}\right)\hfill \\ {\theta }_3=2\times ({\theta }_2+{90}^{\circ })\hfill \\ {\theta }_4={\theta }_2-acos\left(r^{{}^{\prime }}32\right)\hfill \\ {\theta }_5=atan2(r^{{}^{\prime }}33,-r^{{}^{\prime }}31)\hfill \end{array}\right.$$

(9)

According to the values of ${\theta }_1=0^{\circ }$, ${\theta }_6=0^{\circ }$ and Formula (9), we can figure out $(x_m,y_m,{\theta }_0)$ by Formula (8):

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{x}_m=p_{x_i}^{{}^{″}}\hfill \\ y_m=p_{y_i}^{{}^{″}}\hfill \\ {\theta }_0=atan2(r^{{}^{\prime }}21,-r^{{}^{\prime }}11)\hfill \end{array}\right.\end{array}$$

(10)

The above solutions for the joint angles of the manipulator and the mobile platform pose do not include the case ${\theta }_5=0^{\circ }$. When it comes to the special case ${\theta }_5=0^{\circ }$, taking into account the mechanical characteristics, an additional constraint ${\theta }_4={\theta }_2+{180}^{\circ }$ was added to tackle this problem. After solving the parameter $(x_m,y_m,{\theta }_0)$, the differential two-wheel kinematics model can be used to calculate the two-wheel angular velocity ${\omega }_l$ and ${\omega }_r$.

Define $\mathbf{IKC}$ as a function that represents the constraint mapping relation from $P_i$ to $(x_m,y_m,{\theta }_0,{\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5,{\theta }_6)$. Define a set ${\mathbf{P}}_2$, which means that the set of all points ${\mathcal{X}}_i$ satisfied the $\mathbf{IKC}$.

2.5. Path Generation

The above two sections, Next View Planner and Inverse Kinematics Constraints, describe the constraints. For the robot path planning, considering a real-world scene and a series of waypoints set $\mathbf{X}$ employed to fully cover $\mathcal{D}$, the goal of the coverage path planning can be shown as follows:

$$\begin{array}{cc}\hfill & \min \mathcal{L}\left(\mathbf{X}\right)=\sum _{i=1}^k{D}_i\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\parallel {\mathcal{X}}_i-{\mathcal{A}}_i\parallel =\mathbf{h}\hfill \\ \hfill & {\mathcal{X}}_i{\mathcal{F}}_i=\mathbf{0}\hfill \\ \hfill & {\mathcal{X}}_i\in {\mathbf{P}}_1\cup {\mathbf{P}}_2\hfill \end{array}$$

(11)

where X is the path set $[{\mathcal{X}}_1{\mathcal{X}}_2\cdots {\mathcal{X}}_n]$, $D_i$ is the euclidean distance between ${\mathcal{X}}_{n-1}$ and ${\mathcal{X}}_n$, $\mathbf{h}$ is the equidistant constraint, ${\mathcal{X}}_i{\mathcal{F}}_i=\mathbf{0}$ is the frontal constraint, ${\mathbf{P}}_1$ is the set of all points ${\mathcal{X}}_i$ that satisfy the $\mathbf{NVP}$, and ${\mathbf{P}}_2$ is the set of all points ${\mathcal{X}}_i$ that satisfy the $\mathbf{IKC}$.

2.6. Algorithm of Path Planning under Photogrammetric and Kinematics Constraints (PP-PKC)

The path planning algorithm under photogrammetric and kinematics constraints (PP-PKC) is shown in Algorithm 1. M is the pre-calibrated data between LiDAR and the end of the manipulator. $SRC\_Dat{a}_i$ is the original data of LiDAR. $Calc\_n$ represents Formula (5). $Eucl\_d$ means the euclidean distance between ${\mathcal{X}}_{j-1}$ and ${\mathcal{X}}_j$, as shown in Formula (6). In addition, $\mathbf{NVP}$ is defined in Section 2.3, and $\mathbf{IKC}$ is defined in Section 2.4.

Algorithm 1: Path Planning Algorithm under Photogrammetric and Kinematics Constraints (PP-PKC)

3. Robotic Multi-Model Perception System (RMMP)

3.1. Robotic System Overall Structure

The robotic multi-model perception system (RMMP) proposed in this paper is made up of four parts: a mechanical module, driving module, control module, and sensing module. The mechanical module includes a mobile platform, a manipulator, and a 3-D scanner; the drive module consists of several DC servo motors; the control module includes three parts: navigation and positioning, a next frame planner, and motion planning; and the sensing module consists of sensors belonging to a 3-D scanner. All of these modules are shown in the upper left of Figure 5.

3.2. Mechanical Module and Self-Designed Scanner

The mechanical module in RMMP contains the mobile platform, manipulator, and 3-D scanner. The mobile platform is responsible for the plane movement of the robot system in 3-D space, which has the advantage of an unlimited movement range. The manipulator is responsible for the 3-D movement of the end scanner in space, which can ensure that the scanner makes a flexible movement in 3-D space. Moreover, the 3-D scanner is responsible for the data acquisition of the object to be scanned.

Among them, the 3-D scanner is a novel self-designed platform, as shown in the upper right of Figure 5. The designed scanner has an axisymmetric structure with a projector in the center that can be specified to project structured light. The projector has two binocular vision systems, named as BVS1 and BVS2, respectively. The BVS1 is made up of Camera1 and Camera4, and the BVS2 is made up of Camera2 and Camera3. LiDAR is on the top of projector, and Realsense is beneath the projector.

For small-sized objects, we used the binocular systems made up of Camera2 and Camera3; in addition, the projector and Realsense were chosen to assist in data perception. For medium and large objects, the binocular systems made up of Camera1 and Camera4 were selected; in additoin, LiDAR was used to collect the object point cloud data. Throughout the 3-D scanning process, the scanner attitude information can be collected using the inertial measurement unit (IMU) integrated in Realsense.

3.3. Control Module

The control module of RMMP is the key part to accomplish the autonomous scanning task, as shown in Figure 6. The control module designed in this paper contained navigation and positioning, the next view planner, and motion planning. Navigation and positioning were used to calculate the robot pose, obstacle avoidance, and other functions. The next view planner was used to calculate the ideal shooting pose for the next frame, and motion planning was used to calculate and control the robotic system movement.

4. Experiment

In order to verify the effectiveness of the RMMP system and the method described above, we took the real natural scene as a 3-D scanning experimental object and carried out the 3-D scanning experiment.

4.1. Experimental Environment

The hardware composition of the RMMP system is shown in Figure 5. The mobile platform takes differential two wheel drive mode, the mechanical manipulator adopts 6-DOF serial flexible mechanical mode, and the end of the mechanical manipulator controls the self-developed 3-D scanner device. Moreover, the system was equipped with a minicomputer to control the system. Therefore, the robot system can improve the flexibility and automation level of 3-D scanning.

4.2. Task Workflow

The workflow of the 3-D scanning and reconstruction task is shown in Figure 7, and is demonstrated as follows:

(1) First, preset the path of the mobile platform according to the shape of the object to be scanned. For example, for a car, a circular path or elliptical path needs to be preset to control the movement of the mobile platform; for a square column, a square path is needed; for a small object, directly presetting a circular path is correct. During the movement of the robot in the preset path, the LiDAR and Realsense located on the scanner begin to work. After the preset path is completed, we can derive L, W, and n;

(2) Based on the data obtained in the previous step, path planning is started to acquire image data. In this process, not only do the laser radar and depth camera start to work, but BVS1, BVS2, or both also begin to work. After the path is completed, at a distance of h, a set of pictures of the scanned object can be obtained. Compared with using images obtained by the traditional method of path planning, a better 3-D reconstruction can be realized by this set of images;

(3) Decrease the distance h and scan the object again. A better 3-D reconstruction can be obtained by using this set of images and images formed by process (2);

(4) Judge the reconstruction result of the object. Paper [43] gives an evaluation method. If the reconstruction result is reasonable, we can move to the next process. However, if the result is unsatisfactory, repeat process (3) until the reconstruction effect of the object reaches the requirements;

(5) Return the robot to the starting point and carry out the post-processing of the 3-D model. After the post-processing, if holes and other problems exist in the model, repeat process (3), (4), and (5) until the 3-D model meets the requirements;

(6) Finish the experiment.

4.3. Multi-Model Data Perception

In this paper, a real natural scene was scanned and reconstructed in three dimensions. Specifically, for the flower bed and the pine tree inside, a suborbicular motion path was preset. During the motion of the RMMP system, the LiDAR and Realsense located on the scanner started to work. After completing the preset path, L, W, and n were obtained. Considering the actual situation, we chose 2.5 m and 2 m as the equidistant constraint h, combined with kinematic constraints, to generate the planning results of Path 1 and Path 2, respectively. The path planning effect of the RMMP system in the real scene is shown in Figure 8.

Table 1. Experimental data of different paths.

	Num of ${\mathcal{X}}_{\mathit{i}}$	Constrainst h (m)	Path Length (m)	$\mathit{L}\times \mathit{W}$ (m)	Triangular Facet
Preset path	50	—	25.40	—	—
Path1	47	2.5	21.73	$6.49\times 2.88$	1,703,424
Path2	44	2	18.47	$6.30\times 2.78$	2,982,178

shows the relevant experimental data information of different paths.

Table 1 shows the relevant experimental data information of different paths. From Table 1, it can be seen that the effect of path planning is significantly improved when photogrammetric constraints are added (both of the path lengths under constraints are smaller than the preset path length), and the effect of path planning is further improved with the reduction in equidistant constraint h. In addition, the effect of path planning can be seen intuitively from Figure 8. Furthermore, from the triangular facet data in Table 1, it can be seen that the effect of reconstruction is further improved with the reduction in equidistant constraint h.

4.4. Three-Dimensional Real Scene Reconstruction

After using RMMP system for data perception, the collected data were used in 3-D reconstruction. The 3-D reconstruction result under different h is shown in Figure 9. As the left side of Figure 9 shows, the pictures taken in the equidistant constraint h, the reconstructed model has an uneven surface and bulging edge; when distance h is reduced, the reconstructed effect is better handled for both the plane and the edge. The right side of Figure 9 shows the reconstructed results, demonstrating a better performance.

5. Conclusions

This paper introduced a RMMP system, focusing on the path-planning-based next view planner and inverse kinematics constraints. Under photogrammetric constraints, the proposed next view planner can figure out the problems of image deformation and resolution loss caused by the shape change of the objects to be reconstructed. In addition, the proposed inverse kinematics constraints make the robot able to reach an ideal pose. Compared with the traditional scanning method, the proposed robotic system improves the 3-D scanning automation and efficiency. In the RMMP system, the mobile platform was used for plane movement in two-dimensional space, and the flexible manipulator was used to ensure the motion ability of the scanner in 3-D space. In addition, the designed scanner can collect multi-model information of different-size objects. Preliminary experiments show that the correctness of the proposed inverse kinematics and the rationality of the proposed next view planner provide a guarantee for the robot to complete automatic data acquisition.

It must be pointed out that the proposed RMMP system has the characteristics of a high efficiency and universality, as well as being very suitable for the 3-D data acquisition of large scenes or objects. The future work will focus on further improving the level of the automation of the robotic system and achieving efficient 3-D reconstruction.