Real-Time 3D Map Building in a Mobile Robot System with Low-Bandwidth Communication

Abstract

This paper presents a new 3D map building technique using a combination of 2D SLAM and 3D objects that can be implemented on relatively low-cost hardware in real-time. Recently, 3D visualization of the real world became increasingly important. In robotics, it is not only required for intelligent control, but also necessary for operators to provide intuitive visualization. SLAM is generally applied for this purpose, as it is considered a basic ability for truly autonomous robots. However, due to the increase in the amount of data, real-time processing is becoming a challenge. Therefore, in order to address this problem, we combine 2D data and 3D objects to create a new 3D map. The combination is simple yet robust based on rotation, translation, and clustering techniques. The proposed method was applied to a mobile robot system for indoor observation. The results show that real-time performance can be achieved by the system. Furthermore, we also combine high and low-bandwidth networks to deal with network problems that usually occur in wireless communication. Thus, robust wireless communication can be established, as it ensures that the missions can be continued even if the system loses the main network.

1. Introduction

Representing the real world in 3D visualization became increasingly important. In robotics, 3D maps unlock smarter capabilities for the robots [1]; for example, environmental exploration and localization [2], search and rescue [3], motion planning and automation [4], monitoring [5], and some others. Utilization of 3D maps remains challenging, especially for real-time implementation using mobile devices. Due to limited computing and memory resources, the amount of data brings a significant burden to the onboard processing, which can slow down overall performance [6,7]. On the other hand, missions and data collection have to be continued. Therefore, data compression is required to handle this issue.

In ground exploration missions, mobile robots are one of the most widely used technologies due to their mobility and adaptation to environments. Wheeled [8,9] or crawler robots [10,11] all became essential parts for both domestic and industrial applications. In domestic applications, they are more popular as service robots, for example, cleaning kitchens [12], cleaning floors such as the Roomba, and indoor observation [13]. Meanwhile, for industrial applications, many of them are intended for agriculture [14,15], hazardous tasks, saving human lives, and disaster response [16].

In order to work intelligently, maps are required in robot operations, even more so for autonomous control [17,18]. Simultaneous localization and mapping (SLAM) is generally applied for this purpose. It builds a map of the environment, and at the same time, locates the robot position on the map [19]. In addition, SLAM is also considered a basic ability and is highly encouraged for explorer robots [20,21]. Apart from the reasons above, SLAM can be used to provide an intuitive visualization (i.e., maps) of what the environment looks like [22]. It helps humans better understand the current situation. These are the reasons why the development and applications of SLAM were a major research topic for at least the last decade. SLAM is not only used by the robots, but is also useful for the humans.

In this paper, we focus on how to achieve real-time implementation of 3D map building. We believe that nowadays the development of SLAM and its related matters should be applicable and real-time oriented. There were several techniques related to this concern. In [23], a combination of perpendicular 2D data from two 2D laser scanners is proposed. The observations are accomplished in indoor regions with two mobile robots in parallel. Then, translational and rotational values are applied to the perpendicular data to construct a 3D point cloud map. However, real-time computation cannot be achieved since the combination is achieved offline, which requires around fifteen minutes; similarly as in [24,25], where offline computation is performed. Next, the fusion of a 2D light detection and ranging (LiDAR) and a depth camera [26] or a stereo camera [27] was developed. The cameras are used to improve the accuracy of 2D SLAM. Although additional information was combined, only a 2.2% to 3.6% improvement can be achieved. Furthermore, it also results in increased computational requirements.

Real-time implementation of large amounts of data is challenging, especially for processing requirements. As mentioned previously, one of the solutions is to compress the amount of data. The other solution is, of course, increasing the processing capability. In [2,22], real-time computation of 3D map building can be achieved using mobile robots. They use an Intel Core i7-4790K CPU 32 GB RAM and a Core i7-3555LE CPU 8 GB RAM, respectively. The method requires 1.56 s out of 3 s to update the map with 1.33× of data downsampling [22]. A mobile robot system in [28] performed 3D mapping with high quality and repeatability. It employs a Velodyne Puck LITE 3D laser scanner and an inertial measurement unit (IMU) sensor to run an online SLAM algorithm. However, in the implementation, the mobile robot was steered remotely using a wireless joystick, and the amount of data will be nearly impossible to implement on low-cost hardware to perform real-time observation. Modern SLAM algorithms, such as ORB-SLAMs [29,30,31], may also run in real time on embedded systems [32,33]. Nevertheless, they rely on multithreading, which is also limited in this system.

In real-world situations, degraded communication may occur during missions [2], especially in restricted areas that require kilometers of operating distance [10]. Therefore, to handle this issue, we also combine high and low-bandwidth networks in the system, i.e., Wi-Fi and long-range (LoRa). Wi-Fi, due to its high bandwidth, has a short communication range and is thus easily affected by environments; for example, buildings, trees, tunnels, underground, sea water, and some others. LoRa, on the other hand, has a much longer communication range and is more robust to the environments [34]. Furthermore, LoRa can maintain its communication for several kilometers even in urban areas [35], and its network can be scaled to increase the number of nodes [36]. Similar to [37], which combines Wi-Fi and a radio-frequency (RF) module (200 kbps) for sharing data between two mobile robots and constructing a 3D map, network limitations force the system to compress the data for real-time performance. The difference is that we use a lower communication bandwidth of about 6.01 kbps–22 kbps, which requires further data compression. Furthermore, they still require the high-bandwidth network to construct a full 3D map.

There are two restrictions in the proposed system, i.e., limited onboard computing resources and low-bandwidth communication. Based on the state-of-the-art above, a new technique is required in order to perform real-time 3D map building. Therefore, we propose a new real-time 3D map building technique using the combination of 2D SLAM and 3D objects. The technique is applied and focused on the mobile robot system with Wi-Fi and LoRa as the communication networks, as shown in Figure 1. Wi-Fi is used as the main communication network, with LoRa serving as a backup. Multi-bandwidth networks proved their impact on signal interference, shadowing, and wireless communication attenuation [38,39]. Hence, robust wireless communication can be established.

The contributions of this research can be summarized as follows: (1) The system utilizes low-cost devices, i.e., a Raspberry Pi controller and LoRa devices on the mobile robot, and an entry-level PC controller with an Intel Core i5-10210U CPU 8 GB RAM for controlling and monitoring the mobile robot. (2) We combine high and low-bandwidth networks to establish a robust wireless communication system. All observation data need to be sent wirelessly, creating a new data size restriction, and hence data compression is required. Lastly, (3) lightweight and efficient algorithms must be developed. All developed algorithms have to be simple, can be installed on low-cost hardware, and have real-time capabilities. These introduce the main target of the system, which performs the proposed method in real-time at a refresh rate of at least 1 Hz for both communication bandwidths. A real-time system is essential for operators as it gives direct results about environmental conditions. 3D maps will also increase intuitiveness to better understand the environment. Finally, a fully wireless communication system enables a barrier between the operators and observation fields, which increases safety for some crucial missions, such as search and rescue and disaster response.

The rest of this paper is organized as follows. In Section 2, the system configuration is described. Section 3 explains the proposed approach in detail. Section 4 shows the experimental results with its discussion. Last but not least, Section 5 is the conclusion and future work.

2. System Configuration

2.1. Hardware Design

In this system, there is a PC controller, a LoRa transceiver, and a mobile robot, as shown in Figure 1. The PC controller is used by an operator to control and monitor the mobile robot. It is also connected to the LoRa transceiver to communicate with the mobile robot through the low-bandwidth communication using LoRa devices. Figure 2 shows several devices installed on the mobile robot. We use a 2D LiDAR and a LiDAR camera as the main sensors to observe the environment. By using both sensors, 2D and 3D data can be provided from the mobile robot.

The hardware specifications are summarized in

Table 1. System hardware specifications.

PC Controller	Core i5-10210U CPU 8 GB RAM; IEEE 802.11ac (353 kbps–9 Mbps)
LoRa Transceiver	ES920LR 920.6-928.0 MHz ×2 (6.01 kbps–22 kbps); ES920ANT ×2 Arduino Mega 2560
Mobile Robot	Dimension: 350 mm × 200 mm × 200 mm Rear-drive brushed DC motor Servo motor with Ackermann steering Li-Fe battery 6.6 V/1450 mAh ×3 H-bridge MOSFET motor driver Raspberry Pi 4B 4 GB RAM; IEEE 802.11ac (353 kbps–9 Mbps) IMU module LSM9DS1 (3-DoF accel-gyro-magneto) @100 Hz Rotary encoder SG2A174 ×2 (80 ppr, 200 μs rates) @5 kHz A/D converter module ADS1115 @860 Hz ES920LR 920.6-928.0 MHz ×2 (6.01 kbps–22 kbps); ES920ANT ×2 Hokuyo 2D LiDAR UBG-04LX-F01 (240° scan angle, 0.06–4 m) @25 fps Intel RealSense LiDAR camera L515 (depth: 640 × 480 0.5–9 m, color: 1920 × 1080) @30 fps

. The LoRa transceiver is basically used to convert serial data from LoRa devices to USB signal. Thus, the PC controller can read the serial data and communicate with the mobile robot. The same as in [40], there are two LoRa devices used on the PC controller and the mobile robot to increase the communication bandwidth. This is intended for sending large data, such as 2D and 3D data when using the low-bandwidth communication.

2.2. Software Design

We developed a graphical user interface (GUI) using MATLAB, as shown in Figure 3. The GUI is used by the operator to interact with the mobile robot wirelessly. There are monitoring gauges, a connection menu, command buttons, and a canvas for drawing the map result in real time.

The mobile robot has two missions or tasks, these are the observation task and command task. Figure 4 depicts the task flow in the system. First, the operator starts the mission by using the start button on the GUI. The GUI then sends a start command to the mobile robot for the observation task.

In this task, the mobile robot observes the environment autonomously by sending all observation data to the PC controller. They communicate through the communication thread. This task is intended to build a map for the first time. After finishing the observation, both the PC controller and the mobile robot enter the command task. It enables the command buttons on the GUI, and thus the operator can command the mobile robot based on the first map result. In this task, the mobile robot performs the operator’s commands and all the observation data are also sent to the PC controller to update the map result. This is an iterative task and will be terminated when the operator presses the stop button on the GUI.

In this system, a shared control method is applied. It means that the algorithms are divided to both the PC controller and the mobile robot, as explained in Figure 4. For example, 2D SLAM, a rapidly exploring random tree (RRT) path planning, 3D map building, and GUI are installed in the PC controller. Meanwhile, the mobile robot runs the movement and observation algorithms, such as 3D object extraction, 2D data observation, fuzzy logic control (FLC)-based wall follower and path follower, and obstacle avoidance. This method is especially useful for small robots that are unlikely to carry heavy computers [7]. Furthermore, parallel computing can also be performed. This paper, however, focuses more on the real-time 3D map building using the proposed system, especially with the low-bandwidth communication. In addition, all algorithms in the mobile robot are written in C++, which is preferred for real-time applications.

2.3. Experimental Environment

We applied the proposed method for indoor observation in our laboratory. Figure 5 shows the environmental condition. It has 16.59 m² of large with a lot of cluttered objects inside it. All experiments were carried out in this field, including our datasets for simulation purposes when developing the algorithms. Based on hardware specifications, the objects should have a minimum height of 10 cm, and hence the 2D LiDAR sensor can detect them. In this system, the mobile robot is able to reach an average speed of 0.15 m/s (maximum 0.46 m/s). Therefore, all algorithms need to have real-time capabilities.

3. Proposed Approach

In order to generate a 3D map, 2D SLAM and 3D objects are combined, as depicted in Figure 6. This method enables the system to perform real-time observation even though the low-bandwidth communication is used. The exchanged data can be compressed while maintaining the map accuracy. The next subsections explain important parts of the approach. As mentioned earlier, a shared control method is applied in the system. Therefore, the 3D object extraction algorithm is performed by the mobile robot, while 2D SLAM and 2D–3D combination method are performed by the PC controller.

3.1. 2D Simultaneous Localization and Mapping (SLAM)

Although this paper is focused on real-time 3D map building, it will be difficult to understand without explaining how and what kind of 2D data are used in the system. Hence, this subsection will briefly explain how to observe and manage the 2D data. First, 2D observation data from the LiDAR sensor is compressed for about three-time downsampling [40]. The main reason is, of course, the use of LoRa as low-bandwidth communication. This gives the system a balance between data size and map accuracy. Then, the compressed 2D data will be encoded with the 3D data to be sent to the PC controller.

On the PC controller side, we used MATLAB to develop the mapping algorithms. They include 2D SLAM and 2D–3D combination to construct a 3D map. MATLAB uses graph-based SLAM to create a 2D map [41,42]. A pose estimation of the mobile robot may have an error with the real pose, as well as the estimated map since they are related [43]. The error will increase as the mobile robot wanders throughout the environment. They are usually caused by the sensors’ accuracy and the environmental condition. That is why the graph-based SLAM has a loop closure parameter that will always try to correct the errors. It determines whether or not the current mobile robot pose was previously visited. This is calculated within a small radius by matching the current measurement with all the previous measurements. SLAM is also considered as a basic fundamental problem for a truly autonomous robot [20], where a precise pose is needed to build a map, and the map is also needed to estimate the pose. Therefore, the SLAM problem can be formulated as follows:

$$P\left(x_{1:t},m|z_{1:t},u_{1:t}\right)$$

(1)

where P in (1) is the graph-based SLAM probability distribution, x_t is the mobile robot pose at a specific time or measurement, m represents the map, z_t is observation data from the sensors, and u_t is the controls of the system. Both pose and map are unknown, and they can be solved by calculating the observation data at the beginning of the algorithm. Then, probabilistic models of the observation and motion can be formulated as (2) and (3), respectively.

$$P\left(z_t|x_t\right)$$

(2)

$$P\left(x_t|x_{t-1},u_t\right)$$

(3)

where in (2), the observation relates to the pose, and the new pose is described by the previous pose and its control (3). In addition to the pose graph map, MATLAB also has an occupancy grid map as the output of the 2D SLAM algorithm. Figure 7a,b shows an example of both maps.

The occupancy grid map is generated from the pose graph map, where every cell has a probabilistic value ranging from 0 (empty space) to 1 (occupied space). In the 2D–3D combination process, we use both the pose graph map and the occupancy grid map as the references for 3D objects to be combined. The pose graph map generates positions and orientations of the mobile robot relative to the map, and the occupancy grid map generates information of empty, occupied, or untraveled space. In the system, the 2D SLAM is executed every time step to build a 2D map.

3.2. 3D Object Extraction

The LiDAR camera has two outputs, i.e., color and depth. In this approach, we only use the depth information to extract objects using 3D analysis [44]. Figure 8 shows the image frame projection from the camera.

First, it calculates the real-world coordinates (x_c, y_c, and z_c) using the right-angled triangle principle (4–6).

$$x_c=z_g\tan \left[\left(x_g-\frac{{res}_w}{2}\right)\frac{{FoV}_w}{{res}_w}\right]$$

(4)

$$y_c=z_g\tan \left[\left(y_g-\frac{{res}_h}{2}\right)\frac{{FoV}_h}{{res}_h}\right]$$

(5)

$$z_c=z_g$$

(6)

where x_g and y_g are the pixel coordinate, while z_g is the depth measurement at x_g, y_g. FoV_w and FoV_h are the camera field of view which are 70° and 50°, respectively. res_w and res_h are the camera resolutions which are 640 pixels and 480 pixels, respectively.

The 3D object extraction flowchart is shown in Figure 9. We developed a simplification of the fast-growing neural gas (FastGNG) algorithm, called simplified FastGNG, to extract 3D objects from the depth information or point clouds. The standard GNG algorithm uses an iterative method as the learning strategy [45]. Thus, it is difficult to increase the learning speed as a sample node is added to a current network after errors are accumulated with many times of sampling data. FastGNG, on the other hand, is a modification of the standard GNG algorithm, which has a new growing method to enhance the learning speed [46,47]. General objectives such as clustering, feature extraction, and topological mapping can be accomplished using this algorithm.

We proposed a simplification of the FastGNG algorithm to further increase the learning speed and match with the main purpose of this paper. The following are notations used in the simplified FastGNG algorithm:

$D$

: a set of 3D point cloud data (input);

$A$

: a set of sample nodes;

$w_i$

: a 3D vector of i-th node;

$c_{i,j}$

: a connection between i-th and j-th node;

$L$

: an internal loop parameter of the algorithm;

$l_{\mathrm{m}\mathrm{i}\mathrm{n}},l_{\mathrm{m}\mathrm{a}\mathrm{x}}$

: parameters to set the minimum and maximum connection length;

$P_{edge}\left(v\right)$

: the probability of a sample v is the edge, v is a 3D vector;

$P_{node}\left(v\right)$

: the probability of adding a sample v to the network.

The learning process is then arranged based on following steps:

Step 1. 

Generate a set of sample nodes $A$ using $P_{edge}\left(v\right)$. This generates edge nodes of the depth input to simplify the learning process. The left, right, top, and bottom node of v should be known. $\alpha$ is a constant. The higher the $\alpha$ value, the higher the difference in $P_{edge}\left(v\right)$ value between the edge node and the non-edge node. Hence, making it easier to extract the edge nodes.

$$P_{edge}\left(v\right)=\max ({ed}_1,{ed}_2)$$

(7)

$${ed}_1=\max (\left|dL-dR\right|-dL \alpha ,\left|dL-dR\right|-dR \alpha )$$

(8)

$${ed}_2=\max (\left|dT-dB\right|-dT \alpha ,\left|dT-dB\right|-dB \alpha )$$

(9)

$$dL=‖v-w_{left\left(v\right)}‖,dR=‖v-w_{right\left(v\right)}‖$$

(10)

$$dT=‖v-w_{top\left(v\right)}‖,dB=‖v-w_{bottom\left(v\right)}‖$$

(11)

Step 2. 

Generate two nodes using $P_{node}\left(v\right)$ and connect them. Initially, set $l_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and $l_{\mathrm{m}\mathrm{a}\mathrm{x}}$ to any proper value with a constant $\beta$, where a higher value results in a higher probability value. Then, initialize the phase level iP to 1.

$$P_{node}\left(v\right)=\left\{\begin{array}{c}1.0 l_{\mathrm{m}\mathrm{i}\mathrm{n}}\le {dS}_1\le l_{\mathrm{m}\mathrm{a}\mathrm{x}}\\ 0.0 {dS}_1<l_{\mathrm{m}\mathrm{i}\mathrm{n}} \\ \tanh \left(\beta \frac{l_{\mathrm{m}\mathrm{a}\mathrm{x}}}{{dS}_1}\right) \mathrm{otherwise} \hfill \end{array}\right.$$

(12)

$${dS}_1=‖v-w_{s_1}‖,s_1=\arg \underset{i\in A}{\min }‖v-w_i‖$$

(13)

$$c_{\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}1,\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}2}=1, iP=1$$

(14)

Step 3. 

Initialize the internal loop counter iL to 0 and calculate the $l_{\mathrm{m}\mathrm{i}\mathrm{n}}$, $l_{\mathrm{m}\mathrm{a}\mathrm{x}}$, and L.

$$l_{\mathrm{m}\mathrm{i}\mathrm{n}}=\left\{\begin{array}{c}{\gamma }_1 iP=1\\ {\gamma }_2 iP=2\\ {\gamma }_3 iP\ge 3\end{array}\right.,l_{\mathrm{m}\mathrm{a}\mathrm{x}}=\left\{\begin{array}{c}{\delta }_1 iP=1\\ {\delta }_2 iP=2\\ {\delta }_3 iP\ge 3\end{array}\right.$$

(15)

$$L=\left\{\begin{array}{c}\mathrm{ceil}⌈\frac{\#A}{{\sigma }_1}⌉ iP=1\\ \mathrm{ceil}⌈\frac{\#A}{{\sigma }_2}⌉ iP=2\\ \mathrm{ceil}⌈\frac{\#A}{{\sigma }_3}⌉ iP\ge 3\end{array}\right.$$

(16)

where ${\gamma }_i$, ${\delta }_i$, and ${\sigma }_i$ (i = {1, 2, 3}) are constants for $l_{\mathrm{m}\mathrm{i}\mathrm{n}}$, $l_{\mathrm{m}\mathrm{a}\mathrm{x}}$, and L, respectively. We set higher values of ${\gamma }_i$, ${\delta }_i$, and lower values of ${\sigma }_i$ for a lower phase. Thus, $l_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and $l_{\mathrm{m}\mathrm{a}\mathrm{x}}$ values are the opposite of the phase level iP. This will speed up the learning process in the first phase, and be slower but more detailed in the higher phases. #A is the number of sample nodes.

Step 4. 

Generate a new node v using $P_{node}\left(v\right)$ (12) and increase the loop counter iL. Then, calculate the nearest node $s_1$ (13) and the second nearest node $s_2$.

$$iL=iL+1$$

(17)

$$s_2=\underset{i\in A\backslash \left\{s_1\right\}}{\mathrm{arg\; min}}‖v-w_i‖$$

(18)

Step 5. 

Add the new node v (step 4) to the network with a new index r, and update the connection c. Skip the connection if it already exists. Go to step 4 if iL is less than L.

$$w_r=v$$

(19)

$$c\left\{\begin{array}{c}{c}_{s_1,r}=1,c_{s_2,r}=1,c_{s_1,s_2}=0 {dS}_1<{dS}_{12}>{dS}_2\hfill \\ c_{s_1,r}=1,c_{s_1,s_2}=1 \mathrm{otherwise} \end{array}\right.$$

(20)

$${dS}_{12}=‖w_{s_1}-w_{s_2}‖$$

(21)

$${dS}_1=‖v-w_{s_1}‖,{dS}_2=‖v-w_{s_2}‖$$

(22)

Step 6. 

If there are nodes having more than two connections, remove the longest connection. Repeat the removal process until all nodes have a maximum of two connections.

Step 7. 

As a result of the connection deletion in step 6, remove the nodes having no connection at all, and increase the phase level iP. Stop the process when the stopping criteria are satisfied or iP is more than the max_phase parameter. Otherwise, return to step 3.

$$iP=iP+1$$

(23)

Now, let us move to the post-processing methods. These are as important as the simplified FastGNG in compressing the depth information and correcting defects from the previous step.

3.2.1. Ground Removal

In this system, we need to remove the ground nodes since we already have the ground information from the 2D SLAM. The set of the 3D point cloud data D is needed for this calculation.

$$Gnd\left(v\right)=\left\{\begin{array}{l}0 {dG}_x<g_x\wedge {dG}_y>g_y\wedge {dG}_z<g_z\\ 1 \mathrm{otherwise} \end{array}\right.$$

(24)

$${dG}_x=\left|v_x-w_{i}x\right|,{dG}_y=\left|v_y-w_{i}y\right|,{dG}_z=\left|v_z-w_{i}z\right|,i\in D$$

(25)

where $g_x$, $g_y$, and $g_z$ are the thresholds based on the minimum distance between input nodes. The ground nodes having $Gnd\left(v\right)=1$ are then removed from their network. They basically have no neighbors on the y-axes, and hence we use this information to determine the removal process.

3.2.2. Node Reduction

This method reduces the nodes in a straight connection to further compress the point cloud data. Thus, an angle between two connections ${\theta }_v$ is used.

$${\theta }_v={\cos }^{-1}\frac{\overrightarrow{a} \overrightarrow{b}}{\left|\overrightarrow{a}\right| \left|\overrightarrow{b}\right|}$$

(26)

Vector $\overrightarrow{a}$ and $\overrightarrow{b}$ can be known from the corresponding nodes as they have xyz-value. If ${\theta }_v$ is more than a specific threshold, i.e., 160°, the corresponding node can be removed and its connections should be updated.

3.2.3. Open Loop Connection

A perfect detection should have closed loop networks around the detected objects. Hence, this method is performed to close the open loop networks. The probability of a node v to be connected to the open loop nodes, $P_{open}\left(v\right)$, is defined as follows:

$$P_{open}\left(v\right)={\eta }_1 {\theta }_v+{\eta }_2 dC$$

(27)

$$dC=‖v-w_{{open}_i}‖, i\in A$$

(28)

where ${\theta }_v$ is the same as (26). By using a positive value of ${\eta }_1$ and a negative value of ${\eta }_2$, the $P_{open}\left(v\right)$ prioritizes nodes having a larger angle and shorter distance to be a closed loop. Then, the node having the highest $P_{open}\left(v\right)$ value is connected to the open loop nodes for closing the network.

3.2.4. Node Evaluation

This method is applied to reduce noises and false detection problems. The node evaluation Ne(v) is defined as follows:

$$Ne\left(v\right)=\left\{\begin{array}{l}\mu =\mu +1 {dN}_x\le n_x\wedge {dN}_y\le n_y\wedge {dN}_z\le n_z\\ \mu =\mu \mathrm{otherwise} \end{array}\right.$$

(29)

$${dN}_x=\left|v_x-w_{i}x\right|,{dN}_y=\left|v_y-w_{i}y\right|,{dN}_z=\left|v_z-w_{i}z\right|,i\in D$$

(30)

The counter μ is initialized as 0. The node v should be removed if the μ is less than a specific threshold. This means that the node v does not have a sufficient number of neighbors from the input data D. The thresholds n_x, n_y, and n_z are set based on the minimum distance between input nodes of the point cloud.

3.2.5. Connection Evaluation

A connection from a node k to l can be evaluated as follows:

$$Ce\left(k,l\right)=\left\{\begin{array}{c}1 {dQ}_x>q_x\wedge {dQ}_y>q_y\wedge {dQ}_z>q_z\hfill \\ 0 \mathrm{otherwise} \end{array}\right.$$

(31)

$${dQ}_x=\left|t_x-w_{i}x\right|,{dQ}_y=\left|t_y-w_{i}y\right|,{dQ}_z=\left|t_z-w_{i}z\right|,i\in D$$

(32)

$$t_x=\frac{k_x+l_x}{2},t_y=\frac{k_y+l_y}{2},t_z=\frac{k_z+l_z}{2}$$

(33)

The same as before, the thresholds q_x, q_y, and q_z are set based on the minimum distance between input nodes. This method compares the center point t of a connection to the input reference D. Therefore, bad connections having $Ce\left(k,l\right)=0$ are removed from the network.

The output of the 3D object extraction is clusters, consisting of nodes and connections, as shown in Figure 10.

Every node and connection are then encoded to be sent from the mobile robot to the PC controller. From more than 10,000 points in a point cloud to only less than 100 outputs (nodes and connections), a high compression ratio can be achieved using this method, which is especially useful for low-bandwidth communication.

FastGNG adds new nodes after mini-batch learning using weight parameters. Although it improves the learning speed in comparison to the original GNG algorithm, this also results in unnecessary nodes being added to the network, which requires an additional node deletion process. Hence, the simplified FastGNG avoids that by improving the correctness of the probability functions ($P_{edge}\left(v\right)$, $P_{node}\left(v\right)$), resulting in additional computational cost savings (see the comparison in the experimental results). Furthermore, due to the higher correctness of the probability functions, a new node and its connections can be added to the network in each step without a node deletion process (steps 4 and 5). Moreover, it also filters the point cloud data at the beginning of the algorithm by applying $P_{edge}\left(v\right)$ to improve the correctness of the sample nodes. Those are the main differences between simplified FastGNG and the FastGNG algorithm. We simplified it from a general-purpose algorithm to a specific purpose in this system.

3.3. 2D–3D Combination

The combination process between 2D SLAM and 3D objects is expressed in Figure 11. It starts from rotation and translation with 2D SLAM as the reference. All 3D objects from the simplified FastGNG algorithm are combined based on their clusters to create a new 3D map. Several clustering methods are also employed in the reconstruction process. The following are the steps of the combination process:

Step 1. 

The pose graph map of 2D SLAM generates the orientations (${\theta }_i^c$) and 2D position (tx_i, ty_i) that are used for the input references of the rotation (34) and translation (35), respectively. tz_i are always zero as the mobile robot only moves on a 2D slat surface.

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{cccc}\cos {\theta }_i^c& -\sin {\theta }_i^c& 0& 0\\ \sin {\theta }_i^c& \cos {\theta }_i^c& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\\ 1\end{array}\right]$$

(34)

$$\left[\begin{array}{c}{x}^{″}\\ y^{″}\\ z^{″}\\ 1\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& {tx}_i\\ 0& 1& 0& {ty}_i\\ 0& 0& 1& {tz}_i\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\\ 1\end{array}\right]$$

(35)

In the equation, x, y, and z are the node coordinates of 3D objects. Meanwhile, the double-apostrophe values are the results of this step. The rotation is completed first, and then followed by the translation.

Step 2. 

A density-based spatial clustering of application with noise (DBSCAN) is performed to make a correction of the first rotation and translation results. From the top point of view, a single 3D object may have an L-shape or a non-linear shape. Therefore, a density-based clustering algorithm is needed, and DBSCAN is a pioneer of density-based clustering algorithms [48]. DBSCAN is still being adopted in modern techniques due to its simplicity and compactness for topological clustering applications [49]. The DBSCAN clustering is explained in Algorithm 1. In this method, we convert every 3D object into a 2D line from the top point of view. It simplifies the clustering method since the z values are always correct. The mobile robot never moves on the z-axes. Thus, only x and y values need to be corrected. There are several parameters used in the DBSCAN clustering, such as minObj for a minimum number of objects, dEps for minimum search radius, and thetaTh for rotation threshold. The object rotation angle ${\theta }_o$ is calculated using its gradient value (36), and the distance between objects $\epsilon$ is calculated using the Euclidean distance (37).

$${\theta }_o={\tan }^{-1}\frac{P_{2}y-P_{1}y}{P_{2}x-P_{1}x}$$

(36)

$$\epsilon =\sqrt{{(P_{2}x-P_{1}x)}^2+{(P_{2}y-P_{1}y)}^2}$$

(37)

where P₁ and P₂ are two points in a single object as an xy-line. The output of this step is the clusterObj (colored lines in Figure 11 step 2) that will be used in the next step, and noiseObj (black lines in Figure 11 step 2) that is removed from the map result.

Step 3. 

The pose graph map of 2D SLAM only generates orientations and positions of the individual map displacements. In other words, they are the mobile robot orientations and positions when scanning the environment or capturing the data. Therefore, to generate a 2D map reference, the occupancy grid map from 2D SLAM is used, as shown in Figure 12. It is generated based on the values of the occupancy grid map, where empty space is represented by low values (near 0), occupied space has high values (near 1), and untraveled space is between 0 and 1 (0.5 initially). We set 0.750 as the threshold to generate the 2D map reference. All nodes of the 2D map reference are then compared to the DBSCAN clustering result. By using a specific threshold, i.e., the Euclidean distance, these nodes can all be known to which DBSCAN cluster they belong. They are shown as black circles in Figure 11 step 3.

Algorithm 1: DBSCAN Clustering

Input:sampleObj, thetaTh, dEps, minObj

Output:clusterObj, noiseObj

clusterObj ← Null

noiseObj ← Null

while True do

temp_clust ← random(sampleObj)

isUpdate ← True; l_start ← 1; l_stop ← 1

while isUpdate do

isUpdate ← False

for i ← l_start to l_stop do

core ← temp_clusti

for j ← 1 to numOf(sampleObj) do

if |sampleObjj.θ – core.θ| ≤ thetaTh and

distEuc(sampleObjj, core.xy) ≤ dEps then

temp_clust.add(sampleObjj)

isUpdate ← True

end if

end for

end for

l_start ← l_stop + 1

l_stop ← numOf(temp_clust)

end while

if numOf(temp_clust) ≥ minObj then

clusterObj.add(temp_clust)

else

if numOf(temp_clust) = 1 then

noiseObj.add(temp_clust)

else

clusterObj.add(temp_clust)

end if

end if

if numOf(noiseObj) + numOf(clusterObj) = numOf(sampleObj) then

break

end if

end while

return clusterObj, noiseObj

Random sample consensus (RANSAC) [50] is then performed to generate 2D reference lines (black lines in Figure 11 step 3) based on the 2D map reference and the cluster. The line equations used in the RANSAC are as follows;

$$y^{rsc}=m^{rsc}{x}^{rsc}+b^{rsc}$$

(38)

$$m^{rsc}=\frac{y_2-y_1}{x_2-x_1}$$

(39)

$${\theta }^{rsc}={\tan }^{-1}{m}^{rsc}$$

(40)

The slope $m^{rsc}$ is calculated using the gradient value of two sample points, (x₁, y₁) and (x₂, y₂). Then, its y-intercept $b^{rsc}$ can be known from the sample point ($x^{rsc},y^{rsc}$), and its angle ${\theta }^{rsc}$ can be known from the $m^{rsc}$ value. This process is also explained in Algorithm 2.

Step 4. 

In this step, DBSCAN clustering is performed to the RANSAC results of step 3. It generates new clusters for the 2D map reference nodes as a correction of the previous results. Next, RANSAC is applied to the new clusters to generate new 2D reference lines (light blue lines in Figure 11 step 4). These 2D reference lines are used as the final references to rotate and translate all 3D objects in the last step.

Step 5. 

All 3D objects are re-rotated and re-translated using the last references from step 4. It is a correction of the rotation and translation in the first step. Finally, ground information from the occupancy grid map of 2D SLAM is combined with the result of this step. This is the final rendering result of 3D surface reconstruction, as shown in Figure 11.

As described in Figure 6, the 2D–3D combination is run on the PC controller. Since the 2D SLAM tends to accumulate errors before a loop closure, this process is executed every three time steps after the loop closure is detected. On the mobile robot, both 2D and 3D data are captured at the same time, and thus they represent the same object. However, due to sensors and the object extraction accuracy, they are not exactly aligned. Therefore, we performed an alignment process in the 2D–3D combination method to improve the 3D map accuracy.

3.4. Communication System

The combination of Wi-Fi and LoRa is configured based on each communication protocol, i.e., transmission control protocol/internet protocol (TCP/IP) and serial protocol, respectively. Figure 13 depicts the communication system architecture. The connection manager algorithm manages all network switching processes using a cFlag parameter, as explained in Algorithm 3.

Algorithm 2: RANSAC

Input:sampleData, maxIteration, inliersTh

Output:bestM, bestB, bestTheta

maxInliers ← –1

fori ← 1 to maxIteration do

rand1 ← random(sampleData)

rand2 ← random(sampleData)

while rand1 = rand2 do

rand2 ← random(sampleData)

end while

mrsc ← $\frac{{rand}_2.y-{rand}_1.y}{{rand}_2.x-{rand}_1.x}$

brsc ← rand1.y – mrsc rand1.x

θrsc ← ${\tan }^{-1}{m}^{rsc}$

countInliers ← 0

for j ← 1 to length(sampleData) do

if |θrsc| ≤ 45.0 then

newX ← sampleDataj.x

newY ← mrsc newX + brsc

else

newY ← sampleDataj.y

newX ← $\frac{newY-b^{rsc}}{m^{rsc}}$

end if

dist ← distEuc(new.XY, sampleDataj)

if dist ≤ inliersTh then

countInliers ← countInliers + 1

end if

end for

if countInliers > maxInliers then

maxInliers ← countInliers

bestM ← mrsc

bestB ← brsc

bestTheta ← θrsc

end if

end for

return bestM, bestB, bestTheta

In addition, different encoding and decoding strategies are applied to the system, i.e., ASCII coding for Wi-Fi and binary coding for LoRa, which are not the focus of this paper. The reason for using ASCII coding for Wi-Fi is that the 2D LiDAR output is already in ASCII format, and the data size can be compensated by the Wi-Fi bandwidth (353 kbps–9 Mbps). On the other hand, binary coding is required for LoRa to compress all data as it has low bandwidth (6.01 kbps–22 kbps). Therefore, real-time applications can be performed by the system using both communication bandwidths.

Algorithm 3: Connection Manager

Input:IPaddr, tcpPort, tcpTimeout, comPort, serBaud, serTimeout

Output:cFlag

connTCP ← tcpconfig(IPaddr, tcpPort, tcpTimeout)

connSer ← serconfig(comPort, serBaud, serTimeout)

cFlag ← 1

while True do

if cFlag = 1 then

try

funcTxRx(connTCP)

catch

cFlag ← 0

end try

else

funcTxRx(connSer)

end if

end while

4. Results and Discussion

4.1. 3D Object Extraction Evaluation

First, the standard GNG algorithm, FastGNG, and simplified FastGNG were compared for the same input dataset in our laboratory. All ran the object detection without post-processing methods.

Table 2. Comparison between standard GNG algorithm, FastGNG, and simplified FastGNG.

Standard GNG
No.	No. of Nodes	No. of Connections	Iteration/Phase	Time [ms]	Mean Time [ms]
1	150	159	3700	58.3	67.9
2	150	165	3780	61.8
3	150	171	4120	85.7
4	150	164	4070	68.3
5	150	166	3840	65.4
FastGNG
1	148	278	3	22.5	22.2
2	149	269	3	23.2
3	152	297	3	21.2
4	151	278	3	21.5
5	150	285	3	22.8
Simplified FastGNG
1	155	146	3	13.6	16.2
2	144	132	3	18.2
3	158	143	3	17.3
4	152	137	3	16.0
5	145	130	3	15.8

summarizes the results.

It is difficult to run for the same number of nodes in each trial, especially for FastGNG and simplified FastGNG, since they use phase level (iP). Based on the results, the standard GNG algorithm has the highest computational cost. It really depends on the number of iterations. On the other hand, the FastGNG algorithm runs faster. It is roughly three times faster than the standard GNG algorithm. The simplified FastGNG has a similar performance as the FastGNG. It uses the same core as in the FastGNG algorithm, however, in a simpler way, to achieve improved performance for the proposed system. In this case, we set the max_phase = 3 as the limit.

In order to evaluate the performance of the 3D object extraction method, 3D point cloud datasets from our laboratory were used. The datasets, which have the same resolution of 640 × 480 pixels, were previously recorded using the LiDAR camera as described in Section 2.1. All parameter settings for the evaluation can be seen in

Table 3. Parameter settings for 3D object extraction.

Parameter	Value	Definition
$\alpha$	3.9	Constant for generating a set of sample nodes A
$\beta$	1.55	$\mathrm{Constant} \mathrm{for} P_{node}\left(v\right)$
${\gamma }_i$	${\gamma }_1=0.225 \mathrm{m},{\gamma }_2=0.150 \mathrm{m},{\gamma }_3=0.100 \mathrm{m}$	Minimum length of a connection in the network for each phase level iP
${\delta }_i$	${\delta }_1=0.300 \mathrm{m},{\delta }_2=0.225 \mathrm{m},{\delta }_3=0.175 \mathrm{m}$	Maximum length of a connection in the network for each phase level iP
${\sigma }_i$	${\sigma }_1=10,{\sigma }_2=8,{\sigma }_3=6$	Internal loop parameter for mini-batch learning
$g_x,g_y,g_z$	$g_x=0.080 \mathrm{m},g_y=0.035 \mathrm{m},g_z=0.080 \mathrm{m}$	Thresholds for ground removal
${\eta }_1,{\eta }_2$	${\eta }_1=0.01,{\eta }_2=-0.95$	Thresholds for open loop connection
$n_x,n_y,n_z$	$n_x=0.2 \mathrm{m},n_y=0.2 \mathrm{m},n_z=0.2 \mathrm{m}$	Thresholds for node evaluation
$q_x,q_y,q_z$	$q_x=0.06 \mathrm{m},q_y=0.06 \mathrm{m},q_z=0.06 \mathrm{m}$	Thresholds for connection evaluation

. Figure 14 shows the detection results, while the performance is summarized in

Table 4. 3D object extraction performance.

No.	Input Image	No. of Points in Point Cloud	No. of Nodes	No. of Connections	No. of Clusters	Min Distance between Nodes	Max Distance between Nodes	Time [ms]
1	Figure 14a	8586	77	80	8	0.100 m	2.430 m	32.7
2	Figure 14b	11,647	48	49	3	0.100 m	3.210 m	27.6
3	Figure 14c	10,455	61	67	5	0.100 m	3.180 m	34.6
4	Figure 14d	10,802	80	86	2	0.102 m	1.575 m	35.6
5	Figure 14e	13,948	54	60	1	0.102 m	1.217 m	32.1
6	Figure 14f	14,094	70	76	2	0.100 m	0.560 m	30.0
Frame Rate:		Mean = $\frac{1 \mathrm{s}}{32.1 \mathrm{m}\mathrm{s}}$ = 31.15 fps; Min = $\frac{1 \mathrm{s}}{35.6 \mathrm{m}\mathrm{s}}$ = 28.09 fps; Max = $\frac{1 \mathrm{s}}{27.6 \mathrm{m}\mathrm{s}}$ = 36.23 fps

. In the experiments, all post-processing methods were also performed. Based on the results, the proposed method can achieve the mean performance of 31.15 fps. That will be difficult to achieve when using the standard GNG algorithm, which has the mean execution time of 67.9 ms (14.73 fps) without any post-processing methods. This implementation is necessary, as we need to maintain all algorithms to be real-time capable in order to realize the target of this system.

Another important thing is the data compression. Remember that we need to send all monitoring data using both Wi-Fi and LoRa networks. It is impossible to send all raw-point cloud data using low-bandwidth communication. Due to bandwidth restrictions, real-time performance will not be achieved, and there will be a data loss problem. Thus, the simplified FastGNG solves the problem by compressing the data size while at the same time maintaining its performance in real time. The trade-off is, of course, the map quality. Higher compression results in faster data transmission time but lower map quality, and vice versa. However, compared to our previous work using particle swarm optimization-seeded region growing (PSO-RG) [51], which has a mean performance of 11.3 fps, the simplified FastGNG offers a better balance between data compression, accuracy, and computational cost. Furthermore, the PSO-RG limits each cluster to only eight nodes, which is much less than the simplified FastGNG results (48–80 in Table 4), making it more accurate to represent the detected objects. Even though the data size is approximately six to ten times larger, the communication bandwidth can still handle it.

4.2. Real-Time Experimental Results

We performed fifteen trials to evaluate the proposed methods. All of them were run on the proposed system in real time. There are configurations of Wi-Fi to LoRa, LoRa only, and Wi-Fi only, each of which are conducted five times. The environmental condition for experiments is shown in Figure 5. However, the placement of the objects in it may change. The parameter settings for the experiments are shown in

Table 5. Parameter settings for 2D–3D combination method.

Parameter	Value	Definition
thetaTh	10°	DBSCAN clustering rotation threshold
dEps	0.10 m (step 2) 0.15 m (step 4)	DBSCAN clustering minimum search radius
minObj	2	DBSCAN clustering minimum number of objects
maxIteration	1.5 n	RANSAC maximum iteration. n is the number of sample data
inliersTh	0.10 m (step 3, 4)	RANSAC inlier threshold

. These parameters are used in the 2D–3D combination method to build a 3D map.

Table 6. Computational costs of real-time experiments.

PC Controller
No.	Network		Data Transmission Time [ms] Mean of 50+ Data			Mean [ms]	2D SLAM and 2D–3D Combination [ms] Mean of 50+ Data	Mean [ms]	Data Size [KB] Mean of 50+ Data		Mean [KB]
1	Wi-Fi to LoRa		273			274	93	98	0.727		0.734
2			266				92		0.740
3			277				106		0.735
4			272				96		0.718
5			280				106		0.752
6	LoRa		570			625	89	99	0.463		0.471
7			703				106		0.472
8			618				102		0.474
9			577				98		0.471
10			659				103		0.475
11	Wi-Fi		17			18	109	112	0.926		0.932
12			23				108		0.928
13			16				116		0.936
14			17				109		0.914
15			18				118		0.936
Mobile Robot
		Hardware			Sampling Rate		Algorithm			Encoding	Total
2D Data		2D LiDAR			25 fps (40 ms)		3× Downsampling: 10 ms			5 ms	55 ms
3D Data		LiDAR Camera			30 fps (33 ms)		Simplified FastGNG: 32.1 ms (mean)			10 ms	75.1 ms
Total Computational Cost
Wi-Fi to LoRa				274 ms + 98 ms + 75.1 ms = 447.1 ms (2.24 Hz)
LoRa				625 ms + 99 ms + 75.1 ms = 799.1 ms (1.25 Hz) (main target)
Wi-Fi				18 ms + 112 ms + 75.1 ms = 205.1 ms (4.88 Hz)

shows the computational costs of the experiments. The mobile robot observes 2D and 3D data in parallel, and hence the computational cost in it will be the longest time. Figure 15 depicts the 3D map results (Figure S1). In Figure 15a, the mobile robot performed both the observation task and the command task. Therefore, the mobile robot final position is different from Figure 15b,c, which only performed the observation task. Similarly, the map density is also different due to the difference in the amount of data sent by the mobile robot to the PC controller. In the experiments, the mobile robot reached an average speed of 0.15 m/s (maximum 0.46 m/s), which depended on the track length and obstacles.

Figure 16 explains the missions of Figure 15a step by step, where the system switched the network automatically from Wi-Fi to LoRa in the middle of the observation task (Video S1). Then, in the command task, the operator commanded the mobile robot to go to a specific position in the map by using the command buttons on the GUI. Of course, there are some delays for monitoring data when using the LoRa network compared with the Wi-Fi network. This is caused by the low bandwidth of LoRa.

Based on the results in Table 6, the data transmission time is different among different network configurations, and obviously Wi-Fi is the fastest network. The 2D SLAM and 2D–3D combination are performed in the PC controller, and hence they all have similar results. Lastly for data size, as mentioned earlier, we used different strategies for data coding methods between Wi-Fi and LoRa networks. ASCII coding is used for Wi-Fi and binary coding is used for LoRa. Therefore, the mean LoRa data size is smaller than the mean Wi-Fi data size. When using the Wi-Fi to LoRa configuration, the data are coded either using ASCII or using binary coding corresponding to the current network that the system uses. On the mobile robot side, there are two kinds of data sent to the PC controller, i.e., 2D and 3D data. A three-time downsampling is applied to the 2D data and the simplified FastGNG is applied to the 3D data to extract 3D objects. These result in a total maximum computational cost of 799.1 ms, which is 1.25 Hz. Thus, the operator can obtain the map update for at least 1.25 Hz for monitoring data. Faster updates can, of course, be achieved when the system uses the Wi-Fi network.

To evaluate the map results, we combined all fifteen maps and plotted them in 2D from the top point of view, as shown in Figure 17. Since the mobile robot only moves in 2D space, only the x and y-axes need to be evaluated. Compared to the ground truth in Figure 15d, they have about the same shape and size, which are 3.95 m in width and 4.2 m in length. At the same time, the results show that the proposed method has good repeatability from all fifteen trials with different network configurations.

There is one more scenario for the system to perform real-time observation and 3D map building. Figure 18 depicts the map results. The environmental condition is the same but with different object placements. The LoRa network was used and resulted in similar performance as in Table 6. These results also confirm the feasibility of the proposed method for performing real-time 3D map building under the system restrictions.

4.3. Discussion

In order to achieve real-time application of the system, all implemented algorithms need to have real-time capabilities, as depicted in Figure 6. The more data for the algorithms, the higher the accuracy of the results. However, the computational costs may also increase. Therefore, it is very important in this system to maintain a balance between the amount of data and its accuracy. In mobile robot operations, at least an operator is needed not only to control, but also to monitor the mobile robot’s behaviors in performing the missions. Hence, real-time capabilities should be considered since they enable not only direct results, but also real-time responses between the operator and the mobile robot. In [22], it defines a real-time target for 3 s. This paper, however, is targeted at 1 s (1 Hz). We considered it based on the implemented hardware in the system, and it was confirmed by the experimental results.

As depicted in Figure 4, there are several tasks running in parallel on the mobile robot while observing the environment. The FLC wall path follower and obstacle avoidance are running in the background to handle the mobile robot’s movements. Regarding the mobile robot speed, we defined in the FLC controller that 0.2 m/s is the medium speed, less than 0.1 m/s is the slow speed, and more than 0.4 m/s is the fast speed. Based on the experimental results, the mobile robot is able to reach a maximum speed of 0.46 m/s with an average speed of 0.15 m/s. This is a car-like mobile robot, and thus it also needs to go backward to avoid the obstacles. All the above experiments, of course, run in parallel with the background programs. In addition, we also installed an active cooling fan to maintain the Raspberry Pi CPU temperature. Figure 19 shows the CPU performance when running the missions. There are configurations of Wi-Fi to LoRa, LoRa only, and Wi-Fi only. The missions were started at around 5 s, and then both CPU temperature and CPU usage started to rise. When using the Wi-Fi network, the CPU usage is lower than when using the LoRa network. This is caused by the program architecture. One additional thread is added for the LoRa network to help the communication thread. It performs timeout recovery algorithms to prevent data loss on the LoRa network [40], as all data are important for the mapping process. It is worth noting that when the CPU temperature rises to 80–85 °C, the CPU will start to slow down and reduce all clocks, as written in the Raspberry Pi documentation. In the worst case, the simplified FastGNG algorithm may drop to 10 fps when the CPU throttling happens. Hence, the implementation of the active cooling is very important.

Modern SLAM algorithms, such as ORB-SLAMs [29,30,31], can also run in real time on embedded systems [32,33]. Since they require three or more threads to achieve real-time performance, it is difficult for this system to be implemented. Instead, we run the mapping algorithms on the PC controller and focus the mobile robot on data observation. Under embedded systems, ORB-SLAMs can achieve a performance of 2.2–12.69 fps (454–79 ms) [32]. If we only compare the mapping algorithms, the proposed method can perform competitively, which is 8.9–10.2 fps (112–98 ms in Table 6).

In contrast to [28], which prioritizes quality and repeatability, this paper prioritizes repeatability (Figure 17) and real-time performance under low-cost hardware limitations to perform 3D mapping. This is one of our novelties, as most of the state-of-the-art focuses on how to construct a 3D map for better quality but ignores real-time processing [23,24,25]. This will make real-time control and observation difficult to realize, even more so for low-cost devices. Of course, this is a trade-off between data size, accuracy, and computational cost. In this paper, we balance all of them to achieve the target of the proposed system. Thus, the drawback of this work is map quality, as it is nearly impossible to send the amount of data under low-bandwidth communication in real time.

The advantage of our proposed approach is the use of low-bandwidth communication. The combination of high and low-bandwidth networks is also developed in [2,32]. All of them confess that the network combination improves the robustness of wireless communication for mobile robot applications. However, they still require the high-bandwidth network to build a full 3D map and complete the missions. Our system, on the other hand, enables complete network switching. This means that all observation data for 3D mapping can be sent to the host using either the high or low-bandwidth network, even when the main network is completely lost. This brings us to another novelty of this paper, as, to the best of the authors’ knowledge, there are no similar works performing real-time 3D map building under low-bandwidth networks.

The first step of the 2D–3D combination method, rotation and translation, is actually sufficient to represent the environment as a 3D map. The reason for applying the other processes is accuracy. Next, RANSAC is also a bit time consuming. In this case, we set the maxIteration parameter to 1.5 n, where n is the number of sample data.

As explained in Section 2, C++ is used in the mobile robot and MATLAB is used in the PC controller. MATLAB has a MATLAB Coder, which generates C/C++ code directly from MATLAB code. This is very important to be implemented as it can run more than ten times faster than native MATLAB code. Table 6 shows how fast the algorithms (2D SLAM and 2D–3D combination) can be executed by the PC controller.

The challenge of the proposed approach is the number of parameters. We found that it is difficult to obtain appropriate results without understanding the input data characteristics. We have to set the parameters step by step based on the purpose of each algorithm. It is difficult to jump to the next parameters if the previous parameters are not stable yet. For example, setting the parameters for six post-processing methods after the simplified FastGNG algorithm. We must first successfully remove the ground nodes before adjusting the second method, and so forth.

The compressed data size for LoRa is around 0.471 KB (mean). This enables a real-time transmission time of 627 ms (625 ms in experiments) with a speed of 6.01 kbps. By using the compression ratio for 2D and 3D data, the raw data size for LoRa can be estimated to be 11.492 KB. It will take around 15.29 s to send the uncompressed data using LoRa, making the real-time target completely impossible when using the low-bandwidth communication. The Wi-Fi transmission speed can also be estimated to be 414.22 kbps, where the data size is 0.932 KB and the transmission time is 18 ms. However, Wi-Fi can certainly perform much faster since it can stream videos in real-time.

There are two restrictions in the proposed system, i.e., limited onboard computing resources and low-bandwidth communication. Compared with [2,22,23,28], it is difficult to achieve the real-time target under those two restrictions. Therefore, we developed the proposed method to address the problems. It proved the ability to achieve the target of 1 Hz refresh rate, which is our design system for enabling real-time control and monitoring for the operator. The combination of Wi-Fi and LoRa makes the system robust against network problems. The developed algorithms are lightweight and can be installed on relatively low-cost hardware. Parallel computing is also performed as the system uses a shared control method to run all the algorithms. Finally, real-time 3D map building can be conducted using both high and low-bandwidth networks.

5. Conclusions

3D visualization of the real world is becoming increasingly important. In robotics, it is not only required for intelligent control, but also necessary for operators to provide intuitive visualization. However, due to the increase in the amount of data, real-time processing is becoming a challenge, even more so for low-cost mobile devices. Therefore, in order to address this problem, we propose a real-time 3D map building based on the combination of 2D SLAM and 3D objects that can be applied to low-cost hardware.

Starting with 2D–3D data observation and 3D object extraction, all observation data are sent wirelessly from the mobile robot to the PC controller to create a 3D map. First, the 2D SLAM algorithm is run by the PC controller to build a 2D map based on the 2D observation data. Then, the simplified FastGNG algorithm, which is run by the mobile robot, compresses 3D point cloud data as 3D objects. It gives the system a balance between data size, accuracy, and computational cost, and thus enables real-time data transmission even though the low-bandwidth communication is used. Lastly, the 2D–3D combination method is applied to create a 3D map in the PC controller. This combines the 2D data from 2D SLAM and 3D objects from the simplified FastGNG algorithm. The low-bandwidth communication using the LoRa network is actually the main target, as it restricts not only the data size, but also the data transmission speed.

The results show that the proposed method can perform a minimum 1.25 Hz refresh rate, which is real-time capable. It means that the operator will obtain the map update for at least 1.25 Hz for one monitoring data. This proves that the target of a 1 Hz refresh rate can be achieved by the proposed method. Of course, a higher refresh rate can be achieved when the system uses the Wi-Fi network.

In addition, the combination of high and low-bandwidth networks also enables the system to maintain its client–server communication. Therefore, the missions can be continued, and real-time monitoring can still be performed even if the system loses the Wi-Fi network completely.

In future work, we will improve the LoRa performance since it is still far from its maximum bandwidth. The data may also need to be compressed using another compression technique that makes it smaller in size. In addition, we want to try more complex environments to see what further improvements are needed for the algorithms.