MFFRand: Semantic Segmentation of Point Clouds Based on Multi-Scale Feature Fusion and Multi-Loss Supervision

Abstract

With the application of the random sampling method in the down-sampling of point clouds data, the processing speed of point clouds has been greatly improved. However, the utilization of semantic information is still insufficient. To address this problem, we propose a point cloud semantic segmentation network called MFFRand (Multi-Scale Feature Fusion Based on RandLA-Net). Based on RandLA-Net, a multi-scale feature fusion module is developed, which is stacked by encoder-decoders with different depths. The feature maps extracted by the multi-scale feature fusion module are continuously concatenated and fused. Furthermore, for the network to be trained better, the multi-loss supervision module is proposed, which could strengthen the control of the training process of the local structure by adding sub-losses in the end of different decoder structures. Moreover, the trained MFFRand network could be connected to the inference network by different decoder terminals separately, which could achieve the inference of different depths of the network. Compared to RandLA-Net, MFFRand has improved mIoU on both S3DIS and Semantic3D datasets, reaching 71.1% and 74.8%, respectively. Extensive experimental results on the point cloud dataset demonstrate the effectiveness of our method.

1. Introduction

The visual sensors are simple in structure and inexpensive, but they are greatly affected by illumination and can hardly work in dark scenes. Compared with the 2D images collected by visual sensors, 3D point clouds obtained by LiDAR have irreplaceable advantages, such as higher measurement accuracy, faster response speed and stronger anti-interference capability, which not only avoid the problems of illumination and object pose in the process of image acquisition, but also obtain the spatial and distance information of the targets. Achieving efficient and accurate semantic segmentation of point clouds is an important research content in autonomous driving.

A series of processing methods have been proposed in the field of point cloud semantic segmentation in recent years, including the projection-based methods [1,2,3,4,5], voxel-based methods [6,7,8,9,10] and point-based methods [11,12,13,14,15,16,17,18]. The projection-based methods are derived from the processing of 2D images; researchers convert irregular point clouds data into conventional multi-views and input them into the deep network architecture for semantic segmentation. Despite mature 2D image processing methods being used after projecting, there is still much spatial geometric information lost in the transformation of point cloud data from 3D to 2D projection. Therefore, there are some inherent limitations for projection-based methods. For voxel-based approaches, the collected point cloud data is transformed into voxels. The voxelization allows point clouds to be processed by using the 3D neural network model, and achieves good results. However, compared with 2D images, the voxelization of point cloud leads to greater computational complexity, so the practicability of this method is relatively low in real scenarios. The point-based methods start from the 3D original data, and the spatial characteristics and relative position information of the point clouds are directly processed through the network. Therefore, there are usually good effects on point clouds’ segmentation with point-based methods. However, due to the design of these network structures, there is still plenty of room for improvement of the segmentation effect.

Most existing networks are still inadequate for point cloud feature extraction and the utilization of high and low levels semantic information. Inspired by the feature processing [19,20] and deep supervision [21,22] in 2D image segmentation, the Multi-Scale Feature Fusion Based on RandLA-Net (MFFRand) is proposed in this paper, which makes better use of the feature information extracted from the network in point clouds.

The main contributions of this paper can be summarized as the following three aspects:

• A multi-scale feature fusion module is proposed, which different levels of encoder-decoders interconnect to achieve effective feature fusion between high and low levels of semantic information.
• A multi-loss supervision module is proposed, of which multiple sub-losses connected to different levels of encoder-decoder for supervision of the network training. By this way, the local structures could be trained more sufficiently to achieve better feature fusion, so as to further optimize the final segmentation results.
• The MFFRand allows a one-time training for deep networks to obtain the outputs of network with different depths. This is so that the optimal network depth could be selected according to the results inferred by MFFRand.

2. Related Work

LiDAR is an important tool for environmental perception in autonomous driving. Many experts and scholars have conducted a lot of researches on how to use point cloud data of LiDAR to achieve efficient and accurate semantic segmentation, which mainly include projection-based, voxel-based and point-based methods.

2.1. Projection-Based Methods

Earlier researchers applied deep learning on point cloud data by projecting point clouds into 2D images with multiple views. Su et al. [1] first proposed the MVCNN network based on 3D shape descriptors, which constructed 2D images from different perspectives of 3D objects, and the improved VGG-M model was used to achieve the final semantic segmentation effect. On the basis of MVCNN, Feng et al. [3] considered the correlation of different views and grouped the visual descriptors extracted by CNN under different perspectives, which made full use of the feature relationships in multiple views. To compensate for the loss of geometric information in the process of projecting 3D objects to 2D images, Qi et al. [2] introduced multi-resolution filtering in 3D to deeply explore the feature information of 3D objects and further enhance the segmentation capability. Meanwhile, to address the problem of long-tailed distribution of LiDAR data across space, the points were quantified in the polar coordinate in PolarNet [23]. As a way to better comprehend driving scenes, Peng et al. [24] proposed a multi-attention mechanism for dense top-view semantic segmentation based on sparse LiDAR data. Meanwhile, Lyu et al. [25] considered how to project the features of 3D point clouds onto 2D more efficiently; the 2D mesh projection was obtained by the Kamada-Kawai (KK) algorithm with integer programming in this method, and they used a hierarchical approximation strategy to improve computational efficiency. The selection of viewpoints plays a crucial role in projection-based methods. Li et al. [26] considered selecting viewpoints in a data-driven manner and designed an end-to-end network to learn local multi-view descriptors of point clouds, which is robust. By introducing the attention mechanism, the model can also mine feature information more effectively, especially context information from multiple views [27,28,29]. Although the projection-based approaches have solved the problem of unstructured the point cloud data, they inevitably lead to the loss of internal geometric structure information.

2.2. Voxel-Based Methods

VoxNet [6] is the first deep learning network to process point clouds with voxelization. For VoxNet, the 3DCNN was used in processing voxels to address the problem of disorder and unstructured point clouds. The prediction of 3DCNN was limited to the rough output at the voxel level; thus, the voxel size became a limiting factor to the overall accuracy. To address this problem, the trilinear interpolation was used in the SegCloud [7] network to send rough predictions back to the original 3D point cloud and inferred the semantic labels of the point cloud through FC-CRF. Compared with 2D images, the voxel-based approaches are computationally expensive due to the additional dimension; to reduce computational cost, the octree [8,30] and Kd-tree structure [9] were both applied to the semantic segmentation of the point clouds. Sparsity and varying the point density are also common problems; solving such problems is commonly performed using the cylindrical model [31,32]. The 3D models’ inherent properties are usually overlooked by mainstream point cloud detection frameworks; Han et al. [33] proposed a 3D point cloud segmentation model based on occupancy perception, in which a geometric appearance constraint is used to divide the point cloud into supervoxels, and occupancy signals are used to guide the network to achieve semantic segmentation. In addition, a data volume decomposition was used by Chew et al. [34] to analyze point clouds and generate useful features for semantic segmentation. To simplify computations and supplement the details of smaller instances, an attention-focusing feature fusion and an adaptive feature selection module were proposed by Cheng et al. [35]. Nevertheless, the practicality of voxel-based methods is still poor due to the high spatial complexity of voxelization. However, with the increasing computation efficiency, there is still much room for development in voxel-based methods.

2.3. Point-Based Methods

To reduce the loss of information and computational complexity, some scholars started to study the point cloud processing based on the original point cloud. PointNet [11] is the pioneer of point-based methods. It addressed the disorder of the point cloud with symmetric functions and T-Net was introduced to handle the rotation invariance. PointNet++ [12] is a hierarchical structure of PointNet. Although this structure realized the perception of the local visual field of the point cloud, it does not yet address the problem of the relationship between points. The Adaptive Feature Adjustment (AFA) was proposed in PointWeb [36] to simulate the interaction between points, and it could realize the information exchange and local feature learning of points. The point-based convolution is difficult to apply on point clouds due to the disorder of point clouds; to address this problem, a transformation matrix x-transformation was predicted in PointCNN [37], which made the prediction after point cloud transformation independent of the order of the input points. Inspired by image convolution, the spatial convolution of the point clouds was realized in KPConv [18] by constructing a Kernel point that could be extended to variable convolution. Different from KPConv, the fuzzy mechanism was introduced into the discrete convolution kernel of 3D point clouds in SegGCN [38], which divided the neighbor occupied space in a deterministic manner, and it could reduce the influence of boundary effects suffered during the spatial discretization. The sampling methods in various schemes were usually reasons of inefficiency in the processing of large-scale point clouds; therefore, a random sampling strategy was used in RandLA-Net [15] network to improve the point cloud processing speed. Lu et al. [39] proposed to utilize different aggregation strategies between the same category and different categories. The ability to perceive detailed features can be effectively improved by the aggregation or enhancement strategies for local features [40,41]. In addition, in order to efficiently learn the features of various types of targets from large-scale point clouds, Fan et al. [42] proposed the SCF-Net, which used the dual-distance attention mechanism and global contextual features to improve the performance of semantic segmentation. The point-based methods directly start from the raw data of 3D point clouds, which can obtain more effective and relevant information.

However, existing methods have limited capability to capture local details of different scales, since they have not utilized the given information fully. The method proposed in this paper achieves feature extraction at different scales by adding cross-layer connections to the extracted backbone network, and a multi-loss supervision for the feature extraction process is designed. Therefore, our method is able to leverage the advantages of point-based methods and mine the local feature information at different scales further, which in turn improves the performance of the network.

3. Methodology

Figure 1a is a multi-scale feature extraction network composed of five down-sampling and five up-sampling module in RandLA-Net. The overall structure of 6-layers of MFFRand is demonstrated in Figure 1b. Compared with RandLA-Net, MFFRand model adopts the combination of multiple encoder–decoder structures for multi-scale feature fusion in the process of 3D point clouds. In Figure 1b, the orange part is the multi-scale feature fusion module, and the blue lines are the multi-loss supervision module.

3.1. Multi-Scale Feature Fusion Module

In many studies, the fusion of different scale features is an important method to improve the segmentation performance. The low-level features are usually with high-resolution and contain more local and detail information. However, due to fewer convolution operations, low-level features have less semantics and more noise. The high-level features are usually with low-resolution and contain stronger semantic information, but their perception for details is weak. How to fuse the semantic features of different levels efficiently is the key to improve the performance of the segmentation network. To meet the requirements of more accurate segmentation in autonomous driving, a feature fusion module based on multiple encoder–decoder structures is proposed in this paper.

After being processed by encoder–decoder, the common networks usually have the same number of points, feature dimensions and feature sizes. However, this is due to multiple down-sampling and up-sampling, which leads to a large semantic gap between the feature maps of encoder and decoder. To deal with this problem, multiple encoder–decoder structures are added to the original RandLA-Net, and the structure is shown in the top of Figure 2. This module is composed of five encoder-decoders with different levels, and the same backbone is used as the encoder in each encoder–decoder structure. In addition to up-sampling at the decoder phase, feature concatenation is performed in each layer (blue rectangular block). Taking the bottom feature layer $F_{0,0}$, $F_{1,1}$, ⋯, $F_{5,5}$ as an example, the concatenation of this layer is demonstrated in the bottom of Figure 2. The feature maps of different encoder–decoders are fused by the skip connection, so as to form a dense feature fusion network. At the same time, the parameters of network could be updated effectively through the back propagation gradient flow and ensure the network could be trained properly.

In terms of presentation, for different levels of encoder–decoder structures, $F_{i,j}$ denotes the feature map obtained by i encoding and j decoding. In the same layer (horizontal blue rectangular block), there are the same number of points and feature dimensions in feature maps. For example, $F_{0,0}$, $F_{1,1}$, ⋯, and $F_{5,5}$ are in the same layer of the network. Although the feature maps belong to different levels of encoder–decoders, the number of points and the dimension of feature in this layer remain the same after multiple encoding and decoding operations.

There is a relationship between different encoder–decoder structures. For the encoder–decoder of the 2-layer in Figure 3a, the calculation order is $F_{0,0}$-$F_{1,0}$-$F_{1,1}$, and for the encoder–decoder of 3-layer in Figure 3b, the calculation order is $F_{0,0}$-$F_{1,0}$-$F_{1,1}$-$F_{2,0}$-$F_{2,1}$-$F_{2,2}$. Similarly, the encoding and decoding operations are carried out on the encoder–decoder of layer 4 (Figure 3c), layer 5 and layer 6 successively. In the process of down-sampling, the random sampling strategy is used to improve the processing speed, and the LFA module is used to enhance the receptive field of network. In the process of up-sampling, each up-sampling is accompanied by the fusion with the information extracted from the previous encoder–decoder. In this way, the semantic gap is continuously bridged. The calculation of the feature fusion is given by Equation (1):

$$F_{i,j}=\left\{\begin{array}{cc}R\left(L\left(F_{i,j}\right)\right),\hfill & \hfill j=0\hfill \\ \hfill C\left(\mathrm{\Delta }(\mathrm{\Delta }{\left(F_{i-j+t,t}\right)}_{t=0}^{j-1},U\left(F_{i,j-1}\right))\right),\hfill & \hfill j>0\hfill \end{array}\right.$$

(1)

where R denotes random sampling, L denotes LFA, C represents convolution, $\mathrm{\Delta }$ represents feature concatenation, and U denotes up-sampling. When $j=0$, the encoding operation is carried out firstly, then the down-sampling and LFA is performed in the backbone for 5 times. When $j>0$, each feature map is obtained through the convolution of $j+1$ inputs. The input is composed of the first j feature maps ($F_{i-j,0}$, $F_{i-j+1,1}$, ⋯, $F_{i-1,j-1}$) in the same layer and the output by the up-sampling with MLP of the feature map ($F_{i,j-1}$) in the previous layer.

3.2. Multi-Loss Supervision Module

Since the network consists of multiple encoder–decoder structures, there are more complex computational steps in the feature fusion process, and the overall training process is more difficult to be controlled. Inspired by DSN [43] and Multi-Scale Structure-Aware network [22], a multi-loss supervision module is introduced to control the local training process. On the basis of the multi-scale feature fusion module, the training process is controlled by adding sub-losses at the end of each decoder separately, which make the local structure trained fully and achieve better feature fusion.

In the five levels of encoder–decoders, the cross-entropy loss function with category weight is added, respectively.

The weight of each category is shown as Equation (2):

$$w_i=\frac{1}{(\frac{nu{m}_i}{{\sum }_{t=1}^{n=8}nu{m}_t}+0.02)},i=1,2,3,\cdots ,8$$

(2)

where $nu{m}_i$ is the number of points belonging to ith category. There are eight categories in the Semantic3D [44] dataset. The decimal 0.02 in the denominator is to avoid the case where the denominator is 0. When the number of points in a category is less, there will be a higher weight in this category.

Depending on the number of point clouds contained in different categories, the cross-entropy loss output of each category accounts for different proportions, and the calculation of cross-entropy loss is given by Equation (3):

$$H\left(p,q\right)=-\sum _{t=1}^c\left(p\left(x_t\right)\times log\left(q\left(x_t\right)\right)\right)\times w_i$$

(3)

where $H(p,q)$ is the cross-entropy loss function with weights per point, $p\left(x\right)$ is the real distribution of the target, $q\left(x\right)$ is the predicted distribution of the target, c is the number of categories, and the output value of each point will be correspondingly multiplied by that weight as the predicted loss of the target.

The sub-losses output of each encoder–decoder is given by Equation (4):

$$sub\_loss={\displaystyle \frac{{\sum }_{i=1}^{n}H{\left(p,q\right)}_i}{n}}$$

(4)

where n is the number of input points, and $H{(p,q)}_i$ is the loss of ith point.

The loss function of the entire network is given by Equation (5):

$$loss={\displaystyle \frac{1}{m}}\sum _{i=1}^{m}sub\_los{s}_i$$

(5)

where m is the number of sub-losses.

The detailed structure of the network with multi-loss supervision is shown in Figure 4.

4. Experiments

To evaluate the performance of MFFRand, a series of relevant experiments are conducted on MFFRand. The experiments are divided into four parts: evaluation of mIoU (mean Intersection-over-Union) and OA (Overall Accuracy), time consumption, ablation study and the effectiveness of network segmentation at different depths. All the experiments are conducted on Intel(R) Xeon(R) Gold 5118 @2.30 GHz CPU and NVIDIA GeForce GTX 1080Ti GPU. Meanwhile, in order to ensure the reliability of experimental results, all results are averaged over three experiments.

4.1. Evaluation of mIoU and OA

S3DIS [45] is a large-scale indoor dataset which consists of 6 large areas; each area contains several scenes, such as offices, hallway, conference rooms, and so on. The whole dataset has around 273 million points annotated with 13 semantic labels. In our experiments, we use a six-fold cross-validation to evaluate the proposed network.

Semantic3D [44] dataset was developed by a research group at ETH Zurich, Switzerland. Semantic3D provides a large labeled 3D point cloud with a total of over 4 billion points, which contains 8 types of objects, including buildings and cars, etc. The dataset is split into 15 training scenes, 15 testing scenes and 4 reduced testing scenes. The reduced-8 is the reduced version of the testing set provided by Semantic3D for a convenient online evaluation, which includes 4 of the 15 testing scenes, and the point clouds are down-sampled at 0.01 m. In our experiments, two of the training scenes are used for validation during training, and reduced-8 is used as the testing set.

In this part, the MFFRand is tested on the S3DIS dataset (6-fold cross-validation) and Semantic3D dataset (reduced-8). Recent representative works are also evaluated on the same dataset. OA and mIoU are used as evaluation metrics.

Table 1. Quantitative results of different networks on the S3DIS (6-fold cross-validation).

	mIoU (%)	OA (%)	Ceil.	Floor	Wall	Beam	Col.	Wind.	Door	Table	Chair	Sofa	Book.	Board	Clut.
DPFA [46]	61.7	89.2	94.6	98.0	79.2	40.7	36.6	52.2	70.8	65.9	74.7	27.7	49.8	51.6	60.6
PointCNN [37]	65.4	88.1	94.8	97.3	75.8	63.3	51.7	58.4	57.2	71.6	69.1	39.1	61.2	52.2	58.6
PointWeb [36]	66.7	87.3	93.5	94.2	80.8	52.4	41.3	64.9	68.1	71.4	67.0	50.3	62.7	62.2	58.5
ShellNet [47]	66.8	87.1	90.2	93.6	79.9	60.4	44.1	64.9	52.9	71.6	84.7	53.8	64.6	48.6	59.4
Liu et al. [48]	68.3	88.6	93.5	96.0	81.5	42.6	46.3	61.0	74.1	67.4	82.7	63.5	59.5	56.6	62.9
MuGNet [49]	69.8	88.5	92.0	95.7	82.5	64.4	60.1	60.7	69.7	82.6	70.3	64.4	52.1	52.8	60.6
RandLA-Net [15]	70.0	88.0	93.1	96.1	80.6	62.4	48.0	64.4	69.4	69.4	76.4	60.0	64.2	65.9	60.1
KPConv [18]	70.6	-	93.6	92.4	83.1	63.9	54.3	66.1	76.6	57.8	64.0	69.3	74.9	61.3	60.3
MFFRand (ours)	71.1	88.9	93.9	94.4	83.6	64.8	54.7	62.7	68.2	71.7	79.3	65.5	64.2	58.8	63.1

presents the quantitative segmentation results of different networks on the S3DIS dataset with 6-fold cross-validation. MFFRand has achieved a mIoU metric of 71.1%, which is better than other methods, as shown in the table. Additionally, the best segmentation results are obtained on categories such as the wall, beam, column, table, and clutter. The experimental results demonstrate that our method is based on multi-scale feature fusion, and the multi-loss supervision performs effectively in 3D point cloud semantic segmentation and is superior to other methods.

Table 2. Quantitative results of different networks on the Semantic3D (reduced-8).

	mIoU (%)	OA (%)	Man-Made	Natural	High Veg	Low Veg	Building	Hard Scape	Scanning Art	Cars
SEGCloud [7]	61.3	88.1	83.9	66.0	86.0	40.5	91.1	30.9	27.5	64.3
MSDeepVoxNet [50]	65.3	88.4	83.0	67.2	83.8	36.7	92.4	31.3	50.0	78.2
PointConv [51]	69.2	91.8	92.2	79.2	73.1	62.7	92.0	28.7	43.1	82.3
ShellNet [47]	69.3	93.2	96.3	90.4	83.9	41.0	94.2	34.7	43.9	70.2
PointGCR [52]	69.5	92.1	93.8	80.0	64.4	66.4	93.2	39.2	34.3	85.3
GACNet [53]	70.8	91.9	86.4	77.7	88.5	60.6	94.2	37.3	43.5	77.8
RandLA-Net [15]	72.7	92.7	96.4	83.9	85.1	40.1	94.8	46.4	61.4	73.8
KPConv [18]	72.8	92.8	92.9	88.6	82.4	42.4	93.2	38.5	64.5	80.1
MFFRand (ours)	74.8	93.7	97.2	91.9	84.2	47.8	94.0	38.8	66.6	77.8

presents the quantitative segmentation results of different networks on the Semantic3D dataset (reduced-8). In terms of mIoU and OA, MFFRand is significantly better than the compared methods. In addition, MFFRand has achieved very good results in four categories, including man-made, natural, etc. The visualization of the prediction results on the testing set is shown in Figure 5.

4.2. Evaluation of Time Consumption

SemanticKITTI [54] dataset was developed by a research group at the University of Bonn, Germany in 2019, which is a semantic segmentation dataset for large 3D point clouds of outdoor streets based on radar sequence scene understanding. The dataset contains 21 sequences, including 19 categories such as cars, bicycles, motorcycles and so on. Sequences 00–07 and 09 of SemanticKITTI dataset are used as the training set; sequence 08 is used as a validation set and sequences 11–21 are used as the testing set.

Different from the complete street scenes in the Semantic3D dataset, SemanticKITTI is a large-scale road sequence scene point clouds that is collected by vehicle-mounted LiDAR. In SemanticKITTI, we could evaluate the network by the number of that frames processed per second, so the SemanticKITTI is used to evaluate the efficiency of MFFRand in this paper. The experiments are conducted on sequence 19, which has a total of 4981 frames of data. The SPVNAS [55] and RandLA-Net [15] are also evaluated as contrast experiments on the sequence 19. Processing time (seconds) is used as evaluation metrics. The quantitative results of the experiments are shown in

Table 3. The inference time of different approaches (Tested on sequence 19 of the SemanticKITTI dataset).

	Total Time (seconds)
SPVNAS	617.33 (8.07 frame/s)
RandLA-Net	169.5 (29.39 frame/s)
MFFRand (ours)	170.4 (29.23 frame/s)

. According to the results of experiments, the MFFRand has almost no change in time consumption, and still maintains a great advantage.

4.3. Ablation Study

To further validate the contribution of each module in MFFRand, the following ablation study was conducted on the Semantic3D dataset:

• Remove multi-loss supervision module. The better training of local structure in MFFRand could be achieved by the multi-loss supervision module. With the removal of the multi-loss supervision module, the training of the entire network is controlled by the loss function of the deepest encoder-decoder structure.
• Remove multi-feature fusion module (& multi-loss supervision module). This module enables the feature of encoder-decoders could be fused more sufficiently, bridging the semantic gap between the high and low layers effectively. With the removal of the multi-feature fusion module, the encoder and the decoder sub-networks are directly connected to each other through the skip connections.

The ablation study of two modules is conducted as shown in

Table 4. Results of ablated network on the Semantic3D (reduced-8).

	mIoU (%)
MFFRand	74.8
Remove multi-loss supervision module	73.8
Remove multi-scale feature fusion module & multi-loss supervision module (RandLA-Net)	72.7

. The experimental results demonstrate that the application of the multi-scale feature fusion module could help improve the segmentation effect significantly, and the mIoU increase by 1.1 on the basis of RandLA-Net. Meanwhile, the multi-loss supervision module also has a gain effect on the semantic segmentation of the network, with another 1.0 mIoU improvement on the basis of the multi-scale feature fusion module.

4.4. Evaluation of Output at Different Network Depths

To verify the semantic segmentation performance of MFFRand at different depths, we conduct experiments on MFFRand and RandLA-Net, respectively. The

Table 5. Influence of the number of network layers on inference results.

Number of Network Layers	mIoU of RandLA-Net (%)	mIoU of MFFRand (%)
3	53.03	53.73
4	67.63	67.63
5	72.70	74.07
6	72.70	74.80

shows the results of the experiment and performance comparison. Compared with RandLA-Net, there is an absolute advantage in inference results of 3-6 layers network of MFFRand.

In addition, in order to obtain the inference results of RandLA-Net (3–6 layers), the RandLA-Net with different depths needs to be trained separately and then the inference is conducted. Therefore, for RandLA-Net, four times of training is performed. Since the multi-scale feature fusion module in MFFRand consists of multiple encoder–decoder structures, the MFFRand only needs to be trained once, and then the inference network is connected to the end of each decoder in trained MFFRand separately. By this way, the four inference results could be obtained on MFFRand with different depths, as shown in Figure 6. Therefore, this is an advantage of MFFRand in network training, which can obtain segmentation results of different layers of network with a one-time training process.

5. Conclusions

MFFRand is proposed in this paper to handle the problem that high and low levels feature information could not be utilized sufficiently. The proposed block mainly consists of two modules, including the multi-scale feature fusion module and the multi-loss supervision module. The multi-scale feature fusion module is introduced to effectively fuse the semantic information between feature maps of different levels, which consists of a dense feature fusion network by concatenating several different levels of encoder–decoder structures. For multi-loss supervision module, the sub-losses are connected, respectively, to the end of each decoder to control the training of the local structure. Quantitative experiments and ablation studies are conducted on the S3DIS and Semantic3D dataset, which demonstrate the effectiveness of MFFRand in 3D point cloud semantic segmentation and the contribution of each module in MFFRand.

The method proposed in this paper improves the performance of semantic segmentation, but there is no breakthrough in efficiency. In the future, we will further explore general and effective methods for improving performance and investigate which cross-layer connections and losses in MFFRand are critical. Moreover, we will extend our method to end-to-end 3D panoptic segmentation and 3D object detection on large-scale point clouds.