GN-CNN: A Point Cloud Analysis Method for Metaverse Applications

Abstract

Metaverse applications often require many new 3D point cloud models that are unlabeled and that have never been seen before; this limited information results in difficulties for data-driven model analyses. In this paper, we propose a novel data-driven 3D point cloud analysis network GN-CNN that is suitable for such scenarios. We tackle the difficulties with a few-shot learning (FSL) approach by proposing an unsupervised generative adversarial network GN-GAN to generate prior knowledge and perform warm start pre-training for GN-CNN. Furthermore, the 3D models in the Metaverse are mostly acquired with a focus on the models’ visual appearances instead of the exact positions. Thus, conceptually, we also propose to augment the information by unleashing and incorporating local variance information, which conveys the appearance of the model. This is realized by introducing a graph convolution-enhanced combined multilayer perceptron operation (CMLP), namely GCMLP, to capture the local geometric relationship as well as a local normal-aware GeoConv, namely GNConv. The GN-GAN adopts an encoder–decoder structure and the GCMLP is used as the core operation of the encoder. It can perform the reconstruction task. The GNConv is used as the convolution-like operation in GN-CNN. The classification performance of GN-CNN is evaluated on ModelNet10 with an overall accuracy of 95.9%. Its few-shot learning performance is evaluated on ModelNet40, when the training set size is reduced to 30%, the overall classification accuracy can reach 91.8%, which is 2.5% higher than Geo-CNN. Experiments show that the proposed method could improve the accuracy in 3D point cloud classification tasks and under few-shot learning scenarios, compared with existing methods such as PointNet, PointNet++, DGCNN, and Geo-CNN, making it a beneficial method for Metaverse applications.

1. Introduction

The point cloud is a popular 3D shape representation. The 3D point analysis has many important applications in the Metaverse, such as classifying [1] and completing [2] the acquired 3D models to be used. With the rapid development of the Metaverse, a large number of new 3D point cloud models are generated and acquired. Many models are novel, have never been seen before, or come unlabeled. Such limited information brings challenges to data-driven 3D point cloud analyses, such as classification, because they strongly rely on sufficiently labeled training data [3,4]. To tackle this problem, we propose a convolution neural network, GN-CNN, with a few-shot learning approach by utilizing a generative adversarial network GN-GAN to extract prior knowledge for augmenting the information.

As shown in Figure 1, we pre-trained the GN-GAN to a warm start [5,6] GN-CNN, which is the 3D point cloud analysis network. GN-GAN obtains the global point cloud feature (experience) vector. Then the vector is plugged in as prior knowledge into the backbone network of GNConv, which is used as the convolution-like operation in GN-CNN. Using the original and downsampled input helps to learn multi-scale point cloud features.

Furthermore, the 3D models in the Metaverse are usually acquired with a focus on the model’s visual appearances instead of the exact point positions, for example, for driving [7] and indoor navigation [8], more attention is usually paid to the model’s appearance to make sure the models look good, instead of the model actual accuracy, which is usually the focus of computer-aided design (CAD). Based on this, unlike existing (few-shot) learning methods [9,10], we propose unleashing the model appearance to further augment the information. The model’s local surface variance and normal information determine how light is reflected, and how the model’s appearance is perceived by the viewers. Therefore, they are utilized in the proposed pipeline: First, in the encoder of the encoder/decoder framework of the GN-GAN generator, we introduced a graph convolution-enhanced combined multilayer perceptron operation (CMLP [11]), namely GCMLP, by adding several graph convolution layers. Second, in GN-CNN, we utilize local neighborhood normal information.

Different from the existing graph convolution-based methods, such as DGCNN [12] and GeoConv [13], we used graph convolution to enhance CMLP [11], which does not handle geometric relationships well. By integrating the capability of CMLP in learning different levels of feature information and the capability of graph convolution in learning local spatial connection relationships, the local surface variance information can be better learned. Together with the normal information, the appearance of the model can be better captured.

In this paper, we focus on classification and few-shot learning (GN-CNN) as well as reconstruction (GU-GAN) tasks of 3D point cloud analysis. Our contributions are summarized as the following:

(1)

We propose a convolution neural network framework GN-CNN that is suitable for Metaverse applications. We propose a few-shot learning approach using an unsupervised generative adversarial network GU-GAN to warm start GN-CNN by generating prior knowledge.

(2)

We propose the graph-enhanced combined multilayer perceptron operation (GCMLP) in the GN-GAN encoder and normal-aware graph convolution operation GNConv to better capture the model appearance information.

(3)

Based on evaluations, state-of-the-art 3D point cloud classification accuracy and few-shot learning accuracy on ModelNet40 and ModelNet10 are achieved.

Please refer to the table of abbreviations for the main abbreviations used in this paper.

2. Related Work

2.1. Deep Learning on 3D Point Cloud Analysis

Unlike the 2D image, the 3D point cloud does not have a straightforward regular grid (pixels), so it cannot use CNN directly. Therefore, some early work converts point clouds to other regularized representations, such as volumetric 2D views [14,15,16,17,18,19] or representations [20,21,22,23,24]. Although voxelization-based methods and view-based methods can achieve good results on point cloud analysis tasks, the resolution of voxel grids always seriously affects the efficiency of the neural network, and view-based methods always consume much time in data preparation. PointNet [1] and PointNet++ [25] first use the 3D coordinates of the point as direct input. Some works [26,27,28,29] also input point clouds directly and construct their own networks based on PointNet [1] and PointNet++ [25]. However, they extract features on each point independently, which cannot easily learn geometric relationships among points of the point cloud.

2.2. Graph-Based Works on Point Cloud Analysis

As a milestone in the use of graph convolution on 3D point clouds, DGCNN [12], dubbed EdgeConv, uses the k-nearest neighbor algorithm (k-NN) to construct a local neighborhood graph and update the graph structure on deeper latent layers. With the stacking of EdgeConv, DGCNN can learn the global features of the 3D point cloud. However, EdgeConv ignores the directional information of the edges formed by the local neighborhood. RSCNN [30] learns the weight matrix by constructing a local neighborhood graph and then multiplies it with the features in the network. GeoConv [13] builds a graph structure and decomposes the edge features in latent layers according to the orthogonal basis. GeoConv uses the directional information to enrich the geometric properties in the graph convolution and achieved better results than EdgeConv. However, GeoConv still only pays attention to the spatial connection relationship between neighboring points when constructing the graph. Therefore, the obtained directional information involves the spatial relationship between the points, the local surface information in different levels (i.e., low and high levels), and normal information are not addressed.

2.3. Point Cloud Reconstruction and Completion Based on Deep Learning

Some works use GAN-based methods on point cloud reconstruction and completion. The L-GAN [31] is the first to use deep learning methods for point cloud completion in the 3D field, but since L-GAN does not focus on the point cloud completion task, the results are not ideal. Some point cloud completion works pay more attention to real-time performances, such as RL-GAN-Net [2], but the accuracy is not high. Some other works plug graph convolution into the GAN-based networks. Valsesia et al. [32] used graph convolution for unsupervised point cloud reconstruction, defining the graph as the output of the generator. Huang et al. [11] further studied 3D point cloud completion and proposed PF-Net based on unsupervised learning, using the pyramid structure to construct the decoder. PF-Net obtains high-precision 3D point cloud completion results by jointly training both the generator and discriminator.

2.4. Summary

Please refer to

Table 1. Comparison with other methods.

	Related Work Features	Our Method
PointNet [1], PointNet++ [25]	Focused on individual point features	Learning geometric relationships between points
RSCNN [30]	Defining convolutional kernels	Not defining convolutional kernels, but using graph convolution
DGCNN [12]	Lacking directional information	Utilizing normal information
GeoConv [13]	Focused on neighboring points	Considering local surface information in different levels and normal information
L-GAN [31]	Not targeted for the point cloud completion	Applicable for reconstruction
RL-GAN-Net [2]	For real-time	Not for real-time
PF-Net [11]	Using GAN with pyramid structure	Using two different resolutions in GN-GAN for warm start

for a comparison with other methods.

Current few-shot learning methods [9,10,33] of 3D point clouds mostly learn semantic information from the support point sets and then generalize it to analyze the query point sets. Different from the existing methods, we use a GAN-based and appearance-aware approach. Our methods can improve the accuracy of results compared to the voxel-based and non-graph-based methods; please refer to the results section.

3. Method

This section presents the proposed GN-CNN and GN-GAN and the core operations in the two networks. We will first introduce our overall architecture, GN-CNN, and warm start strategy, then describe the structure of GCMLP and GN-GAN.

3.1. The Overall Architecture and GN-CNN

The overall network architecture includes GN-CNN, which is the backbone network for point cloud classification and few-shot learning tasks, as well as GN-GAN, which aims at warm starting GN-CNN. GN-GAN can also be used for point cloud reconstruction tasks.

Our GN-CNN pipeline is inspired by Geo-CNN [11], and the network structure is also divided into two branches, as shown in Figure 2. The first branch of GN-CNN is PointNet++, using PointNet to extract features layer by layer. Each layer uses multilayer perceptron (MLP) and max pooling to abstract features to high dimensions, and the three layers of PointNet encode features into dimensions [64, 128, 384], and finally, we obtain a feature size of $N\times 384$, where N is the number of points.

In the second branch of GN-CNN, the input is the same as the first branch. First, the input point cloud feature with a size of $N\times 6$ into an MLP to map the size of the feature to $N\times 64$, and then pass a layer of GNConv to obtain the feature with a size of $N\times 128$, and then pass an MLP to obtain the feature with a size of $N\times 256$ and follow a layer of GNConv to obtain the feature with a size of $N\times 512$. The $N\times 384$ feature obtained by the first branch is connected to obtain a feature of size $N\times 896$, and then input into a layer of GNConv to obtain a feature of size $N\times 768$, and finally passes through an MLP to obtain a feature of size $N\times 1920$. Next, we use max pooling to obtain a vector with length [1920]. The vector with length [1920] is the point cloud global feature representation vector.

Next, we use the prior knowledge extracted by the GN-GAN encoder with the same length of [1920] for feature aggregation with the point cloud global feature vector extracted by GN-CNN encoder, and GN-CNN is warm-started. The aggregation function adopts an element-wise sum. The GN-CNN decoder consists of four FC layers with the vectors of output lengths [1024, 512, 256, C], where C is the number of classes in the point cloud classification tasks.

3.2. Warm Start

The warm start techniques are often used in the field of deep learning [5,6,34] and reinforcement learning. The warm start is to initialize the network parameters to make networks have a particular experience at the beginning of training. This method can accelerate the convergence of the neural network. When the data size is limited, the neural network can make corresponding judgments based on the experience knowledge to obtain better few-shot learning results. Our warm start strategy is shown in Figure 3. $GNCN{N}_E$ is the encoder of GN-CNN, $GNCN{N}_D$ is the decoder of GN-CNN, $F_v$ is the global feature representation extracted by the GN-CNN encoder, and $E_v$ is the experience knowledge. We aggregate $E_v$ and $F_v$, and then input them into the decoder of GN-CNN. $R_{out}$ is the output of the GN-CNN decoder. The output $R_{out}$ of the GN-CNN decoder is defined as:

$$\begin{array}{cc}\hfill R_{out}& =GNCN{N}_D\left(A(F_v,E_v)\right)\hfill \\ \hfill & =GNCN{N}_D\left(A(GNCN{N}_E\left(P\right),G{N}_E\left(P\right))\right),\hfill \end{array}$$

(1)

where $G{N}_E\left(\right)$ represents the encoder of GN-GAN, P is a point cloud, $A\left(\right)$ is the aggregation function, we use element-wise add as the aggregation function.

3.3. GCMLP

CMLP [11] takes the maximum value in each channel on low-level and high-level through MLP and then concatenates the vector after the max pooling. CMLP can integrate low-level and high-level features to learn multi-level information better, but it does not handle the geometric relationship information among points.

Since graph convolution can handle geometric relationships among points, especially for the local neighborhood learning of point clouds, we propose GCMLP, as shown in Figure 4, to improve CMLP performance by using a graph convolution layer named GMLP. In GCMLP, three MLP layers in CMLP are replaced with GMLP layers, while still concatenating the low-level and high-level feature vectors after max pooling. For a GMLP layer, we first use k-NN for each point $p_i(i=1,2,\dots ,N)$ within N points to find the k nearest neighbors $p_j(j=1,2,\dots ,k)$ and take k neighboring points $p_j$ as source points and $p_i$ as target points and construct an edge $p_j-p_i$ (including the self-loop from $p_i$ to $p_i$). Finally, we concatenate $[p_j-p_i,p_i]$, and then go through the MLP layer.

GCMLP first uses an MLP to encode the dimensions of the original point cloud to [64], then uses three GMLP layers to encode the dimensions to [64, 128, 256], and finally is followed by two MLP layers to encode the dimensions to [512, 1024]. We max pool the output of the last four layers and concatenate the four vectors. The lengths of the four vectors are [128, 256, 512, 1024].

3.4. GN-GAN and Loss Function

The overall structure of GN-GAN is shown in Figure 5. We use iterative farthest point sampling (IFPS) to downsample the input point cloud to obtain two resolutions of the point cloud. The encoder of the GN-GAN generator takes both two resolutions as input. The number of points in the high-resolution point cloud is N, and the number in the low-resolution point cloud after down-sampling is $N_2=\frac{N}{K}$. The point clouds with two resolutions are passed through two GCMLPs, respectively, and the generator reconstructs two point clouds with the same resolutions as respective inputs. We concatenate the two feature vectors obtained through GCMLP and input them into an MLP to encode the dimensions of the global feature vector $F_V$ to 1920. CD is the Chamfer distance to evaluate the difference between the reconstructed point cloud and ground truth.

The structure of the decoder in the GN-GAN generator is shown in Figure 6, where $F_V$ is inputted into the decoder. The red box in Figure 6 is $F_V$. We first use two fully connected (FC) layers to encode the dimensions of $F_V$ to [1024, 256]. The [256] vector is used to obtain the vector $F_1$ with the size of $3·N_2$ through an FC layer, and $F_1$ is reshaped to the size $N_2\times 3$ to obtain $P_L$, which is the low-resolution reconstructed point cloud. Next, we use the [1024] vector to obtain $F_2$ which size is $N·N_2$ through the FC layer, reshape $F_2$ into $F_3$ of size $N\times N_2$, and input $F_3$ into a convolution layer to obtain $F_4$ of size $N_2\times \frac{3N}{N_2}$. We reshape $F_4$ to $F_5$ of size $N_2\times \frac{N}{N_2}\times 3$. At this point, we expand $P_L$ to $N_2\times 1\times 3$, and add it to $F_5$. Finally, we reshape it to obtain the high-resolution reconstructed point clouds $P_H$ of size $N\times 3$.

The reconstruction loss of GN-GAN is defined using the Chamfer distance. Chamfer distance is the average nearest squared distance between two point clouds. Define the ground truth point cloud as $P_{real}=\{p_1,p_2,\dots ,p_N\}$, and the reconstructed point cloud is $P_{fake}=\{p_1^{\prime },p_2^{\prime },\dots ,p_N^{\prime }\}$, then Chamfer distance $L_{CD}$ between $P_{fake}$ and $P_{real}$ is divided into two parts $C{D}_1$ and $C{D}_2$:

$$C{D}_1=\frac{1}{P_{real}}\sum _{i=1}^N\underset{j\in [1,N]}{min}\parallel p_i-p_j^{\prime }\parallel {}_2^2$$

(2)

$$C{D}_2=\frac{1}{P_{fake}}\sum _{j=1}^N\underset{i\in [1,N]}{min}\parallel p_i-p_j^{\prime }\parallel {}_2^2.$$

(3)

Finally, $L_{CD}$ is defined as:

$$L_{CD}=C{D}_1+C{D}_2.$$

(4)

The reconstruction loss of GN-GAN is defined as:

$$L_{gen}=L_{C{D}_h}+\alpha L_{C{D}_l},$$

(5)

where $L_{C{D}_h}$ is the Chamfer distance between the high-resolution reconstructed point cloud and the ground truth, and $L_{C{D}_l}$ is the Chamfer distance between the low-resolution reconstructed point cloud and the corresponding ground truth, and $\alpha$ is a hyperparameter.

The structure of the discriminator is shown in Figure 5, and we divide the adversarial loss of the GN-GAN discriminator into two parts, which correspond to inputting the ground truth point cloud into the discriminator and inputting the reconstructed point cloud into the discriminator, respectively. The generator mapping process is defined as $G(·)$ and the discriminator mapping process is defined as $D(·)$, then the adversarial loss is defined as:

$$L_{adv}=\sum _{1\le i\le N}log\left(D\left(x_i\right)\right)+\sum _{1\le i\le N}log\left(D\left(G\left(x_i\right)\right)\right),$$

(6)

where $x_i\in X$, $i-1,2,\dots ,N$. The joint loss of GN-GAN is defined as:

$$L={\lambda }_{gen}{L}_{gen}+{\lambda }_{adv}{L}_{adv},$$

(7)

where ${\lambda }_{gen}$ and ${\lambda }_{adv}$ are weights to balance two parts of joint loss, which satisfy:

$${\lambda }_{gen}+{\lambda }_{adv}=1.$$

(8)

3.5. GNConv

GNConv is designed to capture the visual variance of the local surface formed by the point cloud and augment the information with it. Since the normal of a point can depict the information and appearance of the local neighborhood, so we use normals. As shown in Figure 7, $d_{normal}$ represents the distance between the two normals, reflecting the visual variance of the surface. Conceptually, when $d_{normal}$ is large, the local surface appearances vary violently, which means there are more features (important information).

Inspired by GeoConv [13], GNConv includes the distance of normals between the center point and the neighboring points on the local neighborhood graph in each iteration. Define the point cloud as $P=\{p_1,p_2,...,p_N\}$, where $p_i=\{x_i,y_i,z_i\}$. Taking each point of P as the center point, take the spherical neighborhood $S\left(p_i\right)=\{p_j|\parallel p_i-p_j\parallel \le r\}$ with radius r, and the normal set is $N_P=\{n_{p_1},n_{p_2},\dots ,n_{p_N}\}$. As the iteration progresses, gradually increase r to obtain a larger receptive field. Define $X{}_{p_i}^l\in R^C$ as the feature of $p_i$ obtained after the lth iteration of GNConv, then the definition of GNConv is as follows:

$$X{}_{p_i}^l=W_{c}X{}_{p_i}^{l-1}+W_e{\sum }_{p_j\in S\left(p_i\right)}\frac{f_d(p_i,p_j,r)f_n(p_i,p_j)}{f_d(p_i,p_j,r)}.$$

(9)

As shown in Equation (9), the feature of $p_i$ on the lth layer has two parts. The first part is the output of the previous layer$X{}_{p_i}^{l-1}$ multiplied by the center point weight $W_c$ to extract the center point feature, and the second part is the edge property between the center point and neighboring points in the spherical neighborhood. The corresponding feature is multiplied by the weight $W_e$ and $W_e$ to enlarge the feature dimension to be the same as the dimension of the first part. In the second part, $f_n(p_i,p_j)$ is defined as:

$$f_n(p_i,p_j)=W_{n}n(p_i,p_j)+\sum _{\overrightarrow{b}}co{s}^2\left({\theta }_{\overrightarrow{b}}\right)W_{\overrightarrow{b}}X{}_{e_{ij}}^{l-1}.$$

(10)

Before each iteration, we take each point $p_i$ as the center of the local spherical neighborhood and determine the neighborhood $S\left(p_i\right)$ with a radius of r. The normal edge feature $n(p_i,p_j)$ between the center point and the neighboring points in the spherical neighborhood is defined as:

$$n(p_i,p_j)=n_{p_j}-n_{p_i},$$

(11)

where $p_j\in S\left(p_j\right)$, $W_n$ is the weight of normal edge features. The edge feature $X{}_{e_{ij}}^{l-1}$ is defined as:

$$X{}_{e_{ij}}^{l-1}=X{}_{p_j}^{l-1}-X{}_{p_i}^{l-1}.$$

(12)

For each spherical neighborhood, we construct an orthogonal basis with $p_i$ as the origin and decompose the edge features along with the orthogonal basis, and $cos\left({\theta }_{\overrightarrow{b}}\right)$ is the cosine between each component and the corresponding orthogonal basis. $W_{\overrightarrow{b}}$ is the weight for feature extraction of the components along with three orthogonal bases. After the feature components of the three edges are mapped to the high-dimensional feature space, we aggregate the components by $cos\left({\theta }_{\overrightarrow{b}}\right)$. Finally, we aggregate the edge feature $X{}_{e_{ij}}^{l-1}$ and the normal edge feature $n(p_i,p_j)$ as shown in Equation (10).

In Equation (9), $f_d(p_i,p_j,r)$ is the weight function, and the weight is dynamically changed according to the distance between $p_j$ and $p_i$, which is defined as:

$$f_d(p_i,p_j,r)=(r-\parallel p_i-p_j{\parallel )}^2.$$

(13)

4. Experiments

4.1. Implementation Details

In our experiments, the GPU used in the classification experiment and the ablation experiment was NVIDIA GeForce GTX 2080ti; the GPU used in the few-shot learning experiment and the visualization of GN-GAN reconstruction results was NVIDIA GeForce GTX 1070.

In the 3D point cloud classification experiment and the few-shot learning experiment, the input 3D point cloud of GN-CNN contained 1024 points. The radii for three GNConv layers were 0.15, 0.3, and 0.6. The first branch and the second branch both used the Adam optimizer and the cosine annealing strategy. The training batch_size was 10, and the test batch_size was 8.

When pre-training GN-GAN, the input point cloud resolution was 1024 points and 64 points after IFPS downsampling with K = 16. The generator used the Adam optimizer and StepLR to update the parameters every 40 epochs. The discriminator also used the Adam optimizer and StepLR to update the parameters every 40 epochs. The training batch_size was 16, and the test batch_size was 8.

4.2. Dataset

Our experiments used ModelNet40 [20] and ModelNet10 [20]. For classification, we carried out experiments on both ModelNet10 and ModelNet40. The input point cloud resolution was 1024 points, and we used all samples from the training and test sets. For few-shot learning, we used ModelNet40 and randomly reduced 50%, 70%, 90%, and 95% of the total number of training samples; that is, using 50%, 30%, 10%, and 5% of the total number of training samples. We kept the number of testing samples unchanged, so the few-shot learning experiment used five training set sizes, including the original training set. We used ModelNet40 to pre-train GN-GAN for experience knowledge.

4.3. Classification

We used warm-start GN-CNN to perform classification experiments on ModelNet40 and ModelNet10. The experiment’s results are shown in

Table 2. Classification accuracy on ModelNet40.

Method	MA (%)	OA (%)
3DShapeNets [20]	77.3	84.7
VoxNet [21]	83.0	85.9
PointNet [1]	86.2	89.2
PointNet++ [25]	-	90.7
DGCNN [12]	90.2	92.9
DeepSets [35]	-	90.3
ECC [36]	83.2	87.4
SpiderCNN [37]	-	92.4
Kd-Net [38]	88.5	91.8
PCNN [39]	-	82.3
Geo-CNN [13]	91.1	93.4
Ours	90.5	93.0

and

Table 3. Classification accuracy on ModelNet10.

Method	OA (%)
3DShapeNets [20]	83.5
VoxNet [21]	92.0
MLH-MV [40]	94.8
KCNet [41]	94.4
SO-Net [42]	95.7
LP-3DCNN [43]	94.4
MHBN [19]	95.0
3DCapsule [44]	94.7
VIPGAN [45]	94.1
Point2Sequence [46]	95.3
Ours	95.9

, where MA is the mean per-class accuracy and OA is the overall accuracy.

The results on ModelNet40 are shown in Table 2. Using our method, MA is 90.5% and OA is 93.0%; OA increased by 3.8% compared with PointNet [1], and 2.3% compared with PointNet++ [25]. Through a warm start strategy, the OA of our method increased by 0.1% compared to the milestone work DGCNN [12] in the field of graph convolution. Note that our result did not reach the reported OA of 93.4% in the original Geo-CNN [13] when training 200 epochs using the same network structure and parameter settings, but it is slightly better than the OA of the reproduced Geo-CNN, which is 92.9% (the same as DGCNN). In our opinion, there are two main reasons: first, the reproduction process is different from the original paper, which causes a different training result, and second, the hardware and the number of epochs are different. Using more epochs, similar results as in the original paper should be achieved.

The OA of our method on ModelNet10 can reach 95.9%. As shown in Table 3, the OA increased by 12.4% compared with 3DShapeNets [20], and increased by 1.5% compared with KCNet. Compared with some work proposed in recent years, such as LP-3DCNN, 3DCapsule, VIPGAN, Point2Sequence, and MHBN, our result on ModelNet10 is the best. Moreover, on the official website of Princeton ModelNet, the overall accuracy ranking of the work performed on ModelNet10 for classification tasks is provided as the ModelNet benchmark leaderboard, and the performance of GN-CNN is ranked fifth.

4.4. Few-Shot Learning

Our few-shot learning experiment uses ModelNet40 and the results are shown in

Table 4. Few-shot learning accuracy based on ModelNet40.

Method	100%		50%		30%		10%		5%
	OA	MA	OA	MA	OA	MA	OA	MA	OA	MA
PointNet [1]	89.2	86.2	87.4	82.8	86.4	82.1	79.9	72.8	78.0	67.0
PointNet++ [25]	91.2	-	91.5	89.3	89.7	87.5	84.1	79.5	78.4	72.5
DGCNN [12]	92.9	90.2	89.3	84.0	87.1	79.6	82.5	73.4	79.3	70.2
Geo-CNN [13]	93.4	91.1	91.7	89.0	89.3	85.4	72.7	77.7	80.3	74.2
Ours	93.0	90.5	92.2	89.3	91.8	88.5	85.1	80.0	82.6	74.8

. Moreover, 100%, 50%, 30%, 10%, and 5% represent the five training set sizes after random sampling. Since the other four methods did not conduct few-shot learning, we used the other four methods to perform few-shot learning under the same conditions as the GN-CNN experimental environment.

We can see from Table 4 that our GN-CNN on the original training set can achieve similar results as DGCNN and Geo-CNN, but after the size of the training set is reduced, the OA and MA of GN-CNN are better than Geo-CNN. When the training set size reduces to 30%, the OA of GN-CNN classification increases by 2.5% compared with Geo-CNN, and the MA increases by 3.1%. Moreover, since the number of samples in the ModelNet40 training set is 9843, and the number of samples in the test set is 2468, there are only about 500 samples in the training set when we reduce the size to 5% of the original training set, and the number of training samples only accounts for about 20% of the number of samples in the test set. In this case, compared with DGCNN, the OA of GN-CNN increased by 3.3%, and the MA increased by 4.6%.

4.5. Reconstruction Results of GN-GAN

To justify the prior knowledge extracted by the GN-GAN containing useful point cloud information, we compare the point cloud reconstructed by the decoder with the ground truth. We use the heatmap to visualize the comparison results, as shown in Figure 8. When the point is closer to blue, the difference between the reconstructed and real points is smaller.

As shown in Figure 8, although the reconstructed points deviate from the real points in areas with more details of the 3D point clouds, such as the tips of airplane wings, the neck of bottles, and the plant leaves, GN-GAN can reconstruct point clouds that are very close to the ground truth on the whole, which shows that the prior knowledge extracted by the encoder in the GN-GAN generator is helpful.

4.6. Ablation Study

The ablation study was conducted using ModelNet40. The training set size was 5% of the original. The results are shown in

Table 5. Ablation study based on ModelNet40.

Method	OA (%)
Baseline	80.3
Baseline (GNConv)	81.1
Baseline (GNConv) + CMLP	82.4
Baseline + GCMLP	81.6
GN-CNN	82.6

. Since our GN-CNN and GNConv are inspired by Geo-CNN and GeoConv, the Geo-CNN is chosen as the “Baseline”. To compare the performance of GNConv and GeoConv, in “Baseline (GNConv)”, the GeoConv is replaced with GNConv. Compared with “Baseline”, GN-CNN not only uses GNConv to extract features but also includes prior knowledge from the GN-GAN generator. The encoder of the generator in GN-GAN relies on GCMLP to extract features. To compare the performance of GCMLP and CMLP, in “Baseline (GNConv) + CMLP”, the GCMLP of GN-GAN is replaced with CMLP, and the prior knowledge extracted by CMLP is plugged into “Baseline (GNConv)”. In “Baseline + GCMLP”, the prior knowledge obtained through GN-GAN is plugged into Geo-CNN.

Comparing the OA of “Baseline” and “Baseline (GNConv)”, it is shown that GNConv has a stronger extracting ability than GeoConv. Comparing “Baseline (GNConv)” and “Baseline (GNConv) + CMLP”, as well as “Baseline” and “Baseline + GCMLP”, it is shown that prior knowledge helps improve the accuracy of few-shot learning. Comparing “Baseline (GNConv) + CMLP” and our GN-CNN, it is shown that GCMLP has a stronger feature-extracting ability than CMLP.

To further compare the performance of GCMLP and CMLP, two layers of GCMLP in GN-GAN are replaced with CMLP and named GN-GAN (CMLP). As shown in

Table 6. Comparison with GCMLP and CMLP.

Method	OA (%)
CMLP	0.002403
GCMLP	0.002281

, we use ground truth and predicted (GT-pred) error to evaluate the performance. GT-pred error computes the average squared distance from each point in the ground truth to its closest in the reconstructed point cloud. We observed that when using GCMLP to extract features, the result is smaller, which means that the reconstruction result of GN-GAN is better than that of GN-GAN (CMLP). This indicts that GCMLP has a stronger feature-extracting ability.

5. Conclusions

In this paper, we propose a convolutional neural network framework GN-CNN with normal-aware convolution operation GNConv, which is suitable for Metaverse applications. To solve the problem of few-shot learning of point clouds, we propose an unsupervised generative adversarial network GN-GAN, based on graph convolution-enhanced multilayer perceptron operation GCMLP for point cloud reconstruction (for extracting experience knowledge to warm start GN-CNN). The performance was evaluated on ModelNet40 and ModelNet10.

It is still challenging to reconstruct all local detailed features and the accuracy still has room to improve. In the future, we plan to study more geometric characteristics of the point cloud to improve the feature extraction performance and explore more prior knowledge of point cloud geometric information to promote the learning of the backbone network.