Deep Spatial-Temporal Neural Network for Dense Non-Rigid Structure from Motion

Abstract

Dense non-rigid structure from motion (NRSfM) has long been a challenge in computer vision because of the vast number of feature points. As neural networks develop rapidly, a novel solution is emerging. However, existing methods ignore the significance of spatial–temporal data and the strong capacity of neural networks for learning. This study proposes a deep spatial–temporal NRSfM framework (DST-NRSfM) and introduces a weighted spatial constraint to further optimize the 3D reconstruction results. Layer normalization layers are applied in dense NRSfM tasks to stop gradient disappearance and hasten neural network convergence. Our DST-NRSfM framework outperforms both classical approaches and recent advancements. It achieves state-of-the-art performance across commonly used synthetic and real benchmark datasets.

1. Introduction

Non-rigid structure from motion (NRSfM) recovers the non-rigid 3D structure and camera pose from given monocular 2D point trajectories. It is ill-posed by its inherent ambiguities, and requires motion and deformation cues as well as prior assumptions to achieve 3D reconstruction. Moreover, it can be further subdivided into sparse NRSfM and dense NRSfM. Dense NRSfM is more complex and advances more slowly because it must reconstruct thousands of times more feature points compared with the sparse NRSFM. Most algorithms that address the dense NRSfM issue [1,2,3,4,5] extend the sparse reconstruction approach, and generally achieve unsatisfactory results. Recent developments have been made in dense NRSfM-specific algorithms [6,7,8,9]. Additionally, gradually introducing the neural network into the NRSfM area has provided new insights for resolving the dense NRSfM issue, despite the current primary focus on tackling sparse NRSfM.

The first 3D reconstruction of dense non-rigid objects from motion using an unsupervised neural network was completed by V. Sidhu [9] in 2020, and he gave the NRSfM network framework its moniker, neural non-rigid structure from motion (N-NRSfM). He developed a neural network framework for fitting deformations, employed latent space constraints for dense NRSfM, and obtained excellent reconstruction results. However, the final results heavily rely on conventional algorithms that first derive shape priors because of the reconstruction requirement for a rigid three-dimensional prior. This severely restricts the neural network’s ability to learn, thereby rendering these approaches less robust.

This study presents a deep spatiotemporal NRSfM framework (DST-NRSfM) network structure to address the dense NRSfM problem. The proposed DST-NRSfM is trained by an entirely unsupervised drawing on the auto-encoder [10,11] principle, which uses layer normalization layers to both prevent gradient disappearance and accelerate neural network convergence; additionally, it uses temporal–spatial constraints to enhance the 3D reconstruction output accuracy. The complete network model is divided into two modules: shape estimation and rotation estimation. These two modules, respectively, derive the 3D structure and camera posture of the reconstructed object from an input 2D point trajectory matrix. The shape-estimation module abandons the shape prior and builds the most appropriate end-to-end network model for dense objects based on the properties of distinct datasets. In the rotation estimation module, considering the uniqueness of the NRSfM problem, we implemented a rotation matrix corrector after the rotation estimation network. This corrector will further orthogonally limit the resulting rotation matrix and eliminate its potential reflection ambiguity. Experiments demonstrated that our proposed method outperformed rival methods and achieved cutting-edge outcomes on the most popular dense datasets. What follows is a summary of the primary contributions of this study.

• We propose a novel deep neural network framework called DST-NRSfM to address the dense NRSfM challenge, which is superior for reconstructing objects with significant deformations because it does not require shape priors.
• We introduce the weighted spatial constraint to minimize the reconstruction error, improve mesh quality, and ensure that the reconstructed item is closer to the original shape. The second set of comparison experiments in Section 4.2 demonstrates the success of the weighted spatial constraint.
• The proposed technique achieves state-of-the-art performance on the most popular dense benchmark datasets. Section 4.3 provides evidence to support this hypothesis.

The remainder of this study is organized as follows. First, Section 2 discusses related work. Section 3 introduces problem settings, reviews the NRSfM problem and subsequently provides a detailed description of our network architecture. Section 4 details the experiments, presents the results, and analyzes the model’s overall performance. Section 5 summarizes the proposed approach.

2. Related Works

NRSfM. Utilizing prior knowledge of shape and camera motion is a popular method for solving the inherently ambiguous NRSfM problem. Note that each frame’s 3D shape is assumed to be composed of a few shape bases [12] or trajectory bases [13,14], and methods, such as probabilistic principal component analysis (PPCA) [15], metric constraints [16], complementary space models [17], block sparse dictionary learning [18], or force-based models [19], are employed to enhance the outcomes of the non-rigid 3D reconstruction produced via factoring. However, because the ideal number of bases is typically unpredictable (different bases occur in various sequences), this strategy is not robust. Recently, NRSfM has advanced significantly as the BMM algorithm [20], isometric priors [21], elastic priors [22], local-rigidity prior [23,24], space–time smooth constraints [25], variational methods [7], and tensor-based models [26] have been successively proposed.

Dense NRSfM. Reconstruction is more challenging for dense NRSfM, which is an extension of the NRSfM problem, owing to the significantly greater number of key points that must be restored. To address this issue, Collins et al. [27] suggested a segmentation approach that first uses a straightforward local model to reconstruct local patches and then stitches all local reconstruction results to create a smooth surface. However, sewing local reconstruction results together presents a hurdle. A template-based approach for reconstructing dense surfaces was proposed by Russell et al. [28] in 2012; however, in this approach, an appropriate 3D template must first be selected. The initial method for sparse reconstruction [1,2,3,4,5] was expanded to address the dense NRSfM problem because of the proliferation of NRSfM problem-solving techniques; however, the outcome remains subpar. Currently, an increasing number of studies are focused on solving the dense NRSfM problem; these include the following: (1) Agudo et al. [6] proposed modeling time-varying shapes using a probabilistic linear subspace of modal shapes obtained from continuum mechanics. (2) Garg et al. [7] treated the dense NRSfM problem as a global variational energy minimization problem by estimating the per-frame and camera motion matrices. (3) Kumar et al. [8,29,30] assumed that non-rigid deformations are collections of local linear subspaces and time, and used Grassmann manifolds to represent shapes as a set of smooth low-dimensional surfaces embedded in higher-dimensional Euclidean space, thus providing a practical solution and new theoretical insights for dense NRSfM. (4) Sidhu et al. [9] completed 3D reconstruction of dense non-rigid motion with an unsupervised neural network; excellent reconstruction results were obtained with the auto-decoder deformation model and latent-space constraints, but their method is not robust because it strongly depends on the mean shape.

NRSfM based on neural networks. Although the application of neural networks in solving dense NRSfM problems is still new, they are crucial in solving numerous NRSfM problems. In 2018, Cha et al. [31] developed a 3D unsupervised reconstruction network that could accurately predict the 3D results of omitted instances from the same category, and successfully reconstructed the 3D results of instances in a certain object category from 2D features using an orthographic camera model. In 2019, Facebook presented the C3DPO algorithm [32], which introduced a fresh regularization technique and substituted neural networks for traditional matrix decomposition. The deep NRSfM and deep NRSfM++ algorithms were proposed by Lucy et al. [33,34] to interpret NRSfM as multilayer sparse coding. By switching out the traditional low-rank assumption [18,35] for hierarchical block sparsity, which is more durable and comprehensible than single-level block sparsity. In 2020, Park et al. [36,37], inspired by the Procrustean alignment scheme [38], created a Procrustean regression network with a loss function that can automatically identify the correct rotation matrix. In 2021, Zeng et al. [39] first introduced a residual recurrent network (RRN) and two new loss compositions: pairwise contrast and pairwise consistency losses. RRN alone can accurately reconstruct non-rigid shapes and further improve the reconstruction results with two new losses. Wang et al. [5] used the 3D depth auto-encoder framework as the NRSfM prior and successfully recovered the 3D shape from a 2D image group with no time sequence and large deformation. Recently, Deng et al. [40] introduced a new technique for applying subspace joint structures, thereby enabling end-to-end batch training, and proposed a sequence-to-sequence translation perspective for deep NRSfM.

Although neural-network-based NRSfM has advanced significantly in recent years, it has focused primarily on the 3D reconstruction of sparse objects. There is still room for the development of dense NRSfM based on neural networks.

3. Methods

This section points out the problem setting of this study, reviews the classical architectures for the NRSfM problem to better explain the design rationale and subsequently describes our method in detail.

3.1. Preliminary

Problem setup. A network framework capable of dense NRSfM with deep neural networks without shape priors is needed. Specifically, given a dense non-rigid 2D trajectory matrix $W\in {\mathbb{R}}^{2F\times P}$, which contains the 2D coordinates of P keypoints in F frame, we want to recover the dense non-rigid 3D structure $S\in {\mathbb{R}}^{3F\times P}$.

Revisiting NRSfM. NRSfM aims to recover the 3D structure of an non-rigid object and the camera pose from a collection of 2D films acquired by a monocular camera. It can be represented by the following linear equation for orthogonal cameras:

$$W=R_{xy}S+t_{xy},$$

(1)

where $W\in {\mathbb{R}}^{2F\times P}$ represents a 2D image sequence with P key-points in F frames and stacks F following $2\times P$ matrix: $W_i=\left[\begin{array}{ccc}{x}_{i1}& \cdots & x_{ip}\\ y_{i1}& \cdots & y_{ip}\end{array}\right]$, where ${(x_{ij},y_{ij})}^T$ denotes the 2D coordinates of j-th key-point captured in the i-th frame; $S\in {\mathbb{R}}^{3F\times P}$ is the stack of 3D shapes $S_i$ of reconstructed objects for each frame, where $S_i$ contains the 3D coordinates of every key-point in the i-th frame and $S_i=\left[\begin{array}{ccc}{x}_{i1}& \cdots & x_{ip}\\ y_{i1}& \cdots & y_{ip}\\ z_{i1}& \cdots & z_{ip}\end{array}\right]$; $R_{xy}\in {\mathbb{R}}^{2F\times 3F}$ and $t_{xy}\in {\mathbb{R}}^{2F}$ both are the $x-y$ component of a rigid transformation. $R_{xy}=\left[\begin{array}{ccc}{R}_1& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & R_F\end{array}\right]$, where $R_i$ is the first two rows of the rotation matrix $R_i^{{}^{\prime }}\in {\mathbb{R}}^{3\times 3}$ for the i-th frame; and the translation parameter $t_{xy}$ of the camera can be eliminated by centralization when all the feature points are visible. Experiments in this study were implemented assuming that the camera translation component was eliminated.

When the reconstruction object of the NRSfM problem is a dense non-rigid object, both global and local features must be considered, and the computations required increase dramatically, thereby complicating the dense NRSfM. Kumar et al. [8,29,30] introduced the Grassmann manifold to address this problem, whereas Sidhu et al. [9] used the average face plus neural network method. This study developed an unsupervised deep neural network based on the basic theory of NRSfM. This network can deeply mine the relationship between dense video stream frames and frames, points, and points and leverage the learning capabilities of the neural network to achieve better reconstruction effects.

3.2. Deep Spatial-Temporal NRSfM

The architecture of the DST-NRSfM model, which operates end-to-end without requiring any real data for supervision, is shown in Figure 1. It takes the 2D feature point trajectory matrix W acquired by video optical flow processing as input and outputs the 3D structure S and the camera’s rotation matrix $R\in {\mathbb{R}}^{3F\times 3F}$ (shown in Figure 2). First, the reconstructor provides us with the initial reconstruction result; subsequently, we further optimize the reconstruction results in time and space according to the traits of high coupling between adjacent feature points of dense reconstruction objects and modest changes in neighboring frames. The outcomes of the second experiment in Section 4.2 indicate the effectiveness of the proposed spatial and temporal constraints.

3.2.1. Shape Estimator in Reconstructor

An illustration of the proposed shape estimator is shown in Figure 2a, which is inspired by the typical autoencoder neural network architecture and N-NRSfM [9]. However, our network structure fully utilizes the neural network’s inherent learning capacity, unlike N-NRSfM. With the addition of a feature encoder $f_e$ and the removal of the shape priors that lead to over-dependence issues, the complete network can learn how to reconstruct the 3D structure of an non-rigid object from its 2D trajectory matrix W. The number B of neurons in the middlemost hidden layer of the whole network framework (i.e., the output layer of $f_e$) is set significantly lower than that of the input and output layers to ensure that the shape estimator first obtains the low-dimensional expression of the input data through $f_e$ and then uses the feature decoder $f_d$ to reconstruct the 3D structure from the low-dimensional expression. This successfully reduces the difficulty of the solution caused by the dense input data and allows the entire shape-estimation network to have a certain level of noise resistance. The entire shape-estimation technique is summarized as follows:

$$S_i^{{}^{\prime }}=f_d\circ f_e\left(W_i\right),$$

(2)

where $W_i$ is reshaped into a $1\times 2P$ feature tensor to be fed into the fully connected layers.

Network structure of the feature decoder $f_d$. To improve the ability of the neural network to fit complex functions and better reconstruct the 3D structure of dense objects, we expanded the number of network layers based on the deformation auto-decoder [9], which is a series of nine fully connected layers with small hidden dimensions (2, 8, 8, 8, 16, 32, 32, 32, $\left|S_i\right|$), set the maximum number of layer nodes except for the output layer to 1024, used exponential linear units (ELU) as the activation units (except for the last layer), and added a layer normalization (LN) layer between each activation function and the neural network’s output. To alleviate the problem of network degradation induced by network deepening, we set the network to its residual form. When the number of neurons in the middle layer B is one, the entire network can be divided into four residual blocks. When $B=64$, the entire network has a large residual block. The main architecture of $f_d$ in Figure 3.

Network structure of the feature encoder $f_e$. Feature encoder $f_e$ is comparable to feature decoder $f_d$ in that it uses a fully connected residual network with a LN layer added before the output of each layer of the neural network enters the activation function ELU. The cone-shaped wiring runs throughout the decoder network. The input layer nodes can reach 1024, whereas the output layer nodes can only go down to B, which is the number of neurons in the middlemost layer determined by the features of various datasets. $f_e$ substitutes for the original shape prior and enhances the neural network’s ability to recover highly deformed objects, as demonstrated by the experimental results on the pants dataset in Section 4.3. The architecture of $f_e$ is shown in Figure 3.

Configuration for the middlemost hidden layer. The size of the number B of neurons in the middlemost hidden layer represents the low-dimensional expression of the reconstructed object, and modifying B influences the function fitting ability of the neural network as the number of neural network parameters changes. A reasonable selection of B allows us to better reconstruct the target object. When B is extremely low, the original information is lost, whereas if B is set extremely high, the reconstruction result is easily disrupted by noise in the input data and severely deformed objects cannot be recovered.

However, the number of neurons in the middlemost hidden layer is not a fixed value. It changes with the characteristics of different datasets, and can only be based on experience or trials to obtain a better value. In this study, B is defined as corresponding to different types of datasets based on the features of various datasets and trials. The settings for B for each dataset are presented in Section 4.

Layer normalization layers. The 2D trajectory matrix of the entire sequence serves as the input to our network to maintain the temporal information of the input sequence, which makes neural network training more challenging. Additionally, although choosing the ELU activation function eliminates the issue of network mean shift, soft saturation in the negative range also contributes to gradient disappearance and sluggish convergence during model training. To address the aforementioned issues, we introduced an LN layer [41] according to the network topology and properties of the incoming data. After every layer of the shape-estimation network (except the last layer), we added LN layers for data normalization. The output of each layer of the shape-estimation network, following the addition of the LN layer, can be stated as follows, assuming that the activation function is f.

$$y=f(\frac{x-E\left(x\right)}{\sqrt{var\left(x\right)-\epsilon }}\times g+b),$$

(3)

where $\epsilon$ is a small decimal value, which prevents the denominator from being 0. Herein, gain (g) and bias (b) ensure that the normalization process does not eliminate the preceding data. Both are altered by the model training and are selected by the neural network itself.

3.2.2. Rotation Estimation Network in Reconstructor

Figure 2b shows the rotation estimator module in the DST-NRSfM model, which can be divided into two sections: a rotation estimation network and rotation matrix corrector. The two sections collaborate to deduce the rotation matrix R. The rotation estimation network, which has only eight fully connected layers, and each layer is followed by an ELU (except the last layer), can be represented by the following formula.

$$\widehat{R}{}_i=g_{\omega }\left(W_i\right).$$

(4)

Simultaneously, because the input data are captured by an orthogonal camera and reflection ambiguity, that is, if the rotation matrix at the i-th frame is assumed to be $R_i$, then both $R_i$ and $-R_i$ are valid rotation matrices, but the choice of different rotation symbols will result in different recovery shapes, that is, $S_f=[S_{fx};S_{fy};S_{fz}]$ and $S_f=[-S_{fx};-S_{fy};-S_{fz}]$ may appear when reconstructing the rotation matrix. Therefore, we solved these problems by adding a rotation matrix corrector after the rotation estimation module. The entire correction process is as follows.

In Figure 4, ① represents the singular value decomposition (SVD) of $\widehat{R_i}$:

$$\widehat{R_i}=U\mathsf{\Sigma }V;$$

(5)

② imposes an orthogonal constraint, that is, $\widetilde{R_i}{\widetilde{R_i}}^T=1$, on the rotation matrix:

$$\widetilde{R_i}⟵U{V}^T;$$

(6)

③ resolves the reflection ambiguity by placing the estimated rotation matrix into a special orthogonal group with $\left|R_i^{{}^{\prime }}\right|=1$:

$$R_i^{{}^{\prime }}⟶U\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& \left|\widetilde{R_i}\right|\end{array}\right]V^T.$$

(7)

Finally, the 3D structure $S_i^{{}^{\prime }}$ estimated by the shape estimator network is reprojected to a 2D shape using the rotation matrix $R_i^{{}^{\prime }}$ to measure the degree of agreement between the projection of the 3D reconstruction result and the given 2D feature point. The reprojection loss was calculated as follows.

$$L_{reproj}=\sum _{i=1}^F{∥W_i-P{R}_i^{{}^{\prime }}{S}_i^{{}^{\prime }}∥}_{\epsilon },$$

(8)

where ${∥·∥}_{\epsilon }$ denotes the Huber loss:

$$L_{\delta }\left(e\right)=\left\{\begin{array}{cc}\frac{1}{2}{e}^2,& if\left|e\right|\le \delta ,\\ \delta ·(\left|e\right|-\frac{1}{2}\delta ),& otherwise,\end{array}\right.$$

(9)

where $\delta =1$, and e usually stands for residuals. Because we performed the experiments using the orthogonal camera model, the projection matrix P can be expressed as:

$$P=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\end{array}\right].$$

(10)

3.3. Temporal Constraint and Spatial Constraint

According to any dense NRSfM’s object that consists of dense feature points, we can assume that points in adjacent fields move in the same direction and most points between adjacent frames do not change significantly. Hence, we introduced temporal and spatial constraints to further improve the reconstruction outcomes.

Temporal constraint. The temporal constraint, also known as time-smoothing loss, limits the 3D trajectory smoothness of dense non-rigid reconstructions from the temporal dimension, as follows.

$$L_{temp}=\sum _{f=1}^{F-1}{\parallel S_{f+1}-S_f\parallel }_{\epsilon }.$$

(11)

Weighted spatial constraint. The spatial constraint aims to constrain the smoothness of 3D structures in each frame from the spatial dimension. Although the spatial smoothing described by Sidhu et al. [9] can achieve this effect, their algorithm causes undesirable deformations. Herein, we presented the weighted spatial constraint to improve the reconstruction performance. This guaranteed that the points in the local field of any vertex had the same motion trend and the local details of the reconstructed object can be preserved. These are all crucial for dense NRSfM, which brings the reconstruction result closer to the real value without deformation while ensuring smoothness. $L_{w\text{\_}spat}$ is given as:

$$L_{w\text{\_}spat}=\sum _{f=1}^F\sum _{i\in S_f}{∥i-\frac{{\sum }_{j\in N\left(i\right)}{\omega }_{ij}*j}{{\sum }_{j\in N\left(i\right)}{\omega }_{ij}}∥}_{\epsilon },$$

(12)

$${\omega }_{ij}=\frac{1}{2}(cot{\alpha }_{ij}+cot{\beta }_{ij}),$$

(13)

where $N\left(i\right)$ represents the set of all vertices in 1-ring field of vertex i, $i\in S_f$; and ${\alpha }_{ij}$ and ${\beta }_{ij}$ are the opposite corners of the two adjoining triangular patches of edge ij.

3.4. Alternative Training

The final training loss function L of DST-NRSfM is:

$$L={\lambda }_1{L}_{reproj}+{\lambda }_2{L}_{temp}+{\lambda }_3{L}_{w\text{\_}spat},$$

(14)

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are the weight coefficients that balance the influence of each loss function on the final structure. We discovered that training the networks with $L_{temp}$ and $L_{w\text{\_}spat}$ yields superior results.

Finally, the priors that solve the dense NRSfM problem are significantly reduced, and the complexity of the rotation estimation network is reduced because of a suitable shape-estimation network.

4. Experiments

4.1. Implementation Details

Datasets. The following three types of datasets were used. (1) Widely used dense benchmarks with 3D ground truth $S_{gt}$: actor mocap (100 frames with 36,349 points) [42], synthetic faces syn3 and syn4 with different camera trajectories (99 frames with 28,000 points) [7], Kinect paper sequences (193 frames with 58,000 points) and t-shirt sequences (313 frames with 77,000 points) [43]. (2) Relatively sparse datasets with 3D ground truth geometry: expressions (384 frames with 997 points) [44] and pants (291 frames with 2859 points) [45], which are significantly deformed. (3) Real video sequence: back (150 frames with 20,474 points) [46], heart surgery sequence (201 frames with 55,115 points) [47], and real face (120 frames with 28,332 points) [7].

Evaluation indicator. We utilized the normalized error (NE) to assess the algorithm’s performance for datasets with three-dimensional true values, defined as

$$e_{3D}=\frac{1}{F}\sum _{f=1}^F\frac{{∥S_{est}^f-S_{gt}^f∥}_F}{{∥S_{gt}^f∥}_F},$$

(15)

where $S_{gt}^f$ denotes the actual 3D structure of the f-th frame of the reconstructed object and $S_{est}^f$ is the f-th frame of our anticipated 3D structure, which is aligned to the ground truth after using the Procrustes algorithm.

Training Details. We employed RProp [48] as the optimizer with a learning rate of 0.0001 because we input the entire dataset during training to effectively retain the spatiotemporal information. All proposed network frameworks were initialized with He [49]. The experiment set the weights to ${\lambda }_1=1000$, ${\lambda }_2=1$, and ${\lambda }_3=1\times {10}^{-5}$ and trained for 60,000 epochs. The number B of neurons in the middlemost layer of the shape-estimation network, which is affected by the essential characteristics of the given dataset, extent of deformation, amount of noise, and degree of density, was selected empirically by the determined ranges $[1,64]$ throughout Section 4.2 and Section 4.3. In general, the larger the noise and deformation variables of the dataset, the closer B is to 1, and the sparser the dataset, and the smaller the value of B.

Experimental environment. DST-NRSfM was programmed using the PyTorch and trained on a server with 12 GB RAM, RTX2080Ti graphics card, and Ubuntu operating system.

4.2. Network Structure Analysis

The necessity for layer normalization layers. To demonstrate the importance of LN layers in the shape-estimation network, we designed a series of controlled trials for shape-estimation networks with and without LN layers using the same loss function on the synthetic faces syn3, the real dataset actor mocap, expressions (sparser), and the Kinect dataset paper. The convergence curves of each dataset are shown in Figure 5.

Figure 5 demonstrates how the LN layers cause the actor mocap and paper datasets to converge at a lower loss while improving the stability and speed of the expressions and syn3 dataset models. We effectively incorporated the LN layers into the proposed shape-estimation network. It improved the model’s problem-solving abilities, increased model stability, and hastened convergence.

Effectiveness of temporal constraint and weighted spatial constraint. We conducted ablation experiments on the loss function of the DST-NRSfM network to confirm the efficiency of the temporal constraint and the newly implemented weighted spatial constraint.

Table 1. Results of ablation experiments.

	syn3	syn4	Expressions	Actor Mocap	Paper
$L_{reproj}$	0.1196	0.1201	0.2205	0.2624	0.0376
$L_{reproj}+L_{temp}$	0.0149	0.0255	0.0273	0.0174	0.0342
$L_{reproj}+L_{temp}+L_{w\text{\_}spat}$	0.0134	0.0241	0.0226	0.0163	0.0296

reports the experimental results. The results show that despite the reconstruction results being considerably improved by the temporal constraints, the extension of the weighted spatial constraint further enhanced the reconstruction accuracy. Figure 6a–d visualize the reconstruction results of actor mocap, expressions, syn3, and syn4 with and without spatial constraints in the presence $L_{reproj}$ and $L_{temp}$ loss conditions. The results following the spatial constraints are more accurate and smoother, as can be seen in Figure 6a,c,d. In contrast, Figure 6b does not exhibit a notable qualitative difference because the expressions are sparser; however, careful inspection of the reconstructed point cloud (reconstructed points are yellow, and the reconstructed points after flipping are black) revealed that there were fewer flipped reconstruction points in the reconstruction results after the weighted spatial constraint. This suggests that the weighted spatial constraint aids in mesh quality improvement.

Analysis of model computational complexity. For smoother deployments in empirical applications, we analyzed the computational complexity of the DST-NRSfM model. As the complexity of the neural network model is mainly determined by two parameters: parameters (Params) and floating-point operations per second (FLOPS), we separately count the two parameters of the shape estimator and the rotation estimator, as shown in

Table 2. Computational complexity analysis of DST-NRSfM model.

	Dataset	Actor Mocap ($\mathit{B}=64$)	syn3 ($\mathit{B}=64$)	Paper ($\mathit{B}=2$)	Expressions ($\mathit{B}=2$)	Heart Surgery ($\mathit{B}=2$)	Back ($\mathit{B}=2$)	Real Face ($\mathit{B}=1$)
Shape Estimator	FLOPS	$8.06\times {10}^{10}$	$6.40\times {10}^9$	$2.51\times {10}^{10}$	$1.88\times {10}^9$	$2.43\times {10}^{10}$	$7.00\times {10}^9$	$7.62\times {10}^9$
	Params	$8.07\times {10}^7$	$6.47\times {10}^7$	$1.31\times {10}^8$	$4.90\times {10}^7$	$1.21\times {10}^8$	$4.67\times {10}^7$	$6.35\times {10}^7$
Rotation Estimator	FLOPS	$7.51\times {10}^9$	$5.93\times {10}^9$	$2.36\times {10}^{10}$	$1.05\times {10}^9$	$2.28\times {10}^{10}$	$6.39\times {10}^9$	$7.05\times {10}^9$
	Params	$7.51\times {10}^7$	$5.99\times {10}^7$	$1.22\times {10}^8$	$2.74\times {10}^6$	$1.14\times {10}^8$	$4.26\times {10}^7$	$5.87\times {10}^7$

. It is clearly visible that the computational complexity of the model differs depending on the number of neurons in the middle layer and the input dimensions. We found that model complexity is strongly influenced by dataset density, and model complexity increases as dataset density increases. Therefore, the model we designed still has a lot of room for improvement in the frame structure.

4.3. Results and Analysis

Actor mocap. The actor mocap sequence, as real face data with $S_{gt}$, allows us to analyze the reconstruction results of our method qualitatively and quantitatively. The visible results are shown in the third image of Figure 6a. Comparing to actual images, our method successfully reconstructed the 3D structure of the actor mocap. Furthermore, we qualitatively compared our approach to face model learning (FML) [50], scalable monocular surface reconstruction (SMSR) [51], consolidating monocular dynamic reconstruction (CMDR) [52,53], optimization neural network (RONN) [54], and N-NRSfM [9]. Moreover, $e_{3D}$ for the actor mocap is listed in

Table 3. $e_{3D}$ on the the actor mocap.

Dataset	FML [50]	SMSR [51]	CMDR [52]	RONN [54]	N-NRSfM [9]	Ours ($\mathit{B}=64$)
actor mocap	0.092	0.054	0.0257	0.0226	0.0181	0.0163

, which revealed that our method is significantly superior to other methods.

Synthetic Faces. The widely usage of synthetic faces allows us to compare DST-NRSfM to more methods. To more intuitively understand the efficacy of our approach, we compared the experimental outcomes of DST-NRSfM with classical sparse NRSfM methods, such as metric projections (MP) [55], complementary rank-3 spaces (CSF2) [17], block-matrix-method (BMM) [20], and organic priors based approach (OP) [3], traditional dense NRSfM methods, such as variational approach (VA) [7], dense spatio-temporal approach (DSTA) [25], CMDR [52,53], Grassmannian manifold (GM) [8], jumping manifolds (JM) [29], SMSR [51], and probabilistic point trajectory approach (PPTA) [14], and the latest neural-based dense NRSfM approaches, such as N-NRSfM [9], and RONN [54].

Table 4. $e_{3D}$ on the synthetic face sequences (syn3 and syn4).

		Classical Sparse NRSfM Methods				Neural Dense NRSFM Methods
Dataset		MP [55]	CSF2 [17]	BMM [20]	OP [3]	N-NRSfM [9]	RONN [54]	Ours ($B=64$)
syn3		0.0611	0.5474	0.0784	0.0279	0.0320	0.0309	0.0134
syn4		0.0762	0.5292	0.0918	0.0419	0.0389	0.0359	0.0263
Conventional Dense NRSfM Methods
Dataset	VA [7]	DSTA [25]	GM [8]	JM [29]	PPTA [14]	CMDR [52]	SMSR [51]	Ours($B=64$)
syn3	0.0346	0.0374	0.0294	0.0280	0.0390	0.0324	0.0304	0.0134
syn4	0.0379	0.0428	0.0309	0.0327	0.052	0.0369	0.0319	0.0263

presents the final comparative experimental results, where OP is the newest method that solves the dense NRSfM problem by extending the sparse approach to the dense domain; however, the accuracy of our proposed framework remained nearly twice as accuracy. Although DSTA also employs space-time limitations to enhance reconstruction accuracy, the reconstruction accuracy of the proposed framework was substantially higher than that of DSTA because our spatial constraint theory fundamentally differs from that proposed by DSTA; moreover, our solution is more flexible. Additionally, our approach performed better than state-of-the-art learning-based dense NRSfM approaches, despite syn4 being more challenging to reconstruct than syn3. The visual reconstruction outcomes for syn3 and syn4 are shown in Figure 6c,d, respectively.

Kinect Sequences. In contrast to the face sequences, the Kinect sequences exhibited larger deformations, more key points, and a simpler camera trajectory. To ensure a fair evaluation, we preprocessed the Kinect sequences as described by Kumar et al. [8] and ran a multi-frame optical flow [56] with default parameters to obtain dense correspondences. $e_{3D}$ for the Kinect sequences are presented in

Table 5. $e_{3D}$ on the Kinect paper and t-shirt sequences.

Dataset	MP [55]	DSTA [25]	GM [8]	JM [29]	N-NRSfM [9]	Ours
paper	0.0827	0.0612	0.0394	0.0338	0.0332	0.0296 ($B=2$)
t-shirt	0.0741	0.0636	0.0362	0.0386	0.0309	0.0396 ($B=2$)

. Visualizations are shown in the second row of Figure 7. Our approach outperformed all competing methods in the Kinect paper sequence. In the t-shirt sequence, owing to insufficient device memory, the weighted spatial constraint cannot be applied, causing a suboptimal reconstruction effect, but the result remains resistant.

Expressions. As the expressions dataset is horizontally shifted and relatively sparse, we set $B=2$. Using this dataset, we can compare our method with two new algorithms: Kernel Shape Trajectory Approach (KSTA) [57] and Global Model with Local Interpretation (GMLI) [44]. Herein, $e_{3D}$ for the expressions are presented in

Table 6. $e_{3D}$ on the expressions dataset.

Dataset	CSF2 [17]	KSTA [57]	GMLI [44]	N-NRSfM [9]	RONN [54]	Ours ($\mathit{B}=2$)
expressions	0.030	0.035	0.026	0.026	0.026	0.023

. The reconstruction results are shown in Figure 6b. We achieved $e_{3D}=0.023$, which is better than the best result of 0.026, for this sequence. Experiments demonstrated that DST-NRSfM can also be applied to sparser datasets with significant horizontal shifts.

Pants. The pants dataset collects sequences that document how pants appear when people jump and rotate. Extremely violent deformation makes the reconstruction extremely challenging. To highlight the benefits of DST-NRSfM in reconstructing severely distorted objects, we used it to reconstruct the pants dataset.

Table 7. $e_{3D}$ on the pants dataset.

Dataset	KSTA [57]	BMM [20]	PPTA [14]	Ours ($\mathit{B}=1$)
pants	0.220	0.183	0.203	0.148

presents the reconstruction results quantitatively, and the visualization is shown in the last row of Figure 7. Experiments demonstrated that the proposed framework had the best reconstruction effect on this dataset, and accuracy was enhanced by nearly 20% over the second-place finisher. Note that these are merely experimental results produced without spatial constraints because the dataset lacks a mesh template.

Real video sequence. We constructed several real video sequences, such as the back, heart surgery sequence, and real face. We adjusted the size of B according to the features of each dataset and expertise. The real face sequence contains a significant amount of noise; therefore, we set $B=1$. Figure 6e shows the effect of the size of B on the reconstruction results. For the back and heart surgery sequences, we assumed that their essential characteristics were 2; hence, $B=2$. The back and heart surgery sequences are shown in the first row of Figure 7. Evidently, the DST-NRSfM generated highly realistic outcomes.

5. Conclusions

This study provided a novel end-to-end unsupervised DST-NRSfM for solving tough dense NRSfM. We discarded shape prior and designed a network architecture that leverages the neural network’s capacity for learning according to the properties of various datasets. Therefore, even for objects with significant deformations, the best reconstruction results may be achieved with only reprojection loss and temporal constraints. To further enhance the reconstruction results, we introduced a new spatial constraint called a weighted spatial constraint. This constraint enhances the reconstruction accuracy and the mesh quality and smooths out the reconstruction. Aiming at the problem of gradient disappearance and model instability found in the experiment, we implemented layer normalization layers in dense NRSfM. Quantitative and qualitative results demonstrated that our method achieves competitive performance with state-of-the-art dense NRSfM methods.

In DST-NRSfM the number of neurons in the middlemost layer of the shape-estimation network can only be hypothesized based on the features of different datasets, which is similar to the bases in the conventional NRSfM technique. The model framework consists of fully connected networks, resulting in redundancy of parameters and high computational complexity. Moreover, because it involves matrices of size $P\times P$, where P is the number of feature points and requires a mesh template for computation, the weighted spatial constraint approach cannot be utilized for arbitrary datasets.

DST-NRSfM substantially enhances the reconstruction accuracy of dense NRSfM and provides a broader perspective of using neural networks to address such issues. Future research will focus on enabling a neural network with low computational complexity to automatically select the most appropriate structure based on the dataset features.