Object-Aware 3D Scene Reconstruction from Single 2D Images of Indoor Scenes

Abstract

Recent studies have shown that deep learning achieves excellent performance in reconstructing 3D scenes from multiview images or videos. However, these reconstructions do not provide the identities of objects, and object identification is necessary for a scene to be functional in virtual reality or interactive applications. The objects in a scene reconstructed as one mesh are treated as a single object, rather than individual entities that can be interacted with or manipulated. Reconstructing an object-aware 3D scene from a single 2D image is challenging because the image conversion process from a 3D scene to a 2D image is irreversible, and the projection from 3D to 2D reduces a dimension. To alleviate the effects of dimension reduction, we proposed a module to generate depth features that can aid the 3D pose estimation of objects. Additionally, we developed a novel approach to mesh reconstruction that combines two decoders that estimate 3D shapes with different shape representations. By leveraging the principles of multitask learning, our approach demonstrated superior performance in generating complete meshes compared to methods relying solely on implicit representation-based mesh reconstruction networks (e.g., local deep implicit functions), as well as producing more accurate shapes compared to previous approaches for mesh reconstruction from single images (e.g., topology modification networks). The proposed method was evaluated on real-world datasets. The results showed that it could effectively improve the object-aware 3D scene reconstruction performance over existing methods.

1. Introduction

Three-dimensional reconstruction from images is a long-standing issue in computer vision and can benefit various applications. For example, the reconstructed mesh can be used for virtual reality (VR) applications [1], games [2], etc. Three-dimensional reconstruction from 3D data, such as point clouds or depth images, is generally more accurate than reconstruction from 2D images because the 3D data provides more information about the shape and geometry of the object. However, obtaining 3D data often requires special devices such as lidar or depth cameras, which can be more expensive or difficult to use than standard 2D cameras. On the other hand, 2D images such as RGB images are easier to obtain, but the reconstruction can be more challenging due to the lack of 3D information. The advancement of deep learning has spurred a range of research addressing issues related to 3D understanding from 2D images, including 3D object detection [3], object tracking [4], object reconstruction [5], and scene reconstruction [6]. Among them, methods that perform 3D scene reconstruction from multiview images or videos have achieved good results. However, when only a single 2D image is available, the reconstructed quality is unsatisfactory because of the scale ambiguity caused by the dimension reduction in the 3D to 2D projection. Additionally, the majority of existing studies [7,8,9,10] have reconstructed objects in images as a single mesh, without separating individual objects within the scene. As a result, these reconstructed scenes are unsuitable for VR applications or games, as they do not allow the identification and interaction of individual objects. Therefore, object-aware 3D reconstruction has been considered for reconstructing an interactable 3D scene.

Object-aware 3D scene reconstruction from a 2D image involves estimating the 3D shape and pose of individual objects depicted in the image, the 3D layout bounding box of the scene, and the camera pose used to capture the input image [6]. The goal is to reconstruct a 3D scene that accurately reflects the real-world scene depicted in the input image. However, 2D image formation is irreversible because a single 2D image may be generated from different 3D scenes [11]. Thus, it is challenging to infer poses accurately from a single 2D image without depth information. Specific methods estimate the 3D poses of objects by using the constraints between objects in a scene. For example, objects should not overlap in the 3D world [12,13]. By adding these constraints to the loss functions during training [12,13], reasonable relative poses of objects can be recovered, but the results still have an enormous scope for improvement. Another challenge is that a 2D image typically shows only a limited portion of a 3D object, with occlusion and hidden surfaces that are not visible in the image. As a result, it can be difficult to accurately reconstruct the entire 3D geometry of an object from a single 2D image since important information is missing. Several experiments [6,14,15,16,17] have been conducted to solve this problem. In the early years of machine learning, various studies reconstructed 3D shapes using voxel representations or point clouds, which can capture the global shape well but may not accurately reproduce surface details. In recent years, implicit function-based methods [5,9,16] have solved the resolution issue. However, they sometimes produce broken meshes.

This study proposes a method to estimate the camera parameters, layout bounding box, 3D object bounding boxes, and 3D object shapes from a 2D image. Unlike previous methods that directly estimated this information from a 2D image, the proposed method first estimates the depth to recover the scale information, enhancing the 3D detection accuracy. For mesh reconstruction, we designed a mesh reconstruction network that outputs a mesh in two different formats followed by a multitask learning scheme. We summarize the contributions of this study as follows:

• A depth-feature generation module is introduced, which can solve the scale ambiguity issue. The generated depth features can be beneficial to recovering the 3D bounding boxes of the layout and objects.
• We propose enhancing the mesh reconstruction results using multitask learning. The mesh reconstruction network comprises the following components: a 2D image encoder that encodes a 2D input to a feature vector; and two decoders, wherein one decodes the feature vector into a point cloud and the other decodes the feature vector into an implicit mesh. The two decoders allow the image encoder to extract more information that can be utilized by the decoders, thereby reducing broken meshes.

The object-aware 3D scene reconstruction pipeline is shown in Figure 1. It consists of two stages: initial estimation and refinement. The input image is first preprocessed to generate 2D object bounding boxes. The initial estimation stage produces coarse estimations of the 3D object bounding boxes, camera pose, layout bounding box, and 3D object meshes. The refinement stage predicts residual errors and improves the coarse estimations to obtain the final 3D scene.

To evaluate our proposed methods, we followed the evaluation metrics and dataset used by [6,16]. We evaluated our methods using the SUN RGBD [18] and Pix3D [19] datasets. The SUN RGBD dataset provided ground-truth data for the evaluation of 3D pose estimation of objects, 3D layout estimation, and camera pose estimation. The Pix3D dataset was used for the evaluation of 3D mesh reconstruction.

The rest of this study is organized as follows. Related work on object-aware 3D scene reconstruction is reviewed in Section 2. Section 3 explains the proposed method and its components. Section 4 describes the experiments and their results. The final section summarizes and concludes the study’s results.

2. Related Work

Object-aware 3D scene reconstruction requires estimating the layout bounding box, 3D object bounding boxes, and object shapes [20]. Early studies focused on layout estimation and 3D object bounding box estimation without estimating 3D object shapes [12,13,21,22]. They estimated the layout bounding box only using the edge [23], area [24], and contour features [25]. Recently, there has been a growing focus on estimating 3D object bounding boxes [12,13,22]. In these studies, various constraints were used to guide the 3D object bounding box estimations. For example, in an indoor scene, objects are physically stable. Therefore, they must be aligned with walls and floors [12], and when 3D bounding boxes are projected onto a 2D image, the projected 2D bounding boxes should be consistent with the ground-truth 2D bounding boxes [22]. When humans exist in the scene, the human–object interaction provides enhanced knowledge of the object placement [13]. In the methods mentioned above, reconstructed objects are represented only by 3D bounding boxes, and the object shapes are not recovered.

Several shape retrieval methods have created 3D scenes with object shapes [26,27,28,29,30,31,32]. A computer-aided design (CAD) model that is closest in similarity to the object in the image is selected from a CAD database based on the distance in the embedding space between the input image and the CAD model. Although these methods allow for creating clean 3D scenes, the reconstruction quality and efficiency depend considerably on the variety and size of the CAD database. If no similar CAD models exist in the database, poorly matched CAD models are selected, resulting in inaccurate reconstructions. Moreover, if the CAD database is extensive, searching for similar 3D models takes a long time.

Single-object reconstruction methods [6,16] are usually combined for 3D scene reconstruction to reconstruct object meshes instead of finding similar CAD models. Early studies reconstructed meshes into point clouds [33,34]. However, point clouds cannot represent mesh topologies. Voxel grid representations have also been widely adopted [35,36,37,38], but the reconstructed meshes lack details because the resolution of the voxel grid and the computational capacity of the device restricts them. Several methods have employed octrees [39,40,41] to obtain a high-resolution voxel grid, but the resolution issue has not yet been fully solved. Several studies [42,43,44] have recovered shapes by deforming a template using single-object mesh reconstruction methods. Good reconstruction results were obtained when the target object had the same topology as the template. However, reconstruction accuracy is sometimes very low if the topologies differ significantly. Therefore, topology modification methods must be applied [6,14], but the reconstructed meshes suffer from self-intersection problems. To solve the resolution and non-watertight issues of the methods mentioned above, signed distance fields [45] and implicit fields [46] have been adopted to represent meshes. These methods are not restricted by resolution and can be adapted to objects with various topologies. Implicit3D [16] adopted implicit fields for mesh reconstruction, which is similar to the proposed method. However, Implicit3D sometimes generates broken meshes, and we propose a method that combines the point-cloud and implicit-field representations in the mesh decoder’s output to improve the reconstructed meshes’ global shapes.

3. Methods

An overview of the proposed object-aware 3D scene reconstruction method is shown in Figure 2. It jointly estimates the camera pose, 3D layout bounding box, 3D object bounding boxes, and 3D object shapes. First, the input images are prepossessed to generate object proposals using a pretrained 2D detector, faster region-based convolutional neural network (Faster R-CNN) [47]. Similar to [16], 3D scene reconstruction is then divided into two stages. The first is the initial estimation stage, which includes layout estimation, 3D object detection, and mesh reconstruction networks. We adopt the layout estimation and 3D object detection networks from [16] and propose a new mesh reconstruction network based on multitask learning. The second is the refined estimation stage, which includes a depth-feature generation network and a scene convolutional network (SGCN). The depth-feature generation network aims to generate depth features for the layout and objects. The SGCN follows the same design from [16], which produces the final camera pose, 3D layout bounding box, and 3D object bounding boxes using a combination of the initial estimations from the first stage and depth features from the depth-feature generation network.

This study follows the setup of the world coordinate system proposed in [16], which places the origin at the center of the camera and aligns the x-axis with the orientation of the camera forward and the y-axis with the direction at a 90-degree angle to the floor. Let $\beta$, and $\gamma$ denote the pitch and roll of the camera, respectively, and the camera pose is represented as $R\left(\beta , \gamma \right)$. The 3D bounding boxes are defined as a combination of the center $C\in {ℝ}^3$, size $s\in {ℝ}^3$, and orientation $\theta \in {ℝ}^3$ in 3D space. Specifically, the 3D layout bounding box is represented as $\left(C^l, s^l, {\theta }^l\right)$, while the 3D object bounding box is represented as $\left(C^o, s^o, {\theta }^o\right)$. Directly predicting the 3D object bounding box center $C^o$ is difficult [22]. We define $\delta \in {ℝ}^2$ as the 2D offset between the 2D projection of the 3D center and the center of the detected 2D object bounding box $c\in {ℝ}^2$ using the 2D object detector, and $d$ as the distance from the center of the 3D object bounding box to the camera center in 3D space. Let $K\in {ℝ}^3$ denote the camera intrinsic matrix, the $C^o$ is then parametrized as Equation (1). Thus, the 3D object bounding box can be decomposed as ($\delta , d, s^o, {\theta }^o$).

$$C^o=R^{-1}\left(\beta , \gamma \right)\cdot d\cdot \frac{K^{-1}{\left[c+\delta , 1\right]}^T}{{\parallel K^{-1}{\left[c+\delta , 1\right]}^T\parallel }_2}$$

(1)

The goal of this study is to estimate the camera pose $R\left(\beta , \gamma \right)$, 3D layout bounding box $\left(C^l, s^l, {\theta }^l\right)$, and 3D object bounding box ($\delta , d, s^o, {\theta }^o$). As directly regressing the orientation, sizes, and distance can be error-prone, classification and regression are combined to obtain the estimations. Specifically, the range of size and orientation of the 3D layout/object bounding box can be evenly divided into a set of small range spaces. The network classifies which small range space the target estimation belongs to and the residual error within that small range space. Same as [16], the $\beta , \gamma , s^l, {\theta }^l, d, s^o$, and ${\theta }^o$ adopts the combination of classification and regression estimation methods, while $C^l$ and $\delta$ are directly estimated. This is because the $C^l$ is computed using a pre-computed 3D layout center and its offset, and $\delta$ is the offset of the detected 2D object bounding box center $c$.

Initial Estimation Stage. The layout estimation network uses Resnet-18 [48] as the backbone, followed by three multilayer perceptron (MLP) networks. The three MLP networks output the classification and regression estimations of camera pose $R\left(\beta , \gamma \right)$, orientation ${\theta }^l$, and size $s^l$ of the layout bounding box, and the regression estimation of the offset of the 3D center of the 3D layout bounding box. The entire 2D image is the input of the layout estimation network. The 3D object detection network takes the 2D proposals detected using the 2D detector and estimates the 3D object bounding boxes. The output is the regression estimation of $\delta$ classification and regression estimations of $s^o, {\theta }^o$. The network architecture design is based on those proposed in previous studies [6,16]. The 3D layout and object bounding boxes are estimated in the camera space. In contrast to the mesh reconstruction network in [16], we designed the mesh reconstruction network based on multitask learning, which includes two decoders that generate shapes in different representations. This will be explained in Section 3.1.

Refined Estimation Stage. The refined estimation stage includes two networks: a depth-feature generation network that extracts depth features from the input image, and an SGCN, which is used for optimizing the coarse estimation of camera pose, 3D layout bounding box, and 3D object bounding boxes. The SGCN represents the objects, the layout, and the relationships between them as graph nodes. Each node contains features extracted at the initial estimation stage, flattened, and concatenated as a node feature vector. The node feature vector is then encoded into the same dimension feature vector using MLP networks and updated iteratively by aggregating the information from neighboring nodes using MLP networks. The architecture is based on the design proposed in [16]. However, in contrast to that study, the global depth feature is included in the layout node and the object depth features are included in the object nodes.

3.1. Mesh Reconstruction Network Based on Multitask Learning

It has been demonstrated in several studies [49,50] that simultaneously learning multiple tasks can improve performance on the original desired task, as certain information shared across tasks may be more effectively extracted through another task. In this study, motivated by multitask learning, we designed a mesh reconstruction network to learn different mesh representations. The architecture of the mesh reconstruction network is shown in Figure 3.

The cropped object is encoded into an image feature vector using Resnet-18, which is then concatenated with a one-hot encoded object class and further encoded into a shape embedding that contains the shape information of the target object. Two different shape decoders were used. One was a Local Deep Implicit Functions (LDIFs) decoder [51] that had the same architecture as that used in [16]. Given the shape embedding and 3D point in a canonical space, it estimates whether the 3D point is inside the target 3D mesh. During the inference time, when 3D points are densely sampled in the canonical space and evaluated using the decoder, the canonical space becomes an occupancy field, which can be processed to generate a triangular mesh using the marching cube algorithm [52]. However, the local deep implicit representation method sometimes generates broken meshes due to insufficient global shape supervision. To solve this problem, we employed an additional point cloud shape decoder to estimate the global shape of the target object. The point cloud decoder adopted in the architecture is described in [33].

3.2. Depth-Feature Generation Network

In the refined estimation stage, we added the global and object depth features to the layout and object nodes of SGCN, respectively. The added depth features helped alleviate the problem of inaccurate 3D perception caused by scale ambiguity. The networks that extract the global and object depth features share the same network architecture but comprise different weights. The network architecture is shown in Figure 4.

The depth-feature extraction network uses the Resnet-18 as the backbone, followed by a two-layer MLP network. We employed the pretrained LapDepthNet [53] to predict depth images from 2D images, as it has been trained on the indoor scene dataset NYU-Depth V2 [54]. The global depth feature was extracted from the global depth image, and the object depth features were extracted from the cropped depth image of the object area. In the SGCN, we concatenate the global depth feature with the original layout feature to create the layout node feature, and we concatenate the object depth features with the original object features to create the object node features. This is in addition to the features used in Implicit3D [16].

3.3. Loss Function

Losses for Mesh Reconstruction Network. The losses for the mesh reconstruction network are designed for two decoders, the Local Deep Implicit Functions (LDIFs) decoder and the point cloud decoder. Let ${ℒ}^{ldif}$ and ${ℒ}^p$ denote the losses for the LDIFs decoder and point cloud decoder, respectively, ${ℒ}^m$ denote the mesh reconstruction loss, and ${\lambda }^p$ denote the weight of ${ℒ}^p$. The mesh reconstruction loss is defined as Equation (2):

$${ℒ}^m={ℒ}^{ldif}+{\lambda }^p{ℒ}^p,$$

(2)

where ${ℒ}^p$ is the chamfer distance [51] calculated between ground-truth 3D points and estimated 3D points using the point cloud decoder. The ${ℒ}^{ldif}$ is further defined as follows:

$${ℒ}^{ldif}={\lambda }^s{ℒ}^s+{\lambda }^u{ℒ}^u+{ℒ}^c,$$

(3)

In Equation (3), ${ℒ}^s$ and ${ℒ}^u$ are the point sampling losses calculated using the L2 loss between the ground-truth labels and predicted labels of near-surface 3D points and uniformly sampled 3D points, respectively [16]. Further, ${\lambda }^s$ and ${\lambda }^u$ are the weights of ${ℒ}^s$ and ${ℒ}^u$, separately, and ${ℒ}^c$ is the shape element loss [51].

Losses for Layout Estimation Network and 3D Object Detection Network. We adopt the same losses from [16] for the layout estimation and 3D object detection networks. Specifically, a hybrid strategy that incorporates both classification loss and regression loss is used to optimize $\left(s^l, s^o,{\theta }^l, {\theta }^o, d, \beta , \gamma \right)$ [22]. The classification loss and regression loss employ Softmax [55] and smooth-L1 (Huber) loss [47], respectively. As the $C^l$ is calculated using its offset and a precomputed layout center, and $\delta$ also represents offset, they are optimized by L2 loss.

Joint Losses. In the refined estimation stage, a joint loss is employed by combining the layout estimation network loss ${ℒ}^l$, 3D object detection network loss ${ℒ}^d$, cooperative loss ${ℒ}^{co}$, and physical violation loss ${ℒ}^{phy}$ [16]. ${ℒ}^{co}$ describes the consistency constraints of the 3D bounding boxes projected onto the 2D bounding boxes. A detailed description is available in [22]. The observation inspires the physical violation loss that objects in the scene should not intersect with each other, and a detailed definition is provided in [16]. The joint loss ${ℒ}^{joint}$ is expressed as Equation (4):

$${ℒ}^{joint}={ℒ}^l+{ℒ}^d+{\lambda }^{co}{ℒ}^{co}+{\lambda }^{phy}{ℒ}^{phy},$$

(4)

where ${\lambda }^{co}$ and ${\lambda }^{phy}$ are the weights of the cooperative and physical violation losses, respectively.

4. Experiments

4.1. Experimental Setup

Dataset. Following previous studies [6,16], the SUN RGB-D dataset [18] and the Pix3D dataset [19] were used in the experiments. The SUN RGB-D dataset contains 10,335 indoor scene images, 3D camera pose annotations, 3D layout bounding boxes, semantic segmentation, and labels. From this dataset, 5050 images were used for testing and 5280 for training. The Pix3D dataset comprises 10,069 real-world images and 395 indoor object models under 9 categories. These images and shapes are annotated using pixel-level 2D–3D alignment. From this dataset, 7556 images were used for training and 2513 for testing. To ensure a fair comparison, we used the same train/test splits as [16].

Implementation. In our implementation, we used the open-source repository of Implicit3D [16] as the software backbone and modified the code to fit our specific needs. Specifically, we added a depth-feature generation network and a multi-task learning mechanism. The depth map estimation in our experiments was based on the implementation and pretrained model of LapDepthNet for the NYU-Depth V2 dataset. We used a Resnet-18 backbone from Implicit3D with a two-layer MLP network added on top for the depth-feature generation network. For the mesh reconstruction network, we based our pointset decoder on the design of the vanilla version of the decoder in [33], which we implemented as a 3-layer MLP network. We also used the evaluation code provided by Implicit3D. The PyTorch version we used was 1.7.0, and the python version was 3.8.13. We used the 2D detection results from [16] as inputs for our networks. The output dimensions of the depth feature from the depth-feature generation network were 1024 for the whole-scene depth maps and 512 for the cropped object depth maps. We first used the method employed in Implicit3D to train the networks in the initial estimation stage. After fixing the network’s weights in the initial estimation stage, the depth-feature extraction and SGCN networks were trained jointly for 30 epochs. Finally, except for the mesh reconstruction network, all networks were jointly trained for 30 epochs to fine-tune their weight. During training, we set the hyperparameters ${\lambda }^p=100.0$, ${\lambda }^s=0.1$, ${\lambda }^u=1.0$, ${\lambda }^{co}=150$, and ${\lambda }^{phy}$ = 20.0. The networks were trained on an Ubuntu server with six NVIDIA GeForce RTX 3090 graphics cards. During inference, only one graphics card was used.

4.2. Quantitative Results

This section presents quantitative evaluation results of the layout estimation, camera estimation, and 3D object detection for the SUN RGB-D dataset and the mesh reconstruction for the Pix3D datasets. A comparison between the proposed methods and existing methods is also presented.

Layout Estimation. The layout estimation for the SUN RGB-D dataset was evaluated using the 3D intersection over union (IoU) metric. We compared the results with previous studies [6,16,22,27], which are listed in Table 1. The proposed method achieved the best score among all the methods.

Camera Pose Estimation. Similar to previous studies [6,16,22], the mean absolute error (MSE) was used to evaluate the accuracy of the estimated camera pose. The distance between the estimated camera pose and ground truth defined by roll and pitch are calculated. The results are presented in Table 1. The best results are highlighted in bold. The proposed method obtained the lowest MSE between the ground truths for pitch estimation among all methods but a slightly higher MSE for roll estimation.

3D Object Detection. To evaluate the 3D object detection performance, we used the mean average precision(mAP) as the evaluation metric, similar to Total3D [6] and Implicit3D [16]. The mAP is calculated based on the 3D IoU of the estimated 3D bounding boxes and ground truth. The detection is considered a true positive when the IoU is greater than 0.15. A quantitative comparison with existing methods is illustrated in Table 2, wherein all methods used the same evaluation metric and dataset. The results indicate that the proposed method achieved the best score in a majority of categories and the best performance on average.

Mesh Reconstruction. We evaluated the mesh reconstruction performance on the Pix3D dataset using Chamfer distance, a metric used for computing the distance between two point clouds. We evaluated the mesh reconstruction network on 256 × 256 × 256 voxel grids and then executed the marching cubes algorithm to extract the triangular meshes. The reconstructed meshes were aligned with the ground-truth mesh using the iterative closest point algorithm. A total of 10 K points were sampled from the estimated and ground-truth meshes, and the distance between the two point clouds was calculated. We conducted experiments using point cloud decoders with different numbers of output points (1024, 2048, and 4096) and compared the use of the chamfer distance (CD) loss to the earth mover’s distance (EMD) loss in Equation (2). The performance of our method was compared to that of several state-of-the-art approaches [6,14,16,56] in

Table 3. Evaluation results of mesh reconstruction on the Pix3D dataset. The bolded results represent the highest scores.

Categories	Desk	Table	Bed	Chair	Wardrobe	Bookcase	Sofa	Tool	Misc.	Mean
TMN [14]	7.08	17.42	7.78	6.86	4.09	5.93	4.25	4.13	23.68	8.43
AtlasNet [56]	8.59	19.46	9.03	8.37	4.78	6.91	6.24	6.95	40.05	10.17
Total3D [6]	5.93	14.19	5.99	5.32	3.83	6.56	3.36	3.12	26.93	6.84
Implicit3D [16]	7.85	11.73	4.11	5.45	4.31	3.96	5.61	2.39	24.65	6.72
Proposed-2048 pts (EMD)	10.59	12.13	5.32	5.03	3.17	5.00	3.54	6.88	21.61	6.55
Proposed-1024 pts (CD)	10.27	13.79	4.84	5.07	2.96	3.90	3.27	4.56	22.71	6.71
Proposed-4096 pts (CD)	10.37	13.34	4.96	5.78	3.13	3.71	3.31	3.41	31.33	6.97
Proposed-2048 pts (CD)	9.80	12.08	4.45	5.24	3.26	3.57	3.58	4.51	26.71	6.46

. We found that the model that used the CD loss and produced 2048 output points outperformed the others on average. In Table 3, the categorical evaluation scores of [6,14,56] were obtained from Total3D [6] and the mean CD score was recalculated across the whole testing samples. The evaluation score of Implicit3D was reported using Implicit3D, which was originally calculated across the whole test set.

Table 4. The number of samples on the test set of pix3D for evaluation.

Categories	Desk	Table	Bed	Chair	Wardrobe	Bookcase	Sofa	Tool	Misc.	Total
Number of samples	175	467	248	959	60	90	486	11	17	2513

shows the number of samples of each category in the test set of the Pix3D dataset.

Table 1. Quantitative evaluation results for layout and camera pose estimations. The bolded results represent the highest scores.

Method	Cam Pitch (MSE)	Cam Roll (MSE)	3D Layout (IoU)
CooP [27]	3.28	2.19	56.9
HoPR [22]	7.60	3.12	54.9
Total3D [6]	3.15	2.09	59.2
Implicit3D [16]	2.98	2.11	64.4
Proposed	2.97	2.16	64.5

Table 1. Quantitative evaluation results for layout and camera pose estimations. The bolded results represent the highest scores.

Method	Cam Pitch (MSE)	Cam Roll (MSE)	3D Layout (IoU)
CooP [27]	3.28	2.19	56.9
HoPR [22]	7.60	3.12	54.9
Total3D [6]	3.15	2.09	59.2
Implicit3D [16]	2.98	2.11	64.4
Proposed	2.97	2.16	64.5

Table 2. Quantitative evaluation results for 3D object detection on the SUN RGB-D dataset. The bolded results represent the highest scores.

Method	Desk	Sofa	Bed	Sink	Dresser	Lamp	Chair	Cabinet	Table	Nightstand	mAP
CooP [27]	19.90	36.67	57.71	15.95	15.98	3.28	15.21	10.47	31.16	11.36	21.77
HoPR [22]	4.79	28.37	58.29	2.18	13.71	2.41	13.56	0.48	12.12	8.80	14.47
Total3D [6]	27.93	44.90	60.65	18.50	21.19	5.04	17.55	14.51	36.48	17.01	26.38
Implicit3D [16]	49.03	69.10	89.32	33.81	29.27	11.90	35.14	33.93	57.35	41.34	45.21
Proposed	49.36	70.26	89.46	31.52	33.60	12.21	35.38	32.29	55.05	46.46	45.56

Table 2. Quantitative evaluation results for 3D object detection on the SUN RGB-D dataset. The bolded results represent the highest scores.

Method	Desk	Sofa	Bed	Sink	Dresser	Lamp	Chair	Cabinet	Table	Nightstand	mAP
CooP [27]	19.90	36.67	57.71	15.95	15.98	3.28	15.21	10.47	31.16	11.36	21.77
HoPR [22]	4.79	28.37	58.29	2.18	13.71	2.41	13.56	0.48	12.12	8.80	14.47
Total3D [6]	27.93	44.90	60.65	18.50	21.19	5.04	17.55	14.51	36.48	17.01	26.38
Implicit3D [16]	49.03	69.10	89.32	33.81	29.27	11.90	35.14	33.93	57.35	41.34	45.21
Proposed	49.36	70.26	89.46	31.52	33.60	12.21	35.38	32.29	55.05	46.46	45.56

Execution Time and Network Parameters. We tested it on 100 images from the test split of the SUN RGBD dataset. The input images were first preprocessed using a pretrained 2D object detection network, Faster R-CNN [47], and the resulting preprocessed data were fed into our 3D reconstruction networks. The networks contained in the initial estimation stage and the refined estimation stage contain a total of 192.59 million parameters. As shown in

Table 5. Execution time and network parameters of the proposed method.

Method	Network Parameters	Average Execution Time
Proposed	192.59 million	3.68 s

, the average time taken to reconstruct a 3D scene from the preprocessed data was 3.68 s, demonstrating the efficiency of our method.

4.3. Qualitative Results

We used the pretrained model in Implicit3D to produce the reconstruction results and compare them with those obtained using the proposed method. Figure 5 shows the visualization results after testing on the SUN RGB-D dataset. The first row shows the input images, and the second row shows the visualization results with the ground-truth layout and 3D object bounding boxes. The third and fourth rows show the results produced using the pretrained model of Implicit3D and the proposed method for 3D detection and layout estimation, respectively. The last two rows show the object-aware 3D scene reconstruction results produced using Implicit3D and the proposed method. We visualized the reconstruction results of the first 100 testing images, and then randomly selected the images containing at least 2 objects that were being detected and reconstructed.

Observing the accuracies of the 3D bounding box estimations from the second to fourth rows is not intuitive. However, combined with the reconstructed scene, the proposed method shows better results than Implicit3D. For example, the wardrobe in the first column has a wrong orientation when produced using Implicit3D. Still, the result of the proposed method estimated the correct orientation. In the second column, evidently, Implicit3D even predicts the wrong orientation for oversized furniture, such as the bed. In the fourth column, the two farthest chairs are also incorrectly orientated with the input image when the reconstructed scene with meshes is compared to the input image. Therefore, the proposed method produces more complete meshes than the reconstructed scene with object shapes.

We also compared the reconstruction results for a single object from the Pix3D dataset. Our mesh reconstruction network consists of an LDIF decoder and a point cloud decoder. The point cloud decoder is only used during training to encourage the ResNet-18 backbone network to extract useful features for generating complete global shapes. The LDIF decoder generates the mesh results during inference, as in [16]. In general, the Implicit3D outputs complete meshes with visually indistinguishable mesh quality. However, in cases where the Implicit3D fails to generate complete meshes, our mesh reconstruction results can still be complete. We visualized the reconstruction results of all test images and put several results corresponding to the cases where Implicit3D generated broken meshes in Figure 6. The first row of the right part shows that our mesh reconstruction quality is visually competitive as in [16], but it failed to handle details such as think structure, as shown in the left part of the first row in Figure 6.

4.4. Discussion

In our experiments, we observed that incorporating depth features into the refined estimation stage resulted in slightly improved estimates for the camera pitch and 3D layout bounding box (Table 1). However, this improvement came at the cost of slightly worse performance in camera roll estimation. This suggests that depth features may not be as beneficial for 3D layout and camera pose estimation. As presented in Table 2, our method achieved better evaluation scores for 7 out of 10 categories and overall better average evaluation results when using depth features in the refined estimation stage. This indicates that depth features can help improve 3D object bounding box estimation.

As shown in Table 3, the reconstructed meshes differ when we set different numbers of output points. When using the chamfer distance (CD) loss with output point numbers 1024, 2048, and 4096, the average performance of the reconstruction increases and then decreases. When the number of output points is 1024, it is likely because the global shape supervision is insufficient, resulting in poor mesh reconstruction. When the output points are 4096, the network extracts too much information for the global shape and ignores the local detail features that are necessary for accurate local surface reconstruction. The experiments also show that both the CD loss and the earth mover’s distance (EMD) loss can improve the reconstruction quality of the mesh reconstruction network, with the CD loss achieving a slightly better average performance than the EMD loss. However, all of our models except the one that outputs 4096 points achieve better average CD scores than the other methods. The best model of our experiments performs well for specific object categories, such as chairs, wardrobes, and bookcases, but performs poorly for others. This is likely because our mesh reconstruction network has difficulty handling thin structures, as demonstrated by the bed frame in the first row of Figure 6. However, our method performs better for objects with large surfaces than Implicit3D, as seen in the results on the Pix3D dataset. Additionally, when the images are taken in certain constrained conditions, our method can infer more accurate global shapes than Implicit3D (Figure 7). We tested the proposed method and Implicit3D on photos captured under constrained conditions, such as a camera held at the bird-view angle, under sheltered environment illumination, and with texture-less background and object and occluded objects. The results are shown in Figure 7, which indicate that Implicit3D failed to recover the shapes accurately. However, the proposed method recovered the shapes with satisfactory quality.

To improve the proposed method’s performance on a wider range of objects, future work could focus on extending the method to handle unseen object categories more effectively. Currently, the method relies on the object categories that have been learned using the mesh reconstruction network. To improve the performance in novel object categories, it may be necessary to develop new techniques for transferring knowledge from known categories or for learning more flexible models that can adapt to a wider range of objects.

5. Conclusions

In this paper, we proposed an object-aware 3D scene reconstruction network that jointly estimates the camera pose, 3D layout, 3D object pose, and object shapes. We propose introducing a depth-feature generation network in the refined estimation stage to address the depth ambiguity issue in 2D to 3D understanding. We also propose using multitasking learning for mesh reconstruction networks to obtain more complete meshes. We evaluated the proposed method on real-world datasets―SUN RGB-D and Pix3D―and compared the results to state-of-the-art methods. Our experimental results on the SUN RGB-D dataset showed that this method improved 3D object bounding box estimation for a majority of object categories. The mesh reconstruction quality on the Pix3D dataset demonstrated that the proposed multitask learning-based mesh reconstruction network can be beneficial for complete shape estimation. One limitation of this study is that current mesh reconstruction networks can only estimate object shapes included in the learned object categories. The proposed method will be extended in future work to more general object reconstruction tasks.