Singular and Multimodal Techniques of 3D Object Detection: Constraints, Advancements and Research Direction

Abstract

Two-dimensional object detection techniques can detect multiscale objects in images. However, they lack depth information. Three-dimensional object detection provides the location of the object in the image along with depth information. To provide depth information, 3D object detection involves the application of depth-perceiving sensors such as LiDAR, stereo cameras, RGB-D, RADAR, etc. The existing review articles on 3D object detection techniques are found to be focusing on either a singular modality (e.g., only LiDAR point cloud-based) or a singular application field (e.g., autonomous vehicle navigation). However, to the best of our knowledge, there is no review paper that discusses the applicability of 3D object detection techniques in other fields such as agriculture, robot vision or human activity detection. This study analyzes both singular and multimodal techniques of 3D object detection techniques applied in different fields. A critical analysis comprising strengths and weaknesses of the 3D object detection techniques is presented. The aim of this study is to facilitate future researchers and practitioners to provide a holistic view of 3D object detection techniques. The critical analysis of the singular and multimodal techniques is expected to help the practitioners find the appropriate techniques based on their requirement.

1. Introduction

Object detection includes identifying both object class and location. Object detection is essential in versatile fields such as medical diagnosis, industrial production inspection, surveillance, etc. Researchers have gained high precision and real-time speed using deep learning-based 2D object detection methods [1,2,3,4,5,6,7,8]. Convolutional Neural Network (CNN) is the backbone of these algorithms. CNN can detect the pattern and shape of different objects. Although 2D object detection can precisely detect objects, it is deficient in depth and height information. Depth and height information is vital in obstacle avoidance, autonomous vehicle navigation, household robots, medical surgery, etc. However, unlike 2D object detection, 3D object detection is more complex in terms of model training, data availability, annotation and processing. Figure 1 illustrates the difference between 2D and 3D object detection of a car. Figure 1 shows that, in 2D object detection, the objects are detected in two-dimensional form (length and width in the image). On the other hand, in 3D object detection, a prediction of depth is also perceived along with length and width.

The most popularly used datasets for 3D object detection are KITTI [9], SUN RGB-D [10], ScanNet [11], nuScenes [12] and the Waymo Open Dataset [13]. Among the unmanned aerial vehicle (UAV) point of view datasets, Shahbazi et al., 2019, presented a 3D dataset generated using both a multi-view stereo camera and a LASER scanner. The site it covers is a gravel mine [14]. Vélez et al. published a high resolution RGB image dataset of a pistachio orchard in Spain captured from UAV [15]. This dataset is intended to use for 3D Photogrammetric Reconstruction Another large dataset containing human activity is produced by Li et al., 2021, in RGB-D format that can be used for 3D detection [16]. 3D detection is being applied in medical sector for internal organ disease detection. In the medical sector, 3D CNN is being used on 3D images obtained by Computed Tomography (CT), Diffusion tensor imaging (DTI), magnetic resonance imaging (MRI), Functional magnetic resonance imaging and Ultrasound [17]. But medical imaging differs in terms of data representation and is difficult to compare with other domains.

Since computers with GPUs are more available now, it has enhanced the scope of 3D detection with deep learning. Several review papers are found on 3D object detection. For example, Fernandes et al., 2021, presented a survey paper on 3D object detection [18]. This paper solely focuses on LiDAR-generated point cloud-based 3D object detection. Moreover, only self-driving vehicle-related methods are discussed in this paper. Zamanakos et al., 2021, published another survey paper on current trends in 3D object detection [19]. This survey exclusively covers LiDAR-based methods and focuses on the self-driving field. Arnold et al., 2019, published a survey article on 3D object detection techniques obtained by sensors other than LiDAR. This article discusses some fusion methods as well [20]. This research emphasizes 3D detection techniques relevant to autonomous driving. After carefully studying the “3D object detection”-related review papers published in Scopus and Web of Science journals, the contents of the most relevant review papers are noted in

Table 1. Relevant review articles on 3D object detection.

Ref.	Modality						Application Domain				Content
	Point Cloud		Camera		RADAR	Multi Modal	Autonomous Vehicles	Robot Vision	Precision Agriculture	Human Pose
	LiDAR	RGB-D	Mono	Stereo
[21]	H	M	H	L	L	L	H	H	L	L	This research focuses on point cloud and monocular camera-based 3D object detection. However, the scope is limited to autonomous vehicle navigation and robot vision.
[22]	H	M	H	H	M	H	H	L	L	L	This research elaborates different modality (camera, LiDAR, RADAR) performance, but the scope is limited to the autonomous vehicle sector.
[23]	H	H	H	M	L	M	H	H	L	L	This paper explores 3D object detection focusing on autonomous vehicles and indoor robot vision.
[24]	H	L	M	L	L	M	H	L	L	L	This article discusses 3D object detection techniques with point cloud methods, especially in the autonomous vehicle navigation sector. However, the other modalities such as stereo sensors are not discussed.
[25]	H	H	H	M	L	M	H	H	L	M	This research focuses on 3D object detection techniques based on point cloud and monocular camera.
[26]	L	H	H	M	L	M	L	H	L	L	This research discusses 3D object detection techniques for room shape assessment.
[27]	H	L	L	L	L	M	L	M	L	H	This article discusses 3D object detection techniques specifically with LiDAR sensors. However, the scope is limited to human detection.

Symbol: H—Detailed discussion. M—Fundamental information. L—no information or few information

. This visual presentation has paved the way to existing research gaps and new research scopes on 3D object detection techniques. Table 1 contains the relevant review papers.

Three-dimensional object detection techniques are recently gaining popularity due to the necessity of object shape and orientation estimation in real-world space. However, Table 1 clearly shows that, a comprehensive review paper on 3D object detection is still lacking which analyses both singular and multimodal techniques in recently popular application domains. This review article is written with a view to reducing this gap to some extent.

1.1. Paper Motivation

The previous survey papers focus on a single application sector (mostly autonomous vehicle navigation) or a singular modality of 3D object detection. Table 1 clearly depicts the scenario. However, 3D object detection is recently growing in other application domains besides autonomous driving. Precision agriculture, household robots, surveillance services, etc. can benefit from 3D object detection. As 3D object detection is rapidly gaining popularity with researchers, there have been some new discoveries as well which are not included in the previously published review papers.

We have studied and analyzed the trend of 3D object detection techniques used in diverse fields since 2017. Before 2017, the research related to 3D object detection is observed to be very scarce. Each technique, whether using LiDAR, monocular camera, stereo camera or RADAR, possesses individual strengths and some limitations. From our observation, there is a lack of a thorough study depicting opportunities and obstacles of singular and multimode 3D object detection techniques in different fields. The motivation of this paper is to present a holistic study to assist future practitioners and researchers to choose the suitable method for their application in 3D object detection research.

1.2. Paper Contribution

To the best of our knowledge, this is the first review paper covering diverse sectors of 3D object detection considering different modalities. The key contributions of this review paper are as follows:

• The fundamental concepts of 3D object detection are illustrated in this paper (Section 2);
• Three-dimensional object detection is booming in different sectors. This paper presents 3D object detection techniques applied in different sectors, benchmark datasets in various fields and compares sensor types (Section 3);
• The most frequently used evaluation metrics are discussed (Section 4);
• This paper compares the speed and performance of different methods in a popular benchmark dataset (Section 5);
• A SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis is presented on the singular and multimodal techniques (Section 5);
• After thorough analysis, future directions and limitations of the existing methods are provided (Section 6).

2. Three-dimensional Object Detection Techniques

Three-dimensional object detection requires inferring height and depth information along with object location. To acquire this knowledge about objects, researchers have used several modalities of data from different kinds of device or sensors. Different benchmark datasets of 3D object detection techniques incorporate different modalities, and they exhibit different scopes and constraints.

Table 2. Benchmark datasets of 3D object detection.

Dataset	Modalities					Annotated Object Category	Constraint
	LiDAR	RGB Camera	Stereo Camera	RADAR	RGB-D
KITTI [9]	✓	✓	✓ (Greyscale)	✗		11 classes	Focuses on road scene. Not suitable for indoor robotic vision research.
nuScenes [12]	✓	✓	✗	✓		23 classes	Focuses on autonomous driving scenario.
Waymo [13]	✓	✓	✗	✓		23 classes	Exclusively built for autonomous driving.
SUN RGB-D [10]	✗	✓	✗	✗	✓	700 classes	Focuses only on indoor scene.
ScanNet [11]	✗	✓	✗	✗	✓	20 classes	Focused on indoor scenes.

briefly discusses the modalities, annotated object category and constraints of the benchmark datasets.

In singular modality only one type of device is used, whereas multimodal methods combine more than one device. These modalities are reviewed in this section. Section 2.1 and Section 2.2, respectively, describe singular modalities and multimodal methods practiced by researchers.

2.1. Singular Modalities

2.1.1. Point Clouds

Numerous 3D object detections are performed from a point cloud. A point cloud can be generated from Kinect devices (SUN RGB-D dataset), CAD models, mobile scanners or LiDAR [28]. Point cloud is the illumination value along with the 3D location of points reflected from the object surface. LiDAR sensors having LASERs can directly produce a point cloud. From the depth information (usually obtained by infrared or Time of Flight camera) of RGB-D datasets, the point cloud is inferred by processing. For 3D detection, point clouds are processed in two ways, i.e., i. Direct point cloud processing; ii. Projection of the point cloud in Bird’s Eye View (BEV) or 2D plane.

Direct Point Cloud Processing

In direct point cloud processing, the point information (such as intensity, 3D coordinate values, color, etc.) of point clouds are directly processed by neural networks to extract features and predict 3D object boundaries. The network has to handle higher dimension data compared to the 2D plane projection-based method. A visual representation is shown in Figure 2. Object segmentation can be performed directly on a point cloud using a neural network [29,30]. In these research works, authors performed semantic segmentation using the features of points by applying max pooling for the overall skeleton of the object. They further improved this process by applying this technique in different layers and detected fine patterns as well. But this process is not demonstrated for instance level segmentation. Instance segmentation can be required when each object needs to be identified individually. Semantic segmentation can locate the same class of all objects with common category names, such as cars, people, bicycles, etc., whereas instance segmentation is capable of distinguishing one car from the others. This is often essential in object tracking or surveillance.

Shi et al. illustrated another point cloud-based network called PointRCNN where points are segmented as foreground and background in the first stage to generate 3D region proposals. Then, coordinates of the proposed regions are identified for performing object detection. However, this network showed poor performance on pedestrian detection compared to vehicles detection. The sparse nature of the point cloud from a small-sized object (here a pedestrian) may be a reason for the lower precision rate [31].

Qi et al. utilized an interesting geometric concept, Hough voting, to identify the core of the 3D object. From the point cloud of the surface of the object, the center can be located using Hough voting. They have successfully demonstrated 3D object detection by locating household items such as a sofa, chair, table, etc. in the SUN RGB-D and ScanNet datasets. But the applicability of this method in long-range detection such as on UAV images is uncertain based on the information of this research [32].

Some researchers have rasterized the information of point clouds, which are called voxels, and then the feature of every voxel is encoded [33,34,35,36]. An improved method of this network is introduced by Zhou, Y. and Tuzel, O. (2018). Instead of handcrafted feature encoding, they have incorporated a region proposal network for detection. They have detected different vehicles and pedestrians in the KITTI dataset, proving this method to be suitable in autonomous vehicles. Interestingly pedestrians detection required close distance LiDAR, compared to vehicles detection, It suggests that small objects such as pedestrians and cyclists generate too sparse point cloud to detect from long distance [37].

Projection of Point Cloud in Bird’s Eye View (BEV) or 2D Plane

Another commonly practiced method for 3D object detection is projecting the 3D point cloud to a pseudo-2D or Birds Eye View plane. After 2D view projection, a neural network is applied to extract features and generate object prediction. Some researchers have applied CNNs after converting the point clouds into 2D planes in their research [38,39]. A similar conversion technique from 3D point cloud to 2D plane is found in [40,41]. After the perspective view conversion, CNN-based feature recognition network is applied in this research.

Fang, Jin et al. used a LiDAR point cloud in the form of VoxelNet or PointPillars and then applied 3D CNN in feature map. A heat map is generated where the local maxima is the object center. Identifying the center made the object tracking faster. The nuScenes and Waymo Open Datasets are used in their research to demonstrate the precision [42]. Simon, Martin et al. (2019) created a Birds Eye View projection from a point cloud and then applied Complex YOLO following the YOLO V2 technique to predict the object class. After that, Euler Region Proposal was used to detect the orientation of the object [43].

In research of Lang, Alex H. et al. (2019), the features of point clouds are stacked in the form of pillars and named PointPillars. A pseudo-2D image is then projected with the PointPillar features. They claimed that any 2D CNN-based network can be applied at this stage to detect the object. In this paper, they applied Single Shot Detector (SSD) [44]. Though research on projecting 3D points to 2D planes has successfully detected road scene objects such as vehicles or pedestrians, the data transformation to another plane may lead to information loss and cause additional computation loss.

2.1.2. Camera/Vision

Vision-based systems have been successfully applied for both object detection and tracking [45,46,47,48]. Mahayuddin et al. performed vision-based moving object detection using semantic convolutional features and achieved a greater detection rate than YOLO V3 and faster than RCNN [46]. Besides single objects, researchers even estimated dynamic crowds from UAV imagery using a vision-based system [49]. While this research focuses on 2D object detection, vision-based moving object detection is also possible in three dimensions [50,51]. Three-dimensional detection using cameras can be performed using two methods, i.e., i. monocular and ii. stereo images.

Monocular (RGB)

Monocular camera is a single camera that is used to generate 2D images. From single 2D images, 2D object detection task i.e., object classification and localization can be done. For 3D object detection, additional depth information is inferred from geometric cues, contrast or prior shape information of the known object.

Several researchers performed 3D object detection only from a single image [52,53,54]. Monocular image-based 3D reconstruction is proposed by Shapii et al., 2020, where multiple images are used for generating a 3D view of a human activity pose [55]. This is the cheapest method for 3D object detection. But the accuracy is lower than stereo and LiDAR-based detection. However, monocular images can be combined with other 3D detection techniques to achieve better precision. This is discussed in Section 2.2, entitled “Multimodal methods”.

Stereo

Stereo cameras utilize two cameras, one capturing the left and another the right image. Its working principle follows that of human vision. By comparing the disparity between the same pixels of two camera images, it is possible to perceive the depth information of the object. Stereo images are proved to be usable for 3D object detection, with a slight compromise in precision compared to LiDAR [56,57,58]. Their research proves that stereo images can be used for 3D object detection precisely and economically.

Zhang et al. applied the CNN-based network TrackleNet for detecting and tracking pedestrians [50]. Stereo R-CNN [59], Pseudo-LiDAR from Visual Depth [56] and Triangulation Learning Network [60] are 3D deep learning networks developed and successfully applied for autonomous vehicle navigation. You et al. utilized stereo images guided by an economical few point laser for improving precision [57].

2.2. Multimodal Methods

Three-dimensional object detection can be performed by applying multiple sensors or devices. Based on the application and requirements, researchers have combined different sensors having different strengths. Recent research works on multimodal methods are briefly discussed in this section.

2.2.1. Two-dimensional Image and Point Cloud

Qi et al., 2018, used both RGB images and depth information for 3D object detection. In this research, CNN is used on an RGB image to generate region proposals, and then in the proposed regions depth information is merged to create a 3D frustum. After that, 3D instance segmentation and “amodal 3D box estimation” is performed to view the object behind an obstacle. This method works at real-time speed and exhibits high recall, even for detecting small objects. But this method relies heavily on the region proposed by the 2D object detector at the beginning. Therefore, an appropriate algorithm for region proposal in the 2D plane is essential for the overall success of the entire process [61]. Similar methods are observed in [62,63,64,65].

LiDAR-only methods (i.e., point cloud-based) are improved by incorporating RGB images in Point-RCNN [31], VoxelNet [37,66] and PointPillars [44]. These methods are similar to PointPainting [67,68,69,70].

Another blended method between a point cloud and an RGB image fuses features of point louds and features of images [71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90]. From collective features, the region of interest is identified. In these regions, 3D object detection is performed afterwards. Although combining RGB and LiDAR point cloud information is complex, it exhibited better performance in outdoor scene 3D object detection than the standalone LiDAR voxel method [91].

2.2.2. RADAR and 2D Image

Radio Detecting and Ranging (RADAR) senses distance using radio signals. As RADAR cannot anticipate color information, it can only assume the shape or size rather than classify. A big advantage of RADAR is being less prone to harsh weather compared to LiDAR or camera images. But combining RADAR with images has successfully improved precision of 3D object detection [92,93,94]. In [92], the object center is detected from a 2D image. Then, targeting that object center, a RADAR point cloud is used for obtaining depth information in frustrum shape. It improved image-based detection in the nuScenes dataset. Nabati et al. (2020) used RADAR-generated point clouds for region proposal for objects. Then this region is imposed on 2D image to perform 3D detection [93].

2.2.3. Other 3D Object Detection Methods

Wang et al., 2020, proposed a multimodal system for leveraging the advantages of images, LiDAR and RADAR. In this method, the first image is used to obtain the regions of interest. Then, in these regions, point clouds from LiDAR are used to obtain depth information along with orientation of the object. RADAR is used for further refinement of 3D detection [95].

Apart from the above-mentioned methods, simultaneous localization and mapping (SLAM) is applied for constructing 3D views. Chen et al. (2022) applied SLAM for waste sorting [96]. They used both image and LiDAR data for the SLAM application. They created their own dataset. Zhang et al. (2019) used SLAM on Multiview stereo images for 3D detection of humans from a UAV [50]. Structure-from-motion (SFM) is another 3D detection and localization approach recently being applied from UAVs. Gené-Mola et al. (2020) applied SFM for 3D detection of fruits [97]. This approach involves capturing multiple 2D images and then constructing a 3D view by combining these images by matching the key features. As this method is time-consuming, instance segmentation is performed to reduce the regions of interest before generating the 3D view. Teng et al. (2023) compared both LiDAR-based SLAM and SFM. In their research, they found SLAM to be faster and more accurate than SFM [98]. For better visualization, an illustration of 3D object detection technique stages is presented in Figure 2 in a very simplified manner.

3. Branches of 3D Object Detection Techniques

Three-dimensional object detection is gaining popularity in various domains. Recent research works on 3D object detection in different application sectors are shown in Figure 3.

The application of 3D object detection is expanding day by day. Figure 3 depicts the commonly practiced sectors of 3D object detection. Among all the four sectors, most research work is conducted in the autonomous vehicle navigation domain. The benchmark datasets are most versatile in this sector as well. Research works in these domains are tabularized in Table 3 and Table 4.

Apart from the above-discussed applications, 3D object detection is also performed in some recent medical research, such as augmented assistance in surgery or diagnosis. The input data format and acquisition process of 3D object detection in the medical field is different than other sectors. Computed Tomography (CT), Diffusion tensor imaging (DTI), magnetic resonance imaging (MRI), Functional magnetic resonance imaging and Ultrasound are three-dimensional. Three-dimensional convolutional networks can be applied to these images for 3D detection of organs for better disease detection. Three-dimensional CNN is successfully applied for disease severity prediction and classification [99,100,101].

However, 3D datasets are very few in number in the medical field. Patient confidentiality may be a reason behind it. Image augmentation is sometimes applied to increase the dataset size [102,103]. In Table 3, we have summarized singular modality 3D object detection methods practiced since 2017.

Multimodal techniques involve multiple sensors for 3D object detection. In Table 4, recent multimodal techniques are mentioned. This table indicates that multimodal techniques usually involve LiDAR or RGB images accompanied by some other modality.

Table 3 and Table 4 exhibit singular and multimodal techniques of 3D object detection in different fields. Observing the trends, the following insights are found.

• Autonomous Vehicle navigation: LiDAR is a very popular modality in this field. It can be used both in singular and multimodal methods. LiDAR exhibits long-range LASER scanning, making it capable of designing a standalone end-to-end 3D object detection system. RGB-D sensors have low range (usually lower than 10 m). Due to this constraint, autonomous vehicle navigation-related research works are not found to use this modality.

Among the vision-based technologies, a monocular RGB camera was used as a singular modality by few researchers, but it could not exhibit as much accuracy as the LiDAR sensor. Moreover, detectability range was lower than LiDAR. Some researchers have mentioned stereo cameras as a modality with high potential in autonomous vehicle navigation. Even the automotive company Tesla is focusing on stereoscopic vision rather than LiDAR, considering it more nature-inspired, economical, and similar to human vision. However, stereoscopic camera ranges are far less than that of LiDAR.

RADAR is not found to be used as a singular modality system for 3D object detection since, it cannot perceive color information of the objects. But in some research works, RADAR was implemented along with other modalities.

• Robot Vision: RGB-D sensor is the most popular sensor, serving both in singular and multimodal techniques of robot vision. Most of these research works are conducted in indoor environments. For this reason, the long-range detection requirement is absent in this field. This makes RGB-D a wonderful option for perceiving both color and depth information of objects in indoor environments. RGB-D cameras are constructed with an RGB camera (for color perception) and infrared sensors (for depth perception).
• Precision Agriculture: In agriculture, LiDAR is found to be used for long-range 3D detection. Specially, precision agriculture involving UAVs from high altitude is benefitted by LiDAR. LiDAR has been applied as singular modality or multimodal technique with other sensors such as RGB cameras or narrow beam SONAR (Sound Navigation and Ranging). However, monocular cameras are being used as a singular modality in the Multiview 3D detection technique. In this method, 2D images captured from different angles around the object contribute to 3D detection. In the case of multimodal detection, RGB cameras can be used with RGB-D sensors. Monocular cameras, having lower range than LiDAR can serve for 3D object detection from a closer range.
• Human activity/pose detection: Monocular RGB cameras are widely used for human pose detection using the Multiview 3D object detection technique. For long-range detection, LiDAR has been used by researchers. To enhance the detectability of LiDAR-based detection, some other modalities such as inertial measurement unit (IMU) are also used in existing research works. However, in indoor robot vision, human activity detection and precision agriculture, RADAR is not usually preferred for 3D object detection technique. The reason may be the low spatial resolution of RADAR (compared to LiDAR or camera) which makes detection of thin objects or close-proximity objects difficult and ambiguous.

Multimodal techniques have evolved for utilizing the benefits of multiple sensors. For example, structural information of objects is well perceived using LiDAR for long range, whereas cameras are good at perceiving fine texture information. However, multimodal techniques come with the additional cost and complexity of synchronizing different formats of data from different sensors.

Table 3. Singular modality techniques of 3D object detection methods from 2017 to 2022.

Domain	Ref.	Sensor Data Type						Dataset
		LiDAR	RGB-D	Monocular (RGB) image	Stereo Image Pair	RADAR	other	KITTI [9]	nuScenes [12]	Waymo [13]	SUN RGB-D [10]	ScanNet [11]	Others
Autonomous Vehicle navigation (Detecting vehicles and people in road scene)	[37]	✓						✓
	[31]	✓						✓
	[43]	✓						✓
	[44]	✓						✓
	[56]				✓			✓
	[59]				✓			✓
	[60]				✓			✓
	[53]			✓				✓
	[42]	✓							✓	✓
	[54]			✓					✓
	[104]							✓
	[105]	✓						✓	✓
Indoor objects (Robotic Vision)	[104]		✓								✓	✓
	[105]		✓								✓	✓
	[29]		✓										[106,107]
	[30]		✓									✓	[107,108]
	[32]		✓								✓	✓
	[109]		✓										[110]
	[111]		✓										Self-made
	[112]		✓										TUM RGB-D
Precision Agriculture	[113]						Multi spectral						Self-made
	[97]	✓											Self-made
	[114]			✓ (Multi View)									Self-made
Human Pose/ Activity Detection	[115]			✓ (Multi View)									[116,117,118]
	[119]			✓ (Multi View)									[116,117,118]

Table 3. Singular modality techniques of 3D object detection methods from 2017 to 2022.

Domain	Ref.	Sensor Data Type						Dataset
		LiDAR	RGB-D	Monocular (RGB) image	Stereo Image Pair	RADAR	other	KITTI [9]	nuScenes [12]	Waymo [13]	SUN RGB-D [10]	ScanNet [11]	Others
Autonomous Vehicle navigation (Detecting vehicles and people in road scene)	[37]	✓						✓
	[31]	✓						✓
	[43]	✓						✓
	[44]	✓						✓
	[56]				✓			✓
	[59]				✓			✓
	[60]				✓			✓
	[53]			✓				✓
	[42]	✓							✓	✓
	[54]			✓					✓
	[104]							✓
	[105]	✓						✓	✓
Indoor objects (Robotic Vision)	[104]		✓								✓	✓
	[105]		✓								✓	✓
	[29]		✓										[106,107]
	[30]		✓									✓	[107,108]
	[32]		✓								✓	✓
	[109]		✓										[110]
	[111]		✓										Self-made
	[112]		✓										TUM RGB-D
Precision Agriculture	[113]						Multi spectral						Self-made
	[97]	✓											Self-made
	[114]			✓ (Multi View)									Self-made
Human Pose/ Activity Detection	[115]			✓ (Multi View)									[116,117,118]
	[119]			✓ (Multi View)									[116,117,118]

Table 4. Multimodal modality techniques of 3D object detection methods from 2017 to 2022.

Domain	Ref.	Sensor Data Type						Dataset
		LiDAR	RGB-D	Monocular (RGB) image	Stereo Image	RADAR	Other	KITTI [9]	nuScenes [12]	Waymo [13]	SUN RGB-D [10]	ScanNet [11]	Others
Autonomous Vehicle navigation	[61]	✓		✓				✓
	[120]	✓		✓				✓
	[90]	✓		✓				✓					[121]
	[67]	✓		✓				✓	✓
	[92]			✓		✓			✓
	[94]			✓		✓			✓				[122]
	[95]	✓		✓		✓			✓
Indoor objects (Robotic Vision)	[61]		✓	✓							✓
Precision Agriculture	[123]		✓	✓									Self-made
	[124]		✓	✓									Self-made
	[125]	✓		✓			narrow beam SONAR						Self-made
Human Pose/ Activity Detection	[126]	✓					inertial measurement unit (IMU)						[127]

Table 4. Multimodal modality techniques of 3D object detection methods from 2017 to 2022.

Domain	Ref.	Sensor Data Type						Dataset
		LiDAR	RGB-D	Monocular (RGB) image	Stereo Image	RADAR	Other	KITTI [9]	nuScenes [12]	Waymo [13]	SUN RGB-D [10]	ScanNet [11]	Others
Autonomous Vehicle navigation	[61]	✓		✓				✓
	[120]	✓		✓				✓
	[90]	✓		✓				✓					[121]
	[67]	✓		✓				✓	✓
	[92]			✓		✓			✓
	[94]			✓		✓			✓				[122]
	[95]	✓		✓		✓			✓
Indoor objects (Robotic Vision)	[61]		✓	✓							✓
Precision Agriculture	[123]		✓	✓									Self-made
	[124]		✓	✓									Self-made
	[125]	✓		✓			narrow beam SONAR						Self-made
Human Pose/ Activity Detection	[126]	✓					inertial measurement unit (IMU)						[127]

4. Evaluation Metrics

A commonly used metric for measuring the performance of object detection is Average Precision [103]. Precision, average precision and mean average precision are widely utilized metrics for evaluating performance of 2D object detection. For 3D object detection, different benchmark competitions use average precision (AP) as well. AP is used for performance comparison in [37,61,128,129]. Average Orientation Estimation is also used as 3D object detection evaluation metric which shows the angle of the object [44,129]. A slightly different version of AP is Mean Average Precision, used by [32,42,44,129].

Wang et al. used Average Precision at different IoU (Intersection over Union) values, i.e., 0.5 or 0.7, for evaluation [56]. IoU is used to measure the amount of matched area between ground truth and the detected object. If there is a perfect match, i.e., the object is perfectly predicted, then the IoU value will be 1.

If TP indicates True Positive (correctly detected object), FP means False Positive (mistakenly detected as object), TN means True Negative (correct mentioning of absence of object) and FN means False Negative (could not predict the presence of object), then, from [130,131], the simplified equations of precision and recall are as follows:

Precision = TP/(TP + FP)

(1)

Recall = TP/(TP + FN)

(2)

The parameters of precision and recall are explained in Figure 4. The figure illustrates the meaning of TP, FP, TN and FN.

In Figure 4a, when the object (in this case the bicycle) is present in the image and it is detected, this scenario is called true positive. In Figure 4b, there was no bicycle in the image, but it was falsely detected. This scenario is called false positive. In Figure 4c, there was no bicycle and there was no detection of bicycles either. It is classed as a true negative. In Figure 4d, there was a bicycle, but it was not detected by the model. This is called false negative.

If the precision and recall curve is p(r), then the formula to compute average precision is:

$$\mathrm{AP}={\int }_0^{1}p\left(r\right)$$

(3)

Mean Average Precision (mAP) is computed by computing the average value of all the AP corresponding to an entire dataset. If a dataset has 10 categories, then after finding the AP for each category the mean is computed for these 20 categories to obtain mAP.

5. Observation and Analysis

Since 2017, most of the 3D object detection methods have been implemented for autonomous vehicle navigation. Numerous competitions have been organized on different open benchmarks, especially on road scenes. Three-dimensional object detection is gradually becoming popular in other domains as well, with the availability of GPU-based computers allowing high computation capabilities.

A comparison of the common devices for 3D detection is presented in

Table 5. Performance comparison of 3D object detection sensors and devices.

Performance Criterion	RADAR	LiDAR	RGB Camera
Object detection	Good	Good	Fair
Object classification	Poor	Fair	Good
Distance inference	Good	Good	Fair
Detecting edge	Poor	Good	Good
Visibility range	Good	Fair	Fair
Adverse weather performance	Good	Fair	Poor
Performance in low light condition	Good	Good	Fair

. Information in this table is collected from the research work of Chen et al. (2021) [132].

The performance of 3D object detection methods depending on the type of sensor can be seen from

Table 6. Comparison of networks in KITTI Vision benchmark Suite.

Modality	Method	Reference	Category	Easy	Moderate	Hard	Run Time
LiDAR+ RGB	VirConv-S	[133]	Car	92.48%	87.20%	82.45%	0.09 s
LiDAR+ RGB	LoGoNet	[134]	Car	91.80%	85.06%	80.74%	0.1 s
LiDAR+ RGB	LoGoNet	[134]	Pedestrian	54.04%	47.43%	44.56%	0.1 s
LiDAR	CasA++	[135]	Car	90.68%	84.04%	79.69%	0.1 s
LiDAR	CasA++	[135]	Pedestrian	56.33%	49.29%	46.70%	0.1 s
Stereo	DSGN++	[58]	Car	83.21%	67.37%	59.91%	0.2 s
Stereo	DSGN++	[58]	Pedestrian	43.05%	32.74%	29.54%	0.2 s
Stereo	DMF	[136]	Car	77.55%	67.33%	62.44%	0.2 s
Stereo	DMF	[136]	Pedestrian	37.21%	29.77%	27.62%	0.2 s
Monocular (RGB)	CIE + DM3D	[137]	Car	35.96%	25.02%	21.47%	0.1 s
Monocular (RGB)	QD-3DT [LSTM on RGB]	[138]	Car	12.81%	9.33%	7.86%	0.03 s
Monocular (RGB)	QD-3DT [LSTM on RGB]	[138]	Pedestrian	5.53%	3.37%	3.02%	0.03 s

. Table 6 contains the top average precision methods exhibited by different modalities in the KITTI vision benchmark leaderboard.

Table 6 shows that LiDAR-based methods exhibited maximum average precision. Multimodal techniques are a recent addition; in particular, LiDAR with camera techniques is showing good potential in terms of precision value. Thin surface objects (such as pedestrians) detection achieved average precision than wide surface cars. The monocular techniques captured only RGB image frames and then the images were trained in sequential network (e.g., LSTM). For dealing with only RGB, the speed was found to be fast, but the average precision was very much less than stereo or LiDAR.

Analyzing the practiced modalities of 3D object detections, following SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis,

Table 7. SWOT analysis on LiDAR (point cloud)-based 3D object detection.

Strengths LiDAR produces a direct point cloud, conveying the three-dimensional information (position and depth) and corresponding illumination value from the object surface. Up to now, LiDAR-based 3D object detection has been proved to have the highest precision.	Weaknesses LiDAR-generated point clouds are sparse. For thin surface objects, the sparsity is greater, which makes the detection process less precise. Thin objects can be detected only at a closer distance from the LiDAR position. Hence, average precision in detecting pedestrians or cyclists is always lower than vehicles.
Opportunities It can be applied for real-time applications. Waymo and some other vehicle companies are using LiDAR-based 3D object detection in self-driving cars.	Threats LiDARs are not commonly available. Researchers rely on open datasets if they cannot use LiDARs of their own. High price of LiDARs is not economic for numerous appli-cations. Majority of open datasets of LiDAR point clouds are fo-cused on road scenes, making it difficult for other sectors to grow.

,

Table 8. SWOT analysis on camera/vision-based 3D object detection.

Strengths It is the most inexpensive method, making it readily available to researchers even for developing new da-tasets for deep learning. Camera-based methods can classify object types with less processing compared to LiDAR or RGB-D. Unlike LiDAR, thin object detection does not suffer from lower precision.	Weaknesses Fog or dusty weather can easily hinder object detection. Depth information in single 2D images is less precise than any other method.
Opportunities While integrating with other modalities, the vi-sion-based method is proved to be very precise.	Threats Using multi-view 2D images for 3D detection (e.g., in the Structure-from-motion technique) is time-consuming, making it difficult for real-time applications.

and

Table 9. SWOT analysis on Multimodal 3D object detection.

Strengths In combined methods, one device or sensor can over-come the individual weakness of another. Hence, it provides better precision than single modality.	Weaknesses Integrating different modality data is challenging. It often requires more time to integrate the stages.
Opportunities With greater availability of sensors and datasets, combined methods are gaining popularity among re-searchers and practitioners.	Threats Failure of one modality can affect the overall success of the entire process. Unavailability of datasets in different sectors can ham-per combined modality research.

are developed.

6. Advancement, Challenges and Future Directions

Analyzing the strengths and weaknesses of existing research works, some valuable insights are learnt. These are listed as follows:

• Point cloud-based 3D object detection can be performed in both indoor and outdoor environments. However, LiDAR can generate point clouds at longer range in variable weather conditions, whereas RGB-D or Kinect-based point clouds face limitations in terms of range and weather conditions. For this reason, point cloud-based 3D object detection is performed with the help of LiDAR sensors in autonomous vehicle navigation research. However, RGB-D sensors are less expensive, and the generated point clouds are successfully implemented in close-range research works on precision agriculture or indoor robotic vision;
• Three-dimensional object detection technology is significantly supported by deep learning. Deep learning networks are comprised of multilayer neural networks which can learn the patterns of data. In significant 3D object detection networks, such as PointNet, PointNet++, VoxelNet, CenterNet, etc., deep learning is used to learn object information from points or group of points. Also, in the two-stage networks where initially RGB images are used for region proposals, deep learning is applied for predicting the regions of objects. Future research work may include deep learning for leveraging more opportunities such as transfer learning in 3D object detection-related research;
• Development of end-to-end 3D object detection networks is becoming popular for their ease of application. End-to-end networks directly need to collect raw sensor data and provide 3D bounding-box output predictions. To develop such networks, it is essential to choose the necessary type of sensor (LiDAR, camera or RADAR), pre-process the data, design a neural network to learn the features of the data and train, validate and evaluate the model. Both hardware and software knowledge are required for the developers of end-to-end networks;
• Data collection and annotation for 3D object detection is more complex compared to 2D object detection. Three-dimensional object detection data collection involves fusing the data from different types of sensors, such as LiDAR, monocular or stereo cameras, RADAR, etc. This process requires calibration of different devices and synchronization of the data. The data annotation for 3D object detection needs to describe not only the object location, but also its dimensions in space, position and orientation. The data description involves parameters such as object length, width, height, yaw, pitch, roll, occlusion amount, etc. More expertise in terms of 3D geometry is necessary to annotate data in the case of 3D object detection;
• The limitation of point cloud-based object detection, especially in outdoor environments, is sparsity. Hence, thin object detection is an open research question in this field. Comparing the average precision values of cars and pedestrians in Table 6, it is clearly visible that detection of the thin surface of pedestrians achieved far less precision. Hence, how to increase precision for thin object detection with point cloud methods is open research question;
• Data scarcity is one of the main constraints in 3D object detection-related research. Due to the support and sponsorship of automobile companies in their pursuit of building self-driving cars, some enriched benchmark datasets such as KITTI, Waymo, nuScenes, etc. are widely available. A few indoor benchmark datasets are also available for robot vision research, such as SUN RGB-D and ScanNet. But there is scarcity of open benchmark datasets for other fields. Specially, 3D object detection is becoming popular in precision agriculture, but the conducted research works are found to be using self-collected datasets. However, these datasets are not publicly available. This is a constraint on conducting 3D object detection research in agriculture.

To reduce the issue of data scarcity, the 2D object detection sector is getting benefit from the new technology Generative Adversarial Networks (GAN). GAN is a type of augmentation technique that can generate new synthetic data. In the near future, 3D object detection techniques can be benefitted by GAN as well. To enhance 3D datasets, more 3D CAD model datasets (e.g., ShapeNet) can emerge in the near future.

7. Conclusions

Three-dimensional object detection is rapidly gaining popularity among researchers and practitioners. As the real world is 3D, hence depth and height information are vital too. Previously the low computational power of computers, unavailability of sensors and lower number of datasets hindered research on 3D object detection. With time, these obstacles are being reduced, and 3D object detection is being performed in numerous sectors. This review paper is aimed to assist researchers and practitioners to be informed about the recent trends in the 3D object detection sector. This study has carefully analyzed the advantages and disadvantages of different modalities so that future researchers can benefit while selecting suitable methods and sensors for their application.