•Notice that no recent literature exists to collect the growing knowledge concerning 3D object detection, we fill this gap by starting with several basic concepts, providing a glimpse of evolution of ...3D object detection, together with comprehensive comparisons on publicly available datasets being manifested, with pros and cons being judiciously presented.•Witnessing the absence of a universal consensus on taxonomy with respect to 3D object detection, we contribute to the maturity of the taxonomy, which keeps a good continuity of existing efforts as well as adapts new branches for dynamics.•We present a case study on fifteen selected models among surveyed works, with regard to runtime analysis, error analysis, and robustness analysis closely. We argue that what mainly restricts the performance of detection is 3D location error based on our findings.
Autonomous driving is regarded as one of the most promising remedies to shield human beings from severe crashes. To this end, 3D object detection serves as the core basis of perception stack especially for the sake of path planning, motion prediction, and collision avoidance etc.. Taking a quick glance at the progress we have made, we attribute challenges to visual appearance recovery in the absence of depth information from images, representation learning from partially occluded unstructured point clouds, and semantic alignments over heterogeneous features from cross modalities. Despite existing efforts, 3D object detection for autonomous driving is still in its infancy. Recently, a large body of literature have been investigated to address this 3D vision task. Nevertheless, few investigations have looked into collecting and structuring this growing knowledge. We therefore aim to fill this gap in a comprehensive survey, encompassing all the main concerns including sensors, datasets, performance metrics and the recent state-of-the-art detection methods, together with their pros and cons. Furthermore, we provide quantitative comparisons with the state of the art. A case study on fifteen selected representative methods is presented, involved with runtime analysis, error analysis, and robustness analysis. Finally, we provide concluding remarks after an in-depth analysis of the surveyed works and identify promising directions for future work.
Deep Learning for 3D Point Clouds: A Survey Guo, Yulan; Wang, Hanyun; Hu, Qingyong ...
IEEE transactions on pattern analysis and machine intelligence,
2021-Dec.-1, 2021-12-1, 20211201, Letnik:
43, Številka:
12
Journal Article
Recenzirano
Odprti dostop
Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, ...deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.
LiDAR point clouds can effectively depict the motion and posture of objects in three-dimensional space. Many studies accomplish the 3D object detection by voxelizing point clouds. However, in ...autonomous driving scenarios, the sparsity and hollowness of point clouds create some difficulties for voxel-based methods. The sparsity of point clouds makes it challenging to describe the geometric features of objects. The hollowness of point clouds poses difficulties for the aggregation of 3D features. We propose a two-stage 3D object detection framework, called MS23D. (1) We propose a method using voxel feature points from multi-branch to construct the 3D feature layer. Using voxel feature points from different branches, we construct a relatively compact 3D feature layer with rich semantic features. Additionally, we propose a distance-weighted sampling method, reducing the loss of foreground points caused by downsampling and allowing the 3D feature layer to retain more foreground points. (2) In response to the hollowness of point clouds, we predict the offsets between deep-level feature points and the object’s centroid, making them as close as possible to the object’s centroid. This enables the aggregation of these feature points with abundant semantic features. For feature points from shallow-level, we retain them on the object’s surface to describe the geometric features of the object. To validate our approach, we evaluated its effectiveness on both the KITTI and ONCE datasets.
The extraction of ground points and breaklines is a crucial step during generation of high quality digital elevation models (DEMs) from airborne LiDAR point clouds. In this study, we propose a novel ...automated method for this task. To overcome the disadvantages of applying a single filtering method in areas with various types of terrain, the proposed method first classifies the points into a set of segments and one set of individual points, which are filtered by segment-based filtering and multi-scale morphological filtering, respectively. In the process of multi-scale morphological filtering, the proposed method removes amorphous objects from the set of individual points to decrease the effect of the maximum scale on the filtering result. The proposed method then extracts the breaklines from the ground points, which provide a good foundation for generation of a high quality DEM. Finally, the experimental results demonstrate that the proposed method extracts ground points in a robust manner while preserving the breaklines.
Display omitted
Surveying techniques such as terrestrial laser scanner have recently been used to measure surface changes via 3D point cloud (PC) comparison. Two types of approaches have been ...pursued: 3D tracking of homologous parts of the surface to compute a displacement field, and distance calculation between two point clouds when homologous parts cannot be defined. This study deals with the second approach, typical of natural surfaces altered by erosion, sedimentation or vegetation between surveys. Current comparison methods are based on a closest point distance or require at least one of the PC to be meshed with severe limitations when surfaces present roughness elements at all scales. To solve these issues, we introduce a new algorithm performing a direct comparison of point clouds in 3D. The method has two steps: (1) surface normal estimation and orientation in 3D at a scale consistent with the local surface roughness; (2) measurement of the mean surface change along the normal direction with explicit calculation of a local confidence interval. Comparison with existing methods demonstrates the higher accuracy of our approach, as well as an easier workflow due to the absence of surface meshing or Digital Elevation Model (DEM) generation. Application of the method in a rapidly eroding, meandering bedrock river (Rangitikei River canyon) illustrates its ability to handle 3D differences in complex situations (flat and vertical surfaces on the same scene), to reduce uncertainty related to point cloud roughness by local averaging and to generate 3D maps of uncertainty levels. We also demonstrate that for high precision survey scanners, the total error budget on change detection is dominated by the point clouds registration error and the surface roughness. Combined with mm-range local georeferencing of the point clouds, levels of detection down to 6mm (defined at 95% confidence) can be routinely attained in situ over ranges of 50m. We provide evidence for the self-affine behaviour of different surfaces. We show how this impacts the calculation of normal vectors and demonstrate the scaling behaviour of the level of change detection. The algorithm has been implemented in a freely available open source software package. It operates in complex 3D cases and can also be used as a simpler and more robust alternative to DEM differencing for the 2D cases.
•The definition of 3D keypoints is analyzed for with deep learning.•Four kinds of 3D keypoints definitions are discussed on large-scale point clouds.•MLP-based definition achieves the best ...performance on indoor and outdoor datasets.
The main solution for large-scale point clouds registration is to first obtain a set of matched 3D keypoint pairs and then accomplish the point cloud registration task based on these matched keypoint pairs. However, at present, many methods study the feature descriptors in the point clouds registration task, but few methods discuss the 3D keypoints detection issues. The commonly used 3D keypoints detection strategy is the voxel-grid-based downsampling method, and the detected 3D keypoints are usually with a relatively huge amount and also with no explicit geometrical properties, which finally leads to a low inlier ratio. In this study, we rethink the 3D keypoints detection problem for large-scale point clouds with deep learning. Specifically, we discuss four kinds of 3D keypoints detection methods based on the joint keypoint detection and description learning framework D3Feat, and carry out extensive analyses on both the indoor large-scale point clouds dataset 3DMatch and the outdoor large-scale point clouds dataset KITTI Odometry. Experimental results demonstrate that the Multi-layer Perceptron (MLP) based method achieves the best inlier ratios under the different numbers of extracted 3D keypoints on both the indoor and outdoor large-scale point clouds. Further, we test these four kinds of keypoints detection methods under the application of large-scale point clouds registration, and the MLP-based method also achieves the state-of-the-art registration performance.
Semantic segmentation, also called scene labeling, refers to the process of assigning a semantic label (e.g. car, people, and road) to each pixel of an image. It is an essential data processing step ...for robots and other unmanned systems to understand the surrounding scene. Despite decades of efforts, semantic segmentation is still a very challenging task due to large variations in natural scenes. In this paper, we provide a systematic review of recent advances in this field. In particular, three categories of methods are reviewed and compared, including those based on hand-engineered features, learned features and weakly supervised learning. In addition, we describe a number of popular datasets aiming for facilitating the development of new segmentation algorithms. In order to demonstrate the advantages and disadvantages of different semantic segmentation models, we conduct a series of comparisons between them. Deep discussions about the comparisons are also provided. Finally, this review is concluded by discussing future directions and challenges in this important field of research.
Point cloud processing has received more attention in recent years. Due to the huge amount of data, using supervoxels to pre-segment the points can improve the performance of point cloud processing ...tasks. There are some supervoxel algorithms generating high-quality results, but their low efficiency hinders the wide application in point cloud processing tasks. In this paper, we try to strike a good balance between the quality and efficiency of point cloud over-segmentation. We propose an algorithm suitable for GPU acceleration, which can generate supervoxel with high efficiency. The algorithm is a seed-based segmentation method, and we carefully design two stages: the clustering stage and optimization stage, each of which can be executed in parallel on the GPU. In the first stage, the algorithm generates an initial segmentation based on well designed energy functions, and the second stage further improves the result by minimizing the segmentation energy. Our method generates good segmentation results and achieves the fastest processing speed compared with the existing methods. We evaluate the supervoxels on three public datasets. Experiments show that our algorithm can generate high-quality segmentation for various point cloud data with high efficiency, which is important for advancing the application of point cloud supervoxels in subsequent processing.
Radar-based human activity classification is crucial for applications such as healthcare monitoring, fall detection and assisted living due to its superior sensing capabilities and privacy ...protection. Traditional classification methods generally retrieve features from the time-range domain or the time-frequency domain. Such two-dimensional representation neglects the underlying dependence between the three radar signal variables of time, range and Doppler frequency, and cannot fully depict the dynamic human motion features. In this paper, we propose a time-range-Doppler radar point clouds (RPC)-based learning model for human activity classification using a frequency-modulated continuous waveform (FMCW) radar. The human echoes are first transformed into a series of three-dimensional (3D) point cloud cubes integrating the motion signatures in three domains, namely time-range, time-Doppler, and range-Doppler domains. The generated RPC cubes are then fed into a newly developed two-layer convolutional multilinear principal component analysis network (CMPCANet) for feature extraction and motion classification. The CMPCANet comprises a simple network architecture with small training parameters, and can be directly implemented on the 3D tensor dataset to extract highly discriminative features. Experimental results demonstrate that proposed framework can achieve superior classification accuracy and noise robustness compared to other methods using multi-domain information, even with small training samples.
PDF
