Reconstruction and expansion, as well as asset management, of highways necessitate the development of a current and highly precise 3D pavement model. Current inverse modeling methods with point ...clouds are laborious, time-consuming, and limited in precision. This article introduces an alternative framework for parametric inverse procedural modeling of highway pavement with standardized alignments seamlessly integrated with off-the-shelf modeling software. It comprises three key steps. (1) Extraction of highway pavement boundaries and lane markings: Initially, we combine grid-based and model-driven methods, followed by line structure-based clustering, to accurately generate road centerlines and layouts. (2) Road centerline generation: The centerline, derived from lane markings, informs highway alignments and parameters based on geometric characteristics such as curvature and slope. We utilize cost functions to facilitate this process. (3) Novel inverse procedural assembly: This innovative step integrates off-the-shelf modeling software. This approach involves extracting vector lines from point clouds and applying constraints at pivotal points on highway pavement cross-sections. Our focus is on refined component-level modeling, allowing for the assembly of diverse highway elements. This method significantly reduces human intervention and achieves high precision. In tests on two highway datasets from Sichuan Province, China, our method achieved excellent results. It attained an average correctness of 98.63% and completeness of 99.66% within a 10 cm error margin. A comparison with the intersection point method indicated minimal errors, with maximum values below 1.2%. The resultant 3D highway pavement model is modular and highly accurate at the centimeter level.
Digital twins (DTs) have been found useful in manufacturing, construction, and maintenance. Adapting DTs to serve cities, the question arises of what an urban digital twin should contain and how it ...should be orchestrated to serve a city’s dynamical ecosystem, along with how to enhance the efficiency of the city. We are aligning with the commonplace idea that the main advantage of using DTs is economical as, for example, DTs can improve the planning of activities thus saving money and time. But how can they be useful for a city? Instead of looking at the DTs as solutions in search of problems to be solved, we start from city needs. Our approach is two-fold. We start by briefly reviewing existing possibilities for meeting some specific needs, but keep the focus on identifying and attempting to close the gap between the needs arising from everyday city functions and the latest DT techniques useful for meeting those needs. DTs are technically different and serve different applications, yet they share a common identity and name, as well as several technical similarities. Adopting computer science terminology, we see a back-end city DT as the container of all information, while any single front-end, visualized or used either by humans or robots, offers a limited but meaningful representation of the DT for a specific application. Alas, there are multiple open questions regarding the realization and benefits of such back-end DT. Nevertheless, we discuss how the back-end DT (or any specific DT) could be updated autonomously from sensor data using artificial intelligence techniques, and how the front-ends could be used for large benefits to the entire city ecosystem.
•There is a call to better match DT technology to meet overall city needs.•City DTs differ from DTs used in manufacturing, construction, and maintenance.•Differences include both technical (BIM-GIS) and human factor-induced complexities.•Novel AI methods could serve in automated updating of city DTs from sensor data.•Human factors and inclusion of third parties need to be considered for city DTs.
A Survey of Visual Transformers Liu, Yang; Zhang, Yao; Wang, Yixin ...
IEEE transaction on neural networks and learning systems,
06/2024, Letnik:
35, Številka:
6
Journal Article
Odprti dostop
Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have ...recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers .
Three-dimensional semantic segmentation is the foundation for automatically creating enriched Digital Twin Cities (DTCs) and their updates. For this task, prior-level fusion approaches show more ...promising results than other fusion levels. This article proposes a new approach by developing and benchmarking three prior-level fusion scenarios to enhance the outcomes of point cloud-enriched semantic segmentation. The latter were compared with a baseline approach that used the point cloud only. In each scenario, specific prior knowledge (geometric features, classified images, or classified geometric information) and aerial images were fused into the neural network’s learning pipeline with the point cloud data. The goal was to identify the one that most profoundly enhanced the neural network’s knowledge. Two deep learning techniques, “RandLaNet” and “KPConv”, were adopted, and their parameters were modified for different scenarios. Efficient feature engineering and selection for the fusion step facilitated the learning process and improved the semantic segmentation results. Our contribution provides a good solution for addressing some challenges, particularly for more accurate extraction of semantically rich objects from the urban environment. The experimental results have demonstrated that Scenario 1 has higher precision (88%) on the SensatUrban dataset compared to the baseline approach (71%), the Scenario 2 approach (85%), and the Scenario 3 approach (84%). Furthermore, the qualitative results obtained by the first scenario are close to the ground truth. Therefore, it was identified as the efficient fusion approach for point cloud-enriched semantic segmentation, which we have named the efficient prior-level fusion (Efficient-PLF) approach.
Three-dimensional (3D) reconstruction techniques have been used to obtain the 3D representations of objects in civil engineering in the form of point cloud models, mesh models and geometric models ...more often than ever, among which, point cloud models are the basis. In order to clarify the status quo of the research and application of the techniques in civil engineering, literature retrieval is implemented by using major literature databases in the world and the result is summarized by analyzing the abstracts or the full papers when required. First, the research methodology is introduced, and the framework of 3D reconstruction techniques is established. Second, 3D reconstruction techniques for generating point clouds and processing point clouds along with the corresponding algorithms and methods are reviewed respectively. Third, their applications in reconstructing and managing construction sites and reconstructing pipelines of Mechanical, Electrical and Plumbing (MEP) systems, are presented as typical examples, and the achievements are highlighted. Finally, the challenges are discussed and the key research directions to be addressed in the future are proposed. This paper contributes to the knowledge body of 3D reconstruction in two aspects, i.e. summarizing systematically the up-to-date achievements and challenges for the applications of 3D reconstruction techniques in civil engineering, and proposing key future research directions to be addressed in the field.
Point clouds, as a kind of 3D objects representation, are the most primitive outputs obtained by 3D sensors. Unlike 2D images, point clouds are disordered and unstructured. Hence the classification ...techniques such as the convolution neural network are not applicable to point cloud analysis directly. To solve this problem, we propose a novel network to extract point clouds feature, named attention-based graph convolutional network (AGCN). Taking the learning process as a message propagation between adjacent points, we specifically introduce attention mechanism to construct a point attention layer for analyzing the relationship between local points feature. The object classification is implemented by stacking multiple layers of point attention layer. In addition, the proposed network is extended to an attention-based encoder-decoder structure for segmentation tasks. We also introduce an additional global graph structure network to compensate for the relative location information of the individual points in the graph structure network. Experimental results show that our network has lower computational complexity and faster convergence speed. Compared with existing methods, the proposed network can achieve comparable performance in classification and segmentation tasks.
Registration of partially overlapping, featureless three-dimensional (3D) point sets with noise is a difficult problem in the applications of large-scale metrology (LSM). Existing approaches use the ...sparse iterative closest points (SICP) method applying the lp norm to decrease the influence of outliers during registration. However, we reveal in this study that sparse point-to-point becomes easily trapped into local minima in featureless point clouds registration, and the error landscape of sparse point-to-plane is too shallow to restrain the sliding due to the lack of constraints in the large flat areas. Also, point clouds sampled from the large flat areas cause the low-rank matrix of linear equation in estimating the transformation matrix. Hence, we propose using the point-to-point lp distance constraints to restrain the sliding along large flat areas. We further define a weighted enhanced lp distance (WELD) error metric to slacken the constraints and escape from the local minima. Moreover, WELD can improve stability with the full-rank linear equation in estimating the transformation matrix. To verify the capability of escaping from the local minima and restraining the sliding, we choose SICP and two other algorithms to be compared with our method in simulated and actual point clouds. The comparisons show that our method successfully can escape from the local minima and restrain the sliding to handle outliers and noisy featureless point clouds effectively. The source code is available at https://github.com/Timbersaw-wangzw/WES-ICP.
Current point cloud registration methods are mainly based on local geometric information and usually ignore the semantic information contained in the scenes. In this article, we treat the point cloud ...registration problem as a semantic instance matching and registration task and propose a deep semantic graph matching method (DeepSGM) for large-scale outdoor point cloud registration. First, the semantic categorical labels of 3D points are obtained using a semantic segmentation network. The adjacent points with the same category labels are then clustered together using the Euclidean clustering algorithm to obtain the semantic instances, which are represented by three kinds of attributes including spatial location information, semantic categorical information, and global geometric shape information. Second, the semantic adjacency graph is constructed based on the spatial adjacency relations of semantic instances. To fully explore the topological structures between semantic instances in the same scene and across different scenes, the spatial distribution features and the semantic categorical features are learned with graph convolutional networks (GCNs), and the global geometric shape features are learned with a PointNet-like network. These three kinds of features are further enhanced with the self-attention and cross-attention mechanisms. Third, the semantic instance matching is formulated as an optimal transport problem, and solved through an optimal matching layer. Finally, the geometric transformation matrix between two point clouds is first estimated by the singular value decomposition (SVD) algorithm and then refined by the ICP algorithm. Experimental results conducted on the KITTI Odometry dataset demonstrate that the proposed method improves the registration performance and outperforms various state-of-the-art methods.
In this study, we propose a high-throughput and low-cost automatic detection method based on deep learning to replace the inefficient manual counting of rapeseed siliques. First, a video is captured ...with a smartphone around the rapeseed plants in the silique stage. Feature point detection and matching based on SIFT operators are applied to the extracted video frames, and sparse point clouds are recovered using epipolar geometry and triangulation principles. The depth map is obtained by calculating the disparity of the matched images, and the dense point cloud is fused. The plant model of the whole rapeseed plant in the silique stage is reconstructed based on the structure-from-motion (SfM) algorithm, and the background is removed by using the passthrough filter. The downsampled 3D point cloud data is processed by the DGCNN network, and the point cloud is divided into two categories: sparse rapeseed canopy siliques and rapeseed stems. The sparse canopy siliques are then segmented from the original whole rapeseed siliques point cloud using the sparse-dense point cloud mapping method, which can effectively save running time and improve efficiency. Finally, Euclidean clustering segmentation is performed on the rapeseed canopy siliques, and the RANSAC algorithm is used to perform line segmentation on the connected siliques after clustering, obtaining the three-dimensional spatial position of each silique and counting the number of siliques. The proposed method was applied to identify 1457 siliques from 12 rapeseed plants, and the experimental results showed a recognition accuracy greater than 97.80%. The proposed method achieved good results in rapeseed silique recognition and provided a useful example for the application of deep learning networks in dense 3D point cloud segmentation.
Digitalization of Nuclear Power Plants (NPPs) is critical for their safe and effective operation and maintenance. Development of Digital Twins (DTs) of NPP legacy assets and subsystems is key to ...achieving this goal. Doing this effectively requires a framework for intelligent allocation of limited resources. This framework is developed here by synthesizing emerging best practices with NPP operators' needs for legacy assets management. Within the framework, a pipeline employs deep-learning object detection to read and locate equipment tags in images. It computes their locations in the corresponding 3D point clouds and then relates that data to an asset management system. The pipeline is premised on preservation and augmentation of existing NPP asset management processes that preclude options such as RFID tags or barcodes. It is a significant step toward more efficient development of DTs of legacy assets. The contributions are framed in the context of a typical Canadian legacy NPP.
•A framework that defines the key aspects of legacy assets for Digital Twins.•Automatic link between photographic records and asset management software.•Performance of tag detection and optical character recognition on real images.•Pipeline for efficient linking of assets in point clouds to management systems.•Identification of asset locations using legacy asset tags in nuclear power plants.
PDF
