Dense Semantic 3D Reconstruction Hane, Christian; Zach, Christopher; Cohen, Andrea ...
IEEE transactions on pattern analysis and machine intelligence,
09/2017, Volume:
39, Issue:
9
Journal Article
Peer reviewed
Both image segmentation and dense 3D modeling from images represent an intrinsically ill-posed problem. Strong regularizers are therefore required to constrain the solutions from being `too noisy'. ...These priors generally yield overly smooth reconstructions and/or segmentations in certain regions while they fail to constrain the solution sufficiently in other areas. In this paper, we argue that image segmentation and dense 3D reconstruction contribute valuable information to each other's task. As a consequence, we propose a mathematical framework to formulate and solve a joint segmentation and dense reconstruction problem. On the one hand knowing about the semantic class of the geometry provides information about the likelihood of the surface direction. On the other hand the surface direction provides information about the likelihood of the semantic class. Experimental results on several data sets highlight the advantages of our joint formulation. We show how weakly observed surfaces are reconstructed more faithfully compared to a geometry only reconstruction. Thanks to the volumetric nature of our formulation we also infer surfaces which cannot be directly observed for example the surface between the ground and a building. Finally, our method returns a semantic segmentation which is consistent across the whole dataset.
Purpose:
Quantum noise properties of CT images are generally assessed using simple geometric phantoms with uniform backgrounds. Such phantoms may be inadequate when assessing nonlinear reconstruction ...or postprocessing algorithms. The purpose of this study was to design anatomically informed textured phantoms and use the phantoms to assess quantum noise properties across two clinically available reconstruction algorithms, filtered back projection (FBP) and sinogram affirmed iterative reconstruction (SAFIRE).
Methods:
Two phantoms were designed to represent lung and soft‐tissue textures. The lung phantom included intricate vessel‐like structures along with embedded nodules (spherical, lobulated, and spiculated). The soft tissue phantom was designed based on a three‐dimensional clustered lumpy background with included low‐contrast lesions (spherical and anthropomorphic). The phantoms were built using rapid prototyping (3D printing) technology and, along with a uniform phantom of similar size, were imaged on a Siemens SOMATOM Definition Flash CT scanner and reconstructed with FBP and SAFIRE. Fifty repeated acquisitions were acquired for each background type and noise was assessed by estimating pixel‐value statistics, such as standard deviation (i.e., noise magnitude), autocorrelation, and noise power spectrum. Noise stationarity was also assessed by examining the spatial distribution of noise magnitude. The noise properties were compared across background types and between the two reconstruction algorithms.
Results:
In FBP and SAFIRE images, noise was globally nonstationary for all phantoms. In FBP images of all phantoms, and in SAFIRE images of the uniform phantom, noise appeared to be locally stationary (within a reasonably small region of interest). Noise was locally nonstationary in SAFIRE images of the textured phantoms with edge pixels showing higher noise magnitude compared to pixels in more homogenous regions. For pixels in uniform regions, noise magnitude was reduced by an average of 60% in SAFIRE images compared to FBP. However, for edge pixels, noise magnitude ranged from 20% higher to 40% lower in SAFIRE images compared to FBP. SAFIRE images of the lung phantom exhibited distinct regions with varying noise texture (i.e., noise autocorrelation/power spectra).
Conclusions:
Quantum noise properties observed in uniform phantoms may not be representative of those in actual patients for nonlinear reconstruction algorithms. Anatomical texture should be considered when evaluating the performance of CT systems that use such nonlinear algorithms.
Metal implants such as hip prostheses and dental fillings produce streak and star artifacts in the reconstructed computed tomography (CT) images. Due to these artifacts, the CT image may not be ...diagnostically usable. A new reconstruction procedure is proposed that reduces the streak artifacts and that might improve the diagnostic value of the CT images. The procedure starts with a maximum a posteriori (MAP) reconstruction using an iterative reconstruction algorithm and a multimodal prior. This produces an artifact-free constrained image. This constrained image is the basis for an image-based projection completion procedure. The algorithm was validated on simulations, phantom and patient data, and compared with other metal artifact reduction algorithms.
Computed tomography (CT) and magnetic resonance imaging (MRI) are two widely used clinical imaging modalities for non-invasive diagnosis. However, both of these modalities come with certain problems. ...CT uses harmful ionising radiation, and MRI suffers from slow acquisition speed. Both problems can be tackled by undersampling, such as sparse sampling. However, such undersampled data leads to lower resolution and introduces artefacts. Several techniques, including deep learning based methods, have been proposed to reconstruct such data. However, the undersampled reconstruction problem for these two modalities was always considered as two different problems and tackled separately by different research works. This paper proposes a unified solution for both sparse CT and undersampled radial MRI reconstruction, achieved by applying Fourier transform-based pre-processing on the radial MRI and then finally reconstructing both modalities using sinogram upsampling combined with filtered back-projection. The Primal–Dual network is a deep learning based method for reconstructing sparsely-sampled CT data. This paper introduces Primal–Dual UNet, which improves the Primal–Dual network in terms of accuracy and reconstruction speed. The proposed method resulted in an average SSIM of 0.932±0.021 while performing sparse CT reconstruction for fan-beam geometry with a sparsity level of 16, achieving a statistically significant improvement over the previous model, which resulted in 0.919±0.016. Furthermore, the proposed model resulted in 0.903±0.019 and 0.957±0.023 average SSIM while reconstructing undersampled brain and abdominal MRI data with an acceleration factor of 16, respectively - statistically significant improvements over the original model, which resulted in 0.867±0.025 and 0.949±0.025. Finally, this paper shows that the proposed network not only improves the overall image quality, but also improves the image quality for the regions-of-interest: liver, kidneys, and spleen; as well as generalises better than the baselines in presence the of a needle.
This article reports, for the first time, a 3-D digital reconstruction method for the geometric characterization of fused silica microshell resonators (MSRs). 3-D digital geometric information of ...fused silica MSRs prepared by a chemical foaming process is acquired by an X-ray microscope (XRM). The surface points of the microshell are selected by pixel contrast to reconstruct the 3-D geometric model. The fourth harmonics distribution of spatial radius and thickness at different shell heights are successfully calculated with the reconstruction model. The finite-element method based on the reconstruction model is also established to analyze the resonant frequency and frequency split of the wineglass mode. Results show that the relative fourth radius harmonic decreases with the height increase, indicating that the chemical foaming process has the capability to restrain the asymmetry for fused silica MSRs. Besides, the wineglass frequency difference between the digital reconstruction model and the experiment is within 3%. The frequency splits from the digital model also correspond well with the experiments. This work provides new insights into 3-D shell information and may enable manufacturing precision improvement of 3-D MSRs.
We address the problem of open-set recognition, where the goal is to determine if a given sample belongs to one of the classes used for training a model (known classes). The main challenge in ...open-set recognition is to disentangle open-set samples that produce high class activations from known-set samples. We propose two techniques to force class activations of open-set samples to be low. First, we train a generative model for all known classes and then augment the input with the representation obtained from the generative model to learn a classifier. This network learns to associate high classification probabilities both when image content is from the correct class as well as when the input and the reconstructed image are consistent with each other. Second, we use self-supervision to force the network to learn more informative featues when assigning class scores to improve separation of classes from each other and from open-set samples. We evaluate the performance of the proposed method with recent open-set recognition works across three datasets, where we obtain state-of-the-art results.
Accelerating the data acquisition of dynamic magnetic resonance imaging leads to a challenging ill-posed inverse problem, which has received great interest from both the signal processing and machine ...learning communities over the last decades. The key ingredient to the problem is how to exploit the temporal correlations of the MR sequence to resolve aliasing artifacts. Traditionally, such observation led to a formulation of an optimization problem, which was solved using iterative algorithms. Recently, however, deep learning-based approaches have gained significant popularity due to their ability to solve general inverse problems. In this paper, we propose a unique, novel convolutional recurrent neural network architecture which reconstructs high quality cardiac MR images from highly undersampled k-space data by jointly exploiting the dependencies of the temporal sequences as well as the iterative nature of the traditional optimization algorithms. In particular, the proposed architecture embeds the structure of the traditional iterative algorithms, efficiently modeling the recurrence of the iterative reconstruction stages by using recurrent hidden connections over such iterations. In addition, spatio-temporal dependencies are simultaneously learnt by exploiting bidirectional recurrent hidden connections across time sequences. The proposed method is able to learn both the temporal dependence and the iterative reconstruction process effectively with only a very small number of parameters, while outperforming current MR reconstruction methods in terms of reconstruction accuracy and speed.
Reconstructing 3D MR volumes from multiple motion-corrupted stacks of 2D slices has shown promise in imaging of moving subjects, e.g ., fetal MRI. However, existing slice-to-volume reconstruction ...methods are time-consuming, especially when a high-resolution volume is desired. Moreover, they are still vulnerable to severe subject motion and when image artifacts are present in acquired slices. In this work, we present NeSVoR, a resolution-agnostic slice-to-volume reconstruction method, which models the underlying volume as a continuous function of spatial coordinates with implicit neural representation. To improve robustness to subject motion and other image artifacts, we adopt a continuous and comprehensive slice acquisition model that takes into account rigid inter-slice motion, point spread function, and bias fields. NeSVoR also estimates pixel-wise and slice-wise variances of image noise and enables removal of outliers during reconstruction and visualization of uncertainty. Extensive experiments are performed on both simulated and in vivo data to evaluate the proposed method. Results show that NeSVoR achieves state-of-the-art reconstruction quality while providing two to ten-fold acceleration in reconstruction times over the state-of-the-art algorithms.
We present a framework for real-time 3D reconstruction of non-rigidly moving surfaces captured with a single RGB-D camera. Based on the variational level set method, it warps a given truncated signed ...distance field (TSDF) to a target TSDF via gradient flow without explicit correspondence search. We optimize an energy that contains a data term which steers towards voxel-wise alignment. To ensure geometrically consistent reconstructions, we develop and compare different strategies, namely an approximately Killing vector field regularizer, gradient flow in Sobolev space and newly devised accelerated optimization. The underlying TSDF evolution makes our approach capable of capturing rapid motions, topological changes and interacting agents, but entails loss of data association. To recover correspondences, we propose to utilize the lowest-frequency Laplacian eigenfunctions of the TSDFs, which encode inherent deformation patterns. For moderate motions we are able to obtain implicit associations via a term that imposes voxel-wise eigenfunction alignment. This is not sufficient for larger motions, so we explicitly estimate voxel correspondences via signature matching of lower-dimensional eigenfunction embeddings. We carry out qualitative and quantitative evaluation of our geometric reconstruction fidelity and voxel correspondence accuracy, demonstrating advantages over related techniques in handling topological changes and fast motions.
Interpreting the climate information recorded by leaf n-alkane carbon isotopes (δ13Calk) in geological sediments facilitates climate prediction. However, this interpretation relies on the calibration ...of modern plant δ13Calk responses to climate change. Here, we present the first continuous calibration at the same place next to a meteorological station, which excludes errors in the commonly used spatial calibrations along environmental gradients, resulting in robust calibrations. δ13Calk was negatively related to precipitation but not temperature. δ13C29 was most related to precipitation, followed by δ13C27–33 (i.e., the mean δ13C from n-C27 to n-C33), δ13C27, δ13C31, and δ13C33. If δ13Calk was only corrected for changes in δ13C of atmospheric CO2 (δ13Catm), the coefficients of δ13Calk vs. annual precipitation were − 0.0048‰/mm, −0.0035‰/mm, and − 0.0033‰/mm for δ13C27, δ13C29, and δ13C27–33, respectively. If correcting for changes in both δ13Catm and atmospheric CO2 concentration to δ13Calk, the coefficients of δ13Calk vs. annual precipitation were − 0.0038‰/mm, −0.0025‰/mm, and − 0.0023‰/mm for δ13C27, δ13C29, and δ13C27–33, respectively. Furthermore, we provided evidence for this relationship using a greenhouse study wherein water availability explained 97% of δ13Calk variance. Therefore, δ13Calk is a good indicator of precipitation, especially δ13C29. Finally, we conducted a reconstruction of pCO2 levels during the late Paleocene (pCO2(initial)) and the height of the carbon isotope excursion (CIE) (pCO2(excursion)) at the Paleocene-Eocene Thermal Maximum (PETM) by coupling the coefficients with other geological indicators. In the scenario where methane hydrates might have been the primary carbon source responsible for PETM warming, the estimated pCO2(initial) and pCO2(excursion) were 201–275 ppm and 911–985 ppm, respectively; whereas, in the scenario where volcanic CO2 emissions were the primary carbon source, they were 398–505 ppm and 1708–1815 ppm, respectively.
•A continuous investigation of plant n-alkane carbon isotope responses to climate change.•n-alkane carbon isotope is negatively related to precipitation but not temperature.•n-alkane carbon isotope is a good indicator of precipitation.•Coefficient of carbon isotope vs. annual precipitation is −0.0048‰/mm for n-C27.•Coefficient of carbon isotope vs. annual precipitation is −0.0035‰/mm for n-C29.
PDF
