The goals of this review paper on deep learning (DL) in medical imaging and radiation therapy are to (a) summarize what has been achieved to date; (b) identify common and unique challenges, and ...strategies that researchers have taken to address these challenges; and (c) identify some of the promising avenues for the future both in terms of applications as well as technical innovations. We introduce the general principles of DL and convolutional neural networks, survey five major areas of application of DL in medical imaging and radiation therapy, identify common themes, discuss methods for dataset expansion, and conclude by summarizing lessons learned, remaining challenges, and future directions.
Full text
Available for:
BFBNIB, FZAB, GIS, IJS, KILJ, NLZOH, NUK, OILJ, SAZU, SBCE, SBMB, UL, UM, UPUK
Scores produced by statistical classifiers in many clinical decision support systems and other medical diagnostic devices are generally on an arbitrary scale, so the clinical meaning of these scores ...is unclear. Calibration of classifier scores to a meaningful scale such as the probability of disease is potentially useful when such scores are used by a physician. In this work, we investigated three methods (parametric, semi-parametric, and non-parametric) for calibrating classifier scores to the probability of disease scale and developed uncertainty estimation techniques for these methods. We showed that classifier scores on arbitrary scales can be calibrated to the probability of disease scale without affecting their discrimination performance. With a finite dataset to train the calibration function, it is important to accompany the probability estimate with its confidence interval. Our simulations indicate that, when a dataset used for finding the transformation for calibration is also used for estimating the performance of calibration, the resubstitution bias exists for a performance metric involving the truth states in evaluating the calibration performance. However, the bias is small for the parametric and semi-parametric methods when the sample size is moderate to large (>100 per class).
Full text
Available for:
NUK, OILJ, SAZU, UKNU, UL, UM, UPUK
Purpose
Multiview two‐dimensional (2D) convolutional neural networks (CNNs) and three‐dimensional (3D) CNNs have been successfully used for analyzing volumetric data in many state‐of‐the‐art medical ...imaging applications. We propose an alternative modular framework that analyzes volumetric data with an approach that is analogous to radiologists’ interpretation, and apply the framework to reduce false positives that are generated in computer‐aided detection (CADe) systems for pulmonary nodules in thoracic computed tomography (CT) scans.
Methods
In our approach, a deep network consisting of 2D CNNs first processes slices individually. The features extracted in this stage are then passed to a recurrent neural network (RNN), thereby modeling consecutive slices as a sequence of temporal data and capturing the contextual information across all three dimensions in the volume of interest. Outputs of the RNN layer are weighed before the final fully connected layer, enabling the network to scale the importance of different slices within a volume of interest in an end‐to‐end training framework.
Results
We validated the proposed architecture on the false positive reduction track of the lung nodule analysis (LUNA) challenge for pulmonary nodule detection in chest CT scans, and obtained competitive results compared to 3D CNNs. Our results show that the proposed approach can encode the 3D information in volumetric data effectively by achieving a sensitivity >0.8 with just 1/8 false positives per scan.
Conclusions
Our experimental results demonstrate the effectiveness of temporal analysis of volumetric images for the application of false positive reduction in chest CT scans and show that state‐of‐the‐art 2D architectures from the literature can be directly applied to analyzing volumetric medical data. As newer and better 2D architectures are being developed at a much faster rate compared to 3D architectures, our approach makes it easy to obtain state‐of‐the‐art performance on volumetric data using new 2D architectures.
Full text
Available for:
BFBNIB, FZAB, GIS, IJS, KILJ, NLZOH, NUK, OILJ, SAZU, SBCE, SBMB, UL, UM, UPUK
Deep two-dimensional (2-D) convolutional neural networks (CNNs) have been remarkably successful in producing record-breaking results in a variety of computer vision tasks. It is possible to extend ...CNNs to three dimensions using 3-D kernels to make them suitable for volumetric medical imaging data such as CT or MRI, but this increases the processing time as well as the required number of training samples (due to the higher number of parameters that need to be learned). In this paper, we address both of these issues for a 3-D CNN implementation through the development of a two-stage computer-aided detection system for automatic detection of pulmonary nodules. The first stage consists of a 3-D fully convolutional network for fast screening and generation of candidate suspicious regions. The second stage consists of an ensemble of 3-D CNNs trained using extensive transformations applied to both the positive and negative patches to augment the training set. To enable the second stage classifiers to learn differently, they are trained on false positive patches obtained from the screening model using different thresholds on their associated scores as well as different augmentation types. The networks in the second stage are averaged together to produce the final classification score for each candidate patch. Using this procedure, our overall nodule detection system called DeepMed is fast and can achieve 91% sensitivity at 2 false positives per scan on cases from the LIDC dataset.
Purpose
Most state‐of‐the‐art automated medical image analysis methods for volumetric data rely on adaptations of two‐dimensional (2D) and three‐dimensional (3D) convolutional neural networks (CNNs). ...In this paper, we develop a novel unified CNN‐based model that combines the benefits of 2D and 3D networks for analyzing volumetric medical images.
Methods
In our proposed framework, multiscale contextual information is first extracted from 2D slices inside a volume of interest (VOI). This is followed by dilated 1D convolutions across slices to aggregate in‐plane features in a slice‐wise manner and encode the information in the entire volume. Moreover, we formalize a curriculum learning strategy for a two‐stage system (i.e., a system that consists of screening and false positive reduction), where the training samples are presented to the network in a meaningful order to further improve the performance.
Results
We evaluated the proposed approach by developing a computer‐aided detection (CADe) system for lung nodules. Our results on 888 CT exams demonstrate that the proposed approach can effectively analyze volumetric data by achieving a sensitivity of > 0.99 in the screening stage and a sensitivity of > 0.96 at eight false positives per case in the false positive reduction stage.
Conclusion
Our experimental results show that the proposed method provides competitive results compared to state‐of‐the‐art 3D frameworks. In addition, we illustrate the benefits of curriculum learning strategies in two‐stage systems that are of common use in medical imaging applications.
Full text
Available for:
BFBNIB, FZAB, GIS, IJS, KILJ, NLZOH, NUK, OILJ, SAZU, SBCE, SBMB, UL, UM, UPUK
The performance of a classifier is largely dependent on the size and representativeness of data used for its training. In circumstances where accumulation and/or labeling of training samples is ...difficult or expensive, such as medical applications, data augmentation can potentially be used to alleviate the limitations of small datasets. We have previously developed an image blending tool that allows users to modify or supplement an existing CT or mammography dataset by seamlessly inserting a lesion extracted from a source image into a target image. This tool also provides the option to apply various types of transformations to different properties of the lesion prior to its insertion into a new location. In this study, we used this tool to create synthetic samples that appear realistic in chest CT. We then augmented different size training sets with these artificial samples, and investigated the effect of the augmentation on training various classifiers for the detection of lung nodules. Our results indicate that the proposed lesion insertion method can improve classifier performance for small training datasets, and thereby help reduce the need to acquire and label actual patient data.
Purpose
Machine learning algorithms are best trained with large quantities of accurately annotated samples. While natural scene images can often be labeled relatively cheaply and at large scale, ...obtaining accurate annotations for medical images is both time consuming and expensive. In this study, we propose a cooperative labeling method that allows us to make use of weakly annotated medical imaging data for the training of a machine learning algorithm. As most clinically produced data are weakly‐annotated – produced for use by humans rather than machines and lacking information machine learning depends upon – this approach allows us to incorporate a wider range of clinical data and thereby increase the training set size.
Methods
Our pseudo‐labeling method consists of multiple stages. In the first stage, a previously established network is trained using a limited number of samples with high‐quality expert‐produced annotations. This network is used to generate annotations for a separate larger dataset that contains only weakly annotated scans. In the second stage, by cross‐checking the two types of annotations against each other, we obtain higher‐fidelity annotations. In the third stage, we extract training data from the weakly annotated scans, and combine it with the fully annotated data, producing a larger training dataset. We use this larger dataset to develop a computer‐aided detection (CADe) system for nodule detection in chest CT.
Results
We evaluated the proposed approach by presenting the network with different numbers of expert‐annotated scans in training and then testing the CADe using an independent expert‐annotated dataset. We demonstrate that when availability of expert annotations is severely limited, the inclusion of weakly‐labeled data leads to a 5% improvement in the competitive performance metric (CPM), defined as the average of sensitivities at different false‐positive rates.
Conclusions
Our proposed approach can effectively merge a weakly‐annotated dataset with a small, well‐annotated dataset for algorithm training. This approach can help enlarge limited training data by leveraging the large amount of weakly labeled data typically generated in clinical image interpretation.
Full text
Available for:
BFBNIB, FZAB, GIS, IJS, KILJ, NLZOH, NUK, OILJ, SAZU, SBCE, SBMB, UL, UM, UPUK
A system for automatic extraction of various feature layers and recognition of the text content of scanned topographic maps is presented here. Linear features which are often intersecting with the ...text are first extracted using a novel line representation method and a set of directional morphological operations. Other graphical objects are then removed in several stages to obtain a text-only image. A custom defect model is subsequently used to create an artificial training set for a Hidden Markov Model-based character recognition engine. Finally, the recovered text is recognized using this multifont segmentation-free optical character recognition (OCR). Extensive testing is conducted to assess the performance of different stages of the proposed system. Furthermore, our custom OCR is shown to achieve a 94% recognition rate for the extracted text, thereby outperforming a commercial OCR used as a benchmark.
Seamless Insertion of Pulmonary Nodules in Chest CT Images Pezeshk, Aria; Sahiner, Berkman; Zeng, Rongping ...
IEEE transactions on bio-medical engineering/IEEE transactions on biomedical engineering,
12/2015, Volume:
62, Issue:
12
Journal Article
Peer reviewed
Open access
The availability of large medical image datasets is critical in many applications, such as training and testing of computer-aided diagnosis systems, evaluation of segmentation algorithms, and ...conducting perceptual studies. However, collection of data and establishment of ground truth for medical images are both costly and difficult. To address this problem, we are developing an image blending tool that allows users to modify or supplement existing datasets by seamlessly inserting a lesion extracted from a source image into a target image. In this study, we focus on the application of this tool to pulmonary nodules in chest CT exams. We minimize the impact of user skill on the perceived quality of the composite image by limiting user involvement to two simple steps: the user first draws a casual boundary around a nodule in the source, and, then, selects the center of desired insertion area in the target. We demonstrate the performance of our system on clinical samples, and report the results of a reader study evaluating the realism of inserted nodules compared to clinical nodules. We further evaluate our image blending techniques using phantoms simulated under different noise levels and reconstruction filters. Specifically, we compute the area under the ROC curve of the Hotelling observer (HO) and noise power spectrum of regions of interest enclosing native and inserted nodules, and compare the detectability, noise texture, and noise magnitude of inserted and native nodules. Our results indicate the viability of our approach for insertion of pulmonary nodules in clinical CT images.
PDF
