Many machine learning and pattern classification methods have been applied to the diagnosis of Alzheimer's disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI). Recently, ...rather than predicting categorical variables as in classification, several pattern regression methods have also been used to estimate continuous clinical variables from brain images. However, most existing regression methods focus on estimating multiple clinical variables separately and thus cannot utilize the intrinsic useful correlation information among different clinical variables. On the other hand, in those regression methods, only a single modality of data (usually only the structural MRI) is often used, without considering the complementary information that can be provided by different modalities. In this paper, we propose a general methodology, namely multi-modal multi-task (M3T) learning, to jointly predict multiple variables from multi-modal data. Here, the variables include not only the clinical variables used for regression but also the categorical variable used for classification, with different tasks corresponding to prediction of different variables. Specifically, our method contains two key components, i.e., (1) a multi-task feature selection which selects the common subset of relevant features for multiple variables from each modality, and (2) a multi-modal support vector machine which fuses the above-selected features from all modalities to predict multiple (regression and classification) variables. To validate our method, we perform two sets of experiments on ADNI baseline MRI, FDG-PET, and cerebrospinal fluid (CSF) data from 45 AD patients, 91 MCI patients, and 50 healthy controls (HC). In the first set of experiments, we estimate two clinical variables such as Mini Mental State Examination (MMSE) and Alzheimer's Disease Assessment Scale-Cognitive Subscale (ADAS-Cog), as well as one categorical variable (with value of ‘AD’, ‘MCI’ or ‘HC’), from the baseline MRI, FDG-PET, and CSF data. In the second set of experiments, we predict the 2-year changes of MMSE and ADAS-Cog scores and also the conversion of MCI to AD from the baseline MRI, FDG-PET, and CSF data. The results on both sets of experiments demonstrate that our proposed M3T learning scheme can achieve better performance on both regression and classification tasks than the conventional learning methods.
Deep Learning in Medical Image Analysis Shen, Dinggang; Wu, Guorong; Suk, Heung-Il
Annual review of biomedical engineering,
06/2017, Volume:
19, Issue:
1
Journal Article
Peer reviewed
Open access
This review covers computer-assisted analysis of images in the field of medical imaging. Recent advances in machine learning, especially with regard to deep learning, are helping to identify, ...classify, and quantify patterns in medical images. At the core of these advances is the ability to exploit hierarchical feature representations learned solely from data, instead of features designed by hand according to domain-specific knowledge. Deep learning is rapidly becoming the state of the art, leading to enhanced performance in various medical applications. We introduce the fundamentals of deep learning methods and review their successes in image registration, detection of anatomical and cellular structures, tissue segmentation, computer-aided disease diagnosis and prognosis, and so on. We conclude by discussing research issues and suggesting future directions for further improvement.
Living in poverty places children at very high risk for problems across a variety of domains, including schooling, behavioral regulation, and health. Aspects of cognitive functioning, such as ...information processing, may underlie these kinds of problems. How might poverty affect the brain functions underlying these cognitive processes? Here, we address this question by observing and analyzing repeated measures of brain development of young children between five months and four years of age from economically diverse backgrounds (n = 77). In doing so, we have the opportunity to observe changes in brain growth as children begin to experience the effects of poverty. These children underwent MRI scanning, with subjects completing between 1 and 7 scans longitudinally. Two hundred and three MRI scans were divided into different tissue types using a novel image processing algorithm specifically designed to analyze brain data from young infants. Total gray, white, and cerebral (summation of total gray and white matter) volumes were examined along with volumes of the frontal, parietal, temporal, and occipital lobes. Infants from low-income families had lower volumes of gray matter, tissue critical for processing of information and execution of actions. These differences were found for both the frontal and parietal lobes. No differences were detected in white matter, temporal lobe volumes, or occipital lobe volumes. In addition, differences in brain growth were found to vary with socioeconomic status (SES), with children from lower-income households having slower trajectories of growth during infancy and early childhood. Volumetric differences were associated with the emergence of disruptive behavioral problems.
One of the major challenges in anatomical landmark detection, based on deep neural networks, is the limited availability of medical imaging data for network learning. To address this problem, we ...present a two-stage task-oriented deep learning method to detect large-scale anatomical landmarks simultaneously in real time, using limited training data. Specifically, our method consists of two deep convolutional neural networks (CNN), with each focusing on one specific task. Specifically, to alleviate the problem of limited training data, in the first stage, we propose a CNN based regression model using millions of image patches as input, aiming to learn inherent associations between local image patches and target anatomical landmarks. To further model the correlations among image patches, in the second stage, we develop another CNN model, which includes a) a fully convolutional network that shares the same architecture and network weights as the CNN used in the first stage and also b) several extra layers to jointly predict coordinates of multiple anatomical landmarks. Importantly, our method can jointly detect large-scale (e.g., thousands of) landmarks in real time. We have conducted various experiments for detecting 1200 brain landmarks from the 3D T1-weighted magnetic resonance images of 700 subjects, and also 7 prostate landmarks from the 3D computed tomography images of 73 subjects. The experimental results show the effectiveness of our method regarding both accuracy and efficiency in the anatomical landmark detection.
Automatic and reliable segmentation of the prostate is an important but difficult task for various clinical applications such as prostate cancer radiotherapy. The main challenges for accurate MR ...prostate localization lie in two aspects: (1) inhomogeneous and inconsistent appearance around prostate boundary, and (2) the large shape variation across different patients. To tackle these two problems, we propose a new deformable MR prostate segmentation method by unifying deep feature learning with the sparse patch matching. First, instead of directly using handcrafted features, we propose to learn the latent feature representation from prostate MR images by the stacked sparse auto-encoder (SSAE). Since the deep learning algorithm learns the feature hierarchy from the data, the learned features are often more concise and effective than the handcrafted features in describing the underlying data. To improve the discriminability of learned features, we further refine the feature representation in a supervised fashion. Second, based on the learned features, a sparse patch matching method is proposed to infer a prostate likelihood map by transferring the prostate labels from multiple atlases to the new prostate MR image. Finally, a deformable segmentation is used to integrate a sparse shape model with the prostate likelihood map for achieving the final segmentation. The proposed method has been extensively evaluated on the dataset that contains 66 T2-wighted prostate MR images. Experimental results show that the deep-learned features are more effective than the handcrafted features in guiding MR prostate segmentation. Moreover, our method shows superior performance than other state-of-the-art segmentation methods.
We previously presented an image registration method, referred to hierarchical attribute matching mechanism for elastic registration (HAMMER), which demonstrated relatively high accuracy in ...inter-subject registration of MR brain images. However, the HAMMER algorithm requires the pre-segmentation of brain tissues, since the attribute vectors used to hierarchically match the corresponding pairs of points are defined from the segmented image. In many applications, the segmentation of tissues might be difficult, unreliable or even impossible to complete, which potentially limits the use of the HAMMER algorithm in more generalized applications. To overcome this limitation, we have used local spatial intensity histograms to design a new type of attribute vector for each point in an intensity image. The histogram-based attribute vector is rotationally invariant, and importantly it also captures spatial information by integrating a number of local intensity histograms from multi-resolution images of original intensity image. The new attribute vectors are able to determine the corresponding points across individual images. Therefore, by hierarchically matching new attribute vectors, the proposed method can perform as successfully as the previous HAMMER algorithm did in registering MR brain images, while providing more generalized applications in registering images of various organs. Experimental results show good performance of the proposed method in registering MR brain images, DTI brain images, CT pelvis images, and MR mouse images.
Abstract
Retinal screening contributes to early detection of diabetic retinopathy and timely treatment. To facilitate the screening process, we develop a deep learning system, named DeepDR, that can ...detect early-to-late stages of diabetic retinopathy. DeepDR is trained for real-time image quality assessment, lesion detection and grading using 466,247 fundus images from 121,342 patients with diabetes. Evaluation is performed on a local dataset with 200,136 fundus images from 52,004 patients and three external datasets with a total of 209,322 images. The area under the receiver operating characteristic curves for detecting microaneurysms, cotton-wool spots, hard exudates and hemorrhages are 0.901, 0.941, 0.954 and 0.967, respectively. The grading of diabetic retinopathy as mild, moderate, severe and proliferative achieves area under the curves of 0.943, 0.955, 0.960 and 0.972, respectively. In external validations, the area under the curves for grading range from 0.916 to 0.970, which further supports the system is efficient for diabetic retinopathy grading.
As shown in the literature, methods based on multiple templates usually achieve better performance, compared with those using only a single template for processing medical images. However, most ...existing multi-template based methods simply average or concatenate multiple sets of features extracted from different templates, which potentially ignores important structural information contained in the multi-template data. Accordingly, in this paper, we propose a novel relationship induced multi-template learning method for automatic diagnosis of Alzheimer's disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI), by explicitly modeling structural information in the multi-template data. Specifically, we first nonlinearly register each brain's magnetic resonance (MR) image separately onto multiple pre-selected templates, and then extract multiple sets of features for this MR image. Next, we develop a novel feature selection algorithm by introducing two regularization terms to model the relationships among templates and among individual subjects. Using these selected features corresponding to multiple templates, we then construct multiple support vector machine (SVM) classifiers. Finally, an ensemble classification is used to combine outputs of all SVM classifiers, for achieving the final result. We evaluate our proposed method on 459 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, including 97 AD patients, 128 normal controls (NC), 117 progressive MCI (pMCI) patients, and 117 stable MCI (sMCI) patients. The experimental results demonstrate promising classification performance, compared with several state-of-the-art methods for multi-template based AD/MCI classification.
•A new measure to characterize the spatial variability of DCN is proposed.•A novel learning framework to integrate both temporal and spatial variabilities of DCNs is developed.•Achieving an accuracy ...of 78.8% and 78.3% for lMCI (late MCI) vs. eMCI (early MCI) classification and eMCI vs. NC (normal control) classification, respectively.•eMCI patients exhibit significant increased spatial variability.
Display omitted
Functional connectivity networks (FCNs) using resting-state functional magnetic resonance imaging (rs-fMRI) have been applied to the analysis and diagnosis of brain disease, such as Alzheimer’s disease (AD) and its prodrome, i.e., mild cognitive impairment (MCI). Different from conventional studies focusing on static descriptions on functional connectivity (FC) between brain regions in rs-fMRI, recent studies have resorted to dynamic connectivity networks (DCNs) to characterize the dynamic changes of FC, since dynamic changes of FC may indicate changes in macroscopic neural activity patterns in cognitive and behavioral aspects. However, most of the existing studies only investigate the temporal properties of DCNs (e.g., temporal variability of FC between specific brain regions), ignoring the important spatial properties of the network (e.g., spatial variability of FC associated with a specific brain region). Also, emerging evidence on FCNs has suggested that, besides temporal variability, there is significant spatial variability of activity foci over time. Hence, integrating both temporal and spatial properties of DCNs can intuitively promote the performance of connectivity-network-based learning methods. In this paper, we first define a new measure to characterize the spatial variability of DCNs, and then propose a novel learning framework to integrate both temporal and spatial variabilities of DCNs for automatic brain disease diagnosis. Specifically, we first construct DCNs from the rs-fMRI time series at successive non-overlapping time windows. Then, we characterize the spatial variability of a specific brain region by computing the correlation of functional sequences (i.e., the changing profile of FC between a pair of brain regions within all time windows) associated with this region. Furthermore, we extract both temporal variabilities and spatial variabilities from DCNs as features, and integrate them for classification by using manifold regularized multi-task feature learning and multi-kernel learning techniques. Results on 149 subjects with baseline rs-fMRI data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) suggest that our method can not only improve the classification performance in comparison with state-of-the-art methods, but also provide insights into the spatio-temporal interaction patterns of brain activity and their changes in brain disorders.
The segmentation of infant brain tissue images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) plays an important role in studying early brain development in health and ...disease. In the isointense stage (approximately 6–8 months of age), WM and GM exhibit similar levels of intensity in both T1 and T2 MR images, making the tissue segmentation very challenging. Only a small number of existing methods have been designed for tissue segmentation in this isointense stage; however, they only used a single T1 or T2 images, or the combination of T1 and T2 images. In this paper, we propose to use deep convolutional neural networks (CNNs) for segmenting isointense stage brain tissues using multi-modality MR images. CNNs are a type of deep models in which trainable filters and local neighborhood pooling operations are applied alternatingly on the raw input images, resulting in a hierarchy of increasingly complex features. Specifically, we used multi-modality information from T1, T2, and fractional anisotropy (FA) images as inputs and then generated the segmentation maps as outputs. The multiple intermediate layers applied convolution, pooling, normalization, and other operations to capture the highly nonlinear mappings between inputs and outputs. We compared the performance of our approach with that of the commonly used segmentation methods on a set of manually segmented isointense stage brain images. Results showed that our proposed model significantly outperformed prior methods on infant brain tissue segmentation. In addition, our results indicated that integration of multi-modality images led to significant performance improvement.
•We study the segmentation of isointense infant brain images.•We integrate multi-modality images.•We employ deep convolutional neural networks.•Integration of multi-modality images improves performance.•Deep convolutional neural networks outperform other methods.
PDF
