International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices ...related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
Display omitted
•Evaluation framework for right ventricle segmentation in short-axis MR images.•The datasets available for training and testing are described, as well as the manual ...reference.•Evaluation measures include technical and clinical indices.•Results obtained by 7 methods are presented.•The framework is open for new submissions.
Magnetic Resonance Imaging (MRI), a reference examination for cardiac morphology and function in humans, allows to image the cardiac right ventricle (RV) with high spatial resolution. The segmentation of the RV is a difficult task due to the variable shape of the RV and its ill-defined borders in these images. The aim of this paper is to evaluate several RV segmentation algorithms on common data. More precisely, we report here the results of the Right Ventricle Segmentation Challenge (RVSC), concretized during the MICCAI’12 Conference with an on-site competition. Seven automated and semi-automated methods have been considered, along them three atlas-based methods, two prior based methods, and two prior-free, image-driven methods that make use of cardiac motion. The obtained contours were compared against a manual tracing by an expert cardiac radiologist, taken as a reference, using Dice metric and Hausdorff distance. We herein describe the cardiac data composed of 48 patients, the evaluation protocol and the results. Best results show that an average 80% Dice accuracy and a 1cm Hausdorff distance can be expected from semi-automated algorithms for this challenging task on the datasets, and that an automated algorithm can reach similar performance, at the expense of a high computational burden. Data are now publicly available and the website remains open for new submissions (http://www.litislab.eu/rvsc/).
Ischemic stroke, triggered by an obstruction in the cerebral blood supply, leads to infarction of the affected brain tissue. An accurate and reproducible automatic segmentation is of high interest, ...since the lesion volume is an important end-point for clinical trials. However, various factors, such as the high variance in lesion shape, location and appearance, render it a difficult task.
In this article, nine classification methods (e.g. Generalized Linear Models, Random Decision Forests and Convolutional Neural Networks) are evaluated and compared with each other using 37 multiparametric MRI datasets of ischemic stroke patients in the sub-acute phase in terms of their accuracy and reliability for ischemic stroke lesion segmentation. Within this context, a multi-spectral classification approach is compared against mono-spectral classification performance using only FLAIR MRI datasets and two sets of expert segmentations are used for inter-observer agreement evaluation.
The results of this study reveal that high-level machine learning methods lead to significantly better segmentation results compared to the rather simple classification methods, pointing towards a difficult non-linear problem. The overall best segmentation results were achieved by a Random Decision Forest and a Convolutional Neural Networks classification approach, even outperforming all previously published results. However, none of the methods tested in this work are capable of achieving results in the range of the human observer agreement and the automatic ischemic stroke lesion segmentation remains a complicated problem that needs to be explored in more detail to improve the segmentation results.
The Sørensen-Dice index (SDI) is a widely used measure for evaluating medical image segmentation algorithms. It offers a standardized measure of segmentation accuracy which has proven useful. ...However, it offers diminishing insight when the number of objects is unknown, such as in white matter lesion segmentation of multiple sclerosis (MS) patients. We present a refinement for finer grained parsing of SDI results in situations where the number of objects is unknown. We explore these ideas with two case studies showing what can be learned from our two presented studies. Our first study explores an inter-rater comparison, showing that smaller lesions cannot be reliably identified. In our second case study, we demonstrate fusing multiple MS lesion segmentation algorithms based on the insights into the algorithms provided by our analysis to generate a segmentation that exhibits improved performance. This work demonstrates the wealth of information that can be learned from refined analysis of medical image segmentations.
Performance of models highly depend not only on the used algorithm but also the data set it was applied to. This makes the comparison of newly developed tools to previously published approaches ...difficult. Either researchers need to implement others' algorithms first, to establish an adequate benchmark on their data, or a direct comparison of new and old techniques is infeasible. The Ischemic Stroke Lesion Segmentation (ISLES) challenge, which has ran now consecutively for 3 years, aims to address this problem of comparability. ISLES 2016 and 2017 focused on lesion outcome prediction after ischemic stroke: By providing a uniformly pre-processed data set, researchers from all over the world could apply their algorithm directly. A total of nine teams participated in ISLES 2015, and 15 teams participated in ISLES 2016. Their performance was evaluated in a fair and transparent way to identify the state-of-the-art among all submissions. Top ranked teams almost always employed deep learning tools, which were predominately convolutional neural networks (CNNs). Despite the great efforts, lesion outcome prediction persists challenging. The annotated data set remains publicly available and new approaches can be compared directly via the online evaluation system, serving as a continuing benchmark (www.isles-challenge.org).
In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of ...five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website22The Challenge Evaluation Website is: http://smart-stats-tools.org/lesion-challenge-2015 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.
•Public lesion data base of 21 training data sets and 61 testing data sets.•Fully automated evaluation website.•Comparison between 14 state-of-the-art algorithms and 2 manual delineators.
Display omitted
•Method for automatic sub-acute stroke lesion segmentation in routine clinical MR.•Flexibility and robustness renders method convenient for database screening.•High volume overlap of ...0.65 Dice's score over 37 clinical cases.•Segmentation accuracy independent of clinical parameters and stroke age.•Multi-spectral investigation established FLAIR as most important sequence.
To analyse the relationship between structure and (dys-)function of the brain after stroke, accurate and repeatable segmentation of the lesion area in magnetic resonance (MR) images is required. Manual delineation, the current gold standard, is time consuming and suffers from high intra- and inter-observer differences.
A new approach is presented for the automatic and reproducible segmentation of sub-acute ischemic stroke lesions in MR images in the presence of other pathologies. The proposition is based on an Extra Tree forest framework for voxel-wise classification and mainly intensity derived image features are employed.
A thorough investigation of multi-spectral variants, which combine the information from multiple MR sequences, finds the fluid attenuated inversion recovery sequence to be both required and sufficient for a good segmentation result. The accuracy can be further improved by adding features extracted from the T1-weighted and the diffusion weighted sequences. The use of other sequences is discouraged, as they impact negatively on the results.
Quantitative evaluation was carried out on 37 clinical cases. With a Dice coefficient of 0.65, the method outperforms earlier published methods.
The approach proves especially suitable to differentiate between new stroke and other white matter lesions based on the FLAIR sequence alone. This, and the high overlap, renders it suitable for automatic screening of large databases of MR scans, e.g. for a subsequent neuropsychological investigation. Finally, each feature's importance is assessed in detail and the approach's statistical dependency on clinical and image characteristics is investigated.
Voxel-wise classification for image segmentation often suffers the drawback, that the learnt global classification model only insufficiently captures sub-problems locally limited in problem space. We ...propose a novel method using spectral clustering to partition the global problem space into strongly connected clusters representing sub-problems. With fuzzy training set sampling, overlapping local problem classifiers are subsequently trained for each. Evaluation on a database of 37 magnetic resonance images displaying ischemic stroke lesions shows a significant improvement in segmentation accuracy compared to standard decision forest.
Developing a satisfactory and effective method for auto-annotating images that works under general conditions is a challenging task. The advantages of such a system would be manifold: it can be used ...to annotate existing, large databases of images, rendering them accessible to text search engines; or it can be used as core for image retrieval based on a query image’s visual content. Manual annotation of images is a difficult, tedious and time consuming task. Furthermore, manual annotations tend to show great inter-person variance: considering an image, the opinions about what elements are significant and deserve an annotation vary strongly. The latter poses a problem for the evaluation of an automatic method, as an annotation’s correctness is greatly subjective. In this paper we present an automatic method for annotating images, which addresses one of the existing methods’ major limitation, namely a fixed annotation length. The proposed method, PATSI, automatically chooses the resulting annotation’s length for each query image. It is held as simple as possible and a build-in parameter optimization procedure renders PATSI de-facto parameter free. Finally, PATSI is evaluated on standard datasets, outperforming various state-of-the-art methods.
PDF
