Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, ...showing extensive context and microscopic detail at the same time. Simultaneously, novel machine-learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, the so-called generative adversarial networks (GANs) applied to histological image data. Besides improving performance, GANs also enable previously intractable application scenarios in this field. However, GANs could exhibit a potential for introducing bias. Hereby, we summarize the recent state-of-the-art developments in a generalizing notation, present the main applications of GANs, and give an outlook of some chosen promising approaches and their possible future applications. In addition, we identify currently unavailable methods with potential for future applications.
The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. While manual examination of these images of considerable size is highly time consuming and error prone, state-of-the-art machine-learning approaches enable efficient, automated processing of whole-slide images. In this paper, we focus on a particularly powerful class of deep-learning architectures, the so-called generative adversarial networks. Over the past years, the high number of publications on this topic indicates a very high potential of generative adversarial networks in the field of digital pathology. In this survey, the most important publications are collected and categorized according to the techniques used and the aspired application scenario. We identify the main ideas and provide an outlook into the future.
Whole-slide scanners digitize microscopic tissue slides and thereby generate a large amount of digital image material. This advocates for methods facilitating (semi-)automated analysis. In this paper, we investigate generative adversarial networks, which are a powerful class of deep-learning-based approaches, useful in, for example, histological image analysis. The most important publications in the field of digital pathology are collected, summarized, and categorized according to the technical approaches employed and the aspired application scenarios. We identify the main findings and furthermore provide an outlook.
AIM: To further improve the endoscopic detection of intestinal mucosa alterations due to celiac disease(CD).METHODS: We assessed a hybrid approach based on the integration of expert knowledge into ...the computerbased classification pipeline. A total of 2835 endoscopic images from the duodenum were recorded in 290 children using the modified immersion technique(MIT). These children underwent routine upper endoscopy for suspected CD or non-celiac upper abdominal symptoms between August 2008 and December 2014. Blinded to the clinical data and biopsy results, three medical experts visually classified each image as normal mucosa(Marsh-0) or villous atrophy(Marsh-3). The experts’ decisions were further integrated into state-of-the-arttexture recognition systems. Using the biopsy results as the reference standard, the classification accuracies of this hybrid approach were compared to the experts’ diagnoses in 27 different settings.RESULTS: Compared to the experts’ diagnoses, in 24 of 27 classification settings(consisting of three imaging modalities, three endoscopists and three classification approaches), the best overall classification accuracies were obtained with the new hybrid approach. In 17 of 24 classification settings, the improvements achieved with the hybrid approach were statistically significant(P < 0.05). Using the hybrid approach classification accuracies between 94% and 100% were obtained. Whereas the improvements are only moderate in the case of the most experienced expert, the results of the less experienced expert could be improved significantly in 17 out of 18 classification settings. Furthermore, the lowest classification accuracy, based on the combination of one database and one specific expert, could be improved from 80% to 95%(P < 0.001).CONCLUSION: The overall classification performance of medical experts, especially less experienced experts, can be boosted significantly by integrating expert knowledge into computer-aided diagnosis systems.
Digital whole slides images contain an enormous amount of information providing a strong motivation for the development of automated image analysis tools. Particularly deep neural networks show high ...potential with respect to various tasks in the field of digital pathology. However, a limitation is given by the fact that typical deep learning algorithms require (manual) annotations in addition to the large amounts of image data, to enable effective training. Multiple instance learning exhibits a powerful tool for training deep neural networks in a scenario without fully annotated data. These methods are particularly effective in the domain of digital pathology, due to the fact that labels for whole slide images are often captured routinely, whereas labels for patches, regions, or pixels are not. This potential resulted in a considerable number of publications, with the vast majority published in the last four years. Besides the availability of digitized data and a high motivation from the medical perspective, the availability of powerful graphics processing units exhibits an accelerator in this field. In this paper, we provide an overview of widely and effectively used concepts of (deep) multiple instance learning approaches and recent advancements. We also critically discuss remaining challenges as well as future potential.
•A review of publications on multiple instance learning in digital pathology.•Due to a strong increase since 2019, we strongly focus on the last three years.•We structured according to the technically innovative building blocks.•We provide a discussion including future potential and limitations.
Frozen sections are rapidly generated during surgical interventions. This allows surgeons to wait for histological findings during the interventions in order to base intra-surgical decisions on the ...outcome of the histology. However, compared to paraffin sections the quality of frozen sections is often strongly reduced, leading to a lower diagnostic accuracy. Deep learning-based image translation technology facilitates a virtual conversion between different native imaging technologies with the potential of translating a frozen section into a virtual paraffin section. Stain normalization can be applied to adjust further unequal image characteristics. We investigated the effect of deep learning-based image translation, conventional image normalization and a combination of these techniques on computer aided decision support systems for thyroid cancer diagnostics. For classification, a bag-of-words approach, based on convolutional neural network features, k-means clustering and a support vector machine were employed. While stain normalization led to a decreased overall classification accuracy (0.703 vs 0.727), image translation led to an increased mean score (0.770). A combination of both, image translation and normalization increased the accuracy even further (0.844) and clearly reduced the gap to the post-operative paraffin sections (0.902). Deep learning-based image translation proved to be a powerful tool to enhance accuracy of computer aided diagnosis which clearly outperformed conventional stain translation. This work provides a strong motivation for performing a study with expert pathologists performing the categorization of frozen sections and the corresponding improved sections, to investigate whether a similar effect is achieved in a clinical setting.
•Image translation is employed to increase classification accuracy of frozen sections.•Assessment of deep learning-based techniques and conventional stain normalization.•A study is performed on computer-aided thyroid cancer diagnosis.•Scores clearly increase and come close to those of superior paraffin sections.
Whole slide images contain a magnitude of quantitative information that may not be fully explored in qualitative visual assessments. We propose: (1) a novel pipeline for extracting a comprehensive ...set of visual features, which are detectable by a pathologist, as well as sub-visual features, which are not discernible by human experts and (2) perform detailed analyses on renal images from mice with experimental unilateral ureteral obstruction. An important criterion for these features is that they are easy to interpret, as opposed to features obtained from neural networks. We extract and compare features from pathological and healthy control kidneys to learn how the compartments (glomerulus, Bowman's capsule, tubule, interstitium, artery, and arterial lumen) are affected by the pathology. We define feature selection methods to extract the most informative and discriminative features. We perform statistical analyses to understand the relation of the extracted features, both individually, and in combinations, with tissue morphology and pathology. Particularly for the presented case-study, we highlight features that are affected in each compartment. With this, prior biological knowledge, such as the increase in interstitial nuclei, is confirmed and presented in a quantitative way, alongside with novel findings, like color and intensity changes in glomeruli and Bowman's capsule. The proposed approach is therefore an important step towards quantitative, reproducible, and rater-independent analysis in histopathology.
The fast acquisition process of frozen sections allows surgeons to wait for histological findings during the interventions to base intrasurgical decisions on the outcome of the histology. Compared ...with paraffin sections, however, the quality of frozen sections is often strongly reduced, leading to a lower diagnostic accuracy. Deep neural networks are capable of modifying specific characteristics of digital histological images. Particularly, generative adversarial networks proved to be effective tools to learn about translation between two modalities, based on two unconnected data sets only. The positive effects of such deep learning-based image optimization on computer-aided diagnosis have already been shown. However, since fully automated diagnosis is controversial, the application of enhanced images for visual clinical assessment is currently probably of even higher relevance.
Three different deep learning-based generative adversarial networks were investigated. The methods were used to translate frozen sections into virtual paraffin sections. Overall, 40 frozen sections were processed. For training, 40 further paraffin sections were available. We investigated how pathologists assess the quality of the different image translation approaches and whether experts are able to distinguish between virtual and real digital pathology.
Pathologists' detection accuracy of virtual paraffin sections (from pairs consisting of a frozen and a paraffin section) was between 0.62 and 0.97. Overall, in 59% of images, the virtual section was assessed as more appropriate for a diagnosis. In 53% of images, the deep learning approach was preferred to conventional stain normalization (SN).
Overall, expert assessment indicated slightly improved visual properties of converted images and a high similarity to real paraffin sections. The observed high variability showed clear differences in personal preferences.
A major challenge in the field of segmentation in digital pathology is given by the high effort for manual data annotations in combination with many sources introducing variability in the image ...domain. This requires methods that are able to cope with variability without requiring to annotate a large amount of samples for each characteristic. In this paper, we develop approaches based on adversarial models for image-to-image translation relying on unpaired training. Specifically, we propose approaches for stain-independent supervised segmentation relying on image-to-image translation for obtaining an intermediate representation. Furthermore, we develop a fully-unsupervised segmentation approach exploiting image-to-image translation to convert from the image to the label domain. Finally, both approaches are combined to obtain optimum performance in unsupervised segmentation independent of the characteristics of the underlying stain. Experiments on patches showing kidney histology proof that stain-translation can be performed highly effectively and can be used for domain adaptation to obtain independence of the underlying stain. It is even capable of facilitating the underlying segmentation task, thereby boosting the accuracy if an appropriate intermediate stain is selected. Combining domain adaptation with unsupervised segmentation finally showed the most significant improvements.
With emerging treatment approaches, it is crucial to correctly diagnose and monitor hereditary and acquired polyneuropathies. This study aimed to assess the validity and accuracy of magnet resonance ...imaging (MRI)-based muscle volumetry.Using semi-automatic segmentations of upper- and lower leg muscles based on whole-body MRI and axial T1-weighted turbo spin-echo sequences, we compared and correlated muscle volumes, and clinical and neurophysiological parameters in demyelinating Charcot-Marie-Tooth disease (CMT) (
= 13), chronic inflammatory demyelinating polyneuropathy (CIDP) (
= 27), and other neuropathy (
= 17) patients.The muscle volumes of lower legs correlated with foot dorsiflexion strength (
< 0.0001), CMT Neuropathy Score 2 (
< 0.0001), early gait disorders (
= 0.0486), and in CIDP patients with tibial nerve conduction velocities (
= 0.0092). Lower (
= 0.0218) and upper (
= 0.0342) leg muscles were significantly larger in CIDP compared to CMT patients. At one-year follow-up (
= 15), leg muscle volumes showed no significant decrease.MRI muscle volumetry is a promising method to differentiate and characterize neuropathies in clinical practice.
Display omitted
•Segmenting whole slide images showing sparse objects-of-interest.•CNN cascades to cope with class imbalance.•Individually optimized convolutional neural networks.•Large experimental ...setting for evaluation on renal histology.•Superior segmentation outcomes.
Due to the increasing availability of whole slide scanners facilitating digitization of histopathological tissue, large amounts of digital image data are being generated. Accordingly, there is a strong demand for the development of computer based image analysis systems. Here, we address application scenarios in histopathology consisting of sparse, small objects-of-interest occurring in the large gigapixel images. To tackle the thereby arising challenges, we propose two different CNN cascade approaches which are subsequently applied to segment the glomeruli in whole slide images of the kidney and compared with conventional fully-convolutional networks. To facilitate unbiased evaluation, eight-fold cross-validation is performed and finally means and standard deviations are reported. Overall, with the best performing cascade approach, single CNNs are outperformed and a pixel-level Dice similarity coefficient of 0.90 is obtained (precision: 0.89, recall: 0.92). Combined with qualitative and further object-level analyses the obtained results are assessed as excellent also compared to previous approaches. We can state that especially one of the proposed cascade networks proved to be a highly powerful tool providing the best segmentation accuracies and also keeping the computing time at the lowest level. This work facilitates accurate automated segmentation of renal whole slide images which consequently allows fully-automated big data analyses for the assessment of medical treatments. Furthermore, this approach can also easily be adapted to other similar biomedical application scenarios.
PDF
