Statistical tools based on machine learning are becoming integrated into chemistry research workflows. We discuss the elements necessary to train reliable, repeatable and reproducible models, and ...recommend a set of guidelines for machine learning reports.
Introducción: la EPOC es altamente subdiagnosticada, por lo que aplicar escalas de predicción clínica como el COPD Diagnostic Questionnaire (CDQ) ayuda a abordar este problema. Sin embargo, se ...desconoce su rendimiento en nuestra población. Objetivo: evaluar la validez de criterio y reproducibilidad intraevaluador del CDQ en una población colombiana. Materiales y métodos: estudio de cohorte prospectivo. La eproducibilidad se valoró con el estadístico Kappa y el coeficiente de correlación intraclase (CCI). La validez se estableció con el puntaje obtenido del cuestionario CDQ y los valores espirométricos VEF1/CVF < 0,7 post B2. Se calculó el área bajo la curva de las características operativas del receptor (ACOR), sensibilidad, especificidad, valor predictivo positivo (VPP) y valor predictivo negativo (VPN). Se consideró una p estadísticamente significativa < 0,05. Resultados: ingresaron 1.423 sujetos, se encontró un CCI entre los dos momentos de la evaluación del cuestionario de 0,781 (IC95%: 0,744-0,813) y un kappa de 0,65 (p < 0,001) para los sistemas de puntuación con baja probabilidad de EPOC CDQ ≤ 16,5, probabilidad intermedia de EPOC CDQ > 16,5 y < 19,5 y probabilidad alta de EPOC CDQ ≥ 19,6. El ACOR fue de 0,68 (IC95%: 0,647-0,712) y se obtuvo una sensibilidad del 79% y una especificidad del 44% para un puntaje ≥ 16,5, con un VPP del 26% y un VPN del 86%. Conclusiones: el cuestionario CDQ tiene un rendimiento regular para el diagnóstico de la EPOC con una ACOR similar a otros cuestionarios, puede ser útil en pacientes subdiagnosticados y el punto de corte óptimo para esta población fue de 16,5.
Artificial intelligence (AI) has shown promise for diagnosing prostate cancer in biopsies. However, results have been limited to individual studies, lacking validation in multinational settings. ...Competitions have been shown to be accelerators for medical imaging innovations, but their impact is hindered by lack of reproducibility and independent validation. With this in mind, we organized the PANDA challenge-the largest histopathology competition to date, joined by 1,290 developers-to catalyze development of reproducible AI algorithms for Gleason grading using 10,616 digitized prostate biopsies. We validated that a diverse set of submitted algorithms reached pathologist-level performance on independent cross-continental cohorts, fully blinded to the algorithm developers. On United States and European external validation sets, the algorithms achieved agreements of 0.862 (quadratically weighted κ, 95% confidence interval (CI), 0.840-0.884) and 0.868 (95% CI, 0.835-0.900) with expert uropathologists. Successful generalization across different patient populations, laboratories and reference standards, achieved by a variety of algorithmic approaches, warrants evaluating AI-based Gleason grading in prospective clinical trials.
Background
The Prostate Imaging Reporting and Data System version 2 (PI‐RADSv2) has been in use since 2015; while interreader reproducibility has been studied, there has been a paucity of studies ...investigating the intrareader reproducibility of PI‐RADSv2.
Purpose
To evaluate both intra‐ and interreader reproducibility of PI‐RADSv2 in the assessment of intraprostatic lesions using multiparametric magnetic resonance imaging (mpMRI).
Study Type
Retrospective.
Population/Subjects
In all, 102 consecutive biopsy‐naïve patients who underwent prostate MRI and subsequent MR/transrectal ultrasonography (MR/TRUS)‐guided biopsy.
Field Strength/Sequences
Prostate mpMRI at 3T using endorectal with phased array surface coils (TW MRI, DW MRI with ADC maps and b2000 DW MRI, DCE MRI).
Assessment
Previously detected and biopsied lesions were scored by four readers from four different institutions using PI‐RADSv2. Readers scored lesions during two readout rounds with a 4‐week washout period.
Statistical Tests
Kappa (κ) statistics and specific agreement (Po) were calculated to quantify intra‐ and interreader reproducibility of PI‐RADSv2 scoring. Lesion measurement agreement was calculated using the intraclass correlation coefficient (ICC).
Results
Overall intrareader reproducibility was moderate to substantial (κ = 0.43–0.67, Po = 0.60–0.77), while overall interreader reproducibility was poor to moderate (κ = 0.24, Po = 46). Readers with more experience showed greater interreader reproducibility than readers with intermediate experience in the whole prostate (P = 0.026) and peripheral zone (P = 0.002). Sequence‐specific interreader agreement for all readers was similar to the overall PI‐RADSv2 score, with κ = 0.24, 0.24, and 0.23 and Po = 0.47, 0.44, and 0.54 in T2‐weighted, diffusion‐weighted imaging (DWI), and dynamic contrast‐enhanced (DCE), respectively. Overall intrareader and interreader ICC for lesion measurement was 0.82 and 0.71, respectively.
Data Conclusion
PI‐RADSv2 provides moderate intrareader reproducibility, poor interreader reproducibility, and moderate interreader lesion measurement reproducibility. These findings suggest a need for more standardized reader training in prostate MRI.
Level of Evidence: 2
Technical Efficacy: Stage 2
Vast 12-lead ECGs repositories provide opportunities to develop new machine learning approaches for creating accurate and automatic diagnostic systems for cardiac abnormalities. However, most 12-lead ...ECG classification studies are trained, tested, or developed in single, small, or relatively homogeneous datasets. In addition, most algorithms focus on identifying small numbers of cardiac arrhythmias that do not represent the complexity and difficulty of ECG interpretation. This work addresses these issues by providing a standard, multi-institutional database and a novel scoring metric through a public competition: the PhysioNet/Computing in Cardiology Challenge 2020.
A total of 66 361 12-lead ECG recordings were sourced from six hospital systems from four countries across three continents; 43 101 recordings were posted publicly with a focus on 27 diagnoses. For the first time in a public competition, we required teams to publish open-source code for both training and testing their algorithms, ensuring full scientific reproducibility.
A total of 217 teams submitted 1395 algorithms during the Challenge, representing a diversity of approaches for identifying cardiac abnormalities from both academia and industry. As with previous Challenges, high-performing algorithms exhibited significant drops (Formula: see text10%) in performance on the hidden test data.
Data from diverse institutions allowed us to assess algorithmic generalizability. A novel evaluation metric considered different misclassification errors for different cardiac abnormalities, capturing the outcomes and risks of different diagnoses. Requiring both trained models and code for training models improved the generalizability of submissions, setting a new bar in reproducibility for public data science competitions.
A wealth of analysis tools are available to fMRI researchers in order to extract patterns of task variation and, ultimately, understand cognitive function. However, this “methodological plurality” ...comes with a drawback. While conceptually similar, two different analysis pipelines applied on the same dataset may not produce the same scientific results. Differences in methods, implementations across software, and even operating systems or software versions all contribute to this variability. Consequently, attention in the field has recently been directed to reproducibility and data sharing. In this work, our goal is to understand how choice of software package impacts on analysis results. We use publicly shared data from three published task fMRI neuroimaging studies, reanalyzing each study using the three main neuroimaging software packages, AFNI, FSL, and SPM, using parametric and nonparametric inference. We obtain all information on how to process, analyse, and model each dataset from the publications. We make quantitative and qualitative comparisons between our replications to gauge the scale of variability in our results and assess the fundamental differences between each software package. Qualitatively we find similarities between packages, backed up by Neurosynth association analyses that correlate similar words and phrases to all three software package's unthresholded results for each of the studies we reanalyse. However, we also discover marked differences, such as Dice similarity coefficients ranging from 0.000 to 0.684 in comparisons of thresholded statistic maps between software. We discuss the challenges involved in trying to reanalyse the published studies, and highlight our efforts to make this research reproducible.
Using nationwide data, this study estimated and compared annual health care expenditures per person between noncancer and cancer patients, and among patients with the 4 most common cancers. Two-part ...models were used to estimate mean expenditures for each group by source of payment and by service type. We found that cancer patients had nearly 4 times higher mean expenditures per person ($16 346) than those without cancer ($4484). These differences were larger among individuals aged 18 to 64 years than those ≥65 years. Medicare was the largest source of payment for cancer patients, especially among those ≥65 years. Among the 4 most common cancers, the most costly cancer was lung cancer. Ambulatory care visits accounted for the majority of health care expenditures for those with breast cancer, while for those with other cancers, inpatient services also contributed to a significant portion of expenditures especially among younger patients. This study demonstrates that cancer patients experience a substantially higher health care expenditure burden than noncancer patients, with lung cancer patients having the highest expenditures. Expenditure estimates varied by age group, source of payment, and service type, highlighting the need for comprehensive policies and programs to reduce the costs of cancer care.
Data independent acquisition (DIA) mass spectrometry is a powerful technique that is improving the reproducibility and throughput of proteomics studies. Here, we introduce an experimental workflow ...that uses this technique to construct chromatogram libraries that capture fragment ion chromatographic peak shape and retention time for every detectable peptide in a proteomics experiment. These coordinates calibrate protein databases or spectrum libraries to a specific mass spectrometer and chromatography setup, facilitating DIA-only pipelines and the reuse of global resource libraries. We also present EncyclopeDIA, a software tool for generating and searching chromatogram libraries, and demonstrate the performance of our workflow by quantifying proteins in human and yeast cells. We find that by exploiting calibrated retention time and fragmentation specificity in chromatogram libraries, EncyclopeDIA can detect 20-25% more peptides from DIA experiments than with data dependent acquisition-based spectrum libraries alone.
The amino acid derivative reactivity assay (ADRA) is an in chemico alternative method that focuses on protein binding as the molecular initiating event for skin sensitization. It is a simple and ...versatile method that has successfully solved some of the problems of the direct peptide reactivity assay (DPRA). The transferability and within‐ and between‐laboratory reproducibility of ADRA were evaluated and confirmed as part of a validation study conducted at four participating laboratories. The transfer of ADRA technology from the lead laboratory to the four participating laboratories was completed successfully during a two‐step training program, after which the skin sensitization potentials of 40 coded chemicals were predicted based on the results of ADRA testing. Within‐laboratories reproducibility was 100% (10 of 10), 100% (10 of 10), 100% (7 of 7) and 90% (9 of 10), or an average of 97.3% (36 of 37); between‐laboratory reproducibility as calculated on the results of three laboratories at the time was 91.9%. The overall predictive capacity comprised an accuracy of 86.9%, sensitivity of 81.5% and specificity of 98.1%. These results satisfied the targets set by the validation management team for demonstrating transferability, within‐ and between‐laboratory reproducibility, and predictive capacity as well as gave a clear indication that ADRA is easily transferable and sufficiently robust to be used in place of DPRA.
ADRA is an in chemico alternative method that focuses on protein binding as the molecular initiating event for skin sensitization. The transferability, within‐laboratory reproducibility (WLR) and between‐laboratory reproducibility (BLR) of ADRA were evaluated and confirmed as part of a validation study. WLR, BLR, and accuracy as one of predictive capacity were an average of 97.3%, 91.9%, and 86.9%, respectively, indicating ADRA is easily transferable and sufficiently robust to be used in place of DPRA.
PDF
