Abstract Objectives It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from ...“different but related” samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models. Study Design and Setting We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting. Results We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings. Conclusion The proposed framework enhances the interpretation of findings at external validation of prediction models.
Infection with Human Immunodeficiency virus (HIV) is an important risk factor for Tuberculosis (TB). Anti-Retroviral Therapy (ART) has improved the prognosis of HIV and reduced the risk of TB ...infected patients. Isoniazid Preventive Therapy (IPT) aims to reduce the development of active TB in patients with latent TB.
Systematically review and synthesize effect estimates of IPT for TB prevention in adult HIV patients. Secondary objectives were to assess the effect of IPT on HIV disease progression, all-cause mortality and adverse drug reaction (ADR).
Electronic databases were searched to identify relevant articles in English available by September 11th 2015.
Research articles comparing IPT to placebo or no treatment in HIV infected adults using randomized clinical trials.
A qualitative review included study-level information on randomization and treatment allocation. Effect estimates were pooled using random-effects models to account for between-study heterogeneity.
This review assessed ten randomized clinical trials that assigned 7619 HIV patients to IPT or placebo. An overall 35% of TB risk reduction (RR = 0.65, 95% CI (0.51, 0.84)) was found in all participants, however, larger benefit of IPT was observed in Tuberculin Skin Test (TST) positive participants, with pooled relative risk reduction of 52% RR = 0.48; 95% CI (0.29, 0.82) and with a prediction interval ranging from 0.13 to 1.81. There was no statistically significant effect of IPT on TB occurrence in TST negative or unknown participants. IPT also reduced the risk of HIV disease progression in all participants (RR = 0.69; 95% CI (0.48, 0.99)) despite no benefits observed in TST strata. All-cause mortality was not affected by IPT although participants who had 12 months of IPT tend to have a reduced risk (RR = 0.65; 95% CI(0.47, 0.90)). IPT had an elevated, yet statistically non-significant, risk of adverse drug reaction RR = 1.20; 95% CI (1.20, 1.71). Only a single study assessed the effect of IPT in combination with ART in preventing TB and occurrence of multi-drug resistant tuberculosis.
IPT use substantially contributes in preventing TB in persons with HIV in general and in TST positive individuals in particular. More evidence is needed to explain discrepancies in the protective effect of IPT in these individuals.
Celotno besedilo
Dostopno za:
DOBA, IZUM, KILJ, NUK, PILJ, PNG, SAZU, SIK, UILJ, UKNU, UL, UM, UPUK
Precision medicine research often searches for treatment‐covariate interactions, which refers to when a treatment effect (eg, measured as a mean difference, odds ratio, hazard ratio) changes across ...values of a participant‐level covariate (eg, age, gender, biomarker). Single trials do not usually have sufficient power to detect genuine treatment‐covariate interactions, which motivate the sharing of individual participant data (IPD) from multiple trials for meta‐analysis. Here, we provide statistical recommendations for conducting and planning an IPD meta‐analysis of randomized trials to examine treatment‐covariate interactions. For conduct, two‐stage and one‐stage statistical models are described, and we recommend: (i) interactions should be estimated directly, and not by calculating differences in meta‐analysis results for subgroups; (ii) interaction estimates should be based solely on within‐study information; (iii) continuous covariates and outcomes should be analyzed on their continuous scale; (iv) nonlinear relationships should be examined for continuous covariates, using a multivariate meta‐analysis of the trend (eg, using restricted cubic spline functions); and (v) translation of interactions into clinical practice is nontrivial, requiring individualized treatment effect prediction. For planning, we describe first why the decision to initiate an IPD meta‐analysis project should not be based on between‐study heterogeneity in the overall treatment effect; and second, how to calculate the power of a potential IPD meta‐analysis project in advance of IPD collection, conditional on characteristics (eg, number of participants, standard deviation of covariates) of the trials (potentially) promising their IPD. Real IPD meta‐analysis projects are used for illustration throughout.
The Framingham risk models and pooled cohort equations (PCE) are widely used and advocated in guidelines for predicting 10-year risk of developing coronary heart disease (CHD) and cardiovascular ...disease (CVD) in the general population. Over the past few decades, these models have been extensively validated within different populations, which provided mounting evidence that local tailoring is often necessary to obtain accurate predictions. The objective is to systematically review and summarize the predictive performance of three widely advocated cardiovascular risk prediction models (Framingham Wilson 1998, Framingham ATP III 2002 and PCE 2013) in men and women separately, to assess the generalizability of performance across different subgroups and geographical regions, and to determine sources of heterogeneity in the findings across studies.
A search was performed in October 2017 to identify studies investigating the predictive performance of the aforementioned models. Studies were included if they externally validated one or more of the original models in the general population for the same outcome as the original model. We assessed risk of bias for each validation and extracted data on population characteristics and model performance. Performance estimates (observed versus expected (OE) ratio and c-statistic) were summarized using a random effects models and sources of heterogeneity were explored with meta-regression.
The search identified 1585 studies, of which 38 were included, describing a total of 112 external validations. Results indicate that, on average, all models overestimate the 10-year risk of CHD and CVD (pooled OE ratio ranged from 0.58 (95% CI 0.43-0.73; Wilson men) to 0.79 (95% CI 0.60-0.97; ATP III women)). Overestimation was most pronounced for high-risk individuals and European populations. Further, discriminative performance was better in women for all models. There was considerable heterogeneity in the c-statistic between studies, likely due to differences in population characteristics.
The Framingham Wilson, ATP III and PCE discriminate comparably well but all overestimate the risk of developing CVD, especially in higher risk populations. Because the extent of miscalibration substantially varied across settings, we highly recommend that researchers further explore reasons for overprediction and that the models be updated for specific populations.
Celotno besedilo
Dostopno za:
DOBA, IZUM, KILJ, NUK, PILJ, PNG, SAZU, SIK, UILJ, UKNU, UL, UM, UPUK
Access to big datasets from e-health records and individual participant data (IPD) meta-analysis is signalling a new advent of external validation studies for clinical prediction models. In this ...article, the authors illustrate novel opportunities for external validation in big, combined datasets, while drawing attention to methodological challenges and reporting issues.
Prognostic factors are associated with the risk of future health outcomes in individuals with a particular health condition or some clinical start point (eg, a particular diagnosis). Research to ...identify genuine prognostic factors is important because these factors can help improve risk stratification, treatment, and lifestyle decisions, and the design of randomised trials. Although thousands of prognostic factor studies are published each year, often they are of variable quality and the findings are inconsistent. Systematic reviews and meta-analyses are therefore needed that summarise the evidence about the prognostic value of particular factors. In this article, the key steps involved in this review process are described.
We present and compare multiple imputation methods for multilevel continuous and binary data where variables are systematically and sporadically missing. The methods are compared from a theoretical ...point of view and through an extensive simulation study motivated by a real dataset comprising multiple studies. The comparisons show that these multiple imputation methods are the most appropriate to handle missing values in a multilevel setting and why their relative performances can vary according to the missing data pattern, the multilevel structure and the type of missing variables. This study shows that valid inferences can only be obtained if the dataset includes a large number of clusters. In addition, it highlights that heteroscedastic multiple imputation methods provide more accurate inferences than homoscedastic methods, which should be reserved for data with few individuals per cluster. Finally, guidelines are given to choose the most suitable multiple imputation method according to the structure of the data.
It is widely recommended that any developed—diagnostic or prognostic—prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When ...multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package “metamisc”.
Validation of prediction models is highly recommended and increasingly common in the literature. A systematic review of validation studies is therefore helpful, with meta-analysis needed to summarise ...the predictive performance of the model being validated across different settings and populations. This article provides guidance for researchers systematically reviewing and meta-analysing the existing evidence on a specific prediction model, discusses good practice when quantitatively summarising the predictive performance of the model across studies, and provides recommendations for interpreting meta-analysis estimates of model performance. We present key steps of the meta-analysis and illustrate each step in an example review, by summarising the discrimination and calibration performance of the EuroSCORE for predicting operative mortality in patients undergoing coronary artery bypass grafting.
Data sharing enhances the value of medical research and builds trust in clinical trials, but more biomedical researchers need to be trained in these approaches, which include meta-research, data ...science and ethical, legal and social issues.
PDF
