Missing data are much studied in epidemiology and statistics. Theoretical development and application of methods for handling missing data have mostly been conducted in the context of prospective ...research data and with a goal of description or causal explanation. However, it is now common to build predictive models using routinely collected data, where missing patterns may convey important information, and one might take a pragmatic approach to optimizing prediction. Therefore, different methods to handle missing data may be preferred. Furthermore, an underappreciated issue in prediction modeling is that the missing data method used in model development may not match the method used when a model is deployed. This may lead to overoptimistic assessments of model performance. For prediction, particularly with routinely collected data, methods for handling missing data that incorporate information within the missingness pattern should be explored and further developed. Where missing data methods differ between model development and model deployment, the implications of this must be explicitly evaluated. The trade-off between building a prediction model that is causally principled, and building a prediction model that maximizes the use of all available information, should be carefully considered and will depend on the intended use of the model.
Clinical prediction models aim to predict outcomes in individuals, to inform diagnosis or prognosis in healthcare. Hundreds of prediction models are published in the medical literature each year, yet ...many are developed using a dataset that is too small for the total number of participants or outcome events. This leads to inaccurate predictions and consequently incorrect healthcare decisions for some individuals. In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model.
Within routinely collected health data, missing data for an individual might provide useful information in itself. This occurs, for example, in the case of electronic health records, where the ...presence or absence of data is informative. While the naive use of missing indicators to try to exploit such information can introduce bias, its use in conjunction with multiple imputation may unlock the potential value of missingness to reduce bias in causal effect estimation, particularly in missing not at random scenarios and where missingness might be associated with unmeasured confounders.
We conducted a simulation study to determine when the use of a missing indicator, combined with multiple imputation, would reduce bias for causal effect estimation, under a range of scenarios including unmeasured variables, missing not at random, and missing at random mechanisms. We use directed acyclic graphs and structural models to elucidate a variety of causal structures of interest. We handled missing data using complete case analysis, and multiple imputation with and without missing indicator terms.
We find that multiple imputation combined with a missing indicator gives minimal bias for causal effect estimation in most scenarios. In particular the approach: 1) does not introduce bias in missing (completely) at random scenarios; 2) reduces bias in missing not at random scenarios where the missing mechanism depends on the missing variable itself; and 3) may reduce or increase bias when unmeasured confounding is present.
In the presence of missing data, careful use of missing indicators, combined with multiple imputation, can improve causal effect estimation when missingness is informative, and is not detrimental when missingness is at random.
No clear guidance exists on handling missing data at each stage of developing, validating and implementing a clinical prediction model (CPM). We aimed to review the approaches to handling missing ...data that underly the CPMs currently recommended for use in UK healthcare.
A descriptive cross-sectional meta-epidemiological study aiming to identify CPMs recommended by the National Institute for Health and Care Excellence (NICE), which summarized how missing data is handled across their pipelines.
A total of 23 CPMs were included through “sampling strategy.” Six missing data strategies were identified: complete case analysis (CCA), multiple imputation, imputation of mean values, k-nearest neighbours imputation, using an additional category for missingness, considering missing values as risk-factor-absent. 52% of the development articles and 48% of the validation articles did not report how missing data were handled. CCA was the most common approach used for development (40%) and validation (44%). At implementation, 57% of the CPMs required complete data entry, whilst 43% allowed missing values. Three CPMs had consistent paths in their pipelines.
A broad variety of methods for handling missing data underly the CPMs currently recommended for use in UK healthcare. Missing data handling strategies were generally inconsistent. Better quality assurance of CPMs needs greater clarity and consistency in handling of missing data.
Deaths in the first year of the Coronavirus Disease 2019 (COVID-19) pandemic in England and Wales were unevenly distributed socioeconomically and geographically. However, the full scale of ...inequalities may have been underestimated to date, as most measures of excess mortality do not adequately account for varying age profiles of deaths between social groups. We measured years of life lost (YLL) attributable to the pandemic, directly or indirectly, comparing mortality across geographic and socioeconomic groups.
We used national mortality registers in England and Wales, from 27 December 2014 until 25 December 2020, covering 3,265,937 deaths. YLLs (main outcome) were calculated using 2019 single year sex-specific life tables for England and Wales. Interrupted time-series analyses, with panel time-series models, were used to estimate expected YLL by sex, geographical region, and deprivation quintile between 7 March 2020 and 25 December 2020 by cause: direct deaths (COVID-19 and other respiratory diseases), cardiovascular disease and diabetes, cancer, and other indirect deaths (all other causes). Excess YLL during the pandemic period were calculated by subtracting observed from expected values. Additional analyses focused on excess deaths for region and deprivation strata, by age-group. Between 7 March 2020 and 25 December 2020, there were an estimated 763,550 (95% CI: 696,826 to 830,273) excess YLL in England and Wales, equivalent to a 15% (95% CI: 14 to 16) increase in YLL compared to the equivalent time period in 2019. There was a strong deprivation gradient in all-cause excess YLL, with rates per 100,000 population ranging from 916 (95% CI: 820 to 1,012) for the least deprived quintile to 1,645 (95% CI: 1,472 to 1,819) for the most deprived. The differences in excess YLL between deprivation quintiles were greatest in younger age groups; for all-cause deaths, a mean of 9.1 years per death (95% CI: 8.2 to 10.0) were lost in the least deprived quintile, compared to 10.8 (95% CI: 10.0 to 11.6) in the most deprived; for COVID-19 and other respiratory deaths, a mean of 8.9 years per death (95% CI: 8.7 to 9.1) were lost in the least deprived quintile, compared to 11.2 (95% CI: 11.0 to 11.5) in the most deprived. For all-cause mortality, estimated deaths in the most deprived compared to the most affluent areas were much higher in younger age groups, but similar for those aged 85 or over. There was marked variability in both all-cause and direct excess YLL by region, with the highest rates in the North West. Limitations include the quasi-experimental nature of the research design and the requirement for accurate and timely recording.
In this study, we observed strong socioeconomic and geographical health inequalities in YLL, during the first calendar year of the COVID-19 pandemic. These were in line with long-standing existing inequalities in England and Wales, with the most deprived areas reporting the largest numbers in potential YLL.
•The computer aided risk scoring system (CARSS), an automated tool that does not require additional data collection, was developed to monitor clinical deterioration in emergency medical ...admissions.•The CARSS model had good performance characteristics and external validation is necessary step to explore the generalisability of CARSS.•This study presents the external validation of the CARSS model using data from another independent but similar NHS trust hospital.•Our results show that the CARSS model has good discrimination and calibration after re-calibrating the calibration-in-the-large.
Clinical prediction models have the potential to improve the quality of care and enhance patient safety outcomes. A Computer-aided Risk Scoring system (CARSS) was previously developed to predict in-hospital mortality following emergency admissions based on routinely collected blood tests and vitals. We aimed to externally validate the CARSS model.
In this retrospective external validation study, we considered all adult (≥18 years) emergency medical admissions discharged between 11/11/2020 and 11/11/2022 from The Rotherham Foundation Trust (TRFT), UK. We assessed the predictive performance of the CARSS model based on its discriminative (c-statistic) and calibration characteristics (calibration slope and calibration plots).
Out of 32,774 admissions, 20,422 (62.3 %) admissions were included. The TRFT sample had similar demographic characteristics to the development sample but had higher mortality (6.1 % versus 5.7 %). The CARSS model demonstrated good discrimination (c-statistic 0.87 95 % CI 0.86–0.88) and good calibration to the TRFT dataset (slope = 1.03 95 % CI 0.98–1.08 intercept = 0 95 % CI −0.06–0.07) after re-calibrating for differences in baseline mortality (intercept = 0.96 95 % CI 0.90–1.03 before re-calibration).
In summary, the CARSS model is externally validated after correcting the baseline risk of death between development and validation datasets. External validation of the CARSS model showed that it under-predicted in-hospital mortality. Re-calibration of this model showed adequate performance in the TRFT dataset.
When developing a clinical prediction model, penalization techniques are recommended to address overfitting, as they shrink predictor effect estimates toward the null and reduce mean-square ...prediction error in new individuals. However, shrinkage and penalty terms (‘tuning parameters’) are estimated with uncertainty from the development data set. We examined the magnitude of this uncertainty and the subsequent impact on prediction model performance.
This study comprises applied examples and a simulation study of the following methods: uniform shrinkage (estimated via a closed-form solution or bootstrapping), ridge regression, the lasso, and elastic net.
In a particular model development data set, penalization methods can be unreliable because tuning parameters are estimated with large uncertainty. This is of most concern when development data sets have a small effective sample size and the model's Cox-Snell R2 is low. The problem can lead to considerable miscalibration of model predictions in new individuals.
Penalization methods are not a ‘carte blanche’; they do not guarantee a reliable prediction model is developed. They are more unreliable when needed most (i.e., when overfitting may be large). We recommend they are best applied with large effective sample sizes, as identified from recent sample size calculations that aim to minimize the potential for model overfitting and precisely estimate key parameters.
•When developing a clinical prediction model, penalization and shrinkage techniques are recommended to address overfitting.•Some methodology articles suggest penalization methods are a ‘carte blanche’ and resolve any issues to do with overfitting.•We show that penalization methods can be unreliable, as their unknown shrinkage and tuning parameter estimates are often estimated with large uncertainty.•Although penalization methods will, on average, improve on standard estimation methods, in a particular data set, they are often unreliable.•The most problematic data sets are those with small effective sample sizes and where the developed model has a Cox-Snell R2 far from 1, which is common for prediction models of binary and time-to-event outcomes.•Penalization methods are best used in situations when a sufficiently large development data set is available, as identified from sample size calculations to minimize the potential for model overfitting and precisely estimate key parameters.•When the sample size is adequately large, any of the studied penalization or shrinkage methods can be used, as they should perform similarly and better than unpenalized regression unless sample size is extremely large and Rapp2 is large.
People living with a long-term condition, such as chronic kidney disease (CKD), often suffer from multiple symptoms simultaneously, making symptom management challenging. This study aimed to identify ...symptom clusters in adults with CKD across treatment groups and investigate their association with people's ability to perform their usual activities.
We conducted a secondary analysis of both cross-sectional and longitudinal data collected as part of a national service improvement programme in 14 kidney centres in England, UK. This data included symptom severity (17 items, POS-S Renal) and the extent to which people had problems performing their usual activities (single item, EQ-5D-5L). We categorised data by treatment group: haemodialysis (n = 1,462), transplantation (n = 866), peritoneal dialysis (n = 127), or CKD without kidney replacement therapy (CKD non-KRT; n = 684). We used principal component analysis to identify symptom clusters per treatment group, and proportional odds models to assess the association between clusters and usual activities.
Overall, clusters related to: lack of energy and mobility; gastrointestinal; skin; and mental health. Across groups, the 'lack of energy and mobility' clusters were associated with having problems with usual activities, with odds ratios (OR) ranging between 1.24 (95% confidence interval CI, 1.21-1.57) for haemodialysis and 1.56 for peritoneal dialysis (95% CI, 1.28-1.90). This association was confirmed longitudinally in haemodialysis (n = 399) and transplant (n = 249) subgroups.
Our findings suggest that healthcare professionals should consider routinely assessing symptoms in the 'lack of energy & mobility' cluster in all people with CKD, regardless of whether they volunteer this information; not addressing these symptoms is likely to be related to them having problems with performing usual activities. Future studies should explore why symptoms within clusters commonly co-occur and how they interrelate. This will inform the development of cluster-level symptom management interventions with enhanced potential to improve outcomes for people with CKD.
Clinical prediction models (CPMs) can inform decision making about treatment initiation, which requires predicted risks assuming no treatment is given. However, this is challenging since CPMs are ...usually derived using data sets where patients received treatment, often initiated postbaseline as “treatment drop‐ins.” This study proposes the use of marginal structural models (MSMs) to adjust for treatment drop‐in. We illustrate the use of MSMs in the CPM framework through simulation studies that represent randomized controlled trials and real‐world observational data and the example of statin initiation for cardiovascular disease prevention. The simulations include a binary treatment and a covariate, each recorded at two timepoints and having a prognostic effect on a binary outcome. The bias in predicted risk was examined in a model ignoring treatment, a model fitted on treatment‐naïve patients (at baseline), a model including baseline treatment, and the MSM. In all simulation scenarios, all models except the MSM underestimated the risk of outcome given absence of treatment. These results were supported in the statin initiation example, which showed that ignoring statin initiation postbaseline resulted in models that significantly underestimated the risk of a cardiovascular disease event occurring within 10 years. Consequently, CPMs that do not acknowledge treatment drop‐in can lead to underallocation of treatment. In conclusion, when developing CPMs to predict treatment‐naïve risk, researchers should consider using MSMs to adjust for treatment drop‐in, and also seek to exploit the ability of MSMs to allow estimation of individual treatment effects.
PDF
