This Viewpoint summarizes the 2019 AI in Healthcare report from the National Academy of Medicine (NAM), which reviews best practices for AI development, adoption, and maintenance and urges ...prioritization of equity, inclusion, and human rights in AI health system implementation.
Abstract
Objective
Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet ...calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.
Materials and Methods
Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.
Results
Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.
Conclusions
Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.
The study of acute kidney injury (AKI) has expanded with the increasing availability of electronic health records and the use of standardized definitions. Understanding the impact of AKI between ...settings is limited by heterogeneity in the selection of reference creatinine to anchor the definition of AKI. In this mini-review, we discuss different approaches used to select reference creatinine and their relative merits and limitations.
We reviewed the literature to obtain representative examples of published baseline creatinine definitions when pre-hospital data were not available, as well as literature evaluating the estimation of baseline renal function, using PubMed and reference back-tracing within known works.
(1) Pre-hospital creatinine values are useful in determining reference creatinine, and in high-risk populations, the mean outpatient serum creatinine value 7-365 days before hospitalization closely approximates nephrology adjudication, (2) in patients without pre-hospital data, the eGFR 75 approach does not reliably estimate true AKI incidence in most at-risk populations, (3) using the lowest inpatient serum creatinine may be reasonable, especially in those with preserved kidney function, but may generously estimate AKI incidence and severity and miss community-acquired AKI that does not fully resolve, (4) using more specific definitions of AKI (e.g., KIDGO stages 2 and 3) may help to reduce the effects of misclassification when using surrogate values and (5) leveraging available clinical data may help refine the estimate of reference creatinine.
Choosing reference creatinine for AKI calculation is important for AKI classification and study interpretation. We recommend obtaining data on pre-hospital kidney function, wherever possible. In studies where surrogate estimates are used, transparency in how they are applied and discussion that informs the reader of potential biases should be provided. Further work to refine the estimation of reference creatinine is needed.
Artificial intelligence (AI) has demonstrated promise in predicting acute kidney injury (AKI), however, clinical adoption of these models requires interpretability and transportability. ...Non-interoperable data across hospitals is a major barrier to model transportability. Here, we leverage the US PCORnet platform to develop an AKI prediction model and assess its transportability across six independent health systems. Our work demonstrates that cross-site performance deterioration is likely and reveals heterogeneity of risk factors across populations to be the cause. Therefore, no matter how accurate an AI model is trained at the source hospital, whether it can be adopted at target hospitals is an unanswered question. To fill the research gap, we derive a method to predict the transportability of AI models which can accelerate the adaptation process of external AI models in hospitals.
Observational research promises to complement experimental research by providing large, diverse populations that would be infeasible for an experiment. Observational research can test its own ...clinical hypotheses, and observational studies also can contribute to the design of experiments and inform the generalizability of experimental research. Understanding the diversity of populations and the variance in care is one component. In this study, the Observational Health Data Sciences and Informatics (OHDSI) collaboration created an international data network with 11 data sources from four countries, including electronic health records and administrative claims data on 250 million patients. All data were mapped to common data standards, patient privacy was maintained by using a distributed model, and results were aggregated centrally. Treatment pathways were elucidated for type 2 diabetes mellitus, hypertension, and depression. The pathways revealed that the world is moving toward more consistent therapy over time across diseases and across locations, but significant heterogeneity remains among sources, pointing to challenges in generalizing clinical trial results. Diabetes favored a single first-line medication, metformin, to a much greater extent than hypertension or depression. About 10% of diabetes and depression patients and almost 25% of hypertension patients followed a treatment pathway that was unique within the cohort. Aside from factors such as sample size and underlying population (academic medical center versus general population), electronic health records data and administrative claims data revealed similar results. Large-scale international observational research is feasible.
CONTEXT Currently most automated methods to identify patient safety occurrences rely on administrative data codes; however, free-text searches of electronic medical records could represent an ...additional surveillance approach. OBJECTIVE To evaluate a natural language processing search–approach to identify postoperative surgical complications within a comprehensive electronic medical record. DESIGN, SETTING, AND PATIENTS Cross-sectional study involving 2974 patients undergoing inpatient surgical procedures at 6 Veterans Health Administration (VHA) medical centers from 1999 to 2006. MAIN OUTCOME MEASURES Postoperative occurrences of acute renal failure requiring dialysis, deep vein thrombosis, pulmonary embolism, sepsis, pneumonia, or myocardial infarction identified through medical record review as part of the VA Surgical Quality Improvement Program. We determined the sensitivity and specificity of the natural language processing approach to identify these complications and compared its performance with patient safety indicators that use discharge coding information. RESULTS The proportion of postoperative events for each sample was 2% (39 of 1924) for acute renal failure requiring dialysis, 0.7% (18 of 2327) for pulmonary embolism, 1% (29 of 2327) for deep vein thrombosis, 7% (61 of 866) for sepsis, 16% (222 of 1405) for pneumonia, and 2% (35 of 1822) for myocardial infarction. Natural language processing correctly identified 82% (95% confidence interval CI, 67%-91%) of acute renal failure cases compared with 38% (95% CI, 25%-54%) for patient safety indicators. Similar results were obtained for venous thromboembolism (59%, 95% CI, 44%-72% vs 46%, 95% CI, 32%-60%), pneumonia (64%, 95% CI, 58%-70% vs 5%, 95% CI, 3%-9%), sepsis (89%, 95% CI, 78%-94% vs 34%, 95% CI, 24%-47%), and postoperative myocardial infarction (91%, 95% CI, 78%-97%) vs 89%, 95% CI, 74%-96%). Both natural language processing and patient safety indicators were highly specific for these diagnoses. CONCLUSION Among patients undergoing inpatient surgical procedures at VA medical centers, natural language processing analysis of electronic medical records to identify postoperative complications had higher sensitivity and lower specificity compared with patient safety indicators based on discharge coding.
Predictive analytics have begun to change the workflows of healthcare by giving insight into our future health. Deploying prognostic models into clinical workflows should change behavior and motivate ...interventions that affect outcomes. As users respond to model predictions, downstream characteristics of the data, including the distribution of the outcome, may change. The ever-changing nature of healthcare necessitates maintenance of prognostic models to ensure their longevity. The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade. Improving outcomes can disrupt the association between the model's predictors and the outcome. Model refitting may not always be the most effective response to these challenges. These problems will need to be mitigated by systematically incorporating interventions into prognostic models and by maintaining robust performance surveillance of models in clinical use. Holistically modeling the outcome and intervention(s) can lead to resilience to future compromises in performance.
Objectives This study sought to examine the contemporary incidence, predictors and outcomes of acute kidney injury in patients undergoing percutaneous coronary interventions. Background Acute kidney ...injury (AKI) is a serious and potentially preventable complication of percutaneous coronary interventions (PCIs) that is associated with adverse outcomes. The contemporary incidence, predictors, and outcomes of AKI are not well defined, and clarifying these can help identify high-risk patients for proactive prevention. Methods A total of 985,737 consecutive patients underwent PCIs at 1,253 sites participating in the National Cardiovascular Data Registry Cath-PCI registry from June 2009 through June 2011. AKI was defined on the basis of changes in serum creatinine level in the hospital according to the Acute Kidney Injury Network (AKIN) criteria. Using multivariable regression analyses with generalized estimating equations, we identified patient characteristics associated with AKI. Results Overall, 69,658 (7.1%) patients experienced AKI, with 3,005 (0.3%) requiring new dialysis. On multivariable analyses, the factors most strongly associated with development of AKI included ST-segment elevation myocardial infarction (STEMI) presentation (odds ratio OR: 2.60; 95% confidence interval CI: 2.53 to 2.67), severe chronic kidney disease (OR: 3.59; 95% CI: 3.47 to 3.71), and cardiogenic shock (OR: 2.92; 95% CI: 2.80 to 3.04). The in-hospital mortality rate was 9.7% for patients with AKI and 34% for those requiring dialysis compared with 0.5% for patients without AKI (p < 0.001). After multivariable adjustment, AKI (OR: 7.8; 95% CI: 7.4 to 8.1, p < 0.001) and dialysis (OR: 21.7; 95% CI: 19.6 to 24.1; p < 0.001) remained independent predictors of in-hospital mortality. Conclusions Approximately 7% of patients undergoing a PCI experience AKI, which is strongly associated with in-hospital mortality. Defining strategies to minimize the risk of AKI in patients undergoing PCI are needed to improve the safety and outcomes of the procedure.
OBJECTIVE: Hypoglycemia is associated with adverse outcomes in mixed populations of patients in intensive care units. It is not known whether the same risks exist for diabetic patients who are less ...severely ill. In this study, we aimed to determine whether hypoglycemic episodes are associated with higher mortality in diabetic patients hospitalized in the general ward. RESEARCH DESIGN AND METHODS: This retrospective cohort study analyzed 4,368 admissions of 2,582 patients with diabetes hospitalized in the general ward of a teaching hospital between January 2003 and August 2004. The associations between the number and severity of hypoglycemic (less-than or equal to50 mg/dl) episodes and inpatient mortality, length of stay (LOS), and mortality within 1 year after discharge were evaluated. RESULTS: Hypoglycemia was observed in 7.7% of admissions. In multivariable analysis, each additional day with hypoglycemia was associated with an increase of 85.3% in the odds of inpatient death (P = 0.009) and 65.8% (P = 0.0003) in the odds of death within 1 year from discharge. The odds of inpatient death also rose threefold for every 10 mg/dl decrease in the lowest blood glucose during hospitalization (P = 0.0058). LOS increased by 2.5 days for each day with hypoglycemia (P < 0.0001). CONCLUSIONS: Hypoglycemia is common in diabetic patients hospitalized in the general ward. Patients with hypoglycemia have increased LOS and higher mortality both during and after admission. Measures should be undertaken to decrease the frequency of hypoglycemia in this high-risk patient population.
Medical Devices in the Real World Resnic, Frederic S; Matheny, Michael E
The New England journal of medicine,
02/2018, Volume:
378, Issue:
7
Journal Article
Peer reviewed
Recent legislation directs the FDA to consider how best to use real-world evidence to ensure device safety and effectiveness while accelerating access to new technologies. But rigorous observational ...statistical methods are needed to mitigate limitations of such data.
PDF
