Diabetic retinopathy is a leading cause of preventable blindness, especially in low-income and middle-income countries (LMICs). Deep-learning systems have the potential to enhance diabetic ...retinopathy screenings in these settings, yet prospective studies assessing their usability and performance are scarce.
We did a prospective interventional cohort study to evaluate the real-world performance and feasibility of deploying a deep-learning system into the health-care system of Thailand. Patients with diabetes and listed on the national diabetes registry, aged 18 years or older, able to have their fundus photograph taken for at least one eye, and due for screening as per the Thai Ministry of Public Health guidelines were eligible for inclusion. Eligible patients were screened with the deep-learning system at nine primary care sites under Thailand's national diabetic retinopathy screening programme. Patients with a previous diagnosis of diabetic macular oedema, severe non-proliferative diabetic retinopathy, or proliferative diabetic retinopathy; previous laser treatment of the retina or retinal surgery; other non-diabetic retinopathy eye disease requiring referral to an ophthalmologist; or inability to have fundus photograph taken of both eyes for any reason were excluded. Deep-learning system-based interpretations of patient fundus images and referral recommendations were provided in real time. As a safety mechanism, regional retina specialists over-read each image. Performance of the deep-learning system (accuracy, sensitivity, specificity, positive predictive value PPV, and negative predictive value NPV) were measured against an adjudicated reference standard, provided by fellowship-trained retina specialists. This study is registered with the Thai national clinical trials registry, TCRT20190902002.
Between Dec 12, 2018, and March 29, 2020, 7940 patients were screened for inclusion. 7651 (96·3%) patients were eligible for study analysis, and 2412 (31·5%) patients were referred for diabetic retinopathy, diabetic macular oedema, ungradable images, or low visual acuity. For vision-threatening diabetic retinopathy, the deep-learning system had an accuracy of 94·7% (95% CI 93·0–96·2), sensitivity of 91·4% (87·1–95·0), and specificity of 95·4% (94·1–96·7). The retina specialist over-readers had an accuracy of 93·5 (91·7–95·0; p=0·17), a sensitivity of 84·8% (79·4–90·0; p=0·024), and specificity of 95·5% (94·1–96·7; p=0·98). The PPV for the deep-learning system was 79·2 (95% CI 73·8–84·3) compared with 75·6 (69·8–81·1) for the over-readers. The NPV for the deep-learning system was 95·5 (92·8–97·9) compared with 92·4 (89·3–95·5) for the over-readers.
A deep-learning system can deliver real-time diabetic retinopathy detection capability similar to retina specialists in community-based screening settings. Socioenvironmental factors and workflows must be taken into consideration when implementing a deep-learning system within a large-scale screening programme in LMICs.
Google and Rajavithi Hospital, Bangkok, Thailand.
For the Thai translation of the abstract see Supplementary Materials section.
While several papers have highlighted a lack of evidence to scale social innovations in health, fewer have explored decision-maker understandings of the relative merit of different types of evidence, ...how such data are interpreted and applied, and what practical support is required to improve evidence generation. The objectives of this paper are to understand (1) beliefs and attitudes towards the value of and types of evidence in scaling social innovations for health, (2) approaches to evidence generation and evaluation used in systems and policy change, and (3) how better evidence-generation can be undertaken and supported within social innovation in health.
Thirty-two one-on-one interviews were conducted between July and November 2015 with purposively selected practitioners, policymakers, and funders from low- and middle- income countries (LMICs). Data were analysed using a Framework Analysis Approach.
While practitioners, funders, and policymakers said they held outcome evidence in high regard, their practices only bear out this assertion to varying degrees. Few have given systematic consideration to potential unintended consequences, in particular harm, of the programs they implement, fund, or adopt. Stakeholders suggest that better evidence-generation can be undertaken and supported within social innovation in health by supporting the research efforts of emerging community organizations; creating links between practitioners and academia; altering the funding landscape for evidence-generation; providing responsive technical education; and creating accountability for funders, practitioners, and policymakers.
How better evidence-generation can be undertaken and supported within social innovation in health is a previously under-operationalised aspect of the policy-making process that remains essential in order to refrain from causing harm, enable the optimization of existing interventions, and ultimately, to scale and fund what works.
ObjectiveThe critical shortage of healthcare workers, particularly in rural areas, is a major barrier to quality care for non-communicable diseases (NCD) in low-income and middle-income countries. In ...this proof-of-concept study, we aimed to test a decentralised model for integrated diabetes and hypertension management in rural Bangladesh to improve accessibility and quality of care.Design and settingThe study is a single-cohort proof-of-concept study. The key interventions comprised shifting screening, routine monitoring and dispensing of medication refills from a doctor-managed subdistrict NCD clinic to non-physician health worker-managed village-level community clinics; a digital care coordination platform was developed for electronic health records, point-of-care support, referral and routine patient follow-up. The study was conducted in the Parbatipur subdistrict, Rangpur Division, Bangladesh.ParticipantsA total of 624 participants were enrolled in the study (mean (SD) age, 59.5 (12.0); 65.1% female).OutcomesChanges in blood pressure and blood glucose control, patient retention and patient-visit volume at the NCD clinic and community clinics.ResultsThe proportion of patients with uncontrolled blood pressure reduced from 60% at baseline to 26% at the third month of follow-up, a 56% (incidence rate ratio 0.44; 95% CI 0.33 to 0.57) reduction after adjustment for covariates. The proportion of patients with uncontrolled blood glucose decreased from 74% to 43% at the third month of follow-up. Attrition rates immediately after baseline and during the entire study period were 29.1% and 36.2%, respectively.ConclusionThe proof-of-concept study highlights the potential for involving lower-level primary care facilities and non-physician health workers to rapidly expand much-needed services to patients with hypertension and diabetes in Bangladesh and in similar global settings. Further investigations are needed to evaluate the effectiveness of decentralised hypertension and diabetes care.
Deep learning algorithms promise to improve clinician workflows and patient outcomes. However, these gains have yet to be fully demonstrated in real world clinical settings. In this paper, we ...describe a human-centered study of a deep learning system used in clinics for the detection of diabetic eye disease. From interviews and observation across eleven clinics in Thailand, we characterize current eye-screening workflows, user expectations for an AI-assisted screening process, and post-deployment experiences. Our findings indicate that several socio-environmental factors impact model performance, nursing workflows, and the patient experience. We draw on these findings to reflect on the value of conducting human-centered evaluative research alongside prospective evaluations of model accuracy.
Global health (GH) training is well established overseas (particularly in North America) and reflects an increasing focus on social accountability in medical education. Despite significant interest ...among trainees, GH is poorly integrated with specialty training programs in Australia. While there are numerous benefits from international rotations in resource-poor settings, there are also risks to the host community, trainee and training provider. Safe and effective placements rely on firm ethical foundations as well as strong and durable partnerships between Australian and overseas health services, educational institutions and GH agencies. More formal systems of GH training in Australia have the potential to produce fellows with the skills and knowledge necessary to engage in regional health challenges in a global context.
Introduction
Deep learning (DL) for screening diabetic retinopathy (DR) has the potential to address limited healthcare resources by enabling expanded access to healthcare. However, there is still ...limited health economic evaluation, particularly in low- and middle-income countries, on this subject to aid decision-making for DL adoption.
Methods
In the context of a middle-income country (MIC), using Thailand as a model, we constructed a decision tree-Markov hybrid model to estimate lifetime costs and outcomes of Thailand’s national DR screening program via DL and trained human graders (HG). We calculated the incremental cost-effectiveness ratio (ICER) between the two strategies. Sensitivity analyses were performed to probe the influence of modeling parameters.
Results
From a societal perspective, screening with DL was associated with a reduction in costs of ~ US$ 2.70, similar quality-adjusted life-years (QALY) of + 0.0043, and an incremental net monetary benefit of ~ US$ 24.10 in the base case. In sensitivity analysis, DL remained cost-effective even with a price increase from US$ 1.00 to US$ 4.00 per patient at a Thai willingness-to-pay threshold of ~ US$ 4.997 per QALY gained. When further incorporating recent findings suggesting improved compliance to treatment referral with DL, our analysis models effectiveness benefits of ~ US$ 20 to US$ 50 depending on compliance.
Conclusion
DR screening using DL in an MIC using Thailand as a model may result in societal cost-savings and similar health outcomes compared with HG. This study may provide an economic rationale to expand DL-based DR screening in MICs as an alternative solution for limited availability of skilled human resources for primary screening, particularly in MICs with similar prevalence of diabetes and low compliance to referrals for treatment.
Objective. To evaluate diabetic retinopathy (DR) screening via deep learning (DL) and trained human graders (HG) in a longitudinal cohort, as case spectrum shifts based on treatment referral and ...new-onset DR. Methods. We randomly selected patients with diabetes screened twice, two years apart within a nationwide screening program. The reference standard was established via adjudication by retina specialists. Each patient’s color fundus photographs were graded, and a patient was considered as having sight-threatening DR (STDR) if the worse eye had severe nonproliferative DR, proliferative DR, or diabetic macular edema. We compared DR screening via two modalities: DL and HG. For each modality, we simulated treatment referral by excluding patients with detected STDR from the second screening using that modality. Results. There were 5,738 patients (12.3% STDR) in the first screening. DL and HG captured different numbers of STDR cases, and after simulated referral and excluding ungradable cases, 4,148 and 4,263 patients remained in the second screening, respectively. The STDR prevalence at the second screening was 5.1% and 6.8% for DL- and HG-based screening, respectively. Along with the prevalence decrease, the sensitivity for both modalities decreased from the first to the second screening (DL: from 95% to 90%, p=0.008; HG: from 74% to 57%, p<0.001). At both the first and second screenings, the rate of false negatives for the DL was a fifth that of HG (0.5-0.6% vs. 2.9-3.2%). Conclusion. On 2-year longitudinal follow-up of a DR screening cohort, STDR prevalence decreased for both DL- and HG-based screening. Follow-up screenings in longitudinal DR screening can be more difficult and induce lower sensitivity for both DL and HG, though the false negative rate was substantially lower for DL. Our data may be useful for health-economics analyses of longitudinal screening settings.
Background/aimsDeep learning systems (DLSs) for diabetic retinopathy (DR) detection show promising results but can underperform in racial and ethnic minority groups, therefore external validation ...within these populations is critical for health equity. This study evaluates the performance of a DLS for DR detection among Indigenous Australians, an understudied ethnic group who suffer disproportionately from DR-related blindness.MethodsWe performed a retrospective external validation study comparing the performance of a DLS against a retinal specialist for the detection of more-than-mild DR (mtmDR), vision-threatening DR (vtDR) and all-cause referable DR. The validation set consisted of 1682 consecutive, single-field, macula-centred retinal photographs from 864 patients with diabetes (mean age 54.9 years, 52.4% women) at an Indigenous primary care service in Perth, Australia. Three-person adjudication by a panel of specialists served as the reference standard.ResultsFor mtmDR detection, sensitivity of the DLS was superior to the retina specialist (98.0% (95% CI, 96.5 to 99.4) vs 87.1% (95% CI, 83.6 to 90.6), McNemar’s test p<0.001) with a small reduction in specificity (95.1% (95% CI, 93.6 to 96.4) vs 97.0% (95% CI, 95.9 to 98.0), p=0.006). For vtDR, the DLS’s sensitivity was again superior to the human grader (96.2% (95% CI, 93.4 to 98.6) vs 84.4% (95% CI, 79.7 to 89.2), p<0.001) with a slight drop in specificity (95.8% (95% CI, 94.6 to 96.9) vs 97.8% (95% CI, 96.9 to 98.6), p=0.002). For all-cause referable DR, there was a substantial increase in sensitivity (93.7% (95% CI, 91.8 to 95.5) vs 74.4% (95% CI, 71.1 to 77.5), p<0.001) and a smaller reduction in specificity (91.7% (95% CI, 90.0 to 93.3) vs 96.3% (95% CI, 95.2 to 97.4), p<0.001).ConclusionThe DLS showed improved sensitivity and similar specificity compared with a retina specialist for DR detection. This demonstrates its potential to support DR screening among Indigenous Australians, an underserved population with a high burden of diabetic eye disease.
PDF
