The SPIRIT 2013 statement aims to improve the completeness of clinical trial protocol reporting by providing evidence-based recommendations for the minimum set of items to be addressed. This guidance ...has been instrumental in promoting transparent evaluation of new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate their impact on health outcomes. The SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence) extension is a new reporting guideline for clinical trial protocols evaluating interventions with an AI component. It was developed in parallel with its companion statement for trial reports: CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence). Both guidelines were developed through a staged consensus process involving literature review and expert consultation to generate 26 candidate items, which were consulted upon by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed upon in a consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The SPIRIT-AI extension includes 15 new items that were considered sufficiently important for clinical trial protocols of AI interventions. These new items should be routinely reported in addition to the core SPIRIT 2013 items. SPIRIT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention will be integrated, considerations for the handling of input and output data, the human-AI interaction and analysis of error cases. SPIRIT-AI will help promote transparency and completeness for clinical trial protocols for AI interventions. Its use will assist editors and peer reviewers, as well as the general readership, to understand, interpret and critically appraise the design and risk of bias for a planned clinical trial.
Full text
Available for:
FZAB, GEOZS, IJS, IMTLJ, KILJ, KISLJ, MFDPS, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, SBMB, SBNM, UKNU, UL, UM, UPUK, VKSCE, ZAGLJ
The value of a diagnostic test is not simply measured by its accuracy, but depends on how it affects patient health. This article presents a framework for the design and interpretation of studies ...that evaluate the health consequences of new diagnostic tests
Full text
Available for:
BFBNIB, CMK, NMLJ, NUK, PNG, SAZU, UL, UM, UPUK
Background
The diagnostic challenges associated with the COVID‐19 pandemic resulted in rapid development of diagnostic test methods for detecting SARS‐CoV‐2 infection. Serology tests to detect the ...presence of antibodies to SARS‐CoV‐2 enable detection of past infection and may detect cases of SARS‐CoV‐2 infection that were missed by earlier diagnostic tests. Understanding the diagnostic accuracy of serology tests for SARS‐CoV‐2 infection may enable development of effective diagnostic and management pathways, inform public health management decisions and understanding of SARS‐CoV‐2 epidemiology.
Objectives
To assess the accuracy of antibody tests, firstly, to determine if a person presenting in the community, or in primary or secondary care has current SARS‐CoV‐2 infection according to time after onset of infection and, secondly, to determine if a person has previously been infected with SARS‐CoV‐2. Sources of heterogeneity investigated included: timing of test, test method, SARS‐CoV‐2 antigen used, test brand, and reference standard for non‐SARS‐CoV‐2 cases.
Search methods
The COVID‐19 Open Access Project living evidence database from the University of Bern (which includes daily updates from PubMed and Embase and preprints from medRxiv and bioRxiv) was searched on 30 September 2020. We included additional publications from the Evidence for Policy and Practice Information and Co‐ordinating Centre (EPPI‐Centre) ‘COVID‐19: Living map of the evidence’ and the Norwegian Institute of Public Health ’NIPH systematic and living map on COVID‐19 evidence’. We did not apply language restrictions.
Selection criteria
We included test accuracy studies of any design that evaluated commercially produced serology tests, targeting IgG, IgM, IgA alone, or in combination. Studies must have provided data for sensitivity, that could be allocated to a predefined time period after onset of symptoms, or after a positive RT‐PCR test. Small studies with fewer than 25 SARS‐CoV‐2 infection cases were excluded. We included any reference standard to define the presence or absence of SARS‐CoV‐2 (including reverse transcription polymerase chain reaction tests (RT‐PCR), clinical diagnostic criteria, and pre‐pandemic samples).
Data collection and analysis
We use standard screening procedures with three reviewers. Quality assessment (using the QUADAS‐2 tool) and numeric study results were extracted independently by two people. Other study characteristics were extracted by one reviewer and checked by a second. We present sensitivity and specificity with 95% confidence intervals (CIs) for each test and, for meta‐analysis, we fitted univariate random‐effects logistic regression models for sensitivity by eligible time period and for specificity by reference standard group. Heterogeneity was investigated by including indicator variables in the random‐effects logistic regression models. We tabulated results by test manufacturer and summarised results for tests that were evaluated in 200 or more samples and that met a modification of UK Medicines and Healthcare products Regulatory Agency (MHRA) target performance criteria.
Main results
We included 178 separate studies (described in 177 study reports, with 45 as pre‐prints) providing 527 test evaluations. The studies included 64,688 samples including 25,724 from people with confirmed SARS‐CoV‐2; most compared the accuracy of two or more assays (102/178, 57%). Participants with confirmed SARS‐CoV‐2 infection were most commonly hospital inpatients (78/178, 44%), and pre‐pandemic samples were used by 45% (81/178) to estimate specificity. Over two‐thirds of studies recruited participants based on known SARS‐CoV‐2 infection status (123/178, 69%). All studies were conducted prior to the introduction of SARS‐CoV‐2 vaccines and present data for naturally acquired antibody responses. Seventy‐nine percent (141/178) of studies reported sensitivity by week after symptom onset and 66% (117/178) for convalescent phase infection. Studies evaluated enzyme‐linked immunosorbent assays (ELISA) (165/527; 31%), chemiluminescent assays (CLIA) (167/527; 32%) or lateral flow assays (LFA) (188/527; 36%).
Risk of bias was high because of participant selection (172, 97%); application and interpretation of the index test (35, 20%); weaknesses in the reference standard (38, 21%); and issues related to participant flow and timing (148, 82%). We judged that there were high concerns about the applicability of the evidence related to participants in 170 (96%) studies, and about the applicability of the reference standard in 162 (91%) studies.
Average sensitivities for current SARS‐CoV‐2 infection increased by week after onset for all target antibodies. Average sensitivity for the combination of either IgG or IgM was 41.1% in week one (95% CI 38.1 to 44.2; 103 evaluations; 3881 samples, 1593 cases), 74.9% in week two (95% CI 72.4 to 77.3; 96 evaluations, 3948 samples, 2904 cases) and 88.0% by week three after onset of symptoms (95% CI 86.3 to 89.5; 103 evaluations, 2929 samples, 2571 cases). Average sensitivity during the convalescent phase of infection (up to a maximum of 100 days since onset of symptoms, where reported) was 89.8% for IgG (95% CI 88.5 to 90.9; 253 evaluations, 16,846 samples, 14,183 cases), 92.9% for IgG or IgM combined (95% CI 91.0 to 94.4; 108 evaluations, 3571 samples, 3206 cases) and 94.3% for total antibodies (95% CI 92.8 to 95.5; 58 evaluations, 7063 samples, 6652 cases). Average sensitivities for IgM alone followed a similar pattern but were of a lower test accuracy in every time slot.
Average specificities were consistently high and precise, particularly for pre‐pandemic samples which provide the least biased estimates of specificity (ranging from 98.6% for IgM to 99.8% for total antibodies).
Subgroup analyses suggested small differences in sensitivity and specificity by test technology however heterogeneity in study results, timing of sample collection, and smaller sample numbers in some groups made comparisons difficult. For IgG, CLIAs were the most sensitive (convalescent‐phase infection) and specific (pre‐pandemic samples) compared to both ELISAs and LFAs (P < 0.001 for differences across test methods). The antigen(s) used (whether from the Spike‐protein or nucleocapsid) appeared to have some effect on average sensitivity in the first weeks after onset but there was no clear evidence of an effect during convalescent‐phase infection.
Investigations of test performance by brand showed considerable variation in sensitivity between tests, and in results between studies evaluating the same test. For tests that were evaluated in 200 or more samples, the lower bound of the 95% CI for sensitivity was 90% or more for only a small number of tests (IgG, n = 5; IgG or IgM, n = 1; total antibodies, n = 4). More test brands met the MHRA minimum criteria for specificity of 98% or above (IgG, n = 16; IgG or IgM, n = 5; total antibodies, n = 7). Seven assays met the specified criteria for both sensitivity and specificity.
In a low‐prevalence (2%) setting, where antibody testing is used to diagnose COVID‐19 in people with symptoms but who have had a negative PCR test, we would anticipate that 1 (1 to 2) case would be missed and 8 (5 to 15) would be falsely positive in 1000 people undergoing IgG or IgM testing in week three after onset of SARS‐CoV‐2 infection.
In a seroprevalence survey, where prevalence of prior infection is 50%, we would anticipate that 51 (46 to 58) cases would be missed and 6 (5 to 7) would be falsely positive in 1000 people having IgG tests during the convalescent phase (21 to 100 days post‐symptom onset or post‐positive PCR) of SARS‐CoV‐2 infection.
Authors' conclusions
Some antibody tests could be a useful diagnostic tool for those in whom molecular‐ or antigen‐based tests have failed to detect the SARS‐CoV‐2 virus, including in those with ongoing symptoms of acute infection (from week three onwards) or those presenting with post‐acute sequelae of COVID‐19. However, antibody tests have an increasing likelihood of detecting an immune response to infection as time since onset of infection progresses and have demonstrated adequate performance for detection of prior infection for sero‐epidemiological purposes. The applicability of results for detection of vaccination‐induced antibodies is uncertain.
In many systematic reviews of test accuracy, the majority of included studies are judged at risk of bias through poor study design, inconsistent or unsatisfactory verification methods, lack of ...blinding of test results, or missing data (11). Most test accuracy studies estimate only the accuracy of an individual test. ...test selection is routinely based on evidence akin to noncomparative case-series. The reviews report predicted numbers of correct test results, false negatives and false positives for a cohort of a fixed size on the basis of summary estimates from the review, which provides both patients and clinicians with probabilistic information for alternative consequences they need to aid in decision-making. ...our primary research urgently needs to change gear so that our test evaluations make relevant comparisons between tests to inform decision-making; evaluate tests in defined clinical diagnostic pathways; publish protocols to provide transparency; prospectively register studies to protect against publication bias; and ensure that the results are reported fully, accurately, and without spin.
Background
COVID‐19 illness is highly variable, ranging from infection with no symptoms through to pneumonia and life‐threatening consequences. Symptoms such as fever, cough, or loss of sense of ...smell (anosmia) or taste (ageusia), can help flag early on if the disease is present. Such information could be used either to rule out COVID‐19 disease, or to identify people who need to go for COVID‐19 diagnostic tests. This is the second update of this review, which was first published in 2020.
Objectives
To assess the diagnostic accuracy of signs and symptoms to determine if a person presenting in primary care or to hospital outpatient settings, such as the emergency department or dedicated COVID‐19 clinics, has COVID‐19.
Search methods
We undertook electronic searches up to 10 June 2021 in the University of Bern living search database. In addition, we checked repositories of COVID‐19 publications. We used artificial intelligence text analysis to conduct an initial classification of documents. We did not apply any language restrictions.
Selection criteria
Studies were eligible if they included people with clinically suspected COVID‐19, or recruited known cases with COVID‐19 and also controls without COVID‐19 from a single‐gate cohort. Studies were eligible when they recruited people presenting to primary care or hospital outpatient settings. Studies that included people who contracted SARS‐CoV‐2 infection while admitted to hospital were not eligible. The minimum eligible sample size of studies was 10 participants. All signs and symptoms were eligible for this review, including individual signs and symptoms or combinations. We accepted a range of reference standards.
Data collection and analysis
Pairs of review authors independently selected all studies, at both title and , and full‐text stage. They resolved any disagreements by discussion with a third review author. Two review authors independently extracted data and assessed risk of bias using the QUADAS‐2 checklist, and resolved disagreements by discussion with a third review author. Analyses were restricted to prospective studies only. We presented sensitivity and specificity in paired forest plots, in receiver operating characteristic (ROC) space and in dumbbell plots. We estimated summary parameters using a bivariate random‐effects meta‐analysis whenever five or more primary prospective studies were available, and whenever heterogeneity across studies was deemed acceptable.
Main results
We identified 90 studies; for this update we focused on the results of 42 prospective studies with 52,608 participants. Prevalence of COVID‐19 disease varied from 3.7% to 60.6% with a median of 27.4%. Thirty‐five studies were set in emergency departments or outpatient test centres (46,878 participants), three in primary care settings (1230 participants), two in a mixed population of in‐ and outpatients in a paediatric hospital setting (493 participants), and two overlapping studies in nursing homes (4007 participants). The studies did not clearly distinguish mild COVID‐19 disease from COVID‐19 pneumonia, so we present the results for both conditions together.
Twelve studies had a high risk of bias for selection of participants because they used a high level of preselection to decide whether reverse transcription polymerase chain reaction (RT‐PCR) testing was needed, or because they enrolled a non‐consecutive sample, or because they excluded individuals while they were part of the study base. We rated 36 of the 42 studies as high risk of bias for the index tests because there was little or no detail on how, by whom and when, the symptoms were measured. For most studies, eligibility for testing was dependent on the local case definition and testing criteria that were in effect at the time of the study, meaning most people who were included in studies had already been referred to health services based on the symptoms that we are evaluating in this review.
The applicability of the results of this review iteration improved in comparison with the previous reviews. This version has more studies of people presenting to ambulatory settings, which is where the majority of assessments for COVID‐19 take place. Only three studies presented any data on children separately, and only one focused specifically on older adults.
We found data on 96 symptoms or combinations of signs and symptoms. Evidence on individual signs as diagnostic tests was rarely reported, so this review reports mainly on the diagnostic value of symptoms. Results were highly variable across studies. Most had very low sensitivity and high specificity. RT‐PCR was the most often used reference standard (40/42 studies).
Only cough (11 studies) had a summary sensitivity above 50% (62.4%, 95% CI 50.6% to 72.9%)); its specificity was low (45.4%, 95% CI 33.5% to 57.9%)). Presence of fever had a sensitivity of 37.6% (95% CI 23.4% to 54.3%) and a specificity of 75.2% (95% CI 56.3% to 87.8%). The summary positive likelihood ratio of cough was 1.14 (95% CI 1.04 to 1.25) and that of fever 1.52 (95% CI 1.10 to 2.10). Sore throat had a summary positive likelihood ratio of 0.814 (95% CI 0.714 to 0.929), which means that its presence increases the probability of having an infectious disease other than COVID‐19.
Dyspnoea (12 studies) and fatigue (8 studies) had a sensitivity of 23.3% (95% CI 16.4% to 31.9%) and 40.2% (95% CI 19.4% to 65.1%) respectively. Their specificity was 75.7% (95% CI 65.2% to 83.9%) and 73.6% (95% CI 48.4% to 89.3%). The summary positive likelihood ratio of dyspnoea was 0.96 (95% CI 0.83 to 1.11) and that of fatigue 1.52 (95% CI 1.21 to 1.91), which means that the presence of fatigue slightly increases the probability of having COVID‐19.
Anosmia alone (7 studies), ageusia alone (5 studies), and anosmia or ageusia (6 studies) had summary sensitivities below 50% but summary specificities over 90%. Anosmia had a summary sensitivity of 26.4% (95% CI 13.8% to 44.6%) and a specificity of 94.2% (95% CI 90.6% to 96.5%). Ageusia had a summary sensitivity of 23.2% (95% CI 10.6% to 43.3%) and a specificity of 92.6% (95% CI 83.1% to 97.0%). Anosmia or ageusia had a summary sensitivity of 39.2% (95% CI 26.5% to 53.6%) and a specificity of 92.1% (95% CI 84.5% to 96.2%). The summary positive likelihood ratios of anosmia alone and anosmia or ageusia were 4.55 (95% CI 3.46 to 5.97) and 4.99 (95% CI 3.22 to 7.75) respectively, which is just below our arbitrary definition of a 'red flag', that is, a positive likelihood ratio of at least 5. The summary positive likelihood ratio of ageusia alone was 3.14 (95% CI 1.79 to 5.51).
Twenty‐four studies assessed combinations of different signs and symptoms, mostly combining olfactory symptoms. By combining symptoms with other information such as contact or travel history, age, gender, and a local recent case detection rate, some multivariable prediction scores reached a sensitivity as high as 90%.
Authors' conclusions
Most individual symptoms included in this review have poor diagnostic accuracy. Neither absence nor presence of symptoms are accurate enough to rule in or rule out the disease. The presence of anosmia or ageusia may be useful as a red flag for the presence of COVID‐19. The presence of cough also supports further testing. There is currently no evidence to support further testing with PCR in any individuals presenting only with upper respiratory symptoms such as sore throat, coryza or rhinorrhoea.
Combinations of symptoms with other readily available information such as contact or travel history, or the local recent case detection rate may prove more useful and should be further investigated in an unselected population presenting to primary care or hospital outpatient settings.
The diagnostic accuracy of symptoms for COVID‐19 is moderate to low and any testing strategy using symptoms as selection mechanism will result in both large numbers of missed cases and large numbers of people requiring testing. Which one of these is minimised, is determined by the goal of COVID‐19 testing strategies, that is, controlling the epidemic by isolating every possible case versus identifying those with clinically important disease so that they can be monitored or treated to optimise their prognosis. The former will require a testing strategy that uses very few symptoms as entry criterion for testing, the latter could focus on more specific symptoms such as fever and anosmia.
Studies of diagnostic accuracy require more sophisticated methods for their meta-analysis than studies of therapeutic interventions. A number of different, and apparently divergent, methods for ...meta-analysis of diagnostic studies have been proposed, including two alternative approaches that are statistically rigorous and allow for between-study variability: the hierarchical summary receiver operating characteristic (ROC) model (Rutter and Gatsonis, 2001) and bivariate random-effects meta-analysis (van Houwelingen and others, 1993), (van Houwelingen and others, 2002), (Reitsma and others, 2005). We show that these two models are very closely related, and define the circumstances in which they are identical. We discuss the different forms of summary model output suggested by the two approaches, including summary ROC curves, summary points, confidence regions, and prediction regions.
Background
The standard method of diagnosing HIV in infants and children less than 18 months is with a nucleic acid amplification test reverse transcriptase polymerase chain reaction test (NAT ...RT‐PCR) detecting viral ribonucleic acid (RNA). Laboratory testing using the RT‐PCR platform for HIV infection is limited by poor access, logistical support, and delays in relaying test results and initiating therapy in low‐resource settings. The use of rapid diagnostic tests at or near the point‐of‐care (POC) can increase access to early diagnosis of HIV infection in infants and children less than 18 months of age and timely initiation of antiretroviral therapy (ART).
Objectives
To summarize the diagnostic accuracy of point‐of‐care nucleic acid‐based testing (POC NAT) to detect HIV‐1/HIV‐2 infection in infants and children aged 18 months or less exposed to HIV infection.
Search methods
We searched the Cochrane Central Register of Controlled Trials (CENTRAL) (until 2 February 2021), MEDLINE and Embase (until 1 February 2021), and LILACS and Web of Science (until 2 February 2021) with no language or publication status restriction. We also searched conference websites and clinical trial registries, tracked reference lists of included studies and relevant systematic reviews, and consulted experts for potentially eligible studies.
Selection criteria
We defined POC tests as rapid diagnostic tests conducted at or near the patient site. We included any primary study that compared the results of a POC NAT to a reference standard of laboratory NAT RT‐PCR or total nucleic acid testing to detect the presence or absence of HIV infection denoted by HIV viral nucleic acids in infants and children aged 18 months or less who were exposed to HIV‐1/HIV‐2 infection. We included cross‐sectional, prospective, and retrospective study designs and those that provided sufficient data to create the 2 × 2 table to calculate sensitivity and specificity. We excluded diagnostic case control studies with healthy controls.
Data collection and analysis
We extracted information on study characteristics using a pretested standardized data extraction form. We used the QUADAS‐2 (Quality Assessment of Diagnostic Accuracy Studies) tool to assess the risk of bias and applicability concerns of the included studies. Two review authors independently selected and assessed the included studies, resolving any disagreements by consensus. The unit of analysis was the participant. We first conducted preliminary exploratory analyses by plotting estimates of sensitivity and specificity from each study on forest plots and in receiver operating characteristic (ROC) space. For the overall meta‐analyses, we pooled estimates of sensitivity and specificity using the bivariate meta‐analysis model at a common threshold (presence or absence of infection).
Main results
We identified a total of 12 studies (15 evaluations, 15,120 participants). All studies were conducted in sub‐Saharan Africa. The ages of included infants and children in the evaluations were as follows: at birth (n = 6), ≤ 12 months (n = 3), ≤ 18 months (n = 5), and ≤ 24 months (n = 1). Ten evaluations were field evaluations of the POC NAT test at the point of care, and five were laboratory evaluations of the POC NAT tests.The POC NAT tests evaluated included Alere q HIV‐1/2 Detect qualitative test (recently renamed m‐PIMA q HIV‐1/2 Detect qualitative test) (n = 6), Xpert HIV‐1 qualitative test (n = 6), and SAMBA HIV‐1 qualitative test (n = 3).
POC NAT pooled sensitivity and specificity (95% confidence interval (CI)) against laboratory reference standard tests were 98.6% (96.1 to 99.5) (15 evaluations, 1728 participants) and 99.9% (99.7 to 99.9) (15 evaluations, 13,392 participants) in infants and children ≤ 18 months.
Risk of bias in the included studies was mostly low or unclear due to poor reporting. Five evaluations had some concerns for applicability for the index test, as they were POC tests evaluated in a laboratory setting, but there was no difference detected between settings in sensitivity (−1.3% (95% CI −4.1 to 1.5)); and specificity results were similar.
Authors' conclusions
For the diagnosis of HIV‐1/HIV‐2 infection, we found the sensitivity and specificity of POC NAT tests to be high in infants and children aged 18 months or less who were exposed to HIV infection.
AbstractThe SPIRIT 2013 (The Standard Protocol Items: Recommendations for Interventional Trials) statement aims to improve the completeness of clinical trial protocol reporting, by providing ...evidence-based recommendations for the minimum set of items to be addressed. This guidance has been instrumental in promoting transparent evaluation of new interventions. More recently, there is a growing recognition that interventions involving artificial intelligence need to undergo rigorous, prospective evaluation to demonstrate their impact on health outcomes.The SPIRIT-AI extension is a new reporting guideline for clinical trials protocols evaluating interventions with an AI component. It was developed in parallel with its companion statement for trial reports: CONSORT-AI. Both guidelines were developed using a staged consensus process, involving a literature review and expert consultation to generate 26 candidate items, which were consulted on by an international multi-stakeholder group in a 2-stage Delphi survey (103 stakeholders), agreed on in a consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants).The SPIRIT-AI extension includes 15 new items, which were considered sufficiently important for clinical trial protocols of AI interventions. These new items should be routinely reported in addition to the core SPIRIT 2013 items. SPIRIT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention will be integrated, considerations around the handling of input and output data, the human-AI interaction and analysis of error cases.SPIRIT-AI will help promote transparency and completeness for clinical trial protocols for AI interventions. Its use will assist editors and peer-reviewers, as well as the general readership, to understand, interpret and critically appraise the design and risk of bias for a planned clinical trial.
Background
Early accurate detection of all skin cancer types is essential to guide appropriate management and to improve morbidity and survival. Melanoma and cutaneous squamous cell carcinoma (cSCC) ...are high‐risk skin cancers which have the potential to metastasise and ultimately lead to death, whereas basal cell carcinoma (BCC) is usually localised with potential to infiltrate and damage surrounding tissue. Anxiety around missing early curable cases needs to be balanced against inappropriate referral and unnecessary excision of benign lesions. Computer‐assisted diagnosis (CAD) systems use artificial intelligence to analyse lesion data and arrive at a diagnosis of skin cancer. When used in unreferred settings ('primary care'), CAD may assist general practitioners (GPs) or other clinicians to more appropriately triage high‐risk lesions to secondary care. Used alongside clinical and dermoscopic suspicion of malignancy, CAD may reduce unnecessary excisions without missing melanoma cases.
Objectives
To determine the accuracy of CAD systems for diagnosing cutaneous invasive melanoma and atypical intraepidermal melanocytic variants, BCC or cSCC in adults, and to compare its accuracy with that of dermoscopy.
Search methods
We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials (CENTRAL); MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles.
Selection criteria
Studies of any design that evaluated CAD alone, or in comparison with dermoscopy, in adults with lesions suspicious for melanoma or BCC or cSCC, and compared with a reference standard of either histological confirmation or clinical follow‐up.
Data collection and analysis
Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS‐2). We contacted authors of included studies where information related to the target condition or diagnostic threshold were missing. We estimated summary sensitivities and specificities separately by type of CAD system, using the bivariate hierarchical model. We compared CAD with dermoscopy using (a) all available CAD data (indirect comparisons), and (b) studies providing paired data for both tests (direct comparisons). We tested the contribution of human decision‐making to the accuracy of CAD diagnoses in a sensitivity analysis by removing studies that gave CAD results to clinicians to guide diagnostic decision‐making.
Main results
We included 42 studies, 24 evaluating digital dermoscopy‐based CAD systems (Derm–CAD) in 23 study cohorts with 9602 lesions (1220 melanomas, at least 83 BCCs, 9 cSCCs), providing 32 datasets for Derm–CAD and seven for dermoscopy. Eighteen studies evaluated spectroscopy‐based CAD (Spectro–CAD) in 16 study cohorts with 6336 lesions (934 melanomas, 163 BCC, 49 cSCCs), providing 32 datasets for Spectro–CAD and six for dermoscopy. These consisted of 15 studies using multispectral imaging (MSI), two studies using electrical impedance spectroscopy (EIS) and one study using diffuse‐reflectance spectroscopy. Studies were incompletely reported and at unclear to high risk of bias across all domains. Included studies inadequately address the review question, due to an abundance of low‐quality studies, poor reporting, and recruitment of highly selected groups of participants.
Across all CAD systems, we found considerable variation in the hardware and software technologies used, the types of classification algorithm employed, methods used to train the algorithms, and which lesion morphological features were extracted and analysed across all CAD systems, and even between studies evaluating CAD systems. Meta–analysis found CAD systems had high sensitivity for correct identification of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants in highly selected populations, but with low and very variable specificity, particularly for Spectro–CAD systems. Pooled data from 22 studies estimated the sensitivity of Derm–CAD for the detection of melanoma as 90.1% (95% confidence interval (CI) 84.0% to 94.0%) and specificity as 74.3% (95% CI 63.6% to 82.7%). Pooled data from eight studies estimated the sensitivity of multispectral imaging CAD (MSI–CAD) as 92.9% (95% CI 83.7% to 97.1%) and specificity as 43.6% (95% CI 24.8% to 64.5%). When applied to a hypothetical population of 1000 lesions at the mean observed melanoma prevalence of 20%, Derm–CAD would miss 20 melanomas and would lead to 206 false‐positive results for melanoma. MSI–CAD would miss 14 melanomas and would lead to 451 false diagnoses for melanoma. Preliminary findings suggest CAD systems are at least as sensitive as assessment of dermoscopic images for the diagnosis of invasive melanoma and atypical intraepidermal melanocytic variants. We are unable to make summary statements about the use of CAD in unreferred populations, or its accuracy in detecting keratinocyte cancers, or its use in any setting as a diagnostic aid, because of the paucity of studies.
Authors' conclusions
In highly selected patient populations all CAD types demonstrate high sensitivity, and could prove useful as a back‐up for specialist diagnosis to assist in minimising the risk of missing melanomas. However, the evidence base is currently too poor to understand whether CAD system outputs translate to different clinical decision–making in practice. Insufficient data are available on the use of CAD in community settings, or for the detection of keratinocyte cancers. The evidence base for individual systems is too limited to draw conclusions on which might be preferred for practice. Prospective comparative studies are required that evaluate the use of already evaluated CAD systems as diagnostic aids, by comparison to face–to–face dermoscopy, and in participant populations that are representative of those in which the test would be used in practice.
Objective Adrenal masses are incidentally discovered in 5% of CT scans. In 2013/2014, 81 million CT examinations were undertaken in the USA and 5 million in the UK. However, uncertainty remains ...around the optimal imaging approach for diagnosing malignancy. We aimed to review the evidence on the accuracy of imaging tests for differentiating malignant from benign adrenal masses. Design A systematic review and meta-analysis was conducted. Methods We searched MEDLINE, EMBASE, Cochrane CENTRAL Register of Controlled Trials, Science Citation Index, Conference Proceedings Citation Index, and ZETOC (January 1990 to August 2015). We included studies evaluating the accuracy of CT, MRI, or 18F-fluoro-deoxyglucose (FDG)-PET compared with an adequate histological or imaging-based follow-up reference standard. Results We identified 37 studies suitable for inclusion, after screening 5469 references and 525 full-text articles. Studies evaluated the accuracy of CT (n=16), MRI (n=15), and FDG-PET (n=9) and were generally small and at high or unclear risk of bias. Only 19 studies were eligible for meta-analysis. Limited data suggest that CT density >10HU has high sensitivity for detection of adrenal malignancy in participants with no prior indication for adrenal imaging, that is, masses with ≤10HU are unlikely to be malignant. All other estimates of test performance are based on too small numbers. Conclusions Despite their widespread use in routine assessment, there is insufficient evidence for the diagnostic value of individual imaging tests in distinguishing benign from malignant adrenal masses. Future research is urgently needed and should include prospective test validation studies for imaging and novel diagnostic approaches alongside detailed health economics analysis.
PDF
