Network meta-analysis synthesizes direct and indirect evidence in a network of trials that compare multiple interventions and has the potential to rank the competing treatments according to the ...studied outcome. Despite its usefulness network meta-analysis is often criticized for its complexity and for being accessible only to researchers with strong statistical and computational skills. The evaluation of the underlying model assumptions, the statistical technicalities and presentation of the results in a concise and understandable way are all challenging aspects in the network meta-analysis methodology. In this paper we aim to make the methodology accessible to non-statisticians by presenting and explaining a series of graphical tools via worked examples. To this end, we provide a set of STATA routines that can be easily employed to present the evidence base, evaluate the assumptions, fit the network meta-analysis model and interpret its results.
Systematic reviews that collate data about the relative effects of multiple interventions via network meta-analysis are highly informative for decision-making purposes. A network meta-analysis ...provides two types of findings for a specific outcome: the relative treatment effect for all pairwise comparisons, and a ranking of the treatments. It is important to consider the confidence with which these two types of results can enable clinicians, policy makers and patients to make informed decisions. We propose an approach to determining confidence in the output of a network meta-analysis. Our proposed approach is based on methodology developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group for pairwise meta-analyses. The suggested framework for evaluating a network meta-analysis acknowledges (i) the key role of indirect comparisons (ii) the contributions of each piece of direct evidence to the network meta-analysis estimates of effect size; (iii) the importance of the transitivity assumption to the validity of network meta-analysis; and (iv) the possibility of disagreement between direct evidence and indirect evidence. We apply our proposed strategy to a systematic review comparing topical antibiotics without steroids for chronically discharging ears with underlying eardrum perforations. The proposed framework can be used to determine confidence in the results from a network meta-analysis. Judgements about evidence from a network meta-analysis can be different from those made about evidence from pairwise meta-analyses.
Despite a major increase in the range and number of software offerings now available to help researchers produce evidence syntheses, there is currently no generic tool for producing figures to ...display and explore the risk‐of‐bias assessments that routinely take place as part of systematic review. However, tools such as the R programming environment and Shiny (an R package for building interactive web apps) have made it straightforward to produce new tools to help in producing evidence syntheses. We present a new tool, robvis (Risk‐Of‐Bias VISualization), available as an R package and web app, which facilitates rapid production of publication‐quality risk‐of‐bias assessment figures. We present a timeline of the tool's development and its key functionality.
The evaluation of the credibility of results from a meta-analysis has become an important part of the evidence synthesis process. We present a methodological framework to evaluate confidence in the ...results from network meta-analyses, Confidence in Network Meta-Analysis (CINeMA), when multiple interventions are compared.
CINeMA considers 6 domains: (i) within-study bias, (ii) reporting bias, (iii) indirectness, (iv) imprecision, (v) heterogeneity, and (vi) incoherence. Key to judgments about within-study bias and indirectness is the percentage contribution matrix, which shows how much information each study contributes to the results from network meta-analysis. The contribution matrix can easily be computed using a freely available web application. In evaluating imprecision, heterogeneity, and incoherence, we consider the impact of these components of variability in forming clinical decisions.
Via 3 examples, we show that CINeMA improves transparency and avoids the selective use of evidence when forming judgments, thus limiting subjectivity in the process. CINeMA is easy to apply even in large and complicated networks.
The increase in treatment options creates an urgent need for comparative effectiveness research. Randomized, controlled trials comparing several treatments are usually not feasible, so other ...methodological approaches are needed. Meta-analyses provide summary estimates of treatment effects by combining data from many studies. However, an important drawback is that standard meta-analyses can compare only 2 interventions at a time. A new meta-analytic technique, called network meta-analysis (or multiple treatments meta-analysis or mixed-treatment comparison), allows assessment of the relative effectiveness of several interventions, synthesizing evidence across a network of randomized trials. Despite the growing prevalence and influence of network meta-analysis in many fields of medicine, several issues need to be addressed when constructing one to avoid conclusions that are inaccurate, invalid, or not clearly justified. This article explores the scope and limitations of network meta-analysis and offers advice on dealing with heterogeneity, inconsistency, and potential sources of bias in the available evidence to increase awareness among physicians about some of the challenges in interpretation.
To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials.
Systematic review of meta-epidemiological studies.
We retrieved ...eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic.
We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear.
Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.
Abstract Objective To develop ROBIS, a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). Study Design and Setting We used four-stage approach to develop ...ROBIS: define the scope, review the evidence base, hold a face-to-face meeting, and refine the tool through piloting. Results ROBIS is currently aimed at four broad categories of reviews mainly within health care settings: interventions, diagnosis, prognosis, and etiology. The target audience of ROBIS is primarily guideline developers, authors of overviews of systematic reviews (“reviews of reviews”), and review authors who might want to assess or avoid risk of bias in their reviews. The tool is completed in three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias. Phase 2 covers four domains through which bias may be introduced into a systematic review: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings. Phase 3 assesses the overall risk of bias in the interpretation of review findings and whether this considered limitations identified in any of the phase 2 domains. Signaling questions are included to help judge concerns with the review process (phase 2) and the overall risk of bias in the review (phase 3); these questions flag aspects of review design related to the potential for bias and aim to help assessors judge risk of bias in the review process, results, and conclusions. Conclusions ROBIS is the first rigorously developed tool designed specifically to assess the risk of bias in systematic reviews.
Narrative reviews of paediatric NAFLD quote prevalences in the general population that range from 9% to 37%; however, no systematic review of the prevalence of NAFLD in children/adolescents has been ...conducted. We aimed to estimate prevalence of non-alcoholic fatty liver disease (NAFLD) in young people and to determine whether this varies by BMI category, gender, age, diagnostic method, geographical region and study sample size.
We conducted a systematic review and meta-analysis of all studies reporting a prevalence of NAFLD based on any diagnostic method in participants 1-19 years old, regardless of whether assessing NAFLD prevalence was the main aim of the study.
The pooled mean prevalence of NAFLD in children from general population studies was 7.6% (95%CI: 5.5% to 10.3%) and 34.2% (95% CI: 27.8% to 41.2%) in studies based on child obesity clinics. In both populations there was marked heterogeneity between studies (I2 = 98%). There was evidence that prevalence was generally higher in males compared with females and increased incrementally with greater BMI. There was evidence for differences between regions in clinical population studies, with estimated prevalence being highest in Asia. There was no evidence that prevalence changed over time. Prevalence estimates in studies of children/adolescents attending obesity clinics and in obese children/adolescents from the general population were substantially lower when elevated alanine aminotransferase (ALT) was used to assess NAFLD compared with biopsies, ultrasound scan (USS) or magnetic resonance imaging (MRI).
Our review suggests the prevalence of NAFLD in young people is high, particularly in those who are obese and in males.
The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action. Incremental advances in conventional review ...updating are unlikely to lead to substantial improvements in review currency. A new approach is needed. We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence. Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication. Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.
The assumption of consistency, defined as agreement between direct and indirect sources of evidence, underlies the increasingly popular method of network meta-analysis. No evidence exists so far ...regarding the extent of inconsistency in full networks of interventions or the factors that control its statistical detection.
In this paper we assess the prevalence of inconsistency from data of 40 published networks of interventions involving 303 loops of evidence. Inconsistency is evaluated in each loop by contrasting direct and indirect estimates and by employing an omnibus test of consistency for the entire network. We explore whether different effect measures for dichotomous outcomes are associated with differences in inconsistency, and evaluate whether different ways to estimate heterogeneity affect the magnitude and detection of inconsistency.
Inconsistency was detected in from 2% to 9% of the tested loops, depending on the effect measure and heterogeneity estimation method. Loops that included comparisons informed by a single study were more likely to show inconsistency. About one-eighth of the networks were found to be inconsistent. The proportions of inconsistent loops do not materially change when different effect measures are used. Important heterogeneity or the overestimation of heterogeneity was associated with a small decrease in the prevalence of statistical inconsistency.
The study suggests that changing the effect measure might improve statistical consistency, and that an analysis of sensitivity to the assumptions and an estimator of heterogeneity might be needed before reaching a conclusion about the absence of statistical inconsistency, particularly in networks with few studies.
PDF
