To determine the reproducibility of biomedical systematic review search strategies.
A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in ...MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records.
The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible.
Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.
Regulatory approval of biosimilars often depends on extrapolating evidence from one clinical indication to all of those of the originator biologic. We aimed to develop a quantitative benefit-risk ...analysis to assess whether the resulting increase in the uncertainty in the clinical performance of biosimilars (i.e. risk) may be countered by their lower pricing (benefit).
A 1-year decision-analytic model was developed for the biosimilar infliximab (Inflectra
) for Crohn's disease. The perspective was that of the National Health Service in the UK and costs were valued to 2015/16. A hypothetical cohort of biologic-naïve patients with moderate-to-severe Crohn's disease was simulated through the model. Immunogenicity to infliximab was a key modifier, influencing rates of non-response and infusion reactions. Net health benefit was estimated based on quality-adjusted life-years. A range of sensitivity analyses tested the robustness of the results and explored how the biosimilar price must respond to varying immunogenicity to remain the preferred option.
The base-case analysis predicted a positive incremental net health benefit of 0.04 (95% central range 0.00-0.09) favouring the biosimilar, based on 0.803 quality-adjusted life-years, and costs of £18,087 and £19,176 for the biosimilar and originator, respectively. Two-way sensitivity analyses suggested that if 50% of patients developed antibodies, the value-based price of £410 per vial must be lower than that of the originator (£420), but remain higher than the actual market price (£378).
The model supports the use of Inflecta
for Crohn's disease in the UK, and provides a framework for the quantitative evaluation of biosimilars in the context of a health technology assessment. Value-based pricing using this methodology could protect health systems from the potential risks of biosimilars where they are untested in the approved populations.
AbstractObjectivesThe objective of this study was to assess the uptake of the rheumatoid arthritis core outcome set (RA-COS) using data from multiple data providers, and to investigate factors that ...may influence this uptake. Study Design and SettingAn observational review was carried out on all clinical trials of rheumatoid arthritis that were indexed on the World Health Organization-International Clinical Trials Registry Platform (WHO-ICTRP). Various measures of COS uptake were calculated from information presented in the trial registries and trial publications. Logistic regression was conducted to investigate factors thought to be associated with planned COS uptake. ResultsA total of 341 trials were eligible for the evaluation of RA-COS uptake. In the decade leading up to 2019, the assessment of uptake based on published results was maintained at just over 80%. Trials that were not commercially funded were much less likely to plan to measure the RA-COS than those with industry funding (60% vs. 80%; adjusted OR 0.18, 95% CI 0.11 to 0.32; P < 0.001). ConclusionThis study has demonstrated that the use of the WHO-ICTRP can identify a large and geographically diverse range of trials to include in the evaluation of COS uptake.
Abstract Objectives The aim of the study was to evaluate citation analysis as an approach to measuring core outcome set (COS) uptake, by assessing whether the number of citations for a COS report ...could be used as a surrogate measure of uptake of the COS by clinical trialists. Study Design and Setting Citation data were obtained for COS reports published before 2010 in five disease areas (systemic sclerosis, rheumatoid arthritis, eczema, sepsis and critical care, and female sexual dysfunction). Those publications identified as a report of a clinical trial were examined to identify whether or not all outcomes in the COS were measured in the trial. Results Clinical trials measuring the relevant COS made up a small proportion of the total number of citations for COS reports. Not all trials citing a COS report measured all the recommended outcomes. Some trials cited the COS reports for other reasons, including the definition of a condition or other trial design issues addressed by the COS report. Conclusion Although citation data can be readily accessed, it should not be assumed that the citing of a COS report indicates that a trial has measured the recommended COS. Alternative methods for assessing COS uptake are needed.
Objective To determine the standard of reporting of harms-related data, in randomised controlled trials (RCTs) according to the Consolidated Standards of Reporting Trials (CONSORT) statement ...extension for harms. Design Systematic review. Data sources The Cochrane library, Ovid MEDLINE, Scopus and ISI Web of Knowledge were searched for relevant literature. Eligibility criteria for selecting studies We included publications of studies that used the CONSORT harms extension to assess the reporting of harms in RCTs. Results We identified 7 studies which included between 10 and 205 RCTs. The clinical areas of the 7 studies were: hypertension (1), urology (1), epilepsy (1), complimentary medicine (2) and two not restricted to a clinical topic. Quality of the 7 studies was assessed by a risk of bias tool and was found to be variable. Adherence to the CONSORT harms criteria reported in the 7 studies was inadequate and variable across the items in the checklist. Adverse events are poorly defined, with 6 studies failing to exceed 50% adherence to the items in the checklist. Conclusions Readers of RCT publications need to be able to balance the trade-offs between benefits and harms of interventions. This systematic review suggests that this is compromised due to poor reporting of harms which is evident across a range of clinical areas. Improvements in quality could be achieved by wider adoption of the CONSORT harms criteria by journals reporting RCTs.
ObjectivesA core outcome set (COS) is an agreed standardised minimum collection of outcomes that should be measured and reported in research in a specific area of health. Cochrane systematic reviews ...(‘reviews’) are rigorous reviews on health-related topics conducted under the auspices of Cochrane. This study examines the use of existing COS to inform the choice of outcomes in Cochrane systematic reviews (‘reviews’) and investigates the views of the coordinating editors of Cochrane Review Groups (CRGs) on this topic.MethodsA cohort of 100 recently published or updated Cochrane reviews were assessed for reference to a COS being used to inform the choice of outcomes for the review. Existing COS, published 2 or more years before the review publication, were then identified to assess how often a reviewer could have used a relevant COS if it was available. We asked 52 CRG coordinating editors about their involvement in COS development, how outcomes are selected for reviews in their CRG and their views of the advantages and challenges surrounding the standardisation of outcomes within their CRG.ResultsIn the cohort of reviews from 2019, 40% (40/100) of reviewers noted problems due to outcome inconsistency across the included studies. In 7% (7/100) of reviews, a COS was referenced in relation to the choice of outcomes for the review. Relevant existing COS could be considered for a review update in 35% of the others (33/93). Most editors who responded (31/36, 86%) thought that COS should definitely or possibly be used to inform the choice of outcomes in a review.ConclusionsSystematic reviewers are continuing to note outcome heterogeneity but are starting to use COS to inform their reviews. There is potential for greater uptake of COS in Cochrane reviews.
Core outcome sets can increase the efficiency and value of research and, as a result, there are an increasing number of studies looking to develop core outcome sets (COS). However, the credibility of ...a COS depends on both the use of sound methodology in its development and clear and transparent reporting of the processes adopted. To date there is no reporting guideline for reporting COS studies. The aim of this programme of research is to develop a reporting guideline for studies developing COS and to highlight some of the important methodological considerations in the process.
The study will include a reporting guideline item generation stage which will then be used in a Delphi study. The Delphi study is anticipated to include two rounds. The first round will ask stakeholders to score the items listed and to add any new items they think are relevant. In the second round of the process, participants will be shown the distribution of scores for all stakeholder groups separately and asked to re-score. A final consensus meeting will be held with an expert panel and stakeholder representatives to review the guideline item list. Following the consensus meeting, a reporting guideline will be drafted and review and testing will be undertaken until the guideline is finalised. The final outcome will be the COS-STAR (Core Outcome Set-STAndards for Reporting) guideline for studies developing COS and a supporting explanatory document.
To assess the credibility and usefulness of a COS, readers of a COS development report need complete, clear and transparent information on its methodology and proposed core set of outcomes. The COS-STAR guideline will potentially benefit all stakeholders in COS development: COS developers, COS users, e.g. trialists and systematic reviewers, journal editors, policy-makers and patient groups.
Abstract
Background
Problems continue to exist with the reporting quality and risk of bias in search methods and strategies in systematic reviews and related review types. Peer reviewers who are not ...familiar with what is required to transparently and fully report a search may not be prepared to review the search components of systematic reviews, nor may they know what is likely to introduce bias into a search. Librarians and information specialists, who have expertise in searching, may offer specialized knowledge that would help improve systematic review search reporting and lessen risk of bias, but they are underutilized as methodological peer reviewers.
Methods
This study will evaluate the effect of adding librarians and information specialists as methodological peer reviewers on the quality of search reporting and risk of bias in systematic review searches. The study will be a pragmatic randomized controlled trial using 150 systematic review manuscripts submitted to
BMJ
and
BMJ Open
as the unit of randomization. Manuscripts that report on completed systematic reviews and related review types and have been sent for peer review are eligible. For each manuscript randomized to the intervention, a librarian/information specialist will be invited as an additional peer reviewer using standard practices for each journal. First revision manuscripts will be assessed in duplicate for reporting quality and risk of bias, using adherence to 4 items from PRISMA-S and assessors’ judgements on 4 signaling questions from ROBIS Domain 2, respectively. Identifying information from the manuscripts will be removed prior to assessment.
Discussion
The primary outcomes for this study are quality of reporting as indicated by differences in the proportion of adequately reported searches in first revision manuscripts between intervention and control groups and risk of bias as indicated by differences in the proportions of first revision manuscripts with high, low, and unclear bias. If the intervention demonstrates an effect on search reporting or bias, this may indicate a need for journal editors to work with librarians and information specialists as methodological peer reviewers.
Trial registration
Open Science Framework. Registered on June 17, 2021, at
10.17605/OSF.IO/W4CK2
.
IntroductionCore outcome sets (COSs) are agreed outcomes (domains (subdomains) and instruments) that should be measured as a minimum in clinical trials or practice in certain diseases or clinical ...fields. Worldwide, the number of COSs is increasing and there might be conceptual overlaps of domains (subdomains) and instruments within disciplines. The aim of this scoping review is to map and to classify all outcomes identified with COS projects relating to skin diseases.Methods and analysisWe will conduct a scoping review of outcomes of skin disease-related COS initiatives to identify all concepts and their definitions. We will search PubMed, Embase and Cochrane library. The search dates will be 1 January 2010 (the point at which Core Outcome Measures in Effectiveness Trials (COMET) was established) to 1 January 2024. We will also review the COMET database and C3 website to identify parts of COSs (domains and/or instruments) that are being developed and published. This review will be supplemented by querying relevant stakeholders from COS organisations, dermatology organisations and patient organisations for additional COSs that were developed. The resulting long lists of outcomes will then be mapped into conceptually similar concepts.Ethics and disseminationThis study was supported by departmental research funds from the Department of Dermatology at Northwestern University. An ethics committee review was waived since this protocol was done by staff researchers with no involvement of patient care. Conflicts of interests, if any, will be addressed by replacing participants with relevant conflicts or reassigning them. The results will be disseminated through publication in peer-reviewed journals, social media posts and promotion by COS organisations.
The Delphi method is used in a wide variety of settings as a method of building consensus on important issues. Traditionally, the Delphi method uses multiple rounds of a survey to allow for feedback ...of other participants' survey responses in between rounds. By informing participants about how others answer a question or prioritise specific topics, it allows for diverse opinions to inform the consensus process. For this reason, the Delphi method is popular as a consensus building approach in developing core outcome sets (COS), i.e. the minimum agreed set of standardised outcomes that should be measured and reported in studies on a specific health condition. In a COS setting, participants prioritise the importance of outcomes for inclusion in a COS. This usually involves participating in multiple rounds of a survey that can span several weeks or months. Challenges with participant retention have been highlighted in previous COS. We will compare a three-round with a Real-Time Delphi approach on prioritised outcomes. This trial is embedded within the COHESION study which is developing a COS for interventions treating neonatal encephalopathy.
One hundred and eighty stakeholders (parents/caregivers of infants diagnosed and treated with neonatal encephalopathy, healthcare providers and researchers) will be randomised using stratified randomisation to take part in either the Multi-Round or Real-Time Delphi. Stakeholders will rate the importance of the same set of outcomes in both arms. We will compare the prioritised outcomes at the end of both surveys as well as other parameters such as feedback, initial condition and iteration effects.
This trial will provide evidence to inform decisions on the use of Multi-Round compared to Real-Time Delphi survey methods.
NCT04471103 . Registered on 14 July 2020.
PDF
