AGREE II is a widely used standard for assessing the methodological quality of practice guidelines. This article describes the development of the AGREE Reporting Checklist, which was designed to ...improve the quality of practice guideline reporting and aligns with AGREE II in its structure and content.
Guidelines continue to be underutilized, and a variety of strategies to improve their use have been suboptimal. Modifying guideline features represents an alternative, but untested way to promote ...their use. The purpose of this study was to identify and define features that facilitate guideline use, and examine whether and how they are included in current guidelines.
A guideline implementability framework was developed by reviewing the implementation science literature. We then examined whether guidelines included these, or additional implementability elements. Data were extracted from publicly available high quality guidelines reflecting primary and institutional care, reviewed independently by two individuals, who through discussion resolved conflicts, then by the research team.
The final implementability framework included 22 elements organized in the domains of adaptability, usability, validity, applicability, communicability, accommodation, implementation, and evaluation. Data were extracted from 20 guidelines on the management of diabetes, hypertension, leg ulcer, and heart failure. Most contained a large volume of graded, narrative evidence, and tables featuring complementary clinical information. Few contained additional features that could improve guideline use. These included alternate versions for different users and purposes, summaries of evidence and recommendations, information to facilitate interaction with and involvement of patients, details of resource implications, and instructions on how to locally promote and monitor guideline use. There were no consistent trends by guideline topic.
Numerous opportunities were identified by which guidelines could be modified to support various types of decision making by different users. New governance structures may be required to accommodate development of guidelines with these features. Further research is needed to validate the proposed framework of guideline implementability, develop methods for preparing this information, and evaluate how inclusion of this information influences guideline use.
Objective Providers and patients are most likely to use and benefit from guidelines accompanied by implementation support. Guidelines published in 2007 and earlier assessed with the Appraisal of ...Guidelines, Research and Evaluation (AGREE) instrument scored poorly for applicability, which reflects the inclusion of implementation instructions or tools. The purpose of this study was to examine the applicability of guidelines published in 2008 or later and identify factors associated with applicability. Design Systematic review of studies that used AGREE to assess guidelines published in 2008 or later. Data sources MEDLINE and EMBASE were searched from 2008 to July 2014, and the reference lists of eligible items. Two individuals independently screened results for English language studies that reviewed guidelines using AGREE and reported all domain scores, and extracted data. Descriptive statistics were calculated across all domains. Multilevel regression analysis with a mixed effects model identified factors associated with applicability. Results Of 245 search results, 53 were retrieved as potentially relevant and 20 studies were eligible for review. The mean and median domain scores for applicability across 137 guidelines published in 2008 or later were 43.6% and 42.0% (IQR 21.8–63.0%), respectively. Applicability scored lower than all other domains, and did not markedly improve compared with guidelines published in 2007 or earlier. Country (UK) and type of developer (disease-specific foundation, non-profit healthcare system) appeared to be associated with applicability when assessed with AGREE II (not original AGREE). Conclusions Despite increasing recognition of the need for implementation tools, guidelines continue to lack such resources. To improve healthcare delivery and associated outcomes, further research is needed to establish the type of implementation tools needed and desired by healthcare providers and consumers, and methods for developing high-quality tools.
•AGREE tools support the development, reporting, and appraisal of guidance.•AGREE II tool targets the entire practice guideline process.•AGREE-REX tool targets recommendations.•AGREE-HS tool targets ...health system guidance documents.•To apply the tools, consider type or part of the guideline, goals, and resources.
Abstract
Background
Health care professionals (HCPs) use clinical practice guidelines (CPGs) to make evidence-informed decisions regarding patient care. Although a large number of cancer-related CPGs ...exist, it is unknown which CPG dissemination and implementation strategies are effective for improving HCP behaviour and patient outcomes in a cancer care context. This review aimed to determine the effectiveness of CPG dissemination and/or implementation strategies among HCPs in a cancer care context.
Methods
A comprehensive search of five electronic databases was conducted. Studies were limited to the dissemination and/or implementation of a CPG targeting both medical and/or allied HCPs in cancer care. Two reviewers independently coded strategies using the Mazza taxonomy, extracted study findings, and assessed study quality.
Results
The search strategy identified 33 studies targeting medical and/or allied HCPs. Across the 33 studies, 23 of a possible 49 strategies in the Mazza taxonomy were used, with a mean number of 3.25 (SD = 1.45) strategies per intervention. The number of strategies used per intervention was not associated with positive outcomes. Educational strategies (
n
= 24), feedback on guideline compliance (
n
= 11), and providing reminders (
n
= 10) were the most utilized strategies. When used independently, providing reminders and feedback on CPG compliance corresponded with positive significant changes in outcomes. Further, when used as part of multi-strategy interventions, group education and organizational strategies (e.g. creation of an implementation team) corresponded with positive significant changes in outcomes.
Conclusions
Future CPG dissemination and implementation interventions for cancer care HCPs may benefit from utilizing the identified strategies. Research in this area should aim for better alignment between study objectives, intervention design, and evaluation measures, and should seek to incorporate theory in intervention design, so that behavioural antecedents are considered and measured; doing so would enhance the field’s understanding of the causal mechanisms by which interventions lead, or do not lead, to changes in outcomes at all levels.
The purpose of the AGREE II is more explicitly stated. The new version of the instrument is designed to assess the quality of practice guidelines across the spectrum of health, provide direction on ...guideline development, and guide what specific information ought to be reported in guidelines. The four-point response scale was replaced by a seven-point response scale, in compliance with key methodologic principles of test construction. 5 A score of 1 indicates an absence of information or that the concept is very poorly reported. A score of 7 indicates that the quality of reporting is exceptional and all of the criteria and considerations articulated in the user's manual were met. A score between 2 and 6 indicates that the reporting of the AGREE II item does not fully meet criteria or considerations. As more criteria are met and more considerations addressed, item scores increase (see user's manual below). Finally, modifications, deletions and additions were made to approximately half of the original 23 items (Table 1). From McMaster University (Melissa C. Brouwers PhD, Michelle E. Kho, Steven E. Hanna PhD, Julie Makarski); the Program in Evidence-based Care, Cancer Care Ontario (Brouwers), Hamilton, Ont.; British Columbia Cancer Agency (Browman), Victoria, BC; the Dutch Institute for Healthcare Improvement CBO and IQ Healthcare (Burgers), Radboud University Nijmegen Medical Centre, the Netherlands; St. George's University of London (Cluzeau), London, UK; the University of Bristol (Feder), Bristol, UK; Unité Cancer et Environement (Fervers), Université de Lyon - Centre Léon Bérard, Université Lyon 1, EA 4129, Lyon, France; the Canadian Institutes of Health Re search (Ian D. Graham PhD), Ottawa, Ont.; the Ottawa Hospital Research Institute (Jeremy Grimshaw MBChB PhD), Ottawa, Ont.; the National Institute for Health and Clinical Excellence (Littlejohns), London, UK; and the Can adian Partnership Against Cancer (Louise Zitzelsberger PhD), Ottawa, Ont. Members of the AGREE Next Steps Consortium: Dr. Melissa C. Brouwers, McMaster University and Cancer Care Ontario, Hamilton, Ont.; Dr. George P. Browman MD MSc, British Columbia Cancer Agency, Vancouver Island, BC; Dr. Jako S. Burgers MD PhD, Dutch Institute for Healthcare Improvement CBO, and Radboud University Nijmegen Medical Centre, IQ Healthcare, Netherlands; Dr. Francoise Cluzeau, Chair of AGREE Research Trust, St. George's University of London, London, UK; Dr. Dave Davis, Association of American Medical Colleges, Washington, USA; Prof. Gene Feder, University of Bristol, Bristol, UK; Dr. Béatrice Fervers, Unité Cancer et Environement, Université de Lyon - Centre Léon Bérard, Université Lyon 1, EA 4129, Lyon, France; Dr. Ian D. Graham, Canadian Institutes of Health Research, Ottawa, Ont.; Dr. Jeremy Grimshaw, Ottawa Hospital Research Institute, Ottawa, Ont.; Dr. Steven E. Hanna, McMaster University, Hamilton, Ont.; Ms. Michelle E. Kho, McMaster University, Hamilton, Ont.; Prof. Peter Littlejohns, National Institute for Health and Clinical Excellence, London, UK; Ms. Julie Makarski, McMaster University, Hamilton, Ont.; Dr. Louise Zitzelsberger, Canadian Partnership Against Cancer, Ottawa, Ont.
The American Society of Clinical Oncology (ASCO) guidelines program employs a systematic review-based methodology to produce evidence-based guidelines. This is consistent with the stance of the ...Institute of Medicine on guideline development, which is that high-quality evidence syntheses form the basis for recommendation development. In the absence of high-quality evidence, recommendation development becomes more complex. One option is to provide no recommendations or withdraw a guideline topic. However, it is often the areas of greatest uncertainty in which the evidentiary base is incomplete, and thus, guidelines are needed most. To provide recommendations in such circumstances, an explicit methodology is needed to ensure that a credible process is undertaken, and rigorous, reliable advice is provided. In 2010, the ASCO Board of Directors approved development of guideline recommendations using consensus methodology. A modified Delphi approach to recommendation development, based on the best available data identified in a systematic review, was piloted with an ASCO guideline. Consensus was achieved through the rating of a series of recommendations by a large group of clinicians, including academic and community-based content and methodology experts. A prespecified threshold of agreement was determined to indicate when consensus was achieved. Consensus was defined as agreement by ≥ 75% of raters. The formal consensus methodology used by ASCO enabled development of guideline recommendations on a challenging clinical issue based on limited evidence using a rigorous, transparent, and explicit method. This methodology is proposed for development of future ASCO guidelines on topics for which limited evidence is available.
Clinical practice guidelines (CPGs) may lack rigor and suitability to the setting in which they are to be applied. Methods to yield clinical practice guideline recommendations that are credible and ...implementable remain to be determined.
To describe the development of AGREE-REX (Appraisal of Guidelines Research and Evaluation-Recommendations Excellence), a tool designed to evaluate the quality of clinical practice guideline recommendations.
A cross-sectional study of 322 international stakeholders representing CPG developers, users, and researchers was conducted between December 2015 and March 2019. Advertisements to participate were distributed through professional organizations as well as through the AGREE Enterprise social media accounts and their registered users.
Between 2015 and 2017, participants appraised 1 of 161 CPGs using the Draft AGREE-REX tool and completed the AGREE-REX Usability Survey.
Usability and measurement properties of the tool were assessed with 7-point scales (1 indicating strong disagreement and 7 indicating strong agreement). Internal consistency of items was assessed with the Cronbach α, and the Spearman-Brown reliability adjustment was used to calculate reliability for 2 to 5 raters.
A total of 322 participants (202 female participants 62.7%; 83 aged 40-49 years 25.8%) rated the survey items (on a 7-point scale). All 11 items were rated as easy to understand (with a mean SD ranging from 5.2 1.38 for the alignment of values item to 6.3 0.87 for the evidence item) and easy to apply (with a mean SD ranging from 4.8 1.49 for the alignment of values item to 6.1 1.07 for the evidence item). Participants provided favorable feedback on the tool's instructions, which were considered clear (mean SD, 5.8 1.06), helpful (mean SD, 5.9 1.00), and complete (mean SD, 5.8 1.11). Participants considered the tool easy to use (mean SD, 5.4 1.32) and thought that it added value to the guideline enterprise (mean SD, 5.9 1.13). Internal consistency of the items was high (Cronbach α = 0.94). Positive correlations were found between the overall AGREE-REX score and the implementability score (r = 0.81) and the clinical credibility score (r = 0.76).
This cross-sectional study found that the AGREE-REX tool can be useful in evaluating CPG recommendations, differentiating among them, and identifying those that are clinically credible and implementable for practicing health professionals and decision makers who use recommendations to inform clinical policy.
Guidelines are important tools that inform healthcare delivery based on best available research evidence. Guideline use is in part based on quality of the guidelines, which includes advice for ...implementation and has been shown to vary. Others hypothesized this is due to limited instructions in guideline development manuals. The purpose of this study was to examine manual instructions for implementation advice.
We used a directed and summative content analysis approach based on an established framework of guideline implementability. Six manuals identified by another research group were examined to enumerate implementability domains and elements.
Manuals were similar in content but lacked sufficient detail in particular domains. Most frequently this was Accomodation, which includes information that would help guideline users anticipate and/or overcome organizational and system level barriers. In more than one manual, information was also lacking for Communicability, information that would educate patients or facilitate their involvement in shared decision making, and Applicability, or clinical parameters to help clinicians tailor recommendations for individual patients.
Most manuals that direct guideline development lack complete information about incorporating implementation advice. These findings can be used by those who developed the manuals to consider expanding their content in these domains. It can also be used by guideline developers as they plan the content and implementation of their guidelines so that the two are integrated. New approaches for guideline development and implementation may need to be developed. Use of guidelines might be improved if they included implementation advice, but this must be evaluated through ongoing research.
Guidelines support health care decision-making and high quality care and outcomes. However, their implementation is sub-optimal. Theory-informed, tailored implementation is associated with guideline ...use. Few guideline implementation studies published up to 1998 employed theory. This study aimed to describe if and how theory is now used to plan or evaluate guideline implementation among physicians.
A scoping review was conducted. MEDLINE, EMBASE, and The Cochrane Library were searched from 2006 to April 2016. English language studies that planned or evaluated guideline implementation targeted to physicians based on explicitly named theory were eligible. Screening and data extraction were done in duplicate. Study characteristics and details about theory use were analyzed.
A total of 1244 published reports were identified, 891 were unique, and 716 were excluded based on title and abstract. Among 175 full-text articles, 89 planned or evaluated guideline implementation targeted to physicians; 42 (47.2%) were based on theory and included. The number of studies using theory increased yearly and represented a wide array of countries, guideline topics and types of physicians. The Theory of Planned Behavior (38.1%) and the Theoretical Domains Framework (23.8%) were used most frequently. Many studies rationalized choice of theory (83.3%), most often by stating that the theory described implementation or its determinants, but most failed to explicitly link barriers with theoretical constructs. The majority of studies used theory to inform surveys or interviews that identified barriers of guideline use as a preliminary step in implementation planning (76.2%). All studies that evaluated interventions reported positive impact on reported physician or patient outcomes.
While the use of theory to design or evaluate interventions appears to be increasing over time, this review found that one half of guideline implementation studies were based on theory and many of those provided scant details about how theory was used. This limits interpretation and replication of those interventions, and seems to result in multifaceted interventions, which may not be feasible outside of scientific investigation. Further research is needed to better understand how to employ theory in guideline implementation planning or evaluation.
PDF
