Review
Improvement evident but still necessary in clinical practice guideline quality: a systematic review

https://doi.org/10.1016/j.jclinepi.2016.08.005Get rights and content

Abstract

Objective

To review the quality of clinical practice guidelines (CPGs) from a wide range of health care topics and report any changes seen since 1992.

Study Design and Setting

A literature search in MEDLINE, EMBASE, Web of Science Core Collection, and BIOSIS was conducted in London, Ontario, Canada. Publications were screened to identify those assessing the quality of CPGs using the Appraisal of Guidelines, Research and Evaluation (AGREE) II instrument. Data were gathered regarding year of publication, institution type, health topic, country of origin, domain scores, and final recommendation.

Results

Twenty-five studies met the inclusion criteria. AGREE II scores from 415 individual CPGs published between 1992 and 2014 were obtained. Domain scores increased significantly over time, and the proportion of guidelines being recommended based on AGREE II assessment was significantly greater after 2010. Domain scores in Applicability and Editorial independence had no significant effect on a CPG's final recommendation, whereas other domains had a significant effect. Finally, international development groups produced CPGs with significantly higher scores.

Conclusion

This review found a steady improvement in CPG quality over time. This is particularly evident in guidelines published after 2010. However, certain domains that are integral to the methodological quality of CPGs remain unsatisfactorily low.

Introduction

Influencing almost all fields of health care, clinical practice guidelines (CPGs) aim to improve the quality, consistency, and effectiveness of care by applying evidence-based medicine and providing health care practitioners with expert summaries of the most recent evidence [1]. The purpose of CPGs is to bridge the gap between clinical research and clinical practice and should therefore be based on the best scientific evidence and developed using the most rigorous methodology. Since the 1980s, the number of CPGs has increased dramatically. However, over the past 25 years, evidence suggests that CPG quality may be highly variable, if not low in general, and the rigor with which CPGs follow standardized development methods is unsatisfactory [2], [3], [4], [5]. It was therefore prudent for a common, widely accepted, and standardized method to evaluate CPGs to be developed.

An international collaboration, the Appraisal of Guidelines, Research and Evaluation (AGREE), created a tool that can be used to evaluate the methodological quality of CPG development. The newest version, the AGREE II instrument, was released in 2010 and is the only appraisal tool that has been developed and validated internationally [6], [7]. It provides a standardized framework consisting of a semiquantitative scoring system involving 23 items over six domains of methodological quality: Scope and purpose, Stakeholder involvement, Rigor of development, Clarity of presentation, Applicability, and Editorial independence. The updated AGREE II instrument is an evolution of the original AGREE. Several changes were made and are outlined in the AGREE II technical document [8].

The AGREE II instrument and its predecessor have been prominent in the literature for over a decade, thus giving CPG developers a viable and effective framework from which to base their final product on. Unfortunately, concerns regarding suboptimal quality, a paucity of supporting evidence, the exclusion of relevant stakeholders from the development process, compromised editorial independence, and a lack of CPG applicability persist [9], [10], [11]. These concerns may be negatively affecting the uptake, utilization, and efficacy of CPGs in their health care domains [12]. The purpose of this study is to review the quality of CPGs spanning many different health care topics published since 1990 to analyze trends in the quality of guideline development and assess the potential effect of the availability of the AGREE II instrument on CPG quality.

Section snippets

Literature search and study selection

A predefined search strategy was used to obtain potentially relevant literature from the MEDLINE, EMBASE, and Web of Science Core Collection and BIOSIS databases. The search strategy used only terms relating to the AGREE II instrument and CPGs to target articles that used the AGREE II instrument to review CPGs from any medical field. In addition to database searching, a bibliographic list of studies citing the AGREE II instrument (list maintained by the AGREE trust and available for download at

Literature review

The authors retrieved 515 citations of articles using the AGREE II instrument from the AGREE Trust online database. Systematic searching of MEDLINE, EMBASE, and Web of Science Core Collection and BIOSIS databases retrieved 719 citations. EndNote X7 (Thomson Reuters) was used to remove duplicate citations, of which there were 180. Of the remaining 539 articles, 445 were published in English and of potential relevance. Full-text versions of the articles were sourced and assessed, after which

Discussion

Overall, CPG quality has improved considerably over the past 2 decades. Steady improvement was observed in all AGREE II domain scores. Domain scores for Clarity of presentation and Scope and purpose reached acceptable levels, Stakeholder involvement and Rigor of development were considered borderline, and Editorial independence and Applicability scored quite poorly. Particularly troubling are the suboptimal scores in Editorial independence and Rigor of development, as these two domains have

Acknowledgments

J.J.A. was funded by the Schulich School of Medicine Summer Research Training Program. J.C.M. was funded by a Canadian Institutes of Health Research Chair in Gender, Work and Health; and Dr James Roth Research Chair in Musculoskeletal Measurement and Knowledge Translation. Karen O'Neil contributed to editing the final article.

Contributors: J.J.A. and J.C.M. conceived the idea for this research and designed the study together with A.M.G. and R.S.I. J.J.A., A.M.G., and R.S.I. searched the

References (54)

  • M.J. Field et al.

    Clinical practice guidelines: directions for a new program

    (1990)
  • F.P. Rivara

    Are guidelines following guidelines?

    AAP Gd Rounds

    (1999)
  • A. Qaseem et al.

    Guidelines International Network: toward international standards for clinical practice guidelines

    Ann Intern Med

    (2012)
  • J. Kung

    Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress

    Arch Intern Med

    (2012)
  • Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project

    Qual Saf Health Care

    (2003)
  • J. Vlayen et al.

    A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit

    Int J Qual Health Care

    (2005)
  • M.C. Brouwers et al.

    AGREE II: advancing guideline development, reporting and evaluation in health care

    Can Med Assoc J

    (2010)
  • P. Alonso-Coello et al.

    The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies

    Qual Saf Health Care

    (2010)
  • S. Sabharwal et al.

    High methodologic quality but poor applicability: assessment of the AAOS guidelines using the AGREE II instrument

    Clin Orthop Relat Res

    (2014)
  • A.C. Don-Wauchope et al.

    Applicability of the AGREE II instrument in evaluating the development process and quality of current National Academy of Clinical Biochemistry guidelines

    Clin Chem

    (2012)
  • M. Lugtenberg et al.

    Why don't physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners

    Implement Sci

    (2009)
  • D. Moher et al.

    Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

    PLoS Med

    (2009)
  • K.G. Avin et al.

    Management of falls in community-dwelling older adults: clinical guidance statement from the Academy of Geriatric Physical Therapy of the American Physical Therapy Association

    Phys Ther

    (2015)
  • M.J. Olivera et al.

    Quality assessment of clinical practice guidelines for Chagas disease

    Rev Soc Bras Med Trop

    (2015)
  • P. Serón et al.

    Evaluation of the quality of clinical guidelines for cardiac rehabilitation

    J Cardiopulm Rehabil Prev

    (2015)
  • K.J. Wilby et al.

    Critical appraisal of clinical practice guidelines in pediatric infectious diseases

    Int J Clin Pharm

    (2015)
  • A. Farghali et al.

    Rigorous method to assess quality and generalizability of clinical practice guidelines

    Can J Hosp Pharm

    (2014)
  • Cited by (0)

    Conflict of interest: All the authors declare that they have no conflict of interests.

    View full text