Transparency Matters: Kaiser Permanente’s National Guideline Program Methodological Processes
Winter 2012 - Volume 16 Number 1
Introduction: The practice-guideline process of collecting, critically appraising, and synthesizing available evidence, then developing expert panel recommendations based on appraised evidence, makes it possible to provide high-quality care for patients. Unwanted variability in the quality and rigor of evidence summaries and Clinical Practice Guidelines has been a long-standing challenge for clinicians seeking evidence-based guidance to support patient care decisions.
As the US health care system continues to undergo significant structural and financial change, evidence-based medicine—which, through clinical recommendations and practice guidelines, brings to the bedside the best available evidence of effective testing and treatment—is used increasingly to reduce unwarranted variation in care and to form the basis for efficient, high-quality care. Physicians want to provide the best-quality care; the practice guideline process of collecting, critically appraising, and synthesizing available evidence, then developing expert panel recommendations based on appraised evidence, makes this possible.
At times, national and international health care organizations and professional societies have issued conflicting recommendations about various clinical processes of care. Unfortunately, this inconsistency has made the identification and selection of high-quality clinical guidance a daunting task for any frontline clinician. It also has made cooperation in guideline development difficult, creating some confusion and contributing to the resource intensive nature of guideline development. Differences in the criteria and processes used to appraise and interpret the same body of evidence are a part of the problem, as are inconsistencies in the processes used to translate evidence into recommendations.
How can we, as consumers or clinicians, be certain of the quality and rigor of clinical practice guidelines (CPGs)? One would assume that guideline developers follow a standard, transparent protocol when searching for, evaluating, analyzing, synthesizing, and summarizing relevant data. One might also assume that expert guideline panels formulate recommendations in the same way. However, unwanted variability in the quality and rigor of evidence summaries and CPGs represents a long-standing challenge for clinicians seeking evidence-based guidance to support patient care decisions. Until recently, there have been no universally accepted standards for evidence summaries and guidelines, which has led to significant variability in the way guidelines are developed. Similarly, because there have not been common standards for documentation, there has been a lack of transparency in materials available for review, making informed guideline comparison and adoption difficult.
Kaiser Permanente's (KP's) methodological approach to guideline development, together with the recently published standards of the Institute of Medicine (IOM),1 may provide some assurances. The IOM standards call for the development of unbiased, scientifically valid, and trustworthy evidence reviews and CPGs. As KP and other guideline development organizations analyze and adopt these recommendations, it is not unreasonable to suggest that the new IOM standards might well form the basis against which transparency, objectivity, and consistency of guidelines will be measured (see Sidebar: Institute of Medicine Standards).
Kaiser Permanente's Guideline Development Infrastructure
KP was an early adopter of evidence-based methods. More than a decade of efforts by a network of KP clinicians, pharmacists, methodologists, evidence analysts, project managers, and other experts in evidence-based medicine has positioned KP as a national leader among health care organizations in developing evidence-based CPGs (see Sidebar: Definition of Clinical Practice Guidelines).2,3 To fully appreciate the link between KP and the IOM standards, it is worth briefly tracing the history of KP's guideline development program.
The CPGs, together with a variety of evidence-based clinician and patient tools linked to KP's electronic health record (EHR), provide evidence and reminders to ensure consistent, effective, and up-to-date evidence-based care. The NGP provides the organization with evidence-based recommendations to support care delivery, to help reduce unwarranted variation in care, and to improve clinical outcomes. To ensure that recommendations are framed consistently and accurately reflect the body of scientific evidence, the Guideline Quality (GQ) committee—a subcommittee of the National Guideline Directors—ensures that guideline development follows a set of rigorous, evidence-based, systematic, and transparent processes. Additionally, through active involvement in national and international organizations, members of the GQ committee remain current on evidence-based medicine and the guideline-development processes, and contribute to the ongoing development of the science of evidence-based practice.
In general, the need for evidence and/or guidance regarding a specific clinical problem prompts a search for existing evidence—most commonly in the form of a formal systematic review. Evidence is then appraised and synthesized to create CPGs that provide guidance for clinical decisions. Guidelines based on well-conducted systematic reviews of the evidence provide an explicit linkage between the best evidence and clinical practice.
A Closer Look at the Institute of Medicine Standards
The IOM standards were developed in part in response to the Medicare Improvements for Patients and Providers Act of 20084 when the US Congress asked the IOM to study and to report on best methods used to develop CPGs. The IOM then developed eight standards for the development of rigorous, trustworthy systematic reviews and CPGs (Figure 1).1,5 These two recent IOM reports—Finding What Works in Health Care: Standards for Systematic Reviews6 and Clinical Practice Guidelines We Can Trust1—further describe detailed standards to increase the rigor of evidence reviews and guideline development, documentation, and reporting. Having received immediate attention on an international level, both reports raise the bar for conducting reviews of scientific evidence and developing evidence-based CPGs. It is expected that adherence to these standards will reduce bias, conflicts of interest, and variability in guideline development. Proponents also hope these standards present a unique opportunity to increase national and international collaboration for guideline development and dissemination, and will make the guideline development process less resource intensive.
Historical Perspective on Kaiser Permanente's National Guideline Development Program
KP's development of evidence-based CPGs began in 1991, when the Southern California Region hired David Eddy, MD, a pioneer in evidence-based medicine, to consult on the development of the Region's clinical guideline and technology assessment programs. At that time, most of KP's Regions had been developing practice guidelines based on expert consensus and review of selected studies. Dr Eddy's methodology—detailed in a manual developed in collaboration with the Council of Medical Specialty Societies Task Force on Practice Policies7—emphasized a rigorous and explicit approach based on systematic searching and selection of all available evidence for the topic of interest. He also emphasized the importance of critical appraisal of relevant studies, detailed estimation of an intervention's effect on important health outcomes, and an explicit description of the link between the evidence and eventual guideline recommendations.
By the mid-1990s, a growing interest in evidence-based guideline methodology, coupled with a desire to share guideline development resources across KP's Regions, led to the founding of the KP Interregional Guidelines Steering Group (IRGSG). Influenced by Dr Eddy's explicit approach, the IRGSG developed a position paper for interregional collaboration and a common methodology outlining principles and processes for evidence-based guideline development. By the late 1990s, under the sponsorship of CMI, the IRGSG evolved into the National Guideline Directors with representation from all Regions and agreement on a core set of nationally endorsed guidelines to be developed using rigorous evidence-based methods.
In 2010, KP's NGP became a member of the Guidelines International Network (G-I-N), a collaborative, international, not-for-profit association of organizations and individuals involved in development and use of CPGs. G-I-N seeks to improve the quality of health care by promoting systematic development of clinical guidelines and application of these guidelines in practice. Through G-I-N, the NGP was exposed to a number of international collaborative groups that had been developing frameworks and tools to improve the guideline development process, including:
Although KP's NGP had invested significant resources to create a unique internal guideline development process, the ADAPTE, AGREE, AMSTAR and GRADE frameworks offered opportunities and suggestions to make KP's guideline development process more systematic, transparent, and explicit. Furthermore, the appropriate application of these tools allows KP to extend its rigorous processes to adopt or to adapt preexisting high-quality guidelines and evidence summaries, and to tailor recommendations for KP's specific cultural and organizational context.
In early 2010, backed by KP's CMI, guideline quality methodologists, and KP Regions, the National Guideline Directors agreed that the ADAPTE, AGREE, AMSTAR, and GRADE frameworks should be incorporated into KP's guideline methodology. To reflect these changes, the KP National Guideline GQ Committee revised the NGP methodology. The Kaiser Permanente National Guideline Program Process and Methodology for Systematic Development of Clinical Practice Recommendations12 defines and describes the methods the NGP employs when creating CPGs. The following is a high-level overview of the KP National Guideline development processes.
Guideline Development and Methodology at Kaiser Permanente
KP employs an integrated, evidence-based, systematic, and transparent approach to the development of clinical guideline recommendations. This iterative process involves collection of data to create evidence-based resources, including CPGs and point-of-care decision-support tools within the EHR.
A multidisciplinary group of stakeholders, with representation from all eight KP Regions, is responsible for creating National Guidelines. Each Guideline Development Team (GDT) includes physicians and other clinical experts (such as psychologists, pharmacists, clinical nurse experts, social workers, etc), evidence analysts, and a methodologist. The GDT serves as the expert panel that refines and approves the clinical recommendations that compose each guideline. To ensure that recommendations accurately reflect the body of scientific evidence, and are relevant to clinician and patient needs, the guideline development process follows the process described in Table 1 and schematically in Figure 3.
Choosing Topics for Clinical Guidelines: Challenges and Implications
Each year, the NGP evaluates and selects priority topics to be included in its guideline portfolio. Clinical questions to be addressed are prioritized on the basis of an assessment of several internal and external factors. These factors may include:
Once a topic has been identified, a GDT is assembled; this team helps specify the scope of the guideline, including target populations, comparative interventions, important outcomes, and other clinical issues. This process provides direction for framing specific clinical questions, which commonly address issues of risk, diagnosis, prognosis, therapy, and harm.
Traditionally, KP's CPGs have been created in-house; that is, the GDT maintained control of the entire guideline development process—from defining the clinical question through conducting systematic reviews, evaluating the evidence, and creating clinical recommendations within the framework of a clinical guideline—regardless of other preexisting external evidence reviews or guidelines.
Conducting high-quality systematic reviews and creating clinical guidelines are time-, labor-, and resource-intensive processes, which raises challenges for an organization striving to balance rigor with efficiency. For these reasons, the NGP elected to allow for the identification, assessment, and possible adoption of existing evidence-based guidelines and systematic reviews using the ADAPTE, AGREE, AMSTAR and GRADE frameworks.
If such guidelines are found, the guideline team uses the ADAPTE process to assess congruence with identified clinical questions, analyze guidelines for quality, and make adaptations as needed to fit KP's context and needs (Table 2). Guidelines are further evaluated using the AGREE tool (part of the ADAPTE process), which evaluates guideline quality standards in six domains of guideline quality and usability (Table 3).
No External Guidelines Available
If no acceptable external guidelines are identified, the GDT then systematically searches for relevant high-quality systematic reviews, meta-analyses, and original studies. Existing systematic reviews are assessed for quality using a measurement tool to assess systematic reviews (the AMSTAR systematic review checklist),13 another tool new to KP's methodology (Table 4). If systematic reviews are identified and deemed high quality by the AMSTAR checklist, the GDT may opt to use a preexisting review, rather than complete a systematic review de novo. In these cases, evidence analysts perform a supplementary search to identify any new studies that may have been published following the date of the preexisting review.
Grading Evidence Quality
Following the screening and selection process, the included studies (the "body of evidence") are critically appraised for quality, using the GRADE11 methodology. This approach is widely used internationally, and provides a systematic, transparent, and explicit framework to grade the quality of bodies of evidence. The higher the quality of the available evidence, the more confident one can be that an estimate of effect or association is close to the actual outcome of interest.
Steps in the GRADE approach include:14
• Identifying critical or important clinical outcomes
The quality of evidence for each important outcome includes consideration of six elements: study design, study quality, consistency, directness, precision, and possible reporting bias. The GRADE system classifies the quality of evidence into the following four levels with corresponding implications:
• High quality—further research is very unlikely to change confidence in the estimate of effect
When based on randomized control trials, the body of evidence is initially categorized as high quality. However, the evidence grade may be reduced for reasons including study limitations, inconsistency of results, indirectness, imprecision, and publication bias. When based on observational studies (eg, cohort or case-control studies), the body of evidence is initially categorized as low quality. However, the evidence grade may be increased if, for example, the magnitude of treatment effect is very large or if there is evidence of a dose-response relationship.
Translating Evidence into Recommendations
The evidence is then used to create preliminary clinical recommendations. The strength of these recommendations is graded to reflect the extent to which a guideline panel is confident that the desirable effects of an intervention outweigh undesirable effects (or vice versa) across the range of patients for whom the recommendation is intended. GRADE focuses on four key factors that must be considered when assigning strength to a recommendation: balance between desirable and undesirable effects, quality of evidence, values and preferences, and cost (Table 6).15
The final outcome of this process is the creation of evidence-based clinical recommendations that address the clinical questions posed at the outset of the process. A recommendation may contain one or more parts. Rationale statements—explicitly addressing the four GRADE strength of recommendation domains—are written to explain the logic that links the evidence synthesis and GRADE analysis to the appropriate recommendation (Table 7).