Comparative Health Systems Research among Kaiser Permanente and Other Integrated Delivery Systems: A Systematic Literature Review
Perm J 2014 Summer;18(3):66-77 [Full Citation]
Editor's note: For more on research and comparative health systems, please see the editorial: A Commentary on "Comparative Health Systems Research among Kaiser Permanente and Other Integrated Delivery Systems: A Systematic Literature Review" available at: www.thepermanentejournal.org/issues/2014/summer/5693-kaiser-commentary.html
Context: Because of rising health care costs, wide variations in quality, and increased patient complexity, the US health care system is undergoing rapid changes that include payment reform and movement toward integrated delivery systems. Well-established integrated delivery systems, such as Kaiser Permanente (KP), should work to identify the specific system-level factors that result in superior patient outcomes in response to policymakers' concerns. Comparative health systems research can provide insights into which particular aspects of the integrated delivery system result in improved care delivery.
Objective: To provide a baseline understanding of comparative health systems research related to integrated delivery systems and KP.
Design: Systematic literature review.
Methods: We conducted a literature search on PubMed and the KP Publications Library. Studies that compared KP as a system or organization with other health care systems or across KP facilities internally were included. The literature search identified 1605 articles, of which 65 met the study inclusion criteria and were examined by 3 reviewers.
Results: Most comparative health systems studies focused on intra-KP comparisons (n = 42). Fewer studies compared KP with other US (n = 15) or international (n = 12) health care systems. Several themes emerged from the literature as possible factors that may contribute to improved care delivery in integrated delivery systems.
Conclusions: Of all studies published by or about KP, only a small proportion of articles (4%) was identified as being comparative health systems research. Additional empirical studies that compare the specific factors of the integrated delivery system model with other systems of care are needed to better understand the "system-level" factors that result in improved and/or diminished care delivery.
Rising health care costs,1 wide variation in quality,2 and increased patient complexity led to passage of the Affordable Care Act,3 which has resulted in the US health care system undergoing rapid changes. These changes include payment reform (ie, value-based purchasing, bundled payments)4 and movement toward integrated delivery systems, such as accountable care organizations and patient-centered medical homes.5,6 Because the current US health care system performs poorly relative to those of other countries, alternative models of care delivery have been proposed.5,7
Some of the inefficiencies of the current US health care delivery system stem from the growth of new and expensive medical technologies and the fee-for-service payment of physicians.8 Although physicians aim to provide patient care on the basis of scientific evidence, financial considerations may influence their treatment decisions. Replacement of fee-for-service with capitated payment has been proposed as one way to improve the efficiency of health care delivery. However, changing the physician payment structure by itself may not be enough to achieve the desired outcome. Previous research has shown that although prospective payment has slowed the growth of health care spending at the medical group level, similar results have not been achieved among individual or small practices.8 Physician practices therefore may need to be reorganized and integrated across multispecialty groups and hospitals to be responsive to new payment methods. Thus, health care reform efforts also may need to focus on redesigning integrated systems of care.7
Integrated delivery systems are a model of health care involving an organized, coordinated, and collaborative network that brings together various physicians to deliver coordinated care and a continuum of services to a given patient population.7 Integrated delivery systems are clinically and fiscally accountable for the health status and outcomes for the population served, and they have systems to manage and to improve clinical outcomes. Key attributes of successful integrated delivery systems have been suggested.7 These attributes include: 1) shared values and goals, 2) patient-centeredness and a focus on population health, 3) coordination of care across a continuum of health care services and settings, 4) physician financial incentives that are aligned with patients' goals, 5) use of evidence-based practices, 6) electronic health records (EHR) that are accessible and shared by all physicians to track patients across a continuum, 7) the right mix of primary care and specialist physicians and appropriate medical equipment to serve the given population, and 8) continuous innovation and learning to improve the value of care.
A previous report highlighted that tightly integrated delivery systems with their own health plan may serve as a potential model of high-performing health care systems because the insurance function of these systems allows for greater flexibility and aligned incentives, and helps deliver high-value care.9 More specifically, Kaiser Permanente (KP), the largest nonprofit integrated delivery system in the US, may serve as a model of a high-performing health care system because of several unique aspects. KP serves various geographic populations in the US, including California, Colorado, District of Columbia, Georgia, Hawaii, Maryland, Oregon, Virginia, and Washington, with more than nine million active members in 2013. Patients in the KP system receive comprehensive, multidisciplinary health care, including all medical and surgical specialties as well as pharmacy, radiology, and laboratory services. In many geographic Regions, KP owns its hospitals; in the other Regions, KP has contracts with preferred hospitals. The population in KP is representative of the states they serve; data indicate that members overall are similar to the general population regarding age, sex, and race/ethnicity, with only slight underrepresentation of those in lower and higher income and education categories.10-12
Despite these key features, important questions remain about the "best practices" of integrated delivery systems that achieve superior outcomes. Policymakers are increasingly demanding high-quality research regarding which specific aspects of the integrated delivery model result in superior patient outcomes. For example, there are essential questions regarding how integrated systems are able to coordinate care among different specialties and how the use of information technology and clinical decision support systems are able to support transformational care delivery.7 Consequently, well-established integrated delivery systems, such as KP, should work to identify the specific system-level factors or confluence of factors that improves such services, as well as access, quality, and other such outcomes in an integrated delivery system. The answer to these fundamental questions may serve as a platform to inform and to guide emerging models of care delivery such as accountable care organizations and patient-centered medical homes.
Through a better understanding for which key systems and processes in integrated delivery systems work and the mechanisms by which they function, this knowledge may be translated and disseminated to the larger US health care delivery system. Comparative health systems research involves a comparison of the different approaches used by systems to organize and deliver health care services for a given population. Thus, comparative health systems is one area of research that may be able to provide valuable insights to policymakers and practitioners regarding which particular aspects of the integrated delivery system model result in improved care delivery and patient outcomes.
In response to policymakers' growing interest in this area, KP has embarked on a research agenda for comparative health systems. Therefore, the objective of this literature review was to examine the existing published studies on comparative health systems that relate to integrated delivery systems and KP, to obtain a baseline understanding of the state of comparative health systems research that can provide foundational knowledge. We also sought to identify, to quantify, and to classify the literature in this area.
To gain an understanding of the universe of research studies published on comparative health systems, we conducted a literature search on PubMed and the internal KP Publications Library. The KP Publications Library is a unique, full text searchable database of publications authored or coauthored by KP staff, including investigators, clinicians, and administrators, regardless of journal. The database contains 10,000 records describing journal articles, book chapters, books, letters, and commentaries. The library does not include posters, presentations, or published abstracts. In both literature searches, we sought to include existing studies that compared KP as a system or organization with other health care systems or organizations, or across KP facilities internally, in any topic area. We defined comparative health systems research as any study that compared KP as a system with another health system; any study that compared KP's performance with a state or national benchmark; and any study that compared KP's innovations in care delivery with old or previous models of care. We also included intra-KP studies that compared a system of care in or between another KP Region or in or between KP facilities. Any such types of these comparisons between different systems or models of care were defined as the systems of comparison. The search was inclusive of all subject areas, ranging from quality to information technology. We looked for explicit comparisons between KP and similar health care systems. Studies that included aggregated data from KP and other health care systems were excluded because there were no direct system-level comparisons that would allow us to disentangle the different health care systems.
In PubMed, the MeSH (Medical Subject Headings) search terms included Kaiser Permanente and comparative health services (n = 258), comparative health systems (n = 34), comparative integrated systems (n = 14), health services benchmarking (n = 10), health system benchmarking (n = 7), and quality benchmarking (n = 7; Table 1). From the 330 publications, we identified 16 studies involving direct system comparisons, which we categorized into KP to Other US, KP to Other International, and KP to KP (interregional or intraregional KP comparisons). The PubMed search yielded 16 relevant articles that met the criteria of comparative health systems research involving KP (Figure 1).
On the basis of PubMed results and additional refinement, we expanded our search to 20 keywords and topics to discover both external and internal comparisons from our KP Publications Library. The KP Publications Library search was used to find additional articles that may have been missed through the PubMed search because of differences in tagged words or keywords, articles that are not indexed, or delays in indexing.
We limited the final results from the KP Publications Library to the following criteria: 1) publication type: journal article only (no editorials, letters, and commentaries); 2) abstract: no publications without an abstract unless published in the last two months; and 3) date: no publications before 1995 because of the likelihood of lesser relevance.
The comprehensive search using the same 20 search terms in the KP Publications Library generated 1271 unique citations (Table 1). After limiting the search set to the previously stated criteria, 1132 citations required closer review. These studies were manually reviewed, and articles that were previously identified from PubMed were removed. We examined the results, first considering the study title and abstract and then reviewing the full text article, if necessary, to make a determination of appropriateness. Our examination of 1117 KP abstracts and full publications yielded 45 more publications relevant to the topic of comparative health systems research. We also identified 4 publications that were not found through our literature search (because they did not have a KP author listed or had no keyword hits from our search sets) and were provided to us by KP authors or identified through press releases because they were found to be relevant to the overall topic. We reviewed the final set of articles for agreement on inclusion.
In total, the literature search from the 2 comprehensive databases, in addition to publications identified outside our systematic review, resulted in 65 publications for inclusion in this analysis. Table 2 summarizes the included studies of comparative health systems. For each of the studies reviewed, we evaluated the system of comparison, topical area, and condition type. The topical area was the areas of comparison, the condition type was the disease or diseases of study, and the outcomes were the system outputs. The following topical areas were examined: resource use (ie, cost of care, utilization, length of stay); quality (ie, quality-improvement programs, quality performance, processes of care, patient outcomes of care, patient satisfaction); health information technology (ie, management of health information across computer systems); EHR (ie, electronic health information about patients); clinical decision support (ie, system that assists physicians with decision making related to patients); computerized physician order entry (ie, electronic entry of physician treatment orders); telemedicine (ie, telecommunications systems that provide health care across distances); health system performance (ie, health system delivery of care); self-care (ie, patient self-management of condition); disease management (ie, interventions to help patients cope with a condition); pharmacy consultation (ie, pharmacist counseling of patients regarding their medications); care delivery/care coordination (ie, provision and coordination of health care services); registries (ie, collection of data on patients with a specific condition); clinical integration (ie, integration of clinical information and health care services from different entities); patient safety (ie, prevention of medical errors); medication adherence (ie, patients taking medications as prescribed); and team performance (ie, team functioning).
Of all studies published by or about KP, only 4% of articles were identified as being comparative health systems research. The comparative health systems studies that were reviewed tended to focus mostly on quality of care (n = 30) and health information technology/EHR/clinical decision support/telemedicine (n = 18). Diabetes mellitus was also a common focus of the studies reviewed (n = 11).
Most studies identified in the literature search that met the criteria of comparative health systems research were intra-KP studies (interregional or intraregional that were in or between different KP Regions; n = 42).13-54 These studies either compared one KP Region with another for a particular care topic or compared a system of care in a KP Region that had heterogeneous processes among its different medical centers. Fewer studies (n = 15) were identified that compared KP with another US health care system (ie, fee-for-service, health maintenance organization, and/or Veterans Affairs).55-69 In addition, there were 12 studies that compared KP with international health care systems.55,58,62,63,70-77
Among the different topical areas that the comparative health systems studies covered, the most frequently studied topic was quality of care (n = 30)a and articles that related to health information technology/EHR/clinical decision support/telemedicine (n = 18).b Other commonly studied topics included resource use (cost/utilization; n = 16),c health systems performance (n = 7),14,59,61,70,72,73,76 and disease management (n = 7).13,15,17,25,36,40,47
On the basis of disease or type of condition, diabetes mellitus was the most frequently studied (n = 11).d Other conditions that were commonly studied included cardiovascular disease (n = 7),16,17,36,40,47,66,69 mental health and substance abuse (n = 6),26,30,32,45,57,67 and asthma/chronic obstructive pulmonary disease (n = 4).16,22,35,68
Several themes emerged from the literature as possible factors that may contribute to improved care delivery in integrated delivery systems. Seven studies suggested clinical integration as a possible reason for better performance.19,27,58,61,62,70,74 The use of technology (ie, electronic alerts, health information technology, EHR, secure messaging, remote video technology) was another common attribute cited across studies.e Last, a comprehensive approach to care delivery (ie, multidisciplinary care teams, comprehensive care management, interdisciplinary treatment, multimodal interventions) and self-management were other themes highlighted as possibly improving patient outcomes.f
In our review of the literature on comparative health systems research involving one or more KP entities, we found that most studies to date have focused primarily on intra-KP comparisons. Fewer studies compared KP with other US health care systems or international health care systems. One possible reason for this gap in the literature could be the lack of recognition of an integrated delivery system's ability to deliver high-quality services, the paucity of comparative performance data, and the unwillingness of organizations to share performance data. Furthermore, because most of the US health care system operates under a fee-for-service model, there are a limited number of other similar integrated delivery systems with a health plan component that may serve as suitable comparisons to KP. As a result, there remains much room for growth and additional research in comparative health systems that compare the KP model of care with other US health care systems, including more traditional fee-for-service care models, academic medical centers, Veterans Affairs medical centers, other integrated delivery systems (ie, Intermountain Healthcare based in Salt Lake City, UT; Geisinger Health System headquartered in Danville, PA; Group Health in Seattle, WA), the safety net (federally qualified health centers, community health centers, and free clinics), and international health care systems. Additional research in this area could examine which systems or processes work in improving care delivery and how different systems are able to achieve these outcomes. Improved performance is evidence that key processes contribute to better care. Further investigations into the types of best practices would lead to a more comprehensive understanding of which models or systems of care are most effective.
We also found that the comparative health systems studies we reviewed tended to focus on quality of care and health information technology/EHR/clinical decision support/telemedicine. In addition, some key attributes of integrated delivery systems emerged from the literature as possibly contributing to a higher performance. Clinical integration, the use of technology, comprehensive care, and patient self-management were consistent themes identified as being associated with improved care delivery. These have been areas of emphasis in practice and research in KP, and thus such findings are not surprising. The EHR and population health management programs are considered essential elements of an integrated approach to care that promotes a consistent and reliable care experience.7,9 However, additional research that examines other factors hypothesized to lead to a higher health system performance, such as physician financial incentives, patient-centeredness, and continuous innovation, should be further investigated. Our finding by type of condition revealed that diabetes mellitus and cardiovascular disease were the most commonly studied, likely because these are common and prevalent conditions and areas of research emphasis. Because most of the comparative health systems studies tended to focus on a limited number of condition types, studies that examine other common types of conditions, such as cancer, gastrointestinal diseases, and joint diseases, would further contribute to the body of literature.
There are a few limitations to this systematic review. First, our review of the literature from the KP Publications Library was restricted to studies published after 1995 until the most currently available at the time of the literature search. There may have been additional studies that were published before our study period or after our literature search was conducted. Second, we focused only on studies of comparative health systems that compared KP with other systems of care. There may be other comparative health systems studies that did not explicitly include KP as a comparator. We excluded studies that aggregated KP data with other health systems. Despite our best efforts, we acknowledge that we may have missed some articles in our literature search. However, we also asked other researchers at KP, as part of our systematic review, to ensure a complete and comprehensive literature search. Furthermore, this literature review did not attempt to examine or compare the outcomes of the comparative health systems studies. Rather, we sought only to identify, to classify, and to quantify the studies to help guide future research among large integrated delivery systems.
We found that studies published by or about KP rarely included comparative health systems research. Given the changing health care landscape and movement toward integrated care, additional empirical studies that compare the specific factors of the integrated delivery system model with other systems of care (or in KP if there is heterogeneity of such care) may identify the system-level factors that result in more efficient care delivery. Additionally, more work must be done in partnership with similar health care organizations to demonstrate the benefits of integration toward quality, affordability, accessibility, and effectiveness. Such investigations could seek to understand how systems work to improve clinical outcomes and examine what are the key characteristics of successful systems. By developing the capacity to conduct and communicate the outcomes of comparative health systems research, the health care industry will be able to disseminate and translate the best practices that are able to address issues of quality, affordability, access, and effectiveness. It is important for all to gain organizational commitment to address the research questions that compare each different system's performance with rigor and transparency. The knowledge gained from comparative health systems research will enable the dissemination and translation of best practices that can be adopted by the larger US health care delivery system and ensure high-quality, effective care for all.
a References: 13, 16, 18, 19, 21, 23-25, 27, 28, 31, 35, 36, 40, 41, 44-46, 50, 51, 54, 57, 59, 60, 65, 66, 68, 69, 75, 77.
This study was sponsored by the Kaiser Permanente Community Benefits Program in Oakland, CA. The sponsor did not have any role in the conduct of the literature search, in writing of this report, or in the decision to submit for publication.
The authors wish to acknowledge Raymond J Baxter, PhD, for his generous support of this work and Maria Faer, DrPH, for her assistance with the literature search and organization.
Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.
1. Orszag PR, Ellis P. The challenge of rising health care costs—a view from the Congressional Budget Office. N Engl J Med 2007 Nov 1;357(18):1793-5. DOI: https://doi.org/10.1056/NEJMp078190.
2. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003 Jun 26;348(26):2635-45. DOI: https://doi.org/10.1056/NEJMsa022615.
3. The Patient Protection and Affordable Care Act of 2010. Public Law 111-148, 111th Congress, 124 Stat 119, HR 3590, enacted 2010 Mar 23.
4. Guterman S, Davis K, Schoenbaum S, Shih A. Using Medicare payment policy to transform the health system: a framework for improving performance. Health Aff (Millwood) 2009 Mar-Apr;28(2):w238-50. DOI: https://doi.org/10.1377/hlthaff.28.2.w238.
5. Woolf SH, Aaron L, editors; Panel on Understanding Cross-National Health Differences Among High-Income Countries; Committee on Population (CPOP); Division of Behavioral and Social Sciences and Education (DBASSE); National Research Council; Board on Population Health and Public Health Practice (BPH); Institute of Medicine (IOM). US health in international perspective: shorter lives, poorer health. Washington, DC: The National Academies Press: 2013 Jan 9.
6. Rittenhouse DR, Shortell SM. The patient-centered medical home: will it stand the test of health reform? JAMA 2009 May 20;301(19):2038-40. DOI: https://doi.org/10.1001/jama.2009.691.
7. Enthoven AC. Integrated delivery systems: the cure for fragmentation. Am J Manag Care 2009 Dec;15(10 Suppl):S284-90.
8. Crosson FJ. 21st-century health care—the case for integrated delivery systems. N Engl J Med 2009 Oct 1;361(14):1324-5. DOI: https://doi.org/10.1056/NEJMp0906917.
9. McCarthy D, Mueller K. Organizing for higher performance: case studies of organized delivery systems: series overview, findings, and methods [Internet]. New York, NY: The Commonwealth Fund; 2009 Jul [cited 2013 Oct 1]. Available from: www.commonwealthfund.org/~/media/Files/Publications/Case%20Study/2009/Jul/1288_McCarthy_Overview_report_final.pdf.
10. Krieger N. Overcoming the absence of socioeconomic data in medical records: validation and application of a census-based methodology. Am J Public Health 1992 May;82(5):703-10. DOI: https://doi.org/10.2105/AJPH.82.5.703.
11. Gordon NP. How does the adult Kaiser Permanente membership in Northern California compare with the larger community? [Internet]. Oakland, CA: Kaiser Permanente Northern California Division of Research; 2006 Jun 14 [cited 2013 Sep 6]. Available from: www.dor.kaiser.org/external/uploadedFiles/content/research/mhs/_2011_Revised_Site/Documents_Special_Reports/comparison_kaiser_vs_nonKaiser_adults_kpnc(1).pdf.
12. Koebnick C, Langer-Gould AM, Gould MK, et al. Sociodemographic characteristics of members at a large, integrated health care system: comparison with US Census Bureau data. Perm J 2012 Summer;16(3):37-41. DOI: https://doi.org/10.7812/TPP/12-031.
13. Domurat ES. Diabetes managed care and clinical outcomes: the Harbor City, California Kaiser Permanente diabetes care system. Am J Manag Care 1999 Oct;5(10):1299-307.
14. Horberg M, Hurley LB, Towner WJ, et al. Determination of optimized multidisciplinary care team for maximal antiretroviral therapy adherence. J Acquir Immune Defic Syndr 2012 Jun 1;60(2):183-90. DOI: https://doi.org/10.1097/QAI.0b013e31824bd605.
15. Brown JB, Nichols GA, Glauber HS. Case-control study of 10 years of comprehensive diabetes care. West J Med 2000 Feb;172(2):85-90. DOI: https://doi.org/10.1136/ewjm.172.2.85.
16. Johnston B, Wheeler L, Deuser J, Sousa KH. Outcomes of the Kaiser Permanente Tele-Home Health Research Project. Arch Fam Med 2000 Jan;9(1):40-5. DOI: https://doi.org/10.1001/archfami.9.1.40.
17. Merenich JA, Lousberg TR, Brennan SH, Calonge NB. Optimizing treatment of dyslipidemia in patients with coronary artery disease in the managed-care environment (the Rocky Mountain Kaiser Permanente experience). Am J Cardiol 2000 Feb 10;85(3A):36A-42A. DOI: https://doi.org/10.1016/S0002-9149(99)00937-6.
18. Nichols GA, Glauber HS, Javor K, Brown JB. Achieving further glycemic control in type 2 diabetes mellitus. West J Med 2000 Sep;173(3):175-9. DOI: https://doi.org/10.1136/ewjm.173.3.175.
19. Perry B. Quality improvement in a managed care organization from a medical director's perspective: an interview with Bruce Perry. Interview by Douglas Roblin. Jt Comm J Qual Improv 2000 Oct;26(10):601-10.
20. Lorig KR, Sobel DS, Ritter PL, Laurent D, Hobbs M. Effect of a self-management program on patients with chronic disease. Eff Clin Pract 2001 Nov-Dec;4(6):256-62.
21. Thompson M, Gee S, Larson P, Kotz K, Northrop L. Health and loyalty promotion visits for new enrollees: results of a randomized controlled trial. Patient Educ Couns 2001 Jan;42(1):53-65. DOI: https://doi.org/10.1016/S0738-3991(00)00096-3.
22. Vollmer WM, Feldstein A, Smith DH, et al. Use of health information technology to improve medication adherence. Am J Manag Care 2011 Dec;17(12 Spec No.):SP79-87. DOI: https://doi.org/10.3121/cmr.2011.1020.c-b4-03.
23. Shafer MA, Tebb KP, Pantell RH, et al. Effect of a clinical practice improvement intervention on Chlamydial screening among adolescent girls. JAMA 2002 Dec 11;288(22):2846-52. DOI: https://doi.org/10.1001/jama.288.22.2846.
24. Yuan Y, Hay JW, McCombs JS. Effects of ambulatory-care pharmacist consultation on mortality and hospitalization. Am J Manag Care 2003 Jan;9(1):45-56.
25. Taylor CB, Miller NH, Reilly KR, et al. Evaluation of a nurse-care management system to improve outcomes in patients with complicated diabetes. Diabetes Care 2003 Apr;26(4):1058-63. DOI: https://doi.org/10.2337/diacare.26.4.1058.
26. Finley PR, Rens HR, Pont JT, et al. Impact of a collaborative care model on depression in a primary care setting: a randomized controlled trial. Pharmacotherapy 2003 Sep;23(9):1175-85. DOI: https://doi.org/10.1592/phco.23.10.1175.32760.
27. Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. Effect of electronic health records in ambulatory care: retrospective, serial, cross sectional study. BMJ 2005 Mar 12;330(7491):581. DOI: https://doi.org/10.1136/bmj.330.7491.581.
28. Patel PB, Vinson DR. Team assignment system: expediting emergency department care. Ann Emerg Med 2005 Dec;46(6):499-506. DOI: https://doi.org/10.1016/j.annemergmed.2005.06.012.
29. Lorig KR, Hurwicz ML, Sobel D, Hobbs M, Ritter PL. A national dissemination of an evidence-based self-management program: a process evaluation study. Patient Educ Couns 2005 Oct;59(1):69-79. DOI: https://doi.org/10.1016/j.pec.2004.10.002.
30. Stubbings T, Miller C, Humphries TL, Nelson KM, Helling DK. Telepharmacy in a health maintenance organization. Am J Health Syst Pharm 2005 Feb 15;62(4):406-10.
31. Grypma L, Haverkamp R, Little S, Unützer J. Taking an evidence-based model of depression care from research to practice: making lemonade out of depression. Gen Hosp Psychiatry 2006 Mar-Apr;28(2):101-7. DOI: https://doi.org/10.1016/j.genhosppsych.2005.10.008.
32. McFarland BH, Lynch FL, Freeborn DK, et al. Substance abuse treatment duration for Medicaid versus commercial clients in a health maintenance organization. Med Care 2006 Jun;44(6):601-6. DOI: https://doi.org/10.1097/01.mlr.0000215814.01127.ce.
33. Palen TE, Raebel M, Lyons E, Magid DE. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care 2006 Jul;12(7):389-95.
34. Simon SR, Smith DH, Feldstein AC, et al. Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people. J Am Geriatr Soc 2006 Jun;54(6):963-8. DOI: https://doi.org/10.1111/j.1532-5415.2006.00734.x.
35. Vollmer WM, Kirshner M, Peters D, et al. Use and impact of an automated telephone outreach system for asthma in a managed care setting. Am J Manag Care 2006 Dec;12(12):725-33.
36. McConnell KJ, Zadvorny EB, Hardy AM, Delate T, Rasmussen JR, Merenich JA; Clinical Pharmacy Cardiac Risk Service Study Group. Coronary artery disease and hypertension: outcomes of a pharmacist-managed blood pressure program. Pharmacotherapy 2006 Sep;26(9):1333-41. DOI: https://doi.org/10.1592/phco.26.9.1333.
37. Hornbrook MC, Whitlock EP, Berg CJ, et al. Development of an algorithm to identify pregnancy episodes in an integrated health care delivery system. Health Serv Res 2007 Apr;42(2):908-27. DOI: https://doi.org/10.1111/j.1475-6773.2006.00635.x.
38. Humphries TL, Carroll N, Chester EA, Magid D, Rocho B. Evaluation of an electronic critical drug interaction program coupled with active pharmacist intervention. Ann Pharmacother 2007 Dec;41(12):1979-85. DOI: https://doi.org/10.1345/aph.1K349.
39. McGaw J, Conner DA, Delate TM, Chester EA, Barnes CA. A multidisciplinary approach to transition care: a patient safety innovation study. Perm J 2007 Fall;11(4):4-9. DOI: https://doi.org/10.7812/TPP/07-012.
40. Merenich JA, Olson KL, Delate T, Rasmussen J, Helling DK, Ward DG; Clinical Pharmacy Cardiac Risk Service Study Group. Mortality reduction benefits of a comprehensive cardiac care program for patients with occlusive coronary artery disease. Pharmacotherapy 2007 Oct;27(10):1370-8. DOI: https://doi.org/10.1592/phco.27.10.1370.
41. Neuwirth EE, Schmittdiel JA, Tallman K, Bellows J. Understanding panel management: a comparative study of an emerging approach to population care. Perm J 2007 Summer;11(3):12-20. DOI: https://doi.org/10.7812/TPP/07-040.
42. Raebel MA, Charles J, Dugan J, et al. Randomized trial to improve prescribing safety in ambulatory elderly patients. J Am Geriatr Soc 2007 Jul;55(7):977-85. DOI: https://doi.org/10.1111/j.1532-5415.2007.01202.x.
43. Graetz I, Reed M, Rundall T, Bellows J, Brand R, Hsu J. Care coordination and electronic health records: connecting clinicians. AMIA Annu Symp Proc 2009 Nov 14;2009:208-12.
44. Schmittdiel JA, Uratsu CS, Fireman BH, Selby JV. The effectiveness of diabetes care management in managed care. Am J Manag Care 2009 May;15(5):295-301. DOI: https://doi.org/10.3121/cmr.6.3-4.125-a.
45. Sterling S, Chi F, Campbell C, Weisner C. Three-year chemical dependency and mental health treatment outcomes among adolescents: the role of continuing care. Alcohol Clin Exp Res 2009 Aug;33(8):1417-29. DOI: https://doi.org/10.1111/j.1530-0277.2009.00972.x.
46. Bowman B, Smith S. Primary care DirectConnect: how the marriage of call center technology and the EMR brought dramatic results—a service quality improvement study. Perm J 2010 Summer;14(2):18-24. DOI: https://doi.org/10.7812/TPP/10-012.
47. Delate T, Olson KL, Rasmussen J, et al; Clinical Pharmacy Cardiac Risk Service Study Group. Reduced health care expenditures after enrollment in a collaborative cardiac care service. Pharmacotherapy 2010 Nov;30(11):1127-35. DOI: https://doi.org/10.1592/phco.30.11.1127.
48. Smith N, Iyer RL, Langer-Gould A, et al. Health plan administrative records versus birth certificate records: quality of race and ethnicity information in children. BMC Health Serv Res 2010 Nov 23;10:316. DOI: https://doi.org/10.1186/1472-6963-10-316.
49. Baer D. Patient-physician e-mail communication: the Kaiser Permanente experience. J Oncol Pract 2011 Jul;7(4):230-3. DOI: https://doi.org/10.1200/JOP.2011.000323.
50. Moiel D, Thompson J. Early detection of colon cancer—the Kaiser Permanente Northwest 30-year history: how do we measure success? Is it the test, the number of tests, the stage, or the percentage of screen-detected patients? Perm J 2011 Fall;15(4):30-8. DOI: https://doi.org/10.7812/TPP/11-128.
51. Naber CM, Water-Schmeder O, Bohrer PS, Matonak K, Bernstein AL, Merchant MA. Interdisciplinary treatment for vestibular dysfunction: the effectiveness of mindfulness, cognitive-behavioral techniques, and vestibular rehabilitation. Otolaryngol Head Neck Surg 2011 Jul;145(1):117-24. DOI: https://doi.org/10.1177/0194599811399371.
52. Nelson C, Chand P, Sortais J, Oloimooja J, Rembert G. Inpatient palliative care consults and the probability of hospital readmission. Perm J 2011 Spring;15(2):48-51. DOI: https://doi.org/10.7812/TPP/10-142.
53. Roblin DW, Howard DH, Junling Ren, Becker ER. An evaluation of the influence of primary care team functioning on the health of Medicare beneficiaries. Med Care Res Rev 2011 Apr;68(2):177-201. DOI: https://doi.org/10.1177/1077558710374619.
54. Ferrara A, Hedderson MM, Ching J, Kim C, Peng T, Crites YM. Referral to telephonic nurse management improves outcomes in women with gestational diabetes. Am J Obstet Gynecol 2012 Jun;206(6):491.e1-5. DOI: https://doi.org/10.1016/j.ajog.2012.04.019.
55. Thorner RM. The use of health services by civilian beneficiaries of the military health care system. A comparative study. Med Care 1978 Apr;16(4):267-88. DOI: https://doi.org/10.1097/00005650-197804000-00001.
56. DeCoster CA, Smoller M, Roos NP, Thomas E. A comparison of ambulatory care and selected procedure rates in the health care systems of the Province of Manitoba, Canada; Kaiser Permanente Health Maintenance Organization; and the United States. Healthc Manag Forum 1997 Winter;10(4):26-9, 32-4. DOI: https://doi.org/10.1016/S0840-4704(10)60978-7.
57. Rubenstein LV, Parker LE, Meredith LS, et al. Understanding team-based quality improvement for depression in primary care. Health Serv Res 2002 Aug;37(4):1009-29. DOI: https://doi.org/10.1034/j.1600-0560.2002.63.x.
58. Ham C, York N, Sutch S, Shaw R. Hospital bed utilisation in the NHS, Kaiser Permanente, and the US Medicare programme: analysis of routine data. BMJ 2003 Nov 29;327(7426):1257. DOI: https://doi.org/10.1136/bmj.327.7426.1257.
59. Fishman PA, Hornbrook MC, Meenan RT, Goodman MJ. Opportunities and challenges for measuring cost, quality, and clinical effectiveness in health care. Med Care Res Rev 2004 Sep;61(3 Suppl):124S-43S. DOI: https://doi.org/10.1177/1077558704267512.
60. Kerr EA, Gerzoff RB, Krein SL, et al. Diabetes care quality in the Veterans Affairs Health Care System and commercial managed care: the TRIAD study. Ann Intern Med 2004 Aug 17;141(4):272-81. DOI: https://doi.org/10.7326/0003-4819-141-4-200408170-00007.
61. Kim C, Williamson DF, Mangione CM, et al. Managed care organization and the quality of diabetes care: the Translating Research Into Action for Diabetes (TRIAD) study. Diabetes Care 2004 Jul;27(7):1529-34. DOI: https://doi.org/10.2337/diacare.27.7.1529.
62. Profit J, Zupancic JA, McCormick MC, et al. Moderately premature infants at Kaiser Permanente Medical Care Program in California are discharged home earlier than their peers in Massachusetts and the United Kingdom. Arch Dis Child Fetal Neonatal Ed 2006 Jul;91(4):F245-50. DOI: https://doi.org/10.1136/adc.2005.075093.
63. Smith DH, Perrin N, Feldstein A, et al. The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation. Arch Intern Med 2006 May 22;166(10):1098-104. DOI: https://doi.org/10.1001/archinte.166.10.1098.
64. Stiefel M, Feigenbaum P, Fisher ES. The Dartmouth Atlas applied to Kaiser Permanente: analysis of variation in care at the end of life. Perm J 2008 Winter;12(1):4-9. DOI: https://doi.org/10.7812/TPP/07-120.
65. Horberg M, Hurley L, Towner W, et al. HIV quality performance measures in a large integrated health care system. AIDS Patient Care STDS 2011 Jan;25(1):21-8. DOI: https://doi.org/10.1089/apc.2010.0315.
66. Magid DJ, Ho PM, Olson KL, et al. A multimodal blood pressure control intervention in 3 healthcare systems. Am J Manag Care 2011 Apr;17(4):e96-103.
67. Wisdom JP, Ford JH 2nd, Wise M, Mackey D, Green CA. Substance abuse treatment programs' data management capacity: an exploratory study. J Behav Health Serv Res 2011 Apr;38(2):249-64. DOI: https://doi.org/10.1007/s11414-010-9221-z.
68. Hazelhurst B, McBurnie MA, Mularski RA, Puro JE, Chauvie SL. Automating care quality measurement with health information technology. Am J Manag Care 2012 Jun;18(6):313-9.
69. Schroeder EB, Hanratty R, Beaty BL, Bayliss EA, Havranek EP, Steiner JF. Simultaneous control of diabetes mellitus, hypertension, and hyperlipidemia in 2 health systems. Circ Cardiovasc Qual Outcomes 2012 Sep 1;5(5):645-53. DOI: https://doi.org/10.1161/CIRCOUTCOMES.111.963553.
70. Feachem RG, Sekhri NK, White KL. Getting more for their dollar: a comparison of the NHS with California's Kaiser Permanente. BMJ 2002 Jan 19;324(7330):135-41. DOI: https://doi.org/10.1136/bmj.324.7330.135.
71. Séror AC. Internet infrastructures and health care systems: a qualitative comparative analysis on networks and markets in the British National Health Service and Kaiser Permanente. J Med Internet Res 2002 Dec;4(3):E21. DOI: https://doi.org/10.2196/jmir.4.3.e21.
72. Towill DR. Viewing Kaiser Permanente via the logistician lens. Int J Health Care Qual Assur Inc Leadersh Health Serv 2006;19(4-5):296-315. DOI: https://doi.org/10.1108/09526860610671364.
73. Frølich A, Schiøtz ML, Strandberg-Larsen M, et al. A retrospective analysis of health systems in Denmark and Kaiser Permanente. BMC Health Serv Res 2008 Dec 11;8:252. DOI: https://doi.org/10.1186/1472-6963-8-252.
74. Strandberg-Larsen M, Schiøtz ML, Silver JD, et al. Is the Kaiser Permanente model superior in terms of clinical integration?: a comparative study of Kaiser Permanente, Northern California and the Danish healthcare system. BMC Health Serv Res 2010 Apr 8;10:91. DOI: https://doi.org/10.1186/1472-6963-10-91.
75. Paxton EW, Furnes O, Namba RS, Inacio MC, Fenstad AM, Havelin LI. Comparison of the Norwegian knee arthroplasty register and a United States arthroplasty registry. J Bone Joint Surg Am 2011 Dec 21;93 Suppl 3:20-30. DOI: https://doi.org/10.2106/JBJS.K.01045.
76. Schiøtz M, Price M, Frølich A, et al. Something is amiss in Denmark: a comparison of preventable hospitalisations and readmissions for chronic medical conditions in the Danish Healthcare system and Kaiser Permanente. BMC Health Serv Res 2011 Dec 22;11:347. DOI: https://doi.org/10.1186/1472-6963-11-347.
77. Schiøtz M, Strandberg-Larsen M, Frølich A, et al. Self-management support to people with type 2 diabetes—a comparative study of Kaiser Permanente and the Danish Healthcare System. BMC Health Serv Res 2012 Jun 14;12:160. DOI: https://doi.org/10.1186/1472-6963-12-160.