Assessment of Pharmacy Department Patient Safety Culture with the Use of Validated Work Environment Survey Indices

Assessment of Pharmacy Department Patient Safety Culture  with the Use of Validated Work Environment Survey Indices

 

Julia E Rawlings, PharmD; Sheryl J Herner, PharmD, MHSA;
Thomas Delate, PhD, MS; Kelsey E Palmer, PharmD;
Kelly A Swartzendruber, PharmD

Perm J 2018;22:17-070 [Full Citation]

https://doi.org/10.7812/TPP/17-070
E-pub: 01/03/2018

ABSTRACT

Context: Patient safety culture (PSC) improvement is a strategy that can foster patient well-being. Measuring PSC without using a validated instrument has been proposed.
Objective: To assess the strengths of correlations between the Agency for Healthcare Research and Quality’s validated Community Pharmacy Survey on Patient Safety Culture (CPSPSC) composites and nonvalidated People Pulse (PP) work environment indices.
Design: This was a cross-sectional, anonymous, voluntary survey.
Methods: This survey was conducted online in an ambulatory pharmacy department that had approximately 900 staff members within an integrated health care delivery system. All pharmacy department staff were asked to participate. CPSPSC composites and PP indices were calculated as percent-positive scores (PPS). Thirty-six correlations between PP index and CPSPSC composite PPS were assessed with the Pearson product-moment correlation coefficient. Correlation strengths were interpreted as very weak (± 0.00 to 0.19), weak (± 0.20 to 0.39), moderate (± 0.40 to 0.59), strong (± 0.60 to 0.79), and very strong (± 0.80 to 1.0).
Results: A total of 429 (47.4%) pharmacy staff participated. Overall, correlations between CPSPSC composites and PP indices’ PPS varied widely. Although all correlations were positive and the majority of correlations were statistically significant, no correlations were identified as very strong. Ten correlations were strong, 16 were moderate, 4 were weak, and 3 were very weak.
Conclusion: Although health care system personnel may prefer to measure PSC with a survey instrument that assesses a variety of workplace environment measures, these findings suggest that use of nonvalidated work environment indices will not provide accurate assessment of PSC in a pharmacy department.

INTRODUCTION

Patient safety is a top health care organization priority.1 A thriving patient safety culture (PSC) exists in health care settings that emphasize cooperation and personal commitment. When a PSC is “healthy,” safety is valued and staff members perceive honest dedication to safety by members of their organization, including those in leadership roles.2 For example, in environments in which PSC is a priority, a pharmacy technician who overhears another technician provide incorrect information to a patient has no reservations about reporting the incident and no fear of supervisor retaliation or being shunned by colleagues who discover s/he spoke up.

Among the various aspects of patient safety in ambulatory settings (including infection control, staff competence, and medical equipment), ensuring medication safety and PSC have been identified as some of the most important strategies that can foster patient well-being.3 The Agency for Healthcare Research and Quality (AHRQ) developed and validated the Community Pharmacy Survey on Patient Safety Culture (CPSPSC) (formerly the Pharmacy Survey on Patient Safety Culture) to acknowledge the importance of medication safety and PSC in ambulatory pharmacies.4 

It is essential to use a validated questionnaire to generate data that accurately reflect the reality of what was measured (in this scenario, PSC). Validated questionnaires allow for findings to be applied more broadly than to the population that supplied the data. Numerous techniques can be employed to evaluate a questionnaire’s 1) reliability (the same results are produced on retesting), 2) internal validity (similar questions are responded to in a similar way), 3) external validity (responses in different samples are similar), 4) sensitivity (conditions that are present are identified accurately), 5) specificity (conditions that are not present are identified accurately), 6) discriminant validity (actual differences/nondifferences between groups are detected accurately), and 7) construct validity (the questionnaire accurately measures the things it is designed to measure).5 Although qualitative research (eg, focus groups, structured interviews) can be used to assess PSC, surveying with a validated questionnaire is the most effective means with which to collect data across a large organization.2

The overarching goals of the CPSPSC, with its 11 composite scores, are to assess understanding of the values, beliefs, and norms of ambulatory pharmacy staff and the expected patient safety attitudes and behaviors that are appropriate in community pharmacies.4 Few validated community pharmacy-specific PSC measurement tools have been created or are widely administered.6-8 In 2013, the Kaiser Permanente (KP) Colorado (KPCO) Pharmacy Department administered the CPSPSC to its entire staff to assess CPSPSC goals and to provide baseline data that KPCO and other community pharmacies could use to compare their PSC over time.7,9

Measuring PSC in pharmacy departments without using the validated CPSPSC has been explored nationally by KP using indices from the People Pulse (PP) survey that measure aspects of the work environment.10 Three indices, Speaking Up, Workplace Safety, and Learning Climate have been proposed by the KP Program Office as a substitute for measurement of PSC.11 The rationale for using the nonvalidated PP indices is that KP conducts a survey annually throughout the organization using the entire PP questionnaire. Administering the CPSPSC would necessitate special effort across the various KP Pharmacy Departments. Therefore, these PP indices may provide a more efficient way to measure KP PSC.

Little information exists regarding the validity of this approach. The purpose of this study was to conduct a cross-sectional survey with simultaneous administration of items from both the CPSPSC and the PP Speaking Up, Workplace Safety, and Learning Climate indices to assess the strengths of correlations between the CPSPSC composite and PP index scores. Results from this study can provide valuable information to key stakeholders about the validity of using the three PP indices as a substitute for administering the full CPSPSC to assess PSC among community/outpatient pharmacy staff.

METHODS

Study Design and Setting

This was a cross-sectional survey of KPCO Pharmacy Department staff conducted between July 5, 2016, and September 5, 2016. The survey consisted of 39 items for 11 composites and the 1-item Overall Rating on Patient Safety from the CPSPSC (Appendix A, available at: www.thepermanentejournal.org/files/2018/17-070-Appendix-A.pdf), 22 items for the 3 indices from the PP survey (Appendix B, available at: www.thepermanentejournal.org/files/2018/17-070-Appendix-B.pdf), and demographic questions. The survey was administered online, and no respondent identifiers were collected.

KPCO is a group model, not-for-profit, integrated health care delivery system with more than 630,000 members throughout Colorado who received care at 28 medical offices with an embedded outpatient pharmacy at the time of the study. All study aspects were reviewed and approved by the KPCO institutional review board.

Participants

All full- and part-time Pharmacy Department staff who were KPCO employees at the time of the study (N = 906) were invited to participate. There were no exclusion criteria. The KPCO Pharmacy Department includes dispensing pharmacy services (DPS), for which staff provide care via outpatient and infusion pharmacies and a centralized prescription fill center that also provides mail order services.12 The KPCO Pharmacy Department also includes clinical pharmacy services (CPS), for which staff provide expertise in primary care,13 specialty care,14 acute and chronic disease management,15,16 and telepharmacy17 services. Support personnel within the KPCO Pharmacy Department fill information technology, medication benefits management, project management, compliance, data analysis, research, management, and administrative support roles.

Outcomes

The primary outcome was to assess correlation strength between each of the 11 CPSPSC composite percent-positive scores (PPS) and the 3 PP index PPS. Secondary outcomes included assessment of correlation strength for CPS and DPS staff individually, a description of the CPSPSC composite and PP index PPS overall, and contrast of the CPSPSC composite and PP index PPS by CPS vs DPS. Subgroup analyses were performed to describe and contrast the correlations and PSC in relevant staff groups to provide more granularity of information according to pharmacy practice site or work setting.

Survey Items

The PP Speaking Up index assesses the extent to which people work together, speak up, and learn from mistakes in the workplace environment.11 The PP Workplace Safety index assesses elements that foster a safe working environment.11 The PP Learning Climate index assesses the type of work environment in which people feel valued, have a say in their jobs, and perceive action is taken that is related to their input.11 The PP indices’ responses are scaled: Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree, and Don’t Know or Not Applicable.

The CPSPSC Mistakes Communication, Communication Across Shifts, and Communication Openness composites measure how well staff discuss mistakes that happen and ways to prevent them, information about prescriptions across shifts, and patient safety concerns and staff level of comfort about asking questions.4 These composites’ responses are Never, Rarely, Sometimes, Most of the Time, Always, and Does Not Apply or Don’t Know.

The CPSPSC Organizational Learning, Overall Perceptions (of patient safety), and Patient Counseling composites measure how effectively a pharmacy works to identify problems in the work process that can lead to mistakes and then implements changes to keep mistakes from happening again; if there is a strong focus and emphasis on patient safety; and if pharmacists spend enough time talking to patients, respectively.4 The Organizational Learning and Overall Perceptions composites’ responses are Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree, and Does Not Apply or Don’t Know, and the Patient Counseling composite’s responses are Never, Rarely, Sometimes, Most of the Time, Always, and Does Not Apply or Don’t Know.

The CPSPSC Physical Space and Environment and Responses to Mistakes composites measure how well a pharmacy layout supports good workflow. The composites also examine why mistakes happen, if staff members receive help so they can learn from mistakes, and if they are treated fairly when they make mistakes, respectively.4 These composites’ responses are Never, Rarely, Sometimes, Most of the Time, Always, and Does Not Apply or Don’t Know.

In addition, the CPSPSC Staff Training and Skills, Staffing, Work Pressure and Pace, and Teamwork composites measure how well staff receive the training they need; if they have the skills they need to do their jobs well; if enough staff is available to handle the workload; and if staff treat each other with respect and work together as an effective team, respectively.4 The Staff Training and Skills and Teamwork composites’ responses are Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree, and Does Not Apply or Don’t Know, and the Staffing, Work Pressure, and Pace composite’s responses are Never, Rarely, Sometimes, Most of the Time, Always, and Does Not Apply or Don’t Know.

The CPSPSC one-item Overall Rating (on patient safety) allows participants to rate (Poor, Fair, Good, Very Good, Excellent) their overall perception of the PSC in their respective pharmacies.4

Data Collection

Items from the CPSPSC composites and PP indices were transcribed and combined into a single SurveyMonkey (Palo Alto, CA) online survey format. Basic demographic information (position [pharmacist, management, resident, technician], practice site [eg, DPS, CPS], and length of employment at KPCO) also was collected.

Staff were invited to participate via Email messages sent from the Pharmacy Department’s Acting Executive Director. The message contained information on the purpose of the voluntary study, a link to the online questionnaire, a statement indicating that all responses were anonymous and confidential (responses were not linked to individual staff members), and directions that mentioned the survey could be completed in 15 to 20 minutes during business hours. A separate Email message that was sent to all department supervisors directed them to encourage staff participation and to provide staff with protected time to participate in the survey. Promotional flyers for the survey were distributed by Email for supervisors to post in their pharmacy department service areas.

Data Analysis

Completed questionnaires were reviewed to assess for common deficiencies of missing (eg, only responses to demographic/composite questions), limited (eg, responses were the same value throughout), and problematic (eg, more than one response per item) responses. The response rate was calculated as the count of completed questionnaires divided by the count of eligible staff receiving the survey.

Does Not Apply and Don’t Know responses were recoded as missing values since they are uninformative. Both PP indices and CPSPSC composites were scored by summing the count of participants who responded in the affirmative (eg, agree or strongly agree) for positively worded items or negative (eg, disagree or strongly disagree) for negatively worded items in the index/composite. These values were divided by the number of responders and multiplied by 100 to determine the PPS for each item and the mean PPS for all items in the index/composite. Item responses of Very Good and Excellent were combined into a positive response, and Poor, Fair, and Good were combined into a negative response for the Overall Rating on Patient Safety item.

Composite scores were not used if 3 or fewer respondents answered a particular item from a single group or location.4 This requirement protected anonymity further by preventing identification of individual respondents by job location, job role, length of employment, etc. Missing values were not used to calculate PPS. A PPS can range between 0% and 100%, with higher scores indicating a culture more favorable for patient safety. Subgroup analyses were performed by recalculating the indices, composites, and one-item PPS for only the staff included in each subgroup (CPS and DPS). Descriptive statistics for respondent characteristics and PPS were reported. Comparisons between CPS and DPS PPS were assessed with the Wilcoxon rank-sum test. Correlations between PP indices and CPSPSC composites and one-item PPS were assessed with the Pearson product-moment correlation coefficient. Because the correlations were calculated using individual staff responses, composite scores did not require 4 or more respondents for an item to be included. Correlation coefficient strength was interpreted as ± 0.00 to 0.19, very weak; ± 0.20 to 0.39, weak; ± 0.40-0.59, moderate; ± 0.60 to 0.79, strong; and ± 0.80-1.0, very strong.18 Correlation coefficients were reported overall and by the CPS and DPS groups. Correlation coefficients were compared between the CPS and DPS groups using the Fisher r-to-z transformation.

RESULTS

Of 906 staff members, 429 (47.4%) participated in the survey. Respondents most commonly reported KPCO length of employment for 1 year to 3 years and at least 12 years, worked 32 to 40 hours per week, were employed in the DPS and other pharmacy services, and were pharmacists (Table 1). Because the survey was voluntary and anonymous, data were unavailable to compare respondents with nonrespondents.

The CPSPSC composites and one-item overall safety rating PPS ranged between 41.1% (Staffing and Work Pressure) and 91.3% (Patient Counseling), whereas the PPS for the PP indices ranged between 70.5% (Learning Climate) and 73.9% (Workplace Safety) for all pharmacy staff respondents (Table 2). When compared with DPS staff respondents, CPS staff respondents had higher PPS for the Communication Openness, Perceptions of Safety, Teamwork, and Physical Space and Environment CPSPSC composites, the CPSPSC one-item Overall Safety rating, and the Workplace Safety PP index. The differences between DPS and CPS in each of these survey areas were statistically significant (all p < 0.05). Conversely, the DPS staff respondents had higher PPS for the Staffing and Work Pressure CPSPSC composite (p < 0.001).

Overall, PPS correlations between CPSPSC composite and one-item overall safety rating and PP indices varied widely (Table 3). Although all correlations were positive and the vast majority of composite and index PPS were statistically significantly correlated, no correlations were identified as very strong. The majority (52.8%) of correlations were moderate, 10 (27.8%) were strong, 4 (11.1%) were weak, and 3 (8.3%) were very weak. The strongest correlation was between the Communication Openness composite and Speaking Up index (r = 0.74, p < 0.001). The weakest correlations were between the Work Pressure and Pace composite and Learning Climate (r < 0.01) and Speaking Up (r < 0.01) indices (both p > 0.05). The Overall Rating of Pharmacy Safety had weak to moderate correlations with the 3 PP indices.

Most PPS correlations between the CPSPSC composite and 1-item overall safety rating and PP index PPS for CPS and DPS staff respondents were positive but not statistically significantly different (Table 4). Most correlations between CPSPSC composite and 1-item and PP index PPS were numerically higher for CPS staff respondents when contrasted against DPS staff respondents’ correlations. One correlation was very strong, the CPSPSC Response to Mistakes composite and PP Learning Climate index for CPS staff respondents (r = 0.80), and significantly higher than the moderate correlation for the DPS staff respondents (r = 0.49, p < 0.01). Additionally, the CPSPSC Response to Mistakes composite and PP Workplace Safety index for the CPS staff respondents (r = 0.72) was significantly higher than the correlation for the DPS staff respondents (r = 0.50, p < 0.05). Correlations for Work Pressure and Pace CPSPSC composites for the CPS staff respondents were negative and very weak (PP Learning Climate [r = -0.06] and Speaking Up [r = -0.14] indices). The correlation for the Work Pressure and Pace composite and PP Speaking Up index was significantly higher for the DPS respondents (r = 0.25, p < 0.01).

Assessment of Pharmacy Department Patient Safety Culture  with the Use of Validated Work Environment Survey Indices

Assessment of Pharmacy Department Patient Safety Culture  with the Use of Validated Work Environment Survey Indices

Assessment of Pharmacy Department Patient Safety Culture  with the Use of Validated Work Environment Survey Indices

Assessment of Pharmacy Department Patient Safety Culture  with the Use of Validated Work Environment Survey Indices

DISCUSSION

This cross-sectional PSC survey of 429 diverse pharmacy department staff revealed that although all 36 PPS correlations between CPSPSC composite and 1-item overall safety rating and PP indices were positive, no correlations were identified as very strong, and nearly three-quarters of the correlations were moderate, weak, or very weak. Only one CPSPSC composite, Communications Openness, consistently had strong correlations with all three PP indices. Alternatively, the Work Pressure and Pace CPSPSC composite consistently had very weak correlations with all PP indices. For health system leaders who might prefer (for operational efficiency) to measure PSC with items that were not validated specifically to measure PSC, our findings suggest that use of a nonvalidated instrument will not provide accurate assessments of PSC in a pharmacy department.

Because patient safety is directly related to well-designed systems rather than to individual vigilance and effort alone, PSC measurement provides a framework with which to assess a systematic approach to quality and safety in pharmacy services.19 Although it is understandable that health care system representatives would prefer to measure PSC with a survey instrument that provides information about other measures in the workplace environment, this approach will not work if PSC is not measured correctly. Other validated PSC measurement tools exist; the Pharmacy Safety Climate Questionnaire assesses safety climate among pharmacists in Great Britain’s community pharmacies.6 The Safety Attitudes Questionnaire-Pharmacy Version was developed by the Center for Healthcare Quality and Safety at the University of Texas for general pharmacy use.8 Validation study findings for these questionnaires, especially among integrated health care delivery systems, have not been reported in the US; consequently, their ability to accurately measure PSC in US pharmacies is unknown.

Overall, this study found numerically similar PPS for all CPSPSC composites when compared with our previous CPSPSC survey of KPCO Pharmacy Department staff.9 The current study revealed that CPSPSC composites Patient Counseling (91.3%) and Staffing and Work Pressure (41.1%) had the overall highest and lowest PPS, respectively. These findings are similar to those in our previous survey9 and AHRQ’s pilot of the CPSPSC.20 This study identified that CPS respondents had much higher PPS than DPS respondents for CPSPSC composites Communication Openness, Perceptions of Safety, Teamwork, Space and Environment, and Overall Safety, which is similar to our previous survey.9 Conversely, this study found that PPS for the Staffing and Work Pressure composite among CPS respondents degraded dramatically since our earlier survey (7.2% vs 51.9%). CPS respondents had a substantially lower PPS for this composite than DPS respondents, which was not found previously.9 This degradation in CPS’s PPS possibly is related to an increased workload from health care system membership growth coupled with clinical services’ inability to backfill positions because of the system’s financial constraints.

Considering this study’s findings of numerical differences between CPS and DPS respondents on CPSPSC composites’ PPS, it is not surprising that numerical differences were found between CPSPSC composite and PP index PPS correlations. The CPS staff had statistically higher correlations between two PP indices and CPSPSC composites, whereas one correlation was statistically higher for DPS staff. One correlation, Response to Mistakes composite with the Learning Climate index, was very strong for the CPS respondents (r = 0.80) and statistically significantly higher than the DPS respondents’ moderate correlation (r = 0.49, p < 0.05).

Interestingly, some of the most pronounced differences in the subgroup correlation assessments involved PP indices in which PPS were significantly different between CPS and DPS staff. For example, the correlation of the Workplace Safety index with the Patient Counseling composite for DPS staff was substantially higher than for CPS staff, whereas the PPS for the Workplace Safety index and Patient Counseling composite was and was not, respectively, statistically different between the subgroups. DPS staff are dispensing and providing counseling for far more medications and are more likely to experience patient safety issues that may or may not be readily resolved. The higher volume of potential safety events encountered by the DPS may explain the differences identified in the Workplace Safety index PPS between the subgroups. These findings further emphasize that use of generic workplace environment scales that were not developed specifically to measure PSC are of questionable value because they do not appear to reliably measure PSC.

This study’s limitations should be considered while interpreting results. Although it was the intention to have all KPCO Pharmacy Department staff participate in the survey, approximately 50% did not participate. Because the survey was anonymous and voluntary, we could not assess if there were important differences between respondents and nonrespondents (eg, respondents might have been more conscientious or concerned about patient safety). Also, our study likely was underpowered to detect other statistically significant between-subgroup correlation differences because the sample size was lower than intended. Nevertheless, we received responses from 429 staff members, which provided a large sample from 1 Pharmacy Department with which to assess PSC and the correlations between the PPS of CPSPSC composites and PP indices. We used only 1 instrument to measure PSC and assess the correlations of their PPS with the PP indices. Use of other instruments may have resulted in more preferable (perhaps very strong) correlations. We identified no PSC measurement instruments other than the AHRQ’s CPSPSC that is validated in the US. Finally, this survey was conducted in 1 integrated health care delivery system. Results may differ in other systems.

CONCLUSION

This survey of 429 diverse Pharmacy Department staff revealed that PP indices designed to measure various aspects of the work environment only modestly correlate with AHRQ’s CPSPSC composites that measure PSC. Additionally, this study identified differences in the PP index and CPSPSC composite correlations between pharmacy staffing types. Although it is understandable that a health care system would prefer to measure PSC with a survey instrument that provides information about other measures of the workplace environment, these findings suggest that use of nonvalidated work environment indices will not provide accurate assessments of PSC in a pharmacy department.

Disclosure Statement

This study was funded by the Kaiser Permanente Colorado Pharmacy Department. The funder had no role in the study design, collection, analysis and interpretation of data, writing of the report, or the decision to submit the manuscript for publication.

The author(s) have no conflicts of interest to disclose.

Acknowledgments

Brenda Moss Feinberg, ELS, provided editorial assistance.

How to Cite this Article

Rawlings JE, Herner SJ, Delate T, Palmer KE, Swartzendruber KA. Assessment of pharmacy department patient safety culture with the use of validated work environment survey indices. Perm J 2018;22:17-070. DOI: https://doi.org/10.7812/TPP/17-070

References
1.    National Quality Forum. NQF Projects [Internet]. Washington, DC: National Quality Forum; c2017 [cited 2017 Mar 16]. Available from: www.qualityforum.org/projectlisting.aspx.
    2.    Sexton B, Thomas E. Measurement: Assessing a safety culture. In: Leonard MS, Frankel A, Simmonds T, Vega K, editors. Achieving safe and reliable healthcare: Strategies and solutions. Chicago, IL: Health Administration Press; 2004. p 115-6.
    3.    Shekelle PG, Sarkar U, Shojania K, et al. Patient safety in ambulatory settings. Report no. 16(17)-EHC033-EF [Internet]. Rockville, MD: Agency for Healthcare Research and Quality; 2016 Oct [cited 2017 Aug 17]. Available from: https://ahrq-ehc-application.s3.amazonaws.com/media/pdf/ambulatory-safety_technical-brief.pdf.
    4.    Franklin M, Sorra J. Chapter 1. Introduction. Community pharmacy survey user’s guide [Internet]. Rockville, MD: Agency for Healthcare Research and Quality; reviewed 2014 Apr [cited 2017 Mar 16]. Available from: www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/pharmacy/toolkit/pharmsopsuserguide1.html.
    5.    Cummings SR, Strull W, Nevitt MC, Hulley SB. Planning the measurements: Questionnaires. In: Hulley SB, Cummings SR, editors. Designing clinical research. Baltimore, MD: Williams & Wilkins; 1988. p 48-51.
    6.    Ashcroft DM, Parker D. Development of the pharmacy safety climate questionnaire: A principal components analysis. Qual Saf Health Care 2009 Feb;18(1):28-31. DOI: https://doi.org/10.1136/qshc.2006.022129.
    7.    Nordén-Hägg A, Sexton JB, Kälvemark-Sporrong S, Ring L, Kettis-Lindblad Å. Assessing safety culture in pharmacies: The psychometric validation of the Safety Attitudes Questionnaire (SAQ) in a national sample of community pharmacies in Sweden. BMC Clin Pharmacol 2010 Apr 11;10:8. DOI: https://doi.org/10.1186/1472-6904-10-8.
    8.    Center for Healthcare Quality and Safety. Safety attitudes questionnaire (pharmacy version) [Internet]. Austin, TX: The University of Texas at Austin; 2002 [cited 2017 Mar 16]. Available from: https://med.uth.edu/chqs/files/2012/05/Survey-SAQ-Pharmacy.pdf.
    9.    Herner SJ, Rawlings JE, Swartzendruber K, Delate T. Pharmacy survey on patient safety culture: Benchmarking results. J Patient Saf 2017 Mar;13(1):37-42. DOI: https://doi.org/10.1097/PTS.0000000000000102.
    10.    People Pulse. Staff engagement surveys [Internet]. Sydney, NSW, Australia: People Pulse; c2017 [cited 2017 Mar 16]. Available from: www.peoplepulse.com/solutions/staff-engagement-surveys/.
    11.    Konitsney D, Pole L, Welch K, Levin E. Driving a safety culture at Kaiser Permanente. [Internet] Arlington, VA: Towers Watson; 2013 [cited 2017 Apr 28]. Available from: www.towerswatson.com/assets/17748/Linking-Employee-Survey-Results-to-Key-Business.pdf.
    12.    Helling DK, Nelson KM, Ramirez JE, Humphries TL. Kaiser Permanente Colorado Region pharmacy department: Innovative leader in pharmacy practice. J Am Pharm Assoc (2003) 2006 Jan-Feb;46(1):67-76. DOI: https://doi.org/10.1331/154434506775268580.
    13.    Heilmann RM, Campbell SM, Kroner BA, et al. Evolution, current structure, and role of a primary care clinical pharmacy service in an integrated managed care organization. Ann Pharmacother 2013 Jan;47(1):124-31. DOI: https://doi.org/10.1345/aph.1R495.
    14.    Hill RR, Herner SJ, Delate T, Lyman AE Jr. Ambulatory clinical pharmacy specialty services: The Kaiser Permanente Colorado experience. J Manag Care Spec Pharm 2014 Mar;20(3):245-53. DOI: https://doi.org/10.18553/jmcp.2014.20.3.245.
    15.    Witt DM, Tillman DJ. Clinical pharmacy anticoagulation services in a group model health maintenance organization. Pharm Pract Manag Q 1998 Oct;18(3):34-55.
    16.    Sandhoff BG, Kuca S, Rasmussen J, Merenich JA. Collaborative cardiac care service: A multidisciplinary approach to caring for patients with coronary artery disease. Perm J 2008 Summer;12(3):4-11. DOI: https://doi.org/10.7812/TPP/08-007.
    17.    Stubbings T, Miller C, Humphries TL, Nelson KM, Helling DK. Telepharmacy in a health maintenance organization. Am J Health Syst Pharm 2005 Feb 15;62(4):406-10.
    18.    Evans JD. Straightforward statistics for the behavioral sciences. Pacific Grove, CA: Brooks/Cole Publishing Company; 1996.
    19.    Edwards B, Olsen AK, Whalen MD, Gold MJ. Guiding principles of safety as a basis for developing a pharmaceutical safety culture. Curr Drug Saf 2007 May;2(2):135-9. DOI: https://doi.org/10.2174/157488607780598331.
    20.    Agency for Healthcare Research and Quality. Pharmacy survey on patient safety culture: 2012 preliminary comparative results [Internet]. Rockville, MD: Agency for Healthcare Research and Quality; reviewed 2015 Jan [cited 2017 Mar 16]. Available from: www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/pharmacy/2012/index.html.

The Permanente Journal

Sponsored by the eight Permanente Medical Groups, The Permanente Journal advances knowledge in scientific research, clinical medicine, and innovative health care delivery.

Reprint Permissions

The Permanente Journal welcomes requests for reprints and reproduction. Use of any and all material published in The Permanente Journal is copyrighted and protected.

The Permanente Press

The Permanente Press publishes The Permanente Journal and books related to healthcare. Journal subscriptions are entered for the calendar year. Advance payment in US dollars is required.


ISSN 1552-5775 Copyright © 2018 thepermanentejournal.org.

The Permanente Press. All Rights Reserved.