Impact of Longitudinal Electronic Health Record Training for Residents Preparing for Practice in Patient-Centered Medical Homes

Impact of Longitudinal Electronic Health Record Training  for Residents Preparing for Practice in Patient-Centered  Medical Homes


Jung G Kim, MPH, CPH; Hector P Rodriguez, PhD, MPH; Katherine AT Estlin, MD; Carl G Morris, MD, MPH

Perm J 2017;21:16-122 [Full Citation]
E-pub: 07/07/2017


Introduction: Competence in using an electronic health record (EHR) is considered a critical skill for physicians practicing in patient-centered medical homes (PCMHs), but few studies have examined the impact of EHR training for residents preparing to practice in PCMHs. This study explored the educational outcomes associated with comprehensive EHR training for family medicine residents.
Methods: The PCMH EHR training consisted of case-based routine clinic visits delivered to 3 resident cohorts (N = 18). Participants completed an EHR competency self-assessment between 2011 and 2016 (N = 127), examining 6 EHR/PCMH core skills. We compared baseline characteristics for residents by low vs high exposure to EHR training. Multivariate regression estimated whether self-reported competencies improved over time and whether high PCMH EHR training exposure was associated with incremental improvement in self-reported competencies over time.
Results: Residents completed an average of 8.2 sessions: low-exposure residents averaged 5.3 sessions (standard deviation = 1.5), and high-exposure residents averaged 9.0 sessions (standard deviation = 0.9). High-exposed residents had higher posttest scores at training completion (84.4 vs 70.7). Over time, adjusted mean scores (confidence interval) for both groups improved (p < 0.001) from 12.2 (9.6-14.8), with low-exposed residents having greater score improvement (p < 0.001) because of their much lower baseline scores.
Conclusion: Comprehensive training designed to improve EHR competencies among residents practicing in a PCMH resulted in improved assessment scores. Our findings indicate EHR training as part of resident exposure to the PCMH measurably improves self-assessed competencies, even among residents less engaged in EHR training.


When introduced, electronic health records (EHRs) were regarded as key tools to improve the quality of health care delivery. However, the potential for EHRs to improve quality has been clouded because of growing frustration with unintended consequences, including decreased patient encounter duration, limited capacity to support care management, and technology implementation glitches.1-5 Nevertheless, the functionality and use of EHRs are increasingly important for health care practice and policy. Use of EHRs supports practice transformation efforts such as the patient-centered medical home (PCMH) model for chronic disease management and point-of-care decision making, aids in achieving financial incentives from the Centers for Medicare and Medicaid Services through Meaningful Use, and moves practices closer toward the Institute of Medicine’s vision of building learning health systems that adapt to our nation’s evolving health care needs.6-8

However, the content of existing EHR training curricula focuses predominantly on mastering software functionalities alone rather than applying the EHR to optimize the quality of patient care.9,10 Because EHR training content is generally not embedded in medical education curricula, training has primarily been delivered in limited, discreet sessions, and often taught as a standard Human Resources onboarding component rather than a professional competency.11 Moreover, assessing the impact of EHR training for future physicians is still in its infancy. Reports on EHR learners in both undergraduate medical education (UME) and graduate medical education (GME) state the tremendous value of being a competent EHR user, not only for measuring performance for the Accreditation Council for Graduate Medical Education competencies such as systems-based practice and practice-based learning but also to promote lifelong learning as future physicians enter practice.12-14 In recent years, studies on EHR curriculum in core skills for effective patient care and communication, clinical efficiency, and EHR functionality have been reported to be a useful foundation for teaching residents.12-15

Moreover, longitudinal and continuous training programs have been recently popularized in medical education. The cumulative exposure of learning has been shown to improve both learner outcomes and performance in health systems.15-17 Coupled with simulation training that provides immersive training that “replace and amplify real experiences with guided ones,” EHR training using clinical cases that simulate practice transformation, such as the PCMH, provides a platform to develop a resident’s knowledge and skills, with a potential added benefit to improve health systems.18,19

The purpose for this study is to describe the Group Health Family Medicine Residency (GHFMR) longitudinal PCMH case-based EHR curriculum and to assess the outcomes of the training on residents’ EHR self-assessment scores over time. The EHR training was grounded in clinical cases, taught through an accredited Accreditation Council for Graduate Medical Education program, delivered longitudinally during the three years of the family medicine residency program, and included EHR self-assessments delivered every trimester, totaling nine possible assessments per resident. We hypothesized that residents’ EHR training scores would improve in the context of longitudinal EHR training. We also hypothesized that residents with more exposure to EHR training sessions would achieve greater improvements over time compared with residents with less exposure to training.


Setting and Participants

The GHFMR is an 18-resident (6 per year of residency), community-based, urban GME program affiliated with the University of Washington School of Medicine. The GHFMR outpatient practice mirrors attending physicians in the sponsoring institution, Group Health Cooperative (Group Health), an integrated health care delivery system throughout Washington state. Residents are assigned a panel of patients through their training period of 3 years and use EpicCare (Epic Systems Corp, Verona, WI) for their EHR.

The GHFMR’s EHR is the foundational tool for Group Health’s PCMH, a nationally recognized practice transformation designed to achieve the Triple Aim.20-23 The PCMH optimizes health information technology to improve physician point-of-care decision making in the patient encounter, facilitates better chronic care management, enhances visit preparation, delivers expanded clinician access (secure messaging and telephone encounters), and increases capacity for patient outreach.21,24

From 2003 to 2009, GHFMR’s physician EHR training program was taught by an information technology professional without the knowledge or authority to provide trainees with clinical information on how to use the EHR to improve patient care. The focus of the training was on software navigation. In response to persistent negative program feedback from residents and faculty about EHR training insufficiencies coupled with the lack of a documented GME EHR curriculum in the literature, the GHFMR designed a longitudinal EHR training curriculum in conjunction with implementing the Group Health PCMH care model.25

The GHFMR EHR curriculum was implemented in 2010. All residents were scheduled to a three-hour training provided every trimester each year. The goals were derived from patient-centered care attributes to improve the core skills of EHR use with chronic disease management and team-based care, patient communication, software navigation, and clinical efficiency and quality. Appendix 1 (available online at: describes the curriculum in detail.

Data Sources

All residents (N = 18) from 3 entering family medicine resident cohorts (2011, 2012, and 2013) each completed 3 years of longitudinal EHR training, for a total of 127 self-assessments. We anticipated heterogeneity in our incoming trainees’ EHR knowledge, which may include characteristic differences incorporated during UME; thus, we examined resident demographic differences at baseline.

We developed an EHR self-assessment instrument to measure EHR core clinical skills of central focus to PCMH. The online assessment is a 20-minute self-administered, 73-item form completed after each training session.


Electronic Health Record Competencies

Cronbach a tests were performed for our EHR skill categories. We assessed an overall score (a = 0.99) and 6 separate skill categories: In-Basket Management (a = 0.95), Encounters (a = 0.97), Chart Review (a = 0.93), Other Activities (a = 0.93), Tools (a= 0.93), and Clinical Efficiency and Quality  (a= 0.91). Resident core skills under these categories were assessed using a 4-item ordinal scale: 1) not proficient, 2) needs review, 3) proficient, and 4) mastery. We then converted the assessment scale (1-4) for each of the 6 core skill categories and overall EHR score to a standardized scale from 0 to 100. Table 1 summarizes the core skills categories.

Training Exposure

All residents received training but differed in the number of sessions they attended. We compared residents by the number of EHR training sessions attended, a continuous measure, to clarify the extent to which intensity of exposure had an effect. For regression analyses, we used a dichotomous variable that categorized residents on the basis of median exposure level of eight sessions of a total of nine possible sessions. Residents who attended fewer than eight training sessions were categorized as “low exposed.” Residents attending eight or more sessions were categorized as “high exposed.”

Medical School Electronic Health Record Exposure

Residents were categorized as having prior exposure to the EHR in UME vs no exposure. Residents who attended the University of Washington School of Medicine had prior exposure to the Group Health EHR while on their clinical rotations at our residency program. Residents not attending medical school at University of Washington were classified as not exposed to EHR training in UME, because these students did not rotate at GHFMR for their clinical rotations.

Statistical Analyses

We compared age, sex, UME training location (locally trained vs not), and number of EHR training sessions attended. We performed t-tests for age and number of training sessions, and c2 tests for sex and UME. Results for which p was less than 0.05 were considered statistically significant.

We conducted paired t-tests comparing low vs high exposure resident to EHR training sessions to analyze pretest and posttest mean scores from the baseline assessment to final EHR training assessment. Then, we estimated multivariate linear regression models to examine whether overall EHR scores improved over time, as well as for the six core skill categories. These regression models controlled for EHR exposure in UME, but age and sex were not included as control variables because of collinearity concerns with the modest sample size of residents.26 An interaction examining term EHR training exposure and time was also included in the regression models to examine whether residents with high exposure to EHR sessions improved scores more over time relative to low-exposure residents.

Data were analyzed using Stata/IC software, Version 14 (StataCorp LLC, College Station, TX) and Tableau 10.0 (Tableau Software, Seattle, WA). The curriculum, self-assessment tool, and analysis were approved by the Group Health Human Subjects Division.

Impact of Longitudinal Electronic Health Record Training  for Residents Preparing for Practice in Patient-Centered  Medical Homes


Group Health Residents and Electronic Health Record Training Characteristics

Table 2 summarizes resident characteristics from the 3 cohorts of the GHFMR. Most residents were women (13/18; 72.2%), with a mean age of 28.4 (standard deviation [SD] = 0.50) years. Five (33.3%) of 18 residents had previous EHR exposure through local UME training. No significant differences in residents’ characteristics by EHR training exposure were found. Residents completed an average of 8.2 sessions: low-exposure residents averaged 5.3 sessions (SD = 1.5) and high-exposure residents averaged 9.0 sessions (SD = 0.9).

Pre- and Posttest Scores by Level of EHR Training Exposure

Table 3 summarizes residents’ pre- and posttest EHR competency scores at baseline and training completion, by high vs low exposure. For each EHR core skill category, low-exposed residents reported baseline scores ranging from 5.9 to 38.5 and completion scores ranging from 55.6 to 89.3, with improvement for all categories demonstrating statistically significant improvement. The mean overall EHR score improvement for low-exposed residents was 56.0 points (p < 0.001). High-exposed residents reported higher baseline scores from 35.9 to 48.5 and completion scores ranging from 73.9 to 97.7, with all categories statistically significant. The mean overall EHR score improvement for high-exposed residents was 40.9 points (p < 0.001). Although low-exposed residents reported lower baseline scores and greater improvement in scores than the high-exposed subgroup, high-exposed residents reported overall higher posttest scores at training completion.

Figure 1 compares the pre- and posttest mean scores for each category by low- and high-exposed resident at baseline and completion.

Multivariate Analyses

Our multivariate regression analysis estimated the effect of EHR training over time, controlling for UME exposure to EHR training. Table 4 reports the over-time effect, high- vs low-exposure effect, and the incremental effect of high exposure. For every session a resident attended, improvements ranged from 6.7 to 14.4 points across the 6 skill categories. For the overall EHR score, an improvement score of 12.2 for each session was estimated (p < 0.001). High-exposed residents were more likely to achieve greater improvement for each of the core skill categories (12.9 to 39.8) and more likely to have overall EHR score improvement of 28.3 (p < 0.001). The interaction results examining differential improvement over time for residents with high vs low EHR training exposure confirm the pattern observed in unadjusted analyses; residents with reported low exposure had more improvement in competencies over time, but this was largely driven by low baseline scores.

Impact of Longitudinal Electronic Health Record Training  for Residents Preparing for Practice in Patient-Centered  Medical Homes

Impact of Longitudinal Electronic Health Record Training  for Residents Preparing for Practice in Patient-Centered  Medical Homes

Impact of Longitudinal Electronic Health Record Training  for Residents Preparing for Practice in Patient-Centered  Medical Homes

Impact of Longitudinal Electronic Health Record Training  for Residents Preparing for Practice in Patient-Centered  Medical Homes


We described a longitudinal EHR training using PCMH case-based content, analyzed self-assessed EHR scores, and found that training improved self-reported EHR competencies over time. We designed a case-based EHR longitudinal training curriculum that includes a step-by-step breakdown of the PCMH clinic visit through outpatient teaching cases. We also developed a self-assessed EHR training instrument with a convenient standardized scale (0-100) that measures both core skill categories and an overall EHR training score. This could provide utility in developing faculty/trainer-based assessments for EHR training programs that wish to assess over-time EHR skills in a PCMH.

The Group Health PCMH practice transformation aims to facilitate various types of physician access to patients, including both face-to-face and electronic visits.25 Given a recent study that reported residents spent a 1:5 ratio with patient time vs computer time, training residents to master technologies in the clinical setting has key implications for physicians-in-training and for lifelong learning as they continue to use the EHR in practice.27 Our results argue that EHR training that simulates the PCMH through a case-based curriculum should be incorporated into physician EHR training. The lack of clinically relevant training content in EHR training from our earlier experiences speaks to this experience, and relates to policy efforts to facilitate EHR meaningful use in practice.28-30

Our findings also support the growing medical education literature that longitudinal or cumulative exposure to learning experiences influences rates of learning.15-17,31 Our analysis for differing learning rates based on high vs low exposure to EHR training helps estimate the value of longitudinal training for residents, even when residents are unable to fully engage in all sessions. Low-exposed residents who reported the lowest scores at baseline may be less engaged at the start of training or uncertain about their EHR core skills but subsequently found EHR training more helpful as they attended more sessions.32 Our results indicate that our residents report improvements with their EHR skills, thus speaking to the potential value of EHR training in the core skills related to a PCMH.

This study has important limitations. First, generalizability to other training settings may be limited because of our small sample size in a single GME program, our training program designed around the PCMH model, and self-assessed competencies that are subject to reporting or social desirability bias. Second, we had no control group because the EHR curriculum was a foundational aspect of orientation to the PCMH and it would be impractical to randomize trainees in experimental groups. Despite the limitations of not having a control group, our analyses highlight that residents report improved competencies from EHR training and that improvement is possible for residents with less-than-optimal exposure to the sessions. Our next steps are to explore these differences influenced by EHR training exposure and to study the link between self-assessed resident competencies to faculty-assigned competency achievements and clinical outcome measures related to the PCMH.

Importantly, we found evidence that early physician engagement in longitudinal EHR training can substantially improve EHR self-reported competency scores among family medicine residents. The results highlight the need to incorporate medical education when implementing new technologies and for health systems undergoing practice changes such as the PCMH.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.


The authors thank Robert Winter and Chris Thayer, MD, for their efforts with the development and implementation of the curriculum.

This work was presented at the American Academy of Family Physicians Workshop for Directors of Family Medicine Residencies (PDW), June 3 to 5, 2012, and April 5 to 7, 2013, Kansas City, MO; and at the Society of Teachers of Family Medicine 46th Annual Spring Conference, May 1 to 5, 2013, Baltimore, MD.

Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.

Ethical Approval

The Group Health Human Subjects Division granted an exemption for the use of human subjects for this study on September 3, 2013.

How to Cite this Article

Kim JG, Rodriguez HP, Estlin KAT, Morris CG. Impact of longitudinal electronic health record training for residents preparing for practice in patient-centered medical homes. Perm J 2017;21:16-122. DOI:

1.    Alkureishi MA, Lee WW, Lyons M, et al. Impact of electronic medical record use on the patient-doctor relationship and communication: A systematic review. J Gen Intern Med 2016 May;31(5):548-60. DOI:
    2.    Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res 2010 Aug 6;10:231. DOI:
    3.    Hsu J, Huang J, Fung V, Robertson N, Jimison H, Frankel R. Health information technology and physician-patient interactions: Impact of computers on communication during outpatient primary care visits. J Am Med Inform Assoc 2005 Jul-Aug;12(4):474-80. DOI:
    4.    Nagy VT, Kanter MH. Implementing the electronic medical record in the exam room: The effect on physician-patient communication and patient satisfaction. Perm J 2007 Spring;11(2):21-4. DOI:
    5.    Rizer MK, Kaufman B, Sieck CJ, Hefner JL, McAlearney AS. Top 10 lessons learned from electronic medical record implementation in a large academic medical center. Perspect Health Inf Manag 2015 Jul 1;12:1g.
    6.    Kern LM, Edwards A, Kaushal R. The patient-centered medical home, electronic health records, and quality of care. Ann Intern Med 2014 Jun 3;160(11):
741-9. DOI:
    7. Electronic health records (EHR) incentive programs [Internet]. Baltimore, MD: Centers for Medicare & Medicaid Services; updated 2017 Feb 8 [cited 2016 Dec 30]. Available from: = /EHRIncentivePrograms/30_Meaningful_Use.asp.
    8.    Institute of Medicine of the National Academies. Best care at lower cost: The path to continuously learning health care in America. Washington, DC: The National Academies Press; 2013.
    9.    Duke P, Frankel RM, Reis S. How to integrate the electronic health record and patient-centered communication into the medical visit: A skills-based approach. Teach Learn Med 2013;25(4):358-65. DOI:
    10.    Krist AH. Electronic health record innovations for healthier patients and happier doctors. J Am Board Fam Med 2015 May-Jun;28(3):299-302. DOI:
    11.    Pantaleoni JL, Stevens LA, Mailes ES, Goad BA, Longhurst CA. Successful physician training program for large scale EMR implementation. Appl Clin Inform 2015 Feb 11;6(1):80-95. DOI:
    12.    Nuovo J, Hutchinson D, Balsbaugh T, Keenan C. Establishing electronic health record competency testing for first-year residents. J Grad Med Educ 2013 Dec;5(4):658-61. DOI:
    13.    Tierney MJ, Pageler NM, Kahana M, Pantaleoni JL, Longhurst CA. Medical education in the electronic medical record (EMR) era: Benefits, challenges, and future directions. Acad Med 2013 Jun;88(6):748-52. DOI:
    14.    Hersh WR, Gorman PN, Biagioli FE, Mohan V, Gold JA, Mejicano GC. Beyond information retrieval and electronic health record use: Competencies in clinical informatics for medical education. Adv Med Educ Pract 2014 Jul 1;5:205-12. DOI:
    15.    Bowen JL, Hirsh D, Aagaard E, et al. Advancing educational continuity in primary care residencies: An opportunity for patient-centered medical homes. Acad Med 2015 May;90(5):587-93. DOI:
    16.    Hirsh DA, Holmboe ES, ten Cate O. Time to trust: Longitudinal integrated clerkships and entrustable professional activities. Acad Med 2014 Feb;89(2):201-4. DOI:
    17.    Pisano GP, Boomer RM, Edmonson AC. Organizational differences in rates of learning: Evidence from the adoption of minimally invasive cardiac surgery. Manage Sci 2001 Jun;47(6):752-68. DOI:
    18.    Lateef F. Simulation-based learning: Just like the real thing. J Emerg Trauma Shock 2010 Oct;3(4):348-52. DOI:
    19.    Schmidt E, Goldhaber-Fiebert SN, Ho LA, McDonald KM. Simulation exercises as a patient safety strategy: A systematic review. Ann Intern Med 2013 Mar 5;158(5 Pt 2):426-32. DOI: Erratum in: Ann Intern Med 2013 Jul 16;159(2):160. DOI:
    20.    Reid RJ, Coleman K, Johnson EA, et al. The Group Health medical home at year two: Cost savings, higher patient satisfaction, and less burnout for providers. Health Aff (Millwood) 2010 MAY;29(5):835-43. DOI:
    21.    Hsu C, Coleman K, Ross TR, et al. Spreading a patient-centered medical home redesign: A case study. J Ambul Care Manage 2012 Apr-Jun;35(2):99-108. DOI:
    22.    McCarthy D, Mueller K, Tillmann I. Group Health Cooperative: Reinventing primary care by connecting patients with a medical home. New York, NY: The Commonwealth Fund; 2009 Jul 2.
    23.    Charles D, Gabriel M, Furukawa MF. Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008-2013. ONC Data Brief, no. 16 [Internet]. Washington, DC: The Office of the National Coordinator for Health Information Technology; 2014 May [cited 2017 Feb 17]. Available from:
    24.    Kim JG, Morris CG, Heidrich FE. A tool to assess family medicine residents’ patient encounters using secure messaging. J Grad Med Educ 2015 Dec;7(4):649-53. DOI:
    25.    Kim JG, Morris CG, Ford P. Teaching today in the practice setting of the future: Implementing innovations in graduate medical education. Acad Med 2017 May;92(5):662-5. DOI:
    26.    Hardin JW. Variance inflation factors and variance-decomposition proportions. Stata Technical Bulletin 1995;4(24):17-22.
    27.    Mamykina L, Vawdrey DK, Hripcsak G. How do residents spend their shift time? A time and motion study with a particular focus on the use of computers. Acad Med 2016 Jun;91(6):827-32. DOI:
    28.    Goveia J, Van Stiphout F, Cheung Z, et al. Educational interventions to improve the meaningful use of electronic health records: A review of the literature: BEME guide no. 29. Med Teach 2013 Nov;35(11):e1551-60. DOI:
    29.    Rouf E, Whittle J, Lu N, Schwartz MD. Computers in the exam room: Differences in physician-patient interaction may be due to physician experience. J Gen Intern Med 2007 Jan;22(1):43-8. DOI:
    30.    Schenarts PJ, Schenarts KD. Educational impact of the electronic medical record. J Surg Educ 2012 Jan-Feb;69(1):105-12. DOI:
    31.    Pusic MV, Boutis K, Hatala R, Cook DA. Learning curves in health professions education. Acad Med 2015 Aug;90(8):1034-42. DOI:
    32.    Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: A conceptual model. Acad Med 2010 Jul;85(7):1212-20. DOI:



The Permanente Journal is celebrating it's 20th anniversary year. We look forward to continuing to bring you more high-quality content during the next 20 years.


Sponsored by the National Permanente Medical Groups, The Permanente Press publishes The Permanente Journal and books related to Kaiser Permanente and health care.


25,000 print readers per quarter, 7,628 eTOC readers, and in 2016, 1.4 million page views on TPJ articles in PubMed from a broad international readership.


Articles, editorials, letters to the editor, and other material represent the opinion of the authors. Send your comments to

Copyright 2017 The Permanente Journal - Kaiser Permanente. All Rights Reserved.