Utilization of Secure Messaging to Primary Care Departments

Jose Yakushi, MD¹; Mose Wintner, PhD²; Naomi Yau²; Lina Borgo, MPH²; Edwin Solorzano, MD¹

Perm J 2020;24:19.177 [Full Citation]

https://doi.org/10.7812/TPP/19.177

E-pub: 04/29/2020

ABSTRACT

Introduction: Secure messaging is a platform for email communication between patients and their physicians. Although patient-generated emails are associated with increased use of clinical services, greater member retention, and improved quality of care, secure messaging has a marked impact on primary care physicians’ workload.

Objective: To understand how the email topic and volume vary by demographics and clinical factors among members of a managed care organization.

Methods: We analyzed all secure messages sent to primary care departments by adult members of Kaiser Permanente Southern California (KPSC) in 2017.

Results: Members with a higher volume of office visits and telephone appointment visits generated a higher volume of emails to primary care physician. Members with a Centers of Medicare and Medicaid Services Hierarchical Condition Category diagnosis history sent 3 times as many emails as those without such a diagnosis history. Women accounted for nearly two-thirds of emails despite making up only half of the KPSC member population. Less than one-fourth (21.4%) of members sent 2.3 million total emails to their physician. Medical advice was the most common reason for sending secure messages (24.7%) in a sample studied (n = 2397).

Discussion: These findings confirm the need for additional research to more accurately quantify the additional burden from secure message utilization on primary care physicians. Knowing the factors associated with secure messaging usage and message content could assist in building more efficient staffing models and creating more efficient routing that matches the message content with a physician’s scope of practice.

INTRODUCTION

Secure messaging is a protected electronic communication service between patients and physicians. Known for its ability to foster physician-patient relationships, secure email has become a ubiquitous communication tool in health care settings.¹ Studies have shown that patient satisfaction has generally increased since adopting secure messaging in family medicine, with satisfaction scores higher on questions related to “ease of communicating with the physician” and “understanding of diagnosis/treatment plan”.² Some organizations initially adopted secure messaging as a way to address low-acuity medical issues, assuming this might reduce unnecessary telephone or in-person office visits.^3,4 Others have targeted young and healthy patients in an early attempt to increase use of secure messaging, assuming the millennial population is more likely to use this virtual service than the older population.^2,5 The rapid adoption of secure messaging has furthered the focus on and improved the quality of patient-centered care^6,7, a pillar of Kaiser Permanente’s (KP) care delivery system.⁸

KP’s secure messaging system was implemented in 2005, first beginning with the KP Hawaii Region. Secure messaging is facilitated through KP’s member portal (https://kp.org), which is integrated with KP’s electronic health record, HealthConnect. HealthConnect represents KP’s implementation of the Epic suite of software (Epic Systems Corp, Verona, WI). To access the member portal, patients are required to register for a kp.org account and be authenticated users.⁸ In the first quarter of 2017, more than 67% KP Southern California (KPSC) members were registered on the member portal, an 8.3% growth from the prior year. In the fourth quarter of that same year, 23.1% of all portal-registered members in KPSC sent 1 or more secure email messages to their physicians; an average of 1272 emails were sent per 1000 registered members in a year. When the 3.6% membership increase from the previous year is factored in, 2017 saw an 11% increase in secure messages from 2016, with a total of 5,807,833 secure messages sent.⁸ Secure messaging is now a common form of physician-patient interaction through which care is provided and common concerns are addressed by the care team.

This study analyzed all adult-generated secure messages in the KPSC Region. The purpose of this study was to better understand the usage of secure messaging between KPSC members and primary care departments. This was accomplished by 1) comparing the use of secure messaging against the utilization of office visits and telephone appointment visits (TAVs) in primary care, 2) examining the demographic characteristics of patients who use KP’s secure messaging platform, and 3) investigating the content of secure messages that KP members sent to their primary care physicians (PCPs).

METHODS

Study Design and Population

After obtaining KP institutional review board approval, we conducted a retrospective study analyzing patient demographic data (eg, age, sex), clinical factors (eg, Health Plan type, time between a sent message and nearest completed office visit), and primary care utilization data (eg, encounter volume).

All data examined in this study pertain to the utilization of primary care services in Southern California, by adult KPSC members from January 2017 to December 2017. Primary care included both family medicine and internal medicine specialties. Data from the Pediatrics and Obstetrics/Gynecology Departments were not included in this study.

At KP, members seeking care for primary care services have several options. For the purposes of this study, we focused on 3 main primary care services: Office visits, TAVs, and secure messaging. To quantify the utilization of these modalities, we counted instances of the following primary care services for each individual who held a KPSC membership in 2017 (4.7 million KP members):

1. Office visits in 2017 (n = 4,898,596) to primary care departments by adult KPSC members

2. Telephone appointment visits (n = 1,185,590) with primary care departments by adult KPSC members in 2017

3. Secure messages (n = 2,316,309) from adult KPSC members to primary care departments in 2017 that were not replies to a previous message (ie, initiations of message threads by members, not by physicians).

Furthermore, we performed random sampling of secure messages (n = 2397 of total 2,316,309) for email topic analysis.

In this study, patients with a chronic condition were defined as having a Centers for Medicare and Medicaid Services (CMS) Hierarchical Condition Category (HCC) diagnosis history (used in the CMS risk-adjustment model¹⁰) listed in their medical record. The CMS HCC was originally designed to estimate future health care costs for patients and is now often used to communicate patient complexity.

Of the 2.3 million email messages sent in 2017, we analyzed the content and corresponding metadata of 2397 messages. Our researchers tagged each email message with up to 3 primary message topic tags, using a detailed description of each message topic. The detailed description and table were modified from the Veterans Affairs study by Shimada et al.¹¹ Table 1 depicts the various email topics.

Statistical Analysis

We performed a negative binomial regression to assess the number of messages sent by a patient. Independent variables included primary care utilization counts; geographic area of the patient’s medical center; and patient’s age, sex, number of chronic illnesses, race/ethnicity, and Health Plan product type. The reference patient we compared all variables to was white, male, age 52 years (median age in our dataset), had no chronic illnesses, and had a KP Health Plan in his home Region. This regression accounts for each variable independent of one another. For example, although being a Medicare member and age may be related, this regression represents them as 2 separate variables, independent of one another. Transformed versions of office visit counts and TAV counts were used in the regression, namely log(variable + 1).

Table 1. Primary topic categories
Category	Description	Example
Medical advice	Physical or mental health problem or concern not coded as a medication issue or test issue	“My right shoulder has been extremely sore for more than a month. I can’t lift my right arm up and to the side without great pain. This has been over a month and is getting worse.”
New or change in medication request	Request for a new medication or a change of current medication	“My current medication isn’t working for me anymore but my sister-in-law recommended [medication name]. Can you prescribe me that?”
Medication renewal or refill	Request for refill or renewal of a medication or medical supply	“Please renew my prescription to oxycodone HCL 5 mg tab. Please have pharmacy mail [it] to me.”
Medication issue/question	Medication or supply issue that is not a refill/renew request or a problem with receiving the medication in mail (eg, questions about dose or side effects)	“I’m sending a list of vitamins and supplements I take. Please have the doctor make sure they are okay to take and don’t counteract what I take for my Parkinson’s [disease].”
Imaging request	Request for imaging/test	“I’ve been having terrible headaches and I want a CT scan to see what the problem is.”
Imaging issue/question	Patient questions about imaging, choosing among imaging options, why a particular imaging procedure is important, preparing for a scan/imaging procedure	“Am I allowed to eat before my MRI tomorrow?”
Imaging result	Request for or discussion of imaging results	“Hello, I got the ultrasound [scan result] last week and I want to know what are the next steps to take?”
Lab/test request	Request for lab test	“I need some bloodwork done.”
Lab/test issue/question	Patient questions about testing, choosing among testing options, why a test is important, preparing for a test	“Did you want me to get lab work done before I see you?”
Lab/test results	Request for or discussion of test results	“Thanks for sending me the test results. I still have a question for the doctor about the numbers that you gave me.”
Surgery or procedure issue/question	Patient questions about a surgery or procedure	“Good morning. I am still having a lot a lot of pain in my knees. The injections did not seem to help. But I heard the second time around it usually helps a lot more. So, I was wondering if I can get another injection in my knees??”
FYI informing	Patient sharing information with no additional discussion that does not fit other category (eg, inform about non-VA care or test result)	“Just to keep you posted. EKG done yesterday. Bloodwork and urine done this AM.”
Scheduling	Schedule an appointment, test, or procedure	“I would like to set up an appointment for a physical.”
Referral	Request referral to a specialist	“I would like to be referred to MOVE! to help me with some weight loss. I also need you to set up an appointment for [examination of] my knees and lower back. I’m having a great deal of pain in both, been injured in the past. Thank you.”
Note request	Request doctor’s note	“I’m supposed to go on vacation this week but I’m too sick to fly. Can you write me a note for my travel insurance?”
Administrative	Administrative issues other than appointment scheduling	“I didn’t receive my medication in the mail.”
Life issue	Any contextual issues that are not strictly biomedical and are about the patient’s life context	“We’re having lots of changes with my mom. Hospice people are coming here but now they’re talking about putting her in a nursing home. They say they can help her all the time.”
Callback request	Simple callback request	“Please call me at xxx-xxx-xxxx when you get the chance.”
Thanks	Expressing appreciation	“I do want to take a minute to thank you for your help. It is a different world going through a pain-free day. You have been kind and caring and I do really appreciate it!!!”
Complaint	Complaining about care, services, KP, etc	“First and foremost … and I would think you would be aware of this by now, [patient first name] is NOT my Dad … he’s my husband.”
Other	Content not captured by the above categories	“Both of my glucose] meters are reading E-9 when trying to get a reading. I think it’s the battery, but I don’t know how to fix it. Tell me how or what to do.”
AM = morning; CT = computed tomography; EKG = electrocardiogram (ECG); FYI = for your information; HCL = hydrochloride; KP = Kaiser Permanente; lab = laboratory; MOVE! = Veterans Affair’s National Weight Management Program; MRI = magnetic resonance imaging; tab = tablet; VA = Veterans Affairs.

RESULTS

Utilization of Primary Care Services

Main Drivers for Secure Message Utilization

As shown in Figure 1, the number of completed primary care office visits by a member was most strongly associated with a member sending a greater number of secure messages to the primary care department. The number of TAVs was the second strongest factor. Other factors associated with a member sending more secure messages included being a Medicare member, being a woman, and having a prior CMS HCC diagnosis. Because this regression looks at each variable independently, this study’s findings show that having an office visit was more indicative of sending a secure message than being a Medicare member or having a chronic disease. Conversely, being a KP-Medicaid member and self-identifying as black and/or Hispanic were factors associated with fewer secure messages sent.

Average counts of office visit utilizations were similar between racial groups, but propensities for secure messaging varied widely (Table 2). Though there was only a small difference between average numbers of office visits between black, Hispanic, and white members, members who self-identified as black and/or Hispanic sent fewer messages than other races with similar average number of office visits.

As seen in Table 3, members who had at least 1 office visit had, on average, more than 2 office visit in 2017. Members who had at least 1 TAV had, on average, 1.63 TAVs. Members who initiated a secure message thread initiated nearly 3 messages on average. Table 3 also shows the average counts for members who did not have any office visits, TAVs, or secure messages sent.

Correlation between Secure Messaging and Office Visits/TAVs

Figure 2 illustrates how age and sex were associated with utilization of different primary care services. The spike in office visit utilization correlates with a spike in utilization of TAVs and secure messages. The observed spike is likely related to the free Medicare annual wellness office visit that members are encouraged by KP to schedule once they become a Medicare member. Additionally, KP PCPs often use TAVs to follow-up with Medicare members after a Medicare wellness visit and often encourage members to use secure messaging to follow-up after their Medicare wellness visit.

In Figure 3, the largest volume of messages is highlighted, detailing that 7.69% of messages sent by adult members to primary care departments were sent within 1 day of a PCP office visit. Another 14.85% of messages were sent within 3 days of a PCP office visit, and 37.5% of messages were sent within 7 days of a PCP office visit.

Utilization by Chronic Condition Diagnosis

Of KPSC members, 42.5% had a chronic condition, yet this population accounted for two-thirds of primary care service utilization. Those KP members with a chronic condition used TAVs and secure messaging more than in-person office visits. In Table 4, patients with a chronic condition diagnosis history sent more than 3 times as many secure messages to their PCP and had 2 times as many office visits compared with patients without any chronic condition diagnosis.

Utilization by Sex

Approximately half of KPSC members (51.7%) in 2017 were women (Table 4), but female members accounted for almost two-thirds of primary care utilization. On average, 5 of 8 messages sent to a PCP were sent by women. In a comparison of the 3 different primary care services, the biggest difference in utilization between sexes was seen in TAVs, with female members accounting for 65% of TAV encounters.

Utilization of All Service Modalities

Less than one-fourth of the member population was responsible for the sum of 2.3 million secure messages to PCPs in 2017 (Table 5).

According to Figure 4, younger men used primary care services the least. For example, less than 5% of 18-year-old men used secure messaging. Overall, women across all ages used primary care office visits and TAVs more than men did. However, in secure messaging, after age 70 years, men consistently sent more messages to their primary care department than did women.

Message Content

Nearly 1 in 4 secure messages to the primary care office fell into the medical advice category. The top categories included medication issues and questions, informing the physician (ie, messages not requiring an action or reply), medication refills, and scheduling (Figure 5). Nearly 25% of all messages examined were about medications.

Table 2. Average encounter volume of utilization per patient by self-reported race/ethnicity
Race/ethnicity	No. of members	Office visits	Telephone appointment visits	Secure messages
White	1,779,631	0.984	0.265	0.748
Native American/Inuit/Aleutian	19,334	0.957	0.271	0.562
Asian/Pacific Islander	521,774	0.942	0.215	0.522
Black/African American	412,163	0.958	0.291	0.434
Other	75,775	0.795	0.196	0.417
Hispanic/Latino	1,525,834	0.906	0.223	0.289
Declined to state	489,833	0.392	0.094	0.189

 

Table 3. Average utilization counts per patient
Parameter	Office visits	Telephone appointment visits (TAVs)	Secure messages
Members with at least 1 office visita	2.15	0.47	0.93
Members with at least 1 TAVa	2.46	1.63	1.46
Members who sent at least 1 secure messagea	2.17	0.61	2.92
Members with 0 office visitsb	—	0.08	0.14
Members with 0 TAVsb	1.04	—	0.42
Members who sent 0 secure messagesb	1.09	0.24	—
a Average counts of utilizations per patient compared against having at least 1 office visit, having 1 telephone appointment visit (TAV), or sending 1 secure message, respectively. b Average counts of utilizations per patient compared against having 0 office visits, 0 TAVs or sending 0 secure messages, respectively.

 

DISCUSSION

This study demonstrates the interdependency of patient demographics and clinical factors on secure message utilization. Studies have shown that secure messages and TAVs were implemented to create more efficient primary care services.^3,4,12 Our study findings reveal that members with a higher utilization of office visits and TAVs sent more secure messages than those who did not use those clinical services. Through our variable-independent regression analysis, we found that having an office visit and having a TAV are more indicative of sending secure messages than being a Medicare member or having a chronic condition, by a factor of 6 and 13, respectively, although being a Medicare member and having a chronic condition are still significant in attributing to larger secure message volumes. At KPSC, our PCPs often prompt patients at the end of their office visits to email them if they have any further questions. Whereas our study results suggest an association between the volume of secure messaging and office visits, other studies have shown that the relationship between secure messaging, office visits, and TAVs are complex. Some studies have shown that messaging can replace the number of office visits or TAVs; other studies have suggested that secure messaging either increases or does not alter the utilization of office visits or telephone calls.^4,12-20 Further studies must be done to confirm these correlations.

Studies have found that patients with certain demographic backgrounds used care services overall more than others did.^21,22 Our study findings confirm previous research by demonstrating that certain patient variables, such as being a woman, being white, having a chronic condition, and being older, contribute to higher secure messaging utilization. Moreover, results of this study reveal the magnitude at which certain clinical and patient demographic factors affect secure message utilization. A member having a CMS HCC diagnosis history may send 3 times as many emails to primary care than a member without an HCC diagnosis history. Women also account for almost two-thirds of secure messaging to primary care departments excluding Obstetrics/Gynecology. Because of this exclusion, our results likely underestimate the message gap between sexes. In a comparison of the 3 different primary care services, the biggest difference in utilization between sexes was seen in virtual services, with female members using 65% of TAV encounters and 62.4% of secure messaging encounters.

Many studies have researched the heavy adoption of secure messaging in outpatient medicine and obstetrics/gynecology.^4,12-22 Data shows that the adoption of the secure messaging platform is not yet widespread. Nearly 25% of the KPSC member population used the secure messaging platform in 2017. This group sent approximately 2.3 million messages in 2017. This finding can also be interpreted as more than 75% of KPSC members have yet to adopt secure messaging. Past KP trends data suggest, however, that the use of secure messaging increases each year.^8,9 The primary care team needs to be well positioned to account for the increased general adoption of secure messaging.

Our study adds to existing research by looking at message topics to primary care departments. Consistent with other studies, medical advice is ranked as one of the top message topics that members are inquiring about.^22,24,25 The top 6 message topics following medical advice are very similar to the message topic study conducted at Veterans Affairs medical centers,¹¹ with a slight difference in topic ranks. The variation could be caused by the small sample size and difference in the service population. Knowing what types of messages patients are sending will help when building, prioritizing, or updating routing protocols.

Practical Applications

Today, drivers of secure message utilization are not commonly considered in certain aspects of the secure message management system in primary care settings. Knowing the factors associated with secure messaging utilization and the message content could assist in the following: 1) building more efficient staffing models; 2) creating more efficient routing that matches the secure message content with maximum scope of practice allowed for nurses, pharmacists, and advanced practice physicians; and 3) using message volume as a weighted factor that influences a PCP’s panel ceiling, or cap.

Staffing Models

Our analysis suggests that certain populations tend to send a larger number of secure messages and therefore produce additional virtual work for physicians. The factors in these populations should be considered by health systems when developing staffing and resource allocation protocols to address this imbalance. For example, if a physician is projected to have a heavier secure message workload based on his/her panel, nurses experienced with handling secure messaging can be assigned to assist.

Routing Secure Messages

Understanding what patients are messaging about can aid in designing a more efficient secure messaging model to route and prioritize messages, stratifying them for the appropriate recipient. For example, if messages about medications have a high volume, there may be value in routing messages about medication to pharmacists directly, bypassing the primary care team.

Optimized Panel Calculation

To account for workload owing to messaging volumes, our research findings suggest that some patient variables should be considered when health systems calculate PCP panel sizes. PCPs whose panels contain white, Medicare-eligible women with a chronic condition likely have a heavier workload because of the increased volume of secure messages they receive from these patient populations.

Limitations

This study has several limitations. First, our study included secure messages through https://kp.org only to the Family Medicine and Internal Medicine Departments, excluding the Pediatrics and Obstetrics/Gynecology Departments. Our analysis did not include any messages to specialty departments. Second, we examined data from only a single KP Region, KPSC. Our study does not include messaging, office visit, and TAV data trends from other KP Regions.

Studies have found that the relationship between secure messaging, office visits, and TAVs is complex. Some study findings have shown that messaging can replace the number of office visits or TAVs; other studies have suggested that secure messaging either increases or does not alter the utilization of office visits or telephone calls.^4,12-20 Therefore, our study, being an observational study, cannot claim whether messaging drives other forms of care or vice versa. To make such a claim about causality, we would need a randomized controlled trial. The results in this article speak only to associations.

Last, it is important to note that the total number of secure messages sampled in this study reflects only the 21.4% of KPSC members who used secure messaging and does not reflect the population as a whole. As more and more members adopt the service, further studies are needed to determine if the observations found in this study stay consistent.

CONCLUSION

This study examined several factors that were often associated with utilization of secure messaging, ranging from having an office visit to having a chronic condition. Factors most strongly associated with increased utilization of secure messaging were the numbers of primary care office visits and TAVs. Demographic factors associated with secure message utilization included sex, age, HCC diagnosis history, and Health Plan type. Being white, female, and older and having a Medicare Health Plan type and a previous HCC diagnosis were associated with higher utilization of secure messages. The findings of this study reveal the magnitude of these factors on secure message utilization and challenge many existing assumptions around secure messaging.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.

Acknowledgments

The authors thank the physicians and operational directors in the Kaiser Permanente Southern California Region, the Health Innovation team under the Southern California Permanente Medical Group and Peter Jung from Kaiser Permanente Business Systems and Reporting for their assistance.

Kathleen Louden, ELS, of Louden Health Communications performed a primary copy edit.

Author Affiliations

¹ Department of Family Medicine, Downey Medical Center, CA
² Department of Health Innovation, Southern California Permanente Medical Group, Tustin, CA

Corresponding Author

Jose Yakushi, MD (jose.a.yakushi@kp.org)

How to Cite this Article

Yakushi J, Wintner M, Yau N, Borgo L, Solorzano E. Utilization of secure messaging to primary care departments. Perm J 2020;24:19.177. DOI: https://doi.org/10.7812/TPP/19.177

References

1.    Garrido T, Meng D, Wang JJ, Palen TE, Kanter MH. Secure e-mailing between physicians and patients: Transformational change in ambulatory care. J Ambul Care Manage 2014 Jul-Sep;37(3):211-8. DOI: https://doi.org/10.1097/JAC.0000000000000043 PMID:24887522
    2.    Franklin R. Secure messaging: Myths, facts, and pitfalls. Fam Pract Manag 2013 Jan-Feb;20(1):21-4. PMID:23418834
    3.    Kane B, Sands DZ. Guidelines for the clinical use of electronic mail with patients. The AMIA Internet Working Group, Task Force on Guidelines for the Use of Clinic-Patient Electronic Mail. J Am Med Inform Assoc 1998 Jan-Feb;5(1):104-11. DOI: https://doi.org/10.1136/jamia.1998.0050104 PMID:9452989
    4.    Chen C, Garrido T, Chock D, Okawa G, Liang L. The Kaiser Permanente Electronic Health Record: Transforming and streamlining modalities of care. Health Aff (Millwood) 2009 Mar-Apr;28(2):323-33. DOI: https://doi.org/10.1377/hlthaff.28.2.323 PMID:19275987
    5.    Ralston JD, Rutter CM, Carrell D, Hecht J, Rubanowice D, Simon GE. Patient use of secure electronic messaging within a shared medical record: A cross-sectional study. J Gen Intern Med 2009 Mar;24(3):349-55. DOI: https://doi.org/10.1007/s11606-008-0899-z PMID:19137379
    6.    Ralston JD, Martin DP, Anderson ML, et al. Group health cooperative’s transformation toward patient-centered access. Med Care Res Rev 2009 Dec;66(6):703-24. DOI: https://doi.org/10.1177/1077558709338486 PMID:19549993
    7.    Anand SG, Feldman MJ, Geller DS, Bisbee A, Bauchner H. A content analysis of e-mail communication between primary care providers and parents. Pediatrics 2005 May;115(5):1283-8. DOI: https://doi.org/10.1542/peds.2004-1297 PMID:15867036
    8.    Zhou YY, Kanter MH, Wang JJ, Garrido T. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health Aff (Millwood) 2010 Jul;29(7):1370-5. DOI: https://doi.org/10.1377/hlthaff.2010.0048 PMID:20606190
    9.    Report Kaiser Permanente Digital Metrics. Southern California Region. 2017-2018 annual report. Portland, OR: Kaiser Permanente Digital Experience Center Analytics; 2017
    10.    2017 Model software/ICD-10 mappings [Internet]. Atlanta, GA: Centers for Medicare & Medicaid Services; 2017 [cited 2019 Sep 29]. Available from: www.cms.gov/Medicare/Health-Plans/MedicareAdvtgSpecRateStats/Risk-Adjustors-Items/Risk2017.html?DLPage=1&DLEntries=10&DLSort=0&DLSortDir=descending
    11.    Shimada SL, Petrakis BA, Rothendler JA, et al. An analysis of patient-provider secure messaging at two Veterans Health Administration medical centers: Message content and resolution through secure messaging. J Am Med Inform Assoc 2017 Sep 1;24(5):942-9. DOI: https://doi.org/10.1093/jamia/ocx021 PMID:28371896
    12.    Liederman EM, Lee JC, Baquero VH, Seites PG. Patient-physician web messaging. The impact on message volume and satisfaction. J Gen Intern Med 2005 Jan;20(1):52-7. DOI: https://doi.org/10.1111/j.1525-1497.2005.40009.x PMID:15693928
    13.    de Lusignan S, Mold F, Sheikh A, et al. Patients’ online access to their electronic health records and linked online services: A systematic interpretative review. BMJ Open 2014 Sep 8;4(9):e006021. DOI: https://doi.org/10.1136/bmjopen-2014-006021 PMID:25200561
    14.    Goldzweig CL, Towfigh AA, Paige NM, et al. Systematic review: Secure messaging between providers and patients, and patients’ access to their own medical record: Evidence on health outcomes, satisfaction, efficiency and attitudes [Internet]. Washington, DC: US Department of Veterans Affairs; 2012 Jul [cited 2019 Oct 23]. Available from: www.ncbi.nlm.nih.gov/books/NBK100359/
    15.    Roter DL, Larson S, Sands DZ, Ford DE, Houston T. Can e-mail messages between patients and physicians be patient-centered? Health Commun 2008;23(1):80-6. DOI: https://doi.org/10.1080/10410230701807295 PMID:18443995
    16.    Zhou YY, Garrido T, Chin HL, Wiesenthal AM, Liang LL. Patient access to an electronic health record with secure messaging: Impact on primary care utilization. Am J Manag Care 2007 Jul;13(7):418-24. PMID:17620037
    17.    Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. JAMA 2012 Nov 21;308(19):2012-9. DOI: https://doi.org/10.1001/jama.2012.14126 PMID:23168824
    18.    Katz SJ, Moyer CA, Cox DT, Stern DT. Effect of a triage-based E-mail system on clinic resource use and patient and physician satisfaction in primary care: A randomized controlled trial. J Gen Intern Med 2003 Sep;18(9):736-44. DOI: https://doi.org/10.1046/j.1525-1497.2003.20756.x PMID:12950483
    19.    Lin C-T, Wittevrongel L, Moore L, Beaty BL, Ross SE. An Internet-based patient-provider communication system: Randomized controlled trial. J Med Internet Res 2005 Aug 5;7(4):e47. DOI: https://doi.org/10.2196/jmir.7.4.e47 PMID:16236699
    20.    Bavafa H, Hitt LM, Terwiesch C. The impact of e-visits on visit frequencies and patient health: Evidence from primary care. SSRN 2017 July 22. DOI: https://doi.org/10.2139/ssrn.2363705
    21.    Wallace LS, Angier H, Huguet N, et al. Patterns of Electronic Portal Use among Vulnerable Patients in a Nationwide Practice-based Research Network: From the OCHIN Practice-based Research Network (PBRN). J Am Board Fam Med 2016 Sep-Oct;29(5):592-603. DOI: https://doi.org/10.3122/jabfm.2016.05.160046 PMID:27613792
    22.    Bertakis KD, Azari R, Helms LJ, Callahan EJ, Robbins JA. Gender differences in the utilization of health care services. J Fam Pract 2000 Feb;49(2):147-52. PMID:10718692
    23.    Cronin RM, Davis SE, Shenson JA, Chen Q, Rosenbloom ST, Jackson GP. Growth of Secure Messaging Through a Patient Portal as a Form of Outpatient Interaction across Clinical Specialties. Appl Clin Inform 2015 Apr 29;6(2):288-304. DOI: https://doi.org/10.4338/ACI-2014-12-RA-0117 PMID:26171076
    24.    Stiles RA, Deppen SA, Figaro MK, et al. Behind-the-scenes of patient-centered care: Content analysis of electronic messaging among primary care clinic providers and staff. Med Care 2007 Dec;45(12):1205-9. DOI: https://doi.org/10.1097/MLR.0b013e318148490c PMID:18007171
    25.    Sittig D. Results of a content analysis of electronic messages (email) sent between patients and their physicians. BMC Med Inform Decis Mak 2003 Oct 1:3:11. DOI: https://doi.org/10.1186/1472-6247-3-11 PMID:14519206

Keywords: family medicine, patient-generated emails, primary care email topics, physician-patient communication, physician-patient relationship, secure messaging