etoc emailClick here to join the eTOC list or text TPJ to 22828. You will receive an Email notice with the Table of Contents of The Permanente Journal.

Effectiveness of Best Practice Alerts for Potentially Inappropriate Medication Orders in Older Adults in the Ambulatory Setting


Taylor Ota, PharmD1; Rachana J Patel, PharmD1; Thomas Delate, PhD, MS1

Perm J 2020;24:19.041 [Full Citation]
E-pub: 11/22/2019


Introduction: Information is limited about the effectiveness of best practice alerts (BPAs) for potentially inappropriate medications (PIMs) in improving clinical outcomes in older adults.
Objective: To assess clinical outcomes of 11 BPAs for PIMs in older adults in the ambulatory setting.
Methods: A retrospective cohort study was conducted at an integrated health care delivery system with computerized provider order entry. Patients aged 65 years and older were included if they had a BPA triggered when a prescriber attempted to order a sedating PIM in the ambulatory setting. Patients were categorized into dispensed and nondispensed groups if they did and did not, respectively, have the study PIM for which the BPA was triggered dispensed within 30 days of the alert. Rates of fall, fracture, or other injury and cognitive impairment were measured during 180-day follow-up.
Results: A total of 2704 patients were included: 1373 (50.8%) and 1331 (49.2%) in the dispensed and nondispensed groups, respectively. The dispensed group had a lower unadjusted rate of fall/fracture/injury (3.4% vs 5.3%, p = 0.019), but this difference was attenuated with multivariable adjustment (adjusted odds ratio = 0.77, 95% confidence interval = 0.51-1.13). There was no difference in the rate of cognitive impairment between groups (4.6% vs 4.4%, adjusted odds ratio = 1.40, 95% confidence interval = 0.95-2.05).
Conclusion: No association was identified between PIM dispensing after a prescriber was alerted with a BPA and reduced rates of falls/fractures/injuries and cognitive impairment.


The goal of the 2015 American Geriatrics Society (AGS) Beers Criteria for Potentially Inappropriate Medication (PIM) Use in Older Adults is to improve care in older adults by reducing their exposure to PIMs (eg, anticholinergic and psychotropic medications).1 The Beers Criteria recommend that any PIM with strong sedating effects be avoided in older adults because of the increased risk of injury and impaired cognition.2-5 Despite these recommendations, PIM use in older adults continues to be widespread.6

Health information technology, including clinical decision support (CDS) tools for computerized provider order entry (CPOE), is available that uses a best practice alert (BPA) to warn prescribers of a PIM being ordered for a vulnerable patient (eg, an older patient).7 When a prescriber attempts to order a PIM, a BPA is triggered and a dialog box with an alert appears on the prescriber’s screen. The BPA warns of the PIM’s potential adverse effect or effects, provides alternate medication recommendations, and allows the prescriber to choose to override the BPA (ie, continue the PIM order), stop the PIM order, and/or order an alternate medication.

Although CDS tools have a demonstrated benefit in reducing PIM prescribing,8 literature is mixed on whether they help improve patient outcomes. Brenner and colleagues,9 who conducted a meta-analysis of 69 studies of CDS tools in the clinical setting, reported that 25 studies found benefit on outcomes, 43 studies found nonsignificant or mixed findings, and 1 study found a detrimental effect. Gurwitz and colleagues10 reported that a CDS tool did not reduce the adverse drug event rate in their randomized clinical trial in the long-term care setting and noted that alert burden may have affected the efficacy of the CDS tool. A cost-effectiveness analysis of CPOE CDS tools in a midsized multidisciplinary medical group identified that use of the tools was a cost-effective strategy to improve medication safety.11

These mixed results are further limited by a dearth of information on the clinical outcomes of CDS tools in the ambulatory and older patient populations.9 Thus, the purpose of this study was to assess the clinical outcomes of PIM BPAs in older adults in the ambulatory setting. Results of this study provide practitioners and policy makers additional information on the effectiveness of BPAs in an understudied population.


Study Design and Setting

This was a retrospective cohort study of adults aged 65 years and older receiving care in an ambulatory setting who had a PIM BPA triggered during computerized medication order entry. The study was conducted at Kaiser Permanente Colorado (KPCO), an integrated health care delivery system providing care to more than 660,000 patients in Colorado at 31 medical offices. KPCO uses an electronic health record that provides e-prescribing capabilities and has a BPA (Figure 1) for 11 PIMs with sedation effects. The study BPA was triggered when a prescriber attempted to order a PIM for a patient aged 65 years or older (no matter the patient’s health history). A BPA could be overridden (by clicking “Keep” on the alert window) or canceled (by clicking “Remove” on the alert window) for the current prescription, with the override in effect for the specific PIM for 1 year unless canceled earlier. A BPA would be triggered if a different PIM was attempted to be ordered at any time. If the BPA was overridden, the prescription was sent to a pharmacy to be filled. There was no PIM BPA in the electronic pharmacy informatics system.

An index date was assigned to each patient on the trigger date of his/her PIM BPA during the study period. If a patient had multiple BPAs triggered during the study period, only the first eligible PIM BPA was included. Patients were followed-up for as long as 180 days to assess for a study outcome.

All KPCO medical offices have a pharmacy that dispenses subsidized prescription medications to KPCO members. Information on prescriptions dispensed from these pharmacies is maintained in a KPCO administrative database. Coded and free-text medical, laboratory, Emergency Department, hospitalization, and membership data from within the delivery system, as well as from other contracted and affiliated facilities, are captured in KPCO’s administrative and claims databases. The KPCO institutional review board reviewed and approved all study activities. Because this was a retrospective evaluation, informed consent was not required.


Study Population

All KPCO patients aged 65 years and older who had a PIM BPA triggered between January 1, 2016, and May 31, 2017, were eligible for inclusion. The PIMs with anticholinergic and sedating effects included were amitriptyline, chlorzoxazone, cyclobenzaprine, doxepin at a dosage greater than 6 mg/d, hydroxyzine, imipramine, metaxalone, methocarbamol, nortriptyline, orphenadrine, and promethazine.1 Patients had KPCO membership during the 6 months before the index date (to allow for assessment of potential confounders) and 30 days after the index date (to allow for assessment of PIM dispensing). Patients who had the prescription written to a non-KPCO pharmacy, for an in-office administration, or for a compounded PIM were excluded. Patients were categorized into dispensed and nondispensed groups if they did and did not, respectively, have the study PIM for which the BPA was triggered dispensed within 30 days of the index date. Patients in the dispensed group were followed-up from the study PIM dispensing date until 180 days from the index date, study outcome date, KPCO membership termination date, or death date, whichever came first. Patients in the nondispensed group were followed-up from the index date until 180 days, study outcome date, KPCO membership termination date, or death date, whichever came first.

Outcome Measures

The primary outcome measure was a fall, fracture, or other injury during the 180 days after the index date (follow-up period). Falls, fractures, and injuries were identified from diagnoses recorded in both the ambulatory and inpatient settings. Secondary outcome measures included cognitive impairment resulting in a medical office visit, Emergency Department visit, or inpatient stay during follow-up. Injuries were defined as damage to the body caused by external force. Cognitive impairment was defined as confusion, altered mental status, delirium, or memory status changes that were not related to progressive diseases (eg, dementia, Parkinson disease), electrolyte abnormalities, or infections.

All clinical outcomes were validated by manual review of the electronic health record by a clinician referee blinded to the study group. The referee was instructed not to assess information about medication use unless the potential outcome was an actual event based on the written notes of a physician or midlevel prescriber (ie, nurse practitioner, physician’s assistant) from the encounter. The notes were copied verbatim into a spreadsheet. Any potential outcome event with encounter information that did not support the diagnosis or had missing encounter information was deemed a nonvalidated event. Any other potential event with ambiguous information regarding the event was reviewed by another referee. Only validated events were included in the final analysis. Multivariable logistic regression models were constructed on the outcomes to adjust for potentially confounding factors and to identify factors associated with increased likelihood of an outcome. Subanalyses were performed for patients who did and did not have a dispensing of the study PIM during baseline (180 days before the index date).

Data Collection and Analysis

Data were collected from queries of KPCO’s electronic, integrated, administration databases. Information on patients who had a BPA triggered was obtained from the electronic health record. Information on study outcomes and patient characteristics was obtained using the International Classification of Diseases Ninth and Tenth Revisions codes (codes available on request). Data on patient characteristics were collected during the 180 days before the index date (baseline).

Because this was a naturalistic, observational study, no a priori sample size or power calculations were performed. Thus, all patients meeting the inclusion criteria and not having the exclusion criteria were included. Age was calculated as of the index date. Patients were categorized into dispensed and nondispensed groups. A chronic disease score, a measure of a patient’s chronic illness burden, was calculated from ambulatory medication dispensings during baseline.12 A Charlson Comorbidity Index was calculated from diagnoses that were recorded during baseline.13

Percentages of fall, fracture, or other injury and cognitive impairment issues were determined by summing all validated respective outcomes during the study period and dividing this value by the number of study participants in each group. A patient could contribute only 1 fall, fracture, or injury and 1 cognitive impairment to the outcomes. Patient characteristics and outcomes were summarized using descriptive statistics: Means (standard deviation) for continuous variables and percentages for categorical variables. Comparisons were performed between groups using c2 tests of association for categorical variables and either independent samples t-tests or Wilcoxon signed rank tests, as appropriate, to compare continuous variables.

Because of differences between the nondispensed and dispensed groups in patient characteristics and potentially confounding covariates, multivariable logistic regression models were constructed to adjust the validated outcomes. Covariates were selected on the basis of having a p value less than 0.2 in the univariate analysis or on the basis of clinical judgment and the presence of at least 5 patients in both groups with the exposure. Covariates included in the models were age; sex; baseline dispensings of an antidepressant, benzodiazepine, narcotic, and skeletal muscle relaxant medication; chronic disease score; Charlson Comorbidity Index; white race; Hispanic ethnicity; baseline diagnoses of delirium, depression, fall/fracture/injury, and cognitive impairment; and Medicaid insurance status. Analyses were performed with statistical analysis software (SAS version 9.4, SAS Institute, Cary, NC). The a was set at 0.05.


A total of 2704 patients had at least 1 PIM BPA triggered during the study period. Of these, 1373 patients (50.8%) had the PIM dispensed within 30 days and 1331 (49.2%) did not have the PIM dispensed (Table 1). The most common PIMs to trigger a BPA were nortriptyline (28.4%), cyclobenzaprine (26.4%), amitriptyline (11.8%), and hydroxyzine (11.0%). Overall, patients were primarily women and white and had a moderate burden of chronic disease.

Patients in the nondispensed group were older (p < 0.001) and had a higher burden of chronic disease and mean count of unique chronic disease medications (p < 0.001; Table 1). In addition, they were more likely to have had a comorbidity of delirium, dementia, and Alzheimer disease; baseline dispensing of an antidepressant and skeletal muscle relaxant; and previous cognitive impairment (all p < 0.05). Patients in the dispensed group were more likely to be white (p < 0.001) and have had a baseline benzodiazepine dispensing (p < 0.05).

Although patients in the dispensed group had a lower unadjusted percentage of fall, fracture, or injury (3.4% vs 5.3% in nondispensed group, p = 0.019; Table 2), this difference was not statistically significant with multivariable adjustment (adjusted odds ratio [AOR] = 0.77, 95% confidence interval [CI] = 0.51-1.13; Table 3). There was no statistically significant difference between the groups in the rate of cognitive impairment in unadjusted analyses (4.6% vs 4.4%) or adjusted analyses (AOR = 1.40, 95% CI = 0.95-2.05).

In the subanalysis between patients in the dispensed (n = 419) and nondispensed (n = 510) groups who did have a prior dispensing of a study PIM during baseline, results identified no difference in outcomes (4.5% vs 6.3%, p = 0.247 for fall, fracture, or injury and 5.0% vs 4.9%, p = 0.939 for cognitive impairment). Similarly, in the subanalysis between dispensed (n = 954) and nondispensed (n = 821) group patients who did not have a prior dispensing of a study PIM during baseline, the results identified no difference in outcomes (2.9% vs 4.5%, p = 0.079 for fall, fracture, or injury and 4.4% vs 4.0%, p = 0.689 for cognitive impairment).

Factors independently related to an increased likelihood of fall, fracture, or injury included increasing age (AOR = 1.05, 95% CI = 1.02-1.07) and a fall, fracture, or injury during baseline (AOR = 2.74, 95% CI = 1.67-4.50; Table 3). Factors independently related to an increased likelihood of cognitive impairment included increasing age (AOR = 1.04, 95% CI = 1.01-1.07) and delirium (AOR = 2.49, 95% CI = 1.25-4.95), antidepressant dispensing (AOR = 1.69, 95% CI = 1.11-2.57), and history of cognitive impairment (AOR = 3.56, 95% CI = 2.09-6.09) during baseline.




In this retrospective study of the clinical effectiveness of PIM BPAs in older adults, we identified no statistically significant association between PIM dispensing after a BPA was triggered and an adverse outcome. Our findings are important because although there is literature demonstrating that BPAs reduce ambulatory prescribing of PIMs in older adults, little data are available that have examined clinical outcomes associated with BPAs for PIM prescribing in older adults.9,14

Systematic reviews provide limited supporting evidence for health information technology in the ambulatory setting on patient outcomes. Brenner and colleagues9 assessed the effectiveness of health information technology (eg, CPOE, CDS) in 10 studies in ambulatory settings, and they identified only 1 study15 that demonstrated a decrease in adverse effects (ie, a significant decrease in asthma exacerbations). Iankowitz and colleagues14 performed a systematic review of 5 studies of CDS for PIM prescribing at hospital discharge and subsequent unplanned Emergency Department visits or hospital readmissions in community-dwelling older adults. Although they found that CDS reduced the rate of PIM prescribing, they identified no studies with a decrease in unplanned Emergency Department visits or hospital readmissions.14

It is plausible that prescribers in our study were more comfortable prescribing a PIM to patients for whom they believed the PIM would be tolerated. This is probably because patients in the dispensed group were younger and healthier. We were unable to assess prescribers’ comfort levels with their patients’ ability to tolerate the PIM. If there was differential prescribing, we would expect additional outcomes without the BPA because prescribers would not be alerted to the risk. We identified that only increasing age and a history of fall, fracture, or injury were related independently with a follow-up fall, fracture, or injury in multivariable regression modeling. Similarly, increasing age, delirium comorbidity, antidepressant dispensing during baseline, and a history of cognitive impairment were related independently with cognitive impairment at follow-up. Neither of the health risk scores (ie, chronic disease score, Charlson Comorbidity Index) was associated with either outcome, suggesting that sicker patients are at no higher risk of one of these outcomes. These findings do not negate any positive effects of a BPA in alerting prescribers to the risk of prescribing a PIM.

In clinical practice with older adults, our findings highlight the importance of the prescriber tailoring a medication regimen, including a medication review and modification, for each patient. In addition, the identification and mitigation of other risk factors (eg, concurrent medication use, environmental hazards, gait or vision impairment, hypotension, sedentary lifestyle, age-related decline in balance) can prevent falls and injuries.16 Furthermore, advising older adults and/or their caregivers on the benefits of cognitive training and physical exercise (eg, resistance training) can aid in preventing cognitive decline.17

A follow-up of 30 days to assess if the PIM was dispensed limited our study. Although this would be a rare event, it is possible that patients had their PIMs dispensed later than 30 days and were misclassified into the nondispensed group. We attempted to control for potential confounding medications and disease states in our regression modeling; however, unknown confounding still may have been present. In addition, we were unable to assess whether the dispensed PIM was ingested. Furthermore, we could not assess for study outcomes that did not come to the attention of the health care system. These outcomes were likely minor events that could not be identified without surveying patients directly.


This retrospective evaluation of older adults in the ambulatory setting identified no statistically significant association between the dispensing of a PIM after the PIM prescriber was alerted to the anticholinergic and sedating effects of the PIM and reduced fall/fracture/injury and cognitive impairment. Although a PIM BPA at the time of medication ordering can be a tool to aid prescribers in making safer clinical decisions for their patients, our findings suggest that the relationship between PIM use and adverse events is complicated and deserves additional research to elucidate. Our study findings do not support the use of PIMs or advise against BPA use in older adults but do buttress the need for evaluating the patient holistically before initiating a PIM order.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.

This study was funded by Kaiser Permanente Colorado. The funder had no role in the study design, collection, analysis and interpretation of data, writing of the report, or the decision to submit the manuscript for publication.


Kathleen Louden, ELS, of Louden Health Communications performed a primary copy edit.

Preliminary results of this study were presented at the American Society of Health-System Pharmacists Midyear Clinical Meeting on December 5, 2018, in Anaheim, CA.

How to Cite this Article

Ota T, Patel RJ, Delate T. Effectiveness of best practice alerts for potentially inappropriate medication orders in older adults in the ambulatory setting. Perm J 2020;24:19.041. DOI:

Author Affiliations

1 Pharmacy Department, Kaiser Permanente Colorado, Aurora

Corresponding Author

Thomas Delate, PhD, MS (

Authors’ Contributions

Taylor Ota, PharmD, designed the research, extracted information from medical records, interpreted the data analysis, drafted the initial version of the manuscript, and revised the manuscript. Rachana J Patel, PharmD, designed the research, supervised the research team, interpreted the analysis, and revised the manuscript. Thomas Delate, PhD, MS, designed the research, extracted information from electronic data sources, performed the statistical analysis, interpreted the data analysis, and revised the manuscript. All authors have given final approval to the manuscript.

1.    American Geriatrics Society 2015 Beers Criteria Update Expert Panel. American Geriatrics Society 2015 Updated Beers Criteria for Potentially Inappropriate Medication Use in Older Adults. J Am Geriatr Soc 2015 Nov;63(11):2227-46. DOI: (Updated: J Am Geriatr Soc 2019 Apr;67(4):674-94. DOI:
    2.    Carrière I, Fourrier-Reglat A, Dartigues JF, et al. Drugs with anticholinergic properties, cognitive decline, and dementia in an elderly general population: The 3-city study. Arch Intern Med 2009 Jul 27;169(14):1317-24. DOI:
    3.    Landi F, Onder G, Cesari M, Barillaro C, Russo A, Bernabei R; Silver Network Home Care Study Group. Psychotropic medications and risk for falls among community-dwelling frail older people: An observational study. J Gerontol A Biol Sci Med Sci 2005 May;60(5):622-6. DOI:
    4.    Falls are leading cause of injury and death in older Americans: Healthcare providers play an important role in falls prevention [press release, Internet]. Atlanta, GA: Centers for Disease Control and Prevention; 2016 Sep [cited 2018 Dec 17]. Available from:
    5.    Falls [fact sheet, Internet]. Geneva, Switzerland: World Health Organization; 2018 Jan 16 [cited 2019 Aug 19]. Available from:
    6.    Davidoff AJ, Miller GE, Sarpong EM, Yang E, Brandt N, Fick DM. Prevalence of potentially inappropriate medication use in older adults using the 2012 Beers criteria. J Am Geriatr Soc 2015 Mar;63(3):486-500. DOI:
    7.    Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007 Mar-Apr;14(2):141-5. DOI:
    8.    Smith DH, Perrin N, Feldstein A, et al. The impact of prescribing safety alerts for elderly persons in an electronic medical record: An interrupted time series evaluation. Arch Intern Med 2006 May 22;166(10):1098-104. DOI:
    9.    Brenner SK, Kaushal R, Grinspan Z, et al. Effects of health information technology on patient outcomes: A systematic review. J Am Med Inform Assoc 2016 Sep;23(5):1016-36. DOI:
    10.    Gurwitz JH, Field TS, Rochon P, et al. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc 2008 Dec;56(12):2225-33. DOI:
    11.    Forrester SH, Hepp Z, Roth JA, Wirtz HS, Devine EB. Cost-effectiveness of a computerized provider order entry system in improving medication safety ambulatory care. Value Health 2014 Jun;17(4):340-9. DOI:
    12.    Von Korff M, Wagner EH, Saunders K. A chronic disease score from automated pharmacy data. J Clin Epidemiol 1992 Feb;45(2):197-203. DOI:
    13.    Quan H, Li B, Couris CM, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol 2011 Mar 11;173(6):676-82. DOI:
    14.    Iankowitz N, Dowden M, Palomino S, Uzokwe H, Worral P. The effectiveness of computer system tools on potentially inappropriate medications ordered at discharge for adults older than 65 years of age: A systematic review. JBI Libr Syst Rev 2012;10(13):798-831. DOI:
    15.    McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet Med 2001 Jul-Sep;26(3):191-201.
    16.    Risk factors for falls [fact sheet, Internet]. Atlanta, GA: Centers for Disease Control and Prevention; 2017 [cited 2019 Feb 1]. Available from:
    17.    Naqvi R, Liberman D, Rosenberg J, Alston J, Straus S. Preventing cognitive decline in healthy older adults. CMAJ 2013 Jul 9;185(10):881-5. DOI:

Keywords: accidental falls, cognitive impairments, computerized provider order entry, medical informatics, potentially inappropriate medications

Reprint Permissions

The Permanente Journal welcomes requests for reprints and reproduction. Use of any and all published materials is copyrighted and protected.


Journal subscriptions for The Permanente Journal are entered for the calendar year. Advance payment in US dollars is required.


27,000 print readers per quarter, 15,350 eTOC readers, and in 2018, 2 million page views of TPJ articles in PubMed from a broad international readership.


Indexed in MEDLINE, PubMed Central, HINARI, EMBASE, EBSCO Academic Search Complete, rdrb, CrossRef, and SciVerse/Scopus.




ISSN 1552-5767 Copyright © 2019

All Rights Reserved.