Challenges in Evaluating All-Cause Hospital Readmission Measures for Use as National Consensus Standards

Alexis Morgan, MPH; Adeela Khan, MPH; Taroon Amin, MA, MPH, PhD(c)

Perm J 2013 Fall;17(4):14-18


Context: The National Quality Forum (NQF) aims to improve the quality of health care for all Americans through fulfillment of its three-part mission. The NQF uses its formal Consensus Development Process to evaluate and endorse consensus standards, including performance measures, best practices, frameworks, and reporting guidelines.
Objective: To understand the opportunities and challenges in endorsing measures addressing all-cause readmissions to hospitals for use as national voluntary consensus standards for accountability and quality-improvement purposes.
Design: Report of standards development process.
Main Outcome Measures: The Consensus Development Process was used to evaluate 3 candidate standards using the NQF Measure Evaluation Criteria. A 21-member steering committee rated each standard according to the criteria and made initial endorsement recommendations for all measures.
Results: Through the evaluation of measures for endorsement, several overarching issues in measuring all-cause readmissions were identified, including statistical modeling and the usability of the measures for quality improvement and accountability. Additionally, it was decided that, for the first time, quality monitoring and accountability of readmissions will take place at the health-plan level. Measuring at various levels of accountability reinforces the idea that multiple stakeholders have a responsibility and a role to reduce readmissions.
Conclusions: These NQF-endorsed measures are a major step in promoting better understanding of readmissions and a reduction in hospital readmission rates, when appropriate. These measures can help reduce the substantial financial and emotional stress that readmissions place on the health care system, and patients will be able to communicate hospital-level performance on this important quality indicator.


The National Quality Forum (NQF) is a not-for-profit membership organization created to develop and to implement a national strategy for health care quality measurement and accountability. Membership consists of national, state, regional, and local groups representing consumers, public and private purchasers, health care professionals, provider organizations, health plans, accrediting bodies, labor unions, suppliers, and other organizations involved in health care research or quality measurement and improvement. The NQF has focused on several areas and aims to improve the quality of health care in the US through fulfillment of its three-part mission:

1. setting national priorities and goals for performance improvement

2. endorsing national consensus standards for measuring and publicly reporting on performance

3. promoting the attainment of national goals through education and outreach programs.

To achieve quality health care across the continuum, there is a need for robust measures that specifically address outcomes of care provided in our nation's health care system. Before the start of this project, the NQF had endorsed more than 100 outcome measures through various Consensus Development Process projects, including the multiphase Patient Outcomes Project. However, many gaps remain, including those related to all-cause readmissions.

The private and public sector spends billions of dollars each year on hospital readmissions, with one-fifth of Medicare beneficiaries readmitted to a hospital within 30 days. Medicare payments for unplanned readmissions alone cost the Centers for Medicare & Medicaid Services (CMS) $17.4 billion in 2004.1 Although patients who are privately insured are less likely to experience a readmission than those with public health insurance, the cost of readmitting commercially insured patients is higher because of higher commercial payment rates.2 In 2008, 8% of patients who were privately insured in New York State were readmitted, costing private payers $568.9 million, or 15.2% of the state's total readmissions cost.3

To curb spending across the public and private sector on avoidable readmissions, lawmakers, purchasers, and health plans have proposed incentives to reduce unplanned readmissions through public reporting, making it clear that readmissions, as an outcome indicator of quality, is important to patients.4-7 The Patient Protection and Affordable Care Act,8 which enacted the Hospital Readmissions Reduction Program under Medicare, began penalizing hospitals for high rates of readmissions in Fiscal Year 2013. Condition-specific and/or all-cause readmission measures will be used across several federal programs, including the National Patient Safety Initiative, the Hospital Readmission Reduction Program, and the community-based Care Transition Program to better understand and ultimately reduce unplanned hospital readmission rates across the country.

To support these initiatives, there is an important need to endorse consensus-based performance measures for all-cause readmissions that can be used across various quality reporting and accountability applications.


Consensus Development Process

In October 2011, the Department of Health and Human Services requested that the NQF launch a Consensus Development Process project to identify and endorse cross-cutting, noncondition-specific quality measures (candidate consensus standards) that addressed all-cause hospital readmissions.

As part of the NQF's maintenance of endorsed performance measures, consensus standards related to all-cause hospital readmissions and endorsed before June 2009 were evaluated in this same project. Endorsement maintenance provides the opportunity to update and to harmonize specifications, ensuring that the best available measures are endorsed and the NQF portfolio of voluntary consensus standards remains current.

Submitted measures were developed using national guidelines for publicly reported outcome measures and following the technical guidance set forth by the NQF guidance for outcome measures.9 Candidate consensus standards were evaluated for their suitability on the basis of the four NQF Measure Evaluation Criteria (Table 1) and by using the NQF's Consensus Development Process.10

The Consensus Development Process is intended to consider the interests of stakeholder groups from across the health care industry. The NQF's Consensus Development Process is designed to satisfy the requirements of a voluntary consensus standards-setting organization as defined by the National Technology Transfer and Advancement Act11 and as implemented by the Office of Management and Budget Circular A-119.

For the All-Cause Readmissions Consensus Development Process Project, a 21-member steering committee and the Consensus Standards Approval Committee, composed of clinical and methodologic experts from public and private health care sectors, evaluated the submitted candidate consensus standards (Table 2).

Statistical Modeling

Although the NQF's Measure Evaluation Criteria do not require, or prefer, a specific statistical modeling approach, all measures submitted to the NQF must be tested to ensure that they are reliable and valid. Additionally, the risk adjustment used must be evidence based, relying on intrinsic patient factors.

Each measure developer submitted different methods to capture a readmission. To control for differences in patient case-mix, measures used various approaches to logistic regression modeling. One particular approach used was hierarchical linear modeling, which helps to correct for the shared variance seen in hierarchically structured data. In other words, hierarchical linear modeling would account for the fact that patients who are nested in hospitals may not be independent of each other since patients would be treated by similar physicians in that hospital. Hierarchical linear modeling using a Bayesian estimator results in two coefficient estimates of the regression model for each level (patient and hospital).12

Generalized Bayesian estimators13 can be used to account for the simultaneous relationships and shared variance among hierarchical levels by computing a weighted combination of the two levels.12 Furthermore, classic estimation theories do not assume prior knowledge of a distribution for variables in the regression model. In contrast, Bayesian estimation may be used to predict hospital performance from a common prior distribution, such as performance from a previous year, or from the grand mean of the population of all hospitals.




Of the candidate consensus standards (Table 2), 2 measures, 1789: Hospitalwide all-cause unplanned readmission measure and 1768: Plan all-cause readmissions, were subsequently endorsed by the NQF Board of Directors.

Throughout the review process, several challenges to measuring and endorsing all-cause hospital readmissions were identified, and important overarching themes were noted throughout the evaluation.

Statistical Modeling

Stakeholders raised concerns that the risk-adjusted rates using hierarchical linear modeling with Bayesian estimators to predict performance for low-volume hospitals would be adjusted toward the mean performance of all hospitals. This shrinkage toward the mean could potentially cause bias for small-volume hospitals, which would seem to have average rates. However, others argued that if the measures did not use a Bayesian estimation technique, hospitals with small volumes would have large confidence intervals around their performance score that would overlap with the average performance of all hospitals. This would result in performance scores for low-volume hospitals that would not allow for statistically significant and practically meaningful differences in performance.

It was broadly recognized that shrinkage toward the mean should theoretically be less of a concern since there is a greater sample size per hospital for all-cause readmissions than condition-specific readmissions.

Hospital Volume

Major concerns also were raised about the lack of risk adjustment for hospital volume. Experts noted that growing literature demonstrates a relationship between hospital volume and quality performance. Two principal hypotheses have been advanced to explain the relationship. The first hypothesis is that physicians (and hospitals) develop more effective skills if they treat more patients (higher volumes), and the second hypothesis is that physicians (and hospitals) achieving better outcomes receive more referrals and thus accrue larger volumes.14,15 This relationship would demonstrate that including volume, as a covariate, would improve the risk adjustment models performance. However, in developing risk adjustment models for performance measures, there should be consideration on why conceptually stakeholders should expect a priori differences in hospital performance on all-cause readmissions rates based on their patient volume and whether such differences are justified. A systematic review to understand what is known about how volume and outcomes are associated, and by what causal pathways they might be related, did not determine that any specific factor was able to explain outcome differences between high- and low-volume hospitals.16 The review found that no longitudinal studies address the important question of how much of the variability, especially among low-volume hospitals, is due to chance.

Usability for Quality Improvement

Throughout the Consensus Development Process, questions surrounded the usability and implementation of an all-cause readmission measure using Medicare claims data. Physicians expressed concern about the inability to produce strategies for effective performance evaluation and quality-improvement activities because of the timeliness of reporting by the CMS and others. Delays in receiving performance results means hospitals are unable to produce strategies for improvement, effectively making it difficult for consumers to make timely and informed decisions when selecting a physician. As such, there was consensus that if these measures are to be used for public reporting, accountability, or payment, the CMS and other users must provide timely feedback to hospitals to support rapid-cycle improvement. Equally as important, timely reporting of quality performance will help to inform consumers about higher-quality hospitals.

The CMS has indicated that they are working to develop methods to provide timely feedback and information on all readmissions, through the CMS dry run, hospital-level results will be shared with hospitals to better understand the performance on the number of readmissions, the patient risk factors, comorbidities, and the name of the hospital where the patient was readmitted.

Of note, the usability of the health plan-level measure was also a point of discussion. The measure was seen as useful for holding health plans responsible for readmissions, recognizing that responsibility for effective care coordination is not solely the job of the hospital or physician.

Risk Adjustment and Socioeconomic Status

Measures submitted for this project were not risk adjusted for a patient's socioeconomic status. In accordance with the NQF guidance, inclusion of socioeconomic status factors in risk-adjustment models is inappropriate because it assumes that differences in outcomes based on those factors are acceptable. Risk adjustment for socioeconomic status may conceal disparities in care by race or ethnicity. The NQF guidance advises that distinctions among population groups should be emphasized to acknowledge disparities, and that including items such as race or ethnicity in a risk-adjustment model would mask differences based on these characteristics.17

Although the steering committee agreed that the NQF guidance was appropriate, there was concern regarding evidence indicating that a patient's socioeconomic status may affect his/her posthospitalization recovery owing to inaccessibility of community resources.18,19 As such, hospitals argue that they cannot be solely responsible for readmissions because multiple factors influence their patients' likelihood of being readmitted.20 Many methodologic challenges inhibit including socioeconomic status as a variable in a measure, since it is especially complex to interpret what this variable is correcting for in a predictive model. To overcome this challenge, some agreed that to support fair and appropriate comparisons, hospital performance on this measure could be reported in like comparison groups (eg, disproportionate-share hospitals). Doing so would highlight disparities in care and improve policy decisions, specifically potential unintended consequences such as the removal of community-level resources from vulnerable communities.

However, it should also be noted that stratification, by disproportionate-share hospitals in particular, might be inappropriate since payments to disproportionate-share hospitals are dependent on Medicaid eligibility and coverage, which vary across states. Also, there is no evidence to suggest that disproportionate-share hospitals should perform a priori differently than do nondisproportionate-share hospitals, or that the difference seen is justifiable. Recognizing that measuring readmissions without adjustment for socioeconomic status holds a hospital partially responsible for coordination and collaboration across care transitions, neither readmissions measure seeks to isolate simply the hospitals' role, but rather seeks to encourage shared accountability for care transitions. In short, developers of both measures recognized that readmissions are not a quality issue solely for hospitals but also for communities and for local health systems.


The NQF process resulted in endorsement of both a health plan- and hospital-level 30-day all-cause hospital readmission measure. To address many of the overarching issues identified throughout the Consensus Development Process, the NQF developed guidance language to reflect the multiple perspectives voiced during the discussion, particularly related to the hospital-level measure (1789). This language sought to reinforce the multifactorial nature of readmissions and the importance of hospital-community collaboration to reduce readmission rates.

The NQF board provided the following guidance language for measure 1789:

Multiple factors affect readmission rates and other measures including: the complexity of the medical condition and associated therapies; effectiveness of inpatient treatment and care transitions; patient understanding of and adherence to treatment plans; patient health literacy and language barriers; and the availability and quality of postacute and community-based services, particularly for patients with low income. Readmission measurement should reinforce national efforts to focus all stakeholders' attention and collaboration on this important issue.21

The NQF acknowledges the concerns raised by physicians and hospitals on the potential negative impact to hospitals that disproportionately treat patients with complex social comorbidities. Ultimately, an all-cause readmission measure provides an opportunity to highlight important quality outcomes that are important to patients.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.


Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.

1. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med 2009;360(14): 1418-28.
2. Steiner C, Barrett M, Hunter K. Hospital readmissions and multiple emergency department visits, in selected states, 2006-2007. Healthcare Cost and Utilization Project statistical brief #90 [monograph on the Internet]. Rockville, MD: Agency for Healthcare Research and Quality; 2010 May [cited 2013 Jun 5]. Available from:
3. Chollet D, Barrett A, Lake T; Mathematica Policy Research. Reducing hospital readmissions in New York State: a simulation analysis of alternative payment incentives [monograph on the Internet]. New York: NYS Health Foundation; 2011 Sep [cited 2013 Jun 5]. Available from:
4. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med 2011 Oct 18;155(8):520-8. DOI:
5. Corrigan JM, Martin JB. Identification of factors associated with hospital readmission and development of a predictive model. Health Serv Res 1992 Apr;27(1):81-101.
6. Ashton CM, Del Junco DJ, Souchek J, Wray NP, Mansyur CL. The association between the quality of inpatient care and early readmission: a meta-analysis of the evidence. Med Care 1997 Oct;35(10):1044-59. DOI:
7. Halfon P, Eggli Y, Prêtre-Rohrbach I, Meylan D, Marazzi A, Burnand B. Validation of the potentially avoidable hospital readmission rate as a routine indicator of the quality of hospital care. Med Care 2006 Nov;44(11):972-81. DOI:
8. The Patient Protection and Affordable Care Act of 2010. Public Law 111-148, 111th Congress, 124 Stat 119, HR 3590, enacted 2010 Mar 23.
9. Guidance for evaluating the evidence related to the focus of quality measurement and importance to measure and report [monograph on the Internet]. Washington, DC: National Quality Forum; 2011 Jan [cited 2013 Jun 6]. Available from:
10. Consensus development process [Web page on the Internet]. Washington, DC: National Quality Forum; c2013 [cited 2013 Jun 6]. Available from:
11. National Technology Transfer and Advancement Act of 1995, Pub. L. No. 104-113, 110 Stat. 775 (1996 Mar 7).
12. Woltman H, Feldstain A, MacKay JC, Rocchi M. An introduction to hierarchical linear modeling. Tutor Quant Methods Psychol 2012;8(1):52-69.
13. Berger JO. Statistical decision theory and Bayesian analysis. 2nd Ed. New York: Springer-Verlag; 1985.
14. Flood AB, Scott WR, Ewy W. Does practice make perfect? Part I: the relation between hospital volume and outcomes for selected diagnostic categories. Med Care 1984 Feb;22(2):98-114. DOI:
15. Flood AB, Scott WR, Ewy W. Does practice make perfect? Part II: the relation between volume and outcomes and other hospital characteristics. Med Care 1984 Feb;22(2):115-25. DOI:
16. Hewitt M. Interpreting the volume-outcome relationship in the context of health care quality: workshop summary. Washington, DC: The National Academies Press; 2000. p 4-10.
17. Weissman JS, Betancourt JR, Green AR, et al. Commissioned paper: healthcare disparities measurement [monograph on the Internet]. Washington, DC: National Quality Forum; 2012 Feb [cited 2013 Jun 6]. Available from:
18. Arbaje AI, Wolff JL, Yu Q, Powe NR, Anderson GF, Boult C. Postdischarge environmental and socioeconomic factors and the likelihood of early hospital readmission among community-dwelling Medicare beneficiaries. Gerontologist 2008 Aug;48(4):495-504. DOI:
19. Joynt KE, Orav EJ, Jha AK. Thirty-day readmission rates for Medicare beneficiaries by race and site of care. JAMA 2011 Feb 16;305(7):675-81. DOI:
20. Joynt KE, Jha AK. Thirty-day readmissions—truth and consequences. N Engl J Med 2012 Apr 12;366(15):1366-9. DOI:
21. NQF endorses all-cause unplanned readmissions measure [press release on the Internet]. Washington, DC: National Quality Forum; 2012 Apr 24 [cited 2013 Aug 15]. Available from:


Click here to join the eTOC list or text ETOC to 22828. You will receive an email notice with the Table of Contents of The Permanente Journal.


2 million page views of TPJ articles in PubMed from a broad international readership.


Indexed in MEDLINE, PubMed Central, EMBASE, EBSCO Academic Search Complete, and CrossRef.




ISSN 1552-5775 Copyright © 2021

All Rights Reserved