Quality Measurement and Improvement in General Surgery


Marisa Cevasco, MD, MPH, Stanley W Ashley, MD

Fall 2011 - Volume 15 Number 4



Increasing emphasis is being placed on health care quality measurement and improvement in the US. Within general surgery, several sophisticated quality-measurement and outcomes systems have been developed. These include the National Surgical Quality Improvement Program, the use of selective referral and centers of excellence, the Surgical Care Improvement Project, and the World Health Organization Surgical Safety Checklist. This article reviews each of these quality-improvement initiatives, highlights their relative contributions, and discusses future directions of quality improvement within general surgery.

Measuring and improving the quality of health care is an increasingly important goal in American medicine. Patients and their families request information on outcomes, payers require health care systems to address variations in quality of care, and credentialing agencies demand evidence that hospitals meet performance standards. Within general surgery, the American Board of Surgery has implemented new standards for maintenance of certification, requiring surgeons to monitor their own performance.1 The Joint Commission also tracks key indicators of surgical safety and monitors surgeon-specific performance as part of its credentialing process.2 Payers in both the public and private sectors are rapidly implementing centers of excellence and pay-for-performance programs, further driving the need to systematically track and improve the quality of surgical care.3

The importance of measuring and improving surgical quality is well established, but it is unclear how best to accomplish these objectives. Time-consuming review of medical records, the tardiness of retrospective quality measures, and the accuracy and fairness of public report cards are frequently cited shortcomings of current methods.4 However, surgeons and national surgical organizations, such as the American College of Surgeons (ACS), are developing more precise and timely methods of measuring surgical quality, with the goal of ultimately improving outcomes. This article discusses current initiatives in quality measurement and improvement within general surgery and highlights future directions within the field of surgical outcomes.

The National Surgical Quality Improvement Program

The ACS National Surgical Quality Improvement Program (NSQIP) is the most widely recognized quality-measurement system for noncardiac surgery in the US.2 It was originally developed in the Department of Veterans Affairs health system in response to a 1986 congressional mandate for the system to compare its risk-adjusted surgical outcomes with those in the private sector.5 NSQIP's overarching purpose is to improve the quality of surgical care not by identifying bad surgeons but rather by focusing on the processes and structure of care at the levels of surgical subspecialty and surgical service. Outcomes of interest include 30-day postoperative mortality and morbidity, chosen primarily because of their importance and relative ease of data collection. From 1991 through 2001, the Department of Veterans Affairs saw a 27% decline in postoperative mortality and a 45% decrease in postoperative morbidity, along with decreased length of stay and improved patient satisfaction scores.6

After validation in three academic medical centers, NSQIP was launched in the private sector in 1999, and in 2004, the system partnered with the ACS with the goal of nationwide involvement. Data abstraction is conducted by trained clinical nurse abstractors according to well-tested procedures and rigorously defined variables. A comprehensive set of clinical and laboratory risk factors are assessed for every patient, and submitted data are externally audited to ensure their completeness and accuracy.6 Well-validated risk-adjustment models incorporate data from a random mix of procedures.

Participating hospitals receive robust, risk-adjusted surgical outcomes, expressed relative to other hospitals as "observed to expected" (O/E) ratios. (An O/E ratio <1 indicates that the hospital is performing better than expected, given the comorbidities of its patient population and surgical case complexity; an O/E ratio >1 indicates that the hospital is performing poorer than expected. For example, an O/E mortality ratio <1 means that fewer deaths occurred than anticipated in comparison with peer hospital performance.) These biannual comprehensive reports are blinded, allowing participating centers to compare their risk profiles and outcomes with those of peer medical centers and national averages.6 Multiple studies have demonstrated that institutions can enhance outcomes by directing performance-improvement initiatives in areas where they seem to be outliers.7–9

Currently, more than 300 hospitals participate in the ACS NSQIP, and in 2002, the Institute of Medicine named the NSQIP "the best in the nation" for measuring and reporting surgical quality and outcomes.

To ensure continued success of this program, future efforts include focusing on specific, more complex procedures and limiting the number of clinical data collected, thereby decreasing the data-collection burden without sacrificing the accuracy of the outcomes models.1 Currently, a large number of variables are required to accurately adjust for differences in risk factors when evaluating outcomes among many procedures. However, as seen in cardiac surgery, using fewer than ten main variables affects risk adjustment when restricting analysis to the single procedure of coronary artery bypass grafting, and initial studies in general surgery confirm that limited models using as few as five variables generate adequate risk adjustment for procedure-specific outcomes.10 Additional future steps include incorporating information about processes of care, which describe the care that patients actually receive. For example, processes of care related to surgical procedures include appropriate preoperative imaging and the perioperative use of beta-blockers. Process-of-care measures provide member hospitals with more reliable performance measures and more actionable outcomes data.1

Selective Referral and Centers of Excellence

Better patient outcomes have been reported when complex surgical procedures are performed at high-volume hospitals and academic medical centers. This phenomenon was first noted in 1979, when Luft et al reported that the mortality rate for certain surgeries was inversely proportional to the number of procedures performed.11 Since then, numerous studies have continued to describe this relationship, including several reviews suggesting that a large number of deaths may be attributed to elective, high-risk surgery performed in low-volume centers.12–14

The strength of this evidence has prompted health care researchers, advocacy groups, and other organizations to call for regionalization, the process of concentrating high-risk procedures at high-volume centers. For instance, the Leapfrog Group for Patient Safety, a national consortium of private and public purchasers of health insurance, has encouraged patients undergoing any one of six surgical procedures (coronary artery bypass grafting, abdominal aortic aneurysm repair, aortic valve replacement, weight-loss surgery, esophagectomy, and pancreatectomy) to choose a hospital "that has a high success rate for [stated procedure] and also has lots of experience with the procedure." Several consumer-oriented Web sites, including one provided by the Leapfrog Group, provide patients with procedure-specific outcomes for local hospitals, listing rates of postoperative complications and compliance with selected perioperative processes of care.15

Although it has been more than 30 years since that report by Luft et al, with ample demonstration of the benefit of volume-based referral for complex procedures, referral patterns of patients have not completely evolved. Patients with operable pancreatic cancer are still treated in low-volume centers, with varying results, and a 2002 report of a study investigating mortality after esophagectomy noted a 23% mortality rate at low-volume centers, nearly 3 times as high as that seen in high-volume centers.12 More recently, another study noted that of the 874 hospitals performing esophagectomies, more than 90% performed fewer than 3 esophagectomies annually.13 In a final example involving patients undergoing coronary artery bypass grafting, one study showed that more than 300 deaths could be averted in the US annually if the mortality rate at very-low-volume hospitals was equal to that seen in very-high-volume centers.14

Improved outcomes are not necessarily limited to having a surgical procedure performed in a high-volume center. Perhaps unsurprisingly, there is also evidence noting that the outcomes of surgical procedures depend on individual surgeon volume. One notable 2003 study concluded that for 8 complex procedures, patients treated by high-volume surgeons had decreased mortality rates when compared with those undergoing surgery performed by low-volume surgeons. Moreover, the authors noted that individual surgeon volume was responsible for a large proportion of the effect of being treated in a high-volume hospital.16 "Practice" does confer better outcomes.

Despite these important findings, volume alone is not a guarantee of quality. In reflecting on 15 years of surgical outcomes as studied through the NSQIP, Itani concluded that volume of surgery in individual specialties and specific procedures should not be used as a surrogate for quality of care.17 The findings of earlier studies support this, cautioning that high-volume hospitals could still deliver poor care if the quality of systems are inadequate.18 More investigation is needed into this controversial topic.17

The Surgical Care Improvement Project

The Surgical Care Improvement Project (SCIP) is a national partnership of organizations seeking to reduce complications of surgical care. Developed by the Centers for Medicare & Medicaid Services to decrease surgical morbidity, SCIP uses evidence-based medicine to establish surgical practice guidelines.19 Collaborating with other national organizations such as the American Hospital Association, the Centers for Disease Control, and the Joint Commission, the SCIP has developed quality measures pertaining to perioperative processes of care associated with a high incidence and cost of complications. To date, the Centers for Medicare & Medicaid Services has required only data collection and reporting for institutions to receive their full market basket (a structured payment based on an index-weighted mix of goods and services as evaluated over a period of time); however, beginning in 2011, payments were to depend on performance metrics. Data for these measures are publicly reported.

Targeted areas of clinical improvement include appropriate use of antibiotics to reduce surgical-site infection; perioperative use of beta-blockers to reduce the risk of cardiac events; proper prophylaxis for deep vein thrombosis; optimal temperature and blood glucose control, particularly in patients undergoing cardiac surgery; and processes of care that aim to reduce postoperative pneumonia.19

SCIP measures employ a multidisciplinary approach to reduce the incidence of surgical complications. To illustrate, in an effort to eliminate surgical-site infections, practice guidelines target prophylactic antibiotic administration within one hour before surgical incision, the appropriate antibiotic choice in light of the surgery, use of clippers (not razors) for hair removal, immediate normothermia after surgery, and postoperative normoglycemia for patients undergoing cardiac surgery. Rather than focusing on antibiotics alone, SCIP targets several key aspects of perioperative care to reduce the risk of surgical-site infections.

SCIP measures are continually being reassessed and revised on the basis of the most up-to-date medical guidelines and evidence-based medicine. Although SCIP measures are evidence-based, some studies fail to demonstrate a link between adherence to SCIP measures and improvement in surgical outcomes such as surgical-site infections.20 There has not been convincing evidence that improvements in compliance are associated with better outcomes.

The World Health Organization Surgical Safety Checklist

Avedis Donabedian was a pioneer in the field of health care quality. One of his lasting contributions is the basic framework that he developed with which to think about health care quality. Known as Donabedian's triad, it emphasizes the key elements of structure, process, and outcome when evaluating the quality of health care.21 The three initiatives already described in this article each fit into one of these categories. Selective referrals and centers of excellence aim to change the structure of health care delivery by establishing regional centers of expertise for complex surgeries. The SCIP measures promote adherence to specific perioperative processes of delivering care, thereby falling under the domain of a process initiative. The ACS NSQIP focuses on health care outcome, specifically the morbidity and mortality for surgical procedures. Although each of these initiatives is a valuable contribution to improving surgical quality of care, what we describe next is a novel approach that does not fit neatly into any of Donabedian's categories.

In 2008, in an effort to improve surgical safety and reduce surgical errors, the World Health Organization (WHO) released a safety checklist identifying multiple recommended practices to ensure the safety of surgical patients. This was subsequently formatted into a 19-item checklist focusing on perioperative events that occurred before three crucial steps of surgery: before induction of anesthesia, before skin incision, and before the patient leaves the operating room (OR).22

Before the induction of anesthesia, confirmation of patient identity and planned procedure, site marking, and identification of any known allergies or risk of excessive blood loss are among the checklist items that must be covered. Before skin incision, reconfirmation of the patient's name and procedure, announcement of any anticipated critical events, and introduction of all members of the OR staff must occur. Finally, before the patient leaves the OR, the circulating nurse must verbally confirm the name of the procedure, announce completion of correct instrument and sponge counts, and ensure appropriate specimen labeling. The complete Surgical Safety Checklist is shown in Figure 1.

Introduction of the Surgical Safety Checklist was associated with significant improvements in surgical outcomes. Most notably, a reduction in postoperative complications and deaths of >30% was seen in a landmark trial conducted in 8 socioeconomically diverse hospitals from around the world (Toronto, Canada; New Delhi, India, Amman, Jordan; Auckland, New Zealand; Manila, Philippines; Ifakara, Tanzania; London, England; and Seattle, WA, US). A decrease in postoperative surgical-site infections, unplanned returns to the OR, deaths, and overall complications occurred at all sites, independent of surgical case mix and whether the center was located in a high-income or low-income area.22 Although evidence supporting decreased surgical complications and deaths was strong, it was not clear what precisely led to this improvement. Authors of the initial validation study hypothesized that use of the checklist "involved both changes in systems and changes in the behavior of individual surgical teams"; the mechanism for improvement was multifactorial rather than consisting of simply checking off boxes in the OR. Indeed, at one site, the checking of intraoperative pulse oximetry rates increased by 50%, and at a second site, the rate of implementation of sponge counts increased from 0% to 92.4%. Finally, the overall rate of appropriate use of antibiotics increased from 56% to 83%; this intervention is a well-known means of reducing surgical-site infections and certainly contributed to the decreased rate of postoperative complications.22

Although the WHO Surgical Safety Checklist has been demonstrated to improve the quality of surgical care, its impact depends on how effectively it is implemented. In addition, use of the checklist is not universal, and some hospitals prefer to maintain other safeguards for patient safety, including presurgery checks that were in use before the WHO Surgical Safety Checklist was developed.23 Some remain unconvinced that compliance with these commonsense practices could lead to such drastic reduction in morbidity and mortality, whereas others contend that the surgical Hawthorne effect—that is, outcomes improve when surgeons know they are being evaluated—is responsible for the observed effects.24

Recently, a more comprehensive perioperative safety system (formally entitled the Surgical Patient Safety System [SURPASS]) involving nearly 11 checklists and 100 items was developed and implemented in 6 regional hospitals in a study conducted in the Netherlands. Some items were identical to those seen on the WHO surgical checklist, such as confirmation of patient identification and verification of surgical site, but this more comprehensive checklist included other items such as communication of postoperative instructions between caregivers and clarification of discharge instructions to the patient. In the study, the rate of complications declined by one-third, from 15.4% to 10.6%, and mortality declined by nearly one-half, from 1.5% to 0.8%.25

Authors of the Netherlands trial built on prior experience and designed a study that avoided some of the limitations seen in earlier work. Specifically, because the safety system was implemented in hospitals that already tracked and monitored surgical outcomes, results could not as easily be attributed to the surgical Hawthorne effect.24 Additionally, a strong correlation was noted between how thoroughly the checklist was completed and the rate of complications; patients with incomplete checklists were more likely to develop a complication than were those with completed checklists. Finally, five similar hospitals serving as study controls went without implementation of a comprehensive checklist; morbidity and mortality rates remained unchanged in that group.25

Shortcomings of the SURPASS approach were reported to include the onus of completing 11 checklists and the potential for alert fatigue in caregivers as the novelty of such an extensive intervention wanes over time.25 Indeed, the true value of such a comprehensive checklist may become apparent only over time.


Future Directions and Conclusions

Although great strides have been made in the measurement and improvement of surgical outcomes, there is still work to do. Outcomes studies completed thus far generally have explored a defined set of perioperative variables and how they relate to postoperative morbidity and mortality and have been less likely to investigate processes of care. This has been largely because of the difficulty inherent in capturing the multifaceted information required to report on processes of care. However, although the ACS NSQIP currently does not collect information on processes of care, future efforts will likely incorporate data on those process measures required by the SCIP guidelines.1

There are several benefits to reporting on process-of-care measures. Process-based feedback may be more quickly incorporated into surgical practice, as opposed to analysis of morbidity and mortality rates, which do not necessarily provide a direct interpretation of why some outcomes are better than others. Additionally, incorporation of SCIP measures into the ACS NSQIP will help simplify data-collection efforts because the SCIP guidelines are important components of several pay-for-performance initiatives. Finally, process-of-care analysis will help identify best practices, an important component of surgery improvement efforts currently missing from outcomes measures.

In addition to incorporating information on processes of care, there are increasing calls for standardized measurement of communication and information transfer. Several reports of studies cite communication breakdown as a common cause of surgical errors and adverse events, and it is known that deficits in information transfer adversely affect processes of care.26 Effective and standardized communication among all members of the OR team, in addition to preoperative and postoperative caregivers, will facilitate surgical safety. The researchers who conducted the SURPASS trial hypothesized that rates for postoperative complications decreased as a result of more effective communication and hand-offs between clinicians.

Finally, direct measurement of surgical skills may play a growing role in future quality measurement and improvement efforts. Traditionally, surgical skills have been assessed in the OR through direct observation by mentors and peers, but there is mounting pressure for more formal measurement.27 Precisely how to measure these skills has not been clearly established, and reasons for this include lack of objective assessment methodology, lack of proper infrastructure for implementation, and high costs associated with individual performance analysis. Regardless, it is clear that quality measurement and improvement in general surgery is here to stay.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.


Katharine O'Moore-Klopf, ELS, of KOK Edit provided editorial assistance.

1.    Birkmeyer JD, Shahian DM, Dimick JB, et al. Blueprint for a new American College of Surgeons: National Surgical Quality Improvement Program. J Am Coll Surg 2008 Nov;207(5):777–82.
2.    The Joint Commission [homepage on the Internet]. Oakbrook Terrace, IL: The Joint Commission; © 2011 [cited 2011 Jun 20]. Available from: www.jointcommission.org/.
3.    Birkmeyer NJ, Birkmeyer JD. Strategies for improving surgical quality—should payers reward excellence or effort? N Engl J Med 2006 Feb 23;354(8):864–70.
4.    Ilminen GR. Improving healthcare quality measurement. Quality Progress 2003 Dec;36(12):62–6.
5.    Khuri SF, Henderson WG, Daley J, et al; Principal Site Investigators of the Patient Safety in Surgery Study. The patient safety in surgery study: background, study design, and patient populations. J Am Coll Surg 2007 Jun;204(6):1089–102.
6.    American College of Surgeons National Surgical Quality Improvement Program [homepage on the Internet]. Chicago, IL: American College of Surgeons; © 2011 [cited 2011 Jun 20]. Available from: www.acsnsqip.org.
7.    Gibbs J, Clark K, Khuri S, Henderson W, Hur K, Daley J. Validating risk-adjusted surgical outcomes: chart review of process of care. Int J Qual Health Care 2001 Jun;13(3):187–96.
8.    Daley J, Forbes MG, Young GJ, et al. Validating risk-adjusted surgical outcomes: site visit assessment of process and structure. National VA Surgical Risk Study. J Am Coll Surg 1997 Oct;185(4):341–51.
9.    Raval MV, Hamilton BH, Ingraham AM, Ko CY, Hall BL. The importance of assessing both inpatient and outpatient surgical quality. Ann Surg 2011 Mar;253(3):611–8.
10.    Dimick JB, Osborne NH, Hall BL, Ko CY, Birkmeyer JD. Risk adjustment for comparing hospital quality with surgery: how many variables are needed? J Am Coll Surg 2010 Apr;210(4):503–8.
11.    Luft HS, Bunker JP, Enthoven AC. Should operations be regionalized? The empirical relation between surgical volume and mortality. N Engl J Med 1979 Dec 20;301(25):1364–9.
12.    Lemmens VE, Bosscha K, van der Schelling G, Brenninkmeijer S, Coebergh JW, de Hingh IH. Improving outcome for patients with pancreatic cancer through centralization. Brit J Surg 2011 Oct;98(10):1455–62.
13.    Funk L, Gawande AA, Semel ME, et al. Esophagectomy outcomes at low-volume hospitals: the association between system characteristics and mortality. Ann Surg 2011 May;253(5):912–7.
14.    Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortality in the United States. N Engl J Med 2002 Apr 11;346(15):1128–37.
15.    The Leapfrog Group [homepage on the Internet]. Hospital comparisons. Washington, DC: The Leapfrog Group; © 2011 [cited 2011 Jun 20]. Available at www.leapfroggroup.org/cp.
16.    Birkmeyer JD, Stukel TA, Siewers AE, Goodney PP, Wennberg DE, Lucas FL. Surgeon volume and operative mortality in the United States. N Engl J Med 2003 Nov 27;349(22):2117–27.
17.    Itani KM. Fifteen years of the National Surgical Quality Improvement Program in review. Am J Surg 2009 Nov(5 Suppl);198:S9–S18.
18.    Khuri SF, Henderson WG. The case against volume as a measure of quality of surgical care. World J Surg 2005 Oct;29(10):1222–9.
19.    Surgical Care Improvement Project [monograph on the Internet]. Oakbrook Terrace, IL: The Joint Commission; 2011 Feb 7 [cited 2011 Jun 20]. Available from: www.jointcommission.org/surgical_care_improvement_project/.
20.    Ingraham AM, Cohen ME, Raval MV, Ko CY, Nathens AB. Comparison of hospital performance in emergency versus elective general surgery operations at 198 hospitals. J Am Coll Surg 2011 Jan;212(1):20–8.e1. Erratum in: J Am Coll Surg 2011 Jun;212(6):1100-1.
21.    Donabedian A. The quality of care. How can it be assessed? JAMA 1988 Sep 23–30;260(12):1743–8.
22.    Haynes AB, Weiser TG, Berry WR, et al; Safe Surgery Saves Lives Study Group. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 2009 Jan 29;360(5):491–9.
23.    Abdel-Galil K. The WHO surgical safety checklist: are we measuring up? Br J Oral Max Surg 2010 Jul;48(5):397–8.
24.    Birkmeyer JD. Strategies for improving surgical quality—checklists and beyond. N Engl J Med 2010 Nov 11;363(20):1963–5.
25.    de Vries EN, Prins HA, Grolla RM, et al; SURPASS Collaborative Group. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med 2010 Nov 11;363(20):1928–37.
26.    Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA 2007 Feb 28;297(8):831–41.
27.    Reznick RK, MacRae H. Teaching surgical skills—changes in the wind. N Engl J Med 2006 Dec 21;355(25):2664–9.


25,000 print readers per quarter, 6900 eTOC readers, and in 2015, 1.4 million page views on TPJ articles in PubMed from a broad international readership


The Permanente Journal (ISSN 1552-5767) is published quarterly by The Permanente Press. The Permanente Journal is available online (ISSN 1552-5775) at www.thepermanentejournal.org.


Articles, editorials, letters to the editor, and other material represent the opinion of the authors. Send your comments to permanente.journal@kp.org.

Copyright 2016 The Permanente Journal - Kaiser Permanente. All Rights Reserved.