The Power of the National Surgical Quality Improvement Program—Achieving A Zero Pneumonia Rate in General Surgery Patients
The National Surgical Quality Improvement Program (NSQIP) of the American College of Surgeons provides risk-adjusted surgical outcome measures for participating hospitals that can be used for performance improvement of surgical mortality and morbidity. A surgical clinical nurse reviewer collects 135 clinical variables including preoperative risk factors, intraoperative variables, and 30-day postoperative mortality and morbidity outcomes for patients undergoing major surgical procedures. A report on mortality and complications is prepared twice a year.
This article summarizes briefly the history of NSQIP and how its report on surgical outcomes can be used for performance improvement within a hospital system. In particular, it describes how to drive performance improvement with NSQIP data using the example of postoperative respiratory complications—a major factor of postoperative mortality. In addition, this article explains the benefit of a collaborative of several participating NSQIP hospitals and describes how to develop a “playbook” on the basis of an outcome improvement project.
The National Surgical Quality Improvement Program (NSQIP) is a surgical outcomes database of the American College of Surgeons (ACS) designed to measure risk adjusted outcomes of surgical interventions so as to compare results between hospitals.1 This is achieved with a validated risk adjustment using a logistic regression model. This risk adjustment allows unbiased comparison of results between hospitals of different sizes serving different patient populations.2,3
NSQIP was born in the mid-1980s from a US government mandate4 to improve surgical outcomes among 133 Veterans Administration (VA) hospitals because of a high-observed rate of mortality and surgical complications. This provision was hampered by the absence of a risk-adjusted database and of a reliable national average for surgical outcomes. Therefore the legislative act provided for the development of a risk-adjustment model that would take into account the severity of a patient’s illness and thus enable “apple-to-apple” comparisons of results from participating hospitals.
Following the National VA Surgical Risk Study conducted by the Department of Veterans Affairs between October 1, 1991, and December 31, 1993, parameters of risk adjustment were established5 and the VA-NSQIP was created in 1994 allowing risk-adjusted comparison of the 133 VA hospitals.6 This constituted the birth of NSQIP. By mandate all VA hospitals were required to participate. The program has proved so successful—with NSQIP participation, mortality and morbidity in the VA system was reduced by 27% and 45%, respectively5—that the ACS adopted NSQIP and tested it in a small number of academic medical centers and private hospitals. The results in this initial cohort of private hospitals validated the VA results and were very well received. In 2004, NSQIP of the ACS (ACS NSQIP) was initiated at the national level (Figure 1). Currently there are over 350 participating hospitals in the US and the Middle East.
How It Works
NSQIP is based on 135 variables collected preoperatively and up to 30 days postoperatively. The variables are collected from the following categories: 1) demographics, 2) surgical profile, 3) preoperative, 4) intraoperative, and 5) postoperative data. Each hospital submits an average of 1600 major operations per year into the NSQIP database. Nine categories of complications are reported: 1) overall mortality; 2) overall complications; 3) cardiac complications; 4) postoperative pneumonia; 5) intubations required within 48 hours postsurgery (>48-hour intubations); 6) unplanned intubations; 7) pulmonary embolism and venous thrombosis; 8) renal dysfunction; and 9) surgical-site infections including superficial, fascia, and deep infections.
The collection of variables is performed by a Surgical Clinical Reviewer trained by NSQIP in the analysis of medical records. It is noteworthy that in hospital systems with established electronic medical records, such as Kaiser Foundation Hospitals, reliability of the data records is consistently very high. The Surgical Clinical Reviewer is audited yearly to ensure reliability of abstraction to at least 95%. The strength of this system lies in the strict definition of complications and validation of risk adjustment. For example, the definition of postoperative pneumonia is predetermined for all participants. Appropriate adherence to these definitions is monitored through regular audits carried out by the central NSQIP office.
The variables are subjected to logistic regression to determine their ability to predict the risk of complications.7 Table 1 shows the ranking by importance of the first nine variables. The consistency of the three primary factors (albumin, presence of cancer, American Society of Anesthesiologists [ASA] score) demonstrated a robust system. The risk adjustment allows precise comparison of hospitals. Figure 2 represents the importance of risk adjustment according to two hospital reviews of surgical mortality. This ranking is reversed when the risk adjustment is introduced, ie, the ratio of observed results and expected results. The expected results are based on NSQIP data using the risk-adjustment variables described above. Without interference from the participating hospitals, Outcomes Sciences, Inc (Cambridge, MA), an independent data management company, performs the calculation of the risk-adjusted outcomes using the patient variables submitted by each hospital.
Display of National Surgical Quality Improvement Program Results
NSQIP results are presented through a “caterpillar” chart depicting observed vs expected ratio for each participating hospital and for each category of complication. Each line corresponds to the result of one particular hospital with the confidence interval (CI) included. The most successful hospitals are to the left of the graph in Figure 3, the worst performers to the right. When the results of a particular hospital are significantly better than the average, the CI line is entirely below the mean (horizontal black line); when the results of a particular hospital are significantly worse than the average, the CI line is entirely above the mean. This “Risk-Adjusted Report” is generated every six months for each category of complication for all participating hospitals. More recently this information has been depicted in bar graphs, which allows for a more concise visual demonstration of these findings (Figure 4).
A graph is generated for each hospital showing the risk-adjusted results for all of the categories of complications (mortality, overall morbidity, cardiac complications, pneumonia, unplanned intubations, >48-hour intubations, deep venous thrombosis and pulmonary emobolism [DVT/PE], renal failure, urinary tract infections [UTIs], and surgical site infections) with hospital-specific ranking indicated by a large arrow (Figure 5). This allows each hospital to evaluate its own performance as compared with other participating hospitals.8 Each category graph includes a small insert that shows the performance of the specific hospital throughout the period of participation. Each individual hospital’s performance can thus be followed over time.
Intubation >48 Hours
After receiving the first 12 months of risk-adjusted data from NSQIP, the Kaiser Permanente Northern California (KPNC) Walnut Creek Medical Center NSQIP group chose its first performance-improvement project on the basis of outcome information on >48-hour intubations. This was an obvious choice because Walnut Creek Medical Center ranked among the significantly worst performers for this complication. The observed incidence was almost twice that of the expected rate after risk adjustment (Figure 6). We benefited from the simplicity of the processes involved to improve this parameter compared with other complications that required more complex processes, such as pneumonia or surgical-site infection. When choosing a first performance-improvement project as a new participant of NSQIP, it is important to consider the complexity of the processes involved and to choose one in which the parts involved are relatively easy to decipher. For example, UTI of >48-hour intubations is a complication with relatively low levels of complexity and a low number of involved processes compared with pneumonia or mortality.
In response to our risk-adjusted results for >48-hour intubations, we implemented an improvement project with 3 main steps (see Sidebar: Principal steps of the performance-improvement project: >48-hours intubation): 1) creation of a working group comprising all stakeholders; 2) identification of all operational problems; 3) development of processes to change practice (small test of change); and 4) verification of results using the NSQIP database, both the risk-adjusted results (Figure 6) and the raw data results (Figure 7). It is important to mention here that both risk-adjusted data and raw data from the NSQIP database are helpful in guiding a performance-improvement project. Whereas risk-adjusted data gives the most “objective” rank of performance and allows comparison to group performance (Figure 6), raw data (Figure 7) allow a rapid assessment of progress made on a short-term basis with a “small test of change” improvement project. Critical adjustments and changes can be made on the basis of the results of raw (run chart) data.
The main results of this project were the identification of major communication shortcomings among the clinicians caring for patients in the immediate postoperative period. The lack of efficient communication between the treatment team and the respiratory therapy team, and the relative absence of respiratory care during postoperative recovery of patients because of staffing issues were discussed and presented to the individual department and the hospital leadership. It is important to note that all problems identified were related to systems and not individual performance of a surgeon, nurse, or respiratory therapist (RT). This is the case for almost all problems identified by NSQIP.
Performance improved within one year after the implementation of a number of small tests of change. The changes implemented included: identifying patients at risk, expanding respiratory care coverage, and avoiding oversedation of patients scheduled to be weaned. We have moved from right to left on the caterpillar graph with an important reduction in the absolute and risk-adjusted incidence of prolonged postoperative intubations (Figure 6). Our observed versus expected quotient is now as expected on risk-adjusted analysis. Interestingly, the mortality associated with this complication was reduced as well (Figure 7).
The success of this project encouraged us to expand and to address the reduction of postoperative pneumonia, a more complex complication with less well-defined operational issues. Again, we implemented an improvement program including pre-, intra- and postoperative interventions with education and enforcement both in the surgery clinic, Post-Aesthesia Care Unit and on the wards (see Sidebar: Principal steps of the performance-improvement project: postoperative pneumonias). We included in our program for the prevention of postoperative pneumonia the processes listed in the Sidebar: The postoperative-pneumonia prevention bundle, including description of all perioperative interventions. This bundle was initially implemented in high-risk patients undergoing major upper abdominal surgery but has since been expanded to include any major abdominal surgery requiring hospital admission beyond one day. It is apparent that this project exceeds the complexity of the previous project by several magnitudes. The difficulty resides in the details of implementation and ensuring close to 100% adherence to the processes in clinical practice. This requires a coordinated approach by the nursing leadership and strong motivation and buy-in of the involved caregivers: surgeons, anaesthesiologists, nurses, nursing assistants, RTs, and medical assistants.
The incidence of postoperative pneumonia declined considerably and currently places our hospital among the best performers (Figure 8). More compelling is the raw data on the incidence of postoperative pneumonia. The absolute number of patients with pneumonia following general surgery remained at zero from May 2010 until February 2011. This represents an average reduction of 2 to 3 postoperative pneumonias per month in general surgery patients (Figure 9). Since February 2011, we have seen a few pneumonias in our patient population, but the observed incidence places us well below the mean observed versus expected ratio in the entire NSQIP cohort. Clearly our next challenge is to sustain the achieved reduction of postoperative pneumonias in our patient population
National Surgical Quality Improvement Program Consortium
The role of a NSQIP consortium is to create a synergy between hospitals participating in a region.9,10 It has been shown that regional cooperation can reduce expense and surgical complications.11
Create a Playbook
A playbook is a compilation of all elements necessary to implement a performance-improvement project after successful testing at one or two beta sites. For example, our KPNC collaborative developed a playbook on the basis of the successful implementation of a performance-improvement project focused on wound classification in the operating room. The goal of this project was the correct classification of wounds to assure greatest accuracy of the risk-adjusted reporting of postoperative wound infection into the NSQIP database. This is a good example of how to address one of the key issues with outcomes databases such as NSQIP: how to ensure validity and accuracy rate of the reported data. Our goal was an accuracy of at least 90% in postoperative documentation of wound class.
In this example, the process of Plan-Do-Study-Act (PDSA) was used. An improvement plan was set in motion with successive re-evaluations and adjustments. More rapid cycles of PDSA followed to improve the education process and verification of the classification of wounds (Figure 10). Figure 11 shows the gradual achievement of at least 90% accuracy in wound classification. As part of the playbook, posters with the definition of wound classification (per the ACS)12 are distributed in all operating rooms. The classification of wounds is now part of the formal debriefing after surgery. The educational materials, posters, and debriefing checklists are now used by all operating rooms in the KPNC collaborative.
NSQIP is central to our efforts to achieve high-quality and effective surgical care for our patients. It has been shown that by using processes similar to the ones described in this article, participating institutions can reduce postoperative mortality and complications by an average of 27% or 45%, respectively. Each participating NSQIP hospital has been shown to eliminate an average of 250 postoperative complications and 12 to 27 surgical deaths per year.8 The economic implications are impressive. The Walnut Creek Medical Center has saved more than 1 million USD per year since the implementation of the NSQIP. This far exceeds the cost of the program of (100,000 to 150,000 USD per year): 35,000 USD for the processing of data and 50,000 to 100,000 USD to hire the NSQIP Surgical Nurse Reviewer (US wages) for data collection.
The author(s) have no conflicts of interest to disclose.
1. National Surgical Quality Improvement Program [home page on the Internet]. Chicago, IL: American College of Surgeons; 2005 [cited 2012 Jan 17]. Available from: www.acsnsqip.org.
The Permanente Journal values your opinion. Please take this time to give us your input, which will help us to serve you better. Click here to complete the survey and for a chance to win one of two Apple iPad Minis.
Copyright 2013 The Permanente Journal - Kaiser Permanente. All Rights Reserved.