Owing to adverse publicity within the National Health Service (NHS), in particular the Bristol Royal Infirmary inquiry¹ and the events surrounding Harold Shipman,² pressure is increasing for publication of outcome data for individual hospitals and operators. This has recently resulted in all the cardiac surgeons in the north west of England publishing named surgeon risk stratified mortality data following a request by The Guardian newspaper to the entire cardiac surgical community in the UK under the Freedom of Information Act.^3,4The Guardian newspaper went on to publish named surgeon mortality data from 244 cardiac surgeons in the UK and highlighted that this was only the beginning of releasing the results of all individual physicians in the NHS.⁵

Operator specific outcomes will become available for percutaneous coronary interventions (PCIs) with the introduction of the new Central Cardiac Audit Database (CCAD).⁶ This, combined with the Freedom of Information Act, will eventually push these data into the public arena.⁴ One important limitation of this eventuality, however, unlike for cardiac surgery, is the absence of meaningful risk stratification in the PCI setting. Failure to account for differences in case mix when comparing hospitals and operators may encourage risk averse behaviour and deny interventions to high risk patients who have the most to gain from treatment.

Several risk prediction tools exist within PCI; however, the majority of these examine only in-hospital mortality as an outcome.^7,8,9,10 Owing to the low incidence of in-hospital death after PCI,¹¹ comparisons can be unstable and large numbers over many years have to be collated to provide any meaningful comparisons.

We therefore aimed at developing and validating a multivariate prediction model for major adverse cardiac events (MACE) after PCI in a multicentre UK setting to establish a contemporaneous tool for the adjustment of risk.

METHODS

Patient population and data

The North West Quality Improvement Programme in Cardiac Interventions (NWQIP) is a regional consortium of all four centres (Blackpool Victoria Hospital, Blackpool; The Cardiothoracic Centre, Liverpool; Manchester Royal Infirmary, Manchester; and South Manchester University Hospital, Manchester) performing adult cardiac surgery and PCIs in the north west of England. The goal of the group is to improve continuously the quality of care for patients receiving cardiac interventions by using a regionally based systems approach.

Data were collected on a total of 9914 consecutive patients undergoing PCI between 1 August 2001 and 31 December 2003 in the north west of England. Data collection methods and definitions are available from the NWQIP website.¹² In brief, each intervention had a dataset collected, which included demographics, heart disease severity, acuity, co-morbidity, procedural details, and outcome. Data were validated in each centre, which consisted of checking each record for completeness and flagging back to the relevant cardiology team any erroneous data. All records entered on to the databases were also cross checked against finance activity lists and catheter laboratory books to ensure capture of all cases. Data were collected in each hospital and returned to a central source for analysis every six months. Data would be returned to the providing hospital if data completeness did not achieve a rate of 98% or above. Any missing risk factor data after acceptance into the central registry was treated as absent and this occurred in less than 2% of records.

Data were collected on the following variables: (1) demographic data: age and sex; (2) medical history: New York Heart Association (NYHA) class, diabetes, renal dysfunction, peripheral vascular disease, cerebrovascular disease, and prior coronary artery bypass grafting (CABG); (3) cardiac anatomy and function: significant stenosis of the left main stem (> 50%), number of other diseased native vessels (> 70%), and ejection fraction; (4) indication for PCI: stable angina, unstable angina (inability to perform any physical activity without discomfort; angina may occur at rest), primary treatment for acute myocardial infarction (MI), and cardiogenic shock (blood pressure < 100 mm Hg, pulse > 100 beats/min, patient cool, clammy or requiring intravenous inotropes or intra-aortic balloon pump to support blood pressure);(5) priority of PCI: elective (routine admission from the waiting list), urgent (urgent but able to be scheduled for intervention within current admission; patient cannot be sent home without intervention), and emergency (unplanned—that is, out of hours procedure requiring the laboratory or theatre to reopen or in-hours procedure for patient requiring immediate treatment); and (6) treatment factors: restenosis of attempted lesion, American Heart Association (AHA) class, attempted lesion location, number of vessels treated, use of glycoprotein inhibitors, and device used.¹³

Major adverse cardiac events

The primary outcome for the NWQIP PCI registry is MACE, which is a composite outcome comprising the number of in-hospital deaths, Q wave MI, emergency CABG surgery, and cerebrovascular accident expressed as a percentage of total activity.

In-hospital mortality has been defined as death within the same hospital admission regardless of cause after PCI. Emergency CABG was defined as the decision to send a patient to surgery because of unstable haemodynamic status, ongoing ischaemia, threatening dissection, etc, within 24 hours of receiving PCI. Q wave MI was defined as a new pathological Q wave with creatine kinase more than twice the laboratory upper limit of normal with increased creatine kinase MB fraction or troponin T (ignoring non-Q wave MIs because of the subjectivity involved). Cerebrovascular accident was defined as a persistent neurological deficit at the time of patient discharge.

Statistical analysis

Continuous data are shown as median values with 25th and 75th centiles. Categorical variables are shown as a percentage and comparisons were made by χ² tests as appropriate. Standard statistical tests were used to calculate odds ratios and 95% confidence intervals. A multivariate logistic regression analysis was undertaken, by using the forward stepwise technique, to identify independent risk factors for MACE.¹⁴ Candidate variables were entered into the model with p < 0.1. The area under the receiver operating characteristic (ROC) curve and the Hosmer-Lemeshow goodness of fit statistic were calculated to assess the performance and calibration of the model, respectively.^14,15 The statistical model was internally validated by using the technique of bootstrap resampling.¹⁶ This technique is efficient and provides nearly unbiased estimates of the predictive accuracy of the model.¹⁷ We developed the multivariate model on the entire dataset and then drew 100 samples of 70% at random with replacement. The ROC curve was calculated for each sample. This allowed the calculation of the standard deviation of the mean ROC statistic. The model was externally validated on 1786 consecutive PCI cases available from one of the four hospitals in the north west of England covering the time period 1 January to 31 December 2004. In all cases p < 0.05 was considered significant. All statistical analysis was performed with SAS for Windows version 8.2 (SAS Institute, Cary, North Carolina, USA).

RESULTS

Outcomes and patient data

Of the 9914 patients who underwent PCI in the north west of England, 129 (1.3%) developed an in-hospital MACE. The in-hospital MACE rate varied between hospitals from 1.1–1.4% (p  =  0.57). Sixty six patients (0.7%) died and 36 (0.4%) had a Q wave MI, 15 (0.15%) had an emergency CABG, and 20 (0.2%) had a cerebrovascular accident (numbers are not mutually exclusive).

Table 1 reports the patient characteristics, and table 2 the treatment factors. The majority of patients were men (70.9%) and the median age of the cohort was 61 (25th and 75th centiles 54–68.2) years. Stent use was high at 93% as was the use of glycoprotein inhibitors at 61.6%.

Univariate association with MACE

Tables 1 and 2 show the univariate association with MACE. Significant pre-intervention characteristics were age, sex, NYHA class, cerebrovascular disease, prior CABG, ejection fraction, extent of disease, significant stenosis of the left main stem, indication for PCI, and priority. Renal dysfunction and diabetes had no univariate association with in-hospital MACE (table 1). Significant univariate treatment factors were AHA C-type lesions, LMS and graft lesions, presence of thrombus, and occlusion (table 2).

Independent risk factors for MACE

Table 3 shows the independent risk factors for MACE, along with coefficients, standard errors, odds ratios, confidence limits, and p values. The ROC curve for the multivariate prediction model was 0.76 (fig 1). The predicted risks of individual patients were rank ordered and divided into eight groups. Within each group of estimated risk, the number of MACE predicted was compared with the number of observed MACE. The Hosmer-Lemeshow goodness of fit statistic across groups of risk was not significant (p  =  0.43; fig 2), indicating little departure from a perfect fit.

Table 3 shows the logistic regression equation for calculation of predicted risk of MACE. When this equation was used, the predicted MACE rate varied between hospitals from 1.1% to 1.5% (p < 0.001).

Validation of the model

To examine the ability of the multivariate model to discriminate, the ROC curves of 100 samples (with bootstrapping) were calculated. The area under the “average” curve was 0.74, with a standard deviation of 0.032, indicating a good ability to discriminate between patients who developed MACE and those who did not.

Applying the logistic regression equation to data for 1786 consecutive PCIs performed after December 2003 showed a ROC curve of 0.72, which was similar to previous values, indicating a good discrimination power.

DISCUSSION

Since the introduction of the Freedom of Information Act, individual named surgical results have been published for 244 surgeons performing CABG surgery in the UK.^4,5 There is an assumption, therefore, that it is only a matter of time before operator outcome specific data are requested for PCI.

We feel that any moves to produce unadjusted outcome data for PCI by named operator or hospital will be misleading and will therefore encourage risk averse behaviour. Without appropriate risk adjustment methods being applied to outcome data for PCI, an individual operator’s performance may be adversely skewed by high risk cases to an unsatisfactory level compared with their peers and jeopardise their career or even lead to suspension.

PCI risk models in the literature

Several risk models have been developed on large cohorts of patients undergoing PCI and have been extensively validated on other cohorts.^{7,8,9,10,18,19,20} These models, however, predict the risk only of in-hospital mortality and were based on cohorts from the 1990s before the rapid uptake of medications such as clopidogrel and glycoprotein inhibitors. Furthermore, these models were developed on US cohorts and thus their application in a UK setting may not be relevant, although this is yet to be tested. The biggest limitation of applying these models is, however, that with such a low incidence of death after PCI in the UK,¹¹ large amounts of data would have to be accumulated to support any meaningful analyses, especially when comparing individual operators.

Kimmel et al,²¹ Resnic et al,²² and Singh et al²³ have developed simplified risk models to predict the probability of developing major complications after PCI. Kimmel et al²¹ and Resnic et al,²² however, examined only death, emergency CABG, and MI, unlike our own study, which also included cerebrovascular accidents as did Singh and associates.²³ NWQIP initially used the Singh risk prediction model to adjust for risk when comparing the results of the four participating hospitals; however, we found that the model would substantially overpredict the incidence of major complications, with a predicted MACE of 4.5% against our observed MACE rate of 1.3%.¹² The primary reason for this overprediction is the time period that Singh’s model was based on, which was before the extensive use of stent devices in PCI. The increase in stent use in the late 1990s has led to a dramatic reduction in MACE outcomes and, therefore, models such as those of Singh et al,²³ Kimmel et al,²¹ and Resnic et al²² would all tend to overpredict risk.

One prediction tool already exists for predicting the probability of MACE based on patients from a UK setting.²⁴ This model uses Bayes theorem and was developed on 1500 consecutive patients who underwent PCI starting from January 1995. The variables identified as predictive of MACE were cardiogenic shock, ejection fraction < 30%, thrombolysis in the last 24 hours, renal dysfunction, and age > 76 years. We did not have information available on whether thrombolysis was administered before PCI and could not find an independent association between ejection fraction and renal dysfunction with the development of MACE. Apart from being developed on a cohort of patients from the mid 1990s, de Belder’s model is limited by the Bayesian assumption that all risk factors are independent and therefore would have a tendency to overestimate the risk.²⁴

Strengths and meaning of the NWQIP model

The NWQIP multivariate prediction model we describe here is based on a contemporary PCI series from a geographically defined area with four hospitals. The time period covered incorporates a large proportion of patients with the latest technologies in PCI with a high use of stents and glycoprotein inhibitors. Although the number of drug eluting stents used in this series will be low, we do not expect this new technology to alter MACE rates significantly.

Several strong risk factors for in-hospital MACE after PCI have been identified. The multivariate prediction equation can be easily programmed into appropriate software resident on desktops and handheld computers and may be useful for patient consent and quality improvement initiatives. The NWQIP tool will allow for calculation of the expected risk of MACE based on the presence of risk factors. It will permit meaningful comparisons of performance between hospitals and operators who undertake high risk interventions and more conservative operators and hospitals who are less willing, or do not have the opportunity, to undertake complex PCI.

The dataset needed to calculate the predicted risk of MACE is small and feasible to collect. There is clear potential for inclusion of these variables in routine datasets collected in each hospital provider of PCI; indeed, these factors are already specified in the new CCAD dataset for PCI.⁶ A comparison of risk adjusted hospital and operator specific outcomes will therefore be possible not just across the north west of England but across the entire country.

A cautionary note on such risk adjusted analyses is needed, however. Narins and associates²⁵ recently published results from a questionnaire looking at the influence of public reporting of outcome data on decision making among interventional cardiologists in New York, which showed that 83% believed that such reporting influenced an operator’s willingness to take on high risk cases. Also, 85% felt that the risk adjustment model used in New York was not sufficient to protect those operators willing to take on high risk cases. Therefore, careful consideration is still needed when deciding on how best to apply risk adjustment models and public reporting of outcomes.

Limitations of the NWQIP model

Several limitations of this model need to be considered. Although the data used to develop the model were locally validated and have the confidence of clinicians, they were not externally validated. Another limitation is that our data contained only 129 (1.3%) MACE outcomes and may not be sufficient for an accurate prediction in some of the smaller patient subgroups. Furthermore, the low MACE rate observed within PCI and the high sample size thus required to reach significance will result in limited application of the model at the operator level. Data on ejection fraction may also be a limitation, as we had sufficient data to examine only ejection fractions less than 50% with MACE; however, a lower cut off of 30% might have reached significance. A further limitation may be the low rate of preoperative renal dysfunction in our sample and that a larger number of patients presenting with this co-morbidity may have achieved significance. Also, although our study incorporates four outcomes after PCI (death, emergency CABG, Q wave MI, and cerebrovascular accidents), it is important to point out that other important outcomes also occur after PCI, which we have not examined, such as vascular complications and contrast induced nephropathy.^26,27 And, lastly, the performance of the model to predict MACE outside of the north west of England is still to be tested and a process of external validation is necessary to check the validity of the model across the UK.

Conclusions

We have produced a multivariate prediction model for MACE after PCI and recommend use of this tool when comparing hospitals and operators. We caution against performing crude unadjusted analyses. The Freedom of Information Act will potentially lead to all procedures in the NHS being under intense public scrutiny and, therefore, it is essential that methods of adjusting for differences in risk be developed not just in cardiac interventions but all areas, for example, lung resections.