External validation of novel clinical likelihood models to predict obstructive coronary artery disease and prognosis

Patients and study design

This study included data from the SCOT-HEART trial.8 9 Patients were randomly allocated to either (1) standard care or (2) standard care with additional CACS and coronary CT angiography (CTA) (CTA arm). Similar to a previous study that proposes a novel exercise electrocardiography-weighted CL model,10 the current validation study was restricted to patients without known CAD allocated to the CTA arm (n=1796) with CACS and CTA data available (final population; n=1585).

In short, the proportion of patient classified to very low (≤5%), low (5–15) or moderate/high (>15%) CL categories were compared between the ESC-PTP model, the RF-CL model and the CACS-CL models, respectively. Second, calibration and discrimination were investigated against obstructive CAD defined by coronary CTA and revascularisation within 120 days following coronary CTA. Finally, prognosis was investigated according to CL classification.

Definitions of variables

The ESC-PTP variables included sex, age and type of chest symptoms at referral, the latter further categorised as either typical, atypical and non-anginal chest pain. Typical chest pain covered (1) discomfort in the chest, neck, jaw, shoulder or arm of constricting character; (2) symptoms provoked by exertion or emotional stress; and (3) symptoms relieved by rest or nitroglycerine. Atypical chest pain was defined as meeting two out of three criteria for typical chest pain. Non-anginal chest pain encompassed all other chest pain symptoms.

Risk factors included in the RF-CL were family history of ischaemic heart disease, smoking, dyslipidaemia, hypertension and diabetes. The definition of family history included first-degree relatives with early signs of ischaemic heart disease, men <55 years of age and women <65 years of age. Smoking was defined as current smoker or a history of smoking. Dyslipidaemia, hypertension and diabetes were defined as either diagnosed by a physician or if receiving medical treatment for these conditions. Additionally, information was collected from the national registers for diagnosis codes or registered prescribed medicine for the specific conditions. CACS was calculated based on non-contrast enhanced CT scan using the Agatston method.

Calculation of CL models

The ECT-PTP model was calculated from sex, age and type of chest pain as recommended in the 2019 ESC guidelines on chronic coronary syndrome.1 The RF-CL was additionally calculated from the number of risk factors ranging from 0 to 5 (see above),4 and the CACS-CL incorporated CACS to the RF-CL model.4

All models were divided into groups of very-low (≤5%), low (>5%–15%) and moderate-high (>15%) likelihood of obstructive CAD.

Reference standards and clinical endpoint

Obstructive CAD was defined as >70% diameter stenosis in a major epicardial vessel on CTA as previously reported.8 Second, revascularisation within 120 days after coronary CTA was used as reference standard.

The clinical endpoint was non-fatal myocardial infarction and cardiovascular death at 5 years.

Statistical analyses

Variables are expressed as mean±SD or median (range), categorical variables reported as frequencies (percentages). The RF-CL and CACS-CL models were compared with the ESC-PTP model.

Discrimination was assessed using the area under the receiver operating characteristic curve (ROC-AUC), and AUCs were compared using the DeLong algorithm. Additionally, diagnostic performance was evaluated by sensitivity, specificity, positive and negative predictive values (NPV) using a ≤5% CL cut-off. Comparison of sensitivities and specificities was tested using McNemar’s test and a weighted generalised score statistics for comparison of predictive values of diagnostic tests.

For time-to-event analyses, Cox-regression assumptions were met and HRs were calculated. The cumulative incidence of the primary outcome, non-fatal myocardial infarction and cardiovascular death, was estimated using the Aalen-Johansen method with non-cardiovascular death as competing risk, while the Kaplan-Meier method was used for non-fatal myocardial infarction and all-cause death.

Calibration of the CL models

Calibration of the ESC-PTP, RF-CL and CACS-CL models against obstructive CAD on coronary CTA is shown in figure 1A. Overall, all models underestimated the probability of obstructive CAD defined by coronary CTA which was more pronounced for the RF-CL and CACS-CL models compared with the ESC-PTP model.

Request permission

Calibration of the ESC-PTP, RF-CL and CACS-CL models against revascularisation within 120 days following coronary CTA is shown in figure 2A. Overall, all models overestimated the probability of revascularisation, with the RF-CL model showing the smallest degree of revascularisation probability overestimation. In general, all models showed increasing probability overestimation with increasing CL.

Request permission

Discrimination of the CL models

ROC-AUC curves for the discrimination of obstructive CAD by the ESC-PTP, RF-CL and CACS-CL models are shown in figure 1B. Compared with the ESC-PTP model, both the RF-CL and CACS-CL models showed superior discrimination (AUC 0.78 (0.76 to 0.81) vs 0.80 (0.77 to 0.82), p<0.01, and 0.89 (0.87 to 0.90), p<0.001, respectively). Compared with the RF-CL model, the CACS-CL model showed improved discrimination (p<0.001).

ROC-AUC curves for the discrimination of revascularisation within 120 days following coronary CTA by the ESC-PTP, RF-CL and CACS-CL models are shown in figure 2B. Compared with the ESC-PTP model, both the RF-CL and CACS-CL models showed superior discrimination (0.79 (0.76 to 0.83) vs 0.82 (0.78 to 0.86), p<0.05, and 0.86 (0.83 to 0.89), p<0.001, respectively). Compared with the RF-CL model, the CACS-CL model showed improved discrimination (p<0.001).

Reclassification of the CL models

Patient distribution and obstructive CAD prevalence at coronary CTA for the ESC-PTP, RF-CL and CACS-CL models are shown in figure 1C and online supplemental table 1). Compared with the ESC-PTP model, both the RF-CL and CACS-CL models classified more patients to a very-low CL group (CL ≤5%) with preserved prevalences of obstructive CAD. Using a ≤5% cut-off for obstructive CAD rule-out, both the RF-CL and CACS-CL models showed lower sensitivities compared with the ESC-PTP model (figure 1D). Overall, the RF-CL and CACS-CL models showed higher specificities compared with the ESC-PTP model with all models reporting high NPVs between 93% and 97%.

Patient distribution and prevalence of revascularisation within 120 days following coronary CTA for the ESC-PTP, RF-CL and CACS-CL models are shown in figure 2C and online supplemental table 1). Compared with the ESC-PTP model, both the RF-CL and CACS-CL models classified more patients to a very-low CL group (CL ≤5%) with preserved low prevalences of revascularisation. Using a ≤5% CL cut-off for revascularisation rule-out, the RF-CL model showed lower sensitivity compared with the ESC-PTP model while the CACS-CL model had similar sensitivity (figure 2D). Overall, the RF-CL and CACS-CL models showed higher specificities compared with the ESC-PTP model with all models reporting high NPVs between 99% and 100%.

CL models and prognosis

The clinical endpoint of non-fatal myocardial infarction and cardiovascular death occurred in 17/1585 (1.1%) patients. Overall, event rates increased with increasing ESC-PTP, RF-CL and CACS-CL (table 2 and figure 3). Importantly, patients with very-low CL across all CL models had very low event rates ≤0.1%, and no event rate differences were observed for patients categorised with very-low, low or intermediate/high CL between the ESC-PTP, RF-CL and CACS-CL models, respectively.

Request permission

Comparing the models by Harrell’s C and Somers’ D, the RF-CL and CACS-CL models were superior for prediction of the clinical endpoint compared with the ESC-PTP model (online supplemental table 2).

CL estimation in chronic coronary syndrome

Recognised as a gap in evidence by the 2019 ESC guidelines on chronic coronary syndrome,1 the RF-CL and CACS-CL models were developed as tabulated, simple and clinically useful tools for improved prediction of obstructive CAD in patients with de novo chest pain.4 Other models have been proposed, including the CAD consortium (CADC) clinical and CADC+CACS models11 but their clinical utility is limited by the need for online calculation.

In general, both the RF-CL and CACS-CL models improved discrimination of obstructive CAD compared with the currently recommended ESC-PTP model (figure 1B).1 Importantly, the RF-CL and CADC-CL models classified more patients (41.4% and 52.2%, respectively) at very-low CL of obstructive CAD (≤5%), where neither European nor North-American guidelines recommend downstream testing.1 2 Against a reference standard of obstructive CAD, sensitivity was slightly impaired for the RF-CL and CACS-CL models compared with the ESC-PTP model but across all CL tables, the NPVs remained high (93% to 97%) and obstructive CAD prevalences similar (figure 1C,D). In alignment with the original derivation study,4 the CACS-CL model also reduced the number of patients with low (5%–15%) CL of obstructive CAD in whom diagnostic testing can be considered without clear guideline recommendations.1

In patients with moderate/severe myocardial ischemia but without high-risk CAD by coronary CTA, the International Study of Comparative Health Effectiveness With Medical and Invasive Approaches authors reported similar prognosis but improved quality of life in patients undergoing revascularisation compared with patients allocated to guideline-directed medical therapy.12 13 Thus, one could argue that prognostic CAD should be ruled out by coronary CTA, and referral for invasive coronary angiography (ICA) and potentially revascularisation should be restricted to patients with medically refractory symptoms. CACS is known to be a very strong predictor of obstructive CAD,14 and discrimination of obstructive coronary lesions improves by the CACS-CL compared with models without CACS utilisation (figure 1).4 5 In general, CACS only represents a surrogate for calcified coronary arteriosclerosis and is not a diagnostic test, and the utility of CACS as a tool to stratify patients is debatable.

Definition of obstructive CAD

In our study, the ESC-PTP underestimated the observed prevalence of obstructive CAD at coronary CTA less than the RF-CL and CACS-CL models (figure 1A). However, model calibration improved across all risk tables against revascularisation (figure 2A). Based on the 2019 ESC guidelines on chronic coronary syndrome, the ESC-PTP model reflects the probability of obstructive CAD in a pooled analysis from three studies: the PROMISE trial (n=4415),15 the CONFIRM registry (n=8106)16 and a study by Reeh et al (n=3291).17 In the PROMISE and CONFIRM populations, coronary CTA was used as reference standard of obstructive CAD, while Reeh et al used a mixed endpoint of either non-invasive or invasive tests. As (1) the RF-CL and CACS-CL models were derived against ICA as reference for obstructive CAD,4 and (2) coronary CTA overestimates lesion severity by invasive measures,18 the improved calibration of the RF-CL and CACS-CL models against revascularisation compared with obstructive CAD at coronary CTA is expected. Importantly, the calibration of CL models is impacted by the chosen reference standard where CL estimates contribute to PTP assessment after advanced non-invasive diagnostic testing.3 As abnormal non-invasive tests guide potential revascularisation, CL models should be calibrated against ICA, alternatively inducible myocardial ischaemia19 and ultimately revascularisation.4 20

Prognosis in chronic coronary syndrome

In our study, the CACS-CL model was superior for the prediction of the clinical endpoint compared with the ESC-PTP (online supplemental table 2). Overall, all CL models found patients with very-low CL to have a 5-year absolute risk <1.0% of myocardial infarction and cardiovascular death (figure 3). Importantly, the ESC-PTP, RF-CL and CACS-CL models are all calibrated against obstructive CAD whereas myocardial infarction also occurs in non-obstructive lesions in 1 of 10 patients.21 Patients with non-obstructive lesions are presumably missed by models not integrating CACS though CACS only surrogates coronary atherosclerosis by outlining coronary calcification. However, event rates of the clinical endpoint were similar for low and moderate/high CL patients by the CACS-CL model (table 2).

Overall, our findings are consistent with previous studies, and the improved very-low CL categorisation by the RF-CL and CACS-CL models seem safe across various study cohorts.6

Limitations

Our cohort consisted exclusively of Caucasian patients, and results should be extrapolated carefully. All patients had chest pain but dyspnoea was not recorded in SCOT-HEART and CL estimates are solely based on chest pain typicality. Our cohort underwent coronary CTA which improves prognosis by early initiation of guideline-directed medical therapy and this could impact subsequent cardiac outcomes.9