Article Text
PDF
Abstract
Objective To assess the accuracy of semi-automatic assisted telephone triage in patients with acute chest discomfort against the diagnosis of acute coronary syndrome (ACS) or other life-threatening events (LTEs).
Methods A cross-sectional study was performed of telephone conversations with 2023 patients with acute chest discomfort (pain, pressure, tightness or discomfort) who called out-of-hours services for primary care (OHS-PC) between 2014 and 2016. Sensitivity, specificity, positive and negative predicted values were calculated for a high urgency (patient seen within one hour) against the diagnoses of ACS and other LTEs. Diagnoses were retrieved from the patients' medical records in general practice, including hospital specialists' discharge letters.
Results Of 2023 patients who called because of chest discomfort, 227 (11.2%) had an ACS (men 14.9%, women 8.2%) and 58 (2.9%) had another LTE (men 3.6%, women 2.3%). The sensitivity and specificity of a high Netherlands Triage System (NTS) urgency allocation against ACS/other LTEs were 0.73 (95% CI 0.68 to 0.78) and 0.43 (95% CI 0.40 to 0.45), respectively. In 13.2% of the calls the triage nurse overruled the NTS urgency, mostly by upscaling (11.0%). The sensitivity and specificity of the final urgency allocation were 0.86 (95% CI 0.81 to 0.90) and 0.34 (95% CI 0.32 to 0.37). The positive and negative predictive values of the final urgency were 0.18 (95% CI 0.17 to 0.19) and 0.94 (95% CI 0.92 to 0.95), respectively.
Conclusions The semi-automatic triage NTS tool underestimated the urgency in 27% of patients with ACS/other LTEs. Overruling by triage nurses improved safety, but still 14% of men and women with ACS/other LTEs received too low urgency, while efficiency remained poor.
Trial registration number NTR7331.
- general practice
- acute coronary syndrome
- emergency medicine
- primary care
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key questions
What is already known about this topic?
The accuracy of semi-automatic assisted telephone triage with the Netherlands Triage System (NTS) in out-of-hours primary care (OHS-PC) has never been validated against clinical outcomes.
Two-thirds of all ambulance dispatches from OHS-PC are because of chest discomfort.
Around 10% of patients with chest discomfort referred to the hospital actually have an acute coronary syndrome (ACS).
What does this study add?
Primary care semi-automatic assisted telephone triage of patients with chest discomfort is not safe or efficient.
The results were similar for men and women.
Intervening by the triage nurse improved safety.
How might this impact on clinical practice?
‘Blind trust’ in decision support systems should be replaced by critical use of it.
Further research into symptom presentation and how to ask for symptoms in the OHS-PC could help improve semi-automatic assisted triage tools for patients who call with chest discomfort.
Introduction
Adequate triage followed by early diagnosis is vital in patients with acute chest discomfort. They might have an acute coronary syndrome (ACS) or other life-threatening event (LTE) such as pulmonary embolism. Telephone triage of such patients as done in out-of-hours services primary care (OHS-PC) and emergency medical services (EMS or ambulance dispatch centres) is, however, challenging because, based on symptoms only, it is difficult to differentiate ACS/other LTEs from other causes of chest pain.1 In the Netherlands, 80% of people with chest discomfort first call the general practitioner (GP) or OHS-PC and 20% directly call EMS (call 112) for an ambulance or are self-referrals to the emergency department (ED).2
In most Western countries, semi-automatic computer triage systems are used to support telephone decision making.3 Triage nurses fill out a standard list of hierarchically ordered questions that pops up after the triage nurse chooses the patient’s main complaint in the computer system, and the computer automatically generates a recommended level of urgency (one out of six urgency levels).3 4 These triage systems have never been validated in the OHS-PC setting against clinical outcomes as the reference.4
Since 2011, most OHS-PC in the Netherlands and half of the EMS use the Netherlands Triage System (NTS).5 In Dutch EDs, both the NTS and the Manchester Triage System (MTS) are used.6 The NTS was developed by an expert panel that integrated knowledge from (1) the MTS, (2) the Dutch national guidelines for general practice (‘NHG triage index’) and (3) the Dutch national standard for ambulance dispatch centres (‘LSMA’).4 The aim of the NTS is similar to other triage systems—that is, to guarantee efficient and safe care.7 The NTS seems safe with 0.006% serious adverse events (SAEs) a year at OHS-PCs (SAE is an unintended or unexpected event resulting in death or a severe harmful event).8 However, most SAEs are of cardiovascular origin (46.2%) and most often missed myocardial infarctions and sudden cardiac death.9 10 More questions are asked about the efficiency of the NTS for patients with chest pain. It is the most common reason for allocating an ambulance within 15 min (U1) to patients calling OHS-PC (60.7% of all dispatches), but only around 10% actually have an ACS .11–13
The aim of our study was to assess the accuracy of telephone triage for patients with chest discomfort who call the OHS-PC with the clinical outcome ACS/other LTE as the reference. Both the accuracy of the NTS tool and the ‘final’ urgency, including overruling by the triage nurses, were evaluated.
Methods
The study design and data collection have been published in our study design paper.14 In short, we performed a cross-sectional study in 2023 patients who contacted one of nine OHS-PCs in the Netherlands with acute chest discomfort (pain, pressure, tightness or discomfort) between 2014 and 2016. We selected calls on the basis of the International Classification of Primary Care (ICPC; a WHO worldwide code system for primary care) codes (K01, K02, K03, K24, K74, K75, K76, K77, K93, L04, P74, R02, R98) and keywords (thoracic pain, chest pain, myocardial infarction, heart attack and their common abbreviations).15 16 We deliberately sampled broadly to capture the entire domain of patients that could be suspected of ACS.
We used electronic health record data of OHS-PC (‘Callmanager’) and re-listened to telephone triage recordings in order to collect information about symptoms, patient and conversation characteristics of patients, and urgency allocation. The final diagnoses were retrieved from the patients’ GP, and these were based on hospital specialist discharge letters or GPs' diagnoses. For the diagnosis of ACS we used the cardiologist’s diagnosis. Patients without raised levels of (high-sensitivity) troponin were classified as unstable angina pectoris (UAP), and those with raised troponin levels as myocardial infarction, divided into STEMI (ST elevation myocardial infarction) and NSTEMI (non-ST elevation myocardial infarction).
Other life-threatening diagnoses were classified by an expert panel of experienced GPs (DZ, FR, RD) based on medical specialist letters (majority of votes).
Context
Triage assessment with the NTS starts with a mandatory ‘ABCD check’ (Airway, Breathing, Circulation and Disability); this check determines whether there are direct life-threatening problems for which an ambulance must be sent straight away.17 When the patient is ‘ABCD-stable’, the triage nurse has to choose one out of 56 main complaints to continue the urgency assessment. Based on the triage nurses’ input, the NTS automatically generates an urgency allocation ranging from U0 (resuscitation) to U5 (telephone self-care advice) (see online supplementary table 1). The triage nurse can overrule this recommendation and up- or downscale the urgency allocation, often after consulting the supervising GP.5 For clarification of the work process in the OHS-PC, see an adapted SEIPS (Systems Engineering Initiative for Patient Safety) framework (online supplementary figure 1).18
Supplemental material
Supplemental material
In the Netherlands there is a strong primary care and GPs take care of 93% of all healthcare problems for 3% of the total healthcare budget.19 The GP has a gatekeeper’s role to hospital care, although people may call EMS (112) in case they suspect a life-threatening situation.19
NTS and final urgency levels
The NTS urgency level is the automatically NTS-generated level of urgency based on the input of the triage nurse. The final urgency level is the ‘overruled’ level of urgency or the NTS urgency level (when not adjusted). In around 15% the final urgency allocation was unclear because of incomplete registration of overruled urgencies. For these cases, an expert panel of three experienced GPs (DZ, FR, IK) independently decided on the final urgency level blinded for the outcome (majority of votes) (see also online supplementary figure 2).
Data analyses
We dichotomised urgency allocation into high (U1 and U2) and low (U3, U4 and U5) urgency levels and calculated the accuracy of the NTS and final urgency for allocating a high level of urgency to patients with the outcome ACS/other LTEs in terms of sensitivity, specificity, positive and negative predictive values. For patient and call characteristics the χ2 test was used for comparison of dichotomous variables and the independent sample t-test or Mann–Whitney U test for continuous variables. We compared baseline characteristics between patients in whom the eventual diagnosis was retrieved and those in whom the GP did not provide these outcomes, to get an impression on whether selection bias could have occurred. Sample size was calculated using Harrell’s rule of thumb, which can be found in our study design paper.14 All data analyses were performed using SPSS statistical software programme Version 25.0.
Patient and public involvement
No patients were involved in setting the research question or the outcome measures, or in developing plans for design; however, they were involved in the implementation of the study. In addition, they were asked to advise on interpretation and writing up of results. The results will be shared and discussed with the national patient community of cardiovascular diseases (‘Harteraad’).
Results
Baseline characteristics
Of the 2023 patients with acute chest discomfort and a retrieved diagnosis, 227 (11.2%) had an ACS (women 8.2%, men 14.9%) and 57 (2.8%) had another LTE (women 2.3%, men 3.5%) (figure 1). The mean (SD) age was 58.9 (19.5) years and 54.9% were women. The NTS allocated 59.7% (men 57.6%, women 61.5%) to high urgency (U1/U2), and this changed to 68.5% (men 68.2%, women 68.7%) for the final urgency. Triage nurses overruled the NTS level of urgency in 13.2% of patients, mostly upscaling (11.0% of all calls) (figure 2).
Flowchart study population.
Overview of NTS urgency allocation and final urgency allocation after overruling.
Patients allocated to a high urgency level were significantly older (mean (SD) 62.7 (18.2) years vs 53.1 (19.9) years, p<0.001), had a shorter mean duration of calls (6:57 min vs 8:30 min, p<0.001) and more often someone else called on behalf of the patient (58.9% vs 38.9%, p<0.001) than those allocated a low urgency level. Nearly all callers expressed concerns (92.1% vs 86.4%, p=0.003).
In 2.1% of the patients labelled as ‘ABCD-unstable’, the NTS urgency allocation was downgraded by the triage nurse/GP. When the triage nurse chose shortness of breath as the main complaint, relatively more patients were given a high urgency level than when chest pain was the main complaint (73.8% vs 56.5%, p<0.001).
Patients classified as a high urgent level more often had acute shortness of breath (71.0% vs 53.9%, p<0.001), chest pain lasting for less than 12 hours (81.3% vs 62.5%, p<0.001), heavier chest pain (Numeric Rating Scale >7, 46.7% vs 22.1%, p<0.001) and sweating (47.7% vs 31.5%, p<0.001). They also more often recognised their symptoms from a previous cardiac event than those who were classified as a low level of urgency (30.9% vs 21.5%, p=0.002). Patients with stabbing pain (33.8% vs 15.0%, p<0.001) or pain located on either the left or right side of the thorax (53.4% vs 28.6%, p<0.001) were more often classified as of low urgency (table 1).
Baseline characteristics of 2023 patients with chest discomfort (chest pain, pressure, tightness, discomfort) calling OHS-PC, divided into high (U1–U2) and low (U3–-5) NTS levels of urgency
There were no clinically meaningful differences in patient or call characteristics between the 2023 patients with a retrieved diagnosis and the 972 patients without a diagnosis (online supplementary table 2).
Diagnoses
Of the 227 patients with an ACS (91 women, 136 men), 67 (29.5%) patients had a STEMI (women 22.1%, men 33.8%), 93 (40.0%) a NSTEMI (women 46.2%, men 37.5%), 52 (22.9%) UAP (women 20.9%, men 24.3%) and 15 (6.6%) unspecified ACS, including three sudden cardiac deaths in women (women 9.9%, men 4.4%) (table 2). In nearly all cases (97.0%) the ACS diagnosis was from the cardiologist and based on symptom presentation, troponin levels and electrocardiography. Three patients died before arrival of the ambulance (they were classified as acute cardiac death) and one patient died after resuscitation at the ED. Three patients were classified as ACS by the GP, while not referred because of short life expectancy due to cancer. Of the 58 patients with other LTEs (2.9%), 13 had a pulmonary embolism, 5 an acute abdominal aneurysm, 5 a thoracic aortic dissection and 35 other diagnoses (eg, diabetic ketoacidosis, sepsis). The majority of patients had non-urgent medical conditions (85.9%) including: non-urgent cardiovascular diseases (18.7%), non-cardiac chest pain (not further specified) (17.3%), musculoskeletal pain (17.9%), psychogenic disorders (11.2%), gastrointestinal disorders (7.1%), respiratory disorders (5.5%) and other non-urgent diagnoses (8.4%).
Diagnosis of 2023 patients who contacted the OHS-PC for chest discomfort (pain, pressure, tightness or discomfort), divided into NTS high urgency (U1–U2) and low urgency (U3–U5)
Relationship between diagnosis and urgency allocation
Of the patients with an ACS, 75.3% (women 74.7%, men 75.7%) were assigned a NTS high urgency level and 88.5% (women 89.0%, men 88.2%) were assigned a final high urgency level. Patients with another LTE were assigned a NTS high level of urgency in 65.5% (women 76.0%, men 57.6%) and a final high level of urgency in 74.1% (women 84.0%, men 66.7%). Of the 41 ACS/other LTE patients who were assigned a final low urgency level, most were categorised as U3 (87.8%) and a minority as U4 (7.3%) or U5 (4.9%) (table 3).
Overview of the NTS and final urgency allocation of 285 patients with acute coronary syndrome or life-threatening events
Patients with non-urgent cardiovascular diseases were more often classified as high urgent (21.3% vs 14.8%, p<0.001) and patients with psychogenic disorder as low urgent (9.6% vs 13.5%, p=0.006).
Accuracy of NTS and final urgency levels
The NTS high urgency had a sensitivity of 0.73 (95% CI 0.68 to 0.78), specificity of 0.43 (95% CI 0.40 to 0.45), positive predictive value of 0.17 (95% CI 0.16 to 0.19) and negative predictive value of 0.91 (95% CI 0.89 to 0.92) for the outcome ACS/other LTE (table 4). For the final urgency level, sensitivity was 0.86 (95% CI 0.81 to 0.90), specificity 0.34 (95% CI 0.32 to 0.37), positive predictive value 0.18 (95% CI 0.17 to 0.19) and negative predictive value 0.94 (95% CI 0.92 to 0.95).
Accuracy outcomes of the NTS and final high urgency allocation (U1/U2) against ACS or other LTEs (ACS/LTE prevalence 14.1%)
Discussion
The safety and efficiency of the Dutch NTS decision support tool were found to be poor for telephone triage at OHS-PCs in patients with acute chest discomfort, with a sensitivity of 0.73 and specificity of 0.43 for high urgency allocation of patients who had an ACS/other LTEs. Safety improved after intervention by the triage nurse and the final high urgency level had a sensitivity of 0.86 and specificity of 0.34.20 Women and men with ACS/other LTEs received equally high urgencies.
At a prevalence of ACS/other LTEs of 14.1% (11.2% and 2.9%, respectively), the positive predictive value for both the NTS and final urgency was low (0.17 and 0.18, respectively) and the negative predictive value was high (0.91 and 0.94, respectively).
Clearly, the triage system is calibrated on the safe side. Most GPs highlight over triage, but in our opinion, in the critical domain of patients with chest discomfort, under triage should receive more attention.5 19 21 It is evident that patients with an ACS and those classified with other LTEs should receive a high urgency allocation, but the NTS classified 27% of the ACS/LTE patients as low urgency, and this was still rather high for the final urgency (14%).
We were able to analyse the original and very first conversation with patients, and these conversations were assessed without knowledge of the diagnosis; the assessment of symptoms was not affected by recall or hindsight bias. Moreover, we were able to include a large sample of patients and missing outcome data did not lead to patient or call selection. We therefore are confident our results are generalizable to Dutch settings and also to similar OHS-PC settings in, for example, the UK and Scandinavian countries.1 Our results may also be generalizable to EMS settings, because the chance of having an ACS among those calling for chest discomfort is similar in the EMS setting and the OHS-PC setting.13 22
A limitation of the study is that, in 15% of calls, the overruled urgency was unclear. In these cases, an expert panel (blinded to the clinical outcome) decided on all the available information to be heard in the calls.
We could not find studies evaluating the accuracy of telephone triage in the OHS-PC among patients with chest discomfort against clinical outcomes.
A systematic review of six studies evaluating the MTS in the ED setting reported that the MTS had allocated a high urgency level in patients with ACS with a sensitivity of 0.70–0.80.23 One of these six studies also reported specificity (0.59) among patients with chest pain referred to a Portuguese ED.24 Thus, the MTS, with the advantage of face-to-face contact, seems to have a similar sensitivity and somewhat higher specificity to that in our study. Interestingly, the prevalence of ACS among the 233 patients with chest pain in the Portuguese study was lower than in our study (9.4% vs 11.2%).24 None of the studies reported positive and negative predictive values, which prevented us from comparing these to ours. The prior risk of ACS and LTEs of 14.1% changed only marginally to 17% and 18% for the NTS and final high urgency, respectively, and to 9% and 6% for the NTS and final low urgency allocation.
To the best of out knowledge, there are no accuracy studies performed in the EMS setting in which urgency allocation with EMS (112 or 911) telephone triage was compared with the presence or absence of ACS. In an observational Swedish study among 14 454 patients calling EMS for chest discomfort, 63.5% received an urgency 1 ambulance and 11.5% turned out to have an ACS (women 9% and men 14%).22 An EMS study in the USA with 3007 calls from patients with acute chest pain to the Medical Priority Dispatch System (MPDS) for decision support reported a prevalence of acute myocardial infarction of 8.7%.25
In a primary care study, the GP’s clinical risk estimate was more accurate in predicting ACS than a prediction rule (c-statistic 0.75 vs 0.66) in patients with acute chest discomfort.20 This is in line with our finding that the sensitivity of the final urgency level (including overruled cases) was higher than that of the NTS level of urgency alone (0.86 vs 0.73).
Previous studies have found that the risk of ACS in women is underestimated, a finding in contrast to our result in which women and men with ACS received similarly high urgency allocations.26 27 One of the former mentioned EMS studies also found that women and men received similarly high urgency allocations and the other EMS study did not report sex-stratified analyses.22 25
Missing an ACS is the most common reason for malpractice claims worldwide.9 28 There are no generally accepted guidelines that define what rate of missed ACS is considered acceptable, and this threshold will likely differ among different healthcare providers. A survey performed among 1029 ED doctors in the USA, New Zealand and Australia showed that they considered on average a rate of 0.1%–1% (range 0%–10%) as acceptable.29 We do not have such information from GPs or lay people, but we suspect that this will be similar in the Netherlands. Alarmingly, this number is not in line with our finding that 27% of patients with ACS/other LTEs received a NTS low urgency allocation and a 14% final low urgency allocation. Indeed, a low urgency allocation does not necessarily mean the ACS is being missed, but that an undesirable risk to patient safety has been taken with the hazard of life-threatening ventricular arrhythmias and sudden death due to severe pumping failure of the heart in those with acute myocardial infarction.30 31 On the other hand, triage nurses and their supervising GPs increased triage safety by overruling the decision support system adequately, although with significantly increasing over triage. Apparently, the ‘human factor’ is conducive to safety in the current telephone triage process.18 32 Nonetheless, potential room for improvement should be studied, both of the decision support tool itself by developing better diagnostic prediction rules as well as by improving its use by triage nurses and management.33
Conclusion
The semi-automatic triage NTS tool underestimated the level of urgency in 27% of patients with ACS/other LTEs. Overruling by triage nurses improved safety, but still 14% of men and women with ACS/other LTEs were allocated too low a level of urgency while efficiency remained poor.
Acknowledgments
The authors would like to thank the OHS-PC foundation ‘Primair Huisartsenposten’ and all employees of the participating locations for their cooperation in this study, notably for providing data and technical support. We thank general practitioner I Kagenaar for her contribution to the expert panel.
References
Supplementary materials
Press release
Footnotes
Funding This study was funded by an unrestricted grant from (i) the Department of General Practice of the University Medical Centre Utrecht, (ii) a personal promotion grant to DZ, (iii) the The Netherlands Triage Standard Foundation and (iv) the Stoffels-Hornstra Foundation.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The Medical Ethics Committee Utrecht, the Netherlands (reference number WAG/mb/16/003208; National Trial Register identification number: NTR7331) approved this study. In addition, a waiver of informed consent was given as the study involves no more than minimal risk to subjects and this study could not be carried out practicably without the waiver. All personal and research data were handled and stored according to the European General Data Protection Regulation.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. The data can be made available for researchers whose proposed use of the data has been approved at request of the corresponding author, with a signed data access agreement.