Introduction

Simulation-based training is being increasingly deployed for both technical and non-technical skill acquisition in healthcare with the aim of reducing medical error and patient harm. There is a need for an evidence-based approach to such training to ensure that the resources used can reliably deliver a quantifiable improved skill set rather than just an enhanced educational experience. Proficiency-based progression (PBP) training is a form of outcomes-based training that involves training individuals to achieve a proficiency benchmark. The process involves ‘deliberate’ practice against a set of clearly defined objective metrics. The proficiency benchmark is set as the mean performance of clinicians who undertake the procedure regularly in clinical practice. It has been shown to improve the performance of individuals undertaking technical procedures.1–7 Metrics are operationally defined to facilitate objective scoring. For example, in the study by Cates et al demonstrating improved performance of carotid angiography, predefined metric errors include ‘number of diagnostic catheters used to obtain diagnostic pictures’ and ‘catheter advancing without a guide-wire in front of it’.6 Despite these results, PBP methodology has not previously been applied to simulation-based training for non-technical skills yet communication failures are a significant source of medical error and preventable adverse events equal if not greater than errors due to lack of technical skill.8–10 Escalation of care for an acutely deteriorating patient demands the most efficient, concise and accurate flow of information among healthcare workers of different disciplines for the best outcome to be achieved.

Early Warning Scores facilitate early detection of deterioration by categorising a patient’s severity of illness and prompting escalation of care at specific trigger points using a structured communication tool such as ISBAR (Identification, Situation, Background, Assessment, Recommendation). This enables a more timely response using a common language.11 Ireland was one of the first countries to agree and implement a standardised Early Warning Score (National Early Warning Score (NEWS)) across the entire acute hospital sector. NEWS uses the ISBAR tool as the recommended structured communication tool for the acutely deteriorating patient.12 13 The NEWSe-learning programme is recommended as the interdisciplinary education programme for healthcare professionals working in acute services in Ireland. The programme teaches ISBAR as the standardised tool to escalate care in the context of the acutely deteriorating patient.

The primary aim of this study was to determine if the addition of a PBP simulation training programme to the national NEWS e-learning module results in better performance of clinical communication of a deteriorating patient than either the e-learning module alone or in combination with standard simulation.

Methods

Participants

Eligible participants were 109 third year nursing and 201 final year medical students who were scheduled to undertake interdisciplinary NEWS training in September 2016 as part of their undergraduate curriculum. This comprised the entire undergraduate nursing and medical classes except for 31 medical students who were scheduled to undertake this training at a later time in the curriculum (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Consort diagram outlining selection, allocation and follow-up of undergraduate medical and nursing participants in a study comparing the effect of e-learning alone (E), e-learning plus standard simulation (E+S) and e-learning plus proficiency-based progression simulation (E+PBP) on clinical communication. NEWS, National Early Warning Score.

Interventions

All third year nursing and final year medical students were emailed prior to training and instructed to undertake the NEWS e-learning programme. Written informed consent was obtained from all participants. On the day of training, participants were required to submit a certificate of successful completion of the e-learning programme. A 15 min lecture on the ISBAR tool was delivered before participants undertook training as per their allocated groups. Students were not notified as to which study group they were allocated. The study flow is outlined in figure 2.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Outline of experimental design and study flow indicating training interventions and assessment of the three study training groups (E, E+S, E+PBP) of undergraduate medical and nursing participants. ISBAR, Identification, Situation, Background, Assessment, Recommendation; NEWS, National Early Warning Score.

The three training groups were as follows:

e-learning only group (E)

Participants in this group proceeded immediately following the 15 min lecture to the high-fidelity suite for performance assessment. After outcome assessment was complete, participants undertook simulation training similar to the E+S group as outlined below in order to ensure that all students were afforded the same training opportunity from a curriculum perspective.

e-learning plus standard simulation group (E+S)

Participants worked in pairs of a medical student and nursing student. If a participant did not have a partner, then a non-study peer student was asked to pair with that individual for the purposes of training. Data from the non-study student were not included in the analysis.

Training consisted of a series of simulated phone calls using four standardised paper cases for each discipline. Case materials included case notes, NEWS charts and a blank ISBAR template indicating the categories and type of information that should be communicated. Each scenario had a deteriorating patient event that necessitated an ISBAR telephone communication. Participants alternated between making and receiving simulated phone calls. A standardised script was given to the recipient. Two facilitators conducted the simulation training. Both facilitators were experienced clinicians and educators who had previously undergone the ‘Train the Trainer NEWS programme’ and regularly facilitate NEWS training and healthcare simulation. The facilitators offered support and feedback in line with standard NEWS training by listening to simulated phone calls and offering guidance on the ISBAR framework and by answering questions as they arose. Participants were required to work through all four cases with their partner. Towards the end of the training session, the participants presented to the facilitator to repeat a simulated phone call for either case 3 or 4. The training session was 3.5 hours in duration, and the participants were required to stay until the end of the training regardless of progress. If an individual had completed all the cases, they were asked to assist by continuing to be the recipient of phone calls for their partner or by continuing to practice by repeating the cases if required.

e-learning plus proficiency-based progression simulation group (E+PBP)

Participants underwent a training programme of the same structure, duration (3.5 hours), content and facilitator: student ratio as the E+S group. The same two facilitators facilitated both the E+S and the E+PBP training. However, in the E+PBP group, partners scored each other’s phone calls during training against a series of predefined metrics (quantified as steps, errors and critical errors for each case) on a score sheet to ascertain if the proficiency benchmark for that case was reached. Partners shared the results of the metrics and proficiency scores with each other as feedback at the end of each simulated phone call. If proficiency was not achieved, the case was repeated before progressing to the next case. Participants were required to reach proficiency on all four cases with their partner before performing case 3 or 4 with the facilitator and demonstrating proficiency again. If proficiency was not achieved with the facilitator, then the participant returned to repeat cases with their partner and present for reassessment to the facilitator until proficiency was demonstrated. The training session was 3.5 hours in duration, and the participants were required to stay until the end of the training regardless of progress. If an individual had completed all the cases, they were asked to assist by continuing to be the recipient of phone calls for their partner or by continuing to practice by repeating the cases if required.

Outcomes

The primary outcome was the ability to reach the proficiency benchmark on the standardised high-fidelity simulation assessment case. The secondary outcomes were the number of successfully completed steps, errors and critical errors performed by each group.

Performance metrics were developed for the training cases and for the high-fidelity simulation assessment case as part of a pilot study in the previous year. Each case presented a different but commonly encountered clinical scenario of an acutely deteriorating patient. As an example, the outline of the nursing component of the high-fidelity simulation assessment case is shown in figure 3.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Outline of the high-fidelity simulation performance assessment case for nursing undergraduates.

The metrics were derived for each of the training and assessment cases according to the five components of the ISBAR tool and were specific to each case.

The performance metrics were validated through a modified Delphi expert panel consisting of nine senior nurses and eight medical staff who regularly facilitate NEWS/ISBAR communication training. Delphi panel members reviewed the performance metric for each of the simulation cases, and the high-fidelity performance outcome case and metric units were included, excluded or modified by consensus. Each metric unit was then classified as a step, error or critical error by consensus. The majority of metrics were common to both medicine and nursing. The number of metrics per case ranged from 24 to 26.

The proficiency benchmark was set as the mean performance of qualified personnel from the respective disciplines on each case. Nine nursing and five medically qualified practitioners (who regularly escalate care in the acute healthcare setting and with a mean years of experience=3 years) underwent the high-fidelity simulation case. The proficiency benchmark for the assessment case was set as the mean performance for each discipline as scored by two independent assessors using the predefined metrics. An extract from the metric scoring sheet and proficiency benchmark for the high-fidelity simulation assessment case is shown in figure 4.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Extract from the nursing metric scoring sheet illustrating some of the metrics and the proficiency benchmark for the high-fidelity simulation assessment case.

Digital recordings of each participant’s performance of the standardised case in the high-fidelity assessment suite were reviewed and scored by two independent assessors (experienced acute care nurses) using the predefined metrics and proficiency benchmark.

The assessors underwent training on scoring the material using 10 recordings of the same case obtained from non-study participants. Assessment of the digital recordings was undertaken within 2 months of study participation. An inter-rater reliability of >85% was achieved prior to commencing scoring study material. The assessors were not part of the investigator group, were blinded to the study group allocations and had no prior knowledge of any of the participants.

Results

Baseline characteristics with respect to age, gender, discipline, nationality and first language of the participants in each group are shown in table 1.

Figure 5 shows percentages of participants in each group who demonstrated the proficiency benchmark following assessment in the high-fidelity simulation suite. At the end of training, 6.9% (2/29) of the e-learning only (E) group and 13% (3/23) of the standard simulation (E+S) group demonstrated proficiency. In comparison, 60% (15/25) of proficiency-based progression simulation (E+PBP) group were proficient. The difference between the E group and the E+S group was not statistically significant (χ²=0.55, 99% CI 0.63 to 0.66, p=0.63) but was significant for the difference between the E group and the E+PBP group (χ²=22.25, CI 0.00 to 0.00, p<0.000) and between the E+S group and the E+PBP group (χ²=11.04, CI 0.00 to 0.00, p=0.001).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

The percentages reaching the proficiency benchmark at the end of training of the three study training groups: e-learning alone (E), e-learning plus standard simulation training (E+S) and e-learning plus proficiency-based progression simulation training (E+PBP).

On logistic regression analysis (figure 6), it was found that in comparison to the E group, the E+PBP trained group were more than 20 times as likely to demonstrate proficiency and the difference was statistically significant (Ext (B)=20.25, 95% CI 3.91 to 105, p<0.000).

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

Logistic regression analysis for the relative differences between the three study training groups of undergraduate medical and nursing participants: e-learning alone (E), e-learning plus standard simulation training (E+S) and e-learning plus proficiency-based progression simulation training (E +PBP).

The E+PBP group completed significantly more steps, mean 8.5 (1.7), than either the E group, mean 5.8 (1.6), p<0.000, or the E+S group, mean 6.3 (2.1), p<0.000. Similarly, combined errors and critical errors were significantly less in the E+PBP group, mean 3.7 (1.6), than either the E group, mean 5.9 (2.1), p<0.000, or the E+S group, mean 5.2 (1.5), p<0.01. Inter-rater reliability of the two assessors was 97%.

Discussion

Our results show that addition of a PBP simulation programme to an e-learning module can deliver a superior set of skills for ISBAR communication in relation to a deteriorating patient than an e-learning module either alone or in combination with standard simulation. Furthermore, this benefit is seen within the same resources, that is, materials, timeframe and facilitators as standard simulation. The Irish health service like its international counterparts has prioritised clinical communication as a key part of the patient safety agenda.12–16 Clinical communication is now viewed as an essential skill and training is recommended as mandatory for all health and social care professionals.13 All participants were required to produce a recent certificate of successful completion of the e-learning programme but only 6.9% of the group who undertook the e-learning module only demonstrated the proficiency benchmark. The addition of standard simulation did not significantly improve performance with only 13% of the E+S group reaching the benchmark.

It could be argued that exposure to metrics-based scoring in the practice cases resulted in better performance in the assessment case for the E+PBP group. However, this is precisely the desired effect, that is, that trainees know what skills need to be achieved, practice to achieve them to an objective predefined standard and transfer that training to a dynamic scenario. The E+S and E+PBP groups differed in only two respects: (i) practice was ‘repeated’ in the E+S cohort as opposed to ‘deliberate’ in E+PBP cohort, that is, focused on pre-defined metrics and (ii) the E+PBP group was required to reach proficiency benchmarks to progress through simulation cases whereas the E+S group were not. Our results demonstrate that proficiency-based training can achieve skill acquisition rates of the order of 60%, similar to those seen with technical skills using this approach. In a study of similar experimental design, Angelo et al found that there were 56% fewer intraoperative errors and 69% fewer critical errors when compared with traditional training.2 To our knowledge, our study is the first randomised trial of PBP training of a non-technical skill.

The main strength of the study is the use of robust methodology to determine the effectiveness of an educational intervention on objectively assessed performance outcomes. The study combines the rigour of a randomised controlled trial with that of an outcomes based-training approach (PBP) to clinical communication. A significant body of evidence already exists in relation to the use of PBP for technical skill acquisition.7–13 Our results support the use of PBP training for communication skills also.

Weaknesses of the study include the single-centre design and the application to the undergraduate population only, although the training programme was designed for qualified nurses and doctors also. Since the completion of study, the programme has been applied successfully to both nursing and medical undergraduate programmes in the university setting and to doctors in training in the hospital setting. There is a need for future research on the application of the programme in different clinical settings and its impact on patient outcomes.

The study was limited by the restriction on training time. The duration of simulation training for both E+S and E+PBP groups was extended to 3.5 hours from the initial pilot (1.5 hours), but was still restricted by the existing undergraduate curriculum rather than that which would ideally be required to train a fundamental skill. Skills consolidation is an important part of the learning process particularly for new skills.17 In the study by Angelo et al,2 trainees had a weekend in which to acquire, refine and consolidate their skills before their proficiency assessment at the end of training. Another difficulty, which may have impinged on the effectiveness of training, was the disparity in fidelity between the paper-based training environment and the assessment undertaken in the high fidelity simulation environment. This disparity is challenging for those with limited clinical experience such as the undergraduate population. Van Sickle et al 3 and Gallagher and O’Sullivan4 have commented on the detrimental impact that this disparity can have on proficiency demonstration by trainees.

It is now widely recognised that clinical communication skills underpin patient safety. Implementation of a training programme in relation to clinical communication has already been shown to reduce medical error and preventable adverse events.18 There is a need for valid, reliable, cost-efficient clinical communication training programmes to address this need and the impact on patient as well as healthcare provider outcomes.

In summary, our study shows that PBP is a more effective way to teach clinical communication for the deteriorating patient than e-learning either alone or in combination with standard simulation. Furthermore, improved performance with PBP simulation was achieved with the same training time and facilitator/student ratio as standard simulation.