You are here

Article Text

Article menu
Interventional cardiology
Original research
The BE-ALIVE score: assessing 30-day mortality risk in patients presenting with acute coronary syndromes
  1. Alexander Tindale1,2 and
  2. Vasileios Panoulas1,2
  1. 1National Heart and Lung Institute, Imperial College London, London, UK
  2. 2Department of Cardiology, Harefield Hospital, Guy's and St Thomas' NHS Foundation Trust, London, UK
  1. Correspondence to Dr Alexander Tindale; alexander.tindale09{at}imperial.ac.uk

Abstract

Aim To create and validate a simple scoring system for predicting 30-day mortality in patients presenting with acute coronary syndromes (ACS) at their moment of admission.

Methods and results 2407 consecutive patients presenting to Harefield Hospital with measured arterial blood gases, from January 2011 to December 2020, were studied to build the training set. 30-day mortality in this group was 17.2%. A scoring algorithm that was built using binary logistic regression of variables available on admission was then converted to an additive risk score. The resultant scoring system is the BE-ALIVE score, which incorporates the following factors:

Base Excess (1 point for <−2 mmol/L), Age (<65 years: 0 points, 65–74: 1 point, 75–84: 2 points, ≥85: 3 points), Lactate (<2 mmol/L: 0 points, 2–4.9: 1 point, 5–9.9: 3 points, ≥10: 6 points), Intubated (2 points), Left Ventricular function (mildly impaired or better: −1 point, moderately impaired: 1 point, severely impaired: 3 points) and External/out of hospital cardiac arrest 2 points).

The scoring system was validated using a testing set of 515 patients presenting to Harefield Hospital in 2021. The validation metrics were excellent with a c-statistic of 0.9, Brier’s score 0.06 vs a naïve classifier of 0.15, Spiegelhalter’s z-statistic probability of 0.267 and a calibration slope of 1.08.

Conclusion The BE-ALIVE score is a simple and accurate scoring system to predict 30-day mortality in patients presenting with ACS. Appreciating this mortality risk can allow prompt involvement of appropriate care such as the shock team.

  • Acute Coronary Syndrome
  • Myocardial Infarction
  • Out-of-Hospital Cardiac Arrest

Data availability statement

Data are available on reasonable request. The data are available from the corresponding author, (AT), on reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/openhrt-2023-002313

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Fast and accurate identification of patients at high risk of death after acute coronary syndromes allows more efficient triage and allocation of downstream care. This improves patient outcomes. However, identifying those at risk can be difficult in the acute situation, especially if they are near to the middle of the probability spectrum.

    WHAT THIS STUDY ADDS

    • This study creates a quick and easy scoring system (The BE-ALIVE score) that allows a clinician to accurately assess the 30-day mortality risk in patients presenting with acute coronary syndromes.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

    • The BE-ALIVE score will assist clinicians in assessing the 30-day mortality risk of patients as they present to a heart attack centre. This expedites the allocation of the right care in the shortest possible time.

    Introduction

    The decline in mortality in acute coronary syndromes (ACS) patients has stalled in recent years, largely because tackling cardiogenic shock in ACS remains difficult and frequently unsuccessful despite advances in timeous reperfusion and support devices. Short-term mortality in these patients remains almost 50%.1 Part of the difficulty is quickly identifying which patients are shocked because initial information can be contradictory and difficult to assess in the acute setting. For example, a raised lactate can be accompanied by normotension, potentially delaying escalation to dedicated shock teams.2

    This is important because identifying patients at risk of poor outcomes and early activation of a dedicated shock team to deliver acute care results in better outcomes. For example, the introduction of protocolised management of shocked patients in our heart attack centre was associated with an almost 50% reduction in the 30-day mortality of this cohort.3 However, this relies on the effective identification of patients who may benefit.

    Several well-validated risk scores have been used to stratify ACS patients, including Thrombolysis in Myocardial Infarction (TIMI), HEART and Global Registry of Acute Coronary Events (GRACE), with c-statistics ranging from 0.73 (GRACE) to 0.86 (HEART) for predicting major adverse cardiac events at 6 weeks.4 However, these scores also require information that is not available on admission, including cardiac biomarkers and renal function, in addition to subjective risk factor inclusion in the history.5–7

    The original Society for Cardiovascular Angiography and Interventions (SCAI) classification sought to stratify patients into different shock categories.1 This work was advanced by the Cardiogenic Shock Working Group (CSWG) who introduced the CSWG-SCAI classification. The SCAI-CSWG classification added further objectivity into the framework,8 although it does include alanine transaminase (which has the same immediate availability issue as detailed above) and lacks echocardiographic data which is essential to identify cardiogenic shock and its aetiology.

    This study, therefore, sets out to create a risk score that is accurate and can be used for immediate risk stratification so that the most appropriate management, including shock team involvement, can be promptly initiated. We have purposefully limited variables to those that can be easily measured prior to angiography, including point-of-care blood gas analysis, echocardiographic findings, and clear, objective features in the history such as the presence and location of cardiac arrest.

    The aim is that the scoring system can be used to improve patient outcomes by expediting appropriate multidisciplinary management.

    Methods

    This was a retrospective, observational study to design a simple risk score to define a 30-day mortality for patients presenting with ACS.

    The inclusion criteria for the study were as follows: (1) aged ≥18 years, (2) presented to Harefield Hospital with an ACS between 1 January 2011 and 1 January 2022 and (3) had a measured arterial blood gas within 4 hours of admission and prior to invasive coronary angiography. As the aim was to assess all patients with ACS on arrival, there were no exclusion criteria.

    The original derivation cohort included 2407 patients who presented with ACS between 2011 and 2020 with full datasets. The test cohort comprised 515 patients with full blood gas data who presented in 2021, and their data were separated and not analysed until the scoring model had been built.

    Clinical and outcome data

    The clinical data were taken from routine audit fields mandated for every admission with ACS at our institution. Laboratory and blood gas analysis was imported from our own hospital’s database. Echocardiographic assessment of left ventricular function was categorised by the overall visual impression of the performing clinician. Mortality data were obtained from the UK National Health Service (NHS) spine in collaboration with the Office for National Statistics.

    The primary endpoint for the scoring system development was mortality at 30 days. Validation of the risk score is discussed below.

    Statistical methods and creation of the risk score

    Baseline demographics were compared using Student’s t-test and Mann-Whitney U for continuous variables, and χ2 and Fisher’s exact test for categorical variables.

    All candidate predictive variables were initially assessed using binary logistic regression with a forward conditional approach. This then identified which variables were significant in predicting mortality at 30 days.

    The risk score was then created using a framework laid down in previous publications.9 10 In essence, each candidate variable was first divided into clinically meaningful categories established by other publications, and then each category was weighted according to its regression coefficient (online supplemental table 1). The intercept (constant) from the regression equation was adjusted by adding back in the predicted mortality contribution of the lowest-risk category for each variable. Using previously validated category boundaries is important because it prevents overfitting of the model to the training data.

    The performance of the model was assessed for the training and test sets using markers that assess both discrimination and calibration. Discrimination was assessed using the c-statistic (or area under the receiver operator characteristic curve). The Brier score was used to assess both discrimination and calibration, and Spiegelhalter’s Z score to assess calibration.

    All statistical analyses were performed by using R and SPSS (V 29.0). Much of the analysis and the figures of predicted vs actual mortality were created using Frank Harrell’s ‘rms’ package in R (https://hbiostat.org/R/rms/). All data are reported according to the Strengthening the Reporting of Observational studies in Epidemiology guidelines.11

    Results

    Baseline characteristics and overall mortality

    The baseline characteristics of the training (derivation) and tested cohort are shown in table 1. The 30-day mortality rate in the training cohort was 414/2407 (17.2%).

    View this table:
    Table 1

    Baseline characteristics and univariate analysis of patients in the training set

    Creation of the risk score

    Binary logistic regression was performed using a forward conditional method. All variables that were significantly associated with 30-day mortality on univariate analysis were included. In the final risk model there were six significant predictors (table 2). These were then used to create the BE-ALIVE score (figure 1).

    View this table:
    Table 2

    Multivariate analysis showing predictors of 30-day mortality in the training set

    Figure 1
    Figure 1

    The BE-ALIVE score.

    The maximum score is 17. The risk of 30-day mortality can be derived for any given points score by the following equation:

    Embedded Image

    Embedded Image can be approximated by using our risk model where:

    Embedded Image

    And hence the predicted mortality for any given points total becomes:

    Embedded Image

    The predicted mortality for all patients plotted against their actual mortality, for the training set is shown in figure 2. A histogram of predicted versus actual mortality for the training set by points score is shown in figure 3. Low risk refers to all patients with a score of 3 or less, whose overall mortality is under 5%. High risk refers to patients with a score of 10 and above, where overall mortality was 86%. The patients with scores of 4–9 (inclusive) have a 30-day mortality risk of 36%.

    Figure 2
    Figure 2

    Predicted versus actual mortality as calculated using the BE-ALIVE score.

    Figure 3
    Figure 3

    predicted mortality by BE-ALIVE points (---) and actual mortality (histogram) by BE-ALIVE points for the training cohort.

    Validation with an internal training set

    The risk score was validated on 515 ACS patients presenting to Harefield Hospital in 2021. The discriminatory ability was excellent with an area under the curve (AUC) of 0.90, in addition to a Brier’s score of 0.06 against a naïve classifier of 0.145 (table 3 and figure 4). The calibration slope was 1.08 and a Spiegelhalter’s Z statistic was −1.1 (p=0.267) indicating good calibration (figure 5).

    View this table:
    Table 3

    Assessment metrics for the BE-ALIVE score

    Figure 4
    Figure 4

    Receiver operator characteristic curve for the BE-ALIVE score applied to the testing set. AUC, area under the curve.

    Figure 5
    Figure 5

    Predicted versus actual mortality if the BE-ALIVE score as applied to the testing set. ROC, receiver operator characteristic.

    A BE-ALIVE score of 3 or less had a negative predictive value for 30-day mortality of 97.4% in the test set. The sensitivity of predicting 30-day mortality using this single threshold value was 89.4% with a specificity of 78%. There were only 13 patients in the test set with a BE-ALIVE score of 10 or above—10/13 died and hence the postive predictive value (PPV) was 77% for this small validation cohort.

    Comparison with SCAI-CSWG categories

    All patients in the training cohort were classified according to the SCAI-CSWG classification.8 The mean score for patients in category A was 1.3 (0.05) rising to mean score of 6.9 (0.15) for those in category E. There was a significant difference (<0.001) between all groups on pairwise comparisons (figure 6).

    Figure 6
    Figure 6

    BE-ALIVE score by SCAI-CSWG categories. p<0.001 across all comparisons (Kruskal-Wallis). p<0.001 for all pairwise comparisons except for C-D, where p=0.52 (Dunn).

    Using the SCAI-CSWG category to predict 30-day mortality resulted in an AUC of 0.791 (0.767–0.815) for the training set. Applying the SCAI-CSWG categories to the test set using the published in-hospital mortality probabilities, the AUC was 0.739 with a Brier score of 0.09 and a Spiegelhalter’s Z statistic of −4.501 (p<0.001) indicating poor calibration for this data set.

    Discussion

    The BE-ALIVE score is a simple, practical score for predicting 30-day mortality on admission, that is both highly discriminatory (AUC 0.9) and well calibrated (as measured by the Brier’s score and Spiegelhalter’s Z statistic). Using a threshold of three or below to designate patients as low risk has a 97.4% negative predictive value of death within 30 days.

    Mortality after ACS is a probability spectrum and appreciating this will lead to better care

    The patient presenting with ACS is not either ‘shocked’ or ‘not shocked’: there is no binary classifier for cardiogenic shock. However, we are preconditioned from clinical trial enrolment to look for defined markers, most commonly a systolic blood pressure of 90 mmHg. In reality, all patients presenting with ACS are at a high risk of mortality compared with both the population baseline and their own personal risk prior to the acute event.

    If we view patients through this probabilistic lens, then shock is a point where the chance of dying imminently rises rapidly—the gradient on the curve starts becoming exponentially steep. This idea ties in with the physiology of shock, where hypoperfusion leads to ischaemia, reduced contractility, raised filling pressures and pro-inflammatory cytokines which further accelerate the process12—and therefore, the patient moves further along the exponential mortality curve with an ever-accelerating risk of death. Thus, the aim of this score is not purely to define a threshold of treatment, but to define a mortality risk that can trigger treatment initiation sooner than may have been thought necessary.

    However, often binary decisions need to be made for the shocked patient, that rely on the patient being ‘shocked’ or ‘not shocked’, even if there are different gradations within the SCAI classification. Therefore, it is useful to have a better understanding of the patient’s trajectory post myocardial infarction (MI), and where they are on this hypothetical shock curve. This is the clinical utility of the BE-ALIVE score, which gives us an accurate visual representation of the trajectory of the patient from the moment that they arrive in the heart attack centre and therefore gives the clinician both more time and more information to divert the patient away from accelerating decline.

    Components of the BE-ALIVE score

    Following on from this, it is not surprising that (A) the risk of short-term mortality rises in this exponential fashion as points, representing markers of hypoperfusion and ischaemia, accumulate and (B) that the significant risk markers are base excess, lactate, age, intubation, presence of out of hospital cardiac arrest (OOHCA) and left ventricular (LV) dysfunction. The time to reperfusion is an important prognostic factor in ACS patients,13 and at the most basic level, an out-of-hospital arrest is likely to have delayed reperfusion in addition to variable quality cardiopulmonary resuscitation (CPR) and associated hypoxia. Ventilation on arrival to a receiving heart attack centre is similarly a marker of hypoxia and/or a relatively prolonged arrest. Both OOHCA and ventilation are thus unsurprisingly associated with increased mortality rates.2 14

    Lactate is the biochemical marker of prolonged tissue hypoperfusion, exacerbated by hepatic dysfunction: another example of the exponential rise in risk as the shock cascade progresses. Lactate as a strong predictor of mortality is not a new finding in either the non-ACS15 or ACS populations.16 However, it is absent from many scoring systems in ACS including the aforementioned GRACE, TIMI and HEART.5–7

    Similarly, base excess as a marker of mortality is, we believe, a powerful and under-represented marker of mortality in ACS patients. Metabolic acidosis is multifactorial, and pH is successfully buffered longer than base excess: hence its drop is a late marker and associated with very poor outcomes. For this reason, pH<7.2 is used in the CSWG-SCAI classification to define patients in category E. In contrast, base excess not only changes early but is also a marker of renal dysfunction, which both occurs early in cardiogenic shock17 and is a powerful predictor of mortality.9 Although creatinine added more accuracy to the training set in early iterations of the BE-ALIVE score, it is not universally available at the index time of a patient’s admission, and therefore, BE is an able substitute for the purposes of this scoring system.

    Finally, the risk scores discussed above do not include left ventricular function as part of the scoring systems, perhaps due to the lack of that data in derivation cohorts based outside of dedicated cardiac centres. As a point of admission predictor, it is powerful, and has been shown to be predictive of mortality in several registry studies.2 16 18

    Assessing the BE-ALIVE score: moving away from binary methods

    In terms of assessing the accuracy of the proposed risk score, we have purposefully not included terms such as ‘accuracy’ or an overall sensitivity/specificity for 30-day mortality, aside from the negative predictive value at very low risk scores. Giving predictions of overall accuracy are easy to understand but unsophisticated because they arbitrarily dichotomise patients into binary categories on an individual basis, whereas we are seeking to visualise the post-ACS hazard as a probability continuum.

    For example, if a patient has a BE-ALIVE score of 7, the predicted 30-day mortality risk is around 50%, whereas it is 99% for a score of 17. A ‘misclassification’ (ie, predicting death but the patient surviving) of patients scoring 7 should happen almost half the time in a perfectly functional model, whereas a misclassification of a patient with a score of 17 should happen in fewer than 1% of such patients. A more confident ‘wrong’ prediction should be penalised more than a misclassification in the middle of the range, and therefore, we have used Brier’s score and Spiegelhalter’s Z statistic to show how well calibrated the risk predicted by the model is compared with the actual risk in the tested population.

    The other part of testing a model is the discrimination between groups, and this is what the c-statistic measures. Essentially, this is a measure of how well the model ranks patient according to risk.19 Using our model as an example, if one randomly chose to look at the data for a patient who died within 30 days and another who survived 30 days, there is a 90% chance that our model would give the non-survivor a higher BE-ALIVE score.20 Both the metrics we have chosen for measuring calibration and discrimination are robust measures that seek to minimise the effect of the underlying population distribution on assessing the accuracy of the results—which is key when the dataset is unbalanced.

    The metrics chosen show that the BE-ALIVE score compares favourably to the SCAI-CSWG categories in terms of discrimination and calibration with better AUC (0.9 vs 0.74), Brier score and Spiegelhalter’s Z-statistic. The BE-ALIVE score is not a replacement for the SCAI or SCAI-CSWG categories but should be used alongside them to facilitate both the recognition of shock and to give a common framework to enhance communication between clinicians.

    Limitations

    The main limitation of this study is its retrospective, observational and single-centre nature. More specifically, it relies on blood gas measurements being taken and recorded, which immediately selects a higher-risk population. This explains why the 30-day mortality in this ACS cohort was 17%. The implication of this is that any particular points score may be overly pessimistic when applied to a lower-risk population.

    Every ACS patient at our institution now has a blood gas when the arterial sheath is inserted and therefore future local validation will enhance this score’s credibility. This is also why we attempted to move beyond the assessment of the system using simple binary classification metrics, which would limit the applicability to patients with a different prevalence of 30-day mortality. However, the most important future work will be to validate and iterate the model using data from institutions with different risk profiles to confirm and improve the clinical validity of this model for a wider population.

    Conclusion

    The BE-ALIVE score is an accurate and simple scoring system for predicting mortality at 30 days. The aim is to assist clinicians in immediate risk assessment of ACS patients with the downstream effect of initiating the most appropriate care in the shortest time.

    Data availability statement

    Data are available on reasonable request. The data are available from the corresponding author, (AT), on reasonable request.

    Ethics statements

    Patient consent for publication

    Ethics approval

    All patient-identifiable information was removed before analysis. Our local audit office assigned institutional support for this project. As this was analysis of anonymised information taken from required audit data, we were advised that no further ethical approval was required.

    Acknowledgments

    Andy Ryan in the Harefield audit office for support of the project.

    References

    1. ↵
      1. Baran DA,
      2. Grines CL,
      3. Bailey S, et al
      . SCAI clinical expert consensus statement on the classification of cardiogenic shock: this document was endorsed by the American college of cardiology (ACC), the American heart Association (AHA), the society of critical care medicine (SCCM), and the society of Thoracic Surgeons (STS) in April 2019. Catheter Cardiovasc Interv 2019;94:29–37. doi:10.1002/ccd.28329
      OpenUrlCrossRef
    2. ↵
      1. Panoulas V,
      2. Ilsley C
      . Rapid classification and treatment algorithm of cardiogenic shock complicating acute coronary syndromes: the SAVE ACS classification. J Interv Cardiol 2022;2022:9948515. doi:10.1155/2022/9948515
    3. ↵
      1. Vela MM,
      2. Rosengerb A,
      3. Fiorelli F, et al
      . TCT-74 using a Protocolized approach and a dedicated shock team leads to significant reductions in 30-day mortality in acute coronary syndrome patients with cardiogenic shock. Journal of the American College of Cardiology 2022;80:B30–1. doi:10.1016/j.jacc.2022.08.090
      OpenUrl
    4. ↵
      1. Poldervaart JM,
      2. Langedijk M,
      3. Backus BE, et al
      . Comparison of the GRACE, HEART and TIMI score to predict major adverse cardiac events in chest pain patients at the emergency Department. Int J Cardiol 2017;227:656–61. doi:10.1016/j.ijcard.2016.10.080
      OpenUrlCrossRefPubMed
    5. ↵
      1. Fox KAA,
      2. Dabbous OH,
      3. Goldberg RJ, et al
      . Prediction of risk of death and myocardial infarction in the six months after presentation with acute coronary syndrome: prospective multinational observational study (GRACE). BMJ 2006;333:1091. doi:10.1136/bmj.38985.646481.55
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Six AJ,
      2. Backus BE,
      3. Kelder JC
      . Chest pain in the emergency room: value of the HEART score. Neth Heart J 2008;16:191–6. doi:10.1007/BF03086144
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Antman EM,
      2. Cohen M,
      3. Bernink PJ, et al
      . The TIMI risk score for unstable angina/non-ST elevation MI: A method for prognostication and therapeutic decision making. JAMA 2000;284:835–42. doi:10.1001/jama.284.7.835
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Kapur NK,
      2. Kanwar M,
      3. Sinha SS, et al
      . Criteria for defining stages of cardiogenic shock severity. J Am Coll Cardiol 2022;80:185–98. doi:10.1016/j.jacc.2022.04.049
      OpenUrlCrossRef
    9. ↵
      1. Tindale A,
      2. Vela MM,
      3. Panoulas V
      . Using base excess, albumin, lactate, and renal function to predict 30-day mortality in patients requiring Impella monotherapy for left-sided mechanical circulatory support: the BALLAR score. Cardiovasc Revasc Med 2022;41:129–35. doi:10.1016/j.carrev.2021.12.008
      OpenUrl
    10. ↵
      1. Sullivan LM,
      2. Massaro JM,
      3. D’Agostino RB
      . Presentation of multivariate data for clinical use: the Framingham study risk score functions. Stat Med 2004;23:1631–60. doi:10.1002/sim.1742
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. von Elm E,
      2. Altman DG,
      3. Egger M, et al
      . The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 2007;147:573–7. doi:10.7326/0003-4819-147-8-200710160-00010
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Reynolds HR,
      2. Hochman JS
      . Cardiogenic shock: Current concepts and improving outcomes. Circulation 2008;117:686–97. doi:10.1161/CIRCULATIONAHA.106.613596
      OpenUrlFREE Full Text
    13. ↵
      1. Park J,
      2. Choi KH,
      3. Lee JM, et al
      . “Prognostic Implicationsof door-to-balloon time and onset-to-door time on mortality in patients with ST -Segment-elevation myocardial infarction treated with primary percutaneous coronary intervention. J Am Heart Assoc 2019;8:e012188. doi:10.1161/JAHA.119.012188
    14. ↵
      1. Pareek N,
      2. Kordis P,
      3. Beckley-Hoelscher N, et al
      . A practical risk score for early prediction of neurological outcome after out-of-hospital cardiac arrest: Miracle2. Eur Heart J 2020;41:4508–17. doi:10.1093/eurheartj/ehaa570
      OpenUrlPubMed
    15. ↵
      1. Kruse O,
      2. Grunnet N,
      3. Barfod C
      . Blood lactate as a Predictor for in-hospital mortality in patients admitted acutely to hospital: a systematic review. Scand J Trauma Resusc Emerg Med 2011;19:74. doi:10.1186/1757-7241-19-74
    16. ↵
      1. Frydland M,
      2. Møller JE,
      3. Wiberg S, et al
      . Lactate is a Prognostic factor in patients admitted with suspected ST- elevation myocardial infarction. Shock 2019;51:321–7. doi:10.1097/SHK.0000000000001191
      OpenUrl
    17. ↵
      1. Schefold JC,
      2. Filippatos G,
      3. Hasenfuss G, et al
      . Heart failure and kidney dysfunction: Epdemiology, mechanisms and management. Nat Rev Nephrol 2016;12:610–23. doi:10.1038/nrneph.2016.113
      OpenUrlCrossRefPubMed
    18. ↵
      1. Delmas C,
      2. Puymirat E,
      3. Leurent G, et al
      . Design and Pre- Liminary results of FRENSHOCK 2016: a prospective Na- Tionwide Multicentre Registry on cardiogenic shock. Arch Cardiovasc Dis 2019;112:343–53. doi:10.1016/j.acvd.2019.02.001
      OpenUrl
    19. ↵
      1. Huang Y,
      2. Li W,
      3. Macheret F, et al
      . A Tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Inform Assoc 2020;27:621–33. doi:10.1093/jamia/ocz228
      OpenUrlPubMed
    20. ↵
      1. Spiegelhalter D
      . The art of Statistics, pelican. London, UK, 2019.

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors AT: concept, design, data collection, data analysis, statistics, data interpretation, drafted article, critical revision of article, guarantor. VP: concept, design, analysis, interpretation of data, critical revision.

    • Funding This study was funded by Imperial College London (PhD student).

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.