Article Text
PDF
Abstract
Objective To provide brief guidance on how to design accrual monitoring activities in a clinical trial protocol.
Setting Two completed clinical trials that did not achieve the planned sample size, the Cost of Strategies After Myocardial Infarction (COSTAMI) trial and the Biventricular Pacing After Cardiac Surgery (BiPACS) trial.
Design A Bayesian monitoring tool, the constant accrual model, is applied retrospectively to accrual data from each case study to illustrate how the tool could be used to identify problems with accrual early in the trial period and to frame the conditions in which the approach can be used in practice.
Results After 312 days and 155 patients enrolled in the COSTAMI trial, accrual could be classified as ‘off target’ on the basis of statistical criteria outlined in the protocol. As for the BiPACS trial, after 2 years, it was already evident that the accrual was ‘considerably off target’.
Conclusions Prompt awareness of a high risk of accrual failure could trigger different interventions to overcome protocol-related, patient-related or investigator-related barriers to recruitment or ultimately contribute to an early stopping decision due to recruitment futility.
Accrual prediction models should be included as standard tools for routine monitoring activities in cardiovascular research. Among them, methods relying on the Bayesian approach are particularly attractive, as they can naturally update past evidence when actual accrual data becomes available.
- poor accrual
- bayesian monitoring
- clinical trial
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Key questions
What is already known about this subject?
Slow patient accrual in clinical trials can have significant negative impact on their success. Careful monitoring and prediction of accrual success (or failure) is crucial for improving enrolment in clinical trials and thus increasing the likelihood of their overall success.
What does this study add?
This study showcases the several different Bayesian options for accrual monitoring available to investigators and frames the conditions in which the different approaches can be used in practice.
How might this impact on clinical practice?
Proposing a standard operating procedure for the application of Bayesian monitoring tools will hopefully encourage their use in general clinical trial monitoring.
Introduction
Lack of timely accrual presents a significant challenge to the performance of cardiovascular trials,1–3 generating resource-consuming underpowered trials that are unable to meet their intended objectives meaningfully. Unsuccessful recruitment poses ethical, economic and scientific problems for the clinical research enterprise.4
The ability to monitor and accurately predict clinical trial accrual allows for greater support for decisions to be made about the management and conduct of clinical trials.5 Measuring the likelihood of achieving the projected accrual within a reasonable time frame may prompt trial decisions in different directions. These range from adding further resources to implement interventions that may improve accrual performance to closing studies early to release resources for more promising trials.
Some authors4 5 suggest that mechanisms for terminating trials are not sensitive enough for recruitment futility and advocate the use of predictors of accrual performance to allow for better allocation of research resources and ensure that subjects enrolled in trials will actually play a role in advancing medical knowledge. With good monitoring tools, clearly identified and detailed in the planning phase of a trial, researchers would be able to define realistic targets for their sample sizes and gain an early warning when accrual rates are suffering.
From a technical standpoint, sophisticated statistical approaches6 7 and tools8 for accrual monitoring and prediction are available in the literature but are probably underused. As acknowledged by Barnard and colleagues,6 the ease of implementation of a model would also contribute to the choice of method to be used. In this trade-off between usability and accuracy, Bayesian models are probably at increased complexity of use compared with the relative simplicity of deterministic models (ie, the total accrual time is estimated by dividing the planned sample size by the number of subjects expected to enrol in the study during each time unit) or frequentist Poisson-based models (ie, number of participants recruited within a fixed time follows a Poisson distribution). However, accrual monitoring methods relying on the Bayesian approach are particularly attractive.9 10
We recall that any Bayesian analysis starts with a prior probability distribution for the value of interest, based on existing knowledge, and adds new evidence as data accumulate to produce a posterior predictive distribution, which is the distribution for future predicted data based on the data observed so far. The attractive key element of the Bayesian approach relies on this natural updating process. In fact, researchers’ experience from similar studies can be incorporated as prior knowledge, and when actual accrual data become available, the predictive distribution becomes the weighted average of the prior distribution and the actual observed data. As more data are collected, the weighting of the current observed data increases, while the weighting of prior information decreases.
Such an approach provides an effective assessment of the accrual process. In fact, one interesting feature of these models is that if the researcher’s a priori confidence in the subject accrual rate is strong, the posterior distribution will be weighted heavily towards the prior distribution, especially early in the trial. This prevents an undesirable alarm when the enrolment of the first few subjects is slower than expected. However, if the researcher has weak confidence, early evidence of slow accrual will be given greater weight, prompting a rapid response to address the slow accrual.
We present brief guidance on how to embed an accrual monitoring framework in a study protocol and how to assess patient accrual by retrospectively applying a Bayesian monitoring tool to Cost of Strategies After Myocardial Infarction (COSTAMI) trial11 12 and Biventricular Pacing After Cardiac Surgery (BiPACS)13 trial data.
Guidance for accrual monitoring
In line with the policy of major research institutions such as the National Heart, Lung and Blood Institute (NHLBI) (http://www.nhlbi.nih.gov/research/funding/human-subjects/accrual-guidelines), it is important for clinical trial protocols to be conducted according to guidelines on monitoring and evaluating the adequacy of accrual. Our proposal is to embed a statistical tool in a clinical trial protocol for accrual monitoring and associated decision rules. That notwithstanding, a statistical stopping rule should be regarded as only one useful objective component in an inevitably more challenging decision-making process.
Accrual benchmark
Before initiating any study, the investigators, in collaboration with the monitoring entity, if applicable, and the biostatistics advisors shall agree on the best practices for quality assurance. This requires them to outline plans and benchmarks for monitoring recruitment, including the statistical model, projected recruitment time duration and final recruitment target. Although there is no obvious choice, table 1 highlights some important aspects of prior distribution choices for the Bayesian constant accrual model.9 10
Contingency table between choice of prior for Bayesian constant accrual model9 10 and investigator beliefs (sceptical vs optimistic) and/or trial features (small vs large sample size at interim reviews of accrual performance)
Conduct
Formal reviews may occur at given time points of the projected recruitment period or continuously, depending on the characteristics of the study, including such factors as the total length of time for recruitment and the level of risk. Ideally, these formal reviews serve as the minimum number of time points at which action will be taken.
Criteria and actions
On the basis of statistical decision rules, recruitment is classified as falling into one of four categories: ‘on target’, ‘slightly off target’, ‘considerably off target’ and ‘off target’. Evidence of inadequate accrual requires that a corrective plan come into play.
Available software
Notwithstanding the availability of statistical models for evaluating accrual, their implementation into an easily accessible, user-friendly software program is not well established. To the best of our knowledge, the only software widely accessible is the R package ‘accrual’,14 which implements accrual Bayesian models. It is also available as a web-based calculator and an Android smartphone application.
Case studies
The COSTAMI trial
The COSTAMI trial11 12 is a randomised multicentre parallel trial aiming to evaluate the reduction in costs associated with the management of patients with acute myocardial infarction by means of a non-invasive strategy intended to identify patients at low risk very early, thus allowing early hospital discharge. The anticipated sample size was 1506 subjects to be recruited in a 3-year period.
The trial terminated after a 3-year accrual with 686 patients recruited. Only some of the planned comparisons were addressed with satisfactory power. Individual data were accessible to authors.
The following is a proposal of how key elements pertaining to accrual monitoring could have been presented in the trial protocol. Recruitment targets and duration are taken from the original protocol.
Protocol proposal
The trial is expected to accrue 1506 patients (753 per arm) in a 3-year period. Accrual monitoring will be performed using the Bayesian constant accrual model.9 10 This model assumes that the waiting time for each successive patient follows an exponential distribution, specified by a single parameter θ, which represents the average accrual time. This Bayesian model incorporates subjective knowledge about subject accrual rates through an informative prior distribution for θ, assumed to be inverse gamma. This distribution depends on two parameters, which are N, the planned number of subjects to recruit in T days (ie, 1506 patients in 1096 days), and P, the investigator’s confidence in the original plan, measured on a 0–1 scale. Given the presumably large sample sizes (n) at which the interim reviews will be carried out, we chose the so-called adaptive accelerated prior10 (ie, P=1 n/N).
The Bayesian constant accrual model9 10 will be applied to predict the waiting time for the rest of the sample size continuously throughout the trial.
Let us define the upper and lower limits of the credibility interval around the point estimate of the mean predicted total duration time (t) as lupper and llower , respectively. We assume that an acceptable delay in accrual completion is represented by 1370 days (ie, tolerance of 25%). Based on this assumption, the accrual performance will be categorised as:
‘On target’ if lupper <1370.
‘Slightly off target’ if t<1370 and lupper >1370.
‘Considerably off target’ if t>1370 and lupper >1370.
‘Off target’ if llower >1370.
Retrospective application
The results of the retrospective application of the chosen Bayesian monitoring model are shown in figure 1. Accrual remained on target for 91 days (24 patients by that time), with a predicted total duration of 1242 days (95% credibility interval 1161–1326). Afterwards, accrual performance progressively worsened. The transition from ‘slightly off target’ to ‘considerably off target’ occurred immediately after 196 days and 83 patients enrolled, with a predicted total duration at that time of 1360 days (95% credibility interval 1277–1446). After 312 days and 155 patients enrolled, accrual could be classified as ‘off target’, with 1474 predicted days (95% credibility interval 1391–1562).
Results of Bayesian models for accrual prediction for COSTAMI trial. Ninety-five per cent credibility intervals and point estimates of predicted total accrual duration. In green ‘on-target’ accrual, in yellow ‘slightly off-target’ accrual, in orange ‘considerably off-target’ accrual and in red ‘off-target’ accrual.
Implications
An early alert on the slow accrual could have noted the clinical trial’s failure to include enough patients to achieve the declared study power.
After 196 days, the predicted sample size at the end of the trial ranged from 799 to 945 patients, meaning a reduction in the power of the study from a minimum of 20 percentage points to a maximum of 27 percentage points. At 312 days, things worsened as the power, based on the predicted sample size achieved at the end of trial, was far below 60% (ranging from 51% to 57%).
Within the first year of the clinical trial, a monitoring procedure could have promptly posed questions about early study termination for futility due to lack of power.
The BiPACS trial
The BiPACS13 trial is a randomised trial aiming to determine whether optimised biventricular pacing increases the cardiac index in patients at risk of left ventricular dysfunction after cardiopulmonary bypass. The BiPACS protocol was conducted between 1 April 2007 and 29 February 2012, and it was terminated for slow accrual, with 61 patients randomised prior to reaching the endpoint target of 196 in approximately 5 years. The numbers of enrolled patients at given time points were derived from figure 3 of the article,13 using plotDigitizer free software, which digitises scanned plots of functional data and is available online (http://plotdigitizer.sourceforge.net/).
The following is a proposal of how key elements pertaining to accrual monitoring could have been presented in the trial protocol. Recruitment targets and duration are taken from the original protocol.
Protocol proposal
The trial is expected to accrue 196 patients (treatment allocation ratio of 1:1) in a 5-year period. Accrual monitoring will be performed using the Bayesian constant accrual model.9 10 This model assumes that the waiting time for each successive patient follows an exponential distribution, specified by a single parameter θ, which represents the average accrual time. This Bayesian model incorporates subjective knowledge about subject accrual rates through an informative prior distribution forθ, assumed to be inverse gamma. This distribution depends on the planned number N of subjects to recruit in T days (ie, 196 in 1826 days) and on the investigator’s confidence P, measured on a 0–1 scale. Given the presumably small sample sizes (n) at which the interim reviews will be carried out, three choices of P will be considered: P=0.5 (sceptical), P=1 (optimistic) and P=1 n/N (adaptive accelerated prior). If P equals 0.5, the prior is given weight equivalent to half the proposed sample size. This means that halfway through the study, the prior and the actual subject accrual data are given equal weight. If P equals 1, the prior is given weight equivalent to the proposed sample size of the study. The adaptive accelerated prior performs similar to the strong prior when the accrual is on target.
The Bayesian constant accrual model9 10 will be applied to predict the waiting time for the rest of the sample size at every 10 patients enrolled.
Let us define (for each prior choice) the upper and lower limits of the credibility interval around the point estimate of the mean predicted total duration time (t) as lupper and llower , respectively. We also assume that an acceptable delay in accrual completion is 2283 days (ie, tolerance of 25%). The accrual performance will be categorised as:
‘On target’ if lupper <2283 consistently across all different choices of priors.
‘Slightly off target’ if t<2283 and lupper >2283 for at least two of three priors.
‘Considerably off target’ if t>2283 and lupper >2283 consistently across all different choices of priors.
‘Off target’ if llower >2283 consistently across all different choices of priors.
Retrospective application
Up to the second evaluation point (30/196), two out of three prior choices consistently indicate that the accrual is ‘considerably off target’ and therefore that it was very unlikely to reach the target sample size in a reasonable time frame, even tolerating a delay of 25%. After 40 patients enrolled, accrual was markedly ‘off target’, with a mean predicted total accrual duration of 3835 days (95% credibility interval 3410–4329) for the accelerated prior. Of note, the choice of prior does not affect this conclusion (figure 2).
Results of Bayesian models for accrual prediction for BiPACS trial. Ninety-five per cent credibility intervals and point estimates of predicted total accrual duration. In green ‘on-target’ accrual, in yellow ‘slightly off-target’ accrual, in orange ‘considerably off-target’ accrual and in red ‘off-target’ accrual. From left to right: P=0.5, P=1 and accelerated prior. BiPACS, Biventricular Pacing After Cardiac Surgery.
Implications
At the first interim evaluation (20 patients in approximately 400 days), the predicted sample size at the end of the second year, computed using the accrual model, provides a potential early alert on trial accrual. On the basis of the constant accrual model with a sceptical prior (which was considered as the interim evaluations are based on small numbers), 41–65 patients are expected.
The number of patients actually enrolled at the end of the second year is slightly inferior (30 patients), implying an expected power for the study ranging from 49% to 66%. While a study with a power of approximately 70% can still be considered acceptable, an early alert would have triggered some interventions such as incentives or recruitment programmes, if feasible.
At the end of the third year, the number of patients is again under the minimum sample size predicted by the model (40 patients), showing an inherently slow accrual process. Projecting these data to the end of the study indicates a study power ranging from 44% to 58%.
Discussion
Patient recruitment difficulties are caused by a number of factors, including high financial costs, competing demands on clinicians, regulatory barriers, narrow eligibility criteria and cultural attitudes towards research.15 Previous studies16 17 have examined several strategies directed at addressing slow accrual from both a patient perspective and healthcare provider standpoint.
This paper provides a formal framework for accrual monitoring that should be envisaged in a clinical trial protocol and illustrates the use of some statistical monitoring tools. This is in line with the requirements of major health institutions, such as the NHLBI, which are increasingly encouraging investigators to use operational performance measures, such as enrolment rates, to regularly track progress and enhance trial efficiency.18 Statistical accrual monitoring accomplishes this task well, as it is a practical tool for continuously inspecting accrual performance, thus preventing unnecessary loss of resource utilisation.
Study investigators and ethical oversight bodies should have the needed and appropriate tools for monitoring accrual to either fix poorly accruing trials or terminate trials that cannot accrue a reasonable number of patients in a reasonable time frame to best reallocate human and economic resources to more promising studies. In the latter case, complete follow-up of subjects already enrolled for endpoints assessment and ancillary information should be guaranteed.
The use of simplistic and unsatisfactory monitoring tools should not be justified by a lack of statistical expertise. Alternatively, recruitment prediction services should be offered by local clinical trials units or research design services, which may allow models to be better tailored to the clinical trial needs.
The allocation of sufficient time for participant recruitment is a fundamental aspect of planning a clinical trial. We agree with Carlisle and colleagues4 when they state that the number of trials that are terminated early for unsuccessful accrual ideally should exceed the number of trials that are continued to completion but with enrolments that are far below the target sample size. Nevertheless, we acknowledge that ineffective accrual practices, such as those in the COSTAMI and BiPACS trials, do not necessarily reflect a failure on the part of investigators, sponsors or oversight bodies, and the simple fact that these studies are underpowered does not make them unethical. Obviously, even studies that fail to achieve their recruitment target still convey some information on secondary endpoints and safety observations.4
The recruitment needs for clinical trials, often conducted simultaneously and involving patients with similar conditions, results in competition for eligible subjects.19 Enrolling patients in studies with diminished likelihood of answering the underlying scientific question meaningfully raises several concerns both on the side of study participants and of enrolment site staff: should accrual results be disclosed to them to enable them to make a truly informed choice (ie, whether to enter the clinical trial or whether to enrol patients in that trial, respectively)?
As acknowledged by others,20 there is a need to reconsider whether it is always essential to keep the interim results of randomised clinical trials confidential, including interim accrual data.
Conclusions
As our examples illustrate, problems with patient accrual can become evident early, during the first years of the clinical trial. A prompt alert on this issue could trigger different interventions to overcome protocol-related, patient-related or investigator-related barriers to recruitment, such as amending the inclusion criteria or implementing recruitment incentives and effective communications, just to mention a few.
We do not advocate making decisions solely on accrual performance; rather, we advise using accrual-monitoring metrics to complement scientific judgement when making decisions about the management of trials. Specifically, we advocate applying this tool into a continuous and integrated decision-making process to optimise the utilisation of resources throughout the trial.
References
Footnotes
Contributors IB and PB designed the study, wrote the manuscript and performed the statistical analysis. All authors read and approved the final manuscript.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.