Original Article
A simple formula for the calculation of sample size in pilot studies

https://doi.org/10.1016/j.jclinepi.2015.04.014Get rights and content

Abstract

One of the goals of a pilot study is to identify unforeseen problems, such as ambiguous inclusion or exclusion criteria or misinterpretations of questionnaire items. Although sample size calculation methods for pilot studies have been proposed, none of them are directed at the goal of problem detection. In this article, we present a simple formula to calculate the sample size needed to be able to identify, with a chosen level of confidence, problems that may arise with a given probability. If a problem exists with 5% probability in a potential study participant, the problem will almost certainly be identified (with 95% confidence) in a pilot study including 59 participants.

Introduction

A pilot study can be defined as a small-scale study that helps to examine the practicality and feasibility of the methods to be used in a subsequent larger and more comprehensive investigation [1]. Because conducting an adequately powered study often requires the inclusion of a large number of participants and therefore may be very costly in terms of time and money, piloting a study on a smaller scale can help to identify unforeseen problems that could compromise the quality or flow of the study [2].

For example, one may encounter nonanticipated reasons why potential participants have to be excluded, questionnaire items that are interpreted in unintended ways by the participants or whose answer options are not sufficiently comprehensive, or unclear information about the delivery of the intervention (eg, dosing or visiting schedules).

If such problems are discovered during the course of a pilot study, the necessary steps can be taken before the actual large-scale study is started to minimize or entirely avoid their negative impact. For example, the study protocol or materials (eg, questionnaire items) could be adapted accordingly, or contingency plans could be set up ahead of time to handle any problems adequately and in a timely manner. However, this is only possible if such problems are actually discovered during the conduct of the pilot study. Therefore, a pilot study aimed at discovering such problems should have sufficient power to do so, or in other words, the sample size of a pilot study must be sufficiently large, such that the probability of detecting such problems is high.

Existing methods for sample size calculations typically focus on how to select an appropriate sample size for a pilot study such that various parameters of interest can be estimated with sufficient precision (eg, the effect size, the standard deviation of the outcome measure, its reliability, or adherence or attrition rates) [1], [3], [4], [5]. Such calculations may also play an important role in deciding whether to proceed with the primary trial in the first place [6], [7]. These considerations have led to various guidelines for choosing an appropriate sample size for a pilot study, such as 12 participants per group [3], values in the range of 10 to 40 participants per group depending on the parameter of interest [4], [5], at least 9% of the main trial's sample size [6], or at least 50 participants [8].

However, none of these approaches is directly applicable when the goal of a pilot study was the detection of unforeseen problems. Therefore, in this article, we take a different approach and describe a simple method for determining the sample size necessary to identify problems with a chosen level of confidence in pilot studies.

Not surprisingly, the sample size determined in this manner depends not only on the confidence level with which we would like to detect a particular problem but also the actual probability that the problem manifests itself in a potential study participant. Because the true problem probability is unknown in practice, what we really need to consider is a lower bound for the problem probability: if the true probability is in fact this low (or higher), then we achieve (or exceed) the desired confidence level, but if it is really lower (so that we are more likely than desired to miss the problem), then it would be infrequent enough to not be considered a problem worthy of detection. Choice of this value therefore depends on the context and on how detrimental a problem would be to a trial. We will return to this issue further in the following.

Section snippets

Required sample size to detect a problem in a pilot study

For now, assume that a particular problem has a given probability of occurring in a potential study participant. For example, if there is a 0.15 probability of encountering unanticipated reasons for exclusion in a given participant, then there is 0.85 probability that this problem does not manifest itself. In a group of n participants, there is then a 0.85n probability that the problem will not occur at all. Therefore, the probability of observing at least one occurrence of this problem in n

Sample size calculations in practice

To use Equation (1) for deciding on a sample size for a pilot study, we must first choose values for γ and π. By convention, we could adopt a confidence level of 95% (analogous to the level commonly chosen for confidence intervals) or choose the confidence level in accordance with the severity of the issue that we want to detect. (ie, a potentially disastrous problem that could ruin the entire study should be detected with higher confidence than a problem that would merely be a nuisance to deal

Discussion

Despite the extensive levels of planning involved in the design of a study, experience shows that unforeseen problems can and typically will arise during the conduct of a study that must be handled with care. A pilot study provides an excellent opportunity to uncover such problems ahead of time, minimizing the need to adapt procedures or to develop contingency plans on short notice when the larger study is being conducted. In this article, we described a simple method for choosing a sample size

References (14)

There are more references available in the full text version of this article.

Cited by (312)

View all citing articles on Scopus

W.V. derived the equation. W.V. wrote the first draft and produced the figure. All authors contributed to revising the article and approved the final version. W.V. is the guarantor.

Conflict of interest/Financial disclosure: Neither conflict of interest nor financial support regarding the present study.

View full text