Sample Size Determination in Pilot Studies

Why pilot-study sample size is a different problem

A definitive study is commonly sized to detect or estimate a clinically important effect. A pilot or feasibility study usually has a different purpose: to reduce uncertainty about whether a future definitive study can be conducted, whether it should proceed, and how its design should be refined.

The governing principle is therefore:

A pilot study does not escape sample size justification. Its sample size must be justified against its pilot objectives, intended analyses, and decision rules—not against a ritual percentage of the future trial or an efficacy hypothesis it is not designed to answer.

The CONSORT extension for randomised pilot and feasibility trials asks investigators to report the rationale for the number of participants rather than assuming that “no formal calculation” is sufficient.[1] Contemporary methodological guidance likewise recommends matching the calculation to the objective and interpretation of the pilot.[2]

This guide focuses mainly on health-research pilot studies conducted in preparation for a future definitive trial. The principles also apply, with appropriate modification, to non-randomised feasibility studies, diagnostic pathways, implementation studies, and studies piloting data-collection procedures.

Learning objectives

By the end of this guide, the reader should be able to:

1. Distinguish pilot and feasibility objectives from definitive efficacy objectives.
2. Select a sample size method that corresponds to a prespecified source of uncertainty.
3. Calculate sample sizes for precision of feasibility proportions and detection of uncommon operational problems.
4. Quantify uncertainty in a pilot estimate of a continuous-outcome standard deviation.
5. Design progression criteria using their sampling operating characteristics rather than thresholds alone.
6. Explain why a pilot treatment-effect estimate is usually unsuitable for powering the definitive trial.
7. Adapt the justification for internal pilots, cluster-randomised studies, repeated measures, and multiple feasibility objectives.
8. Write a transparent, reproducible pilot-study sample size justification.

1. Begin with the decision, not the formula

Before calculating anything, write the decision that the pilot will inform. Examples include:

• Can the target population be recruited at the required rate?
• Can at least 80% of randomised participants provide the primary outcome?
• Can intervention fidelity exceed a clinically defensible threshold?
• Is an outcome measure sufficiently complete, variable, and responsive for the definitive study?
• Is a multicentre or cluster-randomised workflow operationally deliverable?
• Which of two recruitment or data-collection strategies should be taken forward?
• Is an adverse operational event sufficiently rare to permit progression?

Then define an analysis-to-decision chain:

This prevents a common mismatch: calculating sample size for one quantity, analysing another, and making a progression decision about a third.

2. External pilot, internal pilot, and feasibility study

External pilot

An external pilot is completed before the definitive study, and its participants are not included in the definitive analysis. It allows substantial redesign but requires additional time and recruitment.

Internal pilot

An internal pilot is an initial stage of the definitive study. Its eligible observations contribute to the final analysis. It is efficient, but adaptations must be prespecified and statistically compatible with valid final inference. Blinded nuisance-parameter reassessment is different from unblinded adaptation based on an interim treatment effect; the latter may require formal adaptive methods to control type I error.[3]

Feasibility study

A feasibility study asks whether and how a future study can be done. A pilot study is a subset in which the future study, or part of it, is conducted on a smaller scale.[1] Terminology varies, so the operational definition and objectives matter more than the label.

3. A taxonomy of sample size objectives

Most defensible pilot sample size justifications fall into one or more of five families.

If several objectives are genuinely co-primary, calculate the requirement for each and adopt the largest feasible requirement, while also checking site, cluster, and time constraints. Do not multiply sample sizes merely because several outcomes are recorded.

4. Precision for a feasibility proportion

Suppose the primary feasibility parameter is a retention probability p, estimated by

$$\hat p = \frac{X}{n}, \qquad X \sim \operatorname{Binomial}(n,p).$$

For a two-sided confidence level 1 − α and desired approximate half-width d, a planning approximation is

$$n \approx \frac{z_{1-\alpha/2}^{2}p_0(1-p_0)}{d^2},$$

where p₀ is a plausible planning value. If no credible value is available, p₀ = 0.5 maximises the binomial variance and is conservative for this approximation.

Worked example: retention precision

Investigators expect 80% of recruited participants to provide the Week-12 outcome and want an approximate 95% confidence interval with half-width 0.08:

$$n \approx \frac{1.96^2(0.80)(0.20)}{0.08^2}=96.04.$$

The result is rounded up to 97 recruited participants. The final interval should use a binomial method such as Wilson or an appropriate exact interval, not an unqualified Wald interval. Because the parameter is retention among recruited participants, 97 is the denominator required for that objective; mechanically inflating it for “dropout” would double-count the event being estimated.

Important refinements

• The formula is an approximation; actual interval width depends on the observed count and the interval method.
• Near 0 or 1, or with small n, enumerate candidate sample sizes using the intended binomial interval.
• If precision is required separately by trial arm, site, demographic subgroup, or delivery mode, the relevant denominator is the subgroup sample size—not the total sample.
• Recruitment rate per unit time is not a simple binomial proportion. It may require a Poisson, negative-binomial, or site-level model that reflects exposure time and heterogeneity.
• If participants are clustered, the binomial independence assumption is invalid and the correlation structure must be incorporated.

5. Detecting an operational problem

Sometimes the objective is not to estimate a prevalence precisely but to have a high probability of seeing at least one problem if its true probability is unacceptably large.

Let p* be the smallest event probability worth detecting and let γ be the desired probability of observing at least one event. Under independent Bernoulli observations,

$$\Pr(\text{at least one event})=1-(1-p_*)^n.$$

Therefore,

$$n \geq \frac{\log(1-\gamma)}{\log(1-p_*)},$$

rounded upward.[4]

Worked example: data-collection failure

If a technical failure occurring in 5% of procedures would threaten the definitive trial, and the team wants a 95% probability of observing at least one such failure,

$$n \geq \frac{\log(0.05)}{\log(0.95)}=58.40.$$

At least 59 independent procedures are required.

This criterion answers only whether the problem is likely to be seen. It does not estimate its probability precisely, determine causation, or guarantee that every class of problem will be detected. Dependence among repeated procedures or centres reduces the effective information and must be modelled.

6. Progression criteria are statistical decision rules

Traffic-light criteria are often written as if the observed percentage were the truth:

• green: retention at least 80%;
• amber: 65% to less than 80%;
• red: less than 65%.

But the observed category is random. If X follows a binomial distribution with parameters n and p, a green rule requiring X/n ≥ g has probability

$$\Pr(\text{green}\mid p)= \Pr\{X\geq \lceil ng\rceil\mid p\}.$$

Similarly, the probabilities of amber and red can be computed for each plausible true value of p. These are the rule’s operating characteristics.

Design progression criteria in four steps

1. Define an unacceptable value pU and an acceptable value pA, with clinical, operational, and stakeholder justification.
2. Choose candidate red/amber/green boundaries and candidate n.
3. Calculate the probability of each decision under pU, pA, and intermediate values.
4. Select a design whose risks of inappropriate progression and inappropriate stopping are acceptable.

For a simpler go/no-go rule with green if X ≥ c, two useful quantities are

$$\alpha_F=\Pr(X\geq c\mid p=p_U)$$

and

$$\beta_F=\Pr(X<c\mid p=p_A).$$

Here, αF is the risk of progressing when feasibility is unacceptable, and βF is the risk of stopping when feasibility is acceptable. These are analogous to—but should not be confused with—the type I and type II errors of an efficacy hypothesis test.

Progression should not be a mechanical vote across isolated indicators. Criteria should be mapped to feasibility objectives, evaluated jointly, interpreted with qualitative evidence and external changes, and linked to explicit actions.[5]

7. Estimating a continuous-outcome standard deviation

A pilot may estimate the standard deviation σ needed to size a future continuous-outcome trial. Under independent normally distributed observations,

$$\frac{\nu S^2}{\sigma^2}\sim\chi^2_{\nu},$$

where S² is the variance estimate and ν is its degrees of freedom. A two-sided 100(1 − α)% confidence interval for σ is

$$\left( S\sqrt{\frac{\nu}{\chi^2_{1-\alpha/2,\nu}}}, S\sqrt{\frac{\nu}{\chi^2_{\alpha/2,\nu}}} \right).$$

With small ν, this interval is asymmetric and wide. Plugging S directly into the definitive-trial formula treats an uncertain estimate as known and creates a material risk of underpowering the future trial.

More defensible strategies include:

• using a prespecified one-sided upper confidence limit for σ;
• inflating the definitive-trial size to account for uncertainty in S;
• combining pilot data with relevant external evidence through meta-analytic or Bayesian methods;
• evaluating the probability that the definitive trial achieves its target power, called assurance;
• using an internal-pilot nuisance-parameter reassessment when scientifically and statistically appropriate.

Simulation work has shown that moderately sized external pilots can still estimate design parameters imprecisely, and uncertainty should be propagated rather than hidden.[6] Methods that optimise the combined pilot-plus-definitive sample size are available for normally distributed continuous outcomes, but their recommendations are conditional on their objective and assumptions.[7]

8. Why “12 per arm” is not a universal answer

Rules of thumb are attractive because they require few assumptions. A commonly cited rule is 12 participants per treatment arm.[8] Other recommendations are larger. The apparent disagreement reflects different objectives: learning study logistics, estimating an SD, controlling an interval width, or optimising total recruitment do not generate the same answer.

For example, Whitehead and colleagues proposed stepped per-arm sizes for a two-arm external pilot with a normally distributed continuous outcome when the aim is to estimate variability efficiently for a future trial: roughly 75, 25, 15, and 10 per arm for extra-small, small, medium, and large standardised target effects, respectively.[7] These are not universal minimums; they arise from a specific optimisation framework.

Use a rule of thumb only when:

• its original objective matches the pilot objective;
• its assumptions fit the proposed design;
• its uncertainty is acceptable;
• resource constraints and expected losses are addressed; and
• the protocol explains why that rule is relevant.

“Ten percent of the definitive trial” is especially weak unless that fraction happens to meet a defined precision or decision criterion.

9. Do not power the future trial from the pilot treatment effect

In a small pilot, the treatment-effect estimate has high sampling variability. Conditional on being noteworthy or published, it may also be subject to selection and optimism. Substituting that estimate into a conventional power formula can yield a severely under- or oversized definitive trial.

The definitive trial’s target difference should usually be a clinically important difference justified by patient, clinical, and policy considerations—not the effect that happened to appear in the pilot. Pilot treatment-effect estimates may be displayed descriptively with uncertainty, but formal efficacy testing is generally inconsistent with the feasibility purpose.[1][9]

An effect-based progression rule is defensible only if it has been designed as a genuine decision problem, with adequate precision or a calibrated Bayesian decision model and explicit consequences. Calling an underpowered efficacy analysis “exploratory” does not make it reliable.

10. Binary event rates and other nuisance parameters

For a future trial with a binary primary outcome, the control-group event probability may affect the required sample size. A small pilot estimate can be unstable, particularly for rare outcomes. Better practice is to:

1. synthesise relevant trials, registries, or cohort data;
2. assess transportability to the proposed population and follow-up time;
3. define a plausible range rather than a single convenient value;
4. calculate the definitive sample size across that range; and
5. consider blinded internal reassessment when allowed by the design.

The same principle applies to dropout, adherence, within-person correlation, count overdispersion, and covariate-outcome correlation: uncertainty in a nuisance parameter should be reflected in design sensitivity analyses.

11. Cluster-randomised pilot studies

In a cluster design, increasing participants within a small number of clusters cannot replace learning across clusters. Feasibility outcomes may be correlated within sites, practices, schools, wards, or communities. For equal cluster size m and intracluster correlation ρ, the familiar design-effect approximation is

$$DE=1+(m-1)\rho,$$

but pilot design should not consist of mechanically multiplying an individually randomised sample size by DE.

The number of clusters, variability in cluster size, cluster-level recruitment and retention, site heterogeneity, and degrees of freedom all matter. External cluster pilots are commonly too small to estimate the ICC precisely; a single pilot ICC should not be treated as known when sizing the definitive cluster trial.[10] For a binary feasibility outcome, dedicated methods can select numbers of clusters and participants per cluster to achieve a desired maximum likely error while allowing for intracluster correlation.[11]

Practical priorities include:

• representativeness and number of clusters;
• ability to recruit and randomise clusters;
• intervention delivery and contamination across clusters;
• cluster and participant retention;
• feasibility of the intended cluster-level or mixed-model analysis;
• sensitivity of the definitive sample size to a credible ICC distribution.

12. Repeated measures and longitudinal pilots

Longitudinal definitive trials may require assumptions about:

• marginal variances at each visit;
• correlations between visits;
• variance of change from baseline;
• treatment-by-visit patterns;
• missingness and treatment discontinuation;
• convergence of the intended mixed model.

A small pilot covariance matrix is noisy and may not be positive definite under an elaborate unstructured model. Do not estimate every covariance element and then treat it as fixed truth. Instead:

• identify which parameters materially change the definitive sample size;
• combine pilot and external longitudinal evidence;
• use parsimonious covariance assumptions for planning;
• run scenario analyses over plausible covariance structures;
• simulate the definitive design using the planned analysis and missing-data process.

Simulation is particularly useful when feasibility rules are multivariate, denominators vary over time, recruitment is site dependent, or the final analysis is nonlinear.

13. Attrition and unequal allocation

If an objective requires nE evaluable participants and the probability of evaluability is 1 − r, a simple recruitment target is

$$n_R=\left\lceil\frac{n_E}{1-r}\right\rceil.$$

However, apply this only when nE genuinely refers to evaluable observations. If attrition or outcome completion is itself the feasibility parameter, the recruited sample is its denominator and should not be inflated for the same attrition a second time.

For a two-arm pilot, unequal allocation may be appropriate when one arm supplies most of the information—for example, intervention fidelity or a new device’s failure rate. The allocation ratio should follow the information need, ethics, costs, and intended comparison rather than automatically copying the definitive trial.

14. A complete worked design

Consider an external, individually randomised pilot of a behavioural intervention. The future trial requires:

• Week-12 outcome availability near 80%;
• reliable delivery of a new electronic assessment;
• an estimate of the continuous outcome SD;
• a progression decision agreed with patient and clinical stakeholders.

Objective A: estimate outcome availability

Desired approximate 95% confidence-interval half-width: 0.08. With p₀ = 0.80, the requirement is 97 recruited participants.

Objective B: observe technical failures

A failure probability of 5% is considered important and the desired detection probability is 95%. The requirement is 59 independent electronic assessments. If each participant completes repeated assessments, within-person dependence must be considered; 59 assessments do not necessarily equal 59 independent opportunities.

Objective C: estimate SD

The team considers the Whitehead framework, prior studies, and sensitivity of the definitive sample size to the SD. Its candidate requirement is 50 evaluable participants, with the definitive calculation to use an uncertainty-adjusted SD rather than the raw pilot SD.

Objective D: progression

Stakeholders define 65% outcome availability as unacceptable and 80% as acceptable. Candidate sample sizes and integer cut-offs are evaluated using binomial probabilities. Amber results trigger prespecified improvements to reminders and visit windows; green supports progression; red prompts redesign or stopping. Fidelity and qualitative acceptability are interpreted jointly rather than by an automatic checklist vote.

Final target

The largest direct requirement is 97 recruited participants. The team rounds to 100 to preserve equal allocation and simplify site targets, documents that this provides about 49–50 per arm, and reports the precision and progression-rule operating characteristics. The round number is the consequence of a calculation—not the justification itself.

15. Reproducible R calculations

The following base-R functions reproduce common building blocks. They are planning aids, not substitutes for design-specific validation.

  # Approximate total n for a binomial proportion CI half-width
n_precision_proportion <- function(p, half_width, conf = 0.95) {
  alpha <- 1 - conf
  z <- qnorm(1 - alpha / 2)
  ceiling(z^2 * p * (1 - p) / half_width^2)
}

  # Independent opportunities needed to observe >=1 problem
n_detect_problem <- function(problem_probability, detection_probability = 0.95) {
  ceiling(
    log(1 - detection_probability) /
      log(1 - problem_probability)
  )
}

  # Probability of a green decision: observed proportion >= green_cutoff
prob_green <- function(n, true_p, green_cutoff) {
  critical_count <- ceiling(n * green_cutoff)
  pbinom(critical_count - 1, size = n, prob = true_p, lower.tail = FALSE)
}

  # Multiplicative one-sided upper confidence factor for a normal SD
sd_upper_factor <- function(df, conf = 0.95) {
  sqrt(df / qchisq(1 - conf, df = df))
}

n_precision_proportion(p = 0.80, half_width = 0.08)
n_detect_problem(problem_probability = 0.05)
prob_green(n = 100, true_p = 0.80, green_cutoff = 0.80)
sd_upper_factor(df = 48)

For the final design, enumerate integer cut-offs and exact probabilities, reproduce the intended confidence-interval method, and test edge cases. For a complex study, simulation code should generate data under each scenario, apply the intended pilot analysis and progression rule, and tabulate decision probabilities.

16. Bayesian and assurance-based extensions

A Bayesian design can combine external information with pilot data and express decisions directly as probabilities. For example, one might proceed if

$$\Pr(p\geq p_A\mid\text{pilot data})\geq q,$$

where pA is an acceptable feasibility rate and q is a prespecified posterior probability threshold.

Before using the rule, calculate its prior-predictive or frequentist operating characteristics under plausible values of p. A strong prior can dominate a small pilot, so prior-data conflict and sensitivity to prior choice must be examined.

Assurance averages the conditional power of the future trial over uncertainty in nuisance parameters or treatment effects. Unlike ordinary power—which conditions on fixed assumed values—assurance answers a design-stage question such as: “Given present uncertainty, what is the probability that the future trial will meet its success criterion?” This can be more honest than choosing a favourable point estimate, but it requires a defensible probability distribution for the uncertain parameters.

17. Common errors and their repairs

18. Protocol-ready sample size template

The primary pilot objective is to estimate [feasibility parameter] among [analysis population and denominator]. We anticipate a value of [planning value] and require a [confidence level] interval with approximate half-width [value]. Using [method and distribution], this requires [number] analysable units. We will recruit [number] units after accounting for [distinct loss mechanism, if applicable]. The chosen design also gives [probability] probability of observing at least one [problem] if its true probability is [value]. Progression thresholds were agreed with [stakeholders]. For each threshold, we calculated the probabilities of proceed, amend, and stop under unacceptable, intermediate, and acceptable scenarios. Because [clusters/repeated observations/sites] create dependence, the calculation incorporates [ICC/correlation/model or simulation]. The pilot is not powered for treatment efficacy; treatment-effect estimates will be descriptive and will not be used alone to determine the definitive-trial target difference.

Replace every bracketed field. If the sample size is constrained, add the precision actually attainable and explain why it remains decision-useful.

19. Analysis and reporting checklist

• State whether the study is an external pilot, internal pilot, or another feasibility design.
• Identify the future definitive study and the uncertainties the pilot addresses.
• Define each feasibility parameter, numerator, denominator, time window, and analysis population.
• Distinguish participant, observation, site, practitioner, and cluster sample sizes.
• Name the interval method, confidence level, planning value, and desired precision.
• Give formulas, software, version, and reproducible code or simulation settings.
• Round every requirement upward and explain any later operational rounding.
• State assumptions about attrition, clustering, repeated observations, and missing data.
• Prespecify progression thresholds, associated actions, and decision operating characteristics.
• Explain how multiple progression criteria will be interpreted jointly.
• Separate clinically important target effects from pilot effect estimates.
• Propagate uncertainty in SDs, event rates, ICCs, and correlations into definitive-trial scenarios.
• Report estimates with uncertainty even when a traffic-light category is used.
• Document deviations from the planned pilot design and their implications for progression.
• Follow the CONSORT pilot and feasibility extension for a randomised pilot trial.[1]

20. Questions for doctoral-level reflection

1. Is the pilot’s principal inferential target a parameter, a decision, or the performance of a process?
2. Which error is more costly: progressing an infeasible trial or abandoning a feasible one?
3. Does the sample size criterion average over plausible nuisance-parameter uncertainty or condition on one selected value?
4. Are the pilot setting and participants transportable to the future trial?
5. Would additional sites, clusters, or calendar time provide more information than additional participants at existing sites?
6. How will qualitative findings modify a quantitative progression decision?
7. If several feasibility outcomes are jointly required, what is the probability of an overall green decision under realistic dependence?
8. What information would be gained by an internal rather than external pilot, and what adaptations would require error-control methods?
9. At what sample size does further pilot recruitment cease to be more efficient than beginning the definitive trial?
10. Is the proposed pilot answering a feasibility question, or disguising an underpowered efficacy study?

Key takeaways

1. There is no universally correct pilot sample size.
2. Every number should trace back to a pilot objective, analysis, acceptable uncertainty, and decision.
3. Precision and decision-error calculations are usually more appropriate than efficacy power calculations.
4. Treatment-effect estimates from small pilots should not ordinarily determine the definitive target effect.
5. Nuisance-parameter uncertainty must be propagated rather than ignored.
6. In cluster and longitudinal designs, the number and structure of independent units matter as much as the total count of observations.
7. A constrained design can be defensible only when its attainable information is shown to be useful.
8. Transparent progression criteria, reproducible calculations, and sensitivity analyses turn a pilot from a small study into a rigorous design stage.

References

1. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, Lancaster GA. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239. Available from: https://doi.org/10.1136/bmj.i5239

2. Ying X, Freedland KE, Powell LH, Stuart EA, Ehrhardt S, Mayo-Wilson E. Determining sample size for pilot trials: a tutorial. BMJ. 2025;390:e083405. Available from: https://doi.org/10.1136/bmj-2024-083405

3. Friede T, Kieser M. Blinded sample size reassessment in non-inferiority and equivalence trials. Statistics in Medicine. 2003;22(6):995–1007. Available from: https://doi.org/10.1002/sim.1456

4. Viechtbauer W, Smits L, Kotz D, Budé L, Spigt M, Serroyen J, Crutzen R. A simple formula for the calculation of sample size in pilot studies. Journal of Clinical Epidemiology. 2015;68(11):1375–1379. Available from: https://doi.org/10.1016/j.jclinepi.2015.04.014

5. Mellor K, Albury C, Dutton SJ, Eldridge S, Hopewell S. Recommendations for progression criteria during external randomised pilot trial design, conduct, analysis and reporting. Pilot and Feasibility Studies. 2023;9:59. Available from: https://doi.org/10.1186/s40814-023-01291-8

6. Teare MD, Dimairo M, Shephard N, Hayman A, Whitehead A, Walters SJ. Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: a simulation study. Trials. 2014;15:264. Available from: https://doi.org/10.1186/1745-6215-15-264

7. Whitehead AL, Julious SA, Cooper CL, Campbell MJ. Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Statistical Methods in Medical Research. 2016;25(3):1057–1073. Available from: https://doi.org/10.1177/0962280215588241

8. Julious SA. Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics. 2005;4(4):287–291. Available from: https://doi.org/10.1002/pst.185

9. Sim J, Lewis M. The size of a pilot study for a clinical trial should be calculated in relation to considerations of precision and efficiency. Journal of Clinical Epidemiology. 2012;65(3):301–308. Available from: https://doi.org/10.1016/j.jclinepi.2011.07.011

10. Eldridge SM, Costelloe CE, Kahan BC, Lancaster GA, Kerry SM. How big should the pilot study for my cluster randomised trial be? Statistical Methods in Medical Research. 2016;25(3):1039–1056. Available from: https://doi.org/10.1177/0962280215588242

11. Hemming K, Taljaard M, Gkini E, Bishop J. Sample size determination for external pilot cluster randomised trials with binary feasibility outcomes: a tutorial. Pilot and Feasibility Studies. 2023;9:163. Available from: https://doi.org/10.1186/s40814-023-01384-1