Pilot Studies and Operational Pretesting

Overview

Before a study launches at full scale, researchers should test whether the planned processes work in the environment where they will actually be used. Pilot and feasibility work can examine recruitment, consent, measurement, follow-up, data capture, staffing, timing, acceptability and system integration. Their primary purpose is to reduce uncertainty about whether and how the definitive study can be conducted, not to provide a small underpowered test of the final hypothesis.[1,2]

Operational pretesting is broader still. A team can test a form, SOP, contact workflow or electronic database without conducting a formal pilot study. What matters is that critical assumptions are challenged before they affect hundreds of participants.

Why this matters

Desk review rarely exposes every operational problem. A clinic may not have a private interviewing space at the planned time. The laboratory identifier expected in the protocol may not be available on discharge. A branching rule may trap users in the database. A follow-up call may take 25 minutes rather than the planned 8 minutes.

Discovering these issues during a short, controlled pretest allows correction. Discovering them halfway through recruitment creates inconsistent data, amendments, staff frustration and potentially avoidable participant burden.

Pilot, feasibility and pretest are not synonyms

Terminology varies, but a useful distinction is:

• Feasibility study: asks whether something can be done and how it should be done.
• Pilot study: a smaller-scale version of part or all of a future definitive study, often conducted to test feasibility.
• Instrument pretest: examines whether questions, forms or measurements function as intended.
• Cognitive interview: explores how respondents understand and answer questions.
• System/user acceptance testing: checks whether an electronic system and its configured workflow behave correctly.
• Dry run or simulation: uses fictitious or training cases to rehearse operations without enrolling research participants.

The label matters less than specifying the uncertainty being addressed.[1]

Identify critical uncertainties

Prioritise assumptions that could make the study infeasible or undermine validity. Examples include:

• Are enough eligible participants encountered each week?
• Can consent occur without disrupting urgent care?
• Are required source records available at the planned time?
• Can measurements be completed within the visit duration?
• Does the instrument show unacceptable missingness or misunderstanding?
• Can participants be contacted using approved channels?
• Does the EDC workflow support repeated visits and corrections?
• Can sites resolve data queries within the operational timeline?
• Are staffing and equipment demands realistic?

ICH E8(R1) encourages attention to critical-to-quality factors during study design; feasibility testing is one way to examine them empirically before full implementation.[3]

Progression criteria

When a pilot informs a larger study, define how the results will guide decisions. Progression criteria can be framed as:

• Proceed — operations meet acceptable thresholds.
• Proceed with modifications — problems are correctable without changing the essential design.
• Reassess — major uncertainties remain.
• Do not proceed as planned — core assumptions are not feasible or acceptable.

Thresholds should be justified rather than chosen solely for convenience. Some criteria are quantitative, such as a minimum recruitment rate; others are qualitative, such as unacceptable participant burden or an unresolved privacy concern.

PROTOCOL CHECKLIST

Operational pilot plan

• Feasibility questions and rationale.
• Setting, participants or test cases.
• Procedures to be rehearsed.
• Measures of recruitment, timing, completeness and failures.
• Staff and participant feedback where appropriate.
• Progression or modification criteria.
• Rules for documenting protocol or form changes.
• Decision about whether pilot participants/data can enter the main study.
• Ethics and consent implications of pilot participation.

Test normal and abnormal pathways

A database that works for ideal participants may still fail in real use. Build test cases for:

• ineligible screening;
• partial dates;
• missed visits;
• participant withdrawal;
• repeated laboratory tests;
• contradictory sources;
• external results;
• values at plausible extremes;
• skipped or refused questionnaire items;
• corrections after form completion; and
• transfer or downtime scenarios.

Likewise, field teams should rehearse foreseeable exceptions. A pilot is valuable precisely because it challenges the happy path.

Pilot the workflow, not only the questionnaire

Researchers often say they “piloted the questionnaire” when the main risks lie elsewhere. Review the entire participant journey from identification through final follow-up. Map handoffs between clinical staff, research assistants, laboratory systems and data personnel. Observe waiting time and competing responsibilities.

WHO field-survey programmes emphasise interviewer preparation and testing of operational procedures before full fieldwork.[4]

Do not overinterpret pilot effectiveness estimates

Small pilot studies typically produce highly uncertain estimates of treatment effects, prevalence or associations. A nominally significant result can be unstable; a non-significant result does not prove absence of effect. Pilot sample sizes should be justified by feasibility objectives rather than by pretending the pilot is a small definitive trial.[1,2]

If estimating a nuisance parameter such as a standard deviation for later sample-size planning, recognise that small samples provide uncertain estimates. Use external evidence and sensitivity analysis where possible rather than relying on one unstable pilot value.

Can pilot participants enter the main study?

An internal pilot is designed so that eligible pilot data contribute to the definitive study under prespecified conditions. An external pilot is separate. Combining pilot and main-study data may be reasonable when the protocol, intervention, outcome definitions and procedures remain sufficiently unchanged and the design accounts for the pilot appropriately.[2]

If pilot findings lead to substantial changes in eligibility, measurement, intervention or follow-up, pooling may be inappropriate. Decide prospectively where possible and document the rationale.

Worked example

WORKED EXAMPLE

From pilot findings to operational changes

The postpartum pilot enrols 18 participants and uses 10 additional fictitious database scenarios.

Findings include:

• median baseline workflow takes 31 minutes rather than the planned 15;
• six participants cannot identify the clinic name expected in one questionnaire item;
• the discharge appointment field is absent from one site’s standard record;
• two database branches hide the external-laboratory source field incorrectly;
• weekend discharges cannot be contacted by the planned research staff; and
• no participant objects to the proposed reminder message after consent.

The team shortens non-essential baseline items, rewrites the clinic-access question after cognitive review, creates a site-specific source rule for appointment documentation, fixes the branching logic and adds a Monday workflow for weekend discharges. None of these changes is interpreted as evidence about the study’s eventual exposure-outcome association.

Documenting pilot-driven changes

Maintain a structured issues log containing the problem, evidence, decision, responsible person, action, date and affected documents. This prevents pilot feedback from becoming an undocumented collection of informal suggestions.

When a change affects the ethics-approved protocol, participant information, consent, eligibility or significant procedures, follow the applicable approval process before implementation. Changes to forms, manuals and electronic systems should also be version controlled.[5]

DECISION GUIDE

Choose the lightest adequate pretest

1. Can the uncertainty be answered with document review?
2. If not, can fictitious cases or a dry run answer it?
3. Is representative staff testing required?
4. Is participant involvement necessary to assess comprehension, acceptability or workflow?
5. Does the uncertainty require a formal pilot of recruitment and follow-up?
6. What evidence will trigger proceed, modify or stop decisions?

Interface with data quality assurance

Pilot findings should become preventive controls. If the pilot reveals frequent date errors, add a date-consistency check. If staff cannot find a source field, revise the abstraction manual. If follow-up fails because contact details are incomplete, improve baseline verification rather than accepting high attrition as inevitable.

The next lecture treats ongoing data-quality assurance and monitoring. Piloting is the pre-launch component of the same quality-by-design philosophy supported in current GCP guidance.[5]

Pilot sample size follows the feasibility questionMMed extension

There is no single universal sample-size formula for pilot studies. A pilot intended to estimate recruitment yield may require enough calendar time or sites to represent recruitment variation; a pilot assessing a binary process failure may need enough observations to have a reasonable chance of observing that failure; a cognitive pretest may use iterative small rounds until major comprehension problems have been addressed.

The correct justification therefore begins with the parameter or process being learned, the precision or decision needed, and the cost of proceeding with uncertainty. This is different from powering a definitive efficacy hypothesis.[1,2]

Designing progression criteria that support decisions

Progression criteria should be linked to what would make the definitive study credible and feasible. Avoid a long list of arbitrary green–amber–red thresholds simply because this format is fashionable. For each criterion, state the consequence of falling below the threshold and whether modification is realistically possible.

Recruitment illustrates the point. A pilot recruitment rate below target may be acceptable if one easily correctable bottleneck is identified; the same numerical rate may be unacceptable if the eligible population is smaller than assumed. Decision criteria therefore need contextual interpretation rather than mechanical scoring.

Use process metrics, not only participant counts

Operational piloting can generate useful metrics such as:

• proportion of screened patients for whom eligibility can be determined;
• proportion of consent encounters completed in the available workflow;
• median time per research procedure;
• proportion of critical fields available from the proposed source;
• frequency and type of database errors;
• follow-up contact yield by attempt and channel;
• number and age of unresolved data queries; and
• staff or participant reasons for workflow failure.

These metrics identify where a process breaks, which is often more actionable than a single overall completion percentage.

Pilot across important variation

A multicentre study should consider whether one pilot site represents the others. If sites differ in information systems, staffing, language, patient volume or follow-up pathways, test the aspects most likely to vary. It may not be necessary to run a full pilot at every site; targeted site-readiness testing can complement a central pilot.

Similarly, test procedures at difficult times or circumstances when these are expected during the main study. A workflow that functions only on weekday mornings has not been adequately tested for a study recruiting around the clock.

Decide what must be frozen before launch

The purpose of piloting is to allow change, but endless refinement can delay the study and create version instability. Before opening full recruitment, identify which elements must be finalised: primary outcome definition, critical forms, database structure, consent materials, core SOPs, access permissions and quality reports.

Lower-risk wording or administrative details can still change through controlled processes, but the team should not enter full-scale recruitment with unresolved critical definitions.

Learn from near failures

A pilot incident that does not cause actual harm or data loss can still reveal vulnerability. Examples include a participant nearly receiving the wrong survey link, a record almost being duplicated, or a laboratory unit being misinterpreted but caught by a vigilant staff member. Record and analyse these near failures because the same weakness may produce a real error at larger scale.

APPRAISAL CHECKLIST

Appraise a pilot or operational pretest

• Are the feasibility uncertainties stated explicitly?
• Do measures correspond to those uncertainties?
• Are realistic difficult pathways included?
• Does the pilot represent important site or workflow variation?
• Are progression decisions specified before results are interpreted?
• Are pilot findings prevented from being overread as definitive effectiveness evidence?
• Are resulting changes documented and version controlled?
• Is there a clear point at which critical systems are ready for full launch?

Separate feasibility findings from protocol amendments

Not every pilot observation requires a protocol amendment. Some findings can be addressed by clarifying a manual, improving staff scheduling or fixing database configuration without altering the scientific design. Other findings—such as changing eligibility, the primary outcome, participant procedures or consent information—may require formal governance review.

Classifying changes by their effect helps the team respond proportionately. Keep a pilot decision log showing the finding, its methodological importance, the proposed change, the documents affected and whether ethics or other approval is required. This also creates a transparent explanation if readers later ask how the definitive study differed from the pilot.

A final readiness meeting should confirm ownership of every unresolved pilot issue and record why any residual uncertainty is acceptable before recruitment expands.

Common mistakes

Knowledge check

What is the main question of a feasibility study?

Answer and reasoning

Whether and how the proposed research can be conducted as intended. Feasibility work should examine operational uncertainties rather than primarily test the final scientific hypothesis.

Why should a database pretest include contradictory and missing values?

Answer and reasoning

Real studies contain exceptions. Testing only ideal records cannot reveal whether validation rules, branching, query workflows and missing-data states behave safely under realistic conditions.

A pilot finds p = 0.03 for the intervention effect. Should the team conclude the intervention works?

Answer and reasoning

No. A feasibility pilot is usually not designed or powered to provide a reliable definitive effect estimate. The result is highly vulnerable to imprecision and should not displace the planned definitive evaluation.

When may pilot data be incorporated into the main study?

Answer and reasoning

When this is compatible with the design, eligibility, procedures and outcomes, and the decision has been planned or justified. Major pilot-driven changes can make pooling inappropriate.

Key take-home messages

SUMMARY

Key take-home messages

• Define the operational uncertainty before choosing a pilot or pretest.
• Test recruitment, workflow, instruments, systems and failure pathways—not only ideal cases.
• Use progression criteria to turn pilot findings into explicit decisions.
• Do not overinterpret unstable effectiveness estimates from small feasibility samples.
• Convert pilot problems into revised procedures and preventive quality controls.
• Keep pilot-driven changes ethically governed, documented and version controlled.

References

1. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1. https://doi.org/10.1186/1471-2288-10-1
2. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239. https://doi.org/10.1136/bmj.i5239
3. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH harmonised guideline: General considerations for clinical studies E8(R1). 6 October 2021. https://database.ich.org/sites/default/files/ICH_E8-R1_Guideline_Step4_2022_0204%20%281%29.pdf
4. World Health Organization. WHO STEPS surveillance manual: the WHO STEPwise approach to noncommunicable disease risk factor surveillance. Geneva: World Health Organization. https://www.who.int/teams/noncommunicable-diseases/surveillance/systems-tools/steps
5. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH harmonised guideline: Good Clinical Practice E6(R3). Step 4. 6 January 2025. https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf