Pilot Testing and Cognitive Interviewing

Overview

Questionnaire problems are often invisible to their authors. A sentence may appear clear yet trigger different interpretations, require information respondents cannot retrieve or offer response options that do not fit. Cognitive interviewing administers draft questions while collecting verbal evidence about how respondents understand and answer them [1–3].

A pilot test examines how the broader data-collection process works under conditions resembling the planned study. It can identify timing, recruitment, skip-pattern, device, workflow and data-management problems. A feasibility study addresses whether and how the main study can be undertaken, often using prespecified progression criteria. These activities overlap but answer different questions.

Pretesting improves an instrument before definitive use. It is not a ceremonial final step and does not validate a questionnaire merely because a small group completed it without complaint.

Why this matters

Ambiguous questions create systematic response error. Broken skip logic creates missing data. Long forms produce fatigue and straight-lining. Interviewers may reinterpret difficult items inconsistently. Electronic forms may truncate labels or prevent correction. These problems can be discovered before the main study if testing is designed to reveal them.

Iterative pretesting is more efficient than collecting a large dataset with flawed items. It also respects participants by reducing unnecessary burden and avoiding questions that are stigmatising, intrusive or impossible to answer accurately.

How respondents answer questions

A useful model describes four broad cognitive stages [4].

Comprehension

The respondent interprets the words, grammar, time frame and task. Problems arise from technical terms, vague references, double negatives, hidden assumptions and unclear units.

Retrieval

The respondent searches memory for relevant information. Recall becomes difficult when the period is long, events are repetitive or records are unavailable. Questions may need landmarks, bounded periods or recognition cues.

Judgement

The respondent evaluates retrieved information, estimates frequency or decides what counts. Social desirability, uncertainty and subjective thresholds influence judgement.

Response mapping

The respondent maps the judgement to the available options. Categories may be incomplete, overlapping, unbalanced or ordered differently from the respondent’s mental scale.

Cognitive-interviewing techniques

Think-aloud interviewing

Participants verbalise what comes to mind as they answer. This can reveal spontaneous interpretation and retrieval but may feel unnatural and requires practice. The interviewer should avoid teaching the participant how to answer.

Verbal probing

The interviewer asks targeted follow-up questions. Common probes include:

• comprehension: “What does ‘follow-up care’ mean to you here?”
• paraphrasing: “Can you repeat the question in your own words?”
• recall: “How did you remember the number of visits?”
• confidence: “How certain are you about that answer?”
• category selection: “Why did you choose ‘sometimes’ rather than ‘often’?”
• sensitivity: “How comfortable was this question?”

Probes may be scripted, emergent or combined. Scripted probes ensure coverage, while emergent probes investigate unexpected behaviour.

Observation

Hesitation, requests for clarification, changes of answer, skipped items and visible discomfort provide additional evidence. These observations should be interpreted cautiously and discussed with the participant rather than assumed to have one meaning.

Illustrative cognitive-interview guide

illustrative

Table 1. Probes for an access item

Stage	Draft item problem	Probe
Comprehension	“usual provider” may be unclear	Who were you thinking about when you answered?
Retrieval	several appointments may blend together	How did you remember which visit this question referred to?
Judgement	“difficult” has a personal threshold	What made the experience count as difficult or not difficult?
Response	categories may not fit	Was there another answer you wanted to give but could not?

Sampling for cognitive interviews

Cognitive-interview samples are purposive rather than statistically representative. Recruit participants likely to reveal different interpretations and difficulties. Relevant variation may include:

• age and education;
• language variety and bilingual use;
• literacy and digital experience;
• disease or care experience;
• rural and urban pathways;
• disability or sensory needs; and
• whether the question applies directly to the participant.

Sample size is determined by the diversity of the target population, complexity of the instrument, number of rounds and recurrence of important problems. Several small iterative rounds often produce more useful revisions than one large round conducted after wording is fixed [1–3].

Interviewer role and training

Cognitive interviewers need to remain curious and non-judgemental. They should not explain the intended meaning before the respondent has described their interpretation. Leading probes such as “You understood that this meant the glucose test, right?” suppress evidence of misunderstanding.

Training includes practising neutral probes, recording observations, distinguishing participant difficulty from interviewer error and documenting decisions. Audio recording can support analysis when consent and data-security arrangements permit.

Analysing cognitive-interview data

Analysis focuses on the relationship between the intended question and actual response processes. A matrix can organise findings by item, participant, cognitive stage, severity and proposed revision. Teams should distinguish:

• isolated personal preferences;
• recurring comprehension or response problems;
• problems affecting particular subgroups;
• construct-threatening problems; and
• operational issues unrelated to item meaning.

Frequency alone is not the decision rule. One rare misunderstanding may be critical if it reverses score meaning or affects a vulnerable subgroup. Revisions should be tested in another round because solving one problem can create another.

APPRAISAL CHECKLIST

Appraise cognitive-interview evidence

• Did participants reflect important diversity in the target population?
• Were probes designed around the item intent and response process?
• Did interviewers avoid teaching or leading respondents?
• Were findings analysed systematically across cases?
• Were revisions linked to evidence and retested?
• Were unresolved problems and subgroup differences documented?

Pilot testing the complete process

A pilot test moves beyond item cognition to the operational pathway. It may evaluate:

• eligibility screening and recruitment;
• consent and privacy procedures;
• questionnaire length and fatigue;
• interviewer instructions;
• electronic layout, branching and validation rules;
• linkage between forms and participant identifiers;
• missing, inconsistent and out-of-range data;
• time and staffing requirements;
• equipment and connectivity;
• multilingual administration; and
• transfer into the analysis dataset.

The pilot should resemble intended field conditions. Testing an electronic form only on the research team’s devices will not reveal problems with participants’ phones, screen sizes or internet access.

Testing interviewer-administered and electronic instruments

Interviewer-administered tools require testing of both the question and the interaction. Interviewers may skip introductory text, paraphrase complex wording, signal preferred answers or probe inconsistently. Behaviour coding, observation and audio review can identify these deviations. Standardised training should be revised when several interviewers encounter the same difficulty.

Electronic forms require more than checking that every button works. Test:

• display on common screen sizes;
• keyboard and assistive-technology access;
• language fonts and line wrapping;
• whether response options are visible without scrolling;
• prevention of impossible entries without blocking legitimate exceptions;
• saving, reconnection and duplicate submission;
• date and unit handling;
• skip logic after changing an earlier answer; and
• transfer into the analysis dataset.

The CDC cognitive-interviewing programme illustrates how question evaluation can combine systematic probing with documentation of response error [5]. General pretesting research also stresses that different methods identify different classes of problem [6]. Patient-focused guidance places participant understanding and burden within the broader context of identifying what matters to patients [7].

Turning findings into revision decisions

A revision log should record the item version, evidence, severity, proposed action and result of retesting. Problems can be classified as:

• critical: changes the construct, reverses score meaning or excludes an important subgroup;
• major: produces frequent ambiguity, missingness or operational failure; or
• minor: affects style or convenience without changing interpretation.

Not every participant preference requires a change. Contradictory suggestions should be resolved using item intent and patterns across participants. Retain unresolved issues when no wording works equally well for all groups, and consider interviewer support, branching or separate versions only when conceptually justified.

Pilot success criteria should be concrete. “The questionnaire worked well” is not auditable. Examples include median completion time, proportion of items requiring clarification, frequency of broken skip paths, proportion of submissions with identifier mismatch and number of critical cognitive problems remaining after the final round.

Sampling and analysis for an operational pilot

Pilot participants should encounter the same eligibility criteria, recruitment routes and field conditions planned for the main study. Include settings likely to challenge the workflow, such as busy clinics, low-connectivity locations and participants who require interviewer assistance. A pilot restricted to cooperative staff and highly connected participants gives false reassurance.

Prespecify operational outcomes and denominators. Examples include the proportion of eligible people approached, consent completion, median administration time, item non-response, follow-up success, duplicate identifiers and unresolved data queries. Present estimates with their uncertainty, recognising that a small pilot cannot estimate rare failures precisely.

Qualitative debriefing of participants and staff complements numerical summaries. A completion time may look acceptable while participants report fatigue or discomfort. Staff may complete the workflow only by using unofficial workarounds that would not be sustainable at scale.

The pilot report should distinguish problems fixed before launch from limitations that remain. If the main protocol changes materially, repeat the affected components rather than assuming that one pilot covers the revised process.

Pilot data and the main study

Whether pilot participants can be included in the main analysis depends on the design and changes made. Inclusion is problematic when eligibility, measurement, intervention or procedures change materially after the pilot. The protocol should decide prospectively or state criteria for inclusion. Any inclusion should be transparent and should not compromise blinding or independent confirmation.

Progression and stopping decisions

For a feasibility or internal pilot, prespecified progression criteria may address recruitment, retention, completion, fidelity or data quality. Thresholds should guide judgement rather than create automatic decisions detached from context. Red, amber and green zones can make decisions explicit, but rationales and uncertainties remain important.

DECISION GUIDE

Choose the appropriate pretesting activity

1. Is the main concern item meaning and response processes? Use cognitive interviewing.
2. Is the concern questionnaire layout or interviewer administration? Use structured pretesting and observation.
3. Is the concern recruitment, workflow, data capture or logistics? Use a pilot test.
4. Is the concern whether the whole study can proceed and how? Use a feasibility study with progression criteria.
5. Are substantive measurement properties being estimated? Plan a dedicated field evaluation with an adequate design.

PROTOCOL CHECKLIST

Plan iterative pretesting

• version and item intents to be tested;
• participant sampling characteristics;
• interview mode and recording plan;
• think-aloud and probing strategy;
• interviewer training;
• analysis matrix and severity criteria;
• revision and retesting cycles;
• pilot operational outcomes;
• progression or stopping criteria; and
• rules for including pilot data in the definitive study.

Behaviour coding and respondent-interviewer interactionMMed extension

In interviewer-administered surveys, behaviour coding records events such as interviewer misreading, respondent requests for clarification, inadequate answers and interviewer probing. It complements cognitive interviewing by identifying how questions function in more standardised field administration. Audio or video coding requires clear consent, secure storage and trained coders.

Deciding readiness for the main study

Readiness should be based on the absence of unresolved critical problems and the feasibility of correcting major problems before launch. Minor wording preferences can remain when they do not change interpretation or burden. The team should confirm that the final version, training materials, electronic form and data dictionary all match.

A brief final verification can test the locked materials without reopening development automatically. Any new problem should be triaged according to severity and documented. This prevents uncontrolled last-minute changes while preserving a pathway for genuine safety or validity concerns.

The final readiness decision and responsible approver should be recorded.

Interface with statistical analysis

Cognitive interviews generate qualitative evidence and are not analysed as prevalence surveys. Pilot data can be summarised to identify missingness, completion time, distributional problems and technical errors, but estimates are often too unstable for definitive inferential conclusions.

Field evaluation of a questionnaire requires a separate analysis plan for dimensionality, reliability and validity. Item deletion based only on small-sample correlations risks overfitting and loss of content.

Worked example

WORKED EXAMPLE

Three rounds of cognitive testing

Round 1: Six women complete the bilingual draft. Four interpret “follow-up appointment” as the baby’s visit. The team replaces the phrase with an explicit reference to maternal glucose testing and adds a defined time window.

Round 2: Five different women understand the target visit, but the response “not applicable” is selected when no appointment was offered. The team separates “I was not given an appointment” from difficulty ratings.

Round 3: Five women use the revised categories consistently. Electronic testing reveals that long Malay labels wrap poorly on small screens, so layout and font size are adjusted without changing wording.

Outcome: The process identifies construct, category and operational problems. It does not yet establish structural validity or reliability, which are evaluated in a later field sample.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

What does cognitive interviewing examine that ordinary pilot completion may miss?

Answer and reasoning

It examines how respondents comprehend the question, retrieve information, form a judgement and map it to a response. A respondent can complete every item while using an unintended interpretation.

Why are several small rounds often preferable to one large round?

Answer and reasoning

Iterative rounds allow the team to revise identified problems and test whether the revision works or creates new problems. One final round provides no opportunity to evaluate changes.

Should cognitive-interview participants be statistically representative?

Answer and reasoning

No. They are purposively selected to expose diverse response processes and likely problems. Population prevalence is not the target of cognitive interviewing.

What can a pilot study establish about reliability?

Answer and reasoning

A pilot may reveal gross instability or data problems, but a small convenience sample usually cannot provide precise, generalisable reliability evidence. A dedicated study with appropriate participants, repeats and confidence intervals is needed.

Key take-home messages

SUMMARY

Key take-home messages

• Cognitive interviewing investigates how answers are produced, not only whether questions are completed.
• Comprehension, retrieval, judgement and response mapping can each generate error.
• Use purposive, iterative rounds and retest revisions.
• Pilot testing examines the complete operational pathway under realistic conditions.
• Pretesting identifies problems but does not replace formal measurement-property evaluation.

References

1. Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Public Opin Q. 2007;71(2):287–311. https://doi.org/10.1093/poq/nfm006
2. Willis GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks: Sage; 2005. https://doi.org/10.4135/9781412983655
3. Presser S, Couper MP, Lessler JT, Martin E, Martin J, Rothgeb JM, et al. Methods for testing and evaluating survey questions. Public Opin Q. 2004;68(1):109–130. https://doi.org/10.1093/poq/nfh008
4. Tourangeau R, Rips LJ, Rasinski K. The psychology of survey response. Cambridge: Cambridge University Press; 2000. https://doi.org/10.1017/CBO9780511819322
5. Centers for Disease Control and Prevention, National Center for Health Statistics. Cognitive interviewing. Hyattsville: CDC; 2024. https://www.cdc.gov/nchs/ccqder/question-evaluation/cognitive-interviewing.html
6. Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12(3):229–238. https://doi.org/10.1023/A:1023254226592
7. US Food and Drug Administration. Patient-focused drug development: methods to identify what is important to patients. Silver Spring: FDA; 2022. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-methods-identify-what-important-patients