Designing Questionnaire-Based Surveys

Overview

A questionnaire-based survey collects standardised information by asking respondents a planned set of questions. The questionnaire is only one component of the survey design. The study also requires a defined population, sampling or recruitment method, mode of administration, contact strategy, consent process, data-quality plan and analysis aligned with the research question.[1–3]

Good questionnaire design begins with the construct and intended use of each item. It does not begin by writing as many questions as possible. Existing validated instruments should be considered before new items are created, but evidence of validity must fit the intended population, language, mode and interpretation.

This lecture focuses on designing a questionnaire-based survey as a complete research procedure. Instrument validation and cultural adaptation are addressed in Part 5.

Why this matters

A poorly designed questionnaire can create systematic error that no large sample or advanced statistical model can correct. Respondents may interpret vague words differently, forget events outside a realistic recall period, select an answer that does not fit, or abandon an unnecessarily long form.

Survey error may arise from:

• coverage of the wrong population;
• selection and non-response;
• question comprehension;
• memory retrieval;
• judgement and estimation;
• response mapping;
• interviewer or mode effects;
• data processing; and
• misleading analysis or reporting.

Key concepts

Begin with a construct map

Each research objective should map to a construct or factual variable, and each questionnaire item should have a purpose. A construct map can classify content as:

• eligibility and screening;
• factual characteristics;
• behaviours and events;
• knowledge;
• attitudes or beliefs;
• experiences and satisfaction;
• barriers and facilitators;
• open-ended elaboration; and
• administrative metadata.

Items that will not inform an objective, eligibility decision, adjustment set or quality check should usually be removed.

Questions are cognitive tasks

To answer, respondents must understand the question, retrieve relevant information, judge what answer is appropriate and map that answer to the offered response options. Problems at any stage produce measurement error.

When to use this approach

DECISION GUIDE

Conditions that support a questionnaire survey

• The required information is known or perceived by respondents and cannot be obtained more accurately from another source.
• Standardised questions can capture the intended variables across the population.
• The administration mode is accessible and acceptable.
• Recall periods are realistic for the events being measured.
• The burden is proportionate to the study value.
• The team can pretest the instrument and monitor the field process.

Questionnaires are less suitable when direct measurement, observation, records or qualitative exploration would answer the question more validly. For a new or poorly understood phenomenon, interviews may be needed before closed-response items are finalised.

Research question and methodological fit

The survey objective is refined to:

Describe the prevalence and distribution of prespecified access, knowledge and service barriers among women eligible for postpartum glucose testing, and examine how these barriers differ by completion status.

The survey will not claim that current barrier reports caused earlier non-completion unless the timing and wording establish that the barrier existed before the testing deadline.

The team decides that a single total barrier score is not automatically meaningful. Distinct domains may need separate reporting unless scale-development evidence supports a total score.

Design, setting and study population

DESIGN PROFILE

Questionnaire-based cross-sectional survey

Core structure: A defined sample provides standardised responses once through a self-administered or interviewer-administered questionnaire.

Best suited for: Knowledge, attitudes, reported behaviours, experiences and barriers that require respondent report.

Important limitations: Recall, social-desirability, comprehension, mode, non-response and common-method biases.

The intended respondent must be specified. Questions written for clinicians, patients and caregivers require different language and assumptions. Proxy response should be allowed only when methodologically appropriate, and the proxy status should be recorded.

Sampling and recruitment

The survey plan should define:

• how eligible respondents are identified;
• whether the sample is probability-based, consecutive or purposive;
• who makes first contact;
• number and timing of reminders;
• incentives or reimbursement;
• alternative modes for accessibility;
• handling of duplicate responses; and
• disposition codes for non-participation.

PROTOCOL CHECKLIST

Survey contact and administration plan

• Use a standard invitation that identifies the study and legitimate sender.
• Protect sensitive information in email, SMS and envelopes.
• Separate reminders from coercive repeated contact.
• Record eligibility, contact, refusal, partial completion and completion.
• Prespecify whether respondents can pause, return or skip questions.
• Offer language and accessibility options where feasible.
• Pilot the complete contact-to-completion process, not only the item text.

Variables, constructs and measurements

Write focused questions

Questions should generally address one idea, use concrete language and state the reference period. Avoid:

• double-barrelled items;
• leading or loaded wording;
• unexplained abbreviations;
• assumptions that do not apply to all respondents;
• vague frequency terms;
• unnecessarily sensitive detail; and
• negative or double-negative constructions.

Design response options

Response options should be mutually exclusive, collectively adequate and ordered when the construct is ordered. Include “not applicable”, “do not know” or “prefer not to answer” only when these are legitimate states and plan how they will be analysed.

Agreement scales are not appropriate for every construct. Factual frequency is often better measured with frequency categories or counts. Knowledge is better assessed with answerable statements or scenarios than self-rated agreement such as “I know the guideline”.

Revising problematic survey items

illustrative

Table 1. Illustrative item revision

Problematic item	Problem	Revised approach
“The clinic was convenient and staff were helpful.”	Two constructs	Ask separate access and communication items
“Did you regularly attend appointments?”	Undefined frequency and period	“How many scheduled visits did you attend during the first 6 months after delivery?”
“You knew testing was important, didn’t you?”	Leading wording	“Before discharge, how important did you think postpartum testing was?”
“I had no difficulty not arranging childcare.”	Double negative	“How difficult was it to arrange childcare for the testing visit?”
“What was your glucose result?”	Recall may be inaccurate	Ask whether tested; obtain result from records with permission

Note: Final wording requires cognitive testing with the intended population.

Questionnaire order and visual design

Begin with relevant, engaging and non-threatening questions. Group related items, use transitions and place sensitive or burdensome items after trust is established. Screening and branching rules must prevent respondents from seeing irrelevant questions.

Visual design should make the response task apparent. Consistent alignment, spacing, typography and instructions reduce errors. On paper, avoid page breaks that separate a question from its response options. On digital forms, test different screen sizes and assistive technologies.

Language, interviewer administration and accessibility

Translation should preserve conceptual meaning, response task and cultural relevance rather than substitute words mechanically. The process may include forward translation, reconciliation, review, cognitive testing and documentation of changes. A previously translated instrument still needs evaluation when the population, administration mode or health-system context differs.

Interviewer administration can include respondents with limited literacy and allows clarification, but interviewers can influence answers through tone, paraphrasing, probing or visible reactions. Training should define exactly when neutral clarification is allowed and how answers are recorded. Supervisors can review duration, missingness and interviewer-specific response patterns without encouraging staff to coach respondents.

Accessibility should be designed rather than added after launch. Consider font size, contrast, screen-reader labels, plain language, language choice, motor limitations, hearing and visual needs, and whether a proxy or assisted response is methodologically acceptable. The report should state when assistance was provided because assisted and unassisted responses may differ.

Cognitive interviewing provides direct evidence about how respondents process questions.[4] Field piloting then tests the full survey under realistic conditions, including contact, consent, routing, completion time and data export. These are complementary stages rather than interchangeable labels.[5]

Procedures and data collection

Development should include:

1. evidence review and construct map;
2. use or adaptation of existing items where appropriate;
3. expert review for relevance and comprehensibility;
4. cognitive interviews with intended respondents;
5. technical and usability testing;
6. pilot administration under field conditions;
7. revision with documented decisions; and
8. version control and final programming checks.

The team should document who developed the instrument, source of adapted items, permissions, languages, administration mode, completion time and scoring rules.[1,2]

Bias, validity and quality assurance

Quality assurance includes interviewer training, scripted neutral probes, monitoring of missing or patterned responses, checking branching logic, and reviewing open-text fields for unexpected interpretations.

APPRAISAL CHECKLIST

Appraising a questionnaire-based survey

• Does every major item or scale map to a stated objective or construct?
• Were items pretested with the intended population and mode?
• Are recall periods and response options appropriate?
• Is the instrument available or sufficiently described for appraisal?
• Were the sampling, contact and non-response processes reported?
• Are scores interpreted only within the available validity evidence?

Ethical and governance considerations

Interface with statistical analysis

The analysis plan should distinguish individual items, prespecified scales and derived scores. Reverse coding, score ranges, minimum items required and treatment of “not applicable” should be fixed before analysis.

Descriptive summaries should match the measurement scale. Reliability does not establish validity. Factor analysis, internal consistency and score comparisons should be undertaken only when the instrument and sample support those purposes.

Mode effects and measurement equivalenceMMed extension

Responses can differ between interviewer, paper, telephone and online modes because privacy, visual presentation, interviewer presence and response navigation change. A mixed-mode survey improves access only if mode-related differences are anticipated, monitored and, where important, evaluated for measurement equivalence.

Worked example

WORKED EXAMPLE

Reducing a 74-item draft to an answerable survey

The team creates a matrix linking each objective to required variables. Twenty-two copied items have no role and are removed. Factual testing completion is obtained from records rather than recall. Barrier items are organised into transport, time, childcare, communication and emotional domains without assuming they form one total scale.

Six women complete cognitive interviews in Malay and English. They reveal that “follow-up clinic” is interpreted as the infant clinic by some respondents, so the wording is revised to specify maternal glucose testing. The programmed survey is tested on mobile phones, and branching removes irrelevant items. Median completion time in the field pilot is 11 minutes.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

Why is ‘The appointment was convenient and the staff were helpful’ a weak item?

Answer and reasoning

It asks about two constructs. A respondent may find the appointment inconvenient but the staff helpful and cannot express both accurately in one response.

When should testing completion come from records rather than respondent recall?

Answer and reasoning

Records are preferable when they are sufficiently complete and accurately identify the event, because dates and results may be difficult to recall. External testing and incomplete records still need planned verification.

Does a high Cronbach alpha prove that a questionnaire is valid?

Answer and reasoning

No. Internal consistency describes relationships among items under assumptions; it does not prove that the intended construct is captured, that content is adequate or that the score supports the proposed use.

What is the purpose of cognitive interviewing?

Answer and reasoning

It examines how intended respondents understand, retrieve, judge and answer questions, revealing interpretation and response-process problems that expert review alone may miss.

Key take-home messages

SUMMARY

Key take-home messages

• Questionnaire design begins with objectives, constructs and intended interpretation.
• The questionnaire is one part of a complete survey design.
• Questions and response options should match respondents’ cognitive task and real experience.
• Pretesting must include intended respondents, the administration mode and the full workflow.
• Large samples cannot repair systematic measurement error created by weak items.

References

1. Artino AR Jr, La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No. 87. Med Teach. 2014;36(6):463–474. https://doi.org/10.3109/0142159X.2014.889814
2. Burns KEA, Duffett M, Kho ME, Meade MO, Adhikari NKJ, Sinuff T, et al. A guide for the design and conduct of self-administered surveys of clinicians. CMAJ. 2008;179(3):245–252. https://doi.org/10.1503/cmaj.080372
3. Kelley K, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. Int J Qual Health Care. 2003;15(3):261–266. https://doi.org/10.1093/intqhc/mzg031
4. Tourangeau R, Rips LJ, Rasinski K. The psychology of survey response. Cambridge: Cambridge University Press; 2000.
5. Willis GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks: Sage Publications; 2005.