Online and Digital Surveys

Overview

Online and digital surveys use web forms, mobile applications, email links, QR codes, messaging platforms or electronic kiosks to collect questionnaire data. Digital delivery can reduce data-entry work, enable branching and reach geographically dispersed populations. It can also produce severe selection, duplication, authentication, privacy and device-compatibility problems.[1–3]

An online survey is not a sampling method. Posting an open link on social media usually creates a self-selected sample with an unknown denominator. A secure invitation sent to a probability sample is methodologically different even though both surveys appear in a browser.

This lecture explains how to design, secure, test and report digital surveys without assuming that technological convenience equals methodological quality.

Why this matters

Digital surveys are attractive because they are fast and inexpensive to distribute. The true cost may appear later as unusable responses, exclusion of people with limited digital access, uncertainty about who responded and inability to calculate a meaningful response rate.

Potential digital-specific problems include:

• duplicate or automated submissions;
• forwarded personalised links;
• open-link participation outside the target population;
• incompatibility with mobile devices or screen readers;
• platform metadata that collect more information than intended;
• insecure storage or third-party transfer;
• break-off caused by long forms;
• rapid fraudulent completion for incentives; and
• undercoverage of people with limited connectivity or literacy.

Key concepts

Distinguish closed and open surveys

A closed survey restricts access to a defined invited sample, commonly through unique links, authentication or tokens. The denominator and contact history may be known.

An open survey allows anyone with the link to respond. It may be useful for public consultation or exploratory recruitment but usually lacks a known probability of selection and response denominator.

Access control and anonymity are separate decisions

A survey can verify eligibility or prevent duplicate entry without storing identity in the analysis dataset. For example, an invitation system can mark a token as used while the response database stores only a random study identifier.

Absolute anonymity may prevent follow-up, withdrawal of submitted data, linkage and duplicate checking. Researchers should promise only the level of anonymity or confidentiality the system actually provides.

When to use this approach

DECISION GUIDE

Conditions that support digital administration

• The target population has adequate access to and familiarity with the selected technology.
• The sampling and invitation process can be documented.
• The questionnaire is suitable for self-administration.
• Privacy and data-hosting requirements can be met.
• Mobile, browser and accessibility testing can be completed.
• Duplicate, fraudulent and ineligible entries can be addressed proportionately.
• Alternative modes are available when digital exclusion would materially bias the study.

Digital administration may be inappropriate for highly vulnerable populations without safe private access, urgent clinical assessments, complex consent or tasks requiring direct examination.

Research question and methodological fit

The hospital’s objective is to estimate burnout prevalence and associated work characteristics among employed clinical staff. A closed survey using the current human-resources staff list is preferable to an open QR code because the target population and invitation denominator can be defined.

If management is involved, the team must address fear of identification and power relationships. A third-party or research-unit invitation process, minimal metadata and aggregated reporting may support trust.

Design, setting and study population

DESIGN PROFILE

Closed online staff survey

Core structure: A defined invited sample receives secure access to a digital questionnaire during a specified field period.

Best suited for: Literate, digitally connected populations where self-administration and automated data capture are appropriate.

Important limitations: Digital coverage, break-off, device effects, authentication, privacy concerns and non-response.

The protocol should identify the survey platform, hosting location, data controller, access permissions, retention, audit trail and whether IP addresses, cookies, timestamps or device data are collected.

Sampling and recruitment

A probability sample or census invitation can be implemented digitally. The contact plan should include:

• pre-notification where appropriate;
• individual invitation;
• clear sender identity;
• study purpose and estimated time;
• unique access process;
• non-coercive reminders;
• fieldwork start and close dates; and
• disposition records.

PROTOCOL CHECKLIST

Digital invitation and access controls

• Define whether the survey is open or closed.
• Use unique tokens when a known sample is invited.
• Separate invitation identifiers from response content where possible.
• State whether links can be forwarded or reused.
• Prespecify reminder frequency and stop reminders after completion.
• Provide a non-digital or assisted option when exclusion is important.
• Document complete, partial, refused, undeliverable and unknown cases.

A completion rate among people who opened the survey is not the same as a response rate among eligible invited people. Standard disposition definitions should be used when applicable.[4]

Variables, constructs and measurements

Digital questionnaires can use branching, validation, range checks and calculated fields. These features reduce some errors but can introduce hidden logic failures. Required fields should be used sparingly; forcing answers to sensitive questions can increase inaccurate responses or abandonment.

Digital survey quality-control matrix

illustrative

Table 1. Illustrative controls

Risk	Preventive control	Review indicator
Duplicate submission	Single-use token	Reused-token attempts
Ineligible response	Eligibility screen linked to invitation frame	Screen-out count
Automated/fraudulent response	Rate limits and plausibility checks	Implausibly rapid or patterned submissions
Mobile display failure	Responsive testing on common devices	Device-specific break-off
Sensitive-data exposure	Data minimisation and encrypted hosting	Access logs and incident review
Branching error	Test cases covering every path	Missing or contradictory fields

Note: Fraud indicators should not be used as automatic deletion rules without review.

Procedures and data collection

Before launch, conduct:

1. content and cognitive pretesting;
2. usability testing on desktop and mobile;
3. accessibility testing;
4. branching and validation test cases;
5. security and permission review;
6. invitation and reminder testing;
7. database export and coding verification; and
8. a soft launch to a small subset where appropriate.

During fieldwork, monitor delivery failures, starts, completions, break-off location, unusual timing and help requests. Do not repeatedly inspect substantive outcome comparisons and change recruitment reactively without documentation.

Digital paradata and response quality

Digital platforms can record paradata about the response process, such as invitation wave, start and completion time, device class, break-off page and edits. These data can help identify technical failures and unusual patterns, but they are not direct measures of honesty or validity.

Quality review should combine several indicators. A rapid completion time may be concerning when accompanied by straight-lining, contradictory responses, failed attention checks and implausible open text. Any exclusion algorithm should be fixed before substantive analysis, manually reviewed where feasible and reported with the number of responses affected.

Attention checks should be used cautiously. Confusing trick questions can exclude attentive respondents, people using assistive technologies or those with limited language proficiency. Direct consistency and plausibility checks linked to the study content are often more defensible than arbitrary instructions hidden in long item blocks.

Adaptive questionnaires can reduce burden by showing only relevant sections. The exported dataset must retain enough routing information to distinguish a structurally skipped item from a displayed but unanswered item. Otherwise, valid skip patterns may be misclassified as missing data.

The CHERRIES framework emphasises reporting development, recruitment, administration, participation, prevention of multiple entries and analysis for web surveys.[1] CROSS provides complementary reporting items for survey studies more broadly.[5]

Bias, validity and quality assurance

Digital surveys can also create mode effects. Respondents may disclose more sensitive information privately online than to an interviewer, but they may misunderstand complex questions without assistance.

APPRAISAL CHECKLIST

Appraising an online survey

• Was the survey open or closed, and how was eligibility established?
• What population and denominator does the invitation process support?
• Were duplicate, automated and partial responses handled transparently?
• Was the form tested across devices, languages and accessibility needs?
• What metadata were collected and where were data stored?
• Were view, participation, completion and response metrics defined correctly?

Ethical and governance considerations

Prize draws require separation of contact details from survey responses where anonymity is promised. Staff surveys should avoid reporting small combinations of role, unit and demographic characteristics that could enable re-identification.

Interface with statistical analysis

The analysis dataset should retain non-identifying process variables needed for quality assessment, such as completion status, duration, device class or invitation wave, when ethically approved. Exclusion rules for duplicates, implausible responses and partial completions should be prespecified and accompanied by sensitivity analyses.

For known samples, calculate and label outcome rates using a recognised definition. For open surveys, report participation counts and recruitment reach honestly; do not fabricate a response rate from social-media followers or estimated page views.

Poststratification does not fully solve opt-in selectionMMed extension

Weighting an opt-in sample to match age and sex totals can improve balance on those variables but does not guarantee balance on unmeasured factors related to response and outcome. Model-based adjustment requires strong assumptions and should not be presented as converting a convenience sample into a probability sample.

Managing partial responses

Define what constitutes a complete response before launch. Some studies require the primary outcome and key eligibility items rather than every optional question. Report partial completions separately and describe whether their available data were analysed. Excluding all partial responses can discard valid information and increase selection if break-off relates to burden, device or the topic. Including them without transparent denominators can create inconsistent samples across analyses.

Worked example

WORKED EXAMPLE

Redesigning the hospital burnout survey

The team obtains an authorised list of 1,420 eligible clinical staff and generates unique invitation tokens. The invitation system stores staff email and completion status, while responses are stored under random identifiers without names. Staff can complete the survey on mobile or desktop, and a paper option is available through an independent research office.

Two reminders are sent only to non-completers. The questionnaire does not collect IP addresses. The report gives the number invited, undeliverable, ineligible, partial and complete, and calculates the selected response-rate definition. Sensitivity analyses compare estimates with and without partial responses and examine variation by invitation wave.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

What is the methodological difference between a closed and open online survey?

Answer and reasoning

A closed survey restricts participation to a defined invited sample, supporting a known contact process and denominator. An open survey allows self-selection by anyone with the link and usually lacks known inclusion probabilities.

Can an anonymous survey prevent duplicate responses?

Answer and reasoning

It can, if access control is separated from response content—for example, a single-use token is marked as used while the response database stores no direct identity. The actual privacy architecture must be described accurately.

Why is the number of social-media followers not a valid response-rate denominator?

Answer and reasoning

Followers may not see the post, may be ineligible, may include duplicate or inactive accounts and are not a defined invited sample. The probability of exposure and eligibility is unknown.

What should be done with unusually fast responses?

Answer and reasoning

They should be reviewed using prespecified, multi-indicator quality criteria. Speed alone does not prove fraud; short legitimate paths or highly familiar respondents may complete quickly.

Key take-home messages

SUMMARY

Key take-home messages

• Online administration is a mode, not a sampling method.
• Closed surveys support stronger denominator and duplicate control than open links.
• Privacy promises must match platform metadata and data architecture.
• Digital surveys require device, accessibility, logic and security testing.
• Response metrics and exclusion rules must be defined and reported transparently.

References

1. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004;6(3):e34. https://doi.org/10.2196/jmir.6.3.e34
2. Bethlehem J. Selection bias in Web surveys. Int Stat Rev. 2010;78(2):161–188. https://doi.org/10.1111/j.1751-5823.2010.00112.x
3. Couper MP. Web surveys: a review of issues and approaches. Public Opin Q. 2000;64(4):464–494. https://doi.org/10.1086/318641
4. American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 10th ed. Alexandria: AAPOR; 2023. https://aapor.org/wp-content/uploads/2023/05/Standards-Definitions-10th-edition.pdf
5. Sharma A, Minh Duc NT, Luu Lam Thang T, Nam NH, Ng SJ, Abbas KS, et al. A consensus-based checklist for reporting of survey studies (CROSS). J Gen Intern Med. 2021;36(10):3179–3187. https://doi.org/10.1007/s11606-021-06737-1