Non-Response, Attrition and Selection Bias

Overview

Non-response and attrition are not merely reductions in sample size. They can change who remains observable and therefore change the estimate itself.

Unit non-response occurs when an eligible sampled person provides no study data. Item non-response occurs when some variables are missing. Attrition is loss of previously enrolled participants over time. Analytical exclusion occurs when available participants are removed from a particular analysis.

This lecture explains how these processes create selection bias, why response rate alone is insufficient, and how prevention, documentation, weighting, missing-data methods and sensitivity analyses should be combined.

Why this matters

Suppose women who face the greatest barriers to postpartum testing are also hardest to contact. The observed completion rate among respondents will tend to overstate completion. A conventional confidence interval may be narrow, but it reflects only random uncertainty under the observed-data model; it does not automatically include bias from missing non-respondents.

The direction and magnitude of bias depend on:

• who is missing;
• why they are missing;
• what was observed before loss;
• whether missingness is related to the exposure and outcome;
• the analysis model; and
• assumptions used to recover or bound missing information.

A 90% response rate can still be biased if the missing 10% is highly distinctive. A 60% response rate can sometimes yield acceptable estimates after strong follow-up and adjustment when non-response is well explained by measured variables. The rate is an operational signal, not a complete bias diagnosis.

Key concepts

Unit non-response

A sampled eligible unit does not participate at all. The researcher may know only limited frame information, such as age, location or facility.

Item non-response

A participant contributes some data but not a specific item or variable. Causes include refusal, recall difficulty, skip-pattern errors, technical failure or interviewer omission.

Attrition

An enrolled participant ceases to provide later observations. Attrition is especially important in cohort and trial research because it can be related to prognosis, intervention experience or the outcome.

Administrative censoring and planned missingness

Not all absence is problematic non-response. Follow-up may end administratively at a common study date, and planned subsamples may complete expensive measurements by design. These mechanisms must be distinguished from unplanned missingness.

How selection bias arises

Let $S=1$ indicate inclusion in the analysis. Conditioning on $S=1$ can create a distorted association when selection is influenced by variables related to both exposure and outcome, or by consequences of them.

Examples include:

• participants with severe symptoms are more likely to withdraw from an intervention study;
• people with both low income and poor outcomes are less likely to respond to a survey;
• diagnostic-study participants receive the reference standard only after a positive index test;
• complete-case analysis excludes participants because a covariate affected by exposure is missing; and
• researchers remove “non-adherent” participants after randomisation.

The same missingness proportion can have different effects under different mechanisms.

DECISION GUIDE

Investigate non-response and attrition

1. Identify the stage at which data became missing.
2. Record the reason, timing and variables affected.
3. Compare observed baseline characteristics by response status.
4. Examine whether the missingness pathway may involve exposure, outcome, prognosis or their causes.
5. Assess whether auxiliary data are available for adjustment.
6. Choose an analysis consistent with explicit assumptions.
7. conduct sensitivity analyses for plausible departures from those assumptions.
8. report uncertainty and limitations without claiming that a test proves no bias.

Response rates and disposition categories

A response rate requires a defined denominator. Researchers should distinguish:

• known eligible respondents;
• known eligible non-respondents;
• known ineligible units;
• units of unknown eligibility;
• unreachable units;
• refusals;
• partial respondents; and
• cases excluded for protocol or data-quality reasons.

Changing the denominator definition can change the reported rate. The calculation and disposition categories should therefore be specified.

Response rates should also be examined by site, time and key frame characteristics. A stable overall rate may conceal worsening response in one subgroup.

Outcome-response assessment

illustrative

Table 1. Baseline characteristics by outcome availability

Characteristic	Outcome observed, n = 300	Outcome unavailable, n = 60
Rural residence	28%	52%
No private transport	22%	47%
Returned to work before 6 weeks	35%	58%
Received documented reminder	81%	43%

Note: Values are illustrative. Differences describe observed patterns; they do not prove the missingness mechanism or eliminate unmeasured differences.

Prevention before analysis

Prevention is usually more credible than statistical repair. Strategies include:

• inclusive, accurate sampling frames;
• participant-centred contact procedures;
• multiple modes of data collection;
• short, clear instruments;
• real-time missing-item checks where appropriate;
• outcome linkage independent of continued intervention adherence;
• contact verification;
• flexible follow-up windows;
• collection of strong auxiliary variables;
• monitoring by site and subgroup; and
• rapid correction of technical or workflow failures.

Researchers should collect reasons for non-response without pressuring participants to justify withdrawal.

Analytical approaches

Complete-case analysis

Complete-case analysis uses participants with all variables required for a model. It can be valid under specific assumptions, but it is not universally unbiased. It also reduces precision and may change the target population.

Available-case analysis

Different summaries use all participants with the relevant variable observed. Denominators vary and covariance structures may become inconsistent.

Weighting

Non-response weights increase the contribution of respondents who represent similar sampled non-respondents. They require response-propensity information, adequate overlap and correct weight construction. Highly variable weights can reduce precision.

Multiple imputation

Multiple imputation replaces each missing value with several plausible values drawn from an imputation model, analyses each completed dataset and combines results. It requires an appropriate model, inclusion of variables related to missingness and outcome, and compatibility with the substantive analysis.

Model-based likelihood approaches

Mixed models and other likelihood methods can use partially observed repeated data under assumptions about the missingness mechanism. Their ability to use incomplete trajectories does not make missingness irrelevant.

Sensitivity analysis

Sensitivity analysis examines results under assumptions not identified by the observed data, such as worse outcomes among non-respondents, pattern-mixture shifts, tipping-point values or bounds.

Attrition in trials and longitudinal studies

The participant-flow diagram should separate:

• intervention discontinuation;
• withdrawal of consent;
• loss to contact;
• death;
• administrative end of follow-up;
• unavailable primary outcome;
• major protocol deviation; and
• analytical exclusion.

For treatment-policy questions, outcome collection should usually continue after treatment discontinuation if consent permits. Per-protocol analyses may be supplementary but can introduce selection because adherence is post-randomisation.

In observational cohorts, loss may depend on both exposure and prognosis. Baseline adjustment alone may not resolve this, especially when time-varying factors affect both later response and outcome.

PROTOCOL CHECKLIST

Non-response and attrition plan

• disposition categories and denominator definitions;
• prevention and contact procedures;
• variables available for non-respondents;
• reasons and timing of missingness;
• subgroup and site monitoring;
• primary analysis population;
• rules for partial records;
• weighting, imputation or likelihood methods;
• variables included in missing-data models;
• sensitivity analyses;
• reporting of outcome-specific denominators; and
• governance of post hoc analytical decisions.

Ethical and governance considerations

Interface with statistical analysis

Missing-data handling is part of the estimand and analysis plan, not a cleaning step performed after results are seen.

Analysts should specify:

• the population to which inference is intended;
• which variables are missing and at what times;
• the assumed missingness mechanism;
• the primary method;
• auxiliary variables;
• number and diagnostics of imputations where used;
• design weights and clusters;
• sensitivity analyses; and
• comparison of conclusions across assumptions.

See the related Biostatistics lecture on missing data. The methodological lecture identifies the selection process; the Biostatistics lecture develops analytical implementation.

Missing at random is an assumption conditional on the modelMMed extension

“Missing at random” means that, after conditioning on included observed information, the probability of missingness does not depend on the missing value itself. It does not mean missingness occurred randomly in everyday language. The assumption becomes more plausible when the model includes strong predictors of missingness and outcome, but it cannot be confirmed from observed data alone.

Worked example

WORKED EXAMPLE

Assessing the possible direction of attrition bias

Observed result: 216 of 300 women with known status completed testing: 72%.

Missing group: 60 women are more likely to live rurally, lack transport and have no documented reminder. These factors plausibly reduce completion.

Complete-case interpretation: 72% is the observed proportion among women with ascertainable status, not automatically the cohort proportion.

Simple sensitivity scenarios:

• If 72% of the missing group completed, the full-cohort proportion remains 72%.
• If 50% completed, the full-cohort proportion is $(216+30)/360=68.3\%$.
• If 30% completed, it is $(216+18)/360=65.0\%$.

Next analysis: use linked auxiliary data and a prespecified missing-data method, then conduct sensitivity analyses over plausible outcome differences.

Conclusion: The plausible direction is downward, but the exact bias cannot be identified from observed data alone.

Suggested protocol or reporting wording

Evidence basis and critical appraisal

Causal work on selection bias demonstrates why restricting analysis to observed participants can create or distort associations [1]. Meta-analytic evidence shows that response rate and non-response bias are only weakly and inconsistently related; the mechanism matters more than one threshold [2]. STROBE explanatory guidance therefore emphasises participant flow, missing data and discussion of bias rather than reassurance based on response percentage [3].

Official statistical-quality frameworks treat non-response as one component of total survey error and encourage documentation, evaluation and adjustment [4]. Modern missing-data texts provide the foundations for likelihood, weighting, imputation and sensitivity methods [5].

Critical appraisal should distinguish three questions:

1. How much is missing? Report denominators by variable and time.
2. Why is it missing? Use reasons, timing, design knowledge and auxiliary variables.
3. How sensitive is the conclusion? Compare methods and plausible departures from the primary assumption.

Diagnosing patterns before modelling

A missingness report should display outcome availability by randomised group or exposure, site, calendar period and baseline predictors. It should also examine monotone dropout versus intermittent missingness and whether technical failures affected batches or devices.

These descriptions do not identify the mechanism, but they guide prevention and model specification. For example, missing laboratory results caused by one interface outage require a different response from missing results among participants who avoided follow-up after adverse events.

Analysts should avoid stepwise decisions based only on statistical tests of missingness predictors. Variables can be important auxiliary predictors even when their individual p-values are not small.

Reporting uncertainty about bias

Researchers should avoid writing that missing data “did not affect the results” unless this is supported under clearly stated assumptions. A more defensible report explains which analyses were performed, how conclusions changed, and what unmeasured departures remain possible.

Common mistakes

Knowledge check

Why can a 90% response rate still be biased?

Answer and reasoning

The missing 10% may be systematically different in a way strongly related to the outcome or association. Bias depends on the selection mechanism, not only the fraction missing.

What is item non-response?

Answer and reasoning

A participant provides some study information but one or more specific items or variables are missing.

Does a non-significant responder–non-responder comparison prove comparability?

Answer and reasoning

No. The comparison may have low power, cannot examine unmeasured variables and does not directly test differences in unobserved outcomes.

Why are sensitivity analyses needed after multiple imputation?

Answer and reasoning

Multiple imputation commonly relies on a missing-at-random assumption conditional on included variables. Sensitivity analyses examine plausible departures, such as systematically worse missing outcomes.

What is the first response to preventable attrition?

Answer and reasoning

Improve the study design and follow-up process. Statistical methods should complement, not replace, prevention and outcome ascertainment.

Key take-home messages

SUMMARY

Key take-home messages

• Separate unit non-response, item non-response, attrition and analytical exclusion.
• Bias depends on why data are missing and how selection relates to study variables.
• A response rate is an operational indicator, not proof of validity.
• Prevent missingness and collect informative auxiliary data.
• State the missing-data assumptions and analysis population explicitly.
• Use sensitivity analyses when untestable assumptions could affect the conclusion.

References

1. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15(5):615–625. https://doi.org/10.1097/01.ede.0000135174.63482.43
2. Groves RM, Peytcheva E. The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opin Q. 2008;72(2):167–189. https://doi.org/10.1093/poq/nfn011
3. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4(10):e297. https://doi.org/10.1371/journal.pmed.0040297
4. Federal Committee on Statistical Methodology. A framework for data quality. Washington, DC: Office of Management and Budget; 2023. https://nces.ed.gov/FCSM/
5. Little RJA, Rubin DB. Statistical analysis with missing data. 3rd ed. Hoboken, NJ: Wiley; 2019.