Bias in Diagnostic Accuracy Studies

Overview

Bias in diagnostic accuracy studies is not a single problem. It can enter when participants are selected, when the index test is interpreted, when the reference standard is chosen or read, or when participants move through verification and analysis. QUADAS-2 organises these concerns into four domains: participant selection, index test, reference standard, and flow and timing.[1]

These design mechanisms can change sensitivity, specificity and related measures even when the calculations are mathematically correct. Empirical studies have shown that case-control sampling, differential verification and other design features can be associated with different accuracy estimates.[2,3]

For advanced readers, the goal is to move beyond memorising bias labels. A useful appraisal asks: what process changed who was classified, how they were classified, or who entered the analysis—and in what direction might that distort the target estimate?

Why this matters

Diagnostic studies are especially susceptible to bias because the index test and reference standard are applied to the same individuals and because the decision to verify disease may depend on earlier test results. A study may therefore create a feedback loop in which the index result influences whether or how the reference diagnosis is established.

Bias also affects downstream translation. If a test appears more accurate than it really is, clinicians may adopt an unsafe pathway. If an estimate is biased downward, a useful test may be abandoned. Risk-of-bias assessment should therefore inform protocol design, not merely be added during a later systematic review.

A causal way to think about diagnostic bias

Bias often arises because a design decision creates dependence between test results and classification or inclusion. Three questions help:

1. Who enters the study? Selection may depend on known disease or test results.
2. How is target-condition status determined? The reference may depend on the index test or use different methods for different participants.
3. Who enters the final analysis? Missing verification, indeterminate results or exclusions may depend on both index and disease status.

Domain 1 — participant selection

Participant-selection bias arises when inclusion or exclusion creates a systematically unrepresentative diagnostic contrast for the target question. Important mechanisms include:

• separate recruitment of obvious cases and healthy controls;
• excluding diagnostically difficult participants;
• selecting participants according to prior index-test results;
• inappropriate exclusions after enrolment; and
• convenience sampling that is related to target status and test performance.

The two-gate diagnostic case-control design can exaggerate accuracy when extreme groups are compared.[2,3] This is partly a spectrum problem and partly a selection problem. The relevant question is whether the recruited participants represent the clinical uncertainty in which the test will be used.

Domain 2 — index-test bias

Index-test interpretation should follow the intended future procedure. Bias can arise when:

• readers know the reference diagnosis when interpreting a subjective index test;
• thresholds are selected after examining accuracy in the same dataset;
• failed or indeterminate results are excluded selectively;
• index-test methods differ between participants; or
• technical processing is influenced by known clinical status.

A prespecified threshold protects against optimistic selection of the best-looking operating point. If threshold exploration is a genuine objective, label it as development and validate it in independent data.

Domain 3 — reference-standard bias

A reference standard must classify the target condition accurately enough for the intended question. Common problems include:

Incorporation bias

The index test contributes to the reference diagnosis. Agreement may increase mechanically because the test helps define the comparator.

Review bias

Reference assessors know the index result, or index assessors know the reference result, and this information influences subjective interpretation.

Imperfect reference classification

The reference standard itself makes errors. Observed sensitivity and specificity are then relative to an imperfect comparator rather than biological truth. More complex methods may be considered, but they require additional assumptions and cannot automatically reconstruct an unobserved perfect diagnosis.[4]

No gold standard does not mean no design standardMMed extension

When no single definitive reference exists, investigators may use expert consensus, composite rules, latent-class approaches or longitudinal clinical outcomes. Each strategy changes the estimand and introduces assumptions. The protocol should make those assumptions visible and conduct sensitivity analyses where feasible rather than labelling a convenient comparator “gold standard”.

Domain 4 — flow and timing

Partial verification bias

Only a subset receives the reference standard. If verification depends on index results, symptoms or other predictors of disease, the verified subset can yield biased accuracy estimates.[5]

Differential verification bias

Different participants receive different reference standards, often because index-positive participants undergo an invasive definitive procedure while index-negative participants receive clinical follow-up. Bias depends on the accuracy of each reference strategy and how assignment is related to disease and index status.

Disease progression or recovery

A long interval between index and reference procedures allows true target status to change. Discordance may then reflect timing rather than test error.

Exclusion after testing

Participants with indeterminate, conflicting or missing results may be removed from the main analysis. If these states are related to true disease or test difficulty, complete-case accuracy can be optimistic.

DESIGN PROFILE

QUADAS-2 risk-of-bias structure

Participant selection: Was the enrolled spectrum appropriate and free from avoidable selection mechanisms?

Index test: Was the index test conducted and interpreted without knowledge or data-driven decisions that would bias performance?

Reference standard: Was target-condition classification sufficiently valid and independent?

Flow and timing: Did participants receive appropriate verification with acceptable timing, and were all relevant participants included?

Designing out bias before recruitment

PROTOCOL CHECKLIST

Bias-prevention decisions to prespecify

• Use one clinically relevant recruitment gate where the target question is routine diagnostic accuracy.
• Define index-test thresholds and interpretation rules before examining outcome data.
• State exactly what clinical information is available to index and reference assessors.
• Apply the same reference strategy to all participants where ethically and practically possible.
• Set a clinically justified maximum interval between index and reference procedures.
• Record indeterminate, failed and missing results as explicit states.
• Maintain a screening and participant-flow log from eligibility through analysis.
• Separate algorithm development, tuning and evaluation datasets in AI-centred studies.

Worked mechanism map

Illustrative diagnostic-bias map

illustrative

Table 1. From design decision to possible distortion

Design decision	Mechanism	Main concern
Verify mostly index-positive participants	Index result affects reference availability	Partial verification bias
Use invasive reference for positives, follow-up for negatives	Different reference accuracy by index group	Differential verification
Include index result in expert diagnosis	Index contributes to comparator	Incorporation bias
Reader sees final diagnosis	Knowledge influences index interpretation	Review bias
Exclude borderline cases	Difficult cases removed	Selection/applicability bias
Reference performed months later	Disease status may change	Flow-and-timing bias

The table is a reasoning aid. More than one mechanism can operate at once, and the magnitude or direction of bias is not always predictable from the label alone.

AI-centred diagnostic accuracy studies

AI diagnostic studies add familiar and new pathways to bias: dataset curation, class imbalance, annotation quality, repeated images from the same participant, leakage between training and evaluation data, model versioning, subgroup performance and distribution shift. STARD-AI, published in 2025, extends diagnostic reporting for AI-centred studies with additional attention to dataset practices, the AI index test, evaluation, algorithmic bias and fairness.[6] The online article was updated following an Author Correction published on 13 July 2026.[7]

These reporting standards do not replace sound design. A perfectly documented data leak is still a biased evaluation. In particular, all observations from a participant should be allocated consistently across development and evaluation partitions unless the design explicitly models repeated observations.

Appraising direction and consequence

APPRAISAL CHECKLIST

Move from label to consequence

• What variable or process determines study entry, verification or analysis inclusion?
• Is that process related to the index result, target condition or both?
• Could test interpretation be influenced by knowledge unavailable in intended practice?
• Are difficult or indeterminate participants removed disproportionately?
• Does the reference strategy differ across groups?
• Could timing permit the target condition to change?
• Is the principal concern internal bias, applicability, or both?

Not every bias has a simple direction. For example, an imperfect reference standard can lower or sometimes alter apparent accuracy in more complex ways depending on correlated errors. Statements such as “bias always increases sensitivity” should therefore be avoided unless the mechanism and assumptions justify that conclusion.

Ethical and governance considerations

Interface with statistical analysis

Statistical correction cannot generally restore information that the design never collected. Inverse-probability or multiple-imputation approaches may address some missing verification under assumptions, and latent-class models may address imperfect references under assumptions, but sensitivity analysis should make dependence on those assumptions explicit.

Report raw participant flow before adjusted estimates. When multiple centres, readers or repeated observations are present, account for clustering. When thresholds are developed and evaluated, separate these stages. For AI models, retain a locked evaluation dataset and document model version, preprocessing and decision threshold.

Bias mechanisms in greater depth

Selective availability of the reference standard

Verification is rarely random in routine practice. Clinicians are more likely to order invasive procedures for people with alarming symptoms or positive preliminary tests. If a retrospective diagnostic study uses only people who happened to receive definitive verification, it conditions on a clinical decision influenced by information related to disease. The verified sample may therefore be systematically different from the population in whom the index test is intended to operate.

A prospective study can reduce this mechanism by offering a common verification strategy. When universal verification is unethical, the protocol can prespecify a defensible alternative and collect enough information on unverified participants to characterise selection. Statistical methods for verification bias require assumptions about the missing reference status; they should supplement, not replace, design reasoning.

Correlated errors between index and reference procedures

Errors are not always independent. Two imaging methods may both miss very small lesions; two assays may share susceptibility to specimen degradation. If index and reference errors are correlated, apparent agreement can be high even when both are wrong. This is especially important when a new method is technically similar to the comparator. Researchers should discuss shared mechanisms rather than assuming that disagreement is the only sign of reference-standard limitation.

Data-dependent analysis populations

Excluding participants can itself depend on the observed data. Removing “outliers”, borderline reference values or cases with poor-quality images after seeing results can create a cleaner and more separable sample. Quality exclusions should use prespecified criteria whenever possible, with counts and sensitivity analyses that show what happens when ambiguous cases are retained.

Applying QUADAS-2 prospectively

QUADAS-2 is usually encountered during systematic review, but its domain structure can be used prospectively as a design audit. Before recruitment, the team can ask whether participant sampling avoids inappropriate exclusions, whether the index threshold is prespecified, whether reference interpretation is sufficiently independent, and whether flow and timing are consistent. Every signalling question that cannot be answered favourably becomes a design issue to resolve or justify.

This prospective use improves protocol quality because it converts abstract “risk of bias” language into operational decisions. It also creates documentation for later reporting: who was blinded, how long tests could be separated, which participants could receive alternative verification, and what constituted an interpretable result.

Sensitivity analyses for diagnostic bias

Sensitivity analysis should be targeted to plausible mechanisms. Examples include treating indeterminate results as positive versus negative, repeating analyses after including protocol deviations, evaluating alternative assumptions for unverified participants, or comparing results under different defensible reference definitions. The aim is not to generate many estimates until one is favourable. It is to show how conclusions change under prespecified uncertainty about classification or flow.

Bias versus applicability

A study can have low risk of internal bias but high applicability concerns. For example, a carefully blinded study in a tertiary referral sample may estimate accuracy correctly for that referral sample yet remain poorly applicable to primary care. Conversely, a study can recruit the intended population but use biased verification. These dimensions should be reported separately rather than collapsed into one vague judgement of “quality”.

Advanced AI considerations

AI-centred diagnostic studies require particular care with reference labels because many models are trained on labels derived from routine reports. If those labels were generated using information unavailable to the model or reflect inconsistent expert practice, development and evaluation can inherit label bias. External evaluation should also consider device changes, preprocessing, prevalence shifts and subgroup performance. STARD-AI improves transparency, but transparent reporting cannot substitute for independent data, appropriate labels and a clinically realistic test pathway.[6]

Documenting residual bias after prevention

No diagnostic study is completely free of bias. After implementing prevention measures, the protocol should identify residual mechanisms that remain unavoidable. For example, a reference standard may still be imperfect, some participants may refuse verification, or readers may need limited clinical information to reproduce real practice. These should be described as design features with expected consequences rather than hidden as generic limitations.

Quantitative bias analysis can sometimes show how estimates change under plausible assumptions about missing verification or reference misclassification. Such analyses are most informative when assumptions are grounded in external evidence and ranges are prespecified. They should not be presented as a mathematical cure for unknown bias.

When several biases operate simultaneously, their directions may oppose one another. Selective verification could raise one accuracy measure while reference error lowers another. Therefore, a final judgement should integrate mechanisms and uncertainty rather than summing checklist scores. QUADAS-2 deliberately avoids a numerical total score for this reason: domain-specific judgement is more informative than a single quality number.

Reporting uncertainty about bias

Risk-of-bias judgements should not be translated into unsupported numerical corrections. When the likely magnitude of a bias is unknown, state that uncertainty directly. A transparent report can explain why an estimate may be optimistic or difficult to transport without pretending that the exact corrected sensitivity is known. This distinction between recognised mechanism and quantifiable magnitude is central to responsible appraisal.

Protocol deviations are diagnostic evidence

Record deviations by participant and by procedure. If blinding failed, verification was delayed or an alternative reference was used, retain those participants in the flow record and evaluate prespecified sensitivity analyses. Deleting deviations can convert a conduct problem into hidden selection bias.

A final protocol review should assign responsibility for each bias-control procedure so that blinding, timing and verification rules are operational rather than aspirational.

Worked example

WORKED EXAMPLE

Redesigning the postpartum verification plan

The original plan verified all index-positive participants but only index-negative participants who later developed concerning symptoms. The team recognises that the probability of definitive reference assessment would depend strongly on the index result.

They redesign the study so that all enrolled participants receive the same laboratory reference procedure during the same visit. The index threshold is prespecified. Device operators do not see the reference classification, laboratory assessors do not see the index classification, and failed tests remain in the participant-flow dataset.

If some women decline the reference procedure, their characteristics and index results remain recorded. The team will report the extent and reasons for missing verification and will conduct a prespecified sensitivity analysis if missingness is substantial. The redesign prevents an important bias rather than relying on later statistical repair.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

What is partial verification bias?

Answer and reasoning

It arises when only a subset of enrolled participants receives the reference standard and the probability of verification is related to the index result, disease status or their predictors. The verified subset may then give a distorted accuracy estimate.

How does differential verification differ from partial verification?

Answer and reasoning

Partial verification concerns some participants lacking reference assessment. Differential verification concerns participants receiving different reference standards whose accuracies may differ, often according to index results or clinical features.

Why can incorporation bias inflate apparent diagnostic performance?

Answer and reasoning

Because the index test contributes information to the reference classification used to judge that same test. This creates circular agreement rather than an independent comparison.

Can a latent-class model prove the true disease state when no perfect reference exists?

Answer and reasoning

No. It can estimate latent classifications under model assumptions, but those assumptions are not equivalent to observing biological truth. Results should be interpreted as model-dependent and supported by sensitivity analyses.

What is a key AI-specific evaluation threat when multiple images come from one participant?

Answer and reasoning

Participant-level data leakage. If related observations appear in both development and evaluation datasets, performance may be optimistic because the model is not being tested on genuinely independent participants.

Key take-home messages

SUMMARY

Key take-home messages

• Organise diagnostic bias around participant selection, index test, reference standard, and flow and timing.
• Explain the mechanism and consequence of a bias rather than only naming it.
• Prefer prevention through design over uncertain post-hoc statistical correction.
• Treat indeterminate results, verification patterns and timing as core study data.
• For AI-centred studies, prevent leakage and report dataset, model, fairness and evaluation details transparently.

References

1. Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
2. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282(11):1061–1066. https://doi.org/10.1001/jama.282.11.1061
3. Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174(4):469–476. https://doi.org/10.1503/cmaj.050090
4. Rutjes AWS, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PMM. Evaluation of diagnostic tests when there is no gold standard: a review of methods. Health Technol Assess. 2007;11(50):iii, ix–51. https://doi.org/10.3310/hta11500
5. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39(1):207–215. https://doi.org/10.2307/2530820
6. Sounderajah V, Guni A, Liu X, et al. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2025;31:3283–3289. https://doi.org/10.1038/s41591-025-03953-8
7. Sounderajah V, Guni A, Saria S, et al. Author Correction: The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2026. https://doi.org/10.1038/s41591-026-04570-9