Diagnostic Accuracy Study Design

Overview

A diagnostic accuracy study asks how well an index test classifies or quantifies a target condition when compared with an appropriate reference standard. The test may be a laboratory assay, imaging procedure, clinical score, device, algorithm or combination of findings. Accuracy is not an intrinsic property of a test alone: estimates depend on the population, clinical setting, positivity threshold, reference standard, participant flow and timing.[1,2]

The most useful accuracy study therefore begins with the intended use. A test used to rule out disease in primary care faces a different spectrum of patients and a different clinical consequence of errors from a confirmatory test used in a specialist clinic. STARD 2015 emphasises transparent description of intended use, participant selection, test methods, reference standards, flow and accuracy estimates so readers can judge both validity and applicability.[1]

This lecture focuses on design. Detailed calculation of sensitivity, specificity, predictive values, likelihood ratios and receiver-operating-characteristic measures belongs to the linked Biostatistics lectures.

Why this matters

A large sample and a precise confidence interval cannot rescue an unsuitable design. Accuracy may appear exaggerated if clearly diseased cases are compared with healthy controls, if only index-positive participants receive definitive verification, or if the reference standard is interpreted with knowledge of the index test. Empirical research has shown that design features of diagnostic studies can materially alter estimates.[3,4]

For clinicians, these distortions matter because an apparently excellent test can perform much less well in the intended pathway. For researchers, design decisions made before recruitment determine whether the two-by-two table has a defensible clinical interpretation.

Key concepts

Start with the target condition and intended role

The target condition is the condition that the index test is intended to detect, exclude or quantify. It must be operationally defined before recruitment. The intended role may be replacement, triage or add-on testing. These roles affect acceptable trade-offs between false-positive and false-negative results and influence where the test should be evaluated.[2,5]

Index test

The index test is the test being evaluated. Its procedure, specimen handling, reader training, software version and positivity threshold should be prespecified where possible. A threshold selected after inspecting the study data can overstate performance and requires transparent reporting.

Reference standard

The reference standard is the best available method for classifying the target condition in the study. It need not be perfect. The protocol should explain why it is appropriate, how indeterminate results are handled, and whether its interpretation can be kept independent of the index test.[2]

Intended population and pathway

Accuracy estimates apply most directly to people resembling those enrolled. Recruitment should therefore reflect the population in whom the test would actually be used, including realistic disease severity, alternative diagnoses and clinical uncertainty. This is an applicability issue as well as a statistical one.

When to use this approach

DECISION GUIDE

When a diagnostic accuracy study fits the question

• The principal question concerns how well an index test identifies a prespecified target condition.
• Participants can receive both the index test and a suitable reference standard with acceptable timing.
• Recruitment can approximate the clinical population and pathway in which the test will be used.
• The study can prespecify thresholds or clearly distinguish prespecified from exploratory thresholds.
• Test interpretation can be standardised and, where appropriate, blinded.
• Ethical and practical consequences of verification procedures are acceptable.

A diagnostic accuracy study is not sufficient when the main question is whether introducing a test improves patient outcomes, service use or cost-effectiveness. Those are clinical utility or impact questions and usually require a comparative implementation or experimental design.[5]

Research question and methodological fit

A design-ready question for the scenario is:

Among women attending 12-week postpartum follow-up after gestational diabetes, how accurately does the prespecified point-of-care test identify persistent dysglycaemia defined by the laboratory reference procedure?

The unit of observation is the participant. The index-test result and reference classification should refer to the same clinically relevant state. If several specimens, readers or repeated measurements are included per participant, the additional correlation must be reflected in design and analysis rather than treating all readings as independent people.

The primary estimands might include sensitivity and specificity at a prespecified threshold with 95% confidence intervals. Predictive values can also be clinically useful but depend on the prevalence of the target condition in the evaluated population. Threshold-free summaries such as area under the receiver-operating-characteristic curve answer a different question and should not replace threshold-specific performance when clinical decisions use a threshold.

Design, setting and study population

DESIGN PROFILE

Prospective single-gate diagnostic accuracy study

Core structure: A clinically relevant series of participants enters through one recruitment pathway and receives both the index test and reference standard.

Best suited for: Estimating test accuracy in a population that resembles intended use.

Important limitations: Verification burden, imperfect reference standards, missing or indeterminate results, timing effects and changes in disease state between tests.

A strong protocol describes the clinical setting before listing sample-size calculations. Specify whether participants are symptomatic, high risk, screen-detected or referred; whether the test is used before or after other investigations; and whether recruitment occurs in one or several centres. Restricting the sample to obvious disease and obvious non-disease may improve logistical efficiency but can seriously damage clinical applicability.

Sampling and recruitment

PROTOCOL CHECKLIST

Recruitment and flow items to prespecify

• Target population, source population and recruitment setting.
• Consecutive, random or other recruitment method and recruitment dates.
• Eligibility criteria that do not depend inappropriately on known index-test results.
• Whether all participants receive the same reference standard.
• Maximum acceptable interval between index and reference procedures.
• Management of withdrawals, missing tests and indeterminate results.
• Whether participants receive additional clinical care because of either test result.

Consecutive recruitment is often attractive because it reduces investigator discretion and better represents the intended spectrum. However, “consecutive” should be operationalised: eligible clinic sessions, screening logs, reasons for non-enrolment and exclusions after enrolment should be documented.

Variables, measurements and the participant-level dataset

Illustrative diagnostic accuracy data structure

illustrative

Table 1. Minimum participant-level fields

Field	Example operationalisation	Timing
participant_id	Unique study identifier	Enrolment
index_result	Continuous point-of-care measurement	Visit
index_positive	Prespecified threshold: yes/no	Derived after test
reference_result	Laboratory measurement(s)	Same visit
target_condition	Reference-standard positive/negative	After reference assessment
indeterminate_status	Index/reference/none	Recorded explicitly
verification_complete	Yes/no and reason	End of diagnostic work-up

Note: Values and thresholds are intentionally not specified here because the example is methodological rather than clinical guidance.

Preserve continuous index results even when the primary analysis uses a threshold. Record the reason for every missing or indeterminate test. Avoid building the final analytical table only from complete pairs, because this can hide systematic exclusions.

Procedures and data collection

The sequence of procedures matters. Ideally the index test is performed using the protocol intended for future practice; the reference standard follows within an interval short enough that true disease status is unlikely to change. Readers should receive only the clinical information that would normally be available in the intended pathway. If the index or reference interpretation is subjective, reader training, reproducibility procedures and adjudication rules should be stated before data collection.

A diagnostic study also needs a participant-flow record. The final report should make visible how many people were screened, enrolled, received the index test, received the reference standard, produced indeterminate results and entered each analysis.[1]

Bias, validity and quality assurance

Other major concerns include differential verification, partial verification, incorporation of the index test into the reference standard, review bias, inappropriate exclusions, prespecified versus data-derived thresholds, and long intervals between index and reference procedures. QUADAS-2 organises risk-of-bias assessment around participant selection, index test, reference standard, and flow and timing.[2]

APPRAISAL CHECKLIST

Appraising a diagnostic accuracy design

• Does the participant spectrum match the intended use of the test?
• Were the index-test threshold and interpretation rules prespecified?
• Is the reference standard appropriate and interpreted independently where feasible?
• Did all or nearly all participants receive verification using the same strategy?
• Is the interval between tests short enough for the target condition to remain stable?
• Are missing, indeterminate and excluded results transparent?

Ethical and governance considerations

Interface with statistical analysis

For a binary threshold, participants can be cross-classified as true positive, false positive, false negative and true negative relative to the reference classification. Sensitivity and specificity describe different conditional proportions; predictive values answer different, prevalence-dependent questions. Confidence intervals should accompany accuracy estimates.[1]

The linked Biostatistics lectures cover the two-by-two table, sensitivity, specificity, predictive values and diagnostic sample size. Statistical analysis should also reflect clustering by centre or reader, paired comparisons between tests, repeated thresholds or multiple observations per participant where these occur.

Accuracy is only one stage of test evaluationMMed extension

A technically accurate test may still fail to improve care if it is slow, inaccessible, poorly accepted, costly, difficult to interpret or does not change management. Conversely, a modestly discriminating test may be useful when it safely reduces unnecessary procedures. A staged evaluation therefore moves from analytical validity and diagnostic accuracy to clinical utility, impact, implementation and economic consequences rather than treating a high area under the curve as the endpoint of translation.[5]

Planning accuracy around the intended decision

A protocol becomes stronger when it states what the test will do after a result is available. For a triage test, false negatives may be especially important because a negative result could stop further assessment. For an add-on test, false positives may create unnecessary invasive procedures. These consequences should shape threshold choice, required precision and which accuracy measures receive primary emphasis.

The intended decision also determines the relevant comparator. A new rapid test may be evaluated against a reference standard for accuracy, but implementation may ultimately compare new pathway versus existing pathway. Do not confuse these two comparisons. The first asks whether classifications agree sufficiently with target-condition status; the second asks whether changing the pathway improves care.

Multi-centre studies require additional planning. A common protocol should standardise eligibility, specimen handling, reader training, reference verification and data definitions. Centre-specific differences should still be recorded because they may reveal clinically important heterogeneity. Pooling all sites without examining how recruitment and procedures differ can conceal a test that works well only in particular settings.

Prospective registration and protocol discipline

Diagnostic accuracy studies benefit from prospective protocol specification even when formal registration is not mandatory. Prespecify the primary threshold, primary accuracy measures, sample-size rationale, handling of indeterminate results and subgroup analyses. Changes should be dated and justified before results are interpreted. This reduces selective reporting of the threshold or subgroup with the most attractive findings.

Finally, the analysis population should be named explicitly. A “complete-case” diagnostic table can differ from the enrolled cohort if verification is missing. Report the number enrolled, the number with interpretable index results, the number verified, and the number included in each primary analysis. This denominator trail is essential for understanding how the final accuracy estimate was produced.

Precision is part of design

Sample size should deliver useful precision for the primary accuracy measures in the target-condition positive and negative groups, allowing for expected prevalence, missing verification and indeterminate results.

Worked example

WORKED EXAMPLE

Planning the postpartum triage study

The investigators define persistent dysglycaemia using a prespecified laboratory reference procedure. They enrol consecutive eligible women attending participating postpartum clinics and perform the point-of-care test before the reference result is known. The index threshold is selected from prior evidence rather than optimised on the same dataset.

Each participant receives both procedures at the same visit. Operators are trained, device lot numbers are recorded, and laboratory assessors do not receive the point-of-care result. The screening log records non-enrolment. The analysis will report sensitivity and specificity with 95% confidence intervals, the two-by-two counts, predictive values for the study population and a transparent account of indeterminate or missing results.

The protocol explicitly states that this study evaluates accuracy, not whether implementing the point-of-care pathway improves long-term diabetes outcomes. A later impact study would be needed for that question.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

Why is a consecutive single-gate sample usually preferable to obvious cases plus healthy controls?

Answer and reasoning

It better represents the spectrum and competing diagnoses encountered in the intended pathway. Obvious cases and healthy controls often differ on many features besides target-condition status, which can exaggerate apparent separation between groups.

What should be specified before recruitment when a continuous index test will be used at one clinical threshold?

Answer and reasoning

The intended threshold and its interpretation rule should be prespecified where evidence permits. Data-driven optimisation on the same sample can produce overly optimistic performance and must be described as exploratory if performed.

Why should predictive values not be treated as fixed characteristics of the test?

Answer and reasoning

They depend on the frequency of the target condition in the population being tested. The same sensitivity and specificity can therefore correspond to different positive and negative predictive values in different settings.

A study shows excellent sensitivity and specificity. Has it demonstrated clinical utility?

Answer and reasoning

No. Accuracy describes classification performance relative to a reference standard. Clinical utility requires evidence that using the test improves decisions, outcomes or resource use relative to a relevant alternative strategy.

Key take-home messages

SUMMARY

Key take-home messages

• Define intended use, target condition, index test and reference standard before selecting the analysis.
• Recruit participants who resemble the population in whom the test will be used.
• Apply verification, timing and interpretation procedures consistently and transparently.
• Report participant flow, indeterminate results and uncertainty, not only headline accuracy estimates.
• Treat diagnostic accuracy as one stage of test evaluation rather than proof of patient benefit.

References

1. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. https://doi.org/10.1136/bmj.h5527
2. Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
3. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282(11):1061–1066. https://doi.org/10.1001/jama.282.11.1061
4. Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174(4):469–476. https://doi.org/10.1503/cmaj.050090
5. Bossuyt PMM, Reitsma JB, Linnet K, Moons KGM. Beyond diagnostic accuracy: the clinical utility of diagnostic tests. Clin Chem. 2012;58(12):1636–1643. https://doi.org/10.1373/clinchem.2012.182576