Index Tests, Reference Standards and the Two-by-Two Framework

Overview

The familiar diagnostic two-by-two table is the end product of a measurement and classification process, not the starting point of study design. Before a participant appears in one of its four cells, investigators must define the target condition, perform and interpret the index test, apply a reference standard, resolve thresholds and indeterminate results, and decide which participants enter the analysis.[1]

These upstream decisions determine the meaning of every sensitivity, specificity and predictive-value estimate. A neat table can conceal serious problems if the reference standard changes according to the index result, if assessors know each other’s classifications, or if indeterminate results are quietly removed.[1,2]

This lecture therefore focuses on the participant-level logic behind the table. Detailed formulas and calculation are covered in the linked Biostatistics lectures.

Why this matters

Diagnostic terminology is often used loosely. Researchers may call a routine comparator a “gold standard”, dichotomise a continuous measurement without justifying the cut-off, or label discordant pairs as errors without considering uncertainty in the reference classification. These choices can produce confident but misleading estimates.

Understanding the framework also prevents a common analytical mistake: interpreting the four cells without first asking how each label was generated. The reference standard is a method, not truth itself. When it is imperfect, the observed classifications are relative to that comparator.[2,3]

Key concepts

Index test and index-test result

The index test is the procedure being evaluated. Its raw result may be continuous, ordinal, categorical or image based. If clinical use requires a binary decision, a threshold or decision rule converts the raw result into index-positive and index-negative states.

Preserve the original measurement whenever possible. This permits audit, alternative threshold analyses, calibration checks for model-based tests and assessment of technical failures.

Reference standard and target-condition classification

The reference standard is the best available method selected to determine whether the target condition is present. It may comprise one test, expert adjudication, longitudinal follow-up, a composite rule or multiple procedures. The study should document exactly how reference-positive, reference-negative and indeterminate classifications are assigned.[2]

Calling a comparator a gold standard implies near-perfect classification. That description should be avoided unless it is genuinely defensible. Imperfect reference standards can misclassify participants and distort apparent index-test performance.[3]

The four analysable states

When both classifications are binary and available, each participant contributes to one of four cells:

• true positive: index positive, reference positive;
• false positive: index positive, reference negative;
• false negative: index negative, reference positive; or
• true negative: index negative, reference negative.

The words “true” and “false” are conventional shorthand relative to the reference classification. They do not prove biological truth.

When to use this framework

DECISION GUIDE

When a two-by-two framework is appropriate

• The target condition can be classified into two clinically meaningful states for the primary analysis.
• The index test has a prespecified binary result or a defensible threshold can be applied.
• Each participant receives a reference classification that corresponds to the same target condition and time state.
• Participants contribute one independent primary classification, or correlation is handled explicitly.
• Indeterminate and missing results are documented separately rather than silently forced into a binary cell.

Not every diagnostic problem should be reduced to two categories. Multiclass diagnosis, disease staging, continuous biomarker estimation and probabilistic AI outputs may need broader performance measures. A two-by-two analysis can still be useful for a specific clinical decision threshold, but it should not erase the original information.

Research question and methodological fit

The relevant question is not merely “What are the values in the table?” It is:

At the prespecified point-of-care threshold, how does the index classification compare with the laboratory reference classification among women in the intended postpartum follow-up population?

The primary unit is the participant. If both breasts, eyes, lesions or images are analysed, investigators must decide whether the target condition and decision are participant-level or unit-level. Treating multiple correlated units as independent participants can narrow confidence intervals incorrectly.

Test procedures and interpretation

PROTOCOL CHECKLIST

Index-test items to prespecify

• Specimen, device, software or reader procedure.
• Quality-control and technical-failure rules.
• Threshold or decision rule and whether it was prespecified.
• Clinical information available to the index assessor.
• Handling of repeated measurements or multiple readers.
• Definition and coding of indeterminate results.

PROTOCOL CHECKLIST

Reference-standard items to prespecify

• Rationale for choosing the reference standard.
• Exact components and classification rule.
• Clinical information available to reference assessors.
• Independence from the index test where feasible.
• Adjudication procedure for disagreement or borderline results.
• Management of reference-standard failure or non-completion.

One principle is blinding of interpretation, sometimes called masking. If reference assessors know the index result, their judgement may shift toward agreement; if index readers know the final diagnosis, the same problem occurs. Complete blinding may not always reflect clinical practice, so the protocol should state what information each reader receives and why.

Data structure and the two-by-two table

Illustrative classification table

illustrative

Table 1. Participant-level cross-classification

	Reference positive	Reference negative
Index positive	True positive	False positive
Index negative	False negative	True negative

Note: The labels are defined relative to the selected reference standard. Indeterminate and missing results should be reported outside this table and accounted for in participant flow.

From these cells, sensitivity conditions on reference-positive participants, while specificity conditions on reference-negative participants. Positive and negative predictive values condition on the index result and therefore vary with the prevalence of the target condition in the tested population. Likelihood ratios express how much more or less likely a test result is in participants with versus without the target condition. The linked Biostatistics lectures derive and interpret these measures.

Why totals matter

Always report the cell counts, not only percentages. They show denominators, reveal sparse cells and allow readers to reconstruct measures. If different analyses have different denominators because of missing data or alternative thresholds, make that explicit.

Thresholds and indeterminate results

A threshold converts a continuum into a decision. Moving the threshold usually changes sensitivity and specificity in opposite directions. The clinically appropriate threshold depends on intended use and the relative consequences of false negatives and false positives, not solely on the point that maximises a statistical index.

Indeterminate results are part of test performance. Excluding them can make a test appear more usable than it is. Investigators should report their frequency, reasons and clinical handling. Sensitivity analyses may examine plausible alternative coding rules when appropriate.[1]

Three categories may be better than a forced binary decisionMMed extension

Some pathways deliberately use positive, negative and indeterminate zones. An intermediate zone can trigger repeat testing or a more definitive procedure. Evaluating such a strategy requires reporting the size and composition of the indeterminate group and the downstream pathway, rather than forcing every result into a positive or negative cell for convenience.

Bias, validity and quality assurance

Reference-standard bias also arises when the comparator does not classify the target condition accurately enough for the intended question. Differential verification occurs when different reference standards are used according to index results or other participant characteristics. Partial verification occurs when only a subset receives reference assessment. These are explored in RM-P10-M04.

APPRAISAL CHECKLIST

Questions behind every diagnostic table

• Were index and reference classifications defined before analysis?
• Was the index threshold prespecified or clearly labelled exploratory?
• Was the same target condition assessed by both procedures?
• Were interpretations independent where appropriate?
• Were indeterminate and missing results retained in the study flow?
• Did all participants have an appropriate opportunity for verification?

Ethical and governance considerations

Interface with statistical analysis

The two-by-two table supports sensitivity, specificity, predictive values, likelihood ratios and diagnostic odds ratios, each with uncertainty. Comparisons of two tests applied to the same participants require paired methods. Multiple readers, lesions or sites introduce clustering. Continuous or ordinal tests may be assessed across thresholds using receiver-operating-characteristic methods, but threshold-independent summaries should be interpreted together with clinically relevant operating points.

For sample-size planning, the required number may be driven by precision for sensitivity, precision for specificity, comparison of tests, expected prevalence and the design’s recruitment structure. The linked sample-size lecture treats these requirements explicitly.

Special classification problems

Repeated reference measurements

Some target conditions cannot be established from one measurement. A reference definition may require repeated laboratory values, multiple imaging views or confirmation on a second day. The protocol should identify which component determines final status, how discordant components are resolved, and the time window within which they are considered to represent the same clinical state. Post-hoc adjudication rules can change who is called reference positive.

Composite reference standards

A composite reference standard combines several sources, for example imaging, pathology, microbiology and follow-up. This may be necessary when no single test is sufficiently accurate or feasible. The combination rule should be defined before index results are examined. “Expert panel diagnosis” is not enough description: report panel membership, information available, decision rule, handling of disagreement and whether the index test is withheld from the panel.

Multiple index thresholds

If a test has several intended categories—such as low, intermediate and high risk—the primary evaluation should reflect that pathway. Forcing every result into one binary table may hide the purpose of the intermediate zone. Researchers can present several two-by-two analyses at prespecified thresholds, but each threshold has a different clinical interpretation and can produce different denominators if indeterminate states are handled inconsistently.

Unit-of-analysis problems

Diagnostic datasets often contain more than one observation per participant: two eyes, several lesions, repeated images, multiple readers or multiple specimens. The clinical decision might nevertheless be participant-level. The protocol should state the unit at which target condition and index result are defined. If lesion-level accuracy is scientifically important, analysis must account for within-participant correlation and should not imply that lesion-level sensitivity is identical to patient-level detection.

Reporting denominators and uncertainty

A transparent diagnostic report enables the reader to trace every proportion back to counts. Sensitivity must show the number of reference-positive participants contributing to its denominator; specificity must show the reference-negative denominator. The STARD explanation-and-elaboration guidance provides detailed examples of how these design and reporting decisions should be made transparent.[4] Confidence intervals communicate sampling uncertainty but do not capture uncertainty caused by an imperfect reference standard or biased verification. When selection for definitive verification depends on earlier test information, the resulting accuracy estimates can be biased unless the verification process is appropriately designed and analysed.[5]

When prevalence has been manipulated by sampling, positive and negative predictive values calculated from the study should not be exported to routine care. If predictive values are needed for another prevalence, the assumptions used to derive them should be explicit. Even then, spectrum differences may prevent simple mathematical transport.

Interpreting discordance responsibly

Discordant results deserve clinical and technical investigation rather than automatic relabelling. Review whether specimens were collected at comparable times, whether either method has known failure modes, and whether disease could have changed between procedures. If an adjudication panel reviews discordant cases, its information set and rules must be specified in advance to avoid selectively favouring the index test.

In some studies the reference standard is itself continuous and is dichotomised for the target condition. That threshold should be clinically justified and consistently applied. Sensitivity analyses around genuinely uncertain reference thresholds may be informative, but they should not be used to search for the definition that maximises index-test performance.

Finally, report technical failures as a performance dimension. A test with high sensitivity among interpretable results but a substantial failure rate may be less useful than a slightly less accurate test that produces reliable results for nearly everyone. Accuracy and testability should therefore be presented together when failure is non-trivial.

Preserve provenance of every classification

For auditability, the dataset should retain who interpreted each procedure, when interpretation occurred, which software or assay version was used and whether repeat testing changed the final classification. This provenance is especially important when an adjudicated result differs from the first reading. A reproducible two-by-two table should be traceable back to source measurements rather than reconstructed from undocumented manual recoding.

Worked example

WORKED EXAMPLE

From raw postpartum results to analysable classifications

For each participant, the team stores the raw point-of-care measurement, the prespecified index classification, all laboratory reference components, the final reference classification and flags for technical failure or indeterminate status. The index operator records the result before the laboratory classification is available.

A participant with an index-positive and reference-positive result enters the true-positive cell. A participant with an index-positive but reference-negative result enters the false-positive cell relative to the laboratory reference. A participant whose laboratory procedure is incomplete does not automatically enter the reference-negative column; the missing verification is reported separately.

This dataset permits transparent reconstruction of the two-by-two table and supports later threshold analyses without losing the original measurements.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

A participant is index positive and reference negative. What is the conventional cell label?

Answer and reasoning

False positive, relative to the selected reference standard. The label does not prove that the index test is biologically wrong if the reference standard itself can misclassify disease.

Why should the original continuous index result be stored when a binary threshold is prespecified?

Answer and reasoning

It preserves information for audit, quality control and transparent secondary threshold analyses, and prevents irreversible loss of measurement detail.

What is incorporation bias?

Answer and reasoning

It occurs when the index test contributes to the reference classification used to judge that same index test, creating circularity that can inflate apparent agreement or accuracy.

Should a technically unreadable index result simply be removed from the dataset?

Answer and reasoning

No. It should be retained as an explicit test outcome or missingness state, reported in participant flow and handled according to a prespecified analysis strategy because test failure is part of real-world usability.

Key take-home messages

SUMMARY

Key take-home messages

• Build the two-by-two table from explicit participant-level test and reference classifications.
• Treat the reference standard as the best available comparator, not automatic truth.
• Prespecify thresholds and retain raw measurements where possible.
• Report indeterminate, failed and unverified results rather than hiding them.
• Match statistical measures to the clinical decision and data structure.

References

1. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. https://doi.org/10.1136/bmj.h5527
2. Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
3. Rutjes AWS, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PMM. Evaluation of diagnostic tests when there is no gold standard: a review of methods. Health Technol Assess. 2007;11(50):iii, ix–51. https://doi.org/10.3310/hta11500
4. Cohen JF, Korevaar DA, Altman DG, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 2016;6(11):e012799. https://doi.org/10.1136/bmjopen-2016-012799
5. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39(1):207–215. https://doi.org/10.2307/2530820