Sample Size for Diagnostic Studies

Overview

A diagnostic-accuracy study evaluates an index test against a reference standard in participants who represent the intended clinical use.

Sample size depends on the primary accuracy objective:

• estimate sensitivity;
• estimate specificity;
• estimate predictive values;
• estimate an area under the receiver operating characteristic curve;
• test accuracy against a minimum acceptable value; or
• compare two diagnostic tests.

A total sample alone is insufficient. Sensitivity requires participants with the target condition, while specificity requires participants without it.

The design must also address prevalence, spectrum of disease, indeterminate tests, verification, reference-standard error and paired observations.

Why this matters

A study may recruit 500 participants yet contain only 20 with the target condition. Sensitivity would then be based on 20 observations and may be imprecise despite the large total sample.

Conversely, a study that recruits equal numbers with and without disease can estimate sensitivity and specificity efficiently but creates an artificial condition prevalence. Predictive values calculated directly from that sample do not represent routine clinical practice unless corrected using an appropriate target prevalence.

Diagnostic sample-size planning therefore begins with the intended metric and recruitment design, not one universal formula.

Define the diagnostic question

The protocol should specify:

• target condition;
• intended-use population;
• clinical setting;
• index test and threshold;
• reference standard;
• timing;
• whether readers are blinded;
• handling of indeterminate results;
• primary accuracy measure; and
• recruitment design.

Changing the threshold after examining the same data can produce optimistic accuracy and invalidates a sample-size calculation based on a prespecified threshold.

Estimating sensitivity

Sensitivity is:

$$\text{Sensitivity} = \frac{\text{true positives}} {\text{all participants with the target condition}}$$

For a large-sample precision approximation:

Using $Se=0.90$, $d_{Se}=0.05$ and 95% confidence:

$$n_D = \frac{ 1.960^2(0.90)(0.10) }{ 0.05^2 } = 138.29$$

Round upward to 139 participants with the target condition.

Estimating specificity

Specificity is:

$$\text{Specificity} = \frac{\text{true negatives}} {\text{all participants without the target condition}}$$

Using $Sp=0.85$ and $d_{Sp}=0.05$:

$$n_{\bar D} = 195.91$$

(The arithmetic producing 195.91 participants without the target condition from the displayed formula and inputs is not shown here.)

Round upward to 196 participants without the target condition.

These normal-approximation calculations should be verified using an interval method with suitable coverage, especially when accuracy is near 0 or 1 [1], [2].

Translating to a total consecutive sample

If prevalence is $\pi$, the expected total needed to obtain $n_D$ condition-positive participants is:

$$N_D=\frac{n_D}{\pi}$$

The total needed for $n_{\bar D}$ condition-negative participants is:

$$N_{\bar D}=\frac{n_{\bar D}}{1-\pi}$$

Use the larger requirement.

For prevalence 30%:

$$N_D=\frac{139}{0.30}=463.33$$ $$N_{\bar D}=\frac{196}{0.70}=280.00$$

The sensitivity requirement governs. Round upward to 464 evaluable participants.

At 30% prevalence, 464 participants are expected to provide approximately:

• 139 with the condition;
• 325 without the condition.

If 10% may have an uninterpretable reference standard or otherwise lack evaluable classification:

$$464/0.90=515.56$$

Recruit 516 participants.

Case–control accuracy sampling

A diagnostic case–control design may recruit known cases and controls separately.

Advantages include:

• efficient accumulation of condition-positive participants;
• control over the case-to-control ratio; and
• feasibility for rare conditions.

Limitations include:

• artificial prevalence;
• possible spectrum bias;
• difficulty estimating predictive values directly;
• potential overrepresentation of clear-cut disease and healthy controls; and
• limited representation of the intended clinical pathway.

Sensitivity and specificity may be estimable if cases and controls represent the relevant spectra, but predictive values require a clinically relevant prevalence and appropriate modelling.

Predictive-value precision

Positive predictive value uses participants with a positive index test as its denominator. Negative predictive value uses those with a negative index test.

Planning therefore depends on:

• prevalence;
• sensitivity;
• specificity;
• expected proportion of positive and negative index-test results; and
• desired precision.

A sample designed only for sensitivity and specificity may not estimate predictive values precisely.

Simulation is often useful when several metrics must meet precision requirements simultaneously. Reviews of diagnostic-study planning emphasise matching the calculation to the target accuracy measure, prevalence and sampling design [3].

Testing sensitivity or specificity against a minimum

Some studies test whether sensitivity exceeds a minimum acceptable value.

Planning requires:

• null sensitivity;
• alternative sensitivity;
• one-sided or two-sided type I error;
• power;
• condition-positive sample;
• and total recruitment based on prevalence.

The minimum acceptable value should be clinically justified. It should not be chosen after seeing pilot accuracy.

A high-power test can still produce an interval too wide for clinical use; precision should also be reported.

ROC-area studies

The area under the receiver operating characteristic curve, abbreviated AUC, summarises discrimination across thresholds.

Sample size for AUC may target:

• estimation of one AUC with desired precision;
• testing AUC against 0.50 or another value;
• comparing two independent AUCs; or
• comparing paired AUCs from tests applied to the same participants.

Inputs include:

• anticipated AUC;
• condition-positive and condition-negative sample sizes;
• sampling ratio;
• distributional assumptions or variance method;
• correlation between paired tests; and
• partial versus full AUC.

Obuchowski developed methods for ROC-study sample size and reviewed diagnostic-accuracy designs [4], [5].

Comparing diagnostic tests

When two index tests are applied to the same participants, results are paired.

Power depends on:

• accuracy difference;
• prevalence;
• within-participant correlation;
• discordant classifications;
• threshold strategy; and
• analysis method.

Calculating as though test results came from independent samples wastes pairing information and may misstate power.

For multiple readers or repeated images, additional correlation levels require specialised design and analysis.

Imperfect reference standards and verification

If the reference standard is imperfect, sensitivity and specificity are measured relative to that standard, not necessarily the true condition.

Sample-size planning may require:

• latent-class methods;
• composite reference standards;
• adjudication;
• repeated testing;
• validation subsamples; or
• sensitivity analyses.

Partial verification occurs when only some participants receive the reference standard. Verification related to the index-test result can bias accuracy estimates.

The sample must include sufficient verified participants across index-test results and clinical spectrum.

Diagnostic planning quantities

simulated data

Table 1. Required evaluable classifications

Objective	Planning value	Margin of error	Required stratum
Sensitivity	90%	5 percentage points	139 condition-positive
Specificity	85%	5 percentage points	196 condition-negative
Both under 30% prevalence	—	—	464 total evaluable
After 10% non-evaluable	—	—	516 recruited

Note: Large-sample precision approximation; all values simulated.

Assumptions and sensitivity analysis

Why disease prevalence affects recruitment but not the definition of sensitivity MMed extension

Sensitivity is conditional on having the target condition and specificity is conditional on not having it.

Their definitions do not contain prevalence. However, prevalence determines how many condition-positive and condition-negative participants are expected under consecutive recruitment.

Therefore, prevalence affects total recruitment required to achieve the stratum-specific sample sizes.

Diagnostic-accuracy protocols and reports should state how the sample size was determined, how participants were recruited and how indeterminate or missing results were handled [6].

Suggested reporting

Common mistakes

Knowledge check

Why are separate sample sizes needed for sensitivity and specificity?

Answer and reasoning

Sensitivity is estimated only among participants with the target condition, while specificity is estimated only among those without it. Their precision depends on different denominators.

Why does prevalence affect total recruitment?

Answer and reasoning

Under consecutive sampling, prevalence determines the expected proportion with the condition. Lower prevalence means more total participants are needed to obtain the required condition-positive sample.

Can predictive values be calculated directly from a 1:1 case–control accuracy sample?

Answer and reasoning

The arithmetic can be performed, but the results reflect the artificial 50% prevalence and generally do not represent routine clinical predictive values. Target-population prevalence must be incorporated.

Why must paired test comparison use a paired calculation?

Answer and reasoning

Both tests are measured in the same participants, so their errors are correlated. The variance of the difference depends on that within-participant relationship.

What happens if the observed prevalence is lower than planned?

Answer and reasoning

The study may recruit too few participants with the condition, producing a wider sensitivity confidence interval or lower power than intended.

Key take-home messages

References

1. Buderer NMF. Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity. Acad Emerg Med. 1996;3(9):895–900. https://doi.org/10.1111/j.1553-2712.1996.tb03538.x
2. Flahault A, Cadilhac M, Thomas G. Sample size calculation should be performed for design accuracy in diagnostic test studies. J Clin Epidemiol. 2005;58(8):859–862. https://doi.org/10.1016/j.jclinepi.2004.12.009
3. Hajian-Tilaki K. Sample size estimation in diagnostic test studies of biomedical informatics. J Biomed Inform. 2014;48:193–204. https://doi.org/10.1016/j.jbi.2014.02.013
4. Obuchowski NA. Computing sample size for receiver operating characteristic studies. Invest Radiol. 1994;29(2):238–243. https://doi.org/10.1097/00004424-199402000-00020
5. Obuchowski NA. Sample size calculations in studies of test accuracy. Stat Methods Med Res. 1998;7(4):371–392. https://doi.org/10.1177/096228029800700405
6. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. https://doi.org/10.1136/bmj.h5527