Choosing a Clinically Meaningful Effect Size

Overview

A power-based sample-size calculation requires an effect under the alternative hypothesis.

This value is often called the:

• target difference;
• effect to detect;
• clinically important difference;
• minimal important difference;
• non-inferiority margin; or
• anticipated effect.

These terms are not synonyms.

The target effect is the value used for planning. It should represent an effect that is important enough to influence interpretation or decision-making and plausible enough to justify designing the study around it.

The target should be selected before examining the study’s comparative outcomes. It should not be chosen merely because it yields an affordable sample. Structured target-difference guidance treats this choice as a documented scientific judgement supported by several sources of evidence [1].

Why this matters

For a two-group mean comparison, required sample size varies approximately with the inverse square of the target difference.

If all other assumptions remain fixed:

• planning for 3 units requires far more participants than planning for 5;
• planning for 8 requires far fewer participants than planning for 5.

An exaggerated target effect produces a small but potentially uninformative study. An extremely small target may produce an infeasible study whose additional precision has limited decision value.

The investigator must balance:

• patient importance;
• clinical decision consequences;
• existing evidence;
• outcome reliability;
• harms and burden;
• costs;
• feasibility; and
• uncertainty.

Distinguish three effects

Clinically meaningful effect

A clinically meaningful effect is large enough to matter for patients, clinicians, services or policy under a defined context.

Its importance depends on:

• outcome;
• baseline risk or severity;
• duration;
• treatment burden;
• harms;
• cost;
• alternatives;
• patient preferences; and
• population.

Expected effect

The expected effect is the effect considered plausible based on:

• prior trials;
• meta-analysis;
• observational evidence;
• biological rationale;
• pilot data; and
• intervention fidelity.

It may be larger or smaller than the minimum meaningful effect.

Observed effect

The observed effect is estimated after the study.

It is uncertain and should be reported with a confidence interval. It should not be used retrospectively to redefine what the trial was designed to detect.

Use clinically interpretable scales

Preferred target effects include:

• mean difference in mmHg;
• risk difference in percentage points;
• risk ratio accompanied by baseline risk;
• additional symptom-free days;
• difference in median or restricted mean survival time;
• sensitivity or specificity difference;
• ratio of geometric means; and
• change on a validated patient-reported outcome.

A standardised effect such as Cohen’s $d$ can help compare scales but is less directly interpretable clinically.

Whenever possible, convert a standardised effect back into clinical units.

Absolute and relative effects

Suppose an intervention is expected to reduce a 20% event risk to 15%.

This can be expressed as:

• absolute risk difference: −5 percentage points;
• risk ratio: 0.75;
• relative risk reduction: 25%;
• number needed to treat under stated assumptions: 20.

If baseline risk is 2%, the same risk ratio of 0.75 produces an absolute reduction of only 0.5 percentage points.

Therefore, a relative target cannot determine sample size for two proportions without a baseline risk.

Minimal important difference

The minimal important difference, abbreviated MID, is commonly defined as the smallest difference perceived as important by patients or that would support a change in management under a specified context.

Jaeschke and colleagues developed an anchor-based approach linking score change to a global assessment [2].

The MID is not a universal constant. It can vary by:

• population;
• baseline severity;
• within-person versus between-group interpretation;
• follow-up period;
• anchor;
• direction of change;
• treatment context; and
• analytical method.

For patient-reported outcomes, Revicki and colleagues recommend integrating anchor-based and distribution-based evidence rather than relying on one method [3].

Individual change versus group difference

An individual-level meaningful change threshold is not automatically the same as a meaningful difference between group means.

A group mean difference reflects the average contrast across participants. Many response patterns can produce the same mean difference.

The planning target should match the estimand.

Methods for choosing a target effect

Anchor-based methods

An anchor is an external measure with interpretable meaning, such as:

• patient’s global rating of change;
• clinician judgement;
• return to work;
• treatment escalation;
• functional milestone; or
• established disease category.

A credible anchor should relate sufficiently to the outcome and represent the intended concept.

Distribution-based methods

Distribution-based quantities include:

• fractions of a standard deviation;
• standard error of measurement;
• effect-size indices; and
• reliable-change thresholds.

They describe statistical magnitude or measurement properties. They do not establish importance by themselves.

Evidence synthesis

A systematic review or meta-analysis can inform plausible effects and variability.

Potential problems include:

• publication bias;
• small-study effects;
• different populations;
• different outcome definitions;
• poor intervention fidelity;
• short follow-up; and
• optimistic early trials.

The meta-analytic mean should not be copied uncritically as the target. Earlier guidance similarly recommends an explicit process for combining clinical importance, plausibility and uncertainty [4].

Expert and stakeholder elicitation

Structured input can be obtained from:

• patients;
• carers;
• clinicians;
• policymakers;
• methodologists; and
• service managers.

The process should define:

• decision context;
• outcome scale;
• plausible values;
• uncertainty;
• whose perspectives were represented; and
• how disagreement was handled.

Health-economic and decision-based approaches

A target may be informed by whether expected benefits justify:

• harms;
• treatment burden;
• opportunity cost;
• programme cost; and
• downstream resource use.

This is particularly relevant when a small clinical effect could be worthwhile because an intervention is cheap and safe, or when a larger benefit is required to justify toxicity or high cost.

Worked example

Target difference in non-inferiority trials

A non-inferiority margin is the largest loss of efficacy considered acceptable in exchange for other benefits.

It must preserve a clinically important fraction of the active comparator’s established effect.

The margin should consider:

• historical evidence;
• constancy of trial conditions;
• assay sensitivity;
• outcome definition;
• clinical judgement;
• potential advantages of the new treatment; and
• regulatory guidance.

It should not be selected merely to make the trial smaller. Active-control margins should be justified using the clinical and historical framework of the intended non-inferiority claim [6].

The expected true difference and non-inferiority margin are separate inputs.

Binary outcomes

For binary outcomes, specify both group risks.

A target such as “20% relative reduction” is incomplete without the control risk.

Same relative effect under different baseline risks

simulated data

Table 2. Risk ratio 0.75 under different control risks

Control risk	Intervention risk	Absolute risk difference
40%	30%	−10 percentage points
20%	15%	−5 percentage points
4%	3%	−1 percentage point

Note: Clinical importance and sample size can differ despite the same risk ratio.

Survival outcomes

A hazard ratio should be translated into clinically relevant survival quantities where possible.

The same hazard ratio can correspond to different:

• median survival gains;
• milestone survival differences;
• restricted mean survival-time differences; and
• numbers needed to treat.

If proportional hazards are doubtful, a constant hazard-ratio target may not represent the treatment effect adequately.

Measurement reliability

An observed effect is attenuated or made noisier by measurement error.

The target should be considered alongside:

• reliability;
• within-person variability;
• responsiveness;
• floor and ceiling effects;
• minimally detectable change; and
• measurement burden.

A target smaller than the instrument’s meaningful resolution may be difficult to interpret even if statistically detectable.

Sensitivity and uncertainty

The DELTA2 framework MMed extension

The DELTA2 guidance recommends a structured process for specifying the target difference in a randomised trial [1].

The process includes:

• clarifying the estimand;
• identifying candidate methods;
• reviewing evidence;
• considering uncertainty;
• involving relevant stakeholders;
• documenting the final value; and
• performing sensitivity analyses.

The target difference is a scientific judgement supported by evidence, not a value produced automatically by software.

Suggested reporting

Common mistakes

Knowledge check

What is the difference between a target and an expected effect?

Answer and reasoning

The target is the effect the study is designed around because it is important and plausible. The expected effect is a prediction of what may occur. They may be informed by similar evidence but serve different purposes.

Why is a risk ratio of 0.75 insufficient for sample-size planning?

Answer and reasoning

The variance and absolute clinical benefit depend on both group risks. A control risk is needed to derive the intervention risk and absolute difference.

Can one minimal important difference be applied to every setting?

Answer and reasoning

No. Importance may vary by population, baseline severity, anchor, follow-up, intervention burden and whether the target concerns individual change or a group difference.

Why is an 8 mmHg target concerning when the plausible range is 3–7 mmHg?

Answer and reasoning

It may make the study smaller by assuming an effect larger than credible evidence supports. The study could then be underpowered for a worthwhile but more realistic effect.

What should be concluded if the observed estimate is 4 mmHg when the trial was planned for 5 mmHg?

Answer and reasoning

Interpret the 4 mmHg estimate with its confidence interval and clinical context. The planning target is not a threshold that determines whether the observed effect matters.

Key take-home messages

References

1. Cook JA, Julious SA, Sones W, Hampson LV, Hewitt C, Berlin JA, et al. DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. BMJ. 2018;363:k3750. https://doi.org/10.1136/bmj.k3750
2. Jaeschke R, Singer J, Guyatt GH. Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10(4):407–415. https://doi.org/10.1016/0197-2456(89)90005-6
3. Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008;61(2):102–109. https://doi.org/10.1016/j.jclinepi.2007.03.012
4. Cook JA, Hislop J, Altman DG, Briggs AH, Fayers P, Norrie JD, et al. Specifying the target difference in the primary outcome for a randomised controlled trial: guidance for researchers. Trials. 2015;16:12. https://doi.org/10.1186/s13063-014-0526-8
5. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet. 2005;365(9467):1348–1353. https://doi.org/10.1016/S0140-6736(05)61034-3
6. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. Choice of control group and related issues in clinical trials E10. Geneva: ICH; 2000. https://database.ich.org/sites/default/files/E10_Guideline.pdf