Statistical Significance versus Clinical Importance

Overview

Statistical significance concerns compatibility with a specified null hypothesis under a statistical model and decision threshold.

Clinical importance concerns whether the magnitude of an effect is meaningful for patients, clinicians, health services or public-health decisions.

The concepts are related but not equivalent.

A small effect can be statistically significant in a large study. A clinically important effect can be statistically non-significant in a small or imprecise study. Interpretation should therefore prioritise effect estimates and confidence intervals rather than an isolated threshold [1], [2].

Why this matters

A report may state:

The intervention significantly reduced systolic blood pressure, $p<0.001$.

This wording does not reveal whether the reduction was:

• 0.5 mmHg;
• 5 mmHg; or
• 20 mmHg.

It also does not state the confidence interval, adverse effects, burden, cost, adherence or applicability.

Statistical significance answers a model-based question. Clinical importance requires a decision context.

Two different questions

Statistical significance asks:

Are the data sufficiently incompatible with a specified null value under the analysis and significance threshold?

Clinical importance asks:

Is the effect large enough, durable enough and relevant enough to influence care or policy after considering benefits and disadvantages?

Clinical importance may depend on:

• baseline risk or severity;
• outcome type and time horizon;
• harms and adverse events;
• treatment burden;
• costs and resources;
• available alternatives;
• patient preferences;
• equity;
• implementation feasibility; and
• certainty and applicability of the evidence.

Four possible combinations

Statistical and clinical interpretations

simulated data

Table 1. Four combinations of statistical significance and clinical importance

Result	Statistical interpretation	Possible clinical interpretation
Large, precise benefit	Null excluded	Likely important if harms and burden are acceptable
Tiny, precise benefit	Null excluded	May be clinically trivial despite small p
Large, imprecise benefit	Null included	Potentially important but uncertain
Small, precise near-null effect	Null may be included or excluded	Can rule out important benefit if interval is narrow enough

Note: Final interpretation requires the actual effect scale and context.

Statistically significant but clinically trivial

Suppose a very large study estimates a reduction of 0.6 mmHg with 95% CI 0.3 to 0.9 mmHg lower and $p<0.001$.

The result is statistically incompatible with zero but may be too small to justify a costly or burdensome programme.

Clinically important but statistically non-significant

Suppose a small study estimates a 6 mmHg reduction with 95% CI 2 mmHg higher to 14 mmHg lower and $p=0.14$.

The data remain compatible with an important benefit, no effect and some harm. The correct conclusion is uncertainty, not that the clinically important effect is absent.

Use the confidence interval

A confidence interval can be compared with:

• the null value; and
• one or more clinically meaningful values.

For a mean difference where negative values favour treatment, let:

• 0 mmHg = no effect;
• −5 mmHg = planning target for a worthwhile reduction.

The simulated trial CI is −8.5 to −3.3 mmHg.

This interval:

• excludes no effect;
• includes reductions larger than 5 mmHg;
• also includes reductions smaller than 5 mmHg.

The trial supports benefit but leaves some uncertainty about whether the population mean reduction reaches the planning target.

Worked example

Minimal important differences

A minimal important difference, abbreviated MID, is a context-specific threshold intended to represent the smallest effect considered important.

It can be informed by:

• patient or clinician anchors;
• outcome-distribution information;
• existing trials;
• decision modelling;
• costs and harms; and
• stakeholder consultation.

An MID is uncertain and may differ by population, baseline severity, follow-up time and intervention burden [3]. Target differences for trials should therefore be justified through a structured process and sensitivity analysis rather than chosen from feasibility alone [4].

An individual-level meaningful change threshold is not automatically the same as a meaningful difference between group means.

Relative and absolute importance

A relative effect can appear impressive while the absolute benefit is small.

For example, a 25% relative risk reduction means:

• 20% to 15%: absolute reduction 5 percentage points;
• 4% to 3%: absolute reduction 1 percentage point.

Clinical decisions usually require both absolute and relative measures, together with time horizon and baseline risk.

Surrogate and patient-important outcomes

A statistically significant change in a biomarker may not translate into improved symptoms, functioning, quality of life or survival.

Clinical importance depends partly on whether the outcome is:

• directly experienced by patients;
• a validated surrogate;
• a process measure; or
• an intermediate biological measure.

A larger effect on a weak surrogate may be less important than a modest effect on a patient-important outcome.

Harms and burden

A clinically important benefit cannot be evaluated in isolation.

Consider:

• serious adverse events;
• inconvenience;
• treatment complexity;
• financial cost;
• opportunity cost;
• equity of access;
• clinician workload; and
• duration of benefit.

A 3 mmHg reduction from a free, safe and scalable intervention may be worthwhile. A 6 mmHg reduction from a toxic or costly intervention may not be.

Decision thresholds and uncertainty MMed extension

A decision threshold may be represented as a distribution rather than one fixed number because patients and clinicians differ in the benefit they consider worthwhile.

Decision analysis can combine uncertain treatment effects with harms, costs and preferences. This makes explicit why statistical significance alone is insufficient for treatment or policy decisions.

Suggested reporting

Common mistakes

Knowledge check

Can an effect be statistically significant but clinically unimportant?

Answer and reasoning

Yes. A large sample can detect a very small effect precisely. Statistical significance concerns incompatibility with a null value, not whether the magnitude matters.

What does a large but non-significant estimated benefit mean?

Answer and reasoning

It may indicate potentially important benefit with insufficient precision. The confidence interval should show whether no effect, benefit and harm remain compatible.

Why should a risk ratio be accompanied by absolute risks?

Answer and reasoning

The same relative effect can produce very different absolute benefits at different baseline risks. Clinical decisions usually depend strongly on absolute outcome differences.

Does an observed effect below the planning target automatically lack clinical importance?

Answer and reasoning

No. The planning target is one design input. Final interpretation should use the estimate, interval, harms, burden, costs and stakeholder values.

Why is a patient-reported outcome often more directly interpretable than a surrogate biomarker?

Answer and reasoning

It reflects symptoms, function or quality of life experienced by patients. A surrogate requires evidence that changing it reliably predicts patient-important benefit.

Key take-home messages

References

1. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. https://doi.org/10.1080/00031305.2016.1154108
2. Sullivan GM, Feinn R. Using effect size—or why the P value is not enough. J Grad Med Educ. 2012;4(3):279–282. https://doi.org/10.4300/JGME-D-12-00156.1
3. Jaeschke R, Singer J, Guyatt GH. Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10(4):407–415. https://doi.org/10.1016/0197-2456(89)90005-6
4. Cook JA, Julious SA, Sones W, Hampson LV, Hewitt C, Berlin JA, et al. DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. BMJ. 2018;363:k3750. https://doi.org/10.1136/bmj.k3750