Effect Size Measures

Overview

An effect size quantifies the magnitude of a difference, association or treatment effect.

Examples include:

• mean difference in mmHg;
• standardised mean difference in standard-deviation units;
• risk difference in percentage points;
• risk ratio;
• odds ratio;
• correlation coefficient; and
• number needed to treat over a stated period.

No single effect measure is best for every question. The measure should match the estimand, outcome scale, study design and intended interpretation.

Effect sizes should be reported with confidence intervals. A point estimate alone does not show precision [1], [2].

Why this matters

A p-value cannot tell readers how large an effect is. It can become smaller because the sample is larger even when the clinical difference is unchanged.

Effect measures provide complementary information:

• raw measures preserve clinical units;
• standardised measures permit comparison across scales;
• absolute measures communicate event burden;
• relative measures communicate proportional change.

A complete report often presents an absolute and a relative effect for a binary outcome.

Raw effect measures

A raw effect measure remains on the original clinical scale.

Examples include:

• mean difference in mmHg;
• median difference in days;
• risk difference in percentage points;
• difference in quality-of-life score; and
• restricted mean survival-time difference in months.

Raw effects are often the most directly interpretable.

Mean difference

The contrast direction must be stated.

In the simulated trial:

telemonitoring minus usual care = −5.9 mmHg.

Negative values favour telemonitoring because lower systolic blood pressure is desired.

Standardised mean difference

A standardised mean difference, abbreviated SMD, divides a mean difference by a chosen standard deviation.

For −5.9 mmHg and pooled SD 12.1 mmHg:

$$d=\frac{-5.9}{12.1}=-0.49$$

A small-sample correction produces Hedges’ $g$, which is slightly closer to zero. With this sample size, $g$ is approximately −0.49.

The SMD is useful when studies measure the same underlying construct using different scales. It sacrifices the original unit and depends on the chosen standardiser [2].

Generic labels such as small, medium and large should not replace comparison with effects relevant to the clinical field.

Absolute effects for binary outcomes

Risk difference

Let $R_1$ and $R_0$ be the event risks in the two groups.

For blood-pressure control:

$$R_1=\frac{74}{114}=0.649$$ $$R_0=\frac{54}{112}=0.482$$ $$RD=0.649-0.482=0.167$$

The observed risk difference is 16.7 percentage points. A simulated Newcombe 95% CI is 4.0 to 28.7 percentage points.

A positive value favours telemonitoring because the event is beneficial control.

Absolute effects depend on baseline risk and are often valuable for clinical and public-health decisions.

Number needed to treat

For a beneficial binary outcome, the number needed to treat, abbreviated NNT, can be calculated as:

Using $RD=0.167$:

$$NNT=\frac{1}{0.167}=5.99$$

Approximately six participants would need the intervention for one additional participant to achieve control by 12 weeks, under the trial conditions and assuming a causal interpretation is justified.

The NNT must include:

• outcome definition;
• time horizon;
• comparator;
• population; and
• confidence interval.

Inverting the simulated risk-difference interval gives an approximate NNT range from 4 to 25, with endpoint order reversed by the inversion. Calculation of the risk-difference interval and corresponding NNT range is not shown here. NNT intervals can become discontinuous when the risk-difference interval crosses zero, so they should be derived and reported carefully [3].

Relative effects for binary outcomes

Risk ratio

For the simulated data:

$$RR=\frac{0.649}{0.482}=1.35$$

Participants receiving telemonitoring were 1.35 times as likely to achieve control as those receiving usual care among participants with observed outcomes.

A risk ratio greater than 1 favours telemonitoring because the event is beneficial. If the event were harm, the clinical direction would differ.

Relative risk increase or reduction

For a beneficial event, a risk ratio of 1.35 corresponds to a relative increase of approximately 35%:

$$(RR-1)\times100\%=35\%$$

Relative language should not replace the group risks.

Odds ratio

Odds are event probability divided by non-event probability.

For the simulated control outcome:

• telemonitoring controlled: 74;
• telemonitoring not controlled: 40;
• usual care controlled: 54;
• usual care not controlled: 58.

$$OR=\frac{74\times58}{40\times54}=1.99$$

The odds of control are approximately twice as high with telemonitoring.

The odds ratio is not the risk ratio. When outcomes are common, interpreting an OR as an RR exaggerates the relative comparison [4]. Effect measures and uncertainty should be interpreted without reducing evidence to a threshold [5].

Correlation as an effect measure

The Pearson correlation coefficient $r$ measures the strength and direction of linear association between two numerical variables.

It ranges from −1 to 1.

Its squared value, $r^2$, is the proportion of variance in one variable accounted for by a simple linear relationship with the other in that sample-model context.

Correlation is not:

• agreement;
• causation;
• a mean difference; or
• a treatment effect automatically.

The clinical meaning of $r=0.50$ depends on measurement reliability, range and purpose.

Effects for more than two groups

Analysis of variance may report:

• eta squared, $\eta^2$;
• partial eta squared, $\eta_p^2$;
• omega squared, $\omega^2$; or
• Cohen’s $f$.

These measures are not interchangeable. Partial eta squared can be much larger than generalised measures in multifactor or repeated designs.

Pairwise contrasts in clinical units should accompany omnibus effect measures when they answer the clinical question [2].

Worked example

Choosing an effect measure

Non-collapsibility of the odds ratio MMed extension

An adjusted odds ratio can differ from an unadjusted odds ratio even without confounding because the odds ratio is non-collapsible.

Therefore, a numerical change after covariate adjustment does not by itself prove confounding. The target conditional or marginal effect and the analysis model should be stated.

Risk differences and risk ratios have different collapsibility properties and may answer different population questions.

Suggested reporting

Common mistakes

Knowledge check

Why is the raw mean difference usually easier to interpret than an SMD?

Answer and reasoning

It remains in the clinical unit, such as mmHg. The SMD expresses the difference relative to a chosen SD and requires additional context.

What does a risk difference of 16.7 percentage points mean?

Answer and reasoning

The observed probability of control was 16.7 percentage points higher with telemonitoring than with usual care over the stated 12-week period.

Why is OR = 1.99 not the same as RR = 1.35?

Answer and reasoning

The OR compares odds, while the RR compares probabilities. Because control is common, odds and risk differ substantially.

What information must accompany an NNT of 6?

Answer and reasoning

The report should state the outcome, time horizon, comparator, population, underlying risks and uncertainty. Without these, the NNT is not transportable or fully interpretable.

Why can the same raw difference produce different SMDs in two populations?

Answer and reasoning

The standardised effect divides by a variability measure. If the populations have different SDs, the denominator and therefore the SMD differ even when the raw difference is identical.

Key take-home messages

References

1. Sullivan GM, Feinn R. Using effect size—or why the P value is not enough. J Grad Med Educ. 2012;4(3):279–282. https://doi.org/10.4300/JGME-D-12-00156.1
2. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. https://doi.org/10.3389/fpsyg.2013.00863
3. Altman DG. Confidence intervals for the number needed to treat. BMJ. 1998;317(7168):1309–1312. https://doi.org/10.1136/bmj.317.7168.1309
4. Davies HTO, Crombie IK, Tavakoli M. When can odds ratios mislead? BMJ. 1998;316(7136):989–991. https://doi.org/10.1136/bmj.316.7136.989
5. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. https://doi.org/10.1007/s10654-016-0149-3