Type I Error, Type II Error and Statistical Power

Overview

A hypothesis test makes a decision using data generated under uncertainty.

Two decision errors are possible:

• type I error: rejecting the null hypothesis when the specified null scenario is true;
• type II error: not rejecting the null hypothesis when a specified alternative scenario is true.

The probability of type I error is commonly denoted $\alpha$. The probability of type II error under a particular alternative is denoted $\beta$.

Statistical power is:

$$1-\beta$$

It is the probability that the prespecified test rejects the null when the specified alternative and other planning assumptions are true.

These are long-run operating characteristics. They are not posterior probabilities about hypotheses after one study has been completed.

Why this matters

A study with low power may fail to detect an effect that would matter clinically. A study with excessive type I error may generate too many false-positive decisions under null scenarios.

However, power is not a general measure of study quality. A highly powered study can be biased, use the wrong outcome or estimate a clinically trivial effect precisely.

Error rates are meaningful only when the hypotheses, design, test, effect size and assumptions are stated.

Decision table

Truth scenario and testing decision

simulated data

Table 1. Type I and type II errors

Underlying scenario for the defined parameter	Reject $H_0$	Do not reject $H_0$
Null scenario true	Type I error	Correct non-rejection
Specified alternative true	Correct rejection	Type II error

Note: “True” refers to the parameter scenario under the statistical model, not certainty about the real clinical mechanism.

The table describes repeated use of the same procedure. One completed study produces one decision; it does not reveal whether that decision is an error.

Type I error

If $\alpha=0.05$, the testing procedure is designed so that under the null model, approximately 5% of repeated studies reject the null, assuming all test conditions hold.

This interpretation requires:

• a prespecified hypothesis;
• a fixed testing procedure;
• correct model assumptions;
• no unaccounted multiple testing; and
• no data-dependent stopping or analysis selection.

If investigators test 20 outcomes and report only the smallest p-value, the probability of at least one false-positive finding can be much greater than 5%.

Type II error

A type II error occurs when the test does not reject the null under a specified alternative effect.

The value of $\beta$ is not one universal property of a study. It changes with the true effect.

A trial may have:

• 90% power for a 5 mmHg effect;
• less power for a 3 mmHg effect;
• more power for an 8 mmHg effect.

Therefore, a statement such as “the study has 90% power” is incomplete without the alternative effect and other assumptions.

Statistical power

Power answers:

If the specified alternative were true and the study were repeated under the planning assumptions, how often would the test reject the null?

Power does not answer:

• What is the probability that the intervention works?
• What is the probability the completed result is correct?
• What is the probability the effect exceeds the clinical threshold?
• What effect will the study observe?

What affects power?

Effect size

Larger departures from the null are easier to detect.

Sample size

More independent information reduces standard error and generally increases power.

Outcome variability

Greater variability makes a fixed mean difference harder to detect.

Significance level

A larger $\alpha$ makes rejection easier and increases power, but also increases type I error under the null.

One-sided or two-sided testing

A one-sided test concentrates the type I error in one direction and has greater power for that prespecified direction, but cannot be chosen after observing the data.

Design efficiency

Pairing, prognostic covariate adjustment, balanced allocation and stratification can improve power when used appropriately.

Clustering, missing outcomes, measurement error and unequal allocation can reduce power.

Worked example

Non-significance and absence of evidence

A non-significant result can arise when:

• the effect is absent;
• the effect is smaller than planned;
• the study is imprecise;
• event frequency is lower than expected;
• missing data reduce information;
• the model is inefficient; or
• assumptions are violated.

“Absence of evidence is not evidence of absence” is especially important when the CI remains compatible with clinically important effects [2]. Error rates and power are properties of prespecified procedures under assumed parameter values, not posterior probabilities about a completed study [3].

A sufficiently narrow interval around the null can provide evidence that important effects are unlikely. This is an estimation argument, not merely a large p-value.

Power and sample size

Prospective power calculations require:

• target effect;
• outcome variability or event rates;
• significance level;
• sidedness;
• allocation ratio;
• study design;
• analysis method; and
• anticipated missingness.

Changing the primary outcome or analysis can invalidate the calculation. Classical power analysis formalises the relationship among effect size, sample size, significance level and variability [4].

Sample-size planning is developed in Part XIV.

Multiple testing

When several hypotheses form one family of confirmatory claims, type I error should be defined for the family.

Possible error criteria include:

• per-comparison error rate;
• family-wise error rate;
• false discovery rate.

A nominal 0.05 test for each of many outcomes does not maintain a 5% family-wise error rate.

Type S and type M errors

In low-powered studies that achieve significance:

• a type S error is estimating the wrong direction;
• a type M error is exaggerating the magnitude.

These concepts highlight that low power affects more than whether a test rejects. Significant estimates from small studies can be unstable and inflated [5].

Conditional, predictive and assurance concepts MMed extension

Conditional power evaluates the chance of eventual rejection given interim information and assumptions about future data.

Predictive power or Bayesian predictive probability averages over uncertainty about future parameters under a specified model.

Assurance averages conventional power over uncertainty in planning parameters.

These quantities answer different questions and should not be labelled simply as power without definition.

Suggested reporting

Common mistakes

Knowledge check

What is a type I error?

Answer and reasoning

It is rejection of the null under the defined null scenario. Its probability is controlled prospectively by the testing procedure when assumptions and multiplicity rules are respected.

What does 90% power for a 5 mmHg effect mean?

Answer and reasoning

If the true effect were 5 mmHg and all planning assumptions held, approximately 90% of repeated studies would reject the specified null using the planned test.

Does a non-significant result prove a type II error occurred?

Answer and reasoning

No. The null may be approximately true, or an effect may exist but remain undetected. The true scenario is unknown, so the decision cannot be labelled an error with certainty.

Why does greater outcome variability reduce power?

Answer and reasoning

It increases the standard error for a fixed sample size, making the effect less distinguishable from sampling fluctuation.

Why should observed post hoc power not be reported?

Answer and reasoning

When based on the observed effect, it is largely a transformation of the test statistic and p-value. The estimate and confidence interval communicate magnitude and uncertainty directly.

Key take-home messages

References

1. Hoenig JM, Heisey DM. The abuse of power: the pervasive fallacy of power calculations for data analysis. Am Stat. 2001;55(1):19–24. https://doi.org/10.1198/000313001300339897
2. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. https://doi.org/10.1136/bmj.311.7003.485
3. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. https://doi.org/10.1007/s10654-016-0149-3
4. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale (NJ): Lawrence Erlbaum Associates; 1988.
5. Gelman A, Carlin J. Beyond power calculations: assessing Type S (sign) and Type M (magnitude) errors. Perspect Psychol Sci. 2014;9(6):641–651. https://doi.org/10.1177/1745691614551642