Understanding the P-value

Overview

A p-value is a probability calculated from a statistical model. It measures how unusual the observed test statistic, or a more extreme statistic, would be if the null hypothesis and other assumptions used in the calculation were true.

For a two-sided test, extremeness is assessed in both directions according to the chosen test statistic.

A p-value does not measure:

• the probability that the null hypothesis is true;
• the probability that the alternative hypothesis is true;
• the size or clinical importance of an effect;
• the probability that the result occurred by chance alone;
• study quality; or
• the probability that another study will replicate the result.

The American Statistical Association highlighted these distinctions because misinterpretation of p-values is widespread [1].

Why this matters

A report that states only $p=0.03$ does not tell the reader:

• whether the effect is large or small;
• whether the estimate is precise;
• whether the design supports causal inference;
• whether many tests were performed;
• whether the analysis was prespecified;
• whether assumptions are reasonable; or
• whether the result matters clinically.

Scientific conclusions should not depend only on whether a p-value crosses 0.05 [1], [2]. Reviews of common inferential errors [3] and medical-research critiques of significance testing [4] reinforce this concern.

The conditional definition

The p-value can be written conceptually as:

$$P(\text{test statistic at least as extreme as observed}\mid H_0\text{ and model assumptions})$$

The condition appears after the vertical bar. The calculation assumes the null model and asks about possible data or statistics.

It does not reverse the condition to calculate:

$$P(H_0\mid\text{observed data})$$

That reverse probability requires additional information and a different inferential framework.

Worked example

What makes a p-value smaller?

For a simple Wald statistic, the p-value is determined by the estimate relative to its SE.

A smaller p-value can arise from:

• a larger estimated effect;
• a smaller standard error;
• a larger sample size;
• lower outcome variability;
• a more efficient design;
• a one-sided rather than two-sided test; or
• a different model or hypothesis.

Therefore, a very small p-value does not necessarily indicate a large effect.

Large samples and small effects

A very large study can produce $p<0.001$ for a mean difference of 0.5 mmHg if the SE is sufficiently small. The result may be statistically incompatible with zero but clinically negligible.

Small samples and important effects

A small study can estimate a clinically important effect but yield $p>0.05$ because the CI is wide. Non-significance does not make the estimated effect unimportant; it indicates unresolved uncertainty.

p-values near 0.05

A result of $p=0.049$ and one of $p=0.051$ provide nearly the same graded information under comparable analyses.

Labelling the first a discovery and the second no effect creates an artificial discontinuity.

Two nearly identical simulated results

simulated data

Table 1. Results around the conventional threshold

Study	Mean difference	95% CI	p-value
A	−3.1 mmHg	−6.2 to −0.01	0.049
B	−3.0 mmHg	−6.0 to 0.02	0.051

Note: Values are simulated and rounded. The results should not be treated as qualitatively opposite.

The estimates, intervals, design and prior evidence should be compared directly.

A large p-value

A large p-value means the observed statistic is not unusual under the specified null model. It does not show that the null is true.

Possible explanations include:

• the effect is near the null;
• the study has low precision;
• the data are highly variable;
• the test has low power for the relevant effect;
• model assumptions obscure the pattern; or
• the effect exists on another scale or in another population.

The confidence interval reveals whether clinically important effects remain compatible.

A small p-value

A small p-value indicates stronger incompatibility with the null model than a larger one, all else being equal.

It does not protect against:

• bias;
• confounding;
• measurement error;
• selective outcome reporting;
• data fabrication;
• multiple testing;
• model misspecification; or
• an implausible hypothesis generated after seeing the data.

A precisely calculated p-value from a biased analysis remains biased evidence.

Exact values and inequalities

Report exact p-values when they are informative, commonly to three decimal places:

• $p=0.032$;
• $p=0.410$.

For very small values, report $p<0.001$; software output rounded to zero should not be written as an exact probability of zero.

The phrase “highly significant” is usually unnecessary. The estimate and interval provide more useful information.

Multiple testing and selective analysis

If many hypotheses are tested, at least one small p-value becomes more likely even when all null hypotheses hold.

Interpretation changes when researchers:

• test many outcomes;
• try several transformations;
• examine many subgroups;
• select covariates according to significance;
• stop data collection after significance; or
• report only favourable tests.

The nominal p-value usually does not account for this selection unless the procedure explicitly incorporates it.

Multiple comparisons are developed in BIO-P04-M09.

Compatibility across null values MMed extension

A single p-value tests one null value. A p-value function calculates compatible two-sided p-values across a range of parameter values.

Its 0.05 cut points correspond to the endpoints of a compatible 95% confidence interval when the test and interval use matching methods.

This reinforces that inference should not focus exclusively on the point null of zero.

Suggested reporting

Common mistakes

Knowledge check

What probability is calculated under a null model?

Answer and reasoning

It calculates the probability, under the null model and assumptions, of a test statistic at least as extreme as the observed statistic. It does not calculate the probability that the null is true.

Why can a tiny effect produce a very small probability value?

Answer and reasoning

A very large sample can produce a small standard error. The effect can then lie many standard errors from zero even when its clinical magnitude is trivial.

Does a reported value of 0.40 support the conclusion that the treatments are equivalent?

Answer and reasoning

No. A large p-value indicates limited incompatibility with the null. Equivalence requires prespecified margins and sufficiently precise confidence-interval evidence.

How should reported values of 0.049 and 0.051 be compared?

Answer and reasoning

They provide nearly continuous evidence and should be interpreted with their estimates, intervals and study context. A rigid label should not make them qualitatively opposite.

Why does multiple testing affect interpretation?

Answer and reasoning

Testing many hypotheses creates more opportunities for an apparently small p-value by chance under the null. A single nominal value may not represent the full selection process.

Key take-home messages

References

1. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. https://doi.org/10.1080/00031305.2016.1154108
2. Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p < 0.05”. Am Stat. 2019;73(Suppl 1):1–19. https://doi.org/10.1080/00031305.2019.1583913
3. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. https://doi.org/10.1007/s10654-016-0149-3
4. Sterne JAC, Davey Smith G. Sifting the evidence—what’s wrong with significance tests? BMJ. 2001;322(7280):226–231. https://doi.org/10.1136/bmj.322.7280.226