Hypothesis Testing

Overview

A statistical hypothesis test is a prespecified procedure for assessing how compatible observed data are with a statistical hypothesis.

The procedure usually includes:

1. a null hypothesis;
2. an alternative hypothesis;
3. a test statistic;
4. a probability model for the statistic under the null;
5. a significance level; and
6. a decision rule.

Hypothesis testing does not determine whether a clinical intervention is worthwhile. It addresses a narrower statistical question under a model. Good reporting therefore combines the test with an effect estimate, 95% confidence interval, assumptions and clinical interpretation.

Why this matters

A test result is often reduced to “significant” or “not significant”. This binary label discards information.

Two studies can both reject the null while estimating very different effect sizes. Two studies can both fail to reject the null while one excludes important effects and the other remains highly uncertain.

Hypothesis testing is useful when the decision rule is prespecified and interpreted within the research design. It becomes misleading when treated as a substitute for estimation or scientific judgement [1].

From clinical question to statistical hypothesis

The clinical question is:

Does allocation to telemonitoring change mean 12-week systolic blood pressure compared with usual care?

The population parameter is:

$$\Delta=\mu_{\text{telemonitoring}}-\mu_{\text{usual care}}$$

The hypotheses are:

$$H_0:\Delta=0$$ $$H_1:\Delta\ne0$$

$H_0$ is the null hypothesis. It specifies the value used to generate the reference distribution for the test.

$H_1$ is the alternative hypothesis. In a two-sided test, it includes effects in either direction.

The hypotheses must match the estimand and effect scale. A test of zero mean difference is not a test of equal medians or equal individual responses.

Test statistic

A test statistic summarises the distance between the estimate and the null value relative to its standard error.

A value far from zero indicates that the estimate is many standard errors from the null.

Depending on the analysis, the statistic may follow approximately a standard normal, t, chi-square, F, exact permutation or another reference distribution.

Significance level and rejection region

The significance level, $\alpha$, controls the long-run probability of rejecting the null when the complete null model is true and the test assumptions hold.

For a two-sided standard-normal test with $\alpha=0.05$, the rejection region is approximately:

$$z<-1.96\quad\text{or}\quad z>1.96$$

The decision rule is selected before examining the result.

The significance level does not mean that 5% of rejected hypotheses are necessarily false. That proportion also depends on which hypotheses are tested, their plausibility, power, bias and selective reporting.

Worked example

What rejection means

Rejecting $H_0$ at $\alpha=0.05$ means:

• the statistic falls in the prespecified rejection region;
• the compatible two-sided p-value is below 0.05; and
• the data are relatively incompatible with the null model.

It does not mean:

• the probability that $H_0$ is true is below 5%;
• the probability the result occurred by chance is below 5%;
• the treatment works for every patient;
• the effect is clinically important;
• the study is unbiased; or
• replication is guaranteed.

What non-rejection means

If $p\ge0.05$, state that the null was not rejected at the prespecified significance level. Do not state that it was accepted.

Non-rejection can occur because:

• the true effect is near the null;
• the study is imprecise;
• outcome variability is large;
• fewer events occurred than expected;
• missingness reduced information; or
• model assumptions were unsuitable.

Suppose another simulated study estimates a mean difference of −2.0 mmHg, 95% CI −5.2 to 1.2, with $p=0.22$. The interval includes no effect, a potentially useful reduction and a small increase. The study does not establish equivalence or absence of benefit.

Tests and confidence intervals

For compatible two-sided procedures:

• a 95% CI excluding the null corresponds to $p<0.05$;
• a 95% CI including the null corresponds to $p\ge0.05$.

The CI adds the range and direction of compatible effects. Hypothesis testing should not replace estimation.

Composite null hypotheses

Not every null is a single point.

Examples include:

• non-inferiority null: treatment is worse by at least the margin;
• equivalence null: effect lies outside acceptable margins;
• goodness-of-fit null: data follow a specified distribution; and
• interaction null: treatment effects are equal across subgroups.

The decision rule and confidence interval must match the hypothesis structure.

Assumptions

A valid test requires more than a formula. Relevant assumptions may include:

• correct randomisation or sampling process;
• independent units or correctly modelled dependence;
• correctly specified outcome and contrast;
• appropriate variance estimation;
• suitable missing-data handling;
• no severe model misspecification; and
• no outcome-driven choice among many tests.

A very small p-value does not repair an inappropriate design.

Fisherian evidence and Neyman–Pearson decisions MMed extension

Modern practice often combines two historical approaches.

Fisher used the p-value as a graded measure of incompatibility with a null model. Neyman–Pearson theory formalised decision rules with type I error, type II error and power under specified alternatives.

Combining a fixed threshold with language of graded evidence can create confusion. Reports should state the prespecified decision rule while retaining the continuous estimate and uncertainty [2]. Broader critiques of significance testing likewise emphasise estimation, study design and scientific context [3]. Standard medical-statistical guidance describes how the test statistic and reference distribution must match the design [4].

Suggested reporting

Common mistakes

Knowledge check

What does the null hypothesis H0: Δ = 0 specify?

Answer and reasoning

It specifies that the population treatment contrast on the defined mean-difference scale is zero. It does not claim that every participant has an identical response.

What does z = −4.50 represent?

Answer and reasoning

The estimated mean difference lies 4.50 estimated standard errors below the null value. The sign indicates direction; the magnitude determines incompatibility with the null reference distribution.

Does a reported value of 0.22 establish that there is no treatment effect?

Answer and reasoning

No. It indicates non-rejection at the chosen level. The confidence interval is needed to determine whether important beneficial or harmful effects remain compatible with the data.

Why must the significance level be selected before seeing the result?

Answer and reasoning

Choosing the threshold or sidedness after seeing the data makes the stated long-run error rate invalid because the selection process is omitted from the calculation.

Why should an effect estimate accompany the test result?

Answer and reasoning

The test addresses compatibility with a null value, while the estimate and confidence interval show the magnitude, direction and precision needed for clinical interpretation.

Key take-home messages

References

1. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. https://doi.org/10.1080/00031305.2016.1154108
2. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. https://doi.org/10.1007/s10654-016-0149-3
3. Sterne JAC, Davey Smith G. Sifting the evidence—what’s wrong with significance tests? BMJ. 2001;322(7280):226–231. https://doi.org/10.1136/bmj.322.7280.226
4. Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.