Multiple Comparisons and Subgroup Analyses

Overview

Every statistical test creates an opportunity to reject a null hypothesis.

When investigators test many outcomes, time points, treatment arms or subgroups, the probability of at least one false-positive rejection can increase substantially under a family of null hypotheses.

This problem is called multiplicity.

A subgroup analysis estimates or compares effects within categories such as sex, age, disease severity or biomarker status. Subgroups can be clinically valuable, but data-driven subgroup claims are especially vulnerable to chance, low power and selective reporting [1], [2].

The appropriate response is not to ban secondary or subgroup analyses. It is to define the inferential family, prespecify important questions, control error rates where confirmatory claims require it and label exploratory findings honestly.

Why this matters

If ten independent null hypotheses are each tested at $\alpha=0.05$, the probability of at least one rejection is:

$$1-(1-0.05)^{10}=0.401$$

This is approximately 40.1%, not 5%.

The exact probability differs when tests are correlated, but the principle remains: more opportunities can produce more apparently positive findings.

For subgroups, a common error is:

The treatment was significant in women but not significant in men; therefore, the treatment works only in women.

This conclusion is invalid unless the difference between subgroup effects is assessed directly, usually with an interaction estimate and confidence interval.

Define the family of hypotheses

A family is a collection of hypotheses for which a joint error criterion is scientifically relevant.

Possible families include:

• all primary treatment comparisons;
• co-primary outcomes;
• multiple doses versus one control;
• several key secondary outcomes;
• repeated primary time points; or
• confirmatory subgroup claims.

Not every analysis in a paper must belong to one universal family. The family should follow the claims and decision structure [3].

Error-rate concepts

Per-comparison error rate

The probability of a type I error for one test, commonly 0.05.

Family-wise error rate

The family-wise error rate, abbreviated FWER, is the probability of one or more false rejections within a defined family under the relevant null configuration.

False discovery rate

The false discovery rate, abbreviated FDR, is the expected proportion of false rejections among all rejected hypotheses under the procedure.

FWER control is common for confirmatory clinical trials. FDR control may be useful in high-dimensional exploratory research where some false discoveries are tolerated in exchange for greater sensitivity.

These criteria answer different questions.

Bonferroni correction

For $m$ tests and desired FWER no greater than $\alpha$, Bonferroni tests each hypothesis at:

For ten tests with family-wise $\alpha=0.05$:

$$0.05/10=0.005$$

Bonferroni is valid regardless of test dependence but can be conservative, especially when tests are strongly correlated.

Holm procedure

The Holm procedure orders the p-values from smallest to largest and compares them sequentially with increasingly less restrictive thresholds.

It controls FWER and is never less powerful than simple Bonferroni for the same family.

The detailed algorithm is usually implemented by software, but the protocol should state that Holm adjustment is planned and define the family.

Hierarchical testing

A hierarchical or gatekeeping strategy tests hypotheses in a prespecified order.

For example:

1. test the primary outcome;
2. if successful, test the first key secondary outcome;
3. continue only while the hierarchy permits.

This approach can preserve type I error while reflecting clinical priority.

The order must be prespecified. Reordering after seeing the results invalidates the procedure.

Co-primary outcomes

If a trial requires both co-primary outcomes to succeed, multiplicity may differ from a situation in which either outcome can establish success.

Planning must state:

• whether all or any outcomes must succeed;
• the type I error allocation;
• correlations among outcomes;
• sample-size implications; and
• handling of missing values.

Repeated time points

Testing treatment effects separately at weeks 4, 8 and 12 creates several opportunities.

Possible strategies include:

• designate one primary time point;
• model the full trajectory and test one prespecified contrast;
• use a hierarchical sequence;
• adjust the family of time-specific tests; or
• label other time points exploratory.

Selecting the time with the smallest p-value after analysis is outcome-driven selection.

Worked example: ten secondary outcomes

Subgroup analyses

A subgroup analysis asks whether the treatment effect differs across levels of another variable.

The primary statistical question is not:

Is treatment significant in subgroup A and non-significant in subgroup B?

It is:

How different are the treatment effects between A and B?

This is assessed using an interaction.

For a numerical outcome:

$$Y=\beta_0+\beta_1T+\beta_2G+\beta_3(T\times G)+\varepsilon$$

where:

• $T$ is treatment;
• $G$ is subgroup;
• $\beta_3$ estimates the difference between treatment effects across subgroups.

Worked example: subgroup interaction

Credibility of subgroup effects

A subgroup claim is more credible when:

• the subgroup variable was measured before treatment;
• the direction and hypothesis were prespecified;
• only a small number of subgroup hypotheses were planned;
• the interaction is statistically supported;
• the effect is large relative to chance variation;
• findings are consistent across related outcomes or studies;
• a credible biological or clinical rationale exists; and
• the effect is replicated [1], [2], [4].

Post-randomisation subgroups, such as adherence or response, can be affected by treatment and require causal methods beyond ordinary interaction analysis.

Continuous subgroup variables

Age, baseline risk and biomarkers should not be divided at arbitrary cut points merely to create subgroups.

Dichotomisation:

• loses information;
• reduces power;
• can create cut-point-dependent results; and
• may conceal non-linear effect modification.

Model continuous interactions using clinically appropriate functional forms, then present interpretable predicted effects across meaningful values.

Forest plots

A forest plot can display subgroup-specific estimates and confidence intervals.

It should also show:

• subgroup denominators;
• interaction p-value or interaction CI;
• prespecified versus exploratory status; and
• consistent effect direction.

Visual non-overlap or one CI crossing the null is not a valid interaction test.

Confirmatory and exploratory analyses

Confirmatory analyses should have:

• prespecified hypotheses;
• controlled error rates;
• appropriate sample size;
• planned analysis; and
• interpretation aligned with the protocol.

Exploratory analyses can generate hypotheses but should be labelled as such. Their uncertainty includes the selection process, which ordinary nominal intervals may not reflect.

Selective inference beyond multiplicity adjustment MMed extension

Multiplicity is not solved completely by adjusting p-values.

Data-dependent choices involving outcomes, models, transformations and subgroups create selection uncertainty. Transparent protocols, registered analysis plans, complete reporting and independent validation are therefore as important as formal correction.

A prespecified estimand and hierarchy can be more effective than testing a large undifferentiated collection of endpoints.

Suggested reporting

CONSORT 2025 recommends identifying prespecified outcomes and analyses, reporting estimates with uncertainty and distinguishing analyses that were not prespecified [5].

Common mistakes

Knowledge check

Why does testing ten hypotheses increase false-positive opportunity?

Answer and reasoning

Each test provides another chance to reject under a null scenario. Under independence and separate 0.05 thresholds, the probability of at least one rejection is about 40.1%.

What does Bonferroni control?

Answer and reasoning

It controls the family-wise probability of one or more false rejections by dividing the family alpha across tests. It may be conservative when tests are correlated.

Why is significant in women and non-significant in men not evidence of a sex difference?

Answer and reasoning

The two subgroup tests assess separate null hypotheses. Effect modification requires a direct comparison of the subgroup effects using an interaction estimate.

What makes a subgroup claim more credible?

Answer and reasoning

Prespecification, few hypotheses, baseline subgroup definition, a supported interaction, plausible mechanism, consistent evidence and replication all strengthen credibility.

Must every exploratory analysis receive a multiplicity adjustment?

Answer and reasoning

Not necessarily. It can be presented as exploratory with estimates and uncertainty, but it must not be promoted as a controlled confirmatory finding. The inferential family and claim determine the need for formal adjustment.

Key take-home messages

References

1. Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine—reporting of subgroup analyses in clinical trials. N Engl J Med. 2007;357(21):2189–2194. https://doi.org/10.1056/NEJMsr077003
2. Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, et al. Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ. 2012;344:e1553. https://doi.org/10.1136/bmj.e1553
3. Bender R, Lange S. Adjusting for multiple testing—when and how? J Clin Epidemiol. 2001;54(4):343–349. https://doi.org/10.1016/S0895-4356(00)00314-0
4. Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ. 2010;340:c117. https://doi.org/10.1136/bmj.c117
5. Hopewell S, Chan AW, Collins GS, Hróbjartsson A, Moher D, Schulz KF, et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ. 2025;389:e081123. https://doi.org/10.1136/bmj-2024-081123