Sample Size for Correlation and Regression

Overview

Correlation and regression studies are planned for different purposes:

• estimating the strength of association between two variables;
• testing whether a population correlation differs from a null value;
• estimating one adjusted regression coefficient;
• explaining variation in an outcome;
• developing a multivariable prediction model; or
• externally validating an existing prediction model.

These purposes require different sample-size methods.

A formula based only on the number of predictors is rarely sufficient. Planning may also require:

• anticipated correlation or partial effect;
• outcome variance;
• predictor variance;
• correlation among predictors;
• event proportion;
• number of predictor parameters;
• anticipated model performance;
• desired precision; and
• intended validation procedure.

Why this matters

A regression model containing ten named variables may use more than ten parameters.

Examples include:

• a four-category predictor requiring three parameters;
• a continuous predictor modelled with four spline parameters;
• a treatment-by-sex interaction requiring an additional parameter; and
• missing-indicator or site effects adding further parameters.

For binary outcomes, the total number of participants is not the only constraint. A dataset of 1,000 participants with only 20 events may provide less information for event-prediction coefficients than a smaller dataset with a common outcome.

Traditional rules such as ten events per variable do not guarantee adequate model performance and can be either too permissive or unnecessarily restrictive [1].

Planning a correlation study

Testing one Pearson correlation

Fisher’s transformation converts a correlation to an approximately normal scale:

For testing $H_0:\rho=\rho_0$ against $\rho=\rho_1$, an approximate sample size is:

This approximation is most directly associated with Pearson correlation under suitable distributional assumptions.

Worked example: correlation

Estimating a correlation precisely

A study may aim to estimate $\rho$ rather than test it.

Bonett and Wright developed sample-size procedures for confidence intervals for Pearson, Kendall and Spearman correlations [2]. Planning requires:

• anticipated correlation;
• desired confidence level;
• target interval width; and
• correlation type.

A test-based sample may still give an interval too wide for interpretation. The objective should determine whether power or precision is primary.

Pearson, Spearman and partial correlation

Pearson correlation measures linear association. Spearman correlation measures monotonic rank association.

Sample-size methods differ. A Pearson calculation should not be relabelled as a Spearman calculation without methodological support.

A partial correlation adjusts for specified covariates. Its precision depends on sample size, number and distribution of adjustment variables, and their relationships. Dedicated software or simulation is preferable.

Regression planning begins with purpose

Estimating one regression coefficient

Suppose the question is:

What is the adjusted mean difference in outcome per 10 mmHg higher systolic blood pressure?

The sample-size method should target the coefficient’s precision or power. Important inputs include:

• residual outcome variance;
• predictor variance;
• correlation between the predictor of interest and other covariates;
• clinically meaningful coefficient;
• significance level; and
• power.

Collinearity reduces the independent information about the predictor.

Overall model or incremental explained variation

For multiple linear regression, Cohen’s $f^2$ is sometimes defined as:

Planning must specify whether the target is:

• all predictors jointly;
• one coefficient;
• an added block of predictors; or
• model prediction.

These are not interchangeable.

Logistic regression

For a binary outcome, sample-size requirements depend on:

• outcome event proportion;
• predictor distribution;
• expected odds ratio or coefficient;
• correlation with other predictors;
• coding and non-linearity;
• separation risk; and
• purpose of the model.

Hsieh and colleagues provided approximations for testing coefficients in linear and logistic regression [3].

A study of one prespecified exposure effect differs from developing a prediction model with many candidate predictors.

Count parameters, not variable names

Candidate prediction-model parameters

simulated data

Table 1. Example parameter count

Predictor	Planned representation	Parameters
Age	Restricted cubic spline with 4 basis terms	4
Sex	Binary indicator	1
Smoking status	Never, former, current	2
Diabetes	Binary indicator	1
Systolic blood pressure	Linear	1
Kidney function	Linear	1
LDL cholesterol	Linear	1
HDL cholesterol	Linear	1
Total		12

Note: Reference categories do not require separate coefficients. LDL = low-density lipoprotein; HDL = high-density lipoprotein.

Selecting predictors from univariable p-values does not reduce the scientific need to account for the candidate parameter space.

Prediction-model development

A prediction model estimates an individual’s future or current outcome probability.

The development sample should be large enough to limit:

• overfitting;
• optimism in model performance;
• imprecision of the model intercept or overall risk;
• imprecision of residual variance for continuous outcomes; and
• instability of predictor effects.

Riley and colleagues proposed criteria for minimum sample size rather than one fixed events-per-parameter rule [4], [5].

For continuous outcomes, criteria include:

• shrinkage;
• optimism in $R^2$;
• precision of residual standard deviation; and
• precision of the mean outcome.

For binary and time-to-event outcomes, criteria include:

• shrinkage;
• optimism in model fit; and
• precision of overall outcome risk.

Inputs may include:

• number of candidate parameters;
• anticipated Cox–Snell $R^2$;
• event proportion or rate;
• outcome variance; and
• follow-up distribution.

External validation is a different objective

External validation evaluates an existing model in new data.

Sample size should support precise estimation of:

• calibration-in-the-large;
• calibration slope;
• discrimination;
• overall outcome risk; and
• clinical utility where relevant.

A rule such as 100 events and 100 non-events may be an initial heuristic but does not guarantee adequate precision. Modern methods calculate requirements for the chosen performance measures [6].

Assumptions and sensitivity analysis

Why shrinkage matters MMed extension

A model fitted in a limited sample often describes random noise as though it were signal. Coefficients become too extreme and apparent performance becomes optimistic.

A global shrinkage factor less than 1 indicates that predictions may be too extreme. Sample-size criteria targeting a shrinkage factor of at least 0.90 aim to limit this overfitting at the design stage [4], [5].

Shrinkage criteria do not remove the need for internal validation and careful model-building procedures.

Suggested reporting

Common mistakes

Knowledge check

Why does a correlation study require complete pairs?

Answer and reasoning

A correlation is calculated from paired values on the same participant. A participant missing either measurement does not contribute to the basic complete-pair correlation.

Why may a predictor with four categories require three parameters?

Answer and reasoning

One category serves as the reference, and three indicator coefficients compare the remaining categories with it. Parameter count, not the variable label, determines model complexity.

Why is ten events per variable inadequate for prediction-model planning?

Answer and reasoning

It ignores anticipated model performance, shrinkage, parameter coding, event proportion and precision of overall risk. Different models may need fewer or many more events per parameter.

How does collinearity affect planning for one regression coefficient?

Answer and reasoning

When the predictor of interest is strongly correlated with other covariates, less independent information remains for estimating its coefficient, increasing variance and sample-size requirements.

Why does external validation need its own calculation?

Answer and reasoning

Validation aims to estimate calibration, discrimination and possibly clinical utility precisely in new data. These objectives differ from estimating model coefficients during development.

Key take-home messages

References

1. van Smeden M, de Groot JAH, Moons KGM, Collins GS, Altman DG, Eijkemans MJC, et al. No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med Res Methodol. 2016;16:163. https://doi.org/10.1186/s12874-016-0267-3
2. Bonett DG, Wright TA. Sample size requirements for estimating Pearson, Kendall and Spearman correlations. Psychometrika. 2000;65(1):23–28. https://doi.org/10.1007/BF02294183
3. Hsieh FY, Bloch DA, Larsen MD. A simple method of sample size calculation for linear and logistic regression. Stat Med. 1998;17(14):1623–1634. https://doi.org/10.1002/(SICI)1097-0258(19980730)17:14%3C1623::AID-SIM871%3E3.0.CO;2-S
4. Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE Jr, Moons KGM, et al. Minimum sample size for developing a multivariable prediction model: Part I—continuous outcomes. Stat Med. 2019;38(7):1262–1275. https://doi.org/10.1002/sim.7993
5. Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE Jr, Moons KGM, et al. Minimum sample size for developing a multivariable prediction model: Part II—binary and time-to-event outcomes. Stat Med. 2019;38(7):1276–1296. https://doi.org/10.1002/sim.7992
6. Riley RD, Debray TPA, Collins GS, Archer L, Ensor J, van Smeden M, et al. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med. 2021;40(19):4230–4251. https://doi.org/10.1002/sim.9025