A Practical Framework for Choosing a Statistical Test

Overview

Choosing a statistical test is not a matter of matching one variable name to one item in a table. The appropriate analysis follows from the research question, study design, outcome, comparison structure, estimand and assumptions.

A useful sequence is:

research objective → estimand → outcome type → data structure → dependence → effect measure → statistical model → assumptions → reporting

The estimand is the population quantity that answers the research question. Examples include:

• a mean difference;
• a risk difference;
• a risk ratio;
• an odds ratio;
• a median difference;
• a correlation;
• a rate ratio;
• a hazard ratio; or
• a difference in restricted mean survival time.

The test or model should estimate the intended quantity and provide an uncertainty measure. Beginning with the name of a test can obscure what is actually being estimated [1].

Why this matters

Several methods may be technically possible for the same dataset, but they do not necessarily answer the same question.

For example, two independent groups with a numerical outcome could be analysed using:

• Welch’s independent-samples t-test;
• linear regression;
• analysis of covariance;
• a Mann–Whitney test;
• quantile regression;
• a transformation-based model; or
• a robust method.

These methods differ in:

• the effect they estimate;
• the assumptions they require;
• whether covariates can be included;
• how they handle unequal variance;
• and how results should be interpreted.

A Mann–Whitney test is not simply a “non-parametric test of medians”. It assesses a rank-based distributional contrast, and a median interpretation requires additional conditions. Similarly, a statistically significant correlation does not demonstrate agreement or causation.

The purpose of this framework is therefore not to memorise a long list of tests. It is to make the reasoning behind the choice explicit.

When to use this framework

Use the framework before:

• writing the statistical-analysis section of a protocol;
• calculating sample size;
• building the analysis dataset;
• selecting software procedures;
• analysing study outcomes; or
• reviewing a manuscript’s statistical methods.

The framework is useful for:

• descriptive studies;
• clinical trials;
• cohort and case–control studies;
• surveys;
• diagnostic studies;
• repeated-measurement studies;
• prognostic research; and
• laboratory or method-comparison studies.

Not every research question requires a hypothesis test. A descriptive objective may be answered adequately by an estimate and confidence interval. Prediction, agreement and diagnostic accuracy also require methods beyond conventional group-comparison tests.

Research question and variables

A clear analysis plan identifies four elements:

1. the population;
2. the outcome;
3. the explanatory variable or grouping structure; and
4. the intended contrast or association.

Outcome variable

The outcome is the variable whose distribution or expected value is being described or modelled.

Common outcome types include:

• numerical;
• binary;
• nominal categorical;
• ordinal;
• count;
• rate;
• time to event; and
• repeated or longitudinal outcome.

Variable type is important, but it is not enough. The same numerical variable can be:

• an outcome;
• an exposure;
• an adjustment variable;
• a matching variable; or
• an effect modifier.

Explanatory variable

The main explanatory variable may represent:

• treatment allocation;
• exposure status;
• diagnostic-test result;
• time;
• dose;
• patient characteristic;
• intervention period; or
• another predictor.

Its role in the research question matters more than whether it is stored as text or a numerical code.

Comparison structure

Determine whether the question involves:

• one population;
• two independent groups;
• three or more independent groups;
• paired observations;
• repeated measurements;
• matched sets;
• clusters;
• several predictors;
• or no group comparison.

Estimand and effect measure

The effect measure should be stated in clinical units where possible.

Examples include:

• telemonitoring minus usual-care mean difference in mmHg;
• absolute risk difference in percentage points;
• risk ratio;
• change in median length of stay;
• incidence-rate ratio;
• odds ratio;
• correlation coefficient;
• mean measurement bias and limits of agreement; and
• hazard ratio or restricted mean survival-time difference.

Data structure

Variables in the simulated telemonitoring trial

simulated data

Table 1. Variable roles and structures

Variable	Data type	Role in selected analyses	Structure
Allocated group	Binary categorical	Main explanatory variable	One value per participant
Baseline systolic blood pressure	Numerical	Adjustment variable	One value per participant
Week-12 systolic blood pressure	Numerical	Primary outcome	One value per participant
Blood-pressure control	Binary categorical	Secondary outcome	One value per participant
Monthly home blood pressure	Numerical	Repeated outcome	Several values per participant
Adherence rating	Ordinal categorical	Secondary outcome	One ordered category per participant
Unscheduled clinic visits	Count	Secondary outcome	Non-negative integer
Time to readmission	Time to event	Secondary outcome	Time plus event indicator
Clinic	Nominal categorical	Stratification or cluster variable	Participants nested within clinic

Note: The appropriate method depends on the research role and dependence structure, not only the data type.

Wide and long formats

A participant-level analysis of the 12-week outcome may use one row per participant.

A repeated-measurement analysis should usually use a long structure with one row per participant-time observation.

The data arrangement does not determine the analysis by itself, but it must represent:

• participant identifiers;
• time;
• clustering;
• pairing;
• and missing observations correctly.

A practical framework

Step 1: State the objective

First classify the objective.

Is it primarily:

• description or estimation;
• comparison;
• association;
• agreement;
• prediction;
• causal effect estimation;
• diagnostic accuracy;
• or time-to-event analysis?

These objectives are not interchangeable.

For example:

• Pearson correlation assesses linear association;
• Bland–Altman analysis assesses agreement between measurement methods;
• linear regression estimates an adjusted numerical association;
• a causal effect requires additional design and identification assumptions;
• a prediction model targets future or current outcome prediction.

Step 2: Define the estimand

State the exact population quantity.

Poor specification:

Compare blood pressure between groups.

Better specification:

Estimate the telemonitoring minus usual-care difference in mean clinic systolic blood pressure at 12 weeks among all randomised participants.

The improved wording identifies:

• outcome;
• time point;
• contrast direction;
• summary measure;
• and analysis population.

Step 3: Identify the outcome type

The outcome distribution often determines the model family.

Table 2. Common outcome types and modelling families

Note: The specific method also depends on design, dependence and estimand.

Step 4: Determine independence or dependence

Observations are independent when one participant’s outcome does not determine or share structured variation with another observation under the design.

Dependence arises from:

• before-and-after measurements;
• repeated visits;
• matching;
• twins or family members;
• several lesions from one participant;
• eyes, teeth or joints within a participant;
• participants within clinics;
• repeated readers; and
• multiple diagnostic tests on the same person.

The independent unit must be identified before choosing a formula.

Paired and clustered data require methods that model or account for their correlation. Treating all rows as independent commonly produces incorrect standard errors.

Step 5: Count groups and time points

For numerical outcomes:

• one sample versus a reference value differs from two independent groups;
• two independent groups differ from paired groups;
• three groups require an overall or contrast-based method;
• repeated time points require a repeated-measurement model.

Conducting several two-group tests for three or more groups increases multiplicity and does not provide one coherent overall analysis.

Step 6: Decide whether adjustment is required

A simple test may estimate an unadjusted association. Regression is often preferable when the analysis must:

• adjust for baseline outcome;
• account for stratification variables;
• control prespecified confounders;
• model several predictors;
• test interaction;
• incorporate non-linear terms;
• or produce adjusted predictions.

Adjustment variables should be selected using design and substantive knowledge, not solely from univariable p-values.

Step 7: Select the effect measure

Choose the effect measure before selecting the software command.

For a binary outcome, possible measures include:

• risk difference;
• risk ratio;
• odds ratio.

Logistic regression estimates odds ratios naturally, but an odds ratio should not be described as a risk ratio. Modified Poisson regression or standardisation may be used when risk ratios or risk differences are the target.

For a numerical outcome, common targets include:

• mean difference;
• mean change difference;
• median difference;
• ratio of geometric means;
• quantile difference.

Step 8: Assess assumptions

Assumptions apply to the method and estimand, not to the variable in isolation.

For linear models, relevant issues may include:

• independence;
• linearity;
• suitable residual distribution for the intended inference;
• variance structure;
• influential observations;
• correct functional form;
• and appropriate handling of missing data.

For logistic regression, raw outcomes are binary and cannot be normally distributed. Relevant assumptions instead include:

• correct link and functional form;
• adequate event information;
• no complete separation;
• independence or modelled clustering;
• and valid adjustment.

For Cox regression, proportional hazards may be important if one hazard ratio is the intended summary.

A formal normality-test p-value should not be used as the sole decision rule [2], [3].

Step 9: Consider missing data and analysis population

The method must be compatible with:

• intention-to-treat or other analysis population;
• complete or incomplete outcomes;
• repeated-measurement missingness;
• structural missing values;
• and the prespecified missing-data assumptions.

Deleting every incomplete record may change the target population and reduce precision.

Step 10: Address multiplicity

Multiplicity arises when many:

• outcomes;
• groups;
• time points;
• models;
• thresholds;
• or subgroups

are tested.

The plan should identify which analyses are primary, supportive or exploratory and whether type I error control is required.

Step 11: Report estimates and uncertainty

The final output should prioritise:

• descriptive summaries;
• effect estimate;
• 95% confidence interval;
• denominator;
• analysis population;
• exact or appropriately bounded p-value;
• assumptions;
• and clinical interpretation.

Common method choices

The following tables are starting points, not automatic rules.

Table 3. Numerical outcomes

Note: GEE = generalised estimating equations. Rank tests do not automatically estimate median differences.

Table 4. Categorical, count and time-to-event outcomes

Note: Fisher’s exact test addresses sparse contingency-table inference but does not replace reporting an effect estimate and confidence interval.

Table 5. Association, agreement, diagnosis and prediction

Schober and colleagues provide guidance on choosing and interpreting correlation coefficients [4]. Bland and Altman’s method-comparison work demonstrates why agreement requires analysis of measurement differences rather than correlation alone [5].

Statistical assumptions

Parametric versus non-parametric is not the first decision

The first decision is not:

Are the data normal?

The first decisions are:

• What is the question?
• What is the estimand?
• What is the dependence structure?

Parametric methods estimate parameters in a specified model. Non-parametric and rank-based methods use fewer or different distributional assumptions, but they still have assumptions and target particular contrasts [2].

Simulation work has shown that analysis of covariance can remain useful for randomised comparisons with non-normally distributed outcomes when the mean effect is the target, whereas a rank test answers a different question [3].

How the framework works

A practical one-sentence template is:

To estimate [effect measure] for [outcome] between or across [groups, times or predictors] in [independent, paired, repeated or clustered observations], use [model or method], provided [key assumptions] are reasonable.

Examples:

To estimate the adjusted mean difference in 12-week systolic blood pressure between two independently randomised groups, use baseline-adjusted linear regression, provided the model form and residual-based inference are suitable.

To estimate the risk ratio for blood-pressure control between two independent groups, use a binomial log-link model or modified Poisson regression with robust standard errors.

To estimate the association between monthly time and repeated blood pressure within participants, use a longitudinal mixed-effects model or GEE.

This wording forces the method to remain connected to the estimand.

Worked example

Interpreting the output

Why the choice is defensible

The selected method aligns:

• clinical question;
• numerical outcome;
• independent randomised groups;
• baseline adjustment;
• effect measure;
• missing-data plan;
• and reporting scale.

The method is not selected solely because a normality test is non-significant or because it produces the smallest p-value.

Presenting the results

A complete result should include:

• group-specific descriptive summaries;
• number observed;
• analysis population;
• effect estimate;
• 95% confidence interval;
• p-value where relevant;
• adjustment variables;
• direction of contrast;
• and missing-data method.

Table 6. Simulated primary result

Note: The effect is telemonitoring minus usual care from baseline- and clinic-adjusted linear regression using the prespecified multiple-imputation analysis of all 240 randomised participants. SD = standard deviation.

The observed group means and the model estimate have different denominators and purposes. This distinction should be stated.

Other questions from the same dataset

Binary outcome

Question:

Does telemonitoring increase the probability of blood-pressure control at 12 weeks?

Possible primary estimates:

• risk difference;
• risk ratio.

A chi-square test could assess an unadjusted association, but a regression model may be preferable when adjustment or a specific effect measure is required.

Repeated numerical outcome

Question:

How do home blood-pressure trajectories differ between groups over 12 weeks?

Use a longitudinal method such as:

• linear mixed-effects model; or
• GEE.

The analysis should represent within-participant correlation and specify whether the target is a time-specific difference, average effect or treatment-by-time interaction.

Count outcome

Question:

Does telemonitoring reduce unscheduled clinic visits?

Use a count model appropriate to the mean–variance relationship and follow-up time:

• Poisson regression;
• negative-binomial regression when overdispersion is material;
• an offset when participants contribute different exposure times.

Time-to-event outcome

Question:

Does telemonitoring alter time to first cardiovascular readmission?

Possible methods include:

• Kaplan–Meier curves;
• log-rank test;
• Cox regression;
• restricted mean survival-time analysis.

The appropriate choice depends on the estimand and proportional-hazards assumption.

Adherence rating

Question:

Is the ordered adherence distribution better with telemonitoring?

An ordinal regression model may preserve the category order. Dichotomising the rating into adherent and non-adherent loses information unless a clinically justified threshold is required.

Situations in which a simple chart is insufficient

Clustered studies

If clinics rather than participants are randomised, participant outcomes within a clinic are correlated. A cluster-aware analysis is required.

Complex surveys

Probability surveys may require:

• weights;
• strata;
• clusters;
• finite-population information;
• and design-based variance estimation.

An ordinary chi-square or regression model may give incorrect population inference if the survey design is ignored.

Several outcomes or models

The investigator should not try many tests and report whichever is significant. Primary outcomes, contrasts and analysis methods should be prespecified.

Small or sparse datasets

Sparse binary tables, rare outcomes or small numbers of clusters may require:

• exact methods;
• penalised regression;
• small-sample corrections;
• simplified models;
• or acknowledgement that reliable inference is not possible.

Software convergence does not guarantee a trustworthy estimate.

A model-first perspective MMed extension

The phrase “choosing a statistical test” can encourage unnecessary separation between estimation and testing.

A model-first approach asks:

1. What is the outcome distribution?
2. What parameter is of interest?
3. How are observations related?
4. Which predictors and design variables are required?
5. How should uncertainty be estimated?
6. Which contrast answers the question?

The hypothesis test then becomes one component of the fitted model rather than the entire analysis.

Regression modelling also allows consistent extension from unadjusted to adjusted analysis, provided the model is prespecified and appropriate [6].

Suggested reporting

The SAMPL guidelines recommend naming statistical methods, identifying variables and reporting estimates with uncertainty rather than relying on vague descriptions such as “appropriate tests were used” [7].

Common mistakes

Knowledge check

A trial records systolic blood pressure before and after treatment in two independent groups. Should the researcher perform one paired t-test in each group?

Answer and reasoning

No. Separate within-group tests do not test whether the treatment effects differ between groups. The primary analysis should compare groups directly, commonly using baseline-adjusted linear regression or a model of the between-group difference in change.

A binary outcome is compared between two independent groups. Is logistic regression always the only appropriate method?

Answer and reasoning

No. The method depends on the target effect. Logistic regression estimates an odds ratio naturally, while binomial or modified Poisson models can target risk differences or risk ratios. A contingency-table method may be adequate for an unadjusted analysis.

Why should monthly measurements from one participant not be analysed as independent observations?

Answer and reasoning

Measurements from the same participant are correlated. Ignoring that dependence generally produces incorrect standard errors and may give excessive weight to participants with more measurements.

A researcher wants to know whether two blood-pressure devices can be used interchangeably. Is Pearson correlation sufficient?

Answer and reasoning

No. Correlation assesses linear association and can be high even when one device gives systematically different values. Agreement should be assessed using method-comparison approaches such as Bland–Altman analysis with clinically acceptable limits.

Why is the Mann–Whitney test not automatically a test of medians?

Answer and reasoning

It is a rank-based test of a distributional contrast. Interpreting it as a median comparison requires additional assumptions about distribution shape and the nature of the group difference.

Key take-home messages

References

1. Ranganathan P. An introduction to statistics: choosing the correct statistical test. Indian J Crit Care Med. 2021;25(Suppl 2):S184–S186. https://doi.org/10.5005/jp-journals-10071-23815
2. Altman DG, Bland JM. Parametric v non-parametric methods for data analysis. BMJ. 2009;338:a3167. https://doi.org/10.1136/bmj.a3167
3. Vickers AJ. Parametric versus non-parametric statistics in the analysis of randomized trials with non-normally distributed data. BMC Med Res Methodol. 2005;5:35. https://doi.org/10.1186/1471-2288-5-35
4. Schober P, Boer C, Schwarte LA. Correlation coefficients: appropriate use and interpretation. Anesth Analg. 2018;126(5):1763–1768. https://doi.org/10.1213/ANE.0000000000002864
5. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
6. Harrell FE Jr. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. Cham: Springer; 2015. https://doi.org/10.1007/978-3-319-19425-7
7. Lang TA, Altman DG. Basic statistical reporting for articles published in biomedical journals: the SAMPL guidelines. Int J Nurs Stud. 2015;52(1):5–9. https://doi.org/10.1016/j.ijnurstu.2014.09.006