Critical Appraisal of Statistical Methods

Overview

Critical appraisal asks whether a study’s statistical methods support its stated conclusions.

A structured appraisal considers:

1. research question and estimand;
2. design and sampling;
3. analytical unit and dependence;
4. outcome and effect measure;
5. descriptive and inferential methods;
6. model specification and assumptions;
7. missing data;
8. multiplicity and subgroup analyses;
9. robustness and sensitivity;
10. interpretation and reporting.

A well-reported method can still be inappropriate. Conversely, a valid method may be reported too poorly to evaluate. Reporting quality and methodological validity must be distinguished.

Why this matters

Peer review often focuses on whether the named test is familiar:

They used a t-test, so the statistics are acceptable.

This is insufficient.

A t-test can be inappropriate because:

• observations are paired;
• participants are clustered;
• the outcome is not the target mean;
• baseline adjustment was required;
• repeated measurements were ignored;
• the comparison does not answer the objective.

Critical appraisal should begin with the question and design, not the software procedure.

When to use this framework

Use the framework when:

• reviewing a journal manuscript;
• appraising evidence for clinical practice;
• assessing a thesis;
• reviewing a protocol;
• evaluating a statistical-analysis plan;
• deciding whether a study can inform a meta-analysis.

The depth should match the study’s importance and complexity.

Domain 1: Research question and design

Ask:

• Is the objective descriptive, comparative, causal, predictive, diagnostic or prognostic?
• Is the study design capable of addressing it?
• Is the target population defined?
• Are exposure, intervention, comparator, outcome and time point clear?
• Does the conclusion exceed what the design can support?

Examples:

• a cross-sectional study cannot establish temporal incidence;
• an uncontrolled before–after study cannot isolate intervention effects;
• a diagnostic case–control design may exaggerate accuracy through spectrum differences;
• a prediction study needs validation, not only significant coefficients.

Domain 2: Sampling and analytical population

Assess:

• source population;
• inclusion and exclusion;
• sampling method;
• response rate;
• loss to follow-up;
• analysis population;
• representativeness.

A narrow confidence interval does not correct selection bias.

For trials, compare randomised, treated, followed and analysed counts. For observational studies, ask how exclusions alter the target population.

Domain 3: Analytical unit and dependence

Identify the independent unit:

• participant;
• eye;
• lesion;
• tooth;
• clinic;
• household;
• repeated visit;
• matched pair.

Ask whether the analysis accounts for:

• pairing;
• clustering;
• repeated measurements;
• multiple events;
• multiple readers.

Pseudoreplication commonly inflates sample size and understates uncertainty.

Domain 4: Outcome and effect measure

Check:

• outcome definition;
• measurement validity;
• time point;
• censoring;
• competing events;
• clinically meaningful scale.

Confirm that the effect measure matches the design:

• mean difference for numerical outcomes;
• risk difference or ratio for cohort risks;
• odds ratio for case–control sampling;
• rate ratio for person-time;
• hazard ratio or restricted mean survival-time difference for survival;
• sensitivity and specificity for diagnostic accuracy.

Domain 5: Descriptive methods

Ask whether:

• numerical summaries match their clinical meaning;
• categorical denominators are explicit;
• missing values are reported;
• distributions are displayed adequately;
• baseline tables are descriptive rather than tested routinely in randomised trials.

A mean can remain relevant for a skewed cost outcome if total resource burden is the estimand.

Domain 6: Inferential methods

Check alignment between:

• number of groups;
• independent or paired structure;
• outcome type;
• planned effect;
• model family;
• confidence interval;
• sidedness.

Do not accept “non-parametric” as sufficient description. Rank methods also have assumptions and specific estimands.

Domain 7: Regression model

Evaluate:

• model purpose;
• candidate predictors;
• parameter count;
• functional forms;
• reference categories;
• interactions;
• collinearity;
• influence;
• overfitting;
• validation.

Univariable p-value screening and stepwise selection are warning signs [1].

For causal models, assess whether covariates are confounders, mediators or colliders.

For prediction, assess calibration, discrimination and validation.

Domain 8: Missing data

Ask:

• how much is missing;
• in which variables and groups;
• why it may be missing;
• who enters each model;
• whether complete records differ;
• whether imputation is suitable;
• whether sensitivity analyses address departures.

“Participants with missing data were excluded” is not enough.

Domain 9: Multiplicity and subgroup analysis

Identify:

• number of outcomes;
• time points;
• treatment comparisons;
• subgroup analyses;
• model variants;
• cut-offs.

Determine which analyses were prespecified and how type I error or interpretation was controlled.

A significant result in one subgroup and non-significant result in another does not establish interaction.

Domain 10: Assumptions and diagnostics

Assess whether the paper evaluates assumptions relevant to the method.

Examples:

• linearity;
• residual variance;
• proportional hazards;
• proportional odds;
• overdispersion;
• separation;
• influential observations;
• calibration.

A normality test alone is rarely adequate.

Domain 11: Sensitivity and robustness

Strong analyses examine conclusions under plausible alternatives:

• missing-data assumptions;
• alternative outcome definitions;
• influential observations;
• model forms;
• confounder sets;
• competing risks;
• clustering assumptions.

Sensitivity analysis should be planned to investigate uncertainty, not to search for a favourable result.

Domain 12: Interpretation

Ask whether conclusions reflect:

• effect magnitude;
• confidence interval;
• design limitations;
• clinical importance;
• generalisability;
• totality of evidence.

Statistical significance does not remove bias or prove clinical importance [2].

Data example

Critical-appraisal worksheet

simulated data

Domain	Manuscript statement	Appraisal question
Objective	“Predictors of readmission”	Prediction or adjusted association?
Outcome	Any readmission	What time horizon and competing events?
Model	Logistic regression	Was follow-up equal for all participants?
Age	Dichotomised at 60 years	Why discard continuous information?
Missing data	Complete cases	How many excluded and under what assumption?
Effect	Adjusted OR 1.8	What were group risks?
Conclusion	“Low activity caused readmission”	Does the observational design support causality?

Worked example

An appraisal decision path

Reporting quality versus risk of bias

A reporting guideline asks whether required information is disclosed.

Risk-of-bias appraisal asks whether the design, conduct and analysis may distort the result.

Complete reporting makes bias easier to evaluate; it does not guarantee low bias.

For example:

• a fully reported convenience sample remains selected;
• a completely described stepwise model remains unstable;
• a transparent uncontrolled study remains vulnerable to confounding.

Interpreting appraisal findings

Presenting appraisal comments

Suggested reporting

Design-specific reporting guidelines help identify missing information, while SAMPL supports appraisal of statistical reporting [3]. They should be used as aids rather than substitutes for methodological judgement. General evidence-based reading frameworks can supplement this domain-based appraisal [4].

Common mistakes

Knowledge check

What is the first statistical appraisal question?

Answer and reasoning

What population quantity is the study trying to estimate, and can the design address it?

Why is complete reporting not proof of validity?

Answer and reasoning

A biased or inappropriate method can be reported completely. Transparency enables appraisal but does not remove bias.

What is pseudoreplication?

Answer and reasoning

Treating correlated observations from the same independent unit or cluster as separate independent units.

Why is univariable screening a major concern?

Answer and reasoning

It can define the model through chance sample associations, omit confounders and understate uncertainty.

How should appraisal concerns be prioritised?

Answer and reasoning

By their potential to alter the estimand, effect estimate, uncertainty, validity or main conclusion.

Key take-home messages

References

1. Heinze G, Wallisch C, Dunkler D. Variable selection—a review and recommendations for the practicing statistician. Biom J. 2018;60(3):431–449. https://doi.org/10.1002/bimj.201700067
2. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. https://doi.org/10.1080/00031305.2016.1154108
3. Lang TA, Altman DG. Basic statistical reporting for articles published in biomedical journals: the SAMPL guidelines. Int J Nurs Stud. 2015;52(1):5–9. https://doi.org/10.1016/j.ijnurstu.2014.09.006
4. Greenhalgh T. How to read a paper: the basics of evidence-based medicine and healthcare. 6th ed. Hoboken (NJ): Wiley-Blackwell; 2019.