Common Statistical Errors in Medical Papers

Overview

Many statistical errors in medical papers are not advanced mathematical failures. They arise from mismatches between:

• question and method;
• design and analysis;
• estimate and interpretation;
• software output and manuscript wording.

Common errors can be grouped into:

1. descriptive reporting;
2. hypothesis testing;
3. effect measures;
4. dependence and design;
5. regression modelling;
6. missing data;
7. multiplicity and selective reporting;
8. reproducibility.

These errors can exaggerate evidence, conceal uncertainty or make findings impossible to verify.

Why this matters

A small p-value can coexist with:

• trivial effect size;
• biased sampling;
• inappropriate independence assumptions;
• selective outcome reporting;
• model overfitting;
• poor measurement.

A large sample can make a wrong analysis highly precise.

Critical appraisal should therefore ask whether the analysis estimates the intended population quantity under credible assumptions.

When to use this framework

Use this error framework when:

• reviewing a manuscript, thesis or report;
• checking a statistical-analysis plan against completed analyses;
• preparing a point-by-point response to reviewers;
• verifying tables generated from statistical software;
• teaching researchers how to distinguish reporting problems from validity threats.

Prioritise errors that could change the estimand, effect estimate, uncertainty or main conclusion. Minor wording and formatting issues should be addressed after validity concerns.

Error 1: reporting only p-values

A p-value does not show effect magnitude or precision [1].

Correction:

• report the effect estimate;
• report a confidence interval;
• explain the outcome scale;
• interpret clinical importance.

Error 2: misinterpreting the p-value

Incorrect:

There is a 3% probability that the null hypothesis is true.

Correct:

Under the null model and assumptions, the probability of a result at least as incompatible with the null as observed is 0.03.

Even the corrected definition does not quantify bias or model plausibility [2].

Error 3: writing p=0.000

Software rounding does not mean the probability is exactly zero.

Write:

$p<0.001$.

Error 4: equating non-significance with no effect

Incorrect:

There was no association because $p=0.18$.

A wide confidence interval may remain compatible with substantial benefit and harm.

Absence of evidence is not evidence of absence [3].

Error 5: statistical significance equals clinical importance

A 0.4 mmHg difference can become statistically significant in a huge sample but remain clinically negligible.

Clinical interpretation requires:

• effect magnitude;
• interval;
• harms;
• costs;
• baseline risk;
• patient priorities.

Error 6: testing within groups instead of between groups

Incorrect conclusion:

The intervention group improved significantly, but control did not; therefore, the intervention worked.

The correct test compares changes or outcomes directly between groups.

Different p-values do not prove different effects.

Error 7: using the wrong independence assumption

Examples include treating:

• repeated visits;
• both eyes;
• several lesions;
• participants in the same clinic;
• matched pairs

as independent observations.

This commonly underestimates standard errors.

Error 8: choosing parametric versus non-parametric methods from a normality test alone

Formal normality tests are sensitive to sample size and do not identify the estimand.

Choose the method from:

• research question;
• outcome;
• dependence;
• effect measure;
• distribution and influence;
• model robustness.

Error 9: interpreting an odds ratio as a risk ratio

For common outcomes, the odds ratio can exaggerate the relative probability contrast.

The manuscript must use the correct terminology and provide absolute risks where possible.

Error 10: arbitrary categorisation

Dividing age, BMI or laboratory values into categories:

• loses information;
• reduces power;
• leaves residual confounding;
• imposes artificial cut-offs;
• encourages cut-point searching [4].

Use continuous modelling with appropriate functional form unless a clinically established threshold defines the estimand.

Error 11: univariable screening

Selecting model variables because univariable $p<0.20$ can:

• omit confounders;
• retain chance associations;
• bias coefficients;
• invalidate ordinary uncertainty;
• produce unstable models [5].

Variable selection should follow model purpose, subject knowledge and appropriate validation.

Error 12: stepwise regression interpreted as confirmatory

Stepwise procedures search many models. The final coefficient table does not account fully for that search.

Reported p-values and confidence intervals are usually too optimistic.

Error 13: overfitting

A model with too many candidate parameters can fit noise and yield optimistic, unstable findings [6].

Warning signs include:

• very large coefficients;
• unstable signs;
• implausibly high apparent performance;
• sparse events;
• poor validation.

Error 14: ignoring non-linearity

A continuous coefficient assumes a specific functional form.

Categorising the variable is not a satisfactory diagnostic.

Use plots, flexible terms and meaningful contrasts.

Error 15: inappropriate adjustment

Adjustment can create bias when controlling for:

• mediators;
• colliders;
• post-treatment variables.

“Adjusted for all available variables” is not a strength.

Error 16: missing data handled by silent deletion

Complete-record analysis changes the analysis sample and may bias results.

Report:

• missingness by variable and group;
• number included;
• assumptions;
• primary handling;
• sensitivity analyses.

Error 17: multiple testing without context

Testing many outcomes, subgroups or cut-offs increases false-positive findings.

Prespecify primary analyses and label exploratory findings.

Error 18: subgroup conclusions based on separate significance

A direct interaction test is required.

Subgroups also need adequate information and credible prior rationale.

Error 19: post hoc power

Observed post hoc power is largely determined by the p-value and does not clarify a completed study.

Use the effect estimate and confidence interval.

Error 20: baseline significance testing in randomised trials

Chance differences are expected after randomisation. Baseline tables should describe, not test, the groups.

Error 21: regression table without coding

A coefficient cannot be interpreted without:

• unit;
• reference category;
• transformation;
• outcome scale;
• adjustment set.

Error 22: causal language from observational association

“Predicts”, “increases”, “leads to” and “protects against” can imply causality.

Use association language unless design and assumptions support a causal estimand.

Error 23: software warnings ignored

Convergence, separation, singularity and rank-deficiency warnings require investigation.

A coefficient table is not trustworthy merely because it printed.

Error 24: selective reporting

Reporting only favourable models, subgroups or outcomes creates a distorted record.

Registration, protocols and analysis plans help readers identify deviations.

Error 25: non-reproducible analysis

Manual spreadsheet editing, undocumented recoding and copied table values create hidden opportunities for error.

Use scripts, version control and automated outputs.

Error-to-correction map

simulated data

Error	Misleading wording	Better approach
P-value only	“Significant difference, $p=0.03$”	Report effect and 95% CI
Non-significance	“No effect”	Describe compatible effect range
Odds ratio	“Risk was 2.1 times higher”	State odds ratio; provide risks
Separate group tests	“Only intervention changed”	Test the between-group contrast
Univariable screening	“Variables with $p<0.20$ entered”	Prespecify by model purpose
Missing data	“Incomplete cases were excluded”	Report amount, assumptions and sensitivity

Worked example

A statistical error audit

Question-method alignment:
Independent analytical unit:
Outcome and effect measure:
Denominators and missing data:
Confidence intervals:
Reference categories:
Functional forms:
Interactions:
Multiplicity:
Model diagnostics:
Sensitivity analyses:
Causal language:
Reproducible code and outputs:

Interpreting manuscript errors

Presenting corrected analyses

Suggested reporting

The ASA statement emphasises that scientific conclusions should not depend only on whether a p-value crosses a threshold [1]. Statistical review should prioritise design, effect estimation, uncertainty and transparency.

Common mistakes

Knowledge check

Why are separate within-group tests misleading?

Answer and reasoning

They do not test whether the group effects differ. A direct between-group contrast is required.

What should replace observed post hoc power?

Answer and reasoning

The effect estimate and confidence interval.

Why is univariable screening problematic?

Answer and reasoning

It uses outcome-driven significance to define the model and can omit confounders and produce unstable estimates.

Does adjustment for more variables always reduce bias?

Answer and reasoning

No. Adjusting for mediators, colliders or poorly measured variables can increase bias.

Which errors deserve highest priority in peer review?

Answer and reasoning

Those that threaten the estimand, independence, effect measure, uncertainty, selection or interpretation and could change conclusions.

Key take-home messages

References

1. Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. https://doi.org/10.1080/00031305.2016.1154108
2. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. https://doi.org/10.1007/s10654-016-0149-3
3. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. https://doi.org/10.1136/bmj.311.7003.485
4. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25(1):127–141. https://doi.org/10.1002/sim.2331
5. Heinze G, Wallisch C, Dunkler D. Variable selection—a review and recommendations for the practicing statistician. Biom J. 2018;60(3):431–449. https://doi.org/10.1002/bimj.201700067
6. Babyak MA. What you see may not be what you get: overfitting in regression-type models. Psychosom Med. 2004;66(3):411–421. https://doi.org/10.1097/01.psy.0000127692.23278.a9