Confidence Intervals

Overview

A point estimate gives one best estimate of a population quantity. A confidence interval, abbreviated CI, adds information about sampling uncertainty.

Examples include:

• mean difference with a 95% CI;
• risk difference with a 95% CI;
• risk ratio with a 95% CI;
• prevalence with a 95% CI; and
• regression coefficient with a 95% CI.

A CI should be interpreted together with effect magnitude, direction, clinical relevance, statistical model, study design and possible bias. Confidence intervals are generally more informative than an isolated p-value because they show which effect sizes remain compatible with the observed data and assumptions [1], [2].

Why this matters

Consider two trial results:

1. mean difference −6 mmHg, 95% CI −10 to −2;
2. mean difference −6 mmHg, 95% CI −25 to 13.

The point estimates are identical, but the second trial is much less precise. Its data are compatible with substantial benefit, little effect or harm.

A conclusion based only on whether $p<0.05$ would hide this difference.

The frequentist meaning

A 95% confidence procedure is designed so that, under repeated sampling and the stated assumptions, approximately 95% of intervals constructed by that procedure contain the fixed population parameter.

After one interval has been calculated, the interval endpoints and parameter are fixed. The interval either contains the parameter or does not.

The standard frequentist interpretation is therefore not:

There is a 95% probability that the true effect lies inside this particular interval.

A useful interpretation is:

The interval contains parameter values reasonably compatible with the observed data and model assumptions, using a procedure with 95% long-run coverage.

Approximate confidence interval formula

For an estimate whose sampling distribution is approximately normal:

Some analyses use a t critical value or another distribution rather than 1.96. The method must match the estimator and sample structure.

Worked example

Null values depend on the effect scale

For differences:

• mean difference null value: 0;
• risk difference null value: 0;
• regression slope null value: 0.

For ratios:

• risk ratio null value: 1;
• odds ratio null value: 1;
• hazard ratio null value: 1.

An interval excluding its null value corresponds to rejection of the matching two-sided hypothesis test at the same significance level when the interval and test use compatible methods.

The CI remains more informative because it shows magnitude and precision.

Confidence interval for a ratio

Ratios are usually analysed on a logarithmic scale because their sampling distributions are asymmetric on the original scale.

For a risk ratio:

1. calculate $\ln(\widehat{RR})$;
2. construct the CI on the log scale;
3. exponentiate the endpoints.

A ratio CI is naturally asymmetric around the point estimate on the original scale. Do not interpret it as a percentage-point difference.

Confidence interval for a proportion

The familiar Wald interval:

$$\hat{p}\pm1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

can have poor coverage, particularly with small samples or proportions near 0 or 1.

Wilson score or exact methods are usually preferable in these settings. Wilson’s method was developed to improve interval behaviour for binomial proportions [3]. The chosen interval method should be stated. Established medical-statistical treatments provide analysis-specific interval procedures beyond the simple normal approximation [4].

What determines width?

A confidence interval becomes narrower when:

• standard error decreases;
• independent information increases;
• outcome variability decreases;
• allocation becomes more efficient;
• adjustment explains relevant variation; or
• a lower confidence level is selected.

A 99% interval is wider than a 95% interval for the same estimate and SE because it aims for higher long-run coverage.

Same estimate with different precision

simulated data

Table 1. Hypothetical confidence intervals for a −5.9 mmHg effect

Study	Estimate	Standard error	95% CI
A	−5.9 mmHg	1.31 mmHg	−8.5 to −3.3
B	−5.9 mmHg	3.00 mmHg	−11.8 to 0.0
C	−5.9 mmHg	5.00 mmHg	−15.7 to 3.9

Note: Values are simulated. Small differences in endpoints can arise from rounding.

The point estimate does not reveal precision by itself.

Clinical thresholds and confidence intervals

Suppose a 5 mmHg reduction is considered clinically worthwhile for planning.

For the interval −8.5 to −3.3:

• it excludes the null value 0;
• it includes effects larger than 5 mmHg;
• it also includes effects smaller than 5 mmHg.

The study supports a benefit but leaves uncertainty about whether the true average benefit reaches the 5 mmHg planning target.

The planning target is not a rule that makes every smaller effect unimportant.

Confidence intervals do not contain every uncertainty

A conventional CI may not include uncertainty from:

• outcome-definition choice;
• missing-data assumptions;
• unmeasured confounding;
• measurement error;
• model selection;
• multiple testing;
• subgroup selection;
• transport to another population; or
• data-quality failures.

Sensitivity analyses can show how conclusions change under alternative assumptions.

Confidence distributions and compatibility intervals MMed extension

Some methodologists use the term compatibility interval to emphasise that the interval shows parameter values compatible with the data under a model, rather than guaranteeing that all values inside are equally plausible [2].

A confidence curve or p-value function displays compatibility across many parameter values rather than only two endpoints.

These tools do not remove dependence on the study design and assumptions.

Suggested reporting

Common mistakes

Knowledge check

What is the repeated-sampling meaning of a 95% confidence interval?

Answer and reasoning

If the same sampling and interval procedure were repeated under the assumptions, approximately 95% of the resulting intervals would contain the fixed population parameter.

How is the 95% interval −8.5 to −3.3 mmHg interpreted?

Answer and reasoning

It indicates that mean effects from about 3.3 to 8.5 mmHg lower with telemonitoring are reasonably compatible with the data and model. It does not assign 95% probability to this particular range.

What is the null value for a risk ratio?

Answer and reasoning

The null value is 1 because risks are equal when their ratio is one. The null value for a risk difference is 0.

Does a confidence interval crossing zero prove that no effect exists?

Answer and reasoning

No. It indicates that the data and method do not exclude zero at the selected confidence level. Important beneficial or harmful effects may also remain compatible if the interval is wide.

Why may a narrow interval still be misleading?

Answer and reasoning

The interval may quantify random sampling uncertainty while ignoring selection bias, measurement error, confounding, model selection or other systematic problems.

Key take-home messages

References

1. Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. BMJ. 1986;292(6522):746–750. https://doi.org/10.1136/bmj.292.6522.746
2. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. https://doi.org/10.1007/s10654-016-0149-3
3. Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22(158):209–212. https://doi.org/10.1080/01621459.1927.10502953
4. Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.