Sampling Variability and Standard Error

Overview

Medical researchers usually observe a sample rather than every member of the population. A sample mean, prevalence or treatment effect is therefore unlikely to equal the corresponding population value exactly.

Sampling variability is the random variation in an estimate that arises because different probability samples contain different observations. If the sampling process were repeated many times under the same design, each sample would usually produce a somewhat different estimate.

A standard error, abbreviated SE, quantifies the expected sampling variability of an estimator under a stated sampling design and statistical model. Smaller standard errors indicate more precise estimates, but they do not establish that an estimate is unbiased or clinically important.

Why this matters

Suppose two clinics each sample adults with hypertension from the same district. One sample estimates mean systolic blood pressure as 137 mmHg and another estimates 141 mmHg.

The difference may arise even when:

• both samples were selected correctly;
• both clinics used the same measurement protocol; and
• the underlying population mean did not change.

Statistical inference uses the sampling distribution of an estimator to distinguish expected sample-to-sample fluctuation from patterns that would be unusual under a specified model.

Confusing standard deviation with standard error can produce misleading reports. Standard deviation describes variability among participant values. Standard error describes variability of an estimate across hypothetical repeated samples [1].

Population parameters and sample estimates

A population parameter is a numerical feature of the target population.

Examples include:

• population mean systolic blood pressure, $\mu$;
• population prevalence of uncontrolled blood pressure, $p$; and
• population mean treatment effect, $\Delta$.

A sample estimate, also called a statistic, is calculated from observed data.

Examples include:

• sample mean, $\bar{x}$;
• sample proportion, $\hat{p}$; and
• estimated mean difference, $\hat{\Delta}$.

The parameter is fixed but usually unknown in the frequentist framework. The estimate varies from sample to sample. These distinctions underpin standard medical-statistical inference [2].

The sampling distribution

A sampling distribution is the distribution of a statistic across all hypothetical repeated samples generated under the same design and population model.

It is not the distribution of participant values.

For example:

• the participant distribution may have SD 18 mmHg;
• the sampling distribution of the sample mean may have SE 1.8 mmHg when $n=100$.

The sampling distribution is usually narrower because an average is more stable than an individual observation.

Five hypothetical samples from one population

simulated data

Table 1. Sample means from repeated simple random samples

Hypothetical sample	Sample size	Mean systolic blood pressure
1	100	137.8 mmHg
2	100	140.1 mmHg
3	100	138.9 mmHg
4	100	141.0 mmHg
5	100	139.3 mmHg

Note: These are simulated repeated samples from the same unchanged population.

The means differ because the selected participants differ. A larger collection of repeated means would form the sampling distribution of $\bar{x}$.

Standard error of a mean

For an independent simple random sample, the estimated standard error of the sample mean is:

The formula shows two influences:

• greater participant variability increases the SE;
• larger sample size decreases the SE.

Because the denominator contains $\sqrt{n}$, quadrupling the sample approximately halves the SE. Doubling the sample does not halve it.

Worked example

Standard error of a proportion

For a simple random sample and a binary variable, an approximate estimated SE is:

Suppose 64 of 100 sampled adults have controlled blood pressure:

$$\hat{p}=0.64$$ $$SE(\hat{p}) = \sqrt{ \frac{0.64(0.36)}{100} } = 0.048$$

The estimated SE is 0.048, or 4.8 percentage points.

This large-sample formula may perform poorly with small samples or proportions near 0 or 1. Interval methods are developed in BIO-P04-M02 and BIO-P07-M01.

Standard error of a difference

A treatment comparison is often based on a difference between two estimates.

For independent estimates:

$$SE(\hat{\theta}_1-\hat{\theta}_0) = \sqrt{ SE(\hat{\theta}_1)^2+SE(\hat{\theta}_0)^2 }$$

The covariance term is zero only when the estimates are independent under the design.

For paired, repeated or clustered observations, correlation changes the standard error. Treating related observations as independent can make the SE incorrect.

Standard deviation versus standard error

Table 2. Standard deviation and standard error

Note: BP = blood pressure.

A standard error should not be used to make participant measurements appear less variable in a descriptive table.

What affects standard error?

Sample size

More independent information generally reduces the SE.

However, repeated measurements from one participant or many participants from one cluster are not equivalent to the same number of independent participants.

Population variability

More variable outcomes produce less precise means and group differences for a fixed sample size.

Study design

Precision depends on:

• simple random, stratified or cluster sampling;
• independent or paired observations;
• equal or unequal allocation;
• repeated measurements;
• missing outcomes;
• sampling weights; and
• covariate adjustment.

Statistical model

Different estimators can have different standard errors.

Examples include:

• crude and adjusted mean differences;
• ordinary and heteroscedasticity-robust standard errors;
• model-based and cluster-robust standard errors; and
• complete-record and multiple-imputation analyses.

The SE must correspond to the estimate and model being reported. Large-sample robustness depends on the estimator, design and influence of unusual observations rather than raw normality alone [3].

Bias and standard error are different

An estimate can have:

• small bias and large SE;
• large bias and small SE;
• both; or
• neither.

A very large convenience sample may produce a tiny SE around an estimate that is systematically unrepresentative of the target population.

Sampling formulas commonly condition on the assumed design and model. They do not automatically include:

• nonresponse bias;
• measurement error;
• confounding;
• model misspecification; or
• selective reporting.

Model-based, robust and design-based standard errors MMed extension

A model-based SE relies on the variance assumptions of the fitted model.

A robust SE relaxes selected variance assumptions while retaining other requirements, such as independent clusters and suitable sample size.

A design-based SE is derived from the probability-sampling design and may incorporate strata, clusters and weights.

The choice should match how the data were generated and what population inference is intended.

Suggested reporting

Common mistakes

Knowledge check

Why do sample means differ across repeated random samples?

Answer and reasoning

Each sample contains a different set of participants. Their values differ, so the calculated mean changes even when the underlying population is unchanged.

What is the difference between an SD of 18 mmHg and an SE of 1.8 mmHg?

Answer and reasoning

The SD describes variation among individual blood-pressure values. The SE describes the expected sample-to-sample variation of the estimated mean under the sampling design.

What happens approximately to the SE of a mean if the independent sample size increases from 100 to 400 while the SD remains 18 mmHg?

Answer and reasoning

It falls from $18/\sqrt{100}=1.8$ mmHg to $18/\sqrt{400}=0.9$ mmHg. Quadrupling the sample halves the SE.

Does a very small standard error establish that the sample represents the target population?

Answer and reasoning

No. The SE describes random precision under the assumed design. A large biased sample can have a very small SE and still estimate the wrong population quantity.

Why must a paired analysis use a paired standard error?

Answer and reasoning

Measurements within a pair are correlated. The variance of their difference includes that correlation, so an independent-samples formula does not represent the data structure correctly.

Key take-home messages

References

1. Altman DG, Bland JM. Standard deviations and standard errors. BMJ. 2005;331(7521):903. https://doi.org/10.1136/bmj.331.7521.903
2. Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.
3. Lumley T, Diehr P, Emerson S, Chen L. The importance of the normality assumption in large public health data sets. Annu Rev Public Health. 2002;23:151–169. https://doi.org/10.1146/annurev.publhealth.23.100901.140546