Outliers and Influential Observations

Overview

An outlier is an observation that appears unusual relative to other observations under a stated definition or model. It may be a data error, a valid extreme clinical value, a member of a different population or evidence that the chosen model does not describe the data well.

An influential observation is one whose inclusion materially affects a statistical estimate, fitted value, standard error or conclusion. An observation can be unusual without being influential, and influential without having the most extreme outcome.

The correct response is investigation, not automatic deletion. A valid participant with severe disease remains part of the population simply because their value is inconvenient for a model. Conversely, a plausible-looking value assigned to the wrong participant may require correction even if no outlier rule detects it.

Why this matters

Removing unusual observations can change:

• the estimated treatment effect;
• the width of the confidence interval;
• the apparent distribution;
• the fitted regression line;
• whether model assumptions appear reasonable; and
• the clinical population represented by the analysis.

If deletion depends on which version gives the preferred p-value, the reported inference is no longer trustworthy.

Outlier assessment should therefore separate four questions:

1. Is the value recorded correctly?
2. Does the observation belong to the study population and analysis?
3. Is it unusual under a descriptive or model-based definition?
4. Does it materially influence the target estimate?

Different meanings of “outlier”

Data error

A data error is an incorrect representation of the intended observation.

Examples include:

• a misplaced decimal point;
• pounds entered as kilograms;
• systolic and diastolic values transposed;
• an incorrect participant identifier;
• a duplicate import; and
• a date interpreted using the wrong format.

Errors should be corrected using verified source information, not statistical convenience.

Valid extreme observation

A valid extreme observation is correctly recorded and belongs to the target population but lies far from most values.

A systolic blood pressure of 238 mmHg may be rare, but it is medically possible. Such participants may be especially important to clinical interpretation.

Observation from another population

A record may be valid in itself but incompatible with the study population.

For example, a paediatric record accidentally merged into an adult study is not removed because it is statistically extreme. It is excluded because it does not meet the prespecified population definition.

Unusual model residual

A residual is the observed outcome minus the outcome predicted by a fitted model.

A participant may have a common baseline value but an outcome very different from the model prediction. This produces a large residual.

High leverage

Leverage describes how unusual an observation’s explanatory-variable pattern is relative to the data used to fit a regression model.

HTM156 has high leverage because baseline systolic blood pressure of 238 mmHg is far from most baseline values. High leverage does not automatically mean the observation is influential; its outcome may still align with the fitted relationship.

Influence

Influence concerns how much the fitted analysis changes when an observation is perturbed or removed.

Cook’s distance combines information about residual size and leverage to assess influence in linear regression [1].

Start with clinical and data verification

Before calculating a diagnostic statistic, review:

• eligibility;
• units;
• measurement method;
• source record;
• timing;
• participant linkage;
• duplicate status;
• protocol deviations;
• related measurements; and
• whether the value is possible in the clinical context.

The data-cleaning process should record the investigation and resolution [2].

A value should not be changed to the nearest permitted limit. Winsorisation replaces extreme values with less extreme boundary values. It changes observed data and should not be used routinely without a clear estimand, prespecified rule and justification.

Descriptive detection methods

Graphical assessment

Useful displays include:

• dot plots;
• histograms;
• box plots;
• scatter plots;
• plots against time; and
• plots by site or treatment group.

Graphs reveal context that one numerical rule may miss. A high laboratory value may be isolated globally but typical within a subgroup.

Interquartile-range rule

A common exploratory rule flags values below:

$$Q_1 - 1.5(IQR)$$

or above:

$$Q_3 + 1.5(IQR)$$

where $Q_1$ is the first quartile, $Q_3$ is the third quartile and $IQR = Q_3 - Q_1$.

The rule was designed for box-plot display. It does not define invalidity and may flag many legitimate observations in skewed medical data.

Standard-deviation rules

Rules such as “more than three standard deviations from the mean” depend on the mean and standard deviation, both of which are affected by extreme values. They also rely on distributional expectations that may not suit skewed variables.

Median absolute deviation

The median absolute deviation, abbreviated MAD, is a robust measure of spread based on distances from the median.

Modified-score rules based on the MAD are less affected by extreme values than mean-and-standard-deviation rules. They remain screening tools, not automatic deletion criteria [3], [4].

If many observations are identical, the MAD may be zero. No one rule is universally suitable.

Model-based diagnostics

Studentised residuals

A studentised residual scales a residual by an estimate of its standard deviation. Large absolute values indicate outcomes poorly predicted by the fitted model.

Thresholds such as 2 or 3 are heuristics. With large datasets, some large residuals are expected by chance.

Leverage values

In linear regression, leverage values come from the diagonal of the model’s hat matrix. Average leverage is related to the number of model parameters divided by the sample size.

Heuristics based on multiples of average leverage can identify observations for review. Their meaning depends on the model and predictor distribution.

Cook’s distance

Cook’s distance measures the combined effect of an observation’s residual and leverage on the fitted regression [1].

Large values relative to the rest of the dataset deserve investigation. Fixed cut-offs such as 1 or $4/n$ should not be treated as universal laws.

DFBETAs and DFFITS

• DFBETA assesses how much one regression coefficient changes when an observation is omitted.
• DFFITS assesses how much the fitted value for an observation changes when it is omitted.

These diagnostics can show whether an observation mainly influences the treatment coefficient, a baseline adjustment or predictions. Their interpretation should be model-specific and considered together rather than reduced to one automatic cut-off [5].

Outliers in different analyses

Outlier assessment depends on the method and on the clinical quantity being estimated [6].

Numerical summaries

The mean and standard deviation can change substantially with extreme values. The median and interquartile range are more resistant but answer different descriptive questions.

Group comparisons

An extreme value may alter a mean difference and Welch’s t-test. The correct response is not automatically to switch to a rank test: the estimand changes, and rank methods do not simply estimate a median difference in every setting.

Logistic regression

A binary outcome cannot be numerically extreme because it has only two values. Influence may instead arise from an unusual predictor pattern, sparse combination or near-perfect prediction.

Survival analysis

Influence can arise from unusual covariate patterns, early events or long follow-up. Diagnostics must match the survival model and censoring structure.

Repeated and clustered data

Deleting one row may leave other observations from the same participant. Influence may need to be assessed at participant or cluster level rather than row level.

Data structure

Simulated diagnostic summary

simulated data

Table 1. Selected observations from the primary regression

Participant	Group	Baseline SBP	Week-12 SBP	Studentised residual	Leverage	Cook’s distance
HTM041	Usual care	148 mmHg	171 mmHg	2.8	0.018	0.07
HTM092	Telemonitoring	151 mmHg	132 mmHg	−0.4	0.017	0.00
HTM156	Telemonitoring	238 mmHg	176 mmHg	2.2	0.214	0.64
HTM203	Usual care	145 mmHg	119 mmHg	−2.5	0.016	0.05

Note: Values are simulated. No diagnostic statistic independently determines exclusion.

(Calculations for the studentised residuals, leverage values and Cook’s distances are not shown here.)

HTM041 has a large residual but ordinary leverage. HTM156 has both a large residual and high leverage, giving the largest Cook’s distance.

Worked example

Robust methods do not eliminate the need for investigation MMed extension

Robust methods reduce sensitivity to particular deviations but do not make data problems irrelevant.

Examples include:

• robust location and scale summaries;
• heteroscedasticity-robust standard errors;
• quantile regression;
• robust regression estimators; and
• models designed for heavy-tailed distributions.

A robust method may be preferable when its estimand and assumptions answer the research question. It should not be selected only because it produces a desired p-value.

Comparing a primary analysis with a robust supportive analysis can show whether conclusions depend strongly on extreme observations or variance assumptions.

Decision process

Suggested reporting

Common mistakes

Knowledge check

Is every outlier an influential observation?

Answer and reasoning

No. An observation may be extreme but have little effect on the target estimate. Influence depends on the model, predictor pattern, residual and the specific quantity being estimated.

What is the difference between a large residual and high leverage?

Answer and reasoning

A large residual means the observed outcome differs from the model prediction. High leverage means the explanatory-variable pattern is unusual relative to other observations. A point with both can be especially influential.

Should a verified systolic blood pressure of 238 mmHg be deleted because it is extreme?

Answer and reasoning

No. If it is correctly measured, eligible and relevant to the target population, it should ordinarily be retained. Its influence should be assessed and reported.

What does Cook's distance tell the analyst?

Answer and reasoning

It summarises how strongly one observation affects the fitted linear-regression model through its residual and leverage. It flags observations for investigation but does not establish error or justify deletion.

Why is an analysis excluding an influential participant usually considered a sensitivity analysis?

Answer and reasoning

It assesses how dependent the result is on that observation. If the participant is valid and eligible, exclusion changes the analysed population and should not silently replace the prespecified primary analysis.

Key take-home messages

References

1. Cook RD. Detection of influential observation in linear regression. Technometrics. 1977;19(1):15–18. https://doi.org/10.1080/00401706.1977.10489493
2. Van den Broeck J, Cunningham SA, Eeckels R, Herbst K. Data cleaning: detecting, diagnosing, and editing data abnormalities. PLoS Med. 2005;2(10):e267. https://doi.org/10.1371/journal.pmed.0020267
3. Leys C, Delacre M, Mora YL, Lakens D, Ley C. How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. Int Rev Soc Psychol. 2019;32(1):5. https://doi.org/10.5334/irsp.289
4. Rousseeuw PJ, Croux C. Alternatives to the median absolute deviation. J Am Stat Assoc. 1993;88(424):1273–1283. https://doi.org/10.1080/01621459.1993.10476408
5. Belsley DA, Kuh E, Welsch RE. Regression diagnostics: identifying influential data and sources of collinearity. New York: Wiley; 1980.
6. Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.