Prediction Model Validation and Updating

Overview

A prediction model is not established by performing well in the data that created it. External validation evaluates a locked model in participants whose outcomes were not used to estimate, tune or select that model. The goal is to determine whether predicted risks remain accurate and clinically informative in new data from the intended population or setting.[1,2]

Validation is followed, when necessary, by updating. Updating can range from correcting the overall prediction level to recalibrating predictor effects or revising parts of the model. The degree of modification should be proportional to the observed problem and supported by sufficient new data.[3,4]

This lecture treats validation as a new study with its own target population, sample size, missing-data strategy and reporting—not as a final checkbox attached to development.

Why this matters

Prediction performance commonly deteriorates when a model is transported. Baseline outcome incidence may differ, predictor distributions may change, laboratory assays may be calibrated differently, coding practices may shift and clinical pathways may select different patients. Even a statistically stable model can become miscalibrated when these conditions change.[2,5]

A frequent response is to rebuild a new model immediately. This wastes prior information and can create a smaller, more unstable model. When the original model remains useful, targeted updating may be more efficient.[3]

Key concepts

Apparent performance

Performance evaluated in the same data used for final model fitting. It is optimistic and should not be described as validation.

Internal validation

Resampling or data-splitting methods estimate reproducibility within the development source. Bootstrap and cross-validation can quantify optimism if they repeat the full modelling process.

Internal-external validation

When data come from several clusters such as hospitals or countries, the model can be repeatedly developed on some clusters and evaluated in others. This explores heterogeneity and transportability while retaining structured separation.[1]

External validation

The locked model is evaluated in a new dataset not used for development. “External” can refer to temporal, geographic, setting or investigator separation; the key is independent outcome information and relevance to intended use.

Designing an external validation study

DECISION GUIDE

Validation data should recreate intended use

• Recruit from the population in which the model is intended to operate.
• Use the same prediction time point and outcome horizon.
• Measure predictors using compatible definitions and units.
• Reproduce the original model equation, intercept/baseline hazard and transformations exactly.
• Ascertain the outcome without knowledge of predicted risk where feasible.
• Prespecify performance measures, subgroup checks and updating triggers.

A validation study that cannot reconstruct a required predictor should not silently substitute another measure. Substitution changes the model. If a local mapping is unavoidable, it becomes part of an adaptation study and should be evaluated transparently.

Sample size for validation

Validation sample size is driven by precision of performance estimates rather than a generic events-per-variable rule. Adequate numbers of participants and outcome events are needed to estimate calibration, discrimination and overall performance with useful confidence intervals.[1,2]

Small validation samples can produce noisy calibration slopes and misleading subgroup comparisons. A model should not be rejected or extensively revised solely because one small dataset yields an unstable estimate.

Applying the locked model

PROTOCOL CHECKLIST

Before calculating validation performance

• Obtain the complete published model specification.
• Confirm units, coding, transformations and interactions for every predictor.
• Reproduce the original intercept or baseline survival component.
• Calculate predictions without refitting coefficients.
• Document missing predictor and outcome data.
• Check whether the validation cohort meets original eligibility and prediction-time definitions.
• Record deviations from the original measurement process.

For machine-learning models, validation also requires the exact model object or executable implementation, preprocessing pipeline, feature order, software dependencies and decision rules. A textual description of architecture may not be sufficient to reproduce predictions.

What performance should be examined?

Calibration-in-the-large

Compares the average predicted risk with the observed outcome frequency. Systematic overprediction or underprediction can arise when baseline risk differs between settings.

Calibration slope

A slope below one often indicates predictions are too extreme in the validation sample; a slope above one indicates they may not be extreme enough. The slope should be interpreted with its uncertainty and case mix.[5]

Calibration plot

A flexible plot across the risk range shows where miscalibration occurs. A single calibration test should not replace graphical assessment.

Discrimination

Measures such as the c-statistic assess ranking. Discrimination can change simply because the validation population has a narrower or broader case mix even when predictor effects transport reasonably well.[2]

Overall performance

The Brier score or related measures summarise prediction error, while decision-analytic measures are needed when clinical usefulness at thresholds is the question.

Illustrative external-validation results framework

illustrative

Table 1. Questions to answer before updating

Performance domain	Diagnostic question
Average risk	Is the model systematically too high or too low?
Calibration slope	Are predictions too extreme or too moderate?
Calibration curve	Where in the risk range does miscalibration occur?
Discrimination	Does the model rank higher-risk participants appropriately?
Overall error	How accurate are probabilities overall?
Clinical thresholds	Would decisions using the model have potential net benefit?

Why performance changes

Performance differences do not necessarily mean the original model was badly developed. Consider:

• case mix: predictor distributions and severity differ;
• baseline risk: outcome incidence differs because of population or healthcare changes;
• measurement: assays, devices or coding definitions differ;
• treatment paradox: predictor values trigger treatment that then modifies outcome;
• calendar time: disease patterns and management evolve;
• selection: referral or attendance pathways differ; and
• implementation mismatch: the validation cohort is not the population for which use is claimed.

Updating strategies

Updating should begin with the simplest modification capable of addressing the observed problem.[3,4]

Recalibration-in-the-large

Adjust the intercept for a binary model or baseline hazard/survival component for a time-to-event model when average risk differs but relative predictor effects appear transportable.

Logistic recalibration

Adjust the overall prediction level and calibration slope. This can correct systematic over- or under-extremity while retaining relative predictor structure.

Model revision

Re-estimate selected coefficients, transformations or predictor effects when evidence indicates specific relationships differ.

Model extension

Add new predictors only when they are available in practice, improve relevant performance and have sufficient supporting data.

Redevelopment

Develop a new model when transportability is poor, the intended population or outcome has fundamentally changed, or the original model cannot be reconstructed. Redevelopment should not be the automatic first response.

Updating creates a new model versionMMed extension

Once recalibration or revision is performed, the updated model itself requires validation. Version control matters: implementation should identify which equation, baseline hazard, coefficients, preprocessing steps and thresholds are active. Continual silent updating can make clinical performance unauditable.

Risk of bias and reporting

PROBAST+AI applies to studies that develop, validate or update prediction models and emphasises risk-of-bias and applicability concerns across participants, predictors, outcome and analysis.[6] TRIPOD+AI provides current reporting guidance for both regression and machine-learning prediction models.[7]

APPRAISAL CHECKLIST

Appraising external validation

• Was the original model locked and fully reproduced before evaluation?
• Does the cohort represent the intended target setting and prediction time point?
• Were predictors and outcomes measured compatibly with development?
• Was validation sample size justified for precise performance estimation?
• Are calibration and discrimination reported with uncertainty?
• Were model modifications made only after original performance was shown?
• If updated, is the new model specification reported and separately validated or flagged for validation?

Ethical and governance considerations

Interface with statistical analysis

The linked Biostatistics prediction-model material covers calibration, discrimination and related measures. Validation analysis should provide confidence intervals and preserve the model’s original outcome scale and horizon. For survival models, censoring and baseline survival must be handled consistently. For clustered multi-centre data, internal-external validation can quantify heterogeneity across sites.

When updating, separate data used to estimate the update from data used to assess the updated model whenever possible, or use appropriate resampling and future validation.

Validation should be prospective in its logic even when data are retrospective

A validation dataset may already exist, but the analysis should be planned as though the model were being deployed. Before examining outcomes, define eligibility, predictor extraction, time zero, outcome horizon, missing-data handling and performance measures. If researchers explore the data repeatedly and choose the subset where the model performs best, the validation becomes another development exercise.

Temporal validation

Temporal validation uses patients from a later period in the same setting. It tests robustness to calendar change but may share staff, measurement systems and referral mechanisms with development. It is stronger than apparent performance but not equivalent to geographic or setting validation.

Geographic and setting validation

Validation in another hospital, region or healthcare system can test broader transportability. Differences in prevalence and case mix are expected. The aim is not to find a population identical to development but to test the model in a population for which use is genuinely contemplated.

Internal-external validation in multi-centre data

When several centres contribute data, leave-one-centre-out or related internal-external approaches can evaluate heterogeneity across clusters. This can show whether the model repeatedly transports to held-out centres and identify settings with systematic miscalibration before a final model is selected.

Reconstructing the model faithfully

Published prediction models are sometimes impossible to validate because coefficients, intercepts, baseline hazards or transformations are missing. Validators should not guess these quantities. Contact authors when feasible and document any assumptions. A model that cannot produce the same risk for a published example is not being faithfully reproduced.

For machine-learning models, faithful reconstruction can be even harder. The preprocessing pipeline, tokenizer, feature extractor, model weights, software versions and thresholds may all matter. Independent validation should freeze these components before outcomes are analysed.

Interpreting calibration patterns

Different calibration failures suggest different mechanisms:

• Calibration-in-the-large error with slope near one: baseline risk differs while relative effects may transport.
• Slope below one: predictions are too extreme; development overfitting or weaker predictor effects may be present.
• Slope above one: predictions may be insufficiently extreme or predictor effects stronger in the new setting.
• Nonlinear calibration curve: miscalibration varies across the risk range and may not be fixed by one intercept or slope.

These patterns guide updating but do not prove their cause. Case mix, measurement and treatment differences should be investigated.

Updating hierarchy and information requirements

The more extensively a model is updated, the more new information is consumed. Recalibrating an intercept requires less information than re-estimating every coefficient or fitting new interactions. An update dataset too small for the chosen complexity will overfit just as a development dataset can.

A useful strategy is to predefine a hierarchy: no update if performance is acceptable; intercept/baseline-risk correction for systematic level differences; intercept plus slope recalibration when predictions are too extreme; targeted coefficient revision when specific predictor effects differ; extension only for well-justified new predictors; and full redevelopment only when the original structure is unsuitable.

PROTOCOL CHECKLIST

Updating plan to prespecify before seeing results

• Define acceptable ranges or decision-relevant criteria for calibration and discrimination.
• Preserve performance of the unmodified model as the primary validation result.
• Specify the order of candidate updating strategies.
• Match update complexity to the amount of new outcome information.
• Use resampling or separate data to estimate performance of the updated model.
• Assign a new model version and document changed parameters.
• Plan future validation and monitoring of the updated version.

Transportability versus reproducibility

A model may be reproducible—another analyst obtains the same predictions—yet not transportable—the predictions are poorly calibrated in a new setting. These are separate qualities. Reproducibility is necessary to conduct validation; transportability is what the validation study estimates.

The decision to adopt a model should therefore consider both. A black-box tool whose exact version cannot be reconstructed creates governance problems even if a vendor reports high external performance.

Subgroup validation without overinterpretation

Performance should be examined in groups where clinically important differences are plausible, but subgroup validation requires caution. Small strata produce imprecise calibration and discrimination estimates, and repeatedly searching many groups can generate chance findings. Prespecify key groups based on intended use, measurement differences or equity concerns and report uncertainty.

A subgroup performance gap may arise from different baseline risk, predictor measurement, case mix or treatment, not necessarily from a single “biased coefficient”. Investigate mechanisms before modifying the model separately for each group. Group-specific updating can itself overfit and can create operational complexity.

Prospective monitoring after validation

External validation is a snapshot. If a model is adopted, ongoing calibration monitoring is needed because prevalence, treatments and coding can change. Define what data will be collected, how often performance will be reviewed and what degree of deterioration triggers investigation. Monitoring should distinguish random fluctuation from sustained drift and should avoid automatic retraining without governance.

For high-stakes models, shadow-mode deployment can be useful: predictions are generated prospectively without influencing care while calibration, data availability, latency and technical reliability are evaluated. This tests implementation conditions without yet exposing patients to model-guided decisions. It remains observational performance evaluation, not proof of clinical utility.

Validation under data shift

A model may encounter covariate shift, where predictor distributions change; prior probability shift, where baseline outcome risk changes; or broader concept shift, where relationships between predictors and outcome change. These labels can organise investigation, but real healthcare changes often combine several mechanisms. Validation should compare predictor distributions and outcome incidence with development data before attributing poor performance to the algorithm alone.

Measurement shift deserves special attention. A laboratory assay may change calibration, an imaging device may change resolution, or coding rules may be revised. If the model was developed on measurements that are no longer comparable, apparent transport failure may be a data-definition problem. Harmonisation can be considered, but it should be performed without using outcome information where possible and documented as part of adaptation.

Validation at the intended decision threshold

Global calibration and discrimination can look acceptable while performance near the treatment threshold is poor. When a model will trigger action around a narrow risk range, examine calibration and classification consequences in that region with appropriate uncertainty. This does not replace overall evaluation; it connects validation to the decision the model is intended to support.

Report uncertainty around updating decisions

An estimated calibration slope or intercept is itself uncertain. Updating rules should not react to small random deviations as though they were certain drift. Confidence intervals, calibration plots and replication over time help distinguish a genuine transportability problem from sampling fluctuation.

When several external datasets are available, report heterogeneity rather than averaging immediately. Consistent performance across distinct settings provides stronger transportability evidence than one pooled estimate dominated by the largest site. Transportability requires repeated evidence.

Worked example

WORKED EXAMPLE

Validating before recalibrating the postpartum model

The second health system obtains the original equation, including all transformations and the development intercept. It calculates five-year diabetes risks in a new cohort without changing any coefficients.

Discrimination remains similar, but the model systematically underpredicts observed risk. The calibration slope is close to one, suggesting that relative predictor effects are reasonably preserved while baseline risk differs. Rather than rebuilding the model, the team estimates a local recalibration-in-the-large.

They report performance of the original model first, then the recalibrated version as an update. The updated intercept and version are documented, and the team plans continued prospective monitoring. This preserves evidence about transportability and uses the simplest correction consistent with the observed problem.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

What is the defining feature of external validation?

Answer and reasoning

The model is applied in genuinely new outcome data not used to estimate or tune it, with the model specification locked before performance is calculated.

A model has similar discrimination but systematically underpredicts risk in a new setting. What should be considered before redevelopment?

Answer and reasoning

Calibration should be examined in detail. If predictor effects transport reasonably but baseline outcome risk differs, recalibration-in-the-large may be an efficient and defensible first update.

Why can discrimination change between settings even when coefficients remain useful?

Answer and reasoning

Discrimination depends partly on case mix. A population with a narrower range of predictor values may be harder to rank, while a more heterogeneous population can produce higher discrimination.

Why must updated models be versioned?

Answer and reasoning

Updating changes the prediction function. Clinical users, auditors and future validators need to know exactly which coefficients, preprocessing and baseline-risk components generated each prediction.

Does external validation demonstrate clinical benefit?

Answer and reasoning

No. It demonstrates predictive performance in new data. Whether using the model improves decisions or outcomes is a clinical utility and impact question addressed in RM-P10-M09.

Key take-home messages

SUMMARY

Key take-home messages

• Apply a locked model to new data before modifying it.
• Assess calibration, discrimination and overall performance with adequate precision.
• Explain performance changes using population, baseline-risk, measurement and pathway differences.
• Update proportionately, starting with simple recalibration when appropriate.
• Treat every updated prediction function as a new version requiring evaluation and governance.

References

1. Collins GS, Dhiman P, Ma J, et al. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ. 2024;384:e074819. https://doi.org/10.1136/bmj-2023-074819
2. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230. https://doi.org/10.1186/s12916-019-1466-7
3. Janssen KJM, Moons KGM, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol. 2008;61(1):76–86. https://doi.org/10.1016/j.jclinepi.2007.04.018
4. Binuya MAE, Engelhardt EG, Schats W, et al. Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review. BMC Med Res Methodol. 2022;22:316. https://doi.org/10.1186/s12874-022-01801-8
5. Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users’ guides to the medical literature. JAMA. 2017;318(14):1377–1384. https://doi.org/10.1001/jama.2017.12126
6. Moons KGM, Damen JAA, Kaul T, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. https://doi.org/10.1136/bmj-2024-082505
7. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://doi.org/10.1136/bmj-2023-078378