Part 10 · Module 7
Clinical Prediction Model Development
Learn how to develop a clinical prediction model for individual risk with a defined target population, outcome, predictors, sample size, internal validation and performance assessment.
Learning outcomes
After completing this lecture, you should be able to:
- Define the target population, prediction time point, outcome horizon and intended use of a clinical prediction model.
- Distinguish prediction-model development from prognostic-factor research and causal modelling.
- Select and encode candidate predictors based on availability at the intended decision point rather than univariable significance.
- Plan sample size, missing-data handling, model complexity, shrinkage and internal validation to reduce overfitting.
- Evaluate and report model performance using calibration, discrimination and appropriate overall measures in line with current reporting guidance.
Overview
A clinical prediction model combines multiple pieces of information to estimate an individual probability or risk of a current diagnosis or future outcome. Examples include the probability of pulmonary embolism at assessment, 30-day readmission after discharge, or five-year type 2 diabetes after gestational diabetes.[1,2]
Model development is not simply “running multivariable regression”. The design begins with the intended use: who will receive a prediction, at what time point, for which outcome and horizon, using predictors that are genuinely available at that moment. The modelling strategy then has to control overfitting, preserve predictor information, handle missing data, quantify uncertainty and assess predictive performance.[1,3]
Modern guidance such as TRIPOD+AI applies to prediction models developed using regression or machine-learning methods and emphasises transparent reporting of data sources, predictors, outcomes, modelling choices, evaluation and intended use.[1]
Why this matters
A model can appear highly accurate in the data used to build it yet perform poorly in new patients. This optimism arises when the model learns both reproducible signal and idiosyncratic noise. Small datasets, large numbers of candidate parameters, repeated data-driven decisions and evaluation on the same data all increase this risk.[3,4]
Prediction also has different goals from explanation. A variable can improve prediction without being a causal factor, while an important causal factor may add little predictive information once other measurements are available. Selecting predictors only because they have small univariable p-values therefore answers neither goal well.
Key concepts
Prediction time point
The prediction time point or time origin is when the model will be applied. All predictors must be available then. A variable measured six months later cannot be used in a model intended to guide decisions at 12 weeks postpartum unless the model is explicitly dynamic.
Target population
Define who is eligible to receive the prediction. A model developed only in tertiary-care patients may not apply to community follow-up. Selection criteria influence baseline risk, predictor distributions and model performance.
Outcome and horizon
The outcome must be reproducible and clinically meaningful. For prognosis, the horizon matters: a five-year risk model is not automatically a one-year or ten-year model. For diagnostic models, the outcome is current target-condition status rather than future events.
Predictor
A predictor is information used to calculate risk. It can be causal, non-causal or a proxy. It should be defined and measured as it would be in intended practice.
Prediction versus explanation and causation
Clarify the modelling objective
- Prognostic factor research: How is a specific factor associated with future outcome?
- Causal research: What would happen to outcome if an exposure or intervention were changed?
- Prediction development: What is this individual’s probability of the outcome using information available now?
- Prediction impact: Does using the prediction model improve decisions or outcomes?
These questions can use overlapping variables but require different design and analysis logic. Prediction models generally prioritise accurate absolute-risk estimation and transportability, not unbiased estimation of each predictor’s causal coefficient.
Development cohort and data source
Clinical prediction model development study
Core structure: A representative dataset containing the target population, predictors available at the prediction time point and reliably measured outcomes is used to estimate a multivariable prediction function.
Best suited for: Estimating individual probability of a diagnosis or future outcome to support a defined decision.
Important limitations: Overfitting, missing data, measurement heterogeneity, dataset shift, selective predictor coding and optimistic internal performance.
Development data may come from a prospective cohort, registry, trial dataset or routinely collected records. The source should fit intended use. A highly curated trial population can be useful but may have narrower case mix and more complete measurements than routine practice. Routinely collected data can be larger but may have inconsistent definitions and missingness.
Candidate predictors
Prespecify predictor handling
- Define each candidate predictor and why it is available at the prediction time point.
- Preserve continuous predictors rather than automatically categorising them.
- Prespecify clinically plausible transformations or use flexible functions with sufficient data.
- Define reference categories and coding of categorical predictors.
- Avoid screening predictors solely by univariable p-values.
- Decide how repeated measurements are summarised before modelling.
- Define handling of missing predictors at development and at future use.
- Distinguish candidate predictors from parameters: multi-level variables and nonlinear terms consume multiple degrees of freedom.
Predictor selection should combine subject-matter knowledge, intended implementation and statistical discipline. Including a laboratory marker that will not be available at the decision point can make apparent performance irrelevant to practice. Conversely, excluding a known strong predictor because its univariable p-value exceeds 0.05 can destabilise a model.
Sample size and model complexity
Simple “10 events per variable” rules are inadequate as a universal planning strategy. Sample size should consider the outcome frequency, total number of candidate predictor parameters, expected model fit and desired control of optimism and estimation precision.[3,4]
For binary and time-to-event models, contemporary approaches specify criteria such as limiting overfitting, controlling shrinkage and estimating overall outcome risk precisely. The sample-size plan should be conducted before development and should count the full parameter burden rather than only the number of named variables.[4]
Why parameter count matters more than a list of variable namesMMed extension
A predictor with five categories needs several model parameters. A nonlinear spline for age uses multiple parameters. Interactions add more. Machine-learning models may have far greater effective flexibility. Sample-size justification must reflect the degrees of freedom exposed to outcome data, because overfitting is driven by model flexibility relative to available information.
Missing data
Complete-case analysis can waste information and create selection bias when complete records differ systematically from incomplete records. Development protocols should examine missingness by predictor, outcome and participant characteristics, and use appropriate methods such as multiple imputation where assumptions are defensible.
The implementation problem also matters: if a future model requires five predictors but one is often unavailable in routine care, the deployment strategy must define what happens. Development should not assume perfect data availability when intended use does not.
Model building without chasing the sample
Common overfitting practices include testing many transformations and interactions, selecting the best apparent subset, tuning hyperparameters on the full dataset and repeatedly checking performance until a favourable result appears. The more the outcome data influence decisions, the greater the need for internal validation of the entire modelling process, not merely the final fitted equation.
Shrinkage or penalisation can reduce extreme fitted coefficients. Regression splines can model non-linearity more realistically than arbitrary cut-points. Machine-learning methods can capture complex patterns but still require sufficient sample size, rigorous tuning and evaluation.[1,2]
Internal validation
Randomly splitting a modest dataset into a development and test subset is often statistically inefficient: the model is developed on fewer participants and the test-set performance can be imprecise. Resampling methods such as bootstrapping can estimate optimism while using all data for development. Cross-validation can be appropriate, especially for tuning and more complex algorithms, but the resampling procedure must repeat all data-driven modelling steps.[2]
Internal validation assesses reproducibility within the development data source. It is not a substitute for evaluation in genuinely new data.
Performance measures
A model needs more than one performance dimension.
Calibration
Calibration concerns agreement between predicted probabilities and observed outcome frequencies. At minimum, inspect calibration-in-the-large, calibration slope and a flexible calibration plot. A model can rank people well but systematically overpredict absolute risk.[5]
Discrimination
Discrimination concerns separation between participants with different outcomes, often summarised by the c-statistic or area under the receiver-operating-characteristic curve for binary outcomes. Good discrimination does not guarantee good calibration.
Overall accuracy
Measures such as the Brier score combine aspects of calibration and discrimination. Their interpretation depends on outcome frequency and should be complemented by more interpretable displays.
Illustrative model-development specification
illustrativeTable 1. Prespecified development elements
| Element | Illustrative choice |
|---|---|
| Population | Women assessed 12 weeks after gestational diabetes |
| Prediction time | Standardised postpartum assessment |
| Horizon | Five years |
| Outcome | Incident type 2 diabetes |
| Candidate predictors | Age, BMI, fasting glucose, pregnancy treatment, family history |
| Missing data | Multiple imputation under prespecified assumptions |
| Functional form | Continuous predictors retained; nonlinear terms considered prespecifically |
| Internal validation | Bootstrap of the full modelling process |
| Performance | Calibration plot/slope, discrimination, overall error |
Risk of bias and reporting
PROBAST was developed to assess risk of bias and applicability in prediction-model studies, and PROBAST+AI updated the framework in 2025 for prediction models using regression or AI methods.[6,7] Important concerns include participant selection, predictor definition, outcome definition and analysis. High apparent performance from a small or poorly validated model is a risk-of-bias issue, not merely a reporting inconvenience.
TRIPOD+AI, published in 2024, provides harmonised reporting guidance for prediction-model studies regardless of whether modelling uses regression or machine learning.[1] Reporting should permit others to understand exactly how the model was derived and, when appropriate, calculate predictions.
Appraising a development study
- Is the target population and prediction time point clinically coherent?
- Are all predictors available at the moment the model is intended to be used?
- Is the outcome defined without predictor-dependent circularity?
- Was sample size justified for the candidate parameter burden?
- Were continuous predictors and missing data handled appropriately?
- Did internal validation repeat the complete modelling strategy?
- Are calibration and discrimination reported rather than only classification accuracy?
- Is the final model sufficiently specified for independent validation?
Ethical and governance considerations
Interface with statistical analysis
For a binary outcome at a fixed horizon, logistic regression is one possible method; for censored time-to-event outcomes, survival models are usually required. Machine-learning approaches may also be used. The methodological requirements—clear target, sufficient information, appropriate validation, calibration, discrimination and transparent reporting—apply regardless of algorithm family.[1]
The linked Biostatistics prediction-model lecture covers coefficients, probabilities and performance measures in greater computational detail.
Additional development decisions that determine reproducibility
Outcome prevalence and ascertainment
Prediction models learn from the outcome definition they are given. If outcome ascertainment is incomplete or differs according to predictors, model performance can reflect measurement artefact. For routinely collected data, investigators should verify coding validity and consider whether people who attend more often have more opportunities to be diagnosed. A precise algorithm for outcome ascertainment belongs in the protocol and final report.
Predictor measurement harmonisation
A variable name such as “creatinine” or “blood pressure” does not guarantee measurement equivalence. Units, assay platforms, timing, posture, device calibration and preprocessing can differ. Development should choose predictor definitions that can be reproduced in future settings. For imaging or AI features, preprocessing and feature extraction are part of the model and require version control.
Interactions and nonlinearities
Clinical relationships are often nonlinear, but unrestricted searching for nonlinear terms and interactions consumes information rapidly. Pre-specify plausible nonlinear forms or use controlled flexible methods with penalisation and adequate sample size. Interactions should usually have a substantive rationale and should be validated as part of the complete model rather than interpreted from isolated p-values.
Optimism, shrinkage and model stability
Overfitting is not merely “too many variables”. It is instability in the fitted prediction function caused by too much flexibility for the available information. Bootstrap samples can reveal how much performance changes when the development process is repeated. Shrinkage or penalisation pulls unstable estimates away from extreme values and can improve future calibration.
Stability also includes the selected predictors and functional forms. If small perturbations of the dataset produce very different models, a single final equation can give a false sense of certainty. Researchers should report modelling procedures clearly enough that this uncertainty can be understood and should avoid presenting one data-driven selected subset as if it were uniquely established.
Machine learning does not remove classical prediction requirements
Machine-learning algorithms can handle complex interactions and high-dimensional inputs, but they still require a well-defined outcome, independent evaluation and calibration. Hyperparameter tuning is model development. If the same test data are repeatedly consulted while tuning, the test set effectively becomes part of development.
Nested resampling or a separate locked evaluation set is needed when tuning is extensive. Class imbalance should not be “fixed” by altering prevalence in a way that makes predicted probabilities no longer correspond to the intended population without subsequent calibration. Accuracy, F1 score or other classification metrics should not replace probability calibration when the clinical use depends on absolute risk.
From a fitted model to a usable model object
A prediction model is more than a list of statistically significant variables. Reproducibility requires the full equation or executable algorithm, all coefficients, intercept or baseline survival, transformations, handling of missing inputs and output definition. If an online calculator is created, its implementation should be checked against the published equation using test cases.
Model reproducibility items to preserve
- Exact predictor names, units and measurement windows.
- Coding of categories, nonlinear functions and interactions.
- Coefficients and intercept or baseline survival quantities.
- Preprocessing and imputation steps needed at prediction time.
- Software/model version and random seeds where relevant.
- Output horizon and probability scale.
- Test cases that verify implementation against the analytical model.
Reporting model uncertainty
Predicted probabilities are estimates. Although individual prediction intervals are not routinely available for every model, uncertainty in coefficients, baseline risk and performance should influence interpretation. A development study should avoid categorising patients into precise risk groups with unjustified certainty, especially near decision thresholds.
From development to the next evidence stage
Before declaring development complete, freeze the final model specification and create a validation-ready package. This includes the prediction equation or model object, data dictionary, preprocessing rules, outcome horizon and example calculations. The next dataset should not be used to keep tuning the same model while simultaneously being called external validation. A clear version boundary protects the distinction between development and validation.
Avoid outcome-blind convenience preprocessing
Even preprocessing decisions can leak outcome information when they are chosen after inspecting predictive performance. Feature selection, imputation tuning, image normalisation and threshold choice should be nested within internal validation when they are data driven. The final reported performance should therefore represent the whole pipeline that future patients will encounter, not only the last fitted algorithm.
Freeze the target before comparing algorithms
Comparing several algorithms is meaningful only when they use the same eligible population, outcome definition, prediction horizon and evaluation data. Otherwise a performance difference may reflect a changed prediction problem rather than a better modelling method.
Worked example
Developing a five-year postpartum diabetes model
The team defines the prediction moment as the 12-week postpartum assessment. Only information available at that visit is eligible as a predictor. They retain age, body mass index and fasting glucose as continuous variables, prespecify coding for pregnancy treatment and family history, and avoid univariable screening.
Sample size is planned according to the expected outcome proportion and candidate parameter count rather than a fixed events-per-variable slogan. Missing predictor values are addressed with a prespecified imputation strategy. The model-development procedure includes any transformation and penalisation decisions.
Bootstrap internal validation repeats the complete modelling process to estimate optimism. The final report shows a calibration plot, calibration slope, discrimination and overall error, provides the full model specification, and clearly labels the model as internally validated development work requiring external evaluation before clinical use.
Suggested protocol or reporting wording
Common mistakes
Knowledge check
What is the most important first step before choosing logistic regression, random forests or another algorithm?
Answer and reasoning
Define the prediction problem: target population, prediction time point, outcome, horizon and intended use. Algorithm choice cannot repair an incoherent clinical target.
Why is univariable p-value screening a poor default for selecting prediction-model predictors?
Answer and reasoning
It is unstable, ignores joint predictive information, encourages data-driven selection and can discard clinically established predictors. Prediction focuses on overall risk estimation rather than significance of each isolated association.
Why should calibration be reported even when the model has a high c-statistic?
Answer and reasoning
Discrimination measures ranking, not whether predicted probabilities are numerically correct. A model can rank participants well while systematically overpredicting or underpredicting absolute risk.
What does internal validation estimate?
Answer and reasoning
It estimates how much the model-development process has capitalised on the development data and how performance may degrade for similar new participants from the same underlying setting. It does not demonstrate transportability to another setting.
Why is model complexity relevant to sample-size planning?
Answer and reasoning
More candidate parameters give the model more opportunity to fit noise. Adequate information is needed to estimate those parameters stably, control overfitting and obtain reliable predictions.
Key take-home messages
Key take-home messages
- Define the clinical prediction target before selecting the modelling method.
- Use predictors available at the intended decision point and avoid significance-driven screening.
- Plan sample size around outcome information and model complexity.
- Handle missing data and continuous predictors deliberately and validate the full development process internally.
- Report calibration, discrimination and complete model specification, then proceed to external validation.
References
- Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://doi.org/10.1136/bmj-2023-078378
- Efthimiou O, Seo M, Chalkou K, Debray T, Egger M, Salanti G. Developing clinical prediction models: a step-by-step guide. BMJ. 2024;386:e078276. https://doi.org/10.1136/bmj-2023-078276
- Riley RD, Snell KIE, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med. 2019;38(7):1276–1296. https://doi.org/10.1002/sim.7992
- Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441. https://doi.org/10.1136/bmj.m441
- Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230. https://doi.org/10.1186/s12916-019-1466-7
- Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51–58. https://doi.org/10.7326/M18-1376
- Moons KGM, Damen JAA, Kaul T, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. https://doi.org/10.1136/bmj-2024-082505
These materials support learning and research planning. They require adaptation to the study context and do not replace methodological, statistical, ethical, or regulatory consultation.