Developing a Statistical Analysis and Data Management Plan

Overview

A statistical analysis plan (SAP) converts the research questions and protocol into explicit rules for creating analysis variables, defining populations, estimating effects, handling missing information and producing the primary tables and figures. Published SAP guidance for clinical trials recommends a minimum set of prespecified content, while ICH E9(R1) emphasises defining the treatment effect or estimand before choosing the estimator.[1,2] Current good clinical practice also emphasises reliable records and fit-for-purpose data systems.[3] A data management plan (DMP) complements the SAP by describing how data are captured, structured, validated, protected, versioned, shared and preserved.[4–6]

These documents are most useful when developed together. An elegant analysis cannot recover variables that were not collected, timestamps that were overwritten, or outcome definitions that changed during follow-up. Conversely, a perfectly curated dataset does not protect against selective modelling if the analysis rules remain vague until results are visible.

Why this matters

Analytical flexibility is unavoidable in complex research, but undisclosed flexibility can bias inference. “Use multiple imputation if needed” leaves unanswered which variables enter the imputation model, how many imputations are used, how derived variables are treated and what happens if the algorithm does not converge. “Adjust for confounders” is similarly incomplete if the adjustment set is chosen after inspecting p-values. Prespecification should therefore be sufficiently detailed to constrain consequential choices while still allowing transparent responses to unforeseen data problems.

Data management has the same logic. A final spreadsheet may contain the numbers used for analysis, but without provenance it can be impossible to determine where a corrected value came from, which raw fields produced a derived variable, or which version of the code generated a table. FAIR principles emphasise findability, accessibility, interoperability and reusability, but FAIR does not mean that confidential health data must be openly downloadable.[4] Access must remain compatible with consent, governance and privacy.

Key concepts

The estimand describes the quantity the study aims to estimate. In a randomised trial, this includes the population, treatment conditions, outcome, handling of intercurrent events and population-level summary.[2] The estimator is the statistical procedure used to estimate that quantity. Keeping these concepts separate prevents the common error of choosing a familiar model first and then retrospectively interpreting whatever it estimates.

The SAP should define analysis populations, outcome derivations, covariate coding, descriptive summaries, primary and secondary models, effect measures, confidence intervals, multiplicity, missing data, sensitivity analyses, subgroup analyses and table shells. A high-quality SAP also states what will happen when assumptions fail or algorithms do not converge. “Use a non-parametric test if data are non-normal” is usually too crude; model diagnostics and the estimand should guide alternatives.

The DMP covers the lifecycle of data rather than the final dataset alone. It should identify source systems, data ownership or stewardship, variable naming, coding standards, validation rules, queries, corrections, audit trails, transfer procedures, access roles, backups, retention and controlled sharing. The SAP and DMP meet at the analysis dataset: every derived field should have a traceable source and reproducible derivation.

DECISION GUIDE

Build the SAP and DMP around five linked objects

• the estimand or descriptive target for each primary objective;
• the raw variables and provenance required to construct that target;
• the analysis dataset and prespecified derivation rules;
• the estimator, uncertainty measure and sensitivity analyses;
• the version-controlled workflow that connects raw data to reported results.

A step-by-step methodological workflow

1. Freeze the scientific target before the model

Write the primary research question in estimand language where relevant: population, conditions being compared, outcome and timing, handling of important intercurrent events, and the population-level contrast. For observational studies, define the target association, causal contrast, prediction target or descriptive parameter with equal precision. The primary model should be selected only after this target is clear.

2. Define analysis populations and data states

Specify who contributes to the primary analysis and why. In trials, distinguish randomised, treated, safety and per-protocol sets only when they correspond to meaningful questions. In observational studies, specify the cohort entry rule, required baseline data and follow-up contributions. Define data cut-off, database lock or equivalent milestones so that the chronology of analysis is recoverable.

3. Create derivation specifications

For every derived variable, document source fields, transformations, time windows, hierarchy rules and handling of conflicting values. Examples include baseline averages, change scores, composite outcomes, event dates and categorical thresholds. Derivations should be implemented in code rather than by manual editing whenever feasible, with unit tests or validation checks for high-impact variables.

4. Prespecify models and analytical contingencies

Describe the primary model, effect measure, confidence interval, covariates, functional forms, repeated-measures or clustering structure, diagnostics and software. Prespecify the response to foreseeable problems such as sparse cells, non-convergence, influential observations or strong departures from model assumptions. Sensitivity analyses should target assumptions that could materially change interpretation, especially missing-data and estimand assumptions.

5. Engineer data provenance and controlled correction

The DMP should preserve raw or source-aligned data, maintain an audit trail for corrections, define roles and permissions, and avoid overwriting values without trace. Specify how external datasets are received and verified, how identifiers are separated, and how codebooks and data dictionaries are versioned. Backups should be independent and recoverable rather than simple file synchronisation.

6. Plan reproducible outputs and controlled disclosure

Generate tables and figures from code linked to a fixed analysis dataset. Record software versions, package dependencies and random seeds where relevant. Decide prospectively what data, code, metadata and documents can be shared, under what access conditions, and for how long. A data-sharing plan must reflect participant consent and governance, not an abstract expectation that all data should be public.

PROTOCOL CHECKLIST

SAP and DMP items to settle before outcome analysis

• primary and secondary estimands or target parameters;
• analysis populations and data cut-off;
• raw-to-derived variable specifications;
• primary models, effect measures and uncertainty intervals;
• covariate, interaction and subgroup rules;
• missing-data, multiplicity and sensitivity-analysis strategy;
• data provenance, query, correction and audit-trail procedures;
• access control, versioning, backup, retention and sharing;
• table, figure and listing shells with reproducible code workflow.

An auditable planning document

Analysis-decision register

illustrative

Table 1. Illustrative SAP/DMP decision register

Decision	Prespecified rule	Data-management implication
Primary outcome	Completed qualifying test by day 84	Link laboratory and verified external results
External test	Accept if date, test type and source are documented	Store verification status and source
Missing outcome	Primary assumption plus specified sensitivity scenario	Preserve reason and last contact date
Intervention switching	Address according to stated estimand strategy	Capture exposure and switching timestamps
Covariates	Fixed prespecified set	Standardise coding before database lock
Analysis dataset	Generated by version-controlled script	No manual editing of derived fields

Note: This register is illustrative and should be expanded for the actual study.

Quality, validity and governance

Prespecification has degrees. A plan written before seeing comparative outcome data is more protective than one written after investigators have examined treatment differences. The SAP should therefore record its version, approval date and relationship to database lock or unblinding. Changes are possible, but each change should state whether outcome data were available and why the change was necessary.

The strongest SAPs distinguish primary, sensitivity, supplementary and exploratory analyses. Sensitivity analyses test robustness to assumptions while keeping the scientific target as stable as possible; supplementary analyses may answer related but different questions. This distinction is central to ICH E9(R1).[2] For non-trial research, the same discipline helps separate confirmatory from exploratory modelling.

Data quality should be evaluated according to fitness for the intended analysis. A variable can be complete but invalid, or valid but captured too late to serve as a baseline covariate. A data-quality dashboard should therefore monitor critical variables, timing, duplicates, logical inconsistencies, missingness and protocol deviations. Provenance should allow a reviewer to trace a reported estimate backwards from table to analysis dataset, derivation code and source variables.

APPRAISAL CHECKLIST

Appraise whether the analysis and data workflow are truly prespecified

• Is the primary estimand or target parameter explicit before the estimator is described?
• Could two statisticians independently implement the primary analysis and obtain the same result?
• Are derived variables traceable to raw fields and time windows?
• Are missing-data and sensitivity analyses tied to substantive assumptions?
• Is the timing of SAP approval relative to data access documented?
• Can the reported results be regenerated from a version-controlled dataset and codebase?

Practical decision audit

Audit outcome derivations

Composite or time-window outcomes are especially vulnerable to hidden discretion. List every data source that can establish the outcome, the priority when sources conflict, whether late observations are admissible, and how partial components are handled. Validate derivation code using deliberately constructed edge cases.

Audit covariate and subgroup freedom

Avoid choosing covariates by univariable p-values. Prespecify confounders from subject-matter reasoning, or define a reproducible selection procedure if the objective is prediction. Subgroup variables, cut points and interaction tests should be distinguished from post hoc exploratory stratification.

Audit missing-data assumptions

Do not write “missing data will be excluded”. Identify why data may be missing, which variables predict missingness, the primary analytic assumption, and plausible departures. If multiple imputation is planned, specify the imputation model at enough detail to prevent many undocumented implementations.

Audit reproducibility and access

A reproducible pipeline should not depend on one person’s desktop or memory. Store code, dictionaries, environment information and immutable analysis snapshots in controlled locations. For sensitive data, reproducibility can be achieved through governed access and executable code without public disclosure of participant-level records.

Estimand, estimator and sensitivity analysis are different layersMMed extension

In advanced trial work, a common analytical error is to treat a model as if it defines the scientific question. The estimand framework reverses that order. First define the treatment effect of interest, including how events such as treatment discontinuation, rescue therapy or death are handled. Then choose an estimator whose assumptions target that estimand. Finally, use sensitivity analyses to assess robustness to assumptions required for estimation.

The same conceptual separation extends beyond randomised trials. In observational causal research, the target trial or causal estimand should precede the regression model. In prediction, the prediction target and deployment setting should precede algorithm choice. In descriptive research, the target population and parameter should precede weighting or summarisation. This discipline makes the SAP a scientific document rather than a catalogue of statistical commands.

From statistical intentions to executable specifications

An advanced SAP should be written at the level at which an independent statistician could implement the primary analysis without making scientifically consequential choices. This does not mean coding every software command into prose. It means specifying the choices that alter the estimand, estimator, denominator or uncertainty. For a continuous repeated outcome, for example, the plan should identify the outcome scale, visit structure, treatment-by-time terms, baseline adjustment, covariance structure strategy, degrees-of-freedom method, contrast at the primary time point and handling of measurements outside the visit window. Merely writing “a mixed model will be used” leaves many of these choices unresolved.

Define what is locked and what can still be data-informed

Some choices can legitimately depend on blinded or pooled data. A team may assess the prevalence of sparse categories, distribution of a baseline variable or overall missingness before unblinding. The SAP should distinguish these blinded data-informed decisions from choices that will be made after treatment differences or outcome associations are visible. A decision rule can be prespecified—for example, collapsing a category when total counts fall below a threshold—while preserving blindness to comparative outcomes. Recording who had access to what information when the decision was made protects credibility.

Separate estimand sensitivity from model sensitivity

Changing a covariance structure or link function may test model robustness while targeting approximately the same scientific question. Changing from a treatment-policy estimand to a per-protocol effect changes the question itself. The SAP should label these differently. Sensitivity analyses should interrogate assumptions needed to estimate the primary estimand, such as departures from missing-at-random assumptions. Supplementary analyses can estimate alternative clinically relevant effects. Conflating them can make a collection of alternative results look like robustness when they actually answer different questions.

Use analysis dataset specifications as a contract

For complex studies, define one or more analysis dataset specifications before the final derivation. Each variable should have a name, type, label, source, derivation, permissible range, missing code and role in the analysis. The specification can include row structure—for example, one row per participant or one row per participant-visit. When the derivation script is run, automated checks can compare the produced dataset with the specification. This makes data management and statistical analysis meet at a controlled interface.

Prespecify multiplicity according to the claim structure

Multiplicity is not only about the number of p-values. It arises when several hypotheses could independently support the same confirmatory claim, when multiple doses or time points are tested, or when a hierarchy of outcomes determines success. The SAP should explain which claims require error-rate control and which analyses are descriptive or exploratory. Gatekeeping, hierarchical testing or other procedures should be linked to the scientific decision rather than added mechanically. In many observational studies, emphasis on effect estimates and uncertainty may be more informative than attempting to “correct” every analysis with a blanket procedure.

Build failure-handling into the statistical workflow

Algorithms can fail. Multiple imputation may not converge, a mixed model covariance matrix may be singular, separation may occur in logistic regression, or a planned subgroup may contain too few observations. The SAP should identify important foreseeable failures and either specify fallback procedures or establish a blinded decision process. A fallback should preserve the scientific target as far as possible. Choosing whichever alternative gives the most favourable p-value is not an acceptable contingency plan.

Database lock is a governance event, not a magical boundary

A database may be “locked” while unresolved external data remain, or a correction may be scientifically necessary after lock. The DMP should define what lock means, who authorises unlock, how changes are audited and when a new analysis snapshot is created. For non-trial research without a formal database lock, an equivalent immutable analysis cut should be created so the exact dataset used for publication can be reconstructed.

Quality control should include independent reproduction of the primary result

Where resources permit, a second statistician can independently reproduce the primary derivation or model from the specification. Full duplicate programming is not always necessary, but high-impact calculations deserve independent verification. At minimum, code review and reasonableness checks should verify denominators, treatment labels, missingness, effect direction and confidence intervals. Automated pipelines reduce transcription errors but do not guarantee that the scientific specification is correct.

Pre-freeze analysis-plan review

Before the analysis plan is frozen, conduct a structured review in which each primary objective is traced to its estimand, analysis population, outcome derivation, model, uncertainty measure and sensitivity analysis. A second reviewer should be able to identify which dataset fields are required for every analysis and how each derived variable will be constructed. Ambiguous phrases such as “appropriate transformation will be used” or “covariates will be adjusted as necessary” should be replaced by explicit decision rules whenever feasible.

This review should also identify analyses that are genuinely exploratory. Labeling them prospectively protects the primary analysis from being diluted by a large set of post hoc alternatives while still allowing scientifically useful exploration after the prespecified work is complete. The goal of plan freezing is not to prevent learning from unexpected data patterns; it is to preserve a transparent boundary between confirmatory decisions made before outcome inspection and analyses motivated by what the investigators later observed.

Worked example

WORKED EXAMPLE

Planning the postpartum trial analysis before database lock

The team defines the primary estimand as the difference between randomised groups in the probability of completing the qualifying glucose test by day 84 under the prespecified strategy for post-randomisation events. External test results are accepted only when the date, test type and verifiable source are documented. The SAP specifies the effect measure, confidence interval, regression adjustment set, analysis population and sensitivity analysis for uncertain external outcomes.

The DMP stores laboratory results, external verification records and participant contacts in separate source-aligned tables with stable identifiers. A derivation script creates the day-84 outcome and flags conflicts for review. The team never overwrites a laboratory date to “fix” a discrepancy; corrections are made through auditable source resolution.

Before database lock, the statistician generates dummy table shells and tests the code on simulated records containing late tests, duplicate results, missing dates and treatment switching. The SAP is signed and dated before treatment-group outcome summaries are viewed. When an unforeseen coding problem is discovered after lock, the team documents the correction and reruns the pipeline rather than manually altering the analysis dataset.

Suggested protocol and manuscript wording

Common mistakes

Knowledge check

What is the difference between an estimand and an estimator?

Answer and reasoning

The estimand is the target quantity that answers the scientific question; the estimator is the statistical procedure used to estimate it from observed data.

Why should SAP timing be documented?

Answer and reasoning

Because prespecification is most credible when consequential analysis decisions were fixed before investigators could choose them in response to observed comparative results.

What does a data provenance chain allow a reviewer to do?

Answer and reasoning

Trace a reported result back through the analysis dataset, derivation rules and source data, including corrections and versions.

Can a study be reproducible if participant-level data cannot be made public?

Answer and reasoning

Yes. Controlled-access data, comprehensive metadata, versioned code and a documented computing environment can support reproducibility while respecting privacy and governance.

Key take-home messages

SUMMARY

Key take-home messages

• Define the scientific target before choosing the statistical model.
• Prespecification should constrain consequential analytical choices, not merely name a test.
• The SAP and DMP meet at a traceable, reproducibly derived analysis dataset.
• Sensitivity analyses should probe important assumptions rather than serve as a collection of alternative results.
• Reproducibility and confidentiality can coexist through controlled access, metadata and versioned workflows.

References

1. Gamble C, Krishan A, Stocken D, et al. Guidelines for the content of statistical analysis plans in clinical trials. JAMA. 2017;318(23):2337–2343. https://doi.org/10.1001/jama.2017.18556
2. International Council for Harmonisation. ICH E9(R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. 2019. https://www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials-scientific-guideline
3. International Council for Harmonisation. ICH E6(R3) guideline for good clinical practice: principles and Annex 1. Step 4. 2025. https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf
4. Wilkinson MD, Dumontier M, Aalbersberg IJJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. https://doi.org/10.1038/sdata.2016.18
5. National Institutes of Health. Data Management & Sharing Policy overview. Updated 2026. https://grants.nih.gov/policy-and-compliance/policy-topics/sharing-policies/dms/policy-overview
6. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Updated January 2026. https://www.icmje.org/recommendations/