Reproducible Research and Open Science

Overview

Reproducible research is research whose analytic pathway can be inspected and, where appropriate, rerun from the documented data, code, software environment and decision record. Open science is broader: it includes practices such as preregistration, open protocols, data and code sharing, transparent materials, preprints and open access, implemented in ways that remain compatible with participant rights and legitimate governance. FAIR principles emphasise that digital research objects should be findable, accessible under stated conditions, interoperable and reusable.[1] Current NIH data-sharing policy, OSF preregistration resources, TOP guidance, ICMJE publication recommendations and ICH E6(R3) all reinforce prospective transparency and reliable records, although their formal scope differs.[2–6]

Open science should not be reduced to “put everything on the internet”. In health research, unrestricted release of individual-level data may breach consent, confidentiality, contractual obligations or data-protection law. The aim is maximal responsible transparency, not maximal disclosure.

Why this matters

A result can be statistically correct yet impossible to reproduce because the final dataset was manually edited, package versions were not recorded, analysis steps were performed interactively, or excluded records cannot be reconstructed. Reproducibility failures waste time, hinder peer review and make honest error difficult to detect. They also make later updates and secondary analyses fragile.

Open-science practices can reduce selective reporting and improve cumulative research. Preregistration creates a time-stamped record of planned questions and analyses, helping readers distinguish confirmatory from exploratory work. Sharing code allows others to inspect derivations even when data remain controlled. Persistent identifiers and metadata make datasets discoverable and reusable. These practices work best when designed into the study rather than added as an afterthought at publication.

Key concepts

Computational reproducibility usually means obtaining the same results from the same data and analytic procedures. Replicability often refers to whether a new study or new data produce a substantively consistent finding, although terminology varies by field. Transparency means exposing the decisions, materials and provenance needed to understand the work. Open science is an ecosystem of practices and policies that can support all three.

Preregistration records planned hypotheses, outcomes, sampling and analyses before the relevant results are known. It does not guarantee that the plan is good, and deviations are not misconduct. Its value lies in preserving chronology. A registered report adds peer review of the research question and methods before results are known, where the journal supports that format. Exploratory analyses remain scientifically valuable when they are clearly labelled.

FAIR is also frequently misunderstood. The “A” is accessible, not necessarily “open”. Metadata can be public while sensitive participant data are available only through a controlled-access process. Reusability requires licences, provenance, data dictionaries and documentation sufficient to understand what variables mean and what restrictions apply.[1]

DECISION GUIDE

Choose transparency practices according to the research object

• preregister decisions whose chronology matters;
• version-control protocols, code and metadata;
• preserve immutable or protected raw/source-aligned data;
• generate derived datasets and outputs from scripts rather than manual edits;
• share metadata, code and materials as openly as possible;
• use controlled access when participant privacy or governance prevents public data release.

A step-by-step methodological workflow

1. Create a project structure before data arrive

Separate raw/source-aligned data, derived data, code, outputs, documents and confidential identifiers. Define naming and versioning conventions. Raw data should be protected from accidental overwriting. A README or project index should explain how the components fit together so that a new team member can navigate the project without relying on oral history.

2. Record prospective decisions

Register or timestamp the protocol and analysis plan where appropriate. Include hypotheses, outcomes, eligibility, sampling, exclusion rules and primary analysis at a level that constrains consequential choices. If plans change, preserve the original and add an amendment rather than replacing history. Preregistration distinguishes confirmatory and exploratory work; it does not prohibit learning from the data.[3]

3. Use scripted data transformations

Cleaning, recoding, derivation and exclusions should be implemented in code when feasible. The script should create a new analysis dataset from protected source data. High-impact transformations should include assertions or validation checks. Manual review may still be needed for source discrepancies, but the decision and corrected value should be recorded in an auditable system rather than silently changed in the analysis file.

4. Capture the computing environment

Record software, package versions and dependencies. Set and document random seeds for stochastic procedures where this supports exact reruns. For complex analyses, environment-lock files or containers may improve longevity. Reproducibility does not require every reader to use the same software, but the original environment should be sufficiently documented to regenerate the published results.

5. Publish research objects responsibly

At publication, consider the protocol, SAP, code, data dictionary, synthetic example data, materials and analysis outputs. Use persistent identifiers and repositories suited to the object. When participant-level data cannot be public, publish metadata and a data-access statement describing who may apply, for what purposes, what review occurs and what restrictions apply.

6. Plan maintenance, correction and reuse

A reproducible project should survive staff turnover. Archive the final analysis snapshot and code that generated the publication. Keep licences and data-use agreements with the metadata. If an error is discovered, correct the code and regenerate outputs rather than editing the manuscript table directly. Document the correction so the provenance chain remains intact.

PROTOCOL CHECKLIST

Reproducibility and openness plan

• preregistration or protocol timestamping strategy;
• repository and version-control structure;
• raw-data protection and reproducible derivation pipeline;
• software and dependency capture;
• code review and high-impact validation checks;
• metadata, persistent identifiers and licensing;
• public versus controlled-access research objects;
• retention, maintenance and correction procedures.

An auditable planning document

Research-object sharing matrix

illustrative

Table 1. Illustrative openness plan for sensitive health research

Object	Default visibility	Reason
Protocol/SAP	Public after appropriate registration	Preserves planned decisions
Analysis code	Public or controlled if revealing sensitive logic	Enables inspection and reuse
Data dictionary	Public where feasible	Supports interpretation
Participant identifiers	Restricted	Direct privacy risk
De-identified individual data	Controlled access	Residual re-identification risk
Aggregate results	Public	Supports verification and reuse
Software environment file	Public	Supports computational reproduction

Note: Actual sharing must follow consent, institutional policy and applicable law.

Quality, validity and governance

Reproducibility should be tested rather than assumed. Before submission, ask someone who did not write the analysis to regenerate a key table from the archived instructions. Missing paths, unrecorded manual steps and undocumented package dependencies become visible immediately. This “fresh environment” test can be performed without sharing confidential data outside the authorised team.

Preregistration quality also matters. A vague registration stating that “regression will be used” leaves almost as much flexibility as no registration. Prespecify outcomes, exclusions, covariates, interactions and missing-data strategy at a level appropriate to the study. Nevertheless, avoid false precision: if a decision genuinely depends on observed data quality or model diagnostics, prespecify the decision rule rather than inventing an arbitrary choice.

Open science is not ethically neutral. Sharing may create re-identification risks, group harms or breaches of community expectations. Current policies emphasise data management and responsible sharing, not indiscriminate release.[2] Participant consent language, repository controls and data-use agreements should match the actual sharing plan.

APPRAISAL CHECKLIST

Appraise a project for reproducibility

• Can the published tables be regenerated from archived data and code?
• Are raw data protected and derivations scripted or auditable?
• Can the chronology of protocol and analysis decisions be reconstructed?
• Are software dependencies and versions documented?
• Do data and code availability statements match what is actually accessible?
• Are restrictions on health-data sharing justified and transparent rather than used as a blanket excuse for poor documentation?

Practical decision audit

Audit every manual step

Ask analysts to list actions performed outside scripts: spreadsheet corrections, file merges, recodes, exclusions and figure edits. Either automate these actions or document them in a controlled audit trail. A single unrecorded manual deletion can make the entire downstream workflow irreproducible.

Audit preregistration specificity

Compare the registered analysis to the final code. Determine which decisions were fixed, which were amended before results, and which were exploratory. A preregistration is a chronology tool, not a ritual; its usefulness depends on whether it meaningfully constrains the decisions readers care about.

Audit the sharing boundary

For each research object, state why it is public, controlled or closed. “Ethics” is not a sufficiently specific reason by itself. Determine whether the limitation arises from consent, re-identification risk, contractual terms, indigenous or community governance, intellectual property, or another legitimate restriction, and describe the access route where possible.

Audit long-term usability

A repository full of unlabeled CSV files is not reusable. Include variable definitions, units, missing-value codes, provenance, licences, dates and relationships between tables. Persistent identifiers help discovery, but semantic documentation determines whether the object can be interpreted correctly.

Open science changes the distinction between discovery and confirmationMMed extension

One of the most valuable consequences of prospective transparency is that exploratory work no longer has to masquerade as confirmation. Researchers can preregister the primary question, then openly explore unexpected heterogeneity, nonlinear effects or new hypotheses. The resulting exploratory findings can motivate replication or a new confirmatory study.

This separation is particularly important in high-dimensional analyses and machine learning, where the number of potential specifications can be enormous. Reproducible pipelines, locked evaluation datasets, external validation, registered analysis plans and transparent reporting provide complementary safeguards. No single open-science practice guarantees truth; the value comes from a system in which decisions, data transformations and evidence claims can be inspected.

Designing a reproducibility stack

Reproducibility is strongest when several safeguards reinforce one another. Prospective transparency records what was intended. Version control preserves the evolution of code and documents. Data provenance preserves the source and derivation of values. Environment capture records the software context. Automated outputs reduce transcription. Independent checks detect mistakes. Responsible sharing allows external scrutiny. None of these is sufficient alone, but together they create a reproducibility stack that is resilient to staff turnover and software change.

File naming is not version control

Folders containing analysis_final, analysis_final2 and analysis_FINAL_revised do not provide reliable history. A version-control system records changes, authorship, timestamps and prior states. For teams uncomfortable with command-line tools, institutional platforms can still enforce controlled check-in/check-out or immutable snapshots. The methodological principle is that a prior state can be recovered and changes can be attributed.

Reproducible code requires stable inputs

A script cannot reproduce a result if the source file changes in place. Raw exports should receive a checksum, timestamp or immutable version identifier. External data transfers should be documented with receipt date and validation. If a source system refreshes nightly, the study needs a defined extraction date or snapshot. This is especially important for routinely collected data, where the same database query may return different records months later because late coding and corrections continue.

Literate workflows reduce divergence between analysis and report

Tools such as Quarto, R Markdown or Jupyter notebooks can combine code, output and explanatory text so that tables and figures are regenerated from the analysis. The tool itself does not guarantee quality; poorly structured notebooks can be hard to audit. A robust workflow separates reusable functions, data cleaning, analysis and reporting, and avoids hidden state from manually executed cells. The final report should be buildable from a clean session.

Synthetic and simulated data can support openness

When real participant-level data cannot be shared, a carefully constructed synthetic dataset can demonstrate the structure expected by the code and allow readers to test the pipeline. Synthetic data must be clearly labelled and should not be presented as preserving the empirical distribution perfectly. It complements, rather than replaces, controlled access to real data when validation of the exact result is required.

Open materials improve intervention reproducibility

For behavioural, educational or implementation interventions, sharing only analysis code is insufficient. Manuals, prompts, questionnaires, training materials and intervention logic may determine whether another team can reproduce the study. Intellectual-property constraints should be addressed during planning. If materials cannot be openly licensed, describe how qualified researchers can obtain them.

Reproducibility also requires semantic stability

Variable names can remain unchanged while meaning changes. A laboratory field may switch assay, a diagnosis category may change coding rules, or a survey instrument may be revised. Metadata should therefore capture units, versions, coding systems and calendar periods. Reproducibility of computation is not enough if the underlying variables are misunderstood.

Privacy-preserving reproducibility can be engineered

Secure enclaves, federated analyses, remote execution and controlled-access repositories can permit verification without distributing raw data. Analysts can sometimes submit code to an environment where data reside and receive vetted outputs. The key is to make the access mechanism explicit and feasible. A statement that “data are unavailable due to privacy” is weaker than a governed process explaining how legitimate verification requests are considered.

Reproducibility should be tested at project milestones

Do not wait until publication. After the first 20 participants, test whether the data pipeline regenerates monitoring tables. Before database lock, rebuild the analysis dataset from raw inputs. Before submission, regenerate the manuscript outputs in a clean environment. At archiving, verify that the final snapshot contains all code, metadata and documentation. These rehearsals convert reproducibility from aspiration into quality control.

Reproducibility under realistic research constraints

Reproducibility planning should identify the smallest set of research objects needed to regenerate each published result. For a quantitative study this may include a frozen analysis dataset or a documented derivation pipeline, analysis scripts, software and package versions, random seeds where relevant, and a machine-readable record linking outputs to code. For qualitative or mixed-methods work, reproducibility may instead emphasise transparent coding frameworks, analytic decisions, versioned codebooks and carefully governed examples because raw narratives may be impossible to share safely.

Teams should also distinguish computational reproducibility from independent replication. Re-running the same code on the same data checks whether the reported result can be regenerated; collecting new data to examine whether a finding recurs addresses a different scientific question. Neither substitutes for sound design.

A practical governance model is to classify every research object as public, controlled-access, restricted or non-shareable, with a reason recorded for the classification. This prevents open-science aspirations from overriding consent, confidentiality, intellectual-property obligations or legitimate community interests. Where data cannot be released, teams can often still share variable dictionaries, analytic code, synthetic examples, metadata, workflow diagrams and the process for requesting controlled access. The objective is not maximal disclosure at any cost, but the greatest responsible transparency consistent with participants’ rights and the study’s governance obligations.

Reproducibility as a team property

Reproducibility fails when knowledge exists only in one analyst’s memory. Teams should therefore make important transformations, exclusions, recoding decisions and output-generation steps discoverable to another authorised researcher. A lightweight reproducibility review can ask a colleague who did not write the analysis to regenerate one primary table and one figure from the documented workflow. Failure at this stage is useful: it reveals missing paths, undocumented software dependencies, hidden manual edits or unclear data versions before publication.

Long-lived projects also need ownership rules. Specify who can approve changes to shared code, who maintains controlled datasets after staff leave, where immutable analysis snapshots are stored and how corrections will be propagated to manuscripts or public repositories. Open science is more sustainable when responsibility for research objects survives changes in personnel.

Worked example

WORKED EXAMPLE

Rebuilding the cohort workflow for reproducibility

For the next multicentre cohort, the team creates a Git repository for code and text documents, while confidential data remain in the secure institutional environment. Raw exports are write-protected. A scripted pipeline checks variable types, creates harmonised site variables, applies prespecified exclusions and generates an analysis dataset with a unique version identifier.

The analysis plan is registered before the primary outcome models are run. The team records R and package versions and stores a lock file. Table 2 is generated directly from the analysis script into a publication-ready format. A second analyst can reproduce the table from the same controlled dataset without manual instructions.

At publication, the protocol, SAP, code, data dictionary and synthetic example dataset receive repository links. Participant-level data are not posted publicly because of governance constraints; instead, the data-availability statement explains the controlled-access application process. The project is more open than before, but it remains privacy-preserving.

Suggested protocol and manuscript wording

Common mistakes

Knowledge check

Does FAIR require that all research data be openly downloadable?

Answer and reasoning

No. FAIR requires accessible data or metadata under clearly defined conditions; access may legitimately be controlled.

What is the main value of preregistration?

Answer and reasoning

It preserves the timing of planned questions and analyses, helping distinguish prospective confirmation from later exploration.

Can an analysis be reproducible if the raw data remain inside a secure data enclave?

Answer and reasoning

Yes. Reproducible code, metadata, documented environments and controlled re-execution can support reproducibility without public data release.

Why protect raw data from overwriting?

Answer and reasoning

Because source-aligned data provide the reference point from which cleaning and derivation decisions can be reconstructed and audited.

Key take-home messages

SUMMARY

Key take-home messages

• Design reproducibility into the project before data collection and analysis.
• Preserve decision chronology through preregistration, versioning and amendments.
• Generate derived data and outputs from auditable code rather than undocumented manual edits.
• FAIR and open are not synonyms; sensitive data may require controlled access.
• Responsible openness includes protocols, code, metadata, materials and clear access procedures, not only datasets.

References

1. Wilkinson MD, Dumontier M, Aalbersberg IJJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. https://doi.org/10.1038/sdata.2016.18
2. National Institutes of Health. Data Management & Sharing Policy overview. Updated 2026. https://grants.nih.gov/policy-and-compliance/policy-topics/sharing-policies/dms/policy-overview
3. Center for Open Science. Preregistration. https://www.cos.io/initiatives/prereg
4. Center for Open Science. Transparency and Openness Promotion Guidelines. https://www.cos.io/initiatives/top-guidelines
5. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Updated January 2026. https://www.icmje.org/recommendations/
6. International Council for Harmonisation. ICH E6(R3) guideline for good clinical practice: principles and Annex 1. Step 4. 2025. https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf