Data Dictionaries, Coding and Metadata

Overview

A dataset is useful only when its variables retain meaning. A data dictionary records what variables represent and how they are encoded. Metadata are data that describe other data: definitions, units, provenance, timestamps, instrument versions, relationships, processing history and other information needed to interpret or manage a dataset. ICH E6(R3) explicitly includes metadata in its discussion of records and data integrity, while FAIR principles emphasise rich description that makes research objects findable and reusable under appropriate access conditions.[1,2]

Without documentation, even the original research team can forget whether smoke=2 meant “former smoker” or “current smoker”, whether weight was recorded in kilograms, or whether a score used the revised or original questionnaire. Metadata convert a collection of values into an interpretable research object.

Why this matters

Data cleaning often becomes difficult not because values are wrong but because their meaning is uncertain. A column labelled status may refer to eligibility, follow-up, disease or database completion. A variable may change coding midway through a study. A blank may mean not asked, not known or zero.

These ambiguities threaten reproducibility and can change results. A clear dictionary also improves communication between clinicians, data managers and statisticians before analysis begins.

What belongs in a data dictionary

A practical dictionary can include:

Not every project needs every column, but the dictionary should contain enough information to reconstruct the meaning of the analytical variables.

Naming variables

Good names are predictable and stable. Examples such as delivery_date, site_id and fasting_glucose_mmol_l are more informative than q1, var23 or newstatus.

Useful conventions include:

• lowercase letters and underscores;
• no spaces or punctuation likely to cause software problems;
• consistent suffixes for units or time points where helpful;
• stable prefixes for instrument families; and
• avoidance of names that encode temporary form order.

Do not rename variables casually after analysis scripts exist. If a correction is necessary, document the change and update dependent code systematically.

Coding categorical variables

Codes are storage representations, not scientific quantities. If site = 1, 2, 3, those numbers do not imply that Site 3 is three times Site 1. Maintain explicit code labels.

For binary variables, a common scheme is 0 = No and 1 = Yes, but consistency matters more than convention. Do not use 0 for “No” in one variable and “Unknown” in another without clear documentation.

Avoid collapsing categories prematurely. If research participants report detailed ethnicity categories, for example, preserve the collected response and create a separate derived grouping for a particular analysis when scientifically justified. This retains information and makes alternative groupings possible.

Missing values and special codes

Missingness should be represented consistently. Text such as NA, N/A, missing, -, . and blank cells mixed within one column create avoidable cleaning problems. Spreadsheet data-organisation guidance recommends consistent missing-value representation and one type of information per column.[3]

If a system uses numeric special codes such as 9, 99 or 999, document them and ensure they cannot be mistaken for legitimate values. Prefer separate labelled status fields or software-native missing representations when possible, especially for continuous measurements.

DECISION GUIDE

Before assigning a special missing code

1. Can the system represent missingness directly?
2. Do different reasons for missingness need to be distinguished?
3. Could the proposed code fall within the valid measurement range?
4. Will statistical software treat the code as data unless explicitly recoded?
5. Is the same convention used consistently across the project?

Units and measurement metadata

Never rely on memory for units. Record whether temperature is °C, creatinine is μmol/L or mg/dL, weight is kilograms or pounds, and time is days or weeks. In multisite or international research, unit harmonisation should be planned before analysis.

If measurement devices matter, metadata may also include manufacturer, model, calibration information, assay method or laboratory reference range. The amount of detail should be proportional to the research question and potential for measurement differences.

Metadata and provenance

Provenance describes where data came from and what happened to them. Relevant provenance can include:

• source system or instrument;
• collection date and time;
• collector or authorised data originator where appropriate;
• instrument or form version;
• original unit;
• transformation or recoding;
• query and correction history; and
• dataset release from which an analytical variable was produced.

Electronic-source guidance emphasises traceability of data origin and changes.[4] Not every provenance element belongs in the analysis table; some may reside in system logs, metadata files or controlled documentation.

Derived variables

Every important derived variable should have a reproducible specification. For example:

age_delivery_years = (delivery_date - date_of_birth) / 365.25

But a formula alone may be insufficient. Define what happens when the birth date is partial, which date takes precedence if delivery dates conflict, and whether age is truncated or kept continuous.

For questionnaire scores, document included items, reverse scoring, missing-item rules, valid range and instrument version. Prefer generating derived variables through code or controlled calculations rather than manual spreadsheet editing.

WORKED EXAMPLE

Documenting an access-burden score

The study’s six-item access-burden scale ranges from 0 to 24.

The dictionary records:

• items ab1 through ab6, each coded 0–4;
• the wording and response labels for instrument version 1.0;
• item ab4 is reverse scored before summation;
• total score ab_total is calculated only when at least five items are non-missing;
• if one item is missing, the prespecified scoring rule is applied and flagged by ab_prorated = 1;
• if more than one item is missing, ab_total remains missing; and
• higher scores represent greater access burden.

The scoring metadata allow an analyst to reproduce the total without relying on undocumented institutional memory.

Metadata for repeated and longitudinal data

Repeated data need identifiers for participant, time point and occurrence. A long-format dataset may contain one row per participant per visit; a repeating-event export may also require an instance number. Document which fields together form a unique key.

Do not encode all time information only in variable names such as glucose_wk6 and glucose_wk12 if the study contains irregular timing that needs actual dates. Preserve observed dates or elapsed time where analytically relevant.

Data dictionary and EDC metadata

Metadata-driven platforms such as REDCap can export field definitions and project dictionaries.[5] This is valuable, but the automatically generated dictionary may not contain all scientific definitions, derivation decisions or provenance required for future reuse. Maintain study-level documentation in addition to system metadata.

When production changes occur, save dated or versioned metadata exports. A later analyst should be able to determine which field definitions and branching rules were in force during a particular period.

PROTOCOL CHECKLIST

Data-dictionary minimum set

• Stable variable name and readable label.
• Operational definition.
• Data type and permitted range or categories.
• Code labels for categorical fields.
• Unit and measurement scale.
• Source and time point.
• Missing and not-applicable rules.
• Derivation or scoring algorithm where applicable.
• Instrument/form version.
• Notes on changes, provenance or known limitations.

FAIR principles and sensitive health data

FAIR stands for Findable, Accessible, Interoperable and Reusable.[2] “Accessible” does not mean unrestricted public access. Sensitive participant-level health data may appropriately require authentication, data-use agreements, controlled repositories or other governance protections. Rich metadata can make a dataset discoverable and understandable even when access to the data themselves is restricted.

A good data-management plan therefore aligns metadata, sharing expectations, privacy and long-term stewardship.[6]

Versioning and codebooks

Assign versions to important dictionaries and codebooks. When coding changes, document:

• old and new coding;
• effective date;
• reason for change;
• whether historical data were migrated;
• scripts used for migration; and
• implications for prior analyses.

Do not silently reuse a code with a different meaning. If code 3 originally meant “Other” and later means “Unknown”, downstream datasets become uninterpretable.

Interface with analysis

The analysis team should review the dictionary before data collection is final. They can identify variables that need explicit reference categories, continuous measurements stored as text, repeated records without unique keys, or derived outcomes that cannot be reconstructed.

A documented dictionary also supports reproducible data cleaning because code can validate the observed dataset against expected types, ranges and levels. The Biostatistics data-management lectures extend this downstream perspective.

Machine-readable metadata and semantic interoperabilityMMed extension

For large collaborations, a human-readable spreadsheet dictionary may be insufficient. Machine-readable schemas, controlled terminologies and standard data models can support automated validation and interoperability across systems. The appropriate standard depends on discipline and purpose.

Do not adopt a complex standard solely because it exists. Map variables only when the target concept is genuinely equivalent, and retain study-specific provenance where information would otherwise be lost. Interoperability should preserve meaning rather than merely make fields technically exchangeable.

Designing a dictionary for humans and software

A useful dictionary serves two audiences. Researchers need readable definitions and clinical context; software needs predictable names, types, permitted values and relationships. Where feasible, maintain one authoritative specification from which both human-readable documentation and machine checks can be produced.

For example, a dictionary can store the expected type numeric, unit mmol/L, minimum plausible value, maximum plausible value and missing policy. A validation script can compare the actual dataset with this specification and report unexpected character values or categories automatically.

Document analytical recoding separately from collection coding

Collection categories should reflect what was measured. Analytical categories reflect a particular research question. Keep these layers distinct. If educational level is collected in five categories and one analysis combines them into two, document a derived variable such as education_group_analysis1 rather than overwriting the original field.

This separation helps when another manuscript needs a different grouping. It also makes it clear which transformations were analysis choices rather than features of the original measurement.

Define unique keys and relational structure

For multi-table or repeated datasets, document what uniquely identifies a record. A participant table may use study_id; a visit table may require study_id + visit_code; a repeating contact table may require study_id + contact_instance.

Failure to define keys can create duplicate merges, accidental many-to-many joins and inflated participant counts. These errors are particularly dangerous because they may produce plausible-looking datasets without obvious missing values.

Preserve provenance for external data

When importing laboratory, registry or administrative data, record the source system, extraction date, file version and linkage process. If external coding systems change over time, retain the original code and document the mapping to study categories.

Do not assume a variable with the same label has the same meaning across systems. “Admission date”, for example, may represent registration, ward arrival or billing episode depending on source.

Metadata for reproducible sharing

A shared dataset should travel with enough documentation for an authorised secondary user to understand population, time frame, variable definitions, coding, missingness, derivations and known limitations. A readme file can describe dataset-level context while the data dictionary covers variable-level details.

The aim is not documentation for its own sake. Good metadata reduce the risk that future users unknowingly reverse category codes, mix units or infer that structural missingness represents participant refusal.

APPRAISAL CHECKLIST

Appraise a data dictionary and metadata package

• Are collection variables and analytical derivations distinguished?
• Are codes and category labels unambiguous?
• Are units and measurement time points explicit?
• Are unique keys defined for every table or repeated structure?
• Is external-data provenance recorded?
• Can a new analyst reproduce important scores and derived outcomes?
• Are version changes and migrations documented?
• Does the documentation explain known limitations as well as ideal definitions?

Treat the dictionary as a controlled study document

The dictionary should evolve through explicit versions rather than private copies maintained by different analysts. Assign responsibility for approving important definition changes and communicate them to form designers, data managers and analysts. If the same variable appears in several datasets, use one authoritative definition wherever possible.

A change log should describe not only what changed but whether previously collected records were affected. This prevents a later analyst from assuming that one code or definition applied uniformly throughout the study when it did not.

Common mistakes

Knowledge check

What is the difference between data and metadata?

Answer and reasoning

Data are the observed or derived values themselves. Metadata describe those values and their context—for example definitions, units, codes, provenance, instrument versions and transformations.

Why is coding clinic A, B and C as 1, 2 and 3 not an interval measurement?

Answer and reasoning

The numbers are category labels. Their numerical spacing has no substantive meaning unless the underlying construct is genuinely ordered and measured that way.

Should a derived score replace its component items in the stored dataset?

Answer and reasoning

Usually no. Retaining the components allows the score to be checked, reproduced or recalculated if the scoring rule changes.

Can a confidential clinical dataset still follow FAIR principles?

Answer and reasoning

Yes. It can have findable and reusable metadata and a documented controlled-access process. FAIR accessibility can be conditional and governed; it does not require unrestricted public access.

Key take-home messages

SUMMARY

Key take-home messages

• A data dictionary preserves the meaning of variables across collection, cleaning, analysis and reuse.
• Document codes, units, missing states, provenance and derivation rules explicitly.
• Keep categorical codes distinct from the measurement concepts they represent.
• Retain raw components and versioned metadata so important derived variables are reproducible.
• Use EDC-generated metadata as a foundation, not a substitute for scientific documentation.
• Apply FAIR principles together with appropriate confidentiality and access governance.

References

1. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH harmonised guideline: Good Clinical Practice E6(R3). Step 4. 6 January 2025. https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf
2. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. https://doi.org/10.1038/sdata.2016.18
3. Broman KW, Woo KH. Data organization in spreadsheets. Am Stat. 2018;72(1):2–10. https://doi.org/10.1080/00031305.2017.1375989
4. US Food and Drug Administration. Guidance for industry: Electronic source data in clinical investigations. Silver Spring (MD): FDA; 2013. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/electronic-source-data-clinical-investigations
5. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. https://doi.org/10.1016/j.jbi.2008.08.010
6. Michener WK. Ten simple rules for creating a good data management plan. PLoS Comput Biol. 2015;11(10):e1004525. https://doi.org/10.1371/journal.pcbi.1004525