Reliability and Reproducibility

Overview

Repeated measurement rarely produces identical results. Participants change, observers differ, instruments fluctuate and administration conditions vary. Reliability and agreement studies examine whether this variation is small enough for the intended use and where it arises [1–3].

Reliability concerns the ability of a measurement to distinguish people or objects despite measurement error. It is commonly expressed as the proportion of total observed variation attributable to real differences between the measured units. Agreement concerns the closeness of repeated values or ratings in the original scale. Reproducibility is a broader term describing consistency when relevant conditions change, such as observer, device, site, day or laboratory.

These concepts are related but not interchangeable. A measure can show high reliability in a heterogeneous sample while repeated values differ too much for individual monitoring. Conversely, repeated values may be close in a homogeneous group, yet the reliability coefficient is modest because participants differ little from one another.

Why this matters

Measurement inconsistency can dilute associations, obscure change and misclassify participants. It may also create false confidence when researchers report one familiar coefficient without describing the repeat conditions or the intended interpretation. The choice of reliability design and statistic should follow the measurement process.

Internal consistency, test-retest reliability and inter-rater reliability answer different questions. A high internal-consistency coefficient does not show that scores are stable over time, that raters agree or that the construct is valid. Reliability evidence should be evaluated alongside agreement, validity, responsiveness and feasibility.

Sources of variation

An observed value can be thought of as containing the target signal plus variation arising from multiple sources:

• temporary fluctuations within the participant;
• item wording and response processes;
• observer interpretation;
• device or laboratory variation;
• administration mode and environment;
• data-entry or scoring errors; and
• genuine change between measurement occasions.

The repeatability study must control or deliberately vary these conditions according to the intended use. If the instrument will be used by many nurses across clinics, a study involving one highly trained rater under ideal conditions provides incomplete evidence.

Test-retest reliability

Test-retest reliability evaluates whether scores are consistent across occasions when the construct is expected to remain stable. The interval should be long enough to reduce recall but short enough to avoid genuine change. No universal interval is correct. Symptom intensity may change within hours, while a stable knowledge construct may justify a longer interval.

Researchers should document evidence that participants were stable. This may include a transition question, clinical criteria or absence of an intervention. A transition question is itself imperfect and should not be treated as an error-free standard.

Repeated administration can create learning, sensitisation or memory effects. Changes in mode, interviewer or setting can also become part of the observed difference unless deliberately studied.

Inter-rater and intra-rater reliability

Inter-rater reliability examines variation between observers rating the same units. Intra-rater reliability examines whether the same observer produces consistent ratings on separate occasions. Appropriate designs require:

• the same cases or specimens to be rated;
• independent ratings when agreement is the target;
• blinding to previous or other raters’ results where feasible;
• representative case difficulty and category prevalence;
• clear criteria and training; and
• a planned method for handling indeterminate ratings.

Using only easy cases can overstate performance. Discussing each case until consensus produces a final classification but does not estimate independent rater reliability.

Illustrative reliability-study matrix

illustrative

Table 1. Matching the question to the repeatability design

Question	Repeat condition	Example outcome
Are access-burden scores stable?	same participant, two occasions, no expected change	continuous domain score
Do research assistants classify referral completeness consistently?	same record, independent raters	complete/partial/absent
Does one reviewer reproduce a classification?	same record, same rater after washout	ordinal quality category
Are questionnaire items coherent within a domain?	one administration, relationships among items	item covariance and domain score

Internal consistency

Internal consistency concerns the interrelatedness of items intended to measure the same construct [1]. Cronbach’s alpha is commonly reported, but its interpretation requires caution. Alpha is influenced by the number of items, average inter-item covariance and assumptions about the measurement model. A high alpha does not establish unidimensionality, validity or absence of redundant items.

The dimensional structure should be examined before calculating one coefficient for a total score. If the instrument contains service access, financial burden and caregiving domains, a single alpha across all items may be misleading. Domain-specific coefficients may be appropriate after structural evidence supports the domains.

Omega and model-based reliability estimates can be useful when their assumptions match the scale, but replacing alpha with another coefficient does not solve poor content or multidimensionality.

Reliability statistics for continuous measurements

Intraclass correlation coefficient

The intraclass correlation coefficient (ICC) is used for continuous or approximately continuous measurements when repeated values are organised by participants and raters or occasions. Multiple ICC forms exist. Selection depends on:

• whether raters are fixed or considered a sample from a wider population;
• whether the analysis concerns a single rating or an average of several ratings; and
• whether consistency or absolute agreement is required [4].

The ICC form, model, unit and definition should be reported explicitly with a confidence interval. Reporting “ICC = 0.82” without these details is incomplete.

An ICC is a relative statistic. It compares between-participant variation with measurement-error variation. The same measurement error can produce a higher ICC in a heterogeneous sample than in a homogeneous one.

Agreement in original units

For repeated continuous measurements, the standard error of measurement, within-person standard deviation and limits of agreement describe absolute error. Bland-Altman analysis examines the distribution of differences against the magnitude of measurement and can reveal systematic bias or changing variability [5].

Correlation is not agreement. Two methods can correlate perfectly when one is consistently higher than the other.

Reliability statistics for categorical and ordinal ratings

Percentage agreement is transparent but does not adjust for agreement expected under the chosen chance model. Kappa statistics are often used for nominal categories, with weighted versions for ordered categories. Kappa depends on category prevalence and rater marginal distributions; a low kappa can occur despite high raw agreement when one category dominates [6].

The statistic should match the design and decision. Report the cross-tabulation or category-specific agreement where possible rather than hiding all information in one coefficient. For clinically important categories, positive and negative agreement may be more informative than an overall value.

DECISION GUIDE

Choose the reliability or agreement summary

1. Is the measurement continuous, ordinal or nominal?
2. Is the question relative ranking or absolute closeness?
3. Are ratings from one rater, a fixed set or interchangeable raters?
4. Will decisions use a single rating or an average?
5. Is the sample representative of the intended range and category prevalence?
6. Which error magnitude would be acceptable for the intended use?

Design details that materially affect estimates

Participant and case spectrum

Reliability studies should include the range of values, disease severity and case difficulty expected in practice. A heterogeneous spectrum can raise relative reliability, while a restricted spectrum can lower it even when absolute error is unchanged. Report the sample distribution so readers can understand the context of the coefficient.

Number and selection of raters

If future users are interchangeable clinicians, the study should include a defensible sample of clinicians rather than only the instrument developer. If one specialist will make all future decisions, fixed-rater performance may be relevant, but generalisation should be limited accordingly. The distinction affects ICC model selection and interpretation.

Stable versus changing participants

Test-retest designs should separate genuine change from measurement error. Excluding every participant who reports any change can create a highly selected sample, while ignoring change can understate reliability. Describe the stability criterion, the number excluded and whether excluded participants differed from those analysed.

Missing repeat measurements

Participants who return for retesting may be healthier, more engaged or easier to measure. Complete-case reliability estimates can therefore be unrepresentative. Report recruitment, completion and reasons for missing repeats. Consider whether the repeat procedure itself caused burden or selective attrition.

Training and operational use

Reliability obtained after intensive central training may not reflect routine implementation. A useful evaluation can distinguish performance immediately after training from performance after several months, or compare routine users with expert raters. Health-measurement guidance stresses aligning the repeatability study with the actual use conditions rather than an idealised laboratory setting [7].

Individual change

When a score monitors one person, the relevant question is whether observed change exceeds measurement error. The standard error of measurement and smallest detectable change can be expressed in score units. These quantities should not be relabelled as minimally important change: statistical detectability and clinical importance are different concepts.

Maintaining reproducibility after the study

A one-time reliability study does not guarantee stable performance during a long project. Staff turnover, equipment replacement, software updates and changing case mix can alter measurement. Quality assurance may include periodic duplicate ratings, calibration checks, blinded reference cases and review of disagreement patterns.

Monitoring should focus on clinically or methodologically important drift rather than repeatedly testing every measurement. Control charts or summary indicators can show changes in completion, error or rater disagreement over time. When retraining occurs, document whether earlier and later data remain comparable.

Adjudication is useful for obtaining a final study value but should be separated from independent reliability assessment. The protocol should state which cases are adjudicated, who adjudicates them, what information is available and whether the adjudicated result replaces or supplements the original ratings.

For multicentre studies, site differences may reflect training, equipment, population or workflow. Pooling all ratings into one coefficient can hide a poorly performing site. Report site-specific or stratified evidence when the number of observations permits, and investigate systematic differences before attributing them to participants.

Interpreting magnitude

Generic thresholds such as poor, moderate or excellent can provide rough orientation, but acceptability depends on consequences. A measure used to compare group means may tolerate more individual error than a measure used to decide whether one patient receives treatment. Confidence intervals should be considered because a point estimate can conceal substantial uncertainty.

Reliability should be judged against:

• intended use;
• observed error in original units;
• between-person heterogeneity;
• decision threshold;
• consequences of disagreement; and
• alternative available instruments.

PROTOCOL CHECKLIST

Plan a reliability and agreement study

• measurement target and intended use;
• participant or case spectrum;
• raters, devices and occasions to be varied;
• training and standardisation procedures;
• repeat interval and stability criteria;
• blinding and order of assessment;
• selected statistic and exact form;
• confidence-interval method;
• absolute agreement or error summary; and
• criterion for acceptable performance.

Generalizability theoryMMed extension

Classical reliability studies often isolate one source of error at a time. Generalizability theory decomposes variation across several facets, such as participant, rater, occasion and item, and can estimate how reliability changes when the number of raters or items changes. It is valuable for complex assessments but requires a design that crosses or nests the relevant facets appropriately.

Practical interpretation note

Reliability evidence belongs to a defined workflow. A coefficient obtained from duplicate research ratings may support data collection but does not automatically justify clinical use. Before deployment, confirm that routine users, case complexity, equipment and decision consequences resemble the evaluation conditions. When they do not, describe the evidence as preliminary and plan monitoring after implementation.

A decision rule should therefore identify both the acceptable coefficient and acceptable error in meaningful units.

Interface with statistical analysis

Reliability and agreement analyses should include estimates and confidence intervals, not only hypothesis tests. The unit of analysis is usually the participant, specimen, image or record receiving repeated measurements. Repeated ratings are correlated and should remain linked by identifier.

Sample-size planning should reflect the expected coefficient or error, desired precision, number of raters and design. The Biostatistics section should contain detailed calculations; the methodology protocol must specify the target property and exact study structure.

Worked example

WORKED EXAMPLE

Evaluating a referral-documentation checklist

Measurement: Two trained research assistants independently classify 80 postpartum referral records as complete, partial or absent.

Design: Records span all clinics and include straightforward and ambiguous examples. Raters use a version-controlled manual and are blinded to each other’s decisions. Twenty records are re-rated after three weeks to assess intra-rater reliability.

Analysis plan: Report the 3 × 3 cross-tabulation, exact agreement, weighted kappa with confidence interval and category-specific disagreement. The protocol prespecifies that disagreement between complete and absent is more consequential than disagreement between complete and partial.

Interpretation: A high weighted kappa would support reproducible ordering under the study conditions, but the team would still inspect systematic disagreements and whether the sample reflects routine record quality.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

How can an instrument have high reliability but poor agreement?

Answer and reasoning

If participants vary widely, between-person variation can dominate the reliability coefficient even when repeated measurements differ substantially in original units. The measure may preserve ranking while being unsuitable for detecting individual change.

Why is Pearson correlation insufficient for comparing two measurement methods?

Answer and reasoning

Correlation measures linear association, not equality. One method can be consistently higher than another and still correlate perfectly. Agreement requires examining differences and error in the original units.

What must be established before interpreting internal consistency for a total score?

Answer and reasoning

The items should represent a coherent construct and the proposed dimensional structure should be supported. A coefficient calculated across unrelated domains has no clear interpretation as total-score reliability.

Why can kappa be low when percentage agreement is high?

Answer and reasoning

Kappa is affected by category prevalence and the raters’ marginal distributions. When nearly all observations fall in one category, expected chance agreement can be high, reducing kappa despite high raw agreement. Both the table and coefficient should be reported.

Key take-home messages

SUMMARY

Key take-home messages

• Reliability concerns differentiation; agreement concerns closeness in the original scale.
• The repeatability design must match the raters, occasions, devices and conditions of intended use.
• Internal consistency is not evidence of validity or temporal stability.
• ICC and kappa values depend on study design, sample variation and category prevalence.
• Report uncertainty and absolute error, not only a single coefficient.

References

1. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–745. https://doi.org/10.1016/j.jclinepi.2010.02.006
2. de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press; 2011. https://doi.org/10.1017/CBO9780511996214
3. Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96–106. https://doi.org/10.1016/j.jclinepi.2010.03.002
4. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. https://doi.org/10.1016/j.jcm.2016.02.012
5. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
6. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276–282. https://doi.org/10.11613/BM.2012.031
7. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. 5th ed. Oxford: Oxford University Press; 2015. https://doi.org/10.1093/med/9780199685219.001.0001