Validity of Measurements and Instruments

Overview

Validity is the degree to which evidence and theory support the interpretations of measurements for their intended uses. It is not a permanent certificate owned by a questionnaire, device or score. An instrument may support one interpretation in adults attending specialist clinics but lack evidence for adolescents, another language, remote administration or individual clinical decision-making [1–3].

Modern validity thinking therefore begins with an interpretive argument: what does the score or classification represent, for whom, under which conditions, and for what decision? Evidence is then assembled to examine whether the interpretation is defensible. Content relevance, response processes, internal structure, relationships with other variables and consequences of use can all contribute [1].

Health-measurement frameworks commonly classify measurement properties into validity, reliability and responsiveness domains. Within validity, content validity is foundational because statistical associations cannot compensate for missing or irrelevant content [2–4].

Why this matters

Invalid measurement can distort every later stage of research. An instrument may omit an important dimension, contain culturally inappropriate items, use an unsupported scoring structure or correlate with another measure for reasons unrelated to the intended construct. Calling it “validated” can hide these limitations and encourage use outside the populations and purposes for which evidence exists.

Validity evidence should be planned rather than assembled opportunistically after data collection. A list of correlations is not a validity strategy. Researchers should state expected directions and magnitudes before examining the data, select comparison measures for clear conceptual reasons and explain what each result can and cannot support.

Validity begins with the intended interpretation

A validity plan should specify:

• the construct and its boundaries;
• the target population and context;
• the administration mode and language;
• the score or classification to be interpreted;
• the intended use, such as group comparison, screening or monitoring change; and
• plausible alternative explanations for the observed score.

Content validity

Content validity concerns whether the instrument’s content is relevant, comprehensive and comprehensible for the construct, target population and context of use [3,4]. It is assessed through construct definition, literature or theory, input from people with lived experience, and review by experts who understand the content and measurement purpose.

Relevance

Each item should contribute to the intended construct and be appropriate to the population and application. An item about paid parking may be irrelevant in settings where transport is provided, while a question about childcare may be essential for postpartum attendance.

Comprehensiveness

The item set should cover the important aspects of the construct without major omissions. A score called access burden that includes only travel distance under-represents appointment availability, cost, service hours and caregiving responsibilities.

Comprehensibility

Participants should understand items, instructions, recall periods and response options as intended. Expert review alone cannot establish comprehensibility; cognitive interviewing with target respondents provides direct evidence about response processes.

Face validity is the judgement that an instrument appears appropriate or acceptable. It can help identify obvious problems and influence engagement, but it is usually treated as a limited aspect of content appraisal rather than a sufficient validity property.

Illustrative content-validity matrix

illustrative

Table 1. Linking domains, items and evidence sources

Domain	Proposed content	Participant evidence	Expert evidence
Service access	appointment availability and opening hours	interviews describe repeated failed booking attempts	health-services experts confirm pathway relevance
Practical burden	transport, cost and childcare	women identify these as distinct barriers	clinicians confirm they are not clinical-severity items
Information	clarity of referral and test instructions	participants report uncertainty about where to attend	diabetes educators review factual accuracy

APPRAISAL CHECKLIST

Appraise content validity

• Was the construct defined before item selection?
• Were all important domains identified?
• Were members of the target population involved?
• Did experts have relevant content and measurement expertise?
• Were relevance, comprehensiveness and comprehensibility assessed separately?
• Were revisions and unresolved disagreements documented?

Evidence based on response processes

Response-process evidence examines how respondents or observers understand and produce the measurement. For questionnaires, cognitive interviews explore comprehension, retrieval, judgement and response mapping. For observer ratings, training exercises can reveal whether assessors apply criteria as intended. For performance tests, researchers examine whether participants understand the task and whether unintended skills affect performance.

A correct-looking answer does not prove the intended process. A participant may select “no difficulty” because someone else arranged the appointment, not because the service was accessible. This distinction may require additional items or revised wording.

Structural validity

Structural validity concerns whether the dimensional structure of the scores reflects the construct. If access burden is intended to contain service, cost and caregiving domains, analyses should examine whether the scoring structure is compatible with that model. Exploratory factor analysis can help investigate uncertain structure; confirmatory factor analysis evaluates a prespecified model. Item response theory or Rasch-family models offer additional ways to examine item functioning [2,5].

Internal consistency should be interpreted only after the relevant items are shown to form a sufficiently coherent unidimensional scale or subscale. A high coefficient across multidimensional items may reflect many correlated questions rather than one valid total score.

Construct validity through hypotheses testing

When no definitive criterion exists, validity evidence often comes from relationships with other variables. The key is to formulate a priori hypotheses based on theory.

Convergent evidence

Scores should relate more strongly to measures of closely related constructs. Access burden might correlate moderately with reported difficulty attending appointments but should not be expected to correlate perfectly because the measures capture different perspectives.

Discriminant evidence

Scores should have weaker relationships with distinct constructs. Access burden should not show a strong association with an unrelated preference such as satisfaction with ward meals, after considering plausible common causes.

Known-groups evidence

Groups expected to differ should show differences in the predicted direction. Participants living far from services or lacking transport may report greater burden. Such comparisons support validity only when the grouping variable is credible and the hypothesis was specified in advance.

Hypothesis specificity

A useful hypothesis states the direction and, where defensible, relative magnitude. “The score will correlate significantly with other measures” is weak because almost any result can be rationalised. COSMIN guidance emphasises explicit hypotheses and an overall pattern of findings [2,3].

DECISION GUIDE

Build a construct-validity hypothesis

1. What aspect of the construct should relate to the comparison measure?
2. Should the relationship be positive or negative?
3. Should it be weak, moderate or strong relative to other comparisons?
4. Could shared method, timing or wording inflate the relationship?
5. What result would challenge the intended interpretation?

Criterion validity

Criterion validity compares a measurement with an adequate external criterion that represents the target construct. A genuine criterion is uncommon for many patient-reported constructs. There is no error-free gold standard for pain, trust or access burden.

Laboratory methods and diagnoses may have reference procedures, but the criterion must be appropriate to the same target, time point and population. Comparing a brief screening score with a clinician diagnosis can estimate classification performance, yet disagreement may arise from error in either method or from different target definitions.

Cross-cultural validity and measurement invariance

A translated or adapted instrument should not be assumed to retain the same structure and interpretation. Cross-cultural validity or measurement invariance concerns whether item performance and score relationships are comparable across language or cultural groups [2]. Differences may reflect translation, unfamiliar examples, response styles or genuine differences in how the construct is expressed.

Evidence may include qualitative testing, factor-structure comparisons and differential item functioning. A total-score comparison across groups is difficult to interpret when items do not function equivalently.

Responsiveness as longitudinal validity

When an instrument is used to measure change, the validity question becomes whether changes in score reflect changes in the intended construct. Responsiveness should be evaluated with a priori hypotheses about change and relevant comparison measures. A significant before-after difference alone does not prove responsiveness because natural history, regression to the mean, intervention effects on another domain or measurement drift could explain the change.

Consequences and fairness

Validity also includes attention to the consequences of measurement use. A screening threshold may determine referral, eligibility or resource allocation. Researchers should examine false-positive and false-negative consequences, subgroup performance, accessibility and whether the measure reinforces structural disadvantage.

Threats to validity evidence

Validity studies can be misleading even when they use familiar analyses. Common-method variance can inflate relationships when the new instrument and comparator are completed at the same sitting, use similar wording and share the same response scale. A strong correlation may then reflect response style or item overlap rather than the intended construct.

Range restriction can weaken expected relationships. A score evaluated only among participants with uniformly good access may show little association with appointment difficulty because the relevant variation is absent. Conversely, selecting extreme groups can exaggerate known-groups differences and may not reflect routine performance.

Criterion contamination occurs when the new measure contributes to the criterion or when assessors know the new score while assigning the criterion. Independence is especially important in observer ratings and diagnostic classifications. When a criterion is based partly on the same questionnaire items, agreement is built into the design.

Post hoc hypothesis construction is another threat. Researchers may examine many correlations and then describe the observed pattern as expected. Scale-development guidance recommends defining the construct and expected relationships before analysis, while regulatory guidance similarly links evidence to a prespecified concept of interest and context of use [6,7].

Validity evidence can also fail to transport. A relationship established in one healthcare system may depend on service organisation, language or eligibility rules. Replication across settings should examine whether the same interpretation remains defensible rather than treating a previous coefficient as a universal benchmark.

Interpreting contradictory or negative evidence

Not every hypothesis will be supported. A weak relationship may indicate poor measurement, but it may also reflect an unsuitable comparator, restricted range, timing mismatch or a faulty theoretical expectation. The response should be diagnostic rather than defensive: examine which part of the interpretive argument failed and whether revision, additional evidence or a narrower claim is required.

Validity decisions should not be reduced to a percentage of hypotheses confirmed without considering their importance. Failure of a central content or structural claim may outweigh several minor supporting correlations. Conversely, one unexpected association should not automatically invalidate an otherwise coherent programme. Report the complete pattern, uncertainty and any changes to the intended interpretation.

A cumulative evidence plan

PROTOCOL CHECKLIST

Specify a validity-evidence programme

• intended score interpretation and use;
• content-validity methods and stakeholder groups;
• response-process evaluation;
• prespecified dimensional structure;
• hypotheses for convergent, discriminant and known-groups evidence;
• proposed external criteria and their limitations;
• cross-cultural or subgroup comparisons;
• responsiveness hypotheses when change is measured; and
• decision rules for revising, retaining or rejecting the instrument.

The unitary view of validityMMed extension

Historical labels such as content, construct and criterion validity can suggest separate types of validity. Contemporary standards treat validity as one integrated argument supported by multiple sources of evidence. The categories remain useful for organising studies, but no single coefficient establishes validity. Evidence should be judged for coherence, relevance and threats to the intended interpretation [1].

Interface with statistical analysis

Validity analyses depend on the claim. Factor analysis addresses internal structure; correlations and group comparisons address prespecified relationships; classification measures address agreement with a criterion; and longitudinal models examine expected change. P-values alone are insufficient. Estimates, uncertainty and the pattern across hypotheses are more informative.

Researchers should avoid using the same sample for unrestricted item development and strong confirmatory claims without acknowledging overfitting. Independent or split-sample confirmation is preferable when feasible, especially for new scales.

Worked example

WORKED EXAMPLE

Building validity evidence for an access-burden score

Intended interpretation: Higher scores indicate greater practical and service-related difficulty completing postpartum testing.

Content evidence: Interviews with women identify transport, childcare, appointment availability, working hours and cost. Experts review relevance and coverage. Cognitive interviews lead to removal of an item interpreted as clinical anxiety rather than access.

Structural hypothesis: Service-access and practical-burden items form two correlated domains rather than one undifferentiated total score.

Construct hypotheses: Scores should be higher among participants reporting failed appointment attempts and lack of transport; associations with general health literacy should be weaker than associations with appointment difficulty.

Criterion evidence: No error-free criterion exists for perceived burden. Test completion is an important outcome but is not the construct itself, because a participant may complete testing despite high burden.

Conclusion: The validity argument combines content, response-process, structural and relationship evidence. Correlation with completion is informative but not decisive.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

What is being validated: the instrument or the interpretation?

Answer and reasoning

The interpretation and intended use of the resulting measurement are evaluated. The instrument is part of that argument, but evidence does not automatically transfer to every population, language, mode or decision.

Why is content validity foundational for a questionnaire?

Answer and reasoning

If important domains are missing, irrelevant items dominate or respondents do not understand the content as intended, statistical modelling cannot reconstruct the intended construct. Relevance, comprehensiveness and comprehensibility must therefore be addressed early.

Does a strong correlation with test completion prove that an access-burden score is valid?

Answer and reasoning

No. Completion is influenced by many factors and is not identical to perceived burden. The association may support one hypothesis, but content, response-process, structural and other construct evidence are still required.

What makes a construct-validity hypothesis stronger than 'the correlation will be significant'?

Answer and reasoning

A stronger hypothesis specifies the expected direction, approximate strength or comparative pattern and explains why the relationship should occur. It also identifies results that would challenge the interpretation.

Key take-home messages

SUMMARY

Key take-home messages

• Validity is a cumulative argument supporting a defined interpretation and use.
• Content validity examines relevance, comprehensiveness and comprehensibility.
• Structural and construct evidence should follow prespecified theory and hypotheses.
• A criterion must represent the same target and should not be assumed error-free.
• Reliability and internal consistency cannot substitute for validity.

References

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.
2. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–745. https://doi.org/10.1016/j.jclinepi.2010.02.006
3. Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, et al. COSMIN manual for systematic reviews of PROMs. Version 2.0. Amsterdam: COSMIN; 2024. https://www.cosmin.nl/wp-content/uploads/COSMIN-manual-V2_final.pdf
4. Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–1170. https://doi.org/10.1007/s11136-018-1829-0
5. de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press; 2011. https://doi.org/10.1017/CBO9780511996214
6. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149. https://doi.org/10.3389/fpubh.2018.00149
7. US Food and Drug Administration. Patient-focused drug development: selecting, developing, or modifying fit-for-purpose clinical outcome assessments. Silver Spring: FDA; 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-selecting-developing-or-modifying-fit-purpose-clinical-outcome