Content and Construct Validity

Overview

Validity concerns the degree to which evidence supports the interpretation of scores for an intended purpose.

It is not a permanent label attached to an instrument.

The same questionnaire may be valid for:

• one population;
• one language;
• one use;
• one score interpretation;

and inadequate for another.

COSMIN organises validity evidence into:

• content validity;
• structural validity;
• hypotheses testing for construct validity;
• cross-cultural validity or measurement invariance;
• criterion validity when a defensible gold standard exists [1].

Why this matters

A reliable instrument can measure the wrong construct consistently.

A high Cronbach alpha or ICC does not establish validity.

Content validity is foundational because missing or irrelevant content cannot be repaired by sophisticated statistical modelling.

Construct validity requires a network of prespecified hypotheses rather than one favourable correlation.

When to evaluate validity

Evaluate content validity:

• during instrument development;
• during translation or adaptation;
• before relying on internal consistency.

Evaluate construct validity whenever score interpretation depends on a latent construct without a true gold standard.

Validity evidence should accumulate across studies. Messick’s unified validity framework emphasises evidence supporting score interpretation rather than a permanent instrument label [2].

Research question and variables

Content-validity questions include:

• Are items relevant?
• Is important content missing?
• Are instructions, items and response options understandable?

Construct-validity hypotheses include:

• strong correlation with closely related constructs;
• moderate correlation with related but distinct constructs;
• weak correlation with unrelated constructs;
• expected differences between known groups;
• expected factor structure.

Data structure

Validity evidence sources

simulated data

Dataset	Variables
Expert content review	Item, expert, relevance rating 1–4
Patient cognitive interviews	Comprehension, interpretation, missing content
Construct study	Instrument score and comparator measures
Known-groups study	Instrument score and prespecified group
Structural study	Item-level responses for factor analysis

Quantitative indices should not replace qualitative patient and expert feedback.

Statistical assumptions

Content validity

COSMIN defines content validity through:

• relevance;
• comprehensiveness;
• comprehensibility [3].

Patients are essential because experts may overlook wording, burden or missing lived-experience content.

Item content validity index

If experts rate relevance from 1 to 4 and ratings 3–4 count as relevant:

Scale CVI average

$$S\text{-}CVI/Ave = \frac{ \sum I\text{-}CVI }{ \text{Number of items} }$$

Universal agreement

$$S\text{-}CVI/UA = \frac{ \text{Items rated relevant by every expert} }{ \text{Number of items} }$$

Chance agreement and panel size should be considered.

Worked example: content validity

Construct validity

Construct validity is evaluated through patterns predicted by theory.

Convergent validity

The instrument should correlate strongly with measures of similar constructs.

Discriminant validity

Correlations with distinct constructs should be lower.

Known-groups validity

Groups expected to differ should show differences in the predicted direction and magnitude.

Structural validity

The item response structure should support the proposed dimensional model.

Exploratory factor analysis explores potential dimensions.

Confirmatory factor analysis evaluates a prespecified model.

Worked example: construct validity

Worked analysis in R

content_data <- read.csv(
  "data/part-12-content-validity-data.csv"
)

validity_data <- read.csv(
  "data/part-12-construct-validity-data.csv"
)

content_data$relevant <- as.integer(
  content_data$relevance_rating >= 3
)

item_cvi <- aggregate(
  relevant ~ item,
  data = content_data,
  FUN = mean
)

scale_cvi_average <- mean(
  item_cvi$relevant
)

universal_item <- aggregate(
  relevant ~ item,
  data = content_data,
  FUN = min
)

scale_cvi_universal <- mean(
  universal_item$relevant
)

construct_correlations <- cor(
  validity_data[
    c(
      "instrument_total",
      "related_construct",
      "moderately_related",
      "unrelated_construct"
    )
  ]
)

known_group_test <- t.test(
  instrument_total ~ known_group,
  data = validity_data,
  var.equal = FALSE
)

item_cvi
scale_cvi_average
scale_cvi_universal
construct_correlations
known_group_test

Selected analysis output

I-CVI
Items 1–8 and 10 = 1.00
Item 9 = 0.00

S-CVI/Ave = 0.90
S-CVI/UA = 0.90

Construct correlations
Related = 0.937
Moderately related = 0.810
Unrelated = 0.152

Known-group mean difference = 4.973
Welch t = 4.640
p < 0.001

Visualising construct hypotheses

The figure should display confidence intervals in a publication analysis.

Cognitive interviews

Content validation should include cognitive interviews examining:

• how participants interpret each item;
• whether recall periods are workable;
• whether response options fit;
• whether important content is missing;
• whether wording is offensive or burdensome.

The 2025 COSMIN content-validity guidance emphasises relevance, comprehensiveness and comprehensibility [4].

Structural validity

Factor analysis should follow the scoring model.

For ordinal items, estimators based on polychoric correlations may be appropriate.

Report:

• factor model;
• estimator;
• loading pattern;
• residual correlations;
• fit indices;
• model modifications;
• cross-validation.

A good-fitting model does not compensate for poor content validity.

Cross-cultural validity and invariance

Translated or adapted instruments should preserve:

• meaning;
• item function;
• response interpretation;
• score comparability.

Measurement invariance can be assessed across:

• language;
• sex;
• age;
• clinical group;
• time.

Without sufficient invariance, group score comparisons may be biased.

Criterion validity

Criterion validity requires a defensible gold standard.

Many constructs such as quality of life, fatigue and health literacy have no gold standard.

Correlation with another imperfect questionnaire is construct validity evidence, not criterion validity.

Hypothesis testing

A strong hypothesis states:

• comparator;
• expected direction;
• expected magnitude;
• subgroup difference;
• rationale.

Example:

The new fatigue score will correlate at least 0.60 with an established fatigue instrument and below 0.30 with social desirability.

Post hoc reinterpretation of unexpected correlations weakens evidence.

Interpreting validity evidence

Presenting the results

Suggested reporting

Report:

• construct definition;
• intended population and use;
• development process;
• patient and expert involvement;
• content-validity methods;
• item-level results;
• prespecified construct hypotheses;
• structural model and estimator;
• cross-cultural testing;
• all supportive and contradictory evidence.

COSMIN treats content validity as a central measurement property when selecting instruments [1], [3], [4].

Common mistakes

Knowledge check

What are the three main aspects of content validity?

Answer and reasoning

Relevance, comprehensiveness and comprehensibility.

Does a high S-CVI/Ave excuse one irrelevant item?

Answer and reasoning

No. Item-level problems can be concealed by the average.

What makes construct-validity evidence strong?

Answer and reasoning

A prespecified pattern of relationships and group differences supported across studies.

Why is correlation with another questionnaire not automatically criterion validity?

Answer and reasoning

The comparator is usually another imperfect construct measure rather than a gold standard.

What does measurement invariance support?

Answer and reasoning

Comparable score meaning across groups, languages or time points.

Key take-home messages

References

1. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties. Qual Life Res. 2010;19(4):539–549. https://doi.org/10.1007/s11136-010-9606-8
2. Messick S. Validity of psychological assessment. Am Psychol. 1995;50(9):741–749. https://doi.org/10.1037/0003-066X.50.9.741
3. Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures. Qual Life Res. 2018;27(5):1159–1170. https://doi.org/10.1007/s11136-018-1829-0
4. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. Content validity: judging the relevance, comprehensiveness, and comprehensibility of instruments’ content. J Clin Epidemiol. 2025;183:111759. https://doi.org/10.1016/j.jclinepi.2025.111759
5. Mokkink LB, Prinsen CAC, Bouter LM, de Vet HCW, Terwee CB. The COnsensus-based Standards for the selection of health Measurement INstruments and how to select an outcome measurement instrument. Braz J Phys Ther. 2016;20(2):105–113. https://doi.org/10.1590/bjpt-rbf.2014.0143