Selecting an Existing Instrument

Overview

Selecting an existing instrument is usually preferable to developing a new one, but only when the instrument is fit for the study’s construct, population, context and intended use. Selection is not a search for the questionnaire with the highest reliability coefficient. It is a structured comparison of what the instrument measures, how well it performs for the proposed use, and whether it can be implemented lawfully and feasibly [1–4].

The first decision is the outcome or construct, not the instrument. Researchers should define the concept, relevant domains, respondent, recall period, administration mode, score interpretation and intended decision. They can then search for available instruments and evidence about their measurement properties.

COSMIN and related guidance recommend considering conceptual fit, content validity, other measurement properties and feasibility together. A short convenient instrument with poor content is not rescued by good reliability, while a strong instrument may still be unsuitable if the language, licensing or administration burden is incompatible with the study.

Why this matters

An unsuitable instrument can under-represent the construct, impose excessive burden, fail to detect change or produce scores that cannot be compared with relevant studies. Modifying an instrument without permission or evidence may also invalidate its scoring and create legal or ethical problems.

Transparent selection improves reproducibility. Future readers should be able to see which alternatives were considered, what evidence supported the chosen tool, and which limitations remain. “We used a validated questionnaire” is not an adequate justification.

Step 1: define the measurement requirement

Before searching, specify:

• construct and important domains;
• target population and setting;
• respondent or observer;
• intended use, such as description, group comparison, screening or monitoring change;
• administration mode and language;
• relevant time frame or recall period;
• acceptable burden and resources; and
• required score interpretation.

Step 2: identify candidate instruments

Searches can include:

• systematic reviews of measurement instruments;
• COSMIN’s database and search filters;
• core outcome set resources;
• bibliographic databases;
• clinical guidelines and trial protocols;
• instrument repositories and developer websites; and
• consultation with patients, clinicians and measurement experts.

Search terms should combine the construct, population and instrument concepts. Searching only the exact phrase used in the research question can miss instruments using another conceptual label. The literature review should distinguish evidence about the construct from evidence about a specific instrument.

Illustrative candidate-instrument register

illustrative

Table 1. Initial register for access-related instruments

Candidate	Main construct	Population and language	Key concern
Instrument A	general access barriers	adults in urban outpatient care, English	postpartum content not established
Instrument B	treatment burden	chronic disease populations, several languages	construct broader than appointment access
Instrument C	maternal-care access	postpartum women, translated version available	limited structural and reliability evidence
Instrument D	service navigation	mixed populations, proprietary	licence, cost and scoring restrictions

Step 3: examine conceptual and content fit

Content validity should receive priority. Ask whether items are relevant, comprehensive and comprehensible for the target construct and population [2,3]. Review the original construct definition, development methods, item content and scoring domains. Do not rely only on the title of the instrument.

An instrument may contain:

• irrelevant domains that add burden;
• missing domains central to the local pathway;
• items tied to another healthcare system;
• wording unsuitable for the literacy level;
• a recall period that does not match the question; or
• a total score that combines dimensions the study needs separately.

Patient or public involvement can identify mismatches that expert review misses. A questionnaire about access should reflect how participants actually navigate the service, not only how researchers conceptualise it.

APPRAISAL CHECKLIST

Appraise conceptual fit

• Does the instrument define the same construct?
• Are all important domains represented?
• Are any items irrelevant or culturally dependent?
• Does the recall period match the study?
• Is the respondent capable of reporting the target directly?
• Does the scoring structure support the required interpretation?
• Is content evidence available from the intended population?

Step 4: appraise measurement properties

Evidence should be evaluated for the intended population, language and application. Important properties include:

Structural validity and internal consistency

Does the proposed dimensional structure fit the data, and are reliability estimates reported for coherent domains? A high alpha without structural evidence is insufficient.

Test-retest and rater reliability

Are scores stable under conditions where no change is expected? Does the repeatability design reflect the planned administration?

Measurement error and agreement

Is the error small enough for the intended decision? Group comparisons and individual monitoring can require different tolerances.

Construct or criterion validity

Do prespecified relationships support the score interpretation? Is any criterion appropriate and sufficiently strong?

Responsiveness

If the study evaluates change, can the instrument detect relevant change in the construct? Cross-sectional validity does not establish responsiveness.

Cross-cultural validity

For translated versions, is there evidence that items and scores function comparably across groups?

Mokkink and colleagues emphasise evaluating methodological quality of the evidence, not only whether published coefficients exceed a threshold [1–3].

Step 5: consider interpretability

Researchers need to know:

• score range and direction;
• meaning of subscales;
• reference distributions;
• floor and ceiling effects;
• meaningful within-person or between-group differences where established;
• threshold uncertainty; and
• handling of missing items.

A score can have acceptable measurement properties but remain difficult to interpret. Published cut-offs may not apply to another population or purpose.

Step 6: assess feasibility and burden

Feasibility includes more than completion time.

• reading and cognitive demands;
• accessibility for disability or low literacy;
• self-, interviewer- or proxy administration;
• paper, electronic or mixed-mode compatibility;
• staff training and equipment;
• scoring complexity and turnaround time;
• participant acceptability;
• data-security requirements;
• cost and licensing; and
• availability of authorised language versions.

Step 7: check permissions and version control

Questionnaires and scoring systems may be copyrighted even when freely downloadable. Researchers should identify the rights holder, permitted uses, fees, translation rights, reproduction requirements and whether electronic conversion or item modification is allowed.

Use the exact version, language and scoring manual. Record version numbers and dates. Unauthorised deletion, rewording or response-option changes can alter measurement properties and breach licence terms.

Step 8: compare candidates transparently

DECISION GUIDE

Select among candidate instruments

1. Exclude instruments with poor construct or content fit.
2. Compare evidence for validity, reliability and responsiveness in relevant contexts.
3. examine interpretability, burden, accessibility and administration requirements.
4. Confirm language availability, permissions, cost and scoring rules.
5. Identify important evidence gaps and whether local pilot work can address them.
6. Choose the best-supported feasible option, or justify adaptation or development if none is adequate.

Illustrative selection matrix

illustrative

Table 2. Comparing three candidate instruments

Criterion	Candidate A	Candidate B	Candidate C
Content fit	partial	good but overly broad	good
Evidence in postpartum population	none	limited	moderate
Malay version	none	authorised version available	local translation only
Responsiveness	not evaluated	supported	not evaluated
Burden	low	high	moderate
Permission	free with attribution	licence required	developer permission needed
Decision	reject	retain for comparison	provisional preferred candidate

Note: Ratings are illustrative and should be supported by documented evidence rather than an unweighted score alone.

Weighing evidence when no candidate is ideal

Instrument selection rarely produces one candidate that dominates every criterion. A transparent decision distinguishes essential requirements from desirable features. Essential requirements may include adequate content for the primary construct, a usable language version, permission for the intended mode and a scoring procedure that can support the primary endpoint. Desirable features may include shorter completion time, published reference values or extensive cross-national use.

Do not create an unweighted total quality score that allows severe content failure to be compensated by convenience. A candidate that omits a central domain should not win because it is free and brief. Instead, record whether each requirement is met, uncertain or not met, and explain the consequence.

Evidence may conflict. One study may support reliability while another reports poor structure. Differences can arise from population, administration, sample spectrum or methodological quality. Practical measurement texts recommend examining the design behind each coefficient rather than averaging results mechanically [6,7].

Regulatory fit-for-purpose guidance similarly emphasises the concept of interest, context of use and whether proposed modifications remain supported [5]. This is particularly important when an instrument will contribute to a primary endpoint or an individual decision.

Publication bias can affect the evidence base. Instrument developers and users may publish favourable properties more often than null or poor findings. Search for systematic reviews, manuals, theses and independent evaluations, and note when evidence comes primarily from the development team.

A decision can be provisional. The team may select the best candidate while planning cognitive interviews and a field study to address specific uncertainty. The protocol should state which findings would lead to revision, rejection or use only for exploratory analysis.

When no instrument can be recommended confidently

The evidence review may show that every candidate has major limitations. The team should resist selecting one merely because the protocol deadline is approaching. Options include narrowing the research question, measuring a more specific observable construct, using separate domain measures, adapting the best candidate with formal evaluation or beginning new development.

A temporary exploratory use should be labelled accordingly. Scores can be described without claiming established thresholds or individual interpretation, provided participants are not exposed to unsupported clinical decisions. The manuscript should state that measurement limitations constrain the substantive conclusions.

Consulting measurement specialists and target-population representatives can clarify whether the gap is conceptual or technical. Sometimes the proposed construct is too broad to support one score. Dividing it into service availability, practical burden and information clarity may produce more interpretable measurements than forcing a total access score.

The decision record should explain why alternatives were rejected and what evidence would change the choice. This protects against retrospective claims that the selected instrument was obviously superior.

Local evaluation after selection

Selecting an existing instrument does not remove the need for local appraisal. At minimum, verify comprehension, administration feasibility, score calculation and content relevance. Depending on the evidence gap and intended use, the study may need evaluation of structural validity, reliability, cross-cultural validity or responsiveness.

Local evaluation should be proportionate. Repeating every measurement-property study is inefficient when strong evidence already exists in a closely matched context. Conversely, a small convenience pilot cannot establish all properties for a major clinical decision.

PROTOCOL CHECKLIST

Document the instrument-selection decision

• construct, domains and intended use;
• search sources and date;
• candidate instruments and reasons for exclusion;
• evidence for content and other measurement properties;
• target-population and language relevance;
• score interpretation and missing-item rules;
• burden, accessibility and administration mode;
• permission, licence and cost;
• selected version and scoring manual; and
• planned local evaluation and unresolved limitations.

Systematic reviews of measurement instrumentsMMed extension

When instrument choice is central and several candidates exist, a systematic review of measurement properties may be justified. COSMIN provides methods for searching, appraising risk of bias, rating measurement results and grading the body of evidence. The review question specifies the construct, population and instrument type, and conclusions should consider both evidence quality and content validity rather than ranking coefficients mechanically [2].

Practical interpretation note

The preferred instrument is the most defensible feasible option for the stated purpose, not a universally best measure.

Interface with statistical analysis

Instrument selection affects score distribution, dimensionality, missing-item rules, detectable change and sample-size assumptions. The analysis plan should use the authorised scoring procedure and preserve domains where the instrument is not supported as one total scale.

Comparisons across studies are strongest when the same well-supported outcome and instrument are used, but standardisation should not override poor local validity. If different instruments measure the same construct, score harmonisation requires additional assumptions and belongs in advanced statistical or evidence-synthesis work.

Worked example

WORKED EXAMPLE

Selecting rather than immediately developing an access measure

Requirement: Measure practical and service-related barriers to postpartum glucose testing during the first 12 weeks after delivery.

Search: The team identifies 11 instruments and three systematic reviews. Six instruments measure broad healthcare access, three focus on treatment burden and two address maternal follow-up.

Content screening: Most broad instruments omit childcare and referral coordination. One maternal instrument covers these domains but uses healthcare-system-specific insurance items.

Evidence review: The maternal instrument has participant-informed development, a coherent two-domain structure and acceptable test-retest reliability in postpartum populations. Its local-language version is not authorised.

Decision: Request permission to translate and culturally adapt the maternal instrument rather than develop a new scale. Retain the original domain structure provisionally and plan cognitive interviews and field evaluation before using the scores in the main study.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

What should be defined before searching for an instrument?

Answer and reasoning

The construct, relevant domains, target population, context, respondent, time frame, administration mode and intended use should be defined. Without these, there is no defensible basis for deciding whether a candidate is fit for purpose.

Why should content validity be prioritised?

Answer and reasoning

An instrument that omits important domains or includes irrelevant content does not adequately represent the target. Strong reliability or sophisticated modelling cannot recover missing content.

Can researchers delete two inconvenient items and retain the original validity evidence?

Answer and reasoning

Not automatically. Item deletion changes the content and potentially the dimensionality, score range and reliability. Permission may also be required. The modified version needs a new rationale and proportionate evaluation.

When may local evaluation still be necessary for an established instrument?

Answer and reasoning

It is needed when the population, language, mode, setting or intended use differs materially, or when important properties such as responsiveness have not been established. The scope should match the evidence gap.

Key take-home messages

SUMMARY

Key take-home messages

• Define the construct and intended use before searching for an instrument.
• Prioritise conceptual and content fit before comparing coefficients.
• Evaluate evidence quality, interpretability, burden, accessibility, language and permissions together.
• Use the exact authorised version and scoring rules.
• Adapt or develop a new instrument only when existing options cannot adequately support the research purpose.

References

1. Mokkink LB, Prinsen CAC, Bouter LM, de Vet HCW, Terwee CB. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) and how to select an outcome measurement instrument. Braz J Phys Ther. 2016;20(2):105–113. https://doi.org/10.1590/bjpt-rbf.2014.0143
2. Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, et al. COSMIN manual for systematic reviews of PROMs. Version 2.0. Amsterdam: COSMIN; 2024. https://www.cosmin.nl/wp-content/uploads/COSMIN-manual-V2_final.pdf
3. Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–1170. https://doi.org/10.1007/s11136-018-1829-0
4. Prinsen CAC, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to select outcome measurement instruments for outcomes included in a Core Outcome Set—a practical guideline. Trials. 2016;17:449. https://doi.org/10.1186/s13063-016-1555-2
5. US Food and Drug Administration. Patient-focused drug development: selecting, developing, or modifying fit-for-purpose clinical outcome assessments. Silver Spring: FDA; 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-selecting-developing-or-modifying-fit-purpose-clinical-outcome
6. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. 5th ed. Oxford: Oxford University Press; 2015. https://doi.org/10.1093/med/9780199685219.001.0001
7. de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press; 2011. https://doi.org/10.1017/CBO9780511996214