Developing a New Questionnaire or Scale

Overview

Developing a questionnaire or scale is a programme of construct definition, item development, qualitative evaluation and quantitative testing. It is not the act of writing several plausible questions and calculating Cronbach’s alpha. A new instrument is justified only when existing measures cannot support the intended construct, population, language, setting or use after careful selection and feasible adaptation [1–4].

A robust development pathway is iterative. Researchers define the construct and intended interpretation, generate a deliberately broad item pool, obtain input from the target population and experts, select response formats, conduct cognitive interviews, field test the provisional instrument, evaluate dimensionality and measurement properties, revise the instrument and confirm the final structure in new data where feasible.

The process should preserve content while improving measurement. Statistical item reduction can produce a short internally consistent scale that no longer represents the construct. Every deletion or modification requires a conceptual as well as statistical rationale.

Why this matters

New questionnaires are common because they appear easy to create. Weak instruments then circulate with unsupported claims of validity, inconsistent scoring and no evidence that respondents interpret items as intended. This wastes participant effort and can generate misleading research conclusions.

Rigorous development is particularly important when scores will guide screening, referral, eligibility or individual decisions. The evidence required for a descriptive research scale may differ from that required for high-stakes clinical classification, but both begin with clear construct and content work.

Stage 1: justify new development

Before writing items, document:

• existing instruments and why they are unsuitable;
• whether translation or adaptation could address the gap;
• intended users and decisions;
• resources for qualitative and quantitative evaluation;
• how the new instrument will improve research or care; and
• plans for maintenance, permissions and future access.

DECISION GUIDE

Develop, adapt or select

1. Does a well-supported instrument measure the same construct for the intended population and use?
2. Could a permitted translation or limited adaptation resolve the mismatch?
3. Would adaptation replace so much content that the result is effectively new?
4. Is the remaining gap important enough to justify development resources?
5. Can the team conduct participant-informed and quantitative evaluation rather than stopping after item drafting?

Stage 2: define the construct and intended use

Construct definition sets the boundaries of the scale. For access burden, decide whether the construct includes:

• perceived difficulty;
• actual resource costs;
• service availability;
• emotional burden;
• social support; and
• consequences such as missed testing.

These elements are related but not identical. Test completion is an outcome of burden and support, not necessarily an item within the burden score.

Specify the population, setting, time frame, respondent, administration mode and whether scores are intended for group comparisons, screening or monitoring change. A broad label such as “barriers questionnaire” is insufficient.

Illustrative construct specification

illustrative

Table 1. Proposed boundaries of postpartum access burden

Include	Exclude from the scale	Measure separately
difficulty contacting services	diabetes knowledge	confirmed test completion
difficulty obtaining a suitable appointment	clinical glucose result	counselling exposure
transport, time, cost and childcare burden	general satisfaction with maternity care	social support resources
clarity of where and how to attend	depressive symptoms	clinic-level availability

Stage 3: generate the item pool

Item generation should combine several sources:

• qualitative interviews or focus groups with the target population;
• literature and existing instruments;
• theory or conceptual frameworks;
• clinician and service-provider input;
• patient advocates and community partners; and
• direct observation of the care pathway.

The initial pool should be broader than the final scale. It should cover each domain and capture different expressions of the construct. At this stage, premature concern about length can remove important content.

Items should generally express one idea, use a defined time frame and avoid assumptions, jargon, double negatives and double-barrelled wording. Positively and negatively worded items should not be mixed merely to manipulate response patterns; confusing reversals can create method effects.

Stage 4: content-validity evaluation

Content validity evaluates relevance, comprehensiveness and comprehensibility [2,5]. Target-population participants and experts contribute different evidence. Participants identify lived relevance and interpretation; experts can assess construct coverage, clinical accuracy and redundancy.

Quantitative ratings such as content-validity indices can summarise judgements, but they do not replace qualitative reasoning. High agreement among a small homogeneous panel may overlook missing domains. The protocol should describe panel composition, rating criteria, decision rules and how comments influenced revision.

APPRAISAL CHECKLIST

Appraise item content before field testing

• Does each item map to a defined domain?
• Is the content relevant to the target population and context?
• Does the full pool cover all important aspects?
• Are items understandable and answerable?
• Are important perspectives missing from the review panel?
• Would item deletion create construct under-representation?
• Are causes, manifestations and consequences being confused?

Stage 5: select response formats and scoring hypotheses

Response formats should match the item target: frequency, severity, difficulty, agreement or occurrence. The recall period must be appropriate. Fully labelled categories can improve interpretability, while too many categories create unsupported precision.

Before field testing, state provisional hypotheses:

• which items belong to each domain;
• whether one total score is intended;
• score direction;
• whether items are reflective or formative indicators;
• how missing responses may be handled; and
• what external relationships are expected.

Do not finalise weights or cut-offs without evidence. Equal item weighting is common but still an assumption.

Stage 6: cognitive interviewing and pilot testing

Cognitive interviews evaluate comprehension, retrieval, judgement and response mapping. Iterative rounds allow revised items to be retested. Pilot testing then examines layout, burden, skip logic, administration and data capture.

Items that participants interpret differently from the intended concept should be revised or removed even when experts consider them elegant. Conversely, an unusual but important item should not be deleted only because it reduces internal consistency.

Stage 7: design the field test

The field-test sample should reflect the intended population and include sufficient variation across the construct. Recruitment only from one high-performing clinic may create floor effects and limit evaluation of item functioning.

There is no universal rule such as ten participants per item that guarantees an adequate factor-analysis sample. Requirements depend on communalities, number of factors, items per factor, response distributions, model complexity, estimator, missingness and desired precision [1,3,6]. Plan through simulation or evidence-based design reasoning where possible, and acknowledge uncertainty.

The dataset should preserve item-level responses, administration mode, timing and relevant subgroup characteristics. Split-sample or independent confirmation is preferable when the same data are used for extensive exploratory decisions.

Stage 8: examine item performance

Initial item analysis describes:

• missing and not-applicable responses;
• response-category use;
• floor and ceiling effects;
• item distributions;
• item-total and inter-item relationships;
• redundancy;
• subgroup differences; and
• comments or operational problems.

Rigid cut-offs should not determine deletion automatically. A rare-response item may identify an important severe barrier. A very high item-total correlation may indicate duplication. Decisions should integrate content and statistical evidence.

Stage 9: evaluate dimensionality

Exploratory factor analysis

Exploratory factor analysis is appropriate when the latent structure is uncertain. Decisions include the correlation matrix for ordinal items, factor-retention method, extraction, rotation and interpretation. Principal component analysis summarises observed variance but is not identical to a latent factor model.

Confirmatory factor analysis

Confirmatory factor analysis evaluates a prespecified structure. Fit indices are considered together with parameter estimates, residuals, theory and sample characteristics. Adding correlated errors or moving items solely to improve fit can conceal misspecification.

Item response theory and Rasch models

These models relate item responses to a latent trait and can examine category thresholds, information and differential item functioning. They require additional assumptions and expertise. They are tools for evaluating a defined construct, not substitutes for content development.

Reflective and formative scalesMMed extension

Reflective items are manifestations of a latent construct and are expected to covary. Formative indicators jointly define an index, so internal consistency may be inappropriate. Practical access burden may include formative components such as travel time, cost and childcare availability. The choice affects item retention, weighting and interpretation and should be made conceptually before modelling.

Stage 10: evaluate reliability and measurement error

For coherent reflective domains, evaluate internal consistency after dimensionality. Test-retest reliability requires participants expected to remain stable and an appropriate interval. Agreement and measurement error should be considered when the score will monitor individuals.

A high coefficient is not the goal in isolation. Reliability should be sufficient for the intended use without sacrificing content or creating redundant items. Confidence intervals should be reported.

Stage 11: evaluate validity and responsiveness

Build cumulative evidence:

• content relevance, comprehensiveness and comprehensibility;
• response processes;
• structural validity;
• a priori convergent, discriminant and known-groups hypotheses;
• criterion evidence when an appropriate criterion exists;
• cross-cultural validity or subgroup fairness; and
• responsiveness when change scores will be interpreted.

Test completion may relate to access burden, but it is not an error-free criterion because motivated women can overcome severe burden and some low-burden women may not attend for other reasons.

Advanced decisions in item reduction and model revision

Item reduction should preserve representation across domains and respondent experiences. Statistical signals are conditional on the development sample. An item with a low loading may be poorly worded, may measure a rare but important experience, or may belong to a formative index rather than a reflective factor. Before deletion, examine cognitive evidence, subgroup response patterns and whether the item protects content breadth.

Redundancy also requires judgement. Two highly correlated items may appear duplicative, but one may be accessible to respondents with different experiences or may be needed for reliable coverage across the trait range. Item response information and qualitative evidence can help select between them.

Model modification should be limited and documented. Correlated residuals can reflect similar wording, shared context or an omitted subdomain. Adding many post hoc correlations may improve fit while weakening generalisability. Each modification should have a substantive explanation and should be evaluated in new data.

Sample-size planning by analysis purpose

One sample may be inadequate for every development task. Content interviews, cognitive interviews, exploratory factor analysis, confirmatory analysis, test-retest reliability and responsiveness each have different information requirements. A development programme can therefore use several linked samples rather than searching for one universal number.

For factor models, simulation can explore expected factor loadings, category distributions, missingness, number of factors and estimator performance. When simulation is not feasible, justify the planned sample using empirical guidance and anticipated data quality rather than a rigid ratio. Report achieved sample characteristics and uncertainty.

Fairness and subgroup functioning

A scale intended for diverse populations should examine whether items are accessible and function similarly across relevant groups. Differential item functioning may arise from language, literacy, disability, care pathway or cultural experience. Statistical differences are interpreted alongside qualitative evidence; automatic deletion can remove content important to one group.

Open and reproducible instrument development

Development reports should provide the construct specification, item-generation methods, version history, scoring algorithm and sufficient item-level results for appraisal, subject to copyright and participant confidentiality. Clark and Watson emphasise that scale construction requires iterative construct refinement rather than coefficient maximisation [7]. Current fit-for-purpose guidance also links documentation and evidence to the proposed context of use [8].

A public instrument manual should state permitted uses, current version, languages, missing-item rules, evidence base and known limitations. Future changes require dated versions so studies do not unknowingly use different instruments under the same name.

Governance of item and model decisions

A development committee should include construct experts, measurement specialists and target-population representation. Major item deletion, domain redefinition and scoring changes should be decided against prespecified principles and documented with dissent where relevant.

Analysts should provide statistical evidence without allowing software output to become the sole decision maker. The committee should review whether changes preserve the intended construct, accessibility and future comparability. A locked analysis plan for the confirmation sample reduces opportunities to continue modifying the model until fit appears favourable.

The publication should separate planned and post hoc decisions. Provide a flow of items from the initial pool to the final version, reasons for removal and the stage at which each change occurred. This allows users to judge whether the final scale reflects participant-informed content or mainly the idiosyncrasies of one dataset.

Stage 12: interpretability, scoring and thresholds

Specify the final items, domains, response options, scoring algorithm and missing-item rules. Report observed score distributions, floor and ceiling effects and reference values. Meaningful-change estimates should be evaluated for the intended population and use.

Screening thresholds require separate classification evidence and careful evaluation of consequences. Selecting a cut-off that maximises performance in the development sample produces optimistic results; external validation is necessary.

Stage 13: confirmation, revision and maintenance

Instrument development does not end with publication. Confirm structure and measurement properties in new samples, languages, modes and settings. Maintain version control and document modifications. Major wording or scoring changes create a new version requiring re-evaluation.

PROTOCOL CHECKLIST

Specify the development programme

• justification for a new instrument;
• construct, domains, population and intended use;
• qualitative item-generation methods;
• item pool and content-validity procedures;
• cognitive-interview and pilot cycles;
• response formats and provisional scoring model;
• field-test sampling and sample-size rationale;
• exploratory and confirmatory analysis plan;
• reliability, validity and responsiveness hypotheses;
• item-retention and revision criteria;
• independent confirmation or validation strategy; and
• permissions, version control and dissemination plan.

Practical interpretation note

The final scale is a versioned measurement system, not merely a list of retained items. Its manual should travel with the instrument.

Interface with statistical analysis

Scale development is multidisciplinary. Statistical analysis should follow construct and content work. It includes item distributions, appropriate correlations for response types, exploratory or confirmatory factor models, reliability estimates with uncertainty, and prespecified validity hypotheses.

Avoid selecting items solely by p-values, alpha-if-deleted or arbitrary loading thresholds. The analysis plan should account for ordinal responses, missingness, clustering and repeated measurements where applicable. Detailed factor-analysis and psychometric modelling procedures belong in specialised statistical teaching and collaboration.

Worked example

WORKED EXAMPLE

Developing the Postpartum Testing Access Burden Scale

Construct: Perceived practical and service-related burden experienced when arranging and attending maternal glucose testing within 12 weeks after delivery.

Item generation: Interviews with 24 women and service staff identify appointment access, information, transport, time, cost and childcare domains. Existing instruments contribute wording ideas after permission and citation review. The initial pool contains 42 items.

Content evaluation: Women and multidisciplinary experts rate relevance and identify missing content. Twelve redundant or irrelevant items are removed, and six are rewritten.

Cognitive testing: Three iterative rounds reveal confusion between maternal and infant visits, and between lack of an appointment and difficulty attending. The provisional version contains 22 items.

Field testing: A diverse multicentre sample completes the scale. Exploratory analysis suggests three correlated domains. Items are retained or removed using content, category functioning, residuals and loading patterns rather than one threshold.

Confirmation: A new sample evaluates the three-domain model, reliability, measurement error and a priori relationships. The final instrument contains 15 items. No screening cut-off is proposed because the intended use is research description and group comparison.

Suggested protocol or reporting wording

Common mistakes

Knowledge check

When is a new instrument justified?

Answer and reasoning

It is justified when existing instruments cannot support the defined construct, population, context and intended use after serious consideration of selection and feasible adaptation, and when the team can conduct a full participant-informed evaluation programme.

Why should item generation deliberately begin with a broad pool?

Answer and reasoning

A broad pool protects content coverage and allows qualitative and quantitative evaluation. Starting with only a few convenient items risks omitting important dimensions that later statistics cannot recover.

Is ten participants per item an adequate universal sample-size rule?

Answer and reasoning

No. Adequacy depends on communalities, factors, item distributions, estimator, model complexity, missingness and precision. The sample-size rationale should reflect the planned model and data conditions rather than one fixed ratio.

Why might deleting an item that lowers alpha be a mistake?

Answer and reasoning

The item may represent an important but distinct aspect of the construct. Alpha can increase when content is narrowed or items are duplicated. Item decisions should consider content, dimensionality, response functioning and intended interpretation.

What is the value of an independent confirmation sample?

Answer and reasoning

It evaluates whether the structure and performance persist beyond the data used for item selection and model modification. This reduces optimistic claims caused by overfitting to the development sample.

Key take-home messages

SUMMARY

Key take-home messages

• Develop a new instrument only when selection or adaptation cannot address an important measurement gap.
• Define the construct and intended use before generating items.
• Target-population input, content evaluation and cognitive interviewing precede statistical modelling.
• Dimensionality, reliability, validity and responsiveness provide complementary evidence.
• Item reduction must preserve content and should not be driven by one coefficient or threshold.
• Separate exploratory development from confirmatory evaluation whenever feasible.

References

1. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149. https://doi.org/10.3389/fpubh.2018.00149
2. Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–1170. https://doi.org/10.1007/s11136-018-1829-0
3. DeVellis RF, Thorpe CT. Scale development: theory and applications. 5th ed. Thousand Oaks: Sage; 2021.
4. Artino AR Jr, La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No. 87. Med Teach. 2014;36(6):463–474. https://doi.org/10.3109/0142159X.2014.889814
5. Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, et al. COSMIN manual for systematic reviews of PROMs. Version 2.0. Amsterdam: COSMIN; 2024. https://www.cosmin.nl/wp-content/uploads/COSMIN-manual-V2_final.pdf
6. Worthington RL, Whittaker TA. Scale development research: a content analysis and recommendations for best practices. Couns Psychol. 2006;34(6):806–838. https://doi.org/10.1177/0011000006288127
7. Clark LA, Watson D. Constructing validity: basic issues in objective scale development. Psychol Assess. 1995;7(3):309–319. https://doi.org/10.1037/1040-3590.7.3.309
8. US Food and Drug Administration. Patient-focused drug development: selecting, developing, or modifying fit-for-purpose clinical outcome assessments. Silver Spring: FDA; 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-selecting-developing-or-modifying-fit-purpose-clinical-outcome