Delphi and Formal Consensus Methods

Overview

Delphi studies use structured, iterative questioning of a panel, usually with controlled feedback between rounds, to explore judgement and develop convergence when direct empirical evidence is incomplete or decisions require synthesis of distributed expertise. RAND methodological guidance provides cross-disciplinary recommendations for designing, conducting and critically appraising Delphi panels; CREDES provides detailed conduct and reporting guidance developed in palliative-care guideline work; and methodological reviews have highlighted persistent variation in how consensus is defined.[1–3] Classic health-research guidance additionally emphasises transparent panel selection, iterative rounds and careful interpretation.[4,5]

Consensus is not a substitute for evidence. A panel can agree on a poor recommendation if its members share the same blind spots or if the evidence supplied to them is incomplete. Formal consensus methods are most defensible when the purpose genuinely requires judgement—for example prioritisation, core outcome selection, terminology, guideline recommendations under uncertainty, or specification of professional standards. Healthcare-focused methodological guidance also stresses deciding first whether Delphi is appropriate for the question.[6]

Why this matters

Unstructured expert meetings are vulnerable to hierarchy, group pressure, selective memory and poorly documented decision rules. Delphi methods can reduce some of these problems by allowing independent rating, controlled feedback and iterative reconsideration. They also create an auditable record of how views changed.

However, “Delphi” is sometimes used loosely for any survey conducted in several rounds. Methodological credibility depends on who qualifies as an expert, how items are generated, whether panelists receive comparable information, how anonymity is handled, how consensus is defined, how non-response is managed and when the process stops. These decisions should be prespecified rather than adjusted until the desired answer appears.

Key concepts

A classical Delphi typically involves multiple rounds of anonymous or quasi-anonymous questionnaires. Panelists rate or comment on items, receive aggregate feedback, and reconsider their responses. A modified Delphi may begin with items generated from prior literature or an existing framework rather than an open first round. Other formal consensus methods include nominal group techniques, structured consensus conferences and RAND/UCLA-type appropriateness methods; the best choice depends on whether interaction, anonymity, evidence review or voting is central.

Panel expertise is multidimensional. The relevant panel may require lived experience, clinical practice, methodology, implementation, policy or service-user perspectives. A large homogeneous panel does not necessarily produce better judgement than a deliberately diverse panel whose expertise matches the question. Sampling should therefore be purposive and justified.

Consensus must be operationally defined before data collection. Common rules combine a proportion above a threshold with limits on disagreement or dispersion. There is no universal 70% or 80% threshold that automatically makes a study valid.[3] The threshold should reflect the decision stakes and be accompanied by a rule for items that remain uncertain.

DECISION GUIDE

Use Delphi when these conditions are present

• the question requires judgement rather than a directly estimable empirical parameter;
• relevant expertise is distributed across people or stakeholder groups;
• independent initial judgement is valuable before group feedback;
• iteration may improve clarity or convergence;
• the team can define transparent consensus and stopping rules;
• the final product will be interpreted as consensus guidance rather than empirical proof.

A step-by-step methodological workflow

1. Define the decision and why consensus is needed

State the product: priorities, competencies, core outcomes, definitions, appropriateness ratings or recommendations. Explain why empirical evidence alone cannot resolve it. Review existing evidence first so the panel is not asked to recreate knowledge that could be obtained systematically.

2. Construct the panel purposively

Define expertise domains and eligibility criteria. Decide whether stakeholder groups need balanced representation or separate analysis. Record recruitment source, invitations, conflicts of interest and reasons for panel composition. Avoid convenience panels made only of colleagues who already share the organisers’ views.

3. Generate and pilot the items

Items can come from literature, interviews, an open round or existing standards. Write one proposition per item where possible and pilot for ambiguity. Decide whether panelists can propose new items and how these are edited between rounds. The steering group should not quietly alter item meaning while summarising feedback.

4. Prespecify rating, feedback and anonymity

Choose the rating scale and what each anchor means. Decide which statistics and qualitative comments are returned to panelists. Anonymity can reduce dominance but may be partial when panelists know membership. State who can see individual responses, whether panelists can identify comments, and how confidentiality is protected.

5. Define consensus and stopping before Round 1

Specify thresholds for consensus in, consensus out and uncertain items; rules for disagreement; number or maximum number of rounds; and conditions for stopping because ratings stabilise or response falls too low. Do not move the threshold after seeing which items would otherwise fail.

6. Analyse attrition and report the process

Report invitations, participation and retention by round. Compare characteristics of completers and non-completers when possible. Present rating distributions, changes and final consensus rather than only a list of accepted items. Interpret unresolved disagreement as informative rather than a methodological embarrassment.

PROTOCOL CHECKLIST

Delphi protocol items

• purpose and rationale for formal consensus;
• panel expertise domains, eligibility and recruitment;
• conflict-of-interest management;
• item-generation and piloting process;
• rating scale and feedback content;
• anonymity or confidentiality model;
• consensus, disagreement and stopping rules;
• planned number of rounds and attrition handling;
• analysis, stakeholder subgroup reporting and dissemination.

An auditable planning document

Prespecified consensus rule example

illustrative

Table 1. Illustrative decision rules

Classification	Prespecified rule	Action
Consensus in	At least 80% rate 7–9 and less than 15% rate 1–3	Retain
Consensus out	At least 80% rate 1–3 and less than 15% rate 7–9	Exclude
Uncertain	Neither rule met	Re-rate with feedback
Persistent disagreement	Polarised ratings after final round	Report disagreement explicitly

Note: These thresholds are illustrative. There is no universal percentage that defines valid consensus.

Quality, validity and governance

Attrition can change the panel. If dissenting members disproportionately leave after Round 1, apparent consensus may increase because the composition changed rather than because opinions converged. Report response and retention by round and examine whether dropout differs by stakeholder group.

Controlled feedback is another quality dimension. Returning only the median can obscure polarisation; adding the interquartile range, distribution and anonymised rationale can make disagreement visible. However, excessive feedback can burden panelists and steer them toward the organisers’ preferred interpretation. The protocol should define what feedback is standardised for all members.

Conflict of interest matters because formal consensus may shape guidelines or policy. Financial, professional and intellectual conflicts should be disclosed and managed. Stakeholder participation can broaden legitimacy, but representation should not be tokenistic. Service users may need tailored materials or support to participate on equal terms.

APPRAISAL CHECKLIST

Appraise a Delphi or consensus study

• Was consensus genuinely needed for the question?
• Does the panel represent the expertise and stakeholder perspectives required?
• Were items generated and edited transparently?
• Were consensus thresholds, feedback and stopping rules prespecified?
• Was attrition reported and considered as a source of bias?
• Are consensus results interpreted as expert judgement rather than proof of effectiveness?

Practical decision audit

Audit panel construction

Create an expertise matrix before recruitment. Ask which perspectives could change the decision and whether each is represented. Avoid defining “expert” only by academic title when practical or lived expertise is essential.

Audit threshold sensitivity

Before data collection, simulate how plausible response distributions would be classified under the proposed rule. An extremely permissive rule may label weak agreement as consensus; an excessively strict rule may prevent any decision. If multiple rules are scientifically defensible, prespecify a sensitivity summary rather than changing the rule after results.

Audit item drift

Track each item across rounds. If wording changes, document whether it is clarification or a substantive new proposition. Panelists should not be told that an item “reached consensus” when the final item differs materially from what they first rated.

Audit disagreement as a result

Some questions remain legitimately contested. Report the distribution and reasons for disagreement, stakeholder differences and evidence uncertainty. Forcing every item into a consensus category can erase clinically or ethically important pluralism.

Consensus thresholds are decision rules, not measurement propertiesMMed extension

Researchers often debate whether consensus should be 70%, 75% or 80%. The deeper question is what decision error the rule is intended to control. A high-stakes safety recommendation may require stronger agreement than a preliminary research-priority list. A rule combining central tendency with a measure of disagreement may be more informative than a percentage alone.

Statistical indices of agreement can supplement but not replace substantive interpretation. Stability across rounds does not prove correctness; it may indicate that opinions have stopped changing. Likewise, convergence can reflect social pressure if anonymity is weak. The report should therefore present process information, distributions and rationale rather than a single “consensus achieved” label.

Advanced design choices in formal consensus research

The Delphi label covers substantial methodological variation. Some studies seek forecasting, others priority setting, definition development, core outcome selection, appropriateness judgement or guideline recommendation. The design should match that purpose. A panel deciding whether procedures are clinically appropriate may need structured evidence summaries and explicit benefit-risk judgement; a panel generating research priorities may need broader stakeholder representation and open idea generation.

Expertise should be defined before names are nominated

Begin with expertise categories, not a list of famous people. Specify what knowledge is required: years of clinical practice, methodological scholarship, programme implementation, policy responsibility, patient experience or other qualifications. Then identify individuals who satisfy those categories. This reduces prestige bias and makes panel composition auditable. Where stakeholder groups may value outcomes differently, prespecify whether consensus is assessed overall, within groups or both.

Panel size is a design judgement, not a power calculation

There is no universal sample-size formula for Delphi studies. Larger panels can broaden perspectives but increase attrition and coordination burden. Small expert panels may be appropriate for highly specialised questions but can be fragile if one or two members withdraw. Justify panel size through diversity of expertise, anticipated attrition, subgroup needs and feasibility rather than importing a conventional survey calculation.

Round 1 can be generative or confirmatory

An open first round allows panelists to generate items without being constrained by the steering group, but it creates substantial qualitative coding work and may produce redundant items. A modified Delphi can start from a systematic review or existing framework, which is efficient but risks inheriting the biases of the source. Explain why the chosen approach fits the maturity of the field.

Feedback design can influence convergence

Showing the median and interquartile range informs panelists about central tendency and dispersion. Showing the full distribution can expose polarisation. Providing anonymised comments gives reasons but may disproportionately highlight eloquent or extreme views. Personalised feedback showing each panelist’s prior rating relative to the group can encourage reflection. Because feedback itself is an intervention, standardise it and report exactly what participants received.

Consensus and stability answer different questions

Consensus concerns the degree of agreement at a time point. Stability concerns whether ratings are changing across rounds. A panel can be stably divided, or it can reach apparent consensus that is still shifting. Stopping rules can therefore consider both threshold classification and stability. A maximum round count protects participants from endless iteration and prevents the steering group from continuing until a preferred outcome emerges.

Statistical summaries should preserve distributions

Means can hide polarised ordinal ratings. Medians, interquartile ranges and proportions in predefined rating bands are often more interpretable. For core outcome or appropriateness work, separate “consensus in”, “consensus out” and “no consensus” categories can preserve uncertainty. Avoid converting every item into a binary decision if the distribution contains meaningful disagreement.

Qualitative comments are data, not decoration

Open comments can identify ambiguity, missing conditions and reasons for disagreement. Analyse them systematically enough to justify item revisions. Keep a revision log showing original item, comments, steering-group decision and new wording. If comments reveal two distinct concepts, split the item rather than averaging incompatible views.

Steering committees need governance safeguards

The steering group can strongly influence item selection, wording and interpretation. Define membership, conflicts, decision rules and whether panelists can challenge revisions. Where recommendations affect commercial products or professional scope, conflict management is especially important. A formally structured Delphi can still be biased if the steering committee controls the process opaquely.

Consensus should be validated when possible

A consensus competency framework can be piloted in training; a consensus outcome set can be tested for feasibility and measurement properties; appropriateness recommendations can be compared with patient outcomes or future evidence. Consensus is often the beginning of an empirical programme rather than the final answer.

Designing consensus rules that remain interpretable

The protocol should prespecify what happens to items in every possible rating pattern. A rule such as “70% agreement” is incomplete unless the response scale, denominator, missing ratings, definition of agreement, treatment of abstentions and handling of polarisation are specified. For example, an item can have a high median yet still divide the panel into opposing groups. Reporting only the median or the proportion above a threshold can conceal that disagreement.

Consider defining at least three states: consensus for inclusion, consensus for exclusion, and no consensus. The last category is substantive information, not a failed result. Sensitivity analyses can show whether conclusions change under nearby plausible thresholds, especially when the final recommendation hinges on one or two borderline items. Stability across successive rounds is another useful concept: a panel may cross a numerical consensus threshold while individual ratings are still changing substantially.

Attrition should be examined by round and, where possible, by stakeholder group and earlier rating pattern. If dissenting participants disproportionately leave, apparent convergence can be produced by panel composition rather than persuasion or genuine reconsideration. Feedback should therefore be standardised and transparent, showing participants an agreed summary of the previous round without steering them towards the investigators’ preferred answer.

For multidisciplinary panels, consensus may also be examined within prespecified stakeholder groups. A global threshold can otherwise hide systematic disagreement between patients, clinicians, methodologists and service managers. The purpose is not to force every subgroup to agree, but to preserve information about whose judgement supports the final recommendation. Strong Delphi research treats disagreement, stability, attrition and subgroup patterns as findings that help readers judge the legitimacy and transferability of the consensus.

Protecting the legitimacy of the final consensus

The steering group should define how conflicts of interest, sponsorship and item ownership are managed. Panelists may have financial, professional or intellectual commitments to particular recommendations. Disclosure does not automatically disqualify expertise, but the protocol should state when a conflict affects participation, voting or interpretation. Sponsors should not be able to remove inconvenient items or redefine consensus after seeing round results.

The final report should preserve the denominator for every item and round. “80% agreement” is uninterpretable if only a shrinking subset of the original panel responded. Present response counts, attrition, relevant stakeholder composition and the distribution of ratings, especially for disputed items. Where the method is used to create healthcare guidance, readers should also be able to see why a Delphi approach was appropriate for the decision rather than empirical research or another consensus technique.[6]

A final governance check should ask who has authority to approve the consensus output and whether that approval step was part of the Delphi itself or a separate organisational decision. Keeping those stages distinct prevents a later committee endorsement from being misrepresented as panel consensus.

Worked example

WORKED EXAMPLE

Designing the research-coordinator competency Delphi

The steering group reviews existing competency frameworks and interviews coordinators from several settings to generate an initial item pool. It defines five expertise domains and purposively recruits panelists across public and private research sites. Members disclose organisational and financial interests. A pilot panel revises ambiguous items before Round 1.

The protocol specifies a 9-point rating scale, aggregate distribution plus anonymised comments as feedback, and a maximum of three rounds. Consensus-in and consensus-out rules are fixed prospectively. Panelists can suggest new items after Round 1; these are clearly marked as new and undergo the same rating process.

After Round 3, most competencies reach consensus but two remain polarised between sponsor and public-sector representatives. The working group reports this disagreement rather than lowering the threshold. The final document states that the competency set represents structured stakeholder consensus and should be evaluated in future training research; it does not claim that the competencies have been empirically shown to improve trial outcomes.

Suggested protocol and manuscript wording

Common mistakes

Knowledge check

Why is a Delphi panel usually sampled purposively rather than randomly from the general population?

Answer and reasoning

The method seeks informed judgement from people with expertise or stakeholder perspectives relevant to the decision, not a population prevalence estimate.

Is 80% agreement a universally valid consensus threshold?

Answer and reasoning

No. Thresholds vary and should be justified and prespecified according to the decision context.

How can attrition create false convergence?

Answer and reasoning

If dissenting panelists disproportionately stop responding, the remaining panel can appear more consensual even though individual opinions did not converge.

What should researchers do with persistent disagreement after the final round?

Answer and reasoning

Report it transparently, including distributions and reasons where available. Disagreement can be an important finding.

Key take-home messages

SUMMARY

Key take-home messages

• Use Delphi for structured judgement when empirical evidence alone cannot resolve the decision.
• Panel composition is part of the method and must match the expertise required.
• Prespecify feedback, anonymity, consensus thresholds and stopping rules.
• Analyse attrition and preserve item history across rounds.
• Consensus supports a recommendation or shared judgement; it does not prove effectiveness or truth.

References

1. Khodyakov D, Grant S, Kroger J, Bauman M. Methodological guidance for conducting and critically appraising Delphi panels. Santa Monica, CA: RAND Corporation; 2023. https://doi.org/10.7249/TLA3082-1
2. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care. Palliat Med. 2017;31(8):684–706. https://doi.org/10.1177/0269216317690685
3. Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002
4. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x
5. Keeney S, Hasson F, McKenna H. The Delphi technique in nursing and health research. Chichester: Wiley-Blackwell; 2011.
6. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness. World J Methodol. 2021;11(4):116–129. https://doi.org/10.5662/wjm.v11.i4.116