Part 14 · Module 9
Risk-of-Bias and Study-Quality Appraisal
Appraise risk of bias using design-appropriate tools, distinguish bias from reporting quality and evidence certainty, and avoid invalid numerical study-quality scores.
Learning outcomes
After completing this lecture, you should be able to:
- Distinguish risk of bias, methodological quality, reporting quality, applicability and certainty of evidence.
- Select an appraisal tool that matches the study design and result being assessed.
- Use signalling questions and domain judgements rather than summing checklist items into an arbitrary score.
- Explain why risk-of-bias assessment is often result-specific rather than one label for an entire article.
- Integrate appraisal findings into interpretation and evidence synthesis transparently.
Overview
Critical appraisal should ask whether a specific result is likely to be biased, whether the study applies to the review or clinical question, and whether the report contains enough information to judge those issues. These are different from assigning a vague “quality score”. Modern risk-of-bias tools use structured domains and signalling questions because different flaws have different consequences and cannot be meaningfully added as if they were points on one scale.[1]
Tool choice depends on design and question. Cochrane recommends RoB 2 for randomised trials. For non-randomised studies of interventions, ROBINS-I V2 is available as a revised draft, with the riskofbias.info site noting that the November 2025 version remains subject to change.[2] ROBINS-E addresses non-randomised follow-up studies of exposure effects and was published in 2024.[3]
Why this matters
A paper can be beautifully reported yet highly biased; poor reporting can also make risk of bias impossible to judge. A diagnostic study requires different appraisal domains from a prediction-model study. Using one generic checklist for every design can therefore produce false reassurance.
Risk-of-bias assessment matters because evidence synthesis should not treat all estimates as equally credible. Cochrane’s framework focuses on bias in a specific result, acknowledging that missing data, outcome measurement or selective reporting can differ across outcomes and time points.[1]
Key concepts
Risk of bias concerns systematic deviation of a result from the truth or target estimate because of design, conduct, analysis or reporting. Applicability concerns whether the study matches the question of interest. Reporting quality concerns completeness and transparency of the article. Certainty of evidence is a body-of-evidence judgement that incorporates risk of bias together with other domains such as inconsistency, indirectness and imprecision; GRADE is used for that broader task.[4]
A numerical checklist score loses this structure. Two studies with “8/10” can have completely different fatal flaws, and an item about sample-size reporting should not necessarily have the same weight as outcome ascertainment bias.
RoB 2 for randomised trials
RoB 2 evaluates bias arising from the randomisation process, deviations from intended interventions, missing outcome data, outcome measurement and selection of the reported result.[1,5] Judgements are outcome/result specific. The correct domain pathway can depend on the effect of interest—for example, the effect of assignment versus an adherence-based effect.
Do not mark a trial “low risk” solely because the abstract says it was randomised and double blind. Appraise sequence generation or allocation process, deviations, missingness, measurement and reporting.
ROBINS-I for non-randomised intervention effects
ROBINS-I compares a non-randomised intervention result conceptually with a hypothetical well-conducted randomised trial addressing the same causal effect. The tool considers confounding and selection as central domains in addition to intervention classification, deviations, missing data, outcome measurement and reporting.
As of August 2026, riskofbias.info provides a revised ROBINS-I V2 draft posted in November 2025 and explicitly notes that it is still subject to change.[2] A lecture or review protocol should record the exact version used.
ROBINS-E for exposure effects
ROBINS-E was developed for non-randomised follow-up studies estimating effects of exposures. A key feature is explicit specification of the causal effect estimated by the result under assessment.[3] The tool addresses confounding, selection, exposure classification, post-exposure interventions, missing data, outcome measurement and reported-result selection through structured signalling questions.
Its use is more demanding than a simple cohort checklist because it requires a causal model and explicit target effect.
QUADAS-2 for diagnostic accuracy
Diagnostic accuracy studies require attention to patient selection, index-test conduct and interpretation, reference-standard conduct and interpretation, and participant flow/timing. QUADAS-2 assesses both risk of bias and applicability concerns.[6] It should not be replaced by a generic observational-study checklist because diagnostic verification and threshold handling are design-specific.
PROBAST+AI for prediction models
Prediction-model studies have risks that differ from etiological studies: inappropriate participant selection, unavailable predictors, outcome definition, inadequate sample size, overfitting, data leakage, optimistic validation and improper performance assessment. PROBAST+AI, published in 2025, updates the original PROBAST framework for regression and artificial-intelligence prediction methods.[7]
A model can have excellent reported discrimination and still be at high risk of bias because evaluation occurred on data that were not sufficiently independent from model development.
Reporting guidelines are not risk-of-bias tools
CONSORT, STROBE, STARD, TRIPOD+AI, SRQR and other reporting guidelines improve transparency and completeness. They are not interchangeable with RoB tools. Checking whether an article reports randomisation does not establish that randomisation was adequate. Conversely, missing reporting may lead to “some concerns” or “no information” because the underlying conduct cannot be determined.
Study-level versus result-level judgement
Older appraisals often assigned one quality label to a whole paper. Modern tools increasingly focus on a specific result because risk can differ by outcome, time point and analysis. A trial may have complete objective mortality data but high risk of bias for a subjective unblinded symptom score. A cohort may measure one exposure accurately and another poorly.
Systematic reviews should define which outcomes and results are appraised and avoid silently transferring one judgement to every analysis.
Signalling questions and domain judgements
Signalling questions structure the evidence used to make a judgement; they are not a questionnaire score. Reviewers should cite page numbers, quotations or extracted evidence supporting each answer. Domain algorithms can guide but do not replace judgement. Disagreements should be resolved through discussion or an agreed adjudication process.
Avoid numerical quality scores
Summing checklist items assumes equal weights, compensation between unrelated domains and a meaningful arithmetic scale—assumptions that are rarely justified. A severe confounding problem cannot be cancelled by good description of the study setting. Present domain-level judgements and explain the mechanisms most likely to affect the result.
Applicability and external validity
Risk of bias can be low while applicability is poor. A diagnostic study in tertiary referral patients may be methodologically sound yet not apply to primary-care screening. A prediction model validated only in one health system may be low risk for that setting but have uncertain transportability. Keep applicability judgements separate from internal bias.
Risk of bias versus certainty of evidence
GRADE operates at the body-of-evidence level and considers risk of bias together with inconsistency, indirectness, imprecision and publication bias.[4] One high-risk study does not mechanically determine the certainty rating, and a low-risk study does not guarantee high certainty. The synthesis question, effect sizes and consistency across studies matter.
Automation and AI-assisted appraisal
Software and language models may help retrieve relevant text or organise signalling questions, but risk-of-bias assessment requires design-specific reasoning and access to the relevant reports, protocols and analysis plans. Automated summaries can omit key details or infer conduct that was not reported. Human methodological judgement remains necessary, especially for causal structure and result-specific domains.
Version control for appraisal tools
Risk-of-bias tools evolve. Record the tool name, exact version/date, adaptations and any domain-specific guidance used. If a protocol is registered before a major update, document whether the review will retain the prespecified version or migrate assessments and why. This is especially relevant to ROBINS-I V2, whose current official page still identifies the November 2025 version as a draft.[2]
Choose the appraisal framework
- Randomised intervention result → RoB 2.
- Non-randomised intervention effect → ROBINS-I, recording the exact current version.
- Non-randomised exposure effect in follow-up data → ROBINS-E when appropriate.
- Diagnostic accuracy → QUADAS-2.
- Prediction model development or evaluation → PROBAST+AI.
- Body-of-evidence certainty → GRADE after study-level risk-of-bias assessment.
- Use reporting guidelines separately to assess completeness of reporting.
Design-to-tool map
illustrativeTable 1. Appraisal tasks are not interchangeable
| Research result | Primary appraisal framework | Main distinctive concerns |
|---|---|---|
| Randomised trial effect | RoB 2 | Randomisation, deviations, missing outcomes, measurement, selective reporting |
| Non-randomised intervention effect | ROBINS-I | Confounding, intervention classification, selection and post-baseline processes |
| Exposure effect in cohort | ROBINS-E | Confounding, exposure classification, selection and post-exposure processes |
| Diagnostic accuracy | QUADAS-2 | Patient selection, index test, reference standard, flow/timing |
| Prediction model | PROBAST+AI | Participants/data, predictors, outcome, analysis, overfitting and evaluation |
| Overall evidence certainty | GRADE | Risk of bias plus inconsistency, indirectness, imprecision and publication bias |
Quality-assurance questions for a risk-of-bias assessment
- Was the tool matched to the study design and result?
- Was the exact tool version recorded?
- Did reviewers support signalling answers with evidence from the article, protocol or supplementary material?
- Were domain judgements made independently by more than one reviewer when feasible?
- Were disagreements resolved transparently?
- Did synthesis and conclusions actually incorporate risk-of-bias findings?
Result-specific causal targets in ROBINS toolsMMed extension
ROBINS-I and ROBINS-E require reviewers to articulate the causal effect that the result aims to estimate. This changes appraisal from checklist compliance to causal reasoning. Confounding domains depend on specifying important confounders; selection domains depend on how participants enter the analysis; deviations or post-exposure intervention domains depend on the target effect. Review teams should therefore agree the causal target and important bias variables before rating individual studies.
Current tool status also matters. The official ROBINS-I V2 page states that the revised November 2025 release remains a draft subject to change, whereas ROBINS-E has a peer-reviewed 2024 publication and an official tool site.[2,3]
Matching the appraisal tool to the causal question
The correct appraisal tool is determined not only by a paper’s label but by the result being appraised. A cohort can be used to estimate an exposure effect, a prognostic association, a diagnostic pathway or a treatment effect. Those questions have different bias structures. Selecting an appraisal checklist solely because the Methods section says “retrospective cohort” can therefore be misleading.
Appraisers should first identify the target result: What intervention, exposure, test or prediction is being evaluated? What comparator is relevant? What outcome and time point matter? Which analysis produces the estimate that will enter the review? Only then should the appropriate domain-based tool be selected. This result-level orientation is especially important when one article reports several outcomes or analyses with different risks of bias.
Keeping risk of bias separate from reporting completeness
Poor reporting can make risk of bias impossible to judge, but the concepts are not identical. A well-conducted study may omit enough methodological detail that reviewers must judge a domain as having insufficient information. Conversely, a paper can report every design decision transparently while describing methods that create a high risk of bias. Reporting guidelines such as CONSORT, STROBE, STARD or TRIPOD help authors disclose what was done; they do not replace a domain-specific appraisal of whether those methods threaten the result.
This distinction also argues against generic numerical “quality scores”. Adding one point for describing randomisation, another for blinding and another for complete follow-up assumes that different domains have comparable weights and that a high total can compensate for a critical flaw. Domain-based tools instead ask how specific mechanisms could bias the result and derive an overall judgement from those mechanisms.
Recording tool versions and decision rules
Risk-of-bias tools evolve. A reproducible review should record the tool name, version or release date, any extension used, the target result, and any review-specific interpretation rules agreed before appraisal. This is particularly important for current non-randomised intervention appraisal: the official ROBINS-I V2 page states that the revised version posted on 20 November 2025 is still a draft and subject to change.[2]
Teams should preserve completed forms rather than only the final colour-coded figure. The form documents signalling-question responses, support for judgement and disagreements. If guidance changes during a long review, the team can then determine whether re-appraisal is required rather than guessing how an earlier judgement was reached.
Calibration exercises are valuable before duplicate appraisal begins. Two reviewers can independently assess the same small set of studies, compare reasoning domain by domain, and agree how protocol-specific issues will be handled. The objective is not forced agreement but consistent application of the chosen framework.
Independent assessment, consensus and auditability
Risk-of-bias appraisal involves judgement, so a reproducible review should make the judgement process visible. For evidence syntheses, two reviewers commonly assess studies independently or use another prespecified verification process, then resolve disagreements through discussion or a third reviewer. The important feature is not the ritual of duplicate checking by itself, but protection against one person silently applying inconsistent thresholds across studies.
Before full appraisal, reviewers should calibrate on several studies that represent the expected range of designs. They should compare not only final domain labels but the reasoning behind signalling-question responses. Review-specific decision rules—for example, how to judge deviations from intended intervention when treatment switching is common—should be documented without rewriting the official tool. Completed forms, supporting quotations or page references, consensus notes and tool versions should be retained as an audit trail.
Agreement statistics can be reported when useful, but a high agreement coefficient does not establish that the judgements are correct. Conversely, genuine methodological ambiguity can produce reasonable disagreement. The quality goal is transparent, defensible and consistently applied reasoning rather than maximising a numerical agreement score.
Interface with statistical analysis
Risk-of-bias judgements inform synthesis, subgrouping, sensitivity analyses and certainty assessment; they are not statistical weights by themselves unless a prespecified evidence-synthesis method uses them. Avoid excluding all “high risk” studies automatically without considering the review question and analysis plan. The Biostatistics critical-appraisal lectures cover statistical errors; this lecture focuses on design-specific bias appraisal.
Worked example
Appraising a mixed evidence base without a quality score
A systematic review includes 6 randomised reminder trials, 9 observational exposure cohorts, 4 diagnostic studies and 7 prediction-model studies. A novice reviewer proposes a single 10-item checklist. The review team instead defines result-specific appraisal pathways.
Randomised intervention effects are assessed with RoB 2. Cohorts estimating effects of access burden use ROBINS-E because the review question concerns an exposure effect. Diagnostic studies use QUADAS-2, and prediction models use PROBAST+AI. Two non-randomised studies compare service interventions, so the team uses the current ROBINS-I framework and records that the November 2025 V2 release is a draft version.
Each assessment cites evidence and gives domain judgements. The review does not calculate a mean quality score. In synthesis, the team presents main estimates alongside risk-of-bias patterns and performs a prespecified sensitivity analysis excluding results at critical or high risk where the tool and review protocol support that distinction. GRADE is then used for certainty of the body of evidence, incorporating but not duplicating the study-level risk-of-bias assessments.
The final review is more work than scoring checkboxes, but the appraisal now explains why particular results are less trustworthy.
Suggested protocol or reporting wording
Common mistakes
Knowledge check
A randomised trial reports its methods incompletely. Is poor reporting the same as high risk of bias?
Answer and reasoning
Not necessarily. Poor reporting can make bias impossible to judge and lead to concerns or no-information judgements, but reporting quality and actual conduct are distinct. Additional protocol information may clarify the risk.
Why is a 7/10 quality score problematic?
Answer and reasoning
It treats heterogeneous domains as equally weighted and allows strengths in trivial items to compensate mathematically for severe bias. Domain-specific mechanisms are more informative.
Which tool is appropriate for a diagnostic accuracy study?
Answer and reasoning
QUADAS-2 is the established design-specific risk-of-bias and applicability tool for diagnostic accuracy studies.
What is the current status of ROBINS-I V2 as of August 2026?
Answer and reasoning
The official riskofbias.info page provides a revised version posted in November 2025 and states that it remains a draft subject to change. Reviews should record the exact version used.
How does GRADE differ from RoB 2?
Answer and reasoning
RoB 2 assesses risk of bias in a specific randomised-trial result. GRADE assesses certainty across a body of evidence and includes risk of bias plus inconsistency, indirectness, imprecision and publication bias.
Key take-home messages
Key take-home messages
- Risk of bias, reporting quality, applicability and evidence certainty are distinct appraisal tasks.
- Match the appraisal tool to the study design and target result.
- Use domain judgements and signalling questions rather than arbitrary total quality scores.
- Risk of bias is often result-specific, not one permanent label for an article.
- Record tool versions, especially for evolving frameworks such as ROBINS-I V2.
- Appraisal findings should change interpretation and synthesis, not remain an isolated checklist exercise.
References
- Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
- Risk of Bias Tools. ROBINS-I V2 tool: revised draft version posted 20 November 2025. Accessed 20 August 2026. https://www.riskofbias.info/welcome/robins-i-v2
- Higgins JPT, Morgan RL, Rooney AA, Taylor KW, Thayer KA, Silva RA, et al. A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E). Environ Int. 2024;186:108602. https://doi.org/10.1016/j.envint.2024.108602
- Schünemann HJ, Vist GE, Higgins JPT, Santesso N, Deeks JJ, Glasziou P, et al. Chapter 14: Completing Summary of Findings tables and grading the certainty of the evidence. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-14
- Risk of Bias Tools. RoB 2: current version. Accessed 20 August 2026. https://www.riskofbias.info/welcome/rob-2-0-tool/current-version-of-rob-2
- Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529-536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
- Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro CL, Dhiman P, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. https://doi.org/10.1136/bmj-2024-082505
These materials support learning and research planning. They require adaptation to the study context and do not replace methodological, statistical, ethical, or regulatory consultation.