Screening and Organising the Literature

Overview

Screening determines which retrieved records contribute to the review. It applies pre-specified eligibility criteria in stages, commonly title-and-abstract screening followed by full-text assessment. The process should be consistent, auditable and sufficiently inclusive to avoid discarding relevant evidence on the basis of incomplete citation information.

Screening is not merely administrative. Ambiguous criteria, inconsistent reviewer judgement and undocumented exclusions can change the evidence base as substantially as an inadequate search. Cochrane guidance therefore links pre-specified selection criteria, duplicate assessment and transparent record flow [1].

Why this matters

Titles and abstracts are brief and may omit setting, sample characteristics, outcomes or study design. An apparently irrelevant abstract may contain eligible data in the full report. Conversely, a promising title may fail key criteria after full-text review.

Review teams therefore need operational definitions, training, calibration and a structured method for resolving uncertainty. PRISMA 2020 requires transparent reporting of record flow and reasons for full-text exclusion in systematic reviews [2].

From question to eligibility criteria

Eligibility criteria should specify the domains that determine relevance.

Population

Define who or what is eligible. Clarify diagnostic definitions, age range, pregnancy history, disease stage or mixed-population rules. If mixed samples are eligible only when subgroup data can be separated, state this prospectively.

Concept, exposure or intervention

Define what counts as the phenomenon of interest. For barriers, decide whether quantitative predictors, qualitative experiences and implementation determinants are all eligible.

Comparator

A comparator may be required for an intervention-effect review but unnecessary for descriptive or qualitative questions.

Outcomes or evidence contribution

Define the minimum relevant information. A paper may concern the population but provide no data addressing the review question.

Context and setting

Specify care settings, countries, service levels or time periods only when these boundaries are relevant to the purpose.

Study and source types

State eligible designs and publication forms. Avoid excluding designs solely because they are unfamiliar. The appropriate evidence type follows from the question.

Time, language and publication status

Justify restrictions and plan how translated or unpublished records will be handled.

Operational eligibility criteria

illustrative

Table 1. Criteria for the postpartum follow-up review

Domain	Include	Exclude
Population	Women after a pregnancy affected by gestational diabetes	Studies limited to pre-existing type 1 or type 2 diabetes
Concept	Patient, provider or system barrier/facilitator to postpartum glucose testing	General diabetes knowledge with no link to postpartum testing
Context	Maternity, primary-care, community or linked follow-up services	Prevention programmes starting beyond the defined postpartum pathway unless testing barriers are reported
Evidence	Qualitative findings, quantitative determinants, implementation data	Completion rate alone without information on determinants
Source	Full articles, theses, reports and eligible abstracts	Editorials with no empirical or programme data

Note: The exact postpartum interval must be defined in the protocol.

Title-and-abstract screening

The first stage removes records that are clearly ineligible. It should favour sensitivity because information is incomplete.

A practical decision set is:

• include or retrieve full text;
• exclude; or
• uncertain.

Uncertain records should normally proceed to full text. Excluding only when a criterion is clearly violated reduces premature loss.

DECISION GUIDE

Exclude at title and abstract only when clearly ineligible

Examples of clear exclusion may include:

• wrong population with no relevant subgroup;
• wrong concept or intervention;
• non-human research when human studies are required;
• clearly ineligible publication type; or
• no plausible relationship to the review question.

Do not infer absence of an outcome, context or subgroup merely because the abstract does not mention it.

Full-text eligibility assessment

At full text, apply all criteria and record one primary reason for exclusion. A hierarchical reason list prevents inconsistent coding. For example:

1. wrong population;
2. wrong concept or intervention;
3. wrong outcome or evidence contribution;
4. wrong context;
5. wrong design or source type;
6. duplicate report with no additional information; or
7. insufficient information after reasonable attempts.

The primary reason should be the first decisive criterion in the hierarchy, not every possible problem.

Reviewer processes

For formal systematic and scoping reviews, independent assessment by at least two reviewers is commonly used to reduce individual error and subjective selection. Exact arrangements may vary by review type and resources, but deviations should be justified.

Training and calibration

Before full screening, reviewers should independently assess a pilot set and discuss disagreements. Revise ambiguous guidance before proceeding. Calibration is intended to improve shared interpretation, not force identical judgement without discussion. Best-practice guidance recommends piloting and documenting decision rules, particularly in large reviews [3].

Resolving disagreements

Resolve disagreements through discussion, consulting the protocol and, when necessary, a third reviewer. Agreement statistics may describe consistency, but they do not replace examination of why reviewers interpreted criteria differently [4]. Record substantive rule clarifications in a decision log and apply them consistently to earlier records if needed.

Blinding

Screening tools such as Rayyan may conceal the other reviewer’s decision until both have voted [5]. Blinding to journal, authors or results is more difficult and not routinely complete, but decisions should remain criterion-based.

Single screening and prioritisation toolsMMed extension

Machine learning and prioritisation tools can reduce workload, but human oversight and transparent stopping rules remain essential. The acceptable workflow depends on review purpose, risk tolerance and validation. A high-stakes review aiming for comprehensive recall should not silently replace independent screening with one unverified reviewer or an opaque automated decision.

Managing full texts

Track whether each full text is:

• obtained;
• unavailable after stated attempts;
• awaiting translation;
• linked to another report;
• excluded with reason; or
• included.

Document attempts to retrieve inaccessible reports. Do not automatically exclude potentially eligible evidence because the institution lacks a subscription.

Organising included literature

An evidence table converts a pile of papers into a structured knowledge base. Common fields include:

• study and report identifiers;
• country and setting;
• design and study period;
• population and sample size;
• recruitment or data source;
• definitions and measurements;
• intervention or exposure;
• outcomes or themes;
• main findings;
• methodological limitations; and
• relevance to the review question.

The table should distinguish extracted facts from reviewer interpretation.

Initial evidence table

illustrative

Table 2. Example organisation of included studies

Study ID	Design	Setting	Evidence contribution	Linked reports
S001	Qualitative interviews	Urban maternity clinic	Travel, childcare and uncertainty about referral	Thesis; article
S002	Retrospective cohort	Regional health system	Distance and appointment timing associated with completion	Main article
S003	Cluster trial	Primary-care clinics	Reminder and navigation intervention	Registry; abstract; article

Note: Detailed appraisal and results extraction would use additional fields.

Screening reviews versus primary-study literature reviews

A proposal background review may use a less formal process than a standalone systematic review. Even so, the researcher should define source-selection principles, keep notes on why key papers were included and avoid selecting only supportive evidence. Claims of comprehensive identification require methods capable of supporting that claim.

Quality assurance

PROTOCOL CHECKLIST

Screening quality-assurance plan

• Pilot the eligibility criteria on a diverse sample.
• Create a written decision guide with examples.
• Define who screens each stage and how disagreements are resolved.
• Preserve source and record identifiers.
• Record one standardised primary reason for each full-text exclusion.
• Maintain a decision and amendment log.
• Track unavailable and translated reports.
• Link multiple reports to one study.
• Recheck records affected by a clarified rule.
• Reconcile final counts before producing the flow diagram.

Worked example

WORKED EXAMPLE

Screening the postpartum barrier literature

Record A: “Postpartum experiences after gestational diabetes”—abstract does not mention testing. The record proceeds to full text because testing barriers may be reported.

Record B: Audit of test-completion proportion only. The full text confirms no determinant, experience or implementation information. It is excluded for wrong evidence contribution.

Record C: Survey of clinician guideline knowledge. It includes questions about referral barriers and follow-up systems, so it is included as provider-level evidence.

Record D: Reminder trial. It is included because implementation barriers and intervention delivery data are reported, even though the primary trial outcome is completion.

Record E: Narrative review. It is not included as primary evidence but is retained for citation searching and contextual interpretation.

The decision log records that provider-level evidence is eligible when it directly concerns postpartum testing pathways.

Appraising screening methods

APPRAISAL CHECKLIST

Screening appraisal questions

• Are eligibility criteria explicit and operational?
• Were criteria defined before results were known?
• Was title-and-abstract screening appropriately inclusive?
• Were full-text exclusions assigned standard reasons?
• Were reviewers trained and disagreements resolved transparently?
• Were inaccessible and non-English reports handled systematically?
• Were multiple reports linked to one study?
• Can the reported flow counts be reconstructed?
• Did any exclusion depend on the study’s findings?

Suggested protocol or reporting wording

Common mistakes

Knowledge check

An abstract does not state the postpartum testing interval. Should it be excluded at title-and-abstract screening?

Answer and reasoning

Usually not. The missing interval may be reported in the full text. If the record is otherwise potentially relevant, retrieve it and apply the interval criterion at full-text assessment.

Why is ‘poor quality’ usually not an appropriate full-text exclusion reason?

Answer and reasoning

Eligibility determines whether a study addresses the review question. Methodological quality or risk of bias is generally assessed after inclusion using explicit criteria. Excluding studies through an undefined quality judgement can introduce selective bias.

A registry record and journal article describe the same trial. How should they be handled?

Answer and reasoning

Retain both as linked reports of one underlying study. Use the combined information during appraisal and extraction, but count the participants and study only once in synthesis.

What is the purpose of reviewer calibration?

Answer and reasoning

Calibration identifies ambiguous criteria and develops a shared interpretation before large-scale screening. It improves consistency and allows the decision guide to be refined prospectively.

Key take-home messages

SUMMARY

Key take-home messages

• Operational eligibility criteria convert the review question into reproducible decisions.
• Early screening should be inclusive when citation information is incomplete.
• Full-text exclusions require one transparent, standardised reason.
• Reviewer training, calibration and documented disagreement resolution improve consistency.
• Organise included evidence at both report and study levels.

References

1. Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Metzendorf M-I, et al. Technical Supplement to Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5. Cochrane; 2024.
2. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71
3. Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–342. https://doi.org/10.1002/jrsm.1354
4. Belur J, Tompson L, Thornton A, Simon M. Interrater reliability in systematic review methodology: exploring variation in coder decision-making. Sociol Methods Res. 2021;50(2):837–865. https://doi.org/10.1177/0049124118799372
5. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:210. https://doi.org/10.1186/s13643-016-0384-4