Part 9 · Module 10
Quasi-Experimental Designs
Evaluate interventions when randomisation is unavailable using controlled before–after, interrupted time-series, regression-discontinuity and difference-in-differences designs.
Learning outcomes
After completing this lecture, you should be able to:
- Explain why quasi-experiments seek a credible counterfactual without random assignment.
- Compare major quasi-experimental designs.
- Identify design-specific identifying assumptions.
- Plan timing, comparison groups and sensitivity analyses.
- Avoid attributing all before–after change to an intervention.
Overview
Quasi-experimental designs estimate intervention effects without random assignment by exploiting comparison groups, repeated time points, thresholds, phased policies or natural allocation processes. Their credibility depends on a transparent assignment mechanism and design-specific assumptions, not merely statistical adjustment.
The study question, design, participant pathway, measurements and analysis must be specified as one coherent system. A recognised design label is useful only when the operational details preserve the comparison or prediction that the study is intended to support.[1–3]
Why this matters
A simple before–after comparison cannot separate intervention effects from secular trends, seasonality, regression to the mean, changes in measurement or co-occurring policies. Strong quasi-experiments make the counterfactual visible and testable using pre-intervention patterns, negative controls, unaffected outcomes or threshold diagnostics.
Core concepts
Controlled before–after designs add a concurrent comparison
Outcome changes in intervention sites are compared with changes in similar unexposed sites. Baseline differences can remain, and parallel trends must be plausible.
Interrupted time series models level and slope changes
Multiple observations before and after a clearly dated intervention estimate immediate and gradual changes while accounting for baseline trend and seasonality. Other events at the interruption can threaten attribution.
Difference-in-differences compares changes
The intervention-group change is contrasted with the control-group change. The key assumption is that, without intervention, outcome trends would have evolved in parallel. Pre-trend data and event-study plots support but do not prove this.
Regression discontinuity exploits a threshold
Assignment determined by a continuous score crossing a cut-off can create locally comparable groups. Manipulation around the threshold, incorrect functional form and limited local generalisability are central concerns.
Natural experiments require a plausibly exogenous mechanism
Policy timing, lotteries or administrative rules can approximate random variation when unrelated to potential outcomes. The mechanism must be documented and alternative pathways examined.
When to use this approach
Selecting a quasi-experimental design
- Is there a clearly dated intervention or assignment rule?
- Are enough pre-intervention observations available?
- Is a credible concurrent comparison population available?
- Does a threshold determine exposure with limited manipulation?
- Which co-interventions or shocks could coincide with assignment?
Working through the key design decisions
Decision 1: Is there a clearly dated intervention or assignment rule?. This point should be resolved before recruitment, extraction or assignment begins. For quasi-experimental designs, the protocol should state the operational rule, who applies it, when it is applied and what source documents demonstrate compliance. The investigators should also consider how a different choice would change the eligible population, comparison, timing, measurement or interpretation. A pilot review of realistic records or participant pathways can reveal ambiguities before they become irreversible exclusions or post hoc analytical choices. The decision should be reviewed across sites and subgroups because a rule that works in one clinic may operate differently where referral, resources or disease spectrum differ. Any permitted exception should be explicit, dated and reported with its frequency and likely consequence. The corresponding estimand or target contrast should be written in words, and the planned estimator should be checked against the sampling, clustering, matching or censoring mechanism. Sensitivity analyses should target plausible departures from the design assumptions rather than provide an undirected collection of alternative models.
Decision 2: Are enough pre-intervention observations available?. This point should be resolved before recruitment, extraction or assignment begins. For quasi-experimental designs, the protocol should state the operational rule, who applies it, when it is applied and what source documents demonstrate compliance. The investigators should also consider how a different choice would change the eligible population, comparison, timing, measurement or interpretation. A pilot review of realistic records or participant pathways can reveal ambiguities before they become irreversible exclusions or post hoc analytical choices. The decision should be reviewed across sites and subgroups because a rule that works in one clinic may operate differently where referral, resources or disease spectrum differ. Any permitted exception should be explicit, dated and reported with its frequency and likely consequence. The corresponding estimand or target contrast should be written in words, and the planned estimator should be checked against the sampling, clustering, matching or censoring mechanism. Sensitivity analyses should target plausible departures from the design assumptions rather than provide an undirected collection of alternative models.
Decision 3: Is a credible concurrent comparison population available?. This point should be resolved before recruitment, extraction or assignment begins. For quasi-experimental designs, the protocol should state the operational rule, who applies it, when it is applied and what source documents demonstrate compliance. The investigators should also consider how a different choice would change the eligible population, comparison, timing, measurement or interpretation. A pilot review of realistic records or participant pathways can reveal ambiguities before they become irreversible exclusions or post hoc analytical choices. The decision should be reviewed across sites and subgroups because a rule that works in one clinic may operate differently where referral, resources or disease spectrum differ. Any permitted exception should be explicit, dated and reported with its frequency and likely consequence. The corresponding estimand or target contrast should be written in words, and the planned estimator should be checked against the sampling, clustering, matching or censoring mechanism. Sensitivity analyses should target plausible departures from the design assumptions rather than provide an undirected collection of alternative models.
Decision 4: Does a threshold determine exposure with limited manipulation?. This point should be resolved before recruitment, extraction or assignment begins. For quasi-experimental designs, the protocol should state the operational rule, who applies it, when it is applied and what source documents demonstrate compliance. The investigators should also consider how a different choice would change the eligible population, comparison, timing, measurement or interpretation. A pilot review of realistic records or participant pathways can reveal ambiguities before they become irreversible exclusions or post hoc analytical choices. The decision should be reviewed across sites and subgroups because a rule that works in one clinic may operate differently where referral, resources or disease spectrum differ. Any permitted exception should be explicit, dated and reported with its frequency and likely consequence. The corresponding estimand or target contrast should be written in words, and the planned estimator should be checked against the sampling, clustering, matching or censoring mechanism. Sensitivity analyses should target plausible departures from the design assumptions rather than provide an undirected collection of alternative models.
Decision 5: Which co-interventions or shocks could coincide with assignment?. This point should be resolved before recruitment, extraction or assignment begins. For quasi-experimental designs, the protocol should state the operational rule, who applies it, when it is applied and what source documents demonstrate compliance. The investigators should also consider how a different choice would change the eligible population, comparison, timing, measurement or interpretation. A pilot review of realistic records or participant pathways can reveal ambiguities before they become irreversible exclusions or post hoc analytical choices. The decision should be reviewed across sites and subgroups because a rule that works in one clinic may operate differently where referral, resources or disease spectrum differ. Any permitted exception should be explicit, dated and reported with its frequency and likely consequence. The corresponding estimand or target contrast should be written in words, and the planned estimator should be checked against the sampling, clustering, matching or censoring mechanism. Sensitivity analyses should target plausible departures from the design assumptions rather than provide an undirected collection of alternative models.
The approach should be chosen because its structure answers the research question, not because the available dataset happens to resemble a familiar table. The investigator should state the target population, time origin, comparison, outcome and intended measure before collecting or extracting data.
Design and participant pathway
Quasi-experimental intervention evaluation
Core structure: Use non-random but structured intervention assignment and a design-specific counterfactual to estimate change attributable to the intervention.
Best suited for: Policies, service rollouts, thresholds and system interventions where randomisation is infeasible.
Principal threats: Confounding by time, non-parallel trends, concurrent policies, anticipation, regression to the mean, threshold manipulation and model extrapolation.
The protocol should be written before outcome inspection when possible and include a causal diagram, intervention timeline and catalogue of concurrent events. More than one design feature can strengthen inference: an interrupted series with a control series, unaffected outcomes or staggered adoption. Analytical flexibility should be limited through prespecified time windows and functional forms.
Protocol specification
Decisions to prespecify
- Document the assignment mechanism and intervention date.
- Define intervention and comparison populations.
- Collect adequate pre- and post-intervention observations.
- Model baseline trend, seasonality and autocorrelation.
- Assess parallel trends or threshold continuity.
- Plan falsification, negative-control and sensitivity analyses.
Protocol item 1: Document the assignment mechanism and intervention date. This item requires a reproducible definition rather than a broad intention. The study manual should identify the responsible role, data field or procedural record, permissible time window, quality check and response to deviations. Where judgement is unavoidable, assessors should receive examples and escalation rules, and a sample of decisions should be reviewed for consistency. The analysis dataset should retain enough provenance to distinguish planned exclusions, measurement failure, participant withdrawal and genuinely absent information. Because advanced designs often use weights, risk sets, clusters, thresholds or time-varying states, the derivation should be independently reproduced from raw dates and identifiers. Simulation or mock-data testing before database lock can verify that the prespecified estimator answers the stated question under the intended design.
Protocol item 2: Define intervention and comparison populations. This item requires a reproducible definition rather than a broad intention. The study manual should identify the responsible role, data field or procedural record, permissible time window, quality check and response to deviations. Where judgement is unavoidable, assessors should receive examples and escalation rules, and a sample of decisions should be reviewed for consistency. The analysis dataset should retain enough provenance to distinguish planned exclusions, measurement failure, participant withdrawal and genuinely absent information. Because advanced designs often use weights, risk sets, clusters, thresholds or time-varying states, the derivation should be independently reproduced from raw dates and identifiers. Simulation or mock-data testing before database lock can verify that the prespecified estimator answers the stated question under the intended design.
Protocol item 3: Collect adequate pre- and post-intervention observations. This item requires a reproducible definition rather than a broad intention. The study manual should identify the responsible role, data field or procedural record, permissible time window, quality check and response to deviations. Where judgement is unavoidable, assessors should receive examples and escalation rules, and a sample of decisions should be reviewed for consistency. The analysis dataset should retain enough provenance to distinguish planned exclusions, measurement failure, participant withdrawal and genuinely absent information. Because advanced designs often use weights, risk sets, clusters, thresholds or time-varying states, the derivation should be independently reproduced from raw dates and identifiers. Simulation or mock-data testing before database lock can verify that the prespecified estimator answers the stated question under the intended design.
Protocol item 4: Model baseline trend, seasonality and autocorrelation. This item requires a reproducible definition rather than a broad intention. The study manual should identify the responsible role, data field or procedural record, permissible time window, quality check and response to deviations. Where judgement is unavoidable, assessors should receive examples and escalation rules, and a sample of decisions should be reviewed for consistency. The analysis dataset should retain enough provenance to distinguish planned exclusions, measurement failure, participant withdrawal and genuinely absent information. Because advanced designs often use weights, risk sets, clusters, thresholds or time-varying states, the derivation should be independently reproduced from raw dates and identifiers. Simulation or mock-data testing before database lock can verify that the prespecified estimator answers the stated question under the intended design.
Protocol item 5: Assess parallel trends or threshold continuity. This item requires a reproducible definition rather than a broad intention. The study manual should identify the responsible role, data field or procedural record, permissible time window, quality check and response to deviations. Where judgement is unavoidable, assessors should receive examples and escalation rules, and a sample of decisions should be reviewed for consistency. The analysis dataset should retain enough provenance to distinguish planned exclusions, measurement failure, participant withdrawal and genuinely absent information. Because advanced designs often use weights, risk sets, clusters, thresholds or time-varying states, the derivation should be independently reproduced from raw dates and identifiers. Simulation or mock-data testing before database lock can verify that the prespecified estimator answers the stated question under the intended design.
Protocol item 6: Plan falsification, negative-control and sensitivity analyses. This item requires a reproducible definition rather than a broad intention. The study manual should identify the responsible role, data field or procedural record, permissible time window, quality check and response to deviations. Where judgement is unavoidable, assessors should receive examples and escalation rules, and a sample of decisions should be reviewed for consistency. The analysis dataset should retain enough provenance to distinguish planned exclusions, measurement failure, participant withdrawal and genuinely absent information. Because advanced designs often use weights, risk sets, clusters, thresholds or time-varying states, the derivation should be independently reproduced from raw dates and identifiers. Simulation or mock-data testing before database lock can verify that the prespecified estimator answers the stated question under the intended design.
A protocol should make these decisions before outcome-dependent choices become tempting. It should also distinguish the design population from the analysis population, document exclusions after enrolment or sampling, and explain how missing or indeterminate observations will be handled. Any deviation should be dated, justified and retained in the study record.[2–4]
Measurements, timing and analysis interface
Outcome definitions and data systems must remain stable across the interruption and groups. Coding changes can create artificial level shifts. Implementation may be gradual rather than instantaneous, requiring transition periods or dose measures. Exposure should be assigned from policy eligibility or actual rollout according to the target question.
The analysis should reflect how participants or clusters entered the study, how observations are related, and how the effect or performance measure was defined. Statistical adjustment cannot repair a comparison that was invalid at the design stage. Conversely, a sound design can still be undermined by an analysis that ignores matching, clustering, censoring, sampling fractions or prespecified thresholds.
Worked example
Controlled interrupted time series
The team models monthly testing proportions for 36 pre-policy and 24 post-policy months in both regions. The model includes baseline level and trend, seasonality, region, intervention timing and interactions representing differential level and slope changes. The neighbouring region’s pre-policy trend is examined, coding changes are audited and a negative-control outcome unrelated to reminders is analysed.
The example should be interpreted in the population and time frame created by the design. The numerical result does not by itself establish causation, transportability or clinical usefulness; those conclusions require additional assumptions and evidence.
Appraising methodological quality
Questions for critical appraisal
- Assignment: Why did intervention occur where and when it did?
- Counterfactual: What comparison represents no intervention?
- Pre-trends: Are baseline trajectories and shocks described?
- Measurement: Did outcome definitions remain stable?
- Robustness: Were alternative dates, bandwidths or control outcomes examined?
A reporting guideline can improve transparency, but it is not a substitute for sound design. Appraisal should ask whether the reported methods could generate an unbiased, clinically meaningful answer—not merely whether every checklist item has been mentioned.[3–6]
Suggested reporting
Reporting should identify the design in the title or abstract when appropriate, describe the source and flow of participants, define all key variables, report missing and indeterminate observations, and present estimates with uncertainty. The direction and denominator of every comparison must be explicit.
Emulating a target trialMMed extension
Even without randomisation, the study can specify eligibility, strategies, assignment, follow-up, outcomes, causal contrast and analysis as a target trial. Quasi-experimental assignment mechanisms may strengthen exchangeability, but contamination, adherence and time-varying confounding still require explicit treatment.
Common mistakes
Knowledge check
Why is one before and one after point weak?
Answer and reasoning
It cannot separate intervention effects from background trend, seasonality, random fluctuation or regression to the mean.
What is the key difference-in-differences assumption?
Answer and reasoning
Without intervention, the outcome trends in intervention and comparison groups would have evolved in parallel.
What does regression discontinuity estimate?
Answer and reasoning
A local treatment effect near the assignment threshold under continuity and no manipulation assumptions.
How can an interrupted series be strengthened?
Answer and reasoning
Add a concurrent control series, stable measurements, sufficient pre-points, modelling of seasonality and falsification outcomes.
Key take-home messages
Key take-home messages
- Build a visible counterfactual from assignment and comparison.
- Use repeated pre-intervention data.
- State design-specific assumptions.
- Audit measurement and concurrent events.
- Use robustness analyses tied to plausible threats.
References
- Des Jarlais DC, Lyles C, Crepaz N; TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004;94(3):361–366. https://doi.org/10.2105/AJPH.94.3.361
- Sterne JAC, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. https://doi.org/10.1136/bmj.i4919
- Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions. Int J Epidemiol. 2017;46(1):348–355. https://doi.org/10.1093/ije/dyw098
- Wing C, Simon K, Bello-Gomez RA. Designing difference in difference studies. Annu Rev Public Health. 2018;39:453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507
- Craig P, Cooper C, Gunnell D, et al. Using natural experiments to evaluate population health interventions. J Epidemiol Community Health. 2012;66(12):1182–1186. https://doi.org/10.1136/jech-2011-200375
- Lee DS, Lemieux T. Regression discontinuity designs in economics. J Econ Lit. 2010;48(2):281–355. https://doi.org/10.1257/jel.48.2.281
These materials support learning and research planning. They require adaptation to the study context and do not replace methodological, statistical, ethical, or regulatory consultation.