Aligning Question, Framework, Design and Analysis

Overview

A coherent study has a traceable chain:

problem → evidence gap → research question → objectives → framework → design → population → variables and measurements → data structure → analysis → interpretation

Alignment means that each link supports the next. The analysis should estimate the quantity implied by the objective, using data that the design can actually generate. The framework should explain why the selected variables and pathways matter. The protocol should not promise causal, predictive or experiential conclusions that its design cannot support.

This lecture integrates the earlier work on questions, constructs, operational definitions, variable roles and frameworks. It prepares the reader for later parts on measurement, sampling and specific study designs.

Why this matters

Many protocol weaknesses are alignment failures rather than isolated technical mistakes. Examples include:

• an objective to determine incidence using a cross-sectional survey;
• a causal objective with no comparison group or time order;
• a prediction objective with no validation plan;
• a qualitative question answered only by closed questionnaire items;
• an outcome named in the objective but absent from the data-collection form;
• repeated measurements analysed as if they were independent; and
• a sample-size calculation based on a different primary outcome from the one analysed.

An alignment review performed before data collection can prevent expensive redesign, missing variables and uninterpretable results.

Start with the intended inference

Before choosing a design, state what kind of conclusion is sought.

Descriptive inference

What exists, how often, in whom, where or when? Examples include prevalence, incidence, distribution and service coverage.

Associational or analytical inference

How are variables related after describing a comparison structure? Association does not automatically imply causation.

Causal inference

What would happen to the outcome under one intervention or exposure condition compared with another? This requires a well-defined contrast, temporality and assumptions about exchangeability, consistency and positivity.

Predictive inference

How accurately can future or unknown outcomes be estimated for new individuals or settings using information available at a defined prediction time?

Interpretive inference

How do people understand, experience or navigate a phenomenon? This may require qualitative methods and attention to context and reflexivity.

Implementation inference

How, why and under what contextual conditions is an intervention adopted, delivered and sustained?

The alignment chain

1. Problem and evidence gap

State the consequential uncertainty, not only the topic. Explain why existing evidence does not resolve it in the relevant population or context.

2. Research question

Define population, exposure or intervention, comparison, outcome or phenomenon, time and setting as appropriate.

3. Objectives

Each specific objective should contain one principal action and correspond to an obtainable result. Avoid combining incompatible purposes in one sentence.

4. Framework

Identify concepts, causal assumptions, mechanisms, context and measurement domains. The framework should justify the selected variables and anticipated relationships.

5. Design

Choose the design that can create the required comparison, time order, sampling pathway or experiential evidence.

6. Population and sampling

Ensure that eligible and recruited participants can represent the target of inference. A causal design can still be poorly applicable if the sample is narrowly selected.

7. Variables and measurement

Operationalise every outcome, exposure, intervention, predictor, confounder and process measure needed for the objectives.

8. Data structure

Identify units, clusters, repeated measurements, time-to-event data, qualitative sources and linked records.

9. Analysis

For each objective, specify the estimand or output, summary, comparison, uncertainty and method class. Detailed syntax can follow later.

10. Interpretation

State the conclusion the design can support and important alternatives it cannot exclude.

DECISION GUIDE

Run the backward test

For each planned conclusion, ask:

1. Which estimate or qualitative finding supports it?
2. Which variables or data produce that result?
3. Which design creates the necessary comparison or perspective?
4. Which objective authorised the analysis?
5. Which research question and evidence gap make the conclusion relevant?

If any link is missing, revise before data collection.

The objective-to-analysis matrix

A protocol benefits from a matrix that links every objective to its data and output.

Objective-to-analysis matrix

illustrative

Table 1. Alignment matrix for a staged postpartum study

Objective	Design and population	Main variables or data	Target output	Analysis interface
Estimate completion by day 112	linked-record cohort of eligible deliveries	operationally defined completion and eligibility	proportion with 95% CI	estimation of one proportion
Identify stages of pathway loss	service-process cohort	referral, booking, attendance and result stages	stage-specific proportions	descriptive flow and clustered summaries
Explore barriers and facilitators	purposively sampled interviews	narratives from women and staff	themes and explanatory account	qualitative analysis
Estimate effect of booking	randomised or defensible quasi-experimental comparison	booking strategy, outcome, baseline covariates	causal contrast with 95% CI	effect estimation aligned with design
Develop a risk model	prospective prediction cohort	baseline predictors available at discharge	validated predicted risk	prediction-model development and validation

The matrix makes clear that one design may not answer every objective. A programme of linked studies can be more credible than a single overloaded project.

Defining the estimand

An estimand is the precise target quantity. For a treatment-effect question, it clarifies the population, conditions being compared, outcome variable, handling of events after assignment and population-level summary [1].

For an observational or service study, a practical estimand statement can specify:

• target population;
• exposure or intervention contrast;
• outcome and time horizon;
• treatment or exposure strategies;
• handling of competing or intercurrent events;
• effect measure; and
• analysis population.

The estimator is the statistical method used to estimate that target. The estimate is the numerical result. Selecting an estimator before defining the estimand invites analysis-driven questions.

From objective to estimand

illustrative

Table 2. Clarifying the treatment-effect target

Attribute	Specification
Population	eligible women discharged after a pregnancy with gestational diabetes
Strategies	booked postpartum test appointment before discharge versus usual referral
Outcome	laboratory-confirmed testing completion by day 112
Intercurrent events	transfer of care, repeat pregnancy, inability to attend and death considered explicitly
Summary	risk difference and risk ratio between strategies
Target interpretation	effect of adopting the booking strategy in the eligible population under the specified design assumptions

Common alignment patterns by purpose

Descriptive study

Question: What proportion completes testing?

Needs: a defined denominator, representative or clearly described sampling pathway, operational outcome and uncertainty interval.

Does not require: labelling every correlate a risk factor or claiming causation.

Aetiological or causal observational study

Question: What is the effect of an exposure on an outcome?

Needs: temporality, target effect, causal framework, confounder measurement, selection assessment and design-compatible estimator.

Does not gain credibility merely by adding many covariates.

Intervention study

Question: What would happen under intervention versus comparison?

Needs: intervention specification, assignment mechanism, comparator, outcome, follow-up, adherence/intercurrent-event strategy and analysis aligned with allocation.

Prediction study

Question: Can outcome risk be estimated at a defined decision point?

Needs: prediction time, candidate predictors available then, outcome horizon, adequate events, validation and performance measures.

Does not require predictors to be causal.

Qualitative study

Question: How is the pathway experienced and understood?

Needs: appropriate qualitative design, purposive sampling, rich data, reflexivity, analytic transparency and context.

Does not become mixed methods merely because participant characteristics are tabulated.

Checking temporal alignment

Draw a timeline containing:

• eligibility assessment;
• baseline measurement;
• exposure or intervention assignment;
• mediator or process measurements;
• outcome measurements;
• censoring, withdrawal and competing events; and
• analysis time origin.

A predictor measured after the outcome cannot predict it prospectively. A confounder measured after exposure may already be affected by exposure. A mediator measured at the same time as outcome may not establish pathway order.

Checking population alignment

Distinguish target, source and study populations. The available sample must support the intended generalisation.

Ask:

• Does the recruitment setting exclude people central to the question?
• Is participation related to exposure and outcome?
• Does routine-data coverage differ across groups?
• Are findings intended for all eligible women, only service users or only interview participants?
• Does the effect depend on clinic resources absent elsewhere?

Internal validity and applicability are connected but distinct. A precise estimate in a selected sample may not answer the service’s target question.

Checking measurement alignment

For every objective, verify that:

• the primary concept is operationally defined;
• instrument and source match the intended construct;
• measurement timing supports the role;
• categories and thresholds are justified;
• data quality is feasible; and
• missingness can be distinguished from absence.

A data-collection form should be reviewed against the objective matrix. Each essential variable must have a source, field and quality rule. Each non-essential field should have a justification because unnecessary data increase burden and governance risk.

PROTOCOL CHECKLIST

Pre-data-collection alignment review

• one principal question for each objective;
• intended inference stated;
• framework pathway linked to each variable;
• target, source and study populations distinguished;
• design supports comparison and time order;
• outcome, exposure and covariates operationalised;
• unit of observation and analysis identified;
• clustering and repeated measures recognised;
• estimand or qualitative output specified;
• sample-size basis matches the primary objective;
• analysis method class matches the data structure; and
• planned conclusion is no stronger than the design permits.

Alignment with feasibility and governance

A methodologically aligned design must also be deliverable. Confirm access to the sampling frame, expected eligible numbers, instrument permissions, laboratory capacity, linkage identifiers, staff expertise and the time required for approvals. If an essential data source is unavailable, revise the objective rather than leaving the gap to be solved during analysis.

Governance should follow the data pathway. The protocol must identify who collects, links, derives, accesses and retains each dataset. The planned analysis population should be achievable under the consent and data-access arrangements.

Alignment with sample-size planning

Sample size should be based on the main estimand or information need. Common mismatches include:

• calculating for prevalence but declaring a multivariable prediction model as primary;
• calculating for a mean difference while analysing a binary outcome;
• ignoring clustering in a cluster design;
• ignoring attrition or missing records;
• selecting an effect size unrelated to a clinically meaningful difference; and
• powering a pilot study for definitive effectiveness.

The Research Methodology section defines the design inputs; the Biostatistics section performs the calculation.

Alignment with reporting

The Methods section should allow readers to reconstruct the alignment chain. Reporting guidelines can identify expected items but do not substitute for sound design. A manuscript should not introduce new primary objectives or causal language after results are known.

Ethical and governance considerations

Misalignment can be an ethical problem. Participants may provide time, biological samples or sensitive information that cannot answer the stated question. Collecting unnecessary variables increases privacy risk. A study advertised as testing effectiveness should not expose participants to intervention burden if the design can assess only feasibility.

Interface with statistical analysis

A statistical analysis plan should be developed from the objective matrix, not from the software menu. It should define:

• analysis population and denominator;
• estimand for each main objective;
• descriptive summaries;
• effect measures and confidence intervals;
• model appropriate to outcome and design;
• clustering, repeated measurements and sampling weights;
• missing-data assumptions;
• multiplicity and subgroup strategy;
• diagnostics and sensitivity analyses; and
• planned tables and figures.

The related Biostatistics lecture develops these elements in statistical detail.

Target-trial thinking for observational causal questionsMMed extension

When randomisation is unavailable, observational causal research can be clarified by specifying the trial that would answer the question: eligibility, treatment strategies, assignment, time zero, follow-up, outcomes, causal contrast and analysis [2]. Aligning eligibility, treatment assignment and start of follow-up helps prevent immortal-time and related biases [2,3]. Target-trial emulation does not solve unmeasured confounding or poor measurement, but it makes design choices more explicit.

Worked example

WORKED EXAMPLE

Replacing an overloaded protocol with an aligned programme

Original plan: Use 200 retrospective records to identify causes, evaluate a booking intervention, explore experiences and develop a prediction model.

Alignment review: Retrospective records can estimate documented completion and pathway stages but cannot capture unrecorded experiences reliably. Without assigned or quasi-experimental booking, causal effectiveness is weak. Two hundred records may be inadequate for prediction-model development and validation.

Revised phase 1: Linked-record cohort to estimate completion and pathway loss using clear operational definitions.

Revised phase 2: Interviews with women and staff to explain barriers, referral processes and intervention acceptability.

Revised phase 3: Feasibility study of booking and navigation, measuring delivery, recruitment, fidelity and outcome ascertainment.

Future phase: Adequately designed comparative evaluation of effectiveness, followed by prediction research only if a clear decision use and sufficient sample are available.

Benefit: Each objective now has an appropriate population, design, measurement and analysis, while the phases form a coherent programme.

Common mistakes

Knowledge check

A cross-sectional survey aims to determine whether low health literacy causes non-attendance. What is the alignment problem?

Answer and reasoning

Exposure and outcome are measured at the same time, so temporality and reverse causation are unresolved. The design can estimate an association but cannot by itself establish the claimed causal effect.

Why create an objective-to-analysis matrix before data collection?

Answer and reasoning

It confirms that every objective has a corresponding population, variable, source and output. It exposes missing measurements, incompatible designs and unnecessary data fields before they become irreversible.

What is the difference between an estimand and an estimator?

Answer and reasoning

The estimand is the precise target quantity implied by the research question. The estimator is the statistical procedure used to estimate it. A model can be technically correct yet estimate the wrong target if the estimand was not defined.

Can one project contain quantitative and qualitative objectives?

Answer and reasoning

Yes, when each strand has an appropriate design and they are integrated around a coherent mixed-methods question. Merely adding interviews to an unrelated quantitative study does not guarantee alignment or integration.

Key take-home messages

SUMMARY

Key take-home messages

• Coherent research has a traceable chain from problem to interpretation.
• Begin with the intended inference and target quantity, not the preferred software.
• Each objective needs a matching population, design, measurement and output.
• Timelines, objective matrices and backward checking reveal alignment failures early.
• A staged research programme is often stronger than one overloaded study.

References

1. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH E9(R1): addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. Geneva: ICH; 2019. https://database.ich.org/sites/default/files/E9-R1_Step4_Guideline_2019_1203.pdf
2. Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183(8):758–764. https://doi.org/10.1093/aje/kwv254
3. Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70–75. https://doi.org/10.1016/j.jclinepi.2016.04.014