Advanced Randomisation Inference: From the Assignment Law to Auditable Causal Decisions

The elementary randomisation test is easy to describe: hold the outcomes fixed under a sharp null, reshuffle treatment according to the experiment, recompute a statistic, and count how often a result at least as extreme appears. The advanced discipline begins when each phrase in that sentence becomes a technical object. What exactly is the probability law over assignments? Which estimand is the statistic trying to recover? Which assignments are impossible? What if treatment probabilities differ? What if effects differ across units? What if whole clusters are assigned, outcomes are missing, or one person’s treatment changes another person’s outcome?

The purpose of this section is not to make randomisation inference look complicated. It is to preserve the original simplicity when the design becomes complicated. The governing rule remains the same: the reference distribution must be generated by the probability mechanism that actually created treatment assignment, and the statistic must still answer the scientific question that matters.

The assignment law in notation

Let Z denote the full treatment-assignment vector and let Ω be the set of assignments that the design could produce. An assignment mechanism can be written abstractly as π(z) = Pr(Z = z), for z in Ω. Complete randomisation is the special case where all assignments with the required treated count have equal probability. Other designs can assign unequal probabilities or exclude large parts of the mathematical treatment space.

This notation clarifies a crucial point. Randomisation inference is not fundamentally “take the observed labels and permute them”. It is integrate or sum over the assignment distribution π(z). Equal-weight label permutations are correct only when the design gives equal probability to those treatment vectors. When π(z) varies, the randomisation distribution must carry those unequal weights.

For a test statistic T(z,Y), a one-sided randomisation p-value under a sharp null can be expressed conceptually as the total assignment probability of all legal z whose statistic is at least as extreme as the observed statistic. In an equal-probability complete randomisation this reduces to a count divided by the number of assignments. In an unequal-probability design it is a probability-weighted sum, not a simple fraction of unique permutations.

The support of the design matters before any calculation

The set Ω is the support of the assignment mechanism. In a matched-pair trial, Ω contains only assignments with one treated unit per pair. In a blocked design, Ω preserves the block-specific treatment counts. In a cluster trial, Ω is defined over cluster labels. Under rerandomisation, Ω can contain only allocations that pass the pre-specified balance criterion.

A permutation program that samples outside Ω is answering a different experiment. This error can be subtle because the code still runs, produces a smooth histogram and returns a number called a p-value. The problem is conceptual, not computational: the simulated worlds include assignments the study design prohibited.

A reliable implementation should therefore begin with a function that generates legal assignments, not with a generic shuffle routine. The analysis becomes easier to audit when the design generator is a first-class part of the code.

Weighted randomisation p-values under unequal assignment probabilities

Suppose assignment A had probability 0.20, B had probability 0.10 and other legal assignments had their own probabilities. If A and B are the only assignments at least as extreme as the observed result, the tail probability is not automatically 2 divided by the number of legal assignments. It is the sum of the design probabilities attached to the qualifying assignments, subject to the exact test definition.

This becomes important under Bernoulli designs with unit-specific probabilities, adaptive designs, covariate-adaptive randomisation or administrative lotteries where participants have unequal selection chances. Equal-weight permutation can be badly calibrated even when every treatment vector in the sample looks superficially comparable.

The assignment file should therefore preserve more than final treatment labels. Where probabilities differ, preserve enough information to reconstruct those probabilities. A final CSV containing only “treated” and “control” can destroy part of the inferential design.

Horvitz–Thompson reasoning: inverse probability as design information

When treatment probabilities differ across units, design-based estimation can use inverse-probability logic. A Horvitz–Thompson-type estimator weights observed treated outcomes by the inverse of their treatment probabilities and observed control outcomes by the inverse of their control probabilities. Units that were unlikely to receive the assignment they actually received carry more design weight.

The principle is familiar from unequal-probability sampling: an observation representing a rare design event can stand for more of the assignment space. The estimator can remain unbiased under the assignment mechanism even when treatment probabilities differ, provided the probabilities are correctly specified and positive.

Large inverse weights can create high variance. This is not merely a numerical inconvenience. It signals weak design overlap: some units had very little chance of one treatment condition. Design should therefore be judged not only by unbiasedness but by whether the treatment probabilities support stable comparison.

Hájek-style normalisation trades exact unbiasedness for stability

Ratio or Hájek-style estimators normalise inverse-probability weights within treatment arms rather than using raw Horvitz–Thompson totals. They can reduce variance and produce more interpretable weighted means, especially when treatment probabilities vary materially.

The cost is that exact finite-sample unbiasedness can be lost. In many applications the bias is small relative to the variance improvement, but that is a statistical trade-off rather than a free upgrade.

Randomisation inference can still calibrate a Hájek-type statistic by recomputing it across the assignment mechanism. Again, validity and estimand are separate questions: the design supplies the randomisation law, while the estimator defines what feature of the outcomes is being summarised.

The Neyman variance formula reveals treatment-effect heterogeneity

Under complete randomisation with n_T treated and n_C control units, the randomisation variance of the difference-in-means estimator contains three conceptual pieces: finite-population variance of treated potential outcomes divided by n_T, finite-population variance of control potential outcomes divided by n_C, and a subtraction involving the finite-population variance of individual treatment effects divided by N.

The final term is important because heterogeneous treatment effects can reduce the randomisation variance of the average-effect estimator relative to a formula that ignores that covariance structure. Yet S²_τ is generally not identifiable from observed outcomes: each unit reveals only one potential outcome.

The familiar Neyman conservative variance estimator drops the negative S²_τ/N term and replaces the potential-outcome variances with observed arm variances. “Conservative” here has a precise design-based meaning under the framework. It does not imply every finite-sample confidence interval is automatically perfect under every design or outcome distribution.

Effect heterogeneity is not a nuisance term to be wished away

If treatment effects differ across units, the average treatment effect can remain a useful estimand while hiding important structure. An intervention might help low-baseline units and harm high-baseline units, or benefit one site while doing little elsewhere.

Fisher’s sharp null is especially sensitive to any individual-level departure from zero. A weak null about the average can hold even while treatment-effect heterogeneity is large. This explains why different design-based tests can answer different causal questions without contradiction.

Subgroup and heterogeneity analyses should be planned carefully. Searching across many subgroups after seeing outcomes can manufacture apparently dramatic interactions. Randomisation gives a probability foundation; multiplicity and model-selection problems still remain.

Why naive permutation can fail under a weak null

Under the Fisher sharp null, every missing potential outcome is known, so treatment can be reassigned exactly. Under a weak null such as zero average effect, individual effects may differ and the missing potential outcomes are not known. Simply applying the sharp-null permutation distribution to a raw mean difference can therefore have the wrong sampling behaviour when variances or treatment effects differ.

This is one reason advanced randomisation inference uses studentised statistics and asymptotic randomisation theory. A statistic scaled by an appropriate estimated standard error can have a permutation distribution that approximates the correct weak-null distribution even when the raw outcomes are not exchangeable in the naive sense.

The distinction protects a common boundary: exact Fisher test for a sharp null ≠ automatically exact test for every average-effect null. The computation may look almost identical while the theorem supporting it is different.

Studentisation is a calibration device

Suppose treated outcomes are much more variable than control outcomes. Permuting a raw mean difference treats the observed outcomes as exchangeable under label switching, an assumption that can fail under a weak null. A Welch-type studentised statistic incorporates arm-specific variance and can make the randomisation distribution better match the target null distribution.

Studentisation does not fix a wrong randomisation unit, unmeasured interference or post-treatment conditioning. It calibrates a statistic; it does not repair a broken design.

A useful implementation audit therefore separates two questions: “Did we generate the right assignments?” and “Did we recompute a statistic whose null behaviour supports the claim?” Both must be answered correctly.

Conditional randomisation and the logic of relevant comparisons

Conditional randomisation inference compares the observed assignment with a subset of legal assignments sharing a chosen property. For example, under Bernoulli assignment an analyst might condition on the realised number treated. In a rerandomised experiment the conditioning set may be the allocations satisfying the same covariate-balance rule.

The intuition is similar to conditioning on an ancillary feature: once a design feature is observed, compare the experiment with alternative assignments that preserve that feature rather than mixing in assignments that are less relevant to the realised design state.

Conditioning can improve power and interpretability, but it must not become outcome-driven cherry-picking. The conditioning rule should arise from the design or be pre-specified from covariates and assignment information that existed before outcome inspection.

Rerandomisation and Mahalanobis balance

A common rerandomisation idea measures covariate imbalance using a Mahalanobis distance between treatment and control baseline means and accepts only allocations below a threshold. The design intentionally excludes highly imbalanced assignments before treatment is delivered.

That improves covariate balance by construction. It also means the randomisation distribution is not the ordinary complete-randomisation distribution. A valid design-based test should sample from assignments that would have passed the same acceptance rule.

The acceptance threshold belongs to the protocol, not the post-hoc analysis. Choosing it after looking at outcomes converts design improvement into outcome-informed selection.

Pair matching changes both power and the legal universe

Matched pairs can dramatically reduce noise when pair members have similar potential outcomes. The inferential benefit comes from comparing treatment assignments within the matched structure rather than across arbitrary units.

For K independent pairs with one treated unit per pair, the assignment space has 2^K allocations. With ten pairs, that is 1,024 legal treatment patterns. The minimum attainable two-sided exact p-value can therefore still be coarse relative to very large individually randomised experiments.

Matching is not automatically beneficial if pairs are poorly constructed. A weak match can add operational complexity without materially reducing outcome variation. Design quality should be judged by how the matching variables relate to the outcome and implementation.

Block weights define which average the analysis estimates

In blocked experiments, block-specific effects can be aggregated using weights proportional to block size, equal block weights or other design-based choices. These weighting schemes can estimate different causal averages when blocks differ in size.

A large school and a small school should not receive equal weight if the estimand is the average student effect unless the design or target population says otherwise. Equal weighting may instead correspond more closely to an average school effect.

The randomisation distribution can be correct under either statistic. Correct design calibration does not answer the separate question of which population average the statistic represents.

Cluster size can be part of the treatment effect problem

Cluster size is sometimes merely structural and sometimes informative. Large clinics can differ systematically from small clinics. Large classes can have different baseline outcomes or treatment responsiveness. When cluster size is related to potential outcomes, weighting choices become causal rather than cosmetic.

Recent 2026 cluster-trial work continues to emphasise that heterogeneity in cluster sizes and structures can materially distort conventional inference and that design-based benchmarking is valuable. That does not make one cluster estimator universally correct; it makes the estimand and weighting rule impossible to ignore.

A strong report states whether it targets the average individual, average cluster, cluster-period effect or another quantity, then chooses statistics and weights accordingly.

Small cluster counts create a granularity ceiling

Ten clusters with five treated generate only 252 assignments under simple complete cluster randomisation. If a two-sided statistic is perfectly symmetric, the smallest attainable exact two-sided p-value can be on the order of two assignments out of 252, about 0.00794, before any additional constraints or ties.

With fewer clusters the granularity becomes much coarser. This places an information ceiling on hypothesis testing. Thousands of individual observations inside clusters cannot manufacture new assignment patterns.

This is why cluster-trial planning should focus on number of clusters, allocation structure and cluster-size balance, not only total participant count. The participant count determines outcome information; the cluster count determines where treatment randomisation entered.

Cluster-level versus individual-level statistics

A simple cluster-level analysis computes one outcome summary per cluster and applies randomisation inference to those cluster summaries. This makes the assignment structure transparent and can work well with few clusters. It may sacrifice information when individual-level covariates are highly predictive or cluster sizes vary substantially.

Individual-level model-assisted statistics can preserve more information while the reassignment still occurs at cluster level. The model can adjust baseline variables, period effects or precision weights; the randomisation engine still moves whole clusters according to the design.

The distinction echoes the entire article: analysis can be sophisticated without forgetting where randomisation happened.

Stepped-wedge designs require period-aware statistics

In a stepped-wedge trial, more clusters receive treatment as time progresses. A raw treated-versus-control difference can therefore confound intervention status with calendar period because late periods contain more treated observations than early periods.

Randomisation inference solves the assignment part by reassigning clusters to permissible sequences. The statistic still needs to account for period structure so it does not attribute secular time trends to treatment.

Current randomisation-based stepped-wedge methods exploit exactly this division of labour: the design defines the reference distribution; a period-aware statistic targets the marginal treatment effect.

Confidence interval inversion is a search over causal worlds

To invert a randomisation test for a constant additive effect τ, choose a candidate τ, impute the missing potential outcomes implied by that sharp hypothesis, run the randomisation test, and record whether τ is rejected. Repeat across a grid or use an efficient search algorithm.

The accepted τ values form a confidence set at the corresponding level. This is conceptually richer than attaching a normal-theory standard error to a point estimate because each candidate effect is evaluated through the experiment’s actual assignment mechanism.

The p-value function need not always behave as a perfectly smooth monotone curve in finite discrete designs. Confidence sets can inherit discreteness, asymmetry or even unusual shapes under complex statistics. Software should not assume the inversion problem is always solved by one symmetric plus-or-minus interval.

Discrete p-values and the temptation of mid-p

Exact randomisation p-values can be conservative because the finite assignment space may not permit a rejection probability exactly equal to α. Some statistical traditions use mid-p modifications that subtract part of the probability mass tied at the observed statistic.

Mid-p can improve average calibration or power in some discrete problems, but it gives up the same unconditional exactness guarantee as the ordinary tail probability. It should therefore be labelled as a modified procedure rather than described simply as “the exact randomisation p-value”.

For an educational library the important principle is transparency: discreteness is evidence about limited assignment information, not a defect that must always be smoothed away.

Monte Carlo precision can be planned before the analysis

When B independent assignments are sampled from the reference distribution, the Monte Carlo count of extreme assignments behaves approximately like a binomial count conditional on the observed data and test setup. If the true tail probability is p, the Monte Carlo standard error of the naive proportion is approximately sqrt[p(1−p)/B].

At p = 0.05 and B = 10,000, that standard error is about 0.00218. At B = 100,000 it falls to about 0.000689. These are simulation uncertainties, not sampling uncertainties from the experiment.

This allows computational planning. If the scientific interpretation would change between p = 0.049 and p = 0.052, a few hundred randomisations are inadequate. Increase B or use a sequential Monte Carlo procedure until the computational uncertainty is small relative to the decision.

Sequential Monte Carlo can stop when the decision is stable

A sequential Monte Carlo randomisation test samples assignments in batches and monitors evidence about the tail probability. If it becomes overwhelmingly clear that p is far above or below the decision threshold, computation can stop without spending resources on millions of additional assignments.

The stopping rule must be designed to preserve the intended error properties. Informally stopping because “the p-value looks small enough now” is another form of optional stopping.

This is a useful general lesson: computational efficiency is itself a statistical design problem when the computation is stochastic.

Randomisation-based maxT procedures preserve dependence across outcomes

Suppose an experiment measures twenty related outcomes. Testing each at 0.05 inflates the chance of at least one false positive. A maxT randomisation procedure computes all twenty test statistics under each reassignment, records the largest suitably standardised statistic, and compares each observed statistic with that joint null distribution.

Because the outcomes are permuted together under the same assignment, their dependence is preserved. This can be more powerful than corrections that ignore the correlation structure while still controlling a familywise error criterion under the procedure’s assumptions.

The hypothesis family must still be defined before inspecting results. Randomisation cannot rescue a family that expands every time an interesting outcome appears.

Model-assisted randomisation inference

A flexible prediction model can estimate outcomes from baseline covariates, and a randomisation statistic can then compare treatment groups using residuals or adjusted outcomes. If the model captures prognostic variation, the resulting statistic can be more powerful than the raw mean difference.

The design-based perspective gives the model a limited job. It helps compress baseline information and improve precision; the treatment randomisation remains the inferential anchor. This can reduce dependence on the literal truth of the outcome model.

Machine-learning adjustment raises additional issues: overfitting, data reuse, hyperparameter choice and computational cost across thousands of assignments. Cross-fitting or pre-specified training procedures can help, but every randomised replicate must reproduce the intended analysis rule consistently.

Regression adjustment should honour treatment interactions

In randomised experiments, regression adjustment can improve precision, but rigid common-slope models can behave poorly when covariate relationships differ by treatment arm. Including treatment-by-covariate interactions or using arm-specific adjustment can provide more robust design-based performance.

The point is not that one regression specification is universally mandatory. It is that “randomised” does not make every regression adjustment harmless. The adjustment procedure should be part of the pre-specified estimator and examined under randomisation.

A useful check is simulation under the planned design: generate plausible potential outcomes with treatment-effect heterogeneity, run the complete adjustment procedure, and examine bias, variance and confidence-interval coverage before the real outcome data are analysed.

Subgroup effects require interaction tests, not parallel significance stories

A treatment can be statistically significant in one subgroup and not significant in another even when the subgroup effects do not differ significantly. Comparing two p-values is not a test of interaction.

Randomisation inference can directly test an interaction statistic by reassigning treatment according to design and recomputing the difference in treatment effects across pre-specified subgroups.

Subgroup labels must be pre-treatment for ordinary causal interpretation. If subgroups are defined by post-treatment response, compliance or survival, the estimand changes and additional assumptions become necessary.

Non-inferiority and equivalence need more than failure to reject zero

Randomisation inference is often introduced through tests of no effect. Non-inferiority and equivalence ask different questions: whether an effect lies above a lower margin, below an upper margin, or inside a pre-specified equivalence region.

Failing to reject the sharp null of zero effect does not establish equivalence. A wide confidence interval can contain both meaningful benefit and meaningful harm.

Test inversion provides a natural route: construct a design-based confidence set for the relevant effect parameter and compare it with the pre-specified non-inferiority or equivalence margins. The margin itself is a substantive decision, not a number the randomisation distribution can choose.

Treatment misclassification is different from noncompliance

Noncompliance means assigned and received treatments differ. Treatment misclassification means the recorded assignment or received-treatment variable is wrong. The first is a behavioural or implementation phenomenon; the second is a measurement failure.

Randomisation inference relies on knowing the realised assignment. If the assignment record itself is corrupted, the reference distribution can be anchored to the wrong treatment vector.

Preserve randomisation logs, allocation-system records and audit trails independently of downstream data entry. The assignment mechanism is evidence and should be protected like primary measurement data.

Broken allocation concealment can compromise the design before treatment begins

A sequence can be generated randomly and still be compromised if investigators can predict the next assignment and selectively enrol participants. Allocation concealment protects the randomisation process from manipulation before assignment.

If enrolment depends on knowledge of upcoming assignments, the final treatment groups may not have arisen from the advertised mechanism. A later permutation test using the planned design cannot restore an assignment process that was not actually followed.

Design-based inference therefore depends on design integrity, not merely design documentation. The randomisation log should be consistent with the operational history of enrolment and assignment.

Interference requires an exposure mapping

When outcomes depend on other units’ assignments, the full vector Z can be too large to interpret directly. An exposure mapping compresses the assignment vector into a scientifically meaningful state for each unit: untreated with no treated neighbours, untreated with some treated neighbours, treated in a low-saturation cluster, treated in a high-saturation cluster, and so on.

The mapping is a causal model. If it omits the feature through which spillovers operate, the estimated “indirect effect” may combine different exposure states. Network distance, classroom membership, household contact and geographic proximity can imply different mappings.

Randomisation inference can compare assignments that change these exposure states while respecting the original treatment design. The design provides variation; the exposure mapping defines which causal contrasts are meaningful.

Randomised saturation designs identify direct and spillover effects

A saturation design can randomise the fraction of units treated within groups and then randomise individuals inside groups. This creates variation in both own treatment and peer treatment intensity.

Such a design can support direct effects, indirect spillover effects and total effects under appropriate exposure assumptions. A conventional two-arm individual-level test would discard much of the deliberate design information.

This illustrates the recurring principle at a larger scale: when the experiment randomises a structured treatment mechanism, the analysis should preserve that structure rather than flatten it into one binary label.

Attrition sensitivity can be embedded inside the design analysis

When outcomes are missing, analysts can supplement the primary randomised comparison with sensitivity analyses that vary assumptions about the unseen outcomes. For binary outcomes this can include best-case and worst-case bounds. For continuous outcomes it can use bounded shifts, pattern-mixture assumptions or inverse-probability weighting.

The assignment mechanism remains known, but the complete outcome schedule is not. Sensitivity analysis therefore has two layers: randomisation uncertainty from assignment and missing-data uncertainty from unobserved outcomes.

These uncertainties should not be merged into one unexplained standard error. Report which part is design-based and which part depends on unverifiable missingness assumptions.

Noncompliance and randomisation-based instrumental variables

When assignment affects treatment received but compliance is imperfect, assignment can serve as an instrument. Under additional assumptions, the ratio of the intention-to-treat effect on outcome to the intention-to-treat effect on treatment received can estimate a complier average causal effect.

Randomisation supplies the exogenous instrument. It does not supply the exclusion restriction, monotonicity or stable treatment-version assumptions needed for the complier interpretation.

Randomisation tests can target sharp instrumental-variable hypotheses or invert tests for effect-ratio parameters, but the causal interpretation remains conditional on those substantive assumptions. An exact p-value cannot make an exclusion restriction true.

Natural experiments need an assignment-mechanism audit before any permutation

A lottery is the cleanest natural experiment because the assignment probabilities can sometimes be reconstructed directly. Other institutional mechanisms are murkier: threshold rules with discretionary exceptions, rotating queues, oversubscribed programmes, weather shocks or administrative timing.

Before applying randomisation inference, reconstruct who was eligible, which assignments were possible, whether actors could manipulate timing or eligibility, whether probabilities varied, and whether treatment was assigned at individual, household, school or geographic level.

If the mechanism cannot be defended, describe the study as observational and use the appropriate causal tools. Calling an uncertain mechanism “as-if random” does not make it so.

Permutation diagnostics can test the randomisation implementation

Before analysing outcomes, the same assignment generator can be used to study baseline balance statistics, treatment counts, cluster-size distributions and other design properties. The realised assignment should look possible under the documented mechanism.

An extremely unusual baseline balance statistic is not proof of manipulation; rare assignments do occur. But several impossible or near-impossible design features can reveal coding errors, implementation deviations or an incorrectly reconstructed assignment mechanism.

This creates a useful audit sequence: verify the design generator before asking it to calibrate an outcome test.

Simulation can validate the randomisation code itself

A randomisation program should be tested under synthetic sharp-null data where the correct type-I error behaviour is known in principle. Repeatedly generate assignments from the design, apply the full test, and check whether rejection rates are consistent with the nominal level up to simulation error.

Then stress the code: unequal block sizes, repeated outcome values, missing clusters, extreme weights, zero variance within an arm, odd numbers of units, very small assignment spaces and restricted designs.

Software verification cannot prove the scientific assumptions, but it can prevent an implementation bug from masquerading as a methodological result.

A reproducible randomisation test has three seeds, not always one

In a modern computational experiment, several random-number streams can appear: the original treatment-assignment seed, a simulation seed used for Monte Carlo randomisation inference, and possibly a machine-learning or imputation seed used inside each analysis replicate.

These should be distinguished. Reusing one seed casually across nested random processes can create accidental dependence or make the audit trail confusing.

Reproducibility means another analyst can regenerate the same assignment space and analysis result from the documented protocol. It does not mean the original treatment assignment should be rerun; that historical assignment is fixed evidence once the experiment begins.

The assignment mechanism should be versioned like source code

Large trials sometimes change randomisation procedures during recruitment: new strata are added, sites open, allocation ratios change, blocks are exhausted or emergency rules are invoked. These changes can alter π(z).

A final analysis that assumes one static mechanism can then misrepresent the experiment. Preserve dated protocol versions, site-specific rules and the mapping from each recruited unit to the mechanism operating at its assignment time.

Randomisation inference is unusually sensitive to provenance because the design is the probability model. If the design history is lost, part of the inferential model is lost with it.

Protocol deviations should be classified before they are analysed

Not every deviation has the same consequence. A treatment delivered late differs from a treatment assignment generated outside the randomisation system. An outcome measured one day late differs from a participant enrolled after investigators knew the assignment.

Classify deviations by whether they affect assignment integrity, treatment delivery, outcome measurement, missingness or analysis. Intention-to-treat inference can often tolerate imperfect delivery while preserving the random assignment comparison. It cannot automatically tolerate corrupted assignment itself.

This classification helps determine which sensitivity analysis is relevant and prevents every protocol deviation from being treated as the same generic “violation”.

Randomisation inference can be a benchmark for model-rich analyses

Modern experiments often use generalized linear models, mixed models, generalized estimating equations, targeted estimators or machine learning. These methods can improve efficiency and accommodate complex outcomes, but they introduce modelling assumptions.

A design-based analysis can serve as a benchmark. If a sophisticated method and a transparent randomisation-based analysis agree materially, confidence in the substantive conclusion can increase. If they disagree, the disagreement becomes a diagnostic question rather than a reason to select the preferred answer.

The 2026 CARE framework for cluster-randomised trials explicitly advances this kind of design-based benchmarking logic in a modern trial-analysis context. The broader principle is durable: richer models should earn their added assumptions by changing precision or interpretation in a defensible way.

Design-based inference does not make modelling optional for generalisation

The experimental assignment can identify a treatment contrast for the randomised units. Generalising that contrast to another population often requires modelling effect modifiers, reweighting populations or combining experimental data with external information.

This creates a healthy division of labour. Use randomisation to protect the internal causal comparison. Use transport models, sampling weights and substantive knowledge to ask where that causal result should travel.

Do not weaken the internal design argument by asking it to prove external validity it was never built to provide.

When the randomisation test is less useful than the estimate

A study can have an enormous sample in which the sharp null is rejected decisively for a scientifically tiny effect. Another can have a small assignment space where p-values are coarse but the estimated effect and uncertainty are still decision-relevant.

Randomisation tests should therefore not become another p-value ritual. Report effect estimates, design-based uncertainty, clinically or operationally meaningful thresholds, and the randomisation p-value where it answers a useful hypothesis.

The test is strongest when it contributes to a decision: evidence of any effect, compatibility with a target effect, non-inferiority, robustness of a model-based result, or detection of assignment-sensitive conclusions.

A high-quality randomisation-inference report should be reconstructible

1. Design: name the randomisation unit, allocation ratio, blocks, strata, pairs, clusters and restrictions.
2. Assignment probabilities: state whether legal assignments were equally likely.
3. Estimand: state the causal quantity of interest.
4. Null: distinguish sharp from weak null hypotheses.
5. Statistic: give the exact formula or analysis rule.
6. Reassignment algorithm: explain how legal assignments were generated.
7. Enumeration: report total legal assignments when exhaustive.
8. Monte Carlo: report B, seed and tail-count convention when sampled.
9. Tails: define one-sided or two-sided extremeness.
10. Adjustment: state baseline covariates and model-assisted procedures.
11. Clustering: show how the randomisation unit was preserved.
12. Missingness: report attrition and assumptions used to address it.
13. Compliance: separate assignment effects from treatment-received effects.
14. Interference: state the no-interference assumption or exposure mapping.
15. Multiplicity: identify hypothesis families and adjustments.
16. Effect estimate: report magnitude, not only the p-value.
17. Uncertainty: provide a design-based interval or confidence set when appropriate.
18. External validity: state the population to which the conclusion is intended to apply.
19. Code: preserve a reproducible implementation.
20. Deviations: document departures from the assignment protocol.

Advanced worked example: unequal probabilities

Imagine four units with treatment probabilities 0.8, 0.6, 0.4 and 0.2 under independent Bernoulli assignment. A realised assignment of treated, treated, control, control does not have the same probability as every other two-treated allocation. Its assignment probability is 0.8 × 0.6 × 0.6 × 0.8 = 0.2304 because units 3 and 4 must be untreated.

An allocation treated, control, treated, control has probability 0.8 × 0.4 × 0.4 × 0.8 = 0.1024. Counting those allocations equally in an unconditional Bernoulli randomisation test would discard the design probabilities.

If an analyst conditions on exactly two treated units, the conditional probabilities among two-treated assignments must be renormalised from their original Bernoulli probabilities. Conditioning changes the reference distribution; it does not magically make all six two-treated assignments equally likely.

Advanced worked example: p-value granularity in a cluster trial

Ten clusters are completely randomised, five to treatment and five to control. There are 252 equally likely allocations. Suppose the observed statistic is so extreme that only the observed assignment and its complete treatment reversal are at least as extreme in absolute value.

The exact two-sided p-value is 2/252 ≈ 0.00794. Adding ten thousand more participants inside those same ten clusters does not create more assignment vectors. It can make the cluster summaries more precise, but the treatment-assignment granularity remains governed by ten clusters.

This example makes clear why “sample size” has multiple meanings in cluster experiments. Participant count, cluster count and assignment-space size answer different information questions.

Advanced worked example: Monte Carlo precision

Suppose the randomisation p-value is near 0.05. With B = 10,000 Monte Carlo assignments, the approximate simulation standard error at p = 0.05 is sqrt(0.05 × 0.95 / 10,000) ≈ 0.00218. A 95 per cent Monte Carlo interval based on a simple normal approximation would span roughly plus or minus 0.0043.

If that computational uncertainty is too large for the decision, increase the number of assignments. With B = 100,000, the standard error falls by sqrt(10), to about 0.000689.

The lesson is not to worship a threshold. It is to avoid presenting simulation noise as experimental evidence.

Advanced practice: diagnose the design before the statistic

Problem A. Twelve clinics are paired into six matched pairs and one clinic per pair is randomised to treatment. An analyst shuffles all twelve clinic labels while preserving six treated clinics. What is wrong?

Answer. The legal design has 2⁶ = 64 pairwise assignments, not C(12,6) = 924 unrestricted assignments. The analyst invented treatment allocations that break the matching protocol.

Problem B. Treatment probabilities vary by unit, but the analyst draws every treatment vector with equal probability. What assumption has been changed?

Answer. The assignment mechanism. The reference distribution no longer follows the design probabilities, so the resulting test is calibrated to a different experiment.

Problem C. A trial tests the weak null of zero average effect using an unstudentised sharp-null permutation distribution when treatment effects and variances are heterogeneous. What should be checked?

Answer. Whether the chosen permutation procedure is valid for the weak null. A studentised randomisation statistic or design-based asymptotic method may be needed; exact Fisher logic does not automatically transfer.

Problem D. A cluster trial has fifteen thousand participants but only eight randomised hospitals. Why might a conventional individual-level p-value look more certain than the design warrants?

Answer. Treatment was assigned eight times, not fifteen thousand times. Ignoring cluster assignment treats correlated outcomes as independent randomisation information and can dramatically understate uncertainty.

Problem E. A study rerandomised until baseline Mahalanobis distance was below a threshold but the analysis uses unrestricted complete randomisation. What information is discarded?

Answer. The acceptance rule that defined the actual assignment support. The randomisation distribution should usually preserve that restriction.

Problem F. A model-assisted statistic is fit once using the observed assignment and then treatment labels are shuffled without refitting the model. Is that always valid?

Answer. Not necessarily. The randomisation replicates must reproduce the analysis rule the method requires. Some procedures can freeze nuisance quantities by construction; others require refitting. The algorithm, not convenience, decides.

Problem G. The primary randomisation test gives p = 0.40 with a very wide effect interval. Can the study claim equivalence?

Answer. No. Failure to reject no effect does not establish a sufficiently narrow effect range. Equivalence or non-inferiority requires pre-specified margins and interval/test procedures targeted to those margins.

Problem H. A treatment affects peer behaviour, but the analysis treats each student’s outcome as depending only on their own assignment. What has been misspecified?

Answer. The potential-outcome exposure structure. An interference model or exposure mapping is needed to represent peer treatment.

The advanced release test

1. Can the assignment mechanism be regenerated exactly from protocol records?
2. Does the code generate only legal assignments?
3. Does it attach the correct probability to each assignment?
4. Does every replicate preserve pairs, blocks, strata, clusters and restrictions?
5. Is the causal null sharp or weak, and is the test valid for that null?
6. Does the statistic target the estimand rather than merely produce a convenient p-value?
7. Are weights consistent with the intended individual- or cluster-level population?
8. Does covariate adjustment use pre-treatment information and a reproducible rule?
9. Are outcome missingness, noncompliance and interference treated as separate causal problems?
10. Are Monte Carlo error and discrete p-value resolution visible?
11. Can confidence intervals be traced to the same design logic as the test?
12. Are multiplicity and subgroup searches accounted for?
13. Has the implementation been validated under synthetic null cases?
14. Are seeds, code versions and protocol versions preserved?
15. Does the final claim stay inside the population and design that the evidence can support?

If those conditions are met, randomisation inference becomes more than an alternative way to calculate a p-value. It becomes an auditable chain from experimental design to causal claim.

The advanced compression

Randomisation inference is simplest when the experiment is simple, but its governing logic survives complexity. Represent the assignment law explicitly. Preserve its support and probabilities. Distinguish sharp from weak causal nulls. Use studentisation when the broader null requires it. Carry block, pair and cluster structure into every reassignment. Let weighting express the estimand. Treat Monte Carlo error as computational uncertainty. Model interference, missingness and noncompliance rather than pretending randomisation erased them.

The method’s deepest strength is accountability. The reference distribution is not a generic statistical fiction. It is a map of what the experiment was allowed to do. When the analysis remains faithful to that map, every p-value, confidence set and sensitivity check can be traced back to the mechanism through which chance entered the study.

That is the standard for world-class design-based evidence: not “we permuted the data many times”, but “we reconstructed the experiment’s legal counterfactual assignments, used a statistic matched to the causal estimand, tested the right null, and preserved enough provenance that another analyst can reproduce the entire inferential universe.”

Randomisation Inference Design Clinic: Worked Architectures, Estimands and Failure Checks

The fastest way to understand randomisation inference is to watch the same core logic move across different experiment architectures. In every case the workflow is unchanged: recover the assignment law, define the causal claim, choose a statistic, regenerate only legal assignments, and interpret the resulting distribution within the boundaries of the design. What changes is the geometry of the assignment space.

Clinic 1: a two-arm classroom experiment

Twenty students are randomly assigned, ten to a new vocabulary method and ten to the usual method. The primary outcome is a pre-specified test score two weeks later. Complete randomisation gives C(20,10) = 184,756 legal assignments. The primary statistic is the treated-minus-control mean score.

Under Fisher’s sharp null, every student would have the same score under either method, so the observed score vector can be reused under every legal assignment. Exhaustive enumeration is computationally easy at this scale. The result can be reported with the observed effect estimate, the exact two-sided randomisation p-value and, if desired, a confidence set obtained by inverting constant-effect tests.

If baseline vocabulary score is strongly predictive, an adjusted statistic can improve precision. But the labels are still reassigned according to the ten-of-twenty design. Regression adjustment changes the lens; it does not change the assignment universe.

Clinic 2: the same study with matched pairs

Now the twenty students are matched into ten pairs using baseline vocabulary score, and one student in each pair is randomly assigned to treatment. The number of legal assignments falls from 184,756 to 2¹⁰ = 1,024.

A pair-aware statistic can average the ten within-pair treatment–control differences. Under a sharp null, the exact randomisation distribution is generated by flipping treatment within each pair. An unrestricted ten-of-twenty permutation would erase the very structure introduced to improve precision.

The matched design has fewer assignment vectors but often less outcome noise. Assignment-space size and statistical power therefore do not move mechanically together. Good matching can increase power even while the exact p-value grid becomes coarser.

Clinic 3: four blocks with different treatment counts

Suppose a study uses four grade levels as blocks. Grade 1 assigns 3 of 6 students to treatment, Grade 2 assigns 4 of 8, Grade 3 assigns 5 of 10 and Grade 4 assigns 6 of 12. The legal assignment count is the product C(6,3) × C(8,4) × C(10,5) × C(12,6), because randomisation occurs independently within blocks under this simplified design.

The statistic can combine block-specific mean differences using weights proportional to block size when the target is the average individual effect across the study population. Equal block weights would answer a different question, closer to an average block effect.

The randomisation program should generate treatment labels separately within each grade. A single global shuffle that happens to preserve the total treated count is not the same experiment.

Clinic 4: twelve schools, six treated

Twelve schools are completely randomised with six assigned a literacy programme. Each school contains many pupils, but there are only twelve independent treatment assignments. The simple cluster assignment space contains C(12,6) = 924 legal allocations.

A transparent analysis could compute one adjusted outcome mean per school and use the difference in mean cluster outcomes as the statistic. A more efficient analysis could use pupil-level regression adjustment while treatment reassignment still occurs at school level.

The danger is pseudoreplication. A dataset with 6,000 pupil records can produce tiny individual-level standard errors if school dependence is ignored. Randomisation inference keeps the treatment information attached to the twelve schools where it entered.

Clinic 5: paired cluster randomisation

The twelve schools are instead matched into six pairs by size, baseline performance and neighbourhood context, then one school per pair is treated. The legal assignment space is 2⁶ = 64 allocations, not 924.

Only sixty-four assignments means the p-value grid is coarse. A two-sided exact test cannot manufacture arbitrary precision. Yet strong matching can substantially reduce between-school noise, so the design can still be efficient for estimating a treatment contrast.

Reporting should therefore emphasise effect estimates and uncertainty, not treat the inability to produce p = 0.0001 as a weakness. The assignment space describes the evidence the design actually created.

Clinic 6: unequal school sizes

Imagine the six treated schools include one with 1,800 pupils and another with 120. An unweighted mean of school-level outcomes treats those schools equally. A pupil-weighted statistic gives the large school much more influence.

If the target is the average effect on a randomly selected pupil from the enrolled population, pupil weighting is natural. If the target is the average effect on a randomly selected school, equal cluster weighting may be more natural. Neither choice is “more randomised”. They answer different estimands.

This is why design-based inference cannot be reduced to an algorithm. The assignment mechanism calibrates uncertainty; the scientific question chooses the estimand.

Clinic 7: a three-arm experiment

Thirty units are assigned equally to control, Treatment A and Treatment B. The primary hypothesis is that all three potential-outcome schedules are identical under the sharp null. An omnibus statistic such as between-arm dispersion can be calibrated across all legal three-arm assignments.

If the omnibus null is rejected, planned pairwise contrasts can compare A with control and B with control. Because more than one contrast is being reported, multiplicity should be handled according to the pre-specified family of claims.

A randomisation-based maxT procedure can use the joint distribution of the pairwise statistics across the same reassignments, preserving their dependence rather than adjusting each contrast as though it were unrelated to the others.

Clinic 8: a 2 × 2 factorial experiment

A factorial experiment randomises two binary factors, A and B, creating four treatment combinations. The design can estimate the main effect of A, the main effect of B and the A×B interaction.

A randomisation test for the full sharp null can reassign the four treatment combinations according to the factorial design. A test specifically targeting interaction should use a statistic representing the difference in A effects across levels of B.

The phrase “permute the treatment label” becomes ambiguous because treatment is now a vector of factor levels. The randomisation generator should reproduce the factorial assignment rule, not collapse the experiment into treated versus untreated.

Clinic 9: crossover treatment sequences

Participants receive A then B or B then A, with sequence randomised. The assignment unit is the participant’s treatment sequence. Outcomes occur within participant across periods.

A valid randomisation test swaps or reassigns sequences according to the protocol. Shuffling period-level observations across participants destroys both the within-person dependence and the assignment mechanism.

Carryover remains a scientific issue. If Treatment A changes the participant in a way that persists into the B period, random sequence assignment does not make the carryover disappear. Washout and period modelling remain part of the design argument.

Clinic 10: Bernoulli assignment with unequal probabilities

Suppose high-risk units are assigned treatment with probability 0.70 and low-risk units with probability 0.30. The total treatment count varies. A complete-randomisation test that fixes the observed treatment count and weights all allocations equally no longer reproduces the unconditional design.

One valid path is to sample treatment vectors from the original independent Bernoulli probabilities. Another is to perform a carefully defined conditional randomisation test given the realised count or another design feature. Conditional probabilities must be derived from the Bernoulli law rather than replaced with equal weights by habit.

Inverse-probability statistics can account for unequal chances of treatment when the target estimand requires it. The analysis should state both the assignment probabilities and the weighting rule.

Clinic 11: rerandomisation for baseline balance

An engineering field trial repeatedly generates fifty–fifty allocations until a baseline Mahalanobis distance is below a fixed threshold. The accepted allocation is then used. This is not ordinary complete randomisation even though the final treated count is the same.

The analysis should simulate or enumerate only assignments that satisfy the same balance rule. Doing so recognises that extremely imbalanced allocations were never eligible to be observed.

Because rerandomisation improves baseline balance, an analysis that ignores the restriction can be conservative or inefficient. More importantly, it misstates the design that generated the evidence.

Clinic 12: noncompliance after perfect randomisation

A job-training programme is randomly offered, but only 70 per cent of those offered attend. Ten per cent of control participants obtain similar training elsewhere. The comparison by original offer remains randomised and estimates an intention-to-treat effect of assignment.

A comparison of attendees versus non-attendees is not randomised. Attendance reflects motivation, access and other characteristics. To estimate a treatment-received effect using assignment as an instrument, additional assumptions are needed.

The design-based lesson is not “ignore compliance”. It is “do not surrender the randomised comparison while analysing compliance”. Preserve assignment and received-treatment variables separately.

Clinic 13: differential attrition

A school intervention is randomised correctly, but outcome data are missing for 18 per cent of treated pupils and 4 per cent of controls. Complete-case treatment labels are still the original labels, but the complete-case sample is no longer guaranteed to be a randomised subset.

The primary report should show attrition by arm and reasons where available. Sensitivity analyses can examine weighting, imputation or plausible missing-outcome shifts. A complete-case randomisation p-value alone does not resolve whether missingness changed the causal comparison.

Randomisation protects how treatment was assigned. It does not force outcomes to remain observable.

Clinic 14: network spillovers

A social-learning intervention is individually randomised within classrooms, but treated students share materials with friends. An untreated student with five treated friends is not causally equivalent to an untreated student with none.

The analysis can define exposure categories based on own treatment and peer treatment. The randomisation mechanism then determines how often those exposure states arise across legal assignments.

The validity of the spillover analysis depends on the exposure mapping. If friendship links are missing or the true mechanism operates through study groups rather than declared friends, the mapping can misclassify causal exposure even though treatment itself was randomised.

Clinic 15: a stepped-wedge rollout

Eight clinics all eventually receive an intervention, but the order of crossover is randomised across four rollout periods. The treatment label is structurally entangled with time because every later period contains at least as much treated exposure as the period before it.

Randomisation inference reassigns clinics to legal crossover sequences rather than independently flipping clinic-period observations. The statistic should adjust or contrast within periods so secular time trends are not mistaken for treatment effects.

This design demonstrates why the assignment mechanism can involve an entire trajectory rather than one static label.

Clinic 16: a binary outcome with a rare event

A safety intervention is randomised across small industrial sites and the outcome is whether a rare incident occurs. A logistic mixed model may be unstable with very few events and clusters.

A randomisation test can use a simple cluster-level incident statistic under a sharp null, providing a transparent design-based benchmark. The resulting p-value may be coarse because the assignment space is small, but it does not depend on asymptotic normality of a rare-event regression coefficient.

Effect estimation remains important. A risk difference or incidence-rate difference with an appropriate interval can be more informative for decisions than the hypothesis test alone.

Clinic 17: survival outcomes with censoring

A treatment is randomised by clinic and the outcome is time to service failure. Some systems remain failure-free when follow-up ends. A log-rank-type or model-assisted survival statistic can be recalculated across legal clinic assignments.

The treatment assignment is randomised; censoring may not be. If censoring differs because treated systems are monitored more closely or are withdrawn for treatment-related reasons, the survival statistic can inherit censoring bias.

Design-based calibration and censoring assumptions therefore coexist. One cannot substitute for the other.

Clinic 18: repeated outcomes

Participants are randomised once and measured weekly for twelve weeks. The twelve observations per participant are not twelve treatment assignments. A valid statistic can summarise each person’s trajectory, use a repeated-measures model or estimate an area-under-the-curve contrast.

Across the randomisation distribution, the entire participant assignment moves as one unit while all of that participant’s repeated outcomes remain together. Permuting individual weekly rows would create treatment histories the study never generated.

The rule generalises: longitudinal measurement increases outcome information, not the number of independent randomisations.

Clinic 19: an A/B test with triggered exposure

A digital product randomly assigns users to interface A or B, but only users who visit a particular feature actually see the change. Analysing only triggered users can introduce post-randomisation selection if treatment affects whether users reach the feature.

The intention-to-treat comparison across all randomised users remains protected by assignment, though it may dilute the effect of actual exposure. Triggered analyses require careful design definitions—such as pre-treatment eligibility or principal-stratum reasoning—rather than simply dropping unexposed users.

Randomisation inference is widely applicable outside classical laboratories precisely because the key object is assignment, not the scientific domain.

Clinic 20: a natural lottery

A programme receives 5,000 eligible applications for 1,000 places and allocates places by a documented lottery. If every eligible applicant has equal chance and the lottery is faithfully implemented, the assignment mechanism can support randomisation inference even though the study was not designed by researchers.

The audit should verify eligibility rules, duplicate applications, priority categories, withdrawals and whether administrators overrode lottery results. If priority categories existed, the mechanism may be stratified or unequal-probability rather than one simple thousand-of-five-thousand draw.

The credibility of the natural experiment lives in the operational assignment record. A statistical shuffle performed years later is only as trustworthy as that reconstruction.

Clinic 21: why a tiny p-value can coexist with a tiny effect

In a very large individually randomised experiment, a mean difference of 0.2 points on a 100-point scale can produce an extremely small randomisation p-value if outcome variability is modest and the sample is huge.

The p-value says the sharp no-effect assignment pattern is inconsistent with the observed statistic. It does not say the 0.2-point effect is educationally, medically or operationally important.

Effect size and decision threshold belong beside the test. Statistical detectability and practical importance are different dimensions of evidence.

Clinic 22: why a large p-value can coexist with meaningful uncertainty

A small cluster trial estimates an effect large enough to matter, but the confidence interval is wide and the exact p-value is 0.18. The correct interpretation is not “there is no effect”. The data remain compatible with a range that may include meaningful benefit, little effect and possibly harm.

The study may simply be underpowered. Decision-makers should examine the interval, design resolution and prior evidence rather than turning non-rejection into proof of absence.

Randomisation inference makes uncertainty honest; it does not make sparse evidence decisive.

Clinic 23: testing code with a null experiment

Before analysing the real data, construct synthetic outcomes that are fixed and unrelated to assignment. Repeatedly generate treatment assignments from the planned design and run the entire randomisation-test pipeline.

Across many synthetic experiments, the fraction of p-values below 0.05 should be compatible with the procedure’s nominal type-I error, allowing for Monte Carlo variation and discreteness. A systematic excess can reveal an invalid assignment generator, incorrect tail counting, misuse of clustering or a bug in the statistic.

This is statistical unit testing: test the inferential machinery before trusting it on outcomes whose truth is unknown.

Clinic 24: test inversion as a full p-value curve

Instead of calculating only the p-value for τ = 0, evaluate a grid of constant additive effects: −5, −4.9, −4.8 and so on through plausible positive effects. Plot the randomisation p-value against τ.

The curve shows which hypothesised effects are most compatible with the assignment evidence and which are rejected. The confidence set is the region where the p-value remains above the chosen α level.

This makes randomisation inference visibly estimative rather than merely dichotomous. Zero becomes one candidate effect among many, not the centre of the statistical universe.

Clinic 25: a pre-specified robustness ladder

A strong experimental report can pre-specify several analyses with different assumption loads: unadjusted design-based effect estimate, covariate-adjusted design-based statistic, model-based regression estimate, and sensitivity analyses for missing outcomes or noncompliance.

The purpose is not to vote among analyses. It is to ask which assumptions materially change the conclusion. Agreement across methods strengthens robustness; disagreement identifies the assumption that owns the result.

This turns method comparison into scientific diagnosis rather than methodological competition.

A compact implementation algorithm

1. Load the immutable observed outcomes, baseline covariates and realised treatment assignment.
2. Load the exact design specification and any block, pair, cluster, sequence or balance restrictions.
3. Define the causal null and determine whether it is sharp.
4. Impute missing potential outcomes only when the null logically permits it.
5. Define the estimand and a statistic aligned with it.
6. Compute the observed statistic using the full analysis pipeline.
7. Generate one legal alternative assignment from the design law.
8. Recompute every assignment-dependent part of the statistic required by the method.
9. Record whether the replicate is at least as extreme as observed under the pre-specified tail rule.
10. Repeat exhaustively or by Monte Carlo.
11. Convert the tail count or probability mass into a p-value using the chosen exact or simulation convention.
12. For interval estimation, repeat the test over candidate effect values and invert.
13. Run implementation diagnostics and synthetic-null tests.
14. Store the design generator, analysis code, seed information and output together.

What should never be hidden in a randomisation-inference appendix

• The randomisation unit.
• The number of legal assignments or the way they were sampled.
• Whether assignment probabilities were equal.
• Whether blocks, pairs, clusters or rerandomisation restricted the design.
• The exact test statistic.
• The null hypothesis actually tested.
• The tail definition.
• The number of Monte Carlo draws.
• How missing outcomes were handled.
• Whether treatment assignment and treatment received differed.
• Whether interference was assumed absent.
• Which population the causal estimate describes.

These are not minor computational settings. They are the architecture of the inference. Readers should not need to reverse-engineer them from software syntax.

Final design-clinic compression

Across individuals, pairs, blocks, schools, stepped wedges, factorials, networks and natural lotteries, the rule remains stable: reconstruct what chance was actually allowed to change. Then make every simulated or enumerated world obey that rule.

The best randomisation analysis is not the one with the most permutations. It is the one whose possible assignments, causal null, estimand, statistic, computation and final claim all refer to the same experiment.