Activation Patching & Causal Tracing | Mechanistic Interpretability, Causal Mediation and Circuit Localisation

Activation patching is a mechanistic interpretability method that asks a causal question inside a neural network: if we replace one internal activation with the activation produced by a different run, how much does the model’s behaviour change? The modern vocabulary—activation patching, causal tracing, mechanistic interpretability, causal mediation, path patching, clean and corrupted runs, activation intervention, circuit localisation—describes a family of controlled internal interventions rather than one universal recipe.

In causal tracing, researchers typically compare a clean input that produces a desired behaviour with a corrupted or counterfactual input that disrupts it. Internal states from one run are patched into the other at selected layers, token positions, attention heads, MLP outputs, residual-stream sites or feature activations. If the patch restores or removes a target behaviour, that site is causally relevant under the intervention. Work on factual recall, circuit discovery and activation patching best practices shows that the result depends strongly on corruption method, patch direction, evaluation metric and the unit of analysis.

The SEO terms—activation patching LLM, causal tracing language model, mechanistic interpretability, activation patching tutorial, causal intervention neural network, path patching, attribution patching, circuit analysis transformer—name genuine mechanisms, but the 2026 research frontier adds an important warning. Single-component patch effects can contain interaction terms with other components, temporally distributed reasoning can require sequential patching, and local linear approximations can fail when the patched state moves far from the clean activation. A patching score is therefore an intervention result under a specified counterfactual—not an automatic statement that one component “contains” or “causes” the entire computation.

1. Start with two runs that differ in one behaviourally meaningful way

Suppose a language model correctly completes “The Eiffel Tower is in the city of” with “Paris”. Create a counterfactual or corrupted run whose output is wrong or whose factual signal is disrupted. Save the internal activations from both runs.

Activation patching asks: if we insert the clean activation from one internal site into the corrupted run, does the correct answer return? Or in the opposite direction, if we insert the corrupted activation into the clean run, does the correct answer disappear?

The asymmetry matters. Clean-to-corrupt restoration and corrupt-to-clean destruction are related and not identical experimental designs.

2. The clean run defines the behaviour we want to explain

The clean prompt should produce the target behaviour reliably. Otherwise patch effects are measured against an unstable reference.

Define the outcome before looking at internal components: target logit, logit difference, probability, sequence-level answer, loss, classification score or another metric.

Mechanistic analysis is strongest when the behaviour itself is sharply operationalised.

3. The corrupted run defines the counterfactual contrast

Corruption can replace a subject, scramble tokens, add noise to embeddings, alter one factual relation or construct a matched counterfactual prompt. Different corruptions remove different information.

A patch restores only what the corruption made relevant. If the corruption destroys several mechanisms at once, patching one component can measure compensation for a complex perturbation rather than one natural computation.

4. Kevin Meng and colleagues used causal tracing to localise factual recall

Locating and Editing Factual Associations in GPT introduced a causal-tracing procedure for factual predictions and reported important effects in middle-layer feed-forward computations at subject tokens. The work then connected localisation to the ROME editing method.

The broader contribution was methodological: intervene on internal activations to test which hidden states mediate a behavioural difference rather than inferring mechanism from attention weights or correlations alone.

5. Patching is stronger than observing an activation

A neuron or head can correlate with a behaviour because both respond to a third variable. Patching changes internal state and observes downstream effect, moving the analysis toward causal evidence.

This does not make patching omnipotent. The intervention can be unnatural, interactions can confound attribution and the counterfactual itself encodes assumptions.

6. The patch site must be typed precisely

“Layer 12” can contain residual input, attention output, attention head result, MLP input, MLP output, residual after addition, normalisation state and many token positions. Patching these sites asks different questions.

Specify model, layer, sublayer, tensor, token position and whether the entire vector or one component is replaced.

7. Residual-stream patching changes all downstream readers of that state

Replace the residual vector at one token and layer and every downstream component reading that state can be affected. The intervention measures the total downstream influence of that state under the new value.

It does not isolate which feature inside the residual vector caused the effect.

8. Head-output patching narrows the unit of analysis

An attention head produces an output contribution to the residual stream. Patching one head’s output can test whether that head’s contribution mediates the target behaviour.

The effect can depend on other heads, MLPs and the state into which the head writes. One head is not an isolated module.

9. MLP patching tests transformed feature content

MLP outputs can encode or transform features before writing them into the residual stream. Patching MLP outputs can localise factual, syntactic or behavioural computations that are not cleanly attributable to individual neurons.

Neuron-level patching increases resolution and multiplies compute and interaction complexity.

10. Token position is part of the causal site

In transformers, the same component at different positions can carry different information. Patching a head at the subject token asks a different question from patching it at the final answer token.

Layer×position heatmaps are common because mechanism is distributed across both depth and sequence.

11. The outcome metric can change the apparent importance map

One experiment can measure probability of the correct token. Another uses logit difference between correct and incorrect answers. Another uses loss. Because softmax saturates and metrics have different scale, patch rankings can change.

Towards Best Practices of Activation Patching in Language Models: Metrics and Methods systematically shows that methodological choices including evaluation metric and corruption can produce substantially different localisation results.

12. Logit difference often gives a more linear behavioural target

For a binary contrast such as “Paris” versus “Rome”, define Δ = logit(Paris)-logit(Rome). Patching changes Δ directly before softmax normalisation.

This can be easier to interpret than probability when one token already dominates or when unrelated vocabulary logits shift.

13. Normalised patch scores require a denominator with meaning

A common idea is to scale the patch effect between corrupt and clean performance: zero means no restoration; one means full restoration. This makes examples more comparable and can become unstable when clean–corrupt difference is small.

Filter or flag examples with weak corruption effects rather than allowing tiny denominators to create extreme scores.

14. Corruption strength controls intervention distance

A severe corruption creates a large clean–corrupt activation difference. Patching then moves the network far through activation space. The intervention can enter nonlinear regimes or states rarely seen naturally.

A mild counterfactual can be more local and can produce a smaller behavioural contrast. Experimental design must balance signal and intervention realism.

15. Noise corruption and semantic counterfactuals answer different questions

Add Gaussian noise to embeddings and ask which activations restore the original fact: this tests robustness to an artificial perturbation. Replace one subject or relation with a meaningful counterfactual: this tests the representation of a structured semantic difference.

Do not generalise a mechanism located under one corruption to every form of counterfactual change without testing.

16. Clean-to-corrupt and corrupt-to-clean patching can disagree

Restoring one clean component into a broken system asks whether that component is sufficient to recover some behaviour in the corrupted context. Injecting corrupted state into a clean system asks whether the component is necessary enough to damage behaviour.

Redundancy and nonlinear interactions can make sufficiency and necessity asymmetric.

17. Whole-vector patching mixes all features at the site

A residual vector can contain many differences between clean and corrupted runs. If behaviour changes after patching the whole vector, we know the site carries some causal difference and not which feature is responsible.

Feature-level patching, sparse dictionaries or directional interventions can refine the hypothesis.

18. Neuron patching can be too coordinate-specific

Superposition means one semantic feature can be distributed across neurons and one neuron can participate in several features. Patching a single neuron can underestimate distributed mechanisms.

Neuron-level localisation is useful when axes are meaningful and should not be assumed universal.

19. Sparse feature patching changes the unit from neuron to learned feature

With an SAE or transcoder, one can patch candidate feature coefficients instead of raw neurons. This can make interventions more semantically interpretable.

The result inherits SAE reconstruction error, feature non-uniqueness and off-manifold intervention risk. Feature patching is a stronger interface and not ground truth.

20. Path patching tries to isolate an edge rather than a node

A node intervention measures all downstream routes from that component. Path patching or edge intervention attempts to change only the signal passed along a selected connection while leaving other routes intact.

This can distinguish direct and indirect effects and requires careful implementation because residual-stream additions create many parallel paths.

21. Causal mediation provides a useful language for patching

Think of input treatment X, internal mediator M and behavioural outcome Y. Patching asks how changing M between counterfactual values affects Y while other parts of the computation follow specified states.

Terms such as natural indirect effect and controlled direct effect help clarify what is being held fixed. Neural networks complicate standard mediation assumptions because mediators interact strongly and are deterministic functions of upstream state.

22. Total effect, direct effect and indirect effect are different causal objects

Change the input and observe Y: total effect. Change one internal mediator while allowing all downstream consequences: mediator intervention effect. Patch an edge while fixing alternative paths: a more direct effect.

Mechanistic claims should state which object was measured.

23. Component rankings can hide interaction effects

In June 2026, The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching re-derived activation patching through causal mediation and showed that individual patch effects can include interactions with the states of other components.

A component can look unimportant alone and matter strongly when another component is in a particular state. Another can look important because the clean–corrupt intervention creates a large interaction term.

24. Interaction is not merely nuisance—it can reveal conditional circuits

If component A only matters when component B is active, that conditionality is part of the mechanism. Trying to assign one context-free scalar importance to A can destroy the structure we wanted to discover.

Measure pair or group patches for components whose single-site effects vary strongly across prompts.

25. Interaction grows with intervention distance

The 2026 multiple-mediators analysis shows interaction terms grow with distance between clean and patched component activations and become small in locally affine regimes.

This connects patching to local linearity: smaller counterfactual moves can be easier to attribute and can produce weaker behavioural signal.

26. Greedy component search can miss conjunctive mechanisms

If two heads matter only together, neither ranks highly under single-head patching. Selecting top individual components can therefore miss the pair.

Combinatorial search is expensive. Interaction diagnostics, attribution methods and circuit hypotheses can narrow which groups deserve joint testing.

27. Attribution patching approximates many interventions with gradients

Full activation patching requires a forward pass per patched component. Attribution patching replaces downstream computation with a first-order approximation using gradients, making large-scale localisation much cheaper.

The approximation is local. Strong nonlinearities, saturated softmax, layer normalisation and cancellation between effects can make it inaccurate.

28. Gradient approximation is a screening tool, not automatic causal ground truth

Use attribution patching to rank candidate sites efficiently, then validate important candidates with real activation patching.

Anthropic’s 2024 circuits update describes failure modes and improvements to attribution patching, including nonlinear softmax effects and cancellation between direct and indirect contributions.

29. Local linearity can be measured empirically

Scale the patch difference by coefficient α from zero to one and measure outcome. If effect grows approximately linearly for small α, a first-order attribution is plausible locally. Curvature or sign reversals signal nonlinear response.

May 2026 work on Continuous-Depth Field Theory for Transformer Patching explicitly studies bounded local linear regimes and propagation of patch effects across depth/token position.

30. Patching effects propagate anisotropically

A local residual change does not spread equally to every later site. Attention patterns, residual pathways and MLP transformations create structured propagation.

Measuring the downstream field of one perturbation can reveal not only whether a site matters but where its influence travels.

31. Attribution graphs formalise local feature-to-feature influence

Anthropic’s 2025 circuit-tracing work constructs local replacement models over interpretable features and uses constrained interventions to test predicted edges. The graphs are explicitly local to prompts and depend on frozen attention/normalisation approximations.

This is a useful evolution from node importance toward interaction structure and remains bounded by local-model assumptions.

32. Constrained patching limits knock-on effects inside a patch range

When intervening across several layers, recomputing every intermediate feature can amplify reconstruction errors. Constrained patching can hold selected feature values fixed through a range to isolate direct effects predicted by an attribution graph.

The intervention answers a more artificial but cleaner causal question. Iterative patching answers a more natural propagation question and can accumulate error.

33. Iterative patching allows indirect consequences and compounds approximation error

Recompute downstream sparse features after an early feature intervention and the intervention can propagate through the replacement model. This captures indirect effects within the patched range.

If the sparse model is imperfect, each encode/decode step introduces error. A long intervention path can drift far from the native model.

34. Causal tracing can localise information and still fail to identify the best edit site

Work asking Does Localization Inform Editing? reported that causal-tracing localisation and edit success can be weakly related in factual editing experiments.

This is an important boundary. The site mediating factual recall under one intervention is not necessarily the site whose weight update yields the safest or most effective persistent edit.

35. Patching locates causal flow under a run, not permanent storage in one address

Information can be distributed, reconstructed or redundantly encoded. A middle-layer MLP can be causally important to recall without being the only place a fact is stored.

Use “mediates this behaviour under this counterfactual” rather than “stores the fact here” unless additional evidence supports storage claims.

36. Sequential reasoning makes one-token patching incomplete

Reasoning unfolds across multiple generated positions. A mechanism can carry a state through several tokens before affecting the answer. One static patch position can miss the temporal pathway.

August 2026 work on Sequential Activation Patching studies chain-of-thought-conditioned effects across reasoning trajectories and groups of attention heads.

37. Sequential patching treats reasoning as a time-indexed causal process

Instead of one activation tensor, define sites across token positions and layers as the model generates. Patch states in temporal order or analyse their aggregated effect.

The method must specify whether later states are recomputed after earlier patches or independently substituted. These correspond to different causal interventions.

38. Multi-head patching tests distributed subcircuits

If reasoning support is spread over several heads, patching one at a time underestimates the group. Joint patches can test sufficiency and interaction.

Group definition should ideally be chosen independently of the outcome metric to avoid circular discovery and validation.

39. Random activation controls are essential

Patch a random site’s activation with a vector of matched norm or patch a randomly selected clean example. The target intervention should outperform these controls if semantic state matters.

Without controls, any large activation replacement can perturb logits substantially.

40. Mean activations create a different counterfactual from exemplar patches

Instead of copying one clean activation, average activations from many examples representing a target value and patch the difference between means. This reduces exemplar noise and creates a concept-level intervention.

Mean vectors can blur multimodal concepts and move the model into an activation state no individual example produces.

41. Patch magnitude should be compared with natural activation variability

Measure the norm of clean–corrupt difference relative to the distribution of within-class activation differences. A patch ten standard deviations away is a much more invasive intervention than a typical natural change.

Locality of the intervention is part of causal credibility.

42. The patch can create a hybrid state that never occurs naturally

Copy one component from clean run while leaving all other components in corrupted state. The resulting combination may be internally inconsistent.

This is both the method’s strength and its limitation. It isolates a mediator by constructing a surgical counterfactual. The counterfactual can be off-manifold.

43. Off-manifold risk grows with distributed representations

If several features co-adapt, changing one component without its partners can create a state outside the training distribution. Large behavioural effects can reflect inconsistency rather than ordinary causal function.

Use matched group patches, small interpolation steps and natural exemplar swaps to assess robustness.

44. Patching and probing form an evidence ladder

A probe finds where information is accessible. Patching asks where changing the state affects behaviour. Sparse feature analysis proposes what the state means. Path patching asks how influence moves.

No one tool replaces the others. Together they move from correlation toward mechanism.

45. Patching and steering differ in purpose

Patching usually replaces an activation with a counterfactual state from another run to test causal localisation. Steering deliberately adds or rotates a direction to control behaviour at inference.

The same activation geometry can support both, but the evaluation standards differ: explanation versus reliable control.

46. The activation-patching audit

What target behaviour is being explained?
What clean and corrupted inputs create the contrast?
Does the corruption change only the intended semantic variable?
What activation site is patched?
What token positions?
What patch direction: clean→corrupt or corrupt→clean?
What outcome metric?
How is patch effect normalised?
How large is the intervention relative to natural variation?
What random or matched controls?
Are whole vectors, neurons, heads or sparse features patched?
Are pair/group interactions tested?
Is attribution patching validated with exact patches?
Does the conclusion survive alternative corruption methods?
Does sequential reasoning require time-indexed interventions?
What claim is supported: information flow, causal mediation, necessity, sufficiency or storage?

47. What students should remember

Activation patching is an internal causal intervention, not a correlation map.
Clean and corrupted runs define the counterfactual being tested.
Corruption method and outcome metric can change localisation results.
Whole-vector patches localise a site but not necessarily a semantic feature.
Path patching tries to isolate causal routes.
Attribution patching is a gradient-based approximation that needs validation.
Interaction effects can make single-component rankings misleading.
Sequential reasoning can require sequential patching across generated positions.
Patching localises causal mediation under an intervention; it does not automatically identify permanent storage.
Off-manifold hybrid states and intervention magnitude must be audited.

48. The deep principle

Patching is powerful because it refuses to infer causality from observation alone. It changes the internal state and watches the computation respond.

A component is causally important only relative to a counterfactual, a patch, an outcome and the other components around it. Mechanism lives in interactions and paths, not in one heatmap cell.

Advanced Technical Expansion I | Causal Estimands, Patch Metrics and Intervention Geometry

The readable spine explains what patching does. This first technical layer asks what quantity it estimates. Mechanistic interpretability often inherits causal language—total effect, indirect effect, mediator, intervention—without writing down the corresponding estimand. That omission matters because different patching procedures hold different parts of the model fixed. Once the causal object is explicit, disagreements among patching methods become interpretable rather than mysterious.

A. Write a minimal structural causal model for the network slice

Let X represent an input intervention, M an internal mediator and Y a behavioural outcome. The network computes M=f_M(X,U_M) and Y=f_Y(X,M,U_Y), where the U variables summarise other upstream state. In a deterministic forward pass, the U terms can be understood as fixed context inherited from the rest of the computation.

A patching experiment constructs counterfactual combinations such as “corrupted X with clean M”. This combination is not generated by the ordinary model. It is produced by intervention.

The causal meaning therefore comes from what is held at its natural value and what is surgically replaced.

B. Total effect measures the behavioural difference between two inputs

Let X=clean and X=corrupt define two runs. The total effect is the difference in Y between those runs. Before localising a mechanism, verify that the total effect is large and stable enough to explain.

If the clean and corrupt outputs barely differ, a large normalised patch score can be driven by a tiny denominator and has little behavioural meaning.

C. Restoration patching resembles an indirect-effect experiment

Run the corrupted input while replacing M with the value observed under the clean input. Compare the resulting Y with ordinary corrupted Y. The difference asks how much of the clean–corrupt behavioural gap can be restored by assigning M its clean value.

The interpretation depends on the rest of the system remaining in corrupted state at the patch boundary. Downstream components then respond naturally to the hybrid state.

D. Destruction patching asks a different counterfactual

Run the clean input while assigning M its corrupted value. The resulting damage measures whether that mediator is necessary enough, in the clean context, to sustain the behaviour.

Redundant pathways make restoration and destruction asymmetric. One clean component can be sufficient to rescue a corrupted system while corrupting the same component has little effect because another clean pathway compensates.

E. Controlled versus natural mediator values change the causal question

A natural indirect effect uses mediator values that arise naturally under alternative treatments. A controlled direct effect fixes a mediator to one externally chosen value. Neural patching frequently mixes these ideas.

Patching from another prompt uses a naturally observed activation from that prompt but creates an unnatural combination with the current run. Mean-difference interventions use a synthetic mediator value not directly observed in one run.

State the counterfactual source explicitly.

F. Patching one mediator while others remain natural includes interaction terms

The 2026 multiple-mediators analysis makes this precise. If Y depends on both M1 and M2 with interaction, changing M1 while M2 remains at one contextual state produces a different effect from changing M1 while M2 is in another state.

A patch score for M1 therefore includes not just an isolated “importance” but the state-dependent interaction of M1 with the rest of the mechanism.

This is not statistical noise. It can be the computational logic.

G. A two-mediator toy model makes the interaction visible

Let outcome Y=M1 + M2 + γ M1M2. The effect of changing M1 from 0 to 1 is 1 + γM2. If M2=0, effect is one. If M2=1, effect is 1+γ.

No context-free scalar captures M1’s contribution unless γ is zero. A component can look weak on prompts where its partner is absent and strong where the partner is active.

This tiny equation explains why prompt-specific patch maps can legitimately differ.

H. Group patching estimates a different causal object than summing single patches

Patch M1 and M2 together. The group effect includes their joint contribution. Sum the separate effects and the interaction term can be double-counted, missed or assigned inconsistently depending on the baseline.

Therefore “top-k component effect” is not necessarily the sum of top-k single-component scores. Joint validation is required for circuit sufficiency claims.

I. Shapley-style attribution is one possible interaction-aware allocation

Game-theoretic Shapley values average marginal contribution across component orderings, distributing interactions among participants. In principle this gives one fair allocation under its axioms.

Exact computation is combinatorial and still depends on the coalition intervention semantics. It produces an attribution convention, not a unique causal decomposition written into the network.

Use such methods when a component-allocation score is operationally needed, while preserving the underlying interactions separately.

J. Normalised restoration can exceed one

A patch can improve the corrupted run beyond clean baseline on the chosen metric, producing restoration greater than 100 percent. It can also worsen behaviour, yielding negative restoration.

These values are not automatically errors. The hybrid state can create synergistic or pathological effects. Clip only for visual convenience if raw values are preserved and explained.

K. Outcome scale affects patch comparability

Probability changes depend on softmax context. A one-unit logit shift has different probability impact when the baseline probability is 0.01 versus 0.9. Logit difference is closer to an additive pre-softmax scale.

Sequence-level exact match is behaviourally meaningful and discontinuous. Loss is smooth and can include unrelated tokens. Choose the metric that matches the causal question, and test whether conclusions survive a second sensible metric.

L. Contrastive logit metrics need a justified distractor

For IOI or factual tasks, researchers often use the logit difference between the correct and a specific incorrect answer. This focuses the mechanism on one contrast.

If the patch shifts probability toward a third answer, the binary metric can miss important behaviour. Inspect full output distribution or several distractors for a diagnostic subset.

M. Patch effect should be reported with the clean–corrupt gap

A normalised score of 0.8 from a clean–corrupt difference of 0.05 logits is not equivalent to 0.8 from a difference of 10 logits. Always report raw total effect and raw patch effect alongside normalised restoration.

This prevents a clean visual heatmap from hiding weak behavioural contrast.

N. Intervention distance can be measured in several geometries

Euclidean norm of clean–corrupt activation difference is simplest. Mahalanobis distance normalises by activation covariance. Cosine distance focuses on direction. SAE feature changes can measure semantic displacement.

Each geometry answers a different notion of “how far off-manifold did we move?”. Compare with natural within-class and between-class distances.

O. Interpolate between corrupted and clean activation to obtain a dose–response curve

Set M(α)=M_corrupt + α(M_clean-M_corrupt) for α from zero to one, and optionally modest values beyond. Plot Y(α).

A near-linear curve supports local first-order attribution. Thresholds, saturation, curvature and sign reversal reveal nonlinear mechanism or interaction.

Dose–response is a much stronger patch characterisation than one endpoint effect.

P. Patch direction itself can be decomposed

The vector ΔM=M_clean-M_corrupt may combine many semantic changes. Project it onto probe directions, SAE features, principal components or known concept vectors. Measure which projections reproduce the patch effect.

This moves from site localisation to feature attribution and must account for interactions among projected components.

Q. Feature-projected patching can test semantic mediation

If a clean–corrupt difference is hypothesised to involve “subject identity”, project onto a validated subject-identity feature subspace and patch only that component. Does behaviour restore?

Success is stronger evidence that the semantic feature mediates the effect than whole-vector patching. Failure can mean the feature model is incomplete or the effect relies on other correlated state.

R. Off-manifoldness can be tested through reconstruction or density proxies

After patching, pass the hybrid activation through an SAE and measure reconstruction error, or compare it with nearest natural activations. Large anomaly scores suggest the intervention left familiar state space.

These are proxies, not proofs. A model can validly enter rare states absent from the reference corpus.

S. Exemplar patching preserves a natural component value and not a natural joint state

Copying M from one real run gives a plausible marginal activation at that component. Combined with the rest of another run, the joint state can still be impossible under the model’s ordinary dynamics.

This distinction matters for claims that activation patching is always “on manifold” because the patched vector came from real data.

T. Mean-difference patching trades exemplar noise for synthetic-state risk

Averaging many target examples can estimate a stable concept displacement. It also erases multimodality and can produce a mean state that no individual run takes.

Compare exemplar, matched-pair and mean-difference interventions. Consistent effects across methods strengthen the claim.

U. Matched counterfactual pairs reduce nuisance differences

Construct clean/corrupt prompts differing only in the variable of interest while preserving length, syntax, tokenisation and unrelated semantics. The activation difference then has fewer nuisance components.

Perfect matching is often impossible. Document residual differences and use multiple pairs.

V. Tokenisation can make a semantic counterfactual structurally different

Replace one name with another and the tokenizer can produce a different number of tokens. Position alignment then becomes ambiguous. Patching “the subject token” across runs may compare non-equivalent sequence sites.

Choose matched tokenisations where possible or define span-level aggregation and alignment explicitly.

W. Position shifting creates downstream attention changes before the patch

If a corruption changes sequence length or positions, attention patterns and rotary/positional encodings differ everywhere. A patch effect can then reflect a much larger counterfactual than the intended semantic change.

Keep structural prompt form matched for localisation experiments whenever possible.

X. The corruption should produce the intended behavioural failure consistently

Some corrupted prompts remain correct because the model uses redundant cues or memorised priors. Exclude or separately analyse examples where the corruption fails to alter behaviour.

Otherwise the patch map mixes “mechanism recovery” cases with cases that needed no recovery.

Y. Behavioural heterogeneity should be preserved, not averaged away immediately

Average patch effects across prompts only after inspecting the distribution. A component can have positive effects for one subgroup and negative effects for another, yielding a misleading near-zero mean.

Cluster prompts by patch-effect profile. Heterogeneity can reveal multiple circuits solving the same task.

Z. Bootstrap prompts to estimate uncertainty on patch maps

Resample prompt pairs and recompute average effects. Confidence intervals reveal which heatmap differences are stable. Multiple sites and layers create many comparisons, so tiny differences should not receive strong stories.

For prompt families with shared templates, bootstrap at the independent template or semantic-instance level.

AA. Patch-map smoothing can create structures not present in raw estimates

Averaging neighbouring layers or token positions improves readability and can blur sharp transitions. If smoothing is used, show raw and smoothed maps or specify the kernel.

Mechanistic boundaries should not be manufactured by visualisation choices.

AB. Multiple-comparison correction matters when declaring isolated “hot” components

Thousands of heads, positions or neurons can be tested. The maximum patch effect can be large under noise. Use held-out validation, permutation baselines or statistical correction when significance claims are made.

Circuit discovery and circuit validation should preferably use different data.

AC. Discovery and validation should be separated

Use one set of prompts to find candidate components. Freeze the hypothesis. Test patch effects on new prompts, new templates and ideally new task variants.

This prevents an elaborate patch search from overfitting one benchmark’s quirks.

AD. The causal-estimand evidence card

Treatment: clean/corrupt input relation.
Mediator: exact activation site and unit.
Outcome: metric and behavioural contrast.
Patch direction: restoration or destruction.
Counterfactual source: exemplar, mean, noise or synthetic vector.
Normalisation: raw and scaled effect.
Intervention distance: relative to natural activation variation.
Interactions: single, pair and group tests.
Prompt uncertainty: bootstrap and heterogeneity.
Off-manifold checks: reconstruction/density proxies.
Validation: held-out counterfactual pairs.

The first advanced layer therefore reframes activation patching from “copy an activation and see what happens” into a causal measurement with explicit treatment, mediator, outcome and counterfactual. The second layer turns those measurements into circuits: how to search efficiently, validate edges, handle interacting mediators, trace reasoning over time and know when a local mechanism generalises beyond the prompt that revealed it.

Advanced Technical Expansion II | Circuit Discovery, Interaction Structure, Sequential Patching and Validation

The first technical layer treated patching as a causal estimand. This second layer asks how those estimands become a circuit claim. A circuit is not a list of components with high patch scores. It is a structured account of how information is transformed, routed and combined over components, positions and time. The research problem therefore changes from ranking nodes to testing paths, interactions, sufficiency, necessity, generalisation and temporal order.

AE. Circuit discovery should separate candidate generation from causal confirmation

Use cheap signals—attention patterns, probe maps, attribution gradients, SAE activations or prior mechanistic hypotheses—to generate candidate components. Then use exact patching on held-out examples to test those candidates.

Discovery and confirmation on the same examples risks selecting components that happen to explain idiosyncratic prompts. The stronger circuit is the one that survives new templates and semantic variants.

AF. Attribution patching is an efficient screen, not the final circuit

First-order attribution can estimate the effect of many candidate interventions in one or a few backward passes. This makes it attractive for searching thousands of heads, positions or sparse features.

Validate top candidates, random candidates and interaction-sensitive candidates with exact forward-pass patching. Report the precision of the screen: what fraction of high-attribution sites produce large exact effects?

A screening method earns trust by predicting interventions it did not itself execute.

AG. False negatives in attribution matter as much as false positives

A component can have near-zero first-order attribution because two downstream effects cancel locally or because its influence is nonlinear. Exact group patching can reveal an important mediator missed by gradient screening.

Sample some low-attribution components for exact validation. Otherwise a search pipeline can only estimate its precision, not its recall.

AH. Necessity tests remove the proposed circuit from a clean run

Once a candidate circuit is identified, corrupt, ablate or replace its components while preserving the rest of the clean computation. If behaviour collapses, the circuit is necessary under that intervention.

Redundancy can make a real circuit fail necessity: another route can compensate. Necessity is strong evidence when present and not a required property of every causal contributor.

AI. Sufficiency tests isolate the proposed circuit inside a corrupted or ablated model

A sufficiency test keeps or restores the candidate circuit while disrupting alternative pathways. If the target behaviour survives, the circuit is sufficient under that constructed context.

Exact sufficiency can be unrealistic for highly distributed networks. Partial sufficiency—recovering a large fraction of the clean behavioural gap—can still support a useful mechanism.

AJ. Completeness asks how much unexplained effect remains outside the circuit

A circuit can be necessary but incomplete. Measure the residual behavioural difference after preserving the circuit and randomising or ablating the rest. Measure how much behaviour alternative components can still restore.

Mechanistic explanations need a missing-mass concept just as sparse dictionaries need reconstruction error. “These heads matter” is weaker than “these heads and paths account for most of the measured effect.”

AK. Minimality is different from completeness

A complete circuit can contain redundant components. Remove each component or group and measure whether explanatory power remains. A smaller circuit that preserves effect can be easier to understand.

Do not optimise minimality so aggressively that distributed redundancy is erased from the explanation. Redundancy can be a real property of the model.

AL. Interaction graphs should contain hyperedges when pairwise edges are insufficient

If A and B jointly matter while neither matters alone, an ordinary directed graph with scalar node importance can misrepresent the mechanism. Hyperedges or interaction annotations can represent conjunctive dependencies.

Higher-order interactions become combinatorial. Use domain hypotheses, pairwise screens and local nonlinear-response tests to decide which groups deserve explicit measurement.

AM. Interaction tests should use factorial patches

For two candidate mediators A and B, evaluate four states: neither patched, A only, B only, both. Compare the joint effect with the sum of individual effects.

The deviation is an operational interaction term under that counterfactual. Repeat across prompts to determine whether the interaction is stable or context-dependent.

AN. Prompt-conditioned circuits can be legitimate mechanisms

A model need not use one universal circuit for every instance of a task. Different prompts can recruit different heads, layers or features while implementing the same input-output relation.

Cluster examples by patch profiles. A family of circuits can be a more faithful explanation than one averaged circuit.

Generalisation claims should state whether the mechanism is universal, dominant, subtype-specific or exemplar-local.

AO. Template generalisation is the first test beyond prompt memorisation

Discover a circuit on one linguistic template and test it on paraphrased templates expressing the same relation. If localisation moves radically, the mechanism can depend on syntax or token position rather than the abstract task.

Both outcomes are informative. A template-stable circuit suggests reusable computation; template-specific circuits reveal interface dependence.

AP. Semantic generalisation is stronger than template generalisation

Test new entities, relations, value ranges and domains while holding the computational task fixed. A factual-recall circuit found for capitals should not automatically be claimed for dates, professions or multi-hop knowledge.

Mechanisms can be domain-specialised even when output format is identical.

AQ. Counterfactual generalisation tests whether the circuit tracks the variable rather than the answer token

Construct examples where the correct answer changes while superficial prompt form remains. A real variable-tracking mechanism should patch according to the underlying relation, not one memorised token identity.

Balanced answer tokens reduce the risk that a circuit simply promotes a frequent output.

AR. Time should be treated as another circuit dimension in autoregressive reasoning

For generated reasoning, index component activations by generation step as well as layer and token position. A feature can be irrelevant at early steps, become critical in the middle and then be copied forward.

The circuit is then a space-time graph, not a static layer diagram.

AS. Sequential patching must decide whether patched history changes future activations naturally

One design patches an early state and lets the model regenerate subsequent states, testing downstream causal consequences. Another independently patches a sequence of later states from the clean trajectory, asking whether reproducing the trajectory rescues behaviour.

The first is closer to an intervention on a dynamical system; the second is closer to trajectory transplantation. Both can be useful and answer different questions.

AT. Chain-of-thought text and hidden-state reasoning are different observables

Generated reasoning tokens influence future computation and also provide a public trace. Patching hidden activations can alter later hidden state without visibly changing the intermediate text, or alter text while preserving the final answer.

Evaluate hidden mechanism, generated rationale and final behaviour separately. A model’s verbal explanation is not a complete window into its internal causal path.

AU. Teacher-forced and free-running sequential patching answer different questions

Under teacher forcing, the same reference tokens are fed after each step, isolating hidden-state changes. Under free running, patched states can change generated tokens, which then alter all future computation.

Free running is behaviourally realistic and creates branching trajectories that are harder to compare. Report the generation regime.

AV. Circuit discovery at one temperature may not transfer to another decoding regime

Autoregressive sampling changes future token trajectories. A circuit discovered under greedy decoding can face different contexts under stochastic sampling.

For mechanisms claimed to govern open-ended generation, test several decoding settings or condition the claim on deterministic runs.

AW. Feature circuits inherit sparse-replacement-model error

If attribution graphs use SAEs or transcoders, each edge is analysed through an approximate replacement model. Reconstruction error can create, erase or reroute apparent influence.

Measure replacement-model fidelity at every patched layer and compare selected causal effects in the native model. Do not treat the surrogate circuit as exact because its nodes are interpretable.

AX. Constrained patching tests direct graph edges and suppresses indirect alternatives

If an attribution graph predicts A→B, intervene on A while clamping other candidate mediators so B’s change can be attributed more directly. This creates a controlled local test of the edge.

The clamping is artificial. It establishes a controlled causal relationship, not the unmodified network’s full natural trajectory.

AY. Iterative patching tests whether the surrogate can reproduce natural propagation

Let the sparse replacement model recompute each downstream feature after an intervention. Compare the predicted output effect with the native model’s exact intervention.

Agreement supports the surrogate circuit as a model of propagation. Divergence localises where the replacement model loses important nonlinear or uninterpreted state.

AZ. Cross-model circuit comparison needs representation alignment

Two independently trained models can solve the same task with different head numbering and rotated feature spaces. Compare circuits through functional roles, representation similarity, matched sparse features or intervention effects rather than raw component IDs.

A replicated mechanism is stronger evidence when analogous causal roles recur across seeds or architectures.

BA. Model scale can change whether mechanisms are dedicated or distributed

A small model can rely on one head because capacity is constrained; a larger model can distribute the same function across several heads and MLPs. Directly comparing sparsity of patch maps across scale can confuse redundancy with absence.

Measure group sufficiency and redundancy as well as individual effects.

BB. Component ablation can trigger compensation not present under patching

Zeroing a head removes both clean and corrupt information and creates a value the model rarely sees. Replacing with a matched baseline or mean activation can be a gentler necessity test.

Compare several ablation baselines. Necessity claims that survive them are stronger.

BC. Zero ablation, mean ablation and resample ablation define different counterfactuals

Zero says “remove the signal and replace it with nothing”. Mean says “replace with typical background state”. Resample says “replace with another natural state”. The output effect can differ dramatically.

There is no universally neutral baseline. Choose the one closest to the scientific question and test robustness to alternatives.

BD. Circuit metrics should include task fidelity and localisation fidelity

A circuit can reproduce output behaviour while localising the wrong components due to redundancy. Another can identify real causal components and fail to reproduce behaviour because important interactions are missing.

Evaluate circuit sufficiency/necessity and component-level causal precision separately.

BE. Sparse circuits should be compared with random circuits of equal size

If a proposed ten-head circuit preserves 70 percent of behaviour, compare with many random ten-head sets. This shows whether the selected circuit is unusually explanatory relative to size.

Also compare with simple heuristics such as highest activation or attention weight. Mechanistic discovery should outperform cheap correlational selection.

BF. Circuit edge validation should use interventions on edges or mediators, not correlation alone

Two components can co-activate because both receive the same upstream signal. An edge claim says influence flows from one to the other.

Path patching, mediator clamping or controlled feature intervention can test the directional relationship. Correlation provides candidates, not edges.

BG. Causal scrubbing tests whether an abstract computational graph explains behaviour

One can hypothesise an abstract circuit and resample internal variables in ways that preserve the variables the hypothesis says matter while randomising irrelevant details. If behaviour remains, the abstraction gains support.

The resampling distribution and correspondence between abstract variables and network activations are part of the hypothesis. Causal scrubbing is strong when the abstraction makes risky predictions that survive.

BH. Explanation fidelity should be evaluated on counterfactuals not used to build the explanation

A circuit discovered on one set of factual substitutions should predict patch outcomes for new substitutions, paraphrases and controlled perturbations.

Mechanistic explanations should generalise like models: discovery data and evaluation data need separation.

BI. Explanation compression can be quantified

Count components, edges and conditions required to reproduce a target fraction of behaviour. A smaller validated circuit provides greater explanatory compression.

Do not use compactness alone as truth. A simple but incomplete story can be misleading. Plot explanatory fidelity against circuit complexity.

BJ. Circuit explanations can be hierarchical

At a high level: retrieve subject, combine relation, promote answer. At a low level: particular heads and MLP features implement each stage. Both levels can be valid.

Hierarchy prevents a 500-node graph from being mistaken for the only possible explanation. The receiver can zoom from mechanism summary to causal components.

BK. A circuit can be locally faithful and globally wrong

Attribution graphs and prompt-specific patch analyses can accurately explain one example while failing elsewhere. Local faithfulness should be labelled local.

Aggregate local circuits across examples only after measuring which motifs recur and which are example-specific.

BL. Mechanistic motifs can recur without exact component identity

Several prompts can use “copy previous entity”, “compare two candidates” or “retrieve factual value” motifs implemented by different concrete heads or features.

Define motifs functionally and test their interventions. This supports higher-level mechanistic generalisation without requiring one universal sparse subgraph.

BM. Intervention order can reveal causal precedence

Patch an early hypothesised feature, then measure whether a later feature changes naturally. Patch the later feature alone and see whether the early feature remains unchanged. This asymmetry can support directional flow.

Residual connections and feedback-free transformer depth simplify some causal ordering, while token-generation recurrence reintroduces temporal loops across decoding steps.

BN. Sequential reasoning creates cross-token feedback through generated text

At decoding step t, the model emits a token that becomes input at t+1. Hidden-state interventions can therefore influence future computation through both internal activations and changed external token history.

A full causal graph of generation includes the emitted token as a mediator. Teacher-forced analyses remove that route; free-running analyses include it.

BO. Long-form reasoning needs outcome metrics beyond final-answer correctness

A patch can alter intermediate reasoning, length, consistency or confidence while leaving the final answer unchanged. Conversely, one token change can flip the final answer while most reasoning remains intact.

Evaluate intermediate factual states, generated rationale properties and final answer separately when claiming a reasoning mechanism.

BP. Safety-related patching needs especially strong semantic controls

Features associated with refusal, harmful intent or deception can be correlated with topic, style or policy language. A patch that changes refusal may alter broad conversational state rather than one safety computation.

Educational analysis should focus on validation methodology and avoid operational recipes for bypassing safeguards. Safety mechanisms deserve causal evidence plus deployment controls.

BQ. The circuit evidence card

Task: exact behaviour and metric.
Discovery: method used to nominate components.
Validation data: held-out prompts/templates/domains.
Nodes: heads, MLPs, residual sites or sparse features.
Edges: intervention evidence for direction.
Interactions: pair/group/higher-order effects.
Necessity: behaviour loss under removal.
Sufficiency: behaviour preserved/restored by circuit.
Completeness: residual unexplained effect.
Minimality: redundant components and size–fidelity curve.
Temporal scope: one pass or autoregressive sequence.
Surrogate fidelity: SAE/transcoder replacement error.
Generalisation: templates, entities, domains and model seeds.
Safety boundary: operational details withheld where necessary.

BR. Worked experiment: interaction-aware two-head circuit

Create a synthetic transformer-style module where head A writes signal only interpretable when head B sets a gate. Single patch A in the wrong B state has little effect; joint A+B patch restores behaviour.

Compare single-head ranking, pairwise factorial patches and attribution gradients. The experiment demonstrates why greedy top-component selection can miss conjunctive mechanism.

BS. Worked experiment: attribution screen followed by exact validation

Measure exact patch effects for a manageable model across all heads. Separately compute attribution patching estimates. Evaluate rank correlation, top-k precision and false-negative rate.

Then increase corruption magnitude and observe when the gradient approximation degrades. This maps the local-linear regime empirically.

BT. Worked experiment: sequential reasoning trajectory

Design a multi-step arithmetic or symbolic task with known intermediate states. At each generated step, patch a candidate hidden feature from a counterfactual trajectory and measure both next-step state and final answer.

Compare teacher-forced and free-running patching. The divergence quantifies how much causal effect flows through generated tokens versus hidden-state propagation.

BU. Worked experiment: discovery/validation split for a factual circuit

Use half of factual prompts to identify top mediators. Freeze the circuit. On held-out relations and paraphrases, test restoration, destruction and group sufficiency. Compare with random circuits of equal size.

A mechanism that generalises across entities and templates earns a stronger label than one that only fits the discovery set.

BV. Final release gate for activation patching

The clean–corrupt behavioural contrast is stable and meaningful.
The corruption preserves unrelated structure as far as possible.
Patch site, token position and tensor are explicit.
Raw and normalised outcomes are both reported.
Intervention distance is compared with natural variability.
Multiple sensible outcome metrics are checked.
Restoration and destruction are not conflated.
Single-component rankings are checked for interactions.
Approximate attribution is validated with exact patching.
Off-manifold/hybrid-state risks are acknowledged.
Discovery and validation data are separated.
Necessity, sufficiency and completeness are evaluated separately.
Sequential tasks include temporal patching where appropriate.
Surrogate sparse models report reconstruction fidelity.
Mechanism claims are scoped to prompts/domains/models actually tested.
Localisation is not automatically equated with storage or best edit location.
Safety-related analysis avoids becoming a safeguard-bypass recipe.

The longform conclusion is that activation patching is best understood as experimental causal science conducted inside a deterministic computational system. The network gives us extraordinary access—we can intervene on hidden states with precision impossible in many natural sciences. That access does not remove the need for causal discipline. Counterfactual design, mediator interactions, hybrid-state validity, outcome metrics, statistical uncertainty and generalisation still determine whether the resulting story is a mechanism or merely an attractive heatmap.

The strongest circuit explanation is not the one with the brightest patched component. It is the one that predicts new interventions, survives new examples, accounts for interactions and leaves little unexplained behaviour behind.

Sources and research boundaries

Continue in the Representation & Cognitive Tools library

Return to the World Representation & Cognitive Tools canonical owner. Related reading: Linear Probing & Diagnostic Classifiers · Sparse Autoencoders for Neural Representation · Representation Similarity Analysis.

Activation Patching & Causal Tracing | How Interventions Locate Computation Inside Neural Networks