Advanced Technical Expansion I | Capacity Geometry, Sparse Features and the Mathematics of Interference

The readable spine gives the core intuition: sparse features can share a finite activation space because they rarely collide. This expansion asks how far that idea can be made quantitative. We will treat superposition as a representational allocation problem: choose feature directions, norms and nonlinear thresholds so that expected task error stays low even when feature count exceeds dimension. The calculations are deliberately small enough to inspect, but the principles scale to the interpretability questions raised by modern language models.

A. Begin with an explicit sparse feature model

Let x∈R^n represent n latent features. For the simplest teaching model, each feature is non-negative and active independently with probability p_i. When active, feature i has a magnitude drawn from some distribution. A linear encoder maps features into a d-dimensional hidden space with d<n:

h = W x,     W ∈ R^{d×n}.

The columns w_i of W are feature directions. If n≤d, the model can make them orthogonal. If n>d, some overlap is unavoidable. The decoder or downstream computation must therefore tolerate cross-talk.

This setup separates three quantities that are often blurred in prose: feature count n, activation dimension d, and active features per example. Superposition becomes attractive when n is large, d is limited and simultaneous activity is sparse.

B. The Gram matrix is the interference ledger

With tied linear decoding x_hat=Wᵀh=WᵀWx, define the Gram matrix G=WᵀW. Then reconstructed coordinate i is

x_hat_i = G_ii x_i + Σ_{j≠i} G_ij x_j.

The diagonal term carries the intended feature. Every off-diagonal term is interference from another feature. In an orthonormal code, G=I. In a superposed code, the off-diagonal structure records the price of sharing dimensions.

Because G is directly measurable from a proposed feature dictionary, it is a useful first diagnostic. Pairwise cosine similarities, coherence, eigenvalue spectrum and block structure all tell us how features are packed.

C. Expected interference depends on co-activation probability

Suppose feature activations are independent Bernoulli indicators for the moment and each active magnitude is one. For an inactive feature i, the linear reconstruction receives false signal Σ_j G_ij x_j. The expected squared interference contains terms weighted by the activation probabilities of the other features.

If p is small, most terms are absent on one input. The same non-orthogonality that would be disastrous for dense features can be acceptable for sparse features. This is the central economic calculation: a collision cost is multiplied by the probability that colliding features appear together.

Independence is not required for the argument, but it makes the formula simple. With correlated features, replace products of marginal probabilities by joint co-activation probabilities. The optimiser should care about the co-occurrence graph, not just marginal sparsity.

D. A co-occurrence graph suggests which features can safely share directions

Construct a graph whose nodes are candidate features and whose edge weight between i and j measures how often or how consequentially they co-activate. Features connected by strong edges should interfere as little as possible. Features with weak or zero edges can share representational directions more aggressively.

This creates a graph-embedding view of superposition. We seek feature vectors whose dot products are small where co-occurrence cost is high and can be larger where collisions are unlikely. The optimal geometry need not be globally uniform.

Real data make this important. Programming-language features may co-occur often with syntax features. Musical notation features may rarely appear in the same contexts. The network can allocate overlap differently across these feature families.

E. Feature importance turns geometry into weighted resource allocation

Let feature i carry importance weight α_i in the training objective. Reconstruction or behavioural errors on that feature then cost more. A high-importance feature can earn a larger norm, a more isolated direction or lower collision exposure. Low-importance features can be packed densely or omitted.

This provides a principled explanation for uneven interpretability. A model is not obligated to distribute representational fidelity equally among concepts. Training loss allocates capacity according to frequency, gradient magnitude and consequence.

When an interpretability study finds one crisp feature and one noisy feature, the difference may reflect task importance rather than a failure of the analysis tool.

F. Feature norms change both signal strength and collateral damage

If column w_i has norm greater than one, feature i contributes a stronger hidden activation and receives a stronger reconstruction projection. It also creates larger interference along directions that overlap with it.

Normalising all feature vectors is convenient for studying angular packing and can erase this allocation. Therefore report direction and norm separately. In a learned network, vector length can be part of the code’s resource allocation.

This also matters for sparse autoencoders. Decoder-column normalisation is often needed because activation magnitude and decoder norm otherwise have a scaling degeneracy. The native network representation need not obey the same constraint.

G. ReLU converts symmetric linear error into asymmetric false positives and false negatives

Consider a decoder x_hat=ReLU(WᵀWx+b). Negative interference on an inactive feature can be clipped to zero. Positive interference can create a false activation. A negative bias can act as a threshold, requiring the intended signal to exceed background cross-talk.

This gives the feature geometry a preferred sign structure. Arrangements that cause interference to point negative for inactive features can be cheaper than equally large positive interference. The network can exploit nonlinear thresholds as an error-correcting device.

Superposition is therefore not only a question of vector packing. It is vector packing plus the nonlinear cleanup available downstream.

H. Bias thresholds create a precision–recall trade-off for latent features

Raise the ReLU threshold and weak interference disappears, improving precision for feature detection. Weak true activations also disappear, reducing recall. Lower the threshold and subtle features survive while more cross-talk leaks through.

This looks very much like signal detection. The representation can be viewed as noisy measurement followed by a threshold. The optimum depends on the cost of false activation versus missed activation in the training task.

A toy experiment can sweep bias and measure feature precision, recall and reconstruction loss under different sparsity levels. This makes the nonlinear cleanup mechanism quantitative.

I. Antipodal directions can share one axis when features are mutually exclusive

In a signed hidden space, two mutually exclusive features can sometimes occupy opposite directions on the same axis. If only one is active at once and downstream nonlinearity or sign-sensitive readout separates them, one dimension effectively supports two features.

This is an extreme example of using data statistics to exceed a naïve one-axis-per-feature limit. It fails immediately if both features can co-occur and must be represented simultaneously in a simple additive code.

The example reinforces the rule that feature capacity is conditional on allowed activation patterns.

J. The Welch bound gives a lower bound on how small pairwise coherence can become

For n unit vectors in d-dimensional space with n>d, there are mathematical limits on how small all pairwise inner products can be. The Welch bound provides one such limit on average squared coherence. The exact formula depends on the setup, but the lesson is simple: finite dimension imposes unavoidable overlap.

Well-spread frames can approach this bound under special conditions. Toy superposition geometries sometimes resemble optimal spherical codes or tight frames because those arrangements distribute unavoidable interference evenly.

Real networks need not optimise coherence alone. Feature probabilities, importance and nonlinear behaviour break the symmetry assumed by uniform coding problems.

K. Random high-dimensional directions are already nearly orthogonal

For random unit vectors in high dimension, dot products concentrate near zero. This means a network can place many feature directions with modest pairwise interference before needing carefully engineered geometry.

The phenomenon helps explain why large residual streams can support enormous feature repertoires even when the number of possible semantic features exceeds dimension. Exact orthogonality is unnecessary; low enough coherence under sparse co-activation can suffice.

Do not confuse “nearly orthogonal pairwise” with “independent feature recovery”. Many small cross-terms can accumulate when several features activate simultaneously.

L. The Johnson–Lindenstrauss analogy explains distance preservation, not learned semantics

The Johnson–Lindenstrauss lemma shows that a finite set of points can often be projected into surprisingly low dimension while approximately preserving pairwise Euclidean distances. Random projection works because high-dimensional geometry has redundancy for finite datasets.

This is a useful analogy for why dimensionality does not bound the number of distinguishable states as tightly as naïve axis counting suggests. But JL does not say a neural network can recover arbitrarily many semantic features from arbitrary sparse mixtures. It preserves point distances under a random projection subject to a distortion bound.

Use the analogy for geometric intuition, not as proof of neural feature identifiability.

M. Compressed sensing adds sparsity-based recovery conditions

Compressed sensing goes closer to the superposition problem. A k-sparse high-dimensional signal can sometimes be recovered from far fewer linear measurements than ambient dimension when the measurement matrix satisfies properties such as restricted isometry or low coherence.

This suggests a theoretical route from superposition to sparse recovery: if neural features are sufficiently sparse and their directions sufficiently incoherent, an overcomplete sparse code can be recoverable. The 2026 Nature Machine Intelligence perspective A unifying framework from neural superposition to sparse interpretable codes explicitly connects superposition, identifiability and compressed-sensing ideas.

The boundary remains important. Neural activations are generated by nonlinear learned computations, feature dictionaries are unknown and sparsity can be structured. Classical recovery guarantees do not transfer automatically.

N. Restricted isometry is an idealised property worth translating into empirical tests

A restricted-isometry condition says that sparse vectors should approximately preserve norm after projection. In neural terms, distinct sparse feature mixtures should remain distinguishable after being packed into the activation space.

We rarely know the true sparse feature basis, so exact RIP testing is unavailable. But a proposed dictionary can be stress-tested: sample sparse combinations of learned features, encode or reconstruct them, and measure whether distances and supports remain distinguishable.

This converts a theoretical analogy into an empirical falsification test without pretending the theorem’s assumptions hold exactly.

O. Feature co-occurrence creates structured, not isotropic, interference

If features come in modules or semantic families, the covariance of x is not diagonal. Expected hidden covariance is W Cov(x) Wᵀ. The network can orient features to exploit correlations, anticorrelations and conditional exclusivity.

This means one global sparsity percentage is insufficient. Two models with the same marginal activation rate can have very different collision loads because one contains strongly co-occurring feature bundles.

A serious capacity study should estimate the feature co-occurrence matrix or higher-order statistics of the recovered dictionary.

P. Mutually exclusive features can share more aggressively than independent features

If two features never activate together, their directions can overlap substantially because the network never needs to represent their sum. Their distinction only has to survive under their separate contexts.

This is common in categorical alternatives: singular versus plural morphology, mutually exclusive token types, or states of one finite-state variable. The code can reuse representational resources across incompatible states.

When an interpretability tool finds two meanings in one neuron, check whether those meanings are contextually exclusive. Polysemanticity can be computationally efficient rather than arbitrary.

Q. Co-active features need cleaner separation

Features that often appear together must survive additive mixing. If their directions are highly correlated, downstream readout cannot easily tell whether one, the other or both are active. Optimisation therefore has reason to keep commonly co-active features more distinct.

This suggests a falsifiable prediction: recovered feature pairs with high empirical co-activation should on average have lower dangerous overlap than equally important mutually exclusive pairs, after controlling for other factors. Real networks can be tested for such structure.

R. Feature capacity can be measured by expected reconstruction or behavioural cost

There is no universal scalar called “number of features a dimension can hold”. Capacity depends on the tolerated error. One operational definition fixes a sparse feature distribution, a decoder and an acceptable reconstruction error, then asks how many features can be supported.

A stronger definition uses behavioural cost: pack features until interference changes the model’s output beyond a specified tolerance. This includes nonlinear cleanup and downstream redundancy rather than treating hidden reconstruction as the final goal.

Capacity should therefore be reported relative to an error criterion and a feature distribution.

S. Effective dimension and participation ratio describe how activation variance occupies space

Given covariance eigenvalues λ_i, one effective-dimension measure is the participation ratio (Σλ_i)^2 / Σλ_i². It is high when variance is spread across many directions and low when a few dominate.

This statistic does not count semantic features. A layer can have low effective dimension and many sparse superposed features whose activations are rare. Conversely, high variance dimension can reflect nuisance variation.

Use effective dimension as a geometric descriptor alongside feature-level analysis, not as a substitute for it.

T. Feature dimension and activation dimension can change in opposite directions

A model layer can have fixed residual width while the number of semantic distinctions it represents grows during training. Superposition predicts that feature count can increase without proportional activation dimension growth by packing more directions and tolerating more structured overlap.

This is one reason training dynamics matter. Measure proposed feature dictionaries across checkpoints. Does expansion factor needed for an SAE rise? Do neuron interpretations become more polysemantic? Does co-activation sparsity change?

The answers can reveal whether representation complexity is increasing through new dimensions, new features in the same dimensions or both.

U. Width sweeps can separate interference pressure from feature discovery

Train otherwise matched models with different hidden widths. If the task and data encourage roughly the same feature repertoire, wider models should be able to reduce feature overlap. If wider models instead learn more rare or nuanced features, superposition pressure can persist.

Compare task performance, recovered feature count, sparsity, feature coherence and polysemanticity across widths. The result can distinguish “more room for the same ideas” from “more room creates more ideas”.

This is a much stronger test than simply correlating model size with neuron interpretability.

V. Depth can move features between spaces rather than only add them

Features are not static objects painted onto one residual stream. Early layers can represent lexical or local patterns. Middle layers combine them into syntactic and semantic state. Later layers prepare output-relevant computations.

A feature can emerge, split, merge or disappear across layers. Superposition pressure can differ because each layer serves a different computation and has different activation statistics.

Layer-wise feature matching should therefore allow transformations in semantics rather than assuming feature identity is constant from input to output.

W. Residual addition encourages linear feature communication and creates collision risk

Transformer components add vectors into a shared residual stream. Addition is convenient because later components can read linear combinations of earlier writes. It also means simultaneously written features occupy one common basis and can interfere.

Layer normalisation, attention routing and MLP nonlinearities modulate this simple picture, but the residual stream’s additive structure makes linear feature directions a natural interpretability hypothesis.

This is one reason sparse autoencoders are often trained on residual-stream activations: they attempt to recover an overcomplete sparse dictionary whose linear combination reconstructs this communication state.

X. Attention heads can read superposed features without owning them

An attention head’s query, key or value projections can be sensitive to directions in residual space. The semantic feature is not necessarily “inside the head”; the head can read a feature stored in a shared stream, route information based on it and write another feature.

This matters for mechanistic claims. A head that activates in contexts containing a concept may be a consumer, producer or transporter of the feature rather than its unique storage location.

Causal tracing across read and write operations is needed to identify the computation.

Y. MLP neurons can be polysemantic even when MLP feature directions are cleaner

An MLP expands residual state into a larger intermediate dimension, applies a nonlinearity, then projects back. Individual intermediate neurons sometimes have interpretable activation patterns, but the semantic basis can still be distributed across many units.

The expansion layer provides more dimensions, potentially reducing superposition temporarily. The projection back into the residual stream compresses that information again. Interpretability can differ dramatically between the MLP hidden space and residual stream.

Study both spaces rather than treating “the model’s features” as one homogeneous dictionary.

Z. Memorisation competes for representational resources

Memorised exceptions are effectively rare features or patterns. When capacity is limited, representing them can interfere with broader reusable structure. As model capacity crosses interpolation regimes, the allocation between general features and memorised examples can change.

Anthropic’s superposition and double-descent work uses toy settings to explore how models can transition among ignoring rare features, representing them in superposition and memorising individual examples.

This provides a representation-level perspective on double descent: capacity changes not only test error but the internal allocation strategy.

AA. Feature phase diagrams are more informative than one trained model

Vary sparsity, importance, hidden dimension and correlation systematically in a toy model. For each setting, record which features are represented, their norms, pairwise angles and reconstruction error. Plot the result as a phase diagram.

Regions can emerge where features are monosemantic, superposed, antipodal, partially represented or dropped. Boundaries between regions reveal which data statistics trigger representational reorganisation.

This approach is more scientifically useful than one attractive geometry snapshot because it explains when and why the geometry changes.

AB. Worked laboratory: five features in two dimensions

Construct five unit feature vectors in a plane. Begin with equally spaced directions. Generate sparse non-negative feature activations with probability p. Encode by summing active feature vectors and reconstruct with dot products plus ReLU and a tunable threshold.

Sweep p from very sparse to dense. At low p, reconstruction can remain surprisingly good because one feature usually appears alone. As p rises, cross-talk grows rapidly. Plot error against p and compare with an orthogonal two-feature baseline.

The experiment demonstrates the exact condition superposition exploits: feature repertoire can exceed dimension only because simultaneous demand is sparse.

AC. Worked laboratory: importance-weighted capacity allocation

Use the same five-feature model but assign one feature ten times the loss weight of the others. Optimise feature directions and norms. Observe whether the high-importance feature receives a cleaner direction or larger norm while low-importance features absorb more interference.

Then make one low-importance feature common and one high-importance feature rare. The geometry now balances probability against consequence. This is a compact model of how training objectives allocate representational fidelity.

AD. Worked laboratory: correlated versus mutually exclusive features

Create two pairs of features. Pair A is mutually exclusive: only one member can be active. Pair B is positively correlated and often co-active. Give all features the same marginal activation frequency and importance.

Optimise a low-dimensional code. The mutually exclusive pair can tolerate more overlap because the network never represents their sum. The co-active pair benefits from cleaner separation. This isolates why co-occurrence statistics matter beyond sparsity.

AE. Worked laboratory: threshold cleanup and false activation

Fix a superposed dictionary and vary decoder bias. For each threshold, measure true-positive feature recovery, false-positive activations and squared reconstruction error. Plot precision-recall curves separately for rare and common features.

The model can prefer different thresholds depending on feature importance. A single global ReLU bias is therefore a compromise across features with different interference distributions.

AF. Worked laboratory: width sweep with fixed feature distribution

Hold the synthetic feature repertoire and data distribution fixed while increasing hidden dimension. Track minimum achievable loss, feature coherence and number of near-dedicated directions. This shows how extra width buys lower interference when the feature set itself does not grow.

Then run a second experiment in which feature repertoire expands with width. Comparing the two curves demonstrates why real-model scaling cannot be interpreted from width alone.

AG. The capacity evidence card

• Activation dimension: size of the native space.
• Feature ontology: how candidate features are defined or recovered.
• Feature count: estimate and uncertainty.
• Sparsity: marginal activation frequency and magnitude.
• Co-occurrence: joint activation structure.
• Importance: behavioural or loss sensitivity.
• Feature norms: allocation of signal scale.
• Coherence: pairwise feature overlap.
• Nonlinearity: cleanup mechanism and thresholds.
• Interference: reconstruction and behavioural cost.
• Width: capacity available to reduce overlap.
• Layer: residual, attention, MLP or other space.
• Training stage: checkpoint dependence.
• Source evidence: synthetic ground truth or recovered-feature method.

The first advanced layer leaves us with a precise picture: superposition is an optimisation response to finite dimensionality, sparse demand and unequal feature importance. It is not synonymous with opacity. Its opacity comes from using a basis optimised for computational capacity rather than human semantics. The next layer asks the harder question: when we propose a semantic feature basis after the fact, what evidence would justify believing it?

Advanced Technical Expansion II | Feature Recovery, Interpretability Evidence and the Limits of a “True” Basis

The mathematical case for superposition is only half of the story. In a toy model, the features are known before training, so we can measure exactly how they are packed. In a real language model, the features are the unknown. Any claim that the network represents “irony”, “Python list comprehensions”, “legal disclaimers” or “deception-related planning” depends on an analysis method that proposes a basis and an evaluation method that decides whether the proposal deserves a semantic name. This expansion is therefore about evidence: how to move from dense activations to candidate features without quietly assuming the answer.

AH. A feature is an explanatory hypothesis before it is an ontology item

Suppose a direction activates strongly on references to court rulings. It is tempting to call it “legal precedent”. But high-activation examples may share other properties: formal prose, citations, case names or quotation structure. The direction might detect one of those, a mixture, or a broader abstraction.

The correct first label is provisional: “direction associated with these activation contexts”. Semantic naming is an inference step. Stronger evidence comes from contrastive examples, causal interventions, distribution shifts and predictions on examples chosen before inspecting the activation.

This discipline protects interpretability from storytelling. A feature becomes a scientific object when its interpretation generates tests that can fail.

AI. Activation maximisation reveals exemplars and can hide conditional meaning

A common interpretation method finds the tokens or examples that produce the largest activation for a neuron or candidate feature. This is useful for discovering a theme. It overrepresents extreme cases and can miss the conditions under which moderate activation matters.

A relation feature may activate only when two entities stand in a particular configuration, yet the top examples can look like a collection of entity names. A negative-valued or inhibitory role may be invisible if the analysis only inspects positive maxima.

Inspect activation distributions, negative examples, near-threshold examples and matched controls. Feature meaning is a decision boundary, not merely a gallery.

AJ. Contrast sets make feature descriptions falsifiable

For a proposed feature “French language”, do not only show French text. Include Italian, Spanish, French names inside English sentences, code comments in French and transliterated phrases. Predict in advance which should activate.

A feature interpretation improves when it explains both positives and negatives. Contrast sets reveal whether the feature tracks the intended concept or a correlated surface cue.

The best labels often become more specific after contrast testing: not “French”, but perhaps “French function words”, “French prose register” or “language-switch into French”. Narrow labels are stronger labels when the evidence is narrow.

AK. Linear probes test accessibility and can manufacture a basis after the fact

A linear probe can recover a labelled variable from a distributed representation even when no individual feature direction corresponds to that variable. The probe learns a new readout direction.

Therefore “a linear probe predicts sentiment” proves that sentiment information is linearly accessible, not that the model already contains one sentiment feature. A candidate feature basis requires stronger structural evidence.

Use probes to map information content, then distinguish probe directions from native computational directions through causal tests and alignment with model weights.

AL. Causal intervention separates readable features from used features

If a candidate direction genuinely participates in computation, changing activation along it should alter downstream behaviour under suitable contexts. Activation addition, ablation, patching or feature clamping provide interventions.

But interventions can leave the natural activation manifold. Adding a large vector can create states the model never encounters. Behaviour then demonstrates a network response to an artificial state, not necessarily the ordinary role of the feature.

Use dose–response curves, small perturbations, matched random directions and naturally observed activation ranges. Causal evidence is strongest when intervention magnitude stays inside the model’s plausible state distribution.

AM. Necessity and sufficiency are separate

A feature direction is sufficient for a behaviour if activating it can induce the behaviour. It is necessary if removing it reliably prevents the behaviour. Neural networks can contain redundant routes, so sufficiency does not imply necessity.

Ablating one representation direction may have little effect because another layer or correlated feature carries the same information. Conversely, strongly injecting one direction can trigger output even if the model usually reaches that output through another route.

Interpretability reports should state which causal claim was tested: association, sufficiency, necessity, mediation or contribution.

AN. Mediation asks whether a feature carries influence between two parts of the computation

Suppose changing an input fact changes the final answer. To test whether feature f mediates that influence, intervene on the upstream state and then patch or hold f constant. If the downstream effect disappears, f may carry part of the causal pathway.

This is stronger than showing f correlates with the input and output. Mediation analysis in neural networks is complicated by distributed redundancy and nonlinear interaction, but the causal framing clarifies the question.

A superposed representation can contain many mediators in overlapping directions. Sparse feature decompositions aim to make those pathways easier to separate.

AO. Dictionary learning proposes an overcomplete semantic coordinate system

Sparse autoencoders and classical dictionary learning assume dense activations can be reconstructed from a larger collection of feature vectors with only a few active per example. The decoder columns are candidate feature directions; sparse coefficients describe which features are present.

This is well matched to the superposition hypothesis. If the native representation is an overlaid mixture of sparse features, an overcomplete sparse dictionary may undo part of the packing.

The fit does not prove uniqueness. Several dictionaries can reconstruct activations with similar sparsity and error. The next owner, Sparse Autoencoders for Neural Representation, examines these ambiguities in depth.

AP. Identifiability depends on assumptions about the source features

Sparse coding theory provides conditions under which a true dictionary can be recovered up to permutations and scaling. These conditions can involve sparsity, incoherence, sample diversity and generative assumptions.

A trained neural network is not guaranteed to satisfy a textbook sparse generative model. Features can be nonlinear, hierarchical, context-dependent and correlated. The Nature Machine Intelligence 2026 perspective on neural superposition and sparse interpretable codes is valuable precisely because it makes identifiability assumptions part of the conversation rather than treating sparse coding as automatic ground truth.

When claiming recovered features, state the assumptions that make recovery plausible and the equivalence class under which two dictionaries count as the same.

AQ. Feature splitting is not always a mistake

One conceptual feature can split into several learned dictionary atoms. A “Python code” concept might divide into indentation, list comprehensions, imports and exception handling. This can represent genuine semantic substructure.

Splitting becomes problematic when the intended receiver wants one stable control and the decomposition changes unpredictably across training seeds or contexts. It can be useful when finer components support distinct circuits.

Evaluate hierarchical relationships among features rather than assuming one human noun should correspond to one atom.

AR. Feature merging can hide separate mechanisms

An SAE or neuron can respond to two concepts because they co-occur, share downstream use or happen to fit one dictionary direction under the sparsity–reconstruction trade-off. A broad feature may therefore merge several mechanisms.

Use contrast sets and higher expansion factors to test whether the merged feature resolves into subfeatures. If splitting improves reconstruction but produces unstable or uninterpretable atoms, the broader feature may be the more useful receiver unit.

There is no universal optimal granularity independent of the question being asked.

AS. Feature granularity is a resolution choice

“Animal”, “dog”, “golden retriever” and “golden retriever with wet fur” can all be legitimate features at different resolutions. A sparse dictionary with enormous expansion can split semantic space finely; a smaller dictionary produces broader atoms.

Interpretability metrics should account for this hierarchy. A broad feature is not less interpretable merely because a larger SAE can divide it. A fine feature is not better if its meaning is unstable and rarely causally relevant.

The correct resolution follows the receiver: explanation, monitoring, steering, circuit analysis or scientific discovery.

AT. Feature consistency across random seeds tests whether the dictionary is canonical

Train several decomposition models on the same activations with different initialisations. Match learned feature directions by cosine similarity, activation correlation or semantic agreement. Some features may recur reliably; others may split, merge or rotate.

Stable recurrence is evidence that the activation distribution supports that feature strongly. Instability suggests multiple nearly equivalent decompositions or insufficient data. The latter is not necessarily a failure if all dictionaries support the receiver equally well.

Report match rates and unmatched mass rather than showing one dictionary as if unique.

AU. Stability across data slices tests semantic scope

A feature learned on web prose may split differently on code, mathematics or multilingual text. Train or evaluate dictionaries on several data slices. Does the feature activate with the same meaning? Does its direction remain similar?

A feature stable only on one domain should be labelled with that scope. Domain-specific features can be valuable; overgeneralising their meaning is the problem.

This also tests whether superposition is data-distribution dependent. Change the distribution and the economically optimal feature packing can change.

AV. Stability across model checkpoints tests feature persistence through learning

Learn or match feature dictionaries across checkpoints. Some low-level features may appear early and persist. Others emerge only after capabilities develop. Some disappear as the model reorganises its representation.

A stable semantic label across checkpoints does not require an identical coordinate direction; the basis itself can rotate. Representation-similarity methods and matched activation patterns can help track evolving features.

Feature emergence trajectories can link interpretability to capability development rather than treating the final model as a static object.

AW. Stability across model versions matters for interpretability tooling

A production monitoring system might depend on feature 4312 as a “prompt injection” detector. After model retraining, feature numbering and geometry can change completely. Reusing the threshold silently is unsafe.

Version feature dictionaries with the model. Re-match semantic features, recalibrate thresholds and re-run causal tests. Interpretability features are interfaces with compatibility requirements.

A feature label should carry model version, layer, dictionary version and training distribution as provenance.

AX. Feature activation frequency has a long tail

Large language models can plausibly contain many rare features. A dictionary trained on finite tokens may never observe enough examples to interpret or even learn them reliably. Common features dominate reconstruction loss because they contribute more often.

Rare-feature coverage should be estimated explicitly. Plot activation frequency distributions, dead-feature rates and data requirements. Targeted sampling can enrich rare domains without pretending the resulting frequency matches deployment.

The unseen tail places a hard limit on claims such as “we decomposed the model into all features”.

AY. Feature absence can be as informative as feature activation

Some computations depend on the absence of an expected signal. A sparse feature ontology focused only on positive activations can miss inhibitory, subtractive or comparison-based mechanisms.

Signed features, paired positive/negative features or downstream interaction analysis may be needed. ReLU-based SAEs naturally favour non-negative coefficients and can represent negative directions through separate decoder atoms, but the semantic interpretation should acknowledge this construction.

“Feature not active” is not automatically equivalent to “concept absent”.

AZ. Nonlinear features can evade a linear dictionary

The superposition hypothesis often assumes features are approximately linear directions in activation space. Some useful states may be curved, multiplicative or conditional combinations that no single direction captures.

An SAE can approximate nonlinear structure with several atoms, potentially producing feature splitting that reflects method limitation rather than true semantic multiplicity. Nonlinear probes may recover information absent from any one linear direction.

Test the linear-feature assumption rather than defining feature to mean “whatever a linear SAE can discover”.

BA. Context-dependent features challenge one global dictionary

A direction can mean one thing in mathematical text and another in dialogue because the surrounding activation subspace changes. A single global linear dictionary assumes feature directions are sufficiently stable across contexts.

Conditional dictionaries, mixture dictionaries or hierarchical feature models can represent context-specific bases. The price is increased complexity and a harder matching problem.

Before expanding architecture, test whether apparent context dependence persists after controlling for co-active features. Superposition itself can make a stable feature look context-dependent through interference.

BB. Feature circuits ask how sparse components interact, not only what they mean alone

A feature dictionary becomes mechanistically useful when we can trace transformations among features. One feature may trigger attention to a location; another may represent retrieved information; a later feature may encode the decision. The computation is a circuit through sparse states.

Build graphs whose nodes are candidate features and edges represent causal influence or weight-mediated communication. Validate edges by patching or ablation. Co-activation alone is not enough because correlated features can rise together without direct interaction.

Mechanistic interpretability is therefore not solved by naming a large dictionary. It requires understanding how features compose into algorithms.

BC. Sparse feature graphs can still be enormous

Even if one activation contains only tens of active features, a model can have millions of possible features across layers. Circuit analysis must narrow the graph to the receiver job.

Use causal relevance, activation timing, attention paths and output sensitivity to prune. A complete global circuit map may be computationally infeasible and cognitively useless.

Interpretability itself needs a routing system: find the features relevant to this behaviour, not every feature the model can express.

BD. Automated feature descriptions are hypotheses generated by another model

Large language models can inspect top-activating examples and generate natural-language feature descriptions. This scales interpretation and introduces another model into the evidence chain.

Evaluate descriptions on held-out examples. Ask the description to predict activation. Compare with human or programmatic labels. An eloquent description can overfit the examples just as a human can.

Store the description generator, prompt, examples and evaluation result as provenance. Automated interpretability should not be confused with automatic truth.

BE. Simulation scores test whether a description predicts the feature

One evaluation asks an interpreter model to predict feature activation from the natural-language description and new text. If predictions correlate with actual activation, the description captures some functional behaviour.

Simulation can reward descriptions that predict surface correlations without capturing causal role. Combine it with intervention and contrast tests. Different descriptions can achieve similar predictive scores at different semantic granularity.

A good description is useful because it compresses feature behaviour for a human receiver, not because one score certifies its ontology.

BF. Feature steering tests controllability and can expose collateral effects

Add a candidate feature direction to residual activations and observe output changes. If a “formal tone” feature reliably increases formality, the feature supports a control interpretation.

Measure collateral changes: factuality, verbosity, refusal rate, language, topic and unrelated behaviours. Superposition means one direction can overlap with others. Steering can therefore move several semantic variables.

Useful steering requires target efficacy plus low unintended change. It is an intervention test and a safety test.

BG. Feature ablation can create off-manifold states too

Setting one feature coefficient to zero after sparse decomposition may create a reconstructed activation that the native model never produced naturally. The model’s response can reflect the artificial combination as well as removal of the feature.

Use reconstruction-aware controls: compare ablation through the SAE with matched reconstruction error, random feature ablations and activation patching from natural examples. The causal claim should exceed the artefact introduced by the intervention tool.

Interpretability intervention has its own representation error budget.

BH. “Loss recovered” is an important but incomplete interpretability metric

If replacing a native activation with its SAE reconstruction increases language-model loss, reconstruction has removed behaviourally relevant information. Loss recovered measures how much of the native performance drop between original and some ablated baseline the SAE reconstruction restores.

A dictionary can recover most model loss and still have poor semantic feature quality. Another can yield beautiful features while losing important computation. Interpretability quality and reconstruction fidelity are separate axes.

This metric belongs more fully in the SAE owner but is relevant here because it measures whether a proposed sparse basis still carries the superposed computation.

BI. Feature count can be inflated by redundant dictionary atoms

An overcomplete dictionary can contain several near-duplicate atoms. Counting atoms then exaggerates distinct semantic feature count. Conversely, one atom can merge several features and undercount.

Cluster decoder directions, compare activation correlation and test whether atoms substitute for one another. Feature-count estimates should collapse obvious duplicates or report a resolution-dependent count.

“Millions of learned SAE features” is a statement about dictionary width, not necessarily millions of independent concepts.

BJ. Dead features reveal optimisation and coverage limits

A dictionary atom that never activates is dead. Dead features waste capacity and can indicate overly strong sparsity, poor initialisation, insufficient data coverage or competition where a few features dominate.

Measure dead-feature rate and revival procedures. A low dead rate is not enough; features can activate extremely rarely or redundantly. Frequency spectrum matters.

In superposition terms, a dead learned atom tells us nothing about whether the native model lacks the feature. It may mean the decomposition failed to allocate an atom to it.

BK. Rare features need targeted evidence without corrupting prevalence estimates

If a feature fires once per million tokens, ordinary random sampling provides too few examples for interpretation. Search or synthetic prompts can enrich the feature and reveal meaning.

Targeted data change the sampling distribution, so activation frequency observed in the enriched set cannot be treated as prevalence. Keep discovery and prevalence estimation separate.

Rare safety-relevant features may justify expensive targeted evaluation even when they contribute little to average reconstruction loss.

BL. Feature names can become reified and outlive the evidence

Once an interpretability dashboard labels a feature “deception”, later users can forget that the label came from a small example set and uncertain causal tests. The name becomes more confident than the underlying evidence.

Use scoped names: “activates on examples containing planning to conceal information” can be more accurate than “deception neuron”. Include confidence, counterexamples and version information.

Interpretability interfaces should preserve uncertainty rather than turn hypotheses into ontology through typography.

BM. Human concepts are not guaranteed to be the model’s natural factorisation

A model may use distinctions humans do not have convenient names for: mixtures of syntax, position, task state and statistical regularity. Forcing every feature into one noun can make the representation appear messy when the underlying computation is efficient.

Conversely, human concepts can be useful interfaces even if the native code is distributed. Interpretability should distinguish native computational features from human-facing explanatory concepts.

The best analysis may map between the two rather than demanding they coincide perfectly.

BN. Feature hierarchies can connect broad concepts to specific implementations

A broad “quotation” feature can contain subfeatures for dialogue quotes, legal citations, code strings and scare quotes. Hierarchical dictionaries or clustering can represent this nested structure.

Circuits can operate at multiple levels: one high-level state may route behaviour while low-level subfeatures supply evidence. Interpretability tools should permit movement up and down the hierarchy.

This reduces conflict between feature splitting and semantic usefulness. Several atoms can be valid children of one human concept.

BO. Features can be compositional rather than independent

A model can represent “red”, “car” and “moving left” separately and combine them. It can also contain a conjunction feature “red car” or “car moving left”. Sparse dictionaries often recover both broad components and conjunctions.

This is not necessarily redundancy. Conjunction features can make downstream nonlinear computation cheaper. The model’s representational vocabulary may contain reusable primitives and cached compositions.

Feature ontologies should therefore allow overlap and compositional relations rather than enforcing one mutually exclusive partition.

BP. Superposition can hide conjunctions especially well

If a conjunction feature only activates when several base features co-occur, it is extremely sparse. The network can pack it aggressively because collisions are rare. This makes rare high-level states plausible candidates for heavy superposition.

Safety-relevant planning or deception-related states can be rare and context-dependent, precisely the regime where average activation analysis is weak. Targeted causal testing becomes important.

This remains a hypothesis about capacity allocation, not evidence that any particular safety concept has one hidden feature.

BQ. Cross-layer redundancy complicates necessity tests

A concept can be represented at several layers, perhaps in progressively transformed forms. Ablating one layer’s feature may be compensated downstream or reconstructed from context.

Multi-layer interventions can test redundancy but increase off-manifold risk. Cross-layer transcoders and feature-transition models attempt to map how information is rewritten through the network.

The mechanistic unit may therefore be a trajectory of feature states rather than one static direction.

BR. Cross-model feature matching tests whether different networks discover analogous structure

Train models with different seeds or architectures on similar data. Recover candidate feature dictionaries and match them through activation patterns, decoder directions after alignment, or semantic labels. Recurrent feature families suggest the task/data distribution favours those distinctions.

Failure to match does not prove one model lacks the concept. The representation can be rotated, split or implemented by a different circuit. Representation Similarity Analysis, the fourth article in this batch, provides tools for comparing spaces without requiring neuron-to-neuron identity.

Feature matching and whole-space similarity answer complementary questions.

BS. A universal feature dictionary may not exist

The notion of “the model’s true features” is attractive because it suggests one final ontology. But several sparse dictionaries can be equally predictive and reconstructive, especially when features are correlated or hierarchical.

There may be a family of useful coordinate systems rather than one privileged basis. Identifiability theory tells us when uniqueness can be expected under assumptions; empirical stability tells us how strongly real data constrain the solution.

The strongest public claim is often “this dictionary provides an interpretable sparse decomposition with these measured properties”, not “we found the one true feature basis”.

BT. The 2026 unifying perspective makes the assumptions explicit

The July 2026 Nature Machine Intelligence perspective A unifying framework from neural superposition to sparse interpretable codes connects three layers: identifiability of latent features up to mixing, sparse recovery under compressed-sensing-like assumptions, and behavioural interpretability metrics that test whether extracted features correspond to meaningful concepts.

This is useful because it refuses to let one layer stand in for the others. Theoretical identifiability does not automatically provide a practical dictionary. Sparse recovery does not automatically provide human semantics. Human-readable descriptions do not automatically prove behavioural relevance.

A mature superposition programme needs all three: a reason features are recoverable, a method that recovers candidate structure, and tests that show the recovered structure matters.

BU. Superposition claims should be separated into levels

• Level 1 — Geometry: activations show many non-orthogonal useful directions.
• Level 2 — Sparse generative account: activations can be reconstructed well from a sparse overcomplete dictionary.
• Level 3 — Stable features: candidate atoms recur across seeds, data and methods.
• Level 4 — Semantic features: atoms support predictive human-readable descriptions.
• Level 5 — Mechanistic features: interventions show causal participation in model computation.
• Level 6 — Complete account: the recovered feature system explains the relevant behaviour with low residual uncertainty.

Most current interpretability results establish some subset of these levels. Writing them separately prevents a reconstruction result from being promoted directly into a complete mechanistic explanation.

BV. Worked laboratory: rotate a known feature basis and watch neurons become polysemantic

Create a synthetic activation with four independent features aligned to four coordinates. Every neuron is perfectly monosemantic by construction. Multiply activations by a random orthogonal matrix and adjust downstream linear weights by the inverse rotation so the network function is unchanged.

Now individual coordinates respond to mixtures of the original features. The function is identical; neuron interpretability has changed. A linear dictionary learner with the right assumptions can recover a basis aligned with the original features up to permutation/sign.

This experiment demonstrates why coordinate semantics are not functionally invariant and why an interpretability method needs an inductive bias to select a basis.

BW. Worked laboratory: two equally good dictionaries

Create correlated sparse sources where two features almost always co-occur. Fit overcomplete dictionaries from several seeds. You may find dictionaries that represent the pair separately and others that use a shared combined atom plus residual features with similar reconstruction error.

Compare semantic interpretability and downstream intervention. The experiment shows how data correlation weakens uniqueness even when sparse reconstruction is excellent.

Additional environments where the features separate can resolve the ambiguity. Data design changes identifiability.

BX. Worked laboratory: association versus causal use

Train a small network where one hidden direction correlates with the label because of a spurious feature, while another direction carries the causal signal. A probe may read both. Ablation or counterfactual data reveals which direction the network actually relies on.

This demonstrates why activation correlation and probe accuracy are insufficient for mechanistic interpretation. The model can contain information it does not use.

BY. Worked laboratory: feature splitting across dictionary width

Fit sparse dictionaries with expansion factors 2×, 4×, 8× and 16× to the same synthetic or model activations. Match atoms across widths. Broad atoms may split into narrower children as dictionary capacity grows.

Measure reconstruction, sparsity, semantic description consistency and causal efficacy. The result identifies which features are robust to resolution and which exist only at one dictionary granularity.

BZ. Worked laboratory: feature stability across distribution shift

Train a feature decomposition on general prose and evaluate activations on code, mathematics and another language. For matched candidate features, compare activation examples, frequency and causal steering.

A feature that remains semantically coherent across domains earns a broader label. A feature whose meaning changes should be scoped or split. This provides an operational definition of semantic stability.

CA. Interpretability safety: steering tools can be dual-use and misleading

Feature steering can help diagnose model behaviour and can also be used to manipulate capabilities or safeguards. Public interpretability work should distinguish conceptual mechanisms from operational instructions when an intervention could materially facilitate harmful misuse.

Even benign steering can create false confidence. A feature that suppresses one observed failure mode may produce another. Interpretability is evidence about internal state, not a replacement for system-level evaluation and access controls.

For educational treatment, focus on principles, measured effects and evaluation rather than deployable bypass recipes.

CB. The superposition interpretation evidence card

• Feature proposal: neuron, direction, SAE atom, probe direction or other construct.
• Layer and site: exact activation tensor and model version.
• Description: scoped semantic hypothesis.
• Positive evidence: top and representative activations.
• Negative evidence: contrast cases and non-activating examples.
• Predictive test: held-out activation prediction from the description.
• Causal test: ablation, addition, patching or mediation.
• Intervention range: natural versus extrapolated activation magnitude.
• Stability: across seeds, data slices, checkpoints and dictionaries.
• Granularity: broad feature, child feature or conjunction.
• Redundancy: alternative features supporting the same behaviour.
• Reconstruction fidelity: information retained by the proposed basis.
• Behavioural scope: tasks for which the feature matters.
• Uncertainty: competing interpretations and failure cases.

CC. Final release gate for a 20,000+ word superposition owner

1. The article distinguishes neurons, directions, features and dictionary atoms.
2. It explains why sparsity makes feature packing economically possible.
3. It derives interference through feature-vector overlap rather than relying on metaphor alone.
4. It includes co-occurrence and importance, not only marginal sparsity.
5. It explains the role of nonlinearity in suppressing interference.
6. It states mathematical analogies such as compressed sensing with their limits.
7. It connects toy models to real residual-stream questions without claiming proof.
8. It distinguishes superposition from collapse, entanglement, routing and sparse autoencoding.
9. It treats feature recovery as an inference problem.
10. It separates association, accessibility, sufficiency, necessity and mediation.
11. It discusses splitting, merging, granularity and non-uniqueness of dictionaries.
12. It requires stability across seeds, datasets and model versions for strong semantic claims.
13. It includes causal and behavioural evaluation.
14. It preserves uncertainty when no unique feature basis is identified.
15. It routes the reader to the dedicated sparse-autoencoder and representation-similarity owners rather than cannibalising them.

Superposition is ultimately a theory about how intelligence spends representational space. It predicts that a finite neural substrate can support a feature repertoire much larger than its coordinate count when activation is sparse and interference is cheaper than dedicated capacity. Interpretability then faces the inverse problem: recover useful semantic structure from that compressed code. The inverse problem can be tractable, but only under assumptions, data and evaluation strong enough to justify the recovered basis.

The important question is not whether one neuron has one meaning. It is whether the model’s overlapping feature geometry can be recovered well enough that a human can predict, test and intervene on the computation without mistaking a convenient dictionary for the world inside the network.