Advanced Technical Expansion I | Steering-Vector Estimation, Geometry and Context Dependence

The readable spine establishes representation steering as inference-time control. This first technical layer asks how a control direction is estimated and what geometry justifies using it. The central issue is often hidden by the simple equation h'=h+αv: one constant vector assumes the attribute corresponds to a sufficiently stable displacement across a region of representation space. The quality of the controller therefore depends on contrast design, covariance, local curvature, layer choice and the scale at which the concept is approximately linear.

A. Difference-in-means is an estimator with sampling uncertainty

Let positive activations have mean μ+ and negative activations μ-. The estimated steering direction is v_hat=μ_hat+−μ_hat-. Finite data introduce variance. Bootstrap the contrast examples, recompute v and measure the distribution of vector norms, angles and downstream effects.

A direction whose angle changes by forty degrees across resamples is weaker evidence than one stable within a few degrees. Behavioural stability can still exceed geometric stability if several nearby directions work similarly.

Report both vector stability and steering-effect stability.

B. Matched pairs reduce nuisance variance and expose treatment heterogeneity

For paired examples sharing content, define individual differences d_i=h_i^+−h_i^-. Their mean is the global direction; their covariance shows how much the local direction varies from pair to pair.

Large variance orthogonal to the mean indicates context dependence. Cluster the difference vectors. Several clusters can imply multiple local steering regimes rather than one concept axis.

C. Difference-vector PCA can reveal a concept subspace rather than one axis

Stack paired differences into a matrix and compute PCA or SVD. If one singular direction explains most variance, a one-vector controller is plausible. If several directions are required, the attribute may occupy a low-dimensional subspace.

Steering can then use a projection or context-conditioned combination within that subspace. A subspace preserves more variation and increases control complexity.

D. Covariance-weighted directions separate mean shift from background variance

If positive and negative activations share covariance Σ, Fisher’s linear discriminant direction is proportional to Σ^{-1}(μ+−μ-). Directions with large background variance are downweighted; stable discriminative directions are emphasised.

This can improve diagnostic separation and can produce poor steering if the downstream model responds to native mean displacement rather than whitened discrimination. Compare raw mean difference, covariance-normalised direction and probe normal empirically.

E. Whitening changes the control metric

Steering in whitened coordinates corresponds to a Mahalanobis-aware move in native space. A unit step along a low-variance direction becomes relatively large under whitening.

This can surface semantically important small-variance factors and can amplify brittle directions. The metric defining “small intervention” should be chosen relative to model behaviour, not only statistical convenience.

F. Probe normals solve a supervised separation problem

A linear probe direction w is selected to predict labels under a loss. It can exploit directions whose means differ little if covariance structure makes them predictive. It can combine several semantic cues into one decision axis.

Steering along w assumes moving in the decision-normal direction creates the semantic change. This is not guaranteed. Test the direction causally with dose–response and collateral metrics.

G. TCAV-style concept vectors measure sensitivity and are not automatically steering controls

Concept Activation Vectors are learned from concept examples and used with directional derivatives to measure model sensitivity to the concept direction. The method creates a human-defined concept subspace or vector and asks whether outputs depend on it.

Turning the same direction into an inference-time intervention adds a new claim: not only is the output sensitive to the concept direction, but deliberately moving along it changes behaviour specifically and predictably.

H. Decoder or unembedding directions provide native task-aligned controls

In language models, output-token logits are linear functions of late residual states under common architectures. Differences between unembedding vectors define native directions that promote one token relative to another.

These are guaranteed to affect immediate logits at the final layer and need not correspond to high-level concepts or stable earlier-layer controls. Steering earlier layers along an output direction can behave differently because intervening computation transforms it.

I. Causal patch directions can seed a steering hypothesis

If activation patching shows that clean–corrupt difference at one site restores a behaviour, the difference vector is a natural candidate steering direction. Average such causal differences across examples to estimate a robust control vector.

This has stronger causal motivation than an arbitrary probe direction and remains tied to the counterfactual used in patching.

J. Sparse-autoencoder features offer semantically interpretable steering handles

An SAE feature has a decoder direction and sparse coefficient. Increasing the coefficient and decoding the change yields a native activation intervention aligned with a learned feature.

This can improve semantic specificity and inherits SAE feature splitting, reconstruction error and non-uniqueness. Feature steering should be evaluated with the same rigor as raw vector steering.

K. Native feature directions and SAE decoder directions can disagree

Two SAE features can reconstruct similar native directions while activating in different contexts. Steering by decoder direction alone ignores encoder-side presence semantics.

A context-conditioned controller can activate an SAE feature only when the current state indicates its semantic preconditions are satisfied.

L. The linear-feature assumption should be tested with local perturbations

If a concept is represented linearly, small moves along v should change a concept score predictably over nearby states. Evaluate local directional derivatives and dose–response around many natural activations.

Large variation in derivative sign indicates the global direction crosses regions where its semantic meaning changes.

M. Concept geometry can curve through representation space

Imagine positive and negative states separated by a curved boundary. One global vector is tangent to the desired change only in a limited region. Far away, the same vector can cross the boundary obliquely or move in the wrong semantic direction.

This is the geometric motivation for context-aware steering fields: the control direction follows local representation structure instead of imposing one Euclidean translation everywhere.

N. A concept-score gradient defines the Euclidean steepest-ascent direction

For differentiable score s(h), the first-order change under perturbation δ is Δs≈∇s(h)·δ. Under a fixed Euclidean norm budget, choosing δ parallel to ∇s maximises first-order score increase.

This is a local optimisation result. It says nothing about preserving model quality or semantic specificity. The gradient can exploit adversarially sensitive directions.

O. Constrained local steering can preserve protected quantities to first order

If protected scores g1,…,gm should remain unchanged, project the target gradient into the null space of their gradients. The resulting direction increases the target while producing zero first-order change in protected scores under the local linear approximation.

Nonlinear effects and incomplete protected-score models limit the guarantee. This is a useful control design, not a universal safety proof.

P. Natural-gradient steering uses a state-dependent metric

If hidden-state coordinates have anisotropic sensitivity, Euclidean distance may not reflect meaningful intervention size. A metric matrix G(h) can define cost δᵀGδ. The steepest direction becomes proportional to G^{-1}∇s.

Choosing G from covariance, decoder sensitivity or a learned local metric creates different controllers. The metric encodes what variation should be considered cheap.

Q. Tangent-space steering tries to remain on a learned activation manifold

Estimate a local tangent subspace from nearby natural activations and project the steering direction into it. This can reduce moves into directions unsupported by data.

Local PCA is only an approximation and can remove rare but valid control directions. Evaluate whether tangent projection improves quality and target efficacy.

R. Spherical steering assumes norm should be preserved

If layer normalisation and empirical activation statistics make direction more semantically relevant than norm, rotating on a sphere can isolate angular change. The assumption should be tested: norm can carry information.

Compare additive and norm-preserving interventions at matched angular or behavioural effect. The better geometry is the one with lower collateral cost for the receiver.

S. Geodesic steering can be defined on other manifolds too

If a learned metric or manifold model is available, steer along geodesics or constrained paths under that geometry rather than straight Euclidean lines. This generalises the spherical idea.

Estimating the manifold reliably is difficult in the high-dimensional state spaces where steering is used. Geometry-aware control adds modelling assumptions that need validation.

T. Context dependence can be measured before building a vector field

Estimate local paired-difference vectors across prompts and compute their angular dispersion. Fit one global vector and measure per-example target effect. If effect sign frequently reverses or magnitude varies wildly, a static controller is poorly specified.

Context-aware complexity should be justified by measured heterogeneity, not added automatically.

U. Local vector clustering provides a simple context-aware controller

Cluster hidden states or paired-difference directions and learn one steering vector per cluster. At inference, select the vector associated with the nearest or most probable regime.

This is simpler than a fully continuous vector field and introduces routing errors near cluster boundaries.

V. A learned vector field is itself a policy model

When a network predicts v(h), steering is no longer one transparent vector. The controller has learned a nonlinear mapping from hidden state to intervention. It can overfit and encode unintended strategies.

Audit the controller with held-out contexts, smoothness tests, local direction visualisation and ablations. Control complexity reintroduces an interpretability problem.

W. Field smoothness is a useful robustness prior

Nearby hidden states should not receive wildly different steering directions unless the concept boundary genuinely changes sharply. Penalise or monitor local direction variation.

Excessive smoothness can blur real regime changes. As always, the prior trades flexibility against robustness.

X. Vector-field integrability is not required for control

A context-dependent steering field need not be the gradient of one global concept scalar. Non-conservative fields can still define local control. If the interpretation claims one global potential or concept score, integrability becomes relevant.

Test loop behaviour when a global semantic scalar is part of the claim.

Y. Multi-attribute control can be formulated as constrained optimisation

Rather than simply adding vectors, choose δ to increase target scores subject to limits on protected scores and intervention norm. In a local linear regime this becomes a small quadratic or linear optimisation problem.

This makes trade-offs explicit. It also depends on the quality of every score function used as a constraint.

Z. Commutativity of steering interventions should be tested

Apply steering A then B and compare with B then A. Pure addition at one fixed site commutes mathematically. In a nonlinear recurrent generation process, the order can change subsequent hidden states and outputs.

Order sensitivity is evidence of interaction between control objectives.

AA. Orthogonality does not guarantee behavioural independence

Two vectors can have zero dot product and still interact downstream through nonlinear computation. Conversely, non-orthogonal vectors can produce largely independent behaviours if the shared component is ignored by downstream readers.

Geometry is a diagnostic, not a behavioural guarantee.

AB. Steering direction can change across layers because the representation changes basis

A concept direction estimated at layer ℓ should not be copied numerically to layer ℓ+1 even when dimensionality matches. Intermediate components transform the representation.

Estimate layer-specific directions or learn an alignment map. Compare whether the concept becomes more linearly accessible or more causally steerable through depth.

AC. Cross-layer steering can follow a concept trajectory

Match concept directions across layers using paired activations, CCA/Procrustes alignment or semantic feature matching. Apply small interventions along the trajectory rather than one fixed vector.

This resembles following a field through depth and can reduce mismatch between layer-specific bases.

AD. Steering-vector stability should include effect stability, not only cosine

Two estimated vectors can have moderate cosine similarity and nearly identical behavioural effects because the downstream network is insensitive to their difference. Conversely, near-identical vectors can have different effects at a nonlinear boundary.

Always compare geometry with intervention outcome.

AE. Worked laboratory: probe normal versus mean-difference vector

Generate two Gaussian classes with shared anisotropic covariance. Compute raw mean difference and Fisher/probe direction. Evaluate classification and steering through a downstream nonlinear function designed to respond to the generative displacement.

The classifier direction can separate classes better while the mean displacement can steer the intended generative factor better. This demonstrates diagnosis/control divergence.

AF. Worked laboratory: curved concept boundary

Create points on a curved manifold where the target attribute corresponds to moving radially outward. A global horizontal vector helps one region and harms another. A local gradient field follows the correct radial direction everywhere.

Compare efficacy and intervention norm. The lab makes context dependence geometrically visible.

AG. Worked laboratory: additive versus spherical steering

Use hidden states with approximately fixed norm and a target score dependent on angle. Compare Euclidean addition and same-angle geodesic rotation. Measure target score, norm distortion and downstream function.

The experiment shows what norm-preserving steering buys when the task is truly angular.

AH. The steering-geometry evidence card

• Concept contrast: positive/negative or paired design.
• Estimator: mean difference, probe, SAE feature, gradient or patch-derived vector.
• Uncertainty: bootstrap vector and effect stability.
• Dimension: one vector, subspace or field.
• Metric: Euclidean, covariance-weighted, spherical or manifold-aware.
• Layer: intervention site and normalisation context.
• Scale: α relative to natural state variation.
• Context dependence: sign/magnitude heterogeneity across inputs.
• Interactions: multi-attribute geometry and order effects.
• Off-manifold risk: natural-neighbour or reconstruction diagnostics.

The first advanced layer therefore replaces “one concept, one vector” with a more careful hierarchy: some attributes genuinely support stable global directions; some need a subspace; some need context-dependent local directions; some are better treated as nonlinear controls. The correct representation is the simplest controller whose effect remains stable under the contexts, lengths and combinations the receiver actually needs.

Advanced Technical Expansion II | Control Evaluation, Long-Form Stability, Multi-Attribute Interference and Safe Deployment

A steering method becomes useful only after it is evaluated as a controller rather than as an interpretability demonstration. This second technical layer therefore focuses on reliability: target efficacy, collateral effects, long-form drift, context-dependent failures, calibration, multi-attribute interference, reversibility, distribution shift and safe operating boundaries. A steering vector is not judged by whether one cherry-picked completion changes. It is judged by whether the intervention behaves predictably across the operating region while preserving the rest of the model’s competence.

AI. Define efficacy with an independent target measure

Do not evaluate steering solely with the same classifier or concept score used to construct the direction. If a probe defines v and the same probe measures success, the intervention can exploit probe-specific geometry without changing the intended human concept.

Use independent labels, human evaluation, task outcomes or a separately trained evaluator. Agreement among several target measures strengthens the result.

AJ. Specificity requires named protected behaviours

A statement such as “minimal side effects” is empty until the protected set is defined. For style steering, protect factual accuracy, task completion and language. For code preference steering, protect test pass rate, security checks and syntactic validity.

The protected set should follow the deployment receiver, not whichever metrics are convenient to compute.

AK. Plot a target–collateral Pareto frontier over steering strength

For each α, measure target change and a composite or vector of collateral costs. One method can dominate another by achieving equal target effect with lower damage.

There may be no single optimal α. The frontier lets the user choose an operating point according to acceptable trade-off.

AL. Quality should be measured independently of semantic specificity

An intervention can leave protected semantic attributes unchanged while reducing grammaticality, coherence or code execution. Include generic quality metrics and human inspection.

Perplexity changes are informative and can miss task-specific degradation. Use receiver-relevant outcomes.

AM. Dose–response should be monotonic only where the representation supports it

For each prompt, record target effect across α. Some examples show a monotonic region followed by saturation or reversal. Define the safe operating interval from the data instead of assuming larger intervention means more of the concept.

Context-aware controllers can choose α within this local interval.

AN. Per-example backfire rate is more informative than average improvement alone

A global vector can improve mean target score while moving ten or twenty percent of examples in the wrong direction. Report the fraction of examples with negative target effect, not only average effect.

The 2026 vector-field motivation is precisely this heterogeneity: control quality should improve locally, not only on average.

AO. Effect variance can be decomposed into prompt, layer and sampling variance

Repeat generation seeds for each prompt when decoding is stochastic. Repeat vector estimation on resampled contrast sets. Compare layers. A controller can be stable geometrically and noisy behaviourally because generation randomness dominates.

Use hierarchical or mixed-effects analysis when several variance sources matter.

AP. Long-form evaluation should track the attribute through time

Segment long outputs by token window, paragraph or semantic section. Measure the target attribute and quality in each segment.

Three failure modes become visible: decay, where control weakens; runaway amplification, where the attribute grows excessively; and oscillation, where the generated context pushes the model between regimes.

AQ. Generated context makes steering a closed-loop system even when the vector is open-loop

The controller can add the same vector every step, but the model’s next state depends on text produced under previous interventions. The plant being controlled changes with the controller’s own outputs.

Control-theoretic thinking therefore becomes useful: monitor state, adapt gain and avoid unstable positive feedback.

AR. Open-loop steering uses a fixed schedule

Apply a predetermined α over tokens or layers. This is simple, reproducible and unable to react when the target has already been achieved or the current state becomes uncertain.

Open-loop controllers are appropriate when response is predictable and state measurement is unreliable.

AS. Closed-loop steering measures the current state and adjusts intervention

Use an attribute detector, concept score or state estimator to choose α or v(h) at each step. The controller can reduce intervention after reaching a target region.

The detector can be fooled or miscalibrated. Closed-loop robustness depends on both plant and sensor.

AT. A proportional controller gives a simple feedback baseline

If current target score is s and desired score s*, choose α=k(s*−s) within safety limits. This adjusts strength to error.

Nonlinear response, measurement delay and autoregressive feedback can cause overshoot. The simple controller is useful as a transparent baseline rather than a universal solution.

AU. Gain scheduling adapts control to context classes

Use different α ranges for different prompt types, layers or confidence regimes. This captures some context dependence without learning a full vector field.

Routing errors near regime boundaries should be evaluated explicitly.

AV. Multi-attribute steering needs factorial evaluation

For two attributes A and B, test four conditions: none, A only, B only, A+B. The interaction is the joint effect beyond what would be expected from the single conditions.

Repeat over α and β. One attribute can help another at weak strength and conflict at strong strength.

AW. Multi-objective control should report a surface, not one hand-picked coefficient pair

Plot target A, target B, quality and protected metrics over a grid of coefficients. Identify Pareto-efficient regions.

This makes interference visible and prevents a convenient single example from defining the method’s capability.

AX. Concept antagonism can be real rather than representational interference

Two attributes can be intrinsically hard to satisfy together—extreme brevity and exhaustive detail, for example. Steering trade-offs then reflect the task, not poor feature geometry.

Compare with prompted or fine-tuned baselines to distinguish inherent objective conflict from steering-specific interference.

AY. Prompting is a necessary baseline for steering

If a natural-language instruction can achieve the same target with lower collateral damage, activation steering may not be worth its complexity. Compare against prompt engineering, system instructions and decoding controls.

Steering is most compelling when it adds controllability unavailable or unreliable through ordinary interfaces.

AZ. Fine-tuning and adapters are persistent-control baselines

Low-rank adaptation or supervised fine-tuning can create stable behaviour at training cost and with parameter changes. Steering avoids retraining and can be less stable.

Compare data, compute, reversibility, specificity and deployment overhead. No control mechanism is universally best.

BA. Decoding parameters can imitate some style controls

Temperature, top-p and repetition penalties alter diversity and style. A steering method controlling verbosity or determinism should be compared with these simpler decoding knobs.

Control should be attributed to the representation intervention only after cheaper interfaces are ruled out as explanations.

BB. Distribution shift can rotate the effective concept direction

A vector estimated on English prose can perform differently on code, another language or dialogue. Re-estimate local differences and measure cosine/effect changes across domains.

Domain-specific controllers can be appropriate. The label should include the domain scope.

BC. Cross-lingual steering is a strong test of semantic abstraction

Estimate a concept direction in one language, align or directly apply it in another and measure effect. Transfer suggests some shared geometry; failure can reflect language-specific bases or concept differences.

Do not assume translation labels define identical cultural or pragmatic concepts across languages.

BD. Model updates invalidate steering calibration

Retraining, instruction tuning, quantisation or distillation can shift activation scales and concept geometry. Recalibrate layer, vector and α after model changes.

A steering controller is versioned software tied to a particular internal interface.

BE. Quantisation can change intervention scale without changing nominal vector values

Low-precision activations and altered layer norms change the effective perturbation. Compare dose–response before and after deployment compression.

Internal-control methods require deployment-environment validation, not only full-precision research-model results.

BF. Throughput overhead matters for context-aware steering

A constant vector add is nearly free. A vector field requiring gradient computation or another neural controller can add latency and memory. Per-token closed-loop measurement can be expensive in long sequences.

Report inference cost alongside control quality. The best research controller may be impractical for production.

BG. Cached keys and values complicate layer-wise steering

During autoregressive inference, key–value caches store earlier attention states. Steering the current residual stream does not retroactively change cached representations unless the method explicitly does so.

This creates temporal asymmetry: earlier context remains encoded under the unsteered model while later states are controlled. Persistent steering can affect future cache entries only.

BH. Cache-aware evaluation is required for long sequences

Compare steering from the start of generation with steering introduced midway. Measure whether the controller can overcome already cached context.

Stateful serving systems should document whether hooks apply before or after cached-state construction.

BI. Reversibility should be tested after removing the controller mid-generation

Turn steering on for a segment, then off. Does the model immediately return to baseline behaviour, or does the generated context preserve the steered mode?

The latter is not parameter persistence; it is environmental persistence through context. For users, the distinction can still matter operationally.

BJ. Steering can create path dependence

Apply the same current-state controller after two different steering histories. The generated contexts and hidden states differ, so outcomes can differ. The system is path-dependent even with a deterministic vector rule.

Long-form evaluation should include different intervention schedules, not only one constant policy.

BK. Human evaluation should be blinded to steering condition where possible

If raters know which outputs were steered toward “helpfulness” or “formality”, expectation can bias judgement. Randomise and blind output order.

Collect separate scores for target property, quality and unintended changes rather than one overall preference.

BL. Automated evaluators can share biases with the steering estimator

If the same LLM generates contrast labels, builds the concept score and evaluates output, correlated errors can exaggerate efficacy. Use independent evaluators or human checks on a subset.

Evaluation independence is especially important for subtle concepts such as honesty, uncertainty or deference.

BM. Safety controls should sit outside the steering vector

A controller intended to increase safety can fail under distribution shift. External policy enforcement, permissions, filtering and human oversight should not depend solely on an internal steering direction.

Representation steering is one control signal inside a broader system, not the final authority.

BN. A safe-control evaluation should include adversarial and ambiguous contexts

Test prompts near the concept boundary, prompts combining conflicting goals and inputs unlike the contrast data. The controller should reduce strength or abstain when its semantic model is uncertain.

Focus on robustness and failure detection rather than operational bypass demonstrations.

BO. Worked experiment: global vector versus local vector field

Construct a synthetic curved concept geometry or use a dataset with measurable context regimes. Compare one mean-difference vector with a learned local gradient field.

Report mean target effect, backfire rate, intervention norm, protected-score drift and compute cost. A field earns its complexity if it reduces context-dependent failures materially.

BP. Worked experiment: long-form drift

Generate outputs of increasing length under fixed steering, decaying steering and closed-loop steering. Score the target attribute and quality in windows.

Plot decay, overshoot and variance. This identifies whether the controller needs feedback rather than stronger α.

BQ. Worked experiment: two-attribute interference surface

Choose two benign measurable attributes and sweep α,β over a grid. Measure both targets plus a quality metric.

Compare raw vector addition, orthogonalised vectors and locally constrained optimisation. The surface shows whether geometric separation translates into behavioural separation.

BR. Worked experiment: norm-preserving versus additive control

Match additive and spherical interventions for initial target effect. Generate across many prompts and lengths. Compare hidden-state norm, quality, backfire rate and target persistence.

If spherical steering helps, the benefit should appear at matched control efficacy rather than merely from a weaker effective perturbation.

BS. The steering deployment evidence card

• Target evaluator: independent measure of desired effect.
• Protected metrics: named collateral behaviours.
• Quality: fluency/task correctness/human judgement.
• Dose curve: efficacy and damage over α.
• Backfire rate: per-example wrong-direction effect.
• Long-form profile: decay, overshoot and drift.
• Context policy: global, clustered, field or closed-loop.
• Multi-control: interaction surface and ordering.
• Shift tests: domains, languages and unfamiliar states.
• Versioning: model and controller calibration.
• Compute: latency/memory overhead.
• Reversibility: parameter and contextual persistence.
• Safety gates: external controls and abstention.

BT. Final release gate for representation steering

1. The target concept has an operational definition and matched contrast design.
2. Direction estimation uncertainty is measured.
3. Probe, mean-difference and feature directions are not conflated.
4. Layer and token-position selection are validated out of sample.
5. Intervention size is normalised to natural activation scale.
6. Dose–response and backfire rates are reported.
7. Norm shift and geometric side effects are measured.
8. Static-vector context dependence is tested.
9. Context-aware fields earn their added complexity empirically.
10. Long-form drift and autoregressive feedback are evaluated.
11. Multi-attribute interactions are mapped rather than assumed additive.
12. Efficacy, specificity and quality are separate axes.
13. Prompting/fine-tuning/decoding baselines are included.
14. Distribution shift and model-version drift are tested.
15. Controller overhead is reported.
16. External safety and authority controls remain independent of steering.
17. No safeguard-bypass procedure is provided as part of educational analysis.

The longform conclusion is that representation steering should be treated as control engineering over learned state rather than as a clever interpretability trick. A useful internal direction must be estimated, calibrated, constrained and evaluated like any other actuator. The model’s representation is nonlinear, context-dependent and dynamic under autoregressive generation. The simplest successful controller may be one global vector; the harder cases require subspaces, local vector fields, geometry-aware paths or feedback. Complexity is justified only when it reduces measured failure.

Steering succeeds when an internal intervention becomes a reliable control interface: predictable in dose, narrow in side effects, stable across context, reversible where promised, and humble enough to abstain when its representation of the concept no longer applies.