A dense neural network applies roughly the same parameter blocks to every token. A mixture-of-experts model asks a different question: which subset of parameters should process this token?
Expert routing is the representation-and-compute decision that maps token states to specialised expert modules. The token remains the same input unit, but the path through the model becomes conditional.
This article continues the eduKateSingapore Representation and Tokenisation series. It follows naturally from adaptive tokenisation: one mechanism changes which tokens survive; expert routing changes which parameters process the surviving tokens.
The Expert Routing Route
TOKEN REPRESENTATION → ROUTER SCORES → TOP-k EXPERT CHOICE → SPARSE EXPERT COMPUTATION → COMBINE EXPERT OUTPUTS → NEXT LAYER CONDITIONAL COMPUTATION = MANY AVAILABLE PARAMETERS + FEW ACTIVE PARAMETERS PER TOKEN
1. Experts Are Parameter Blocks
An expert is usually a feed-forward subnetwork or related parameter block that can process token representations.
Several experts can exist in parallel while only a small subset activates for one token.
2. The Router Is a Classifier Over Experts
A router maps the current token representation to scores over available experts.
Those scores decide where the token’s computation should go next.
3. Routing Is Conditional Computation
A model can contain a very large number of total parameters while activating only a fraction for each token.
This changes the relationship between parameter capacity and per-token compute.
4. Switch Transformer Simplified Sparse Routing
Switch Transformers route each token to a single selected expert in the Switch layer, simplifying earlier MoE designs while retaining sparse activation.
The core representation idea is simple: one token can select a specialised computational path rather than paying for every expert.
5. Top-1 Routing Is Efficient and Brittle
Sending each token to one expert keeps activation sparse and communication simple.
If the router chooses poorly, the token receives no backup expert at that layer.
6. Top-k Routing Adds Redundancy
Other MoE systems send a token to two or more experts and combine their outputs.
This increases compute while reducing dependence on one routing decision.
7. Expert Choice Depends on the Current Token State
The same surface token can route differently in different contexts because earlier layers have changed its representation.
Routing therefore operates on contextual state, not raw token ID alone.
8. Experts Need Not Correspond to Human Topics
One expert is not automatically “the mathematics expert” or “the French expert”.
Specialisation can emerge in distributed, overlapping and difficult-to-name ways.
9. Interpretability Should Be Measured, Not Assumed
Researchers can inspect which tokens route to which experts and search for patterns.
Observed specialisation does not guarantee a clean semantic partition.
10. Load Balancing Is a Routing Constraint
If the router sends nearly every token to one expert, the sparse architecture loses parallel capacity and can overload that expert.
Auxiliary losses and routing controls encourage tokens to spread across experts.
11. Balanced Load Is Not the Same as Useful Specialisation
Perfectly even traffic can be computationally convenient while forcing unnatural routing.
The system must balance expert utilisation without erasing content-sensitive choice.
12. Expert Capacity Limits Create Queue-Like Behaviour
Implementations may limit how many tokens one expert processes in a batch.
When too many tokens choose the same expert, some can be dropped, rerouted or handled under overflow rules.
13. Capacity Factor Is an Operational Representation Budget
More spare expert capacity reduces overflow risk and increases memory or communication cost.
Routing quality and systems capacity are coupled.
14. Token Dropping Can Become Information Loss
If an overloaded expert cannot process all assigned tokens, dropped tokens may receive degraded computation.
Throughput engineering can therefore become representational fidelity.
15. Communication Cost Matters in Distributed MoE
Experts are often distributed across devices. Tokens must be sent to the devices hosting their selected experts and outputs returned.
Sparse arithmetic can still incur substantial network movement.
16. Model FLOPs and System Latency Are Different Metrics
Activating few experts can reduce arithmetic while all-to-all communication becomes a bottleneck.
MoE efficiency must be measured on real serving hardware, not only by theoretical active-parameter count.
17. Routing Stability Matters
Small changes in a token representation can sometimes switch the selected expert.
That discontinuity can create brittle behaviour near routing boundaries.
18. Expert Boundaries Are Learned Decision Surfaces
The router partitions representation space into regions favouring different experts.
The partition is learned from training objectives rather than manually defined domain taxonomies.
19. Expert Collapse Is a Failure Mode
If only a few experts become useful, others may receive little meaningful training.
The model then carries dormant capacity rather than productive specialisation.
20. Dead Experts Resemble Dead Codebook Entries
In vector quantisation, unused codebook entries waste representational capacity. In MoE, unused experts waste parameter capacity.
Both systems need utilisation monitoring.
21. Expert Diversity Can Improve Model Capacity
Different experts can learn different transformations so the model represents more functions without applying all of them to every token.
Sparsity creates conditional expressivity.
22. Diversity Without Routing Quality Is Useless
A brilliant expert does not help a token that never reaches it.
Router competence is therefore part of model competence.
23. Routing Error Is a Distinct Failure Layer
A token can be well represented and the expert can be capable, yet performance fails because the router sends the token elsewhere.
Diagnose routing separately from expert quality.
24. Expert Routing Is Different From Attention Routing
Attention chooses which token information to combine. MoE routing chooses which parameter block transforms a token.
One routes information; the other routes computation.
25. Expert Routing Is Different From Adaptive Tokenisation
Adaptive tokenisation changes how many token units survive or merge.
MoE usually keeps token count while changing which expert parameters activate.
26. The Two Can Work Together
A system can first reduce redundant tokens and then route the remaining tokens to specialised experts.
Compute can be sparse along both token and parameter dimensions.
27. Routing Can Be Hierarchical
A top-level router can choose an expert group and a second router choose one expert inside the group.
Hierarchical routing can reduce routing cost and reflect multi-scale specialisation.
28. Expert Granularity Is a Design Choice
Experts can replace whole feed-forward blocks, smaller modules or larger subnetworks.
Coarse experts concentrate more capacity behind each routing decision.
29. Expert Count Is Not Active Capacity
A model may contain hundreds of experts while one token sees only one or two at a layer.
Total parameter count and active parameter count should be reported separately.
30. Data Distribution Shapes Expert Specialisation
If some languages, domains or token types dominate training, routing patterns can reflect those imbalances.
Minority inputs may receive undertrained expert paths.
31. Load Balance Can Hide Quality Imbalance
Two experts can receive equal token counts and very different semantic difficulty.
Measure per-expert performance, not traffic volume alone.
32. Routing Needs Observability
Useful metrics include expert utilisation, overflow, router entropy, token-expert patterns and per-domain routing distributions.
Conditional computation is difficult to debug when the conditions remain invisible.
33. The Expert Routing Audit
- What expert modules exist?
- What representation does the router receive?
- Is routing top-1, top-2 or another top-k?
- How are router scores normalised?
- How is load balancing encouraged?
- What capacity limit applies per expert?
- What happens to overflow tokens?
- How much communication does routing create?
- Are routing choices stable under small perturbations?
- Do experts develop productive specialisation?
- Are some experts dead or overloaded?
- How does routing quality vary by language or domain?
- What is total versus active parameter count?
- Can routing failures be separated from expert failures?
34. What Students Should Remember
- Mixture-of-experts models activate only a subset of parameters for each token.
- The router chooses expert paths from contextual token representations.
- Switch Transformer simplified sparse expert routing with top-1 expert choice.
- Load balance, expert capacity and communication are central systems constraints.
- Expert routing changes parameter flow, not token identity.
- Routing failure is a distinct failure layer.
35. The Deep Principle
Mixture-of-experts models make parameter use conditional.
Scale does not require every token to traverse every capability. Expert routing works when the model can recognise which computation this token needs—and send it there without losing balance, stability or evidence about the path it took.