Context is not only about which tokens exist. It is also about which tokens are allowed to interact. Token interaction topology is the pattern of visibility that determines where information can flow inside a model.
Two models can receive the same tokens in the same order and still build different contextual representations because their attention masks, local windows, global tokens or sparse connections differ.
This article extends the eduKateSingapore Representation and Tokenisation series into the geometry of token-to-token information flow.
The Interaction Route
TOKENS + POSITION → VISIBILITY RULES → LOCAL / GLOBAL / CAUSAL CONNECTIONS → ATTENTION GRAPH → CONTEXT PROPAGATION → CONTEXTUAL REPRESENTATIONS → OUTPUT
1. Attention Defines a Communication Graph
At one layer, each token can be treated as a node. An allowed attention connection is an edge through which information can flow.
The attention mask therefore defines a graph over the sequence.
2. Full Self-Attention Creates a Dense Graph
In ordinary bidirectional full attention, every token can interact directly with every other token in the same layer.
This maximises direct connectivity and grows expensive as sequence length increases.
3. Dense Connectivity Is Not Always Necessary
Many relationships are local. Nearby words, adjacent image patches and neighbouring time steps often carry most of the immediately useful context.
Sparse topologies exploit this structure to reduce compute.
4. Causal Masks Create One-Way Information Flow
Autoregressive language models typically allow each position to attend only to itself and earlier positions during training.
The resulting topology is directed: later tokens can depend on earlier ones, but earlier representations cannot depend on future tokens.
5. Causality Here Is Architectural, Not Philosophical
A causal attention mask enforces prediction-time information order. It does not prove that earlier content causes later content in the real world.
Visibility constraint and causal inference are different ideas.
6. Bidirectional Attention Allows Mutual Context
Encoder-style models can allow a token to use evidence from both left and right context.
This is well suited to representation learning and classification tasks where the whole sequence is already available.
7. Padding Masks Remove Artificial Nodes
Sequences in a batch often contain padding so shapes align.
A padding mask prevents those placeholder positions from influencing real tokens.
8. Local Windows Restrict Direct Attention
A token can be allowed to attend only to positions within a fixed neighbourhood.
This reduces cost and creates a locality prior similar to convolution.
9. Locality Preserves Nearby Structure Efficiently
Syntax, phrase structure, visual texture and short temporal motion often depend heavily on local context.
Local windows spend compute where many strong correlations live.
10. Local Windows Delay Long-Range Communication
If token A and token Z lie outside each other’s window, information may still reach them through several layers of intermediate tokens.
Path length replaces direct connection.
11. More Layers Expand the Effective Receptive Field
A local window of radius r can influence progressively farther positions as layers stack.
Depth becomes one route to broader context.
12. Global Tokens Create Shortcuts
A global token can attend broadly and be attended to broadly.
This creates a hub through which distant information can interact without restoring full dense attention everywhere.
13. Longformer Combines Local and Global Attention
Longformer uses local windowed attention plus task-motivated global attention, reducing the cost of long-document modelling while preserving direct access for selected important positions.
The deeper principle is topology design: most tokens can communicate locally while a small set receives broader connectivity.
14. Choosing Global Tokens Is a Representation Decision
Question tokens, classification tokens, document headings or designated landmarks can receive global visibility.
What receives global status determines what can route information efficiently.
15. Sparse Attention Is a Family, Not One Pattern
Local windows, dilated links, random links, blocks and global hubs are all sparse connectivity patterns.
Different sparse graphs create different communication properties.
16. Connectivity Should Be Measured, Not Assumed
A sparse topology can look efficient while isolating important regions or requiring too many layers for information to travel.
Ask how many hops connect distant evidence under the actual mask.
17. Random Links Can Shorten Graph Diameter
A small number of long-range connections can dramatically reduce the number of hops between distant tokens.
This is a graph-theoretic reason sparse attention can still support long-range interaction.
18. Block-Sparse Attention Matches Hardware Better Than Arbitrary Sparsity
Some theoretically sparse patterns are difficult to execute efficiently on accelerators.
Practical topology design must consider both information flow and computational layout.
19. Attention Topology and Position Are Coupled
A local window is defined by position. Relative positional information helps the model distinguish which neighbour is one step away and which is ten.
See Positional Representation.
20. Hierarchical Topology Can Connect Levels Instead of Every Token
Tokens can first communicate within sentences or patches, then sentence-level or region-level summaries communicate globally.
This replaces one enormous flat graph with several connected scales.
21. Hierarchy Is a Topological Shortcut
A local token can reach distant information through its local summary, a global summary and another local summary.
Higher-level nodes shorten long-range paths.
22. Retrieval Creates External Topology
Instead of connecting every token inside one giant context, a system can retrieve relevant external chunks and add them as new neighbours.
The context graph is expanded selectively from outside the model.
23. Tool Calls Create Another Kind of Edge
A model can route from current context to a calculator, database or search system, then incorporate the returned result.
This is not attention in the narrow architectural sense, but it is a broader information-flow topology for intelligent systems.
24. Attention Weight Is Not Guaranteed Explanation
A large attention weight shows one pattern of internal interaction, not necessarily the causal reason for the final output.
Topology explains where information could flow; it does not by itself prove why a decision was made.
25. Mask Errors Can Leak Forbidden Information
An incorrect causal mask can expose future target tokens during training.
The model can then appear powerful because the topology accidentally allowed information it should not have seen.
26. Leakage Is a Representation Failure
The token sequence may be correct while the interaction graph is wrong.
Evaluation must therefore verify visibility rules as well as data content.
27. Document Boundaries Need Interaction Rules
If several documents share one batch or packed sequence, tokens from one document should not automatically attend to another unless the task intends cross-document reasoning.
Boundary tokens without matching masks may not be enough.
28. Multimodal Tokens Need Cross-Modal Edges
Text can attend to image tokens, audio tokens or video tokens through cross-attention or unified self-attention.
Which modalities can see which others is part of the model’s topology.
29. Early Fusion and Late Fusion Are Topology Choices
Early fusion allows modalities to interact deeply from the start. Late fusion builds separate representations and connects them later.
The difference changes both compute and the kinds of cross-modal relationships that can form.
30. Agent Systems Have Interaction Topology Too
Planner, retriever, tool, critic and memory modules can be connected in different graphs.
Intelligence at system scale also depends on which components can exchange information and when.
31. More Connectivity Is Not Always Better
Dense interaction can increase cost, amplify distractors and blur local structure.
A good topology exposes the right neighbours, not every possible neighbour.
32. Too Little Connectivity Creates Blind Spots
If distant evidence cannot reach the decision token within the available layers, the model effectively cannot use it.
Nominal context length can exceed effective information reach.
33. Effective Context Is a Graph Property
The useful question is not only “how many tokens fit?” but “how can information travel among those tokens before the output is produced?”
Capacity depends on connectivity as well as count.
34. The Token Interaction Topology Audit
- Which tokens can attend to which others?
- Is the graph causal, bidirectional or mixed?
- What local window size is used?
- Which tokens receive global visibility?
- How many hops connect distant evidence?
- Are random, dilated or block-sparse links used?
- Does the sparsity pattern map efficiently to hardware?
- Are padding and document boundaries masked correctly?
- Can future or label information leak through the mask?
- How do position encodings interact with the topology?
- Are multimodal cross-connections explicit?
- Does the effective receptive field match the receiver’s task?
35. What Students Should Remember
- Attention masks define an information-flow graph.
- Causal attention is directional; bidirectional attention is not.
- Local windows reduce cost while lengthening some information paths.
- Global tokens create long-range shortcuts.
- Hierarchy can connect local and global context efficiently.
- Mask errors can leak forbidden information.
- Effective context depends on connectivity, not just token count.
36. The Deep Principle
A context window is not a bag of mutually visible facts. It is a communication network.
Token interaction topology decides which evidence can meet. Good context architecture does not connect everything to everything; it creates the shortest trustworthy routes between the distinctions the receiver needs to combine.