The same token can enter a model with the same vocabulary identity and leave each layer with a different contextual representation.
This is one of the central reasons modern language models can handle polysemy, reference, syntax and long-range relationships. The token is not interpreted once and frozen. Its internal state is repeatedly transformed as information from the surrounding sequence is integrated.
This article follows Token IDs and Embeddings in the eduKateSingapore Representation and Tokenisation series.
The Core Transition
TOKEN ID → INITIAL EMBEDDING → POSITION / ORDER → LOCAL CONTEXT → LONGER-RANGE CONTEXT → UPDATED INTERNAL STATE → TASK-RELEVANT REPRESENTATION
1. Surface Identity Is Only the Starting Point
The surface token bank may use the same vocabulary entry in “river bank” and “central bank.” The initial model-side state begins from the same token identity, but the surrounding sequence differs. Later computation uses that difference to construct different contextual states.
This makes a useful distinction: lexical identity is stable; contextual interpretation is conditional.
2. Position Changes Role
Sequence order matters. “The teacher praised the student” and “The student praised the teacher” contain similar words but assign different roles. A model therefore needs access to ordering information as well as token identity.
Position does not create meaning alone, but it helps determine which relationships are possible.
3. Neighbouring Tokens Constrain Interpretation
Nearby words often resolve ambiguity quickly. Pitch near football, music or sales suggests different interpretations. Local context acts as a constraint system.
But some ambiguities cannot be resolved locally. A pronoun may refer to a noun several clauses earlier; a technical term may depend on a definition introduced paragraphs before. Context therefore operates at multiple scales.
4. Attention Is Relationship Selection
Attention mechanisms allow positions in a sequence to weigh information from other positions. This does not mean the model “looks” at text like a human eye. It means one internal state can be updated using weighted information from other states.
The representational consequence is important: a token state can become partly about other tokens that matter to its current role.
5. Contextual Meaning Is Distributed
The model does not need one internal slot labelled “meaning of bank.” Useful interpretation can be distributed across many dimensions and many interacting positions. Some information may be carried in the current token state, some in nearby states, some in broader sequence structure.
Meaning-like behaviour therefore emerges from the system of relationships, not from one token acting alone.
6. Syntax Changes Representation
Grammar constrains which words modify, govern or depend on others. “The dog that chased the cat barked” requires the system to distinguish the main subject from a noun inside a relative clause.
Contextual representations can encode these structural relationships because training rewards states that support accurate prediction and transformation.
7. Reference Requires Memory Across the Sequence
In “Maya put the book on the table because it was unstable,” the word it is ambiguous without broader context. Resolving reference depends on competing candidates, plausibility and surrounding information.
Reference is therefore a representation-linking problem: which earlier object should this later token connect to?
8. Polysemy Is a Context Test
Words often have multiple related senses. A contextual model does not need to assign a separate vocabulary token to every sense. It can begin from one lexical identity and let context move the internal state toward the relevant usage.
This shifts complexity from the vocabulary into the sequence model.
9. Homonymy Makes the Difference Even Sharper
Two meanings can share spelling by historical accident. Bat can refer to an animal or sporting equipment. The surface token alone cannot settle the interpretation. Context must.
This is why static word lookup is an incomplete model of language understanding.
10. A Token State Can Encode Its Role, Not Just Its Word
As contextual processing proceeds, an internal state may carry information about whether a token acts as a subject, object, modifier, entity mention, numerical value, quotation, instruction or other sequence role.
The same surface token can therefore occupy different functional regions of the model’s internal space.
11. Layers Can Increase Abstraction
Early representations may remain closely tied to local form, while later layers can integrate broader relationships. It is risky to assign one universal function to every layer, but the general pattern is useful: repeated transformation allows representations to become increasingly conditioned on the whole task context.
A token’s state is therefore historical: it records the transformations it has undergone.
12. Context Has a Finite Horizon
A model cannot use information that is not available inside its accessible context or memory system. If an important definition lies outside the context capacity, the model cannot directly condition on it unless retrieval or another memory mechanism brings it back.
This links contextual representation to Token Budgets and Chunking.
13. More Context Is Not Always Better
Adding irrelevant text can introduce competing cues, distract from important evidence or create contradictory instructions. Context quality therefore matters alongside context quantity.
A useful context preserves the distinctions required for the current task and suppresses unrelated material.
14. Recency Can Compete With Earlier Evidence
In long sequences, newer statements can modify, contradict or override earlier ones. Systems therefore need ways to represent temporal and logical relationships rather than merely treating all previous tokens as equally authoritative.
Context is not a bag of facts. It is a structured history.
15. Instructions Are Context Too
System instructions, user requests, examples and tool outputs all enter the model as contextual representations. Their roles differ even when they are all encoded as token sequences.
The interface therefore needs protocol-level structure so the model can distinguish types of context.
16. Quotation Changes the Interpretation of Words
The sentence “She said ‘the plan failed’” differs from the speaker directly asserting “the plan failed.” The words inside the quotation may be identical, but discourse context changes who owns the claim.
Good contextual representation therefore includes stance and attribution, not only lexical content.
17. Negation Changes Nearby Representations
“The medicine is effective” and “The medicine is not effective” differ through one small token with large semantic consequence. A representation system must propagate the effect of negation to the proposition it modifies.
This is a useful reminder that token frequency or size does not determine conceptual importance.
18. Modality Changes Certainty
“It will rain,” “it may rain,” and “it probably will not rain” refer to related events with different certainty. Contextual representation must preserve these differences if downstream decisions depend on them.
Collapsing modality turns uncertain claims into false precision.
19. Time Changes Meaning
“The bridge is closed” has different operational meaning depending on whether it was written yesterday, today or ten years ago. Time expressions and document timestamps therefore belong to context when the represented world changes.
Contextual fidelity requires temporal location, not just text content.
20. Jurisdiction Can Change Interpretation
Words such as legal, school year, tax or licence can depend on country, state or institution. A sequence that omits jurisdiction may leave the model with multiple plausible worlds.
Context resolves not only language ambiguity but world-state ambiguity.
21. Domain Context Reweights Vocabulary
Derivative means different things in mathematics, finance and chemistry. Domain context changes which relationships become relevant. The token remains the same surface form while its active conceptual neighbourhood changes.
22. Context Can Be Explicit or Inferred
Sometimes the domain is named directly: “In calculus…” Sometimes it must be inferred from surrounding vocabulary. Explicit context is generally safer when the decision matters because it reduces the burden on inference.
Strong prompting and strong technical writing make important context visible.
23. Retrieval Supplies External Context
Retrieval systems can add documents or passages to a model’s current context. This does not permanently rewrite the model’s parameters. It temporarily changes the evidence available for contextual representation.
The quality of retrieval therefore directly affects the world the model can represent in that turn.
24. Bad Retrieval Creates Bad Context
If retrieval returns outdated, irrelevant or duplicated evidence, the model receives a distorted representation of the knowledge environment. Better reasoning cannot guarantee correction when the evidence supplied is systematically poor.
Context quality is an upstream responsibility.
25. Chunk Boundaries Can Break Context
A passage may be retrieved without the heading that defines its topic, the paragraph that contains its exception or the table column that gives its units. The tokens inside the chunk are intact, yet the contextual representation is damaged.
This is why document chunking is a representation-design problem rather than only a storage problem.
26. Context Can Contain Contradictions
Real documents disagree. A contextual system must sometimes represent multiple competing claims rather than collapse them prematurely into one answer.
Good context therefore preserves source identity and provenance so disagreement remains visible.
27. Contextual Representation Is Not Guaranteed Truth
A model can form a coherent contextual state around a false claim. Internal consistency is not external verification. The representation must still be tested against authoritative evidence and world return.
This separates representation quality from factual correctness.
28. Prompt Injection Is a Context-Authority Problem
When untrusted retrieved text contains instructions, a system must distinguish content from authority. The words may look like legitimate commands at token level while belonging to a lower-trust source.
Safe systems represent not only what text says, but what role and authority the text has.
29. Context Compression Is Lossy
Long conversations and documents are sometimes summarised to fit within bounded capacity. Summarisation creates a smaller representation. It can preserve decisions and key facts while losing wording, uncertainty, provenance or minority viewpoints.
Compressed context should therefore declare what invariants must survive.
30. Memory Is Curated Context
Persistent memory systems select information from past interactions and make it available later. This creates another representation layer: what is remembered, what is omitted, how it is summarised and when it is retrieved.
Memory quality depends on relevance, accuracy, freshness and consent—not merely storage.
31. Human Reading Works Similarly
Humans also reinterpret words through context. The sentence before a metaphor changes how it is understood. A chapter heading changes how a paragraph is framed. Prior knowledge changes which distinctions a reader notices.
The analogy is imperfect, but educationally useful: context is part of representation, not decoration around it.
32. Teaching Builds Context Deliberately
A good teacher prepares context before introducing a difficult concept. Definitions, examples, diagrams and prerequisite knowledge create a representational environment in which the new idea can be interpreted correctly.
When students appear to “know the words but not understand,” the missing layer may be contextual structure.
33. Assessment Without Context Can Misdiagnose
A wrong answer can arise from vocabulary, representation, procedure, memory or transfer. The answer token alone does not identify the failure. Diagnostic assessment needs the surrounding reasoning path.
Context increases diagnostic resolution.
34. The Context Audit
- What token or object is being interpreted?
- What local context surrounds it?
- What longer-range context matters?
- What is the sequence order?
- Which earlier definitions or references constrain interpretation?
- What domain, date and jurisdiction apply?
- What is quoted versus asserted?
- What uncertainty or modality must survive?
- Which sources have authority?
- Is relevant evidence missing because of chunking or context limits?
- Are contradictions visible?
- Does the final interpretation survive world return?
35. The Deep Principle
Tokenisation gives a system units. Embeddings give those units numerical form. Context tells the system what those units are doing here, now, in relation to everything else available.
The same token can remain the same symbol while becoming a different working representation because meaning is relational, situated and conditional on context.