A token tells a model what unit is present. Position tells the model where that unit sits in relation to everything else. Without positional information, many Transformer-style attention mechanisms would treat a reordered sequence as containing the same collection of token vectors.
That creates a foundational representation problem. “Dog bites man” and “man bites dog” contain almost the same lexical inventory. Their meaning changes because order changes. Positional representation is the machinery that lets sequence models preserve that distinction.
This article continues the eduKateSingapore Representation and Tokenisation series. The focus is not token identity but the second coordinate every sequence needs: where is this token?
The Positional Route
TOKEN IDENTITY + POSITION / RELATION → POSITION-AWARE REPRESENTATION → ATTENTION / INTERACTION → CONTEXTUAL STATE → ORDER-SENSITIVE OUTPUT
1. Self-Attention Does Not Supply Order by Itself
The original Transformer architecture in Attention Is All You Need added positional encodings because attention alone does not contain recurrence or convolution that automatically imposes sequence order.
The model therefore needs an explicit representation of position or relation.
2. Absolute Position Answers “Which Slot?”
Absolute position assigns each token an index such as 0, 1, 2, 3 and so on.
The representation can then combine token identity with a vector associated with that slot.
3. Learned Position Embeddings Treat Slots Like a Vocabulary
A model can learn one vector for position 1, another for position 2 and so forth.
This is simple and flexible inside the trained range, but positions beyond that range may not have learned representations.
4. Sinusoidal Encodings Use a Deterministic Pattern
The original Transformer used sine and cosine functions at different frequencies so positions receive structured vectors without requiring a learned table for every slot.
The design lets nearby and distant positions be represented through a reusable mathematical pattern.
5. Absolute Position Is Not the Same as Distance
Tokens at positions 100 and 101 are adjacent even though their absolute indices are large.
Many linguistic and structural relationships depend more on relative distance than on absolute slot number.
6. Relative Position Answers “How Far Apart?”
Self-Attention with Relative Position Representations introduced attention mechanisms that explicitly represent distances or relations between sequence elements.
Instead of asking only where each token sits globally, the model can ask how one token is positioned relative to another.
7. Relative Position Often Matches Language Structure Better
A modifier appearing two words before a noun may behave similarly whether the pair occurs near the beginning or end of a document.
Relative distance can capture this reusable structural pattern.
8. Direction Matters
Two tokens can be three positions apart while one lies before and the other after.
Relative position should preserve sign or direction when order matters.
9. Distance Buckets Compress Large Separations
Some architectures represent small distances precisely and group large distances into broader buckets.
This reflects a useful prior: exact local distance may matter more than whether two paragraphs are 800 or 801 tokens apart.
10. Positional Resolution Can Be Uneven Deliberately
Fine local position and coarse global position can coexist.
Representation resolution should match the relationships the task needs to distinguish.
11. Rotary Position Encodings Represent Relation Through Rotation
Rotary approaches apply position-dependent rotations to query and key representations so relative offset affects attention interactions.
The important conceptual move is that position is woven into how tokens compare rather than simply added once to token embeddings.
12. Position Can Live in the Interaction, Not Only the Token
This distinction matters. A positional vector attached to a token says “I am at slot 37.” A relation-aware interaction says “I am five places after you.”
Both are positional representations, but they live at different layers.
13. Context Extension Is Partly a Position Problem
A model trained on short sequences may encounter positional configurations far outside its training distribution when deployed on longer contexts.
Supporting a larger nominal context window therefore requires more than increasing an input limit.
14. Extrapolation Tests Whether Position Generalises
A positional method that performs well inside its training range can degrade when sequence length grows substantially.
Long-context evaluation should therefore test actual retrieval and reasoning at distant positions.
15. Position Is Not Importance
Being first, last or near the centre does not automatically make a token more relevant.
Position says where evidence is. Attention and task structure decide whether it matters.
16. Yet Models Can Develop Positional Bias
Training data and architecture can produce stronger use of information at some context locations than others.
Long-context tests should therefore vary where decisive evidence appears.
17. Document Position Has More Than One Coordinate
A word can have a token index, sentence position, paragraph position, section identity and page location simultaneously.
One linear coordinate cannot express the full document hierarchy.
18. Hierarchical Position Preserves Structural Address
A token may be represented as section 3 → paragraph 2 → sentence 4 → token 7.
This connects positional representation to the companion article on Hierarchical Representation.
19. Images Need Two-Dimensional Position
An image patch has row and column coordinates rather than one natural reading position.
Flattening patches into one sequence is an implementation convenience; the model still needs to recover 2D geometry.
20. Video Adds a Third Positional Axis
A video token can have x, y and time coordinates.
Spatial location and temporal order must survive together.
21. 3D Models Add Physical Coordinates
Point clouds and meshes may use x, y and z coordinates plus local orientation or connectivity.
Position here can be metric geometry rather than sequence index.
22. Graphs Use Relational Position
A graph node may not have any meaningful absolute sequence slot. Position can instead mean shortest-path distance, structural role or graph neighbourhood.
This shows that “position” is broader than counting tokens from the left.
23. Sets May Need No Position at All
If the input is genuinely an unordered set, adding arbitrary order can inject false structure.
See Set and Permutation Representation.
24. Position Should Represent Real Structure, Not Serialization Accident
A database row order, JSON key order or arbitrary node index can be stable in one file while carrying no semantic meaning.
Models should not learn importance from accidental order.
25. Special Tokens Can Define Local Coordinate Systems
Segment markers, separators or document boundaries can reset or contextualise position.
Position can therefore be interpreted relative to structural landmarks.
26. Padding Should Not Become Meaningful Position
Batching often inserts padding tokens so sequences share a length.
Masks should prevent those artificial positions from influencing real tokens.
27. Position and Visibility Work Together
Knowing that two tokens are far apart matters only if the architecture allows them to interact directly or indirectly.
This connects position to Token Interaction Topology.
28. Causal Models Add Directional Visibility
A token at position 10 may see positions 1–9 but not 11–20 during autoregressive training.
Sequence position and attention permission jointly define the information available at each step.
29. Position Is a Coordinate System for Context
The model needs enough geometry to distinguish before from after, near from far, inside from outside and local from global.
Different modalities require different coordinate systems.
30. The Positional Representation Audit
- Does the input have meaningful order?
- Is position absolute, relative or relational?
- What is the maximum trained position range?
- How are positions beyond training evaluated?
- Are local distances represented more precisely than long distances?
- Does direction matter?
- Are segment, page or section coordinates also needed?
- Does the modality require 2D, 3D or temporal coordinates?
- Could serialization order inject false structure?
- Are padding positions masked?
- Can decisive evidence appear anywhere in the context?
- How does position interact with attention topology?
31. What Students Should Remember
- Token identity alone does not preserve sequence order.
- Absolute position identifies slots; relative position identifies relationships between slots.
- Position can live in token embeddings or in attention interactions.
- Long-context performance requires positional generalisation, not merely a larger limit.
- Images, video, 3D data and graphs need richer coordinates than one line of numbers.
- Unordered sets should not receive fake sequence meaning.
32. The Deep Principle
A token without position tells the model what exists. Position tells it how existence is arranged.
Representation becomes structure when identity acquires coordinates. The right positional system preserves exactly the spatial, temporal or relational distinctions the receiver needs—and no accidental ordering it does not.