A token ID is an address. An embedding is a numerical representation associated with that address. Neither is the meaning of the token by itself.
This distinction sits at the heart of modern machine learning. Human-visible text becomes tokens. Tokens become vocabulary identities. Those identities are mapped into numerical states on which the model can perform mathematics. From there, context transforms the initial representation again and again.
This article continues the eduKateSingapore Representation and Tokenisation series and follows directly from How Tokenisation Works.
The Route
VISIBLE INPUT → TOKENISATION → TOKEN PIECES → TOKEN IDs → EMBEDDING LOOKUP → POSITION / ORDER INFORMATION → CONTEXTUAL TRANSFORMATION → OUTPUT REPRESENTATION → DECODED RESULT
1. Token IDs Are Vocabulary Addresses
After a tokenizer has segmented text, each recognised piece is mapped to an integer identifier. If a vocabulary contains 50,000 entries, every token piece needs a stable address within that vocabulary. The token ID performs that role.
The number itself is arbitrary in the semantic sense. Token 8421 is not inherently “closer in meaning” to token 8422 than to token 17. Integer adjacency usually reflects storage or vocabulary construction order, not conceptual proximity.
2. Vocabulary Identity Must Be Stable
The model is trained with a particular mapping between token pieces and token IDs. If that mapping changes, the model would receive the wrong addresses. A token ID therefore has meaning only inside a specific tokenizer–vocabulary–model contract.
This is why tokenizer files travel with model releases. The vocabulary is not a convenience layer that can be swapped casually after training.
3. The Embedding Table Converts Addresses Into Vectors
An embedding table can be imagined as a matrix with one learned row for each token identity. When a token ID arrives, the model retrieves the corresponding numerical vector. That vector becomes the token’s initial model-side representation.
The result is a major representational transition:
SYMBOLIC IDENTITY TOKEN #8421 ↓ NUMERICAL STATE [0.13, -0.42, 0.07, ...]
4. Why Use Vectors?
Vectors allow learned systems to operate through addition, multiplication, projection, similarity, attention and other numerical transformations. A discrete token identity can only say “this is token X.” A vector can participate in a geometry where many properties are distributed across dimensions.
This does not mean every dimension has a clean human label. Learned representations can encode useful structure without providing one interpretable axis per concept.
5. Embeddings Are Learned, Not Hand-Written Dictionaries
Traditional dictionaries assign explicit human definitions. Embeddings are learned numerical parameters optimised during training. Their values change because doing so helps the model perform its training objective.
The resulting vector can capture statistical relationships that support prediction, but it does not contain a readable definition hidden in decimal form.
6. Similarity Is Representation-Dependent
Two vectors can be compared using a distance or similarity measure. If they are close under that representation and metric, they may behave similarly for some learned purposes. But “close” is always relative to the geometry produced by the model.
Distance in an embedding space is therefore not universal meaning-distance. Another model, layer, task or metric may place the same items differently.
7. Initial Embeddings Are Context-Free
The first embedding retrieved for a token depends on its token ID, not on the sentence in which it appears. The visible word bank begins from the same vocabulary entry whether the sentence refers to a river bank or a financial institution.
Context enters through later sequence processing. This is why the initial embedding should not be mistaken for the model’s final interpretation of the token.
8. Position Is a Separate Problem
A bag of token embeddings would lose sequence order. Yet order can reverse meaning: dog bites man and man bites dog contain the same visible words. Models therefore need some representation of position or relative order.
Different architectures encode position differently, but the representational need is universal: identity alone is insufficient when sequence matters.
9. Identity Plus Position Still Is Not Context
Knowing which token appears and where it appears gives the model a structured input, but contextual meaning still depends on relationships among tokens. Sequence layers repeatedly exchange information across positions so each token state becomes informed by its neighbours.
This is the point where a static vocabulary item becomes a contextual representation.
10. Contextual States Are New Representations
After one transformation layer, the vector at a token position is no longer simply the initial embedding. After many layers, it may encode information about syntax, reference, topic, local relationships and long-range dependencies relevant to the model’s objective.
It is therefore more accurate to talk about a sequence of representations than one fixed embedding for meaning.
11. A Word Can Occupy Different Contextual Regions
The word cell in biology, prisons, spreadsheets and batteries has the same surface form but different contextual roles. Modern contextual models can transform its internal representation differently in each sentence because surrounding evidence changes the state.
This is developed further in Contextual Representation | How the Same Token Changes Across a Sequence.
12. Subword Tokens Complicate Human Interpretation
Many vocabulary entries are not full words. A single human word may be represented by several subword tokens, each with its own embedding. The model must compose those parts through context.
This reinforces the main rule: token-level vectors should not be read as if they were dictionary entries for human words.
13. Rare Tokens Can Still Have Useful Vectors
A token does not need to be frequent to participate in a model, but the quality of its learned representation depends on the evidence available during training and on how information is shared through subword structure and context.
For uncommon names and technical terms, several smaller pieces may provide a route into the model even when no whole-word token exists.
14. Special Tokens Have Embeddings Too
Control tokens, separators, role markers and other special identities can also map to learned vectors. These may represent protocol states rather than ordinary lexical content.
This shows that embeddings can encode model-interface functions as well as language pieces.
15. Embedding Dimension Is a Capacity Choice
The length of an embedding vector determines the dimensionality of the model’s initial token representation. More dimensions provide more numerical capacity but also increase parameter and computation requirements.
There is no universal “correct” dimension independent of architecture, scale and task.
16. Bigger Vectors Do Not Automatically Mean Better Meaning
A high-dimensional representation can carry more degrees of freedom, but useful structure still depends on training, data, objective and architecture. Numerical capacity without the right evidence does not create semantic truth.
Representation quality must be tested through behaviour and world return, not inferred from vector length.
17. Embeddings Can Support Retrieval
Separate embedding models are often used to represent sentences, paragraphs, images or documents so similar items can be retrieved by vector similarity. This is related to token embeddings but operates at a different representational scale.
A document embedding represents an entire text object, while a token embedding begins at one vocabulary identity. Mixing these levels creates conceptual confusion.
18. Representation Scale Must Be Named
Whenever someone says “the embedding,” ask: embedding of what? A token, sentence, passage, image, user, product, graph node or document? The object being represented determines what similarity and distance can reasonably mean.
This is the same fidelity discipline developed in Representation Fidelity | What Models Preserve, Omit and Distort.
19. Geometry Can Encode Useful Relations Without Human Labels
Learned spaces can support clustering, analogy-like structure or task-specific separations even when individual dimensions resist interpretation. Distributed representation means a property can be encoded across many dimensions rather than located in one slot.
This is powerful but also means visualising a few projected dimensions can oversimplify the full geometry.
20. Projection Is Another Representation Layer
When high-dimensional embeddings are reduced to two or three dimensions for a chart, the visualisation is a new representation. Techniques such as PCA, t-SNE or UMAP preserve different structures and can create different apparent clusters.
A pretty embedding plot therefore needs the same fidelity questions as any map: what was preserved, what was distorted, and what does distance mean after projection?
21. Contextual Similarity Can Differ From Lexical Similarity
Two different words may occupy similar contextual roles in one sentence, while two identical words may have very different contextual states in different sentences. The model’s useful geometry emerges from how the words function in context, not only how their surface forms look.
22. Embeddings Inherit Training Data
Learned vectors reflect patterns in the data and objective used to train them. If the training distribution contains biased, stale or incomplete associations, those patterns can influence the representation.
Embeddings should therefore be treated as learned statistical artefacts, not neutral maps of reality.
23. Embeddings Can Drift Across Models and Versions
The same token can have very different numerical representations in two independently trained models. Even retraining the same architecture can rotate or reorganise the latent space while preserving similar behaviour.
Raw embedding coordinates are therefore not portable universal meanings.
24. Model Upgrades Can Break Vector Compatibility
If a retrieval system stores vectors created by one embedding model and later switches query vectors to a different model, similarity may become meaningless unless the spaces are compatible. Production systems must version the embedding model alongside stored vectors.
This is analogous to versioning the tokenizer vocabulary.
25. Quantisation Changes the Numerical Representation
Models can store or compute representations using reduced numerical precision to save memory and accelerate inference. Quantisation changes how precisely numerical states are represented while aiming to preserve enough behaviour for the task.
This is another fidelity trade-off: smaller numerical representation in exchange for controlled approximation.
26. Tokens Can Share Statistical Structure Without Sharing IDs
Two tokens have different identifiers but can develop nearby or functionally related embeddings because training repeatedly places them in similar contexts. The ID says they are different entries. The learned geometry can say they behave similarly under some conditions.
Identity and similarity are therefore distinct relations.
27. One Token Can Participate in Many Meanings
A single vocabulary item may be polysemous. The initial embedding is shared, but context differentiates usage. This allows the model to avoid requiring a separate vocabulary token for every possible sense of every word.
The price is that meaning resolution moves into contextual computation.
28. One Meaning Can Be Expressed Through Many Tokens
Synonyms, paraphrases and translations can express similar ideas with different token sequences. The model therefore cannot rely on token identity alone if it is to recognise semantic equivalence.
Useful representation requires convergence across different surface forms.
29. Multilingual Models Make This Especially Visible
Two languages can express comparable meaning with very different scripts and token counts. Shared or aligned multilingual representations must bridge those surface differences through training.
This is explored in Multilingual Tokenisation | Why Equal Meaning Can Consume Unequal Tokens.
30. Embeddings Are Interfaces Between Symbol and Computation
Tokenisation converts source text into discrete symbolic identities. Embedding converts those identities into continuous numerical states. This bridge allows discrete human-readable sequences to enter a system built from numerical optimisation.
The embedding layer is therefore not the final meaning representation. It is the beginning of model-side computation.
31. Debugging Requires Layer Separation
If a model mishandles a string, several layers could be responsible: bad source extraction, surprising tokenisation, poor token frequency, weak learned embedding, insufficient context, architecture limits or output decoding. Blaming “the embedding” without locating the failure layer is too coarse.
Good debugging follows the representation chain.
32. The Token-ID Audit
- Which tokenizer and vocabulary produced the IDs?
- Is the vocabulary version exactly compatible with the model?
- Are special tokens mapped correctly?
- Are unknown or rare strings being decomposed as expected?
- Are IDs being passed in the correct sequence order?
- Are padding, masking and control tokens used correctly?
33. The Embedding Audit
- What object does the vector represent?
- Which model and version produced it?
- At which layer was it extracted?
- Is the vector static or contextual?
- What distance metric is being used?
- Are compared vectors from the same representational space?
- Was dimensionality reduction applied before visual interpretation?
- Has the representation been tested against real downstream outcomes?
34. What Students Should Remember
- A token ID identifies a vocabulary entry.
- An embedding is a learned numerical representation.
- Token IDs are not meaning coordinates.
- Initial embeddings are not final contextual meaning.
- Position matters because sequence order matters.
- Context transforms the representation.
- Similarity depends on the learned space and metric.
- Vectors inherit training data and model design.
35. The Deep Principle
The path from symbols to vectors is powerful because it turns discrete identities into mathematical objects that can interact. But mathematical manipulability should never be confused with direct access to reality.
Token IDs tell the model where to look. Embeddings give it a numerical place to begin. Context does the work of turning that beginning into a representation useful for this particular sequence.