Representation makes a world, message or problem available in another form. Tokenisation divides that representation into units a system can address, count, compare and transform.
The distinction is foundational. A representation asks what form will stand in for the source. Tokenisation asks where the working boundaries will be drawn inside that form. Every later operation inherits these choices.
World → Representation → Token → Context → World Return
The world does not arrive inside a model intact. It is observed, encoded, selected, simplified and bounded. A model therefore operates on a representation of a representation. The canonical eduKateSingapore owner for this wider idea is World Representation & Cognitive Tools | World → Representation → Receiver.
What Representation Does
A map stands in for geography. A graph stands in for relationships among quantities. A photograph stands in for a visual scene. An equation stands in for a mathematical relationship. A database record stands in for selected facts about an entity. None preserves everything. A useful representation preserves the distinctions needed by a receiver while suppressing details that would obstruct the task.
What Tokenisation Does
Tokenisation creates addressable units. In text systems these units may be characters, bytes, words, subwords, punctuation-associated fragments or special symbols. A token is not automatically a word and is not automatically an atom of meaning. It is a working unit produced by a boundary rule.
Why Boundaries Matter
Boundaries make indexing, storage, counting and transformation possible. But the boundary is an engineering hypothesis rather than proof of a natural division. Speech is acoustically continuous although writing encourages us to perceive separate words. Landscapes are continuous although maps divide them into parcels and jurisdictions. Learning is continuous although curricula divide it into subjects and lessons.
Never confuse the usefulness of a boundary with proof that the boundary is natural, unique or complete.
Why Subword Tokenisation Exists
A vocabulary containing every possible word becomes huge and still struggles with new names, compounds and technical forms. Character-level systems have broad coverage but produce longer sequences. Subwords occupy the middle ground: common patterns remain compact while rare forms decompose into reusable pieces. This design became influential in neural language processing through work such as Sennrich, Haddow and Birch (2016) and Kudo and Richardson’s SentencePiece (2018).
Representation Comes Before Tokenisation
Text already has representational structure before a tokenizer sees it. Unicode normalisation illustrates this: equivalent-looking strings can have different underlying code-point sequences. The Unicode Consortium specifies normalisation forms in Unicode Standard Annex #15. The route is therefore source → encoding → normalisation where appropriate → tokenisation → token identity → numerical representation → contextual processing.
Token Identity Is Not Meaning
A token ID is an address in a vocabulary, not a semantic definition. Its usefulness emerges when the system combines token identity with position, neighbouring units, learned parameters and task context. The visible word bank changes interpretation between river bank and central bank because context changes the working representation.
TOKEN ≠ MEANING TOKEN_ID ≠ MEANING NUMERICAL REPRESENTATION ≠ COMPLETE MEANING TOKEN + POSITION + CONTEXT + LEARNED PARAMETERS → CONTEXTUAL REPRESENTATION
The Layered Representation Stack
- world or source;
- observation or capture;
- human-visible representation;
- encoded characters or bytes;
- token sequence;
- token identifiers;
- numerical representation;
- contextual internal states;
- output distribution;
- decoded output;
- human interpretation;
- world return.
Calling every layer simply “the data” hides where information is transformed or lost. Strong analysis asks which layer created the distinction being observed.
What Can Be Lost
If a representation removes a crucial feature upstream, later reasoning cannot guarantee its recovery. Downsample an image until a mark disappears, discard a timestamp from a measurement, or remove tone and overlap from a transcript: downstream systems may infer what was missing, but inference is not recovery of preserved evidence.
Information removed upstream cannot be guaranteed downstream.
Tokenisation Changes Computational Cost
When a system has bounded sequence or context capacity, segmentation determines how much visible material fits. The same human-level amount of content can occupy different numbers of tokens depending on language, notation and tokenizer. There is therefore no universal words-to-tokens conversion.
Beyond Text
The same design problem appears in image patches, audio frames, biological sequences, event streams, music and structured records. Too coarse a unit hides internal structure. Too fine a unit creates long or noisy sequences. The useful unit lies between indivisible detail and unusable bulk.
Learning Is Also a Representation Problem
A novice often sees many isolated marks where an expert sees one meaningful chunk: an algebraic pattern, grammatical clause, circuit motif or biological pathway. Expertise changes the effective tokenisation of a problem. Good teaching therefore helps learners form better representations and better units inside those representations.
The Representation Test
- What is the source?
- Who is the receiver?
- What distinctions must survive?
- What has been omitted?
- Who chose the boundaries?
- Can the source be reconstructed?
- What errors become easier because of this representation?
- Can a second representation expose the blind spots?
- Does the result survive world return?
Series Route
- How Tokenisation Works | Boundaries, Vocabularies, Subwords and Reconstruction
- Representation Fidelity | What Models Preserve, Omit and Distort
- Why Token Boundaries Matter | Cost, Context, Language and Meaning
Representation makes the world workable. Tokenisation makes the representation addressable. Context makes the units useful. World return tests whether the chain deserved our trust.
More articles in this collection
Representation and tokenisation
- Causal Representation Learning | How Models Search for Latent Variables That Survive Interventions and Change
- Context Sufficiency | When Retrieved Evidence Is Enough to Answer
- Expert Routing | How Mixture-of-Experts Models Decide Which Parameters Process Each Token
- Hierarchical Representation | How Local Tokens Become Sentences, Sections, Scenes and Global Memory
- Latent Bottlenecks | How Models Compress Many Inputs Into a Small Working Representation
- Memory Representation | How Models Store, Compress, Forget and Retrieve Context
- Object-Centric Representation | How Scenes Become Persistent Objects, Slots, Relations and Composable World Models
- Positional Representation | How Models Know Where a Token Is
- Predictive Representation Learning | How Models Learn by Predicting Hidden Latent Structure Instead of Reconstructing Pixels
- Representation Collapse | Why Learned Embeddings Can Lose Diversity—and How Models Prevent It
- Retrieval Representation | How Queries and Documents Become Sparse, Dense and Hybrid Search Signals
- Set and Permutation Representation | How Models Work With Unordered Collections Without Inventing Sequence
- State-Space Representation | How Selective Recurrent State Carries Sequence History Without Full Attention
- Token Interaction Topology | How Attention Masks, Local Windows and Global Tokens Shape Context
- Uncertainty Representation | How Models Express Confidence, Ambiguity and When to Abstain
- World-State Representation | How Models Turn Observations and Actions Into Predictive Latent State