Tokenisation rarely fails by producing no tokens at all. More often, it fails by producing a technically valid representation that is inefficient, misleading, brittle or badly matched to the downstream task.
This matters because many tokenisation problems hide behind apparently successful model inputs. The system accepts the sequence, the model runs, the decoder returns text—and yet names are misspelled, numbers drift, code becomes fragile, one language consumes far more context than another, or an important structure is split across boundaries.
This article continues the eduKateSingapore Representation and Tokenisation series.
The Failure Model
SOURCE → ENCODING → NORMALISATION → TOKEN BOUNDARIES → TOKEN IDS → MODEL CONTEXT → OUTPUT → DETOKENISATION → RECEIVER FAILURE CAN ENTER AT ANY TRANSFORMATION
1. Failure Mode: Excessive Fragmentation
A rare word, specialist term or unfamiliar script can be split into many small pieces. The text remains representable, but sequence length increases and the model must integrate more positions to recover the larger unit.
Fragmentation is not automatically harmful. It becomes a problem when it systematically wastes capacity or degrades exact handling of important strings.
2. Failure Mode: Overly Coarse Units
The opposite problem occurs when a vocabulary uses large units that hide useful internal structure. A whole technical string may be compact but share little representational structure with related forms.
Good tokenisation balances compactness with compositional reuse.
3. Failure Mode: Unknown-Token Collapse
Older or restricted vocabularies may map unsupported strings into one generic unknown token. Distinct inputs then become indistinguishable at the token layer.
This is a severe information loss because the original spelling cannot be reconstructed from the unknown token alone.
4. Failure Mode: Name Fragmentation
Uncommon personal names, place names and organisation names can split into several pieces. A model may understand the surrounding context yet still reproduce the name incorrectly.
When exact identity matters, preserve the source string and use canonical identifiers rather than relying on free regeneration.
5. Failure Mode: Numerical Fragmentation
Numbers can be segmented in ways that do not align with place value, decimal structure or mathematical units. The model must learn arithmetic relationships through a representation that was optimised for text statistics rather than mathematics.
This can make exact numerical copying or manipulation more brittle even when the tokenisation is fully reversible.
6. Failure Mode: Date Ambiguity Survives Perfect Tokenisation
04/09/2026 can be tokenised and reconstructed perfectly while remaining ambiguous across date conventions. Tokenisation preserves symbols; it does not automatically preserve locale semantics.
The missing information is contextual, not lexical.
7. Failure Mode: Unicode Equivalence Produces Different Sequences
Visually equivalent Unicode strings can have different underlying code-point sequences. Without an appropriate normalisation policy, they may receive different tokenisations.
Unicode normalization is formalised in Unicode Standard Annex #15. The correct policy depends on which distinctions the application must preserve.
8. Failure Mode: Harmful Normalisation
Normalisation can solve variation while creating new losses. Lowercasing can erase acronyms or case-sensitive identifiers. Removing diacritics can merge distinct words or names. Whitespace cleanup can break code or tabular structure.
A preprocessing improvement for one domain can be a fidelity failure in another.
9. Failure Mode: Whitespace Misinterpretation
Spaces and line breaks are not always decorative. They can separate words, define indentation, format tables or encode document structure. Tokenisers and preprocessors that treat whitespace casually can damage downstream meaning.
Code and fixed-width data are especially sensitive.
10. Failure Mode: Punctuation Fragmentation
Punctuation carries structure in prose, mathematics, URLs, file paths, code and markup. A tokenizer can represent every symbol correctly while still creating long, awkward sequences around punctuation-dense inputs.
Efficiency and reversibility must be evaluated separately.
11. Failure Mode: Code Tokenisation Mismatch
A tokenizer developed primarily on prose may represent source code less efficiently than one whose vocabulary includes frequent programming patterns. Identifiers, operators and indentation can fragment heavily.
This does not prove the model cannot code, but it can increase sequence length and exact-string difficulty.
12. Failure Mode: URL and Path Explosion
URLs, query strings and file paths combine domains, slashes, punctuation, numbers and identifiers. One human-level address can become many model tokens.
Long exact strings should be handled through source-preserving mechanisms when possible.
13. Failure Mode: Multilingual Token Density
A shared multilingual vocabulary can represent some languages more compactly than others. Equivalent human-level content may consume different numbers of tokens.
This affects context capacity, latency and cost. See Multilingual Tokenisation.
14. Failure Mode: Dialect and Informal Text Fragmentation
Dialect spellings, phonetic chat, abbreviations and community-specific forms may appear less often in tokenizer training data and therefore receive more fragmented representation.
Representation efficiency reflects corpus history, not linguistic legitimacy.
15. Failure Mode: Emoji and Grapheme Splitting
A user-perceived emoji can contain several Unicode code points joined into one displayed symbol. Token boundaries can fall inside this underlying sequence.
Decoding may still reconstruct the emoji correctly, but partial streaming or malformed sequences can expose the complexity.
16. Failure Mode: Tokenizer–Model Mismatch
This is one of the most serious configuration errors. If token IDs are generated using a vocabulary different from the one expected by the model, valid integers can point to the wrong learned embeddings.
The model then receives structurally legal but semantically scrambled input.
17. Failure Mode: Version Drift
A tokenizer upgrade can alter vocabulary, normalisation or special-token behaviour. Historical token counts and model inputs may change even when visible source text remains identical.
Production systems should version tokenizers explicitly.
18. Failure Mode: Incorrect Special-Token Serialization
If sequence boundaries, role tokens or separators are missing, duplicated or misplaced, a chat or tool-using model can misinterpret the structure of the conversation.
See Special Tokens and Control Tokens.
19. Failure Mode: Hidden Protocol Injection
If raw user text can imitate or inject reserved protocol syntax, content and control may become confused. Safe serialization keeps trusted structure separate from untrusted content.
This is an interface security problem as much as a tokenisation problem.
20. Failure Mode: Excessive Token Overhead
Chat templates, role markers, repeated headers and metadata can consume context without adding equivalent user-visible content. Hidden overhead becomes significant in long conversations or repeated tool loops.
Measure the serialized sequence rather than estimating from visible words.
21. Failure Mode: Context Budget Waste
Fragmented tokenisation can cause the same source material to occupy more of a fixed context window. That can force earlier truncation of conversation history or retrieved evidence.
A tokenizer failure can therefore appear later as a memory or retrieval failure.
22. Failure Mode: Bad Chunk Size From Token Assumptions
Document chunking often uses token counts. If a system assumes one language or content type has a universal words-per-token ratio, chunks can become too short or too long in other domains.
Chunking should use the actual tokenizer and preserve document structure. See Token Budgets and Chunking.
23. Failure Mode: Boundary-Crossing Retrieval
A definition can sit in one chunk while its exception sits in the next. Tokenisation itself may be perfect, but token-count-driven chunk boundaries can still damage the larger representation.
Boundary design exists at several scales.
24. Failure Mode: Exact-Copy Drift
A model can understand an entity while reproducing its identifier incorrectly. This often matters for citations, URLs, filenames, phone numbers, codes and account strings.
Exact-copy tasks should use constrained mechanisms or direct source transfer whenever possible.
25. Failure Mode: Detokenisation Artifacts
Decoding can introduce unwanted spaces, strip protocol markers incorrectly or mishandle byte sequences if the wrong decoder is used.
See Detokenisation and Reconstruction.
26. Failure Mode: Streaming Partial Characters
Streaming one token at a time can expose incomplete byte or character sequences if the display layer assumes each token corresponds to a self-contained visible unit.
Streaming decoders need buffering rules consistent with the tokenizer.
27. Failure Mode: Mistaking Tokens for Linguistic Units
A subword boundary does not prove a morpheme boundary. A one-token phrase does not prove semantic unity. Treating tokenizer output as linguistic analysis creates false conclusions.
Segmentation and parsing are different layers.
28. Failure Mode: Mistaking Token Count for Information Amount
Ten tokens in one language or domain may carry a different amount of human-visible content from ten tokens in another. Token count measures representation length under one tokenizer, not universal information value.
Cost and fairness analyses should compare completed tasks as well as raw token totals.
29. Failure Mode: Mistaking Efficient Tokenisation for Understanding
A common word can be represented by one token and still be misunderstood in context. A rare technical word can use several tokens and still be handled accurately.
Representation efficiency and model competence are distinct variables.
30. Failure Mode: Mistaking Embedding Problems for Tokenisation Problems
A tokenizer may segment an input sensibly while the learned model representation remains weak because training evidence was sparse or misleading. Debugging must separate boundary formation from learned representation.
31. Failure Mode: Mistaking Context Problems for Token Problems
The token bank is not defective because it is ambiguous. Ambiguity belongs to language. Contextual processing must resolve which sense applies.
See Contextual Representation.
32. Failure Mode: Silent Truncation
If a long token sequence exceeds the permitted input length, systems may truncate from the beginning, end or another region. The tokenizer has worked correctly, but important information disappears at the context boundary.
Truncation policy should be explicit and tested.
33. Failure Mode: Wrong Side Truncation
Keeping the first part of a document may drop the conclusion. Keeping the last part may drop definitions and instructions. Conversation systems can lose the system prompt, old decision or latest user request depending on the policy.
There is no universally safe truncation direction.
34. Failure Mode: Token Counting With the Wrong Tokenizer
A planner can estimate context using one tokenizer while the serving model uses another. The predicted budget then becomes inaccurate and requests fail or truncate unexpectedly.
Budget with the tokenizer attached to the actual deployed model.
35. Failure Mode: Testing Only English Prose
A tokenizer can look excellent on clean English sentences while behaving poorly on multilingual text, code, equations, URLs and user-generated content.
Evaluation must represent the real workload.
36. Failure Mode: No Round-Trip Tests
If encode → decode is never tested, basic reconstruction errors can survive into production. Exact invariants should be checked at byte, code-point, canonical-equivalence or rendered-text level as appropriate.
Round-trip testing is cheap compared with debugging corrupted output downstream.
37. Failure Mode: No Offset Mapping
When a tokenizer does not preserve offsets back to source spans, debugging named entities, highlights and annotations becomes harder. The transformed representation loses its easy route back to the source.
Traceability is a form of representation safety.
38. Failure Mode: No Provenance Around Tokenised Data
Token sequences can be stored without source document identity, date or preprocessing version. Later analysts then cannot reproduce how the sequence was created.
Tokenisation should produce a receipt: source, tokenizer version, normalisation policy and transformation metadata.
39. Failure Mode: Optimising Token Count Alone
A tokenizer can be more compact while creating worse task boundaries. Compression is useful only when it preserves the structure the model needs.
Lowest token count is not the same as highest representation quality.
40. Failure Mode: Ignoring the Receiver
A tokenizer optimised for general prose may be wrong for genomic sequences, source code or mathematical notation. The correct representation depends on what the downstream receiver must do.
Universal tokenisation is an engineering compromise, not a metaphysical optimum.
41. A Proper Tokenisation Test Suite
- ordinary prose;
- rare personal and place names;
- long integers and decimals;
- dates and times;
- URLs and file paths;
- source code;
- mathematical expressions;
- mixed scripts;
- emoji and combining characters;
- unusual whitespace;
- dialect and informal chat;
- special-token-like user strings;
- maximum-length inputs;
- round-trip reconstruction;
- multilingual token-density comparisons.
42. The Failure-Diagnosis Ladder
SOURCE ERROR? ↓ ENCODING ERROR? ↓ NORMALISATION ERROR? ↓ TOKEN BOUNDARY ERROR? ↓ TOKENIZER–MODEL MISMATCH? ↓ CONTEXT / TRUNCATION ERROR? ↓ MODEL REPRESENTATION ERROR? ↓ GENERATION ERROR? ↓ DETOKENISATION / RENDERING ERROR?
Diagnose the earliest failing layer. Repairing a later layer cannot reliably restore information lost upstream.
43. The Deep Principle
Tokenisation failures are often not catastrophic crashes. They are small representational mismatches that accumulate: an extra split here, a lost distinction there, a context budget quietly consumed, a name made harder to copy.
A tokeniser can be technically correct and still be wrong for the job. The real test is whether its boundaries preserve the distinctions the receiver needs at acceptable cost.