Two sentences can communicate the same human-level meaning and still consume very different numbers of model tokens.
This happens because tokenisation is not a universal linguistic law. It is a learned or engineered boundary system built from a finite vocabulary, a training corpus, an algorithm and a representation policy. Different languages and scripts interact with that system differently.
This article continues the eduKateSingapore Representation and Tokenisation series and extends the boundary analysis from Why Token Boundaries Matter.
The Main Idea
SAME HUMAN INTENT → DIFFERENT LANGUAGE / SCRIPT → DIFFERENT CHARACTER PATTERNS → DIFFERENT VOCABULARY COVERAGE → DIFFERENT TOKEN SEGMENTATION → DIFFERENT TOKEN COUNT → DIFFERENT CONTEXT AND COST FOOTPRINT
1. Meaning Is Not Measured in Tokens
Tokens are model-interface units, not units of human meaning. A short phrase in one language may map to more tokens than a longer-looking phrase in another. The number depends on how the tokenizer’s vocabulary represents the visible string.
This means token count cannot be used as a language-neutral measure of information without qualification.
2. Writing Systems Create Different Surface Patterns
Alphabetic, syllabic and logographic writing systems organise visible symbols differently. Languages also differ in spacing conventions, inflection, compounding and punctuation. A tokenizer trained on surface patterns must allocate its finite vocabulary across these different structures.
The result is that one vocabulary cannot represent every language with identical compactness.
3. Whitespace Is Not Universal Word Segmentation
Many English-oriented text pipelines historically treated spaces as convenient word boundaries. That assumption does not transfer cleanly to every language. Some writing systems do not mark word boundaries in the same way, and compounds or clitics can challenge the idea of a universal whitespace-delimited word.
This is one reason SentencePiece became influential: it can train subword models directly from raw sentences without requiring a language-specific pre-tokenisation stage that assumes spaces already define words.
4. Vocabulary Space Is Finite
A multilingual tokenizer has a fixed or bounded number of token identities. Those identities must cover recurring patterns across many languages and scripts. If one language occupies a large share of the training corpus, frequent strings from that language may receive many compact vocabulary entries.
Less represented patterns may be decomposed into smaller pieces, increasing sequence length.
5. Frequency Buys Compression
Subword tokenisers tend to give compact representation to recurring sequences. High-frequency patterns are more likely to become reusable vocabulary pieces. Rare patterns remain assembled from smaller components.
This means corpus frequency indirectly determines which linguistic forms receive representation efficiency.
6. Token Efficiency Is Not Language Quality
A language that uses more tokens under one tokenizer is not more complicated, less expressive or less efficient for humans. It is simply less compact under that particular computational representation.
Tokenizer efficiency describes the model interface, not the worth or sophistication of the language.
7. Equal Meaning Can Have Unequal Context Cost
If one translation requires 40 tokens and another requires 70, the second consumes more of a fixed context window even if both communicate comparable human-level content.
This creates a practical asymmetry in long documents, dialogue history, retrieval and prompt construction.
8. Equal Meaning Can Have Unequal Financial Cost
Where API usage, storage or compute scales with token volume, higher tokenisation density can translate into higher operating cost for the same human-level task.
This is an infrastructure property, not merely a billing detail. Product economics can inherit representation choices.
9. Context Windows Become Unequal in Visible Length
A fixed context limit provides the same number of model tokens to everyone, but not necessarily the same number of words, characters, sentences or pages. Users working in languages that require denser tokenisation may fit less visible material.
Equal token capacity is therefore not automatically equal document capacity.
10. Translation Benchmarks Need Token-Aware Interpretation
If one language’s inputs are consistently longer in model tokens, latency, memory use and context truncation can differ even when translation quality is similar. Benchmarking only accuracy can hide representation-efficiency differences.
A fuller evaluation should include token counts, visible content length and compute characteristics.
11. Character Count Is Not a Universal Fix
Comparing characters across scripts is also difficult because one visible symbol may carry different amounts of linguistic information, and Unicode code points do not always correspond one-to-one with user-perceived characters.
No single surface measure automatically solves cross-language comparison.
12. Word Count Is Not Universal Either
Languages differ in compounding and segmentation conventions. One language may express as one orthographic word what another expresses through several. Word counts therefore depend on linguistic convention as well as meaning.
This is why multilingual system evaluation should report multiple measures rather than rely on one universal denominator.
13. Morphology Changes Compression Opportunities
Highly inflected languages can generate many surface forms from one root. Subword tokenisation can share pieces across these forms, but how efficiently it does so depends on the learned vocabulary and whether the token boundaries align with recurring morphology.
A good subword system can reduce the whole-word vocabulary problem without guaranteeing perfect morphological segmentation.
14. Productive Compounding Stresses Whole-Word Vocabularies
Languages that readily create long compounds expose the limits of word-level tokenisation. Subwords help by representing familiar components inside previously unseen compounds.
This is a strong example of why compositional tokenisation matters across languages.
15. Proper Names Cross Language Boundaries Poorly
Names can be transliterated, translated, abbreviated or written in multiple scripts. A tokenizer may treat each surface form differently even when all refer to the same person or place.
Entity systems should therefore preserve canonical identity and aliases rather than rely only on token similarity.
16. Transliteration Adds a Representation Layer
When text moves from one script into another, transliteration changes the surface representation before tokenisation. The resulting token count and segmentation can change dramatically.
Transliteration can improve interoperability in one system while losing orthographic or phonological distinctions in another. It is a fidelity trade-off, not a neutral copy.
17. Unicode Normalisation Matters Across Scripts
Equivalent-looking text can have different underlying Unicode sequences. Normalisation can reduce representational variation before tokenisation. The Unicode Consortium formalises these transformations in Unicode Standard Annex #15.
In multilingual systems, normalisation policy should be tested across scripts rather than assumed safe from English examples.
18. Diacritics Can Carry Meaning
Removing accents or diacritics may simplify matching in some tasks but can collapse distinctions in others. What appears like harmless normalisation from one language perspective may alter words, pronunciation or identity in another.
Representation design must be language-aware.
19. Case Folding Is Not Universally Harmless
Lowercasing can improve vocabulary sharing, but case carries information in names, acronyms and some writing systems. Case transformations also behave differently across languages.
Normalisation should preserve distinctions that matter to the downstream receiver.
20. Punctuation Conventions Vary
Quotation marks, decimal separators, thousands separators, sentence punctuation and spacing conventions differ internationally. Tokenisation therefore interacts with locale-specific formatting even when the underlying concept is the same.
Numbers and dates are especially vulnerable because visually familiar strings can be interpreted differently across locales.
21. Dates Need Locale Context
A date such as 04/09/2026 can represent different calendar conventions. Tokenisation preserves the visible symbols but cannot determine the intended date without contextual information.
Multilingual representation therefore needs semantic metadata in addition to token boundaries.
22. Shared Subwords Can Enable Cross-Language Transfer
Related languages, borrowed words, digits, names and shared scripts can reuse vocabulary pieces. Shared subwords may provide statistical bridges across languages.
But cross-language transfer also depends on training objective and model architecture; shared tokens alone do not guarantee aligned meaning.
23. Different Scripts Can Still Converge Internally
Two translations may begin from completely different token identities but still develop similar task-relevant internal representations after contextual processing. This is one reason multilingual models can perform cross-language tasks despite surface differences.
The representation stack therefore separates surface form from higher-level contextual structure.
24. Token Efficiency Can Influence Latency
Longer token sequences can require more sequential or attention computation depending on architecture and implementation. Even when response quality is equal, denser tokenisation can affect latency.
Multilingual product testing should therefore measure speed by language, not assume one global latency profile.
25. Token Efficiency Can Influence Energy Use
At scale, extra computation has energy and infrastructure consequences. A tokenizer that produces systematically longer sequences for some languages can increase resource use across millions of requests.
Representation efficiency therefore participates in sustainability as well as cost.
26. Token Efficiency Can Influence Retrieval Quality
If retrieval chunks are sized by tokens, languages with denser tokenisation may fit less visible text per chunk. That can change how often definitions, examples and qualifiers remain together.
Chunking strategies should therefore be validated multilingually.
27. Fixed Token Chunks Are Not Fixed Semantic Chunks
A 500-token chunk may represent a short passage in one language and a longer passage in another. Equal numerical chunk size does not imply equal semantic coverage.
Document structure should help determine chunk boundaries alongside token counts.
28. Multilingual Prompt Templates Need Real Testing
Translating an English prompt into another language can change token count, instruction order and ambiguity. The translated prompt may also interact differently with examples and system constraints.
Prompt engineering should therefore test meaning fidelity and token behaviour together.
29. Language Mixing Adds Another Boundary Problem
Real users switch languages inside one sentence, combine English with local expressions, insert code, numbers or names and use transliterated speech. Tokenizers must handle these mixed sequences without assuming one clean language label per input.
Code-switching is a practical stress test for multilingual representation.
30. Dialect and Informal Spelling Can Fragment More
Non-standard spelling, dialect forms, phonetic writing and chat abbreviations may appear less frequently in tokenizer training data. They can therefore be decomposed into more pieces even when they are perfectly familiar to human communities.
Representation efficiency reflects corpus history, not linguistic legitimacy.
31. Evaluation Must Separate Quality From Efficiency
A model can produce equally good answers across two languages while requiring more tokens in one. Conversely, efficient tokenisation does not guarantee accurate output. Quality and efficiency are separate axes.
Good multilingual evaluation reports both.
32. Compare Normalised Cost Per Task
When operating systems at scale, compare the cost of completing equivalent user jobs rather than only price per token. A task-level measure can reveal whether representation differences create systematic economic asymmetries.
This is more meaningful than assuming one token is the same amount of human work across languages.
33. Compare Context Retention Per Task
Ask how much relevant history survives before truncation for users in each language. A fixed model context can yield different conversational depth if token density varies.
Long-running assistants should test this explicitly.
34. Compare Retrieval Coverage Per Task
For multilingual retrieval, measure whether equivalent documents preserve comparable structural units inside chunks. If one language regularly splits headings from paragraphs or examples from definitions because of token density, retrieval quality can diverge.
Token-aware chunking is necessary but not sufficient; structure-aware chunking matters too.
35. Multilingual Fairness Includes Representation Efficiency
Fairness discussions often focus on output quality and harmful bias. Representation efficiency adds another dimension: how much context, compute and cost a user receives for the same human-level task.
This does not make token parity a universal fairness requirement, but it makes asymmetry measurable and therefore designable.
36. The Multilingual Tokenisation Audit
- Which languages and scripts does the product serve?
- What real token-per-character and token-per-word distributions occur?
- How much visible text fits into the context for each language?
- How do prompt templates change after translation?
- Which languages fragment names and technical terms heavily?
- How are mixed-language inputs handled?
- Do normalisation rules preserve important diacritics and script distinctions?
- How does token density affect retrieval chunk size?
- How does it affect latency and cost per completed task?
- Are evaluation datasets balanced by task rather than raw token count?
- Does output quality remain comparable?
- Are users paying an unintended representation penalty?
37. What Students Should Remember
- Tokens are not universal units of meaning.
- Different languages can require different token counts for equivalent ideas.
- Vocabulary frequency influences compression.
- Equal token limits do not guarantee equal visible-text capacity.
- Efficiency and quality are different measures.
- Normalisation and script handling matter.
- Multilingual systems need language-specific testing.
38. The Deep Principle
A multilingual model can share one computational system across many languages, but it cannot erase the fact that those languages arrive through different surface representations. The tokenizer must distribute a finite vocabulary across that diversity.
Equal meaning does not imply equal token cost because tokens measure the representation chosen by the system, not the amount of human meaning carried by the language.