A tokenizer should not be judged by how neat its pieces look. It should be judged by whether its representation is complete, efficient, reversible where required, balanced across real workloads and useful for the tasks the model must perform.

That requires more than one metric. A tokenizer can achieve excellent compression while fragmenting names badly. It can provide universal coverage while consuming too much context in some languages. It can round-trip text perfectly while still creating inconvenient boundaries for code or mathematics.

This article continues the eduKateSingapore Representation and Tokenisation series and turns the previous articles into an evaluation framework.

The Evaluation Stack

SOURCE REPRESENTATION
→ COVERAGE
→ NORMALISATION FIDELITY
→ SEGMENTATION QUALITY
→ TOKEN DENSITY
→ ROUND-TRIP RECONSTRUCTION
→ CONTEXT EFFICIENCY
→ MULTILINGUAL / DOMAIN PARITY
→ DOWNSTREAM TASK PERFORMANCE
→ WORLD RETURN

1. Start With the Real Workload

A tokenizer benchmark should represent the application it will serve. If users submit English prose, Mandarin, Malay, Tamil, code, URLs, equations and names, the test corpus must contain those forms.

Benchmarking only clean English paragraphs can produce a beautifully measured tokenizer that is badly matched to reality.

2. Separate Tokenizer Quality From Model Quality

Tokenization determines units and sequence length. The model learns representations and behaviours above those units. A tokenizer can be efficient while the model remains weak; a model can perform well despite awkward token boundaries.

Evaluation should therefore include tokenizer-only metrics and end-to-end model metrics without confusing them.

3. Coverage Is the First Hard Test

Can every valid input in the target workload be represented? Character, subword and byte-level systems differ in how they achieve coverage. Unknown-token rates are a direct warning when unsupported strings collapse into one generic identity.

For general-purpose systems, broad coverage is usually non-negotiable.

4. Unknown-Token Rate Should Approach the Design Target

If a tokenizer uses an unknown token, measure how often it appears by language and domain. A near-zero global average can hide severe failures for a small user population or specialist notation.

Always break aggregate metrics down by receiver group and workload type.

5. Token Count Is the Simplest Efficiency Metric

Apply the tokenizer to a representative corpus and count output tokens. Fewer tokens usually mean more compact representation under that tokenizer.

But raw token count alone says nothing about what was lost, whether languages are treated evenly or whether the segmentation helps downstream tasks.

6. Characters per Token Measures Surface Compression

Characters per token provides an intuitive view of how much visible text each token represents on average. Higher values indicate more compact segmentation.

This metric is useful within a language or corpus family, but scripts differ in how much linguistic content is carried by one character.

7. Bytes per Token Adds an Encoding-Level View

Bytes per token compares sequence length with encoded source size. It can be useful across Unicode text because bytes provide a machine-level denominator.

It remains a representation-efficiency measure rather than a semantic-information measure.

8. Token Fertility Measures Fragmentation

Token fertility is commonly used to describe how many tokenizer pieces correspond to a reference unit such as a word. A higher number means greater fragmentation.

Because word segmentation differs across languages, define the denominator carefully and supplement it with character- or byte-based measures.

9. Distribution Matters More Than the Mean

Average tokens per word can hide long tails. Most words may be compact while rare names, specialist terms or URLs explode into many pieces.

Report percentiles and maximums, not just means.

10. Benchmark Names Separately

Create a test set of personal names, place names, organisations and culturally diverse names. Measure fragmentation, round-trip fidelity and exact-copy performance.

Identity-sensitive strings deserve their own benchmark because one spelling error can matter more than several stylistic errors in ordinary prose.

11. Benchmark Numbers Separately

Test integers, decimals, percentages, currencies, dates, scientific notation and long identifiers. Measure how boundaries align with digit groups and how the downstream model performs on copying and arithmetic.

Tokenisation of numbers should be evaluated both structurally and behaviourally.

12. Benchmark Code Separately

Code includes identifiers, operators, punctuation, indentation and domain-specific repetition. Compare token density across programming languages and test exact reconstruction.

Then evaluate whether alternative tokenizers change code-completion accuracy, compile rate or context utilisation.

13. Benchmark URLs and Paths

URLs, filenames and paths stress tokenizers with punctuation and rare substrings. Measure how many tokens they consume and whether source strings can be copied exactly.

These cases often matter in practical assistants even if they are a small fraction of training text.

14. Benchmark Emoji and Grapheme Clusters

Emoji, combining characters and zero-width joiner sequences reveal whether a tokenizer handles user-perceived characters robustly. Test both segmentation and detokenisation.

Visual equality alone is insufficient; compare code points or normalized forms when exactness matters.

15. Round-Trip Fidelity Is a Fundamental Test

Run encode → decode across the corpus. Compare reconstructed output with the intended invariant: exact bytes, code points, canonical Unicode equivalence or rendered text.

See Detokenisation and Reconstruction.

16. Normalisation Must Be Included in the Evaluation Receipt

A tokenizer can appear non-reversible because the preprocessing pipeline intentionally normalises text before segmentation. Evaluate the whole encode–decode contract rather than one stage in isolation.

See Text Normalisation Before Tokenisation.

17. Measure Context Efficiency Directly

Take representative documents and prompts, then calculate how much visible material fits into a fixed context budget. This turns tokenizer compression into a product-level measure.

A tokenizer that saves 15% of tokens may effectively provide 15% more document capacity for the same model context, subject to protocol overhead and output reservation.

18. Measure Hidden Protocol Overhead

Chat templates, role tokens and separators can add model tokens that users never see. Include them when measuring real context utilisation.

See Special Tokens and Control Tokens.

19. Evaluate Multilingual Token Density by Language

Measure characters per token, tokens per sentence and tokens per task across each served language. Report the distribution rather than a single global average.

This exposes representational asymmetries that can affect context, latency and cost.

20. Compare Equivalent Translations

Use professionally aligned or carefully curated translations of the same content. Compare token counts across languages while remembering that no translation is perfectly identical in information structure.

The goal is practical parity, not a metaphysical measurement of meaning.

21. Measure Cost per Completed Task

If inference cost scales with token volume, compare how many tokens are required to complete equivalent tasks across languages and domains.

This is more meaningful than price per token when representation density varies.

22. Measure Latency per Completed Task

Longer sequences can increase processing time. Benchmark end-to-end latency across the same task categories under alternative tokenizers where model compatibility permits.

Representation efficiency becomes user experience.

23. Measure Truncation Risk

For long documents and conversations, calculate how often important information falls outside the context because of token density. Compare by language and domain.

Token efficiency can influence memory quality indirectly.

24. Benchmark Retrieval Chunk Coverage

If chunks are sized by tokens, measure how many sentences, paragraphs or logical units fit into a chunk under each language and document type.

A tokenizer that fragments one language heavily can change retrieval coherence even before embedding quality is considered.

25. Benchmark Boundary-Crossing Questions

Create questions whose answer depends on information near chunk or token boundaries: a rule plus exception, a table header plus row, a compound technical term or a code identifier split into several pieces.

Adversarial boundary tests reveal weaknesses hidden by average cases.

26. Measure Vocabulary Utilisation

Count how often each vocabulary entry appears in the training or evaluation corpus. A huge tail of almost-unused tokens may indicate inefficient vocabulary allocation.

Utilisation helps explain whether a larger vocabulary is earning its embedding cost.

27. Measure Frequency Concentration

A small number of tokens often account for a large share of sequence positions. Understanding this distribution helps identify whether common forms are represented compactly and whether many entries provide little benefit.

Vocabulary economics should be measured empirically.

28. Measure Special-Token Reliability

Test whether beginning markers, separators, roles and end tokens serialize correctly in every supported chat or tool configuration. A tokenizer can score perfectly on ordinary prose while failing the model’s protocol.

Interface correctness is part of tokenizer correctness.

29. Measure Tokenizer–Model Compatibility Explicitly

Verify that token IDs map to the embedding rows expected by the deployed model. This sounds obvious, but version mismatch is catastrophic because all other tokenizer metrics become irrelevant once identities are scrambled.

Compatibility is a hard gate, not a quality score.

30. Compare Alternative Vocabulary Sizes

When designing a tokenizer, train several vocabulary sizes and compare sequence length, embedding-table cost, fragmentation and downstream model results.

The best vocabulary size is a system optimum, not the maximum possible vocabulary.

31. Compare Algorithm Families Under the Same Corpus

BPE, WordPiece and Unigram-style methods can produce different boundaries even when trained on similar data. Controlled comparison helps separate algorithm effects from corpus effects.

Use the same normalization and evaluation suite where possible.

32. Compare Normalisation Policies

Train or evaluate tokenizers under alternative case, Unicode and whitespace policies. Measure compression gains against the information distinctions that are lost.

A normalisation policy is justified only when its savings are worth its fidelity cost.

33. Evaluate Downstream Accuracy

Ultimately, tokenization exists to support model tasks. Compare downstream performance on language modelling, translation, retrieval, code, arithmetic or other relevant objectives.

A tokenizer that is elegant in isolation but harms the task has failed its receiver.

34. Evaluate Exact Copy Separately From General Generation

Names, citations and identifiers require exact reproduction. Measure character-perfect accuracy rather than semantic similarity.

This benchmark often exposes fragmentation costs hidden by ordinary text metrics.

35. Evaluate Arithmetic Separately From Text Fluency

A model can write fluent explanations while mishandling long numbers. Token boundaries around digits can interact with arithmetic learning in ways not visible from language-generation scores.

Use controlled numeric test sets.

36. Evaluate Code Parse and Compile Rates

For code models, compare whether generated output parses, compiles or passes tests. Token-level compression is only useful if structural correctness survives.

See Segmentation vs Parsing.

37. Evaluate Robustness to Noisy Input

Test typos, repeated characters, mixed casing, unusual whitespace, malformed Unicode, OCR errors and informal chat. Tokenizers encounter messy real-world representations, not only edited corpora.

Robustness means graceful representation under imperfection.

38. Evaluate Distribution Shift

A tokenizer trained on one era or domain may face new terminology later. Benchmark current vocabulary, emerging technologies and newly important names to see whether fragmentation has increased.

Tokenizer quality can drift as the world changes.

39. Evaluate Before and After Re-Tokenisation

If considering a tokenizer migration, process the same historical dataset with both versions. Compare token counts, source offsets, special-token handling and downstream model compatibility.

A migration changes the searchable and model-ready representation of the corpus.

40. Keep a Reproducible Evaluation Receipt

Record corpus versions, tokenizer files, normalization settings, special-token configuration, code version and metrics. Without this, benchmark numbers cannot be reproduced after a tokenizer update.

Evaluation is part of edition control.

41. Do Not Optimise One Metric to Destruction

A tokenizer optimised only for shortest sequences may choose very large pieces and sacrifice compositional sharing. One optimised only for vocabulary size may create extremely long sequences.

Representation quality is multi-objective.

42. Build a Scorecard, Not a Single Score

43. The Tokenizer Benchmark Suite

1. clean general prose;
2. all supported languages and scripts;
3. proper names and organisations;
4. numbers, dates and currencies;
5. source code;
6. URLs, email addresses and file paths;
7. mathematical notation;
8. emoji and complex Unicode;
9. dialect and informal chat;
10. OCR and noisy text;
11. maximum-context documents;
12. chat-template and special-token serialization;
13. encode–decode round trips;
14. retrieval chunks;
15. downstream task evaluations.

44. What Students Should Remember

• No single metric defines a good tokenizer.
• Coverage, efficiency, fidelity and task fit are separate dimensions.
• Average token counts can hide rare but important failures.
• Multilingual and domain-specific evaluation is essential.
• Round-trip reconstruction should be tested explicitly.
• Context capacity turns token efficiency into a product-level issue.
• Downstream performance is the final receiver test.

45. The Deep Principle

A tokenizer is good when its boundaries make the intended work easier without hiding unacceptable losses. That judgement requires several measures because representation quality is never one-dimensional.

Measure whether the tokenizer can represent the world, how much capacity that representation consumes, what distinctions survive the round trip and whether the downstream receiver performs better because of the choices made.

Representation & Tokenisation Series

• Token Vocabulary Design | How Systems Choose Which Pieces Deserve Their Own Identity
• Text Normalisation Before Tokenisation | What Should Be Preserved, Collapsed or Canonicalised?
• Byte-Level Tokenisation | How Bytes Give Models a Route to Almost Any Text
• Canonical owner: World Representation & Cognitive Tools