A tokenizer vocabulary is a finite inventory of pieces that a model can address directly. Designing that inventory means deciding which recurring forms deserve their own identity and which should be composed from smaller units.

This is not just a list-making exercise. Vocabulary design changes sequence length, coverage, memory, training efficiency, multilingual balance, handling of code and numbers, compatibility with model weights and the kinds of patterns the system can represent compactly.

This article continues the eduKateSingapore Representation and Tokenisation series and sits beside How Tokenisation Works.

The Design Problem

TRAINING CORPUS
→ NORMALISATION POLICY
→ BASE UNITS
→ FREQUENCY / PROBABILITY MODEL
→ VOCABULARY SELECTION
→ TOKEN IDS
→ MODEL EMBEDDINGS
→ SEQUENCE LENGTH / COVERAGE / COST
→ DOWNSTREAM PERFORMANCE

1. A Vocabulary Is a Compression Dictionary

A tokenizer vocabulary gives short symbolic identities to recurring patterns. Common pieces can be represented directly. Less common strings are assembled from smaller pieces. The vocabulary therefore acts like a compression dictionary tuned to the corpus from which it was learned.

The more useful patterns the vocabulary captures, the shorter many sequences become. But every additional entry consumes model capacity because each token identity needs an associated representation.

2. Vocabulary Size Creates a Core Trade-Off

A larger vocabulary can encode common words and phrases more compactly. A smaller vocabulary forces more composition from subword or byte pieces. Neither extreme is universally best.

LARGER VOCABULARY
+ shorter sequences for frequent forms
+ more whole words and domain strings
− more embedding rows
− more sparse rare entries

SMALLER VOCABULARY
+ fewer token identities
+ strong compositional coverage
− longer sequences
− more positions needed for the same visible content

3. Vocabulary Entries Are Not Concepts

A vocabulary entry is a recurring surface unit selected for computational usefulness. It is not automatically a concept, morpheme or semantic atom. A token can represent part of a word, punctuation plus surrounding space, a byte-derived string or a special protocol marker.

This is one of the central representation rules: compact identity does not prove conceptual unity.

4. Corpus Frequency Shapes What Gets a Direct Identity

Subword vocabulary construction usually rewards recurring sequences. Patterns that appear often enough can become economical units. Rare strings remain decomposed.

Frequency therefore buys representational privilege: common patterns receive shorter routes through the model.

5. The Training Corpus Is Part of Vocabulary Design

A tokenizer trained on news, literature, source code and scientific papers will learn a different inventory from one trained mainly on casual conversation. The corpus determines which strings appear common enough to justify dedicated pieces.

Vocabulary design is therefore inseparable from corpus composition.

6. Domain Balance Changes Representation Efficiency

If a corpus contains substantial programming code, repeated programming fragments may receive compact token identities. If biomedical literature is common, scientific morphemes and terminology may become more economical. If those domains are scarce, the same strings will fragment more heavily.

Domain representation begins before model training proper: it begins in the tokenizer’s vocabulary.

7. Multilingual Vocabularies Allocate Scarce Space

A multilingual vocabulary must distribute finite entries across many scripts and languages. Patterns common in high-resource languages may receive compact identities while lower-frequency languages use smaller pieces.

This contributes to unequal token density across languages, explored in Multilingual Tokenisation.

8. Coverage and Compression Are Different Objectives

Coverage asks whether every relevant input can be represented. Compression asks how economically it can be represented. Byte-level approaches can achieve extremely broad coverage while still producing long sequences for unfamiliar patterns.

A vocabulary can therefore have perfect coverage and uneven efficiency.

9. Unknown Tokens Represent Coverage Failure

When a tokenizer maps unsupported text to a generic unknown token, different source strings collapse into the same identity. This destroys reconstructability and makes exact handling impossible.

Subword and byte-level methods became attractive partly because they reduce or eliminate this failure mode.

10. Base Units Determine the Floor of Decomposition

A tokenizer may begin from characters, bytes or another primitive alphabet. These base units determine the lowest level to which an unseen string can be decomposed.

The choice defines the ultimate coverage guarantee and affects how multilingual scripts, unusual symbols and malformed text are handled.

11. BPE Builds Upward From Smaller Units

Byte Pair Encoding-style tokenisation repeatedly merges frequent neighbouring units. This creates larger pieces where the corpus provides enough repetition to justify them.

Sennrich, Haddow and Birch’s influential 2016 work on subword units for neural machine translation demonstrated why compositional vocabularies are useful for rare and unseen words.

12. Unigram Selects Useful Pieces Probabilistically

Unigram approaches can begin from a large candidate vocabulary and remove pieces while preserving a probabilistic model of likely segmentations. This creates a different route to the same general objective: a compact inventory that represents the corpus efficiently.

The vocabulary is optimised, not discovered as a natural linguistic truth.

13. SentencePiece Separates Vocabulary Learning From Language-Specific Word Splitting

SentencePiece treats raw text as the starting point and can train language-independent subword models without requiring a pre-existing whitespace word segmentation.

This matters because “word boundary” is not represented identically across all languages.

14. Vocabulary Size Changes the Embedding Matrix

Each token identity usually corresponds to a learned embedding row. Increasing vocabulary size therefore increases the number of token embeddings the model must store and train.

A vocabulary decision becomes a model-parameter decision.

15. Rare Whole-Word Tokens Can Waste Capacity

A very large vocabulary can contain entries that occur too rarely to learn robust representations. Those entries occupy rows in the embedding table while offering little compression advantage.

Subwords solve this by sharing statistical evidence across related strings.

16. Too-Small Vocabularies Spend Capacity in Sequence Length

If the vocabulary contains only very small pieces, ordinary text expands into long sequences. The embedding table is smaller, but every example uses more token positions.

The system trades parameter memory for sequence computation.

17. There Is No Universal Best Vocabulary Size

The optimal point depends on corpus diversity, model scale, target languages, context length, hardware, training objective and application domain.

Vocabulary design must be evaluated as part of the whole system rather than optimised in isolation.

18. Numbers Need Deliberate Evaluation

A vocabulary can represent common numeric patterns compactly while still producing segmentations that do not align with place value. Dates, decimals, scientific notation and identifiers create different requirements.

Numeric tokenisation should be tested against arithmetic, exact copying and structured parsing tasks.

19. Code Deserves Its Own Coverage Analysis

Programming languages contain repeated keywords, operators, indentation patterns and identifier fragments. A vocabulary exposed to substantial code can compress these patterns better than one trained only on prose.

Code capability therefore interacts with both model training and tokenizer design.

20. URLs and Paths Are Adversarial Vocabulary Objects

URLs combine common domains with rare paths, IDs, punctuation and parameters. A tokenizer needs compositional coverage even when the entire string has never appeared before.

Whole-string vocabulary entries are useless here; smaller reusable pieces are essential.

21. Proper Names Reveal Corpus History

Frequently occurring names may receive compact tokens while uncommon names fragment. This is not a judgement of importance. It reflects how often the surface form appeared relative to competing vocabulary candidates.

Identity-sensitive systems should never infer social importance from token compactness.

22. Vocabulary Design Can Encode Historical Bias

If the corpus overrepresents some languages, regions, technologies or communities, the vocabulary can inherit that imbalance as representational efficiency.

The result is not necessarily harmful by itself, but it affects context use, cost and ease of handling across groups.

23. Special Tokens Consume Reserved Vocabulary Space

Beginning markers, end markers, masks, separators, roles and control tokens need reserved identities. These entries may never appear as ordinary text, but they are still part of the vocabulary contract.

See Special Tokens and Control Tokens.

24. Vocabulary Extension Is a Model Change

Adding a new token identity requires a new embedding row and possibly output parameters. The tokenizer can recognise the new token immediately, but the model has not automatically learned how to use it.

Vocabulary extension therefore usually requires training or adaptation.

25. Removing Tokens Can Break Compatibility

If token IDs shift after entries are removed or reordered, model weights no longer align with vocabulary identities. Even a visually identical token can point to the wrong embedding.

Stable IDs are part of the trained interface contract.

26. Merging Tokens Changes Sequence Statistics

A vocabulary revision that creates larger tokens shortens sequences and changes where boundaries occur. This can alter position distributions, attention patterns and training dynamics.

Tokenizer migration is therefore not a cosmetic preprocessing update.

27. The Tokenizer and Model Must Be Co-Designed

The tokenizer determines input identities and sequence lengths. The model determines how those identities are embedded and processed. Designing them independently can create avoidable inefficiency.

The vocabulary is the model’s alphabet of addressable pieces.

28. Context Length Changes the Vocabulary Trade-Off

When context is scarce, compact tokenisation becomes more valuable because every saved position can hold additional evidence. With abundant context, sequence length pressure may be lower, though compute cost can still matter.

Vocabulary design therefore interacts with context architecture.

29. Compute Architecture Changes the Trade-Off Too

Sequence computation and embedding-table size scale differently. Hardware constraints and architecture determine whether reducing sequence length or reducing vocabulary parameters yields the larger benefit.

There is no tokenizer design independent of the machine that runs it.

30. Vocabulary Quality Needs More Than Compression Ratio

A compact vocabulary can still create awkward boundaries for morphology, code or names. Evaluation should combine coverage, token density, downstream accuracy, exact-copy behaviour, multilingual parity and robustness.

This is developed in Tokenizer Evaluation and Benchmarking.

31. Token Fertility Is One Useful Metric

One way to describe fragmentation is to measure how many model tokens are produced per human-level word or other reference unit. Higher fertility means more model pieces are needed on average.

The metric is useful but language-dependent because “word” itself is not universal.

32. Characters per Token Is Another View

Characters per token can reveal compression efficiency without depending on word segmentation. But scripts differ in how much linguistic content a character carries, so it should not be interpreted as a universal fairness measure.

Multiple metrics give a more faithful picture.

33. Bytes per Token Matters for Storage-Layer Comparison

When systems operate across Unicode scripts, bytes per token can provide another machine-level efficiency measure. It connects encoded input size with sequence length.

Again, it measures representation efficiency rather than semantic information.

34. Downstream Task Fit Is the Final Test

A vocabulary can look excellent under compression metrics while performing poorly on the actual tasks the model must solve. Code completion, translation, retrieval, arithmetic and conversational generation stress different boundary properties.

Token design earns its place through downstream outcomes.

35. Vocabulary Design Is an Allocation Problem

Every token identity consumes finite representational capacity. The designer must decide which strings deserve direct access, which can share substructure and which should remain decomposed.

The vocabulary is therefore a budget over recurring form.

36. The Vocabulary Audit

1. What corpus trained the tokenizer?
2. Which languages and domains are represented?
3. What normalisation policy was used?
4. What base units guarantee coverage?
5. What vocabulary size was selected?
6. How many entries are special or reserved?
7. How fragmented are common words, names, code and numbers?
8. How does token density vary across languages?
9. How many rare entries are barely used?
10. What is the embedding-parameter cost?
11. How does vocabulary choice affect context utilisation?
12. What downstream tasks improve or worsen under alternative vocabularies?

37. What Students Should Remember

• A token vocabulary is finite.
• Common patterns receive compact identities.
• Rare strings are composed from smaller pieces.
• Larger vocabularies shorten some sequences but enlarge embedding tables.
• Smaller vocabularies improve compositional sharing but lengthen sequences.
• Corpus composition shapes representational efficiency.
• Vocabulary entries are not automatically concepts.
• The tokenizer and model form one compatibility contract.

38. The Deep Principle

A vocabulary is a decision about what deserves a name inside the machine. Give a recurring pattern its own token and the system can address it directly. Leave it decomposed and the system must reconstruct it from smaller parts.

Vocabulary design is representational economics: a finite budget of identities distributed across an effectively unlimited world of possible strings.

Continue the Representation & Tokenisation Series

• Text Normalisation Before Tokenisation | What Should Be Preserved, Collapsed or Canonicalised?
• Byte-Level Tokenisation | How Bytes Give Models a Route to Almost Any Text
• Tokenizer Evaluation and Benchmarking | How to Measure Coverage, Efficiency, Fidelity and Task Fit
• Canonical owner: World Representation & Cognitive Tools