How Byte Pair Encoding Works | Frequent Pairs to Subword Vocabulary

Byte Pair Encoding turns frequently recurring neighbouring units into larger reusable pieces. Repeating that operation builds a vocabulary that compresses common strings while preserving a route to rare or unseen strings through smaller components.

For tokenisation, BPE is important because it solves a practical problem: whole-word vocabularies are too large and brittle, while character- or byte-level sequences can be unnecessarily long. BPE builds a middle layer.

This article extends the eduKateSingapore Representation and Tokenisation series and the vocabulary layer described in Token Vocabulary Design.

The BPE Idea in One Line

START WITH SMALL UNITS
→ COUNT ADJACENT PAIRS
→ MERGE A HIGH-VALUE PAIR
→ RECOUNT
→ MERGE AGAIN
→ BUILD A VOCABULARY OF REUSABLE LARGER PIECES
→ APPLY THE LEARNED MERGE RULES TO NEW TEXT

1. BPE Begins With a Primitive Alphabet

A BPE tokenizer needs a starting set of units. Depending on implementation, those units may be characters, bytes or pre-tokenized symbols. The starting alphabet determines what the system can always fall back to when no larger learned piece applies.

The primitive layer therefore establishes the coverage floor.

2. The Corpus Supplies the Frequency Evidence

BPE learns from a training corpus. It observes which units occur next to one another and how often. Repeated neighbouring patterns become candidates for compression.

A vocabulary trained on law, code and medicine will therefore develop different high-frequency merges from one trained on children’s stories or social media.

3. The First Merge Creates a New Unit

Suppose the pair t + h appears very frequently. A BPE learner may create a new unit th. Every eligible occurrence can then be represented using that larger piece.

The vocabulary has grown by one identity, and the corpus representation has become slightly shorter.

4. Repeated Merges Build Hierarchy

After th exists, the learner may later merge it with e to create the, or with another neighbouring unit inside some other recurring sequence. Larger tokens therefore emerge from earlier smaller tokens.

The final vocabulary reflects a history of merge decisions.

5. The Merge List Is Part of the Tokenizer

A trained BPE tokenizer does not simply store its final vocabulary. It also needs the ordered merge rules or equivalent structure that determines which combinations are permitted and in what precedence.

This is why BPE tokenizers are versioned artifacts rather than just word lists.

6. Applying BPE to New Text Replays Learned Structure

At inference time, the tokenizer does not relearn pair frequencies from the new sentence. It applies the vocabulary and merge logic learned earlier.

The segmentation of new text is therefore shaped by historical corpus statistics.

7. Common Strings Become Compact

A frequent word may eventually become one token or a small number of tokens. Common prefixes, suffixes and recurring fragments can also become compact pieces.

This reduces sequence length for the patterns that consume the most corpus volume.

8. Rare Words Remain Composable

A rare word does not need a dedicated whole-word token. It can be assembled from smaller subwords or primitive units already in the vocabulary.

This was one of the central motivations behind the use of subword units in neural machine translation. Sennrich, Haddow and Birch’s 2016 paper Neural Machine Translation of Rare Words with Subword Units helped establish the approach.

9. BPE Solves the Open-Vocabulary Problem by Composition

Natural language can always generate new names, compounds and inflections. A fixed whole-word vocabulary cannot enumerate them all. BPE keeps the vocabulary finite by representing uncommon forms through known smaller pieces.

The system does not need to know every future word in advance.

10. Compression and Meaning Are Not the Same

A BPE merge is chosen because a sequence is useful statistically, not because a linguist has declared it a morpheme or semantic unit.

The learned pieces can align with meaningful language structure, but that alignment is a side effect rather than a guarantee.

11. A Frequent Non-Morpheme Can Still Be a Good Token

If a character sequence occurs repeatedly across the corpus, compressing it can save sequence positions even when it has no standalone human interpretation.

BPE optimises a representation objective, not a dictionary.

12. A Genuine Morpheme Can Still Be Split

A linguistically meaningful root or suffix may remain fragmented if the corpus statistics do not reward a merge that isolates it cleanly.

Tokenizer output therefore should not be used as automatic morphological analysis.

13. Vocabulary Size Controls How Far Merging Proceeds

If the target vocabulary is small, BPE stops after relatively few merges and leaves many strings fragmented. If the target vocabulary is larger, more recurring sequences become direct tokens.

Vocabulary size therefore controls the balance between embedding-table size and sequence length.

14. More Merges Do Not Automatically Produce a Better Tokenizer

A very large vocabulary can create rare tokens with weak training evidence and consume many embedding parameters. It can also reduce compositional sharing across related forms.

The optimum depends on the model, corpus and receiver job.

15. Corpus Duplication Can Distort Merge Priorities

If the training corpus contains duplicated documents or overrepresented sources, their repeated strings can gain excessive influence over pair frequencies.

Tokenizer training therefore inherits data-quality problems from the corpus.

16. Multilingual BPE Is a Shared Budget Problem

In a multilingual corpus, merge opportunities compete across languages and scripts. High-frequency languages can consume more of the finite merge budget, leaving lower-resource languages represented by smaller pieces.

This can increase token density for underrepresented languages even when they remain fully representable.

17. BPE Can Be Character-Level or Byte-Level at Its Base

The phrase “BPE tokenizer” does not uniquely specify the primitive representation. Some systems start from characters or pre-tokenized text; others use bytes or byte-derived symbols.

See Byte-Level Tokenisation for the lower-layer coverage trade-off.

18. Pre-Tokenisation Can Restrict Which Merges Are Possible

If a pipeline first splits on whitespace or punctuation, BPE may be prevented from learning tokens that cross those boundaries. Another implementation may represent whitespace directly and permit different merge patterns.

Two BPE tokenizers can therefore differ substantially because their preprocessing pipelines differ.

19. Normalisation Changes the Frequency Table

Lowercasing or Unicode canonicalisation can combine frequencies that would otherwise be split across multiple surface forms. This changes which pairs are considered frequent enough to merge.

Tokenizer vocabulary and normalisation policy must be analysed together.

20. Spaces Can Be Encoded as Part of Tokens

Some BPE implementations make whitespace visible to the tokenizer so common word-start patterns become distinguishable from the same letters inside words.

This improves reconstructability and allows the vocabulary to model boundary context explicitly.

21. BPE Tokenisation Is Deterministic Once the Rules Are Fixed

Standard application of a fixed BPE merge list produces a stable segmentation for the same normalised input. This is valuable for reproducibility and caching.

Variants such as BPE dropout deliberately introduce stochasticity during training to expose the model to alternative segmentations.

22. BPE Dropout Treats Segmentation as a Training Variable

Instead of always applying every eligible merge, BPE dropout can randomly skip some merges during training. The same surface string can then appear under several subword segmentations.

This can reduce overdependence on one exact boundary pattern and improve robustness in some settings.

23. Alternative Segmentations Can Act Like Data Augmentation

When a model sees several valid decompositions of related strings, it is encouraged to learn more robust relationships across smaller units rather than memorise one segmentation path.

This idea is related to subword regularisation, developed more explicitly for Unigram-style models by Taku Kudo in Subword Regularization.

24. BPE Works Especially Well When Repetition Is Strong

Language, code and structured text contain enormous repetition. BPE turns that repetition into direct vocabulary access.

The algorithm therefore converts corpus regularity into sequence compression.

25. Names Reveal the Long Tail

Common names may receive compact tokenisation while rare names remain fragmented. The system stays open-vocabulary, but exact copying can become harder for long multi-token identities.

Keep authoritative source strings when identity matters.

26. Numbers Reveal the Representation Mismatch

Digit strings can acquire merges according to corpus frequency rather than place value. A common year can become compact while another number is split differently.

BPE sees recurring strings. Mathematics sees quantities. The two representational logics are not identical.

27. Code Can Gain Useful Domain Tokens

If source code appears frequently during tokenizer training, repeated language keywords, operators and identifier fragments can receive dedicated merged tokens.

This can reduce sequence length for code-heavy workloads.

28. URLs Remain Highly Compositional

Domains and protocol strings may become common tokens, while unique paths and query parameters remain decomposed. This is exactly the kind of open-ended string space BPE handles well through composition.

29. BPE Does Not Solve Contextual Meaning

Once BPE has chosen pieces, the model still has to determine what those pieces mean in context. A token such as bank remains ambiguous until surrounding sequence information is integrated.

See Contextual Representation.

30. BPE Does Not Solve Syntax

Subword segmentation decides pieces. Parsing decides relationships among pieces at higher structural levels.

See Segmentation vs Parsing.

31. BPE Does Not Guarantee Equal Multilingual Efficiency

A shared vocabulary can represent every language while still allocating more merges to high-frequency scripts and languages. Token parity must be measured rather than assumed.

See Multilingual Tokenisation.

32. Decoding Reverses the Learned Segmentation

The decoder maps token IDs back to pieces and recombines them under the tokenizer’s boundary conventions. When normalization is compatible, BPE can support reliable text reconstruction.

See Detokenisation and Reconstruction.

33. Tokenizer–Model Compatibility Is a Hard Constraint

The learned token IDs correspond to specific model embeddings. Changing the BPE vocabulary or merge rules without retraining can scramble the model’s input identities.

The tokenizer and model form one trained interface.

34. BPE Evaluation Needs More Than Compression

Measure coverage, token density, long-tail fragmentation, multilingual parity, exact-copy behaviour, round-trip fidelity and downstream task performance.

See Tokenizer Evaluation and Benchmarking.

35. BPE vs WordPiece

Both produce subword vocabularies, but their merge-selection logic differs. BPE classically prioritises frequent adjacent pairs, while WordPiece-style training uses a likelihood-oriented score rather than raw pair frequency alone.

Continue to How WordPiece Works.

36. BPE vs Unigram

BPE builds its vocabulary through successive merges. Unigram approaches can begin with many candidate pieces and optimise a probabilistic inventory, allowing multiple possible segmentations under one vocabulary.

Continue to How Unigram Tokenisation Works.

37. The BPE Audit

What primitive units does the tokenizer start from?
What corpus supplies pair frequencies?
What normalization and pre-tokenisation occur first?
How large is the target vocabulary?
How are merge candidates scored and ordered?
What share of the vocabulary is used frequently?
How fragmented are names, numbers and code?
How does token density vary across languages?
Is BPE dropout or another stochastic variant used?
Can the tokenizer round-trip the required source invariants?
Is the vocabulary exactly matched to the model?
Does the segmentation improve downstream tasks?

38. What Students Should Remember

BPE begins with small units.
Frequent neighbouring pairs are merged into larger pieces.
Repeated merges create a finite subword vocabulary.
Common strings become compact.
Rare strings remain composable.
BPE tokens are statistical pieces, not guaranteed morphemes.
Vocabulary size controls the sequence-length trade-off.
Corpus frequency determines which patterns receive compression.

39. The Deep Principle

BPE turns repetition into reusable identity. What appears often enough earns a shorter path; what remains rare is reconstructed from smaller pieces.

Byte Pair Encoding is a compression history turned into a vocabulary: the corpus teaches the tokenizer which neighbouring pieces deserve to become one.

How Byte Pair Encoding Works | From Frequent Pairs to a Learned Subword Vocabulary