WordPiece is a subword tokenisation family that represents common forms compactly while decomposing less familiar words into reusable pieces. At application time, WordPiece-style tokenizers are especially associated with greedy longest-match segmentation over a learned vocabulary.

WordPiece became widely known through BERT-family systems, where continuation pieces are often displayed with a marker such as ##. The marker is a vocabulary convention showing that a piece continues a word rather than beginning one; it is not part of the human word’s meaning.

This article belongs to the eduKateSingapore Representation and Tokenisation branch and follows the broader comparison in How Tokenisation Works.

The WordPiece Route

SOURCE TEXT
→ NORMALISATION
→ PRE-TOKENISATION / WORD-LIKE REGION
→ LEARNED WORDPIECE VOCABULARY
→ LONGEST VALID PIECE FROM CURRENT POSITION
→ CONTINUE UNTIL REGION IS COVERED
→ TOKEN IDS
→ MODEL

1. WordPiece Lives Between Words and Characters

Whole-word vocabularies cannot enumerate every possible name, inflection or technical term. Character vocabularies cover almost everything but create long sequences. WordPiece uses subword units so frequent strings can remain compact while unfamiliar forms are built compositionally.

The design objective is the same broad middle ground as other subword systems, even though the training and application rules differ.

2. The Vocabulary Contains Start and Continuation Forms

In common WordPiece conventions, a piece that begins a word may have one vocabulary form while a piece that continues inside a word carries a continuation marker such as ##. This allows the tokenizer to distinguish the same letter sequence at different boundary positions.

The marker is protocol information about segmentation, not visible source text.

3. Application Often Uses Greedy Longest Match

Given a word-like region, a WordPiece tokenizer commonly searches for the longest vocabulary entry that matches from the current position. Once found, it emits that piece and continues with the remaining suffix.

If the whole word exists as one token, the search can end immediately. If not, the tokenizer backs off to shorter pieces until it finds a valid route.

4. Greedy Means Local Choice, Not Global Semantic Analysis

Longest-match segmentation chooses a locally valid vocabulary decomposition. It does not compare every possible segmentation according to sentence meaning before selecting one.

The model receives the resulting sequence and performs contextual interpretation later.

5. A Whole Word Can Be One Piece

If a frequent word is in the vocabulary as a complete starting token, WordPiece can represent it compactly. This saves sequence positions and gives that surface form a direct vocabulary identity.

Frequency and vocabulary budget determine which words earn this privilege.

6. An Unseen Word Can Be Several Pieces

Suppose a word is absent as a whole but contains familiar fragments. WordPiece can emit a starting piece followed by one or more continuation pieces. The model can therefore receive a workable representation without requiring a dedicated token for the full word.

This is the open-vocabulary advantage of subword tokenisation.

7. If No Valid Decomposition Exists, an Unknown Token May Appear

Classic WordPiece vocabularies often include an unknown token such as [UNK]. If the tokenizer cannot represent a word-like region using available pieces, the entire region can collapse to that unknown identity.

This is a stronger information loss than ordinary fragmentation because the exact original string is no longer recoverable from the token sequence alone.

8. Coverage Therefore Depends on the Primitive Inventory

A WordPiece vocabulary with sufficient character or script coverage can decompose many unseen words. A limited alphabet can still produce unknown-token failures for unsupported symbols or scripts.

Coverage must be measured on the actual languages and domains the system serves.

9. WordPiece Training and WordPiece Application Are Different Questions

The inference procedure—longest matching pieces from the learned vocabulary—is easier to describe than the historical vocabulary-training algorithm. Modern libraries provide WordPiece training implementations, but implementation details can vary.

For that reason, claims about WordPiece should distinguish the general family and common modern implementations from one allegedly universal historical recipe.

10. WordPiece-Style Training Does Not Simply Copy BPE

Modern educational descriptions often contrast BPE’s raw pair-frequency merging with WordPiece-style scoring that favours pairs whose joint occurrence is informative relative to their individual frequencies. The Hugging Face tokenisation guide illustrates this distinction.

The important systems point is that the vocabulary can be learned under a different merge objective even when the final product is still a subword inventory.

11. Pair Frequency and Pair Informativeness Are Not the Same

A very common token can create frequent pairs simply because it appears everywhere. A likelihood-oriented score can instead favour pairs that occur together more strongly than their independent frequencies would suggest.

Different scoring rules create different vocabularies from the same corpus.

12. Vocabulary Training Still Inherits Corpus Bias

Whatever the exact learning objective, the corpus decides which strings are frequent enough to influence the vocabulary. Languages, names and domains that appear often receive more opportunities for compact representation.

Tokenizer training cannot escape the representational distribution of its data.

13. Pre-Tokenisation Matters Strongly

Many WordPiece pipelines first divide text into word-like regions according to whitespace and punctuation rules. The subword model then operates inside those regions.

This means the tokenizer’s first boundaries constrain which WordPiece tokens can ever cross.

14. WordPiece Does Not Define a Universal Word

The pre-tokenizer’s idea of a word is implementation-dependent and can be awkward across languages with different spacing conventions.

This is one reason raw-text approaches such as SentencePiece became attractive for multilingual systems.

15. Continuation Markers Preserve Boundary State

A piece such as ##ing communicates that it is expected inside an already-started word. This prevents the tokenizer from treating every substring occurrence as equivalent regardless of word position.

The marker is a small example of representation carrying state.

16. Continuation Markers Also Affect Decoding

When tokens are reconstructed into text, the decoder needs to know which pieces continue a word and which begin after a boundary. The visible marker is not normally printed literally.

See Detokenisation and Reconstruction.

17. Greedy Longest Match Can Prefer a Larger Piece Over Linguistic Structure

If a long vocabulary item matches the current string, the tokenizer can choose it even if a linguist would prefer several morphemes. The algorithm is satisfying vocabulary coverage, not producing a morphological parse.

Tokenisation and linguistic analysis remain separate layers.

18. The Same Surface Letters Can Have Different Token Forms by Position

A substring at the start of a word can map to a different vocabulary entry from the same letters inside a word because continuation state matters.

This allows the vocabulary to model some boundary-sensitive frequency patterns.

19. BERT Made WordPiece Familiar

BERT-family tokenizers popularised the visible pattern of full words mixed with ##-prefixed continuation pieces. But WordPiece should not be reduced to BERT alone; it is a tokenizer family whose implementations can appear in different model pipelines.

The model-specific tokenizer configuration remains the authoritative contract.

20. Special Tokens Sit Alongside WordPieces

BERT-style sequences commonly use reserved tokens for classification, separation, masking and padding in addition to ordinary WordPiece vocabulary entries.

These protocol tokens are discussed in Special Tokens and Control Tokens.

21. Masked-Language Modelling Used WordPiece Units as Prediction Targets

In BERT-style masked-language modelling, selected token positions are hidden or altered and the model learns to predict the vocabulary identity. If one human word spans several WordPieces, prediction occurs at the subword-token level unless a training strategy groups them.

Token boundaries therefore interact with the learning objective.

22. Whole-Word Masking Is a Response to Subword Boundaries

Some training variants mask all WordPieces belonging to one human word together. This changes the prediction task without changing the underlying tokenizer vocabulary.

It illustrates an important design pattern: model training can compensate for segmentation properties at a higher layer.

23. Names Can Fragment Heavily

An uncommon name absent from the vocabulary may become many pieces. The model retains a route to the string if its characters are covered, but exact reproduction can become harder.

Identity-sensitive systems should preserve canonical source strings.

24. Numbers Can Follow Vocabulary Accidents

Common digit sequences can become vocabulary pieces for statistical reasons rather than mathematical structure. A familiar year may tokenise compactly while a nearby number splits differently.

Numerical reasoning must therefore operate above arbitrary surface segmentation.

25. Code Can Be Awkward Under Prose-Oriented Word Boundaries

Programming identifiers, punctuation and operators do not always behave like natural-language words. A WordPiece pipeline whose pre-tokenizer assumes prose-like boundaries can represent code correctly but inefficiently.

Domain fit must be measured.

26. Multilingual Efficiency Depends on Vocabulary Allocation

A shared WordPiece vocabulary can devote many entries to high-resource languages and fewer to others. Lower-resource scripts can therefore use more pieces for equivalent visible content.

See Multilingual Tokenisation.

27. Vocabulary Size Controls Fragmentation

A larger WordPiece inventory can contain more full words and long fragments, reducing average sequence length. A smaller inventory pushes more strings into continuation pieces.

The same embedding-table versus sequence-length trade-off appears across subword families.

28. WordPiece Segmentation Is Normally Deterministic Once Configured

Given the same normalized input, pre-tokenisation and vocabulary, greedy longest-match application produces a stable segmentation. This supports reproducibility and caching.

Unigram systems differ because their probabilistic model can naturally support multiple candidate segmentations.

29. WordPiece vs BPE Is Not Merely a Branding Difference

BPE is usually described through an ordered sequence of frequent-pair merges. WordPiece-style systems use a learned vocabulary and commonly apply greedy longest-match segmentation, with modern training descriptions using a different criterion for vocabulary construction.

Compare with How Byte Pair Encoding Works.

30. WordPiece vs Unigram Separates Greedy Matching From Probabilistic Segmentation

WordPiece application generally takes the longest valid piece at each step. Unigram assigns probabilities to vocabulary pieces and evaluates complete segmentations under a probabilistic model.

Continue to How Unigram Tokenisation Works.

31. Token IDs Still Need the Correct Model

Every WordPiece vocabulary entry maps to a model-side ID and embedding. Swapping in another tokenizer changes those identities and breaks the trained interface unless the model is adapted.

Tokenizer compatibility is a hard constraint.

32. Evaluation Should Include Unknown-Token Risk

Measure not only average token counts but how often unsupported inputs collapse to [UNK]. Test languages, emoji, names, code and malformed text rather than only edited English.

Coverage failure can hide behind excellent average compression.

33. Evaluation Should Include Continuation Fragmentation

Count how many pieces common and rare words require, and inspect long-tail cases. A tokenizer may have no unknown tokens on a corpus while still producing inefficient chains of continuation pieces.

See Tokenizer Evaluation and Benchmarking.

34. The WordPiece Audit

1. What normalization happens first?
2. How does the pre-tokenizer define word-like regions?
3. Which symbols can appear as starting pieces?
4. How are continuation pieces marked?
5. How large is the vocabulary?
6. What training implementation produced it?
7. How does longest-match segmentation behave on rare strings?
8. When does the unknown token appear?
9. How fragmented are names, numbers and code?
10. How does efficiency vary across languages?
11. Do special-token and chat conventions match the model?
12. Does the tokenizer improve downstream task performance?

35. What Students Should Remember

• WordPiece is a subword tokenizer family.
• Common implementations use a learned vocabulary plus greedy longest-match application.
• Continuation pieces can be marked with forms such as ##.
• Whole words can remain one token when present in the vocabulary.
• Rare words are decomposed into reusable pieces.
• Unsupported regions may collapse to an unknown token.
• Vocabulary training and vocabulary application are separate stages.
• Token pieces are not automatically linguistic morphemes.

36. The Deep Principle

WordPiece takes a finite vocabulary and searches for a path through an open-ended language. It prefers large known pieces when they fit, then backs off to smaller continuation pieces when they do not.

WordPiece turns vocabulary membership into a greedy segmentation path: use the largest known piece available here, then continue until the surface string has a model-ready form.

Continue the Representation & Tokenisation Series

• How Byte Pair Encoding Works | From Frequent Pairs to a Learned Subword Vocabulary
• How Unigram Tokenisation Works | From Many Candidate Pieces to a Probabilistic Subword Model
• Tokenizer Training Corpus Design | How the Data Before the Model Shapes Every Token Boundary After It
• Canonical owner: World Representation & Cognitive Tools