What if a language model did not begin by deciding what the words or subwords are? Tokenisation-free models push the boundary downward. Instead of first applying a learned subword vocabulary, they can read characters or raw bytes directly and let the model learn useful linguistic groupings internally.
The promise is attractive: fewer language-specific preprocessing assumptions, no out-of-vocabulary word problem in the ordinary subword sense, and a tiny primitive alphabet. The price is equally real: much longer sequences and therefore more computation unless the architecture learns how to compress those low-level inputs efficiently.
This article is the counter-case inside the eduKateSingapore Representation and Tokenisation series. A library about tokenisation is incomplete if it never asks whether explicit learned token boundaries are necessary at all.
The Tokenisation-Free Route
RAW TEXT → BYTES OR CHARACTERS → SMALL PRIMITIVE ALPHABET → EARLY LOCAL PROCESSING / DOWNSAMPLING → CONTEXTUAL REPRESENTATIONS → LEARNED HIGHER-LEVEL STRUCTURE → TASK OUTPUT → TEXT / LABEL / ACTION NO FIXED SUBWORD VOCABULARY REQUIRED
1. “Tokenisation-Free” Still Uses Units
The phrase does not mean the model receives an indivisible stream of language. Bytes or characters are still discrete input units.
What disappears is the separate learned subword-tokenizer stage that decides in advance which multi-character pieces deserve vocabulary entries.
2. Bytes Give a Tiny Universal Machine Alphabet
A byte has 256 possible values. That is dramatically smaller than a subword vocabulary containing tens of thousands of entries.
Embedding-table size can therefore shrink while sequence length grows.
3. Character Models Use Human-Oriented Symbols
Character-level systems can operate on Unicode characters or grapheme-like units rather than bytes.
This can be easier to interpret than bytes but requires a larger and more complicated alphabet across writing systems.
4. Byte-Level Representation Avoids a Fixed Language Vocabulary
English, Chinese, Tamil, emoji, code and newly invented strings can all be represented as byte sequences without adding vocabulary entries.
The representation handles coverage uniformly even though different scripts can require different numbers of bytes.
5. Coverage Is Not Efficiency
A byte model can represent almost any encoded text while using several times more positions than a compact subword tokenizer.
Universal coverage shifts the problem from vocabulary design to sequence processing.
6. ByT5 Made the Byte Trade-Off Concrete
ByT5 showed that a largely standard Transformer could operate directly on UTF-8 bytes and remain competitive with token-level counterparts while gaining robustness on character-sensitive tasks.
The paper also makes the cost explicit: byte sequences are longer, changing training FLOPs and inference speed.
7. CANINE Removed Explicit Tokenisation at the Encoder Input
Google Research’s CANINE operates directly over character sequences without an explicit subword vocabulary, using downsampling before deep contextual processing.
The design demonstrates the central engineering answer to long primitive sequences: compress them inside the model rather than outside it.
8. Removing the Tokenizer Removes One Hard Boundary System
A subword tokenizer decides that one string is one unit and another must be split. A byte or character model postpones that grouping decision.
The model can learn useful multi-character structure from context instead of inheriting a fixed vocabulary cut.
9. This Can Reduce Vocabulary Lock-In
New names, technical terms and spelling variants do not require new subword entries to become representable.
The primitive alphabet remains stable while the model’s learned internal representations adapt.
10. It Can Also Reduce Tokenizer Migration Risk
A system without a large learned vocabulary does not face the same token-ID-to-embedding migration problem described in Tokenizer Migration and Version Compatibility.
But byte encoding, character normalization and model architecture still require versioning.
11. Tokenisation-Free Does Not Mean Preprocessing-Free
Text must still be encoded. Unicode still exists. Line endings, normalization and invalid byte handling still need defined policies.
The pipeline becomes simpler in one place and remains governed elsewhere.
12. UTF-8 Makes Character Cost Uneven
Many ASCII characters use one byte while other Unicode characters use several.
Byte-level universality therefore does not automatically guarantee equal sequence efficiency across languages.
13. Subword Inequality Can Become Byte-Length Inequality
Removing one source of multilingual imbalance does not remove every representation difference.
Measure real byte and character lengths by language rather than assuming neutrality.
14. Spelling Errors Become Local Changes
A one-character typo alters only a small number of primitive units. Under a subword tokenizer, the same typo can completely change the segmentation of an entire word.
This helps explain why byte-level models can be robust on noisy or spelling-sensitive tasks.
15. Morphology Must Be Learned Internally
A subword vocabulary may hand the model recurring fragments such as prefixes or suffixes. A primitive byte model must discover comparable groupings through representation learning.
Tokenisation-free systems relocate inductive bias rather than eliminate the need for structure.
16. Word Boundaries Can Still Be Useful
Spaces and punctuation remain visible bytes or characters. The model can learn that they often mark linguistic boundaries.
The difference is that those boundaries are cues rather than mandatory vocabulary segmentation.
17. Long Primitive Sequences Create Attention Cost
If one word becomes four or more characters and perhaps more bytes, the sequence can become several times longer than a subword representation.
Naively applying full deep attention to every primitive position is computationally expensive.
18. Downsampling Moves From Preprocessing Into Architecture
CANINE reduces sequence length before expensive deep layers. Other designs use pooling, convolution, learned merging or funnel-like structures.
The system still needs compression; it simply learns or applies that compression after primitive input.
19. Learned Compression Can Be Context-Sensitive
A fixed tokenizer uses the same segmentation for a substring wherever it occurs. An internal learned compression mechanism can potentially group information differently depending on surrounding context.
This connects directly to Adaptive Tokenisation.
20. The Model Can Build Several Granularities
Early layers can operate on bytes, middle layers on learned local groups and later layers on semantic spans.
Granularity becomes hierarchical rather than fixed by one tokenizer file.
21. Primitive Inputs Improve Exact Character Access
Tasks such as spelling correction, transliteration, pronunciation-sensitive generation and exact string manipulation benefit when characters or bytes remain directly addressable.
A subword system can still learn these tasks, but the character layer is less explicit.
22. Exact Access Does Not Guarantee Exact Reasoning
A model that sees every digit separately can still perform arithmetic incorrectly.
Representation fidelity and reasoning competence remain different layers.
23. Byte Models Can Read Code Naturally as Raw Text
Operators, punctuation and unusual identifiers are all representable without special vocabulary expansion.
But formal program structure still requires parsing or learned contextual reconstruction.
24. They Can Also Read Mixed-Script Text Without Vocabulary Surgery
A sentence mixing English, Chinese, emoji and code remains a byte stream.
The burden moves from vocabulary coverage to learning useful patterns over longer heterogeneous sequences.
25. Small Vocabularies Change Parameter Allocation
A byte vocabulary needs far fewer input embeddings than a 30,000- or 100,000-entry subword vocabulary.
Those saved parameters can potentially be spent on deeper or wider contextual modelling.
26. Output Representation Can Still Be Expensive
An autoregressive byte decoder must predict many more output positions to produce the same visible paragraph.
Input simplicity can become generation latency.
27. Hybrid Systems Are Possible
A model can consume primitive bytes, compress internally and decode through another granularity.
“Token-free” and “token-based” are ends of a design spectrum rather than mutually exclusive religions.
28. MrT5 Shows Dynamic Merging as a Modern Direction
MrT5 addresses byte-model inefficiency by learning to remove or merge many encoder positions dynamically, reducing sequence length while preserving task performance.
This is exactly the frontier the library needs to recognise: primitive input and learned token compression can coexist.
29. The Boundary Can Become a Model Decision
Traditional tokenisation says “these boundaries are fixed before inference.” Adaptive systems increasingly ask “which information should remain separate for this input?”
Tokenisation becomes part of intelligence rather than only preparation for intelligence.
30. Evaluation Must Include Representation Cost
A byte model can improve robustness and still be impractical if latency or memory rises too far.
Accuracy, sequence length, throughput, multilingual behaviour and energy cost should be measured together.
31. Compare Equal Tasks, Not Equal Token Counts
A byte token and a subword token carry different average amounts of visible text.
Cost comparisons should use documents, characters or completed receiver jobs as common denominators.
32. Robustness Tests Should Include Corruption
Typos, Unicode variation, transliteration, casing, spacing and noisy text reveal representation brittleness that clean benchmarks can hide.
Primitive models should be tested where their claimed advantages actually matter.
33. Multilingual Evaluation Must Be Per-Language
Average scores can hide scripts or languages that remain computationally expensive or difficult.
Representation equality is an empirical question.
34. Tokenisation-Free Does Not Mean Meaning-Free
The model still needs to discover words, phrases, entities, syntax and concepts internally.
Removing explicit subword boundaries does not remove language structure from the world.
35. It Changes Where the Structure Is Learned
A tokenizer externalises one layer of compression. A tokenisation-free architecture internalises more of that compression into learned computation.
The design question is therefore not “tokens or no tokens?” but “where should grouping happen, under which constraints, and at what cost?”
36. The Tokenisation-Free Audit
- What primitive unit is used: byte, character or grapheme?
- What encoding and normalization rules apply?
- How long are sequences compared with a subword baseline?
- How large is the primitive vocabulary?
- Where does downsampling or compression occur?
- Can grouping adapt to context?
- How does the model behave across languages and scripts?
- How robust is it to typos and noisy text?
- What exact-character tasks improve?
- What training and inference costs increase?
- Does output generation become slower?
- What hidden linguistic units emerge internally?
- How are byte positions mapped back to human-visible text?
- Does the representation improve the receiver’s job after total cost is counted?
37. What Students Should Remember
- Tokenisation-free models still operate on discrete primitive units.
- Bytes provide a tiny universal vocabulary.
- Primitive sequences are much longer than subword sequences.
- CANINE and ByT5 show that competitive models can operate without explicit learned subword tokenisation.
- Downsampling and learned compression recover efficiency.
- Byte models can be robust to spelling and noisy text.
- Removing a tokenizer relocates—not removes—the need to learn structure.
- The real question is where grouping should happen and what it costs.
38. The Deep Principle
Subword tokenisation freezes one theory of useful boundaries before the model begins. Tokenisation-free systems gamble that intelligence can discover better groupings from smaller primitives.
The absence of a tokenizer does not remove boundaries from language. It moves the responsibility for discovering those boundaries deeper into the model.