Most tokenizers make their boundary decisions before the model understands the input. Adaptive tokenisation asks a more ambitious question: what if the system could learn, from the current input and task, which pieces deserve to remain separate and which can be merged, pooled, dropped or represented more coarsely?
This turns tokenisation from a fixed preprocessing rule into a computational allocation strategy. Easy, redundant regions can be compressed. Rare, uncertain or task-critical regions can retain more resolution. The model no longer spends the same representational budget everywhere simply because the grid or vocabulary was designed that way in advance.
This article extends the eduKateSingapore Representation and Tokenisation series into learned, variable-resolution boundaries.
The Adaptive Representation Route
RAW OR FIXED-GRANULARITY INPUT → INITIAL TOKENS → EARLY CONTEXT → IMPORTANCE / MERGE / DELETE / POOL DECISION → SMALLER TASK-ADAPTED TOKEN SET → EXPENSIVE DEEP PROCESSING → OUTPUT → PROVENANCE RETURN TO ORIGINAL INPUT
1. Fixed Tokenisation Spends Compute Uniformly
A fixed image patch grid gives blank wall and human face the same number of tokens per square centimetre. A fixed subword tokenizer applies the same segmentation to a string regardless of the current task.
The representation is predictable but indifferent to informational value.
2. Adaptive Tokenisation Spends Compute Selectively
An adaptive system can keep more detail where the input appears important and compress regions that are repetitive or irrelevant.
Token count becomes a learned resource allocation.
3. Importance Is Task-Dependent
For face recognition, a person’s face may deserve high resolution. For road navigation, lane boundaries may matter more than faces.
There is no universal saliency map independent of the receiver’s job.
4. TokenLearner Made Learned Visual Selection Explicit
Google Research’s TokenLearner learns a small set of informative tokens from visual features rather than relying only on a hand-designed uniform split.
The result is fewer tokens entering expensive later computation while maintaining useful spatial-temporal information.
5. Adaptive Tokens Can Summarise Regions, Not Merely Select Them
A learned token can pool information from several input locations into one representation.
This differs from simply deleting low-scoring patches because the surviving unit can integrate what it compresses.
6. Learned Pooling Moves Granularity Into the Network
The model begins with fine tokens and later constructs coarser units after seeing context.
Granularity becomes dynamic rather than fixed at the input boundary.
7. Dynamic Merging Is Another Adaptive Strategy
Nearby or redundant token representations can be combined so deeper layers process a shorter sequence.
The system keeps information while reducing the number of independently tracked positions.
8. MrT5 Applies Dynamic Compression to Byte-Level Language
MrT5 introduces a learned deletion mechanism inside a byte-level encoder, allowing the model to shorten its sequence dynamically after early processing.
This directly addresses the central cost of tokenisation-free models: primitive byte sequences are flexible but long.
9. Adaptive Compression Can Follow Context
A character sequence that is predictable inside one word can be compressed differently from an unusual identifier or misspelling requiring finer inspection.
Context-sensitive grouping can therefore outperform one universal boundary map in principle.
10. Token Deletion Is Not Information Deletion Automatically
If information from a removed position has already been mixed into neighbouring representations, the position can disappear while some of its information survives.
But survival is empirical, not guaranteed.
11. Compression Creates an Information Bottleneck
Every merge, pool or deletion reduces the number of explicit positions available downstream.
The bottleneck is useful when it removes redundancy and harmful when it removes distinctions needed later.
12. The Hard Problem Is Knowing What Will Matter Later
An apparently unimportant token early in processing can become decisive after later context arrives.
Adaptive tokenisation therefore risks premature compression.
13. Early Layers Have Less Context Than Late Layers
A compression decision made immediately after input sees less semantic structure than one made after several contextual layers.
Compressing later may be smarter and more expensive.
14. Adaptive Systems Trade Early Compute for Later Savings
The model must first process enough information to decide which tokens are redundant.
Net efficiency depends on whether later savings exceed the cost of learning that decision.
15. Sparse Selection Can Reduce Attention Cost
When fewer tokens enter deep self-attention layers, memory and compute can fall substantially.
The benefit grows when the original sequence is large, as in high-resolution vision, video and byte-level text.
16. Video Is a Natural Adaptive Domain
Many frames contain redundant background while a few moments carry the decisive event.
Adaptive spatiotemporal tokens can reduce repeated information while retaining motion and event structure.
17. Long Documents Have Similar Redundancy
Boilerplate, repeated headers and predictable syntax can consume tokens without changing the user’s answer.
Adaptive document compression can preserve high-value evidence while representing routine structure more coarsely.
18. But Retrieval and Token Compression Are Different Decisions
Retrieval decides which external passages enter context. Adaptive tokenisation decides how the selected information is represented inside the model.
One filters sources; the other allocates internal resolution.
19. Adaptive Tokenisation Can Operate Within Retrieved Evidence
A retrieved page may contain one relevant sentence and much supporting boilerplate.
Internal compression can reduce processing while preserving a pointer back to the full source.
20. Provenance Becomes Harder After Merging
If one learned token pools information from many original positions, which source span does it represent?
High-stakes systems need a crosswalk from adaptive tokens back to their contributing evidence.
21. Many-to-One Tokens Need Contribution Maps
An adaptive visual token may summarise several patches. A merged byte token may contain information from several characters.
Contribution weights or retained source ranges can make the compression auditable.
22. Adaptive Boundaries Can Be Unstable
A tiny input change can sometimes alter which positions survive or merge.
Representation stability should therefore be tested under small perturbations.
23. Instability Can Be Useful or Dangerous
If a small change is semantically decisive, changing token allocation is appropriate. If it is harmless noise, large boundary changes indicate brittleness.
Evaluate adaptive sensitivity against task meaning, not token identity alone.
24. Adaptive Compression Creates Variable Sequence Lengths
Two inputs of equal raw length can produce different numbers of deep tokens.
Compute becomes dependent on content complexity, not only input size.
25. Variable Compute Can Be Economically Attractive
Easy inputs can become cheap while difficult inputs retain more capacity.
This resembles adaptive routing: spend resources where they alter the outcome.
26. Variable Compute Needs Operational Budgets
A production service still needs latency and memory limits.
Adaptive policies should operate within maximum token and compute envelopes.
27. The System Can Fail by Compressing Rare Things
Training objectives often reward average performance. Rare names, minority scripts or unusual visual details may receive too little representational capacity.
Efficiency must be audited across uncommon but important cases.
28. Frequency Is Not Importance
A rare safety warning can matter more than a repeated background texture.
Adaptive token importance should be driven by receiver value, not mere statistical commonness.
29. Uncertainty Can Be a Reason to Preserve Detail
When the model is unsure which region or span matters, aggressive compression can destroy the evidence needed to recover.
Uncertain areas may deserve more tokens, not fewer.
30. Confidence-Aware Granularity Is a Powerful Design Direction
Allocate coarse representation to predictable regions and fine representation to ambiguous regions.
The system turns uncertainty into computational resolution.
31. Adaptive Tokenisation Can Become Hierarchical
A model can maintain fine local tokens, medium-level grouped tokens and a small number of global summary tokens simultaneously.
Different layers answer different questions about the same source.
32. Hierarchy Protects Fine Detail Without Processing It Everywhere
Coarse tokens can guide global reasoning while a later mechanism returns to fine source positions only when needed.
This is computational zoom.
33. Adaptive Tokenisation Is Different From Attention
Attention can assign low weight to a token while still carrying that token through every layer.
Adaptive tokenisation changes the representation itself by reducing, merging or reorganising the token set.
34. Attention Says “Look Less”; Token Compression Says “Carry Less”
The distinction matters for memory and compute.
A token that remains in the sequence can continue to consume resources even if its attention weight is small.
35. Adaptive Tokenisation Is Different From Mixture-of-Experts Routing
Expert routing decides which parameters process a token. Adaptive tokenisation decides which token units continue to exist or at what resolution.
Both allocate compute, but along different axes.
36. The Two Can Be Combined
A system can first compress redundant positions and then route the surviving representations to specialised experts.
Efficient intelligence can be both token-adaptive and parameter-adaptive.
37. Evaluation Must Measure What Compression Erases
Average accuracy and FLOPs are not enough.
Test rare entities, fine visual details, exact copying, adversarial noise, minority languages, long-range dependencies and provenance recovery.
38. Compare Against Fixed Baselines at Equal Outcome Quality
A 50% token reduction is not valuable if the task error doubles.
Measure compute saved at equivalent receiver performance.
39. Latency Variance Matters in Production
Average compute can fall while difficult inputs become much slower.
Adaptive systems need percentile latency measurements, not only mean cost.
40. Reproducibility Needs the Adaptive Policy Version
If token selection is learned, changing model weights changes representation boundaries.
Historical evaluations should record the exact model and adaptive-token configuration.
41. Adaptive Tokenisation Changes the Meaning of “Context Length”
A model may accept a large raw input while compressing it to a much smaller deep sequence.
Raw length, early-token length and deep-token length become separate capacity measures.
42. Effective Context Depends on What Survives Compression
A million-character document is not meaningfully “in context” if decisive evidence was dropped before deep reasoning.
Capacity should be measured by retained task-relevant information, not input size alone.
43. Adaptive Representation Is a Form of Learned Attention to Reality
Human experts do something similar. A doctor does not inspect every pixel of every scan equally; a reader does not reread every word when searching for one clause.
Expertise includes knowing where detail is worth preserving.
44. But Human Selectivity Also Creates Blind Spots
Experts can miss unexpected evidence because their learned attention is biased toward familiar patterns.
Adaptive models inherit the same general danger: efficient selection can become systematic omission.
45. World Return Is the Antidote to Over-Compression
When a high-level adaptive token drives a consequential conclusion, the system should be able to return to the original pixels, bytes, frames or source spans supporting it.
Compression should accelerate reasoning without severing evidence.
46. The Adaptive Tokenisation Audit
- What fixed or primitive tokens enter the system?
- At which layer does adaptation begin?
- Does the model select, merge, pool or delete?
- What signal determines token importance?
- Is importance task-conditioned?
- How much sequence reduction occurs?
- What compute is spent before compression?
- What information can be lost permanently?
- How are rare and high-impact details protected?
- Does uncertainty trigger higher resolution?
- How stable are token decisions under small perturbations?
- What maximum compute and latency budgets apply?
- Can adaptive tokens be mapped back to source positions?
- Are raw, early and deep context lengths reported separately?
- Does the adaptive representation improve cost per correct receiver outcome?
47. What Students Should Remember
- Fixed tokenizers choose boundaries before understanding the current input.
- Adaptive tokenisation lets the model change representational resolution.
- Selection, merging, pooling and deletion are different compression mechanisms.
- TokenLearner shows learned visual token selection; MrT5 shows learned compression in byte-level language modelling.
- Fewer deep tokens can reduce compute and memory.
- Premature compression can erase rare but decisive evidence.
- Uncertainty can justify preserving more detail.
- Good adaptive systems keep a provenance route back to the source.
48. The Deep Principle
Fixed tokenisation asks how the world should be divided in general. Adaptive tokenisation asks how this world, for this task, deserves to be divided now.
Intelligence is partly the allocation of resolution. The mature tokenizer does not merely cut information into pieces; it learns which distinctions deserve continued computational life.