Most meaningful structures are larger than one token. Words form phrases, phrases form sentences, sentences form paragraphs, paragraphs form sections, frames form shots, shots form scenes, points form objects, and events form processes.
Hierarchical representation is the strategy of preserving several of those scales at once instead of forcing every reasoning step to happen on one flat stream of primitive tokens.
This article extends the eduKateSingapore Representation and Tokenisation series into multi-scale memory and long-context structure.
The Hierarchical Route
PRIMITIVE TOKENS → LOCAL GROUPS → SENTENCE / PATCH / EVENT / OBJECT UNITS → SECTION / SCENE / PROCESS UNITS → GLOBAL REPRESENTATION → REASONING → RETURN DOWN THE HIERARCHY TO SOURCE EVIDENCE
1. Flat Sequences Treat Every Unit at One Scale
A flat token sequence gives every subword or patch a position in one long list.
This is convenient for a Transformer and inefficient when the task concerns higher-level structure such as chapter argument, scene identity or process stage.
2. Hierarchies Compress Repeated Local Detail
A paragraph representation can summarise dozens of word tokens. A scene representation can summarise many frames.
Higher levels shorten the distance between semantically related large units.
3. Compression Is Useful Only if Lower-Level Evidence Remains Reachable
A section summary may support global reasoning, but a precise factual claim should still be traceable to its sentence or source page.
Hierarchy should create abstraction without severing provenance.
4. Sentences Are Higher-Level Text Units
A sentence usually contains a more complete proposition than one token.
Sentence embeddings can therefore support semantic search or document-level reasoning at a coarser scale.
5. Sentence Boundaries Are Useful and Imperfect
One claim can span several sentences, while one sentence can contain several claims.
Editorial units are practical boundaries, not universal semantic atoms.
6. Paragraphs Preserve Local Discourse
A paragraph often keeps a claim, explanation and example together.
Paragraph-level tokens can therefore be stronger retrieval units than arbitrary fixed-length chunks.
7. Sections Preserve Scope
A statement under “Limitations” has a different role from one under “Conclusion”.
Section identity gives paragraphs a structural parent.
8. Documents Become Trees, Not Merely Strings
Title → section → subsection → paragraph → sentence → token forms a natural hierarchy.
Representing that tree can reduce accidental dependence on absolute token distance.
9. Hierarchical Models Separate Local Encoding From Global Encoding
A common architecture encodes short chunks with one model, compresses each chunk into a vector, then runs another model over those chunk vectors.
Recent long-document work, including hierarchical Transformer families, uses this basic principle to make document-scale context more tractable.
10. Higher-Level Tokens Reduce Sequence Length
A 10,000-token document might become 100 paragraph tokens or 20 section tokens at a higher layer.
Global reasoning can then operate over a much shorter sequence.
11. Shorter Global Sequences Reduce Long-Range Path Length
Two sentences thousands of primitive tokens apart can become neighbouring paragraph representations at the next level.
Hierarchy compresses distance as well as count.
12. But Higher-Level Tokens Are Lossy
A paragraph vector cannot preserve every exact number, name and qualifier unless the architecture retains a path back to lower levels.
Semantic compression increases the importance of retrieval and source return.
13. Hierarchy Can Be Fixed by Document Structure
Headings, paragraphs, pages and chapters provide explicit boundaries.
These structures are available before model inference and can be used directly.
14. Hierarchy Can Also Be Learned
A model can discover recurring groups, motifs or semantic regions that do not match editorial boundaries.
Adaptive tokenisation can therefore feed hierarchical representation.
15. Learned Hierarchy Needs Interpretability Controls
If one latent region represents a mix of several source spans, human verification becomes harder.
Contribution maps and source crosswalks protect auditability.
16. Images Have Natural Hierarchies Too
Pixels form patches, patches form regions, regions form objects and objects form scenes.
Computer vision has long benefited from multi-scale features because edges, parts and objects live at different resolutions.
17. Video Adds Temporal Hierarchy
Frames form clips, clips form shots, shots form scenes and scenes form episodes.
Long-video understanding is naturally hierarchical because event meaning spans several temporal scales.
18. Audio Has Frames, Phonetic Units, Words and Utterances
A waveform can be represented at acoustic-frame level and later compressed into speech or semantic units.
Higher levels preserve meaning while lower levels preserve exact sound.
19. Graphs Have Nodes, Communities and Whole-Graph Representations
Local neighbourhoods can be pooled into communities or subgraphs.
Graph hierarchy reduces large relational systems into multi-scale structure.
20. Workflows Have Actions, Stages and Whole Processes
Dozens of low-level actions can belong to one review stage. Several stages belong to one workflow.
Operational reasoning is easier when the system can move between those scales.
21. Hierarchical Memory Supports Long-Term Context
Older detail can be compressed into summaries while recent detail remains fine-grained.
This creates a memory pyramid rather than one indefinitely growing flat transcript.
22. Summary Memory Must Preserve Edition and Time
A summary created before a correction may become stale.
Higher-level memory should record source versions and timestamps so the system can know when to rebuild it.
23. Global Tokens Are a Minimal Hierarchy
A classification token or global document token acts as a coarse node connected to many local tokens.
This is a one-level hierarchy even when no explicit paragraph or section tokens exist.
24. Multiple Global Tokens Can Specialise
One summary token can represent topic, another chronology and another entity state.
Global memory does not need to collapse every dimension into one vector.
25. Hierarchy and Interaction Topology Are Coupled
Local tokens may interact densely within a group while summary tokens connect groups globally.
See Token Interaction Topology.
26. Hierarchical Position Gives Tokens Structural Coordinates
A token can know its sentence, paragraph and section parents in addition to its flat index.
See Positional Representation.
27. Retrieval Can Operate at Several Levels
A system can retrieve a document summary first, then drill into the most relevant section and finally the exact paragraph.
This reduces search cost while preserving evidence depth.
28. Coarse-to-Fine Retrieval Is Computational Zoom
The system spends broad low-resolution compute first and detailed high-resolution compute only where needed.
Hierarchy therefore connects representation design to cost control.
29. Fine-to-Coarse Reasoning Creates Summaries
Evidence can flow upward from tokens to sentences to sections to a global conclusion.
Coarse-to-fine verification should then flow downward to test that conclusion.
30. Upward Compression and Downward Verification Form a Loop
A mature hierarchical system should not reason only upward.
It should be able to reopen detail whenever a coarse representation becomes uncertain or consequential.
31. Hierarchy Can Fail Through Premature Summary
If a local exception disappears before global reasoning, the summary can become confidently wrong.
Compression should preserve exceptions, uncertainty and provenance more aggressively than redundant prose.
32. Different Tasks Need Different Hierarchies
A legal question may need clause → section → document structure. A video task may need frame → shot → scene. A biology task may need residue → motif → domain → protein.
Hierarchy should follow the structure of the world and receiver job.
33. Hierarchies Can Overlap
A sentence belongs to one paragraph, but an entity can span several paragraphs. A video action can cross a shot boundary.
Tree structure is useful and not universally sufficient.
34. Cross-Cutting Graph Links Complement Hierarchy
Entity references, citations, callbacks and prerequisite relations can connect distant branches.
The strongest representation can combine hierarchy with graph edges.
35. Hierarchy Changes the Meaning of Context Length
A model can retain millions of source tokens indirectly if many are compressed into higher-level memories.
But indirect retention is not equivalent to direct access to every original detail.
36. Effective Memory Needs Fidelity Labels
The system should know whether a fact is stored verbatim, extracted, summarised or inferred.
Representation level is part of epistemic status.
37. The Hierarchical Representation Audit
- What are the primitive input units?
- What natural local groups exist?
- What higher-level units should be represented directly?
- Are boundaries explicit or learned?
- What information is lost at each compression step?
- Can higher-level units return to lower-level evidence?
- How are positions represented across levels?
- How do local and global tokens communicate?
- Can retrieval operate coarse-to-fine?
- Are summaries versioned and time-aware?
- Are exceptions and uncertainty preserved?
- Do cross-cutting graph links supplement the hierarchy?
38. What Students Should Remember
- Most meaningful structures contain several scales.
- Hierarchical representation compresses local detail into larger units.
- Higher-level tokens shorten global reasoning paths.
- Compression is lossy and needs source return.
- Hierarchy appears in text, images, video, audio, graphs and workflows.
- Coarse-to-fine retrieval spends resolution selectively.
- Good hierarchies preserve exceptions, uncertainty and provenance.
39. The Deep Principle
Intelligence rarely reasons at one resolution. It zooms out to see structure and zooms in to verify detail.
Hierarchy is useful when abstraction makes the world smaller without making the evidence unreachable. The best global memory remains connected to the local facts from which it was built.