Memory is not one thing inside an AI system. There is active context, the key-value cache used during generation, retrieved external evidence, compressed summaries, recurrent state, persistent databases and learned model weights. Each remembers in a different way, for a different duration, at a different cost.
Memory representation is the design problem of deciding what information should remain explicit, what can be compressed, what can be forgotten, what should be retrieved later and what evidence must remain traceable to source.
This article continues the eduKateSingapore Representation and Tokenisation series. The question here is what happens after information has become tokens and context begins to outgrow immediate attention.
The Memory Representation Route
INPUT TOKENS → ACTIVE CONTEXT → KV CACHE / WORKING MEMORY → RETAIN / COMPRESS / SUMMARISE / EVICT → BOUNDED MEMORY OR EXTERNAL STORE → RETRIEVAL → RECONSTRUCTED CONTEXT → ANSWER / ACTION → SOURCE RETURN
1. Context Is Working Memory
The current prompt, conversation history, retrieved evidence and tool results form the model’s immediate working context.
This memory is explicit and expensive because every retained token competes for context capacity and inference compute.
2. The KV Cache Is an Inference Memory
Autoregressive Transformers commonly cache key and value representations from earlier tokens so they do not need to recompute the full past at every generation step.
The cache therefore stores a transformed memory of prior context rather than the raw text itself.
3. KV Cache Growth Creates a Memory Bottleneck
Longer contexts require more cached representations. Memory use can become a dominant serving constraint even when the model’s parameter count stays fixed.
This is why current research treats KV-cache compression as a first-class systems problem rather than a minor implementation detail.
4. Bounded Memory Changes the Question
If memory has a fixed number of slots, the system cannot simply keep every earlier representation.
It must decide what deserves to remain and how new information should modify old state.
5. TRELLIS Learns Dynamic KV-Memory Compression
Google Research’s TRELLIS replaces an ever-growing KV cache with fixed-size memory and learns recurrent compression with a forget gate at test time.
The important representation lesson is that memory can become an adaptive summary of prior token states rather than a full archive of them.
6. Lattice Treats Memory Update as Novelty Preservation
Google Research’s Lattice uses a fixed number of memory slots and an update rule designed to incorporate novel information while reducing interference with what is already stored.
This reframes remembering as controlled compression, not indefinite accumulation.
7. Forgetting Is a Feature When Memory Is Finite
A bounded system must forget something. The engineering problem is not whether forgetting occurs but whether it discards redundancy before it discards decision-critical evidence.
Useful forgetting removes what can be reconstructed or safely ignored.
8. Forget Gates Are Learned Eviction Policies
A forget mechanism controls how strongly old memory survives when new information arrives.
This is adaptive tokenisation at a later stage: the system decides which contextual representations continue to deserve computational life.
9. Compression Can Preserve Meaning and Lose Exact Wording
A summary may remember that “the contract expires in June” while forgetting the exact clause language, exceptions and date definition.
Semantic retention and verbatim retention are different memory jobs.
10. Exact-Copy Tasks Need Source Memory
Serial numbers, legal clauses, code and quotations should not rely on lossy semantic memory when exact reproduction matters.
Keep a route to the authoritative source representation.
11. Summary Memory Is a Hierarchical Representation
Conversation turns can be compressed into episode summaries; episode summaries can become long-term notes.
This connects memory to Hierarchical Representation.
12. Summary Layers Need Edition Control
If a source fact is corrected, every summary derived from the old fact can become stale.
Memory objects should preserve source version, timestamp and dependency when correctness matters.
13. External Retrieval Is Memory by Address
Instead of carrying everything in active context, a system can store documents or records externally and retrieve relevant items when needed.
This turns memory from continuous possession into selective re-access.
14. Retrieval Memory Preserves More Detail Than Summary Memory
A document store can retain the original source while an index stores compact keys for finding it.
The key is compressed; the value can remain high-fidelity.
15. Keys and Values Serve Different Memory Jobs
A retrieval key answers “where should I look?” A memory value answers “what information should I recover?”
Confusing the retrieval index with the evidence itself leads to weak source return.
16. Compressed Values Can Expand Retrieval Capacity
Google Research has used Perceiver-style compression in retrieval-augmented systems to condense large memory values into fewer tokens before fusion with the query.
The trade-off is clear: more memories can fit, while each memory carries less raw detail.
17. Persistent Memory Is Different From Model Weights
A database record can be edited directly and timestamped. A fact absorbed into model parameters is distributed across weights and difficult to update surgically.
Long-term factual memory and pretrained knowledge are therefore not interchangeable.
18. Model Weights Are Slow Memory
Training changes what the model tends to know or generate across many future prompts.
This memory is broad, expensive to write and poor at preserving explicit provenance for one fact.
19. Context Is Fast Memory
Prompt context is cheap to write, immediately available and temporary.
It is powerful precisely because it can change from one request to the next without retraining.
20. Retrieval Stores Are Addressable Long-Term Memory
Documents, vectors, records and graphs can persist outside the model and be retrieved selectively.
They are easier to update and audit than model weights.
21. Agent Memory Adds Experience
Long-running agents can accumulate interaction histories, preferences, plans, errors and successful procedures.
Unchecked accumulation eventually becomes another context problem.
22. Experience Compression Converts Episodes Into Reusable Knowledge
Recent agent-memory research increasingly frames memory, skills and rules as different compression levels of experience.
The more compressed the representation becomes, the more transferable it can be—and the more episode-specific detail it loses.
23. Episodic Memory Preserves What Happened
An episode can record a concrete interaction: task, context, actions, outcome and timestamp.
This is high-detail memory with limited transfer.
24. Procedural Memory Preserves How to Do Something
Several episodes can be compressed into a reusable procedure or skill.
The representation forgets incidental details and keeps action structure.
25. Declarative Memory Preserves Generalised Claims
Repeated experience can become a rule such as “verify the identifier before destructive action”.
This is extreme compression: broad utility, low event specificity.
26. Memory Write Policy Matters as Much as Retrieval Policy
If every observation becomes memory, noise accumulates. If only frequent observations survive, rare critical events disappear.
The system needs criteria for what deserves persistence.
27. Salience Is Not Frequency
A one-time safety failure can matter more than one thousand routine successes.
Memory importance should reflect consequence, novelty and receiver need—not occurrence count alone.
28. Novelty Can Guide Memory Update
If new information adds nothing not already stored, compressing or discarding it can save memory.
This is the intuition behind several modern bounded-memory approaches.
29. Contradiction Should Trigger Memory Revision
When new evidence conflicts with stored memory, merely appending both facts can leave an unresolved contradiction.
Memory needs revision, supersession or uncertainty state.
30. Forgetting Needs Provenance Too
If an item is removed because it was stale, duplicated or low-value, that policy can matter during later debugging.
High-stakes memory systems should record why information disappeared.
31. Quantisation Compresses Memory Numerically
KV-cache vectors can be stored at lower precision to reduce memory footprint while trying to preserve retrieval and attention behaviour.
Google Research’s TurboQuant work illustrates how current long-context engineering increasingly treats vector compression and KV-memory footprint as central efficiency concerns.
32. Numerical Compression and Semantic Compression Are Different
Quantisation stores approximately the same vector at lower precision. Summarisation creates a new representation with less semantic detail.
Their error modes differ and should be evaluated separately.
33. Memory Failure Has Several Forms
- Omission: important evidence was never stored.
- Eviction: useful evidence was removed too early.
- Distortion: compression changed the meaning.
- Staleness: old memory survives after the world changes.
- Collision: unrelated memories interfere.
- Retrieval failure: the memory exists but cannot be found.
- Provenance loss: the memory cannot return to its source.
34. More Memory Can Make Reasoning Worse
Irrelevant history consumes context, creates distractors and increases retrieval competition.
Memory quality depends on curation, not raw accumulation.
35. Context Engineering Is Memory Engineering
Anthropic’s context engineering framing emphasises that context is finite and must be curated for utility rather than filled indiscriminately.
Memory systems therefore need policies for selection, compression and retrieval—not just storage.
36. Memory Evaluation Needs Recall and Fidelity
A compressed memory can be small and useless if it cannot recall critical facts. It can recall facts and still be unsafe if wording or qualifiers are distorted.
Measure both what remains retrievable and how faithfully it returns.
37. The Memory Representation Audit
- What information lives in active context?
- What information lives in KV cache?
- What memory is bounded and what grows?
- What gets compressed, summarised or quantised?
- What policy decides retention versus eviction?
- How are novelty and importance estimated?
- What is stored externally for later retrieval?
- How are stale or contradictory memories revised?
- Can exact source evidence be recovered?
- Are summaries versioned?
- What is the measured recall of important facts?
- What fidelity is lost under compression?
- How does memory cost scale with context length?
- Can the system explain why something was remembered or forgotten?
38. What Students Should Remember
- Context, KV cache, external retrieval and model weights are different kinds of memory.
- Bounded memory requires forgetting or compression.
- Compression can preserve meaning while losing exact wording.
- Retrieval memory stores information by address rather than keeping it all active.
- Memory write policy matters as much as retrieval.
- Staleness and contradiction are memory failures, not only reasoning failures.
- World return requires a route from compressed memory to source evidence.
39. The Deep Principle
A memory system is an agreement about what the future is allowed to forget.
Intelligence does not require perfect retention. It requires preserving the distinctions that future decisions will need, discarding redundancy without erasing evidence, and knowing when compressed memory is no longer enough.