Every representation has a cost. In language models, tokenisation turns that cost into something measurable: sequence positions that consume compute, memory, bandwidth, context capacity, storage and often money.
This does not mean the cheapest representation is the best. Over-compression can remove distinctions the task needs. The real engineering objective is to preserve sufficient meaning and evidence at the lowest acceptable representation cost.
This article continues the eduKateSingapore Representation and Tokenisation series.
The Cost Chain
SOURCE CONTENT → REPRESENTATION CHOICE → TOKEN COUNT → CONTEXT OCCUPANCY → MEMORY + COMPUTE → LATENCY + THROUGHPUT → INFRASTRUCTURE COST → USER EXPERIENCE → BUSINESS / SYSTEM DECISION
1. Token Count Is a Representation Cost Meter
Two texts with similar human meaning can produce different token counts. The difference reflects tokenizer vocabulary, language, formatting and domain.
Token count therefore measures how expensive the chosen model representation is—not how much meaning the user produced in any universal sense.
2. Input Tokens Consume Context Capacity
Every input token occupies part of a model’s available context. System instructions, conversation history, retrieved evidence, code and multimodal tokens all compete for this finite space.
A token saved in one part of the prompt can be spent on evidence somewhere else.
3. Output Tokens Also Need Capacity
Generation consumes sequence positions too. A system that fills all capacity with input may leave insufficient room for a complete answer depending on the model interface.
Representation budgeting must include both entry and return paths.
4. Longer Sequences Require More Processing
More tokens mean more model states to compute. The exact scaling depends on architecture and implementation, but longer sequences generally require more work.
Tokenizer efficiency therefore influences inference economics even before pricing is considered.
5. Attention Makes Sequence Length Especially Important
Classical full self-attention compares many positions with many others, making long contexts computationally expensive. Modern systems use architectural and serving optimisations, but sequence length remains a major resource variable.
Representation granularity has system-wide consequences.
6. KV Cache Memory Grows With Active Context
Autoregressive serving commonly stores key/value representations from prior tokens so they do not need to be recomputed at every generation step. Longer active contexts therefore require larger cache memory.
Token count becomes memory occupancy.
7. Memory Pressure Reduces Concurrency
If each request consumes more accelerator memory, fewer requests can run concurrently on the same hardware.
One verbose prompt can therefore affect not only its own latency but overall system throughput.
8. Context Length Changes Capacity Planning
A service supporting long-context workloads needs different memory and scheduling assumptions from one serving short chat turns.
Nominal model context is therefore an infrastructure design parameter, not merely a product feature.
9. Tokenisation Efficiency Changes Effective Context
If tokenizer A represents a workload using 20% fewer tokens than tokenizer B, the same model-side context can hold roughly more visible source material under tokenizer A, subject to protocol and modality overhead.
A better tokenizer can therefore increase effective capacity without changing model architecture.
10. Multilingual Users Can Pay Different Representation Costs
Equivalent meaning can require different numbers of tokens across languages. Users writing in a more fragmented language can consume more context and potentially more billable units for the same task.
See Multilingual Tokenisation.
11. Cost per Token Is Not Cost per Meaning
A pricing model can charge the same rate per token across languages while the amount of human-visible content per token differs.
Business analysis should therefore compare cost per completed user job as well as raw token price.
12. Cost per Task Is a Better Receiver Metric
Ask how much it costs to summarise one document, answer one support case, analyse one contract or tutor one student to a successful outcome.
Task-level accounting incorporates representation density, model choice and output requirements together.
13. Prompt Boilerplate Has Recurring Cost
System prompts, policies, schemas and examples can be repeated on every request. Even small inefficiencies multiply across millions of calls.
Reusable prompt architecture should be measured for information value per token.
14. Hidden Protocol Tokens Count Too
Role markers, separators and chat templates can add token overhead users never see. Cost accounting should use the serialized model input, not just visible user text.
See Special Tokens and Control Tokens.
15. Repeated Context Can Be Expensive Waste
If a long policy document is appended unchanged to every turn, the system repeatedly processes the same representation. Caching, retrieval or structured references can reduce duplicated context depending on model and platform capabilities.
Representation reuse is an economic optimisation.
16. Retrieval Can Save Context by Selecting Only Relevant Evidence
Instead of sending an entire knowledge base into the model, retrieval selects the passages most relevant to the current question.
The index spends compute upstream to reduce context cost downstream.
17. Bad Retrieval Can Waste Tokens
Irrelevant or duplicate chunks consume context without helping the answer. A top-k retrieval system that returns five near-identical passages can spend five times the representation budget for one evidence source.
Context efficiency depends on evidence diversity and relevance.
18. Chunk Size Is a Cost Parameter
Large chunks preserve context but consume more tokens when retrieved. Small chunks are cheaper but can require more items to reconstruct a complete argument.
See Token Budgets and Chunking.
19. Overlap Is Paid Redundancy
Overlapping retrieval chunks protect boundary information by repeating text. That repetition consumes storage, embedding compute, index space and prompt tokens.
Overlap should be large enough to preserve continuity and no larger.
20. Embedding Generation Has Its Own Representation Cost
Every document chunk sent through an embedding model consumes compute. Smaller chunks create more vectors; larger chunks create fewer but potentially less precise vectors.
Retrieval architecture moves cost between indexing and inference stages.
21. Vector Storage Is Another Cost Layer
High-dimensional embeddings require storage. Large corpora multiplied by many chunks can produce substantial vector indexes.
Chunking granularity becomes storage economics.
22. Search Indexes Also Pay for Token Granularity
Character n-grams improve fuzzy recall but create many more index terms than whole-word indexing. Storing term positions also increases index size.
See Tokenisation in Search and Indexing.
23. Model Vocabulary Size Has Parameter Cost
A larger token vocabulary can shorten sequences but requires more embedding rows and often a larger output projection.
Token compression therefore moves cost from sequence length into model parameters.
24. Smaller Vocabularies Move Cost Back Into Sequences
A small vocabulary is cheap to store and shares parameters broadly, but it may fragment ordinary text heavily.
The cost trade-off is the same one explored in Token Vocabulary Design.
25. Training Cost Also Depends on Tokenisation
A training corpus converted into longer token sequences requires more model steps to process the same visible amount of text. More compact tokenisation can therefore reduce training sequence volume.
But aggressive compression can create rare tokens with poor statistical support.
26. Corpus Size Should Be Measured in Several Units
Training datasets can be described in bytes, characters, words, documents or model tokens. Each metric reveals a different aspect of scale.
Model-training cost is most directly connected to model-side sequence units, while human data coverage may be better understood through documents and languages.
27. Output Length Can Dominate Interactive Cost
A concise prompt can produce a long report. In generative workloads, output tokens may dominate latency and cost even when input is small.
Response design is therefore part of representation economics.
28. Verbosity Is a Cost Choice
A longer answer can provide more explanation, examples and safety boundaries. It can also waste time and money if the receiver needs only one precise fact.
Good systems calibrate output resolution to the user job.
29. Compression Must Protect Necessary Resolution
Summarising a legal rule into one sentence can save hundreds of tokens while omitting the exception that makes the answer safe. The cheapest representation can be the most expensive decision if it causes error.
Cost accounting must include consequence, not only compute.
30. Error Cost Can Dwarf Token Cost
Saving a few cents by truncating evidence is irrational if the resulting wrong decision costs hours of work, lost revenue or human harm.
Representation economics must be bounded by risk.
31. Low-Stakes Tasks Can Optimise More Aggressively
A casual brainstorming prompt can tolerate summarisation and smaller retrieval budgets. A medical, legal or financial workflow needs stronger fidelity and source preservation.
The risk class determines the acceptable compression ratio.
32. Latency Is Partly Representation Latency
Longer prompts take longer to ingest. Longer outputs take longer to generate. Larger multimodal inputs take longer to encode.
User-perceived speed therefore reflects representation size as well as hardware speed.
33. Throughput Is the System-Level Receiver
A service that shortens average sequence length can handle more requests per unit time on the same infrastructure, depending on batching and model architecture.
Representation efficiency scales operational capacity.
34. Bandwidth Matters in Distributed Systems
Tokens, embeddings and multimodal features may move between clients, gateways, accelerators and storage services. Larger representations consume more network bandwidth.
Compression can reduce communication cost as well as compute.
35. Storage Cost Accumulates Across Logs and Caches
Prompt logs, token traces, embeddings, KV caches and generated outputs can become large at production scale.
Retention policies should preserve what is needed for observability and compliance without storing unnecessary representation indefinitely.
36. Multimodal Inputs Have Different Cost Units
An image may create hundreds or thousands of visual tokens; audio has a token rate over time; video multiplies spatial and temporal units.
Shared context requires a cross-modal accounting system rather than assuming one visible item equals one unit of cost.
37. Resolution Is a Cost Dial
Higher image resolution, denser video sampling and higher audio bitrate preserve more information while increasing representation size.
The correct setting is the lowest resolution that still preserves decision-critical evidence.
38. Dynamic Budgets Can Match Task Difficulty
A simple query may need no retrieval and a short answer. A difficult research question may justify more documents, longer reasoning and a larger output.
Adaptive representation budgeting spends resources where they change the result.
39. Fixed Budgets Are Easy but Inefficient
Giving every request the same retrieval count, context allocation and maximum output length wastes resources on easy tasks and starves difficult ones.
Cost-aware systems should route by task complexity and risk.
40. Token Budgets Can Be Treated Like Financial Budgets
Allocate capacity across instruction, evidence, memory and output. Track planned versus actual use. Identify repeated waste. Reserve contingency for complex cases.
Representation management becomes an operating discipline.
41. A Useful Cost Equation
TOTAL REPRESENTATION COST ≈ INPUT PROCESSING + ACTIVE-CONTEXT MEMORY + OUTPUT GENERATION + RETRIEVAL / EMBEDDING + STORAGE / BANDWIDTH + LATENCY COST + ERROR / CONSEQUENCE COST
No single provider price captures all of these components.
42. Cost per Correct Outcome Is the Strongest Measure
A cheaper model or shorter prompt can look efficient until quality falls enough to require retries or human repair. A more expensive representation can be cheaper overall if it produces reliable completion on the first attempt.
Optimise cost per successful receiver outcome.
43. Measure Retries
Poor representation often produces follow-up clarification, regeneration or correction. Those extra calls add tokens that should be attributed to the original design failure.
First-pass success is an economic metric.
44. Measure Human Repair Time
An answer that saves tokens but requires ten minutes of editing may be more expensive than a slightly longer accurate output.
Human time belongs in representation cost accounting.
45. Measure Opportunity Cost
If context is filled with low-value boilerplate, useful evidence cannot enter. The cost is not only compute spent; it is information excluded.
Finite context creates opportunity cost for every token.
46. Measure Cost by Language and Domain
Average global cost can hide high-cost minority languages, code-heavy users or document types with poor token efficiency.
Representation accounting should be segmented just like financial reporting.
47. Measure Cost Before and After Tokenizer Changes
A tokenizer migration can reduce sequence length while increasing vocabulary parameters or changing retrieval chunks. Compare the full stack.
See Tokenizer Migration and Version Compatibility.
48. Cost Optimisation Should Preserve Provenance
Deleting source metadata can save a few tokens while making answers impossible to verify. Compress prose before compressing evidence identity.
Auditability is part of output value.
49. Cost Optimisation Should Preserve Uncertainty
Turning a range into one number or several hypotheses into one claim saves representation space while creating false certainty.
Never buy compression by spending epistemic integrity.
50. The Representation Cost Audit
- How many visible words or characters become model tokens?
- How does that ratio vary by language and domain?
- How much context is instructions, history, evidence and output reserve?
- How much hidden protocol overhead exists?
- How much duplication comes from retrieval overlap?
- What embedding and vector-storage costs exist?
- What KV-cache memory does the active context require?
- What latency is attributable to long inputs or outputs?
- How does token density affect concurrency?
- What storage and bandwidth are consumed by representations?
- What retry rate comes from weak context or compression?
- How much human repair time is required?
- What evidence is excluded by low-value context?
- What error consequences arise from over-compression?
- What is the cost per correct completed task?
51. What Students Should Remember
- Tokens consume context, compute and memory.
- Token cost is not the same as meaning cost.
- Multilingual token density can create unequal system costs.
- Vocabulary size trades parameter cost against sequence cost.
- Retrieval can save tokens by selecting relevant evidence.
- Compression can become expensive if it creates errors.
- The strongest metric is cost per correct receiver outcome.
52. The Deep Principle
A model never receives “meaning” for free. Meaning is carried through a representation, and that representation consumes finite resources.
Representation economics asks one disciplined question: how much compute, memory, time and money must we spend to preserve enough of the world for the receiver to make the right distinction?