A multimodal model receives several kinds of represented world at once. Text arrives as linguistic tokens, images as visual units, audio as temporal features or acoustic tokens, and video as spatial-temporal sequences. The model must preserve which modality each unit came from while learning relationships across them.
Multimodal tokenisation is therefore not simply “put every input into one list”. It is the architecture that converts heterogeneous signals into compatible model-ready units, marks their boundaries, allocates context capacity and establishes routes for cross-modal alignment.
This article sits inside the eduKateSingapore Representation and Tokenisation series.
The Multimodal Route
TEXT → TEXT TOKENS ┐ IMAGE → VISUAL TOKENS ├→ MODALITY-AWARE EMBEDDINGS → SHARED CONTEXT → CROSS-MODAL REPRESENTATION AUDIO → AUDIO TOKENS ┤ VIDEO → SPACE-TIME TOKENS ┘
1. Different Modalities Begin With Different Physical Sources
Text is symbolic before the model sees it. Images come from sampled light. Audio comes from sampled pressure variation. Video combines visual sampling with time. Their raw representations therefore begin at different layers.
A multimodal system must not erase these differences merely to force everything into one numerical space.
2. Modality-Specific Encoders Solve Different Entry Problems
Text needs tokenisation and lexical embeddings. Images may need patch projection or a vision encoder. Audio may need waveform or spectrogram processing. Video may need frame sampling and temporal aggregation.
Multimodal intelligence begins with several specialised representation front doors.
3. Shared Numerical Dimension Does Not Mean Shared Meaning
A text token vector and an image patch vector can be projected to the same dimensional size so common Transformer machinery can process them. But equal vector length does not make the two units semantically equivalent.
The model still needs modality identity and cross-modal training evidence.
4. Modality Markers Preserve Source Type
Systems can add modality embeddings, special boundary tokens or structured attention patterns so the model knows which units came from text, vision, audio or another channel.
This is analogous to role tokens in chat: source type is part of the context.
5. Text Tokens Carry Symbolic Language Structure
Text enters through the token vocabulary described across this series. It carries words, punctuation, identifiers and instructions through a finite symbolic sequence.
6. Visual Tokens Carry Spatial Structure
Images may become fixed patches, regions or learned visual units. Their positions preserve spatial arrangement so the model can reconstruct objects and layouts.
See Visual Tokenisation.
7. Audio Tokens Carry Temporal Structure
Audio is inherently time-based. Frames, latent features or codec tokens must preserve order and often duration.
See Audio Tokenisation.
8. Video Combines Space and Time
A video model can create visual patches for several sampled frames, spatiotemporal tube tokens or compressed latent units. The representation must answer both where a feature is and how it changes.
Video therefore places particularly heavy pressure on shared context capacity.
9. Context Capacity Must Be Allocated Across Modalities
If a high-resolution image creates thousands of visual units, fewer positions remain for text, audio history or additional images in a model with bounded context.
Multimodal context is a budget, not an infinite canvas.
10. One Image Can Cost More Context Than a Long Paragraph
Visible size and model-side representation size are different. A seemingly simple screenshot can produce many visual tokens because the system preserves fine spatial detail.
Users see one image; the model may see a long sequence.
11. Image Resolution Becomes a Context-Allocation Decision
Higher visual resolution can improve small-text and small-object recognition while consuming more compute and context. Lower resolution saves capacity while risking information loss.
The correct resolution depends on the question the image must answer.
12. Audio Duration Becomes a Context-Allocation Decision
A ten-minute recording contains far more temporal content than a ten-second clip. Systems may compress, summarise, transcribe or downsample audio before inserting it into shared context.
Every compression choice decides which acoustic distinctions survive.
13. Video Frame Rate Becomes a Context-Allocation Decision
Sampling more frames captures brief actions and motion but increases token count. Sampling fewer frames saves capacity while risking missed events.
The model’s “memory of the video” is constructed through a sampling policy.
14. Alignment Is the Central Multimodal Problem
To answer “What is written on the red sign?”, the system must connect text-language concepts with a visual region. To answer “Who is speaking when the door closes?”, it must align acoustic, temporal and possibly visual evidence.
Multimodal reasoning requires relationships across representations, not only representations in parallel.
15. Paired Data Teaches Cross-Modal Correspondence
Image captions, videos with transcripts, audio with text labels and documents containing both layout and language provide evidence linking modalities.
Training data determines which cross-modal relationships the model can learn reliably.
16. CLIP Shows Alignment Without One Shared Tokeniser
Models such as CLIP learn to align image and text representations in a shared embedding space even though the two modalities have different encoders and entry units.
This is a useful reminder: multimodal alignment does not require one literal tokenizer for every modality.
17. Shared Embedding Space Is One Alignment Strategy
If matching image and text representations are trained to lie near each other, the system can perform retrieval or zero-shot classification through vector similarity.
But shared-space similarity is still task-specific geometry, not universal semantic truth.
18. Cross-Attention Is Another Alignment Strategy
A model can allow text states to attend directly to visual or audio states, building relationships dynamically for the current query.
This lets the meaning of a phrase such as “that object” depend on which visual units are relevant now.
19. Early Fusion Mixes Modalities Sooner
Some architectures convert modalities into compatible unit sequences and process them together relatively early. This can support rich interaction but creates long shared sequences and demands careful modality markers.
Early fusion spends more shared computation on joint representation.
20. Late Fusion Preserves Specialist Encoders Longer
Other architectures process modalities largely separately and combine higher-level representations later. This can reduce shared sequence length and preserve domain-specific processing.
The trade-off is less fine-grained early interaction.
21. OCR Bridges Vision Into Text
For documents and screenshots, OCR can convert visible words into text tokens while the vision encoder preserves layout and appearance.
The same source therefore creates two representations: what the text says and where/how it appears.
22. Speech Recognition Bridges Audio Into Text
Speech can be transcribed into text tokens while an audio encoder preserves prosody, speaker and background sound.
A transcript alone can lose acoustic evidence that matters to the task.
23. Redundant Modalities Can Cross-Check One Another
If a slide contains text and a speaker reads it aloud, OCR and speech recognition provide two routes to similar content. Agreement can increase confidence; disagreement can identify an error.
Multimodality enables evidence triangulation.
24. Modalities Can Also Contradict One Another
A caption can be wrong, a dubbed voice can conflict with the video, or a graph label can contradict the plotted values. A multimodal system should not assume that every channel is mutually consistent.
Source identity and confidence must survive fusion.
25. Cross-Modal Hallucination Can Begin With Alignment Error
If the model associates a text phrase with the wrong image region, the later answer can be fluent but visually unsupported.
Multimodal debugging should distinguish perception, alignment and language-generation failure.
26. Spatial Grounding Requires More Than Caption Similarity
Knowing that an image contains a bicycle is different from locating which pixels belong to the bicycle. Grounding requires mapping language references to spatial regions.
Different tasks need different representational resolution.
27. Temporal Grounding Requires Time Alignment
In video or audio, “when” matters. A model answering when a bell rang must connect a language query to a temporal region.
Time codes, frame positions or segment timestamps become part of the token context.
28. Modality Compression Can Be Asymmetric
A system may retain every text token while aggressively compressing an image into a small number of visual latents. Another system may preserve high-resolution vision but summarise a long transcript.
The architecture decides which modality receives representational privilege.
29. Learned Resamplers Can Compress Variable-Length Visual Inputs
Some multimodal architectures use learned mechanisms to convert a large set of visual features into a smaller fixed or bounded number of latent units before language-model processing.
This is adaptive visual compression into a model-side token budget.
30. Compression Can Hide Small Evidence
If many visual patches are compressed into a few latents, tiny text or subtle objects may disappear. The same risk appears when audio is summarised or video frames are sparsely sampled.
Compression must protect the invariants required by the task.
31. Discrete Tokens Can Unify Modalities Architecturally
Text already uses discrete IDs. Images and audio can also be quantised into finite codebook identities, allowing several modalities to be represented as discrete sequences.
This is explored in Discrete Latent Tokens.
32. One Vocabulary for All Modalities Is Not Required
A model can use separate token ID ranges or separate encoders for text, image and audio while projecting all outputs into compatible hidden dimensions.
Shared computation does not require shared primitive vocabulary.
33. Position Needs More Than One Axis
Text has sequence position. Images have horizontal and vertical position. Video has horizontal, vertical and temporal position. Audio has time and potentially frequency position.
Multimodal position is a geometry problem, not one universal counter.
34. The Model Must Know Which Positions Are Comparable
Text position 50 does not naturally correspond to image patch 50. Cross-modal relationships must be learned through alignment signals, explicit timestamps, coordinates or attention.
Sequence index is not semantic alignment.
35. Multimodal Prompts Are Representation Programs
A user can provide text plus an image and ask a question about one region. The prompt specifies not only content but how multiple representations should be interpreted together.
Good interface design preserves modality roles and reference links clearly.
36. Ordering Modalities Can Influence Processing
Depending on architecture, placing image tokens before or between text segments can change how the sequence is contextualised. Serialized order becomes another protocol decision.
Multimodal templates should be treated as versioned interfaces.
37. Tool Outputs Are Another Modality-Like Channel
Structured tables, maps, code execution results and sensor data are not ordinary prose even when serialized as text. A broader multimodal architecture can preserve their typed structure rather than flatten everything into one undifferentiated string.
Representation should match source structure.
38. Multimodal Evaluation Must Separate Perception and Reasoning
If a model answers a chart question incorrectly, the failure may be OCR, spatial grounding, numerical reading, cross-modal alignment or reasoning. One end-to-end score hides the causal layer.
Evaluate each transformation separately.
39. Measure Context Cost by Modality
Track how many model-side units each image, second of audio or second of video consumes. Compare that cost with the useful information retained.
Representation efficiency becomes multimodal economics.
40. The Multimodal Tokenisation Audit
- Which modalities enter the system?
- What encoder and unit definition does each modality use?
- How is modality identity preserved?
- How is spatial or temporal position represented?
- How many units does each modality consume?
- What compression occurs before shared context?
- Can text be grounded to image regions?
- Can language be aligned to audio/video time?
- What happens when modalities contradict?
- Are source provenance and timestamps retained?
- Can one modality cross-check another?
- Which failures belong to perception versus reasoning?
- Do context budgets systematically privilege one modality?
- Does the final answer survive world return?
41. What Students Should Remember
- Different modalities begin from different source representations.
- Each modality can have its own tokenizer or encoder.
- Shared vector size does not imply shared primitive meaning.
- Modality identity and position must remain visible.
- Images, audio and video can consume large context budgets.
- Alignment connects information across modalities.
- Compression can erase small but important evidence.
- Cross-modal agreement can strengthen evidence; disagreement must remain visible.
42. The Deep Principle
Multimodal intelligence is not achieved by pretending every signal is the same. It is achieved by preserving what makes each modality different long enough for the model to learn where their representations genuinely correspond.
Text, image, audio and video can share a model context only after each has been given a faithful computational form—and the bridges between those forms must be learned, not assumed.