An image does not arrive inside a vision model as “a cat”, “a road” or “a diagram”. It arrives as a numerical representation that must be divided into workable visual units before the model can build higher-level structure.
That division is visual tokenisation in the broad sense: the system decides what counts as one addressable piece of visual information. Depending on architecture, the pieces may be fixed image patches, regions, object proposals, learned discrete codes, feature-map locations or frame-patch combinations in video.
This article extends the eduKateSingapore Representation and Tokenisation series beyond text.
The Visual Representation Route
WORLD SCENE → SENSOR / CAMERA → PIXEL ARRAY → RESIZE / CROP / NORMALISE → PATCHES / REGIONS / VISUAL UNITS → NUMERICAL EMBEDDINGS → POSITION + CONTEXT → VISUAL REPRESENTATION → CLASSIFICATION / DESCRIPTION / ACTION → WORLD RETURN
1. Pixels Are Already a Representation
A camera measures light through a sensor and converts those measurements into digital pixel values. The pixel grid is not the world itself. It is a sampled representation shaped by lens, sensor, exposure, colour processing, resolution and viewpoint.
Visual tokenisation therefore begins after an earlier chain of representation decisions has already occurred.
2. One Pixel Is Usually Too Small to Be a Useful Semantic Unit
A single pixel contains very little local structure. Objects, edges, textures and symbols emerge from relationships among many neighbouring pixels.
Vision systems therefore often group pixels into larger units before deep contextual processing.
3. Patches Turn an Image Into a Sequence
Vision Transformer architectures popularised a simple and powerful move: divide an image into fixed-size patches, flatten or project each patch into a vector and process the resulting patch sequence with Transformer-style computation.
Dosovitskiy and colleagues described this approach in An Image is Worth 16×16 Words. The title makes the analogy explicit: image patches can play a role similar to tokens in a text sequence.
4. A Patch Is Not an Object
A 16×16 patch can contain part of an eye, a piece of sky, half a letter or several tiny objects. Patch boundaries are geometric convenience, not proof of visual semantics.
The model must combine many patch representations to recover larger structures.
5. Patch Size Controls Resolution
Smaller patches preserve finer spatial detail but create longer sequences. Larger patches reduce sequence length but can blur or compress small features inside one unit.
SMALL PATCHES + finer detail + better localisation potential − longer sequence − more compute LARGE PATCHES + shorter sequence + cheaper processing − coarser local resolution − small features can be diluted
6. Image Resolution and Patch Size Interact
A 224×224 image divided into 16×16 patches creates a different sequence length from a 1024×1024 image using the same patch size. Increasing image resolution without changing patch size increases the number of visual units.
Visual context therefore has a token-budget problem analogous to text.
7. Resizing Can Destroy Information Before Tokenisation
If a high-resolution image is aggressively downsampled, small text, hairline cracks, distant signs or fine medical structures can disappear before patches are formed.
No later visual reasoning can guarantee recovery of detail that the input representation removed.
8. Cropping Changes the World the Model Sees
A crop can remove distracting background and improve focus. It can also remove the context needed to interpret the central object. A traffic sign without the road, or a graph without its axis labels, can become ambiguous.
Preprocessing is part of representation fidelity.
9. Colour Normalisation Is Another Upstream Choice
Vision pipelines often normalise pixel channels according to training statistics. This improves numerical conditioning but creates a model-specific input contract.
A model trained under one pixel-normalisation scheme should not casually receive another.
10. Patch Projection Creates Model-Side Visual Embeddings
Each image patch is transformed into a numerical vector. That vector is not a label saying what the patch contains. It is an initial representation that later layers can contextualise using information from other patches.
This parallels Token IDs and Embeddings, except the starting unit is visual rather than lexical.
11. Position Matters Because Images Are Spatial
The same patch content in the top-left and bottom-right positions can play different roles. Vision models therefore need spatial position information so they do not treat the image as an unordered bag of patches.
Identity plus position becomes the starting point for contextual vision.
12. Context Makes a Patch Meaningful
A patch containing a curved dark edge could belong to a wheel, an eye, a letter or a shadow. Surrounding patches constrain the interpretation.
Visual meaning is relational just as word meaning is contextual.
13. Edges Cross Patch Boundaries
A physical object does not respect the artificial grid. A face can span dozens of patches, and one patch boundary can cut directly through an eye or mouth.
Good models must reconstruct continuity across those arbitrary cuts.
14. Fixed Grids Are Simple but Blind to Object Boundaries
Patch tokenisation is computationally regular because every image can be divided according to the same grid. But the grid does not adapt to the content.
This is analogous to fixed-length text chunking: predictable size, imperfect semantic alignment.
15. Region-Based Representations Try to Follow Objects
Other vision systems represent candidate regions or object proposals rather than fixed patches. A region can align more closely with a meaningful object while producing a variable number of units.
The trade-off is greater preprocessing complexity and dependence on the region-selection mechanism.
16. Object Tokens Are Higher-Level Representations
A learned object representation can compress many pixels or patches into one unit corresponding approximately to an entity. This reduces sequence length and can improve reasoning about relationships among objects.
But object discovery itself can fail, especially for unusual scenes or tiny objects.
17. Hierarchical Vision Uses Multiple Resolutions
Some architectures build representations at several spatial scales: fine local features early, broader regional features later. This resembles hierarchical chunking in text.
Different tasks require different visual resolutions.
18. OCR Creates a Second Tokenisation Route
An image containing text can be processed visually as patches and linguistically through OCR followed by text tokenisation. The two routes preserve different information.
Patch tokens preserve appearance and layout; OCR tokens preserve recognised characters and words. Combining both can improve document understanding.
19. OCR Errors Become Text-Token Errors
If OCR mistakes “0” for “O”, the text tokenizer receives the wrong symbol. The later language model may repair it from context, but the source representation has already been altered.
The failure chain should be diagnosed at the earliest layer.
20. Charts Need Layout and Text Together
A chart cannot be understood from OCR words alone. Axis positions, line geometry, legend colour and spatial correspondence matter. Visual tokens carry relationships that flat text extraction can lose.
Multimodal document understanding requires several coordinated representations.
21. Diagrams Expose Structural Relationships
Arrows, boxes, relative positions and grouping can encode meaning that no individual patch contains. A diagram is a structured visual language.
Visual tokenisation must preserve enough geometry for those relations to be reconstructed.
22. Medical Images Raise the Resolution Stakes
In radiology, pathology or retinal imaging, tiny spatial patterns can be clinically important. Downsampling and patch size therefore become safety-relevant representation choices.
A vision system should be validated at the resolution of the abnormalities it is expected to detect.
23. Satellite Images Raise the Scale Stakes
A satellite scene can contain kilometres of geography. Small patches preserve local detail but create huge sequences; large patches compress the scene while potentially hiding small structures.
Hierarchical or tiled processing can help manage this scale.
24. Tiling Is Visual Chunking
Large images are often divided into tiles that are processed separately. This is the visual analogue of document chunking.
A structure near a tile boundary can be split across two contexts, so overlap and later stitching become important.
25. Overlapping Tiles Protect Boundary Features
Processing overlapping image regions allows an object near one tile edge to appear fully inside another tile. The cost is repeated computation and duplicated evidence.
The same overlap trade-off appears in text retrieval chunks.
26. Video Adds Time to the Tokenisation Problem
A video is a sequence of image frames. A model can tokenise spatial patches inside each frame, temporal segments across frames, or learned spatiotemporal tubes.
The unit must preserve both where something is and how it changes.
27. Frame Rate Is a Temporal Resolution Choice
Sampling fewer frames reduces token volume but can miss brief actions. Sampling every frame preserves more temporal detail but creates enormous sequences.
Video tokenisation therefore balances spatial and temporal resolution simultaneously.
28. Motion Can Be a More Useful Unit Than Frames
Some video representations model changes between frames, optical flow or learned temporal features. For action recognition, movement may be more informative than static appearance alone.
The correct visual unit depends on the task.
29. Discrete Visual Tokens Compress Continuous Features
Vector-quantised models can map continuous visual features into discrete codebook identities, creating token-like units that can be modelled sequentially.
This is developed in Discrete Latent Tokens | How Continuous Signals Become Codebook Identities.
30. Generative Image Models Need a Return Path
If an image is represented through latent or discrete visual tokens, generation eventually requires a decoder that reconstructs pixels from those internal units.
The quality of the generated image depends on both the model over tokens and the fidelity of the visual decoder.
31. Reconstruction Loss Can Hide Behind Plausibility
A decoded image can look realistic while failing to reproduce exact source details. Plausibility is not the same as identity preservation.
For creative generation that may be acceptable. For evidence, documents or medical imaging it may not be.
32. Visual Tokens Can Be Continuous or Discrete
Patch embeddings in Vision Transformers are continuous vectors. Vector-quantised image models use discrete codebook IDs. Both can be called token-like because they create addressable units, but their mathematical properties differ.
The representation type should be named precisely.
33. Continuous Patches Do Not Need a Fixed Visual Vocabulary
A patch can be projected directly into a vector without first assigning it to one of a finite set of visual token IDs. This preserves continuous variation.
Discrete visual tokenisers intentionally compress that continuous space into a finite codebook.
34. Visual Token Count Is Not Object Count
An image with three objects may contain hundreds or thousands of patch tokens. Another architecture may represent those objects through a smaller number of region or object tokens.
Token count measures the chosen representation, not the number of meaningful entities in the scene.
35. Visual Tokenisation Affects Multimodal Context
When text and images share one model context, the number of visual units determines how much computational capacity the image consumes relative to words.
This is explored in Multimodal Tokenisation | How Text, Images, Audio and Video Share One Model Context.
36. The Visual Tokenisation Audit
- What sensor representation enters the model?
- What resolution, crop and colour normalization are applied?
- What counts as one visual unit?
- What patch or region size is used?
- How is spatial position represented?
- Can small objects survive preprocessing?
- Do text and layout require OCR or document parsing?
- Are large images tiled?
- Is overlap used at tile boundaries?
- How are video time and frame rate represented?
- Are visual tokens continuous vectors or discrete IDs?
- What information is lost during reconstruction?
- How much model context does each image consume?
- Does the representation preserve what the receiver must decide?
37. What Students Should Remember
- Pixels are already a sampled representation of a scene.
- Visual tokenisation creates workable units such as patches or regions.
- A patch is not an object.
- Smaller patches preserve detail but create longer sequences.
- Position and context reconstruct larger visual structure.
- OCR provides a parallel text-token route for images containing language.
- Video adds temporal tokenisation.
- Visual tokens can be continuous or discrete.
38. The Deep Principle
Vision models need cuts just as language models do. The camera samples the world into pixels; the model groups those pixels into workable visual units; contextual computation rebuilds objects, scenes and relationships above those artificial boundaries.
The patch is not the object, just as the text token is not the meaning. It is a computational cut that gives the model somewhere to begin.