A photograph freezes one arrangement of light. A video adds the harder dimension: change. Video tokenisation is the representation problem of deciding how many moments, how much spatial detail and which motion patterns deserve to become the finite units a model can process.
A video can be represented as every pixel of every frame, sampled image patches, short spatiotemporal tubelets, tracked objects, scene changes, actions, captions or learned latent tokens. Each representation preserves a different answer to the question: what happened?
This article extends the eduKateSingapore Representation and Tokenisation series beyond the broader Visual Tokenisation owner. The job here is specifically temporal: how moving images become model-ready sequences without confusing frame count with understanding.
The Video Representation Route
WORLD EVENT → CAMERA / SENSOR → FRAME SEQUENCE → TEMPORAL SAMPLING → PATCHES / TUBELETS / OBJECTS / EVENTS → VIDEO TOKENS + POSITION → CONTEXTUAL MODEL → ACTION / EVENT / RELATION INFERENCE → ANSWER / RETRIEVAL / CONTROL → RETURN TO TIMESTAMPED EVIDENCE
1. A Video Is Not Just a Large Image
A still image asks what is visible at one moment. Video asks what persists, changes, enters, leaves, causes, follows and repeats across time.
The temporal axis creates relationships that cannot be recovered reliably from independent frames alone.
2. Frames Are the First Obvious Temporal Units
A digital video commonly arrives as a sequence of frames sampled at a frame rate. Each frame is already a discretisation of continuous optical change.
Frame rate is therefore an upstream tokenisation decision before the model sees anything.
3. Frame Rate Sets the Fastest Visible Change
A low frame rate can miss a brief gesture, ball contact or machine fault. A high frame rate preserves short events while multiplying storage and model input.
Temporal resolution should match the fastest event the receiver needs to distinguish.
4. Processing Every Frame Is Often Wasteful
Neighbouring frames can be highly redundant. A person sitting still for ten seconds may produce hundreds of nearly identical images.
Video systems therefore sample, pool or compress time before or during modelling.
5. Uniform Temporal Sampling Is Simple and Blind
Taking one frame every second guarantees regular coverage but does not know where important actions occur.
A critical half-second event can fall between samples.
6. Event-Aware Sampling Spends Tokens Where Change Happens
Motion, scene changes, audio cues or learned saliency can guide denser sampling around important moments.
Adaptive sampling converts attention into a representation budget.
7. Image Patches Can Be Repeated Across Time
A straightforward video transformer can divide every selected frame into image patches and treat each patch as one visual token.
But multiplying spatial patches by many frames causes token counts to rise quickly.
8. Video Token Count Is Spatial Resolution × Temporal Resolution
If one frame produces 576 patch tokens and a clip uses 32 frames, the raw visual sequence can exceed eighteen thousand tokens before special tokens or text are added.
Video is therefore a representation-budget problem by construction.
9. Tubelets Combine Space and Time
A tubelet is a small three-dimensional block extending across height, width and several frames. Instead of tokenising one 2D patch independently at each moment, the model receives a local spatiotemporal unit.
The token can therefore encode both appearance and short motion.
10. Tubelet Size Controls Motion Granularity
Short tubelets preserve rapid temporal changes. Longer tubelets compress more frames and can blur distinct actions inside one unit.
The same scale trade-off seen in subwords, image patches and time-series windows reappears here.
11. Sparse Video Tubes Reduce Redundant Compute
Google Research’s Sparse Video Tubes work demonstrated a strategy that sparsely samples learnable 3D tube shapes at larger spatial and temporal strides, reducing token count while retaining joint spatial-temporal information.
The important principle is broader than one architecture: not every location in every frame deserves equal computational resolution.
12. Different Tube Shapes Preserve Different Information
A tube can emphasise spatial detail, long temporal extent or a balance of both.
Token shape itself becomes a prior about what kinds of change matter.
13. Position Must Include Time
Two identical patches appearing in different frames do not have identical roles. One may occur before a door opens and another after.
Temporal position is therefore part of token meaning.
14. Absolute Time and Relative Time Serve Different Jobs
Absolute timestamp answers when in the clip? Relative timing answers how long before or after another event?
Action recognition, causal inference and retrieval can require both.
15. Motion Is a Relationship Across Frames
A moving object cannot be defined from one frame alone. Motion emerges from changes in position, shape or appearance through time.
Motion is therefore a contextual relation rather than a primitive pixel property.
16. Optical Flow Is One Explicit Motion Representation
Optical flow estimates apparent image motion between frames. It can provide a dense movement field that is easier to analyse for some tasks than raw images.
But flow is an inferred representation and can fail under occlusion, lighting change or texture ambiguity.
17. Tracked Objects Are Higher-Level Video Tokens
A tracked object ID can connect the same car, person or ball across many frames.
This compresses thousands of patch observations into one persistent entity trajectory.
18. Tracking Adds Identity Claims
If a tracker switches identities between two people, downstream reasoning can assign actions to the wrong person.
Object persistence must therefore carry confidence and source timestamps.
19. Actions Are Coarser Temporal Tokens
“Open door”, “pour water” or “score goal” can span many frames and involve several objects.
Action tokens compress motion into semantic event categories.
20. Action Boundaries Are Often Fuzzy
When exactly does “standing up” begin—the first muscle movement, the moment the torso rises or the moment the person is fully upright?
Human-labelled action segments contain interpretive boundary choices.
21. Scene Cuts Create Strong Temporal Boundaries
A hard edit can jump instantly to another place or time.
Treating pixels before and after the cut as one continuous physical motion is misleading.
22. Shots Are Natural Video Chunks
A shot groups frames recorded continuously between edits. It is often a better retrieval unit than an arbitrary fixed-duration interval.
Shot detection is therefore a form of video segmentation.
23. Scenes Are Coarser Than Shots
Several shots can belong to one scene or narrative episode.
Higher-level grouping helps long-video reasoning while depending on semantic inference.
24. Long Videos Create Context Explosion
A two-hour video sampled densely can produce orders of magnitude more visual tokens than an image or short text prompt.
No practical system should assume every frame deserves equal attention.
25. Hierarchical Video Representation Solves Different Scales Separately
A model can represent local patches, short clips, shots, scenes and a whole-video summary simultaneously.
This lets coarse tokens route attention toward fine evidence when needed.
26. Summaries Should Preserve Timestamp Return
A statement such as “the cyclist fell” is useful only if the system can return to the seconds that support the claim.
Long-video compression needs provenance, not merely semantic density.
27. Audio Is a Parallel Temporal Channel
Speech, music, impact sounds and silence can disambiguate what the image stream alone cannot.
Video understanding is often audio-visual token alignment rather than vision alone.
28. Subtitles Are Another Representation, Not Ground Truth
Subtitles can be missing, mistranscribed, delayed or summarised.
They are useful evidence but should not replace the original audio-visual timeline.
29. OCR Creates Text Tokens Inside Video
Signs, slides, captions and scoreboards can be converted into text and aligned to timestamps.
This creates a bridge from visual tokenisation to lexical retrieval.
30. OCR Text Needs Temporal Lifetime
A sign may be visible for three seconds and then disappear. The extracted text should remain linked to that interval rather than treated as globally true for the entire video.
31. Video Retrieval Needs Event-Level Indexing
Users rarely want “the video”. They want the moment when the experiment failed, the lecturer defined a term or the player scored.
Timestamped event and shot tokens make long media addressable.
32. Similar Frames Are Not Similar Events
Two frames showing a closed door can belong to “about to open” and “just closed”.
Temporal context changes interpretation even when visual appearance is nearly identical.
33. Temporal Order Can Carry Causal Evidence
Ball strikes glass before glass shatters is different from the reverse ordering.
Chronology constrains explanation, although it still does not prove causality by itself.
34. Adaptive Tokenisation Can Learn Where Compute Belongs
Google Research’s TokenLearner learns to extract a small set of informative visual tokens rather than relying only on fixed hand-designed patch partitions.
This points toward a broader principle explored in the companion article on adaptive tokenisation: representation boundaries themselves can become learned decisions.
35. Compression Must Be Task-Aware
A sports highlight model can discard most idle footage. A safety investigation cannot discard the quiet seconds that show how a hazard developed.
The same video deserves different token budgets for different receiver jobs.
36. Generated Video Adds a Reconstruction Problem
Discrete or latent video tokens can be predicted and decoded back into frames.
Visual plausibility does not guarantee physical consistency, identity persistence or factual fidelity.
37. Physics Is a World-Return Test
An object should not teleport, change identity without cause or violate the constraints relevant to the intended world.
Video generation is strongest when representation is checked against temporal and physical coherence rather than frame beauty alone.
38. The Video Tokenisation Audit
- What event or task must the video representation support?
- What original frame rate and resolution exist?
- How densely is time sampled?
- Are tokens 2D patches, tubelets, objects, actions or learned latent units?
- What temporal span does one token cover?
- How is position represented across space and time?
- Are scene cuts treated as boundaries?
- Can objects be tracked across frames without identity switching?
- Are audio, OCR and subtitles aligned to timestamps?
- How is long-video context compressed hierarchically?
- What events could sparse sampling miss?
- Are generated or inferred frames distinguished from observed frames?
- Can every summary claim return to timestamped evidence?
- Does the final representation preserve the temporal distinction the receiver actually needs?
39. What Students Should Remember
- Video adds time to visual representation.
- Frame rate is an upstream temporal discretisation.
- Tubelets combine local space and time.
- Dense video tokens become expensive quickly.
- Objects, actions, shots and scenes are higher-level temporal units.
- Adaptive sampling can spend compute on informative moments.
- Audio and OCR provide parallel evidence streams.
- Long-video compression should preserve timestamp return.
40. The Deep Principle
A video token is a wager about what part of change deserves to survive. Too fine, and the model drowns in nearly identical moments. Too coarse, and the event disappears between the cuts.
Video intelligence is not seeing more frames. It is preserving the right changes across time, then keeping a route back to the moment where the world actually showed them.