Robots do not act in words. They act through motion, force, position, velocity, gripper state and control signals. Robotics action tokenisation is the representation problem of turning those continuous physical commands into model-ready units that can be predicted, sequenced and executed safely.
The action token can be as fine as a small change in joint angle or as coarse as “pick up the cup”. The correct granularity depends on the robot, controller and task.
This article continues the eduKateSingapore Representation and Tokenisation series and connects the representation stack to physical action.
The Robotics Action Route
WORLD OBSERVATION → VISION / PROPRIOCEPTION / LANGUAGE → INTERNAL REPRESENTATION → ACTION POLICY → CONTINUOUS OR DISCRETE ACTION TOKENS → LOW-LEVEL CONTROLLER → MOTORS / GRIPPER → PHYSICAL CHANGE → SENSOR FEEDBACK → WORLD RETURN
1. An Action Token Is a Command Representation
A model may output a token that corresponds to moving the end effector left, closing the gripper or increasing wrist rotation.
The token is not the physical movement itself. It is an instruction interpreted by a controller.
2. Robots Often Begin With Continuous Action Spaces
Joint position, torque, velocity and Cartesian pose are naturally continuous-valued controls.
To use next-token style prediction, those values can be quantised into finite bins.
3. Quantisation Creates Discrete Action Vocabulary
A continuous translation value can be divided into bins such as strongly negative, slightly negative, zero, slightly positive and strongly positive.
Each bin becomes one action ID.
4. Finer Action Bins Preserve Control Resolution
More bins allow smaller command differences but increase the output vocabulary and can require more data to learn each action reliably.
Coarser bins simplify prediction while making motion less precise.
5. RT-1 Demonstrated Tokenised Robot Actions at Scale
RT-1: Robotics Transformer for Real-World Control at Scale represented robot actions through discretised dimensions and learned them alongside visual and language-conditioned context.
The larger lesson is architectural: continuous robot control can be expressed as a finite sequence of predictable action symbols.
6. RT-2 Made the Token Bridge Even More Explicit
RT-2 expressed robotic actions as text-like tokens so vision-language-action models could train on web-scale visual-language data and robot trajectories inside a more unified output format.
This does not make physical action equivalent to language. It creates a shared symbolic interface.
7. The Shared Interface Is Powerful Because It Reuses Sequence Modelling
A model already capable of predicting token sequences can be extended to predict commands represented as tokens.
Representation convergence allows one architecture to bridge perception, language and action.
8. But Action Tokens Have Physical Consequences
A wrong word can confuse a reader. A wrong motion command can damage equipment or injure someone.
Embodied token generation therefore requires stronger safety boundaries than ordinary text output.
9. Low-Level Controllers Constrain the Token
A model can request a motion while a controller enforces joint limits, velocity limits and stability constraints.
The token should not directly bypass the safety layer.
10. Joint-Space and Cartesian-Space Actions Are Different Representations
A robot can command individual joint angles or command the end effector to move in x, y, z and orientation.
Both can produce similar physical movement through different control representations.
11. Cartesian Actions Are More Task-Intuitive
“Move the hand 2 cm upward” is closer to task geometry than “rotate joint 3 by 0.04 radians”.
Inverse kinematics can translate Cartesian intent into joint-level control.
12. Joint Actions Are Closer to Actuator Reality
Joint-level commands expose the actual degrees of freedom available to the robot.
They can be harder for high-level task models to reason about directly.
13. Gripper State Is a Discrete Action Naturally
Open, close or hold can often be represented through a small categorical vocabulary.
Other grippers require continuous force or aperture control.
14. Force Is Different From Position
Pressing, wiping or grasping fragile objects requires force control, not position alone.
An action tokenisation that ignores force can be adequate for free-space motion and unsafe for contact tasks.
15. Contact Is a State Transition
A robot can move through free space, make contact, slide, grasp and release.
These contact modes are higher-level action states above continuous motor commands.
16. Skills Are Coarser Action Tokens
Pick, place, open_drawer and pour can each represent multi-step motor behaviours.
Skill tokens compress trajectories into reusable action abstractions.
17. Skill Tokens Move Complexity Into the Skill Controller
A high-level model only needs to choose pick, but another controller must execute grasp planning, approach, closure and lift.
Coarser tokens reduce sequence length by delegating detail.
18. Trajectories Are Action Sequences
A trajectory records actions over time. Two trajectories reaching the same final pose can differ greatly in safety, speed and efficiency.
Final state alone is an incomplete representation of behaviour.
19. Temporal Resolution Controls Smoothness
Predicting one action every 20 milliseconds gives finer control than one every half-second.
Higher action frequency increases sequence length and controller bandwidth.
20. Action Chunking Predicts Several Steps Together
Instead of predicting one tiny control at a time, a policy can predict a short chunk of future actions.
This reduces decision frequency while increasing the consequence of one prediction.
21. Open-Loop Chunks Need Feedback Boundaries
If a robot executes a long action chunk without checking sensors, small errors can accumulate.
Closed-loop control repeatedly returns to observation.
22. Proprioception Is the Robot’s Internal State Stream
Joint positions, velocities and forces tell the policy what the robot itself is doing.
Action tokens without proprioceptive context can become physically incoherent.
23. Vision Grounds Action in the External World
Visual tokens provide object location, pose and scene context.
This creates the perception → action bridge linking 3D Tokenisation with embodied control.
24. Language Specifies Goals, Not Motor Details
“Put the red cup in the sink” defines an outcome. It does not specify joint angles.
A robotics model must translate between semantic goal representation and control representation.
25. Object-Centric Actions Are Another Granularity
Commands such as grasp(object_17) or place(object_17, region_4) combine entity tokens with action types.
This is more structured than free-form natural language.
26. Object IDs Need Stable Perception
If the vision system changes which cup is object_17 between frames, the action can target the wrong entity.
Identity tracking is part of action safety.
27. Coordinate Frames Must Be Explicit
A translation of +0.1 m means something different in world coordinates, camera coordinates and gripper coordinates.
Action tokens need a frame contract.
28. Relative Actions Can Generalise Better
Move left by 2 cm can transfer across workspaces more naturally than move to one absolute coordinate.
Relative action tokens encode change rather than location.
29. Absolute Actions Are Useful for Known Frames
Industrial workcells with calibrated fixtures can benefit from precise absolute targets.
Representation should match environmental stability.
30. Action Validity Is State-Dependent
Close_gripper is sensible when an object lies between the fingers and meaningless in empty space.
The same action token can be safe or unsafe depending on current state.
31. Constraints Should Be Enforced Outside the Generative Policy
Workspace boundaries, collision checks and emergency stops should be formal controls rather than optional text instructions.
Physical safety needs deterministic gates around probabilistic action generation.
32. Simulation Provides a Safer Evaluation Representation
Policies can be tested in simulation before physical execution.
But simulation is only an approximation of real friction, compliance, sensor noise and object variability.
33. Sim-to-Real Is a Representation Gap
A policy can succeed in simulated observations and fail when real sensor distributions differ.
World return is the final test of embodied representation.
34. Demonstrations Are Action Training Corpora
Robot trajectories collected from humans or existing policies create examples of observation → action sequences.
Dataset diversity determines which action tokens receive useful coverage.
35. Rare Recovery Actions Matter
Most demonstrations show successful behaviour. Real robots also need to recover from slips, collisions and failed grasps.
A dataset containing only happy paths creates brittle action vocabulary usage.
36. Action Tokens Need Temporal Provenance
For debugging, record observation timestamp, predicted action, safety-filtered action and executed action separately.
What the model proposed and what the robot physically did are different records.
37. The Robotics Action Tokenisation Audit
- What robot and controller interpret the action?
- Is the action space continuous or discrete?
- What dimensions are tokenised?
- How fine are the action bins?
- What coordinate frame applies?
- Are commands absolute or relative?
- What gripper, force and contact states exist?
- Is the token one control, a short chunk or a high-level skill?
- How often does the policy reobserve the world?
- What proprioceptive state is available?
- How are object identities grounded?
- What hard safety constraints filter actions?
- What simulation and real-world tests exist?
- Are predicted and executed actions logged separately?
- Can failures return to the exact sensory evidence that produced the command?
38. What Students Should Remember
- Robot actions are physical control representations.
- Continuous actions can be quantised into discrete tokens.
- Finer action bins preserve precision but enlarge the output space.
- Skills are coarse action tokens.
- Coordinate frames and robot state are part of action meaning.
- Vision and language ground actions in goals and objects.
- Safety constraints should sit outside probabilistic token generation.
- World return means the robot actually performs the intended safe action.
39. The Deep Principle
Action tokenisation closes the representation loop. The model no longer merely describes the world; its symbols can alter the world.
When a token becomes a motor command, representation fidelity becomes physical responsibility. The action vocabulary must preserve enough control to act usefully and enough constraint to act safely.