Three-dimensional geometry does not arrive as one natural sequence. A 3D scene can be represented as points, voxels, triangles, surfaces, objects or learned latent regions. Tokenisation in 3D is the problem of deciding which of those geometric pieces should become the units a model can address directly.
That choice matters because each representation preserves a different part of the world. Point clouds preserve sampled surface locations without a built-in mesh. Voxels impose a regular grid. Meshes encode explicit surface connectivity. Object-level regions compress many lower-level measurements into semantic units.
This article extends the eduKateSingapore Representation and Tokenisation series beyond text, images and spatial maps.
The 3D Representation Route
PHYSICAL WORLD → DEPTH / LiDAR / MULTIVIEW CAPTURE → COORDINATES + FEATURES → POINT CLOUD / VOXEL GRID / MESH → LOCAL PATCHES / REGIONS / OBJECTS → 3D TOKENS OR EMBEDDINGS → CONTEXTUAL MODEL → CLASSIFICATION / SEGMENTATION / PLANNING → ACTION OR RECONSTRUCTION → WORLD RETURN
1. Point Clouds Are Sets, Not Sentences
A point cloud is a collection of sampled 3D coordinates, often with colour, intensity or other features. The ordering of those points is usually arbitrary.
This creates a major difference from text: sequence order is not inherently meaningful.
2. Permutation Invariance Is a Core Requirement
If the same set of points is listed in a different order, the represented shape should remain the same. Models that consume raw point sets therefore need permutation-invariant or permutation-equivariant structure.
PointNet made this requirement explicit by designing a network that directly consumes point sets while respecting their unordered nature.
3. One Point Is Not One Object
A single LiDAR point might land on a car door, a tree trunk or a wall. Object identity emerges only from relationships among many points.
Fine geometric tokens need contextual aggregation before semantic meaning appears.
4. Sampling Density Changes Representation
Nearby surfaces may contain many points while distant surfaces contain few. Different sensors and distances produce different densities.
Point count is therefore not a direct measure of object size or importance.
5. Downsampling Is 3D Compression
Large point clouds can contain millions of points. Downsampling reduces compute but can erase thin structures, small objects or sharp boundaries.
The right density depends on the smallest geometry the receiver must still recognise.
6. Farthest-Point and Voxel Sampling Create Different Biases
Some sampling methods spread retained points across the shape; others select one representative per spatial cell.
The sampling algorithm determines which local detail survives.
7. Local Neighbourhoods Are Natural 3D Chunks
Nearby points can be grouped into neighbourhoods using radius, k-nearest-neighbour or learned region rules.
These local regions become higher-level token-like units.
8. Neighbourhood Size Controls Geometric Granularity
Small neighbourhoods preserve fine curvature and edges. Large neighbourhoods capture object-level shape while smoothing local detail.
The same granularity trade-off reappears across every modality.
9. Voxels Turn 3D Space Into a Regular Grid
Voxelisation divides three-dimensional space into cubic cells. Each cell can store occupancy, density, colour or learned features.
Voxel IDs become regular spatial tokens, making convolution and indexing easier.
10. Voxel Resolution Is Expensive
Doubling resolution along x, y and z can increase the number of grid cells by roughly eight times.
Fine 3D grids become memory-intensive very quickly.
11. Sparse Voxels Avoid Paying for Empty Space
Most large 3D scenes are mostly empty. Sparse voxel representations store only occupied or relevant cells.
This is representation efficiency through selective addressability.
12. Meshes Add Connectivity
A mesh represents surfaces through vertices, edges and faces, often triangles.
Unlike an unordered point cloud, a mesh explicitly tells the system which local points are connected.
13. Mesh Faces Are Higher-Level Surface Tokens
A triangle packages three vertices and their connectivity into one surface element.
Mesh tokenisation can therefore operate over vertices, faces, patches or whole components.
14. Mesh Quality Affects Model Quality
Irregular triangles, holes and non-manifold geometry can create unstable neighbourhoods.
Clean geometry is upstream representation quality.
15. Surface Patches Compress Local Shape
Nearby faces can be grouped into patches based on curvature, connectivity or learned similarity.
A patch becomes a middle layer between triangles and objects.
16. Normals Add Orientation Information
A surface normal tells the model which way a local surface faces. Two identical coordinate patterns can represent different geometry if their orientations differ.
Position alone is not enough.
17. Coordinate Frames Are Part of the Contract
Robot coordinates, camera coordinates and world coordinates can describe the same point differently.
Transform matrices preserve the bridge among these frames.
18. Rotation Changes Coordinates Without Changing Shape
A chair rotated 90 degrees remains a chair while every point coordinate changes.
Models can learn rotation invariance, use canonical orientation or represent orientation explicitly depending on the task.
19. Scale Matters Too
A 3D model can be measured in millimetres, metres or arbitrary normalized units.
Removing absolute scale can help shape recognition and break tasks requiring real-world dimensions.
20. 3D Objects Are Coarse Tokens
A detected chair or vehicle can compress thousands of points into one object representation with class, pose and extent.
Object tokens are efficient but depend on correct detection and segmentation.
21. Instance Segmentation Protects Identity
Two adjacent chairs share one semantic class but remain two physical entities.
Instance IDs distinguish object identity from category.
22. Scene Graphs Add Relationships
Object tokens can be connected by relations such as on, inside, behind or adjacent_to.
The 3D scene then becomes a graph above the geometric token layer.
23. Occlusion Creates Missing Geometry
A camera or LiDAR cannot observe surfaces hidden behind other objects.
Missing points should not be mistaken for empty space.
24. Completion Is Inference
Models can infer hidden surfaces from learned priors, but reconstructed geometry is a hypothesis rather than direct observation.
World-return interfaces should distinguish measured from completed regions.
25. Registration Aligns Multiple Views
Several scans can be transformed into one shared coordinate frame.
Registration errors can create duplicated surfaces or false geometry even when each individual scan is accurate.
26. Temporal 3D Adds Motion
Repeated 3D frames create dynamic point clouds or 4D scenes. Tokens now require both spatial and temporal context.
A moving vehicle is not merely a series of unrelated object detections.
27. Tracking Creates Persistent Object Tokens
A tracked object ID connects the same physical entity across frames.
Identity persistence is a higher-level representation above per-frame geometry.
28. 3D Tokens Can Feed Robotics
Robots can use object poses, occupancy cells, grasps and trajectories as action-relevant spatial tokens.
This connects to Robotics Action Tokenisation.
29. Point Tokens and Image Tokens Can Cross-Walk
RGB-D and multimodal models can connect image pixels or patches with 3D points through camera calibration.
The same object can therefore have 2D and 3D representations simultaneously.
30. Texture and Geometry Are Different Channels
A surface can keep the same shape while colour changes. Another can keep the same colour while shape changes.
Models should not collapse appearance and geometry into one undifferentiated signal.
31. 3D Generation Needs a Decoder or Renderer
If a model predicts latent shape tokens, another system must reconstruct a mesh, point cloud, radiance field or other usable geometry.
Generation quality depends on both token prediction and representation return.
32. Plausible Geometry Is Not Measured Geometry
A generated chair can look convincing while dimensions are wrong.
Engineering, medical and survey applications require stronger geometric validation than creative rendering.
33. Coordinate Quantisation Creates Discrete 3D Tokens
Continuous coordinates can be binned into cells or quantised codebook entries so they can be predicted as discrete IDs.
Quantisation makes sequence modelling easier while sacrificing exact position.
34. Fine Quantisation Is Expensive
More bins preserve detail and expand the spatial vocabulary.
Coarser bins reduce token complexity and increase geometric error.
35. 3D Retrieval Needs Representation Choice
Searching by object class, point-cloud similarity, mesh shape or spatial relation produces different neighbours.
Retrieval quality depends on the representation matched to the query.
36. Provenance Matters in 3D
A scan should record sensor, timestamp, calibration, coordinate frame and processing history.
Without that receipt, a precise-looking geometry can be difficult to verify.
37. The 3D Tokenisation Audit
- What physical geometry is represented?
- Which sensor or reconstruction method produced it?
- What coordinate frame and units apply?
- Is the primitive representation points, voxels, mesh or another structure?
- What sampling density or voxel resolution is used?
- How are local neighbourhoods formed?
- Is point order treated as irrelevant?
- How are rotation and scale handled?
- How are missing and occluded regions represented?
- Are object identities tracked across time?
- What quantisation error is introduced?
- Can 2D and 3D observations be cross-walked?
- What provenance and calibration data survive?
- Can generated or inferred geometry be distinguished from direct measurement?
38. What Students Should Remember
- Point clouds are unordered sets.
- Voxels impose a regular 3D grid.
- Meshes preserve surface connectivity.
- Local patches and objects are higher-level tokens.
- Coordinate frames, scale and orientation are part of meaning.
- Occlusion creates missing evidence.
- Quantisation trades precision for discrete addressability.
- 3D world return requires calibrated geometry, not merely plausible shape.
39. The Deep Principle
Three-dimensional tokenisation decides how a continuous shape becomes a finite set of addressable geometric units. Every representation gains structure by choosing what detail to ignore.
The point is not the object, the voxel is not the room, and the mesh is not the physical surface. They are computational handles whose value depends on whether the geometry needed by the receiver survives the cut.