Structured data already arrives with boundaries. Rows, fields, columns, keys, types and schemas tell the system where one unit ends and another begins. The challenge is not merely to tokenize the text inside the table, but to preserve the structure that gives each value its meaning.
A cell containing 42 is almost meaningless alone. It becomes useful when the system knows the row identity, column name, unit, data type, timestamp and source. Structured-data tokenisation therefore operates above ordinary text tokenisation.
This article continues the eduKateSingapore Representation and Tokenisation series.
The Structured Data Route
SOURCE SYSTEM → SCHEMA → RECORDS / ROWS → FIELDS / COLUMNS → TYPES + UNITS + IDS → SERIALISATION OR TABLE ENCODING → MODEL-READY TOKENS / EMBEDDINGS → REASONING / PREDICTION / RETRIEVAL → STRUCTURED OUTPUT → VALIDATION
1. Structure Is Already Metadata
A table tells us which values belong together. A JSON object tells us which keys name which values. A database schema declares field types and relationships.
Flattening all of this into plain text can preserve characters while destroying meaning.
2. Rows Usually Represent Instances
One row may represent a student, transaction, patient visit, product or sensor reading.
The row is therefore a higher-level token-like unit.
3. Columns Represent Attributes
A column name such as price, score or timestamp tells the receiver what the value means.
Column identity is part of every cell’s context.
4. A Cell Is Not Meaningful Without Its Schema
The value 17 can mean age, quantity, temperature, score or an identifier fragment.
Schema information restores semantic type.
5. Data Types Preserve Operations
An integer can be added. A categorical code should not necessarily be added. A date can be ordered chronologically. A string may need lexical comparison.
Type determines which transformations are valid.
6. Numeric-Looking Fields Can Be Identifiers
Phone numbers, postal codes, school IDs and account numbers can contain only digits while functioning as categorical identifiers.
Converting them to numbers can strip leading zeros or invite meaningless arithmetic.
7. Units Belong With Numeric Fields
A measurement column should preserve whether values are metres, kilograms, dollars or seconds.
Separating a number from its unit creates false comparability.
8. Dates Need Typed Temporal Representation
Storing dates as strings can preserve display while making ordering and interval arithmetic harder.
A structured system can store a canonical date or timestamp and render the user’s preferred format separately.
9. Missing Values Need Their Own Semantics
Missing can mean unknown, not measured, not applicable, redacted or delayed.
One generic null marker can hide important distinctions.
10. Zero Is Not Missing
A score of zero is evidence. A missing score is absence of evidence.
Replacing nulls with zero without a justified model can corrupt the representation.
11. Empty String Is Not Always Null
An empty string can represent a deliberately blank field while null means no recorded value.
Structured systems should preserve the distinction when it matters.
12. Primary Keys Create Stable Row Identity
A primary key gives each record a machine identity independent of its display name.
This is entity tokenisation at the table level.
13. Foreign Keys Encode Relationships
A student row can point to a class row; an order can point to a customer; a transaction can point to an account.
Relational structure survives through stable IDs rather than repeated names.
14. Joins Are Representation Reconstruction
Normalised databases distribute one logical object across several tables. A join reconstructs the combined view when needed.
The stored representation is deliberately decomposed.
15. Denormalisation Trades Repetition for Simpler Reads
Repeating customer information on every order can make one record self-contained while increasing duplication and update risk.
Database normalisation and model tokenisation solve different problems but share the same representation economics.
16. Wide Tables Create Feature Explosion
A table with thousands of columns can make one row extremely high-dimensional.
Feature selection, grouping and learned embeddings compress the representation.
17. Long Tables Create Sequence Explosion
Millions of rows cannot be serialized into one language-model context.
Filtering, aggregation, retrieval and query execution should happen before prompt assembly.
18. SQL Is a Structured Retrieval Language
A SQL query can select, filter, join and aggregate data without exposing every row to a language model.
Tool use preserves formal data operations while reducing token cost.
19. Serialising a Table as CSV Preserves Some Structure
CSV records rows and columns compactly, but escaping, quoting, delimiter choice and missing values can create ambiguity.
The parser must know the serialization contract.
20. JSON Preserves Keys More Explicitly
JSON repeats field names and can represent nested structures. This improves self-description while increasing text length.
Human/model readability trades against token efficiency.
21. Repeated Keys Consume Language-Model Tokens
Serializing 10,000 records with the key student_id repeated 10,000 times is wasteful if the receiving model already knows the schema.
Schema-once representations can reduce context cost.
22. Positional Table Formats Are More Compact and More Fragile
A row such as [17, 82, 2026-09-05] is compact only if the receiver knows exactly which column each position means.
Compression removes labels and increases dependence on schema alignment.
23. Column Order Should Not Become Accidental Meaning
A database query can return columns in a different order while representing the same record.
A model should rely on field identity rather than memorised column position when schemas can change.
24. Schema Names Are Semantic Tokens
gross_margin communicates more than field_17.
Descriptive schema names improve machine and human interpretability.
25. Schema Descriptions Add Context
A column called score is ambiguous without range, assessment type or unit.
Data dictionaries preserve the semantics that bare field names omit.
26. Schema Versioning Is Essential
If status=1 means active in one version and verified in another, historical data becomes misleading without version context.
Schema changes are representation migrations.
27. Category Codes Need Lookup Tables
A value such as 3 can represent “Secondary 3” or “high risk” depending on the field.
Category IDs should never be interpreted numerically without the codebook.
28. One-Hot Encoding Creates Explicit Category Dimensions
Classical machine learning can represent each category as a separate binary feature.
This removes false numeric ordering but becomes expensive for high-cardinality fields.
29. Categorical Embeddings Compress Large Vocabularies
High-cardinality categories such as product IDs or user IDs can map to learned vectors.
The stable ID preserves identity; the embedding captures learned similarity.
30. Continuous Features Need Scaling
Income, temperature and distance can have very different numeric ranges.
Standardization or normalization can improve model optimisation while requiring stored transformation parameters for world return.
31. Binning Converts Continuous Fields Into Tokens
Age can become child, teen, adult or senior; income can become quantile bands.
Binning creates discrete categories while sacrificing within-bin detail.
32. Learned Bins Can Optimise Task Performance
Instead of fixed thresholds, a model can learn where numeric boundaries should fall.
The categories become task-dependent rather than universal.
33. Tabular Models Need Column Identity
Unlike text, swapping two columns should not silently change which feature a value represents.
Column embeddings, named fields or fixed schema positions preserve identity.
34. Row Order Is Often Irrelevant
Many tables represent a set of independent records. Reordering rows should not change the dataset’s meaning.
Serialising rows as one long sequence can accidentally introduce order where none exists.
35. Some Row Order Is Meaningful
Transaction histories and time-series tables use row order or timestamps to represent sequence.
The schema must declare whether order is semantic.
36. Table Relationships Can Become Graphs
Foreign keys connect rows across tables. Those relationships can be projected into a graph for relational reasoning.
See Graph Tokenisation.
37. Event Tables Can Become Sequences
Rows containing timestamped activity can be treated as event tokens and ordered into histories.
See Event Tokenisation.
38. Sensor Tables Can Become Time Series
Repeated measurements indexed by time can be windowed and modelled as temporal sequences.
39. Structured Data Should Usually Be Queried Before Being Prompted
If a database contains a million rows, the model should not receive all million as text. SQL, filters, aggregations and retrieval should reduce the data to the evidence relevant to the current question.
Formal tools protect context and numerical fidelity.
40. Schema-Aware Prompting Reduces Ambiguity
When structured data does enter an LLM context, include enough schema information to explain field meaning, units and category codes.
A self-describing representation costs more tokens and prevents larger semantic errors.
41. Generated Structured Output Needs Validation
A model can produce syntactically valid JSON with the wrong fields, wrong units or impossible values.
Schema validation should check types, required fields and allowed ranges.
42. Constraints Should Be Enforced Outside Prose
If age must be non-negative or status must come from a fixed enum, formal validators can enforce those rules more reliably than descriptive instructions alone.
Representation structure should carry operational constraints.
43. Provenance Should Travel With Rows
A value can be accurate and still unusable if its source, date or collection method is unknown.
Metadata preserves why the system should trust a field.
44. Structured Data Quality Is Multi-Dimensional
Accuracy, completeness, consistency, timeliness, validity and uniqueness describe different failure modes.
A table can be perfectly tokenised and still contain bad data.
45. Schema Drift Can Break AI Pipelines
A renamed field or changed unit can cause a model or tool to interpret values incorrectly.
Detect schema changes before they become silent representation errors.
46. Structured Data Can Be More Reliable Than Prose for Exact Facts
A typed database field can preserve a price or timestamp more precisely than a paragraph describing it.
Strong AI systems route factual operations to the representation best suited to the task.
47. Prose Is Better for Context and Explanation
Tables are excellent at exact fields and comparison. Prose explains causes, assumptions and meaning.
Multirepresentational systems use each where it is strongest.
48. Structured Data Tokenisation Audit
- What does one row or record represent?
- What stable key identifies it?
- What does each field mean?
- What data type applies?
- What units or category codebooks are required?
- How is missingness represented?
- Does row or column order carry meaning?
- What foreign-key relationships exist?
- What schema version produced the record?
- How is the table serialized for models?
- Which repeated labels can be compressed safely?
- Should the data be queried or aggregated before prompting?
- What provenance accompanies each value?
- How are generated outputs schema-validated?
- Can every model result return to authoritative rows?
49. What Students Should Remember
- Structured data already contains semantic boundaries.
- Rows, fields and schemas are higher-level tokens.
- Values require column names, types and units for meaning.
- Missing is not the same as zero.
- Primary and foreign keys preserve identity and relationships.
- Serialisation trades self-description against token efficiency.
- Formal queries should filter large datasets before LLM prompting.
- Generated structured output should be validated against schema.
50. The Deep Principle
Structured data demonstrates that tokenisation does not always begin by discovering boundaries. Sometimes the boundaries are already there, and the real job is to keep them intact as information moves between databases, models and human explanations.
A table cell is not a fact by itself. The fact emerges from value + field + type + unit + row identity + provenance. Good structured-data tokenisation keeps that bundle together.