Data Serialisation and File Formats | JSON, CSV, Parquet, Avro, Encoding, Compression and Interchange

Data serialisation is the process of converting structured information into a representation that can be stored, transmitted or reconstructed later. File formats define how that representation is organised so different systems can read the same bytes with the same intended meaning.

A format is not merely a container. It is a contract between the writer of data and every future reader that must reconstruct the intended structure.

Formats sit at the boundary between data and software. A table may become CSV, JSON, Parquet or another representation. Each choice changes portability, precision, speed, schema control, compression, streaming behaviour and long-term readability. Good data management therefore treats format choice as an architectural decision, not an export afterthought.

ARTICLE ID: DATA.MANAGEMENT.043
Canonical function: data representation for storage, transport and machine interoperability
Owner boundary: this article owns serialisation, encoding and file-format trade-offs. Data Integration and Interoperability owns system-to-system exchange; Data Contracts and Data Products owns producer–consumer promises; Data Lakes and Lakehouses owns analytical object-storage architecture.

The Simple Answer

A serialisation route is:

Logical Data → Schema → Encoding Rules → Bytes / Text → Storage or Transport → Reader → Decode → Validate → Reconstruct Logical Data

Trust depends on both sides agreeing about the format and enough schema or metadata surviving to reconstruct meaning correctly.

Logical Data vs Physical Representation

A customer record with ID, name and balance is a logical structure. CSV, JSON and Parquet are different physical representations of that structure.

Changing format does not automatically change business meaning, but some formats preserve types and relationships better than others.

Text Formats

Text formats are human-readable or semi-readable representations built from characters.

CSV;
TSV;
JSON;
XML;
line-delimited JSON;
plain text with conventions.

They are easy to inspect and exchange, but may be verbose and can leave type interpretation to the reader.

Binary Formats

Binary formats encode data into machine-oriented byte structures. They can preserve types more compactly and support faster parsing or richer schemas.

Examples include Avro-style row formats, Protocol Buffers-style messages and columnar analytical formats such as Parquet.

CSV

CSV is simple, widely supported and excellent for flat tabular interchange. Its strengths are also its limits.

easy to open;
easy to generate;
works across many tools;
good for simple tables;
poor at nested structures;
does not inherently preserve data types;
requires conventions for quoting, delimiters and nulls.

CSV Is More Ambiguous Than It Looks

A value such as 01/02/03 can be interpreted as text or several possible dates. 00123 may be an identifier that must preserve leading zeros rather than an integer.

A CSV file should therefore travel with a schema or data dictionary when types matter.

Delimiter and Quoting Rules

CSV fields can contain commas, quotes and line breaks. Producers and consumers must agree on escaping and quoting rules.

Naive splitting on commas can corrupt legitimate text.

Headers

Header names help identify columns but do not fully define them. A column named amount still needs unit, currency, nullability and semantic definition.

JSON

JSON represents objects, arrays, strings, numbers, booleans and nulls. It is widely used in APIs and event systems because it is flexible and easy to inspect.

JSON can represent nested data naturally, but its flexibility means schema discipline must come from contracts or validation rather than the syntax alone.

JSON Numbers

Numeric interpretation can differ between languages and runtimes, especially for very large integers and decimal precision.

Financial or scientific values may need explicit string, decimal or schema conventions to avoid precision loss.

Dates in JSON

JSON has no dedicated universal date type. Dates are commonly represented as strings under an agreed convention.

The contract should state timezone, precision and whether the value represents event time, effective time or another clock.

Null vs Missing

In many serialised structures, an absent field and a field explicitly set to null can mean different things.

The producer–consumer contract should state whether missing means “unknown”, “not applicable”, “default value” or “not supplied”.

XML

XML supports nested structure, namespaces, attributes and formal schemas. It remains important in many standards-heavy and enterprise domains.

Its verbosity can be a disadvantage for high-volume data, but explicit namespaces and mature schema tooling can be valuable in complex interchange.

Row-Oriented Formats

Row-oriented representations store the fields of one record together. They are well suited to message transport, event logs and workloads that read complete records.

Avro-style serialisation is a common example of schema-aware row representation used in data pipelines.

Columnar Formats

Columnar formats store values from the same column together. Analytical queries often read only a small subset of columns across many rows, making columnar layout efficient for scan-heavy workloads.

Parquet is a widely used example in analytical data estates.

Why Parquet Works Well for Analytics

column pruning reduces unnecessary reads;
similar values compress efficiently;
typed schemas are preserved;
nested structures can be represented;
metadata can support query pruning.

Parquet is less convenient than JSON for small human-inspected messages. Format selection should follow the job.

Avro-Style Schema-Aware Records

Schema-aware row formats can separate the schema from the encoded records and support controlled schema evolution.

This is useful in event and streaming environments where messages must remain compact while old and new consumers coexist.

Schema Evolution

Serialised data can outlive the software version that created it. Schema evolution therefore matters.

adding optional fields;
adding required fields with defaults;
renaming fields;
changing types;
removing fields;
changing enum values;
changing business semantics without schema change.

Compatibility should be tested against real consumers and historical files.

Backward Compatibility

Backward compatibility generally means a newer reader can interpret older data.

This matters when historical records remain in storage for long periods.

Forward Compatibility

Forward compatibility generally means an older reader can tolerate data written under a newer schema.

This is useful during rolling upgrades where not every consumer updates at once.

Semantic Compatibility

A schema can remain technically compatible while meaning changes. A field called status can keep the same string type while its allowed values or business interpretation changes.

Technical schema compatibility cannot replace semantic governance.

Schema Registries

A schema registry stores versions of message or record schemas and can enforce compatibility rules before new producers deploy.

Registries are especially useful for streaming and shared interfaces, but the business meaning of fields still needs documentation and ownership.

Character Encoding

Text becomes bytes through character encoding. UTF-8 is widely used because it can represent Unicode text efficiently and interoperably.

If the reader assumes the wrong encoding, characters can become corrupted even when the original bytes were transmitted correctly.

Unicode Normalisation

Visually identical text can sometimes have different underlying Unicode sequences. Normalisation can improve matching and deduplication.

Normalisation should be applied carefully where exact original representation matters.

Byte Order

Some binary numeric encodings depend on byte order. Modern libraries usually abstract this, but low-level interchange specifications should state endianness where relevant.

Compression

Compression reduces storage and transfer size.

whole-file compression;
block compression;
column-level compression;
dictionary encoding;
run-length encoding;
delta encoding.

Compression should be evaluated alongside CPU cost, random access and splittability for parallel processing.

Compression Is Not Encryption

Compressed data is still readable by anyone who can decompress it. Compression does not provide confidentiality.

Splittable Formats

Large analytical files can be processed more efficiently when different workers can read independent blocks without scanning from the beginning.

Splittability matters for distributed processing.

Record Boundaries

Line-delimited formats create explicit record boundaries, which can simplify streaming and parallel ingestion.

A single enormous JSON array may be harder to process incrementally than one JSON object per line.

Self-Describing Formats

Some formats carry field names or schema metadata with the data. This improves portability but increases size.

Other formats rely on an external schema, producing smaller messages but stronger dependence on schema governance.

Human Readability vs Machine Efficiency

Human-readable formats are excellent for debugging, teaching and simple interchange. Machine-efficient formats reduce cost and latency at scale.

Many estates use both: JSON at APIs, Parquet in analytics, CSV for simple exports and schema-aware binary messages in streaming.

File Extensions Are Not Proof

A file named .csv may contain malformed or unexpected content. A robust ingestion pipeline validates actual structure and, where appropriate, content type rather than trusting the filename alone.

Magic Bytes and Signatures

Some binary formats begin with identifying byte sequences. These can support format detection, though they do not prove the file is safe or semantically correct.

File-Level Metadata

Useful metadata can include:

format;
schema version;
row count;
creation time;
producer;
compression;
checksum;
partition;
classification;
source dataset version.

This metadata improves validation and lineage.

Checksums

Checksums help detect corruption during transfer or storage.

A checksum should be calculated over a defined representation. Reformatting equivalent JSON can change the bytes and therefore the checksum even if logical content remains equivalent.

Canonical Serialisation

Some systems need a deterministic canonical representation so logically equivalent data produces the same byte sequence for signing, hashing or comparison.

Canonicalisation rules should be explicit because ordinary serializers may vary field order or whitespace.

Decimal Precision

Financial and scientific data often require exact decimal semantics. Binary floating-point representations can introduce small approximations.

The format and schema should state precision and scale where exact decimal behaviour matters.

Units

A number without a unit is incomplete when the domain depends on measurement. Formats rarely solve this by themselves.

Units belong in schema, metadata or semantic contracts.

Enumerations

Enumerated codes need controlled definitions. A code 3 has no meaning without a reference vocabulary and version.

See Master Data and Reference Data.

Nested Data

JSON and some binary or columnar formats represent nested arrays and objects naturally. Flattening nested data into CSV can duplicate parent values or require multiple related files.

The best representation depends on whether hierarchy is central to the receiver’s job.

Wide vs Long Representations

The same logical data can be represented in wide or long form. Wide tables use many columns; long tables use repeated attribute rows.

Format choice should preserve the model expected by downstream tools rather than forcing arbitrary transformations at every handoff.

File Size and Partitioning

Thousands of tiny analytical files can create more metadata overhead than a smaller number of well-sized files.

One enormous file can make parallel processing and selective access inefficient. File sizing is an operational trade-off.

Long-Term Preservation

For long-lived data, prefer formats with durable specifications, broad tooling and enough metadata to reconstruct meaning later.

Proprietary formats may still be necessary, but preservation planning should account for dependency on specific software.

Open Formats

Openly documented formats can reduce dependency on one vendor and support long-term interoperability.

Open specification alone does not guarantee quality, security or universal tool compatibility.

Data Exchange

When data crosses organisational boundaries, the exchange package should include more than a file:

format specification;
schema;
field definitions;
reference codes;
encoding;
compression;
checksum;
version;
rights and classification;
contact or owner.

APIs and Serialisation

APIs frequently use JSON or binary message formats. The interface contract should define the representation and compatibility behaviour.

See Data APIs and Data Services.

Streaming and Serialisation

Streaming systems benefit from compact, schema-aware formats where millions of messages may be exchanged continuously.

Historical consumers may replay old messages, so schema evolution is especially important.

See Data Streaming and Event-Driven Systems.

Lakehouse Formats

Lakehouses commonly use columnar files beneath table metadata because analytical readers benefit from efficient scans and typed schemas.

The file format stores data; the table abstraction manages snapshots, commits and schema evolution across files.

Security

File formats can contain embedded scripts, formulas, macros or external links. Ingestion should treat complex file types according to their security risk.

CSV and spreadsheet exports can also create formula-injection risks when opened in interactive tools if untrusted fields are interpreted as formulas.

Privacy

Changing format does not remove sensitivity. Converting a restricted table from a database into CSV usually makes it easier to copy, not safer.

Classification and access should follow the data, not the extension.

Validation

Validation should check both syntax and semantics:

file parses successfully;
required fields exist;
types are valid;
encoding is correct;
row counts match expectations;
reference codes are valid;
checksums match;
schema version is supported;
business invariants hold.

Testing Format Evolution

Tests should include old files, new files, mixed-version consumers, unexpected nulls, large values, multilingual text, precision boundaries and corrupted records.

See Data Testing and Reliability Engineering.

Education Example

An education platform exports student attendance for analysis. A CSV version is suitable for a small controlled spreadsheet workflow, while a typed columnar version is used for long-term analytics. Both carry the same canonical field definitions and classification.

The identifier column is explicitly typed as text so leading zeros are not lost when opened in spreadsheet software.

Research Example

A research dataset uses a durable open tabular format plus a machine-readable schema, codebook, units and checksums. The representation is chosen so future researchers can reconstruct both values and meaning without the original laboratory software.

Common Failure Modes

Extension equals schema: a filename is trusted without validation.
CSV simplicity illusion: types, nulls and quoting are ambiguous.
JSON flexibility without contract: every producer emits different shapes.
Precision loss: large identifiers or decimals pass through unsuitable numeric types.
Encoding mismatch: text corrupts across systems.
Schema compatible, meaning broken: semantic change goes undetected.
Compression equals security: confidential data remains exposed.
One format for every job: human interchange and analytical storage are forced into the same representation.
Proprietary archive without plan: future readers depend on obsolete software.
Format conversion drops metadata: units, codes or rights disappear.

A Format Selection Checklist

What logical structure must be represented?
Who will read the data?
Is human readability important?
Is the data flat or nested?
Which types and precision must survive?
How large is the dataset?
Will it be streamed or batch processed?
Is column pruning important?
What compression is appropriate?
How will schemas evolve?
Are old readers expected to remain compatible?
Which encoding applies?
What metadata and checksums accompany the file?
What security and privacy risks follow the format?
Can a future reader reconstruct the intended data without hidden vendor knowledge?

A Maturity Ladder

Exported: data can be written into files.
Specified: encoding, delimiter and structure rules are documented.
Typed: schemas preserve data types and constraints.
Versioned: schema evolution is controlled.
Validated: syntax, semantics and checksums are tested.
Interoperable: formats fit APIs, streams and analytical systems intentionally.
Preservable: long-lived representations have durable specifications and metadata.
Adaptive: workload, tooling and receiver needs drive deliberate format evolution.

The Deeper Principle: Bytes Need a Shared Interpretation

Every stored or transmitted dataset eventually becomes bytes. Bytes alone are not meaningful. Meaning reappears only when a future reader has the rules required to reconstruct the intended structure, types, units, codes and relationships.

Good serialisation therefore preserves more than data values. It preserves the possibility that another system, another team or another decade can read the representation without silently inventing what the bytes were meant to mean.

Data Management Series

Final idea: serialisation is the discipline of making data reconstructible outside the memory of the system that created it. A good format carries enough structure and governance that moving the bytes does not destroy the meaning.