Data Integration and Interoperability

Data integration is the controlled work of bringing data from different sources into a usable relationship. Interoperability is the broader ability of separate systems, organisations or domains to exchange data and use what they receive correctly.

Moving data is easy. Preserving meaning across the move is the hard part.

An integration can be technically successful and still fail. A file can transfer, an API can return HTTP 200, an event can arrive, a pipeline can finish green—and the receiver can still misunderstand the data. The central problem is therefore not connection alone. It is connection with identity, context, semantics, timing, quality and authority intact.

ARTICLE ID: DATA.MANAGEMENT.009
Canonical function: cross-system exchange and semantic interoperability
Series route: What Is Data Management? → Data Architecture → Data Integration and Interoperability.

The Simple Answer

Suppose a school uses one system for enrolment, another for assessment, another for billing and another for parent communication. Each system can work perfectly on its own. The difficulty appears when the systems need to agree that the same learner is the same learner, the same class is the same class, and the same status carries the same meaning.

Integration connects the systems. Interoperability makes the connection usable.

Four Levels of Interoperability

1. Technical interoperability

Can systems physically exchange data? This level concerns networks, protocols, APIs, files, events, authentication and transport.

2. Syntactic interoperability

Do sender and receiver agree on representation? This includes JSON, XML, CSV, field types, date formats, encodings and schemas.

3. Semantic interoperability

Do they mean the same thing by the values exchanged? A field named status is useless if the sender means “account enabled” and the receiver means “customer relationship active”.

4. Organisational interoperability

Do the organisations have compatible responsibilities, policies, timing, governance and decision rights? A technically compatible interface can still fail if nobody owns correction, access approval or schema change.

True interoperability requires enough success at all four levels for the intended receiver and purpose.

Integration Is a Handoff

Every integration creates a producer–receiver relationship:

Producer → Contract → Transport → Receiver → Interpretation → Action

Each arrow can fail. A producer can emit incomplete data. A contract can be ambiguous. Transport can duplicate messages. A receiver can apply an old schema. Interpretation can use the wrong unit. Action can occur before late-arriving corrections.

Integration design should therefore model the entire handoff rather than only the wire in the middle.

The Identity Problem

Separate systems often assign separate identifiers to the same real-world entity. One student may have an enrolment ID, billing ID and learning-platform ID. One supplier may have different account numbers across business units.

Integration needs a reliable identity strategy: common identifiers, crosswalks, matching rules or a master-data service. Joining records by name alone is fragile.

See Master Data and Reference Data.

The Semantic Problem

Two systems can store the same value while meaning different things. One can store 1 for “active”; another can store 1 for “pending”. One can measure weight in kilograms; another in pounds. One can treat a cancelled order as complete for fulfilment history; another excludes it from completion metrics.

Semantic integration therefore requires:

• explicit definitions;
• units;
• code sets;
• reference-data mappings;
• effective dates;
• versioned contracts;
• rules for ambiguity and unmapped values.

A transformation should never silently claim equivalence where only approximation exists.

The Time Problem

Systems may disagree because they are looking at different moments. A student may have withdrawn in the enrolment system while the billing system still holds yesterday’s active state. A warehouse may refresh nightly while an operational application updates instantly.

Interoperability therefore needs freshness expectations and temporal semantics:

• event time;
• processing time;
• effective time;
• last refresh;
• expected latency;
• rules for late or out-of-order data.

“Both systems are correct” can be true if they represent different valid times.

Batch Integration

Batch integration moves groups of data at scheduled intervals. It is common for nightly reporting, finance processing, historical loads and large transformations.

Strengths include simplicity, controllable windows and efficient processing of large volumes. Weaknesses include latency, larger failure batches and the possibility that receivers act on stale snapshots.

Streaming and Event Integration

Streaming systems move events continuously or near continuously. They are useful where timeliness matters: payments, telemetry, fraud signals, logistics, notifications or live operational state.

Streaming introduces additional questions:

• Can events arrive twice?
• Can they arrive out of order?
• Can the receiver replay them?
• How is idempotency handled?
• How long are events retained?
• What happens if the consumer is offline?
• How are schemas evolved?

Real time is not automatically better. It should be justified by receiver need.

APIs

APIs expose data or capabilities through explicit interfaces. They can provide controlled access, validation, authentication and versioning.

A good API contract explains more than endpoint paths. It should define resource identity, field semantics, errors, permissions, pagination, rate limits, compatibility and deprecation.

An API becomes interoperable when receivers can use it without reverse-engineering hidden assumptions.

File-Based Integration

CSV, spreadsheets, flat files and secure file transfers remain common because they are simple and universal. They can be entirely appropriate.

The risks are weak schema enforcement, manual edits, unclear versions, duplicate transfers and limited error signalling. File integration therefore benefits from manifests, checksums, timestamps, stable naming, validation and acknowledgement of receipt.

ETL and ELT

ETL means extract, transform, load: data is transformed before entering its target. ELT means extract, load, transform: source data is first loaded into a target platform and transformations occur there.

The architectural choice affects lineage, performance, recovery and how much raw source state remains available. Neither pattern removes the need to document transformations.

Data Contracts

A data contract makes producer obligations and receiver expectations explicit. It can include:

• schema;
• field definitions;
• units;
• required values;
• freshness;
• quality thresholds;
• ownership;
• security classification;
• allowed changes;
• deprecation notice;
• support and escalation.

Contracts reduce the chance that a producer improves its own system while unknowingly breaking every downstream receiver.

Schema Evolution

Schemas change. New fields appear. Types change. Required fields become optional. Categories are retired. The question is not whether change occurs but whether receivers can survive it.

Good schema evolution uses compatibility rules, versioning, migration periods and explicit deprecation. Breaking changes should be treated as managed transitions, not surprises.

Canonical Models and Their Limits

A canonical data model creates a common representation for exchange. It can reduce the number of pairwise mappings between systems.

But a universal model can become too abstract or force distinct domain concepts into false equivalence. Canonical models work best when they standardise what truly needs to be shared while allowing domain-specific detail to remain local.

Crosswalks

A crosswalk maps one identifier, classification or vocabulary to another. It is often the practical bridge between systems that cannot adopt one standard immediately.

Crosswalks should support:

• one-to-one matches;
• one-to-many and many-to-one relationships;
• partial matches;
• unmapped values;
• confidence or review state;
• effective dates;
• version changes.

A crosswalk that forces every concept into an exact match can conceal semantic loss.

Reconciliation

After data moves, reconciliation checks whether expected records, totals and relationships arrived correctly.

Useful reconciliation tests include:

• record counts;
• control totals;
• hashes or checksums;
• key coverage;
• duplicate detection;
• exception counts;
• source-to-target sampling;
• business invariants.

Transport success is not evidence of semantic or quantitative completeness. Reconciliation provides a receiver-side proof.

Idempotency

An idempotent process can safely receive the same instruction more than once without creating duplicate effects. This matters because networks and distributed systems may retry after uncertain failures.

If a payment event is retried, the receiver should not charge twice. If a data load reruns, it should not duplicate every row. Stable event identifiers and controlled merge logic help.

Exactly Once Is Harder Than It Sounds

Distributed systems often cannot simply guarantee that an event will be delivered and processed exactly once in every failure scenario. Practical designs frequently use at-least-once delivery plus idempotent processing, or transactional boundaries that provide equivalent business outcomes.

The architectural lesson is to design for retries, duplicates and uncertainty rather than assuming perfect transport.

Integration and Data Quality

Integration can improve quality by consolidating identities and exposing inconsistencies. It can also amplify poor quality by distributing errors across more systems.

A strong integration therefore carries quality metadata and creates clear exception paths rather than silently coercing bad records into acceptable-looking values.

See Data Quality.

Integration and Lineage

Every integration should contribute to lineage. A receiver should be able to determine which source, version and transformation produced an important field.

Without lineage, integration creates convenient copies that become detached from authority.

See Metadata and Data Lineage.

Integration and Security

Every interface creates a new attack and exposure surface. Integration therefore needs authentication, authorisation, encryption, secret management, logging, least privilege and controlled error messages.

Security should be applied to data flows, not only storage endpoints.

Integration and Privacy

Joining datasets can create privacy risk even when each source seems harmless alone. Combined records may reveal identity, behaviour or sensitive patterns that were not visible separately.

Data minimisation, purpose limitation, access control and lifecycle rules should therefore be reviewed at the integrated view, not only at source.

Worked Example: Student Systems

An enrolment system creates the student identity and current class membership. An assessment system records marks. A billing system records invoices. A communication system sends notices.

A strong integration model:

1. uses a stable student identifier;
2. treats enrolment status as authoritative from the enrolment domain;
3. maps subject and class reference data explicitly;
4. publishes assessment events with assessment identity and version;
5. keeps billing facts owned by finance;
6. uses access rules that prevent unnecessary exposure of marks or financial data;
7. records lineage when analytics combines the domains.

The objective is not one giant database. It is coherent exchange between specialised owners.

Worked Example: Scientific Data

A research programme may integrate instrument output, sample registries, laboratory records, code repositories and publication systems.

Interoperability requires stable sample identity, standard units, documented instrument metadata and reproducible transformations. A result cannot be trusted merely because all files were successfully copied into one repository.

Worked Example: AI Retrieval

An AI retrieval system integrates source documents, access-control metadata, chunking, embeddings, search indexes and model prompts.

The retrieval layer should preserve document identity, source version, permissions and update status. Otherwise the integration may be fast while returning stale or unauthorised knowledge.

The Interoperability Paradox

More integration does not automatically create a better shared model of reality.

Connecting more sources can increase contradictions, duplicate identities, semantic conflict and false confidence. The shared system becomes better only when differences are resolved or represented honestly.

Integration increases connectivity. Interoperability increases usable understanding. They are not the same achievement.

Common Failure Modes

• Transport-only success: data arrives but meaning is wrong.
• Name-based joins: unstable labels are treated as identity.
• Unit loss: numeric values travel without measurement context.
• Silent coercion: incompatible values are forced into a target type.
• Schema surprise: producers change fields without receiver coordination.
• Stale snapshot: receivers assume old data is current.
• Duplicate event: retries create repeated business effects.
• Crosswalk certainty: approximate mappings are presented as exact.
• Unowned interface: nobody is accountable for contract failures.
• Integration sprawl: point-to-point connections multiply beyond understanding.

An Interoperability Checklist

1. Who owns the source data?
2. What is the stable identity?
3. Which schema version applies?
4. Are field definitions and units explicit?
5. Which reference-data version is used?
6. What freshness does the receiver require?
7. Can messages arrive late, twice or out of order?
8. How are exceptions represented?
9. How is source-to-target reconciliation performed?
10. How are changes communicated?
11. What security and privacy controls apply?
12. Can a downstream value be traced back to source?

A Maturity Ladder

1. Connected: systems can transfer data.
2. Structured: schemas and formats are explicit.
3. Mapped: identities and reference values are crosswalked.
4. Semantic: definitions and units are shared or translated deliberately.
5. Contracted: producers and receivers have versioned expectations.
6. Reconciled: receipt and completeness are verified.
7. Traceable: lineage spans interfaces.
8. Adaptive: change is governed without breaking receiver outcomes.

The Deeper Principle: Preserve Difference Until You Can Justify Equivalence

Integration is often motivated by a desire to make everything uniform. That can be useful, but premature uniformity destroys information.

If two source concepts are not actually the same, the integrated system should preserve the difference until a governed mapping can justify how they relate. This protects the shared model from becoming simpler than the world it is supposed to represent.

Data Management Series

• What Is Data Management?
• How Data Management Works
• The Data Lifecycle
• Data Governance
• Data Architecture
• Data Quality
• Metadata and Data Lineage
• Master Data and Reference Data
• Data Integration and Interoperability
• Data Security and Privacy
• Data Warehousing and Analytics
• Data Backup, Recovery and Resilience

---

Final idea: data integration succeeds when the receiver gets more than a copy of the sender’s values. It succeeds when enough identity, meaning, provenance, timing and authority survive the journey that the receiver can use the data without inventing missing context.