eduKate Singapore

Data Architecture | How Information Systems Are Structured, Connected and Kept Coherent

Data Architecture

Data architecture is the deliberate structure that determines where data lives, how it is represented, how it moves, which systems own which facts, how different domains connect, and how the whole information estate remains understandable as it grows.

Architecture is the map that prevents a data estate from becoming a pile of useful systems that no longer know how they fit together.

Every organisation has a data architecture whether it designed one or not. If a business has spreadsheets, databases, SaaS tools, shared drives, reports, APIs, dashboards, exports and cloud storage, those components already form an architecture. The only question is whether the architecture is explicit, governed and resilient—or accidental, undocumented and fragile.

ARTICLE ID: DATA.MANAGEMENT.005
Canonical function: structural design of the data estate
Series route: What Is Data Management? → How Data Management Works → The Data Lifecycle → Data Governance → Data Architecture.

The Simple Answer

If data management is the discipline of turning recorded reality into reliable organisational memory, data architecture is the plan for where that memory is placed and how the pieces are connected.

A good architecture can answer five basic questions:

If nobody can answer those questions, the organisation has systems but not yet a coherent architecture.

Architecture Begins with Boundaries

The first architectural problem is not choosing a database. It is deciding where one responsibility ends and another begins.

Examples of boundaries include:

Each domain may have its own systems, vocabulary, update rate, sensitivity and lifecycle. Architecture should connect these domains without pretending they are all the same thing.

The Canonical Source Principle

One of the most important architectural rules is to make authority explicit. A fact may be copied into many systems, but there should be a known source of authority for the version that matters.

For example:

Architecture becomes fragile when every application stores its own unmanaged version of the same fact. Copies are not the problem. Unknown authority is.

The Main Layers of Data Architecture

1. Source layer

The source layer contains systems where data is first created or authoritatively maintained. These can include transactional applications, sensors, forms, enterprise systems, scientific instruments, web services and external providers.

2. Integration layer

The integration layer moves and transforms data through APIs, messaging, event streams, file transfers, pipelines or replication processes. It is the wiring between systems.

3. Storage layer

The storage layer includes operational databases, analytical stores, object storage, warehouses, lakes, search indexes, document repositories and archives.

4. Semantic layer

The semantic layer gives shared meaning to data through models, definitions, controlled vocabularies, metrics, dimensions and mappings.

5. Access layer

The access layer exposes data to applications, analysts, reports, dashboards, models, APIs and authorised users.

6. Governance and control layer

This layer includes ownership, security, privacy, quality, metadata, lineage, lifecycle and audit controls.

These layers may be implemented by many tools or only a few. The conceptual separation matters because different questions belong to different layers.

Operational Databases

Operational databases support current activity. They record orders, payments, enrolments, bookings, inventory movements, submissions and other live processes.

Their priorities often include:

Operational systems are usually not ideal for every analytical workload. Architecture separates the workload that runs the business from the workload that studies the business.

Data Warehouses

A data warehouse organises integrated historical data for reporting and analysis. It often reshapes operational data into structures designed for questions rather than transactions.

Warehouses are valuable when the organisation needs stable metrics, historical comparison and common definitions across many source systems.

The architectural danger is to assume that moving data into a warehouse automatically resolves semantic conflict. It does not. If two source systems disagree about what “customer” means, the warehouse still needs an explicit model or mapping.

Data Lakes

A data lake stores large volumes of data in flexible forms, often close to source structure. It can hold structured, semi-structured and unstructured data and may support analytics, research, machine learning and archival use.

The strength of a lake is flexibility. The weakness is that flexibility can become disorder if metadata, ownership, quality and lifecycle are neglected.

A lake without governance can become a storage system that remembers everything except what anything means.

Lakehouse Architecture

Lakehouse designs aim to combine the flexible storage characteristics of data lakes with stronger table management, reliability and analytical performance traditionally associated with warehouses.

The important architectural lesson is not the label. It is that organisations repeatedly seek a balance between flexibility and control.

Event-Driven Architecture

In event-driven systems, changes are represented as events that other systems can consume. Examples include “order placed”, “payment received”, “student submitted assignment” or “sensor threshold exceeded”.

Events can reduce tight coupling because producers do not need to know every consumer. But event architecture requires strong contracts: event names, schemas, identifiers, timestamps, ordering expectations and versioning rules must be explicit.

Events are powerful because they preserve change through time. They are dangerous when receivers interpret the same event differently.

Batch vs Streaming

Batch systems move data at intervals. Streaming systems process events continuously or near continuously. Neither is automatically superior.

The architecture should match the freshness requirement. Daily finance reconciliation may be sufficient for one use. Real-time fraud detection or machine monitoring may require seconds.

Real-time architecture has a cost: more operational complexity, more state management and more difficult failure handling. Freshness should be justified by consequence rather than fashion.

APIs as Architectural Contracts

An API is not merely a technical interface. It is a promise about how one system exposes capabilities and data to another.

A robust data API should make explicit:

APIs support architectural independence when contracts are stable. They create architectural fragility when hidden dependencies proliferate.

Data Models: Structure Before Storage

A data model describes entities, attributes, relationships and constraints. It can be conceptual, logical or physical.

Separating these levels helps prevent technology choices from dictating the meaning of the domain.

Normalisation and Denormalisation

Normalisation reduces duplication by separating facts into related structures. Denormalisation intentionally duplicates or combines data to improve performance or simplify access.

Neither is morally superior. Architecture is about trade-offs. The key is to know which copy is authoritative and how duplicated values remain synchronised or traceable.

Master Data in Architecture

Master data represents important shared entities such as customers, products, suppliers, employees, students, locations or assets. Architecture must decide which systems can create, update and distribute those identities.

Without explicit master-data architecture, duplicate identities and conflicting attributes spread through the estate.

The dedicated companion article Master Data and Reference Data develops this in depth.

Metadata Architecture

Metadata architecture determines how the organisation describes its datasets, fields, ownership, quality, lineage, sensitivity and lifecycle.

A mature metadata layer acts like a map legend for the data estate. Without it, even a technically elegant architecture becomes difficult to navigate.

See Metadata and Data Lineage.

Data Quality Architecture

Quality should be designed into the flow rather than added after delivery. Architecture should identify where quality is checked, where exceptions are stored, who owns repair and how corrections propagate to downstream systems.

See Data Quality for the full framework.

Security Architecture

Security architecture controls who can reach data, from where, under what identity and with what privileges. It includes network boundaries, authentication, authorisation, encryption, secrets, audit trails and recovery controls.

Good security architecture is layered. If one control fails, another should reduce consequence.

Privacy Architecture

Privacy architecture shapes where personal data is collected, which fields are necessary, where identifiers can be separated, how access is limited, how sharing occurs and how data can be deleted when retention is no longer justified.

Privacy improves when the architecture reduces unnecessary copies. Every duplicate containing sensitive information creates another place that must be protected and eventually governed through its lifecycle.

The Problem of Hidden Copies

Exports are among the most common sources of architectural drift. A user downloads a CSV, emails it, edits it, stores it locally and builds a report. The data has left the governed route.

Copies are sometimes necessary. But architecture should make controlled routes easier than shadow routes. If the official system is too slow, too hard to access or too inflexible, unmanaged copies will multiply.

Shadow Data

Shadow data is information created, copied or stored outside normal governance and visibility. It may live in personal drives, notebooks, local databases, temporary cloud buckets or unsanctioned SaaS platforms.

Shadow data often appears because official architecture does not meet real user needs. The solution is not merely punishment. Architecture should understand the workflow pressure that created the shadow route.

Architectural Coupling

Systems are tightly coupled when a change in one easily breaks another. Loose coupling reduces direct dependence through stable contracts, events, APIs or intermediary layers.

Loose coupling does not mean no relationship. It means the relationship is explicit enough to change safely.

The Cost of Point-to-Point Integration

As organisations grow, direct integrations between every pair of systems can create a web of dependencies. Each new connection adds another place where schema, credentials, timing and error behaviour can break.

Architecture may therefore introduce shared integration patterns, event buses, API gateways or canonical exchange models. These reduce duplication but can also become central bottlenecks if poorly designed.

Centralised vs Federated Architecture

A centralised architecture places more data and control in shared platforms. A federated architecture allows domains greater autonomy while preserving cross-domain standards and contracts.

Centralisation can simplify governance and tooling. Federation can preserve domain expertise and speed. The strongest design often separates shared infrastructure from domain ownership.

Data Mesh as a Governance Idea

Data-mesh thinking emphasises domain-oriented ownership, data as a product, self-service infrastructure and federated governance. The useful architectural idea is that the team closest to a domain often understands its meaning best, while the organisation still needs common standards for interoperability and trust.

The label matters less than the design question: how can local ownership coexist with enterprise coherence?

Data Products

A data product treats a dataset or service as something with explicit users, owners, contracts, quality expectations, documentation and lifecycle.

This is useful because it shifts thinking from “we published a table” to “we provide a maintained capability to receivers”.

Architecture and AI

AI systems add new architectural components: training datasets, feature stores, vector indexes, model registries, prompt logs, evaluation datasets, retrieval pipelines and feedback loops.

The architectural risk is to create a parallel AI estate disconnected from normal data governance. AI components should inherit the same principles of ownership, provenance, lifecycle, quality and access control as the rest of the organisation.

Architecture and Knowledge Retrieval

Retrieval systems often combine documents, metadata, embeddings and search indexes. The original source should remain distinguishable from the index representation. Embeddings are useful representations for similarity, but they do not replace provenance or canonical source content.

A mature retrieval architecture can answer: which source produced this chunk, which version was indexed, when it was refreshed and whether access rights still permit retrieval.

Architecture and Time

Architecture must support both current truth and historical truth. A source may overwrite current state, while an event store or warehouse preserves earlier states. A report may need to reproduce what was known at month-end rather than today’s corrected values.

This is why versioning, effective dates and lineage belong inside architecture rather than being added later.

Architecture and Resilience

Resilient data architecture assumes components will fail. It designs for recovery, graceful degradation, replication, retry, idempotency, backup, replay and controlled failover.

Resilience also means semantic recovery. Restoring a database from backup is not enough if the organisation cannot determine which version of a reference table was valid when the incident occurred.

The Architecture Drift Problem

Architectures drift when new systems are added faster than the map is updated. Teams create exceptions. Temporary integrations become permanent. Old systems remain because one forgotten report still depends on them.

Architecture management therefore requires continuous discovery and dependency mapping, not only diagrams produced during project launch.

A Data Architecture Map Should Show More Than Boxes

A useful architecture map should show:

A diagram without these meanings can look impressive while remaining operationally weak.

A Worked Example: Education Data Architecture

Consider a school or tuition organisation. It may have separate systems for enrolment, attendance, assessment, billing, learning content and communication.

A coherent architecture might define:

The architecture should make it difficult for an old spreadsheet to silently become the new source of truth.

A Worked Example: Scientific Data Architecture

A research environment may connect instruments, laboratory systems, notebooks, code repositories, raw data stores, processed datasets and publication repositories.

Strong architecture preserves the route:

Instrument → Raw Observation → Metadata → Processing Code → Derived Dataset → Analysis → Figure → Publication → Repository

If that route can be reconstructed, reproducibility becomes much stronger.

Architecture Anti-Patterns

A Practical Architecture Review

  1. List the major data domains.
  2. Name the authoritative system for each critical fact.
  3. Map where copies are made.
  4. Map interfaces and transfer mechanisms.
  5. Identify shared identifiers and reference data.
  6. Identify semantic mismatches across domains.
  7. Trace one important dashboard metric back to source.
  8. Trace one sensitive field across every copy.
  9. Identify lifecycle rules for each major store.
  10. Identify single points of failure.
  11. Identify systems with unknown dependencies.
  12. Identify exports and shadow routes.
  13. Review whether freshness requirements justify architectural complexity.
  14. Review whether current access boundaries match ownership and sensitivity.
  15. Update the target-state map and transition plan.

The Target-State Problem

Architectures are often drawn as perfect future states that ignore the messy systems that already exist. A useful target state must include a migration path.

Good transition architecture distinguishes:

The hard part is rarely drawing the destination. It is crossing from old to new without losing evidence, service or authority.

Architecture as Organisational Memory Design

At the deepest level, data architecture is memory design.

It decides which parts of organisational memory are local, which are shared, which are temporary, which are authoritative, which are analytical and which must survive beyond current systems.

A good architecture allows information to travel without losing its route back to meaning. A poor architecture allows data to multiply faster than understanding.

Data Management Series

Final idea: the purpose of data architecture is not to centralise everything or modernise everything. It is to make the route between source, meaning, storage, authority and receiver explicit enough that the organisation can change without forgetting how its information system works.

eduKate Asia

Subscribe to
our newsletter

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading