Data APIs and Data Services

A data API is a controlled machine-readable interface through which authorised consumers request, retrieve or exchange data. A data service is the broader maintained capability behind that interface: ownership, business semantics, access policy, reliability, versioning, documentation, monitoring and lifecycle.

An API is not merely a door into a database. It is a promise about what a machine can ask for, what it will receive, and how that promise changes through time.

Direct database access can be convenient, but it couples consumers to storage schemas that were designed for internal implementation rather than stable reuse. Data APIs create an explicit boundary. They allow producers to control representation, access, change and service quality while giving consumers a stable route to data.

ARTICLE ID: DATA.MANAGEMENT.034
Canonical function: governed machine-to-machine data access and service interfaces
Series route: Data Contracts and Data Products → Data APIs and Data Services.

The Simple Answer

A good data API creates a stable interface between producer and consumer:

Authorised Consumer → Request → API Contract → Data Service → Validation → Response → Monitoring → Versioned Change

The goal is not merely to move JSON or rows. The goal is to preserve meaning, access and reliability while systems evolve independently.

API vs Database Access

Database schemas expose internal tables, indexes and relationships. APIs can expose domain concepts instead.

a database may store several tables for a student;
an API may expose one coherent Student resource;
a database may rename columns during migration;
the API can preserve the external contract while implementation changes behind it.

This separation reduces coupling and makes responsibility clearer.

Resource-Oriented Interfaces

Many APIs organise access around resources such as students, products, orders, datasets or metrics. A resource should have stable identity and predictable representation.

Resources should follow domain meaning rather than mirror every internal table mechanically.

Query Interfaces

Some data services expose flexible query capabilities rather than fixed resource endpoints. This can reduce the number of bespoke interfaces but increases the need for query governance, cost limits and security.

The more expressive the query language, the more responsibility the service has to protect performance and data boundaries.

Contracts

An API contract should define what consumers can rely on:

endpoint or operation identity;
request fields;
response fields;
types;
required vs optional values;
error behaviour;
pagination;
authentication;
authorisation;
rate limits;
versioning;
deprecation policy;
service expectations.

Schema is only part of the contract. Field semantics, units and lifecycle matter too.

Stable Identity

Resource identifiers should be stable enough that consumers can reference the same entity across calls and over time.

Internal surrogate keys may be unsuitable if they can change during migration. External identifiers should be chosen deliberately.

See Master Data and Reference Data.

Representation Is an API Decision

The API decides how domain state is represented: field names, nesting, code values, timestamps and links.

Changing representation can be breaking even if the underlying database remains unchanged.

Timestamps

Time fields should state what they mean. Created time, effective time, event time, last-modified time and ingestion time are not interchangeable.

Machine-to-machine interfaces should use consistent, unambiguous time representation and document timezone assumptions.

Pagination

Large collections should usually be paginated so one request does not attempt to return the entire dataset.

Pagination design affects correctness. If records are inserted while a consumer pages through results, simple offset-based pagination can skip or duplicate items. Cursor or key-based approaches can provide more stable traversal for changing datasets.

Filtering

Filters should be explicit and bounded. Consumers need to know which fields are filterable and whether comparisons are exact, case-sensitive, range-based or semantic.

Unbounded query flexibility can create performance and information-disclosure risk.

Sorting

Sorting should define deterministic tie-breaking where pagination depends on stable order. A collection ordered only by a non-unique timestamp can change unpredictably between requests.

Partial Responses

Some APIs allow consumers to request only needed fields. This can reduce payload size and exposure, but field-level access rules should still apply.

Authentication

Authentication establishes who or what is calling the API. Machine identities should be managed explicitly rather than sharing one long-lived credential across many applications.

Credentials should be rotated, protected and scoped appropriately.

Authorisation

Authorisation determines what the authenticated caller may access or do.

which resources;
which fields;
which operations;
which populations;
which time periods;
whether bulk export is allowed.

Authentication without authorisation only proves identity; it does not establish permission.

Least Privilege

Service credentials should receive the narrowest access required. One reporting application should not need the same permissions as an administrative platform.

See Data Security and Privacy.

Rate Limits

Rate limits protect service capacity and can reduce abuse. They should be designed around legitimate workloads and return useful information when limits are reached.

Consumers need retry guidance so throttling does not cause uncontrolled request storms.

Quotas

Quotas bound longer-term usage by consumer, account or application. They are useful when one consumer could otherwise impose disproportionate cost on shared infrastructure.

Caching

Caching can reduce repeated computation and latency, but cached data creates freshness questions.

Responses should make cache behaviour and staleness expectations clear when those properties matter to the receiver.

Idempotency

For write operations, idempotency helps protect against duplicate outcomes when clients retry after uncertain network failures.

Stable request identifiers or resource semantics can allow the service to recognise repeated logical operations.

Errors Are Part of the Contract

A good API returns errors that are structured enough for machines to handle predictably.

authentication failure;
authorisation failure;
invalid request;
resource not found;
conflict;
rate limit;
temporary service failure;
validation failure.

Error messages should be informative without leaking sensitive implementation detail.

Versioning

APIs should evolve without breaking every consumer unexpectedly.

Versioning can occur through explicit version identifiers, compatible additive change, feature negotiation or other controlled mechanisms. The exact technique matters less than the contract: consumers must know which changes are safe and which require migration.

See Data Versioning and Change Management.

Breaking Changes

Breaking changes can include:

removing fields;
renaming fields;
changing types;
changing identifiers;
changing units;
changing error behaviour;
changing pagination semantics;
changing business meaning while leaving schema unchanged.

Semantic changes are particularly dangerous because automated compatibility checks may not detect them.

Deprecation

Deprecated versions should have a clear replacement, migration guide and retirement date. Keeping every historical API forever creates support and security debt.

Consumer Inventory

Before retiring an API, producers should know who consumes it. API gateways, service registration, telemetry and explicit contracts can help build a consumer inventory.

Unknown consumers make safe change difficult.

Documentation

Machine interfaces still need human-readable documentation. Useful documentation includes:

purpose;
domain concepts;
authentication;
request examples;
response examples;
field definitions;
error handling;
pagination;
limits;
versioning;
support route;
known limitations.

Machine-Readable Specifications

Formal interface specifications can allow tools to generate clients, validate requests and test compatibility. They strengthen technical consistency but do not replace business definitions.

Observability

Useful service signals include:

request volume;
latency;
error rate;
availability;
rate-limit events;
authentication failures;
consumer-specific failure patterns;
data freshness;
contract violations.

Technical health should be connected to data health. A fast API serving stale data is not healthy for the receiver.

See Data Observability and Monitoring.

Service-Level Objectives

Critical APIs can define service-level objectives for availability, latency, freshness and incident response according to receiver consequence.

Not every API needs extremely high availability. Reliability should be proportional to the importance of the consuming workflow.

Bulk Export vs API Access

APIs are not always the best interface for large historical datasets. Bulk files may provide simpler and more reproducible access for analytical consumers.

A mature data service can expose several interfaces for different jobs:

low-latency API;
bulk snapshot;
event stream;
analytical table;
controlled query service.

The product should define which interface is authoritative for which use.

APIs and Event Streams

APIs are often request-response interfaces. Event streams provide push-style or subscription-based change delivery.

Consumers may use an API to retrieve current state and an event stream to keep that state updated.

See Data Streaming and Event-Driven Systems.

API Gateways

Gateways can centralise authentication, routing, rate limiting, logging and policy enforcement across APIs.

A gateway is infrastructure, not governance by itself. It cannot decide whether a consumer should receive a sensitive field unless the relevant policy and ownership have been defined.

Data Classification and APIs

API design should reflect data sensitivity. Highly restricted data may need narrower endpoints, stronger authentication, field-level control, audit logging and lower export limits.

See Data Classification and Sensitivity.

Data Sovereignty and APIs

API calls can move data across regions even when the database remains local. Service endpoints, caching, logging and support infrastructure should be included in sovereignty review.

See Data Sovereignty, Residency and Jurisdiction.

Data Mesh and APIs

In federated architectures, APIs can become stable product interfaces between domains. A domain can evolve its internal storage while preserving the contract consumed by other domains.

See Data Mesh and Federated Data Ownership.

Testing Data APIs

Testing should cover both interface behaviour and data correctness:

contract compatibility;
authentication and authorisation;
pagination stability;
edge cases;
rate limiting;
idempotency;
error responses;
data freshness;
known reference cases;
deprecation behaviour.

See Data Testing and Reliability Engineering.

AI and Tool Use

AI agents increasingly use APIs as tools. Machine-readable specifications and stable semantic contracts help models invoke data services safely.

For AI consumers, services should expose enough metadata to answer:

what the operation does;
which fields are required;
what the response means;
which permissions apply;
which errors are recoverable;
how current the data is.

An AI should not have to infer critical business semantics from field names alone.

Education Example

An education organisation can expose a Student Profile API that returns authorised enrolment and class information using a stable student identifier. Teaching systems receive academic fields; finance systems receive billing-relevant identifiers through separate authorised views.

The API keeps each consumer from reading the entire student database directly.

Commerce Example

A Product API can provide canonical product identity, lifecycle state and public attributes to e-commerce, inventory and partner systems. Internal cost fields remain unavailable to external consumers even though they live in related source tables.

Research Example

A research repository API can expose metadata openly while requiring authenticated approval for restricted files. One service can therefore support findability and controlled access without making every research object public.

Common Failure Modes

Database over HTTP: internal tables are exposed directly without a stable domain contract.
Authentication equals permission: any logged-in caller can access too much.
Unlimited collection endpoints: one request can exhaust service capacity.
Pagination drift: changing datasets cause skipped or duplicated records.
Error chaos: clients cannot distinguish invalid requests from temporary failure.
Version forever: old endpoints accumulate without retirement.
Schema compatibility only: semantic changes break consumers invisibly.
Fast but stale: latency metrics hide outdated data.
Unknown consumers: producers cannot deprecate safely.
API as universal interface: massive historical exports are forced through inefficient request-response patterns.

A Data API Checklist

What data product does the API represent?
Who owns the service?
Who are the intended consumers?
Are resource identities stable?
Are request and response semantics explicit?
How are authentication and authorisation separated?
How is sensitive data minimised?
How are large collections paginated?
What filters and queries are allowed?
What rate limits or quotas apply?
How are errors structured?
How is the API versioned?
What deprecation route exists?
Can consumers be identified before retirement?
Are latency, availability and freshness monitored together?

A Maturity Ladder

Exposed: systems can retrieve data through an interface.
Contracted: request, response and error behaviour are documented.
Secured: identity, authorisation and least privilege are enforced.
Versioned: change and deprecation are controlled.
Observable: latency, errors, freshness and consumer impact are visible.
Productised: ownership, support and service expectations are explicit.
Federated: APIs act as stable domain boundaries across the estate.
Machine-actionable: tools and AI agents can invoke services safely from governed specifications.

The Deeper Principle: Interfaces Preserve Independence

A good API allows producer and consumer systems to change without forcing each side to know the other’s internal implementation. That independence is not free. It depends on stable identity, explicit semantics, predictable errors, security and disciplined versioning.

Data services become durable infrastructure when the interface is treated as a product promise rather than a convenient technical shortcut.

Data Management Series

Final idea: data APIs turn internal data into governed machine-to-machine capability. Their real job is not transport. It is to preserve a dependable boundary of meaning, permission and change between systems that need to cooperate without becoming tightly coupled.

Data APIs and Data Services | Interfaces, Contracts, Versioning, Access and Reliable Machine-to-Machine Data