eduKate Singapore

Data Synchronisation and Reconciliation | Replication, Change Capture, Drift, Conflict Resolution and Convergence

Data synchronisation is the controlled process of keeping two or more representations of data sufficiently aligned for their intended jobs. Data reconciliation is the evidence process used to compare those representations, explain differences and determine whether they have converged to an acceptable state.

Synchronisation moves state. Reconciliation proves whether the resulting states still represent the same intended reality.

Modern organisations rarely have one copy of important data. A customer record may exist in an operational application, a warehouse, a lakehouse, a reporting system, a search index, a cache and an AI retrieval service. Some copies are authoritative. Some are derived. Some are temporary. The engineering problem is not to make every copy identical at every instant. It is to define which differences are acceptable, how long they may persist, and how the system detects when ordinary delay becomes actual divergence.

ARTICLE ID: DATA.MANAGEMENT.037
Canonical function: multi-system state alignment, divergence detection and proof of convergence
Owner boundary: this article owns synchronisation and reconciliation. Data Integration and Interoperability owns ongoing exchange between unlike systems; Data Migration and Legacy Modernisation owns bounded source-to-target transition; Data Streaming and Event-Driven Systems owns continuous event processing.

The Simple Answer

A synchronised system usually follows this loop:

Authoritative Change → Capture → Transfer → Apply → Observe → Compare → Explain Difference → Repair → Verify Convergence

Moving data is only the middle of the loop. The beginning requires authority: which state is allowed to lead? The end requires proof: did the receiver reach the intended state?

Synchronisation Is Not Duplication

Copying a file creates another representation. Synchronisation adds an obligation to manage change through time.

A copied dataset may be correct at 09:00 and stale at 09:05. Whether that is acceptable depends on the receiver. A daily planning report may tolerate hours of lag. An account-balance service may not.

Define the Authority Direction

The first design question is not technical. It is: which system is authoritative for which fact?

Without an authority model, conflict resolution becomes political guesswork.

One-Way Replication

One-way replication is the simplest model. Changes flow from an authoritative source into one or more receivers. Receivers may query or transform the data but do not write authoritative changes back.

This pattern is common for analytics, search indexes, caches and reporting copies because it preserves a clear direction of truth.

Bidirectional Synchronisation

Bidirectional synchronisation is harder because two locations can change the same logical entity before either has seen the other’s change.

A design must define:

Change Data Capture

Change Data Capture records inserts, updates and deletes from an operational source so downstream systems can apply incremental changes rather than repeatedly copying the entire dataset.

CDC can reduce latency and transfer cost, but it also carries source-system semantics. A row update says what changed physically; it may not explain why the business state changed.

Full Refresh vs Incremental Synchronisation

A full refresh rebuilds the receiver from a complete source snapshot. Incremental synchronisation applies only changes since a known point.

Mature systems often use incremental updates during normal operation and periodic full or independent reconciliation to detect accumulated drift.

Checkpoints

A checkpoint records how far a receiver has successfully applied the source change sequence. It may use a log position, event offset, timestamp, version number or another monotonic marker.

Checkpoint advancement should occur only after the corresponding receiver state is durably accepted. Advancing too early can create invisible gaps after failure.

Idempotency

Retries are normal in distributed systems. An idempotent apply operation ensures that repeating the same logical change does not create repeated effects.

Useful tools include stable event IDs, version checks, upserts and deduplication ledgers.

Ordering

Some changes must be applied in order. A record cannot be deleted before its creation event has been observed. A balance correction may depend on an earlier transaction.

Ordering should be preserved at the smallest domain boundary that actually requires it. Global total ordering is expensive and often unnecessary.

Eventual Consistency

Eventually consistent systems allow replicas to differ temporarily but are designed to converge if no new changes occur.

Eventual consistency is not permission for permanent unexplained mismatch. A useful design states the expected convergence window and what monitoring identifies replicas that remain outside it.

Strong Consistency

Some operations require a stronger guarantee that consumers observe a coordinated state before proceeding. Stronger consistency can simplify reasoning but usually increases latency, coupling or availability trade-offs across distributed systems.

The correct consistency model follows the consequence of stale or conflicting state.

Drift

Data drift between replicas is an unintended difference that remains after normal synchronisation should have converged.

Causes include:

Reconciliation Is Independent Evidence

Reconciliation should not merely ask the synchronisation process whether it succeeded. It should compare source and receiver state independently enough to detect failures in the transfer logic itself.

Useful techniques include:

Counts Are Not Enough

Two systems can contain the same number of records while holding different records. Two financial totals can match because errors cancel each other.

Strong reconciliation combines several independent signals.

Hash Reconciliation

Deterministic hashes can provide compact evidence that normalised representations match. Large datasets can be compared by partition or key range so mismatches are localised.

A matching hash supports equality of the hashed representation; it does not prove that either representation is semantically correct.

Normalisation Before Comparison

Two systems may represent the same meaning differently: timezone formats, decimal precision, code aliases or field ordering.

Reconciliation should compare canonicalised representations where appropriate while preserving the original values needed to explain differences.

Conflict Detection

A conflict occurs when two legitimate changes cannot both be applied without a decision.

Examples:

Last-Write-Wins

Last-write-wins chooses the change with the latest accepted timestamp or version. It is simple, but it is safe only when later really means more authoritative.

Clock skew, delayed delivery or different business authority can make a later write less legitimate than an earlier one.

Field-Level Merge

Some conflicts can be merged when independent fields changed. A phone number updated in one system and a postal address updated in another may coexist if both systems are authorised for those fields.

The merge rule should come from domain ownership, not convenience.

Human Conflict Resolution

High-consequence or semantically ambiguous conflicts may require human review. The reviewer should see both versions, provenance, timestamps, authority and downstream impact.

The resolution itself becomes a governed event that must propagate to every replica.

Convergence

Convergence means replicas have reached the intended compatible state under the defined authority and consistency rules.

A release or recovery process should record evidence of convergence rather than assuming it from successful message delivery.

Repair Strategies

The safest repair depends on whether the receiver is authoritative, derived or temporarily co-authoritative.

Read Repair

Some distributed systems repair stale replicas when a read discovers disagreement. Read repair can improve convergence but should not hide persistent systemic drift from monitoring.

Anti-Entropy Processes

Periodic anti-entropy processes compare replicas and exchange missing or divergent state even if ordinary change propagation appeared successful.

This embodies an important reliability principle: do not let the mechanism that writes state be the only mechanism that proves state.

Reconciliation Windows

Large estates may reconcile continuously for high-risk data and periodically for lower-risk products.

The interval should reflect:

Reconciliation Status Must Be Visible

Consumers should be able to distinguish:

A complete-looking dashboard should not conceal an unresolved reconciliation state.

Observability

Useful signals include:

See Data Observability and Monitoring.

Testing Synchronisation

Tests should include failure conditions rather than only normal flow:

See Data Testing and Reliability Engineering.

Synchronisation and Migration

Migrations often use temporary synchronisation while old and new systems coexist. An initial bulk load is followed by incremental change capture until the target catches up to a known consistency point.

Cutover should require reconciliation evidence, not merely a statement that the replication queue is empty.

Synchronisation and Caches

Caches are deliberately derived copies. They may remain stale for bounded periods. Cache invalidation is therefore a specialised synchronisation problem with an explicit freshness contract.

See Data Caching and Materialisation.

Synchronisation and Search Indexes

A search index may receive updates asynchronously from an authoritative content store. Reconciliation should detect missing documents, stale versions and documents that should have been deleted.

Synchronisation and AI Retrieval

AI retrieval systems create derived chunks, embeddings and indexes. When a source document changes or is revoked, the derived representations must converge to the new authoritative state.

A deleted source that remains retrievable through an old vector index is a reconciliation failure, not merely an AI issue.

Security and Privacy

Every replica expands the security and privacy surface. Synchronisation design should therefore control which fields are copied, where replicas live, who can access them and how deletion propagates.

Replicas should inherit appropriate classification and retention requirements.

Deletion Propagation

Deletion is often harder to synchronise than creation because downstream systems may have transformed or aggregated the original record.

A mature deletion route maps which derived products can remove the record directly, which require recomputation and which legitimately retain independent records under their own authority.

Education Example

A student-information system owns enrolment status. The learning platform receives changes asynchronously. If a student transfers class, the update propagates to the platform, analytics and attendance services.

A nightly reconciliation compares active student–class memberships across systems. Any mismatch remains visible until repaired. This prevents an old class assignment from surviving simply because the original event was missed.

Finance Example

A payment service publishes settled transactions into finance reporting. The reporting warehouse applies each event idempotently and reconciles daily counts, values and key coverage against the authoritative settlement system.

A successful load is not the receipt. Reconciliation is the receipt.

Common Failure Modes

A Synchronisation Checklist

  1. Which system owns each fact?
  2. Is synchronisation one-way, bidirectional or domain-split?
  3. How are changes captured?
  4. What checkpoint proves progress?
  5. Are updates idempotent?
  6. Which ordering guarantees matter?
  7. What lag is acceptable?
  8. How are conflicts detected?
  9. Which conflicts can resolve automatically?
  10. What independent reconciliation runs?
  11. How are mismatches quarantined and repaired?
  12. How is convergence proven?
  13. How are deletions propagated?
  14. Which replicas inherit sensitive-data controls?
  15. Can every unresolved mismatch remain visible until resolution?

A Maturity Ladder

  1. Copied: downstream representations are refreshed manually.
  2. Replicated: changes move automatically.
  3. Checkpointed: progress is durable and restartable.
  4. Idempotent: retries do not duplicate logical effects.
  5. Reconciled: independent evidence detects drift.
  6. Conflict-aware: concurrent changes follow explicit authority rules.
  7. Convergence-proven: repair ends with verified compatible state.
  8. Adaptive: mismatches produce stronger tests, contracts and synchronisation design.

The Deeper Principle: Distributed Truth Requires a Return Path

Distributed systems inevitably create multiple representations. The danger is not multiplicity itself. The danger is losing the route that tells the organisation which representation is authoritative, how differences arose and whether repair actually restored the intended state.

Synchronisation moves the change forward. Reconciliation provides the return path from receiver to source authority. Together they turn replication from hopeful copying into controlled distributed memory.

Data Management Series

Final idea: trustworthy synchronisation is not measured by how quickly copies are produced. It is measured by whether authority is clear, divergence is observable, conflicts are resolvable and the system can prove that distributed representations eventually return to a coherent state.

eduKate Asia

Subscribe to
our newsletter

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading