Data Deduplication and Entity Resolution | Matching, Record Linkage, Survivorship, False Merges and Golden Records

Data deduplication is the process of identifying records or objects that represent the same underlying information closely enough to be treated as duplicates. Entity resolution is the broader discipline of deciding whether multiple records, names, identifiers or observations refer to the same real-world entity. The central risk is symmetrical: false merges combine different entities incorrectly, while false splits leave one entity fragmented across several identities.

Identity resolution is not the art of making records look tidy. It is the controlled decision about when two representations are allowed to point to one thing.

Organisations accumulate duplicates naturally. Customers sign up twice, names are spelled differently, suppliers change identifiers, records arrive from acquisitions, systems use different keys, documents are copied, and historical data contains old addresses or aliases. Some duplication is redundant. Some represents legitimate history. Some indicates two different people who merely resemble each other. Good resolution preserves those distinctions.

ARTICLE ID: DATA.MANAGEMENT.050
Canonical function: duplicate detection, entity identity resolution and controlled consolidation
Owner boundary: this article owns matching and resolution. Master Data and Reference Data owns canonical shared identities after governance; Knowledge Graphs and Semantic Data owns graph relationships; Data Quality owns overall fitness and defect measurement.

The Simple Answer

A trustworthy resolution route is:

Standardise → Generate Candidates → Compare Evidence → Score or Decide → Review Ambiguity → Link or Merge → Apply Survivorship → Preserve Provenance → Monitor False Merges and False Splits

The purpose is not to collapse aggressively. It is to make identity decisions with enough evidence that future receivers can understand and, where necessary, reverse them.

Duplicate Records vs Duplicate Entities

Two byte-identical files may be exact duplicates. Two customer rows can differ substantially yet still represent the same person.

Deduplication therefore ranges from simple representation equality to difficult identity inference.

Exact Duplicates

Exact duplicates match under a defined representation: same checksum, same canonical key, or identical normalised record.

Exact matching is high precision but can miss duplicates whose spelling, formatting or identifiers differ.

Near Duplicates

Near duplicates differ superficially while carrying substantially the same content or identity.

“Tan Wei Ming” vs “Wei Ming Tan”;
“Acme Pte Ltd” vs “ACME Private Limited”;
old and new postal addresses;
PDF and DOCX manifestations of one document;
two product descriptions with minor wording changes.

Similarity is evidence for review, not proof of identity.

Stable Identifiers Beat Guessing

When a trustworthy unique identifier exists and is governed correctly, it should usually dominate fuzzy matching.

Problems arise when identifiers are missing, reused, mistyped, local to one system or changed during migration.

Natural Keys and Surrogate Keys

A natural key comes from the domain, such as a government-issued identifier or external publication identifier. A surrogate key is created by the system.

Surrogate keys simplify databases but do not prove that two source records represent different real entities. Two systems can assign two surrogate IDs to one person.

Normalisation Before Matching

Records often need normalisation before comparison:

trim whitespace;
standardise case;
normalise punctuation;
standardise phone formats;
separate titles from names;
normalise dates;
standardise address components;
map known abbreviations.

Normalisation should preserve original source values so future reviewers can see what was actually received.

Over-Normalisation

Normalisation can erase distinctions. Removing all punctuation, accents or suffixes may make genuinely different names look identical.

Every normalisation rule should be appropriate to the domain and language.

Candidate Generation

Comparing every record with every other record becomes computationally expensive as datasets grow. Candidate generation reduces the search space by identifying plausible pairs.

Common strategies include:

blocking by postal code;
same surname prefix;
same email domain;
same birth year;
phonetic keys;
approximate nearest-neighbour retrieval;
shared external identifiers.

Blocking Has Recall Risk

If candidate generation never places two true duplicates together, later matching cannot recover the missed pair.

Blocking therefore trades computational efficiency against recall and should be tested on difficult cases.

Deterministic Matching

Deterministic rules decide matches from explicit conditions.

same national identifier;
same verified email and date of birth;
same supplier registration number;
same document checksum.

Rules are interpretable and easy to audit, but can become brittle when source quality varies.

Probabilistic Record Linkage

Probabilistic methods combine several pieces of evidence and estimate how strongly a pair resembles a true match versus a non-match.

Name agreement, address similarity, date of birth and phone number can each contribute differently depending on how common or reliable the field is.

Common Values Carry Less Evidence

Matching on a rare surname can be more informative than matching on a very common surname. Likewise, a shared household address may provide weaker evidence where many unrelated people live in one building.

Evidence strength depends on frequency and context.

Fuzzy String Matching

Fuzzy string methods measure similarity between names, addresses or text values.

They can handle typographical variation but should not be used as identity proof alone. Two different people can have nearly identical names.

Phonetic Matching

Phonetic algorithms map names to sound-like representations. They can help with spelling variants but may perform unevenly across languages and naming conventions.

Multilingual identity requires domain-aware matching rather than assuming one English-centric phonetic rule works globally.

Address Matching

Addresses contain hierarchy: unit, building, street, postal code, city, country. Standardisation and geocoding can improve matching.

Shared addresses do not imply shared identity. Families, tenants and businesses can occupy the same location.

Email and Phone Matching

Email addresses and phone numbers are often useful but can be shared, recycled, mistyped or changed.

Verified contact channels carry stronger identity evidence than unverified self-entered fields.

Temporal Context

Identity attributes change over time. Addresses, names, employment and contact details can all evolve.

A record from 2018 should not be rejected as a match to a 2026 record merely because the address changed, if other evidence supports continuity.

Historical Aliases

Known aliases, former names and old identifiers should be preserved with valid periods and provenance.

This allows matching through time without overwriting historical identity evidence.

Households Are Not People

Two people can share contact information because they belong to one household. Household resolution and person resolution should remain separate entity jobs.

Collapsing them can create serious privacy and communication errors.

Organisations and Branches

One corporate group can contain many legal entities, branches and trading names. Matching should preserve the level of identity required by the receiver.

“Same brand” is not always “same legal entity”.

Products and Variants

A product family, model, size and SKU can be distinct entity levels. Deduplication should not collapse legitimate variants merely because descriptions are similar.

Documents and Editions

Exact file duplicates can be safely linked in many contexts, but near-identical documents may represent meaningful revisions.

A corrected policy document differs materially from the previous edition even if 99.9% of text is unchanged.

Matching Thresholds

Probabilistic systems often use thresholds:

above high threshold: auto-match;
below low threshold: non-match;
between thresholds: manual review.

Thresholds should follow consequence. A marketing mailing list can tolerate different error trade-offs from a financial account merge.

False Merge

A false merge occurs when different entities are combined as one.

This can expose one person’s data to another, combine financial histories, contaminate analytics or create irreversible downstream errors.

High-consequence identity systems should generally bias toward review when evidence is ambiguous.

False Split

A false split occurs when records belonging to one entity remain separate.

This can undercount customers, fragment history, duplicate outreach or prevent complete risk assessment.

The Cost of Each Error Is Different

False merges and false splits rarely have equal consequence. Resolution design should explicitly state which error is more dangerous for the use case.

Clerical Review

Ambiguous matches can be routed to trained human reviewers.

Review interfaces should show evidence symmetrically:

matching fields;
conflicting fields;
source systems;
historical values;
match score components;
prior merge history;
privacy-safe context.

Do Not Show Reviewers the Model Verdict First

Where independent judgement matters, showing a high-confidence model recommendation before evidence can anchor human reviewers.

Review design should balance efficiency against genuine independence.

Link vs Merge

Resolution does not always require destructive merging.

link: preserve source records and assert that they refer to one entity;
merge: consolidate records into one representation;
cluster: group records believed to represent one entity while retaining membership evidence.

Linking is often safer because source provenance remains intact.

Golden Records

A golden record is a governed canonical representation assembled from several sources for shared organisational use.

It should not erase source disagreement. The golden record needs field-level lineage showing why each selected value is considered authoritative.

See Master Data and Reference Data.

Survivorship Rules

Survivorship rules determine which source value becomes the canonical value after records are resolved.

most authoritative source;
most recently verified value;
highest-quality source;
non-null preference;
human-approved override;
domain-specific hierarchy.

“Latest wins” is only safe when the latest source is also legitimate.

Field-Level Survivorship

Different sources may be authoritative for different fields. Finance may own billing address, identity services may own legal name, and communications may own preferred contact channel.

One source need not win the entire record.

Preserve Source Values

Canonicalisation should not destroy original source values. They are needed for audit, dispute resolution and future rule changes.

Merge History

Every consequential merge should record:

records involved;
match evidence;
score or rule;
reviewer where applicable;
survivorship decisions;
time;
resolution version.

See Data Audit Trails and Change History.

Unmerge

A mature identity system needs a controlled path to undo a false merge.

Unmerge is difficult after downstream systems have copied the canonical identity, so merge lineage and dependent-product mapping matter.

Resolution Versioning

Matching rules and models change. The same pair can receive a different decision under a new version.

Resolution outputs should therefore record which rule or model version produced the decision.

Re-Resolution

When a matching algorithm improves, organisations may re-evaluate historical records. Re-resolution should be staged and reconciled because changing identity clusters can affect many downstream products.

Identity Graphs

Entity resolution can be represented as a graph linking source records to canonical entities and aliases.

The graph should distinguish observed source claims from derived same-entity assertions.

See Knowledge Graphs and Semantic Data.

Transitive Matching Is Dangerous

If A matches B and B matches C, it does not automatically follow that A and C are the same entity under every rule.

Cluster formation should test consistency across the full entity rather than relying on unchecked transitivity.

Householding

Householding groups related people or accounts without merging their identities.

This is useful for family, address or organisation-level analysis while preserving person-level distinction.

Privacy

Entity resolution can increase privacy risk because combining records reveals a richer profile than either source alone.

Access should be justified by purpose, and matching data should be minimised where possible.

Protected Characteristics

Sensitive attributes may improve match accuracy but also create unnecessary exposure. Use should be proportionate to consequence and subject to appropriate governance.

Security

Identity-resolution systems are attractive targets because they connect identifiers across sources. Protect match tables, crosswalks and source links with strong access controls and audit.

Data Minimisation

Not every downstream consumer needs the complete identity graph. Publish only the canonical identifiers and fields required by the consumer’s job.

Deduplication in Analytics

Duplicate entities can inflate counts and distort metrics.

“Unique customers” requires a governed definition of customer identity. Counting distinct email addresses is only one proxy.

Deduplication in AI Training

Near-duplicate examples can distort training and evaluation by over-representing repeated content or leaking similar examples across train and test sets.

AI datasets may therefore need exact and near-duplicate controls while preserving meaningful revisions and variants.

See AI Data Management.

Embeddings for Candidate Retrieval

Vector similarity can help generate candidate pairs for names, products or documents. It is useful for recall but should not be treated as final identity evidence without domain-specific validation.

Match Quality Metrics

Useful measures include:

precision of predicted matches;
recall of known true matches;
false merge rate;
false split rate;
manual-review rate;
unmerge rate;
coverage of canonical IDs;
match quality by population or source.

Aggregate accuracy can hide poor performance for names, languages or source systems underrepresented in evaluation data.

Gold-Standard Match Sets

Resolution systems need labelled reference pairs or clusters for evaluation. The labels themselves should be reviewed because identity can be ambiguous and source evidence incomplete.

Monitoring Drift

Match behaviour can change when new source systems, naming conventions, countries or customer populations enter the estate.

Monitor score distributions, manual-review rates and error rates over time.

Resolution Incidents

A false-merge incident should trigger:

Detect → Contain → Identify Affected Records → Unmerge → Correct Canonical Identity → Reconcile Downstream Systems → Review Rule or Model → Add Regression Case

Education Example

A student appears in two systems as “Alicia Tan” and “Tan Alicia”, with different local IDs. A verified parent contact and date of birth support linking the records to one canonical student identity. The original source IDs remain preserved.

Another student with the same name and birth year but a different verified guardian remains separate. Similarity does not override conflicting identity evidence.

Commerce Example

A customer buys under two email addresses but the same verified account and delivery history. The retailer links both profiles to one customer entity for analytics while preserving separate contact-channel histories.

Document Example

A repository contains five identical copies of a PDF and one later corrected edition. Checksums collapse the exact copies into one storage identity, while the corrected edition remains a distinct version linked to the same logical document.

Common Failure Modes

Name equality equals identity: different people are merged.
Surrogate IDs equal real entities: one person stays split across systems.
Normalisation erases distinction: meaningful differences disappear.
Blocking too narrow: true duplicates never become candidates.
Similarity score equals proof: probabilistic evidence becomes certainty.
Latest value always wins: low-authority sources overwrite canonical fields.
Merge destroys source: provenance is lost.
No unmerge path: false merges become permanent.
Transitive collapse: weak pairwise matches create one incorrect giant cluster.
Identity graph overexposed: resolution creates privacy risk beyond the original sources.

An Entity Resolution Checklist

What real-world entity is being resolved?
Which identifiers are genuinely authoritative?
Which source fields need normalisation?
What candidate-generation strategy protects recall?
Which deterministic rules are safe?
Which probabilistic evidence contributes to matching?
What are the false-merge and false-split consequences?
Which thresholds require human review?
Should records be linked or physically merged?
What survivorship rule applies to each field?
Are original source values preserved?
Can merge history be audited and reversed?
How are privacy and access controlled?
How is match quality monitored across populations and sources?
Can downstream systems reconcile after an identity correction?

A Maturity Ladder

Exact-matched: obvious duplicates are removed.
Normalised: comparable representations improve matching.
Candidate-aware: scalable blocking preserves likely pairs.
Evidence-scored: deterministic and probabilistic signals are explicit.
Reviewable: ambiguous cases receive human adjudication.
Survivorship-governed: canonical fields follow source authority.
Reversible: merge provenance supports correction and unmerge.
Adaptive: observed false merges and splits continuously improve resolution rules.

The Deeper Principle: Identity Is a Claim, Not a Formatting Trick

Entity resolution looks technical because it uses keys, similarity scores and matching algorithms. Its real job is epistemic: deciding when separate records are allowed to make one claim about the same thing.

Trust comes from preserving evidence, uncertainty, source authority and reversibility. The best resolution system is not the one that produces the fewest rows. It is the one that consolidates identity only as far as the evidence legitimately supports.

Data Management Series

Final idea: deduplication removes redundant representations; entity resolution governs identity. Keep those jobs distinct enough that efficiency never becomes permission to merge things the evidence cannot safely prove are the same.