Key takeaways
- Deduplication identifies repeated records and reconciles them under an explicit identity rule.
- A merge must preserve relationships, history and the correct surviving values.
- Prevent duplicate creation as well as repairing duplicates that already exist.
Overview
Exact identifiers make some duplicates straightforward, but names and domains can be ambiguous. Define what counts as the same entity before matching. A parent company and subsidiary may be related without being duplicates. Merge rules must also decide which values survive, how relationships move and how the operation can be reviewed or reversed.
How it works
Normalize useful identifiers and generate candidate duplicate pairs.
Apply matching rules with confidence thresholds and review paths.
Merge or link approved duplicates while preserving provenance and relationships.
Define what makes two records the same
Two records can share a company name without representing the same organization, or describe the same person under different names and employers. Decide whether the entity being deduplicated is a person, an employment relationship, a company or an event. The definition changes which identifiers and time relationships matter.
Use exact stable identifiers when they are reliable, then add controlled candidate matching for incomplete records. A matching email can be helpful but may be shared, reassigned or obsolete. A similar name is a reason to investigate, not sufficient evidence to merge automatically. Preserve the evidence and confidence behind each decision.
| Candidate pair | Why it may match | What to check before merging |
|---|---|---|
| Same CRM external ID | A repeated import of one source record | Identifier scope and source integrity |
| Same name and company | Possibly one person entered twice | Profile, role and other identity evidence |
| Parent and subsidiary | Related names and domains | Whether they should remain separate accounts |
| Repeated event ID | A retried delivery | Event source and intended operation scope |
Design the surviving record and relationships
Choose field-level survivorship rules rather than keeping whichever record was created first. A newer record may contain a corrected phone number while an older record holds the opportunity history. Preserve activities, ownership, source attribution and suppression status when relationships are moved. A merge that reduces rows but loses those connections is not a successful cleanup.
Record the merged identifiers and the reason for the decision so later imports can resolve to the survivor. Where possible, retain a reversible mapping or audit record. Ambiguous candidates should remain in a review queue instead of being forced into a merge to meet a duplicate-count target.
Stop the same duplicates returning
An illustrative cleanup merges 80 duplicate records, but the next nightly import recreates 50 of them because it uses a name as the only lookup key. The durable fix is to repair source identifiers and creation logic, then reconcile existing duplicates. Idempotency also prevents repeated processing of one intended action from creating multiple records.
Measure incorrect merges as well as missed duplicates. A false merge can be more damaging than leaving a duplicate for review because it combines unrelated histories and permissions. Test repeat imports, concurrent creation and renamed companies. A lower duplicate count is useful only if the surviving records still represent the correct entities.
What this looks like in practice
Two CRM contacts share the same verified professional profile but have different old email addresses. The team links them to one person and preserves employment history rather than discarding the older evidence.
Examples explain the concept; they are not reported customer results.What to check
Measure false merges and missed duplicates separately. False merges can be more damaging than leaving a duplicate for review, especially for permissions or account ownership.
Common mistake
Merging everyone with the same name or every company sharing a domain without checking entity context.
Data deduplication vs. Entity resolution
Entity resolution determines which records refer to the same real-world entity across sources. Deduplication uses that decision to manage repeated records within a dataset or system.
Read the Entity resolution definition →Questions answered
What is Data deduplication?
Data deduplication identifies records representing the same entity or event and applies rules to merge, suppress or link those duplicates without losing important information.
Should duplicate records always be deleted?
No. Linking or merging with an audit trail may preserve important history. Deletion should follow explicit retention and recovery rules.
Can a unique database constraint solve deduplication?
It can prevent exact duplicates on a chosen key, but it cannot determine whether two different keys represent the same person or company.
Should duplicate records be deleted or merged?
When both records contain useful relationships or history, a controlled merge is often preferable to deletion. The right action depends on the entity and system. Preserve necessary identifiers, source attribution and exclusions so the cleanup does not break downstream references or reintroduce records that should remain suppressed.
Can fuzzy matching safely automate deduplication?
It can help generate candidates and, with validated thresholds, automate some clear cases. Similarity is not identity. Evaluate false merges and missed matches on representative records, use stronger identifiers where available and route ambiguous cases to review before combining consequential histories or permissions.
References and further reading
Primary documentation and source material for this topic. Sources checked September 14, 2026; provider requirements can change.
Continue reading on the blog
Explore all articles and guides →Put the concept to work.
Explore the relevant AstroFabric workflow and see how the pieces connect.
Help keep this guide useful. Suggest a correction or browse the full glossary.