Duplicates in Integration: Why They Happen and How to Actually Handle Them

Before you can fix duplicates, you need to understand how they get created in the first place. There are a few usual suspects:

Race conditions. System A creates a record and immediately syncs it to System B. System B processes the sync and creates the record. But before it finishes, a user in System B has already created the same record manually. Now you have two.

Missing or mismatched keys. Your integration uses Email to match contacts, but System A stores john.doe@company.com and System B stores John.Doe@Company.com. The lookup fails, a new record gets created, and now you have twins.

Bidirectional sync without a lock. Record gets updated in System A → syncs to System B → triggers an update event in System B → syncs back to System A → repeat. If you're not careful, this loop also creates phantom records along the way.

Initial load collisions. You're migrating historical data into a new system that already has some records. Nobody mapped the existing data to the incoming data. Duplicates at scale.

System restarts and retries. A sync fails halfway through, the job retries, and the record gets written twice. Especially common when your integration platform doesn't track what was already committed.

The Matching Strategy

Before you write a single mapping or toggle a single field, you need to define how your integration will recognize that two records in different systems represent the same real-world entity.

This is your matching key and it's the single most important design decision you'll make on a duplicate-prone integration.

Common options:

• External ID / Cross-reference ID : System A stores System B's ID (and vice versa). This is the gold standard. Once a record is linked, it will never be confused for another one.
• Business key : A natural identifier that both systems share: email address, VAT number, order number, SKU. Works well when the field is truly unique and consistently formatted.
• Composite key : A combination of fields (e.g., Company Name + Order Number + Line Number).
• Fuzzy matching : Similarity algorithms that catch near-matches and can be implemented in the systems that should be integrated. 

The Architecture Choices That Prevent Duplicates

Duplicate handling isn't just a data problem. It's an architecture problem. These patterns help:

1. Upsert over Insert

Never blindly insert. Always try to match first, then update if found, insert if not. Most integration platforms and APIs support upsert natively.

2. Write Back the ID

When System A creates a record in System B, System B returns an ID. Write that ID back into System A immediately. This creates a permanent, unambiguous link between the two records. Future syncs follow the link instead of trying to re-match by field values. This is especially important in bidirectional integrations. If you skip this step, every sync is a potential duplicate.

3. Single Direction of Truth (at least per entity)

Decide which system owns each record type. Customers might be created in the CRM and pushed to the ERP. Orders might flow the other way. Pick a direction and enforce it. When both systems can create the same type of record independently and sync bidirectionally without guardrails, duplicates are almost inevitable.

4. Deduplication at the Gate

If you have a system that can receive records from multiple sources, build a deduplication check at the entry point before the record is written. Check for an existing match. If found, merge or reject with a clear log message. Don't let the record through and deal with the mess later.

5. Idempotent Sync Jobs

Design your integration jobs so that running them twice produces the same result as running them once. This means: if a record was already synced, re-syncing it updates it in place rather than creating a new one. Idempotency is your safety net for retries, failures, and reruns.

The Summary

Duplicates in integrations are a design problem more than a technical one. They appear when there's ambiguity about identity, ownership, or sequencing and they tend to compound over time.

The fundamentals:

• Define your matching key before anything else
• Use upsert, write back IDs, and keep sync jobs idempotent
• Establish a single system of record per entity type
• Build exception handling for the cases you can't match
• Have the business ownership conversation early

Get those right, and duplicates become the exception rather than the default.