--- title: "Duplicates in Integration: Why They Happen and How to Actually Handle Them" description: "Duplicates in Integration: Why They Happen and How to Actually Handle Them" keywords: "integration duplicates,duplicate records, matching key,data quality, record matching, deduplication strategy,single source of truth" url: "https://myrapidi.com/wiki/duplicates_in_integration_why_they_happen_and_how_to_actually_handle_them" locale: "en" published: "2026-06-30T11:43:00Z" modified: "2026-06-30T11:49:32Z" --- Before you can fix duplicates, you need to understand how they get created in the first place. There are a few usual suspects: **Race conditions.** System A creates a record and immediately syncs it to System B. System B processes the sync and creates the record. But before it finishes, a user in System B has already created the same record manually. Now you have two. **Missing or mismatched keys.** Your integration uses Email to match contacts, but System A stores john.doe@company.com and System B stores John.Doe@Company.com. The lookup fails, a new record gets created, and now you have twins. **Bidirectional sync without a lock.** Record gets updated in System A → syncs to System B → triggers an update event in System B → syncs back to System A → repeat. If you're not careful, this loop also creates phantom records along the way. **Initial load collisions.** You're migrating historical data into a new system that already has some records. Nobody mapped the existing data to the incoming data. Duplicates at scale. **System restarts and retries.** A sync fails halfway through, the job retries, and the record gets written twice. Especially common when your integration platform doesn't track what was already committed. ### **The Matching Strategy** Before you write a single mapping or toggle a single field, you need to define how your integration will recognize that two records in different systems represent the same real-world entity. This is your **matching key** and it's the single most important design decision you'll make on a duplicate-prone integration. Common options: - **External ID / Cross-reference ID** : System A stores System B's ID (and vice versa). This is the gold standard. Once a record is linked, it will never be confused for another one. - **Business key** : A natural identifier that both systems share: email address, VAT number, order number, SKU. Works well when the field is truly unique and consistently formatted. - **Composite key** : A combination of fields (e.g., Company Name + Order Number + Line Number). - **Fuzzy matching** : Similarity algorithms that catch near-matches and can be implemented in the systems that should be integrated.  ### **The Architecture Choices That Prevent Duplicates** Duplicate handling isn't just a data problem. It's an architecture problem. These patterns help: **1. Upsert over Insert** Never blindly insert. Always try to match first, then update if found, insert if not. Most integration platforms and APIs support upsert natively. **2. Write Back the ID** When System A creates a record in System B, System B returns an ID. Write that ID back into System A immediately. This creates a permanent, unambiguous link between the two records. Future syncs follow the link instead of trying to re-match by field values. This is especially important in bidirectional integrations. If you skip this step, every sync is a potential duplicate. **3. Single Direction of Truth (at least per entity)** Decide which system _owns_ each record type. Customers might be created in the CRM and pushed to the ERP. Orders might flow the other way. Pick a direction and enforce it. When both systems can create the same type of record independently and sync bidirectionally without guardrails, duplicates are almost inevitable. **4. Deduplication at the Gate** If you have a system that can receive records from multiple sources, build a deduplication check at the entry point before the record is written. Check for an existing match. If found, merge or reject with a clear log message. Don't let the record through and deal with the mess later. **5. Idempotent Sync Jobs** Design your integration jobs so that running them twice produces the same result as running them once. This means: if a record was already synced, re-syncing it updates it in place rather than creating a new one. Idempotency is your safety net for retries, failures, and reruns. **The Summary** Duplicates in integrations are a design problem more than a technical one. They appear when there's ambiguity about identity, ownership, or sequencing and they tend to compound over time. The fundamentals: - Define your matching key before anything else - Use upsert, write back IDs, and keep sync jobs idempotent - Establish a single system of record per entity type - Build exception handling for the cases you can't match - Have the business ownership conversation early Get those right, and duplicates become the exception rather than the default.