{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":["admonition","partial"]},"type":"markdown"},"seo":{"title":"Duplicate Guard Integration","description":"How a Duplicate Guard Monitor, the link database, and connected-component grouping fit together when you consume duplicate detection from your own system."},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"duplicate-guard-integration","__idx":0},"children":["Duplicate Guard Integration"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"overview","__idx":1},"children":["Overview"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Duplicate Guard finds duplicate business partners continuously and stores what it finds as ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["links"]},": scored pairs of"," ","records that appear to describe the same company. Everything CDQ publishes about duplication - the reports, the review"," ","apps, the API - is built on that one primitive."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you consume Duplicate Guard from your own system rather than from the CDQ Cloud Apps, there is one conceptual step"," ","you have to own: turning those pairs into ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["groups"]},". A supplier that exists five times in your ERP does not arrive as"," ","one cluster of five. It arrives as up to ten separate pairs, each with its own score. Deciding which of those pairs"," ","belong to the same real-world company, and which record of that company survives, happens on your side, deliberately."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This article explains why the boundary sits there, what each part of the integration surface does, and what the"," ","grouping step has to get right to be safe."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["In this article, you will understand:"]}]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Why Duplicate Guard returns scored pairs rather than finished duplicate groups"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["What the matching configuration, the monitor, and the link database each contribute"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["How links are retrieved and which identifier connects them back to your own records"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["How connected-component grouping turns pairs into groups, and where that technique is dangerous"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["How review decisions and the monitor lifecycle keep the result current"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Available in:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/apis/data-matching-api/api-v2"},"children":["Data Matching API"]}," - the matching configuration, the links, and the review status"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/apis/data-exchange-api/api-v5"},"children":["Data Exchange API"]}," - the data mirror, the data sources, and the monitor"]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"what-a-duplicate-guard-integration-consists-of","__idx":2},"children":["What a Duplicate Guard integration consists of"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Three things belong to CDQ, one belongs to you."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"text","header":{"controls":{"copy":{}}},"source":"  CDQ                                                        Your system\n\n  Matching configuration\n  (what counts as a duplicate)\n          |\n          v\n  Duplicate Guard Monitor  ---->  Link database   ---->  read the links   ---->  connected components\n  (runs continuously)             A <-> B  0.93          A <-> B  0.93           { A, B, C }\n                                  B <-> C  0.88          B <-> C  0.88           { D, E }\n                                  D <-> E  0.95          D <-> E  0.95                  |\n                                                                                        v\n                                          ^                                    elect a survivor\n                                          |                                             |\n                                          +----  review decisions  <-------------  merge / report\n","lang":"text"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["matching configuration"]}," is a stored definition of what similarity means for your data: which records are compared"," ","at all, which attributes are scored, with which cleaners and comparators, and where the thresholds sit. It is created"," ","once in the Duplicate Guard Configurator app and then referenced by ID."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["monitor"]}," is a standing object bound to that configuration and to one or more data sources. Creating it starts a"," ","scan of everything in scope; after that it keeps watching. A changed record is rematched, and links that stop being"," ","valid are removed without you asking."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["link database"]}," holds the result. Each entry is one pair, its score, its confidence label, its review status, and"," ","a per-attribute explanation of how the score came about."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Grouping is yours."]}," So is the decision about which record survives a merge, because only your system knows what a"," ","surviving record has to satisfy."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info","name":"Curated records, not raw rows"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A Duplicate Guard Monitor matches the curated business partners in your data mirror. A data source that has never been"," ","processed has nothing for the monitor to compare, and the link database stays empty."]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"why-the-api-returns-pairs-not-groups","__idx":3},"children":["Why the API returns pairs, not groups"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The honest answer is that similarity is not transitive, and a grouped result would hide that."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["A"]}," resembles ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["B"]}," with a score of ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.93"]},", and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["B"]}," resembles ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["C"]}," with ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.88"]},", nothing follows about ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["A"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["C"]},". They"," ","may be near-identical. They may be two different subsidiaries that happen to share a street with the same head office."," ","They may never have been compared at all."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The moment the API returned ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["{A, B, C}"]}," as a finished group, that judgement would be baked in and you could not"," ","reconstruct what it was based on. By returning the pairs and their explanations, Duplicate Guard leaves you the two"," ","decisions that genuinely depend on your business: at which score a pair counts, and what to do with the cluster that"," ","results."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This is also why the same link set can legitimately produce different groupings. A migration that must not lose a"," ","single record and a reporting job that counts duplication rates will reasonably read the same links at different"," ","thresholds."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"the-integration-surface","__idx":4},"children":["The integration surface"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"the-matching-configuration","__idx":5},"children":["The matching configuration"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Two settings in the configuration shape everything you will see downstream, so it is worth knowing them before you"," ","interpret any result."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["candidateSearchConfiguration.maxCandidates"]}," caps how many other records any single record is ever compared against."," ","It exists because comparing everything with everything does not scale: a population of 100,000 records has five billion"," ","possible pairs. A small set of plausible candidates is selected first, and only those are scored. The consequence for"," ","your integration is that this value is a ceiling on recall. With ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["maxCandidates"]}," set to 10, a record can never carry"," ","more than ten links, however many near-identical twins it has."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["generalMatchingConfiguration.threshold"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["thresholdMaybe"]}," decide the label a scored pair receives. At or above"," ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["threshold"]}," a pair is a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["MATCH"]},"; between ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["thresholdMaybe"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["threshold"]}," it is a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["MAYBE_MATCH"]},". Pairs below are"," ","discarded, so the link database contains only ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["MATCH"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["MAYBE_MATCH"]},"."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info","name":"Configuration in depth"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The full structure, the available cleaners and comparators, and the default configuration are described in"," ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/articles/processes/duplicate-guard-matching-process"},"children":["Duplicate Guard Matching Process"]},"."," ","To set one up through the apps, see"," ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/tutorials/duplicate-guard/ddg-business-partners-deduplication-monitor"},"children":["How to set up and activate Duplicate Guard Monitor"]},"."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"the-monitor","__idx":6},"children":["The monitor"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A monitor is created against a data mirror with a type of ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["DUPLICATE"]},", a configuration, and the data sources in scope:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"{\n    \"type\": { \"technicalKey\": \"DUPLICATE\" },\n    \"dataSources\": [\n        { \"id\": \"YOUR_DATA_SOURCE_ID\" }\n    ],\n    \"enabled\": \"true\",\n    \"configuration\": { \"id\": \"YOUR_MATCHING_CONFIGURATION_ID\" },\n    \"executions\": [\n        { \"type\": { \"technicalKey\": \"TRIGGER\" } }\n    ],\n    \"scopes\": [\n        { \"technicalKey\": \"AUGMENTED_BUSINESS_PARTNERS\" }\n    ]\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSources"]}," - between one and seven data sources of the same data mirror"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["configuration.id"]}," - the matching configuration the monitor uses"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["executions[].type.technicalKey"]}," - ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["TRIGGER"]}," rematches a record whenever its data changes"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["scopes[].technicalKey"]}," - ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AUGMENTED_BUSINESS_PARTNERS"]}," matches the curated records"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["That ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSources"]}," is an array is not incidental. It is the mechanism for detecting duplicates ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["between"]}," systems. One"," ","monitor pointed at the data sources of two legacy ERPs will link the same customer across both, with no prior"," ","consolidation of the data. Every link names the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSourceId"]}," on both sides, so cross-source pairs stay"," ","distinguishable from pairs inside a single source."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"warning","name":"One duplicate monitor per data source"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A data source can carry only one monitor of a given type. An integration that creates monitors as part of its setup"," ","should list the existing monitors of the data mirror first and look for one of type ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["DUPLICATE"]}," covering that data"," ","source, rather than creating a second."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Creating the monitor begins a background scan. The progress endpoint reports it as a breakdown: ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["processingStatuses"]}," is"," ","a ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["list of buckets"]},", each naming a processing state and carrying its own record count and share, and together they"," ","account for all records in scope. It is not a single overall verdict, and reading the first bucket as one is the most"," ","common misreading of that response."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["You do not have to wait for the last record before consuming anything. Links appear as they are produced, so a job that"," ","reads on a schedule simply sees the set grow and then settle."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"retrieving-links","__idx":7},"children":["Retrieving links"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Links are read per monitor. Two query parameters are mandatory: the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataMonitorId"]},", and a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["matchingScore"]}," that acts as"," ","a floor on the score. That floor filters the response; it does not change what the monitor stored, so the same link"," ","database can be read at several thresholds without re-running anything."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A single link looks like this:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"json","header":{"controls":{"copy":{}}},"source":"{\n    \"id\": \"YOUR_LINK_ID\",\n    \"dataMonitorId\": \"YOUR_DATA_MONITOR_ID\",\n    \"entityA\": {\n        \"dataSourceId\": \"SAP_ERP\",\n        \"businessPartnerId\": \"YOUR_BUSINESS_PARTNER_ID_A\",\n        \"externalId\": \"0001004211\"\n    },\n    \"entityB\": {\n        \"dataSourceId\": \"SAP_ERP\",\n        \"businessPartnerId\": \"YOUR_BUSINESS_PARTNER_ID_B\",\n        \"externalId\": \"0001009873\"\n    },\n    \"confidence\": {\n        \"overallScore\": 0.95,\n        \"status\": \"MATCH\",\n        \"explanation\": {\n            \"attributeMatchingExplanation\": [\n                {\n                    \"jsonPath\": \"businessPartner.name\",\n                    \"valueA\": \"Acme Corporation GmbH\",\n                    \"valueB\": \"ACME Corp. GmbH\",\n                    \"similarity\": 0.92\n                },\n                {\n                    \"jsonPath\": \"businessPartner.address.street\",\n                    \"valueA\": \"123 Main Street\",\n                    \"valueB\": \"123 Main St.\",\n                    \"similarity\": 0.98\n                }\n            ]\n        }\n    },\n    \"linkStatus\": \"INFERRED\"\n}\n","lang":"json"},"children":[]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["entityA"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["entityB"]}," - the two linked records, each with its data source, the CDQ business partner ID, and your own ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["externalId"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["confidence.overallScore"]}," - similarity of the pair, between ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["1"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["confidence.status"]}," - ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["MATCH"]}," or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["MAYBE_MATCH"]},", derived from the configuration thresholds"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["confidence.explanation.attributeMatchingExplanation"]}," - per attribute, both raw values and their similarity"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["linkStatus"]}," - the review state of the link"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"which-identifier-to-join-on","__idx":8},"children":["Which identifier to join on"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Both identifiers are returned for a reason. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["businessPartnerId"]}," is the key CDQ uses internally; ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["externalId"]}," is the key"," ","your own system supplied. Grouping and any join back onto your records has to run on ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["externalId"]},"."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"danger","name":"The silent join failure"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Joining links to your records through ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["businessPartnerId"]}," produces no matches and no error. Your system will report"," ","zero duplicates on a population full of them. This is the single most common integration defect, and it is invisible"," ","unless you assert that the join produced rows."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"reading-the-whole-result","__idx":9},"children":["Reading the whole result"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The endpoint pages with a cursor: take ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["nextStartAfter"]}," from the response and send it as ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["startAfter"]}," on the next"," ","request, stopping when no cursor comes back. Two defaults are worth overriding deliberately. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["limit"]}," defaults to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["20"]}," ","and accepts up to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["250"]},", and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["total"]}," is only returned when ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["NUMBER_OF_TOTAL"]}," is requested through ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["featuresOn"]},"."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"python","header":{"controls":{"copy":{}}},"source":"import requests\n\nBASE = \"https://api.cdq.com/data-matching/rest\"\n\n\ndef fetch_links(api_key, data_monitor_id, matching_score=0.8):\n    \"\"\"Return every link of one Duplicate Guard Monitor, following the paging cursor.\"\"\"\n    params = {\n        \"dataMonitorId\": data_monitor_id,\n        \"matchingScore\": matching_score,\n        \"limit\": 250,\n        \"sort\": \"-matchingScore\",\n        \"featuresOn\": \"NUMBER_OF_TOTAL\",\n    }\n    headers = {\"X-API-KEY\": api_key}\n    links, total = [], 0\n\n    while True:\n        response = requests.get(\n            f\"{BASE}/entityresolution/entitymatchinglinks\",\n            headers=headers,\n            params=params,\n            timeout=60,\n        )\n        response.raise_for_status()\n        page = response.json()\n\n        values = page.get(\"values\", [])\n        links.extend(values)\n        total = page.get(\"total\", total)\n\n        cursor = page.get(\"nextStartAfter\")\n        if not cursor or not values:\n            break\n        params[\"startAfter\"] = cursor\n\n    return links, total\n","lang":"python"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Links can also be narrowed to one data source, to specific records through ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["externalIds"]}," or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["businessPartnerIds"]},", to a"," ","confidence label, or to a review status. Filtering on ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["status=INFERRED"]}," is what makes an incremental review possible,"," ","as described further below."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info","name":"Keep the explanation"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["attributeMatchingExplanation"]}," is the only place that records why a pair scored as it did. Storing it next to the link"," ","costs very little and turns every later question about a false positive into a lookup instead of an investigation."]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"from-links-to-duplicate-groups","__idx":10},"children":["From links to duplicate groups"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A set of pairs is a graph: records are nodes, links are edges. A duplicate group is a ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["connected component"]}," of that"," ","graph, and the standard way to compute connected components incrementally is a disjoint-set structure, usually called"," ","union-find. Each link merges the sets of its two records; what remains at the end are the groups. With path compression"," ","it runs in near-linear time, so it stays comfortable well beyond the volumes a monitor covers."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"python","header":{"controls":{"copy":{}}},"source":"from collections import defaultdict\n\n\nclass UnionFind:\n    \"\"\"Disjoint set with path compression and union by rank.\"\"\"\n\n    def __init__(self):\n        self._parent, self._rank = {}, {}\n\n    def find(self, item):\n        if item not in self._parent:\n            self._parent[item], self._rank[item] = item, 0\n        if self._parent[item] != item:\n            self._parent[item] = self.find(self._parent[item])\n        return self._parent[item]\n\n    def union(self, left, right):\n        root_left, root_right = self.find(left), self.find(right)\n        if root_left == root_right:\n            return\n        if self._rank[root_left] < self._rank[root_right]:\n            root_left, root_right = root_right, root_left\n        self._parent[root_right] = root_left\n        if self._rank[root_left] == self._rank[root_right]:\n            self._rank[root_left] += 1\n\n\ndef record_key(entity):\n    \"\"\"Group on your own identifier, falling back to the CDQ one only if absent.\"\"\"\n    return str(entity.get(\"externalId\") or entity.get(\"businessPartnerId\", \"\")).strip()\n\n\ndef build_groups(links, min_score=None):\n    \"\"\"Turn pairwise links into duplicate groups.\n\n    min_score drops weak links BEFORE grouping, so a group only ever contains\n    records connected by links that meet your bar.\n    \"\"\"\n    if min_score is not None:\n        links = [\n            link for link in links\n            if link.get(\"confidence\", {}).get(\"overallScore\", 0.0) >= min_score\n        ]\n\n    union_find = UnionFind()\n    for link in links:\n        left = record_key(link.get(\"entityA\", {}))\n        right = record_key(link.get(\"entityB\", {}))\n        if left and right:\n            union_find.union(left, right)\n\n    members = defaultdict(set)\n    for link in links:\n        for side in (\"entityA\", \"entityB\"):\n            key = record_key(link.get(side, {}))\n            if key:\n                members[union_find.find(key)].add(key)\n\n    group_links = defaultdict(list)\n    for link in links:\n        left = record_key(link.get(\"entityA\", {}))\n        if left:\n            group_links[union_find.find(left)].append(link)\n\n    groups = []\n    for group_id, (root, member_set) in enumerate(sorted(members.items())):\n        if len(member_set) < 2:\n            continue\n        scores = [\n            link.get(\"confidence\", {}).get(\"overallScore\", 0.0)\n            for link in group_links[root]\n        ]\n        groups.append({\n            \"group_id\": group_id,\n            \"members\": sorted(member_set),\n            \"survivor\": min(member_set),\n            \"links\": group_links[root],\n            \"max_score\": max(scores) if scores else 0.0,\n        })\n    return groups\n","lang":"python"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Records that appear in no link are singletons and are not groups. Whether you carry them through as groups of one is a"," ","choice about your own downstream model, not about the data."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"where-connected-components-become-dangerous","__idx":11},"children":["Where connected components become dangerous"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Connected components follow chains, and that is both why the technique works and where it bites."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"danger","name":"A group's best score does not vouch for its members"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Given ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["A <-> B"]}," at ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.93"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["B <-> C"]}," at ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.88"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["C <-> D"]}," at ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.86"]},", all four records land in a single group whose"," ","best score is ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.93"]},". Yet ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["A"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["D"]}," may have scored far below your threshold, or may never have been compared at all"," ","because the candidate search never brought them together. Deleting non-survivors because the ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["group"]}," score is high will"," ","eventually destroy a record that was never similar to the one it was merged into."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Two habits contain this, and both are cheap."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Apply the threshold to the links, not to the finished group."]}," That is what ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["min_score"]}," does above: weak edges are"," ","removed before the components are computed, so every edge inside a group is one you accepted. The chain still exists,"," ","but each link in it meets your bar."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Check each member's strongest link within its group before merging or deleting."]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"python","header":{"controls":{"copy":{}}},"source":"def member_best_scores(group):\n    \"\"\"For each member, the best score it reaches with any other member of the group.\"\"\"\n    best = {member: 0.0 for member in group[\"members\"]}\n    for link in group[\"links\"]:\n        left = record_key(link[\"entityA\"])\n        right = record_key(link[\"entityB\"])\n        score = link.get(\"confidence\", {}).get(\"overallScore\", 0.0)\n        if left in best:\n            best[left] = max(best[left], score)\n        if right in best:\n            best[right] = max(best[right], score)\n    return best\n\n\ndef all_members_qualify(group, threshold):\n    return all(score >= threshold for score in member_best_scores(group).values())\n","lang":"python"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Reporting both figures is more useful than reporting either: how many groups contain at least one qualifying pair, and"," ","how many groups have every member qualifying. The difference between the two numbers is precisely the set that deserves"," ","a human look, and it is usually much smaller than either total."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Before fixing a threshold at all, group the same links at several values and watch where large clusters break apart. A"," ","cluster of 300 records at ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.85"]}," that splits into 40 clusters at ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["0.90"]}," was never one company; it was a chain through a"," ","shared street or a common name fragment."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"electing-a-survivor","__idx":12},"children":["Electing a survivor"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Each group needs exactly one record that survives, and the rule is a business decision rather than a technical one:"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Rule"},"children":["Rule"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Behavior"},"children":["Behavior"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Lowest identifier"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The lexicographically smallest external ID. Deterministic and trivially auditable, which is why it is the common default."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Most linked"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The member appearing in the most links, that is, the most central record of the group."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Most complete"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The member with the highest field completeness in your own system."]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Whichever rule you choose, it has to be deterministic. If the same links can yield a different survivor on a later run,"," ","merge results are not reproducible and no past delivery can be explained. If your process already excludes certain"," ","records from surviving, such as blocked partners or billing-only addresses, elect the survivor among the remaining"," ","members; otherwise a group can lose its survivor to a later filter and vanish from the result entirely."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"keeping-the-result-current","__idx":13},"children":["Keeping the result current"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A Duplicate Guard Monitor is not a job that finishes. After the initial scan it moves into continuous monitoring: new"," ","and changed records are matched, links that are no longer valid disappear, and deleting a record removes every link it"," ","appeared in. An integration therefore re-reads rather than accumulates."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Two events change the result wholesale and are worth handling explicitly. Changing the matching configuration rebuilds"," ","every ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INFERRED"]}," link from scratch, so a cached copy is stale rather than merely incomplete. Re-evaluation can also be"," ","triggered deliberately when you want the monitor to reprocess without waiting for record changes."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"review-status","__idx":14},"children":["Review status"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Every link carries a status that records what a human has concluded about it."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"width":"18%","data-label":"Status"},"children":["Status "]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Meaning"},"children":["Meaning"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{"align":"center"},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INFERRED"]}," "]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Found by the matching algorithm. Not yet reviewed."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{"align":"center"},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ASSERTED"]}," "]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Confirmed correct through review."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{"align":"center"},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["RETRACTED"]}," "]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Confirmed incorrect through review."]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A link moves from ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INFERRED"]}," to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ASSERTED"]}," or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["RETRACTED"]},", and between those two in either direction. It never returns"," ","to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INFERRED"]},": once a person has judged a pair, that judgement is a fact about the pair, not a state the algorithm may"," ","overwrite."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Two consequences matter for an integration. Reviewed links survive a configuration change, while ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INFERRED"]}," links are"," ","rebuilt, so the effort your reviewers invest is not lost when the configuration is tuned. And because the status is"," ","filterable, a review cycle can ask only for ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["INFERRED"]}," links and start where the previous cycle stopped, instead of"," ","re-presenting every pair each time."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"summary","__idx":15},"children":["Summary"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Duplicate Guard owns detection and keeps it current; your system owns grouping and the merge decision. The boundary"," ","sits at the pair because similarity is not transitive, and a grouped result would conceal the judgement that made it."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Three things decide whether an integration is correct rather than merely working. Join on ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["externalId"]},", because the"," ","alternative fails silently. Apply your threshold to the links before computing connected components, because a group's"," ","best score says nothing about its weakest member. And elect the survivor deterministically, because an unreproducible"," ","merge cannot be explained after the fact."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["With those in place, the groups are yours to act on: merge them in your master data system, route the uncertain ones to"," ","a reviewer, or report duplication per country, per data source, or per business unit."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"related-resources","__idx":16},"children":["Related Resources"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/articles/processes/duplicate-guard-matching-process"},"children":["Duplicate Guard Matching Process"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/tutorials/duplicate-guard/ddg-business-partners-deduplication-monitor"},"children":["How to set up and activate Duplicate Guard Monitor"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/apis/data-matching-api/api-v2"},"children":["Data Matching API"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/apis/data-exchange-api/api-v5"},"children":["Data Exchange API"]}]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"your-opinion-matters","__idx":17},"children":["Your opinion matters!"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!"]}]},"headings":[{"value":"Duplicate Guard Integration","id":"duplicate-guard-integration","depth":1},{"value":"Overview","id":"overview","depth":2},{"value":"What a Duplicate Guard integration consists of","id":"what-a-duplicate-guard-integration-consists-of","depth":2},{"value":"Why the API returns pairs, not groups","id":"why-the-api-returns-pairs-not-groups","depth":2},{"value":"The integration surface","id":"the-integration-surface","depth":2},{"value":"The matching configuration","id":"the-matching-configuration","depth":3},{"value":"The monitor","id":"the-monitor","depth":3},{"value":"Retrieving links","id":"retrieving-links","depth":2},{"value":"Which identifier to join on","id":"which-identifier-to-join-on","depth":3},{"value":"Reading the whole result","id":"reading-the-whole-result","depth":3},{"value":"From links to duplicate groups","id":"from-links-to-duplicate-groups","depth":2},{"value":"Where connected components become dangerous","id":"where-connected-components-become-dangerous","depth":3},{"value":"Electing a survivor","id":"electing-a-survivor","depth":3},{"value":"Keeping the result current","id":"keeping-the-result-current","depth":2},{"value":"Review status","id":"review-status","depth":3},{"value":"Summary","id":"summary","depth":2},{"value":"Related Resources","id":"related-resources","depth":2},{"value":"Your opinion matters!","id":"your-opinion-matters","depth":2}],"frontmatter":{"seo":{"title":"Duplicate Guard Integration","description":"How a Duplicate Guard Monitor, the link database, and connected-component grouping fit together when you consume duplicate detection from your own system."},"toc":{"enable":true},"cdqContentMetadata":{"product":"Duplicate Guard","type":"article","subtype":"integration","lastUpdate":"2026-09-25T00:00:00.000Z","author":"Simon Schlosser"}},"lastModified":"2026-10-21T13:36:59.830Z","pagePropGetterError":{"message":"","name":""}},"slug":"/documentation/_public/product-manuals/duplicate-guard-api-integration","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}