Duplicate Guard finds duplicate business partners continuously and stores what it finds as links: scored pairs of records that appear to describe the same company. Everything CDQ publishes about duplication - the reports, the review apps, the API - is built on that one primitive.
If you consume Duplicate Guard from your own system rather than from the CDQ Cloud Apps, there is one conceptual step you have to own: turning those pairs into groups. A supplier that exists five times in your ERP does not arrive as one cluster of five. It arrives as up to ten separate pairs, each with its own score. Deciding which of those pairs belong to the same real-world company, and which record of that company survives, happens on your side, deliberately.
This article explains why the boundary sits there, what each part of the integration surface does, and what the grouping step has to get right to be safe.
In this article, you will understand:
- Why Duplicate Guard returns scored pairs rather than finished duplicate groups
- What the matching configuration, the monitor, and the link database each contribute
- How links are retrieved and which identifier connects them back to your own records
- How connected-component grouping turns pairs into groups, and where that technique is dangerous
- How review decisions and the monitor lifecycle keep the result current
Available in:
- Data Matching API - the matching configuration, the links, and the review status
- Data Exchange API - the data mirror, the data sources, and the monitor
Three things belong to CDQ, one belongs to you.
CDQ Your system
Matching configuration
(what counts as a duplicate)
|
v
Duplicate Guard Monitor ----> Link database ----> read the links ----> connected components
(runs continuously) A <-> B 0.93 A <-> B 0.93 { A, B, C }
B <-> C 0.88 B <-> C 0.88 { D, E }
D <-> E 0.95 D <-> E 0.95 |
v
^ elect a survivor
| |
+---- review decisions <------------- merge / reportThe matching configuration is a stored definition of what similarity means for your data: which records are compared at all, which attributes are scored, with which cleaners and comparators, and where the thresholds sit. It is created once in the Duplicate Guard Configurator app and then referenced by ID.
The monitor is a standing object bound to that configuration and to one or more data sources. Creating it starts a scan of everything in scope; after that it keeps watching. A changed record is rematched, and links that stop being valid are removed without you asking.
The link database holds the result. Each entry is one pair, its score, its confidence label, its review status, and a per-attribute explanation of how the score came about.
Grouping is yours. So is the decision about which record survives a merge, because only your system knows what a surviving record has to satisfy.
A Duplicate Guard Monitor matches the curated business partners in your data mirror. A data source that has never been processed has nothing for the monitor to compare, and the link database stays empty.
The honest answer is that similarity is not transitive, and a grouped result would hide that.
If A resembles B with a score of 0.93, and B resembles C with 0.88, nothing follows about A and C. They may be near-identical. They may be two different subsidiaries that happen to share a street with the same head office. They may never have been compared at all.
The moment the API returned {A, B, C} as a finished group, that judgement would be baked in and you could not reconstruct what it was based on. By returning the pairs and their explanations, Duplicate Guard leaves you the two decisions that genuinely depend on your business: at which score a pair counts, and what to do with the cluster that results.
This is also why the same link set can legitimately produce different groupings. A migration that must not lose a single record and a reporting job that counts duplication rates will reasonably read the same links at different thresholds.
Two settings in the configuration shape everything you will see downstream, so it is worth knowing them before you interpret any result.
candidateSearchConfiguration.maxCandidates caps how many other records any single record is ever compared against. It exists because comparing everything with everything does not scale: a population of 100,000 records has five billion possible pairs. A small set of plausible candidates is selected first, and only those are scored. The consequence for your integration is that this value is a ceiling on recall. With maxCandidates set to 10, a record can never carry more than ten links, however many near-identical twins it has.
generalMatchingConfiguration.threshold and thresholdMaybe decide the label a scored pair receives. At or above threshold a pair is a MATCH; between thresholdMaybe and threshold it is a MAYBE_MATCH. Pairs below are discarded, so the link database contains only MATCH and MAYBE_MATCH.
The full structure, the available cleaners and comparators, and the default configuration are described in Duplicate Guard Matching Process. To set one up through the apps, see How to set up and activate Duplicate Guard Monitor.
A monitor is created against a data mirror with a type of DUPLICATE, a configuration, and the data sources in scope:
{
"type": { "technicalKey": "DUPLICATE" },
"dataSources": [
{ "id": "YOUR_DATA_SOURCE_ID" }
],
"enabled": "true",
"configuration": { "id": "YOUR_MATCHING_CONFIGURATION_ID" },
"executions": [
{ "type": { "technicalKey": "TRIGGER" } }
],
"scopes": [
{ "technicalKey": "AUGMENTED_BUSINESS_PARTNERS" }
]
}dataSources- between one and seven data sources of the same data mirrorconfiguration.id- the matching configuration the monitor usesexecutions[].type.technicalKey-TRIGGERrematches a record whenever its data changesscopes[].technicalKey-AUGMENTED_BUSINESS_PARTNERSmatches the curated records
That dataSources is an array is not incidental. It is the mechanism for detecting duplicates between systems. One monitor pointed at the data sources of two legacy ERPs will link the same customer across both, with no prior consolidation of the data. Every link names the dataSourceId on both sides, so cross-source pairs stay distinguishable from pairs inside a single source.
A data source can carry only one monitor of a given type. An integration that creates monitors as part of its setup should list the existing monitors of the data mirror first and look for one of type DUPLICATE covering that data source, rather than creating a second.
Creating the monitor begins a background scan. The progress endpoint reports it as a breakdown: processingStatuses is a list of buckets, each naming a processing state and carrying its own record count and share, and together they account for all records in scope. It is not a single overall verdict, and reading the first bucket as one is the most common misreading of that response.
You do not have to wait for the last record before consuming anything. Links appear as they are produced, so a job that reads on a schedule simply sees the set grow and then settle.
Links are read per monitor. Two query parameters are mandatory: the dataMonitorId, and a matchingScore that acts as a floor on the score. That floor filters the response; it does not change what the monitor stored, so the same link database can be read at several thresholds without re-running anything.
A single link looks like this:
{
"id": "YOUR_LINK_ID",
"dataMonitorId": "YOUR_DATA_MONITOR_ID",
"entityA": {
"dataSourceId": "SAP_ERP",
"businessPartnerId": "YOUR_BUSINESS_PARTNER_ID_A",
"externalId": "0001004211"
},
"entityB": {
"dataSourceId": "SAP_ERP",
"businessPartnerId": "YOUR_BUSINESS_PARTNER_ID_B",
"externalId": "0001009873"
},
"confidence": {
"overallScore": 0.95,
"status": "MATCH",
"explanation": {
"attributeMatchingExplanation": [
{
"jsonPath": "businessPartner.name",
"valueA": "Acme Corporation GmbH",
"valueB": "ACME Corp. GmbH",
"similarity": 0.92
},
{
"jsonPath": "businessPartner.address.street",
"valueA": "123 Main Street",
"valueB": "123 Main St.",
"similarity": 0.98
}
]
}
},
"linkStatus": "INFERRED"
}entityAandentityB- the two linked records, each with its data source, the CDQ business partner ID, and your ownexternalIdconfidence.overallScore- similarity of the pair, between0and1confidence.status-MATCHorMAYBE_MATCH, derived from the configuration thresholdsconfidence.explanation.attributeMatchingExplanation- per attribute, both raw values and their similaritylinkStatus- the review state of the link
Both identifiers are returned for a reason. businessPartnerId is the key CDQ uses internally; externalId is the key your own system supplied. Grouping and any join back onto your records has to run on externalId.
Joining links to your records through businessPartnerId produces no matches and no error. Your system will report zero duplicates on a population full of them. This is the single most common integration defect, and it is invisible unless you assert that the join produced rows.
The endpoint pages with a cursor: take nextStartAfter from the response and send it as startAfter on the next request, stopping when no cursor comes back. Two defaults are worth overriding deliberately. limit defaults to 20 and accepts up to 250, and total is only returned when NUMBER_OF_TOTAL is requested through featuresOn.
import requests
BASE = "https://api.cdq.com/data-matching/rest"
def fetch_links(api_key, data_monitor_id, matching_score=0.8):
"""Return every link of one Duplicate Guard Monitor, following the paging cursor."""
params = {
"dataMonitorId": data_monitor_id,
"matchingScore": matching_score,
"limit": 250,
"sort": "-matchingScore",
"featuresOn": "NUMBER_OF_TOTAL",
}
headers = {"X-API-KEY": api_key}
links, total = [], 0
while True:
response = requests.get(
f"{BASE}/entityresolution/entitymatchinglinks",
headers=headers,
params=params,
timeout=60,
)
response.raise_for_status()
page = response.json()
values = page.get("values", [])
links.extend(values)
total = page.get("total", total)
cursor = page.get("nextStartAfter")
if not cursor or not values:
break
params["startAfter"] = cursor
return links, totalLinks can also be narrowed to one data source, to specific records through externalIds or businessPartnerIds, to a confidence label, or to a review status. Filtering on status=INFERRED is what makes an incremental review possible, as described further below.
attributeMatchingExplanation is the only place that records why a pair scored as it did. Storing it next to the link costs very little and turns every later question about a false positive into a lookup instead of an investigation.
A set of pairs is a graph: records are nodes, links are edges. A duplicate group is a connected component of that graph, and the standard way to compute connected components incrementally is a disjoint-set structure, usually called union-find. Each link merges the sets of its two records; what remains at the end are the groups. With path compression it runs in near-linear time, so it stays comfortable well beyond the volumes a monitor covers.
from collections import defaultdict
class UnionFind:
"""Disjoint set with path compression and union by rank."""
def __init__(self):
self._parent, self._rank = {}, {}
def find(self, item):
if item not in self._parent:
self._parent[item], self._rank[item] = item, 0
if self._parent[item] != item:
self._parent[item] = self.find(self._parent[item])
return self._parent[item]
def union(self, left, right):
root_left, root_right = self.find(left), self.find(right)
if root_left == root_right:
return
if self._rank[root_left] < self._rank[root_right]:
root_left, root_right = root_right, root_left
self._parent[root_right] = root_left
if self._rank[root_left] == self._rank[root_right]:
self._rank[root_left] += 1
def record_key(entity):
"""Group on your own identifier, falling back to the CDQ one only if absent."""
return str(entity.get("externalId") or entity.get("businessPartnerId", "")).strip()
def build_groups(links, min_score=None):
"""Turn pairwise links into duplicate groups.
min_score drops weak links BEFORE grouping, so a group only ever contains
records connected by links that meet your bar.
"""
if min_score is not None:
links = [
link for link in links
if link.get("confidence", {}).get("overallScore", 0.0) >= min_score
]
union_find = UnionFind()
for link in links:
left = record_key(link.get("entityA", {}))
right = record_key(link.get("entityB", {}))
if left and right:
union_find.union(left, right)
members = defaultdict(set)
for link in links:
for side in ("entityA", "entityB"):
key = record_key(link.get(side, {}))
if key:
members[union_find.find(key)].add(key)
group_links = defaultdict(list)
for link in links:
left = record_key(link.get("entityA", {}))
if left:
group_links[union_find.find(left)].append(link)
groups = []
for group_id, (root, member_set) in enumerate(sorted(members.items())):
if len(member_set) < 2:
continue
scores = [
link.get("confidence", {}).get("overallScore", 0.0)
for link in group_links[root]
]
groups.append({
"group_id": group_id,
"members": sorted(member_set),
"survivor": min(member_set),
"links": group_links[root],
"max_score": max(scores) if scores else 0.0,
})
return groupsRecords that appear in no link are singletons and are not groups. Whether you carry them through as groups of one is a choice about your own downstream model, not about the data.
Connected components follow chains, and that is both why the technique works and where it bites.
Given A <-> B at 0.93, B <-> C at 0.88 and C <-> D at 0.86, all four records land in a single group whose best score is 0.93. Yet A and D may have scored far below your threshold, or may never have been compared at all because the candidate search never brought them together. Deleting non-survivors because the group score is high will eventually destroy a record that was never similar to the one it was merged into.
Two habits contain this, and both are cheap.
Apply the threshold to the links, not to the finished group. That is what min_score does above: weak edges are removed before the components are computed, so every edge inside a group is one you accepted. The chain still exists, but each link in it meets your bar.
Check each member's strongest link within its group before merging or deleting.
def member_best_scores(group):
"""For each member, the best score it reaches with any other member of the group."""
best = {member: 0.0 for member in group["members"]}
for link in group["links"]:
left = record_key(link["entityA"])
right = record_key(link["entityB"])
score = link.get("confidence", {}).get("overallScore", 0.0)
if left in best:
best[left] = max(best[left], score)
if right in best:
best[right] = max(best[right], score)
return best
def all_members_qualify(group, threshold):
return all(score >= threshold for score in member_best_scores(group).values())Reporting both figures is more useful than reporting either: how many groups contain at least one qualifying pair, and how many groups have every member qualifying. The difference between the two numbers is precisely the set that deserves a human look, and it is usually much smaller than either total.
Before fixing a threshold at all, group the same links at several values and watch where large clusters break apart. A cluster of 300 records at 0.85 that splits into 40 clusters at 0.90 was never one company; it was a chain through a shared street or a common name fragment.
Each group needs exactly one record that survives, and the rule is a business decision rather than a technical one:
| Rule | Behavior |
|---|---|
| Lowest identifier | The lexicographically smallest external ID. Deterministic and trivially auditable, which is why it is the common default. |
| Most linked | The member appearing in the most links, that is, the most central record of the group. |
| Most complete | The member with the highest field completeness in your own system. |
Whichever rule you choose, it has to be deterministic. If the same links can yield a different survivor on a later run, merge results are not reproducible and no past delivery can be explained. If your process already excludes certain records from surviving, such as blocked partners or billing-only addresses, elect the survivor among the remaining members; otherwise a group can lose its survivor to a later filter and vanish from the result entirely.
A Duplicate Guard Monitor is not a job that finishes. After the initial scan it moves into continuous monitoring: new and changed records are matched, links that are no longer valid disappear, and deleting a record removes every link it appeared in. An integration therefore re-reads rather than accumulates.
Two events change the result wholesale and are worth handling explicitly. Changing the matching configuration rebuilds every INFERRED link from scratch, so a cached copy is stale rather than merely incomplete. Re-evaluation can also be triggered deliberately when you want the monitor to reprocess without waiting for record changes.
Every link carries a status that records what a human has concluded about it.
| Status | Meaning |
|---|---|
INFERRED | Found by the matching algorithm. Not yet reviewed. |
ASSERTED | Confirmed correct through review. |
RETRACTED | Confirmed incorrect through review. |
A link moves from INFERRED to ASSERTED or RETRACTED, and between those two in either direction. It never returns to INFERRED: once a person has judged a pair, that judgement is a fact about the pair, not a state the algorithm may overwrite.
Two consequences matter for an integration. Reviewed links survive a configuration change, while INFERRED links are rebuilt, so the effort your reviewers invest is not lost when the configuration is tuned. And because the status is filterable, a review cycle can ask only for INFERRED links and start where the previous cycle stopped, instead of re-presenting every pair each time.
Duplicate Guard owns detection and keeps it current; your system owns grouping and the merge decision. The boundary sits at the pair because similarity is not transitive, and a grouped result would conceal the judgement that made it.
Three things decide whether an integration is correct rather than merely working. Join on externalId, because the alternative fails silently. Apply your threshold to the links before computing connected components, because a group's best score says nothing about its weakest member. And elect the survivor deterministically, because an unreproducible merge cannot be explained after the fact.
With those in place, the groups are yours to act on: merge them in your master data system, route the uncertain ones to a reviewer, or report duplication per country, per data source, or per business unit.
- Duplicate Guard Matching Process
- How to set up and activate Duplicate Guard Monitor
- Data Matching API
- Data Exchange API
We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!