# How to set up and activate Duplicate Guard Monitor ## Overview This tutorial explains how to configure, activate, and analyze results from a Duplicate Guard Monitor in CDQ. **Learning Goals** In this tutorial, you will learn how to: * set up a deduplication configuration * activate a duplicate monitor * generate and interpret deduplication reports ## Prerequisites CDQ supports a default matching configuration for Duplicate Monitors. It consists of three search attributes and five matching attributes. Default configuration The default configuration is designed to systematically identify duplicate records based on specified attributes and matching criteria, using various cleaning and comparison techniques to ensure accuracy. **Key components:** The following JSON code represents the default configuration for duplicate matching, which can be used by a Duplicate Monitor to identify and handle duplicate records. Here is the list of the key components: * `"name": "Default Matching Configuration"`: the name of the configuration * `"configuration"`: the structure of the configuration, which contains at least two objects: * `candidateSearchConfiguration` * `generalMatchingConfiguration` * `"candidateSearchConfiguration"`: defines how candidate records are searched for potential duplicates and includes: * `"maxCandidates": 10`: specifies the maximum number of candidate records to consider (in this case, `10`) * `"searchAttributes"`: an array of attributes used to search for candidates, each defined by a JSON path: * `{"jsonPath": "$.names[0].value"}`: searches using the first name value * `{"jsonPath": "$.addresses[0].country.shortName"}`: searches using the country ISO code in the first address * `{"jsonPath": "$.addresses[0].thoroughfares[0].value"}`: searches using the thoroughfare value in the first address * `"generalMatchingConfiguration"`: defines the criteria for matching records and includes: * `"threshold": 0.85`: the score above which records are considered a definite match * `"thresholdMaybe": 0.65`: the score above which records are considered a possible match; the score must be below the `threshold` value — in this case, between `0.65` and `0.85` * `"matchingAttributes"`: an array of attributes used for matching: * `"name"`: descriptive name of the attribute being matched; this name will be used in reports * `"jsonPath"`: path to the attribute in the JSON structure * `"cleaners"`: list of cleaning operations applied to the attribute before matching (e.g., `LowerCaseNormalizer`, `PunctuationsCleaner`) * `"high"` and `"low"`: thresholds for matching scores specific to the attribute * `"comparator"`: method used to compare the attribute values (e.g., `QGramComparator`, `ExactComparator`, `Levenshtein`) **Example default configuration:** ```json { "name": "Default Matching Configuration", "configuration": { "candidateSearchConfiguration": { "maxCandidates": 10, "searchAttributes": [ { "jsonPath": "$.names[0].value" }, { "jsonPath": "$.addresses[0].country.shortName" }, { "jsonPath": "$.addresses[0].thoroughfares[0].value" } ] }, "generalMatchingConfiguration": { "threshold": 0.85, "thresholdMaybe": 0.65, "matchingAttributes": [ { "name": "Business Partner Name", "jsonPath": "businessPartner.names[0].value", "cleaners": [ { "name": "LowerCaseNormalizer" }, { "name": "PunctuationsCleaner" }, { "name": "LegalFormCleaner", "parameters": [ { "name": "configProperty", "value": "$.addresses[0].country.shortName" } ] } ], "high": 0.8, "low": 0.3, "comparator": { "name": "QGramComparator", "parameters": [ { "name": "tokenizer", "value": "BASIC" }, { "name": "formula", "value": "DICE" }, { "name": "q", "value": "3" } ] } }, { "name": "Business Partner Country", "jsonPath": "businessPartner.addresses[0].country.shortName", "cleaners": [ { "name": "LowerCaseNormalizer" } ], "high": 0.5, "low": 0, "comparator": { "name": "ExactComparator" } }, { "name": "Business Partner City", "jsonPath": "businessPartner.addresses[0].localities[0].value", "cleaners": [ { "name": "LowerCaseNormalizer" }, { "name": "PunctuationsCleaner" } ], "high": 0.6, "low": 0.2, "comparator": { "name": "Levenshtein" } }, { "name": "Business Partner Street", "jsonPath": "businessPartner.addresses[0].thoroughfares[0].value", "cleaners": [ { "name": "LowerCaseNormalizer" }, { "name": "PunctuationsCleaner" } ], "high": 0.7, "low": 0.3, "comparator": { "name": "Levenshtein" } }, { "name": "Business Partner Postal Code", "jsonPath": "businessPartner.addresses[0].postCodes[0].value", "cleaners": [ { "name": "DigitsOnlyCleaner" } ], "high": 0.6, "low": 0.3, "comparator": { "name": "ExactComparator" } } ] }, "scopedMatchingConfiguration": {} } } ``` ## Step 1: Log into CDQ Cloud Apps Check your CDQ apps account. 1. Log into the CDQ [Cloud Apps](https://apps.cdq.com) ![](/assets/auth1.69ceac19aff9aec9330cc1beda18e5bf60d84699d415ea6b4be08c7ac393b369.6fde6558.png) *No account?* 1. If your organization is already a CDQ customer, ask your internal point of contact to create a CDQ dedicated account or [request account](https://support.cdq.com). Account detail will be sent by email. 2. If your organization is not yet a CDQ customer, please [contact us](https://www.cdq.com/about/contact) to get started. ## Step 2: Set up a Deduplication Configuration In order to create the default configuration for the Duplicate Guard Monitor, do the following in the Duplicate Guard Configurator application: 1. Navigate to the **Duplicate Guard Configurator** app. ![Figure 1. Duplicate Guard Configurator](/assets/bpdm-img001.b2a04dc4acdd9c689c014c8b26af1d043420bbbc5cc75d40a65b3e303ccd09f4.feb4434b.png) 1. Click the **Create New Configuration** button. 2. Provide a **Configuration Name** and click **Create**. ![Figure 2. Setting Duplicate Guard Monitor](/assets/bpdm-img002.e8348ff4bedda70a323d24036b80689bd10fc7bf32db3b15692d00feb43cd819.feb4434b.png) Creation of default configuration The default configuration is created. Browse and adjust if necessary (for this example, leave it as is). ## Step 3: Create a Duplicate Monitor Duplicate Monitor allows for asynchronous processing of data stored in a data mirror (that contains data sources - previously known as storage) in order to find duplicated business partners. The Duplicate Monitor can scan multiple data sources in order to detect duplicates. However, each data source can have only one dedicated Duplicate Monitor. **Required attributes:** * Duplicate Matching Configuration ([see above](#step-2-set-up-a-deduplication-configuration)) * Selection of data sources within one data mirror ### Data Source for Tests To simplify understanding, a sample data set is provided with 100 Business Partners: * 40 are different (according to the default configuration) * 60 are duplicates The basis for the set was the CDQ sample data file, which contains 10 different Business Partners. We extended it by adding 9 additional entries for each existing Business Partner. As a result, each original Business Partner has the following set of relationships with those additional 9: * Three `NO_MATCH` results (2nd, 6th, and 7th after the original Business Partner) * Three `MAYBE_MATCH` links (1st, 8th, and 9th) * Three `MATCH` links (3rd, 4th, and 5th) Worth noting Such results will be present only when using the default configuration. The applied matching configuration has always impact on the results. With different settings completely different results will be obtained. ### Set up a duplicate monitor 1. Navigate to the **Data Mirror Management** app. 2. Use the **Create Data Source** button to create a new data source: * Name it **Example Deduplication Data**. * Set the **Default mapping**. 3. Upload the example data using the **Import Data** button. Click the [`SampleDataForDeduplication.xlsx`](/assets/sampledatafordeduplication.dc1299b149c74440eb404e09940ff4946eca753d0a68761658ff61e99bb5c4db.6ca150ba.xlsx) to download the example data set. 1. Navigate to the **Data Clinic** app. 2. Add a new Data Monitor using the **Add New Data Monitor** button: * Set the data source to **Example Deduplication Data**. * Choose monitor type: **Duplicate**. * Choose configuration: **Example Matching Configuration**. * Click **Create New Data Monitor**. ![Figure 2. Setting Duplicate Guard Monitor](/assets/bpdm-img003.d3d1cdd53079c15c096e43d37e98513e2fc726aa028125c141ea96f8a745de60.feb4434b.png) Monitor results The example data set contains 60 duplicates: 30 `MATCH` links and 30 `MAYBE_MATCH` links. Each business partner contains both (3 and 3, respectively). ## Step 4: Generate a Deduplication Report In order to analyze Duplicate Guard Monitor results we need to generate an appropriate report. The way of generating it follows the usual way of report generation in CDQ apps. Duplicate Guard reports contain data columns that were provided in duplicate guard configuration. However, we also allow for having additional custom columns in the resulting report (for analysis purposes). A user can add up to 10 additional columns by providing a name of it as well as path to a data field in the original data. To generate a report: 1. In the **Data Clinic** application, click the **Reports** tab. 2. Click **Generate New Report**, then do the following: * Provide the report title (for example, **Example Report**). * Choose report type: **Duplicate Matching Report**. * Choose the appropriate data monitor: **Example Deduplication Data** (only `Duplicate` monitors are listed). * Select file format: **Excel**. * Leave other options as they are. * Click **Generate**. ![Figure 3. Generating Duplicate Matching Report](/assets/bpdm-img004.9bf8ba489df133514b12e779788ce95fea549dd7ac422b7ccf10c661e9418bd5.feb4434b.png) Report availability After a couple of minutes (it depends on the size of Duplicate Guard Monitor results) a report will available for download. ### Duplicate Guard Report Format Duplicate Guard report structure strictly relays on the configuration which was used by the Monitor itself. It contains all data fields indicated in the configuration. We provide fields like the following ones (based on the default Duplicate Guard configuration): * `Data Source ID` - indicates a data source from which a record comes from * `External ID` - external identifier of a record * `Matching Status` - the status of the match between the pattern record and a duplicate * `Linkage Status` - the status of the link which connects the pattern record with a duplicate * `Overall Score` - the similarity score between the pattern record and a duplicate * `Data Monitor Id` - the identifier of a Duplicate Guard Monitor * The list of matching attributes with their original values and similarity scores * In case of selecting the feature `SHOW_MATCHING_EXPLANATION` a report will contain the Matching Explanation column in which we provide the explanation of score calculation. The feature is set `OFF` by default. Worth noting Each group of Pattern+Duplicates is separated by a blank (empty) row. Moreover, patterns have light-green background while duplicates have white background. ## Your opinion matters! We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!