Setup and Synchronize a Data Mirror

Overview

The sequence of activities for setting up a data mirror and continuously synchronization with the local source system.

info

This instruction is based on Setup and Synchronize a Data Mirror use case published on the Wiki.

Scenario

In this instruction you will learn how to:

Create a Data Source
Synchronize your data with the CDQ Data Model

Prerequisites

Data Storage

To create a Data Source a Storage or Data Mirror is needed! Create one before further step and use its storageId value as YOUR_STORAGE_ID in the next steps of this instruction.

attention

Every created Storage or Data Mirror has a default Data Source.

Step 1: Data Mirror Configuration

To configure a Data Mirror a Data Source and a Data Mapper Definition must be set.

To create Data Source:

Use the Create Data Source endpoint to send the below request.

Payload
curl

application/json

{"name": "dataSourceExample",
"dataMapperDefinitionId": "default",
"prefix": "Custom prefix"
}

Check the above example in the Try It Console now!
Set value of:
- dataSource to YOUR_DATASOURCE_ID
- dataMapperDefinitionId to default
- storageId to YOUR_STORAGE_ID in Parameters section

Details of created Data Source are provided:

Copy

Copied

{
    "id": "YOUR_DATASOURCE_ID",
    "name": "YOUR_DATASOURCE_NAME",
    "dataMapperDefinitionId": "default"
}

Congratulations

Your Data Source is created!

Step 2: Synchronize Data

Data synchronization process is dependent to the size of the data to be updated:

If the update contains many Business Partners and the file size <= 10 MB read: File upload instruction.
If the update contains many Business Partners and the file size > 10 MB read: Request upload instruction.
If the update is for one Business Partner go to Single record update.

First time data upload

To start the import job of a csv or xlsx file:

Use the Start Import Job endpoint to send the below request.

curl

Check the above example in the Try It Console now!
Set value of:
- dataMapperDefinitionId to default
- dataSource to YOUR_DATASOURCE_NAME
- featuresOn to UPSERT_BY_EXTERNAL_ID
- storageId to YOUR_STORAGE_ID
Point the file path.

In the response you will find 3 important parameters:

The id with the value representing YOUR_JOB_ID - the unique number of every upload job.
The progress: 0 represents the progress of the job.
The status: SCHEDULED - the job is in the job queue.

attention

Note down received YOUR_JOB_ID for polling the job status. Check how to poll the job status here.

Mass data update

To perform mass data update follow the same steps from the First time data upload subchapter.

warning

The key to successfully uploading is UPSERT_BY_EXTERNAL_ID feature. The update file with correctly mapped External Id column after upload will result in data update for matching externalIds and new record creation for new externalIds.

Single record update

attention

Before the update of the existing records the Business Partner's externalId is needed. Be sure that the UPSERT_BY_EXTERNAL_ID feature is active.

To update existing records:

Use the Upsert Business Partners endpoint to send the below request.

info

Check the below example based on default mapping.

Payload
curl

application/json

{"dataSource": "YOUR_DATASOURCE_ID",
"featuresOn": ["UPSERT_BY_EXTERNAL_ID"
],
"businessPartners": [{"externalId": "YOUR_EXTERNAL_ID",
"record": "{\"Customer number\":\"YOUR_EXTERNAL_ID\", \"Name\":\"COMPANY_NAME\", \"Country\":\"COUNTRY_CODE\", \"City\":\"CITY\", \"Street\":\"STREET\", \"Postal code\":\"POSTAL_CODE\"}"
}
]
}

Check the above example in the Try It Console now!
Select example Single record update.
Set the YOUR_DATASOURCE_ID.
Replace below values with the Business Partner data to be updated:
- BP_EXTERNAL_ID
- COMPANY_NAME
- COUNTRY_CODE
- CITY
- STREET
- POSTAL_CODE
Set the YOUR_STORAGE_ID in Parameters section.

Results

3 types of results can be expected:

New record created.
Existing record updated.
Something went wrong:
- The response includes a message about the root of the issue.

New record createdExisting record updatedSomething went wrong

Copy

Copied

{
"numberOfAccepted": 1,
"numberOfInserts": 1,
"numberOfUpdates": 0,
"numberOfFailed": 0
}

Copy

Copied

{
"numberOfAccepted": 1,
"numberOfInserts": 0,
"numberOfUpdates": 1,
"numberOfFailed": 0
}

Copy

Copied

{
"numberOfInserts": 0,
"numberOfUpdates": 0,
"numberOfFailed": 1,
"failures":
[

    {
        "externalId": "0053",
        "message": "Incorrect record quote escaping"
    }

]
}

Congratulations

Now you know how to update a single record.

Additional activities

Read updated Business Partner

To start the update of the existing records:

Use the Read Business Partners endpoint to send the below request.

curl

Check the above example in the Try It Console now!
Set values of:
- storageId to YOUR_STORAGE_ID
- dataSource to YOUR_DATASOURCE_ID
- externalId to BUSINESSPARTNER_EXTERNALID

Clear a complete data source

To clear a data source:

Use the Clear Data Source endpoint to send the below request.

curl

danger

Remember that your ALL data within cleared Data Source will be deleted.

Check the above example in the Try It Console now!
Set values of:
- storageId to YOUR_STORAGE_ID
- dataSource to YOUR_DATASOURCE_ID

Delete a data source

To delete a data source:

Use the Delete Data Source endpoint to send the below request.

curl

danger

Remember that your Data Source and its data will be deleted.

Check the above example in the Try It Console now!
Set values of:
- storageId to YOUR_STORAGE_ID
- dataSource to YOUR_DATASOURCE_ID

Delete a Business Partner Record

To delete a data source:

Use the Delete Business Partners endpoint to send the below request.

Payload
curl

application/json

{"dataSource": "YOUR_DATASOURCE_ID",
"businessPartners": [{"externalId": "BP_EXTERNAL_ID"
}
],
"featuresOn": ["DELETE_BY_EXTERNAL_ID"
]
}

danger

Remember that your Business Partner record will be deleted permanently.

Check the above example in the Try It Console now!
Select Delete Business Partner Based on External Id example.
Set values of:
- externalId to BP_EXTERNAL_ID
- dataSource to YOUR_DATASOURCE_ID
Set the YOUR_STORAGE_ID in Parameters section.

Your opinion matters!

We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!

Mail our developer-portal team: developer-portal@cdq.com