Data Mirror Integration

Everything CDQ does happens against the Data Mirror: the copy of your business partner data that CDQ holds. An integration therefore has exactly two jobs. Get records in, and get results back out.

This article covers both directions as a developer meets them: which route to take for writing, how to read incrementally so the integration still works when the dataset is large, and the handful of operational details that account for most of the time lost during a first integration.

In this article, you will understand:

• The three routes for writing records into the mirror, and which one suits which cadence
• The two payload shapes, raw record and CDQ data model, and what each one needs in place first
• How records are identified, and why that makes every write repeatable
• How to read results back out incrementally, with paging and the fields outside the standard model
• The failure modes worth knowing before you hit them

Available in:

• Data Exchange API - the programmatic interface to the mirror

An integration needs four things, all established when the workspace is set up:

Base URL: https://api.cdq.com/data-exchange/rest

A record is identified by external ID + data source, not by the id CDQ assigns it. Sending the same external ID into the same data source twice updates the existing record rather than creating a second one.

That is what makes a write safe to retry, and it is why a failed or partially completed load can simply be sent again.

Three routes, differing in latency, volume and integration effort. They are not exclusive, and most integrations use two of them.

One endpoint, two shapes. This is the choice most integrations get wrong first, because the endpoint accepts both and does something quite different with each.

Neither is more correct. The raw record keeps the translation in one place that a non-developer can change, and it preserves what your system actually held. The data model shape gives you exact control and needs no setup. A record can even carry both, in which case the model attributes are used and the raw columns are kept alongside them.

The record field holds a JSON string of your own columns, with your own column names:

curl -X PUT \
  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: {apiKey}' \
  -d '{
    "dataSource": "{dataSourceId}",
    "featuresOn": ["UPSERT_BY_EXTERNAL_ID"],
    "businessPartners": [
      {
        "externalId": "0000012345",
        "record": "{\"Customer number\":\"0000012345\", \"Name\":\"Acme Corporation\", \"Country\":\"DE\", \"City\":\"Berlin\", \"Street\":\"Friedrichstrasse 123\", \"Postal code\":\"10115\"}"
      }
    ]
  }'

Nothing here is in the CDQ data model. Customer number and Postal code are your names for those columns, and the data source's mapping is what turns them into a business partner. If no mapping is assigned, there is nothing to translate the record with.

This is also the shape to use for a partial update. Sending a record with only the external ID and the one field that changed updates that field and leaves the rest alone.

curl -X PUT \
  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: {apiKey}' \
  -d '{
    "dataSource": "{dataSourceId}",
    "featuresOn": ["UPSERT_BY_EXTERNAL_ID"],
    "businessPartners": [
      {
        "externalId": "0000012345",
        "names": [
          { "value": "Acme Corporation",
            "type": { "technicalKey": "LOCAL" } }
        ],
        "addresses": [
          { "country": { "shortName": "DE" },
            "localities": [{ "value": "Berlin", "type": { "technicalKey": "CITY" } }],
            "postCodes": [{ "value": "10115", "type": { "technicalKey": "REGULAR" } }],
            "thoroughfares": [{ "value": "Friedrichstrasse 123", "type": { "technicalKey": "STREET" } }] }
        ],
        "identifiers": [
          { "value": "DE123456789",
            "type": { "technicalKey": "EU_VAT_ID_DE" } }
        ]
      }
    ]
  }'

No mapping is involved, and the stored business partner looks exactly like the request.

Where dataSource goes. At the top level it applies to every record in the request, which is the usual case. It can instead be set per business partner, so one request can write to several data sources. If the storage has exactly one data source, it can be omitted. Supplying a data source that does not exist, or omitting it when the storage has more than one, is an error rather than a default.

Batch size. Up to 500 business partners per request.

The raw record shape and the import job both depend on a mapping, so it is worth knowing where that lives and what happens when it changes.

A mapping describes which of your fields becomes which CDQ attribute. It is created once and then assigned to a data source, which is why two source systems with different layouts need two data sources rather than one.

Changing a mapping does not retranslate what is already stored. Records that arrived under the old mapping keep the attributes it produced. To bring them in line, start a transformation job against the data source:

curl -X POST \
  'https://api.cdq.com/data-exchange/rest/jobs/transformationjobs' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: {apiKey}' \
  -d '{
    "storageId": "{storageId}",
    "dataSourceId": "{dataSourceId}",
    "featuresOn": ["UPSERT_CHANGES_ONLY"],
    "countries": [{ "shortName": "CH" }]
  }'

Narrowing by country is worth doing when a mapping change only affected some of them, because the job then has less to re-process.

A file carries your columns rather than the data model, so like the raw record shape it is translated through the data source's mapping on arrival. The difference is only that the data travels as a file instead of a request body.

The flow is four steps, because the file does not travel through the API itself:

1. POST /files/upload          ->  returns a pre-signed upload link and a file reference
2. PUT  {uploadLink}           ->  the raw file bytes, sent to the link, without your API key
3. POST /jobs/importjobs       ->  starts the job, referencing the file and the target data source
4. GET  /jobs/importjobs/{id}  ->  poll until the job reaches a terminal state

Two things about this flow surprise people:

• Step 2 does not go to CDQ. The pre-signed link points at object storage. Sending your X-API-KEY to it is both unnecessary and undesirable.
• Step 3 is multipart/form-data, not JSON. The fields are sent as form fields. Setting a JSON content type results in the fields being rejected or ignored.

Writing into a data source that mirrors a system requires the job to upsert by external ID rather than insert, which means telling the job which column holds that ID.

For the step-by-step of each route, see How to synchronize data with the Data Mirror.

This is the half that is easy to under-plan. Writing data in is a one-off piece of work; reading results back is something your integration does for as long as the mirror exists.

Results are read from the data source they were written to:

curl -X GET \
  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners?dataSource={dataSourceId}&limit=500' \
  -H 'X-API-KEY: {apiKey}'

Re-downloading the full dataset on every cycle works until the dataset is large, and then it stops working. The modifiedAfter parameter takes an ISO-8601 timestamp and returns only records modified since:

curl -X GET \
  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners?dataSource={dataSourceId}&modifiedAfter=2026-10-01T09:00:00Z&limit=500' \
  -H 'X-API-KEY: {apiKey}'

The sequence that makes this safe is to take the timestamp before you start reading, not after you finish:

1. Record the current time.
2. Download everything modified after the timestamp you stored on the previous run.
3. Store the timestamp from step 1 as the new watermark.

Results are paged. When the response reports more records than the page size, it carries a cursor for the next page, which you pass back as startAfter. Page size defaults to 500 and the maximum is 1,000.

The CDQ data model covers names, addresses, identifiers and the other standard attributes. An engagement often produces more than that - per-field before and after values, duplicate group assignments, review decisions - and those do not all have a home in the standard model.

They are preserved on the record anyway. Requesting the FETCH_RECORD feature returns a record field on each business partner containing the full stored row as JSON, including anything outside the standard attributes:

...&featuresOn=FETCH_RECORD

If your integration needs the detail behind a curation rather than just the curated values, this is where it is.

None of these are conceptually interesting. All of them cost time when they are discovered during an integration rather than before it.

An integration with the Data Mirror is two directions of travel. Records go in through the upsert API for a live connection, through an import job for bulk and file-shaped data, or through the apps for one-off loads.

The upsert endpoint takes either your raw columns in record, translated by the data source's mapping, or a record already in the CDQ data model, stored as sent. Whichever you choose, UPSERT_BY_EXTERNAL_ID is what makes the call an update rather than an insert, and a mapping change needs a transformation job before it reaches records that are already stored.

Results come back out of the data source they were written to. An integration that reads with modifiedAfter, takes its watermark before the read rather than after it, and pages through the result will keep working as the dataset grows. FETCH_RECORD is where the detail lives when the standard attributes are not enough.

• Data Mirror
• How to synchronize data with the Data Mirror
• Uploading CSV/XLS data
• Uploading CSV/XLS with Request Upload
• How to Define and Use Data Transformation Definitions
• Data Exchange API

We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!