{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"metadata":{"markdoc":{"tagList":["admonition","partial"]},"type":"markdown"},"seo":{"title":"Data Mirror Integration","description":"How to write business partners into a Data Mirror, read curated results back out incrementally, and the operational details that cause most integration failures."},"dynamicMarkdocComponents":[],"compilationErrors":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"data-mirror-integration","__idx":0},"children":["Data Mirror Integration"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"overview","__idx":1},"children":["Overview"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Everything CDQ does happens against the ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Data Mirror"]},": the copy of your business partner data that CDQ holds. An"," ","integration therefore has exactly two jobs. Get records in, and get results back out."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This article covers both directions as a developer meets them: which route to take for writing, how to read"," ","incrementally so the integration still works when the dataset is large, and the handful of operational details that"," ","account for most of the time lost during a first integration."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["In this article, you will understand:"]}]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["The three routes for writing records into the mirror, and which one suits which cadence"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["The two payload shapes, raw record and CDQ data model, and what each one needs in place first"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["How records are identified, and why that makes every write repeatable"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["How to read results back out incrementally, with paging and the fields outside the standard model"]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["The failure modes worth knowing before you hit them"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Available in:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/api/data-exchange/quickstart"},"children":["Data Exchange API"]}," - the programmatic interface to the mirror"]}]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This article assumes the concepts. For what a Data Mirror is, how storages and data sources relate, and how to"," ","decide your data source design, start with ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/articles/expertise/data-mirror"},"children":["Data Mirror"]},"."]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"before-you-start","__idx":2},"children":["Before you start"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["An integration needs four things, all established when the workspace is set up:"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"What"},"children":["What"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Where it comes from"},"children":["Where it comes from"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["API key"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Created in the workspace. Sent as the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["X-API-KEY"]}," header on every call."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Storage ID"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The storage you are integrating with. Appears in the path of most calls."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Data source ID"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The data source you write to and filter by when reading."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["A mapping"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Needed whenever you send your own columns, as a raw record or in a file. Not needed when you send records already in the CDQ data model."]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Base URL:"]}," ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["https://api.cdq.com/data-exchange/rest"]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"what-identifies-a-record","__idx":3},"children":["What identifies a record"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A record is identified by ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["external ID + data source"]},", not by the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["id"]}," CDQ assigns it. Sending the same external"," ","ID into the same data source twice updates the existing record rather than creating a second one."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["That is what makes a write safe to retry, and it is why a failed or partially completed load can simply be sent"," ","again."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"writing-records-into-the-mirror","__idx":4},"children":["Writing records into the mirror"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Three routes, differing in latency, volume and integration effort. They are not exclusive, and most integrations use"," ","two of them."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Route"},"children":["Route"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Shape"},"children":["Shape"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Suits"},"children":["Suits"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Upsert API"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["PUT /v4/storages/{storageId}/businesspartners"]},", carrying either your raw record or a record already in the CDQ data model."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["A live or near-live connection, where your system emits an event on create and change."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Import job"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Upload a CSV or Excel file, then start a job that maps its columns into the data model."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Scheduled bulk loads, initial loads, and any case where data already leaves your system as a file."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Upload in Cloud Apps"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The same import job, started from the user interface instead of the API."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["One-off loads, and getting started before an automated route exists."]}]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"the-upsert-route","__idx":5},"children":["The upsert route"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["One endpoint, two shapes. This is the choice most integrations get wrong first, because the endpoint accepts both"," ","and does something quite different with each."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Shape"},"children":["Shape"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"You send"},"children":["You send"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Who owns the mapping"},"children":["Who owns the mapping"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Prerequisite"},"children":["Prerequisite"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Raw record"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Your own columns, exactly as your system names them, as a JSON string in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["record"]},"."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["CDQ. The data source's mapping translates your columns into the data model."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["A mapping must be assigned to the data source."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["CDQ data model"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["names"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["addresses"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["identifiers"]}," and the other model attributes."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["You. The record is stored as sent."]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["None."]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Neither is more correct. The raw record keeps the translation in one place that a non-developer can change, and it"," ","preserves what your system actually held. The data model shape gives you exact control and needs no setup. A record"," ","can even carry both, in which case the model attributes are used and the raw columns are kept alongside them."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":4,"id":"sending-a-raw-record","__idx":6},"children":["Sending a raw record"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["record"]}," field holds a JSON ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["string"]}," of your own columns, with your own column names:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"curl -X PUT \\\n  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners' \\\n  -H 'Content-Type: application/json' \\\n  -H 'X-API-KEY: {apiKey}' \\\n  -d '{\n    \"dataSource\": \"{dataSourceId}\",\n    \"featuresOn\": [\"UPSERT_BY_EXTERNAL_ID\"],\n    \"businessPartners\": [\n      {\n        \"externalId\": \"0000012345\",\n        \"record\": \"{\\\"Customer number\\\":\\\"0000012345\\\", \\\"Name\\\":\\\"Acme Corporation\\\", \\\"Country\\\":\\\"DE\\\", \\\"City\\\":\\\"Berlin\\\", \\\"Street\\\":\\\"Friedrichstrasse 123\\\", \\\"Postal code\\\":\\\"10115\\\"}\"\n      }\n    ]\n  }'\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Nothing here is in the CDQ data model. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Customer number"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Postal code"]}," are your names for those columns, and the"," ","data source's mapping is what turns them into a business partner. If no mapping is assigned, there is nothing to"," ","translate the record with."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This is also the shape to use for a partial update. Sending a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["record"]}," with only the external ID and the one field"," ","that changed updates that field and leaves the rest alone."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":4,"id":"sending-a-record-in-the-cdq-data-model","__idx":7},"children":["Sending a record in the CDQ data model"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"curl -X PUT \\\n  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners' \\\n  -H 'Content-Type: application/json' \\\n  -H 'X-API-KEY: {apiKey}' \\\n  -d '{\n    \"dataSource\": \"{dataSourceId}\",\n    \"featuresOn\": [\"UPSERT_BY_EXTERNAL_ID\"],\n    \"businessPartners\": [\n      {\n        \"externalId\": \"0000012345\",\n        \"names\": [\n          { \"value\": \"Acme Corporation\",\n            \"type\": { \"technicalKey\": \"LOCAL\" } }\n        ],\n        \"addresses\": [\n          { \"country\": { \"shortName\": \"DE\" },\n            \"localities\": [{ \"value\": \"Berlin\", \"type\": { \"technicalKey\": \"CITY\" } }],\n            \"postCodes\": [{ \"value\": \"10115\", \"type\": { \"technicalKey\": \"REGULAR\" } }],\n            \"thoroughfares\": [{ \"value\": \"Friedrichstrasse 123\", \"type\": { \"technicalKey\": \"STREET\" } }] }\n        ],\n        \"identifiers\": [\n          { \"value\": \"DE123456789\",\n            \"type\": { \"technicalKey\": \"EU_VAT_ID_DE\" } }\n        ]\n      }\n    ]\n  }'\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["No mapping is involved, and the stored business partner looks exactly like the request."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":4,"id":"what-both-shapes-need","__idx":8},"children":["What both shapes need"]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"warning"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["featuresOn: [\"UPSERT_BY_EXTERNAL_ID\"]"]}," is what makes the call an update rather than an insert. Without it, sending"," ","the same external ID a second time can create a second record instead of updating the first. It belongs in every"," ","upsert an integration makes."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Where ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSource"]}," goes."]}," At the top level it applies to every record in the request, which is the usual case."," ","It can instead be set per business partner, so one request can write to several data sources. If the storage has"," ","exactly one data source, it can be omitted. Supplying a data source that does not exist, or omitting it when the"," ","storage has more than one, is an error rather than a default."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Batch size."]}," Up to 500 business partners per request."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"mapping-and-transformation","__idx":9},"children":["Mapping and transformation"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The raw record shape and the import job both depend on a mapping, so it is worth knowing where that lives and what"," ","happens when it changes."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["mapping"]}," describes which of your fields becomes which CDQ attribute. It is created once and then ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["assigned to"," ","a data source"]},", which is why two source systems with different layouts need two data sources rather than one."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Step"},"children":["Step"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Where"},"children":["Where"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Create the mapping"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["The Data Mirror Management app, starting from the downloadable example, or the Data Mapper Definition endpoints."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Assign it to a data source"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Set when the data source is created, and changeable afterwards."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Send data"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Raw records and import jobs are translated through it on arrival."]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Changing a mapping does not retranslate what is already stored."]}," Records that arrived under the old mapping keep"," ","the attributes it produced. To bring them in line, start a transformation job against the data source:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"curl -X POST \\\n  'https://api.cdq.com/data-exchange/rest/jobs/transformationjobs' \\\n  -H 'Content-Type: application/json' \\\n  -H 'X-API-KEY: {apiKey}' \\\n  -d '{\n    \"storageId\": \"{storageId}\",\n    \"dataSourceId\": \"{dataSourceId}\",\n    \"featuresOn\": [\"UPSERT_CHANGES_ONLY\"],\n    \"countries\": [{ \"shortName\": \"CH\" }]\n  }'\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Narrowing by country is worth doing when a mapping change only affected some of them, because the job then has less"," ","to re-process."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"info"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Changing the assigned mapping through the Data Mirror Management app starts this transformation automatically."," ","Changing it through the API does not, so an API-driven change needs the job started explicitly."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"the-import-job-route","__idx":10},"children":["The import job route"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A file carries your columns rather than the data model, so like the raw record shape it is translated through the"," ","data source's mapping on arrival. The difference is only that the data travels as a file instead of a request body."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The flow is four steps, because the file does not travel through the API itself:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"1. POST /files/upload          ->  returns a pre-signed upload link and a file reference\n2. PUT  {uploadLink}           ->  the raw file bytes, sent to the link, without your API key\n3. POST /jobs/importjobs       ->  starts the job, referencing the file and the target data source\n4. GET  /jobs/importjobs/{id}  ->  poll until the job reaches a terminal state\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Two things about this flow surprise people:"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Step 2 does not go to CDQ."]}," The pre-signed link points at object storage. Sending your ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["X-API-KEY"]}," to it is"," ","both unnecessary and undesirable."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Step 3 is ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["multipart/form-data"]},", not JSON."]}," The fields are sent as form fields. Setting a JSON content type"," ","results in the fields being rejected or ignored."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Writing into a data source that mirrors a system requires the job to upsert by external ID rather than insert, which"," ","means telling the job which column holds that ID."]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"warning"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["An import job is asynchronous, and reaching a terminal state is not the same as succeeding. Read the record counts"," ","on the final poll: a job can finish having failed a subset of its records, and a job that cannot find a column the"," ","mapping expects stops rather than importing partial records."]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["For the step-by-step of each route, see"," ",{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/instructions/dsp-how-to-sync-with-data-mirror"},"children":["How to synchronize data with the Data Mirror"]},"."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"reading-results-back-out","__idx":11},"children":["Reading results back out"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This is the half that is easy to under-plan. Writing data in is a one-off piece of work; reading results back is"," ","something your integration does for as long as the mirror exists."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Results are read from the data source they were written to:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"curl -X GET \\\n  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners?dataSource={dataSourceId}&limit=500' \\\n  -H 'X-API-KEY: {apiKey}'\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"reading-only-what-changed","__idx":12},"children":["Reading only what changed"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Re-downloading the full dataset on every cycle works until the dataset is large, and then it stops working. The"," ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["modifiedAfter"]}," parameter takes an ISO-8601 timestamp and returns only records modified since:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"curl -X GET \\\n  'https://api.cdq.com/data-exchange/rest/v4/storages/{storageId}/businesspartners?dataSource={dataSourceId}&modifiedAfter=2026-10-01T09:00:00Z&limit=500' \\\n  -H 'X-API-KEY: {apiKey}'\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The sequence that makes this safe is to ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["take the timestamp before you start reading, not after you finish"]},":"]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Record the current time."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Download everything modified after the timestamp you stored on the previous run."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Store the timestamp from step 1 as the new watermark."]}]},{"$$mdtype":"Tag","name":"Admonition","attributes":{"type":"warning"},"children":[{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Taking the timestamp after the download finishes loses any record that CDQ writes while the download is in progress."," ","On a large result set that window is minutes wide, and the records lost are silently absent rather than reported as"," ","an error."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"paging","__idx":13},"children":["Paging"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Results are paged. When the response reports more records than the page size, it carries a cursor for the next page,"," ","which you pass back as ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["startAfter"]},". Page size defaults to 500 and the maximum is 1,000."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"reading-fields-that-are-not-in-the-data-model","__idx":14},"children":["Reading fields that are not in the data model"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The CDQ data model covers names, addresses, identifiers and the other standard attributes. An engagement often"," ","produces more than that - per-field before and after values, duplicate group assignments, review decisions - and"," ","those do not all have a home in the standard model."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["They are preserved on the record anyway. Requesting the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["FETCH_RECORD"]}," feature returns a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["record"]}," field on each"," ","business partner containing the full stored row as JSON, including anything outside the standard attributes:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"...&featuresOn=FETCH_RECORD\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If your integration needs the detail behind a curation rather than just the curated values, this is where it is."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"other-filters-worth-knowing","__idx":15},"children":["Other filters worth knowing"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Parameter"},"children":["Parameter"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Effect"},"children":["Effect"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["dataSource"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Restricts the read to one data source. The usual way each consuming system collects only its own records."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["modifiedAfter"]}," / ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["modifiedBefore"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Bounds the read by modification time. The basis of incremental retrieval."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["countryCode"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Restricts to one country, for phased rollouts."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["externalId"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Fetches specific records by their key rather than scanning."]}]}]}]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"operational-details-that-cause-most-of-the-failures","__idx":16},"children":["Operational details that cause most of the failures"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["None of these are conceptually interesting. All of them cost time when they are discovered during an integration"," ","rather than before it."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Area"},"children":["Area"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"What to know"},"children":["What to know"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Excel files"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["An uploaded workbook should contain exactly one sheet of data. Extra sheets cause import problems."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["File names"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Stick to letters, digits, hyphens, underscores and dots. Square brackets and other special characters are rejected by the underlying storage."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Column headers"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Must match what the mapping expects, and must not contain dots. A missing column stops the job rather than importing a partial record."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Escaping"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Fields containing commas, semicolons, line breaks or double quotes have to be quoted."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Job status"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Poll until a terminal state and read the record counts. Handle both the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["CANCELED"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["CANCELLED"]}," spellings if you match on the string."]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Deletions"]}]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Removing a record from your system does not remove it from the mirror. Deletion is its own call."]}]}]}]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"summary","__idx":17},"children":["Summary"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["An integration with the Data Mirror is two directions of travel. Records go in through the upsert API for a live"," ","connection, through an import job for bulk and file-shaped data, or through the apps for one-off loads."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The upsert endpoint takes either your raw columns in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["record"]},", translated by the data source's mapping, or a record"," ","already in the CDQ data model, stored as sent. Whichever you choose, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["UPSERT_BY_EXTERNAL_ID"]}," is what makes the call"," ","an update rather than an insert, and a mapping change needs a transformation job before it reaches records that are"," ","already stored."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Results come back out of the data source they were written to. An integration that reads with ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["modifiedAfter"]},", takes"," ","its watermark before the read rather than after it, and pages through the result will keep working as the dataset"," ","grows. ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["FETCH_RECORD"]}," is where the detail lives when the standard attributes are not enough."]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"related-resources","__idx":18},"children":["Related Resources"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/articles/expertise/data-mirror"},"children":["Data Mirror"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/instructions/dsp-how-to-sync-with-data-mirror"},"children":["How to synchronize data with the Data Mirror"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/instructions/uploading-cvs-xls-data"},"children":["Uploading CSV/XLS data"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/instructions/url-upload"},"children":["Uploading CSV/XLS with Request Upload"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/_public/instructions/define-data-transformation-definitions"},"children":["How to Define and Use Data Transformation Definitions"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"MarkdownLink","attributes":{"href":"/documentation/api/data-exchange/quickstart"},"children":["Data Exchange API"]}]}]},{"$$mdtype":"Tag","name":"hr","attributes":{},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"your-opinion-matters","__idx":19},"children":["Your opinion matters!"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!"]}]},"headings":[{"value":"Data Mirror Integration","id":"data-mirror-integration","depth":1},{"value":"Overview","id":"overview","depth":2},{"value":"Before you start","id":"before-you-start","depth":2},{"value":"What identifies a record","id":"what-identifies-a-record","depth":3},{"value":"Writing records into the mirror","id":"writing-records-into-the-mirror","depth":2},{"value":"The upsert route","id":"the-upsert-route","depth":3},{"value":"Sending a raw record","id":"sending-a-raw-record","depth":4},{"value":"Sending a record in the CDQ data model","id":"sending-a-record-in-the-cdq-data-model","depth":4},{"value":"What both shapes need","id":"what-both-shapes-need","depth":4},{"value":"Mapping and transformation","id":"mapping-and-transformation","depth":3},{"value":"The import job route","id":"the-import-job-route","depth":3},{"value":"Reading results back out","id":"reading-results-back-out","depth":2},{"value":"Reading only what changed","id":"reading-only-what-changed","depth":3},{"value":"Paging","id":"paging","depth":3},{"value":"Reading fields that are not in the data model","id":"reading-fields-that-are-not-in-the-data-model","depth":3},{"value":"Other filters worth knowing","id":"other-filters-worth-knowing","depth":3},{"value":"Operational details that cause most of the failures","id":"operational-details-that-cause-most-of-the-failures","depth":2},{"value":"Summary","id":"summary","depth":2},{"value":"Related Resources","id":"related-resources","depth":2},{"value":"Your opinion matters!","id":"your-opinion-matters","depth":2}],"frontmatter":{"seo":{"title":"Data Mirror Integration","description":"How to write business partners into a Data Mirror, read curated results back out incrementally, and the operational details that cause most integration failures."},"toc":{"enable":true},"cdqContentMetadata":{"product":"CDQ Platform","type":"article","subtype":"integration","lastUpdate":"2026-10-02T00:00:00.000Z","author":"Simon Schlosser"}},"lastModified":"2026-10-21T09:50:10.000Z","pagePropGetterError":{"message":"","name":""}},"slug":"/documentation/_public/product-manuals/data-mirror-integration","userData":{"isAuthenticated":false,"teams":["anonymous"]},"isPublic":true}