Uploading CSV/XLS with Request Upload

Overview

This step-by-step instruction guides the user through the data upload process using Upload Request. Uploaded data can be used with Use Case Enablers.

Scenario

In this instruction user will learn how to:

  1. Create a Data Storage,
  2. Request a file upload
  3. Upload desired data,
  4. Read uploaded data,
  5. Delete unused data storage.

Step 1. Create a storage

attention
Skip this step if a Data Storage was created before. You can use it multiple times for your files. Get the id of your Data Storage and go directly to the Step 2

To upload desired files storage is needed! Before upload:

  1. Use the Create Storage endpoint to send the below request
  2. Select the Create a storage example
  3. Replace the dataStorageExample and the dataSourceExample with new names
application/json
{
  • "name": "dataStorageExample",
  • "dataMatchingDefinitionId": "string",
  • "dataSources": [
    • {
      • "dataSourceName": "dataSourceExample",
      • "dataMapperDefinitionId": "61a5d4dc773d1012a3dfa5aa"
      }
    ],
  • "featuresOn": [
    • "LOOKUP",
    • "DATA_MIRROR"
    ],
  • "sharedWithOrganization": false,
  • "labels": [
    • "string"
    ]
}
  1. Check the above example in the Try It Console now:
Loading...

Important information:

  • Your storage is created if the status: CREATED_SUCCESSFULLY is displayed in the response body.
  • The id refers to the unique identifier of a newly created storage (for example id:3a10645d8be23f53a20b30bfa936e63d).
  • The storage is empty by default.
Congratulations!
The Data Storage is successfully created. Use storage id in the next step

Step 2. Request the file upload

Get the upload link

To start the import job of a csv or xlsx file:

  1. Use the Request file upload endpoint to send the below request
  2. Change the name of the imported file if needed
application/json
{
  • "url": "file.xlsx"
}
  1. Check the above example in the Try It Console now:
Loading...

In the response you will find 3 important parameters:

  • The uploadLink with the link to be used in the next step
  • The uploadMethod : with the method to be used during the upload
  • The url with the link with your uploaded file
Copy
Copied
{
  "uploadLink": "https://cdl-runtime-data.s3.eu-central-1.amazonaws.com/customer-uploads/cdq/53e219a1-3a79-4e67-b88c-cfcdddf57eab%2540generated.cdq.com/file.xlsx?x-amz-meta-created_at=2022-03-18T10%3A09%3A34.054784152Z&x-amz-meta-username=53e219a1-3a79-4e67-b88c-cfcdddf57eab&x-amz-meta-organization=cdq&X-Amz-Security-Token=FwoGZXIvYXdzEFMaDJYA5%2BuN%2FcqOVq%2Fa6yK1AUGjuYNn55Up9v0mTV50eOE8GVQoKktkwdo9B88Xg3Q%2Bl9JnIysGuQI3ne0RkvirEy9M0G9tKZgPKrxFLuSHOsrwaa8uw2enwK7Y%2F60v7kNN5uU%2BsVW4SuQ2MmcvZ%2BFRUtGmghEl7sySKKq0YPv%2F1jiafkCaT%2BomRkvep3AS%2BV7Uf9alDCUDwCgNdwDxJ0NWxH%2BTxGf5wyfnQ7FxkUiVmKU4ioRKuI%2Bxph9UUzRx0YE88AhXLmcorLHRkQYyLbXiNxFzcz0hZeP%2Fx5nOQ0E2aoWGoZhCjTZ82HZ%2B7ctgsuYyPHWq%2BH5ZDBBdzQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20220318T100934Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIARBIBNLCAPF4MTEU7%2F20220318%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=6a9b92b231a24352a4b05cb056f97fe8e46dabc35bf980e7ee06eac6792710c9",
  "uploadMethod": "PUT",
  "url": "customer-uploads/cdq/53e219a1-3a79-4e67-b88c-cfcdddf57eab%40generated.cdq.com/file.xlsx"
}
attention
Use uploadLink and uploadMethod during the upload. Use url in the next step.
warning

The validation of the link is temporary. Upload the file as fast as possible.

Upload the file

Uploading the file must be performed by an external client like Insomnia.

warning

Remember to authenticate. Check how to get an API-Key.

To upload the file:

  1. Create a new request
  2. Set the uploadMethod and paste the uploadLink
UploadIns1
Defining the request parameters
  1. Change the body of the request to Binary File
  2. Select the file
UploadIns2 UploadIns3
Setting the request bodySelecting the file
  1. Send the request
info

No request body expected. Go to the next step

Step 3. Upload data to the storage

Use received url link

To start the import job of the uploaded file:

  1. Use the Start Import Job endpoint to send the below request
  2. Paste the existing storage id
  3. Paste the url from step 2.
  1. Check the above example in the Try It Console now:
Loading...

In the response you will find 3 important parameters:

  • The id with the value representing YOUR JOB ID - the unique number of every upload job.
  • The progress : 0 representing the progress of the job.
  • The status : SCHEDULED - the job is in the job queue.
attention
Note down received YOUR JOB ID for polling the job status.

Pool for the completion status

To check the import job status:

  1. Use the Poll Import Job endpoint to send the below request
  2. Use the YOUR JOB ID from the previous step and replace the jobId
  1. Check the above example in the Try It Console now:
Loading...
  • If the job is still running the status with the progress will be displayed:
Copy
Copied
{
  "id" : "YOUR JOB ID",
  "progress" : "71",
  "status" : "RUNNING"
}
info

The import job runtime is dependent on the size of the imported file. Poll the status regularly to find out when the import is done.

  • When the job is done the FINISHED status will be displayed
Copy
Copied
{
  "id" : "YOUR JOB ID",
  "progress" : "100",
  "status" : "FINISHED"
}
success

The data is fully imported to your storage now. The uploaded data can be processed now by the wide range of CDQ Solutions.

Step 4. Read data

To read uploaded data:

  1. Use the Read Business Partners endpoint to send the below request
  2. Use the existing storage id
  1. Check the above example in the Try It Console now:
Loading...
In the response the Business Partners data will be displayed in the value object:
Copy
Copied
{
    "values" : [
      {
        `businessPartnerData1`
      },
      {
        `businessPartnerData2`
      }
    ]
}
attention
The businessPartnerData1 and businessPartnerData2 represent the sets of parameters for a particular Business Partner. It is used for simplification.

Step 5. Delete storage

GOOD PRACTISES

Delete the storage after you no longer need it.

  1. Use the Delete Storage endpoint to send the below request
  2. Use the existing storage id no longer in use and replace the storageId in the request
  1. Check the above example in the Try It Console now:
Loading...

Your opinion matters!

We are constantly working on providing an outstanding user experience with our products. Please share your opinion about this tutorial!

Mail our developer-portal team: developer-portal@cdq.com