How to perform bulk upload from Thin Edge to Cumulocity?

Forum
1

Product/components used and version/fix level are you on:

Detailed explanation of the problem:

Hi everyone,

I’m currently working on a task where I need to upload large amounts of data from multiple devices connected to a Thin Edge platform to Cumulocity. I was wondering if anyone could provide some guidance on how to perform bulk uploads from Thin Edge to Cumulocity using the Cumulocity REST API or MQTT protocol.

Specifically, I’m looking for information on how to:

  • Create batch requests in the Cumulocity REST API to perform operations on multiple objects at once
  • Use MQTT publish topics to efficiently upload large amounts of data in batches

Any tips, advice, or sample code would be greatly appreciated. Thank you in advance for your help!

Error messages / full error message screenshot / log fileL

Is your question related to the free trial, or to a production (customer) instance?

Have you installed all the latest fixes for the products and systems you are using?

2

To answer your questions specifically:

  • Create batch requests in the Cumulocity REST API to perform operations on multiple objects at once

The REST API only supports batching on some APIs like Measurements Cumulocity IoT - OpenAPI Specification. For other like inventory, events etc. you need to do single requests

  • Use MQTT publish topics to efficiently upload large amounts of data in batches

Thin-edge has multiple ways to integrate data. You can either use thin-edge format https://thin-edge.github.io/thin-edge.io/html/architecture/thin-edge-json.html or using smartREST directly e.g. tedge mqtt pub c8y/s/us 211,20.

For the second you can send multiple different static SmartREST Messages within one MQTT Message (just add multiple lines in your payload) But you have to be aware of the 16 KB payload size.

REST doesn’t have that limit so for large amount of data I would suggest REST for data integration.

3

Thanks for the answer

is there any guide to follow to push data in batch using SMART Rest

4

No, but it’s pretty simple. Just make sure to use line breaks for each message e.g.
Publish s/us (timestamp is optional, when not provided it’s likely they will get the same server timestamp and overwrite themselves on same measurements).

211,20,2019-09-30T12:21:30.593Z
211,40,2019-09-30T12:21:45.593Z

you can also mix messages:
Publish s/us (Creates a measurement + event)

211,20,2019-09-30T12:21:30.593Z
400,c8y_MyEvent,"Something was triggered"