# Dynamic Mapper - Map (MQTT) device data in a zero-code approach!

**URL:** <https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043>\
**Category:** Knowledge Base\
**Tags:** dynamic-mapper, thin-edge-io, mqtt, messaging-service, platform-services, mqtt-service, device-connectivity, cumulocity\
**Created:** [April 30, 2024, 12:37pm UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043 "2024-04-30T12:37:37Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stefan\_Witschel](https://dub1.discourse-cdn.com/flex005/user_avatar/community.cumulocity.com/stefan_witschel/32/31955_2.png) [@Stefan\_Witschel](https://community.cumulocity.com/u/Stefan_Witschel)\
**Post date:** [April 30, 2024, 12:37pm UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043/1 "2024-04-30T12:37:37Z")

</div>

## Introduction

Since years Cumulocity IoT is providing a MQTT endpoint dedicated to the Cumulocity Domain Model to connect MQTT devices. But what about devices using a pre-defined topic structure and any custom payload which cannot be modified because they are so called “closed”? Since the latest release of Cumulocity there is a new Service in private-preview - the **MQTT Service** - which is a full-fledged MQTT Broker without the limitation of pre-defined topic structure and payload. Still, it does not solve the problem to map device data to the Cumulocity Domain Model. Therefor we @Christof_Strack and myself implemented an open-source tool called **Dynamic Mapper** which allows you to map any data bi-directional in a zero-code approach.

In the first part of this article I will explain some patterns and concepts behind Device Connectivity and MQTT devices. In the second part I will provide a step-by-step guide how to connect a sample MQTT device and to send telemetry data to Cumulocity IoT.

**Let’s go!**

## MQTT Endpoint

Cumulocity provides an MQTT endpoint that let you connect devices via MQTT but they must support a pre-defined topic structure and the SmartREST data model or Cumulocity domain model via JSON.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/e/ae56e811df292a7049821dde8bbb466a814771eb.png)

This MQTT endpoint concept therefore only works for devices that can be customized or allow an installation of an agent like [thin-edge.io](http://thin-edge.io) but not for devices that are _closed_ and cannot be modified with an additional agent. If you wanted to connect devices with a pre-defined topic structure and payload to Cumulocity you had to use an externally hosted MQTT Broker and add some logic to map it to Cumulocity IoT.

## MQTT Service

Since the latest release in Q2 2024, the **MQTT Service** feature will be available as a private preview, which covers this gap as part of the platform.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/7/5/751d802f321ac6409ee7f9e933b1538ecabfc727.png)

Still, it will not cover the **map** and the **transport** part of an agent. So this must be covered by a custom component (microservice) which I want to highlight as a potential solution in this article:

> **[The Dynamic Mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper)**

## Dynamic Mapper

As stated in the chapter before the **Dynamic Mapper** tries to address the need to dynamically map **any** topic structure and data payload to the Cumulocity IoT domain model. We implemented it in a modular approach so it can support multiple message brokers and any data format like JSON, Protobuf, Binary, CSV, and many more.

### Architecture

To use the dynamic mapper you need to use a message broker either hosted externally or using the MQTT Service hosted by Cumulocity.

The Dynamic MQTT Mapper itself provides two artifacts:

1. A **microservice** which can be uploaded to Cumulocity IoT and
2. A **frontend** as a plugin which can extend existing Applications like the Administration App.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/7/f/7f90c6d38cbc2575a8cde6fbc08e0a991bc8b817.png)

### Components

The Dynamic Mapper has multiple components:

- **Connector clients** - the mapper has a modular connector concept which allows to add multiple connectors for each broker with required properties. Out of the box it supports **MQTT** , **MQTT Service** and **Kafka**. They allow to connect, publish and subscribe to topics.
- **Data Mapper** - handling of inbound & outbound messages from/to Broker in regards to the Cumulocity IoT Domain Model.. Also includes an expression runtime [JSONata](https://jsonata.org/) to execute expressions. Data Mappers again can be separated into Processors. For example, there is a _JSONProcessor_ handling any JSON payload and _FlatFileProcessor_ for processing flat-file payload (e.g. 10,Temperatur,20,°C) and it can be also easily extended with processor extensions.
- **C8Y Client** - implements part of the Cumulocity IoT REST API to integrate data. Also subscribes on the Notification 2.0 API to receive messages from Cumulocity IoT and forwards it to the data mapper.
- **REST Endpoints** - custom endpoints that are used by the MQTT frontend plugin or can be used to add mappings programmatically
- **Mapper Frontend** - A plugin for Cumulocity IoT to provide a UI for Broker Configuration, Data Mapping, Testing, Monitoring, and much more.

Also, the mapper microservice is fully **multi-tenant enabled**. You can deploy it once in your enterprise tenant and subscribe to subtenants to manage connections and mapping per tenant without using additional resources.

Let’s summarize the key features of the dynamic mapper:

- No-code data mapping capabilities
- Supports multiple types of message brokers like MQTT, MQTT Service or Kafka and can be extended easily.
- Supports multiple broker connections in parallel
- Bi-directional messaging: Broker ↔ Mapper ↔ C8Y
- Multi-data payload support like JSON, Protobuf, CSV, binary, and others
- Fully multi-tenant enabled
- Open-source developed and maintained
- Easy to extend (Broker connectors, payload processors, etc.)

No let’s check how to use it in practice. In the following I will guide you step-by-step to make use of that Dynamic Mapper.

## Step-by-step guide

### 1. Prerequisites

To follow this guide you need the following:

- A [Cumulocity IoT Tenant](https://signup.softwareag.cloud/#/basic-b?product=cumulocity&utm_referrer=TechCommunity) with microservice feature enabled. In trial tenants this feature is disabled. Please get in contact with your sales rep or support to request that feature for your tenant.
- A MQTT Client Tool like [MQTTx](https://mqttx.app)
- A publicly available MQTT Broker or MQTT Service. You can set up your own one or use existing sandboxes like [test.mosquitto.org](https://test.mosquitto.org/) or [others](https://mntolia.com/10-free-public-private-mqtt-brokers-for-testing-prototyping/). For this guide, I will use `test.mosquitto.org`.

### 2. Setup & Deployment

#### 1. Download of artifacts

As a next step we gonna download the artifacts of the [Dynamic Mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper): [Releases · Cumulocity-IoT/cumulocity-dynamic-mapper · GitHub](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/releases)  
Make sure to select the latest release. In the assets part of the release you’ll find two zip files:

1. `dynamic-mapping-service.zip` - This is the microservice of the dynamic mapper.
2. `dynamic-mapping-ui.zip` - This is the frontend plugin of the dynamic mapper.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/2/4/24d3eade33db9d17715d13a555686426f4ae812e.png)

Download them and store them somewhere on your disk.

#### 2. Upload of artifacts

Now we go the our Cumulocity IoT Tenant and upload both assets.

##### Microservice

We start with the microservice `dynamic-mapping-service.zip`.

In the Application Switcher we go to **Adminstration**. Afterwards we select **Ecosystem** → **Microservices**. Now we should see the “Add microservice” button on top right. If this is not the case please check the [Prerequisites](#prerequisites) that the microservice-hosting-feature is assigned to your tenant.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/2/8/28b6562dfa53dfc7e0521f0fb6958ab1c76bdbb9.png)

Select the `dynamic-mapping-service.zip` or drag & drop it in the area of the popup. When asked to subscribe please click the “Subscribe” button.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/4/0/409aa1b20675f271956e46e65012ef760edf2ffe.png)

After a couple of seconds, the microservice should be uploaded and activated. We can check that by selecting Dynamic-mapping-service in the list of microservice and checking the status.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/d/b/dbb1f6ec243340a3a046fbea23a6d6d328e66e90.png)

##### Frontend Plugin

Let’s continue with the frontend plugin (dynamic-mapping-ui.zip).

Now we can navigate to **Ecosystem** → **Applications** → **Extensions**.

Select the `Dynamic-mapping-ui` plugins or click on **Add extension package** and upload the `dynamic-mapping-ui.zip` manually.  
You should see the Dynamic-mapping packages in the packages list afterward.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/0/c/0cb49b54cc1b3b2b7801e1f4eaf48b9612ecd459.png)

Now we want to assign that plugin to the Administration App. Go to **Applications** and click on **Add application**. Click on **Duplicate existing application** and **Administration** afterward.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/0/a056247129adf9d8da91e026e2ae0b486f8be17e.png)

Click on `Duplicate`

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/3/4/34ed1dd636cf8b85948de52704a4fe5e1db1e758.png)

It will take a few seconds and you should see the new custom application **Administration** in the Application list. Select it by clicking on the name.

Go to the **Plugins** Tab. In the bottom right you see a button **Install plugin**. When clicked you see all available plugins in your tenant.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/c/e/ce5296bd4fec0c2ca21893fbfa4ac6532dd7497c.png)

Select **Dynamic Mapping Widget** and click **Install**.  
Now we need to refresh the page by pressing F5 or click the reload button in the browser.

In the left menu go to **Settings**. There you should see now **Dynamic Mapping** which is the plugin we have just installed.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/6/b/6b6187728c7f9930ad71c0dd3d59e2e16e1cc2cd.png)

**Congratulations you’ve successfully installed both components in your tenant!**

### 3.1 MQTT Service configuration

> **Please note:** MQTT Service is currently in private preview and is only available on request on your tenants. If you haven’t it enabled please proceed with chapter [3.2 MQTT Broker configuration](#h-32-mqtt-broker-configuration-16)

For adding MQTT Service you just need to add a MQTT Connector by clicking on **Connector** → **Add configuration**. Select **MQTT Service** and click **Save**

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/b/f/bf3ba07f6e1d2114d4d3b679a9c881093ade19e5.png)

Now activate that connector by clicking on the Action Button.

![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/2/e/2e4547130a1295791b41fa2aed49c1cf2b591435.png)

After a few seconds the status should change to ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/d/0/d0a4ea0c934724162ee86ab594c635ff32efb33d.png)

### 3.2 MQTT Broker configuration

To add an external MQTT Broker we use the following connection details of a public mosquitto MQTT broker:

```auto
Protocol: mqtts:/
MQTT Host: test.mosquitto.org
MQTT Port: 8886
ClientId: dynamic-mapping-client-{{anyNumber}}

```

Make sure to select **mqtts://** as protocol enter the **Mqtt Host** , **Mqtt Port** , **Client Id** and click on **Save**.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/e/e/ee9bf05ba58c5fd3a0de8464d05bfaa94343ed49.png)

Now activate that connector by clicking on the Action Button.

![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/2/e/2e4547130a1295791b41fa2aed49c1cf2b591435.png)

After a few seconds the status should change to ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/d/0/d0a4ea0c934724162ee86ab594c635ff32efb33d.png)

### 4. Create a device mapping

Now let’s assume we have a device, a CCU installed on a bus, that sends some JSON data like this:

```auto
topic: device/express/berlin_01
{ 
    "line": "Bus-Berlin-Rom", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 64, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International" 
} 

```

We go to the **Mapping Inbound** tab and click on **Add mapping**. In the next pop-up we select **JSON**.

#### Define Topic

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/d/8/d8fd68cc8e15076f84c0ea095b277e2d7b3a23b0.png)

- **Subscription Topic** : `device/#`
- **Mapping Topic** : `device/+/+`
- **Mapping Topic Sample** : `device/express/berlin_01`
- **Target API** : `Inventory`.
- **Map Device Identifier** & **Update Existing Device** : checked

Explanation:

- In **Subscription Topic** we enter `device/#` because we want to subscribe on all devices not only for express busses or the bus berlin\_01. This is the topic we subscribe to on the MQTT Broker.
- In **Mapping Topic** `device/+/+` is entered. The meaning is that this mapping will handle all messages for exactly 2 levels of sub-topics. As our topic is `device/express/berlin_01` we change that to the value `device/+/+`
- **Mapping Topic Sample** We define a concrete sample that is used for the testing. If we wouldn’t provide that testing is not possible.
- In Target API we select **Inventory** to create a new device in Cumulocity.
- Check **Map Device Identifier** & **Update Existing Device** - Map Device Identifier states that we want to correlate a unique ID to an external ID in Cumulocity and Update Existing Device states that it will also update the existing object in Cumulocity when it already exists.

Click on **Next**.

#### Define template

In the next view you see on the left the **source template** and on the right the **target template**. With the below we can manage the mappings.  
Let’s start by defining the source template.

Click on **Tree** in **Source** and change it to **Code**. We don’t touch the `_TOPIC_LEVEL_` but we add our example payload, copy & paste the following JSON message:

```auto
{ 
    "line": "Bus-Berlin-Rom", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 64, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International",
    "_TOPIC_LEVEL_": [
       "device",
       "express",
       "berlin_01"
  ]
} 

```

Now change back from **Code** to **Tree**.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/7/9/79a374912031bc4b98588b518cdbb181e38ae72c.png)

Let’s start with the important info the device identifier. We know that it is part of the topic _device/express/ **berlin\_01** _ so we select on the left side `berlin_01` and on the right side `id` and click on **+ Add substitution** to add the substitution.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/4/a4839f0341c99054900b93ab6d828a8364268290.png)

In **Substitutions** list we see this afterward:  
`00. [* _TOPIC_LEVEL_[2] --> id ]`

The second mapping will be the `customType` to `type`.  
Select `customType` on the left and `type` on the right. Again click on **+ Add Substitution**.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/0/d/0d07e0e615ebebfc31f2d6bea4895da9a6e496c4.png)

For the `name` we use now two properties from the left `operator` and `line` to map it to one property of the right `name`.  
This can be done by using an expression language called [JSONata](https://jsonata.org/). We select `operator` on the left and `name` on the right. Next we go to the field **Evaluate Expression on Source** and add the following:

```auto
operator&"-"&line

```

In the field **Result Type** you see the preview how the expression result will look like. Click on **+Add substitution** to add this substitution.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/5/d/5d06d3644451c3478e8805d03ab19ce34929aa63.png)

You can click on **?** to get more examples for expressions or full spec of the expression language.

At last, we map the `capacity` from the left to a new property on the right.  
In the **Target Tree** go with the mouse to the line below “id” and click the arrow down to open the menu. Scroll down an click on **+Value**

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/6/0/60a3f14c8bd270448eb46f526caa767268873016.png)

Enter `capacity` as the field and `100` as the value. The 100 value will be overwritten by the value of the source mapping so it doesn’t matter what we enter here.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/b/e/be9271f6a9fbf080c41a4c2f59c0b38e7c7f265e.png)

Now we have 4 substitutions and we are done with our mapping.  
Click on **Next**

#### Test mapping

In the next view we can evaluate how our mapping works. First click on **Transform Test Message** which will execute all substitutions based on provided source data.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/b/a/ba540c4ef82a631176f40e146eb564b9c39d42e1.png)

With **Show Next Test Result** we can iterate through all messages if we have an array mapped containing multiple device information. In our case, we just have one message.

Click on **Send Test Message**. It will run the mapping and send it to Cumulocity IoT. After some time you will see the response of Cumulocity IoT in the **Cumulocity Response**.

Everything looks good so we **Confirm** our mapping.  
Now in the Mapping list, you should see your mapping. By clicking on the **Active** Toggle Button you can enable your mapping.  
Our mapping is finished and we can continue sending device data to the MQTT Broker.

#### Send device data

Now let’s simulate that we connect our device to the generic MQTT Broker we have configured earlier.  
Start MQTTx client on your desktop. On **Connections** click on the **+** to create a new connection.  
We provide the following information:

```auto
Name: mosquitto_sandbox
Client_ID: (don't touch)
Host: mqtts://test.mosquitto.org
Port: 8886

```

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/1/9/1931a6a0b2937c4c8bddc0e1b295c7a4d04273e0.png)

Click on **Connect**.

> **Please note** : The mosquitto sandbox will terminate the connection after a few seconds with no activity. So you might have to re-connect.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/d/add005b8ef3cd032647212cf783454385b040c4e.png)

In the bottom right corner change the **Payload** to JSON. Enter the **topic** `device/express/berlin_01` with the payload:

```auto
{
    "line": "Bus-Berlin-Rom", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 64, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International2" 
}

```

> Please recognize that we changed the customType to `type_International2`. This will result in an update of the existing Device we have created during Testing.

Click on the bottom right button to send the message.

#### Confirm Device creation

Let’s check what happened! Go back to the Cumulocity IoT Dynamic Mapping App. Click on **Monitoring**.  
There you should see now an entry:

![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/optimized/3X/1/3/13a5c83cebe8f8a31033e604aa0dde0fb7a32ba4_2_690x28.png)

This means at least 1 message has been received without error.

Switch the application to **Device Management** using the Application Switcher in top right corner.

Click on **Devices** → **All devices** in the left navigation.  
Here you see a new device has been created of type `type_International2` we have just sent via MQTT Broker.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/7/7/77608bbb842aafba74944682a9ae7521fbc2b3e8.png)

We can now repeat the last steps by sending the same message with different values on different topics e.g. `device/express/berlin_02` with

```auto
{
    "line": "Bus-Prag-Leipzig", 
    "operator": "EuroBus", 
    "customFragment": { 
        "customFragmentValue": "Express" 
    }, 
    "capacity": 128, 
    "customArray": [ 
        "ArrayValue1", 
        "ArrayValue2" 
    ], 
    "customType": "type_International" 
}

```

### 5. Sending Measurements Events Alarms (MEAs)

In [4.](#4-create-a-device-mapping) we created mappings for our device definitions. Now we could also send telemetric data which is mapped to Cumulocity IoT as Measurements, Events or Alarms. Let’s start with an event. I’ll skip the parts I already have explained in detail.

#### Event Mapping

The following event data is sent by busses:

```auto
topic: event/berlin_01
{ 
    "msg_type": "c8y_BusStopEvent", 
    "txt": "Bus stopped at petrol station today!", 
    "td": "2024-04-026T15:21:53.389+02:00", 
    "ts": "1714149124000" 
} 

```

We recognize that the device identifier is again part of the topic.

##### Define topic

We create a new mapping:

- **Subscription Topic** : event/#
- **Template Topic** : event/+
- **Template Topic Sample** : event/berlin\_01
- **Target API** : Event
- **Active** , **Map Device Identifier** , **Create Non Existing Device** : Enabled

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/7/5/752f121aac9b7e250096dffea4cca718fea86d35.png)

##### Define template

Source Code:

```auto
{ 
    "msg_type": "c8y_BusStopEvent", 
    "txt": "Bus stopped at petrol station today!", 
    "td": "2024-04-026T15:21:53.389+02:00", 
    "ts": "1714149124000", 
    "_TOPIC_LEVEL_": [
       "event",
       "berlin_01"
  ]
} 

```

Substitutions:

```auto
00. [* _TOPIC_LEVEL_[1] -> source.id ]
01. [txt -> text]
02. [msg_type -> type]
03. [$now() -> time]

```

$now() is a JSONata function using the date-time when the event is processed by the mapper. As there is a timestamp `ts` or even a data `td` we can also map them. Cumulocity IoT expected always a date-time in the format: yyyy-MM-dd’T’HH:mm:ss.SSSZ. We can use the following function to get that date format from a UNIX timestamp:

```auto
$fromMillis(ts)

```

Click **Next**.  
**Test** the substitution.  
**Confirm** & **activate** the mapping.

##### Send events via MQTT Broker

Go to MQTTx and send on topic `event/berlin_01`the the following payload:

```auto
{ 
    "msg_type": "c8y_BusStopEvent", 
    "txt": "Bus stopped at petrol station today!", 
    "td": "2024-04-026T15:21:53.389+02:00", 
    "ts": "1714149124000" 
} 

```

##### Confirm events are processed

In **MQTT Mapping** we check the **Monitoring** again if any error occurs. If not we switch to **Device Management** and click on our device `EuroBus-Bus-Berlin-Rom`. In the **Events** Tab we should find now at least two events.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/9/2/92665c6240f62b8cc53947a563b4e3f8022be8f9.png)

#### Measurement Mapping

The following measurement data is sent by busses:

```auto
topic: measurement/berlin_01/gasoline
{ 
    "fuel": 65, 
    "ts": "2024-04-26T10:14:49.389+02:00", 
    "mea": "c8y_FuelMeasurement" 
} 

```

We recognize that the device identifier is again part of the topic.

##### Define topic

We create a new mapping:

- **Subscription Topic** : measurement/#
- **Template Topic** : measurement/+/gasoline
- **Template Topic Sample** : measurement/berlin\_01/gasoline
- **Target API** : Measurement
- **Active** , **Map Device Identifier** , **Create Non Existing Device** : Enabled

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/c/9/c9d7fdb1cb4d523e234ceaf6aa4e17c0ffbacbbc.png)

##### Define template

Source Template:

```auto
{ 
    "fuel": 65, 
    "ts": "2024-04-26T10:14:49.389+02:00", 
    "mea": "c8y_FuelMeasurement", 
    "_TOPIC_LEVEL_": [
       "measurement",
       "berlin_01",
       "gasoline"
  ]
} 

```

Target Template:

```auto
{ 
    "c8y_FuelMeasurement": { 
        "L": { 
            "value": 110, 
            "unit": "L" 
        } 
    }, 
    "time": "2024-04-26T10:14:49.389+02:00", 
    "source": { 
        "id": "909090" 
    }, 
    "type": "c8y_FuelMeasurement" 
}

```

Substitutions:

```auto
00. [* _TOPIC_LEVEL_[1] -> source.id ]
01. [fuel -> c8y_FuelMeasurement.L.value]
02. [$now() -> time]

```

Click **Next**.  
**Test** the substitution.  
**Confirm** the mapping.  
**Activate** the mapping.

##### Send measurement via MQTT Broker

Go to MQTTx and send on topic `measurement/berlin_01/gasoline` the following payload:

```auto
{ 
    "fuel": 40, 
    "ts": "2024-04-26T10:14:49.389+02:00", 
    "mea": "c8y_FuelMeasurement" 
} 

```

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/0/1/017e6c58d0bd73783c80145b0702fef9547438db.png)

##### Confirm measurements are processed

In **Dynamic Mapping** we check the **Monitoring** again if any error occurred. If not we switch to **Device Management** and click on our device `EuroBus-Bus-Berlin-Rom`. In the **Measurements** Tab we should find now a series of values.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/1/a1ba49457dbea1ac572e13279460045e52c1c848.png)

#### Alarm Mapping

The following alarm data is sent by busses:

```auto
topic: alarm/tires
{ 
  "msg_type": "c8y_FlatTireAlarm", 
  "tx": "Left rear tire loses air!", 
  "bus_id": "berlin_01" 
} 

```

We recognize that the device identifier is now part of the payload.

##### Define topic

We create a new mapping:

- **Subscription Topic** : alarm/tires
- **Template Topic** : alarm/tires
- **Template Topic Sample** : alarm/tires
- **Target API** : Alarm
- **Active** , **Map Device Identifier** , **Create Non Existing Device** : Enabled

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/2/5/2530ef35c36054f5521b2dacf42e6fa5e493778f.png)

##### Define template

Source Template:

```auto
{ 
  "msg_type": "c8y_FlatTireAlarm", 
  "tx": "Left rear tire loses air!", 
  "bus_id": "berlin_01",
  "_TOPIC_LEVEL_": [
    "alarm",
    "tires"
  ]
} 

```

Substitutions:

```auto
00. [* bus_id -> source.id]
01. [msg_type -> type]
02. [tx -> text]
03. [$now() -> time]

```

Click **Next**.  
**Test** the substitution.  
**Confirm** the mapping.  
**Activate** the mapping.

##### Send alarm via MQTT Broker

Go to MQTTx and send on topic `alarm/tires` the following payload:

```auto
{ 
  "msg_type": "c8y_FlatTireAlarm", 
  "tx": "Left rear tire loses air!", 
  "bus_id": "berlin_01"
}

```

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/5/4/54f2a3518bb1061984ba9cb5b9c0e4fe0971d64d.png)

##### Confirm alarms are processed

In **Dynamic Mapping** we check the **Monitoring** again if any error occurred. If not we switch to **Device Management** and click on our device `EuroBus-Bus-Berlin-Rom`. In the **Alarms** Tab we should find 1 alarm with a count of 2 or more now

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/8/a813fcdde83be21c8a4754b3acb371cc4237cf60.png)

### 6. Outbound Operation Mapping

Let’s try the other way around by creating an operation and using the Dynamic mapper to forward this operation to a specific device.

In Cumulocity we create the following operation:

```auto
{
  "description": "Reset all metrics of the Bus CCU",
  "deviceId": "berlin_01",
  "bus_command": "RESET_METRICS"
}

```

To start the mapping go to **Mapping outbound** Tab and click **Add mapping**. Here only JSON is supported.

#### Define topic

- **Publish Topic** : opp/outbound/berlin\_01
- **Publish Topic Sample** : opp/outbound/berlin\_01
- **Filter Outbound** : bus\_command
- **Target API** : Operation
- **Active** , **Map Device Identifier**

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/c/b/cb8bd7d6d228ad92e8f6a61f0a7cd35cebac4164.png)

#### Define template

Source Template:

```auto
{
  "description": "Reset all metrics of the Bus CCU",
  "deviceId": "berlin_01",
  "bus_command": "RESET_METRICS"
}

```

Target Template:

```auto
{
  "description": "dummy operation",
  "command": "Do something!",
  "type": "bus_operation",
  "_TOPIC_LEVEL_": [
    "opp",
    "outbound",
    "berlin_01"
  ]
}

```

Substitutions:

```auto
00. [* deviceId -> _TOPIC_LEVEL_[2] ] --> Important: Activate "Resolve to externalId" here
01. [bus_command -> command]
02. [description -> description]

```

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/a/f/af0fc31e1deef08f015c1b20d6faebbe3870f2ec.png)

Click **Next**.  
**Test** the substitution.  
**Confirm** the mapping.  
**Activate** the mapping.

#### Activate device subscription

Before the the mapper can process the operation we need to activate the device subscription. Click on ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/3/0/30ae930d62bd827aefb2d734378d278fb84e4a59.png) in the menu bar.  
Search for `berlin_01` and select the device `EuroBus-Bus-Berlin-Rom`.  
Finally, click on **Save** to add the subscription. In the **Subscription** table you should now see an entry:

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/e/2/e2ab8f0f6ceaef839be1b6e4b5379175c8fff57e.png)

##### Subscribe on opp/outbound/berlin\_01

Go to MQTTx click on **New Subscription** and enter `opp/outbound/berlin_01`

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/6/7/67573bd16d75ba555c207bc816b25e06c7e09446.png)

Now we create the operation via REST. Please note that you need to use the internal ID of your device for deviceId. As we activated the option it will be resolved to externalID automatically.

The internal device can be found by searching for `berlin_01` in device management. In the URL you can find the ID behind the `/device/` path.

```auto
POST {{url}}/devicecontrol/operations/
{
  "description": "Reset all metrics of the Bus CCU",
  "deviceId": "<<internal ID of device>>",
  "bus_command": "RESET_METRICS"
}

```

#### Confirm operation is received by device

In **Dynamic Mapping** we check the **Monitoring** again if any error occurred. If not we switch to **Device Management** and click on our device `EuroBus-Bus-Berlin-Rom`. In the **Device Control** Tab we should find 1 operation with a status successful.

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/c/3/c371d372792bcce26d5a13c10f94db7fcfbc3f9b.png)

In MQTTx check the received message. You should have one message received:

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/4/1/41e04f237ae0ced49612d628bceb2b202a845c71.png)

## Summary & Next Steps

In this article, I described how you can connect any device using MQTT and other brokers to Cumulocity IoT. With the dynamic mapper, you have a flexible way to map any data to the Cumulocity IoT domain model. It bridges the gap between handling proprietary data models and the Cumulocity IoT domain model.

I also showed in a step-by-step approach how you can try out the dynamic mapper yourself. This is just a starting point! There are many more features that I did not cover in this article. Here are a few noteworthy:

- [Support of FLAT\_FILES & GENERIC\_BINARY](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper#wizzard-to-define-a-mapping): Flat payloads like `90` or `oil,100,1666863595` or binary payloads like `5a75207370c3a47420303821` are mapped to a JSON you can transform again with JSONata.
- [Payload Snooping](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper?tab=readme-ov-file#enable-snooping-payloads-on-source-topic): You can enable snooping at the beginning of the mapping and later on use the snooped messages as templates in the template editor.
- **Mapping Tree overview** : In the UI there is a tab called **Mapping Tree**. It lists the whole tree of mappers you’ve created.
- **Logging** : In the Microservice Tab you can fully inspect the log file of the dynamic mapper. In case of any error you can research what happened and either correct your mapping or create an [issue at github](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/issues) if an issue with the component itself
- [REST API](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper?tab=readme-ov-file#api-documentation): There is a full REST API for all functions we are using from the UI. You can use it to programmatically manage your mappings within your tenants

If you want to read the full documentation please have a look here:

> [GitHub - Cumulocity-IoT/cumulocity-dynamic-mapper: The ultimate Mapper for building the bridge between any Message Broker and Cumulocity IoT in a zero-code approach!](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper)

Is there anything missing? Any connector you want to use or payload format which is currently not supported?

**We are very happy about feedback and contribution! So if you have an idea please approach us either here or using the GitHub project!**

* * *

> This article is part of the TECHniques newsletter blog - technical tips and tricks for the Software AG community. [Subscribe](https://info.softwareag.com/TechCommunity-Subscription-Page.html) to receive our quarterly updates or read the [latest issue](https://tech.forums.softwareag.com/techniques-latest).

---

<div class="post-metadata">

**Author:** ![Stefan\_Witschel](https://dub1.discourse-cdn.com/flex005/user_avatar/community.cumulocity.com/stefan_witschel/32/31955_2.png) [@Stefan\_Witschel](https://community.cumulocity.com/u/Stefan_Witschel)\
**Post date:** [July 26, 2024, 10:53am UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043/3 "2024-07-26T10:53:17Z")

</div>

4.4.0 of the dynamic mapper is now available with a lot of improvements

> **[Release Release v4.4.0 · Cumulocity-IoT/cumulocity-dynamic-mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/releases/tag/v4.4.0)**
>
> New Features
> 
> Reviewed complete UIs to improve first user experience #171:
> 
> include creating connectors as one step when creating a mapping
> added a second stepper for the snooping process, this imp...

---

<div class="post-metadata">

**Author:** ![Stefan\_Witschel](https://dub1.discourse-cdn.com/flex005/user_avatar/community.cumulocity.com/stefan_witschel/32/31955_2.png) [@Stefan\_Witschel](https://community.cumulocity.com/u/Stefan_Witschel)\
**Post date:** [September 6, 2024, 12:24pm UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043/4 "2024-09-06T12:24:00Z")

</div>

4.5.0 is now available which supports to activate/deactivate mappings per connector and tons of UX improvements.

> **[Release Release v4.5.0 · Cumulocity-IoT/cumulocity-dynamic-mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/releases/tag/v4.5.0)**
>
> New & Improved Features
> 
> Reviewed the UI to increase UX
> Activate Mappings per Connector #184
> Added Landing Page #250
> Disable/Enable buttons on step 5 test mapping #249
> Proper removal of Notificatio...

---

<div class="post-metadata">

**Author:** ![Stefan\_Witschel](https://dub1.discourse-cdn.com/flex005/user_avatar/community.cumulocity.com/stefan_witschel/32/31955_2.png) [@Stefan\_Witschel](https://community.cumulocity.com/u/Stefan_Witschel)\
**Post date:** [November 14, 2024, 1:40pm UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043/5 "2024-11-14T13:40:12Z")

</div>

And here we have the 4.6.0 release shining with the brand new Cumulocity branding, lot of scalability improvements and tons of bugs fixed!

> **[Release Release v4.6.0 · Cumulocity-IoT/cumulocity-dynamic-mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/releases/tag/v4.6.0)**
>
> New & Improved Features
> 
> Please note: This release is for Cumulocity 2021.0.x which is currently deployed on eu-latest environment. For earlier Cumulocity version please use 4.5.x version of the ma...

Thanks to all the contributors who created feature request, reported bugs and helped us to improve the mapper!

@Harald_Meyer @Yishu_Zhong @Vachaspati_Diwevedi2 and many more.

![Video gif. A toddler sits at a table with a cracker in her hands. She looks at us with a big excited smile and then grins while giving a big thumbs up. Text, “Thank you!”](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/b/2/b27cd28ae8c8928b67c1818b0f2252e161907b9c.gif)

---

<div class="post-metadata">

**Author:** ![Stefan\_Witschel](https://dub1.discourse-cdn.com/flex005/user_avatar/community.cumulocity.com/stefan_witschel/32/31955_2.png) [@Stefan\_Witschel](https://community.cumulocity.com/u/Stefan_Witschel)\
**Post date:** [January 7, 2025, 2:41pm UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043/6 "2025-01-07T14:41:37Z")

</div>

We just published 4.7.0 release of Dynamic Mapper.  
This time it includes unfortunately breaking changes because we cleaned up the mapping data format which was quite dirty due to historical changes in the past.  
Also it includes some UI and scalability improvements by supporting Spring Boot 3.3.5 and using virtual threads now to scale better.

> **[Release Release v4.7.0 · Cumulocity-IoT/cumulocity-dynamic-mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/releases/tag/v4.7.0)**
>
> What's Changed
> 
> !! Breaking Change !!
> Caution this release contains breaking changes which makes it necessary to migrate existing mappings! For details please read on.
> 
> The format used for serializ...

If you have feedback or questions do not hesitate to comment or create an issue in the repository!

![Parks And Recreation Wink GIF](https://europe1.discourse-cdn.com/flex005/uploads/cumulocity/original/3X/2/3/237af32ef783b5f7b85757eae39b95c0fe82c9d1.gif)

---

<div class="post-metadata">

**Author:** ![Stefan\_Witschel](https://dub1.discourse-cdn.com/flex005/user_avatar/community.cumulocity.com/stefan_witschel/32/31955_2.png) [@Stefan\_Witschel](https://community.cumulocity.com/u/Stefan_Witschel)\
**Post date:** [January 14, 2025, 4:42pm UTC](https://community.cumulocity.com/t/dynamic-mapper-map-mqtt-device-data-in-a-zero-code-approach/3043/7 "2025-01-14T16:42:04Z")

</div>

## 4.7.1 Pre-Release is available.

Main new feature of this pre-release is a new **generic HTTP/REST endpoint** that can be used to receive any payload on any sub-path.

The endpoint for the new `Default Http Connector` can be accessed at the url `https://<YOUR_CUMULOCITY_TENANT>/service/dynamic-mapping-service/httpConnector/<MAPPING_TOPIC>`

For example if a json payload sent to `https://<YOUR_CUMULOCITY_TENANT>/service/dynamic-mapping-service/httpConnector/temp/berlin_01` it will be resolved to a mapping with mapping topic: `temp/berlin_01`

So mappings will be applied same as when using MQTT but just for the sub-path of the URL where the payload is sent to.

> **[Release Release v4.7.1\_pre · Cumulocity-IoT/cumulocity-dynamic-mapper](https://github.com/Cumulocity-IoT/cumulocity-dynamic-mapper/releases/tag/v4.7.1_pre)**
>
> What's New
> 
> New feature to handle HTTP/REST request via a generic REST endpoint.
> Guide
> PR #300
> 
> What's Changed
> Full Changelog: v4.7.0...v4.7.1\_pre

Is there anything missing? Do you think this is useful?  
Looking forward for your feedback!