Overview
Dynamic Mapper is evolving with the latest release 6.1.0 to 6.1.5.
Since 6.1.0, new code-based mappings were introduced, which will also be used in an upcoming product by the platform: Smart Functions.
Disclaimer: This is an AI-generated video by NotebookLM
Besides that, many other features have been added:
- AMQP Connector support
- MQTT Service Pulsar connector
- A new connector to support C8Y to C8Y mappings
- and many more
Check out the full release notes for all the features & updates:
In this article, I want to highlight two things:
- The new smart function code mappings
- The updated extensions capability
Let’s start with the Smart Functions.
Smart Functions to replace code mappings
Before Smart Functions, we supported JavaScript Code Mappings with an own interface very similar to the one we are using for graphical mappings. It was based on substitutions.
Here is an example:
/**
* @name Default template, one measurement
* @description Default template, one measurement
* @templateType INBOUND_SUBSTITUTION_AS_CODE
* @direction INBOUND
* @defaultTemplate true
* @internal true
* @readonly true
* sample to generate one measurement
* payload
* {
* "temperature": 139.0,
* "unit": "C",
* "externalId": "berlin_01"
* }
* topic 'testGraalsSingle/berlin_01'
*/
function extractFromSource(ctx) {
//This is the source message as json
const sourceObject = JSON.parse(ctx.getPayload());
tracePayload(ctx);
// Define a new Measurement Value for Temperatures by assigning from source
const fragmentTemperatureSeries = {
value: sourceObject['temperature'],
unit: sourceObject['unit']
};
// Assign Values to Series
const fragmentTemperature = {
T: fragmentTemperatureSeries
};
// Create a new SubstitutionResult with the HashMap
const result = new SubstitutionResult();
// Add time with key 'time' to result.getSubstitutions()
// const time = new SubstitutionValue(sourceObject['time'], 'TEXTUAL', 'DEFAULT', false);
// addSubstitution(result, 'time', time);
// Define temperature fragment mapping temperature -> c8y_Temperature.T.value/unit
const temperature = new SubstitutionValue(fragmentTemperature, TYPE.OBJECT, RepairStrategy.DEFAULT, false);
// Add temperature with key 'c8y_TemperatureMeasurement' to result.getSubstitutions()
addSubstitution(result, 'c8y_TemperatureMeasurement', temperature);
// Define Device Identifier
const deviceIdentifier = new SubstitutionValue(sourceObject['_TOPIC_LEVEL_'][1], TYPE.TEXTUAL, RepairStrategy.DEFAULT, false);
// Add deviceIdentifier with key ctx.getGenericDeviceIdentifier() to result.getSubstitutions()
addSubstitution(result, ctx.getGenericDeviceIdentifier(), deviceIdentifier);
return result;
}
We received feedback from multiple parties that this concept is working fine but is improvable in the way that we should allow more “native” and intuitive JavaScript Code Mappings. In previous code mappings, the main function was the addSubstitution function, which was provided by Dynamic Mapper. Still, it was hard to implement and test code-mappings that way.
Luckily, in internal discussions about a new product to be implemented, the concept of Smart Functions arose.
Smart Functions are much more clean and native than the old mappings:
/**
* @name Create either measurement or event
* @description Create either measurement or event
* @templateType INBOUND_SMART_FUNCTION
* @direction INBOUND
* @defaultTemplate false
* @internal true
* @readonly true
*
*/
function onMessage(msg, context) {
var payload = msg.getPayload();
console.log("Context: " + context.getStateAll());
console.log("Payload Raw: " + payload);
console.log("Payload messageId: " + payload.get("messageId"));
let result;
const payloadType = payload["payloadType"];
if (payloadType == "telemetry") {
result = {
cumulocityType: "measurement",
action: "create",
payload: {
"time": new Date().toISOString(),
"type": "c8y_TemperatureMeasurement",
"c8y_Steam": {
"Temperature": {
"unit": "C",
"value": payload["sensorData"]["temp_val"]
}
}
},
externalSource: [{"type":"c8y_Serial", "externalId": payload.get("clientId")}]
}
} else {
// if type == "error"
result = {
cumulocityType: "event",
action: "create",
payload: {
"time": new Date().toISOString(),
"type": "c8y_ErrorEvent",
"text": payload["logMessage"],
"severity": "MAJOR",
"status": "ACTIVE"
},
externalSource: [{"type":"c8y_Serial", "externalId": payload.get("clientId")}]
}
}
return result;
}
The main function is the onMessage function that provides the msg and context.
As part of the msg, you get mainly the payload, while the context provides additional data like topic. clientId and helper functions.
As an output, one or multiple messages are expected in the format:
{
cumulocityType: "{The API to call e.g. event, measurement, alarm etc.}"
action: "{create, update}"
payload: {
"{The payload that is expected by the API}"
},
externalSource: [{"type":"c8y_Serial", "externalId": "{external Identifier of the device}"}]
}
So here you are very flexible in the following regards:
- You can create multiple messages for the same input message
- You can call multiple APIs for the same input message
- You are totally free how you implement your mapping, be it helper functions or just replace values of the json with variables.
- You can easily test your function as the input and output has a clear definition.
Therefore, with 6.1.5, we deprecated the “old” code-based mappings and will replace them with the smart functions.
Also by using Smart Function you might get a jump-start when the new processing component is available as part of the product because it will be using the same or similar interface.
As a runtime, we are still using the same: GraalJS. As the microservice runtime is still implemented in Java, we execute JavaScript by leveraging GraalJS, which has some performance impact for interpreting & executing the JavaScript code.
Our performance tests showed that JavaScript Mappings are having half of the performance as a native Java mapping.
For that reason, we also invested some time in modernizing the Extension capability of the Dynamic Mapper.
Extensions for high performance mappings
Since some time already, the Dynamic Mapper could be extended with Java code mappings.
The idea was to implement a Java interface, build a jar, and deploy it to the runtime. Here the Java interface was kind of complicated to understand as it was following a similar concept the code-based mappings were by using the concept of substitutions:
@Override
public void extractFromSource(ProcessingContext<byte[]> context)
throws ProcessingException {
try {
Map jsonObject = (Map) Json.parseJson(new String(context.getPayload(), "UTF-8"));
context.addSubstitution("time", new DateTime(
jsonObject.get("time"))
.toString(), TYPE.TEXTUAL, RepairStrategy.DEFAULT,false);
Map fragmentTemperatureSeries = Map.of("value", jsonObject.get("temperature"), "unit",
jsonObject.get("unit"));
Map fragmentTemperature = Map.of("T", fragmentTemperatureSeries);
context.addSubstitution("c8y_Fragment_to_remove", null, TYPE.TEXTUAL,
RepairStrategy.REMOVE_IF_MISSING_OR_NULL, false);
context.addSubstitution("c8y_Temperature",
fragmentTemperature, TYPE.OBJECT, RepairStrategy.DEFAULT,false);
context.addSubstitution("c8y_Temperature",
fragmentTemperature, TYPE.OBJECT, RepairStrategy.DEFAULT,false);
// as the mapping uses useExternalId we have to map the id to
// _IDENTITY_.externalId
context.addSubstitution(context.getMapping().getGenericDeviceIdentifier(),
jsonObject.get("externalId")
.toString(),
TYPE.TEXTUAL, RepairStrategy.DEFAULT,false);
Number unexpected = Float.NaN;
if (jsonObject.get("unexpected") != null) {
// it is important to use RepairStrategy.CREATE_IF_MISSING as the node
// "unexpected" does not yet exists in the target payload
Map fragmentUnexpectedSeries = Map.of("value", jsonObject.get("unexpected"), "unit", "unknown_unit");
Map fragmentUnexpected = Map.of("U", fragmentUnexpectedSeries);
context.addSubstitution("c8y_Unexpected",
fragmentUnexpected, TYPE.OBJECT, RepairStrategy.CREATE_IF_MISSING, false);
unexpected = (Number) jsonObject.get("unexpected");
}
log.info("{} - New measurement over json processor: {}, {}, {}, {}", context.getTenant(),
jsonObject.get("time").toString(),
jsonObject.get("unit").toString(), jsonObject.get("temperature"),
unexpected);
} catch (Exception e) {
throw new ProcessingException(e.getMessage());
}
}
With now Smart Functions introduced, we changed the interface to a very similar one we are also using in JavaScript smart functions but leveraging available Java capabilities like builder:
@Override
public CumulocityObject[] onMessage(Message<byte[]> message, DataPreparationContext context) {
try {
// Parse JSON payload
String jsonString = new String(message.getPayload(), "UTF-8");
@SuppressWarnings("unchecked")
Map<String, Object> payload = (Map<String, Object>) Json.parseJson(jsonString);
log.info("{} - Context state: {}", context.getTenant(), context.getStateAll());
log.debug("{} - Payload raw: {}", context.getTenant(), jsonString);
log.info("{} - Processing message, messageId: {}",
context.getTenant(), payload.get("messageId"));
// Extract common data
String clientId = (String) payload.get("clientId");
String payloadType = (String) payload.get("payloadType");
// Decide which type of Cumulocity object to create based on payloadType
if ("telemetry".equals(payloadType)) {
// Create measurement
@SuppressWarnings("unchecked")
Map<String, Object> sensorData = (Map<String, Object>) payload.get("sensorData");
Number tempVal = (Number) sensorData.get("temp_val");
log.info("{} - Creating temperature measurement: {} C for device: {}",
context.getTenant(), tempVal, clientId);
return new CumulocityObject[] {
CumulocityObject.measurement()
.type("c8y_TemperatureMeasurement")
.time(new DateTime().toString())
.fragment("c8y_Steam", "Temperature", tempVal.doubleValue(), "C")
.externalId(clientId, "c8y_Serial")
.build()
};
} else {
// Create event for error or other payload types
String logMessage = (String) payload.get("logMessage");
log.warn("{} - Creating error event for device: {}, message: {}",
context.getTenant(), clientId, logMessage);
return new CumulocityObject[] {
CumulocityObject.event()
.type("c8y_ErrorEvent")
.text(logMessage)
.time(new DateTime().toString())
.externalId(clientId, "c8y_Serial")
.build()
};
}
} catch (Exception e) {
String errorMsg = "Failed to process inbound message: " + e.getMessage();
log.error("{} - {}", context.getTenant(), errorMsg, e);
context.addWarning(errorMsg);
return new CumulocityObject[0];
}
}
Feedback wanted: This extension interface is pretty new and we are looking forward for your feedback about it
Also here we added the flexibility of Smart Functions to have multiple messages as an output.
In a YAML file, you define all the extensions you have implemented. It will be bundled as part of the jar and used by the runtime to select it for mappings.
extensions:
- eventName: ConditionalMeasurementOrEvent
className: dynamic.mapper.processor.extension.external.inbound.ProcessorExtensionSmartInbound04
description: Conditional processor creating either measurement or event based on payload type
version: "2.0"
When creating a new mapping, you can select extension as mapping type and select one of the available mappings:
Now why I am stating this as “high performance”? Because it is just native Java code that is executed in a Java runtime. Meaning there is almost no execution overhead.
In our performance tests, we achieved twice the performance of JavaScript Mappings by using an extension. This means Java extension mappings are very efficient, and you get much more throughput for the assigned resources when using them instead of JavaScript mappings.
When to use what?!
Graphical Mappings and JavaScript Code-Mappings are the way to start with Dynamic Mapper if you are new to it. It will give you the easiest and best user experience.
If you are not expecting heavy load for your mappings and JavaScript is your language to define mappings, go for the Smart Functions JavaScript Mappings. In most cases, it will be performant enough to fulfill the job of your requirements.
Also, if you need more guidance by using our AI agents, JavaScript might be the way to go, as we have an integrated agent that suggests code based on your provided samples messages and input.
If you are expecting heavy load on your mappings logic, there is no way around using Java extensions. The same is if you feel more comfortable implementing stuff in Java and you don’t need built-in AI to generate code for you.
It’s fair to mention that JSONata graphical mappings achieve nearly the same performance as java extension mappings. So if you are not a fan of “coding” but “clicking” your (simple) mappings, it will come with no performance drawback. It is that way because we use a java JSONata lib that executes JSONata expressions natively in java.
Try it out!
Now we are very interested in your feedback! Give it a try and play around with both new features. For the Smart Functions, we have added one example as part of the samples when clicking on
For the extensions, you can find a lot of examples in the repository:
You can build the extension by using mvn package, which will generate a jar file that can be uploaded in the UI of Dynamic Mapper.