Error while trying firmware OTA

Forum
, , ,
1

Hello,

I am testing the firmware OTA operation in Cumulocity and although the procedure was working ok yesterday, today I get this error:

Operation updated: status: "FAILED", failure reason: "Unexpected failure of operation. Details: Cannot invoke "com.cumulocity.lwm2m.agent.server.device.nodevisitors.C8yLwM2mNodeVisitor.getValue(String)" because "this.visitor" is null ...", description: "Update firmware to:

So it seems that there is an issue with Cumulocity itself as I get this error as soon as I press the install firmware button.

thanks

4

Hi Haris,

We investigated the issue and found that the operations are failing because the server evaluates the device as offline.

The Root Cause:
The device is currently configured with a registration lifetime of only 1 second (lifetimeInSec=1). This means that exactly one second after the device do a new registration or registration update, the server marks it as offline, blocking any subsequent operations.

How to Resolve This:
To fix this, the registration lifetime needs to be increased. You can do this in one of two ways:

  • Device-Side Update: Increase the lifetime value directly within the device’s configuration.
  • Via LwM2M Configuration UI from the Device management: Go to the device’s LwM2M Configuration UI → Servers written during bootstrap, update the “Registration lifetime” field (which is written during bootstrap), and then trigger a new bootstrap for the device.

Note: We recognize that the error message you encountered wasn’t very clear, and our team will be updating it to ensure it provides better context in the future.

5

Weird. It was working yesterday and as I stated the error is immediate after the button press. The operation is normally scheduled for next registration so the device is not registered when we press the button for OTA.

After I press the button I get this:

  • Jun 4, 2026, 1:20:03 PMOperation created.

    Operation created: status: “PENDING’, description=‘Update firmware to: “ads700_v5.10.1” (version: 5.10.1)’, device name='ADS700”.

  • Jun 4, 2026, 1:20:03 PMOperation updated.service_lwm2m-agent

    Operation updated: status: “EXECUTING”, description: “Update firmware to: “ads700_v5.10.1” (version: 5.10.1)”", device name: “ADS700”.

It states Executing without the device having received the command.

6

When an operation is in the Executing status, the LWM2M Service sends the command directly to the device. If a response is not received until time out, the service is designed to automatically revert the operation back to a PENDING status so it can be retried as soon as the device reconnects during its next registration.
The timeout period is by default is 3 minutes and this is also configurable in device’s LWM2M Configuration UI.

If the service already detects that a device is offline, it intentionally holds back the operation to prevent unnecessary failed delivery attempts. However, in your case, instead of gracefully moving the operation back to PENDING to wait for the next check-in, the service failed it directly.

This is not the intended behavior, and we will soon updating the handling on the Service side so that these operations are correctly queued for retry in the future.

We still highly recommend increasing your device’s registration lifetime value. The currently set to 1 second (lifetimeInSec=1) window is incredibly narrow; it causes the device to be considered as offline almost instantly from the service’s perspective.

7

Gotcha. Thanks.

One more thing, it seems that if I include both objects 5000 and 5 (firmware) then the registration fails. But when I include only one of the two, registration is successful. Do you recommend anything here?:

objArray\[0\] = get_security_object_x509(

    123,

    "coaps://lwm2m.eu-latest.cumulocity.com:5784",

    NULL, 0, NULL, 0, false);

objArray\[1\] = get_server_object(123, "U", 1, 0);

objArray\[2\] = get_object_device();

objArray\[3\] = get_object_senml();

objArray\[4\] = get_object_firmware();

I am using the wakaama library. I did change the lifetime variable btw that’s just old code

8

Cumulocity’s LWM2M Service don’t have a restriction regarding the objects and you can include all the supported objects.

However, in order to do firmware updates with Cumulocity object 5 should be available.

9

I am having a hard time testing the OTA procedure as I think Cumulocity keeps messing the OTA state machine up.

Please give it a look once you get the chance as it’s difficult trying to debug my application when I have outside errors as well.

Sometimes deleting the device and then reregistering again solves the issue. Sometimes not. My update delivery method is push only btw.

For example I managed to receive 44 blocks of my firmware before the procedure failed for some reason. From that point I do a device power cycle (no states saved anywhere) and I delete my device in Cumulocity then create again and I cannot start another OTA since.

thanks

10

Hi Haris,

From the LWM2M Service logs, the LWM2M Service sends the firmware package to the 5/0/0 resource blockwise.

The device responds with 2.31 Continue but without acknowledging the block.

Therefore, the LWM2M Service tries to send the first block again a couple of times and in the end it fails due to the unexpected response.

This is the last sent block before failure and the acknowledgement received from your device:

2026-06-06T12:14:36.575Z TRACE 1 — [erver(main)#120] c.lwm2m.leshan.server.CoapMessageTracer  : LWM2M-SERVER /*** <== req CON-PUT    MID=40896, Token=A47EDD6E10FBF102, OptionSet={“Uri-Path”:[“5”,“0”,“0”], “Content-Format”:“application/octet-stream”, “Block1”:“(szx=5/512, m=true, num=0)”, “Size1”:187660}, EB 17 A6 03 08 00 00 00 00 00 00 03 00 00 00 00 F4 0A 0A F4 1C 00 00 00 10 00 00 00 01 00 00 00..512 bytes


2026-06-06T12:14:41.213Z TRACE 1 --- [erver(main)#110] o.e.californium.core.network.UdpMatcher  : received response ACK-2.31   MID=40896, Token=A47EDD6E10FBF102, OptionSet={}, <empty data> from DTLS(***:14672,ID:54494D453A)

When doing the similar blockwise firmware push with our Leshan simulation device:

2026-06-09T12:44:20.102Z TRACE 1 — [erver(main)#153] c.lwm2m.leshan.server.CoapMessageTracer  : LWM2M-SERVER /*** <== req CON-PUT    MID=34224, Token=FCCF7B1F4166CAF7, OptionSet={“Uri-Path”:[“5”,“0”,“0”], “Content-Format”:“application/octet-stream”, “Block1”:“(szx=5/512, m=true, num=833)”}, BB FA EA D0 A6 D7 A0 FF C2 C2 42 DF F1 EA 8E 1D 93 FA FA 7A F3 A9 A8 A8 30 ED 4A 4B 4B 3B 2C 58..512 bytes


2026-06-09T12:44:20.157Z TRACE 1 --- [erver(main)#347] o.e.californium.core.network.UdpMatcher  : received response ACK-2.31   MID=34224, Token=FCCF7B1F4166CAF7, OptionSet={"Block1":"(szx=5/512, m=true, num=833)"}, <empty data> from UDP(***:16766)

The received response from our simulation device contains the “OptionSet” property with information about the block.
Then block acknowledgement gets recognized and continues with the next block.

11

Thanks for that.

I fixed it now and today I had a successful OTA but I cannot recreate it.

Can you please check the logs again to see what is happening?

From my end everything seems to be identical on my device.

Working sequence:

  1. package write called ← Cumulocity writes to /5/0/0
  2. Full object read — all resources including state, result, pkg_name, delivery_method
  3. OBSERVE option present for /5/0/3 (state)
  4. OBSERVE option present for /5/0/5 (result)
  5. First block arrives — (UDP DATA:1,541)

On the failed attempts the agent skips the full read and goes straight to failing, so steps 3 and 4 are missing most of the time.

Also after changing the LWM2M OTA Config Tab in Cumulocity to Push only i consistently get this error:
Operation updated: status: “FAILED”, failure reason: “[DELIVER FIRMWARE] Couldn"t establish observation of /5/0/3 resource with device ID [type=com_cumulocity_model_idtype_GId, value=59458264]', description: “Update firmware to: “0000” (version: 0000)””, device name: “ADS700”.

thanks

12

Hi Haris,

I see for this device new registrations and de-registration happening very frequently.

De-registration interrupted observation request sent to the device.

13

I am testing it quite frequently but since I do a fresh register each time and sometimes I successfully get the whole firmware OTA I still don’t see how that could happen.

Meaning I register, try OTA and deregister upon timeout.

thanks

14

The device stays alive for 30+ seconds after the full object read. During these 30 seconds before Cumulocity fails (because we de-register), no CoAP packet arrives at the device — not even the Observe request. Cumulocity is not transmitting the CoAP Observe to the device. Can you check your server logs to see why the Observe CoAP packet is not being sent?

In one successful OTA session on Friday, we did receive the Observe requests for /5/0/3 and /5/0/5, and the full transfer completed successfully.

thanks

15

Hi,

After the de-registration, no operations will be sent to the device.

I see during the firmware package delivery step, this device re-connects with new registration.

However, as this is PUSH-based firmware update, the device should be online until firmware package is delivered.

In the LWM2M standard, resuming firmware image delivery is not possible for the PUSH firmware updates.

If you would like to have retrying / resuming firmware image delivery downloads, this is supported with PULL-based firmware delivery.

16

I found the root cause on our side. The lwm2m_send (SenML data report) sent immediately after registration was interfering with Cumulocity’s OTA observation setup. We now detect whether there is a pending OTA operation on the first incoming packet after registration, and skip the SenML send if OTA is in progress. OTA now works reliably.

Thanks so much for the help

17

One more thing, does Cumulocity automatically send an Execute firmware command after getting state=2?

Or should I do the execution on my own at a different stage?

I’ve tried waiting for the execute command but I never receive it. And then when I update the result to 1 (successful) Cumulocity retried the downloading of the new firmware again.

thanks

18

Hi Haris,

Yes, the service automatically sends the execute when it receives state=2 (Downloaded).

If this didn’t happen, it could be that an error faced while delivering the firmware.

19

I update the object’s state to 2 (visible in the object’s page in Cumulocity) but I never receive the execute command. So I correctly change the state from 0 to 1 and then 2 when i get the last block.

Does this off by one discrepancy play any role?

image
image415×477 19.6 KB

If I use this state machine instead of the standard 0=idle… I get this:

Operation updated: status: “FAILED”, failure reason: “Another firmware update process is still in progress. Current state: DOWNLOADING”, description: “Update firmware to: “ADS700” (version: 8.8.10)”", device name: “ADS700”.

which makes sense since 1=downloading normally.

thanks

20

Hi Haris,

Since the state is still showing as ‘Downloading’, it is likely that the firmware image did not completely transfer.

To resolve this, I suggest canceling the current firmware update process:

Once canceled, please restart the update from the beginning.

21

Hi,

Here is a fresh session:

  • Jul 1, 2026, 12:50:37 PMOperation created.

    Operation created: status: “PENDING’, description=‘Update firmware to: “ADS700” (version: 0.0.4)’, device name='ads700dev”.

  • Jul 1, 2026, 12:52:23 PMOperation updated.service_lwm2m-agent

    Operation updated: status: “EXECUTING”, description: “Update firmware to: “ADS700” (version: 0.0.4)”", device name: “ads700dev”

After the last block I change state:

image
image285×253 7 KB

Cumulocity never sends an execute command and goes back to pending.

  • Jul 1, 2026, 1:00:44 PMOperation updated.service_lwm2m-agen t

    Operation updated: status: “PENDING”, description: “Update firmware to: “ADS700” (version: 0.0.4)”", device name: “ads700dev”.

Can you please check the logs to see if an Execute command had been sent at that time?

If it has then the issue is on my end.

thanks

22

Hi Haris,

Based on our service logs, the firmware file delivery was successfully completed, and the device acknowledged receipt. We also confirmed that the device reported a successful “Downloaded” state.

However, it appears the actual “execute firmware” request was never sent to the device.

To help us investigate, we have increased the logging level on our end. Could you please try triggering the firmware update one more time? Additionally, please let us know if this behavior happens every time you attempt an update on this device, or if it occurs intermittently.