Hello,
what I know from postman is that I can get from Cumulocity 2000 managed objects maximum in the same page. Going back to EPL, it seems that the limit is 1000, is it correct? thats the first question. The 2nd question is: should I handle pagination within EPL by writing the corresponding code? or EPL handles the pagination automatically?
note: as you can see generating the reqID again is commented in the last attempt, I was suspecting that maybe the handler is not updating to new reqID so I tested commenting it but I tested both generating a new reqID for findmanagedobject11 and using the old one, both options didn’t succeed in fetching all the expected pages
in most cases you should not be doing pagination yourself. Here you can find an explanation of the parameters:
If you do not provide pageSize and currentPage, Apama will automatically do the pagination for you with a pageSize of 1000. You can modify the pageSize to up to 2000 but that will not change the behavior you observe but will only lead to fewer request.
If you provide a current page, only that page will be fetched, but there are very few use cases for that.
Also, it does not look like you are listening to the right events. You have to listen for the Response to get the actual managed objects and for the ResponseAck to detect when you received the last event.
Typically the pattern should look like this:
FindManagedObject findManagedObject := new FindManagedObject;
findManagedObject.reqId := Util.generateReqId();
findManagedObject.params.add("fragmentType", configurationFragmentName);
monitor.subscribe(FindManagedObjectResponse.SUBSCRIBE_CHANNEL);
on all FindManagedObjectResponse(reqId=findManagedObject.reqId) as resp and not FindManagedObjectResponseAck(reqId=findManagedObject.reqId) {
// Process existing managed objects
}
on FindManagedObjectResponseAck(reqId=findManagedObject.reqId) {
monitor.unsubscribe(FindManagedObjectResponse.SUBSCRIBE_CHANNEL);
}
send findManagedObject to FindManagedObject.SEND_CHANNEL;
In reality pagination was not occurring automatically and we were receiving only 1000 managed objects. We had to do the pagination inside the EPL by sending a new API request under “on FindManagedObjectResponseAck” and removing reID=fmoReqId from both “FindManagedObjectResponse” and “on all FindManagedObjectResponseAck” because ReqID should change for each page while the handler is setup for only the 1st ReqID of the 1st page (now ReqID part is moved using an if inside “on all”
the initial code didn’t include current page, this was the initial one:
FindManagedObject findManagedObject := new FindManagedObject;
findManagedObject.reqId := Util.generateReqId();
findManagedObject.params.add("pageSize", "2000");
/* construct query to get all workers*/
// individual workers query => filter c8y_IsDevice
findManagedObject.params.add(“query”, “$filter=(has(c8y_IsDevice))”);
send findManagedObject to FindManagedObject.SEND_CHANNEL;
/* subscribe for query response*/
/* receive response and get all workers*/
monitor.subscribe(FindManagedObjectResponse.SUBSCRIBE_CHANNEL);
//why do we need "on all" here? it should be "on"
on all FindManagedObjectResponse(reqId=findManagedObject.reqId) as fmo and not FindManagedObjectResponseAck(reqId=findManagedObject.reqId) {
If you are still using currentPage, please remove it as suggested by @Sandeep_Komarneni. The documentation I linked to above clearly states that if it is provided no pagination takes place.
I did a quick test and the below code returns all 1201 matching devices on my tenant:
action onload() {
FindManagedObject findManagedObject := new FindManagedObject;
findManagedObject.reqId := Util.generateReqId();
findManagedObject.params.add("fragmentType", "c8y_TestDevice");
sequence<string> ids := new sequence<string>;
monitor.subscribe(FindManagedObjectResponse.SUBSCRIBE_CHANNEL);
on all FindManagedObjectResponse(reqId=findManagedObject.reqId) as resp and not FindManagedObjectResponseAck(reqId=findManagedObject.reqId) {
ids.append(resp.managedObject.id);
}
on FindManagedObjectResponseAck(reqId=findManagedObject.reqId) {
monitor.unsubscribe(FindManagedObjectResponse.SUBSCRIBE_CHANNEL);
log "Count: " + ids.size() at INFO;
}
send findManagedObject to FindManagedObject.SEND_CHANNEL;
}
you should only do the “send” after you have established the listener like you can see in my example as you otherwise might get responses before you have a listener reacting to them.
ok thanks, I will test that. So conclusion : in EPL to get more than 3000+ managed objects there is no need for pageSize and Pagination (currentPage) params in order to get the full response, Pagination happens automatically?
unlike API calls for example from postman or python (where maximum page size is 2000 managed objects) correct?
Initially we thought that the 2000 managed objects limit applies to EPL just like how it applies to inventory API, but the initial search (which might have been missleading) indicated that API calls to Inventory REST API directly have a limit of 2000 managed object per page, while Apama is going through the Cumulocity Java SDK / Microservice layer which enforces a 1000-item cap.
Correct, no need to do anything with pageSize or currentPage (in most cases).
EPL like the JavaSDK are still bound to the 2000 item page size limit of Cumulocity but they provide abstractions on top of it that mean you do not have to care about it yourself.
The 1000 item limit use by EPL is compromise to efficiently load many items once but not going to the 2000 item page limit of the API, which could actually be less efficient when processing in EPL.
ok thanks, from your perspective, what was the reason why my initial code before pagination didn’t get all the managed objects?
could it be because pagesize was defined?
my code: pagesize defined to 2000, while it seems EPL supports only 1000 in one page
your code: no explicit page size, could it be the reason? and what would be the pagesize here? default 5 like postman? while pagination happens automatically
in our case now the expected Managed objects are greater than 2000, we are receiving now 3125 managed objects and this number will increase daily, does it change anything in the previous feedback? or you already tested a number greater than 2000 ?
I tested with around 2100 managed objects not providing a page size, so the default of 1000 was used. I received all expected managed objects in my EPL App.
