How to Connect Local AI Models to Cumulocity AI Agent Manager

Rahul_Talreja · August 20, 2026, 8:45am

Authors: @Hendrik_Naether1 and @Rahul_Talreja

If you have experimented with local AI models, you already know the appeal. You get full control over the models you use, no per-request cloud costs, and the freedom to test a wide range of open-source models.

Connecting those models to a platform like Cumulocity, however, can be a challenge. Many integrations assume you are using a cloud-based AI service, which usually means an API key and a wait on your IT team.

Cumulocity AI Agent Manager can connect to any model server that exposes an OpenAI-compatible API. This includes locally hosted models running with tools such as Ollama, LM Studio, LocalAI, or vLLM. You can point AI Agent Manager to a model running on your own machine and start experimenting with AI agents using a setup you already have.

In this post, we will show you how to connect a locally hosted model to AI Agent Manager and test agents without a cloud provider or API key. We will cover two approaches: a quick tunnel for a first experiment and the local-ai-proxy microservice for a setup that feels like a native part of your tenant.

Step 1: Prepare Your Local Model Engine

To host local AI models, you need a runner that exposes an OpenAI-compatible API endpoint. This is the key detail that makes everything else in this post work: any inference server that supports the OpenAI /v1/chat/completions API, such as Ollama, LM Studio, vLLM, or LocalAI, will connect to AI Agent Manager without any extra setup on the Cumulocity side.

Depending on whether you prefer a command line tool or a graphical interface, you have two easy options.

Ollama (CLI-based) Install Ollama by following the setup steps in the official Ollama documentation. Once installed, run your preferred model (for example, ollama run gemma3:4b). This automatically exposes an OpenAI-compliant local endpoint at: http://192.168.178.10:11434/v1
LM Studio (GUI-based) If you prefer a visual interface, download and set up the app using the official LM Studio documentation. You can download any model directly inside the app, then go to the Developer / Local Server tab and start the local server. This exposes an OpenAI-compatible endpoint on your chosen local port.

image2206×524 58.2 KB

If You Are on Cumulocity Edge
Cumulocity Edge runs on your own infrastructure, so you can skip the tunneling step entirely and point AI Agent Manager directly at your local endpoint (for example http://192.168.178.10:11434/v1) without exposing anything to the public internet. On cloud-based Cumulocity, continue with Step 2 — the cloud platform needs a public URL to reach your machine.

Step 2: Expose Your Local Endpoint (Reverse Proxy)

Cloud platforms cannot reach localhost directly, so you need to expose your local port securely using a tunnel or reverse proxy tool such as ngrok or localtunnel.

ngrok is a cross-platform tool that lets you expose a local web server to the internet. Think of it as a secure door from your local machine to the outside world. Once it is set up correctly, you can reach your machine from anywhere.
`

You can find setup instructions here: ngrok + Ollama or ngrok + LM Studio

Run the following command to expose your Ollama port:

bash

ngrok http 11434

Copy the generated HTTPS forwarding URL, for example:

https://UniqueURL.ngrok-free.dev/v1

Note: This URL changes every time you restart ngrok. If you want a persistent URL, follow the steps in ngrok’s guide on free static domains. A free ngrok account lets you claim one static domain.

A quick word on security here, since you are exposing a local service to the internet. The free tier of ngrok already gives you some protection, since it assigns a new URL every time you start it, which limits how long that door stays open. If you plan to keep a persistent URL, it is worth looking into related authentication features so you can control who is able to reach your endpoint.

Step 3: Configure the Local Provider in Cumulocity

In the Cumulocity Administration app, go to AI Agent Manager and define a local provider JSON override for your agent.

Set the local provider JSON configuration using the OpenAI format:

json

{
  "name": "openai",
  "baseURL": "https://UniqueURL.ngrok-free.dev/v1",
  "model": "google/gemma-4-e4b"
}

name: Set to "openai" so AI Agent Manager formats requests according to the OpenAI Chat Completions API standard.
baseURL: Your public reverse proxy URL, pointing to the local /v1 endpoint.
model: The model identifier loaded inside Ollama or your local inference engine.

Save the configuration, and your agent will now route its requests to your local model. The best part is that you can switch back and forth between hosted providers like Anthropic or OpenAI and your local, on-premise, or edge-hosted model just by changing this configuration, without rewriting any application code.

The same logs can also be seen in the developer logs of the relevant tooling, in this case, LM Studio.

Beyond ngrok: the local-ai-proxy microservice

The ngrok approach above is perfect for a quick experiment, but it has drawbacks for anything longer-lived: your local model server is exposed on a public URL, the URL changes on every restart (unless you claim a static domain), and authentication is bolted on rather than integrated with your tenant. If you want a setup that feels like a native part of Cumulocity, you can replace the ngrok tunnel with a small Cumulocity microservice: local-ai-proxy.

How it works

local-ai-proxy inverts the direction of the connection. Instead of exposing your machine to the internet, a small client on your machine opens a single outbound WebSocket to the microservice running on your tenant. The microservice then relays AI requests back through that tunnel:

What this buys you over a plain reverse proxy:

Nothing on your machine is ever exposed. LM Studio stays bound to localhost; the tunnel client only makes outbound connections. There is no public URL to your laptop at all.
Tenant-native authentication. The /v1 endpoint sits behind the Cumulocity gateway like any other microservice, so the agent authenticates with a regular technical user and role (ROLE_LOCAL_AI_PROXY_ACCESS). The tunnel itself is protected by a shared secret stored as a tenant option.
A stable URL. https://<your-tenant>/service/local-ai-proxy/v1 never changes, no matter how often your laptop reconnects.
Full OpenAI surface, including streaming. Every /v1/* call, including models, chat/completions, responses, and embeddings, is relayed transparently, and stream: true responses are forwarded chunk-by-chunk as real server-sent events, so you see tokens as they are generated.
A stateful-API shim for the Responses API. Some agents drive the OpenAI Responses API in stateful mode: after a model turn, the follow-up request does not resend prior output items. It sends { "type": "item_reference", "id": "..." } and expects the server to remember them. Local engines like LM Studio are stateless and reject those references. The proxy plays the server’s role: it captures every output item as it streams past and expands item_reference entries back into full items before forwarding, and it normalizes replayed items (reasoning, messages, function calls) into the stricter input shapes local engines accept. Multi-turn tool-calling agents work against a local model because of this.
Resource hygiene. If the agent disconnects mid-generation or a request times out, the proxy sends a cancel over the tunnel and the client aborts the LM Studio request, so your GPU stops generating tokens nobody will read.

The tunnel client keeps itself alive with heartbeats, reconnects automatically with exponential backoff, and proactively recycles its connection before the platform’s request-duration ceiling, so an idle overnight tunnel is still connected in the morning. One deliberate limitation: the tunnel relays text (JSON and SSE). Binary endpoints such as /v1/audio/* are not supported.

Configuring local-ai-proxy on your tenant

Here is the full setup, end to end. You need: admin access to your tenant, the local-ai-proxy microservice binary (built from the project with pnpm build), and LM Studio (or another OpenAI-compatible server) running locally.

1 · Deploy and subscribe the microservice

Upload the microservice in Administration › Ecosystem › Microservices (or deploy it with your usual CI/CD or the c8y CLI) and subscribe your tenant to it. After subscribing you should see it listed with:

Name: local-ai-proxy
Path: /service/local-ai-proxy
Isolation: Single tenant. The tunnel bridge is in-memory, so the service runs as a single replica. This is enforced by the manifest, not something you need to configure.

The subscribed microservice in Administration: name local-ai-proxy, path /service/local-ai-proxy, single-tenant isolation.

2 · Set the tenant options

The microservice reads its configuration from tenant options in the category local.ai:

Key	Required	Purpose
`tunnel.secret`	yes	Shared secret the tunnel client must present. Generate one with `openssl rand -hex 32`.
`agentUser`	optional	Technical username, used only to auto-fill the provider JSON returned by `/provider-config`.
`credentials.agentPassword`	optional	That user’s password (stored encrypted by the platform).
`publicBaseUrl`	optional	Override if your tenant’s runtime base URL is not the externally reachable domain.

Set them via REST, for example:

curl -u "<admin-user>:<admin-password>" \
  -H "Content-Type: application/json" \
  -X POST "https://<your-tenant>/tenant/options" \
  -d '{"category": "local.ai", "key": "tunnel.secret", "value": "<your-generated-secret>"}'

3 · Create the technical user for the agent

AI Agent Manager needs credentials to call the proxy through the gateway. Create a dedicated user (for example svc-local-ai-proxy) with a strong password, and assign it a global role that contains only the ROLE_LOCAL_AI_PROXY_ACCESS permission. This user can call this one microservice and nothing else.

4 · Start the tunnel client on your machine

On the machine running LM Studio, configure the client’s .env:

# Where the deployed microservice listens for the tunnel
PROXY_WSS_URL=wss://<your-tenant>/service/local-ai-proxy/agent-tunnel

# Must match the tunnel.secret tenant option
TUNNEL_SECRET=<your-generated-secret>

# Your local OpenAI-compatible server
LMSTUDIO_URL=http://127.0.0.1:1234

# C8Y Basic auth for the gateway on the WebSocket upgrade
# printf '<user>:<password>' | base64
C8Y_AUTH=Basic <base64 of user:password>

Then start it (pnpm start). You should see tunnel connected. Verify from the outside:

curl -u "svc-local-ai-proxy:<password>" https://<your-tenant>/service/local-ai-proxy/health
# » {"status":"ok","tunnel":"connected"}

5 · Point your agent at the proxy

The microservice will even write the provider config for you: GET /service/local-ai-proxy/provider-config returns paste-ready JSON with the base URL, a suggested model id (read live from LM Studio through the tunnel), and, if you set the optional tenant options in step 2, the filled-in Authorization header.

In Administration › AI Agent Manager, open your agent, go to Local provider, and paste:

{
  "name": "openai",
  "baseURL": "https://<your-tenant>/service/local-ai-proxy/v1",
  "apiKey": "unused-dummy",
  "model": "google/gemma-4-26b-a4b-qat",
  "headers": {
    "Authorization": "Basic <base64 of svc-local-ai-proxy:password>"
  }
}

Two differences from the ngrok variant in Step 3: the apiKey is a dummy (the real authentication is the Authorization header checked by the Cumulocity gateway), and the model must be the exact id LM Studio reports under /v1/models.

Important: the model id must match your LM Studio setup. The value shown here (google/gemma-4-26b-a4b-qat) is just an example. Set model to whatever model you currently have loaded in LM Studio. Copy the id verbatim from GET /v1/models (or let /provider-config fill it in for you, since it reads the id live through the tunnel). If you switch models in LM Studio later, update this field to match.

Save, hit Test, and your agent is now talking to the model on your desk, through your tenant, with no public exposure of your machine.

Troubleshooting

Symptom	Likely cause
`503 tunnel_offline` from `/v1/*`	The tunnel client is not running or is not connected. Check its console output.
Client can’t connect (401/404 on the WebSocket upgrade)	`C8Y_AUTH` missing or wrong; the gateway requires valid Basic auth on the upgrade.
Client connects, then is closed with code 1008	`TUNNEL_SECRET` does not match the `tunnel.secret` tenant option.
`422 unresolved_item_reference`	The agent referenced an output item the proxy no longer has (proxy restarted mid-conversation, or the item expired). Start a fresh conversation.
Agent errors on long non-streaming requests	Raise the buffered timeout via `TUNNEL_BUFFERED_TIMEOUT_MS` on the microservice (default 120 s; `TUNNEL_FIRST_BYTE_TIMEOUT_MS` and `TUNNEL_STREAM_IDLE_TIMEOUT_MS` control the streaming path).

Give this a try and let us know in the comments which local models you got working with AI Agent Manager. We would love to hear about your setup and any issues you ran into along the way.

martin.hall-may · August 26, 2026, 2:36pm

This is very interesting. I have set up a local model as described (using gemma3:4b on Ollama), but testing the agent gives an error with no particular details:

How can I debug this? Also, is it possible to add the Basic Auth header to the requests as described in Expose and Secure Your Self-Hosted Ollama API - ngrok documentation?

Rahul_Talreja · August 27, 2026, 12:57pm

Hi Martin, thanks for trying this out.

Regarding the error, could you check the developer logs of the relevant tooling? In my case, the LM Studio developer logs should show the actual request coming from AI Agent Manager and can help identify whether the issue is with the endpoint, model name, or request format.

You can also test the exposed endpoint independently using curl or directly in the browser. I have now added a sample screenshot for both of these to the article as well.

martin.hall-may · August 27, 2026, 1:27pm

Hi Rahul,

The endpoint seems to work as expected when tested with cURL:

curl https://$NGROK_DOMAIN/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gemma3:4b",
        "messages": [
          {
            "role": "system",
            "content": "You are a helpful assistant."
          },
          {
            "role": "user",
            "content": "Hello!"
          }
        ]
      }'

Response:

{
    "id": "chatcmpl-597",
    "object": "chat.completion",
    "created": 1787753360,
    "model": "gemma3:4b",
    "system_fingerprint": "fp_ollama",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello there! How can I help you today? 😊 \n\nDo you have a question, need some information, or just want to chat? Let me know!"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 22,
        "completion_tokens": 34,
        "total_tokens": 56
    }
}

However, when I send a test message from Cumulocity, there appears to be no inbound request in the Ollama logs. This seems to suggest there is a problem sending the request. Is there a way I can debug on the Cumulocity side?

Rahul_Talreja · August 31, 2026, 2:34pm

Hi Martin,

Since the request from AI Agent Manager is not reaching LLM, the issue is most likely between AI Agent Manager and ngrok. A few things to check:

Check the ngrok inspector at http://127.0.0.1:4040. If the request doesn’t appear there, it’s likely not reaching ngrok. If it appears in ngrok but not LLM, the issue is likely between ngrok and the local LLM setup.

image1410×935 52.8 KB
Double-check the connector endpoint URL in Agent Manager — especially the ngrok domain, https vs. http, and any trailing slash. Small differences here can cause the request to fail. I initially ran into this myself as well.

Regards
Rahul

martin.hall-may · September 3, 2026, 7:53am