The client talks to the ShareLLM server through a small HTTP API. It only makes outbound requests and never accepts connections. HTTP/1.1 over TLS is enough; no WebSockets, MQTT or callbacks are needed.
Every server request carries Authorization: Bearer <SHARELLM_WORKER_KEY>.
- Discover. Read installed models from local Ollama:
GET {OLLAMA_URL}/api/tags. - Register.
POST /internal/workers/registerwith{"clientId": "...", "models": ["..."]}. This repeats everyMODEL_REFRESH_SECONDS, so newly pulled models appear automatically. - Heartbeat.
POST /internal/workers/heartbeatwith{"clientId": "..."}everyHEARTBEAT_SECONDS. Heartbeats run on their own thread, so they continue while an inference is running. - Claim.
GET /internal/jobs/claim?clientId=...&waitSeconds=...long-polls for work. A new job is only claimed after the current one has been fully handled. - Execute. Run the job's payload on local Ollama with
"stream": falseforced, so the response is a single JSON object. - Report.
POST /internal/jobs/{id}/completeorPOST /internal/jobs/{id}/fail, always including the job'sleaseToken.
A claimed job looks like this:
{
"id": "job-42",
"leaseToken": "opaque-fencing-token",
"endpoint": "chat",
"model": "llama3.2:latest",
"payload": "{\"model\":\"llama3.2:latest\",\"messages\":[...]}"
}payload is the requester's original JSON body, sent either as a string or as an embedded object.
Job endpoint |
Ollama path |
|---|---|
chat, openai-chat |
/api/chat |
generate, openai-completion |
/api/generate |
show |
/api/show |
embed, openai-embedding |
/api/embed |
embeddings |
/api/embeddings |
Any other endpoint is reported as a failure (Unsupported endpoint: <name>).
| Call | Status | Client reaction |
|---|---|---|
| claim | 204 No Content |
No work. Pause 250 ms, then poll again. |
| claim, heartbeat | 404 Not Found |
The server forgot this worker. Re-register. |
| complete / fail | 409 Conflict |
The lease was lost to another worker. Discard the result, no retry. |
| any | other non-2xx, or a network error | Log it. Claims retry after 2 s; results retry up to RESULT_POST_RETRIES times. |
| Scenario | What happens |
|---|---|
| Ollama returns an error or times out | The client reports fail so the server can retry on another worker. |
| Job payload is not a JSON object | Reported as fail. |
| Server temporarily unreachable while delivering | Retried every 2 s, up to RESULT_POST_RETRIES extra attempts. |
| Delivery still fails after all retries | The client gives up. The server requeues the job when the lease expires. |
| Laptop sleeps, crashes or loses network | Heartbeats stop. The server requeues the job after its lease or heartbeat threshold. |
| Old worker comes back with a stale result | The server answers 409, because the job has a new lease token. The stale result is dropped. |
| Client is shut down mid-job | Nothing is reported. The lease expires and the job is requeued. |
Because of requeuing, a prompt can occasionally run twice, but only the current lease holder's result is ever accepted.
Each client runs one request at a time. CPU, GPU and memory use is decided by your Ollama installation, not by ShareLLM. When several workers offer the same model, the first worker to claim a job gets it. If no live worker offers the requested model, the request stays queued on the server.