Skip to content

Latest commit

 

History

History
90 lines (68 loc) · 3.9 KB

protocol.md

File metadata and controls

90 lines (68 loc) · 3.9 KB

Worker protocol & reliability

← Back to README

The client talks to the ShareLLM server through a small HTTP API. It only makes outbound requests and never accepts connections. HTTP/1.1 over TLS is enough; no WebSockets, MQTT or callbacks are needed.

Every server request carries Authorization: Bearer <SHARELLM_WORKER_KEY>.

Lifecycle

  1. Discover. Read installed models from local Ollama: GET {OLLAMA_URL}/api/tags.
  2. Register. POST /internal/workers/register with {"clientId": "...", "models": ["..."]}. This repeats every MODEL_REFRESH_SECONDS, so newly pulled models appear automatically.
  3. Heartbeat. POST /internal/workers/heartbeat with {"clientId": "..."} every HEARTBEAT_SECONDS. Heartbeats run on their own thread, so they continue while an inference is running.
  4. Claim. GET /internal/jobs/claim?clientId=...&waitSeconds=... long-polls for work. A new job is only claimed after the current one has been fully handled.
  5. Execute. Run the job's payload on local Ollama with "stream": false forced, so the response is a single JSON object.
  6. Report. POST /internal/jobs/{id}/complete or POST /internal/jobs/{id}/fail, always including the job's leaseToken.

Job format

A claimed job looks like this:

{
  "id": "job-42",
  "leaseToken": "opaque-fencing-token",
  "endpoint": "chat",
  "model": "llama3.2:latest",
  "payload": "{\"model\":\"llama3.2:latest\",\"messages\":[...]}"
}

payload is the requester's original JSON body, sent either as a string or as an embedded object.

Supported endpoints

Job endpoint Ollama path
chat, openai-chat /api/chat
generate, openai-completion /api/generate
show /api/show
embed, openai-embedding /api/embed
embeddings /api/embeddings

Any other endpoint is reported as a failure (Unsupported endpoint: <name>).

Result messages

// POST /internal/jobs/{id}/complete
{ "clientId": "my-laptop-001", "leaseToken": "...", "response": { /* Ollama JSON */ } }

// POST /internal/jobs/{id}/fail
{ "clientId": "my-laptop-001", "leaseToken": "...", "error": "local model inference returned HTTP 500" }

Server status codes

Call Status Client reaction
claim 204 No Content No work. Pause 250 ms, then poll again.
claim, heartbeat 404 Not Found The server forgot this worker. Re-register.
complete / fail 409 Conflict The lease was lost to another worker. Discard the result, no retry.
any other non-2xx, or a network error Log it. Claims retry after 2 s; results retry up to RESULT_POST_RETRIES times.

Failure scenarios

Scenario What happens
Ollama returns an error or times out The client reports fail so the server can retry on another worker.
Job payload is not a JSON object Reported as fail.
Server temporarily unreachable while delivering Retried every 2 s, up to RESULT_POST_RETRIES extra attempts.
Delivery still fails after all retries The client gives up. The server requeues the job when the lease expires.
Laptop sleeps, crashes or loses network Heartbeats stop. The server requeues the job after its lease or heartbeat threshold.
Old worker comes back with a stale result The server answers 409, because the job has a new lease token. The stale result is dropped.
Client is shut down mid-job Nothing is reported. The lease expires and the job is requeued.

Because of requeuing, a prompt can occasionally run twice, but only the current lease holder's result is ever accepted.

Capacity

Each client runs one request at a time. CPU, GPU and memory use is decided by your Ollama installation, not by ShareLLM. When several workers offer the same model, the first worker to claim a job gets it. If no live worker offers the requested model, the request stays queued on the server.