Skip to content

Latest commit

 

History

History
189 lines (141 loc) · 9.4 KB

docker.md

File metadata and controls

189 lines (141 loc) · 9.4 KB

Running with Docker

← Back to README

The client is meant to run on your own laptop or desktop, next to the models in your local Ollama. Docker is an easy way to keep it running in the background. It starts again with Docker when you log in or reboot, and you don't need Java on your machine.

The container only makes outbound connections: it publishes no ports and needs no inbound firewall rules.

Choose your setup

Your laptop Ollama Compose files COMPOSE_FILE in .env
macOS or Windows (Docker Desktop) Installed natively (recommended) compose.yaml (leave unset)
Linux (Docker Engine) Installed natively compose.yaml + compose.linux.yaml compose.yaml:compose.linux.yaml
Any In a container as well compose.yaml + compose.ollama.yaml compose.yaml:compose.ollama.yaml

Tip

On a Mac, install Ollama natively. Containers on macOS cannot use the Apple GPU, so a containerized Ollama runs on the CPU only and is much slower. On Linux with an NVIDIA GPU, the containerized Ollama can use the GPU (see Ollama in a container).

Quick start

Prerequisites: Docker Desktop (macOS/Windows) or Docker Engine with the Compose plugin (Linux), plus Ollama with at least one model.

ollama pull llama3.2                     # 1. make a model available on the laptop

cp .env.example .env                     # 2. configure (see below)
$EDITOR .env

docker compose up -d --build             # 3. build the image and start the worker in the background
docker compose logs -f                   # 4. watch it register: "Registered my-laptop-001 with 1 models"

The first build compiles the project and runs the full test suite inside Docker, which takes a few minutes. Later builds reuse the cached Maven dependencies.

Configure .env

Docker Compose reads .env from the project folder automatically. At minimum, set:

SHARELLM_SERVER_URL=https://sharellm.example.org
SHARELLM_WORKER_KEY=worker-secret-from-your-server
SHARELLM_CLIENT_ID=my-laptop-001

All three are required in a container, and Compose refuses to start with a clear message if one is missing. SHARELLM_CLIENT_ID is required here because otherwise the container would get a new random ID on every restart, and the server would see a new worker each time.

Container-specific settings

Variable Default Description
COMPOSE_FILE compose.yaml Which overlays to use, see Choose your setup.
CONTAINER_OLLAMA_URL http://host.docker.internal:11434 (http://localhost:11434 with the Linux overlay) How the container reaches Ollama. Kept separate from OLLAMA_URL, so the same .env also works for running natively.
SHARELLM_CLIENT_MEMORY 512m Memory limit of the client container. The JVM uses up to 75% of it.
SHARELLM_CLIENT_IMAGE sharellm-client:local Image name to build or run.
OLLAMA_HOST_PORT 11434 Laptop port for the containerized Ollama (Ollama overlay only).
OLLAMA_IMAGE_TAG latest ollama/ollama image tag (Ollama overlay only).

All the regular settings (HEARTBEAT_SECONDS, INFERENCE_TIMEOUT_SECONDS, …) are passed through as well. See the configuration reference.

Reaching a ShareLLM server on the same laptop

Inside a container, localhost means the container itself. If you run the ShareLLM server on the same machine (for example for development), use:

SHARELLM_SERVER_URL=http://host.docker.internal:8080

With the Linux overlay, the container shares the host network, so http://localhost:8080 works there. Plain HTTP to host.docker.internal does not trigger the security warning, because the traffic never leaves your machine.

How the container reaches Ollama

Ollama listens on 127.0.0.1:11434 by default, which is the laptop's loopback interface. A container cannot reach that directly.

  • Docker Desktop (macOS/Windows): the special name host.docker.internal forwards to the laptop's loopback, so the default CONTAINER_OLLAMA_URL works without any change.
  • Linux: host.docker.internal points to the Docker bridge, which cannot reach a service bound to 127.0.0.1. The compose.linux.yaml overlay therefore runs the client with network_mode: host, so localhost is the laptop again. This is safe here because the client still opens no ports.
    • Alternative: keep the bridge network and make Ollama listen on the bridge with OLLAMA_HOST=0.0.0.0 (systemctl edit ollama). Only do this if your firewall blocks port 11434 from the outside, because it exposes Ollama on your network.

Check that Ollama is reachable from the client's point of view:

docker compose run --rm --entrypoint wget sharellm-client -qO- http://host.docker.internal:11434/api/tags

Ollama in a container

If you don't want to install Ollama on the laptop, the compose.ollama.yaml overlay runs it next to the client. Models are stored in the ollama_models volume, so they survive restarts.

echo 'COMPOSE_FILE=compose.yaml:compose.ollama.yaml' >> .env
docker compose up -d --build
docker compose exec ollama ollama pull llama3.2

The client re-reads the model list every MODEL_REFRESH_SECONDS (default 120), so a newly pulled model is offered to the server within two minutes. Restart the client to pick it up straight away: docker compose restart sharellm-client.

Ollama is published on 127.0.0.1:11434 (only reachable from the laptop itself), so the ollama command and other local tools on the laptop can still use it. If a native Ollama already uses that port, set OLLAMA_HOST_PORT to another port.

NVIDIA GPU (Linux): install the NVIDIA Container Toolkit and uncomment the deploy block in compose.ollama.yaml.

Everyday commands

Task Command
Start in the background docker compose up -d
Follow the logs docker compose logs -f sharellm-client
Stop sharing (keeps the container) docker compose stop
Remove the containers docker compose down
Rebuild after updating the code docker compose up -d --build
Faster rebuild without running tests docker compose build --build-arg SKIP_TESTS=true
Status docker compose ps

The service uses restart: unless-stopped. It starts automatically whenever Docker starts, until you stop it yourself. On macOS and Windows, enable Start Docker Desktop when you sign in in the Docker Desktop settings.

Laptop sleep: while the laptop sleeps, heartbeats stop, and the server requeues any running job to another worker. After wake-up the client reconnects and re-registers on its own.

Without Compose

docker build -t sharellm-client:local .

docker run -d --name sharellm-client --restart unless-stopped \
  --add-host host.docker.internal:host-gateway \
  --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges:true \
  -e SHARELLM_SERVER_URL=https://sharellm.example.org \
  -e SHARELLM_WORKER_KEY=worker-secret-from-your-server \
  -e SHARELLM_CLIENT_ID=my-laptop-001 \
  sharellm-client:local

On Linux with a native Ollama, replace --add-host ... with --network host and add -e OLLAMA_URL=http://localhost:11434.

Image details

Property Value
Base image eclipse-temurin:25-jre-alpine (JRE only), built with maven:3.9-eclipse-temurin-25
Build Multi-stage. Runs mvn verify, including the 100% line-coverage gate, unless SKIP_TESTS=true
Layers Spring Boot layered jar: dependencies are cached separately from application code
User Non-root, UID/GID 10001
Ports None
Default OLLAMA_URL http://host.docker.internal:11434
JVM -XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError (restart on OOM instead of hanging)

The compose service is hardened further: read-only root filesystem with a tmpfs for /tmp, all Linux capabilities dropped, no-new-privileges, a memory limit, and rotated logs (3 × 10 MB).

There is no Docker HEALTHCHECK. The client exposes no endpoint to probe, and it recovers from server or Ollama outages by itself. Check its health in the logs (Registered …, Heartbeat failed …).

Troubleshooting

Symptom in the logs Cause and fix
required variable SHARELLM_… is missing a value Set that variable in .env.
Initial registration failed: … Connection refused (Ollama) The container cannot reach Ollama. Is Ollama running (ollama list)? On Linux, use the Linux overlay. Test with the wget command above.
Registered … with 0 models Ollama is reachable but has no models. Run ollama pull <model> (or docker compose exec ollama ollama pull <model>).
Heartbeat failed: … Connection refused (server) SHARELLM_SERVER_URL is wrong or the server is down. Use host.docker.internal instead of localhost for a server on the same laptop.
register returned HTTP 401 / 403 SHARELLM_WORKER_KEY does not match the server's worker key.
PKIX path building failed The server's HTTPS certificate is not trusted by the JVM in the image. Use a certificate from a public CA.
Container restarts repeatedly Check docker compose logs. An invalid setting stops the app at startup with APPLICATION FAILED TO START.