diff --git a/.claude/skills/olla-validate/SKILL.md b/.claude/skills/olla-validate/SKILL.md new file mode 100644 index 00000000..6d8ddd3b --- /dev/null +++ b/.claude/skills/olla-validate/SKILL.md @@ -0,0 +1,288 @@ +--- +name: olla-validate +description: > + Full automated validation of Olla via parallel agents against mock backends + (no Docker, no real LLMs - CI/CD safe). Gates on make ready first, then boots + an ollamock fleet + Olla and fans out per-area validation agents covering + core routing, OpenAI/Anthropic surfaces, observability, health/recovery and + failure handling. Use as the pre-release gate or after major changes. + Trigger when the user asks to validate Olla, run the validation suite, + run regression/nightly validation, or gate a release. + Runs on Sonnet regardless of the session model; area agents are balanced + between Sonnet and Haiku for token efficiency. +argument-hint: "[--quick|--nightly] [--soak-minutes=N]" +model: sonnet +--- + +# /olla-validate - Olla Validation Harness + +One reusable suite, two depths: + +| Mode | Target time | Scope | +|---|---|---| +| `--quick` | 5–10 min | `make ready` gate + live smoke of every area (trimmed checklists) | +| `--nightly` | 2–4 h | full test suite + stress + exhaustive checklists + chaos + soak + sherpa pass + translation-forced pass + bench snapshot | + +**If neither flag is given, ask** with AskUserQuestion: "Which validation depth?" - options **Quick (Recommended after major changes)** (5–10 min smoke gate) and **Nightly** (multi-hour exhaustive pre-release gate). `--soak-minutes=N` overrides the nightly soak duration (default 30). + +The harness never touches real backends. All traffic goes to `test/cmd/ollamock` +instances (stdlib Go mock speaking OpenAI, Ollama-native, LM Studio, Lemonade +and Anthropic protocols, with a `/_mock/` fault-injection control plane). + +## Model assignment (token efficiency) + +The orchestration runs on Sonnet (`model: sonnet` above) - it makes the +judgement calls, sequences nightly passes and writes the report. Subagents +cannot spawn subagents, so orchestration must stay here; never try to +delegate the whole run to a single agent. + +Spawned area agents get an explicit `model` on every Agent call: + +- **haiku** - mechanical curl-and-assert checklists that never mutate state: + core-routing, openai-api, observability, limits-failures (client section). +- **sonnet** - anything that mutates mock/Olla state or needs protocol- + sequence judgement: resilience, anthropic (both sections - SSE event-order + and translation validation), limits-failures (upstream section). + +Escalation rule: if a Haiku agent returns malformed report JSON, dies, or its +failures look like agent error rather than product error (e.g. it asserted +the wrong endpoint set), re-run that one area once on Sonnet before trusting +a FAIL. Record the escalation in the report. + +## Topology (fixed ports) + +One ollamock instance per endpoint - Olla keys endpoint identity on the URL, +so endpoints sharing a URL silently collapse to one (last declared wins). + +| Port | Process | Endpoint (type) | Models | +|---|---|---|---| +| 19431 | ollamock `mock-a` | mock-openai-a (openai-compatible) | test-model, shared-model | +| 19432 | ollamock `mock-b` | mock-openai-b (openai-compatible) | test-model, beta-model | +| 19433 | ollamock `mock-c` | mock-ollama-c (ollama, native `/api/*`) | llama3.1:8b, shared-model | +| 19434 | ollamock `mock-d` | mock-lmstudio-d (lm-studio) | phi-4, shared-model | +| 19435 | ollamock `mock-e` | mock-vllm-e (vllm - anthropic passthrough target) | test-model, shared-model | +| 19436 | ollamock `mock-f` | mock-litellm-f (litellm - anthropic translation target) | test-model, beta-model | +| 19437 | ollamock `mock-g` | mock-llamacpp-g (llamacpp) | phi-4, shared-model | +| 41141 | olla (main) | `test/validate/config.validate.yaml` - 7 endpoints, unifier, sticky, anthropic translator | | +| 41142 | olla (limits) | `test/validate/config.validate.limits.yaml` - 256KB body cap, 30 req/min per-IP, backed by mock-a | | + +Mock fault injection: `POST http://127.0.0.1:194NN/_mock/behaviour` with +`{"mode":"ok|error|flaky|hang|slow","error_status":503,"error_rate":0.5,"latency_ms":0,"fail_health":false,"drop_mid_stream":false,"malformed_json":false}`; +`POST /_mock/reset` restores defaults; `GET /_mock/stats` gives per-path request +counts. See `test/cmd/ollamock/README.md`. + +## Phase 0 - Setup + +```bash +MODE=quick|nightly # from args or AskUserQuestion +SOAK_MINUTES=${SOAK_MINUTES:-30} # nightly only +RUN_TS=$(date -u +%Y%m%d-%H%M%S) +GIT_SHA=$(git rev-parse --short HEAD) +RESULTS=test/results +REPORT=$RESULTS/olla-validate-$RUN_TS.md +LOGDIR=$RESULTS/logs/olla-validate-$RUN_TS +mkdir -p "$LOGDIR" +``` + +Check ports 19431–19437, 41141–41142 are free (`netstat -ano | grep ` or +curl probe); if occupied, kill stale ollamock/olla-validate processes from a +previous run, otherwise abort with a clear message. + +Track every PID you start. **Teardown (Phase 7) must always run**, including on +abort. + +## Phase 1 - Static gate (fail fast) + +Both modes: + +```bash +make ready 2>&1 | tee "$LOGDIR/make-ready.log" +``` + +Nightly additionally: + +```bash +make test 2>&1 | tee "$LOGDIR/make-test.log" +make test-stress 2>&1 | tee "$LOGDIR/make-test-stress.log" +``` + +Any failure here → **abort**: skip the live phases, go straight to Phase 8 and +write a FAILED report quoting the failing output. Do not "fix and continue" - +this skill is a gate, not a repair loop. Report failures; the user decides. + +## Phase 2 - Build + +```bash +EXE=$(go env GOEXE) +go build -o "build/validate/olla$EXE" . +go build -o "build/validate/ollamock$EXE" ./test/cmd/ollamock +``` + +## Phase 3 - Boot the fleet + +Start each process with the Bash tool in background mode, logging to `$LOGDIR`: + +```bash +build/validate/ollamock$EXE --addr 127.0.0.1:19431 --name mock-a --models "test-model,shared-model" > "$LOGDIR/mock-a.log" 2>&1 +build/validate/ollamock$EXE --addr 127.0.0.1:19432 --name mock-b --models "test-model,beta-model" > "$LOGDIR/mock-b.log" 2>&1 +build/validate/ollamock$EXE --addr 127.0.0.1:19433 --name mock-c --models "llama3.1:8b,shared-model" > "$LOGDIR/mock-c.log" 2>&1 +build/validate/ollamock$EXE --addr 127.0.0.1:19434 --name mock-d --models "phi-4,shared-model" > "$LOGDIR/mock-d.log" 2>&1 +build/validate/ollamock$EXE --addr 127.0.0.1:19435 --name mock-e --models "test-model,shared-model" > "$LOGDIR/mock-e.log" 2>&1 +build/validate/ollamock$EXE --addr 127.0.0.1:19436 --name mock-f --models "test-model,beta-model" > "$LOGDIR/mock-f.log" 2>&1 +build/validate/ollamock$EXE --addr 127.0.0.1:19437 --name mock-g --models "phi-4,shared-model" > "$LOGDIR/mock-g.log" 2>&1 +build/validate/olla$EXE --config test/validate/config.validate.yaml > "$LOGDIR/olla-main.log" 2>&1 +build/validate/olla$EXE --config test/validate/config.validate.limits.yaml > "$LOGDIR/olla-limits.log" 2>&1 +``` + +Readiness gates (poll up to 60s, 1s interval; abort to teardown on timeout): + +1. Each mock: `curl -sf http://127.0.0.1:194NN/health` +2. Olla main: `curl -sf http://127.0.0.1:41141/internal/health` +3. Olla limits: `curl -sf http://127.0.0.1:41142/internal/health` +4. All 7 endpoints healthy: poll `/internal/status/endpoints` until every entry reports healthy/routable +5. Model discovery done: poll `/olla/models` until it lists `shared-model` + +Record boot time in the report. If any process dies during boot, capture the +tail of its log into the report. + +## Phase 4 - Wave 1: parallel happy-path agents + +Spawn **five agents in parallel** (single message, multiple Agent calls, +`subagent_type: general-purpose`, `model` per the table below). Each agent +prompt must contain: + +- The mode (`quick` or `nightly`) and instruction to execute the matching + checklist in its area file (path below) - read the file first. +- The topology table above (URLs, ports, which mock backs which endpoint). +- The reporting contract: *"Return ONLY a JSON object: + `{"area":"","pass":N,"fail":N,"warn":N,"skip":N,"failures":[{"check":"","expected":"","actual":"","evidence":""}],"warnings":[...],"notes":""}`. + Every checklist item is exactly one of pass/fail/warn/skip. Use warn for + behaviour that works but looks off; never silently skip - record skips with a + reason in notes. Do not modify any repo files. Do not kill or restart any + process. Do not call any `/_mock/behaviour` endpoint unless your area file + explicitly says to."* + +| Agent | Area file | Model | +|---|---|---| +| core-routing | `.claude/skills/olla-validate/areas/core-routing.md` | haiku | +| openai-api | `.claude/skills/olla-validate/areas/openai-api.md` | haiku | +| anthropic (passthrough section) | `.claude/skills/olla-validate/areas/anthropic.md` | sonnet | +| observability | `.claude/skills/olla-validate/areas/observability.md` | haiku | +| limits-failures (client section - 41142 only) | `.claude/skills/olla-validate/areas/limits-failures.md` | haiku | + +Wave 1 agents are read-only against the mocks: **no fault injection**, so they +can run concurrently without poisoning each other's assertions. + +## Phase 5 - Wave 2: resilience agent (solo) + +After wave 1 completes, `POST /_mock/reset` to all seven mocks (ports 19431-19437), confirm all +endpoints healthy again, then spawn **one** agent (`model: sonnet`) for +`.claude/skills/olla-validate/areas/resilience.md` (mode-appropriate +checklist). This agent **is** allowed to inject faults via `/_mock/behaviour` +and (nightly) to ask you to kill/restart a mock - for process kill/restart +steps the agent reports back what it needs and you perform the kill/restart +yourself, or simpler: give the agent the PID table and let it use taskkill/kill +itself, then verify afterwards that all seven mocks are running again (restart +any that are not, from the same command lines as Phase 3). + +After wave 2: reset all mock behaviours, re-confirm all endpoints return to +healthy within 60s (this is itself a recovery assertion - record it; health +probes tick globally every 30s regardless of per-endpoint check_interval). + +## Phase 6 - Nightly-only extended passes (sequential) + +Skip this phase entirely in quick mode. + +### 6a. Upstream-failure section of limits-failures +Spawn the limits-failures agent again (`model: sonnet` - it mutates mock-a +behaviour), pointing it at the **upstream section** of its area file (runs +solo). Reset mocks after. + +### 6b. Translation-forced Anthropic pass +```bash +sed 's/passthrough_enabled: true/passthrough_enabled: false/' \ + test/validate/config.validate.yaml > "$LOGDIR/config.translation.yaml" +``` +Restart olla-main with that config, wait for readiness (Phase 3 gates 2/4/5), +zero mock stats (`POST /_mock/reset` on all), then spawn the anthropic agent +(`model: sonnet`) against the **translation-forced section** of its area file. Restart olla-main +on the original config afterwards and re-confirm readiness. + +### 6c. Sherpa engine pass +```bash +sed 's/engine: "olla"/engine: "sherpa"/' \ + test/validate/config.validate.yaml > "$LOGDIR/config.sherpa.yaml" +``` +Restart olla-main on it, wait for readiness, then run **quick** checklists of +core-routing, openai-api and anthropic (passthrough) - three parallel agents +(haiku, haiku, sonnet respectively, as in wave 1). +Sherpa is maintenance-mode: quick depth is deliberate. Restore the original +config and readiness afterwards. + +### 6d. Soak + chaos +Run for `$SOAK_MINUTES` minutes against olla-main (olla engine, original +config): + +- Background load: a loop issuing ~5 req/s mixed traffic (non-stream chat, + streaming chat, anthropic messages, /olla/models) - a small bash loop is + fine; log status-code counts. +- Every 60s sample `/internal/process` (goroutines, heap) to + `$LOGDIR/soak-samples.jsonl`. +- Chaos: every ~3 minutes pick one mock, inject `fail_health` or `mode=error`, + hold for 60s, then reset. (Health probes tick globally every 30s, so + shorter faults may never be observed by the checker.) Never fault more than + one mock at a time, and never mock-a and mock-b simultaneously (they are + the only openai-compatible-typed pair). + +Pass criteria: zero client-visible 5xx for requests issued while at least one +compatible backend was healthy and un-faulted (allow a small transition window +of ≤5s after each injection); goroutine count at end within 25% of the +pre-soak baseline after a 60s settle; heap not monotonically growing across +the final third of samples. Anything outside → FAIL with the samples quoted. + +### 6e. Bench snapshot +```bash +make bench-balancer 2>&1 | tee "$LOGDIR/bench-balancer.log" +``` +Record results in the report (informational - WARN if any benchmark fails to +run, never FAIL on numbers). + +## Phase 7 - Teardown (always) + +Kill every started PID (olla instances first, then mocks). On Windows Git +Bash, `kill ` works for processes you started; fall back to +`taskkill //F //PID ` if needed. Verify ports are released. Remove +`build/validate/` binaries only if the user's tree was clean of them before. + +## Phase 8 - Report & verdict + +Aggregate every agent JSON plus the phase-level checks you performed yourself +(gates, boot, recovery-after-reset, soak) into `$REPORT`: + +```markdown +# Olla Validation - - +- Commit: Branch: Engine passes: olla[, sherpa] +- Verdict: PASS | FAIL +- Totals: P/F/W/S + +| Area | Pass | Fail | Warn | Skip | +|---|---|---|---|---| +... + +## Failures + + +## Warnings / Notes / Soak samples summary / Bench snapshot +``` + +Append one line to `$RESULTS/last-runs.md` +(create with a header if missing): + +``` + | olla-validate () | PASS|FAIL |

P/F/W | | report: +``` + +**Verdict rule:** any FAIL anywhere (including the Phase 1 gate, boot, soak or +a dead agent) → overall FAIL. Warnings never fail the gate but must all appear +in the report. Finish by telling the user the verdict, the totals, the three +most important findings in plain sentences, and the report path. diff --git a/.claude/skills/olla-validate/areas/anthropic.md b/.claude/skills/olla-validate/areas/anthropic.md new file mode 100644 index 00000000..3dd8844a --- /dev/null +++ b/.claude/skills/olla-validate/areas/anthropic.md @@ -0,0 +1,68 @@ +# Area: anthropic + +Validates the Anthropic Messages API surface: passthrough to natively capable +backends and (nightly) the forced translation path. Target: olla-main on +`http://127.0.0.1:41141`. **Read-only - never call `POST /_mock/behaviour`.** +`GET /_mock/stats` is allowed. + +The orchestrator tells you which section to run: **passthrough** (default +config, `passthrough_enabled: true`) or **translation-forced** (Olla restarted +with `passthrough_enabled: false`). Run only that section. + +Standard body: +`{"model":"test-model","max_tokens":32,"messages":[{"role":"user","content":"ping"}]}` +Headers: `Content-Type: application/json`, `x-api-key: validate`, +`anthropic-version: 2023-06-01`. + +## Passthrough section + +### Quick checklist + +1. `POST /olla/anthropic/v1/messages` (non-stream) → 200; + `type:"message"`, `role:"assistant"`, `content[0].type:"text"` with + non-empty text containing `BACKEND:`; `stop_reason` set; + `usage.input_tokens > 0` and `usage.output_tokens > 0`. +2. The same response carries `X-Olla-Mode: passthrough` plus the standard + `X-Olla-Endpoint` / `X-Olla-Request-ID` headers, and the serving endpoint + is an anthropic-capable one (not mock-litellm-f; litellm goes via + translation). +3. Streaming (`"stream":true`) → 200 `text/event-stream`; events arrive in + valid order: `message_start` → `content_block_start` → + `content_block_delta`(×N) → `content_block_stop` → `message_delta` → + `message_stop`; deltas assemble to non-empty text. +4. `GET /olla/anthropic/v1/models` → 200, Anthropic-format model list + (non-empty `data[]`). +5. `POST /olla/anthropic/v1/messages/count_tokens` with the standard body → + 200 with `input_tokens > 0`. +6. Invalid body (`{"model":"test-model"}` - no messages/max_tokens) → 4xx + with an Anthropic-style error object (`type:"error"` or similar); FAIL on + 5xx or hang. +7. Sticky on the translator route: two requests with + `X-Olla-Session-ID: validate-anthropic-1` → miss then hit, same endpoint. + +### Nightly additions + +8. Confirm wire-level passthrough via mock stats: note `/_mock/stats` before + and after a burst of 5 messages - the serving mock's `/v1/messages` count + rises and its `/v1/chat/completions` count does not (for those requests). +9. 20 parallel non-stream messages → all 200, all passthrough. +10. Streaming with a multi-block conversation (system + 3 user/assistant + turns) → valid event stream. +11. `/internal/stats/translators` → anthropic translator present; passthrough + counter consistent with the traffic you sent (record numbers). + +## Translation-forced section (nightly; orchestrator restarted Olla with passthrough_enabled: false and reset mock stats) + +1. Non-stream message → 200 with a **valid Anthropic-shape** response + (`type:"message"`, `content[0].text` non-empty, `usage` populated) even + though the backend spoke OpenAI. +2. `X-Olla-Mode` is absent or not `passthrough`. +3. Wire-level proof via `/_mock/stats`: after your burst, the serving mock's + `/v1/chat/completions` count rose; `/v1/messages` did not. +4. Streaming → 200 with a syntactically valid Anthropic SSE event sequence + (same ordering rules as passthrough check 3) synthesised from OpenAI + chunks; assembled text non-empty. +5. `count_tokens` and `/olla/anthropic/v1/models` still work (200). +6. `/internal/stats/translators` shows translation (non-passthrough) counts + rising; record passthrough/translation split and any fallback reasons. +7. Invalid body → 4xx Anthropic-style error (no 5xx). diff --git a/.claude/skills/olla-validate/areas/core-routing.md b/.claude/skills/olla-validate/areas/core-routing.md new file mode 100644 index 00000000..eb430251 --- /dev/null +++ b/.claude/skills/olla-validate/areas/core-routing.md @@ -0,0 +1,73 @@ +# Area: core-routing + +Validates the proxy core: provider-scoped routes, model routing, response +headers, load balancing and sticky sessions. Target: olla-main on +`http://127.0.0.1:41141`. **Read-only - never call `/_mock/behaviour`.** + +Expected backend ownership (assert via `X-Olla-Endpoint` header and the +`BACKEND:` marker in response content): + +| Route prefix | Eligible endpoints | Marker(s) | +|---|---|---| +| `/olla/proxy/` | all seven | any | +| `/olla/openai/`, `/olla/openai-compatible/` | inclusive **by design**: any endpoint whose profile declares OpenAI compatibility; with model `test-model` that narrows to mock-openai-a/b, mock-vllm-e, mock-litellm-f | mock-a, mock-b, mock-e, mock-f | +| `/olla/ollama/` | mock-ollama-c | mock-c | +| `/olla/lmstudio/` (+ `/olla/lm-studio/`, `/olla/lm_studio/`) | mock-lmstudio-d | mock-d | +| `/olla/vllm/` | mock-vllm-e | mock-e | +| `/olla/litellm/` | mock-litellm-f | mock-f | +| `/olla/llamacpp/` | mock-llamacpp-g | mock-g | + +Standard chat body (OpenAI-shaped routes): +`{"model":"test-model","messages":[{"role":"user","content":"ping"}],"max_tokens":20,"stream":false}` +Ollama route body (`/olla/ollama/api/chat`): +`{"model":"llama3.1:8b","messages":[{"role":"user","content":"ping"}],"stream":false}` + +## Quick checklist + +1. `POST /olla/proxy/v1/chat/completions` → 200; body contains `BACKEND:`; + headers `X-Olla-Endpoint`, `X-Olla-Model`, `X-Olla-Backend-Type`, + `X-Olla-Request-ID`, `X-Olla-Response-Time` all present and non-empty. +2. Each provider route in the table above → 200 AND the serving endpoint is in + that route's eligible set (both header and body marker agree). For ollama + use the native body/path. One request per route is enough. +3. Provider aliases: `/olla/lm-studio/v1/chat/completions` and + `/olla/lm_studio/v1/chat/completions` both 200 from mock-d. +4. `X-Olla-Backend-Type` matches the provider for at least the ollama, + lm-studio and vllm routes. +5. Model routing on `/olla/proxy/`: model `llama3.1:8b` → served by + mock-ollama-c; model `phi-4` → served by mock-d or mock-g (the only + hosts). If `X-Olla-Routing-*` + headers are present, record their values; FAIL only if the request lands on + an endpoint that does not serve the model. +6. Sticky: two POSTs to `/olla/proxy/v1/chat/completions` with header + `X-Olla-Session-ID: validate-core-1` → turn 1 `X-Olla-Sticky-Session: miss` + + `X-Olla-Sticky-Key-Source: session_header`; turn 2 `hit` + same + `X-Olla-Endpoint` + same body marker. `X-Olla-Session-ID` echoed back. +7. Distribution sanity: 10 requests to `/olla/openai-compatible/...` with + *unique* session IDs → at least 2 distinct endpoints observed (candidates + for `test-model`: mock-a/b/e/f). WARN (not FAIL) if all 10 land on one. +8. `GET /olla/nonexistent/v1/models` → 404 (clean error, no hang). + +## Nightly additions + +9. Repeat check 2 with `"stream":true` on every route (ollama native streams + NDJSON; others SSE) - each stream completes with its terminator and carries + the right backend marker. +10. Model `shared-model` 15× on `/olla/proxy/` → only ever served by + mock-a / mock-c / mock-d / mock-e / mock-g (never mock-b or mock-f, + which don't host it). +11. Model `beta-model` 5× → only mock-b or mock-f. +12. Sticky prefix_hash fallback: two identical POSTs (same messages, no + session header, no auth header) → turn 2 `X-Olla-Sticky-Session: hit` with + `X-Olla-Sticky-Key-Source: prefix_hash`. +13. Balance distribution: 100 unique-session requests on + `/olla/openai-compatible/` with model `test-model` → at least 3 of the + four candidates (mock-a/b/e/f) receive traffic and no single endpoint + serves more than 60%. +14. Concurrency: 50 parallel POSTs to `/olla/proxy/` → all 200, all carry + unique `X-Olla-Request-ID` values. +15. `GET /olla/proxy/v1/models` → 200, contains `test-model`, `shared-model`, + `beta-model`, `llama3.1:8b`, `phi-4`. +16. Cross-check `/_mock/stats` on each mock (`GET` only - allowed) against + where you sent traffic: every mock you observed in markers shows non-zero + counts on the paths you exercised. diff --git a/.claude/skills/olla-validate/areas/limits-failures.md b/.claude/skills/olla-validate/areas/limits-failures.md new file mode 100644 index 00000000..3a43a3a7 --- /dev/null +++ b/.claude/skills/olla-validate/areas/limits-failures.md @@ -0,0 +1,73 @@ +# Area: limits-failures + +Validates request limits, rate limiting and bad-input handling. Two sections - +the orchestrator tells you which to run: + +- **Client section** (wave 1, both modes): targets the dedicated tight-limits + Olla instance on `http://127.0.0.1:41142` (256KB body cap, 30 req/min per-IP, + burst 5, backed by mock-a). **No `/_mock/behaviour` calls.** Do not hammer + olla-main on 41141 - its rate limits are shared with the other agents. +- **Upstream section** (nightly, wave 3, runs solo): mutates mock-a behaviour + to test how Olla handles a misbehaving upstream. Targets 41142 (short 10s + response timeout makes hang tests fast). Reset mock-a when done. + +Large payloads: build them in a file and send with `curl --data @file` - +inlining hundreds of KB on the command line fails with "argument list too +long" on Git Bash/Windows. + +## Client section + +### Quick checklist + +1. Baseline: `POST http://127.0.0.1:41142/olla/proxy/v1/chat/completions` + with the standard small body → 200. +2. Body cap, declared length: same route with a ~300KB body (pad one message + with filler) → rejected. Current contract: the security chain rejects a + declared oversized Content-Length with a blanket **403** ("Security + validation failed", `internal/app/handlers/application.go`); 413 is also + acceptable. Any 2xx or 5xx = FAIL. +3. Just-under cap (~200KB) → 200. +4. 429 rate limit: fire 45 rapid sequential requests → at least one 429; + record at which request it first appears (expect after roughly + burst+window allowance) and whether a Retry-After or rate-limit header is + present (note, don't fail on header absence). +5. Health exemption: while rate-limited, `GET /internal/health` on 41142 + still 200 (health has its own generous limit). +6. Malformed JSON (`{"model":`) → 4xx, not 5xx. +7. Empty body POST → 4xx, not 5xx. +8. Wrong method: `DELETE /olla/proxy/v1/chat/completions` → 404/405, no 5xx. + +### Nightly additions + +9. Oversized **chunked** body (no Content-Length, stream >256KB chunked; + `curl -H 'Transfer-Encoding: chunked'` with `--data-binary @file`) → + rejected with 413 once the cap is crossed mid-read (recent hardening; + regression guard - this path must NOT be a 403, it is detected during + streaming). +10. Header size: a single ~80KB header (cap 64KB) → 431 or connection + rejection - record actual; 5xx or hang = FAIL. +11. Anthropic message size: `POST /olla/anthropic/v1/messages` on 41142 with + a ~2MB message (translator cap 1MB) → 4xx, record the status. +12. 429 recovery: after check 4, wait ~65s → requests succeed again. +13. Burst parallelism: 20 parallel requests immediately after recovery → + mix of 200s and 429s, never 5xx, olla-limits stays healthy. + +## Upstream section (nightly, solo) + +1. Malformed upstream JSON: set `{"malformed_json":true}` on mock-a → + `POST 41142 /olla/proxy/v1/chat/completions`: record what the client gets + (passthrough of garbage or a 502 are both observable outcomes - note + which); olla must not crash and `/internal/health` stays 200. Reset. +2. Upstream hang: `{"mode":"hang","latency_ms":60000}` on mock-a → request + fails within ~15s (10s response timeout + margin), client receives a + gateway-style error, no goroutine leak across 10 repeats + (`/internal/process` before/after within 25%). Reset. +3. Upstream slow first byte: `{"mode":"slow","latency_ms":3000}` → request + succeeds (3s < 5s response_header_timeout on 41142); record latency. + Then `{"mode":"slow","latency_ms":8000}` → fails with a timeout error, + not a hang. Reset. +4. Upstream 503 with `fail_health` false: `{"mode":"error","error_status":503}` + → 5xx surfaces to client (current contract), endpoint not flapped into + permanent removal - after reset, next request 200 within 10s. +5. Final: `POST /_mock/reset` on mock-a; confirm default behaviour and a + clean 200 on 41142. Mandatory. diff --git a/.claude/skills/olla-validate/areas/observability.md b/.claude/skills/olla-validate/areas/observability.md new file mode 100644 index 00000000..81b335cb --- /dev/null +++ b/.claude/skills/olla-validate/areas/observability.md @@ -0,0 +1,51 @@ +# Area: observability + +Validates internal/status/stats endpoints, the unified model registry and +version info. Target: olla-main on `http://127.0.0.1:41141`. **Read-only - +never call `/_mock/behaviour`.** Other wave-1 agents are generating traffic +concurrently, so assert structure and plausibility, not exact counts. + +## Quick checklist + +1. `GET /internal/health` → 200, valid JSON. +2. `GET /internal/status` → 200, valid JSON; FAIL on any 5xx (this endpoint + has a known history of panics - treat errors here as serious). +3. `GET /internal/status/endpoints` → 200; exactly 7 endpoints + (mock-openai-a, mock-openai-b, mock-ollama-c, mock-lmstudio-d, + mock-vllm-e, mock-litellm-f, mock-llamacpp-g); all healthy/routable. +4. `GET /internal/status/models` → 200, non-empty. +5. `GET /internal/stats/models` → 200, valid JSON. +6. `GET /internal/stats/sticky` → 200 and `enabled: true` (sticky is on in + the harness config). +7. `GET /internal/stats/translators` → 200, anthropic translator listed. +8. `GET /internal/process` → 200; goroutines > 0, memory stats present. +9. `GET /version` → 200 with version info. +10. `GET /olla/models` → 200; includes a unified entry covering + `shared-model`; that entry (or its detail view) references multiple + endpoints/providers (it is served by mock-a, mock-c and mock-d). +11. `GET /olla/models/` for one unified model id from check 10 → 200 with + the same model; an unknown id → 404. +12. Two requests to `/internal/health` → distinct `X-Olla-Request-ID` values + (if the header is set on internal routes; skip with note if not). + +## Nightly additions + +13. Unifier alias resolution: `llama3.1:8b` resolves via + `/olla/models/llama3.1:8b` (or its documented alias lookup) to a unified + model that lists the ollama endpoint. If alias lookup is not supported on + that route, record actual behaviour as a note, not a FAIL. +14. Consistency: send 10 chat requests yourself to `/olla/proxy/` with model + `test-model`, then confirm `/internal/stats/models` reflects activity for + that model (counter increased vs your earlier reading). +15. `/internal/status/endpoints` per-endpoint stats: endpoints you know + received traffic (cross-reference `GET /_mock/stats`) show non-zero + request counts. +16. Sticky stats: after the core-routing agent has run (it always does in + wave 1), `/internal/stats/sticky` shows `insertions > 0` and `hits > 0`. + If you run before it finishes, generate two same-session requests + yourself first. +17. Malformed queries: `/olla/models?bogus=1` and `/internal/status/` → + no 5xx; sane 200/404. +18. Poll `/internal/process` 5 times over 2 minutes while wave-1 load runs → + goroutine count is bounded (no monotonic climb of >50% across samples); + record the series. diff --git a/.claude/skills/olla-validate/areas/openai-api.md b/.claude/skills/olla-validate/areas/openai-api.md new file mode 100644 index 00000000..88419277 --- /dev/null +++ b/.claude/skills/olla-validate/areas/openai-api.md @@ -0,0 +1,53 @@ +# Area: openai-api + +Validates the OpenAI-compatible API surface and streaming behaviour through +olla-main on `http://127.0.0.1:41141`. **Read-only - never call +`/_mock/behaviour`** (the slow-stream check below uses ollamock's startup +pacing only in nightly via instructions to the orchestrator; if you cannot do +a check without mutating mock state, mark it skip with a note). + +## Quick checklist + +1. `GET /olla/proxy/v1/models` → 200, `object: "list"`, `data[]` non-empty, + ids include `test-model`. +2. Non-stream chat: `POST /olla/proxy/v1/chat/completions` with + `{"model":"test-model","messages":[{"role":"user","content":"say hi"}],"max_tokens":32}` + → 200; `choices[0].message.role == "assistant"`; non-empty content; + `choices[0].finish_reason == "stop"`; `usage.prompt_tokens > 0` and + `usage.completion_tokens > 0`; response `model` echoes the request model. +3. Streaming chat: same body plus `"stream":true` → 200 with + `Content-Type: text/event-stream`; multiple `data:` chunks of + `chat.completion.chunk`; deltas assemble to non-empty content; a chunk with + `finish_reason:"stop"`; terminates with `data: [DONE]`. +4. X-Olla headers present on the streaming response too (they are sent before + the body). +5. Ollama native non-stream: `POST /olla/ollama/api/chat` + (`"stream":false`) → 200, `done: true`, `message.content` non-empty, + `prompt_eval_count`/`eval_count` > 0. +6. Malformed body to `/olla/proxy/v1/chat/completions` (`{"model":`) → 4xx + (FAIL if 5xx or hang). + +## Nightly additions + +7. Ollama native streaming: `POST /olla/ollama/api/chat` with stream + defaulted (omit the field) → NDJSON lines, last line `done:true` with eval + counts. +8. Ollama generate: `POST /olla/ollama/api/generate` + `{"model":"llama3.1:8b","prompt":"ping"}` → streams, final `done:true`. +9. `GET /olla/ollama/api/tags` → 200, models array includes `llama3.1:8b`. +10. `POST /olla/openai/v1/completions` (legacy completions) + `{"model":"test-model","prompt":"ping","max_tokens":16}` → 200 with a + `choices[0].text`. +11. Concurrency under streaming: 20 parallel streaming chats → all complete + with `[DONE]`, none hang past 30s, all content non-empty. +12. Large prompt: single message with ~3MB of text (under the 5MB cap) → + 200 (stream or non-stream). FAIL on 413 (cap is 5MB) or 5xx. Build the + body in a file and send with `curl --data @file` (inline args this big + fail on Git Bash/Windows). +13. Anthropic-format body sent to the OpenAI route (wrong-shape input: + `{"model":"test-model","max_tokens":10,"messages":[{"role":"user","content":[{"type":"text","text":"hi"}]}]}`) + → must not crash Olla; record the status observed (2xx/4xx both + acceptable; 5xx = FAIL). +14. Response-time sanity: median of 20 sequential non-stream requests + < 250ms (mocks are instant; this catches gross proxy regressions). + WARN, not FAIL, if exceeded - machines vary. diff --git a/.claude/skills/olla-validate/areas/resilience.md b/.claude/skills/olla-validate/areas/resilience.md new file mode 100644 index 00000000..a5805454 --- /dev/null +++ b/.claude/skills/olla-validate/areas/resilience.md @@ -0,0 +1,91 @@ +# Area: resilience + +Validates health checking, failover, recovery, sticky repinning and proxy +behaviour under backend failure. Target: olla-main on +`http://127.0.0.1:41141`. **This is the only wave-2 agent: you ARE allowed to +inject faults via `POST /_mock/behaviour` and `POST /_mock/reset`.** + +Rules of engagement: +- Fault at most ONE mock at a time. Never fault mock-a and mock-b together + (they are the only openai-compatible pair - faulting both removes all + failover headroom). +- After EVERY scenario: `POST /_mock/reset` on the faulted mock, then poll + `/internal/status/endpoints` until all 7 endpoints are healthy again before + starting the next scenario. The recovery itself is an assertion each time + (healthy within ~60s of reset = pass). +- Health probe timing: despite `check_interval: 2s` in the config, the health + checker runs on a global 30s ticker (`DefaultHealthCheckInterval` in + `internal/adapter/health/checker.go`), so allow up to **40s** for any + health transition driven by probes. Request-path connection failures mark + an endpoint unhealthy immediately - those are fast. +- `/olla/openai/` and `/olla/openai-compatible/` routes are inclusive by + design: any OpenAI-compatible endpoint may serve them. Failover assertions + on those routes must check "not the faulted endpoint", not a specific + survivor. + +Documented current behaviour (do not "fix" expectations to taste): +- Retry applies to **connection failures** (refused/reset/timeout), not HTTP + responses: an upstream HTTP 5xx with a healthy connection is **forwarded to + the client** (known gap, issue #144). Assert accordingly. +- 429/401/403 from a backend must NOT trip health/circuit breaking. + +## Quick checklist (~3–5 min; dominated by the 30s probe tick) + +1. Baseline: all 7 endpoints healthy in `/internal/status/endpoints`. +2. Health-fail failover: set `{"fail_health":true}` on mock-b (19432) → + within 40s mock-openai-b goes non-routable. While it is down, 10 requests + to `/olla/openai-compatible/v1/chat/completions` → all 200, none served + by mock-b. +3. During the transition window itself, issue requests continuously (one per + 250ms from the moment you set the fault until the endpoint is marked + down): fail_health only fails the probe route, so every request should + still return 200 throughout. Any client error here = FAIL. +4. Recovery: reset mock-b → all endpoints healthy within 60s; subsequent + unique-session requests reach mock-b again (loop until you see its marker, + max 40 tries). +5. Request-time 5xx (healthy connection): set + `{"mode":"error","error_status":500}` on mock-c (19433) → a request to + `/olla/ollama/api/chat` returns 5xx to the client (current contract); + olla-main itself stays healthy (`/internal/health` 200) and other routes + are unaffected. Reset. + +## Nightly additions + +6. Connection-refused failover: ask the orchestrator's PID table - kill the + mock-b process outright. Immediately issue 20 requests to + `/olla/openai-compatible/` → all 200, none from mock-b (retry-on- + connection-failure must hide the dead backend; brief first-request latency + is fine). `/internal/status/endpoints` marks mock-openai-b unhealthy + (request-path failures mark immediately). Restart mock-b with its original + command line; healthy within 60s; traffic returns. +7. Sticky repin: pin session `validate-repin-1` (two turns, note the + endpoint). Fault that endpoint's mock with `fail_health`. Wait for + non-routable, then send turn 3 with the same session → 200 from a + different endpoint and `X-Olla-Sticky-Session` is `repin` (or `miss` - + record which; a 5xx or routing to the dead backend = FAIL). Reset. +8. Rate-limit semantics: set `{"mode":"error","error_status":429}` on mock-d + → requests to `/olla/lmstudio/` surface 429; the endpoint must NOT be + marked unhealthy/offline in `/internal/status/endpoints` (RateLimited is + not a health failure). Reset. Same check with 401 (ConfigError class). +9. Hang/timeout: set `{"mode":"hang","latency_ms":30000}` on mock-c → a + request to `/olla/ollama/api/chat` fails within the proxy response + timeout (60s config; expect an error well before 70s - record actual) and + olla-main remains responsive to other routes throughout (probe every 2s + while waiting). Reset. +10. Mid-stream drop: set `{"drop_mid_stream":true}` on mock-e (19435) → a + streaming request to `/olla/vllm/v1/chat/completions` terminates + (truncated is acceptable; an indefinite hang or olla crash = FAIL); + olla-main healthy afterwards; next streaming request after reset + completes normally. +11. Flaky backend: set `{"mode":"flaky","error_rate":0.5,"error_status":500}` + on mock-e, send 30 requests to `/olla/vllm/` → roughly half fail with + 5xx (current 5xx-forwarding contract), olla stays healthy, no goroutine + runaway (`/internal/process` before/after within 25%). Reset. +12. Repeated kill/restore cycling (mini-chaos): 5 cycles of + fail_health(mock-b) → wait down → reset → wait up (each cycle can take + ~2 min given the 30s probe tick - budget ~10 min). After the 5th cycle, + all endpoints healthy, sticky stats endpoint still serves, goroutines + within 25% of your baseline from check 1. +13. Final state assertion: all seven mocks report default behaviour + (`GET /_mock/behaviour`), all 7 endpoints healthy, `/internal/status` + 200. This is mandatory - you must leave the fleet clean. diff --git a/AGENTS.md b/AGENTS.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/AGENTS.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 74267205..cf001a21 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,279 +1,125 @@ # CLAUDE.md +> Single source of truth for agent guidance. `AGENTS.md` is a symlink to this file - edit here only. + ## Overview -Olla is a high-performance proxy and load balancer for LLM infrastructure, written in Go. It intelligently routes requests across local inference nodes (Ollama, LM Studio, LiteLLM, vLLM, vLLM-MLX, SGLang, llama.cpp, Lemonade, LMDeploy, Docker Model Runner, and OpenAI-compatible endpoints). The Anthropic Messages API is also supported via passthrough or translation. +Olla is a high-performance proxy and load balancer for LLM infrastructure, written in Go. It routes requests across local inference nodes (Ollama, LM Studio, LiteLLM, vLLM, vLLM-MLX, SGLang, llama.cpp, Lemonade, LMDeploy, Docker Model Runner, and OpenAI-compatible endpoints). The Anthropic Messages API is supported via passthrough or translation. -The project provides two proxy engines: Sherpa (simple, maintainable) and Olla (high-performance with advanced features). +Two proxy engines: **Sherpa** (simple, maintenance-mode) and **Olla** (high-performance, where all new engine work goes). Olla is local-only - hosted API providers (Groq, Gemini, etc.) are out of scope. -Full documentation available at: https://thushan.github.io/olla/ +Full documentation: https://thushan.github.io/olla/ ## Commands ```bash -make ready # Run before commit (test-short + test-race + fmt + vet + lint + align) -make ready-tools # Check code with tools only (fmt + vet + lint + align) -make test # Run all tests -make test-race # Run tests with race detection -make test-stress # Run comprehensive stress tests -make bench # Run all benchmarks -make bench-balancer # Run balancer benchmarks -make build # Build optimised binary with version info -make build-local # Build binary to ./build/ (fast, for testing) -make run # Run with version info -make run-debug # Run with debug logging -make docker-build # Build Docker image with goreleaser (requires goreleaser) -make docker-build-local # Build Docker image locally for amd64 (no goreleaser required) -make docker-build-local-arm64 # Build Docker image locally for ARM64 (no goreleaser required) -make docker-run # Run Docker image with local config (amd64) -make ci # Run full CI pipeline locally -make help # Show all available targets +make ready # Pre-commit gate: test-short + test-race + fmt + vet + lint + align +make ready-tools # Tools only: fmt + vet + lint + align +make test # All tests make test-race # With race detection +make test-stress # Stress tests make bench # All benchmarks +make build # Optimised binary make build-local # Fast build to ./build/ +make run # Run make run-debug # Run with debug logging +make docker-build-local # Build amd64 image locally (no goreleaser) +make docker-build-local-arm64 # Build ARM64 image locally (no goreleaser) +make ci # Full CI pipeline locally +make help # All targets ``` +Always run `make ready` before reporting work complete or committing. -## Project Structure -``` -olla/ -├── main.go # Entry point, initialises services -├── config.yaml # Default configuration -├── config/ -│ ├── profiles/ # Provider-specific profiles -│ │ ├── ollama.yaml # Ollama configuration -│ │ ├── llamacpp.yaml # llama.cpp configuration -│ │ ├── lmstudio.yaml # LM Studio configuration -│ │ ├── lemonade.yaml # Lemonade SDK configuration -│ │ ├── litellm.yaml # LiteLLM gateway configuration -│ │ ├── lmdeploy.yaml # LMDeploy configuration -│ │ ├── vllm.yaml # vLLM configuration -│ │ ├── vllm-mlx.yaml # vLLM-MLX (Apple Silicon) configuration -│ │ ├── sglang.yaml # SGLang configuration -│ │ ├── dmr.yaml # Docker Model Runner configuration -│ │ └── openai-compatible.yaml # OpenAI-compatible generic profile (type: "openai" is an alias) -│ ├── models.yaml # Model configurations -│ └── config.local.yaml # Local configuration overrides (user, not committed to git) -├── internal/ -│ ├── core/ # Domain layer (business logic) -│ │ ├── domain/ # Core entities -│ │ ├── ports/ # Interface definitions -│ │ └── constants/ # Application constants -│ ├── adapter/ # Infrastructure layer -│ │ ├── balancer/ # Load balancing strategies -│ │ ├── converter/ # Model format converters -│ │ ├── discovery/ # Service discovery -│ │ ├── factory/ # Factory patterns -│ │ ├── filter/ # Request/response filtering -│ │ ├── health/ # Health checking & circuit breakers -│ │ ├── inspector/ # Request inspection -│ │ ├── metrics/ # Metrics collection -│ │ ├── proxy/ # Proxy implementations -│ │ │ ├── sherpa/ # Simple, maintainable proxy -│ │ │ ├── olla/ # High-performance proxy -│ │ │ └── core/ # Shared proxy components -│ │ ├── registry/ # Model & profile registries -│ │ ├── security/ # Security features (rate/size limits) -│ │ ├── stats/ # Statistics collection -│ │ ├── translator/ # API translation layer (OpenAI ↔ Provider) -│ │ └── unifier/ # Model unification -│ ├── app/ # Application layer -│ │ ├── handlers/ # HTTP handlers -│ │ │ ├── server.go # HTTP server setup -│ │ │ ├── server_routes.go # Route registration -│ │ │ ├── handler_proxy.go # Main proxy handler -│ │ │ ├── handler_provider_*.go # Provider-specific handlers -│ │ │ ├── handler_translation.go # Translation handler -│ │ │ ├── handler_status*.go # Status endpoints -│ │ │ ├── handler_health.go # Health endpoints -│ │ │ └── handler_version.go # Version information -│ │ ├── middleware/ # HTTP middleware -│ │ └── services/ # Application services -│ ├── config/ # Configuration management -│ ├── env/ # Environment handling -│ ├── integration/ # Integration tests -│ ├── logger/ # Logging framework -│ ├── router/ # Routing logic -│ ├── util/ # Utilities -│ └── version/ # Version management -├── pkg/ # Reusable packages -│ ├── container/ # Dependency injection -│ ├── eventbus/ # Event bus (pub/sub) -│ ├── format/ # Formatting utilities -│ ├── nerdstats/ # Process statistics -│ ├── pool/ # Object pooling -│ └── profiler/ # Profiling support -└── test/ - └── scripts/ # Test scripts - ├── auth/ # Authentication tests - ├── cases/ # Test cases - ├── load/ # Load testing - ├── logic/ # Logic & routing tests - ├── platform/ # Platform-specific tests - ├── security/ # Security tests - └── streaming/ # Streaming tests +Specific test patterns: +```bash +go test -v ./internal/adapter/proxy -run TestAllProxies # or TestSherpa / TestOlla ``` -## Key Files -- `main.go` - Application entry point -- `config.yaml` - Main configuration -- `internal/app/handlers/server_routes.go` - Route registration & API setup -- `internal/app/handlers/handler_proxy.go` - Request routing logic -- `internal/app/handlers/handler_translation.go` - Translation handler with passthrough logic -- `internal/adapter/proxy/sherpa/service.go` - Sherpa proxy implementation -- `internal/adapter/proxy/olla/service.go` - Olla proxy implementation -- `internal/adapter/translator/` - API translation layer (OpenAI ↔ Provider formats) -- `internal/adapter/translator/types.go` - PassthroughCapable interface and translator types -- `internal/adapter/translator/anthropic/` - Anthropic translator implementation -- `internal/adapter/stats/translator_collector.go` - Translator metrics collector -- `internal/adapter/balancer/sticky.go` - Sticky session wrapper -- `internal/app/handlers/handler_stats_sticky.go` - Sticky session stats endpoint +## Project Structure +Hexagonal architecture - domain (`internal/core`) → infrastructure (`internal/adapter`) → application (`internal/app`): + +- `main.go` - entry point +- `config.yaml` - default config; `config/config.local.yaml` - user overrides (gitignored) +- `config/profiles/*.yaml` - one profile per backend type (ollama, vllm, vllm-mlx, sglang, llamacpp, lemonade, litellm, lmdeploy, dmr, openai-compatible). `type: "openai"` aliases `openai-compatible`. +- `config/models.yaml` - model configuration +- `internal/core/` - domain layer: `domain/` (entities), `ports/` (interfaces), `constants/` +- `internal/adapter/` - infrastructure: `balancer/`, `health/` (+ circuit breakers), `proxy/` (`sherpa/`, `olla/`, shared `core/`), `registry/`, `translator/` (OpenAI ↔ provider), `unifier/`, `stats/`, `security/`, `discovery/`, `inspector/`, `metrics/`, `converter/`, `filter/` +- `internal/app/` - `handlers/` (HTTP), `middleware/`, `services/` +- `internal/` (other) - `config/`, `router/`, `logger/`, `version/`, `integration/`, `env/`, `util/` +- `pkg/` - reusable: `container/` (DI), `eventbus/` (pub/sub), `pool/` (object pooling), `format/`, `nerdstats/`, `profiler/` +- `test/cmd/ollamock/` - multi-protocol mock backend with fault injection; `test/cmd/mockbackend/` - minimal auth-test backend +- `test/validate/` - `/olla-validate` harness configs; `test/scripts/` - e2e scenarios (auth, load, logic, platform, security, streaming) + +### Key Files +- `internal/app/handlers/server_routes.go` - route registration & API setup +- `internal/app/handlers/handler_proxy.go` - request routing logic +- `internal/app/handlers/handler_translation.go` - translation handler with passthrough logic +- `internal/adapter/proxy/{sherpa,olla}/service.go` - proxy engine implementations +- `internal/adapter/translator/` - translation layer; `types.go` (PassthroughCapable interface), `anthropic/` (impl) +- `internal/adapter/stats/translator_collector.go` - translator metrics collector +- `internal/adapter/balancer/sticky.go` - sticky session wrapper - `internal/core/constants/translator.go` - TranslatorMode and FallbackReason constants -- `internal/core/ports/stats.go` - StatsCollector interface with translator tracking +- `internal/core/ports/stats.go` - StatsCollector interface (incl. translator tracking) - `internal/core/domain/profile_config.go` - AnthropicSupportConfig for backend profiles -- `config/profiles/*.yaml` - Backend profiles with `anthropic_support` sections -- `internal/version/version.go` - Version information embedded at build time -- `/test/scripts/logic/test-model-routing.sh` - Test routing & headers +- `internal/version/version.go` - build-time version info +- `test/scripts/logic/test-model-routing.sh` - routing & header tests ## API Endpoints +**Internal** (`/internal/...`): `health`, `status`, `status/endpoints`, `status/models`, `stats/models`, `stats/translators`, `stats/sticky` (`{"enabled":false}` when off), `process`. Plus `/version`. -### Internal Endpoints -- `/internal/health` - Health check endpoint -- `/internal/status` - Endpoint status -- `/internal/status/endpoints` - Endpoints status details -- `/internal/status/models` - Models status details -- `/internal/stats/models` - Model statistics -- `/internal/stats/translators` - Translator statistics -- `/internal/stats/sticky` - Sticky session statistics (returns `{"enabled":false}` when disabled) -- `/internal/process` - Process statistics -- `/version` - Version information - -### Unified Model Endpoints -- `/olla/models` - Unified models listing with filtering -- `/olla/models/{id}` - Get unified model by ID or alias +**Unified models**: `/olla/models` (listing with filtering), `/olla/models/{id}` (by ID or alias). -### Proxy Endpoints -- `/olla/proxy/` - Olla API proxy endpoint (POST) -- `/olla/proxy/v1/models` - OpenAI-compatible models listing (GET) +**Proxy**: `/olla/proxy/` (POST), `/olla/proxy/v1/models` (GET, OpenAI-compatible). -### Provider Proxy Prefixes -Profile-driven per-backend namespaces (see `internal/app/handlers/server_routes.go`): -- `/olla/ollama/` -- `/olla/lmstudio/` (also `/olla/lm-studio/`, `/olla/lm_studio/`) -- `/olla/vllm/` -- `/olla/vllm-mlx/` -- `/olla/sglang/` -- `/olla/lmdeploy/` -- `/olla/llamacpp/` -- `/olla/lemonade/` -- `/olla/litellm/` -- `/olla/dmr/` -- `/olla/openai/` and `/olla/openai-compatible/` (both served by `openai-compatible.yaml`) +**Provider prefixes** (profile-driven, see `server_routes.go`): `/olla/{ollama,vllm,vllm-mlx,sglang,lmdeploy,llamacpp,lemonade,litellm,dmr}/`, `/olla/lmstudio/` (+ `lm-studio`, `lm_studio` aliases), `/olla/openai/` and `/olla/openai-compatible/` (both served by `openai-compatible.yaml`). -### Translator Endpoints -Dynamically registered based on configured translators (e.g., Anthropic Messages API) - -- `/olla/anthropic/v1/messages` - Anthropic Messages API (POST) - supports passthrough and translation modes -- `/olla/anthropic/v1/models` - List models in Anthropic format (GET) -- `/olla/anthropic/v1/messages/count_tokens` - Token count estimation (POST) +**Translator** (dynamically registered): `/olla/anthropic/v1/messages` (POST, passthrough + translation), `/olla/anthropic/v1/models` (GET), `/olla/anthropic/v1/messages/count_tokens` (POST). ## Response Headers -- `X-Olla-Endpoint`: Backend name -- `X-Olla-Model`: Model used -- `X-Olla-Backend-Type`: ollama, lm-studio, litellm, vllm, vllm-mlx, sglang, llamacpp, lmdeploy, lemonade, openai, openai-compatible, docker-model-runner, omlx -- `X-Olla-Request-ID`: Request ID -- `X-Olla-Response-Time`: Total processing time -- `X-Olla-Mode`: Translator mode used (`passthrough` or absent for translation) - set on Anthropic translator requests -- `X-Olla-Routing-Strategy`: Routing strategy used (when model routing is active) -- `X-Olla-Routing-Decision`: Routing decision made (routed/fallback/rejected) -- `X-Olla-Routing-Reason`: Human-readable reason for routing decision -- `X-Olla-Sticky-Session`: Sticky session status (hit/miss/repin/disabled) -- `X-Olla-Sticky-Key-Source`: Key source used (session_header/prefix_hash/auth_header/ip/none) -- `X-Olla-Session-ID`: Echoed session ID when client supplies one - -## Testing - -### Testing Strategy -1. **Unit Tests**: Components in isolation -2. **Integration Tests**: Full request flow through proxy engines -3. **Benchmark Tests**: Performance comparison (balancers, proxy engines, repositories) -4. **Security Tests**: Rate limiting and size restrictions (see `/test/scripts/security/`) -5. **Stress Tests**: Comprehensive testing under load -6. **Script Tests**: End-to-end scenarios in `/test/scripts/` - -### Testing Commands -```bash -# Core test commands -make test # Run all tests -make test-race # Run with race detection -make test-stress # Run stress tests -make test-cover-html # Generate coverage HTML report +`X-Olla-Endpoint` (backend name), `X-Olla-Model`, `X-Olla-Backend-Type` (ollama, lm-studio, litellm, vllm, vllm-mlx, sglang, llamacpp, lmdeploy, lemonade, openai, openai-compatible, docker-model-runner, omlx), `X-Olla-Request-ID`, `X-Olla-Response-Time`, `X-Olla-Mode` (`passthrough` or absent for translation), `X-Olla-Routing-{Strategy,Decision,Reason}`, `X-Olla-Sticky-Session` (hit/miss/repin/disabled), `X-Olla-Sticky-Key-Source`, `X-Olla-Session-ID`. -# Benchmark commands -make bench # Run all benchmarks -make bench-balancer # Run balancer benchmarks -make bench-repo # Run repository benchmarks +## Architecture Notes +- **Translator layer** - API format translation (e.g. OpenAI ↔ Anthropic) with passthrough optimisation. **Passthrough mode**: when a backend natively supports the Anthropic Messages API (vLLM, llama.cpp, LM Studio, Ollama), requests bypass translation entirely. +- **Translator metrics** - thread-safe per-translator stats (passthrough/translation rates, fallback reasons, latency, streaming breakdown) in `stats/translator_collector.go`. +- **Sticky sessions** - optional decorator on the endpoint selector that pins multi-turn conversations to the backend that handled the first turn, maximising KV-cache reuse. 64-bit FNV-1a hashed keys, TTL + LRU bounded, purged on routable→non-routable health transitions (`balancer/sticky.go`). +- **Load balancing** - priority-based recommended for production. +- **Version management** - build-time injection via `internal/version`. -# Specific test patterns -go test -v ./internal/adapter/proxy -run TestAllProxies -go test -v ./internal/adapter/proxy -run TestSherpa -go test -v ./internal/adapter/proxy -run TestOlla -``` +## Testing Strategy +Unit (isolation) · integration (full request flow through engines) · benchmark (balancers, engines, repos) · security (rate/size limits, `test/scripts/security/`) · stress (under load) · script (e2e, `test/scripts/`). -Always run `make ready` before committing changes. +### Validation Harness (`/olla-validate`) +Agent-driven end-to-end validation against mock backends - no Docker, no real inference, CI-safe. Defined in `.claude/skills/olla-validate/`. -## Architecture Notes +- `/olla-validate --quick` - 5–10 min gate after major changes +- `/olla-validate --nightly` - multi-hour pre-release gate (chaos, soak, Sherpa pass, forced-translation pass, benchmarks) +- No flag - prompts for depth -### Hexagonal Architecture -- **Domain Layer** (`internal/core`): Business logic, entities, and interfaces -- **Infrastructure Layer** (`internal/adapter`): Implementations (proxies, balancers, registries) -- **Application Layer** (`internal/app`): HTTP handlers, middleware, and services +Gates on `make ready` first, then boots seven `test/cmd/ollamock` instances (ports 19431–19437) plus two Olla instances (`test/validate/config.validate*.yaml`, ports 41141/41142) and fans out parallel agents per area. Reports land in `test/results/`. ollamock speaks OpenAI, Ollama-native, LM Studio, Lemonade and Anthropic wire formats with runtime fault injection via `/_mock/behaviour` (see `test/cmd/ollamock/README.md`). Full docs: `docs/content/development/validation.md`. -### Key Components -- **Translator Layer**: Enables API format translation (e.g., OpenAI ↔ Anthropic) with passthrough optimisation for backends with native support -- **Passthrough Mode**: When a backend natively supports the Anthropic Messages API (vLLM, llama.cpp, LM Studio, Ollama), requests bypass translation entirely -- **Translator Metrics**: Thread-safe per-translator statistics tracking passthrough/translation rates, fallback reasons, latency, and streaming breakdown (`internal/adapter/stats/translator_collector.go`) -- **Sticky Sessions**: Optional decorator on the endpoint selector that pins multi-turn LLM conversations to the backend that handled the first turn, maximising KV-cache reuse. 64-bit FNV-1a hashed keys, TTL + LRU bounded, purged on routable→non-routable health transitions (`internal/adapter/balancer/sticky.go`) -- **Proxy Engines**: Choose Sherpa (simple) or Olla (high-performance) -- **Load Balancing**: Priority-based recommended for production -- **Version Management**: Build-time version injection via `internal/version` - -### Development Guidelines -- Go 1.24 (do not bump to 1.25; see Dependencies for held-back packages) -- Australian English for comments and documentation -- Comment on **why** rather than **what** -- Always run `make ready` before committing -- Use `make help` to see all available commands +## Development Guidelines +- **Go 1.24** - do not bump to 1.25 (see pins below). +- Australian English for comments and documentation. No em-dashes. +- Comment on **why**, not **what**. Concise and direct. +- Production code must not panic - guard closes with CAS or `sync.Once`. +- Always run `make ready` before committing. ## Dependencies (Endorsed) - +Do not add dependencies unless explicitly asked. ```go "github.com/docker/go-units" // Human-readable sizes -"github.com/json-iterator/go" // High-performance JSON encoding/decoding +"github.com/json-iterator/go" // High-performance JSON "github.com/puzpuzpuz/xsync/v4" // Concurrent maps/counters "github.com/tidwall/gjson" // Fast JSON parsing -"github.com/jellydator/ttlcache" // Time-to-live cache -"github.com/rs/cors" // CORS middleware for browser clients +"github.com/jellydator/ttlcache" // TTL cache +"github.com/rs/cors" // CORS middleware (off by default) "golang.org/x/sync" // errgroup "golang.org/x/time" // rate limiting ``` -Do not add additional dependencies unless explicitly asked. - ### Go 1.24 Compatibility Pins +From the versions below onward the upstream `go` directive moves to 1.25, so these are held back. `go get -u ./...` silently bumps the toolchain by pulling them - afterwards re-pin in `go.mod`, or use `go get -u=patch ./...`. -Olla targets Go 1.24. From the versions listed below onward, the upstream `go` directive moves to 1.25, so these packages are held back: - -- `golang.org/x/sys` at v0.41.0 (v0.42.0+ requires Go 1.25) -- `golang.org/x/term` at v0.40.0 -- `golang.org/x/text` at v0.34.0 -- `golang.org/x/sync` at v0.19.0 -- `golang.org/x/time` at v0.14.0 -- `atomicgo.dev/keyboard` at v0.2.9 - -`go get -u ./...` will silently bump the toolchain to 1.25 by pulling these. After running it, check `go.mod` and pin the affected packages back to the versions above, or use `go get -u=patch ./...` to limit upgrades to patch releases only. - -## SUB-AGENT DELEGATION - -CRITICAL: Always delegate tasks to the appropriate subagent. Do NOT perform work directly in the main context. +- `golang.org/x/sys` v0.41.0 · `golang.org/x/term` v0.40.0 · `golang.org/x/text` v0.34.0 +- `golang.org/x/sync` v0.19.0 · `golang.org/x/time` v0.14.0 · `atomicgo.dev/keyboard` v0.2.9 -- Code Review → Use the appropriate language subagent (Eg. Go Architect) or reviewer subagent -- Code changes → Use the appropriate language subagent (Eg. Go Architect) or implementer subagent -- Research/exploration → Use the explore subagent -- Testing → Use the test subagent +## Sub-Agent Delegation +CRITICAL: delegate work to the appropriate subagent; use the main context only for orchestration and task decomposition. -Only use the main context for orchestration and task decomposition. +- Code review / code changes → language subagent (e.g. Go Architect) or reviewer/implementer +- Research / exploration → explore subagent +- Testing → test subagent diff --git a/docs/content/development/testing.md b/docs/content/development/testing.md index 65149ffe..c4fb2b5b 100644 --- a/docs/content/development/testing.md +++ b/docs/content/development/testing.md @@ -30,7 +30,8 @@ This guide covers testing patterns and strategies used in Olla. │ │ ├── logic/ # Routing logic tests │ │ ├── security/ # Security tests │ │ └── streaming/ # Streaming tests -│ └── cmd/ # Helper programs for testing (e.g. mock backend) +│ ├── cmd/ # Helper binaries (mockbackend, ollamock) +│ └── validate/ # Validation harness configs (see Validation Harness) └── (benchmarks live alongside code as *_bench_test.go files in internal/) ``` diff --git a/docs/content/development/validation.md b/docs/content/development/validation.md new file mode 100644 index 00000000..4b5ef0a7 --- /dev/null +++ b/docs/content/development/validation.md @@ -0,0 +1,78 @@ +--- +title: Validation Harness - Agent-Driven Release Gating +description: How to validate Olla end-to-end with the olla-validate skill and the ollamock mock backend, without Docker or real inference infrastructure. +keywords: olla validation, release gate, mock backend, ollamock, regression testing +--- + +# Validation Harness + +Beyond unit and integration tests, Olla has an agent-driven validation harness +that exercises a running Olla against a fleet of mock backends. It covers the +ground that in-process tests cannot: real HTTP routing across every provider +namespace, streaming, Anthropic translation and passthrough, sticky sessions, +health transitions, failover and recovery, and request limits. + +It needs no Docker and no real inference backends, so it runs the same way on +a laptop and in CI. + +## The `/olla-validate` skill + +The harness is driven by a Claude Code skill at +`.claude/skills/olla-validate/`. It has two depths: + +| Mode | Time | Use | +|---|---|---| +| `/olla-validate --quick` | 5–10 min | Gate after major changes | +| `/olla-validate --nightly` | 2–4 h | Pre-release gate: exhaustive checklists, chaos, soak, Sherpa pass, forced-translation pass, benchmarks | + +Without a flag it asks which depth to run. + +The skill pins itself to Sonnet (`model: sonnet` in its frontmatter), so it +costs the same regardless of which model the session is using. Area agents +are balanced for token efficiency: mechanical curl-and-assert checklists run +on Haiku, while anything that mutates mock state or validates protocol +sequences (resilience, the Anthropic areas) runs on Sonnet. + +Every run starts with `make ready` as a hard gate, then boots the mock fleet +and two Olla instances, fans out parallel validation agents (one per area +checklist in `.claude/skills/olla-validate/areas/`), and writes a report to +`test/results/olla-validate-.md` with a one-line history entry in +`test/results/last-runs.md`. Any failure anywhere fails the gate. + +## ollamock + +`test/cmd/ollamock` is a stdlib-only mock LLM backend that speaks the wire +formats of every provider Olla fronts: OpenAI chat completions (including SSE +streaming), Ollama's native `/api/*` protocol (NDJSON streaming), LM Studio, +Lemonade and the Anthropic Messages API. Responses carry real token-usage +fields so metrics extraction works, and every response is tagged with a +`BACKEND:{name}` marker plus an `X-Ollamock-Instance` header so tests can +assert exactly which backend served a request. + +Fault injection is controlled at runtime through `/_mock/behaviour`: forced +error statuses, flaky error rates, hangs, slow first byte, mid-stream +connection drops, malformed JSON and health-check failure. `/_mock/stats` +exposes per-path request counters. See `test/cmd/ollamock/README.md` for +flags and examples. + +## Harness topology + +`test/validate/config.validate.yaml` wires seven endpoints (openai-compatible +×2, ollama, lm-studio, vllm, litellm, llamacpp) across seven ollamock +instances on ports 19431–19437 into an Olla on 41141, with the unifier, +sticky sessions and the Anthropic translator enabled. The model lists overlap +deliberately so unification and model routing are observable. +`test/validate/config.validate.limits.yaml` runs a second Olla on 41142 with +a 256KB body cap and a tight per-IP rate limit so 413/429 behaviour is +testable in seconds. + +Each endpoint gets its own ollamock instance because Olla keys endpoint +identity on the URL: two endpoints sharing a URL silently collapse into one. + +## Timing expectations + +Health probes run on a global 30-second ticker regardless of the configured +`check_interval`, so checklists allow up to 40 seconds for probe-driven +health transitions and up to 60 seconds for recovery. Request-path connection +failures mark an endpoint unhealthy immediately, so failover assertions are +fast; only probe-driven transitions need the generous windows. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 30a590ca..a8ab8f08 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -184,6 +184,7 @@ nav: - Circuit Breaker: development/circuit-breaker.md - Contributing: development/contributing.md - Testing: development/testing.md + - Validation Harness: development/validation.md - Benchmarking: development/benchmarking.md - Notes: - Anthropic Inspector: notes/anthropic-inspector.md diff --git a/test/cmd/ollamock/README.md b/test/cmd/ollamock/README.md new file mode 100644 index 00000000..2e576ae3 --- /dev/null +++ b/test/cmd/ollamock/README.md @@ -0,0 +1,164 @@ +# ollamock + +A multi-protocol mock LLM backend for end-to-end validation of Olla's proxy and routing logic without requiring real inference infrastructure. + +It speaks Ollama, LM Studio, OpenAI-compatible, Lemonade, and Anthropic wire formats and supports controllable fault injection via an HTTP control plane. + +## Running + +```bash +go run ./test/cmd/ollamock +``` + +With all flags: + +```bash +go run ./test/cmd/ollamock \ + --addr 127.0.0.1:19431 \ + --name mock-a \ + --models llama3.2,phi4 \ + --ttft-ms 50 \ + --tps 20 \ + --stream-chunks 5 +``` + +## Flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--addr` | `127.0.0.1:19431` | Listen address | +| `--name` | `mock-a` | Instance marker embedded in every response body and `X-Ollamock-Instance` header | +| `--models` | `test-model` | Comma-separated model names served in all list endpoints | +| `--ttft-ms` | `0` | Delay before first streamed byte (milliseconds) | +| `--tps` | `0` | Token-per-second pacing for stream chunks (0 = instant) | +| `--stream-chunks` | `5` | Number of content chunks per streamed response | + +## Protocol Endpoints + +| Route | Protocol | +|-------|----------| +| `GET /health` | Health check | +| `GET /` | Ollama root liveness (`"Ollama is running"`) | +| `GET /api/tags` | Ollama model listing | +| `GET /api/version` | Ollama version | +| `POST /api/chat` | Ollama chat (streaming default) | +| `POST /api/generate` | Ollama generate (streaming default) | +| `GET /api/v0/models` | LM Studio model listing | +| `GET /v1/models` | OpenAI-compatible model listing | +| `POST /v1/chat/completions` | OpenAI chat completions | +| `POST /v1/completions` | OpenAI text completions (legacy) | +| `GET /api/v1/models` | Lemonade model listing | +| `POST /api/v1/chat/completions` | Lemonade chat (OpenAI shape) | +| `POST /v1/messages` | Anthropic Messages API | + +## Behaviour Modes + +| Mode | Description | +|------|-------------| +| `ok` | Normal operation (default) | +| `error` | All requests return `error_status` with a mock error body | +| `flaky` | Returns errors randomly at `error_rate` frequency | +| `hang` | Blocks all requests indefinitely (until client disconnects) | +| `slow` | Adds `latency_ms` delay to every request | + +## Behaviour Fields + +| Field | Type | Description | +|-------|------|-------------| +| `mode` | string | One of the modes above | +| `error_status` | int | HTTP status for error/flaky modes (default 500) | +| `error_rate` | float | Probability of error in flaky mode, 0.0–1.0 (default 0.5) | +| `latency_ms` | int | Additional latency in slow mode (milliseconds) | +| `fail_health` | bool | Return 503 on `/health` and `/` only | +| `drop_mid_stream` | bool | Close connection after half the stream chunks | +| `malformed_json` | bool | Return truncated JSON (`{"broken":`) with 200 status | + +## Control Plane + +All `/_mock/*` routes are immune to behaviour - they always respond normally. + +### Get current behaviour + +```bash +curl http://127.0.0.1:19431/_mock/behaviour +``` + +### Set behaviour (partial merge) + +```bash +# Switch to error mode +curl -X POST http://127.0.0.1:19431/_mock/behaviour \ + -H 'Content-Type: application/json' \ + -d '{"mode":"error","error_status":503}' + +# Make it flaky at 30% error rate +curl -X POST http://127.0.0.1:19431/_mock/behaviour \ + -d '{"mode":"flaky","error_rate":0.3}' + +# Add 200ms latency +curl -X POST http://127.0.0.1:19431/_mock/behaviour \ + -d '{"mode":"slow","latency_ms":200}' + +# Fail only health checks (circuit-breaker testing) +curl -X POST http://127.0.0.1:19431/_mock/behaviour \ + -d '{"fail_health":true}' +``` + +### Reset to defaults + +```bash +curl -X POST http://127.0.0.1:19431/_mock/reset +``` + +### Get request stats + +```bash +curl http://127.0.0.1:19431/_mock/stats +# {"total":42,"by_path":{"/v1/chat/completions":30,"/api/tags":12}} +``` + +## Example Requests + +```bash +# Ollama model list +curl http://127.0.0.1:19431/api/tags + +# OpenAI model list +curl http://127.0.0.1:19431/v1/models + +# OpenAI non-streaming chat +curl -X POST http://127.0.0.1:19431/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"test-model","stream":false,"messages":[{"role":"user","content":"hello"}]}' + +# OpenAI streaming chat +curl -X POST http://127.0.0.1:19431/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"test-model","stream":true,"messages":[{"role":"user","content":"hello"}]}' + +# Ollama streaming chat (stream: absent defaults to true) +curl -X POST http://127.0.0.1:19431/api/chat \ + -d '{"model":"test-model","messages":[{"role":"user","content":"hello"}]}' + +# Anthropic non-streaming +curl -X POST http://127.0.0.1:19431/v1/messages \ + -H 'Content-Type: application/json' \ + -d '{"model":"test-model","max_tokens":100,"messages":[{"role":"user","content":"hello"}]}' + +# Anthropic streaming +curl -X POST http://127.0.0.1:19431/v1/messages \ + -H 'Content-Type: application/json' \ + -d '{"model":"test-model","stream":true,"max_tokens":100,"messages":[{"role":"user","content":"hello"}]}' +``` + +## Running Multiple Instances + +For load-balancer and routing tests, run several instances on different ports: + +```bash +go run ./test/cmd/ollamock --addr 127.0.0.1:19431 --name mock-a --models llama3.2 & +go run ./test/cmd/ollamock --addr 127.0.0.1:19432 --name mock-b --models llama3.2,phi4 & +go run ./test/cmd/ollamock --addr 127.0.0.1:19433 --name mock-c --models phi4 & +``` + +Each instance embeds its name in every response body (`BACKEND:mock-a`) and in the `X-Ollamock-Instance` header, so the validation harness can confirm which backend served each request. diff --git a/test/cmd/ollamock/behaviour.go b/test/cmd/ollamock/behaviour.go new file mode 100644 index 00000000..d0602dfa --- /dev/null +++ b/test/cmd/ollamock/behaviour.go @@ -0,0 +1,283 @@ +package main + +import ( + "encoding/json" + "math/rand" + "net/http" + "sync" + "sync/atomic" +) + +// Mode describes how ollamock should behave when processing requests. +type Mode string + +const ( + // ModeOK is normal operation - all requests succeed. + ModeOK Mode = "ok" + // ModeError causes all requests to return a fixed error status. + ModeError Mode = "error" + // ModeFlaky returns errors at a configurable rate so tests can exercise + // retry and circuit-breaker logic. + ModeFlaky Mode = "flaky" + // ModeHang blocks requests indefinitely (until context cancellation) to + // simulate a hung backend. + ModeHang Mode = "hang" + // ModeSlow adds a fixed latency to every response. + ModeSlow Mode = "slow" +) + +// validModes is the closed set of accepted mode strings. +var validModes = map[Mode]bool{ + ModeOK: true, + ModeError: true, + ModeFlaky: true, + ModeHang: true, + ModeSlow: true, +} + +// Behaviour captures the current fault-injection configuration. +// Field order is largest-to-smallest for alignment. +type Behaviour struct { + Mode Mode `json:"mode"` + ErrorRate float64 `json:"error_rate"` + LatencyMS int `json:"latency_ms"` + ErrorStatus int `json:"error_status"` + FailHealth bool `json:"fail_health"` + DropMidStream bool `json:"drop_mid_stream"` + MalformedJSON bool `json:"malformed_json"` +} + +func defaultBehaviour() Behaviour { + return Behaviour{ + Mode: ModeOK, + ErrorStatus: 500, + ErrorRate: 0.5, + } +} + +// behaviourState holds the mutable behaviour alongside a seeded RNG for flaky +// mode. Both are protected by the same mutex so reads and writes are always +// consistent - the rand source is not goroutine-safe on its own. +type behaviourState struct { + rng *rand.Rand + b Behaviour + seed int64 + mu sync.RWMutex +} + +func newBehaviourState(seed int64) *behaviourState { + return &behaviourState{ + b: defaultBehaviour(), + rng: rand.New(rand.NewSource(seed)), //nolint:gosec // deterministic seed for reproducible test scenarios + seed: seed, + } +} + +func (s *behaviourState) get() Behaviour { + s.mu.RLock() + defer s.mu.RUnlock() + return s.b +} + +// merge applies a partial JSON patch into the current behaviour. +// Only fields present in the patch are updated; others retain their value. +func (s *behaviourState) merge(patch Behaviour, hasPatch patchFields) error { + s.mu.Lock() + defer s.mu.Unlock() + + if hasPatch.mode { + if !validModes[patch.Mode] { + return errUnknownMode + } + s.b.Mode = patch.Mode + } + if hasPatch.errorStatus { + s.b.ErrorStatus = patch.ErrorStatus + } + if hasPatch.errorRate { + s.b.ErrorRate = patch.ErrorRate + } + if hasPatch.latencyMS { + s.b.LatencyMS = patch.LatencyMS + } + if hasPatch.failHealth { + s.b.FailHealth = patch.FailHealth + } + if hasPatch.dropMidStream { + s.b.DropMidStream = patch.DropMidStream + } + if hasPatch.malformedJSON { + s.b.MalformedJSON = patch.MalformedJSON + } + return nil +} + +func (s *behaviourState) reset() { + s.mu.Lock() + defer s.mu.Unlock() + s.b = defaultBehaviour() + // Re-seed with the original seed so behaviour after reset is reproducible. + s.rng = rand.New(rand.NewSource(s.seed)) //nolint:gosec +} + +// float64 returns a random float64 in [0,1). Caller must not hold the mutex. +func (s *behaviourState) float64() float64 { + s.mu.Lock() + defer s.mu.Unlock() + return s.rng.Float64() +} + +// patchFields tracks which JSON fields were explicitly set in a PATCH body. +// This lets us distinguish "error_rate: 0" (explicit zero) from absent field. +type patchFields struct { + mode bool + errorStatus bool + errorRate bool + latencyMS bool + failHealth bool + dropMidStream bool + malformedJSON bool +} + +// parsePatch decodes a partial behaviour JSON and records which fields were present. +func parsePatch(data []byte) (Behaviour, patchFields, error) { + var raw map[string]json.RawMessage + if err := json.Unmarshal(data, &raw); err != nil { + return Behaviour{}, patchFields{}, err + } + + var b Behaviour + var pf patchFields + + if v, ok := raw["mode"]; ok { + if err := json.Unmarshal(v, &b.Mode); err != nil { + return b, pf, err + } + pf.mode = true + } + if v, ok := raw["error_status"]; ok { + if err := json.Unmarshal(v, &b.ErrorStatus); err != nil { + return b, pf, err + } + pf.errorStatus = true + } + if v, ok := raw["error_rate"]; ok { + if err := json.Unmarshal(v, &b.ErrorRate); err != nil { + return b, pf, err + } + pf.errorRate = true + } + if v, ok := raw["latency_ms"]; ok { + if err := json.Unmarshal(v, &b.LatencyMS); err != nil { + return b, pf, err + } + pf.latencyMS = true + } + if v, ok := raw["fail_health"]; ok { + if err := json.Unmarshal(v, &b.FailHealth); err != nil { + return b, pf, err + } + pf.failHealth = true + } + if v, ok := raw["drop_mid_stream"]; ok { + if err := json.Unmarshal(v, &b.DropMidStream); err != nil { + return b, pf, err + } + pf.dropMidStream = true + } + if v, ok := raw["malformed_json"]; ok { + if err := json.Unmarshal(v, &b.MalformedJSON); err != nil { + return b, pf, err + } + pf.malformedJSON = true + } + + return b, pf, nil +} + +// statsStore tracks per-path request counts using atomic int64s inside a +// sync.Map so hot paths never contend on a single lock. +type statsStore struct { + m sync.Map // map[string]*atomic.Int64 +} + +func (s *statsStore) inc(path string) { + v, _ := s.m.LoadOrStore(path, &atomic.Int64{}) + if counter, ok := v.(*atomic.Int64); ok { + counter.Add(1) + } +} + +func (s *statsStore) snapshot() (int64, map[string]int64) { + byPath := make(map[string]int64) + var total int64 + s.m.Range(func(k, v any) bool { + counter, ok := v.(*atomic.Int64) + if !ok { + return true + } + key, ok := k.(string) + if !ok { + return true + } + n := counter.Load() + byPath[key] = n + total += n + return true + }) + return total, byPath +} + +func (s *statsStore) reset() { + s.m.Range(func(_, v any) bool { + if counter, ok := v.(*atomic.Int64); ok { + counter.Store(0) + } + return true + }) +} + +// controlHandlers registers the /_mock/* control plane routes onto mux. +// Control routes are immune to behaviour - they always respond normally. +func (srv *mockServer) controlHandlers(mux *http.ServeMux) { + mux.HandleFunc("GET /_mock/behaviour", srv.handleGetBehaviour) + mux.HandleFunc("POST /_mock/behaviour", srv.handlePostBehaviour) + mux.HandleFunc("POST /_mock/reset", srv.handleReset) + mux.HandleFunc("GET /_mock/stats", srv.handleStats) +} + +func (srv *mockServer) handleGetBehaviour(w http.ResponseWriter, _ *http.Request) { + b := srv.bstate.get() + writeJSON(w, http.StatusOK, b) +} + +func (srv *mockServer) handlePostBehaviour(w http.ResponseWriter, r *http.Request) { + var body [4096]byte + n, _ := r.Body.Read(body[:]) + _ = r.Body.Close() + + patch, pf, err := parsePatch(body[:n]) + if err != nil { + writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()}) + return + } + if err := srv.bstate.merge(patch, pf); err != nil { + writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()}) + return + } + writeJSON(w, http.StatusOK, srv.bstate.get()) +} + +func (srv *mockServer) handleReset(w http.ResponseWriter, _ *http.Request) { + srv.bstate.reset() + srv.stats.reset() + writeJSON(w, http.StatusOK, map[string]string{"status": "reset"}) +} + +func (srv *mockServer) handleStats(w http.ResponseWriter, _ *http.Request) { + total, byPath := srv.stats.snapshot() + writeJSON(w, http.StatusOK, map[string]any{ + "total": total, + "by_path": byPath, + }) +} diff --git a/test/cmd/ollamock/handlers.go b/test/cmd/ollamock/handlers.go new file mode 100644 index 00000000..2f90a2f6 --- /dev/null +++ b/test/cmd/ollamock/handlers.go @@ -0,0 +1,308 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "hash/fnv" + "net/http" + "time" +) + +// errUnknownMode is returned when a PATCH body sets an unrecognised mode. +var errUnknownMode = errors.New("unknown mode") + +// serverConfig holds static configuration set at startup. +// Field order largest-to-smallest for alignment. +type serverConfig struct { + name string + models []string + ttftMS int + tps int + streamChunks int +} + +// mockServer is the central state container wired into the HTTP mux. +type mockServer struct { + bstate *behaviourState + stats statsStore + cfg serverConfig +} + +func newServer(cfg serverConfig) *mockServer { + // Seed from FNV-1a hash of the instance name so behaviour is deterministic + // per-instance but distinct across instances in a multi-node test fleet. + h := fnv.New64a() + _, _ = h.Write([]byte(cfg.name)) + seed := int64(h.Sum64()) + + return &mockServer{ + cfg: cfg, + bstate: newBehaviourState(seed), + } +} + +// handler builds and returns the ServeMux for this server instance. +func (srv *mockServer) handler() http.Handler { + mux := http.NewServeMux() + + // Control plane - always healthy, immune to behaviour. + srv.controlHandlers(mux) + + // LLM protocol routes. + mux.HandleFunc("GET /health", srv.wrap(srv.handleHealth)) + mux.HandleFunc("GET /", srv.wrap(srv.handleRoot)) + + // Model listing endpoints - one per protocol. + mux.HandleFunc("GET /v1/models", srv.wrap(srv.handleOpenAIModels)) + mux.HandleFunc("GET /api/tags", srv.wrap(srv.handleOllamaTags)) + mux.HandleFunc("GET /api/version", srv.wrap(srv.handleOllamaVersion)) + mux.HandleFunc("GET /api/v0/models", srv.wrap(srv.handleLMStudioModels)) + mux.HandleFunc("GET /api/v1/models", srv.wrap(srv.handleLemonadeModels)) + + // Inference endpoints - streaming handled in streaming.go. + mux.HandleFunc("POST /api/chat", srv.wrap(srv.handleOllamaChat)) + mux.HandleFunc("POST /api/generate", srv.wrap(srv.handleOllamaGenerate)) + mux.HandleFunc("POST /v1/chat/completions", srv.wrap(srv.handleOpenAIChat)) + mux.HandleFunc("POST /v1/completions", srv.wrap(srv.handleOpenAICompletion)) + mux.HandleFunc("POST /api/v1/chat/completions", srv.wrap(srv.handleOpenAIChat)) + mux.HandleFunc("POST /v1/messages", srv.wrap(srv.handleAnthropicMessages)) + + return mux +} + +// wrap returns a handler that records stats, sets the instance header, and +// applies the current behaviour before delegating to the inner handler. +func (srv *mockServer) wrap(inner http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + srv.stats.inc(r.URL.Path) + w.Header().Set("X-Ollamock-Instance", srv.cfg.name) + if !srv.applyBehaviour(w, r) { + return + } + inner(w, r) + } +} + +// applyBehaviour enforces the current Mode, returning false when the request +// should be short-circuited (response already written). +func (srv *mockServer) applyBehaviour(w http.ResponseWriter, r *http.Request) bool { + b := srv.bstate.get() + + isHealthPath := r.URL.Path == "/health" || r.URL.Path == "/" + + // fail_health only gates the health endpoints - all other routes are unaffected. + if b.FailHealth && isHealthPath { + w.WriteHeader(http.StatusServiceUnavailable) + _, _ = fmt.Fprintf(w, `{"status":"unhealthy"}`) + return false + } + + switch b.Mode { + case ModeOK: + // No-op - proceed to handler. + + case ModeHang: + // Block until the client disconnects or context is cancelled. + select { + case <-r.Context().Done(): + case <-time.After(10 * time.Minute): + } + return false + + case ModeError: + writeErrorResponse(w, b.ErrorStatus) + return false + + case ModeFlaky: + if srv.bstate.float64() < b.ErrorRate { + writeErrorResponse(w, b.ErrorStatus) + return false + } + + case ModeSlow: + if b.LatencyMS > 0 { + select { + case <-r.Context().Done(): + return false + case <-time.After(time.Duration(b.LatencyMS) * time.Millisecond): + } + } + } + + if b.MalformedJSON { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + _, _ = fmt.Fprint(w, `{"broken":`) + return false + } + + return true +} + +func writeErrorResponse(w http.ResponseWriter, status int) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _, _ = fmt.Fprintf(w, `{"error":{"message":"mock error","type":"mock_error"}}`) +} + +// writeJSON serialises v as JSON and writes it with the given status. +func writeJSON(w http.ResponseWriter, status int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(v) +} + +// --- Health / liveness --- + +func (srv *mockServer) handleHealth(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, map[string]string{"status": "ok"}) +} + +func (srv *mockServer) handleRoot(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/plain") + w.WriteHeader(http.StatusOK) + _, _ = fmt.Fprint(w, "Ollama is running") +} + +// --- Ollama protocol --- + +func (srv *mockServer) handleOllamaTags(w http.ResponseWriter, _ *http.Request) { + type details struct { + Family string `json:"family"` + ParameterSize string `json:"parameter_size"` + QuantizationLevel string `json:"quantization_level"` + } + type model struct { + Details details `json:"details"` + ModifiedAt string `json:"modified_at"` + Digest string `json:"digest"` + Name string `json:"name"` + Model string `json:"model"` + Size int64 `json:"size"` + } + type response struct { + Models []model `json:"models"` + } + + now := time.Now().UTC().Format(time.RFC3339) + models := make([]model, len(srv.cfg.models)) + for i, m := range srv.cfg.models { + models[i] = model{ + Name: m, + Model: m, + ModifiedAt: now, + Size: 4_000_000_000, + Digest: fmt.Sprintf("sha256:ollamock%s%s", srv.cfg.name, m), + Details: details{ + Family: "llama", + ParameterSize: "7B", + QuantizationLevel: "Q4_0", + }, + } + } + writeJSON(w, http.StatusOK, response{Models: models}) +} + +func (srv *mockServer) handleOllamaVersion(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, map[string]string{"version": "0.6.0-ollamock"}) +} + +// --- LM Studio protocol --- + +func (srv *mockServer) handleLMStudioModels(w http.ResponseWriter, _ *http.Request) { + type model struct { + Type string `json:"type"` + Publisher string `json:"publisher"` + Arch string `json:"arch"` + State string `json:"state"` + ID string `json:"id"` + Object string `json:"object"` + MaxContextLength int64 `json:"max_context_length"` + } + type response struct { + Object string `json:"object"` + Data []model `json:"data"` + } + + models := make([]model, len(srv.cfg.models)) + for i, m := range srv.cfg.models { + models[i] = model{ + ID: m, + Object: "model", + Type: "llm", + Publisher: "ollamock", + Arch: "llama", + State: "loaded", + MaxContextLength: 4096, + } + } + writeJSON(w, http.StatusOK, response{Object: "list", Data: models}) +} + +// --- OpenAI-compatible protocol --- + +func (srv *mockServer) handleOpenAIModels(w http.ResponseWriter, _ *http.Request) { + type model struct { + OwnedBy string `json:"owned_by"` + ID string `json:"id"` + Object string `json:"object"` + Created int64 `json:"created"` + } + type response struct { + Object string `json:"object"` + Data []model `json:"data"` + } + + now := time.Now().Unix() + models := make([]model, len(srv.cfg.models)) + for i, m := range srv.cfg.models { + models[i] = model{ + ID: m, + Object: "model", + Created: now, + OwnedBy: "ollamock", + } + } + writeJSON(w, http.StatusOK, response{Object: "list", Data: models}) +} + +// --- Lemonade protocol --- + +func (srv *mockServer) handleLemonadeModels(w http.ResponseWriter, _ *http.Request) { + type model struct { + ID string `json:"id"` + Object string `json:"object"` + OwnedBy string `json:"owned_by"` + Checkpoint string `json:"checkpoint"` + Recipe string `json:"recipe"` + Created int64 `json:"created"` + Downloaded bool `json:"downloaded"` + } + type response struct { + Object string `json:"object"` + Data []model `json:"data"` + } + + now := time.Now().Unix() + models := make([]model, len(srv.cfg.models)) + for i, m := range srv.cfg.models { + models[i] = model{ + ID: m, + Object: "model", + OwnedBy: "lemonade", + Checkpoint: fmt.Sprintf("amd/%s", m), + Recipe: "oga-cpu", + Created: now, + Downloaded: true, + } + } + writeJSON(w, http.StatusOK, response{Object: "list", Data: models}) +} + +// contextDone returns a closed channel when ctx is already done, otherwise the +// ctx.Done() channel. Used in select statements that must compile on Go 1.24. +func contextDone(ctx context.Context) <-chan struct{} { + return ctx.Done() +} diff --git a/test/cmd/ollamock/main.go b/test/cmd/ollamock/main.go new file mode 100644 index 00000000..ee969aed --- /dev/null +++ b/test/cmd/ollamock/main.go @@ -0,0 +1,98 @@ +// ollamock is a multi-protocol mock LLM backend for end-to-end validation of +// Olla's proxy and routing logic without requiring real inference infrastructure. +// +// It speaks Ollama, LM Studio, OpenAI-compatible, Lemonade, and Anthropic +// wire formats and supports controllable failure injection via a control plane. +// +// Usage: +// +// go run ./test/cmd/ollamock \ +// --addr 127.0.0.1:19431 \ +// --name mock-a \ +// --models llama3.2,phi4 \ +// --ttft-ms 50 \ +// --tps 20 \ +// --stream-chunks 5 +package main + +import ( + "context" + "errors" + "flag" + "log/slog" + "net/http" + "os" + "os/signal" + "strings" + "syscall" + "time" +) + +func main() { + addr := flag.String("addr", "127.0.0.1:19431", "listen address") + name := flag.String("name", "mock-a", "instance marker embedded in every response") + models := flag.String("models", "test-model", "comma-separated list of model names to serve") + ttftMS := flag.Int("ttft-ms", 0, "delay in milliseconds before first streamed byte") + tps := flag.Int("tps", 0, "tokens per second pacing (0 = instant)") + streamChunks := flag.Int("stream-chunks", 5, "number of content chunks per streamed response") + flag.Parse() + + modelList := parseModels(*models) + + cfg := serverConfig{ + name: *name, + models: modelList, + ttftMS: *ttftMS, + tps: *tps, + streamChunks: *streamChunks, + } + + srv := newServer(cfg) + + httpSrv := &http.Server{ + Addr: *addr, + Handler: srv.handler(), + ReadHeaderTimeout: 5 * time.Second, + } + + slog.Info("ollamock started", + "addr", *addr, + "name", *name, + "models", modelList, + ) + + quit := make(chan os.Signal, 1) + signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM) + + go func() { + if err := httpSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + slog.Error("ollamock listen error", "err", err) + os.Exit(1) + } + }() + + <-quit + slog.Info("ollamock shutting down") + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + if err := httpSrv.Shutdown(ctx); err != nil { + slog.Error("ollamock shutdown error", "err", err) + } +} + +func parseModels(s string) []string { + parts := strings.Split(s, ",") + out := make([]string, 0, len(parts)) + for _, p := range parts { + p = strings.TrimSpace(p) + if p != "" { + out = append(out, p) + } + } + if len(out) == 0 { + out = []string{"test-model"} + } + return out +} diff --git a/test/cmd/ollamock/ollamock_test.go b/test/cmd/ollamock/ollamock_test.go new file mode 100644 index 00000000..56eb2faa --- /dev/null +++ b/test/cmd/ollamock/ollamock_test.go @@ -0,0 +1,428 @@ +package main + +import ( + "bufio" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// testServer creates an isolated httptest.Server for each test, so tests never +// share mutable state. The server is closed automatically via t.Cleanup. +func testServer(t *testing.T, models ...string) *httptest.Server { + t.Helper() + if len(models) == 0 { + models = []string{"test-model"} + } + cfg := serverConfig{ + name: "test", + models: models, + ttftMS: 0, + tps: 0, + streamChunks: 3, + } + srv := newServer(cfg) + ts := httptest.NewServer(srv.handler()) + t.Cleanup(ts.Close) + return ts +} + +// postJSON sends a POST with a JSON body and returns the response. +func postJSON(t *testing.T, url, body string) *http.Response { + t.Helper() + resp, err := http.Post(url, "application/json", strings.NewReader(body)) + if err != nil { + t.Fatalf("POST %s: %v", url, err) + } + return resp +} + +// getURL sends a GET and returns the response. +func getURL(t *testing.T, url string) *http.Response { + t.Helper() + resp, err := http.Get(url) + if err != nil { + t.Fatalf("GET %s: %v", url, err) + } + return resp +} + +// decodeJSON decodes the response body into v, closing the body. +func decodeJSON(t *testing.T, resp *http.Response, v any) { + t.Helper() + defer resp.Body.Close() + if err := json.NewDecoder(resp.Body).Decode(v); err != nil { + t.Fatalf("decode JSON: %v", err) + } +} + +// --- Tests --- + +func TestOllamaTagsFormat(t *testing.T) { + t.Parallel() + ts := testServer(t, "llama3.2", "phi4") + + resp := getURL(t, ts.URL+"/api/tags") + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + + var body struct { + Models []struct { + Name string `json:"name"` + Model string `json:"model"` + Digest string `json:"digest"` + Details struct { + Family string `json:"family"` + } `json:"details"` + } `json:"models"` + } + decodeJSON(t, resp, &body) + + if len(body.Models) != 2 { + t.Fatalf("want 2 models, got %d", len(body.Models)) + } + m := body.Models[0] + if m.Name == "" { + t.Error("name field is empty") + } + if m.Model == "" { + t.Error("model field is empty") + } + if m.Digest == "" { + t.Error("digest field is empty") + } + if m.Details.Family == "" { + t.Error("details.family field is empty") + } +} + +func TestOpenAIModelsList(t *testing.T) { + t.Parallel() + ts := testServer(t, "gpt-mock") + + resp := getURL(t, ts.URL+"/v1/models") + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + + var body struct { + Object string `json:"object"` + Data []struct { + ID string `json:"id"` + Object string `json:"object"` + } `json:"data"` + } + decodeJSON(t, resp, &body) + + if body.Object != "list" { + t.Errorf("want object=list, got %q", body.Object) + } + if len(body.Data) != 1 { + t.Fatalf("want 1 model, got %d", len(body.Data)) + } + if body.Data[0].ID != "gpt-mock" { + t.Errorf("want id=gpt-mock, got %q", body.Data[0].ID) + } + if body.Data[0].Object != "model" { + t.Errorf("want object=model, got %q", body.Data[0].Object) + } +} + +func TestOpenAINonStreamChat(t *testing.T) { + t.Parallel() + ts := testServer(t) + + resp := postJSON(t, ts.URL+"/v1/chat/completions", + `{"model":"test-model","stream":false,"messages":[{"role":"user","content":"hi"}]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + + var body struct { + Object string `json:"object"` + Choices []struct { + Message struct { + Role string `json:"role"` + Content string `json:"content"` + } `json:"message"` + FinishReason string `json:"finish_reason"` + } `json:"choices"` + Usage struct { + PromptTokens int `json:"prompt_tokens"` + CompletionTokens int `json:"completion_tokens"` + TotalTokens int `json:"total_tokens"` + } `json:"usage"` + } + decodeJSON(t, resp, &body) + + if body.Object != "chat.completion" { + t.Errorf("want object=chat.completion, got %q", body.Object) + } + if len(body.Choices) == 0 { + t.Fatal("no choices in response") + } + if body.Choices[0].FinishReason != "stop" { + t.Errorf("want finish_reason=stop, got %q", body.Choices[0].FinishReason) + } + if body.Usage.TotalTokens == 0 { + t.Error("usage.total_tokens is 0") + } +} + +func TestOpenAIStreamChat(t *testing.T) { + t.Parallel() + ts := testServer(t) + + resp := postJSON(t, ts.URL+"/v1/chat/completions", + `{"model":"test-model","stream":true,"messages":[{"role":"user","content":"hi"}]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + defer resp.Body.Close() + + var chunks []string + var sawDone bool + var sawContent bool + + scanner := bufio.NewScanner(resp.Body) + for scanner.Scan() { + line := scanner.Text() + if line == "" { + continue + } + if line == "data: [DONE]" { + sawDone = true + break + } + if !strings.HasPrefix(line, "data: ") { + continue + } + data := strings.TrimPrefix(line, "data: ") + chunks = append(chunks, data) + + var chunk struct { + Choices []struct { + Delta struct { + Content string `json:"content"` + } `json:"delta"` + } `json:"choices"` + } + if err := json.Unmarshal([]byte(data), &chunk); err == nil { + if len(chunk.Choices) > 0 && chunk.Choices[0].Delta.Content != "" { + sawContent = true + } + } + } + + if !sawDone { + t.Error("stream did not end with data: [DONE]") + } + if !sawContent { + t.Error("no chunk with delta content found") + } + if len(chunks) == 0 { + t.Error("no SSE chunks received") + } +} + +func TestOllamaChatStreamDefault(t *testing.T) { + t.Parallel() + ts := testServer(t) + + // Ollama defaults to streaming when stream field is absent. + resp := postJSON(t, ts.URL+"/api/chat", + `{"model":"test-model","messages":[{"role":"user","content":"hi"}]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + defer resp.Body.Close() + + var lastLine map[string]any + scanner := bufio.NewScanner(resp.Body) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if line == "" { + continue + } + var obj map[string]any + if err := json.Unmarshal([]byte(line), &obj); err != nil { + t.Fatalf("invalid NDJSON line: %q: %v", line, err) + } + lastLine = obj + } + + if lastLine == nil { + t.Fatal("no NDJSON lines received") + } + + done, ok := lastLine["done"].(bool) + if !ok || !done { + t.Errorf("final line has done != true: %v", lastLine["done"]) + } +} + +func TestOllamaChatNonStream(t *testing.T) { + t.Parallel() + ts := testServer(t) + + resp := postJSON(t, ts.URL+"/api/chat", + `{"model":"test-model","stream":false,"messages":[{"role":"user","content":"hi"}]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + + var body map[string]any + decodeJSON(t, resp, &body) + + done, ok := body["done"].(bool) + if !ok || !done { + t.Errorf("want done=true, got %v", body["done"]) + } + if _, ok := body["message"]; !ok { + t.Error("missing message field in non-stream chat response") + } +} + +func TestAnthropicStreamEventOrder(t *testing.T) { + t.Parallel() + ts := testServer(t) + + resp := postJSON(t, ts.URL+"/v1/messages", + `{"model":"test-model","stream":true,"messages":[{"role":"user","content":"hi"}],"max_tokens":100}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("want 200, got %d", resp.StatusCode) + } + defer resp.Body.Close() + + // Collect event types in order. + var eventOrder []string + scanner := bufio.NewScanner(resp.Body) + for scanner.Scan() { + line := scanner.Text() + if strings.HasPrefix(line, "event: ") { + eventOrder = append(eventOrder, strings.TrimPrefix(line, "event: ")) + } + } + + // Validate required sequence. + want := []string{ + "message_start", + "content_block_start", + "content_block_stop", + "message_delta", + "message_stop", + } + // Find each required event in order (content_block_delta(s) sit between + // content_block_start and content_block_stop). + pos := 0 + for _, w := range want { + found := false + for pos < len(eventOrder) { + if eventOrder[pos] == w { + pos++ + found = true + break + } + pos++ + } + if !found { + t.Errorf("missing event %q in stream; got sequence: %v", w, eventOrder) + } + } + + // Confirm at least one content_block_delta was emitted. + hasDelta := false + for _, e := range eventOrder { + if e == "content_block_delta" { + hasDelta = true + break + } + } + if !hasDelta { + t.Error("no content_block_delta events in stream") + } +} + +func TestBehaviourModeError(t *testing.T) { + t.Parallel() + ts := testServer(t) + + // Switch to error mode with status 503. + patchResp := postJSON(t, ts.URL+"/_mock/behaviour", + `{"mode":"error","error_status":503}`) + if patchResp.StatusCode != http.StatusOK { + patchResp.Body.Close() + t.Fatalf("PATCH behaviour want 200, got %d", patchResp.StatusCode) + } + patchResp.Body.Close() + + // Any subsequent request should get 503. + resp := getURL(t, ts.URL+"/v1/models") + resp.Body.Close() + if resp.StatusCode != http.StatusServiceUnavailable { + t.Errorf("want 503 under error mode, got %d", resp.StatusCode) + } +} + +func TestMockStats(t *testing.T) { + t.Parallel() + ts := testServer(t) + + path := "/v1/chat/completions" + const requestCount = 3 + + for range requestCount { + resp := postJSON(t, ts.URL+path, + `{"model":"test-model","stream":false,"messages":[{"role":"user","content":"hi"}]}`) + resp.Body.Close() + } + + statsResp := getURL(t, ts.URL+"/_mock/stats") + if statsResp.StatusCode != http.StatusOK { + statsResp.Body.Close() + t.Fatalf("/_mock/stats want 200, got %d", statsResp.StatusCode) + } + + var stats struct { + Total int64 `json:"total"` + ByPath map[string]int64 `json:"by_path"` + } + decodeJSON(t, statsResp, &stats) + + if stats.Total < requestCount { + t.Errorf("want total >= %d, got %d", requestCount, stats.Total) + } + if stats.ByPath[path] < requestCount { + t.Errorf("want by_path[%q] >= %d, got %d", path, requestCount, stats.ByPath[path]) + } +} + +func TestFailHealth(t *testing.T) { + t.Parallel() + ts := testServer(t) + + // Enable fail_health. + patchResp := postJSON(t, ts.URL+"/_mock/behaviour", `{"fail_health":true}`) + if patchResp.StatusCode != http.StatusOK { + patchResp.Body.Close() + t.Fatalf("PATCH behaviour want 200, got %d", patchResp.StatusCode) + } + patchResp.Body.Close() + + // Health endpoint should now return 503. + healthResp := getURL(t, ts.URL+"/health") + healthResp.Body.Close() + if healthResp.StatusCode != http.StatusServiceUnavailable { + t.Errorf("want 503 on /health with fail_health, got %d", healthResp.StatusCode) + } + + // Non-health routes must be unaffected - fail_health is health-path only. + modelsResp := getURL(t, ts.URL+"/v1/models") + modelsResp.Body.Close() + if modelsResp.StatusCode != http.StatusOK { + t.Errorf("want 200 on /v1/models with fail_health, got %d", modelsResp.StatusCode) + } +} diff --git a/test/cmd/ollamock/streaming.go b/test/cmd/ollamock/streaming.go new file mode 100644 index 00000000..800ed14b --- /dev/null +++ b/test/cmd/ollamock/streaming.go @@ -0,0 +1,616 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "time" +) + +// inferenceRequest is the minimal request shape shared across all protocols. +// Stream nil means "use protocol default" - true for Ollama, false for OpenAI. +type inferenceRequest struct { + Stream *bool `json:"stream"` + Model string `json:"model"` +} + +func parseInferenceRequest(r *http.Request) (inferenceRequest, error) { + var req inferenceRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil && err != io.EOF { + return inferenceRequest{}, err + } + return req, nil +} + +// streamBool is true if stream==nil (use given default) or stream==value. +func resolveStream(s *bool, defaultStream bool) bool { + if s == nil { + return defaultStream + } + return *s +} + +// --- Ollama chat / generate --- + +func (srv *mockServer) handleOllamaChat(w http.ResponseWriter, r *http.Request) { + var req inferenceRequest + body := readBody(r) + _ = json.Unmarshal(body, &req) + + model := req.Model + if model == "" && len(srv.cfg.models) > 0 { + model = srv.cfg.models[0] + } + + stream := resolveStream(req.Stream, true) // Ollama streams by default + + if stream { + srv.streamOllamaChat(w, r, model) + } else { + srv.nonStreamOllamaChat(w, model) + } +} + +func (srv *mockServer) handleOllamaGenerate(w http.ResponseWriter, r *http.Request) { + var req inferenceRequest + body := readBody(r) + _ = json.Unmarshal(body, &req) + + model := req.Model + if model == "" && len(srv.cfg.models) > 0 { + model = srv.cfg.models[0] + } + + stream := resolveStream(req.Stream, true) + + if stream { + srv.streamOllamaGenerate(w, r, model) + } else { + srv.nonStreamOllamaGenerate(w, model) + } +} + +func (srv *mockServer) nonStreamOllamaChat(w http.ResponseWriter, model string) { + now := time.Now().UTC().Format(time.RFC3339) + resp := map[string]any{ + "model": model, + "created_at": now, + "message": map[string]any{ + "role": "assistant", + "content": fmt.Sprintf("BACKEND:%s model:%s", srv.cfg.name, model), + }, + "done": true, + "prompt_eval_count": 10, + "eval_count": 20, + "total_duration": 1_000_000_000, + "load_duration": 100_000_000, + "prompt_eval_duration": 200_000_000, + "eval_duration": 700_000_000, + } + writeJSON(w, http.StatusOK, resp) +} + +func (srv *mockServer) nonStreamOllamaGenerate(w http.ResponseWriter, model string) { + now := time.Now().UTC().Format(time.RFC3339) + resp := map[string]any{ + "model": model, + "created_at": now, + "response": fmt.Sprintf("BACKEND:%s model:%s", srv.cfg.name, model), + "done": true, + "prompt_eval_count": 10, + "eval_count": 20, + "total_duration": 1_000_000_000, + "load_duration": 100_000_000, + "prompt_eval_duration": 200_000_000, + "eval_duration": 700_000_000, + } + writeJSON(w, http.StatusOK, resp) +} + +func (srv *mockServer) streamOllamaChat(w http.ResponseWriter, r *http.Request, model string) { + w.Header().Set("Content-Type", "application/x-ndjson") + rc := http.NewResponseController(w) + + applyTTFT(r.Context(), srv.cfg.ttftMS) + + n := srv.cfg.streamChunks + b := srv.bstate.get() + dropAt := (n + 1) / 2 // ceil(n/2) for drop_mid_stream + + now := time.Now().UTC().Format(time.RFC3339) + + for i := range n { + if b.DropMidStream && i == dropAt { + // Truncate mid-stream - client gets EOF without a done:true final line. + tryHijackClose(w) + return + } + + chunk := map[string]any{ + "model": model, + "created_at": now, + "message": map[string]any{ + "role": "assistant", + "content": fmt.Sprintf("The answer is 42. BACKEND:%s model:%s chunk:%d", srv.cfg.name, model, i), + }, + "done": false, + } + writeNDJSON(w, chunk) + _ = rc.Flush() + applyTPS(r.Context(), srv.cfg.tps) + } + + // Final done:true line with metrics. + final := map[string]any{ + "model": model, + "created_at": now, + "message": map[string]any{ + "role": "assistant", + "content": "", + }, + "done": true, + "prompt_eval_count": 10, + "eval_count": 20, + "total_duration": 1_000_000_000, + "load_duration": 100_000_000, + "prompt_eval_duration": 200_000_000, + "eval_duration": 700_000_000, + } + writeNDJSON(w, final) + _ = rc.Flush() +} + +func (srv *mockServer) streamOllamaGenerate(w http.ResponseWriter, r *http.Request, model string) { + w.Header().Set("Content-Type", "application/x-ndjson") + rc := http.NewResponseController(w) + + applyTTFT(r.Context(), srv.cfg.ttftMS) + + n := srv.cfg.streamChunks + b := srv.bstate.get() + dropAt := (n + 1) / 2 + now := time.Now().UTC().Format(time.RFC3339) + + for i := range n { + if b.DropMidStream && i == dropAt { + tryHijackClose(w) + return + } + + chunk := map[string]any{ + "model": model, + "created_at": now, + "response": fmt.Sprintf("The answer is 42. BACKEND:%s model:%s chunk:%d", srv.cfg.name, model, i), + "done": false, + } + writeNDJSON(w, chunk) + _ = rc.Flush() + applyTPS(r.Context(), srv.cfg.tps) + } + + final := map[string]any{ + "model": model, + "created_at": now, + "response": "", + "done": true, + "prompt_eval_count": 10, + "eval_count": 20, + "total_duration": 1_000_000_000, + "load_duration": 100_000_000, + "prompt_eval_duration": 200_000_000, + "eval_duration": 700_000_000, + } + writeNDJSON(w, final) + _ = rc.Flush() +} + +// --- OpenAI chat completions --- + +func (srv *mockServer) handleOpenAIChat(w http.ResponseWriter, r *http.Request) { + var req inferenceRequest + body := readBody(r) + _ = json.Unmarshal(body, &req) + + model := req.Model + if model == "" && len(srv.cfg.models) > 0 { + model = srv.cfg.models[0] + } + + stream := resolveStream(req.Stream, false) // OpenAI defaults to non-streaming + + if stream { + srv.streamOpenAIChat(w, r, model) + } else { + srv.nonStreamOpenAIChat(w, model) + } +} + +func (srv *mockServer) nonStreamOpenAIChat(w http.ResponseWriter, model string) { + now := time.Now().Unix() + resp := map[string]any{ + "id": fmt.Sprintf("chatcmpl-mock-%s", srv.cfg.name), + "object": "chat.completion", + "created": now, + "model": model, + "choices": []map[string]any{ + { + "index": 0, + "message": map[string]any{ + "role": "assistant", + "content": fmt.Sprintf("BACKEND:%s reply", srv.cfg.name), + }, + "finish_reason": "stop", + }, + }, + "usage": map[string]any{ + "prompt_tokens": 10, + "completion_tokens": 20, + "total_tokens": 30, + }, + } + writeJSON(w, http.StatusOK, resp) +} + +func (srv *mockServer) streamOpenAIChat(w http.ResponseWriter, r *http.Request, model string) { + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + rc := http.NewResponseController(w) + + applyTTFT(r.Context(), srv.cfg.ttftMS) + + id := fmt.Sprintf("chatcmpl-mock-%s", srv.cfg.name) + now := time.Now().Unix() + n := srv.cfg.streamChunks + b := srv.bstate.get() + dropAt := (n + 1) / 2 + + for i := range n { + if b.DropMidStream && i == dropAt { + tryHijackClose(w) + return + } + + chunk := map[string]any{ + "id": id, + "object": "chat.completion.chunk", + "created": now, + "model": model, + "choices": []map[string]any{ + { + "index": 0, + "delta": map[string]any{ + "role": "assistant", + "content": fmt.Sprintf("BACKEND:%s reply: chunk %d", srv.cfg.name, i), + }, + "finish_reason": nil, + }, + }, + } + data, _ := json.Marshal(chunk) + writeSSEData(w, data) + _ = rc.Flush() + applyTPS(r.Context(), srv.cfg.tps) + } + + // Final chunk signals stop. + final := map[string]any{ + "id": id, + "object": "chat.completion.chunk", + "created": now, + "model": model, + "choices": []map[string]any{ + { + "index": 0, + "delta": map[string]any{}, + "finish_reason": "stop", + }, + }, + } + data, _ := json.Marshal(final) + writeSSEData(w, data) + _, _ = fmt.Fprint(w, "data: [DONE]\n\n") + _ = rc.Flush() +} + +// --- OpenAI text completions (legacy) --- + +func (srv *mockServer) handleOpenAICompletion(w http.ResponseWriter, r *http.Request) { + var req inferenceRequest + body := readBody(r) + _ = json.Unmarshal(body, &req) + + model := req.Model + if model == "" && len(srv.cfg.models) > 0 { + model = srv.cfg.models[0] + } + + stream := resolveStream(req.Stream, false) + + if stream { + // Stream legacy completions using SSE - same shape as chat but with + // text field instead of delta.content. + srv.streamOpenAICompletion(w, r, model) + return + } + + now := time.Now().Unix() + resp := map[string]any{ + "id": fmt.Sprintf("cmpl-mock-%s", srv.cfg.name), + "object": "text_completion", + "created": now, + "model": model, + "choices": []map[string]any{ + { + "text": fmt.Sprintf("BACKEND:%s reply", srv.cfg.name), + "index": 0, + "finish_reason": "stop", + }, + }, + "usage": map[string]any{ + "prompt_tokens": 10, + "completion_tokens": 20, + "total_tokens": 30, + }, + } + writeJSON(w, http.StatusOK, resp) +} + +func (srv *mockServer) streamOpenAICompletion(w http.ResponseWriter, r *http.Request, model string) { + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + rc := http.NewResponseController(w) + + applyTTFT(r.Context(), srv.cfg.ttftMS) + + id := fmt.Sprintf("cmpl-mock-%s", srv.cfg.name) + now := time.Now().Unix() + n := srv.cfg.streamChunks + + for i := range n { + chunk := map[string]any{ + "id": id, + "object": "text_completion", + "created": now, + "model": model, + "choices": []map[string]any{ + { + "text": fmt.Sprintf("BACKEND:%s chunk %d", srv.cfg.name, i), + "index": 0, + "finish_reason": nil, + }, + }, + } + data, _ := json.Marshal(chunk) + writeSSEData(w, data) + _ = rc.Flush() + applyTPS(r.Context(), srv.cfg.tps) + } + + _, _ = fmt.Fprint(w, "data: [DONE]\n\n") + _ = rc.Flush() +} + +// --- Anthropic Messages API --- + +func (srv *mockServer) handleAnthropicMessages(w http.ResponseWriter, r *http.Request) { + var req inferenceRequest + body := readBody(r) + _ = json.Unmarshal(body, &req) + + model := req.Model + if model == "" && len(srv.cfg.models) > 0 { + model = srv.cfg.models[0] + } + + stream := resolveStream(req.Stream, false) + + if stream { + srv.streamAnthropic(w, r, model) + } else { + srv.nonStreamAnthropic(w, model) + } +} + +func (srv *mockServer) nonStreamAnthropic(w http.ResponseWriter, model string) { + resp := map[string]any{ + "id": "msg_mock_001", + "type": "message", + "role": "assistant", + "model": model, + "content": []map[string]any{ + { + "type": "text", + "text": fmt.Sprintf("BACKEND:%s reply", srv.cfg.name), + }, + }, + "stop_reason": "end_turn", + "usage": map[string]any{ + "input_tokens": 10, + "output_tokens": 20, + }, + } + writeJSON(w, http.StatusOK, resp) +} + +func (srv *mockServer) streamAnthropic(w http.ResponseWriter, r *http.Request, model string) { + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + rc := http.NewResponseController(w) + + applyTTFT(r.Context(), srv.cfg.ttftMS) + + n := srv.cfg.streamChunks + b := srv.bstate.get() + dropAt := (n + 1) / 2 + + // 1. message_start + writeSSEEvent(w, "message_start", map[string]any{ + "type": "message_start", + "message": map[string]any{ + "id": "msg_mock_001", + "type": "message", + "role": "assistant", + "model": model, + "usage": map[string]any{ + "input_tokens": 10, + "output_tokens": 0, + }, + }, + }) + _ = rc.Flush() + + // 2. content_block_start + writeSSEEvent(w, "content_block_start", map[string]any{ + "type": "content_block_start", + "index": 0, + "content_block": map[string]any{ + "type": "text", + "text": "", + }, + }) + _ = rc.Flush() + + if b.DropMidStream { + // Stop after content_block_start + ceil(n/2) deltas - no closing events. + for i := range dropAt { + chunk := fmt.Sprintf("The answer is 42. BACKEND:%s model:%s chunk:%d", srv.cfg.name, model, i) + writeSSEEvent(w, "content_block_delta", map[string]any{ + "type": "content_block_delta", + "index": 0, + "delta": map[string]any{ + "type": "text_delta", + "text": chunk, + }, + }) + _ = rc.Flush() + applyTPS(r.Context(), srv.cfg.tps) + } + tryHijackClose(w) + return + } + + // 3. N× content_block_delta + for i := range n { + chunk := fmt.Sprintf("The answer is 42. BACKEND:%s model:%s chunk:%d", srv.cfg.name, model, i) + writeSSEEvent(w, "content_block_delta", map[string]any{ + "type": "content_block_delta", + "index": 0, + "delta": map[string]any{ + "type": "text_delta", + "text": chunk, + }, + }) + _ = rc.Flush() + applyTPS(r.Context(), srv.cfg.tps) + } + + // 4. content_block_stop + writeSSEEvent(w, "content_block_stop", map[string]any{ + "type": "content_block_stop", + "index": 0, + }) + _ = rc.Flush() + + // 5. message_delta + writeSSEEvent(w, "message_delta", map[string]any{ + "type": "message_delta", + "delta": map[string]any{ + "stop_reason": "end_turn", + "stop_sequence": nil, + }, + "usage": map[string]any{ + "output_tokens": 20, + }, + }) + _ = rc.Flush() + + // 6. message_stop + writeSSEEvent(w, "message_stop", map[string]any{ + "type": "message_stop", + }) + _ = rc.Flush() +} + +// --- Shared streaming helpers --- + +// writeSSEEvent writes an Anthropic-style named SSE event with JSON data. +func writeSSEEvent(w http.ResponseWriter, eventType string, data any) { + payload, _ := json.Marshal(data) + _, _ = fmt.Fprintf(w, "event: %s\ndata: %s\n\n", eventType, payload) +} + +// writeSSEData writes an OpenAI-style SSE data line (no event: prefix). +func writeSSEData(w http.ResponseWriter, data []byte) { + _, _ = fmt.Fprintf(w, "data: %s\n\n", data) +} + +// writeNDJSON writes a single NDJSON line (newline-delimited JSON). +func writeNDJSON(w http.ResponseWriter, v any) { + data, _ := json.Marshal(v) + _, _ = fmt.Fprintf(w, "%s\n", data) +} + +// applyTTFT sleeps for ttftMS milliseconds before the first token. +// The sleep is context-aware so a cancelled request doesn't block. +func applyTTFT(ctx context.Context, ttftMS int) { + if ttftMS <= 0 { + return + } + select { + case <-ctx.Done(): + case <-time.After(time.Duration(ttftMS) * time.Millisecond): + } +} + +// applyTPS paces token emission. When tps > 0 it sleeps for 1000/tps ms +// between chunks to simulate realistic streaming throughput. +func applyTPS(ctx context.Context, tps int) { + if tps <= 0 { + return + } + delay := time.Duration(1000/tps) * time.Millisecond + select { + case <-ctx.Done(): + case <-time.After(delay): + } +} + +// tryHijackClose attempts to abruptly close the TCP connection to simulate a +// mid-stream backend failure. When hijack is unavailable (e.g. httptest with +// HTTP/2), it silently falls through - the truncated write still achieves the +// test goal of presenting an incomplete response. +func tryHijackClose(w http.ResponseWriter) { + hj, ok := w.(http.Hijacker) + if !ok { + return + } + conn, _, err := hj.Hijack() + if err != nil { + return + } + _ = conn.Close() +} + +// readBody reads the entire request body, returning an empty slice on error. +// Errors are intentionally swallowed because missing bodies are treated as +// empty requests in the mock - we still want to return a valid response. +func readBody(r *http.Request) []byte { + if r.Body == nil { + return nil + } + buf := make([]byte, 0, 512) + tmp := make([]byte, 512) + for { + n, err := r.Body.Read(tmp) + if n > 0 { + buf = append(buf, tmp[:n]...) + } + if err != nil { + break + } + } + _ = r.Body.Close() + return buf +} diff --git a/test/validate/config.validate.limits.yaml b/test/validate/config.validate.limits.yaml new file mode 100644 index 00000000..d9fa7889 --- /dev/null +++ b/test/validate/config.validate.limits.yaml @@ -0,0 +1,88 @@ +# Tight-limits Olla instance for the /olla-validate limits & failure agent. +# +# Deliberately small body cap and aggressive per-IP rate limit so 413 and 429 +# behaviour can be asserted in seconds without disturbing the main harness +# instance on 41141. Single backend keeps routing out of the picture. + +server: + host: "127.0.0.1" + port: 41142 + read_timeout: 20s + read_header_timeout: 5s + write_timeout: 0s + shutdown_timeout: 5s + request_logging: true + request_limits: + max_body_size: 262144 # 256KB: a ~300KB payload trips 413 cheaply + max_header_size: 65536 + rate_limits: + global_requests_per_minute: 600 + per_ip_requests_per_minute: 30 # ~burst+30 requests trip 429 in seconds + health_requests_per_minute: 600 + burst_size: 5 + cleanup_interval: 5m + trust_proxy_headers: false + trusted_proxy_cidrs: ["127.0.0.0/8"] + +proxy: + engine: "olla" + profile: "auto" + load_balancer: "least-connections" + stream_buffer_size: 8192 + connection_timeout: 5s + # Short response timeouts so hang-mode upstream tests resolve quickly. + response_timeout: 10s + read_timeout: 10s + response_header_timeout: 5s + retry: + enabled: true + on_connection_failure: true + max_attempts: 0 + sticky_sessions: + enabled: false + +discovery: + type: "static" + refresh_interval: 10s + health_check: + initial_delay: 1s + static: + endpoints: + - url: "http://127.0.0.1:19431" + name: "mock-limits-a" + type: "openai-compatible" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + model_discovery: + enabled: false + +model_registry: + type: "memory" + enable_unifier: false + unification: + enabled: false + routing_strategy: + type: "optimistic" + options: + fallback_behavior: "all" + discovery_timeout: 2s + discovery_refresh_on_miss: false + +translators: + anthropic: + enabled: true + passthrough_enabled: true + max_message_size: 1048576 + inspector: + enabled: false + +logging: + level: "info" + format: "text" + output: "stdout" + +engineering: + show_nerdstats: false diff --git a/test/validate/config.validate.yaml b/test/validate/config.validate.yaml new file mode 100644 index 00000000..f43e2877 --- /dev/null +++ b/test/validate/config.validate.yaml @@ -0,0 +1,184 @@ +# Olla validation harness configuration (/olla-validate skill). +# +# Points at seven ollamock instances (test/cmd/ollamock) so every provider +# route, the unifier, sticky sessions and the Anthropic translator have live +# mock backends. The model lists deliberately overlap (shared-model) and +# diverge (per-instance uniques) so unification and model routing decisions +# are both observable. +# +# One instance per endpoint: Olla keys endpoint identity on the URL, so two +# endpoints sharing a URL silently collapse to one (last declared wins). +# +# Topology (one ollamock process per port, started by the skill): +# 19431 mock-a openai-compatible models: test-model,shared-model +# 19432 mock-b openai-compatible models: test-model,beta-model +# 19433 mock-c ollama models: llama3.1:8b,shared-model +# 19434 mock-d lm-studio models: phi-4,shared-model +# 19435 mock-e vllm models: test-model,shared-model +# 19436 mock-f litellm models: test-model,beta-model +# 19437 mock-g llamacpp models: phi-4,shared-model + +server: + host: "127.0.0.1" + port: 41141 + read_timeout: 20s + read_header_timeout: 10s + write_timeout: 0s + shutdown_timeout: 5s + request_logging: true + request_limits: + # 5MB so the limits agent can trip 413 with a ~6MB body without the + # harness paying for huge uploads elsewhere. + max_body_size: 5242880 + max_header_size: 524288 + rate_limits: + # Generous on purpose: rate-limit behaviour is exercised against the + # separate config.validate.limits.yaml instance, not this one. + global_requests_per_minute: 100000 + per_ip_requests_per_minute: 50000 + health_requests_per_minute: 100000 + burst_size: 1000 + cleanup_interval: 5m + trust_proxy_headers: false + trusted_proxy_cidrs: ["127.0.0.0/8"] + +proxy: + # The skill runs the full suite against the olla engine; the nightly pass + # swaps this to sherpa (maintenance mode) for a trimmed second run. + engine: "olla" + profile: "auto" + # least-connections spreads traffic across instances, which gives the + # affinity and diversity assertions something real to observe. + load_balancer: "least-connections" + stream_buffer_size: 8192 + connection_timeout: 10s + response_timeout: 60s + read_timeout: 60s + response_header_timeout: 10s + retry: + enabled: true + on_connection_failure: true + max_attempts: 0 + sticky_sessions: + enabled: true + idle_ttl_seconds: 600 + max_sessions: 1000 + key_sources: [session_header, prefix_hash] + prefix_hash_bytes: 512 + +discovery: + type: "static" + refresh_interval: 10s + health_check: + initial_delay: 1s + static: + endpoints: + # ── openai-compatible pair: failover + balancing target ───────────── + - url: "http://127.0.0.1:19431" + name: "mock-openai-a" + type: "openai-compatible" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + - url: "http://127.0.0.1:19432" + name: "mock-openai-b" + type: "openai-compatible" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + + # ── ollama: native /api/* protocol ─────────────────────────────────── + - url: "http://127.0.0.1:19433" + name: "mock-ollama-c" + type: "ollama" + priority: 100 + model_url: "/api/tags" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + + # ── lm-studio ───────────────────────────────────────────────────────── + - url: "http://127.0.0.1:19434" + name: "mock-lmstudio-d" + type: "lm-studio" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + + # ── vllm: Anthropic passthrough target ────────────────────────────── + - url: "http://127.0.0.1:19435" + name: "mock-vllm-e" + type: "vllm" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + + # ── litellm: Anthropic translation target ─────────────────────────── + - url: "http://127.0.0.1:19436" + name: "mock-litellm-f" + type: "litellm" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + + # ── llamacpp ────────────────────────────────────────────────────────── + - url: "http://127.0.0.1:19437" + name: "mock-llamacpp-g" + type: "llamacpp" + priority: 100 + model_url: "/v1/models" + health_check_url: "/health" + check_interval: 2s + check_timeout: 1s + + model_discovery: + enabled: true + interval: 30s + timeout: 5s + concurrent_workers: 4 + retry_attempts: 2 + retry_backoff: 1s + +model_registry: + type: "memory" + enable_unifier: true + unification: + enabled: true + stale_threshold: 24h + cleanup_interval: 10m + routing_strategy: + # optimistic + all keeps the harness resilient to discovery timing while + # still emitting X-Olla-Routing-* headers for routing assertions. + type: "optimistic" + options: + fallback_behavior: "all" + discovery_timeout: 2s + discovery_refresh_on_miss: false + +translators: + anthropic: + enabled: true + # The nightly translation-forced pass swaps this to false to drive the + # full Anthropic <-> OpenAI translation path. + passthrough_enabled: true + max_message_size: 10485760 + inspector: + enabled: false + +logging: + level: "info" + format: "text" + output: "stdout" + +engineering: + show_nerdstats: false