feat(agentos): discovery endpoint, OTLP gateway, cak_ ingest auth

Three server-side additions that complete the "one key" SDK contract:

Discovery (§2.6d)
- `routes/discovery.ts` — GET /agentos/api/discovery (public, no auth).
  Returns { version, endpoints: { ingest, api, otel, messages } } with
  absolute URLs derived from the request origin (X-Forwarded-* honoured;
  AGENTOS_PUBLIC_URL pins it behind a proxy). Endpoint paths live here only
  — the single source of truth — so moving a route never forces an SDK release.
- Mounted in app.ts before the auth guards (OIDC .well-known pattern).

OTLP trace gateway
- `routes/otlp.ts` — POST /agentos/api/otel/v1/{traces,metrics,logs}.
  Reads the raw request stream (no body parser) and forwards verbatim to
  OTEL_GATEWAY_UPSTREAM_ENDPOINT with OTEL_GATEWAY_UPSTREAM_HEADERS. Binary
  protobuf/gzip pass through untouched; mirrors upstream status. 503 when
  upstream unset; 16 MB cap.
- `otel-auth.ts` — fail-closed cak_ guard (invalid/missing → 401, Mongo-down → 503).
- Mounted as a SERVICE boundary before the global json parser.

cak_ ingest auth refactor
- `cak-bearer.ts` — shared `verifyCakBearer()` helper used by both
  ingest-auth and otel-auth (DRY, single validation path).
- `ingest-auth.ts` — refactored to use the shared helper; behaviour unchanged.

Tests: 164 pass (21 files), typecheck clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: abhi-bhat-lyzr

CLAUDE.md

@@ -105,10 +105,10 @@ The Mongo cluster is the source of truth for AgentOS. **Only the `agentos-server
 Resources stamped with `ownerGroup`/`ownerUser` are **hard-isolated**: a non-admin sees only their own or their group's; admins (`*`) see all (§2.6b).
 
 **Credentials required:**
-- On the **SDK** side: `AGENTOS_INGEST_URL` (e.g. `https://<host>/agentos/api/ingest/events`) + optional `AGENTOS_INGEST_TOKEN` (sent as `Authorization: Bearer …`). No Mongo creds.
+- On the **SDK** side: **`AGENTOS_DISCOVERY_URL`** (e.g. `https://<host>/agentos/api/discovery`) + the `cak_` key (`COMPUTERAGENT_HARNESS_TOKEN`). The SDK reads the ingest URL (and api/otel) from the discovery document — no endpoint path is baked into the SDK (§2.6d). A full-URL `AGENTOS_INGEST_URL` override still wins for the ingest endpoint. No Mongo creds.
 - On the **server** side: `MONGO_URL` + `MONGO_DATABASE` (this is the DB the collections above live in).
 
-**Behaviour:** when `AGENTOS_INGEST_URL` is set, the SDK's default telemetry pipeline auto-attaches `AgentOSHttpSink` (gated on the `[agentos]` extra, which is now `httpx`-based). Each event carries a stable `event_id` so the server's writes are idempotent on retry. **Ingest auth** (`ingest-auth.ts`): the SDK presents the **same `cak_` API key it uses everywhere** as the ingest `Bearer`, and the server validates it via `apiKeyStore.verify` (the same path the CAS introspection uses). A legacy static `AGENTOS_INGEST_TOKEN` string is still accepted for back-compat. ⚠️ Only when **no `cak_` is presented AND `AGENTOS_INGEST_TOKEN` is unset** does the route fall **open** (anonymous writes) — present a key (or set the token) on any network-exposed deployment. A presented `cak_` is always validated (invalid → 401, Mongo-down → 503), never waved through.
+**Behaviour:** when an ingest URL resolves (from `AGENTOS_DISCOVERY_URL` or an `AGENTOS_INGEST_URL` override), the SDK's default telemetry pipeline auto-attaches `AgentOSHttpSink` (gated on the `[agentos]` extra, which is now `httpx`-based). Each event carries a stable `event_id` so the server's writes are idempotent on retry. **Ingest auth** (`ingest-auth.ts`): the SDK presents the **same `cak_` API key it uses everywhere** as the ingest `Bearer`, and the server validates it via `apiKeyStore.verify` (the same path the CAS introspection uses). A legacy static `AGENTOS_INGEST_TOKEN` string is still accepted for back-compat. ⚠️ Only when **no `cak_` is presented AND `AGENTOS_INGEST_TOKEN` is unset** does the route fall **open** (anonymous writes) — present a key (or set the token) on any network-exposed deployment. A presented `cak_` is always validated (invalid → 401, Mongo-down → 503), never waved through.
 
 ---
 
@@ -147,7 +147,20 @@ Resources stamped with `ownerGroup`/`ownerUser` are **hard-isolated**: a non-adm
 
 **Required env:**
 - Server: `AGENTOS_CREDENTIALS_KEY` (base64 of 32 random bytes; **fail-closed** — credentials CRUD/resolve 503 without it). Optional `AGENTOS_CREDENTIALS_KEY_OLD` for rotation.
-- SDK: `AGENTOS_API_URL` (e.g. `https://<host>/agentos/api/v1`) + the same `cak_` key it already uses (`COMPUTERAGENT_HARNESS_TOKEN` / `AGENTOS_INGEST_TOKEN`). The key's role must include `git-credentials:read`.
+- SDK: `AGENTOS_DISCOVERY_URL` (the SDK reads the api endpoint from it; `AGENTOS_API_URL` overrides) + the same `cak_` key it already uses (`COMPUTERAGENT_HARNESS_TOKEN`). The key's role must include `git-credentials:read`.
+
+---
+
+### 2.6d AgentOS discovery — one SDK config, server-owned endpoint paths
+
+> So the SDK never hardcodes an endpoint path: change a route server-side and existing SDKs follow without a release. The OIDC `.well-known` pattern. Code: `routes/discovery.ts` (server), `computeragent/agentos.py` (SDK).
+
+- **Server.** `GET /agentos/api/discovery` — **public** (URLs only, no secrets), mounted before the auth guards (`app.ts`). Returns `{ version, endpoints: { ingest, api, otel, messages } }` with **absolute** URLs built from the request origin (honours `X-Forwarded-Proto`/`-Host`; pin with `AGENTOS_PUBLIC_URL`). The endpoint **paths live here only** — the single source of truth — kept in sync with the `app.ts`/`dashboard.ts` mounts.
+- **SDK.** A consumer sets just **`AGENTOS_DISCOVERY_URL`** (the full discovery URL) + the `cak_` key. `agentos.py` GETs the document once (memoized) and resolves ingest / api / otel from it — reading the **absolute URLs verbatim**, never constructing a path. Resolution order per endpoint: explicit full-URL override env (`AGENTOS_INGEST_URL`/`AGENTOS_API_URL`/`AGENTOS_OTEL_URL`) → discovery → unconfigured (no baked fallback). The remote-harness URL is **not** discovered (separate service; `harness_url=` / `COMPUTERAGENT_HARNESS_URL`).
+
+**Required env:**
+- Server: none (derives the origin from the request; optional `AGENTOS_PUBLIC_URL` to pin it behind a proxy).
+- SDK: `AGENTOS_DISCOVERY_URL` + `COMPUTERAGENT_HARNESS_TOKEN` (the `cak_`).
 
 ---
 
@@ -317,12 +330,16 @@ GITCLAW_MODEL_BASE_URL=https://api.lyzr.ai/v1
 OPENAI_API_KEY=sk-...
 
 # AgentOS persistence — SDK POSTs telemetry to the server; the server writes Mongo.
-# On the SDK (library/worker) side:
-AGENTOS_INGEST_URL=https://<agentos-host>/agentos/api/ingest/events
-AGENTOS_INGEST_TOKEN=<shared-secret>          # optional; must match the server's
-AGENTOS_API_URL=https://<agentos-host>/agentos/api/v1   # for private-GAP credential resolve (§2.6c)
-COMPUTERAGENT_HARNESS_TOKEN=cak_...           # the AgentOS API key the SDK presents (role needs git-credentials:read)
+# On the SDK (library/worker) side — ONE discovery URL + the cak_ key (§2.6d):
+AGENTOS_DISCOVERY_URL=https://<agentos-host>/agentos/api/discovery  # SDK reads ingest/api/otel from it
+COMPUTERAGENT_HARNESS_TOKEN=cak_...           # the AgentOS API key the SDK presents everywhere (role needs git-credentials:read)
+# Optional full-URL overrides (win over discovery; for pinning one endpoint):
+# AGENTOS_INGEST_URL=https://<agentos-host>/agentos/api/ingest/events
+# AGENTOS_API_URL=https://<agentos-host>/agentos/api/v1
+# AGENTOS_OTEL_URL=https://<agentos-host>/agentos/api/otel
+# COMPUTERAGENT_HARNESS_URL=https://<harness-host>   # remote harness/CAS (separate service, not discovered)
 # On the agentos-server side (NOT the SDK):
+# AGENTOS_PUBLIC_URL=https://<agentos-host>   # optional — pin the origin in discovery URLs behind a proxy
 MONGO_URL=mongodb+srv://user:pass@cluster.mongodb.net
 MONGO_DATABASE=computeragent
 # AgentOS auth / RBAC (§2.6b) — SSO via Keycloak (Okta brokered), DB-backed roles:
@@ -586,8 +603,10 @@ kustomize edit set image \
 | AgentOS auth / OIDC / BFF + refresh | `packages/agentos-server/src/auth/{oidc,authenticate,authorize,ownership,keycloak-admin}.ts`, `routes/auth.ts` |
 | Permission catalog + role seeds | `packages/agentos-server/src/auth/permissions.ts`, `stores/role-store.ts` |
 | Route composition / trust boundaries | `packages/agentos-server/src/app.ts`, `routes/dashboard.ts` |
+| AgentOS discovery (one SDK config, server-owned paths) | `packages/agentos-server/src/routes/discovery.ts` (`GET /agentos/api/discovery`, public, absolute URLs); SDK resolver `computeragent-py/src/computeragent/agentos.py` (`AGENTOS_DISCOVERY_URL`) — §2.6d |
 | Anthropic model gateway (cak_-authed proxy) | `packages/agentos-server/src/routes/messages.ts` (`POST /agentos/api/v1/messages`); smoke `computeragent-smoke/scripts/07_local_via_agentos_gateway.py` |
 | Run-an-agent-by-id (resolve handle) | `packages/agentos-server/src/routes/agents.ts` (`POST /agentos/api/v1/agents/resolve`, by `agentId`, group-scoped); SDK `computeragent-py/src/computeragent/harness/agent_resolve_client.py`; smoke `…/scripts/08_*.py` |
+| OTLP trace gateway (cak_-authed forwarder) | `packages/agentos-server/src/routes/otlp.ts` (`POST /agentos/api/otel/v1/{traces,metrics,logs}`) + `otel-auth.ts` + shared `cak-bearer.ts`. Env: `OTEL_GATEWAY_UPSTREAM_ENDPOINT`/`_HEADERS` (server-held vendor/collector creds). SDK side: `AGENTOS_OTEL_URL` → `OtelSink` auto-injects the `cak_` Bearer (`telemetry/sinks/otel.py`); smoke `…/scripts/09_*.py` |
 | Git-credential store + resolve endpoint | `packages/agentos-server/src/crypto/secret-box.ts`, `stores/git-credential-store.ts`, `routes/git-credentials.ts` |
 | SDK private-repo clone (PAT + SHA) | `computeragent-py/src/computeragent/harness/git_credential_client.py`, `substrates/local.py` |
 | Keycloak provisioning script | `packages/agentos-server/scripts/provision-keycloak.mjs` (`pnpm provision:keycloak`) |

packages/agentos-server/src/app.smoke.test.ts

@@ -51,6 +51,16 @@ describe("app routing + auth gate", () => {
     expect(j.error.code).toBe("UNAUTHENTICATED");
   });
 
+  it("401s on the OTLP trace gateway when unauthenticated (cak_-gated, fail-closed)", async () => {
+    delete process.env["AGENTOS_DEV_AUTH"];
+    const r = await fetch(`${base}/agentos/api/otel/v1/traces`, {
+      method: "POST",
+      headers: { "content-type": "application/x-protobuf" },
+      body: Buffer.from("x"),
+    });
+    expect(r.status).toBe(401);
+  });
+
   it("401s on the agent-resolve gateway when unauthenticated (cak_-gated)", async () => {
     delete process.env["AGENTOS_DEV_AUTH"];
     const r = await fetch(`${base}/agentos/api/v1/agents/resolve`, {
@@ -67,6 +77,18 @@ describe("app routing + auth gate", () => {
     expect(r.status).toBe(401);
   });
 
+  it("GET /agentos/api/discovery is public and returns absolute endpoint URLs", async () => {
+    delete process.env["AGENTOS_DEV_AUTH"]; // no auth — discovery is public
+    const r = await fetch(`${base}/agentos/api/discovery`);
+    expect(r.status).toBe(200);
+    const j = (await r.json()) as { version: string; endpoints: Record<string, string> };
+    expect(j.version).toBe("1");
+    // Absolute URLs (origin derived from the request) so the SDK does no path math.
+    expect(j.endpoints.ingest).toBe(`${base}/agentos/api/ingest/events`);
+    expect(j.endpoints.api).toBe(`${base}/agentos/api/v1`);
+    expect(j.endpoints.otel).toBe(`${base}/agentos/api/otel`);
+  });
+
   it("login redirects to Keycloak only when OIDC is configured (else 503)", async () => {
     delete process.env["AGENTOS_DEV_AUTH"];
     const r = await fetch(`${base}/agentos/api/v1/auth/login`, { redirect: "manual" });

packages/agentos-server/src/app.ts

@@ -14,8 +14,11 @@ import cors from "cors";
 
 import { requireIngestAuth } from "./ingest-auth.js";
 import { requireIntrospectionAuth } from "./introspection-auth.js";
+import { requireOtelAuth } from "./otel-auth.js";
+import { discoveryRouter } from "./routes/discovery.js";
 import { ingestRouter } from "./routes/ingest.js";
 import { keysIntrospectRouter } from "./routes/keys-introspect.js";
+import { otlpRouter } from "./routes/otlp.js";
 import { mountDashboard } from "./routes/dashboard.js";
 import { mountObs } from "./routes/obs.js";
 import { errorHandler } from "./http/error-handler.js";
@@ -31,11 +34,22 @@ export function buildApp(): Express {
     .filter(Boolean);
   app.use(cors({ origin: corsOrigins.length ? corsOrigins : false, credentials: true }));
 
+  // ── DISCOVERY — public, no auth, no secrets (OIDC `.well-known` style). The
+  // SDK is given only this URL + its cak_ key and reads every other endpoint's
+  // absolute URL from here, so route changes never force an SDK release. No body
+  // parser (GET only). Mounted first so it's reachable regardless of the guards. ──
+  app.use("/agentos/api", discoveryRouter);
+
   // ── SERVICE — machine-to-machine, own guards, BEFORE the global json/cookie ──
   // Ingest takes a larger batch body; introspection is tiny. Both bypass the
   // dashboard's cookie auth and use their own bearer guards.
   app.use("/agentos/api/ingest", express.json({ limit: "5mb" }), requireIngestAuth, ingestRouter);
   app.use("/agentos/api/keys", express.json({ limit: "16kb" }), requireIntrospectionAuth, keysIntrospectRouter);
+  // OTLP trace gateway — cak_-authed, forwards the binary OTLP body (protobuf,
+  // maybe gzip) verbatim to the upstream backend. NO body parser: the router
+  // reads the raw stream itself so bytes pass through untouched regardless of
+  // content-encoding (routes/otlp.ts).
+  app.use("/agentos/api/otel", requireOtelAuth, otlpRouter);
 
   app.use(express.json({ limit: "1mb" }));
   app.use(cookieParser());

packages/agentos-server/src/cak-bearer.ts

@@ -0,0 +1,23 @@
+// Shared `Bearer cak_…` validation — the primitive behind the machine-to-machine
+// guards (ingest, OTLP gateway). Each guard wraps this with its own policy
+// (ingest falls open / accepts a legacy token; the OTLP gateway is fail-closed).
+
+import { apiKeyStore, KEY_PREFIX } from "./stores/api-key-store.js";
+
+export type CakResult =
+  | "valid" //    a live, non-revoked, non-expired cak_ key
+  | "invalid" //  a cak_-shaped token that isn't a known active key
+  | "not-cak" //  no `Bearer cak_…` present (caller decides: legacy token / open / 401)
+  | "error"; //   the store (Mongo) couldn't be reached — caller should fail 503
+
+/** Validate the `Authorization` header's `Bearer cak_…` against `api_keys`. */
+export async function verifyCakBearer(authHeader: string | undefined): Promise<CakResult> {
+  const m = /^Bearer\s+(.+)$/i.exec(authHeader ?? "");
+  const presented = m ? m[1]!.trim() : "";
+  if (!presented.startsWith(KEY_PREFIX)) return "not-cak";
+  try {
+    return (await apiKeyStore.verify(presented)) ? "valid" : "invalid";
+  } catch {
+    return "error";
+  }
+}

packages/agentos-server/src/ingest-auth.ts

@@ -19,32 +19,29 @@
 
 import type { RequestHandler } from "express";
 import { timingSafeEqual } from "node:crypto";
-import { apiKeyStore, KEY_PREFIX } from "./stores/api-key-store.js";
+import { verifyCakBearer } from "./cak-bearer.js";
 
 function unauthenticated(res: Parameters<RequestHandler>[1]): void {
   res.status(401).json({ error: { code: "UNAUTHENTICATED" } });
 }
 
 export const requireIngestAuth: RequestHandler = async (req, res, next) => {
   const header = req.header("authorization") ?? "";
-  const prefix = "Bearer ";
-  const presented = header.startsWith(prefix) ? header.slice(prefix.length) : "";
 
   // (1) API key — the same `cak_` key the SDK uses everywhere.
-  if (presented.startsWith(KEY_PREFIX)) {
-    try {
-      const result = await apiKeyStore.verify(presented);
-      if (result) return next();
-      return unauthenticated(res); // recognized shape, but inactive/revoked/unknown
-    } catch (err) {
-      // Validation infra (Mongo) is down — fail closed, but distinguish from a
-      // bad key so the caller can retry rather than treating it as a 401.
-      console.warn("[agentos-server] ingest key verification failed:", (err as Error).message);
-      return res.status(503).json({ error: { code: "KEY_VERIFICATION_UNAVAILABLE" } });
-    }
+  const cak = await verifyCakBearer(header);
+  if (cak === "valid") return next();
+  if (cak === "invalid") return unauthenticated(res); // cak_-shaped but inactive/unknown
+  if (cak === "error") {
+    // Validation infra (Mongo) is down — fail closed, distinct from a bad key.
+    console.warn("[agentos-server] ingest key verification unavailable");
+    return res.status(503).json({ error: { code: "KEY_VERIFICATION_UNAVAILABLE" } });
   }
+  // cak === "not-cak" → fall through to the legacy/open paths.
 
   // (2) Back-compat: legacy static shared ingest token.
+  const prefix = "Bearer ";
+  const presented = header.startsWith(prefix) ? header.slice(prefix.length) : "";
   const expected = process.env["AGENTOS_INGEST_TOKEN"];
   if (expected) {
     if (presented) {

packages/agentos-server/src/otel-auth.ts

@@ -0,0 +1,20 @@
+// Machine-to-machine auth for the OTLP trace gateway (routes/otlp.ts).
+//
+// Unlike ingest (which falls open / accepts a legacy token), the OTLP gateway is
+// FAIL-CLOSED: it forwards to a vendor/collector using a server-held credential,
+// so an unauthenticated caller must never get through. The SDK presents the same
+// `cak_` key it uses everywhere as `Authorization: Bearer` (set via the OTLP
+// exporter headers). Missing/invalid → 401; store unreachable → 503.
+
+import type { RequestHandler } from "express";
+import { verifyCakBearer } from "./cak-bearer.js";
+
+export const requireOtelAuth: RequestHandler = async (req, res, next) => {
+  const cak = await verifyCakBearer(req.header("authorization"));
+  if (cak === "valid") return next();
+  if (cak === "error") {
+    return res.status(503).json({ error: { code: "KEY_VERIFICATION_UNAVAILABLE" } });
+  }
+  // "invalid" or "not-cak" → reject (fail-closed; no legacy/open fallback).
+  return res.status(401).json({ error: { code: "UNAUTHENTICATED" } });
+};

packages/agentos-server/src/routes/discovery.test.ts

@@ -0,0 +1,59 @@
+// Unit test for the discovery document. Mounts the router in isolation and
+// drives it via fetch. The absolute URLs must reflect the request origin
+// (honouring X-Forwarded-* behind a proxy) or the AGENTOS_PUBLIC_URL override.
+
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import express from "express";
+import type { Server } from "node:http";
+
+import { discoveryRouter } from "./discovery.js";
+
+let server: Server;
+let base = "";
+
+beforeEach(async () => {
+  const app = express();
+  app.use("/agentos/api", discoveryRouter);
+  await new Promise<void>((r) => {
+    server = app.listen(0, "127.0.0.1", () => r());
+  });
+  const addr = server.address();
+  base = `http://127.0.0.1:${typeof addr === "object" && addr ? addr.port : 0}`;
+});
+
+afterEach(() => {
+  server?.close();
+  delete process.env["AGENTOS_PUBLIC_URL"];
+});
+
+describe("discovery document", () => {
+  it("returns absolute URLs from the request origin + a cache header", async () => {
+    const r = await fetch(`${base}/agentos/api/discovery`);
+    expect(r.status).toBe(200);
+    expect(r.headers.get("cache-control")).toContain("max-age=300");
+    const j = (await r.json()) as { version: string; endpoints: Record<string, string> };
+    expect(j.version).toBe("1");
+    expect(j.endpoints.ingest).toBe(`${base}/agentos/api/ingest/events`);
+    expect(j.endpoints.api).toBe(`${base}/agentos/api/v1`);
+    expect(j.endpoints.otel).toBe(`${base}/agentos/api/otel`);
+    expect(j.endpoints.messages).toBe(`${base}/agentos/api/v1/messages`);
+  });
+
+  it("honours X-Forwarded-Proto/Host so URLs are correct behind an ingress", async () => {
+    const r = await fetch(`${base}/agentos/api/discovery`, {
+      headers: { "x-forwarded-proto": "https", "x-forwarded-host": "agentos.example.com" },
+    });
+    const j = (await r.json()) as { endpoints: Record<string, string> };
+    expect(j.endpoints.ingest).toBe("https://agentos.example.com/agentos/api/ingest/events");
+    expect(j.endpoints.otel).toBe("https://agentos.example.com/agentos/api/otel");
+  });
+
+  it("AGENTOS_PUBLIC_URL pins the origin and trims a trailing slash", async () => {
+    process.env["AGENTOS_PUBLIC_URL"] = "https://canonical.example.com/";
+    const r = await fetch(`${base}/agentos/api/discovery`, {
+      headers: { "x-forwarded-host": "ignored.example.com" },
+    });
+    const j = (await r.json()) as { endpoints: Record<string, string> };
+    expect(j.endpoints.api).toBe("https://canonical.example.com/agentos/api/v1");
+  });
+});