P9c: AI keys and budgets change through litellm:keys and litellm:key #202

A budget or limit change is one pull request to infrastructure/litellm/keys.yaml plus mise run litellm:keys, and a new app's key is one row there plus mise run litellm:key <app>, which tells LiteLLM the value already in that app's Secret.

Caller

• P9: legal pages, person export and erasure, AI keys and expiry checks move from hal #173, slice P9c, its Done when and its invariants 1, 2 and 4 ("Commands print names, counts and outcomes; a person's data or a key's value is never printed."). Platform on k3s and Flux: from an empty PC to every app live, with hal archived #1, the guiding sentence ("AI keys and budgets") and outline row P9.
• Real callers: after hal leaves, nothing else changes a number on hopin's, presentator's or songmaker's key or the person budget. P7: Postgres, Keycloak, LiteLLM and backups run on the platform, tried on the PC with copies of hal's backups #116's cutover step 4 gives hopin its litellm_key, and this command proves that value matches the row in the loaded copy.
• The cut's example litellm:key skat is stale: skat's repository names no LiteLLM, and the front admits only hopin, presentator, songmaker and gatus.

Neighbours

P9b landed (#185) and shares infrastructure/litellm/kustomization.yaml. P9d and P9f are untouched. P9c-2, the admin command below, is a row of #173.

Predecessors

hal's key step in hal apply, its one-off litellm-keys, the [api] tables of ~/.config/hal/config.toml and ~/.secrets/litellm/apps/* keep serving hal's own LiteLLM. They leave with hal (#1, P11; #173, invariant 2). These commands act only on the platform's LiteLLM.

What binds

• Ruled, 08.11.2026 on P9: legal pages, person export and erasure, AI keys and expiry checks move from hal #173: "P9c: a new admin's unlimited AI budget is one command of his per admin." This slice does not prove it. It is deferred to P9c-2, which needs Keycloak, a person's fingerprint and a customer row. Until then P9: legal pages, person export and erasure, AI keys and expiry checks move from hal #173 stays open on that sentence.
• Ruled, P7: Postgres, Keycloak, LiteLLM and backups run on the platform, tried on the PC with copies of hal's backups #116, card "Untergruppe admin", 07.11.2026, A: "The realm keeps /services/litellm/admin as the admin budget marker". Nothing here changes the group.
• P9: legal pages, person export and erasure, AI keys and expiry checks move from hal #173's plan row: "A key or budget change is one pull request plus one command."
• The head's rule for this slice, beyond invariant 4: a key's value is never printed, never passed as an argument and never written outside its Secret. It overtakes the cut's 0600 file on P9: legal pages, person export and erasure, AI keys and expiry checks move from hal #173.

What hal does today, from hal/src/hal/stack/litellm_keys.py and hal/docs/agent-service.md: alias is the app; max_budget from [api], default 10 dollars; max_parallel_requests, default 5; budget_duration 1mo; allowed_routes the front's three; models as /v1/models lists them; no rpm or tpm; a person budget of 2 dollars per 1mo. Only changed fields are sent, so a pure limit change keeps the period's reset. The first failed write stops the rest. No reply body and no key or hash is ever printed.

Tools checked instead of building (#173, invariant 1):

• LiteLLM's config.yaml holds no keys.
• The litellm-proxy CLI at v1.106.6 offers keys list, generate, delete and import, with no update and no caller-chosen value.
• BerriAI/terraform-provider-litellm (latest GitHub release v1.108.3, listed on registry.terraform.io; not found at the registry.opentofu.org path tried) exists, which corrects Platform on k3s and Flux: from an empty PC to every app live, with hal archived #1's "no provider (unverified)". Its litellm_key generates the value and keeps it in state as key, so it can neither take the Secret's value nor keep the value out of a second place.
• A Job like config-cli would need every key's value in a second Secret (the cut, rejected).

What remains is a small client of LiteLLM's own admin routes, in bash, curl and yq, and no Python in this layer.

Proposals of this draft

• infrastructure/litellm/keys.yaml holds period, person_budget and one row per app with budget and parallel. Its first numbers are hal's code defaults. Not in the table, each from its one owner: the person budget's id (max_end_user_budget_id in litellm-config.yaml), the routes (the Path rules of front-route.yml) and the models (/v1/models).
• It reaches the cluster as a ConfigMap litellm-keys through configMapGenerator, without a hash suffix since no pod reads it. The commands read that ConfigMap and print the Git revision Flux applied. Weighed: a file in the checkout would apply stale numbers from an old PC checkout, and a clean-main guard like apply_github_settings.sh's has no path through CI.
• One script, scripts/litellm_keys.sh, behind two tasks. It reaches LiteLLM as tests/litellm/key_list_count_equals_key_table_rows.sh does: a port-forward, the master key from the Secret litellm, and curl's --config - on stdin, each step under a deadline, with KUBECONFIG defaulting to k3s's own as the keycloak: tasks do.
• litellm:keys [--plan] reads each key by alias (/key/list) and the person budget, then sends only the changed fields: /key/update with the stored hash, once exactly one row was counted under the alias, so a change of alias between the read and the write cannot redirect it, and /budget/new or /budget/update. It prints one line per change with name, field, old and new, then a count. It never reads an app's Secret. A table row without a key is named with run litellm:key <app>, and the run ends 1 after the rest. A failed write stops the rest and names what was done and what was not. --plan writes nothing and exits 0 with no change, 2 with changes and 1 on failure, as apply_github_settings.sh does.
• litellm:key <app> takes only a table row. It reads key litellm_key of Secret <app> in namespace <app>, the file hopin's manifest names. It refuses a value that is absent, lacks sk- or is under 16 characters, which is LiteLLM's rule for a chosen value, naming only the app and the condition.
  • No key under the alias: /key/generate with the value and the row's fields.
  • The stored hash equals the value's sha256: unchanged, and no write.
  • Another hash under the alias: refused, changing nothing, as hal's name taken row is.
• How a value travels. From kubectl's stdout through a shell variable to curl's stdin, in a request body, never in a URL, an argument, a temporary file or the environment of a child. The answer of /key/generate, which repeats the value, goes to /dev/null, and only the status is read. A failure names the step and the status, never a reply body. LiteLLM stores only the hash.

Gaps, each with its owner

• A replaced key value (rotation) is refused naming the app; it waits for a first need.
• rpm_limit and tpm_limit are not managed: the table has no column for them, so a loaded copy keeps whatever hal's keys hold there; AGENTS.md says so.
• The printed revision is Flux's lastAppliedRevision, its last successful apply: when Flux applied the ConfigMap and its health check then failed, the revision can be older than the table. The read before and after the table does not cover that.
• Deleting a retired app's key stays the operator's call through LiteLLM, as on hal.
• tests/litellm/key_list_count_equals_key_table_rows.sh keeps its own port-forward lines, since that script's job ends with the cutover's count; sharing them is a follow-up of this item, if the script stays.

Decided by the head, 08.11.2026

None of these changes a ruling.

• The numbers. The table starts at hal's code defaults, which spend nothing in k3d. --plan on the PC's copy lists every difference from what the operator set in hal. A write to the loaded copy is a live-data write and waits for his spoken yes on those numbers.
• A new key touches two repositories. This repository is public, so the value goes to platform-secrets through secrets:put first and the row comes here. A budget change stays literally one pull request and one command.
• A replaced key value is refused, naming the app. Nobody has asked to rotate a key. LiteLLM's /key/regenerate with new_key could do it; that waits for the need.
• P9c-2 owns the ruled admin command. It needs Keycloak, a person's fingerprint and a customer row, and shares neither table nor way in with this slice.
• AGENTS.md's line on tests/litellm/key_list_count_equals_key_table_rows.sh drops "is the operator's, since it reads the master key". The operator's ruling on Agents run every platform step that needs neither sudo nor a secret the operator fetched #140 lets an agent run every step that needs neither sudo nor a secret he fetches from outside, and the master key comes from the cluster's Secret.

Files

infrastructure/litellm/keys.yaml, infrastructure/litellm/kustomization.yaml and its header comment, scripts/litellm_keys.sh, mise.toml, tests/litellm/, .github/workflows/ci.yml, AGENTS.md.

Size: four production files and about 150 lines. It does not split, because both commands share one table and one way in, and the Done when needs both halves.

Cells

In k3d, a new step after "P9b cells", as tests/litellm/keys_table_drives_budgets_and_the_app_key.sh. It suspends the Kustomization litellm for the step and resumes it on every exit, and it adds a generated dummy litellm_key to Secret hopin, removed on every exit. Rows standing in for hal's copy are made with the master key under the aliases presentator and songmaker, with other numbers.

• --plan lists the differences, exits 2 and changes no row.
• A table row without a key is named and the run ends 1.
• litellm:keys sets the numbers on a key it did not make, and a second run prints 0 and exits 0.
• After a changed budget in the ConfigMap, only that field changes and budget_reset_at stays.
• The person budget takes its declared number under the id litellm-config.yaml names.
• litellm:key hopin makes the row with the table's fields, and a client Pod in hopin gets 200 from the front's GET /v1/models with the dummy value.
• A second litellm:key hopin prints unchanged.
• A changed value, one without sk-, a Secret without the key and an app outside the table are each refused and change no row.
• No output of any run holds the dummy value, the master key or the value's sha256.

In checks, against fake kubectl and curl, as tests/litellm/failed_write_names_what_is_done.sh: a write that fails midway stops the rest, names what was done and what was not, and prints no reply body.

Not a cell, for the reviewers: no value appears in an argument or a file, read from the script.

Risk

Material risk: secrets (the master key and app keys pass through the command), money (a budget decides what may be spent) and persistent data (it writes LiteLLM's rows). It takes the final gate on the other provider column and the scenario matrix the head writes.

PC, on the loaded copy: the layer is Ready and mise run litellm:keys --plan lists its differences by name and number. An agent runs it: the script takes the master key from the cluster's Secret and prints no value. A write follows the operator's yes on the numbers. litellm:key has nothing to do on the PC until hopin's litellm_key arrives at #116's cutover step 4.

version = 1
now = "Landed on 08.11.2026 (#211, bf570f0): keys.yaml, litellm:keys and litellm:key are on main."
next = "An agent runs litellm:keys --plan on the PC and records what it lists; a write on the PC waits for the operator's yes on the numbers."
done_when = "In k3d a changed row of keys.yaml reaches LiteLLM through litellm:keys alone with only the changed fields and a second run reports 0, litellm:key registers the value of the app's Secret so that it passes the front, and no output holds a key's value or hash; on the PC litellm:keys --plan lists the loaded copy's differences; hal's key step and key files leave with hal (#1, P11); and the ruled admin command is an open row P9c-2 on #173."

scope = [".github/workflows/ci.yml", "AGENTS.md", "infrastructure/litellm/keys.yaml", "infrastructure/litellm/kustomization.yaml", "mise.toml", "scripts/litellm_keys.sh", "tests/litellm"]

size = "M"

whole = "One table and its two commands: a key without its budget row, or a table no command applies, changes nothing a person can rely on."