AI keys and budgets change through litellm:keys and litellm:key

.github/workflows/ci.yml

@@ -66,6 +66,9 @@ jobs:
       - name: Check that legal:check passes hopin's legal files at the tag it runs, names file and line of a broken one and quotes no text
         run: mise exec -- tests/legal/legal_check_names_file_and_line_and_quotes_no_text.sh
 
+      - name: Check that a failed LiteLLM write stops the rest and names what was made, and that no key value reaches an argument or an output
+        run: mise exec -- tests/litellm/failed_write_names_what_is_done.sh
+
       - name: Check that the Renovate report check fails naming a missing report, a report with no dependency and each missing or skipped pin
         run: tests/renovate/report_check_names_each_missing_pin.sh
 
@@ -391,6 +394,14 @@ jobs:
           LITELLM_NAMESPACE: litellm
         run: tests/litellm/expire_deletes_only_what_is_past_90_days.sh
 
+      # The step suspends the Kustomization litellm, which would otherwise restore the table it changes.
+      - name: P9c cells
+        if: matrix.part == 'postgres'
+        env:
+          LITELLM_NAMESPACE: litellm
+          APP_NAMESPACE: hopin
+        run: tests/litellm/keys_table_drives_budgets_and_the_app_key.sh
+
       - name: Assert that /impressum reaches a Pod beside cloudflared through Traefik with the committed bytes and header and everything else gets 404
         if: matrix.part == 'core'
         env:

AGENTS.md

@@ -366,6 +366,20 @@
   Job made from the CronJob: it prints one deletion in each count and leaves the fresh rows and the idle customers
   with a budget or with spend, a second run prints zeros, and the first log holds no planted value; its Pod ran in a
   Namespace enforcing `restricted` with its Postgres's image.
+  The step "P9c cells", in the part `postgres`, suspends the Kustomization `litellm` and resumes it on every exit, removing
+  its keys, person budget and Secret key: `mise run litellm:keys --plan` lists the differences between the table and two
+  keys made with the master key, exits 2 and changes no row; the run sets the numbers, a second run prints 0 changes and a
+  raised budget changes that field alone and keeps `budget_reset_at`; the person budget takes the table's number; a row
+  without a key is named and ends 1 after the other rows were made equal, with the models and routes LiteLLM and the
+  front's rules name; `litellm:key hopin` makes the row from a generated value in the Secret `hopin`, a client Pod in
+  `hopin`, which reads the header line from a mounted Secret through `curl --header @file`, gets 200 from the front's `GET /v1/models` with it, a second run prints `unchanged`, and a changed
+  value, one without `sk-`, a missing key and an app outside the table are refused and change no row; no output holds a
+  value, the master key or the value's sha256. The job `checks` runs `tests/litellm/failed_write_names_what_is_done.sh`
+  against a fake kubectl and curl: a write that fails midway stops the rest, names what was made, the one that failed,
+  also when its answer was lost, and what was not made, and prints no reply body; a budget-only change sends no period;
+  a table without a number for the person budget or a row's limits, an absent revision, one that changes while the table
+  is read and a value with a newline are refused before any write; and the value reaches curl's stdin and no argument
+  or output, a run under `bash -x` included.
 - `clusters/pc/flux-system/`: Flux's own components and its sync to this repository. `gotk-components.yaml` is
   generated by `mise run flux:export` for the components `FLUX_COMPONENTS` in `mise.toml` lists, which hold no Helm reconciler
   (every chart is rendered into Git); it is not edited by hand, and `cluster` fails when a regeneration differs.
@@ -480,10 +494,25 @@
   layer's Postgres, which a replacement copies, and it prints the five counts by name, never a row; a first `select 1`,
   tried up to 30 times a second apart inside the Pod since a fresh Pod can be refused until the NetworkPolicy controller admits
   its address, shows psql's stderr, so a failed connection names itself in the log, while the statement's stderr is discarded because
-  psql quotes row values. `mise exec -- tests/litellm/key_list_count_equals_key_table_rows.sh <namespace>` is the operator's, since it
-  reads the master key: through the PC's k3s and a port-forward it prints the keys LiteLLM's `/key/list` counts and the
+  psql quotes row values. `mise exec -- tests/litellm/key_list_count_equals_key_table_rows.sh <namespace>`
+  reads the master key from the cluster's Secret: through the PC's k3s and a port-forward it prints the keys LiteLLM's `/key/list` counts and the
   key table's rows, those of the dashboard's UI session (which `/key/list` omits) and the rest, then `equal` or `DIFFERS`
   for the count against the rest; each step runs under a deadline and names itself when it fails.
+  `keys.yaml` is the table of the person budget and each app's budget and parallel limit; the ConfigMap `litellm-keys`
+  carries it, without a hash suffix, as no pod reads it. `mise run litellm:keys [--plan]` makes the person budget (the id
+  of `max_end_user_budget_id`) and each row's key equal to it, sending only the fields that differ, so a pure budget
+  change keeps the period's reset; it prints the revision Flux applied to the table it read, which it refuses when absent or changed during the read, one
+  line per field with old and new, and a count; it refuses a table whose person budget or a row's budget or parallel
+  limit is no number, naming the row; a row without a key is named with `run litellm:key <app>` and ends 1 after the
+  rest; a failed write stops the rest and names what was made, the failed write, whose outcome is unknown when its
+  answer was lost, and what was not made; `--plan` writes nothing and exits 0 with no change, 2 with changes, 1 on a
+  failure. `mise run litellm:key <app>` registers key `litellm_key` of Secret `<app>` in namespace `<app>` (one printable
+  line without a trailing newline, `sk-` and at least 16 characters) as the row's key, prints `unchanged` when the stored hash equals the value's,
+  and refuses another value under the alias, changing nothing; a key's new value goes to `platform-secrets` first. Both
+  run through `scripts/litellm_keys.sh` with a port-forward and the master key: a value and the master key travel from
+  kubectl's stdout through shell variables to curl's stdin, never in an argument, a file or a child's environment, and are
+  never printed; the reply of `/key/generate` goes nowhere; a failure names the step and the status, never a body. The
+  rpm and tpm limits are not in the table and stay as they are.
 - `infrastructure/legal/`: the imprint at `hallucinai.de/impressum`, nginx's unprivileged image (a vendor entry) in a
   Namespace that enforces Pod Security `restricted`, one pod with `Recreate` and a read-only root. `built/impressum.html`
   is hal's `legal/frame/impressum.html` byte for byte, which holds no template field, and `default.conf` serves only

infrastructure/litellm/keys.yaml

@@ -0,0 +1,16 @@
+# The budgets and limits LiteLLM holds for the apps' keys and the person budget, applied by `mise run litellm:keys`
+# (`--plan` lists the differences first); `mise run litellm:key <app>` registers an app's key from its Secret. One
+# row per app, named like the app's namespace and Secret. The person budget's id is `max_end_user_budget_id` of
+# litellm-config.yaml, the routes are those of front-route.yml and the models those LiteLLM lists. Amounts are dollars.
+period: 1mo
+person_budget: 2
+apps:
+  hopin:
+    budget: 10
+    parallel: 5
+  presentator:
+    budget: 10
+    parallel: 5
+  songmaker:
+    budget: 10
+    parallel: 5

infrastructure/litellm/kustomization.yaml

@@ -1,6 +1,6 @@
 # LiteLLM, its front and its egress, beside its Postgres from the base. LiteLLM reads its file and each Traefik its
 # static file once, at start, while each Traefik watches its dynamic directory; so each ConfigMap's name carries its
-# content's hash and a change rolls the pod.
+# content's hash and a change rolls the pod. The ConfigMap litellm-keys is the exception: `mise run litellm:keys` reads it.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: litellm
@@ -32,6 +32,12 @@ configMapGenerator:
   - name: litellm-expire
     files:
       - expire.sql
+  # No pod reads the table; the commands read it from the cluster, so its name carries no hash.
+  - name: litellm-keys
+    files:
+      - keys.yaml
+    options:
+      disableNameSuffixHash: true
   - name: agent-egress
     files:
       - traefik.yml=egress-traefik.yml

mise.toml

@@ -853,6 +853,16 @@ echo "keycloak:admin: open $admin_url/admin/master/console/ in a private window;
 exec kubectl --namespace keycloak port-forward --address 127.0.0.1 service/keycloak "${BASH_REMATCH[1]}:http"
 '''
 
+[tasks."litellm:keys"]
+description = "Make LiteLLM's person budget and the app keys' limits equal to infrastructure/litellm/keys.yaml; --plan only lists the differences (exit 2 with changes)"
+quiet = true
+run = "scripts/litellm_keys.sh keys"
+
+[tasks."litellm:key"]
+description = "Register the litellm_key of Secret <app> as the key of the row <app> in infrastructure/litellm/keys.yaml"
+quiet = true
+run = "scripts/litellm_keys.sh key"
+
 [tasks."gatus:open"]
 description = "Forward Gatus's status page to this PC's loopback and print its address"
 quiet = true