Activity

1. FlexOr2

   added the

   Container

   issue type on Nov 6, 2026
2. FlexOr2 commented on Nov 6, 2026

   

   FlexOr2

   on Nov 6, 2026

   MemberAuthor

   More actions

   Evidence: architecture analysis by Opus 5.5 at max effort, 06.11.2026. It was run read-only against hal at bac221a1, from the brief in hal head chat, and is posted unchanged. It recommends k3s plus Flux. One correction: its gate S6, the agent service's containment, no longer applies, because hal#1086 removes that service.

   ---

   hal's boundaries, and whether to run on k3s (read-only analysis, 06.11.2026)

   Repository read: /home/felix-hummert/git/hal-worktrees/arch-boundary-analysis (bac221a1). I changed no file and ran nothing live. Where I could not confirm something, it says "unverified".

   Short answers

   • If hal is finished, do we have k3s? No. A finished hal is a deploy controller for Compose on one machine with safety rails. That is Flux's job, done without Flux's community. It never gets a scheduler, network policy, RBAC, multi-node or rolling updates. Groups 2–6 below are that controller: 8,652 of 28,617 src lines (30%). Its own spec is docs/architecture.md §8, "The applied record".
   • hal breaks its own guiding rule (G1: the shipped command applies the declared security and merge settings of the managed repositories #20) today. The rule is "hal builds nothing a maintained tool already does". Four areas break it: the controller (groups 2–6), backup scheduling (11), the check framework (12) and the bootstrapper (15). The parts that are really hal's own are legal pages, the people contract (CI's reads of GitHub's meta API carry the job's token #256), LiteLLM keys and expiry, the secret checks, the restore test and the agent account. No tool does those, k3s included.
   • "Switch today, which command?" Two lines would do it: curl -sfL https://get.k3s.io | sh - and flux bootstrap github …. But nothing would run afterwards. No image is in a registry and no manifests exist. Slices S1–S9 below build exactly that.
   • Recommendation: (C) k3s plus Flux. The path starts with slices that also pay off on Compose: private GHCR images, a mise bootstrap, SOPS secrets and logs. If he stops after S4, he has (B) and nothing is wasted.
   • One hard gate comes before any stateful move. The agent service's containment must be re-proven on k3s in a VM. If it fails, stop at (B).
   • What is left of hal is a small toolbox for its own jobs, about 10% of today's src (estimate). That is (D) in all but name.
   • Shipping afterwards is a tag push in the app repo plus one merge, or zero commands with auto-merge (section 6).

   1. Inventory

   Line counts come from wc -l over src/hal, 28,617 lines in total. Tests are about 79.8k lines and probes about 29k (rough counts by file name).

   #	Job (modules, lines)	Who should own it	Named tool (source)	Does hal rebuild it today?
   1	Catalog: stack/catalog.py (622)	hal's own: the one description	OpenTofu already reads fleet.yaml (external/*.tf)	Partly. run.enabled/setup/build duplicate Compose profiles or Flux Kustomizations. Hosts, exposure, deny_public and legal have no tool equivalent.
   2	Unit reconcile: apply 779, unit_plans 707, core 523, compose 411, core_phase 216, service_recreate 208, built_image 133, environment 118, setup 111, plan 68 (= 3,274)	Compose itself; Kubernetes standard	docker compose up -d --wait recreates on config or image change and keeps volumes (Compose up). Flux kustomize-controller has prune, wait, healthChecks, dependsOn (Kustomization).	Yes. It compares config hashes per container, keeps followers and an applied record, and handles "created, never started" (docs/architecture.md §8).
   3	App deploy: apps 594, app_contract 341, app_compose 172, deploy_checkout 152, releases 67 (= 1,326)	Maintained tool; contract is own policy	Flux plus a digest in Git; on Compose, doco-cd or Komodo	Yes for deploy (apps.py _run: checkout tag, build, create, start, wait, record). No for the contract's rules, but enforcing them is standard (PSS, CI).
   4	Live checkout and update: checkout 477, update 378, stack/cli 128, stack_commands 153, apply_lock 40, waiting 28, terminal 23 (= 1,227)	Kubernetes standard	Flux source-controller	Yes. hal update is a Git pull behind a yes (stack/update.py docstring).
   5	Plan and result rows: plan_rows 613, result_rows 527 (= 1,140)	The PR diff; flux get	GitHub PR, Flux CLI	Yes. It exists only because row 2 exists.
   6	Emergency stop, barrier, switches, tunnel: emergency 799, barrier 424, switches 198, switch_events 160, tunnel 104 (= 1,685)	Kubernetes standard; systemd	A failed readiness probe removes the pod from EndpointSlices (probes); flux suspend (CLI); systemctl stop cloudflared	Yes in Kubernetes terms. On Compose no maintained tool keeps an "off" through a redeploy, so (B) would need a small copy of it.
   7	Realm and snapshot first: realm/observe 799, realm/reconcile 462, realm/__init__ 104, realm/declared 97, pre_apply 141 (= 1,603)	Maintained tool plus standard hooks	keycloak-config-cli is already used; it is idempotent, keeps an import checksum and ships a Kubernetes Job example (repo). A Helm pre-upgrade hook runs "before any resources are updated" (hooks). The Keycloak Operator's realm import "does not update or delete" (doc), so it is no substitute.	Partly. Triggers and ordering are what hooks do. observe (list what exists only in Keycloak) is hal's own; whether config-cli has a diff mode is unverified.
   8	External publish: external/*, 10 files (= 2,407)	Maintained tool	OpenTofu with the Cloudflare provider, run as a mise task or in CI	Partly. The wrapper (env allowlist, plan handling) is glue. host_check/host_probe (the ready guard) are hal's own but moot behind readiness. cloudflare_read (undeclared records) is hal's own.
   9	Legal and appkit: legal_section 421, legal_step 422, legal_pages 299, appkit_step 213, legal_commands 65 (= 1,420)	hal's own	None. In the target it is a CI job that builds an nginx image.	No.
   10	LiteLLM: litellm_keys 489, litellm_admin 429, api_limits 59, litellm_expiry 156, expire_commands 104 (= 1,237)	Own glue on LiteLLM's admin API	LiteLLM's retention setting covers only spend logs (doc), not the four tables litellm_expiry.py deletes. I found no LiteLLM Terraform provider (unverified).	No.
   11	Backup and restore: backup/* plus backup_commands, 13 files (= 3,392)	Maintained tool; restore test is hal's own	resticprofile has profiles, hooks, schedules and retention (docs). docker-volume-backup stops containers during backup by label (docs). k8up streams pg_dump via k8up.io/backupcommand, schedules backup, check and prune, and has a Restore resource, but no restore test (docs).	Yes for orchestration, schedules, retention and status. No for the restore into a throwaway Postgres (backup/verify.py).
   12	Daily check, alarm, mail: check 707, hook_check 200, platform_check 160, report 139, backup_check 131, logs_check 131, mail 111, alarm 89, exposure_check 24, check_commands 135, alarm_commands 44 (= 1,871)	Maintained framework; own probes	Gatus: checks, conditions, endpoints pushed with a heartbeat, email alerts (repo). Prometheus plus Alertmanager (#4).	Yes for scheduler, aggregation and mail. No for the hook, secret and log probes.
   13	Exposure: exposure/* plus exposure_rows, 9 files (= 817)	hal's own on Compose; mostly standard in (C)	Pod Security Standards Baseline forbids hostPath, hostPorts and hostNetwork (PSS); plus a host firewall	No on Compose. In (C) it shrinks to a host-listener check.
   14	Secrets: secrets_view 822, secrets_setup 767, secrets_rows 261, hidden_paste 131, manifest_fields 58, secrets_commands 53, secrets_create 49 (= 2,141)	Maintained tool for storage; checks are hal's own	SOPS with age (docs); Flux decrypts it natively (guide)	Partly. The wizard writes files sops edit would hold. The mode and expiry checks are hal's own.
   15	Workstation setup: setup 740, system_step 421, user_units 347, layout 112, jobs 111, path_link 93, ports 84, hal_events 80, setup_commands 68, tunnel_config 40 (= 2,096), plus system/playbook.yml (546)	Maintained tools	mise: tools, lockfile, tasks (lock, tasks); Ansible is already used	Yes. It is a bootstrapper around both (setup.py even pins MISE_VERSION itself).
   16	Rename and migrate: rename_commands 799, rename_rows 182, migrate_commands 142, project_move 117 (= 1,240)	Transitional	none	No; it dies after the rename window (#835).
   17	CLI, output, config, version (= 1,119)	Shell of the CLI	none	Shrinks with the CLI.
   none	Outside src: keycloak/go-live 937, step-ca/init 540, agent-service/setup 546	One-time setups	Jobs or Ansible	No.
   none	Fleet: songmaker scripts/auto-deploy.sh (1,771 lines) plus two units	none	Same as row 3	Yes. A second deploy controller in the fleet (#1016, #660).

   hal's own docs say it: Compose swap cost is "high for edge (apply, check, backup labels, barrier and the Traefik provider all speak Compose)" (docs/architecture.md §10, swap table).

   2. The boundary

   hal is:

   • the one description of what is public and which rules hold (fleet.yaml, read by OpenTofu and the legal build);
   • the repository that holds the desired state;
   • a few own-domain tools.

   hal must never:

   • start, stop, recreate or order containers;
   • decide health, rollout or rollback;
   • keep markers about the runtime's state;
   • wrap a maintained tool's CLI to restate its output;
   • build images on the PC.

   What hal hands the platform is one Git commit on main:

   • manifests with digest-pinned images, which Flux pulls;
   • SOPS-encrypted Secrets from a private repo;
   • fleet.yaml, which OpenTofu applies to Cloudflare.

   What the platform owns alone: running, health, rollout, rollback, routing, secret decryption, volumes, backup schedules, log and metric retention, and alerts.

   The same boundary holds under (B); only the component that pulls the commit changes. That is why the boundary work is never wasted.

   flowchart LR
     subgraph GH["GitHub, org overnightworks (Team)"]
       APP["app repo, public<br/>git tag vYYYY.MM.DD.N"]
       WF["shared image workflow in quality<br/>build, SBOM, provenance"]
       REG[("GHCR, private<br/>image@sha256")]
       PR["bump PR in hal<br/>CI: attestation, policy, legal build"]
       HAL["hal repo = desired state<br/>cluster/, fleet.yaml, legal/, realm, own tools"]
       SEC["private repo<br/>SOPS-encrypted secrets"]
     end
     subgraph PC["the PC"]
       subgraph K3S["k3s: the platform alone runs, heals, rolls out, routes, stores"]
         FLUX["Flux controllers"]
         WL["apps, Keycloak + Postgres,<br/>LiteLLM, agent service, step-ca"]
         OPS["k8up, Gatus, Loki + Alloy, Grafana"]
         JOBS["hal jobs: LiteLLM expiry, restore test,<br/>secret check, daily summary"]
         TR["Traefik on 127.0.0.1"]
       end
       CFD["cloudflared, systemd<br/>global off switch"]
       OPR["operator: git, gh, mise run"]
     end
     TOFU["OpenTofu, mise task"]
     CF["Cloudflare: DNS, Access,<br/>tunnel config, rulesets"]
     APP --> WF --> REG
     WF --> PR
     OPR -- "merge = the yes" --> PR
     PR --> HAL
     HAL -- "Flux pulls main" --> FLUX
     SEC -- "pulled, decrypted" --> FLUX
     FLUX --> WL
     FLUX --> OPS
     FLUX --> JOBS
     REG -- "pulled by digest" --> WL
     HAL -- "fleet.yaml" --> TOFU --> CF
     CF --> CFD --> TR --> WL
   

   Loading

   3. The alternatives

   • (A) Compose plus hal as it is.
   • (B) Compose plus maintained tools, with hal thin.
   • (C) k3s plus Flux.
   • (D) No hal at all; same platform as (C).
   • (E) Podman Quadlet. I considered it seriously and rejected it: podman auto-update rolls back a failed restart by default (man page), but Git pull and digest pins would again need hal-like glue.
   • Argo CD vs Flux: both are CNCF graduated (2022). Flux wins here: it is CLI-first, has SOPS built in, and has suspend, Helm hooks and remediation. Argo CD's core install drops its UI, RBAC and notifications (core).

   3.1 Day to day

   Task	(A)	(B)	(C) and (D)
   Ship an app version	Push a tag in the app repo, then hal apply and yes on the PC. songmaker deploys itself by its own timer.	Push the tag. CI builds the image and opens the bump PR. Run gh pr merge <n> --squash. The Compose deploy tool runs the update.	Push the tag, merge the bump PR (or let auto-merge do it). Flux applies on its next interval; to apply now: flux reconcile ks <app> --with-source.
   Change a config	PR and merge, then hal update (yes), then hal apply (yes)	PR and merge	PR and merge; Flux applies it with no release tag (#987)
   Add an app	fleet.yaml entry, backup/apps.toml, realm client, the app's legal part; then hal update and hal apply	apps/<name>/compose.yaml, catalog entry, backup profile; merge; mise run tofu:apply	cluster/apps/<name>/ (start from kompose convert, Kompose), host entry, k8up Schedule, realm client; merge; mise run tofu:apply
   Restore after a broken update	Keycloak and LiteLLM: hal restore <unit> --live from the named pre-apply snapshot (§8). An app has no rollback today (#660).	git revert <bump> && git push (Compose has no rollback option). Data: restic restore, then docker compose up -d.	Stateless: git revert <bump> && git push. Stateful: flux suspend hr keycloak, then kubectl -n keycloak scale deploy keycloak --replicas=0, then mise run restore -- keycloak <snapshot>, then revert, then flux resume hr keycloak.
   Emergency stop of one app	hal stop <app>; everything public: hal offline	docker compose -p <app> stop. It stays off only if the deploy tool honours a hold (doco-cd: unverified).	flux suspend ks <app> && kubectl -n <app> scale deploy --all --replicas=0, back with flux resume ks <app>. This holds while Git does not declare suspend (Flux). Everything public: sudo systemctl stop cloudflared.

   3.2 Cost on this machine (12 cores, 60 GB)

   	Memory and CPU	Upkeep	Failure modes	Learning
   A	Nothing extra	28.6k src plus about 109k test and probe lines	Controller races (#409, #1081, #908); no app rollback; ~/.secrets in no backup (docs/architecture.md §6)	hal's own semantics
   B	Plus a log store, Gatus and a deploy tool. Komodo needs Core, Periphery and MongoDB or FerretDB (setup); doco-cd is one container (about 1.8k stars, repo). Sizes unverified.	Several small tools plus glue for the rails	No rollback; persisted "off" and snapshot-first stay hal's own	Low
   C/D	k3s server with workload: 1,596 MB and about 6% of a core (profiling). Flux and the observability stack: unverified. Estimate 3–5 GB in total.	k3s upgrades pinned with INSTALL_K3S_VERSION (env); chart bumps by Renovate (flux manager); datastore backup (doc)	local-path uses reclaimPolicy: Delete (manifest); Flux prune deletes what leaves Git unless the resource carries kustomize.toolkit.fluxcd.io/prune: disabled; Docker and k3s together (iptables, unverified); 6443/10250 bind scope (unverified)	High for the operator, familiar to agents. A common pattern: onedr0p/cluster-template

   What of hal dies:

   • (A): nothing.
   • (B): rows 2–5, the deploy part of 3, 11's scheduling, 12's framework, 14's wizard and 15. About 55–65% of src (estimate).
   • (C)/(D): all of the above plus 6, 7's reconcile, 8's wrapper, 13 and 16. About 85–90% (estimate). What remains: legal about 720–900 lines, LiteLLM about 1,000, snapshot and restore test about 300, secret checks about 300, a small CLI.

   On D14 (agents are root through the docker group): k3s does not fix this by itself. Through a bind mount, an agent can still read the root-only kubeconfig. What (C) adds is RBAC: a limited kubeconfig for the agents' account, which playbook.yml already creates dormant, once that account leaves the docker group.

   3.3 Risks for stateful services

   	Keycloak with Postgres	App databases	Backup and restore
   A	Proven: snapshot first plus restore test (stack/pre_apply.py)	Proven; no pause during the dump (#678)	Same disk only; D3 not built
   B	Snapshot-first must be redone as a deploy hook	docker-volume-backup's stop label solves #678	resticprofile; must be re-proven
   C/D	One dump-and-restore migration. Use a StatefulSet on a Retain StorageClass with prune disabled. A pre-upgrade hook takes the snapshot and runs the restore test; a post-upgrade hook runs config-cli. No automatic rollback because of schema migrations (Keycloak downgrade: unverified).	Restore into PVCs. The GPU worker uses runtimeClassName: nvidia (k3s). local-path has no capacity limits (README).	k8up. Not Velero: its file-system backup does not support hostPath and needs object storage (doc). CloudNativePG only later, once D3 has an object store; it does not manage pg_dump (doc).

   3.4 Logs and the abuse trail

   Today's settings explain the unexplained 401:

   • Docker logs go to journald, vacuumed at 28 days (system_step.py: DOCKER_LOG_DRIVER, JOURNAL_VACUUM_DAYS).
   • All three Traefik configs have only log: and no access log (traefik.yml, agent-service/litellm/front/traefik.yml, agent-service/egress/traefik.yml).
   • LiteLLM logs nothing useful: turn_off_message_logging, disable_spend_logs and disable_error_logs are on (agent-service/litellm/config.yaml), and LITELLM_LOG: CRITICAL is set (compose.yaml).

   The gateway lines are #1084, not planned here.

   	Where lines live	Retention (#167)	Abuse trail
   A	journald only; search with journalctl	Met by the daily vacuum	Missing at the edge, for identity and on the AI route
   B/E	journald, collected by Alloy's loki.source.journal into Loki (Promtail is end of life since 2026-03-02, doc), or VictoriaLogs (doc); searched in Grafana	Loki's compactor with retention_period: 720h; the default 0s keeps logs forever (doc). VictoriaLogs -retentionPeriod=30d (its default is 7d).	All five sources below
   C/D	Container output in /var/log/pods. Kubernetes rotates these by size only (10Mi × 5), has no log storage of its own, and logs vanish with evicted pods (doc). An Alloy DaemonSet ships them to Loki.	As in B, plus an age rule for /var/log/pods: a quiet container's live log file can hold lines older than 30 days. Whether the kubelet can rotate by age is unverified.	Five sources, plus Flux events and the Kubernetes audit log (optional)

   The five sources of the abuse trail:

   1. Edge. Traefik JSON access log, keeping the Cf-Connecting-Ip and Cf-Ray headers (that the tunnel passes them on is unverified). Set fields.queryParameters.defaultMode: drop, because the default is keep and Keycloak's ?code= would be logged. Headers are dropped by default (doc).
   2. Identity. Keycloak events (configuration unverified).
   3. AI route. The gateway (#1084).
   4. Apps. Audit lines carrying the person's sub.
   5. Platform changes. Git history.

   Fleet-wide rule (it lives in quality, not in hal code):

   • Logs are JSON lines on stdout.
   • Never log content, prompts, keys, bearer tokens, cookies or query strings.
   • A person appears only as their Keycloak sub, as docs/people.md already does.
   • IP addresses appear only in the edge log.
   • Retention is 30 days everywhere, and logs never go into backups.
   • The log UI sits behind Keycloak and Cloudflare Access.
   • The privacy notice names the access log (#1027).

   4. Recommendation: (C) through a (B) prefix, ending as (D) in substance

   Why the complexity earns its keep:

   1. It deletes about 8.7k controller lines, the realm, backup and check frameworks, and songmaker's 1,771-line deployer. Standard, public complexity replaces them.
   2. hal's rails all have standard equivalents. Snapshot first becomes Helm hooks. The barrier and ready guard become readiness. The switch becomes suspend. Rollback becomes git revert, which apps lack today (#660). Hardening becomes PSS; isolated networks become NetworkPolicy.
   3. The re-ask triggers of G1: the shipped command applies the declared security and merge settings of the managed repositories #20 get cheap. A second machine (#755) becomes a k3s agent join; more apps cost no new code.
   4. Now is the cheapest moment: there are no users and no important data.
   5. Slices S1–S4 are useful on Compose too.

   What would change it:

   • S6 is red;
   • Docker and k3s cannot share the host cleanly in the VM;
   • the operator rejects kubectl in daily use after the S5 rehearsal.

   In each case: stay on (B). A GHCR price change only changes the registry.

   Slices. Each lists what is added, what of hal is deleted, and how it is proven without touching the live system.

   Slice	Adds	Deletes in hal	Proof without the live system
   S0	The operator re-rules #20; controller work is frozen	none	none
   S1 images	The image pipeline (details below); songmaker moves onto the same path (#1016)	built_image.py, run.build, the build branch of apps.py and app_compose.py, appkit_step.py; legal_step.py becomes a CI step; deploy_checkout.py; songmaker's auto-deploy.sh	Build on a PR without push; test tag on speech-runtime; gh attestation verify; a pull with the read-only token inside scripts/system-proof/; the Keycloak image starts in CI
   S2 bootstrap	bootstrap.sh: mise pinned by sha256, mise install --locked, mise run setup (Ansible). mise.toml and mise.lock with kubectl, flux, helm, sops, age, opentofu, gh, k3d, kubeconform, restic.	Most of workstation/setup.py, setup_commands.py, path_link.py; the units move into Ansible	scripts/system-proof/ (Ubuntu 26.04 container with systemd) runs bootstrap.sh in CI
   S3 secrets	SOPS with age; a private repo for the encrypted files; sops -d writes today's file paths; a small check script that also watches token expiry	secrets_setup.py, hidden_paste.py, secrets_create.py, most of secrets_view; closes "~/.secrets in no backup"	CI decrypts with a throwaway key; the operator compares sha256sum of decrypted and current files
   S4 logs and checks	Access logs, Keycloak events, Loki with Alloy, Grafana, Gatus, the fleet log rule	platform_check, report, mail, alarm, alarm_commands; check.py shrinks	A CI compose job sends ?code=secret and an Authorization header and asserts the stored line contains neither
   S5 cluster skeleton	cluster/pc, plus an infrastructure layer. Traefik runs with hostNetwork: true and an entry point on 127.0.0.1:<port> (chart, entry points). Also a Retain StorageClass, k8up, Gatus, Loki, Grafana, and an Ansible role for k3s with --disable=traefik --disable=servicelb (doc) plus a firewall.	none	kubeconform and kustomize build on every PR; a k3d job (k3d) installs Flux and waits until everything is ready; a VM with both Docker and k3s
   S6 gate	The agent service in the VM under PSS restricted: read-only root, drop ALL, memory emptyDir, PVCs for the CLI logins. The egress gateway gets a pinned ClusterIP plus hostAliases, with dnsPolicy: None. NetworkPolicies are default deny, enforced by k3s's embedded kube-router netpol (doc). step-ca and traefik-mcp stay, since cert-manager's CA issuer is "for … demos or for advanced users" (doc).	none	The intents of the 16k-line probe/agent-service ported to the VM: no DNS, no direct egress, no foreign SNI, no reach into other namespaces, the smoke green. Red: stop at (B).
   S7 stateless	legal, speech-runtime, hopin, with readiness on the catalog's ready paths	none	VM end-to-end tests through 127.0.0.1 with Host headers
   S8 stateful	Keycloak as a local chart with pre- and post-upgrade hook Jobs and no auto-rollback; Postgres StatefulSets; LiteLLM with a key Job and an expiry CronJob; k8up to a local backend (mountPath, API); a restore-test CronJob (verify.py logic)	none	The operator copies the latest restic snapshots into the VM; restore; config-cli reports no change; a deliberately broken tag shows the hook snapshot and that restore works
   S9 songmaker, presentator	GPU worker, Postgres 16, Redis, audio PVC; songmaker's own Prometheus and Grafana :latest join #4; presentator's SQLite PVC; closes gap 35	none	VM end-to-end test, nvidia-smi in a pod, one song generated
   S10 cutover (the one live step)	hal offline, final backups, docker compose -p hal stop and the same for each app project, mise run k3s:install, flux bootstrap github --owner=overnightworks --repository=hal --branch=main --path=cluster/pc (doc), restore, check on loopback, systemctl start cloudflared	none	Rollback: /usr/local/bin/k3s-uninstall.sh ("deletes the local cluster datastore", doc), then docker compose -p hal start; the old volumes are untouched until S11
   S11 delete	none	After two stable weeks and one green restore test: rows 2–6 with their tests, realm reconcile, the backup runner, exposure, the external wrapper, rename and migrate (#835), docs/stack.md, most of docs/architecture.md §8	The CI suite of the remaining toolbox

   S1 image pipeline in detail:

   • One shared workflow. It lives at overnightworks/quality/.github/workflows/image.yml, is pinned by SHA, and each repo's release.yml calls it on v* tags after CI is green. A shared reusable workflow is GitHub's documented route to SLSA v1.0 Build Level 3 (attestations).
   • Build settings. docker/build-push-action with push: true, provenance: mode=max and sbom: true. Build args are visible in the provenance, so secrets go in as mounts (Docker). Then actions/attest with push-to-registry: true and the permissions id-token: write, attestations: write, packages: write.
   • Registry. GHCR packages are private by default, and org owners can restrict creation to private only. A package once made public cannot be made private again (access).
   • Cost on the Team plan.
     • GHCR storage and bandwidth are "currently free", with at least a month's notice before any change (billing).
     • Standard Actions runners are free on public repos; larger runners are billed (Actions).
     • Attestations are available on Team only for public repos. Ours are public, so they work, but image names and digests land in the public Sigstore log. The image content stays private.
   • How the PC authenticates. GHCR accepts only a classic PAT. Pulling needs read:packages and nothing more (GHCR). Set an expiry and let the secret check watch it.
     • On Compose: docker login ghcr.io.
     • On k3s: /etc/rancher/k3s/registries.yaml, which needs a restart (doc).
     • A machine user would narrow the token's reach but costs a seat (unverified).
   • Digest pinning and the bump PR.
     • Every image is pinned as ghcr.io/overnightworks/<app>:<tag>@sha256:<digest>.
     • The workflow's last job opens the bump PR in hal using a GitHub App token, so the digest is exact.
     • hal's CI runs gh attestation verify oci://… -R overnightworks/<app>.
     • Merging is the yes. Under #1016, auto-merge on green.
     • Base images and actions are bumped by Dependabot, or by Renovate, which covers docker-compose, flux and mise.
   • Special images.
     • Keycloak: theme plus appkit, built with kc.sh build and run with start --optimized (containers); this fixes #908.
     • Agent service: stays private (licensing unverified).
     • Legal: built from the pinned app tags.
     • songmaker's CUDA images: runner disk and build time unverified; prove on a PR build first.
   • Supply-chain levels.
     • Required: build in CI only; digest pins; private-only packages; a read-only pull token with expiry; actions pinned by SHA; least-privilege permissions:; base images pinned and bumped by a bot; no secrets in build args.
     • Recommended: provenance attestation plus verification; SBOM; a vulnerability scan report.
     • Optional: cosign; admission-time verification on k3s; pruning of old package versions.

   Own-domain jobs and their homes (the "no hal" question):

   Job	Home in the target
   Legal pages	A CI job using legal_section.py and legal_pages.py; output is an nginx image
   Person export and erasure (#256)	A small CLI: mise run people -- export <name>
   Secret checks	A CronJob that pushes to Gatus
   Agent account	Ansible (already in playbook.yml)
   Daily check	Gatus plus a summary job (#415)
   LiteLLM budgets	A key Job and an expiry CronJob, from today's code
   Restore test and snapshot	A toolbox image used as a CronJob and a hook
   Terms re-acceptance (#1029)	A small script

   No single tool does these jobs. They are about 2.5–3k lines either way. Keep them in one small package with one test suite rather than six scripts.

   5. Open items

   • Superfluous:
     • The controller's apply, stop, status and output: kustomize-controller polls a waiting Kustomization every 5 s and runs ten reconciles at once #361, #1063, #595, most of P5: docs/github-settings.md holds the six ruled sentences on what src/opentofu/github declares and how a change reaches GitHub, and AGENTS.md loses them #296, #1081, presentator's web holds its secrets and its service-account token in one volume #409, The platform shows presentator's legal texts of v2026.11.10.1 #395, The realm declares the agent client agent-leonardo #394, Only attested app images and digest-pinned listed vendor images start on the PC #50.
     • Runtime state names: #790, #951, #819.
     • The secrets wizard: The cluster job's limit is 15 minutes with room for a slow runner #407, #991, #676.
     • #660, which becomes the digest pin plus auto-merge.
     • #1071, since the check and alarm timers go.
     • #1079 is already closed.
   • Changed:
     • #1016: tag, image, bump PR; songmaker's deployer goes.
     • #1051: git tag as a mise task.
     • #1082, #1037 and #908: baked into images.
     • #987: values in Git.
     • #610: a Helm pre-upgrade hook.
     • #678: a backup hook.
     • #639: the cause is .github/dependabot.yml ignoring semver-minor, which is Postgres's in-series step. Fix the bot rule.
     • #615 and #1056: owned by quality and the bootstrap.
     • skat runs release 2026.11.09.1 after a backup, with its new legal texts #291, The GitHub configuration drops scratch-landing-drive #374, The app blocks come from a declared list and all three legal pages are built #233 and The Kustomization apps stops pruning before one Kustomization per app replaces it #147: bootstrap, mise and Ansible; PSS.
     • S1b-2: the PreBackupPod stages songmaker's files beside the dump #280 and The platform shows presentator's legal texts of v2026.11.10.1 #396: manifest policy in CI.
     • P1: every app release yields an attested private image in GHCR #4, presentator has its repository on the rest server and a monthly restore test of its SQLite copy #415, Run hopin image v2026.11.11.1 #428 and P7d: k8up backups into private rest-server repositories, the restore test and backup:fetch #167: the standard stack; Traefik's errors middleware (not re-verified); store retention plus the /var/log/pods age rule.
     • k8up backs up keycloak and litellm into private repositories, with a monthly restore test #184, songmaker's web reaches the platform's AI front with a key of its own #339, #901 and #710: content stays; re-proven in S6.
     • #755 and An organisation ruleset blocks the merge of a pull request that brings a new high or critical CodeQL alert #310: a k3s agent join; GPU sharing (unverified).
     • The bootstrap job's apt calls retry a failed download #257, A bootstrap part raises the PC's inotify limits #207 and #835: shrink, made moot, deleted.
   • Stays:
     • People and the operator token: CI's reads of GitHub's meta API carry the job's token #256, #680.
     • Terms and legal: #1029, The cells share one until_true and one Prometheus reader, and the expire cell's wait fails loud #268, SCRATCH, do not merge: #103 C9 trustedIPs removed, whoami echo #110, #957, #1021, #1025, #1027 (which must also cover the new access log).
     • Realm content: #903, #748.
     • LiteLLM keys: The PreBackupPod stages songmaker's audio beside the dump, so one k8up snapshot holds both #299.
     • Restore-test content: #709.
     • presentator's content: Bump cloudflare/cloudflare from 5.28.0 to 5.28.1 in /src/opentofu/cloudflare #446, #1002.
     • Other independent items: #665, litellm_keys.sh's port-forward log exists before its first read #345, Seven steps of the cluster job run in parts with room #397, G4: A pull request that brings a new severe code-scanning alert cannot merge #308, The cluster job's two in-cluster sources take addresses read from the Docker network #316, The cluster job proves Flux decrypts a SOPS-age Secret through platform-secrets in k3d #24, #931.

   6. "Welchen Befehl führe ich aus?"

   Ship an app version on the target:

   git tag -a v2026.11.06.1 -m v2026.11.06.1 && git push origin v2026.11.06.1
   gh pr merge --squash --auto <PR-number> -R overnightworks/hal
   

   With auto-merge on green ruled under #1016, the first line alone ships. A config change is gh pr create --fill && gh pr merge --squash --auto.

   A fresh PC: git clone https://github.com/overnightworks/hal ~/git/hal && ~/git/hal/bootstrap.sh.

   Unverified:

   • Flux and observability memory;
   • whether 6443 and 10250 bind beyond loopback, and the kubeconfig's default mode;
   • Docker and k3s iptables on the same host;
   • CUDA builds on standard runners;
   • that no LiteLLM Terraform provider exists;
   • whether config-cli has a diff mode;
   • the Keycloak downgrade rule;
   • whether the tunnel passes Cf-* headers;
   • age-based kubelet log rotation;
   • the machine-user seat cost;
   • the agent-service image licensing;
   • k3s's current CNCF level.
3. FlexOr2 commented on Nov 6, 2026

   

   FlexOr2

   on Nov 6, 2026

   MemberAuthor

   More actions

   Evidence: architecture analysis by Codex gpt-6-astra at max effort, 06.11.2026. It was run read-only against hal at bac221a1, from the same brief, and is posted unchanged. It recommends Compose plus maintained tools. The operator chose k3s plus Flux. Its slices 0 to 3 (images, bootstrap, secrets, and logs and checks) appear in this epic as P1, P2, P4 and P5.

   ---

   Choose B: keep Compose, replace hal’s general deployment machinery with maintained tools, and keep hal only for fleet-specific policy.

   Hal has not rebuilt k3s. It has built a substantial deployment manager around Compose. Much of that manager can disappear without introducing Kubernetes. K3s becomes the stronger choice when you need continuous reconciliation, placement across machines, or rolling deployments—not simply because hal has become large.

   The PC can accommodate k3s. The harder cost is migrating storage, network isolation, emergency controls and operational knowledge. Installing k3s does not resolve those boundaries automatically.

   Evidence is from bac221a130d02c055c2fff4ba08580d4d3c0c552. No files changed, tests ran, containers started, or live systems were inspected. Shell access to GitHub failed; issue bodies and closed rulings remain unverified. The separate web tool could retrieve public documentation, so linked external claims below were re-read on 6 November 2026. Other repositories were read only through local origin/main, which may be stale. Proposed commands and replacement configurations have not been executed.

   1. Inventory. The Python inventory is 118 files, 28,617 physical lines, 23,369 nonblank lines. The table assigns every src/hal/**/*.py file exactly once. Counts include comments and docstrings.

   “Mixed” means a standard tool owns the mechanism, while a specific fleet requirement still needs custom policy. Replacement recommendations are follow-up architecture work requiring plan review, not permission to remove existing protections.

   Job and present implementation	Physical / nonblank lines	Professional owner and named tool	Does hal rebuild it today?
   Fleet schema, app restrictions and release grammar: stack/catalog.py, app_contract.py, releases.py; catalog.apps, app_contract.check	1,030 / 837	Mixed: hal owns fleet relationships and restrictions; Compose owns Compose syntax and semantics	Partly. Checking fleet requirements is legitimate. Maintaining a reduced interpretation of Compose creates another compatibility surface.
   Deployment: apply, apps, app_compose, checkout, deploy_checkout, compose, core, core_phase, unit_plans, service_recreate, built_image, environment, plan, pre_apply, setup, locks and waiting	4,878 / 4,114	Maintained tools on Compose: Compose plus mise’s Compose support; Kubernetes standard: Flux and Kubernetes controllers	Substantial overlap: image/config comparison, dependency ordering, recreation, fetching releases and health polling. Backup-before-change and stop priority remain policy. Evidence: core.dependency_order, Compose.config_hashes, AppsPlan.run, apply._plan.
   Emergency controls: emergency, barrier, switches, switch_events, tunnel; stop_unit, Switches, Barrier	1,685 / 1,398	Mixed: Compose/Traefik/systemd perform actions; hal owns durable emergency intent and precedence over deployment	Not wholly replaceable. “Stop stays stopped, including during deployment and after reboot” exceeds a plain container stop. Kubernetes introduces another reconciler that must respect this intent.
   Deployment presentation: stack/cli.py, plan_rows.py, result_rows.py, terminal.py	1,291 / 1,050	Native tool output and task entrypoints; custom presentation only for remaining domain commands	Yes, considerable wrapping. No standard supplies the exact ruled wording, but that wording is an operator-interface choice, not a deployment capability.
   Realm application and observation: stack/realm/; plan_realm, Observer, observe	1,462 / 1,206	Maintained tool: keycloak-config-cli; hal owns backup gating, protected people and any required change report	Mixed. Config-cli already performs the writes. Hal adds its own input/checksum tracking, observation and reporting. The Keycloak Kubernetes operator’s realm import is not a replacement for updating existing realms.
   External infrastructure: stack/external/; apply_plan, check_hosts, CloudflareRead, tofu	2,407 / 1,990	Maintained tool: OpenTofu and its Cloudflare provider; hal owns exposure-policy assertions	Mixed. Plan execution and formatting wrap native functionality. Detecting undeclared resources and testing actual exposure is different from reconciling resources tracked in state.
   Legal composition and shared assets: legal_pages, legal_section, legal_step, appkit_step; plan_legal, plan_appkit	1,355 / 1,128	Hal domain: assembling the fleet’s approved sections and release association; ordinary build tooling packages assets	The fleet contract is custom. Runtime copying, version directories and deployment planning should largely become build-time work.
   LiteLLM provisioning and retention: litellm_keys, litellm_admin, api_limits, root litellm_expiry, expire_commands	1,237 / 994	Maintained tool: LiteLLM enforces budgets; hal domain: app/person mapping, Keycloak admin-group policy and retention	Hal does not implement the budget engine. KeysPlan configures native APIs. litellm_expiry.statement directly depends on LiteLLM’s database schema; that maintenance obligation survives k3s.
   Backup and restore: backup/, backup_commands.py; runner.backup, restore_test, live.restore_live, sqlite_copy.copy/load	3,392 / 2,795	Maintained tools: restic, resticprofile, PostgreSQL/SQLite backup interfaces; hal/app domain: consistency and restore acceptance	Mixed. Scheduling, retention arguments and status reporting overlap maintained tools. Coordinating a consistent app snapshot, selecting the proven snapshot and verifying a restored application do not vanish.
   Workstation installation: workstation/setup, system_step, callback, path_link, user_units, jobs, tunnel_config, setup_commands	1,900 / 1,542	Maintained tools: mise for tools/tasks; Ansible for host configuration; systemd for scheduling	Much is replaceable: downloads, installation detection, unit rendering and a second presentation layer around Ansible. Keep the declared host policy.
   Secret inventory and onboarding: secrets_view, secrets_setup, secrets_create, manifest fields, ports and command/output modules	2,094 / 1,633	Mixed: native provider login/secret tools; hal owns inventory, reader relationships and the guided workflow	No reviewed tool exactly replaces this workflow. It is also not a secret scanner: secrets_view primarily checks presence and metadata; secrets_setup provisions missing entries.
   Daily check and alarm: workstation checks, alarm, mail, report, layout and root command modules	1,983 / 1,535	Maintained tools: Prometheus/Alertmanager and systemd; hal supplies fleet-specific findings and any required daily report	Generic service/disk/backup alarms, persistence and notification deduplication overlap monitoring tools. Secret inventories, hook installation and the exact daily report are separate requirements.
   Exposure inspection: exposure/, exposure_rows.py; Problem, check	817 / 640	Mixed: native network/container controls plus a small fleet exposure audit	The particular “only approved exposure, valid routing names” comparison is custom. Moving to Kubernetes changes its inputs; it does not prove the policy automatically.
   Checkout update, relocation and migrations: rename_commands, rename_rows, migrate_commands, stack/project_move, stack/update	1,618 / 1,308	Git/native provisioning; finite migration programs where data actually needs moving	Migration code should expire when its callers disappear. rename_commands.make_plan handles real local relocation details, not a Kubernetes responsibility. Source-checkout deployment creates part of this work.
   Shared command interface and output: output/, root CLI/config/grammar/version and remaining door modules	1,468 / 1,199	Standard CLI libraries and native output; small shared interface for retained domain commands	Much shrinks with the commands it serves. Moving the same interface into shell scripts would not remove its complexity.
   Total	28,617 / 23,369

   The principal replacement capabilities are documented directly: Compose already recreates changed services and offers health waiting; mise now documents Compose project state, model-hash comparison and one-shot services. Ansible also has a maintained Compose module, but these should not become two owners of the same deployment. Compose, mise Compose projects, Ansible Compose module.

   Other areas must not disappear from the accounting:

   Outside the Python subtotal	Physical lines	Ownership assessment
   compose.yaml, fleet.yaml, both Traefik configurations	992	Configuration for maintained tools. Preserve the service contracts; replace runtime-specific definitions only when their runtime changes.
   external/*.tf	457	Already the appropriate OpenTofu owner. Keep.
   tunnel/, mcp/, dynamic/	97	Native service/proxy configuration, including emergency-stop integration. Keep the policy; adapt its representation if necessary.
   Workstation Ansible configuration/playbook; secret inventory	551; 672	Declarative host policy and secret metadata. These are not redundant merely because their Python launcher is.
   keycloak/	2,410	Realm/theme configuration plus bootstrap code. GoLive alone is 937 lines. Config-cli and Keycloak remain the mechanism; bootstrap policy needs a separate reduction pass.
   step-ca/	560	CA configuration plus the 540-line StepCaInit wrapper. Keep CA trust/material ownership; simplify orchestration around the maintained CA.
   agent-service/	5,042	Includes runtime configuration, image dependencies, setup/signing helpers and substantial proof code. The agent runtime itself belongs to agent-runtime; this tree is not another container scheduler.
   legal/	208	Approved text, frames, styling and nginx configuration. Keep or package into an immutable artifact.
   Person export/erasure	0 implemented command lines	The orchestration is still a requirement in docs/people.md, not an existing capability k3s could replace.
   tests/, probe/, scripts/system-proof/	79,898; 29,022; 2,241	Verification assets, outside production savings. Retire tests with removed implementations while preserving behavioural proof.

   The code already has structural separation: pyproject.toml declares import-layer and independence contracts. That is useful, but module separation does not decide whether hal should own the underlying job.

   2. The boundary. Hal should own fleet meaning and exceptional policy, then hand ordinary configuration and release artifacts to maintained tools. It should not maintain a parallel container-reconciliation engine, dependency scheduler or general deployment-state database.

   The handoff should be a reviewed release bundle: OCI image digests, native Compose definitions, configuration and legal assets, selected by the fleet declaration. Credentials travel separately through the existing secret boundary. On a future Kubernetes target, the deployment handoff becomes native manifests consumed by Flux; the apps’ images and domain contracts remain.

   flowchart TD
       A["App repositories<br/>images and Compose release bundles"]
       F["Fleet repository<br/>fleet.yaml, settings, realm and policy"]
       H["Small hal<br/>validate fleet relationships and build legal assets"]
       B["Reviewed deployment artifacts<br/>native configuration and fixed releases"]
       O["Operator<br/>mise task"]
       T["Maintained execution tools<br/>Compose, OpenTofu, Ansible, restic"]
       R["Running services and durable data"]
       P["Explicit domain operations<br/>people, budgets, restore proof, emergency priority"]
   
       A --> B
       F --> H
       H --> B
       O --> T
       B --> T
       T --> R
       P --> T
       P --> R
   

   Loading

   Proposed ownership boundary for the operator. Derived from the existing catalog, Compose, realm, backup and domain modules; it is not a picture of today’s implementation. Mermaid rendering was not verified.

   Two details prevent this from becoming the same system under another name:

   • One writer per resource. Compose manages containers; OpenTofu manages declared Cloudflare resources; config-cli manages declared realm configuration; restic manages backup storage. Hal checks and supplies policy around those owners.
   • Keep distinct facts distinct. A receipt saying “release X last deployed successfully” may remain for existing consumers. It must not become another claim about what is running now. That distinction is already documented in docs/architecture.md, “The deployed record,” and implemented by apps.DeploymentRecord.

   There is concrete coupling to remove. apply._apply_changes writes legal/appkit files before deploying apps. Consequently, a failed app update can leave its new legal text served with the old app. Packaging and promoting release artifacts gives that association an explicit owner.

   3. The real alternatives. Compose is a supported production deployment option; Kubernetes is not a prerequisite for professional operation. Docker’s production guidance.

   Alternative	Fit and upkeep	Machine cost	What becomes dead
   A. Compose plus hal as it is	Lowest transition effort. Continues maintaining a bespoke deployment manager, its state transitions and its presentation. Current readiness/partial-run/stateful-update work remains hal’s responsibility.	Hal’s CLI and timer jobs are intermittent; no continuous hal controller is declared in workstation/jobs.py. Actual CPU/RSS unmeasured. Current app builds also run on this PC through AppCompose.build.	Nothing substantial. Growth continues in the current owners.
   B. Compose plus maintained tools; small hal	Best fit for the current machine and existing app constraints. mise owns tools/tasks and native Compose reconciliation; Ansible keeps host policy; OpenTofu, config-cli and restic keep their existing jobs. Integration policy remains explicit.	No new cluster control plane. Tool runs are intermittent. Monitoring adds persistent processes under every serious alternative. Actual memory/CPU unmeasured. CI-built images remove local build load.	Most deployment planning/recreation/polling and presentation; custom tool installation; much scheduling/notification plumbing; obsolete migration paths. Domain and safety policies remain.
   C. k3s now plus Flux	Clean standard owner for continuous deployment reconciliation, workload state and later placement. Adds cluster/API/CNI/storage/RBAC/certificate operations. Retain Keycloak, LiteLLM, CA, external infrastructure and application-aware restore.	Official server minimum: 2 cores, 2 GB RAM, excluding workloads. Budget additional capacity for Flux and monitoring; 4–8 GiB is a planning allowance, not a measurement. Hardware is not the decisive objection here.	Compose deployment engine, Docker-label discovery, Docker-specific backup discovery and much routing configuration. Also creates migration work for volumes, secrets, networks and emergency controls.
   D. No hal executable: configured standards plus repository	Viable if custom domain work lives in a few clearly owned commands. A strict “configuration only, no custom code” target does not cover the fleet contracts examined here. Moving hal into miscellaneous scripts would preserve the maintenance cost.	Similar to B on Compose, or C on Kubernetes. Removing a CLI name does not save meaningful runtime resources.	The hal command interface can disappear. The domain behaviours cannot be assumed to disappear with it.
   E. Coolify as the integrated deployment product	Serious option when a deployment UI is the priority. Offers Compose applications, configuration and deployment controls. Its own configuration/state becomes an additional operational owner; fleet-specific privacy, identity and stop rules still need integration.	Official minimum 2 cores, 2 GB RAM, 10 GB disk, excluding app needs. Actual use here unmeasured. Adds its own application/database upkeep.	Much of hal’s generic deployment UI and execution machinery. Not the cross-app policies or proven recovery procedures.

   Sources for C: k3s requirements, Flux deployment model, k3s networking. Sources for E: Coolify installation requirements, Compose support.

   For B, mise’s newer documentation also covers machine bootstrap. I would initially retain the existing Ansible host role because its specific account, tunnel and logging policies already have repository proofs. Mise replaces its Python installation/presentation wrapper. A later replacement of that role must demonstrate equivalent behaviour; mise and Ansible must never manage the same host resource concurrently. mise bootstrap; local evidence: workstation/system/playbook.yml, scripts/system-proof/.

   Day-to-day operation. Commands marked target are interfaces to implement and prove, not commands available today. Release/configuration changes first pass through the deployment repository’s review.

   Alternative	Ship a version	Change configuration	Add an app	Recover from a broken update	Emergency stop
   A	Publish an eligible release tag; hal apply. Update hal’s own checkout first with hal update when required.	Merge configuration change; hal update; hal apply. Local [api] settings follow the existing separate path.	Add Compose contract, fleet entry and required identity/secrets/backup/legal declarations; hal update; hal apply.	Failed app release currently requires a newer fixed tag. Database recovery uses hal stop keycloak, hal restore keycloak --live --snapshot ID, then the matching image/configuration and hal start keycloak.	hal stop hopin
   B	Merge a promotion selecting an immutable release; update deployment checkout; target: mise run deploy -- hopin.	Merge deployment settings; same deploy task. External exposure changes use target: mise run publish.	Add the release bundle and fleet relationships; deploy; publish only after readiness succeeds.	Revert the promotion and redeploy for compatible changes. Stateful recovery retains the explicit stop/restore/start procedure.	Retain hal stop hopin, with its policy guard shared by the new deployment path.
   C	Merge the image/configuration promotion. Flux applies it; optional immediate command: flux reconcile kustomization hopin --with-source.	Merge ConfigMap/Secret-reference/configuration change; Flux reconciles.	Merge namespace, workload, service, policy, storage and identity configuration; Flux reconciles.	Revert the Git promotion. For incompatible data changes, suspend reconciliation and restore matching data before resuming.	Target: mise run stop -- hopin, closing ingress, suspending the relevant reconciler and stopping workloads, with observed confirmation.
   D, Compose implementation	Same Git promotion; target: mise run deploy -- hopin.	Same task after reviewed settings change.	Same native configuration and app relationships as B.	Target: mise run restore -- keycloak ID, invoking the retained domain recovery command.	Target: mise run stop -- hopin, invoking the retained emergency policy.
   E	Open app → Deploy, after selecting the approved source/image.	Environment Variables → Save → Deploy.	Create a Compose application from its repository and configure its required integrations.	For supported image deployments: Rollback → Available images → Roll back to this image. For Compose, restore the prior image/configuration definition and redeploy. Restore data separately.	App Stop is the product action; equivalence to hal’s persistent emergency contract remains unverified.

   A is grounded in docs/stack.md, docs/backup.md, apps.AppsPlan and emergency.stop_unit. Flux’s immediate reconciliation command is documented here.

   For C, suspending Flux alone does not stop an app. Scaling it down alone does not establish a durable stop against another controller. Parent configuration can also overwrite a suspension declared differently in Git. The emergency procedure must establish and prove the complete policy. Flux suspension semantics.

   Coolify explicitly says rollback changes the application image, not databases or persistent storage. Its application-level rolling updates exclude Compose applications. Rollback, rolling updates.

   Stateful services and isolation are the migration cost:

   Concern	What remains necessary under every option
   Keycloak and Postgres	Preserve the database, issuer, signing keys and client identity. Keep a tested pre-change backup and the matching application/database versions for recovery. backup/apps.toml, pre_apply.Backup and live.restore_live encode this today. A realm import is not a full user/database recovery. The official Kubernetes realm-import resource only creates realms; it does not update existing ones. Keycloak operator.
   App databases and files	Migrate data explicitly. backup/apps.toml describes Postgres plus audio for songmaker and SQLite for presentator. A Git revert cannot reverse data changes. PostgreSQL dumps are internally consistent, but that does not synchronize separately copied files. PostgreSQL backup documentation.
   Backup consistency	Current songmaker backup reads audio live. backup/verify.py checks counts and referenced-file presence; it does not prove that database and files represent one transactional moment. #678 remains necessary.
   k3s storage	Local-path volumes remain tied to their node. Adding another node does not make those bytes available there. The cluster datastore and recovery token also need backup; that is separate from app data. k3s storage, cluster backup.
   Presentator’s worker isolation	presentator@origin/main:compose.yaml, service build-worker, declares network_mode: none and a shared Unix socket. Putting web and worker into one Kubernetes Pod would share their network namespace. A faithful port therefore requires a deliberate containment design. Kubernetes Pods.
   Service network restrictions	compose.yaml and speech-runtime@origin/main:compose.yaml use separate internal networks. Kubernetes requires corresponding policies and verified enforcement; namespace separation alone is insufficient. NetworkPolicy.
   LiteLLM	Preserve keys, spend and budget state. docs/backup.md, “Restore LiteLLM,” explicitly says restoring loses spend recorded after the snapshot. Kubernetes does not change that consequence.

   If the operator installed k3s today, the official command would install a cluster:

   curl -sfL https://get.k3s.io | sh -

   It would migrate none of these services or data. It is not the recommended deployment command, and no k3s migration command exists in hal.cli.COMMANDS. Official k3s quick start.

   For D, the supposedly “hal-only” jobs need this accounting:

   Job	Home without hal
   Legal pages	A deterministic build command in the deployment repository using the existing section contract; nginx serves the artifact. No reviewed platform assembles these exact sections by deployed release.
   Person export/erasure	A dedicated fleet administration command calling app APIs and Keycloak, preserving the tombstone/replay order in docs/people.md. This is substantive custom work, still unbuilt.
   Secret scanning	Already owned by quality and its scanner engines; native Git hooks invoke it. Evidence: workstation/hooks/commit-hook, quality@origin/main:README.md.
   Secret inventory/onboarding	Native provider logins plus a small inventory/onboarding command. Encryption or a secret store alone does not check all current reader, presence and revocation requirements.
   Agent account	Ansible/Linux permissions; later Kubernetes RBAC for cluster access. Keep the host boundary too. Current role creates a dormant account; that alone does not move active agents into it.
   Daily check	Monitoring handles generic health and alerts. A small audit command supplies secret/hook/exposure findings and any specifically required daily digest. Alertmanager already owns grouping, deduplication and notification routing. Alertmanager.
   LiteLLM budgets	LiteLLM’s native budget/key APIs plus a small provisioning command for the fleet’s identity mapping. Retention remains separately owned. LiteLLM budgets.

   4. Recommendation. Adopt B as a deliberate reduction, rather than continuing A while calling it temporary.

   The boundary is the reason for the choice:

   • The apps already have Compose deployment definitions and specific isolation requirements.
   • Native tools can remove much of the custom execution layer now.
   • The genuinely custom fleet requirements remain under either runtime.
   • Kubernetes would add a second substantial migration before those requirements have been separated.

   The six broad buckets covering deployment, deployment presentation, setup, daily checks, migrations/update and shared CLI contain 13,138 physical lines—46% of src/hal. This is an opportunity inventory, not a promised deletion count: some policy survives, and replacement configuration also costs maintenance. There is further simplification potential in backup and external wrappers. Do not preserve code to protect an investment; do not rewrite it line-for-line into Ansible or mise tasks either.

   I would change the recommendation when any of these becomes concrete:

   1. A service needs deployment or placement on another machine.
   2. A service must update without downtime and can support that behaviour.
   3. Continuous reconciliation becomes a requirement.
   4. The reduced Compose implementation still needs a general controller to satisfy the accepted workflow.

   The existing “about ten apps” trigger remains a useful review point, not a technical capacity limit.

   The ordered plan is:

   Slice	Add/change	Delete when its replacement is proven	Proof without the live system
   0. Freeze the target contract	Record native-tool ownership, release promotion, emergency precedence, backup gates and app/data ownership. Identify which existing UI/notification rules change. Obtain independent plan review.	No safety mechanism yet. Stop extending the old generic deployment design.	Review every ruled behaviour against an owner and a scenario. Resolve issue bodies unavailable in this analysis.
   1. One fresh-PC entrypoint	Ship scripts/bootstrap: verified pinned mise installation, locked tools, existing Ansible host role, credentials onboarding and resumable setup. Keep host resources solely owned by Ansible.	Python download/install detection in workstation/setup.py; unnecessary Ansible output parsing/callback; custom unit writing once standard configuration owns it.	Disposable Ubuntu VM: first install, second unchanged run, interrupted install/retry, wrong credentials, dormant agent account restrictions, reboot and disabled tunnel. Reuse the scenarios in scripts/system-proof/.
   2. Build releases in CI	Each app publishes an immutable OCI image and production Compose bundle. Bundle required files and release metadata; select the release explicitly in the fleet declaration. Keep runtime settings outside app rebuilds.	On-PC app builds and newest-tag discovery: relevant built_image, deploy_checkout, checkout, releases paths after their final caller moves.	In CI, deploy the published artifact without a source checkout or compiler. Verify image revision, health, settings override and same-artifact redeployment. GitHub supports image publication from CI. Documentation.
   3. Replace the deployment engine, one app at a time	Generate only native deployment inputs from fleet.yaml. Let mise/Compose perform model reconciliation. Integrate the existing emergency policy as a narrow guard. Start with a disposable app, then hopin.	core hash bookkeeping, recreation planning, polling and corresponding plan/result presentation for migrated callers. Remove the old apply dispatcher after the last caller moves.	Throwaway stack: unchanged rerun, config/image update, failed health, interrupted deployment, and stop during preparation/start/wait. Prove volumes survive disable/re-enable and stop wins.
   4. Separate backup mechanics from app recovery	Introduce resticprofile for standard profile/schedule/retention/reporting work. Keep app-owned snapshot creation, verification and explicit restore. Preserve exact successful-snapshot selection and pre-change backup gates.	Covered argument/scheduling/status plumbing in backup/restic.py, runner.py and timer generation. Delete only after equivalence, not merely because a backup exits zero.	Synthetic Postgres/SQLite/file data: concurrent writes, incomplete backup, corrupted copy, missing key, wrong snapshot, failed restore and clean-machine recovery. Test the existing retention policy behaviour. Resticprofile provides hooks and scheduling, not the app consistency proof. Documentation.
   5. Extract domain operations from deployment	Run realm configuration through config-cli; retain required backup/reporting policy. Make LiteLLM provisioning an explicit job. Build legal pages and Keycloak theme assets as immutable artifacts. Keep people operations separately specified.	Redundant realm input-state machinery where native config-cli proves equivalent behaviour; runtime asset copying and the legal/appkit branches of apply.	Disposable Keycloak/LiteLLM with synthetic identities: preserve undeclared people/memberships, repeat an interrupted job, retain spend on a limit change, match legal assets to the promoted release.
   6. Standard monitoring	Implement #4 with native metrics/alerts; feed fleet audit findings into it. Retain a daily report only for its distinct question.	Generic service alarm state and delivery plumbing once the chosen semantics match the accepted contract.	Fake failures and recovery through a test mail sink: deduplication, failed delivery, stopped-by-operator services, missing backup and a broken check itself. Exact sample-count behaviour must be retained or explicitly replaced by a new ruling.
   7. Rehearse, switch ownership, remove predecessors	Rebuild a second disposable machine from repository plus synthetic secret/backup inputs. Update docs and callers. At a later authorized cutover, transfer each runtime resource to its sole new owner.	Old deployment entrypoint/state, superseded timers including songmaker’s independent deploy timer, dead migration paths and their implementation-specific tests. Preserve any public deployment receipt until readers migrate.	Complete fresh-machine setup and restore; then targeted failure scenarios. CI runs required checks and the pinned hygiene gate. Each deletion has evidence that no caller still uses the predecessor.

   The host remains usable after an interrupted provisioning run; it must report partial completion and resume. A collection of tools cannot supply an atomic transaction across Cloudflare, Keycloak, databases and containers. The plan must make recovery explicit rather than imply that one command creates atomicity.

   Existing safety distinctions are worth preserving:

   • Config-cli performs the realm writes; the surrounding observer is not itself another identity provider. config-cli.
   • LiteLLM performs budget enforcement; hal’s provisioner is integration policy.
   • Restic stores backups; app recovery verification has a different owner.
   • Emergency intent is distinct from ordinary desired deployment state.

   5. Open items. All 70 supplied titles are accounted for below. These are proposed dispositions from titles and code, not closure decisions; the bodies were unreachable.

   Items	Effect of the recommendation
   #50, #1016, #660, #987, #233	Re-cut around the target. One deployment path, fixed release selection, separately supplied settings and simple commands remain valid. Implement them through native artifacts/tasks. #1016 must retire songmaker’s separate timer.
   #361, #1051, #1063	Supersede the old implementation proposals. Native task/deployment workflow replaces further work on hal’s generic apply dialogue and release-command wrapper. Preserve required approval/outcome semantics in the new contract.
   #1081, #409, #595, #395, #394, #610	Carry the behaviour into acceptance tests. Interruption, readiness, observed stops and backup-before-stateful-change remain. Status should use native observations plus emergency intent. #610 needs a stale-body check: current service_recreate/pre_apply already implement substantial backup gating.
   #291, #374, #147	Core setup work, changed owner. Fresh-PC setup, sudo compatibility and tool updates belong to bootstrap/mise/Ansible.
   #407, #676	Still valid onboarding requirements. Implement beside the retained credentials workflow, using native provider login commands where available.
   #615, #1056	Move hook responsibility to quality/bootstrap. Avoid making hal another owner of repository quality policy.
   #678, #709	Keep. Snapshot consistency and restored-file verification survive every runtime choice.
   #110, #957, #1021, #1025, #1027, #268, #1029	Keep the legal/consent requirements. Rendering moves to the build; identity actions remain explicit jobs. #1025 needs a body check because a speech privacy section already exists.
   #256, #680, #345	Keep domain need; narrow implementation. People operations and operator authentication need a home. Evaluate the proposed maintained API CLI without turning hal into a generated universal app CLI.
   #184, #299, #339, #710, #748, #901, #903	Keep runtime/security/identity contracts. Image publication changes; the contracts and their proofs remain.
   #308	Check for stale scope. Shared speech is already represented by fleet.yaml and the speech-runtime deployment definition. The body may contain remaining work.
   #310	Leave as an idea. Do not build a GPU allocation system as part of this reduction. Kubernetes adoption alone would not prove the requested GPU-memory policy.
   #755	Keep as a future architecture trigger. A real second-machine requirement should reopen the runtime choice.
   #4, #415, #1071, #167	Keep outcomes; consolidate owners. Standard monitoring owns service alerts; retained commands emit suitable events; host/app configuration owns log retention.
   #428	Independent optional feature. A maintenance page belongs to ingress configuration and the emergency-state presentation.
   #296, #1079	Shrink to surviving domain commands. Do not finish presentation work for a dispatcher being removed. Changes to ruled output require a new ruling.
   #1082, #1037, #908, #931, #639	Dependencies and CI remain valid. #1037 needs a stale-body check: pyproject.toml already pins appkit 0.14.0. Immutable theme packaging addresses #908’s underlying coupling. Dependency-update automation should own #639.
   #207, #24, #790, #951, #819, #835	Re-cut migration scope. Do not migrate deployment bookkeeping that will be deleted. Preserve required durable-data/name transitions; remove finite migration tools after their windows. Avoid combining database-major upgrades with this reduction.
   #280, #396, #397, #665	Keep. App contracts, cross-owner consistency, external infrastructure as code and API contracts remain useful. Native configuration should be validated by its own tools first.
   #316, #257, #991	Keep. Carry this analysis into the existing audit owner; update documentation with each landing; retain the secret-command process-lifetime regression work.
   #1002, #446	Unverified remaining scope. #1002’s title is insufficient. Presentator is already enabled in fleet.yaml, so #446 requires its body before judging what remains.

   6. “Welchen Befehl führe ich aus?” For the recommended target, CI publishes the release, and a reviewed promotion selects it in the deployment repository. After merging that promotion:

   git -C ~/srv/hal pull --ff-only
   mise -C ~/srv/hal run deploy -- hopin

   deploy is a proposed repository task. It prepares the selected immutable bundle, applies the narrow safety policy, and calls the maintained deployment tool. It does not implement another reconciliation engine.

   On a fresh supported PC, from the repository checkout:

   ./scripts/bootstrap

   That is also proposed. It installs the pinned tools, configures the host, guides necessary account/secret setup and resumes after interruption.

   Today’s existing command remains hal apply. The target is complete only when the new commands are proven and the old deployment machinery is removed.
4. FlexOr2

   added a sub-issue on Nov 6, 2026

   • P0: the skeleton and the fleet's quality gate, from the first pull request #2
5. FlexOr2 commented on Nov 6, 2026

   

   FlexOr2

   on Nov 6, 2026

   MemberAuthor

   More actions

   Evidence: the fleet heads' replies to the platform switch, 06.11.2026, as relayed to hal's head. Each head keeps its own items. This comment gathers what the platform has to plan for.

   Local image sizes on the PC, 06.11.2026 (unpacked; docker image inspect):

   Image	Size
   hopin	0.2 GB
   keycloak 26.8.1	0.5 GB
   songmaker web	1.2 GB
   songmaker music-worker	0.9 GB
   songmaker scoring-worker	5.9 GB
   songmaker acestep-worker-0	9.3 GB
   speech-runtime (live)	1.2 GB
   speech-runtime since G3	9.8 GB
   presentator	2.2 GB

   Replies:

   • aco: nothing changes; aco-board is a user unit he starts by hand (systemd-run, loopback), not on compose/hal.
   • marketplace: nothing pins the platform; audit skill platform lens gets a one-line addition after the switch; send standards through docs/standards.md.
   • appkit: nothing to freeze; no item superfluous. Correction: appkit does NOT build in CI — scripts/publish_release builds wheel/npm/tokens on his machine and pushes the tag over SSH, no CI/verdict check. pnpm 10 drops integrity on warm re-resolve (pnpm update rewrites lockfile dropping integrity for URL/tarball deps -> later install fails with ERR_PNPM_MISSING_TARBALL_INTEGRITY pnpm/pnpm#14351). CI-built shape exists in speech-runtime and agent-runtime (dispatched workflow gates before tag, creates tag via API). appkit#98 holds that idea; agent-runtime proposes quality as owner of one shared release gate. Release immutability is on org-wide since today (his choice). Fair standard: CI builds and gates, assets on immutable GitHub Release, consumers pin URL + hash.
   • hopin: no machinery. Items changing owner: #493/#498 legal part (needs a legal-composition owner on the platform), #495, hopin runs on the platform from its attested private image, routed by Traefik #46 gates hal#167/The cells share one until_true and one Prometheus reader, and the expire cell's wait fails loud #268/S1b-2: the PreBackupPod stages songmaker's files beside the dump #280 (need a home on the platform board), The GitHub settings root rolls the merge queue out to the gated repositories #459 l5 (readiness probe vs manifest), The GitHub settings root rolls the merge queue out to the gated repositories #459 l10 (GHCR push in per-app item), presentator's layer runs at its release 2026.11.11.2 and its web is routed through Traefik #475 (speech on new platform, model volume). Per-app facts: GET /health, HOPIN_* env + *_FILE secrets, no persistent store, needs litellm-front:4000, Keycloak back-channel, Cloudflare TURN.
   • presentator: freeze ok. Superfluous: #710 (closes when #782 lands). Changes: The P9c cell's write to the Secret hopin survives a platform-secrets reconcile #243 build-worker sandbox is Docker-socket based → needs new isolation plan on k3s (B4 live proof keeps value now); The job images checks each app ref's tag, attestation and signer commit with the locked gh #67 moot for old stack; Gatus shows every service's state, opened only at the PC #197 Done-when names hal apply tunnel → Flux ingress; #508 closes on today's platform after S9l + #782, k3s move is a new item; compose.yaml/test_compose/runbooks become predecessors (exit item at switch). Has health endpoint, env+file config, no weights in image. KI-Dienst: wants follower item on presentator board naming both legal parts, blocked by hal's rename slice.
   • songmaker: freeze ok, deletions in flight continue (#1476 l8-11). Closing/rewriting needs his word in songmaker session (asking him there). Images: acestep 9.3 GB has NO weights (CUDA torch + ACE-Step deps; checkpoints bind-mounted, download_models.sh with HF_TOKEN) → volume, no image change; base images built locally by scripts/build_images.sh → CI/GHCR. scoring 5.9 GB HAS weights baked (Whisper large-v3 ~3 GB + audiobox ckpt, build secret hf_token; Dockerfile:25-38) → volume saves 3+ GB and HF token from build. Superfluous: #1789/#2277 (logging), #2003, #1711. Changes: #2284 exit becomes Flux; #2206 secrets source becomes platform secrets, own monitoring exit becomes platform monitoring; #2271/#2270 mechanism becomes k8s volumes/NetworkPolicy/Redis ACL (paused); #1476 host steps continue. Wants image workflow template + weight volume layout in per-app items.
   • agent-runtime: no machinery, no item superfluous. #672 rows mirror hal's HAL-2 (key as secret file fits). After #749 ships one wheel on the GitHub release; public, no token needed. AGENTS.md Releasing sentence about hal's image goes stale; #749 drops it after HAL-R.
   • speech-runtime: the weights are baked into the image, and the service runs with no egress. Since G3 (8b15a1c) the image is 9.8 GB with Qwen and CUDA. A volume would need a fill step with egress and a revision check at start. The board drops merge_method, since aco squashes by default #155 G4 extends compose, and its head asks the operator to stop it. G4c is superfluous if G4 stops. ci.yml keeps only job structure; the cluster and bootstrap cells live as scripts under tests/ #156 G4b stays. SCRATCH, do not merge: #103 C9 trustedIPs removed, whoami echo #110's image and compose from a release tag become CI, GHCR and this repository's manifests. The PC creates its age key once and reaches platform-secrets with its own read-only deploy key #63 (scale by instances) fits k3s. The old image proof refuses to start while the hal_speech_* networks exist.
   • webauth: there is no machinery. Its library release files lose their last consumer with hal#1086 (webauth#595 row 13). Its hand-run end-to-end proof takes Keycloak from a pinned hal checkout (hal_sha.txt), so it needs the realm's new home; webauth opens that item when this repository's realm slice is cut. Libraries stay on git tags as webauth#543 ruled on 02.11.2026, because Dependabot fails on release-asset URLs.
   • quality: it owns the shared image workflow and the shared release gate (both workflows share one check, "this commit is main's head with green CI"). It is opening a fleet-standard item with the operator's cards.
6. FlexOr2

   marked this as blocking on Nov 6, 2026

   • Remove the compose-port rule after the platform cut-over quality#365
7. FlexOr2

   added a sub-issue on Nov 6, 2026

   • P1: every app release yields an attested private image in GHCR #4
8. FlexOr2 commented on Nov 6, 2026

   

   FlexOr2

   on Nov 6, 2026

   Last edited by FlexOr2

   MemberAuthor

   More actions

   Inputs for later slices, from the heads on 06.11.2026. Each one is named in its slice's body when that slice is cut.

   • P1 is P1: every app release yields an attested private image in GHCR #4. Its image workflow is One release path for the fleet: quality's release gate, image build and tag workflows quality#364, recorded as P1: every app release yields an attested private image in GHCR #4's blocker.
   • P1, P8 and P11, from the fleet audit (Distributor: fleet audit of CI and scripts (06.11.2026) marketplace#484, rows R10 and R11): the build-only image jobs of hopin, presentator and speech-runtime become callers of quality's image, and their image proofs stay as tests; songmaker's build_images.sh and check_pinned_images_match.py leave at P8 and P11.
   • P2: the bootstrap keeps the global git hook that scans git commit and merge commits for secrets, which hal setup installs today (hal AGENTS.md; docs/home.md, „Was der Hook nicht garantiert“), or names its replacement. hal setup installs a commit-msg hook that refuses a commit without a body hal#1056 reopens if the bootstrap ships without it.
   • P2 and P5: logs are deleted after 30 days (Server logs on this PC are deleted by age after 30 days, for every app hal#167): the host journal keeps at most 28 days, and the log path (Alloy into Loki) keeps the same maximum; each slice says what holds it. On 04.11 and 05.11 a kernel USB flood of 2.7 million lines filled journald's default 4 GB cap, so a size cap alone does not hold the time limit (details on hal#167).
   • Closed on hal on 06.11.2026 with a reopen trigger on their row: skat runs release 2026.11.09.1 after a backup, with its new legal texts #291 (P2, P3), P1: every app release yields an attested private image in GHCR #4 and #1071 (P5), #1029 and #680 (P7), #957 and #1021 (P9), #676 (P4).
   • P3: the k3s port rule (hostPort, NodePort, LoadBalancer) waits for real manifests. Remove the compose-port rule after the platform cut-over quality#365 removes the compose-port rule after the cutover (quality's head).
   • P0's follow-up: once main has its first SonarCloud analysis, [gate] legacy in quality/config.toml goes back to "fail", as aco, quality and hal have it. It is "report" only because an unanalysed main makes "fail" unjudgeable (quality docs/sonar.md). This lands as a fix lane right after P0.
   • P3: the checks job pins check-jsonschema but not its dependencies. When P3 extends checks, its tools get a lock or a pinned constraint (review of platform#3).
   • P5: One monitoring for every service: e-mail, Prometheus and Grafana in the platform hal#4, one monitoring for every service, folds into P5. hal's head closes it with a pointer once P5 has its number.
   • P6: speech-runtime's voices are a list set at start: Piper alone by default, Qwen on the GPU only when the operator adds it, so the GPU stays free for songmaker (GPU voices: the service loads the selected GPU voice and fails visibly when the card is taken speech#2, ruled 04.11.2026). On the platform that list is a value in this repository that Flux applies on merge. The operator's settings for an app reach it from one place hal reads, without a release tag hal#987 is closed as overtaken; it is history only.
   • P7: the platform's backup path keeps every existing restic snapshot restorable at least as long as hal's policy does: weekly forget --keep-weekly 4 --keep-monthly 3 --keep-tag pre-apply, pre-apply snapshots --keep-last 10, anything older than 90 days forgotten by ID (hal src/hal/backup/restic.py). presentator dates a data-upgrade removal by it: the last presentator backup from before 2026-11-06 stays restorable until the first forget past 2027-02-04. A shorter policy, or dropping the existing snapshots, goes to presentator's head first (hal's head).
   • P7, the realm (HAL-1: the realm holds one public client for agents, an MCP scope and an MCP group per app hal#903, HAL-1, moves here): the realm export P7 takes over keeps what hal#905 landed: the public agents client, the mcp-<app> scopes and the empty /apps/<app>/mcp groups. Ruled sentences (agent-runtime#672 and hal#903): one pre-registered public client with PKCE S256, no secret, DCR off, consent required, no refresh tokens; an audience mapper per app, https://<app>.<EDGE_DOMAIN>/mcp; the realm's 300-second access token lifespan with no per-client override; no role claim in the token; the groups stay empty until each app's privacy notice names the person's own agent. The operator confirmed MCP stays on 06.11.2026 in agent-runtime's session („ok mcp“). Left of HAL-1: /mcp and the well-known path on each app's host (P6 routing), and two questions from agent-runtime#672: how Codex behaves as an MCP client, and where Claude Code stores its MCP tokens on Linux. agent-runtime's head repoints #672's HAL-1 row once P7 has its number.
   • P8: songmaker's head records the songmaker app item as the blocker of songmaker#2284, #2206, #2270 and #2271 once it exists.
   • P5, log retention beside the journal (plan for P2 to P4, v3, D7): Loki keeps the same maximum as the host journal; the kubelet's pod log files under /var/log/pods rotate by size only (containerLogMaxSize, containerLogMaxFiles), so a quiet long-lived pod can keep lines past 30 days on the host. P5 says what deletes them.
   • P10, ownership at the swap (same plan, unknowns of P2: scripts/bootstrap_pc.sh sets up a machine from nothing #12 and P3: the cluster skeleton reconciles in k3d on every pull request #11): the live PC's daemon.json, ~/.local/bin/mise, the global core.hooksPath and hal's journal timer each have two owners on the day of the switch, and whether the live PC runs ufw is read only then, with the operator's leave.
9. FlexOr2

   added 4 commits that reference this issue on Nov 6, 2026

   The README opens with the guiding sentence of platform#1

   ...

   b8a0fe1

   Add the repository skeleton and the fleet's quality gate (#3)

   ...

   90fdadc

   The gate fails on legacy Sonar findings now that main is analysed (#7)

   ...

   2f7a814

   Three unused tokens leave hal's secret list (#1091)

   ...

   b6ae0be
10. FlexOr2 commented on Nov 6, 2026

    

    FlexOr2

    on Nov 6, 2026

    MemberAuthor

    More actions

    The operator's rulings of 06.11.2026 in platform's head tab, on the questions from the plan for P2, P3, P4 and #4. The head put each line to him in German with options and a recommendation; his words follow each line. Each line moves into its slice's body when that slice is cut.

    • Licence: MIT, like every other fleet repository except hal. „ja passt alles!“
    • Q1, the age key's backup: for now one key on the PC plus a copy in his password manager. A second, offline recovery key (both recipients in .sops.yaml, the recovery key on paper or USB) is required before the cutover, P10. „ja passt alles!“, answering the head's proposal „jetzt 1a, und 1b ist Pflicht vor der Umstellung (P10)“.
    • Q2, the gate of platform-secrets: quality's hygiene with the secret scan plus a check that every file is encrypted; no Sonar and no CodeQL, which on a private repository would cost money. „ja passt alles!“
    • Q3, the per-machine deploy key: the bootstrap shows the public key, and he adds it once, read-only, in GitHub's web interface, so it hangs on no token. „ja passt alles!“
    • Q4, the bump path: right after a release, the release's last job opens the bump pull request in platform through an organisation GitHub App that is installed only on platform and may write code and open pull requests. Several releases a day update the same pull request. He merges, then Flux deploys. Dependabot's docker ecosystem is not used for app images. His words: „weil wir releasen öfters am tag gerade? … so dass ich sofort updaten kann wenn ich will?“ and „also das muss auch möglich sein? das macht ja hal apply gerade bzw hal update und hal apply? das schon komfortabel? sonst muss ich ja warten?“
    • Q6, the GHCR pull credential: his own classic PAT with only read:packages and an expiry, watched by P9's expiry check. „ja passt alles!“
    • Q7, Docker on the host: Docker stays beside k3s for good, for agents' k3d runs and test containers, and the bootstrap owns it. „ja passt alles!“
    • Q8, write access to main is cluster-admin on the PC: accepted and recorded for now. Bumps are merged by hand until P10. Reopen trigger: auto-merge of bumps, or the agent account. „ja passt alles!“
    • Q9, the layout: Flux's documented layout, clusters/, infrastructure/, apps/ and host/ at the root, as four declared deviations from the root-placement rule for this repository. „ja passt alles!“

    The head decided these without a question: no Ansible, using mise bootstrap with a reopen trigger; scripts/bootstrap_pc.sh in place of ./bootstrap.sh, since quality's script-name and root rules forbid a script at the root; bump pull requests merged by hand until P10, re-asked at P10.
11. FlexOr2

    added a commit that references this issue on Nov 6, 2026

    The repository is licensed under MIT (#9)

    ...

    747581a
12. FlexOr2

    added a sub-issue on Nov 6, 2026

    • Every fleet repository's GitHub settings and the organisation rulesets live as code in platform #10
13. 43 remaining items

    Load more
14. FlexOr2

    added a sub-issue on Nov 9, 2026

    • The agents runner runs Mozart on the platform #281
15. FlexOr2

    added 5 commits that reference this issue on Nov 9, 2026

    hal setup installs quality v0.10.0 and its scanners; the hook runs ch…

    ...

    be3d3aa

    Stream every chat completion and read each round whole before it is j…

    ...

    540befa

    The delete test cites only the sentences it proves

    ...

    1aa4590

    docs/github-settings.md holds the six ruled sentences on GitHub setti…

    ...

    fe101f8

    docs/VISION.md states what the platform promises, for whom, and what …

    ...

    ef03296
16. FlexOr2

    added a sub-issue on Nov 9, 2026

    • The cluster job's two in-cluster sources name addresses, so a lost DNS record cannot fail the job #315
17. FlexOr2

    added a commit that references this issue on Nov 9, 2026

    A request waits for a running key-set refresh when no current key set…

    ...

    1297cf9
18. FlexOr2

    added a sub-issue on Nov 9, 2026

    • presentator and presentator-talk serve from the platform, so that hal can stop #322
19. FlexOr2

    added a commit that references this issue on Nov 9, 2026

    An organisation ruleset blocks the merge of a pull request that bring…

    ...

    e0d3586
20. FlexOr2

    unmarked this as blocking 2 issues on Nov 9, 2026

    • songmaker's secrets are files the platform mounts; none stays in .env songmaker#2206
    • The GPU worker writes only its output folder and sits on its own network (#1301 F1-C) songmaker#2271
21. FlexOr2

    added a commit that references this issue on Nov 9, 2026

    webauth ships its in-memory stores as a public adapter (#699)

    ...

    a02e283
22. FlexOr2

    added a sub-issue on Nov 9, 2026

    • The platform's stateful pods are not the kernel's first victim when the PC runs out of memory #370
23. FlexOr2

    added a commit that references this issue on Nov 10, 2026

    One standard wiring installs webauth with its defaults and the app's …

    ...

    10fad4c