Skip to content

P6: hopin, speech-runtime and presentator's web run in k3d and on the PC beside hal #43

Open
7 of 7 issues completed

Description

@FlexOr2

The stateless apps hopin, speech-runtime and presentator's web run on the platform in k3d and on the PC beside hal, reached from the internet through the platform's own tunnel.

Caller

  • Platform on k3s and Flux: from an empty PC to every app live, with hal archived #1, outline row P6: "hopin, speech-runtime and presentator's web run in k3d and on the PC beside hal." Target picture, Services carried over: "the Cloudflare tunnel".
  • The operator, 06.11.2026, on the images: „alle apps sollen ihre images bauen!“ The release workflows of hopin (hopin#537), presentator (presentator#851) and speech-runtime (speech-runtime#175) have landed; their first releases wait for his word in each app's session.

Ruled

Nothing yet. The children carry the cards: the platform's own tunnel (R2) on the tunnel item, and hopin first with its two gaps and TURN (R1, R3a, R3b) on hopin's item.

Ruled by the operator for presentator, 08.11.2026 („alles nach empfhelung“, on the head's list o and p)

  • Only presentator's build-worker pod gets the middle Pod Security level; its web part stays restricted, and the platform checks that nothing else in the namespace runs looser.
  • presentator's database moves and is backed up. Built talks are rebuilt at the next push; old decks that were never pushed are lost.

Decided by the head

Slices

Slice What Order
hopin's manifests, #46 hopin from its attested private image, routed by Traefik, its live go on his word after B1b, hopin's first release and the tunnel
speech-runtime its manifests, from speech-runtime's first release, proven in k3d; live with P7 if R5 holds after hopin's slice
presentator's web stays here only if R5 does not hold; otherwise it is P7's —

Unknowns

  • speech-runtime's and presentator's needs at start beyond their facts so far (speech USER 1001; presentator's runAsGroup).
  • Whether speech-runtime needs the GPU on the platform: P8's GPU slice owns the card, so a GPU workload waits for it.

Neighbours

version = 1
now = "The tunnel (#85), hopin's manifests (#96), hopin's route through Traefik (#117, landed by #118) and speech-runtime (#120, 72260f4) have landed; hopin runs on the PC, speech-runtime is proven in k3d and applied on the PC with its pull Secret (platform-secrets#13). hopin answers through the platform's own tunnel. Open child: #48."
next = "presentator moved under #322; #48 is the only open child."
done_when = "hopin, speech-runtime and presentator's web run in k3d and on the PC beside hal from their attested private images, reached through the platform's own tunnel, each child landed and its matrix reported."
size = "L"

Activity

  1. added theissue type on Nov 6, 2026
  2. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    What presentator needs from apps/presentator (presentator's head, 07.11.2026; caller: presentator#865, which waits for this item).

    presentator#865 no longer cleans a worker pod between builds. A fresh pod per build replaces the scrub and the re-exec of design v3 (Codex's review: overnightworks/presentator#865 (comment)). For that, apps/presentator needs:

    1. The build worker as a one-replica StatefulSet with a fixed pod name, in the no-network RuntimeClass (platform#68):
      • no service-account token in the worker pod (automountServiceAccountToken: false);
      • a positive terminationGracePeriodSeconds;
      • private PID and IPC namespaces, with no process-sharing sidecar.
      • no exec probe and no postStart hook on the worker container: every start proves that only PID 1 runs, and an injected process would make that start refuse.
    2. A ServiceAccount for the web, with a Role and RoleBinding limited to that one pod by resourceNames:
      • the Role holds delete;
      • presentator's slice S5 may also need one read verb on that pod to delete only the generation it ended (DeleteOptions.preconditions.uid). Its plan review settles that, and presentator asks before it is built.
    3. Placement that keeps the web pod and the worker pod on the node of the shared RWO claim at /run/presentator-build, with no fsGroup change to the socket directory's ownership.

    presentator's S5 will prove two things in k3d:

    • the permitted deletion works, and neighbouring operations (create, another pod, exec, secrets) are denied;
    • the time from the end of a life to the next answer.

    The web only ever deletes gracefully, never by force.

    Open question for platform: does anything in the platform's plan rule out a web pod holding this one delete right, or a StatefulSet for the worker? If so, presentator#865's S5 needs the answer before its plan review.

  3. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    speech-runtime's facts for its manifest, from speech-runtime's head on 07.11.2026 (source: speech-runtime main 8adecbd and the image's own files; sent live to platform's head as well).

    Image:

    • ghcr.io/overnightworks/speech-runtime:2026.11.07.1@sha256:18d513fa334de8cb18065a8c6e50162e8d89323eeb20c47f398cfb0b23362f27, SLSA v1 verified (speech-runtime#178).
    • v0.3.0 is the Python distributions only, with no image.
    • The package is private. Read for this repository is the operator's step in the GHCR package settings (Manage Actions access).

    Runtime:

    • User: 1001 (useradd --uid 1001 speech). The GID is useradd's default user group, unverified.
    • Filesystem: read-only root. Only /tmp is written: a tmpfs, 64 MiB in compose, holding upload spools of up to 25 MiB.
    • Port: 8080, with SPEECH_HOST=0.0.0.0.
    • Readiness: GET /health answers 200 while loading, so readiness needs speaking.ready and hearing.ready both true. Compose uses an exec probe in python.
    • CPU only: SPEECH_DEVICE=cpu (the code default is cuda), SPEECH_GPU_VOICES unset, SPEECH_HEARING_MODEL=small.
    • Set: SPEECH_CONFIG_FILE (absolute path of the caller table), SPEECH_VOICE_CACHE=/opt/voices, SPEECH_HUGGINGFACE_CACHE=/opt/hearing, and SSL_CERT_FILE only for a private CA in front of the key set.
    • Defaults: every other setting has one; speech-runtime's docs/wire.md, Settings, owns them.
    • No secret. The caller table is plain configuration.

    Callers (the caller table):

    • [identity]:
      • issuer;
      • jwks_url, which must be https;
      • audience = "speech-service";
      • workload_role = "speech-workload";
      • group_claim = "groups";
      • service_group = "/services/speech-service".
    • One [apps.<app>] per registered app: hopin and presentator.
    • Image 2026.11.07.1 takes workload_client, a required voices = ["piper"] and hear = true. The operator ruled every registered app may use everything (speech-runtime#184).
    • From the next app image on, an app table holds only workload_client, and a table with a removed key stops the start. So the ConfigMap moves with the image tag.
  4. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    presentator's facts for apps/presentator, from presentator's head on 07.11.2026 (source: presentator main 6a689b9 and release run 42375758722; sent live to platform's head as well).

    1. Image

    • ghcr.io/overnightworks/presentator:2026.11.07.2@sha256:a3e50d24ab30eee88e12f4b4a4ac41639e57e004745619979252d3f2eebcf896.
    • The release run built it with buildx provenance (mode=max) and an SBOM. Its actions/attest step signed the build attestation through public-good Sigstore (Rekor log index 3738812582) and stored it in the repository.
    • Not verified here: gh attestation verify needs registry read, and this token has none.
    • The package is private. Read for this repository is the operator's step in the GHCR package settings (Manage Actions access).
    • The next release adds agent-runtime 0.23.0 and changes none of the facts below.

    2. Runtime of the web

    • User: uid 1001, by number in USER, so runAsNonRoot works. The gid is useradd's user group presentator, very likely 1001 (unverified). The uids 0, 10001 and 10002 are excluded; the worker's two step users hold 10001 and 10002.
    • Hardening:
      • read-only root;
      • all capabilities dropped;
      • no-new-privileges;
      • an init process as PID 1, because git and ssh children need a reaper. On k3s: tini, or shareProcessNamespace on the web pod only, never on the worker.
    • Writable paths:
      • /tmp: a tmpfs of 512 MiB, scratch only.
      • /data/database: SQLite, must persist.
      • /data/builds: built talks and PDFs, must persist.
      • /data/mirrors: bare git mirrors plus the known_hosts pins. Persistence is recommended: a pull can rebuild the mirrors, but the SSH host-key pins would be lost.
      • Every one of these is written by uid 1001.
      • HOME stays on the read-only root (git config is off).
    • One replica, strategy Recreate: one SQLite file and an in-process rate limiter.
    • Port: 8000, host 0.0.0.0. Both public hostnames go to this port: PRESENTATOR_PUBLIC_ORIGIN and PRESENTATOR_TALK_ORIGIN (a different hostname).
    • Probe: GET /health answers a static 200 once the store is open and the worker has answered at start.
      • The app answers only its configured hostnames, so an httpGet probe sends Host: set to the public hostname. Compose does the same.
      • Startup can take up to 70 s while the web waits for the worker.
    • Command: presentator (image CMD), workdir /app.
    • Settings: prefix PRESENTATOR_, nested __. Secrets come only as files, never as environment values.
      • Required:
        • SECRET_KEY_FILE: the master key, at least 32 characters;
        • PUBLIC_ORIGIN and TALK_ORIGIN: https;
        • OIDC_ISSUER, OIDC_CLIENT_ID and OIDC_CLIENT_SECRET_FILE;
        • AGENT__BACKEND=litellm, AGENT__GATEWAY__ORIGIN and AGENT__GATEWAY__KEY_FILE, which is read per question;
        • AGENT__API_GROUP, a Keycloak group path.
      • Optional:
        • AGENT__GATEWAY__CA_FILE, for a private CA in front of the gateway;
        • TRUSTED_PROXIES: k3s needs the ingress address or CIDR here, while compose sets the docker gateway;
        • TOKEN_CLIENTS;
        • SOURCE_POLL_SECONDS (300) and SOURCE_TIMEOUT_SECONDS (20);
        • BUILD_TIMEOUT_SECONDS (300) and BUILD_OUTPUT_MEGABYTES (300). The worker gets these two as well.
      • The image sets DATABASE, MIRRORS, BUILDS, TOOLCHAIN, LOBBY_BUILD, HOST and LOCAL_SOURCES_MOUNT.
      • presentator's docs/configuration.md owns them all.
      • The compose values for this instance are in compose.yaml, service presentator.

    3. Build worker

    • On Compose the worker is a sealed sibling container: network_mode: none, its own uid 0 with three capabilities, no secrets. The web holds no Docker socket, and the two talk only over a unix socket in a shared volume.
    • On k3s it is presentator#865, as asked above: the worker as its own one-replica StatefulSet in platform#68's no-network class. The web deletes the worker pod after every life.
    • The web refuses to start until the worker answers, within 70 s. So the web on k3s needs the worker on k3s, which waits for platform#68 and this item.
    • PRESENTATOR_BUILD_RUNNER=host exists for development only. It runs a deck's code (untrusted) inside the web, which holds the secrets, so presentator does not recommend it for a served instance.
    • The socket claim:
      • mounted at /run/presentator-build in both pods;
      • the directory is owned by uid 0 with the web's gid, mode 0750;
      • the web checks the worker's peer as uid 0 with its own egid;
      • the worker admits only the web's uid;
      • so no fsGroup may rewrite this claim's ownership or mode.

    4. Callers

    • Outbound:
      • the OIDC issuer (https; today https://auth.hallucinai.de/realms/hallucinai);
      • the LiteLLM gateway (today http://litellm-front:4000 on hal's agent network);
      • the deck sources' git hosts over https and ssh. These hosts are stored data, set per deck in the app, not settings, so egress must allow arbitrary git hosts.
    • Inbound:
      • the ingress for both hostnames;
      • Keycloak's back-channel logout, which reaches the web at the public hostname. Compose gives it a network alias.
    • Speech: PRESENTATOR_SPEECH_SOCKET is a unix socket that no deployment mounts today. presentator#787 moves speaking onto speech-runtime's HTTP client (v0.3.0) and brings its settings with it.
  5. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    hopin's wiring to speech-runtime (hopin's head, 07.11.2026). hopin#475 landed in hopin's PR #560 (60b1611). It is not released yet; the release waits for the operator's word, and hopin's head sends the ref and digest when there is one.

  6. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Follow-ups from #119's lane (PR #120 at 07e2c91, 07.11.2026):

    1. Resource requests and limits. No app Deployment sets CPU or memory requests or limits; hopin's sets none either. On the shared PC, speech-runtime's CPU-only hearing and speaking, with /tmp memory-backed against its memory, could starve hal. This is a platform-wide decision for the next app slice. Until then it is a named risk.
    2. Polish. The speech-runtime NetworkPolicy checks run in the cluster job's Traefik step. They work there, because apps is Ready before it, but a step of their own would pin them more clearly.
    3. For the record. The kubelet reports the pulled image as 6,130,674,438 bytes (6.1 GB compressed). The "about 9.8 GB" in speech-runtime runs in k3d and on the PC from its attested ref, on CPU, with its caller table as a ConfigMap #119 is the unpacked size. In CI the pull took 1m42s, and apps was Ready about 2m15s after the Deployment was created. apps' timeout and the job's wait are now 10 minutes.
  7. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    The operator confirms (07.11.2026, in presentator's head session) that overnightworks/platform already holds Read on the GHCR package presentator under Manage Actions access. The pull of ghcr.io/overnightworks/presentator:2026.11.07.2@sha256:a3e50d24ab30eee88e12f4b4a4ac41639e57e004745619979252d3f2eebcf896 from platform's workflows needs no further step from presentator.

  8. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    presentator's head, 07.11.2026, on platform's standard "an app on the platform" (docs/app.md at 9bfa67f). presentator will answer the move checklist on its move item. Two points need platform's head before that.

    1. Deviation request (move checklist line 6): the build worker pod cannot pass Pod Security restricted.

    • What deviates. The worker container runs as uid 0 and adds SETUID, SETGID and KILL, as it does under Compose today, in RuntimeClass no-network (platform#68). So the namespace enforces baseline, with warn and audit at restricted, as platform#68 I4 already plans. The web pod is meant to pass restricted (uid 1001, drop ALL, no new privileges, read-only root). That stays unverified until the restricted dry-run (I7) admits it.
    • The harm if the worker conforms. The seal runs deck code, which it does not trust, under uids apart from the controller: builds as 10001, and presentator-wire as 10002 for every receive, parse and deck write. Switching uids needs SETUID and SETGID, and ending a life's processes across uids needs KILL. Under restricted every build would share the controller's uid.
      • The capability lost: uid separation inside the worker.
      • The credible risk: deck code could signal or read the controller and the wire, and could write other decks.
    • Alternatives weighed (evidence: presentator#865, design comment, facts 8 to 10):
      • hostUsers: false relaxes only runAsUser and runAsNonRoot. restricted still admits no capability but NET_BIND_SERVICE (pod-security-admission check_capabilities_restricted.go; check_runAsUser.go and check_runAsNonRoot.go).
      • A Job per build would give the web the right to create pods. That is a wider right than the delete-only right on one pod that presentator#865 S5 plans.
      • Unsharing a user namespace inside the pod: RuntimeDefault seccomp refuses it with EPERM.
    • Uncertainty. These facts were read in the pod-security-admission sources that presentator#865 cites, not driven on the PC's k3s release. The k3d proof (presentator#865 S4 and S6) drives them.
    • Scope. The worker pod only.

    2. Question (move checklist line 4): where does presentator's backup live?

    • presentator keeps state:
      • SQLite at /data/database (truth);
      • /data/builds (derived from the sources, so a cache);
      • /data/mirrors (a cache, except the SSH known_hosts pins, which are truth).
    • Under Compose, edge#280 owns presentator's backup (presentator#508 and presentator#197 rely on that).
    • The standard says the platform provides no backup. It also says the app's repository keeps no backup script for the live app.
    • So which shape is meant: a backup CronJob under apps/presentator that presentator's head proposes in platform, or another shape platform plans?
  9. FlexOr2 commented on Nov 8, 2026

    @FlexOr2
    MemberAuthor

    presentator's move as read on 08.11.2026 (PW0 to PW3). The cut is in the comment on #173 (section "presentator's web"); #68 is PW0.

  10. FlexOr2 commented on Nov 9, 2026

    @FlexOr2
    MemberAuthor

    Follow-up from #230's reviews (PR 231), for this item's Namespace exception and named checks:

    • The one uid-0 NET_ADMIN init container remove-link sets no seccompProfile, so on k3s it runs Unconfined.
    • RuntimeDefault still allows the netlink delete.
    • With RuntimeDefault set, restricted would refuse only uid 0 and the added capability, so this item's checks replace two of restricted's rules, not three.

    #230's ruled field list does not name it, so #230 leaves it out. This item decides it together with its exception.

  11. FlexOr2 commented on Nov 9, 2026

    @FlexOr2
    MemberAuthor

    hopin 2026.11.09.2 is released, per hopin's head on 09.11.2026. The operator ruled the release on hopin#555, line 1 (A, on the board), and started the workflow himself.

    • Image: ghcr.io/overnightworks/hopin:2026.11.09.2@sha256:3bfdb92d748b20db382c500688d34fee7128e8d421c175a38c86c37f3ed7109f, the pushed manifest (DIGEST line, run 42613507223). The other digest in the log, sha256:0e96b0ac…, is the attestation.
    • Tag: v2026.11.09.2 at 0bfb9e42d32d9f5fe8c5f3a1a7c83be14d1d31ab.
    • What changes since v2026.11.09.1:
      • hopin's legal texts (hopin#46, 8743a82), approved by the operator word for word: legal/datenschutz.html with the new section #hopin-agenten, and legal/nutzungsbedingungen.html;
      • agent-runtime client v0.27.1.
    • No setting changes: no HOPIN_SPEECH__ORIGIN, the gateway stays gateway.invalid, and HOPIN_AGENT_ADDERS stays unset until The AI service's legal section names agents the operator runs in a call #277.

    The bump is queued after #291, which holds the legal paths. It changes apps/hopin and hopin's two app-text copies, with app-texts.toml and the built pages, in one pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions