Skip to content

P1: every app release yields an attested private image in GHCR #4

Open
2 of 2 issues completed
Open

Description

@FlexOr2

Every app release, and the Keycloak image of the platform, yields an attested private image in GHCR that the platform pins by digest.

Caller

  • platform#1, slice P1, target picture under Images: every app image is built in its own CI and stored privately in GHCR, and the PC pulls it with a read-only token.
  • The operator, hal head chat, 06.11.2026, on model weights: „den besten weg gehen … wenn sie in ein volume gehören sollten wir das so machen oder nicht idk“. Where weights live is decided per service here, by what serves best.
  • One release path for the fleet: quality's release gate, image build and tag workflows quality#364 owns the reusable image workflow as its own slice, with hopin as first caller. Platform's inputs for it are posted there (comment of 06.11.2026); quality's head confirmed they fit its cut. The bump from a release to Flux stays this item's.

What is known

  • The interface (proposal on quality#364): one image per call; outputs digest and ref; platforms linux/amd64; one optional build-secrets secret as BuildKit mounts; push: false for the pull request proof.
  • Tags: apps keep their v* tags; images are built only for a release (hal ruling of 16.10.2026: only a v* tag goes live).
  • Sizes of the images on the PC: Platform on k3s and Flux: from an empty PC to every app live, with hal archived #1, third comment.

Ruled

  • The bump path (the operator, 06.11.2026, platform's head tab; recorded on Platform on k3s and Flux: from an empty PC to every app live, with hal archived #1): right after a release, the release's last job opens the bump pull request in platform through an organisation GitHub App installed only on platform, with write access to code and pull requests. Several releases a day update the same pull request; he merges, then Flux deploys. Dependabot's docker ecosystem is not used for app images. His words: „so dass ich sofort updaten kann wenn ich will?“ and „das macht ja hal apply gerade bzw hal update und hal apply? … sonst muss ich ja warten?“
  • The guard: platform's CI refuses a bump whose app tag is missing or points at another commit than the attested image; quality#364's image builds no guard of its own (quality's head, 06.11.2026).

Open, settled at this item's cut

  • Weights per service: speech-runtime bakes them so it runs with no egress; songmaker's scoring worker bakes 3 GB of Whisper weights and uses an HF token in the build; songmaker's acestep image holds no weights. Decided after the runner-disk measurement.
  • GHCR: private-only package creation in the organisation, and the PC's read-only pull credential, which is a secret and lands with P4.

Neighbours

version = 1
now = "quality#364's image workflow releases hopin and speech-runtime as attested private images, which run on the PC. Children landed: #49 (the images job judges every ref) and #83 (21046c9, it refuses HelmRepository and HelmChart too). quality#364 stays open for the rest of the fleet's release path."
next = "Each remaining app (presentator, songmaker, skat, Keycloak's theme image) gets its attested release at its move; the bump path's plan follows the app standard (#126)."
done_when = "quality's image workflow ships; every app's release and the Keycloak image yield an attested private GHCR image; the bump path from a release to a platform pull request is decided, built and proven once; the weights of each service have their decided home."
size = "L"

Activity

  1. added theissue type on Nov 6, 2026
  2. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    Inputs for this item's cut, from the plan for P2 to P4 (v3, 06.11.2026; plan review Codex gpt-6-astra high and Opus 5.5 high: APPROVE).

    The operator's ruling on the bump path (the "Ruled" section above, a GitHub App opening the pull request right after a release) binds over the plan text below. So the plan's Dependabot route (D5 route A) and its Dependabot spike B1 are not built. What carries over to the cut:

    • the release-eligibility check: an attested image, and its tag resolving to the attested commit;
    • the attestation flags and cells E7 to E9;
    • the four-part tag form;
    • the unknowns.

    Verbatim below.


    D5. The bump path from an app release to a platform pull request

    Releases flow: release-gate, then image (outputs digest and ref), then release-tag (quality#364). So an image can
    exist in GHCR whose release-tag step failed afterwards.

    Route Latency Rights needed New code Failure modes
    A. Release job with a GitHub App token (actions/create-github-app-token) opens the pull request in platform minutes An org GitHub App with contents and pull-requests write on platform only, its private key an environment secret on a protected release environment in each app repository About 40 lines: a reusable bump workflow owned by platform The release succeeds but the bump fails (re-run, idempotent branch). A leaked App key can open pull requests in platform. The key needs rotation.
    B. Dependabot docker ecosystem on the Flux tree its schedule, daily at best (cron minimum unverified) None new. Dependabot reads private GHCR packages through its GITHUB_TOKEN once each package grants platform Read under "Manage Actions access", one grant per package (https://docs.github.com/en/code-security/dependabot/working-with-dependabot/configuring-access-to-private-registries-for-dependabot; Opus table row 14; API for the grant unverified) About 10 lines of dependabot.yml Updates Kubernetes YAML and keeps @sha256 (Opus table row 13). Orders vYYYY.MM.DD.N numerically once v is stripped, but only among tags of the same precision: a tag without .N is silently skipped (Opus table row 15). It may propose an image whose release-tag step failed; the eligibility check below makes that pull request red.
    C. Flux image automation reflector interval plus a pull request A write deploy key to platform on the PC, within agents' reach (D14), and a token to open the pull request: events caused by GITHUB_TOKEN start no new workflow runs except workflow_dispatch and repository_dispatch (https://docs.github.com/en/actions/concepts/security/github_token) ImageRepository, ImagePolicy, ImageUpdateAutomation per app, plus a pull request action A cluster credential can write the source of truth. Rejected.
    D. Renovate (hosted app, or self-hosted through its GitHub Action) minutes to hours (unverified) A token with write on platform and read on the private packages renovate.json A second bot beside the fleet's Dependabot. quality docs/gate.md names Dependabot for workflow pins; it bans no other image tool, so this is upkeep, not a rule.

    Recommendation: B, gated by the spike in B1 (an app's real release yields a Dependabot pull request in platform
    from …:vA@sha256:x to …:vB@sha256:y). If the spike fails, or the operator wants minutes instead of hours (Q4), B2
    first weighs Renovate's self-hosted action and a maintained pull request action against A before anything is written.

    Release eligibility (revised for Opus v2 follow-up 2). A bump is mergeable only for a completed release. Platform
    CI gets a job images, in mandatory_jobs and quality.needs (plan invariant 4), that runs on every pull request and
    push and checks every ghcr.io/overnightworks/… ref in the Flux tree (apps/ and infrastructure/, so the
    platform-built Keycloak image is covered; not only changed refs, so it needs no diff logic and a later tag deletion
    also shows). Third-party images (the local-path provisioner, Traefik's chart) are outside it. Per ref
    ghcr.io/overnightworks/<repo>[/<name>]:<version>@sha256:<digest>:

    1. The tag. git ls-remote https://github.com/overnightworks/<repo> refs/tags/<version> (quality#364's tag is
      lightweight, so no peeling) yields the tag's commit; no tag means red, naming the missing tag. <repo> is the
      image path's first segment after the owner: an app image is named after its repository.
    2. The attestation, enforced only through flags on unforgeable fields. gh's own help says "only the
      signature.certificate and the verifiedTimestamps properties contain values that cannot be manipulated by the
      workflow that originated the attestation" and warns about statement.predicate; so the job reads nothing from the
      predicate and runs gh attestation verify oci://<ref> --repo overnightworks/<repo> --signer-workflow overnightworks/quality/.github/workflows/<image workflow> --source-digest <tag's commit> --source-ref refs/heads/main --predicate-type <the type quality#364's actions/attest writes> --deny-self-hosted-runners (flags from gh attestation verify --help, gh 2.103.2). --repo ties the image to its
      own app's repository; --source-digest ties it to the tagged commit; --source-ref holds because quality#364's
      gate refuses a ref other than main. gh defaults --predicate-type to https://slsa.dev/provenance/v1; the job
      passes the value explicitly, matched in B1 against quality#364's landed workflow (its body says "provenance
      mode=max, an SBOM and actions/attest", so an SBOM attestation of another type may sit beside it).
    3. The signer's version. --signer-workflow pins no ref. Pinning one quality commit with --signer-digest is
      rejected: each app pins its own quality sha and Dependabot moves it, so platform would hold a copy of every app's
      pin. Instead the job reads the signer digest from signature.certificate (unforgeable; exact field name unverified,
      B1) and requires it to be a commit on quality's main (GitHub's compare API, ahead or identical), so an image
      workflow edited on a quality branch does not pass.
    4. Keycloak. Its image is built by platform's own release.yml calling quality's image workflow (quality#364
      lists "platform's Keycloak" as an app caller), so its <repo> is platform and its tag a platform v* tag. Its
      name must keep the first-segment rule (for example ghcr.io/overnightworks/platform/keycloak; GHCR names with
      several segments unverified); naming it is the Keycloak item's at P6, and B1 builds nothing for it without that
      caller.

    Because quality#364 creates the tag last, an existing tag at the attested commit means gate, image and tag all
    completed. The failed-release case (image pushed, tag step failed) has no tag: the job is red and the Dependabot pull
    request cannot merge (main requires quality / verdict). If the tag step is re-run successfully, a re-run of the job
    turns green; the next release supersedes the pull request either way.

    Alternative weighed: quality#364 pushes only a digest and adds the GHCR v* tag after the git tag, so Dependabot never
    sees an unreleased version. It would spare red pull requests, but it changes quality#364's design, and platform must
    verify what it deploys anyway. Named to quality#364 as an option, not a demand.

    Constraints on P6's layout and the apps: image refs stay in a Deployment's image: field (Dependabot support for
    kustomize images: or HelmRelease values is unverified); app tags keep four parts until quality#364 decides the app
    grammar; an image's first path segment is its source repository; each private package grants platform Read (for
    Dependabot and for the images job's packages: read).

    Item #4: bump path (its open part)

    Invariants kept: 2, 3; the bump never changes anything but an image ref; merging stays a person's yes (head decision,
    section 4) until P10 re-asks.

    • B1 (first slice). Done when an app's real release (hopin, after quality#364's image slice) yields a Dependabot
      pull request in platform that moves ghcr.io/overnightworks/hopin:vA@sha256:x to vB@sha256:y, and the images
      job (D5) holds every cell below.
      • Files: .github/dependabot.yml (the docker entries for the Flux tree's directories, schedule);
        .github/workflows/ci.yml (job images with permissions: contents: read, packages: read; quality.needs
        gains images); quality/config.toml (mandatory_jobs plus images); scripts/verify_image_release.sh
        (about 30 lines: parse each ref, git ls-remote, gh attestation verify with D5's flags, the signer-digest check;
        it only calls gh and git).

      • Outside the diff: the package grant (platform Read under "Manage Actions access"), one per package, by the operator
        or the head.

      • Settled against quality#364's landed workflow before the job is written: the image workflow's file name, the
        --predicate-type value, and the certificate field holding the signer digest.

      • Depends on quality#364 (image), P6's hopin manifest, P4d.

      • Cells (release eligibility, material risk):

        Cell State Expected
        E1 a completed release vB (tag at the attested commit) images green; the Dependabot pull request mergeable
        E2 failed release: a real attested digest whose version has no git tag (driven with a real digest and an untagged version string; if quality#364's own matrix leaves a real pushed-but-untagged image, also against that) images red naming the missing tag; quality / verdict red
        E3 the tag exists but points at another commit than the attestation's source (vB@<digest of vA>) red (--source-digest)
        E4 a digest not signed by quality's image workflow (another workflow, or no attestation) red (--signer-workflow)
        E5 a Dependabot-triggered run the job reads tags and attestations with its read-only token; packages: read works or the gap is named
        E6 a four-part tag next to a three-part one in GHCR Dependabot proposes only the four-part line (documents the grammar constraint)
        E7 an image whose attestation names another app's repository than its image name (hopin's name, a digest built for another app) red (--repo)
        E8 a digest whose signer digest is not on quality's main (a quality branch commit, if one can be produced without touching quality's main; otherwise judged by reading the check against this row) red
        E9 an attestation whose --predicate-type differs (for example the SBOM attestation alone) red; the job never reads statement.predicate
      • Documentation: README's "shipping an app" line, once true; the four-part tag and first-segment naming constraints
        to quality#364 and the app heads.

      • Verification: the observed pull request; CI; the cells.

      • Review: material risk (release eligibility is what keeps "only a v* tag goes live"): cross-provider gate.

      • Rollback: remove the entry and the job (a declared loosening for the removed mandatory job).

      • Deletes: songmaker's auto-deploy and the others in P11, named on Platform on k3s and Flux: from an empty PC to every app live, with hal archived #1.

      • Named gaps: who may create v* tags in an app (a tag ruleset) is the app's and quality#364's (its open creation
        rule); a fork pull request in platform could read private images through packages: read, held by GitHub's
        approval for outside contributors' workflows; the Keycloak image's name and tag scheme (D5 point 4) are the
        Keycloak item's at P6.

    • B2. Only if B1's spike fails or Q4 picks (b): first compare Renovate's self-hosted action and a maintained pull
      request action against route A, then build the cleanest; done when the same pull request appears within minutes and
      E1 to E4 and E7 to E9 still hold.

    Unknowns rows for this item (plan section 5):

    | Unknown | Settled in |
    | packages: read for Dependabot-triggered runs; the package-grant API | P4d, B1 |
    | Dependabot's cron minimum; the image workflow's file name, its predicate type and the certificate field holding the signer digest | B1 |

  3. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    Facts from quality's head (06.11.2026) for B1's check. The operator gave quality#364's image workflow top priority in quality's session; it is planned and built now, beside slice 1b (release-tag) and the gate's app version form.

    • App grammar: vYYYY.MM.DD.N, which is today's practice. hopin, presentator, songmaker and speech-runtime tag this way, read through the tags API. Month and day are zero-padded; N starts at 1. The gate adopts it in its own small lane.
    • Image path: ghcr.io/overnightworks/<repo>[/<name>], formed from the caller's repository. The first segment is the app repository, as B1 assumes.
    • Provenance: SLSA provenance v1 through actions/attest-build-provenance, pushed to the registry. Still to be confirmed once quality's plan is written: the exact action, and whether the SBOM is a second GitHub attestation or BuildKit's in-image SBOM.
    • Source ref: refs/heads/main for a dispatched release, since quality's gate refuses any other ref. Dispatch is the standard's only path.
    • Signer: quality's .github/workflows/image.yml. B1 passes --signer-workflow overnightworks/quality/.github/workflows/image.yml. It also requires the certificate's signer digest to be a commit on quality's main.
    • Tag dependency: B1 needs a lightweight git tag at the attested commit. That tag comes from release-tag (1b), or from an app's hand-pushed dated tag until 1b lands.
  4. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    quality's image workflow has landed: quality#402, PR #404, merge 89069a1a97e1ef9da92f1cc37b45064e21962f77 (quality's head, 06.11.2026).

    • Call it pinned by SHA: uses: overnightworks/quality/.github/workflows/image.yml@89069a1a97e1ef9da92f1cc37b45064e21962f77. quality has no tag before The legal sections and their declared Stand move to platform #228's R.
    • Inputs: context, file, target, image, version, build-args, push, cache.
      • image is the sub-name below the repository's path; empty means the repository itself.
      • version is a valid Docker tag.
      • push: false only builds.
      • cache is off by default.
    • Secret: only build-secrets, passed by name. There is no secrets: inherit.
    • Outputs: digest, and ref = ghcr.io/<owner>/<repository>[/<image>]:<version>@<digest>, lowercased.
    • The caller's job grants: attestations: write, contents: read, id-token: write, packages: write.
    • Build: github.sha, linux/amd64 only.
    • Attestation:
      • actions/attest writes SLSA provenance v1, pushed to the registry, with no storage record.
      • BuildKit adds provenance mode=max and an in-image SBOM (sbom: true). The SBOM is not separately signed.
      • B1 verifies with --predicate-type https://slsa.dev/provenance/v1 (the default).
    • Next in quality:
    • Still open:
      • The first real app dispatch (push and attestation) waits for the operator's yes.
      • Package visibility and Actions access per package are settings of the app or of platform.
  5. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    presentator's caller of quality's image workflow is overnightworks/presentator#851 (template hopin#537, blocked by quality#406 and quality#407). It builds one image: the sealed build worker runs the same image with its own command. What a Deployment needs (uid, volumes, port and health, one replica, file secrets, the worker's seal, and the open question of where the worker runs) is in overnightworks/presentator#851 (comment).

  6. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    GHCR storage and retention (platform's head, 06.11.2026).

    • Cost today. GitHub's billing page (docs.github.com/en/billing/concepts/product-billing/github-packages, read 06.11.2026) says: "Container image storage and bandwidth for the Container registry is currently free. If you use Container registry, you'll be informed at least one month in advance of any change to this policy." The general private-package quota (Team: 2 GB storage shared with Actions artifacts, 10 GB transfer per month) is the documented fallback should that change. A pull from Actions with GITHUB_TOKEN never counts.
    • Gap: retention. GitHub deletes no old package version on its own. songmaker's releases push about 15 GB of fresh layers each (acestep worker about 9 GB, scoring worker about 6 GB, uncompressed). platform decides which versions run, so platform owns a cleanup rule that keeps every version a manifest on main references, plus a rollback window. That rule has to exist before GitHub's one-month notice could make storage cost money. It is not a blocker for the first releases.
    • Image names confirmed to songmaker: ghcr.io/overnightworks/songmaker/{web,music-worker,scoring-worker,acestep-worker}.
    • Callers on quality 7715afe5c911a00ae3076057691e0b5d4e4a65f7: hopin#537, presentator#851, speech-runtime#175, songmaker#2394. Per quality#364 outline line 7, platform's CI owns the guard against an image without its release tag.
  7. changed the issue type fromtoon Nov 6, 2026
  8. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Caller: overnightworks/hopin#561, blocked by this item (hopin's head, 07.11.2026). It adds the bump workflow as the last job of hopin's release.yml and passes the Image job's ref and digest. hopin's release.yml has only the jobs Gate, Image and Tag today, so the bump pull request into platform does not exist yet; until it does, a pin changes by a hand-written platform pull request the operator merges. Once the workflow has a ref, hopin's head starts #561.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions