Skip to content

The platform's own Cloudflare tunnel runs cloudflared in k3s in front of Traefik #45

Closed
3 of 3 issues completed
Closed

Description

@FlexOr2

The platform's own Cloudflare tunnel runs cloudflared in k3s and sends hopin.hallucinai.de to Traefik and every other host to 404, its configuration in Git, and Traefik's web port admits only the tunnel and trusts its forwarded headers.

Caller

Ruled

Decided by the head

Plan-reviewed, because this is a new boundary and a new source of truth for the edge. Revised 06.11.2026 after the Codex plan review (REVISE): its findings 1, 2, 4 and 5 are answered in decisions 5 and 12, the cells C10 and C12 and #46; finding 3 in the card. The Codex delta re-check of the same day approved the revision (APPROVE: all five blocking findings repaired, none new; its one follow-up, decision 12's option qualifier, is applied), as #43 records.

  1. A second tunnel owned by platform, cloudflared in k3s. The tunnel is called platform; hal's tunnel, its cloudflared unit and its catalog stay untouched and leave with hal (P11). The card (R2) ruled where the tunnel's lines live: in Git (A, the operator, 06.11.2026).

    • Rejected: hal's tunnel credential in k3s with a catch-all back to 127.0.0.1:8081. That moves hal's secret, breaks hal offline, and needs a host port into hal (live-go plan, section 5).
    • Rejected: hal's tunnel sending one host to the cluster through a NodePort or hostPort. P3d forbids both, and the port check enforces that.
  2. The credential is a tunnel token, read from a file.

    • Run: cloudflared tunnel --no-autoupdate --metrics 0.0.0.0:2000 --config <config file> run --token-file <token file>.
    • Why a token: cloudflared 2026.10.7 starts the same server from a token as from a credentials JSON (cmd/cloudflared/tunnel/subcommands.go, runCommand; subcommand_context.go, runWithCredentials). The token carries the tunnel's id, so the config in Git names no id and can land before the tunnel exists.
    • The Secret: tunnel-token in namespace cloudflared, key token, at namespaces/cloudflared/tunnel-token.yaml in platform-secrets. The operator creates it once (Operator actions).
    • A file, not an environment variable, as hopin reads its secrets.
    • Rejected: the credentials JSON with tunnel: <id> in the config. The id would have to be in Git before the slice lands, and the Secret would hold the same secret in a second form.
  3. The lines in a ConfigMap. ingress: hopin.hallucinai.de → http://traefik.traefik.svc.cluster.local:80, then service: http_status:404.

    • Why one line per host rather than a catch-all to Traefik: as on hal, a host reaches the internet only through a reviewed line in the tunnel, not through an Ingress alone. Both mismatches fail safe: a line without an Ingress gets Traefik's 404, and an Ingress without a line gets the tunnel's 404. So no check compares the two lists.
  4. The pod.

    • Deployment: cloudflared, one replica. The default rolling update starts the new pod before the old one stops.
    • Image: docker.io/cloudflare/cloudflared:<version>@sha256:<index digest>. Its repository joins the vendor list in this slice; the list's one owner is infrastructure/admission/image-vendors.yaml (since B1a, 0f4cdaf), read by the job images and later by the admission check (Only attested app images and digest-pinned listed vendor images start on the PC #50). The image's own user 65532 runs it, NO_AUTOUPDATE=true is set in the image, and its Dockerfile at 2026.10.7 sets CONTAINER_BUILD=1. That makes the metrics server bind 0.0.0.0 on one of 20241 to 20245 (metrics/metrics.go). --metrics 0.0.0.0:2000 fixes the port for the probe.
    • Probe: a readiness probe on /ready at 2000. It answers 200 only while the tunnel has at least one edge connection (metrics/readiness.go). There is no liveness probe: cloudflared reconnects on its own, and a restart would not create a connection.
    • Security: runAsNonRoot, read-only root, no privilege escalation, all capabilities dropped, RuntimeDefault seccomp, automountServiceAccountToken: false.
  5. Namespaces that receive Secrets from platform-secrets get their own layer, and Flux never prunes them.

    • The problem: a Secret in namespace cloudflared cannot be applied before that namespace exists. If the Namespace lived with cloudflared's manifests and tunnel depended on platform-secrets, platform-secrets would fail on the missing namespace. If instead platform-secrets depended on tunnel, it would wait for a pod that waits for the Secret: a deadlock.
    • The layer: a Kustomization namespaces (./infrastructure/namespaces) holds infrastructure/namespaces/namespaces.yaml with the Namespace cloudflared, which enforces Pod Security restricted as P3d's does. platform-secrets gains dependsOn: [namespaces], and tunnel depends on platform-secrets. hopin's namespace joins in hopin runs on the platform from its attested private image, routed by Traefik #46.
    • The rollback problem (the review's finding 1): a namespace's Secrets are declared in another repository. A revert here that removed a Namespace would let Flux delete it, and with it its Secrets. platform-secrets applies its whole source (path: ./, prune: true), so it would then fail on the missing namespace, and every layer that depends on it would wait.
    • Chosen: every Namespace in the layer carries kustomize.toolkit.fluxcd.io/prune: disabled. Flux skips such an object both when it prunes and when its Kustomization is deleted (kustomize-controller v1.10.1, the version the committed gotk-components.yaml pins: deleteObjects excludes that annotation, called from prune and finalize). A revert of a workload, or of this whole slice, then leaves the Namespace, and platform-secrets keeps applying its Secrets into it and stays Ready. P3: the cluster skeleton reconciles in k3d on every pull request #11's D2 marks PVCs the same way, for the same reason: losing the object costs more than keeping it.
    • The guard: the job fails, naming the object, when a Namespace the Kustomization namespaces applies lacks the annotation (read from the cluster). Every later namespace keeps the rule without anyone having to remember it.
    • The cost, and the exit: a Namespace removed from Git stays in the cluster with its Secrets. It leaves for good only after its Secrets have left platform-secrets, then by a deletion on the PC. That exit is a named gap until the first app leaves the platform; no caller needs it today.
    • Weighed: a coordinated removal order across both repositories (first the Secrets leave platform-secrets, then the Namespace leaves here). It rests on every agent and the operator remembering an order that no check can see: this job serves its own dummy Secrets, and platform-secrets' gate does not read this tree. A revert under pressure is when the order is forgotten. It stays the order of the exit above, not the protection.
    • Proven by hopin runs on the platform from its attested private image, routed by Traefik #46's cell H15, which reverts hopin's workload while its Secrets stay, and its control without the annotation.
    • Rejected: Namespaces in the root's ./clusters/pc. The order would then rest on Flux applying Namespaces first (fluxcd/pkg ssa/manager_apply.go, ApplyAllStaged), and clusters/pc/ would hold more than layer files.
    • Rejected: a Namespace inside platform-secrets. Two owners of one object, and P4b1's check accepts only Secrets.
    • Rejected: an apply that is retried until the namespace exists. Every fresh bootstrap would go red first.
  6. One Kustomization per directory under infrastructure/, in clusters/pc/infrastructure.yaml, as Flux's documented layout splits infrastructure. The file gains namespaces and tunnel (./infrastructure/cloudflared, dependsOn: [platform-secrets]). Each has wait: true, a timeout and prune: true. P3d's revision names its own traefik the same way.

    • tunnel does not depend on traefik: cloudflared is Ready on its edge connection, and a request finds Traefik when it arrives.
  7. CI: in k3d the tunnel's manifests apply, but cloudflared runs no pod.

    • The patch: the job root's spec.patches (D4 of P3: the cluster skeleton reconciles in k3d on every pull request #11) gain one patch on the Kustomization tunnel. It sets that Kustomization's own spec.patches to scale the Deployment cloudflared to 0. The job first checks that the committed tunnel carries no spec.patches, as P4a's step does for the root.
    • The proof: the job builds a throwaway Pod in namespace cloudflared from the Deployment's own pod template, read from the cluster. It keeps the image, the security context and the config volume; it drops the token volume and replaces the arguments with tunnel --config <file> ingress validate, then ingress rule https://hopin.hallucinai.de and ingress rule for a host the config does not name.
    • What that proves: the template passes restricted admission, the committed config parses, and hopin's host goes to the rule a second client Pod reaches as Traefik's web (P3d's 404), while the unknown host gets http_status:404. The client Pods run P3d's client image, docker.io/curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93, with a numeric user of their own, since namespace cloudflared enforces restricted.
    • Weighed:
      • a dummy token in k3d: every run would call Cloudflare's edge with a forged credential, and the pod would crash-loop or never become Ready, with no stable observable;
      • a real test tunnel with a CI secret: a secret for a test, a Cloudflare account coupling and red runs from forks, and still not the PC's tunnel;
      • the Kustomization deleted in k3d (the self-Kustomization's treatment): nothing of the tunnel would be applied or checked;
      • cloudflared as a mise tool run on the runner: a second install path, the config read out of a ConfigMap by hand, and no admission check.
    • What k3d does not prove: the edge connection, the token mount, /ready turning 200, and whether a token run routes by the committed lines. The PC proves them in host cells beside hal: C10 and C12 (decision 12).
  8. Traefik admits only the tunnel (P3d decision 7, carried out here):

    • A NetworkPolicy in namespace traefik, podSelector: {}, admits ingress only to the web container port and only from namespace cloudflared, matched on kubernetes.io/metadata.name, which the API server sets and no one can forge on another namespace. Traffic from the node, and with it the kubelet's probes, is always allowed (Kubernetes "Network Policies"). Everything else is denied, Traefik's own traefik and metrics ports included; P5 opens metrics when it needs them.
    • A second NetworkPolicy in namespace cloudflared denies all ingress, so no pod reads cloudflared's metrics, /ready or diagnostics.
    • ports.web.forwardedHeaders.trustedIPs: [10.42.0.0/16], k3s's default cluster CIDR. P3b's host/k3s/config.yaml (landed in fa2f78e) sets no cluster-cidr. hopin runs on the platform from its attested private image, routed by Traefik #46 checks the same relationship against hopin's setting.
    • What the trust admits stands in P3d decision 7. The remaining risk is a compromised pod in namespace cloudflared, or a NetworkPolicy that k3s's kube-router controller does not enforce. Cell C8 drives the controller in k3d, and C10 on the PC.
  9. hal's entrypoint hardening travels with the internet. hal's traefik.yml sets on its tunnel entrypoint:

    • aliasHeadersStrategy: delete;
    • sanitizePath: true;
    • every encodedCharacters.allow… set to false.

    Its comment reads: "Traefik 3.7 allows encoded path characters by default; reject them so no app sees a traversal". P3d does not carry these settings. This slice sets them on web, because this is where the internet first reaches Traefik. The chart's value path for them is not verified: ports.web.http… or additionalArguments.

    • Rejected: leaving them out. Every app on the platform would lose a protection it has on hal today.
  10. The bump owner of the cloudflared pin is the vendor-pin bump item Every vendor pin in platform's manifests has a maintained bump path #47; until it lands the pin moves by hand (Q8).

  11. Cut into two pull requests, the head's choice (07.11.2026, S4a first).

    • S4a, cloudflared runs: decisions 2 to 7, about 160 lines in six files.
    • S4b, Traefik admits only the tunnel: decisions 8 and 9, about 60 lines in four files, plus Traefik's regenerated render.

    Between them, nothing reaches Traefik through the tunnel: the one DNS record pointing at it is C12's tunnel-check.hallucinai.de (hal at 641e072e, 07.11.2026, this item's journal), which S4a's lines answer with the tunnel's 404, and hopin's record stays on hal's tunnel until O10. Weighed: one pull request (about 220 lines in nine files), and S4b first (trustedIPs with no sender, which P3d rejects).

  12. Routing through the real tunnel is proven on the PC before the live go (the review's finding 5). k3d checks the committed lines offline (decision 7), and C10 shows an edge connection, but neither shows that a token run routes by those lines. That rests on source and on a cloudflared maintainer's statement that the edge pushes no configuration to a tunnel created with the CLI (cloudflared#1029, 05.09.2023); it is not verified offline.

    • The cell C12, a host step of the head on the PC beside hal, after S4b, hopin runs on the platform from its attested private image, routed by Traefik #46's S5b and its H13 have held, before O10:
      1. a pull request the operator merges adds one line ahead of hopin's: hostname tunnel-check.hallucinai.de, path: ^/health$, Traefik's Service, and originRequest.httpHostHeader: hopin.hallucinai.de (both keys exist in cloudflared 2026.10.7: ingress/rule.go, Matches; config/configuration.go);
      2. a DNS record for that host points at the tunnel platform: hal#1103 (PR #1104, 641e072e) put the host into hal's catalog, and the operator's hal update and hal apply created its CNAME on 07.11.2026 (this item's journal); until a connector serves it, Cloudflare answers 530;
      3. from the PC, GET https://tunnel-check.hallucinai.de/health gets hopin's 200, and GET https://tunnel-check.hallucinai.de/ gets the tunnel's 404;
      4. the revert of that line, merged by the operator, and the record deleted again, both before O10.
    • What it proves: under R2's option A the check line exists only in the committed config, so the 200 shows that the edge reached this tunnel, the token run applied the local lines, cloudflared reached Traefik's Service, and Traefik's route sent hopin's host to hopin. The rewrite is needed because hopin's Ingress is bound to hopin's host. The 404 on / shows the path bound holds: the check host exposes nothing but /health, which needs no session, so the zone's block on hopin's back-channel path is not needed there.
    • From proof to live: until O10 no visitor's host points at the platform; C12's host serves only /health, and only between its two pull requests. The live go (O10, the operator's word) is the one step from proof to live.
    • Weighed:
      • the live go's first request as the check: a fault would reach visitors before the rollback;
      • the check line to Traefik without the host rewrite: Traefik's 404 and the tunnel's 404 would differ only in their bodies, a weaker observable;
      • a check host that stays: a public name with no caller after the cell;
      • hopin's own host: its record is the live switch itself.

What is known

  • cloudflared, read 06.11.2026:
    • The latest release is 2026.10.7 (GitHub, 30.10.2026). Docker Hub's API gives cloudflare/cloudflared:2026.10.7 the index digest sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba, the same as latest, and 2026.10.7-amd64 the digest sha256:32f933aa6fe1f69c5885b93f3c92cc3f60cca8f0530868bc3bc2cc446114a385. The builder re-reads both at build time.
    • The image is distroless base-debian13:nonroot with USER 65532:65532 (Dockerfile at 2026.10.7).
    • cloudflared tunnel token <name> prints the token of a tunnel created with cloudflared 2022.3.0 or later, and tunnel ingress validate and tunnel ingress rule <url> exist (ingress_subcommands.go).
    • An ingress rule matches on hostname and an optional path regular expression, and originRequest.httpHostHeader sets the Host sent to the origin (2026.10.7 source, read 06.11.2026).
    • A tunnel managed by Cloudflare cannot be moved back to local management (a cloudflared maintainer, cloudflared#1029, 11.09.2023). Deleting a locally managed tunnel keeps its configuration, while deleting a remotely managed one deletes its configuration too (Cloudflare's Tunnels FAQ, read 06.11.2026).
    • Whether the image is signed was not read.
  • The fixtures, Docker Hub's API, 06.11.2026 (the builder re-reads both; named with repository and digest so the admission item can list them):
    • the client Pods' image, P3d's: docker.io/curlimages/curl:8.23.0@sha256:3fb7e13e5d179a845c97d78204f3f1064d1907037f26a3ebf91710571151fd93, the same digest as latest;
    • C9's echo server: docker.io/traefik/whoami:v1.12.0@sha256:c4717a8d1f0134a7444e24f881160e033991f23027c6c5a9a3f8fd22e70d1d44, the same digest as latest.
  • hal, read at origin/main e15da11:
    • hal's tunnel is ruled to be managed by Cloudflare (D6). Whether it has switched (config_src) is not known here; that is O10's question to the operator.
    • Cloudflare's zone-wide deny_public rule blocks /auth/backchannel-logout on hopin's host before any tunnel (docs/external.md), so it covers this tunnel too, as long as hal's catalog declares it (R4).
  • k3s embeds kube-router's network policy controller, on unless --disable-network-policy (k3s docs, "Network Policy Controller"); P3b's host/k3s/config.yaml at fa2f78e does not disable it.
  • Invariants kept.
    • No secret in the repository, a log or a test. In k3d no token exists, and the job's patch is the only difference from the PC.
    • cluster keeps contents: read and persist-credentials: false.
    • No new mandatory job.
    • Every new Kustomization sets wait: true and a timeout.
    • hal's 8081, 8090, tunnel and unit are untouched.
  • Rollback. Revert the commit. The root's prune deletes the Kustomizations, and with them the Deployment, the ConfigMap and the NetworkPolicies. The Namespace cloudflared stays (decision 5), and platform-secrets keeps applying the token into it and stays Ready. On the PC, the live DNS record is the live switch, not this slice (O10 at the live go).
  • Deletes. Nothing in the tree. hal's tunnel leaves with hal (P11).
  • Size.
    • S4a, about 160 lines in six files: infrastructure/cloudflared/cloudflared.yaml about 70, infrastructure/namespaces/namespaces.yaml about 12, clusters/pc/infrastructure.yaml about 30, clusters/pc/secrets.yaml 2, the cluster job about 42, infrastructure/admission/image-vendors.yaml 2.
    • S4b, about 60 lines in four files, plus infrastructure/traefik/rendered.yaml regenerated by mise run charts:render: infrastructure/traefik/values.yaml about 10, infrastructure/traefik/network-policy.yaml about 20, infrastructure/traefik/kustomization.yaml 1, the cluster job about 30.
    • C12's two pull requests, about 7 lines in infrastructure/cloudflared/cloudflared.yaml, added and reverted.
    • AGENTS.md changes in S4a and S4b.

Operator actions

  • The card below (R2) is ruled: A, the operator, 06.11.2026, in chat.
  • O4, once, after the card, P4b (A and B) and The PC creates its age key once and reaches platform-secrets with its own read-only deploy key #61's SB6: the tunnel platform exists since 06.11.2026 (this item's journal), so O4 has no create step and no hal#1098 precondition.
    • Store the token: pipe cloudflared tunnel token platform into mise run secrets:put cloudflared tunnel-token token in the worktree of the platform-secrets value item, opened at P4b step 8 (no value reaches a terminal), which writes namespaces/cloudflared/tunnel-token.yaml; the head merges that item's pull request by hand once the namespace cloudflared is Ready on the PC.
    • The credentials JSON is gone: the operator ran cloudflared tunnel create --cred-file <mktemp path> platform and deleted that file, so it never touched his cloudflared directory, and the token in platform-secrets will be the secret's one living copy (this item's journal, 06.11.2026).
  • O11, for C12, before O10: merge the check line's pull request; after the cell, have hal's head remove the record for tunnel-check.hallucinai.de through hal's catalog (hal#1100 step 2), and merge the revert. The record already points at the tunnel platform (hal#1103 and the operator's hal apply, 07.11.2026).
  • R4 is settled (platform's OpenTofu takes over the zone hallucinai.de from hal at the cutover #48, 06.11.2026): hal's OpenTofu stays the zone's one owner until the cutover. hopin's record moves to this tunnel through hal's catalog line (hal#1098) and the operator's hal apply (O10), and the zone's block on hopin's /auth/backchannel-logout stays in hal's ruleset.
  • No DNS writer here: every ingress line lives in the tunnel's config (decision 3), never as a dashboard public-hostname route, because that route also writes or overwrites the host's DNS record, which hal owns (hal's head, 06.11.2026). O11's check record was set through hal's catalog (hal#1103), not with cloudflared tunnel route dns.

Files

  • infrastructure/cloudflared/cloudflared.yaml (new): the ConfigMap, the Deployment, and the NetworkPolicy denying all ingress in namespace cloudflared. C12 adds its check line and reverts it.
  • infrastructure/namespaces/namespaces.yaml (new): the Namespace cloudflared with pod-security.kubernetes.io/enforce: restricted and kustomize.toolkit.fluxcd.io/prune: disabled.
  • clusters/pc/infrastructure.yaml: the Kustomizations namespaces and tunnel.
  • clusters/pc/secrets.yaml: dependsOn: [namespaces] on platform-secrets.
  • infrastructure/traefik/values.yaml (S4b): trustedIPs and hal's hardening on web, rendered into infrastructure/traefik/rendered.yaml by mise run charts:render.
  • infrastructure/traefik/network-policy.yaml (new, S4b): Traefik's NetworkPolicy, which the chart does not render, listed in infrastructure/traefik/kustomization.yaml.
  • .github/workflows/ci.yml, job cluster:
    • S4a: the root patch for tunnel, the check that tunnel carries no committed patches, the annotation guard on the layer's Namespaces, and the config check;
    • S4b: the client Pods move to namespace cloudflared, plus the NetworkPolicy cells and the encoded-slash cell.
  • infrastructure/admission/image-vendors.yaml, the vendor list's one owner: docker.io/cloudflare/cloudflared, with its comment naming this manifest and Docker Hub's cloudflare/cloudflared page.
  • AGENTS.md:
    • the Layout lines for infrastructure/cloudflared/ and infrastructure/namespaces/, and that a namespace receiving a Secret from platform-secrets is declared there and nowhere else, never pruned, and leaves only after its Secrets have left platform-secrets;
    • the cluster line on the tunnel's treatment in k3d.

Verification: CI on the pull requests and the landing pushes; quality hygiene . printing hygiene: clean; the driven cells; the PC cells C10 after O4 and C12 before O10; the independent plan review of decisions 1, 5, 7, 8 and 12, held before the build (Codex, REVISE, then delta APPROVE, 06.11.2026).

Done when

  • S1 Flux reconciles the Kustomizations namespaces, platform-secrets and tunnel in that order, each with wait: true and a timeout. A namespaces that is not Ready holds platform-secrets back with Flux's dependency reason, and cluster turns red. Every Namespace namespaces applies carries kustomize.toolkit.fluxcd.io/prune: disabled, and the job fails naming one without it.
  • S2 tunnel applies the ConfigMap, the Deployment cloudflared (one replica on the PC, scaled to 0 only by the job's root patch) and the NetworkPolicy of namespace cloudflared. The image is pinned by digest and its repository is on the vendor list.
  • S3 In the job, a Pod built from the Deployment's template is admitted under restricted. In it, ingress validate passes on the committed config, ingress rule sends hopin's host to a URL where a client Pod in namespace cloudflared reaches Traefik's web port, and an unknown host gets http_status:404. A config without the final 404 rule turns cluster red.
  • S4 The run line reads the token from the file of the Secret tunnel-token, readiness is /ready on port 2000, and no Cloudflare API token, tunnel id or credential is in the tree.
  • S5 (S4b) Only pods in namespace cloudflared reach Traefik's web port; a client Pod in another namespace gets no answer. A request with an encoded slash gets 400. Traefik keeps the X-Forwarded-For that cloudflared's namespace sends.
  • S6 AGENTS.md describes the tunnel, the namespaces layer and its prune rule, and the k3d treatment, and quality hygiene . prints hygiene: clean.
  • S7 On the PC, once P3c1 (The PC runs k3s and Flux beside running hal, its ufw keeping the k3s ports off the LAN #41), P4b (A and B), The PC creates its age key once and reaches platform-secrets with its own read-only deploy key #61's SB6 and O4 are done, tunnel is Ready at main's revision and the cloudflared pod is Ready, which /ready allows only with an edge connection. Once S4b has landed, a client Pod in another namespace gets no answer from Traefik's web there too.
  • S8 C12 on the PC, before O10: through the real tunnel, /health on the check host gets hopin's 200 and / gets the tunnel's 404; afterwards the check line and its DNS record are gone.

Scenario matrix

Material risk (the internet-facing edge and its trust): the first review drives the matrix, then the cross-provider gate. Driven cells run on scratch pull requests of the lane, closed after with their branches deleted. Host cells run on the PC beside running hal, under #1's invariant 1.

Cell What is driven Expected observable Proven by
C1 S1, pull request namespaces, platform-secrets and tunnel Ready at the candidate revision; the Namespace cloudflared carries the prune annotation; the Deployment cloudflared at 0 replicas in the job CI
C2 S3, pull request the template Pod is admitted; ingress validate exits 0; hopin's rule reaches Traefik's 404 from namespace cloudflared; the unknown host gets http_status:404 CI
C3 S1, scratch, two parts: (a) an invalid label value on the Namespace cloudflared; (b) its prune annotation removed (a) namespaces not Ready; platform-secrets reports its dependency not ready; cluster red. (b) the guard turns cluster red naming the Namespace driven
C4 S3, scratch: the final http_status:404 rule removed ingress validate fails; cluster red naming the check driven
C5 S2 and S3, scratch: runAsUser: 0 in the template the template Pod is refused by Pod Security; cluster red driven
C6 S4, the diff the run line, the token mount, the /ready probe, the security context and the image digest; no tunnel id, credential or API token; the vendor entry and its comment diff review
C7 S2, the job's patch the committed tunnel has no spec.patches, and the patch touches only replicas CI and diff review
C8 S5, pull request (S4b) a client Pod in namespace cloudflared gets 404 from Traefik's web; one in another namespace gets no answer within its timeout; one with /a%2Fb gets 400 CI
C9 S5, scratch (S4b): docker.io/traefik/whoami:v1.12.0@sha256:c4717a8d… behind a scratch Ingress; a request from namespace cloudflared with X-Forwarded-For: 192.0.2.1; then the same with trustedIPs removed first, whoami's echo shows 192.0.2.1 ahead of the client's address; then only the client Pod's address driven
C10 S7, the PC after O4 tunnel and the cloudflared pod Ready on the PC; once S4b has landed, a throwaway client Pod in another namespace gets no answer from Traefik's web; no visitor's DNS record points at the tunnel (only C12's tunnel-check.hallucinai.de does, answered by the tunnel's 404) the head, a host step
C11 S1 to S6, push to main cluster green with C1, C2 and C8's observables CI, read at landing
C12 S8, the PC after S4b, #46's S5b and H13, before O10: the check line and its record (decision 12) GET https://tunnel-check.hallucinai.de/health gets 200 with hopin's health body; / gets the tunnel's 404; after the revert and the record's deletion, the check host no longer reaches the platform the head, a host step

Driven cells: C3, C4, C5, C9 (four), plus the host steps C10 and C12.

Unknowns

Unknown Settled in
ingress validate and ingress rule run offline and without the token file C2
Whether a token run routes by the committed lines (read from source and cloudflared#1029; not verified offline) C12
Whether k3s's network policy controller enforces in k3d, and on the PC C8 in k3d, C10 on the PC
The chart's value paths for hal's hardening and trustedIPs (the container port is named web in rendered.yaml) S4b's build, against the chart P3d pins
QUIC to the edge (UDP 7844) from a pod under Docker's FORWARD policy on the PC; cloudflared falls back to HTTP/2 C10, with P3c1's cells
Whether cloudflared writes anything under a read-only root C10
Whether the zone's certificate covers tunnel-check.hallucinai.de, a first-level name like hopin's C12
Which numeric user the client image runs as (the Pod sets its own) the build
Whether the cloudflared image carries a signature not read; a named gap for the vendor-pin item
hal's tunnel mode and whether hopin's record is its own or a wildcard O10

Neighbours

version = 1
now = "S4a landed as 1d7457c (#81) and S4b as 5607fe5 (#103, PR #108): cloudflared runs on the PC, and Traefik's web admits only namespace cloudflared and trusts its forwarded headers. C8, C9 and C11 held (journal). C12 landed as #128 (6466f1b, #130) and #132 then dropped the check line."
next = "C10's second part on the PC and the real-tunnel request to hopin (overnightworks/hal#1151 is closed), the operator's probe."
done_when = "cloudflared runs in k3s on the platform's own tunnel from a token in platform-secrets, with its lines in Git sending hopin.hallucinai.de to Traefik and every other host to 404; the namespaces layer lets platform-secrets apply before the tunnel and never prunes its Namespaces, which the job guards; in k3d the tunnel's manifests apply and its config is checked without a pod; only namespace cloudflared reaches Traefik's web port, whose forwarded headers it trusts and which refuses encoded path characters; the tunnel is Ready on the PC; a request through the real tunnel reached hopin through Traefik's route before the live go; and the matrix C1 to C12 is reported."

scope = [".github/workflows/ci.yml", "AGENTS.md", "clusters/pc/infrastructure.yaml", "clusters/pc/secrets.yaml", "infrastructure/admission/image-vendors.yaml", "infrastructure/cloudflared/cloudflared.yaml", "infrastructure/namespaces/namespaces.yaml", "infrastructure/traefik/kustomization.yaml", "infrastructure/traefik/network-policy.yaml", "infrastructure/traefik/rendered.yaml", "infrastructure/traefik/values.yaml"]

size = "L"

whole = "S4a does not split further: the namespaces layer, the tunnel's manifests and their k3d check have no caller apart, and the Secret cannot be applied without the layer; S4b is the cut that serves the trust boundary, and C12's two pull requests are a host cell, not a slice."

[[expectation]]
text = "Die neue Plattform bekommt einen eigenen Tunnel ins Internet; seine Weiterleitungen stehen im Git und ändern sich per Pull Request."
asked_on = 2026-11-06
ruling = "A"
ruled_on = 2026-11-06
card_hash = "sha256:7049ef598eb8a008b2f89b36fe9b0c90ce03f932185a9c90f6fabbff9b1f4d5f"
via = "chat"
question = "Sollen die Weiterleitungen der neuen Plattform ins Internet wie alles andere per Pull Request geändert und zurückgedreht werden?"
problem = "Besucher erreichen die neue Plattform ohne eigenen Tunnel nicht; der heutige Tunnel gehört dem alten Stand."
decision_kind = "technical"
area = "Plattform"
topic = "Eigener Tunnel"
recommended = "A"
reason = "Die neue Plattform wendet alles aus dem Git selbst an und braucht dafür keinen Cloudflare-Schlüssel. Laut einem Entwickler von Cloudflares Tunnel-Programm (11.09.2023) lässt sich ein von Cloudflare verwalteter Tunnel nicht zurück auf eigene Verwaltung stellen; A hält also beide Wege offen. Wird ein Tunnel gelöscht, bleiben Weiterleitungen im Git erhalten, bei Cloudflare verwaltete gehen mit ihm verloren (Cloudflares Hilfeseite zu Tunneln). Für den alten Stand hast du am 18.10.2026 „Cloudflare steuert“ entschieden; das gilt nur dort."

[[expectation.option]]
name = "Weiterleitungen im Git"
effect = "Ein zweiter Tunnel gehört der neuen Plattform. Seine Weiterleitungen stehen im Git, und die Plattform wendet sie selbst an. Der alte Tunnel bleibt unberührt und geht mit dem alten Stand."
notice = "Eine neue Adresse kommt als Pull Request, den du mergst. Einmal legst du den Tunnel mit deiner Cloudflare-Anmeldung an und legst seinen Schlüssel verschlüsselt ab."
cost = "Anders als beim alten Stand stehen die Weiterleitungen nicht bei Cloudflare. Die Adresseinträge setzt du weiter von Hand, bis eine eigene Aufgabe sie übernimmt."
undo = "reversible"
undo_how = "Später auf Verwaltung durch Cloudflare umstellen; nur dieser Wechsel geht nicht zurück."
contract = "S4 creates the platform's own tunnel `platform` as a locally managed tunnel (O4: `cloudflared tunnel create platform` under the operator's existing login, its token encrypted into `platform-secrets` as `namespaces/cloudflared/tunnel-token.yaml`, the credentials JSON deleted). cloudflared runs in k3s as the Deployment `cloudflared` in namespace `cloudflared` with `--token-file`; its ingress is the ConfigMap in `infrastructure/cloudflared/cloudflared.yaml` (`hopin.hallucinai.de` to Traefik's Service, then `http_status:404`), applied by Flux, changed only by a pull request the operator merges (Q8). No Cloudflare API token enters the cluster, CI or this repository. DNS records stay set by the operator by hand (O10 at each live go, O11 for C12) until an item declares them. hal's tunnel, its cloudflared unit, 127.0.0.1:8081 and 127.0.0.1:8090 stay untouched and leave with hal (P11)."

[[expectation.option]]
name = "Cloudflare verwaltet die Weiterleitungen"
effect = "Ein zweiter Tunnel gehört der neuen Plattform. Seine Weiterleitungen stehen bei Cloudflare wie beim alten Stand, und du änderst sie im Cloudflare-Dashboard. Der alte Tunnel bleibt unberührt."
notice = "Eine neue Adresse richtest du im Cloudflare-Dashboard ein; im Git steht nur, dass der Tunnel läuft."
cost = "Ein zweiter Ort neben dem Git hält die Weiterleitungen, und zurückdrehen geht nur von Hand im Dashboard. Sollen sie später aus dem Git geschrieben werden, braucht das einen Schlüssel mit Schreibrecht bei Cloudflare und eine eigene Entscheidung; dieses Ja erlaubt ihn nicht."
undo = "reversible"
undo_how = "Nur mit einem neuen Tunnel und einem Adresswechsel."
contract = "S4 creates the tunnel `platform` in the Cloudflare dashboard (remotely managed, as hal's D6). cloudflared runs in k3s as in option A with `--token-file`, without the ConfigMap or its k3d check; the operator sets the lines in the dashboard, C12's check line included. This yes chooses dashboard management only: a writer of the lines from code (an OpenTofu root of platform with a Cloudflare token holding Tunnel Edit) is not authorised by it and needs its own item and card under #1. The token Secret, the namespaces layer, Traefik's NetworkPolicy and trust, and hal's tunnel are as in option A."

Activity

  1. added theissue type on Nov 6, 2026
  2. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    Journal, 06.11.2026: the edge order with hal (hal's head, from hal#1098's plan)

    1. hal's S1 (hal#1099) lands: hal finds its own tunnel by id, no longer as the only one in the account. hal's head tells platform.
    2. O4: the operator creates the tunnel platform.
    3. hal's S2 lands: the tunnel host key, plus one temporary host line tunnel-check with tunnel: platform and the ready path /health. hal writes only its CNAME, and the tunnel config here serves it. The operator runs hal apply, then C12 runs. hal's head tells platform.
    4. hopin's line replaces the tunnel-check line. The operator's hal apply flips hopin (O10) and removes the check record in the same run.
  3. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    Journal, 06.11.2026: O4's precondition holds in code. hal's head reports that hal#1099 landed at 47275516 (hal PR #1101): from that commit on, hal finds its own tunnel by the id in /etc/cloudflared/config.yml, in tf and in hal check hosts, so a second tunnel no longer breaks it. It is not live yet: the operator's ~/srv/hal is at c2f8ccf. Until his hal update, a second tunnel makes hal apply and hal check hosts fail loud before anything is applied; no timer runs either, and hal's cloudflared and DNS records keep serving. The next hal apply this item needs (O11, O10) comes after hal#1098 and so after that update. O4's tunnel creation may therefore run now; storing its token waits for platform-secrets and its recipient (P4b2, P4c). The tunnel is named exactly platform, because hal#1098 looks it up by that name.

  4. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    Journal, 06.11.2026: O4's first half is done. The operator ran cloudflared tunnel create --cred-file <mktemp path> platform and then deleted that file. The tunnel platform exists with id 9f3f03a7-0f38-4bcc-8f96-8f4235f10629 and no connections yet. The credentials JSON never touched his cloudflared directory, and it is gone.

    Still open:

    • O4's token step (cloudflared tunnel token platform piped into sops) waits for platform-secrets, which now exists, to get its recipient (P4b2, P4c).
    • His cloudflared CLI is 2026.10.3; it recommends 2026.10.7. hal owns that host binary, so platform does not change it.
  5. FlexOr2 commented on Nov 6, 2026

    @FlexOr2
    MemberAuthor

    Journal, 07.11.2026: hal's head reports hal#1103 (PR #1104) landed at 641e072e. The line {name: tunnel-check, exposure: public, tunnel: platform, ready: /health, deny_public: [/auth/backchannel-logout]} sits under hopin's entry in hal's catalog. It goes live with the operator's hal update, hal check hosts and hal apply, whose plan creates the CNAME tunnel-check.hallucinai.de to the tunnel platform, extends the back-channel deny rule to that name, and leaves hal's tunnel config and Access unchanged. Until the platform's cloudflared connects with an ingress for that host, the name answers with Cloudflare's tunnel error, and hopin's own name is unaffected. The ingress line for the check host and its removal after C12 are this item's (decision 12).

  6. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Fact from hal's head, 07.11.2026: the operator's hal update and hal apply put hal at 641e072e. Since then tunnel-check.hallucinai.de resolves and is proxied by Cloudflare, routed to the tunnel platform.

    hal's head's read-only public check:

    C12 is this item's to drive. Once it holds, hal reverts its temporary host (hal#1100 step 2) and moves hopin to tunnel: platform (step 3).

  7. changed the issue type fromtoon Nov 7, 2026
  8. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Journal (head), follow-ups from S4a's lane (#81, tip 6ae9d4e), routed here and not built now:

    • The cluster job's tunnel probe copies hopin's host (ROUTED_HOST) instead of reading every hostname rule from the applied ConfigMap. Each later line is checked only once someone extends that list. The pull: the second routed host (the next app's line) reads the hosts from the ConfigMap.
    • A wildcard line such as *.hallucinai.de would give the probe's client Pod a name that is not a valid DNS name. The job would then fail loud at kubectl create and never pass wrongly. The pull: a ruled wildcard line.
  9. 8 remaining items

  10. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Two more from S5b's delta review (#117, PR #118 at 6510a9a), for the lane that moves the client-Pod runner into scripts/:

    • .github/workflows/ci.yml: the Traefik step's ROUTED_APP_MANIFEST repeats one entry of the job-level APP_MANIFESTS. The routed app can be found as the manifest in APP_MANIFESTS that holds an Ingress, so a moved hopin manifest changes in one place.
    • AGENTS.md, the infrastructure/traefik/ line: in "sets no cluster-cidr (cluster checks it against the nodes' pod CIDRs)", "it" still reads as cluster-cidr. Put the check on the trusted CIDR itself: "web trusts the forwarded headers of 10.42.0.0/16, which cluster checks against the nodes' pod CIDRs: the pod CIDR k3s uses because host/k3s/config.yaml sets no cluster-cidr".
  11. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Two follow-ups from C12's lane (#128, PR #130 at 7a7bb9a), for the lane that moves the client-Pod runner into scripts/. Neither blocks.

    • Off-path paths. The tunnel step judges only / as a path outside a path-restricted line. Offline, the explorative test proved that /health/x, /healthz, /health/ and /HEALTH match the final http_status:404, but CI does not judge them.
    • The routed-request check. It accepts any status from 100 to 599 except 404 as "reaches its route", so /health is not pinned to 200. This behaviour is unchanged by the lane. On the PC, C12's D2 reads the 200.
  12. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    C12 held on the PC on 07.11.2026 after #130 (6466f1b): /health on tunnel-check.hallucinai.de got hopin's 200 and / the tunnel's 404 (evidence: #128's journal). Remaining before O10: the revert (#128 D3, the operator merges), then hal's head removes the record (O11); hal's head has been asked to prepare that and the O10 catalog line, and to answer the two O10 questions (tunnel mode, own record or wildcard).

  13. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    O10's two questions, answered by hal's head (07.11.2026, from hal's repository and board, no Cloudflare call):

    • hal's tunnel is remotely managed: config_src is cloudflare with remote_config: true since 20.10.2026 (hal#168, the operator's read after the switch). hal's OpenTofu owns its whole ingress (external/tunnel.tf, cloudflare_zero_trust_tunnel_cloudflared_config.this); a catalog host with tunnel: platform gets no line in hal's ingress, so its line is this tunnel's.
    • hopin has its own record, no wildcard: external/cloudflare.tf writes one proxied CNAME per catalog host, hopin.hallucinai.de → <tunnel id>.cfargotunnel.com; tunnel: platform swaps only the target id, resolved by name (hal#1098). fleet.yaml has no * host and tunnel.tf refuses one.
    • R4 holds by construction: the deny_public rule for /auth/backchannel-logout hangs on the host line, not the tunnel, so it stays in hal's ruleset when the target changes.

    hal's head opens two children of hal#1100: the record removal for tunnel-check.hallucinai.de (O11, landed only after the operator merged #128's revert) and hopin's catalog line to the tunnel platform (O10, unlanded until the operator's word).

  14. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    hal's head opened the two hal lanes: hal#1150 (O11, removes the tunnel-check catalog line; lands once #128's revert PR #132 is merged, then the operator's hal update and hal apply delete the record, which closes C12's last cell) and hal#1151 (O10, tunnel: platform on hopin's host, blocked by hal#1150, landed only on the operator's word; rollback: the line flipped back, hal update, hal apply, hal start hopin). #45 now records hal#1150 as its blocker.

  15. FlexOr2 commented on Nov 7, 2026

    @FlexOr2
    MemberAuthor

    Live-go order agreed with hal's head (07.11.2026): (1) step 0, the head reports https://tunnel-check.hallucinai.de/health answering hopin's 200 through the tunnel platform; (2) on the operator's word, #132 (#128's revert) is merged, Flux rolls cloudflared, and the check host gets the tunnel's 404; (3) hal's head lands hal#1153, which now carries both catalog changes (hal#1150 folded into hal#1151); (4) the operator runs hal stop hopin, then one hal update and hal apply, whose one plan deletes the check record, keeps hopin in the deny rule and moves hopin's record. Neither catalog change may reach hal's main before the live go. #45 now waits for hal#1151.

  16. FlexOr2 commented on Nov 8, 2026

    @FlexOr2
    MemberAuthor

    The matrix, reported on 08.11.2026. Evidence is main's ci run 42516492663 at 69a62e3, the present tree and this journal.

    Cell Result Evidence
    C1 held cluster step on namespaces and tunnel; platform-secrets Ready only through the reconcile's wait on all Kustomizations
    C2 held cluster step on cloudflared's ingress; hopin's Ingress now answers 401 where the row says Traefik's 404
    C3 not driven No drive recorded. Part (b) has a standing guard in tests/workloads/layers_depend_on_admission.sh; part (a) has none
    C4 not driven No drive recorded. The cell's refusal exists in tests/cloudflared/ingress_from_pods_built_from_its_template.sh and was never shown red
    C5 not driven No drive recorded. The Namespace enforces restricted and the template Pod runs through the same step
    C6 held Read in the present infrastructure/cloudflared/cloudflared.yaml and the vendor list; no tunnel id or credential
    C7 held tests/flux/root_and_tunnel_carry_no_spec_patches.sh green; the overlay's patch touches only replicas
    C8, C9, C11 held Journal, 07.11.2026 13:29Z
    C10 held First part 07.11.2026 12:36Z. Second part 07.11.2026 13:53Z and again 08.11.2026: a Pod in a throwaway namespace, which resolved DNS, got no answer from Traefik. Its DNS clause was overtaken by the live go
    C12 held 07.11.2026 17:59Z, before the live go: /health on the check host got hopin's 200 through the tunnel and / the tunnel's 404

    After the live go, on 08.11.2026: one GET https://hopin.hallucinai.de/health from the PC answered 200 and cloudflared_tunnel_total_requests of the platform's tunnel went from 31 to 32.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions