Skip to content

The cluster job's two in-cluster sources name addresses, so a lost DNS record cannot fail the job #315

Closed

Description

@FlexOr2

Parent: #1

The cluster job's two in-cluster sources name addresses, so a lost DNS record cannot fail the job

Caller: an incident. Three cluster cells failed alike within the last 23 finished runs of ci.yml: run 42628861287 (e375b80, a red trunk), run 42630017625 (b0f3e2a) and run 42633968554 (29cfee6, #310). Each costs a re-run of the job, and a red trunk stops every lane behind it.

What happens

  • The step "Assert that Flux reconciled the candidate commit and decrypted the dummy" runs out its 3-minute wait on gitrepository/flux-system.
  • source-controller reports, on every try for those 3 minutes, lookup host.k3d.internal on 10.43.0.10:53: no such host for the GitRepository and lookup platform-registry on 10.43.0.10:53: no such host for the OCIRepository platform-secrets. (Read from the job logs of the first two runs; An organisation ruleset blocks the merge of a pull request that brings a new high or critical CodeQL alert #310's annotation carries the same ArtifactFailed line.)
  • k3d 5.9.0 prints Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap... in failing and passing jobs alike, so the records are lost or never take effect afterwards.

Not known: what removes them. The job prints nothing of CoreDNS's NodeHosts. This item does not need the answer: it removes the dependency.

What it builds

Only two objects inside the cluster reach the job's hosts by name (git grep at 11d9311):

  1. tests/flux/serve_the_candidate_commit.sh:47 writes CANDIDATE_SOURCE_URL with host.k3d.internal, though the script already holds that very address as bind_address (:29), the Docker network's gateway.
  2. tests/secrets/serve_encrypted_secrets_as_platform_secrets.sh:24 writes the OCIRepository's url with ${REGISTRY_NAME}:${REGISTRY_PORT}.

Each takes an address read from the Docker network k3d-${CLUSTER_NAME} instead: the gateway for the first, the registry container's address on that network for the second. Reading fails loud when the address is empty, as :30 does for the gateway.

The registry name stays where it is used outside CoreDNS: k3d cluster create --registry-create and the node's containerd mirror, which resolve through Docker's own DNS.

Alternatives weighed.

  • A wait after k3d cluster create for both names in NodeHosts: it shortens the failure and does not remove it.
  • A re-injection or a coredns-custom *.server block for the two zones: more moving parts in the job to keep a name nobody needs.
  • Raising the 3-minute wait: the lookups fail on every try, so time does not help.

The address form needs no new part and deletes the dependency.

Proof

  • cluster is green on all three parts on the lane's tip, the steps that read those two urls included.
  • The builder's trace: git grep for host.k3d.internal and REGISTRY_NAME at the lane's base names every consumer, and the pull request lists each with what it now uses. One flake's absence cannot be proven by one run; the proof is that no in-cluster request names a host CoreDNS must know.
  • No cluster runs on the PC. CANDIDATE_SOURCE_URL's other readers (tests/flux/flux_reconciled_the_candidate_and_decrypted_the_dummy.sh, which compares the url with the variable) follow by reading the variable.

Risk

None material: CI only. No shipped manifest, PC step or public contract changes. A wrong address fails the job loudly at its first reconcile.

Deliberately untouched

  • AGENTS.md: no sentence names either host.
  • .github/workflows/ci.yml changes only in its comment on the registry's name, which said the cluster reaches the registry by name; no step changes.
  • k3d's own record injection: it keeps running, and nothing depends on it any more.
  • tests/flux/committed_source_url_turns_ready.sh: it proves the committed root's own url, unless the trace shows that it uses one of the two names.

Done when

  • The GitRepository's and the OCIRepository's urls in cluster hold addresses read from the Docker network, and no in-cluster request of the job names host.k3d.internal or the registry's name.
  • cluster is green on all three parts on the merged result.
version = 1
now = "Opened on 09.11.2026 from three cluster failures with no such host for the job's two hosts, a red trunk among them."
next = "Build: both sources take an address read from the Docker network."
done_when = "The GitRepository's and the OCIRepository's urls in cluster hold addresses read from the Docker network, no in-cluster request of the job names host.k3d.internal or the registry's name, and cluster is green on all three parts on the merged result."

scope = [".github/workflows/ci.yml", "tests/flux/serve_the_candidate_commit.sh", "tests/secrets/serve_encrypted_secrets_as_platform_secrets.sh"]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions