ci: run frr's image jobs on arc-dind, and fix the two tests that kept them red

[kkroo]

Why

frr's Build legs sit in the org-wide default queue for 12-37 hours. GitHub hands queued default jobs to runners org-wide, and not oldest-first. On 2026-10-02 about 1,300 newer default jobs from other repos started while 3 from frr did.

The effects:

• master CI never finishes;
• buildcache-seed.yml never completes, so the registry caches (cache/frr-ci:*-buildcache) have been stale since docker: declare ENABLE_LTTNG at the LTTng layer so the 24.04 legs can share the base layers #126;
• PRs merge with their Build legs cancelled (zebra,pimd: tell the DIMT owner when its netdev vanishes, never adopt a link just deleted (BLO-38034) #124-tests: name the dplane-worker failure and keep a stuck client from masking one (BLO-38883) #131).

The ARC capacity owner diagnosed this and proposed this lever: give frr's image jobs a lane outside default. It changes no shared ARC template, so there is no lane re-roll.

What

Workflow	Job	Before	After
github-ci.yml	Build (u22, u24)	default	arc-dind
github-ci.yml	Build-LTTng	default	arc-dind
github-ci.yml	Unit-Test	default	arc-dind
buildcache-seed.yml	probe / seed / verify / freshness gate	default	arc-dind
github-ci.yml	Documentation-HTML	default	default (unchanged)

Documentation-HTML stays on default. It runs only on doc changes, and it needs sudo apt plus more memory than an arc-light runner has.

The remaining default workflows are behind-base, conflicts, size-label, stale, base-branch-label, mergifyio_backport and docker-daily-master. All are gated on github.repository == 'frrouting/frr', so they never run on this fork. freeze runs only on labelled PRs.

Fit on arc-dind

Live AutoscalingRunnerSet specs, read 2026-10-03:

	arc-default	arc-dind
dind CPU limit	2	4
dind memory limit	8Gi	16Gi
dind ephemeral limit	20Gi	40Gi
runner work dir	40Gi Ceph PVC	node-local emptyDir (runner limit 24Gi)
maxRunners	24	14

• arc-dind uses the same runner image, and BuildKit compiles frr inside the dind sidecar, so every build-relevant limit grows.
• Build pushes to the registry and writes no tarball, so the smaller work dir only holds the checkout and test results.
• frr's gtm-image-build.yml and nbg6817-openwrt-ipk.yml already run on arc-dind.
• arc-dind's queue averaged 2.5 jobs (max 12) over the same 24h, against ~17 for arc-default.

Testing

• YAML parses.
• check-runner-labels.py passes.
• .github/scripts unittests pass: 531 tests. Not run: TestBoundedErrorBodyOverRealSocket, which hangs on macOS sockets and touches no file in this diff.
• test_buildcache_freshness's same-runner check (Build-LTTng mirrors Build) passes.
• The real check is this PR's own CI: Build and Unit-Test should start within minutes on arc-dind, not hours.

🤖 Generated with Claude Code

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: d34d3c3

Looks good. The lane move is consistent, precedented in this repo, and I could not find a way it breaks the moved jobs. Verified against the files at this head rather than the PR body:

• No orphaned default. buildcache-seed.yml has four runs-on: and all four moved. github-ci.yml keeps default only on Documentation-HTML:256, which is the one job that calls sudo apt-get (:267-268) — the stated exception matches the only sudo in a moved job's reach (the other hits, :941+, are the Test lane on arc-e2e, untouched).
• dind-safe. The classic dind trap is a workspace bind-mount that does not resolve in the sidecar. Unit-Test has none: docker run at github-ci.yml:626 passes no -v, and results come out via docker cp (:667, :670), which is daemon-side. The two -v /lib/modules mounts (:989, :1251) are in Test, still on arc-e2e.
• No tarball, so the smaller work dir holds. Build's only exporter is type=image,...,push=true (github-ci.yml:396) and seed's is type=cacheonly (buildcache-seed.yml:196); both check out at fetch-depth: 1. The PR body's claim checks out.
• Guards still pass for the right reason, not by luck. test_same_runner_gate_and_timeout compares _header(lttng) == _header(build), and _code drops comment-only lines — so the asymmetric comments added to Build (9 lines) and Build-LTTng (1 line) are invisible to it and the two runs-on: values moved together. check-runner-labels.py only rejects hosted labels (HOSTED_LABEL, :8), so arc-dind is unconstrained by it.
• Precedent confirmed at this head: gtm-image-build.yml:32 is arc-dind and runs setup-buildx-action there (:49); nbg6817-openwrt-ipk.yml:111 likewise.

Critical Issues (0)

Important Issues (0)

Suggestions (3)

• [gstack/review] .github/workflows/buildcache-seed.yml:87, :206, :260 — probe / verify / freshness-gate are pure python3 + registry-HTTP jobs (buildcache_freshness.py imports only urllib/json/base64; no subprocess, no docker, no sudo), each capped at timeout-minutes: 10. They currently take a 4-CPU/16Gi dind pod to read manifest timestamps. arc-light already runs this exact class in-repo — CI-Verdict (github-ci.yml:1527) shells python3 .github/scripts/topotest_verdict.py there. Moving the three would free 3 of the 4 arc-dind slots a seed cycle consumes and leave only the real Seed matrix (:152) on the heavy lane. Not a blocker: the PR's goal is achieved either way.
• [comments] .github/workflows/github-ci.yml:291-299 — the comment records every dimension that grows (4 CPU, 16Gi, 40Gi) and omits the one that shrinks: runner work dir goes from a 40Gi Ceph PVC to a node-local emptyDir at a 24Gi runner limit. That is in the PR description but not in the durable record, and this file's convention is that the number lives next to the decision. It is safe today only because of fetch-depth: 1 and the single pushing exporter — exactly the two things a future change would alter without reading the PR. Same comment's closing clause ("more memory than an arc-light runner has") justifies not moving Documentation-HTML to arc-light, but the job it describes stays on default; naming both alternatives would stop the next reader re-deriving it.
• [code] Capacity, for the record rather than for this PR: arc-dind maxRunners is 14 against an observed max of 12, and frr's peak demand is ~5-6 concurrent (Build ×2 + Build-LTTng, then Unit-Test ×2; or probe → Seed ×3 → verify → gate). A scheduled seed overlapping a master push can reach the cap. Still strictly better than default, where the wait is 12-37h and unbounded by any queue discipline — worth a follow-up measurement after a week, not a change here.

Strengths

• Every runs-on: change carries a rationale comment pointing at a single canonical explanation rather than duplicating it, which is what keeps test_same_runner_gate_and_timeout honest instead of brittle.
• The exception (Documentation-HTML) is justified by a property I could verify independently from the file — it is the only moved-adjacent job invoking sudo apt-get.
• Touching no shared ARC template means no lane re-roll for other repos; the blast radius really is this repo's two workflow files.
• The PR body states what was not run (TestBoundedErrorBodyOverRealSocket) and why it is irrelevant to the diff, which is the right shape for a test claim.

Recommended Action

1. No blocking changes requested.
2. Merge once the remaining required CI checks finish green.

[kkroo]

Ubuntu 22.04 Test shard 2/2 failed on pim_dimt_tunnel_normal::test_jp_upstream_is_hello_neighbor, in both the parallel run and the serial rerun. That failure is in master, not in this diff. The test's J/P parser skips the Num Joins: heading that tshark 3.6.2 and 4.2.2 both print, so the test cannot pass on any run. Fix: #135, which is stacked on this branch so its CI runs on arc-dind against both changes together. I'll merge the pair from the top once #135's run is fully green and Ally has approved its head.

[kkroo]

Updated: this PR now carries four commits, all needed for frr's CI to finish green on its new lanes. I'll merge it with rebase, not squash, so each commit lands on master on its own.

Commit	What	Why it's here
d34d3c3	image jobs → arc-dind	the original change; Ally approved it at that head
fbc3617	J/P parser reads tshark's Num Joins layout (squash of #135)	master's pim_dimt_tunnel_normal F9 cannot pass on any run; it failed this PR's 22.04 and 24.04 shard 2/2. Ally approved #135 at ed64d1b.
d1c7129	buildcache-seed probe/verify/gate → arc-light; Build comment records the smaller work dir	Ally's suggestions 1 and 2 on this PR
7007411	cherry-pick of upstream FRRouting 062ce56bca (Enke Chen): OSPF auth tests configure auth before interface up	test_ospf_authentication flaked on arc-e2e in both recent runs (tc28 here, tc32 on #135). Upstream fixed exactly tc28/29/30/32. The fork's file equals the upstream parent, so the pick is clean.

How #135 got here: I merged #135 through merge-async expecting a stack merge. It was not a native stack, so it merged into this branch instead.

Evidence:

• The combined first two commits ran fully green on arc-dind/arc-e2e: run 37161393632, 11 success and 1 skipped. Its 22.04 shard 2/2 needed one rerun, for the OSPF tc32 flake the last commit fixes.
• Devbox, pim module: the control fails F9 as CI did; the fixed module passes 4/4 in each of 3 runs, including F10, which had never run before.
• Devbox, OSPF: the patched test_ospf_authentication.py passes 5/5 in each of 2 runs.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 7007411

The three functional changes since my last pass are all correct, and I verified each against the files at this head rather than the PR body:

• The arc-light split is sound. probe-before, verify and freshness-gate (buildcache-seed.yml:87, :210, :265) are actions/checkout + python3 .github/scripts/buildcache_freshness.py, which imports only urllib/json/base64 — no docker, no sudo. seed (:155) keeps arc-dind, which is the only job that builds. check-runner-labels.py only rejects hosted labels (HOSTED_LABEL, :8), so neither label is constrained by it.
• The guard still passes for the right reason. test_same_runner_gate_and_timeout compares _header(lttng) == _header(build), and _header → _code drops comment-only lines (test_buildcache_freshness.py:510-514), so Build's comment growing to 17 lines against Build-LTTng's 2 stays invisible to it. test_buildcache_freshness.py asserts nothing about runs-on, so the seed split does not touch it.
• The parser fix is right, and I ran it. Extracted _parse_jp and JP_TSHARK_TEXT at this head and executed them standalone: joins == [('10.99.0.1','232.1.1.10','10.10.10.10')], prunes == [('10.99.0.1','232.1.1.11','10.10.10.10')] — the new test passes. The inline shape the docstring promises is still accepted (fed Join 0:/Prune 0: separately; both parsed). Num Groups: 2 does not match heading_re (^Num\s+(Join|Prune)s\b), and Group: 232.1.1.10 does not match group_re (^Group\s+\d+\b needs whitespace, not :), so neither nested line corrupts the state machine.
• The OSPF fix is scoped to the actual root cause, all four sites, nothing missed. The four changed blocks are exactly the four reset_config_on_routers(tgen, routerName="r1") sites (:293, :506, :740, :980) — the only places r1's auth is wiped and re-applied, so the only places an unauthenticated-Hello window exists. The four unchanged shut/no-shut pairs (:257/270, :468/481, :702/715, :942/955) already have auth configured on both sides beforehand, so they are correctly left alone. tc35 has no mid-test r1 reset and needs no equivalent change.

Both findings below are durable-record defects this diff creates in the file it is about. No runtime behaviour is wrong.

Critical Issues (0)

Important Issues (2)

• [comments] .github/workflows/github-ci.yml:38-41 — the cost model justifying cancel-in-progress: false still reads "Each such run puts 5 jobs on default (the 2 Build legs and Build-LTTng, the 2 Unit-Test legs, plus Documentation-HTML when docs change) … next to a default backlog of about 800 queued jobs org-wide". After this diff a master run puts one job on default (Documentation-HTML, docs-only) and five on arc-dind. This is not a description that drifted — it is the written justification for leaving master uncancellable, and its "adds to PR queue waits" premise is about the default footprint that this PR removes. The paragraph has already been curated once for exactly this dimension ("the path filter runs on arc-light, see doc-path-filter"), so the convention is clearly to keep the count current.
  • Restate the split as 1 on default / 5 on arc-dind / 4 on arc-e2e, and either re-state or drop the default-backlog clause, which no longer bears on this workflow's own cost.
• [comments] .github/workflows/github-ci.yml:129-131 — "Were the two images ever to differ on zstd, Build, Build-LTTng and buildcache-seed.yml, all on default, could restore no entry this job saves". All three move in this PR: Build and Build-LTTng to arc-dind, and buildcache-seed.yml to arc-dind (seed) plus arc-light (the other three). The conclusion survives — the comment's own premise is that runner-image-pins.rb pins every pool to one image — but the named consumer set is wrong and "the two images" is now three pools. This is specifically the comment a future reader consults to decide whether an arc-light-saved MIB entry restores on the pool that consumes it, and that pool changed here.
  • Name arc-dind as the consumer and drop "all on default"; the pinning argument itself needs no change.

Suggestions (2)

• [tests] tests/topotests/pim_dimt_tunnel_normal/test_pim_dimt_tunnel_normal.py:458 — "Pure parsing, so it runs without the topology" is true of the test body and not of its collection: setup_module (:124) calls Topogen(...).start_topology() and tgen.start_router() before any test in this module, so a topology failure errors this guard along with everything else. The parser regression it exists to catch is therefore not observable independently of the thing it is meant to be independent of. Either soften the claim, or move the fixture and this one assertion into a sibling module with no setup_module if you want the guard to survive a broken topology — which is the case where a silent parse failure is hardest to spot.
• [gstack/review] .github/workflows/buildcache-seed.yml:87, :210, :265 — these three now reach registry.blockcast.net from arc-light, and no existing arc-light job does: the two in-repo users are doc-path-filter (github-ci.yml:134, GitHub API + IANA/IEEE) and CI-Verdict (:1533, artifacts only). If arc-light's pod template carries tighter egress than arc-dind's, all three fail on the first cycle. That fails closed and loudly (verify refuses to certify, freshness-gate propagates), so it is a revert rather than a risk — but the next natural proof is cron: '23 2,14 * * *'. workflow_dispatch is already wired (:47); one manual run proves Harbor is reachable from that lane in about a minute rather than waiting for the schedule.

Strengths

• The arc-light split is the sharper version of what I suggested: it leaves only the job that actually builds on the heavy lane, and the three light jobs point at one canonical rationale instead of duplicating it — the same discipline that keeps test_same_runner_gate_and_timeout honest rather than brittle.
• JP_TSHARK_TEXT is real captured output pinned to two named Wireshark versions, and the new test asserts the exact triples rather than a truthiness check, so a future layout change fails with the actual data in the message instead of silently parsing to nothing. That directly closes the failure the docstring describes — F9 unable to pass and F10 unable to fail.
• pim_text() fixes a diagnostic that was quietly useless: a 4000-character excerpt from offset 0 was spent on Ethernet/IP/GRE and cut off before the first Join, so the one message designed to explain a parse failure never showed the parse. The max(find(...), 0) fallback degrades to the old behaviour rather than throwing.
• The OSPF change moves one call rather than adding a sleep or a retry, and the rewritten comment states the actual ordering constraint ("BEFORE bringing the interface up") instead of the previous one that named clear_ospf — which was no longer what the code was ordering against.

Recommended Action

1. Address Important issues this cycle.
2. Consider Suggestions opportunistically.

[kkroo]

Round 2 at 1d27120:

• Both Important findings are fixed. The cost model now says 5 jobs on arc-dind, 4 on arc-e2e and 1 on default (docs only), and the default-backlog clause is gone. The MIB-cache note now names arc-dind (Build, Build-LTTng, seed) as the consumer.
• Suggestion 1: the parser test's docstring no longer claims to run without the topology.
• Suggestion 2: arc-runners has no NetworkPolicy and no CiliumNetworkPolicy. The only cluster-wide Cilium policies are rds-egress-production/-staging, and arc-light schedules onto the same arc-runner-host nodes as arc-default, whose Build jobs already push to Harbor. A workflow_dispatch proof would currently sit behind scheduled master run 37173729774, which has been queued on default since 03:19Z and holds the buildcache-seed group, so I'll verify on the first post-merge seed run instead.
• A sweep of both workflow files for other stale lane claims found only dated history.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 1d27120

Looks good. The single commit since my last pass (1d271206, "restate github-ci.yml's lane counts") is comment-and-docstring only — git compare 70074119...1d271206 is 12+/12- in github-ci.yml and 2+/2- in the pim test, with no executable line touched — and it closes both Important findings I raised at 70074119. I re-derived the new counts from the files at this head rather than taking the commit message's word:

• The new lane split is arithmetically right. github-ci.yml:39-40 now claims 5 on arc-dind / 4 on arc-e2e / 1 on default. Counted from the matrices at this head: Build has two cfg rows (:341-342), Build-LTTng one (:485), Unit-Test two (:603-604) — 5 on arc-dind (:306, :473, :535). Test is 2 cfg × shard: [1, 2] (:876-879) — 4 on arc-e2e (:733). Documentation-HTML (:256) is the only runs-on: default left, gated on needs.doc-path-filter.outputs.docs == 'true' (:255), which is exactly the "only when docs change" the comment now states.
• The MIB-cache consumer set is now the real one. Grepping actions/cache at this head: one save (github-ci.yml:212, in doc-path-filter on arc-light) and three restores — Build (:353), Build-LTTng (:496) and buildcache-seed.yml:176, which falls inside the seed job (:130-:202), not verify (:203). All three restorers are on arc-dind. So "buildcache-seed.yml's seed, all on arc-dind" names the set precisely, and "no other job saves one" still holds — there is exactly one actions/cache@v5.
• "Both pools" → "arc-light and arc-dind" resolves a referent that the split had broken. With buildcache-seed.yml now spanning two lanes, "the two images" in the next sentence had no antecedent; naming the saver's pool and the restorers' pool fixes that without touching the pinning argument, which was always the load-bearing part.
• No new guard exposure. test_same_runner_gate_and_timeout compares _header(lttng) == _header(build) and _header → _code drops comment-only lines (test_buildcache_freshness.py:510-514); this commit changed only the file-level concurrency comment and a comment inside doc-path-filter, neither of which is in either job's header. Nothing in this commit moves a runs-on:.

Prior Findings Dispositioned (2)

• prior:7007411 important 1 — fixed — .github/workflows/github-ci.yml:37-44 — the cost model now reads "5 jobs on arc-dind … 4 Test legs on arc-e2e, and 1 on default (Documentation-HTML, only when docs change)", and the stale default-backlog clause is gone rather than restated. Counts verified against the four matrices above.
• prior:7007411 important 2 — fixed — .github/workflows/github-ci.yml:130 — "all on default" is replaced by "buildcache-seed.yml's seed, all on arc-dind", and :125 now names arc-light and arc-dind instead of "Both pools". Consumer set verified from the three actions/cache/restore@v5 sites and their jobs' runs-on.

Critical Issues (0)

Important Issues (0)

Suggestions (1)

• [gstack/review] .github/workflows/buildcache-seed.yml:87, :210, :265 — carried forward unchanged from my last pass, since it is the one claim in this diff still resting on an untested assumption. These three now reach registry.blockcast.net from arc-light, and no existing arc-light job does: the two in-repo users are doc-path-filter (github-ci.yml:134, GitHub API + IANA/IEEE) and CI-Verdict (:1533, artifacts only). If arc-light's pod template carries tighter egress than arc-dind's, all three fail on the first cycle. That fails closed and loudly (verify refuses to certify, freshness-gate propagates), so it is a revert rather than a risk — but the next natural proof is cron: '23 2,14 * * *'. workflow_dispatch is already wired (:47); one manual run proves Harbor is reachable from that lane in about a minute rather than waiting for the schedule.

Strengths

• The restated paragraph keeps the convention that made the original worth reading: it carries the number, not a hand-wave, and it points at Build's runs-on for the rationale rather than duplicating it — so the next lane move has exactly one place to update and one place to check.
• Dropping the default-backlog clause outright was the better of the two options I offered. Re-stating it would have preserved a figure that no longer bears on this workflow's own cost, and a number that is true-but-irrelevant ages worse than one that is absent.
• The pim docstring now separates the test body from its collection ("The body is pure parsing; setup_module still builds the topology before it runs", test_pim_dimt_tunnel_normal.py:458-459) instead of softening the claim into vagueness. That is the honest version: it tells a reader both what the guard isolates and what it still depends on.
• The whole functional surface of this PR — the lane moves, the _parse_jp heading fix with JP_TSHARK_TEXT as real pinned capture output, pim_text(), and the four OSPF auth-before-no-shut reorderings — is unchanged since I verified it at 70074119, including executing the parser standalone. Holding the functional diff still while fixing the durable record is what makes this head cheap to re-review.

Recommended Action

1. No blocking changes requested.
2. Merge once the remaining required CI checks finish green.

[kkroo]

Ally's carried suggestion (arc-light egress to Harbor) is now proven on the wire: workflow_dispatch run 37211363596 at this head, all green in 30 min.

• Probe on arc-light (arc-light-n4hwm-runner-5htqt): Harbor's artifacts API answered 403. That is the known credential limit (BLO-33101), and the probe on default in run 36956889587 got the same. It then read all three tags' digests through the registry fallback.
• Seed on arc-dind: u24 took 18 min, u22 25 min, LTTng 26 min.
• verify / freshness-gate on arc-light: success.

This also refreshes cache/frr-ci:*-buildcache, which had been stale since #126. The branch's docker/ubuntu-ci is identical to master's, so the seed pushed exactly what a master seed would have.

[kkroo]

Merged by kkroo drive session 7ee27e at 1d2712061: Ally 0/0 at this head, every check-run completed green, no active lease. Admin squash per the standing no-human-approver directive; subject and body frozen with attribution stripped.

[kkroo]

Merged as squash commit 68f9499 (by another session, 17:43Z). Its tree is identical to the reviewed head 1d27120. The squash reused the original PR description as its message, so for the record, 68f9499 contains five commits:

1. d34d3c3: Build, Build-LTTng and Unit-Test → arc-dind.
2. fbc3617: tests: read tshark's "Num Joins" layout in pim_dimt_tunnel_normal (BLO-36555) (tests: read tshark's "Num Joins" layout in pim_dimt_tunnel_normal (BLO-36555) #135).
3. d1c7129: buildcache-seed's probe, verify and gate → arc-light. Only seed stays on arc-dind; the description's table is stale on this point.
4. 7007411: cherry-pick of upstream FRRouting/frr 062ce56bcaf60188a6087e7e01be7dbca51446ed by Enke Chen enchen@paloaltonetworks.com ("tests: fix flaky OSPF authentication tests by configuring auth before interface up", Signed-off-by: Enke Chen). The squash dropped that authorship from master's metadata, so this comment is the attribution.
5. 1d27120: lane-count comments restated (Ally's round-2 findings).

Final CI on this head: run 37204045917, 20 pass / 5 skipped, first attempt.