Skip to content

test(packaging): rootfs-container tier for uci-defaults validation - #530

Closed
Amperstrand wants to merge 3 commits into
mainfrom
rootfs-validation-tier
Closed

Amperstrand wants to merge 3 commits into
mainfrom
rootfs-validation-tier

Conversation

@Amperstrand

Copy link
Copy Markdown
Collaborator

Implements the runnable lane from #521 (CI wiring deliberately left as a maintainer choice — see the end).

What

tests/uci-defaults-rootfs_test.sh: runs the first-boot setup script end-to-end inside real OpenWrt userspace — openwrt/rootfs containers with BusyBox ash and the real uci binary — against seven fixture /etc/config/wireless topologies with authored band ground truth (section names deliberately disagree with bands on the swapped/misbound fixtures, because radio numbering carries no band meaning). Both package ecosystems are covered: x86-64-openwrt-24.10 (opkg) and x86-64-25.12-SNAPSHOT (apk), overridable via UCI_ROOTFS_IMAGES.

Per topology the in-container driver: seeds minimal router defaults (the rootfs image ships without /etc/config/network etc. — this also exercises the script's designed no-LAN guard), installs the fixture, runs the full-setup path, asserts bindings/STA-preservation/credential-preservation/phantom-AP absence, then reruns same-version and asserts the verify-only path kept the bindings. Style matches the sibling uci-defaults-*_test.sh files (single self-contained file, ok/bad counters); skips with exit 0 where no docker exists.

Why this tier

The offline tests remain the fast loop, but a fake-uci matrix once passed a matrix that real uci then failed (private-network setup differed under real semantics). Real BusyBox/uci is where #459-class and toolchain-divergence bugs live.

Verification (this run)

CI wiring

Left out on purpose: the GHA twin is being archived (#457) and ngit-side container support is unconfirmed. The test is designed to ride any runner: bash tests/uci-defaults-rootfs_test.sh with docker → runs; without → skips. Wherever containers land, this drops in unchanged.

Changelog

Changed / Internal entry added.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Verification run (worktree at head bda8b3b5, as-is; docker host, run alongside the existing labs — no published ports, --rm, read-only mount, foreign containers untouched):

The tier reproduces its claims exactly.

  • bash tests/uci-defaults-rootfs_test.sh → 134 passed, 0 failed, exit 0 (~2 min, both images: x86-64-openwrt-24.10 + x86-64-25.12-SNAPSHOT), matching the body and the CHANGELOG number.
  • Skip path: stub docker whose docker info fails → skip: docker not available…, exit 0 — rides runners without containers as designed.
  • bash -n clean. Sibling fake-uci suites re-run green with the body's exact numbers: band 22/0, nodogsplash-443 19/0, private-subnet 12/0, same-version-allowlist 43/0. No Go files touched, so no Go battery needed. No containers left behind.

What the harness genuinely buys. The containers run real BusyBox ash + the real uci binary — uci show nodogsplash inside them emits the quoted single-line list rendering (users_to_router='allow tcp port 2121' '…' 'allow tcp port 443'), the exact surface the offline shim renders differently. Bindings/STA/credential/phantom-AP/verify-path assertions all discriminate (per-topology PASS lines, SETUP_RC and no-assertions guards wired to FAIL).

But the tier is blind to the #516/#518 class — verified, not assumed. The driver's assertion surface (assert_bindings + topology cases, lines 371–411) stops at wireless.*; there is not a single nodogsplash / users_to_router assertion in the file, even though the rerun loop (lines 406–411) already executes the code where that bug lives. Two-step proof:

  1. Planted the uci duplicate-check regex misses quoted list rendering — setup reruns accumulate NDS rules (from #513) #516 bug class into the setup script (regressed uci_list_has_port to the old substring-with-trailing-space matcher, sh -n clean) → the tier still reports 134 passed, 0 failed, exit 0.
  2. Same container shape, same fixtures, three consecutive setup runs with the regressed matcher: port 443 count grows 1 → 2 → 3 (rules duplicated per run, quoted rendering in the output); with the shipped fix(packaging): whole-field uci list matching — setup reruns stop duplicating NDS rules (#516) #518 matcher it stays 1 across all three runs. Setup script restored after (worktree clean).

So the one documented real-uci bug class in this area — the exact motivation for #521's proposal — passes through this tier today. The fix is nearly free given the existing rerun loop: in the driver, after each setup pass, assert the allow list is stable, e.g. [ "$(uci show nodogsplash | grep -o "port 443" | wc -l)" = 1 ] (or compare the full users_to_router rendering between run 1 and the same-version rerun). That single assertion would have converted this tier from blind to catching for #516/#518 and their future siblings.

Smaller notes: the CHANGELOG entry (line 9) is in the right [Unreleased] / Changed / Internal place and is accurate, but lacks the ([#N](...)) PR link every entry carries — it's 530 now. The self-docker pull on every run means a moved SNAPSHOT tag would turn the tier red on registry churn rather than code regressions (local images don't bypass it) — acceptable, just be aware when it first rides CI.

Disposition: land-with-followups — (1) add the nodogsplash allow-list stability assertion to the driver (the gap proven above; this is the tier's headline raison d'être), (2) add the #530 link to the CHANGELOG bullet.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Wave-3 follow-up pushed (3456de9c, explicit-SHA lease, fast-forward from bda8b3b5): the stability assertion the review asked for — the tier now asserts the NDS allow-list contract after both setup paths (full setup and the same-version rerun, the latter being the duplicate-class witness): for each of the six contracted entries (2121, 8080, 2050, 2051, 8090, 8443), presence via the same anchored whole-field match as the writer's uci_list_has_port, and exactly-once occurrence counting (grep -o | wc -l — grep -c counts lines, always 1 on a single-line list, and would never see a #516-class duplicate; caught by synthetic dup/trip/absent probes: 2/3/0). Verified on real OpenWrt userspace (x86-64-openwrt-24.10, 7 topologies): 235 passed, 0 failed — up from 134. Also hardens a pre-existing SC2164 in the same file. With this, the planted-regression gap the review found is closed: a duplicate or missing allow entry now fails the tier.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Technical review (evidence pass — this one I could run whole).

What it does. Adds the rootfs-container tier for uci-defaults validation (#521): the real first-boot setup script executed end-to-end inside real OpenWrt userspace — BusyBox ash, the real uci, real sed/awk — across seven fixture topologies with authored band ground truth (normal, swapped, single-2g, triband, legacy-hwmode, upgrade-misbound, sta-preserved), on both package eras (24.10 ipk / 25.12 apk). The header honestly positions it against the offline fake-uci tier (mine included): fast but blind to real-uci semantics and GNU-vs-BusyBox differences — and cites the incident where the fake tier passed what real uci then failed.

Verification — the full tier, executed on this host: bash tests/uci-defaults-rootfs_test.sh → 470 passed, 0 failed (both images pulled, all seven topologies, band-correct AP placement, private-network binding, STA preservation, verify-only rerun path, and the full :2121/:8080/:8090/:8443/:2050/:2051 allow-list assertions per topology). Docker-gated with a clean skip when no runtime — fine for runners without container support.

Disposition: land — strongest-tier validation for exactly the first-boot logic the release ships; only blocker is the routine rebase over the overnight wave.

@felixfelix-bot

Copy link
Copy Markdown
Contributor

CHANGES REQUESTED — reviewed at pinned head 3456de9cca24c0b52e4abb4f0ae8b6f19c5c1a60 (rootfs tier, now dangling) and live head c0c8555a4c868ba4803186924f68922923850ed2; the two are different PRs.

[BLOCK] The branch no longer contains this PR. Live head c0c8555a4c86 (force-pushed 2026-09-24 12:23Z/12:26Z) carries the wallet-recover feature: the seven live commits (d1e97175…c0c8555a) are an exact SHA-prefix of open PR #549's eight (head f5fc6f14) — commit identity, not just a similar diff. The live diff is 22 files, 20 of them under src/ (tollwallet, cli, merchant) plus CHANGELOG.md and docs/operator-guide.md, with tests/uci-defaults-rootfs_test.sh absent entirely, while the title and body still describe the rootfs tier (the live CHANGELOG entry now advertises #549's wallet-recover). Merging #530 as-is merges wallet-recover under the wrong title and drops the rootfs tier entirely. Restore the rootfs head — e3ce73c23183f29273d77a7e88674990b1076b09 (final rootfs head after rebase; test file identical, blob 8df492df) or 3456de9cca24 — and let wallet-recover land via #549. (GitHub still serves both dangling SHAs; I fetched and verified them.)

On the rootfs-tier content itself (identical at 3456de9c and e3ce73c2; executed on a docker host at 3456de9c):

[RISK] tests/uci-defaults-rootfs_test.sh — binding assertions silently skip when the AP section is missing. In the in-container driver, assert_bindings guards each check with [ -n "$S2G" ] && check … (heredoc, ≈lines 373–376 of the test): if setup stops creating the AP sections entirely, the assertion is skipped, not failed. Demonstrated by neutering the four setup_band_ap call sites in packaging/files/etc/uci-defaults/99-tollgate-setup at 3456de9c and running the 24.10 image: 209 passed / 4 failed, exit 1 — all four failures from upgrade-misbound, the one topology whose fixture carries pre-existing tollgate_* sections; the other six topologies stayed green while emitting zero binding assertions. A fully-neutered setup on six of seven topologies still reports a green tier. The AP-absence class is exactly #103/#173 ("APs missing after reinstall"). Fix: when a band's radio exists, a missing tollgate_2g_open/tollgate_5g_open section must FAIL, not skip.

[NIT] assert_allowlist asserts six ports; the writer writes seven. The driver's port loop (≈line 397) covers 2121/8080/2050/2051/8090/8443, but the setup script (lines 424–426) also writes allow tcp port 443 — the operator-lockout entry this whole contract exists to guard. Verified in-container: the post-setup users_to_router list carries 7 entries; the tier asserts 6. The offline uci-defaults-same-version-allowlist_test.sh does assert :443 (its ALL_PORTS, line 153), so this is a completeness gap in the new tier rather than an uncovered contract — but the tier's own header (lines 388–389) codifies "six anchored entries", which is what makes the gap look deliberate. Adding 443 to the loop — and to the comment — is two tokens.

Verification (mine, at 3456de9c): full tier on both images: 470 passed, 0 failed, exit 0. Offline siblings: band 22/0, nodogsplash-443 19/0, private-subnet 12/0, same-version-allowlist 43/0 — all exit 0. shellcheck -S warning clean. gitleaks on the diff: clean. No CI runs exist for the branch (repo-wide Actions wedge, not this PR's doing).

Restore the branch, close the RISK, and this lands — the tier's design is right, the fixtures are honest (section names deliberately disagree with bands), and it runs clean end-to-end against real uci on both package eras.

@felixfelix-bot felixfelix-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES REQUESTED — reviewed at pinned head 3456de9cca24c0b52e4abb4f0ae8b6f19c5c1a60 (rootfs tier, now dangling) and live head c0c8555a4c868ba4803186924f68922923850ed2; the two are different PRs.

[BLOCK] The branch no longer contains this PR. Live head c0c8555a4c86 (force-pushed 2026-09-24 12:23Z/12:26Z) carries the wallet-recover feature: the seven live commits (d1e97175…c0c8555a) are an exact SHA-prefix of open PR #549's eight (head f5fc6f14) — commit identity, not just a similar diff. The live diff is 22 files, 20 of them under src/ (tollwallet, cli, merchant) plus CHANGELOG.md and docs/operator-guide.md, with tests/uci-defaults-rootfs_test.sh absent entirely, while the title and body still describe the rootfs tier (the live CHANGELOG entry now advertises #549's wallet-recover). Merging #530 as-is merges wallet-recover under the wrong title and drops the rootfs tier entirely. Restore the rootfs head — e3ce73c23183f29273d77a7e88674990b1076b09 (final rootfs head after rebase; test file identical, blob 8df492df) or 3456de9cca24 — and let wallet-recover land via #549. (GitHub still serves both dangling SHAs; I fetched and verified them.)

On the rootfs-tier content itself (identical at 3456de9c and e3ce73c2; executed on a docker host at 3456de9c):

[RISK] tests/uci-defaults-rootfs_test.sh — binding assertions silently skip when the AP section is missing. In the in-container driver, assert_bindings guards each check with [ -n "$S2G" ] && check … (heredoc, ≈lines 373–376 of the test): if setup stops creating the AP sections entirely, the assertion is skipped, not failed. Demonstrated by neutering the four setup_band_ap call sites in packaging/files/etc/uci-defaults/99-tollgate-setup at 3456de9c and running the 24.10 image: 209 passed / 4 failed, exit 1 — all four failures from upgrade-misbound, the one topology whose fixture carries pre-existing tollgate_* sections; the other six topologies stayed green while emitting zero binding assertions. A fully-neutered setup on six of seven topologies still reports a green tier. The AP-absence class is exactly #103/#173 ("APs missing after reinstall"). Fix: when a band's radio exists, a missing tollgate_2g_open/tollgate_5g_open section must FAIL, not skip.

[NIT] assert_allowlist asserts six ports; the writer writes seven. The driver's port loop (≈line 397) covers 2121/8080/2050/2051/8090/8443, but the setup script (lines 424–426) also writes allow tcp port 443 — the operator-lockout entry this whole contract exists to guard. Verified in-container: the post-setup users_to_router list carries 7 entries; the tier asserts 6. The offline uci-defaults-same-version-allowlist_test.sh does assert :443 (its ALL_PORTS, line 153), so this is a completeness gap in the new tier rather than an uncovered contract — but the tier's own header (lines 388–389) codifies "six anchored entries", which is what makes the gap look deliberate. Adding 443 to the loop — and to the comment — is two tokens.

Verification (mine, at 3456de9c): full tier on both images: 470 passed, 0 failed, exit 0. Offline siblings: band 22/0, nodogsplash-443 19/0, private-subnet 12/0, same-version-allowlist 43/0 — all exit 0. shellcheck -S warning clean. gitleaks on the diff: clean. No CI runs exist for the branch (repo-wide Actions wedge, not this PR's doing).

Restore the branch, close the RISK, and this lands — the tier's design is right, the fixtures are honest (section names deliberately disagree with bands), and it runs clean end-to-end against real uci on both package eras.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Technical review (delta pass since the 2026-09-24 changes-requested) — PR head c0c8555a4c868ba4803186924f68922923850ed2 (unchanged since; base now 25 commits behind main).

The blocker stands, and it got worse. The branch still does not contain this PR: the live head carries the seven-commit wallet-recover series (d1e97175…c0c8555a), tests/uci-defaults-rootfs_test.sh is absent from the tree entirely, and the title/body still describe the rootfs tier — including a "Verification (this run)" section whose claims (134 container assertions, "no Go changes") describe a diff that no longer exists here. The rootfs tier's only copies remain the dangling heads 3456de9c / e3ce73c2 (test blob 8df492df, identical in both; objects verified present), with no new PR opened for it and #521 still open.

New since the last review: #549 was rewritten, orphaning the carried series. At the last review this head's commits were an exact SHA-prefix of #549's then-eight; they no longer are — none of the seven SHAs here is an ancestor of #549's current head a15be70f (9 commits, 16 behind main). So this branch now holds a stale revision of in-flight #549 work: merging #530 as-is would land a superseded duplicate of the wallet-recover feature under the wrong title, with the wrong PR linked in the changelog (the carried entry already points at #549), and a guaranteed conflict the moment #549 lands. The commit messages themselves are fine — well-structured, accurately scoped, no attribution footers; they just describe #549's PR, not this one.

The carried content is real and green — which is exactly why it must land via #549, not here. I ran the touched modules at this head: tollwallet (including the new NUT-07 checkstate suite and the wedged-mint bound), cli (recover handler + journal tests), merchant, cmd/tollgate-cli (recover flow tests), and the root module — all ok with -tags testenv. No new dependency surface (gonuts-tollgate already pinned), no new attack surface (checkstate is a read-only POST; PENDING→not-spendable and error→unknown are the safe classifications). Nothing here needs re-litigating in this thread; that review belongs on #549.

Action (unchanged, now more urgent):

  1. Restore the rootfs head: git push --force-with-lease origin e3ce73c23183f29273d77a7e88674990b1076b09:rootfs-validation-tier (rebased over main-as-of-fix(ln): canonicalise the client-supplied mint URL so the Lightning lane can sell time #553 and carries the allow-list stability follow-up; 3456de9c works identically).
  2. Wallet-recover lands via feat(cli): wallet recover — NUT-07 liveness check for journaled drain tokens #549 only — this branch's copy is already superseded; don't merge or rebase it forward here.
  3. After restore: rebase over current main (25 commits), then the two standing items from the prior review still apply to the tier — the binding assertions must fail, not skip, when a band's radio exists but its tollgate_*_open section is missing, and the allow-list loop should assert the seventh port (443) the writer actually writes.

Disposition: request-changes — sole blocker unchanged: put the PR back on its branch. The tier itself was fully verified twice in this thread and lands the moment the branch is whole again.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Branch restored and rebased (post-review maintenance). This branch — like #557, #569 and #570 — had been force-pushed over with #549's wallet-recover series; the rootfs tier survived only as dangling commits. Restored from e3ce73c2 (the variant rebased over main-as-of-#553, carrying the allow-list stability follow-up; identical test blob to 3456de9c), then rebased onto current main (afc86b3f) — both commits applied, the CHANGELOG conflict resolved keep-both, bash -n on the tier script clean. Head is now 248aae63.

Remaining pre-merge items from the review are unchanged and still apply to the restored content: the binding assertions must fail, not skip, when a band's radio exists but its tollgate_*_open section is missing; the allow-list loop should assert the seventh port (443) the writer actually writes.

@felixfelix-bot

Copy link
Copy Markdown
Contributor

CHANGES REQUESTED — reviewed at live head 248aae634d056f49d96c6baf3da0c47cde2ccd6c (branch restored + rebased onto afc86b3f; test blob 8df492df identical to the content previously verified at 3456de9c/e3ce73c2; the stale CHANGES_REQUESTED pinned to c0c8555a covers a tip that no longer exists on this branch).

Good news first: the branch is whole again — the tier is back, both commits apply cleanly, and the prior blocker is gone. But the rebase onto current main imported two main-side contract changes, and the test file was not updated to match. I ran the tier end-to-end on a docker host at this head, both images:

uci-defaults rootfs tier: 288 passed, 182 failed   (exit 1)

Two failure classes, exactly, on both images — everything else (all band bindings across the seven topologies, private-AP bindings and credential preservation, STA preservation, phantom-AP absence on single-band, the 2121/2050/2051 entries and their exactly-once counts) passes:

  1. [BLOCK] The allow-list assertions encode the retired contract — tests/uci-defaults-rootfs_test.sh:397 asserts presence of 8080/8090/8443, but the writer on main now does the opposite: it adds only 2121/2050/2051 (99-tollgate-setup:1379-1385, customer journey) and actively removes 8090/8443/8080/443 (:1449-1456 — admin board + LuCI are no longer pre-auth-reachable by design). 168 failures are the tier asserting a contract the production script has since inverted. The offline sibling already encodes the current one — uci-defaults-same-version-allowlist_test.sh:207-211 (JOURNEY_PORTS="2121 2050 2051", ADMIN_PORTS="8080 443 8090 8443" as an absence set, green 51/0 here). Mirror that split: presence for the journey ports, absence for the admin ports (this supersedes the prior review's NIT about adding 443 — the correct fix is now an absence assertion; 443 is asserted nowhere in the tier today).
  2. [BLOCK] The rerun probe greps a log string that no longer exists — the driver greps Flag matches (tests/uci-defaults-rootfs_test.sh:433), deleted on main by fix(setup): compare the setup version marker by order, so a roll-back stops re-running full setup #578 (75bc323, ancestor of this head), which replaced it with Setup branch $BRANCH — $REASON (now :2048). So every topology reports FAIL: rerun re-ran full setup even though the verify path was taken (14 failures). Three sibling suites already use the boundary anchor Setup branch VERIFY([[:space:]]|$) (uci-defaults-same-version-allowlist_test.sh:253, tests/packaging/guest-client-isolation_test.sh:268, tests/packaging/admin-board-requires-credential_test.sh:257) — use the same one, and keep the property that it does not match VERIFY_REPAIR.
  3. [BLOCK] The CHANGELOG entry ships a wrong assertion count — CHANGELOG.md:1184 (added by this PR) says "134 assertions in ~2 minutes"; the tier actually emits ~470 assertion results (the identical blob measured 470 passed / 0 failed at 3456de9c; at this head 288+182=470, exit 1). The figure never matched this test blob's output — same for the PR body's "134 passed, 0 failed" verification section. Please refresh both when the tier is green again.
  4. [RISK] Carried from the 2026-09-24 review, still open — the binding assertions still skip rather than fail when a band's radio exists but its tollgate_*_open section is missing (tests/uci-defaults-rootfs_test.sh:373-374, [ -n "$S2G" ] && check …); a neutered setup stays green on 6 of 7 topologies today. Your restore note already concedes this applies.

One observation worth its own line: .github/workflows/test.yml enumerates its test invocations explicitly (no glob), so this tier is referenced by no workflow — the red state above could not have been caught automatically. Wherever CI wiring lands (#457 is still open), this file needs a line.

Verified clean at this head: bash -n, shellcheck -S warning, the skip path (exit 0), gitleaks on the diff, and the offline siblings (band 22/0, same-version-allowlist 51/0, nodogsplash-convergence 29/0). No CI result exists for this head (ngit carries none; the GHA workflows are active but wedged). The tier's design remains right — fixtures, topology coverage and the driver structure are good; both root-cause fixes are confined to the assert_allowlist port loop and the rerun anchor. Fix 1+2+3 (and ideally 4), rerun (expect ~470 green), and this lands.

@felixfelix-bot felixfelix-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES REQUESTED at head 248aae6 — see the review comment for details and evidence: the tier is empirically red at its own head (288 passed / 182 failed, exit 1, executed on a docker host at this head) from two rebase-imported contract drifts: (1) the allow-list loop at tests/uci-defaults-rootfs_test.sh:397 asserts presence of 8080/8090/8443, which the writer on main now actively removes (99-tollgate-setup:1449-1456) — mirror the offline sibling's JOURNEY/ADMIN split; (2) the rerun probe at :433 greps 'Flag matches', deleted by #578 — use the boundary anchor 'Setup branch VERIFY([[:space:]]|$)' like the three sibling suites; plus (3) CHANGELOG.md:1184 ships '134 assertions' — the real count is ~470; and (4, carried) the binding assertions at :373-374 skip instead of fail when the AP section is missing. Design is right; all fixes are confined to the driver heredoc and the CHANGELOG figure.

Amperstrand added 3 commits October 1, 2026 09:31
The offline uci-defaults tests drive the setup script's functions
through a fake uci: fast, but blind to real-uci semantics and
GNU-vs-BusyBox differences — a fake-uci matrix once passed what real
uci then failed on private-network setup.

Add tests/uci-defaults-rootfs_test.sh: it runs the real script
end-to-end inside openwrt/rootfs containers (BusyBox ash, the real uci
binary) against seven fixture wireless topologies with authored band
ground truth, on both the 24.10 (opkg) and 25.12-snapshot (apk)
images, including the same-version verify path. The rootfs image ships
without /etc/config defaults, so the driver seeds minimal ones — which
incidentally exercises the script's designed no-LAN guard. Skips with
exit 0 where no docker exists, so it can ride runners without
container support (#521).
…tract

The wave-3 review's follow-up: the tier ran 134/0 yet a planted
#516-class regression still passed — zero nodogsplash allow-list
assertions, the exact class the tier exists to catch.

assert_allowlist() now runs after both setup paths (full setup and the
same-version rerun — the rerun being the duplicate-class witness):
for each of the six contracted entries (2121, 8080, 2050, 2051, 8090,
8443) it asserts presence via the same anchored whole-field match as
the writer's uci_list_has_port (:8443 cannot satisfy a :443-class
check) and asserts the entry appears exactly once (occurrence-counted
with grep -o | wc -l — grep -c counts lines, which is always 1 on a
single-line list and would never see a duplicate).

Also hardens a pre-existing SC2164 shellcheck warning in the same file
(cd without || exit).

Verified on real OpenWrt userspace (openwrt/rootfs:x86-64-openwrt-24.10,
7 topologies): 235 passed, 0 failed — up from 134, +12 assertions per
topology (6 presence + 6 exactly-once) after each of the two setup
paths. Synthetic dup/trip/absent probes: counts 2/3/0 as expected.
… narrowed allow-list contract

Review response (2026-09-30 changes-requested at 248aae6, plus the two
standing items from the 2026-09-26/27 passes):

assert_bindings(): a band whose radio exists but whose tollgate_*_open
section is missing now FAILs instead of silently skipping — the
#103/#173 reinstall class. Previously a fully-neutered AP setup stayed
green on 6 of 7 topologies.

assert_allowlist(): mirrors the contract main now ships (and the offline
sibling's JOURNEY_PORTS/ADMIN_PORTS split) instead of the retired one.
Journey ports 2121/2050/2051 asserted present exactly once; admin
surfaces 8080/443/8090/8443 asserted ABSENT from the pre-auth list —
this supersedes the earlier "add 443" item: main's writer removes 443
(operator-lockout entry), so the tier asserts its absence. 443 is now
asserted on every topology, both setup paths.

Rerun probe: "Flag matches" no longer exists (#578 replaced the branch
log with "Setup branch $BRANCH — $REASON"). The raw script this
container runs is never packaged, so its recorded marker is unorderable
and the same-version rerun legitimately lands on VERIFY_REPAIR, not
plain VERIFY; the assertion accepts both verify-family tokens with a
boundary after the token (never matching a hypothetical other verdict)
and fails on FULL or a missing verdict line.

CHANGELOG: the entry counted 134 assertions; the tier emits 414 — and
it now names the binding and allow-list contracts it asserts. PR link
(#530) added per house style.

Verified on real OpenWrt userspace, both images (24.10 + 25.12-SNAPSHOT)
x seven topologies: 414 passed, 0 failed, exit 0 (~20 s warm; skip path
exit 0 without docker). Discrimination controls, one image: neutered AP
creation fails all seven topologies (was green on 6 of 7); a writer that
re-admits allow tcp port 443 fails all seven topologies. Offline
siblings re-run green: band 22/0, same-version-allowlist 51/0,
nodogsplash-convergence 29/0, private-subnet 12/0. bash -n and
shellcheck -S warning clean. No Go files touched.
@Amperstrand
Amperstrand force-pushed the rootfs-validation-tier branch from 248aae6 to 3f254b9 Compare October 1, 2026 07:38
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Fix-and-rebase for the standing review items — new head 3f254b91 (rebased onto current main 74cd6f99, which moved past afc86b3f via #624/#625; CHANGELOG applied clean, one stray double blank line from the prior rebase normalized).

Resolutions, per finding:

  1. Binding assertions skip → FAIL (standing item, 2026-09-24 RISK, Bug: TollGate AP interfaces are not created when the setup flag already exists #103/AP setup is non-recoverable if sections are missing on reinstall/upgrade (real fix for #103) #173 class): assert_bindings no longer guards the band checks with [ -n "$S2G" ] &&. When a band's radio exists but its tollgate_2g_open/tollgate_5g_open section is missing, the driver now emits FAIL: <band> radio <X> exists but tollgate_<band>_open AP section missing — after both the full setup and the rerun. Control: the reviewer's neutered-setup_band_ap mutant, which previously stayed green on 6 of 7 topologies, now fails all seven (24.10 image: 181 passed / 26 failed, exit 1).
  2. The seventh port (443) is now asserted — as ABSENCE, per the 2026-09-30 supersession. assert_allowlist mirrors the offline sibling's split (uci-defaults-same-version-allowlist_test.sh): journey ports 2121/2050/2051 present exactly once (occurrence-counted, duplicate class uci duplicate-check regex misses quoted list rendering — setup reruns accumulate NDS rules (from #513) #516), admin surfaces 8080/443/8090/8443 asserted absent — main's writer now removes them, so presence assertions would encode the retired contract. Control: a mutant writer that re-adds allow tcp port 443 after the convergence del_lists fails all seven topologies (193 passed / 14 failed, exit 1). Note: an insert before the convergence step is healed by the script's own del_list re-assertion — the tier's assertion targets the unhealed state, correctly.
  3. Rerun probe re-anchored to the fix(setup): compare the setup version marker by order, so a roll-back stops re-running full setup #578 log vocabulary, with one environment-driven deviation from the suggested anchor. In-container evidence: the raw, unpackaged script records an unorderable marker, so the same-version rerun legitimately lands on VERIFY_REPAIR (Setup branch VERIFY_REPAIR — not an orderable release marker: recorded=unsubstituted… relation UNORDERABLE), never plain VERIFY — the bare-VERIFY sibling anchor cannot match this environment. The assertion therefore accepts both verify-family tokens with a boundary after the token — grep -qE "Setup branch (VERIFY|VERIFY_REPAIR)([[:space:]]|\$)" — and fails on FULL or a missing verdict line, which is the script's own documented taxonomy ("everything that is not FULL is the verify/repair path"). The 14 rerun re-ran full setup failures are gone.
  4. CHANGELOG refreshed: count corrected 134 → 414, the entry now names the binding and allow-list contracts the tier asserts, and the ([#530](...)) link is added per house style. The PR body's stale "134 passed" verification section describes the pre-stability-assertion blob and should be read as superseded by the numbers below (body edits were outside this session's write allowance).

Verification (this run, at 3f254b91, docker host):

  • bash tests/uci-defaults-rootfs_test.sh — 414 passed, 0 failed, exit 0 (both images x86-64-openwrt-24.10 + x86-64-25.12-SNAPSHOT, all seven topologies; ~20 s with warm images).
  • Discrimination controls (24.10): neutered AP creation → fails 7/7 topologies; :443 re-admission → fails 7/7 topologies.
  • Offline siblings: band 22/0, same-version-allowlist 51/0, nodogsplash-convergence 29/0, private-subnet 12/0 — all exit 0.
  • bash -n + shellcheck -S warning clean; skip path (no docker) exits 0. No Go files touched, so no Go battery.

The tier's CI wiring (#457) remains the known open follow-up — no workflow invokes this file yet.

@c03rad0r
c03rad0r self-requested a review October 6, 2026 18:28
@felixfelix-bot

Copy link
Copy Markdown
Contributor

Rebased onto main @ 977fa96f (35 commits) — branch pushed to felixfelix-bot/tollgate-module-basic-go as pr/rootfs-validation-tier-rebased, head f686768b.

What conflicted: CHANGELOG.md only, in ### Changed / Internal — main's new #556 bullet (packaging/local-build-ipk.sh refuses to build without staged portal bundles) landed exactly where this PR's rootfs-tier bullet is added. Resolved by keeping both, main's bullet unchanged and this PR's bullet untouched; the same hunk collided on commits 1 and 3 of 3, both resolved the same way. tests/uci-defaults-rootfs_test.sh did not conflict.

Nothing was rewritten but history: the replayed script is byte-identical to the pre-rebase head (git diff upstream/rootfs-validation-tier HEAD -- tests/uci-defaults-rootfs_test.sh → empty), and git diff of the rebased tree against the pre-rebase head shows only main's own commits' content. Post-rebase diff vs main: 2 files, +529.

Verified on the rebased tree:

  • tests/uci-defaults-rootfs_test.sh run for real — docker, both images (x86-64-openwrt-24.10, x86-64-25.12-SNAPSHOT) × 7 topologies against current main's 99-tollgate-setup: 414 passed, 0 failed, exit 0.
  • shellcheck -S warning clean, bash -n clean.

To move this PR's head (whoever has write on the org branch — the branch head lives in OpenTollGate/tollgate-module-basic-go, which is not writable from the felixfelix-bot account):

git fetch https://github.com/felixfelix-bot/tollgate-module-basic-go pr/rootfs-validation-tier-rebased
git checkout rootfs-validation-tier
git reset --hard FETCH_HEAD
git push --force-with-lease origin rootfs-validation-tier

A force-push may dismiss the existing approval if the branch protection dismisses stale reviews — re-approving is the price of a clean rebase. If you would rather not wait on the author, say so and I will open the rebased version as a replacement PR against main instead (it would need a fresh review, and this PR would close as superseded).

@felixfelix-bot

Copy link
Copy Markdown
Contributor

Rebased content offered as #671 (base main, head pr/rootfs-validation-tier-rebased @ f686768b) — one commit, 2 files, mergeable, test blob byte-identical to this PR's head, tier re-run green (414 passed / 0 failed on both rootfs images × 7 topologies). I cannot move this PR's own head: the branch lives in this repo and felixfelix-bot has no write access. So either merge #671 and close this one as superseded, or force-push the same commit here:

git fetch https://github.com/felixfelix-bot/tollgate-module-basic-go pr/rootfs-validation-tier-rebased
git checkout rootfs-validation-tier
git reset --hard FETCH_HEAD
git push --force-with-lease origin rootfs-validation-tier

Both routes land the same single commit. I will close #671 myself if you prefer the force-push.

c03rad0r pushed a commit that referenced this pull request Oct 6, 2026
…rebase of #530 (#671)

* test(packaging): rootfs-container tier for uci-defaults validation

The offline uci-defaults tests drive the setup script's functions
through a fake uci: fast, but blind to real-uci semantics and
GNU-vs-BusyBox differences — a fake-uci matrix once passed what real
uci then failed on private-network setup.

Add tests/uci-defaults-rootfs_test.sh: it runs the real script
end-to-end inside openwrt/rootfs containers (BusyBox ash, the real uci
binary) against seven fixture wireless topologies with authored band
ground truth, on both the 24.10 (opkg) and 25.12-snapshot (apk)
images, including the same-version verify path. The rootfs image ships
without /etc/config defaults, so the driver seeds minimal ones — which
incidentally exercises the script's designed no-LAN guard. Skips with
exit 0 where no docker exists, so it can ride runners without
container support (#521).

* test(packaging): rootfs tier asserts the NDS allow-list stability contract

The wave-3 review's follow-up: the tier ran 134/0 yet a planted
#516-class regression still passed — zero nodogsplash allow-list
assertions, the exact class the tier exists to catch.

assert_allowlist() now runs after both setup paths (full setup and the
same-version rerun — the rerun being the duplicate-class witness):
for each of the six contracted entries (2121, 8080, 2050, 2051, 8090,
8443) it asserts presence via the same anchored whole-field match as
the writer's uci_list_has_port (:8443 cannot satisfy a :443-class
check) and asserts the entry appears exactly once (occurrence-counted
with grep -o | wc -l — grep -c counts lines, which is always 1 on a
single-line list and would never see a duplicate).

Also hardens a pre-existing SC2164 shellcheck warning in the same file
(cd without || exit).

Verified on real OpenWrt userspace (openwrt/rootfs:x86-64-openwrt-24.10,
7 topologies): 235 passed, 0 failed — up from 134, +12 assertions per
topology (6 presence + 6 exactly-once) after each of the two setup
paths. Synthetic dup/trip/absent probes: counts 2/3/0 as expected.

* test(packaging): rootfs tier fails on absent AP sections, asserts the narrowed allow-list contract

Review response (2026-09-30 changes-requested at 248aae6, plus the two
standing items from the 2026-09-26/27 passes):

assert_bindings(): a band whose radio exists but whose tollgate_*_open
section is missing now FAILs instead of silently skipping — the
green on 6 of 7 topologies.

assert_allowlist(): mirrors the contract main now ships (and the offline
sibling's JOURNEY_PORTS/ADMIN_PORTS split) instead of the retired one.
Journey ports 2121/2050/2051 asserted present exactly once; admin
surfaces 8080/443/8090/8443 asserted ABSENT from the pre-auth list —
this supersedes the earlier "add 443" item: main's writer removes 443
(operator-lockout entry), so the tier asserts its absence. 443 is now
asserted on every topology, both setup paths.

Rerun probe: "Flag matches" no longer exists (#578 replaced the branch
log with "Setup branch $BRANCH — $REASON"). The raw script this
container runs is never packaged, so its recorded marker is unorderable
and the same-version rerun legitimately lands on VERIFY_REPAIR, not
plain VERIFY; the assertion accepts both verify-family tokens with a
boundary after the token (never matching a hypothetical other verdict)
and fails on FULL or a missing verdict line.

CHANGELOG: the entry counted 134 assertions; the tier emits 414 — and
it now names the binding and allow-list contracts it asserts. PR link
(#530) added per house style.

Verified on real OpenWrt userspace, both images (24.10 + 25.12-SNAPSHOT)
x seven topologies: 414 passed, 0 failed, exit 0 (~20 s warm; skip path
exit 0 without docker). Discrimination controls, one image: neutered AP
creation fails all seven topologies (was green on 6 of 7); a writer that
re-admits allow tcp port 443 fails all seven topologies. Offline
siblings re-run green: band 22/0, same-version-allowlist 51/0,
nodogsplash-convergence 29/0, private-subnet 12/0. bash -n and
shellcheck -S warning clean. No Go files touched.

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
@felixfelix-bot

Copy link
Copy Markdown
Contributor

Closing as landed...

@felixfelix-bot

Copy link
Copy Markdown
Contributor

There is nothing left to resolve here: this PR's content is already in main, via the rebase that landed as #671 (squash 0dbae452, merged 18:45Z).

Proof against the current tree (6247731f):

  • tests/uci-defaults-rootfs_test.sh — this branch's 3f254b91 and main carry the same blob (d53d93bf); 513 lines, byte-identical.
  • CHANGELOG.md — every line this branch adds (15 of them, the rootfs-tier entry) is present in main; main has the Rootfs-container tier for uci-defaults validation bullet exactly once. The only delta is the blank separator line (main's section already supplies it) — that is the +15 here vs +16 in the landed commit.
  • Nothing else: the branch's full diff against its merge-base is those two files, and both are accounted for above.

The CONFLICTING flag is therefore cosmetic — GitHub is comparing a base from before the landing. A merge would try to re-apply a change that is already there.

Fresh evidence on today's main: the tier itself re-run in a clean worktree of 6247731f — 414 passed, 0 failed, exit 0 — across x86-64-openwrt-24.10 and x86-64-25.12-SNAPSHOT × the seven fixture topologies. Note packaging/files/etc/uci-defaults/99-tollgate-setup has not changed since the landing, so this run exercises the same script in the same surroundings.

Recommended action: close this PR as landed. (I'd close it myself, but felixfelix-bot lacks the permission — ClosePullRequest returns a permissions error.)

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Superseded by #671 — same rootfs-container tier for uci-defaults validation, rebased onto current main and merged there. Closing to keep the queue clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants