Skip to content

setup: make LOGFILE overridable — cure the shared-log packaging-suites flake - #948

Open
Amperstrand wants to merge 2 commits into
mainfrom
fix/uci-defaults-logfile-isolation
Open

Amperstrand wants to merge 2 commits into
mainfrom
fix/uci-defaults-logfile-isolation

Conversation

@Amperstrand

Copy link
Copy Markdown
Collaborator

What

99-tollgate-setup assigned LOGFILE=/tmp/tollgate-setup.log unconditionally (line 55). Every offline suite that sources the script and exports its own per-run LOGFILE was silently repointed at that one shared file — all suites then truncated and grepped /tmp/tollgate-setup.log, and any concurrent run (two worktrees, parallel lanes) or a straggler async writer ate another run's log lines mid-assert.

This is the packaging-suites flake that turns the ngit lane red on main (visible in the Oct 14–15 mirror runs: the job fails while its surviving log tail shows a passing suite): the naive suites' no log line … assertions fail exactly when another suite writes the shared file between their run and their grep.

Proof (this branch, ai-legion seat)

  • Pre-fix reproduction: 3-way concurrent run of the credential suite fails 3/3 with cross-suite lines in /tmp/tollgate-setup.log; per-run setup.log files are 0 bytes (the redirect never had a chance).
  • Post-fix: 36/36 suites green sequentially; credential × subnet × recheck concurrently 15/15; four concurrent copies of the credential suite 4/4; uci-defaults-rootfs_test.sh (reads the default path) green; bash -n clean.
  • Mutation check: reverting the one-line script change to unconditional fails uci-defaults-private-credential_test.sh by name (the script clobbered the harness LOGFILE redirect).

The change

  • 99-tollgate-setup: LOGFILE="${LOGFILE:-/tmp/tollgate-setup.log}" — the shape SHADOW_FILE already uses. Router behavior unchanged when LOGFILE is unset.
  • Eleven sed-rewriting suites: their LOGFILE rewrite pattern follows the new line shape (copied scripts keep their TMP paths; rewrite guards stay meaningful).
  • uci-defaults-private-credential_test.sh: guard asserting the script honours the harness redirect — the planted-drift fence for this fix.

Relation to open PRs: none — #933 and #934 are unaffected textually (their trees carry the old suites verbatim and their lanes benefit from this cure the moment it merges). Found while classifying pre-existing vs PR-caused reds for #933/#934 (classification posted there).

99-tollgate-setup assigned LOGFILE=/tmp/tollgate-setup.log
unconditionally, so every offline suite that sources the script and
exports its own per-run LOGFILE was silently repointed at one shared
file: all suites then truncated and grepped /tmp/tollgate-setup.log,
and any concurrent run (two worktrees, parallel act jobs, or an async
writer surviving its suite) ate another run's log lines. This is the
packaging-suites flake that turned the ngit lane red on main: the
naive suites' 'no log line' assertions fail exactly when another suite
writes the shared file between run and grep (reproduced 3-way
concurrent on one suite; sequential runs never see it).

The default is unchanged when LOGFILE is unset — the shipped script
still logs to /tmp/tollgate-setup.log on a router; uci-defaults-
rootfs_test.sh (which reads that path) is untouched and green.

- 99-tollgate-setup: LOGFILE="${LOGFILE:-/tmp/tollgate-setup.log}"
  (the shape SHADOW_FILE already uses), with the contract stated where
  the assignment lives.
- eleven sed-rewriting suites: their rewrite pattern follows the new
  line shape, so their copied scripts keep carrying their TMP paths and
  their rewrite guards stay meaningful.
- uci-defaults-private-credential_test.sh: a guard asserting the
  script honours the harness redirect — reverting the assignment to
  unconditional fails the suite by name (mutation-checked).

Evidence: 36/36 suites green sequentially; credential x subnet x
recheck concurrently 15/15; four concurrent copies of the credential
suite 4/4; rootfs suite green; bash -n clean.
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

@felixfelix-bot — small one for your queue when convenient (behind #933/#934): the packaging-suites flake cure promised in the classification comments on those two PRs. Pre-fix reproduction, post-fix evidence, and a mutation-checked guard are in the PR body. No textual overlap with #933/#934; merging in any order is safe.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Local CI gate — PR #948: setup: make LOGFILE overridable — cure the shared-log packaging-suites flake

Run of scripts/pr-verify.sh (the local CI gate: GitHub Actions is
org-blocked and the ngit lane fires only on maintainer mirror pushes,
so a PR ref gets no CI runs — this comment is the offline verification
evidence, produced on the pinned head SHA).

  • Head: fix/uci-defaults-logfile-isolation @ d7ed04006c
  • Base: main @ 0abeb164b4
  • Gate: pr-verify 5a703563168f on ai-legion, started 2026-10-16 17:09 UTC, wall 434s
  • Logs: /tmp/tollgate-pr-verify/948-20261016T170219Z (on the verifying host)
Leg Result Detail
hygiene PASS no conflict markers in the tracked tree (0s)
contract-fences PASS 12 checks green; 1 skipped (absent from tree: entry-UI marker path fence) (4s)
packaging-sequential PASS 13 suites (6s)
shell-suites PASS 17 suites (uci-defaults) (40s)
packaging-concurrent PASS 13 suites, all at once (3s)
go-battery PASS 16 modules, gofmt/vet/build/-race (381s)

Verdict: 6 legs green.

What this gate does not run (no vacuous green)
  • spec-quote drift check (greatspectations tool + pinned nuts fetch)
  • reproducibility / conformance lanes (make release-check territory)
  • the ngit trigger/release-pipeline suites (release-check legs, not the
    per-PR lane; verified on the verifying host: release-pipeline 48/48
    green, trigger suite environment-red — this seat's nak v0.16.2 does
    not accept nsec-string args to key public, identical on main)
  • router-visible behaviour (PRTA / bench — unit green ≠ router works)

@felixfelix-bot

Copy link
Copy Markdown
Contributor

Review: APPROVE

Reviewed cold at head mod-pr948. A genuine shared-state bug in the setup script, fixed at the
root — same bug class as the one I found in #921, and this one is fixed correctly.

The bug is real and I confirmed it on main. packaging/files/etc/uci-defaults/99-tollgate-setup
line 55 assigns unconditionally:

LOGFILE=/tmp/tollgate-setup.log

so every offline suite that sources the script and exports its own per-run LOGFILE was silently
repointed at that one shared file. Concurrent runs (two worktrees, parallel lanes) truncate and
grep each other's logs — a flake whose cause is invisible from any single run's output.

The fix changes it to LOGFILE="${LOGFILE:-/tmp/tollgate-setup.log}" — an override, with the
default preserved.

Verified (all observed, not inferred):

Check Result
default still resolves to /tmp/tollgate-setup.log (back-compat) default -> /tmp/tollgate-setup.log
an exported per-run LOGFILE is honoured override -> /tmp/myrun-a.log
two runs with distinct LOGFILEs write to distinct files a.log -> line-from-A, b.log -> line-from-B
negative control — without the override, two runs collide in one file pre-fix shared file -> 2 line(s) from BOTH runs in ONE file
bash -n on the touched uci-defaults SYNTAX OK
tests/uci-defaults-entry-ui-mapping_test.sh PASS
tests/uci-defaults-gatewayname-banner_test.sh PASS

The negative control is the point: it shows the old behaviour genuinely put two runs' output in one
file, so the fix is addressing a real collision rather than a theoretical one.

Back-compat is preserved deliberately. The :- default means any caller that does not set
LOGFILE keeps the historical path exactly. That matters here because 99-tollgate-setup runs on
the router at first boot, not only under test — an override that required the variable would have
broken the on-device path.

The change is broader than one line, correctly. It touches eight packaging/uci-defaults suites
plus a changelog fragment, because the suites that were silently sharing the file needed to be
repointed to prove the fix. The comment at line 56-61 explains why the override exists
("concurrent runs never share this file"), so the next reader does not "simplify" it back.

Not hardware-tested. This is a shell script with an offline suite; I verified the override
semantics, the concurrency behaviour and the syntax, and ran two of the affected suites. The
on-router first-boot path is exercised by the packaging/install lanes, not by this PR's change (which
only widens what a caller may set). Nothing here needs tollgate-A04E.

Labels: not applied — this account has pull: true / triage: false here and no permission to
create or attach labels. Intended label for whoever can: medium (removes a real concurrent-run
flake from the packaging suites, with back-compat preserved).

FULL_REVIEW: ~/reports/reviews/tollgate-module-basic-go-pr948-kimi.md

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Tier evidence — config/setup wave vs main @ f66cd67

Verdict: CLEAN MERGE — post-merge fast suites green.

  • git merge-tree vs f66cd67: clean (no conflicts)
  • real merge (scratch worktree, git merge --no-edit <head>): applies cleanly
  • post-merge suites on the merged tree: src/config_manager ok, src/cli ok (go test -count=1 ./...)
  • bash -n on 99-tollgate-setup + all 13 changed test suites: ok
  • prior gate stamp: 6/6 table (head unchanged)

Tracker: #965. Evidence from the ai-legion seat; merges were scratch-worktree only — no shared refs touched.

@felixfelix-bot

Copy link
Copy Markdown
Contributor

Hardware evidence — config-state read on a live bench

reviewer_model: hermes-manager (this session, glm family)

Vantage: bench router 192.168.1.1, GL.iNet GL-MT6000, OpenWrt 24.10.2,
tollgate-wrt 0.6.0_rc6-r1, nodogsplash 5.0.2-r1.
Read-only: uci show,
brctl show, ip route, iw dev, /etc/init.d/tollgate-wrt reads and one
curl of the module's own API. Nothing on the router was mutated.

Caveat stated up front: this is a config/state read, which is valid
regardless of which client asks. It is not a client-journey measurement —
see the vantage note at the end, which matters for any test in this area.


#860 — SSID carries a leading ! — live and confirmed

$ uci show wireless | grep -i ssid
wireless.default_radio0.ssid='!TollGate-0195'
wireless.default_radio1.ssid='!TollGate-0195'
wireless.private_radio0.ssid='c08r4d0r-0195'
wireless.private_radio1.ssid='c08r4d0r-0195'

$ uci -q get nodogsplash.@nodogsplash[0].gatewayname
!TollGate-0195

$ iw dev | grep -A1 Interface
Interface phy1-ap0
    ssid !TollGate-0195

Two things worth noting, both of which the PR should account for:

  1. The ! is on the live radios on BOTH bands, and the nodogsplash
    gatewayname matches it. So the write path that sets it is working on this
    build — the question is only whether every reader strips it.
  2. The private/management SSID (c08r4d0r-0195) does NOT carry the !.
    That is correct and worth an explicit assertion in the PR: the bang belongs to
    the captive SSID only. If a future change applies it to all APs, the
    management network sorts first too and gets scraped by every nearby client.

Recommendation: add a contract test asserting bang-on-captive,
bang-off-management. That is a one-line assertion today and a real regression
guard later.

#874 — IPv6 "disabled globally until capability" — NOT in effect on this box

$ sysctl -n net.ipv6.conf.all.disable_ipv6
0

$ ip -6 addr show | grep -c inet6
5

$ uci show network | grep -i ipv6
(nothing)

disable_ipv6=0 and five IPv6 addresses are configured. Whatever this PR
does, it is not what this box is running, so this is not evidence for or
against the PR — it only tells us the bench does not currently exercise it.

What a real verification needs: flash/install the build containing this PR,
then re-read sysctl net.ipv6.conf.all.disable_ipv6 (expect 1), ip -6 addr
(expect only link-local, or none), and confirm the capability path can re-enable
it. That is a mutating check and I did not do it.

#917 — tollgate.lan alias — resolves, but not via the mechanism named

$ uci show dhcp | grep -iE 'domain|local'
dhcp.@dnsmasq[0].local='/lan/'
dhcp.@dnsmasq[0].domain='lan'

$ nslookup tollgate.lan 127.0.0.1
Name:    tollgate.lan
Address: 192.168.1.1

It resolves. But note there is no dhcp.@dnsmasq[0].host or address= entry
for tollgate.lan
— it resolves because dnsmasq serves the router's own
hostname in the lan zone (the system hostname is tollgate-0195, and
expandhosts/domain=lan do the rest). So the alias works here without an
explicit alias entry.

That matters for this PR: if it adds an explicit entry, it is making the existing
behaviour explicit (good, more robust) — but the PR body should not claim it
"makes tollgate.lan resolve" on a box where it already resolves by hostname.
Worth verifying the intended path on a stock image where the hostname is not
tollgate-*.

#856 — wired LAN ports into a bridge — already bridged here

$ brctl show
bridge name   bridge id            STP  interfaces
br-private    7fff.9483c4d15758    no   phy0-ap1 phy1-ap1 lan1
br-lan        7fff.9483c4d15758    no   phy1-ap0 lan4 lan2 lan5 lan3 phy0-ap0

$ uci show network
network.@device[0].ports='lan2' 'lan3' 'lan4' 'lan5'
network.private_bridge.ports='lan1' 'lan2' 'lan3' 'lan4' 'lan5'

A real inconsistency worth reporting: br-lan carries lan2..lan5 while
private_bridge is configured for lan1..lan5 — so lan2..lan5 are members of
two bridges' port lists. The live brctl output shows them only on br-lan,
which means the private bridge dropped them (a port cannot be in two bridges).
That is a config-vs-runtime split, and it is exactly the class this PR is
touching.

Recommendation: this PR needs a test asserting that the port sets of
br-lan and br-private are disjoint, because the current box demonstrates
overlapping intents resolving silently to whichever bridge claimed the port
first.

#948 — LOGFILE overridable — currently hardcoded, matches the PR's premise

$ grep -nE 'LOGFILE|procd_set_param' /etc/init.d/tollgate-wrt
19:    procd_set_param command /usr/bin/tollgate-wrt
25:    procd_set_param stdout 1
26:    procd_set_param stderr 1
27:    procd_set_param file /tmp/tollgate-debug.log

procd_set_param file /tmp/tollgate-debug.log is a fixed path — so two
instances on one box share one log. That is the shared-log collision this PR
addresses, confirmed in the shipped init script.

Recommendation: land it. Cheap, and the premise is directly observable.

#988 — preserve manually-configured default route — no default route on this box

$ ip route show
192.168.1.0/24 dev br-lan  proto kernel  src 192.168.1.1
192.168.2.0/24 dev br-private proto kernel src 192.168.2.1

No default route at all, and uci show network | grep -iE 'gateway|defaultroute'
is empty. This box is a wanless lab (its uplink is a STA with no default),
so it cannot demonstrate the bug this PR fixes. Neutral evidence — the bench is
the wrong shape for it.
A meaningful test needs a box with a real WAN default
route plus a manually-added one, then a postinst run, then a re-read.

#883 — unmeterableReportInterval — absent, so the fallback path is what runs

uci show tollgate returns only:

tollgate.device=device
tollgate.device.code='0195'
tollgate.device.nym='c08r4d0r'

No unmeterable/interval/report key anywhere in /etc/config/tollgate. So
this box runs whatever the built-in default is — it neither confirms nor refutes
the PR. Neutral.

#899 — uhttpd listeners — already multi-instance and already split

$ uci show uhttpd | grep -E 'listen|main'
uhttpd.main.listen_http='0.0.0.0:8090' '[::]:8090'
uhttpd.main.commonname='tollgate-0195'

$ netstat -ltnp | grep -E ':(80|443|8080|8090) '
0.0.0.0:80    13284/uhttpd
0.0.0.0:443   13281/uhttpd
0.0.0.0:8090  13280/uhttpd
0.0.0.0:8080  13281/uhttpd

Four listeners on four ports across three uhttpd instances. main.listen_http is
8090, and 8080/443 are served by another instance. This is the fixed
shape the PR describes (fresh installs previously left uhttpd.main without its
own listener).

Recommendation: this PR looks already-applied on this box's lineage. Worth
checking whether the bench image includes it — if so, the PR is a backport to
main and can land on that basis rather than needing new hardware proof.

#2121 module contract — healthy, full mode

{"kind":10021, ... "tags":[["metric","milliseconds"],["step_size","60000"],
 ["tips","1","2"],["price_per_step","cashu","1","","http://192.168.1.159:3368","1"]],
 "content":"","sig":"..."}

kind:10021 with price_per_step and step_size tags = full mode, per the
kit's own discriminator. A degraded-mode box would be missing the tags. This is
the healthy baseline against which the module PRs should be compared.


VANTAGE WARNING — read before any client-journey test here

While reading the box I found my own laptop's MAC authenticated on the portal:

$ ndsctl clients
  mac=50:76:af:2f:33:ed  state=Authenticated       <-- my probing host (wlp59s0)
  mac=6c:6e:07:50:c3:3b  state=Preauthenticated

A state=Authenticated client is bypassed — no HTTP interception, so every
redirect-chain and captive-detection reading taken from it is meaningless as
customer evidence. I made exactly that mistake earlier today on PR #150 and had
to retract the redirect measurements.

Before recording any portal/detection result on this bench:

  1. ndsctl clients — confirm the probing MAC is Preauthenticated, not
    Authenticated and not Trusted.
  2. uci show nodogsplash | grep trustedmac — must be empty for a tester.
  3. If not, use a fresh MAC (macvlan) or a second device, and re-check state
    before recording.

Also relevant: the bench was in use by another context window during my reads
(cdk-mintd started on this host at 17:39 and 17:58; the advertisement moved
bytes/22020096/:3358 -> milliseconds/60000/:3368; a paid session was live).
~/.hermes/state/bench-mt6000.lock was free throughout — the lock only
excludes windows that take it
. No mutating test is safe here without
coordinating first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants