[Amperstrand]

Summary

Slice 2 of the storage-degraded family (#583/#1035 — the storage half itself stays out of scope). Two defects in the degraded state machine kept a broken-wallet (with perfectly reachable mints) degraded for life and lying about why:

1. The recovery trigger is a one-shot latch. With mints REACHABLE and wallet init failing (the Wallet never initializes on jffs2-overlay devices (squashfs NOR targets): bbolt requires shared mmap; silent degraded mode + misleading "no reachable mints" #583/Wallet InitStorage fails with EINVAL on armv7 + overlayfs/jffs2: backend permanently degraded despite healthy mint #1035 storage class; any persistent wallet error), the tracker's onFirstReachable fires once, the upgrade attempt fails, hadReachableMint is set, and nothing ever fires again — degraded until a manual restart.
2. Every degraded stub hardcodes wallet not initialized: no reachable mints — while the mints probe 200 OK. Wallet never initializes on jffs2-overlay devices (squashfs NOR targets): bbolt requires shared mmap; silent degraded mode + misleading "no reachable mints" #583/Wallet InitStorage fails with EINVAL on armv7 + overlayfs/jffs2: backend permanently degraded despite healthy mint #1035 triage burned days on that misdirection (23 PRTA failures chased to one root cause).

What changes

• MerchantDegraded carries a degrade reason — set by both of New()'s degraded branches (no reachable mints vs wallet initialization failed: …), updated to the last failed attempt's error — and every API stub surfaces it (notInitializedError()). Empty default keeps the legacy text for direct constructions.
• WireWalletRetry(): a bounded, backing-off wallet-init retry loop (30 s initial → 5 min cap; skips its tick while the reachable set is empty — that class stays with the first-reachable trigger), armed idempotently wherever the trigger is wired (WireStartupDegraded, WireRecoveryTrigger) and in New()'s degraded branches. Single-flighted against the trigger through AttemptUpgrade's mutex; stopped on successful upgrade or Shutdown.
• New()'s two degraded branches replace their inline duplicate of the trigger wiring with WireRecoveryTrigger (identical behavior, one owner).
• onUpgrade is now guarded — the retry loop can be mid-attempt while the caller is still attaching the callback (data race caught by -race in this suite).

Fund-safety statement

No monetary operations added or reordered; the retry re-runs wallet construction only. A failed attempt leaves the degraded merchant serving exactly as before (its offline wallet was already shut down by the pre-existing upgrade flow). The loop cannot spin: backoff caps at the proactive cadence, and it never runs while no mint is reachable.

Evidence (ai-legion, go1.26.8; /tmp/opencode/tg-storage/evidence/go-battery-degraded-retry.log)

• make go-battery — 18 modules green (gofmt/vet/build/go test -race -count=1 -tags testenv per module).
• New tests (heal-within-window via the retry loop, never the 5-min trigger; honest-reason assertions; quiet-while-unreachable; single-upgrade). Retry-loop race hammered 5/5 clean under -race after the callback guard.
• Mutation checks: retry wiring removed → heal test fails stuck-degraded (10 s window); honest-reason reverted to the hardcoded string → reason test fails.
• Honest: unit level; the 30 s→5 min cadence and the reason text deserve one bench QEMU boot look before release (noted in the tracker row).

Independent of the #1063 → #908 → #1077 chain (different files; based on current main, pin v0.13.0 — the honest-keyset fixture id used survives the v0.14.x guard). OWNER-GATED: no self-merge.

[Amperstrand]

Review — verdict: BLOCKERS(1) + CONCERNS(1); disposition: request-changes

The honest-reason half of this PR is good and well-tested: the degrade reason is set by both of New()'s degraded branches, every API stub surfaces it, the unreachable-class keeps its meaning and its trigger, and the reason tests bite in both directions (the mutation checks in the body are real). The suite passes here (go test -race -count=1 -tags testenv ./... in src/merchant → ok, 350s; the three new tests green in 18s). But the retry-loop half does not do what the PR body claims in the PR's own target scenario — the bounded backoff never happens, because of how the retry path interacts with newFullMerchant's degraded-fallback return. I reproduced this empirically; evidence below.

Blocker — every "failed" wallet retry actually succeeds with a fresh degraded instance, cascading forever at the initial interval.

newFullMerchant's wallet-init-failure branch returns the degraded fallback with a nil error (merchant.go#L499-L513 — return deg, nil). On main this quirk was capped by the one-shot latch (the exact "degraded for life" bug this PR documents). With this PR's loop, AttemptUpgrade treats that fallback as a successful upgrade (merchant_degraded.go#L245-L266): m.upgraded = true, stop own loop, fire onUpgrade(newDegradedInstance) — and that new instance was wired with its own retry loop at construction (deg.WireRecoveryTrigger() at L512). Result on a persistently-broken wallet with reachable mints (the #583/#1035 class this PR targets): an infinite chain of degraded-merchant generations, each retrying at the initial interval — the 30 s→5 min backoff, the cap, and the single-flight story are all void; a router gets a new MerchantDegraded + wallet construction + mint keyset HTTP calls every ~30 s forever, flapping the operator surface with a fresh "degraded" banner per generation.

Reproduced with a never-healing reachable mint (scratch test, test-shrunk constants initial=100ms, max=200ms, timestamps from the mint server and the loop's own log): the loop's log prints every 100ms on every tick, forever — never 200 ms — and /v1/keys hits arrive in pairs every ~124 ms without end; with initial=25ms/max=100ms, 26 hits in 700 ms. That cadence is one generation per tick, not one backing-off loop. None of the existing tests catch this because the heal test heals via a genuinely-successful path and no test pins attempt counts or cadence — exactly the green-suite/dead-semantics class PR-REVIEW.md's mutation doctrine exists for.

Suggested fix (either closes it): make the wallet-fail branch of newFullMerchant return an error (and construct the degraded merchant explicitly at its call sites, as New() already does), or have AttemptUpgrade reject a *MerchantDegraded result and treat it as a failed attempt. Then add a cadence test: never-healing mint, shrunk constants, assert the attempt count over initial + 2·initial + max + slack is exactly the backoff schedule (my probe pattern, kept as a scratch example in the evidence below).

Concern — failed attempts of the transient classes kill the loop after one try.

AttemptUpgrade calls m.Shutdown() up front, and Shutdown unconditionally calls stopWalletRetry() (merchant_degraded.go#L273-L274), closing the retry channel. When newFullMerchant returns a real error (advertisement failure, mkdir failure — plausibly transient), the loop's next tick sees the closed channel and exits: one attempt, then the original stuck-degraded-for-life behavior returns for those classes. The wallet class masks this today only because of the blocker above (err is nil there). Suggested fix: stop the retry loop only on successful upgrade — e.g. drop the stopWalletRetry() from Shutdown (keep it in AttemptUpgrade's success path and wherever a full shutdown of a retired merchant happens), or use a non-stopping internal shutdown inside AttemptUpgrade.

Notes, not concerns: the macAddress→"" change visible in the two-dot diff on PurchaseSession's notice is a stale-base artifact — the branch sits on the pre-#968 stack line and does not touch that line; the merge-tree landing keeps main's "" (verified). onUpgrade's RW-lock guard is correct and needed. The changelog fragment is present and correctly shaped, but its "bounded backoff" claim inherits the blocker. Body's make go-battery/18-modules-green claim wasn't re-verified file-by-file; the merchant module (the only one touched) is green here.

evidence (all at head fc40332, worktree)

$ go test -race -count=1 -tags testenv ./...          # src/merchant
ok  	github.com/OpenTollGate/tollgate-module-basic-go/src/merchant	350.454s

# Scratch probe: never-healing mint (keysets 200 OK, keys garbage), reachable set non-empty,
# retry constants shrunk to initial=100ms max=200ms, 1.3s window; loop's own log captured:
$ go test -run TestScratchRetryLogV4 -v .
    retry-loop lines (10):
      ... Degraded wallet retry: ... (every 100ms while it fails)     x10  <- never 200ms
    attempt lines: 10

# Same probe, /v1/keys timestamps (ms since start) — pairs (2 calls per attempt), steady ~124ms:
    53 64 | 179 189 | 300 310 | 422 433 | 549 561 | 673 683 | 802 813 | 924 934 | 1053 1064 | 1176 1187 | 1298 1309
# Backoff would read: 100 | 200 | 200 | ...  (attempt starts ~53, 253, 453, ...) — not observed.

# With initial=25ms/max=100ms: 26 /v1/keys hits in 700ms — steady ~46ms cycle, no cap.

$ git merge-tree --write-tree --no-messages origin/main HEAD; echo rc=$?   # landing clean
rc=0
$ git show <tree>:src/merchant/merchant_degraded.go | grep -A1 service-unavailable
	"TollGate is initializing. No reachable mints. Please try again in a few minutes.", "")   <- main's #950 fix survives

— agent evidence pass (review-debt wave R1); not an approval.

[c03rad0r]

Technical review — head fc40332 (content approved; branch needs a rebase before it can merge).

This fixes a real stuck-state: the first-reachable trigger was a one-shot latch, so a wallet that failed to construct stayed degraded for life even with every mint reachable — and every stub then blamed "no reachable mints" while the probes said 200. Carrying an explicit degrade reason through both branches and surfacing it on every stub is the honest-reasoning half of the storage-degraded family, and the bounded retry (30 s → 5 min, single-flighted through AttemptUpgrade's mutex, skipped while the reachable set is empty) is the right shape. Guarding onUpgrade against the retry/callback race is a genuine -race find, not cleanup.

Verified at this head:

cd src/merchant && go test -tags testenv -count=1 -run 'Degraded|WalletRetry|Reason' ./...
ok  github.com/OpenTollGate/tollgate-module-basic-go/src/merchant  221.853s

The fund-safety statement is accurate: construction only, no monetary operation reordered, and a failed attempt leaves the merchant serving exactly as before.

The only reason this is not "merge now": the branch currently conflicts with main (CONFLICTING/DIRTY). Rebase it (the merchant files moved under #1108/#1117's storage-verdict work) and I will flip to an approval. The rebase should keep #1108's walletDirFor seam if #1080 lands first — walletDirFor is the same function this branch calls for construction, so order does not matter as long as the branch does not reintroduce filepath.Dir(...) at those sites.

Disposition: approve content; rebase required (I will re-approve on the rebased head).

[c03rad0r]

Rebased and hand-merged. Head f970bc9.

Main's storage-verdict work and this PR's retry machinery are now combined:

• walletInitErr (the storage-mmap class) is kept, and degradeReason is added beside it; every degraded stub goes through the reason-aware notInitializedError().
• the bounded retry (30s → 5min, single-flighted, quiet while no mint is reachable) is wired through WireRecoveryTrigger in both New() degraded branches;
• the upgrade hand-off keeps both semantics: upgraded = true + stopWalletRetry() + the getOnUpgrade guard, and the no-consumer Shutdown fallback that prevents leaked background work;
• the onUpgrade mutex guard is retained (the retry can read it while the caller is attaching).

Verified: go build ./...; the three retry/reason tests (TestDegradedWalletRetryHealsBrokenWallet, TestDegradedReasonIsHonestWhenWalletFailsWithReachableMints, TestDegradedWalletRetryQuietWhileNoMintReachable) pass, plus the targeted -race subset. (A broad -run Degraded selection hits the known merchant flake family, not this change.)

[Amperstrand]

Local CI gate — PR #1078: fix(merchant): degraded mode retries wallet init (bounded backoff) and surfaces the honest degrade reason

Run of scripts/pr-verify.sh (the local CI gate: GitHub Actions is
org-blocked and the ngit lane fires only on maintainer mirror pushes,
so a PR ref gets no CI runs — this comment is the offline verification
evidence, produced on the pinned head SHA).

• Head: fix/degraded-stuck-retry @ f970bc9cb5
• Base: main @ 92a3c18228
• Gate: pr-verify 6f304ed0e72d on ai-legion, started 2026-10-31 21:03 UTC, wall 90s
• Logs: /tmp/tollgate-pr-verify/1078-20261031T210146Z (on the verifying host)
• Retrospective stamp: PR merged 2026-10-27T00:01:07Z without a pre-merge gate run; this verification targets the merged PR head SHA, posted as backlog evidence (campaign frontier: every merged PR down to the campaign start gets a stamp)

Leg	Result	Detail
hygiene	PASS	no conflict markers in the tracked tree (0s)
contract-fences	FAIL	red: changelog fragments (assemble check) doc-facts contract dependency sync — see contract-fences.log; 13 checks green (5s)
packaging-sequential	FAIL	red (3/17): backend-api-owner-network_test.sh entry-ui-install-sequence_test.sh rebrand-literal-gutter_test.sh — logs in /tmp/tollgate-pr-verify/1078-20261031T210146Z/packaging-sequential (13s)
shell-suites	FAIL	red (2/23): uci-defaults-gatewayname-banner_test.sh uci-defaults-lan-private-wired_test.sh — logs in /tmp/tollgate-pr-verify/1078-20261031T210146Z/shell-suites (67s)
packaging-concurrent	FAIL	red (3/17): backend-api-owner-network_test.sh entry-ui-install-sequence_test.sh rebrand-literal-gutter_test.sh — logs in /tmp/tollgate-pr-verify/1078-20261031T210146Z/packaging-concurrent (5s)
go-battery	FAIL	19 modules — see go-battery.log (0s)

Verdict: 5 leg(s) RED — 1 green.

What this gate does not run (no vacuous green)

• spec-quote drift check (greatspectations tool + pinned nuts fetch)
• reproducibility / conformance lanes (make release-check territory)
• router-visible behaviour (PRTA / bench — unit green ≠ router works)

failing-leg log tails

contract-fences — tail

🔴 google.golang.org/genproto/googleapis/rpc — 2 versions:
   root: v0.0.0-20260928230214-8a89bd6388cc
   tollwallet: v0.0.0-20260720211330-0afa2a65878a

🔴 google.golang.org/grpc — 2 versions:
   root: v1.83.2
   tollwallet: v1.82.1

Fix: update all go.mod files to the same version.
     Run 'go mod tidy' in each sub-module after updating.
== gonuts pin fence (offline identity)
ok   an aligned tree (block + single-line carriers) passes
ok   a block-form carrier on a different version
ok   a single-line carrier left behind (the token-recovery shape)
ok   a go.sum that never recorded the truth version (half-update)
ok   a self-consistent pseudo-version integration pin passes OFFLINE (the --latest leg is the release gate for it)
ok   a manifest without the .gonuts key
ok   a tree whose manifest names a pin no carrier has

7 cases, 0 failed
== gonuts pin carriers
  manifest pin (packaging/build-inputs.json .gonuts.version): v0.14.1
  carriers discovered by glob: 5 module(s)
✅ one pin, everywhere: github.com/OpenTollGate/gonuts-tollgate v0.14.1 across 5 carrier(s).
== mirror/relay sync
ok    BLOSSOM_SERVERS: 13 copies identical (https://blossom.primal.net https://blossom.psbt.me https://blossom2.orangesync.tech)
ok    relay list: 13 copies identical (wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://relay2.orangesync.tech)
== stale import paths
✅ All 344 Go source files use correct import paths.

packaging-sequential — tail

--- backend-api-owner-network_test.sh.log
ok   the fragment guards exactly :2121
ok   packaging/files/etc/nftables.d/30-backend-firewall.nft is an include fragment (no own table declaration)
ok   the chain hooks input at priority -1 (the drop is authoritative)
ok   chain 'backend_input_firewall' is declared once
== the admin-board and LuCI guards are still captive-bridge-scoped
ok   31-admin-board-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
ok   32-luci-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
== the recipe installs the rule
ok   local-build-ipk.sh installs 30-backend-firewall.nft
ok   packaging/Makefile installs the 30-backend-firewall.nft rule via its *.nft glob
ok   packaging/Makefile FILES_ lists /etc/nftables.d/30-backend-firewall.nft
== the fragment parses
FAIL nft -c rejects the fragment — 'nft -f' would fail the whole fw4 reload

passed=17 failed=1
--- entry-ui-install-sequence_test.sh.log

== scenario 4b: converged box, root hash emptied — 999 re-refuses the board
ok   scenario 4b: :8080 is unbound — the refused board's own pair
ok   scenario 4b: :443 is unbound — the refused board's own pair
ok   scenario 4b: exactly one section binds :8090 (main)
ok   scenario 4b: exactly one section binds :8443 (main)
ok   scenario 4b: uhttpd.main keeps at least one listener
ok   scenario 4b: uhttpd.portal keeps at least one listener
ok   scenario 4b: uhttpd.trusted keeps at least one listener
ok   scenario 4b: the admin board is deliberately refused (fail-closed gate), not torn
ok   scenario 4b: LuCI (uhttpd.main) answers :8090 on both families
ok   scenario 4b: LuCI (uhttpd.main) answers :8443 on both families
ok   scenario 4b: the board is refused — its pair is intentionally unbound

tests: 72 passed, 1 failed
--- rebrand-literal-gutter_test.sh.log

== negative control: the scan pattern sees a planted occurrence
ok   the pattern matches a planted occurrence (the gutter is not inert)
ok   the pattern does not match a near-miss brand name (it is anchored to the banned literal)
ok   scan_tree ignores the control file (it is outside the tracked tree)

== the shipped uci-defaults reference only known uhttpd sections
ok   every uhttpd.<name> in the uci-defaults' active code is a known section (or a cert/key path)

== control: the section tokenizer ignores comments, not code
ok   comment-only section names (full-line and trailing) are not flagged
ok   known sections in active code are still extracted
ok   a foreign section in active code is still extracted (the check keeps its teeth)

tests: 8 passed, 1 failed

shell-suites — tail

--- uci-defaults-gatewayname-banner_test.sh.log
ok   the FULL path's banner IS the canonical one from the shared table
ok   reinstall (same version) run exits 0
ok   the same-version reinstall took the VERIFY branch
ok   the repair path COMMITS the banner '!TollGate-OQ3Q Portal'
ok   the repair path's banner is the SAME value the full path writes
ok   no gatewayname op is left pending in the /tmp/.uci-style delta
FAIL the repair path's captive SSID converged on the same code (got '!tollgate-0GLK', want '!TollGate-OQ3Q')
FAIL the banner is exactly '<captive SSID> Portal' (got '!TollGate-OQ3Q', want '!tollgate-0GLK')
ok   the operator's PSK is untouched
ok   drifting reinstall run exits 0
ok   the drifting repair path commits the SAME banner
ok     ...and the drifted allow-list rule was repaired (the commit did fire)
ok     ...with no gatewayname op left pending

27 passed, 2 failed
--- uci-defaults-lan-private-wired_test.sh.log
ok   silent no-op: the log names the missing port
== empty source and empty target: reported as an error, nothing written
ok   empty source+target: writer returns non-zero (1)
ok   empty source+target: private_bridge still has no ports
ok   empty source+target: log reports the broken state
== bridge device section beyond index 23 is still found and moved
ok   late bridge: writer returns 0
ok   network.private_bridge lists eth1 exactly once
ok   late bridge: the source section was cleared
== negative control: a script with a no-op writer fails the assertions
ok   negative control: the writer is neutralised in the copied script
ok   negative control: without the writer the port stays on the captive bridge
ok   negative control: the captive bridge keeps its port (the move assertion would fail)

lan-private-wired: 8 check(s) FAILED, 35 passed

packaging-concurrent — tail

--- backend-api-owner-network_test.sh.log
ok   the fragment guards exactly :2121
ok   packaging/files/etc/nftables.d/30-backend-firewall.nft is an include fragment (no own table declaration)
ok   the chain hooks input at priority -1 (the drop is authoritative)
ok   chain 'backend_input_firewall' is declared once
== the admin-board and LuCI guards are still captive-bridge-scoped
ok   31-admin-board-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
ok   32-luci-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
== the recipe installs the rule
ok   local-build-ipk.sh installs 30-backend-firewall.nft
ok   packaging/Makefile installs the 30-backend-firewall.nft rule via its *.nft glob
ok   packaging/Makefile FILES_ lists /etc/nftables.d/30-backend-firewall.nft
== the fragment parses
FAIL nft -c rejects the fragment — 'nft -f' would fail the whole fw4 reload

passed=17 failed=1
--- entry-ui-install-sequence_test.sh.log

== scenario 4b: converged box, root hash emptied — 999 re-refuses the board
ok   scenario 4b: :8080 is unbound — the refused board's own pair
ok   scenario 4b: :443 is unbound — the refused board's own pair
ok   scenario 4b: exactly one section binds :8090 (main)
ok   scenario 4b: exactly one section binds :8443 (main)
ok   scenario 4b: uhttpd.main keeps at least one listener
ok   scenario 4b: uhttpd.portal keeps at least one listener
ok   scenario 4b: uhttpd.trusted keeps at least one listener
ok   scenario 4b: the admin board is deliberately refused (fail-closed gate), not torn
ok   scenario 4b: LuCI (uhttpd.main) answers :8090 on both families
ok   scenario 4b: LuCI (uhttpd.main) answers :8443 on both families
ok   scenario 4b: the board is refused — its pair is intentionally unbound

tests: 72 passed, 1 failed
--- rebrand-literal-gutter_test.sh.log

== negative control: the scan pattern sees a planted occurrence
ok   the pattern matches a planted occurrence (the gutter is not inert)
ok   the pattern does not match a near-miss brand name (it is anchored to the banned literal)
ok   scan_tree ignores the control file (it is outside the tracked tree)

== the shipped uci-defaults reference only known uhttpd sections
ok   every uhttpd.<name> in the uci-defaults' active code is a known section (or a cert/key path)

== control: the section tokenizer ignores comments, not code
ok   comment-only section names (full-line and trailing) are not flagged
ok   known sections in active code are still extracted
ok   a foreign section in active code is still extracted (the check keeps its teeth)

tests: 8 passed, 1 failed

go-battery — tail

=== module: src
gofmt: unformatted files (fix with: gofmt -w .):
cli/network_radioless_test.go
merchant/unmeterable_session_cadence_test.go
wireless_gateway_manager/connector.go
make: *** [Makefile:13: go-battery] Error 1