Skip to content

fix(merchant): event-driven mint-health transitions — recovery in seconds, not poll cycles (#747) - #777

Closed
Amperstrand wants to merge 1 commit into
mainfrom
fix/747-mint-health-transitions
Closed

Amperstrand wants to merge 1 commit into
mainfrom
fix/747-mint-health-transitions

Conversation

@Amperstrand

Copy link
Copy Markdown
Collaborator

Diagnosis (from the tracker code)

Health transitions rode the 5-minute proactive poll in both directions:

  1. Recovery waited for poll-cadence confirmation. recoveryThreshold (3) successes at the 5-minute cadence = up to ~15 min to readmit a returned mint — exactly the issue's headline: the prober answered ok=true twice, five minutes apart, while discovery still served no-reachable-mints. The aggressive 15-second loop existed but was one-shot per downgrade with a 5-minute window: an outage that outlived the window left recovery on the poll cadence — why repeated block/unblock cycles stopped recovering inside any test budget (finding 3).
  2. No success-side event. MarkUnreachable fired on payment-observed failures, but a mint that just served a complete token swap still counted as down until probes said otherwise (finding 1's mirror image).
  3. A poll answering OK for an unreachable mint triggered nothing — the recovery signal was observed and then left to age on the cadence.

The fix — event-driven transitions, steady-state rates untouched

The design respects the rate-limiter realities the issue names: the steady-state cadence stays at 5 minutes; the burst runs only while a recovery is actually pending, honours every Retry-After via the existing probeDue gate, and stops itself the moment the mint is readmitted.

  1. MarkReachable(mintURL) — the success-side health event. A completed swap is the strongest reachability evidence there is, stronger than any probe: readmit immediately, void the stale Retry-After hold, reset both counters (successes seeded to threshold, like a boot probe), fire the set-changed and first-reachable callbacks. Wired into PurchaseSession right after a successful Receive.
  2. Poll-OK arms the recovery burst. runProactiveCheck now collects mints that answered OK while the tracker still considers them down and arms the 15-second loop — confirmation lands within seconds, and the re-arm closes the expired-window dead end: after a burst times out mid-recovery, the next poll's OK arms a fresh one and the success counter accumulates across bursts.
  3. Flap protection kept. The burst readmits at threshold 1 only when the reachable set is empty (the historical aggressive behaviour, unchanged); with the set non-empty, recoveryThreshold successes are still required — a single 15-second answer must not readmit what three probes would still be weighing.

Bonus fix (found by the new tests)

Registering the degraded first-reachable trigger on a tracker that already had a reachable mint armed a spurious fire on the next poll — SetOnFirstReachableForDegraded cleared the latch unconditionally. The guard suite was self-contradictory about this: FiredOnce and FiredAfterReset pinned opposite behaviours, and one of them passed only by beating the callback goroutine to its assertion (a load-dependent flake — reproduced on pristine main). Registration now pre-sets the latch when a mint is already reachable — the documented "after starting with none" contract — and the two contradictory tests pin the consistent semantics from both sides. Production is unaffected: the degraded path registers at zero-reachable by construction.

Tests (6 new + 2 reworked, all -race)

TestMarkReachable_ReadmitsImmediatelyAndResetsBackoff   — immediate readmit, Retry-After voided, callback fired
TestMarkReachable_FiresFirstReachableForDegraded        — the degraded upgrade trigger fires
TestRunProactiveCheck_OKForUnreachableMintArmsRecoveryBurst — the issue's headline: ok-for-unreachable poll → readmitted in <2s, no external arming
TestRecoveryBurstKeepsFlapProtectionWhenSetNotEmpty     — readmission counted server-side against recoveryThreshold probe successes
TestRecoveryBurstReArmsAfterWindowExpiry                — finding 3's mechanism: expired window is not a dead end
TestSuccessfulPaymentMarksMintReachable                 — a full PurchaseSession readmits a marked-down mint through the real path
TestSetOnFirstReachableForDegraded_* (reworked pair)    — both sides of the registration latch, deterministic

Merchant suite green except the two pre-existing timing failures documented since #728 (proven on pristine main). make go-battery mirrors that picture.

Fund-safety note (per AGENTS.md)

Health-state only — no money-path semantics change: thresholds, probe cadence, Retry-After honouring, throttle demotion, and the advertisement rules are untouched. The point of no return, intent journal, and duplicate guards are not in this diff's scope.

Fixes #747

…onds, not poll cycles (#747)

Health transitions rode the 5-minute proactive poll in both directions:
a mint answering ok=true on consecutive probes stayed out of the reachable
set until recoveryThreshold successes accumulated at the poll cadence
(~15 min — the issue measured ok=true twice while discovery still served
degraded), the aggressive loop was one-shot per downgrade so an outage
that outlived its 5-minute window left recovery on the poll cadence
(repeated block/unblock cycles stopped recovering inside any test
budget), and a mint that just served a full swap still counted as down
until probes said otherwise.

Three changes, none of which touch the steady-state probe rate (the
rate-limiter reality the design respects — the burst runs only while a
recovery is actually pending, honours every Retry-After via probeDue,
and stops itself the moment the mint is readmitted):

- MarkReachable: the success-side health event. A completed swap is the
  strongest reachability evidence there is — readmit immediately, void
  the stale Retry-After hold, reset both counters, fire the set-changed
  and first-reachable callbacks. Wired into PurchaseSession after a
  successful Receive.
- runProactiveCheck arms the 15-second recovery burst when a probe
  answers OK for a mint the tracker still considers down — confirmation
  happens in seconds, and the re-arm closes the expired-window dead end.
- The burst keeps flap protection: readmit at threshold 1 only when the
  reachable set is empty (the historical aggressive behaviour);
  otherwise recoveryThreshold successes are still required.

Also fixed, discovered by the new tests: registering the degraded
first-reachable trigger on a tracker that already has a reachable mint
armed a spurious fire on the next poll — SetOnFirstReachableForDegraded
cleared the latch unconditionally. The guard suite was
self-contradictory about this (FiredOnce vs FiredAfterReset pinned
opposite behaviours; one passed only by beating the callback goroutine
to its assertion). Registration now pre-sets the latch when a mint is
already reachable — the documented "after starting with none" contract —
and the two contradictory tests pin the consistent semantics.

Tests: 6 new (MarkReachable immediate readmission + backoff reset +
callbacks; the degraded trigger fire; the headline ok-for-unreachable
poll arms a working burst; flap protection counted server-side;
window-expiry re-arm; a full PurchaseSession readmitting a marked-down
mint) plus the reworked registration pair. Merchant suite green except
the two pre-existing timing failures documented on #728/#737.

Fixes #747
@Amperstrand
Amperstrand force-pushed the fix/747-mint-health-transitions branch from e817dfe to 4c66719 Compare October 8, 2026 21:26
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

@felixfelix-bot — requesting cross-identity approval for this PR (fix lane for #747).

What: Three tracker changes, no steady-state rate increase: (1) MarkReachable — the success-side health event; a completed swap readmits the mint immediately, voids its stale Retry-After, and fires the degraded-upgrade callbacks (wired into PurchaseSession post-Receive); (2) runProactiveCheck arms the 15-second recovery burst when a probe answers OK for a mint the tracker still considers down — and re-arms after a burst window expires mid-recovery, closing the one-shot dead end that killed repeated block/unblock recovery; (3) the burst keeps flap protection (threshold 1 only with an empty reachable set; recoveryThreshold otherwise). Bonus: fixed the self-contradictory first-reachable registration semantics (unconditional latch clear armed a spurious fire; one guard test passed only by goroutine-scheduling luck — reproduced on pristine main).

Why: #747 — both directions of health transitions rode the 5-minute poll: readmission needed 3 poll successes (~15 min) while the prober already answered ok=true, and the aggressive loop was one-shot per downgrade.

Evidence: 6 new -race tests including the headline scenario (ok-for-unreachable poll → readmitted <2s, no external arming), server-side-counted flap protection, window-expiry re-arm, and a full PurchaseSession readmitting a marked-down mint. Rate-limiter realities respected: burst only while recovery is pending, every Retry-After honoured via probeDue, self-stopping on readmit. Health-state only — no money-path semantics touched (fund-safety note in the PR body).

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

CHANGES-REQUESTED — the #747 core (event-driven transitions, flap protection, re-arm) is verified sound and merge-ready, but the PR's "bonus fix" independently re-fixes #732, which #774 owns, reworks the same two tests under different names (guaranteed textual conflict), and its supporting claim "production is unaffected" is wrong — the fix is needed in production, and #774's analysis of that is the correct one.

The #747 machinery itself I verified end-to-end in the head worktree: MarkReachable sits strictly on the post-Receive success path (merchant.go#L1774-L1779), mirrors MarkUnreachable's discipline exactly (set-change-only callbacks, exact count bookkeeping, go cb() outside the lock — mint_health_tracker.go#L427-L472), the burst honors every Retry-After via probeDue even mid-burst (mint_health_tracker.go#L727-L731), self-terminates the moment a mint is readmitted or the 5-minute window expires, and the armed-flag prevents stacking. On the three questions this review was asked to settle: recovery-in-seconds is real — the arming poll fires within one 5-minute cycle of the mint returning, then readmission is ~15 s (empty reachable set) or ~3×15 s (flap-protected) versus the ~15 minutes the issue measured; load stays bounded — the burst arms only on an OK-for-down probe (mint_health_tracker.go#L634-L637), so a sustained outage never bursts; transitions survive restart in the only sense that matters here — health state is in-memory and fully rebuilt by RunInitialProbe at boot, which is consistent with the repo's rule that non-money state be reconcilable from an external authority; nothing in this PR changes that. The flap-protection test is the strongest in the diff: it counts the mint's served probes server-side and fails if readmission preceded recoveryThreshold of them. All ten new/reworked tests pass under -race in a clean worktree (I ran them). The changelog correctly ships as a changelog.d fragment.

  1. (overlap with fix(merchant): de-flake the suite trio — late-outcome picker, first-reachable re-arm, owed-grant retry clock (#731, #732, #733) #774 — must be resolved before either merges) The bonus fix re-implements fix(merchant): de-flake the suite trio — late-outcome picker, first-reachable re-arm, owed-grant retry clock (#731, #732, #733) #774's merchant suite: TestSetOnFirstReachableForDegraded_FiredOnce intermittent — recovery callback fires for a mint already reachable from the initial probe #732 latch fix: same semantics (SetOnFirstReachableForDegraded pre-arms when a mint is already reachable — yours loops the map at mint_health_tracker.go#L482-L494; fix(merchant): de-flake the suite trio — late-outcome picker, first-reachable re-arm, owed-grant retry clock (#731, #732, #733) #774's is the one-liner hadReachableMint = reachableCount > 0), and both PRs rework the same two tests in merchant_degraded_test.go under different names — a guaranteed conflict, since both branch off 4614ac2. Worse, the body's justification — "Production is unaffected: the degraded path registers at zero-reachable by construction" — is incorrect: the wallet-failure degraded path registers the callback while mints are still reachable (we only reach it after passing the zero-reachable check, merchant.go#L511-L517), and each failed upgrade re-enters newFullMerchant, re-creates the degraded merchant, and re-registers — the every-5-minutes rebuild loop fix(merchant): de-flake the suite trio — late-outcome picker, first-reachable re-arm, owed-grant retry clock (#731, #732, #733) #774's changelog describes. Requested: drop the bonus fix and the reworked TestOnFirstReachable_* pair from this PR and rebase on fix(merchant): de-flake the suite trio — late-outcome picker, first-reachable re-arm, owed-grant retry clock (#731, #732, #733) #774 (which owns merchant suite: TestSetOnFirstReachableForDegraded_FiredOnce intermittent — recovery callback fires for a mint already reachable from the initial probe #732 by issue assignment and carries the correct production analysis). If the maintainer prefers the reverse sequencing, that's an owner call — but only one of the two may carry the latch change.

  2. (minor, criterion 4) The failure-branch comment at mint_health_tracker.go#L768-L770 still says "aggressive mode only runs when the reachable set is empty" — now false by this PR's own design (the readmitThreshold logic 30 lines up exists precisely for the non-empty case). One-line comment fix while you're in the file.

  3. (precision note, optional) "The extra probe rate exists only while a recovery is actually pending" is true of the window, not the scope: each burst tick probes every configured mint, healthy ones included, at 15 s for up to 5 minutes. Bounded and self-terminating, but if you want the claim airtight, scope the burst's probe set to the pendingRecovery mints or soften the comment.

No fund-safety concerns: health-state and callbacks only — thresholds, cadence, Retry-After honoring, throttle demotion, advertisement rules, and everything on the money path are untouched (verified line by line against the diff).

Freeze compatibility: yes — a router that gets stuck degraded for 15 minutes after its mint returns is a support-ticket generator in a fresh release, and this is the minimal, well-fenced fix. Land it once the #732 duplication with #774 is resolved.

Amperstrand pushed a commit that referenced this pull request Oct 9, 2026
…e changelog rides the fragment convention (#778 review)

Review mechanicals:

- build-purity's Tests 1-2 now audit rateLimitTestBypass = true writers
  with the same guard discipline as TOLLGATE_TEST_CONFIG_DIR: a non-
  _test.go, non-build-tagged writer fails purity. The PR's "enforced by
  the build-purity contract" claim was ahead of the script; it is true
  now. The audit is verified both ways: a spoofed unguarded writer
  fails 2/2 checks, the clean tree passes 3/3.
- The direct CHANGELOG.md edit is reverted in favour of a changelog.d
  fragment (the #769/#771/#777 convention), carrying the corrected
  claim — rig-verified FIXED on #748, purity-audited writer.

No runtime behavior touched: the shipped 10 RPM default, its semantics
and the three rate-limit proof tests are byte-identical (9/9 passes).
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Dynamic rel-verify evidence for this PR (lane rc1-777, QEMU rig on v0.6.0-rc1, OpenWrt 25.12.0 x86-64, pristine snapshot boots, single-mint config against a controllable mock mint serving NUT-01 keysets; outage = HTTP 503): issue #747's findings 1+3 reproduced on pristine rc1, eliminated on this PR's build. Full verdict with the event-by-event timeline: #747 (comment)

Patch path used: real build of head 4c66719 through the pinned SDK pipeline (scripts/build-sdk-package.sh, go1.26.8, digest-pinned openwrt/sdk x86-64) → tollgate-wrt-0.6.0_pre777-r0.apk, sha256 fbfc2263be211af23f9d1b4df65b1cbdf2235280fff3af00794fac085de4eab3. Baseline = shipped rc1 bytes (d1b1ba5e…, commit 659de8a).

metric (single mint; all times UTC, logread + mint-side request log) rc1 baseline PR #777
restore (aggressive window expired) → healthy 13 m 07 s (poll OK 1/3 at +3 m, 2/3 at +8 m, 3/3 at +13 m; kind 21023 served while probes answered ok=true — finding 1 live for 10 min) 3 m 43 s / 4 m 28 s / 3 m 32 s (3 consecutive cycles)
first probe-OK → readmitted 10 m 00 s (3×300 s confirmations) 15 s every cycle — probe answered OK for unreachable mint(s) — arming the recovery-burst loop → readmit 15 s later
repeated cycles cycle 2+ dead-ends (finding 3) identical 15 s recovery on cycles 1–3, incl. after a 21-min sustained outage — no backoff residue
sustained 21-min outage, probe load 15 s burst one-shot 5 min, then 300 s same bounded cadence: burst only while recovery pending (mint-side hits every 15.000 s for exactly 5 min), then 300 s — 29 probes total, no busy-loop
restart mid-outage — sane degraded-from-boot, window expiry while down, restore → burst → wallet re-init + Upgrading from degraded to full merchant in 16 s after probe-OK
outage → degraded 13 m 12 s (3×300 s failure threshold) unchanged by design (steady-state cadence untouched)

Regression on the patched build: logread clean of new error classes (0 panic/fatal; only the by-design degraded-wallet line rc1 also emits), NDS + nft enforcement stable through all 5 transitions, advertisement flipped 10021↔21023 exactly at logged set-changes — no flapping.

Scope notes: MarkReachable (swap-success readmission) needs a real purchase flow and is covered by this PR's own tests, not this probe-driven rig leg; patched apk = main@cb4db7f + 4c66719 (src/merchant delta vs rc1 is exactly 4c66719). Raw logs + transcript: ~/tollgate-vm/rel-777/ on ai-legion.

@felixfelix-bot — base policy blocks author-merge (REVIEW_REQUIRED), and per house rules the maintainer lands it: this lane's dynamic evidence says the fix works and costs nothing in steady state. Please review + merge when you're satisfied.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Release-position note (lane rc1-777, final comment) — for/against getting this and the related open queue in before 0.6.0-final, and the risks. Rebase first: none needed — head 4c66719 sits on the current main tip (4614ac2), GitHub reports MERGEABLE, blocked only by the required review. (The rel-verify build in this lane was made against exactly that parent, so the measured evidence is against today's base, not a stale one.)

The case FOR landing this queue before 0.6.0

This PR (#777 / #747) — field-visible liveness defect, measured:

  • On pristine rc1, every mint restart (a normal event: deployments, CDK upgrades) opens a window where recovery dead-ends: measured 13 m 07 s restore→healthy, with the :2121 advertisement serving kind-21023 no-reachable-mints for 10 minutes while the prober itself answered ok=true — sales stop while the mint is healthy. Repeated outages never recover inside the issue's 420 s test budgets.
  • The fix costs nothing in steady state (measured: 29 probes in a 21-minute outage; the 15 s burst exists only while a recovery is pending; poll cadence, thresholds, Retry-After handling, throttle demotion untouched) and recovery becomes 15 s from first probe-OK, every cycle, no escalation residue, including after restart-mid-outage.
  • Release-engineering reason: Mint-health transitions are 5-min-poll driven with escalating backoff — repeated mint outages make recovery exceed any test budget #747's origin is test flakiness (HEALTH_POLL_TIMEOUT=420 / module timeout 900 workarounds). A release gate that cannot test mint restarts reliably cannot certify any mint-touching change in 0.6.0 or beyond. Landing fix(merchant): event-driven mint-health transitions — recovery in seconds, not poll cycles (#747) #777 makes the gate itself trustworthy.

The related queue, three clusters:

The case AGAINST (and the real risks)

Recommendation

Land #777 and the test-reliability cluster for 0.6.0 — they close a measured field-visible defect and make the gate meaningful, at measured-zero steady-state cost. Treat the wallet bumps and the bench-verify quartet as wanted in, but behind an rc2 that carries the full queue and a re-run of the dynamic lanes (the harnesses exist: this lane's mint-health rig ~3 h, bench-verify's four repros) — then tag final from rc2. The one outcome to avoid is a final tag that is neither the validated rc1 bytes nor a re-validated superset.

— lane rc1-777 (transcript + raw logs: ~/tollgate-vm/rel-777/ on ai-legion)

@c03rad0r c03rad0r added status:changes-requested Reviewer flagged blocking items; author action needed wave:4 Covered by the wave-4 review pass labels Oct 10, 2026
@c03rad0r

Copy link
Copy Markdown
Contributor

🏷️ Next action — changes-requested

Changes requested — author addresses the blocking item(s) in the review above, then request a re-review.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

Review: APPROVE

Reviewed cold at head mod-pr777. A health-transition fix on the money path, with a test suite that
genuinely defends both directions. I verified the success-side event by removing it.

Diagnosis is precise and splits the bug in two — which is what makes the fix reviewable:

  1. Recovery rode the poll cadence. recoveryThreshold (3) successes at the 5-minute proactive
    poll meant up to ~15 min to readmit a returned mint, and the aggressive 15-second loop was
    one-shot per downgrade with a 5-minute window — so an outage outliving the window left
    recovery back on the poll cadence. That matches the issue's headline symptom: the prober answered
    ok=true twice, five minutes apart, while discovery still served no-reachable-mints.
  2. No success-side event. MarkUnreachable fired on payment-observed failures, but a mint that
    just served a complete token swap produced no symmetric "it's back" event.

The fix adds MarkReachable as the success-side event, and the semantics are right: it sets
consecutiveSuccesses[mintURL] = t.recoveryThreshold — i.e. it satisfies the threshold rather than
bypassing it, so immediate readmission does not weaken the poll-side rule. SetChangedCallback then
fires so the merchant reacts at once instead of waiting for a poll.

I removed the success-side event and the tests caught it:

--- FAIL: TestMarkReachable_ReadmitsImmediatelyAndResetsBackoff
    MarkReachable must readmit immediately — recovery must not wait for probe thresholds (#747)
--- FAIL: TestMarkReachable_FiresFirstReachableForDegraded
    MarkReachable must fire the first-reachable callback — the degraded upgrade path depends on it

Restored → green. Both failure messages name the behaviour and the consequence (the degraded
upgrade path depending on the callback), which is the useful kind.

Suite scale and balance: mint_health_tracker_test.go carries 37 tests, and they cover both
sides rather than only the new path — MarkUnreachable (including FiresSetChangedCallback,
ResetsConsecutiveSuccesses, and an unknown-mint no-op/no-panic guard), RunInitialProbe across all
reachable / none / mixed / connection-refused, and ProactiveCheck_RecoveryRequiresThreeConsecutiveSuccesses.
The unknown-mint cases matter here specifically: a callback-driven design is exactly where a
nil/unknown key panics, and those are pinned. Full merchant package under -race is green.

Honest note on suite cost: TestMerchantDegraded_GetMintHealthTracker takes 101.7s (it
exercises the real unreachable-mint path with DNS failure and the offline fallback). Not a defect —
it is a genuinely integration-shaped test and its assertions are meaningful — but it dominates the
package's wall time, so a -short split is worth considering if package runtime becomes a CI
concern.

Scope boundary stated correctly by the PR: recovery is now seconds-not-poll-cycles; the
durable side (restart loses session metering / health is in-memory) is the known gap already
recorded in AGENTS.md, and this PR does not claim to close it. I read it the same way — mint_health_tracker
state is process memory, so a restart re-derives health from the initial probe. That is consistent
with the documented gap rather than a new one.

Not hardware-tested. The behavioural claim is about recovery latency against a real mint going
away and coming back. The unit tests model the transitions deterministically (which is what the
tracker is for), and tests/cloud-lab/ is the house lane for mint-failure/rotation scenarios rather
than a bench board; a router run would not exercise mint health more faithfully than the cloud lab.
If the maintainer wants the end-to-end — block a mint mid-session on tollgate-A04E, confirm
readmission in seconds — say so and I will run it under the console mutex.

Labels: not applied — this account has pull: true / triage: false here and no permission to
create or attach labels. Intended label for whoever can: high (money-path availability: a mint that
recovered stayed excluded for up to ~15 minutes, and repeated block/unblock cycles could stop
recovering within any test budget).

FULL_REVIEW: ~/reports/reviews/tollgate-module-basic-go-pr777-kimi.md

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Local CI gate — PR #777: fix(merchant): event-driven mint-health transitions — recovery in seconds, not poll cycles (#747)

Run of scripts/pr-verify.sh (the local CI gate: GitHub Actions is
org-blocked and the ngit lane fires only on maintainer mirror pushes,
so a PR ref gets no CI runs — this comment is the offline verification
evidence, produced on the pinned head SHA).

  • Head: fix/747-mint-health-transitions @ 4c6671906e
  • Base: main @ 4614ac274d
  • Gate: pr-verify 294317251383 on ai-legion, started 2026-10-18 12:36 UTC, wall 378s
  • Logs: /tmp/tollgate-pr-verify/777-20261018T123032Z (on the verifying host)
Leg Result Detail
hygiene PASS no conflict markers in the tracked tree (0s)
contract-fences PASS 9 checks green; 4 skipped (absent from tree: entry-UI marker path fence SSID naming contract gonuts pin fence (offline identity) gonuts pin carriers) (3s)
packaging-sequential PASS 11 suites (7s)
shell-suites FAIL red (1/16): ngit-ci-trigger_test.sh — logs in /tmp/tollgate-pr-verify/777-20261018T123032Z/shell-suites (48s)
packaging-concurrent PASS 11 suites, all at once (3s)
go-battery FAIL 16 modules — see go-battery.log (317s)

Verdict: 2 leg(s) RED — 4 green.

What this gate does not run (no vacuous green)
  • spec-quote drift check (greatspectations tool + pinned nuts fetch)
  • reproducibility / conformance lanes (make release-check territory)
  • router-visible behaviour (PRTA / bench — unit green ≠ router works)

Note: the ngit trigger suite red carries the nak key public
nsec-arg signature — the portability break the NGIT_TRUST
generalization fixed. Trees predating that change reproduce it
on this seat; it is environmental, not PR-caused.

failing-leg log tails
shell-suites — tail
--- ngit-ci-trigger_test.sh.log
ok   - refuses a workflow file that is absent from the worktree
ok   - refuses a workflow that is not present at the declared commit
ok   - refuses a signing key that is not the maintainer key
ok   - never prints the private key on refusal
ok   - refuses a key file with no nsec in it
invalid hex key: '0nsec1c2lkdjfv5mjw66gpd7u0aazaujnvyylaxf2x7e0wacqm2qkeej9smny4dt' is not valid hex: encoding/hex: invalid byte: U+006E 'n'
ok   - prints the sha256 of the file's content at the declared commit
FAIL - fails loudly when every relay refuses the event
       rc=1 out=workflow : .ngit/act/workflows/fake.yml
commit   : d6f0d469e367ac57ecc4e765fd6da1f8e7e6a62d
sha256   : 0a2be7daf3741ba278eca69d6d25755580a1f4223c3c808b5e7e4d621e953544
ERROR: /tmp/tmp.cL9zdFFHV8/wrong.key is not the maintainer key (signer ..., expected 9cd14d9acdbab7ca...)
ok   - never prints the private key on the happy path

8 passed, 1 failed
go-battery — tail
time="2026-10-18T14:36:51+02:00" level=error msg="Gate close UNRESOLVED: ndsctl never confirmed the deauthorization of this client and the close budget is spent, so the module stops re-attempting it here — it cannot keep hammering the ndsctl socket, because that storm is what wedges it and stops a PAID purchase from being authorised at all. The gate STAYS TRACKED and the client's access state is UNVERIFIED: ndsctl answered with an error, not with an answer about the client. The reconciliation re-attempts this close whenever it verifies the client's state with ndsctl, and the record is forgotten only when a new session replaces it. OPERATOR ACTION: inspect nodogsplash (ndsctl status) — restart it or reload the module to clear the wedge" abandoned_closes=3 attempts=8 budget=8 last_error="exit status 1" mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
time="2026-10-18T14:36:51+02:00" level=warning msg="Attempted to close a gate that was not open." mac_address="aa:bb:cc:dd:ee:62" module=valve
FAIL
FAIL	github.com/OpenTollGate/tollgate-module-basic-go/src/merchant	285.698s
FAIL
make: *** [Makefile:7: go-battery] Error 1

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Closing as superseded (close-prep-verified, re-validated today at current main): the event-driven mint-health machinery is on main at symbol level — MintHealthTracker (src/merchant/mint_health_tracker.go) with StartProactiveChecks and ArmAggressiveRetry, plus its test suite green in the merchant battery. The reactive-polling design this PR replaced is gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:post-0.6.0 Deferred until after the v0.6.0 release status:changes-requested Reviewer flagged blocking items; author action needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mint-health transitions are 5-min-poll driven with escalating backoff — repeated mint outages make recovery exceed any test budget

3 participants