Repository navigation
fix(merchant): event-driven mint-health transitions — recovery in seconds, not poll cycles (#747) - #777
fix(merchant): event-driven mint-health transitions — recovery in seconds, not poll cycles (#747)#777Amperstrand wants to merge 1 commit into
Conversation
…onds, not poll cycles (#747) Health transitions rode the 5-minute proactive poll in both directions: a mint answering ok=true on consecutive probes stayed out of the reachable set until recoveryThreshold successes accumulated at the poll cadence (~15 min — the issue measured ok=true twice while discovery still served degraded), the aggressive loop was one-shot per downgrade so an outage that outlived its 5-minute window left recovery on the poll cadence (repeated block/unblock cycles stopped recovering inside any test budget), and a mint that just served a full swap still counted as down until probes said otherwise. Three changes, none of which touch the steady-state probe rate (the rate-limiter reality the design respects — the burst runs only while a recovery is actually pending, honours every Retry-After via probeDue, and stops itself the moment the mint is readmitted): - MarkReachable: the success-side health event. A completed swap is the strongest reachability evidence there is — readmit immediately, void the stale Retry-After hold, reset both counters, fire the set-changed and first-reachable callbacks. Wired into PurchaseSession after a successful Receive. - runProactiveCheck arms the 15-second recovery burst when a probe answers OK for a mint the tracker still considers down — confirmation happens in seconds, and the re-arm closes the expired-window dead end. - The burst keeps flap protection: readmit at threshold 1 only when the reachable set is empty (the historical aggressive behaviour); otherwise recoveryThreshold successes are still required. Also fixed, discovered by the new tests: registering the degraded first-reachable trigger on a tracker that already has a reachable mint armed a spurious fire on the next poll — SetOnFirstReachableForDegraded cleared the latch unconditionally. The guard suite was self-contradictory about this (FiredOnce vs FiredAfterReset pinned opposite behaviours; one passed only by beating the callback goroutine to its assertion). Registration now pre-sets the latch when a mint is already reachable — the documented "after starting with none" contract — and the two contradictory tests pin the consistent semantics. Tests: 6 new (MarkReachable immediate readmission + backoff reset + callbacks; the degraded trigger fire; the headline ok-for-unreachable poll arms a working burst; flap protection counted server-side; window-expiry re-arm; a full PurchaseSession readmitting a marked-down mint) plus the reworked registration pair. Merchant suite green except the two pre-existing timing failures documented on #728/#737. Fixes #747
e817dfe to
4c66719
Compare
|
@felixfelix-bot — requesting cross-identity approval for this PR (fix lane for #747). What: Three tracker changes, no steady-state rate increase: (1) Why: #747 — both directions of health transitions rode the 5-minute poll: readmission needed 3 poll successes (~15 min) while the prober already answered ok=true, and the aggressive loop was one-shot per downgrade. Evidence: 6 new |
|
CHANGES-REQUESTED — the #747 core (event-driven transitions, flap protection, re-arm) is verified sound and merge-ready, but the PR's "bonus fix" independently re-fixes #732, which #774 owns, reworks the same two tests under different names (guaranteed textual conflict), and its supporting claim "production is unaffected" is wrong — the fix is needed in production, and #774's analysis of that is the correct one. The #747 machinery itself I verified end-to-end in the head worktree:
No fund-safety concerns: health-state and callbacks only — thresholds, cadence, Retry-After honoring, throttle demotion, advertisement rules, and everything on the money path are untouched (verified line by line against the diff). Freeze compatibility: yes — a router that gets stuck degraded for 15 minutes after its mint returns is a support-ticket generator in a fresh release, and this is the minimal, well-fenced fix. Land it once the #732 duplication with #774 is resolved. |
…e changelog rides the fragment convention (#778 review) Review mechanicals: - build-purity's Tests 1-2 now audit rateLimitTestBypass = true writers with the same guard discipline as TOLLGATE_TEST_CONFIG_DIR: a non- _test.go, non-build-tagged writer fails purity. The PR's "enforced by the build-purity contract" claim was ahead of the script; it is true now. The audit is verified both ways: a spoofed unguarded writer fails 2/2 checks, the clean tree passes 3/3. - The direct CHANGELOG.md edit is reverted in favour of a changelog.d fragment (the #769/#771/#777 convention), carrying the corrected claim — rig-verified FIXED on #748, purity-audited writer. No runtime behavior touched: the shipped 10 RPM default, its semantics and the three rate-limit proof tests are byte-identical (9/9 passes).
|
Dynamic rel-verify evidence for this PR (lane rc1-777, QEMU rig on v0.6.0-rc1, OpenWrt 25.12.0 x86-64, pristine snapshot boots, single-mint config against a controllable mock mint serving NUT-01 keysets; outage = HTTP 503): issue #747's findings 1+3 reproduced on pristine rc1, eliminated on this PR's build. Full verdict with the event-by-event timeline: #747 (comment) Patch path used: real build of head
Regression on the patched build: logread clean of new error classes (0 panic/fatal; only the by-design degraded-wallet line rc1 also emits), NDS + nft enforcement stable through all 5 transitions, advertisement flipped 10021↔21023 exactly at logged set-changes — no flapping. Scope notes: @felixfelix-bot — base policy blocks author-merge (REVIEW_REQUIRED), and per house rules the maintainer lands it: this lane's dynamic evidence says the fix works and costs nothing in steady state. Please review + merge when you're satisfied. |
|
Release-position note (lane rc1-777, final comment) — for/against getting this and the related open queue in before 0.6.0-final, and the risks. Rebase first: none needed — head The case FOR landing this queue before 0.6.0This PR (#777 / #747) — field-visible liveness defect, measured:
The related queue, three clusters:
The case AGAINST (and the real risks)
RecommendationLand #777 and the test-reliability cluster for 0.6.0 — they close a measured field-visible defect and make the gate meaningful, at measured-zero steady-state cost. Treat the wallet bumps and the bench-verify quartet as wanted in, but behind an rc2 that carries the full queue and a re-run of the dynamic lanes (the harnesses exist: this lane's mint-health rig ~3 h, bench-verify's four repros) — then tag final from rc2. The one outcome to avoid is a final tag that is neither the validated rc1 bytes nor a re-validated superset. — lane rc1-777 (transcript + raw logs: |
|
🏷️ Next action — changes-requested Changes requested — author addresses the blocking item(s) in the review above, then request a re-review. |
Review: APPROVEReviewed cold at head Diagnosis is precise and splits the bug in two — which is what makes the fix reviewable:
The fix adds I removed the success-side event and the tests caught it: Restored → green. Both failure messages name the behaviour and the consequence (the degraded Suite scale and balance: Honest note on suite cost: Scope boundary stated correctly by the PR: recovery is now seconds-not-poll-cycles; the Not hardware-tested. The behavioural claim is about recovery latency against a real mint going Labels: not applied — this account has FULL_REVIEW: ~/reports/reviews/tollgate-module-basic-go-pr777-kimi.md |
Local CI gate — PR #777: fix(merchant): event-driven mint-health transitions — recovery in seconds, not poll cycles (#747)Run of
Verdict: 2 leg(s) RED — 4 green. What this gate does not run (no vacuous green)
failing-leg log tailsshell-suites — tailgo-battery — tail |
|
Closing as superseded (close-prep-verified, re-validated today at current main): the event-driven mint-health machinery is on main at symbol level — |
Diagnosis (from the tracker code)
Health transitions rode the 5-minute proactive poll in both directions:
recoveryThreshold(3) successes at the 5-minute cadence = up to ~15 min to readmit a returned mint — exactly the issue's headline: the prober answeredok=truetwice, five minutes apart, while discovery still servedno-reachable-mints. The aggressive 15-second loop existed but was one-shot per downgrade with a 5-minute window: an outage that outlived the window left recovery on the poll cadence — why repeated block/unblock cycles stopped recovering inside any test budget (finding 3).MarkUnreachablefired on payment-observed failures, but a mint that just served a complete token swap still counted as down until probes said otherwise (finding 1's mirror image).The fix — event-driven transitions, steady-state rates untouched
The design respects the rate-limiter realities the issue names: the steady-state cadence stays at 5 minutes; the burst runs only while a recovery is actually pending, honours every
Retry-Aftervia the existingprobeDuegate, and stops itself the moment the mint is readmitted.MarkReachable(mintURL)— the success-side health event. A completed swap is the strongest reachability evidence there is, stronger than any probe: readmit immediately, void the staleRetry-Afterhold, reset both counters (successes seeded to threshold, like a boot probe), fire the set-changed and first-reachable callbacks. Wired intoPurchaseSessionright after a successfulReceive.runProactiveChecknow collects mints that answered OK while the tracker still considers them down and arms the 15-second loop — confirmation lands within seconds, and the re-arm closes the expired-window dead end: after a burst times out mid-recovery, the next poll's OK arms a fresh one and the success counter accumulates across bursts.recoveryThresholdsuccesses are still required — a single 15-second answer must not readmit what three probes would still be weighing.Bonus fix (found by the new tests)
Registering the degraded first-reachable trigger on a tracker that already had a reachable mint armed a spurious fire on the next poll —
SetOnFirstReachableForDegradedcleared the latch unconditionally. The guard suite was self-contradictory about this:FiredOnceandFiredAfterResetpinned opposite behaviours, and one of them passed only by beating the callback goroutine to its assertion (a load-dependent flake — reproduced on pristine main). Registration now pre-sets the latch when a mint is already reachable — the documented "after starting with none" contract — and the two contradictory tests pin the consistent semantics from both sides. Production is unaffected: the degraded path registers at zero-reachable by construction.Tests (6 new + 2 reworked, all
-race)Merchant suite green except the two pre-existing timing failures documented since #728 (proven on pristine main).
make go-batterymirrors that picture.Fund-safety note (per AGENTS.md)
Health-state only — no money-path semantics change: thresholds, probe cadence, Retry-After honouring, throttle demotion, and the advertisement rules are untouched. The point of no return, intent journal, and duplicate guards are not in this diff's scope.
Fixes #747