Skip to content

Mint-health transitions are 5-min-poll driven with escalating backoff — repeated mint outages make recovery exceed any test budget #747

Open
Open
Mint-health transitions are 5-min-poll driven with escalating backoff — repeated mint outages make recovery exceed any test budget#747

Description

@Amperstrand
Amperstrand
opened on Oct 8, 2026

Migrated from Amperstrand/tollgate-module-basic-go#87 as part of the fork retirement — OpenTollGate/tollgate-module-basic-go is the single source of truth; the fork issue carries the original discussion and any lab evidence. Retirement plan: see the tracking issue linked in the migration comment there.]

Evidence (local virtual lab, 2026-09-04, cdk-mintd 0.18.0 fakewallet)

Live probes during a failing tests/api/test_merchant_provider.py run:

  1. Wallet state vs prober disagree. After the mint work-dir is wiped and restarted (same mnemonic → same keysets), discovery served kind 21023 no-reachable-mints while:

    • proactive prober: mint probe: url=http://10.99.99.2:8383/v1/info status=200 ok=true (twice, 5 min apart)
    • curl http://10.99.99.2:8383/v1/info from the router: 200
    • tollgate status reported wallet_ok=true while discovery said degraded
      Only /etc/init.d/tollgate-wrt restart flipped discovery to kind 10021 (~10s). The wallet's cached mint state (wallet.db) is not re-validated on prober success.

  2. Health transitions ride the 300s proactive poll. iptables-blocking all mints does not flip discovery to degraded until the next probe cycle (up to 5 min); same for unblock→full. Test budgets of 180s expired before the first probe fired. (Worked around test-side: HEALTH_POLL_TIMEOUT 420, module timeout 900 — prta fix/local-lab-green 90bf9c6.)

  3. Recovery after repeated block/unblock cycles exceeds any test budget. In one file run: cycle 1 recovered well within 420s; cycles 2–4 did not recover within 420s (Service did not recover, swap-test stop reason test window ended, recovered=False). Escalating backoff on the mint URL after repeated failures is the likely mechanism; it does not reset on unblock.

Impact

  • Degraded/recovery tests are timing-flaky against any mint that restarts (mints DO restart — new deployments, CDK upgrades).
  • Discovery can serve no-reachable-mints while the mint is healthy, for up to the next probe cycle + wallet re-validation delay (currently: forever, absent a backend restart).

Suggested direction

  • Event-driven health transition: flip degraded/full immediately on swap/probe failure+recovery (probe success already detected per-cycle — propagate it to the wallet/discovery state).
  • Reset per-mint backoff when a probe or swap succeeds.
  • Re-validate cached wallet mint state when the prober reports ok=true but the wallet considers the mint down.

Activity

  1. Amperstrand
    added a commit that references this issue on Oct 8, 2026

    fix(merchant): event-driven mint-health transitions — recovery in sec…

    4c66719

  2. Amperstrand commented on Oct 9, 2026

    @Amperstrand
    Amperstrand
    on Oct 9, 2026
    CollaboratorAuthor

    Rel-verify verdict on v0.6.0-rc1 + PR#777: FIXED — event-driven mint-health transitions verified dynamically on the QEMU rig (OpenWrt 25.12.0 x86-64, pristine snapshot boots, single-mint config against a controllable mock mint serving NUT-01 keysets; outage = HTTP 503). Issue #747's findings 1+3 reproduced on pristine rc1, then eliminated on the PR build. All times UTC from logread + mint-side request log.

    Patch path used: real build — PR head 4c66719 (fix/747-mint-health-transitions) through the pinned SDK pipeline (scripts/build-sdk-package.sh, go1.26.8, openwrt/sdk x86-64 digest-pinned) → tollgate-wrt-0.6.0_pre777-r0.apk, sha256 fbfc2263be211af23f9d1b4df65b1cbdf2235280fff3af00794fac085de4eab3. Baseline = shipped rc1 bytes (d1b1ba5e…, commit 659de8a).

    Before (pristine rc1):

    • outage → degraded: 13 m 12 s (failureThreshold 3 × 300 s polls — unchanged by the PR, by design).
    • restore after the one-shot 5-min aggressive window expired → healthy: 13 m 07 s — poll OK at +3 m (1/3), +8 m (2/3), +13 m (3/3) while :2121 kept serving kind 21023 (finding 1's signature, live for 10 minutes). First-probe-OK → readmit = 10 m 00 s.
    • This is the mechanism behind finding 3: the aggressive loop is one-shot per downgrade; an outage that outlives the window leaves recovery on the poll cadence with 3-confirmation threshold — every repeated cycle, which is why cycles 2–4 blew the 420 s budget.

    After (PR #777):

    • Same dead-end schedule (degrade → aggressive period ended → restore): poll-OK → arming the recovery-burst loop → readmitted in 15 s, every cycle — restore→healthy 3 m 43 s / 4 m 28 s / 3 m 32 s across 3 consecutive cycles (the residual is purely the wait for the next scheduled 300 s poll, which the PR intentionally keeps).
    • No escalation residue: cycle 3 followed a 21-minute sustained outage and recovered identically (15 s from probe-OK).
    • Sustained outage (21 m 06 s): no busy-loop — 15 s burst only while a recovery is pending (exactly 5 min, mint-side hits every 15.000 s), 300 s cadence otherwise; 29 probes total.
    • Restart mid-outage: sane degraded-from-boot (no stale reachable set), window expiry while down, then restore → poll-OK → burst → wallet re-init + Upgrading from degraded to full merchant in 16 s.
    • Regression: logread clean of new error classes (0 panic/fatal; only the by-design degraded-wallet line also present in rc1); NDS + nft enforcement stable through all 5 transitions; ad flipped 10021↔21023 exactly at logged set-changes, no flapping between.

    Scope notes: the MarkReachable swap-success path needs a real purchase flow and is covered by the PR's own tests, not this probe-driven rig leg. PR head = main@cb4db7f + 4c66719 (src/merchant delta vs rc1 is exactly 4c66719; base carries 3 unrelated commits: brands registry, :2121 content-type, portal pin).

    Full transcript + raw logs: ~/tollgate-vm/rel-777/ on ai-legion (lane rc1-777).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions