Skip to content

test(tollwallet): tolerate an unreachable stored mint at startup - #1142

Open
c03rad0r wants to merge 1 commit into
mainfrom
fix/stale-mint-resilience
Open

c03rad0r wants to merge 1 commit into
mainfrom
fix/stale-mint-resilience

Conversation

@c03rad0r

Copy link
Copy Markdown
Contributor

Why

A wallet DB can still hold a keyset (and a Cashu proof) for a mint that is currently down. In the field, mint.minibits.cash returned EOF while five other accepted mints were healthy; wallet.LoadWallet() failed on that stale keyset and the merchant stayed in degraded mode, so the captive portal showed no-reachable-mints and could not sell access.

The underlying fix is in gonuts: isNetworkError() did not recognise io.EOF, so loadWalletMints() treated the unreachable mint as fatal instead of skipping the keyset and retaining its DB record/proof.

Test

src/tollwallet/stale_mint_test.go — TestNewToleratesUnreachableStoredMint

  • seed a wallet DB with a keyset, derivation counter and 1-sat proof for a mint that accepts the TCP connection and closes it (EOF);
  • add a healthy mint as the second accepted mint;
  • New(dir, []string{healthy, dead}, false) must succeed and register the healthy mint;
  • the dead mint’s record is retained by the underlying wallet (no delete, no overwrite).

RED on v0.14.1:

--- FAIL: TestNewToleratesUnreachableStoredMint
    New must tolerate an unreachable stored mint, got:
    failed to create wallet: error getting keyset from mint:
    Get "http://127.0.0.1:…/v1/keys/…": EOF

GREEN with the fix:

ok  github.com/OpenTollGate/tollgate-module-basic-go/src/tollwallet

Note

TestV2Keyset_BothIdSpellingsReachSwapWithFullId and TestV2Keyset_V4TokenShortIdResolves also fail on unmodified v0.14.1 in this workspace; they are pre-existing and unrelated to this change.

Follow-up (separate)

GetBalanceByMints() reports 0 for a mint whose keysets were all skipped, even though the proofs remain in the DB. A follow-up could retain the offline mint in walletMints so the balance (and the portal) shows the stranded amount. The fund-safety guarantee — records and proofs are kept — is covered here.

Add TestNewToleratesUnreachableStoredMint: seed a wallet DB with a keyset, counter and proof for a mint that accepts connections and closes them (EOF), plus a healthy mint; assert tollwallet.New() succeeds, the reachable mint is registered, and the dead mint's record is retained by the underlying wallet.

Bump gonuts-tollgate to the isNetworkError(EOF) fix from OpenTollGate/gonuts-tollgate#45 (v0.14.2-0.20261030003712-ae77396240a2). On v0.14.1 the test fails with the exact production error: 'failed to create wallet: error getting keyset from mint: Get ...: EOF'; with the fix it passes.

Note: TestV2Keyset_BothIdSpellingsReachSwapWithFullId and TestV2Keyset_V4TokenShortIdResolves fail on unmodified v0.14.1 as well; they are pre-existing and unrelated to this change.
@c03rad0r

Copy link
Copy Markdown
Contributor Author

E750 A/B verification (physical GL-E750)

Cross-references: gonuts-tollgate#45 · tollgate-module-basic-go#1142 · physical-router-test-automation#330

The unit/integration tests were reproduced on the real router with the actual wallet.db that triggered the report. No DB reset: the 1 sat held at minibits had to survive.

Setup

wallet.db        keeps two inactive minibits keysets (00107937db0cc865, 00500550f0494146)
accepted mints   coinos, macadamia, westernbtc, kashu.me, cubabitcoin
stale mint       mint.minibits.cash -> EOF (mint down)
binary           v0.6.0 tag, GOOS=linux GOARCH=mips GOMIPS=softfloat, CGO_ENABLED=0

A — old binary v0.6.0-g9364f2e (md5 09a2f0b781be717086569e52eed3f147)

GET / -> {"kind":21023, tags: [["level","warning"],["code","no-reachable-mints"]],
          content:"TollGate is initializing. No reachable mints detected."}
log    WARNING: Wallet initialization failed (failed to create wallet:
       error getting keyset from mint: Get ".../v1/keys/00107937db0cc865": EOF)
       — starting in degraded mode
       Merchant started in degraded mode

This is the reported failure.

B — fixed binary (md5 286b8f28eb8bbeece43d85bfe7dedbbe)

log    Skipping keyset 00107937db0cc865 until connection is restored.
       Skipping keyset 00500550f0494146 until connection is restored.
       TollWallet: registered mint https://mint.macadamia.cash/
       ... westernbtc / kashu.me / cubabitcoin
GET / -> {"kind":10021, tags: price_per_step ...}
portal http://10.0.0.1:2051/splash.html -> 200
wallet Total: 1 sats across 1 mints
       https://mint.minibits.cash/Bitcoin: 1

The stale keysets are skipped, the healthy mints are registered, the merchant is in full mode, and the 1 sat is still visible and spendable when minibits returns. wallet.db was never modified.

Reproduce

# gonuts
go test ./wallet/ -run 'TestIsNetworkErrorEOF|TestLoadWalletSkipsUnreachableStoredMint' -count=1

# module
cd src/tollwallet && go test . -run TestNewToleratesUnreachableStoredMint -count=1

# PRTA (local QEMU lab or physical)
python3 -m pytest tests/api/test_mint_stale_keyset_offline.py -q

@c03rad0r

Copy link
Copy Markdown
Contributor Author

Local QEMU lab A/B — the regression test catches the bug

Ran tests/api/test_mint_stale_keyset_offline.py on the PRTA local lab
(OpenWrt 24.10.1 x86_64, QEMU/KVM) with the same wallet state as the field
report: an accepted mint that advertises an inactive keyset stored without
public keys, then goes EOF.

Cross-references: OpenTollGate/gonuts-tollgate#45 · #1142

A — stock tollgate-wrt v0.6.0 (md5 5b12703af453cc11fa00a4020f0d92fb)

FAILED tests/api/test_mint_stale_keyset_offline.py::test_stale_keyset_survives_unreachable_mint
E   AssertionError: {"kind":21023,"id":"d200aa24…","tags":[["level","warning"],
    ["code","no-reachable-mints"]],"content":"TollGate is initializing. No reachable mints"
E   assert 'no-reachable-mints' not in …
1 failed in 68.75s

The test fails exactly at the portal-mode assertion — the reported symptom.

B — v0.6.0 + gonuts isNetworkError(EOF) (md5 32a53cd9cb2fd26ec302b154bc2704fe)

tests/api/test_mint_stale_keyset_offline.py .
1 passed in 53.83s

With the fix the stale keyset is skipped, the wallet initializes, the backend
serves kind 10021 and the inactive keyset record stays in wallet.db.

Test is now self-contained

The fixture no longer needs TOLLGATE_LOCAL_MINT_URL: it starts two
FakeMintServers (healthy + stale), so the case runs on any lab with no
cdk-mintd and no external mint. Also fixed the fake NUT-01 reply: /v1/keys
returns active keysets with keys (an entry without a keys object made
gonuts fail with unexpected end of JSON input), and /v1/keys/<id> now
returns a distinct, derivable inactive keyset.

Reproduce

# local QEMU lab
python3 scripts/virtual-lab.py start-poc --host localhost --workdir ~/tollgate-virtual-lab-2410
TOLLGATE_VIRTUAL_LAB=1 TOLLGATE_VIRTUAL_HOST=10.99.99.2 \
TOLLGATE_SSH_HOST=10.99.99.1 TOLLGATE_SSH_PASSWORD=… \
  python3 -m pytest tests/api/test_mint_stale_keyset_offline.py -q

(If the host firewall drops the lab bridge, allow 10.99.99.0/24 on the lab
interface — that was the only environment-specific step here.)

@c03rad0r

Copy link
Copy Markdown
Contributor Author

Follow-up: reproducible device build (same results)

The E750 runs TollGate v0.6.0, which pins gonuts v0.13.0, while this PR targets main (v0.14.x). The device binary is therefore the v0.13.0 backport of the same change, built by e750-repair/scripts/build-tollgate-wrt.sh + patches/gonuts-isnetworkerror-eof.patch.

Rebuilt and reinstalled on the GL-E750 (no wallet.db change):

md5 /usr/bin/tollgate-wrt   2a73aa2fc90a9b269d9b3c256bfc5f08
GET /                       "kind":10021
ubus call tollgate health   config_ok true, status ok, wallet_ok true
portal :2051/splash.html    200
tollgate wallet info        Total: 1 sats across 1 mints
                            https://mint.minibits.cash/Bitcoin: 1
logread                     Skipping keyset 00107937db0cc865 …
                            Skipping keyset 00500550f0494146 …
e750-health                 RESULT OK

The QEMU-lab RED/GREEN result is unchanged (the lab binary is the same source built for amd64). Cross-refs: gonuts#45, module#1142, PRTA#330, e750-repair#45.

@Amperstrand

Copy link
Copy Markdown
Collaborator

Local CI gate — PR #1142: test(tollwallet): tolerate an unreachable stored mint at startup

Run of scripts/pr-verify.sh (the local CI gate: GitHub Actions is
org-blocked and the ngit lane fires only on maintainer mirror pushes,
so a PR ref gets no CI runs — this comment is the offline verification
evidence, produced on the pinned head SHA).

  • Head: fix/stale-mint-resilience @ a672c70606
  • Base: main @ 6aabea2d36
  • Gate: pr-verify 294317251383 on ai-legion, started 2026-10-30 20:59 UTC, wall 116s
  • Logs: /tmp/tollgate-pr-verify/1142-20261030T205727Z (on the verifying host)
Leg Result Detail
hygiene PASS no conflict markers in the tracked tree (0s)
contract-fences FAIL red: changelog fragments (assemble check) doc-facts contract build purity dependency sync gonuts pin carriers — see contract-fences.log; 13 checks green (6s)
packaging-sequential FAIL red (3/17): backend-api-owner-network_test.sh entry-ui-install-sequence_test.sh rebrand-literal-gutter_test.sh — logs in /tmp/tollgate-pr-verify/1142-20261030T205727Z/packaging-sequential (17s)
shell-suites FAIL red (2/23): uci-defaults-gatewayname-banner_test.sh uci-defaults-lan-private-wired_test.sh — logs in /tmp/tollgate-pr-verify/1142-20261030T205727Z/shell-suites (85s)
packaging-concurrent FAIL red (3/17): backend-api-owner-network_test.sh entry-ui-install-sequence_test.sh rebrand-literal-gutter_test.sh — logs in /tmp/tollgate-pr-verify/1142-20261030T205727Z/packaging-concurrent (8s)
go-battery FAIL 19 modules — see go-battery.log (0s)

Verdict: 5 leg(s) RED — 1 green.

What this gate does not run (no vacuous green)
  • spec-quote drift check (greatspectations tool + pinned nuts fetch)
  • reproducibility / conformance lanes (make release-check territory)
  • router-visible behaviour (PRTA / bench — unit green ≠ router works)
failing-leg log tails
contract-fences — tail

🔴 google.golang.org/grpc — 2 versions:
   root: v1.83.2
   tollwallet: v1.82.1

Fix: update all go.mod files to the same version.
     Run 'go mod tidy' in each sub-module after updating.
== gonuts pin fence (offline identity)
ok   an aligned tree (block + single-line carriers) passes
ok   a block-form carrier on a different version
ok   a single-line carrier left behind (the token-recovery shape)
ok   a go.sum that never recorded the truth version (half-update)
ok   a self-consistent pseudo-version integration pin passes OFFLINE (the --latest leg is the release gate for it)
ok   a manifest without the .gonuts key
ok   a tree whose manifest names a pin no carrier has

7 cases, 0 failed
== gonuts pin carriers
  manifest pin (packaging/build-inputs.json .gonuts.version): v0.14.1
  carriers discovered by glob: 5 module(s)

❌ GONUTS PIN DRIFT — 1 finding(s):
  🔴 src/tollwallet/go.mod: pins github.com/OpenTollGate/gonuts-tollgate v0.14.2-0.20261030003712-ae77396240a2, manifest says v0.14.1 — run scripts/bump-gonuts.sh v0.14.1

Fix: scripts/bump-gonuts.sh <version> rewrites the manifest and every carrier together. Never edit one go.mod by hand — that is how the pin drifted (#791).
== mirror/relay sync
ok    BLOSSOM_SERVERS: 13 copies identical (https://blossom.primal.net https://blossom.psbt.me https://blossom2.orangesync.tech)
ok    relay list: 13 copies identical (wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://relay2.orangesync.tech)
== stale import paths
✅ All 359 Go source files use correct import paths.
packaging-sequential — tail
--- backend-api-owner-network_test.sh.log
ok   the fragment guards exactly :2121
ok   packaging/files/etc/nftables.d/30-backend-firewall.nft is an include fragment (no own table declaration)
ok   the chain hooks input at priority -1 (the drop is authoritative)
ok   chain 'backend_input_firewall' is declared once
== the admin-board and LuCI guards are still captive-bridge-scoped
ok   31-admin-board-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
ok   32-luci-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
== the recipe installs the rule
ok   local-build-ipk.sh installs 30-backend-firewall.nft
ok   packaging/Makefile installs the 30-backend-firewall.nft rule via its *.nft glob
ok   packaging/Makefile FILES_ lists /etc/nftables.d/30-backend-firewall.nft
== the fragment parses
FAIL nft -c rejects the fragment — 'nft -f' would fail the whole fw4 reload

passed=17 failed=1
--- entry-ui-install-sequence_test.sh.log

== scenario 4b: converged box, root hash emptied — 999 re-refuses the board
ok   scenario 4b: :8080 is unbound — the refused board's own pair
ok   scenario 4b: :443 is unbound — the refused board's own pair
ok   scenario 4b: exactly one section binds :8090 (main)
ok   scenario 4b: exactly one section binds :8443 (main)
ok   scenario 4b: uhttpd.main keeps at least one listener
ok   scenario 4b: uhttpd.portal keeps at least one listener
ok   scenario 4b: uhttpd.trusted keeps at least one listener
ok   scenario 4b: the admin board is deliberately refused (fail-closed gate), not torn
ok   scenario 4b: LuCI (uhttpd.main) answers :8090 on both families
ok   scenario 4b: LuCI (uhttpd.main) answers :8443 on both families
ok   scenario 4b: the board is refused — its pair is intentionally unbound

tests: 72 passed, 1 failed
--- rebrand-literal-gutter_test.sh.log

== negative control: the scan pattern sees a planted occurrence
ok   the pattern matches a planted occurrence (the gutter is not inert)
ok   the pattern does not match a near-miss brand name (it is anchored to the banned literal)
ok   scan_tree ignores the control file (it is outside the tracked tree)

== the shipped uci-defaults reference only known uhttpd sections
ok   every uhttpd.<name> in the uci-defaults' active code is a known section (or a cert/key path)

== control: the section tokenizer ignores comments, not code
ok   comment-only section names (full-line and trailing) are not flagged
ok   known sections in active code are still extracted
ok   a foreign section in active code is still extracted (the check keeps its teeth)

tests: 8 passed, 1 failed
shell-suites — tail
--- uci-defaults-gatewayname-banner_test.sh.log
ok   the FULL path's banner IS the canonical one from the shared table
ok   reinstall (same version) run exits 0
ok   the same-version reinstall took the VERIFY branch
ok   the repair path COMMITS the banner '!TollGate-OQ3Q Portal'
ok   the repair path's banner is the SAME value the full path writes
ok   no gatewayname op is left pending in the /tmp/.uci-style delta
FAIL the repair path's captive SSID converged on the same code (got '!tollgate-0GLK', want '!TollGate-OQ3Q')
FAIL the banner is exactly '<captive SSID> Portal' (got '!TollGate-OQ3Q', want '!tollgate-0GLK')
ok   the operator's PSK is untouched
ok   drifting reinstall run exits 0
ok   the drifting repair path commits the SAME banner
ok     ...and the drifted allow-list rule was repaired (the commit did fire)
ok     ...with no gatewayname op left pending

27 passed, 2 failed
--- uci-defaults-lan-private-wired_test.sh.log
ok   silent no-op: the log names the missing port
== empty source and empty target: reported as an error, nothing written
ok   empty source+target: writer returns non-zero (1)
ok   empty source+target: private_bridge still has no ports
ok   empty source+target: log reports the broken state
== bridge device section beyond index 23 is still found and moved
ok   late bridge: writer returns 0
ok   network.private_bridge lists eth1 exactly once
ok   late bridge: the source section was cleared
== negative control: a script with a no-op writer fails the assertions
ok   negative control: the writer is neutralised in the copied script
ok   negative control: without the writer the port stays on the captive bridge
ok   negative control: the captive bridge keeps its port (the move assertion would fail)

lan-private-wired: 8 check(s) FAILED, 35 passed
packaging-concurrent — tail
--- backend-api-owner-network_test.sh.log
ok   the fragment guards exactly :2121
ok   packaging/files/etc/nftables.d/30-backend-firewall.nft is an include fragment (no own table declaration)
ok   the chain hooks input at priority -1 (the drop is authoritative)
ok   chain 'backend_input_firewall' is declared once
== the admin-board and LuCI guards are still captive-bridge-scoped
ok   31-admin-board-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
ok   32-luci-not-guest-reachable.nft is scoped by $tg_portal_if (the captive bridge's define)
== the recipe installs the rule
ok   local-build-ipk.sh installs 30-backend-firewall.nft
ok   packaging/Makefile installs the 30-backend-firewall.nft rule via its *.nft glob
ok   packaging/Makefile FILES_ lists /etc/nftables.d/30-backend-firewall.nft
== the fragment parses
FAIL nft -c rejects the fragment — 'nft -f' would fail the whole fw4 reload

passed=17 failed=1
--- entry-ui-install-sequence_test.sh.log

== scenario 4b: converged box, root hash emptied — 999 re-refuses the board
ok   scenario 4b: :8080 is unbound — the refused board's own pair
ok   scenario 4b: :443 is unbound — the refused board's own pair
ok   scenario 4b: exactly one section binds :8090 (main)
ok   scenario 4b: exactly one section binds :8443 (main)
ok   scenario 4b: uhttpd.main keeps at least one listener
ok   scenario 4b: uhttpd.portal keeps at least one listener
ok   scenario 4b: uhttpd.trusted keeps at least one listener
ok   scenario 4b: the admin board is deliberately refused (fail-closed gate), not torn
ok   scenario 4b: LuCI (uhttpd.main) answers :8090 on both families
ok   scenario 4b: LuCI (uhttpd.main) answers :8443 on both families
ok   scenario 4b: the board is refused — its pair is intentionally unbound

tests: 72 passed, 1 failed
--- rebrand-literal-gutter_test.sh.log

== negative control: the scan pattern sees a planted occurrence
ok   the pattern matches a planted occurrence (the gutter is not inert)
ok   the pattern does not match a near-miss brand name (it is anchored to the banned literal)
ok   scan_tree ignores the control file (it is outside the tracked tree)

== the shipped uci-defaults reference only known uhttpd sections
ok   every uhttpd.<name> in the uci-defaults' active code is a known section (or a cert/key path)

== control: the section tokenizer ignores comments, not code
ok   comment-only section names (full-line and trailing) are not flagged
ok   known sections in active code are still extracted
ok   a foreign section in active code is still extracted (the check keeps its teeth)

tests: 8 passed, 1 failed
go-battery — tail
=== module: src
gofmt: unformatted files (fix with: gofmt -w .):
merchant/unmeterable_session_cadence_test.go
make: *** [Makefile:13: go-battery] Error 1

@felixfelix-bot felixfelix-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the diagnosis and the fork fix are right, and the RED/GREEN differential is real evidence rather than a claim. Two things need to land before this can merge; one of them currently breaks main.

1. BLOCK — the gonuts pin moved in one carrier, not all of them

src/tollwallet/go.mod goes to v0.14.2-0.20261030003712-ae77396240a2, but packaging/build-inputs.json (.gonuts.version) and the other four carriers stay at v0.14.1. That is the #791 drift shape, and it is not cosmetic — it fails two contract jobs and the root module build:

$ python3 tests/contract/check-gonuts-pin.py
  manifest pin (packaging/build-inputs.json .gonuts.version): v0.14.1
  carriers discovered by glob: 5 module(s)
❌ GONUTS PIN DRIFT — 1 finding(s):
  🔴 src/tollwallet/go.mod: pins github.com/OpenTollGate/gonuts-tollgate v0.14.2-0.20261030003712-ae77396240a2, manifest says v0.14.1
$ python3 tests/contract/check-deps-sync.py
🔴 github.com/OpenTollGate/gonuts-tollgate — 2 versions:
   cli: v0.14.1
   merchant: v0.14.1
   root: v0.14.1
   tollwallet: v0.14.2-0.20261030003712-ae77396240a2
$ cd src && go build ./...
go: updates to go.mod needed; to update it:
	go mod tidy

Both fences run in the deps-and-imports job of .github/workflows/test.yml, so CI will be red on the first run. The build failure is the sharp one: src/ is the root module and it still requires v0.14.1 while src/tollwallet requires the pseudo-version, so the module graph no longer resolves. The same command on 6aabea2d (this PR's base) exits 0.

I staged the mechanical fix on fix/stale-mint-resilience in my fork rather than describing it — manifest plus all five carriers (scripts/token-recovery, src/cli, src, src/merchant, src/tollwallet) to the same pseudo-version, then go mod tidy per module:

felixfelix-bot@7073012

That also resolves two stale go.sum fragments in src/tollwallet/go.sum — the superseded v0.14.1 pair, and a v0.15.1-0.20261030003228-d6b9e39c686a pair that no go.mod in the tree names (looks like carry-over from a different branch; worth confirming it was intentional).

After it:

✅ one pin, everywhere: github.com/OpenTollGate/gonuts-tollgate v0.14.2-… across 5 carrier(s).
gonuts-pin_test.sh            -> 7 cases, 0 failed
cd src && go build ./...      -> 0   (was 1)
go vet ./...                  -> 0

check-deps-sync.py still reports 8 entries, but they are byte-identical to the base commit's set (Go directive on tollgate-ndsctl, x/mod, x/sync, x/tools, grpc, genproto×2) — pre-existing, and out of scope here.

Pseudo-version rather than the tag is right, and I checked it rather than assuming: gonuts-tollgate#45 is still open, and v0.15.0 does not carry the fix — isNetworkError at that tag still has no io.EOF arm. So bumping to the newest tag would silently drop the fix. One flag for whoever merges: #1151 (deps/gonuts-v0.15.0) moves the same files, so these two will conflict and need sequencing.

2. SHOULD-FIX — the test does not assert the retention it claims

The body says the dead mint's record is retained ("no delete, no overwrite"), and the test seeds a counter and a proof — but it never reads either back. The only assertions are that New() returned no error and the healthy mint was registered:

https://github.com/OpenTollGate/tollgate-module-basic-go/blob/a672c706/src/tollwallet/stale_mint_test.go#L100-L109

That gap is load-bearing, not theoretical. Deleting both the IncrementKeysetCounter(deadID, 2) and the SaveProofs(...) seed leaves the test green:

$ go test -run TestNewToleratesUnreachableStoredMint .    # counter+proof seeding removed
ok  …/src/tollwallet  16.136s

The claim itself does hold — I verified it independently by reading the DB back after New() (counter=2, keyset present, proofs=1 for deadID). But nothing in the suite enforces it, so a future regression that clobbers the counter or drops the proof would pass silently. Given AGENTS.md's monotonic-counter rule (#480), that is the exact class this test should be pinning. Suggested close: after tw.Shutdown(), reopen storage and assert GetKeysetCounter(deadID) == 2 and len(GetProofsByKeysetId(deadID)) == 1, in the shape the fork's own load_unreachable_mint_test.go already uses.

Test evidence

Same harness (.github/scripts/go-test-summary.sh), this PR's head vs its base:

  • base 6aabea2d: 83 passed / 4 failed / 3 skipped
  • head a672c706: 84 passed / 4 failed / 3 skipped
  • failure set byte-identical: TestV2Keyset_BothIdSpellingsReachSwapWithFullId (+ its two subtests) and TestV2Keyset_V4TokenShortIdResolves — the two the body calls pre-existing, confirmed pre-existing here.

RED/GREEN differential on the new test, done by reverting only the pin:

  • head: --- PASS: TestNewToleratesUnreachableStoredMint (16.87s)
  • go.mod/go.sum reverted to v0.14.1: --- FAIL: stale_mint_test.go:102: New must tolerate an unreachable stored mint, got: failed to create wallet: error getting keyset from mint: Get "http://127.0.0.1:40701/v1/keys/0092812ee35fa269": EOF

Also: no CI has ever run on this branch (no checks reported on 'fix/stale-mint-resilience'), so CI status is unknown rather than green.

Happy to open a small PR carrying the pin fix against your branch if that is easier than rebasing it yourself — say which you prefer.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

Thanks — the diagnosis and the fork fix are right, and the RED/GREEN differential is real evidence rather than a claim. Two things need to land before this can merge; one of them currently breaks main.

1. BLOCK — the gonuts pin moved in one carrier, not all of them

src/tollwallet/go.mod goes to v0.14.2-0.20261030003712-ae77396240a2, but packaging/build-inputs.json (.gonuts.version) and the other four carriers stay at v0.14.1. That is the #791 drift shape, and it is not cosmetic — it fails two contract jobs and the root module build:

$ python3 tests/contract/check-gonuts-pin.py
  manifest pin (packaging/build-inputs.json .gonuts.version): v0.14.1
  carriers discovered by glob: 5 module(s)
❌ GONUTS PIN DRIFT — 1 finding(s):
  🔴 src/tollwallet/go.mod: pins github.com/OpenTollGate/gonuts-tollgate v0.14.2-0.20261030003712-ae77396240a2, manifest says v0.14.1
$ python3 tests/contract/check-deps-sync.py
🔴 github.com/OpenTollGate/gonuts-tollgate — 2 versions:
   cli: v0.14.1
   merchant: v0.14.1
   root: v0.14.1
   tollwallet: v0.14.2-0.20261030003712-ae77396240a2
$ cd src && go build ./...
go: updates to go.mod needed; to update it:
	go mod tidy

Both fences run in the deps-and-imports job of .github/workflows/test.yml, so CI will be red on the first run. The build failure is the sharp one: src/ is the root module and it still requires v0.14.1 while src/tollwallet requires the pseudo-version, so the module graph no longer resolves. The same command on 6aabea2d (this PR's base) exits 0.

I staged the mechanical fix on fix/stale-mint-resilience in my fork rather than describing it — manifest plus all five carriers (scripts/token-recovery, src/cli, src, src/merchant, src/tollwallet) to the same pseudo-version, then go mod tidy per module:

felixfelix-bot@7073012

That also resolves two stale go.sum fragments in src/tollwallet/go.sum — the superseded v0.14.1 pair, and a v0.15.1-0.20261030003228-d6b9e39c686a pair that no go.mod in the tree names (looks like carry-over from a different branch; worth confirming it was intentional).

After it:

✅ one pin, everywhere: github.com/OpenTollGate/gonuts-tollgate v0.14.2-… across 5 carrier(s).
gonuts-pin_test.sh            -> 7 cases, 0 failed
cd src && go build ./...      -> 0   (was 1)
go vet ./...                  -> 0

check-deps-sync.py still reports 8 entries, but they are byte-identical to the base commit's set (Go directive on tollgate-ndsctl, x/mod, x/sync, x/tools, grpc, genproto×2) — pre-existing, and out of scope here.

Pseudo-version rather than the tag is right, and I checked it rather than assuming: gonuts-tollgate#45 is still open, and v0.15.0 does not carry the fix — isNetworkError at that tag still has no io.EOF arm. So bumping to the newest tag would silently drop the fix. One flag for whoever merges: #1151 (deps/gonuts-v0.15.0) moves the same files, so these two will conflict and need sequencing.

2. SHOULD-FIX — the test does not assert the retention it claims

The body says the dead mint's record is retained ("no delete, no overwrite"), and the test seeds a counter and a proof — but it never reads either back. The only assertions are that New() returned no error and the healthy mint was registered:

https://github.com/OpenTollGate/tollgate-module-basic-go/blob/a672c706/src/tollwallet/stale_mint_test.go#L100-L109

That gap is load-bearing, not theoretical. Deleting both the IncrementKeysetCounter(deadID, 2) and the SaveProofs(...) seed leaves the test green:

$ go test -run TestNewToleratesUnreachableStoredMint .    # counter+proof seeding removed
ok  …/src/tollwallet  16.136s

The claim itself does hold — I verified it independently by reading the DB back after New() (counter=2, keyset present, proofs=1 for deadID). But nothing in the suite enforces it, so a future regression that clobbers the counter or drops the proof would pass silently. Given AGENTS.md's monotonic-counter rule (#480), that is the exact class this test should be pinning. Suggested close: after tw.Shutdown(), reopen storage and assert GetKeysetCounter(deadID) == 2 and len(GetProofsByKeysetId(deadID)) == 1, in the shape the fork's own load_unreachable_mint_test.go already uses.

Test evidence

Same harness (.github/scripts/go-test-summary.sh), this PR's head vs its base:

  • base 6aabea2d: 83 passed / 4 failed / 3 skipped
  • head a672c706: 84 passed / 4 failed / 3 skipped
  • failure set byte-identical: TestV2Keyset_BothIdSpellingsReachSwapWithFullId (+ its two subtests) and TestV2Keyset_V4TokenShortIdResolves — the two the body calls pre-existing, confirmed pre-existing here.

RED/GREEN differential on the new test, done by reverting only the pin:

  • head: --- PASS: TestNewToleratesUnreachableStoredMint (16.87s)
  • go.mod/go.sum reverted to v0.14.1: --- FAIL: stale_mint_test.go:102: New must tolerate an unreachable stored mint, got: failed to create wallet: error getting keyset from mint: Get "http://127.0.0.1:40701/v1/keys/0092812ee35fa269": EOF

Also: no CI has ever run on this branch (no checks reported on 'fix/stale-mint-resilience'), so CI status is unknown rather than green.

Happy to open a small PR carrying the pin fix against your branch if that is easier than rebasing it yourself — say which you prefer.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

Following up on the earlier review — I independently reproduced findings 1 and 2 (pin half-update breaks cd src && go build, manifest+4 carriers still at v0.14.1; and the retention claim is unasserted — the test seeds counter=2 and a proof and never reads either back, though I confirmed by reading the DB back after New() that the behaviour itself is correct: counter=2, proofs=1). No need to repeat those here.

One thing neither that review nor the PR body covers, and which changes the shape of the fix: the ordering the test does not exercise is the one that regresses.

TestNewToleratesUnreachableStoredMint passes []string{healthy, dead}, so the unreachable mint is acceptedMints[1]. When the down mint is listed first it becomes wallet.Config.CurrentMintURL (tollwallet.go:80), and at this pin gonuts' offline path returns the wallet early without setting isErr = false, so its own deferred if isErr { db.Close() } (gonuts wallet/wallet.go:90-97) closes the DB of the wallet it just handed back.

Self-contained probe, identical fixtures, both orders, no external network (-race, 2/2 identical runs at head):

                          dead-second                dead-first (dead == CurrentMintURL)
base 6aabea2d  New()      ok                         REFUSED: error adding new mint:
                                                     error getting active keysets from mint: …/v1/keysets: EOF

head a672c706  New()      ok                         ok   ← returns success
               healthyRegistered   true                 false
               healthyAfterRetry   true                 false       (registerMint retried, still fails)
               GetBalanceByMints   {healthy:0}            map[]       (no mint entries at all)
               log                 —                    failed to register mint …: database not open

So on this ordering the PR converts base's loud refusal into a quiet broken wallet: New() succeeds, the portal receives a wallet with zero registered mints and an empty balance map, and the merchant looks healthy while being unable to sell — a harder failure to diagnose than the no-reachable-mints this PR set out to remove.

To be precise about causality, because my first pass got this wrong: the closed-DB defect is pre-existing in the fork, and connection refused reaches it on base too. But on base that path is only reached after New() has already failed. io.EOF was the one error class that kept the failure loud. Classifying it as a network error is what routes the exact field failure into the broken path — so the fork bug is inherited, and the regression is in how this PR reaches it.

Two closes, either acceptable:

  1. Reject at the seam (small, fits in this PR): in tollwallet.New, treat "zero mints successfully registered" as a construction failure. It is indistinguishable from nothing working, and right now the code hides it.
  2. Fix the fork: both offline early-returns in LoadWallet set isErr = false before returning, so the degraded wallet is actually usable. Needs a gonuts PR + tag.

Either way please add the ordering case: []string{dead.URL, healthy.URL}, asserting New either errors or returns a wallet that can register the healthy mint. That is a ~10-line extension of the test this PR already adds.

Two sequencing notes, both verified rather than assumed:

Evidence: /home/c08r4d0r/reports/tmbg-pr-sweep/pr1142/ — REPORT.md, logs/probe-order-{head,base}.log (both orders, raw), logs/tollwallet-full-{head,base,fixed}.log, logs/gonuts-pin-{HEAD-FAIL,fixed}.log, and pr1142-pin-fix.diff (the 10-file pin fix, which does turn the contract fence and both build lanes green).

@felixfelix-bot felixfelix-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review delta — independently reproduced, verdict unchanged: BLOCK.

I re-ran the full pass on a672c706 in a clean worktree (own worktree, -race, serialized Go lanes). Confirming the prior review rather than repeating it:

  • Pin half-update — reproduced. check-gonuts-pin.py → ❌ GONUTS PIN DRIFT … manifest says v0.14.1, rc=1. Root-module build breaks: cd src && go build ./... → go: updates to go.mod needed, rc=1 (base 6aabea2d: rc=0). Release lane go build -C src -o … -trimpath -buildvcs=false . (build-package.yml:355) rc=1, and the main package (testenv) lane (test.yml:77) rc=1 — 227/227 pass once the pin is aligned. The 10-file fix (manifest + all five carriers + go mod tidy) turns the fence green and every lane to rc=0.
  • Retention unasserted — reproduced, and the behaviour is correct. Test seeds counter=2 and a 1-sat proof and reads neither back. Reading the DB back after New()/Shutdown() gives counter=2, proofs=1, keyset record present — so the claim holds, it is just not pinned. Given AGENTS.md's monotonic-counter rule (#480), that gap is the class this test should be closing.
  • RED/GREEN is real. Reverting only the pin (test untouched): --- FAIL … Get "…/v1/keys/…": EOF; head --- PASS (16.64s). gofmt/go vet clean, -race clean, no assertion weakened or skipped, full package count 83→84 passed with the same 4 pre-existing failures.
  • Pre-existing failures are the same 4 at base and head — and already owned by #1162.

New, and the reason this stays BLOCK: the ordering the test does not cover is the one that regresses. With the dead mint at acceptedMints[0] it becomes CurrentMintURL, gonuts' offline early-return leaves isErr true, and its deferred close shuts the DB of the wallet it just returned — so New() now succeeds with zero registered mints and an empty balance map (database not open on every later write), where base refused loudly with the exact production error. Detail, reproducer and the two suggested closes are in my comment above.

Net: the test is worth landing, the gonuts#45 diagnosis is right, and the branch needs the pin aligned plus a decision on the dead-mint-first ordering before it can merge.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

Stacked the pin fix from my review as a PR against your branch, since the branch lives in the org repo and I can't push to it (maintainer_can_modify: false):

#1182 — deps(wallet): one gonuts pin, every carrier → #1182

Base is fix/stale-mint-resilience, so merging it into your branch leaves #1142's diff exactly as authored plus the pin alignment. Dependency metadata only (10 files, +13/−17, no Go source). It takes cd src && go build ./... from rc=1 to rc=0, the two build-package lanes and main package (testenv) from rc=1 to rc=0, and turns the gonuts pin fence green.

Feel free to cherry-pick, merge, or ignore — whichever suits your workflow. It deliberately does not touch the two open review blockers (unasserted retention, and the dead-mint-first ordering), since both need your call.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

BLOCKER found on review — the branch is red on main's own guard

Verdict: request changes. This is a real, reproducible regression, and it is a one-line fix.

The failure

$ git checkout refs/remotes/upstream/pr-1142
$ cd src/tollwallet && go test -count=1 -run TestV2Keyset .
--- FAIL: TestV2Keyset_BothIdSpellingsReachSwapWithFullId (0.02s)
    --- FAIL: .../cdk_native_short_id_resolves_to_full
    --- FAIL: .../wallet-side_full_id_passes_through_unchanged
    v2_keyset_spellings_test.go:130: tollwallet.New: failed to create wallet:
      error adding new mint: mint advertised keyset id 01e73e75a0809604372487732ee917e8f881cc4c882ccd9fbac094fbfd3e3a42e6
      but its published keys derive to 0131b33abca29b2977eeb24789c0baf0250aa6dcffe226b0cf9f20c9c7d2e86be7
      — refusing (forged keyset id, tollgate #705)
--- FAIL: TestV2Keyset_V4TokenShortIdResolves (0.01s)

Pristine upstream/main (95835413), same selection, same machine:

ok  	github.com/OpenTollGate/tollgate-module-basic-go/src/tollwallet	0.077s

Cause

The stub-mint fixture hardcodes the advertised keyset id as a constant:

-const v2KeysetFull = "01e73e75a0809604372487732ee917e8f881cc4c882ccd9fbac094fbfd3e3a42e6"
+const v2KeysetFull = "01e73e75a0809604372487732ee917e8f881cc4c882ccd9fbac094fbfd3e3a42e6"   (this branch)

while leaving stubKeysJSON's keys untouched. The production guard derives the id from the published keys and refuses a mismatch — correctly. The keys in the fixture derive to 0131b33a… (which is what main carries), not to 01e73e75…. main is green only because its constant happens to match its keys.

Fix

Do not hand-pick an id. Derive it, the way PR #1162 does:

var v2KeysetKeys = map[uint64]string{
	1: "0279be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798",
	2: "02c6047f9441ed7d6d3045406e95c07cd85c778e4b8cef3ca7abac09b95c709ee5",
	4: "02e493dbf1c10d80f3581e4904930b1404cc6c13900ee0758474fa94abe8c4cd13",
}
var v2KeysetFull = deriveHonestKeysetID(v2KeysetKeys)

honest_keyset_test.go already provides the helper (#1156). #1162 exists specifically to make this fixture honest and I have reviewed it as land — it is green on this same selection (ok … 0.080s). Cheapest resolution: rebase onto #1162 once it lands, or cherry-pick its v2_keyset_spellings_test.go hunk.

On the rest of the PR

The subject — test(tollwallet): tolerate an unreachable stored mint at startup — is a legitimate and important behaviour to pin: a wallet DB holding a keyset and proofs for a mint that is down with io.EOF must not fail initialization. stale_mint_test.go's eofMint helper (hijack the connection and close it, which is what a crashed/restarting mint looks like) is the right way to model it, and asserting the healthy mint is still registered while the unreachable record is retained is the correct shape.

But the same branch also pins src/tollwallet/go.mod to gonuts v0.14.2-0.20261030003712-ae77396240a2 while packaging/build-inputs.json and the other four carriers stay on v0.14.1. That is the drift #1182 was opened to paper over, and per the doctrine the resolution is not a pseudo-version on main while a stable v0.15.0 exists — it is to bump via scripts/bump-gonuts.sh (#1175 does this correctly). If the io.EOF tolerance genuinely requires an unreleased fork commit, the fork needs a tag first.

Not verified

No restart/kill test at the transaction boundary — the cloud-lab crash lanes are where this belongs and I did not run them. No router.

Context

main is red for unrelated src/merchant reasons (#1160's fix), and .github/workflows/test.yml has not run since 2026-08-27 — nothing would have told you about the tollwallet red either. Run make go-battery locally before pushing.


Evidence: go test -count=1 -run TestV2Keyset ./src/tollwallet on this branch vs. pristine upstream/main, same machine, Go 1.26.0 module-pinned.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

Status of my CHANGES_REQUESTED — here is the exact unblock, so this does not sit.

Re-checked your HEAD just now: still a672c706, unchanged since my review. The fence is still red:

$ python3 tests/contract/check-gonuts-pin.py
  carriers discovered by glob: 5 module(s)

❌ GONUTS PIN DRIFT — 1 finding(s):
  🔴 src/tollwallet/go.mod: pins github.com/OpenTollGate/gonuts-tollgate
     v0.14.2-0.20261030003712-ae77396240a2, manifest says v0.14.1
     — run scripts/bump-gonuts.sh v0.14.1

So the review state is intentional, not stale. But I do not want it to be a dead end, so:

Two ways to clear it, both already built

Option A — merge my stacked fix, #1182 (recommended, zero work for you)

#1182 — base is your branch fix/stale-mint-resilience, currently MERGEABLE, head d1bff42a, 11 files +14/−17. It moves the manifest and all 5 carriers to v0.14.2-0.20261030003712-ae77396240a2 and drops the stale go.sum pairs. Merging it into your branch turns the fence green and I will convert my review to approval.

Verified on that branch: check-gonuts-pin.py rc=0 (5 carriers agree), check-deps-sync.py clean, release build rc=0 (18,733,517 B), CLI build rc=0, gonuts-pin_test.sh 7/0.

Option B — do it yourself in two commands

scripts/bump-gonuts.sh v0.14.2-0.20261030003712-ae77396240a2
python3 tests/contract/check-gonuts-pin.py    # expect rc=0, 5 carriers

Note that the manifest, not src/tollwallet/go.mod, is the source of truth — editing the go.mod by hand is what produced the drift (#791). That is why the fix touches packaging/build-inputs.json plus every module that imports gonuts, not just the one your test needed.

Why I am holding the review rather than approving on the test alone

Your test is genuinely load-bearing — I proved it. Reverting only the pin (leaving your test intact) turns

PASS 16.87s   →   FAIL with the production `EOF` error

So the work is real and the merged thing should keep it. But main currently builds green because the carriers agree, and merging a branch where one carrier has drifted re-breaks the release build path — the exact failure the fence exists to catch. Approving that would be signing off on a red gate.

The diff is 2 lines of dependency metadata away from mergeable. Option A costs you one click.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants