Skip to content

fix(merchant): boot resume replays journaled swap intents — close the #719 crash window (#497) - #771

Closed
Amperstrand wants to merge 2 commits into
OpenTollGate:mainfrom
Amperstrand:fix/719-boot-swap-resume
Closed

Amperstrand wants to merge 2 commits into
OpenTollGate:mainfrom
Amperstrand:fix/719-boot-swap-resume

Conversation

@Amperstrand

Copy link
Copy Markdown
Collaborator

Closes #719. Fund-safety lane: the P0 swap crash window (#497).

Root cause (why the lane is red on main)

The crash-window acceptance (tests/cloud-lab/run-crash-injection.sh) fails on main because only the customer half of #502's business-transaction record landed:

So after SIGKILL between mint-signed and wallet-save: the customer gets owed service, but the operator's value stays destroyed — no pending swap … recovered line ever appears, which is exactly what the lane greps for.

What this PR does (three coordinated moves)

  1. Fork pin: gonuts-tollgate v0.13.0 → d7591d2 (fork main = Duplicate tollgate SSIDs #31 pre-persisted swap intents + Create identities.json #30 KeysetFeesForMint) across all four carrying go.mod files (src, src/cli, src/merchant, src/tollwallet). Integration-ref pin per the 7cd1882 precedent; maintainer action: tag the fork release (v0.14.0) and repin — one line ×4, no code change.
  2. WalletPort.ResumePendingSwaps:
  3. newFullMerchant calls the resume once at boot, before serving payments and beside the merchant-half restores. Recovered value is operator balance. Every failure shape is non-fatal: intents stay journaled, the next boot retries, a down mint never wedges boot. A degraded boot (no wallet) skips it and the intents wait for the upgrade — nothing is deleted, nothing is regenerated.

Verification

Out of scope (tracked)

  • gonuts-tollgate Feature/test joel frontend #33 (mint + melt intents) — the mint/melt crash windows remain open at the fork level; this PR recovers the customer-facing swap path the lane pins.
  • The long-running-daemon sweep (resume only runs at boot today; an operator/CLI sweep for a wallet that stayed up through a failed swap is future work).

Amperstrand added 2 commits October 8, 2026 21:53
…ollGate#719 crash window closes at boot (OpenTollGate#497)

OpenTollGate#719: the crash-window lane fails on main because only the CUSTOMER half
of OpenTollGate#502's business-transaction record landed — the merchant restores
receive intents and converges owed service, but nothing ever drives the
WALLET half, so value the mint already issued is never recovered: the
swap-intent machinery (gonuts-tollgate #31, merged but unreleased) is not
in the v0.13.0 pin and the daemon calls no resume.

Three coordinated moves:

- bump gonuts-tollgate v0.13.0 -> d7591d2 (fork main: #31 pre-persisted
  swap intents + #30 KeysetFeesForMint) in all four carrying go.mod
  files. The maintainer tags the fork release (v0.14.0) and repins; the
  integration-ref pin follows the 7cd1882 precedent.
- WalletPort grows ResumePendingSwaps: the gonuts adapter delegates to
  the fork's replay (exact request bytes re-POSTed; a deterministically
  signing mint re-issues identical signatures, verified against
  cdk-mintd 0.17.6 in OpenTollGate#497); cdk is an honest no-op (its saga resumes at
  load); the sidecar reports its protocol gap as a typed error.
- newFullMerchant calls the resume once at boot, before serving
  payments: recovered value is operator balance, every failure shape is
  non-fatal (intents stay journaled, next boot retries, a down mint
  never wedges boot).

Verification: unit tests at both seams (delegation + non-fatal boot
shapes); full go-battery green across all 16 modules (three documented
pre-existing merchant flakes skipped — OpenTollGate#731/OpenTollGate#732/OpenTollGate#733; OpenTollGate#731 reproduced
failing on pristine main at the same rate, 4/5, so not introduced here;
the OpenTollGate#726 ndsctl-on-PATH requirement worked around with the cloud-lab
stub). The docker crash-window lane itself cannot run on this host (no
docker access) — it must be run by the lane owner; the success log line
it greps for is exactly the one the fork's resume emits.
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

CHANGES-REQUESTED — the mechanism is fund-safe (I verified in the pinned fork source that the replay can never re-expose deterministic derivation outputs), but the PR's central recovery claim is contradicted by the author's own deeper research in #497: beyond the mint's NUT-19 replay-cache TTL the re-POST is refused and the value is not recovered, and the changelog fragment ships the unqualified claim into v0.6.0 release notes.

I audited the fork at the pinned commit (d7591d2f3cb361f2661eb79ca54b7abf9f6b2953) end-to-end, because the fund-safety substance lives there, not in this diff. The brick-class questions all come back clean: resumeOneSwap re-POSTs the journaled RequestBytes verbatim (wallet/pending_swaps.go#L57) and reconstructs proofs from the journaled Outputs/Secrets/Rs — there is no derivation code in the resume path at all, so #257/#266/#480 cannot trigger. The counter range and the intent are reserved in one bbolt transaction before the POST (ReserveKeysetRangeWithIntent, wallet/storage/bolt.go#L672-L717) — the #266 discipline made atomic. The live-path 10002 handler deletes the superseded intent and regenerates from a freshly incremented counter (wallet/wallet.go#L973-L996), exactly per the house rule. Resume is idempotent (SaveProofs secret-keyed; a crash between save and intent-delete replays benignly), per-intent failures retain the intent, and boot never wedges — I also verified the degraded-boot path returns before the resume call and gets it on the mint-reachable upgrade, and ran the new seam tests plus the merchant boot test in a worktree: all pass.

  1. (criteria 1/4 — the recovery claim is wider than the mechanism) The body, port.go#L260-L263 and the fork comment at pending_swaps.go#L24-L28 all state that a deterministically-signing mint returns identical signatures on re-POST, "verified against cdk-mintd 0.17.6". Your own later research in Priority: P0 — research-first: the swap crash window can silently destroy received value; the correct fix shape (pre-persisted swap intents) must be designed, not improvised. #497 (comment 2, 2026-09-22 20:34) corrected the earlier finding: the idempotent 200 comes from NUT-19 cached responses, cdk-mintd's default backend is in-memory with a 60-second TTL, and on cache expiry the mint returns 11001 "Proofs already spent" — replay is then refused forever. The Priority: P0 — research-first: the swap crash window can silently destroy received value; the correct fix shape (pre-persisted swap intents) must be designed, not improvised. #497 harness runs that verified recovery were quick docker restarts, inside that window; a router power-loss or any restart minutes later is not recovered by replay, and a mint without NUT-19 never is. CDK's saga — the reference model this repo's AGENTS.md names — falls back to NUT-09 /restore for exactly this case (#497 comment 5); resumeOneSwap has no such fallback (the fork's wallet/restore.go is mnemonic-level only). The mechanism is still a strict improvement over rc1 — today the value is destroyed and unjournaled, while the intent preserves everything a future restore-fallback needs — but the changelog headline "A crash between the mint signing a swap and the wallet saving the proofs no longer destroys the received value" (changelog.d/771-boot-swap-resume.fixed.md#L1-L3) is false beyond the TTL. Requested: scope the fragment (e.g. "…recovers it when the mint honours NUT-19 replay, and journals the full recovery record for mints that don't"), and soften the same claim in port.go's doc comment so the API contract future callers code against is honest. Whether to add the NUT-09 restore fallback in the fork before v0.6.0 (no tollgate-side diff beyond the repin — the intent already stores Outputs/Secrets/Rs) is an owner decision; the journaled-for-future-recovery framing makes it a clean follow-up either way.

  2. (criterion 7 / release process — the pin must become a tag before stable) The pseudo-version pin v0.13.1-0.20261006144824-d7591d2f3cb3 across the four go.mod files follows the 7cd1882 integration-ref precedent and the body flags the maintainer action, so it is fine for merge sequencing — but v0.14.0 does not exist yet on the fork (latest tag is v0.13.1; verified against the fork's refs), and AGENTS.md's letter is tag-then-bump. This PR must not ride into the v0.6.0 stable cut on a commit pseudo-version: repin to the tag once the sibling lane tags it. Note for whoever repins: fork main has moved past d7591d2 with two more wallet fixes (f64d7f3 refuse-empty-outputs swap, 1eae260 NUT-13 residue guard — the tollgate Priority: P2 — NUT-13 residue-collision guard is dead code in gonuts v0.13.0 (Conduition disclosure): CheckCollidingKeysets never invoked; wallet-level fix belongs in gonuts-tollgate, tracked here #705 fix), so a v0.14.0 tagged from current main carries more than the swap-intent feature; both look like strict improvements but the repin diff should say so.

  3. (criterion 8 — the acceptance lane has not run on this branch) The body states plainly that the docker crash-injection lane could not run on the author's host. The lane (tests/cloud-lab/run-crash-injection.sh) is the acceptance crash-window lane: no value recovery after restart (SIGKILL between mint-signed and wallet-save) — #502/#700 acceptance FAILS on main #719 exists for, and its grep line matches the resume's log line at pending_swaps.go#L98 — but it has not produced evidence against this branch. Make its green run a merge condition (the lane owner run in the PR body covers this).

  4. (minor, criterion 4) merchant.go#L527-L529: balance := tw.GetBalance() snapshots before the resume, so the boot log's "Wallet Balance" line excludes recovered value an operator reconciling boot logs would expect to see. One-line reorder (resume before the snapshot) if you touch the file anyway; not blocking.

  5. (fragility note, not a change request) A stale intent against a non-NUT-19 mint retries one POST per boot forever (30s timeout each, pending_swaps.go#L18) — bounded and boot-only, but a resume-pass time budget plus eventual NUT-09 classification (recovered / restoreable / refused) would keep a handful of stale intents from adding minutes to every future boot.

No overlap concerns: this is the only boot-resume PR in the queue, the customer half it complements (#700) is already on main, and the changelog fragment correctly follows the changelog.d convention (unlike #778's direct edit — the two PRs should converge on the fragment style).

Freeze compatibility: exactly stop-ship-sized — this is the P0 of the batch and the design is right — but the release-notes claim must be honest before this lands, the fork tag + repin must precede the stable cut, and the crash-injection lane needs one green run on the repinned branch.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Rel-verify: cloud-lab crash-injection lane on #771 head b3ce6d5 — PASS per scenario (lane rc1-771, ai-legion, docker 29.x; transcript: /root/src/rel-771-transcript.md on the lane host).

The acceptance run the author could not run (no docker on their host) — tests/cloud-lab/run-crash-injection.sh against this head, exit 0, plus scenario coverage beyond the stock lane:

  • (a) crash pre-POST (intent journaled, never sent) — PASS. Request-swallow killer variant (lane-local, see disclosure): the tollgate's /v1/swap request never reached the mint, SIGKILL on marker, restart → boot resume replayed the journaled request byte-identically (sha256 900e1598…, len 3654 — same hash pre-kill and post-restart via killer instrumentation), mint processed it as first exposure, wallet: pending swap 6720e108… recovered 50 sats, 0 unrecoverable this pass, owed grant applied (allotment 3,000,000 ms = 50 sat × 60 s exactly), post-recovery purchase 200.
  • (b) crash post-POST pre-save (the crash-window lane: no value recovery after restart (SIGKILL between mint-signed and wallet-save) — #502/#700 acceptance FAILS on main #719 window) — PASS, twice. Stock lane verbatim (exit 0, pending swap 6957346c… recovered 50 sats from http://killer:8085 ~0.5 s after restart) and an instrumented rerun proving the replay re-POSTs identical bytes (0a132ae9…, len 3654, pre-kill swallow = post-restart pass-through) with the deterministic mint answering 200 — idempotent per NUT-19, no 10002/duplicate-output errors anywhere (mint logs clean in all runs). Operator wallet held 80 sats (50 recovered + 30 health) after each run; customer side converged via the owed-grant machinery (state: granted, attempts 0).
  • (c) end-to-end purchase after resume — PASS in every crash run: fresh payment → HTTP 200 with a kind 1022 session event (allotment/metric/start-time tags), Amount after swap: 30, fake-ndsctl AUTH <mac> recorded for both the crashed customer (post-restart owed grant) and the post-recovery buyer.

Standard cloud-lab suite on this head: 17 passed / 21 skipped / 6 failed. All 6 failures are pre-existing and not from this PR: 5 fee-allotment assertions break on cumulative in-memory sessions per static client MAC (reproduced identically against a main-built upstream image, and the first case passes on a freshly restarted upstream — an ordering/memory artifact of the combined pytest invocation, amplified by the clientd renewal daemon), plus 1 clientd sats/step string drift in files the PR does not touch. Fund-safety-adjacent suites that matter here passed: smoke (kind-1022 session + gate opened), mint-failure recovery, and test_replayed_token_grants_no_second_session.

Scratch-tree disclosures (lane-local, none part of the PR): (1) the #728 ARG GO_VERSION hoist in Dockerfile.client — #728 unmerged, image unbuildable without it on docker 29.x; (2) build.network: host added to the per-run override builds — this host's build bridge has no DNS for proxy.golang.org (same workaround the prior #719 triage session used here); (3) two lane-only killer variants (hash-logging; request-swallow-from-tollgate-IP) and a variant driver selecting them, kill/restart/assertion sequence byte-identical to stock; (4) the standard-suite upstream image was tagged from this tree's earlier successful build after a host resolver flake (fd21:240c:c58b::1 unreachable) aborted a rebuild — same commit, same context. Gonuts pin verified resolving on-host: v0.13.1-0.20261006144824-d7591d2f3cb3 in all four carrying go.mod files.

Not covered by this run: mint/melt crash windows (gonuts-tollgate #33, explicitly out of scope per the PR), the long-running-daemon sweep, QEMU/router-surface behavior, and the gated cloud-lab lanes that skipped (rotation phases, mint matrix, external mints, two-router).

Amperstrand pushed a commit that referenced this pull request Oct 9, 2026
…e changelog rides the fragment convention (#778 review)

Review mechanicals:

- build-purity's Tests 1-2 now audit rateLimitTestBypass = true writers
  with the same guard discipline as TOLLGATE_TEST_CONFIG_DIR: a non-
  _test.go, non-build-tagged writer fails purity. The PR's "enforced by
  the build-purity contract" claim was ahead of the script; it is true
  now. The audit is verified both ways: a spoofed unguarded writer
  fails 2/2 checks, the clean tree passes 3/3.
- The direct CHANGELOG.md edit is reverted in favour of a changelog.d
  fragment (the #769/#771/#777 convention), carrying the corrected
  claim — rig-verified FIXED on #748, purity-audited writer.

No runtime behavior touched: the shipped 10 RPM default, its semantics
and the three rate-limit proof tests are byte-identical (9/9 passes).
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Review item applied — the recovery claim is scoped — head c0f638b.

  • Changelog fragment: the headline now reads "no longer destroys the received value — when the mint honours NUT-19 replay within its cache window", and the body states the boundary and its consequence: beyond the window (cdk-mintd default: in-memory, ~60 s TTL) a mint that already processed the swap refuses with 11001 already-spent, and the intent stays journaled — visible, not lost — pending the NUT-09 /restore fallback (a fork-level follow-up, tracked separately).
  • WalletPort.ResumePendingSwaps doc comment: the same scoping, in the same breath as the idempotence/never-lost guarantees, so the interface's contract carries the boundary wherever it's read.

No logic, no pin hunks — the repin rides #780's merge order as established. Verified after the edit: changelog-assemble.py check (9 fragments, no findings), the fragments contract suite 18/0, full src/tollwallet suite ok 51.4s (-race), and the merchant resume rows (TestKilledAttempt|TestBoot|TestReceiveIntent) ok.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

[release support 2026-10-09] Fork tag prerequisite SATISFIED — gonuts-tollgate v0.14.0 is live; repin when ready.

  • v0.14.0 (tag b00bc231 → commit 1eae260) — identical tree to v0.13.2, which the fork lane tagged tonight carrying everything this PR needs: journaled swap intents + boot resume (d7591d2, the exact commit this PR pins), refuse-empty-outputs (f64d7f3), KeysetFeesForMint (c2054b5), and the NUT-13 guard (1eae260). v0.14.0 exists as the minor-version alias the train repins to; either tag resolves identically — use v0.14.0.
  • Repin = the four go.mod entries (src, src/cli, src/merchant, src/tollwallet) → github.com/OpenTollGate/gonuts-tollgate v0.14.0 + go mod tidy. Module verified resolving at the tag (GOPROXY=direct and default proxy).
  • The tag was verified before pushing: go build/go vet clean, go test ./... 12/12 packages green (one load-flake of TestVerifyKeysetIdRejectsForgedV1 in a single parallel full-tree run; 3/3 standalone and on repeat full runs — flagged to the fork lane for follow-up).
  • Remaining merge conditions from lane-A are unchanged: scope the changelog claim to the NUT-19-replay reality (+ port.go doc comment), and one green crash-injection lane run on the repinned branch. Related: feat(tollwallet): recover crashed swaps on wallet load — wire ResumePendingSwaps (#497 resume half, #703 item 1) #793 (new) wires ResumePendingSwaps on wallet load — same family; the maintainer should pick the single carrier (comment posted there).

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Carrier adjudication (#665 addendum, #771 vs #793): recommendation — supersede this PR in favor of #793. Same standard as the #718-vs-#774 call: one mechanism wins on the merits, the loser closes with an explicit supersession rather than a silent drop. The comparison, all points verified on both current heads:

Dimension #771 (head b3ce6d54) #793 (head 13f97e5e)
Wiring site synchronous resumePendingSwapsAtBoot in newFullMerchant — full boot only; costs 30 s/intent on the path to serving when an intent's mint is slow at boot background pass in TollWallet.New — the single funnel (full boot, degraded boot, degraded→full upgrade all construct through it); never blocks boot, test-pinned
Pin state four go.mod pseudo-version pin hunks (v0.13.1-0.20261006144824-d7591d2f3cb3) — obsolete now that fork v0.14.0 exists and #780 delivers the pin zero pin hunks (stacked on #780)
Test coverage seam tests + merchant exactly-once test (stubbed resume); real-recovery evidence is the external crash-injection lane in-repo -race end-to-end tests exercising the actual recovery (I re-ran both: PASS)
Changelog claim unqualified on the current branch — the scoped reword (commit c0f638b) is not on the branch; the fragment at head still says "no longer destroys the received value … a deterministically signing mint re-issues identical signatures" unscoped, plus a now-obsolete "pinned … pending its release tag" note unqualified in fragment, comment, and body — but the fix is a lift of the already-written scoped text from your own comment above

What transfers to #793: (1) the scoped NUT-19 fragment wording from the reword comment — please re-push that text somewhere durable (it currently exists only as a PR comment); (2) the crash-injection lane green — your PASS run validated the fork mechanism both PRs share, and #793's review asks for one re-run against its head since its wiring differs (recovery concurrent with serving). What is deliberately dropped: the WalletPort.ResumePendingSwaps surface and its cdk/sidecar adapters — under #793's site only the gonuts backend needs the call (cdk self-resumes on open; the sidecar daemon owns its intents), so the port method would be dead surface. If an operator-triggered resume is ever wanted (CLI), the surface can return then.

Suggested close: once #793 lands the scoped claim and the lane run, close this PR as superseded by #793 with a one-line note pointing at this comment — same shape as #718→#774. The engineering here wasn't wasted: this PR's review produced the TTL scoping the feature's changelog now needs, and its lane run is the standing evidence the mechanism works.

@c03rad0r

Copy link
Copy Markdown
Contributor

🏷️ Next action — changes-requested

Changes requested — author addresses the blocking item(s) in the review above, then request a re-review.

@c03rad0r c03rad0r removed the wave:4 Covered by the wave-4 review pass label Oct 10, 2026
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #873 — migrated to an upstream head as part of the fork deprecation (identical content, same branch name). Continuing review in #873.

Amperstrand added a commit that referenced this pull request Oct 12, 2026
…pin to the NUT-09 restore fallback — #866

Draft pin: gonuts-tollgate PR #40 branch commit d42a98cf8825
(pseudo-version v0.14.1-0.20261011150228-d42a98cf8825) carried by all
five go.mod carriers AND packaging/build-inputs.json — hand-edited per
the #771 branch-pseudo-pin pattern; bump-gonuts.sh (which refuses
pseudo-versions by design) is the flip path at tag time.
check-gonuts-pin.py: one pin, everywhere, across 5 carriers.

Scenario (e') asserts the restore fallback end-to-end (red on current
main by design — the fallback arrives with this repin);
Dockerfile.mint018 bridges 0.18's DB-authoritative config from the
0.17.6 env-var flow; the research writeup lives at docs/research/ per
contributing guidance.

Single clean commit rebased onto main f89c879 (v0.6.0-rc2 train, gonuts
base v0.13.1) — supersedes the earlier branch state whose review fixes
had failed to push (refspec mistake; disclosed in the PR thread).

Closes #866
Amperstrand added a commit that referenced this pull request Oct 24, 2026
…et on Unknown Keyset with a reachable mint

Wallet init on a DB caching keysets the mint has rotated out failed
with 'error getting keyset from mint: Unknown Keyset', the merchant
degraded, and the degraded path's first-reachable trigger never fired
(mint reachable all along) — the router bricked until wallet.db was
moved aside, discarding proofs (#818).

State machine at the New() seam: init fail → classify with the fork's
IsUnknownKeysetError → if unknown-keyset class AND the current mint
answers the bounded NUT-02 /v1/keysets probe (mintServesKeysets),
re-fetch and retry the load ONCE → proceed or degrade. Everything else
(storage errors, unreachable mint, real refusals) degrades exactly as
before — the storage-degraded family (#583/#1035) is a separate slice.

Pin: gonuts-tollgate v0.14.1-0.20261024073441-2c1154c68048
(pseudo-version integration pin, #771 pattern) — fork PR #44 carries the
loadWalletMints tolerance; stacks on #1063 (v0.14.0 + honest fixtures);
owner tags the fork, then scripts/bump-gonuts.sh finalizes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:changes-requested Reviewer flagged blocking items; author action needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

crash-window lane: no value recovery after restart (SIGKILL between mint-signed and wallet-save) — #502/#700 acceptance FAILS on main

2 participants