Repository navigation
crash-window lane: no value recovery after restart (SIGKILL between mint-signed and wallet-save) — #502/#700 acceptance FAILS on main #719
Description
Summary
The crash-window acceptance lane for the payment-convergence work (#502/#700, the business-transaction record) fails on current main: after the tollgate is SIGKILLed between "mint signed the swap" and "wallet has persisted", the boot-time intent resume does not recover the value within the lane's window. This is the exact invariant the lane exists to hold.
Severity: S1/S2 candidate (funds-adjacent). The log shows the #502 machinery engaging (intent restored, NUT-07 reconciliation started) — so the failure is either reconciliation-not-completing in the lane's wait window (timing) or reconciliation completing without recovering the value (lost value). Distinguishing those two is the first triage step.
Verified on
- Commit:
68ad5144ab09cb4723d8f3f655cbba1ac7ea9de5(main, 2026-10-07) + release-prep metadata commit (code-identical) - Host: docker 29.1.3, linux/amd64; lane run via
tests/cloud-lab/run-crash-injection.sh - Campaign evidence: release-labgrid-testing, 2026-10-07
Evidence (verbatim, from the lane transcript)
== paying in the background (the swap response will be swallowed)
== waiting for the swap response marker (mint has signed)
marker: /v1/swap
== SIGKILL the tollgate (mid-wait: mint signed, wallet has NOT processed)
killed at 18:53:33
== restarting the tollgate; boot resume must replay the intent
FAIL: no recovery after restart — the crash window destroyed value:
2026/10/07 16:53:33 PurchaseSession: Receive completed, amount=0, err=mint did not answer; the outcome is unknown: could not swap proofs: mint http://killer:8085/v1/swap did not answer; the request may have been processed — reconcile before retrying: Post "http://killer:8085/v1/swap": EOF
2026/10/07 16:53:34 Restored 1 pending receive intent(s) — reconciling against their mints (NUT-07)
Note the two log lines: the pending intent IS restored and NUT-07 reconciliation IS started one second after restart — then the lane's recovery assertion fails.
Reproduce
Prerequisites: a linux docker host (docker 29.x needs the Dockerfile.client ARG fix from the companion issue "cloud-lab client image unbuildable on docker/buildkit 29" — or build the client image manually first). If your docker build bridge has no working DNS for proxy.golang.org (symptom: dial tcp: lookup proxy.golang.org ... network is unreachable inside RUN go build), pre-build images with build.network: host.
git clone https://github.com/OpenTollGate/tollgate-module-basic-go && cd tollgate-module-basic-go
git checkout 68ad5144ab09cb4723d8f3f655cbba1ac7ea9de5
cd tests/cloud-lab
export TG_GO_VERSION="$(jq -r '.go.version' ../../packaging/build-inputs.json)" # 1.26.8
./run-crash-injection.sh ; echo EXIT=$?Expected (per #497/#502/#676 acceptance): exit 0 — intent resume recovers the value AND a fresh payment afterwards still succeeds.
Actual: FAIL: no recovery after restart — the crash window destroyed value and exit 1.
Troubleshooting guide (for the fixing agent)
- Read the lane first:
tests/cloud-lab/run-crash-injection.sh— the killer proxy (killerproxy.pyfrom PRTA, mounted read-only) swallows every/v1/swapresponse once the mint has signed; the tollgate is SIGKILLed when the/marker/kill.markerfile appears; the assertion block is after the restart. Find the exact wait window it gives the reconcile loop. - Distinguish timing vs lost value: instrument or watch the upstream container logs after restart — does the NUT-07 reconcile (
Restored 1 pending receive intent(s)) ever complete? If it completes but the proofs are gone, that's lost value; if it needs longer than the lane waits, the lane's window is too tight for the reconcile path (still a release problem, different fix). - Code pointers: the receive-intent journal and resume live in the Priority: P1 — the TollGate-level business-transaction record; generalizes the still-open #403 into a mechanism. #502/fix(merchant): the business-transaction record — an ambiguous or interrupted payment converges to service or spendable value (#502) #700 business-transaction work —
src/merchant/(purchase-session guard + journal; seepurchasesession_duplicate_guard_test.gofor the intended semantics) andsrc/tollwallet/(swap-intent machinery; gonuts-tollgate v0.13.0 persrc/tollwallet/go.mod). The "outcome is unknown … reconcile before retrying" string in the log above is the unknown-outcome path — that is the state the resume must resolve. - Reference material:
docs/architecture/bearer-instrument-port.mdandresearch/bearer-port-demo/describe the intended derivation-replay recovery (NUT-07 answer is about consumption, never existence;SPENT/UNSPNTonly from what the authority said). - Unit-level repro that passes today:
cd src/merchant && go test -tags testenv -run 'TestPurchaseSessionGuardHoldsThroughTheOutcomeUnknownWindow' -v(needs anndsctlshim on PATH on a bare host — see companion issue). The docker lane is where it fails; the unit test seeds a scripted wallet, the lane uses a real mint behind the killer.
Fix hints
- If timing: widen/loop the lane's post-restart wait on a functional signal (e.g. poll the owed-grant/session event rather than a fixed sleep), and check the reconcile backoff constants the boot resume uses.
- If lost value: the boot-time replay must re-derive the blinded proofs from the journal and re-drive the swap (or recover via NUT-09 token state) — start where
Restored N pending receive intent(s)is logged and follow what that reconcile actually does for a swap whose response was swallowed. - Regression guard: this lane IS the regression guard — once fixed, keep it in the release gate (
make release-checkwithTOLLGATE_RELEASE_CHECK_CONFORMANCE=1on the machine that owns the lane).
References
- Priority: P0 — research-first: the swap crash window can silently destroy received value; the correct fix shape (pre-persisted swap intents) must be designed, not improvised. #497 (crash-window acceptance), Priority: P1 — the TollGate-level business-transaction record; generalizes the still-open #403 into a mechanism. #502/fix(merchant): the business-transaction record — an ambiguous or interrupted payment converges to service or spendable value (#502) #700 (business-transaction record), test(cloud-lab): #497 crash-window harness — SIGKILL between mint-signed and wallet-save — rebase of #569 #676 (harness), docs(wallet): derive NUT-07/NUT-09 for any bearer instrument (research + runnable demo) #631 (bearer-instrument port research)
Activity
Amperstrand commentedon Oct 8, 2026
Fix up in PR #771 (branch
Amperstrand:fix/719-boot-swap-resume).Triage answer to the issue's own first question — timing vs lost value: neither; the wallet half never ran. The merchant half of #502 (receive intents + owed grants, merged in #700) converged the customer, but the wallet-side resume was never wired and the fork's swap-intent machinery (gonuts-tollgate #31) is merged-but-unreleased, so the v0.13.0 pin never had it. The
Restored 1 pending receive intent(s)line you see in the failure log is the merchant half working as designed; the missingpending swap … recoveredline is the unwired wallet half — that is the lost-value direction, fixed by replaying the journaled intent bytes at boot.Maintainer action requested in the PR: tag the fork release (v0.14.0 = fork main d7591d2) and repin — the PR pins the integration ref per the 7cd1882 precedent so it is mergeable either way.
Lane note: the docker lane cannot run on this host (no docker access); the lane owner's run is the remaining acceptance. Local evidence: full go-battery green (16 modules; the three documented merchant flakes #731/#732/#733 skipped — #731 reproduced failing 4/5 on pristine main too, so pre-existing), unit tests at both seams. Fork-level follow-up for the mint/melt windows is gonuts-tollgate #33, unchanged.
docs(changelog): fragment for the OpenTollGate#719 boot swap resume
Amperstrand commentedon Oct 9, 2026
[release support 2026-10-09] Fork half fully tagged. gonuts-tollgate
v0.14.0(identical tree tov0.13.2, tagged tonight) carries the journaled-intent + boot-resume mechanism lane-A audited (re-POST of journaled outputs — no derivation re-exposure; counter range + intent reserved in one bbolt transaction). Remaining on #771 before this issue closes with it: thev0.14.0repin, the changelog-claim scoping to the NUT-19 replay TTL, and one green crash-injection run on the repinned branch.Amperstrand commentedon Oct 12, 2026
Adjudication: #793 carries the fix — land it stacked on #781, close #873 as superseded. The crash-injection lane is green on #793's head (
e1f83305); evidence below.Both carriers drive the fork's
ResumePendingSwaps(fork #31 machinery), so the point of no return, the durable-before-POST intent, the retain-on-failure policy, and the secret-keyed idempotent saves are identical — that is not the axis of choice. The axes that separate them:newFullMerchant, before the API serves: a mint down at boot costs ~30 s per journaled intent on the path to serving. feat(tollwallet): recover crashed swaps on wallet load — wire ResumePendingSwaps (#497 resume half, #703 item 1) #793 runs the pass as a background goroutine inTollWallet.New; boot never pays the replay timeout — pinned byTestWalletLoadKeepsIntentWhenMintStaysDown(≈16 s dead-mint load; the assertion fails above 30 s, a synchronous replay adds ≥30 s per intent). On a router, serve-first is the right default.TollWallet.Newis the single funnel — full boot, degraded boot (DefaultWalletFactory,merchant_degraded.go:249), and the degraded→full upgrade all heal. fix(merchant): boot resume replays journaled swap intents — close the #719 crash window (#497) #873'snewFullMerchantcall site skips the degraded boot entirely (intents wait for the upgrade).BootSwapReplayDone(), with a three-case test matrix. fix(merchant): boot resume replays journaled swap intents — close the #719 crash window (#497) #873 avoids the race only by being synchronous — i.e. by paying cost 1.v0.13.1-0.2026…d7591d2, which must not outlive its tag onmainnow thatv0.14.0exists. feat(tollwallet): recover crashed swaps on wallet load — wire ResumePendingSwaps (#497 resume half, #703 item 1) #793 carries no pin hunks of its own.port.godoc still say a mint "signs deterministically … re-issues identical signatures" — the refuted reading. feat(tollwallet): recover crashed swaps on wallet load — wire ResumePendingSwaps (#497 resume half, #703 item 1) #793's fragment, call-site comment, and PR body are NUT-19-scoped end to end.-raceend-to-end recovery tests; fix(merchant): boot resume replays journaled swap intents — close the #719 crash window (#497) #873's tests are seam/stub-only, and its body states the docker lane could not run on its host.Worth salvaging from #873 as comments on #793 (not blockers): the explicit adapter-contract prose (cdk honest no-op, sidecar typed
ErrSidecarResumeUnsupported). Under #793's wiring aWalletPort.ResumePendingSwapssurface would be dead code, but that honesty argument belongs at theBootSwapReplayDonetype assertion.Repin: #781 is still required and lands first. #793's stack embeds a bare
v0.14.0go.mod repin (including the fifth carrier) but none of #781's fence: the.gonuts.versionmanifest truth,check-gonuts-pin.py,bump-gonuts.sh, therelease-checkgate. Merge #781, rebase #793 onto it (drop the embedded repin commits, keep the wiring), retarget #793 tomain.Lane evidence —
tests/cloud-lab/run-crash-injection.shon #793 heade1f83305,CONCURRENT=1, exit 0:/v1/swap→ SIGKILL → restart →RECOVERED: wallet: pending swap 491ed781… recovered 50 sats from http://killer:8085, one second after the kill — inside the mint's NUT-19 cache window;200(the review's item-4 scenario, live);200.Changelog: the surviving fragment must scope the claim to the NUT-19 replay window (as #793's already does) — never "deterministic signing".
Known residual, deliberately out of scope: beyond the mint's NUT-19 TTL (or after a mint restart), replay is refused
11001and the intent stays journaled pending the NUT-09/v1/restorefallback — #866 / fork #40 / draft lane #886. A green lane closes the lane's window, not the class.Full research brief:
issue-briefs/719-*on theresearch/issue-briefsbranch.