Skip to content

wallet: swap-timeout retry re-exposes derivation outputs (measured by the #535 conformance lane on current main) — the #257/#266/#480 brick class #640

Closed

Description

@Amperstrand

Summary

The conformance fast-subset lane (#535), re-run on current main (post-#558/#595, branch head rebase 2026-10-05), measured the swap-timeout-retry scenario re-exposing deterministic derivation outputs after an ambiguous outcome — the exact wallet-brick class of #257/#266/#480, forbidden by the fund-safety rules in AGENTS.md ("Never reuse deterministic Cashu derivation outputs... Re-derivation triggers mint error 10002... and has repeatedly bricked wallets").

What the lane measured

The PRTA fault proxy (which observes every blinded message the daemon sends to the mint) recorded the daemon's swap-route derivation outputs sighted twice after the proxy processed the first swap and dropped the response (the ambiguous-outcome window): the same output set was re-sent and the retry succeeded — the payment returned 1022 anyway. A mint that enforces duplicate-output detection (error 10002) would have bricked that derivation range instead; the lab mint tolerates it, which is why payments keep working while the invariant is violated.

Evidence (from the lane run, .conformance/evidence/swap_timeout_retry.log + the proxy observation log): blinded-message digests with total: 2, phase_delta: 2 on swap routes; verdict row swap-timeout-retry / no-output-reuse=fail, restart-converges=pass.

The open question the lane cannot answer

Whether the re-send is gonuts-tollgate retry logic re-deriving from the same counter range, or Go's net/http transport replaying the request body after the connection was cut — the fix differs (fresh-counter regeneration on ambiguous outcome vs. disabling request replay / idempotency keys). Per the fund-safety rules, on timeout the wallet must query proof/quote state before regenerating (NUT-07 checkstate), and any retry must derive from a freshly incremented counter. The reference model is CDK's wallet saga (crates/cdk/src/wallet/swap/saga/) — the ambiguous-outcome reconciliation is exactly what its crash-safe state machine exists for.

Reproduce

cd tests/cloud-lab
./conformance/run-conformance.sh     # needs docker + a PRTA sibling checkout
# row: swap-timeout-retry  no-output-reuse=fail  (evidence in .conformance/evidence/)

Filed per the #535 review's follow-up list ("file the swap-timeout re-exposure follow-up against current main, not the fork point") — the original observation predated #558/#595 and is now re-confirmed against them: neither changed the re-exposure behavior.

Activity

  1. Amperstrand commented on Oct 5, 2026

    @Amperstrand
    CollaboratorAuthor

    Root cause found (code-level, verified against gonuts-tollgate v0.12.1 as bumped into this module).

    The re-send is the wallet client's own transport-error retry, not net/http replay and not swapWithRetry

    Three candidates were checked:

    1. Wallet.swapWithRetry (wallet/wallet.go:885): retried — it only fires on the mint's BlindedMessageAlreadySignedErrCode (10002), re-derives fresh blinded messages, and increments the keyset counter before sending. That path is disciplined (fix: bump gonuts-tollgate v0.7.1 → v0.7.4 for swap-counter race #266's fix); it is not the re-exposure.
    2. Go net/http transport-level replay (GetBody + reused connection): ruled out empirically — a minimal repro (POST with a *bytes.Reader body over a pooled connection that the server kills while idle, same client shape as the fork's) shows the transport opens a fresh connection and sends the request once; a non-GET/HEAD without Idempotency-Key is not replayable in Go's transport.
    3. httpPostWithRetry (wallet/client/client.go:459-474): this is it. On a transport error it sleeps a backoff and re-POSTs the identical serialized body:
    resp, err := httpClient.Post(url, contentType, bytes.NewReader(bodyBytes))
    if err != nil {
        if attempt < maxRetries {
            time.Sleep(backoffDuration(attempt, 0))
            return httpPostWithRetry(url, contentType, bytes.NewReader(bodyBytes), attempt+1, 0) // same outputs
        }
        return nil, err
    }

    A dropped response after the mint processed the swap (the lane's drop_response) surfaces as exactly such a transport error: the same derivation outputs are re-sent to the mint. The observed chain, matching every lane symptom: first swap processed by the mint → response dropped → transport error → same-body retry (the re-exposure, digests sighted twice) → mint answers 10002 for the already-signed outputs → swapWithRetry's legitimate fresh re-derivation completes the swap → payment returns 1022. The lab mint tolerates the sequence, which is why payments keep working while the invariant is broken; a mint with strict duplicate-output enforcement is the brick scenario the fund-safety rules describe.

    Why this is the wrong retry, per the repo's own rules

    AGENTS.md: "Ambiguous network results must be reconciled, never blindly retried. A swap/melt timeout does not mean failure — the mint may have processed it. On timeout, query proof/quote state before regenerating or retrying." A transport error on a money-moving POST is precisely an ambiguous outcome. The same-body retry is also applied to /v1/mint/bolt11, /v1/melt/quote/bolt11 and /v1/melt/bolt11 — melt retries are the same class (a melt whose response is dropped may have paid the invoice).

    Fix direction (fork-level — OpenTollGate/gonuts-tollgate, then the three-go.mod bump)

    • The transport-error and 5xx retry must be per-route: never for the money-moving POSTs (/v1/swap, /v1/mint/bolt11, /v1/melt/quote/bolt11, /v1/melt/bolt11). For those, an ambiguous outcome returns an error that the caller reconciles — NUT-07 checkstate for the swap outputs' inputs, quote state for melts — before any regeneration (which must use a freshly incremented counter, per the existing 10002 path).
    • Retrying GETs (/v1/keys, /v1/checkstate, keyset endpoints) on transport errors is fine and should stay.
    • The 429 retry-with-Retry-After on POSTs is a separate question: a 429 means the mint did not process the request (it refused it), so a same-body retry there is safe in principle — but only if the mint's 429 is answered before processing, which the current single-status-check shape cannot distinguish for sure. Keep it, but document the assumption.

    Repro: tests/cloud-lab/conformance/run-conformance.sh, row swap-timeout-retry / no-output-reuse=fail (evidence under .conformance/evidence/swap_timeout_retry.log + the fault-proxy observation log).

  2. Amperstrand commented on Oct 5, 2026

    @Amperstrand
    CollaboratorAuthor

    Fix state: root-caused, fixed, and lane-proven — awaiting review/merge chain.

    • Root cause (comment above): the fork's httpPostWithRetry re-POSTed the identical body on any transport error.
    • Fix: gonuts-tollgate#35 — state-changing POSTs are single-shot on no-answer (new AmbiguousOutcomeError, reconcile-first contract), 429 keeps the same-body retry, checkstate keeps its full retry.
    • End-to-end proof (posted on the PR): the module's conformance lane re-run with the fork branch pinned (v0.12.2-0.20261005164740-3c054dfc880d) — swap-timeout-retry / no-output-reuse flips fail → pass; each blinded-message digest is now sighted exactly once even when the swap response is dropped after the mint processed it.

    Merge chain to actually close this: fork #35 → tag → the three-go.mod bump PR in this repo → re-run the lane on the bumped main. Two follow-throughs for the bump PR: teach isAmbiguousMintOutcomeError in src/merchant to match client.AmbiguousOutcomeError (so recordLateReceiveOutcome labels the no-answer case precisely), and expect the swap-timeout-retry row to remain fail on service-or-refund only — that is #403/#258, deliberately not masked here.

  3. Amperstrand commented on Oct 5, 2026

    @Amperstrand
    CollaboratorAuthor

    Chain closed through the tag: fork #34 + fork #35 merged → tagged v0.13.0 → bump PR #643 open (module half of this issue; auto-closes on merge). The lane re-run on the bumped tree confirms swap-timeout-retry / no-output-reuse = pass — the re-exposure is gone through the whole chain, not just on the branch. Remaining red on that row is service-or-refund = the #403/#258 window, owned there.

  4. Amperstrand commented on Oct 7, 2026

    @Amperstrand
    CollaboratorAuthor

    For the record: CDK hit this exact class and fixed it twice. cdk#1278 ("Fix missing try_proof_operation_or_reclaim wrapping of a swap") patched the missed call site — a swap path whose reconciliation wrapper was simply absent, our #640 verbatim. They then went structural: #1456 removed the wrapper from receive ops and #1457 slimmed the saga table, with spend operations now running as persisted sagas where the reconciliation cannot be forgotten — a spend op without its saga state doesn't compile.

    Collected in #703's incident map (Class A) with the lesson spelled out: wrapper discipline rots when someone adds a call site under pressure; the durable fix shape is persisted intents + boot-time resume.

  5. Amperstrand commented on Oct 7, 2026

    @Amperstrand
    CollaboratorAuthor

    Closing the loop with a measurement: the #535 conformance lane re-run on the RC-candidate base with the gonuts v0.13.0 pin shows swap-timeout-retry ... no-output-reuse=pass (full table on #665, run of 2026-10-07T09:20Z). The proxy observed the first swap processed-and-dropped, and the retry path no longer re-exposes deterministic outputs — v0.13.0's AmbiguousOutcomeError contract ("a state-changing POST whose answer never arrived is never re-sent; reconcile first; any regeneration from a freshly incremented counter") holds under the fault injection. The class stays pinned by the lane permanently; the structural saga-shaped fix direction remains tracked in #703.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions