Skip to content

fix(merchant): reconcile late Receive completions after the response deadline (#498) - #511

Closed
Amperstrand wants to merge 32 commits into
fix/500-pending-proof-reclaimerfrom
fix/498-late-receive-reconciliation
Closed

Amperstrand wants to merge 32 commits into
fix/500-pending-proof-reclaimerfrom
fix/498-late-receive-reconciliation

Conversation

@Amperstrand

Copy link
Copy Markdown
Collaborator

Stacked on #507 (same files, trivial context). Fixes #498 (scoped).

What

  • The 30s timeout branch no longer drops the still-running Receive: reconcileLateReceive adopts its outcome — a late success grants the session the customer paid for (visible via GET /usage); a late failure is logged with the mint identity.
  • Timeout notice copy now says the payment may still complete and to check usage before retrying (no more "please try again" invitation to double-pay).
  • receiveTimeout package var + openGateForSession test seam (repo's standard stub conventions).

Why

Every slow-mint crossing of the deadline consumed the token with no session, no record, and a message telling the customer to pay again — value loss on each occurrence (#498).

Scope

Closes the common case (process survives the timeout). The crash-during-window case is the durable payment record (#502); reconcileLateReceive documents itself as crash-unsafe pending that.

Verification

  • Full merchant suite green under -race (224.8s, testenv).
  • New tests: late-success grants (waits on the gate stub, race-clean), late-failure grants nothing.

felixfelix-bot and others added 5 commits September 22, 2026 11:17
Portal main now carries #53 (both Playwright lanes actually run; admin wallet +
wifi coverage) and #54 (descriptor-driven VITE_BRAND brand slot). Pinning to the
current main tip makes the next release the pre14 cut; the build-portal CI job
builds these assets from source at this SHA at packaging time (upstream #335).

Co-authored-by: Felix <felix@c03rad0r.dev>
…) (#475)

* fix(merchant): arm the aggressive probe loop on runtime downgrades (#429)

The runtime downgrade path wired the recovery trigger (#400) but only
the 5-minute proactive check ever probed again — a transient mint blip
left the service degraded for up to a proactive cycle (~13 min observed
live). The aggressive 15 s loop the startup path already uses is now
armed on the downgrade (ArmAggressiveRetry, idempotent, single-flight)
and fires the same first-reachable/set-changed callbacks, so a wired
recovery triggers within seconds of the mint returning.

The three aggressive timing constants moved to per-tracker struct
fields (constructor defaults unchanged): tests shorten them per
instance before any loop starts, eliminating the package-level mutable
state whose restore raced live loop goroutines under -race — the race
was real and caught by the new test's count=10 race runs.

* docs(changelog): move the #429 entry to [Unreleased]

Post-tag change placed under the v0.6.0-alpha3 section — belongs in
[Unreleased] (caught by the independent PR verification sweep).

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…west metric (#454) (#468)

* fix(upstream_detector): match both default-route encodings; prefer lowest metric (#454)

netlink v1.3.x decodes kernel default routes (no RTA_DST) with Dst as
the parsed 0.0.0.0/0 or ::/0 IPNet rather than nil, so the detector's
route.Dst == nil comparisons never matched for IPv4: both route-based
gateway lookups were dead code and x.x.x.1 inference was effectively
the only selection path — pointing reseller mode at the wrong gateway
whenever the real default route's next hop is not the subnet's .1
(live lab case: default via 172.29.0.10 metric 100 ignored in favor of
inferred 172.29.0.1).

isDefaultRoute accepts both encodings for both families;
bestDefaultGateway selects the lowest-metric default (kernel
route-selection order) instead of first-dump-order. Root cause proven
with a netns probe against the exact pinned library version, covering
both the netlib Dst=nil add form and the literal 'ip route add default'
command from the lab evidence.

* docs(changelog): move the #454 entry to [Unreleased]

Post-tag change placed under the v0.6.0-alpha3 section — belongs in
[Unreleased] (caught by the independent PR verification sweep).

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
/#465) (#478)

* fix(config): lenient default prepay — 2.5 GB tank, renew near half

Raise preferred_session_increments_bytes 500,000,000 -> 2,500,000,000
and bytes_renewal_offset 125,000,000 -> 1,225,000,000 (49% of the
increment, just under the #442 clamp so it stays dormant at common
step sizes). The default should just work with a large margin;
per-link tuning stays an operator task (tests/sim/RESULTS.md).

Grounded in the #460/#465 simulation:
- the 10 s renewal trigger throttle caps sustained throughput at
  increment/10 s — 2.5 GB lifts the ceiling to ~2 Gbps (2x line-rate
  headroom); the old 500 MB tank capped gigabit uplinks at ~39%
- the offset covers the sim's offset >= 2 x RTT x link_rate rule up to
  ~5 s payment RTT at 1 Gbps (and the 8 s pathological cell below
  ~610 Mbps); lab-measured payments are 35-53 ms local, 841 ms with
  +800 ms mint netem (#460 comment)
- renewing near half a tank matches the DHCP-T1-at-50% precedent the
  clamp itself cites

Trade-off, accepted: more capital locked per renewal (113 steps at the
reference 22,020,096-byte step, ~1 sat/step gateways) and a higher
minimum reseller balance to bootstrap. RESULTS.md's operator table
says who should tune down (3G/4G/satellite and ~$10/GB-class links).

* docs: link the lenient-defaults PR from changelog and sim results

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…uth (#448)

* ci(ngit): align the go-test lane's Go with the reproducibility pin

The test workflow pinned go-version 1.25.0 (commented as tracking
src/go.mod's declaration) while the release build lane and the #383
reproducibility pin (packaging/build-inputs.json) are on 1.25.8 — the
test suite was not exercising the toolchain that builds the shipped
binaries. The pin now tracks the reproducibility pin, and the comment
says so (src/go.mod's minimum stays 1.25.0; contributor-facing
constraints are unchanged).

Follow-up recorded in the #434 review.

* ci(ngit): align build and test lanes' Go with the reproducibility pin

The cross-compile lane (build-package-binaries.yml GO_VERSION) and the
go-test workflow still pinned 1.25.0 while packaging/build-inputs.json
(#383) and repro-check.yml are on 1.25.8 — the shipped binaries were
built by a toolchain the reproducibility pin contradicts, and the test
suite did not exercise it either. Both literals now say 1.25.8, with
comments anchoring them to the pin. test.yml deliberately resolves from
src/go.mod (the module minimum) and is unchanged.

Also lands the bash-not-sh runbook note for the release driver scripts
(ngit-ci-release.sh / ngit-ci-trigger.sh use bash arrays; dash dies on
them) — the one-liner invited when #434 closed as superseded by #445's
shard architecture, which had orphaned the build-lane pin bump this
carries.

* build(pin): follow the official OpenWrt SDK Go feed; manifest becomes the only Go source

The released 25.12.0 SDK feed ships golang 1.26.8 (golang1.26-1.26.8-r1),
not 1.25.8: the pin moves to what the pinned SDK actually builds Go
packages with. packaging/build-inputs.json now records the official golang
per OpenWrt release line (.openwrt_sdk.go_per_release: 23.05 -> 1.21.13,
24.10 -> 1.23.12, 25.12 -> 1.26.8), audited against the live
openwrt/packages branches and released feeds by the new
scripts/sdk-go-version.sh (print/released/check/update; check fails on
drift, update refreshes the map only).

Every CI lane that needs the pin's Go resolves it from the manifest at
run time -- ngit go-test, ngit repro-check, ngit build-package-binaries
(whose workflow-level GO_VERSION literal had drifted to a stale 1.25.0
after #434 closed unmerged) and the GitHub build-package lane -- so no
lane-local literal can go stale again. build-env.sh gains sdk_series()
and sdk_go_version() so build scripts can ask for a release line's
official Go without re-reading the manifest.

* ci(repro): gate the reproducibility lanes on the SDK Go audit

scripts/sdk-go-version.sh check now runs in both repro-check twins,
right where the pin is already verified against the installed
toolchain, so a pin that stops matching the pinned SDK release's feed
(or a go_per_release entry that drifts from the openwrt/packages
branch) fails the lane instead of waiting for a maintainer to run the
audit by hand.

* ci(version-sync): refuse lane-local Go literals; manifest is the only source

check-version-sync.sh (pre-commit hook + release precondition) gains a
section 10 matching the branch's manifest-derived design: any literal
go-version/GO_VERSION in a workflow fails — even one that matches
today's manifest, since it would go stale on the next pin bump — and
the 16 module go.mod directives are checked for internal consistency
(the module minimum is a separate invariant and may trail the pin).

The enforcement half of the manifest-as-single-source design stacked
on top of the derivation commits.

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…ld (#514)

build-package-binaries.yml invoked portal-build.sh with
PORTAL_REF="${PORTAL_REF:-main}"; since the pin-hardening made a
floating ref a hard error, that default fails the build-portal job on
every main push ('PORTAL_REF main does not match the pinned portal
commit 992cf7f1') while the pinned build itself is green. Verified on
an isolated runner: the verbatim floating invocation reproduces the
error; with the default (manifest pin) the full portal build stages
all five products from 992cf7f1 under the pinned node 22.17.0, and
the cross-compile job is green at the tip under the pinned Go 1.26.8
— so this one default was the whole stage-1 failure.

Co-authored-by: Amperstrand <amperstrand@localhost>
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Verification: PR #511 (late-Receive reconciliation, #498)

Tested as-is at head 0d319076ccaa788ae2067ee5e1a2c2767bf98583 (mergeable, stacked on #507's branch fix/500-pending-proof-reclaimer, 5 commits behind main). The mechanism itself is correct and verified under race: the late-completion grant rides the exact inline grant path, in-process exactly-once holds, and the concurrency division of labor with #507 is confirmed from this diff. But the head tree fails the repo's pre-PR gate in src/cli (inherited from the stack's base, identical to what #507's verification found), and the CHANGELOG entry is missing — so this needs the same small stack-level revision before it can land.

What I ran

  • make go-battery from the repo root: fails at module src/cli — go: updates to go.mod needed; to update it: go mod tidy (exit 2); root module green (ok … 14.9s).
  • Per-module battery (gofmt -l / go vet / go build / go test -race -count=1 -tags testenv) on the modules the diff touches or reaches: src/merchant green end-to-end (ok 220.217s), src/tollwallet green (ok 34.867s), src/cli fails vet and build with the go.mod error above.
  • gonuts sweep over all 16 go.mod files (only 4 reference gonuts): src/merchant / src / src/tollwallet pin v0.12.1 ✓; src/cli still carries v0.11.2 // indirect ✗ — the chore(deps): bump gonuts-tollgate v0.11.2 → v0.12.0 (melt/swap-retry counter reservation + #480 landing) #506 gap class, present in this tree via the base branch.
  • Fix proof: cd src/cli && go mod tidy (flips gonuts indirect to v0.12.1) makes vet+build green there; reverted afterwards.
  • Focused race on the PR's own scenario: go test -race -count=10 -tags testenv -run TestPurchaseSessionLateReceive → 20/20 PASS (2 tests × 10), ok 8.186s, exit 0.
  • No cloud-lab run: no lane exercises PurchaseSession against a slow mint (the lab has renewal_e2e.sh and run-keyset-rotation.sh only); Priority: P1 — ambiguous-outcome class at the TollGate layer. Independent of gonuts fixes. #498's slow-mint proxy lane doesn't exist yet and is fairly follow-up work, not verification of this diff.

Reconciliation semantics — grants the session late, via the inline path

reconcileLateReceive blocks on the same buffered channel the abandoned goroutine feeds, then on success runs identically to the inline success arm: calculateAllotment(res.amount, mint) → grantSessionAccess. That's the best arm of the choice — the customer paid, they get service, visible via GET /usage; no refund-path complexity imported from #403.

Collisions land correctly because grantSessionAccess is additive: AddAllotment on an existing session does Allotment += amount (with a start-time reset) — a late grant stacking on a retry that already succeeded lands as a top-up through the same path as any second payment, and a gate-open failure rolls back to the snapshotted previous session. Same-token retry is safe by mint-side serialization: the two Receives race on the same proofs, exactly one can win; the loser surfaces as ErrTokenAlreadySpent (inline error notice or late-failure log), so exactly one grant per payment in every interleaving. A late failure only logs (mint identity + error) — no state change, no grant.

Idempotency and crash window — honest, explicitly scoped

In-process exactly-once is airtight: the channel is buffered size 1 with exactly one consumer — the inline select if the result arrived in time, otherwise the single reconcile goroutine spawned only in the timeout arm; the goroutine's panic-recover also routes into the channel, so the reconciler always gets a terminal result. Across a daemon restart inside the window the completion is lost — accepted with rationale, documented at the code site ("crash-unsafe by design — the durable version is the payment-record saga (#502)") and in the PR body; #502 (open, P1, business-transaction record) is the right home for the durable version. The PR does not pretend otherwise, which is the correct shape for a P1 stopgap.

Ordering vs #507 and #475 — no new lock interaction; interplay verdict confirmed

I re-derived #507's interplay verdict from this diff and confirm it: reconcileLateReceive calls no wallet methods at all — it touches neither the pending bucket nor the wallet mutex; the only wallet interaction remains the original Receive call (pre-existing, covered by v0.12.1's serialized counter derivation). Lock-wise the reconciler holds nothing while blocked on the channel, takes sessionMu briefly inside AddAllotment/snapshotSession, and shells out to ndsctl with no lock held — no cycle with the reclaimer's wallet-mu sweeps, no shared state, no starvation (the reclaimer's 60 s ticker and this per-timeout goroutine are independent). #475-class rebuilds: sessions are in-memory per-Merchant with no persistence anywhere in the package, so a rebuild between timeout and late completion lands the grant in the old merchant's map — a pre-existing class for any in-flight grant (same family as #507's stale-goroutine note), not a defect this PR introduces.

Tests — the race is pinned, one assertion is too loose

TestPurchaseSessionLateReceiveGrantsSession pins the exact #498 interleaving: deadline shrunk to 50 ms, Receive completes at 250 ms — asserts the timeout notice went to the customer and the completion landed and the eventual state (GetSession().Allotment > 0) and the gate opened; the failure arm asserts nothing is granted. Race-clean at -count=10. One gap: the response assertion is Contains("timed out"), which the old copy also satisfies — assert "may still complete" so #498's copy requirement (message must say the payment may still complete and to check usage) is actually pinned.

Findings

Blocking

  1. Head tree fails make go-battery at src/cli — identical to finding 1 on feat(merchant): pending-proof reclaimer — ambiguous send/melt outcomes self-heal (#500) #507: the stack's dep-bump commits (75a566c, 320896b) updated src/, src/merchant, src/tollwallet but not src/cli, which replace-pins ../merchant/../tollwallet and still carries gonuts v0.11.2 indirect + stale go.sum. Not introduced by this commit, but it's the tree this stack would merge. Fix is go mod tidy in src/cli (verified above) — ideally in chore(deps): bump gonuts-tollgate v0.11.2 → v0.12.0 (melt/swap-retry counter reservation + #480 landing) #506 so the whole stack inherits it.
  2. No CHANGELOG entry. User-visible twice over: late payments now grant the paid-for session, and the timeout notice copy changed. One bullet under [Unreleased] → ### Fixed (which sits above the already-tagged [v0.6.0-alpha3] — the entry must not land under the alpha3 section), linking the PR per house style.

Strongly recommended

  1. "Fixes Priority: P1 — ambiguous-outcome class at the TollGate layer. Independent of gonuts fixes. #498" will auto-close the issue, but Priority: P1 — ambiguous-outcome class at the TollGate layer. Independent of gonuts fixes. #498 is only partially delivered. The issue's acceptance criteria include crash-restart reconciliation (durable record) and the slow-mint cloud-lab lane; both are explicitly deferred here to Priority: P1 — the TollGate-level business-transaction record; generalizes the still-open #403 into a mechanism. #502. Reword to "Part of Priority: P1 — ambiguous-outcome class at the TollGate layer. Independent of gonuts fixes. #498 (common case)" — or the maintainer reopens it post-merge — so the durable-saga remainder keeps a tracking issue.
  2. Strengthen the notice-content assertion to "may still complete" (see Tests).

Notes

  • The timeout-arm comment says a late failure is logged "with the same classification the inline path uses" — it's actually logged raw (mint identity included, which suffices for manual classification) without running the inline branches (MarkUnreachable, token-spent, rate-limit). Harmless since the customer is already answered; just wording.
  • res, ok := <-ch; if !ok is unreachable (nothing closes ch) — fine to keep, just defensive.
  • The receiveTimeout/openGateForSession test seams match the repo's stub conventions (valve.runNdsctl); production behavior of openGateForSession is unchanged by the func→var conversion, and its only caller outside tests (grantSessionAccess) is unchanged. Single commit, no AI attribution, no drive-by changes.

Disposition

request-changes — both blockers are stack-level hygiene, not reconcile-mechanism defects: (1) the src/cli go mod tidy (land it in #506 so #507 and #511 inherit it), (2) the [Unreleased] CHANGELOG bullet, plus ideally the #498 linkage rewording and the copy assertion. The mechanism itself verified green under race and I'd flip to land-with-followups (follow-ups: #502 durable saga, slow-mint cloud-lab lane) as soon as those land.

…counter reservation + #480 landing) (#506)

* chore(deps): bump gonuts-tollgate v0.11.2 → v0.12.0 — melt/swap-retry counter reservation + #480 landing

Picks up the fund-safety train (gonuts-tollgate#28, tagged v0.12.0):

- the #480 fix lands: canonical mint-URL bucket keys, one-time
  max-counter alias merge on open, SaveKeyset monotonic-counter guard —
  service restarts no longer replay swap outputs on alias-carrying
  wallets (#480, #496)
- Melt reserves its blank-output derivation range before PostMeltBolt11;
  a lost Paid-with-change response no longer bricks the melt path when
  the caller retries with a fresh quote (#494)
- swapWithRetry reserves its retry range; the counter no longer trails
  the last exposed range after a 10002-retry (#495)
- writer-audit pin: keyset rewrites can never rewind a counter

Verification: go build/vet clean; race suites green in the two modules
that consume gonuts (src/tollwallet 35s, src/merchant 432s) plus the
root module (21s), all -tags testenv.

* chore(deps): retarget gonuts bump to v0.12.1 (serialized ReclaimUnspentProofs swap)

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
Amperstrand and others added 4 commits September 22, 2026 13:42
#506 bumped the direct declarations (merchant, tollwallet, root) but
src/cli's indirect pin stayed v0.11.2, failing make go-battery at that
module. Tidied; cli battery green under race.

Co-authored-by: Amperstrand <amperstrand@localhost>
* ci(ngit): token-emit the package-job SOURCE_DATE_EPOCH expression

The shard generator rendered the package-job env line from inside the
package-job f-string, where ${{ collapses to ${ — all eleven committed
shards carried

  SOURCE_DATE_EPOCH: ${ needs.resolve-inputs.outputs.source_date_epoch }

The runner passes that through literally and packaging/build-env.sh's
epoch validation (non-integer => tg_die) fails every package job, so the
first ngit-lane release run would have died packaging anything (#441
landed it; the lane has not run since, so it never surfaced — found
during the #474 verification sweep).

The line is now emitted through the @@token@@ mechanism like every other
brace-bearing template (the module docstring's own rule), the eleven
shards are regenerated, and the pipeline test pins both the ${{ }} form
(11/11 shards) and the absence of the collapsed form.

Lane-replica smoke: sourcing build-env.sh with the collapsed value dies
with 'SOURCE_DATE_EPOCH must be a unix epoch integer, got
"${ needs..."'; with a resolved numeric epoch it exports
TG_STRICT_INPUTS=1 and BUILD_TIME_UTC deterministically. Negative
control: reverting one shard to the collapsed form fails the suite (14
checks red).

* docs: link the shards epoch fix from the changelog

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
The release the alpha3 tag could not be: cut per the #339 decision —
alpha3 was tagged but held back before a single artifact shipped (the
ngit mirror was re-keyed mid-migration and the portal pin had drifted
past its manifest), so alpha4 supersedes both it and the never-tagged
alpha2 as the first release to actually publish from the Nostr CI
lane.

Section rollup from current main (bea1c88): the [v0.6.0-alpha3]
section becomes [v0.6.0-alpha4] with the post-tag additions (#463 apk
setup re-run fix, #465 renewal-policy sim) and the former [Unreleased]
entries (#462 bundle-location ADR, #466 full pinned portal bundle)
folded in — 61 entries total. RELEASE-NOTES retitled and extended
(two-predecessor intro, the full-bundle bullet, #463/#465/#462
additions, do-not-reuse extended to alpha1/2/3). VERSION to
v0.6.0-alpha4; compare links updated.

Validation: check-version-sync v0.6.0-alpha4 green; the Go tree is
unchanged since the make go-battery 16/16-green run (the delta since
is packaging/docs/tests/changelog only).

Co-authored-by: Amperstrand <amperstrand@localhost>
…re-auth :443, collision-aware private subnet (#513)

* fix(packaging): depend on nodogsplash instead of replacing it

The SDK package definition declared `REPLACES:=nodogsplash base-files` while
depending on libc alone, and every .ipk recipe stamped the same
`Replaces: nodogsplash` into the generated control file. Replaces means "my
files supersede yours" and the package manager removes the replaced package,
so installing tollgate-wrt deleted nodogsplash -- the captive-portal daemon
the module gates the network with -- and a fresh install came up with no
portal until it was reinstalled by hand.

The shipping-path feed definition (net/tollgate-wrt/Makefile) never had the
bug: `DEPENDS:=+nodogsplash +jq`, no REPLACES. The module's recipes now match
it -- DEPENDS names nodogsplash and jq, the virtual nodogsplash-files
ownership stays (the module still ships files into nodogsplash's config/doc
space), and Replaces narrows to base-files, where only the payload-file
ownership overlap is intentional. Every recipe is covered: packaging/Makefile
(SDK/apk path), packaging/local-build-ipk.sh (deterministic .ipk), the GitHub
build-package lane, and scripts/ngit-gen-shards.py plus its eight generated
ngit CI shards (regenerated; `ngit-gen-shards.py --check` passes).

tests/packaging/package-nodogsplash-dependency_test.sh (offline: no SDK, no
network) checks each recipe, guards the packaging/CI trees against drift, and
builds a real .ipk to assert that the control file opkg consumes now reads
`Depends: libc, nodogsplash, jq` / `Replaces: base-files`.

* fix(captive-portal): allow pre-auth TCP 443 so the :8080 redirect reaches LuCI

uhttpd.main.redirect_https is derived: with a readable cert/key pair it is 1,
and the :8080 LuCI port answers `307 Location: https://<router>/`. The
pre-auth allow list written by setup_nodogsplash() covered :2121, :8080,
:2050, :2051, :8090 and :8443 -- every port the script owns except the one the
redirect targets -- so nodogsplash REJECTed :443 for an unauthenticated
captive-LAN client, the redirect dead-ended on a blocked port, and the
operator could not reach LuCI before authenticating at all: exactly the
lockout docs/architecture/uhttpd-redirect-https-ownership-decision.md exists
to prevent. Reproduced on hardware on the pre15 build.

The rule was not a matter of disagreement but of a missing writer. The Go CLI
already writes it for its own TLS path (src/cmd/tollgate-cli/ssl.go:
allowPort443/removePort443Allow around `ssl enable`/`ssl disable`), and :8443
- the admin board's opt-in HTTPS - has been in the list for the same reason
since the admin board was folded in. First boot, the path every freshly
flashed router takes, was the only path without it.

The check matches the whole field rather than an end-anchored "port 443$": a
list ending in `allow tcp port 8443` must never satisfy the :443 check, and
uci may render a list option either one entry per line or space separated
(verified with BusyBox grep -E). Nothing changes for redirect_https=0
routers: this is an allow rule, not a listener, and the redirect stays
derived instead of being dropped.

tests/uci-defaults-nodogsplash-443_test.sh (offline: fake uci shim, no daemon,
no network) covers the fresh install, duplicate-freedom across re-runs and
against rules tollgate-cli already wrote, the :8443 trap in both list
renderings, and that the six pre-existing rules survive untouched.

See docs/architecture/luci-https-pre-auth-reachability-decision.md.

* feat(setup): collision-aware, randomised management-subnet selection

The private network's /24 was derived deterministically from the LAN
(x.y.(z+1).0/24) with no look at what the router is attached to upstream. A
WAN side that is itself a private LAN in that same /24 -- a hotel, a site
uplink, another router's LAN -- put the management subnet and the uplink
subnet in one network, and the private bridge's default route then pointed at
an address the router also owned: traffic for the upstream's client range was
routed back into the router instead of out of the WAN.

The LAN-adjacent /24 stays the answer whenever it is safe (operators expect it
and the DNS/DHCP defaults assume it); what changes is the check and the
fallback:

- the candidate is checked against the LAN's *real* mask -- a 192.168.0.1/16
  LAN already contains the 192.168.1.0/24 the old rule derived -- and against
  every upstream-side network;
- upstream networks are read from both places an address can live: the
  configured WAN address in UCI (static setups) and every non-LAN/private
  address in `ip -4 -o addr show` (DHCP, STA, PPP, FIPS). The field failure
  was a DHCP upstream, which UCI never sees;
- a collision draws a random non-overlapping /24 from 10/8 (the same BusyBox
  hexdump idiom as the RSSID suffix), retries up to 32 draws, walks 10/8 as a
  last resort, and logs the collision and the pick so a field report shows why
  a router is off the expected subnet.

Everything downstream is unchanged: network.private stays a static
255.255.255.0 with the same DHCP pool, firewall zone and forwarding rules.

The Go side stays out of scope: IPAddressRandomized in
src/config_manager/config_manager_install.go is only ever logged
(src/main.go) and never set or acted upon, and identity.DeriveIPv4's CGNAT
space is not wired into runtime network setup -- the follow-up (the service
owning and persisting the choice) is recorded in the decision doc.

tests/uci-defaults-private-subnet_test.sh (offline: fake uci, fake ip, fake
hexdump, no network) covers all of the above: candidate kept when free, a
colliding DHCP or static upstream, a colliding draw being retried, a wider
than /24 LAN mask, the .255 step-down, two runs over the same collision
picking different /24s, and a router with no upstream information at all.

See docs/architecture/private-subnet-collision-avoidance-decision.md.

* docs(packaging): correct the nodogsplash-purge rationale to what the artifacts show

The packaging commit in this branch attributed the missing captive portal to
`REPLACES:=nodogsplash`. The raw build log for the artifact we actually
installed says otherwise: the apk was produced with

    apk mkpkg --info "provides:tollgate-wrt-any nodogsplash-files=0.6.0_alpha3-r0" \
              --info "depends:libc"

and carries no `replaces:` field at all, so the apk lane cannot have removed the
daemon through Replaces. What the artifact does show is the missing DEPENDS:
nothing pulled or retained nodogsplash, which is the same failure class
packaging/preinst already documents for an undeclared runtime dependency that a
maintainer script needs (issue #93).

Corrected in the recipes' comments (packaging/Makefile,
packaging/local-build-ipk.sh, scripts/ngit-gen-shards.py), the CHANGELOG bullet
and the test header.

The fix itself is unchanged and remains correct: declare the runtime dependency
the way the shipping path does (`DEPENDS:=+nodogsplash +jq`), and never claim to
replace the daemon the module gates the network with. The .ipk lane's
`Replaces: nodogsplash` did supersede the named package and stays removed.

* fix(setup): re-assert the captive-portal allow list on every setup path

The same-version branch of 99-tollgate-setup — the path taken when
/etc/tollgate-setup-done already equals SETUP_VERSION — verified the
wireless APs and re-asserted the uhttpd contract, but never wrote
nodogsplash's users_to_router allow list. An install whose version string
is unchanged (the pre15 round over pre14: both v0.6.0-alpha3, same apk
version, flag left equal) therefore inherited the previous install's list
verbatim and came out without the pre-auth :443 entry: nodogsplash
REJECTed the :8080 -> https:// redirect hop, LuCI stayed unreachable and
the post-install sweep scored 4/9, while a clean install of the same
build scored 9/9.

Extract the writer into assert_nodogsplash_allow_entries() and call it
from setup_nodogsplash() (first boot / full setup — behaviour unchanged)
and from the same-version branch, next to the uhttpd re-assertion. The
writer is idempotent: each rule is added only when its whole entry is
absent, so re-running adds nothing and duplicates nothing, and a config
that has :8443 but not :443 gains exactly :443. The anchored
'port 443'-as-a-whole-field match is preserved so :8443 can never satisfy
the :443 check.

Offline tests (fake uci shim): tests/uci-defaults-same-version-allowlist_test.sh
covers the same-version repair, idempotence (exactly one entry per port
after two runs) and the :8443 anchoring. The seed() helper of the
existing 443 test word-split its argument and seeded a non-faithful list;
fixed so the existing assertions run against an honest fixture. No
existing assertion was weakened or removed.

* chore(release): bump VERSION to v0.6.0-alpha4

Installing the captive-portal fixes over an alpha3 install was not an
upgrade: both artifacts carried 0.6.0_alpha3-r0, so apk had nothing to do,
/etc/tollgate-setup-done stayed equal and the setup script's same-version
short branch was the only branch that ran — the branch that, before the
previous commit, never wrote the nodogsplash allow list. A changed version
string makes the install a real upgrade (0.6.0_alpha4-r0) and puts the
repaired full path back in charge.

VERSION stays the single source of truth: packaging/Makefile,
scripts/build-sdk-package.sh, packaging/local-build-ipk.sh and the
__TOLLGATE_VERSION__ substitution all derive from it, and no version
literal is added anywhere else.

- CHANGELOG.md: [Unreleased] folded into the '## [v0.6.0-alpha4]' heading
  as docs/release-process.md prescribes, with a short preamble naming what
  this packaging-fix alpha contains, and the compare links renumbered.
- RELEASE-NOTES.md: retitled for v0.6.0-alpha4 with a preamble plus a
  "What v0.6.0-alpha4 changes" section; the v0.6.0-alpha3 note is kept
  verbatim below it, because alpha4 changes packaging only and the
  capability, known-issue and verification detail there still applies.

Gates: scripts/check-version-sync.sh green (20 ok, 0 fail),
python3 scripts/ngit-gen-shards.py --check green (11 shards match — they
carry no version literal, so there was nothing to regenerate).

---------

Co-authored-by: Felix <felix@c03rad0r.dev>
Amperstrand pushed a commit that referenced this pull request Sep 22, 2026
Verification-sweep follow-up: the entry was missing; post-alpha3-tag
change belongs in [Unreleased]. Also note: Refs #498 rather than Fixes
— the durable-saga criteria remain open in #502.
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Stack-level follow-ups resolved: cli tidy via #515, CHANGELOG entry added at branch head (bb7cf679). Remaining recommendations: reword the auto-close footer to Refs #498 (durable-saga criteria live in #502) and pin the updated notice copy in the assertion. Mechanism already race-proven — land-with-followups.

@Amperstrand
Amperstrand force-pushed the fix/498-late-receive-reconciliation branch from bb7cf67 to b159a86 Compare September 22, 2026 13:40
Amperstrand pushed a commit that referenced this pull request Sep 22, 2026
Verification-sweep follow-up: the entry was missing; post-alpha3-tag
change belongs in [Unreleased]. Also note: Refs #498 rather than Fixes
— the durable-saga criteria remain open in #502.
felixfelix-bot and others added 9 commits September 22, 2026 17:49
…c decode + guard the shipped bytes (#517)

* fix(packaging): pin the portal to the keyset-agnostic decode and regenerate the committed bundle

The shipped captive portal is built from .portal.commit (992cf7f1), whose
token-validation path decodes with getDecodedToken(token) - a call that needs
a MintKeyset list and therefore throws "A short keyset ID v2 was encountered,
but got no keysets to map it to" on every v4 cashuB token carrying a short
keyset id, which is what coinos/minibits hand out. The portal surfaced that
as #CU102 and the payment could not be made.

Reproduced against the pinned dependency (@cashu/cashu-ts 2.9.0) with the
coinos v4 fixture the portal itself carries in tests/unit/mint-fee.test.js:
getDecodedToken(token) throws the keyset error, getTokenMetadata(token)
returns the proofs.

.portal.commit moves 992cf7f1 -> d699367 (upstream main, portal #55), which
decodes through the keyset-agnostic getTokenMetadata first, and the committed
bundle was regenerated from that pin with 'bash packaging/portal-build.sh'
(pinned node v22.17.0 / npm 10.9.2).

Regenerating also repairs the drift in the checked-in copy: its
asset-manifest.json still listed assets/index-DxBkINUB.js - an asset no
current pin builds - it carried a welcome.html the build no longer produces,
and it was missing the logo192/512.png the PWA manifest.json references. The
committed copy now matches the pinned build output byte for byte, so
.pre-commit-config.yaml's whitespace fixers are told not to rewrite it and the
pin manifest's public sha256 pins are recorded in .secrets.baseline.

* test(packaging): guard the shipped portal bundle and the production mint list

tests/packaging/assert-portal-bundle-contract.sh fails the build when

  A. the revision pinned in packaging/build-inputs.json validates tokens
     with a keyset-requiring decode (getTokenMetadata absent), which rejects
     every v4 cashuB token with a short keyset id - the #CU102 regression -
     and
  B. the committed bundle no longer matches what the pin builds (staged
     build from a different revision, or a dirty tracked copy under
     packaging/files/tollgate-captive-portal-site|tollgate-admin).

Check B is what keeps the checked-in copy honest: it is the drift that let a
stale asset-manifest.json (index-DxBkINUB.js, asset missing) sit in the tree
while every build produced something else.

It runs on every pull request (contract-lint in test.yml, which is not gated
on same-repo PRs) and again in build-portal after packaging/portal-build.sh,
where the staged bytes exist. Note for the record: the two cashu-ts strings
in the original report ("short keyset ID v2 was encountered" / "no keysets to
map it to") are dependency internals that are present in bundles built from
BOTH the regressed and the fixed revisions - they are not a usable gate, so
the pinned source is checked instead and the bundle markers are printed for
the record.

config_manager_production_mints_test.go pins the release-line mint defaults
against the test-mint hostnames (testnut.cashu.space, nofee/nofees variants,
testnut.cashu.exchange) and keeps the IsDevBuild() gate that owns
testnut.cashu.exchange under test, so a test mint cannot reach a paying
customer's default configuration unnoticed.

---------

Co-authored-by: felixfelix-bot <felixfelix-bot@users.noreply.github.com>
…licating NDS rules (#516) (#518)

The users_to_router duplicate guards grepped plain substring or
end-of-line 'port N' patterns, but uci renders list options as a quoted
single-line list, so the guards never matched and each rerun appended
its rule again — live router evidence: 27x port 2051, 4x port 443.

All nine guards now share uci_list_has_port: a whole-field matcher
covering both uci renderings, keeping 8443-vs-443 and prefix traps
closed. Ten sandbox cases across both renderings; live disease
captured on-router.

Co-authored-by: Amperstrand <amperstrand@localhost>
…ating main/node (#477)

* ci(ngit): build the stage-1 portal from the pinned toolchain, not floating main/node

build-portal overrode PORTAL_REF with a floating 'main' and setup-node
with a floating '20'. Both are refused by portal-build.sh's
reproducibility gates (portal commit and node/npm pins live in
packaging/build-inputs.json), so every stage-1 run on main since the

  ERROR: PORTAL_REF='main' does not match the pinned portal commit '…'.

Drop the override — portal-build.sh already defaults PORTAL_REF to the
manifest pin — and pin node 22.17.0 plus the twin's npm verification
step, matching .github/workflows/build-package.yml.

Evidence, run locally at this commit:
- PORTAL_REF=main bash packaging/portal-build.sh  -> gate error, rc=1
- system node v22.22.3                            -> toolchain gate, rc=1
- pinned toolchain (manifest tarball, sha256-verified) + no override
  -> full portal build from the pinned commit, rc=0

* docs(changelog): record the stage-1 portal toolchain fix (#477)

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…allet (#486)

The wallet's accepted-mint set was frozen at construction from the boot
probe: a configured mint that was unreachable while the tollgate started
(mint outage during router boot) rejected tokens forever — even after it
came back healthy. test_mint_failure passes because mints accepted at
boot re-probe and recover; only the never-admitted case was stuck.

- WalletPort gains AcceptMint (idempotent, best-effort registration);
  implemented by the gonuts wallet, the cdk adapter and the sidecar RPC
- the accepted set read in Receive is now lock-guarded — AcceptMint
  grows it at runtime while payments read it
- the reachable-set-changed callback in main grows the wallet with every
  mint the tracker sees reachable (same 3-probe threshold as recovery)

Cloud-lab proof (staggered start, no restart): boot with mint-fees down
-> token rejected (400 'not accepted'); mint-fees up; three probe cycles
later the wallet logs 'AcceptMint: admitted mint http://mint-fees:8085/'
and the same payment path returns 200 (#481).

Co-authored-by: Amperstrand <amperstrand@localhost>
…oken (#512)

* test(cloud-lab): pin that an outage-refused payment never burns the token

The #423-class property, pinned at the tollgate HTTP surface. New
three-phase lane run from the host (the client has no docker control,
and 'compose run' would restart a stopped mint through depends_on — the
outage phase runs with --no-deps):

- A1 (mint up): fund and stash the token
- A2 (mint down): payment refused without a crash; a 1022 here would
  mean offline acceptance exists and must be consciously re-pinned
- B (mint back): every proof UNSPENT at the mint (NUT-07) and the same
  token still buys a session, byte for byte

Plus a default-lane replay test: a token that already bought a session
never buys another. test_mint_failure covers degradation with fresh
tokens per attempt; the same-token-across-outage leg was pinned nowhere.

* docs(changelog): link the rejection-safety lane PR (#512)

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…ion default (#492)

The shipped defaults live in two tables — NewDefaultConfig (what a
fresh install gets) and the Default fields of GetConfigSchema (what
the config UI presents) — and nothing kept them in lockstep: the
lenient-defaults change (#478) had to update both by hand.
TestSchemaDefaultsMatchNewDefaultConfig walks the schema tree and
requires every leaf Default to equal the shipped default (numbers,
strings, bools, duration-like strings vs time.Duration nanoseconds),
treating omitempty zero-values as consistent and skipping list-shaped
fields whose Children model item shape.

Writing it caught live drift: the schema's config_version default was
still v0.0.7 while NewDefaultConfig and the migration stamp (and now
the README) say v0.0.8. Fixed; the test pins both classes, verified by
mutation: a one-sided default change and a version regression both red
the suite.

Co-authored-by: Amperstrand <amperstrand@localhost>
…ane shareable (#485)

* test(cloud-lab): make the default client run green and the rotation lane shareable

- the two-router module now skips as a whole when the reseller service
  is not up, instead of failing two of its three tests with connection
  errors in a default 'docker compose run --rm client' invocation
- the quick start brings up the full default topology: the fee tests
  need mint-fees, and a bare 'mint upstream' start left them red
- register the requires_docker pytest mark (no more unknown-mark
  warnings on every run)
- run-keyset-rotation.sh honors COMPOSE_PROJECT_NAME-based project
  isolation, layers an optional CLOUD_LAB_EXTRA_COMPOSE override (e.g.
  stripping host port bindings when another service holds 8085/8087),
  and takes the settle-probe URL from MINT_ROTATE_PROBE_URL

Found running the full suite against main on a shared dev box (#482).

* docs(changelog): record the cloud-lab suite hygiene fixes (#485)

* test(cloud-lab): layer compose files in the rotation lane; drop the two-router skip hunk

CLOUD_LAB_EXTRA_COMPOSE built 'docker compose -f $EXTRA' — an explicit
-f replaces the default compose file set instead of layering, so the
documented port-stripping override use case died with 'service reseller
has neither an image nor a build context', and once any -f is passed the
auto-loaded docker-compose.override.yml stops loading (#436's trap; the
same fix renewal_e2e.sh carries). The array now passes the base file
explicitly, re-appends the override, then the extra file.

The two-router skip moves to #484 wholesale: its module-level skipif is
the dedicated implementation (rebased on current main and re-verified),
so this branch drops the overlapping autouse-fixture hunk and its
changelog clause to avoid a second-merge conflict.

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
* ci(ngit): pin UPX in the release shards via fetch-upx.sh

The upx shards install upx-ucl from apt unpinned, but upx output IS
part of the artifact bytes, so those legs were reproducible only as
long as the CI hosts' apt snapshots agreed — while
packaging/build-inputs.json already pins UPX 5.2.1 (tarball + sha256)
and scripts/fetch-upx.sh fetches exactly that. The GitHub twin's
Install UPX steps already use the fetcher; this ports the same pin to
the ngit shards, in the generator, and re-renders all eleven.

Also fixes a latent break in the apk SDK lane: the ultrabrute leg sets
USE_UPX=1 and packaging/Makefile then runs upx from PATH inside the
SDK container — which ships no upx and had nothing provisioning one
(the leg has never run: the lane has not published a release yet). The
shard now streams the pinned binary into /usr/local/bin/upx in the
container before the build.

tests/ngit-release-pipeline_test.sh now asserts every rendered shard
avoids apt upx-ucl, that any upx-compression shard uses the fetcher,
and that the apk ultrabrute shard provisions the container binary.

Found by the v0.6.0-alpha3 lane replica (#471).

* changelog: pin UPX in the ngit release shards

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…ow-up) (#479)

Review wording nit on the merged entry: 'no longer import the concrete
wallet package' was true of merchant_token_flow_test.go but misleading
about the package — tokenfixture_test.go still imports gonuts' cashu,
which is the whole design (one place, not zero places). Now phrased as
the single place a wallet swap touches. [Unreleased] copy only; the
alpha3 release section is frozen history.

Co-authored-by: Amperstrand <amperstrand@localhost>
Amperstrand pushed a commit that referenced this pull request Sep 22, 2026
Verification-sweep follow-up: the entry was missing; post-alpha3-tag
change belongs in [Unreleased]. Also note: Refs #498 rather than Fixes
— the durable-saga criteria remain open in #502.
Amperstrand and others added 9 commits September 22, 2026 18:11
…ng (#484)

* test(cloud-lab): skip two-router autopay unless the reseller is running

A default 'docker compose run --rm client' failed its two reseller
tests on connection-refused because the reseller lives behind the
two-router profile — every fresh checkout's first lab run looked red
with no defect present. The module now probes the reseller at
collection (5s grace so a profile run with a slow-booting reseller is
not wrongly skipped) and skips with the profile start command as the
reason; the first test's own internal skip becomes redundant but
stays (it also covers upstream).

Default suite: green with the module skipped; profile run: module
executes and passes — both verified on an isolated runner.

* docs(changelog): two-router skip gating entry (#484)

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
The [Unreleased] entries accumulated since the team's alpha4
redefinition all belong to this release: the v4 (cashuB) short-keyset
token fix (#517's portal repin to d699367 — coinos/minibits customers
could not pay), the stage-1 toolchain pin (#477), the portal-bundle
regeneration + contract guard (#517), the production-mint guard (#517),
and the shard SOURCE_DATE_EPOCH fix (#491). Folded into the
[v0.6.0-alpha4] section (24 entries), fresh empty [Unreleased] left on
top. RELEASE-NOTES gains the v4 token-fix bullet in Notable bug fixes.

VERSION unchanged (v0.6.0-alpha4); check-version-sync green.

After merge: delete the stale v0.6.0-alpha4 tag (fd58297 — never
mirrored, never announced; it predates this wave and carries the
node-pin defect), re-tag this merge commit, then DQ05 (mirror push +
drive) per #339.

Co-authored-by: Amperstrand <amperstrand@localhost>
…in) (#476)

* test(cloud-lab): live external-mint lane against testnut (nutshell main)

First committed coverage against a Cashu mint this repo does not
control: testnut.cashu.exchange, nutshell's main branch with a
FakeWallet backend — free test ecash, but production HTTP surface,
keyset handling and fee semantics (active sat keyset charges
input_fee_ppk=10, plus a real rotated/inactive keyset history).

- New profile-gated 'upstream-ext' service: a second upstream whose
  accepted_mints also lists testnut, so live-mint payments have their
  own router and the default upstream stays offline-only. Host port
  21214.
- run-external-mints.sh drives it (KEEP honored); EXTERNAL_MINTS=1
  gates the module out of default runs, and every test skips with a
  reason when the mint is unreachable — a canary lane, not a gate.
- Pins, against the real mint: the active keyset's fee policy; a 1-sat
  token terminated by the #409 below-swap-fee pre-check (terminal
  payment-error-below-swap-fee); and a 10-sat token crediting a
  fee-deducted session (live fee, decomposition-independent range).
  Real-money mints are deliberately never probed.

Validated live twice (release-tag VM run in #461's evidence, and this
branch): 3/3 green; default suite unchanged (20 passed, 6 skipped —
the new module skips by design).

* docs(changelog): external-mint lane entry (#476)

* test(cloud-lab): move the stray mid-module import to the header

Self-review finding: 'import sys' sat after the pytestmark block,
between a MAC-related comment and _mac() — imports belong in the
header. Mechanical move, no behavior change.

* docs(changelog): move the external-mint lane entry under [Unreleased]

The entry landed after the #405 bullet inside the v0.6.0-alpha2 section
— post-tag changes belong under [Unreleased] → Changed / Internal
(AGENTS.md: CI/test work goes in that category).

---------

Co-authored-by: Amperstrand <amperstrand@localhost>
…up script (#470)

script's case pattern: the packaging substitutes the placeholder
globally, so the pattern was rewritten to the real version, matched the
substituted assignment, and forced the fallback (degenerating to
'unsubstituted' on apk) on every shipped build. #463 made the gate
itself substitution-proof; this adds the missing regression guard so a
future second code occurrence fails check-version-sync.sh before it
ships. Mentions inside comments are exempt (#463's fix carries one).

Co-authored-by: Amperstrand <amperstrand@localhost>
… counter reservation + #480 landing

Picks up the fund-safety train (gonuts-tollgate#28, tagged v0.12.0):

- the #480 fix lands: canonical mint-URL bucket keys, one-time
  max-counter alias merge on open, SaveKeyset monotonic-counter guard —
  service restarts no longer replay swap outputs on alias-carrying
  wallets (#480, #496)
- Melt reserves its blank-output derivation range before PostMeltBolt11;
  a lost Paid-with-change response no longer bricks the melt path when
  the caller retries with a fresh quote (#494)
- swapWithRetry reserves its retry range; the counter no longer trails
  the last exposed range after a 10002-retry (#495)
- writer-audit pin: keyset rewrites can never rewind a counter

Verification: go build/vet clean; race suites green in the two modules
that consume gonuts (src/tollwallet 35s, src/merchant 432s) plus the
root module (21s), all -tags testenv.
…s self-heal (#500)

Proofs that enter the pending/reserved bucket through an ambiguous
send/melt outcome (interrupted token hand-off, lost melt response) were
stranded until a human ran the gonuts CLI: invisible in balance,
skewing payout math, indistinguishable from lost funds. ReclaimUnspentProofs
existed but nothing in the daemon ever called it.

The merchant now sweeps pending proofs after a boot grace period and
then once a minute: unspent proofs are re-swapped back into spendable
balance; spent-but-still-pending proofs stay put and surface through a
persistent log line (the operator signal that a payout/melt needs manual
reconciliation). Reclaim errors are retried on the next sweep.

Capability-discovered by type assertion on a narrow interface rather
than by growing WalletPort: the sidecar/CDK backends do not implement
pending-proof recovery yet, and extending the wire protocol is its own
change (noted on #500). Requires gonuts >= v0.12.1, whose
ReclaimUnspentProofs now serializes its swap against concurrent wallet
operations (gonuts-tollgate#29).

Tests: boot reclaim, no-op when nothing pending, error-then-retry
convergence, unsupported-backend no-op (race-clean).
…deadline (#498)

PurchaseSession's 30s timeout abandoned the still-running Receive
goroutine: a swap that completed at t=31s landed the funds with no
session, no record, and a "timed out, try again" notice — the customer
paid twice for one session, every time a slow mint crossed the deadline.

The timeout branch now hands the channel to reconcileLateReceive: a late
success grants the session the customer paid for (visible via GET
/usage); a late failure is logged with the mint identity. The timeout
notice copy now says the payment may still complete and to check usage
before retrying. The response deadline is a package var (receiveTimeout)
and gate opening gains the standard test seam (openGateForSession).

Scope note: this closes the value-loss window for the common case (the
process survives the timeout). The crash-during-window case needs the
durable payment record (#502) — reconcileLateReceive is deliberately
documented as crash-unsafe pending that.

Tests: late success grants (race-clean, waits on the gate stub);
late failure grants nothing. Full merchant suite green under -race
(224.8s), testenv tag.
Verification-sweep follow-up: the entry was missing; post-alpha3-tag
change belongs in [Unreleased]. Also note: Refs #498 rather than Fixes
— the durable-saga criteria remain open in #502.
@Amperstrand
Amperstrand force-pushed the fix/498-late-receive-reconciliation branch from f5fb4ee to d795e86 Compare September 22, 2026 16:17
@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Integration verification against this branch in the cloud-lab (frozen-mint scenario, docker pause):

What works: the scoped fix is sound — unit coverage (late-success grants / late-failure grants nothing) plus live boot is clean, and the recovery path is correct: after the mint thawed, the exact token that lived through the wedge paid 200 in 0.1 s, no residue.

Adjacent gap found (not this PR's scope, filing separately): with a wedged/frozen mint, payment requests never reach the receiveTimeout wrapper. Goroutine dumps (SIGQUIT, two in-flight payments, minutes deep):

HandleRootPost → PurchaseSession (merchant.go:482) → SwapFeeSats
→ mintKeysetFees → GetMintActiveKeyset → getMintActiveKeysetFallback → client.go:134

The fee precheck's keyset fetch blocks first: each HTTP call is 30 s-timeout × up to 5 attempts with backoff, and GetMintActiveKeyset chains two endpoints — a wedged mint stalls the precheck for ~5+ minutes, so the "fall through and let Receive classify" path (which needs the calls to return) never fires and the 30 s timeout below it is unreachable. Customer sees a multi-minute hang instead of the new notice. Fix shape candidates: bound the precheck with a short context deadline and reuse the existing error fall-through; or serve SwapFeeSats from the wallet's cached keyset (InputFeePpk is already in the local DB — the network fetch looks avoidable entirely).

Worth noting for review ordering: this doesn't weaken the PR's claims (its window is Receive-start-to-deadline, unit-proven); it's the next window over.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Mergeability note: GitHub's cache has reported this PR CONFLICTING for hours while git merge-tree against current main proves the tree CLEAN at every pushed head (including after an empty-commit push to force recomputation, c5c50596). All content blockers from the verification pass are resolved (cli tidy via #515, CHANGELOG at head). If the UI still refuses, merging via the API with a fresh recompute or a one-line revert-and-reapply of the entry hunk will clear it — the code itself is conflict-free and race-proven.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Live validation of the late-reconciliation claim (cloud-lab, this PR stacked with #532's precheck budget — previously your coverage was unit-level only):

step result
payment with the mint frozen 400 in 33.0 s, payment-processing-timeout, "may still complete…" copy
mint thawed 9 s later PurchaseSession: late Receive completed … amount=50 — granting the paid-for session → 50 steps converted (3,000,000 ms), gate opened, MAC authorized
replay of the same token 400 / 21023 — no double grant

Without #532 this scenario cannot reach your timeout wrapper (the fee precheck parks first — see #525); the two compose exactly as designed.

One integration note for your rebase: your branch predates #486, and merging it with current main silently drops TollWallet.AcceptMint from tollwallet.go (textual overlap in the file) — the root module stops compiling (w.inner.AcceptMint undefined). Worth rebasing before review completes.

@Amperstrand Amperstrand left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evidence pass (unattended run) — request-changes: the branch tip does not compile.

go vet ./... in src/tollwallet fails:

gonuts_wallet.go:261:17: w.inner.AcceptMint undefined
(type *TollWallet has no field or method AcceptMint)

The branch pins gonuts-tollgate v0.12.1 — the same tag main pins (verified both go.mod files) — and v0.12.1's TollWallet has no AcceptMint method. The new delegation (GonutsWallet.AcceptMint) targets an API that exists in no released tag this branch declares: it needs either a newer gonuts release (retarget + refreshed go.sum) or different wiring. The merchant suite cannot run past this (module dependency), so the reconciliation logic is unverified-by-build at tip.

Stack hygiene: the branch carries 6 unique commits — a duplicated dependency pair (the v0.12.0/v0.12.1 bumps that already landed on main differently via #506/#515), the #500 reclaimer, the #511 fix, its changelog entry, and a mergeability-recompute trigger. That duplication is why the PR diff balloons to AGENTS.md/README/workflow files. Recommend: rebase onto current main, drop the duplicated dep commits (keep main's pins), and let the diff shrink to the real surface — the pending-proof reclaimer (+78/+98), the late-receive test (+167), and the reconciliation wiring — then rebase the AcceptMint call against whatever tag actually ships it.

The design shape (a reclaimer that self-heals ambiguous send/melt outcomes + late-Receive reconciliation) is the right skeleton for #498's class — but it needs to compile before the logic can be credited.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Extensive-test pass (per the critical-but-high-risk protocol: test, don't merge).

Headline: the branch does not build against current main. tollwallet fails vet/compile at gonuts_wallet.go:261 — w.inner.AcceptMint undefined (type *TollWallet has no field or method AcceptMint). Root cause: this branch's TollWallet refactor predates #486 (merged) and drops the AcceptMint surface that Merchant.AdmitReachableMints now calls. The PR's CONFLICTING status was the textual symptom; this is the semantic one — none of its tests are runnable until it rebases (the full make go-battery on this tree dies at the same line, EXIT=2). Fix on rebase: restore the AcceptMint passthrough in the refactored TollWallet (one method), then the suites below can actually run.

What I could verify without building — the design, against the repo's own fund-safety doctrine:

  • pendingProofReclaimer resolves ambiguous outcomes via mint-reported proof states — reconciles, never blindly retries (spent-at-mint proofs stay pending and surface as the operator's manual-reconciliation signal, not a silent retry). That is exactly the AGENTS.md rule for ambiguous swap/melt results.
  • The capability is discovered by type assertion on a deliberately narrow interface instead of growing WalletPort and the sidecar wire protocol — right call; backends that can't resolve ambiguity degrade to a logged skip.
  • Boot-delay + minute sweep; the no-pending fast path is one in-memory bucket read.

Against current main this PR is much smaller than it looks: +434/−39 across 8 files (the earlier 4.6k-line footprint was against its old base — the overnight wave absorbed its workflow churn). That materially lowers its risk profile once it compiles.

Test plan ready to execute the moment it rebases and builds: full make go-battery; merchant + tollwallet race suites; its two new test files (purchasesession_late_receive_test.go, pending_proof_reclaimer_test.go); the cloud-lab payment lanes incl. a frozen-mint scenario (per its own verification notes). I'll run the full pass on request after the rebase lands.

Disposition: do not merge — blocked at compile; rebase + restore AcceptMint, then re-request the extensive pass.

@felixfelix-bot felixfelix-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — CHANGES REQUESTED at head c5c505961c5bfa22d6bd65a93955c6d76e9541d4 (all evidence re-derived first-hand this run: fresh clone, rev-parse verified; gate 2.5 cold audit by an independent cross-family reviewer confirmed verdict + findings).

The reconciliation mechanism itself is sound. The late grant rides the exact inline success path (calculateAllotment → grantSessionAccess, src/merchant/merchant.go:667-674); in-process exactly-once holds — the buffered channel has exactly one consumer (the inline select, or the reconciler spawned only in the timeout arm; the panic-recover send can't double-send); AddAllotment is additive under sessionMu, so a late grant stacking on an already-succeeded retry lands as a top-up; the reconciler never re-submits the token (no derivation-range re-use); a late failure only logs; no lock is held across the ndsctl call; the crash window is honestly scoped to #502. gofmt clean on both touched modules; gitleaks 8.21.2 over the true stack diff (a99d983..c5c5059): no leaks.

But the tree this PR would merge does not build, does not merge cleanly, and now competes with a different interim policy already on main:

  1. [BLOCK] The branch does not compile — root cause is in this PR's own stack, not in main moving underneath. go vet ./... from src/tollwallet (and every module importing it, including src/merchant under -tags testenv): src/tollwallet/gonuts_wallet.go:261:17: w.inner.AcceptMint undefined (type *TollWallet has no field or method AcceptMint). Stack commit 1e7107f rewrote tollwallet.go and dropped TollWallet.AcceptMint + mintMu (both present in the merge-base via #486) while leaving the GonutsWallet delegation at gonuts_wallet.go:261. Blast radius: accept_mint_test.go and merchant_late_mint_test.go also fail to build in every default configuration — none of this PR's (or the reclaimer's) tests can run at head. Control at origin/main: vet clean, tollwallet suite ok 32.165s. Fix in this PR: restore the AcceptMint surface + mintMu locking in the rewritten TollWallet (or split the reclaimer commits out) — a plain rebase will not bring the method back.

  2. [BLOCK] Real merge conflict whose resolution is a policy decision. git merge-tree --write-tree origin/main pr-511 → exit 1, CONFLICT (content): src/merchant/merchant.go. The conflict is the timeout arm itself: main has since adopted the opposite interim behavior (#548): payment-outcome-unknown notice, an operator receiveReference, explicit "Do not send this e-cash note again — a second attempt will be refused" copy, grant-nothing, with the late-grant journal explicitly deferred — and src/merchant/late_receive_notice_test.go on main pins that no-grant contract. This branch auto-grants and tells the customer "may still complete. Check your usage before retrying." Resolving the markers means choosing one design (and updating main's test to match); you cannot keep both.

  3. [RISK] The #486 regression is semantic, not just textual. Branch src/merchant/merchant.go:206 (AdmitReachableMints) still calls tollwallet.AcceptMint(...), but the rewritten Receive reads w.acceptedMints with no lock (main: mintMu-protected read) while the health tracker appends at runtime — the exact data race #486's mintMu closed. Whatever design wins, the locking must come back with the method.

  4. [RISK] The PR body's verification claim is stale at head. "Full merchant suite green under -race (testenv)" is not reproducible at c5c5059 — the merchant package fails to build under testenv because its tollwallet dependency does not compile. The run was real at the earlier pre-stack-move head (documented in your 09-22 comment); re-run and re-state it after the build is fixed.

  5. [NIT] CHANGELOG.md:87 — the new entry drifted into the tagged, released [v0.6.0-alpha4] section ([Unreleased] at line 11 is empty on this branch). Belongs under [Unreleased].

  6. [NIT] src/merchant/purchasesession_late_receive_test.go:90,135 — Contains("timed out") is satisfied by the old copy too; pin whichever notice copy survives finding 2 (today that's contested between this branch and main).

  7. [NIT] PR body "Fixes #498 (scoped)" — auto-closes #498 on merge while its durable criteria live in #502 (both open). Refs #498 keeps the tracking honest.

No CI evidence exists for this head (GitHub check-runs: 0; ngit kind-9842: no results) — a documented absence, not a pass.

Disposition: fix the AcceptMint surface in-PR, decide the timeout-arm policy against main's #548 (I'd argue the auto-grant here is the better end-state for the customer — it is what #548's own comment says the journal should eventually do — but it must be a deliberate supersession, updated tests and all), move the CHANGELOG entry, then re-request review.

@felixfelix-bot

Copy link
Copy Markdown
Collaborator

CHANGES REQUESTED at head c5c505961c5bfa22d6bd65a93955c6d76e9541d4 — formal review posted alongside; this is the summary. All evidence re-derived first-hand this run (fresh clone, rev-parse verified); independent gate-2.5 cold audit confirmed verdict and findings (2 corrections incorporated).

One line: the late-Receive reconciliation mechanism is sound and arguably the better customer end-state, but this PR's own stack deleted TollWallet.AcceptMint (branch doesn't compile), the timeout-arm conflict with main's #548 is a policy choice, and the CHANGELOG entry landed in a released section.

Findings: 2×[BLOCK] (compile break from stack commit 1e7107f — not fixable by rebase alone; merge conflict = competing timeout-arm designs), 2×[RISK] (#486 locking regression; stale "suite green" verification claim in the PR body), 3×[NIT] (CHANGELOG section, test copy assertion, Fixes→Refs #498).

Evidence: go vet FAIL gonuts_wallet.go:261:17 (control at origin/main clean, suite ok 32.165s); git merge-tree exit 1 (src/merchant/merchant.go); branch tests not runnable at head — recorded as such, never claimed green. gofmt clean; gitleaks 8.21.2 on a99d983..c5c5059: no leaks. No CI evidence for the head (check-runs 0, ngit kind-9842 no results) — documented absence, not a pass.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Technical review (evidence pass) — PR head c5c505961c5bfa22d6bd65a93955c6d76e9541d4 (stacked on fix/500-pending-proof-reclaimer, which is not main; the branch is 34 commits behind main).

The change itself — reconcileLateReceive adopting the outcome of a Receive that outlived the response deadline, a late success granting through the shared grant path, honest crash-unsafety scoping to #502 — is well built, and I verified it green (details below). But the branch cannot land as it stands: the head does not compile, the stack it sits on merged somewhere other than main, and main has since landed #558, which rewrote the exact same deadline branch with different semantics and pinned them with tests. Three blockers, then the good parts.

The head does not build

At c5c505961c5bfa22d6bd65a93955c6d76e9541d4, go build ./... from src/merchant (and src/tollwallet) fails:

../tollwallet/gonuts_wallet.go:261:17: w.inner.AcceptMint undefined
(type *TollWallet has no field or method AcceptMint)

Cause: the branch carries cherry-picks of #507's commits, and the reclaimer pick 1e7107feea791c64f4703a386190a745dc8325c0 resolved src/tollwallet/tollwallet.go to the pre-#486 file — it deletes TollWallet.AcceptMint and the mintMu field — while gonuts_wallet.go:259-262 still delegates to the method. The merge base (a99d983a) has the method and origin/main has it, so this branch removes a merged fix and breaks the default (!cdk_wallet) build of two modules. Fix: drop the stale picks (615adbf4, ea21f6ca, 1e7107fe) and rebuild the branch on a base that already contains their content.

Where this PR actually sits

The base branch fix/500-pending-proof-reclaimer is #507's head branch, but #507 merged into deps/gonuts-v0.12.0 (2026-09-22, merge commit 2239211), not into main. So the reclaimer machinery this branch duplicates (PendingBalance/ReclaimPendingProofs on TollWallet, pending_proof_reclaimer.go) is not on main at all, and GitHub's CONFLICTING state here is the cherry-pick criss-cross, not a real content conflict. The PR needs a deliberate retarget: onto deps/gonuts-v0.12.0's successor into main if that chain lands first, or onto main directly once it carries the reclaimer — and then the duplicated commits vanish from the diff (the PR's own delta is a clean 4 files: merchant.go, lightning.go, the new test, CHANGELOG.md).

#558 landed on the same lines after this PR opened

While this branch sat, #558 (3249d0973999937e9129b2b3cfbdc81dda2740cb, merged 2026-09-24) took the same #498 window from the other direction: it is deliberately log-only — recordLateReceiveOutcome consumes the very channel reconcileLateReceive wants, the notice became payment-outcome-unknown with a reference fingerprint and an explicit "Do not send this e-cash note again", and the behavior is pinned by src/merchant/late_receive_notice_test.go and late_receive_outcome_test.go (untagged, so the CI lane runs them). main's deadline comment explicitly reserves granting for the future journal.

That makes the rebase a design merge, not a textual one:

What I verified of the change itself

To separate the stack breakage from the PR's own logic, I grafted #486's AcceptMint back into tollwallet.go at the head (verification shim only, not proposed) and ran: go build, go vet, gofmt clean; go test -race -count=1 -tags testenv . in src/merchant green, 234.6s, matching your claim (224.8s), including both new tests. The grant path is the shared one — grantSessionAccess → AddAllotment extends the allotment under sessionMu with snapshot/restore on gate failure — so a late grant and a subsequent on-time purchase compose instead of clobbering, and the goroutine is race-clean against the sessions map. openGateForSession as a package-level var matches the valve.runNdsctl convention. Test scoping is right (unit, testenv && !cdk_wallet, same tags as merchant_token_flow_test.go), no new dependencies, no token material in the new log lines, and the CHANGELOG.md entry is present and styled correctly.

Two smaller items for the rebase:

  • The late-failure arm only logs (merchant.go:661-664); the inline path also calls mintHealthTracker.MarkUnreachable for non-token-spent failures. Either feed the tracker or drop "same classification the inline path uses" from the comment — a mint that fails at t=35s is exactly the one the tracker should hear about.
  • In purchasesession_late_receive_test.go:71-77, t.Cleanup blocks unconditionally on <-gateCalled; if the grant path regresses, the suite hangs to the 10-minute go-test timeout instead of failing. Make it a select with a deadline.

And one PR-body fix: it says "Fixes #498 (scoped)", but your own changelog commit (d795e865) correctly says "Refs #498 rather than Fixes — the durable-saga criteria remain open in #502". Once retargeted to main, "Fixes" would auto-close a P1 whose durable-record criteria this PR knowingly leaves open. Change the body to Refs #498.

Disposition

request-changes — (1) rebuild the branch so it compiles: drop the stale #507 cherry-picks that revert #486's AcceptMint; (2) retarget onto the branch that actually carries the reclaimer into main; (3) rebase as a deliberate composition with #558: keep the recorder + reference + no-resubmit notice, layer the grant on top, update the operator procedure, and say explicitly why an in-memory grant is the right interim ahead of #502. The reconciliation logic itself is in good shape — this is rebase-and-decide work, not a redesign.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Closing in favor of a fresh rework. The review found this head does not build (the #486 cherry-pick reverted TollWallet.AcceptMint while gonuts_wallet.go still calls it), the branch stacks on fix/500-pending-proof-reclaimer whose content merged into a deps branch rather than main, and #558 has since landed a different design for the same timeout path (record + no-resubmit guidance). A late-Receive reconciliation is still worth building on post-#558 main — the review's design-merge guidance (record and grant, keep the no-resubmit notice, update the operator guide) is the starting point for the rework. The PR's own logic was verified sound under -race with only the AcceptMint shim, so the rework is mostly mechanical on a correct base.

@Amperstrand
Amperstrand deleted the fix/498-late-receive-reconciliation branch October 17, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants