Real findings from running Nirium in production. No marketing copy here, just what broke, how we found it, and where it stands.
What we found. On 5 August 2026, our mainnet x402 endpoint returned HTTP 402 with an empty body to a payment retry, meaning we told the caller their payment was rejected. About 30 seconds later, that same payment settled on-chain anyway, from a different account in the facilitator's relayer pool than the one that settled the caller's second, separate attempt. Net effect: the caller was charged for a request we had already told them failed, and received nothing for that charge.
How we diagnosed it. The report came from a third-party integrator, AgentLedger (agentpayments.fi), testing our x402 endpoint from a real client. When we investigated, they independently tested our endpoint alongside two unrelated x402 sellers on Stellar pubnet and found the same failure signature across all three: 402 with an empty payment-required challenge and zero settlement. Reproducible with a plain curl request against a rejected or missing payment, no signed transaction required. That cross-vendor consistency is what pointed at the facilitator layer (we use @x402/express's paymentMiddlewareFromConfig against OpenZeppelin Channels, and never call verify or settle ourselves) rather than at any single integration, including ours.
We filed the issue against the upstream SDK: x402-foundation/x402#3148
Update, 14 August 2026: we had the header wrong. Our original report included "no payment-error header despite it being advertised in access-control-expose-headers" as part of the evidence. A community reviewer, whawk46, traced the SDK source across TypeScript, Python, and Go and found that payment-error is not a header any x402 SDK emits at all; it's the internal result.type process discriminator. We verified that independently in the installed @x402/express@2.17.0 dist rather than take it on faith, confirmed it, and fixed our own CORS exposedHeaders list, which had hardcoded payment-error/payment-verified as if they were real header names. Credited in the issue thread. The header confusion is resolved; the underlying race (rejected payment settling anyway) is real and stays open, unrelated to the header.
Where it stands today. We cannot fix a race condition inside a facilitator we do not operate. What we built instead is GET /api/reporting/reconcile, which compares on-chain settlements to the treasury against payment receipts our own route handlers logged. Read honestly: it currently reports matchable: false for most of its history, because our receipts did not record the settlement transaction hash until we built this endpoint, so most existing receipts cannot be matched to a specific on-chain payment one to one. It compares totals in the meantime and will match individually going forward as receipts carry the hash. That gap is not fixed yet. It is the next thing to close.
Production patterns proposed back to the Stellar agentic-payments skill. Kaan Kaçar had closed an earlier PR of ours against the same skill (stellar/stellar-dev-skill#14) because the files it targeted had been restructured out from under it, and invited a fresh, smaller PR against whichever of those patterns were still missing. Re-verified each one against what's actually running in production rather than copy the old diff, and opened stellar/stellar-dev-skill#97 with four: multi-route pricing, recipient resolution that fails open instead of crashing, an optional dual-intent server, and a discovery endpoint. Open, awaiting review.
@x402/core leaks a raw internal error on a malformed payment payload. paymentRequirementsMatchAccepted(), called from every @x402/express resource server on every incoming payment, destructures the client-supplied accepted field without checking it exists first. Send a payload that omits accepted (or sets it to null) and the 402 comes back with a raw V8 TypeError in the error field instead of a protocol-level reason. Confirmed live against our own production endpoint, and confirmed still present in the latest published release. Filed: x402-foundation/x402#3171.
Update, 17 August 2026: fixed, by someone else. JasonColapietro, unaffiliated with this team, opened PR #3180 treating a missing/null accepted as no match instead of letting it throw, with end-to-end regression coverage for it. Maintainer phdargen reviewed and approved it — "Thanks for the patch @JasonColapietro, LGTM 🚀" — before merging at commit ab1a31a, 55/55 tests passing. GitHub closed the issue automatically two seconds later. Closed, fixed upstream.
OpenZeppelin's fee-abstraction module: an expiration check that checks the wrong thing. Read closely because it's the module backing our facilitator's fee-sponsor mechanism. In Lazy approval mode, when an existing token allowance is already large enough, the crate validates that the caller's newly passed expiration_ledger parameter is in the future, but that value is never actually applied to the real, on-chain allowance, which keeps whatever expiration a prior approve() call set. The crate's own test suite demonstrates the resulting false rejection: a fee collection against a genuinely still-valid allowance panics anyway, purely because of the unrelated parameter. Filed: OpenZeppelin/stellar-contracts#840. Open, awaiting response.
Nirium is listed in Stellar's official developer skills catalog (skills.stellar.org), reviewed and merged by SDF DevRel: stellar/stellar-dev-skill#96. Skill source: skills/nirium-agentic-payments/SKILL.md.
A real CORS bug, found by external QA, fixed same day. External QA testing our x402 endpoint reported that any Origin outside our allowlist returned 500 — including on preflight OPTIONS, which killed the paid request before the browser ever attempted it. Reproduced live before touching anything: /health returned 200 for a disallowed Origin, the payment route returned 500. Traced the difference to routing, not to the payment route being special: /health is answered by our MASTER proxy before the request reaches the Express app; the payment route sits behind our CORS middleware, whose origin callback called callback(new Error(...)) on rejection. The installed cors@2.8.6 propagates that as an uncaught error via next(err) for any route behind it, preflight or not, and with no dedicated error handler it falls through to Express's generic 500. Fixed to callback(null, false) — the package then omits CORS headers without throwing, letting the real request continue normally (the browser correctly blocks reading the response) and the preflight respond cleanly instead of crashing. Re-verified live on both our testnet and mainnet boxes after deploying: disallowed Origin now returns 200/501 (route-appropriate, not a crash) on preflight and the real route's own status on the actual request; an allowed Origin still gets a clean 204 with full CORS headers, unchanged.
Corroborated the reasoning-code DX gap on #3148 with a real case. x402-foundation/x402#3148 — the same finding this devlog covers in the 13 August entry — had settled into a residual ask: rejection reasons live only inside a base64-encoded PAYMENT-REQUIRED field, so curl-level triage requires decoding first. We hit exactly that ourselves while diagnosing the CORS bug above and a separate facilitator-sponsor outage the same week, and added a real case: during the sponsor outage, the envelope came back as a literal empty {} — a fourth distinguishable failure mode (facilitator itself down, not the payment) beyond what was already documented. Comment.
Filed a question to OpenZeppelin, not a fix — status: open, no response yet. Our facilitator sponsor account (GA5SXMFJ...) went silent for 205+ hours; a different account (GD6ZLGJJ...) appeared active on pubnet 3 days later with a matching activity pattern, but we have no way to confirm ownership from the outside. Filed OpenZeppelin/relayer-plugin-x402-facilitator#47 as an open question, not a claim: if the new account is theirs, it's a config fix on our side (we're pointed at a retired account); if it's not, the outage is broader than just us. Open, awaiting OpenZeppelin's response.
Confirmed root cause independently before touching either issue — same discipline as the original reports, applied backward. Before writing a fix for #3148 or #840, re-verified both from scratch against live sources instead of trusting our own prior write-ups. That split them into two different situations.
#3148: there was no code bug left to fix. Re-read the full thread live. We had already retracted the original finding on 14 August — the repro was broken, not the SDK, and the mechanism to distinguish a rejected payment from an unpaid one already exists (it's just base64-encoded inside PAYMENT-REQUIRED). What's actually open is a DX request for a stable reason-code vocabulary, which is a protocol design decision for the x402-foundation maintainers, not something we can code and PR unilaterally. Posted a concrete starting proposal instead of another PR — a five-value enum, with the two design constraints the original reviewer already raised (stable codes, not raw internals; organized around retry semantics, not exhaustiveness) — explicit that a PR only gets written if there's rough agreement on the shape. Comment. Open, no PR intended yet.
#840: root cause confirmed twice, independently, before writing the fix. First ran the crate's own cited test against main at the exact commit the issue names (fbfde38) — it panics with Error(Contract, #5006), and the diagnostic log shows the real allowance was 100, genuinely sufficient and non-expired, at the moment of the false rejection. Then wrote two new, isolated tests — not just re-running theirs — to check both directions the issue describes separately: a real allowance valid for 99 more ledgers still gets rejected on an unrelated parameter, and an arbitrary expiration value nowhere near the real allowance's actual expiration gets accepted just as readily. Both held up against real, compiled, run code. Only then wrote the fix: OpenZeppelin/stellar-contracts#844, implementing option (a) from the issue's own suggested directions (drop the check — the SAC's own transfer_from expiry enforcement already covers the real allowance) over option (b) (real tightening via a conditional approve()), with the fork documented explicitly in the PR description in case the maintainers want the other one instead. 22/22 tests passing, cargo clippy -D warnings clean, cargo +nightly fmt --check clean. Open, awaiting review.
Both outcomes came from the same rule: confirm root cause with real, run evidence first, and let what you find decide whether there's a fix to write at all.
Independent corroboration of the facilitator sponsor outage — not from us this time. A comment on our own OZ#47 issue — another x402 buyer on pubnet, credited here only by link, matching how it framed itself. Verified live before quoting: four of their mainnet settlements on 5–6 August had fees paid by the exact sponsor account we flagged (GA5SXMFJ...), each fee_account checked against Horizon — the account wasn't just active, it was sponsoring settled payments for a second, unrelated buyer on this same path. Two more facts stated as independently verifiable, not asked to be trusted: their own daily canary against a different pubnet seller (different facilitator) has settled first-attempt every day since 18 August — scoping the problem to this sponsor/config, not to pubnet or the exact scheme generally — and a second sponsor account has been active since 13 August, three days after the flagged one went quiet, the same three-day gap we'd already noted. Silence duration as of the comment: 234 hours. Closed with a concrete ask: document which sponsor account is current, to save the next integrator the same detour. Still open, still awaiting OpenZeppelin's own response — this is corroboration, not resolution.
The CORS fix (13 August entry) holds — confirmed independently, on both mainnet and testnet, by the same third party.
The #3148 vocabulary proposal now has five real scenarios behind it, not two. Follow-up comment, grounding the enum shape proposed the day before in measurements from another integrator running x402 against multiple pubnet sellers in production — credited without a name, no public link exists for the raw data. Three of the five we'd already confirmed ourselves (No matching payment requirements, invalid_exact_stellar_payload_malformed, the empty-{} facilitator-down case); two are new. The new ones matter more than the old ones: an undecodable payment header lands in the same branch as no payment at all, which is the mechanism behind why our original #3148 repro proved nothing — and, sharper, a JSON payload missing x402Version hits an uncaught throw in findMatchingRequirements, so whether a caller catches it decides whether the client sees a 402 with a reason or a bare 500 — two different pubnet sellers on the same protocol version producing two different HTTP status codes for identical malformed input. That's not a documentation gap, it's proof the DX ask isn't hypothetical.
#97 and #103, status update. stellar/stellar-dev-skill#97 (the 15 August entry above) had picked up a real merge conflict against upstream/main, which had independently restructured the same file this PR touches. Resolved by rebase, keeping both sides' additions rather than picking one — gh pr view now reports mergeStateStatus: CLEAN. #103, a separate small fix (27 of 28 ECOSYSTEM_CARDS entries pointed at a GitHub blob HTML page instead of raw content, so llms.txt was shipping a rendered-HTML URL to any agent that fetched it), is also open and mergeable, no dependency on #97. Both flagged to the reviewer in one comment. Awaiting a maintainer review on all three PRs in this entry — nothing left on our side.
x402#3169, closed by the maintainer. The double percent-encoding bypass in isValidRouteTemplate() we filed 15 August (/foo/%252e%252e%252fsecret surviving one decode, then revealing itself as /foo/../secret on the second) was fixed upstream and the issue closed as completed by phdargen, via PR #3213 and #3422. Nothing left on our side.
relayer-plugin-x402-facilitator#47, a response after a month. Our mainnet x402 facilitator sponsor account (GA5SXMFJ...) went silent in early August; by early September it was three independent integrators (us, AgentPayments.fi, NovaCorpAI) converging on the same finding across two official OpenZeppelin channels (this issue and their support forum, filed 13 August, still unpublished in moderation weeks later) with zero response from either. Verified the repo's real commit history before escalating further: no human commit since March 2026, so the silence read as nobody-watching, not deliberate. Tagged the two maintainers who were actually active elsewhere in the OpenZeppelin org that week — collins-w (pushing to the Channels infra repo) and zeljkoX (active on a sibling repo) — by name, directly, having verified their activity via the GitHub API first rather than guessing. zeljkoX replied within a day: "Apologies for the slow response and thanks for the direct tag. Thanks for the detailed report, looking into it now and will update here shortly." The account is still dead as of this entry — a response is engagement, not a fix — but the month of silence is over. Direct, verified tags to maintainers who are demonstrably active elsewhere beat a thread or a forum post that nobody happens to be watching.
Two Stellar mainnet SAC constants that were never valid addresses. While wiring MPP's mainnet USDC handling, @stellar/mpp's exported USDC_SAC_MAINNET (54 characters) and XLM_SAC_MAINNET (55 characters, not a truncation of the real 56-character address) both failed StrKey.isValidEd25519PublicKey() — testnet-only bug, so it had been shipping unnoticed in every version including the then-latest 0.7.1. We'd already worked around it on our side with a manual override (USDC_SAC_OVERRIDE in middleware/mpp.ts) rather than trust the package's own constant. Filed stellar/stellar-mpp-sdk#68 with the exact byte counts and the correct addresses, then opened PR #69 fixing both. Merged the same day. The workaround stays in our own code until a release actually ships the fix — a merge is not a release — but the upstream constant itself is no longer wrong.