Skip to content

ci(spec): spec-quote drift checking in CI + vacuous-pass fix + drift repair - #357

Closed
Amperstrand wants to merge 1 commit into
OpenTollGate:mainfrom
Amperstrand:spec/wire-speccheck
Closed

Amperstrand wants to merge 1 commit into
OpenTollGate:mainfrom
Amperstrand:spec/wire-speccheck

Conversation

@Amperstrand

Copy link
Copy Markdown
Collaborator

Wires the greatspectations quote set into CI as a drift-blocking check — and fixes two real problems found while doing it.

Found while wiring

  1. The invocation matters critically: --comment-start "//" (no trailing space) matches zero quotes — the marker NUT never follows the comment start, so the check exits 0 having verified nothing. "/ " is required. The make/speccheck.sh carried by open feat: token recovery tool + AI audit prompts + speccheck (rebased onto main) #354 has exactly this bug: it passes vacuously. This PR ships a corrected script (with the trap documented in a comment) and supersedes that copy — worth folding into feat: token recovery tool + AI audit prompts + speccheck (rebased onto main) #354 or dropping theirs.
  2. One genuine spec drift: the NUT-03 swap quote (from chore: add NUT spec quotes (greatspectations Layer 1) #327) no longer matches cashubtc/nuts HEAD — the spec now writes `Proofs` with backticks ("generate new Proofs" → "generate new `Proofs`"). Updated; 9/9 quotes now verify verbatim against current spec.
  3. Two malformed markers aborted whole-repo runs: comment lines starting // NUT-00 ... / // NUT-04 spec... (a dedup pointer from chore: add NUT spec quotes (greatspectations Layer 1) #327 and a doc note from feat(tollwallet): WalletPort interface + GonutsWallet adapter + token flow tests #299) parse as broken quote markers — one syntax error makes the tool return empty results. Rephrased both.

What this adds

  • contract-lint CI step: install greatspectations, clone nuts HEAD, run the check over all non-test sources. Drift fails the build — exit-code semantics verified by test (deliberately drifted quote → exit 1; clean → exit 0).
  • NUT-05 melt quote at GonutsWallet.Melt (the feat(tollwallet): WalletPort interface + GonutsWallet adapter + token flow tests #299 site) — the quote set now covers NUT-00/03/04/05 at their construction sites.
  • make/speccheck.sh (correct version) for local runs: exit 0 always, prints the drift report.

Verification

  • greatspectate check over 56 source files: 9/9 quotes pass, 0 syntax errors against nuts @ 734f60e (current HEAD)
  • Drift-injection test: bad quote → exit 1 (CI would fail); clean → exit 0
  • gofmt clean on touched files; import-paths checker green; tollwallet builds + cross-vector test green
  • CHANGELOG entry under Changed / Internal

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Findings from review + verification:

E2E cloud-lab run for this head is in flight; results will follow. (CI-only change, so the run is a regression formality — the value here is the drift gate itself.)

@felixfelix-bot

Copy link
Copy Markdown
Contributor

Thanks — drift checking in CI is the right call, and the --comment-start "// " vacuous-pass fix is clearly documented (including why the trailing space matters). Four findings:

  1. [BLOCK] make/speccheck.sh:22-23 re-introduces the vacuous pass this PR exists to fix. greatspectate check ... || true followed by unconditional exit 0 means the script can never fail — spec drift, a broken specquotes.toml, a greatspectate CLI/flag change, or a regression of the --comment-start marker all report identical success. The CI step fails the build, but the local entry point (the one run before pushing) is a rubber stamp, and the two will silently diverge the first time the tool errors instead of drifting. Please distinguish outcomes: let tool/config/usage errors exit non-zero (drop || true, or branch on the exit code), and if local drift must stay non-fatal, keep exit 0 only for the "drift found" path with a loud SPEC DRIFT DETECTED banner — not for every failure mode.

  2. [RISK] Unpinned upstream deps make CI non-reproducible and add a supply-chain surface. .github/workflows/test.yml:62 pip-installs git+https://github.com/rustyrussell/greatspectations.git at HEAD, and :64 clones cashubtc/nuts at HEAD. Any upstream change — a spec rewording or a tool CLI change — flips this job red on an untouched commit, and the CI is executing whatever code sits at that repo's HEAD at run time. Consider pinning both to commit SHAs, with a scheduled (daily) bump job that refreshes the pins and opens a PR when upstream moves. You keep the canary (drift caught within a day) but push CI becomes reproducible and a red run comes with a diff to read instead of a debugging session.

  3. [NIT] NUT-05 quote anchored to the wrong function. src/tollwallet/gonuts_wallet.go:160 places the "POST /v1/melt/quote/{method}" quote above Melt() (:161), which executes an existing quote. The quote-request semantics belong on RequestMeltQuote at :156. As-is, the drift check verifies the right quote text against the wrong anchor site, and a future reader gets pointed at the wrong operation.

  4. [NIT] Silenced install output hides root causes. Both test.yml:62 and make/speccheck.sh:11 discard pip stderr (>/dev/null 2>&1). When both install attempts fail (e.g. upstream repo renamed, network blocked), the only symptom is greatspectate: command not found with exit 127 and no hint why. Keep the fallback, but let the second attempt's stderr through, or echo a one-line "primary install failed, retrying without --user" note before falling back.

No E2E run for this PR — CI workflow + source comments only, no web/portal/UI surface touched.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

E2E results: deployed head ee50e34 as ci-pr-357.194.ee50e34 (CI-built ipk) on the OpenWrt x86_64 QEMU lab:

Suite Result
api/test_quote_persistence 4/4 passed
api/test_lightning_backoff 3/3 passed

Regression formality as noted — this PR is CI-only. One incidental: while wiring artifacts for these runs we confirmed the drift gate this PR adds would have caught the current go.mod drift on main (being fixed by #361's cleanup).

…markers

- contract-lint job now installs greatspectations, clones cashubtc/nuts
  HEAD, and runs the check over all non-test sources — DRIFT FAILS THE
  BUILD (exit-code semantics verified: drifted quote exit 1, clean 0)
- critical invocation fix: --comment-start must be "// " (trailing
  space) — with "//" the marker never matches and the check passes
  vacuously. make/speccheck.sh (new on this branch; supersedes the
  version in open OpenTollGate#354 which carries the vacuous-pass bug) documents
  this in a comment
- fixed the one genuinely drifted quote: NUT-03 swap (spec now writes
  `Proofs` with backticks); 9/9 quotes verified against current spec
- added NUT-05 melt-quote at GonutsWallet.Melt (OpenTollGate#299 site)
- repaired two comment lines that parsed as malformed markers and
  aborted whole-repo runs (merchant.go dedup pointer, port.go doc
  comment)
@c03rad0r
c03rad0r force-pushed the spec/wire-speccheck branch from 88eb163 to 5b8efc9 Compare August 30, 2026 13:24

@felixfelix-bot felixfelix-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of current head 5b8efc9b (force-pushed 2026-08-30). The prior review's findings are still live on this head, and the branch is now conflicting with main.

Blocking

  1. make/speccheck.sh:22-23 still re-introduces the vacuous pass this PR exists to fix. greatspectate check ... || true followed by unconditional exit 0 means the local entry point can never fail — spec drift, a broken specquotes.toml, a CLI/flag change, or a regression of the --comment-start marker all report identical success. The CI step fails the build but the local script is a rubber stamp; the two will silently diverge the first time the tool errors instead of drifting. Distinguish outcomes: let tool/config/usage errors exit non-zero (drop || true, or branch on the exit code); if local drift must stay non-fatal, keep exit 0 only for the drift-found path with a loud SPEC DRIFT DETECTED banner.

Conflict (must resolve before merge)

  1. make/speccheck.sh conflicts with main. main merged #354 which added its own make/speccheck.sh (uses spectate, --comment-start '//', plus a spectate coverage step). This PR's version supersedes that copy (correct --comment-start "// "), but the branch needs a rebase/merge to resolve. Note main's version carries the exact vacuous-pass bug this PR fixes — after resolving, keep this PR's corrected invocation and consider folding in the coverage step.

Non-blocking (still live)

  1. [RISK] Unpinned upstream deps — test.yml:62 pip-installs greatspectations at HEAD and :64 clones cashubtc/nuts at HEAD. Any upstream change flips CI red on an untouched commit. Consider pinning to commit SHAs with a scheduled bump job.
  2. [NIT] NUT-05 quote anchored to wrong function — gonuts_wallet.go:160 places the melt-quote-request quote above Melt() (:161, which executes a quote) instead of RequestMeltQuote (:156). Move it to the request site.
  3. [NIT] Silenced install output — both install attempts discard stderr; when both fail the only symptom is greatspectate: command not found. Let the fallback's stderr through.

CI

No CI runs exist for the current head 5b8efc9b (last branch run was on 88eb1638, 2026-08-23). After resolving the conflict, a fresh push is needed to get CI on the final head.

Cannot merge in this state (conflicting + blocking finding unaddressed).

c03rad0r pushed a commit that referenced this pull request Sep 7, 2026
… fixes) (#376)

* fix(spec): fix drifted NUT-03 quote, malformed markers, NUT-05 comment placement

- NUT-03 swap quote: 'generate new Proofs' -> 'generate new `Proofs`'
  (spec added backticks; the one genuinely drifted quote found by review)
- NUT-05 melt quote added at GonutsWallet.RequestMeltQuote (per reviewer
  feedback: above the request-method, not above Melt())
- merchant.go: reword 'NUT-00' marker prefix to 'Spec (NUT 00)' so it
  doesn't parse as a malformed quote marker
- port.go: reword 'NUT-04 spec' to 'spec (NUT 04)' for the same reason
- CHANGELOG: note spec-quote drift checking entry superseding #357
- .gitignore: add *.cov to exclude greatspectate coverage artifacts

* ci(spec): wire spec-quote drift check with vacuous-pass fix + coverage step

- Contract-lint job: new 'Spec-quote drift check' step that installs
  greatspectations, clones cashubtc/nuts, and verifies all NUT spec
  quotes in source comments against spec — drift fails the build.
- make/speccheck.sh: supports both local drift-report mode and CI
  fail mode. Vacuous-pass fix: tool/config/usage errors (exit 2+)
  propagate non-zero; exit 0 only for clean runs or drift-only
  (exit 1, loud banner). Coverage step folded in from main's #354
  but corrected for the real greatspectate CLI (--coverage=FILE).
- Correct command name: greatspectations installs 'greatspectate'
  as its only console script (no 'spectate'). Both CI and local
  scripts use the correct name.
- Correct --comment-start '// ' (trailing space) — without the
  trailing space the checker passes vacuously (matches nothing).
- Pinned deps: greatspectations at SHA 0f226495 with scheduled-bump
  comment; cashubtc/nuts at SHA 49a909c in CI for reproducibility.
- No silenced pip output (install logs visible).

---------

Co-authored-by: Felix <301398501+felixfelix-bot@users.noreply.github.com>
@c03rad0r
c03rad0r self-requested a review September 7, 2026 22:36
@felixfelix-bot

Copy link
Copy Markdown
Contributor

Status: superseded by #376 — no merge needed, conflicts are now a no-op.

I investigated the conflicts on this PR (.github/workflows/test.yml, CHANGELOG.md, make/speccheck.sh). Root cause: #376 already merged the reviewed, fixed copy of this exact work into main (commit 3fa8336, "ci(spec): spec-quote drift checking in CI (copy of #357 with reviewer fixes)"), so this branch is now fully redundant.

What #376 carried forward (all reviewer-corrected):

  • vacuous-pass fix in make/speccheck.sh (correct // comment-start; tool/config errors propagate, drift-only exits 0 with banner)
  • pinned greatspectations ref + pinned cashubtc/nuts snapshot (reproducible CI)
  • NUT-03 drift repair (backticked Proofs)
  • NUT-05 comment placed at the correct site (RequestMeltQuote)
  • merchant.go/port.go malformed-marker repairs

After resolving all three conflicts against current main, the ONLY remaining delta from this branch is a duplicate NUT-05 quote at GonutsWallet.Melt — main already carries the fixed, correctly-placed version at RequestMeltQuote (corrected by reviewer feedback in #376). Merging would introduce that duplicate and break the very drift check this PR adds.

Recommendation: close this PR as superseded by #376. Nothing is lost — the branch is preserved and all unique content landed (in corrected form) via #376.

cc @Amperstrand — thanks for the original; your contribution is in, just via the reviewed copy.

@Amperstrand

Copy link
Copy Markdown
Collaborator Author

Marking this superseded by #376 (clean copy with reviewer fixes, merged as 3fa8a2/3fa8336) — thanks for carrying it across the finish line. For the record: this branch was also rebased onto post-#369 main locally and validated (build+vet), but #376 includes equivalent changes plus the speccheck coverage merge, so nothing further is needed from this branch. Maintainer feel free to close, or we will.

@felixfelix-bot

Copy link
Copy Markdown
Contributor

Closing as superseded by #376 (already merged to main as 3fa8336).

Every change in this PR already landed, in corrected form, via #376:

  • spec-quote drift check step in CI (pinned greatspectations ref + pinned cashubtc/nuts snapshot)
  • make/speccheck.sh 'vacuous-pass' fix (correct // comment-start)
  • NUT-03 drift repair (Proofs backticks) — src/tollwallet/tollwallet.go, on main
  • NUT-05 quote — placed at RequestMeltQuote (reviewer-corrected site; this PR's version put it at Melt)
  • merchant.go + port.go malformed-marker repairs — on main

The two remaining conflicts vs current main (test.yml, CHANGELOG.md, make/speccheck.sh) are all cases where main's #376 version is the reviewed+improved one. Merging this PR would regress those to the unpinned originals and re-add a misplaced duplicate NUT-05 quote. Nothing is lost — branch preserved at 5b8efc9.

Thanks @Amperstrand — the contribution is in via the reviewed copy #376.

@c03rad0r c03rad0r closed this Sep 8, 2026
felixfelix-bot added a commit to felixfelix-bot/tollgate-module-basic-go that referenced this pull request Sep 20, 2026
…ith reviewer fixes) (OpenTollGate#376)

* fix(spec): fix drifted NUT-03 quote, malformed markers, NUT-05 comment placement

- NUT-03 swap quote: 'generate new Proofs' -> 'generate new `Proofs`'
  (spec added backticks; the one genuinely drifted quote found by review)
- NUT-05 melt quote added at GonutsWallet.RequestMeltQuote (per reviewer
  feedback: above the request-method, not above Melt())
- merchant.go: reword 'NUT-00' marker prefix to 'Spec (NUT 00)' so it
  doesn't parse as a malformed quote marker
- port.go: reword 'NUT-04 spec' to 'spec (NUT 04)' for the same reason
- CHANGELOG: note spec-quote drift checking entry superseding OpenTollGate#357
- .gitignore: add *.cov to exclude greatspectate coverage artifacts

* ci(spec): wire spec-quote drift check with vacuous-pass fix + coverage step

- Contract-lint job: new 'Spec-quote drift check' step that installs
  greatspectations, clones cashubtc/nuts, and verifies all NUT spec
  quotes in source comments against spec — drift fails the build.
- make/speccheck.sh: supports both local drift-report mode and CI
  fail mode. Vacuous-pass fix: tool/config/usage errors (exit 2+)
  propagate non-zero; exit 0 only for clean runs or drift-only
  (exit 1, loud banner). Coverage step folded in from main's OpenTollGate#354
  but corrected for the real greatspectate CLI (--coverage=FILE).
- Correct command name: greatspectations installs 'greatspectate'
  as its only console script (no 'spectate'). Both CI and local
  scripts use the correct name.
- Correct --comment-start '// ' (trailing space) — without the
  trailing space the checker passes vacuously (matches nothing).
- Pinned deps: greatspectations at SHA 0f226495 with scheduled-bump
  comment; cashubtc/nuts at SHA 49a909c in CI for reproducibility.
- No silenced pip output (install logs visible).

---------

Co-authored-by: Felix <301398501+felixfelix-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants