From 21544de8b905680a961be4763177b2bbb6f87e71 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 28 May 2026 19:02:43 +0000 Subject: [PATCH 1/3] fix(wheelstest): capture spec-load errors against the bundle and emit positive error count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec files whose run() called it() outside a describe() block, or whose beforeAll() threw during load, were silently dropped (the run bubbled up as an anonymous BundleRunnerMajorException) or surfaced as the legacy "-1 error(s)" sentinel in the summary, with no file context. BDDRunner now starts bundle stats before invoking the target and wraps both the suite-descriptor pass (target.run / getTestSuites) and the bundle execution (beforeAll, suite iteration, afterAll) in a single catch that records the throw against bundleStats.globalException and increments the error count by one. Tooling that walks bundleStats.path / globalException already surfaces the filename and exception detail. Fixes the silent-failure reports in issue #2829 for both shapes — a malformed orphan-it() spec and a load-time exception from a setup hook. Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> --- CHANGELOG.md | 1 + .../wheelstest/BeforeAllErrorFixture.cfc | 18 +++++ .../resources/wheelstest/OrphanItFixture.cfc | 9 +++ .../BDDRunnerErrorReportingSpec.cfc | 51 ++++++++++++ .../wheelstest/system/runners/BDDRunner.cfc | 80 +++++++++++-------- 5 files changed, 124 insertions(+), 35 deletions(-) create mode 100644 vendor/wheels/tests/resources/wheelstest/BeforeAllErrorFixture.cfc create mode 100644 vendor/wheels/tests/resources/wheelstest/OrphanItFixture.cfc create mode 100644 vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc diff --git a/CHANGELOG.md b/CHANGELOG.md index 42b59a9bc9..cc35dc131a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ All historical references to "CFWheels" in this changelog have been preserved fo - The Debian/Ubuntu `apt` install instructions now pipe the distribution key through `sudo gpg --dearmor` before writing `/usr/share/keyrings/wheels.gpg` instead of `tee`-ing it verbatim. The key published at `apt.wheels.dev/wheels.gpg` is ASCII-armored, and modern `apt` rejects an armored key in a `signed-by=` keyring with an "unsupported filetype" warning followed by `NO_PUBKEY` — so `apt update` failed signature verification and the install never worked. Corrected across the install guide, the CLI installation reference, the release-channels guide, the `apt.wheels.dev` landing page, and the `tools/distribution-drafts/` repo templates (#2838) - The `apt.wheels.dev` publishing template (`tools/distribution-drafts/apt-repo/`) no longer wipes the `stable` package index when a `bleeding-edge` snapshot publishes. `regenerate-apt-metadata.sh` rebuilt *both* channels on every run while the workflow synced only the dispatched channel's pool into the runner, so a frequent bleeding-edge publish scanned an empty local `pool/stable/`, produced an empty `Packages`, and the unscoped upload overwrote the good stable index on R2 — leaving `apt install wheels` with "Unable to locate package wheels" even though the `.deb` was present in the pool. The regen now honors a `CHANNELS` env (the workflow passes only the dispatched channel) and the upload is scoped to that channel's `dists/` subtree, so the two channels can no longer clobber each other (#2838) - The Wheels CLI test suite (`cli/lucli/tests/specs`, served at `/wheels/cli/tests`) is green again after the BDDRunner error-count fix unmasked 13 pre-existing failures the old `-1` bundle-error sentinel had been arithmetically cancelling (a negative error total netted real failures down to `<= 0`, so the CI gate read the suite as passing). The eight `*CommandSpec` bundles that instantiate `new cli.lucli.Module()` no longer fail to load with `can't find component [modules.BaseModule]`: a lightweight `BaseModule` test double under `cli/lucli/tests/_modules/` plus a `/modules` mapping (added alongside the existing `/modules/wheels`, which longest-prefix resolution keeps authoritative for the wheels module) lets `Module.cfc` instantiate under TestBox — resurrecting the Db/Info command specs as real behavioral coverage. The stale `AdminSpec` route assertion now expects `.namespace("admin")` (the service's current named-route-prefixed output) instead of the legacy `.scope(path="admin")`. Command specs that need the LuCLI runtime, a running Wheels server, CodeGen harness fixtures, or the CLI bash wrapper (Deploy/Destroy/Generate/Packages, plus the server-dependent Migrate/Test cases) and the unbuilt-feature specs (Doctor #2260 mixin-detail, Scaffold route-model-binding) are `xdescribe`/`xit`-skipped with documented reasons, pending a command-by-command CLI test audit. Finally, `tools/ci/run-tests.sh` now clamps a negative error count for its pass/fail decision and fails explicitly when it sees one, so this masking class of bug can never silently turn a red suite green again (#2829) +- WheelsTest BDD runner now captures spec-load and bundle-execution errors against the offending bundle instead of bubbling out as an anonymous `BundleRunnerMajorException`, and reports the resulting error count as a positive number (was the `-1` sentinel) so summaries read "1 error(s)" with the bundle path and `globalException` populated — covers both `it()` called outside a `describe()` body and a `beforeAll()` that throws during spec load (#2829) ---- diff --git a/vendor/wheels/tests/resources/wheelstest/BeforeAllErrorFixture.cfc b/vendor/wheels/tests/resources/wheelstest/BeforeAllErrorFixture.cfc new file mode 100644 index 0000000000..8b5e349270 --- /dev/null +++ b/vendor/wheels/tests/resources/wheelstest/BeforeAllErrorFixture.cfc @@ -0,0 +1,18 @@ +component extends="wheels.WheelsTest" { + + function beforeAll() { + throw( + type = "Test.Setup.MissingDep", + message = "application.wo.functionDoesNotExist is undefined" + ); + } + + function run() { + describe("foo", function() { + it("bar", function() { + expect(true).toBeTrue(); + }); + }); + } + +} diff --git a/vendor/wheels/tests/resources/wheelstest/OrphanItFixture.cfc b/vendor/wheels/tests/resources/wheelstest/OrphanItFixture.cfc new file mode 100644 index 0000000000..2e51082251 --- /dev/null +++ b/vendor/wheels/tests/resources/wheelstest/OrphanItFixture.cfc @@ -0,0 +1,9 @@ +component extends="wheels.WheelsTest" { + + function run() { + it("orphan", function() { + expect(true).toBeTrue(); + }); + } + +} diff --git a/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc b/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc new file mode 100644 index 0000000000..6cf5ddc594 --- /dev/null +++ b/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc @@ -0,0 +1,51 @@ +component extends="wheels.WheelsTest" { + + function run() { + + describe("BDDRunner load/run-time error reporting", function() { + + it("captures an it() call outside describe() against the bundle instead of bubbling out", function() { + // Reproduces issue #2829, case 1: a spec whose run() calls + // it() at the top level (no enclosing describe) currently + // bubbles up as a BundleRunnerMajorException, leaving the + // CLI summary at "0 passed" with no filename or message. + var testBox = new wheels.wheelstest.system.TestBox( + bundles = ["wheels.tests.resources.wheelstest.OrphanItFixture"] + ); + var state = {threw: false, results: ""}; + try { + state.results = testBox.runRaw(); + } catch (any e) { + state.threw = true; + } + expect(state.threw).toBeFalse(); + expect(isObject(state.results)).toBeTrue(); + expect(state.results.getTotalError()).toBe(1); + var bs = state.results.getBundleStats(); + expect(arrayLen(bs)).toBeGT(0); + expect(bs[1].totalError).toBe(1); + expect(bs[1].path).toInclude("OrphanItFixture"); + expect(isStruct(bs[1].globalException) || isObject(bs[1].globalException)).toBeTrue(); + }); + + it("uses a positive error count when beforeAll() throws during spec load", function() { + // Reproduces issue #2829, case 2: a spec whose beforeAll() + // throws records totalError = -1, which sums into the global + // count as "-1 error(s)" with no file context. + var testBox = new wheels.wheelstest.system.TestBox( + bundles = ["wheels.tests.resources.wheelstest.BeforeAllErrorFixture"] + ); + var results = testBox.runRaw(); + expect(results.getTotalError()).toBeGT(0); + var bs = results.getBundleStats(); + expect(arrayLen(bs)).toBeGT(0); + expect(bs[1].totalError).toBeGT(0); + expect(bs[1].path).toInclude("BeforeAllErrorFixture"); + expect(isStruct(bs[1].globalException) || isObject(bs[1].globalException)).toBeTrue(); + }); + + }); + + } + +} diff --git a/vendor/wheels/wheelstest/system/runners/BDDRunner.cfc b/vendor/wheels/wheelstest/system/runners/BDDRunner.cfc index d98c9218ec..70d80270ea 100755 --- a/vendor/wheels/wheelstest/system/runners/BDDRunner.cfc +++ b/vendor/wheels/wheelstest/system/runners/BDDRunner.cfc @@ -47,25 +47,33 @@ component structKeyExists( tartetAnnotations, "displayName" ) ? tartetAnnotations.displayname : targetMD.name ); - // Execute the suite descriptors - arguments.target.run( testResults = arguments.testResults, testbox = variables.testbox ); - - // Discover the test suite data to use for testing - var testSuites = getTestSuites( arguments.target, targetMD ); - var testSuitesCount = arrayLen( testSuites ); - - // Start recording stats for this bundle + // Start recording stats for this bundle BEFORE invoking the target, + // so a throw from `target.run()` (e.g. `it()` called outside a + // `describe()` body — see issue #2829) can be recorded against this + // bundle instead of bubbling out anonymously as a + // `BundleRunnerMajorException` with no file context. var bundleStats = arguments.testResults.startBundleStats( bundlePath = targetMD.name, name = bundleName ); - // Verify we can run this bundle - if ( - canRunBundle( - bundlePath = targetMD.name, - testResults = arguments.testResults, - targetMD = targetMD - ) - ) { - try { + // Wrap the suite-descriptor pass and bundle execution in a single + // try so both load-time errors (target.run, getTestSuites) and + // run-time errors (beforeAll/afterAll, suite iteration) land in the + // same catch and report the offending bundle. + try { + // Execute the suite descriptors + arguments.target.run( testResults = arguments.testResults, testbox = variables.testbox ); + + // Discover the test suite data to use for testing + var testSuites = getTestSuites( arguments.target, targetMD ); + var testSuitesCount = arrayLen( testSuites ); + + // Verify we can run this bundle + if ( + canRunBundle( + bundlePath = targetMD.name, + testResults = arguments.testResults, + targetMD = targetMD + ) + ) { // execute beforeAll() for this bundle, no matter how many suites they have. if ( structKeyExists( arguments.target, "beforeAll" ) ) { arguments.target.beforeAll(); @@ -142,26 +150,28 @@ component for ( var afterAllMethod in afterAllAnnotationMethods ) { invoke( arguments.target, "#afterAllMethod.name#" ); } - } catch ( Any e ) { - bundleStats.globalException = e; - // For a righteous man falls seven times, and rises (tests) again :) - // The amount doesn't matter, nothing can run at this point, failure with before/after aspects that need fixing - bundleStats.totalError = -1; - arguments.testResults.incrementStat( type = "error", count = bundleStats.totalError ); - - // Module call backs - variables.testbox.announceToModules( - "onSuiteError", - [ - e, - arguments.target, - arguments.testResults, - isNull( thisSuite ) ? {} : thisSuite - ] - ); } + // end if we can run bundle + } catch ( Any e ) { + bundleStats.globalException = e; + // Use a positive count so the summary reports "1 error(s)" rather + // than the legacy "-1 error(s)" sentinel — issue #2829. The bundle's + // globalException still carries the underlying throw for tooling + // that wants the full detail. + bundleStats.totalError = 1; + arguments.testResults.incrementStat( type = "error", count = 1 ); + + // Module call backs + variables.testbox.announceToModules( + "onSuiteError", + [ + e, + arguments.target, + arguments.testResults, + isNull( thisSuite ) ? {} : thisSuite + ] + ); } - // end if we can run bundle // finalize the bundle stats arguments.testResults.endStats( bundleStats ); From 0f5188ce965bcd8b190ae62cfbd616cdd5dedc16 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 28 May 2026 19:05:50 +0000 Subject: [PATCH 2/3] docs(web/guides): note BDD runner load-time errors in common failure modes Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> --- .../src/content/docs/v4-0-0/testing/running-tests-locally.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/web/sites/guides/src/content/docs/v4-0-0/testing/running-tests-locally.mdx b/web/sites/guides/src/content/docs/v4-0-0/testing/running-tests-locally.mdx index ab6ce0f783..7397994cc3 100644 --- a/web/sites/guides/src/content/docs/v4-0-0/testing/running-tests-locally.mdx +++ b/web/sites/guides/src/content/docs/v4-0-0/testing/running-tests-locally.mdx @@ -202,6 +202,7 @@ These are the errors you'll actually hit. Fix-ups first, diagnosis second. - **Populate SQL errors** — read the error. "Table already exists" means a stale table from a previous engine-specific run; add a `DROP TABLE IF EXISTS` first. "Syntax error near CURRENT_TIMESTAMP" means cross-engine SQL drift; use `NOW()` — it works on every supported engine. - **Flaky `wheels new` in parallel harness runs** — a known LuCLI race (framework gap tracker #11). Retry once, or serialize the runs until the fix lands. - **Server not responding after `compose up`** — engines need 30–90 seconds to cold-start. `curl -I http://localhost:60007/` returns connection-refused until the engine is ready. If it stays down after two minutes, check `docker compose logs `. +- **"1 error(s)" with a bundle path but no failing assertion** — the BDD runner caught a load-time or setup error for that spec file. Two common causes: (1) `it()` called directly inside `run()` without an enclosing `describe()` block, or (2) `beforeAll()` throwing an exception. The runner captures both against the bundle and populates `globalException` — check the spec file at the path shown in the summary for the underlying throw. ## Cleanup From c22c2cfa7345f0c33805d4e4c46bdb16d068e375 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Thu, 28 May 2026 19:19:36 +0000 Subject: [PATCH 3/3] fix(wheelstest): tighten BeforeAllErrorFixture assertions to toBe(1) Address Reviewer A/B consensus from PR #2831 round 1: both reviewers agreed the BeforeAllErrorFixture case in vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc should match the OrphanItFixture case's exact-count style. The catch block in BDDRunner.cfc unconditionally records totalError = 1 per failed bundle, so toBeGT(0) was weaker than the invariant the fix guarantees. - vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc lines 39 and 42: toBeGT(0) -> toBe(1) for results.getTotalError() and bs[1].totalError in the BeforeAllErrorFixture case. Verified: wheelstest layer 143 pass / 0 fail / 0 error against the running CI server (sqlite). Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> --- .../tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc b/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc index 6cf5ddc594..bf2970e0a5 100644 --- a/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc +++ b/vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc @@ -36,10 +36,10 @@ component extends="wheels.WheelsTest" { bundles = ["wheels.tests.resources.wheelstest.BeforeAllErrorFixture"] ); var results = testBox.runRaw(); - expect(results.getTotalError()).toBeGT(0); + expect(results.getTotalError()).toBe(1); var bs = results.getBundleStats(); expect(arrayLen(bs)).toBeGT(0); - expect(bs[1].totalError).toBeGT(0); + expect(bs[1].totalError).toBe(1); expect(bs[1].path).toInclude("BeforeAllErrorFixture"); expect(isStruct(bs[1].globalException) || isObject(bs[1].globalException)).toBeTrue(); });