Deployment activation and certification tests fail in full-suite runs while comparison cases pass in isolation #3293

Observed

On macOS, the complete main unit suite fails deployment activation and certification cases that pass when run together in isolation.

Observed on 19 October 2026 with Apple M4 Pro, Node 24.16.0 and npm 11.21.1. The matched full-suite runs used a canonical macOS temporary-directory path:

• The unchanged baseline commit and the branch in Add full-text and autocomplete snapshot continuation, implementing Support full-text and autocomplete continuation in Table.search and REST both have reported failures in the four transient-rename cases in unitTests/components/deployActivation.test.js during the complete main run.
• Both full-suite runs have also reported failures in unitTests/components/deployCertification.test.js, including cleanup hooks. The particular certification cases differ between runs.
• A comparison subset of seven cases (the four rename tests and three certification tests) passes together in isolation on both baseline and feature: seven passing in about 10 seconds on each. This subset does not cover every failure in the full runs.
• An earlier completed feature run (before canonicalizing the temporary path) reported no-retry assertions in two rename cases and certification timeouts of 30 seconds. That earlier run is separate evidence; it is not the four-case count above. Neither observation establishes a production deployment defect.

Completed full-gate results: feature 7,111 passing / 218 pending / 17 failing in 44 minutes; baseline 7,108 passing / 218 pending / 17 failing in 27 minutes. Some failed cases differ, so these totals do not establish that every feature-side failure is pre-existing.

A further full-suite symptom needs isolation: selected ordinary SQL cases became very slow late in the feature run (about 69–283 seconds), while the two selected alias queries passed in fresh processes on both branches. The first selected query took 48 ms on baseline and 49 ms on feature. The reductions below establish shared watcher interference for one activation failure and two SQL probes; attribution of every original full-run failure remains open. The SQL suite sets its own timeout to zero, overriding the runner's default timeout.

Reproduce and reduce

Use npm 11.21.1 and a canonical temporary path to keep npm 12 install-policy changes and macOS path-alias assertions out of the comparison.

npm run test:unit:main -- --timeout 60000
npx mocha unitTests/components/deployActivation.test.js unitTests/components/deployCertification.test.js --grep "retries a transient failure|completes the swap after putting|keeps the previous version in place while it retries|does not fence a release another process|restores only while the release|makes a certified release durable" --timeout 60000

The reduced activation failure now has a confirmed background-work cause, described below. The other full-suite failures remain incompletely reduced; this is not a passing full-main gate. The rename fixtures use permission changes and a fixed 150 ms hold, which is one concrete timing dependency to check.

Reduced watcher mechanism (19 October)

The unchanged baseline and draft feature both reproduce a smaller prefix with 1,087 passing / 20 pending / one failing in about two minutes. The failure is retries a transient failure moving the candidate into place; two SQL probes take several seconds on both branches. This diagnostic omits the separately reproduced detached-process hang and does not replace the full gate.

A call-through observer identifies seven loader-test watchers still open after the ComponentLoader Status Integration fixture directory is deleted. Their watched paths include the temporary directory's ancestors. A SQL CPU profile is dominated by filesystem/path work, Chokidar and readdirp. The leak exists on baseline; the relevant loader test and watch modules are unchanged on current main.

In a matched feature control, closing only those seven deleted-fixture watchers at the start of the first activation-recovery case produces 1,088 passing / 20 pending / zero failing in 41 seconds. The SQL probes take 46 and 33 ms. This establishes the leak as a cause of interference in the reduced cases. Fix fixture ownership by closing the scopes/watchers before deleting their directories; do not retain a global cleanup hook as the repair. The other original certification/OptionsWatcher failures and the full 44-minute run still need qualification after that fix.

The newly observed Bun continuation CI failure in Add full-text and autocomplete snapshot continuation has a different cause: worker-local high-resolution clock origins. It is an implementation blocker owned by that draft, not explained by this watcher result.

Acceptance

• Reduce the full-suite failure to the required preceding tests and explain why isolation passes.
• Fix the owning test or shared state without weakening assertions or excluding cases from the gate.
• Bound waits and clean up workers, processes, watchers and files on setup, assertion and teardown failures.
• Run both the isolated cases and the complete main gate after the fix; distinguish platform-specific results.

Parent: Deployment tracking, replicated payload delivery, and rollback. The separately tracked unitTests/components/applicationSpawn.test.js hangs after its last test, no open-handle report was independently reproduced on the baseline and requires a timeout to bound the complete gate. It is separate from the deployment-test failures reported here.

Priority: P2, contained test-gate unreliability with no confirmed production impact. Effort: High, because the complete set of full-suite interactions still needs qualification after repairing the reproduced watcher leak. No release is committed; there is no customer or security incident associated with this report.

Comment generated by kAIle (GPT-6)