test(core): restore the default timeout for the code executor tests - #794
Conversation
AmaadMartin
left a comment
There was a problem hiding this comment.
Approve. I verified the diff against the file at head 34f7d70: it deletes only the vi.setConfig({testTimeout: 30_000}) override and its comment, and vi stays in use. The two slowest cases set their own executor timeouts (0.5s and 1s), well under the restored 5s default, so the change is sound. One open item: should execute shell code and return stdout now runs on the 5s default on Windows, and #793 does not bound a cold PowerShell start; the author acknowledges this and plans a soak, so watch run-tests (windows-latest). CI: only auto-assign and cla/google had reported at review time, both pass.
Stacked on fix/code-executor-process-teardown. The 30s budget was raised in #662 to stop the Windows flake in #622, on the theory that a cold PowerShell start was blowing Vitest's 5s default. It did not work, because the failure was never slowness: a surviving grandchild held the stdio pipes open and 'close' never fired, so the wait was unbounded and no budget could contain it. The parent commit fixes that at the source. Two reasons to put the budget back rather than leave it: The raised value masks real regressions. A hang now fails in ~0.5s with the executor's own "Code execution timed out" message; under a 30s harness budget it would sit there for 30s and report a harness abort instead, which is what made #622 so hard to read. 30s also collided exactly with the executor's own default timeout of 30s, so even a merely-slow run was a coin flip between the two deadlines. That is why the CI failures alternated between "Test timed out in 30000ms" and "Code execution timed out after 30 seconds". A harness budget has to be strictly greater than the thing under test, never equal. Headroom at the restored default: the whole file runs in ~2.0s locally, and the two slowest cases (1009ms, 510ms) are bounded by their own explicit executor timeouts rather than by interpreter start-up, so they do not get slower on a loaded runner. The real-interpreter cases finish in tens of milliseconds. Leaves tests/integration/app_loader/app_loader_test.ts alone. Its 60s covers a genuine `npm install` per fixture, it did not fail in the runs behind #622, and its pre-#633 value of 40000 sat below the project's own INTEGRATION_TEST_TIMEOUT_MS of 60000 -- restoring it would restore a bug. Bug: #622
34f7d70 to
f0fb612
Compare
AmaadMartin
left a comment
There was a problem hiding this comment.
Approve. The change deletes the 30s testTimeout override and restores Vitest's 5s default. #793 is merged into main and fixes the hang at its source, so the inflated budget is no longer needed. I verified the head file: the vi import stays in use, and no source or suppression changed. CI was all green at review time, including run-tests (windows-latest), the job that carried the #622 flake.
Link to Issue or Description of Change
fix/code-executor-process-teardown, notmain. Merge fix(core): stop a surviving grandchild from hanging code execution #793 first; GitHub will retarget this tomainautomatically.Problem:
vi.setConfig({testTimeout: 30_000})was added in #662 to stop the Windows flake in #622, on the theory that a cold PowerShell start was blowing Vitest's 5s default. It did not work, because the failure was never slowness — a surviving grandchild held the stdio pipes open so'close'never fired, and the wait was unbounded. #793 fixes that at the source, which makes the inflated budget both unnecessary and actively harmful:Code execution timed outmessage. Under a 30s harness budget the same hang sits for 30s and reports a harness abort instead — which is exactly what made Flaky CI: app_loader and unsafe_local_code_executor time out intermittently on macOS/Windows (7 PRs affected) #622 so hard to read for two months.timeoutSeconds = 30. Even a merely-slow run was a coin flip between the two deadlines, which is why the CI failures alternated betweenTest timed out in 30000msandCode execution timed out after 30 seconds. A harness budget must be strictly greater than the thing under test, never equal.Solution:
Delete the override and its comment, restoring the pre-#662 state (Vitest's 5000ms default).
Headroom at the restored default. The whole file runs in ~2.0s locally, and the only two cases over 100ms are bounded by their own explicit executor timeouts rather than by interpreter start-up, so they do not stretch on a loaded runner:
The real-interpreter cases finish in tens of milliseconds.
What this deliberately does not touch.
tests/integration/app_loader/app_loader_test.tskeeps its 60s. It runs a genuinenpm installper fixture (app_loader_test.ts:65,112), it did not fail in the runs behind #622, and its pre-#633 value of40000sat below the project's ownINTEGRATION_TEST_TIMEOUT_MS = 60000— #622 itself flagged that as unintentional, so "restoring" it would restore a bug.Residual risk, stated plainly. A cold PowerShell or Python start on a loaded Windows runner can take 1–3s, so 5s has less margin than 30s. That is the point: with #793 in place a genuine hang is impossible, so a 5s failure would be real signal about Windows spawn latency rather than noise. If CI disagrees, the follow-up is a deliberate intermediate (10s — still clear of the executor's 30s so the deadlines cannot race), chosen from data rather than pre-emptively.
Suggested sequencing: merge #793, let
mainsoak a few Windows runs, then merge this and watchrun-tests (windows-latest)for ~10 runs before closing #622.Testing Plan
Unit Tests:
At the restored 5s default:
unsafe_local_code_executor_test.ts— 29 passed, file total 1959ms. Fullnpm run test:unit— 241 files, 3643 tests passed.ts:checkclean.Manual End-to-End (E2E) Tests:
Not applicable — test-configuration only, no source change.
Checklist