Skip to content

test(core): restore the default timeout for the code executor tests - #794

Merged
ScottMansfield merged 1 commit into
mainfrom
test/restore-code-executor-timeout
Aug 22, 2026
Merged

test(core): restore the default timeout for the code executor tests#794
ScottMansfield merged 1 commit into
mainfrom
test/restore-code-executor-timeout

Conversation

@ScottMansfield

Copy link
Copy Markdown
Member

Link to Issue or Description of Change

Problem:

vi.setConfig({testTimeout: 30_000}) was added in #662 to stop the Windows flake in #622, on the theory that a cold PowerShell start was blowing Vitest's 5s default. It did not work, because the failure was never slowness — a surviving grandchild held the stdio pipes open so 'close' never fired, and the wait was unbounded. #793 fixes that at the source, which makes the inflated budget both unnecessary and actively harmful:

  1. It masks real regressions. With fix(core): stop a surviving grandchild from hanging code execution #793 a hang now fails in ~0.5s carrying the executor's own Code execution timed out message. Under a 30s harness budget the same hang sits for 30s and reports a harness abort instead — which is exactly what made Flaky CI: app_loader and unsafe_local_code_executor time out intermittently on macOS/Windows (7 PRs affected) #622 so hard to read for two months.
  2. 30s collided exactly with the executor's own default timeoutSeconds = 30. Even a merely-slow run was a coin flip between the two deadlines, which is why the CI failures alternated between Test timed out in 30000ms and Code execution timed out after 30 seconds. A harness budget must be strictly greater than the thing under test, never equal.

Solution:

Delete the override and its comment, restoring the pre-#662 state (Vitest's 5000ms default).

Headroom at the restored default. The whole file runs in ~2.0s locally, and the only two cases over 100ms are bounded by their own explicit executor timeouts rather than by interpreter start-up, so they do not stretch on a loaded runner:

✓ times out even when the script leaves a child holding the pipes open   510ms   (0.5s executor timeout)
✓ should respect timeout                                                1009ms   (1s executor timeout)

The real-interpreter cases finish in tens of milliseconds.

What this deliberately does not touch. tests/integration/app_loader/app_loader_test.ts keeps its 60s. It runs a genuine npm install per fixture (app_loader_test.ts:65,112), it did not fail in the runs behind #622, and its pre-#633 value of 40000 sat below the project's own INTEGRATION_TEST_TIMEOUT_MS = 60000#622 itself flagged that as unintentional, so "restoring" it would restore a bug.

Residual risk, stated plainly. A cold PowerShell or Python start on a loaded Windows runner can take 1–3s, so 5s has less margin than 30s. That is the point: with #793 in place a genuine hang is impossible, so a 5s failure would be real signal about Windows spawn latency rather than noise. If CI disagrees, the follow-up is a deliberate intermediate (10s — still clear of the executor's 30s so the deadlines cannot race), chosen from data rather than pre-emptively.

Suggested sequencing: merge #793, let main soak a few Windows runs, then merge this and watch run-tests (windows-latest) for ~10 runs before closing #622.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

At the restored 5s default: unsafe_local_code_executor_test.ts29 passed, file total 1959ms. Full npm run test:unit — 241 files, 3643 tests passed. ts:check clean.

Manual End-to-End (E2E) Tests:

Not applicable — test-configuration only, no source change.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

@AmaadMartin AmaadMartin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. I verified the diff against the file at head 34f7d70: it deletes only the vi.setConfig({testTimeout: 30_000}) override and its comment, and vi stays in use. The two slowest cases set their own executor timeouts (0.5s and 1s), well under the restored 5s default, so the change is sound. One open item: should execute shell code and return stdout now runs on the 5s default on Windows, and #793 does not bound a cold PowerShell start; the author acknowledges this and plans a soak, so watch run-tests (windows-latest). CI: only auto-assign and cla/google had reported at review time, both pass.

Base automatically changed from fix/code-executor-process-teardown to main August 22, 2026 02:49
Stacked on fix/code-executor-process-teardown.

The 30s budget was raised in #662 to stop the Windows flake in #622, on
the theory that a cold PowerShell start was blowing Vitest's 5s default.
It did not work, because the failure was never slowness: a surviving
grandchild held the stdio pipes open and 'close' never fired, so the
wait was unbounded and no budget could contain it. The parent commit
fixes that at the source.

Two reasons to put the budget back rather than leave it:

The raised value masks real regressions. A hang now fails in ~0.5s with
the executor's own "Code execution timed out" message; under a 30s
harness budget it would sit there for 30s and report a harness abort
instead, which is what made #622 so hard to read.

30s also collided exactly with the executor's own default timeout of
30s, so even a merely-slow run was a coin flip between the two
deadlines. That is why the CI failures alternated between "Test timed
out in 30000ms" and "Code execution timed out after 30 seconds". A
harness budget has to be strictly greater than the thing under test,
never equal.

Headroom at the restored default: the whole file runs in ~2.0s locally,
and the two slowest cases (1009ms, 510ms) are bounded by their own
explicit executor timeouts rather than by interpreter start-up, so they
do not get slower on a loaded runner. The real-interpreter cases finish
in tens of milliseconds.

Leaves tests/integration/app_loader/app_loader_test.ts alone. Its 60s
covers a genuine `npm install` per fixture, it did not fail in the runs
behind #622, and its pre-#633 value of 40000 sat below the project's own
INTEGRATION_TEST_TIMEOUT_MS of 60000 -- restoring it would restore a bug.

Bug: #622

@AmaadMartin AmaadMartin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. The change deletes the 30s testTimeout override and restores Vitest's 5s default. #793 is merged into main and fixes the hang at its source, so the inflated budget is no longer needed. I verified the head file: the vi import stays in use, and no source or suppression changed. CI was all green at review time, including run-tests (windows-latest), the job that carried the #622 flake.

@ScottMansfield
ScottMansfield merged commit 1585e10 into main Aug 22, 2026
13 checks passed
@ScottMansfield
ScottMansfield deleted the test/restore-code-executor-timeout branch August 22, 2026 04:06
@AmaadMartin AmaadMartin mentioned this pull request Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants