Add an agent session driver and Gadget durability tests - #270
Closed
AshishKumar4 wants to merge 7 commits into
Closed
Add an agent session driver and Gadget durability tests#270AshishKumar4 wants to merge 7 commits into
AshishKumar4 wants to merge 7 commits into
Conversation
This was referenced Aug 19, 2026
Maximo-Guk
reviewed
Aug 19, 2026
AshishKumar4
force-pushed
the
evals/toolkit
branch
2 times, most recently
from
August 24, 2026 17:34
bf05017 to
e10294c
Compare
AgentSession drives one agent session over the same Cap'n Web API the browser uses: a fresh account and workspace, one chat across turns, complete paginated history, and an optional source snapshot. Two methods support tests that need a known implementation rather than whatever an agent produced. seedGadget() writes hand-authored source into the workspace. restartGadgets() restarts every Gadget server by applying an empty code update, which is what the platform does on every code change. gadget-durability.test.ts uses both to pin platform behaviour with no model involved. Storage survives a restart and memory does not, outstanding stubs become invalid, and the data holds across five restarts and across one that interrupts a write. It also shows that a check-then-write implementation oversells under concurrent calls. startHarness() gains enableGadgetExecution, which keeps the Worker Loader so Gadget code can run. It defaults to false, so the existing suites are unchanged.
A handler receives the URL, the method, and the headers, but never the body, so it cannot stand in for a host a suite has to reach with a real POST. passThroughHosts exempts such a host before the request is taken apart. Every other host still throws.
Every .js file in a Gadget becomes a module in its Worker, so workerd parses client.js at load even though the server never imports it. A test that checks a Gadget through its RPC therefore already covers the syntax of both files, and a separate parse step would add nothing.
AshishKumar4
force-pushed
the
evals/toolkit
branch
from
August 24, 2026 17:59
e10294c to
035b216
Compare
Contributor
Author
|
The reduced eval implementation and its shared session driver now belong to one review unit. #271 contains this branch and is being retargeted to main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the reusable out-of-process driver for Workshop integration tests and live agent evals.
Everything in this PR is deterministic and runs in
pnpm test. It does not call a model.AgentSession
AgentSessiondrives the same public Cap'n Web API as the browser. It:Local sessions create an isolated password account. A deployed preview can attach an Access application JWT to the WebSocket handshake and call
authenticateFromCfAccess().Shared harness
startHarness()can retain the Worker Loader when an agent eval needs to execute generated Gadgets. Existing deterministic integration tests continue to use the same multi-Worker workerd and gatekeeper setup.Verification
pnpm --filter @gadgets/integration-tests buildpnpm --filter @gadgets/integration-tests test:runThe live eval package that consumes this driver is stacked in #271.