Repository navigation
test(directsdk): the session tests write Claude Code's userID into their throwaway home, so per-call and kept carry one device id on Windows too - #411
Open
metahacker wants to merge 157 commits into
Conversation
…ess (NousResearch#329) The plugin started one claude per model call with no limit. A delegate_task fan-out (up to 10 children by default) plus auxiliary calls started 10+ processes at the same moment, about 250 MB each, all on one subscription, with Claude Code's own retries off, so a burst could trip the plan's rate limits for every caller together. Every turn now takes a place in one process-wide pool before its claude starts (the extra-usage check included) and gives it back only once that process has exited, on the fast-return drain thread too, so a crashed or killed native never holds one. The pool lives under one sys.modules key because Hermes imports the plugin once per profile home; a per-module pool would let a multiplexed gateway run max_concurrent per profile. A call past the limit waits in arrival order within its own timeout. It is not failed and not sent to a fallback provider. Cancel ends the wait within 0.1 s without spawning. While waiting it yields the plugin's existing empty liveness chunk once a second; without it Hermes' stale timer would kill the waiting call and requeue it at the back. A wait over 2 s is logged (one warning a minute, info after), since a model provider has no door to the status line. /usage, the picker handshake and auth status take no place: they send no Messages request, end within seconds, and /usage must answer while a fan-out holds every place. Setting: max_concurrent in plugin.yaml config_schema, default 4, read per call from the serving profile.
Appends from several processes to one file are not atomic on Windows; the Windows lane lost one fake's start line (4 of 5 recorded, all 5 calls served).
…rrency feat(directsdk): cap concurrent Claude Code processes
…iction during a park falls back to per-call bytes
…rounds, compaction, edit, park) and replay-cost bench
… texts merge into one wire message; one process per per-call step
…pped client's process ends with it With max_sessions 1, three agents starting together kept three processes: the cap was checked under _live_lock but the place was taken only when the launched native joined _live. A _Reservation now holds the place from the check. The idle timer held the client's bound _end_session, so a client Hermes dropped without close() stayed referenced, and its process with it, for the idle hour. The timer calls through a WeakMethod and a finalizer ends the process a collected client still held. _keep refuses to publish on a client closed mid-step; settled() asks the reader thread instead of poll(), which reaped the child behind the reader's group kill.
…ll before the park
…vious mode's drains, budgets each turn by its own rounds
…Hermes' frames is refused before upstream and replayed
…rocess state together; a pid that vanishes mid-check counts as gone
…the parked row's pid for survival; a bare claude resolves on PATH
…ssion ends it; ownership moves under both locks Review M2 on NousResearch#361 (and Greptile's "Cancellation misses the kept process"). The ownership-transfer change is the reviewer's patch, kept together with the weak idle timer. The test pauses a step just before the hand-over, stops the client from another thread and expects the cancellation error, no kept session, no process and no place held.
…equest; control writes are bounded like frames Review M3 on NousResearch#361 (and Greptile's "Tool replies bypass the timeout"). The bound on _write is the reviewer's patch: one timer per control write, as send() already has per frame. A write that gives up on the reader thread now reaches the step as the timeout error too, not as bad output. The test parks a call on the real CLI, stops the child, brings a 250,000 byte result with a 0.2 s timeout and expects the timeout error with nothing left running; the same result on a healthy child arrives in full on the same process.
…and its place back at once Review m1 on NousResearch#361 (and Greptile's "Dead processes consume slots"). The reader thread ends the session through the client when native exits, and release() no longer joins the reader from the reader. A process that exits just before it becomes the session is ended by the hand-over. Tests: kill the idle child, the place is free and a second agent with max_sessions 1 keeps a process; kill it at the hand-over, the step returns and no place stays held.
Review m3 on NousResearch#361 (and Greptile's "Failed launches leave relays"). The launch directory is made inside the relay's cleanup scope, as the reviewer suggested. The test spawns nothing: _launch_dir raises OSError and the relay's thread, its listener and its place against max_sessions must all be gone.
…against max_sessions back Follow-up to review M1 on NousResearch#361. The place was taken under the lock but released only inside _launch's own cleanup, so a relay that could not start kept it for the life of the process. _run now gives it back however the launch ends (the reviewer's release on every outcome), and _launch still hands the place from the reservation to native in one step under the lock, so the count never reads one high. The test fails the relay with OSError and expects no place held.
…stream; the extension check has its own case Review m5 on NousResearch#361. tests/test_admission_session_drift.py is the reviewer's regression, added as written in the review. On this branch a history that does not line up with Hermes' frames is refused first, and that test's drift case has no frames to line up with, so either refusal answers it. The new case in test_admission_session_history.py changes the model on rows that do line up: it fails when the extension check stops refusing and passes without the alignment refusal.
…ls/ci.sh Review m4 on NousResearch#361 (and Greptile's "Session tests miss CI"). A runner without claude on PATH skips every test in tests/test_directsdk_session*.py and the suite still exits 0. ci.sh already installs the pin for the evals, so it runs those three files with CLAUDE_CODE_BIN set to it, on Linux and macOS as the evals do.
…bug level launch, edited, results, steer, exited or diverged: resumes() answers with the reason instead of None, and one _restart() kills, releases and logs. The divergence line moves from info to debug with the others. The session test asserts no restart line at debug level during the conversation, and the reason for its edit of the opening turn.
…s with per-call's bytes resumes() has a third answer for the expected results followed by one text block: the step writes the steer as a user frame, then delivers the results. Native reads its stdin in order, so it sends one request with both, the steer as its own system row (the same wording on 2.1.292, 2.1.322 and 2.1.323). canonical_history() puts that row back as the text block after the results, in the newest frame and in native's history after it, and refuses a request with a steer row it cannot match, so a Claude Code that words it otherwise restarts into per-call bytes. A steer of more than one text block restarts at once. The real-CLI tests (one and three parked calls, both restarts) run in evals/ci.sh.
…rted and per-call --steer 50,200 has Hermes deliver a steer after those tool rounds and adds a session_restart mode, session mode with the steer not carried (as before), so the step's cost is reported three ways and each session mode's bytes are compared with per-call's.
…; the steer bench and its numbers The transport list says what a steer does on a kept process and what it costs at round 200, kept, restarted and per-call; the steering paragraph no longer says the provider adds nothing; ci.sh's session files include _steer.py.
…imit_event; declare and document it The reserve only took effect on a get_usage probe, which runs only with no current answer. After any turn the process holds the turn's own rate_limit_event answer (using_overage false below 100%) for up to 15 minutes, so a running gateway kept sending past the reserve until the plan wall: the setting worked only on a fresh process. The event summary now carries the fuller account-wide window's utilization (unifiedWindows five_hour/seven_day), and under ask/stop with permitted overage a window at the reserve counts as spilling. The refusal says it was the reserve. The setting is declared in plugin.yaml (Desktop settings form) and documented in the README; CHANGELOG line under Added.
…hread per replayed frame Every replayed frame is one stdin write, and each write started a threading.Timer (one thread) as its stall guard: 602 thread starts for a 300-round history. One watchdog thread per request now guards every write. A write records when it began; the thread never sleeps past the timeout, so it wakes before the deadline of any write that began while it slept, then sleeps until exactly that deadline. Plugin CPU per call (4 vCPU Linux, Claude Code 2.1.327, loopback upstream): 100 rounds 0.19 -> 0.10 s, 300 rounds 0.42 -> 0.23 s. Claude Code's own CPU is unchanged.
… at 5m, not 1h Core clamps a delegated child to the 5m cache tier (_apply_child_cache_ttl: a subagent calls every few seconds for minutes and is gone, so the 2x write buys retention it never collects). Core can only clamp the cache_control markers it builds. On this route native places them, and a `claude -p` turn is Claude Code's main-conversation bucket, which asks for the one-hour tier on a subscription within plan usage. Every child request therefore wrote at 1h: 2x the input rate instead of 1.25x. The profile hook reads the cache scope core stamps on a child (sub_<hash>, prompt_cache_scope.is_sibling_cache_scope) and returns hermes_cache_ttl=5m; the spawn then starts native with CLAUDE_CODE_PROMPT_CACHE_TTL=5m (documented, Claude Code 2.1.242+). A tier the user exported wins. Every other request keeps native's default. CLAUDE_CODE_SUBAGENT_PROMPT_CACHE_TTL is deliberately not used: it governs requests outside the main bucket, and a -p turn is inside it.
…ss, not every call
…r loop's next 50 ms poll
…, as a real step does
…r a connection that is gone
…ts start, not later in its thread
…n three Claude Code versions
…riting under its temporary HOME at cleanup
… after its deadline or its cancellation
…er's deadline, inside a long poll
Native's limit message names a clock time ("resets 3:45pm") that core
cannot parse, so core never knew when a used-up window reopens. The
relay now keeps the reset of a 429 whose unified status is "rejected",
and the error carries it as body["resets_at"], the field core reads for
its fallback bench and limit notice.
…ession-steer: main's fixes carried onto the kept process NousResearch#445 lands 15 fixes on main. Where a kept process takes the same path, it gets the same fix: close() no longer refuses or cancels a call, it ends the kept process (NousResearch#378); a delegated child's 5m cache tier is in the launch env and so in what a kept process is matched on (NousResearch#416); an incomplete upstream answer keeps its transport phrase and plan-window reset (NousResearch#396, NousResearch#387); and one stdin watchdog per request serves a kept step's frames too (NousResearch#419). The watchdog ends with its step, so a kept process between steps holds neither a thread nor its client. One-shots mount no MCP server here, so the frozen-copy test checks that a one-shot names none instead of the inert helper.
…rozen copy is deleted: plugin.yaml is read at import
… process leaves the step's answer standing; the process still ends
…ate_limit_event is read as it arrives, and a step it reports nothing on feeds the last state
…d child's 5m cache tier and for a vendor-prefixed model id
…ce, so a client dropped right after a step is collected and its kept process ends
…rving profile's settings, not the gateway's
…o fix/directsdk-session-tests-windows: no conflict
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #402 (which is stacked on #361): only
d992f56is new here, with a merge of #402's head74a5ae5(main72eb3b8merged in) that had no conflict. It changes the test harness and the bench, not the plugin.On Windows two kept-process tests failed. The only difference in the bytes was
metadata.user_id.device_id, which differed between the per-call run and the kept run. Claude Code puts on the wire theuserIDit keeps in its config directory's.claude.json. In a fresh directory it makes one up and writes it back. Here, each process can make up its own before one of those writes lands, because the plugin ends a process right after its answer. On Windows two processes starting at once (a kept process and the per-call build that proves it) each kept their own.A real install already has a
userIDfrom its login, so this is the tests' fresh directory, not the plugin.Fix. The tests'
homefixture and the session bench write a fixeduserIDinto the throwaway home up front.device_id), 2 skippedBoth runs were on a scratch branch of the fork: the PR 402 head, the Windows step from #410, and the matrix cut to this plugin on Windows. The two skipped are the blocked-write test's two cases, which stop Claude Code with SIGSTOP. On Linux at
bb3c58bthe session test files pass with the fix: 130 on 2.1.292 and 130 on 2.1.328. CI atbb3c58bpasses on Linux, macOS and Windows; the Windows job has 71 passed, 2 skipped (https://github.com/NousResearch/hermes-official-plugins/actions/runs/44617415186). #402 at74a5ae5, without this commit, fails 1 test on Windows on the samedevice_iddifference (https://github.com/NousResearch/hermes-official-plugins/actions/runs/44616312164).