Skip to content

test(runtime): Claude 5.x lifecycle — compaction, reload, switches, shared clients #4654

Closed
Task
#4689
Closed
test(runtime): Claude 5.x lifecycle — compaction, reload, switches, shared clients#4654
Task
#4689
Assignees
aheritier
Labels
area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic modelsarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionarea/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)area/testingTest infrastructure, CI/CD, test runners, evaluation

Description

@aheritier
aheritier
opened on Oct 19, 2026

Goal

Check that the Claude 5.x core holds up across the real session lifecycle: compaction, reload, model and transport switches, and several sessions sharing one client.

Test cost matters. Each behavior is pinned by the cheapest test that covers it: unit or converter level first, then request capture through a transport wrapper (as in pkg/model/provider/providers/claude5_requests_test.go), and a full runtime with a fake server only where the runtime integration itself is the point (compaction, reload, continuation). No new fake server per transport: in particular, no Bedrock event-stream fake server.

Delivery: three PRs, in order

PR Scope Size
1 Pricing (Haiku 5.5 tier boundary, serving model after reload) and concurrent sessions on one client; no new harness needed S
2 End-to-end runtime harness (factory, runtime, existing Anthropic SSE script, SQLite reload); compaction lifecycle on Anthropic Direct, plus two compaction bug fixes M
3 Switch and reload across Direct, Vertex and Bedrock at the request level; results table S–M

Scope

  • Local compaction triggered by the token threshold, with thinking-bearing turns: the summary goes through a checked clone, is persisted, survives a SQLite reload, and the first continuation is valid.
    • Under the default drop_block binding the continuation succeeds: the server drops thinking whose prefix changed.
    • An edited thinking turn inside the active tool loop is refused locally, and the context is not cleared.
  • Native on-demand compaction (Direct only): real client, persisted signed block, reload, and the continuation starts with the verbatim block. A refusal or a missing summary leaves the context intact. Iteration usage reaches the stored usage and cost.
    • The stream shape (whole block at content_block_start, no deltas) is already covered at the client level.
    • Vertex rejects native_compaction at construction (vertex.go). Bedrock lacks the capability, and the runtime reports it (AWS compaction).
  • Switches: route, then switch model or transport (a new provider) and clone (summary and title calls), add an attachment, reload, and continue, on each of the three transports. Covers what happens to recorded thinking across transports: replayed verbatim, flattened, or omitted. Bedrock is covered by request capture and the existing adapter-level tests, not by streamed responses.
  • Shared clients: two concurrent sessions on one client, with different attachments, tools and system prompts, don't leak state into each other (AGENTS.md, "Client and Session Lifetimes"). Run under -race.
  • Pricing:
    • Haiku 5.5's 100,000-token tier at 99,999 / 100,000 / 100,001 prompt tokens, cache reads and writes included, on Direct, Vertex and Bedrock. The tier mechanism already exists; only tests are missing.
    • The serving model from fix: price routed turns using the serving model #4579 survives save and reload.
    • Vertex Claude is priced at Claude API rates as a best estimate. Regional and multi-region surcharges are not modelled; the docs say so.

Bugs found while planning (fixed in PR 2)

  1. A native compaction block can be replayed to Vertex. ReplayableCompaction only compares the provider name (compaction.go), which Direct and Vertex share. After a Direct native compaction and a switch to Vertex, the signed block and its beta are sent to Vertex.
  2. Native compaction picks the price tier from summed iterations. Iteration usage is added up (compaction.go) before the tier is chosen, so two iterations of 60k tokens are priced as one 120k request. Each request should be priced on its own (pricing).

Out of scope (separate issues)

Acceptance criteria

  • Each scenario above pinned by the cheapest test that covers it; end to end (real factory, runtime, fake server) where the runtime integration is the point. Testing a hook on its own doesn't count for those.
  • Both bugs above fixed, each with a test that failed first.
  • go test -race -shuffle=on passes on the runtime and provider packages.
  • A results table per model × transport × feature: pass / fail / not established / expected refusal.
  • task build, task test, task lint pass.

Parent: #4583 · Depends on #4651, #4652, #4653 (all done)

Activity

  1. aheritier
    added
    area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic models
    area/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)
    area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)
    on Oct 19, 2026
  2. aheritier
    added a parent issue on Oct 19, 2026
  3. aheritier
    marked this as blocked by 3 issues on Oct 19, 2026
  4. aheritier
    added
    area/runtimeRuntime engine, agent loop execution, tool dispatch, loop detection
    area/testingTest infrastructure, CI/CD, test runners, evaluation
    on Oct 19, 2026
  5. aheritier
    added the
    Task
    issue type on Oct 19, 2026
  6. aheritier
    self-assigned this
    on Oct 19, 2026

  7. aheritier commented on Oct 21, 2026

    @aheritier
    aheritier
    on Oct 21, 2026
    CollaboratorAuthor

    Results: Claude 5.x lifecycle (#4654)

    Legend:

    • ✅ pass, pinned by the named test;
    • 🚫 expected refusal, tested;
    • ❔ not established, with the reason.

    Models: Fable 5 / 5.1, Mythos 5 / 5.1, Opus 5 / 5.5, Sonnet 5 / 5.5, Haiku 5.5. "Prefix-checked" models are the ones whose thinking blocks the API checks against their prefix: Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5.

    Feature Claude API Vertex AI Bedrock
    Model table and safety defaults (all nine models) ✅ TestLookupClaude5, TestClaude5Capabilities, TestClaude5Predicates ✅ same table (@version IDs) ✅ same table (regional and -vN IDs); 🚫 Mythos 5 outside Mantle: TestNewClient_RefusesMythos5
    Thinking request shape ✅ TestClaude5FinalRequests (standard, beta, binding) ✅ TestClaude5FinalRequests/vertex ✅ TestClaude5FinalRequests/bedrock_converse, TestBuildRequest_Claude5
    drop_block binding on prefix-checked models ✅ TestThinkingBindingBehavior, TestApplyThinkingBinding ✅ TestThinkingBindingBehavior/vertex, TestClaude5SwitchAfterReload ❔ the binding controls are documented for the Claude API and Google Cloud only
    Recorded thinking replayed after reload ✅ TestReplayContent_*, TestClaude5SwitchAfterReload/same_model ✅ TestClaude5SwitchAfterReload/transport_switch_to_vertex ✅ TestNativeReasoning_RoundTrip, TestConvertAssistantContent_ReplaysNativeBlocks
    Switch: Sonnet 5.5 → Haiku 5.5 keeps sending thinking (the API drops what it can't read) ✅ TestClaude5SwitchAfterReload/model_switch_to_haiku_5.5 ✅ same Anthropic converter ❔ not tested on Bedrock
    Switch: Claude API → other transport, then reload — ✅ verbatim block: TestClaude5SwitchAfterReload/transport_switch_to_vertex ✅ signed reasoning: TestClaude5SwitchAfterReload/transport_switch_to_bedrock
    Switch: Bedrock → Claude API ✅ TestForeignProviderStateKeepsThinking ✅ same Anthropic converter —
    Thinking from another account ❔ dropped by the API, per the docs; not observable in tests ❔ same ❔ Bedrock behavior not documented
    Attachment after a switch ✅ TestClaude5SwitchAfterReload ✅ same ✅ same (with the models.dev catalogue)
    Edited thinking in the active tool loop 🚫 refused before sending: TestUnreplayableRawState 🚫 same Anthropic converter 🚫 TestUnreplayableNativeBlocks
    Local compaction, reload, continue ✅ TestClaude5LocalCompactionLifecycle ❔ the runtime path is the same, and replay is covered above; not run end to end ❔ same; no Bedrock fake server, by design
    Native compaction, reload, continue ✅ TestClaude5NativeCompactionLifecycle, TestCompaction_* 🚫 option rejected: TestRejectClaudeAPIOnlyFeatures; the block isn't replayed: TestVertexDoesNotReplayCompactionBlock 🚫 not a native compactor: TestNativeCompactionRejectsUnsupportedModel (generic provider)
    Native compaction: refusal or missing summary keeps the context ✅ TestCompactConversation_Safeguards (including refusal), TestNativeCompactionErrorLeavesTranscriptIntact, TestNativeCompactionEmptySummaryIsNoop — —
    Native compaction cost, priced per iteration ✅ TestNativeCompactionPricesEachIteration, TestCompactionCostUnpricedIteration — —
    Haiku 5.5 100k price tier (99,999 / 100,000 / 100,001) ✅ TestEmbeddedSnapshotHaiku55ContextTier ✅ priced from the Claude API catalogue (best estimate; regional surcharges not modelled) ✅ global. and us. rows
    Serving model and cost after reload ✅ TestRoutedPricing/catalog_leaf_survives_reload (runtime, transport-independent) ✅ same ✅ same
    Concurrent sessions on one client ✅ TestSharedClientConcurrentSessions (-race) ✅ same client type as the Claude API ❔ the client holds no per-session state; not run, as that needs a Bedrock fake server

    Delivered by #4686 (PR 1, merged), #4688 (PR 2, merged) and #4689 (PR 3). Out of scope: #4683, #4684, #4685.

    Follow-up to consider: Google Cloud now documents on-demand compaction (beta), while docker-agent rejects native_compaction on Vertex.

  8. aheritier
    closed this as completedin #4689on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/providers/anthropicFor features/issues/fixes related to the usage of Anthropic modelsarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionarea/sessionsFor features/issues/fixes related to session lifecycle (resume, persistence, export)area/testingTest infrastructure, CI/CD, test runners, evaluation

Type

Task

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions