Skip to content

fix(cache): deterministic Responses ids, stable session anchor, structural Anthropic breakpoints - #251

Draft
lecommander wants to merge 1 commit into
ltmoerdani:mainfrom
lecommander:feature/prompt-cache-stability
Draft

lecommander wants to merge 1 commit into
ltmoerdani:mainfrom
lecommander:feature/prompt-cache-stability

Conversation

@lecommander

Copy link
Copy Markdown
Contributor

Summary
Reduce provider prompt-cache misses caused by client-side prefix instability. Three independent fixes, each verified by the existing suite plus two new regression tests. No behavior change for cache-ineligible routes.

Changes

  • Responses item ids are now deterministic (P0a): responsesFunctionCallItemId maps non-fc ids to fc + sha256(call id, 32 hex) instead of randomUUID per request, fc ids pass through verbatim. Same history now replays byte-identically every turn. Previously the prefix cache broke at the first function_call item on every tool turn (same class as OpenHands SDK 2905).
  • Stable session anchor (P0b): the x-opencode-session fallback no longer hashes the first 3 messages (messages 1-2 shift on every history trim by design). It now hashes first message + sorted tool names + model id, memoized per conversation for its lifetime. Legacy conversationAnchor kept for compatibility.
  • Structural Anthropic breakpoints (P1): buildAnthropicMessages placed cache_control on the first 4 content blocks in emit order, so any added text/image part shifted all later marks and consumed the 4-slot budget gateways also inject into. Now at most 2 marks at structural boundaries: end of message 0 (static prefix) and end of message N-2 (tail checkpoint); only the last block of a marked message carries the mark.

Before / After

  • Before: Responses replay emitted a fresh fc id per request, session id re-hashed over shifting messages, Anthropic marks drifted with content growth.
  • After: replay payloads are deterministic, session header is constant across trims, Anthropic marks sit on stable boundaries. DeepSeek-class implicit routes get stability with zero new markers.

Verification

  • npx tsc -p ./ clean
  • Targeted: 24/24 pass (responsesRequest, session-header, headers)
  • Full npm test: 497/497 pass
  • Live gate to run after install: 30-turn run, prefix hash and session constant, cacheHit at 98 percent plus after turn 2, exactly one dip across a trim event.

Notes

  • Explicit markers remain gated to Anthropic-native and GPT-5.6-family routes by follow-up work; DeepSeek/Qwen/Kimi/GLM/MiniMax/MiMo/Muse stay marker-free (implicit only).
  • No keepalive, no previous_response_id chaining, no 1h TTL stamp in this PR.
  • Draft status: asking for direction on breakpoint policy and per-route gating before packaging a vsix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant