Skip to content

finalizeAllTurnCheckpoints has no total deadline: a slow remote can outlast the agent's hook timeout #2148

Description

@Soph

Split out of #2146, where the acute dead-network case is fixed by memoizing an exhausted blob fetch. This is the case the memos cannot catch.

The gap

finalizeAllTurnCheckpoints (strategy/manual_commit_hooks.go) loops over state.TurnCheckpointIDs against a single store, logging and continuing past each failure. Each iteration can make bounded network calls — a ref fetch and, since #2146, a blob fetch — each capped at remote.WriteProbeFetchBudget (15s).

Two memos bound the dead network: gitRefsStore.fetchFailure for refs, and hookBlobFetcher's exhaustion memo for blobs (#2146). Both trigger only when a fetch burns its whole budget.

A slow-but-alive remote trips neither. Every fetch returns just under budget, nothing is memoized, and the cost accumulates per checkpoint. 5 × 14s is as fatal to a hook as 5 × 15s. Worse, FetchingTree.File fetches one hash at a time, so a read touching several missing blobs multiplies within a single iteration.

Nothing caps the total:

  • newGitHookContext (hooks_git_cmd.go:41) wraps logging and a perf span only — no deadline.
  • lifecycle.go's condenseDeadline is a different path and does not enclose this loop.

So the effective ceiling is the agent killing the hook — roughly 60s for Claude Code — after which everything downstream of the loop is skipped, silently.

Why this is more than a latency nit

The stop hook is what the user's agent waits on. Being killed mid-loop means some checkpoints keep their provisional mid-turn transcript instead of the full session transcript, with only Warn-level breadcrumbs. It is also the failure mode most likely in the field: flaky wifi, a VPN, a throttled or loaded remote are all commoner than a cleanly dead network.

Direction, not a decision

An enclosing context.WithTimeout over the loop is the obvious shape, but two things need deciding rather than assuming:

  1. The budget. It has to fit inside the tightest host deadline, and those differ — Claude Code's ~60s hook timeout versus Codex's agent.SessionEndBudgeter 3s cap, where the process tree is killed on expiry. Existing reference points: StatusWalkBudget 20s, WriteProbeFetchBudget 15s, ReadChainBudget 3min.
  2. What happens to checkpoints past the deadline. Almost certainly fine to leave unfinalized: finalize is already best-effort (errCount++; continue), and PostCommit retries an incomplete condense. But that should be confirmed rather than assumed, and whatever it is should be visible — entire status already surfaces capture degradation via SessionState.CaptureDegradedAt, which may be the right precedent.

A cheaper partial mitigation worth considering either way: have FetchingTree.File fall back to PreFetch-style batching on a miss, so one read costs one round-trip instead of one per blob.

Not a regression

This predates #2146 — the ref path had the same unbounded-loop shape with only a dead-network memo. #2146 adds a second fetch per iteration, which raises the constant but is not the cause.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions