fix(kiro): store the transcript as JSONL so checkpoint scoping works - #73
Open
suhaanthayyil wants to merge 2 commits into
Open
fix(kiro): store the transcript as JSONL so checkpoint scoping works#73suhaanthayyil wants to merge 2 commits into
suhaanthayyil wants to merge 2 commits into
Conversation
Entire slices external-agent transcripts by LINE offset before compacting
them (transcript.SliceFromLine, cmd/entire/cli/transcript/parse.go:130,
applied at cmd/entire/cli/explain.go:1883 and
cmd/entire/cli/strategy/manual_commit_condensation.go:1006). Kiro stored
its transcript as one pretty-printed JSON document, so every checkpoint
compacted a fragment of a single object and wrote nothing at all: 1116 B
of transcript in, 0 B of transcript.jsonl out.
Line shape matters as much as line count. The generic JSONL compactor
keeps a line only when normalizeKind (compact.go:197) finds a TOP-LEVEL
"type"/"role" of user or assistant AND parseMessage (compact.go:617)
finds content under a TOP-LEVEL "message" object. Kiro satisfied neither:
its content sat under the "user"/"assistant" keys of a paired history
entry, and there was no top-level "type" at all.
A paired entry therefore becomes TWO lines. normalizeKind yields exactly
one kind per line, so {"user":...,"assistant":...} cannot survive as one
line without discarding a half; the halves are emitted as separate
records sharing an "entry" index, which is what decoding groups them back
by. Session-level fields are stamped on every record rather than kept in
a header, because a scoped slice starting past line 0 has no header.
That split is why get-transcript-position now reports a LINE count rather
than len(History). The unit change is safe because a position and the
bytes it indexes are always stored together: a checkpoint keeps its own
transcript blob alongside its own CheckpointTranscriptStart, and Entire
re-applies that position only to that blob -- "CheckpointTranscriptStart
indexes the stored format" (cmd/entire/cli/explain.go:2132). Historical
checkpoints keep reading in their own units forever. The one crossing
window is a session already in flight at upgrade, whose stale entry-count
position skips too few lines and so over-includes -- the bounded head
overlap Entire's compactor documents as tolerable -- and never drops
content. It self-heals at the next condensation.
A transcript this build cannot parse is stored verbatim rather than
discarded, a whole-document transcript written by an older build still
reads, and the write is atomic so a concurrent reader never sees a
half-written file.
Measured on the committed fixture through cli@main's real
SliceFromLine + compact.FullWithBoundary: 0 B -> 1084 B, six lines all
carrying content, prompts and responses intact, and tool results inlined
onto their tool_use blocks by tool_use_id.
Every assertion runs the committed fixture through the live encoder (materializeTranscript -> encodeTranscriptJSONL) rather than comparing against a checked-in expected constant, so the tests fail when the encoder changes instead of when a fixture drifts. The suite mirrors the three cli functions that decide whether a stored line survives -- normalizeKind (compact.go:197), parseMessage (compact.go:617) and SliceFromLine (transcript/parse.go:130) -- and asserts that real message TEXT reaches message.content, not merely that the file is non-empty. A non-empty file of empty envelopes is the failure mode these tests exist to catch, so a line carrying a kind but no content is an explicit error. Also covered: the paired entry splitting into two records that share an entry index; tool_use blocks carrying the type/id/name/input that stripAssistantContent keeps; tool_result blocks carrying the snake_case tool_use_id and string content extractUserContent reads; every scoped slice staying valid JSONL and keeping the half that survived a mid-entry cut; position and extraction agreeing with what Entire itself sees after slicing at the same offset; legacy whole-document transcripts still reading in their original unit; and the capture path actually writing JSONL to disk, not just the encoder being correct in isolation. Tests that assert on the on-disk file now read it through parseTranscript rather than unmarshalling one JSON document, because the stored format is what this change replaces.
Contributor
Author
|
CI note: That is Locally, with the repo
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Kiro checkpoints were writing a 0-byte
transcript.jsonl. This is the lastof the six adapters with that failure (omp, kilo #70, amp #69, goose #71 and
qwen #72 cover the others).
Entire slices external-agent transcripts by line offset before compacting
them —
transcript.SliceFromLine(cmd/entire/cli/transcript/parse.go:130),applied at
cmd/entire/cli/explain.go:1883andcmd/entire/cli/strategy/manual_commit_condensation.go:1006. Kiro stored itstranscript as one pretty-printed JSON document, so every checkpoint compacted a
fragment of a single object and produced nothing.
Note that an adapter's own
compact-transcriptprotocol method never runs onthis path:
cmd/entire/cli/checkpoint/persistent.go:1136always uses Entire'sgeneric compactor. Kiro's
CompactTranscriptwas correct and irrelevant.Both halves of the contract
The generic JSONL compactor keeps a line only when both hold:
type, falling back torole, ofuser/assistantcompact.go:197normalizeKindtypeat allmessageobjectcompact.go:617parseMessageuser/assistantkeys of a paired entryKiro failed both, so JSONL alone would have produced the right number of lines
carrying nothing.
One history entry becomes two lines
kiroHistoryEntryis paired —{"user":…,"assistant":…}— andnormalizeKindyields exactly one kind per line, so a pair cannot survive as asingle line without discarding a half. The halves are emitted as separate
records sharing an
entryindex, which is what decoding groups them back by.Session-level fields (
conversation_id,cli_version) are stamped on everyrecord rather than kept in a header:
SliceFromLinehands a mid-sessioncheckpoint a slice that starts past line 0, where a header is gone.
Payloads this build cannot project into content blocks are still written (so
kiro's own extractors keep seeing them) but carry no
type, sonormalizeKinddrops them instead of emitting an empty envelope.
Why the position unit can change safely
get-transcript-positionnow reports a line count instead oflen(History).That is the unit
SliceFromLineconsumes, and it matches every one of Entire'sbuilt-in agents.
The reason this does not mis-scope existing checkpoints: a position and the
bytes it indexes are always stored together. A checkpoint keeps its own
transcript blob (
checkpoint.SessionContent.Transcript,api/checkpoint/metadata.go:376) alongside its ownCheckpointTranscriptStart,and Entire re-applies that position only to that blob —
"
CheckpointTranscriptStartindexes the stored format"(
cmd/entire/cli/explain.go:2132). A historical checkpoint written by an olderbuild therefore keeps reading in its own units forever; nothing ever applies an
entry-count position to a line-counted file.
The one window where units can cross is a session already in flight when the
binary is upgraded, whose live state still holds an entry-count
CheckpointTranscriptStart. Applying it to a line-counted file skips toofew lines, so that single checkpoint over-includes some already-checkpointed
content — the bounded head overlap Entire's own compactor documents as tolerable
(
compact.go:139-147) — and never drops any. It self-heals at the nextcondensation, which rewrites the position in the new unit
(
manual_commit_hooks.go:1529,1548).A whole-document transcript written by an older build still parses, reported in
its original history-entry unit, so nothing breaks before that rewrite happens.
Measured
Kiro's own committed fixture (
testCLIAnalyzerTranscript) run throughcli@
main's realSliceFromLine+compact.FullWithBoundary:transcript.jsonlContent assertion, not just byte count — every prompt and response survives, and
tool results are inlined onto their
tool_useblocks bytool_use_id(
inlineToolResults,compact.go:454):Scoping works too: at
StartLine=4the compactor reportsboundary=4and a387 B delta that still carries the final response.
Testing
Every assertion runs the fixture through the live encoder, never against a
checked-in expected constant, so the tests fail when the encoder changes rather
than when a fixture drifts. The suite mirrors
normalizeKind,parseMessageand
SliceFromLine, and asserts real message text reachesmessage.content— a non-empty file of empty envelopes is the failure modethese tests exist to catch, so a line with a kind but no content is an explicit
error.
Also covered: the two-record split and its shared
entryindex;tool_useblocks carrying the
type/id/name/inputthatstripAssistantContentkeeps;
tool_resultblocks carrying the snake_casetool_use_idand stringcontentthatextractUserContentreads; every scoped slice staying validJSONL and keeping the half that survived a mid-entry cut; position and
extraction agreeing with what Entire itself sees after slicing at the same
offset; legacy whole-document transcripts still reading; and the capture path
actually writing JSONL to disk rather than the encoder merely being correct in
isolation.
Verified by mutation — each of these compiles and is caught:
type(half 1)messagewrapper (half 2)toolUseIdgo test ./...127 pass,go vetclean,golangci-lint(repo config, 2.11.3)0 issues —
mainis also 0 for this agent.Compatibility
No protocol or CLI contract changes. Reads are back-compatible; the stored
format changes on the next capture. Tests that assert on the on-disk file now
read it through
parseTranscriptrather than unmarshalling one JSON document,because the stored format is what this change replaces.
Relationship to #58
No file overlap. #58 touches
kiro/paths.goandkiro/paths_test.go; thistouches
transcript.go,compact.go,AGENT.mdand two new/one modified testfiles. They compose: #58 sanitizes
ResolveSessionFile, which is upstream ofcacheTranscriptPath, and the atomic write introduced here creates itstemporary file inside that same sanitized directory.