Skip to content

fix(kiro): store the transcript as JSONL so checkpoint scoping works - #73

Open
suhaanthayyil wants to merge 2 commits into
mainfrom
fix/kiro-jsonl-transcript
Open

fix(kiro): store the transcript as JSONL so checkpoint scoping works#73
suhaanthayyil wants to merge 2 commits into
mainfrom
fix/kiro-jsonl-transcript

Conversation

@suhaanthayyil

Copy link
Copy Markdown
Contributor

Summary

Kiro checkpoints were writing a 0-byte transcript.jsonl. This is the last
of the six adapters with that failure (omp, kilo #70, amp #69, goose #71 and
qwen #72 cover the others).

Entire slices external-agent transcripts by line offset before compacting
them — transcript.SliceFromLine (cmd/entire/cli/transcript/parse.go:130),
applied at cmd/entire/cli/explain.go:1883 and
cmd/entire/cli/strategy/manual_commit_condensation.go:1006. Kiro stored its
transcript as one pretty-printed JSON document, so every checkpoint compacted a
fragment of a single object and produced nothing.

Note that an adapter's own compact-transcript protocol method never runs on
this path: cmd/entire/cli/checkpoint/persistent.go:1136 always uses Entire's
generic compactor. Kiro's CompactTranscript was correct and irrelevant.

Both halves of the contract

The generic JSONL compactor keeps a line only when both hold:

Requirement cli reference kiro before
top-level type, falling back to role, of user/assistant compact.go:197 normalizeKind no top-level type at all
content under a top-level message object compact.go:617 parseMessage content under the user/assistant keys of a paired entry

Kiro failed both, so JSONL alone would have produced the right number of lines
carrying nothing.

One history entry becomes two lines

kiroHistoryEntry is paired{"user":…,"assistant":…} — and
normalizeKind yields exactly one kind per line, so a pair cannot survive as a
single line without discarding a half. The halves are emitted as separate
records sharing an entry index, which is what decoding groups them back by.

Session-level fields (conversation_id, cli_version) are stamped on every
record rather than kept in a header: SliceFromLine hands a mid-session
checkpoint a slice that starts past line 0, where a header is gone.

Payloads this build cannot project into content blocks are still written (so
kiro's own extractors keep seeing them) but carry no type, so normalizeKind
drops them instead of emitting an empty envelope.

Why the position unit can change safely

get-transcript-position now reports a line count instead of len(History).
That is the unit SliceFromLine consumes, and it matches every one of Entire's
built-in agents.

The reason this does not mis-scope existing checkpoints: a position and the
bytes it indexes are always stored together.
A checkpoint keeps its own
transcript blob (checkpoint.SessionContent.Transcript,
api/checkpoint/metadata.go:376) alongside its own CheckpointTranscriptStart,
and Entire re-applies that position only to that blob —
"CheckpointTranscriptStart indexes the stored format"
(cmd/entire/cli/explain.go:2132). A historical checkpoint written by an older
build therefore keeps reading in its own units forever; nothing ever applies an
entry-count position to a line-counted file.

The one window where units can cross is a session already in flight when the
binary is upgraded, whose live state still holds an entry-count
CheckpointTranscriptStart. Applying it to a line-counted file skips too
few
lines, so that single checkpoint over-includes some already-checkpointed
content — the bounded head overlap Entire's own compactor documents as tolerable
(compact.go:139-147) — and never drops any. It self-heals at the next
condensation, which rewrites the position in the new unit
(manual_commit_hooks.go:1529,1548).

A whole-document transcript written by an older build still parses, reported in
its original history-entry unit, so nothing breaks before that rewrite happens.

Measured

Kiro's own committed fixture (testCLIAnalyzerTranscript) run through
cli@main's real SliceFromLine + compact.FullWithBoundary:

raw transcript.jsonl lines carrying content
before 1116 B 0 B 0
after 2412 B 1084 B 6 / 6

Content assertion, not just byte count — every prompt and response survives, and
tool results are inlined onto their tool_use blocks by tool_use_id
(inlineToolResults, compact.go:454):

{"v":1,"agent":"kiro",…,"type":"user","content":[{"text":"Create a hello.go file"}]}
{"v":1,"agent":"kiro",…,"type":"assistant","id":"msg-1","content":[{"text":"I'll create that file for you.","type":"text"}]}
…
{"v":1,"agent":"kiro",…,"type":"assistant","id":"msg-2","content":[{"id":"tu-1","input":{"path":"/repo/hello.go",…},"name":"fs_write","result":{"output":"ok","status":"success"},"type":"tool_use"}]}

Scoping works too: at StartLine=4 the compactor reports boundary=4 and a
387 B delta that still carries the final response.

Testing

Every assertion runs the fixture through the live encoder, never against a
checked-in expected constant, so the tests fail when the encoder changes rather
than when a fixture drifts. The suite mirrors normalizeKind, parseMessage
and SliceFromLine, and asserts real message text reaches
message.content — a non-empty file of empty envelopes is the failure mode
these tests exist to catch, so a line with a kind but no content is an explicit
error.

Also covered: the two-record split and its shared entry index; tool_use
blocks carrying the type/id/name/input that stripAssistantContent
keeps; tool_result blocks carrying the snake_case tool_use_id and string
content that extractUserContent reads; every scoped slice staying valid
JSONL and keeping the half that survived a mid-entry cut; position and
extraction agreeing with what Entire itself sees after slicing at the same
offset; legacy whole-document transcripts still reading; and the capture path
actually writing JSONL to disk rather than the encoder merely being correct in
isolation.

Verified by mutation — each of these compiles and is caught:

mutation caught by
drop the top-level type (half 1) 4 tests
rename the message wrapper (half 2) 8 tests
collapse a paired entry back to one line 12 tests
report position in entry units 3 tests
camelCase toolUseId 1 test
write path skips materialization 1 test

go test ./... 127 pass, go vet clean, golangci-lint (repo config, 2.11.3)
0 issues — main is also 0 for this agent.

Compatibility

No protocol or CLI contract changes. Reads are back-compatible; the stored
format changes on the next capture. Tests that assert on the on-disk file now
read it through parseTranscript rather than unmarshalling one JSON document,
because the stored format is what this change replaces.

Relationship to #58

No file overlap. #58 touches kiro/paths.go and kiro/paths_test.go; this
touches transcript.go, compact.go, AGENT.md and two new/one modified test
files. They compose: #58 sanitizes ResolveSessionFile, which is upstream of
cacheTranscriptPath, and the atomic write introduced here creates its
temporary file inside that same sanitized directory.

Entire slices external-agent transcripts by LINE offset before compacting
them (transcript.SliceFromLine, cmd/entire/cli/transcript/parse.go:130,
applied at cmd/entire/cli/explain.go:1883 and
cmd/entire/cli/strategy/manual_commit_condensation.go:1006). Kiro stored
its transcript as one pretty-printed JSON document, so every checkpoint
compacted a fragment of a single object and wrote nothing at all: 1116 B
of transcript in, 0 B of transcript.jsonl out.

Line shape matters as much as line count. The generic JSONL compactor
keeps a line only when normalizeKind (compact.go:197) finds a TOP-LEVEL
"type"/"role" of user or assistant AND parseMessage (compact.go:617)
finds content under a TOP-LEVEL "message" object. Kiro satisfied neither:
its content sat under the "user"/"assistant" keys of a paired history
entry, and there was no top-level "type" at all.

A paired entry therefore becomes TWO lines. normalizeKind yields exactly
one kind per line, so {"user":...,"assistant":...} cannot survive as one
line without discarding a half; the halves are emitted as separate
records sharing an "entry" index, which is what decoding groups them back
by. Session-level fields are stamped on every record rather than kept in
a header, because a scoped slice starting past line 0 has no header.

That split is why get-transcript-position now reports a LINE count rather
than len(History). The unit change is safe because a position and the
bytes it indexes are always stored together: a checkpoint keeps its own
transcript blob alongside its own CheckpointTranscriptStart, and Entire
re-applies that position only to that blob -- "CheckpointTranscriptStart
indexes the stored format" (cmd/entire/cli/explain.go:2132). Historical
checkpoints keep reading in their own units forever. The one crossing
window is a session already in flight at upgrade, whose stale entry-count
position skips too few lines and so over-includes -- the bounded head
overlap Entire's compactor documents as tolerable -- and never drops
content. It self-heals at the next condensation.

A transcript this build cannot parse is stored verbatim rather than
discarded, a whole-document transcript written by an older build still
reads, and the write is atomic so a concurrent reader never sees a
half-written file.

Measured on the committed fixture through cli@main's real
SliceFromLine + compact.FullWithBoundary: 0 B -> 1084 B, six lines all
carrying content, prompts and responses intact, and tool results inlined
onto their tool_use blocks by tool_use_id.
Every assertion runs the committed fixture through the live encoder
(materializeTranscript -> encodeTranscriptJSONL) rather than comparing
against a checked-in expected constant, so the tests fail when the
encoder changes instead of when a fixture drifts.

The suite mirrors the three cli functions that decide whether a stored
line survives -- normalizeKind (compact.go:197), parseMessage
(compact.go:617) and SliceFromLine (transcript/parse.go:130) -- and
asserts that real message TEXT reaches message.content, not merely that
the file is non-empty. A non-empty file of empty envelopes is the failure
mode these tests exist to catch, so a line carrying a kind but no content
is an explicit error.

Also covered: the paired entry splitting into two records that share an
entry index; tool_use blocks carrying the type/id/name/input that
stripAssistantContent keeps; tool_result blocks carrying the snake_case
tool_use_id and string content extractUserContent reads; every scoped
slice staying valid JSONL and keeping the half that survived a mid-entry
cut; position and extraction agreeing with what Entire itself sees after
slicing at the same offset; legacy whole-document transcripts still
reading in their original unit; and the capture path actually writing
JSONL to disk, not just the encoder being correct in isolation.

Tests that assert on the on-disk file now read it through parseTranscript
rather than unmarshalling one JSON document, because the stored format is
what this change replaces.
@suhaanthayyil

Copy link
Copy Markdown
Contributor Author

CI note: lint-agents fails on all six agents on this branch, including the five this PR does not touch (amp, goose, kilo, omp, qwen), and lint-e2e too. The errors are typecheck failures inside internal/protocol/*.go — files this PR does not modify:

could not load export data: internal error in importing "internal/goarch"
(cannot decode "internal/goarch", export data version 4 is greater than
maximum supported version 2)

That is setup-go: stable resolving to Go 1.27.0, whose export data golangci-lint cannot read. It is the pre-existing repo-wide breakage #68 fixes, not something introduced here.

Locally, with the repo .golangci.yaml and the pinned golangci-lint 2.11.3 from mise.toml, this branch reports 0 issues for entire-agent-kiro — the same as main.

test-agents (entire-agent-kiro) and fmt pass on this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant