fix(qwen): give sidecar lines the shape Entire's compact reader needs - #72
Merged
Merged
Conversation
readSidecarRecords used a default bufio.Scanner, capping a single sidecar record at 64 KB. One record embeds a whole tool_input/tool_response, so any large file read or write produced a record over that limit and every transcript operation on the session — get-transcript-position, extract-modified-files, extract-prompts, extract-summary — failed with "bufio.Scanner: token too long". The oversized record stays on the append-only sidecar, so the failure was permanent for that session. The same loop also returned an error for the whole file when any single line failed to unmarshal. The sidecar is appended to while Qwen is running, so a torn final line is normal and must not discard the records already written. Raise the line limit to 10 MB and skip unparseable lines, matching how the other JSONL transcript scanners in this repo already read sessions (kilo/internal/kilo/session_jsonl.go, omp/internal/omp/session.go). Adds regression coverage for both cases; internal/qwen previously had no transcript tests.
Qwen's sidecar was already JSONL and get-transcript-position already reported
the record count, so Entire's line-offset scoping
(transcript.SliceFromLine, cmd/entire/cli/transcript/parse.go:130) landed on
record boundaries. The line count was never the problem. The line SHAPE was:
measured against cli@main, qwen's transcript.jsonl came out at 0 bytes.
Entire compacts an external agent's transcript with the generic JSONL reader in
cmd/entire/cli/transcript/compact/compact.go — an adapter's own
compact-transcript method does not change that, because
checkpoint/persistent.go:1136 always runs the generic compactor over the raw
transcript. It keeps a line only when BOTH of these hold, and a sidecar record
failed both:
- normalizeKind (compact.go:197) reads a TOP-LEVEL "type", falling back to
"role". A sidecar record has neither: its discriminator is "event"
("UserPromptSubmit", "PostToolUse", ...), so every line was dropped.
- parseMessage (compact.go:617) reads content from a TOP-LEVEL "message"
OBJECT: "All JSONL agents nest content inside a 'message' object." The
sidecar's "message" is a STRING — Qwen's notification text — so even a line
that had passed the first check would have carried nothing.
Each record now keeps its native fields and gains a projection built from them:
a top-level "type", a "timestamp" under the key parseLine reads, and the
"message" wrapper.
- UserPromptSubmit becomes a user text block
- PreToolUse becomes an assistant tool_use with type/id/name/input, the only
shape stripAssistantContent (compact.go:704) preserves
- PostToolUse / PostToolUseFailure become a user tool_result with snake_case
tool_use_id and a string "content" (compact.go:665), so Entire inlines the
output into the preceding tool_use (inlineToolResults, compact.go:454)
- Stop / StopFailure become an assistant text block
- every other event stays unprojected and is dropped, as before
Qwen fires PreToolUse before a call and PostToolUse after it, so that pairing is
Claude Code's own tool_use/tool_result shape and each hook still costs exactly
one line, keeping position == line count.
"message" is now stored as raw JSON. That is load-bearing, not cosmetic: typed
as a string, a projected line fails to unmarshal in readSidecarRecords and takes
the entire sidecar read with it. Qwen's notification text is the only thing that
used the key, and it is only sent on events that are never projected, so it is
written exactly as before; a projected record that also carried one would keep
it as the first text block of its content.
Back-compat: the projection is derived data and is ignored on read, so a sidecar
written by an older build still parses and later hooks append projected records
alongside it.
Measured with a full hook sequence (session-start, notification, prompt,
pre-tool-use, post-tool-use, stop, session-end) driven through ParseHook and run
through cli@main's transcript.SliceFromLine + compact.FullWithBoundary:
before start=0 raw 1032B/7 lines transcript.jsonl=0B
after start=0 raw 1674B/7 lines transcript.jsonl=574B, 3 lines with content
after start=3 boundary=1, delta=439B
The compacted lines now read:
{"type":"user","ts":"...02Z","content":[{"text":"Create hello.txt"}]}
{"type":"assistant","ts":"...03Z","id":"tool-1","content":[{"id":"tool-1",
"input":{"content":"hi","file_path":"hello.txt"},"name":"write_file",
"result":{"output":"{\"ok\":true}","status":"success"},"type":"tool_use"}]}
{"type":"assistant","ts":"...05Z","content":[{"text":"Created hello.txt",...}]}
Reverting only the message wrapper reproduces the other failure mode exactly:
468 bytes across 4 lines with every content array empty. A byte count would call
that fixed, so the tests assert the message TEXT, and they drive real hook
payloads through ParseHook rather than comparing against a checked-in constant.
Member
|
Reviewed for functional and security issues. This projection doubles large hook payloads, making the old 64 KiB reader limit even easier to hit. I integrated the reviewed #65 branch, including the larger-record support and physical-line offset fix, so this PR works independently. The original projection remains intact. Module race tests and lint pass, including the >64 KiB and malformed-line regressions. Current main was merged for the shared CI toolchain fix. No unresolved finding in this PR; fresh CI has 36/36 successful checks and GitHub reports MERGEABLE/CLEAN. Prefer merging #65 before this PR to keep the review history clear. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
Qwen's sidecar is already JSONL and
get-transcript-positionalready reports the record count, so Entire's line-offset scoping (transcript.SliceFromLine,cmd/entire/cli/transcript/parse.go:130) already lands on record boundaries. The line count was never the problem.The line shape was. Measured against cli@main, qwen's
transcript.jsonlcame out at 0 bytes.Entire compacts an external agent's transcript with the generic JSONL reader in
cmd/entire/cli/transcript/compact/compact.go. Shipping acompact-transcriptmethod does not avoid it —checkpoint/persistent.go:1136always runs the generic compactor over the raw transcript.That reader keeps a line only when both hold, and a native sidecar record failed both:
normalizeKind(compact.go:197) reads a top-leveltype, falling back toroleevent(UserPromptSubmit,PostToolUse, …), so every line was droppedparseMessage(compact.go:617) reads content from a top-levelmessageobjectmessageis a string (Qwen's notification text), so even a line that passed the first check would carry nothingqwen is not listed in #67 — its table implies qwen is fine. It is not; I have commented there.
The change
Each record keeps its native fields and gains a projection built from them: a top-level
type, atimestampunder the keyparseLinereads (compact.go:391), and themessagewrapper.UserPromptSubmituserPreToolUseassistanttool_usewithtype/id/name/input— the only fieldsstripAssistantContent(compact.go:704) preservesPostToolUse/PostToolUseFailureusertool_resultwith snake_casetool_use_idand a stringcontent(compact.go:665)Stop/StopFailureassistantQwen fires
PreToolUsebefore a call andPostToolUseafter it, so that pairing is Claude Code's own tool_use/tool_result shape:inlineToolResults(compact.go:454) folds the output into the precedingtool_use. Each hook still costs exactly one line, soposition == line countis preserved. An unpairedtool_resultis dropped by compact.go:288, which is cli's own behaviour for Claude Code;install-hooksregisters both hooks together, so the pair is present in any session Entire set up.messageis now raw JSON, and that is load-bearingsidecarRecord.Messagewasstring. With an object under that key,json.UnmarshalinreadSidecarRecordsfails — and on main that error aborts the whole sidecar read, so prompts, modified files and summaries all disappear. Typing it asjson.RawMessageis what makes a projected record round-trip.This also interacts with #65, which changes
readSidecarRecordsto skip unparseable lines instead of erroring: without this type change, a projected line would be silently skipped there. The two changes touch different hunks oftranscript.goand merge cleanly in either order.Qwen's notification text is the only thing that used
message, and it is only sent on events that are never projected, so it is written exactly as before — there is a test pinning that. A projected record that also carried one would keep it as the first text block of its content.Back-compat
The projection is derived data and is ignored on read, so a sidecar written by an older build still parses and later hooks append projected records alongside it. Test: a legacy line plus a freshly projected one, read together.
Measured
A full hook sequence (session-start, notification, prompt, pre-tool-use, post-tool-use, stop, session-end) driven through
ParseHook, then run through cli@main'stranscript.SliceFromLine+compact.FullWithBoundary:The compacted lines now read:
{"type":"user","ts":"2026-05-20T12:00:02Z","content":[{"text":"Create hello.txt"}]} {"type":"assistant","ts":"2026-05-20T12:00:03Z","id":"tool-1","content":[{"id":"tool-1","input":{"content":"hi","file_path":"hello.txt"},"name":"write_file","result":{"output":"{\"ok\":true}","status":"success"},"type":"tool_use"}]} {"type":"assistant","ts":"2026-05-20T12:00:05Z","content":[{"text":"Created hello.txt","type":"text"}]}Lifecycle and notification records are correctly dropped rather than emitted empty.
Why the tests are shaped this way
Reverting only the message wrapper — keeping the top-level
type— reproduces the other failure mode exactly: 468 bytes across 4 lines, every content array empty. A non-zero byte count would call that fixed. The tests therefore assert the message text, and they drive real hook payloads throughParseHookand read what the agent actually wrote, rather than comparing against a checked-in expected constant.Mutations tried, all caught, all compiling:
messagewrapper (keeptype)type(keep the wrapper)tool_use_id->toolUseIdtool_useinput->argumentsmessagetyped as a string againPostToolUsecarries no tool outputmessagedroppedgo test ./...inagents/entire-agent-qwen: 33 pass (was 23).Overlap with my other open qwen PRs
readSidecarRecordsintranscript.goand addstranscript_test.go. This PR touchesappendSidecarin the same file and addssidecar_jsonl.go+jsonl_test.go. Different hunks, different new files — no conflict. The semantic interaction is described above.hooks.go/hooks_test.goonly — no overlap.