Skip to content

fix(qwen): give sidecar lines the shape Entire's compact reader needs - #72

Merged
ashtom merged 6 commits into
mainfrom
fix/qwen-jsonl-transcript-shape
Sep 9, 2026
Merged

ashtom merged 6 commits into
mainfrom
fix/qwen-jsonl-transcript-shape

Conversation

@suhaanthayyil

Copy link
Copy Markdown
Contributor

The defect

Qwen's sidecar is already JSONL and get-transcript-position already reports the record count, so Entire's line-offset scoping (transcript.SliceFromLine, cmd/entire/cli/transcript/parse.go:130) already lands on record boundaries. The line count was never the problem.

The line shape was. Measured against cli@main, qwen's transcript.jsonl came out at 0 bytes.

Entire compacts an external agent's transcript with the generic JSONL reader in cmd/entire/cli/transcript/compact/compact.go. Shipping a compact-transcript method does not avoid it — checkpoint/persistent.go:1136 always runs the generic compactor over the raw transcript.

That reader keeps a line only when both hold, and a native sidecar record failed both:

half what cli reads native record
kind normalizeKind (compact.go:197) reads a top-level type, falling back to role has neither — the discriminator is event (UserPromptSubmit, PostToolUse, …), so every line was dropped
content parseMessage (compact.go:617) reads content from a top-level message object message is a string (Qwen's notification text), so even a line that passed the first check would carry nothing

qwen is not listed in #67 — its table implies qwen is fine. It is not; I have commented there.

The change

Each record keeps its native fields and gains a projection built from them: a top-level type, a timestamp under the key parseLine reads (compact.go:391), and the message wrapper.

event projected as content
UserPromptSubmit user text block with the prompt
PreToolUse assistant tool_use with type/id/name/input — the only fields stripAssistantContent (compact.go:704) preserves
PostToolUse / PostToolUseFailure user tool_result with snake_case tool_use_id and a string content (compact.go:665)
Stop / StopFailure assistant text block with the final message or error details
everything else not projected dropped, as before

Qwen fires PreToolUse before a call and PostToolUse after it, so that pairing is Claude Code's own tool_use/tool_result shape: inlineToolResults (compact.go:454) folds the output into the preceding tool_use. Each hook still costs exactly one line, so position == line count is preserved. An unpaired tool_result is dropped by compact.go:288, which is cli's own behaviour for Claude Code; install-hooks registers both hooks together, so the pair is present in any session Entire set up.

message is now raw JSON, and that is load-bearing

sidecarRecord.Message was string. With an object under that key, json.Unmarshal in readSidecarRecords fails — and on main that error aborts the whole sidecar read, so prompts, modified files and summaries all disappear. Typing it as json.RawMessage is what makes a projected record round-trip.

This also interacts with #65, which changes readSidecarRecords to skip unparseable lines instead of erroring: without this type change, a projected line would be silently skipped there. The two changes touch different hunks of transcript.go and merge cleanly in either order.

Qwen's notification text is the only thing that used message, and it is only sent on events that are never projected, so it is written exactly as before — there is a test pinning that. A projected record that also carried one would keep it as the first text block of its content.

Back-compat

The projection is derived data and is ignored on read, so a sidecar written by an older build still parses and later hooks append projected records alongside it. Test: a legacy line plus a freshly projected one, read together.

Measured

A full hook sequence (session-start, notification, prompt, pre-tool-use, post-tool-use, stop, session-end) driven through ParseHook, then run through cli@main's transcript.SliceFromLine + compact.FullWithBoundary:

before  start=0  raw 1032B / 7 lines   transcript.jsonl=0B
after   start=0  raw 1674B / 7 lines   transcript.jsonl=574B, 3 lines all carrying content
after   start=3  boundary=1, delta=439B

The compacted lines now read:

{"type":"user","ts":"2026-05-20T12:00:02Z","content":[{"text":"Create hello.txt"}]}
{"type":"assistant","ts":"2026-05-20T12:00:03Z","id":"tool-1","content":[{"id":"tool-1","input":{"content":"hi","file_path":"hello.txt"},"name":"write_file","result":{"output":"{\"ok\":true}","status":"success"},"type":"tool_use"}]}
{"type":"assistant","ts":"2026-05-20T12:00:05Z","content":[{"text":"Created hello.txt","type":"text"}]}

Lifecycle and notification records are correctly dropped rather than emitted empty.

Why the tests are shaped this way

Reverting only the message wrapper — keeping the top-level type — reproduces the other failure mode exactly: 468 bytes across 4 lines, every content array empty. A non-zero byte count would call that fixed. The tests therefore assert the message text, and they drive real hook payloads through ParseHook and read what the agent actually wrote, rather than comparing against a checked-in expected constant.

Mutations tried, all caught, all compiling:

mutation caught by
drop the message wrapper (keep type) 3 contract tests
drop the top-level type (keep the wrapper) 3 tests
tool_use_id -> toolUseId tool-shape test
tool_use input -> arguments tool-shape test
message typed as a string again 3 tests (the read-back guard)
PostToolUse carries no tool output 3 tests
notification message dropped notification test

go test ./... in agents/entire-agent-qwen: 33 pass (was 23).

Overlap with my other open qwen PRs

suhaanthayyil and others added 2 commits August 29, 2026 09:20
readSidecarRecords used a default bufio.Scanner, capping a single sidecar
record at 64 KB. One record embeds a whole tool_input/tool_response, so any
large file read or write produced a record over that limit and every
transcript operation on the session — get-transcript-position,
extract-modified-files, extract-prompts, extract-summary — failed with
"bufio.Scanner: token too long". The oversized record stays on the
append-only sidecar, so the failure was permanent for that session.

The same loop also returned an error for the whole file when any single line
failed to unmarshal. The sidecar is appended to while Qwen is running, so a
torn final line is normal and must not discard the records already written.

Raise the line limit to 10 MB and skip unparseable lines, matching how the
other JSONL transcript scanners in this repo already read sessions
(kilo/internal/kilo/session_jsonl.go, omp/internal/omp/session.go).

Adds regression coverage for both cases; internal/qwen previously had no
transcript tests.
Qwen's sidecar was already JSONL and get-transcript-position already reported
the record count, so Entire's line-offset scoping
(transcript.SliceFromLine, cmd/entire/cli/transcript/parse.go:130) landed on
record boundaries. The line count was never the problem. The line SHAPE was:
measured against cli@main, qwen's transcript.jsonl came out at 0 bytes.

Entire compacts an external agent's transcript with the generic JSONL reader in
cmd/entire/cli/transcript/compact/compact.go — an adapter's own
compact-transcript method does not change that, because
checkpoint/persistent.go:1136 always runs the generic compactor over the raw
transcript. It keeps a line only when BOTH of these hold, and a sidecar record
failed both:

  - normalizeKind (compact.go:197) reads a TOP-LEVEL "type", falling back to
    "role". A sidecar record has neither: its discriminator is "event"
    ("UserPromptSubmit", "PostToolUse", ...), so every line was dropped.
  - parseMessage (compact.go:617) reads content from a TOP-LEVEL "message"
    OBJECT: "All JSONL agents nest content inside a 'message' object." The
    sidecar's "message" is a STRING — Qwen's notification text — so even a line
    that had passed the first check would have carried nothing.

Each record now keeps its native fields and gains a projection built from them:
a top-level "type", a "timestamp" under the key parseLine reads, and the
"message" wrapper.

  - UserPromptSubmit becomes a user text block
  - PreToolUse becomes an assistant tool_use with type/id/name/input, the only
    shape stripAssistantContent (compact.go:704) preserves
  - PostToolUse / PostToolUseFailure become a user tool_result with snake_case
    tool_use_id and a string "content" (compact.go:665), so Entire inlines the
    output into the preceding tool_use (inlineToolResults, compact.go:454)
  - Stop / StopFailure become an assistant text block
  - every other event stays unprojected and is dropped, as before

Qwen fires PreToolUse before a call and PostToolUse after it, so that pairing is
Claude Code's own tool_use/tool_result shape and each hook still costs exactly
one line, keeping position == line count.

"message" is now stored as raw JSON. That is load-bearing, not cosmetic: typed
as a string, a projected line fails to unmarshal in readSidecarRecords and takes
the entire sidecar read with it. Qwen's notification text is the only thing that
used the key, and it is only sent on events that are never projected, so it is
written exactly as before; a projected record that also carried one would keep
it as the first text block of its content.

Back-compat: the projection is derived data and is ignored on read, so a sidecar
written by an older build still parses and later hooks append projected records
alongside it.

Measured with a full hook sequence (session-start, notification, prompt,
pre-tool-use, post-tool-use, stop, session-end) driven through ParseHook and run
through cli@main's transcript.SliceFromLine + compact.FullWithBoundary:

  before  start=0  raw 1032B/7 lines  transcript.jsonl=0B
  after   start=0  raw 1674B/7 lines  transcript.jsonl=574B, 3 lines with content
  after   start=3  boundary=1, delta=439B

The compacted lines now read:

  {"type":"user","ts":"...02Z","content":[{"text":"Create hello.txt"}]}
  {"type":"assistant","ts":"...03Z","id":"tool-1","content":[{"id":"tool-1",
   "input":{"content":"hi","file_path":"hello.txt"},"name":"write_file",
   "result":{"output":"{\"ok\":true}","status":"success"},"type":"tool_use"}]}
  {"type":"assistant","ts":"...05Z","content":[{"text":"Created hello.txt",...}]}

Reverting only the message wrapper reproduces the other failure mode exactly:
468 bytes across 4 lines with every content array empty. A byte count would call
that fixed, so the tests assert the message TEXT, and they drive real hook
payloads through ParseHook rather than comparing against a checked-in constant.
@ashtom

ashtom commented Sep 9, 2026

Copy link
Copy Markdown
Member

Reviewed for functional and security issues. This projection doubles large hook payloads, making the old 64 KiB reader limit even easier to hit. I integrated the reviewed #65 branch, including the larger-record support and physical-line offset fix, so this PR works independently. The original projection remains intact. Module race tests and lint pass, including the >64 KiB and malformed-line regressions. Current main was merged for the shared CI toolchain fix. No unresolved finding in this PR; fresh CI has 36/36 successful checks and GitHub reports MERGEABLE/CLEAN. Prefer merging #65 before this PR to keep the review history clear.

@ashtom
ashtom merged commit d92a1a2 into main Sep 9, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants