feat: compress the searchable session archive - #148
Merged
Conversation
Freudator86
force-pushed
the
fm/fm-transcript-compress-archive
branch
from
August 19, 2026 22:08
41b5149 to
d70fe3f
Compare
added 18 commits
August 19, 2026 23:10
The store was 265 MB of plain text and growing daily. Each session is now written as one zstd file at level 3, which took this seat's archive from 271.2 MB to 52.1 MB, and the migration lost nothing: of the 2236 sessions present beforehand, 2144 came back byte-identical and 92 came back longer because the session had continued since the previous build. Level 3 rather than a higher one, measured on this seat's real store rather than on a sample: a full rebuild with verification costs 119.4 s at -3 and 159.6 s at -9, against 89.8 s plain, and -9 spends those 40 extra seconds of every rebuild to save 3.4 MB more - 1.2 percent of what -3 already saved. No index was added and none was needed. The scan still reads every byte of every session, now through ripgrep's -z, so nothing exists that could silently disagree with the content; a decompression that goes wrong is an error rather than a wrong answer. Searches over the whole compressed store measured 0.63 to 0.92 s on real queries, and --since and --cwd narrow the file set exactly as before. What compression would have broken, had the documentation been left alone, is the sentence promising that plain `grep -r` works identically. It does not: it matches nothing in a compressed session and exits reporting no matches over a full archive, which is the one answer this archive must never give - and it half answers, because _index.tsv is the one plain file left, so a phrase in its first-user-message column still matches. That promise is withdrawn everywhere it stood, and the raw-tool alternative that does work, `rg -z`, is named in its place. A rebuild that finds an older archive README still carrying the promise says so rather than overwriting what someone wrote there. The rest is refusal, in the shape this tool already used: - a missing compressor refuses the build instead of writing a store that is half compressed and half plain - a rebuild compresses the sessions the raw store no longer has, where they lie, so nothing is stranded plain - verification reads the store through the decompressor, and a file it cannot read is reported as unread rather than counted as clean - a missing search tool is reported as a missing tool, never as a search that found nothing - the exit status states whether the search matched, not how one batch of it ended
…ocumentation test
The compressed store was searched through ripgrep, and ripgrep turned out to
be the one part of this tool that answered differently depending on the
machine. Two CI rounds failed on the runner's build - which flags it accepted,
whether it could decompress at all - while every build and verification test
passed there. A search tool whose answer depends on the machine it runs on is
the same silent disagreement this archive exists to refuse, so the dependency
is removed rather than pinned to a version.
The scan now decompresses each session into grep, one file at a time, spread
across the machine's cores. Required tools are zstd, grep and xargs, and
nothing else; zstd is already needed to build the store, and the other two are
on any machine that can run this repository's tests. No override names any of
them, after a partial override twice passed its own prerequisite check before
the scan read nothing.
Measured on this seat's live 2273-session archive rather than on a fixture:
Grossreinschiff 75 of 75 matching sessions, exit 0
no-mistakes 2048 matching sessions, exit 0
whole-store scan 1.44 - 1.48 s
That is slower than the 0.63 - 0.92 s recorded for the ripgrep scan, and
docs/session-archive.md now carries the new number rather than the old one.
The defect that made this necessary is worth stating, because it survived
three fix rounds and every existing test: grep stops reading as soon as it can
answer, the decompressor feeding it then dies of a broken pipe, and treating
that ordinary event as a scanner error returned 0 matching sessions where 75
were expected - over a full archive, with the suite green. A one-session
fixture with one match is too small to make anything stop early, which is why
nothing caught it. The suite now plants that case: a dozen sessions with the
match at the top and a long tail behind it, requiring all twelve back.
Two further tests replace assertions that could not have caught this. The
proof that the search does not need ripgrep now runs it on a PATH where
ripgrep cannot be resolved at all, rather than stubbing a flag on a wrapper;
and the missing-tool refusal removes the decompressor from PATH rather than
pointing an override at a nonexistent path.
Everything else holds: 0 when the search matched, 1 when it genuinely matched
nothing, 2 for a usage error, a missing archive, a missing tool or a scanner
failure; --files-only and the context path agree; the session header still
prints with each hit; a plain .txt session left from an older build is still
searched; --since and --cwd still narrow the file set first; and no inverted
index exists anywhere.
CI caught one real assertion: a search matching every session exited 2. It is
the mirror of the defect this branch already fixed once - the old search
reported success while finding nothing, this one found everything and reported
failure - and both make the exit status lie to every caller that trusts it.
The cause was accepting only signal 141 from the decompressor when its
consumer stopped reading early. This machine's zstd dies on the signal; the CI
runner's reports the closed pipe as an ordinary error, so a full match became a
scanner failure there while passing here.
The rule is now stated rather than pattern-matched: the reader's exit status
means something only when the consumer read to the end. When grep reports a
match it deliberately stopped reading, so its reader's status says nothing
about the file and is not consulted. When grep read the session through and
found nothing, the reader had every chance to finish, so a non-zero status is a
genuine failure to read the store and stays an error. The status is not
masked - masking it would leave a search that can no longer report a real
failure at all - it is read only where it carries information.
tests/fm-transcript-archive.test.sh runs the real script against a reader that
hides the signal the way the runner's does, and that test fails against the
previous rule and passes against this one. A deliberately corrupt store file is
proven to still exit non-zero, on both search paths.
Verifying that fix on the live archive surfaced two further defects in the
parallel scan, both of which made the search disagree with itself rather than
merely with its exit status:
Parallel workers wrote into one shared pipe, and buffered writes interleave at
block boundaries rather than line boundaries. Three runs of one query returned
8121, 8105 and 8104 lines with header lines cut mid-path. Each worker now
writes its own file, which the parent gathers afterwards, so the same query
returns the same answer every time.
grep called five sessions binary and printed no lines for them, because a
reduced session quotes material that carries the occasional control byte.
--files-only reported those sessions and the context path did not. The scan
now reads the store as text.
Measured on the live 2273-session archive against ground truth taken by
decompressing every session separately:
no-mistakes 2048 sessions, 88294 matching lines exact
forgejo 123 sessions, 2306 matching lines exact
Grossreinschiff 75 sessions, 232 matching lines exact
exit code 127 1 session, 2 matching lines exact
three runs of one query identical
Whole-store timings move with the extra correctness: 1.50 - 1.54 s for the file
list and 2.31 - 2.64 s with context, and docs/session-archive.md carries those
rather than the earlier single figure.
The compressor-override mismatch is now recorded as a known condition where a
reader of FM_ZSTD meets it: it governs building and verifying, the search uses
the zstd on PATH, and the two have to be able to read each other.
Freudator86
force-pushed
the
fm/fm-transcript-compress-archive
branch
from
August 19, 2026 23:41
d70fe3f to
e1c828f
Compare
Freudator86
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Compress the searchable session archive without giving up the property that makes it trustworthy: a full content scan is the index, and no inverted index may be added. This branch carries that work; this run validates it after a CI failure the user steered on directly.
WHAT THE BRANCH DOES. Each session is written as one zstd file at level 3, chosen on measurement over this seat's real store: a full rebuild with verification costs 89.8 s plain, 119.4 s at -3 (52.1 MB), 159.6 s at -9 (48.7 MB), so -9 spends 40 more seconds of every rebuild for 1.2 percent more saving. The live archive was migrated 271.2 MB to 52.1 MB with no session lost. Verification runs over the compressed output through the decompressor, and a store file it cannot read is reported as unread rather than counted as clean. A rebuild converges retained sessions the raw store no longer has. A missing compressor refuses the build before creating anything. The documentation trap that motivated the task is closed: the old promise that plain 'grep -r' works identically is withdrawn everywhere, because over a compressed store that command reports no matches on a full archive.
THE SEARCH USES ZSTD AND GREP, NOT RIPGREP, by the user's decision after two CI rounds failed on the runner's ripgrep while every build and verification test passed there. Required tools are zstd, grep and xargs. Do not reintroduce ripgrep, an override, or an index.
WHAT THIS RUN'S COMMIT FIXES, all measured rather than reasoned:
A search matching EVERY session exited 2. The scan accepted only signal 141 from the decompressor when grep stopped reading early; the runner's zstd reports that closed pipe as an ordinary error instead. The rule is now that the reader's exit status means something only when the consumer read to the end: grep reporting a match means it deliberately stopped reading, so the reader's status is not consulted; grep reading the session through and finding nothing means the reader had its chance, so a non-zero status is a genuine read failure and stays an error. The status is deliberately NOT masked - masking it would leave a search that can never report a real failure - and a corrupt store file is proven to still exit non-zero on both paths. A test runs the real script against a reader that hides the signal like the runner's; it fails against the old rule and passes against the new one.
Parallel workers wrote into one shared pipe, and buffered writes interleave at block boundaries rather than line boundaries. Three runs of one query returned 8121, 8105 and 8104 lines, with header lines cut mid-path. Each worker now writes its own file which the parent gathers, so a query returns the same answer every time.
grep called five sessions binary and printed no lines for them, because a reduced session quotes material carrying the occasional control byte, so --files-only reported those sessions and the context path did not. The scan now reads the store as text.
MEASURED AGAINST GROUND TRUTH taken by decompressing every session separately, because a passing test is not the deliverable here - earlier rounds certified a search that found nothing: no-mistakes 2048 sessions and 88294 matching lines, forgejo 123 and 2306, Grossreinschiff 75 and 232, exit code 127 one session and 2 lines, all exact, and three runs of one query identical. Whole-store timings are 1.50 to 1.54 s for the file list and 2.31 to 2.64 s with context, and the documentation carries those figures rather than the earlier single one.
FM_ZSTD stays on the reducer by the user's explicit decision, because it is the only way to build a store on a machine without zstd on PATH; the search always uses the zstd on PATH. That split is recorded as a known condition where a reader of the variable meets it, which is the disposition the user chose over extending the withdrawal.
CONSTRAINTS. Shared tracked firstmate material, so firstmate-coding-guidelines applies and the colocated tests move with the change: 37 tests pass and bin/fm-lint.sh is clean. No credential value, pattern literal or transcript content in any commit, test or document. The repository forbids squash, so this lands as a true merge commit. Exit statuses are 0 matched, 1 genuinely no match, 2 usage error or missing archive or missing tool or scanner failure.
What Changed
Risk Assessment
✅ Low: The compression and parallel-search changes are well bounded, and the current code consistently fails closed across reader, worker, gathering, and output-pipeline failures without contradicting the stated intent.
Testing
The focused archive suite passed, and an end-to-end 97-session compressed-store scan exactly matched independent decompression ground truth, remained byte-identical across three parallel runs, preserved a control-byte match in context output, and rejected corrupt input with exit 2 on both query paths.
Evidence: End-to-end compressed archive search evidence
Ground truth: 97 files / 97 matching lines. Three parallel searches produced identical hashes. Context search returned all 97 sessions, including the control-byte session. Corrupt input exited 2 for matching and non-matching queries.Evidence: Rendered context-search output
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed (4) ✅
bin/fm-transcript-search.sh:195---files-onlyignores every reader failure once grep has seen a match. A truncated.zstthat decompresses a matching prefix and then fails therefore prints the path and exits 0, contradicting the required guarantee that a corrupt store file exits non-zero on both search paths. Make this path consume the full stream (for example, avoidgrep -qand discard grep's output) so the reader status is meaningful, then reject any non-zero reader status.🔧 Fix: Fail closed on corrupt matching sessions
1 warning still open:
docs/session-archive.md:67- The documentation still says the scan stops after its first match and that the regression fixture exercises this behavior, but both search paths now intentionally consume every session completely. Update this historical explanation to describe the former failure and the new full-read invariant; otherwise it directly contradicts the implementation and its header documentation.🔧 Fix: Document full-read corruption detection invariant
1 error still open:
bin/fm-transcript-search.sh:215- The final output pipelines return only awk's status (PIPESTATUS[2]), masking failures fromfindorsort. If either producer fails after emitting partial data, awk can still see a match and return 0, so the search reports a successful but incomplete answer, violating the exit-status contract that scanner failures return 2. Capture all pipeline statuses and return 2 whenfindorsortis non-zero, while preserving awk's 0/1 match result.🔧 Fix: Fail closed on output pipeline errors
1 error still open:
bin/fm-transcript-search.sh:150- File-set gathering still masks producer failures.finderrors are redirected and ignored, while the indexed path likewise continues after a failedawk/reader pipeline; either can leave a partial file list that is then reported as a successful search or genuine no-match. Capture each gathering pipeline’s status and exit 2 before scanning when it is non-zero, preserving the required scanner-failure contract.🔧 Fix: Fail closed on file gathering errors
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-transcript-archive.test.shCreated 97 independently zstd-compressed synthetic sessions, including one control-byte line, then established ground truth using per-sessionzstd -dcq | grep -aE.RanFM_TRANSCRIPT_ARCHIVE="$DEMO" ./bin/fm-transcript-search.sh 'fleet needle' --files-onlythree times and compared outputs withcmpandsha256sum.RanFM_TRANSCRIPT_ARCHIVE="$DEMO" ./bin/fm-transcript-search.sh 'fleet needle' -C 0and compared session/header and matching-line counts with decompressed ground truth.Inserted an invalid.txt.zstsession and exercised both matching and no-match--files-onlysearches, confirming scanner-error exit status 2.docs/session-archive.md:68- The documentation says both search paths always consume each session fully, contradicting the authoritative contract that a matching files-only grep may stop early and its reader status must then be ignored. The implementation also unconditionally checks that status, so this cannot be resolved with documentation-only edits.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.