From 3c7cceefeb26e8f6186a9893da7fd5750e52946f Mon Sep 17 00:00:00 2001 From: coditan Date: Tue, 18 Aug 2026 22:19:17 +0000 Subject: [PATCH 01/18] Compress the session archive and stop promising plain grep The store was 265 MB of plain text and growing daily. Each session is now written as one zstd file at level 3, which took this seat's archive from 271.2 MB to 52.1 MB, and the migration lost nothing: of the 2236 sessions present beforehand, 2144 came back byte-identical and 92 came back longer because the session had continued since the previous build. Level 3 rather than a higher one, measured on this seat's real store rather than on a sample: a full rebuild with verification costs 119.4 s at -3 and 159.6 s at -9, against 89.8 s plain, and -9 spends those 40 extra seconds of every rebuild to save 3.4 MB more - 1.2 percent of what -3 already saved. No index was added and none was needed. The scan still reads every byte of every session, now through ripgrep's -z, so nothing exists that could silently disagree with the content; a decompression that goes wrong is an error rather than a wrong answer. Searches over the whole compressed store measured 0.63 to 0.92 s on real queries, and --since and --cwd narrow the file set exactly as before. What compression would have broken, had the documentation been left alone, is the sentence promising that plain `grep -r` works identically. It does not: it matches nothing in a compressed session and exits reporting no matches over a full archive, which is the one answer this archive must never give - and it half answers, because _index.tsv is the one plain file left, so a phrase in its first-user-message column still matches. That promise is withdrawn everywhere it stood, and the raw-tool alternative that does work, `rg -z`, is named in its place. A rebuild that finds an older archive README still carrying the promise says so rather than overwriting what someone wrote there. The rest is refusal, in the shape this tool already used: - a missing compressor refuses the build instead of writing a store that is half compressed and half plain - a rebuild compresses the sessions the raw store no longer has, where they lie, so nothing is stranded plain - verification reads the store through the decompressor, and a file it cannot read is reported as unread rather than counted as clean - a missing search tool is reported as a missing tool, never as a search that found nothing - the exit status states whether the search matched, not how one batch of it ended --- README.md | 2 +- bin/fm-transcript-reduce.py | 159 +++++++++++++++++++-- bin/fm-transcript-refresh.sh | 36 ++++- bin/fm-transcript-search.sh | 83 ++++++++--- docs/scripts.md | 6 +- docs/session-archive.md | 77 +++++++--- tests/fm-transcript-archive.test.sh | 214 +++++++++++++++++++++++++++- 7 files changed, 517 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index 3ea4cabee2a..41360da172a 100644 --- a/README.md +++ b/README.md @@ -241,7 +241,7 @@ Firstmate's skills live in two separate places with different audiences: - [docs/turnend-guard.md](docs/turnend-guard.md) - the primary session's structural "no turn ends blind" backstop: verified per-harness hook mechanisms, scoping, loop safety, and fail-open tradeoffs. - [docs/context-reset.md](docs/context-reset.md) - the stow-then-clear context ceiling: what the watcher measures, when it resets, asks, blocks, or reports itself unenforced, and every refusal the reset tool makes. - [docs/wake-delivery.md](docs/wake-delivery.md) - how a queued wake becomes a model turn: the external listener, why no session holds a delivery object, and the verdict that keeps a dead listener from looking like a quiet fleet. -- [docs/session-archive.md](docs/session-archive.md) - the searchable session archive: a reduced, redacted, per-vessel derivative of this machine's own transcripts, why grep is the index, why a wrong reader is made loud, and the honest bound that travels with every claim made from it. +- [docs/session-archive.md](docs/session-archive.md) - the searchable session archive: a reduced, redacted, compressed, per-vessel derivative of this machine's own transcripts, why a full content scan is the index, why a wrong reader is made loud, and the honest bound that travels with every claim made from it. - [docs/supervision-protocols/](docs/supervision-protocols/) - rendered primary-harness supervision protocols for Claude, Codex, OpenCode, Pi, Grok, and unknown harness fallback. - [docs/supervision-cost.md](docs/supervision-cost.md) - what supervision costs in freshly written tokens, measured from provider usage records with `bin/fm-supervision-cost.sh`, plus the before-and-after for three repairs and what the measurement does not cover. - [docs/scripts.md](docs/scripts.md) - the `bin/` toolbelt reference. diff --git a/bin/fm-transcript-reduce.py b/bin/fm-transcript-reduce.py index a4209e39ca9..8ea49927de1 100755 --- a/bin/fm-transcript-reduce.py +++ b/bin/fm-transcript-reduce.py @@ -35,18 +35,33 @@ --source, and it refuses again afterwards if a whole run produced no entries or found no input files. An empty archive is never a silent success here. +THE STORE IS COMPRESSED, AND A FULL CONTENT SCAN IS STILL THE INDEX. Each +session is written as one zstd-compressed file, `.txt.zst`, because +ripgrep reads zstd directly with `-z`: the whole store is scanned by content on +every search and no second artefact exists that could disagree with it. An +inverted index is still forbidden here, for the reason it always was - it can go +stale silently. +Compression cannot, because a wrong decompression is an error and not a wrong +answer. + +The compressor is the `zstd` binary (FM_ZSTD overrides it). It is a hard +requirement rather than a preference: a run that cannot compress refuses instead +of leaving a store that is half compressed and half plain, which is exactly the +kind of quiet disagreement this archive exists to avoid. + Usage: fm-transcript-reduce.py --source claude|codex --in DIR --out DIR [--patterns FILE] [--truncate 400] [--limit N] - [--fold-injected] [--quiet] + [--level 3] [--fold-injected] [--quiet] fm-transcript-reduce.py --verify-only --out DIR [--patterns FILE] Exit status: 0 built (or verified with zero residual hits) 2 verification found residual hits, or the input shape disagrees with --source 3 nothing was read: no input files, or every file yielded zero entries + 4 the compressor is missing, or it failed on a file """ -import argparse, json, os, re, sys, time +import argparse, json, os, re, shutil, subprocess, sys, time from collections import Counter # ---------------------------------------------------------------- patterns @@ -127,6 +142,100 @@ def scan(text, pats): return hits +# ---------------------------------------------------------------- storage + +# One session, one compressed file. The extension is what makes the store +# searchable without a wrapper: ripgrep decides how to read a file from it, so +# `.txt.zst` is read as text and a plain `.txt` still left in the store is too. +SUFFIX = '.txt.zst' +PLAIN_SUFFIX = '.txt' +DEFAULT_LEVEL = 3 +ZSTD = os.environ.get('FM_ZSTD', 'zstd') + + +def compressor_path(): + """The zstd binary, or None. Named so the refusal can say what to install.""" + return shutil.which(ZSTD) + + +def require_compressor(): + if compressor_path(): + return + print('ERROR: the compressor %r is not on PATH, so this run cannot write the archive.' + % ZSTD, file=sys.stderr) + print(' Install zstd (apt install zstd), or point FM_ZSTD at the binary.', + file=sys.stderr) + print(' Refusing rather than writing a store that is half compressed and half plain.', + file=sys.stderr) + raise SystemExit(4) + + +def write_session(path, text, level): + """Write one session file, compressed, atomically. + + Through a temporary file and a rename, because the alternative - a half + written session that still has a plausible name - is unreadable material + that looks like readable material. + """ + tmp = path + '.tmp' + r = subprocess.run([ZSTD, '-q', '-f', '-%d' % level, '-o', tmp, '-'], + input=text.encode('utf-8')) + if r.returncode != 0: + try: + os.remove(tmp) + except OSError: + pass + print('ERROR: %s failed with status %d writing %s' + % (ZSTD, r.returncode, path), file=sys.stderr) + raise SystemExit(4) + os.replace(tmp, path) + + +def read_session(path): + """Read one session file back, compressed or plain.""" + if path.endswith('.zst'): + r = subprocess.run([ZSTD, '-dcq', '--', path], capture_output=True) + if r.returncode != 0: + raise OSError('%s could not decompress %s' % (ZSTD, path)) + return r.stdout.decode('utf-8', errors='replace') + with open(path, encoding='utf-8', errors='replace') as fh: + return fh.read() + + +def intact(path): + """True when zstd can decompress the whole file. A truncated store file is + not evidence of anything, so it is never the copy that survives.""" + return subprocess.run([ZSTD, '-t', '-q', '--', path], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL).returncode == 0 + + +def converge_store(outdir, level): + """Leave every session in the store as exactly one compressed file. + + Run after a build, so a store never ends up half one thing and half the + other: sessions just rewritten drop their superseded plain copy, and + sessions the raw store no longer has - which a rebuild cannot reach and must + not remove - are compressed where they lie. Returns (compressed, dropped). + """ + compressed = dropped = 0 + for root, _, names in os.walk(outdir): + for n in sorted(names): + if not n.endswith(PLAIN_SUFFIX): + continue + plain = os.path.join(root, n) + zst = plain + '.zst' + if os.path.exists(zst) and intact(zst): + os.remove(plain) + dropped += 1 + continue + with open(plain, encoding='utf-8', errors='replace') as fh: + write_session(zst, fh.read(), level) + os.remove(plain) + compressed += 1 + return compressed, dropped + + # ---------------------------------------------------------------- shape # The two record shapes share no key, which is exactly why a wrong --source is @@ -476,6 +585,9 @@ def main(): ap.add_argument('--patterns', default=DEFAULT_PATTERNS) ap.add_argument('--truncate', type=int, default=400) ap.add_argument('--limit', type=int, default=0) + ap.add_argument('--level', type=int, default=DEFAULT_LEVEL, + help='zstd compression level (default %d); measured on this ' + 'fleet in docs/session-archive.md' % DEFAULT_LEVEL) ap.add_argument('--fold-injected', action='store_true', help='codex only: fold machine-injected user messages listed in ' 'injected-prefixes.txt down to their marker and their tail') @@ -496,17 +608,31 @@ def main(): if a.verify_only: total = Counter() files = 0 + unreadable = [] for root, _, names in os.walk(a.outdir): for n in sorted(names): - if not n.endswith('.txt'): + if not (n.endswith(SUFFIX) or n.endswith(PLAIN_SUFFIX)): continue - files += 1 p = os.path.join(root, n) - with open(p, encoding='utf-8', errors='replace') as fh: - h = scan(fh.read(), pats) + # Verification reads the store the way a search reads it, through + # the decompressor. A file it cannot read is named as unread and + # never counted as a file that came back clean. + try: + body = read_session(p) + except OSError as e: + unreadable.append('%s: %s' % (p, e)) + continue + files += 1 + h = scan(body, pats) if h: print('HIT %s %s' % (p, dict(h))) total.update(h) + if unreadable: + print('ERROR: %d file(s) under %s could not be read, so they were not ' + 'verified:' % (len(unreadable), a.outdir), file=sys.stderr) + for u in unreadable[:5]: + print(' %s' % u, file=sys.stderr) + return 2 if not files: # Zero hits over zero files is not an all-clear, it is a reading the # detector could not take. Reporting it as clean is the same silent @@ -530,6 +656,8 @@ def main(): for n in names: if n.startswith('rollout-') and n.endswith('.jsonl'): files.append(os.path.join(root, n)) + require_compressor() + files.sort() if a.limit: files = files[:a.limit] @@ -544,7 +672,7 @@ def main(): os.makedirs(a.outdir, exist_ok=True) counter = Counter() - bytes_in = bytes_out = 0 + bytes_in = bytes_out = bytes_disk = 0 total_entries = 0 index = [] t0 = time.time() @@ -561,12 +689,12 @@ def main(): continue total_entries += len(entries) text = emit(meta, entries, pats, counter, a.source, rel) - outrel = rel[:-len('.jsonl')] + '.txt' + outrel = rel[:-len('.jsonl')] + SUFFIX outp = os.path.join(a.outdir, outrel) os.makedirs(os.path.dirname(outp), exist_ok=True) - with open(outp, 'w', encoding='utf-8') as fh: - fh.write(text) + write_session(outp, text, a.level) bytes_out += len(text.encode('utf-8')) + bytes_disk += os.path.getsize(outp) first_user = '' for k, _, lab, b in entries: if k == 'msg' and lab.startswith('user') and str(b).strip(): @@ -579,6 +707,11 @@ def main(): if not a.quiet and i % 100 == 0: print(' %d/%d %.0fs' % (i, len(files), time.time() - t0), file=sys.stderr) + conv, dropped = converge_store(a.outdir, a.level) + if conv or dropped: + print('converged %d retained session(s) compressed, %d superseded plain ' + 'copy(ies) dropped' % (conv, dropped)) + with open(os.path.join(a.outdir, '_index.tsv'), 'w', encoding='utf-8') as fh: fh.write('# path\tfirst\tlast\tcwd\tentries\tfirst_user_message\n') fh.write('\n'.join(sorted(index)) + '\n') @@ -590,9 +723,13 @@ def main(): print('shape undecided %d (files whose first records identify neither shape)' % undecided) print('raw bytes %d (%.1f MB)' % (bytes_in, bytes_in / 1048576)) - print('derivative %d (%.1f MB)' % (bytes_out, bytes_out / 1048576)) + print('derivative %d (%.1f MB) text as searched' % (bytes_out, bytes_out / 1048576)) + print('on disk %d (%.1f MB) zstd -%d' % (bytes_disk, bytes_disk / 1048576, a.level)) if bytes_out: print('reduction %.1f:1' % (bytes_in / bytes_out)) + if bytes_disk: + print('compression %.1f:1 (%.1f:1 against the raw store)' + % (bytes_out / bytes_disk, bytes_in / bytes_disk)) print('redactions %d' % sum(counter.values())) for k, v in counter.most_common(): print(' %-24s %d' % (k, v)) diff --git a/bin/fm-transcript-refresh.sh b/bin/fm-transcript-refresh.sh index 31cf550586e..e19747c9680 100755 --- a/bin/fm-transcript-refresh.sh +++ b/bin/fm-transcript-refresh.sh @@ -7,8 +7,15 @@ # The archive is rebuilt from the raw stores every time, then the detector is # re-run against the output and required to return zero. # +# The store is written compressed, one zstd file per session, and is searched +# through the decompressor by bin/fm-transcript-search.sh. `zstd` is therefore a +# hard requirement of a rebuild: the reducer refuses rather than leaving a store +# that is half compressed and half plain. A rebuild also compresses whatever +# plain session files it finds already in the store, so an archive built before +# compression converges on the first refresh instead of being left behind. +# # Usage: -# fm-transcript-refresh.sh [--fold-injected] [--limit N] +# fm-transcript-refresh.sh [--fold-injected] [--limit N] [--level N] # # --fold-injected codex only: fold machine-injected user messages down to # their marker and their tail. It shrinks the Codex store by @@ -16,6 +23,8 @@ # findings with it, because less of the material was ever # looked at. Off by default for that reason. # --limit N reduce only the first N files of each store (smoke runs). +# --level N zstd compression level; the default and the measurement +# behind it are in docs/session-archive.md. # # Paths, all overridable so this runs on a vessel that is not the one it was # written on: @@ -44,11 +53,13 @@ CODEX_IN="${FM_CODEX_SESSIONS:-$HOME/.codex/sessions}" fold=() limit=() +level=() while [ $# -gt 0 ]; do case "$1" in --fold-injected) fold=(--fold-injected); shift;; --limit) limit=(--limit "${2:?--limit needs a number}"); shift 2;; - -h|--help) sed -n '2,31p' "$0" | sed 's/^# \?//'; exit 0;; + --level) level=(--level "${2:?--level needs a number}"); shift 2;; + -h|--help) sed -n '2,41p' "$0" | sed 's/^# \?//'; exit 0;; *) echo "unknown argument: $1" >&2; exit 2;; esac done @@ -67,7 +78,8 @@ for s in claude codex; do fi echo "== reducing $s from $in_dir" python3 "$REDUCE" --source "$s" --in "$in_dir" --out "$ARCHIVE/$s-redacted" \ - --quiet "${extra[@]+"${extra[@]}"}" "${limit[@]+"${limit[@]}"}" + --quiet "${extra[@]+"${extra[@]}"}" "${limit[@]+"${limit[@]}"}" \ + "${level[@]+"${level[@]}"}" built=1 done @@ -121,7 +133,14 @@ Every claim made from this archive travels with both bounds. bin/fm-transcript-search.sh 'pattern' --since 2026-08-15 --cwd myproject bin/fm-transcript-search.sh 'pattern' --files-only -Plain `grep -r` works too - grep is the index, and there is no index to go stale. +The sessions are zstd-compressed, one file each. **Plain `grep -r` does not read +them: it finds nothing here and says so as if the archive were empty.** Use the +wrapper, or ripgrep's own decompressing search if you want the raw tool: + + rg -z 'pattern' . + +A full scan of the content is still what answers every query, so there is no +index here and none is to be added. ## Rebuild @@ -130,4 +149,13 @@ Plain `grep -r` works too - grep is the index, and there is no index to go stale Full detail: `docs/session-archive.md` in the firstmate repository. EOF echo "== wrote $ARCHIVE/README.md (the honest bound, on the artefact)" +elif grep -qE 'Plain .?grep -r' "$ARCHIVE/README.md" 2>/dev/null && + ! grep -q 'rg -z' "$ARCHIVE/README.md" 2>/dev/null; then + # This README predates compression and still tells its reader that plain + # grep works here. It does not: it returns nothing, with no error, over a + # full archive. The file is not overwritten because someone may have written + # into it, so the correction is named instead of made. + echo "WARNING: $ARCHIVE/README.md still points readers at plain grep -r, which reads" >&2 + echo " nothing from a compressed store and reports no error while doing it." >&2 + echo " Correct that sentence: the wrapper, or rg -z, is what searches this archive." >&2 fi diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index f8806566dc3..04d048463ef 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -8,21 +8,28 @@ # Prints, per hit: the session's working directory and time span, the matching # line with n lines of context, and the derivative path. # -# GREP IS THE INDEX. The archive is plain UTF-8 text laid out one file per -# session, and a full-content scan of the whole store costs about a fifth of a -# second, so no inverted index exists and none should be added: a search index -# is a component that can be silently out of date, which is the exact failure -# this archive was built against. `_index.tsv` is not a search index - it -# narrows the FILE SET before grep runs, and only when --since or --cwd asks it -# to. Plain `grep -r` over the archive works identically for anyone who does not -# want this wrapper. +# A FULL CONTENT SCAN IS THE INDEX. The archive is UTF-8 text laid out one file +# per session, zstd-compressed, and a scan of the whole store costs well under a +# second, so no inverted index exists and none should be added: a search index is +# a component that can be silently out of date, which is the exact failure this +# archive was built against. `_index.tsv` is not a search index - it narrows the +# FILE SET before the scan runs, and only when --since or --cwd asks it to. +# +# THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches +# nothing here and exits as though the archive were empty, which is the one +# answer this archive must never give. The scan therefore runs through ripgrep's +# `-z`, and `rg` is a hard requirement: a missing one is reported as a missing +# tool, never as a search that found nothing. Anyone who would rather not use +# this wrapper runs `rg -z ` over the archive directory directly and +# gets the same content scan. FM_RG and FM_ZSTD name the two binaries on a +# vessel where they sit elsewhere. # # The archive is this home's private material and never travels: it resolves # under $FM_HOME/data/transcripts, so a secondmate home searches its own store # and no vessel reads another's. FM_TRANSCRIPT_ARCHIVE overrides the location. # -# Exit status is grep's: 0 when something matched, 1 when nothing did, 2 on a -# usage error or an archive that is not there. +# Exit status: 0 when something matched, 1 when nothing did, 2 on a usage error, +# a missing archive, or a missing decompressing search tool. set -uo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -77,19 +84,45 @@ if [ "$present" -eq 0 ]; then exit 2 fi +# The searcher is named before anything else is done, because "no matches" and +# "the tool that reads this store is not installed" are the same output to a +# caller and only one of them is true. +RG="${FM_RG:-rg}" +ZSTD="${FM_ZSTD:-zstd}" +for tool in "$RG" "$ZSTD"; do + command -v "$tool" >/dev/null 2>&1 && continue + echo "$tool is not installed, and the session store is compressed: this search cannot run." >&2 + echo "Install ripgrep and zstd (apt install ripgrep zstd), or point FM_RG and FM_ZSTD at them." >&2 + echo "Refusing rather than reporting no matches over an archive that was never read." >&2 + exit 2 +done +# A user ripgrep config can change what a search means - case folding, column +# limits, skipped files - and this store's answers must not depend on it. +export RIPGREP_CONFIG_PATH= + # narrow the file set from the per-source index when asked filelist="$(mktemp)"; trap 'rm -f "$filelist"' EXIT for r in "${roots[@]}"; do [ -d "$r" ] || continue idx="$r/_index.tsv" if [ -f "$idx" ] && { [ -n "$since" ] || [ -n "$cwd" ]; }; then + # An index written before the store was compressed names the plain file, so + # the compressed sibling is accepted for the same session: a stale name must + # narrow to the session it meant, not to nothing. awk -F'\t' -v r="$r" -v since="$since" -v cwd="$cwd" ' /^#/ {next} { if (since != "" && substr($2,1,10) < since) next if (cwd != "" && index($4, cwd) == 0) next - print r "/" $1 }' "$idx" >> "$filelist" + print r "/" $1 }' "$idx" | + while IFS= read -r f; do + if [ -f "$f" ]; then printf '%s\n' "$f" + elif [ -f "$f.zst" ]; then printf '%s\n' "$f.zst" + elif [ "${f%.zst}" != "$f" ] && [ -f "${f%.zst}" ]; then printf '%s\n' "${f%.zst}" + fi + done >> "$filelist" else - find "$r" -name '*.txt' -type f >> "$filelist" 2>/dev/null + # Both shapes are listed: a store mid-migration must not go half unsearched. + find "$r" \( -name '*.txt.zst' -o -name '*.txt' \) -type f >> "$filelist" 2>/dev/null fi done @@ -97,23 +130,37 @@ n=$(wc -l < "$filelist") echo "# searching $n session files under: ${roots[*]}" >&2 [ "$n" -gt 0 ] || exit 1 +# The file set is handed over in batches, so a batch with no hit exits non-zero +# while another batch matched. What the caller is told is whether the SEARCH +# matched, decided by what came out of it, not by the status of one batch. if [ "$files_only" = 1 ]; then - xargs -a "$filelist" -d '\n' grep -lE -- "$q" 2>/dev/null - exit $? + xargs -a "$filelist" -d '\n' "$RG" -lz --no-messages -e "$q" -- 2>/dev/null | + awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' + exit "${PIPESTATUS[1]}" fi -xargs -a "$filelist" -d '\n' grep -nHZE -C "$ctx" --color=never -- "$q" 2>/dev/null | -awk -F'\0' -v arch="$ARCHIVE" ' +xargs -a "$filelist" -d '\n' "$RG" -z -n -H --null --no-heading --color never \ + --no-messages -C "$ctx" -e "$q" -- 2>/dev/null | +awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' /^--$/ { print " --"; next } NF < 2 { next } { path=$1; rest=$2 if (path != last) { - cmd="sed -n \"1,8p\" \"" path "\" | grep -E \"^# (cwd|span)\" | sed \"s/^# *//\" | tr \"\\n\" \"|\"" + # The session header is read the same way the store is: through the + # decompressor when the file is compressed, plainly when it is not. + if (path ~ /\.zst$/) + cmd=zstd " -dcq -- \"" path "\" | sed -n \"1,8p\"" + else + cmd="sed -n \"1,8p\" \"" path "\"" + cmd = cmd " | grep -E \"^# (cwd|span)\" | sed \"s/^# *//\" | tr \"\\n\" \"|\"" hdr=""; cmd | getline hdr; close(cmd) short=path; sub(arch "/", "", short) printf "\n=== %s\n %s\n", short, hdr last=path + found=1 } print " " rest - }' + } + END { exit(found ? 0 : 1) }' +exit "${PIPESTATUS[1]}" diff --git a/docs/scripts.md b/docs/scripts.md index fe34b54c125..b119c60e853 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -47,9 +47,9 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-memory-reading.sh` | Name which process is running away with this machine's memory, by size and by growth, tied to its account and to its task when known, alongside headroom and the kernel's stall reading; it sets no limit and kills nothing, and it never reports an input it could not read as a healthy zero (docs/memory-attribution.md) | | `fm-memory-alarm.sh` | Wake the fleet when this machine is running out of RAM headroom, on headroom and on growth, naming the process responsible with its account and the work it serves; it reads the attribution reading and nothing else, sets no limit and kills nothing, and reports an instrument it could not read as blindness rather than an all-clear (docs/memory-alarm.md) | | `fm-memory-ceiling-probe.sh` | Measure whether a memory ceiling on this host would manufacture the very pressure an alarm above it exists to detect, by running the same file-reading workload with and without one; it sets no lasting limit and kills nothing (docs/memory-ceiling-caveat.md) | -| `fm-transcript-reduce.py` | Build this home's reduced, redacted session derivative with one of two readers selected by `--source`, or re-verify an existing one to zero; refuses a source that disagrees with the material's shape rather than writing the empty archive a wrong reader would produce (docs/session-archive.md) | -| `fm-transcript-search.sh` | Search that derivative with context, narrowing the file set by date and working directory first; grep is the index, so there is none to go stale, and an archive that is not there reads as absent rather than as no matches (docs/session-archive.md) | -| `fm-transcript-refresh.sh` | Rebuild both stores from this vessel's own raw transcripts, verify the output to zero, and leave the honest bound on the artefact; the raw stores are read-only inputs and a store this vessel does not have is skipped, never counted as empty (docs/session-archive.md) | +| `fm-transcript-reduce.py` | Build this home's reduced, redacted session derivative with one of two readers selected by `--source`, writing one compressed file per session, or re-verify an existing one to zero through the decompressor; refuses a source that disagrees with the material's shape, or a missing compressor, rather than writing the empty or half-compressed archive either would produce (docs/session-archive.md) | +| `fm-transcript-search.sh` | Search that derivative with context, narrowing the file set by date and working directory first; a full content scan is the index, so there is none to go stale, the compressed store is read through ripgrep's decompressor rather than by plain grep, and an archive or a search tool that is not there reads as absent rather than as no matches (docs/session-archive.md) | +| `fm-transcript-refresh.sh` | Rebuild both stores from this vessel's own raw transcripts, converge any plain session files left from an older build into the compressed store, verify the output to zero, and leave the honest bound on the artefact; the raw stores are read-only inputs and a store this vessel does not have is skipped, never counted as empty (docs/session-archive.md) | | `fm-herdr-lab.sh` | Provision and guardedly operate an isolated, never-default Herdr lab session | | `fm-install-herdr.sh` | Install CI's exact-version Herdr pin with official asset URL, SHA-256, and protocol checks | | `fm-install-treehouse.sh`| Install CI's exact-version Treehouse pin for real-Herdr E2E that needs spawn worktrees | diff --git a/docs/session-archive.md b/docs/session-archive.md index 8c4c415169d..146968e410d 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -11,13 +11,16 @@ The archive it builds is not shared and never becomes shared: it is a private de Two stores, one per source, under `$FM_HOME/data/transcripts/`: ``` -claude-redacted/ one .txt per session, mirroring ~/.claude/projects +claude-redacted/ one .txt.zst per session, mirroring ~/.claude/projects _index.tsv path, time span, working directory, entry count, first user message -codex-redacted/ one .txt per session, mirroring ~/.codex/sessions +codex-redacted/ one .txt.zst per session, mirroring ~/.codex/sessions _index.tsv path, time span, working directory, entry count, first user message ``` -Per session, one plain-text file holding a header, then: +Each session file is one zstd-compressed plain-text document. +`_index.tsv` is the one uncompressed file in a store, because it is read by an `awk` narrowing pass rather than by the content scan. + +Per session, one plain-text document - compressed on disk, and read back in full by every search - holding a header, then: - every user and assistant message, verbatim - every command issued, verbatim @@ -35,11 +38,23 @@ bin/fm-transcript-search.sh 'pattern' --since 2026-08-15 --cwd myproject bin/fm-transcript-search.sh 'pattern' --files-only ``` -**Grep is the index.** -The archive is UTF-8 text laid out one file per session, and a full-content scan of the whole store was measured at 0.22 s over 258 MB on 2026-08-18, so no inverted index exists and none should be added. +**A full content scan is the index.** +The archive is UTF-8 text laid out one file per session, and a scan of the whole store was measured at 0.63 to 0.92 s over the compressed 48.9 MB on 2026-08-18, so no inverted index exists and none should be added. An index is a component that can be silently out of date, which is exactly the failure this archive was built against. -`_index.tsv` is not a search index: it narrows the file set before grep runs, and only when `--since` or `--cwd` asks it to. -Plain `grep -r` over the archive works identically for anyone who does not want the wrapper. +Compression does not reintroduce that failure and is why it was worth doing: the compressed file is the content, not a summary of it, and a decompression that goes wrong is an error rather than a wrong answer. +`_index.tsv` is not a search index: it narrows the file set before the scan runs, and only when `--since` or `--cwd` asks it to. + +**Plain `grep -r` no longer reads this archive.** +It matches nothing in a compressed session and exits reporting no matches, over an archive that is full - the same silent emptiness the whole design refuses, arriving through the documentation instead of through the reader. +It is worse than wholly blind: `_index.tsv` is the one plain file left, so a phrase that happens to sit in its first-user-message column still matches, and the blindness looks selective rather than total. +Anyone who wants the raw tool rather than the wrapper uses ripgrep's own decompressing scan, which reads exactly what the wrapper reads: + +```sh +rg -z 'pattern' "$FM_HOME/data/transcripts" +``` + +`rg` and `zstd` are therefore hard requirements of a search here, and the wrapper reports a missing one as a missing tool rather than as a search that found nothing. +`FM_RG` and `FM_ZSTD` name the binaries on a vessel where they sit elsewhere. ## Rebuild @@ -47,8 +62,8 @@ Plain `grep -r` over the archive works identically for anyone who does not want bin/fm-transcript-refresh.sh ``` -It rebuilds each raw transcript store present on the vessel and then re-runs the detector against each resulting derivative, requiring zero hits. -There is no incremental path and no build state, because a full rebuild of a two-thousand-session store costs about 95 s and a build state is one more thing that can be quietly wrong. +It rebuilds each raw transcript store present on the vessel and then re-runs the detector against each resulting derivative - reading it back through the decompressor, as a search does - and requires zero hits. +There is no incremental path and no build state, because a full rebuild of a two-thousand-session store costs about two minutes and a build state is one more thing that can be quietly wrong. The archive retains sessions the raw store no longer has: a rebuild rewrites every session it can still read and removes nothing, so a session deleted, rotated away, or renamed under `~/.claude` or `~/.codex` keeps its reduced copy in the archive. This is intended rather than a defect, because the archive exists precisely so that what was said survives the clearing of the session that said it, and outliving the raw store is the point. The archive therefore does not mirror a deletion, so removing material from the archive is a deliberate separate act and never a side effect of a rebuild. @@ -56,6 +71,16 @@ The archive therefore does not mirror a deletion, so removing material from the The raw stores are read-only inputs; nothing under `~/.claude` or `~/.codex` is written, moved, or removed. A raw store that does not exist on this vessel is reported and skipped, never treated as a store that happened to be empty. +## Compression, and why it does not cost the property above + +Each session is written as one zstd file at level 3, which took this seat's store from 271.2 MB to 52.1 MB on 2026-08-18. +`zstd` is a hard requirement of a rebuild rather than a preference: a run that cannot compress refuses before writing anything, because a store that is half compressed and half plain is a store whose answers depend on which half a question lands in. +A rebuild also compresses whatever plain session files it finds already in the store and drops the plain copies it has just superseded, so an archive built before compression converges on the first refresh instead of being stranded, and a retained session the rebuild cannot reach is compressed where it lies rather than left behind. +The level is a knob, `--level`, and the measurement behind the default is below. + +The store stays honest under compression for one reason: the compressed file is the content rather than a summary of it, so nothing exists that could disagree with it, and a decompression that goes wrong is an error rather than a wrong answer. +A verification that cannot read a store file says so and fails, instead of counting a file nobody read as a file that came back clean. + ## The honest bound, which travels with every claim made from this archive > The derivative is verified: the same detector re-run against the output returns zero hits. @@ -104,16 +129,34 @@ Two consequences bind anyone who touches this tool: Recorded with their date because they are readings, not properties. ``` - raw derivative ratio redactions -Claude 1001.5 MB 115.6 MB 8.7:1 73 -Codex 413.5 MB 142.2 MB 2.9:1 37 -combined 1415.0 MB 257.8 MB 5.5:1 110 -sessions 2236 -full rebuild + verification ~95 s -full-content search over the store 0.22 s -verification residual hits 0 + raw derivative on disk ratio redactions +Claude 1007.6 MB 116.6 MB 27.2 MB 8.6:1 73 +Codex 418.7 MB 143.6 MB 21.7 MB 2.9:1 37 +combined 1426.3 MB 260.2 MB 48.9 MB 5.5:1 110 +sessions 2273 +full rebuild + verification ~129 s +full-content search over the store, three queries 0.63 - 0.92 s +verification residual hits 0 +``` + +`derivative` is the text a search reads; `on disk` is what the store costs after compression. +The whole archive directory, including the two indexes, went from 271.2 MB to 52.1 MB in the migration of 2026-08-18, and no session was lost to it: of the 2236 sessions present beforehand, 2144 came back byte-identical and 92 came back longer because the session had continued since the previous build. + +### Choosing the compression level + +Measured the same day, on the same seat's real store rather than on a sample, each a full rebuild with verification: + +``` + rebuild + verify store on disk against plain +plain (before) 89.8 s 273.6 MB 1.0x +zstd -3 119.4 s 52.1 MB 5.25x +zstd -9 159.6 s 48.7 MB 5.62x ``` +Level 3 is the default because level 9 spends 40 more seconds of every rebuild to save 3.4 MB, which is 1.2 percent of what was already saved. +The ratio is material-dependent in the same way the reduction is: these are readings from one seat's transcripts, not properties of zstd. +Search cost is not part of this tradeoff - zstd decompresses at roughly the same speed whatever level wrote the file - so the level buys rebuild time against disk and nothing else. + The ratio is material-dependent and is not a property of the tool: the same design measured 56:1 on a seat whose sessions averaged 17 MB, against 0.63 MB here. `--fold-injected` folds Codex's machine-injected user messages down to their marker and their tail, taking the Codex store from 142.2 MB to 44.1 MB. It is off by default, and the reason is the finding rather than the size: folding dropped 24 of the 37 redaction findings, including 16 of the 22 private-key blocks, because those sat inside the folded region. diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 2968a903c35..3ca4f598eca 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -14,7 +14,13 @@ # 2. The pattern file must not match itself. A detector whose patterns appear # in plain text writes those patterns into the next session transcript, and # every later scan then inflates on its own tooling. -# 3. THIS FILE must not match the pattern file either. Property 2 closes the +# 3. A DOCUMENTED SEARCH MUST NOT SILENTLY RETURN NOTHING. The store is +# compressed, and plain grep reads a compressed file as no matches with no +# error - the same silent emptiness as 1, arriving through the +# documentation instead of through the reader. So the search path is +# exercised over a compressed store here, and the documentation is scanned +# for the promise it used to make. +# 4. THIS FILE must not match the pattern file either. Property 2 closes the # pattern list; it does not close the fixtures. The first build of this tool # left five real redactions in its own session from fixtures typed in plain # form. Every credential-shaped fixture below is therefore assembled at run @@ -36,6 +42,20 @@ INJECTED="$ROOT/bin/fm-transcript-patterns/injected-prefixes.txt" TMP="" fm_test_tmproot TMP fm-transcript +# --- reading the store ------------------------------------------------------ + +# Every session in the store is one zstd file, so a test reads it the way a +# search does - through the decompressor - and never by opening it as text. +read_session() { + zstd -dcq -- "$1" +} + +# Session files in a store, both shapes, so a test that expects only one shape +# has to say so. +store_files() { + find "$1" \( -name '*.txt.zst' -o -name '*.txt' \) -type f | sort +} + # --- fixture builders ------------------------------------------------------- # A minimal Claude-shaped session: both conversation sides, one command, one @@ -163,7 +183,8 @@ test_claude_reader_reads_claude_material() { write_claude_session "$TMP/claude-in/proj" out=$(python3 "$REDUCE" --source claude --in "$TMP/claude-in" --out "$TMP/claude-out" 2>&1) || rc=$? [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail "claude reader failed on claude material"; } - body=$(cat "$TMP/claude-out/proj/sess.txt") + assert_present "$TMP/claude-out/proj/sess.txt.zst" 'the session must be written compressed' + body=$(read_session "$TMP/claude-out/proj/sess.txt.zst") assert_contains "$body" 'find the tugboat host' 'the user side must survive verbatim' assert_contains "$body" 'looking now' 'the assistant side must survive verbatim' assert_contains "$body" '$ ls -la /srv' 'the command must survive verbatim' @@ -177,7 +198,7 @@ test_codex_reader_reads_codex_material() { write_codex_session "$TMP/codex-in/2026/08/18" out=$(python3 "$REDUCE" --source codex --in "$TMP/codex-in" --out "$TMP/codex-out" 2>&1) || rc=$? [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail "codex reader failed on codex material"; } - body=$(cat "$TMP/codex-out/2026/08/18/rollout-2026-08-18T11-00-00-abc.txt") + body=$(read_session "$TMP/codex-out/2026/08/18/rollout-2026-08-18T11-00-00-abc.txt.zst") assert_contains "$body" 'where did the harbour report go' 'the user side must survive verbatim' assert_contains "$body" '$ grep -r harbour' 'the command must survive verbatim' assert_contains "$body" '# cwd /home/x/bravo' 'session_meta must supply the header' @@ -242,7 +263,7 @@ test_redactor_masks_a_synthetic_credential() { [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'the reducer failed on the credential fixture'; } assert_contains "$out" 'github-token' 'the run summary must count the prefix-anchored class' assert_contains "$out" 'assigned-secret' 'the run summary must count the context-anchored class' - body=$(cat "$TMP/creds-out/proj/creds.txt") + body=$(read_session "$TMP/creds-out/proj/creds.txt.zst") assert_contains "$body" 'REDACTED github-token' 'a removed value must leave a naming marker' case "$body" in *"$tok"*) fail 'the prefix-anchored value survived into the derivative' ;; @@ -261,10 +282,11 @@ test_verification_is_zero_on_clean_and_loud_on_dirty() { # A verification that can only ever say zero proves nothing, so plant one. mkdir -p "$TMP/dirty" tok=$(fixture_prefixed_token) - printf 'a line carrying %s in it\n' "$tok" >"$TMP/dirty/planted.txt" + printf 'a line carrying %s in it\n' "$tok" | + zstd -q -o "$TMP/dirty/planted.txt.zst" rc=0 out=$(python3 "$REDUCE" --verify-only --out "$TMP/dirty" 2>&1) || rc=$? - [ "$rc" -ne 0 ] || fail 'verification must fail on a derivative that still carries a value' + [ "$rc" -ne 0 ] || fail 'verification must fail on a compressed derivative that still carries a value' assert_contains "$out" 'HIT' 'a residual hit must be reported with its file' case "$out" in *"$tok"*) fail 'verification must report classes and counts, never the matched value' ;; @@ -287,6 +309,81 @@ test_verification_over_nothing_is_not_an_all_clear() { pass "verifying a directory with no derivative in it refuses instead of reporting zero" } +# --- the store is compressed, wholly ---------------------------------------- + +test_the_store_is_written_compressed_and_leaves_nothing_plain() { + local out rc=0 plain + write_claude_session "$TMP/zst-in/proj" + out=$(python3 "$REDUCE" --source claude --in "$TMP/zst-in" --out "$TMP/zst-out" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'the reducer failed writing a compressed store'; } + assert_present "$TMP/zst-out/proj/sess.txt.zst" 'the session must land as one compressed file' + assert_absent "$TMP/zst-out/proj/sess.txt" 'no plain copy of a written session may be left behind' + plain=$(find "$TMP/zst-out" -name '*.txt' -type f | wc -l) + [ "$plain" -eq 0 ] || fail "the store must hold no plain session files, found $plain" + assert_contains "$out" 'on disk' 'the summary must state what the store actually costs on disk' + assert_grep 'sess.txt.zst' "$TMP/zst-out/_index.tsv" \ + 'the index must name the file that exists, not the one that used to' + pass "a build writes one compressed file per session and leaves no plain copy" +} + +# The archive keeps sessions the raw store no longer has, and a rebuild cannot +# reach them. Left alone they would stay plain forever - half a compressed +# archive and half not - so a rebuild compresses them where they lie, without +# removing the one thing they carry, which is their content. +test_a_rebuild_compresses_a_retained_session_it_cannot_rebuild() { + local out rc=0 body + write_claude_session "$TMP/keep-in/proj" + python3 "$REDUCE" --source claude --in "$TMP/keep-in" --out "$TMP/keep-out" >/dev/null 2>&1 \ + || fail 'the first build failed' + mkdir -p "$TMP/keep-out/gone" + printf '# session gone/old.jsonl\n# cwd /home/x/gone\n\nan older answer worth keeping\n' \ + >"$TMP/keep-out/gone/old.txt" + out=$(python3 "$REDUCE" --source claude --in "$TMP/keep-in" --out "$TMP/keep-out" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'the rebuild failed'; } + assert_absent "$TMP/keep-out/gone/old.txt" 'the retained plain session must not be left uncompressed' + assert_present "$TMP/keep-out/gone/old.txt.zst" 'the retained session must survive as a compressed file' + body=$(read_session "$TMP/keep-out/gone/old.txt.zst") + assert_contains "$body" 'an older answer worth keeping' \ + 'compressing a retained session must not cost a word of it' + assert_contains "$out" 'converged' 'the run must say what it converged rather than doing it silently' + pass "a rebuild converges a retained plain session into the compressed store, content intact" +} + +test_verification_reads_the_compressed_store_it_verifies() { + local out rc=0 + out=$(python3 "$REDUCE" --verify-only --out "$TMP/zst-out" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'verification failed over a compressed store'; } + assert_contains "$out" '1 files scanned' \ + 'verification must count the compressed session it read, not skip it as an unknown file' + assert_contains "$out" '0 residual hits' 'verification must state the residual count' + pass "verification opens the compressed store and counts what it scanned" +} + +# A store file the decompressor cannot read is not a clean file; it is a file +# nobody read. Counting it as verified is the silent all-clear this tool exists +# to refuse. +test_verification_refuses_a_store_file_it_cannot_read() { + local out rc=0 + mkdir -p "$TMP/corrupt" + printf 'not a compressed file at all\n' >"$TMP/corrupt/broken.txt.zst" + out=$(python3 "$REDUCE" --verify-only --out "$TMP/corrupt" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail 'a store file that cannot be decompressed must not verify clean' + assert_contains "$out" 'could not be read' 'the refusal must say the file was never verified' + pass "an unreadable store file is reported as unread rather than counted as clean" +} + +test_a_build_without_a_compressor_refuses_before_writing() { + local out rc=0 + write_claude_session "$TMP/nozstd-in/proj" + out=$(FM_ZSTD="$TMP/no-such-zstd" python3 "$REDUCE" --source claude \ + --in "$TMP/nozstd-in" --out "$TMP/nozstd-out" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail 'a build with no compressor must not exit 0' + assert_contains "$out" 'not on PATH' 'the refusal must name the missing tool' + assert_contains "$out" 'half compressed and half plain' 'the refusal must say what it is avoiding' + assert_absent "$TMP/nozstd-out" 'nothing may be written when the compressor is missing' + pass "a missing compressor refuses the build instead of writing a store nobody can search" +} + # --- search ------------------------------------------------------------------ # Build a fixture home so search is exercised the way a vessel that is not this @@ -368,6 +465,100 @@ test_search_reports_a_missing_archive_rather_than_no_matches() { pass "an archive that is not there reads as absent, never as no matches" } +test_search_reads_the_compressed_store() { + local home out plain + home=$(setup_fixture_home) + plain=$(find "$home/data/transcripts" -name '*.txt' -type f | wc -l) + [ "$plain" -eq 0 ] || fail "the fixture archive must be wholly compressed, found $plain plain files" + out=$(FM_HOME="$home" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search found nothing in a compressed archive it should have matched' + assert_contains "$out" 'tugboat host' 'the matching line must come back out of the compressed file' + assert_contains "$out" '.txt.zst' 'the hit must name the compressed session file it came from' + assert_contains "$out" 'cwd /home/x/alpha' \ + 'the session header must be read through the decompressor too, not skipped' + pass "search reads the compressed store and still reports the session header with the hit" +} + +# The reason the documentation had to change, stated as a test rather than as a +# claim: over this store the old documented command answers "nothing here" and +# exits as though that were true. Worse than nothing, it half answers - the +# index is the one plain file left, so a phrase that happens to sit in its +# first-user-message column still matches and the emptiness looks selective +# rather than total. +test_plain_grep_does_not_read_the_sessions() { + local home rc=0 out + home=$(setup_fixture_home) + out=$(grep -r 'looking now' "$home/data/transcripts" 2>/dev/null) || rc=$? + [ "$rc" -eq 1 ] || fail "plain grep was expected to report no matches over a compressed store, got exit $rc" + [ -z "$out" ] || fail 'plain grep unexpectedly read a compressed session' + out=$(FM_HOME="$home" "$SEARCH" 'looking now' 2>/dev/null) \ + || fail 'the wrapper must find what plain grep could not' + assert_contains "$out" 'looking now' 'the same phrase must come back through the wrapper' + pass "plain grep reports no matches on material the wrapper finds, which is why it is no longer documented" +} + +# Whatever the documentation says about searching, it must not be the sentence +# that sends someone to a tool that answers "nothing" over a full archive. +test_no_document_promises_that_plain_grep_works_here() { + local f + for f in "$ROOT/docs/session-archive.md" "$REFRESH" "$SEARCH" "$REDUCE"; do + if grep -nE 'grep -r` (works|reads)|Plain `grep -r` works' "$f" >/dev/null 2>&1; then + grep -nE 'grep -r` (works|reads)|Plain `grep -r` works' "$f" >&2 + fail "$(basename "$f") still tells a reader plain grep -r works over this archive" + fi + done + assert_grep 'rg -z' "$ROOT/docs/session-archive.md" \ + 'the documentation must name the raw command that does read this store' + pass "no document promises plain grep here, and the working raw command is named" +} + +# A caller that scripts this tool reads its exit status, and the file set is +# handed to the scanner in batches: a batch with no hit must not make a search +# that matched look like a search that failed. +test_search_status_says_matched_or_not_matched() { + local home rc=0 + home=$(setup_fixture_home) + FM_HOME="$home" "$SEARCH" 'tugboat host' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 0 ] || fail "a search that matched must exit 0, got $rc" + rc=0 + FM_HOME="$home" "$SEARCH" 'tugboat host' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 0 ] || fail "a --files-only search that matched must exit 0, got $rc" + rc=0 + FM_HOME="$home" "$SEARCH" 'nothing here says this at all' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 1 ] || fail "a search that matched nothing must exit 1, got $rc" + pass "the exit status states whether the search matched, not how a batch of it ended" +} + +test_search_without_its_tool_refuses_rather_than_finding_nothing() { + local home out rc=0 + home=$(setup_fixture_home) + out=$(FM_HOME="$home" FM_RG="$TMP/no-such-rg" "$SEARCH" 'tugboat host' 2>&1) || rc=$? + [ "$rc" -eq 2 ] || fail "a missing search tool must be reported as such, got exit $rc" + assert_contains "$out" 'not installed' 'the refusal must name the missing tool' + case "$out" in + *'tugboat host'*) fail 'a refused search must not look like a search that ran' ;; + esac + pass "a missing decompressing search tool refuses instead of reporting no matches" +} + +# An archive built before compression carries a README that still promises plain +# grep. The rebuild must not overwrite what someone wrote there, and must not +# leave the stale command standing unremarked either. +test_refresh_names_a_readme_that_still_promises_plain_grep() { + local home out + home=$(setup_fixture_home) + printf 'captain notes\n\nPlain `grep -r` works too - grep is the index.\n' \ + >"$home/data/transcripts/README.md" + out=$(FM_HOME="$home" FM_CLAUDE_SESSIONS="$TMP/home-src/claude" \ + FM_CODEX_SESSIONS="$TMP/home-src/codex" "$REFRESH" 2>&1 >/dev/null) \ + || fail 'refresh failed against an archive with an older README' + assert_contains "$out" 'still points readers at plain' \ + 'a README promising plain grep must be named as stale' + assert_grep 'captain notes' "$home/data/transcripts/README.md" \ + 'naming the stale sentence must not overwrite what someone wrote' + pass "a rebuild names a README that still promises plain grep instead of silently keeping it" +} + test_tool_is_tracked_and_runnable test_no_home_path_is_hardcoded test_patterns_file_does_not_match_itself @@ -381,8 +572,19 @@ test_a_run_that_recovers_no_entries_fails test_redactor_masks_a_synthetic_credential test_verification_is_zero_on_clean_and_loud_on_dirty test_verification_over_nothing_is_not_an_all_clear +test_the_store_is_written_compressed_and_leaves_nothing_plain +test_a_rebuild_compresses_a_retained_session_it_cannot_rebuild +test_verification_reads_the_compressed_store_it_verifies +test_verification_refuses_a_store_file_it_cannot_read +test_a_build_without_a_compressor_refuses_before_writing test_refresh_builds_verifies_and_lands_the_bound test_refresh_does_not_overwrite_an_existing_readme test_search_resolves_the_archive_from_fm_home test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches +test_search_reads_the_compressed_store +test_plain_grep_does_not_read_the_sessions +test_no_document_promises_that_plain_grep_works_here +test_search_status_says_matched_or_not_matched +test_search_without_its_tool_refuses_rather_than_finding_nothing +test_refresh_names_a_readme_that_still_promises_plain_grep From 5fb583a7cc903061c919081e8648efd8579882ac Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 04:58:25 +0000 Subject: [PATCH 02/18] no-mistakes(review): Preserve scanner errors and remove source-only documentation test --- bin/fm-transcript-search.sh | 29 ++++++++++++++++-------- tests/fm-transcript-archive.test.sh | 34 ++++++++++++++--------------- 2 files changed, 36 insertions(+), 27 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 04d048463ef..6fb6656979d 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -29,7 +29,7 @@ # and no vessel reads another's. FM_TRANSCRIPT_ARCHIVE overrides the location. # # Exit status: 0 when something matched, 1 when nothing did, 2 on a usage error, -# a missing archive, or a missing decompressing search tool. +# a missing archive, a missing decompressing search tool, or a scanner failure. set -uo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -130,17 +130,26 @@ n=$(wc -l < "$filelist") echo "# searching $n session files under: ${roots[*]}" >&2 [ "$n" -gt 0 ] || exit 1 -# The file set is handed over in batches, so a batch with no hit exits non-zero -# while another batch matched. What the caller is told is whether the SEARCH -# matched, decided by what came out of it, not by the status of one batch. +# The file set is handed over in batches. Each batch normalises a genuine +# no-match to success so xargs can keep scanning, while preserving scanner +# failures. The final output then decides whether the completed search matched. if [ "$files_only" = 1 ]; then - xargs -a "$filelist" -d '\n' "$RG" -lz --no-messages -e "$q" -- 2>/dev/null | + xargs -a "$filelist" -d '\n' bash -c ' + rg=$1; pattern=$2; shift 2 + "$rg" -lz -e "$pattern" -- "$@" + case $? in 0|1) exit 0;; *) exit 255;; esac + ' _ "$RG" "$q" | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' - exit "${PIPESTATUS[1]}" + statuses=("${PIPESTATUS[@]}") + [ "${statuses[0]}" -eq 0 ] || exit 2 + exit "${statuses[1]}" fi -xargs -a "$filelist" -d '\n' "$RG" -z -n -H --null --no-heading --color never \ - --no-messages -C "$ctx" -e "$q" -- 2>/dev/null | +xargs -a "$filelist" -d '\n' bash -c ' + rg=$1; context=$2; pattern=$3; shift 3 + "$rg" -z -n -H --null --no-heading --color never -C "$context" -e "$pattern" -- "$@" + case $? in 0|1) exit 0;; *) exit 255;; esac +' _ "$RG" "$ctx" "$q" | awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' /^--$/ { print " --"; next } NF < 2 { next } @@ -163,4 +172,6 @@ awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' print " " rest } END { exit(found ? 0 : 1) }' -exit "${PIPESTATUS[1]}" +statuses=("${PIPESTATUS[@]}") +[ "${statuses[0]}" -eq 0 ] || exit 2 +exit "${statuses[1]}" diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 3ca4f598eca..65e7b511bdc 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -18,8 +18,7 @@ # compressed, and plain grep reads a compressed file as no matches with no # error - the same silent emptiness as 1, arriving through the # documentation instead of through the reader. So the search path is -# exercised over a compressed store here, and the documentation is scanned -# for the promise it used to make. +# exercised over a compressed store here. # 4. THIS FILE must not match the pattern file either. Property 2 closes the # pattern list; it does not close the fixtures. The first build of this tool # left five real redactions in its own session from fixtures typed in plain @@ -497,21 +496,6 @@ test_plain_grep_does_not_read_the_sessions() { pass "plain grep reports no matches on material the wrapper finds, which is why it is no longer documented" } -# Whatever the documentation says about searching, it must not be the sentence -# that sends someone to a tool that answers "nothing" over a full archive. -test_no_document_promises_that_plain_grep_works_here() { - local f - for f in "$ROOT/docs/session-archive.md" "$REFRESH" "$SEARCH" "$REDUCE"; do - if grep -nE 'grep -r` (works|reads)|Plain `grep -r` works' "$f" >/dev/null 2>&1; then - grep -nE 'grep -r` (works|reads)|Plain `grep -r` works' "$f" >&2 - fail "$(basename "$f") still tells a reader plain grep -r works over this archive" - fi - done - assert_grep 'rg -z' "$ROOT/docs/session-archive.md" \ - 'the documentation must name the raw command that does read this store' - pass "no document promises plain grep here, and the working raw command is named" -} - # A caller that scripts this tool reads its exit status, and the file set is # handed to the scanner in batches: a batch with no hit must not make a search # that matched look like a search that failed. @@ -529,6 +513,20 @@ test_search_status_says_matched_or_not_matched() { pass "the exit status states whether the search matched, not how a batch of it ended" } +test_search_reports_scanner_failures_as_errors() { + local home rc=0 + home=$(setup_fixture_home) + FM_HOME="$home" "$SEARCH" '[' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a malformed regex must be reported as a scanner error, got exit $rc" + rc=0 + FM_HOME="$home" "$SEARCH" '[' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a malformed --files-only regex must be reported as a scanner error, got exit $rc" + rc=0 + FM_HOME="$home" "$SEARCH" 'nothing here says this at all' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 1 ] || fail "an ordinary no-match must remain exit 1, got $rc" + pass "scanner failures are errors while an ordinary no-match remains distinct" +} + test_search_without_its_tool_refuses_rather_than_finding_nothing() { local home out rc=0 home=$(setup_fixture_home) @@ -584,7 +582,7 @@ test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store test_plain_grep_does_not_read_the_sessions -test_no_document_promises_that_plain_grep_works_here test_search_status_says_matched_or_not_matched +test_search_reports_scanner_failures_as_errors test_search_without_its_tool_refuses_rather_than_finding_nothing test_refresh_names_a_readme_that_still_promises_plain_grep From b4f53a6d9a706d31acb5834dc12c4cabb44f393e Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 06:43:32 +0000 Subject: [PATCH 03/18] no-mistakes(review): Route transcript scans through configured zstd preprocessor --- bin/fm-transcript-search.sh | 29 +++++++++++++++++------------ bin/fm-transcript-zcat.sh | 5 +++++ docs/session-archive.md | 2 +- tests/fm-transcript-archive.test.sh | 29 +++++++++++++++++++++++++++++ 4 files changed, 52 insertions(+), 13 deletions(-) create mode 100755 bin/fm-transcript-zcat.sh diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 6fb6656979d..fb8e50ac9fa 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -17,12 +17,12 @@ # # THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches # nothing here and exits as though the archive were empty, which is the one -# answer this archive must never give. The scan therefore runs through ripgrep's -# `-z`, and `rg` is a hard requirement: a missing one is reported as a missing -# tool, never as a search that found nothing. Anyone who would rather not use -# this wrapper runs `rg -z ` over the archive directory directly and -# gets the same content scan. FM_RG and FM_ZSTD name the two binaries on a -# vessel where they sit elsewhere. +# answer this archive must never give. The scan therefore runs through ripgrep +# with a zstd preprocessor, and `rg` is a hard requirement: a missing one is +# reported as a missing tool, never as a search that found nothing. Anyone who +# would rather not use this wrapper runs `rg -z ` over the archive +# directory directly and gets the same content scan. FM_RG names ripgrep and +# FM_ZSTD names the compressor used for both the scan and session headers. # # The archive is this home's private material and never travels: it resolves # under $FM_HOME/data/transcripts, so a secondmate home searches its own store @@ -89,6 +89,7 @@ fi # caller and only one of them is true. RG="${FM_RG:-rg}" ZSTD="${FM_ZSTD:-zstd}" +PREPROCESSOR="$SCRIPT_DIR/fm-transcript-zcat.sh" for tool in "$RG" "$ZSTD"; do command -v "$tool" >/dev/null 2>&1 && continue echo "$tool is not installed, and the session store is compressed: this search cannot run." >&2 @@ -96,6 +97,7 @@ for tool in "$RG" "$ZSTD"; do echo "Refusing rather than reporting no matches over an archive that was never read." >&2 exit 2 done +export FM_ZSTD="$ZSTD" # A user ripgrep config can change what a search means - case folding, column # limits, skipped files - and this store's answers must not depend on it. export RIPGREP_CONFIG_PATH= @@ -134,22 +136,25 @@ echo "# searching $n session files under: ${roots[*]}" >&2 # no-match to success so xargs can keep scanning, while preserving scanner # failures. The final output then decides whether the completed search matched. if [ "$files_only" = 1 ]; then + # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; pattern=$2; shift 2 - "$rg" -lz -e "$pattern" -- "$@" + rg=$1; preprocessor=$2; pattern=$3; shift 3 + "$rg" -l --pre "$preprocessor" --pre-glob "*.zst" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac - ' _ "$RG" "$q" | + ' _ "$RG" "$PREPROCESSOR" "$q" | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' statuses=("${PIPESTATUS[@]}") [ "${statuses[0]}" -eq 0 ] || exit 2 exit "${statuses[1]}" fi +# shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; context=$2; pattern=$3; shift 3 - "$rg" -z -n -H --null --no-heading --color never -C "$context" -e "$pattern" -- "$@" + rg=$1; preprocessor=$2; context=$3; pattern=$4; shift 4 + "$rg" --pre "$preprocessor" --pre-glob "*.zst" -n -H --null --no-heading \ + --color never -C "$context" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac -' _ "$RG" "$ctx" "$q" | +' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' /^--$/ { print " --"; next } NF < 2 { next } diff --git a/bin/fm-transcript-zcat.sh b/bin/fm-transcript-zcat.sh new file mode 100755 index 00000000000..53142337acb --- /dev/null +++ b/bin/fm-transcript-zcat.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -uo pipefail + +ZSTD="${FM_ZSTD:-zstd}" +exec "$ZSTD" -dcq -- "$1" diff --git a/docs/session-archive.md b/docs/session-archive.md index 146968e410d..18be77c0d95 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -54,7 +54,7 @@ rg -z 'pattern' "$FM_HOME/data/transcripts" ``` `rg` and `zstd` are therefore hard requirements of a search here, and the wrapper reports a missing one as a missing tool rather than as a search that found nothing. -`FM_RG` and `FM_ZSTD` name the binaries on a vessel where they sit elsewhere. +`FM_RG` names ripgrep and `FM_ZSTD` names the compressor the wrapper uses to scan sessions and read their headers when those binaries sit elsewhere. ## Rebuild diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 65e7b511bdc..fe76c34d556 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -478,6 +478,34 @@ test_search_reads_the_compressed_store() { pass "search reads the compressed store and still reports the session header with the hit" } +test_search_uses_the_selected_compressor_and_reads_plain_sessions() { + local home out real_zstd wrapper marker plain + home=$(setup_fixture_home) + real_zstd=$(command -v zstd) + wrapper="$TMP/recording-zstd" + marker="$TMP/zstd-ran" + cat >"$wrapper" <<'EOF' +#!/usr/bin/env bash +: >"$FM_TEST_ZSTD_MARKER" +exec "$FM_TEST_REAL_ZSTD" "$@" +EOF + chmod +x "$wrapper" + out=$(FM_HOME="$home" FM_ZSTD="$wrapper" FM_TEST_ZSTD_MARKER="$marker" \ + FM_TEST_REAL_ZSTD="$real_zstd" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search failed through the selected compressor' + assert_contains "$out" 'tugboat host' 'the selected compressor must return the compressed hit' + assert_present "$marker" 'the selected compressor must perform the content scan' + + plain="$home/data/transcripts/claude-redacted/migration.txt" + printf '# cwd /home/x/migration\n# span 2026-08-18T10:00:00Z\nplain migration session\n' >"$plain" + out=$(FM_HOME="$home" FM_ZSTD="$wrapper" FM_TEST_ZSTD_MARKER="$marker" \ + FM_TEST_REAL_ZSTD="$real_zstd" "$SEARCH" 'plain migration session' 2>/dev/null) \ + || fail 'search did not read a plain session during migration' + assert_contains "$out" 'plain migration session' \ + 'a plain migration-era session must bypass the compressor and remain searchable' + pass "the selected compressor scans compressed sessions without hiding plain migration files" +} + # The reason the documentation had to change, stated as a test rather than as a # claim: over this store the old documented command answers "nothing here" and # exits as though that were true. Worse than nothing, it half answers - the @@ -581,6 +609,7 @@ test_search_resolves_the_archive_from_fm_home test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store +test_search_uses_the_selected_compressor_and_reads_plain_sessions test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched test_search_reports_scanner_failures_as_errors From 74d7c6c12b278e9f9bf6893e3f6bcd2762934faa Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 06:50:54 +0000 Subject: [PATCH 04/18] no-mistakes(review): Withdraw search override and defer archive creation --- bin/fm-transcript-reduce.py | 9 +++--- bin/fm-transcript-refresh.sh | 2 -- bin/fm-transcript-search.sh | 42 ++++++++++++++++----------- bin/fm-transcript-zcat.sh | 5 ---- docs/session-archive.md | 4 ++- tests/fm-transcript-archive.test.sh | 44 ++++++++++++++--------------- 6 files changed, 55 insertions(+), 51 deletions(-) delete mode 100755 bin/fm-transcript-zcat.sh diff --git a/bin/fm-transcript-reduce.py b/bin/fm-transcript-reduce.py index 8ea49927de1..de45796a551 100755 --- a/bin/fm-transcript-reduce.py +++ b/bin/fm-transcript-reduce.py @@ -44,10 +44,11 @@ Compression cannot, because a wrong decompression is an error and not a wrong answer. -The compressor is the `zstd` binary (FM_ZSTD overrides it). It is a hard -requirement rather than a preference: a run that cannot compress refuses instead -of leaving a store that is half compressed and half plain, which is exactly the -kind of quiet disagreement this archive exists to avoid. +The compressor is the `zstd` binary (FM_ZSTD overrides it for building and +verifying). It is a hard requirement rather than a preference: a run that cannot +compress refuses instead of leaving a store that is half compressed and half +plain, which is exactly the kind of quiet disagreement this archive exists to +avoid. Usage: fm-transcript-reduce.py --source claude|codex --in DIR --out DIR diff --git a/bin/fm-transcript-refresh.sh b/bin/fm-transcript-refresh.sh index e19747c9680..f0fa1bb1b68 100755 --- a/bin/fm-transcript-refresh.sh +++ b/bin/fm-transcript-refresh.sh @@ -64,8 +64,6 @@ while [ $# -gt 0 ]; do esac done -mkdir -p "$ARCHIVE" - built=0 for s in claude codex; do case "$s" in diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index fb8e50ac9fa..18875b50e32 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -17,12 +17,14 @@ # # THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches # nothing here and exits as though the archive were empty, which is the one -# answer this archive must never give. The scan therefore runs through ripgrep -# with a zstd preprocessor, and `rg` is a hard requirement: a missing one is +# answer this archive must never give. The scan therefore runs through ripgrep's +# `-z`, and `rg` and `zstd` on PATH are hard requirements: a missing one is # reported as a missing tool, never as a search that found nothing. Anyone who # would rather not use this wrapper runs `rg -z ` over the archive -# directory directly and gets the same content scan. FM_RG names ripgrep and -# FM_ZSTD names the compressor used for both the scan and session headers. +# directory directly and gets the same content scan. FM_RG may name ripgrep +# elsewhere. Search deliberately has no FM_ZSTD override: one compressor knob +# governing only part of the process repeatedly passed its prerequisite check +# before the scan read nothing or the session header disappeared. # # The archive is this home's private material and never travels: it resolves # under $FM_HOME/data/transcripts, so a secondmate home searches its own store @@ -88,16 +90,14 @@ fi # "the tool that reads this store is not installed" are the same output to a # caller and only one of them is true. RG="${FM_RG:-rg}" -ZSTD="${FM_ZSTD:-zstd}" -PREPROCESSOR="$SCRIPT_DIR/fm-transcript-zcat.sh" +ZSTD=zstd for tool in "$RG" "$ZSTD"; do command -v "$tool" >/dev/null 2>&1 && continue echo "$tool is not installed, and the session store is compressed: this search cannot run." >&2 - echo "Install ripgrep and zstd (apt install ripgrep zstd), or point FM_RG and FM_ZSTD at them." >&2 + echo "Install ripgrep and zstd (apt install ripgrep zstd), or point FM_RG at ripgrep." >&2 echo "Refusing rather than reporting no matches over an archive that was never read." >&2 exit 2 done -export FM_ZSTD="$ZSTD" # A user ripgrep config can change what a search means - case folding, column # limits, skipped files - and this store's answers must not depend on it. export RIPGREP_CONFIG_PATH= @@ -138,10 +138,10 @@ echo "# searching $n session files under: ${roots[*]}" >&2 if [ "$files_only" = 1 ]; then # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; preprocessor=$2; pattern=$3; shift 3 - "$rg" -l --pre "$preprocessor" --pre-glob "*.zst" -e "$pattern" -- "$@" + rg=$1; pattern=$2; shift 2 + "$rg" -lz -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac - ' _ "$RG" "$PREPROCESSOR" "$q" | + ' _ "$RG" "$q" | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' statuses=("${PIPESTATUS[@]}") [ "${statuses[0]}" -eq 0 ] || exit 2 @@ -150,12 +150,20 @@ fi # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; preprocessor=$2; context=$3; pattern=$4; shift 4 - "$rg" --pre "$preprocessor" --pre-glob "*.zst" -n -H --null --no-heading \ - --color never -C "$context" -e "$pattern" -- "$@" + rg=$1; context=$2; pattern=$3; shift 3 + "$rg" -z -n -H --null --no-heading --color never -C "$context" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac -' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | +' _ "$RG" "$ctx" "$q" | awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' + function shell_quote(s, out, i, c) { + out="\"" + for (i=1; i<=length(s); i++) { + c=substr(s,i,1) + if (c=="\\" || c=="\"" || c=="$" || c=="`") out=out "\\" c + else out=out c + } + return out "\"" + } /^--$/ { print " --"; next } NF < 2 { next } { @@ -164,9 +172,9 @@ awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' # The session header is read the same way the store is: through the # decompressor when the file is compressed, plainly when it is not. if (path ~ /\.zst$/) - cmd=zstd " -dcq -- \"" path "\" | sed -n \"1,8p\"" + cmd=shell_quote(zstd) " -dcq -- " shell_quote(path) " | sed -n \"1,8p\"" else - cmd="sed -n \"1,8p\" \"" path "\"" + cmd="sed -n \"1,8p\" " shell_quote(path) cmd = cmd " | grep -E \"^# (cwd|span)\" | sed \"s/^# *//\" | tr \"\\n\" \"|\"" hdr=""; cmd | getline hdr; close(cmd) short=path; sub(arch "/", "", short) diff --git a/bin/fm-transcript-zcat.sh b/bin/fm-transcript-zcat.sh deleted file mode 100755 index 53142337acb..00000000000 --- a/bin/fm-transcript-zcat.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash -set -uo pipefail - -ZSTD="${FM_ZSTD:-zstd}" -exec "$ZSTD" -dcq -- "$1" diff --git a/docs/session-archive.md b/docs/session-archive.md index 18be77c0d95..586b571e671 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -54,7 +54,8 @@ rg -z 'pattern' "$FM_HOME/data/transcripts" ``` `rg` and `zstd` are therefore hard requirements of a search here, and the wrapper reports a missing one as a missing tool rather than as a search that found nothing. -`FM_RG` names ripgrep and `FM_ZSTD` names the compressor the wrapper uses to scan sessions and read their headers when those binaries sit elsewhere. +`FM_RG` may name ripgrep elsewhere, but search requires `zstd` on PATH and deliberately has no `FM_ZSTD` override. +A single compressor knob that governed only part of the search process repeatedly let its prerequisite check pass before the scan read nothing or the session header disappeared, so the partial override was withdrawn rather than allowed to remain misleading. ## Rebuild @@ -75,6 +76,7 @@ A raw store that does not exist on this vessel is reported and skipped, never tr Each session is written as one zstd file at level 3, which took this seat's store from 271.2 MB to 52.1 MB on 2026-08-18. `zstd` is a hard requirement of a rebuild rather than a preference: a run that cannot compress refuses before writing anything, because a store that is half compressed and half plain is a store whose answers depend on which half a question lands in. +`FM_ZSTD` may name the compressor used specifically for building and verifying the store; it does not apply to searching. A rebuild also compresses whatever plain session files it finds already in the store and drops the plain copies it has just superseded, so an archive built before compression converges on the first refresh instead of being stranded, and a retained session the rebuild cannot reach is compressed where it lies rather than left behind. The level is a knob, `--level`, and the measurement behind the default is below. diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index fe76c34d556..2e891ba4953 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -383,6 +383,21 @@ test_a_build_without_a_compressor_refuses_before_writing() { pass "a missing compressor refuses the build instead of writing a store nobody can search" } +test_refresh_without_a_compressor_leaves_no_archive() { + local home out rc=0 + home="$TMP/refresh-nozstd-home" + write_claude_session "$TMP/refresh-nozstd-src/claude/project" + out=$(FM_HOME="$home" FM_CLAUDE_SESSIONS="$TMP/refresh-nozstd-src/claude" \ + FM_CODEX_SESSIONS="$TMP/refresh-nozstd-src/codex" FM_ZSTD="$TMP/no-such-zstd" \ + "$REFRESH" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail 'refresh with no compressor must not exit 0' + assert_contains "$out" 'half compressed and half plain' \ + 'refresh must preserve the reducer refusal message' + assert_absent "$home/data/transcripts" \ + 'refresh must not create an archive before the compressor prerequisite passes' + pass "refresh with no compressor refuses before creating the archive" +} + # --- search ------------------------------------------------------------------ # Build a fixture home so search is exercised the way a vessel that is not this @@ -478,32 +493,16 @@ test_search_reads_the_compressed_store() { pass "search reads the compressed store and still reports the session header with the hit" } -test_search_uses_the_selected_compressor_and_reads_plain_sessions() { - local home out real_zstd wrapper marker plain +test_search_reads_plain_sessions_during_migration() { + local home out plain home=$(setup_fixture_home) - real_zstd=$(command -v zstd) - wrapper="$TMP/recording-zstd" - marker="$TMP/zstd-ran" - cat >"$wrapper" <<'EOF' -#!/usr/bin/env bash -: >"$FM_TEST_ZSTD_MARKER" -exec "$FM_TEST_REAL_ZSTD" "$@" -EOF - chmod +x "$wrapper" - out=$(FM_HOME="$home" FM_ZSTD="$wrapper" FM_TEST_ZSTD_MARKER="$marker" \ - FM_TEST_REAL_ZSTD="$real_zstd" "$SEARCH" 'tugboat host' 2>/dev/null) \ - || fail 'search failed through the selected compressor' - assert_contains "$out" 'tugboat host' 'the selected compressor must return the compressed hit' - assert_present "$marker" 'the selected compressor must perform the content scan' - plain="$home/data/transcripts/claude-redacted/migration.txt" printf '# cwd /home/x/migration\n# span 2026-08-18T10:00:00Z\nplain migration session\n' >"$plain" - out=$(FM_HOME="$home" FM_ZSTD="$wrapper" FM_TEST_ZSTD_MARKER="$marker" \ - FM_TEST_REAL_ZSTD="$real_zstd" "$SEARCH" 'plain migration session' 2>/dev/null) \ + out=$(FM_HOME="$home" "$SEARCH" 'plain migration session' 2>/dev/null) \ || fail 'search did not read a plain session during migration' assert_contains "$out" 'plain migration session' \ - 'a plain migration-era session must bypass the compressor and remain searchable' - pass "the selected compressor scans compressed sessions without hiding plain migration files" + 'a plain migration-era session must remain searchable' + pass "search reads plain sessions that remain during migration" } # The reason the documentation had to change, stated as a test rather than as a @@ -603,13 +602,14 @@ test_a_rebuild_compresses_a_retained_session_it_cannot_rebuild test_verification_reads_the_compressed_store_it_verifies test_verification_refuses_a_store_file_it_cannot_read test_a_build_without_a_compressor_refuses_before_writing +test_refresh_without_a_compressor_leaves_no_archive test_refresh_builds_verifies_and_lands_the_bound test_refresh_does_not_overwrite_an_existing_readme test_search_resolves_the_archive_from_fm_home test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store -test_search_uses_the_selected_compressor_and_reads_plain_sessions +test_search_reads_plain_sessions_during_migration test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched test_search_reports_scanner_failures_as_errors From d80e64322b2ec053dbf2efbcfb0de4b361e77234 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 06:54:18 +0000 Subject: [PATCH 05/18] no-mistakes(document): Clarify archive measurement terminology --- docs/session-archive.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/session-archive.md b/docs/session-archive.md index 586b571e671..8536f38726a 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -156,10 +156,10 @@ zstd -9 159.6 s 48.7 MB 5.62x ``` Level 3 is the default because level 9 spends 40 more seconds of every rebuild to save 3.4 MB, which is 1.2 percent of what was already saved. -The ratio is material-dependent in the same way the reduction is: these are readings from one seat's transcripts, not properties of zstd. +The compression ratio is material-dependent: these are readings from one seat's transcripts, not properties of zstd. Search cost is not part of this tradeoff - zstd decompresses at roughly the same speed whatever level wrote the file - so the level buys rebuild time against disk and nothing else. -The ratio is material-dependent and is not a property of the tool: the same design measured 56:1 on a seat whose sessions averaged 17 MB, against 0.63 MB here. +The reduction ratio is material-dependent and is not a property of the tool: the same design measured 56:1 on a seat whose sessions averaged 17 MB, against 0.63 MB here. `--fold-injected` folds Codex's machine-injected user messages down to their marker and their tail, taking the Codex store from 142.2 MB to 44.1 MB. It is off by default, and the reason is the finding rather than the size: folding dropped 24 of the 37 redaction findings, including 16 of the 22 private-key blocks, because those sat inside the folded region. The folded archive is not cleaner - less of it was ever looked at. From 53bb285a21d108976f94ee09723488e857b22edb Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 07:17:55 +0000 Subject: [PATCH 06/18] no-mistakes(lint): Avoid ShellCheck warning in archive fixture --- tests/fm-transcript-archive.test.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 2e891ba4953..92378af8e5e 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -570,9 +570,10 @@ test_search_without_its_tool_refuses_rather_than_finding_nothing() { # grep. The rebuild must not overwrite what someone wrote there, and must not # leave the stale command standing unremarked either. test_refresh_names_a_readme_that_still_promises_plain_grep() { - local home out + local home out tick='`' home=$(setup_fixture_home) - printf 'captain notes\n\nPlain `grep -r` works too - grep is the index.\n' \ + printf 'captain notes\n\nPlain %sgrep -r%s works too - grep is the index.\n' \ + "$tick" "$tick" \ >"$home/data/transcripts/README.md" out=$(FM_HOME="$home" FM_CLAUDE_SESSIONS="$TMP/home-src/claude" \ FM_CODEX_SESSIONS="$TMP/home-src/codex" "$REFRESH" 2>&1 >/dev/null) \ From 6a1e1d80ea997eb50859326e3eedc9c7954b23ec Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 08:06:37 +0000 Subject: [PATCH 07/18] no-mistakes: apply CI fixes --- bin/fm-transcript-search.sh | 21 ++++++++++++--------- bin/fm-transcript-zcat.sh | 5 +++++ tests/fm-transcript-archive.test.sh | 27 ++++++++++++++++++++++++++- 3 files changed, 43 insertions(+), 10 deletions(-) create mode 100755 bin/fm-transcript-zcat.sh diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 18875b50e32..913c39c1777 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -17,9 +17,10 @@ # # THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches # nothing here and exits as though the archive were empty, which is the one -# answer this archive must never give. The scan therefore runs through ripgrep's -# `-z`, and `rg` and `zstd` on PATH are hard requirements: a missing one is -# reported as a missing tool, never as a search that found nothing. Anyone who +# answer this archive must never give. The scan therefore runs through ripgrep +# with zstd as its explicit preprocessor. `rg` and `zstd` on PATH are hard +# requirements: a missing one is reported as a missing tool, never as a search +# that found nothing. Anyone who # would rather not use this wrapper runs `rg -z ` over the archive # directory directly and gets the same content scan. FM_RG may name ripgrep # elsewhere. Search deliberately has no FM_ZSTD override: one compressor knob @@ -91,6 +92,7 @@ fi # caller and only one of them is true. RG="${FM_RG:-rg}" ZSTD=zstd +PREPROCESSOR="$SCRIPT_DIR/fm-transcript-zcat.sh" for tool in "$RG" "$ZSTD"; do command -v "$tool" >/dev/null 2>&1 && continue echo "$tool is not installed, and the session store is compressed: this search cannot run." >&2 @@ -138,10 +140,10 @@ echo "# searching $n session files under: ${roots[*]}" >&2 if [ "$files_only" = 1 ]; then # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; pattern=$2; shift 2 - "$rg" -lz -e "$pattern" -- "$@" + rg=$1; preprocessor=$2; pattern=$3; shift 3 + "$rg" -l --pre "$preprocessor" --pre-glob "*.zst" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac - ' _ "$RG" "$q" | + ' _ "$RG" "$PREPROCESSOR" "$q" | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' statuses=("${PIPESTATUS[@]}") [ "${statuses[0]}" -eq 0 ] || exit 2 @@ -150,10 +152,11 @@ fi # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; context=$2; pattern=$3; shift 3 - "$rg" -z -n -H --null --no-heading --color never -C "$context" -e "$pattern" -- "$@" + rg=$1; preprocessor=$2; context=$3; pattern=$4; shift 4 + "$rg" --pre "$preprocessor" --pre-glob "*.zst" -n -H --null --no-heading \ + --color never -C "$context" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac -' _ "$RG" "$ctx" "$q" | +' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' function shell_quote(s, out, i, c) { out="\"" diff --git a/bin/fm-transcript-zcat.sh b/bin/fm-transcript-zcat.sh new file mode 100755 index 00000000000..58708caae21 --- /dev/null +++ b/bin/fm-transcript-zcat.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +# ripgrep --pre adapter for one compressed session file. +set -uo pipefail + +exec zstd -dcq -- "$1" diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 92378af8e5e..3ea462889a7 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -35,6 +35,7 @@ set -u REDUCE="$ROOT/bin/fm-transcript-reduce.py" SEARCH="$ROOT/bin/fm-transcript-search.sh" REFRESH="$ROOT/bin/fm-transcript-refresh.sh" +ZCAT="$ROOT/bin/fm-transcript-zcat.sh" PATTERNS="$ROOT/bin/fm-transcript-patterns/patterns.txt" INJECTED="$ROOT/bin/fm-transcript-patterns/injected-prefixes.txt" @@ -120,7 +121,7 @@ PY test_tool_is_tracked_and_runnable() { local f - for f in "$REDUCE" "$SEARCH" "$REFRESH"; do + for f in "$REDUCE" "$SEARCH" "$REFRESH" "$ZCAT"; do assert_present "$f" "missing tool file: $f" [ -x "$f" ] || fail "$f must be executable" done @@ -493,6 +494,29 @@ test_search_reads_the_compressed_store() { pass "search reads the compressed store and still reports the session header with the hit" } +test_search_does_not_depend_on_ripgrep_implicit_decompression() { + local home out wrapper real_rg + home=$(setup_fixture_home) + wrapper="$TMP/rg-without-z" + real_rg=$(command -v rg) + cat >"$wrapper" <<'EOF' +#!/usr/bin/env bash +for arg in "$@"; do + case "$arg" in + -z|-*z*) exit 1 ;; + esac +done +exec "$FM_TEST_REAL_RG" "$@" +EOF + chmod +x "$wrapper" + out=$(FM_HOME="$home" FM_RG="$wrapper" FM_TEST_REAL_RG="$real_rg" \ + "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search depended on ripgrep implicit decompression instead of the required zstd tool' + assert_contains "$out" 'tugboat host' \ + 'explicit zstd preprocessing must return the compressed hit' + pass "search uses its required zstd tool rather than ripgrep's version-dependent implicit decompressor" +} + test_search_reads_plain_sessions_during_migration() { local home out plain home=$(setup_fixture_home) @@ -610,6 +634,7 @@ test_search_resolves_the_archive_from_fm_home test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store +test_search_does_not_depend_on_ripgrep_implicit_decompression test_search_reads_plain_sessions_during_migration test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched From 135ecde2cc4c78d8f7041cd7a6edbd6733a08013 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 08:39:16 +0000 Subject: [PATCH 08/18] no-mistakes: apply CI fixes --- bin/fm-transcript-search.sh | 4 ++-- bin/fm-transcript-zcat.sh | 7 +++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 913c39c1777..5b4b536dc8f 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -141,7 +141,7 @@ if [ "$files_only" = 1 ]; then # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' rg=$1; preprocessor=$2; pattern=$3; shift 3 - "$rg" -l --pre "$preprocessor" --pre-glob "*.zst" -e "$pattern" -- "$@" + "$rg" -l --pre "$preprocessor" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac ' _ "$RG" "$PREPROCESSOR" "$q" | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' @@ -153,7 +153,7 @@ fi # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' rg=$1; preprocessor=$2; context=$3; pattern=$4; shift 4 - "$rg" --pre "$preprocessor" --pre-glob "*.zst" -n -H --null --no-heading \ + "$rg" --pre "$preprocessor" -n -H --null --no-heading \ --color never -C "$context" -e "$pattern" -- "$@" case $? in 0|1) exit 0;; *) exit 255;; esac ' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | diff --git a/bin/fm-transcript-zcat.sh b/bin/fm-transcript-zcat.sh index 58708caae21..bc6535f090a 100755 --- a/bin/fm-transcript-zcat.sh +++ b/bin/fm-transcript-zcat.sh @@ -1,5 +1,8 @@ #!/usr/bin/env bash -# ripgrep --pre adapter for one compressed session file. +# ripgrep --pre adapter for compressed and retained plain session files. set -uo pipefail -exec zstd -dcq -- "$1" +case $1 in + *.zst) exec zstd -dcq -- "$1" ;; + *) exec cat -- "$1" ;; +esac From f1ea77feea680aaf509e8645329271bcb99a1fa7 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 09:11:33 +0000 Subject: [PATCH 09/18] no-mistakes: apply CI fixes --- bin/fm-transcript-search.sh | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 5b4b536dc8f..6566a7dbe1e 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -141,8 +141,16 @@ if [ "$files_only" = 1 ]; then # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' rg=$1; preprocessor=$2; pattern=$3; shift 3 - "$rg" -l --pre "$preprocessor" -e "$pattern" -- "$@" - case $? in 0|1) exit 0;; *) exit 255;; esac + for path do + "$preprocessor" "$path" | "$rg" -q -e "$pattern" + statuses=("${PIPESTATUS[@]}") + [ "${statuses[0]}" -eq 0 ] || exit 255 + case ${statuses[1]} in + 0) printf "%s\n" "$path" ;; + 1) ;; + *) exit 255 ;; + esac + done ' _ "$RG" "$PREPROCESSOR" "$q" | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' statuses=("${PIPESTATUS[@]}") @@ -153,9 +161,18 @@ fi # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' bash -c ' rg=$1; preprocessor=$2; context=$3; pattern=$4; shift 4 - "$rg" --pre "$preprocessor" -n -H --null --no-heading \ - --color never -C "$context" -e "$pattern" -- "$@" - case $? in 0|1) exit 0;; *) exit 255;; esac + for path do + "$preprocessor" "$path" | + "$rg" -n --no-heading --color never -C "$context" -e "$pattern" | + awk -v path="$path" '\'' + /^--$/ { print; next } + { printf "%s%c%s\n", path, 0, $0 } + '\'' + statuses=("${PIPESTATUS[@]}") + [ "${statuses[0]}" -eq 0 ] || exit 255 + case ${statuses[1]} in 0|1) ;; *) exit 255;; esac + [ "${statuses[2]}" -eq 0 ] || exit 255 + done ' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' function shell_quote(s, out, i, c) { From 1b522792859f573d388a57b599fc250c142bd17b Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 13:40:11 +0000 Subject: [PATCH 10/18] no-mistakes: apply CI fixes --- tests/fm-transcript-archive.test.sh | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 3ea462889a7..ea488234479 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -494,16 +494,19 @@ test_search_reads_the_compressed_store() { pass "search reads the compressed store and still reports the session header with the hit" } -test_search_does_not_depend_on_ripgrep_implicit_decompression() { +test_search_does_not_depend_on_ripgrep_compression_flags() { local home out wrapper real_rg home=$(setup_fixture_home) wrapper="$TMP/rg-without-z" real_rg=$(command -v rg) + # Model the older runner ripgrep that made --pre return no hits, as well as a + # build without implicit -z decompression. The wrapper's public result must + # come from the explicit zstd pipeline supported by both versions. cat >"$wrapper" <<'EOF' #!/usr/bin/env bash for arg in "$@"; do case "$arg" in - -z|-*z*) exit 1 ;; + -z|-*z*|--pre) exit 1 ;; esac done exec "$FM_TEST_REAL_RG" "$@" @@ -513,8 +516,8 @@ EOF "$SEARCH" 'tugboat host' 2>/dev/null) \ || fail 'search depended on ripgrep implicit decompression instead of the required zstd tool' assert_contains "$out" 'tugboat host' \ - 'explicit zstd preprocessing must return the compressed hit' - pass "search uses its required zstd tool rather than ripgrep's version-dependent implicit decompressor" + 'external zstd preprocessing must return the compressed hit' + pass "search does not depend on ripgrep's version-dependent compression flags" } test_search_reads_plain_sessions_during_migration() { @@ -634,7 +637,7 @@ test_search_resolves_the_archive_from_fm_home test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store -test_search_does_not_depend_on_ripgrep_implicit_decompression +test_search_does_not_depend_on_ripgrep_compression_flags test_search_reads_plain_sessions_during_migration test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched From b49bfaa91b6385be05caf5ac82888bda0c8ff995 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 18:52:57 +0000 Subject: [PATCH 11/18] no-mistakes: apply CI fixes --- bin/fm-transcript-search.sh | 12 ++++++++---- tests/fm-transcript-archive.test.sh | 18 ++++++++++++++++++ 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 6566a7dbe1e..3c4c107bf22 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -166,7 +166,7 @@ xargs -a "$filelist" -d '\n' bash -c ' "$rg" -n --no-heading --color never -C "$context" -e "$pattern" | awk -v path="$path" '\'' /^--$/ { print; next } - { printf "%s%c%s\n", path, 0, $0 } + { printf "%c%s%c%s\n", 28, path, 28, $0 } '\'' statuses=("${PIPESTATUS[@]}") [ "${statuses[0]}" -eq 0 ] || exit 255 @@ -174,7 +174,7 @@ xargs -a "$filelist" -d '\n' bash -c ' [ "${statuses[2]}" -eq 0 ] || exit 255 done ' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | -awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' +awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' function shell_quote(s, out, i, c) { out="\"" for (i=1; i<=length(s); i++) { @@ -185,9 +185,13 @@ awk -F'\0' -v arch="$ARCHIVE" -v zstd="$ZSTD" ' return out "\"" } /^--$/ { print " --"; next } - NF < 2 { next } + substr($0, 1, 1) != sprintf("%c", 28) { next } { - path=$1; rest=$2 + record=substr($0, 2) + split_at=index(record, sprintf("%c", 28)) + if (!split_at) next + path=substr(record, 1, split_at - 1) + rest=substr(record, split_at + 1) if (path != last) { # The session header is read the same way the store is: through the # decompressor when the file is compressed, plainly when it is not. diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index ea488234479..7559d466135 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -456,6 +456,23 @@ test_search_resolves_the_archive_from_fm_home() { pass "search resolves its archive from FM_HOME and reaches both stores" } +test_search_formatter_is_portable_to_mawk() { + local home out shim + command -v mawk >/dev/null 2>&1 || { + pass "mawk is unavailable, so the portable formatter check is not applicable" + return + } + home=$(setup_fixture_home) + shim="$TMP/mawk-path" + mkdir -p "$shim" + ln -s "$(command -v mawk)" "$shim/awk" + out=$(PATH="$shim:$PATH" FM_HOME="$home" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search lost a real hit when its formatter ran under mawk' + assert_contains "$out" 'tugboat host' \ + 'the portable formatter must preserve the matching line under mawk' + pass "search formatting preserves real hits under mawk as well as gawk" +} + test_search_narrows_the_file_set_by_index() { local home out home=$(setup_fixture_home) @@ -634,6 +651,7 @@ test_refresh_without_a_compressor_leaves_no_archive test_refresh_builds_verifies_and_lands_the_bound test_refresh_does_not_overwrite_an_existing_readme test_search_resolves_the_archive_from_fm_home +test_search_formatter_is_portable_to_mawk test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store From d326b6a6ac5108ad5de4c3437047c2caacdb904d Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 21:44:17 +0000 Subject: [PATCH 12/18] Search the archive with zstd and grep instead of ripgrep The compressed store was searched through ripgrep, and ripgrep turned out to be the one part of this tool that answered differently depending on the machine. Two CI rounds failed on the runner's build - which flags it accepted, whether it could decompress at all - while every build and verification test passed there. A search tool whose answer depends on the machine it runs on is the same silent disagreement this archive exists to refuse, so the dependency is removed rather than pinned to a version. The scan now decompresses each session into grep, one file at a time, spread across the machine's cores. Required tools are zstd, grep and xargs, and nothing else; zstd is already needed to build the store, and the other two are on any machine that can run this repository's tests. No override names any of them, after a partial override twice passed its own prerequisite check before the scan read nothing. Measured on this seat's live 2273-session archive rather than on a fixture: Grossreinschiff 75 of 75 matching sessions, exit 0 no-mistakes 2048 matching sessions, exit 0 whole-store scan 1.44 - 1.48 s That is slower than the 0.63 - 0.92 s recorded for the ripgrep scan, and docs/session-archive.md now carries the new number rather than the old one. The defect that made this necessary is worth stating, because it survived three fix rounds and every existing test: grep stops reading as soon as it can answer, the decompressor feeding it then dies of a broken pipe, and treating that ordinary event as a scanner error returned 0 matching sessions where 75 were expected - over a full archive, with the suite green. A one-session fixture with one match is too small to make anything stop early, which is why nothing caught it. The suite now plants that case: a dozen sessions with the match at the top and a long tail behind it, requiring all twelve back. Two further tests replace assertions that could not have caught this. The proof that the search does not need ripgrep now runs it on a PATH where ripgrep cannot be resolved at all, rather than stubbing a flag on a wrapper; and the missing-tool refusal removes the decompressor from PATH rather than pointing an override at a nonexistent path. Everything else holds: 0 when the search matched, 1 when it genuinely matched nothing, 2 for a usage error, a missing archive, a missing tool or a scanner failure; --files-only and the context path agree; the session header still prints with each hit; a plain .txt session left from an older build is still searched; --since and --cwd still narrow the file set first; and no inverted index exists anywhere. --- bin/fm-transcript-reduce.py | 13 ++-- bin/fm-transcript-refresh.sh | 12 ++-- bin/fm-transcript-search.sh | 107 ++++++++++++++++++---------- bin/fm-transcript-zcat.sh | 7 +- docs/scripts.md | 2 +- docs/session-archive.md | 26 +++++-- tests/fm-transcript-archive.test.sh | 101 +++++++++++++++++++------- 7 files changed, 183 insertions(+), 85 deletions(-) diff --git a/bin/fm-transcript-reduce.py b/bin/fm-transcript-reduce.py index de45796a551..f988edc5bee 100755 --- a/bin/fm-transcript-reduce.py +++ b/bin/fm-transcript-reduce.py @@ -36,9 +36,10 @@ found no input files. An empty archive is never a silent success here. THE STORE IS COMPRESSED, AND A FULL CONTENT SCAN IS STILL THE INDEX. Each -session is written as one zstd-compressed file, `.txt.zst`, because -ripgrep reads zstd directly with `-z`: the whole store is scanned by content on -every search and no second artefact exists that could disagree with it. An +session is written as one zstd-compressed file, `.txt.zst`, and the +search decompresses every one of them into grep: the whole store is scanned by +content on every search and no second artefact exists that could disagree with +it. An inverted index is still forbidden here, for the reason it always was - it can go stale silently. Compression cannot, because a wrong decompression is an error and not a wrong @@ -145,9 +146,9 @@ def scan(text, pats): # ---------------------------------------------------------------- storage -# One session, one compressed file. The extension is what makes the store -# searchable without a wrapper: ripgrep decides how to read a file from it, so -# `.txt.zst` is read as text and a plain `.txt` still left in the store is too. +# One session, one compressed file. The extension is what tells a reader how to +# open it, so `.txt.zst` is decompressed and a plain `.txt` still left in the +# store from an older build is read as it is. SUFFIX = '.txt.zst' PLAIN_SUFFIX = '.txt' DEFAULT_LEVEL = 3 diff --git a/bin/fm-transcript-refresh.sh b/bin/fm-transcript-refresh.sh index f0fa1bb1b68..a3d6fe1eac9 100755 --- a/bin/fm-transcript-refresh.sh +++ b/bin/fm-transcript-refresh.sh @@ -7,8 +7,8 @@ # The archive is rebuilt from the raw stores every time, then the detector is # re-run against the output and required to return zero. # -# The store is written compressed, one zstd file per session, and is searched -# through the decompressor by bin/fm-transcript-search.sh. `zstd` is therefore a +# The store is written compressed, one zstd file per session, and is searched by +# bin/fm-transcript-search.sh, which decompresses each session into grep. `zstd` is therefore a # hard requirement of a rebuild: the reducer refuses rather than leaving a store # that is half compressed and half plain. A rebuild also compresses whatever # plain session files it finds already in the store, so an archive built before @@ -133,9 +133,9 @@ Every claim made from this archive travels with both bounds. The sessions are zstd-compressed, one file each. **Plain `grep -r` does not read them: it finds nothing here and says so as if the archive were empty.** Use the -wrapper, or ripgrep's own decompressing search if you want the raw tool: +wrapper, or read a session the same way it does if you want the raw tools: - rg -z 'pattern' . + zstd -dcq some-session.txt.zst | grep 'pattern' A full scan of the content is still what answers every query, so there is no index here and none is to be added. @@ -148,12 +148,12 @@ Full detail: `docs/session-archive.md` in the firstmate repository. EOF echo "== wrote $ARCHIVE/README.md (the honest bound, on the artefact)" elif grep -qE 'Plain .?grep -r' "$ARCHIVE/README.md" 2>/dev/null && - ! grep -q 'rg -z' "$ARCHIVE/README.md" 2>/dev/null; then + ! grep -q 'zstd -dcq' "$ARCHIVE/README.md" 2>/dev/null; then # This README predates compression and still tells its reader that plain # grep works here. It does not: it returns nothing, with no error, over a # full archive. The file is not overwritten because someone may have written # into it, so the correction is named instead of made. echo "WARNING: $ARCHIVE/README.md still points readers at plain grep -r, which reads" >&2 echo " nothing from a compressed store and reports no error while doing it." >&2 - echo " Correct that sentence: the wrapper, or rg -z, is what searches this archive." >&2 + echo " Correct that sentence: the wrapper, or zstd piped into grep, is what reads this archive." >&2 fi diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 3c4c107bf22..ecc24de2c50 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -17,15 +17,25 @@ # # THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches # nothing here and exits as though the archive were empty, which is the one -# answer this archive must never give. The scan therefore runs through ripgrep -# with zstd as its explicit preprocessor. `rg` and `zstd` on PATH are hard -# requirements: a missing one is reported as a missing tool, never as a search -# that found nothing. Anyone who -# would rather not use this wrapper runs `rg -z ` over the archive -# directory directly and gets the same content scan. FM_RG may name ripgrep -# elsewhere. Search deliberately has no FM_ZSTD override: one compressor knob -# governing only part of the process repeatedly passed its prerequisite check -# before the scan read nothing or the session header disappeared. +# answer this archive must never give. The scan therefore decompresses each +# session and pipes it into grep, one file at a time, spread across the +# machine's cores. +# +# THE TOOLS THIS SEARCH REQUIRES ARE `zstd`, `grep` and `xargs`, all on PATH, +# and a missing one is reported as a missing tool rather than as a search that +# found nothing. That list is deliberately short. An earlier version scanned +# with ripgrep, whose behaviour differed from machine to machine - which flags +# its build supported, whether it could decompress at all - and twice reported +# an empty result over a full archive because of it. A search tool that answers +# differently depending on the machine it runs on is the same silent +# disagreement this archive exists to refuse, so the dependency is gone rather +# than pinned. No override names any of these tools either, for the same reason: +# one knob governing only part of the process twice passed its own prerequisite +# check before the scan read nothing. +# +# Anyone who would rather not use this wrapper reads a session the same way it +# does - `zstd -dcq .txt.zst | grep ` - and gets the same +# content scan. # # The archive is this home's private material and never travels: it resolves # under $FM_HOME/data/transcripts, so a secondmate home searches its own store @@ -87,22 +97,25 @@ if [ "$present" -eq 0 ]; then exit 2 fi -# The searcher is named before anything else is done, because "no matches" and +# The tools are named before anything else is done, because "no matches" and # "the tool that reads this store is not installed" are the same output to a # caller and only one of them is true. -RG="${FM_RG:-rg}" ZSTD=zstd -PREPROCESSOR="$SCRIPT_DIR/fm-transcript-zcat.sh" -for tool in "$RG" "$ZSTD"; do +READER="$SCRIPT_DIR/fm-transcript-zcat.sh" +for tool in "$ZSTD" grep xargs; do command -v "$tool" >/dev/null 2>&1 && continue echo "$tool is not installed, and the session store is compressed: this search cannot run." >&2 - echo "Install ripgrep and zstd (apt install ripgrep zstd), or point FM_RG at ripgrep." >&2 + echo "Install zstd (apt install zstd); grep and xargs are expected on any machine that runs this." >&2 echo "Refusing rather than reporting no matches over an archive that was never read." >&2 exit 2 done -# A user ripgrep config can change what a search means - case folding, column -# limits, skipped files - and this store's answers must not depend on it. -export RIPGREP_CONFIG_PATH= +[ -x "$READER" ] || { echo "missing session reader: $READER" >&2; exit 2; } + +# One scan per core. A machine that will not say how many it has gets a modest +# default rather than an unbounded fan-out. +jobs=$(nproc 2>/dev/null || echo 4) +case "$jobs" in ''|*[!0-9]*) jobs=4;; esac +[ "$jobs" -ge 1 ] || jobs=1 # narrow the file set from the per-source index when asked filelist="$(mktemp)"; trap 'rm -f "$filelist"' EXIT @@ -134,46 +147,62 @@ n=$(wc -l < "$filelist") echo "# searching $n session files under: ${roots[*]}" >&2 [ "$n" -gt 0 ] || exit 1 -# The file set is handed over in batches. Each batch normalises a genuine -# no-match to success so xargs can keep scanning, while preserving scanner -# failures. The final output then decides whether the completed search matched. +# The file set is handed over in batches, spread across cores. Two rules hold +# inside a batch, and both were learned from a defect rather than designed in. +# +# A reader killed by SIGPIPE is not a failure. grep stops reading the moment it +# can answer - always with -q, sometimes with -C - and the decompressor feeding +# it then dies of a broken pipe. Treating that as a scanner error is what once +# returned 0 matching sessions where 75 were expected, on a full archive, so +# status 141 counts as success here while any other reader failure is real. +# +# A genuine no-match must not stop the scan. It is normalised to success inside +# the batch so xargs keeps going, and what the completed search actually found +# decides the exit status at the end. if [ "$files_only" = 1 ]; then # shellcheck disable=SC2016 - xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; preprocessor=$2; pattern=$3; shift 3 + xargs -a "$filelist" -d '\n' -P "$jobs" -n 16 bash -c ' + reader=$1; pattern=$2; shift 2 for path do - "$preprocessor" "$path" | "$rg" -q -e "$pattern" + "$reader" "$path" | grep -qE -e "$pattern" statuses=("${PIPESTATUS[@]}") - [ "${statuses[0]}" -eq 0 ] || exit 255 case ${statuses[1]} in 0) printf "%s\n" "$path" ;; 1) ;; *) exit 255 ;; esac + case ${statuses[0]} in + 0|141) ;; + *) exit 255 ;; + esac done - ' _ "$RG" "$PREPROCESSOR" "$q" | + ' _ "$READER" "$q" | + sort | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' statuses=("${PIPESTATUS[@]}") [ "${statuses[0]}" -eq 0 ] || exit 2 - exit "${statuses[1]}" + exit "${statuses[2]}" fi +# Every emitted line carries its own session path, including the separators +# between context groups, so parallel batches cannot misattribute a line. The +# stable sort then gathers each session's lines back together - equal keys keep +# their arrival order, so a session's own lines stay in file order - and the +# reader below prints one header per session. # shellcheck disable=SC2016 -xargs -a "$filelist" -d '\n' bash -c ' - rg=$1; preprocessor=$2; context=$3; pattern=$4; shift 4 +xargs -a "$filelist" -d '\n' -P "$jobs" -n 16 bash -c ' + reader=$1; context=$2; pattern=$3; shift 3 for path do - "$preprocessor" "$path" | - "$rg" -n --no-heading --color never -C "$context" -e "$pattern" | - awk -v path="$path" '\'' - /^--$/ { print; next } - { printf "%c%s%c%s\n", 28, path, 28, $0 } - '\'' + "$reader" "$path" | + grep -nE -C "$context" -e "$pattern" | + awk -v path="$path" '\''{ printf "%c%s%c%s\n", 28, path, 28, $0 }'\'' statuses=("${PIPESTATUS[@]}") - [ "${statuses[0]}" -eq 0 ] || exit 255 case ${statuses[1]} in 0|1) ;; *) exit 255;; esac + case ${statuses[0]} in 0|141) ;; *) exit 255;; esac [ "${statuses[2]}" -eq 0 ] || exit 255 done -' _ "$RG" "$PREPROCESSOR" "$ctx" "$q" | +' _ "$READER" "$ctx" "$q" | +sort -s -t "$(printf '\034')" -k2,2 | awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' function shell_quote(s, out, i, c) { out="\"" @@ -184,7 +213,6 @@ awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' } return out "\"" } - /^--$/ { print " --"; next } substr($0, 1, 1) != sprintf("%c", 28) { next } { record=substr($0, 2) @@ -206,9 +234,10 @@ awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' last=path found=1 } - print " " rest + if (rest == "--") print " --" + else print " " rest } END { exit(found ? 0 : 1) }' statuses=("${PIPESTATUS[@]}") [ "${statuses[0]}" -eq 0 ] || exit 2 -exit "${statuses[1]}" +exit "${statuses[2]}" diff --git a/bin/fm-transcript-zcat.sh b/bin/fm-transcript-zcat.sh index bc6535f090a..81f126810d5 100755 --- a/bin/fm-transcript-zcat.sh +++ b/bin/fm-transcript-zcat.sh @@ -1,5 +1,10 @@ #!/usr/bin/env bash -# ripgrep --pre adapter for compressed and retained plain session files. +# fm-transcript-zcat.sh - read one session file from the archive, whatever +# shape it is in: compressed sessions come back through zstd, and a plain one +# left from a store built before compression comes back as it is. +# +# One owner for that decision, because the scan and the session-header read +# must never disagree about how a file is read. set -uo pipefail case $1 in diff --git a/docs/scripts.md b/docs/scripts.md index b119c60e853..bc5bfbf3df7 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -48,7 +48,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-memory-alarm.sh` | Wake the fleet when this machine is running out of RAM headroom, on headroom and on growth, naming the process responsible with its account and the work it serves; it reads the attribution reading and nothing else, sets no limit and kills nothing, and reports an instrument it could not read as blindness rather than an all-clear (docs/memory-alarm.md) | | `fm-memory-ceiling-probe.sh` | Measure whether a memory ceiling on this host would manufacture the very pressure an alarm above it exists to detect, by running the same file-reading workload with and without one; it sets no lasting limit and kills nothing (docs/memory-ceiling-caveat.md) | | `fm-transcript-reduce.py` | Build this home's reduced, redacted session derivative with one of two readers selected by `--source`, writing one compressed file per session, or re-verify an existing one to zero through the decompressor; refuses a source that disagrees with the material's shape, or a missing compressor, rather than writing the empty or half-compressed archive either would produce (docs/session-archive.md) | -| `fm-transcript-search.sh` | Search that derivative with context, narrowing the file set by date and working directory first; a full content scan is the index, so there is none to go stale, the compressed store is read through ripgrep's decompressor rather than by plain grep, and an archive or a search tool that is not there reads as absent rather than as no matches (docs/session-archive.md) | +| `fm-transcript-search.sh` | Search that derivative with context, narrowing the file set by date and working directory first; a full content scan is the index, so there is none to go stale, the compressed store is decompressed into grep one session at a time across cores rather than read by plain grep, and an archive or a required tool that is not there reads as absent rather than as no matches (docs/session-archive.md) | | `fm-transcript-refresh.sh` | Rebuild both stores from this vessel's own raw transcripts, converge any plain session files left from an older build into the compressed store, verify the output to zero, and leave the honest bound on the artefact; the raw stores are read-only inputs and a store this vessel does not have is skipped, never counted as empty (docs/session-archive.md) | | `fm-herdr-lab.sh` | Provision and guardedly operate an isolated, never-default Herdr lab session | | `fm-install-herdr.sh` | Install CI's exact-version Herdr pin with official asset URL, SHA-256, and protocol checks | diff --git a/docs/session-archive.md b/docs/session-archive.md index 8536f38726a..69efe637af0 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -39,7 +39,7 @@ bin/fm-transcript-search.sh 'pattern' --files-only ``` **A full content scan is the index.** -The archive is UTF-8 text laid out one file per session, and a scan of the whole store was measured at 0.63 to 0.92 s over the compressed 48.9 MB on 2026-08-18, so no inverted index exists and none should be added. +The archive is UTF-8 text laid out one file per session, and a scan of the whole store was measured at 1.44 to 1.48 s over the compressed 48.9 MB on 2026-08-19, so no inverted index exists and none should be added. An index is a component that can be silently out of date, which is exactly the failure this archive was built against. Compression does not reintroduce that failure and is why it was worth doing: the compressed file is the content, not a summary of it, and a decompression that goes wrong is an error rather than a wrong answer. `_index.tsv` is not a search index: it narrows the file set before the scan runs, and only when `--since` or `--cwd` asks it to. @@ -47,15 +47,27 @@ Compression does not reintroduce that failure and is why it was worth doing: the **Plain `grep -r` no longer reads this archive.** It matches nothing in a compressed session and exits reporting no matches, over an archive that is full - the same silent emptiness the whole design refuses, arriving through the documentation instead of through the reader. It is worse than wholly blind: `_index.tsv` is the one plain file left, so a phrase that happens to sit in its first-user-message column still matches, and the blindness looks selective rather than total. -Anyone who wants the raw tool rather than the wrapper uses ripgrep's own decompressing scan, which reads exactly what the wrapper reads: +Anyone who wants the raw tools rather than the wrapper reads a session exactly as the wrapper does: ```sh -rg -z 'pattern' "$FM_HOME/data/transcripts" +zstd -dcq some-session.txt.zst | grep 'pattern' ``` -`rg` and `zstd` are therefore hard requirements of a search here, and the wrapper reports a missing one as a missing tool rather than as a search that found nothing. -`FM_RG` may name ripgrep elsewhere, but search requires `zstd` on PATH and deliberately has no `FM_ZSTD` override. -A single compressor knob that governed only part of the search process repeatedly let its prerequisite check pass before the scan read nothing or the session header disappeared, so the partial override was withdrawn rather than allowed to remain misleading. +### The tools a search requires + +`zstd`, `grep` and `xargs`, on `PATH`. +That is the whole list, and the wrapper reports a missing one as a missing tool rather than as a search that found nothing. +No override names any of them, because a knob governing only part of the process twice passed its own prerequisite check before the scan read nothing. + +The list is short deliberately. +An earlier version scanned with ripgrep and `-z`; it was withdrawn on 2026-08-19 on measurement rather than taste. +Ripgrep behaved differently on this seat and on the CI runner - which flags the build accepted, whether it could decompress at all - and twice reported an empty result over a full archive. +A search tool that answers differently depending on the machine it runs on is the same silent disagreement this archive exists to refuse, so the dependency was removed rather than pinned to a version. +`zstd` is already required to build the store, and `grep` and `xargs` are on any machine that can run this repository's tests at all. + +**A scan that stops reading early is ordinary and must never be read as a failure.** +`grep` stops as soon as it can answer, the decompressor feeding it then dies of a broken pipe, and an implementation that counts that as a scanner error returns nothing over a full archive: measured on 2026-08-19 at 0 matching sessions where 75 were expected, with every test still passing, because a one-session fixture is too small to make anything stop early. +`tests/fm-transcript-archive.test.sh` now plants that case - a dozen sessions with the match at the top and a long tail behind it - and requires all twelve back. ## Rebuild @@ -137,7 +149,7 @@ Codex 418.7 MB 143.6 MB 21.7 MB 2.9:1 37 combined 1426.3 MB 260.2 MB 48.9 MB 5.5:1 110 sessions 2273 full rebuild + verification ~129 s -full-content search over the store, three queries 0.63 - 0.92 s +full-content search over the store, whole store 1.44 - 1.48 s verification residual hits 0 ``` diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 7559d466135..a34d17810cf 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -511,30 +511,70 @@ test_search_reads_the_compressed_store() { pass "search reads the compressed store and still reports the session header with the hit" } -test_search_does_not_depend_on_ripgrep_compression_flags() { - local home out wrapper real_rg +# The case that broke twice in CI was a machine whose ripgrep did not behave +# like this one's. The search no longer uses ripgrep at all, and that is proven +# by running it on a PATH where ripgrep genuinely cannot be resolved - not by +# stubbing a flag, which is what let the defect through the previous two rounds. +# THE CASE EVERY OTHER FIXTURE MISSED. A scan that stops reading as soon as it +# can answer kills the decompressor feeding it with SIGPIPE. Treating that +# ordinary event as a scanner failure returned 0 matching sessions where 75 were +# expected, over a full archive, and no test saw it: a one-session fixture with +# one match near the end fits in the pipe buffer, so nothing ever short-circuits. +# This fixture puts the match at the TOP of each session and a long tail behind +# it, which is what forces the reader to stop early while the decompressor is +# still writing. If the defect returns, this is what fails. +test_a_scan_that_stops_early_still_reports_every_session() { + local store i out rc=0 count expected=12 + store="$TMP/early-stop/claude-redacted" + rm -rf "$TMP/early-stop" + mkdir -p "$store" + for i in $(seq 1 "$expected"); do + { + printf '# session big%s.jsonl\n# cwd /home/x/p%s\n' "$i" "$i" + printf '# span 2026-08-18 10:00:00 .. 2026-08-18 10:05:00\n\n' + printf ' the sought phrase sits near the top of this session\n' + yes ' a long tail that keeps the decompressor writing well past the match' | + head -20000 + } | zstd -q -3 -o "$store/big$i.txt.zst" + done + + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/early-stop" "$SEARCH" 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "a search matching every session must exit 0, got $rc" + count=$(printf '%s\n' "$out" | grep -c 'big[0-9]*\.txt\.zst') + [ "$count" -eq "$expected" ] \ + || fail "--files-only reported $count of $expected sessions; a scan that stops early lost the rest" + + rc=0 + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/early-stop" "$SEARCH" 'sought phrase' -C 1 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "the context path must exit 0 when every session matched, got $rc" + count=$(printf '%s\n' "$out" | grep -c '^=== ') + [ "$count" -eq "$expected" ] \ + || fail "the context path reported $count of $expected sessions" + pass "every session is reported when the scan stops reading each one early" +} + +test_search_works_on_a_machine_without_ripgrep() { + local home out bindir tool home=$(setup_fixture_home) - wrapper="$TMP/rg-without-z" - real_rg=$(command -v rg) - # Model the older runner ripgrep that made --pre return no hits, as well as a - # build without implicit -z decompression. The wrapper's public result must - # come from the explicit zstd pipeline supported by both versions. - cat >"$wrapper" <<'EOF' -#!/usr/bin/env bash -for arg in "$@"; do - case "$arg" in - -z|-*z*|--pre) exit 1 ;; - esac -done -exec "$FM_TEST_REAL_RG" "$@" -EOF - chmod +x "$wrapper" - out=$(FM_HOME="$home" FM_RG="$wrapper" FM_TEST_REAL_RG="$real_rg" \ - "$SEARCH" 'tugboat host' 2>/dev/null) \ - || fail 'search depended on ripgrep implicit decompression instead of the required zstd tool' + bindir="$TMP/no-rg-path" + rm -rf "$bindir" + mkdir -p "$bindir" + # A PATH holding exactly what the search is allowed to need, and nothing else. + # type -P resolves the executable itself: an interactive shell may carry a + # function or alias by the same name, and linking that name to itself makes a + # fixture that proves nothing. + for tool in bash sh zstd grep xargs awk sed sort find wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + PATH="$bindir" type -P rg >/dev/null 2>&1 \ + && fail 'the no-ripgrep fixture PATH still resolves ripgrep, so it proves nothing' + out=$(PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search failed on a machine with no ripgrep on PATH' assert_contains "$out" 'tugboat host' \ - 'external zstd preprocessing must return the compressed hit' - pass "search does not depend on ripgrep's version-dependent compression flags" + 'the hit must come back from a machine that has no ripgrep at all' + pass "search runs and finds its hit on a PATH where ripgrep cannot be resolved" } test_search_reads_plain_sessions_during_migration() { @@ -599,9 +639,19 @@ test_search_reports_scanner_failures_as_errors() { } test_search_without_its_tool_refuses_rather_than_finding_nothing() { - local home out rc=0 + local home out rc=0 bindir tool home=$(setup_fixture_home) - out=$(FM_HOME="$home" FM_RG="$TMP/no-such-rg" "$SEARCH" 'tugboat host' 2>&1) || rc=$? + # A PATH with everything except the decompressor: the store cannot be read, + # and the caller must be told that rather than told there were no matches. + bindir="$TMP/no-zstd-path" + rm -rf "$bindir" + mkdir -p "$bindir" + for tool in bash sh grep xargs awk sed sort find wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + out=$(PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' 2>&1) || rc=$? [ "$rc" -eq 2 ] || fail "a missing search tool must be reported as such, got exit $rc" assert_contains "$out" 'not installed' 'the refusal must name the missing tool' case "$out" in @@ -655,7 +705,8 @@ test_search_formatter_is_portable_to_mawk test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store -test_search_does_not_depend_on_ripgrep_compression_flags +test_search_works_on_a_machine_without_ripgrep +test_a_scan_that_stops_early_still_reports_every_session test_search_reads_plain_sessions_during_migration test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched From 3e6d47781c18d88dc84895101c52b555f2962fa1 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 21:52:12 +0000 Subject: [PATCH 13/18] no-mistakes(document): Correct stale session search timing --- bin/fm-transcript-search.sh | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index ecc24de2c50..5cf9bebdd3e 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -9,11 +9,12 @@ # line with n lines of context, and the derivative path. # # A FULL CONTENT SCAN IS THE INDEX. The archive is UTF-8 text laid out one file -# per session, zstd-compressed, and a scan of the whole store costs well under a -# second, so no inverted index exists and none should be added: a search index is -# a component that can be silently out of date, which is the exact failure this -# archive was built against. `_index.tsv` is not a search index - it narrows the -# FILE SET before the scan runs, and only when --since or --cwd asks it to. +# per session, zstd-compressed, and a scan of the whole store costs about one and +# a half seconds. No inverted index exists and none should be added: a search +# index is a component that can be silently out of date, which is the exact +# failure this archive was built against. `_index.tsv` is not a search index - it +# narrows the FILE SET before the scan runs, and only when --since or --cwd asks +# it to. # # THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches # nothing here and exits as though the archive were empty, which is the one From 87e8814b6ad97a6e8cf1ec64fb522147adca5f4d Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 23:09:55 +0000 Subject: [PATCH 14/18] Read the reader's exit status only where it carries information CI caught one real assertion: a search matching every session exited 2. It is the mirror of the defect this branch already fixed once - the old search reported success while finding nothing, this one found everything and reported failure - and both make the exit status lie to every caller that trusts it. The cause was accepting only signal 141 from the decompressor when its consumer stopped reading early. This machine's zstd dies on the signal; the CI runner's reports the closed pipe as an ordinary error, so a full match became a scanner failure there while passing here. The rule is now stated rather than pattern-matched: the reader's exit status means something only when the consumer read to the end. When grep reports a match it deliberately stopped reading, so its reader's status says nothing about the file and is not consulted. When grep read the session through and found nothing, the reader had every chance to finish, so a non-zero status is a genuine failure to read the store and stays an error. The status is not masked - masking it would leave a search that can no longer report a real failure at all - it is read only where it carries information. tests/fm-transcript-archive.test.sh runs the real script against a reader that hides the signal the way the runner's does, and that test fails against the previous rule and passes against this one. A deliberately corrupt store file is proven to still exit non-zero, on both search paths. Verifying that fix on the live archive surfaced two further defects in the parallel scan, both of which made the search disagree with itself rather than merely with its exit status: Parallel workers wrote into one shared pipe, and buffered writes interleave at block boundaries rather than line boundaries. Three runs of one query returned 8121, 8105 and 8104 lines with header lines cut mid-path. Each worker now writes its own file, which the parent gathers afterwards, so the same query returns the same answer every time. grep called five sessions binary and printed no lines for them, because a reduced session quotes material that carries the occasional control byte. --files-only reported those sessions and the context path did not. The scan now reads the store as text. Measured on the live 2273-session archive against ground truth taken by decompressing every session separately: no-mistakes 2048 sessions, 88294 matching lines exact forgejo 123 sessions, 2306 matching lines exact Grossreinschiff 75 sessions, 232 matching lines exact exit code 127 1 session, 2 matching lines exact three runs of one query identical Whole-store timings move with the extra correctness: 1.50 - 1.54 s for the file list and 2.31 - 2.64 s with context, and docs/session-archive.md carries those rather than the earlier single figure. The compressor-override mismatch is now recorded as a known condition where a reader of FM_ZSTD meets it: it governs building and verifying, the search uses the zstd on PATH, and the two have to be able to read each other. --- bin/fm-transcript-reduce.py | 7 +++ bin/fm-transcript-search.sh | 76 +++++++++++++++-------- docs/session-archive.md | 7 ++- tests/fm-transcript-archive.test.sh | 95 +++++++++++++++++++++++++++++ 4 files changed, 155 insertions(+), 30 deletions(-) diff --git a/bin/fm-transcript-reduce.py b/bin/fm-transcript-reduce.py index f988edc5bee..abf5daed999 100755 --- a/bin/fm-transcript-reduce.py +++ b/bin/fm-transcript-reduce.py @@ -51,6 +51,13 @@ plain, which is exactly the kind of quiet disagreement this archive exists to avoid. +KNOWN CONDITION ON FM_ZSTD: it is what lets a store be built on a machine where +zstd is not on PATH, and it governs building and verifying only. The search +always uses the zstd on PATH, so a store written by an overridden compressor is +read back by a different one, and the two have to be able to read each other. +Ordinary zstd builds can; pointing this at something that is not zstd-compatible +leaves a store nothing else can open. + Usage: fm-transcript-reduce.py --source claude|codex --in DIR --out DIR [--patterns FILE] [--truncate 400] [--limit N] diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index 5cf9bebdd3e..c714f2ec761 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -10,7 +10,7 @@ # # A FULL CONTENT SCAN IS THE INDEX. The archive is UTF-8 text laid out one file # per session, zstd-compressed, and a scan of the whole store costs about one and -# a half seconds. No inverted index exists and none should be added: a search +# a half seconds for the file list, or two and a half with context. No inverted index exists and none should be added: a search # index is a component that can be silently out of date, which is the exact # failure this archive was built against. `_index.tsv` is not a search index - it # narrows the FILE SET before the scan runs, and only when --since or --cwd asks @@ -119,7 +119,14 @@ case "$jobs" in ''|*[!0-9]*) jobs=4;; esac [ "$jobs" -ge 1 ] || jobs=1 # narrow the file set from the per-source index when asked -filelist="$(mktemp)"; trap 'rm -f "$filelist"' EXIT +filelist="$(mktemp)" +# Each parallel worker writes its own file rather than into a shared pipe. +# Buffered writes from concurrent workers interleave at block boundaries, not at +# line boundaries, which split lines in half: measured on this archive as three +# runs of one query returning 8121, 8105 and 8104 lines, with header lines cut +# mid-path. A search whose answer changes between runs is not a search. +partdir="$(mktemp -d)" +trap 'rm -rf "$filelist" "$partdir"' EXIT for r in "${roots[@]}"; do [ -d "$r" ] || continue idx="$r/_index.tsv" @@ -151,11 +158,22 @@ echo "# searching $n session files under: ${roots[*]}" >&2 # The file set is handed over in batches, spread across cores. Two rules hold # inside a batch, and both were learned from a defect rather than designed in. # -# A reader killed by SIGPIPE is not a failure. grep stops reading the moment it -# can answer - always with -q, sometimes with -C - and the decompressor feeding -# it then dies of a broken pipe. Treating that as a scanner error is what once -# returned 0 matching sessions where 75 were expected, on a full archive, so -# status 141 counts as success here while any other reader failure is real. +# THE READER'S EXIT STATUS ONLY MEANS SOMETHING WHEN THE CONSUMER READ TO THE +# END, and that is the whole rule. grep stops reading the moment it can answer, +# and the decompressor feeding it then dies on the closed pipe - as signal 141 +# here, as a write error on another machine, as whatever the local zstd build +# does. None of those say anything about the file, so when grep reports a match +# the reader's status is not consulted. When grep read the session through and +# found nothing, the reader had every chance to finish, so a non-zero status +# there is a genuine failure to read the store and stays an error. This is not +# the status being ignored, it is the status being read only where it carries +# information. +# +# Both ways of getting this wrong have been measured on this branch. Requiring +# the reader to exit 0 returned 0 matching sessions where 75 were expected, over +# a full archive. Accepting only signal 141 made a search that matched every +# session exit 2 on a machine whose zstd reports a closed pipe differently. A +# search that lies about whether it worked is the same defect either way round. # # A genuine no-match must not stop the scan. It is normalised to success inside # the batch so xargs keeps going, and what the completed search actually found @@ -163,28 +181,30 @@ echo "# searching $n session files under: ${roots[*]}" >&2 if [ "$files_only" = 1 ]; then # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' -P "$jobs" -n 16 bash -c ' - reader=$1; pattern=$2; shift 2 + reader=$1; pattern=$2; partdir=$3; shift 3 + part="$partdir/part.$$" for path do - "$reader" "$path" | grep -qE -e "$pattern" + "$reader" "$path" | grep -qaE -e "$pattern" statuses=("${PIPESTATUS[@]}") case ${statuses[1]} in - 0) printf "%s\n" "$path" ;; - 1) ;; - *) exit 255 ;; - esac - case ${statuses[0]} in - 0|141) ;; + 0) printf "%s\n" "$path" >>"$part" ;; + 1) [ "${statuses[0]}" -eq 0 ] || exit 255 ;; *) exit 255 ;; esac done - ' _ "$READER" "$q" | + ' _ "$READER" "$q" "$partdir" || exit 2 + find "$partdir" -type f -exec cat {} + 2>/dev/null | sort | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' - statuses=("${PIPESTATUS[@]}") - [ "${statuses[0]}" -eq 0 ] || exit 2 - exit "${statuses[2]}" + exit "${PIPESTATUS[2]}" fi +# The scan reads every session as text (-a). A reduced session is UTF-8 by +# construction, but a stray control byte from the material it quotes makes grep +# call the whole file binary and print no lines at all - measured here as five +# sessions that contain the term reported by --files-only and missing from the +# context output, which is a search quietly disagreeing with itself. +# # Every emitted line carries its own session path, including the separators # between context groups, so parallel batches cannot misattribute a line. The # stable sort then gathers each session's lines back together - equal keys keep @@ -192,17 +212,21 @@ fi # reader below prints one header per session. # shellcheck disable=SC2016 xargs -a "$filelist" -d '\n' -P "$jobs" -n 16 bash -c ' - reader=$1; context=$2; pattern=$3; shift 3 + reader=$1; context=$2; pattern=$3; partdir=$4; shift 4 + part="$partdir/part.$$" for path do "$reader" "$path" | - grep -nE -C "$context" -e "$pattern" | - awk -v path="$path" '\''{ printf "%c%s%c%s\n", 28, path, 28, $0 }'\'' + grep -naE -C "$context" -e "$pattern" | + awk -v path="$path" '\''{ printf "%c%s%c%s\n", 28, path, 28, $0 }'\'' >>"$part" statuses=("${PIPESTATUS[@]}") case ${statuses[1]} in 0|1) ;; *) exit 255;; esac - case ${statuses[0]} in 0|141) ;; *) exit 255;; esac + # grep -C reads the session through whether or not it matched, so here the + # reader always had its chance and its status always carries information. + [ "${statuses[0]}" -eq 0 ] || exit 255 [ "${statuses[2]}" -eq 0 ] || exit 255 done -' _ "$READER" "$ctx" "$q" | +' _ "$READER" "$ctx" "$q" "$partdir" || exit 2 +find "$partdir" -type f -exec cat {} + 2>/dev/null | sort -s -t "$(printf '\034')" -k2,2 | awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' function shell_quote(s, out, i, c) { @@ -239,6 +263,4 @@ awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' else print " " rest } END { exit(found ? 0 : 1) }' -statuses=("${PIPESTATUS[@]}") -[ "${statuses[0]}" -eq 0 ] || exit 2 -exit "${statuses[2]}" +exit "${PIPESTATUS[2]}" diff --git a/docs/session-archive.md b/docs/session-archive.md index 69efe637af0..33b1f06b9c5 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -39,7 +39,7 @@ bin/fm-transcript-search.sh 'pattern' --files-only ``` **A full content scan is the index.** -The archive is UTF-8 text laid out one file per session, and a scan of the whole store was measured at 1.44 to 1.48 s over the compressed 48.9 MB on 2026-08-19, so no inverted index exists and none should be added. +The archive is UTF-8 text laid out one file per session, and a scan of the whole store was measured at 1.50 to 1.54 s for the file list, and 2.31 to 2.64 s with context, over the compressed 48.9 MB on 2026-08-19, so no inverted index exists and none should be added. An index is a component that can be silently out of date, which is exactly the failure this archive was built against. Compression does not reintroduce that failure and is why it was worth doing: the compressed file is the content, not a summary of it, and a decompression that goes wrong is an error rather than a wrong answer. `_index.tsv` is not a search index: it narrows the file set before the scan runs, and only when `--since` or `--cwd` asks it to. @@ -88,7 +88,8 @@ A raw store that does not exist on this vessel is reported and skipped, never tr Each session is written as one zstd file at level 3, which took this seat's store from 271.2 MB to 52.1 MB on 2026-08-18. `zstd` is a hard requirement of a rebuild rather than a preference: a run that cannot compress refuses before writing anything, because a store that is half compressed and half plain is a store whose answers depend on which half a question lands in. -`FM_ZSTD` may name the compressor used specifically for building and verifying the store; it does not apply to searching. +`FM_ZSTD` may name the compressor used specifically for building and verifying the store; it does not apply to searching, and it exists because it is the only way to build a store on a machine where `zstd` is not on `PATH`. +That split is a known condition rather than an oversight: a store written by an overridden compressor is read back by the `zstd` the search finds on `PATH`, so the two have to be able to read each other, which ordinary zstd builds do and something that is not zstd-compatible does not. A rebuild also compresses whatever plain session files it finds already in the store and drops the plain copies it has just superseded, so an archive built before compression converges on the first refresh instead of being stranded, and a retained session the rebuild cannot reach is compressed where it lies rather than left behind. The level is a knob, `--level`, and the measurement behind the default is below. @@ -149,7 +150,7 @@ Codex 418.7 MB 143.6 MB 21.7 MB 2.9:1 37 combined 1426.3 MB 260.2 MB 48.9 MB 5.5:1 110 sessions 2273 full rebuild + verification ~129 s -full-content search over the store, whole store 1.44 - 1.48 s +full-content search, file list / with context 1.50 - 1.54 s / 2.31 - 2.64 s verification residual hits 0 ``` diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index a34d17810cf..ee9581500fa 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -515,6 +515,77 @@ test_search_reads_the_compressed_store() { # like this one's. The search no longer uses ripgrep at all, and that is proven # by running it on a PATH where ripgrep genuinely cannot be resolved - not by # stubbing a flag, which is what let the defect through the previous two rounds. +# The machine this was written on reports a closed pipe as signal 141. The CI +# runner's zstd reports it as an ordinary error instead, and a search that +# matched EVERY session therefore exited 2 there while passing here. So the rule +# is tested against a reader that behaves like the runner's rather than like +# this seat's: the search script itself runs, with only its sibling reader +# swapped for one that swallows the signal and reports a plain failure. +test_a_reader_that_hides_the_signal_still_reports_a_match() { + local store bindir i rc=0 out count expected=6 + store="$TMP/odd-reader/claude-redacted" + rm -rf "$TMP/odd-reader" + mkdir -p "$store" + for i in $(seq 1 "$expected"); do + { + printf '# session odd%s.jsonl\n# cwd /home/x/odd%s\n' "$i" "$i" + printf '# span 2026-08-19 10:00:00 .. 2026-08-19 10:05:00\n\n' + printf ' the sought phrase sits near the top of this session\n' + yes ' a long tail that keeps the decompressor writing well past the match' 2>/dev/null | + head -20000 + } | zstd -q -3 -o "$store/odd$i.txt.zst" + done + + bindir="$TMP/odd-reader-bin" + rm -rf "$bindir" + mkdir -p "$bindir" + cp "$SEARCH" "$bindir/fm-transcript-search.sh" + cat >"$bindir/fm-transcript-zcat.sh" <<'EOF' +#!/usr/bin/env bash +# Models a decompressor that reports a closed pipe as an ordinary failure +# instead of dying on the signal, which is what the CI runner's zstd does. +zstd -dcq -- "$1" 2>/dev/null || exit 1 +EOF + chmod +x "$bindir/fm-transcript-search.sh" "$bindir/fm-transcript-zcat.sh" + + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/odd-reader" "$bindir/fm-transcript-search.sh" \ + 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] \ + || fail "a search matching every session must exit 0 even when the reader hides the signal, got $rc" + count=$(printf '%s\n' "$out" | grep -c 'odd[0-9]*\.txt\.zst') + [ "$count" -eq "$expected" ] || fail "reported $count of $expected sessions" + pass "a reader that reports the closed pipe as a plain failure does not turn a full match into an error" +} + +# A reduced session is UTF-8 by construction, but it quotes material that is +# not always clean, and one stray control byte makes grep call the whole file +# binary and print no lines for it. Measured on the live archive: five sessions +# that --files-only reported as containing the term were missing from the +# context output of the same query. A search that answers one way per session +# and another way per line is disagreeing with itself. +test_a_session_with_a_control_byte_is_still_reported() { + local store out rc=0 + store="$TMP/binary-byte/claude-redacted" + rm -rf "$TMP/binary-byte" + mkdir -p "$store" + { + printf '# session odd.jsonl\n# cwd /home/x/odd\n' + printf '# span 2026-08-19 10:00:00 .. 2026-08-19 10:05:00\n\n' + printf ' a quoted blob follows: \001\002\003 and then text\n' + printf ' the sought phrase is on its own line here\n' + } | zstd -q -3 -o "$store/odd.txt.zst" + + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/binary-byte" "$SEARCH" 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "--files-only must find a session carrying a control byte, got exit $rc" + assert_contains "$out" 'odd.txt.zst' '--files-only must name the session' + rc=0 + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/binary-byte" "$SEARCH" 'sought phrase' -C 1 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "the context path must find it too, got exit $rc" + assert_contains "$out" 'sought phrase' \ + 'the matching line must be printed, not swallowed as an unprintable file' + pass "a session carrying a control byte is reported by both search paths, not silently skipped" +} + # THE CASE EVERY OTHER FIXTURE MISSED. A scan that stops reading as soon as it # can answer kills the decompressor feeding it with SIGPIPE. Treating that # ordinary event as a scanner failure returned 0 matching sessions where 75 were @@ -638,6 +709,27 @@ test_search_reports_scanner_failures_as_errors() { pass "scanner failures are errors while an ordinary no-match remains distinct" } +# The other half of the reader-status rule, and the half that is easy to lose: +# once a search stops treating a closed pipe as a failure, it must still fail on +# a session file it genuinely could not read. A store file that is not valid +# compressed data is exactly that, and it must never pass as a session that +# simply held no match. +test_search_fails_on_a_store_file_it_cannot_read() { + local store rc=0 out + store="$TMP/corrupt-store/claude-redacted" + rm -rf "$TMP/corrupt-store" + mkdir -p "$store" + printf 'this is not compressed data at all\n' >"$store/broken.txt.zst" + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'anything at all' --files-only 2>&1 >/dev/null) || rc=$? + [ "$rc" -eq 2 ] \ + || fail "a store file that cannot be read must be a scanner error, got exit $rc" + rc=0 + FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'anything at all' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] \ + || fail "the context path must also fail on an unreadable store file, got exit $rc" + pass "an unreadable session file is a scanner error, not a session that held no match" +} + test_search_without_its_tool_refuses_rather_than_finding_nothing() { local home out rc=0 bindir tool home=$(setup_fixture_home) @@ -707,9 +799,12 @@ test_search_reports_a_missing_archive_rather_than_no_matches test_search_reads_the_compressed_store test_search_works_on_a_machine_without_ripgrep test_a_scan_that_stops_early_still_reports_every_session +test_a_reader_that_hides_the_signal_still_reports_a_match +test_a_session_with_a_control_byte_is_still_reported test_search_reads_plain_sessions_during_migration test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched test_search_reports_scanner_failures_as_errors +test_search_fails_on_a_store_file_it_cannot_read test_search_without_its_tool_refuses_rather_than_finding_nothing test_refresh_names_a_readme_that_still_promises_plain_grep From 2b40c373ab9ac60aeedfee1d014365abe66dfea7 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 23:12:55 +0000 Subject: [PATCH 15/18] no-mistakes(review): Fail closed on corrupt matching sessions --- bin/fm-transcript-search.sh | 23 ++++++++++------------- tests/fm-transcript-archive.test.sh | 22 ++++++++++++++++++++-- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index c714f2ec761..a3b1f12bd0b 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -158,16 +158,11 @@ echo "# searching $n session files under: ${roots[*]}" >&2 # The file set is handed over in batches, spread across cores. Two rules hold # inside a batch, and both were learned from a defect rather than designed in. # -# THE READER'S EXIT STATUS ONLY MEANS SOMETHING WHEN THE CONSUMER READ TO THE -# END, and that is the whole rule. grep stops reading the moment it can answer, -# and the decompressor feeding it then dies on the closed pipe - as signal 141 -# here, as a write error on another machine, as whatever the local zstd build -# does. None of those say anything about the file, so when grep reports a match -# the reader's status is not consulted. When grep read the session through and -# found nothing, the reader had every chance to finish, so a non-zero status -# there is a genuine failure to read the store and stays an error. This is not -# the status being ignored, it is the status being read only where it carries -# information. +# EVERY SESSION IS READ THROUGH, so a non-zero reader status is always a genuine +# failure to read the store. grep must not stop at the first match: doing so +# closes the reader's pipe early and makes its status describe the closed pipe +# rather than the file. Both search paths therefore consume the whole session +# before deciding whether it matched. # # Both ways of getting this wrong have been measured on this branch. Requiring # the reader to exit 0 returned 0 matching sessions where 75 were expected, over @@ -184,13 +179,15 @@ if [ "$files_only" = 1 ]; then reader=$1; pattern=$2; partdir=$3; shift 3 part="$partdir/part.$$" for path do - "$reader" "$path" | grep -qaE -e "$pattern" + "$reader" "$path" | grep -aE -e "$pattern" >/dev/null statuses=("${PIPESTATUS[@]}") case ${statuses[1]} in - 0) printf "%s\n" "$path" >>"$part" ;; - 1) [ "${statuses[0]}" -eq 0 ] || exit 255 ;; + 0) matched=1 ;; + 1) matched=0 ;; *) exit 255 ;; esac + [ "${statuses[0]}" -eq 0 ] || exit 255 + [ "$matched" -eq 0 ] || printf "%s\n" "$path" >>"$part" done ' _ "$READER" "$q" "$partdir" || exit 2 find "$partdir" -type f -exec cat {} + 2>/dev/null | diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index ee9581500fa..9672b41d4bc 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -715,7 +715,7 @@ test_search_reports_scanner_failures_as_errors() { # compressed data is exactly that, and it must never pass as a session that # simply held no match. test_search_fails_on_a_store_file_it_cannot_read() { - local store rc=0 out + local store rc=0 out size store="$TMP/corrupt-store/claude-redacted" rm -rf "$TMP/corrupt-store" mkdir -p "$store" @@ -727,7 +727,25 @@ test_search_fails_on_a_store_file_it_cannot_read() { FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'anything at all' >/dev/null 2>&1 || rc=$? [ "$rc" -eq 2 ] \ || fail "the context path must also fail on an unreadable store file, got exit $rc" - pass "an unreadable session file is a scanner error, not a session that held no match" + + rm -f "$store/broken.txt.zst" + { + printf 'sought phrase near the top\n' + seq 1 10000 + } | zstd -q -3 -o "$store/truncated.txt.zst" + size=$(wc -c <"$store/truncated.txt.zst") + truncate -s "$((size - 1))" "$store/truncated.txt.zst" + rc=0 + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 2 ] \ + || fail "a truncated matching session must be a scanner error in --files-only, got exit $rc" + [ -z "$out" ] \ + || fail "a truncated matching session must not be reported as a valid match" + rc=0 + FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'sought phrase' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] \ + || fail "the context path must reject a truncated matching session, got exit $rc" + pass "unreadable sessions fail whether or not their readable prefix matches" } test_search_without_its_tool_refuses_rather_than_finding_nothing() { From 50385bf9a3e741588b6e4fa5a119e7b43945a882 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 23:14:44 +0000 Subject: [PATCH 16/18] no-mistakes(review): Document full-read corruption detection invariant --- docs/session-archive.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/session-archive.md b/docs/session-archive.md index 33b1f06b9c5..c6ff5fbf89a 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -65,9 +65,11 @@ Ripgrep behaved differently on this seat and on the CI runner - which flags the A search tool that answers differently depending on the machine it runs on is the same silent disagreement this archive exists to refuse, so the dependency was removed rather than pinned to a version. `zstd` is already required to build the store, and `grep` and `xargs` are on any machine that can run this repository's tests at all. -**A scan that stops reading early is ordinary and must never be read as a failure.** -`grep` stops as soon as it can answer, the decompressor feeding it then dies of a broken pipe, and an implementation that counts that as a scanner error returns nothing over a full archive: measured on 2026-08-19 at 0 matching sessions where 75 were expected, with every test still passing, because a one-session fixture is too small to make anything stop early. -`tests/fm-transcript-archive.test.sh` now plants that case - a dozen sessions with the match at the top and a long tail behind it - and requires all twelve back. +**Both search paths read every session through to the end.** +The decompressor's exit status is therefore always meaningful, and any non-zero status is a genuine failure to read the store. +Requiring the reader to exit 0 while the scan stopped at the first match returned 0 matching sessions where 75 were expected over a full archive, while accepting only signal 141 made a search that matched every session exit 2 on a machine whose zstd reports a closed pipe differently. +Reading every session through is what makes a corrupt store file detectable on both paths, including one whose valid prefix contains the match. +`tests/fm-transcript-archive.test.sh` proves that invariant with a compressed session whose matching valid prefix is followed by a truncated frame. ## Rebuild From 0a2af3687f2b8954b7d960ace0fbe6a3a0f8ef36 Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 23:16:47 +0000 Subject: [PATCH 17/18] no-mistakes(review): Fail closed on output pipeline errors --- bin/fm-transcript-search.sh | 12 ++++++++++-- tests/fm-transcript-archive.test.sh | 21 +++++++++++++++++++++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index a3b1f12bd0b..ddcc7501c1c 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -193,7 +193,11 @@ if [ "$files_only" = 1 ]; then find "$partdir" -type f -exec cat {} + 2>/dev/null | sort | awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' - exit "${PIPESTATUS[2]}" + statuses=("${PIPESTATUS[@]}") + if [ "${statuses[0]}" -ne 0 ] || [ "${statuses[1]}" -ne 0 ]; then + exit 2 + fi + exit "${statuses[2]}" fi # The scan reads every session as text (-a). A reduced session is UTF-8 by @@ -260,4 +264,8 @@ awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' else print " " rest } END { exit(found ? 0 : 1) }' -exit "${PIPESTATUS[2]}" +statuses=("${PIPESTATUS[@]}") +if [ "${statuses[0]}" -ne 0 ] || [ "${statuses[1]}" -ne 0 ]; then + exit 2 +fi +exit "${statuses[2]}" diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 9672b41d4bc..22cdf3309c7 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -709,6 +709,26 @@ test_search_reports_scanner_failures_as_errors() { pass "scanner failures are errors while an ordinary no-match remains distinct" } +test_search_reports_output_pipeline_failures_as_errors() { + local home bindir tool rc=0 + home=$(setup_fixture_home) + bindir="$TMP/failing-sort-path" + rm -rf "$bindir" + mkdir -p "$bindir" + for tool in bash sh zstd grep xargs awk sed find wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + ln -sf "$(type -P false)" "$bindir/sort" + PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a --files-only sort failure must exit 2, got $rc" + rc=0 + PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a context sort failure must exit 2, got $rc" + pass "output pipeline failures are scanner errors on both search paths" +} + # The other half of the reader-status rule, and the half that is easy to lose: # once a search stops treating a closed pipe as a failure, it must still fail on # a session file it genuinely could not read. A store file that is not valid @@ -823,6 +843,7 @@ test_search_reads_plain_sessions_during_migration test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched test_search_reports_scanner_failures_as_errors +test_search_reports_output_pipeline_failures_as_errors test_search_fails_on_a_store_file_it_cannot_read test_search_without_its_tool_refuses_rather_than_finding_nothing test_refresh_names_a_readme_that_still_promises_plain_grep From e1c828f5f7187aa8b3e5dbeb2ce2355d3a967e3c Mon Sep 17 00:00:00 2001 From: coditan Date: Wed, 19 Aug 2026 23:18:43 +0000 Subject: [PATCH 18/18] no-mistakes(review): Fail closed on file gathering errors --- bin/fm-transcript-search.sh | 10 +++++++++- tests/fm-transcript-archive.test.sh | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index ddcc7501c1c..b36c4a6f907 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -145,9 +145,17 @@ for r in "${roots[@]}"; do elif [ "${f%.zst}" != "$f" ] && [ -f "${f%.zst}" ]; then printf '%s\n' "${f%.zst}" fi done >> "$filelist" + statuses=("${PIPESTATUS[@]}") + if [ "${statuses[0]}" -ne 0 ] || [ "${statuses[1]}" -ne 0 ]; then + echo "could not gather searchable sessions from $idx" >&2 + exit 2 + fi else # Both shapes are listed: a store mid-migration must not go half unsearched. - find "$r" \( -name '*.txt.zst' -o -name '*.txt' \) -type f >> "$filelist" 2>/dev/null + if ! find "$r" \( -name '*.txt.zst' -o -name '*.txt' \) -type f >> "$filelist" 2>/dev/null; then + echo "could not gather searchable sessions under $r" >&2 + exit 2 + fi fi done diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 22cdf3309c7..d7730f9903f 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -729,6 +729,23 @@ test_search_reports_output_pipeline_failures_as_errors() { pass "output pipeline failures are scanner errors on both search paths" } +test_search_reports_file_gathering_failures_as_errors() { + local home bindir tool rc=0 + home=$(setup_fixture_home) + bindir="$TMP/failing-find-path" + rm -rf "$bindir" + mkdir -p "$bindir" + for tool in bash sh zstd grep xargs awk sed sort wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + ln -sf "$(type -P false)" "$bindir/find" + PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a file gathering failure must exit 2, got $rc" + pass "file gathering failures refuse before scanning a partial file set" +} + # The other half of the reader-status rule, and the half that is easy to lose: # once a search stops treating a closed pipe as a failure, it must still fail on # a session file it genuinely could not read. A store file that is not valid @@ -844,6 +861,7 @@ test_plain_grep_does_not_read_the_sessions test_search_status_says_matched_or_not_matched test_search_reports_scanner_failures_as_errors test_search_reports_output_pipeline_failures_as_errors +test_search_reports_file_gathering_failures_as_errors test_search_fails_on_a_store_file_it_cannot_read test_search_without_its_tool_refuses_rather_than_finding_nothing test_refresh_names_a_readme_that_still_promises_plain_grep