diff --git a/README.md b/README.md index 3ea4cabee2a..41360da172a 100644 --- a/README.md +++ b/README.md @@ -241,7 +241,7 @@ Firstmate's skills live in two separate places with different audiences: - [docs/turnend-guard.md](docs/turnend-guard.md) - the primary session's structural "no turn ends blind" backstop: verified per-harness hook mechanisms, scoping, loop safety, and fail-open tradeoffs. - [docs/context-reset.md](docs/context-reset.md) - the stow-then-clear context ceiling: what the watcher measures, when it resets, asks, blocks, or reports itself unenforced, and every refusal the reset tool makes. - [docs/wake-delivery.md](docs/wake-delivery.md) - how a queued wake becomes a model turn: the external listener, why no session holds a delivery object, and the verdict that keeps a dead listener from looking like a quiet fleet. -- [docs/session-archive.md](docs/session-archive.md) - the searchable session archive: a reduced, redacted, per-vessel derivative of this machine's own transcripts, why grep is the index, why a wrong reader is made loud, and the honest bound that travels with every claim made from it. +- [docs/session-archive.md](docs/session-archive.md) - the searchable session archive: a reduced, redacted, compressed, per-vessel derivative of this machine's own transcripts, why a full content scan is the index, why a wrong reader is made loud, and the honest bound that travels with every claim made from it. - [docs/supervision-protocols/](docs/supervision-protocols/) - rendered primary-harness supervision protocols for Claude, Codex, OpenCode, Pi, Grok, and unknown harness fallback. - [docs/supervision-cost.md](docs/supervision-cost.md) - what supervision costs in freshly written tokens, measured from provider usage records with `bin/fm-supervision-cost.sh`, plus the before-and-after for three repairs and what the measurement does not cover. - [docs/scripts.md](docs/scripts.md) - the `bin/` toolbelt reference. diff --git a/bin/fm-transcript-reduce.py b/bin/fm-transcript-reduce.py index a4209e39ca9..abf5daed999 100755 --- a/bin/fm-transcript-reduce.py +++ b/bin/fm-transcript-reduce.py @@ -35,18 +35,42 @@ --source, and it refuses again afterwards if a whole run produced no entries or found no input files. An empty archive is never a silent success here. +THE STORE IS COMPRESSED, AND A FULL CONTENT SCAN IS STILL THE INDEX. Each +session is written as one zstd-compressed file, `.txt.zst`, and the +search decompresses every one of them into grep: the whole store is scanned by +content on every search and no second artefact exists that could disagree with +it. An +inverted index is still forbidden here, for the reason it always was - it can go +stale silently. +Compression cannot, because a wrong decompression is an error and not a wrong +answer. + +The compressor is the `zstd` binary (FM_ZSTD overrides it for building and +verifying). It is a hard requirement rather than a preference: a run that cannot +compress refuses instead of leaving a store that is half compressed and half +plain, which is exactly the kind of quiet disagreement this archive exists to +avoid. + +KNOWN CONDITION ON FM_ZSTD: it is what lets a store be built on a machine where +zstd is not on PATH, and it governs building and verifying only. The search +always uses the zstd on PATH, so a store written by an overridden compressor is +read back by a different one, and the two have to be able to read each other. +Ordinary zstd builds can; pointing this at something that is not zstd-compatible +leaves a store nothing else can open. + Usage: fm-transcript-reduce.py --source claude|codex --in DIR --out DIR [--patterns FILE] [--truncate 400] [--limit N] - [--fold-injected] [--quiet] + [--level 3] [--fold-injected] [--quiet] fm-transcript-reduce.py --verify-only --out DIR [--patterns FILE] Exit status: 0 built (or verified with zero residual hits) 2 verification found residual hits, or the input shape disagrees with --source 3 nothing was read: no input files, or every file yielded zero entries + 4 the compressor is missing, or it failed on a file """ -import argparse, json, os, re, sys, time +import argparse, json, os, re, shutil, subprocess, sys, time from collections import Counter # ---------------------------------------------------------------- patterns @@ -127,6 +151,100 @@ def scan(text, pats): return hits +# ---------------------------------------------------------------- storage + +# One session, one compressed file. The extension is what tells a reader how to +# open it, so `.txt.zst` is decompressed and a plain `.txt` still left in the +# store from an older build is read as it is. +SUFFIX = '.txt.zst' +PLAIN_SUFFIX = '.txt' +DEFAULT_LEVEL = 3 +ZSTD = os.environ.get('FM_ZSTD', 'zstd') + + +def compressor_path(): + """The zstd binary, or None. Named so the refusal can say what to install.""" + return shutil.which(ZSTD) + + +def require_compressor(): + if compressor_path(): + return + print('ERROR: the compressor %r is not on PATH, so this run cannot write the archive.' + % ZSTD, file=sys.stderr) + print(' Install zstd (apt install zstd), or point FM_ZSTD at the binary.', + file=sys.stderr) + print(' Refusing rather than writing a store that is half compressed and half plain.', + file=sys.stderr) + raise SystemExit(4) + + +def write_session(path, text, level): + """Write one session file, compressed, atomically. + + Through a temporary file and a rename, because the alternative - a half + written session that still has a plausible name - is unreadable material + that looks like readable material. + """ + tmp = path + '.tmp' + r = subprocess.run([ZSTD, '-q', '-f', '-%d' % level, '-o', tmp, '-'], + input=text.encode('utf-8')) + if r.returncode != 0: + try: + os.remove(tmp) + except OSError: + pass + print('ERROR: %s failed with status %d writing %s' + % (ZSTD, r.returncode, path), file=sys.stderr) + raise SystemExit(4) + os.replace(tmp, path) + + +def read_session(path): + """Read one session file back, compressed or plain.""" + if path.endswith('.zst'): + r = subprocess.run([ZSTD, '-dcq', '--', path], capture_output=True) + if r.returncode != 0: + raise OSError('%s could not decompress %s' % (ZSTD, path)) + return r.stdout.decode('utf-8', errors='replace') + with open(path, encoding='utf-8', errors='replace') as fh: + return fh.read() + + +def intact(path): + """True when zstd can decompress the whole file. A truncated store file is + not evidence of anything, so it is never the copy that survives.""" + return subprocess.run([ZSTD, '-t', '-q', '--', path], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL).returncode == 0 + + +def converge_store(outdir, level): + """Leave every session in the store as exactly one compressed file. + + Run after a build, so a store never ends up half one thing and half the + other: sessions just rewritten drop their superseded plain copy, and + sessions the raw store no longer has - which a rebuild cannot reach and must + not remove - are compressed where they lie. Returns (compressed, dropped). + """ + compressed = dropped = 0 + for root, _, names in os.walk(outdir): + for n in sorted(names): + if not n.endswith(PLAIN_SUFFIX): + continue + plain = os.path.join(root, n) + zst = plain + '.zst' + if os.path.exists(zst) and intact(zst): + os.remove(plain) + dropped += 1 + continue + with open(plain, encoding='utf-8', errors='replace') as fh: + write_session(zst, fh.read(), level) + os.remove(plain) + compressed += 1 + return compressed, dropped + + # ---------------------------------------------------------------- shape # The two record shapes share no key, which is exactly why a wrong --source is @@ -476,6 +594,9 @@ def main(): ap.add_argument('--patterns', default=DEFAULT_PATTERNS) ap.add_argument('--truncate', type=int, default=400) ap.add_argument('--limit', type=int, default=0) + ap.add_argument('--level', type=int, default=DEFAULT_LEVEL, + help='zstd compression level (default %d); measured on this ' + 'fleet in docs/session-archive.md' % DEFAULT_LEVEL) ap.add_argument('--fold-injected', action='store_true', help='codex only: fold machine-injected user messages listed in ' 'injected-prefixes.txt down to their marker and their tail') @@ -496,17 +617,31 @@ def main(): if a.verify_only: total = Counter() files = 0 + unreadable = [] for root, _, names in os.walk(a.outdir): for n in sorted(names): - if not n.endswith('.txt'): + if not (n.endswith(SUFFIX) or n.endswith(PLAIN_SUFFIX)): continue - files += 1 p = os.path.join(root, n) - with open(p, encoding='utf-8', errors='replace') as fh: - h = scan(fh.read(), pats) + # Verification reads the store the way a search reads it, through + # the decompressor. A file it cannot read is named as unread and + # never counted as a file that came back clean. + try: + body = read_session(p) + except OSError as e: + unreadable.append('%s: %s' % (p, e)) + continue + files += 1 + h = scan(body, pats) if h: print('HIT %s %s' % (p, dict(h))) total.update(h) + if unreadable: + print('ERROR: %d file(s) under %s could not be read, so they were not ' + 'verified:' % (len(unreadable), a.outdir), file=sys.stderr) + for u in unreadable[:5]: + print(' %s' % u, file=sys.stderr) + return 2 if not files: # Zero hits over zero files is not an all-clear, it is a reading the # detector could not take. Reporting it as clean is the same silent @@ -530,6 +665,8 @@ def main(): for n in names: if n.startswith('rollout-') and n.endswith('.jsonl'): files.append(os.path.join(root, n)) + require_compressor() + files.sort() if a.limit: files = files[:a.limit] @@ -544,7 +681,7 @@ def main(): os.makedirs(a.outdir, exist_ok=True) counter = Counter() - bytes_in = bytes_out = 0 + bytes_in = bytes_out = bytes_disk = 0 total_entries = 0 index = [] t0 = time.time() @@ -561,12 +698,12 @@ def main(): continue total_entries += len(entries) text = emit(meta, entries, pats, counter, a.source, rel) - outrel = rel[:-len('.jsonl')] + '.txt' + outrel = rel[:-len('.jsonl')] + SUFFIX outp = os.path.join(a.outdir, outrel) os.makedirs(os.path.dirname(outp), exist_ok=True) - with open(outp, 'w', encoding='utf-8') as fh: - fh.write(text) + write_session(outp, text, a.level) bytes_out += len(text.encode('utf-8')) + bytes_disk += os.path.getsize(outp) first_user = '' for k, _, lab, b in entries: if k == 'msg' and lab.startswith('user') and str(b).strip(): @@ -579,6 +716,11 @@ def main(): if not a.quiet and i % 100 == 0: print(' %d/%d %.0fs' % (i, len(files), time.time() - t0), file=sys.stderr) + conv, dropped = converge_store(a.outdir, a.level) + if conv or dropped: + print('converged %d retained session(s) compressed, %d superseded plain ' + 'copy(ies) dropped' % (conv, dropped)) + with open(os.path.join(a.outdir, '_index.tsv'), 'w', encoding='utf-8') as fh: fh.write('# path\tfirst\tlast\tcwd\tentries\tfirst_user_message\n') fh.write('\n'.join(sorted(index)) + '\n') @@ -590,9 +732,13 @@ def main(): print('shape undecided %d (files whose first records identify neither shape)' % undecided) print('raw bytes %d (%.1f MB)' % (bytes_in, bytes_in / 1048576)) - print('derivative %d (%.1f MB)' % (bytes_out, bytes_out / 1048576)) + print('derivative %d (%.1f MB) text as searched' % (bytes_out, bytes_out / 1048576)) + print('on disk %d (%.1f MB) zstd -%d' % (bytes_disk, bytes_disk / 1048576, a.level)) if bytes_out: print('reduction %.1f:1' % (bytes_in / bytes_out)) + if bytes_disk: + print('compression %.1f:1 (%.1f:1 against the raw store)' + % (bytes_out / bytes_disk, bytes_in / bytes_disk)) print('redactions %d' % sum(counter.values())) for k, v in counter.most_common(): print(' %-24s %d' % (k, v)) diff --git a/bin/fm-transcript-refresh.sh b/bin/fm-transcript-refresh.sh index 31cf550586e..a3d6fe1eac9 100755 --- a/bin/fm-transcript-refresh.sh +++ b/bin/fm-transcript-refresh.sh @@ -7,8 +7,15 @@ # The archive is rebuilt from the raw stores every time, then the detector is # re-run against the output and required to return zero. # +# The store is written compressed, one zstd file per session, and is searched by +# bin/fm-transcript-search.sh, which decompresses each session into grep. `zstd` is therefore a +# hard requirement of a rebuild: the reducer refuses rather than leaving a store +# that is half compressed and half plain. A rebuild also compresses whatever +# plain session files it finds already in the store, so an archive built before +# compression converges on the first refresh instead of being left behind. +# # Usage: -# fm-transcript-refresh.sh [--fold-injected] [--limit N] +# fm-transcript-refresh.sh [--fold-injected] [--limit N] [--level N] # # --fold-injected codex only: fold machine-injected user messages down to # their marker and their tail. It shrinks the Codex store by @@ -16,6 +23,8 @@ # findings with it, because less of the material was ever # looked at. Off by default for that reason. # --limit N reduce only the first N files of each store (smoke runs). +# --level N zstd compression level; the default and the measurement +# behind it are in docs/session-archive.md. # # Paths, all overridable so this runs on a vessel that is not the one it was # written on: @@ -44,17 +53,17 @@ CODEX_IN="${FM_CODEX_SESSIONS:-$HOME/.codex/sessions}" fold=() limit=() +level=() while [ $# -gt 0 ]; do case "$1" in --fold-injected) fold=(--fold-injected); shift;; --limit) limit=(--limit "${2:?--limit needs a number}"); shift 2;; - -h|--help) sed -n '2,31p' "$0" | sed 's/^# \?//'; exit 0;; + --level) level=(--level "${2:?--level needs a number}"); shift 2;; + -h|--help) sed -n '2,41p' "$0" | sed 's/^# \?//'; exit 0;; *) echo "unknown argument: $1" >&2; exit 2;; esac done -mkdir -p "$ARCHIVE" - built=0 for s in claude codex; do case "$s" in @@ -67,7 +76,8 @@ for s in claude codex; do fi echo "== reducing $s from $in_dir" python3 "$REDUCE" --source "$s" --in "$in_dir" --out "$ARCHIVE/$s-redacted" \ - --quiet "${extra[@]+"${extra[@]}"}" "${limit[@]+"${limit[@]}"}" + --quiet "${extra[@]+"${extra[@]}"}" "${limit[@]+"${limit[@]}"}" \ + "${level[@]+"${level[@]}"}" built=1 done @@ -121,7 +131,14 @@ Every claim made from this archive travels with both bounds. bin/fm-transcript-search.sh 'pattern' --since 2026-08-15 --cwd myproject bin/fm-transcript-search.sh 'pattern' --files-only -Plain `grep -r` works too - grep is the index, and there is no index to go stale. +The sessions are zstd-compressed, one file each. **Plain `grep -r` does not read +them: it finds nothing here and says so as if the archive were empty.** Use the +wrapper, or read a session the same way it does if you want the raw tools: + + zstd -dcq some-session.txt.zst | grep 'pattern' + +A full scan of the content is still what answers every query, so there is no +index here and none is to be added. ## Rebuild @@ -130,4 +147,13 @@ Plain `grep -r` works too - grep is the index, and there is no index to go stale Full detail: `docs/session-archive.md` in the firstmate repository. EOF echo "== wrote $ARCHIVE/README.md (the honest bound, on the artefact)" +elif grep -qE 'Plain .?grep -r' "$ARCHIVE/README.md" 2>/dev/null && + ! grep -q 'zstd -dcq' "$ARCHIVE/README.md" 2>/dev/null; then + # This README predates compression and still tells its reader that plain + # grep works here. It does not: it returns nothing, with no error, over a + # full archive. The file is not overwritten because someone may have written + # into it, so the correction is named instead of made. + echo "WARNING: $ARCHIVE/README.md still points readers at plain grep -r, which reads" >&2 + echo " nothing from a compressed store and reports no error while doing it." >&2 + echo " Correct that sentence: the wrapper, or zstd piped into grep, is what reads this archive." >&2 fi diff --git a/bin/fm-transcript-search.sh b/bin/fm-transcript-search.sh index f8806566dc3..b36c4a6f907 100755 --- a/bin/fm-transcript-search.sh +++ b/bin/fm-transcript-search.sh @@ -8,21 +8,42 @@ # Prints, per hit: the session's working directory and time span, the matching # line with n lines of context, and the derivative path. # -# GREP IS THE INDEX. The archive is plain UTF-8 text laid out one file per -# session, and a full-content scan of the whole store costs about a fifth of a -# second, so no inverted index exists and none should be added: a search index -# is a component that can be silently out of date, which is the exact failure -# this archive was built against. `_index.tsv` is not a search index - it -# narrows the FILE SET before grep runs, and only when --since or --cwd asks it -# to. Plain `grep -r` over the archive works identically for anyone who does not -# want this wrapper. +# A FULL CONTENT SCAN IS THE INDEX. The archive is UTF-8 text laid out one file +# per session, zstd-compressed, and a scan of the whole store costs about one and +# a half seconds for the file list, or two and a half with context. No inverted index exists and none should be added: a search +# index is a component that can be silently out of date, which is the exact +# failure this archive was built against. `_index.tsv` is not a search index - it +# narrows the FILE SET before the scan runs, and only when --since or --cwd asks +# it to. +# +# THE STORE IS COMPRESSED, SO PLAIN `grep -r` NO LONGER READS IT. It matches +# nothing here and exits as though the archive were empty, which is the one +# answer this archive must never give. The scan therefore decompresses each +# session and pipes it into grep, one file at a time, spread across the +# machine's cores. +# +# THE TOOLS THIS SEARCH REQUIRES ARE `zstd`, `grep` and `xargs`, all on PATH, +# and a missing one is reported as a missing tool rather than as a search that +# found nothing. That list is deliberately short. An earlier version scanned +# with ripgrep, whose behaviour differed from machine to machine - which flags +# its build supported, whether it could decompress at all - and twice reported +# an empty result over a full archive because of it. A search tool that answers +# differently depending on the machine it runs on is the same silent +# disagreement this archive exists to refuse, so the dependency is gone rather +# than pinned. No override names any of these tools either, for the same reason: +# one knob governing only part of the process twice passed its own prerequisite +# check before the scan read nothing. +# +# Anyone who would rather not use this wrapper reads a session the same way it +# does - `zstd -dcq .txt.zst | grep ` - and gets the same +# content scan. # # The archive is this home's private material and never travels: it resolves # under $FM_HOME/data/transcripts, so a secondmate home searches its own store # and no vessel reads another's. FM_TRANSCRIPT_ARCHIVE overrides the location. # -# Exit status is grep's: 0 when something matched, 1 when nothing did, 2 on a -# usage error or an archive that is not there. +# Exit status: 0 when something matched, 1 when nothing did, 2 on a usage error, +# a missing archive, a missing decompressing search tool, or a scanner failure. set -uo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -77,19 +98,64 @@ if [ "$present" -eq 0 ]; then exit 2 fi +# The tools are named before anything else is done, because "no matches" and +# "the tool that reads this store is not installed" are the same output to a +# caller and only one of them is true. +ZSTD=zstd +READER="$SCRIPT_DIR/fm-transcript-zcat.sh" +for tool in "$ZSTD" grep xargs; do + command -v "$tool" >/dev/null 2>&1 && continue + echo "$tool is not installed, and the session store is compressed: this search cannot run." >&2 + echo "Install zstd (apt install zstd); grep and xargs are expected on any machine that runs this." >&2 + echo "Refusing rather than reporting no matches over an archive that was never read." >&2 + exit 2 +done +[ -x "$READER" ] || { echo "missing session reader: $READER" >&2; exit 2; } + +# One scan per core. A machine that will not say how many it has gets a modest +# default rather than an unbounded fan-out. +jobs=$(nproc 2>/dev/null || echo 4) +case "$jobs" in ''|*[!0-9]*) jobs=4;; esac +[ "$jobs" -ge 1 ] || jobs=1 + # narrow the file set from the per-source index when asked -filelist="$(mktemp)"; trap 'rm -f "$filelist"' EXIT +filelist="$(mktemp)" +# Each parallel worker writes its own file rather than into a shared pipe. +# Buffered writes from concurrent workers interleave at block boundaries, not at +# line boundaries, which split lines in half: measured on this archive as three +# runs of one query returning 8121, 8105 and 8104 lines, with header lines cut +# mid-path. A search whose answer changes between runs is not a search. +partdir="$(mktemp -d)" +trap 'rm -rf "$filelist" "$partdir"' EXIT for r in "${roots[@]}"; do [ -d "$r" ] || continue idx="$r/_index.tsv" if [ -f "$idx" ] && { [ -n "$since" ] || [ -n "$cwd" ]; }; then + # An index written before the store was compressed names the plain file, so + # the compressed sibling is accepted for the same session: a stale name must + # narrow to the session it meant, not to nothing. awk -F'\t' -v r="$r" -v since="$since" -v cwd="$cwd" ' /^#/ {next} { if (since != "" && substr($2,1,10) < since) next if (cwd != "" && index($4, cwd) == 0) next - print r "/" $1 }' "$idx" >> "$filelist" + print r "/" $1 }' "$idx" | + while IFS= read -r f; do + if [ -f "$f" ]; then printf '%s\n' "$f" + elif [ -f "$f.zst" ]; then printf '%s\n' "$f.zst" + elif [ "${f%.zst}" != "$f" ] && [ -f "${f%.zst}" ]; then printf '%s\n' "${f%.zst}" + fi + done >> "$filelist" + statuses=("${PIPESTATUS[@]}") + if [ "${statuses[0]}" -ne 0 ] || [ "${statuses[1]}" -ne 0 ]; then + echo "could not gather searchable sessions from $idx" >&2 + exit 2 + fi else - find "$r" -name '*.txt' -type f >> "$filelist" 2>/dev/null + # Both shapes are listed: a store mid-migration must not go half unsearched. + if ! find "$r" \( -name '*.txt.zst' -o -name '*.txt' \) -type f >> "$filelist" 2>/dev/null; then + echo "could not gather searchable sessions under $r" >&2 + exit 2 + fi fi done @@ -97,23 +163,117 @@ n=$(wc -l < "$filelist") echo "# searching $n session files under: ${roots[*]}" >&2 [ "$n" -gt 0 ] || exit 1 +# The file set is handed over in batches, spread across cores. Two rules hold +# inside a batch, and both were learned from a defect rather than designed in. +# +# EVERY SESSION IS READ THROUGH, so a non-zero reader status is always a genuine +# failure to read the store. grep must not stop at the first match: doing so +# closes the reader's pipe early and makes its status describe the closed pipe +# rather than the file. Both search paths therefore consume the whole session +# before deciding whether it matched. +# +# Both ways of getting this wrong have been measured on this branch. Requiring +# the reader to exit 0 returned 0 matching sessions where 75 were expected, over +# a full archive. Accepting only signal 141 made a search that matched every +# session exit 2 on a machine whose zstd reports a closed pipe differently. A +# search that lies about whether it worked is the same defect either way round. +# +# A genuine no-match must not stop the scan. It is normalised to success inside +# the batch so xargs keeps going, and what the completed search actually found +# decides the exit status at the end. if [ "$files_only" = 1 ]; then - xargs -a "$filelist" -d '\n' grep -lE -- "$q" 2>/dev/null - exit $? + # shellcheck disable=SC2016 + xargs -a "$filelist" -d '\n' -P "$jobs" -n 16 bash -c ' + reader=$1; pattern=$2; partdir=$3; shift 3 + part="$partdir/part.$$" + for path do + "$reader" "$path" | grep -aE -e "$pattern" >/dev/null + statuses=("${PIPESTATUS[@]}") + case ${statuses[1]} in + 0) matched=1 ;; + 1) matched=0 ;; + *) exit 255 ;; + esac + [ "${statuses[0]}" -eq 0 ] || exit 255 + [ "$matched" -eq 0 ] || printf "%s\n" "$path" >>"$part" + done + ' _ "$READER" "$q" "$partdir" || exit 2 + find "$partdir" -type f -exec cat {} + 2>/dev/null | + sort | + awk 'NF { print; found=1 } END { exit(found ? 0 : 1) }' + statuses=("${PIPESTATUS[@]}") + if [ "${statuses[0]}" -ne 0 ] || [ "${statuses[1]}" -ne 0 ]; then + exit 2 + fi + exit "${statuses[2]}" fi -xargs -a "$filelist" -d '\n' grep -nHZE -C "$ctx" --color=never -- "$q" 2>/dev/null | -awk -F'\0' -v arch="$ARCHIVE" ' - /^--$/ { print " --"; next } - NF < 2 { next } +# The scan reads every session as text (-a). A reduced session is UTF-8 by +# construction, but a stray control byte from the material it quotes makes grep +# call the whole file binary and print no lines at all - measured here as five +# sessions that contain the term reported by --files-only and missing from the +# context output, which is a search quietly disagreeing with itself. +# +# Every emitted line carries its own session path, including the separators +# between context groups, so parallel batches cannot misattribute a line. The +# stable sort then gathers each session's lines back together - equal keys keep +# their arrival order, so a session's own lines stay in file order - and the +# reader below prints one header per session. +# shellcheck disable=SC2016 +xargs -a "$filelist" -d '\n' -P "$jobs" -n 16 bash -c ' + reader=$1; context=$2; pattern=$3; partdir=$4; shift 4 + part="$partdir/part.$$" + for path do + "$reader" "$path" | + grep -naE -C "$context" -e "$pattern" | + awk -v path="$path" '\''{ printf "%c%s%c%s\n", 28, path, 28, $0 }'\'' >>"$part" + statuses=("${PIPESTATUS[@]}") + case ${statuses[1]} in 0|1) ;; *) exit 255;; esac + # grep -C reads the session through whether or not it matched, so here the + # reader always had its chance and its status always carries information. + [ "${statuses[0]}" -eq 0 ] || exit 255 + [ "${statuses[2]}" -eq 0 ] || exit 255 + done +' _ "$READER" "$ctx" "$q" "$partdir" || exit 2 +find "$partdir" -type f -exec cat {} + 2>/dev/null | +sort -s -t "$(printf '\034')" -k2,2 | +awk -v arch="$ARCHIVE" -v zstd="$ZSTD" ' + function shell_quote(s, out, i, c) { + out="\"" + for (i=1; i<=length(s); i++) { + c=substr(s,i,1) + if (c=="\\" || c=="\"" || c=="$" || c=="`") out=out "\\" c + else out=out c + } + return out "\"" + } + substr($0, 1, 1) != sprintf("%c", 28) { next } { - path=$1; rest=$2 + record=substr($0, 2) + split_at=index(record, sprintf("%c", 28)) + if (!split_at) next + path=substr(record, 1, split_at - 1) + rest=substr(record, split_at + 1) if (path != last) { - cmd="sed -n \"1,8p\" \"" path "\" | grep -E \"^# (cwd|span)\" | sed \"s/^# *//\" | tr \"\\n\" \"|\"" + # The session header is read the same way the store is: through the + # decompressor when the file is compressed, plainly when it is not. + if (path ~ /\.zst$/) + cmd=shell_quote(zstd) " -dcq -- " shell_quote(path) " | sed -n \"1,8p\"" + else + cmd="sed -n \"1,8p\" " shell_quote(path) + cmd = cmd " | grep -E \"^# (cwd|span)\" | sed \"s/^# *//\" | tr \"\\n\" \"|\"" hdr=""; cmd | getline hdr; close(cmd) short=path; sub(arch "/", "", short) printf "\n=== %s\n %s\n", short, hdr last=path + found=1 } - print " " rest - }' + if (rest == "--") print " --" + else print " " rest + } + END { exit(found ? 0 : 1) }' +statuses=("${PIPESTATUS[@]}") +if [ "${statuses[0]}" -ne 0 ] || [ "${statuses[1]}" -ne 0 ]; then + exit 2 +fi +exit "${statuses[2]}" diff --git a/bin/fm-transcript-zcat.sh b/bin/fm-transcript-zcat.sh new file mode 100755 index 00000000000..81f126810d5 --- /dev/null +++ b/bin/fm-transcript-zcat.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# fm-transcript-zcat.sh - read one session file from the archive, whatever +# shape it is in: compressed sessions come back through zstd, and a plain one +# left from a store built before compression comes back as it is. +# +# One owner for that decision, because the scan and the session-header read +# must never disagree about how a file is read. +set -uo pipefail + +case $1 in + *.zst) exec zstd -dcq -- "$1" ;; + *) exec cat -- "$1" ;; +esac diff --git a/docs/scripts.md b/docs/scripts.md index fe34b54c125..bc5bfbf3df7 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -47,9 +47,9 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-memory-reading.sh` | Name which process is running away with this machine's memory, by size and by growth, tied to its account and to its task when known, alongside headroom and the kernel's stall reading; it sets no limit and kills nothing, and it never reports an input it could not read as a healthy zero (docs/memory-attribution.md) | | `fm-memory-alarm.sh` | Wake the fleet when this machine is running out of RAM headroom, on headroom and on growth, naming the process responsible with its account and the work it serves; it reads the attribution reading and nothing else, sets no limit and kills nothing, and reports an instrument it could not read as blindness rather than an all-clear (docs/memory-alarm.md) | | `fm-memory-ceiling-probe.sh` | Measure whether a memory ceiling on this host would manufacture the very pressure an alarm above it exists to detect, by running the same file-reading workload with and without one; it sets no lasting limit and kills nothing (docs/memory-ceiling-caveat.md) | -| `fm-transcript-reduce.py` | Build this home's reduced, redacted session derivative with one of two readers selected by `--source`, or re-verify an existing one to zero; refuses a source that disagrees with the material's shape rather than writing the empty archive a wrong reader would produce (docs/session-archive.md) | -| `fm-transcript-search.sh` | Search that derivative with context, narrowing the file set by date and working directory first; grep is the index, so there is none to go stale, and an archive that is not there reads as absent rather than as no matches (docs/session-archive.md) | -| `fm-transcript-refresh.sh` | Rebuild both stores from this vessel's own raw transcripts, verify the output to zero, and leave the honest bound on the artefact; the raw stores are read-only inputs and a store this vessel does not have is skipped, never counted as empty (docs/session-archive.md) | +| `fm-transcript-reduce.py` | Build this home's reduced, redacted session derivative with one of two readers selected by `--source`, writing one compressed file per session, or re-verify an existing one to zero through the decompressor; refuses a source that disagrees with the material's shape, or a missing compressor, rather than writing the empty or half-compressed archive either would produce (docs/session-archive.md) | +| `fm-transcript-search.sh` | Search that derivative with context, narrowing the file set by date and working directory first; a full content scan is the index, so there is none to go stale, the compressed store is decompressed into grep one session at a time across cores rather than read by plain grep, and an archive or a required tool that is not there reads as absent rather than as no matches (docs/session-archive.md) | +| `fm-transcript-refresh.sh` | Rebuild both stores from this vessel's own raw transcripts, converge any plain session files left from an older build into the compressed store, verify the output to zero, and leave the honest bound on the artefact; the raw stores are read-only inputs and a store this vessel does not have is skipped, never counted as empty (docs/session-archive.md) | | `fm-herdr-lab.sh` | Provision and guardedly operate an isolated, never-default Herdr lab session | | `fm-install-herdr.sh` | Install CI's exact-version Herdr pin with official asset URL, SHA-256, and protocol checks | | `fm-install-treehouse.sh`| Install CI's exact-version Treehouse pin for real-Herdr E2E that needs spawn worktrees | diff --git a/docs/session-archive.md b/docs/session-archive.md index 8c4c415169d..c6ff5fbf89a 100644 --- a/docs/session-archive.md +++ b/docs/session-archive.md @@ -11,13 +11,16 @@ The archive it builds is not shared and never becomes shared: it is a private de Two stores, one per source, under `$FM_HOME/data/transcripts/`: ``` -claude-redacted/ one .txt per session, mirroring ~/.claude/projects +claude-redacted/ one .txt.zst per session, mirroring ~/.claude/projects _index.tsv path, time span, working directory, entry count, first user message -codex-redacted/ one .txt per session, mirroring ~/.codex/sessions +codex-redacted/ one .txt.zst per session, mirroring ~/.codex/sessions _index.tsv path, time span, working directory, entry count, first user message ``` -Per session, one plain-text file holding a header, then: +Each session file is one zstd-compressed plain-text document. +`_index.tsv` is the one uncompressed file in a store, because it is read by an `awk` narrowing pass rather than by the content scan. + +Per session, one plain-text document - compressed on disk, and read back in full by every search - holding a header, then: - every user and assistant message, verbatim - every command issued, verbatim @@ -35,11 +38,38 @@ bin/fm-transcript-search.sh 'pattern' --since 2026-08-15 --cwd myproject bin/fm-transcript-search.sh 'pattern' --files-only ``` -**Grep is the index.** -The archive is UTF-8 text laid out one file per session, and a full-content scan of the whole store was measured at 0.22 s over 258 MB on 2026-08-18, so no inverted index exists and none should be added. +**A full content scan is the index.** +The archive is UTF-8 text laid out one file per session, and a scan of the whole store was measured at 1.50 to 1.54 s for the file list, and 2.31 to 2.64 s with context, over the compressed 48.9 MB on 2026-08-19, so no inverted index exists and none should be added. An index is a component that can be silently out of date, which is exactly the failure this archive was built against. -`_index.tsv` is not a search index: it narrows the file set before grep runs, and only when `--since` or `--cwd` asks it to. -Plain `grep -r` over the archive works identically for anyone who does not want the wrapper. +Compression does not reintroduce that failure and is why it was worth doing: the compressed file is the content, not a summary of it, and a decompression that goes wrong is an error rather than a wrong answer. +`_index.tsv` is not a search index: it narrows the file set before the scan runs, and only when `--since` or `--cwd` asks it to. + +**Plain `grep -r` no longer reads this archive.** +It matches nothing in a compressed session and exits reporting no matches, over an archive that is full - the same silent emptiness the whole design refuses, arriving through the documentation instead of through the reader. +It is worse than wholly blind: `_index.tsv` is the one plain file left, so a phrase that happens to sit in its first-user-message column still matches, and the blindness looks selective rather than total. +Anyone who wants the raw tools rather than the wrapper reads a session exactly as the wrapper does: + +```sh +zstd -dcq some-session.txt.zst | grep 'pattern' +``` + +### The tools a search requires + +`zstd`, `grep` and `xargs`, on `PATH`. +That is the whole list, and the wrapper reports a missing one as a missing tool rather than as a search that found nothing. +No override names any of them, because a knob governing only part of the process twice passed its own prerequisite check before the scan read nothing. + +The list is short deliberately. +An earlier version scanned with ripgrep and `-z`; it was withdrawn on 2026-08-19 on measurement rather than taste. +Ripgrep behaved differently on this seat and on the CI runner - which flags the build accepted, whether it could decompress at all - and twice reported an empty result over a full archive. +A search tool that answers differently depending on the machine it runs on is the same silent disagreement this archive exists to refuse, so the dependency was removed rather than pinned to a version. +`zstd` is already required to build the store, and `grep` and `xargs` are on any machine that can run this repository's tests at all. + +**Both search paths read every session through to the end.** +The decompressor's exit status is therefore always meaningful, and any non-zero status is a genuine failure to read the store. +Requiring the reader to exit 0 while the scan stopped at the first match returned 0 matching sessions where 75 were expected over a full archive, while accepting only signal 141 made a search that matched every session exit 2 on a machine whose zstd reports a closed pipe differently. +Reading every session through is what makes a corrupt store file detectable on both paths, including one whose valid prefix contains the match. +`tests/fm-transcript-archive.test.sh` proves that invariant with a compressed session whose matching valid prefix is followed by a truncated frame. ## Rebuild @@ -47,8 +77,8 @@ Plain `grep -r` over the archive works identically for anyone who does not want bin/fm-transcript-refresh.sh ``` -It rebuilds each raw transcript store present on the vessel and then re-runs the detector against each resulting derivative, requiring zero hits. -There is no incremental path and no build state, because a full rebuild of a two-thousand-session store costs about 95 s and a build state is one more thing that can be quietly wrong. +It rebuilds each raw transcript store present on the vessel and then re-runs the detector against each resulting derivative - reading it back through the decompressor, as a search does - and requires zero hits. +There is no incremental path and no build state, because a full rebuild of a two-thousand-session store costs about two minutes and a build state is one more thing that can be quietly wrong. The archive retains sessions the raw store no longer has: a rebuild rewrites every session it can still read and removes nothing, so a session deleted, rotated away, or renamed under `~/.claude` or `~/.codex` keeps its reduced copy in the archive. This is intended rather than a defect, because the archive exists precisely so that what was said survives the clearing of the session that said it, and outliving the raw store is the point. The archive therefore does not mirror a deletion, so removing material from the archive is a deliberate separate act and never a side effect of a rebuild. @@ -56,6 +86,18 @@ The archive therefore does not mirror a deletion, so removing material from the The raw stores are read-only inputs; nothing under `~/.claude` or `~/.codex` is written, moved, or removed. A raw store that does not exist on this vessel is reported and skipped, never treated as a store that happened to be empty. +## Compression, and why it does not cost the property above + +Each session is written as one zstd file at level 3, which took this seat's store from 271.2 MB to 52.1 MB on 2026-08-18. +`zstd` is a hard requirement of a rebuild rather than a preference: a run that cannot compress refuses before writing anything, because a store that is half compressed and half plain is a store whose answers depend on which half a question lands in. +`FM_ZSTD` may name the compressor used specifically for building and verifying the store; it does not apply to searching, and it exists because it is the only way to build a store on a machine where `zstd` is not on `PATH`. +That split is a known condition rather than an oversight: a store written by an overridden compressor is read back by the `zstd` the search finds on `PATH`, so the two have to be able to read each other, which ordinary zstd builds do and something that is not zstd-compatible does not. +A rebuild also compresses whatever plain session files it finds already in the store and drops the plain copies it has just superseded, so an archive built before compression converges on the first refresh instead of being stranded, and a retained session the rebuild cannot reach is compressed where it lies rather than left behind. +The level is a knob, `--level`, and the measurement behind the default is below. + +The store stays honest under compression for one reason: the compressed file is the content rather than a summary of it, so nothing exists that could disagree with it, and a decompression that goes wrong is an error rather than a wrong answer. +A verification that cannot read a store file says so and fails, instead of counting a file nobody read as a file that came back clean. + ## The honest bound, which travels with every claim made from this archive > The derivative is verified: the same detector re-run against the output returns zero hits. @@ -104,17 +146,35 @@ Two consequences bind anyone who touches this tool: Recorded with their date because they are readings, not properties. ``` - raw derivative ratio redactions -Claude 1001.5 MB 115.6 MB 8.7:1 73 -Codex 413.5 MB 142.2 MB 2.9:1 37 -combined 1415.0 MB 257.8 MB 5.5:1 110 -sessions 2236 -full rebuild + verification ~95 s -full-content search over the store 0.22 s -verification residual hits 0 + raw derivative on disk ratio redactions +Claude 1007.6 MB 116.6 MB 27.2 MB 8.6:1 73 +Codex 418.7 MB 143.6 MB 21.7 MB 2.9:1 37 +combined 1426.3 MB 260.2 MB 48.9 MB 5.5:1 110 +sessions 2273 +full rebuild + verification ~129 s +full-content search, file list / with context 1.50 - 1.54 s / 2.31 - 2.64 s +verification residual hits 0 ``` -The ratio is material-dependent and is not a property of the tool: the same design measured 56:1 on a seat whose sessions averaged 17 MB, against 0.63 MB here. +`derivative` is the text a search reads; `on disk` is what the store costs after compression. +The whole archive directory, including the two indexes, went from 271.2 MB to 52.1 MB in the migration of 2026-08-18, and no session was lost to it: of the 2236 sessions present beforehand, 2144 came back byte-identical and 92 came back longer because the session had continued since the previous build. + +### Choosing the compression level + +Measured the same day, on the same seat's real store rather than on a sample, each a full rebuild with verification: + +``` + rebuild + verify store on disk against plain +plain (before) 89.8 s 273.6 MB 1.0x +zstd -3 119.4 s 52.1 MB 5.25x +zstd -9 159.6 s 48.7 MB 5.62x +``` + +Level 3 is the default because level 9 spends 40 more seconds of every rebuild to save 3.4 MB, which is 1.2 percent of what was already saved. +The compression ratio is material-dependent: these are readings from one seat's transcripts, not properties of zstd. +Search cost is not part of this tradeoff - zstd decompresses at roughly the same speed whatever level wrote the file - so the level buys rebuild time against disk and nothing else. + +The reduction ratio is material-dependent and is not a property of the tool: the same design measured 56:1 on a seat whose sessions averaged 17 MB, against 0.63 MB here. `--fold-injected` folds Codex's machine-injected user messages down to their marker and their tail, taking the Codex store from 142.2 MB to 44.1 MB. It is off by default, and the reason is the finding rather than the size: folding dropped 24 of the 37 redaction findings, including 16 of the 22 private-key blocks, because those sat inside the folded region. The folded archive is not cleaner - less of it was ever looked at. diff --git a/tests/fm-transcript-archive.test.sh b/tests/fm-transcript-archive.test.sh index 2968a903c35..d7730f9903f 100755 --- a/tests/fm-transcript-archive.test.sh +++ b/tests/fm-transcript-archive.test.sh @@ -14,7 +14,12 @@ # 2. The pattern file must not match itself. A detector whose patterns appear # in plain text writes those patterns into the next session transcript, and # every later scan then inflates on its own tooling. -# 3. THIS FILE must not match the pattern file either. Property 2 closes the +# 3. A DOCUMENTED SEARCH MUST NOT SILENTLY RETURN NOTHING. The store is +# compressed, and plain grep reads a compressed file as no matches with no +# error - the same silent emptiness as 1, arriving through the +# documentation instead of through the reader. So the search path is +# exercised over a compressed store here. +# 4. THIS FILE must not match the pattern file either. Property 2 closes the # pattern list; it does not close the fixtures. The first build of this tool # left five real redactions in its own session from fixtures typed in plain # form. Every credential-shaped fixture below is therefore assembled at run @@ -30,12 +35,27 @@ set -u REDUCE="$ROOT/bin/fm-transcript-reduce.py" SEARCH="$ROOT/bin/fm-transcript-search.sh" REFRESH="$ROOT/bin/fm-transcript-refresh.sh" +ZCAT="$ROOT/bin/fm-transcript-zcat.sh" PATTERNS="$ROOT/bin/fm-transcript-patterns/patterns.txt" INJECTED="$ROOT/bin/fm-transcript-patterns/injected-prefixes.txt" TMP="" fm_test_tmproot TMP fm-transcript +# --- reading the store ------------------------------------------------------ + +# Every session in the store is one zstd file, so a test reads it the way a +# search does - through the decompressor - and never by opening it as text. +read_session() { + zstd -dcq -- "$1" +} + +# Session files in a store, both shapes, so a test that expects only one shape +# has to say so. +store_files() { + find "$1" \( -name '*.txt.zst' -o -name '*.txt' \) -type f | sort +} + # --- fixture builders ------------------------------------------------------- # A minimal Claude-shaped session: both conversation sides, one command, one @@ -101,7 +121,7 @@ PY test_tool_is_tracked_and_runnable() { local f - for f in "$REDUCE" "$SEARCH" "$REFRESH"; do + for f in "$REDUCE" "$SEARCH" "$REFRESH" "$ZCAT"; do assert_present "$f" "missing tool file: $f" [ -x "$f" ] || fail "$f must be executable" done @@ -163,7 +183,8 @@ test_claude_reader_reads_claude_material() { write_claude_session "$TMP/claude-in/proj" out=$(python3 "$REDUCE" --source claude --in "$TMP/claude-in" --out "$TMP/claude-out" 2>&1) || rc=$? [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail "claude reader failed on claude material"; } - body=$(cat "$TMP/claude-out/proj/sess.txt") + assert_present "$TMP/claude-out/proj/sess.txt.zst" 'the session must be written compressed' + body=$(read_session "$TMP/claude-out/proj/sess.txt.zst") assert_contains "$body" 'find the tugboat host' 'the user side must survive verbatim' assert_contains "$body" 'looking now' 'the assistant side must survive verbatim' assert_contains "$body" '$ ls -la /srv' 'the command must survive verbatim' @@ -177,7 +198,7 @@ test_codex_reader_reads_codex_material() { write_codex_session "$TMP/codex-in/2026/08/18" out=$(python3 "$REDUCE" --source codex --in "$TMP/codex-in" --out "$TMP/codex-out" 2>&1) || rc=$? [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail "codex reader failed on codex material"; } - body=$(cat "$TMP/codex-out/2026/08/18/rollout-2026-08-18T11-00-00-abc.txt") + body=$(read_session "$TMP/codex-out/2026/08/18/rollout-2026-08-18T11-00-00-abc.txt.zst") assert_contains "$body" 'where did the harbour report go' 'the user side must survive verbatim' assert_contains "$body" '$ grep -r harbour' 'the command must survive verbatim' assert_contains "$body" '# cwd /home/x/bravo' 'session_meta must supply the header' @@ -242,7 +263,7 @@ test_redactor_masks_a_synthetic_credential() { [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'the reducer failed on the credential fixture'; } assert_contains "$out" 'github-token' 'the run summary must count the prefix-anchored class' assert_contains "$out" 'assigned-secret' 'the run summary must count the context-anchored class' - body=$(cat "$TMP/creds-out/proj/creds.txt") + body=$(read_session "$TMP/creds-out/proj/creds.txt.zst") assert_contains "$body" 'REDACTED github-token' 'a removed value must leave a naming marker' case "$body" in *"$tok"*) fail 'the prefix-anchored value survived into the derivative' ;; @@ -261,10 +282,11 @@ test_verification_is_zero_on_clean_and_loud_on_dirty() { # A verification that can only ever say zero proves nothing, so plant one. mkdir -p "$TMP/dirty" tok=$(fixture_prefixed_token) - printf 'a line carrying %s in it\n' "$tok" >"$TMP/dirty/planted.txt" + printf 'a line carrying %s in it\n' "$tok" | + zstd -q -o "$TMP/dirty/planted.txt.zst" rc=0 out=$(python3 "$REDUCE" --verify-only --out "$TMP/dirty" 2>&1) || rc=$? - [ "$rc" -ne 0 ] || fail 'verification must fail on a derivative that still carries a value' + [ "$rc" -ne 0 ] || fail 'verification must fail on a compressed derivative that still carries a value' assert_contains "$out" 'HIT' 'a residual hit must be reported with its file' case "$out" in *"$tok"*) fail 'verification must report classes and counts, never the matched value' ;; @@ -287,6 +309,96 @@ test_verification_over_nothing_is_not_an_all_clear() { pass "verifying a directory with no derivative in it refuses instead of reporting zero" } +# --- the store is compressed, wholly ---------------------------------------- + +test_the_store_is_written_compressed_and_leaves_nothing_plain() { + local out rc=0 plain + write_claude_session "$TMP/zst-in/proj" + out=$(python3 "$REDUCE" --source claude --in "$TMP/zst-in" --out "$TMP/zst-out" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'the reducer failed writing a compressed store'; } + assert_present "$TMP/zst-out/proj/sess.txt.zst" 'the session must land as one compressed file' + assert_absent "$TMP/zst-out/proj/sess.txt" 'no plain copy of a written session may be left behind' + plain=$(find "$TMP/zst-out" -name '*.txt' -type f | wc -l) + [ "$plain" -eq 0 ] || fail "the store must hold no plain session files, found $plain" + assert_contains "$out" 'on disk' 'the summary must state what the store actually costs on disk' + assert_grep 'sess.txt.zst' "$TMP/zst-out/_index.tsv" \ + 'the index must name the file that exists, not the one that used to' + pass "a build writes one compressed file per session and leaves no plain copy" +} + +# The archive keeps sessions the raw store no longer has, and a rebuild cannot +# reach them. Left alone they would stay plain forever - half a compressed +# archive and half not - so a rebuild compresses them where they lie, without +# removing the one thing they carry, which is their content. +test_a_rebuild_compresses_a_retained_session_it_cannot_rebuild() { + local out rc=0 body + write_claude_session "$TMP/keep-in/proj" + python3 "$REDUCE" --source claude --in "$TMP/keep-in" --out "$TMP/keep-out" >/dev/null 2>&1 \ + || fail 'the first build failed' + mkdir -p "$TMP/keep-out/gone" + printf '# session gone/old.jsonl\n# cwd /home/x/gone\n\nan older answer worth keeping\n' \ + >"$TMP/keep-out/gone/old.txt" + out=$(python3 "$REDUCE" --source claude --in "$TMP/keep-in" --out "$TMP/keep-out" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'the rebuild failed'; } + assert_absent "$TMP/keep-out/gone/old.txt" 'the retained plain session must not be left uncompressed' + assert_present "$TMP/keep-out/gone/old.txt.zst" 'the retained session must survive as a compressed file' + body=$(read_session "$TMP/keep-out/gone/old.txt.zst") + assert_contains "$body" 'an older answer worth keeping' \ + 'compressing a retained session must not cost a word of it' + assert_contains "$out" 'converged' 'the run must say what it converged rather than doing it silently' + pass "a rebuild converges a retained plain session into the compressed store, content intact" +} + +test_verification_reads_the_compressed_store_it_verifies() { + local out rc=0 + out=$(python3 "$REDUCE" --verify-only --out "$TMP/zst-out" 2>&1) || rc=$? + [ "$rc" -eq 0 ] || { printf '%s\n' "$out" >&2; fail 'verification failed over a compressed store'; } + assert_contains "$out" '1 files scanned' \ + 'verification must count the compressed session it read, not skip it as an unknown file' + assert_contains "$out" '0 residual hits' 'verification must state the residual count' + pass "verification opens the compressed store and counts what it scanned" +} + +# A store file the decompressor cannot read is not a clean file; it is a file +# nobody read. Counting it as verified is the silent all-clear this tool exists +# to refuse. +test_verification_refuses_a_store_file_it_cannot_read() { + local out rc=0 + mkdir -p "$TMP/corrupt" + printf 'not a compressed file at all\n' >"$TMP/corrupt/broken.txt.zst" + out=$(python3 "$REDUCE" --verify-only --out "$TMP/corrupt" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail 'a store file that cannot be decompressed must not verify clean' + assert_contains "$out" 'could not be read' 'the refusal must say the file was never verified' + pass "an unreadable store file is reported as unread rather than counted as clean" +} + +test_a_build_without_a_compressor_refuses_before_writing() { + local out rc=0 + write_claude_session "$TMP/nozstd-in/proj" + out=$(FM_ZSTD="$TMP/no-such-zstd" python3 "$REDUCE" --source claude \ + --in "$TMP/nozstd-in" --out "$TMP/nozstd-out" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail 'a build with no compressor must not exit 0' + assert_contains "$out" 'not on PATH' 'the refusal must name the missing tool' + assert_contains "$out" 'half compressed and half plain' 'the refusal must say what it is avoiding' + assert_absent "$TMP/nozstd-out" 'nothing may be written when the compressor is missing' + pass "a missing compressor refuses the build instead of writing a store nobody can search" +} + +test_refresh_without_a_compressor_leaves_no_archive() { + local home out rc=0 + home="$TMP/refresh-nozstd-home" + write_claude_session "$TMP/refresh-nozstd-src/claude/project" + out=$(FM_HOME="$home" FM_CLAUDE_SESSIONS="$TMP/refresh-nozstd-src/claude" \ + FM_CODEX_SESSIONS="$TMP/refresh-nozstd-src/codex" FM_ZSTD="$TMP/no-such-zstd" \ + "$REFRESH" 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail 'refresh with no compressor must not exit 0' + assert_contains "$out" 'half compressed and half plain' \ + 'refresh must preserve the reducer refusal message' + assert_absent "$home/data/transcripts" \ + 'refresh must not create an archive before the compressor prerequisite passes' + pass "refresh with no compressor refuses before creating the archive" +} + # --- search ------------------------------------------------------------------ # Build a fixture home so search is exercised the way a vessel that is not this @@ -344,6 +456,23 @@ test_search_resolves_the_archive_from_fm_home() { pass "search resolves its archive from FM_HOME and reaches both stores" } +test_search_formatter_is_portable_to_mawk() { + local home out shim + command -v mawk >/dev/null 2>&1 || { + pass "mawk is unavailable, so the portable formatter check is not applicable" + return + } + home=$(setup_fixture_home) + shim="$TMP/mawk-path" + mkdir -p "$shim" + ln -s "$(command -v mawk)" "$shim/awk" + out=$(PATH="$shim:$PATH" FM_HOME="$home" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search lost a real hit when its formatter ran under mawk' + assert_contains "$out" 'tugboat host' \ + 'the portable formatter must preserve the matching line under mawk' + pass "search formatting preserves real hits under mawk as well as gawk" +} + test_search_narrows_the_file_set_by_index() { local home out home=$(setup_fixture_home) @@ -368,6 +497,335 @@ test_search_reports_a_missing_archive_rather_than_no_matches() { pass "an archive that is not there reads as absent, never as no matches" } +test_search_reads_the_compressed_store() { + local home out plain + home=$(setup_fixture_home) + plain=$(find "$home/data/transcripts" -name '*.txt' -type f | wc -l) + [ "$plain" -eq 0 ] || fail "the fixture archive must be wholly compressed, found $plain plain files" + out=$(FM_HOME="$home" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search found nothing in a compressed archive it should have matched' + assert_contains "$out" 'tugboat host' 'the matching line must come back out of the compressed file' + assert_contains "$out" '.txt.zst' 'the hit must name the compressed session file it came from' + assert_contains "$out" 'cwd /home/x/alpha' \ + 'the session header must be read through the decompressor too, not skipped' + pass "search reads the compressed store and still reports the session header with the hit" +} + +# The case that broke twice in CI was a machine whose ripgrep did not behave +# like this one's. The search no longer uses ripgrep at all, and that is proven +# by running it on a PATH where ripgrep genuinely cannot be resolved - not by +# stubbing a flag, which is what let the defect through the previous two rounds. +# The machine this was written on reports a closed pipe as signal 141. The CI +# runner's zstd reports it as an ordinary error instead, and a search that +# matched EVERY session therefore exited 2 there while passing here. So the rule +# is tested against a reader that behaves like the runner's rather than like +# this seat's: the search script itself runs, with only its sibling reader +# swapped for one that swallows the signal and reports a plain failure. +test_a_reader_that_hides_the_signal_still_reports_a_match() { + local store bindir i rc=0 out count expected=6 + store="$TMP/odd-reader/claude-redacted" + rm -rf "$TMP/odd-reader" + mkdir -p "$store" + for i in $(seq 1 "$expected"); do + { + printf '# session odd%s.jsonl\n# cwd /home/x/odd%s\n' "$i" "$i" + printf '# span 2026-08-19 10:00:00 .. 2026-08-19 10:05:00\n\n' + printf ' the sought phrase sits near the top of this session\n' + yes ' a long tail that keeps the decompressor writing well past the match' 2>/dev/null | + head -20000 + } | zstd -q -3 -o "$store/odd$i.txt.zst" + done + + bindir="$TMP/odd-reader-bin" + rm -rf "$bindir" + mkdir -p "$bindir" + cp "$SEARCH" "$bindir/fm-transcript-search.sh" + cat >"$bindir/fm-transcript-zcat.sh" <<'EOF' +#!/usr/bin/env bash +# Models a decompressor that reports a closed pipe as an ordinary failure +# instead of dying on the signal, which is what the CI runner's zstd does. +zstd -dcq -- "$1" 2>/dev/null || exit 1 +EOF + chmod +x "$bindir/fm-transcript-search.sh" "$bindir/fm-transcript-zcat.sh" + + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/odd-reader" "$bindir/fm-transcript-search.sh" \ + 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] \ + || fail "a search matching every session must exit 0 even when the reader hides the signal, got $rc" + count=$(printf '%s\n' "$out" | grep -c 'odd[0-9]*\.txt\.zst') + [ "$count" -eq "$expected" ] || fail "reported $count of $expected sessions" + pass "a reader that reports the closed pipe as a plain failure does not turn a full match into an error" +} + +# A reduced session is UTF-8 by construction, but it quotes material that is +# not always clean, and one stray control byte makes grep call the whole file +# binary and print no lines for it. Measured on the live archive: five sessions +# that --files-only reported as containing the term were missing from the +# context output of the same query. A search that answers one way per session +# and another way per line is disagreeing with itself. +test_a_session_with_a_control_byte_is_still_reported() { + local store out rc=0 + store="$TMP/binary-byte/claude-redacted" + rm -rf "$TMP/binary-byte" + mkdir -p "$store" + { + printf '# session odd.jsonl\n# cwd /home/x/odd\n' + printf '# span 2026-08-19 10:00:00 .. 2026-08-19 10:05:00\n\n' + printf ' a quoted blob follows: \001\002\003 and then text\n' + printf ' the sought phrase is on its own line here\n' + } | zstd -q -3 -o "$store/odd.txt.zst" + + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/binary-byte" "$SEARCH" 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "--files-only must find a session carrying a control byte, got exit $rc" + assert_contains "$out" 'odd.txt.zst' '--files-only must name the session' + rc=0 + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/binary-byte" "$SEARCH" 'sought phrase' -C 1 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "the context path must find it too, got exit $rc" + assert_contains "$out" 'sought phrase' \ + 'the matching line must be printed, not swallowed as an unprintable file' + pass "a session carrying a control byte is reported by both search paths, not silently skipped" +} + +# THE CASE EVERY OTHER FIXTURE MISSED. A scan that stops reading as soon as it +# can answer kills the decompressor feeding it with SIGPIPE. Treating that +# ordinary event as a scanner failure returned 0 matching sessions where 75 were +# expected, over a full archive, and no test saw it: a one-session fixture with +# one match near the end fits in the pipe buffer, so nothing ever short-circuits. +# This fixture puts the match at the TOP of each session and a long tail behind +# it, which is what forces the reader to stop early while the decompressor is +# still writing. If the defect returns, this is what fails. +test_a_scan_that_stops_early_still_reports_every_session() { + local store i out rc=0 count expected=12 + store="$TMP/early-stop/claude-redacted" + rm -rf "$TMP/early-stop" + mkdir -p "$store" + for i in $(seq 1 "$expected"); do + { + printf '# session big%s.jsonl\n# cwd /home/x/p%s\n' "$i" "$i" + printf '# span 2026-08-18 10:00:00 .. 2026-08-18 10:05:00\n\n' + printf ' the sought phrase sits near the top of this session\n' + yes ' a long tail that keeps the decompressor writing well past the match' | + head -20000 + } | zstd -q -3 -o "$store/big$i.txt.zst" + done + + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/early-stop" "$SEARCH" 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "a search matching every session must exit 0, got $rc" + count=$(printf '%s\n' "$out" | grep -c 'big[0-9]*\.txt\.zst') + [ "$count" -eq "$expected" ] \ + || fail "--files-only reported $count of $expected sessions; a scan that stops early lost the rest" + + rc=0 + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/early-stop" "$SEARCH" 'sought phrase' -C 1 2>/dev/null) || rc=$? + [ "$rc" -eq 0 ] || fail "the context path must exit 0 when every session matched, got $rc" + count=$(printf '%s\n' "$out" | grep -c '^=== ') + [ "$count" -eq "$expected" ] \ + || fail "the context path reported $count of $expected sessions" + pass "every session is reported when the scan stops reading each one early" +} + +test_search_works_on_a_machine_without_ripgrep() { + local home out bindir tool + home=$(setup_fixture_home) + bindir="$TMP/no-rg-path" + rm -rf "$bindir" + mkdir -p "$bindir" + # A PATH holding exactly what the search is allowed to need, and nothing else. + # type -P resolves the executable itself: an interactive shell may carry a + # function or alias by the same name, and linking that name to itself makes a + # fixture that proves nothing. + for tool in bash sh zstd grep xargs awk sed sort find wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + PATH="$bindir" type -P rg >/dev/null 2>&1 \ + && fail 'the no-ripgrep fixture PATH still resolves ripgrep, so it proves nothing' + out=$(PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' 2>/dev/null) \ + || fail 'search failed on a machine with no ripgrep on PATH' + assert_contains "$out" 'tugboat host' \ + 'the hit must come back from a machine that has no ripgrep at all' + pass "search runs and finds its hit on a PATH where ripgrep cannot be resolved" +} + +test_search_reads_plain_sessions_during_migration() { + local home out plain + home=$(setup_fixture_home) + plain="$home/data/transcripts/claude-redacted/migration.txt" + printf '# cwd /home/x/migration\n# span 2026-08-18T10:00:00Z\nplain migration session\n' >"$plain" + out=$(FM_HOME="$home" "$SEARCH" 'plain migration session' 2>/dev/null) \ + || fail 'search did not read a plain session during migration' + assert_contains "$out" 'plain migration session' \ + 'a plain migration-era session must remain searchable' + pass "search reads plain sessions that remain during migration" +} + +# The reason the documentation had to change, stated as a test rather than as a +# claim: over this store the old documented command answers "nothing here" and +# exits as though that were true. Worse than nothing, it half answers - the +# index is the one plain file left, so a phrase that happens to sit in its +# first-user-message column still matches and the emptiness looks selective +# rather than total. +test_plain_grep_does_not_read_the_sessions() { + local home rc=0 out + home=$(setup_fixture_home) + out=$(grep -r 'looking now' "$home/data/transcripts" 2>/dev/null) || rc=$? + [ "$rc" -eq 1 ] || fail "plain grep was expected to report no matches over a compressed store, got exit $rc" + [ -z "$out" ] || fail 'plain grep unexpectedly read a compressed session' + out=$(FM_HOME="$home" "$SEARCH" 'looking now' 2>/dev/null) \ + || fail 'the wrapper must find what plain grep could not' + assert_contains "$out" 'looking now' 'the same phrase must come back through the wrapper' + pass "plain grep reports no matches on material the wrapper finds, which is why it is no longer documented" +} + +# A caller that scripts this tool reads its exit status, and the file set is +# handed to the scanner in batches: a batch with no hit must not make a search +# that matched look like a search that failed. +test_search_status_says_matched_or_not_matched() { + local home rc=0 + home=$(setup_fixture_home) + FM_HOME="$home" "$SEARCH" 'tugboat host' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 0 ] || fail "a search that matched must exit 0, got $rc" + rc=0 + FM_HOME="$home" "$SEARCH" 'tugboat host' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 0 ] || fail "a --files-only search that matched must exit 0, got $rc" + rc=0 + FM_HOME="$home" "$SEARCH" 'nothing here says this at all' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 1 ] || fail "a search that matched nothing must exit 1, got $rc" + pass "the exit status states whether the search matched, not how a batch of it ended" +} + +test_search_reports_scanner_failures_as_errors() { + local home rc=0 + home=$(setup_fixture_home) + FM_HOME="$home" "$SEARCH" '[' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a malformed regex must be reported as a scanner error, got exit $rc" + rc=0 + FM_HOME="$home" "$SEARCH" '[' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a malformed --files-only regex must be reported as a scanner error, got exit $rc" + rc=0 + FM_HOME="$home" "$SEARCH" 'nothing here says this at all' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 1 ] || fail "an ordinary no-match must remain exit 1, got $rc" + pass "scanner failures are errors while an ordinary no-match remains distinct" +} + +test_search_reports_output_pipeline_failures_as_errors() { + local home bindir tool rc=0 + home=$(setup_fixture_home) + bindir="$TMP/failing-sort-path" + rm -rf "$bindir" + mkdir -p "$bindir" + for tool in bash sh zstd grep xargs awk sed find wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + ln -sf "$(type -P false)" "$bindir/sort" + PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a --files-only sort failure must exit 2, got $rc" + rc=0 + PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a context sort failure must exit 2, got $rc" + pass "output pipeline failures are scanner errors on both search paths" +} + +test_search_reports_file_gathering_failures_as_errors() { + local home bindir tool rc=0 + home=$(setup_fixture_home) + bindir="$TMP/failing-find-path" + rm -rf "$bindir" + mkdir -p "$bindir" + for tool in bash sh zstd grep xargs awk sed sort wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + ln -sf "$(type -P false)" "$bindir/find" + PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' --files-only >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] || fail "a file gathering failure must exit 2, got $rc" + pass "file gathering failures refuse before scanning a partial file set" +} + +# The other half of the reader-status rule, and the half that is easy to lose: +# once a search stops treating a closed pipe as a failure, it must still fail on +# a session file it genuinely could not read. A store file that is not valid +# compressed data is exactly that, and it must never pass as a session that +# simply held no match. +test_search_fails_on_a_store_file_it_cannot_read() { + local store rc=0 out size + store="$TMP/corrupt-store/claude-redacted" + rm -rf "$TMP/corrupt-store" + mkdir -p "$store" + printf 'this is not compressed data at all\n' >"$store/broken.txt.zst" + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'anything at all' --files-only 2>&1 >/dev/null) || rc=$? + [ "$rc" -eq 2 ] \ + || fail "a store file that cannot be read must be a scanner error, got exit $rc" + rc=0 + FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'anything at all' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] \ + || fail "the context path must also fail on an unreadable store file, got exit $rc" + + rm -f "$store/broken.txt.zst" + { + printf 'sought phrase near the top\n' + seq 1 10000 + } | zstd -q -3 -o "$store/truncated.txt.zst" + size=$(wc -c <"$store/truncated.txt.zst") + truncate -s "$((size - 1))" "$store/truncated.txt.zst" + rc=0 + out=$(FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'sought phrase' --files-only 2>/dev/null) || rc=$? + [ "$rc" -eq 2 ] \ + || fail "a truncated matching session must be a scanner error in --files-only, got exit $rc" + [ -z "$out" ] \ + || fail "a truncated matching session must not be reported as a valid match" + rc=0 + FM_TRANSCRIPT_ARCHIVE="$TMP/corrupt-store" "$SEARCH" 'sought phrase' >/dev/null 2>&1 || rc=$? + [ "$rc" -eq 2 ] \ + || fail "the context path must reject a truncated matching session, got exit $rc" + pass "unreadable sessions fail whether or not their readable prefix matches" +} + +test_search_without_its_tool_refuses_rather_than_finding_nothing() { + local home out rc=0 bindir tool + home=$(setup_fixture_home) + # A PATH with everything except the decompressor: the store cannot be read, + # and the caller must be told that rather than told there were no matches. + bindir="$TMP/no-zstd-path" + rm -rf "$bindir" + mkdir -p "$bindir" + for tool in bash sh grep xargs awk sed sort find wc mktemp nproc rm cat dirname tr head; do + if type -P "$tool" >/dev/null 2>&1; then + ln -sf "$(type -P "$tool")" "$bindir/$tool" + fi + done + out=$(PATH="$bindir" FM_HOME="$home" "$SEARCH" 'tugboat host' 2>&1) || rc=$? + [ "$rc" -eq 2 ] || fail "a missing search tool must be reported as such, got exit $rc" + assert_contains "$out" 'not installed' 'the refusal must name the missing tool' + case "$out" in + *'tugboat host'*) fail 'a refused search must not look like a search that ran' ;; + esac + pass "a missing decompressing search tool refuses instead of reporting no matches" +} + +# An archive built before compression carries a README that still promises plain +# grep. The rebuild must not overwrite what someone wrote there, and must not +# leave the stale command standing unremarked either. +test_refresh_names_a_readme_that_still_promises_plain_grep() { + local home out tick='`' + home=$(setup_fixture_home) + printf 'captain notes\n\nPlain %sgrep -r%s works too - grep is the index.\n' \ + "$tick" "$tick" \ + >"$home/data/transcripts/README.md" + out=$(FM_HOME="$home" FM_CLAUDE_SESSIONS="$TMP/home-src/claude" \ + FM_CODEX_SESSIONS="$TMP/home-src/codex" "$REFRESH" 2>&1 >/dev/null) \ + || fail 'refresh failed against an archive with an older README' + assert_contains "$out" 'still points readers at plain' \ + 'a README promising plain grep must be named as stale' + assert_grep 'captain notes' "$home/data/transcripts/README.md" \ + 'naming the stale sentence must not overwrite what someone wrote' + pass "a rebuild names a README that still promises plain grep instead of silently keeping it" +} + test_tool_is_tracked_and_runnable test_no_home_path_is_hardcoded test_patterns_file_does_not_match_itself @@ -381,8 +839,29 @@ test_a_run_that_recovers_no_entries_fails test_redactor_masks_a_synthetic_credential test_verification_is_zero_on_clean_and_loud_on_dirty test_verification_over_nothing_is_not_an_all_clear +test_the_store_is_written_compressed_and_leaves_nothing_plain +test_a_rebuild_compresses_a_retained_session_it_cannot_rebuild +test_verification_reads_the_compressed_store_it_verifies +test_verification_refuses_a_store_file_it_cannot_read +test_a_build_without_a_compressor_refuses_before_writing +test_refresh_without_a_compressor_leaves_no_archive test_refresh_builds_verifies_and_lands_the_bound test_refresh_does_not_overwrite_an_existing_readme test_search_resolves_the_archive_from_fm_home +test_search_formatter_is_portable_to_mawk test_search_narrows_the_file_set_by_index test_search_reports_a_missing_archive_rather_than_no_matches +test_search_reads_the_compressed_store +test_search_works_on_a_machine_without_ripgrep +test_a_scan_that_stops_early_still_reports_every_session +test_a_reader_that_hides_the_signal_still_reports_a_match +test_a_session_with_a_control_byte_is_still_reported +test_search_reads_plain_sessions_during_migration +test_plain_grep_does_not_read_the_sessions +test_search_status_says_matched_or_not_matched +test_search_reports_scanner_failures_as_errors +test_search_reports_output_pipeline_failures_as_errors +test_search_reports_file_gathering_failures_as_errors +test_search_fails_on_a_store_file_it_cannot_read +test_search_without_its_tool_refuses_rather_than_finding_nothing +test_refresh_names_a_readme_that_still_promises_plain_grep