feat(bin): batch routine supervision wakes across the watcher and Pi primary - #76
Merged
Conversation
…d untangle endpoint markers
…sion to the default home
… status wakes full
…e-present open decisions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Implement the wake-churn fixes for secondmate and primary homes from the investigation report: batch all due paused and captain-held pane rechecks into one fleet stale wake while preserving per-pane bookkeeping, bounded safety rechecks, and immediate terminal, wedge, and missing or unreadable endpoint wakes; add a configurable Pi follow-up aggregator using config/wake-batch-seconds with default 60 seconds, deduplicated bounded status-path and endpoint lists, and urgent failed, blocked, needs-decision, lost-lock, and watcher-failure bypass; absorb only unchanged repeat routine status and turn-end signals for already-presented durable pauses and holds while keeping first declaration, changed terminal, missing or unreadable endpoint, and lapsed recheck immediate; deliver one drain and one acknowledgement per batch, compact unchanged OPEN DECISIONS to an unchanged-count marker while preserving exact-once UNREAD STATUS and durable acknowledgement; on Pi session_compact emit only lock and watcher ownership, actionable queue and open decisions, active task identities, and the exact next supervision instruction with no full status tails or unchanged context files; harden Pi fm-send Herdr submit so dropped or unconfirmed submits are detected and boundedly retried rather than silently reported unknown, with live Pi reproduction and regression coverage; and suppress the Firstmate repository session-start digest in a registered crew worktree with the exact one-line crew worktree - digest suppressed. Preserve AGENTS.md section 8 safety boundaries including no blind turn ends, presented-before-ack, and home-scoped repair; keep Claude primary behavior unchanged except the drain and open-decision and compact-recovery requirements; keep Bash 3.2 portability and shellcheck cleanliness; add colocated behavior tests and update docs/supervision-protocols/pi.md, docs/turnend-guard.md, docs/configuration.md, and session-start documentation. Report expected versus measured reductions using the investigation replay. Ship one PR through no-mistakes with CI green, do not merge, and do not perform Herdr lifecycle actions.
What Changed
bin/fm-watch.shnow collapses every due paused/captain-held recheck into one bounded fleet stale wake and collects missing or unreadable endpoints from a cycle into one immediate fleet wake, each with its own limit (FM_PAUSED_RESURFACE_BATCH_LIMIT,FM_ENDPOINT_BATCH_LIMIT) and per-pane bookkeeping so an omitted pane stays due; terminal, wedge, and endpoint-loss wakes stay immediate, andbin/fm-classify-lib.shmarksfailed:/blocked:/needs-decision:stale wakes urgent. Away-mode daemon triage keeps the per-window identity it parses..pi/extensions/fm-primary-pi-watch.tsaggregates routine watcher closes over a configurable window (config/wake-batch-seconds,FM_WAKE_BATCH_SECONDS, default 60s) with deduplicated bounded status-path/endpoint lists, one drain and one acknowledgement per batch, urgent-first rendering, urgent-class bypass, and arm-rotation/arm-end flush handling;bin/fm-wake-drain.shcompacts unchanged open decisions to an unchanged-count marker while--compactand--no-presentation-commitdrains keep the full block and withhold the UNREAD STATUS cursor.bin/fm-supervision-instructions.shgains self-contained--next-line/--state-linesoutput feeding a bounded Pisession_compactrecovery digest (ownership, actionable queue, open decisions, task identities, next instruction — no status tails);bin/backends/herdr.shre-observes inconclusive post-Enter surfaces (FM_BACKEND_HERDR_UNKNOWN_RECHECKS) before reporting a submit unknown;bin/fm-session-start.shprints onlycrew worktree - digest suppressedin a registered crew worktree. Docs and colocated behavior tests updated across all four areas.Risk Assessment
Testing
Re-ran the targeted session-start suites plus the watcher, Pi-extension, drain, and Herdr suites that own this change's behavior — all pass, and the round-1 hung-git regression is gone: with a wedged
gitand a 3s bound, session start now returns at 6s with the digest and its loud truncation banner instead of hanging silently to an external 25s kill, and the full session-start file runs in ~4 minutes rather than ~22. End-to-end replays on HEAD still show the intended reductions (6 supervision turns to 2 for a due-paused fleet, several watcher closes to one Pi follow-up with one ack, urgent statuses bypassing the window) and the crew-worktree digest suppression prints its exact one line against a real registered worktree. Evidence transcripts were refreshed in the evidence directory; the worktree is clean of transient artifacts.Evidence: Session start under a hung git: broken vs fixed
Source: Session start under a hung git: broken vs fixed
=== BROKEN (round-1 target bf53bf0) === exit=124 wall=25s - killed externally, NO output at all: the bound never applied. === FIXED (HEAD cc2dc20) (session-start runtime bound: 3s) === exit=0 wall=6s ... SESSION START ... LOCK ... BOOTSTRAP ... ● STARTUP TRUNCATED - SESSION START HIT ITS 3s RUNTIME BOUND ● It stopped during the "bootstrap" stage, so everything above is COMPLETE only up to that point. ● RECONCILE these stages before acting on anything they would have shown: ● bootstrap wake-queue supervision-instructions read-once fleet-state network-checks context next-stepEvidence: Repro script for the hung-git runtime bound
Source: Repro script for the hung-git runtime bound
Evidence: Crew worktree digest suppression (real git worktree)
Source: Crew worktree digest suppression (real git worktree)
--- registered crew worktree --- crew worktree - digest suppressed --- unregistered worktree (no state/*.meta pointing at it): digest still runs --- === SESSION START - /tmp/fm-crew.ikps0e/home === LOCK lock acquired: harness pid 57225Evidence: Crew worktree suppression demo script
Source: Crew worktree suppression demo script
Evidence: Wake-churn replay: 6 supervision turns to 2
Source: Wake-churn replay: 6 supervision turns to 2
RESULT HEAD (cc2dc20): supervision turns spent = 2; stale wake records presented = 2 (turn 2 delivers: "stale: paused fleet recheck (5 due): fleet:crew-2 ...; fleet:crew-6 ..." - every pane still named individually inside the single batched reason)Evidence: Pi follow-up aggregator demo
Source: Pi follow-up aggregator demo
SCENARIO A - routine wakes only: 4 watcher closes -> one follow-up at t+4.2s, "batched 3 watcher wakes" (duplicate status path deduplicated), 1 acknowledgement SCENARIO B - same wakes plus one blocked [key=deploy-window]: delivered at t+1.6s, urgent bypasses the configured windowEvidence: Evidence index
Source: Evidence index
Pipeline
Updates from git push no-mistakes
... (7 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
🔧 Fix: fix(drain): keep compact open decisions and mate status wakes full
2 issues (1 error, 1 warning) still open:
bin/fm-session-start.sh:654- The compact-recovery digest runs"$SCRIPT_DIR/fm-wake-drain.sh" --compact || truewithout redirecting stderr, unlike the ordinary digest at bin/fm-session-start.sh:795 which usesDRAIN_OUT=$("$SCRIPT_DIR/fm-wake-drain.sh" 2>&1).WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through <N> --recovery-generation <G>is printed on stderr only (bin/fm-wake-drain.sh:405 and :452), and the Pi compact route discards stderr entirely: runSessionstartHook spawns fm-sessionstart-run.sh withstdio: ["ignore", "pipe", "ignore"](.pi/extensions/fm-primary-turnend-guard.ts:105). Concrete sequence: Pi emits session_compact -> injectSessionstart("compact") -> fm-sessionstart-run.sh routes to fm-session-start.sh --compact -> the compact drain truncates .wake-queue, prints RAW_ROWS on stdout, and drives fm_recovery_marker_begin_handling so the episode is nowpending:handling:G; the ack line is written to stderr and thrown away. The recovering session receives the wakes but is never told to acknowledge them, so the recovery episode stays unacknowledged and the intent's required "one drain and one acknowledgement per batch" / "durable acknowledgement" property is broken on exactly the new path. The same redirection gap also swallows the drain's own diagnostics ("compact OPEN DECISIONS presentation failed", "decision recovery could not begin handling safely"). Fix: capture the compact drain with2>&1the way the ordinary digest already does, so the ack command and the drain's error text reach the recovering session.bin/fm-wake-drain.sh:167- The unchanged-open-decisions collapse is gated only on the persisted$STATE/.open-decisions-presentationmarker, which is written by commit_open_decisions_presentation (:210) and never cleared by anything except a drain that finds zero open decisions. Only the--compactcaller passesforce. But a brand-new session and a/clearre-emit destroy the session context just as compaction does, and both run the ordinary drain: fm-session-start.sh:795DRAIN_OUT=$("$SCRIPT_DIR/fm-wake-drain.sh" 2>&1)on the full-startup path and on--reemit. Concrete sequence: session S1 drains and presentstask1 [key=api-shape] needs-decision: pick REST or RPCin full, committing the marker; S1 quits (or the captain runs /clear); session S2 starts, takes the lock, and its startup digest's WAKE QUEUE section prints onlyOPEN DECISIONS: unchanged, 1 open- no task, no key, no note, and nobin/fm-send.sh <task> --resolve-key <key>instruction - because the decision set has not changed. S2 has none of S1's context, so the collapse's own stated premise ("the full block is already in this session's context") is false, and this is the same failure the compact fix corrected one caller at a time. The status-tail section is not a substitute: it prints only the last STATUS_TAIL (5) lines per task, so a decision buried under later routine appends - the case tests/fm-wake-drain-open-decisions.test.sh:test_buried_decision_still_surfaces exists for - never appears. The comment at bin/fm-session-start.sh:772-773 still claims "The drain's separate OPEN DECISIONS section remains actionable even when that queue is empty (AGENTS.md sections 3 and 8)", which is no longer true across a process boundary. Fix at the shared boundary: bind the marker to the presenting session (store the session/lock owner identity alongside$digest $countand force the full block when it differs), or passforcefrom the startup and --reemit digest paths as well.🔧 Fix: fix(session-start): carry compact wake ack and re-present open decisions
2 issues (1 warning, 1 info) still open:
bin/fm-session-start.sh:672- The compact-recovery digest replaces the--reemitdigest forcompacton every harness, but it never reports away mode or X mode. The--reemitpath it replaces always emitted the full supervision block (bin/fm-session-start.sh:808-832), which prints- Away mode: active; load /afk and keep normal harness supervision paused while the daemon owns the watcher.and the X-modesource <x-mode.env>line. The compact block instead callsfm-supervision-instructions.sh --harness <h> --next-line, and--next-lineaccepts no--afk/--x-modeandordinary_wake_line(bin/fm-supervision-instructions.sh:167-191) has no away-mode branch, so the output is identical whether or notstate/.afkexists. Concrete sequence: away mode is active (daemon owns the watcher,state/.afkpresent), the Pi primary compacts, fm-sessionstart-run.sh routescompacttofm-session-start.sh --compact; nothing inLOCK AND WATCHER OWNERSHIP(fm-lock.sh has no afk awareness) or in the drain (fm-guard.sh only emits its afk-aware repair line when the watcher is DOWN) mentions away mode, andNEXT SUPERVISION INSTRUCTIONtells the recovering sessionthe Pi extension already owns watcher continuity. The recovering session is handed an attended-supervision instruction while the daemon owns triage, losing exactly the away-mode boundary the intent requires preserving ("Preserve AGENTS.md section 8 safety boundaries"). The same gap hides an active X mode'ssource $CONFIG/x-mode.envrequirement. Confirm whether the compact digest is meant to carry away/X mode state (e.g. print the two state lines, or teach--next-linethe afk wording) or whether omitting them is deliberate.docs/configuration.md:148- docs/configuration.md still says the forced full OPEN DECISIONS block belongs to "A compact-recovery drain (bin/fm-wake-drain.sh --compact)", but the round-6 change generalized it:--session-recovery(bin/fm-wake-drain.sh:50-67) now sets the same force, and bin/fm-session-start.sh:798 passes it on the full-startup and--reemitdigest paths. As written the docs lead an operator to expectOPEN DECISIONS: unchanged, N openfrom a new session's or a/clearre-emit's digest, which no longer happens. The intent requires updating docs/configuration.md; extend that paragraph (and the neighboring--compactsentence) to cover every session-recovery drain.🔧 Fix: fix(session-start): report away and X mode in compact digest
1 error still open:
bin/fm-wake-drain.sh:183- The new unchanged-OPEN-DECISIONS collapse is committed by a caller that never presents the block, so the away-mode return handover degrades to a bare count. bin/fm-supervise-daemon.sh:1320 runsfm-wake-drain.sh > "$out" 2> "$err"and consumes $out strictly as TSV wake rows (case "$epoch" in ''|*[!0-9]*) continue), then rm -f's it: theOPEN DECISIONS:lines are discarded, but print_status_sections still reaches commit_open_decisions_presentation (bin/fm-wake-drain.sh:304) and writes$STATE/.open-decisions-presentation=<digest> <count>. Concrete sequence: away mode active; a crew appendsneeds-decision [key=api-shape]: pick REST or RPC; the watcher wakes, the daemon drains, and the marker is committed while no human sees the block; the captain returns and bin/fm-afk-return.sh:156 runs"$SCRIPT_DIR/fm-wake-drain.sh" 2> "$drain_err"with no --session-recovery, so line 183's[ "$prior" = "$digest $count" ]holds and the returning captain's evidence (append_evidence wake "$drained") reads onlyOPEN DECISIONS: unchanged, 1 open- no task, no key, nobin/fm-send.sh <task> --resolve-key <key>instruction. fm-afk-return.sh's scan_open_blockers (:58) is not a second surface: it matches only[ "$verb" = blocked ], so a needs-decision has none. Before this change every drain reprinted the full block, so the away-return handover always carried it, and the intent'sPreserve AGENTS.md section 8 safety boundariescovers exactly this handover. Fix at the shared boundary: the collapse's premise isalready in this session's context, which is false for a machine consumer and for a captain returning from away mode - do not commit the presentation marker on the daemon's non-presenting drain (or gate the commit on a presenting caller), and pass --session-recovery from bin/fm-afk-return.sh:156.🔧 Fix: fix(drain): never spend open-decisions collapse on unread drains
3 issues (1 warning, 2 infos) still open:
bin/fm-wake-drain.sh:314- The new--no-presentation-commitboundary is half-applied. Its own comment states the invariant generally - "A caller that consumes this drain's rows mechanically and discards the presented text has shown it to nobody, so it must not spend that record" (bin/fm-wake-drain.sh:40-45) - but the flag is only read by commit_open_decisions_presentation (:237). print_status_sections still callsstatus_acknowledge_presented_snapshot(:311) andstatus_commit_presentation_snapshot(:314) unconditionally on the very same drain. bin/fm-supervise-daemon.sh:1320 runs that drain and consumes $out strictly as TSV wake rows (case "$epoch" in ''|*[!0-9]*) continue), then rm -f's it, so everyUNREAD STATUSline is discarded while its presentation cursor is advanced. Concrete sequence: away mode active; crew task1 appendsnote: upstream contract changed, see thread(routine informational, not a decision so the open-decisions guard does not cover it); the watcher wakes, the daemon drains, the UNREAD STATUS block is generated and thrown away, and status_commit_presentation_snapshot advances task1's cursor past those bytes; handle_wake only distills the wake reason, not the status bytes; the captain returns and bin/fm-afk-return.sh:156 drains again - the line is already recorded as presented and never appears. The intent requires "preserving exact-once UNREAD STATUS", and exact-once means presented exactly once to a reader, not consumed by a machine. Same fix shape as the open-decisions one: gate status_commit_presentation_snapshot (and its acknowledge sibling) on PRESENTATION_COMMIT too, so the daemon's non-presenting drain leaves the unread span for whoever reads it next. Flagging as ask-user rather than auto-fix because it changes what a returning captain is shown and the daemon may be intended to own that distillation..pi/extensions/fm-primary-pi-watch.ts:283- wakeIsUrgent classifies a wake either by regex over the message or by resolving*.statuspaths found in the message and reading their last line. The watcher's terminal-stale wake reason carries neither: bin/fm-watch.sh:1456 and :1471 emit the bare stringstale: <window>(no verb, no status path). So a crew whose last status isfailed: ...and whose pane then goes stale producesstale: test:fm-x, which queueWake (:364) treats as routine and holds for the full wake-batch window (default 60s) instead of taking the urgent bypass the intent requires for "urgent failed, blocked, needs-decision, lost-lock, and watcher-failure". The primary delivery path is covered - the same failure normally arrives first assignal: <state>/<task>.status, which wakeIsUrgent does read and does flush immediately - so the impact is bounded to a <=60s delay on the secondary stale path. Confirm whether that is deliberate, or whether the extension should mapstale: <window>back to its task (via state/*.meta, the way the watcher does) before classifying, or the watcher should carry the terminal verb in the stale reason.bin/fm-wake-drain.sh:150- open_decisions_digest introducesgit hash-object --stdinas the only git dependency in the drain's presentation path; every other hashing site in this repo (bin/fm-backlog-receive.sh:33, bin/fm-backlog-handoff.sh:86, bin/fm-backend-hometag-lib.sh:46) uses a shasum/sha256sum fallback pair.git hash-object --stdindoes work outside a repository (verified), so the only failure is git being absent from PATH - but thendigest=$(...) || return 1at :191 aborts print_open_decisions_section, which aborts print_status_sections, so OPEN DECISIONS and RECORD DIVERGENCE are never printed AND status_commit_presentation_snapshot (:314) never runs, leaving the same unread status replayed on every subsequent drain. Low probability, disproportionate blast radius; use the repo's existing shasum/sha256sum fallback so a missing git degrades to a full re-present rather than suppressing the whole section.🔧 Fix: fix(drain): withhold unread cursor and mark urgent stale wakes
2 infos still open:
bin/fm-supervision-instructions.sh:230---next-linestrips only the- Ordinary wake:prefix and prints the harness line verbatim, but three of those lines end in a forward reference to content the compact digest deliberately omits: codex says "take the next foreground bin/fm-watch-checkpoint.sh checkpoint as directed below", grok says "as directed below", and theunknownfallback says "follow the continuation in the harness protocol below". The compact digest ends atNEXT SUPERVISION INSTRUCTIONwith no protocol snippet after it (bin/fm-session-start.sh:678-684), so a recovering session on those harnesses is pointed at instructions that are not present. Reachable for a codex-execcompacthook payload and, more commonly, wheneverfm-harness.shcannot detect the harness during the compact hook and falls back tounknown. The claude and pi lines are self-contained and read correctly. Either give--next-lineself-contained wording for those cases or have the compact digest name where the protocol lives..pi/extensions/fm-primary-pi-watch.ts:110-wakeBatchSecondsis resolved once at module load (const wakeBatchSeconds = configuredWakeBatchSeconds()), so editingconfig/wake-batch-secondshas no effect until Pi is restarted. docs/configuration.md's new "Pi watcher wake batching" section says only "Write one positive integer number of seconds to gitignoredconfig/wake-batch-secondsto change it for that home", which leads an operator to expect the running session to pick it up. The neighbouringFM_*knobs are process-env by nature so the same caveat is implicit there, but a config file is not. Add the restart requirement to that paragraph.🔧 Fix: fix(supervision): make every harness next line self-contained
3 issues (1 warning, 2 infos) still open:
bin/fm-supervision-instructions.sh:206-ordinary_wake_linewas rewritten so every harness line now beginsdrain and handle this wake with bin/fm-wake-drain.sh, then run the exact --ack-through command it printed. That line is printed unconditionally by the full digest at bin/fm-supervision-instructions.sh:255, including on the read-only path, two lines below- Lock: read-only; do not drain, arm, spawn, steer, merge, or repair fleet state here.Verified:bin/fm-supervision-instructions.sh --harness pi --read-only 1emits both lines back to back. Before this change codex/pi/opencode/grok ordinary lines contained no drain instruction at all (the Pi extension already owns watcher continuity; do not arm another cycle.), and claude/cursor said onlydrain and handle the wakewithout naming the command or the acknowledgement, so this is a new and much more actionable contradiction on four harnesses.bin/fm-wake-drain.shhas no lock-ownership gate of its own: a read-only session that follows the instruction truncates$FM_WAKE_QUEUE, advances the UNREAD STATUS cursor and the open-decisions presentation marker, consuming the lock holder's durable wakes - the AGENTS.md section 8 boundary the intent requires preserving. Fix: suppress or reword the drain clause when READ_ONLY=1 (the--next-linecompact path already routes read-only sessions torepair_line, which handles it correctly).bin/fm-session-start.sh:658- The compact-recovery digest prints theACTIONABLE QUEUE AND OPEN DECISIONSheader and then whateverfm-wake-drain.sh --compactemits. With an empty queue and no open decisions the drain is completely silent, so the section is a bare header. Verified by running the compact digest against a fresh home: the header is followed immediately byACTIVE TASK IDENTITIES. The full digest already handles this at bin/fm-session-start.sh:805-809 with an explicit(no queued wakes)fallback. For a digest whose entire purpose is a self-contained handover to a session that just lost its context, an empty section is indistinguishable from a drain that failed or was suppressed. Mirror the(no queued wakes)fallback: capture the drain output and print a positive nothing-pending line when it is empty..pi/extensions/fm-primary-pi-watch.ts:326-confirmHandlingDeliveryWithRetrymoved out ofdeliverActionableWakeand intoflushWakeBatch, so it now runs up towakeBatchSeconds(default 60s) after the wake was produced instead of synchronously with it.bin/fm-watch-arm.sh:404requiresfm_pid_alive "$handling_watcher_pid"beforefm_recovery_marker_begin_handling, andsnapshot()only rescues a stale pid whileowner.childis non-null. Concrete sequence: a routine actionable wake queues recovery {G,B} and starts the 60s timer; within that window B's arm child closes non-actionably,releaseChild()setsowner.child = null, andscheduleRetryeither waits out its backoff or bails (lock lost / retryLimit exceeded) leavingowner.childnull; the flush then confirms against the dead pid B, gets a non-zero status, and pusheswatcher: FAILED - handling delivery confirmation was rejected (...)into the batch as an urgent detail even though nothing about the recovery episode is actually wrong (the drain's ownfm_recovery_marker_begin_handlingat bin/fm-wake-drain.sh:421 still covers the real transition). Bounded to a misleading extra line during an already-degraded window, but it did not exist before batching. Consider capturing the confirmation at queue time, or skipping it when the recorded arm is gone.🔧 Fix: fix(pi): flush a wake batch when its arm ends unreplaced
3 issues (2 warnings, 1 info) still open:
.pi/extensions/fm-primary-pi-watch.ts:390- The new arm-end flush is disabled for any batch that spans more than one arm - i.e. exactly the aggregating case wake batching exists for.owner.wakeBatchArmis set only when the batch is EMPTY (queueWake:390) and is never re-pointed at a successor, while an actionable close deliberately does not flush. Concrete sequence: arm A closes actionably -> successor B started -> deliverActionableWake queues W1 while owner.child===B, so wakeBatchArm=B; 10s later B closes actionably -> successor C started -> W2 queued, batch non-empty so wakeBatchArm STAYS B (already dead, and B will never emit another close); C then closes non-actionably -> the close handler calls flushWakeBatchOnArmEnd(owner, C), the guardowner.wakeBatchArm !== armChildis true, and it returns without flushing. The batch waits out the full wakeBatchSeconds window under a broken chain and then confirms against C's dead watcher pid - the precise failure the round-11 instruction ("A batch never outlives its originating watcher arm unconfirmed") required removing. After the first rotation the ownership guard is permanently unsatisfiable for that batch. Fix: hand ownership to the replacement on an actionable rotation (set wakeBatchArm to the new owner.child when a non-empty batch survives restoreAfterActionableClose), or track the batch against the CURRENT arm rather than the opening one. The round-11 test only exercises a single-wake batch, so it passes with this hole open..pi/extensions/fm-primary-pi-watch.ts:330- When the arm-end flush does fire, its handling confirmation is guaranteed to be rejected and appended as a spurious urgentwatcher: FAILEDline. bin/fm-watch-arm.sh:548-568 printswatcher: started pid=$child ... recovery-generation=...and thenwait "$child", so the arm child process only exits AFTER its watcher pid has exited; the extension'scloseevent therefore fires when recovery.watcherPid is already dead. flushWakeBatchOnArmEnd runs afterreleaseChild()(owner.child===null), so confirmHandlingDeliveryWithRetry's snapshot() falls back to the stale recovery,bin/fm-watch-arm.sh --handling-deliveredfails itsfm_pid_alive "$handling_watcher_pid"gate at :404, and flushWakeBatch:331-332 pusheswatcher: FAILED - handling delivery confirmation was rejected (...)into the delivered batch even though the recovery episode is fine (the drain's own fm_recovery_marker_begin_handling still covers it). Concrete sequence: arm A closes actionably -> successor B -> W1 queued with recovery{G,B}, wakeBatchArm=B -> B closes non-actionably 5s later -> flush -> confirm(dead B) -> rejected -> the supervisor is told the watcher failed for a reason that did not happen. Pre-batching, the confirmation ran synchronously while B was alive and succeeded. The round-11 test's fake arm script exits 0 unconditionally for--handling-delivered, so it cannot detect this. Fix: skip the confirmation (and its failure detail) when the recorded watcher pid is no longer alive - there is nothing left to confirm at that point.bin/fm-session-start.sh:680- The compact-recovery digest passes --afk tofm-supervision-instructions.sh --state-linesbut not to the--next-linecall two sections later, and ordinary_wake_line has no away-mode branch (repair_line does). Under away mode the digest therefore prints- Away mode: active; load /afk and keep normal harness supervision paused while the daemon owns the watcher.and then, as the NEXT SUPERVISION INSTRUCTION,drain and handle this wake with bin/fm-wake-drain.sh, then run the exact --ack-through command it printed; .... The recovering session is handed an attended drain-and-acknowledge order while bin/fm-supervise-daemon.sh owns triage and drains the same queue. Confirm whether --next-line should take --afk and route to the away-mode wording (as repair_line already does), or whether the state line alone is considered sufficient.🔧 Fix: fix(pi): rotate batch arm ownership and skip ended-arm confirms
2 infos still open:
bin/fm-watch.sh:294- The window_key header claims to be the single owner of the per-window marker-file format and enumerates every family (".hash-, .count-, .stale-, .stale-since-, .wedge-escalations-, .paused-, .writing-"), warning that a second copy is how a future change silently orphans a window's markers. The change adds a new family, .endpoint-missing-<key>, which is not listed there and is also not cleared by clear_pause_tracking (bin/fm-watch.sh:615) alongside .stale-/.stale-since-/.wedge-escalations-. Add it to that enumeration so the stated owner stays complete..pi/extensions/fm-primary-pi-watch.ts:313- The arm-end flush skips confirmHandlingDeliveryWithRetry unconditionally rather than on !pidAlive(recovery.watcherPid). The header comment justifies it with "an arm process only closes AFTER its own watcher pid has exited", which holds for both bin/fm-watch-arm.sh paths (started+wait and attach_and_wait) but not for an arm killed out-of-band while its watcher survives. Verified this is not a durability loss: bin/fm-wake-drain.sh:461 performs fm_recovery_marker_begin_handling on the drain the batch orders, so the recovery episode still transitions. Noting the tradeoff only; a pidAlive guard would make the code match its stated premise exactly.🔧 **Test** - 1 issue found → auto-fixed ✅
bin/fm-session-start.sh:267- bin/fm-session-start.sh:267 runs fm_root_is_registered_crew_worktree (which shells out togit rev-parse) BEFORE the "--- 0. runtime bound ---" wrapper, in both the parent and the timed child. With a slow or hung git (stuck index.lock, hung network mount, unresponsive filesystem) session start now emits NO digest at all and never returns within its own runtime bound: measured exit=124 after an external 25s kill with FM_SESSION_START_TIMEOUT=3, versus the base commit which truncated loudly at 3s and still printed LOCK/BOOTSTRAP output. The repository's existing test tests/fm-session-start.test.sh::test_runtime_bound_truncates_loudly_and_exits_zero (which installs a hanginggit) fails deterministically on the target commit and passes on the base commit; it also stretched the suite from seconds to ~22 minutes. Fix: move the crew-worktree suppression check inside the runtime-bound child (or bound/timeout the git calls in fm_root_is_registered_crew_worktree) so the digest's hard bound and truncation contract still hold. Repro transcript: /Users/ivan/.no-mistakes/evidence/01M0VT8T0CHZ154T3SVWEQETHM/session-start-hung-git-repro.txtbin/fm-test-run.sh tests/fm-watch-triage.test.sh tests/fm-pi-watch-extension.test.sh tests/fm-wake-drain-open-decisions.test.sh tests/fm-wake-drain-open-decisions-cursor.test.sh tests/fm-wake-drain-unread-status.test.sh tests/fm-supervision-instructions.test.sh tests/fm-session-start.test.sh tests/fm-sessionstart-nudge.test.sh tests/fm-backend-herdr.test.sh— all green except tests/fm-session-start.test.sh (exit=1)tests/fm-session-start.test.sh::test_runtime_bound_truncates_loudly_and_exits_zeroisolated on the target tree (hangs) and on agit archive d982d45baseline tree (passes in seconds)Watcher replay driving the realbin/fm-watch.shover synthetic 6- and 9-pane paused fleets on base vs target, draining and acknowledging each wake like a primary:wake-churn-replay.sh <tree> <label> <fleet>Pi aggregator demo loading the real.pi/extensions/fm-primary-pi-watch.tswithconfig/wake-batch-seconds=3: routine-only batch vs urgentblocked [key=...]bypass, dedup of repeated status paths, single delivery acknowledgementRealbin/fm-wake-drain.shtwice in one session (full OPEN DECISIONS then unchanged-count marker) andbin/fm-session-start.sh --compactafter a queued wakeRealbin/fm-session-start.shinvoked with FM_ROOT_OVERRIDE pointing at a git worktree registered asworktree=in a primary's state meta (crew-worktree suppression line)Herdr submit reproduction driving the realfm_backend_herdr_send_text_submitwith a transiently unreadable post-Enter surface, base vs targetbin/fm-session-start.shunder a hunggitwith FM_SESSION_START_TIMEOUT=3, base vs target🔧 Fix: bound crew-worktree check inside session-start runtime bound
✅ Re-checked - no issues remain.
bash tests/fm-session-start.test.sh— 50 assertions pass in 3m54s, includingthe pure-Bash watchdog bounds session start, kills its hung grandchild, and emits the truncation contract(the test that failed deterministically in round 1)bash tests/fm-sessionstart-nudge.test.sh— 22 assertions pass, includingsession-start hooks suppress the primary digest in a registered crew worktreebash tests/fm-watch-triage.test.sh(82 ok),bash tests/fm-pi-watch-extension.test.sh(40 ok),bash tests/fm-wake-drain-open-decisions.test.sh(13 ok),bash tests/fm-wake-drain-unread-status.test.sh(15 ok),bash tests/fm-backend-herdr.test.sh(182 ok)Manual repro on HEAD with agitstub that ignores TERM and sleeps 600s,FM_SESSION_START_TIMEOUT=3, wrapped intimeout 25: exit 0 at 6s wall, digest through BOOTSTRAP plus the STARTUP TRUNCATED banner (round-1 target: exit 124, no output)Manual live crew-worktree check: realgit init+git worktree add, worktree recorded in the primary'sstate/task1.meta, driving the realbin/fm-session-start.sh— prints exactlycrew worktree - digest suppressed; an unregistered linked worktree still runs the full digestManual replaywake-churn-replay.sh <tree> 'HEAD (cc2dc20)' 6driving the realbin/fm-watch.sh+bin/fm-wake-drain.sh— 6 supervision turns collapse to 2Manualpi-wake-aggregator-demo.shdriving the real.pi/extensions/fm-primary-pi-watch.ts— 4-5 watcher closes become one batched follow-up with one acknowledgement at t+4.2s; ablocked [key=deploy-window]status bypasses the window at t+1.6s🔧 **Document** - 2 issues found → auto-fixed ✅
bin/fm-session-start.sh:298- Repository lint fails on this branch independently of documentation: bin/fm-lint.sh exits 1 on shellcheck SC2016 at bin/fm-session-start.sh:298 (the single-quoted. "$1/fm-primary-scope-lib.sh"; fm_root_is_registered_crew_worktree "$2"bash -c body, introduced by cc2dc20). The intent requires shellcheck cleanliness. Fixing it means editing an executable file, which is outside this documentation phase.docs/configuration.md:148- Judgment call, left as-is: this change put the OPEN DECISIONS unchanged-collapse and --no-presentation-commit cursor semantics in docs/configuration.md, while docs/architecture.md already owned the drain fold/presentation contract. I treated configuration.md as the owner and reduced architecture.md to a one-line pointer rather than migrating the prose; a later consolidation could move that mechanism rationale into architecture.md and leave configuration.md owning only the knobs.🔧 Fix: silence intentional SC2016 in session-start crew check
✅ Re-checked - no issues remain.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.