fix(afk): backend-aware away-mode delivery and race-free verified daemon arm

[roelofb]

Intent

Fix the AFK Herdr branch readiness race: fm-afk-arm.sh must not report daemon: started or healthy until the supervise daemon has completed supervisor backend/target validation and published an explicit home-scoped readiness marker agreeing with pidfile, lock, and live pid. Startup validation failures must report daemon: FAILED with useful evidence, clean shutdown must remove readiness state, home scoping must be preserved, pkill must not be used, and Herdr readiness/liveness probes must stay read-only without resurrecting a stopped Herdr server. Keep the upstream kunchenguid/firstmate PR paused; do not open an upstream PR yet. It is acceptable for no-mistakes to update its existing fork-side validation branch or PR if its pipeline requires that.

What Changed

• Made the away-mode supervise daemon backend-aware: escalation injection and liveness checks now resolve the task's recorded backend, adding a read-only herdr delivery/liveness path (bin/backends/herdr.sh) that never resurrects a stopped herdr server.
• Added bin/fm-afk-arm.sh, a verified home-scoped daemon arm that reports daemon: started or healthy only after the daemon publishes a readiness marker agreeing with the pidfile, lock owner, and live pid — closing the startup readiness race — and reports daemon: FAILED with captured stderr evidence (the earlier undefined err_tail call now uses failure_evidence) on startup validation failure; --stop removes pid and readiness state cleanly, with no pkill anywhere.
• Covered the new behavior with tests/fm-afk-daemon-herdr-delivery.test.sh and extended herdr backend tests, and synced the afk skill, architecture, configuration, and scripts docs to the readiness-gated arm flow.

Risk Assessment

✅ Low: The only confirmed defect from round 1 (undefined err_tail) was fixed exactly as instructed; the readiness/lock/pidfile ordering, read-only herdr probes, and home-scoped stop all verify correct, and the change ships a dedicated regression test suite.

Testing

Ran the full baseline test suite plus the dedicated afk-daemon-herdr-delivery suite (all green), then manually demonstrated the readiness contract end-to-end in an isolated state home: started is only reported once the readiness marker agrees with pidfile/lock/live pid, re-arm reports healthy, --stop cleanly removes readiness state, and a startup validation failure produces a loud daemon: FAILED with evidence; transcript captured as an artifact.

Evidence: AFK arm readiness lifecycle transcript (started/healthy/stopped/FAILED with state-file evidence)

=== 1. arm inside a herdr pane === arm: daemon: started pid=77207 readiness marker: .supervise-daemon.ready = 77207 pidfile: .supervise-daemon.pid = 77207 lock owner: .supervise-daemon.lock/pid = 77207 === 2. second arm === arm: daemon: healthy pid=77207 === 3. --stop === arm: daemon: stopped pid=77207 attached arm final output: daemon: stopped (clean shutdown) .supervise-daemon.pid: removed .supervise-daemon.ready: removed === 4. failure case === arm: daemon: FAILED - exited rc=1 during startup (see .../.supervise-daemon.err) arm: ...supervisor target firstmate:0 ... does not resolve to a live pane exit code: 1

=== 1. arm inside a herdr pane: must not report started until readiness marker published ===
  arm: daemon: started pid=77207
  readiness marker: .supervise-daemon.ready = 77207
  pidfile:          .supervise-daemon.pid   = 77207
  lock owner:       .supervise-daemon.lock/pid = 77207

=== 2. second arm: reports healthy, does not start a duplicate ===
  arm: daemon: healthy pid=77207

=== 3. --stop: home-scoped clean shutdown removes readiness state ===
  arm: daemon: stopped pid=77207
  attached arm final output: daemon: started pid=77207
  attached arm final output: daemon: stopped (clean shutdown)
  .supervise-daemon.pid: removed
  .supervise-daemon.ready: removed

=== 4. failure case: no resolvable supervisor endpoint -> daemon: FAILED with evidence ===
  arm: daemon: FAILED - exited rc=1 during startup (see /var/folders/jx/qt_gfrgs2tbgs40myz32d6280000gn/T/tmp.QvfKXybddJ/failhome/state/.supervise-daemon.err)
  arm:   fm away-mode daemon FAILED to start at 2026-07-04T10:22:04+0200: supervisor target firstmate:0 (backend tmux, source FALLBACK(firstmate:0)) does not resolve to a live pane
  arm:   warn: could not auto-discover supervisor pane (no FM_SUPERVISOR_TARGET, TMUX_PANE, or HERDR_PANE_ID); falling back to 'firstmate:0' — verify this is firstmate's pane
  arm:   error: supervisor target 'firstmate:0' does not resolve to a live tmux pane; set FM_SUPERVISOR_TARGET (and FM_SUPERVISOR_BACKEND for a non-tmux pane)
  exit code: 1
  daemon stderr evidence file (.supervise-daemon.err):
    warn: could not auto-discover supervisor pane (no FM_SUPERVISOR_TARGET, TMUX_PANE, or HERDR_PANE_ID); falling back to 'firstmate:0' — verify this is firstmate's pane
    error: supervisor target 'firstmate:0' does not resolve to a live tmux pane; set FM_SUPERVISOR_TARGET (and FM_SUPERVISOR_BACKEND for a non-tmux pane)

=== 5. no pkill anywhere in the arm/daemon scripts ===
1

Evidence: Demo script used for the manual verification

#!/usr/bin/env bash
# Manual end-to-end demo of fm-afk-arm.sh readiness gating (LOM-119 fix),
# in an isolated FM_STATE_OVERRIDE home with a fake herdr and no tmux server.
set -u
ROOT=$1
TMP=$(mktemp -d)
trap 'kill "${ARM_PID:-}" 2>/dev/null; wait 2>/dev/null; rm -rf "$TMP"' EXIT
dir="$TMP/home"; fakebin="$dir/fakebin"
mkdir -p "$dir/state" "$fakebin"
printf 'idle prompt\n│ ❯ │\n' > "$dir/pane-content"
cat > "$fakebin/herdr" <<SH
#!/usr/bin/env bash
set -u
CONTENT="$dir/pane-content"
case "\${1:-} \${2:-}" in
  "status --json") printf '{"client":{"version":"0.7.1","protocol":14},"server":{"running":true}}\n' ;;
  "pane get") printf '{"result":{"pane":{"pane_id":"w1:p2"}}}\n' ;;
  "pane read") cat "\$CONTENT" ;;
  "pane send-text") text="\${!#}"; printf 'idle prompt\n│ ❯ %s │\n' "\$text" > "\$CONTENT" ;;
  "pane send-keys") printf 'idle prompt\nsubmitted\n│ ❯ │\n' > "\$CONTENT" ;;
  "agent get") printf '{"result":{"agent":{"agent_status":"idle"}}}\n' ;;
esac
exit 0
SH
chmod +x "$fakebin/herdr"
printf '#!/usr/bin/env bash\nexit 1\n' > "$fakebin/tmux"; chmod +x "$fakebin/tmux"

run() { # <herdr:0|1> cmd...
  local herdr=$1; shift
  local -a markers=(_FM_UNUSED=1)
  [ "$herdr" = 1 ] && markers=(HERDR_ENV=1 HERDR_PANE_ID=w1:p2 HERDR_SESSION=default)
  env -u TMUX -u TMUX_PANE -u FM_SUPERVISOR_TARGET -u FM_SUPERVISOR_BACKEND \
    -u HERDR_ENV -u HERDR_PANE_ID -u HERDR_SESSION -u HERDR_SOCKET_PATH \
    -u HERDR_TAB_ID -u HERDR_WORKSPACE_ID \
    PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$dir/state" "${markers[@]}" \
    FM_ESCALATE_BATCH_SECS=0 FM_HOUSEKEEPING_TICK=1 FM_POLL=1 FM_SIGNAL_GRACE=1 \
    FM_HEARTBEAT=999999 FM_CHECK_INTERVAL=999999 FM_STALE_ESCALATE_SECS=999999 \
    FM_INJECT_CONFIRM_SLEEP=0.1 "$@"
}

state="$dir/state"
date '+%s' > "$state/.afk"

echo "=== 1. arm inside a herdr pane: must not report started until readiness marker published ==="
run 1 "$ROOT/bin/fm-afk-arm.sh" > "$dir/arm.out" 2>&1 &
ARM_PID=$!
for i in $(seq 1 80); do grep -q 'daemon: started pid=' "$dir/arm.out" && break; sleep 0.25; done
sed 's/^/  arm: /' "$dir/arm.out"
echo "  readiness marker: .supervise-daemon.ready = $(cat "$state/.supervise-daemon.ready")"
echo "  pidfile:          .supervise-daemon.pid   = $(cat "$state/.supervise-daemon.pid")"
echo "  lock owner:       .supervise-daemon.lock/pid = $(cat "$state/.supervise-daemon.lock/pid")"

echo
echo "=== 2. second arm: reports healthy, does not start a duplicate ==="
run 1 "$ROOT/bin/fm-afk-arm.sh" 2>&1 | sed 's/^/  arm: /'

echo
echo "=== 3. --stop: home-scoped clean shutdown removes readiness state ==="
run 1 "$ROOT/bin/fm-afk-arm.sh" --stop 2>&1 | sed 's/^/  arm: /'
wait "$ARM_PID" 2>/dev/null; ARM_PID=
sed 's/^/  attached arm final output: /' "$dir/arm.out" | tail -2
for f in .supervise-daemon.pid .supervise-daemon.ready; do
  [ -f "$state/$f" ] && echo "  LEFTOVER: $f still present" || echo "  $f: removed"
done

echo
echo "=== 4. failure case: no resolvable supervisor endpoint -> daemon: FAILED with evidence ==="
faildir="$TMP/failhome"; mkdir -p "$faildir/state"
date '+%s' > "$faildir/state/.afk"
out=$(env -u TMUX -u TMUX_PANE -u HERDR_ENV -u HERDR_PANE_ID -u HERDR_SESSION \
  -u FM_SUPERVISOR_TARGET -u FM_SUPERVISOR_BACKEND \
  PATH="$fakebin:$PATH" FM_STATE_OVERRIDE="$faildir/state" FM_POLL=1 \
  "$ROOT/bin/fm-afk-arm.sh" 2>&1); rc=$?
printf '%s\n' "$out" | sed 's/^/  arm: /'
echo "  exit code: $rc"
echo "  daemon stderr evidence file (.supervise-daemon.err):"
sed 's/^/    /' "$faildir/state/.supervise-daemon.err" 2>/dev/null | tail -5
echo
echo "=== 5. no pkill anywhere in the arm/daemon scripts ==="
grep -n "pkill" "$ROOT/bin/fm-afk-arm.sh" "$ROOT/bin/fm-supervise-daemon.sh" | grep -v '^\s*#' | grep -cv 'NEVER\|never' || echo "  no live pkill invocations (only comments warning against it)"

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ⚠️ bin/fm-afk-arm.sh:160 - err_tail is called but never defined; the only evidence helper is failure_evidence. When a started daemon dies with non-zero rc, the arm prints err_tail: command not found instead of the captured stderr evidence — the exact evidence this fix exists to surface. Replace with failure_evidence (or tail -5 "$ERRFILE").
• ℹ️ bin/fm-afk-arm.sh:142 - : > "$ERRFILE" truncates the previous run's daemon stderr before each new launch, discarding prior failure evidence; the durable state/.subsuper-startup-failed marker still survives, so this is an acceptable tradeoff worth noting.
• ℹ️ bin/fm-supervise-daemon.sh:503 - On herdr, supervisor_input_pending always reports not-pending (no composer read primitive), leaving a bounded captain-mid-typing race; deliberately accepted and documented, with the typed-baseline submit verification as the guard.

🔧 Fix: replace undefined err_tail with failure_evidence in fm-afk-arm
1 info still open:

• ℹ️ bin/fm-afk-arm.sh:124 - The arm path only skips forking when a daemon is fully healthy; a daemon still in its pre-readiness startup window (addressable but not ready, e.g. launched by a concurrent arm) causes this arm to fork a doomed duplicate that loses the singleton lock and exits 1. The confirm loop handles the common case (reports daemon: healthy once the winner publishes readiness), but if the winner dies before this arm's child, the arm reports FAILED with 'another daemon already running' evidence — slightly confusing but honest and self-correcting on re-arm.

✅ **Test** - passed

✅ No issues found.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• Baseline configured command (all tests/*.test.sh under tmux) ran green before this validation
• bash tests/fm-afk-daemon-herdr-delivery.test.sh — herdr delivery, loud startup failure, arm FAILED honesty, and started→healthy→stopped lifecycle all pass
• Manual end-to-end demo (evidence demo.sh): armed the daemon in an isolated FM_STATE_OVERRIDE home with a fake herdr pane and no tmux server, verified daemon: started only appears with .supervise-daemon.ready == pidfile == lock owner pid, second arm reports healthy without duplicating, --stop removes pid and readiness files and the attached arm reports clean shutdown, and a no-endpoint home yields daemon: FAILED (exit 1) with the supervisor-target evidence in .supervise-daemon.err
• grep -n pkill bin/fm-afk-arm.sh bin/fm-supervise-daemon.sh — pkill appears only in a comment warning against its use
• Full suite re-run (for t in tests/*.test.sh) — all green, worktree left clean

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.