fix(afk): backend-aware away-mode delivery and race-free verified daemon arm

.agents/skills/afk/SKILL.md

@@ -23,15 +23,30 @@ batched digest rather than per-wake injections.
    This file survives a firstmate restart: recovery re-enters afk if the
    flag is present.
 
-2. **Ensure the sub-supervisor daemon is running.** Check the pid file; start
-   the daemon only if it is dead or absent:
-   ```sh
-   if [ -f state/.supervise-daemon.pid ] && kill -0 "$(cat state/.supervise-daemon.pid)" 2>/dev/null; then
-     : # daemon already alive - it picks up the flag on its next cycle
-   else
-     nohup bin/fm-supervise-daemon.sh >/dev/null 2>&1 &
-   fi
-   ```
+2. **Ensure the sub-supervisor daemon is running - verified, never
+   fire-and-forget.** Run `bin/fm-afk-arm.sh` as the harness's own tracked
+   background task (the same discipline as `bin/fm-watch-arm.sh`, standalone,
+   nothing else in that call). It no-ops with `daemon: healthy pid=N` when a
+   live daemon already holds this home's lock and has published its
+   post-validation readiness marker (`state/.supervise-daemon.ready`, written
+   only after the supervisor endpoint validates, removed on shutdown), launches
+   and confirms one with `daemon: started pid=N` otherwise, and prints
+   `daemon: FAILED - <reason>`
+   (non-zero exit, with captured startup-failure evidence) when no daemon could
+   be confirmed. Treat that one line, not a pid-file guess, as the truth.
+   Because the arm stays attached to the daemon it started, a later daemon
+   death completes the tracked task and re-notifies firstmate - re-enter this
+   step then.
+
+   **Never** start the daemon with `nohup ... >/dev/null 2>&1 &`: that
+   discards the startup failure the daemon exits with when its supervisor
+   endpoint cannot be validated, and away-mode supervision is silently off
+   (incident LOM-119: exactly this, under the herdr backend). The daemon
+   validates its endpoint on both tmux and herdr (auto-discovered from
+   `$TMUX_PANE` or herdr's `HERDR_ENV`/`HERDR_PANE_ID` markers) and injects
+   through the matching backend adapter; a failed validation leaves a durable
+   `state/.subsuper-startup-failed` marker as evidence.
+
    The daemon is **presence-gated**: it injects escalations only while
    `state/.afk` exists, and stays quiet otherwise.
 
@@ -47,11 +62,13 @@ batched digest rather than per-wake injections.
 No `/back` is needed. The first genuine message is the return signal:
 
 - A message **without** the sentinel marker and **not** starting with `/afk`
-  -> the captain is back. Clear `state/.afk`, stop the daemon, flush one
-  distilled "while you were out" catch-up (drain `state/.wake-queue`, summarize
-  any pending escalations from `state/.subsuper-escalations` and any
-  `state/.subsuper-inject-wedged` marker), and resume full per-wake
-  responsiveness (arm `bin/fm-watch-arm.sh`).
+  -> the captain is back. Clear `state/.afk`, stop the daemon with
+  `bin/fm-afk-arm.sh --stop` (home-scoped: it signals only the pid this home's
+  pid file and lock agree on - never `pkill`), flush one distilled "while you
+  were out" catch-up (drain `state/.wake-queue`, summarize any pending
+  escalations from `state/.subsuper-escalations` and any
+  `state/.subsuper-inject-wedged` or `state/.subsuper-startup-failed` marker),
+  and resume full per-wake responsiveness (arm `bin/fm-watch-arm.sh`).
 - A message **with** the sentinel marker (`FM_INJECT_MARK`, ASCII 0x1f) -> it
   is a daemon escalation; stay afk and process it.
 - Re-invoking `/afk` while already away -> stay afk (refresh the flag); this
@@ -97,13 +114,21 @@ afk mode the composer guard is belt-and-suspenders (no human is typing), but it
 protects against the race window between the captain returning and their
 message landing, and against the daemon's own previous injection sitting unsent.
 
+These checks dispatch on the supervisor pane's backend. The tmux arm is the
+description above, unchanged. On herdr, busy comes from the backend's native
+agent state, there is no composer read (herdr's CLI exposes none - the
+typed-baseline verification inside its submit primitive is the swallowed-Enter
+guard), and the submit goes through `bin/backends/herdr.sh`'s verified
+send-text/Enter sequence with the same empty/pending/unknown verdicts.
+
 **Max-defer escape (the daemon must never silently wedge).**
 If anything stays buffered past `FM_MAX_DEFER_SECS` (default 300), the daemon
 attempts one normal flush, which still requires an idle pane and empty composer.
 If that submit cannot be confirmed, it raises a loud, rate-limited wedge alarm:
 an ERROR in the daemon log, a durable
 `state/.subsuper-inject-wedged` marker (surface it on the "while you were out"
-catch-up if present), and a flash on the supervisor client's status line.
+catch-up if present), and a flash on the supervisor client's status line (a
+tmux-only nicety; on herdr the log and marker are the loud parts).
 So a guard false-positive becomes a visible stall, never an unbounded silent no-op.
 
 ## Submit model
@@ -178,7 +203,8 @@ the marker lets firstmate distinguish it from a real captain message.
   buffered past `FM_MAX_DEFER_SECS` (default 300s), the daemon attempts one
   normal flush, which still requires an idle pane and empty composer. If that
   cannot confirm a submit, it raises a loud, rate-limited wedge alarm: ERROR log,
-  durable `state/.subsuper-inject-wedged` marker, and a status-line flash. A
+  durable `state/.subsuper-inject-wedged` marker, and a status-line flash (tmux
+  only; herdr has no display-message analog). A
   composer false-positive surfaces as a visible stall, never an unbounded silent
   no-op.
 - **Verified type-once submit model** - the digest is typed once via
@@ -195,9 +221,12 @@ the marker lets firstmate distinguish it from a real captain message.
   both check the seen-status marker before escalating, so a status escalated by
   one path is not re-escalated by another in the same digest.
 - **Auto-discovered supervisor pane** - the daemon resolves its injection target
-  from `FM_SUPERVISOR_TARGET`, then `$TMUX_PANE`, then a `firstmate:0` fallback
-  with a warning. The resolution source is logged at startup so a
-  wrong-but-resolving fallback is detectable.
+  from `FM_SUPERVISOR_TARGET`, then `$TMUX_PANE`, then herdr's
+  `HERDR_ENV`/`HERDR_PANE_ID` markers, then a `firstmate:0` tmux fallback with a
+  warning. The pane's backend resolves from the same markers
+  (`FM_SUPERVISOR_BACKEND` overrides; an explicit `FM_SUPERVISOR_TARGET` alone
+  keeps its legacy tmux meaning). The resolution source is logged at startup so
+  a wrong-but-resolving fallback is detectable.
 
 ## Reliability properties
 

AGENTS.md

@@ -745,9 +745,10 @@ Inline facts that must survive without a loaded skill:
 
 - Every daemon injection is prefixed with `FM_INJECT_MARK`, ASCII unit separator `0x1f`, so internal escalations are distinguishable from a captain message.
 - While `state/.afk` exists, the daemon owns the watcher; do not separately arm `fm-watch-arm.sh` or `fm-watch.sh`.
+- Start or stop the daemon only through `bin/fm-afk-arm.sh`, run standalone as the harness's own tracked background task; it verifies the daemon and prints one honest `daemon: started|healthy|FAILED|stopped` line. Never launch it fire-and-forget with `nohup ... &` (a startup failure then dies silently and away-mode supervision is off), and never `pkill` (secondmate homes run the same script).
 - If firstmate receives a marked message while afk is active, it is an internal escalation: stay afk and process it.
 - If the message starts with `/afk`, stay afk and refresh the flag.
-- Any other unmarked message means the captain is back: clear `state/.afk`, stop the daemon, flush catch-up from `state/.wake-queue`, `state/.subsuper-escalations`, and `state/.subsuper-inject-wedged`, then re-arm normal watcher supervision.
+- Any other unmarked message means the captain is back: clear `state/.afk`, stop the daemon with `bin/fm-afk-arm.sh --stop`, flush catch-up from `state/.wake-queue`, `state/.subsuper-escalations`, `state/.subsuper-inject-wedged`, and `state/.subsuper-startup-failed`, then re-arm normal watcher supervision.
 - Afk never changes approval authority; PR merges, ask-user findings, destructive actions, irreversible actions, and security-sensitive choices still require the same approval they required before.
 - Bias ambiguous cases toward exit because a present captain beats token savings and a false exit is self-correcting.
 

CONTRIBUTING.md

@@ -76,6 +76,7 @@ tests/fm-send-secondmate-marker.test.sh   # fm-send from-firstmate marker for ki
 tests/fm-wake-daemon-lifecycle-e2e.test.sh # watcher + daemon lifecycle e2e: restart catch-up, batching, dedupe, stale-pane routing, and digest injection
 tests/fm-composer-ghost.test.sh           # dim-ghost stripping, ghost-only composer detection, and escape-free peek tests
 tests/fm-afk-inject-e2e.test.sh           # private-socket end-to-end test of the afk injection path (partial-input deferral, swallowed-Enter retry)
+tests/fm-afk-daemon-herdr-delivery.test.sh # away-mode daemon under herdr (incident LOM-119): digest delivery to a herdr supervisor pane, loud startup failure with a durable marker, and fm-afk-arm started/healthy/FAILED/--stop honesty
 tests/fm-bootstrap.test.sh                # bootstrap dependency, feature-probe, and crew-dispatch reporting tests
 tests/fm-session-start.test.sh            # fm-session-start.sh: ABSENT vs empty-vs-present digest files, lock-refusal read-only path skipping every mutating step, diagnostics-first section ordering, status-tail bounding, tmux/herdr/zellij endpoint liveness, and composition of the real fm-lock/fm-bootstrap/fm-wake-drain scripts
 tests/fm-grok-harness.test.sh             # grok adapter spawn hook, token guard, teardown cleanup, and session-lock detection tests

bin/backends/herdr.sh

@@ -513,6 +513,21 @@ fm_backend_herdr_target_ready() {  # <target>
   fm_backend_herdr_server_ensure "$FM_BACKEND_HERDR_SESSION" || return 1
 }
 
+# fm_backend_herdr_pane_alive: READ-ONLY liveness probe for one pane target -
+# a single `herdr pane get` call, deliberately NOT routed through
+# fm_backend_herdr_target_ready/fm_backend_herdr_server_ensure, so probing a
+# deliberately stopped server never resurrects it (a bare socket CLI call does
+# not auto-start the server - verified, see server_ensure above). A dead
+# server, an unreachable socket, or a missing pane all fail; alive means the
+# response carries the pane's id. Cheap enough for a ~1s poll loop.
+fm_backend_herdr_pane_alive() {  # <target>
+  fm_backend_herdr_parse_target "$1" || return 1
+  local out pane
+  out=$(HERDR_SESSION="$FM_BACKEND_HERDR_SESSION" herdr pane get "$FM_BACKEND_HERDR_PANE" 2>/dev/null) || return 1
+  pane=$(printf '%s' "$out" | jq -r '.result.pane.pane_id // empty' 2>/dev/null)
+  [ -n "$pane" ]
+}
+
 # fm_backend_herdr_current_path: the live FOREGROUND process's cwd, or empty on
 # any error. Mirrors tmux's pane_current_path poll used for worktree-path
 # discovery after `treehouse get`.

bin/fm-afk-arm.sh

@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# Safe, home-scoped (re-)arm of the away-mode sub-supervisor daemon
+# (bin/fm-supervise-daemon.sh), with honest verification - the daemon's
+# counterpart to bin/fm-watch-arm.sh, born from the same failure mode.
+#
+# WHY THIS EXISTS (incident LOM-119 / afk-daemon-herdr-delivery): the /afk skill
+# used to start the daemon with `nohup ... >/dev/null 2>&1 &` - fire-and-forget.
+# When the daemon exited at startup (under herdr its supervisor-endpoint probe
+# was tmux-only and always failed), both the non-zero exit and the stderr error
+# vanished. Because the daemon owns the watcher while state/.afk exists, its
+# silent death meant NO watcher, a stale liveness beacon, and captain-relevant
+# wakes stranded in state/.wake-queue until the captain happened to type.
+# Reliability requires the same discipline the watcher already has: launch
+# through a mechanism that SURVIVES the call and NOTIFIES on exit (the
+# harness's tracked background task), and VERIFY the outcome before settling in.
+#
+# Run this as the harness's OWN tracked background task, standalone, never
+# bundled onto the tail of another command and never with a shell `&` inside
+# another call. It forks the daemon as a tracked child, confirms the daemon
+# genuinely holds this home's singleton lock and has published its post-startup
+# readiness marker, and prints exactly one unambiguous status line:
+#   daemon: started pid=<N>          - it launched one and confirmed it
+#   daemon: healthy pid=<N>          - a genuinely live daemon already held the lock
+#   daemon: FAILED - <reason>        - could not confirm one (exits non-zero)
+# On started, this stays attached (`wait`) for the daemon's whole life, so a
+# later daemon death completes the tracked task and re-notifies firstmate
+# instead of passing unnoticed. The daemon's stdout/stderr are captured to
+# state/.supervise-daemon.err so a failure always leaves readable evidence
+# (alongside the daemon's own durable state/.subsuper-startup-failed marker).
+#
+# --stop: stop ONLY this FM_HOME's daemon (the pid recorded in THIS home's
+# state/.supervise-daemon.pid, cross-checked against the lock). It resolves and
+# signals exactly that pid, so it can never touch another home's daemon. NEVER
+# `pkill -f fm-supervise-daemon.sh`: secondmate homes run the same script.
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=bin/fm-wake-lib.sh
+. "$SCRIPT_DIR/fm-wake-lib.sh"
+
+DAEMON="$SCRIPT_DIR/fm-supervise-daemon.sh"
+LOCK="$STATE/.supervise-daemon.lock"
+PIDFILE="$STATE/.supervise-daemon.pid"
+READYFILE="$STATE/.supervise-daemon.ready"
+ERRFILE="$STATE/.supervise-daemon.err"
+STARTUP_FAILED="$STATE/.subsuper-startup-failed"
+# How long to wait for a freshly forked daemon to validate its supervisor
+# endpoint and acquire the lock. The herdr probe may ensure a server first
+# (bounded at ~10s inside the adapter), so this sits above that.
+CONFIRM_TIMEOUT=${FM_AFK_ARM_CONFIRM_TIMEOUT:-15}
+
+# A daemon is addressable iff the pidfile names a live process AND the singleton
+# lock's owner pid agrees. This is enough for home-scoped --stop, but not enough
+# to claim startup succeeded: the daemon publishes lock+pid before it validates
+# the supervisor endpoint.
+ADDRESSABLE_PID=
+addressable_daemon() {
+  local pid lock_pid
+  ADDRESSABLE_PID=
+  pid=$(cat "$PIDFILE" 2>/dev/null || true)
+  fm_pid_alive "$pid" || return 1
+  lock_pid=$(cat "$LOCK/pid" 2>/dev/null || true)
+  [ "$lock_pid" = "$pid" ] || return 1
+  ADDRESSABLE_PID=$pid
+  return 0
+}
+
+# A daemon is "healthy" iff pidfile, lock, live pid, and readiness marker all
+# agree. Sets HEALTHY_PID on success. This is the honesty gate: this script
+# never reports a daemon that is only in its pre-validation startup window.
+HEALTHY_PID=
+healthy_daemon() {
+  local ready_pid
+  HEALTHY_PID=
+  addressable_daemon || return 1
+  ready_pid=$(cat "$READYFILE" 2>/dev/null || true)
+  [ "$ready_pid" = "$ADDRESSABLE_PID" ] || return 1
+  HEALTHY_PID=$ADDRESSABLE_PID
+  return 0
+}
+
+failure_evidence() {
+  if [ -s "$STARTUP_FAILED" ]; then
+    sed 's/^/  /' "$STARTUP_FAILED"
+  fi
+  [ -s "$ERRFILE" ] && tail -5 "$ERRFILE" | sed 's/^/  /'
+}
+
+mode=arm
+case "${1:-}" in
+  ''|arm|--arm) mode=arm ;;
+  --stop) mode=stop ;;
+  *) echo "usage: $(basename "$0") [--stop]" >&2; exit 2 ;;
+esac
+
+if [ "$mode" = stop ]; then
+  # Home-scoped stop: only the daemon pid recorded in THIS home's pidfile,
+  # and only when this home's lock agrees it is the daemon. Readiness is not
+  # required for stop: a daemon caught during startup is still this home's
+  # addressable process and should be stoppable.
+  if addressable_daemon; then
+    stop_pid=$ADDRESSABLE_PID
+    kill -TERM "$stop_pid" 2>/dev/null || true
+    i=0
+    while [ "$i" -lt 50 ] && fm_pid_alive "$stop_pid"; do
+      sleep 0.1
+      i=$((i + 1))
+    done
+    if fm_pid_alive "$stop_pid"; then
+      echo "daemon: FAILED - pid $stop_pid did not exit within 5s of SIGTERM"
+      exit 1
+    fi
+    rm -f "$READYFILE" 2>/dev/null || true
+    echo "daemon: stopped pid=$stop_pid"
+  else
+    rm -f "$READYFILE" 2>/dev/null || true
+    echo "daemon: not running"
+  fi
+  exit 0
+fi
+
+# If a genuinely live daemon already holds the lock, do not start a second one -
+# the singleton would refuse anyway. Report it honestly and return success.
+if healthy_daemon; then
+  echo "daemon: healthy pid=$HEALTHY_PID"
+  exit 0
+fi
+
+# Start the daemon as a tracked child and confirm it before settling in. The
+# child stays our child for its whole life: we wait on it, so its eventual exit
+# (crash, --stop, captain-return shutdown) completes this tracked task and the
+# harness re-notifies firstmate.
+child=
+cleanup_child() {
+  if [ -n "$child" ] && fm_pid_alive "$child"; then
+    kill -TERM "$child" 2>/dev/null || true
+  fi
+}
+trap 'cleanup_child; exit 129' HUP
+trap 'cleanup_child; exit 143' TERM INT
+
+: > "$ERRFILE" 2>/dev/null || true
+"$DAEMON" >>"$ERRFILE" 2>&1 &
+child=$!
+
+# Verify the outcome: poll until this child is the confirmed lock-holding
+# daemon, or until some other daemon legitimately holds the singleton (a
+# startup race), or until the child gives up. Only then print the honest line.
+deadline=$(( $(date +%s) + CONFIRM_TIMEOUT ))
+while :; do
+  if healthy_daemon; then
+    if [ "$HEALTHY_PID" = "$child" ]; then
+      echo "daemon: started pid=$child"
+      wait "$child"
+      rc=$?
+      if [ "$rc" -eq 0 ]; then
+        echo "daemon: stopped (clean shutdown)"
+      else
+        echo "daemon: DIED rc=$rc - see $STATE/.supervise-daemon.log and $ERRFILE; re-arm with bin/fm-afk-arm.sh"
+        failure_evidence
+      fi
+      exit "$rc"
+    fi
+    # Another daemon won the singleton; our child stood down. Report the live one.
+    echo "daemon: healthy pid=$HEALTHY_PID"
+    wait "$child" 2>/dev/null || true
+    exit 0
+  fi
+  if ! fm_pid_alive "$child"; then
+    wait "$child" 2>/dev/null
+    rc=$?
+    trap - HUP TERM INT
+    echo "daemon: FAILED - exited rc=$rc during startup (see $ERRFILE)"
+    failure_evidence
+    exit 1
+  fi
+  [ "$(date +%s)" -ge "$deadline" ] && break
+  sleep 0.2
+done
+
+trap - HUP TERM INT
+echo "daemon: FAILED - no confirmed daemon within ${CONFIRM_TIMEOUT}s (see $ERRFILE)"
+failure_evidence
+cleanup_child
+wait "$child" 2>/dev/null || true
+exit 1