fix(lock): bound stale lock recovery with a non-recursive steal mutex

[withally]

Intent

Fix the unbounded recursive stale-lock acquisition defect in bin/fm-wake-lib.sh that caused the 2026-08-24 Mac freeze. Replace the recursive primary-lock-to-.steal call with a dedicated non-recursive steal-mutex acquisition that uses one atomic owner-directory/symlink claim and never calls fm_lock_try_acquire; return typed nonzero results distinguishing contention from invalid parent or owner-record/create failure and never attempt stale recovery for the latter; allow at most one primary-to-steal transition and reject stale recovery for paths already ending in .steal; preserve the identity recheck after mutex acquisition and all existing lock safety, including stale primary-lock reclamation. Harden Grok live-E2E cleanup by recording the coordinator PID, signaling coordinator, arm, and watcher only through the existing safe-lab ownership check, waiting for their exit, and deleting the lab only afterward. Add colocated behavioral regressions proving prompt descendant-free missing-parent failure, prompt owner-record failure, stale primary reclaim, no .steal.steal recursion for stale or malformed steal mutexes, no Grok coordinator/arm/watcher after lab deletion, and a small repeated-failure process-launch budget. Keep the change surgical with no unrelated lock refactor and prove existing lock tests stay green. The known pre-existing remote-secondmate lifecycle failures named failing seed exited before remote provisioning and herdr-preflight-missing-adapter are out of scope and may be ignored.

What Changed

• bin/fm-wake-lib.sh: stale-lock recovery no longer re-enters fm_lock_try_acquire for the .steal mutex. A dedicated fm_lock_try_acquire_steal_mutex claims the mutex through one atomic owner-directory/symlink create, refuses any path already ending in .steal, serializes reclaim through a pid-recording reclaim marker, and retires legacy directory-shaped mutexes and pre-upgrade .steal.steal residue — so at most one primary-to-.steal transition exists and no nested mutex is ever created. Lock creation now returns typed statuses (1 = contended and retryable, 2 = missing/unwritable parent or owner-record failure); status 2 propagates out of fm_lock_try_acquire and fm_lock_acquire_wait instead of spinning, and stale recovery is never attempted for it. Self-abandoned holds, the post-mutex identity recheck, and stale primary-lock reclamation are preserved.
• Callers updated for the typed result: fm-startup-network.sh, fm-x-lib.sh, fm-backlog-handoff.sh (with explicit lock-failure errors and lock release), fm-afk-return.sh, fm-captain-hold.sh, fm-lock.sh, fm-spawn.sh, fm-wake-drain.sh, fm-watch.sh now check fm_lock_acquire_wait instead of assuming the lock is held; fm-remote-secondmate-control.sh accepts reclaim/reclaim.dead.<pid> markers as valid owner-directory residue.
• Tests: tests/fm-watcher-lock.test.sh adds regressions for typed missing-parent failure (prompt, descendant-free, bounded helper launches), owner-record creation failure, stale primary reclaim, no .steal.steal for stale/malformed/legacy steal mutexes, abandoned and contested reclaim markers, and foreign steal targets. tests/fm-grok-continuity-live-e2e.test.sh records the coordinator PID, signals coordinator/arm/watcher only through the existing safe-lab ownership check, waits for their exit, and fails cleanup if the lab or any process survives; docs/watcher-continuity.md, tests/fm-tool-update-check.test.sh, and tests/fm-remote-secondmate-lifecycle-e2e.test.sh updated accordingly.

Risk Assessment

⚠️ Medium: The change rewrites the core stale-lock recovery protocol — a concurrency-critical, hard-to-test surface that took eight pipeline fix rounds to settle — but every intent criterion is satisfied in the current source, I could not substantiate a remaining reachable double-holder or unbounded-wait path, and the added regressions genuinely exercise the repaired failure modes through runtime side effects rather than source text.

Testing

Baseline understanding came from the 7e4fada..1ebeaf4 diff; I then built a standalone end-to-end repro that drives the real fm_lock_try_acquire on a lock with a missing parent directory and measures outcome, external process launches, and .steal nesting depth. Against the base commit it hangs until killed at 20s with 2486 process launches and a 621-level .steal chain (the runaway freeze); against the target it returns typed status 2 immediately with 4 launches and no .steal transition. I ran the credentialed Grok continuity live E2E for real (grok 1.0.5 through tmux), observing 8 lab-owned processes at peak and, after the hardened cleanup, zero surviving coordinator/watcher/arm processes and a fully deleted lab. The targeted lock suites (fm-watcher-lock including all 15 new colocated regressions, fm-wake-queue, fm-session-lock-ancestry, fm-claude-stop-autoarm) all pass, and the worktree is clean with no transient artifacts left behind. This change is shell/CLI-level lock plumbing with no rendered UI surface, so no screenshot or visual artifact applies; the CLI transcripts are the end-user-visible evidence.

Evidence: Freeze repro: before vs after (CLI transcript)

Source: Freeze repro: before vs after (CLI transcript)

### BEFORE FIX - base commit 7e4fada:bin/fm-wake-lib.sh lock path : $WORK/no-such-parent/.wake-queue.lock parent dir exists: no outcome : HUNG - killed after 20s (never returned) process launches : 2486 deepest .steal chain reached: 621 sample runaway path (truncated to 160 chars): /no-such-parent/.wake-queue.lock.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.s ...(continues) expected: unbounded primary->.steal->.steal.steal... recursion (Mac freeze) ### AFTER FIX - target commit 1ebeaf4 bin/fm-wake-lib.sh lock path : $WORK/no-such-parent/.wake-queue.lock parent dir exists: no outcome : returned status 2 process launches : 4 deepest .steal chain reached: 0 status 2 = typed invalid parent / owner-record failure (status 1 would mean contention)

### BEFORE FIX - base commit 7e4fada:bin/fm-wake-lib.sh
lock path        : $WORK/no-such-parent/.wake-queue.lock
parent dir exists: no
outcome          : HUNG - killed after 20s (never returned)
process launches : 2486
deepest .steal chain reached: 621
sample runaway path (truncated to 160 chars):
/no-such-parent/.wake-queue.lock.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.steal.s
  ...(continues)
expected: unbounded primary->.steal->.steal.steal... recursion (Mac freeze)

### AFTER FIX - target commit 1ebeaf4 bin/fm-wake-lib.sh
lock path        : $WORK/no-such-parent/.wake-queue.lock
parent dir exists: no
outcome          : returned status 2
process launches : 4
deepest .steal chain reached: 0
status 2 = typed invalid parent / owner-record failure (status 1 would mean contention)

Evidence: Grok live E2E: lab-owned processes during run and after cleanup

Source: Grok live E2E: lab-owned processes during run and after cleanup

peak lab-owned processes observed while running: 8 --- snapshot of lab-owned processes at peak (paths shortened) --- 79503 tmux -L fm-grok-live-e2e-79486 new-session -d -s grok-live-e2e -c <LAB>/project env FM_HOME='<LAB>/fmhome' ... 83886 bash <LAB>/project/bin/fm-watch.sh 85706 bash <LAB>/project/bin/fm-watch.sh 85709 bash <LAB>/project/bin/fm-inactive-reconcile.sh scan 79504 bash -lc printf "pid=%s\n" "$$" > "<LAB>/fmhome/state/.grok-watch-coordinator"; ... grok --trust --alwa test exit status: 0 test output: ok - grok 1.0.5 (5115b46bc909) live E2E preserved tracked background completion and shared ledger classification --- after cleanup --- lab directory still present: no surviving lab-owned coordinator/watcher/arm processes: 0 RESULT: no Grok coordinator, arm, or watcher survived lab deletion

lab directory created: /Users/ivan/.no-mistakes/worktrees/37852af5566c/01M0SF08ZQ9J6J68AHH9N3X9N6/.grok-live-e2e.79486

peak lab-owned processes observed while running: 8
--- snapshot of lab-owned processes at peak (paths shortened) ---
79503 /opt/homebrew/bin/tmux -L fm-grok-live-e2e-79486 new-session -d -s grok-live-e2e -c <LAB>/project env FM_HOME='<LAB>/fmhome' FM_ROOT_OVERRIDE='<
83886 bash <LAB>/project/bin/fm-watch.sh
85706 bash <LAB>/project/bin/fm-watch.sh
85709 bash <LAB>/project/bin/fm-inactive-reconcile.sh scan
86499 timeout -k 1 11 bash -c \012    status_file=$1\012    shift\012    "$@"\012    command_rc=$?\012    printf "%s\n" "$command_rc" > "$status_file"
86506 bash -c \012    status_file=$1\012    shift\012    "$@"\012    command_rc=$?\012    printf "%s\n" "$command_rc" > "$status_file"\012    exit "$c
86510 bash <LAB>/project/bin/fm-inactive-reconcile.sh _scan-locked 0
79504 bash -lc printf "pid=%s\n" "$$" > "<LAB>/fmhome/state/.grok-watch-coordinator"; printf "%s\n" "$$" > "$FM_HOME/state/.lock"; grok --trust --alwa

test exit status: 0
test output: ok - grok 1.0.5 (5115b46bc909) live E2E preserved tracked background completion and shared ledger classification

--- after cleanup ---
lab directory still present: no
surviving lab-owned coordinator/watcher/arm processes: 0
RESULT: no Grok coordinator, arm, or watcher survived lab deletion

Evidence: Reusable freeze repro harness

Source: Reusable freeze repro harness

#!/usr/bin/env bash
# E2E repro of the 2026-08-24 Mac-freeze defect.
#
# Precondition: a lock whose parent directory does not exist (the state dir was
# never created / was removed under a running watcher). Before the fix,
# fm_lock_try_acquire could not create the primary lock, so it recursed into
# "<lock>.steal", then "<lock>.steal.steal", ... without bound, forking helper
# processes at every level. That is the runaway that froze the Mac.
#
# usage: repro-recursion.sh <path-to-fm-wake-lib.sh>
set -u
LIB=$1
WORK=$(mktemp -d "${TMPDIR:-/tmp}/fm-freeze-repro.XXXXXX")
SHIM="$WORK/shim"; mkdir -p "$SHIM"
COUNT="$WORK/launches"; : > "$COUNT"
# Count every external process the lock path launches, so a runaway is visible
# as a process-launch budget blowout and not only as wall-clock hang.
for tool in mktemp basename dirname cat date mkdir rmdir ln rm mv stat; do
  real=$(command -v "$tool")
  cat > "$SHIM/$tool" <<EOF
#!/bin/sh
echo "$tool \$*" >> "$COUNT"
exec "$real" "\$@"
EOF
  chmod +x "$SHIM/$tool"
done

LOCK="$WORK/no-such-parent/.wake-queue.lock"
ERR="$WORK/child.err"

echo "lock path        : \$WORK/no-such-parent/.wake-queue.lock"
echo "parent dir exists: $([ -d "$(dirname "$LOCK")" ] && echo yes || echo no)"

PATH="$SHIM:$PATH" perl -e '
  $pid = fork();
  if ($pid == 0) { exec(@ARGV) or exit 127 }
  $SIG{ALRM} = sub { kill "KILL", $pid; waitpid($pid,0);
                     print "outcome          : HUNG - killed after 20s (never returned)\n"; exit 99 };
  alarm 20;
  waitpid($pid, 0);
  printf("outcome          : returned status %d\n", $? >> 8);
  exit $? >> 8;
' bash -c '
  . "$1"
  rc=0
  fm_lock_try_acquire "$2" || rc=$?
  exit "$rc"
' _ "$LIB" "$LOCK" 2> "$ERR"
rc=$?

echo "process launches : $(wc -l < "$COUNT" | tr -d ' ')"
deep=$(sed "s|$WORK||g" "$COUNT" "$ERR" 2>/dev/null | awk '{n=gsub(/\.steal/,"x"); if(n>m){m=n}} END{print m+0}')
echo "deepest .steal chain reached: $deep"
if [ "$deep" -gt 0 ]; then
  echo "sample runaway path (truncated to 160 chars):"
  sed "s|$WORK||g" "$ERR" | grep -o '[^ ]*\.steal\(\.steal\)*' | sort -u | tail -1 | cut -c1-160
  echo "  ...(continues)"
fi
rm -rf "$WORK"
exit $rc

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos

• 🚨 bin/fm-wake-lib.sh:430 - fm_lock_try_create decides rc=1 (contention) vs rc=2 (fatal) by re-probing the filesystem after the failure, so a lost race that resolves in the window is misclassified as fatal. Concrete sequence: A and B both see $lockdir absent; A wins ln -s; B's ln -s fails EEXIST, B runs fm_lock_remove_stray_owner_link (forks basename, ~ms); A finishes its short critical section and releases; B's post-check at 430-432 then sees neither $lockdir nor $lockdir.steal and returns 2. Same for the fm_lock_claim_blocked_by_steal branch, where the steal holder releases during fm_lock_discard_owner. rc=2 propagates through fm_lock_try_acquire (833-835) into fm_lock_acquire_wait, which now returns 2 instead of retrying. Callers that never check the result — fm_wake_append (bin/fm-wake-lib.sh:1199), bin/fm-captain-hold.sh:778, bin/fm-backlog-handoff.sh:447/454/488/492, bin/fm-x-lib.sh:942/969/988 — then execute their critical sections without holding the lock (the later fm_lock_release is pid-guarded and silently no-ops), so ordinary contention can silently lose mutual exclusion on the wake queue and registries. Fix by returning 2 only from the branches that proved a fatal cause (invalid parent from fm_lock_owner_dir, fm_lock_prepare_owner failure) and defaulting every post-ln/post-claim failure to 1, rather than inferring the type from a later filesystem probe.
• 🚨 bin/fm-wake-lib.sh:1168 - fm_autoarm_release_abandoned still calls fm_lock_try_acquire "$steal" on a .steal path, which the new guard at bin/fm-wake-lib.sh:869-873 now short-circuits to return 1 for any dead-owner steal mutex. Concrete sequence: a process is SIGKILLed while holding $STATE/.claude-autoarm.lock.steal; the dangling symlink with a dead pid survives. Every later call now returns 1 forever (no other code path reclaims that mutex: fm_lock_release requires a pid match, and only the new fm_lock_try_acquire_steal_mutex reclaims stale steals). Before this change the recursive path reclaimed it via .steal.steal. Result: bin/fm-claude-stop-autoarm.sh:149 exits 0 and bin/fm-turnend-guard.sh:321 returns 1 permanently, so an abandoned auto-arm claim can never be released again — a loss of the existing stale-reclaim safety the intent requires be preserved. Fix: call fm_lock_try_acquire_steal_mutex "$steal" here, which is exactly the non-recursive reclaim path the change introduced.
• 🚨 tests/fm-grok-continuity-live-e2e.test.sh:74 - The intent requires "recording the coordinator PID, signaling coordinator, arm, and watcher only through the existing safe-lab ownership check, waiting for their exit". Nothing in the repository ever writes $HOME_DIR/state/.grok-watch-coordinator (grep over bin/ and tests/ finds only this reader), and the only coordinator-ish pid the test records is the tmux shell's bare $$ written to $FM_HOME/state/.lock at line 108 — which the sed -n 's/^pid=//p' pattern would not match either. So coordinator_pid is always empty, both loops skip it via [ -n "$pid" ] / wait_lab_pid_exit's early return 0, and the coordinator is never signaled, never waited for, and never asserted retired before rm -rf "$LAB". The required regression is vacuous as written; the coordinator PID needs to be genuinely recorded (or read from the file that actually holds it, with a matching parse).
• ⚠️ bin/fm-wake-lib.sh:459 - The reclaim serialization marker mkdir "$ownerdir/reclaim" has no liveness or age fallback, unlike every other stale check in this file (fm_pid_alive / fm_lock_mid_acquire_is_fresh / FM_LOCK_STALE_AFTER). If a process is killed between line 459 and the rm -f "$steal" at 467, $ownerdir/reclaim is left behind permanently; every subsequent reclaimer's mkdir fails with EEXIST and returns 1 forever. Since this helper is the only code that can reclaim a stale steal mutex, the primary lock's stale recovery dies with it: fm_lock_try_acquire returns 1 forever and fm_lock_acquire_wait spins indefinitely — the same unbounded-wait class of failure this change is meant to eliminate. Record the reclaimer's pid inside the marker and treat a dead-pid marker older than FM_LOCK_STALE_AFTER as reclaimable.
• ℹ️ tests/fm-watcher-lock.test.sh:349 - The malformed variant does not exercise the malformed-owner branch it names. The dangling $steal symlink is created moments before the assertion, so pid reads empty and fm_lock_mid_acquire_is_fresh (bin/fm-wake-lib.sh:455) returns true because fm_path_age lstats the fresh symlink (age 0 < the clamped 2s floor) — the function returns 1 there and never reaches the [ -L "$steal" ] / fm_lock_link_owner handling under test. Backdate the symlink (touch -h -t) or raise its apparent age so the malformed path is actually reached; otherwise the test would still pass if that branch regressed.

🔧 Fix: fix lock status typing, steal reclaim, and E2E cleanup
4 issues (1 error, 3 warnings) still open:

• 🚨 bin/fm-wake-lib.sh:448 - fm_lock_try_acquire_steal_mutex treats any existing $lock.steal.steal as permanent contention (line 448, repeated at 463), but after this change nothing in the repo ever creates or removes .steal.steal (grep -rn 'steal\.steal' bin/ matches only these two guards and the comment at 437). The old recursive code did create it, and a SIGKILL while it held the nested mutex — exactly the recursion that caused the 2026-08-24 freeze — leaves a dangling $STATE/.watch.lock.steal.steal symlink behind. Concrete sequence after upgrade: $STATE/.watch.lock holds a dead pid; fm_lock_try_acquire falls through to fm_lock_try_acquire_steal_mutex "$lock.steal"; try_create fails (steal exists with dead owner); line 448 sees the leftover .steal.steal and returns 1; fm_lock_try_acquire returns 1; fm_lock_acquire_wait spins forever. No code path removes the residue, so the primary lock's stale reclamation is permanently dead — the same unbounded-wait failure this change exists to eliminate. Unconditional removal is wrong during a mixed-version window (an old process may legitimately hold it), so apply the file's standard staleness test to it instead: read its owner pid, and when the pid is dead and fm_path_age exceeds FM_LOCK_STALE_AFTER, remove the residue and continue rather than returning 1.
• ⚠️ bin/fm-wake-lib.sh:486 - The abandoned-marker takeover in fm_lock_reclaim_marker_claim is not ownership-checked, so two reclaimers can both end up holding the steal mutex. fm_lock_reclaim_marker_release (504) unconditionally rm -f "$reclaim/pid" then rmdir "$reclaim" — it never verifies the recorded pid is the caller's. Concrete sequence: reclaimer A is killed after mkdir $ownerdir/reclaim, leaving a dead-pid marker older than FM_LOCK_STALE_AFTER. B and C both call fm_lock_reclaim_marker_claim; both mkdir fail; both see fm_lock_reclaim_marker_is_abandoned true. B releases, mkdirs, and writes its pid. C then runs its own fm_lock_reclaim_marker_release, which deletes B's live pid file and rmdirs B's marker, and C's mkdir then succeeds. B and C now both believe they hold the reclaim marker, both pass fm_lock_recheck_stale_owner, and both rm -f "$steal" + fm_lock_try_create "$steal" (454-472) — so C can delete the steal symlink B just created and both proceed into the primary-lock stale-recovery section, where fm_lock_remove_path "$lockdir" + fm_lock_try_create (960-963) lets each clobber the other's primary lock. Fix: make the takeover atomic — rename the abandoned marker to a unique name (mv "$reclaim" "$reclaim.dead.$mypid", which only one racer can win) and remove it, then mkdir; and make fm_lock_reclaim_marker_release a no-op unless $reclaim/pid equals ${BASHPID:-$$}.
• ⚠️ bin/fm-wake-lib.sh:454 - The dangling-owner branch of fm_lock_try_acquire_steal_mutex (454-459) reclaims $steal with a bare check-then-rm, with no serialization at all — unlike the -d $ownerdir branch below it, which takes the reclaim marker. Concrete sequence: $lock.steal is a stale dangling symlink to a removed owner dir (the malformed fixture the new test builds at tests/fm-watcher-lock.test.sh:349). A and B both reach line 455 and both pass fm_lock_points_to_owner. A executes rm -f "$steal" and fm_lock_try_create "$steal", acquiring the mutex with ownerA. B, still inside its window, executes rm -f "$steal" — deleting A's freshly created live link — and its own fm_lock_try_create succeeds with ownerB. Both hold the steal mutex. A's identity recheck at line 936 (fm_lock_points_to_owner "$steal" "$steal_owner") catches this only if A has not already passed it; A can pass the recheck and then be overtaken during fm_lock_recheck_stale_owner / _fm_recovery_marker_publish, after which both reach fm_lock_remove_path "$lockdir" + fm_lock_try_create and mutual exclusion on the primary lock is lost. The prior recursive implementation serialized this same rm behind .steal.steal. Fix: route this case through the same reclaim-marker serialization — mkdir "$ownerdir" (atomic; the loser sees EEXIST) and then fall into the shared fm_lock_reclaim_marker_claim path, instead of keeping a separate unserialized branch.
• ⚠️ bin/fm-wake-lib.sh:976 - fm_lock_acquire_wait can now return non-zero (status 2) where it previously could only block until it acquired, but several call sites still invoke it as a bare statement and continue into their critical section unlocked. Concrete sequence: mktemp -d in fm_lock_owner_dir fails (state dir unwritable / ENOSPC) → fm_lock_try_create returns 2 → fm_lock_try_acquire returns 2 → fm_lock_acquire_wait returns 2. In bin/fm-afk-return.sh:231 (which runs under set -u only, not set -e) the result is discarded, so write_pending_seed and return_reconcile run without the lock and the later fm_lock_release "$LOCK" silently no-ops because it is pid-guarded. Same shape in fm_wake_append (bin/fm-wake-lib.sh:1219), which then mutates the shared wake queue unlocked. The set -eu callers (fm-captain-hold.sh:778, fm-backlog-handoff.sh:447/454/488/492, fm-pr-check.sh:100) abort loudly and are fine. Fix: check the status at the unguarded call sites and fail the operation rather than proceeding lock-free.

🔧 Fix: serialize steal reclaim, retire nested-steal debris, fail closed
2 issues (1 warning, 1 info) still open:

• ⚠️ bin/fm-wake-lib.sh:495 - fm_lock_reclaim_marker_claim still allows two reclaimers to hold the steal mutex simultaneously: the mv takeover never verifies that the directory it retires is still the abandoned marker the caller inspected. Concrete sequence, with $STATE/.watch.lock holding a dead pid and $STATE/.watch.lock.steal a stale symlink whose ownerdir contains a reclaim marker left by a SIGKILLed reclaimer (dead pid, age > FM_LOCK_STALE_AFTER): (1) B and C both fail mkdir "$reclaim" (491) and both pass fm_lock_reclaim_marker_is_abandoned (492) against the same old marker; (2) B mv-retires it, re-mkdirs, writes pid B, returns 0; (3) C then mv-retires B's live marker (495), re-mkdirs, writes pid C, returns 0. C never re-checks abandonment after B replaced the marker, and B never re-verifies ownership afterward, so both proceed past line 468. Both pass fm_lock_recheck_stale_owner (the steal link is still the stale one), both run rm -f "$steal" (474) and fm_lock_try_create "$steal" (481) — C's rm -f can delete the live steal symlink B just created — so B and C both return 0 with FM_LOCK_OWNER_DIR set. Back in fm_lock_try_acquire, the identity recheck at line 966 only catches B if C overtakes it before that line; if C overtakes during _fm_recovery_marker_publish (line 986, real file I/O for $STATE/.watch.lock), B is already past both the recheck at 966 and fm_lock_recheck_stale_owner at 979, so B's fm_lock_remove_path "$lockdir" (991) deletes the primary lock C legitimately created one step earlier, and C keeps running its critical section with no lock present — a later process can acquire .watch.lock concurrently. Note fm_lock_reclaim_marker_release (517) is correctly pid-guarded now, so B's release silently no-ops and leaves C's marker in place; the guard does not prevent the double hold. Fix: make the takeover verify what it retires — capture the abandoned marker's pid at the is_abandoned check, and after mv confirm $retired/pid still holds that same dead value (restore/abort otherwise); additionally re-verify $reclaim/pid equals ${BASHPID:-$$} immediately before the destructive rm -f "$steal" at line 474.
• ℹ️ bin/fm-wake-lib.sh:452 - fm_lock_try_acquire_steal_mutex narrows reclaimable stale steal mutexes to symlink-shaped ones: [ -L "$steal" ] || return 1 (452) means a .steal that is a plain directory with a dead pid can never be reclaimed, so fm_lock_try_acquire returns 1 forever for its primary lock and fm_lock_acquire_wait spins indefinitely. The pre-change recursive path handled this via fm_lock_remove_path, which still carries an explicit non-symlink branch (rmdir), and fm_lock_recheck_stale_owner (573) likewise still accepts a [ -d ] && [ ! -L ] lock — so the directory shape is otherwise treated as supported legacy state. Reachability is weak in practice: symlink locks landed in 0b11b10 (2026-06-24), so a directory-shaped .steal only survives in a state dir untouched since before then, and nothing in the current tree creates one. Flagging it because the change explicitly reasons about mixed-version residue for .steal.steal but not for this shape; if you want symmetry, route a dead-pid directory .steal through fm_lock_remove_path under the same staleness test.

🔧 Fix: bind reclaim takeover to inspected marker, reclaim legacy steal dirs
2 issues (1 error, 1 warning) still open:

• 🚨 tests/fm-grok-continuity-live-e2e.test.sh:125 - The new coordinator gate fails on every run, so the required Grok cleanup regression never executes. The recorded coordinator PID is the bash -lc '...' process spawned at line 111; tmux runs the pane string via /bin/sh -c, which exec-chains through env into bash, replacing argv. That bash's argv is bash -lc <script text>, and the script text contains the literal $FM_HOME (written as \"\$FM_HOME/state/.grok-watch-coordinator\", escaped so the test shell does not expand it) — the $LAB path only ever appears in the env FM_HOME='...' prefix that exec discards. I reproduced this locally: sh -c "env FM_HOME='/tmp/labcheck.N/fmhome' bash -lc 'printf ... > \"$FM_HOME/...\"; sleep 4'" yields ps -p <pid> -o command= == bash -lc printf "pid=%s\n" "$$" > "$FM_HOME/../pidfile" 2>/dev/null; sleep 4, with no lab path anywhere, and ps -eo command | grep labcheck matched nothing. So lab_pid_is_safe "$coordinator_pid" returns 1 and line 125 fails immediately after wait_for_text "Grok Build", aborting the test before the arm/watcher continuity path it exists to prove. Even with that assertion removed, cleanup (line 74) would still skip signaling and wait_lab_pid_exit would return success vacuously — the exact emptiness the earlier round was asked to eliminate. Fix: interpolate the lab path into the pane script at build time (write to "$HOME_DIR/state/.grok-watch-coordinator" with $HOME_DIR expanded, rather than the escaped \$FM_HOME), so the coordinator's own argv contains $LAB and the ownership check passes for the real process.
• ⚠️ bin/fm-wake-lib.sh:510 - fm_lock_reclaim_marker_claim still leaves the marker slot observably empty during a takeover, so a live holder's marker can be destroyed by a bystander and two reclaimers can hold the steal mutex. Concrete sequence with a stale $lock.steal symlink whose ownerdir carries a dead-pid reclaim marker older than FM_LOCK_STALE_AFTER: (1) B and C both fail mkdir "$reclaim" (507) and both observe the same abandoned marker via fm_lock_reclaim_marker_is_abandoned (508); (2) C then forks rm -rf "$retired" (511) — a real exec, milliseconds — during which B completes mv/validate/rm/mkdir and writes its own live pid, returning 0; (3) C's mv "$reclaim" "$retired" (512) now moves B's live marker away, leaving the slot absent while C forks cat "$retired/pid" (513); (4) a third reclaimer D mkdir "$reclaim" succeeds in that gap and writes pid D; (5) C's restore branch (515) sees [ -e "$reclaim" ] true and rm -rf "$retired" — B's marker is gone. If B has already passed fm_lock_reclaim_marker_held (468) it proceeds to rm -f "$steal" (474) and fm_lock_try_create "$steal" (481) while D independently passes held and does the same, so both return 0 with FM_LOCK_OWNER_DIR set. Back in fm_lock_try_acquire, whichever loses the identity recheck at 966 can still be overtaken during _fm_recovery_marker_publish (986) and then fm_lock_remove_path "$lockdir" (991) deletes the primary lock the other legitimately created, leaving its critical section running with no lock present. Fix: never vacate the slot for validation — after mv, immediately mkdir "$reclaim" and write the caller's pid (losing that mkdir simply means return 1), and only then validate $retired/pid against the captured abandoned pid, releasing the caller's own claim on mismatch instead of restoring into a slot another racer may have taken.

🔧 Fix: fix coordinator argv ownership and marker takeover gap
3 issues (2 warnings, 1 info) still open:

• ⚠️ bin/fm-backlog-handoff.sh:447 - with_remote_route_locks still calls fm_lock_acquire_wait bare, and set -e does not protect it because the function's only call site (bin/fm-backlog-handoff.sh:477) is with_remote_route_locks "$id" resume_remote_outbox "$id" "$outbox" || failed=1 - a conditional context, which disables errexit for the entire function body. Concrete sequence: mktemp -d inside fm_lock_owner_dir fails for $STATE/.secondmate-registry.lock.owner.XXXXXX (state dir read-only, ENOSPC, or fd exhaustion) -> fm_lock_try_create returns 2 -> fm_lock_try_acquire returns 2 -> fm_lock_acquire_wait returns 2 at line 447. Before this change that call could only block until it acquired, so the status was irrelevant; now execution falls straight through to the registry read at 448, then line 454 repeats the same for $ACTIVE_HANDOFF_LOCK, and "$operation" "$@" (remote_deliver_outbox) mutates the secondmate registry and the per-id handoff outbox with neither lock held. release_remote_locks -> fm_lock_release is pid-guarded and silently returns 0, so the failure is invisible and rc reports success. This is the same defect class already accepted and fixed at bin/fm-afk-return.sh:231 and bin/fm-wake-lib.sh:1307; guard both calls here (fm_lock_acquire_wait "$ACTIVE_REGISTRY_LOCK" || { release_remote_locks; return 1; }) so the operation fails instead of running lock-free.
• ⚠️ bin/fm-spawn.sh:2823 - Same unguarded-status shape in spawn_record_traceparent. Its only call site is if ! spawn_record_traceparent; then (bin/fm-spawn.sh:2848), so set -eu (line 197) is suppressed inside the body. When fm_lock_acquire_wait "$SPAWN_META_LOCK" returns the new status 2 (owner-record/create failure, e.g. mktemp -d failing under ENOSPC on $STATE), the function continues to SPAWN_META_LOCK_HELD=1 and rewrites $STATE/$ID.meta via awk + mv -f with the meta lock not held, racing concurrent holders in bin/fm-teardown.sh:221 and bin/fm-captain-hold.sh:778 and losing whichever write lands first. fm_lock_release "$SPAWN_META_LOCK" then returns 0 (pid mismatch path), so status stays 0 and the caller is told the traceparent was recorded successfully. Guard the acquire and return 1 on failure.
• ℹ️ bin/fm-wake-lib.sh:486 - In the legacy directory-shaped branch, fm_lock_remove_path "$steal" || return 1 reduces to clean_known_files + rmdir, so any entry in the directory that fm_lock_clean_known_files does not know about makes the rmdir fail permanently and wedges stale recovery for the primary lock (fm_lock_acquire_wait then spins forever). Two such entries are reachable: a reclaim.dead.<pid> directory left behind if a reclaimer dies between the mv (line 511) and the following mkdir in fm_lock_reclaim_marker_claim, and a stray $steal/<basename>.owner.XXXXXX symlink from a pre-upgrade racer whose ln -s landed inside the directory (the case fm_lock_remove_stray_owner_link exists to clean, and which only its creator cleans). The symlink branch is unaffected because its rmdir "$ownerdir" is || true after $steal is already unlinked. Reachability is compound-rare (requires the pre-2026-06-24 directory shape plus a crash or an abandoned stray), so this is informational; if you want symmetry, tolerate unknown residue in the directory branch (e.g. move the directory aside to a unique name before removing it) rather than failing the whole reclaim.

🔧 Fix: guard conditional-context lock waits, retire legacy steal debris
4 issues (3 warnings, 1 info) still open:

• ⚠️ bin/fm-wake-lib.sh:488 - The legacy directory-shaped branch of fm_lock_try_acquire_steal_mutex releases its serialization marker before the destructive removal, and fm_lock_remove_path is shape-agnostic, so it can delete another reclaimer's freshly created live steal mutex and leave two holders. Concrete sequence with $STATE/.watch.lock holding a dead pid and $STATE/.watch.lock.steal a legacy directory whose pid is dead and stale: (1) A claims $steal/reclaim (line 468), passes fm_lock_recheck_stale_owner and fm_lock_reclaim_marker_held, then runs fm_lock_reclaim_marker_release at line 486, which rmdirs $steal/reclaim and leaves the slot empty; (2) A is descheduled (line 487 fm_lock_clean_known_debris and line 488 fm_lock_remove_path both fork rm/rmdir); (3) B enters the same branch, mkdir "$steal/reclaim" now succeeds, B passes recheck+held, releases, cleans, fm_lock_remove_path "$steal" rmdirs the now-empty directory, and B's final fm_lock_try_create "$steal" (line 500) creates a live $steal symlink pointing at B's owner dir — B returns 0 holding the mutex; (4) A resumes and calls fm_lock_remove_path "$steal", whose [ -L "$lockdir" ] branch (line 601) unlinks B's live symlink and discards B's owner dir; A's fm_lock_try_create then succeeds too. Both A and B now hold the steal mutex with FM_LOCK_OWNER_DIR set. In fm_lock_try_acquire, B's identity recheck (fm_lock_points_to_owner "$steal" "$steal_owner") only catches B if A clobbers before B reaches it; if A clobbers while B is inside _fm_recovery_marker_publish, B has already passed both the recheck and fm_lock_recheck_stale_owner, so both reach fm_lock_remove_path "$lockdir" + fm_lock_try_create and mutual exclusion on the primary lock is lost. Fix: after the release at line 486, do not use the shape-agnostic fm_lock_remove_path; clean the known entries and finish with a plain rmdir "$steal" (rmdir fails with ENOTDIR on a symlink, so it can never delete a racer's live mutex), and re-verify [ -d "$steal" ] && [ ! -L "$steal" ] before touching it.
• ⚠️ bin/fm-wake-lib.sh:532 - fm_lock_reclaim_marker_claim's "restore a live racer's pid" path can leave the reclaim marker permanently held by a process that has already given up on it, wedging stale steal recovery for the whole state dir until that process dies. Concrete sequence on an abandoned marker (dead pid X, age > FM_LOCK_STALE_AFTER): (1) B and C both fail mkdir (line 507) and both observe the same abandoned marker, abandoned_pid=X; (2) B takes over (mv/mkdir/write pid B) and returns 0; (3) C then mv-retires B's live marker, mkdirs, and writes pid C; (4) B — still executing fm_lock_reclaim_marker_held "$reclaim" at line 483 — reads pid C, so held fails and B does return 1 without releasing anything; (5) C reads retired_pid=B, sees it alive, and at line 532 writes B's pid back into the marker, then returns 1. The marker now records a live pid that its "owner" B has already abandoned. fm_lock_reclaim_marker_is_abandoned requires a dead pid, so no one can ever take it over, and fm_lock_reclaim_marker_release is pid-guarded so no one can release it. B itself is normally spinning in fm_lock_acquire_wait, and every retry re-enters fm_lock_reclaim_marker_claim, fails mkdir, fails is_abandoned (pid B is alive — it is B), and returns 1 forever: B deadlocks against its own orphaned marker, the same unbounded-wait class this change exists to eliminate. Minimal fix consistent with the file's existing self-held reclaim pattern: treat a marker whose recorded pid equals ${BASHPID:-$$} as reclaimable in fm_lock_reclaim_marker_is_abandoned (the caller only reaches that check when it does not believe it holds the marker), so a process can always break out of a marker it orphaned. Note tests/fm-watcher-lock.test.sh:test_reclaim_marker_takeover_is_bound_to_the_marker_it_inspected asserts the restore behavior, so it needs a companion case for the orphaned-live-pid state rather than treating restore alone as sufficient.
• ⚠️ bin/fm-wake-lib.sh:467 - The new reclaim marker is created inside a lock owner directory (reclaim="$ownerdir/reclaim", plus reclaim.dead.<pid> takeover copies), but retired_lock_owner_dir_valid in bin/fm-remote-secondmate-control.sh:99-111 rejects any owner-dir entry that is not a plain file named pid|fm-home|pid-identity|role|watcher-path. Concrete sequence: a reclaimer is SIGKILLed after fm_lock_reclaim_marker_claim while recovering $TARGET_HOME/config/.fm-inherit-<item>.lock.steal, leaving .fm-inherit-<item>.lock.steal.owner.XXXXXX/reclaim/pid behind. That owner-dir name matches the allowed pattern at bin/fm-remote-secondmate-control.sh:149-150, so the residue check descends into it, hits the reclaim directory, fails retired_plain_file, and retired_inheritance_residue_valid returns 1 — the retired secondmate home is judged to hold foreign residue and retirement is refused, even though the leftover is debris this lock code created. Before this change nothing was ever written inside an owner directory, so the validator's allowlist was complete. Fix: teach retired_lock_owner_dir_valid about a reclaim / reclaim.dead.<pid> marker directory whose only entry is a pid file, or place the reclaim marker as a sibling of the owner directory instead of inside it.
• ℹ️ bin/fm-wake-lib.sh:418 - fm_lock_try_create's rc local is vestigial: it is set to 1 at line 418 and never reassigned on any path that reaches return "$rc" at line 431 (both fatal paths return 2 directly, and the success path returns 0 inline). The added local declaration, the assignment, and the indirect return are equivalent to a plain return 1 and only obscure which failures are typed. Collapse to return 1 and drop rc from the local list.

🔧 Fix: harden legacy steal reclaim, marker self-recovery, owner residue
2 issues (1 warning, 1 info) still open:

• ⚠️ bin/fm-wake-lib.sh:489 - The legacy directory-shaped steal reclaim destroys the mutex's own pid record before it can prove the directory is removable, so any failed rmdir leaves a permanently unreclaimable steal mutex. Sequence: $STATE/.watch.lock holds a dead pid and $STATE/.watch.lock.steal is a legacy directory whose pid is dead and stale. A claims $steal/reclaim, passes recheck+held, then runs fm_lock_clean_known_files "$steal" (489) which unlinks $steal/pid, releases the marker (491), and rmdir "$steal" (492) fails — either because the directory holds content this code does not own (the exact fail-closed case tests/fm-watcher-lock.test.sh:test_legacy_directory_steal_mutex_survives_known_recovery_debris builds), or because a racer B re-mkdired $steal/reclaim in the window between the release at 491 and the rmdir at 492 (B then aborts at fm_lock_recheck_stale_owner, since the pid it compares against is now empty). Either way $steal survives as a directory with no pid. Every later reclaimer reaches the directory branch at 467-470, reads pid='', and hits case "$pid" in ''|*[!0-9]*) return 1, so fm_lock_try_acquire_steal_mutex returns 1 forever, fm_lock_try_acquire returns 1 forever, and fm_lock_acquire_wait spins indefinitely on the primary lock — the same unbounded-wait class this change exists to eliminate, and now unrecoverable even after an operator deletes the unowned residue. Reproduced locally against the current worktree: with $steal a directory containing a dead pid plus not-a-lock-record, one fm_lock_try_acquire deletes $steal/pid, and subsequent calls (after the stale window) return 1 permanently; an empty pid-less legacy .steal directory likewise returns 1 on every call. Note the debris test only asserts rc=1 and that the unowned file survives, so it passes with this defect present. Fix: do not vacate the pid record until removal is guaranteed — attempt the emptiness-proving step first (e.g. clean only reclaim.dead.*/stray owner links, then rename the directory aside to a unique name while still marker-held and remove it there), or restore $steal/pid when the rmdir at 492 fails; additionally accept a directory-shaped .steal whose pid record is absent but whose age is past FM_LOCK_STALE_AFTER as reclaimable, so an already-wedged state dir can recover.
• ℹ️ tests/fm-watcher-lock.test.sh:283 - The descendant-free assertion cannot fail. attempt_pid is the bash -c root of the lock attempt, and it is read from $pidfile only after the $(... ) command substitution has already reaped that shell, so pgrep -P "$attempt_pid" queries a pid that no longer exists and always prints nothing — any child the attempt genuinely leaked would have been reparented to init and would still not be listed. The intent's "prompt descendant-free missing-parent failure" requirement is in practice carried entirely by the launch-count budget and the elapsed-time bound, both of which are real. Either capture the descendant set while the attempt shell is still alive (e.g. have the fixture write pgrep -P $$ output before exiting) or drop the check as subsumed by the budget assertion, rather than leaving an assertion that would still pass if the code regressed.

🔧 Fix: preserve steal-dir owner record, prove descendant-free attempts
2 issues (1 error, 1 info) still open:

• 🚨 bin/fm-wake-lib.sh:447 - fm_lock_try_acquire_steal_mutex dropped the self-held reclaim branch that the recursive fm_lock_try_acquire "$steal" call used to provide, so a steal mutex abandoned by the calling process itself is never reclaimable and fm_lock_acquire_wait spins forever — the exact unbounded-wait class this change exists to eliminate. Concrete sequence: process P calls fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK"; the primary lock records a dead, aged pid, so fm_lock_try_acquire (bin/fm-wake-lib.sh:1058) acquires $FM_WAKE_QUEUE_LOCK.steal. A TERM/INT trap fires inside the recovery section (before fm_lock_remove_path "$lockdir" at line 1104, a window that includes _fm_recovery_marker_publish, which itself takes another lock and can block), so the fm_lock_release "$steal" at line 1114 never runs and $steal is left recording P's own live pid while the primary lock still records the dead pid. P's EXIT path re-acquires the same lock (the documented self-abandonment case at bin/fm-wake-lib.sh:1024-1032, regression-tested in tests/fm-wake-queue.test.sh): fm_lock_try_create fails, the recorded primary pid is dead and stale, so it falls through to fm_lock_try_acquire_steal_mutex "$steal", which reads $steal/pid = P's own pid, fm_pid_alive is true, and returns 1 — forever, for as long as P lives. fm_lock_acquire_wait therefore loops indefinitely. Verified against this worktree: with $lock/pid a dead pid and $lock.steal/pid the caller's own pid, fm_lock_try_acquire returns 1 (and fm_lock_try_acquire_steal_mutex returns 1) on the current code, while the same fixture against base commit 7e4fada returns 0. This contradicts the intent's requirement to "preserve the identity recheck after mutex acquisition and all existing lock safety". Fix: before the liveness check, apply the same self-held reclaim the primary path uses — when $pid equals ${BASHPID:-$$} (under the same BASHPID/BASH_SUBSHELL guard used at line 1025), fm_lock_remove_path "$steal" and retry fm_lock_try_create "$steal" — and add a regression that fails before that fix.
• ℹ️ bin/fm-wake-lib.sh:1025 - The added { [ -n "${BASHPID:-}" ] || [ "${BASH_SUBSHELL:-0}" -eq 0 ]; } guard on the self-held reclaim branch is a behavior change to the primary-lock path that the intent does not ask for ("Keep the change surgical with no unrelated lock refactor") and that no added regression covers. It is defensible — under bash 3.2 a subshell inherits $$, so without it a subshell could reclaim the parent's live lock, and the guard makes bash 3 behave like bash 4+ — but it silently converts that case into permanent contention, and nothing in this change exercises it. Noting it so the scope deviation is a deliberate, recorded choice rather than an unnoticed one.

🔧 Fix: restore self-abandoned steal-mutex reclaim in non-recursive helper
2 infos still open:

• ℹ️ bin/fm-wake-lib.sh:608 - fm_lock_reclaim_marker_is_abandoned's self-pid clause ([ -n "$pid" ] && [ "$pid" = "${BASHPID:-$$}" ]) is the only one of the three self-comparison sites in this file that omits the Bash 3 subshell guard the other two now carry (bin/fm-wake-lib.sh:454 and bin/fm-wake-lib.sh:1036). Under bash 3.2 (still /bin/bash on macOS) a subshell inherits $$, so a child frame would classify a live parent's reclaim marker as abandoned and take it over, defeating the serialization that protects the destructive rm -f "$steal" at line 474. I could not construct a reachable path today: the marker is only held across a synchronous stretch of fm_lock_try_acquire_steal_mutex whose only subshells are $(cat ...), none of which re-enter the lock code, and a marker orphaned by the parent is genuinely reclaimable anyway. Flagging the asymmetry so a future caller that acquires a lock inside a command substitution does not silently reintroduce it; adding && { [ -n "${BASHPID:-}" ] || [ "${BASH_SUBSHELL:-0}" -eq 0 ]; } would make all three sites consistent.
• ℹ️ bin/fm-startup-network.sh:441 - cmd_run's new fm_lock_acquire_wait "$STATE/.lock.acquire" || return 1 is the first early return in this function that happens after the status file was published with state=running pid=$$ (line 411) and after $out/$timings were created via mktemp (lines 429/433). On the status-2 path (state dir unwritable / ENOSPC making fm_lock_owner_dir fail) the function returns without calling publish, so $STATUS_FILE is left reading state=running for a pid that has exited and two temp files leak in $TMPDIR. Impact is bounded — cmd_start's worker_alive check and cmd_harvest both reap a dead running-state record, and await_delivery exits promptly because no live claim file exists — and failing closed is strictly better than the prior unbounded block, so this is a residue nit rather than a hang. Publishing a terminal failed state (or rm -f "$out" "$timings") before returning would keep the early exit consistent with every other terminal path.

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-watcher-lock.test.sh (all lock primitives plus the 15 new regressions: missing-parent typed failure with bounded launches, owner-record failure, no .steal.steal for stale/malformed steal mutexes, legacy steal-dir reclaim, self-abandoned steal-mutex reclaim, reclaim-marker takeover safety)
• FM_GROK_LIVE_E2E=1 bash tests/fm-grok-continuity-live-e2e.test.sh (real grok 1.0.5 + tmux, run twice; second run instrumented to snapshot lab-owned processes during the run and after cleanup)
• bash tests/fm-wake-queue.test.sh
• bash tests/fm-session-lock-ancestry.test.sh
• bash tests/fm-claude-stop-autoarm.test.sh
• Manual before/after freeze repro: bash repro-recursion.sh <lib> run against git show 7e4fada:bin/fm-wake-lib.sh and against the target bin/fm-wake-lib.sh, measuring outcome, external process launches, and deepest .steal chain reached under a 20s hard kill

⚠️ **Document** - 1 warning

• ⚠️ bin/fm-wake-lib.sh:254 - Lint regression introduced by this change, outside the documentation phase's edit permission: shellcheck 0.11.0 now reports SC2100 twice on bin/fm-wake-lib.sh (lines 254 and 260, FM_WATCHER_VERDICT_REASON=stale-beacon), so bin/fm-lint.sh exits 1. Verified it is not pre-existing and not caused by my doc edits: the base commit's copy of the file shellchecks clean, while the target commit 1ebeaf4 copy already produces both warnings. The lint phase owner should resolve it (e.g. a # shellcheck disable=SC2100 directive or quoting the assignment).

🔧 **Lint** - 1 issue found → auto-fixed ✅

• ⚠️ linter found issues (exit code 1)

🔧 Fix: quote stale-beacon verdict literal to fix SC2100
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.