feat(watchdog): launchd beacon backstop for watcher continuity - #1252
Open
hnvivek wants to merge 2 commits into
Open
feat(watchdog): launchd beacon backstop for watcher continuity#1252hnvivek wants to merge 2 commits into
hnvivek wants to merge 2 commits into
Conversation
Add a turn-independent continuity layer that re-arms a silently-dead watcher on a launchd timer, closing the gap where a watcher dies during an idle gap with no Stop hook to re-arm it and state/.last-watcher-beat goes stale for hours. - bin/fm-watchdog-check.sh: checker reusing the Stop auto-arm's gates (primary scope, supervision need, not-AFK, live session lock) plus the shared single-flight owner lock (state/.claude-autoarm.lock), so it never double-arms with the hook and only acts when the beacon is past grace. It foregrounds the real arm wrapper (never shell &) and never writes outcome=rewake, since a launchd arm produces no Claude continuation and that outcome would falsely suppress one. A launchd process tree is not a harness ancestry, so it gates on a live lock holder rather than owned-by-self. - launchd/com.firstmate.watcher-watchdog.plist: per-user agent template (RunAtLoad, 90s StartInterval) the installer stamps per home. - bin/fm-watchdog-install.sh: stamp/load/unload + status; per-home label so homes under one user do not collide; idempotent reload; fails loud off-macOS. - tests/fm-watchdog-check.test.sh: re-arm vs no-op (fresh beacon, each gate, held single-flight) and never-double-arm concurrency, plus the installer and plist-template contract. - docs/watcher-watchdog.md with a cross-reference from watcher-continuity.md; scripts.md, documentation-audiences.json, and the watcher-wake-lock test family wiring.
hnvivek
force-pushed
the
fm/fm-watcher-watchdog
branch
from
July 29, 2026 15:31
bab125c to
2af4a28
Compare
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Build a launchd-backed beacon watchdog for firstmate's watcher continuity. Root cause: the Claude Stop asyncRewake auto-arm only re-arms while Claude is turning, so a watcher that dies silently during an idle gap leaves the liveness beacon stale with nothing to re-arm it (observed ~4h blind). The fix must be independent of Claude's turn cadence and survive firstmate restarts and reboots.
Deliverables: a checker script, a per-user launchd agent template (RunAtLoad + 90s StartInterval), an install/uninstall/status helper, a behavior test, a mechanism+install doc with a cross-reference from the existing watcher-continuity doc, and wiring in the scripts doc, the documentation-audiences inventory, and the test family mapping.
Key deliberate decisions and tradeoffs a reviewer should know:
What Changed
fm-watchdog-check.shreuses the Stop auto-arm's exact gates (primary scope, supervision need, not-AFK, live session lock) and re-arms a watcher when the liveness beacon ages past its grace window, plus a RunAtLoad + 90sStartIntervalLaunchAgent template and a macOS-onlyfm-watchdog-install.sh(install/uninstall/status).docs/watcher-watchdog.mdmechanism+install doc cross-referenced fromdocs/watcher-continuity.md, with entries indocs/scripts.md,docs/documentation-audiences.json, and the test family mapping, plus afm-watchdog-check.test.shbehavior suite.CLAUDE_CONFIG_DIRto crewmates, session-lock resolves ancestry to the outermost Claude pid, andfm-brief.shparses under stock macOS Bash 3.2.Risk Assessment
✅ Low: The one substantive finding (X-mode cadence) from round 1 was fixed correctly by mirroring the Stop auto-arm's x-mode.env sourcing; the watchdog feature and the surrounding fixes are otherwise clean, well-tested, and conform to the stated intent.
Testing
Ran the full behavior suite (12/12 pass) which exercises the real checker against hermetic fixtures covering every gate and the single-flight invariant. Since the suite only greps (never runs) the installer, I additionally drove the real installer install→status→uninstall under a fake launchctl shim and proved the stamped LaunchAgent is valid (
plutil -lint: OK, RunAtLoad=true, StartInterval=90, FM_HOME, real checker path, per-home label). Confirmed exact gate parity with the Stop auto-arm and the absence of both forbidden behaviors (epoch-ledger write, exit 2). All documentation deliverables and cross-references are in place; AGENTS.md is correctly untouched. No issues found.Evidence: Stamped, validated launchd agent produced by the installer
plutil -lint stamped-agent.plist → OK; carries Label, /bin/bash + fm-watchdog-check.sh, FM_HOME, RunAtLoad <true/>, StartInterval <integer>90</integer>, log pathsEvidence: Installer install/status/uninstall transcript (fake launchctl, sandboxed HOME)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
.agents/skills/afk/SKILL.md- branch carries 4 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (83 file(s)) into the PR:Push main to origin, or rebase your branch onto origin/main, before gating.
🔧 Fix applied.
1 warning still open:
.agents/skills/afk/SKILL.md- branch carries 4 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (83 file(s)) into the PR:Push main to origin, or rebase your branch onto origin/main, before gating.
🔧 **Review** - 1 issue found → auto-fixed ✅
bin/fm-watchdog-check.sh:116- The watchdog checker arms a watcher for X-mode relay-poll homes (itsfm_supervision_neededgate admits any home withstate/x-watch.check.sh) but never sourcesconfig/x-mode.envbefore running the arm. The Stop auto-arm sources it ([ -f "$CONFIG/x-mode.env" ] && . "$CONFIG/x-mode.env"in bin/fm-claude-stop-autoarm.sh:131) because it exportsFM_CHECK_INTERVAL=30, which bin/fm-watch.sh:98 consumes (CHECK_INTERVAL=${FM_CHECK_INTERVAL:-300}). The arm itself does not source it. Result: when the watchdog re-arms an X-mode home during an idle gap, the forked watcher runs at the 300s default cadence instead of X-mode's 30s — a 10x relay-poll slowdown for that mode, and an inconsistency with the intent's stated 'foregrounds the real arm wrapper ... exactly like the hook.' Fix: resolveCONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}"and add[ -f "$CONFIG/x-mode.env" ] && . "$CONFIG/x-mode.env"immediately before invoking$WATCH_ARM, mirroring the hook.🔧 Fix: source x-mode.env in watchdog checker before arming
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-watchdog-check.test.sh(12 behavior tests: re-arm when beacon absent/old-mtime, no-op when fresh/idle/AFK/no-live-session-lock/child-worktree/single-flight-held, second-tick-doesn't-double-arm, installer+plist contract, checker never exits 2 / foreground arm)Manual end-to-end installer drive with a fakelaunchctlshim against a fixture primary home:fm-watchdog-install.sh install|status|uninstall; captured the stamped LaunchAgent and validated it withplutil -lint(OK)Gate-parity check: compared checker gate calls against bin/fm-claude-stop-autoarm.sh (same scope/need/AFK/single-flight-owner-lockstate/.claude-autoarm.lock, same foregroundfm-watch-arm.sh, samex-mode.envsourcing)Forbidden-behavior grep on the checker: no epoch-ledger write, noexit 2Doc/cross-reference verification: docs/watcher-watchdog.md present, cross-ref from watcher-continuity.md, wiring in docs/scripts.md, entry in docs/documentation-audiences.json, family mapping in bin/fm-test-run.sh, AGENTS.md intentionally untouched✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.