feat(quota-watch): pause and resume crew on Claude quota pressure

[pereirajair]

Intent

Captain asked firstmate to periodically monitor Claude usage quota and, when it runs high, automatically pause live crewmates (workers) in this home and resume them once quota recovers. Hard, non-negotiable requirement from the captain: the checking mechanism itself must never consume quota - it cannot be a Claude/model agent turn triggered on a schedule (explicitly ruled out the schedule/loop skills, which dispatch a real paid agent turn per firing). It must be a lightweight, pure shell script invoked by an OS-level scheduler (cron or launchd).

Delivered: bin/fm-quota-watch.sh - reads "quota-axi --provider claude --json" (same tool AGENTS.md section 4 already uses for dispatch decisions; researched its actual JSON schema rather than assuming one), takes the max percentUsed across all reported windows (session/weekly/credits), and acts:

• At/above a configurable pause threshold (default 80%, env FM_QUOTA_PAUSE_THRESHOLD or gitignored config/quota-pause-threshold): interrupts every LIVE kind=ship/kind=scout crewmate of this home using its harness's verified interrupt key from the harness-adapters skill (single Escape for claude/codex/pi/pi-signed/kimi, double Escape for opencode, Ctrl-C for grok - refuses rather than guessing for an unrecognized harness), sends nothing further, and records the pause both durably (state/.quota-paused, listing exactly which task ids this script paused) and on the crew's own status log using the ALREADY EXISTING paused: verb (AGENTS.md section 8 / bin/fm-classify-lib.sh) rather than inventing new supervision machinery.
• Below a lower hysteresis resume threshold (default 65%, env FM_QUOTA_RESUME_THRESHOLD or config/quota-resume-threshold) - deliberately below the pause threshold so a reading oscillating near 80% does not flap crew back and forth - sends one short note to every crewmate this script itself paused (never a crewmate that declared its own unrelated paused wait) and clears the flag.
• No usable reading (quota-axi missing, auth_required, malformed output) is always a harmless no-op: never a pause, never a crash. Verified this is the current live state in this environment (quota-axi is already authenticated and returns real data), so no captain decision is blocking right now; docs/quota-watch.md documents the one-time "quota-axi --allow-keychain-prompt" step for if auth ever lapses on a fresh machine.
• Idempotent: rerunning while still above the pause threshold never resends interrupts or duplicates the flag entry for an already-paused task; it only picks up crew spawned since the last pause.
• Never touches kind=secondmate (separate lifecycle) and never writes to projects/ - only this home's own state/ and crewmate pane input, exactly like bin/fm-send.sh already does.

Deliberately added NO new code to AGENTS.md or fm-watch.sh for "supervision awareness of a paused crew". I verified empirically (not just by reading code) that this was unnecessary: appending the existing paused verb to a crew's status log is enough, because bin/fm-crew-state.sh already re-reads the live pane/run-step before ever trusting that status line. I built a throwaway sandbox with a real tmux pane and proved both directions: an idle pane with a trailing paused: line reports state=paused source=status-log (absorbed as an expected wait, not escalated as stale), while a BUSY pane with that same stale paused: line still reports state=working source=pane - so a crewmate that resumes after the quota-freed note is never misread as still paused. This is why the change to bin/fm-classify-lib.sh / AGENTS.md is zero lines.

Also added: bin/fm-quota-watch-install.sh, an optional helper that only PRINTS a crontab line or launchd plist by default (an --install-crontab flag appends it, idempotently, only when the captain explicitly runs it themselves) - installing this repo's changes never starts a cron job or launchd agent on its own, since that is a per-machine action outside what a PR can carry. docs/quota-watch.md is the full behavior/setup writeup, with short pointers added from docs/configuration.md, docs/scripts.md, and README.md, plus the new config/quota-pause-threshold and config/quota-resume-threshold entries in .gitignore (config/ is not wholesale gitignored in this repo, only specific named files are, so I had to add these explicitly) and the docs/documentation-audiences.json inventory/readmeSetupTargets entries the structural doc-audience check requires.

tests/fm-quota-watch.test.sh covers: crossing the pause threshold (ship+scout paused, secondmate never touched), idempotent rerun at the same high pct, a crewmate spawned mid-pause being picked up without resending to already-paused ones, the hysteresis band leaving an existing pause alone, recovery below the resume threshold, auth_required as a harmless no-op, a missing quota-axi binary as a harmless no-op, an unrecognized harness being refused rather than guessed, and --status as a read-only inspection command. All fixture/mock-driven (fake quota-axi JSON, fake sender) - no real quota reading, backend, or live fleet touched by the tests. I also separately ran the real script end-to-end against the real quota-axi (with a sandboxed state dir and fake sender, to avoid disrupting the two real crewmates currently live in this home) and confirmed it correctly read the live quota and no-opped below threshold.

I also registered the new test in bin/fm-test-run.sh's family map and its conservative changed-path map (pure-contract-unit) so --changed and --family keep working for these new files, matching this repo's existing convention of never leaving a new bin/ script or test unclassified in that map.

POST-REVIEW FIX 1 (captain caught live-testing the merged PR): the original window-selection logic took the MAX percentUsed across ALL reported windows, including the "credits" window (paid overage spend, e.g. id extra_usage). That let a high paid-credit reading alone trigger a pause even while the actual free session/weekly quota was low, working against the captain's actual goal of maximizing free session/weekly usage without spending paid credits. Fixed: the jq filter now selects only windows with kind=="session" or kind=="weekly"; credits is deliberately, permanently excluded from the pause/resume decision. Updated docs/quota-watch.md and added tests/fm-quota-watch.test.sh coverage for a high-credits/low-session/weekly reading never pausing and never blocking a resume. The captain explicitly approved this as their own intentional decision when a reviewer flagged the (by-then-stale) original --intent text as contradicting it.

POST-REVIEW FIX 2 (captain caught this in actual production, not testing): the installed cron job read quota fine every time (quota-axi resolved), but every actual pause/resume ACTION silently failed with "herdr server ... did not report running" / "key 'Escape' not sent". Root cause: bin/fm-quota-watch-install.sh only ever resolved quota-axi's own directory into the generated cron/launchd PATH; herdr (needed by bin/fm-send.sh for this home's herdr backend) lives in /opt/homebrew/bin, which was never included, so cron's minimal PATH could not find it. Fixed: the install script now also resolves jq (required) and every session-backend CLI it finds actually installed (tmux, herdr, zellij, cmux - included only if present) into the generated PATH for both the crontab line and the launchd plist, then re-verifies every resolved binary actually resolves through that generated PATH under a scrubbed environment (env -i) before printing anything - catching a stale/incomplete PATH at generation time instead of failing silently on every cron firing. Added tests/fm-quota-watch-install.test.sh (new file, no prior coverage existed for the install script) proving this end to end, and updated docs/quota-watch.md.

What Changed

• Added bin/fm-quota-watch.sh, a pure-shell script (no Claude/model turn) meant for cron/launchd invocation that reads quota-axi --provider claude --json, takes the max percentUsed across only session/weekly windows (excluding paid credits), interrupts live kind=ship/kind=scout crewmates at a configurable pause threshold (default 80%), records the pause in state/.quota-paused and via the existing paused: <reason> status verb, and resumes paused crewmates below a lower hysteresis threshold (default 65%); missing/unauthenticated/malformed quota data is always a no-op, reruns are idempotent, and resume send failures are retried instead of dropping pause tracking.
• Added bin/fm-quota-watch-install.sh, an opt-in helper that prints (or, with --install-crontab, appends) a crontab line or launchd plist, resolving jq and available session-backend CLIs (tmux, herdr, zellij, cmux) into the generated PATH and re-verifying they resolve under a scrubbed environment before printing.
• Registered the new scripts in bin/fm-test-run.sh's family/changed-path maps, added tests/fm-quota-watch.test.sh and tests/fm-quota-watch-install.test.sh, and documented the feature in docs/quota-watch.md with pointers from docs/configuration.md, docs/scripts.md, README.md, and docs/documentation-audiences.json; added gitignore entries for the new threshold config files.

Risk Assessment

✅ Low: The requested fix was applied exactly as specified, is symmetric with the already-correct pause path, and the only remaining gap is missing test coverage for the new retry branch, which is a minor and easily addressed follow-up rather than a merge blocker.

Testing

Ran the two targeted fixture/mock-driven test suites for this change; all 16 cases pass, including a new case I wrote to cover the target commit's previously-untested resume-retry-on-send-failure behavior (partial delivery failure keeps only the undelivered crewmate flagged, without re-notifying the one that already got its resume note, and a follow-up run retries and clears the flag). No manual/live verification was needed since the existing fixture harness (fake quota-axi JSON, fake sender with a configurable failure hook) already exercises the real script end-to-end without touching a live fleet.

Evidence: New resume-retry test passing (tests/fm-quota-watch.test.sh)

ok - resume retries a failed send instead of dropping pause tracking, without re-notifying already-resumed crew

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 issues (1 warning, 1 info)

• ⚠️ bin/fm-quota-watch.sh:268 - In fm_quota_watch_resume (bin/fm-quota-watch.sh:268-285), the pause flag is unconditionally removed with rm -f "$PAUSE_FLAG" after iterating all recorded tasks, even for a task whose resume-note send failed (line 277-280 only logs a warning, does not retain the entry). A transient fm-send.sh failure for one crewmate during a resume cycle therefore permanently drops that crewmate's paused-tracking with no record it needs a retry, while the crewmate itself stays interrupted/idle forever since no future cycle will re-attempt the resume note or re-add it to a future pause. The pause path is symmetric and correct by contrast (bin/fm-quota-watch.sh:254-261): only tasks whose interrupt succeeds are written into tmp_flag, so a failed interrupt is retried on the next run. The test double for fm-send even ships a FM_TEST_SEND_FAIL_ID failure-simulation hook (tests/fm-quota-watch.test.sh:69-74) that is never exercised by any test, suggesting this failure path was not verified.

🔧 Fix: fix(quota-watch): retry resume send failures instead of dropping pause tracking
2 issues (1 warning, 1 info) still open:

• ⚠️ bin/fm-quota-watch.sh:268 - Prior finding: resume path dropped pause tracking on a failed send.
• ℹ️ tests/fm-quota-watch.test.sh:73 - The new retry-on-failed-resume-send path (bin/fm-quota-watch.sh:280-298) has no test coverage: tests/fm-quota-watch.test.sh's fake fm-send already supports FM_TEST_SEND_FAIL_ID to simulate a failed send, but no test sets it to exercise the retry branch (mv vs rm -f, pending/acted counts, or that a subsequent run re-attempts only the still-pending task).

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-quota-watch.test.sh (12 cases, including new test_resume_retries_failed_send)
• bash tests/fm-quota-watch-install.test.sh (4 cases, PATH-resolution and hard-failure coverage for the install script)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[kunchenguid]

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

_{Noted for firstmate#1251 at 0aab61f9.}

[kunchenguid]

Automated reminder: this PR still looks blocked on a rebase or merge conflict fix.

If you are still interested, please rebase onto the current base branch, resolve the conflict, and push.

If I do not hear back, I may close this as inactive.