fix(scheduler): self-healing 5-minute watchdog daemon, configurable stuck request recovery, and authenticated REST API

[thehaven]

Note

Disclosure: This pull request was developed with assistance from Google Antigravity / Gemini CLI (gemini-cli) AI coding assistant. All implementation logic and unit tests (vitest 100% passing) have been empirically verified in production container deployments.

📦 Summary & Problem Solved

1. Self-Healing 5-Minute Scheduler Watchdog Daemon

• Problem: Long-running internal cron scheduler instances could stall silently over extended runtimes, stopping automated tasks like library scans, indexer search retries, and shelf syncing.
• Fix: Added a self-healing 5-minute watchdog interval (src/lib/services/scheduler.service.ts) that monitors scheduled job execution freshness. If any cron job hasn't executed within its expected timeframe, the watchdog automatically re-queues and resumes execution without requiring container restarts.

2. Configurable Stuck Request Recovery (Slow Torrent Safe)

• Problem: Requests trapped in searching or processing due to unhandled worker crashes remained stuck indefinitely, but hardcoded short timeouts could interrupt slow torrent downloads or long-running private tracker seeds.
• Fix: Made stuck request recovery fully configurable via DB (system.stuck_request_timeout_hours) and ENV (STUCK_REQUEST_TIMEOUT_HOURS) with a safe default threshold of 6 hours (and setting to 0 disables recovery completely).

3. Authenticated REST API Endpoints & Self-Documentation

• Problem: Ability to inspect scheduled job status and trigger on-demand execution was missing.
• Fix: Added admin-authenticated REST API endpoints (POST /api/admin/scheduler/trigger, GET /api/admin/scheduler/status) enforcing verifyAccessToken JWT authentication and admin role checks (401 Unauthorized when unauthenticated). Integrated endpoints into the interactive /api-docs interface.

🔒 Authentication Usage Example

Endpoints require a Bearer token in the Authorization header obtained via /api/auth/local/login or via Admin API Tokens (rmab_...):

# 1. Obtain admin access token:
curl -X POST https://your-readmeabook-instance.example.com/api/auth/local/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "your_password"}'

# 2. Trigger on-demand background job execution:
curl -X POST https://your-readmeabook-instance.example.com/api/admin/scheduler/trigger \
  -H "Authorization: Bearer <YOUR_ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"type": "retry_missing_torrents"}'

# 3. Query scheduled job freshness & execution status:
curl -X GET https://your-readmeabook-instance.example.com/api/admin/scheduler/status \
  -H "Authorization: Bearer <YOUR_ACCESS_TOKEN>"

🧪 Unit Test Verification

 ✓ tests/processors/recover-stuck-requests.processor.test.ts (2 tests)
   ✓ should reset stuck searching requests back to awaiting_search
   ✓ should disable recovery when stuckTimeoutHours is configured to 0