Sink Health Monitoring: Threshold Evaluation Worker

[joejohnson123]

Overview

Implement the Oban worker that periodically evaluates all enabled threshold monitors and fires/resolves alerts.

Depends on: #1 (Schema), #2 (Context Module)

Worker: Sequin.Monitors.EvaluateMonitorsWorker

Scheduling

• Oban recurring job, runs every 30 seconds
• Register in Sequin.Application alongside existing Oban workers
• Use unique: [period: 25] to prevent overlap

Evaluation Loop (Threshold Monitors)

For each enabled monitor where monitor_type == :threshold:

1. Fetch current metric value via Monitors.current_metric_value(monitor)
2. Compare against threshold:
   • If value >= threshold_value:
     • Check Redis key monitor:threshold_crossed:{monitor_id} for when threshold was first crossed
     • If key doesn't exist, set it to now with TTL of threshold_window_seconds * 2
     • If key exists and now - first_crossed >= threshold_window_seconds:
       • Check if cooldown is active → skip if yes
       • Check if there's already an active (unresolved) alert → skip if yes
       • Fire alert: create alert record, dispatch notifications
   • If value < threshold_value:
     • Delete Redis key monitor:threshold_crossed:{monitor_id}
     • If there's an active alert AND notify_on_recovery is true:
       • Resolve alert: update alert record, dispatch recovery notification

Redis Keys

• monitor:threshold_crossed:{monitor_id} — stores ISO timestamp of when threshold was first crossed
• TTL: threshold_window_seconds * 2 (auto-cleanup if worker stops)

Error Handling

• If fetching metric value fails (e.g., consumer not running), log warning and skip
• If notification dispatch fails, log error but don't fail the evaluation
• Wrap each monitor evaluation in try/rescue so one failure doesn't block others

Files to Create

• lib/sequin/monitors/evaluate_monitors_worker.ex
• test/sequin/monitors/evaluate_monitors_worker_test.exs

Implementation Notes

• Use Sequin.Redis for threshold-crossed tracking (consistent with existing health event storage)
• Process monitors in batches if there are many (unlikely initially but good practice)
• Emit telemetry events: [:sequin, :monitors, :evaluated] with count of monitors checked
• Log at debug level for normal evaluations, warn for threshold crossings, error for failures

Acceptance Criteria

• Worker runs on schedule via Oban
• Correctly detects when metric crosses threshold
• Respects threshold_window_seconds debounce (doesn't fire immediately)
• Fires alert only once per incident (checks for active alert)
• Resolves alert when metric drops below threshold
• Respects cooldown period
• Handles errors gracefully without blocking other monitors
• No Process.sleep in tests — use telemetry handlers or signal passing
• Tests don't generate extra logs