Sink Health Monitoring: PagerDuty & Incident.io Notification Channels

[joejohnson123]

Overview

Add PagerDuty and Incident.io as notification channel types, reusing the existing Sequin.Pagerduty and Sequin.IncidentIO modules.

Depends on: #4 (Notification Dispatcher & Webhook Channel)

PagerDuty Channel: Sequin.Monitors.Channels.Pagerduty

Implementation

• Reuse Sequin.Pagerduty module's alert/3 and resolve/3 functions
• Channel config provides a per-monitor routing key (overrides global config)
• Dedup key: sink-monitor:{monitor_id} (ensures PagerDuty groups related alerts)

Behavior

• On trigger: Call Pagerduty.alert(dedup_key, summary, severity: :warning, routing_key: config.routing_key)
  • Need to extend Sequin.Pagerduty to accept a per-call routing_key override
• On resolve: Call Pagerduty.resolve(dedup_key, summary, routing_key: config.routing_key)
• Summary format: "[Sequin Monitor] {monitor_name}: {metric} ({value}) exceeded threshold ({threshold}) on sink {sink_name}"
• Include link to sink dashboard in PagerDuty alert links

Config Shape

%{
  "routing_key" => "your-pagerduty-integration-key",
  "severity" => "warning"  # critical | error | warning | info
}

Incident.io Channel: Sequin.Monitors.Channels.IncidentIO

Implementation

• Reuse Sequin.IncidentIO module's create_incident/2 function
• Channel config provides per-monitor API key and severity
• Idempotency key: sink-monitor:{monitor_id}:{alert_id} (prevents duplicate incidents)

Behavior

• On trigger: Create incident with configured severity
• On resolve: No auto-resolve (Incident.io incidents are resolved manually or via their API — add resolve support if their API supports it)

Config Shape

%{
  "api_key" => "inc_...",
  "severity" => "minor"  # minor | major | critical
}

Changes to Existing Modules

Sequin.Pagerduty

• Add routing_key option to alert/3 and resolve/3
• When provided, override the global integration_key from config
• Backwards compatible — existing calls without routing_key still work

Sequin.IncidentIO

• Add optional api_key parameter to create_incident/2
• When provided, override the global API key
• Backwards compatible

Files to Create

• lib/sequin/monitors/channels/pagerduty.ex
• lib/sequin/monitors/channels/incident_io.ex
• test/sequin/monitors/channels/pagerduty_test.exs
• test/sequin/monitors/channels/incident_io_test.exs

Files to Modify

• lib/sequin/pagerduty.ex — add per-call routing key support
• lib/sequin/incident_io.ex — add per-call API key support

Acceptance Criteria

• PagerDuty alerts fire with correct dedup key and severity
• PagerDuty auto-resolves when monitor recovers
• Per-monitor routing key overrides global config
• Incident.io creates incidents with correct severity
• Idempotency prevents duplicate incidents
• Existing PagerDuty/IncidentIO callers unaffected (backwards compatible)
• Tests mock HTTP calls appropriately