Skip to content

Add metrics ingest endpoint for warehouse loader #7293

@gagantrivedi

Description

@gagantrivedi

Summary

Internal endpoint the Lambda loader (#7292) calls after each run to report completion metrics against a pre-created metrics row (seeded by the recurring task). Core persists the update and serves aggregates via the telemetry endpoint (#7289).

Depends on: #7276 (connection model)
Consumed by: #7289 (telemetry read)
Called by: #7292 (Lambda loader)
Row seeded by: the recurring task (separate issue)

Endpoint

PATCH /api/v1/organisations/{organisation_pk}/integrations/warehouse-connections/{id}/metrics/{run_id}/

run_id refers to the metrics row pre-created by the recurring task when it invoked the Lambda. The Lambda receives run_id in its event payload and echoes it back here.

Request body

{
  "completed_at": "2026-04-20T11:00:12Z",
  "events_delivered": 148233,
  "batches_dlq": 0,
  "error": null
}
  • completed_at — when the Lambda run finished
  • events_delivered — total events successfully COPY'd on this run (0 on failure or empty-prefix ticks)
  • batches_dlq — number of batches that exhausted retries on this run and landed in the DLQ
  • error — short error string if the run itself errored out before completing; null otherwise

Response

204 No Content on success. 404 Not Found if run_id does not exist.

Auth

Service-to-service — not user-authenticated. Mechanism to be decided by implementer (internal API key, IAM/SigV4, or shared secret header).

Storage

Rows in a WarehouseConnectionMetrics table (or similar). Each row represents one loader run and moves through two states:

  1. Created by the recurring task at invocation time, with started_at populated and completion fields null
  2. Updated by this endpoint when the Lambda reports back, populating completed_at and the metric fields

Rows where started_at is set but completed_at remains null past an expected window indicate an orphaned/crashed Lambda run. #7289 uses this table to compute rolling 24h / 7d aggregates cheaply.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions