Skip to content

stat_replication: fix "slot_name does not exist" by joining pg_replication_slots#1313

Open
megative wants to merge 1 commit into
prometheus-community:masterfrom
megative:fix/stat-replication-slot-name
Open

stat_replication: fix "slot_name does not exist" by joining pg_replication_slots#1313
megative wants to merge 1 commit into
prometheus-community:masterfrom
megative:fix/stat-replication-slot-name

Conversation

@megative

Copy link
Copy Markdown

Summary

Fixes #1310. The stat_replication collector currently fails every scrape with pq: column "slot_name" does not exist — affects every user on main after #1306, not just Aurora (despite the issue title).

The fix is a LEFT JOIN pg_replication_slots s ON s.active_pid = pg_stat_replication.pid so that slot_name is read from the view where it actually lives, with an empty label when no named slot is in use.

Test plan

  • go build ./..., go vet ./..., go test ./... clean.
  • Live: vanilla PG 17.10.0 — pg_scrape_collector_success{collector="stat_replication"} = 1 (was 0).
  • Live: Aurora PG 17.7.0 — pg_scrape_collector_success{collector="stat_replication"} = 1 (was 0).
Full timeline and root cause

Timeline

  1. Before Refactor stat_replication into standalone collector #1306, pg_stat_replication was scraped through the legacy yaml-driven exporter with a SELECT * FROM pg_stat_replication query. The yaml mapping listed slot_name as a label, but the column does not actually exist on pg_stat_replication — so the yaml exporter silently produced an empty label. No error.

  2. slot_name has never been a column of pg_stat_replication on any supported PG version. It lives on pg_replication_slots. To populate it for an active WAL sender you have to JOIN the two views. The legacy yaml never did. The label was effectively aspirational — described, but always empty. A latent bug sat in queries.yaml for years.

  3. Refactor stat_replication into standalone collector #1306 ("Refactor stat_replication into standalone collector") migrated this query into a Go-native collector. The new SQL was rewritten with an explicit column list, including slot_name:

    SELECT application_name, client_addr::text, state, slot_name, ...
    FROM pg_stat_replication

    That promoted slot_name from "yaml label silently missing" to "explicit SQL column that must exist". It does not.

  4. CI on Refactor stat_replication into standalone collector #1306 went green because unit tests for stat_replication use sqlmock, which returns whatever columns the test declares — the real Postgres schema is never validated. Integration tests apparently do not exercise this collector. The bug shipped.

  5. After Refactor stat_replication into standalone collector #1306 merged, stat_replication fails on every scrape, on every PG version. Because it is defaultEnabled, every user is affected automatically.

Verification that slot_name is not on pg_stat_replication

Three independent sources:

  1. \d pg_stat_replication on postgres:17.10.0 Docker — 20 columns, no slot_name.
  2. \d pg_stat_replication on postgres:18 Docker — 20 columns, no slot_name.
  3. Official docs for PG 17 and PG 18 — 20 columns documented, no slot_name. The PG 18 docs explicitly note slot_name lives on pg_stat_replication_slots.

Fix details

LEFT JOIN pg_replication_slots s ON s.active_pid = pg_stat_replication.pid and read s.slot_name. Behavior:

  • Replication client using a named slot → slot_name populated.
  • Replication client without a slot → slot_name empty (matches legacy yaml behavior).
  • Aurora and other setups without physical replication slots → slot_name empty.

Applied to both the PG ≥ 10 and pre-10 query paths. pg_replication_slots exists since PG 9.4, so the pre-9.4 branch remains affected by the same root cause; that branch is not in CI scope and was already broken before this PR.

…ation_slots

The stat_replication collector currently fails every scrape with:

  pq: column "slot_name" does not exist

slot_name is not a column of pg_stat_replication on any supported PostgreSQL version (verified on PG 17.10.0 Docker, PG 18 Docker, and the official PG 17 and PG 18 docs). It lives on pg_replication_slots and has to be pulled in via a JOIN on active_pid = pid.

Before prometheus-community#1306 the same collector ran via the legacy yaml-driven exporter, which used SELECT * and silently produced an empty slot_name label when the column was missing. prometheus-community#1306 promoted slot_name into an explicit SQL column list, turning that silent no-op into a hard parse-time error. CI did not catch it because the unit tests use sqlmock and do not validate against the real Postgres schema.

stat_replication is defaultEnabled, so every user on main after prometheus-community#1306 sees the collector dead on every scrape, regardless of whether they run Aurora or vanilla Postgres.

Fix: LEFT JOIN pg_replication_slots s ON s.active_pid = pg_stat_replication.pid and read s.slot_name. The label is populated when a named slot is in use, empty otherwise (matching the previous yaml behavior).

Fixes prometheus-community#1310.

Signed-off-by: Pavel K <megativ3@gmail.com>

@ArthurSens ArthurSens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the fix, and apologies for not seeing this PR before making the 0.20.0 release.

There's still some small work to be done to make sure this also works with older versions of PG.

Could you also rebase the PR?

s.slot_name,
(case pg_is_in_recovery() when 't' then pg_xlog_location_diff(pg_last_xlog_receive_location(), replay_location)::float else pg_xlog_location_diff(pg_current_xlog_location(), replay_location)::float end) AS pg_xlog_location_diff
FROM pg_stat_replication
LEFT JOIN pg_replication_slots s ON s.active_pid = pg_stat_replication.pid

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this query only works for postgres 9.5 and above, because active_pid was only added in PG 9.5.x. See https://www.postgresql.org/docs/9.5/view-pg-replication-slots.html and https://www.postgresql.org/docs/9.4/catalog-pg-replication-slots.html

Could you please have a query for 9.4 and below that doesn't include slot_name?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: stat_replication collector crashes ("slot_name" does not exist)

2 participants