fix(tuning): retune shared_buffers behind a pgbouncer PAUSE

[jaredLunde]

The bug

The VM's visible RAM changes at runtime — the guest boots with only part of its memory online and the rest is hot-plugged in later. But beyond-pg only read RAM at boot, derived shared_buffers from whatever was visible then, and never revisited it.

So an instance that grows stays sized for its smallest moment, for the life of the postmaster. Measured on a live VM: MemTotal: 752920 kB at boot → shared_buffers = 183MB, held there permanently even after the VM grew to 1.7 GiB. Nothing anywhere complained.

The fix

The memory watcher now follows RAM across both tuning halves:

• 02-memory.conf (sighup-context: effective_cache_size, work_mem, maintenance_work_mem, per-gather knobs) — rewritten and applied with pg_reload_conf(), as before.
• 01-tuning.conf (postmaster-context: shared_buffers, max_connections, wal_buffers, worker counts) — Postgres fixes these at postmaster start and cannot be talked out of them. The watcher asks the supervisor loop for a retune cycle: rewrite the conf → resize the hugepage reservation → PAUSE every pgbouncer worker (SIGUSR1) → fast-shutdown + respawn the postmaster → RESUME (SIGUSR2) on every path out, including failure (a pooler left paused is an outage).

PAUSE waits for in-flight transactions to be released and then queues new clients rather than dropping them, so the restart is a stall, not an error.

Gated by hysteresis: shared_buffers must move ≥25% (RETUNE_MIN_RATIO), at most once per RETUNE_MIN_INTERVAL — so a flapping balloon can't put the postmaster into a restart loop.

autoprewarm — the restart doesn't cost you a cold pool

A restart would normally drop the buffer pool cold. pg_prewarm.so was already in the image; we just never loaded it. It's now preloaded with its autoprewarm worker on: it periodically dumps the block list in shared_buffers to PGDATA/autoprewarm.blocks and reloads those blocks at startup (block identities, not data — the dump is small). Interval is 60s rather than the 300s default, since a retune or crash can land at any time.

This pays for every postmaster restart, not just retunes: reboots, image swaps, forks, restores.

Latent brick, also fixed

apply_kernel_settings only checked whether the nr_hugepages write succeeded, never whether the kernel actually granted the pages. Writing nr_hugepages is a request, not a guarantee — hugetlb pages must be physically contiguous, so the kernel reserves what it can and silently clamps the rest.

With huge_pages = on, a short reservation makes postgres refuse to start. Resizing shared_buffers upward could therefore have left the postmaster unable to come back. It now reads the count back and falls back to huge_pages = try on any shortfall, clearing the override again once a reservation succeeds.

shared_buffers and its hugepage reservation are now derived from one place (config::shared_buffers_mb / nr_hugepages_for) instead of three, so they can't drift apart.

Verification — on a live VM, not just tests

Retune (memory hot-plugged up underneath a running database, reproduced twice):

	before	after
visible RAM	752,920 kB	1,801,496 kB
shared_buffers	183MB	503MB
postmaster	—	restarted
client querying 1/sec across the retune	—	75 successes, 0 errors

autoprewarm — warmed the pool, then forced a retune 14s later, inside the 60s dump interval so no periodic dump had fired and only the shutdown dump could save it:

	before	after
buffers holding the table	2316	2316 (100%)

Plus 106 bin + 41 lib tests green. Five new tests pin the retune boundary — including that a single small hotplug step is correctly declined as not worth bouncing the postmaster, and that a shrink retunes too.

🤖 Generated with Claude Code