perf(tuning): coalesce a scale-up into one retune, and stop sleeping through the pause

[jaredLunde]

A full scale-up (128 → 1152 MiB of plugged memory, four steps ~80s apart) cost two postmaster restarts and two client stalls of 3783 ms and 3272 ms. Both numbers were self-inflicted.

1. Wait for the pool to drain, don't sleep through it

RETUNE_PAUSE_GRACE was a fixed sleep(3s) after signalling PgBouncer to PAUSE. It was the client-visible stall: of the 3056 ms a client blocked, 3000 ms was that sleep.

PAUSE releases each server connection as it goes idle — normally milliseconds. pg::wait_quiesced now polls Postgres for in-flight work and returns the moment no client backend is executing. The 3s survives only as a cap, so a stuck long-running query can't hold the retune open forever.

2. Wait for RAM to settle before retuning

The host grows a starved VM in steps, not one jump. Retuning on the first threshold crossing therefore guarantees another restart when the later steps land — which is precisely what the two restarts were.

The watcher now tracks when visible RAM last changed and defers the postmaster-context retune until it's been quiet for RETUNE_SETTLE (90s — must exceed the ~80s inter-step interval, or it would fire between steps). The whole climb coalesces into one restart.

Cost, stated honestly: shared_buffers lags the memory by up to ~90s. That's cheap — the reload-safe half (effective_cache_size, work_mem) still applies immediately on every step, and newly-plugged RAM is already doing useful work as page cache the moment it arrives.

Measured — same climb, same load, same client

	before	after
restarts	2	1
client stalls > 1s	2 (3783 ms, 3272 ms)	0
slowest query	3783 ms	253 ms
errors	0 / 1056	0 / 1084
median	25 ms	24 ms

Worst-case client stall drops 15×, and no query even crosses one second.

The trace, on a live VM:

t=  10s   128 MiB   183MB   02:38:27
t=  90s   384 MiB   183MB   02:38:27   ← plug, no restart
t= 170s   640 MiB   183MB   02:38:27   ← plug, no restart
t= 250s   896 MiB   183MB   02:38:27   ← plug, no restart
t= 340s  1152 MiB   183MB   02:38:27   ← plug, no restart
t= 430s  1152 MiB   471MB   02:46:33   ← ONE retune, after RAM went quiet

Settling also sizes better: shared_buffers lands at 471MB instead of 407MB, because it now tunes against the settled RAM rather than a mid-climb snapshot.

Tests: 107 + 41 + 33 green. New test pins the reason for the debounce — a single measured climb crosses the ratio threshold more than once.

🤖 Generated with Claude Code