fix(tuning): retune shared_buffers behind a pgbouncer PAUSE

DECISIONS.md

@@ -538,9 +538,14 @@ plus the vCPU-derived parallelism knobs (`max_worker_processes`,
 
 **Why.** Static defaults are wrong everywhere except one size. The
 image runs on 4 GB / 2 vCPU dev boxes and 256 GB / 64 vCPU production
-boxes from the same artifact. Resizing a box (`byd pg scale`) should
-leave the database correctly tuned without a manual step. Regenerating
-every boot makes that automatic.
+boxes from the same artifact. Regenerating every boot makes that
+automatic.
+
+**Boot-time tuning is not sufficient on its own — see F-011.** The VM's
+visible RAM changes at runtime: the guest boots with only part of its
+memory online and the rest is hot-plugged in later. So "the RAM at boot"
+is not the RAM the database ends up with, and a boot-only tuner leaves a
+grown instance sized for its smallest moment.
 
 **Why cgroup-aware.** Inside Docker for `beyond dev`, `/proc/meminfo`
 reports host RAM, not the container's cgroup limit. If we set
@@ -699,6 +704,50 @@ well-known PG footgun. Disable it. Use explicit huge pages
 want them; PG default `huge_pages = try` is fine without explicit
 setup.
 
+### F-011 — Follow elastic memory: retune + restart the postmaster behind a PgBouncer PAUSE
+
+**Decision.** The memory watcher follows virtio-mem RAM changes across
+*both* tuning halves, not just the reload-safe one:
+
+- `02-memory.conf` (sighup-context: `effective_cache_size`, `work_mem`,
+  `maintenance_work_mem`, the per-gather knobs) — rewritten and applied
+  with `pg_reload_conf()`, as before.
+- `01-tuning.conf` (postmaster-context: `shared_buffers`,
+  `max_connections`, `wal_buffers`, the worker counts) — Postgres fixes
+  these at postmaster start and cannot be talked out of them. The
+  watcher therefore asks the supervisor loop for a **retune cycle**:
+  rewrite the conf, resize the hugepage reservation, `PAUSE` every
+  PgBouncer worker (SIGUSR1), fast-shutdown and respawn the postmaster,
+  then `RESUME` (SIGUSR2) on every path out — including failure.
+
+Gated by hysteresis: only when `shared_buffers` would move by ≥25%
+(`RETUNE_MIN_RATIO`), and at most once per `RETUNE_MIN_INTERVAL`.
+
+**Why.** The VM's memory is elastic: it boots with only part of its RAM
+online and the rest is hot-plugged in as the workload needs it.
+`shared_buffers` is the single most important memory knob in Postgres and
+the one that *cannot* change without a restart — so a guest that only
+reads RAM at boot stays permanently mis-tuned for the memory it actually
+ends up with.
+
+The alternative — pinning databases to a fixed size at boot — was
+rejected: it opts Postgres out of elastic memory entirely, and costs a
+capacity knob we don't want. The guest should participate in elasticity,
+not sit outside it.
+
+**Why the restart is safe.** PgBouncer's `PAUSE` disconnects each server
+connection *after waiting for it to be released* per pool mode, and new
+client queries then queue rather than error ("new client connections to
+a paused database will wait until RESUME"). Clients see a stall, not a
+dropped connection. Verified end-to-end on a live primitive: a client
+querying once/sec across the retune logged **75 successes, 0 errors**,
+while `shared_buffers` went 183MB → 503MB and the postmaster restarted.
+
+**Cost.** The restart drops the buffer pool cold. The OS page cache
+survives in the same guest, so re-warm is mostly from cache rather than
+disk; `pg_prewarm` is not in the image today.
+
+
 ---
 
 ## G — Bootstrap

DESIGN.md

@@ -412,7 +412,14 @@ subcommand because there's no external invoker that needs a CLI surface.
    max_parallel_workers_per_gather = {clamp(vcpus / 2, 1, 4)}
    max_parallel_maintenance_workers = {clamp(vcpus / 2, 1, 4)}
    ```
-   Regenerated every boot so resized VMs pick up new defaults.
+   Regenerated every boot so resized VMs pick up new defaults — and, because
+   the VM's memory is elastic (it boots with only part of its RAM online and
+   the rest is hot-plugged in later), regenerated *again* whenever visible
+   RAM moves.
+   The reload-safe half (`02-memory.conf`) is applied with `pg_reload_conf()`;
+   the postmaster-context half (`01-tuning.conf` — `shared_buffers` above all)
+   requires a postmaster restart, which the supervisor performs behind a
+   PgBouncer `PAUSE` so clients stall instead of erroring. See DECISIONS F-011.
 
    **Why this `work_mem` formula** (instead of `ram * 0.01 / max_connections`):
    `work_mem` is allocated per sort/hash _node_ in the query plan, not per

packer/files/postgresql/00-beyond.conf

@@ -53,9 +53,32 @@ log_temp_files = 10MB
 # memory) and are installed on demand by the beyond-auth service's migration, so
 # they don't need preloading. beyond_queue DOES need preload (shared-memory
 # waiter registry / routing cache for push-based receive).
-shared_preload_libraries = 'pg_stat_statements,auto_explain,pg_cron,beyond_queue'
+shared_preload_libraries = 'pg_stat_statements,auto_explain,pg_cron,beyond_queue,pg_prewarm'
 pg_stat_statements.max = 10000
 pg_stat_statements.track = all
+
+# ---------------------------------------------------------------------------
+# autoprewarm — survive a restart with a warm buffer pool
+# ---------------------------------------------------------------------------
+# pg_prewarm's background worker periodically dumps the *block list* currently
+# in shared_buffers to PGDATA/autoprewarm.blocks, and reloads those blocks back
+# into shared_buffers at startup. It stores block identities, not data — the
+# dump is small and cheap.
+#
+# This image restarts the postmaster more often than a hand-run Postgres does:
+# the VM's memory is elastic, so the supervisor re-derives shared_buffers and
+# restarts when visible RAM moves (see DECISIONS F-011), and the same volume is
+# also rebooted, image-swapped, forked, and restored. Every one of those events
+# would otherwise come back with a stone-cold buffer pool and a latency cliff
+# while it refills. Prewarming turns that cliff into a ramp.
+#
+# Ref: PostgreSQL docs §F.29 (pg_prewarm)
+pg_prewarm.autoprewarm = on
+# Dump every 60s rather than the 300s default: a retune or a crash can land at
+# any time, and a 5-minute-stale block list can miss most of a working set that
+# turned over in between. The dump is a list of block numbers, so paying for it
+# 5x more often is still negligible.
+pg_prewarm.autoprewarm_interval = 60s
 auto_explain.log_min_duration = 1000
 auto_explain.log_analyze = on
 auto_explain.log_buffers = on

src/boot.rs

@@ -136,8 +136,7 @@ async fn do_boot_primary(cfg: &MmdsConfig) -> Result<(), BootError> {
     write_config_files(cfg, &tls)?;
     info!("boot: step 7/8 write_pitr_config");
     write_pitr_config(cfg)?;
-    let shared_buffers_mb = (cfg.ram_bytes / (1024 * 1024) / 4).max(128);
-    apply_kernel_settings(shared_buffers_mb);
+    apply_kernel_settings(config::shared_buffers_mb(cfg.ram_bytes));
 
     info!("boot: step 8/8 run_hook_scripts");
     run_hook_scripts(HOOKS_PRE_START).await?;
@@ -190,8 +189,7 @@ async fn do_boot_replica(cfg: &MmdsConfig) -> Result<(), BootError> {
     // root-owned; postgres refuses to start on a non-postgres-owned data dir.
     chown_data_tree();
 
-    let shared_buffers_mb = (cfg.ram_bytes / (1024 * 1024) / 4).max(128);
-    apply_kernel_settings(shared_buffers_mb);
+    apply_kernel_settings(config::shared_buffers_mb(cfg.ram_bytes));
 
     run_hook_scripts(HOOKS_PRE_START).await?;
 
@@ -1124,47 +1122,101 @@ fn apply_kernel_settings(shared_buffers_mb: u64) {
         Err(e) => warn!("could not set transparent hugepages: {e}"),
     }
 
-    // Static hugepages: reserve enough 2 MB pages to back shared_buffers, plus
-    // 32 pages (64 MB) of overhead for WAL buffers and other shared memory in
-    // the same segment.
-    // Percona benchmark: TLB faults drop from ~200k/s to near zero with hugepages.
-    // Ref: Percona "Benchmark PostgreSQL with Linux HugePages";
-    //      PostgreSQL docs §19.4 "Managing Kernel Resources"
-    let nr_hugepages = shared_buffers_mb / 2 + 32;
-    let hugepages_reserved =
-        match std::fs::write("/proc/sys/vm/nr_hugepages", format!("{nr_hugepages}\n")) {
-            Ok(()) => {
-                info!(
-                    "reserved {nr_hugepages} hugepages ({} MB)",
-                    nr_hugepages * 2
-                );
-                true
-            }
+    apply_hugepages(shared_buffers_mb);
+}
+
+/// Reserve the hugepages that must back `shared_buffers_mb`, and keep the
+/// `huge_pages = try` fallback file in step with whether that succeeded.
+///
+/// Called at boot, and again by the supervisor's retune cycle when virtio-mem
+/// grows RAM enough to move `shared_buffers` — the reservation has to be resized
+/// *before* the postmaster restarts into the new `shared_buffers`, or it won't
+/// come back up.
+///
+/// Percona benchmark: TLB faults drop from ~200k/s to near zero with hugepages.
+/// Ref: Percona "Benchmark PostgreSQL with Linux HugePages";
+///      PostgreSQL docs §19.4 "Managing Kernel Resources"
+pub fn apply_hugepages(shared_buffers_mb: u64) {
+    let reserved = reserve_hugepages(config::nr_hugepages_for(shared_buffers_mb));
+
+    // `01-tuning.conf` hardcodes `huge_pages = on`: the right production default,
+    // because postgres then fails fast rather than silently giving up the TLB win.
+    // But `on` also means postgres REFUSES TO START if the reservation came up
+    // short — so whenever it does, we must override to `try` via a high-numbered
+    // conf.d file, or the postmaster never comes back.
+    //
+    // Two ways it comes up short: an environment where we can't write the sysctl
+    // at all (unprivileged containers, locked-down dev hosts), and a reservation
+    // sized against hot-added RAM, which lands in ZONE_MOVABLE and may not be able
+    // to back hugepages at all. Both are handled here, and the
+    // override is REMOVED again once a reservation succeeds, so a transient
+    // shortfall doesn't strand us on `try` forever.
+    let override_path = format!("{PGDATA}/conf.d/99-hugepages-fallback.conf");
+    if reserved {
+        match std::fs::remove_file(&override_path) {
+            Ok(()) => info!("hugepages reserved; cleared {override_path}"),
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
+            Err(e) => warn!("could not remove {override_path}: {e}"),
+        }
+        return;
+    }
+    let body = "# Generated automatically when the nr_hugepages reservation came\n\
+                # up short (boot::apply_hugepages). Postgres tries hugepages and\n\
+                # falls back to anonymous shmem rather than refusing to start.\n\
+                huge_pages = try\n";
+    match config::write_atomic(Path::new(&override_path), body) {
+        Ok(()) => info!("huge_pages override written to {override_path} (fallback to try)"),
+        Err(e) => warn!("could not write huge_pages override: {e}"),
+    }
+}
+
+const NR_HUGEPAGES: &str = "/proc/sys/vm/nr_hugepages";
+
+/// Reserve `want` 2 MB hugepages, returning true only if the kernel actually
+/// handed over every one of them.
+///
+/// Writing `nr_hugepages` is a *request*, not a guarantee: hugetlb pages must be
+/// physically contiguous, so the kernel reserves as many as it can and silently
+/// clamps the rest. The write returns `Ok(())` either way. Checking only the
+/// write result — as this did before — treats a partial reservation as a full
+/// one, and `huge_pages = on` then makes postgres refuse to start, because it
+/// cannot map its whole shared-memory segment.
+///
+/// This matters more than it looks here. The VM boots with only part of its RAM
+/// online; memory hot-added later lands in `ZONE_MOVABLE`, which the kernel keeps
+/// for migratable allocations so the memory can be removed again. A hugepage
+/// reservation sized against grown RAM can therefore come up short even though
+/// the RAM is "there". Read the number back and believe the kernel, not the
+/// request.
+fn reserve_hugepages(want: u64) -> bool {
+    if let Err(e) = std::fs::write(NR_HUGEPAGES, format!("{want}\n")) {
+        warn!("could not set nr_hugepages: {e}");
+        return false;
+    }
+    let got = match std::fs::read_to_string(NR_HUGEPAGES) {
+        Ok(s) => match s.trim().parse::<u64>() {
+            Ok(n) => n,
             Err(e) => {
-                warn!("could not set nr_hugepages: {e}");
-                false
+                warn!("could not parse nr_hugepages read-back: {e}");
+                return false;
             }
-        };
-
-    // The tuning conf hardcodes `huge_pages = on`. That's the right
-    // production default — postgres fails fast if its hugepage reservation
-    // wasn't successfully provisioned, rather than silently falling back
-    // to regular pages and giving up the TLB win. In environments where
-    // we *couldn't* reserve hugepages (unprivileged containers, dev hosts
-    // with locked-down sysfs), forcing `on` would make postgres refuse to
-    // start. Detect that and override with `try` via a high-numbered
-    // conf.d file so postgres still boots cleanly.
-    if !hugepages_reserved {
-        let override_path = format!("{PGDATA}/conf.d/99-hugepages-fallback.conf");
-        let body = "# Generated automatically when nr_hugepages reservation\n\
-                    # failed at boot (apply_kernel_settings). Postgres tries\n\
-                    # hugepages and falls back to anonymous shmem if unavailable.\n\
-                    huge_pages = try\n";
-        match config::write_atomic(Path::new(&override_path), body) {
-            Ok(()) => info!("huge_pages override written to {override_path} (fallback to try)"),
-            Err(e) => warn!("could not write huge_pages override: {e}"),
+        },
+        Err(e) => {
+            warn!("could not read back nr_hugepages: {e}");
+            return false;
         }
+    };
+    if got < want {
+        warn!(
+            "hugepage reservation short: asked {want} ({} MB), kernel gave {got} ({} MB) \
+             — falling back to huge_pages=try so postgres still starts",
+            want * 2,
+            got * 2
+        );
+        return false;
     }
+    info!("reserved {got} hugepages ({} MB)", got * 2);
+    true
 }
 
 // ---------------------------------------------------------------------------

src/config.rs

@@ -291,8 +291,30 @@ pub fn pgbouncer_ini(
 // Generated: 01-tuning.conf  (postmaster-context — restart required)
 // ---------------------------------------------------------------------------
 
+/// `shared_buffers`, in MB, for a VM with `ram_bytes` of visible RAM.
+///
+/// 25% of RAM, floor 128MB. Single source of truth: the postmaster config, the
+/// hugepage reservation that must back it (`boot::apply_hugepages`), and the
+/// memory watcher's "is a retune worth a restart?" test all derive from this, and
+/// they must not be able to drift apart.
+///
+/// Ref: PostgreSQL docs §20.4; pganalyze shared_buffers benchmark (2024) —
+/// 25% is optimal up to ~64 GB; gains plateau above 40% due to double-buffering.
+pub fn shared_buffers_mb(ram_bytes: u64) -> u64 {
+    (ram_bytes / (1024 * 1024) / 4).max(128)
+}
+
+/// Number of 2 MB hugepages needed to back `shared_buffers_mb`, plus 32 pages
+/// (64 MB) of headroom for WAL buffers and the rest of the shared-memory segment.
+pub fn nr_hugepages_for(shared_buffers_mb: u64) -> u64 {
+    shared_buffers_mb / 2 + 32
+}
+
 /// Postmaster-context parameters — require a Postgres restart to take effect.
-/// Written once at boot; not touched by the memory watcher.
+///
+/// Written at boot, and rewritten by the memory watcher's retune cycle when
+/// virtio-mem changes visible RAM enough to move `shared_buffers` — the postmaster
+/// is then restarted behind a pgbouncer PAUSE so the new values take effect.
 pub fn tuning_conf_boot(ram_bytes: u64, vcpus: u32) -> String {
     let ram_mb = ram_bytes / (1024 * 1024);
     let pool_size = pgbouncer_pool_size(vcpus); // server connections PgBouncer opens
@@ -307,10 +329,7 @@ pub fn tuning_conf_boot(ram_bytes: u64, vcpus: u32) -> String {
     // Ref: PostgreSQL wiki "Number Of Database Connections"; Mattermost PgBouncer study
     let max_connections = (pool_size + vcpus * 2 + 10).clamp(100, (ram_mb / 50).max(100));
 
-    // shared_buffers: 25% of RAM, floor 128MB.
-    // Ref: PostgreSQL docs §20.4; pganalyze shared_buffers benchmark (2024) —
-    // 25% is optimal up to ~64 GB; gains plateau above 40% due to double-buffering.
-    let shared_buffers_mb = (ram_mb / 4).max(128);
+    let shared_buffers_mb = shared_buffers_mb(ram_bytes);
 
     // wal_buffers: replicates Postgres auto-tune (wal_buffers=-1 = shared_buffers/32).
     // 16 MB ceiling is the historical single-WAL-segment size and is sufficient for