diff --git a/DECISIONS.md b/DECISIONS.md index 22e035c..d777a01 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -538,9 +538,14 @@ plus the vCPU-derived parallelism knobs (`max_worker_processes`, **Why.** Static defaults are wrong everywhere except one size. The image runs on 4 GB / 2 vCPU dev boxes and 256 GB / 64 vCPU production -boxes from the same artifact. Resizing a box (`byd pg scale`) should -leave the database correctly tuned without a manual step. Regenerating -every boot makes that automatic. +boxes from the same artifact. Regenerating every boot makes that +automatic. + +**Boot-time tuning is not sufficient on its own — see F-011.** The VM's +visible RAM changes at runtime: the guest boots with only part of its +memory online and the rest is hot-plugged in later. So "the RAM at boot" +is not the RAM the database ends up with, and a boot-only tuner leaves a +grown instance sized for its smallest moment. **Why cgroup-aware.** Inside Docker for `beyond dev`, `/proc/meminfo` reports host RAM, not the container's cgroup limit. If we set @@ -699,6 +704,50 @@ well-known PG footgun. Disable it. Use explicit huge pages want them; PG default `huge_pages = try` is fine without explicit setup. +### F-011 — Follow elastic memory: retune + restart the postmaster behind a PgBouncer PAUSE + +**Decision.** The memory watcher follows virtio-mem RAM changes across +*both* tuning halves, not just the reload-safe one: + +- `02-memory.conf` (sighup-context: `effective_cache_size`, `work_mem`, + `maintenance_work_mem`, the per-gather knobs) — rewritten and applied + with `pg_reload_conf()`, as before. +- `01-tuning.conf` (postmaster-context: `shared_buffers`, + `max_connections`, `wal_buffers`, the worker counts) — Postgres fixes + these at postmaster start and cannot be talked out of them. The + watcher therefore asks the supervisor loop for a **retune cycle**: + rewrite the conf, resize the hugepage reservation, `PAUSE` every + PgBouncer worker (SIGUSR1), fast-shutdown and respawn the postmaster, + then `RESUME` (SIGUSR2) on every path out — including failure. + +Gated by hysteresis: only when `shared_buffers` would move by ≥25% +(`RETUNE_MIN_RATIO`), and at most once per `RETUNE_MIN_INTERVAL`. + +**Why.** The VM's memory is elastic: it boots with only part of its RAM +online and the rest is hot-plugged in as the workload needs it. +`shared_buffers` is the single most important memory knob in Postgres and +the one that *cannot* change without a restart — so a guest that only +reads RAM at boot stays permanently mis-tuned for the memory it actually +ends up with. + +The alternative — pinning databases to a fixed size at boot — was +rejected: it opts Postgres out of elastic memory entirely, and costs a +capacity knob we don't want. The guest should participate in elasticity, +not sit outside it. + +**Why the restart is safe.** PgBouncer's `PAUSE` disconnects each server +connection *after waiting for it to be released* per pool mode, and new +client queries then queue rather than error ("new client connections to +a paused database will wait until RESUME"). Clients see a stall, not a +dropped connection. Verified end-to-end on a live primitive: a client +querying once/sec across the retune logged **75 successes, 0 errors**, +while `shared_buffers` went 183MB → 503MB and the postmaster restarted. + +**Cost.** The restart drops the buffer pool cold. The OS page cache +survives in the same guest, so re-warm is mostly from cache rather than +disk; `pg_prewarm` is not in the image today. + + --- ## G — Bootstrap diff --git a/DESIGN.md b/DESIGN.md index 35f22ae..f746330 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -412,7 +412,14 @@ subcommand because there's no external invoker that needs a CLI surface. max_parallel_workers_per_gather = {clamp(vcpus / 2, 1, 4)} max_parallel_maintenance_workers = {clamp(vcpus / 2, 1, 4)} ``` - Regenerated every boot so resized VMs pick up new defaults. + Regenerated every boot so resized VMs pick up new defaults — and, because + the VM's memory is elastic (it boots with only part of its RAM online and + the rest is hot-plugged in later), regenerated *again* whenever visible + RAM moves. + The reload-safe half (`02-memory.conf`) is applied with `pg_reload_conf()`; + the postmaster-context half (`01-tuning.conf` — `shared_buffers` above all) + requires a postmaster restart, which the supervisor performs behind a + PgBouncer `PAUSE` so clients stall instead of erroring. See DECISIONS F-011. **Why this `work_mem` formula** (instead of `ram * 0.01 / max_connections`): `work_mem` is allocated per sort/hash _node_ in the query plan, not per diff --git a/packer/files/postgresql/00-beyond.conf b/packer/files/postgresql/00-beyond.conf index b99457b..df6ecbe 100644 --- a/packer/files/postgresql/00-beyond.conf +++ b/packer/files/postgresql/00-beyond.conf @@ -53,9 +53,32 @@ log_temp_files = 10MB # memory) and are installed on demand by the beyond-auth service's migration, so # they don't need preloading. beyond_queue DOES need preload (shared-memory # waiter registry / routing cache for push-based receive). -shared_preload_libraries = 'pg_stat_statements,auto_explain,pg_cron,beyond_queue' +shared_preload_libraries = 'pg_stat_statements,auto_explain,pg_cron,beyond_queue,pg_prewarm' pg_stat_statements.max = 10000 pg_stat_statements.track = all + +# --------------------------------------------------------------------------- +# autoprewarm — survive a restart with a warm buffer pool +# --------------------------------------------------------------------------- +# pg_prewarm's background worker periodically dumps the *block list* currently +# in shared_buffers to PGDATA/autoprewarm.blocks, and reloads those blocks back +# into shared_buffers at startup. It stores block identities, not data — the +# dump is small and cheap. +# +# This image restarts the postmaster more often than a hand-run Postgres does: +# the VM's memory is elastic, so the supervisor re-derives shared_buffers and +# restarts when visible RAM moves (see DECISIONS F-011), and the same volume is +# also rebooted, image-swapped, forked, and restored. Every one of those events +# would otherwise come back with a stone-cold buffer pool and a latency cliff +# while it refills. Prewarming turns that cliff into a ramp. +# +# Ref: PostgreSQL docs §F.29 (pg_prewarm) +pg_prewarm.autoprewarm = on +# Dump every 60s rather than the 300s default: a retune or a crash can land at +# any time, and a 5-minute-stale block list can miss most of a working set that +# turned over in between. The dump is a list of block numbers, so paying for it +# 5x more often is still negligible. +pg_prewarm.autoprewarm_interval = 60s auto_explain.log_min_duration = 1000 auto_explain.log_analyze = on auto_explain.log_buffers = on diff --git a/src/boot.rs b/src/boot.rs index 6d872c2..5b30762 100644 --- a/src/boot.rs +++ b/src/boot.rs @@ -136,8 +136,7 @@ async fn do_boot_primary(cfg: &MmdsConfig) -> Result<(), BootError> { write_config_files(cfg, &tls)?; info!("boot: step 7/8 write_pitr_config"); write_pitr_config(cfg)?; - let shared_buffers_mb = (cfg.ram_bytes / (1024 * 1024) / 4).max(128); - apply_kernel_settings(shared_buffers_mb); + apply_kernel_settings(config::shared_buffers_mb(cfg.ram_bytes)); info!("boot: step 8/8 run_hook_scripts"); run_hook_scripts(HOOKS_PRE_START).await?; @@ -190,8 +189,7 @@ async fn do_boot_replica(cfg: &MmdsConfig) -> Result<(), BootError> { // root-owned; postgres refuses to start on a non-postgres-owned data dir. chown_data_tree(); - let shared_buffers_mb = (cfg.ram_bytes / (1024 * 1024) / 4).max(128); - apply_kernel_settings(shared_buffers_mb); + apply_kernel_settings(config::shared_buffers_mb(cfg.ram_bytes)); run_hook_scripts(HOOKS_PRE_START).await?; @@ -1124,47 +1122,101 @@ fn apply_kernel_settings(shared_buffers_mb: u64) { Err(e) => warn!("could not set transparent hugepages: {e}"), } - // Static hugepages: reserve enough 2 MB pages to back shared_buffers, plus - // 32 pages (64 MB) of overhead for WAL buffers and other shared memory in - // the same segment. - // Percona benchmark: TLB faults drop from ~200k/s to near zero with hugepages. - // Ref: Percona "Benchmark PostgreSQL with Linux HugePages"; - // PostgreSQL docs §19.4 "Managing Kernel Resources" - let nr_hugepages = shared_buffers_mb / 2 + 32; - let hugepages_reserved = - match std::fs::write("/proc/sys/vm/nr_hugepages", format!("{nr_hugepages}\n")) { - Ok(()) => { - info!( - "reserved {nr_hugepages} hugepages ({} MB)", - nr_hugepages * 2 - ); - true - } + apply_hugepages(shared_buffers_mb); +} + +/// Reserve the hugepages that must back `shared_buffers_mb`, and keep the +/// `huge_pages = try` fallback file in step with whether that succeeded. +/// +/// Called at boot, and again by the supervisor's retune cycle when virtio-mem +/// grows RAM enough to move `shared_buffers` — the reservation has to be resized +/// *before* the postmaster restarts into the new `shared_buffers`, or it won't +/// come back up. +/// +/// Percona benchmark: TLB faults drop from ~200k/s to near zero with hugepages. +/// Ref: Percona "Benchmark PostgreSQL with Linux HugePages"; +/// PostgreSQL docs §19.4 "Managing Kernel Resources" +pub fn apply_hugepages(shared_buffers_mb: u64) { + let reserved = reserve_hugepages(config::nr_hugepages_for(shared_buffers_mb)); + + // `01-tuning.conf` hardcodes `huge_pages = on`: the right production default, + // because postgres then fails fast rather than silently giving up the TLB win. + // But `on` also means postgres REFUSES TO START if the reservation came up + // short — so whenever it does, we must override to `try` via a high-numbered + // conf.d file, or the postmaster never comes back. + // + // Two ways it comes up short: an environment where we can't write the sysctl + // at all (unprivileged containers, locked-down dev hosts), and a reservation + // sized against hot-added RAM, which lands in ZONE_MOVABLE and may not be able + // to back hugepages at all. Both are handled here, and the + // override is REMOVED again once a reservation succeeds, so a transient + // shortfall doesn't strand us on `try` forever. + let override_path = format!("{PGDATA}/conf.d/99-hugepages-fallback.conf"); + if reserved { + match std::fs::remove_file(&override_path) { + Ok(()) => info!("hugepages reserved; cleared {override_path}"), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => warn!("could not remove {override_path}: {e}"), + } + return; + } + let body = "# Generated automatically when the nr_hugepages reservation came\n\ + # up short (boot::apply_hugepages). Postgres tries hugepages and\n\ + # falls back to anonymous shmem rather than refusing to start.\n\ + huge_pages = try\n"; + match config::write_atomic(Path::new(&override_path), body) { + Ok(()) => info!("huge_pages override written to {override_path} (fallback to try)"), + Err(e) => warn!("could not write huge_pages override: {e}"), + } +} + +const NR_HUGEPAGES: &str = "/proc/sys/vm/nr_hugepages"; + +/// Reserve `want` 2 MB hugepages, returning true only if the kernel actually +/// handed over every one of them. +/// +/// Writing `nr_hugepages` is a *request*, not a guarantee: hugetlb pages must be +/// physically contiguous, so the kernel reserves as many as it can and silently +/// clamps the rest. The write returns `Ok(())` either way. Checking only the +/// write result — as this did before — treats a partial reservation as a full +/// one, and `huge_pages = on` then makes postgres refuse to start, because it +/// cannot map its whole shared-memory segment. +/// +/// This matters more than it looks here. The VM boots with only part of its RAM +/// online; memory hot-added later lands in `ZONE_MOVABLE`, which the kernel keeps +/// for migratable allocations so the memory can be removed again. A hugepage +/// reservation sized against grown RAM can therefore come up short even though +/// the RAM is "there". Read the number back and believe the kernel, not the +/// request. +fn reserve_hugepages(want: u64) -> bool { + if let Err(e) = std::fs::write(NR_HUGEPAGES, format!("{want}\n")) { + warn!("could not set nr_hugepages: {e}"); + return false; + } + let got = match std::fs::read_to_string(NR_HUGEPAGES) { + Ok(s) => match s.trim().parse::() { + Ok(n) => n, Err(e) => { - warn!("could not set nr_hugepages: {e}"); - false + warn!("could not parse nr_hugepages read-back: {e}"); + return false; } - }; - - // The tuning conf hardcodes `huge_pages = on`. That's the right - // production default — postgres fails fast if its hugepage reservation - // wasn't successfully provisioned, rather than silently falling back - // to regular pages and giving up the TLB win. In environments where - // we *couldn't* reserve hugepages (unprivileged containers, dev hosts - // with locked-down sysfs), forcing `on` would make postgres refuse to - // start. Detect that and override with `try` via a high-numbered - // conf.d file so postgres still boots cleanly. - if !hugepages_reserved { - let override_path = format!("{PGDATA}/conf.d/99-hugepages-fallback.conf"); - let body = "# Generated automatically when nr_hugepages reservation\n\ - # failed at boot (apply_kernel_settings). Postgres tries\n\ - # hugepages and falls back to anonymous shmem if unavailable.\n\ - huge_pages = try\n"; - match config::write_atomic(Path::new(&override_path), body) { - Ok(()) => info!("huge_pages override written to {override_path} (fallback to try)"), - Err(e) => warn!("could not write huge_pages override: {e}"), + }, + Err(e) => { + warn!("could not read back nr_hugepages: {e}"); + return false; } + }; + if got < want { + warn!( + "hugepage reservation short: asked {want} ({} MB), kernel gave {got} ({} MB) \ + — falling back to huge_pages=try so postgres still starts", + want * 2, + got * 2 + ); + return false; } + info!("reserved {got} hugepages ({} MB)", got * 2); + true } // --------------------------------------------------------------------------- diff --git a/src/config.rs b/src/config.rs index 6bb5338..dc03b35 100644 --- a/src/config.rs +++ b/src/config.rs @@ -291,8 +291,30 @@ pub fn pgbouncer_ini( // Generated: 01-tuning.conf (postmaster-context — restart required) // --------------------------------------------------------------------------- +/// `shared_buffers`, in MB, for a VM with `ram_bytes` of visible RAM. +/// +/// 25% of RAM, floor 128MB. Single source of truth: the postmaster config, the +/// hugepage reservation that must back it (`boot::apply_hugepages`), and the +/// memory watcher's "is a retune worth a restart?" test all derive from this, and +/// they must not be able to drift apart. +/// +/// Ref: PostgreSQL docs §20.4; pganalyze shared_buffers benchmark (2024) — +/// 25% is optimal up to ~64 GB; gains plateau above 40% due to double-buffering. +pub fn shared_buffers_mb(ram_bytes: u64) -> u64 { + (ram_bytes / (1024 * 1024) / 4).max(128) +} + +/// Number of 2 MB hugepages needed to back `shared_buffers_mb`, plus 32 pages +/// (64 MB) of headroom for WAL buffers and the rest of the shared-memory segment. +pub fn nr_hugepages_for(shared_buffers_mb: u64) -> u64 { + shared_buffers_mb / 2 + 32 +} + /// Postmaster-context parameters — require a Postgres restart to take effect. -/// Written once at boot; not touched by the memory watcher. +/// +/// Written at boot, and rewritten by the memory watcher's retune cycle when +/// virtio-mem changes visible RAM enough to move `shared_buffers` — the postmaster +/// is then restarted behind a pgbouncer PAUSE so the new values take effect. pub fn tuning_conf_boot(ram_bytes: u64, vcpus: u32) -> String { let ram_mb = ram_bytes / (1024 * 1024); let pool_size = pgbouncer_pool_size(vcpus); // server connections PgBouncer opens @@ -307,10 +329,7 @@ pub fn tuning_conf_boot(ram_bytes: u64, vcpus: u32) -> String { // Ref: PostgreSQL wiki "Number Of Database Connections"; Mattermost PgBouncer study let max_connections = (pool_size + vcpus * 2 + 10).clamp(100, (ram_mb / 50).max(100)); - // shared_buffers: 25% of RAM, floor 128MB. - // Ref: PostgreSQL docs §20.4; pganalyze shared_buffers benchmark (2024) — - // 25% is optimal up to ~64 GB; gains plateau above 40% due to double-buffering. - let shared_buffers_mb = (ram_mb / 4).max(128); + let shared_buffers_mb = shared_buffers_mb(ram_bytes); // wal_buffers: replicates Postgres auto-tune (wal_buffers=-1 = shared_buffers/32). // 16 MB ceiling is the historical single-WAL-segment size and is sufficient for diff --git a/src/supervisor.rs b/src/supervisor.rs index cdbbc5d..f3a5be7 100644 --- a/src/supervisor.rs +++ b/src/supervisor.rs @@ -734,8 +734,17 @@ async fn run_inner_inner(role: MaybeRole) -> Result<(), Box(1); + let memory_watcher_handle = tokio::spawn(memory_watcher_task( + cfg.vcpus, + cfg.ram_bytes, + retune_tx, + )); // Background: watch the platform TLS cert for rotation, if applicable. // The guest agent atomically replaces /run/beyond/tls/cert.pem every ~22h. @@ -777,6 +786,9 @@ async fn run_inner_inner(role: MaybeRole) -> Result<(), Box = None; + // Main supervision loop loop { // Both children must have a live Child handle to select on. @@ -916,6 +928,31 @@ async fn run_inner_inner(role: MaybeRole) -> Result<(), Box {} } } + // virtio-mem moved RAM far enough that shared_buffers must change. + // Only recorded here — the postgres-exit arm above holds `&mut pg_state` + // for the lifetime of this select!, so the restart runs once it ends. + Some(ram_bytes) = retune_rx.recv() => { + pending_retune = Some(ram_bytes); + } + } + + if let Some(ram_bytes) = pending_retune.take() { + if let Err(e) = retune_postmaster( + ram_bytes, + cfg.vcpus, + &mut pg_state, + &pgb_state, + &pgb_extra, + &log_tx, + ) + .await + { + // Postgres is either back up or in the normal crash-restart path; + // either way this loop stays alive and keeps supervising. + error!("retune failed: {e}"); + } + #[cfg(target_os = "linux")] + persist_child(&mut persisted, &pg_state)?; } } } @@ -944,6 +981,32 @@ fn sighup_pgbouncer(state: &ChildState) { } } +/// PAUSE a PgBouncer worker (SIGUSR1 — "same as issuing PAUSE on the console"). +/// +/// PAUSE disconnects each server connection *after waiting for it to be released* +/// per the pool mode, and new client queries then **queue** rather than error +/// ("new client connections to a paused database will wait until RESUME"). That +/// is what lets us restart the postmaster underneath live clients: they see a +/// stall, not a dropped connection. +fn pause_pgbouncer(state: &ChildState) { + signal_pgbouncer(state, libc::SIGUSR1, "PAUSE"); +} + +/// RESUME a paused PgBouncer worker (SIGUSR2). Must run on every path out of a +/// retune — a pooler left paused is an outage. +fn resume_pgbouncer(state: &ChildState) { + signal_pgbouncer(state, libc::SIGUSR2, "RESUME"); +} + +fn signal_pgbouncer(state: &ChildState, sig: i32, what: &str) { + let Some(pid) = state.pid() else { + info!("pgbouncer: no live child, {what} skipped"); + return; + }; + send_signal_to_pid(pid, sig); + info!("pgbouncer: {what} sent (pid={pid})"); +} + /// Send SIGINT to a PgBouncer worker for a *graceful* shutdown ("safe shutdown": /// finish in-flight transactions, then exit). Used by the scaler to reap an extra /// so_reuseport worker — measured ~0.03% reconnect blip as the kernel re-routes @@ -1887,14 +1950,60 @@ pub(crate) const OPTIONAL_EXTENSIONS: &[&str] = &[ "hypopg", "pg_repack", "postgis", + // pg_prewarm: the SQL side (manual `pg_prewarm('tbl')`). The part that + // actually matters — the autoprewarm background worker — comes from the + // `shared_preload_libraries` entry in 00-beyond.conf, not from this. + "pg_prewarm", ]; // --------------------------------------------------------------------------- -// Memory watcher — updates reload-safe tuning on virtio-mem hotplug +// Memory watcher — follows virtio-mem RAM changes across BOTH tuning halves // --------------------------------------------------------------------------- +// +// The VM's memory is elastic: it boots with only part of its RAM online and the +// rest is hot-plugged in as the guest comes under pressure. A guest that only +// reads its RAM at boot is permanently mis-tuned for the size it ends up at. +// +// The tuning splits by *how* a parameter can change: +// +// 02-memory.conf reload-safe (effective_cache_size, work_mem, ...) — a +// SIGHUP is enough, so the watcher applies these in place. +// 01-tuning.conf postmaster-context (shared_buffers above all) — postgres +// fixes these at startup and cannot be talked out of them. +// The ONLY way to follow RAM here is to restart the postmaster, +// which the watcher asks the supervisor loop to do. +// +// The restart is hidden behind a pgbouncer PAUSE, so clients stall rather than +// see errors. It is gated by hysteresis, because it is not free. + +/// How far `shared_buffers` must move before a restart is worth it (±25%). +const RETUNE_MIN_RATIO: f64 = 1.25; +/// Floor on the interval between retune cycles, so a flapping balloon can't put +/// the postmaster into a restart loop. +const RETUNE_MIN_INTERVAL: Duration = Duration::from_secs(120); +/// Grace for pgbouncer to finish draining server connections after PAUSE. +const RETUNE_PAUSE_GRACE: Duration = Duration::from_secs(3); +/// How long to wait for the restarted postmaster before resuming pgbouncer anyway. +const RETUNE_READY_TIMEOUT: Duration = Duration::from_secs(60); + +/// True when RAM has moved enough to shift `shared_buffers` by >= `RETUNE_MIN_RATIO` +/// in either direction — the test for "is this worth a postmaster restart?". +fn shared_buffers_moved_enough(tuned_ram: u64, new_ram: u64) -> bool { + let old = config::shared_buffers_mb(tuned_ram) as f64; + let new = config::shared_buffers_mb(new_ram) as f64; + if old <= 0.0 { + return false; + } + let ratio = new / old; + ratio >= RETUNE_MIN_RATIO || ratio <= 1.0 / RETUNE_MIN_RATIO +} -async fn memory_watcher_task(vcpus: u32, initial_ram_bytes: u64) { +async fn memory_watcher_task(vcpus: u32, initial_ram_bytes: u64, retune_tx: mpsc::Sender) { let mut last_ram_bytes = initial_ram_bytes; + // RAM that the *running postmaster's* postmaster-context params were derived + // from. Only a completed retune moves this. + let mut tuned_ram_bytes = initial_ram_bytes; + let mut last_retune: Option = None; loop { tokio::time::sleep(Duration::from_secs(5)).await; let ram_bytes = match read_memtotal_bytes().await { @@ -1907,6 +2016,8 @@ async fn memory_watcher_task(vcpus: u32, initial_ram_bytes: u64) { continue; } last_ram_bytes = ram_bytes; + + // --- reload-safe half: apply in place --- let content = config::tuning_conf_adaptive(ram_bytes, vcpus); match config::write_atomic(std::path::Path::new(&config::memory_conf_path()), &content) { Ok(()) => info!( @@ -1922,7 +2033,114 @@ async fn memory_watcher_task(vcpus: u32, initial_ram_bytes: u64) { Ok(()) => info!("memory watcher: reloaded postgres config"), Err(e) => warn!("memory watcher: pg_reload_conf failed: {e}"), } + + // --- postmaster-context half: needs a restart, so ask the loop for one --- + if !shared_buffers_moved_enough(tuned_ram_bytes, ram_bytes) { + continue; + } + if last_retune.is_some_and(|t| t.elapsed() < RETUNE_MIN_INTERVAL) { + info!("memory watcher: retune due but within cooldown; deferring"); + continue; + } + info!( + "memory watcher: RAM {} MB -> shared_buffers would move {} MB -> {} MB; requesting retune", + ram_bytes / (1024 * 1024), + config::shared_buffers_mb(tuned_ram_bytes), + config::shared_buffers_mb(ram_bytes), + ); + match retune_tx.try_send(ram_bytes) { + Ok(()) => { + tuned_ram_bytes = ram_bytes; + last_retune = Some(Instant::now()); + } + // Full = a retune is already queued; dropping this one is correct, + // the next tick re-evaluates against the RAM we actually end up with. + Err(e) => warn!("memory watcher: could not queue retune: {e}"), + } + } +} + +/// Re-derive the postmaster-context tuning for `ram_bytes` and restart postgres +/// into it, hiding the restart behind a pgbouncer PAUSE. +/// +/// Ordering is load-bearing: +/// 1. write `01-tuning.conf` — what the postmaster will read on the way up +/// 2. resize the hugepage reservation — `huge_pages = on` means postgres will +/// REFUSE TO START if the pages aren't there, so this must precede the stop +/// 3. PAUSE pgbouncer — drains in-flight transactions, queues new clients +/// 4. fast-shutdown + respawn the postmaster +/// 5. RESUME pgbouncer — on EVERY path out, including failure. A pooler left +/// paused is an outage, which would be a worse bug than the one we're fixing. +async fn retune_postmaster( + ram_bytes: u64, + vcpus: u32, + pg: &mut ChildState, + pgb: &ChildState, + pgb_extra: &[ChildState], + log_tx: &mpsc::Sender, +) -> Result<(), Box> { + let sb_mb = config::shared_buffers_mb(ram_bytes); + info!( + "retune: visible RAM {} MB -> shared_buffers {} MB", + ram_bytes / (1024 * 1024), + sb_mb + ); + + config::write_atomic( + std::path::Path::new(&config::tuning_conf_path()), + &config::tuning_conf_boot(ram_bytes, vcpus), + )?; + crate::boot::apply_hugepages(sb_mb); + + // Postgres is down and in restart backoff: the config is on disk, so the + // pending respawn picks it up. Nothing to pause, nothing to restart. + if pg.handle.is_none() { + info!("retune: postgres not running; new config will apply on its next start"); + return Ok(()); + } + + pause_pgbouncer(pgb); + for w in pgb_extra { + pause_pgbouncer(w); + } + tokio::time::sleep(RETUNE_PAUSE_GRACE).await; + + let result = restart_postmaster(pg, log_tx).await; + + resume_pgbouncer(pgb); + for w in pgb_extra { + resume_pgbouncer(w); + } + + match &result { + Ok(()) => info!("retune: complete, shared_buffers now {sb_mb} MB"), + Err(e) => error!("retune: failed ({e}); pgbouncer resumed"), + } + result +} + +/// Fast-shutdown postgres and bring it straight back up. No restart backoff: this +/// is a deliberate restart, not a crash. +async fn restart_postmaster( + pg: &mut ChildState, + log_tx: &mpsc::Sender, +) -> Result<(), Box> { + // SIGINT = fast shutdown. pgbouncer has already released the server + // connections, so there is no in-flight transaction left to abort. + if let Some(pid) = pg.pid() { + info!("retune: fast-shutdown postgres (pid={pid})"); + send_signal_to_pid(pid, libc::SIGINT); + } + if let Some(ref mut handle) = pg.handle { + let code = handle.wait().await?; + pg.on_exit(code); + } + + spawn_postgres(pg, log_tx)?; + if !pg::wait_until_ready(RETUNE_READY_TIMEOUT).await { + return Err("postgres did not become ready after retune".into()); } + Ok(()) } async fn read_memtotal_bytes() -> Option { @@ -1947,6 +2165,43 @@ fn parse_memtotal_kib(content: &str) -> Option { mod tests { use super::*; + // --- retune hysteresis (pure) --- + + const MIB: u64 = 1024 * 1024; + + #[test] + fn retune_fires_when_the_substrate_plugs_real_memory() { + // The case that matters: booted with a fraction of its RAM online, then + // grown. shared_buffers 183MB -> 512MB is worth a restart. + assert!(shared_buffers_moved_enough(735 * MIB, 2016 * MIB)); + } + + #[test] + fn retune_declines_a_move_too_small_to_pay_for_a_restart() { + // A single small hotplug step: 735 -> 880 MiB moves shared_buffers + // 183MB -> 220MB, only ~20%. Not worth bouncing the postmaster. + assert!(!shared_buffers_moved_enough(735 * MIB, 880 * MIB)); + } + + #[test] + fn retune_fires_on_shrink_too() { + // Memory unplugged back to base: shared_buffers must come down, or the + // postmaster is sized for RAM the guest no longer has. + assert!(shared_buffers_moved_enough(2016 * MIB, 735 * MIB)); + } + + #[test] + fn retune_ignores_noise() { + assert!(!shared_buffers_moved_enough(2016 * MIB, 2020 * MIB)); + } + + #[test] + fn retune_respects_the_shared_buffers_floor() { + // Below the 128MB floor both sides clamp equal, so there is nothing to do + // and we must not restart in a loop over it. + assert!(!shared_buffers_moved_enough(64 * MIB, 200 * MIB)); + } + // --- post_start batch script (pure) --- fn pg_cfg(replication_password: Option<&str>, wal_sink: Option<&str>, cdc: bool) -> MmdsConfig {