Persistent Supavisor 28P01 on both pooler modes after a verified database password reset

[marcofalace]

Project ref: mhgnxlrqbpbamdlzkvzv
Region: eu-west-3
Existing support ticket: SU-495314

Password authentication persistently fails with 28P01: password authentication failed for user "postgres" on both:

• Session pooler: port 5432
• Transaction pooler: port 6543

The project ref and region have been verified as correct. The postgres role is login-enabled, uses SCRAM-SHA-256 authentication, and has no password expiration.

The database password reset was verified on the PostgreSQL side: the SCRAM fingerprint changed, confirming that the reset was applied. Both pooler modes continue to fail afterward.

Supavisor logs show DbHandler: Auth error and checkout failed. The problem has been reproduced across multiple Supavisor nodes.

Last observed event: 2026-10-05 12:17:54 UTC.

[Sarvagya-24-chaturvedi]

Hi @marcofalace,

The error log DbHandler: Auth error and checkout failed is the key to understanding what is happening here:

This log message is not Supavisor rejecting your client. It means Supavisor itself is failing to authenticate upstream to your PostgreSQL database when checking out a connection from its pool.

Even though you verified the SCRAM fingerprint changed in PostgreSQL, the pooler is out of sync. Here is why this happens and how to resolve it:

---

Root Causes

1. Password reset was performed via direct SQL (ALTER USER):
   • If the password was changed by running ALTER ROLE postgres WITH PASSWORD '...' directly inside Postgres, pg_authid updates (changing the SCRAM hash), but bypasses Supabase's internal control plane.
   • Supavisor fetches tenant credentials from the Supabase management database/vault. If the control plane didn't execute the change, Supavisor continues trying to reach Postgres using the old secret, throwing 28P01 on every checkout.
2. Control Plane ↔ Supavisor Tenant Desync (Dashboard Reset):
   • If you used the Supabase Dashboard, the background worker responsible for updating the Supavisor tenant record across the eu-west-3 cluster either timed out or hung, leaving Supavisor nodes holding stale credentials in memory.

---

Step-by-Step Fix

Step 1: Force a Control Plane Reset from the Dashboard

If you used SQL before, re-apply the password through the Supabase Dashboard so the control plane captures the change:

• Go to Project Settings → Database → Database password → Reset password.

Step 2: Restart the Project to Flush Supavisor's Tenant Cache

A project restart forces the orchestration layer to rebuild container configurations and flush Supavisor's cached tenant credentials for your project ref:

• Go to Project Settings → General → Restart project (or Pause and Unpause).
• Wait 2–3 minutes for all pooler nodes in eu-west-3 to pick up the tenant reload.

Step 3: Verify Tenant Scoping in the Username

Ensure your connection string explicitly specifies the project ref in the user field:

• Correct format for Supavisor: postgres.mhgnxlrqbpbamdlzkvzv
• Host: aws-0-eu-west-3.pooler.supabase.com
• Note: Connecting as plain postgres through the pooler will fail or route to a default tenant.

Step 4: Verify via Direct Connection

To confirm your password and role are 100% operational independent of the pooler:

• Test connecting directly to db.mhgnxlrqbpbamdlzkvzv.supabase.co:5432 (requires IPv6 or IPv4 add-on). If direct connection succeeds but pooler fails, the issue is confirmed to be isolated to the Supavisor tenant cache.

---

Note for Support Ticket SU-495314

If a project restart does not unstick the pooler, update ticket SU-495314 specifying that:

"The project has a tenant credential desync in Supavisor: PostgreSQL credentials work directly, but DbHandler: Auth error and checkout failed indicates Supavisor's internal tenant vault record for ref mhgnxlrqbpbamdlzkvzv is out of sync."

An engineer can run a tenant reload command (supavisor:reload_tenant) to immediately resolve it.