fix(proxy): bound connection memory and admission

[XuPeng-SH]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Fixes #25856.

Depends on #25857. This is a stacked PR and should be retargeted to main after #25857 merges.

What this PR does / why we need it:

The Proxy previously paid two eager 1 MiB frontend.IOSession buffers per live proxied connection, created one managed allocator per connection, retained an unbounded login packet for migration, and had no global or per-tenant admission boundary. Raising a memory limit alone would only delay exhaustion.

This PR closes the resource lifecycle at multiple layers:

• reduces the two retained Proxy protocol buffers from about 2 MiB to 32 KiB per active connection (16 KiB on each side);
• shares one size-bounded protocol allocator across client, backend, migration, and cached backend sessions;
• adds global and per-tenant live-connection admission with idempotent leases;
• rejects global overload before per-connection protocol allocation with MySQL error 1040;
• caps retained client login packets at 64 KiB and rejects them from the packet header before decoding;
• bounds the complete unauthenticated handshake lifecycle, including TLS, and clears the read deadline before entering the raw tunnel;
• rejects duplicate PROXY protocol headers so they cannot extend the handshake or grow recursion;
• validates connection, protocol-memory, handshake-size, and handshake-timeout configuration as one coherent budget;
• records expected overload as reject metrics without error-log amplification.

The SQL/tunnel data path is unchanged: the limiter lock is touched only during acquire, tenant bind, and release, and no query/transaction idle deadline is introduced. This avoids regressions for long HTAP queries.

Test matrix

• full ./pkg/frontend and ./pkg/proxy unit suites;
• go build ./pkg/frontend ./pkg/proxy;
• go vet ./pkg/frontend ./pkg/proxy;
• new boundary/concurrency tests repeated 100 times;
• full ./pkg/proxy race suite;
• focused frontend race tests.

Covered unhappy paths include allocator exhaustion and recovery, exact-fill and oversized packets, global and tenant saturation, concurrent bind/release, double release, constructor/TLS failure cleanup, silent-client timeout, deadline clearing after successful login, standard client-visible errors, and bounded rejection writes.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[LeftHandCold]

Requesting changes for one availability bypass and three concrete overload/protocol regressions. I reviewed only the stacked delta from da2e56c to f259642. The full proxy test suite, focused race tests, build, and vet pass; additional focused diagnostics reproduced the absolute-deadline bypass and the incorrect ERR packet sequence.

[LeftHandCold]

Re-reviewed the latest head. The four findings from my previous review are fixed. Requesting changes for one newly confirmed PROXY v2 protocol regression. The full proxy/frontend test suites, the full proxy race suite, build, vet, and diff checks pass; a focused diagnostic reproduces this failure deterministically.

[gouhongshen]

Codex automated review

The new admission and memory limits have a concrete OOM gap, and duplicate PROXY v2 detection is incomplete.

P1 - Include retained handshake storage in the memory budget (pkg/proxy/config.go:348)

Validation accounts only for the shared 16 KiB client/backend session buffers. The goetty input buffer and the copied handshakePack.Payload are outside that allocator but remain retained for authenticated connections. With the defaults, 10,000 connections may each retain a valid 64 KiB login twice, already exceeding 1 GiB before the fixed session buffers and tunnel state, so ProtocolMemoryLimit does not prevent OOM.

P2 - Mark consumed LOCAL/UNSPEC PROXY frames for duplicate detection (pkg/proxy/ppv2.go:194)

For valid unspec/unix PROXY v2 frames, the parser consumes the frame but returns ok=false. The wrapper then delegates to the SQL codec, and readPacketBefore never sets proxyHeaderReceived. A LOCAL/UNSPEC frame followed by an address-bearing frame in a later read therefore bypasses duplicate rejection and can overwrite origin metadata used for internal classification and routing.

[LeftHandCold]

Re-reviewed the latest head and approving. The previously reported fragmented PROXY v2 regression is fixed: matching signature prefixes, partial fixed headers, and partial declared bodies now remain buffered without consumption, while definite mismatches still delegate to the bounded MySQL codec. I found no new blocker in the updated delta. The focused fragmentation test passed 100 iterations; the full proxy suite, full proxy race suite, build, vet, and diff checks also pass.

[XuPeng-SH]

Deep-review blocker (GitHub does not allow the PR author to submit a formal REQUEST_CHANGES review on their own PR):

[P1] Reject handshake packet limits that cannot admit a final login

minimumClientHandshakePacketLimit is currently 32 because a protocol 4.1 SSLRequest is 32 bytes, and Config.Validate therefore accepts limits of 32 or 33. However, 32 bytes is only the fixed protocol-4.1 prefix. A final authentication response must additionally contain at least the NUL-terminated username and an auth length/terminator. On TLS connections the 32-byte SSLRequest is followed by that larger final login, and the same SQL codec limit applies to both phases.

I reproduced this at the wire level on the current head: a config with client-handshake-packet-limit = 32 passes FillDefault + Validate, then a valid login is rejected with frontend.ErrPacketTooLarge. Thus an explicitly accepted configuration deterministically makes the proxy unable to authenticate any real client.

Please model the minimum for a complete login rather than the SSLRequest-only size (or otherwise reject unusable limits), and add a boundary test that the smallest configuration accepted by Validate can pass the packet-size gate for a final protocol-4.1 login.

[XuPeng-SH]

@gouhongshen Addressed the latest handshake-limit blocker and the related phase-model gaps in a569dfa; the branch is also merged with the latest main at 80668d9.

Root-cause classification:

• The 32-byte minimum was a configuration-contract bug: 32 bytes describes only the intermediate protocol-4.1 SSLRequest, not a usable terminal login. The accepted minimum is now derived as 32-byte fixed prefix + one-byte username + NUL + one auth length/terminator byte (35 bytes), with exact min/max wire and validation boundaries.
• The recursive TLS path exposed a broader protocol-state-machine gap. Handshake processing is now an explicit bounded phase loop: pre-TLS -> at most one TLS upgrade -> final login. A repeated SSLRequest is rejected instead of recursively layering TLS/stack/CPU work.
• Validation now mirrors runtime composition: plugin routing disables the connection cache, so that mode no longer reserves 1,024 cache sessions that can never exist.

Unhappy-path ownership coverage now proves:

• the intermediate SSLRequest allocation is released before both successful and failed TLS negotiation;
• a partially parsed final login transfers cleanup ownership and is released exactly once by Close;
• duplicate TLS requests are rejected promptly without recursion;
• plaintext and TLS logins at the exact accepted minimum complete parsing;
• allocator, admission, timeout, pipelined-prefix, double-close, and concurrent reader/Close paths remain covered.

Performance: the production delta is limited to startup validation and the unauthenticated handshake. The recursion-to-loop change adds only a stack boolean and performs the same packet parse/allocation work for the normal one- or two-packet flow. It does not touch the established tunnel/query hot path.

Fresh validation after merging latest main:

• focused phase/config tests: 20x
• full pkg/proxy
• full pkg/proxy with -race
• go build -mod=readonly ./pkg/proxy
• go vet -mod=readonly ./pkg/proxy
• make static-check
• git diff --check

During the unhappy-path test addition, it also exposed and fixed the existing "TSL handshake error" typo so production diagnostics now consistently say TLS.

[XuPeng-SH]

Deep self-review found and fixed one additional admission trust-boundary issue in 584ff05.

Root cause:

• The global slot is intentionally acquired before expensive per-connection state, but the per-tenant slot was previously bound as soon as the login packet revealed a tenant name.
• Before backend authentication that name is only an untrusted client claim. An unauthenticated client could repeatedly claim another tenant and occupy the target tenant quota for the backend-auth window, causing targeted admission denial without owning any credentials.

The lifecycle is now:

global admission -> parse claimed tenant -> authenticate via fresh backend or cache authenticator -> bind tenant admission -> expose backend OK to client

This preserves the global memory/work bound during authentication while making tenant fairness depend on an authenticated identity. On tenant saturation, a freshly authenticated backend or cache hit is closed before any OK packet reaches the client. Backend authentication failures consume no tenant quota. Expected admission rejection remains debug-level and uses the existing reject metric, avoiding error-log amplification.

Added regression coverage for:

• two unauthenticated connections claiming the same tenant without consuming its authoritative quota;
• fresh-backend success binding before client OK;
• fresh-backend tenant rejection closing the backend before client OK;
• backend auth failure leaving tenant quota untouched;
• authenticated cache-hit rejection and cleanup.

Validation on the pushed commit:

• focused admission/lifecycle tests: 100x;
• focused race tests: 20x;
• full pkg/proxy;
• full pkg/proxy with -race;
• go build -mod=readonly ./pkg/proxy;
• go vet -mod=readonly ./pkg/proxy;
• make static-check;
• git diff --check.

Performance remains outside the established tunnel/query hot path: one existing limiter mutex operation moves from pre-auth parsing to the successful authentication boundary; rejected unauthenticated claims no longer contend on per-tenant counters.

[LeftHandCold]

Re-reviewed the latest head carefully and approving.\n\nI re-audited the new handshake lifecycle and admission boundary from first principles:\n- retained auth payload ownership is closed: backend connects hold a read lease, timeout closes the transport and joins the worker before returning, and client Close frees the allocator-backed payload exactly once;\n- coalesced plaintext/TLS bytes are handed to the tunnel without loss or stale ByteBuf access;\n- SSLRequest is phase-local, duplicate TLS requests are bounded/rejected, and the final login remains subject to the absolute handshake deadline and packet limit;\n- per-tenant admission is bound only after fresh-backend or cache authentication and before the backend OK is exposed; rejection closes the accepted backend and does not consume unauthenticated tenant quota;\n- PROXY v2 fragmentation/LOCAL/UNSPEC handling, oversized ERR sequencing, shared-memory accounting, cache and migration paths have no new reachable regression found.\n\nFresh validation on this exact head:\n- focused lifecycle/admission/TLS/PROXY/timeout tests: 10x;\n- focused ownership tests with : 10x;\n- full FAIL github.com/matrixorigin/matrixone/pkg/proxy [build failed]
FAIL;\n- full FAIL github.com/matrixorigin/matrixone/pkg/proxy [build failed]
FAIL;\n- ;\n- ;\n- dependent frontend SQL codec test;\n- .\n\nNo new realistic blocker found.

[LeftHandCold]

Correction to my immediately preceding approval: its Markdown command names were expanded by the local shell while submitting the review body. The two displayed build-failed lines were submission-formatting artifacts, not validation results.

This approval is for head 584ff05. The actual fresh validation all passed:

• focused lifecycle, admission, TLS, PROXY, timeout, and packet-sequence tests, 10 iterations;
• focused ownership tests under the race detector, 10 iterations;
• full pkg/proxy test suite;
• full pkg/proxy suite under the race detector;
• pkg/proxy build and vet;
• dependent frontend SQL codec test;
• diff check.

The code-review conclusion is unchanged: retained handshake ownership, timeout worker termination, pipelined plaintext and TLS handoff, authenticated tenant binding, packet bounds, cache and migration interactions form a closed lifecycle. I found no new realistic blocker on this head.

[gouhongshen]

Codex automated review

Valid configurations can fail normal migration/control-plane operations because peak shared allocations exceed the validated budget. The fragmented PROXY-v2 path also bypasses the configured handshake memory limit.

P1 - Account for the transient third protocol session (pkg/proxy/config.go:357)

Validation reserves only maxConnections*2 fixed sessions, but migration allocates the replacement backend session before replaceServerConn closes the old one (tunnel.go:491-496, tunnel.go:371-374). The same third-session peak occurs for connAndExec during upgrade/kill. A configuration accepted by the exact-boundary test can therefore reach the allocator limit and return out of space during legitimate operations. Reserve transient headroom or release/coordinate the old session before allocating its replacement.

P2 - Bound fragmented PROXY-v2 bodies by the handshake budget (pkg/proxy/ppv2.go:158)

The parser now waits for the complete declared body before delegating to the SQL codec. Length is a uint16, so an unauthenticated client can send an UNSPEC frame with a 65535-byte body and make goetty retain roughly 64 KiB outside the shared allocator; line 175 also creates a second temporary copy once complete. ClientHandshakePacketLimit can legally be as low as 35, but it is never applied to this frame, so the new fragmented-header path bypasses the configured handshake memory bound until timeout.