fix: reuse CN pipeline connections after clean teardown

[aptend]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Closes #25690

Related to #25691.

What this PR does / why we need it:

Remote pipeline completion currently couples a logical stream teardown to destruction of its morpc backend and CN-to-CN TCP connection. High-fanout distributed queries therefore create enough connection churn and TIME_WAIT tuples to exhaust Linux ephemeral ports even though the CN processes remain healthy.

This PR adds a negotiated FIN/FIN_ACK lifecycle for successfully drained pipeline streams:

• the client requests FIN/ACK teardown per stream and falls back to the legacy close-connection path when the server does not accept it;
• the server acknowledges FIN only after pipeline and dispatch cleanup crosses a completion barrier;
• morpc atomically retires sent/received sequence state before unlocking the still-exclusive backend for reuse;
• uncertain FIN delivery, timeout, malformed sequencing, unread data, cancellation, and ambiguous errors poison and close the backend;
• lifecycle, FIN/ACK, reuse, close-reason, and sequence-container metrics make teardown outcomes observable;
• disable-stream-reuse provides a rollback switch.

The connection remains exclusively locked while a logical stream is active. This PR does not multiplex active streams on one TCP connection, add query-boundary admission, or change the existing maximum-concurrency policy. Those require separate deadlock/fairness design under #25691.

Validation

• go test -count=1 ./pkg/common/morpc ./pkg/cnservice ./pkg/sql/compile
• targeted FIN/ACK, lifecycle, and negotiated teardown tests with -race
• make -j8 build
• macOS two-CN reproduction: two consecutive 400-statement waves completed 400/400 with result 499999; CN pipeline TIME_WAIT remained zero
• Linux amd64 with independent log/TN/CN1/CN2 processes: two consecutive 400-statement waves completed 400/400; established socket endpoints plateaued at 2,778 then 2,814; TIME_WAIT remained zero during load; idle GC returned established pipeline sockets to zero and CN file descriptors from about 2,895 to 63
• both CNs remained healthy, and the exact join still returned 499999 after idle GC

The design, failure matrix, rollout/rollback plan, and detailed acceptance records are included in docs/design/cn_pipeline_connection_lifecycle.md.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[aptend]

Coverage gap fixed in ea7271bffa. The previous artifact showed remoterun_lifecycle.go at 29.2% (19/65 changed statements), and the existing “without token” test was actually stopped earlier by teardown-mode validation. The new tests separately cover malformed FIN, missing validated token, missing lifecycle, non-finisher session, invalid response, successful FIN_ACK, FinishStream failure, cleanup timeout, message cancellation, connection close, and idempotent cleanup. Focused coverage now reports 91.3% for remoterun_lifecycle.go (63/69 statements), projecting overall patch coverage from 67.3% to about 90.9%. Full pkg/common/morpc, pkg/cnservice, and pkg/sql/compile tests, focused race tests, build, and vet pass. The branch was also merged with current main ddbff5b400 before the coverage change.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-07-16 03:49 UTC · Rule: main · triggered by rule Automatic queue on approval for main
• ✅ Checks skipped · PR is already up-to-date
• ✅ Merged — 2026-07-16 03:49 UTC · at ea7271bffaaa4ce5ba355082fe563b8d3ff5c4f3 · squash

This pull request spent 17 seconds in the queue, including 3 seconds running CI.

Required conditions to merge

• #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • fix: reuse CN pipeline connections after clean teardown #25713
• github-review-approved [🛡 GitHub branch protection]
  • fix: reuse CN pipeline connections after clean teardown #25713
• github-review-decision = APPROVED [🛡 GitHub branch protection]
  • fix: reuse CN pipeline connections after clean teardown #25713
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-neutral = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-skipped = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / UT Test on Ubuntu/x86
  • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
  • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Utils CI / Coverage
  • check-neutral = Matrixone Utils CI / Coverage
  • check-skipped = Matrixone Utils CI / Coverage
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / SCA Test on Linux/arm64
  • check-neutral = Matrixone CI / SCA Test on Linux/arm64
  • check-skipped = Matrixone CI / SCA Test on Linux/arm64