fix(proxy): retry HAKeeper errors during bootstrap

[VioletQwQ-0]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Fixes #25737

What this PR does / why we need it:

Proxy bootstrap had a five-minute retry window, but a single HAKeeper GetClusterState error panicked from an unmanaged goroutine and bypassed that window.

Changes:

• Retry returned GetClusterState errors within the existing five-minute bootstrap window.
• Return controlled cancellation/timeout errors from bootstrap instead of panicking.
• Run bootstrap as a Stopper-managed task so caller and server shutdown cancellation terminate it.
• Add regression coverage for a transient failure followed by success, parent cancellation, actual Stopper shutdown cancellation, and persistent failure/timeout.
• Use generous deadlock guards in asynchronous tests instead of machine-speed assertions.

Semantics:

• Success: credentials are present, the SQL writer and worker configuration is installed, and bootstrap returns nil.
• Cancellation: parent or Stopper cancellation stops the current request and returns the cancellation error without logging a startup failure.
• Transient error: the error is retained for diagnostics and the next interval retries.
• Persistent failure/timeout: the task returns a deadline error joined with the last HAKeeper error, and the managed task logs the controlled failure.

Decision / Non-goals / Risks:

• Proxy startup remains asynchronous. On bootstrap timeout, Proxy remains running, the SQL worker stays unready, and bootstrap does not retry past the existing five-minute window.
• This preserves the current Server/Stopper contract: Stopper tasks do not return errors to NewServer, and changing process/readiness policy is outside this fix.
• MOCluster.Close waiting for an independent first GetClusterDetails refresh is a pre-existing clusterservice lifecycle concern and is not changed here.

Validation:

• go test ./pkg/proxy -run '^TestBootstrap' -count=20
• go test ./pkg/proxy -count=1
• go test -race ./pkg/proxy -run '^TestBootstrap' -count=1
• go vet ./pkg/proxy
• git diff --check upstream/main...HEAD
• mo-self-review full diff gate, including Q1-Q3 lifecycle checks

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

The production change looks sound: transient HAKeeper failures are retried within a real wall-clock bootstrap deadline, request/parent cancellation is propagated, and the task/ticker/error state all have bounded lifecycles. The focused tests, race run, full proxy package, and CI are green.

One blocking regression-coverage gap remains. The lifecycle behavior added in server.go is not tested: TestBootstrapReturnsContextCancellation calls bootstrapWithTimeout with a directly canceled parent context, so it bypasses the new Stopper taskCtx -> context.AfterFunc -> bootstrapCtx chain. If that wiring is removed or broken later, all new tests still pass while Server.Close can fail to terminate an in-flight bootstrap through the intended managed-task path. Please add a deterministic test around the actual Stopper task (or Server.Close): wait until the mock GetClusterState request has started, invoke Stopper.Stop/Close, then prove the request context is canceled, shutdown returns, and the task reaches its terminal state.

Please also make the new tests slow-CI safe while updating them. TestBootstrapReturnsTimeoutAfterPersistentHAKeeperError asserts completion in less than one second, which is a scheduler/performance assertion unrelated to the functional contract and can flake on a loaded runner. Remove that wall-clock assertion and use only a generous timeout as a deadlock guard; likewise, the one-second channel guards in the cancellation test should be generous guards rather than tight scheduling requirements. The 50 ms bootstrap timeout can remain the semantic stimulus—the test should assert DeadlineExceeded plus the retained HAKeeper error, not machine speed.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-07-16 04:06 UTC · Rule: main · triggered by rule Automatic queue on approval for main
• ✅ Checks passed · in-place
• ✅ Merged — 2026-07-16 05:06 UTC · at 1e811f4b30b1472e476e6e6d5348d2378f83e285 · squash

This pull request spent 1 hour 6 seconds in the queue, including 59 minutes 47 seconds running CI.

Required conditions to merge

• #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • fix(proxy): retry HAKeeper errors during bootstrap #25778
• github-review-approved [🛡 GitHub branch protection]
  • fix(proxy): retry HAKeeper errors during bootstrap #25778
• github-review-decision = APPROVED [🛡 GitHub branch protection]
  • fix(proxy): retry HAKeeper errors during bootstrap #25778
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-neutral = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-skipped = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / UT Test on Ubuntu/x86
  • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
  • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Utils CI / Coverage
  • check-neutral = Matrixone Utils CI / Coverage
  • check-skipped = Matrixone Utils CI / Coverage
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / SCA Test on Linux/arm64
  • check-neutral = Matrixone CI / SCA Test on Linux/arm64
  • check-skipped = Matrixone CI / SCA Test on Linux/arm64