Skip to content

[Bug] Proxy bootstrap panics instead of retrying transient HAKeeper errors #25737

Description

@Ariznawlll

Summary

Proxy bootstrap has a five-minute retry loop, but any single GetClusterState error panics immediately. A temporary HAKeeper timeout or network interruption can therefore terminate Proxy instead of being retried.

Affected revision

  • Branch: test/issues-24122-21685-4.0-dev
  • Commit: 7f7de555c49acc3a64995b078025f5526afc9c34

Root cause

pkg/proxy/server.go starts bootstrap asynchronously with go h.bootstrap(ctx). Inside pkg/proxy/bootstrap.go:

for retry < int(BootstrapTimeout/BootstrapInterval) {
    state, err = h.haKeeperClient.GetClusterState(ctx)
    if err != nil {
        panic(moerr.AttachCause(ctx, err))
    }
}

The panic makes the surrounding retry budget ineffective. An unrecovered panic in this goroutine terminates the process.

Actual behavior

The first transient HAKeeper error during task-table credential discovery causes a panic. The configured five-minute bootstrap retry window is not used.

Expected behavior

Retry transient errors until bootstrap succeeds, the parent context is canceled, or the bootstrap timeout expires. Report a controlled startup failure at timeout rather than panicking from an unmanaged goroutine.

Suggested regression test

Use a mock HAKeeper client that fails the first request and succeeds on the second. Assert bootstrap completes without panic. Add cancellation and permanent-failure cases.

Relevant code

  • pkg/proxy/bootstrap.go:32-64
  • pkg/proxy/server.go:108

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingneeds-triageNeeds evaluation before prioritization. Not yet decided whether to proceed

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions