Skip to content

fix: update DN/CN HAKeeper config after LogService failover (#596, #597)#599

Merged
loveRhythm1990 merged 1 commit into
matrixorigin:mainfrom
loveRhythm1990:lr90/sync-cm
Jul 2, 2026
Merged

fix: update DN/CN HAKeeper config after LogService failover (#596, #597)#599
loveRhythm1990 merged 1 commit into
matrixorigin:mainfrom
loveRhythm1990:lr90/sync-cm

Conversation

@loveRhythm1990

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

Fixes #596
Fixes #597

Related: #598

What this PR does / why we need it:

After LogService failover (e.g. log-2 fails and is replaced by log-3 via ReserveOrdinals), DN/CN ConfigMaps may keep stale hakeeper-client.service-addresses pointing at the dead pod. This PR fixes two intertwined bugs:

Bug-1 (#597): MO 3.x incorrectly falls back to service-addresses

MOFeatureDiscoveryFixed was only listed under featureVersions with a 2.0.0 entry. Because versionPrecedes() isolates major versions, HasMOFeature(3.x, DiscoveryFixed) always returned false, so MO 3.x DN/CN kept using legacy service-addresses instead of discovery-address.

Fix:

  • Move MOFeatureDiscoveryFixed to a new featureGlobalMinVersions map (stable cross-major config capability since MO 2.0).
  • Add versionPrecedesCrossMajor() for features that do not need per-major re-verification.
  • Keep other features (LockMigration, PipelineInfo, etc.) in featureVersions unchanged.
  • Add regression tests to prevent accidentally widening unrelated 3.x feature gates.

Bug-2 (#596): HaKeeperAdds() ignored ReserveOrdinals

When service-addresses mode is active (MO 1.x / semver unavailable / pre-2.0), DN/CN ConfigMap builders used HaKeeperAdds(), which always generated [log-0, log-1, log-2] and ignored ordinal holes created during failover.

Fix (no CRD change):

  • Add HaKeeperSvcAddrs(ls, reservedOrdinals) mirroring the existing gossipSeeds() hole-skipping logic.
  • Export LogSetStsName() so DN/CN controllers can read LogSet STS spec.reserveOrdinals.
  • Fetch reserved ordinals only when the service-addresses branch is actually used (skip extra STS GET for MO >= 2.0 discovery path).
  • Propagate STS read errors instead of silently falling back to "no holes".
  • Remove obsolete HaKeeperAdds().

Expected behavior after merge

Component MO 2.x MO 3.x
DN/CN ConfigMap service-addresses with correct ordinal holes discovery-address
LogService ConfigMap unchanged start script V2 when semver gate applies

Special notes for your reviewer:

  • No CRD / API change. DN/CN read LogSet StatefulSet directly instead of adding fields to LogSetStatus.
  • Scope control: only MOFeatureDiscoveryFixed uses cross-major global min version; other features are intentionally not extended to 3.x.
  • Conservative error handling: if LogSet STS cannot be read on the service-addresses path, reconcile retries instead of writing a wrong address list.
  • Duplicate helper: fetchLogSetReservedOrdinals exists in both dnset and cnset controllers; acceptable for now, can dedupe later.

Additional documentation (e.g. design docs, usage docs, etc.):


Test plan

  • go test ./api/core/v1alpha1/...
  • go test ./pkg/controllers/dnset/...
  • go test ./pkg/controllers/cnset/...
  • go test ./pkg/controllers/logset/...
  • Deploy operator to a test cluster with MO 3.x DN/CN
  • Verify DN/CN ConfigMap switches to:
    [hakeeper-client]
    discovery-address = "<logset>-discovery.<ns>.svc:32001"

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes DN/CN HAKeeper client configuration becoming stale after LogService failover by (1) making MOFeatureDiscoveryFixed correctly apply to MO 3.x+ via a cross-major “global min version” gate, and (2) generating service-addresses that skip StatefulSet.spec.reserveOrdinals holes when the legacy address path is used.

Changes:

  • Add featureGlobalMinVersions + versionPrecedesCrossMajor() so MOFeatureDiscoveryFixed is enabled for MO >= 2.0.0 across majors (fixes MO 3.x incorrectly falling back to service-addresses).
  • Replace HaKeeperAdds() with HaKeeperSvcAddrs(ls, reservedOrdinals) and thread reserved ordinals from the LogSet StatefulSet into DN/CN ConfigMap generation (fixes stale addresses after failover).
  • Add/extend regression tests for the new feature-gate behavior and ordinal-hole skipping.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 14 comments.

Show a summary per file
File Description
pkg/controllers/logset/sts.go Export LogSetStsName() so other controllers can find the backing StatefulSet.
pkg/controllers/logset/configmap.go Add HaKeeperSvcAddrs() to generate HAKeeper addresses while skipping reserved ordinals.
pkg/controllers/logset/configmap_test.go Add unit tests covering ordinal-hole skipping for HAKeeper service addresses.
pkg/controllers/dnset/resource.go Thread reserved ordinals into DN ConfigMap generation; use HaKeeperSvcAddrs().
pkg/controllers/dnset/resource_test.go Add regression test ensuring DN service-addresses skips reserved ordinals.
pkg/controllers/dnset/controller.go Conditionally fetch LogSet reserved ordinals only on the legacy service-addresses path; propagate errors.
pkg/controllers/dnset/controller_test.go Update fixtures/tests to include the LogSet StatefulSet and validate error propagation behavior.
pkg/controllers/cnset/resource.go Thread reserved ordinals into CN ConfigMap generation; use HaKeeperSvcAddrs().
pkg/controllers/cnset/resource_test.go Add regression test ensuring CN service-addresses skips reserved ordinals.
pkg/controllers/cnset/controller.go Conditionally fetch LogSet reserved ordinals only on the legacy service-addresses path; propagate errors.
pkg/controllers/cnset/controller_test.go Update fixtures/tests to include the LogSet StatefulSet and validate error propagation behavior.
api/core/v1alpha1/semver.go Introduce cross-major feature gating for stable features; move DiscoveryFixed to global min versions.
api/core/v1alpha1/semver_test.go Add regression tests for DiscoveryFixed across majors and guardrails preventing widening other gates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread api/core/v1alpha1/semver.go
Comment thread api/core/v1alpha1/semver.go
Comment thread api/core/v1alpha1/semver_test.go
Comment thread pkg/controllers/logset/sts.go
Comment thread pkg/controllers/logset/configmap.go
Comment thread pkg/controllers/dnset/controller_test.go
Comment thread pkg/controllers/cnset/resource.go
Comment thread pkg/controllers/cnset/resource_test.go
Comment thread pkg/controllers/cnset/controller.go
Comment thread pkg/controllers/cnset/controller_test.go
@loveRhythm1990
loveRhythm1990 merged commit 2301e3b into matrixorigin:main Jul 2, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants