fix: update DN/CN HAKeeper config after LogService failover (#596, #597)#599
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
There was a problem hiding this comment.
Pull request overview
This PR fixes DN/CN HAKeeper client configuration becoming stale after LogService failover by (1) making MOFeatureDiscoveryFixed correctly apply to MO 3.x+ via a cross-major “global min version” gate, and (2) generating service-addresses that skip StatefulSet.spec.reserveOrdinals holes when the legacy address path is used.
Changes:
- Add
featureGlobalMinVersions+versionPrecedesCrossMajor()soMOFeatureDiscoveryFixedis enabled for MO >= 2.0.0 across majors (fixes MO 3.x incorrectly falling back toservice-addresses). - Replace
HaKeeperAdds()withHaKeeperSvcAddrs(ls, reservedOrdinals)and thread reserved ordinals from the LogSet StatefulSet into DN/CN ConfigMap generation (fixes stale addresses after failover). - Add/extend regression tests for the new feature-gate behavior and ordinal-hole skipping.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/controllers/logset/sts.go | Export LogSetStsName() so other controllers can find the backing StatefulSet. |
| pkg/controllers/logset/configmap.go | Add HaKeeperSvcAddrs() to generate HAKeeper addresses while skipping reserved ordinals. |
| pkg/controllers/logset/configmap_test.go | Add unit tests covering ordinal-hole skipping for HAKeeper service addresses. |
| pkg/controllers/dnset/resource.go | Thread reserved ordinals into DN ConfigMap generation; use HaKeeperSvcAddrs(). |
| pkg/controllers/dnset/resource_test.go | Add regression test ensuring DN service-addresses skips reserved ordinals. |
| pkg/controllers/dnset/controller.go | Conditionally fetch LogSet reserved ordinals only on the legacy service-addresses path; propagate errors. |
| pkg/controllers/dnset/controller_test.go | Update fixtures/tests to include the LogSet StatefulSet and validate error propagation behavior. |
| pkg/controllers/cnset/resource.go | Thread reserved ordinals into CN ConfigMap generation; use HaKeeperSvcAddrs(). |
| pkg/controllers/cnset/resource_test.go | Add regression test ensuring CN service-addresses skips reserved ordinals. |
| pkg/controllers/cnset/controller.go | Conditionally fetch LogSet reserved ordinals only on the legacy service-addresses path; propagate errors. |
| pkg/controllers/cnset/controller_test.go | Update fixtures/tests to include the LogSet StatefulSet and validate error propagation behavior. |
| api/core/v1alpha1/semver.go | Introduce cross-major feature gating for stable features; move DiscoveryFixed to global min versions. |
| api/core/v1alpha1/semver_test.go | Add regression tests for DiscoveryFixed across majors and guardrails preventing widening other gates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
What type of PR is this?
Which issue(s) this PR fixes:
Fixes #596
Fixes #597
Related: #598
What this PR does / why we need it:
After LogService failover (e.g.
log-2fails and is replaced bylog-3viaReserveOrdinals), DN/CN ConfigMaps may keep stalehakeeper-client.service-addressespointing at the dead pod. This PR fixes two intertwined bugs:Bug-1 (#597): MO 3.x incorrectly falls back to
service-addressesMOFeatureDiscoveryFixedwas only listed underfeatureVersionswith a2.0.0entry. BecauseversionPrecedes()isolates major versions,HasMOFeature(3.x, DiscoveryFixed)always returnedfalse, so MO 3.x DN/CN kept using legacyservice-addressesinstead ofdiscovery-address.Fix:
MOFeatureDiscoveryFixedto a newfeatureGlobalMinVersionsmap (stable cross-major config capability since MO 2.0).versionPrecedesCrossMajor()for features that do not need per-major re-verification.LockMigration,PipelineInfo, etc.) infeatureVersionsunchanged.Bug-2 (#596):
HaKeeperAdds()ignoredReserveOrdinalsWhen
service-addressesmode is active (MO 1.x / semver unavailable / pre-2.0), DN/CN ConfigMap builders usedHaKeeperAdds(), which always generated[log-0, log-1, log-2]and ignored ordinal holes created during failover.Fix (no CRD change):
HaKeeperSvcAddrs(ls, reservedOrdinals)mirroring the existinggossipSeeds()hole-skipping logic.LogSetStsName()so DN/CN controllers can read LogSet STSspec.reserveOrdinals.service-addressesbranch is actually used (skip extra STS GET for MO >= 2.0 discovery path).HaKeeperAdds().Expected behavior after merge
service-addresseswith correct ordinal holesdiscovery-addressSpecial notes for your reviewer:
LogSetStatus.MOFeatureDiscoveryFixeduses cross-major global min version; other features are intentionally not extended to 3.x.service-addressespath, reconcile retries instead of writing a wrong address list.fetchLogSetReservedOrdinalsexists in bothdnsetandcnsetcontrollers; acceptable for now, can dedupe later.Additional documentation (e.g. design docs, usage docs, etc.):
unit-agent/docs/handbook/logservice-failover-configmap-bug.mdTest plan
go test ./api/core/v1alpha1/...go test ./pkg/controllers/dnset/...go test ./pkg/controllers/cnset/...go test ./pkg/controllers/logset/...