[Bug] Chaos (aa2a02c): with 2/3 LogStores Up, CN/DN still fail `waitAnyShardReady` and dial deleted `log-0`

[heni02]

Environment

Field	Value
Cluster / Namespace	mo-chaos-main-aa2a02c-202607172328
MO image	commit-826a3e3d2 (ns name still aa2a02c)
Endpoint	10.222.6.252:6001
Artifact	/data1/chaos-test/29592172186_aa2a02c
LogSet	replicas 3, pods log-1 / log-2 / log-3 (no log-0 pod)
Analysis	2026-07-23 ~11:00 CST (cluster already degraded since ~Jul 21 evening)

Scope: MatrixOne runtime HA / HAKeeper membership / CN·DN bootstrap. After one LogStore is Down (CrashLoop), the other two report Up, but CN/DN still cannot become ready: dial non-existent …-log-0…, then wait.tn.ready / waitAnyShardReady timeout → process panic → CrashLoop; surviving CN stays in txn client is in pause / createTxnOpUnsafe, rate(mo_txn_commit_total)≈0.

Out of scope (separate issues / workflow):

• Operator CR Ready=False / NoEnoughReadyStores with 2 Up — Operator issue (sibling).
• Chaos task_mem_log injecting 2GB into Log with limit 2Gi — workflow misconfig (trigger only).
• TPCC durable ytd corruption — not observed before outage (verify was green until freeze).

Summary

Single LogStore failure should be tolerated by a 3-replica log quorum (majority = 2). Observed reality:

Signal	Value
log-1 / log-3	Ready true
log-2	CrashLoopBackOff (OOMKilled historically; exit 137)
LogSet availableStores	2 (log-1, log-3)
LogSet failedStores	1 (log-2)
Client	ERROR 20508 all CN servers are busy, possibly due to too many active transactions
Commit	sum(rate(mo_txn_commit_total{type="cn"}[5m])) ≈ 0
Surviving CN	txn client is in pause + mass createTxnOpUnsafe

Preferred root-cause chain:

1. Symptom: CN/DN CrashLoop or txn create blocked while 2/3 Log pods are Ready.
2. Mechanism: Bootstrap / HAKeeper client still targets mo-chaos-regression-dis-log-0.…svc:32001, which does not exist (DNS no such host). Shard never becomes ready → waitAnyShardReady / wait.tn.ready.timeout → panic in cmd/mo-service.
3. Root cause (fixable): Stale HAKeeper / log member list (or bootstrap seed) retains removed ordinal log-0 after StatefulSet members became log-1..3, and/or CN/DN bootstrap does not skip permanently unreachable members / does not fail soft when quorum of reachable stores is enough.

Evidence

DNS to ghost log-0 (DN previous log)

init remote connection failed ... remote:
  mo-chaos-regression-dis-log-0.mo-chaos-regression-dis-log-headless.<ns>.svc:32001
error: lookup ...log-0... on 192.168.20.10:53: no such host

CN bootstrap then panic

wait.tn.ready.not.ready   (repeated)
wait.tn.ready.timeout
panic: context deadline exceeded
  internal error: waitAnyShardReady
main.main()  cmd/mo-service/main.go

Surviving CN (txn path)

txn client is in pause state, wait for it to be ready
context deadline exceeded
internal error: createTxnOpUnsafe

Proposed fixes (MO server)

Area	Change
HAKeeper / log discovery	On member unreachable (NXDOMAIN / connection refused sustained): evict or mark Down in bootstrap list; never hard-depend on a DNS name with no endpoints
Membership reconcile	When StatefulSet ordinals change (log-0 gone, log-1..n present), rewrite bootstrap / gossip seeds from current Ready pods + discovery Service, not historical pod names
waitAnyShardReady	If ≥ quorum of LogStores are reachable and shards are scheduled on them, proceed; do not block forever on a dead member
Process exit	Startup timeout: return error / os.Exit(1) instead of panic, so CrashLoop reason is clearer and sidecars/probes behave predictably (cmd/mo-service/main.go, launch.go)
Observability	Structured log: expected_members, reachable_members, ghost_members when bootstrap stalls

Verification

Check	Expectation
Repro	3 Log replicas; OOMKill or delete one Log pod only; leave other two Ready ≥10min
Pass	CN/DN Ready; rate(mo_txn_commit_total{type="cn"}) healthy; no dials to missing log-0; client login works
Fail (today)	CN/DN panic CrashLoop or createTxnOpUnsafe / all CN busy with 2/3 Log Up

Related

• Sibling (Operator): LogSet Ready=False / NoEnoughReadyStores despite availableStores=2
• Trigger context: chaos mem_log size 2GB vs Log limit 2Gi (fix in workflow; not this bug’s code fix)

[LeftHandCold]

调查结论

基于当前 issue 中的环境、日志和版本信息，这次故障不是 matrixone PR #25867（offline WAL recovery）导致的。目前更符合的根因是：

1. LogService 发生过一次 failover，原来的 log-0 被替换成了 log-3。
2. CN/TN 仍然使用旧的静态 HAKeeper 地址列表，仍包含 log-0/log-1/log-2，但没有包含替代节点 log-3。
3. 本次 chaos 又让 log-2 OOM/CrashLoop。
4. 于是 CN/TN 看到的静态候选实际上变成：
   • log-0: DNS 不存在；
   • log-1: 存活；
   • log-2: CrashLoop；
   • log-3: 存活，但不在配置列表中。
5. TN 无法完成 HAKeeper/Log shard 的连接和恢复，CN 查询 HAKeeper 时看不到带 shard 的 TN，最终触发 waitAnyShardReady 超时。

所以 issue 中的 wait.tn.ready.timeout 是 TN 未成功恢复后的下游症状，log-0 no such host 是过期地址配置的直接证据。

为什么不是 PR #25867

Issue 中运行的镜像是：

commit-826a3e3d2

对应 commit：

826a3e3

我检查了 commit ancestry：

git merge-base --is-ancestor fed039c2d55e35327cc389bd291b7f10dedeefa2 826a3e3d294ba62c7c4da217d1445f3c568ce4d9
# exit code: 1

fed039c... 是 PR #25867 的第一个提交，运行镜像不包含该提交。PR #25867 直到 2026-07-22 才合入，而 issue 环境使用的镜像 commit 是 2026-07-19 的 commit。

PR 地址：

#25867

因此这次故障不可能由该 PR 的 WAL recovery 代码引入。

waitAnyShardReady 的实际语义

运行版本中的 waitAnyShardReady 检查的是 HAKeeper 返回的 TN store：

details, err := client.GetClusterDetails(ctx)
for _, store := range details.TNStores {
    if len(store.Shards) > 0 {
        return true, nil
    }
}

代码位置：

matrixone/cmd/mo-service/launch.go

Lines 268 to 299 in 826a3e3

	func waitAnyShardReady(client logservice.CNHAKeeperClient) error {
	ctx, cancel := context.WithTimeoutCause(context.TODO(), time.Second*30, moerr.CauseWaitAnyShardReady)
	defer cancel()
	// wait shard ready
	for {
	if ok, err := func() (bool, error) {
	details, err := client.GetClusterDetails(ctx)
	if err != nil {
	if errors.Is(err, context.DeadlineExceeded) {
	err = moerr.AttachCause(ctx, err)
	logutil.Error("wait.tn.ready.timeout", zap.Error(err))
	return false, err
	}
	logutil.Error("wait.tn.ready.failed", zap.Error(err))
	return false, nil
	}
	for _, store := range details.TNStores {
	if len(store.Shards) > 0 {
	return true, nil
	}
	}
	logutil.Info("wait.tn.ready.not.ready")
	return false, nil
	}(); err != nil {
	return err
	} else if ok {
	logutil.Info("wait.tn.ready.ready.completed")
	return nil
	}
	time.Sleep(time.Second)
	}

它不是简单统计 Kubernetes 中有几个 Log Pod 为 Ready，也不是在等待某个固定的 log-0。因此：

• log-1/log-3 Kubernetes Ready，不代表 HAKeeper shard 0 有 leader；
• 也不代表 Log shard 1 有 leader；
• 更不代表 TN 已经向 HAKeeper 上报了有效 shard；
• CN 继续等待 TN shard 是符合当前代码语义的。

同时，HAKeeper client 会遍历配置中的多个 service address，并不是遇到第一个 log-0 DNS 失败就立即停止：

matrixone/pkg/logservice/hakeeper_client.go

Lines 1046 to 1106 in 826a3e3

	func connectToHAKeeper(
	ctx context.Context,
	sid string,
	targets []string,
	cfg HAKeeperClientConfig,
	) (*hakeeperClient, error) {
	if len(targets) == 0 {
	return nil, nil
	}
	pool := &sync.Pool{}
	pool.New = func() interface{} {
	return &RPCRequest{pool: pool}
	}
	respPool := &sync.Pool{}
	respPool.New = func() interface{} {
	return &RPCResponse{pool: respPool}
	}
	c := &hakeeperClient{
	cfg: cfg,
	pool: pool,
	respPool: respPool,
	}
	var e error
	addresses := append([]string{}, targets...)
	rand.Shuffle(len(addresses), func(i, j int) {
	addresses[i], addresses[j] = addresses[j], addresses[i]
	})
	for _, addr := range addresses {
	cc, err := getRPCClient(
	ctx,
	sid,
	addr,
	c.respPool,
	defaultMaxMessageSize,
	cfg.EnableCompress,
	defaultBackendReadTimeout,
	"connectToHAKeeper",
	)
	if err != nil {
	e = err
	continue
	}
	c.addr = addr
	c.client = cc
	isHAKeeper, err := c.checkIsHAKeeper(ctx)
	logutil.Info(fmt.Sprintf("isHAKeeper: %t, err: %v", isHAKeeper, err))
	if err == nil && isHAKeeper {
	return c, nil
	} else if err != nil {
	e = err
	}
	if err := cc.Close(); err != nil {
	logutil.Error("failed to close the client", zap.Error(err))
	}
	}
	if e == nil {
	// didn't encounter any error
	return nil, moerr.NewNoHAKeeper(ctx)
	}
	return nil, e

但是，如果列表本身已经过期，且当前存活/承载有效 Raft replica 的节点没有被列入，客户端就可能无法找到正确的 HAKeeper/LogService 入口。

对应的 operator 已知问题和修复

这个故障模式已经在 matrixone-operator 中有对应 issue：

operator #596

HaKeeperAdds() 忽略 Kruise StatefulSet 的 reserveOrdinals。当 log-0 被 failover、替代 Pod 变成 log-3 后，CN/TN ConfigMap 仍然生成：

service-addresses = [
  "log-0",
  "log-1",
  "log-2",
]

而不会生成包含 log-3 的列表。

matrixorigin/matrixone-operator#596

operator #597

MO 3.x 及后续版本由于 feature gate 没有正确跨 major 生效，错误地回退到了 legacy service-addresses 模式，没有使用 discovery-address。

matrixorigin/matrixone-operator#597

operator PR #599

PR #599 已经合并了对应修复：

• MO 2.x 及以后版本使用 discovery-address；
• legacy service-addresses 模式读取 reserveOrdinals；
• 跳过失效 ordinal；
• 把替代 Log Pod 加入 CN/TN 地址列表；
• 读取 StatefulSet 失败时让 reconcile 重试，而不是静默生成错误地址。

matrixorigin/matrixone-operator#599

该 PR 的 checks 和 e2e 已通过。修复 commit：

matrixorigin/matrixone-operator@2301e3b

本次 issue 需要确认的现场配置

为了最终把结论从“高置信度”闭合为“现场证实”，需要检查以下只读信息：

# 1. 查看 CN/TN 实际使用的是 discovery-address 还是 service-addresses
kubectl -n mo-chaos-main-aa2a02c-202607172328 get cm <cn-configmap> -o yaml
kubectl -n mo-chaos-main-aa2a02c-202607172328 get cm <dn-configmap> -o yaml

# 重点检查：
[hakeeper-client]
discovery-address = "..."
# 或者：
service-addresses = ["...log-0...", "...log-1...", "...log-2..."]

# 2. 查看 LogSet 的 reserve ordinal
kubectl -n mo-chaos-main-aa2a02c-202607172328 \
  get statefulset <log-statefulset> -o jsonpath='{.spec.reserveOrdinals}{"\\n"}'

如果看到：

• reserveOrdinals 包含 0；
• CN/TN 仍然使用 log-0/log-1/log-2；
• 配置中没有 discovery-address，也没有 log-3；

那么可以直接确认这是 operator #596/#597 对应的配置生成/版本未升级问题。

还需要补充 HAKeeper 层面的确认：

• HAKeeper shard 0 当前成员和 leader；
• Log shard 1 当前成员和 leader；
• log-1/log-3 是否真的承载当前 Raft replica；
• TN 是否已经发送有效 heartbeat 并获得 shard。

最终判断

当前不建议修改 MatrixOne：

• 不需要让 waitAnyShardReady 根据 Kubernetes Ready Pod 数量强行放行；
• 不应该让 MO 根据 DNS NXDOMAIN 自动修改 Raft membership；
• 不应该把 log-0 的过期地址问题通过 MO client 的特殊容错掩盖；
• Raft 成员变更、ordinal failover、ConfigMap 地址生成属于 operator/运维控制面职责。

建议优先确认并升级运行中的 matrixone-operator 至包含 PR #599 的版本，然后重新生成/校验 CN/TN 配置。

只有在确认 CN/TN 已经使用 discovery-address 或正确的 log-1/log-2/log-3 地址，并且确认 HAKeeper 和 Log shard 都有有效多数派后，仍然出现 TN 无法 ready，才需要继续作为 MatrixOne runtime bug 分析。

[heni02]

main故障测试没有再复现出该问题，先closed