[Bug] Chaos (33bc7a2): TPCC durable ytd corruption on warehouse 8 after `task_kill_dn` — Payment district/warehouse split never heals

[heni02]

Environment

Field	Value
Cluster	mo-chaos-4-1-dev-33bc7a2-202607131556
Namespace	mo-chaos-4-1-dev-33bc7a2-202607131556
MO version	env name 33bc7a2; running image at incident commit-ce99d329c (S3 prefix still …/commit-33bc7a2/data)
Endpoint	10.222.6.253:6001
Artifact root	/data1/chaos-test/29230227851_33bc7a2
Workloads	TPCC (jTPCC 10 wh / 10 terminals), mo-load, vector/fulltext, chaos-mesh
Incident start	2026-07-17 15:52:14 CST
Analysis point	2026-07-17 17:20 CST (still failing; 0 PASS after incident)

Scope: Permanent TPCC ConsistencyCheck failure on w_ytd ↔ sum(d_ytd) for warehouse id = 8 after chaos task_kill_dn, preceded by 20702 / 20502 and 1062 bursts. Drift does not self-heal and continues to widen as Payments run.

Out of scope / do not merge:

• Transient o_ol_cnt read-skew (self-heal in ≤1 min) — earlier same env (e.g. Jul 17 01:28) and 2865ed6 / 2a00828 class.
• DN merge file … is not found (#25718) — same soak, different defect; do not treat ytd as caused by rawlog/statement_info missing aobj without proof.
• Single New-Order 1062 without sustained ytd fail (#25704 / mem_cn 1062 class).

Summary

On commit-ce99d329c chaos long-run, TPCC verify was healthy through 15:51:40 CST, then starting 15:52:14 every cycle fails ytd invariants for w_id=8. Through 17:20+ (≥1.5h, 290+ Consistency verification failed lines, 0 Consistency verification successfully):

(SELECT d_w_id, sum(d_ytd) FROM bmsql_district GROUP BY d_w_id)
EXCEPT (SELECT w_id, w_ytd FROM bmsql_warehouse);

(SELECT w_id, w_ytd FROM bmsql_warehouse)
EXCEPT (SELECT d_w_id, sum(d_ytd) FROM bmsql_district GROUP BY d_w_id);

Signal	Value	Interpretation
Last PASS	15:51:40	Still green ~7 min after kill_dn cleanup
First FAIL	15:52:14	ytd EXCEPT non-empty for w_id=8
PASS after incident	0	Not read-skew
Failed SQL	ytd only (both EXCEPT directions)	Payment invariant
o_ol_cnt consistency fail (window)	0	Not ol read-skew
Drift @ 15:52	d≈898621750.55 vs w≈898618420.23 (Δ≈3330)	Already durable split
Drift @ 17:20	d≈902367724.57 vs w≈902360474.51 (Δ still thousands, changing)	Payments keep widening/changing gap
Trigger chaos	task_kill_dn 15:44:03–15:44:34; prior task_mem_cn 15:34–15:39	DN hard kill recovery window
Client errors before FAIL	20702 / 20502 storm 15:45–15:48; 1062×3 (incl. w8/w9)	Ambiguous commit / unsafe retry signal

Preferred root-cause chain (symptom → mechanism → root):

1. Symptom: For warehouse 8, sum(d_ytd) ≠ w_ytd forever after 15:52; verify never returns successfully.
2. Mechanism: Payment updates bmsql_warehouse.w_ytd and bmsql_district.d_ytd in one client txn. Under DN kill + lock-bind churn (20702) / backend close (20502), one side of the ytd update is applied (or retried) without the other → permanent split. Ongoing Payments amplify the gap.
3. Root cause (fixable): Same family as 18addf5 / 7cbeacd / 4056142: Payment cross-table (cross-partition) ytd update is not atomic / not safely retriable under DN fault. Secondary: client blind retry after 20702/20502 may worsen split (must not double-apply one side).

TPCC invariants

Invariant	Txn	This incident
sum(d_ytd) per w_id = w_ytd	Payment	Permanent FAIL (w_id=8)
sum(o_ol_cnt) = count(order_line)	NewOrder	0 consistency fails in incident window
D_NEXT_O_ID - 1 = max(o_id)	NewOrder	Not observed failing in sampled FAIL lines

Payment (single JDBC txn):

UPDATE bmsql_warehouse SET w_ytd = w_ytd + :amount WHERE w_id = :w_id;
UPDATE bmsql_district   SET d_ytd = d_ytd + :amount WHERE d_w_id = :w AND d_id = :d;
...
COMMIT;

Timeline (CST)

Time	Event	Evidence
15:34:03–15:39:03	task_mem_cn	mo_chaos_test.log
15:44:03–15:44:34	task_kill_dn	mo_chaos_test.log
15:45:06–15:48:21	20702 / 20502 burst; 1062×3 on bmsql_oorder	tpcc.log
15:45–15:51:40	Verify still successfully	check.log
15:52:14	First ytd FAIL (w_id=8, both EXCEPT directions)	check.log
15:52 → 17:20+	Continuous FAIL, 0 PASS	check.log

Evidence

Log paths

File	Path
TPCC verify	/data1/chaos-test/29230227851_33bc7a2/mo-chaos/test-tool/mo-tpcc/check.log
TPCC run	/data1/chaos-test/29230227851_33bc7a2/mo-chaos/test-tool/mo-tpcc/tpcc.log
Chaos harness	/data1/chaos-test/29230227851_33bc7a2/mo-chaos/logs/mo_chaos_test.log

First FAIL exceptional rows

2026-07-17 15:52:14 ... failed for sql : (select d_w_id, sum(d_ytd) ...) except (Select w_id, w_ytd ...)
The exceptional result are :
8	898621750.55

2026-07-17 15:52:14 ... failed for sql : (Select w_id, w_ytd ...) except (select d_w_id, sum(d_ytd) ...)
The exceptional result are :
8	898618420.23

Still FAIL ~1.5h later

2026-07-17 17:20:20 ... exceptional:
8	902367724.57	  -- district sum side
8	902360474.51	  -- warehouse side

Pre-incident client FATAL sample

2026-07-17 15:45:06 ... [TT_PAYMENT] 20702 lock table bind changed
2026-07-17 15:45:16 ... [TT_PAYMENT] 20502 backend connection closed
2026-07-17 15:47:49 ... [TT_NEW_ORDER] 1062 Duplicate entry '(9,1,60502)' ...
2026-07-17 15:48:21 ... [TT_NEW_ORDER] 1062 Duplicate entry '(8,1,40043)' ...
2026-07-17 15:48:21 ... [TT_NEW_ORDER] 1062 Duplicate entry '(9,7,60609)' ...

Why this is release-blocking

1. Hard consistency gate: w_ytd / d_ytd is a core TPCC Payment invariant; sustained FAIL = data corruption, not chaos noise.
2. No self-heal: Unlike o_ol read-skew (33s–1min PASS), this stays red while commit path still accepts Payments (gap evolves).
3. Repeat family: Same R1 class as 18addf5 / 7cbeacd / 4056142 / 07924fb-line ytd fails under chaos — DN/network fault + Payment split.
4. Trigger is production-realistic: task_kill_dn (hard kill) + lock-table rebind.

Proposed fixes

A. MO server (required)

Area	Change
Payment / txn	Ensure warehouse + district ytd updates commit atomically across partitions under DN restart; no durable one-sided apply
Commit / retry	On 20702 / 20502 / 20638, expose safe resolve so clients do not retry a half-applied Payment
Observability	Structured log: Payment txn-id → applied tables / amounts when DN kill interrupts commit

B. Client / TPCC (defense in depth)

Area	Change
jTPCC Payment	After 20702/20502/20638: do not blindly re-apply +amount to both tables; re-read both ytd or use idempotency token
Gate	Any consecutive ytd FAIL ≥ N (e.g. 3) or FAIL lasting > 2 min → hard fail soak (already effectively hard here)

C. Verification

Check	Expectation
Repro: task_kill_dn + TPCC Payment soak	May see 20702; ytd must return to PASS and stay PASS
Soft	Isolated 1062 without ytd drift → separate retry issues
Hard	Sustained ytd FAIL → block release (this issue)

Severity

P0 — durable TPCC consistency corruption; release blocker.

Related

• Same env merge missing object (orthogonal): #25718
• Same-family durable ytd: issues/matrixone-chaos-18addf5-tpcc-consistency-durable-corruption.md, 7cbeacd, 4056142; soak 07924fb ytd class
• Distinct transient read-skew: 2a00828 / 4164cfb / 83cc163 / earlier 33bc7a2 o_ol self-heal
• 1062 without durable ytd: #25704

[LeftHandCold]

补充完整 RCA、代码结论和 main 残余竞态验证。结论分为两层，避免把事故直接路径和同类残余旁路混在一起。

1. #25845 的直接根因可以确认

这次不是 Payment 跨表事务在 TN 上“一边提交、一边没提交”，而是 TN allocator 重启后，旧 allocator epoch 的 lock-table bind 继续在旧 CN 上服务，同时新 allocator 已把同一张表分配给新 CN。同一热点行因此出现两个互不感知的 lock owner，TPCC Payment 的 read-modify-write 失去串行化，最终形成 lost update。

事故中可对上的两张表和 epoch：

对象	值
bmsql_district table id	272983
bmsql_warehouse table id	273028
旧 allocator version	1784269108857246992
新 allocator version	1784274278278875983
旧 owner	CN 2lsdp，继续持有旧 local bind
新 owner	CN zvdmr，取得新 local bind

关键时间线：

1. 15:45:10 新 allocator 为 bmsql_district 建立新 bind。
2. 15:45:16 新 allocator 为 bmsql_warehouse 建立新 bind。
3. 旧 epoch 的两张 local lock table 直到 15:49:11 才关闭，形成约 4 分钟双 owner 窗口。
4. 15:45:42，fix(lockservice): backport fenced orphan unlock to 4.1-dev #25652 的 fenced orphan unlock 已经正确拒绝一次不安全 unlock；15:47:17 后续 unlock 也携带了非空 fence timestamp。这说明 fix(lockservice): backport fenced orphan unlock to 4.1-dev #25652 本身在工作，但它只能保护“一个 lock table 内的 holder”，不能阻止第二个 allocator epoch 同时提供另一套 lock table。
5. 1062 (8,1,40043) 出现在双 owner 窗口内；现存原始订单行的 o_entry_d=15:48:05。
6. 最终单条 SQL 同一 snapshot 查询得到：w_ytd=903696876.64，sum(d_ytd)=903700206.96，固定差值 3330.32。Issue 中看起来“差值继续扩大”的两条 EXCEPT 是先后两条 SQL、使用了不同 snapshot，不能据此证明差值持续扩大。

机制是 lost update：两个 CN 分别在不同 lock table 上同时读取并更新同一 warehouse 热点行，其中一个 w_ytd += amount 被覆盖；对应 district 行的增量仍保留。因此最终持久化状态违反 warehouse.w_ytd = sum(district.d_ytd)，但不要求 TN 做出非原子的单事务提交。

目前无法从日志唯一命名造成最终 3330.32 差值的那一个 Payment txn：成功 SQL 没有完整记录，system.statement_info 又受 #25718 缺对象问题影响。02c9...60658 只能算候选，不能定责；它在 backend retry 耗尽后 rollback，旧锁之后被 fenced cleanup。

2. 与已有修复的关系

• fix(lockservice): fence stale binds by allocator epoch #24906：本次事故环境 ce99d329cf 没有该修复。它引入 allocator ID/epoch 观测、清理旧 local/remote/proxy bind，并 fence 使用旧 bind 的活跃事务，是 [Bug] Chaos (33bc7a2): TPCC durable ytd corruption on warehouse 8 after task_kill_dn — Payment district/warehouse split never heals #25845 主路径需要的修复。
• fix(lockservice): backport fenced orphan unlock to 4.1-dev #25652：修复 orphan cleanup 空 commit timestamp 的安全问题；本次日志证明它已生效，但不能处理“双 allocator epoch、双 lock owner”。
• fix(lockservice): fence unlock after unknown commit result #25722：处理 ErrTxnUnknown/模糊 commit 的后续 resolve 和锁清理，是配套防线，不是这次双 bind 的直接根因。
• fix(disttae): fence latest logtail reads with pending table apply #25515：修复 pending logtail 的读可见性，可解释校验何时看见结果，不会制造这次持久化 lost update。

因此，对 #25845 事故本身，缺失 #24906 是直接代码缺口；不是 #25652 再次失效，也不能用 #25515 解释真实坏数据。

3. main 上仍存在一个真实、可生产到达的 #24906 残余旁路

重新审计当前 main 后确认：旧 owner 的异步 NewBind 响应此前可以绕过 allocator epoch 校验直接调用 handleBindChanged。真实顺序如下：

1. CN 缓存 allocator A0 的 remote bind，并向旧 owner 发送普通 remote Lock/Unlock/GetLock 或 keeper 的 KeepRemoteLock RPC。
2. RPC 在途期间，标准 keeper/keepalive goroutine 观察到 A1，清理 A0 bind。
3. 网络延迟后，旧 owner 返回 A0 的 NewBind；因为 response allocator ID 与 request bind 都是 A0，旧代码走直接发布。
4. A0 bind 在 A1 purge 之后被重新放回 tableGroups。如果没有新的 allocator 变化事件，之后的同 A1 keepalive可以因 epoch 未再次变化而短路，旧 bind 得以继续服务。

这不是 fake-only 理论路径：标准 CN 初始化必然启动 keeper（service_remote.go:initRemote），keeper 默认周期性发送异步 KeepRemoteLock；普通 remote lock RPC 也会消费 NewBind。TN/allocator 重启与网络延迟足以形成上述交错。

已加入确定性回归：使用真实 MORPC 固定“请求已到旧 owner -> CN 观察 A1 并完成 purge -> 放行 A0 response”的顺序。未修复 main 上两个测试都稳定失败，日志和断言都看到 A0 bind 在 A1 后重新创建；修复后 -race -count=10 和 pkg/lockservice 全包通过。

main 修复 PR：#25847

修复原则：owner response 只作为“bind 已变化”的 point-in-time hint，不能直接作为发布依据；收到 NewBind 后向当前 allocator 做一次 GetBind，再通过 #24906 已有的 handleBindChangedFromAllocator epoch 校验发布。若刷新失败，只返回 ErrLockTableBindChanged，绝不发布 response 中的 bind。

性能和语义边界：额外 RPC 只发生在收到 NewBind 的异常/rebind 路径，正常 Lock、Unlock 和 keepalive 成功路径不增加 RPC；同 allocator 的合法 rebind 在 allocator 确认后仍正常发布；无 wire format、事务提交和锁 ownership 语义变化。

按当前决策，本次只修 main，暂不更新 4.1-dev。4.1 后续如果 backport，完整组合应是已有 #25652 + #24906 + #25847 的 residual-race closure，而不是只机械 cherry-pick #24906 后宣称所有竞态已经关闭。

[LeftHandCold]

主要观察main分支 @heni02