[Bug]: Multi-CN dispatch timeout on DENSE_RANK() OVER (ORDER BY ...) without PARTITION BY (jinpan purchase_receipt)

[heni02]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Related (same error string / registry lifecycle, different workloads):

• [Bug] 30s remote dispatch watchdog can falsely fail slow receiver registration #25816 — 30s remote dispatch watchdog can falsely fail slow receiver registration (design root)
• Hive partition JOIN queries intermittently fail with dispatch timeout (dispatch process not registered within 30s) #25211 — Hive partition JOIN dispatch timeout
• [Bug]: Empty remote dispatch can lose receiver registration and time out #25687 / fix: wait for remote receivers on empty dispatch #25688 — empty remote dispatch registration race (fixed path)
• [Bug]: ErrStreamClosed / context canceled on WITH RECURSIVE + bkpf/bseg join (jinpan payment_details_tree) #25803 — ErrStreamClosed / context canceled on WITH RECURSIVE + bkpf/bseg (same multi-CN pipeline family)

This issue tracks a concrete jinpan Fact+Dim workload repro: global window (DENSE_RANK without PARTITION BY) over a multi-table join.

Branch Name

main / CI image used by jinpan AP Regression (big_data_jinpan)

Commit ID

CI run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/29470315932
Job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/29470315932/job/87586929302 (Fact+Dim Join Query Tests)

Please fill exact MO image SHA from that run's Build MO Image step if needed (historically around jinpan-f8aaa25 / #25713 era for this suite).

Other Environment Information

- Hardware parameters: TKE multi-CN cluster (jinpan_001 AP Regression)
- OS type: Linux
- Others:
  - No CN OOM observed when the error occurred
  - Dataset: jinpan_001 (`dwd_dcp.dwd_s4_ekbe` + dim tables)
  - Case branch: `big_data_jinpan`

Actual Behavior

Query fails with:

ERROR 20101 (HY000): internal error: dispatch process not registered within 30s, remote CN may have failed

Cluster CNs remain healthy (no OOM). The error message's "remote CN may have failed" is misleading here.

Expected Behavior

Query should complete (or fail with a real root-cause error). A delayed multi-CN receiver registration during heavy join + global window preparation must not be reported as remote CN failure after a fixed 30s watchdog.

Steps to Reproduce

Failing SQL (exact hit)

Case: q0121_purchase_goods_receipt_mart
File: tools/mo-regression-test/cases/jinpan_001/join/j_batch_007.sql
(template family: purchase_receipt / purchase_goods_receipt_mart — many similar cases with different vgabe)

SELECT * FROM (
SELECT
  e.ebeln, e.budat, e.menge, e.wrbtr, e.belnr,
  k.frgke, mk.maktx, ma.guige,
  DENSE_RANK() OVER (ORDER BY e.wrbtr DESC) AS receipt_rank
FROM dwd_dcp.dwd_s4_ekbe e
LEFT JOIN dwd_dcp.dwd_s4_ekko k ON e.ebeln = k.ebeln
LEFT JOIN dwd_dcp.dwd_s4_makt mk ON e.matnr = mk.matnr
LEFT JOIN dwd_dcp.dwd_s4_mara ma ON e.matnr = ma.matnr
WHERE e.vgabe = '2' AND e.shkzg = 'H'
  AND e.budat >= '2021-01-01'
) w
WHERE 1 = 1
LIMIT 500;

Why this shape stresses remote dispatch

1. DENSE_RANK() OVER (ORDER BY ...) has no PARTITION BY
   Window operator enters receiveAll and must drain the entire child result before ranking (pkg/sql/colexec/window/window.go).

2. Outer LIMIT 500 does not help
   Global order window prevents meaningful limit pushdown into the join; the engine still has to produce / shuffle a large intermediate for the window.

3. Multi-table LEFT JOIN on large fact (ekbe since 2021)
   On multi-CN this becomes a distributed join + shuffle/sort plan with remote pipeline notify/dispatch.

4. 30s registration watchdog (waitRegistrationTimeout in pkg/sql/compile/remoterunServer.go)
   Receiver waits for PutProcIntoUuidMap. If join/window pipeline placement or registration is delayed >30s (or races with notify), client sees this error even when no CN has failed — see [Bug] 30s remote dispatch watchdog can falsely fail slow receiver registration #25816.

Additional information

Root-cause analysis (current best understanding)

Layer	Finding
Case SQL	Valid; not a schema/syntax issue
OOM / CN crash	Ruled out for this occurrence
Trigger shape	Global window (DENSE_RANK w/o partition) + multi-CN join
Failure site	Remote dispatch UUID not registered within 30s
Likely class	Registration ordering / readiness vs fixed watchdog (#25816), possibly compounded by pipeline lifecycle races (#25687 family)

Suggested fix direction

1. Treat readiness separately from CN liveness (do not map slow registration to "remote CN may have failed") — align with [Bug] 30s remote dispatch watchdog can falsely fail slow receiver registration #25816.
2. Ensure dispatch UUID registration happens before/parallel to expensive child work for window receiveAll + remote join plans (early register paths like fix: register remote dispatch receivers early #25238 / fix: make remote dispatch registration race-safe #25574 / fix: register local dispatches below remote scopes #25664 coverage for this topology).
3. Consider whether global-window plans should avoid fragile remote notify ordering, or register receivers before starting heavy join producers.
4. Add a multi-CN regression: DENSE_RANK() OVER (ORDER BY x) (no partition) over a large join, assert no dispatch process not registered within 30s.

Workaround (case / query)

• Add a meaningful PARTITION BY (if business-correct), or
• Materialize join first / tighten predicates, or
• Rewrite rank via ORDER BY wrbtr DESC + application-side numbering after a smaller limited scan (only if semantics allow — note true global dense_rank still needs full order).

These are mitigations only; engine fix is required for the multi-CN path.

[heni02]

有issue记录了，closed