[Bug]: ErrStreamClosed / context canceled on WITH RECURSIVE + bkpf/bseg join (jinpan payment_details_tree)

[heni02]

Summary

In jinpan_001 Fact+Dim join regression, several payment_details_tree queries fail within 1–3 seconds with:

• (20503) rpc stream closed (ErrStreamClosed) — 4 hits
• (1105) context canceled — 1 hit on the same template

Same template succeeds in 24/34 runs in the same suite, so this is intermittent and not a SQL syntax/schema error. Client timeout is 1800s; these are not client timeouts.

20503 and 1105 here look like two client-visible faces of the same failure (pipeline/stream torn down → query context canceled).

Environment

Item	Value
MO commit	f8aaa25e670d0936b14602becd70d5966b1a87a1 (jinpan-f8aaa25)
Note	This commit is the CN pipeline FIN/ACK / connection-reuse change (#25713) — possibly related
CI	AP Regression #29470315932 / Fact+Dim Join
Case branch	big_data_jinpan
Dataset	jinpan_001

Hits (same template family)

Case	Error	Wall clock to fail
q0385_payment_details_tree	(20503) rpc stream closed	~3s
q0447_payment_details_tree	(20503) rpc stream closed	~2s
q0664_payment_details_tree	(20503) rpc stream closed	~2s
q0911_payment_details_tree	(20503) rpc stream closed	~1s
q0695_payment_details_tree	(1105) context canceled	~2s

Case files under mo-nightly-regression:
tools/mo-regression-test/cases/jinpan_001/join/j_batch_{020,023,034,035,046}.sql

After each failure, the next SQL in the same session often succeeds immediately (cluster not fully dead).

Repro SQL (q0385)

WITH w AS (
WITH RECURSIVE acct_tree AS (
  SELECT cpmb_kgd4b76 AS node_id, parenth1 AS parent_id, 0 AS lvl
  FROM dwd_dcp.dwd_bw_1cpmb_bkgd4b76
  WHERE cpmb_kgd4b76 = '200000'
  UNION ALL
  SELECT c.cpmb_kgd4b76, c.parenth1, t.lvl + 1
  FROM dwd_dcp.dwd_bw_1cpmb_bkgd4b76 c
  INNER JOIN acct_tree t ON c.parenth1 = t.node_id
  WHERE t.lvl < 3
)
SELECT
  CONCAT(b.belnr, '-', b.bukrs, '-', b.gjahr) AS voucher_header_id,
  b.belnr, b.bukrs, b.gjahr, b.bldat,
  s.buzei, s.hkont, s.dmbtr, s.zuonr,
  t.node_id AS acct_node, t.lvl,
  pd.payment_type
FROM dwd_dcp.dwd_s4_bkpf b
INNER JOIN dwd_dcp.dwd_s4_bseg s ON b.bukrs = s.bukrs AND b.belnr = s.belnr AND b.gjahr = s.gjahr
INNER JOIN acct_tree t ON s.hkont = t.node_id
LEFT JOIN jst_receipts_tables.payment_details pd
  ON pd.belnr = b.belnr AND pd.bukrs = b.bukrs AND pd.gjahr = b.gjahr
WHERE b.gjahr = '2024' AND s.dmbtr <> 0
  AND b.bukrs IN ('2000', '3000')
  AND b.belnr >= '8' AND b.belnr < '9'
LIMIT 2000
)
SELECT * FROM w
WHERE 1 = 1
LIMIT 50;

Other hits are the same shape; only recursive root (100000/200000/600000), lvl bound, company-code / belnr filters, and outer LIMIT vary.

Why this looks like an MO bug (not case)

1. Error is engine ErrStreamClosed / canceled context, not parser/schema.
2. Same template succeeds most of the time in the same run → intermittent engine/pipeline path.
3. Failures cluster on WITH RECURSIVE + large fact joins (bkpf/bseg).
4. Fail in 1–3s under 1800s client timeout → not harness timeout.
5. Tested MO build centers on pipeline stream lifecycle (fix: reuse CN pipeline connections after clean teardown #25713); worth regression-checking with disable-stream-reuse / prior commit.

Related but uncertain (appendix)

One additional hit in the same job:

Case	Error
q0190_invoice_acdoca_link	(1105) context canceled (<1s)

Different SQL (multi-CTE invoice ↔ acdoca/bseg, no recursive CTE). Same error class; unclear if same root cause — include for correlation only.

Also seen in same suite (separate)

HashJoin nil panic on bill_collection_p4: #25802 — different stack/template; do not merge unless investigation proves shared cause.

Expected

Query completes (rows or empty set) without ErrStreamClosed / unexpected context canceled. If the CN aborts, the client error should surface the underlying panic/error, not only stream closed.

[heni02]

再一次手动执行也是必现，不到1s报错