[AMD] [AGENTX] [WIP] GLM-5.2: EP=1 interactivity + c12 throughput tuning on MI355X

[giovanniguastiamd]

Summary / 摘要

Tuning follow-up to #2570 (GLM-5.2 FP4 MI355X SGLang Agentic MTP).

Changes / 变更内容

1. TP8 arm: EP=8 → EP=1 (script + yaml)

GLM-5.2 is a MoE model. With EP=8 every forward pass introduces an all-to-all collective between 8 EP ranks to route tokens to the correct experts. At low concurrency (c4, effective batch ≈ 1–2 requests) this collective is overhead on essentially unshared traffic.

Switching to EP=1 (no expert parallelism, all experts replicated across TP=8 ranks) eliminates the all-to-all entirely:

• ITL p50 at c4: 7.3 ms → 6.95 ms (−5%)
• Interactivity P90 at c4: 105 → 110.5 tok/s/user (+5%)

The TP8 concurrency sweep is extended to [4, 6, 8, 10] to cover the mid-range crossover region (previously only [1, 2, 4]).

2. HiCache tuning: ratio 1.5 → 2.5, write_through_selective (script)

The agentic-coding corpus (ISL p50 ≈ 90k tokens) saturates any fixed DRAM KV pool at conc ≥ 10. Raising the host-DRAM ratio to 2.5× delays saturation and keeps throughput stable through conc 12:

• tok/s/GPU at c12: 102 vs 91 at c10 (+12%)

write_through_selective (introduced in #2679) skips DRAM writes for non-reusable KV blocks, reducing host-bus traffic without impacting cache hit rate.

The TP4 concurrency list is trimmed to [1, 2, 4, 8, 10, 12]; conc 16 is removed (untested and beyond the validated range).

3. MAX_RUNNING_REQUESTS 1×CONC → 2×CONC; CUDA_GRAPH_MAX_BS capped at 64 (script)

With MTP num-steps=5 the draft+verify phase transiently batches more tokens than CONC sessions. Doubling the in-flight slot budget prevents scheduler stalls under burst. SGLang auto-interpolates a batch-size list [1..max_bs], so capping cuda-graph-max-bs at 64 keeps graph-capture memory bounded without losing coverage.

---

摘要

对 #2570（GLM-5.2 FP4 MI355X SGLang Agentic MTP）的调优跟进。

1. TP8 臂：EP=8 → EP=1：GLM-5.2 是 MoE 模型，EP=8 在每次前向计算时引入 all-to-all 集合通信。低并发（c4）下该通信几乎无法分摊，EP=1 消除了这一开销：c4 ITL p50 7.3→6.95 ms（−5%），P90 交互性 105→110.5 tok/s/user（+5%）。TP8 并发扫描扩展至 [4, 6, 8, 10]。

2. HiCache 调优：ratio 1.5→2.5，write policy 改为 write_through_selective。agentic-coding 语料在 conc ≥ 10 时会填满任何固定 DRAM 池；更大的 host 端为 c12 提供吞吐支撑（+12% tok/s/GPU）。TP4 并发列表缩减为 [1, 2, 4, 8, 10, 12]。

3. MAX_RUNNING_REQUESTS 1×→2×CONC：MTP draft+verify 阶段瞬时 batch 超过 CONC，加大 in-flight slot 可防止调度器阻塞。CUDA_GRAPH_MAX_BS 上限设为 64（SGLang 已自动插值 bs 列表）。

---

Test plan / 测试计划

• Full sweep on MI355X cluster: TP4+HiCache conc [1,2,4,8,10,12] and TP8+EP=1 conc [4,6,8,10] both produce valid results
• Interactivity P90 at c4 ≥ 108 tok/s/user (within 2% of measured 110.5)
• Throughput at c12 ≥ 98 tok/s/GPU (within 4% of measured 102)
• No OOM on TP8/EP=1 (expert weights replicated across 8 ranks; HBM budget verified pre-run)

🤖 Generated with Claude Code

---

Note

Low Risk
Changes are benchmark recipe tuning (defaults, concurrency lists, EP topology) for a single MI355X agentic config—not production serving or auth/data paths; main operational risk is host DRAM pinning/OOM if cluster memory differs from validated runs.

Overview
Tuning follow-up for glm5.2-fp4-mi355x-sglang-agentic-mtp on MI355X: benchmark script defaults, search-space arms, and perf changelog.

Benchmark script (glm5.2_fp4_mi355x_sglang_mtp.sh): non–DP-attention HiCache default ratio 1.5 → 2.5 and write policy write_through → write_through_selective to delay DRAM KV saturation on long agentic contexts. MAX_RUNNING_REQUESTS is 2×CONC (was CONC) for MTP draft/verify bursts; CUDA_GRAPH_MAX_BS is capped at 64 instead of tracking full MAX_RUNNING_REQUESTS.

Config (amd-master.yaml): TP4+HiCache sweep drops conc 16 (now [1, 2, 4, 8, 10, 12]). TP8 GPU-resident arm switches EP=8 → EP=1 and sweeps [4, 6, 8, 10] (replacing [1, 2, 4]) for a low/mid-concurrency latency arm without MoE all-to-all.

perf-changelog.yaml documents measured gains (e.g. ~5% ITL at c4, ~12% tok/s/GPU at c12) and links PR #2769.

^{Reviewed by Cursor Bugbot for commit b9cec95. Bugbot is set up for automated code reviews on this repo. Configure here.}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

Dispatched trusted sweep run for approved external revision b9cec954bbe233da3c5b312590d0ba1ab2a8c46d.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 b9cec954bbe233da3c5b312590d0ba1ab2a8c46d 调度trusted sweep run。
后续新提交不会自动获得信任；如需批准新的 SHA，请移除并重新添加主扫描标签。

[seungrokj]

@giovanniguastiamd plz reopen in the upstream branch

[giovanniguastiamd]

Closing in favour of a new PR from an upstream branch (collaborator push). Same content, rebased on main.