[AMD][AGENTX] GLM-5.2: GPU-resident KV (no HiCache) Pareto sweep on MI355X - #2853
[AMD][AGENTX] GLM-5.2: GPU-resident KV (no HiCache) Pareto sweep on MI355X#2853giovanniguastiamd wants to merge 3 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
giovanniguastiamd
added
sweep-enabled
AMD
full-sweep-fail-fast
and removed
sweep-enabled
labels
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34100593424 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "TP4/EP4 no-KV arm, conc [2, 4, 6, 8]: validated on run 33724174688 (branch testgg-qr-int4). norm_intvty_p90: c2=67.0, c6=66.3, c8=48.7." | ||
| - "TP8/EP1 no-KV arm, conc [4, 10]: EP=1 eliminates MoE all-to-all collective. c4 validated on run 32986446019 (norm_intvty_p90=69.2, ITL p90=9.05 ms); c10 validated on run 33489967280 via PR#2777 sweep (norm_intvty_p90=38.6)." | ||
| - "Add optional ROCM_QUICK_REDUCE_QUANTIZATION=INT4 / ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16 env-var hooks in the benchmark script (no-op when unset; enables F-2 quick-reduce probe without a script edit)." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX |
There was a problem hiding this comment.
🔴 pr-link uses invented placeholder "pull/XXXX" instead of the documented pre-PR placeholder pr-link: TBD; docs/configuration-procedures.md "Append the changelog safely" step 3 explicitly permits only TBD before a PR exists, so this deviates from convention and, if merged verbatim, ships a dead/wrong URL in the changelog. Fix: replace with pr-link: TBD and update it to the real PR URL immediately after PR creation, per the documented procedure.
Extended reasoning...
AGENTS.md and docs/configuration-procedures.md's "Append the changelog safely" section state the only sanctioned pre-PR placeholder is pr-link: TBD, to be swapped for the real URL right after PR creation. This entry instead hardcodes https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX, a non-standard, non-parseable placeholder that isn't TBD and isn't a real PR number. Since perf-changelog.yaml is append-only and byte-sensitive (never rewritten later per repo convention), if this literal string is not caught in review it risks merging as a permanent, incorrect pr-link rather than being mechanically replaced the way TBD entries are expected to be.
Verification: normal. perf-changelog.yaml:6923 appends pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX (four X's). The repo's own changelog gate rejects this. utils/validate_perf_changelog.py:21-27 sets CANONICAL_PR_LINK = r"https://github\.com/SemiAnalysisAI/InferenceX/pull/\d+" and PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"} (three X's…
| - agentic-coding | ||
| description: | ||
| - "Replace HiCache DRAM offload with GPU-resident KV on both arms: GLM-5.2 MLA KV fits entirely in HBM up to conc 8, eliminating host-DRAM dependency and simplifying deployment. Pareto-optimal interactivity P90 equal to or better than the HiCache baseline across c2–c8." | ||
| - "TP4/EP4 no-KV arm, conc [2, 4, 6, 8]: validated on run 33724174688 (branch testgg-qr-int4). norm_intvty_p90: c2=67.0, c6=66.3, c8=48.7." |
There was a problem hiding this comment.
🟡 (optional) The TP4/EP4 no-KV arm's cited validation run 33724174688 was on branch "testgg-qr-int4" (quick-reduce INT4 probe active), but the merged recipe (configs/amd-master.yaml) sets no ROCM_QUICK_REDUCE_QUANTIZATION default, and the script only exports it when already set in the env, so the default/shipped code path never enables it. The conc [2,4,6,8] narrowing and the "Pareto-optimal ... equal to or better than the HiCache baseline" claim rest on numbers measured with an env var the default recipe doesn't set. Fix: re-validate c2/c6/c8 with the quick-reduce hook unset (the actual default), or set the env var explicitly in the recipe/runner config so the shipped path matches what was measured.
Extended reasoning...
benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh only does [[ -n "${ROCM_QUICK_REDUCE_QUANTIZATION:-}" ]] && export ROCM_QUICK_REDUCE_QUANTIZATION -- a no-op unless the runner's .env already sets it. configs/amd-master.yaml's glm5.2-fp4-mi355x-sglang-agentic-mtp entry has no additional-settings/env entry setting that var, so production runs of this recipe execute with quick-reduce OFF (falls back to NCCL/custom-AR per the script comment). But perf-changelog.yaml:6920 cites run 33724174688, branch "testgg-qr-int4", as the validation for the TP4/EP4 arm's conc [2,4,6,8] with norm_intvty_p90 c2=67.0/c6=66.3/c8=48.7 -- i.e. measured with the probe enabled. The default (unset) path this recipe actually runs going forward has never been shown to reproduce those numbers, undermining the stated Pareto-parity-with-baseline justification for dropping HiCache and narrowing the conc-list.
Verification: nit. The provenance mismatch the candidate describes is factually real and sits on the changed lines. perf-changelog.yaml line 6918 cites the TP4/EP4 validation as "run 33724174688 (branch testgg-qr-int4)" — a branch whose name denotes quick-reduce INT4 active. But glm5.2_fp4_mi355x_sglang_mtp.sh lines 73-74 make quick-reduce strictly opt-in (`[[ -n "${ROCM_QUICK_REDUCE_QUANTIZATION:-}" ]]…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34101340395 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34103958942 |
giovanniguastiamd
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 92a20c1. Configure here.
MLA KV fits in HBM up to conc 8; drop the DRAM offload dependency. Two no-KV arms: - TP4/EP4 conc [2,4,6,8]: validated on run 33724174688 (norm_intvty_p90: c2=67.0, c6=66.3, c8=48.7) - TP8/EP1 conc [4,10]: EP=1 removes MoE all-to-all collective (c4 run 32986446019 norm=69.2; c10 run 33489967280 norm=38.6) Also add ROCM_QUICK_REDUCE_QUANTIZATION env-var hooks in the script (no-op when unset). 性能(glm5.2,mi355x)：将两个 arm 从 HiCache DRAM 卸载改为 GPU 常驻 KV。 GLM-5.2 MLA KV 在 conc≤8 时完全适配 HBM，消除了主机 DRAM 依赖。 两个纯 GPU KV arm： - TP4/EP4 conc [2,4,6,8]：由 run 33724174688 验证 - TP8/EP1 conc [4,10]：EP=1 消除 MoE all-to-all 集合通信， c4 由 run 32986446019 验证，c10 由 PR#2777 sweep 验证 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extend the GLM-5.2 MI355X SGLang recipe from two to three arms covering both the interactivity and throughput regimes: - TP4/EP4 no-KV: conc [1,2,4,6,8] — best interactivity, low-mid conc - TP8/EP1 no-KV: conc [1,2,4] — EP=1 removes MoE all-to-all; best at c4 - TP4/EP4 + HiCache: conc [1,2,4,8,10] — KV offload for throughput regime Restores the TP4/EP4 HiCache arm from PR#2777 sweep (run 33489967280). Drops TP4/EP1 no-KV c10 (OOM risk without KV offload at that concurrency). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
92a20c1 to
514b3f7
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34217604862 |
1 participant
Summary / 摘要
Tuning follow-up to #2777 (GLM-5.2 FP4 MI355X SGLang Agentic MTP).
Changes / 变更内容
1. Replace HiCache DRAM offload with GPU-resident KV on both arms
GLM-5.2 uses MLA (Multi-head Latent Attention), which compresses KV to ~10–20× smaller than standard MHA. The KV working set fits entirely in HBM up to conc 8, making host-DRAM offload unnecessary in this regime.
Removing HiCache:
Validated Pareto points vs PR #2777 HiCache baseline (same runner, same dataset):
2. Search space update
Two no-KV arms replace the previous HiCache + TP8/EP=1 pair:
tp: 4, ep: 4, kv-offloading: none, conc-list: [2, 4, 6, 8]tp: 8, ep: 1, kv-offloading: none, conc-list: [4, 10]3. Optional ROCM_QUICK_REDUCE_QUANTIZATION hooks (script)
Two env-var hooks added to
glm5.2_fp4_mi355x_sglang_mtp.sh; both are no-op when unset. ActivatingROCM_QUICK_REDUCE_QUANTIZATION=INT4in the runner.envenables INT4 quick-reduce for prefill all-reduces (tensors ≥ 16 MB threshold at TP4/bf16) without requiring a script edit.摘要
对 #2777（GLM-5.2 FP4 MI355X SGLang Agentic MTP）的调优跟进。
1. 两个 arm 均改为 GPU 常驻 KV（去除 HiCache）：GLM-5.2 使用 MLA，KV 压缩比约 10–20×，在 conc≤8 时完全适配 HBM，无需主机 DRAM 卸载。去除 HiCache 可消除约 2.9 TB DRAM 固定占用，消除 DRAM→HBM 预取延迟，简化部署。
2. 搜索空间更新：两个纯 GPU KV arm 替换原有的 HiCache + TP8/EP=1 组合：
tp4/ep4 conc [2,4,6,8]和tp8/ep1 conc [4,10]。3. ROCM_QUICK_REDUCE_QUANTIZATION 钩子（脚本）：在 runner
.env中设置ROCM_QUICK_REDUCE_QUANTIZATION=INT4即可激活 INT4 快速规约（TP4/bf16 下张量 ≥16 MB 时生效），不修改脚本。Test plan / 测试计划
kv_gpu_usage≤ 85% at c8 on the TP4/EP4 arm (confirms KV stays in HBM)Note
Low Risk
Benchmark YAML and optional env forwarding only; no serving auth, data, or core runtime logic changes.
Overview
Extends
glm5.2-fp4-mi355x-sglang-agentic-mtpfrom two arms to three so SemiAnalysis can pick a Pareto point per concurrency: TP4/EP4 and TP8/EP1 withkv-offloading: none(GPU-resident MLA KV) for low–mid interactivity, plus a TP4/EP4 + HiCache arm for higher concurrency. Concurrency lists are retuned (e.g. no-KV TP4 adds c6; HiCache drops c12; TP8 no-KV drops c10).The benchmark script
glm5.2_fp4_mi355x_sglang_mtp.shforwards optional runner.envvarsROCM_QUICK_REDUCE_QUANTIZATIONandROCM_QUICK_REDUCE_CAST_BF16_TO_FP16when set (no-op otherwise) for INT4 quick-reduce experiments.perf-changelog.yamlrecords the recipe change and validation run IDs.Reviewed by Cursor Bugbot for commit 514b3f7. Bugbot is set up for automated code reviews on this repo. Configure here.