DIAGNOSTIC: unserialized control for the MI355X serialize experiment (#2641) - #2655
DIAGNOSTIC: unserialized control for the MI355X serialize experiment (#2641)#2655sammshen wants to merge 54 commits into
Conversation
Add an lmcache kv-offload-backend point at TP8 conc 10 on top of the existing DSpark MTP serving stack, mirroring the vllm-simple offload arm for a direct backend comparison. The benchmark script gains an lmcache case arm that installs the LMCache 0.5.4rc1 ROCm wheel (torch/ROCm stack untouched), starts one MP server per the Kimi-K3 recipe (chunk size 768 = K3 unified block size at 8 GPUs, --separate-object-groups for the hybrid KDA/MLA two-group KV layout, --enable-extra-logging, --max-cpu-workers 8 --max-gpu-workers 1), and wires vLLM to it via LMCacheMPConnector.
A separate kimik3-fp4-mi355x-vllm-agentic-mtp-lmcache key lets the changelog select only the LMCache points instead of re-running the resident and vllm-simple arms of the base key. The base key returns to its upstream shape.
The LMCache MP server's L1 lives in /dev/shm and the script rejects budgets above 90% of free shm. mi355x-amds nodes mount ~1.5 TB of shm (cap ~1360 GB), so 0.50's 1499 GB budget failed the check in run 31644286169. 0.40 generates ~1199 GB, which fits with margin.
vLLM sizes the K3 unified attention block to 1536 tokens on the MI355X fp8-KV TRITON_MLA path (attention page >= mamba page), and the MP connector asserts chunk %% block == 0, so the recipe's CUDA-path 768 fails engine init (run 31644990546).
The connector requires the chunk to be a multiple of every engine KV group's tokens_per_block. On this stack the hybrid layout registers attention groups at 1536 and a KDA state group at 3072 (run 31645828378), so 1536 fails registration; 3072 is the minimum valid chunk.
Auto mode loads both transfer paths; pin server-driven STORE/RETRIEVE (as the MiniMax-M3 arm does) so the benchmark measures one deterministic path. The L1 stays shm-backed either way, so the /dev/shm capacity check still applies.
The default 300s read-lock TTL expires under a single GPU worker serializing huge K3 transfers: run 31648224111 logged 57k finish-read-on-non-read-locked-key warnings starting exactly at warmup+300s, followed by a GPU illegal-access crash mid-profile. Match the MiniMax-M3 arm's 7200s read TTL.
This reverts commit 20b4fda.
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
# Conflicts: # perf-changelog.yaml
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
…SemiAnalysisAI/InferenceX into amd/kimi-k3-agentic-perf-tuning
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
Signed-off-by: Sirra <asirra@amd.com>
# Conflicts: # perf-changelog.yaml
The previous merge left conflict markers in the file and my scripted fallback asserted before rewriting it, so the broken version was committed and pushed. Rebuild deterministically as 'upstream/main verbatim + this PR's entry': zero deletions vs main, valid YAML, 733 entries, #2602's entry absent so its sweep no longer runs here.
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Not for merge. Adds kimik3-fp4-mi355x-vllm-agentic-mtp-nokv: identical to the -lmcache key (same image nightly-ac7509e2b1, same script, TP8/EP1, spec-mtp, conc 4/8/10/12/16) with kv-offloading: none, so the run starts no LMCache server and passes no --kv-transfer-config. Purpose: isolate the hipErrorIllegalAddress crashes. They have appeared on mia1-p01-g11 across the LMCache arm, #2602's vllm-simple arm and two LMCache versions -- 0-for-8 on that node -- which points at the node rather than the connector. This runs the same points with the connector absent to confirm that directly. Expected: if the crashes persist with no KV transfer config, they are not LMCache's. If they vanish, the connector is implicated after all.
# Conflicts: # perf-changelog.yaml
Not for merge. Adds kimik3-fp4-mi355x-vllm-agentic-mtp-serialize and a
DIAGNOSTIC env block in kimik3_fp4_mi355x_mtp.sh that sets
AMD_SERIALIZE_KERNEL=3 plus TORCH_USE_HIP_DSA/TORCH_USE_CUDA_DSA.
Every traceback collected for the hipErrorIllegalAddress so far is unusable
for locating it. HIP reports kernel faults asynchronously, so the Python
stack unwinds at whatever CUDA call next checks the error flag. In our logs
that is triton's _init_handles() -> get_current_device() while *loading*
_fwd_kernel_stage2 -- a kernel that had not been launched yet. vLLM prints
the caveat itself ("the stacktrace below might be incorrect"), and the
mid-run variant of the same crash unwinds at SetDevice with no Python frames
at all. Serializing launches makes the fault surface at the kernel that
actually causes it.
Uses vllm-simple rather than LMCache deliberately: SimpleCPUOffloadConnector
is vLLM's own CPU offload and reproduces the identical crash with no LMCache
in the picture, which keeps the repro minimal. Both c8 and c10 failed on that
backend in run 31837518310. The crash reproduces during warmup, so the run
only needs to reach startup; serialized launches make every timing from this
key meaningless.
Also repairs the #2635 changelog entry, whose pr-link had been overwritten by
a stray duplicate fragment of the #2609 entry (duplicate YAML key, last one
winning), leaving #2635 credited to #2609 and #2609 listed twice.
…lize # Conflicts: # perf-changelog.yaml
The branch was cut from kimik3-mi355x-nokv, so it carried that PR's changelog entry as well and the sweep selected both keys -- re-running the five nokv points that already have clean results, under AMD_SERIALIZE_KERNEL=3, on the same MI355X nodes the serialize jobs need. Those runs would be enormously slow and their numbers meaningless. Drop the nokv entry here so only kimik3-fp4-mi355x-vllm-agentic-mtp-serialize sweeps. The entry stays on #2635, which owns it; this removes nothing that exists on main.
Two points is too few to say anything about an intermittent crash. All four of these failed on vllm-simple in run 31837518310, so they are the points with a known unserialized failure to compare against.
Not for merge. Adds kimik3-fp4-mi355x-vllm-agentic-mtp-noserialize: the same vllm-simple DRAM offload points as the -serialize key on #2641 (c8/c10/c12/c16), same recipe, with DIAG_SERIALIZE_KERNEL defaulting to 0 so AMD_SERIALIZE_KERNEL is never set and launches stay asynchronous. The serialized arm did not reproduce the hipErrorIllegalAddress on either point it ran. That is only meaningful against a contemporaneous unserialized control: the crash is intermittent, and the same LMCache config went 4/4 on 08-14 and 0/5 on 08-16, so pass/fail counts from different days are not comparable. This runs the control on the same cluster on the same day. If this arm crashes where the serialized arm did not, serialization is what suppresses the fault, which makes it a race rather than a deterministic out-of-bounds access. If neither arm crashes, the repro itself has drifted and the earlier failures need re-establishing before anything else is concluded.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| _patch() { | ||
| local target="$1" marker="$2" label="$3" | ||
| if [ -z "$target" ] || [ ! -f "$target" ]; then | ||
| echo "[$label] target not found; skipping." | ||
| return 0 | ||
| fi | ||
| if grep -q "$marker" "$target"; then | ||
| echo "[$label] already patched." | ||
| return 0 | ||
| fi | ||
| cp -n "$target" "$target.orig" 2>/dev/null || true | ||
| if $PY - "$target" "$label"; then | ||
| return 0 | ||
| else | ||
| echo "[$label] patch failed; left unchanged." >&2 | ||
| return 0 | ||
| fi | ||
| } |
There was a problem hiding this comment.
🟡 The generic _patch() helper in apply_k3_container_patches.sh (lines 41-58) documents the marker-check/backup/apply/error-handling convention but has zero call sites — grep -n '_patch\\b' only matches its own definition and the doc comment above it. All three real patch functions (patch_aiter_pybind11, patch_triton_mla_cudagraph, patch_kv_blockpool) reimplement that same sequence inline instead of calling it, so a fix to the apply/backup/error-handling logic now needs three edits instead of one. Either wire the three functions through _patch or delete the unused helper.
Extended reasoning...
What's wrong: apply_k3_container_patches.sh defines a generic _patch() helper at lines 41-58 with a clearly documented calling convention — check the target exists, skip if the post-patch marker is already present, back up to $target.orig, run the python heredoc apply, and report/no-op on failure. This is exactly the convention the file's own header comment (lines 3-8) advertises: 'Each patch is idempotent, verifies its own anchor, backs up to <file>.orig, and no-ops if the image already ships the fix.'
However, _patch is never actually called. Running grep -n '_patch\\b' apply_k3_container_patches.sh turns up only the doc comment referencing it and its own definition — zero call sites anywhere in the ~260-line file. It is dead code.
Where the duplication lives: Instead of calling _patch, each of the three real patch functions reimplements the identical sequence inline:
patch_aiter_pybind11: target-exists check,grep -q _use_torch_pybind11marker check,cp -n "$target" "$target.orig", a $PY heredoc apply, and an|| echo '[aiter-pybind11] patch failed; unchanged.'fallback.patch_triton_mla_cudagraph: the same check-target / grep-marker (AttentionCGSupport.UNIFORM_BATCH) /cp -n .orig/ $PY heredoc / failure-echo sequence.patch_kv_blockpool: the same shape again, gated on theKIMI-PATCH-KV-BLOCKPOOLmarker.
Why this matters: Because none of the three functions route through _patch, a bug in that common sequence — e.g. the backup logic, the marker-check ordering, or the failure-reporting behavior — has to be fixed in three separate places rather than one. Conversely, _patch itself can silently drift out of sync with what the real functions do (it already has: the real functions pass the python script via stdin heredoc plus "$label" as an argument, a convention _patch's own comment documents but no caller demonstrates), so nobody would notice if it broke.
Proof by inspection:
grep -n '_patch()' apply_k3_container_patches.sh→ line 41 (definition) only.grep -n '_patch "' apply_k3_container_patches.shorgrep -n '\\b_patch\\b' apply_k3_container_patches.shexcluding the definition and the doc comment at line 39 → no results.- Reading
patch_aiter_pybind11,patch_triton_mla_cudagraph, andpatch_kv_blockpoolin turn shows each hand-rolling:if [ -z "$target" ] || [ ! -f "$target" ],grep -q <marker> "$target",cp -n "$target" "$target.orig" 2>/dev/null || true, then$PY - "$target" <<'EOF' ... EOF || echo '[label] patch failed; unchanged.'— the exact sequence_patch's comment describes.
Fix: Either delete the unused _patch() helper (least effort, but loses the documented convention as a factored point), or refactor the three functions to call it (more invasive, since patch_aiter_pybind11 also has a pybind11-version-mismatch check and JIT-artifact cleanup that don't fit the helper's exact signature, and each function's python heredoc differs in its target file and edit). Given this is explicitly a diagnostic/not-for-merge PR, this is a nit rather than a blocker.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32167736488 |
4 participants
Diagnostic only — not for merge. Control arm for #2641.
Why
#2641 ran the same
vllm-simpleDRAM-offload points withAMD_SERIALIZE_KERNEL=3and did not reproduce thehipErrorIllegalAddress(run 32094050099 — c10 clean, c8 died of an unrelatedHSA_STATUS_ERROR_OUT_OF_RESOURCESthat serialization itself likely caused).That result is only meaningful against a contemporaneous control. The crash is intermittent — the same LMCache config went 4/4 on 08-14 and 0/5 on 08-16 — so comparing a serialized run today against unserialized failures from four days ago proves nothing.
What this is
Identical recipe and search space to the
-serializekey. The only difference:so
AMD_SERIALIZE_KERNELis never exported and launches stay asynchronous.Both keys sweep c8, c10, c12, c16 — all four failed on
vllm-simplein run 31837518310, so every point has a known unserialized failure to compare against.Reading the result
Context
Crashes occur only with DRAM KV offload and are backend-agnostic (LMCache and
SimpleCPUOffloadConnectorproduce byte-identical tracebacks). The no-KV control (#2635) passed 5/5 with DSpark fully active. Full writeup: Confluence.Note:
TORCH_USE_HIP_DSAdoes not work as a runtime env var — logs still print "Device-side assertion tracking was not enabled by user." It needs a torch built with it.