[Klaud Cold] qwen3.8next-fp8-mi355x-sglang-agentic: Qwen3.8-Flash-Next FP8 SGLang AgentX on MI355X / MI355X 上 Qwen3.8-Flash-Next FP8 SGLang AgentX 配方 - #2754
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
…link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add the day-zero Qwen3.8-Flash-Next FP8 AgentX recipe on MI355X with SGLang at TP4 and concurrency 1/4/8/12/16." | ||
| - "Run without speculative decoding because the MI355X SGLang path for this model does not drive MTP yet, so this arm carries no synthetic-acceptance pin." | ||
| - "Serve the FP8 checkpoint because no AMD FP4 checkpoint exists for this model yet." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 New appended entry uses pr-link: TBD, but the repo's canonical placeholder set (utils/validate_perf_changelog.py PR_LINK_PLACEHOLDERS) only recognizes "XXX" or "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX", not "TBD".
Extended reasoning...
When the automated squash-merge tooling (utils/merge_with_reuse.sh -> prepare_perf_changelog_merge.py canonicalize -> canonicalize_appended_links -> compare_entries -> validate_added_pr_link) runs on this PR, it checks link not in PR_LINK_PLACEHOLDERS for the appended entry's pr-link; since "TBD" is neither the placeholder nor the eventual canonical pull/ link, it raises ChangelogValidationError("appended entry N has unexpected pr-link 'TBD'") and the automated merge/canonicalization refuses, blocking this PR from merging via the normal path until the author replaces "TBD" with the conventional "XXX" placeholder.
Verification: normal — the appended entry uses an unrecognized placeholder that the merge/validation tooling rejects. perf-changelog.yaml:6501 (the new appended entry) sets pr-link: TBD. The canonical placeholder set is defined in utils/validate_perf_changelog.py:24-27 as exactly {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"} — "TBD" is not a member. The other placeholder entry in…
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2688 | ||
| - config-keys: |
There was a problem hiding this comment.
🔴 New appended entry (config-keys: qwen3.8next-fp8-mi355x-sglang-agentic) starts immediately after the prior entry's pr-link: line with zero blank-line separator, instead of the required single blank line every other entry in the file uses.
Extended reasoning...
utils/validate_perf_changelog.py's validate_raw_change() computes expected_start = b"\n- config-keys:" (since the pre-PR file does not end in a double newline) and requires the appended suffix to start with that blank line; here the suffix starts directly with - config-keys: with no leading newline, so the CI changelog gate (utils/validate_perf_changelog.py, wired into .github/workflows/run-sweep.yml) raises 'new changelog entries must be separated from history by one empty line and appended at the end' and blocks this PR from passing until the separator is added.
Verification: Severity: normal. The appended entry is missing the mandatory single blank-line separator, and the byte-sensitive append invariant that enforces it is real. FACTS: - Base file ends with a single newline: git show 9e61a20...:perf-changelog.yaml ends .../pull/2688\n (od: 2 6 8 8 \n), not \n\n. - HEAD appends - config-keys: immediately after that line with no blank line (cat -A:…
| # AgentX trace replay for Qwen3.5-397B-A17B MXFP4 on MI355X with SGLang | ||
| # native EAGLE MTP. Throughput uses the committed golden synthetic | ||
| # acceptance length; evaluation retains real target-model verification. |
There was a problem hiding this comment.
🟡 Header comment copy-pasted from qwen3.5_fp4_mi355x_sglang_mtp.sh still describes 'Qwen3.5-397B-A17B MXFP4... native EAGLE MTP' and 'committed golden synthetic acceptance length', but this script serves Qwen3.8-Flash-Next FP8 with no MTP/speculative decoding at all.
Extended reasoning...
Anyone reading this new recipe's header (docs, on-call, or a future engineer diffing scripts) is told it runs MTP with a golden synthetic acceptance length, directly contradicting the PR's own stated design (no spec decoding) and the actual command built below, risking wasted debugging time or incorrect assumptions when comparing throughput numbers across arms.
Verification: nit. Lines 5-7 of the new script benchmarks/single_node/agentic/qwen3.8next_fp8_mi355x_sglang.sh read: "# AgentX trace replay for Qwen3.5-397B-A17B MXFP4 on MI355X with SGLang / # native EAGLE MTP. Throughput uses the committed golden synthetic / # acceptance length; evaluation retains real target-model verification." This is a verbatim copy from the qwen3.5_fp4_mi355x_sglang_mtp.sh source,…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33035484584 |
…entX on MI355X Add the Qwen3.8-Flash-Next AgentX recipe on MI355X, served by SGLang. Two differences from the NVIDIA arms: no speculative decoding, because the MI355X SGLang path for this model does not drive MTP yet, so this arm runs the target model alone and carries no synthetic-acceptance pin; and FP8 rather than FP4, because no AMD FP4 checkpoint exists for this model yet. TP4 puts the 172.8 GiB checkpoint at about 43 GiB per rank. 新增 MI355X 上的 Qwen3.8-Flash-Next AgentX 配方，由 SGLang 提供服务。与 NVIDIA 分支有两点不同：其一，不启用投机解码——MI355X 上该模型的 SGLang 路径尚不支持 MTP，故本分支仅运行目标模型，也不设置合成接受长度；其二，使用 FP8 而非 FP4， 因为该模型目前没有 AMD FP4 权重。TP4 下每卡权重约 43 GiB。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recipe was copied from the Qwen3.5 MXFP4 MI355X sibling and inherited its flags. The SGLang cookbook playground's verified command for Qwen3.8-Flash-Next on MI355X is FP8, balanced, single node at --tp-size 8, with page size 32, automatic KV cache dtype rather than fp8_e4m3, chunked prefill 16384, memory fraction 0.9, watchdog 1200 and multithreaded model loading. Low-latency and high-throughput strategies are not offered for this part and NVFP4 is greyed out, so this is the whole verified AMD surface for the model today. Still no speculative decoding. 本配方原样照搬自 Qwen3.5 MXFP4 MI355X 同类配方并带入了其参数。SGLang cookbook playground 给出的 Qwen3.8-Flash-Next MI355X 验证命令为 FP8、balanced、单节点、 --tp-size 8，并使用 page size 32、KV 缓存 dtype 为 auto（而非 fp8_e4m3）、 chunked prefill 16384、显存占用 0.9、watchdog 1200 以及多线程权重加载。该硬件上 不提供 low-latency 与 high-throughput 策略，NVFP4 亦不可用，因此这就是该模型目前 在 AMD 上全部的已验证配置面。仍不启用投机解码。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
87439df to
ac87237
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33036522644 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| --enable-metrics | ||
| --enable-cache-report | ||
| "${CACHE_ARGS[@]}" | ||
| ) |
There was a problem hiding this comment.
Missing Qwen tool and reasoning parsers
Medium Severity
The SGLang launch command omits --reasoning-parser qwen3 and --tool-call-parser qwen3_coder. Those flags are part of this model's official serve command and are present on the Qwen AgentX SGLang siblings this recipe was based on. Without them, thinking tokens stay in content and tool calls are not turned into structured message.tool_calls, so the eval path and any live-assistant AgentX turns fail to parse Qwen3.8-Flash-Next output.
Reviewed by Cursor Bugbot for commit ac87237. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33036809737 |
The MI355X cells died at server launch with /opt/venv/bin/python3: No module named 'sglang' The qwen38flashnext ROCm image has no pip-installed sglang. It ships the sources under /workspace and puts them on PYTHONPATH: PYTHONPATH=/workspace/sglang-qwen-next/python:/workspace/aiter-pr4882 Reading the image layers confirms sglang/__init__.py exists only at workspace/sglang-qwen-next/python/sglang/, with the patched aiter alongside it. This launcher bind-mounts the repository at /workspace, which masks both trees, so the interpreter finds nothing to import. Mount the repository at /inferencex for images that keep packages under /workspace, and move RESULT_DIR with it, since benchmark-tmpl.yml hard-codes /workspace/results and that is the only /workspace path it sets. The host side of the mount does not change, so the workflow's artifact staging still finds everything under GITHUB_WORKSPACE. Every other image keeps the current /workspace mount untouched. MI355X 各单元在启动服务时失败：/opt/venv/bin/python3 报 No module named 'sglang'。qwen38flashnext ROCm 镜像并未通过 pip 安装 sglang，而是把源码放在 /workspace 下并写入 PYTHONPATH。查阅镜像层可确认 sglang/__init__.py 仅存在于 workspace/sglang-qwen-next/python/sglang/，打过补丁的 aiter 也在其旁。本 launcher 将仓库绑定挂载到 /workspace，恰好把这两棵目录树遮蔽，解释器因而无从导入。 对于把 python 包放在 /workspace 下的镜像，改为将仓库挂载到 /inferencex，并同步 调整 RESULT_DIR（benchmark-tmpl.yml 硬编码了 /workspace/results，且这是它设置的 唯一 /workspace 路径）。挂载的宿主机一侧不变，工作流的产物收集仍能在 GITHUB_WORKSPACE 下找到全部文件。其他镜像的 /workspace 挂载保持原样。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1564665. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33037987773 |
Moving the repo mount to /inferencex fixed the masked sglang, and then install_agentic_deps failed: error: File not found: `/workspace/utils/agentic-benchmark/requirements.txt` benchmark_lib.sh resolves AGENTIC_DIR, AIPERF_DIR and every `cd` into the repo through INFMAX_CONTAINER_WORKSPACE, which defaults to /workspace. That variable already exists as the override knob, so export it alongside RESULT_DIR instead of touching the shared library. I checked benchmark-tmpl.yml for hard-coded /workspace paths before the first attempt but not benchmark_lib.sh. 把仓库挂载点移到 /inferencex 解决了 sglang 被遮蔽的问题，随后 install_agentic_deps 报错找不到 /workspace/utils/agentic-benchmark/ requirements.txt。benchmark_lib.sh 通过 INFMAX_CONTAINER_WORKSPACE 解析 AGENTIC_DIR、AIPERF_DIR 以及所有进入仓库的 cd，其默认值为 /workspace。该变量本就 是预留的覆盖入口，因此与 RESULT_DIR 一并导出，无需改动公共库。上次只检查了 benchmark-tmpl.yml 中硬编码的 /workspace，遗漏了 benchmark_lib.sh。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33039162699 |
MI355X now gets through dependency install and model loading, and dies in the weight loader: AssertionError: Expected 1.0, got 0.00019931793212890625 in skipped model.layers.1.ple.ple_embedding.ngram_embedding.weight_scale qwen4_exp.py:2039 in load_weights The image's model code classifies the PLE ngram embedding as an unquantized module and asserts its weight_scale is 1.0, but the FP8 checkpoint genuinely quantizes it and ships a real scale. Its modules_to_not_convert lists ple.conv1d, ple.key_proj and ple.value_proj and not the ngram embedding, so the skip set is being matched too broadly on the .ple. prefix. That is a bug in this image, no serve flag reaches it, and the tag has not been rebuilt since 2026-08-26. The CUDA image carries a different SGLang build and loads the same checkpoint fine, which is why only the AMD arm hits this. Switch to Qwen/Qwen3.8-Flash-Next, 335.3 GiB with no quantization_config, so there are no scales to mis-handle. TP8 puts that at ~42 GiB per rank of 288 GB. The config key and script are renamed to bf16 accordingly. Revert to FP8 once a fixed ROCm image ships. Also fixes the file header, which still described the Qwen3.5 MXFP4 EAGLE MTP recipe this was derived from: the generator applied that substitution without asserting it matched, so the wrong description shipped from the first commit. MI355X 现已通过依赖安装并进入权重加载，随后在加载器中失败：镜像中的模型代码把 PLE ngram embedding 归类为未量化模块并断言其 weight_scale 为 1.0，而 FP8 权重确实对其 做了量化并带有真实 scale。该权重的 modules_to_not_convert 列出了 ple.conv1d、 ple.key_proj 与 ple.value_proj，并不包含 ngram embedding，说明跳过集合按 .ple. 前缀做了过宽匹配。这是镜像自身的缺陷，任何服务参数都无法绕开，且该标签自 2026-08-26 起未再重建；CUDA 镜像使用的是另一套 SGLang 构建，加载同一权重正常， 因此只有 AMD 分支受影响。 改用 Qwen/Qwen3.8-Flash-Next（335.3 GiB，完全没有 quantization_config），不存在 可被误处理的 scale；TP8 下每卡约 42 GiB（共 288 GB）。配置项与脚本名相应改为 bf16。待修复版 ROCm 镜像发布后再切回 FP8。 同时修正文件头：此前仍在描述其所派生的 Qwen3.5 MXFP4 EAGLE MTP 配方——生成脚本 执行该替换时未做断言，导致错误描述从首次提交起一直存在。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
functionstackx
changed the title
Reverting the BF16 switch: this arm stays FP8 on Qwen/Qwen3.8-Flash-Next-FP8, per instruction. The key, model and script name go back to fp8; the corrected file header is kept and rewritten to describe the FP8 arm. It will keep failing on the current image, and it is worth writing down why so nobody spends the debugging round again. The server aborts in load_weights: AssertionError: Expected 1.0, got 0.00019931793212890625 in skipped model.layers.1.ple.ple_embedding.ngram_embedding.weight_scale The model registers no parameter for the PLE ngram embedding, so the loader takes its "skipped" branch, which assumes any orphaned _scale is a no-op and asserts it is 1.0. Reading the checkpoint directly: that embedding is stored as 128 shard_N.weight tensors of dtype F8_E4M3 with one BF16 weight_scale of ~1.99e-4, and modules_to_not_convert lists ple.conv1d, ple.key_proj and ple.value_proj but not the ngram embedding. So the tensor really is quantized and the model simply cannot consume it yet. Suppressing the assert would not fix it: the shards would load as raw FP8 with the scale never applied, leaving that embedding wrong by roughly 5000x and no error reported. That is silent corruption, so it is deliberately not done. The CUDA image carries a different SGLang build and loads the same checkpoint, so this is specific to the ROCm build, and the tag has not been rebuilt since 2026-08-26. The recipe is otherwise the cookbook's verified balanced command and needs no further change once a fixed image ships. 按指示回退 BF16 改动：本分支恢复为 FP8，使用 Qwen/Qwen3.8-Flash-Next-FP8；配置项、 权重与脚本名均改回 fp8，同时保留并改写了修正后的文件头以描述 FP8 分支。 在当前镜像上它仍会失败，原因值得写清楚，以免重复排查。服务在 load_weights 阶段 中止：模型未为 PLE ngram embedding 注册参数，加载器因而走「skipped」分支，该分支 假定任何无对应参数的 _scale 都是空操作并断言其为 1.0。直接读取权重可见：该 embedding 由 128 个 dtype 为 F8_E4M3 的 shard_N.weight 与一个 ~1.99e-4 的 BF16 weight_scale 组成，而 modules_to_not_convert 只列出 ple.conv1d、ple.key_proj 与 ple.value_proj，并不含 ngram embedding。也就是说该张量确实被量化，只是模型尚无法 使用它。 屏蔽该断言并不能解决问题：这样会以原始 FP8 加载分片而从不施加 scale，使该 embedding 偏差约 5000 倍且不报任何错误。这属于静默的数值破坏，因此刻意不做。CUDA 镜像使用另一套 SGLang 构建，可正常加载同一权重，故该问题仅限 ROCm 构建，且该标签 自 2026-08-26 起未重建。除此之外本配方就是 cookbook 的验证命令，待修复镜像发布后 无需再改。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
functionstackx
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33040096927 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33040362439 |
1 participant
Summary / 摘要
Qwen3.8-Flash-Next AgentX recipe on MI355X, served by SGLang. SGLang is this model's plan-of-record engine per
MODELS.md. This is the AMD arm of the four-chip set and the only one without speculative decoding.MI355X 上的 Qwen3.8-Flash-Next AgentX 配方，由 SGLang 提供服务。按
MODELS.md，SGLang 是该模型的 PoR 引擎。本 PR 是四芯片系列中的 AMD 分支，也是唯一不启用投机解码的一个。Config key / 配置项
qwen3.8next-fp8-mi355x-sglang-agentic—Qwen/Qwen3.8-Flash-Next-FP8,lmsysorg/sglang-rocm:qwen38flashnext,cluster:mi355x-amds, TP8/EP1, conc[1, 4, 8, 12, 16], no KV offloading.Two deliberate differences from the NVIDIA arms / 与 NVIDIA 分支的两点差异
-mtpsuffix, the search space carries nospec-decoding, and the script has no--speculative-*flags and noSGLANG_SIMULATE_ACC_*pin — it is the one Qwen3.8-Flash-Next AgentX arm without a synthetic-acceptance target. Add MTP and the golden AL once the ROCm path supports it. Note this makes the arm not directly comparable to the spec-decode NVIDIA arms on the published frontier.This arm cannot pass on the current ROCm image. The server aborts in
load_weights:The model registers no parameter for the PLE ngram embedding, so the loader takes its "skipped" branch, which assumes any orphaned
_scaleis a no-op and asserts it is 1.0. Reading the checkpoint directly says otherwise: that embedding is 128shard_N.weighttensors of dtypeF8_E4M3with one BF16weight_scaleof ≈1.99e-4, andmodules_to_not_convertlistsple.conv1d,ple.key_proj,ple.value_proj— not the ngram embedding. The tensor really is quantized; the model just cannot consume it yet.Suppressing the assert is not a fix. The shards would load as raw FP8 with the scale never applied, leaving that embedding wrong by ~5000× with no error reported. That is silent corruption, so it is deliberately not done.
The CUDA image carries a different SGLang build and loads the same checkpoint fine, so this is specific to the ROCm build, and
lmsysorg/sglang-rocm:qwen38flashnexthas not been rebuilt since 2026-08-26. Everything else here is the cookbook's verified balanced command; this PR is ready to pass the moment a fixed ROCm image ships, with no further recipe change.本分支在当前 ROCm 镜像上无法通过。 模型未为 PLE ngram embedding 注册参数，加载器走「skipped」分支并断言
_scale为 1.0；而权重中该 embedding 实为 128 个F8_E4M3的shard_N.weight加一个 ≈1.99e-4 的 BF16weight_scale，modules_to_not_convert并未包含它。屏蔽该断言不是修复：分片会以原始 FP8 加载且从不施加 scale，使该 embedding 偏差约 5000 倍且不报错，属静默数值破坏，故刻意不做。CUDA 镜像使用另一套 SGLang 构建可正常加载，问题仅限 ROCm 构建，该标签自 2026-08-26 未重建。其余部分即 cookbook 验证命令，待修复镜像发布后本 PR 无需再改即可通过。-mtp后缀，搜索空间不含spec-decoding，脚本中没有--speculative-*参数，也没有SGLANG_SIMULATE_ACC_*锁定——这是 Qwen3.8-Flash-Next 各 AgentX 分支中唯一没有合成接受目标的一个。待 ROCm 路径支持后再补上 MTP 与黄金 AL。需注意，这使该分支在已发布前沿上与启用投机解码的 NVIDIA 分支不具备直接可比性。Qwen/Qwen3.8-Flash-Next-FP8（172.8 GiB）。Qwen3.5 的 MI355X 同类配方使用amd/Qwen3.5-397B-A17B-MXFP4，此处无对应权重。Recipe decisions / 配方要点
TP4/EP1. 172.8 GiB is about 43 GiB per rank on MI355X's 288 GB, leaving ample room for the 256k-capped agentic traces.
Image.
lmsysorg/sglang-rocm:qwen38flashnext, the ROCm model bring-up tag published 2026-08-26 (verified on Docker Hub).Serve flags are carried from
qwen3.5_fp4_mi355x_sglang_mtp.shminus the speculative block:aiterattention backend,SGLANG_USE_AITER/SGLANG_USE_AITER_UNIFIED_ATTN/AITER_FLYDSL_FORCE, page size 16,fp8_e4m3KV cache, bf16 Mamba SSM, 32768 prefill/chunked-prefill.No launcher change needed:
runners/launch_mi355x-amds.shresolvesagentic/<prefix>_<precision>_mi355x_<framework>.shwith an empty spec suffix whenSPEC_DECODINGis notmtp— verified against the generated matrix (exp-name qwen3.8next_tp4_conc1_kvnone,spec-decoding: none).TP4/EP1：MI355X 288 GB 下每卡权重约 43 GiB，为 256k 上限的智能体轨迹留出充足显存。
镜像：
lmsysorg/sglang-rocm:qwen38flashnext（2026-08-26 发布的 ROCm 模型适配标签，已核实）。服务参数沿用
qwen3.5_fp4_mi355x_sglang_mtp.sh并去掉投机解码部分。无需改动 launcher：
SPEC_DECODING非mtp时后缀为空，已按生成矩阵核实脚本名可解析。Files / 改动文件
benchmarks/single_node/agentic/qwen3.8next_fp8_mi355x_sglang.sh(new)configs/amd-master.yaml— new entry after the Qwen3.5 MI355X SGLang siblingperf-changelog.yaml— appended entryPart of a four-PR set, one per chip: B200 (#2751), B300 (#2752), H200 (#2753), MI355X (this).
本 PR 属于按芯片划分的四个 PR 之一：B200（#2751）、B300（#2752）、H200（#2753）、MI355X（本 PR）。
🤖 Generated with Claude Code
Note
Low Risk
Changes are limited to benchmark scripts, AMD config, changelog, and a container mount workaround for one bring-up image; no auth, data, or serving core logic changes.
Overview
Adds the day-zero AgentX recipe for Qwen3.8-Flash-Next FP8 on MI355X using SGLang (
qwen3.8next-fp8-mi355x-sglang-agentic): new agentic benchmark script,amd-master.yamlentry at TP8/EP1 with conc 1/4/8/12/16, no KV offloading, and a perf-changelog note.Unlike the NVIDIA arms, this AMD path omits speculative decoding (no MTP / synthetic-acceptance pin) because the MI355X SGLang stack for this model does not drive MTP yet. It targets the FP8 checkpoint because there is no AMD FP4 weight and NVFP4 is unavailable on MI355X. Serve flags follow the SGLang cookbook balanced single-node recipe (e.g. TP8, page size 32, auto KV dtype, chunked prefill 16384, mem fraction 0.9, multithreaded load).
launch_mi355x-amds.shnow mounts the repo at/inferencex(not/workspace) forqwen38flashnextimages so the bind mount does not hide the image’s sglang/aiter trees onPYTHONPATH;RESULT_DIRandINFMAX_CONTAINER_WORKSPACEmove with that mount.The script header documents an upstream ROCm image blocker: loading
Qwen/Qwen3.8-Flash-Next-FP8can abort on a PLE ngram embeddingweight_scaleassert until the image correctly loads that quantized tensor.Reviewed by Cursor Bugbot for commit 4b02f48. Bugbot is set up for automated code reviews on this repo. Configure here.