[Klaud Cold] qwen3.8next-fp4-b200-sglang-agentic-mtp: day-zero Qwen3.8-Flash-Next NVFP4 SGLang AgentX on B200 / B200 上 Qwen3.8-Flash-Next NVFP4 SGLang AgentX 首发配方 - #2751
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
…link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33035370026 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add the day-zero Qwen3.8-Flash-Next NVFP4 AgentX recipe on B200 with SGLang native NEXTN MTP at TP4 and concurrency 1/4/8/12/16." | ||
| - "Serve RadixArk/Qwen3.8-Flash-Next-NVFP4 with modelopt_fp4 quantization, the trtllm_mha attention backend, and the flashinfer_trtllm MoE runner, following the Qwen3.5 NVFP4 B200 sibling." | ||
| - "Pin throughput runs to an interim acceptance length of 3.24 measured at three speculative tokens; eval-only runs keep real target verification." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 New changelog entry uses pr-link: TBD, which is not an accepted placeholder.
Extended reasoning...
utils/validate_perf_changelog.py's validate_added_pr_link() only accepts the exact expected PR URL or the literal placeholders "XXX"/".../pull/XXX" (PR_LINK_PLACEHOLDERS in utils/validate_perf_changelog.py:24-27); "TBD" matches neither, so the changelog validation gate raises ChangelogValidationError and this PR's changelog check fails CI until pr-link is changed to XXX or the real PR number.
Verification: normal. The diff appends a changelog entry ending in pr-link: TBD (perf-changelog.yaml:6501, per git diff). validate_added_pr_link (utils/validate_perf_changelog.py:144-160) rejects "TBD" on both code paths: on a PR run (pr_number set) it requires link in PR_LINK_PLACEHOLDERS (only "XXX" or ".../pull/XXX", lines 24-27) or an exact .../pull/{pr_number} match, and on a main-branch…
| - "Required lanes promote exporter startup timeouts, launch failures, and endpoint resolution failures to blocking validation failures, so a recipe cannot publish a result whose power collection never started." | ||
| - "Route only enabled recipes through the immutable producer fork and preserve non-power launcher revisions." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2688 | ||
| - config-keys: |
There was a problem hiding this comment.
🔴 New changelog entry is appended directly after the prior entry's pr-link line (6492) with no blank-line separator, violating utils/prepare_perf_changelog_merge.py's validate_raw_change, which requires appended entries to follow the last historical byte with exactly \n- config-keys: (i.e. one blank line) when the base file doesn't already end in a blank line. Verified via byte inspection: base perf-changelog.yaml ends ...pull/2688\n and HEAD continues immediately with - config-keys: (no blank line).
Extended reasoning...
When merge automation runs canonicalize_appended_links()/resolve_conflict_bytes() (utils/prepare_perf_changelog_merge.py lines 90-97, 147-155) to canonicalize this entry's pr-link, validate_raw_change raises ChangelogValidationError('new changelog entries must be separated from history by one empty line and appended at the end') even after the separately-flagged pr-link placeholder is fixed, blocking merge until a blank line is inserted before line 6493.
Verification: normal (merge-prep automation raises an error the base branch does not). Byte inspection confirms base perf-changelog.yaml ends with a single newline (git show BASE:perf-changelog.yaml | od -> ...p u l l / 2 6 8 8 \n, one \n, no trailing blank line). HEAD appends the new block immediately: the diff shows pr-link: https://.../pull/2688 followed directly by +- config-keys: with no…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33036088519 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33036486451 |
…link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
45baa93 to
ca5dee9
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33036804323 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33036862898 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33037090816 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33038261431 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33038511086 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33039174779 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e2ed089. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33043922030 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33044903141 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33044903141 |
…link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2d0ff52 to
7043888
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7043888. Configure here.
| TOKENIZER_ARGS=() | ||
| if [ "$TP" -ge 4 ]; then | ||
| TOKENIZER_ARGS=(--tokenizer-worker-num 6) | ||
| fi |
There was a problem hiding this comment.
Tokenizer workers skipped at TP1
Medium Severity
Parallel tokenization is gated on TP >= 4, but this recipe only searches tp: 1, so --tokenizer-worker-num never applies. The nearby comment says 256k AgentX warmups miss the client timeout without it, and the same model's H100 arm always sets 6 tokenizer workers. Tokenization is CPU-side, so TP1 does not remove that cost.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 7043888. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33113497147 |
|
/reuse-sweep-run |
…AgentX on B200 Add the first Qwen3.8-Flash-Next AgentX recipe on B200, served by SGLang with native NEXTN MTP. SGLang is the plan-of-record engine for this model per MODELS.md. Serves RadixArk/Qwen3.8-Flash-Next-NVFP4 at TP4 with modelopt_fp4 quantization, following the Qwen3.5 NVFP4 B200 sibling. Throughput runs pin an interim acceptance length of 3.24 at three speculative tokens; eval-only runs keep real target verification. 新增 B200 上首个 Qwen3.8-Flash-Next AgentX 配方，由 SGLang 以原生 NEXTN MTP 提供服务。按 MODELS.md，SGLang 是该模型的 PoR 引擎。使用 RadixArk/Qwen3.8-Flash-Next-NVFP4，TP4，modelopt_fp4 量化，参照 Qwen3.5 NVFP4 B200 同类配方。吞吐运行按 3 个投机 token 锁定临时接受长度 3.24，仅评测运行仍使用 真实目标验证。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All six B200 cells failed in one minute with "Unsupported model prefix/ precision: qwen3.8next/fp4". launch_b200-nscale-compat.sh resolves every model-prefix + precision pair to a pre-staged directory under /scratch/models through a hardcoded if-chain, and qwen3.8next had no branch, so it fell to the else and exited 1 before Slurm was ever touched. Add the branch, following the dsv4 pattern: prefer an explicitly supplied MODEL_PATH, then the first staged candidate directory, so a differently named staging dir needs no further code change. This launcher bind-mounts MODEL_PATH into the container and exports MODEL=$MODEL_PATH, so unlike the B300, H200 and MI355X paths there is no hf-download fallback and the checkpoint has to be staged on the nscale cluster for the sweep to pass. B200 的全部 6 个单元在一分钟内失败，报错 "Unsupported model prefix/precision: qwen3.8next/fp4"。launch_b200-nscale-compat.sh 通过硬编码的 if 链把每组 model-prefix + precision 解析到 /scratch/models 下预置的权重目录，其中没有 qwen3.8next 分支，因此落入 else 并在进入 Slurm 之前即以 1 退出。 参照 dsv4 的写法补上该分支：优先使用显式提供的 MODEL_PATH，其次取第一个存在的 候选目录，这样即使权重目录换名也无需再改代码。该 launcher 会把 MODEL_PATH 绑定挂载进容器并导出 MODEL=$MODEL_PATH，因此与 B300、H200、MI355X 路径不同， 它没有 hf 下载兜底，权重必须预先在 nscale 集群上就位，sweep 才可能通过。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recipe was copied from the Qwen3.5 NVFP4 B200 sibling, which is a much larger model, so it inherited flags this one does not want. The SGLang cookbook playground's verified single-node command for Qwen3.8-Flash-Next on B200 is --tp 1: 126 GiB of NVFP4 weights fit on one 192 GB part, so nothing is sharded. It also takes the hybrid GDN linear-attention backends rather than a single --attention-backend, reads quantization from the checkpoint instead of --quantization modelopt_fp4, and drops the trtllm_mha / flashinfer_trtllm MoE flags that belong to the Qwen3.5 recipe. --max-running-requests stays explicit, since NEXTN silently resets it to 48 when unset. Acceptance length moves from the interim 3.24 to 2.32, the thinking_on value at three speculative tokens from the committed golden curve. 本配方原样照搬自 Qwen3.5 NVFP4 B200 同类配方，而后者模型规模大得多，因此带入了 本模型并不需要的参数。SGLang cookbook playground 给出的 Qwen3.8-Flash-Next B200 单节点验证命令为 --tp 1：126 GiB 的 NVFP4 权重可放入单张 192 GB 卡，无需 切分。该命令改用混合 GDN 线性注意力的独立后端而非单一 --attention-backend，量化 直接从权重读取而不传 --quantization modelopt_fp4，并去掉了属于 Qwen3.5 配方的 trtllm_mha 与 flashinfer_trtllm MoE 参数。--max-running-requests 保持显式设置， 因为 NEXTN 在其未设置时会静默重置为 48。 接受长度由临时值 3.24 改为 2.32，即已提交黄金曲线中 3 个投机 token 的 thinking_on 取值。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Qwen3.8-Flash-Next NVFP4 is not pre-staged under /scratch/models, and every other branch in this launcher resolves to that tree, so the previous commit would have cleared the prefix gate only to fail on the bind mount. Point the branch at a writable cache directory on the shared /scratch filesystem instead and let the bench script's own `hf download --local-dir "$MODEL_PATH"` populate it on the first run. That needs MODEL to stay a HuggingFace repo id, because `hf download` cannot take a local path, while this launcher otherwise overwrites MODEL with MODEL_PATH for every model. KEEP_HF_MODEL_ID gates that overwrite, and the directory is created up front because srun fails outright when a bind mount source is missing. Qwen3.8-Flash-Next NVFP4 并未预置在 /scratch/models 下，而本 launcher 其余分支 都指向该目录树，因此上一个提交虽然过了前缀校验，却会在绑定挂载处失败。改为指向 共享 /scratch 文件系统上的可写缓存目录，由基准脚本自身的 `hf download --local-dir "$MODEL_PATH"` 在首次运行时填充。 这要求 MODEL 保持为 HuggingFace 仓库 id（`hf download` 不接受本地路径），而本 launcher 对其他模型都会把 MODEL 覆盖为 MODEL_PATH。KEEP_HF_MODEL_ID 用于跳过该 覆盖；目录提前创建，因为绑定挂载源缺失时 srun 会直接失败。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified directly on the nscale cluster: /scratch is not the same filesystem on both sides of a job. On the login node it is a symlink to NFS /data/scratch, which is root-owned and holds almost nothing. On a compute node it is node-local /dev/md0 xfs, which holds the staged models and is not writable by the runner account. So the previous commit's /scratch/hf-models path would have been created on NFS by the launcher and then not existed on the compute node where srun evaluates the bind mount. The home directory is the one path that is shared, writable and identical on both: /data/home/sa-shared on 24T of NFS with 5.8T free, confirmed writable from a compute node. Point MODEL_PATH there. Qwen3.8-Flash-Next is staged nowhere on this cluster, so the first run downloads it. 已在 nscale 集群上直接核实：/scratch 在作业两侧并非同一个文件系统。登录节点上它是 指向 NFS /data/scratch 的符号链接，属 root 且几乎为空；计算节点上它是节点本地的 /dev/md0 xfs，存放已预置的权重，且运行账号不可写。因此上一个提交使用的 /scratch/hf-models 路径会由 launcher 在 NFS 上创建，而在 srun 评估绑定挂载的计算 节点上并不存在。 家目录是两侧共享、可写且路径一致的唯一位置：/data/home/sa-shared，位于 24T NFS 上，剩余 5.8T，已确认可从计算节点写入。MODEL_PATH 改指该处。该集群未预置 Qwen3.8-Flash-Next，首次运行将自行下载。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
flashinfer's gated_delta_rule_mtp asserts initial_state.dtype == torch.float32 and aborts CUDA graph capture on the bfloat16 SSM state the cookbook command specifies. Confirmed on the H200 arm, which died at "Capture target verify CUDA graph"; this arm runs the same GDN backend, the same kernel and the same NEXTN, so it would fail identically once it got that far. flashinfer 的 gated_delta_rule_mtp 断言 initial_state 必须为 float32，遇到 cookbook 命令指定的 bfloat16 SSM 状态会在 CUDA graph 捕获阶段中止。该问题已在 H200 分支确认；本分支使用相同的 GDN 后端、相同内核与相同的 NEXTN，运行到同一阶段 必然同样失败。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
My previous commit copied the H200 float32 fix here, and SGLang refused to start at all: ValueError: --linear-attn-decode-backend flashinfer on SM100+ requires --mamba-ssm-dtype bfloat16, got 'float32' The two architectures want opposite things. On SM100+ the flashinfer linear attention decode backend requires a bfloat16 state and validates it in server_args before launch. On Hopper the same backend's MTP verify kernel asserts a float32 state during CUDA graph capture. So the cookbook's bfloat16 is correct here and the H200 arm is the exception, not the other way round. Reverted, with the asymmetry written down so neither arm gets 'fixed' to match the other again. 上一个提交把 H200 的 float32 改动照搬到本分支，导致 SGLang 直接拒绝启动：SM100+ 上使用 flashinfer 线性注意力解码后端时必须搭配 bfloat16。两种架构的要求正好相反： SM100+ 在 server_args 中校验并要求 bfloat16；而 Hopper 上同一后端的 MTP 验证内核 在 CUDA graph 捕获阶段要求 float32。因此 cookbook 的 bfloat16 在此处是正确的， H200 才是例外。现已回退，并把这一不对称写入注释，避免两侧再被互相“对齐”。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
B300 passed its evals; B200 ran the identical recipe and returned
gsm8k exact_match 0.0000 on both strict-match and flexible-extract. The server
started and served, so this was not an infrastructure failure: it was serving a
broken model.
Inspecting the nscale cluster showed why. The shared checkpoint directory held
81 GB of 126 GB, 396 of 419 files, 8 leftover .incomplete files and no
model.safetensors.index.json. The download had stalled, and the guard here only
asked whether the directory was non-empty, so every cell accepted the partial
tree and loaded whatever shards happened to be present. The eval gate is what
caught it; the throughput numbers from those cells would otherwise have been
published as if they were real.
Three changes, all inside the script rather than by pre-staging weights:
* Completeness is now decided by the index and every shard it names, plus the
absence of .incomplete files, instead of by emptiness. hf download is
incremental, so re-running it against a partial tree resumes cheaply.
* The download takes a flock around the shared directory. Checking
completeness alone would have turned "one cell downloads" into "all six
download into the same tree at once", and hf download names .incomplete
files by content hash, so concurrent writers collide on the same paths.
The first holder downloads; the rest wake, find it complete and skip. The
launcher already uses flock this way for squashfs imports.
* HF_HUB_DISABLE_XET=1 for the download. Xet stalled this transfer twice at
exactly 81 GB, once from a login node and once from inside a job; the plain
HTTPS path sustained 861 MB/s.
The stale 81 GB tree has been removed from the cluster so the next run starts
clean. Also verified the completeness helper against an empty directory, an
index with no shards, a partial shard set, a full set, and a leftover
.incomplete file.
B300 的评测通过，而 B200 以完全相同的配方返回 gsm8k exact_match 0.0000。服务本身
正常启动并响应，因此这不是基础设施故障，而是在为一个损坏的模型提供服务。
在 nscale 集群上核查发现：共享权重目录只有 126 GB 中的 81 GB、419 个文件中的 396
个、8 个残留 .incomplete 文件，且缺少 model.safetensors.index.json。下载中途停滞，
而此处的判据仅检查目录是否非空，于是每个单元都接受了这棵残缺的目录树并加载了碰巧
存在的分片。是评测环节兜住了这个问题，否则这些单元的吞吐数据会被当作有效结果发布。
三处改动，全部在脚本内完成，不采用预置权重的做法：完整性改为依据索引及其列出的每个
分片、并要求不存在 .incomplete 文件；下载前对共享目录加 flock（仅做完整性检查会使
「一个单元下载」变成「六个单元同时下载同一棵目录树」，而 hf download 的 .incomplete
文件按内容哈希命名，并发写入会相互冲突）；下载时设置 HF_HUB_DISABLE_XET=1（Xet 两次
都恰好停滞在 81 GB，而普通 HTTPS 路径可达 861 MB/s）。
集群上残留的 81 GB 目录已删除，下次运行将从干净状态开始。完整性判据也已针对空目录、
仅有索引、分片不全、分片齐全以及残留 .incomplete 五种情况验证。
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The completeness check and the flock were not enough. The next run failed with OSError: [Errno 521] Unknown error 521: /data/home/sa-shared/models/Qwen3.8-Flash-Next-NVFP4/layer-00007-experts-0256-0383.safetensors raised from huggingface_hub reading a shard back, i.e. a stale-handle class error on NFSv4 while another cell was writing the same tree from a different node. The shared home was the wrong target. Every cell in the matrix runs on its own node and they were all writing one directory over NFS. /tmp on these compute nodes is /dev/md0 with about 17 TB free, so point MODEL_PATH there: each node fetches its own copy, roughly 2.5 minutes at the 861 MB/s these nodes sustain, and cross-node sharing disappears along with the lock semantics it depended on. The directory is now created with srun so it lands on the allocated node; the login node has an unrelated /tmp, and the bind mount source has to exist where the container starts. The completeness check stays: it still guards a partial tree within a node, and it is what turned this from a silent gsm8k 0.0000 into a visible failure. 完整性检查与 flock 仍不足够。下一次运行报错：huggingface_hub 在回读分片时抛出 OSError: [Errno 521]，即 NFSv4 上的失效句柄类错误——彼时另一个单元正从另一个节点 写入同一棵目录树。 共享家目录本就是错误的落点：矩阵中每个单元各自占用一个节点，却都在通过 NFS 写同一 个目录。这些计算节点的 /tmp 位于 /dev/md0，可用约 17 TB，因此将 MODEL_PATH 指向 该处：每个节点各自下载一份（按这些节点实测的 861 MB/s 约需 2.5 分钟），跨节点共享 及其依赖的锁语义一并消失。 目录改用 srun 创建，以确保建在已分配的计算节点上——登录节点的 /tmp 与之无关，而绑定 挂载的源必须存在于容器启动的那台机器上。完整性检查予以保留：它仍能防止单节点内的 残缺目录树，也正是它把一次静默的 gsm8k 0.0000 变成了可见的失败。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Qwen3.8-Flash-Next NVFP4 is now staged at /scratch/models on the compute nodes, verified on im-b200-c004: 126 GB, index present, 206/206 shards, no .incomplete leftovers. That removes the reason for everything this arm accumulated while the weights were absent. Deleted: the hf-download fallback and KEEP_HF_MODEL_ID in the launcher, the srun mkdir of a node-local target, and the bench script's completeness check, cross-cell flock and HF_HUB_DISABLE_XET. The launcher branch is now the ordinary staged-path lookup with a dsv4-style candidate search, and the bench script is byte-identical to the merged B300 sibling apart from the SKU name. Every one of those existed only to work around a missing checkpoint: the partial 81 GB tree that was served as a finished model and produced gsm8k 0.0000, and the NFSv4 stale-handle error from six nodes writing one shared directory. With the checkpoint staged, none of it applies, and B300 already passes with exactly this recipe. Qwen3.8-Flash-Next NVFP4 现已预置在计算节点的 /scratch/models 下，已在 im-b200-c004 上核实：126 GB、索引齐全、206/206 分片、无 .incomplete 残留。此前为 应对权重缺失而在本分支上累积的各项处理，至此均无必要。 已删除：launcher 中的 hf 下载兜底与 KEEP_HF_MODEL_ID、在节点本地创建目标目录的 srun mkdir，以及基准脚本中的完整性检查、跨单元 flock 与 HF_HUB_DISABLE_XET。 launcher 分支恢复为常规的预置路径查找（沿用 dsv4 式的候选目录搜索），基准脚本除 SKU 名称外与已合入的 B300 同类脚本逐字节一致。 这些处理本就只为绕开权重缺失：被当作完整模型加载、导致 gsm8k 0.0000 的 81 GB 残缺 目录树，以及六个节点同写一个共享目录引发的 NFSv4 失效句柄错误。权重预置后均不再适 用，而 B300 正是以这套配方通过的。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous run failed a single cell, concurrency 1, on the AIPerf coverage gate: TTFT 93.7% and inter-token latency 94.0% against a 95% requirement, with no latency signal in the final 180s of the 3600s window. The other four cells were cancelled by fail-fast rather than failing on their own, and the eval passed. Server startup, the staged checkpoint and the serve flags are all working, so this re-runs to see whether the quiet tail is deterministic. 上一次运行仅有并发 1 这一个单元失败，卡在 AIPerf 覆盖率门限：TTFT 93.7%、 token 间延迟 94.0%，均低于 95% 的要求，且在 3600 秒窗口的最后 180 秒内没有任何 延迟信号。其余四个单元是被 fail-fast 取消而非自身失败，评测单元通过。服务启动、 预置权重与服务参数均已正常，因此重跑一次以确认末段静默是否可复现。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7043888 to
41be4e3
Compare
1 participant
Summary / 摘要
Day-zero Qwen3.8-Flash-Next AgentX recipe on B200, served by SGLang with native NEXTN MTP.
qwen3.8nexthas no master-config entry yet — the lane was registered in #2742 and the golden-AL collector in #2748 — so this is the model's first benchmark recipe. SGLang is its plan-of-record engine perMODELS.md.B200 上首个 Qwen3.8-Flash-Next AgentX 配方，由 SGLang 提供服务，采用原生 NEXTN MTP。
qwen3.8next目前在主配置中尚无任何条目（#2742 注册模型通道，#2748 加入黄金 AL 采集脚本），本 PR 为该模型的首个基准配方。按MODELS.md，SGLang 是其 PoR 引擎。Config key / 配置项
qwen3.8next-fp4-b200-sglang-agentic-mtp—RadixArk/Qwen3.8-Flash-Next-NVFP4,lmsysorg/sglang:qwen38flashnext,cluster:b200-nscale, TP4/EP1, conc[1, 4, 8, 12, 16], no KV offloading.Recipe decisions / 配方要点
Checkpoint.
RadixArk/Qwen3.8-Flash-Next-NVFP4, 126 GiB,quantization_config.quant_method = modelopt— so--quantization modelopt_fp4is the right flag, the same one the Qwen3.5 NVFP4 B200 sibling uses. The checkpoint'signorelist keepsmtp.*unquantized, so NEXTN needs no external drafter.Image.
lmsysorg/sglang:qwen38flashnext, the model bring-up tag published 2026-08-26 (verified on Docker Hub).TP4/EP1. 126 GiB of weights is ~32 GiB per rank on B200, leaving room for the 256k-capped agentic traces. EP1 keeps the day-zero arm off the DeepEP/a2a path.
Serve flags are carried from
qwen3.5_fp4_b200_sglang_mtp.shunchanged:trtllm_mhaattention,flashinfer_trtllmMoE runner,fp8_e4m3KV cache, bf16 Mamba SSM, NEXTN with 3 steps / eagle-topk 1 / 4 draft tokens.No launcher change needed:
launch_b200-nscale-compat.shalready resolvesagentic/<prefix>_<precision>_b200_<framework>_mtp.sh.权重：
RadixArk/Qwen3.8-Flash-Next-NVFP4，126 GiB，quant_method = modelopt，故使用--quantization modelopt_fp4，与 Qwen3.5 NVFP4 B200 同类配方一致；其ignore列表保留mtp.*不量化，因此 NEXTN 无需外部草稿模型。镜像：
lmsysorg/sglang:qwen38flashnext（2026-08-26 发布的模型适配标签，已在 Docker Hub 核实）。TP4/EP1：每卡约 32 GiB 权重，为 256k 上限的智能体轨迹留出充足显存；EP1 让首发分支避开 DeepEP/a2a 路径。
服务参数沿用
qwen3.5_fp4_b200_sglang_mtp.sh：trtllm_mha注意力、flashinfer_trtllmMoE、fp8_e4m3KV 缓存、bf16 Mamba SSM、NEXTN 3 步 / eagle-topk 1 / 4 草稿 token。无需改动 launcher。
Acceptance length / 接受长度
SGLANG_SIMULATE_ACC_LEN=3.24. Three speculative tokens per verification step is the MTP=3 cell, measured thinking=off in speedbench-al run 33031708148 (MTP 1–6 → AL 1.87 / 2.61 / 3.24 / 3.74 / 4.18 / 4.40). This is an interim value: thethinking=oncollection is still running andgolden_al_distribution/qwen3.8next_mtp.yamlis not committed yet. Refresh this constant when that curve lands.SGLANG_SIMULATE_ACC_LEN=3.24。每次验证 3 个投机 token 即 MTP=3 单元，取自 speedbench-al 运行 33031708148 的 thinking=off 实测值。这是临时值：thinking=on采集仍在进行，golden_al_distribution/qwen3.8next_mtp.yaml尚未提交，待该曲线合入后需刷新此常量。Files / 改动文件
benchmarks/single_node/agentic/qwen3.8next_fp4_b200_sglang_mtp.sh(new)configs/nvidia-master.yaml— new entry after the Qwen3.5 NVFP4 B200 siblingperf-changelog.yaml— appended entryPart of a four-PR set, one per chip: B200 (this), B300, H200, MI355X.
本 PR 属于按芯片划分的四个 PR 之一：B200（本 PR）、B300、H200、MI355X。
🤖 Generated with Claude Code
Note
Low Risk
Adds benchmark config, a new shell recipe, and launcher model-path mapping only; no changes to core inference or auth paths.
Overview
Introduces the first agentic-coding benchmark lane for Qwen3.8-Flash-Next NVFP4 on B200 via SGLang with native NEXTN MTP, including a new single-node replay script, master config key
qwen3.8next-fp4-b200-sglang-agentic-mtp, and a perf-changelog entry.The recipe targets TP1/EP1 (cookbook single-node layout), concurrency 1–16, optional HiCache sizing when KV offload is enabled, GSM8K (
lm-eval) for eval-only runs, and throughput runs pinned to golden thinking_on acceptance length 2.32 at three speculative steps. Serve flags follow the Qwen3.8 cookbook: flashinfer linear-attention prefill/decode, bfloat16 Mamba SSM on Blackwell, checkpoint-derived NVFP4 (no explicit--quantization), and NEXTN 3 steps / 4 draft tokens.launch_b200-nscale-compat.shnow resolves theqwen3.8next+ fp4 prefix to staged weights under/scratch/modelsinstead of failing as unsupported.Reviewed by Cursor Bugbot for commit 41be4e3. Bugbot is set up for automated code reviews on this repo. Configure here.