Skip to content

Commit 92debd9

Browse files
seungrokjclaude
seungrokj
and
claude
authored
perf(amd): add DP-attention band to DSV4 ATOM MI355X AgentX MTP (#2778)
* perf(amd): add DP-attention band to DSV4 ATOM MI355X AgentX MTP Split the DeepSeek-V4-Pro FP4 ATOM AgentX MTP sweep into two attention bands per the ROCm/ATOM recipe: tensor-parallel attention for concurrency 1-16, and DP attention (--enable-dp-attention --enable-tbo plus the ATOM_DP_* routing knobs and AIPerf session-affinity vars) for concurrency 48-256. The mode is selected by the config's dp-attn, propagated as DP_ATTENTION, so the recipe drops its DP_ATTENTION=false guard. Bump the ATOM image to nightly_202608280858. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(changelog): record DSV4 ATOM MI355X AgentX DP-attention band Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(agentx): scope DSV4 MI355X ATOM sweep to dp-attn c256 for e2e Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(agentx): raise AIPerf timeouts + realtime metrics for DSV4 ATOM MI355X MTP Bump dataset/profile-configure timeouts to 1800s and cancel-drain to its 300s ceiling for long agentic cold-cache runs, and enable realtime metrics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(agentx): set ATOM keep-alive to 900s for DSV4 MI355X MTP warmup uvicorn's 5s idle keep-alive closes AIPerf's pooled sockets during long-backlog warmup; the reused write hits ECONNRESET (104) and one such reset on a root AgentX warmup request aborts the entire run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(agentx): restore full DSV4 MI355X ATOM sweep search space Re-enable the TP c1-16 and DP-attention c48-256 bands after the keep-alive fix, reverting the temporary dp-attn c256-only e2e scoping. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 99fca10 commit 92debd9

3 files changed

Lines changed: 55 additions & 15 deletions

File tree

‎benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.sh‎

Lines changed: 40 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,10 @@ if [[ -n "${SLURM_JOB_ID:-}" ]]; then
1414
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
1515
fi
1616

17-
if [ "$TP" -ne 8 ] || [ "$EP_SIZE" -ne 1 ] || [ "$DP_ATTENTION" != "false" ]; then
18-
echo "This recipe requires TP=8, EP_SIZE=1, and DP_ATTENTION=false" >&2
19-
exit 1
20-
fi
2117
require_agentic_kv_offload_none
2218

19+
echo "Attention mode: $([ "$DP_ATTENTION" = "true" ] && echo dp || echo tp) (DP_ATTENTION=$DP_ATTENTION, CONC=$CONC)"
20+
2321
if [[ -n "${ROCR_VISIBLE_DEVICES:-}" ]]; then
2422
export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"
2523
fi
@@ -48,14 +46,37 @@ export ATOM_DEBUG_PREFIX_HITS=1
4846
export ATOM_PROFILER_MORE=0
4947
export ATOM_PROFILER_TIMEOUT=1200
5048

51-
# AgentX/AIPerf network, failure, warmup, and trace-gap settings from the
52-
# validated one-hour baseline.
49+
# DP-attention runs layer ATOM's DPA routing and two-batch-overlap knobs on top of
50+
# the TP settings above (recipe section "Server - DP attention"); exported only for
51+
# the DP band. ATOM_DP_SESSION_AFFINITY is not optional: without it a session's
52+
# turns scatter across DP ranks, the prefix KV written by one turn is unreachable
53+
# by the next, and the multi-turn agentic workload collapses to cold prefill.
54+
# GPU_MAX_HW_QUEUES and ATOM_NUMA_BIND are prerequisites of --enable-tbo.
55+
DP_ATTN_ARGS=()
56+
if [ "$DP_ATTENTION" = "true" ]; then
57+
export GPU_MAX_HW_QUEUES=5
58+
export ATOM_NUMA_BIND=1
59+
export ATOM_DP_SESSION_AFFINITY=1
60+
export ATOM_DP_LB_REQ_EQUIV=512
61+
export ATOM_ENABLE_PREFILL_DELAYER=1
62+
export ATOM_PREFILL_DECODE_INTERVAL=10
63+
# Client-side counterpart to session affinity: make AIPerf emit a stable
64+
# session id (x-dynamo-session-id, falling back to the always-sent
65+
# x-correlation-id) so the DPA router pins each conversation to one rank.
66+
export AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true
67+
export AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=true
68+
DP_ATTN_ARGS=(--enable-dp-attention --enable-tbo)
69+
fi
70+
71+
# Raise the AIPerf HTTP TCP user timeout to 900000 ms (15 min), well above the
72+
# aiperf default of 30000 ms (30 s), so long-stalling AgentX request
73+
# connections are not torn down as dead during extended server-side pauses.
5374
export AIPERF_HTTP_TCP_USER_TIMEOUT=900000
54-
export AIPERF_FAILED_REQUEST_THRESHOLD=0.10
55-
export AIPERF_LIVE_FAILED_REQUEST_THRESHOLD=0.10
56-
export AIPERF_TRACE_IDLE_GAP_CAP_SECONDS=300
57-
export AIPERF_WARMUP_REQUESTS_PER_LANE=10
58-
export AIPERF_BENCHMARK_GRACE_PERIOD=30
75+
export AIPERF_TIMING_CANCEL_DRAIN_TIMEOUT=300
76+
export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES=0
77+
export AIPERF_DATASET_CONFIGURATION_TIMEOUT=1800
78+
export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800
79+
export AIPERF_UI_REALTIME_METRICS_ENABLED=true
5980

6081
# Require ATOM Prometheus metrics in every official result.
6182
export AIPERF_SERVER_METRICS_URLS="http://localhost:${PORT}/metrics"
@@ -83,8 +104,7 @@ trap 'exit 143' TERM
83104
MAX_NUM_SEQS=$((2 * CONC))
84105

85106
# golden_al_distribution/dsv4_mtp.yaml: thinking_on, 3 draft tokens -> AL 2.49
86-
# --spec-decode-acceptance-length 2.49.
87-
# https://github.com/ROCm/ATOM/pull/1948
107+
# https://github.com/SemiAnalysisAI/InferenceX/blob/main/golden_al_distribution/dsv4_mtp.yaml
88108
NUM_SPEC_TOKENS=3
89109
SPEC_DECODE_AL=2.49
90110
SPEC_ARGS=(
@@ -102,6 +122,12 @@ ATOM_CMD=(
102122
--served-model-name "$MODEL"
103123
--host 0.0.0.0
104124
--server-port "$PORT"
125+
# uvicorn defaults to a 5s idle keep-alive; AIPerf pools sockets for far
126+
# longer (aiohttp ~15s) and warmup inter-turn gaps under backlog exceed 5s,
127+
# so the server closes an idle pooled socket and the reused write hits
128+
# 'Connection reset by peer' (errno 104). One such reset on a root AgentX
129+
# warmup request aborts the whole run. Outlast the client idle window.
130+
--timeout-keep-alive 900
105131
--tensor-parallel-size "$TP"
106132
--kv-cache-dtype fp8
107133
--index-cache-dtype fp4
@@ -113,6 +139,7 @@ ATOM_CMD=(
113139
--level 3
114140
--cudagraph-mode FULL
115141
"${SPEC_ARGS[@]}"
142+
"${DP_ATTN_ARGS[@]}"
116143
--max-num-seqs "$MAX_NUM_SEQS"
117144
)
118145
write_command "$RESULT_DIR/server_command.txt" "${ATOM_CMD[@]}"

‎configs/amd-master.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1336,7 +1336,7 @@ dsv4-fp4-mi355x-vllm-agentic-mtp:
13361336
# uses the thinking_on golden AL 2.49 for three draft tokens; eval uses real
13371337
# MTP acceptance. max-num-seqs is set to 2x concurrency by the recipe.
13381338
dsv4-fp4-mi355x-atom-agentic-mtp:
1339-
image: rocm/atom-dev:nightly_202608201032
1339+
image: rocm/atom-dev:nightly_202608280858
13401340
model: deepseek-ai/DeepSeek-V4-Pro
13411341
model-prefix: dsv4
13421342
runner: cluster:mi355x-amds
@@ -1346,7 +1346,8 @@ dsv4-fp4-mi355x-atom-agentic-mtp:
13461346
scenarios:
13471347
agentic-coding:
13481348
- search-space:
1349-
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 48] }
1349+
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16] }
1350+
- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [48, 64, 96, 128, 256] }
13501351

13511352
dsr1-fp4-mi355x-sglang-disagg-mtp:
13521353
image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260519

‎perf-changelog.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6577,6 +6577,18 @@
65776577
- "Runner: launch_gb300-nv.sh bumped from NVIDIA/srt-slurm@v1.0.29 to v1.0.72 for the dynamo-trt+qwen3.5+fp4 path."
65786578
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2730
65796579

6580+
- config-keys:
6581+
- dsv4-fp4-mi355x-atom-agentic-mtp
6582+
scenario-type:
6583+
- agentic-coding
6584+
description:
6585+
- "Split the DeepSeek-V4-Pro FP4 ATOM AgentX MTP sweep into two attention bands per the ROCm/ATOM DeepSeek-V4-Agentic-InferenceX.md recipe: tensor-parallel attention at concurrency 1, 2, 4, 8, and 16, and DP attention at concurrency 48, 64, 96, 128, and 256 (previously a single tensor-parallel band at concurrency 1, 2, 4, 8, 16, 32, and 48)."
6586+
- "The DP-attention band adds --enable-dp-attention and --enable-tbo plus ATOM's DPA routing knobs (ATOM_DP_SESSION_AFFINITY, ATOM_DP_LB_REQ_EQUIV, ATOM_ENABLE_PREFILL_DELAYER, ATOM_PREFILL_DECODE_INTERVAL), the two-batch-overlap prerequisites GPU_MAX_HW_QUEUES and ATOM_NUMA_BIND, and the AIPerf session-affinity vars that keep each agentic conversation pinned to one DP rank; the tensor-parallel band is otherwise unchanged."
6587+
- "Session affinity is mandatory for the DP band: without it a session's turns scatter across DP ranks, the prefix KV written by one turn is unreachable by the next, and the multi-turn agentic workload collapses to cold prefill."
6588+
- "The attention band is selected by the config's dp-attn, propagated to the recipe as DP_ATTENTION, so the script's former DP_ATTENTION=false hard-fail guard is removed."
6589+
- "Bump the ATOM image from rocm/atom-dev:nightly_202608201032 to nightly_202608280858; FP8 KV/index caches, prefix caching, 32K state checkpoints, 16K batching/prefill chunks, FULL cudagraph mode, three-token MTP with golden AL 2.49, and max-num-seqs at twice concurrency are unchanged."
6590+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2778
6591+
65806592
- config-keys:
65816593
- dsv4-fp4-b200-sglang-agentic-hicache-mtp
65826594
scenario-type:

0 commit comments

Comments
 (0)