[WIP] feat: add agentx benchmark type for agentic-coding replay - #342
Open
shljessie wants to merge 1 commit into
Open
[WIP] feat: add agentx benchmark type for agentic-coding replay#342shljessie wants to merge 1 commit into
shljessie wants to merge 1 commit into
Conversation
The InferenceX AgentX harness has been reachable only through benchmark.type: custom, which meant every recipe inlined the harness's environment contract by hand. The seven variables agentic_srt.sh requires (MODEL, MODEL_PREFIX, FRAMEWORK, PRECISION, CONC, RESULT_FILENAME, DURATION) were therefore either restated per recipe or exported by whichever launcher submitted the job, so a recipe could be complete on its own terms and still abort on check_env_vars after the workers had loaded. Register an agentx runner that derives that contract from the recipe and reports missing inputs before submission. MODEL, FRAMEWORK, PRECISION, PORT, IS_MULTINODE, RESULT_DIR and the backend's required server metric prefix come from the config; model_prefix, duration_seconds and result_filename are new benchmark fields; benchmark.env is applied last so recipes keep control of the AIPERF_* knobs and the optional harness metadata. The harness itself stays in InferenceX, mounted at /infmax-workspace exactly as lm-eval already expects, so this does not vendor the client stack. Two consequences of that mount are handled here: - AIPERF_SERVER_METRICS_URLS and the SRT_* worker endpoints were gated on type == "custom". agentx polls those endpoints to decide when the deployment has drained between concurrencies, so it needs the same logical-worker view. - The dry-run warning for a missing INFMAX_WORKSPACE was found by scanning benchmark.command for the path. agentx and lm-eval build that path in Python, so both went unwarned; match on benchmark type as well. bench.sh fails with an actionable message naming the clone (submodules included, since the harness installs from utils/aiperf) rather than leaving bash to report a missing file. Signed-off-by: Seonghee Lee <seongheel@nvidia.com>
shljessie
requested review from
alec-flowers,
csahithi,
ishandhanani and
nlevin-ui
as code owners
August 25, 2026 22:27
shljessie
changed the title
Contributor
|
LGTM. cc @ishandhanani |
Collaborator
Author
|
Still work in progress. Will ping again once ready! Want to do more thorough checking to make sure we don't have any issues running it. |
This was referenced Aug 26, 2026
nv-yna
added a commit
that referenced
this pull request
Aug 27, 2026
…353) * feat(benchmark): add agentperf benchmark type for trajectory replay Adds an 'agentperf' benchmark runner that drives the agentperf-client trajectory-replay load generator against an srt-slurm-launched server. This is a different client from the InferenceX AgentX harness (#342): agentperf-client is a standalone uv-managed project with a Rust streaming core, invoked as agentperf/run.py, producing per-phase trajectory outputs plus a per-request requests.jsonl. The bench script is a faithful translation of the standalone disagg-harness invocation it was ported from, so results stay comparable: - isolated job-scoped runtime under /tmp (HOME/CARGO_HOME/RUSTUP_HOME/ UV_PROJECT_ENVIRONMENT), read-only-mount friendly - pinned toolchain preflight (rust via rustup archive, uv via pip), uv sync --frozen + maturin rustcore build, tokenizer cache warm - trajectory/user-assignments datasets staged from shared storage to node-local /tmp before measurement, with config paths rewritten - self-validating READY marker (arch + client commit) and an atomic build lock for relocated runtimes The workload definition lives in the client's own config YAML (benchmark.agentperf_config); srtctl injects endpoint, model and concurrency via CLI overrides. rollup.py normalizes per-phase stats into benchmark-rollup.json. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> * feat(benchmark): add agentperf harness-run porter Adds scripts/agentperf/port_harness_run.py: a host-side tool that ports a standalone disagg-harness AgentPerf run directory to an srt-slurm recipe + client workload YAML, with a provenance report (every generated field -> source artifact) and a TODO list for the few things it cannot infer (dataset originals, gpu_type, cross-user path reachability). Encodes the translations learned porting the DSV4 c1010 baseline: DYN_KV_BLOCK_SIZE -> DYN_TRTLLM_KV_BLOCK_SIZE, DYN_UCX_TLS -> UCX_TLS, /lustre -> /scratch rewrites, harness-managed env drops, frontend flag reconstruction (without the worker-only --dyn-*-parser flags), taskset -> numa_cpu_bind detection, settle-time recovery from output stems. Verified against the real c1010 run dir: generated recipe matches the hand-validated parity recipe on 176/184 fields (all remaining diffs benign and TODO-flagged) and the workload YAML on 19/19 fields. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> --------- Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AgentX recipes currently require an InferenceX clone, because the replay client (agentic_srt.sh, benchmark_lib.sh, and the utils/aiperf submodule) lives there and is not vendored in srt-slurm. A native agentx benchmark type is proposed upstream in NVIDIA/srt-slurm#342; once it lands we can validate these recipes in CI and update them to type: agentx.
Example
Test plan