Adding nemo template by xing-anyscale · Pull Request #1086 · anyscale/templates

xing-anyscale · 2026-09-11T20:52:40Z

I have run the template in workspace to make sure it's runnable.

kouroshHakha · 2026-09-11T21:46:46Z

+        mamba_ssm_cache_dtype="float32",
+        gpu_memory_utilization=0.9,
+        trust_remote_code=True,
+        enable_chunked_prefill=True,


this is the default if I am not mistaken?

Nice catch — confirmed, dropping enable_chunked_prefill=True as SchedulerConfig.enable_chunked_prefill defaults to True in vLLM 0.25.1 (V1 enables chunked
prefill by default), so the line is a no-op.

Keeping the other three — none of them are defaults on 0.25.1:

gpu_memory_utilization=0.9 — default is 0.92, and 0.9 is what NVIDIA's card specifies

mamba_ssm_cache_dtype="float32" — default is "auto"

trust_remote_code=True — default is False, and required for the custom nemotron_h
architecture

kouroshHakha · 2026-09-11T21:46:59Z

+        # async_scheduling=True,
+        # max_cudagraph_capture_size=128,
+        reasoning_parser="nemotron_v3",
+        # enable_auto_tool_choice=True,   # uncomment for agents/tool use


should we make these the default.

Good call. will do.

kouroshHakha · 2026-09-11T21:47:33Z

+        # async_scheduling=True,
+        # max_cudagraph_capture_size=128,


async scheduling should be the default on the latest ray serve llm image. These seem to be referencing an old vllm version, why?

Agreed on both. let me update it.

kouroshHakha · 2026-09-11T21:49:09Z

+        )
+    ),
+    # runtime_env=dict(env_vars={"HF_TOKEN": os.environ.get("HF_TOKEN")}),
+    engine_kwargs=dict(


did you cross check this with the vllm recipes's page?

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8
--trust-remote-code
--kv-cache-dtype fp8
--tensor-parallel-size 8
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
--reasoning-parser nemotron_v3
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16?hardware=h100&features=tool_calling,reasoning,spec_decoding&variant=fp8

there is spec decoding. It's tp8 and no ep here.

Good call — let me follow the recipe and update the config: TP=8, drop expert parallelism, addspeculative_config={"method": "mtp", "num_speculative_tokens": 3}. Taking all three together since that's the combination recipes validated.

kouroshHakha · 2026-09-11T21:51:18Z

+!pip install "ray[serve,llm]==2.57.0"
+!pip install "vllm==0.25.1"


why are we on an old version here?

i discussed this with Kunling as well. Since other templates in this deployment-serve-llm folder are all based on ray=2..57, we wanna keep the same version for this one. lmk if it makes sense. @kunling-anyscale

Aydin-ab

I'd like to make that troubleshooting more concise / nemotron-specific to make it more readable

Otherwise LGTM thank you for adding this 🙏

I believe ray docs should sync automatically but cc @elliot-barn

Aydin-ab · 2026-09-14T19:31:37Z

I can do the bump to 2.58 once this merges

8a8fd8b tightened the Troubleshooting section in README.md directly, which desynced it from README.ipynb and failed the generate-readme hook. The notebook is the generated source, so port the same text there; README.md is unchanged and now matches nbconvert output. Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…#1087) #1086 added the Nemotron-3-Super-120B template but didn't list it here, so the index stops at gpt-oss. Adding it surfaced a second problem: all seven existing tutorial links point at `serve/tutorials/deployment-serve-llm/content/<name>/README.html`, which redirects to the Serve index. Ray publishes these under `_collections/` — per `doc/source/serve/examples.yml`, the link is `../_collections/serve/tutorials/deployment-serve-llm/<name>/README`. Repointed all eight. ## Testing `curl -sL` on each of the seven repointed links returns HTTP 200 at the requested path, against a soft 404 before. The Nemotron link 404s until Ray picks the template up in its next docs sync — the same lag every new tutorial has. Signed-off-by: svc-template-updater <282975472+svc-template-updater@users.noreply.github.com> Co-authored-by: svc-template-updater <282975472+svc-template-updater@users.noreply.github.com>

xing-anyscale force-pushed the template/nemotron-example branch from 754302c to 20d4629 Compare September 11, 2026 21:31

Adding nemo template

926e725

xing-anyscale force-pushed the template/nemotron-example branch from 20d4629 to 926e725 Compare September 11, 2026 21:35

xing-anyscale requested review from Aydin-ab and kunling-anyscale September 11, 2026 21:39

xing-anyscale marked this pull request as ready for review September 11, 2026 21:39

xing-anyscale requested a review from kouroshHakha September 11, 2026 21:39

kouroshHakha reviewed Sep 11, 2026

View reviewed changes

kunling-anyscale reviewed Sep 11, 2026

View reviewed changes

Comment thread templates/deployment-serve-llm/nemotron-3-super-120b/workspace_prod.yaml Outdated

Remove workspace config yaml

f7c2fa0

xing-anyscale force-pushed the template/nemotron-example branch from 609f4c8 to 08eb20e Compare September 12, 2026 00:23

Update per code reviews

8ca95f0

xing-anyscale force-pushed the template/nemotron-example branch from 08eb20e to 8ca95f0 Compare September 12, 2026 00:33

xing-anyscale requested review from kouroshHakha and kunling-anyscale September 12, 2026 00:34

kouroshHakha approved these changes Sep 12, 2026

View reviewed changes

kunling-anyscale approved these changes Sep 12, 2026

View reviewed changes

Aydin-ab approved these changes Sep 14, 2026

View reviewed changes

Comment thread templates/deployment-serve-llm/nemotron-3-super-120b/README.md Outdated

Aydin-ab and others added 2 commits September 14, 2026 12:34

Update templates/deployment-serve-llm/nemotron-3-super-120b/README.md

8a8fd8b

Aydin-ab merged commit b19087d into main Sep 14, 2026
4 checks passed

Aydin-ab deleted the template/nemotron-example branch September 14, 2026 20:00

svc-template-updater mentioned this pull request Sep 14, 2026

docs(deployment-serve-llm): list nemotron, and fix the tutorial links #1087

Merged

		!pip install "ray[serve,llm]==2.57.0"
		!pip install "vllm==0.25.1"

Conversation

xing-anyscale commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xing-anyscale Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Aydin-ab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Aydin-ab commented Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xing-anyscale commented Sep 11, 2026 •

edited

Loading

xing-anyscale Sep 11, 2026 •

edited

Loading