Reward Profiling Infra Update

benchmarks/nemotron_3.5_super/build_eval_container.sh

@@ -10,6 +10,10 @@ INPUT_CONTAINER=$INPUT_CONTAINER
 OUTPUT_CONTAINER=$OUTPUT_CONTAINER
 MOUNTS=$MOUNTS
 GYM_CONFIG=$GYM_CONFIG
+# Benchmark data preparation only applies to configs that declare a `benchmark` dataset.
+# Set to 1 for train-split configs (e.g. reward-profiling sweeps), where `gym eval prepare`
+# would abort with "No benchmark config found".
+SKIP_PREPARE=${SKIP_PREPARE:-0}
 VLLM_ROUTER_WHEEL=$VLLM_ROUTER_WHEEL
 NEMO_GYM_GIT_URL=${NEMO_GYM_GIT_URL:-https://github.com/NVIDIA-NeMo/Gym}
 NEMO_GYM_GIT_REF=${NEMO_GYM_GIT_REF:-main}
@@ -87,7 +91,11 @@ fi
 # END Benchmark specific preparation
 ########################################
 
-gym eval prepare +num_prepare_benchmark_processes=4 --config $GYM_CONFIG
+if [[ "$SKIP_PREPARE" == "0" ]]; then
+    gym eval prepare +num_prepare_benchmark_processes=4 --config $GYM_CONFIG
+else
+    echo ">>> SKIP_PREPARE=1: skipping benchmark data preparation."
+fi
 
 gym env start \
     --config $GYM_CONFIG \

benchmarks/nemotron_3.5_super/reward_profiling/CONTRIBUTING.md

@@ -0,0 +1,201 @@
+# How to Contribute an environment / dataset.
+
+## Index
+0. [Files you touch](#files-you-touch) / [What your data rows need](#what-your-data-rows-need)
+1. [Creating a container.](#01---creating-a-container)
+2. [Creating a single manifest file.](#02---creating-a-single-manifest-file)
+    - a. [Non-judge environments.](#02a---non-judge-environments)
+    - b. [Judge environments.](#02b---judge-environments)
+    - c. [Sandbox environments.](#02c---sandbox-environments)
+3. [Running a single manifest job.](#03---running-a-single-manifest-job)
+    - a. [Non-judge environments.](#03a---non-judge-environments)
+    - b. [Judge environments.](#03b---judge-environments)
+    - c. [Sandbox environments.](#03c---sandbox-environments)
+4. [Adding a manifest entry into manifests/nemotron_3_5_super.yaml](#04---adding-a-manifest-entry-into-manifestsnemotron_3_5_superyaml)
+
+Contribute your environment as a one-entry manifest first, prove it runs, then add it to the shared
+manifest. A one-entry manifest fails in seconds; the same mistake in the shared one costs a full
+spin-up on every shard.
+
+See [README.md](./README.md) for the pipeline itself. `R=benchmarks/nemotron_3.5_super/reward_profiling`
+throughout, and all paths below are relative to the repo root.
+
+### Files you touch
+
+| file | when | what |
+|---|---|---|
+| your data `.jsonl` | always | one row per task, on `/lustre`. Not committed |
+| `resources_servers/<env>/` | only if your env is not already in Gym | verifier + config. See the [environment guide](https://docs.nvidia.com/nemo/gym/latest/contribute/environments) |
+| `$R/manifests/<yours>.yaml` | always | your one-entry manifest. Committed |
+| `$R/manifests/nemotron_3_5_super.yaml` | step 04 | add your entry to the shared sweep |
+| `$R/configs/container_config.yaml` | only if you pulled in a new server | regenerate, do not hand-edit |
+
+Nothing else. In particular you do not edit anything under `nemo_gym/`, and you do not edit
+upstream configs under `resources_servers/*/configs/` to bind a judge — that goes in your manifest
+([02b](#02b---judge-environments)).
+
+### What your data rows need
+
+```json
+{ "agent_ref": { "name": "my_simple_agent" },
+  "responses_create_params": { "input": [ ... ] } }
+```
+
+`agent_ref.name` is what rollout collection dispatches on — it is why 36 environments can share one
+deployment. It must match the `agent` your manifest entry declares, and that agent must be declared
+by one of the entry's `configs`. `validate` checks all three agree.
+
+Add `verifier_metadata` only if your verifier reads it — the reference answer, expected tool call,
+or test cases it scores against. Most entries in the shared manifest do not carry one, because
+their verifiers score the response intrinsically. Nothing in the sweep looks at it.
+
+## 01 - Creating a container.
+
+Only needed if your environment's servers are not already baked into the reward profiling
+container. A server with no baked venv installs at runtime and hangs the run behind connection
+retries rather than failing, so check first:
+
+```bash
+PYTHONPATH=$R python -m infra container-config $R/manifests/<yours>.yaml --output /tmp/mine.yaml
+python -c "
+import yaml
+have = set(yaml.safe_load(open('$R/configs/container_config.yaml'))['config_paths'])
+mine = set(yaml.safe_load(open('/tmp/mine.yaml'))['config_paths'])
+print('\n'.join(sorted(mine - have)) or 'nothing new; reuse the existing container')
+"
+```
+
+A plain `diff` is not the check — the committed config covers 36 entries and yours covers one, so
+they always differ. What matters is whether your `config_paths` are a subset of it. Anything
+printed is a server the container does not have: regenerate `configs/container_config.yaml` over
+all manifests and rebuild, per [README.md § 01](./README.md#01---optional-create-a-container).
+
+## 02 - Creating a single manifest file.
+
+Copy the closest `manifests/example_*.yaml` and change `entries`. Every entry needs `label`, `agent`,
+`configs` and `data`. `owner` is optional in the schema but expected for new entries — it is you, because the `/lustre` path stops
+identifying anyone once the data is copied or a blend is re-cut.
+
+```bash
+PYTHONPATH=$R python -m infra validate $R/manifests/<yours>.yaml
+```
+
+This checks the configs exist, the agent is declared by one of them, the data parses, and each
+row's `agent_ref` matches. That last check is what stops a mispaired dataset and config from
+silently scoring rollouts with the wrong verifier.
+
+### 02a - Non-judge environments.
+
+Nothing beyond the entry itself. See [`manifests/example_basic.yaml`](./manifests/example_basic.yaml).
+
+### 02b - Judge environments.
+
+Shipped configs name a judge but never an endpoint, so bind one under `gym_env_start` in your
+manifest. Skip this and the composed config fails to resolve — the run dies at parse, before a
+single rollout.
+
+Two keys, both in [`manifests/example_judge.yaml`](./manifests/example_judge.yaml): the model server
+itself, and the binding that tells your resources server to use it.
+
+Set `max_concurrent_requests` if the judge is an external endpoint, as the ones in the shared
+manifest are. The transport retries 429s with a flat sleep and extends its retry budget on rate
+limits, so a saturated gateway degrades into a retry storm rather than backpressure. A judge served
+locally has no such limit and does not need it.
+
+### 02c - Sandbox environments.
+
+Two extra things, in [`manifests/example_sandbox_judge.yaml`](./manifests/example_sandbox_judge.yaml):
+
+- Register every `verifier_type` your rows use under `ns_tools.verifiers`. Rows carry it per-row,
+  and an unregistered name is a hard `ValueError` that reaches the driver as a bare 500 — the lane
+  produces zero rollouts rather than bad ones.
+- If the verifier is judge-backed, enable the judge explicitly. `math_with_judge` verifies
+  symbolically via math-verify by default and only consults the judge as a fallback.
+
+## 03 - Running a single manifest job.
+
+```bash
+MANIFEST=$R/manifests/<yours>.yaml OUT_DIR=$R/outputs/sweeps/<name> LIMIT_PER_ENTRY=8 \
+  bash $R/scripts/01_materialize.sh
+
+MODEL=<ckpt> SWEEP_DIR=$R/outputs/sweeps/<name>/<nickname> bash $R/scripts/03_run_single.sh
+```
+
+`LIMIT_PER_ENTRY=8` takes only the first 8 rows of your dataset instead of all of them, so at
+`num_repeats: 8` that is 64 rollouts — enough to exercise every code path in minutes. Drop it for
+a real run.
+
+If a policy endpoint already exists, skip the GPUs entirely — this is the fastest loop for checking
+a new entry, since there is no allocation to wait for:
+
+```bash
+SWEEP_DIR=<sweep> POLICY_BASE_URL=http://host:8000/v1 POLICY_MODEL_NAME=<model> \
+  POLICY_API_KEY=<key> bash $R/scripts/03_run_endpoint.sh
+```
+
+The judges are already a hosted endpoint either way, so only the policy costs GPUs. A sandbox entry
+still needs `NEMO_SKILLS_SANDBOX_HOST/PORT` pointed somewhere reachable, since this path starts no
+sandbox of its own. Every other variable is in [README § Common knobs](./README.md#common-knobs).
+
+If you want a *committed* subset — your dataset has 200k rows and only 10k belong in the sweep —
+that is `limit` on your entry, or `materialize.limit_per_entry` for a manifest-wide default. Set
+`materialize.sample: random` with a `seed` if the first N rows would be a biased sample.
+
+Not `gym eval run --limit`, which looks equivalent: it takes the first N rows of the whole input,
+so against the full sweep in step 04 it never reaches your entry ([why](./README.md#gotchas)).
+
+Then profile and read the result:
+
+```bash
+SWEEP_DIR=$R/outputs/sweeps/<name>/<nickname> bash $R/scripts/05_profile.sh
+cat $R/outputs/sweeps/<name>/<nickname>/by_label/<label>/profile.txt
+```
+
+Check `by_label/split_report.json` for `labels_without_rollouts`. Your label at zero means the lane
+failed rather than scored badly — read `outputs/.../slurm-logs/` for the 500.
+
+Reward is not the only signal worth a look. A `std/reward` of 0 across every task usually means the
+verifier is returning a constant, not that the model is perfectly consistent.
+
+### 03a - Non-judge environments.
+
+Nothing extra.
+
+### 03b - Judge environments.
+
+Export the keys `env.yaml` interpolates, e.g. `NVI_KEY_EVALUATOR`. A bare `VAR=value` in `.bashrc`
+is a shell variable, not an environment one, so sbatch never sees it — and `export` in `.bashrc`
+only reaches a login shell. `03_run_single.sh` and `05_profile.sh` both check this up front and name what
+is missing.
+
+### 03c - Sandbox environments.
+
+The shared manifest's `srun.sandbox_container` already names one; a standalone manifest needs its
+own, or `SANDBOX_CONTAINER=<nemo-skills sqsh>`. Without it `ns_tools` falls back to `127.0.0.1:6000`,
+where nothing listens, and every rollout fails with a bare 500. `03_run_single.sh` starts one and waits
+for `/health` before collecting, so watch that gate rather than the rollout count.
+
+Sandbox capacity is `sandbox_nodes x sandbox_workers` concurrent executions. The image forces one
+uWSGI process per worker, so the worker count *is* the concurrency — a session pins to a worker by
+`X-Session-ID` consistent hashing and holds it. `sandbox_nodes` defaults to every node in the job;
+the sandboxes ride along on the vLLM nodes and are fronted by an nginx balancer hashing the same
+header, so they cost no extra nodes. If your environment executes code, size this rather than
+assuming GPUs are the limit. Only one arm64 build exists:
+`/lustre/fsw/portfolios/llmservice/users/igitman/images/nemo-skills-sandbox-0.7.1-arm64.sqsh`
+
+## 04 - Adding a manifest entry into manifests/nemotron_3_5_super.yaml
+
+Once your entry runs clean alone, move it into
+[`manifests/nemotron_3_5_super.yaml`](./manifests/nemotron_3_5_super.yaml):
+
+- **Append to `entries`, never insert.** Order assigns task indices, so appending keeps every
+  existing `--resume` key valid; inserting in the middle renumbers every entry after yours and
+  invalidates work already collected.
+- **Merge your `gym_env_start` keys into the shared block**, reusing the judge anchors already
+  defined there rather than declaring a second endpoint for the same model.
+- **Regenerate the container config** if you added a server, and rebuild.
+- **Re-validate, then re-run with `LIMIT_PER_ENTRY=2` over the whole manifest** (2 rows per entry, so 576 rollouts across all 36). Your entry can
+  resolve alone and still collide once composed with 35 others — a port, a judge binding, or a
+  server that was never baked into the container.
+
+Open the PR against this branch with the `LIMIT_PER_ENTRY` profile output for your label.