Add golden patch adaptations.

README.md

@@ -358,6 +358,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
 | Swe Agents                                    | coding                | SWE-bench driven by the opencode agent framework for RL training.                                                                                                                                                                                                                                                                                                                                                               | Train software engineering capabilities on SWE tasks using opencode rollouts.                                                         | ✓     | -          | Apache 2.0                                                | <a href='responses_api_agents/swe_agents/configs/swebench_opencode_training.yaml'>swebench_opencode_training.yaml</a>                                                                                                               | -                                                                                                                                                              |
 | Swe Agents                                    | coding                | SWE-bench driven by the opencode agent framework.                                                                                                                                                                                                                                                                                                                                                                               | Eval software engineering capabilities on SWE-bench using opencode.                                                                   | ✓     | ✓          | Apache 2.0                                                | <a href='responses_api_agents/swe_agents/configs/swebench_opencode.yaml'>swebench_opencode.yaml</a>                                                                                                                                 | -                                                                                                                                                              |
 | Swe Pivot                                     | agent                 | SWE pivot verifier for PivotRL on coding agent trajectories                                                                                                                                                                                                                                                                                                                                                                     | Improve coding agent fix-design decisions                                                                                             | ✓     | ✓          | Apache 2.0                                                | <a href='resources_servers/swe_pivot/configs/swe_pivot.yaml'>swe_pivot.yaml</a>                                                                                                                                                     | -                                                                                                                                                              |
+| Swebench Pro                                  | coding                | -                                                                                                                                                                                                                                                                                                                                                                                                                               | -                                                                                                                                     | -     | -          | -                                                         | <a href='resources_servers/swebench_pro/configs/swebench_pro.yaml'>swebench_pro.yaml</a>                                                                                                                                            | -                                                                                                                                                              |
 | Swerl Gen                                     | coding                | Running sandboxed evaluation for SWE-style tasks (either patch generation or reproduction test generation)                                                                                                                                                                                                                                                                                                                      | Improve SWE capabilities useful for benchmarks like SWE-bench                                                                         | ✓     | ✓          | Apache 2.0                                                | <a href='resources_servers/swerl_gen/configs/swerl_gen.yaml'>swerl_gen.yaml</a>                                                                                                                                                     | -                                                                                                                                                              |
 | Swerl Llm Judge                               | coding                | SWE-style multiple-choice LLM-judge tasks scored via <solution>...</solution> choice.                                                                                                                                                                                                                                                                                                                                           | Improve SWE capabilities useful for benchmarks like SWE-bench                                                                         | ✓     | ✓          | MIT                                                       | <a href='resources_servers/swerl_llm_judge/configs/swerl_llm_judge.yaml'>swerl_llm_judge.yaml</a>                                                                                                                                   | -                                                                                                                                                              |
 | Tales                                         | games                 | TALES text-adventure games (textworld, textworld_express, alfworld, scienceworld, jericho) in Gymnasium API style.                                                                                                                                                                                                                                                                                                              | Multi-turn reasoning and progress through open-ended, situated text environments.                                                     | -     | -          | -                                                         | <a href='resources_servers/tales/configs/tales.yaml'>tales.yaml</a>                                                                                                                                                                 | -                                                                                                                                                              |

benchmarks/nemotron_3.5_super/policy_model_override.yaml

@@ -2,6 +2,8 @@ policy_model:
   responses_api_models:
     vllm_model:
       sampling_overrides:
+        temperature: 1.0
+        top_p: 0.95
         max_output_tokens: null
         max_tokens: null
         max_completion_tokens: null

benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh

@@ -189,7 +189,7 @@ set -euo pipefail
 nodes=(\$(scontrol show hostnames "\$SLURM_JOB_NODELIST"))
 
 ALL_NODES="\${nodes[*]}" \
-srun --nodes=$NUM_NODES --ntasks=$NUM_NODES --ntasks-per-node=1 \
+srun --nodes=$NUM_NODES --ntasks=$NUM_NODES --ntasks-per-node=1 --kill-on-bad-exit=1 \
     --container-image=$CONTAINER \
     --container-name=container-on-node \
     --container-mounts=$MOUNTS \

benchmarks/swebench/data/.gitignore

@@ -1 +1,4 @@
 *benchmark.jsonl
+swebench_pro_benchmark.jsonl.tmp
+swebench_pro_image_digests.json
+swebench_pro_upstream/

benchmarks/swebench/pro/opencode.yaml

@@ -0,0 +1,23 @@
+config_paths:
+- responses_api_agents/opencode_sandboxed_agent/configs/opencode_agent.yaml
+- resources_servers/swebench_pro/configs/swebench_pro.yaml
+
+opencode_swebench_pro_resources_server:
+  _inherit_from: swebench_pro_resources_server
+
+opencode_sandboxed_agent_swebench_pro:
+  _inherit_from: opencode_sandboxed_agent
+  responses_api_agents:
+    opencode_sandboxed_agent:
+      remote_opencode_install_script_path: /mnt/s3-data/data/bxyu/opencode/install_cached_opencode.sh
+      remote_opencode_binary_path: /mnt/s3-data/data/bxyu/opencode/opencode-linux-x64
+      remote_opencode_musl_binary_path: /mnt/s3-data/data/bxyu/opencode/opencode-linux-x64-musl-v1.17.11
+      resources_server:
+        type: resources_servers
+        name: opencode_swebench_pro_resources_server
+      datasets:
+        - name: swebench_pro
+          type: benchmark
+          jsonl_fpath: benchmarks/swebench/data/swebench_pro_benchmark.jsonl
+          prepare_script: benchmarks/swebench/pro/prepare.py
+          num_repeats: 3

benchmarks/swebench/pro/prepare.py

@@ -0,0 +1,198 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Prepare SWE-bench Pro data and pin its per-instance evaluator assets."""
+
+import json
+import tarfile
+import urllib.request
+from collections.abc import Iterable, Mapping
+from pathlib import Path
+from time import sleep
+from typing import Any
+from urllib.error import HTTPError
+from urllib.parse import quote
+
+from datasets import load_dataset
+
+from nemo_gym.global_config import get_hf_token
+
+
+UPSTREAM_COMMIT = "ca10a60a5fcae51e6948ffe1485d4153d421e6c5"  # pragma: allowlist secret
+UPSTREAM_ARCHIVE_URL = f"https://github.com/scaleapi/SWE-bench_Pro-os/archive/{UPSTREAM_COMMIT}.tar.gz"
+DATASET_REVISION = "7ab5114912baf22bb098818e604c02fe7ad2c11f"  # pragma: allowlist secret
+EXPECTED_INSTANCE_COUNT = 731
+BENCHMARK_DIR = Path(__file__).parent.parent
+DATA_DIR = BENCHMARK_DIR / "data"
+OUTPUT_FPATH = DATA_DIR / "swebench_pro_benchmark.jsonl"
+UPSTREAM_CACHE_DIR = DATA_DIR / "swebench_pro_upstream"
+PROMPT_TEMPLATE = Path(__file__).with_name("prompt.txt").read_text(encoding="utf-8")
+
+
+def render_prompt(row: Mapping[str, Any]) -> str:
+    """Render the SWE-bench Pro coding prompt."""
+    materialized_prompt = PROMPT_TEMPLATE.format(
+        problem_statement=str(row["problem_statement"]),
+        requirements=str(row["requirements"]),
+        interface=str(row["interface"]),
+    )
+    return materialized_prompt
+
+
+def fetch_upstream_assets(cache_dir: Path = UPSTREAM_CACHE_DIR) -> Path:
+    """Download and extract the pinned evaluator source once."""
+    root = cache_dir / f"SWE-bench_Pro-os-{UPSTREAM_COMMIT}"
+    if root.is_dir():
+        return root
+
+    cache_dir.mkdir(parents=True, exist_ok=True)
+    archive_path = cache_dir / f"{UPSTREAM_COMMIT}.tar.gz"
+    urllib.request.urlretrieve(UPSTREAM_ARCHIVE_URL, archive_path)
+    with tarfile.open(archive_path, "r:gz") as archive:
+        archive.extractall(cache_dir, filter="data")
+    if not root.is_dir():
+        raise FileNotFoundError(f"Expected extracted upstream directory at {root}")
+    return root
+
+
+def _read_asset(upstream_root: Path, relative_path: str) -> str:
+    path = upstream_root / relative_path
+    if not path.is_file():
+        raise FileNotFoundError(f"Missing SWE-bench Pro evaluator asset: {path}")
+    return path.read_text(encoding="utf-8")
+
+
+def fetch_image_digest(dockerhub_tag: str, max_attempts: int = 8) -> str:
+    """Resolve a case-sensitive Docker Hub tag to an immutable digest."""
+    url = f"https://hub.docker.com/v2/repositories/jefzda/sweap-images/tags/{quote(dockerhub_tag, safe='')}"
+    for attempt in range(max_attempts):
+        try:
+            with urllib.request.urlopen(url, timeout=30) as response:
+                metadata = json.load(response)
+            break
+        except HTTPError as exc:
+            if exc.code != 429 or attempt == max_attempts - 1:
+                raise
+            retry_after = exc.headers.get("Retry-After")
+            delay = float(retry_after) if retry_after else min(2**attempt, 60)
+            print(f"Docker Hub rate-limited {dockerhub_tag}; retrying in {delay:g}s")
+            sleep(delay)
+    digest = metadata.get("digest")
+    if not isinstance(digest, str) or not digest.startswith("sha256:"):
+        raise ValueError(f"Docker Hub did not return a digest for {dockerhub_tag}")
+    return digest
+
+
+def _load_digest_cache(*paths: Path) -> dict[str, str]:
+    cache: dict[str, str] = {}
+    for path in paths:
+        if not path.is_file():
+            continue
+        try:
+            if path.suffix == ".json":
+                values = json.loads(path.read_text(encoding="utf-8"))
+                cache.update({str(tag): str(digest) for tag, digest in values.items()})
+            else:
+                for line in path.read_text(encoding="utf-8").splitlines():
+                    row = json.loads(line)
+                    if row.get("dockerhub_tag") and row.get("image_digest"):
+                        cache[str(row["dockerhub_tag"])] = str(row["image_digest"])
+        except (json.JSONDecodeError, OSError):
+            continue
+    return cache
+
+
+def _write_digest_cache(path: Path, cache: Mapping[str, str]) -> None:
+    temporary_path = path.with_suffix(path.suffix + ".tmp")
+    temporary_path.write_text(json.dumps(cache, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    temporary_path.replace(path)
+
+
+def enrich_row(row: Mapping[str, Any], upstream_root: Path, image_digest: str) -> dict[str, Any]:
+    """Attach verifier-only assets to one public dataset row."""
+    instance_id = str(row["instance_id"])
+    enriched = dict(row)
+    enriched.update(
+        {
+            "run_script": _read_asset(upstream_root, f"run_scripts/{instance_id}/run_script.sh"),
+            "parser_script": _read_asset(upstream_root, f"run_scripts/{instance_id}/parser.py"),
+            "base_dockerfile": _read_asset(upstream_root, f"dockerfiles/base_dockerfile/{instance_id}/Dockerfile"),
+            "instance_dockerfile": _read_asset(
+                upstream_root, f"dockerfiles/instance_dockerfile/{instance_id}/Dockerfile"
+            ),
+            "image_digest": image_digest,
+            "responses_create_params": {
+                "input": [
+                    {
+                        "role": "user",
+                        "content": render_prompt(row),
+                    }
+                ],
+            },
+            "subset": "pro",
+            "split": "test",
+            "evaluator_commit": UPSTREAM_COMMIT,
+            "dataset_revision": DATASET_REVISION,
+        }
+    )
+    return enriched
+
+
+def prepare(
+    dataset: Iterable[Mapping[str, Any]] | None = None,
+    upstream_root: Path | None = None,
+    output_fpath: Path = OUTPUT_FPATH,
+    image_digest_resolver=fetch_image_digest,
+    image_digest_cache_fpath: Path | None = None,
+) -> Path:
+    """Materialize the public Pro split as self-contained NeMo Gym JSONL."""
+    if dataset is None:
+        dataset = load_dataset(
+            "ScaleAI/SWE-bench_Pro",
+            split="test",
+            revision=DATASET_REVISION,
+            token=get_hf_token(),
+        )
+    if upstream_root is None:
+        upstream_root = fetch_upstream_assets()
+
+    output_fpath.parent.mkdir(parents=True, exist_ok=True)
+    if image_digest_cache_fpath is None:
+        image_digest_cache_fpath = output_fpath.parent / "swebench_pro_image_digests.json"
+    output_tmp_fpath = output_fpath.with_suffix(output_fpath.suffix + ".tmp")
+    digest_cache = _load_digest_cache(image_digest_cache_fpath, output_fpath, output_tmp_fpath)
+    count = 0
+    with output_tmp_fpath.open("w", encoding="utf-8") as output:
+        for row in dataset:
+            dockerhub_tag = str(row["dockerhub_tag"])
+            image_digest = digest_cache.get(dockerhub_tag)
+            if image_digest is None:
+                image_digest = image_digest_resolver(dockerhub_tag)
+                digest_cache[dockerhub_tag] = image_digest
+                _write_digest_cache(image_digest_cache_fpath, digest_cache)
+            output.write(json.dumps(enrich_row(row, upstream_root, image_digest)) + "\n")
+            count += 1
+    if count == 0:
+        raise ValueError("SWE-bench Pro preparation produced no rows")
+    if dataset is not None and hasattr(dataset, "num_rows") and count != EXPECTED_INSTANCE_COUNT:
+        raise ValueError(f"Expected {EXPECTED_INSTANCE_COUNT} SWE-bench Pro rows, got {count}")
+
+    output_tmp_fpath.replace(output_fpath)
+    print(f"Wrote {count} SWE-bench Pro problems to {output_fpath}")
+    return output_fpath
+
+
+if __name__ == "__main__":
+    prepare()

benchmarks/swebench/pro/prompt.txt

@@ -0,0 +1,32 @@
+<uploaded_files>
+/app
+</uploaded_files>
+I've uploaded a code repository in the directory /app. Consider the following PR description:
+
+<pr_description>
+{problem_statement}
+
+Requirements:
+{requirements}
+
+New interfaces introduced:
+{interface}
+</pr_description>
+
+Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?
+I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!
+Your task is to make the minimal changes to non-tests files in the /app directory to ensure the <pr_description> is satisfied.
+Follow these steps to resolve the issue:
+1. As a first step, it might be a good idea to find and read code relevant to the <pr_description>
+2. Create a script to reproduce the error and execute it using the bash tool, to confirm the error
+3. Edit the sourcecode of the repo to resolve the issue
+4. Rerun your reproduce script and confirm that the error is fixed!
+5. Think about edgecases and make sure your fix handles them as well
+Your thinking should be thorough and so it's fine if it's very long.
+
+# Important rules to follow:
+
+1. Do NOT fetch this task's known fix from the internet. That includes: the fixing PR, issue, or commit and its diff/patch (.../pull/<n>.diff, .patch, gh pr/gh issue, the GitHub API, or searching the issue number or error text); and the post-fix source of the code you are editing — or of its upstream or dependency projects — at any other branch, tag, or release (raw.githubusercontent.com/..., git clone/git fetch of the upstream repo, or using pip/npm/conda to download a different version to read its source).
+2. Do NOT check out or apply a branch, tag, or commit that post-dates this task in order to copy its fix.
+3. Do NOT look up the answer via a web search, Stack Overflow/gist/blog, or another AI model.
+4. Do NOT modify, disable, skip, or special-case the tests or checks used to grade your work.

resources_servers/swebench_pro/.gitignore

@@ -0,0 +1 @@
+logs/