Skip to content

Add golden patch adaptations. - #2853

Draft
ffrujeri wants to merge 7 commits into
mainfrom
ffrujeri/swebench-pro-golden-patch-enhancements
Draft

Add golden patch adaptations.#2853
ffrujeri wants to merge 7 commits into
mainfrom
ffrujeri/swebench-pro-golden-patch-enhancements

Conversation

@ffrujeri

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR improves OpenSandbox runtime parity and failure diagnostics for SWE-bench Pro golden-patch verification
without changing the upstream patch-application contract or exact test-name grading.

Runtime parity adaptations

The verifier adds opt-in compatibility handling for four OpenSandbox divergences reproduced against the official
Docker evaluator:

  • removes inherited OTEL_* configuration for the affected Flipt instance;
  • makes Node prefer IPv4 for the affected NodeBB instance whose server listens on IPv4 while localhost may resolve
    to ::1;
  • waits for a literal Redis PONG for the affected NodeBB instance instead of treating any successful redis-cli
    invocation as readiness;
  • starts Xvfb explicitly and disables pytest-xvfb auto-start for the affected qutebrowser instance.

Each adaptation is restricted to the exact pinned dataset instance where the divergence was observed. Other Flipt,
NodeBB, and qutebrowser tasks retain their original evaluator scripts and environment.

The existing Go module prefetch remains a separate adaptation. It warms the module cache before running Go tests but
does not rewrite the upstream test command.

Verification diagnostics

The generated entry script now records exit codes independently for:

  • repository reset;
  • base-commit checkout;
  • patch application;
  • OpenSandbox runtime setup;
  • benchmark test setup;
  • Go module prefetch;
  • test execution;
  • parser execution.

These statuses are persisted in the evaluation logs and returned by the resources server. Setup, patch, checkout, or
parser failures can therefore be distinguished from ordinary test failures. A failed infrastructure/setup phase
cannot be reported as a completed resolved evaluation, while grade_output() continues to require the exact
fail_to_pass | pass_to_pass test names with PASSED status.

OpenSandbox configuration and evaluation helper

  • Pins SWE-bench Pro sandboxes to native linux/amd64, matching the benchmark images.
  • Enables the runtime-parity layer in both SWE-bench Pro server configurations.
  • Adds optional instance_id_prefixes filtering to apply_golden_patch.py for focused retries and diagnostics.

Validation

  • SWE-bench Pro unit tests: 27 passed.
  • Ruff formatting and lint checks: passed.
  • Scoped pre-commit checks: passed.
  • Four directly affected golden-patch instances: 4/4 resolved at concurrency 2.
  • Full stress run: 718/731 resolved (98.22%) at concurrency 731.

The full stress-run result should not be treated as a strict A/B comparison with the earlier 719/731 run because the
runs used different concurrency and can include unrelated benchmark/runtime flakes. The focused 4/4 run validates
the intended adaptations directly.

No expected test names are normalized, no required tests are skipped, and the three known malformed benchmark
contracts remain unresolved rather than being waived.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass
    when applicable).
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ffrujeri
ffrujeri force-pushed the ffrujeri/swebench-pro branch 4 times, most recently from fd20b40 to f7707d9 Compare August 31, 2026 04:33
Base automatically changed from ffrujeri/swebench-pro to main August 31, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant