Score on your own Intel x86-64 Linux machine matching the documented scoring
configuration: 8 dedicated judge CPUs, 16 GiB of memory for the task, SMT off
(one thread per core), Docker with Buildx, and spare CPUs for the system outside
the judge set. The reference configuration recorded in
judge/hosts/registry.json uses judge CPUs 0-7 and system CPUs 8-15. No cloud
account or particular provider is required; see judge.md for what
claimable means.
Run setup before starting a quiet scoring session, never next to a timed
run. tools/setup_scoring_host.sh is the single setup script.
The initial clone must skip LFS smudging, or Git can hydrate the entire multi-GB corpus before any setup/preflight code can run:
sudo apt-get update && sudo apt-get install -y git git-lfs
GIT_LFS_SKIP_SMUDGE=1 git clone "$REPOSITORY_URL" verifier # the OptimizationBench repository
git -C verifier lfs install --local --skip-smudgeFrom an existing checkout, sh tools/clone_verifier_checkout.sh REPOSITORY_URL DESTINATION
enforces this bootstrap pattern. It never pulls LFS objects. Later public and
authorized retained-heldout acquisition is scoped to exact task paths.
On a fresh machine, provision it once before setup.
tools/provision_scoring_host.sh installs Docker CE and the host tools,
disables background timers and reserves CPUs 8 and up for the system (systemd's
manager, Docker and containerd run there through CPUAffinity) and applies the
Docker MTU/MSS repair described below; rerun that repair after every reboot or
Docker restart. Run the judge
itself under taskset -c 0-7. The judge records both CPU sets, and a score is
claimable only when they are 0-7 and 8-15:
cd verifier
sudo bash tools/provision_scoring_host.sh
sudo JUDGE_VENV=/opt/judge sh tools/setup_scoring_host.sh
sudo /opt/judge/bin/python -m tools.scoring_host check --task sqlglot-mozilla-transpile --stage-publicChoose a writable JUDGE_VENV if /opt/judge is not writable. Setup creates that
venv and installs judge/requirements-interface.lock plus the complete
judge/requirements-host-registry.lock schema dependency closure, with
--require-hashes --only-binary=:all:. The judge must use its Python, not a
system interpreter with unrelated packages. Preflight imports/version-checks
every locked package and initializes all configured grammars plus the registry's
Draft 2020-12 schema validator. The interface lock pins nine parser packages.
These parsers belong on the host, never in agent task images.
judge/host-runtime.lock.json explicitly supports standard stable CPython
3.12 and 3.14, with a dedicated venv and the locked wheel ABI (not a
free-threaded interpreter). Bootstrap checks the interpreter before installing
packages; preflight checks it before parsing dependencies or comparing an audit.
An unsupported version gets a host-interpreter error and concrete
JUDGE_PYTHON=python3.12 JUDGE_VENV=/opt/judge-py312 sh tools/setup_scoring_host.sh
instructions, not a misleading external-command audit failure.
The registry closure pins jsonschema 4.19.2, attrs 25.4.0,
jsonschema-specifications 2025.9.1, referencing 0.37.0, rpds-py 0.30.0 and
typing-extensions 4.15.0. Every supported wheel hash is committed; typing
extensions is installed on both minors for an identical declared closure.
Maintainers may regenerate/review this separate lock with
python3 tools/lock_host_registry_dependencies.py; setup/preflight never resolve
or relock dependencies dynamically.
Read-only check fails with explicit commands for missing python3-venv,
GNU nm and c++filt from binutils, Git/Git LFS, Docker, buildx, iproute2, iptables and
conditionally ffmpeg for selected public PNG/RGB24 acquisition. It checks
local Git LFS configuration, reachable Docker/cgroup v2, readable
CPU/memory/PID/cpuset controllers and Docker bridge network state.
It does not prove the task image works, the host is quiet, or the host is a
registered scoring machine.
Install the Docker daemon through your distro setup before reservation.
Missing buildx has a hash-pinned Linux amd64/arm64 installer:
python3 -m tools.scoring_host install-buildx
docker buildx versionIt installs Buildx v0.24.0 into the active Docker CLI plugin directory, verifies the release SHA256, is idempotent for that exact binary, and refuses to overwrite a different operator-managed plugin or a symlink. It neither starts a builder nor replaces Docker. Other architectures require an explicitly reviewed plugin.
Network repair is a separate, explicitly mutating quiet-host action:
sudo python3 -m tools.scoring_host fix-network --mtu 1400 --confirm-no-timing-sessionIt requires a local Unix-socket Docker daemon and root/CAP_NET_ADMIN. It
enumerates actual Docker bridge interfaces, lowers MTUs only when larger than
the declared maximum, and adds checked, commented TCPMSS
--clamp-mss-to-pmtu rules in each bridge's incoming/outgoing direction.
IPv6-enabled bridges also receive ip6tables rules. Repeated execution does not
append existing rules or raise an already smaller MTU; rule-read permission
errors are not treated as an absent rule.
The repair does not restart Docker, edit daemon configuration, disable firewalls, alter non-Docker interfaces, or claim to be persistent across reboot. Rerun it after host/daemon restarts or creation of another bridge; preflight checks the live state again. Serialize setup commands. This structural check is not a substitute for a real network transfer; a completed public payload acquisition provides its own size/hash evidence.
An image's cache is not the verifier checkout's public corpus. The shared helper checks and materializes missing fetch-derived public files before any generator opens them:
/opt/judge/bin/python -m judge.public_input_preflight \
--task sqlglot-mozilla-transpile --fetch --out /tmp/sqlglot-public-inputs.jsonWithout --fetch, this is a read-only readiness check. With it, the helper calls
tools.stage_payload.stage(task, fetch=True, scope="public") as a function,
then authenticates every real public file/history again. It never runs the
catalog-writing staging CLI. Failed acquisition rolls back its batch.
Rules:
- Only explicit
scope: publicentries under the task environment are staged. Paths/symlinks, exact bytes/SHA256 or authenticated Git-history inventories must verify. Merely having a fetch recipe is not readiness. - Held-out fetch-only bodies remain outside the repository. No held-out contents are fetched, copied from the image, or used to generate cases.
- LFS pointer text is never accepted as an input body. Setup runs
git lfs install --local --skip-smudge, never global smudge/pull. Missing public LFS sources produce exact--include=<public paths> --exclude=''pull/checkout commands; no private/task-wide/global pull is performed. - Payload manifests are SHA-bound and must not change during staging. Expected outputs/manifests are never regenerated to admit new input data.
Integration hooks used by the shared judge:
from tools.scoring_host import scoring_preflight
from judge.public_input_preflight import ensure_public_inputs, needs_ffmpeg
host = scoring_preflight(require_ffmpeg=needs_ffmpeg(task))
public = ensure_public_inputs(task, fetch=True)
# Only now: differential.harness(task), generate(...), source/behavior gates.Call host preflight at the top of scoring before admission/build/samples. Call
the public helper before loading/invoking the generator, including direct
differential entry points; a scored path may call it earlier and reuse the
passing receipt until its inputs change. These helpers do not assign candidate
correctness or performance verdicts. PreflightError/PublicInputsError mean
invalid infrastructure, not candidate failure.
tools/judge_external_commands.json inventories launch sites/Step producers,
literal binary names, explicit dynamic producers, task-configured worker
commands and source hashes. It covers judge modules, shared runner, public
payload staging/acquisition, differential generators, task drivers and setup.
Host control-plane dependencies include Docker, GNU nm/c++filt, Git/LFS and the locked
Python venv; optional public RGB24 conversion uses host ffmpeg. Node/Go/C
compilers, build scripts, generated task programs, task-configured argv and
their descendants normally run in pinned worker images, not on the scoring
host. Explicit LocalExecutor/direct C-preprocessor modes are separately marked,
not silently substituted for an image's pinned toolchain.
Native-surface projection specifically requires c++filt -n; preflight checks
a known Itanium function/vtable and unchanged invalid/unmangled names. Missing/incompatible
demangling fails with the binutils installation command rather than dropping
unproven weak symbols or treating them as absent.
Dynamic argv remains visible rather than inventing a complete static binary list. Pinned-image build/test/process controls must exercise those dependencies; this audit is not a successful-execution claim for arbitrary candidate children. No raw shell scripts, endpoint configurations or credentials are dumped in the manifest.
stage_payload.py --task NAME --fetch changes that task's requested payload
scope only and never calls catalog generation. Its receipt explicitly says
catalog_refreshed: false; failed acquisition rolls back its batch.
Catalog maintenance/checks have an explicit task subset:
python3 tools/update_catalog.py --task sqlglot-mozilla-transpile --check
python3 tools/update_catalog.py --task sqlglot-mozilla-transpile --refresh-bindings
python3 tools/check_tasks.py --task sqlglot-mozilla-transpile --require-readyThe task subset propagates through distribution normalization, binding refresh
and generation. It never rewrites another task's provenance, global README or
dataset selections. Prior global collision disclosures are preserved, not
removed by a local check. Repository-wide rewriting requires explicit --all;
the staging path never requests it.
Unfetched/missing LFS objects are per-task unavailable rows, not JSON parsing
exceptions or an excuse to bind pointer text. Unavailable tasks are not rewritten.
An unavailable requested scope exits 2 after structured reporting, never
claims success, and does not stop unrelated materialized tasks from being checked.
Real materialized byte/SHA corruption or malformed JSON remains fatal; it is
not relabeled missing/unsupported.
Cross-task leak checks still cover materialized packages and explicitly report
complete: false plus unavailable task names when the audit is partial. An
unavailable task's safety is not certified. Full-repository maintenance
must finish the complete audit before making a repository-wide claim.
Git LFS is required for retained held-out inputs and expected-output bodies,
not only public recovery packets. On a verifier checkout, initialize
git lfs install --local --skip-smudge, then recover only the exact listed
retained paths with git lfs pull --include='tasks/NAME/tests/heldout/EXACT_PATH' --exclude=''
and git lfs checkout 'tasks/NAME/tests/heldout/EXACT_PATH'. Never materialize
fetch-only held-out reference bodies in the repository or agent image; those
continue to use verifier acquisition outside the checkout. No global LFS pull
is an implicit setup/staging/check action.
After producers or dependencies change, review them and refresh this audit explicitly:
python3 -m tools.judge_command_audit --check
python3 -m tools.judge_command_audit --write --reviewed
python3 -m tools.judge_command_audit --checkPreflight never silently regenerates the audit. Source changes after review
invalidate it. This source-specific audit refresh is separate from task catalog
generation and never changes task manifests/provenance.
Argument identities use exact UTF-8/LF source segments, not ast.dump or other
interpreter-version-specific AST reprs. Identical checked-out source therefore
produces the same audit on both supported Python minors. A new source commit
still requires one reviewed manifest refresh; do not locally relock a
different source tree just to make a host pass.