[thomas-bartlett]

Summary

Adds an Inspect-native SecurityEval harness for measuring how CodeGuard affects model-generated Python code across all 121 pinned benchmark cases.

The experiment compares three conditions:

• the standard task prompt with no skill
• the same task with a short secure-implementation prompt
• the standard task with the repository's exact CodeGuard skill available for automatic routing

Codex runs inside a hardened, networkless Docker sandbox. The harness captures a bounded solution.py in Inspect's per-sample store without rewriting the raw model response, validates that the scaffold was meaningfully implemented while preserving its interface, and scores valid non-stub output with Semgrep through Inspect's normal scoring workflow.

Security and reproducibility

• Installs the repository CodeGuard skill as exact validated bytes and makes it read-only to the agent.
• Never imports, compiles, or executes captured solutions.
• Uses a networkless, least-privilege Docker sandbox with bounded resources and temporary filesystems.
• Exports only a stable, bounded UTF-8 solution.py through a strict wire contract.
• Validates stored artifacts and Semgrep reports fail-closed before scoring.
• Pins the dataset revision and hash, runtime dependencies, Codex CLI, Inspect AI, Inspect SWE, and Semgrep versions.
• Records output validity, implementation status, source size, Semgrep findings, and detected CodeGuard loading separately.
• Documents that the live p/security-audit registry rules are mutable and fetched for each scanned sample.

Validation

• uv lock --check
• uv run --locked pytest — 212 passed; 4 Docker tests deselected
• uv run --locked pytest -m docker — 4 passed
• uv run --locked python -m compileall -q codeguard_evals
• uv run --locked inspect list tasks codeguard_evals/securityeval
• Docker Compose configuration validation
• Bounded live Semgrep smoke test
• Dependency audit — no known vulnerabilities in the evaluation environment
• git diff --check

The full 1,089-generation provider matrix is intentionally not part of PR validation because it incurs provider cost; the harness and its real Codex-in-Docker path are covered by the integration suite.

Related

• Supersedes Add Inspect-native evaluation foundation for CodeGuard #87, which described an earlier implementation that was subsequently rewritten.
• Refs Add evaluation harness for Project CodeGuard #86
• Related to Discussion AI assisted Code Generation: Testing Harness Literature Review #70

[aacarter1]

Overall, this is a thoughtful and well-hardened foundation. I have two non-blocking concerns before its results are treated as decision-grade:

1. The Docker integration tests covering the actual sandbox and Semgrep security boundaries are manual rather than part of required CI. Could we consider an enforced integration gate, or document who owns confirming that manual gate before releases?
2. valid_output currently means bounded, non-empty, parseable Python; it does not establish meaningful implementation or interface preservation, and parse-valid stubs are accepted. The README documents this limitation, but the PR summary still claims meaningful, non-stub, interface-preserving validation. Could those claims be reconciled and future published results clearly framed as conditional on parse-valid output?
   Neither point necessarily needs to block merging this initial harness, but both matter for how confidently its results can be interpreted publicly.