Skip to content

Improve SecurityEval isolation and generation controls - #128

Merged
thomas-bartlett merged 1 commit into
mainfrom
fix/securityeval-harness
Sep 18, 2026
Merged

thomas-bartlett merged 1 commit into
mainfrom
fix/securityeval-harness

Conversation

@thomas-bartlett

Copy link
Copy Markdown
Contributor

Summary

  • Move Codex state outside the task workspace and give all conditions the same minimal environment notice.
  • Expose and record generation limits; default to 16 turns and 600 seconds, with a 1,200-second task backstop.
  • Support both CodeGuard skill layouts without inferring versions, and strengthen compatibility and sandbox tests.

Dataset, Semgrep contract, scoring semantics, dependencies, and skill contents are unchanged. Comparisons should use matching environment and budget settings.

Validation

  • 286 non-Docker tests passed.
  • Five mock-backed Docker integration tests passed; test containers cleaned up.
  • Lockfile, pinned artifacts, task discovery, compilation, and Compose configuration verified.

Follow-up to #119.

@aacarter1 aacarter1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found, lgtm

@thomas-bartlett
thomas-bartlett merged commit ea88e31 into main Sep 18, 2026
5 checks passed
@thomas-bartlett
thomas-bartlett deleted the fix/securityeval-harness branch September 18, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants