Audit pinned AgentHarm call-response and email-link semantics - #2
Draft
YusefSyed wants to merge 3 commits into
Draft
Audit pinned AgentHarm call-response and email-link semantics#2YusefSyed wants to merge 3 commits into
YusefSyed wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AgentHarm's grading helpers select call arguments and responses separately. This optional audit makes their behavior inspectable on partial, rejected and reordered benign transcripts, and exercises a benign email-link criterion that can award credit when its extracted URL is empty.
The integration imports pinned Inspect Evals and Inspect AI code, checks source and dependency-lock hashes, and produces a byte-reproducible observation packet. It covers 14 helper fixtures, three identical-transcript/different-fixture-effect pairs, and seven grader fixtures with positive and negative controls. Two tests replay messages through an Inspect Task and AgentHarm's combined scorer. A dedicated CI job repeats the tests and artifact verification.
Validation on Python 3.12: 20 audit tests passed; Ruff and strict mypy passed; artifact verification passed. The existing 158 core tests passed with 82% branch-aware coverage, and frozen core artifacts reproduced unchanged. Third-party deprecation warnings remain.
The first clean Linux run exposed a tokenizer cache dependency that was already present locally. An empty-cache reproducer failed at the offline guard as expected. Setup now explicitly prepares the hash-checked
o200k_basevocabulary before the audit; all 20 tests and artifact verification passed using that newly prepared cache, with runtime network access still blocked. The workflow passes actionlint.The fixtures use mocked semantic judgments. Timeout/output-limit labels are injected, file effects belong to the controlled fixture, and the two response-association mismatches concern partial or externally reordered histories. This does not run a normal model/tool loop, estimate model safety, or change upstream scoring policy. AgentHarm's full license notice is retained. The packet supports the discussion in UKGovernmentBEIS/inspect_evals#2290; no incremental upstream scoring PR is proposed.
Implementation and test preparation were assisted by OpenAI Codex.