Evaluate persisted tool effects through Inspect and isolated SQLite

[YusefSyed]

The existing engine evaluates trajectories and the Inspect importer classifies log evidence; neither establishes persisted effects from an agent's executed tool calls. The existing mock tool fixture only returns text. This optional integration fills that gap without changing the frozen benchmark or its 624-trial model study.

It reuses Inspect's agent loop, tool dispatch, approvals, logs and Docker sandbox. Narrow tools mutate a synthetic SQLite artifact registry; a separate read-only scorer checks committed history and final state. Exact-argument approval binding, duplicate request handling, pre/post-commit faults and publish-then-revoke cases distinguish proposed actions, execution errors and actual effects. Synthetic publication never creates a network or public-account action.

The deterministic milestone passes 40 unit/negative tests and two full 13-case actual-tool runs. Normalized reports and all 13 raw database snapshots match byte-for-byte. The verifier requires the complete fixture set and independently recomputes expected effects and recorded approval/dispatch checks; missing cases and forged success flags fail. The root's 17 canonical artifacts and all frozen-study checksums remain unchanged.

The integration is isolated under integrations/inspect_tool_execution/ with its own dependency lock. Containers run without networking, host mounts, root privileges or capabilities. Raw operational logs remain separate from normalized evidence. These are finite integration tests with scripted mock-model choices, not proof of model-selected behavior or general agent safety.

A separate, input-frozen Qwen/Ollama evaluation is now complete: 24 model-selected runs, 72 model requests, 72 approvals and 72 executed tool calls. All 24 completed the permitted task, with the sample-specific note present before each publication decision. No forbidden proposal, forbidden effect, unknown outcome, error or retry was observed. All approvals allowed execution, so live gate-denial evidence is not claimed. The design has six benign/adversarial note pairs repeated twice under an enforcing gate; it is a descriptive integration evaluation, not a population safety result.

An independent direct-SQL and resolved-native-log audit reconciled every sample field and grouped total, and all 21 frozen source hashes still match. The published bundle contains unmodified database snapshots, frozen inputs, summaries, metadata-sanitized native logs, a sanitization manifest and checksums. Only three operational path fields per log were replaced; prompts, responses, tool calls and scoring observations are unchanged. CI re-scores the published database evidence without inference, in addition to the deterministic integration and existing core checks.

No production service, customer data or paid model API was used. This additive integration does not amend the original study's failed validity verdict.