feat: add SecurityEval reporting and CodeGuard 1.4.0 results

.github/workflows/validate-codeguard-evaluations.yml

@@ -41,10 +41,10 @@ jobs:
           version: "0.12.5"
 
       - name: Install locked dependencies
-        run: uv sync --locked --managed-python
+        run: uv sync --locked --managed-python --group reporting
 
       - name: Run non-Docker tests
-        run: uv run --locked pytest
+        run: uv run --locked --group reporting pytest
 
       - name: Discover Inspect tasks
         run: uv run --locked inspect list tasks codeguard_evals/securityeval

docs/results/2026-09-codeguard-1.4.0/index.md

@@ -0,0 +1,45 @@
+# CodeGuard 1.4.0 on SecurityEval
+
+Findings are Semgrep scanner results, not confirmed vulnerabilities. [Evaluation Results](../index.md) explains how the evaluation works and its limits.
+
+Recorded comparison settings match; harness snapshots differ.
+
+## Key numbers
+
+Findings use shared cases; tokens and costs per case use all attempted cases. n/a means a value was not recorded.
+
+### GPT-5.6 Sol (111 shared cases)
+
+Limits: 16 turns; 600 s generation; 1,200 s per case; output token limit 32,768.
+
+| Condition | Findings | High + Medium | Likely vulnerabilities | Tokens per case | Estimated cost per case (USD) | Estimated total cost (USD) | Total USD vs baseline | Unfinished |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Baseline | 42 | 17 | 15 | 42.7K | $0.0872 | $10.55 | — | 1 of 121 (1%) |
+| Secure prompt | 22 | 4 | 3 | 44.9K | $0.1040 | $12.59 | +19.4% | 1 of 121 (1%) |
+| CodeGuard 1.4.0 | 26 | 6 | 6 | 162.4K | $0.2506 | $30.33 | +187.6% | 9 of 121 (7%) |
+
+### GPT-5.6 Luna (116 shared cases)
+
+Limits: 16 turns; 600 s generation; 1,200 s per case; output token limit 32,768.
+
+| Condition | Findings | High + Medium | Likely vulnerabilities | Tokens per case | Estimated cost per case (USD) | Estimated total cost (USD) | Total USD vs baseline | Unfinished |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Baseline | 47 | 18 | 15 | 35.3K | $0.0043 | $0.52 | — | 0 of 121 (0%) |
+| Secure prompt | 30 | 8 | 8 | 39.2K | $0.0054 | $0.65 | +24.4% | 3 of 121 (2%) |
+| CodeGuard 1.4.0 | 35 | 11 | 9 | 70.1K | $0.0071 | $0.86 | +65.1% | 2 of 121 (2%) |
+
+## Charts
+
+![Semgrep findings against estimated total USD cost](findings.png)
+
+![How each attempt ended](outcomes.png)
+
+![Findings by rule confidence](confidence.png)
+
+## Cost estimates
+
+USD estimates use rates checked October 1, 2026, including Sol promotional pricing and cache-read discounts. Inspect 0.3.255 omitted cache-write counts, so all non-cached input is conservatively charged at 1.25 times the ordinary input rate. Actual provider charges may differ.
+
+## Data
+
+[results.csv](results.csv) has the summary numbers for each run and epoch, and [samples.csv](samples.csv) has one row per attempt.

docs/results/2026-09-codeguard-1.4.0/results.csv

@@ -0,0 +1,7 @@
+run,epoch,model,condition,codeguard_version,agent_turn_limit,agent_seconds,sample_seconds_limit,agent_output_tokens,cases,shared_cases,findings,likely_vulnerabilities,tokens_per_case,unfinished,recorded_cost_usd,estimated_cost_usd,cost_per_case_usd,cost_change_pct,confidence_high,confidence_medium,confidence_low
+gpt-5.6-sol-baseline,1,GPT-5.6 Sol,baseline,,16,600,1200,32768,121,111,42,15,42673.57024793389,1,,10.546194,0.08715862809917355,0.0,3,14,25
+gpt-5.6-sol-secure-prompt,1,GPT-5.6 Sol,secure_prompt,,16,600,1200,32768,121,111,22,3,44888.33884297521,1,,12.5871242,0.10402581983471074,19.352291452252814,0,4,18
+gpt-5.6-sol-codeguard-1.4.0,1,GPT-5.6 Sol,codeguard,1.4.0,16,600,1200,32768,121,111,26,6,162380.4958677686,9,,30.327536600000002,0.25064079834710745,187.56854463325823,2,4,20
+gpt-5.6-luna-baseline,1,GPT-5.6 Luna,baseline,,16,600,1200,32768,121,116,47,15,35251.818181818184,0,,0.52378927,0.00432883694214876,0.0,3,15,29
+gpt-5.6-luna-secure-prompt,1,GPT-5.6 Luna,secure_prompt,,16,600,1200,32768,121,116,30,8,39152.89256198347,3,,0.65134714,0.005383034214876034,24.35289863803436,1,7,22
+gpt-5.6-luna-codeguard-1.4.0,1,GPT-5.6 Luna,codeguard,1.4.0,16,600,1200,32768,121,116,35,9,70086.76859504133,2,,0.86474549,0.00714665694214876,65.09415895442075,2,9,24

docs/results/index.md

@@ -0,0 +1,74 @@
+# Evaluation Results
+
+These pages publish results from the
+[CodeGuard evaluation harness](https://github.com/cosai-oasis/project-codeguard/tree/main/evaluations/codeguard-inspect),
+which compares Semgrep findings in code written by an AI coding agent and
+records model usage and available cost data. Results are exploratory.
+
+## How the evaluation works
+
+- **Tasks:** the 121 Python tasks of the
+  [SecurityEval](https://github.com/s2e-lab/SecurityEval) benchmark, each built
+  around a known weakness.
+- **Agent:** the Codex coding agent, working in an isolated container with no
+  network access. Each attempt has a turn and time limit.
+- **Conditions:** every model runs the same tasks three ways:
+    - **Baseline:** the standard task prompt.
+    - **Secure prompt:** the same prompt plus a one-sentence request to write
+      secure code.
+    - **CodeGuard:** the standard prompt with the CodeGuard skill installed. The
+      agent decides whether to use it.
+- **Scoring:** the harness keeps the final file, checks its Python syntax, and scans it with a pinned version of Semgrep and its public rules.
+  Generated code is never run.
+
+## What the pages show
+
+- **Epochs:** each pass over the cases is reported separately.
+  For multi-epoch runs, token and cost totals use each case's final saved
+  attempt. Retry usage recorded only for the whole run is excluded.
+- **Limits:** recorded turn, generation-time, case-time, and output-token caps.
+  Shown on the report and in `results.csv`. Missing limits stay blank in the CSV.
+- **Findings:** Semgrep security findings at all confidence levels, counted on the shared cases,
+  where every run for a model produced changed, valid code. Unchanged or
+  invalid outputs are excluded. An attempt stopped by a limit can still
+  contribute findings if its output passes those checks, and also counts
+  as unfinished. Passing the checks does not establish functional correctness.
+- **Likely vulnerabilities:** findings from rules tagged `vuln`.
+  This is a rule category, not a confidence rating.
+- **High + Medium:** findings from rules rated High or Medium confidence.
+  The confidence chart shows High, Medium, and Low on the same shared cases.
+  Low-confidence findings remain in the total.
+- **Tokens per case:** recorded tokens divided by all attempted cases.
+  Includes cached input and retries. Token counts are not costs.
+- **Costs:** total USD and USD per case, using complete recorded costs or
+  estimates calculated from supplied model prices. When prices are supplied,
+  all runs use those rates. Per-case costs average all attempted cases,
+  including unfinished ones. Every run needs a complete recorded or estimated
+  cost.
+- **Total USD vs baseline:** percentage change in total USD cost from the same
+  model's baseline, calculated from unrounded amounts. Missing costs stay blank.
+- **Unfinished attempts:** attempts that hit any limit or ended with an error.
+
+## Limitations
+
+- **Static analysis only.** Findings are scanner results, not confirmed
+  vulnerabilities, and no findings does not mean the code is secure or even
+  correct. SecurityEval has no functional or exploit tests.
+- **Limited scanner coverage.** The pinned rules flag only 48 of the
+  benchmark's 121 known-insecure reference programs, so many weaknesses go
+  unseen.
+- **One benchmark, one language.** All tasks are Python.
+- **Small numbers.** Each epoch is one pass over 121 cases. A difference
+  of a few findings can be chance.
+- **Limits shape the results.** Attempts that hit a limit leave work unfinished.
+  Compare runs only when their settings match; each page warns when they do
+  not.
+- **Using the skill is up to the agent.** Reading CodeGuard does not prove the
+  agent followed it.
+
+## Result sets
+
+The [results exporter](https://github.com/cosai-oasis/project-codeguard/tree/main/evaluations/codeguard-inspect#export-results)
+creates the page, charts, and public CSVs together from completed harness logs.
+
+- [CodeGuard 1.4.0 on SecurityEval](2026-09-codeguard-1.4.0/index.md)

evaluations/codeguard-inspect/README.md

@@ -315,6 +315,58 @@ unchanged outputs first. Subcategory and confidence columns remain available but
 hidden by default to keep the initial view readable. Finding colors are
 zero-anchored neutral magnitude cues rather than green/red security judgments.
 
+## Export Results
+
+The exporter writes a generated `index.md`, three charts per epoch,
+`results.csv`, and `samples.csv`.
+[Evaluation Results](../../docs/results/index.md) explains the metrics and limits.
+
+First run the full `inspect eval-set` command from [Run](#run), which runs all
+three conditions, with a new `--log-dir` such as `logs/securityeval-report`. It
+writes one log per condition. Then pass those three logs to the exporter:
+
+```bash
+uv run --locked --group reporting python -m codeguard_evals.export_results \
+  --title "CodeGuard 1.5.0 on SecurityEval" \
+  --run "Model A" logs/securityeval-report/BASELINE.eval \
+  --run "Model A" logs/securityeval-report/SECURE_PROMPT.eval \
+  --run "Model A" logs/securityeval-report/CODEGUARD.eval \
+  --output-dir ../../docs/results/2026-10-codeguard-1.5.0
+```
+
+- `--run` takes a public model label and one final log. Repeat for each condition
+  and model, in display order; provider and deployment names are not published.
+- `--epoch N` selects one epoch. By default, every epoch has separate tables and
+  charts. Runs for a model must attempt the same cases in every epoch and have
+  matching epoch counts.
+- `--output-dir` must be new. Nothing is written until all logs pass validation.
+
+Every run needs a complete cost for the findings chart. With the current
+Inspect 0.3.255 pin, OpenAI runs need `--prices FILE.json` when exporting, even
+if costs were recorded. With Inspect 0.3.260 or newer, future runs can record
+costs using [`--model-cost-config`](https://inspect.aisi.org.uk/options.html).
+Prices are per million tokens, keyed by the public `--run` labels; see the
+[1.4.0 prices](reporting/securityeval-1.4-prices.json).
+Keep deployment-specific Inspect configuration private.
+
+With `--prices`, every run uses the supplied rates for tables, charts, and cost
+comparisons. Original recorded costs stay separate in the CSV. Without it, the
+report uses complete recorded costs. `--cost-note TEXT` explains estimate
+assumptions. No prices are fetched.
+For OpenAI logs from Inspect before 0.3.260 (or an unknown version) with missing
+cache-write counts, recorded costs are unreliable: supply `--prices` to estimate
+them. Non-cached input uses the higher input or cache-write rate; cache reads
+retain their discount.
+The 1.4.0 prices were checked October 1, 2026, including
+[Sol promotional pricing](https://developers.openai.com/api/docs/models/gpt-5.6-sol).
+Actual provider charges may differ.
+
+To regenerate a report, export the original logs into a new directory.
+
+Use trusted local logs. Scores are copied, never rescored, and only allowlisted
+fields are exported. Review every generated file before committing, then link the report from
+`docs/results/index.md`. Regenerate and replace the page, charts, and CSVs together.
+
 ## Measurement Details
 
 ### Skill routing
@@ -439,7 +491,7 @@ Run non-live checks:
 
 ```bash
 uv lock --check
-uv run --locked pytest
+uv run --locked --group reporting pytest
 uv run --locked python -m compileall -q codeguard_evals
 uv run --locked inspect list tasks codeguard_evals/securityeval
 docker compose --file sandbox/compose.yaml config --quiet
@@ -458,7 +510,7 @@ a manual pre-merge gate:
 
 ```bash
 uv run --locked python -m codeguard_evals.prefetch
-uv run --locked pytest -m docker --basetemp=logs/pytest-docker
+uv run --locked --group reporting pytest -m docker --basetemp=logs/pytest-docker
 ```
 
 Together, they exercise the real named Semgrep service, a known positive finding,