Skip to content

REFACTOR: Count execution units in pyrit_scan progress bar - #2549

Closed
Richard Lundeen (rlundeen2) wants to merge 1 commit into
copilot/romanlutz-scenario-pr-03-live-upstreamfrom
copilot/rlundeen2-cli-progress-units
Closed

REFACTOR: Count execution units in pyrit_scan progress bar#2549
Richard Lundeen (rlundeen2) wants to merge 1 commit into
copilot/romanlutz-scenario-pr-03-live-upstreamfrom
copilot/rlundeen2-cli-progress-units

Conversation

@rlundeen2

Copy link
Copy Markdown
Contributor

Stacked on #2374. Base branch is copilot/romanlutz-scenario-pr-03-live-upstream, not main. Review only the diff on top of that branch. This change was extracted from #2374 (commit 5b3f8f2d7 removed it there) so it can be reviewed on its own.

What changed

pyrit_scan run prints a live progress bar while it polls a scenario run. The bar counted techniques completed. It now counts execution unitscompleted_attacks / total_attacks on ScenarioRunSummary — and the label changes from techniques: to units:.

An execution unit is (atomic_group_id, seed_group_id): one objective run under one atomic attack group.

Before:

  [███████████████░░░░░░░░░░░░░░░] techniques: 2/4 (50%) | success rate: 30% | IN_PROGRESS

After:

  [███████████████░░░░░░░░░░░░░░░] units: 5/10 (50%) | success rate: 30% | IN_PROGRESS

Why the technique-based bar existed, and why that constraint is gone

The removed code carried this comment:

# The bar tracks techniques completed / total, which is the only ratio we can
# honestly compute mid-run: the server only knows about attacks already persisted,
# so an attacks-based bar would always read 100%.

Techniques were a proxy metric. The server could not report a planned attack total, so total_attacks only ever held the number of attacks already written to memory, and completed / total was always 1.

The backend now persists a normalized run plan and computes canonical planned-vs-completed execution-unit counts. total_attacks is now a real planned total, so the bar can measure the work the user actually cares about instead of a proxy.

Deleted plumbing

total_techniques and scenario_meta were threaded through _run_scenario_async and _poll_until_terminal_async only to compute the old ratio. Both are gone, along with the max(total_techniques, techniques_done) reconciliation that existed because a user-supplied technique could be an aggregate (single_turn) that expands into N concrete techniques on the server.

This also removes a real inconsistency: pyrit_shell.py computed total_techniques from request.techniques only, with no all_techniques fallback, so the shell's bar could disagree with pyrit_scan's for aggregate techniques. Both now read the same server-computed counts.

Files

  • pyrit/cli/_output.pyprint_scenario_run_progress drops the total_techniques parameter and computes the bar from the run's unit counts.
  • pyrit/cli/pyrit_scan.py — removes the total_techniques / scenario_meta threading.
  • pyrit/cli/pyrit_shell.py — same call-site update in the interactive shell.
  • tests/unit/cli/test_output.py, tests/unit/cli/test_pyrit_scan.py — updated assertions and call sites.

Net: 20 insertions, 40 deletions. No frontend or server change; this reads fields that already exist on ScenarioRunSummary.

Tests

uv run pytest tests/unit/cli -q → 516 passed.

pre-commit run --all-files: the two failing hooks (Check Memory Migrations, ty) fail identically on the base branch without this change.

The scan progress bar counted techniques completed / total because the server only knew about attacks already persisted, so an attacks-based bar always read 100%. The backend now persists a normalized run plan, so total_attacks is a real planned total.

Switch the bar to run.completed_attacks / run.total_attacks, relabel the output from 'techniques:' to 'units:', and delete the total_techniques / scenario_meta plumbing that only existed to compute the old ratio.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant