Skip to content

FIX Stop scenario workers when an atomic attack is cancelled - #2851

Merged
Roman Lutz (romanlutz) merged 10 commits into
microsoft:mainfrom
biefan:fix/scenario-worker-cancellation
Oct 6, 2026
Merged

Roman Lutz (romanlutz) merged 10 commits into
microsoft:mainfrom
biefan:fix/scenario-worker-cancellation

Conversation

@biefan

@biefan biefan (biefan) commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Description

When an atomic attack is cancelled, Scenario.run_async() can return and persist CANCELLED while sibling workers still execute. A ready sibling can also start a queued attack. The scenario must stop admitting work and finish worker cleanup before exposing its terminal state.

For example, with concurrency 2, A and B are running and C is queued. If B cancels while A is awaiting target cleanup, A must finish that cleanup and C must never start.

The supervisor now owns cancellation through one controlled path:

  1. Shield the initial worker gather so caller cancellation does not automatically cancel workers before the supervisor handles it.
  2. Stop queue admission. Cancel unfinished workers only if they do not already have an outstanding cancellation request; an unfinished task may already be awaiting cleanup in its finally block.
  3. Shield the drain from subsequent caller cancellation, wait for every worker, and retrieve the original gather's exception as well. Only then propagate cancellation and let the scenario persist its terminal state.

Workers also stop admission immediately when cancellation originates inside an atomic attack. Ordinary failures retain the existing behavior: in-flight work may complete and persist, queued work stops, and multiple failures are reported together. This extends the cancellation lifecycle from #2342 without changing retry or completed-result semantics.

Tests and Documentation

  • Added an event-gated regression through real Scenario, AtomicAttack, AttackExecutor, PromptSendingAttack, and conversation lifecycle code, with a mocked target. One worker exits quickly while another is held inside reset_conversation_async().
  • Both the single-caller-cancel and repeated-caller-cancel cases failed on the previous PR head: the slow execution task received two cancellation requests. They now verify exactly one request, cleanup remaining active until its gate opens, no queued send, and IN_PROGRESS until cleanup finishes followed by CANCELLED with one attempt.
  • Existing regressions retain child-originated cancellation, queue admission, completed-result persistence and duplicate-free resume coverage.
  • Focused scenario/executor/backend/Crescendo tests: 301 passed.
  • make unit-test (Python 3.11, default dependencies): 20,400 passed, 146 skipped, 1 failed. The sole failure is the pre-existing test_get_seed_dataset_summaries_follows_a_trailing_blank_insensitive_collation in tests/unit/memory/memory_interface/test_interface_seed_prompts.py, previously reproduced on unmodified upstream in the same environment.
  • All applicable pre-commit hooks passed with all optional dependencies installed, including repository-wide ty check pyrit.

To run the cancellation regressions:

uv run pytest tests/unit/scenario/core/test_scenario.py tests/unit/scenario/core/test_scenario_partial_results.py -q -k cancellation

Updated the worker-pool docstring to describe cancellation ownership. Tests use local SQLite and controlled target behavior; no live model endpoint was used.

Comment thread pyrit/scenario/core/scenario.py Outdated
Comment thread pyrit/scenario/core/scenario.py Outdated
Comment thread pyrit/scenario/core/scenario.py
@romanlutz Roman Lutz (romanlutz) self-assigned this Oct 3, 2026
Roman Lutz (romanlutz) and others added 4 commits October 4, 2026 23:28
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Stop queue admission on new supervisor cancellation requests and drain the original gather together with all workers. Cover completion callbacks, ready siblings, repeated cancellation, and same-task resume.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roman Lutz (romanlutz) and others added 3 commits October 5, 2026 21:04
Retain caller- and worker-originated cancellation regressions alongside main's async-only test readiness and cleanup handling.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 6, 2026
Merged via the queue into microsoft:main with commit d0367ab Oct 6, 2026
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants