Add live Scenario progress - #2374
Conversation
58c5c56 to
4337415
Compare
4337415 to
80a40b7
Compare
80a40b7 to
7fddb1d
Compare
7fddb1d to
e43a331
Compare
e43a331 to
9d55233
Compare
2fd41d4 to
b6ea257
Compare
b6ea257 to
cda766e
Compare
Behnam (behnam-o)
left a comment
Behnam (behnam-o)
left a comment
There was a problem hiding this comment.
This PR should be scoped to the frontend live-progress experience; no changes under /pyrit are needed here. The progress and cancellation backend APIs come from earlier stack layers, ScenarioRunPlanGroupKind is only consumed by #2378, and the dataset-size override contract is first consumed by #2377. Moving those backend changes to their corresponding follow-up PRs would keep this layer focused and independently reviewable.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5ece2d4e-f111-4a4a-80ba-39a59edd1298
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5ece2d4e-f111-4a4a-80ba-39a59edd1298
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5ece2d4e-f111-4a4a-80ba-39a59edd1298
Drive the attack-technique display projection from the typed identifier markers instead of hand-maintained child-name sets, so a target in a newly declared slot cannot leak operational params such as endpoint. Build the FloatScaleThresholdScorer rationale conditionally rather than patching the dangling heading in the frontend. Group attempts by atomic group id and truncate per group, surface attempts that match no planned unit, share the retry-pressure definition between the CLI and GUI rollups, and split ScenarioRunPage into reusable pieces. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c1f04cc2-edd2-4f51-82a4-7e2fde405b4f
43fecf3 to
6b7e9ae
Compare
The CLI progress bar switch from technique counts to execution-unit counts does not depend on the dashboard work and is user-visible on its own, so it is reviewed separately. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c1f04cc2-edd2-4f51-82a4-7e2fde405b4f
…nario-pr-03-live-upstream # Conflicts: # pyrit/score/true_false/float_scale_threshold_scorer.py
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c1f04cc2-edd2-4f51-82a4-7e2fde405b4f
| ) | ||
| score = None | ||
| if row.score_id is not None: | ||
| scorer_identifier = ComponentIdentifier.model_validate(row.scorer_class_identifier) |
There was a problem hiding this comment.
[GHCP:] This new projection assumes scorer_class_identifier is populated, but Score allows it to be absent and ScoreEntry stores {} in that case. ComponentIdentifier.model_validate({}) then raises, so one valid score can make the whole progress endpoint return 500. Existing readers such as _load_identifier already treat an empty identifier as missing; please follow that pattern here and use Unknown for the display name.
| Returns: | ||
| ScenarioScorerIdentity: Scorer parameters and nested component identities. | ||
| """ | ||
| identity = ScenarioRunService._build_component_identity(component_identifier=scorer_identifier) |
There was a problem hiding this comment.
[GHCP:] This builds scorer details directly from the raw identifier. Unlike the technique-details path below, it skips project_behavioral_identity, so target fields explicitly marked Evaluate.Exclude()—including endpoint, model name, and rate limit—are returned and rendered by the frontend. Please apply the behavioral projection with ScorerIdentifier before mapping the scorer details.
| error_type=delta.error_type, | ||
| error_message=delta.error_message, | ||
| score=delta.score, | ||
| technique_details=( |
There was a problem hiding this comment.
[GHCP:] This attaches the full technique tree to every attempt even though all attempts in an atomic group share it. Those trees can include raw seed and system-prompt content, and the frontend downloads every page, so large runs retain many copies of identical data. Please return the details once per atomic group and resolve them through the existing atomic_group_id relationship.
|
|
||
| first_new_index = len(entry.deltas) | ||
| while True: | ||
| page, has_more = self._memory.get_scenario_attack_result_deltas( |
There was a problem hiding this comment.
[GHCP:] The cache only reads rows after its cursor, but these rows are not actually append-only: an attack result is persisted first, then AtomicAttack._enrich_atomic_attack_identifiers() updates that same row after the batch returns. A live poll can cache the pre-enrichment row and never see its technique details. Please either persist the complete identifier initially or invalidate/refresh cached rows after enrichment.
| self._dataset_config = configured_dataset | ||
| selected_groups = await self._resolve_seed_groups_by_dataset_async(apply_sampling=True) | ||
| self._dataset_config = configured_dataset | ||
| full_groups = await self._resolve_seed_groups_by_dataset_async(apply_sampling=False) |
There was a problem hiding this comment.
[GHCP:] Removing the shared read_only_dataset_resolution() guard means public estimation paths can now fetch and persist missing datasets. Before this change every estimate was protected, and ScenarioService still applies that same guard around one caller. This behavior change is unrelated to live progress; please restore the guard here and move any intentional side-effecting estimate behavior to the PR that consumes it.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c1f04cc2-edd2-4f51-82a4-7e2fde405b4f
Summary
Adds the live run dashboard with incremental polling, active/completed/pending attack groups, outcome accounting, elapsed time, ETA, cancellation, and result navigation. Progress cursors are isolated per run so route changes cannot reuse another run's opaque cursor.
What this layer adds
Preview
Stack
This is 3 of 7 in native GitHub Stack #2390 and depends on #2373.
See the full implementation, screenshots, walkthrough, validation, and operational notes.
Validation