Skip to content

FEAT: Add Dart, Perl, and Raku package hallucination techniques - #2537

Merged
hannahwestra25 merged 2 commits into
microsoft:mainfrom
YusefSyed:feat/garak-package-ecosystems
Sep 2, 2026
Merged

FEAT: Add Dart, Perl, and Raku package hallucination techniques#2537
hannahwestra25 merged 2 commits into
microsoft:mainfrom
YusefSyed:feat/garak-package-ecosystems

Conversation

@YusefSyed

Copy link
Copy Markdown
Contributor

Description

Closes #2531.

This extends the existing garak.package_hallucination scenario with Dart, Perl, and Raku while reusing PyRIT's registered Garak package datasets and PackageHallucinationScorer support.

  • add Dart, Perl, and Raku technique/spec wiring to the existing scenario
  • bump the scenario version from 2 to 3 for the expanded technique contract
  • keep Rust as the only default technique and the only package registry preloaded by the default dataset configuration
  • retain on-demand registry fetching for every non-default technique
  • extend scenario tests for all seven techniques, each scorer ecosystem, all six non-default lazy-fetch paths, and the unchanged Rust default
  • update the synchronized Garak scanner .py and .ipynb documentation

No dataset loader, scorer, or default-loading behavior is added or duplicated.

AI assistance disclosure: AI tools assisted with repository search, implementation drafting, and a separate fresh-context review. I directed the work, reviewed the final four-file diff, verified the existing dataset/scorer contracts and upstream Garak language labels, removed unrelated generated lockfile churn, and ran the checks below.

Tests and Documentation

  • uv run --frozen pytest tests/unit/scenario/garak/test_package_hallucination.py -q — 31 passed
  • uv run --frozen pytest tests/unit/scenario/garak/test_package_hallucination.py tests/unit/score/test_package_hallucination_scorer.py tests/unit/datasets/test_garak_package_hallucination_dataset.py tests/unit/docs/test_scenario_documentation.py -q — 79 passed
  • uv run --frozen -m pytest -n 4 --dist=loadfile tests/unit — 17,067 passed, 10 skipped
  • uv run --frozen pre-commit run --files pyrit/scenario/scenarios/garak/package_hallucination.py tests/unit/scenario/garak/test_package_hallucination.py doc/scanner/garak.py doc/scanner/garak.ipynb — all hooks passed, including Ruff, ty, documentation validation, notebook checks, and async-suffix validation
  • Jupytext logical-content comparison between doc/scanner/garak.py and doc/scanner/garak.ipynb — synchronized; conversion differed only in the locally installed Jupytext version metadata (1.19.4 vs 1.19.5)

The full unit run reported 17 existing dependency/deprecation warnings. No live model target or external registry download was used; the focused tests mock registry fetching and exercise the provider-free scenario wiring.

@hannahwestra25 hannahwestra25 self-assigned this Sep 2, 2026
@YusefSyed

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing!

@hannahwestra25
hannahwestra25 added this pull request to the merge queue Sep 2, 2026
Merged via the queue into microsoft:main with commit 1d1b815 Sep 2, 2026
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT: Complete Garak package hallucination ecosystem coverage

2 participants