Skip to content

feat(judge): add output_matches rule - #173

Merged
zpzjzj merged 2 commits into
alibaba:mainfrom
googs1025:feat/rule-based-output-matches
Aug 11, 2026
Merged

feat(judge): add output_matches rule#173
zpzjzj merged 2 commits into
alibaba:mainfrom
googs1025:feat/rule-based-output-matches

Conversation

@googs1025

Copy link
Copy Markdown
Contributor

Summary

Add output_matches support to the deterministic rule_based judge so eval authors can assert final output with Go regular expressions without changing output_contains literal
matching semantics.

Related issues

Closes #165

Changes

  • Add output_matches rule schema with all, any, and not groups.
  • Validate configured regex patterns during config validation.
  • Evaluate output_matches against the final message in RuleBasedJudge.
  • Preserve existing failure-priority and success-all-pass semantics.
  • Add unit coverage for regex matching, empty groups, failure rules, and invalid regex validation.
  • Update user-facing and Skill author docs.

Test plan

  • make test passes
  • make verify passes (fmt + vet + lint)
  • Manual testing steps (if applicable): N/A

Notes for reviewers

output_contains remains literal substring matching. Tool-set aliases and output section checks from the original issue are intentionally deferred.

@googs1025
googs1025 requested a review from hittyt as a code owner August 8, 2026 02:46
@zpzjzj
zpzjzj requested review from roark47 and zpzjzj August 10, 2026 01:47
Comment thread internal/judge/rule_based.go Outdated

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review summary

  • 🟠 Major: 1
  • 🔴 Blocker: 0

The implementation scope and the normal all / any / not paths look consistent with #165. One reproducible edge-case bug remains in output_matches.not: an empty Go regex is mistaken for the no-match sentinel. I left the details inline.

Overall: conditionally ready after the Major finding is fixed.


Manual source review and focused reproduction @ 13edd02ea043d0379d2cbc4e2d35d7be9b18536a; 1 finding accepted, 0 downgraded, 0 rejected. The configured local automated reviewer was unavailable.

@googs1025
googs1025 force-pushed the feat/rule-based-output-matches branch from 763524b to 2153234 Compare August 11, 2026 03:01

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zpzjzj
zpzjzj added this pull request to the merge queue Aug 11, 2026
Merged via the queue into alibaba:main with commit 5ba8c3a Aug 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(judge): add stronger deterministic rule_based assertions

2 participants