Skip to content

fix(judge): add bounded correction retry for agent judge - #209

Merged
zpzjzj merged 3 commits into
alibaba:mainfrom
DancingCircles:fix/agent-judge-correction-retry
Aug 19, 2026
Merged

fix(judge): add bounded correction retry for agent judge#209
zpzjzj merged 3 commits into
alibaba:mainfrom
DancingCircles:fix/agent-judge-correction-retry

Conversation

@DancingCircles

@DancingCircles DancingCircles commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add exactly one validation-guided correction retry for strict agent_judge decode or semantic validation failures
  • resume the Judge session when supported, with one serialized first-turn fallback message otherwise
  • preserve every raw Judge response and isolate retry engine artifacts under outputs/judge/run/retry/
  • aggregate both attempts into the final Judge session without changing report schemas
  • add Judge, Evaluator, deterministic E2E, and credential-gated QoderCLI coverage

Relationship to #197

This builds on #197, which introduced the strict program-owned response contract and deterministic criterion mapping. This PR completes the maintainer-approved PR2 scope: one bounded correction retry and raw-output artifact preservation.

Closes #192

Behavior

  • retry only after strict JSON decoding or semantic contract validation fails
  • do not retry a valid FAIL judgment, explicit cancellation, context materialization failure, or an agent error without recoverable output
  • share the existing Judge/case timeout budget across both attempts; no timeout reset or backoff
  • use agent.SessionResumer.RunTurn when a session ID is available; otherwise call Run with one serialized user message containing the original prompt, first raw response, and correction instructions
  • keep the last attempt authoritative while aggregating duration, token/turn usage, and stable-deduplicated artifacts
  • keep native structured output, configurable retry policies, and report schema/UI changes out of scope

Artifact layout

outputs/judge/run/
├── stdout.json (or other first-attempt engine artifacts)
├── raw-response-attempt-1.txt
├── raw-response-attempt-2.txt       # only when correction runs
└── retry/
    └── stdout.json (or other retry engine artifacts)

Test plan

  • go test -race ./internal/judge ./internal/evaluator ./internal/cli
  • make fmt
  • make verify
  • go test -tags e2e -timeout 1200s -count=1 ./e2e
  • go run ./cmd/skill-up debug judge examples/judge-debug-agent.json --output /tmp/grading.json
  • verified the deterministic mock performs exactly initial → correction, returns PASS, and preserves both raw responses plus retry artifacts
  • verified the QoderCLI ultimate test skips without full-E2E prerequisites; the trusted Model E2E runner will execute it when QoderCLI and credentials are available

Local Docker note

go test -race ./... passes for every package except the unchanged upstream test internal/runtime.TestNoneRuntime_ExecStaleLockDoesNotHangNextCommand, which reproducibly reports that a background child survives timeout in this Docker environment. internal/runtime has no diff from upstream/main; all other packages pass together under -race.

- 严格协议失败后仅执行一次纠错并共享超时预算
- 保存两次原始响应并隔离重试引擎产物
- 聚合 Judge 会话指标并补齐单元测试与 E2E

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary: 0 Blockers, 2 Major findings, and 1 Info item. I recommend addressing the Major findings before merging.

Coverage: reviewed all 10/10 changed files, including the correction contract, agent adapters and callers, evaluator/report metrics, artifact handling, tests, fixtures, documentation, E2E paths, and CI-only coverage. All current CI checks are green.

Info: please use an English commit subject. Replace fix(judge): 增加有界纠错重试 with fix(judge): add bounded correction retry.

Comment thread internal/judge/agent_judge.go Outdated
Comment thread internal/judge/agent_judge.go
- serialize fallback correction context into one user message
- distinguish independent and resumed retry usage aggregation
- cover CLI and custom prompt rendering regressions
@DancingCircles

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. Both Major findings are addressed in 422bdaf, with focused adapter/aggregation regressions plus deterministic and full quick E2E coverage. The follow-up commit and PR title use English subjects. I kept the already-reviewed first commit intact rather than force-pushing rewritten history; the final PR title is fix(judge): add bounded correction retry for agent judge.

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary: 0 Blockers, 2 Major findings, and 2 Info items. I recommend addressing the Major findings before merging.

Coverage: reviewed all 11/11 changed files at 422bdaf, including the correction contract, all built-in and custom agent paths, metric aggregation, artifact collection and archival, evaluator/report consumers, unit tests, fixtures, documentation, deterministic E2E, credential-gated paths, and current CI. make test, make verify, and the targeted correction E2E pass; all GitHub checks are green. The local full E2E suite reached and passed the new correction test, but the pre-existing QoderCLI live test failed because the successful external run reported zero token usage.

Info:

  • Commit 7c9e1b1 still has a Chinese subject and body. Adding a later English commit and changing the PR title does not make the historical commit information English; please reword that commit.
  • The Summary still says "deterministic three-message fallback", while the implementation and Behavior section now correctly describe one serialized user message. Please update the stale Summary text.

The two Major findings from the previous review have been fixed in 422bdaf; the comments below are distinct issues found during the complete re-review.

Comment thread internal/judge/agent_judge.go Outdated
Comment thread internal/judge/agent_judge.go
- count fresh correction runs from turn one
- snapshot runtime artifacts before retry overwrite
- cover CustomAgent artifact isolation through evaluator
@DancingCircles

Copy link
Copy Markdown
Contributor Author

Both new Major findings are addressed in aed0cdd, with a real evaluator-level CustomAgent regression covering fresh-turn accounting and separate framework input/output snapshots. The stale Summary wording has also been updated. Local verification: focused race tests pass, make verify reports 0 issues, all packages except the documented unchanged Docker-only internal/runtime case pass under race, and the full quick E2E suite passes.

For the remaining historical commit Info item: rewording 7c9e1b1 requires a non-fast-forward force push over the already-reviewed branch. I have not rewritten that remote history. If English-only individual commit history is mandatory, would you prefer a replacement clean PR branch; otherwise an English squash-merge commit using the current PR title preserves an English final history?

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zpzjzj
zpzjzj added this pull request to the merge queue Aug 19, 2026
@zpzjzj
zpzjzj removed this pull request from the merge queue due to a manual request Aug 19, 2026
@zpzjzj
zpzjzj added this pull request to the merge queue Aug 19, 2026
Merged via the queue into alibaba:main with commit 6da1264 Aug 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(judge): enforce a strict program-owned output contract for agent_judge

2 participants