fix(judge): add bounded correction retry for agent judge - #209
Conversation
- 严格协议失败后仅执行一次纠错并共享超时预算 - 保存两次原始响应并隔离重试引擎产物 - 聚合 Judge 会话指标并补齐单元测试与 E2E
There was a problem hiding this comment.
Review summary: 0 Blockers, 2 Major findings, and 1 Info item. I recommend addressing the Major findings before merging.
Coverage: reviewed all 10/10 changed files, including the correction contract, agent adapters and callers, evaluator/report metrics, artifact handling, tests, fixtures, documentation, E2E paths, and CI-only coverage. All current CI checks are green.
Info: please use an English commit subject. Replace fix(judge): 增加有界纠错重试 with fix(judge): add bounded correction retry.
- serialize fallback correction context into one user message - distinguish independent and resumed retry usage aggregation - cover CLI and custom prompt rendering regressions
|
Thanks for the thorough review. Both Major findings are addressed in 422bdaf, with focused adapter/aggregation regressions plus deterministic and full quick E2E coverage. The follow-up commit and PR title use English subjects. I kept the already-reviewed first commit intact rather than force-pushing rewritten history; the final PR title is |
zpzjzj
left a comment
There was a problem hiding this comment.
Review summary: 0 Blockers, 2 Major findings, and 2 Info items. I recommend addressing the Major findings before merging.
Coverage: reviewed all 11/11 changed files at 422bdaf, including the correction contract, all built-in and custom agent paths, metric aggregation, artifact collection and archival, evaluator/report consumers, unit tests, fixtures, documentation, deterministic E2E, credential-gated paths, and current CI. make test, make verify, and the targeted correction E2E pass; all GitHub checks are green. The local full E2E suite reached and passed the new correction test, but the pre-existing QoderCLI live test failed because the successful external run reported zero token usage.
Info:
- Commit
7c9e1b1still has a Chinese subject and body. Adding a later English commit and changing the PR title does not make the historical commit information English; please reword that commit. - The Summary still says "deterministic three-message fallback", while the implementation and Behavior section now correctly describe one serialized user message. Please update the stale Summary text.
The two Major findings from the previous review have been fixed in 422bdaf; the comments below are distinct issues found during the complete re-review.
- count fresh correction runs from turn one - snapshot runtime artifacts before retry overwrite - cover CustomAgent artifact isolation through evaluator
|
Both new Major findings are addressed in For the remaining historical commit Info item: rewording |
Summary
agent_judgedecode or semantic validation failuresoutputs/judge/run/retry/Relationship to #197
This builds on #197, which introduced the strict program-owned response contract and deterministic criterion mapping. This PR completes the maintainer-approved PR2 scope: one bounded correction retry and raw-output artifact preservation.
Closes #192
Behavior
agent.SessionResumer.RunTurnwhen a session ID is available; otherwise callRunwith one serialized user message containing the original prompt, first raw response, and correction instructionsArtifact layout
Test plan
go test -race ./internal/judge ./internal/evaluator ./internal/climake fmtmake verifygo test -tags e2e -timeout 1200s -count=1 ./e2ego run ./cmd/skill-up debug judge examples/judge-debug-agent.json --output /tmp/grading.jsoninitial → correction, returns PASS, and preserves both raw responses plus retry artifactsultimatetest skips without full-E2E prerequisites; the trusted Model E2E runner will execute it when QoderCLI and credentials are availableLocal Docker note
go test -race ./...passes for every package except the unchanged upstream testinternal/runtime.TestNoneRuntime_ExecStaleLockDoesNotHangNextCommand, which reproducibly reports that a background child survives timeout in this Docker environment.internal/runtimehas no diff fromupstream/main; all other packages pass together under-race.