fix(judge): enforce a strict program-owned output contract for agent_judge

[zpzjzj]

Summary

agent_judge currently relies on the judge agent to return free-form text containing a JSON object:

{
  "results": [
    {
      "criterion": "<criterion text>",
      "passed": true,
      "evidence": "..."
    }
  ]
}

The prompt describes this shape clearly, but the program does not enforce it as a strict output contract. In dogfooding with complex Skills and rubrics, judge output can be malformed or structurally valid but semantically misaligned, making evaluation results unstable.

The final grading.json, result.json, pass rate, status, and HTML report are already generated by skill-up. This issue is specifically about hardening the intermediate contract between the judge agent and skill-up.

Background and validation priority

This issue was reported while using skill-up to evaluate complex data-analysis Skills, where the judge must follow detailed rubrics and return a stable machine-readable result.

The first validation target should be the current dogfooding combination:

• Judge Agent engine: Qoder (qodercli)
• Judge model: Qoder's 极致模型 tier/model

Implementation and regression verification should prioritize this combination before expanding the compatibility matrix to other Agent Engines and models. The exact resolved engine/model should be recorded in test evidence so model-specific behavior is not confused with framework-level contract enforcement.

The contract itself must remain engine-independent: native structured output may be used when supported, while strict text-JSON validation and bounded correction retry remain the compatibility path.

Current behavior

The current implementation in internal/judge/agent_judge.go:

• extracts JSON from plain text, Markdown fences, or embedded objects;
• attempts to repair unescaped quotes;
• unmarshals with the permissive default encoding/json behavior;
• validates only that:
  • the number of returned results equals the number of configured criteria;
  • every result has non-empty evidence.

This leaves several gaps:

1. Unknown fields such as name, score, threshold, or description are silently ignored.
2. A missing passed field is indistinguishable from an explicit passed: false because the Go zero value is false.
3. A missing, rewritten, duplicated, reordered, or mismatched criterion is accepted as long as the result count and evidence pass validation.
4. The report uses the agent-returned criterion text instead of the configured criterion as the source of truth.
5. Evidence is only checked for non-emptiness; any stronger evidence contract remains prompt-only.
6. Invalid output immediately becomes a judge ERROR; there is no bounded correction retry.

Proposed direction

Make the judge result contract program-owned rather than prompt-owned.

1. Use stable criterion identifiers

Assign stable IDs or indices before invoking the judge and require the judge to return those identifiers instead of copying criterion text:

{
  "results": [
    {
      "criterion_id": "criterion-1",
      "passed": false,
      "evidence": ["concrete observation"],
      "failures": ["original failure text"]
    }
  ]
}

The configured criteria remain authoritative. Report labels must be populated from configuration, not model-returned text.

2. Enforce strict decoding and validation

• reject unknown fields;
• detect missing required fields, including passed;
• reject unknown, duplicate, missing, or repeated criterion IDs;
• validate result cardinality and mapping;
• reject trailing or unrelated JSON;
• keep malformed contract output as ERROR, never silently convert it into FAIL.

An implementation may use json.Decoder.DisallowUnknownFields(), pointer/custom required-field types, and explicit semantic validation. If duplicate JSON keys must also be rejected, add a small strict decoder check rather than relying on the default last-value-wins behavior.

3. Keep aggregation and rendering deterministic

Skill-up should continue to own:

• configured criterion text;
• totals and pass counts;
• pass-rate and threshold evaluation;
• normalized evidence formatting;
• grading.json, result.json, JUnit, Markdown, and HTML serialization.

The agent should only provide semantic decisions and evidence/failure data.

4. Add one bounded correction retry

When validation fails, optionally retry the same judge once with concrete validation errors, for example:

criterion-2 is missing passed; unknown field score is not allowed. Return only the required schema.

If the corrected response is still invalid, return ERROR and preserve both raw judge outputs in judge artifacts.

5. Support engine capabilities incrementally

• Prefer native structured output / tool-call schema when an Agent Engine supports it.
• Keep strict text-JSON decoding plus one retry as the compatibility path for engines without structured output.
• Do not add another summarization agent to the authoritative grading path.

Acceptance criteria

• Configured criteria are the authoritative source for report labels.
• Judge results map to criteria through stable IDs or indices, not copied criterion prose.
• Missing passed is rejected rather than interpreted as false.
• Unknown fields are rejected.
• Unknown, duplicate, missing, and repeated criterion IDs are rejected.
• Evidence/failure fields have an explicit validated schema.
• Totals, pass rate, overall status, and report files remain program-generated.
• Invalid judge output is classified as ERROR and retains raw artifacts.
• At most one validation-guided correction retry is supported.
• Tests cover malformed JSON, missing fields, unknown fields, criterion mismatch, duplicate IDs, partial results, retry success, and retry exhaustion.
• Existing rule_based and script judge behavior is unchanged.

Non-goals

• Replacing deterministic judges with agent_judge.
• Adding a second summarization agent to generate authoritative results.
• Redesigning the final report formats or HTML UI.

[DancingCircles]

Hi, I'd like to work on this issue if it is still available.

I have experience with Go backend development and AI agent applications. I reviewed internal/judge/agent_judge.go and the existing agent_judge tests. The current CriterionResult/judgeResponse flow, permissive json.Unmarshal, and report mapping align with the gaps described here.

I would start with a focused contract-hardening change:

1. assign stable criterion IDs while keeping configured criteria authoritative;
2. use strict decoding and explicit required-field validation, including distinguishing a missing passed from false;
3. reject unknown fields and unknown, duplicate, missing, or repeated criterion IDs;
4. build report labels from configured criteria rather than model-returned prose;
5. add focused tests for malformed and semantically invalid responses.

The validation-guided correction retry and engine-native structured-output capability handling make the full issue relatively broad. Would you prefer this as one PR, or split into:

• PR 1: strict contract, deterministic mapping, and regression tests;
• PR 2: one bounded correction retry and raw-output artifact preservation/capability integration?

If the issue is available, I'm happy to prepare the focused implementation and follow the maintainers' preferred split.

[zpzjzj]

Thanks for looking into this — the issue is available, and we would be happy to welcome a PR contribution.

Splitting the work sounds good:

• PR 1: strict contract, deterministic criterion mapping, and regression tests.
• PR 2: one bounded correction retry and raw-output artifact preservation.

Please also add a credential-gated QoderCLI E2E test to the repository CI, skipping it when the required credentials are unavailable.

Feel free to start with PR 1. Thanks for taking this on!