feat: Eval results should include failure attribution and improvement suggestions

[JHWang-1997]

Problem / motivation

When an evaluation goes wrong, skill-up currently reports what happened but does not
help the skill author determine why it happened or what to change next:

• Failed criteria: per-criterion results only carry text, passed, and evidence.
  The evidence explains the verdict, but not the likely cause or remediation.
• Errored cases: cases that finish with ERROR may carry only a raw error string,
  and Grading can be nil when execution fails before judging. These are often the
  cases where dependency, environment, engine, and timeout causes matter most.

Authors therefore have to inspect transcripts, workspace diffs, generated files, skill
sources, and runtime errors manually to answer:

1. Was the skill not triggered, incomplete, or misleading?
2. Was the case ambiguous, unreasonable, or dependent on an undeclared environment?
3. Was the failure caused by an unavailable external dependency or by skill-up/runtime
   infrastructure?
4. Did the agent under-perform despite adequate guidance?
5. What concrete edit or operational action should be tried before re-running?

This slows down the core loop that skill-up is meant to accelerate:

evaluate → diagnose → improve → re-evaluate

Goals

• Add structured, machine-readable diagnoses to human and JSON reports.
• Diagnose agent_judge FAIL results at criterion granularity without a second model
  invocation.
• Optionally diagnose final ERROR results at case granularity with a separate,
  explicitly enabled diagnosis invocation.
• Ground diagnoses in the exact evaluated skill source and available execution evidence.
• Preserve the original verdict/error even when diagnosis is missing or fails.
• Make diagnostic cost, artifacts, and uncertainty visible.

Non-goals

• Treating an LLM-produced attribution as ground truth.
• Diagnosing deterministic FAIL paths (expect, rule_based, script, unjudged
  non-zero exit, or multi-turn post-condition failure) in the MVP.
• Diagnosing SKIP cases.
• Aggregating attribution statistics across benchmark runs in the MVP.
• Automatically editing the skill or case.

Shared diagnosis model

Use the same nested shape for criterion-level and case-level diagnoses:

{
  "diagnosis": {
    "failure_attribution": "skill_missing_info",
    "confidence": "high",
    "attribution_evidence": "The criterion requires a fixed output filename, but the evaluated SKILL.md and referenced guidance do not define one.",
    "improvement_suggestion": "Add an output-file naming convention to SKILL.md and reference it from the relevant workflow step."
  }
}

A nested object keeps verdict evidence separate from causal reasoning and leaves room for
future diagnostic fields without expanding every result object.

Field semantics

• failure_attribution: the most likely primary cause from the enum below.
• confidence: low, medium, or high. This describes confidence in the
  attribution, not confidence in the original pass/fail verdict.
• attribution_evidence: why the available evidence supports this attribution.
  This is distinct from the existing criterion evidence, which explains why the
  criterion passed or failed.
• improvement_suggestion: a concrete next action. It may target the skill, case,
  environment/configuration, infrastructure, or simply recommend a re-run.

Reports must label this block as a likely cause / AI-generated diagnosis, not as a
verified root cause.

failure_attribution vocabulary

Value	Meaning
skill_not_triggered	Evidence indicates that the agent never loaded or used the skill when it should have. Absence of telemetry alone is not sufficient.
skill_missing_info	The evaluated skill lacks instructions or knowledge needed to satisfy the criterion.
skill_misleading_info	The evaluated skill contains incorrect, conflicting, or counterproductive guidance that the agent followed.
case_design_issue	The prompt, criterion, fixture, or expected outcome is ambiguous, unreasonable, inconsistent, or untestable.
environment_configuration	The declared sandbox, network policy, credentials, tools, or case setup are incompatible with the task.
external_dependency_unavailable	A declared external service or dependency was unavailable or unhealthy.
infrastructure_error	skill-up, the engine adapter, sandbox provisioning, or another harness/runtime component failed unexpectedly.
agent_capability	The agent failed despite adequate skill guidance, a reasonable case, and a working environment.
undetermined	Available evidence is insufficient to choose a defensible category. This should be preferred over unsupported blame.
other	Evidence supports a cause outside the vocabulary; attribution_evidence must explain it.

A single primary cause is sufficient for the MVP. Ranked or multi-cause diagnoses can be
added later if real evaluations show that the single-cause model is too limiting.

Mechanism A — per-criterion diagnosis from agent_judge

For every failed agent_judge criterion, ask the existing judge invocation to return an
optional diagnosis object alongside criterion, passed, and evidence.

This adds no second model invocation, but it does add prompt/output tokens and must
not be described as zero-cost.

Example:

{
  "criterion": "writes the generated report to report.md",
  "passed": false,
  "evidence": "The final response references summary.txt and no report.md was generated.",
  "diagnosis": {
    "failure_attribution": "skill_missing_info",
    "confidence": "medium",
    "attribution_evidence": "The evaluated skill describes report contents but does not specify the required filename.",
    "improvement_suggestion": "State that the final artifact must be written to report.md."
  }
}

Validation and compatibility

• The verdict protocol remains strict: result count and non-empty verdict evidence
  retain their current validation.
• diagnosis is optional. Missing or malformed diagnosis data must never invalidate an
  otherwise valid grading result.
• Unknown attribution values normalize to other.
• Unknown confidence values are omitted or normalize to low.
• Diagnosis returned for a passed criterion is discarded.
• Incomplete diagnoses may be omitted from reports rather than causing the judge result
  to fail.
• The prompt should explicitly allow undetermined and prohibit inventing evidence.

Evidence prerequisites

Skill source

skill_missing_info and skill_misleading_info require visibility into the exact skill
version used by the evaluated run.

Add a skill_source material to judge context, defaulting to file_ref for the standard
profile. At minimum it must include the evaluated SKILL.md and a manifest/index of the
skill files available during the run. Referenced readable guidance should be made
available subject to the existing context limits; large/binary assets may remain indexed
rather than inlined.

The report-facing context manifest must record which source files were actually made
available so the attribution remains auditable.

Skill usage

Reading the source does not prove whether the evaluated agent loaded or used it.
skill_not_triggered therefore requires positive engine/session evidence when the engine
can expose it, such as skill load/read events. If reliable usage evidence is unavailable,
the judge should use undetermined rather than infer non-triggering from silence.

Benchmark behavior

without_skill is intentionally executed without the skill, so
skill_not_triggered is not a meaningful diagnosis for that configuration.

For MVP:

• Continue generating non-skill diagnoses for without_skill results where useful.
• Disallow skill_not_triggered, skill_missing_info, and
  skill_misleading_info for without_skill.
• Do not materialize skill_source into the baseline diagnosis context.
• Include the configuration (with_skill / without_skill) explicitly in the prompt.

Cross-validating diagnoses across benchmark arms remains future work.

Mechanism B — opt-in case diagnosis for final ERROR results

Add an eval-level configuration block:

diagnosis:
  enabled: true
  model: ""             # optional; effective agent_judge model, then engine model
  timeout_seconds: 60   # independent diagnosis budget

Defaults:

• enabled: false
• model resolution order:
  1. explicit diagnosis.model
  2. effective agent_judge model, when configured
  3. evaluated engine model
• timeout_seconds must be independent from the expired case-attempt timeout.

Case-level overrides are out of scope for the MVP.

When enabled and a case remains ERROR after retry policy is exhausted, run a
lightweight diagnosis over:

• the immutable original error and typed failure metadata when available
  (failure stage, timeout source, exit code, retry history);
• partial agent transcript/final message;
• preserved judge session/output if judging was attempted;
• downloaded workspace diff and selected artifacts;
• evaluated skill source and available skill-usage evidence;
• effective case, environment, engine, judge, and network configuration with secrets
  redacted.

Example case result:

{
  "status": "ERROR",
  "error": "agent execution failed: context deadline exceeded",
  "diagnosis": {
    "failure_attribution": "environment_configuration",
    "confidence": "high",
    "attribution_evidence": "The case requires registry.npmjs.org, while effective network_policy is deny_all and no mock is configured.",
    "improvement_suggestion": "Declare registry.npmjs.org in allowed_egress or provide a mocked dependency for the case."
  }
}

Lifecycle requirements

• Diagnose only after all configured retries are exhausted. Do not spend diagnosis
  tokens on a transient attempt that later succeeds.
• Use a fresh child context derived from the still-live evaluation parent, never the
  expired case-attempt context.
• Use a separate, bounded diagnosis workspace/runtime and host-side preserved artifacts;
  diagnosis must not assume the failed case runtime is still alive or was ever created.
• Store materials under diagnosis/context and agent artifacts under diagnosis/run so
  an agent-judge failure diagnosis cannot overwrite judge/context or judge/run.
• Diagnosis failure is non-fatal: log it, preserve diagnostic artifacts when possible,
  and return the original ERROR unchanged.
• Diagnosis must never rewrite, replace, or weaken the original error.
• If the parent evaluation context is already cancelled, skip diagnosis cleanly.

Typed evaluator/runtime signals should be included as evidence and prompt constraints.
The LLM may add causal interpretation and remediation, but it must not contradict known
facts such as the failure stage or timeout source.

Result and cost accounting

Add Diagnosis *FailureDiagnosis to both:

• evaluator EvalResult
• report CaseResult

The conversion in internal/runner/runner.go must copy it explicitly; it does not flow
into result.json automatically.

Preserve the diagnosis agent session separately (for example
DiagnosisSession *agent.SessionResult) and expose:

• per-case diagnosis duration;
• diagnosis input/output tokens;
• diagnosis artifacts/context;
• a top-level diagnosis token total;
• overall_tokens including tested-agent, judge, and diagnosis tokens.

A failed diagnosis invocation should still contribute its observable duration/token
usage, as failed judge invocations do today.

Reporting

result.json is the source of truth.

• report.html and offline report.md render:
  • failed criterion verdict evidence;
  • likely cause, confidence, attribution evidence, and improvement suggestion;
  • case-level diagnosis for ERROR cases;
  • a clear “AI-generated diagnosis” label.
• HTML embedded report structs must carry the diagnosis fields explicitly.
• JUnit behavior is unchanged in the MVP.
• Keep Anthropic-compatible grading.json strictly Anthropic-shaped in the MVP.
  Diagnostic extensions live in result.json and derived skill-up reports until
  compatibility with strict downstream consumers is verified.

Implementation touch points

Config

• internal/config/schema.go: eval-level DiagnosisConfig.
• internal/config/defaults.yaml: disabled default.
• internal/config/validator.go: model/timeout validation.
• Config examples and writing-evals documentation.

Judge and diagnosis

• internal/judge/judge.go: shared FailureDiagnosis type and optional diagnosis on
  AssertionResult.
• internal/judge/agent_judge.go: response schema, prompt, normalization, and result
  mapping.
• internal/judge/context_materializer.go: skill_source and any supported skill-usage
  material.
• A small diagnosis component/package may reuse JSON extraction/normalization helpers
  without weakening the existing verdict validation.

Evaluator and runner

• internal/evaluator/evaluator.go and multiturn.go: preserve diagnostic inputs.
• The outer retry finalization path: launch ERROR diagnosis only after retry exhaustion.
• internal/runner/runner.go: map diagnosis/session metrics into report.CaseResult
  and aggregate token accounting.

Reports

• internal/report/reporter.go: case diagnosis and diagnosis metrics.
• internal/report/html.go plus templates/report.html: embedded mapping/rendering.
• internal/report/markdown.go: offline Markdown rendering.
• grading.go remains unchanged for MVP interoperability.

Compatibility

• New result fields use omitempty.
• With diagnosis.enabled: false, ERROR execution behavior and output remain unchanged.
• Older/weaker judge models may omit criterion diagnoses without invalidating grading.
• Deterministic judges and deterministic FAIL paths remain byte-identical.
• Existing grading.json consumers remain unaffected.
• Report readers must tolerate older result.json files without diagnosis fields.

Alternatives considered

• Put attribution inside verdict evidence/error strings: no schema change, but not
  independently renderable, auditable, or aggregatable.
• Run a second diagnosis call for every FAIL: simpler judge schema, but duplicates
  context work and adds an invocation for cases where agent_judge already has the
  evidence.
• Use deterministic classification only for ERROR: reliable for some typed failures,
  but too coarse for causal explanation and cannot generate remediation. The proposed
  approach uses deterministic signals as constraints and lets the diagnosis agent add
  interpretation.
• Case-level diagnosis only: loses the fact that different criteria in one case can
  fail for different reasons.
• Force a category when evidence is weak: produces confident-looking blame.
  undetermined and confidence are preferable.

Delivery plan

This can be implemented as two independently reviewable changes:

1. Shared diagnosis schema, skill-source/usage evidence, agent_judge per-criterion
   diagnosis, and report rendering.
2. Opt-in final-ERROR diagnosis lifecycle, runtime/context isolation, artifacts, and
   cost accounting.

Future work

• Attribution aggregation across cases and iterations.
• Cross-validation using with_skill / without_skill benchmark pairs.
• Multiple ranked contributing causes.
• Deterministic FAIL diagnosis behind the same opt-in configuration.
• Suggested patch generation or human-approved automatic edits.
• Calibration/evaluation of attribution accuracy.

Acceptance criteria

• Failed agent_judge criteria may carry an optional nested diagnosis containing
  attribution, confidence, attribution evidence, and an actionable suggestion in
  result.json.
• Existing verdict validation remains strict; missing/malformed diagnosis never
  invalidates a valid grading result.
• The exact evaluated skill source is available and recorded in judge context when
  needed; skill_not_triggered is not emitted without positive usage evidence.
• Baseline without_skill results never report skill-specific attribution categories.
• With diagnosis.enabled: true, a case that is still ERROR after retries receives
  a case-level diagnosis when the diagnosis invocation succeeds.
• ERROR diagnosis uses a fresh timeout/context and does not depend on the failed case
  runtime remaining available.
• Diagnosis failure never masks or changes the original ERROR.
• Diagnosis artifacts and token/duration metrics are preserved and included in
  aggregate cost reporting.
• result.json, report.html, and offline report.md render the new data;
  grading.json remains Anthropic-compatible and unchanged.
• With diagnosis disabled, ERROR behavior/output remains unchanged.
• Deterministic FAIL and SKIP paths remain unchanged.