Context. A recurring weakness in agentic-workflow evaluation (and the named limitation of arXiv:2605.22502) is the closed loop: training data, user simulator, and primary judge are all the same frontier model family. Fathom's evaluate() returns a deterministic decision + rule_trace/module_trace + Ed25519 attestation — a hard-grounded oracle. Wrapping it as a judge gives Task-Success / Consistency a non-self-preferential, signable ground truth instead of an LLM grading its own family.
Proposal. New src/fathom/integrations/judge.py.
- Adapter takes a model transcript + a per-domain fact extractor (registered Python fn via
Engine.register_function, engine.py:720) that lifts turn state into facts.
- Runs
evaluate() at each decision point; compares the model's actual route/terminal to Fathom's deterministic decision: Task-Success = reached the procedurally-correct terminal; Consistency = no turn routed against the rule trace.
- Emits a signed, hash-chained scorecard (
attestation.py + chained_log.py). Ship a worked example judging the ssvc / owasp_agentic packs.
Where.
- Reuse the in-process ASGI harness of
studio/scenarios.py; consume EvaluationResult.rule_trace/module_trace (evaluator.py:70-77, models.py:341).
Acceptance criteria.
Bounding caveat. Grades procedure-following only — cannot judge Naturalness or Information Accuracy (world-knowledge), so it augments, not replaces, an LLM judge. Fact-extraction from free-form dialogue is the weak link and may itself need an LLM — that boundary must be explicit so the determinism claim isn't oversold.
Source: arXiv:2605.22502v1 — "Compiling Agentic Workflows into LLM Weights" (subterranean agents). Distilled from arXiv-research/2605.22502v1/analysis.md; file refs verified against current main by the analysis pass.
Context. A recurring weakness in agentic-workflow evaluation (and the named limitation of arXiv:2605.22502) is the closed loop: training data, user simulator, and primary judge are all the same frontier model family. Fathom's
evaluate()returns a deterministic decision +rule_trace/module_trace+ Ed25519 attestation — a hard-grounded oracle. Wrapping it as a judge gives Task-Success / Consistency a non-self-preferential, signable ground truth instead of an LLM grading its own family.Proposal. New
src/fathom/integrations/judge.py.Engine.register_function,engine.py:720) that lifts turn state into facts.evaluate()at each decision point; compares the model's actual route/terminal to Fathom's deterministic decision: Task-Success = reached the procedurally-correct terminal; Consistency = no turn routed against the rule trace.attestation.py+chained_log.py). Ship a worked example judging thessvc/owasp_agenticpacks.Where.
studio/scenarios.py; consumeEvaluationResult.rule_trace/module_trace(evaluator.py:70-77,models.py:341).Acceptance criteria.
Bounding caveat. Grades procedure-following only — cannot judge Naturalness or Information Accuracy (world-knowledge), so it augments, not replaces, an LLM judge. Fact-extraction from free-form dialogue is the weak link and may itself need an LLM — that boundary must be explicit so the determinism claim isn't oversold.
Source: arXiv:2605.22502v1 — "Compiling Agentic Workflows into LLM Weights" (subterranean agents). Distilled from
arXiv-research/2605.22502v1/analysis.md; file refs verified against currentmainby the analysis pass.