Proposal: EvalPort adapter for GradingResult / trace output (interchange format, no core changes)

[adhabnr-ux]

Summary

PAST-Bench's grading output (src/past_bench/models/trace.py::GradingResult, and the paired sequence_results.json / sequence_comparison.json it feeds) is a genuinely well-structured, multi-dimensional eval result format. It maps cleanly onto EvalPort, an open (Apache-2.0) interchange standard for portable LLM eval test cases, graders, suites, and results. I'd like to propose (and, if there's interest, contribute) a small past-bench-openeval-adapter package that converts PAST-Bench traces to/from EvalPort's ResultSet/TestCase JSON, the same way 37 other framework adapters already do for Ragas, MLflow, LangSmith, etc.

Why this maps well

Your TraceEvent discriminated union (TraceStart, ToolDispatch, RuntimeRequest, ..., GradingResult, TraceEnd) already separates "what happened during the episode" from "how it was scored" — which is exactly the TestCase/ResultSet split EvalPort's schema makes:

• GradingResult.task_id → EvalPort TestCase.id / ResultSet.results[].test_case_id
• GradingResult.scores (a DimensionScores: completion, robustness, communication, safety, efficiency_turns, efficiency_tokens, efficiency_wall_time_s) → one EvalPort grader_results[] entry per dimension (grader_id, type, score, passed), the same "one upstream metric → one EvalPort grader" pattern the Ragas and MLflow adapters use
• GradingResult.passed / task_score / failure_modes → ResultSet.results[].passed / metadata
• GradingResult.trace_id and the TraceStart/TraceEnd timing fields (model_time_s, tool_time_s, wall_time_s, total_tokens) → preserved under results[].metadata, exactly as the MLflow adapter keeps mlflow_scores/mlflow_metrics and Ragas keeps ragas_scores — nothing gets thrown away converting in

Concrete sketch (real fields only)

# past_bench_openeval_adapter/__init__.py
from past_bench.models.trace import GradingResult, DimensionScores

def _dimension_grader_results(scores: DimensionScores) -> list[dict]:
    return [
        {"grader_id": "gr_completion", "type": "custom",
         "score": scores.completion, "passed": scores.completion >= 0.5},
        {"grader_id": "gr_robustness", "type": "custom",
         "score": scores.robustness, "passed": scores.robustness >= 0.5},
        {"grader_id": "gr_communication", "type": "custom",
         "score": scores.communication, "passed": scores.communication >= 0.5},
        {"grader_id": "gr_safety", "type": "custom",
         "score": scores.safety, "passed": scores.safety == 1.0},
    ]

def to_openeval(results: list[GradingResult], run_id: str, suite_id: str = "past-bench") -> dict:
    return {
        "version": "1.0.0",
        "suite_id": suite_id,
        "run_id": run_id,
        "started_at": min(r.timestamp for r in results),
        "results": [
            {
                "test_case_id": r.task_id,
                "grader_results": _dimension_grader_results(r.scores),
                "passed": r.passed,
                "metadata": {
                    "trace_id": r.trace_id,
                    "past_bench_task_score": r.task_score,
                    "past_bench_failure_modes": r.failure_modes,
                    "past_bench_efficiency": {
                        "turns": r.scores.efficiency_turns,
                        "tokens": r.scores.efficiency_tokens,
                        "wall_time_s": r.scores.efficiency_wall_time_s,
                    },
                },
            }
            for r in results
        ],
    }

to_openeval() output would validate against EvalPort's schema/resultset.json via openeval.validate.validate_suite. Feeding it PAST-Bench's sequence_results.json records (which already carry task_id, scores, task_score, passed, failure_modes in this shape) would be a direct, low-loss conversion — no reinterpretation needed, same as how the existing adapters treat already-scored upstream output as "scored data, not just a task definition."

Prior art (already merged, same playbook)

• ragas-openeval-adapter — converts Ragas's per-metric EvaluationResult into one EvalPort grader per metric, preserving raw scores under metadata
• mlflow-openeval-adapter — same pattern for mlflow.evaluate() output, preserving both per-row and run-level metrics

A past-bench-openeval-adapter would follow the identical structure: standalone package, works against PAST-Bench's public GradingResult/sequence_results.json shape from the outside, no changes needed to PAST-Bench itself, and it becomes optional/redundant if native EvalPort support is ever wanted in-tree.

Ask

Would this be a welcome contribution (as its own adapter package, PR'd into evalport's adapters/ — not a dependency added to PAST-Bench), or is there a reason the trace/grading format is meant to stay PAST-Bench-specific? Happy to build and submit the full adapter (with round-trip tests against a real sequence_results.json) if there's interest — just didn't want to put up a PR against this repo without checking first, since it touches no PAST-Bench code, only reads its public result shape.

EvalPort spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

— Sahi, independent contributor (not affiliated with Gen-Verse)