Mark-It-Bench

Markdown-native benchmark framework for the agent era.

Use Markdown instead of JSON as your benchmark dataset format. Agent frameworks are the runner — no external harness, no adapters, no framework-specific protocols.

---

Why Markdown instead of JSON?

Traditional benchmarks use JSON datasets + external test harnesses. This creates friction in the agent era:

	Traditional (JSON)	Mark-It-Bench (Markdown)
Agent reads data	Needs a parser / adapter	LLM natively understands Markdown
Per-framework setup	Write an adapter per framework	Zero — agent reads .md files directly
Test runner	External harness drives evaluation	Agent framework itself is the runner
Evaluation	String matching / BLEU / exact match	LLM-as-judge with rubric-based scoring
Anti-leak	Runtime stripping	Physical file separation (scenarios vs answers)

Mark-It-Bench is a reusable framework for building Markdown-native benchmarks. It provides:

• Data format: YAML-frontmatter Markdown files as test cases, physically separated into scenarios/ (agent-visible) and answers/ (judge-only)
• Manifest generation: Auto-scan frontmatter to build a machine-readable episode index
• LLM-as-judge: Rubric-based scoring via any OpenAI-compatible endpoint (with optional structured decoding)
• Trajectory auditing: Verify that agents followed the prescribed execution protocol
• Result aggregation: Summarize verdicts by metric and category into JSON + Markdown reports

---

Core Concepts

1. Episodes = Paired Markdown Files

Each benchmark episode is a pair of same-stem files:

dataset/
├── scenarios/<category>/<stem>.md   ← Agent can read (blind test input)
└── answers/<category>/<stem>.md     ← Judge only (reference answer + rubric)

Anti-leak by design: answers and rubrics are physically separated from scenarios. The agent under test is only pointed at scenarios/ — it cannot accidentally read answer keys.

2. Scenario Format

---
id: episode-001
name: "Short description"
category: my-category
mode: my-mode
metric: MY_METRIC
domain: "Business Domain"
tags: [tag1, tag2]
---

# Context

[Setup and instructions for the agent]

# Input

[The actual test input / interaction history]

# Task

[What the agent should do]

# Questions

1. [Probe 1]
2. [Probe 2]

3. Answer Format

# Reference Answer

[Expected behavior and ground truth]

# Rubric

1. [!] Critical criterion (must pass for episode to pass)
2. [ ] Regular criterion
3. [!] Another critical criterion

• [!] marks critical items — all must pass for the episode to pass (verdict_pass)
• [ ] marks regular items — contribute to score but don't gate pass/fail

4. Self-Executing Protocol

Any agent framework can run the benchmark by reading TASK.md:

Discovery → Blind Test → Audit → Judge → Summarize

1. Discovery: Read dataset/manifest.json to enumerate episodes
2. Blind Test: For each episode, read only dataset/scenarios/<cat>/<stem>.md and produce an answer
3. Audit: Validate execution trajectory against protocol requirements
4. Judge: Independent model scores answers against rubrics from dataset/answers/
5. Summarize: Aggregate verdicts into summary.json + summary.md

---

Quick Start

Building a benchmark with Mark-It-Bench

1. Define your categories (subdirectories under dataset/scenarios/ and dataset/answers/)
2. Write episode pairs: scenario .md + answer .md with matching stems
3. Run python scripts/build_manifest.py to generate dataset/manifest.json
4. Have agents run the blind test (read scenarios, produce answers)
5. Score with python scripts/judge.py --framework <name>
6. Summarize with python scripts/summarize.py --framework <name>

Running a benchmark

Point any agent at this repo:

Read TASK.md and run the benchmark on yourself. Only read dataset/scenarios/ during the blind test — never read dataset/answers/.

Judge setup

The judge uses any OpenAI-compatible /v1/chat/completions endpoint:

export JUDGE_BASE_URL=http://localhost:8000/v1
export JUDGE_MODEL=your-judge-model
export JUDGE_API_KEY=your-key
python scripts/judge.py --framework <framework>
python scripts/summarize.py --framework <framework>

---

Repository Structure

Mark-It-Bench/
├── TASK.md                    Self-execution protocol (authoritative)
├── AGENTS.md                  Agent framework entry point
├── CLAUDE.md                  Claude Code compatibility shim
├── README.md                  This file
├── docs/
│   └── PROTOCOL.md            Multi-phase execution protocol + metrics
├── dataset/
│   ├── manifest.json          Episode index (generated)
│   ├── scenarios/<category>/  Test inputs (agent-visible)
│   └── answers/<category>/    Reference answers + rubrics (judge-only)
├── scripts/
│   ├── build_manifest.py      Scan frontmatter → manifest.json
│   ├── judge.py               LLM-as-judge scoring
│   ├── summarize.py           Aggregate verdicts → summary
│   ├── audit_trajectory.py    Validate execution trajectories
│   └── start_vllm.sh          Optional: start local vLLM judge endpoint
└── results/<framework>/       Evaluation outputs per framework
    ├── <category>/<stem>.answer.md
    ├── <category>/<stem>.verdict.json
    └── summary.{json,md}

---

Extending Mark-It-Bench

Mark-It-Bench is designed to be forked and customized for specific benchmark domains:

• Add categories: Create new subdirectories under scenarios/ and answers/
• Add metrics: Define new frontmatter metric: values; they flow through to summaries automatically
• Custom execution protocols: Override TASK.md and docs/PROTOCOL.md for domain-specific multi-phase flows
• Custom judge prompts: Modify SYSTEM_PROMPT in judge.py for domain-specific evaluation criteria
• Custom trajectory auditing: Extend audit_trajectory.py with domain-specific checks

See dataset/scenarios/_TEMPLATE.md and dataset/answers/_TEMPLATE.md for episode templates.

---

License

MIT