Skip to content

Latest commit

 

History

History

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Offline eval

Python reimplementation of the selector and scorer prompts. No network, no API keys.

Run

python3 harness.py run

Tracks

Track Input Output
Distilled selector score_examples in src/route-core/model-cards.ts Pick distribution, nominal cost vs always-premium
Selector boundaries Synthetic score vectors Regression on 0.7 threshold logic
Fixture replay fixtures/*.jsonl Scorer system + user prompts

Commands

python3 extract_distilled.py
python3 harness.py run --limit 8 --sessions 2
python3 harness.py replay-session fixtures/example_refactor.jsonl

Limits

  • No live scorer LLM
  • No agent task execution in Docker
  • Gold scores come from hand-written model cards, not production telemetry