Add YYLO Benchmark to AI & LLM Testing - #239
Open
InsightFactoryAPP wants to merge 1 commit into
Open
InsightFactoryAPP wants to merge 1 commit into
InsightFactoryAPP wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi maintainers! Thanks for maintaining this valuable list.
I'd like to propose adding YYLO Benchmark to the AI & LLM Testing section.
What is YYLO Benchmark?
YYLO Benchmark is a flexible, isolated evaluation layer for task prompts and project-owned Workflow Runner YAML. It normalizes every case into one attempt contract, runs each candidate in a private fresh-repository workspace (dedicated repo plus private home/temp/cache roots behind a selective filesystem sandbox), and evaluates the retained evidence with ordered deterministic and/or LLM judge profiles. Results chain receipts, manifests, and hash-verified terminals into immutable records, so regrade/rejudge consume retained evidence instead of re-running candidates.
yylo-benchmarkCLI, also delegated byyy benchmarkfrom the YYLO CLI)Why this section?
The section's focus is exactly this noun: testing frameworks for AI systems (promptfoo, Tenro, nika, crilio). YYLO Benchmark occupies the agent-run side — the agent's executed attempt is the software under test, evaluated deterministically and by LLM judges with retained, tamper-evident evidence. It complements prompt-output regression tools like crilio and workflow snapshot engines like nika.
Contribution notes
### AI & LLM Testingsection, one-line format matching the existing entries, description ends with a full stop, no trailing whitespace.Disclosure
I'm part of the team that builds YYLO. This PR was prepared with an AI coding agent and reviewed by me before submission.