Add YYLO Benchmark to 5a · Eval frameworks & harnesses - #92
Open
InsightFactoryAPP wants to merge 1 commit into
Open
InsightFactoryAPP wants to merge 1 commit into
InsightFactoryAPP wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Affiliation disclosure, up front: I maintain YYLO — this is a self-submission, disclosed per the CONTRIBUTING bar ("If you wrote or maintain the resource, say so in the PR body. Self-submissions are welcome… undisclosed ones are not").
What: one-line entry at the bottom of §5a, following the shape of the two most recent additions to that section (Caliper #34, Coder Eval #61). One change per PR; README only.
Why §5a: YYLO Benchmark is a code-first test-runner. It normalizes every case into one v2 attempt contract, runs each candidate in a private fresh-repository workspace (default candidate subprocesses sit behind a selective filesystem sandbox), and grades retained evidence with ordered deterministic and/or LLM-judge profiles. The angle none of the current §5a entries own is evidence integrity: workspace receipts, post-execution repository manifests, and evaluator provenance are hash-linked, and report/doctor/re-evaluation verify that complete chain before anything is published — "immutable evidence for agent runs" is the repo's own one-line description. Runs as
yylo-benchmark, oryy benchmarkdelegated from the YYLO orchestrator CLI (https://github.com/yylo-dev/yylo).Adoption evidence, honestly stated: the benchmark repo is 18 days old (created 2026-08-17) and currently shows 1★ — the star number alone fails the "evidence someone else uses them" test, and I'd rather say that than have you find it. The external-use evidence I can actually offer:
@yylo/benchmark: 151 downloads in the last month (api.npmjs.org/downloads/point/last-month/@YyLo%2Fbenchmark, checked at submission)yy benchmark(yylo-dev/yylo): ~54 external stars over ~8 months of daily development;@yylo/cli563 downloads last month (same API)If that still reads as "come back later," that's a fair call and I'll re-submit when the benchmark repo's own numbers are stronger. Per "every number must be quotable," the annotation itself carries no figures; every claim above traces to the linked README, the repo About line, or the npm downloads API.