Skip to content

Add YYLO Benchmark to 5a · Eval frameworks & harnesses - #92

Open
InsightFactoryAPP wants to merge 1 commit into
benchflow-ai:mainfrom
InsightFactoryAPP:add-yylo-benchmark
Open

InsightFactoryAPP wants to merge 1 commit into
benchflow-ai:mainfrom
InsightFactoryAPP:add-yylo-benchmark

Conversation

@InsightFactoryAPP

Copy link
Copy Markdown

Affiliation disclosure, up front: I maintain YYLO — this is a self-submission, disclosed per the CONTRIBUTING bar ("If you wrote or maintain the resource, say so in the PR body. Self-submissions are welcome… undisclosed ones are not").

What: one-line entry at the bottom of §5a, following the shape of the two most recent additions to that section (Caliper #34, Coder Eval #61). One change per PR; README only.

Why §5a: YYLO Benchmark is a code-first test-runner. It normalizes every case into one v2 attempt contract, runs each candidate in a private fresh-repository workspace (default candidate subprocesses sit behind a selective filesystem sandbox), and grades retained evidence with ordered deterministic and/or LLM-judge profiles. The angle none of the current §5a entries own is evidence integrity: workspace receipts, post-execution repository manifests, and evaluator provenance are hash-linked, and report/doctor/re-evaluation verify that complete chain before anything is published — "immutable evidence for agent runs" is the repo's own one-line description. Runs as yylo-benchmark, or yy benchmark delegated from the YYLO orchestrator CLI (https://github.com/yylo-dev/yylo).

Adoption evidence, honestly stated: the benchmark repo is 18 days old (created 2026-08-17) and currently shows 1★ — the star number alone fails the "evidence someone else uses them" test, and I'd rather say that than have you find it. The external-use evidence I can actually offer:

  • npm @yylo/benchmark: 151 downloads in the last month (api.npmjs.org/downloads/point/last-month/@YyLo%2Fbenchmark, checked at submission)
  • The parent orchestrator CLI that delegates yy benchmark (yylo-dev/yylo): ~54 external stars over ~8 months of daily development; @yylo/cli 563 downloads last month (same API)
  • Both repos MIT-licensed, pushed daily, with test suites

If that still reads as "come back later," that's a fair call and I'll re-submit when the benchmark repo's own numbers are stronger. Per "every number must be quotable," the annotation itself carries no figures; every claim above traces to the linked README, the repo About line, or the npm downloads API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant