Skip to content

perf(benchmarks): expand level 1 benchmark corpora and task suites - #78

Merged
knight22-21 merged 1 commit into
mainfrom
perf/benchmark-corpora-expansion
Nov 2, 2026
Merged

knight22-21 merged 1 commit into
mainfrom
perf/benchmark-corpora-expansion

Conversation

@knight22-21

Copy link
Copy Markdown
Owner

Summary

Expands benchmark repository definitions and task suites to cover 11 corpora across 220 tasks:

  • Registers 7 new repositories in \�enchmarks/setup_repos.py\ (\ astapi, \pydantic, \express,
    eact, \ ypescript, \gin, \cobra).
  • Adds dedicated task definition files for each new corpus (20 tasks each) and expands existing task suites (\ ixture,
    equests, \ lask, \httpx) to 20 tasks each.
  • Updates \�enchmarks/RESULTS.md\ with Run 11 and Run 12 benchmark records across all 11 corpora.
  • Updates \docs/benchmark-results.md\ summary table with the latest 220-task level 1 token reduction and accuracy results.

Testing

  • Ran \python -m benchmarks.run_token_benchmark\ (20 fixture tasks completed successfully with +10.3% token reduction).
  • Validated JSON syntax across all task definition files in \�enchmarks/tasks/.
  • Verified
    uff check\ and
    uff format --check.

@knight22-21 knight22-21 added the enhancement New feature or request label Nov 2, 2026
@knight22-21
knight22-21 merged commit 74f9df0 into main Nov 2, 2026
4 checks passed
@knight22-21
knight22-21 deleted the perf/benchmark-corpora-expansion branch November 2, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant