Evaluate over-time degradation of LLM APIs
uv syncDocker Engine 28 or newer, because cells run in containers. Each cell is given
the key of the provider it talks to — CLAUDE_CODE_OAUTH_TOKEN,
CODEX_ACCESS_TOKEN, KIMI_MODEL_API_KEY or DEEPSEEK_API_KEY — from the
environment.
The bench runs itself. stupid bench starts
every Sunday and gives each of thirty-five cells — seven models on five seeds —
a twenty-four hour day to make one kernel faster, scored by an evaluator the
agent can reach but cannot change. It leaves one table in the job summary, the same
run at length in a report artifact, and every trajectory in one
cell-<flow>-<seed> artifact per cell. Run it early from the Actions tab,
where it also takes the flows to carry — a JSON array of the names
run --help lists — for a run that is to leave one out.
The same four commands carry one cell on any host:
uv run stupidbench prepare --flow opus5_max --seed 0
uv run stupidbench run --flow opus5_max --seed 0 --seconds 3600
uv run stupidbench redact
uv run stupidbench report --out reportuv run stupidbench --help says what each of them does, and run --help lists
every flow and seed a cell can be.
Open an issue for questions or defects, and a pull request for changes.
AGENTS.md states the requirements code has to meet; these must all pass:
uvx ruff format --check .
uvx ruff check .
uv run pytest -qAGPL-3.0-only © Zijian Zhang