Skip to content

Унификация отчётов: per-date папки results/<date>/ + двуязычная авто-… #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gistrec merged 1 commit into from
Jul 3, 2026
Merged

Унификация отчётов: per-date папки results/<date>/ + двуязычная авто-… #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ node_modules/
.pytest_cache/
.DS_Store

# Сырые артефакты прогонов и авто-сгенерированный отчёт (курируемые .ru/.en.md — под VCS).
# Сырые per-run данные прогонов (отчёты results.ru/en.md в дата-папках — под VCS).
results/**/runs.jsonl
results/runs-*.jsonl
results/model-comparison-grid.generated.md
results/model-comparison-grid.partial.md
23 changes: 12 additions & 11 deletions README.en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ llmbench/
report.py # record aggregation, Stability/Score/Pareto, markdown build (CI-tested)
runner.py # variant grid x cases x repeat, JSONL persistence, report
tests/ # offline self-test: scoring + aggregation + runner pipeline (all in CI)
results/ # curated reports (.ru/.en.md) + generated report and runs-*.jsonl (gitignored)
results/ # per-date folders: results.ru.md + results.en.md (in VCS) + raw runs.jsonl (gitignored)
```

## Running
Expand Down Expand Up @@ -78,15 +78,16 @@ line (don't forget the rate in `core.MODEL_RATES`, or the runner warns).

## Artifacts and re-scoring

Every run writes **`results/runs-<ts>.jsonl`** — one record per run (answer, tool trace,
usage, all scores, errors). This is the source of truth: the report is rebuilt from it for
free, with no repeat model calls —
Every run lands in a dated folder **`results/<date>/`**: the raw `runs.jsonl` (one record per
run — answer, tool trace, usage, all scores, errors) and the bilingual generated report
`results.ru.md` + `results.en.md`. `runs.jsonl` is the source of truth: both report versions
are rebuilt from it for free, with no repeat model calls —
```bash
python -m llmbench.runner --report-from results/runs-20260703-120000.jsonl
python -m llmbench.runner --report-from results/2026-07-03/runs.jsonl
```
The runner writes the generated report to `results/model-comparison-grid.generated.md`
(gitignored) so it never clobbers the hand-curated `model-comparison-grid.ru.md` / `…en.md`
(top-3, prose, bilingual) — those are edited by hand from the generated one.
`runs.jsonl` is gitignored (raw data), the `.md` reports are in VCS. Top-3, prose and takeaways
are added by hand on top of the generated grid in the same dated folder. A second run on the
same day doesn't clobber the first (a time suffix is added on a date collision).

## How Score is computed

Expand All @@ -100,9 +101,9 @@ differing difficulty).

## Latest run results

`results/model-comparison-grid.ru.md` (+ English `…en.md`) — curated summary + Pareto
frontier. ⚠️ Their current numbers are from the 2026-06-29 run, BEFORE the scoring/fixture
fixes (see `REVIEW.md`); regenerate with a fresh run.
The latest run is in the newest dated folder `results/<date>/` (`results.ru.md` + English
`results.en.md`) — curated summary + Pareto frontier. ⚠️ The numbers in `results/2026-06-29/`
predate the scoring/fixture fixes (see `REVIEW.md`); regenerate with a fresh run.

## Known limitations

Expand Down
23 changes: 12 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ llmbench/
report.py # агрегация записей, Stability/Score/Pareto, сборка markdown (тестируется в CI)
runner.py # сетка вариантов × кейсы × repeat, JSONL-персист, отчёт
tests/ # офлайн self-test: скоринг + агрегация + конвейер ранера (всё в CI)
results/ # курируемые отчёты (.ru/.en.md) + авто-отчёт и runs-*.jsonl (в .gitignore)
results/ # per-date папки: results.ru.md + results.en.md (под VCS) + сырой runs.jsonl (.gitignore)
```

## Запуск
Expand Down Expand Up @@ -77,15 +77,16 @@ Env токенов кабинета: `YANDEX_DIRECT_TOKEN` (+ опц. `YANDEX_DI

## Артефакты и пере-скоринг

Каждый прогон пишет **`results/runs-<ts>.jsonl`** — по записи на прогон (ответ, tool-трейс,
usage, все оценки, ошибки). Это источник правды: отчёт пересобирается из него бесплатно,
без повторных вызовов моделей —
Каждый прогон складывается в дата-папку **`results/<date>/`**: сырой `runs.jsonl` (по записи
на прогон — ответ, tool-трейс, usage, все оценки, ошибки) и двуязычный авто-отчёт
`results.ru.md` + `results.en.md`. `runs.jsonl` — источник правды: отчёт (обе версии)
пересобирается из него бесплатно, без повторных вызовов моделей —
```bash
python -m llmbench.runner --report-from results/runs-20260703-120000.jsonl
python -m llmbench.runner --report-from results/2026-07-03/runs.jsonl
```
Ранер пишет авто-отчёт в `results/model-comparison-grid.generated.md` (в `.gitignore`),
чтобы не затирать курируемые вручную `model-comparison-grid.ru.md` / `…en.md` (Топ-3, проза,
двуязычие) — их правят руками из сгенерированного.
`runs.jsonl` — в `.gitignore` (сырые данные), сами отчёты `.md` — под VCS. Топ-3, прозу и
выводы дописывают руками поверх сгенерированного грида в той же дата-папке. Второй прогон за
день не затирает первый (при коллизии по дате к папке добавляется время).

## Как считается Score

Expand All @@ -98,9 +99,9 @@ python -m llmbench.runner --report-from results/runs-20260703-120000.jsonl

## Результаты последнего прогона

`results/model-comparison-grid.ru.md` (+ англ. `…en.md`) — курируемая сводка + Pareto-фронт.
⚠️ Текущие числа в них — от прогона 2026-06-29, ДО правок скоринга/фикстур (см. `REVIEW.md`);
их нужно перегенерировать свежим прогоном.
Последний прогон — в самой свежей дата-папке `results/<date>/` (`results.ru.md` + англ.
`results.en.md`), курируемая сводка + Pareto-фронт. ⚠️ Числа в `results/2026-06-29/` — ДО
правок скоринга/фикстур (см. `REVIEW.md`); их нужно перегенерировать свежим прогоном.

## Известные ограничения

Expand Down
149 changes: 118 additions & 31 deletions llmbench/report.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,19 +83,20 @@ def _f(x):
}


def describe(v):
"""Вариант → (LLM, Thinking, Effort). Thinking: adaptive/reasoning/нет; у GLM effort не рычаг (—)."""
def describe(v, lang="ru"):
"""Вариант → (LLM, Thinking, Effort). Thinking: adaptive/reasoning/нет(en: no); у GLM effort не рычаг (—)."""
llm = MODEL_DISPLAY.get(v["model"], v["model"])
no = "no" if lang == "en" else "нет"
if v.get("reasoning_effort"):
return llm, "reasoning", v["reasoning_effort"]
if v["engine"] == "openai":
return llm, "нет", "—"
thinking = "adaptive" if v.get("thinking") == "adaptive" else "нет"
return llm, no, "—"
thinking = "adaptive" if v.get("thinking") == "adaptive" else no
effort = "—" if v["vendor"] == "zai" else (v.get("effort") or "—")
return llm, thinking, effort


GLOSSARY = """## Термины (как читать таблицу)
GLOSSARY_RU = """## Термины (как читать таблицу)

- **Accuracy** (0–5) — точность чисел: верно ли посчитаны CTR/CPC/CPA/расход, не выдуманы ли
цифры и той ли кампании они приписаны (entity-анкоринг). **В коде** (детерминированно).
Expand Down Expand Up @@ -123,43 +124,129 @@ def describe(v):
"""


def build_md(aggregates, meta):
o = ["# Сравнение моделей для AskAds (Claude / GLM / GPT)\n"]
GLOSSARY_EN = """## Terms (how to read the table)

- **Accuracy** (0–5) — numeric correctness: are CTR/CPC/CPA/spend computed right, nothing made
up, and are the numbers attributed to the right campaign (entity anchoring). **In code**
(deterministic).
- **Tools Use** (0–5) — tool correctness: called the right tools (successfully) in the right
order, nothing extra/forbidden. **Code**.
- **Edge Cases** (0–5) — behavior in edge cases (empty report, refusing to change a bid,
clarifying). **LLM judges** — they also score runs with tool violations.
- **Lang quality** (0–5) — naturalness and clarity of the Russian. Judges.
- **Score** (0–5) — a run's overall score = mean of the available components: Tools Use
(always), Accuracy (if the case has golden facts), Edge Cases/Lang quality (if judges ran).
The component set depends on the case, so Score is comparable across variants (everyone runs
the same cases) but is NOT equal to the mean of the four left columns. Failed runs are
excluded from Score — see Err.
- **Cost per Answer** — mean cost of a successful run (USD); **Score per USD (s/m)** — "quality
per dollar" (Score ÷ cost) for single-/multi-step dialogs; higher = better value.
- **Stability** (0–5) — `5 − mean spread (σ) of Score between repeats of the same case`:
higher = more stable. Meaningful at repeat ≥ 2.
- **Err** — `failed/all runs` (API errors, token-limit truncation); suffix `·NR` — N runs
succeeded only after a retry with the same config. Failed runs are excluded from all metrics,
but their cost is included in the total run cost.
- **Thinking** — whether the model thinks before answering: `adaptive` (Claude/GLM),
`reasoning` (GPT-5), `no`.
- **Effort** — the "effort" budget per answer (`low/medium/high/max`); separate from thinking
(weak effect when thinking is off). Not configurable for GLM (`—`).
- **⭐** — **best quality/price balance**: a variant that can't be beaten — no other is both
better and cheaper. _(In optimization — the "Pareto frontier".)_
"""


# Локализация проз-строк отчёта. Колонки таблицы (Accuracy/Tools Use/…) — англ. в обеих
# версиях, поэтому переводим только заголовки/пояснения/оговорки; числа считает agg().
_LANG = {
"ru": {
"title": "# Сравнение моделей для AskAds (Claude / GLM / GPT)\n",
"cross": "🇷🇺 Русский · [🇬🇧 English](results.en.md)\n",
"runline": ("_Запуск от {ts} × **{nv} вариантов** (модель × thinking/effort) × "
"**{nc} тест-кейсов** × **{rep} повтора** = {total} запусков · режим {mode} · "
"вход одинаковый для всех (фикстуры версии `{fx}`){commit}._\n"),
"commit": " · код `{c}`",
"how": ("**Как считалось.** Claude/GLM — наш агентный движок; GPT — отдельный OpenAI-цикл "
"(askads на Anthropic, GPT в тот же движок не встроить) → его tool-use сопоставим не "
"на 100%. **Tools Use/Accuracy** считает код; **Edge Cases/Lang quality** — LLM-судьи "
"({judges}; нейтрален: **{neutral}**). Судьи вторичны — вес на ключевых метриках.\n"),
"glossary": GLOSSARY_RU,
"variants_h": "## Все варианты (сорт. по Score)\n",
"star": ("\n⭐ — **лучший баланс «качество/цена»** (нельзя стать и качественнее, и дешевле "
"одновременно): **{front}**.\n"),
"baseline": "_Для ориентира: текущий прод askads — {desc}._\n",
"limits_h": "\n## Известные ограничения\n",
"jsonl": ("\n_Сырые per-run данные: `{jsonl}` — отчёт пересобирается из них командой "
"`python -m llmbench.runner --report-from <файл>`._"),
"none": "—",
},
"en": {
"title": "# Model comparison for AskAds (Claude / GLM / GPT)\n",
"cross": "[🇷🇺 Русский](results.ru.md) · 🇬🇧 English\n",
"runline": ("_Run from {ts} × **{nv} variants** (model × thinking/effort) × "
"**{nc} test cases** × **{rep} repeats** = {total} runs · mode {mode} · "
"identical input for all (fixtures version `{fx}`){commit}._\n"),
"commit": " · code `{c}`",
"how": ("**How it was measured.** Claude/GLM — our agentic engine; GPT — a separate OpenAI "
"loop (askads is on Anthropic, GPT can't be plugged into the same engine) → its "
"tool-use isn't 100% comparable. **Tools Use/Accuracy** are computed in code; **Edge "
"Cases/Lang quality** — LLM judges ({judges}; neutral: **{neutral}**). Judges are "
"secondary — weight is on the key metrics.\n"),
"glossary": GLOSSARY_EN,
"variants_h": "## All variants (sorted by Score)\n",
"star": ("\n⭐ — **best quality/price balance** (can't become both better and cheaper at "
"once): **{front}**.\n"),
"baseline": "_For reference: current askads production — {desc}._\n",
"limits_h": "\n## Known limitations\n",
"jsonl": ("\n_Raw per-run data: `{jsonl}` — the report is rebuilt from it with "
"`python -m llmbench.runner --report-from <file>`._"),
"none": "—",
},
}

_TABLE_HEADER = ("| LLM | Thinking | Effort | Accuracy | Tools<br>Use | Edge<br>Cases | "
"Lang<br>quality | Cost<br>per Answer | Score<br>per USD (s) | "
"Score<br>per USD (m) | Stability | Err | Score |")
_TABLE_SEP = "|---|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|"


def build_md(aggregates, meta, lang="ru"):
t = _LANG.get(lang, _LANG["ru"])
total = sum(a["n_runs"] for a in aggregates.values())
judges = ', '.join(meta['judges']) if isinstance(meta['judges'], list) else meta['judges']
commit = f" · код `{meta['git_commit']}`" if meta.get("git_commit") else ""
o.append(f"_Запуск от {meta['ts']} × **{len(meta['variants'])} вариантов** "
f"(модель × thinking/effort) × **{meta['n_cases']} тест-кейсов** × **{meta['repeat']} повтора** "
f"= {total} запусков · режим {meta['mode']} · вход одинаковый для всех "
f"(фикстуры версии `{meta['fixture_version']}`){commit}._\n")
o.append("**Как считалось.** Claude/GLM — наш агентный движок; GPT — отдельный OpenAI-цикл "
"(askads на Anthropic, GPT в тот же движок не встроить) → его tool-use сопоставим не на 100%. "
f"**Tools Use/Accuracy** считает код; **Edge Cases/Lang quality** — LLM-судьи ({judges}; "
f"нейтрален: **{meta['neutral'] or '—'}**). Судьи вторичны — вес на ключевых метриках.\n")
o.append(GLOSSARY)
o.append("## Все варианты (сорт. по Score)\n")
o.append("| LLM | Thinking | Effort | Accuracy | Tools<br>Use | Edge<br>Cases | Lang<br>quality | "
"Cost<br>per Answer | Score<br>per USD (s) | Score<br>per USD (m) | Stability | Err | Score |")
o.append("|---|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|")
nv = meta.get('neutral')
neutral = (', '.join(nv) if isinstance(nv, list) else (str(nv) if nv else "")) or t["none"]
commit = t["commit"].format(c=meta['git_commit']) if meta.get("git_commit") else ""
o = [t["title"], t["cross"]]
o.append(t["runline"].format(ts=meta['ts'], nv=len(meta['variants']), nc=meta['n_cases'],
rep=meta['repeat'], total=total, mode=meta['mode'],
fx=meta['fixture_version'], commit=commit))
o.append(t["how"].format(judges=judges, neutral=neutral))
o.append(t["glossary"])
o.append(t["variants_h"])
o.append(_TABLE_HEADER)
o.append(_TABLE_SEP)
front = set(pareto(aggregates))
by_label = {v["label"]: v for v in meta["variants"]}
for label, a in sorted(aggregates.items(), key=lambda kv: (kv[1]["composite"] is None, -(kv[1]["composite"] or 0))):
llm, thinking, effort = describe(by_label[label])
llm, thinking, effort = describe(by_label[label], lang)
spd = a["score_per_dollar"]
cost = "—" if a["cost_avg"] is None else f"${a['cost_avg']:.5f}"
cost = t["none"] if a["cost_avg"] is None else f"${a['cost_avg']:.5f}"
err = f"{a['errors']}/{a['n_runs']}" + (f" ·{a['retried']}R" if a.get("retried") else "")
stability = _f(round(5 - a["stddev_composite"], 3) if a["stddev_composite"] is not None else None)
o.append(f"| {llm}{' ⭐' if label in front else ''} | {thinking} | {effort} | {_f(a['numeric'])} | "
f"{_f(a['tool'])} | {_f(a['edge'])} | {_f(a['russian'])} | {cost} | {_f(spd['single'])} | "
f"{_f(spd['multi'])} | {stability} | {err} | {_f(a['composite'])} |")
o.append(f"\n⭐ — **лучший баланс «качество/цена»** (нельзя стать и качественнее, и дешевле одновременно): "
f"**{', '.join(front) or '—'}**.\n")
if meta.get("baseline_desc"):
o.append(f"_Для ориентира: текущий прод askads — {meta['baseline_desc']}._\n")
o.append("\n## Известные ограничения\n")
for line in meta["caveats"]:
o.append(t["star"].format(front=', '.join(front) or t["none"]))
baseline = next((v for v in meta["variants"] if v.get("is_baseline")), None)
if baseline:
bl_llm, bl_th, bl_ef = describe(baseline, lang)
o.append(t["baseline"].format(desc=f"{bl_llm} (thinking {bl_th}, effort {bl_ef})"))
o.append(t["limits_h"])
caveats = meta["caveats"]
if isinstance(caveats, dict):
caveats = caveats.get(lang) or caveats.get("ru") or []
for line in caveats:
o.append(f"- {line}")
if meta.get("jsonl"):
o.append(f"\n_Сырые per-run данные: `{meta['jsonl']}` — отчёт пересобирается из них "
f"командой `python -m llmbench.runner --report-from <файл>`._")
o.append(t["jsonl"].format(jsonl=meta['jsonl']))
return "\n".join(o)
Loading
Loading