From d751167608358edac91bc4222b304d23e261ba3a Mon Sep 17 00:00:00 2001 From: Dana Burks Date: Sun, 6 Sep 2026 22:51:38 -0700 Subject: [PATCH 1/2] [bench] release: version 2.1.0; date the changelog; refresh the README cost figures The release commit for v2.1.0. pyproject moves from 2.1.0.dev0 to 2.1.0 and the changelog's 2.1.0 entry is dated 2026-09-06, matching git's local-date convention the other entries follow. The README's What It Costs section is refreshed from the operational chain as of 2026-09-07 UTC. It now quotes both measured weeks: the v2.1 roadmap week (1,797 governed edits, median 34,747 tokens, 33,451 at cached rates, 63.5 seconds) and the release week so far (80 edits, median 19,483 tokens, 18,124 cached, 22.0 seconds), with the caveat stated in the section: the release-week row is a small sample of light edits, mostly docstring changes the Challenger clears in seconds and the Defender then skips, it is not a settled reduction, the roadmap-week row is the number to plan on, and the target of a median under 20,000 tokens stays a target. The wall-time table now covers all 1,635 timed entries rather than the first 78. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0122bFBUvdtXT245QFoYDkMf --- CHANGELOG.md | 2 +- README.md | 27 ++++++++++++++++----------- pyproject.toml | 2 +- 3 files changed, 18 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a0aa7a..e87729a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ tags. Since the runner began recording verdicts, partway through 1.0.0, every change has landed through Bench's own governance pipeline and carries a receipt in the operational ledger; the earliest 1.0.0 work predates that. -## [2.1.0] - unreleased +## [2.1.0] - 2026-09-06 The theme is making the claim survive a skeptic: cheaper verdicts, a real installer, honest numbers, and tests for the properties that used to live in diff --git a/README.md b/README.md index 2bc2be3..6976a4f 100644 --- a/README.md +++ b/README.md @@ -302,25 +302,30 @@ Set `BENCH_PROVIDER=claude_code` to run the pipeline on the subscription that al Every governed `Write`, `Edit`, or `MultiEdit` is three sequential model calls, and Claude Code makes many small edits. The figures below come from Bench's own operational ledger. `python -m cli stats` prints each of them (the "Tokens per edit", "Seconds per edit", and "Seconds by stage" lines), and the viewer's dashboard shows the same distributions, so anyone with a chain can reproduce the table for their own ledger rather than take an estimate. -Tokens, over 1,148 entries that recorded usage, as of 2026-09-04: +Tokens per governed edit, all stages, by week of the operational chain, as of 2026-09-07: -| Figure | Value | -|---|---| -| Median tokens per governed edit, all stages | 36,166 | -| 90th percentile | 47,590 | +| Week | Governed edits | Median tokens | 90th percentile | Median at cached rates | +|---|---|---|---|---| +| 2026-W36, the v2.1 roadmap week | 1,797 | 34,747 | 42,650 | 33,451 | +| 2026-W37, the release week so far | 80 | 19,483 | 36,811 | 18,124 | +| Every entry with usage recorded (2,705) | | 34,967 | 44,486 | 33,979 | + +The release-week row is the first under every 2.1 cost change together, and it is a small sample of light edits: most were docstring changes, which the Challenger clears in seconds and the Defender then skips. It is not a settled reduction. The roadmap-week row is the number to plan on until the release week fills out, and the roadmap's target of a median under 20,000 tokens stays a target. -Wall time, recorded per stage from v2.1 onward. The first 78 timed entries, all on the `claude_code` provider, which cold-starts a `claude` process per stage: +Wall time, recorded per stage, on the `claude_code` provider, which cold-starts a `claude` process per stage. Over the 1,635 timed entries: | Stage | Median seconds | 90th percentile | |---|---|---| -| Challenger | 17.3 | 29.5 | -| Defender | 10.2 | 30.8 | -| Oracle | 21.9 | 29.9 | -| Whole edit | 47.0 | 92.9 | +| Challenger | 21.8 | 38.7 | +| Defender | 15.8 | 38.6 | +| Oracle | 23.1 | 34.9 | +| Whole edit | 61.9 | 107.7 | + +By week, the whole-edit median was 63.5 seconds in 2026-W36 and 22.0 seconds in 2026-W37, with the same caveat on the second figure. The Defender median is low because a CLEAR challenge skips it and records zero seconds. The direct API providers avoid the process start but not the sequential three-call shape. Both tables will drift as the ledger grows; run `python -m cli stats` for the current figures on your own chain, and `python -m cli viewer` for the per-week view. -Roadmap v2.1 targets a median under 20,000 tokens, and two of its steps are in. Each stage receives every constraint's rule without its rationale and commentary (`pipeline.constitution.prompt_view`), and constitution v7 moved procedure and history out of the rules into commentary. On the `anthropic` provider the constitution and the repository context open every stage prompt under one cache breakpoint: they are written to the prompt cache on the first edit and read back at a tenth of the input price on every edit that follows within five minutes. Each stage's `_tokens` record now carries `cache_read` and `cache_creation` beside `input` and `output`, and `cli stats` and the viewer print a second per-edit figure with those counted at the cached rates (reads 0.1x, writes 1.25x), so the cost of a cached edit is read off the ledger rather than estimated. On the `claude_code` provider the CLI caches its system prompt file and not stdin text, so the constitution is folded into that file; the repository context is repository input and stays on stdin at user priority, uncached, because a project's CLAUDE.md must not outrank the stage prompt or the constitution. Measured on the development machine, a Challenger call under that layout reads about 2,470 of 9,340 prompt tokens from cache. The figures above predate both changes; the ledger will show the difference. +Roadmap v2.1 targets a median under 20,000 tokens, and two of its steps are in. Each stage receives every constraint's rule without its rationale and commentary (`pipeline.constitution.prompt_view`), and constitution v7 moved procedure and history out of the rules into commentary. On the `anthropic` provider the constitution and the repository context open every stage prompt under one cache breakpoint: they are written to the prompt cache on the first edit and read back at a tenth of the input price on every edit that follows within five minutes. Each stage's `_tokens` record now carries `cache_read` and `cache_creation` beside `input` and `output`, and `cli stats` and the viewer print a second per-edit figure with those counted at the cached rates (reads 0.1x, writes 1.25x), so the cost of a cached edit is read off the ledger rather than estimated. On the `claude_code` provider the CLI caches its system prompt file and not stdin text, so the constitution is folded into that file; the repository context is repository input and stays on stdin at user priority, uncached, because a project's CLAUDE.md must not outrank the stage prompt or the constitution. Measured on the development machine, a Challenger call under that layout reads about 2,470 of 9,340 prompt tokens from cache. The roadmap-week row above straddles both changes, which landed partway through it; the release-week row is the first entirely under them, and the ledger will show whether its figure holds as it fills with ordinary edits. ## Design Decisions diff --git a/pyproject.toml b/pyproject.toml index b02c07f..bfba44b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "bench-governance" -version = "2.1.0.dev0" +version = "2.1.0" description = "Constitutional governance layer for Claude Code: every edit is challenged, defended, ruled on, and hash-chained into a ledger." readme = "README.md" license = "MIT" From ef4b3ad6709dbceac44a575c5c8848770a2533cb Mon Sep 17 00:00:00 2001 From: Dana Burks Date: Sun, 6 Sep 2026 22:56:59 -0700 Subject: [PATCH 2/2] [bench] docs: say how the weekly token figures are reproduced Codex review on #65: the README claimed the per-week token rows could be reproduced through cli stats and the viewer, and neither exposes tokens by week; cli stats prints the all-entries rows and the dashboard plots verdicts and latency by week. The section now says which interface shows which rows and gives the reproduction for the weekly ones: the same utils.stats helpers over one week's entries, in a snippet that runs as written and reproduces the roadmap-week row exactly. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0122bFBUvdtXT245QFoYDkMf --- README.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 6976a4f..865899e 100644 --- a/README.md +++ b/README.md @@ -300,7 +300,19 @@ Set `BENCH_PROVIDER=claude_code` to run the pipeline on the subscription that al ## What It Costs -Every governed `Write`, `Edit`, or `MultiEdit` is three sequential model calls, and Claude Code makes many small edits. The figures below come from Bench's own operational ledger. `python -m cli stats` prints each of them (the "Tokens per edit", "Seconds per edit", and "Seconds by stage" lines), and the viewer's dashboard shows the same distributions, so anyone with a chain can reproduce the table for their own ledger rather than take an estimate. +Every governed `Write`, `Edit`, or `MultiEdit` is three sequential model calls, and Claude Code makes many small edits. The figures below come from Bench's own operational ledger, through the helpers in `utils/stats.py` that `python -m cli stats` and the viewer use. `cli stats` prints the all-entries rows (the "Tokens per edit", "Seconds per edit", and "Seconds by stage" lines) and the viewer's dashboard plots verdicts and latency by week; the per-week token rows come from the same helpers over one week's entries, which this reproduces on any chain: + +```python +from collections import defaultdict +from ledger.chain import load_ledger +from utils.stats import billed_tokens_per_entry, seconds_by_stage, tokens_per_entry, week_of + +weeks = defaultdict(list) +for entry in load_ledger(): + weeks[week_of(entry.get("timestamp", ""))].append(entry) +for week, entries in sorted(weeks.items()): + print(week, tokens_per_entry(entries), billed_tokens_per_entry(entries), seconds_by_stage(entries)["total"]) +``` Tokens per governed edit, all stages, by week of the operational chain, as of 2026-09-07: