Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ tags. Since the runner began recording verdicts, partway through 1.0.0, every
change has landed through Bench's own governance pipeline and carries a
receipt in the operational ledger; the earliest 1.0.0 work predates that.

## [2.1.0] - unreleased
## [2.1.0] - 2026-09-06

The theme is making the claim survive a skeptic: cheaper verdicts, a real
installer, honest numbers, and tests for the properties that used to live in
Expand Down
41 changes: 29 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,27 +300,44 @@ Set `BENCH_PROVIDER=claude_code` to run the pipeline on the subscription that al

## What It Costs

Every governed `Write`, `Edit`, or `MultiEdit` is three sequential model calls, and Claude Code makes many small edits. The figures below come from Bench's own operational ledger. `python -m cli stats` prints each of them (the "Tokens per edit", "Seconds per edit", and "Seconds by stage" lines), and the viewer's dashboard shows the same distributions, so anyone with a chain can reproduce the table for their own ledger rather than take an estimate.
Every governed `Write`, `Edit`, or `MultiEdit` is three sequential model calls, and Claude Code makes many small edits. The figures below come from Bench's own operational ledger, through the helpers in `utils/stats.py` that `python -m cli stats` and the viewer use. `cli stats` prints the all-entries rows (the "Tokens per edit", "Seconds per edit", and "Seconds by stage" lines) and the viewer's dashboard plots verdicts and latency by week; the per-week token rows come from the same helpers over one week's entries, which this reproduces on any chain:

```python
from collections import defaultdict
from ledger.chain import load_ledger
from utils.stats import billed_tokens_per_entry, seconds_by_stage, tokens_per_entry, week_of

weeks = defaultdict(list)
for entry in load_ledger():
weeks[week_of(entry.get("timestamp", ""))].append(entry)
for week, entries in sorted(weeks.items()):
print(week, tokens_per_entry(entries), billed_tokens_per_entry(entries), seconds_by_stage(entries)["total"])
```

Tokens, over 1,148 entries that recorded usage, as of 2026-09-04:
Tokens per governed edit, all stages, by week of the operational chain, as of 2026-09-07:
Comment thread
dburks-svg marked this conversation as resolved.

| Figure | Value |
|---|---|
| Median tokens per governed edit, all stages | 36,166 |
| 90th percentile | 47,590 |
| Week | Governed edits | Median tokens | 90th percentile | Median at cached rates |
|---|---|---|---|---|
| 2026-W36, the v2.1 roadmap week | 1,797 | 34,747 | 42,650 | 33,451 |
| 2026-W37, the release week so far | 80 | 19,483 | 36,811 | 18,124 |
| Every entry with usage recorded (2,705) | | 34,967 | 44,486 | 33,979 |

Wall time, recorded per stage from v2.1 onward. The first 78 timed entries, all on the `claude_code` provider, which cold-starts a `claude` process per stage:
The release-week row is the first under every 2.1 cost change together, and it is a small sample of light edits: most were docstring changes, which the Challenger clears in seconds and the Defender then skips. It is not a settled reduction. The roadmap-week row is the number to plan on until the release week fills out, and the roadmap's target of a median under 20,000 tokens stays a target.

Wall time, recorded per stage, on the `claude_code` provider, which cold-starts a `claude` process per stage. Over the 1,635 timed entries:

| Stage | Median seconds | 90th percentile |
|---|---|---|
| Challenger | 17.3 | 29.5 |
| Defender | 10.2 | 30.8 |
| Oracle | 21.9 | 29.9 |
| Whole edit | 47.0 | 92.9 |
| Challenger | 21.8 | 38.7 |
| Defender | 15.8 | 38.6 |
| Oracle | 23.1 | 34.9 |
| Whole edit | 61.9 | 107.7 |

By week, the whole-edit median was 63.5 seconds in 2026-W36 and 22.0 seconds in 2026-W37, with the same caveat on the second figure.

The Defender median is low because a CLEAR challenge skips it and records zero seconds. The direct API providers avoid the process start but not the sequential three-call shape. Both tables will drift as the ledger grows; run `python -m cli stats` for the current figures on your own chain, and `python -m cli viewer` for the per-week view.

Roadmap v2.1 targets a median under 20,000 tokens, and two of its steps are in. Each stage receives every constraint's rule without its rationale and commentary (`pipeline.constitution.prompt_view`), and constitution v7 moved procedure and history out of the rules into commentary. On the `anthropic` provider the constitution and the repository context open every stage prompt under one cache breakpoint: they are written to the prompt cache on the first edit and read back at a tenth of the input price on every edit that follows within five minutes. Each stage's `_tokens` record now carries `cache_read` and `cache_creation` beside `input` and `output`, and `cli stats` and the viewer print a second per-edit figure with those counted at the cached rates (reads 0.1x, writes 1.25x), so the cost of a cached edit is read off the ledger rather than estimated. On the `claude_code` provider the CLI caches its system prompt file and not stdin text, so the constitution is folded into that file; the repository context is repository input and stays on stdin at user priority, uncached, because a project's CLAUDE.md must not outrank the stage prompt or the constitution. Measured on the development machine, a Challenger call under that layout reads about 2,470 of 9,340 prompt tokens from cache. The figures above predate both changes; the ledger will show the difference.
Roadmap v2.1 targets a median under 20,000 tokens, and two of its steps are in. Each stage receives every constraint's rule without its rationale and commentary (`pipeline.constitution.prompt_view`), and constitution v7 moved procedure and history out of the rules into commentary. On the `anthropic` provider the constitution and the repository context open every stage prompt under one cache breakpoint: they are written to the prompt cache on the first edit and read back at a tenth of the input price on every edit that follows within five minutes. Each stage's `_tokens` record now carries `cache_read` and `cache_creation` beside `input` and `output`, and `cli stats` and the viewer print a second per-edit figure with those counted at the cached rates (reads 0.1x, writes 1.25x), so the cost of a cached edit is read off the ledger rather than estimated. On the `claude_code` provider the CLI caches its system prompt file and not stdin text, so the constitution is folded into that file; the repository context is repository input and stays on stdin at user priority, uncached, because a project's CLAUDE.md must not outrank the stage prompt or the constitution. Measured on the development machine, a Challenger call under that layout reads about 2,470 of 9,340 prompt tokens from cache. The roadmap-week row above straddles both changes, which landed partway through it; the release-week row is the first entirely under them, and the ledger will show whether its figure holds as it fills with ordinary edits.

## Design Decisions

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "bench-governance"
version = "2.1.0.dev0"
version = "2.1.0"
description = "Constitutional governance layer for Claude Code: every edit is challenged, defended, ruled on, and hash-chained into a ledger."
readme = "README.md"
license = "MIT"
Expand Down