Caveman really does compress model prose — it just doesn't reach your bill.
This is the measurement code behind that claim: the detector, the replay, the pricing, and the paired-trial harness. Every figure below is regenerated by a command in this repository, and the command is printed next to it.
The narrative version is docs/blog-caveman.md. This README is the same
story with the commands attached. REPRODUCING.md goes deeper on each one.
Warning : Yes, this is mostly pair-programmed with Opus 5.
On most corpora caveman cannot save more than 4% of the bill, however well it works. If it fired on every turn it would save +0.67%. As it actually fires, it returns between +0.02% and −0.03% — nothing, and the billing data cannot resolve which side of zero. The reminder injections cost more than the compression saves below roughly 700 English prose tokens per prompt you send.
Where caveman genuinely ran, it did save: $4.74 on $755.46, or 0.6%. That is a measurement. The ±0.03% is a projection onto people who never installed it, and the two are different questions.
git clone git@github.com:jaynapp/jayn-caveman.git
cd jayn-caveman
npm install
npm run cli -- --helpNode 20+. No API key is needed: token counting is offline by default (calibrated BPE). Pass
--exact to any command to use Anthropic's count_tokens endpoint instead, which needs
ANTHROPIC_API_KEY.
Everything reads directories shaped like ~/.claude/projects. Point --root at your own, or at
a frozen copy — a live root grows while it is being read, so two runs over it are not comparable.
# your own transcripts, the fast path
npm run cli -- analyze --root ~/.claude/projects --model claude-opus-5Prose is 4.1%–7.0% of output tokens, and 0.4%–1.1% of the bill counting only the turn that wrote it. But prose stays in the prefix and is paid for again on every later turn — cheaply while the cache holds it, at the full write price on the turns where the cache had lapsed — so the real ceiling, what you would save if caveman deleted all prose and were free, is 2.5% to 4.2% of the bill, and under 3% on three of the four corpora.
npm run cli -- analyze --root <dir> --model claude-opus-5
# Audit → "prose share of bill" — the output side alone, 0.4%–1.1%
npx tsx tools/ceiling.ts <dir>,<dir> [proportional|read|rewrite]
# → the amplified ceiling, 2.5%–4.2%That range is 2.3%–4.3% across the two corners of what billing can identify; see
docs/replay-correction.md.
The corpus topping the range has an unremarkable 6.6% prose share; its sessions are simply long enough for the same tokens to be paid for hundreds of times. That is the whole budget caveman is competing for.
effective_ratio(turn) = p_fire × R + (1 − p_fire) × 1.0
R is how much shorter a turn gets when caveman fires. p_fire is how often it fires at all.
Published caveman figures report something near R and implicitly assume p_fire = 1. On a
single-turn benchmark that is nearly true; on a real session it is not.
A turn counts as caveman-style when its sentence length falls below the 25th percentile of vanilla turns in the same language, size band, shape and model family. Vanilla turns trip that detector 25% of the time by construction, so the floor is measured per bin and subtracted:
p_fire = (observed − floor) / (1 − floor)
Sensitivity — how often caveman fires and the detector catches it — is unknown and taken as 1,
so every p_fire here is a lower bound.
| turn index | closing turns | mid-run turns |
|---|---|---|
| 0–10 | 93% | 39% |
| 10–20 | 80% | 37% |
| 20–40 | 66% | 21% |
| 40–80 | 42% | 11% |
| 80+ | 41% | 13% |
Every cell above is measured directly. None of them is thin enough to need the position gap borrowed from the fitted curve.
npm run cli -- compliance --root <dir>,<dir> --by-position --model claude-opus-5Restricted to Opus 5 and English in both arms: 2,313 turns, 2,257 scoreable, 1,070 with caveman live, across three corpora. Caveman's deletion rules name English function words and were measured not firing on French at all, so a mixed-language corpus measures the instrument rather than the tool — English-only is the default everywhere in this repo.
Splitting turns into "pure text" vs "carries a tool call" looks informative and is an artefact. Over the 2,275 English Opus-5 turns carrying prose:
last=0 last=1
pure=0 1737 313
pure=1 0 225 <- P(last | pure) = 100%
Every pure-text turn is the model's closing turn. And that is where the money is:
| share of turns | share of prose tokens | mean prose tokens/turn | |
|---|---|---|---|
| closing turns | 23.6% | 77.6% | 799 |
| mid-run turns | 76.4% | 22.4% | 72 |
90 real interactive coding sessions — 45 matched ON/OFF pairs, two operators, same prompt pasted into each arm against the same pinned repository state, English only.
| stratum | token-mass R | pair IQR | basis |
|---|---|---|---|
| closing | 0.689 | 0.59–0.77 | 45 pairs, 39,178/56,885 tok |
| mid-run | 0.383 | 0.15–0.93 | 254/325 turns, 961/2,512 tok |
npm run cli -- trial analyze --root <ledger>,<ledger>The mid-run per-turn statistic is 0.490; the token-mass 0.383 is what gets priced, because bills are paid in tokens. Both include silent turns: caveman produced no prose at all on 206 of 254 mid-run turns, against 232 of 325 in control.
To run the trial yourself, see REPRODUCING.md — trial init, trial plan,
trial run, then trial analyze.
The SessionStart ruleset costs 457–467 tokens once per session; the per-prompt reminder costs
34–50 tokens on every user turn. A corpus that never ran caveman has no profile of its own,
so it borrows the midpoint, 462 and 42 tokens (SHIPPED_PROFILE in
src/transcript/injection.ts).
That borrowing is not a detail. Charge nothing for injections and the same replay reports +0.41%; charge them and it reports −0.00%. A projection that omits the tool's cost is not a projection of the tool.
npx tsx tools/injection-term.ts <dirA>,<dirB>,<dirC>,<dirD> claude-opus-5Measured, on the 98 caveman-live sessions of 108 ($755.46 of bill): saved $4.74, 0.6%. The token-weighted fire rate over those turns is 50.6%.
| scenario | saved |
|---|---|
| caveman's advertised 0.35, charged to both strata | $12.64 · 1.7% |
| lower pair quartile (closing 0.59 / mid-run 0.15) | $6.92 · 0.9% |
| pooled interactive trial (0.689 / 0.383) | $4.74 · 0.6% |
| upper pair quartile (closing 0.77 / mid-run 0.93) | $2.72 · 0.4% |
Caveman's advertised 0.35 saves nearly three times the measurement — because it is charged to both strata, while the quartiles leave closing turns at 0.59–0.77 where the trial found them. Closing turns carry 78% of the prose, so most of the advertised saving is a claim about the stratum that holds the money.
Projected, onto four corpora (504 English sessions) that never installed it:
| corpus | bill | if it always fired | with p_fire wired in |
|---|---|---|---|
| corpus A | $4578.70 | +$30.60 · +0.67% | +$2.94 · +0.06% |
| corpus B | $623.58 | +$4.25 · +0.68% | −$3.15 · −0.51% |
| corpus C | $131.84 | +$1.00 · +0.76% | −$0.10 · −0.08% |
| corpus D | $97.65 | +$0.80 · +0.82% | +$0.08 · +0.08% |
| all | $5431.76 | +$36.61 · +0.67% | −$0.24 · −0.00% |
npm run cli -- corpora --root <dirA>,<dirB>,<dirC>,<dirD> --model claude-opus-5
# --placement read | rewrite prices the two corners of the same tableTwo positive, two negative, pooled indistinguishable from zero. Corpus A is 84% of that bill, so read the rows rather than the total.
The pooled p_fire cell is finer than the data can resolve. Billing says what fraction of a
prefix was written a second time, never which tokens those were, so it cannot say whether the
tokens this replay invents would have been among them. Across the two corners that allows, that
cell runs −$1.57 (−0.03%) to +$1.03 (+0.02%): the sign is not identified, the magnitude is.
The always-fired column is robust over the same range, +0.63% to +0.67%.
Note on
--rootlabels. Rows are labelled by directory basename. Pointing at several.../<name>/projectspaths collapses them into one row calledprojects. Give each corpus a distinctly named directory if you want per-corpus rows.
Caveman's cost is charged per prompt; its benefit is a fraction of the English prose in the answer. So it pays exactly when the answer is long enough to cover the reminder that asked for it.
| English prose per prompt | sessions | share of bill | share that gain | money-weighted |
|---|---|---|---|---|
| 0–200 | 92 | 2.7% | 0% | −0.428% |
| 200–400 | 123 | 30.6% | 10% | −0.260% |
| 400–600 | 112 | 28.2% | 39% | −0.002% |
| 600–800 | 71 | 16.1% | 54% | +0.161% |
| 800–1000 | 44 | 7.9% | 57% | +0.187% |
| 1000–1500 | 38 | 10.8% | 63% | +0.291% |
| 1500–2500 | 18 | 2.8% | 78% | +0.462% |
| 2500+ | 6 | 0.9% | 100% | +0.261% |
npm run cli -- breakeven --root <dir>,<dir> --model claude-opus-5
# → "Breaks even between 600 and 800 prose tokens per prompt."
# --placement read moves that to 400–600; the corners are in docs/replay-correction.mdDivide the English prose your model writes by the prompts you send. Well under 700, don't bother. Well over, it might pay.
# 1. the ceiling and the headline band, per corpus
npm run cli -- analyze --root <dir> --model claude-opus-5
# 2. how often it fires, by contributor and turn position
npm run cli -- compliance --root <dir>,<dir> --by-position --model claude-opus-5
# 3. what it would do to someone who never installed it
npm run cli -- corpora --root <dirA>,<dirB> --model claude-opus-5
# 4. the variable that decides the sign
npm run cli -- breakeven --root <dirA>,<dirB> --model claude-opus-5
# 5. R, from the paired trial ledgers
npm run cli -- trial analyze --root <ledger>,<ledger>
# 6. refit the shipped thresholds from data/
npm run cli -- curves build
# 7. the amplified prose ceiling, and the two corners around it
npx tsx tools/ceiling.ts <dirA>,<dirB> proportional
# 8. what charging caveman's own injections is worth
npx tsx tools/injection-term.ts <dirA>,<dirB> claude-opus-5Steps 7 and 8 price tools that do not exist — perfect free compression, and caveman with its
cost side switched off — so they live in tools/ rather than in the CLI.
Useful flags on most commands:
| flag | effect |
|---|---|
--model <family> |
fit p_fire on one model family only — recommended, see below |
--mixed-languages |
keep non-English sessions (off by default, and off for good reason) |
--exact |
count tokens via count_tokens instead of the offline BPE counter |
--by-position |
compliance only: split each curve into closing and mid-run |
--placement <where> |
analyze/corpora/breakeven: where a changed token sits in a re-written prefix |
Checks: npm test (153 tests), npm run typecheck, npm run lint, npm run format:check.
p_fireis not a compliance rate for any population. It is a rate against a vanilla baseline, and the baseline is whoever contributed. Leaving one contributor out moves the pooled rate from 47.1% to anywhere in 44.0%–54.9%, and the contributor moving it down most holds none of the caveman-live turns — they are pure baseline. The shape (position gap, decay) survived every subset we tested; the level did not.- Vanilla terseness spans 28 percentage points between model families. Always pass
--model. A curve fitted across a mixture partly measures the mixture. Ris an English number. Nothing here measured what caveman does to prose in any other language, and it was observed not to fire on French at all.- Thinking tokens are assumed unaffected, and this is the weakest assumption in the model. Transcripts store thinking as empty text, so it is only reachable as a residual. Thinking is roughly 89% of billed output on these corpora — if caveman moves it, everything here is understated by a lot.
- The replay prices a token the bill never contained. Usage totals say how much of a prefix
was written a second time after a cache TTL lapsed; they never say which tokens those were. So
where a counterfactual token sits in that prefix is a modelling choice, and it is worth ±0.03%
of the pooled bill — more than the pooled answer itself.
--placement read|rewriteprices the corners;docs/replay-correction.mdworks through why. - A doubly-registered hook is not the tool's cost. Both our machines registered each caveman hook twice for months, delivering the same reminder twice on 98.9% of injection-carrying turns. That cost $2.67 and is excluded from anything labelled "saved".
compliance record exports your transcripts as a contributable batch of style observations under
data/ — sentence-length statistics only, never prose. curves build refits the shipped
thresholds from every contributed batch. See REPRODUCING.md and
docs/methodology.md.
The transcripts behind the numbers above are not published and will not be: they are other people's work.
Code under LICENSE; contributed observation data under LICENSE-DATA. See NOTICE.