Skip to content

About

In-depth caveman tool analysis

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

jayn-caveman

Caveman really does compress model prose — it just doesn't reach your bill.

This is the measurement code behind that claim: the detector, the replay, the pricing, and the paired-trial harness. Every figure below is regenerated by a command in this repository, and the command is printed next to it.

The narrative version is docs/blog-caveman.md. This README is the same story with the commands attached. REPRODUCING.md goes deeper on each one.

Warning : Yes, this is mostly pair-programmed with Opus 5.


TL;DR

On most corpora caveman cannot save more than 4% of the bill, however well it works. If it fired on every turn it would save +0.67%. As it actually fires, it returns between +0.02% and −0.03% — nothing, and the billing data cannot resolve which side of zero. The reminder injections cost more than the compression saves below roughly 700 English prose tokens per prompt you send.

Where caveman genuinely ran, it did save: $4.74 on $755.46, or 0.6%. That is a measurement. The ±0.03% is a projection onto people who never installed it, and the two are different questions.


Install

git clone git@github.com:jaynapp/jayn-caveman.git
cd jayn-caveman
npm install
npm run cli -- --help

Node 20+. No API key is needed: token counting is offline by default (calibrated BPE). Pass --exact to any command to use Anthropic's count_tokens endpoint instead, which needs ANTHROPIC_API_KEY.

Everything reads directories shaped like ~/.claude/projects. Point --root at your own, or at a frozen copy — a live root grows while it is being read, so two runs over it are not comparable.

# your own transcripts, the fast path
npm run cli -- analyze --root ~/.claude/projects --model claude-opus-5

The findings, and the command that produces each

1. The ceiling: prose is a small slice of the bill

Prose is 4.1%–7.0% of output tokens, and 0.4%–1.1% of the bill counting only the turn that wrote it. But prose stays in the prefix and is paid for again on every later turn — cheaply while the cache holds it, at the full write price on the turns where the cache had lapsed — so the real ceiling, what you would save if caveman deleted all prose and were free, is 2.5% to 4.2% of the bill, and under 3% on three of the four corpora.

npm run cli -- analyze --root <dir> --model claude-opus-5
#   Audit → "prose share of bill" — the output side alone, 0.4%–1.1%

npx tsx tools/ceiling.ts <dir>,<dir> [proportional|read|rewrite]
#   → the amplified ceiling, 2.5%–4.2%

That range is 2.3%–4.3% across the two corners of what billing can identify; see docs/replay-correction.md.

The corpus topping the range has an unremarkable 6.6% prose share; its sessions are simply long enough for the same tokens to be paid for hundreds of times. That is the whole budget caveman is competing for.

2. Two numbers, not one

effective_ratio(turn) = p_fire × R + (1 − p_fire) × 1.0

R is how much shorter a turn gets when caveman fires. p_fire is how often it fires at all. Published caveman figures report something near R and implicitly assume p_fire = 1. On a single-turn benchmark that is nearly true; on a real session it is not.

3. p_fire decays with session depth

A turn counts as caveman-style when its sentence length falls below the 25th percentile of vanilla turns in the same language, size band, shape and model family. Vanilla turns trip that detector 25% of the time by construction, so the floor is measured per bin and subtracted:

p_fire = (observed − floor) / (1 − floor)

Sensitivity — how often caveman fires and the detector catches it — is unknown and taken as 1, so every p_fire here is a lower bound.

turn index closing turns mid-run turns
0–10 93% 39%
10–20 80% 37%
20–40 66% 21%
40–80 42% 11%
80+ 41% 13%

Every cell above is measured directly. None of them is thin enough to need the position gap borrowed from the fitted curve.

npm run cli -- compliance --root <dir>,<dir> --by-position --model claude-opus-5

Restricted to Opus 5 and English in both arms: 2,313 turns, 2,257 scoreable, 1,070 with caveman live, across three corpora. Caveman's deletion rules name English function words and were measured not firing on French at all, so a mixed-language corpus measures the instrument rather than the tool — English-only is the default everywhere in this repo.

4. Position, not content, is what splits the turns

Splitting turns into "pure text" vs "carries a tool call" looks informative and is an artefact. Over the 2,275 English Opus-5 turns carrying prose:

             last=0   last=1
  pure=0       1737      313
  pure=1          0      225   <- P(last | pure) = 100%

Every pure-text turn is the model's closing turn. And that is where the money is:

share of turns share of prose tokens mean prose tokens/turn
closing turns 23.6% 77.6% 799
mid-run turns 76.4% 22.4% 72

5. R, measured in interactive sessions

90 real interactive coding sessions — 45 matched ON/OFF pairs, two operators, same prompt pasted into each arm against the same pinned repository state, English only.

stratum token-mass R pair IQR basis
closing 0.689 0.59–0.77 45 pairs, 39,178/56,885 tok
mid-run 0.383 0.15–0.93 254/325 turns, 961/2,512 tok
npm run cli -- trial analyze --root <ledger>,<ledger>

The mid-run per-turn statistic is 0.490; the token-mass 0.383 is what gets priced, because bills are paid in tokens. Both include silent turns: caveman produced no prose at all on 206 of 254 mid-run turns, against 232 of 325 in control.

To run the trial yourself, see REPRODUCING.md — trial init, trial plan, trial run, then trial analyze.

6. Caveman's own cost, and why it decides the answer

The SessionStart ruleset costs 457–467 tokens once per session; the per-prompt reminder costs 34–50 tokens on every user turn. A corpus that never ran caveman has no profile of its own, so it borrows the midpoint, 462 and 42 tokens (SHIPPED_PROFILE in src/transcript/injection.ts).

That borrowing is not a detail. Charge nothing for injections and the same replay reports +0.41%; charge them and it reports −0.00%. A projection that omits the tool's cost is not a projection of the tool.

npx tsx tools/injection-term.ts <dirA>,<dirB>,<dirC>,<dirD> claude-opus-5

7. What it did, and what it would do

Measured, on the 98 caveman-live sessions of 108 ($755.46 of bill): saved $4.74, 0.6%. The token-weighted fire rate over those turns is 50.6%.

scenario saved
caveman's advertised 0.35, charged to both strata $12.64 · 1.7%
lower pair quartile (closing 0.59 / mid-run 0.15) $6.92 · 0.9%
pooled interactive trial (0.689 / 0.383) $4.74 · 0.6%
upper pair quartile (closing 0.77 / mid-run 0.93) $2.72 · 0.4%

Caveman's advertised 0.35 saves nearly three times the measurement — because it is charged to both strata, while the quartiles leave closing turns at 0.59–0.77 where the trial found them. Closing turns carry 78% of the prose, so most of the advertised saving is a claim about the stratum that holds the money.

Projected, onto four corpora (504 English sessions) that never installed it:

corpus bill if it always fired with p_fire wired in
corpus A $4578.70 +$30.60 · +0.67% +$2.94 · +0.06%
corpus B $623.58 +$4.25 · +0.68% −$3.15 · −0.51%
corpus C $131.84 +$1.00 · +0.76% −$0.10 · −0.08%
corpus D $97.65 +$0.80 · +0.82% +$0.08 · +0.08%
all $5431.76 +$36.61 · +0.67% −$0.24 · −0.00%
npm run cli -- corpora --root <dirA>,<dirB>,<dirC>,<dirD> --model claude-opus-5
#   --placement read | rewrite prices the two corners of the same table

Two positive, two negative, pooled indistinguishable from zero. Corpus A is 84% of that bill, so read the rows rather than the total.

The pooled p_fire cell is finer than the data can resolve. Billing says what fraction of a prefix was written a second time, never which tokens those were, so it cannot say whether the tokens this replay invents would have been among them. Across the two corners that allows, that cell runs −$1.57 (−0.03%) to +$1.03 (+0.02%): the sign is not identified, the magnitude is. The always-fired column is robust over the same range, +0.63% to +0.67%.

Note on --root labels. Rows are labelled by directory basename. Pointing at several .../<name>/projects paths collapses them into one row called projects. Give each corpus a distinctly named directory if you want per-corpus rows.

8. The only number that transfers to you

Caveman's cost is charged per prompt; its benefit is a fraction of the English prose in the answer. So it pays exactly when the answer is long enough to cover the reminder that asked for it.

English prose per prompt sessions share of bill share that gain money-weighted
0–200 92 2.7% 0% −0.428%
200–400 123 30.6% 10% −0.260%
400–600 112 28.2% 39% −0.002%
600–800 71 16.1% 54% +0.161%
800–1000 44 7.9% 57% +0.187%
1000–1500 38 10.8% 63% +0.291%
1500–2500 18 2.8% 78% +0.462%
2500+ 6 0.9% 100% +0.261%
npm run cli -- breakeven --root <dir>,<dir> --model claude-opus-5
#   → "Breaks even between 600 and 800 prose tokens per prompt."
#   --placement read moves that to 400–600; the corners are in docs/replay-correction.md

Divide the English prose your model writes by the prompts you send. Well under 700, don't bother. Well over, it might pay.


Reproducing it end to end

# 1. the ceiling and the headline band, per corpus
npm run cli -- analyze --root <dir> --model claude-opus-5

# 2. how often it fires, by contributor and turn position
npm run cli -- compliance --root <dir>,<dir> --by-position --model claude-opus-5

# 3. what it would do to someone who never installed it
npm run cli -- corpora --root <dirA>,<dirB> --model claude-opus-5

# 4. the variable that decides the sign
npm run cli -- breakeven --root <dirA>,<dirB> --model claude-opus-5

# 5. R, from the paired trial ledgers
npm run cli -- trial analyze --root <ledger>,<ledger>

# 6. refit the shipped thresholds from data/
npm run cli -- curves build

# 7. the amplified prose ceiling, and the two corners around it
npx tsx tools/ceiling.ts <dirA>,<dirB> proportional

# 8. what charging caveman's own injections is worth
npx tsx tools/injection-term.ts <dirA>,<dirB> claude-opus-5

Steps 7 and 8 price tools that do not exist — perfect free compression, and caveman with its cost side switched off — so they live in tools/ rather than in the CLI.

Useful flags on most commands:

flag effect
--model <family> fit p_fire on one model family only — recommended, see below
--mixed-languages keep non-English sessions (off by default, and off for good reason)
--exact count tokens via count_tokens instead of the offline BPE counter
--by-position compliance only: split each curve into closing and mid-run
--placement <where> analyze/corpora/breakeven: where a changed token sits in a re-written prefix

Checks: npm test (153 tests), npm run typecheck, npm run lint, npm run format:check.


What you should not conclude from a clean run

  • p_fire is not a compliance rate for any population. It is a rate against a vanilla baseline, and the baseline is whoever contributed. Leaving one contributor out moves the pooled rate from 47.1% to anywhere in 44.0%–54.9%, and the contributor moving it down most holds none of the caveman-live turns — they are pure baseline. The shape (position gap, decay) survived every subset we tested; the level did not.
  • Vanilla terseness spans 28 percentage points between model families. Always pass --model. A curve fitted across a mixture partly measures the mixture.
  • R is an English number. Nothing here measured what caveman does to prose in any other language, and it was observed not to fire on French at all.
  • Thinking tokens are assumed unaffected, and this is the weakest assumption in the model. Transcripts store thinking as empty text, so it is only reachable as a residual. Thinking is roughly 89% of billed output on these corpora — if caveman moves it, everything here is understated by a lot.
  • The replay prices a token the bill never contained. Usage totals say how much of a prefix was written a second time after a cache TTL lapsed; they never say which tokens those were. So where a counterfactual token sits in that prefix is a modelling choice, and it is worth ±0.03% of the pooled bill — more than the pooled answer itself. --placement read|rewrite prices the corners; docs/replay-correction.md works through why.
  • A doubly-registered hook is not the tool's cost. Both our machines registered each caveman hook twice for months, delivering the same reminder twice on 98.9% of injection-carrying turns. That cost $2.67 and is excluded from anything labelled "saved".

Contributing data

compliance record exports your transcripts as a contributable batch of style observations under data/ — sentence-length statistics only, never prose. curves build refits the shipped thresholds from every contributed batch. See REPRODUCING.md and docs/methodology.md.

The transcripts behind the numbers above are not published and will not be: they are other people's work.


License

Code under LICENSE; contributed observation data under LICENSE-DATA. See NOTICE.

About

In-depth caveman tool analysis

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages