feat: route breadth work to a haiku surveyor, measured against a control arm - #33
Merged
Conversation
The unit tests cover the report and say nothing about the half of this plugin that is prose. This adds the missing half: an eval case that asks the same codebase question twice, once with the plugin loaded and once without, and scores both arms on graders either arm can pass. The cases are written for `claude plugin eval`, which is early access and gated per organization; where the gate is closed `evals/run-local.mjs` reads the same `prompt.md` + `graders/*.md` files and does the two-arm run through `claude -p`. It exists to be deleted once the gate opens. The baseline arm is made genuinely plugin-free by dropping the user's own settings, since `enabledPlugins` is where an installed copy would otherwise reach both arms - the `system/init` event in a baseline trace lists no `model-routing:*` agent, which is the check worth repeating before trusting any run. First result on an opus session, 3 runs per arm: the plugin arm delegated to `scout` in 3 of 3, the plugin-free arm delegated in 0 of 3 despite having `Explore` available, and both arms answered correctly. The same run also prices the trade - identical answers for 2.3x the money and 3x the wall clock on a four-file fixture, which is the plugin routing where its own anchor says a dispatch costs more than the answer. Both halves are recorded rather than only the flattering one, and the README no longer claims the behavioural half has no control group.
The first write-up read the total cost and concluded the plugin arm was a 2.3x net loss. The per-model split says something different: opus cost is flat at $0.140 against $0.135 while opus reads 34% fewer tokens, and the entire increase in the total is the sonnet subagent doing the file reading instead. That is the mechanism working, not the plugin costing - a total that goes up while the top tier holds flat is precisely the shape this plugin is trying to produce. Two corrections to the numbers that were published: the 2.3x came from averaging in the first run of the batch, which paid a cold prompt cache at $0.518 against $0.206 for the two warm runs that followed, and the totals were never split by tier, so the one number that matters was not visible at all. The runner now prints cost and input tokens per model per arm. The structural caveat is worth more than either number: a `-p` run is one shot, so the tokens kept out of the main context have no later turn to be re-read in and the saving has nowhere to accumulate. Every case written against this harness under-measures the plugin, and that belongs in the README rather than in someone's head.
… money The small case could only ever show the cost of delegating, never the return. This adds the case that can show both, and the three things it took to get there - each of which was a measurement that came back negative first. The question has to be wide, because a needle question is answered by one grep and a grep costs the same in any session. The fixture is a twelve-stage pipeline where each stage hands off to exactly one other, no file states the order, and the stage modules are fat enough that walking the chain costs something. It ships as a generator rather than 85 committed modules. The session has to continue, because a single `-p` turn gives the spared context no later turn to be re-read in. Cases now take `followup-*.md` turns, run through `--continue` in the same session. And the subagent's tier has to be cheap enough, which is arithmetic rather than judgement: a fresh subagent pays cache write for everything it reads, a main session pays cache read for what it already has, and read is 12.5x cheaper than write at the same tier. Result over 4 turns, 3 runs per configuration, opus main session. The expensive tier drops either way - opus $1.169 with sonnet and $1.042 with haiku against $1.358 with no plugin - but the shipped `scout` pin ends up 23% more expensive in total, and haiku 9% cheaper. Every haiku run came in under the cheapest baseline run, every sonnet run over the dearest, and correctness held at 3/3 throughout. The plugin dispatched once per run and answered the follow-ups from the summary, which is what the guidance asks for. The `scout` pin is deliberately left as it ships. One workload that rewards breadth over judgement is not evidence that haiku answers a subtle question well enough, and a wrong answer that sends the main session back to re-read is the expensive failure mode this case does not measure. The numbers, the arithmetic, and what they do not license are in evals/README.md; the main README no longer implies the routing pays for itself unconditionally.
AqueGen
changed the title
…rusting it Whether a session delegates at all turns out to be a coin it flips - 3/3, 3/3, 2/3, 2/3 and then 0/3 across otherwise identical batches - so a case meaning to price a tier cannot also be measuring that. This adds the case that takes the decision away: identical prompt and follow-ups to the wide-reading case, delegation required, the subagent's tier the only variable left. Getting "the main session does not read the files itself" to actually hold took three attempts, and the first two produced numbers that looked fine. Asking in the system prompt was ignored. Denying Read, Grep and Glob moved the reading into Bash - three opus runs were spent before a tool census showed `cat`, `grep -n` and `sed -n` doing the work, and those figures were void. Denying the shells left ToolSearch, which loads a deferred Read back in. An allowlist holds, so the runner now takes `tools:` from frontmatter and the grader watches the shells too. The case is committed unfinished and tagged as such. With the allowlist in place the answers come back wrong on both haiku and opus - one dispatch, nothing left to check it with - and a configuration that is cheap and wrong has saved nothing, so the tier comparison is not run and no number from this case is reported. NOTES.md carries the intended configurations, the three failed isolation attempts, and the two candidate fixes. `scout` stays pinned sonnet, as it ships.
…served The wide-reading case made pinning `scout` to haiku look like a free 9% off the session bill. This is the case that checked whether the cheap tier can be trusted with a question that is not merely wide, and the answer is no. It asks two questions whose code contains a confident-looking wrong answer. `withRetryIntake` loops `maxRetries` times, `maxRetries` is 3, and it also breaks the moment `error?.retryable === false` - which every stage error sets - so `validateOrder` is called once and the file has a 3 in it for anyone who reads one line. `applyTaxPolicy` looks up `TAX_POLICY[batch.outcome]` against a table keyed `onWarn`/`onFatal`/`onTimeout`/`onUnknown`, so an outcome of "warn" misses, falls through to abort, and returns null rather than the `continue` the `onWarn` row appears to promise. Three runs a tier, four deterministic graders, no judge model: sonnet 12/12, haiku 11/12, opus 4/4 over one reference run. haiku's miss was substantive - it answered 3 - and the pin therefore stays sonnet. The instrument is new and is what this question needed all along: `agent:` in frontmatter runs the session AS the bundled agent, so the tier is asked directly and graded on what comes back. No main session to argue with about whether it really delegated, and nothing for it to contaminate the result with. That also corrects the diagnosis in forced-dispatch-tier/NOTES.md, which had blamed forced delegation for producing wrong answers. The real cause is that `--tools` propagates to subagents: the allowlist starved `scout` of file tools, the dispatches came back empty, and the main session refused to invent an answer. One transcript says "two scout runs returned zero real tool executions". Isolating a main session without disabling the agents it dispatches remains unsolved, and the Superpowers-by-default question stays open behind it.
…costs Two measurements point in opposite directions and both are real. On a twelve-stage tracing question - breadth, no judgement - `scout` on haiku scored 3 of 3 and took 9% off the whole session bill against no plugin at all. On a question whose code contains a confident-looking wrong answer, the same tier took the bait in 1 run of 3 while sonnet took it in none. One pin cannot serve both, and the plugin already has the mechanism for the difference: an explicit `model=` on the dispatch. So this adds the rule that was missing rather than a second agent, which would have cost every session context for a decision the caller can make: enumerating and tracing goes out with `model=haiku`, working out what code actually does keeps the sonnet pin. The anchor carries the short form with the two numbers, the skill carries the reasoning and where it comes from. The README stops implying the routing pays for itself. What it reliably buys is room in the expensive model's context - opus down 14%, measured - and on a wide-reading session the total bill goes UP 23%, because a fresh subagent pays cache write for what a main session would have paid cache read for. That is the honest framing, and the case for the cheaper total is now a routing decision with a quality bound on it rather than a pin change nobody checked.
AqueGen
changed the title
The README sold tiered routing as saving money, and the evals showed that claim is false in the comparison a reader assumes: against a session that just does the work itself, delegating costs 23% more on a wide-reading question, on either pin. Leaving that as a paragraph three screens down, under a heading about measurement, was the kind of accurate-but-buried that functions as a sales pitch. The comparison the plugin actually wins is the one most users are already in. A subagent starts empty, so everything it reads is a cache write while the main session pays cache read - 12.5x less - for context it already holds, and that penalty lands whether or not the tier is routed down. Routing changes the price per token, not the penalty. So on work that was going to be dispatched anyway - Superpowers, workflows, any unpinned agent inheriting the session model - it is a straight discount with nothing on the other side. Repricing the measured subagent tokens (74k cache write, 451k cache read, 6k output) at opus rates puts that number at $2.013 against the $1.675 measured with the tier routed down, so roughly 17% off a dispatch already leaving the session, against a 23% surcharge on one that did not need to leave. The pricing model reproduces the billed sonnet cost to the cent, which is why the multiplier can be trusted even though the arm itself is still blocked on the isolation problem in forced-dispatch-tier/NOTES.md. Both the repricing and its two limits are written down rather than implied. This goes at the top of the README behind its own quick link, and into the skill, so the model applying these rules knows they pay off on dispatches that already exist and stops short of manufacturing dispatches to collect the discount.
…e data did not carry Two independent reviews went over this branch. Between them they found one defect that invalidated a ground truth, several published sentences that outran their evidence, and a routing rule that contradicted a rule three lines below it. The fixture could not execute. Every generated stage module imported `retry<Stage>` while the helper exports `withRetry<Stage>`, so nothing linked, and the quality case's two asserted answers - `validateOrder` called once, `applyTaxPolicy` returning null - were claims about the runtime behaviour of code that had no runtime. Both are now confirmed by running it: a non-retryable error produces exactly 1 call and a retryable one produces 3, which also proves the `maxRetries: 3` trap is real rather than assumed. `evals/evidence/README.md` carries the two-command check so the next reader need not take the reading on trust either. The guidance change was made the way the plugin's own rules forbid. Dispatching `scout` with `model=haiku` overrides a role agent below its pin, and the floor rule in the same anchor - and the skill, in as many words - says to reach for an agent pinned for the work instead. So breadth work now has one: `surveyor`, haiku, read-only, with its own eval case that scores it 3/3 at $0.156 against sonnet's $0.439 on the same question. That is ~108 always-on tokens, and it replaces a line that told every session to do two contradictory things. The writing has been pulled back to what three runs on one fixture can hold. "Measured $2.01" now says repriced, since that arm never ran. "Correctness held throughout" now says which turn was scored, since three of four were not. "A third of subtle questions" is gone - 1/3 against 0/3 is not a distinguishable rate, and the reason to keep judgement on sonnet is the asymmetry of the failure, not its frequency. "Delegating cost more on either pin" was simply false for haiku, which came in under the baseline. Percentages are qualified as one workload; opus "down 14%" now distinguishes spend from the ~25% drop in input volume it was being conflated with. The runner had four ways to publish a number that was not there. A killed child closes with code null, which `?? 0` reported as a clean run whose truncated trace was then graded and whose missing cost silently left the mean; failed runs are now excluded from scoring and the mean prints its own denominator. `last_message` graders matched the joined text of every turn, so an ordered pattern could assemble a pass out of pieces of several answers; they now match per turn. A generated fixture was reused even after its generator changed. And `CLAUDE_CODE_SUBAGENT_MODEL` was inherited into experiments whose whole subject is the subagent tier - it is now stripped from the child environment. Evidence is committed rather than gitignored. `evals/evidence/` holds the aggregate behind every published table plus the version-stamped `claude plugin details` output, because a reader cannot reproduce a historical sample by rerunning a stochastic model. One reviewer suspicion is rejected with evidence: per-turn costs are not cumulative across `--continue`, and the extra `result` events some invocations emit carry identical `modelUsage` to the final one, with only a stale `total_cost_usd`. Taking the last event per turn is correct, and the dropped event is a duplicate rather than unbilled spend.
"Which files mention X" and "which files import X" sat one line apart in the anchor, routed to different agents, and read as the same question. A rule a session cannot apply is worse than no rule, and this one is paid for in every session. The three routes now split on the answer rather than the phrasing: unverified candidates to look at next go to the built-in Explore, a complete list or a verified ordering goes to `surveyor`, and a judgement about what code does goes to `scout`. The skill says outright that the two lookalike questions are not the same - one is answered by grep and may be wrong at the edges, the other has to be right.
AqueGen
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The unit tests cover the report - given a transcript, does the maths come out right. They cover none of the half of this plugin that is prose, so nothing until now told us whether the routing guidance changes what a session does, or whether the change is worth paying for. This measures both, changes the guidance on the strength of it, and rewrites the README claim that turned out not to survive the measurement.
What ships
A third exploration route,
surveyor(haiku). Breadth work - a complete list, a verified ordering, everything that imports X - runs correctly a tier belowscoutand costs about a third as much. Judgement work does not, and stays onscout. The anchor now splits the three routes by what has to come back rather than by how the question is phrased, because "which files mention X" and "which files import X" read alike and are not the same question.A README that stops implying the routing pays for itself. What it reliably buys is room in the expensive model's context. Whether it lowers the bill depends entirely on whether you were going to dispatch anyway, and that now has its own section at the top rather than a paragraph three screens down.
Four eval cases, a runner, and the evidence behind every number, in the format
claude plugin evalreads (evals/<case>/prompt.mdplusgraders/*.md). That command is early access and gated per organization; the gate is closed here and neither the CLI version nor the environment is the cause, soevals/run-local.mjsreads the same files and does the two-arm run throughclaude -p. It exists to be deleted the day the gate opens.The behavioural claim holds
Opus session, 3 runs per arm: the plugin arm delegated a codebase question to
scout3/3, the plugin-free arm 0/3 and grepped inline despite havingExploreavailable. Both answered correctly. That is the control group the README did not have, and the baseline is not a straw man.Worth knowing: whether a session delegates is a coin it flips - 3/3, 3/3, 2/3, 2/3 and once 0/3 across otherwise identical batches. The guidance is a request, followed most of the time.
The money
Wide-reading question, four-turn session, 3 runs per configuration. Per-run totals rather than means, because with three runs a mean can hide an overlap:
scoutpinned sonnetscoutpinned haikuThe expensive tier drops either way. The total goes up about a fifth against not delegating at all, and down about a sixth against delegating at the session tier - which is what an unpinned subagent inherits, and what Superpowers does by default. The arithmetic: a fresh subagent pays cache write for everything it reads, a main session pays cache read, 12.5x cheaper, for what it already has. Delegation only pays when the tier discount covers that conversion.
So the plugin is for sessions that already dispatch. Where dispatches were not going to happen, it costs money by causing them - and that is now the first thing the README says.
Why the pin did not change
Haiku's 9% looked like the fix until the tier was asked something subtler. Two questions whose code contains a confident wrong answer -
withRetryIntakebreaking onretryable === falsedespitemaxRetries: 3, andapplyTaxPolicymissing its ownonWarnkey - gave sonnet 12/12 and haiku 11/12, the miss being a substantive3.One miss in three runs cannot pin a failure rate. The pin stayed on sonnet because the failure is asymmetric: a cheap right answer saves cents, a cheap wrong one arrives looking identical and sends the main session back to read the files itself.
surveyorexists so the cheap tier still gets the work it is measured to be good at, and so that nothing overrides a role agent below its pin - which the plugin's own floor rule forbids, and which the first draft of this PR did anyway.Two reviews, and what they cost
A fixture that could not run. Every generated stage imported
retry<Stage>while the helper exportswithRetry<Stage>, so nothing linked - and the quality case's ground truth was a claim about the runtime of code with no runtime. Fixed and now confirmed by execution: a non-retryable error yields exactly 1 call, a retryable one yields 3.evals/evidence/README.mdcarries the two-command check.Four ways the runner could publish a number that was not there. A killed child closes with code
null, which?? 0reported as clean - truncated trace graded, missing cost quietly absent from the mean.last_messagegraders matched the joined text of all turns, so an ordered pattern could assemble a pass out of pieces. A generated fixture was reused after its generator changed.CLAUDE_CODE_SUBAGENT_MODELwas inherited into experiments whose subject is the subagent tier.Claims pulled back to what the data carries. "Measured $2.01" now says repriced, since that arm never ran. "Correctness held throughout" says which of the four turns was scored. "A third of subtle questions" is gone. "Delegating cost more on either pin" was false for haiku. Opus "down 14%" now distinguishes spend from the ~25% drop in input volume it was conflated with.
Evidence committed.
evals/evidence/holds the aggregate behind every published table plus version-stampedclaude plugin detailsoutput, because a reader cannot reproduce a historical sample by rerunning a stochastic model.One reviewer suspicion is rejected with evidence: per-turn costs are not cumulative across
--continue, and the duplicateresultevents some invocations emit carry identicalmodelUsagewith only a staletotal_cost_usd. Taking the last event per turn is correct.Still open
forced-dispatch-tieris committed unfinished and tagged so the runner skips it. It would replace the repriced $2.01 with a measured arm, and cannot until there is a way to stop a main session reading without also starving the subagents it dispatches ---toolsdoes both, which is what produced its first, wrong conclusion. NOTES.md has the table of what was tried.Also unmeasured: opus output tokens are the largest single line in every run, and no tier routing touches them. Four turns never compact, while a real session does - so these numbers are the floor of the plugin's case, not the whole of it.
Runs cost real tokens, so nothing here is wired into CI. Unit suite green, 74/74; one of those tests caught the new agent missing from the dispatch counter's pin table.