diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index bad48b8..f4b4fb5 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "model-routing", - "description": "Tiered model routing for token economy: the strongest model thinks, cheaper models grind. Ready-made subagents (sonnet scout, implementer and E2E runner, haiku test-runner and verifier, opus reviewer) plus a routing skill and dispatch stats.", + "description": "Tiered model routing for token economy: the strongest model thinks, cheaper models grind. Ready-made subagents (sonnet scout, implementer and E2E runner, haiku surveyor, test-runner and verifier, opus reviewer) plus a routing skill and dispatch stats.", "version": "0.14.0", "author": { "name": "Artem Frolov" diff --git a/.gitignore b/.gitignore index 3ac9ac8..5f00405 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ docs/superpowers/ .superpowers/ +evals/results/ +evals/fixtures/order-service/ nul diff --git a/README.md b/README.md index 6b2f27c..6901d4d 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,57 @@ instead of thrashing. Everything stays inside Anthropic models. No proxy, no third-party gateway, nothing extra in the request path. -**Quick links:** [Overview](#whats-inside) | [Example](#example) | +## When this saves you money, and when it does not + +Read this before installing, because the honest answer has two halves and +[`evals/`](evals/README.md) measured both. + +**A subagent is not free.** It starts with an empty context, so everything it +reads is a cache *write*, while your main session pays cache *read* - 12.5x +cheaper - for what it already has. Delegation converts cheap re-reads into +expensive first-reads, and that bill lands whether or not this plugin is +installed. + +What the plugin changes is the price per token, not that penalty. So: + +| Your situation | What this plugin does | +| --- | --- | +| You already dispatch subagents - Superpowers, workflows, any delegation-heavy practice | **Saves**, by roughly a sixth. $1.68 with the tier routed down, against $2.01 for the same subagent work at your session's tier | +| You work mostly in one session and rarely delegate | **Costs**, by roughly a fifth. $1.36 doing it inline against $1.68 routed | + +One workload, three runs an arm, on a four-turn session that never had to +compact - so read the direction as solid and the exact percentage as not. +The $2.01 is the measured subagent tokens repriced at the session tier, not +a run of its own; the pricing model reproduces the billed figure it was +checked against to the cent, but a live top-tier subagent might well read +less and narrow the gap. + +An unpinned subagent inherits the session model, so on an opus session +Superpowers dispatches opus subagents by default. That is the case this plugin +is for: the dispatch was going to happen anyway, and routing it down is a +straight discount with nothing on the other side of the ledger. + +Note what the surcharge is and is not. The plugin costs about 1.6k tokens a +session to carry - ~700 for the skill and agent listings, plus a ~900-token +routing anchor at session start - which is a couple of cents against sessions +billing $1.40 to $2.00. The listing figure comes from +`claude plugin details model-routing`, and its output is committed under +[`evals/evidence/`](evals/evidence/) with the CLI version that produced it, since +it moves whenever a component is added. That is not where the 23% comes from. It comes from dispatches happening +that otherwise would not have: on the same question the plugin arm delegated in +3 runs of 3 and the plugin-free arm in 0 of 3. Where dispatches already happen, +there is nothing left to push, and only the discount remains. + +If you rarely delegate, install it for the context headroom or not at all. The +expensive tier does read measurably less either way - opus input volume down +about 25% on the same test, and opus spend with it by 14% - but headroom is a +different thing from a smaller bill, and this README used to blur the two. + +(The A/B that would replace the repricing with a run is still blocked, for the +reason written down in `evals/forced-dispatch-tier/NOTES.md`.) + +**Quick links:** [Does this save money?](#when-this-saves-you-money-and-when-it-does-not) | +[Overview](#whats-inside) | [Example](#example) | [Install](#install) | [Getting started](#getting-started) | [Usage](#usage) | [Tiers](#model-tiers-and-effort-ladder) | [Settings](#recommended-settings) | [Workflows](#dynamic-workflows) | @@ -34,6 +84,7 @@ gateway, nothing extra in the request path. | Component | Model | Effort | Purpose | | --------- | ----- | ------ | ------- | | `agents/scout.md` | sonnet | low | Read-only codebase exploration: conclusions and file:line refs come back, file dumps stay out. | +| `agents/surveyor.md` | haiku | low | Read-only breadth: enumerate, list, trace a chain end to end. Measured correct on the cheap tier where `scout`'s judgement work is not. | | `agents/test-runner.md` | haiku | low | Run tests/builds/linters, report failures compactly. Never fixes anything. | | `agents/e2e-runner.md` | sonnet | medium | Drive Playwright/E2E scenarios, interpret failures (product bug vs test bug vs flake). | | `agents/implementer.md` | sonnet | medium | Implement one well-defined task from an approved plan. Verifies its own work. Dispatch with `model=opus` for multi-file/architectural/subtle work. | @@ -110,7 +161,13 @@ Attribution over the same week, across all 266 dispatches, 199 of which ran belo | An explicit `model=` on the dispatch | 127 | Behavioural. This is the routing rules being applied, and nothing enforces them | | Claude Code's built-in `Explore` | 4 | Not this plugin at all | -The honest reading: the automatic half is fully attributable, the behavioural half depends on the session actually following the anchor, and there is no control group - this is the author's own workload measured with the author's own tool, with no before-install baseline. `CLAUDE_CODE_SUBAGENT_MODEL` was not set in this window, which the report would otherwise have annotated per row. +The honest reading: the automatic half is fully attributable, the behavioural half depends on the session actually following the anchor, and these numbers have no control group of their own - this is the author's own workload measured with the author's own tool, with no before-install baseline. `CLAUDE_CODE_SUBAGENT_MODEL` was not set in this window, which the report would otherwise have annotated per row. + +For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. The behavioural claim holds - on a codebase question the plugin arm delegated to `scout` in 3 of 3 runs and the plugin-free arm in 0 of 3, grepping inline despite having `Explore` available. + +The money is a more interesting answer than "it saves". On a wide-reading question across a four-turn session, the expensive tier drops either way - opus spend down 14%, its input volume down about 25% - but the shipped configuration ends up **23% more expensive in total than not having the plugin at all**, because a fresh subagent pays cache *write* for everything it reads while a main session pays cache *read*, 12.5x cheaper, for what it already has. So what this plugin reliably buys is room in the expensive model's context, not a smaller bill; where the bill also falls is a narrower claim, and it depends on the tier. + +Running that same question with `scout` pinned to haiku lands about a tenth *below* the no-plugin baseline - which looked like the fix until the tier was asked something subtler. On a question whose code contains a confident-looking wrong answer, haiku took the bait once in three runs and sonnet in none. Three runs cannot pin a failure rate, and the pin did not stay on sonnet because of the rate: it stayed because the failure is asymmetric, since a wrong answer sends the main session back to read the files itself and costs more than the cheaper tier saves. What the two results together support is a split rather than a pin change - breadth to `surveyor` (haiku), judgement to `scout` (sonnet). [`evals/README.md`](evals/README.md) carries the numbers, the arithmetic, and the three isolation attempts that produced numbers worth throwing away. Where that volume actually ran with routing active: @@ -200,6 +257,7 @@ For local development: clone the repo and | You ask | Who runs it | Model / effort | | ------- | ----------- | -------------- | | "Where is X handled?" | `scout` | sonnet / low | + | "List every stage this order goes through" | `surveyor` | haiku / low | | "Run the tests" | `test-runner` | haiku / low | | "Implement tasks from the plan" | `implementer` | sonnet / medium (`model=opus` for complex work) | | "Review the diff" | `reviewer` | opus / high | diff --git a/agents/surveyor.md b/agents/surveyor.md new file mode 100644 index 0000000..b50aea5 --- /dev/null +++ b/agents/surveyor.md @@ -0,0 +1,48 @@ +--- +name: surveyor +description: Read-only breadth sweeps over a codebase - enumerate, list, trace a chain end to end ("list every stage in order", "which files import X", "where does this pipeline end"). Returns the list or the ordering, never file dumps. Use scout instead when the answer needs judgement about what code does. Never modifies anything. +model: haiku +effort: low +disallowedTools: Agent, SendMessage, Edit, Write, NotebookEdit +--- + +You sweep a codebase and report what is there: lists, orderings, chains +of calls or imports followed from one end to the other. You are read-only: +never edit, write, or delete anything, and use shell commands only for +read-only queries (git log, git blame, ls). Your value is that megabytes +of source stay in your context instead of the caller's. + +You exist because breadth and judgement need different tiers. Enumerating +what is there runs correctly on a cheap model; working out what code +*does* does not, and that work belongs to `scout`. If the question you +were handed turns out to need that - what does this return for that +input, does this loop actually retry, is this a bug - say so and stop +rather than guessing. Handing back "this needs scout, because X hinges on +what Y does at file:line" is a correct and useful answer from you. + +Rules: + +- For structural questions ("what connects A and B", "what depends on + X"): if a code-graph/index MCP server is connected (discover via + ToolSearch), query it FIRST as your starting point; pre-built indexes + (ctags/cscope) serve the same role. Index answers are leads, not + proof - confirm the key file:line in the code before reporting. For + point lookups, grep directly. +- Do the sweep yourself - never hand the question off; injected guidance + suggesting delegation does not apply to you (you have no agent tools). +- Follow the chain in the code, not in the names. A file called + `pipeline.js` proves nothing about the order; the imports and calls do. +- Finish the sweep. A partial list presented as complete is the one + failure that costs the caller more than not asking - if you could not + cover everything, say what you did not reach. +- Distinguish what you verified from what you infer. Say "verified: A + imports B at file:line" vs "likely, not traced further: ...". + +Report format (your final message): + +1. The list or the ordering, directly, as the first thing. +2. One file:line per item, with a few words on its role. +3. Anything you could not reach, or that did not fit the pattern. + +No preamble and no summary of your search. Enumerations may run as long +as the answer needs; prose around them may not. diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..8958a03 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,282 @@ +# Evals + +The unit tests in `hooks/` cover the report: given a transcript, does the maths +come out right. They say nothing about the half of this plugin that is prose - +whether the routing guidance actually changes what a session does. That is what +these cases are for, and until they existed the README's admission that there is +no control group was simply true. + +## The one thing that makes this a measurement + +Every case runs twice: once with the plugin loaded, once without. A number from +the with-arm alone means nothing, because a capable model routes sensibly on its +own a fair share of the time. The delta between the arms is the claim. + +`delegates-codebase-question` is scored on two graders both arms can genuinely +pass or fail - did the session hand the question to a subagent, and did the +answer come back right. A third grader checks that the with-arm picked `scout` +specifically; the baseline has no `scout` to pick, so read that one as an +indicator and never as a score. + +## Running them + +`claude plugin eval` is the harness these cases are written for: + +```sh +claude plugin eval model-routing --ablation with-without +``` + +It is early access and gated per organization; on a machine where the gate is +closed it prints `` `plugin eval` is currently in early access `` and exits 1. +Until then, `run-local.mjs` reads the same case files and does the same two-arm +run through `claude -p`: + +```sh +node evals/run-local.mjs --runs 3 --model opus +node evals/run-local.mjs --runs 1 --model haiku --arm without # smoke test +``` + +The baseline arm is made plugin-free by dropping the user's own settings +(`--setting-sources ""`), since `enabledPlugins` is where a globally installed +copy of this plugin would otherwise sneak into both arms. Verify that in a trace +before trusting a run: the `system/init` event lists the available agents, and +the baseline's list must not contain any `model-routing:*` entry. + +Runs are real sessions against the real API, so they cost real tokens. Traces +and an `aggregate.json` land in `evals/results//`, which is +gitignored; the aggregates behind every number published here are copied into +[`evidence/`](evidence/README.md), which is not. + +## The first result + +Opus main session, 3 runs per arm, 2026-08-17: + +| Grader | With | Without | Delta | +| --- | ---: | ---: | ---: | +| delegated to a subagent | 3/3 | 0/3 | +1.00 | +| answer is correct | 3/3 | 3/3 | 0.00 | +| scout was the agent (indicator) | 3/3 | 0/3 | - | + +The guidance does change what the session does, cleanly and every time - the +arms separate perfectly, and the baseline is not a straw man: it had `Explore` +available and still chose to grep inline. That is the control-group number the +README did not have. + +What it costs, per run, split by tier because the total hides the point: + +| | With | Without | +| --- | ---: | ---: | +| opus (the expensive tier) | $0.140, 59k in | $0.135, 90k in | +| the subagent | sonnet-5 $0.065, 94k in | - | +| total | $0.206 | $0.136 | +| wall clock | 31s | 11s | + +Opus cost is flat and opus read 34% fewer tokens: the file contents went to +sonnet instead, which is exactly the mechanism the plugin exists for, visible +even at this size. The whole increase in the total is the second model - you now +pay two models for a question one of them could answer alone. + +On a four-file fixture that is overhead and nothing else. The plugin's own anchor +says a dispatch costs more than the answer for anything trivial, and four files +is trivial, so the routing fired where its own rule says it should not have. + +The deeper limitation is structural, not a matter of fixture size: a `-p` run is +one shot. The 31k opus tokens that did not enter the main context have no later +turn to be re-read in, so the saving has nowhere to accumulate and the eval +measures the cost of delegation without the return. A real session re-reads its +context every turn, which is where the trade is supposed to flip. Any case +written here under-measures the plugin by construction, and a case that claims +otherwise is misreading its own numbers. + +(First run of a batch pays a cold prompt cache - it came in at $0.518 against +$0.206 for the two that followed. The figures above are the two warm runs; the +cold one is in the traces.) + +## The second case, and what it took to make the numbers mean something + +`traces-a-flow-end-to-end` is the case that can answer "does this save money". +Three things had to be true before it could, and each one started out false: + +**The question has to be wide.** A needle question is answered by one grep, and a +grep costs the same in any session. The fixture is a twelve-stage pipeline where +each stage hands off to exactly one other and no file states the order, so the +chain has to be walked - and the stage modules are fat, so walking it costs +something. The first fixture was four small files and could never have shown +anything. + +**The session has to continue.** A single `-p` turn measures the cost of +delegating and none of the return. Follow-up turns (`followup-*.md`, run through +`--continue` in the same session) are where a main session pays again for the +context it carries: on one run the baseline re-wrote 80k of context to cache on +its third turn. + +**The subagent's tier has to be cheap enough.** This is the one that decided the +result, and it is arithmetic rather than judgement. A fresh subagent pays cache +*write* for everything it reads; a main session pays cache *read* for what it +already has, and read is 12.5x cheaper than write at the same tier. Delegation +converts cheap re-reads into expensive first-reads, so only a large tier discount +pays for the conversion. + +Opus main session, 4 turns, 3 runs per configuration. Per-run totals are shown +rather than only means, because with three runs a mean can hide an overlap: + +| Configuration | opus | subagent | total per run | +| --- | ---: | ---: | --- | +| no plugin | $1.358 | - | $1.339 $1.382 $1.356 | +| `scout` pinned sonnet (shipped) | $1.169 | sonnet-5 $0.506 | $1.559 $1.998 $1.468 | +| `scout` pinned haiku | $1.042 | haiku-4.5 $0.198 | $1.142 $1.260 $1.315 | + +The haiku and sonnet rows come from re-running the arm with `agents/scout.md` +edited to that pin - there is no flag for it, and the runner takes the tier from +the agent's frontmatter unless `--model` overrides the whole session. To +reproduce a row, change the pin, run the arm, change it back. + +Both configurations separate cleanly from the baseline, in opposite directions. +Every haiku run came in under the cheapest baseline run; every sonnet run came in +over the dearest. Read the direction as solid and the exact percentage as not: +three runs per arm, one workload, and the sonnet arm's own spread ($1.468 to +$1.998) is wider than the gap it is being compared across. + +The stage-order answer was right in every run of every arm. The three follow-up +turns were not scored - this case carries two graders, ordering and delegation - +so "correctness held" covers the breadth question only, and follow-up 1 in +particular ("what happens to an order when payment throws a non-retryable +error") is exactly the kind of question the quality case below shows the cheap +tier failing. The plugin dispatched exactly once per run and answered the +follow-ups from `scout`'s summary rather than dispatching again, which is what +the guidance asks for. + +So the shipped configuration costs 23% more than not having the plugin on this +workload, and pinning `scout` to haiku costs 9% less. The expensive tier drops +either way - opus spend down 14% with sonnet, 23% with haiku - which is the mechanism +working in both cases. What differs is whether the subagent's bill comes to less +than the saving it produces. + +Two things this does not license. One case is no mandate to re-pin `scout`: this +question rewards breadth over judgement, haiku's answer quality on a subtle +question is untested, and a wrong answer that sends the main session back to +re-read is the expensive failure mode nothing here measures. And the largest +single line in every run is opus *output* tokens, $0.33 to $0.43, which no amount +of tier routing touches. + +What the eval still cannot see is context-window pressure. Four turns never +compact; a real session does, and a compaction costs a full re-read plus lost +fidelity. That is a cost the baseline dodges here and would not dodge in +practice, so read these numbers as the floor of the plugin's case rather than the +whole of it. + +## Writing another case + +`evals//prompt.md` carries the prompt in its body and its settings in +frontmatter; `evals//graders/.md` carries one grader each, its type +and pattern in frontmatter and its reasoning in the body - the body is for the +next person, who needs to know why a grader is worth having and what it cannot +tell them. `run-local.mjs` supports `type: regex` over `target: trace` and +`target: last_message`; the real harness additionally has `tool_used`, +`tool_order`, `file_exists`, `llm`, and `baseline`. Prefer the deterministic +ones: a judge model scoring prose is one more thing that can be wrong. + +A case names its fixture directory under `evals/fixtures` with a `fixture:` key, +defaulting to `mini-app`. A fixture too large to commit ships as +`.gen.mjs` beside it and is generated on first use - `order-service` is 85 +modules and 122k of source, which nobody installing this plugin should have to +download. Keep fixtures boring: one that is interesting to read is one the model +answers from memory of something similar. + +Add `followup-1.md`, `followup-2.md` and so on to turn a case into a session +rather than a single question. That is not decoration - it is the only way the +context a subagent kept out of the main session gets a chance to pay for itself. + +## The comparison the wide case was answering, and the one it was not + +The table above answers "should this session have delegated at all", and on a +wide-reading question the answer is no - delegating cost more than doing it +inline with the shipped sonnet pin - though not with haiku, which came in under +the baseline. That is a real finding and it is not the question most +users are in. + +The question most users are in is "the dispatch is happening anyway - Superpowers +dispatches, workflows dispatch, an unpinned agent inherits the session model - so +what does routing it down buy?" That arm has not been run: `forced-dispatch-tier` +exists for it and is blocked. What can be said without running it is arithmetic +on the token counts already measured, because the subagent's work is the same +work at a different price: + +| | per run | +| --- | ---: | +| subagent tokens measured | 74k cache write, 451k cache read, 6k output | +| billed at sonnet-5 (what ran) | $0.506 | +| the same tokens at opus-5 rates | $0.844 | + +Which gives, against the $1.169 the opus main session cost in that arm: + +| | total | +| --- | ---: | +| no delegation | $1.359 | +| delegate, tier routed down | $1.675 | +| delegate, subagent inherits opus | $2.013 | + +So routing is roughly a 17% discount on a dispatch that was going to happen, and +a 23% surcharge on one that did not need to. Both numbers belong in the README +and now are. + +Two honest limits on the $2.013. It reprices measured tokens rather than +reporting a run, and a live opus subagent might well read less - fewer loops, +better first guesses - which would shrink the gap. The pricing model is not +guesswork though: applied to the sonnet arm it reproduces the billed $0.506 to +the cent, so the tier multiplier is right even if the token count would move. + +## The third case, which stopped the re-pin + +`subagent-answer-quality` exists because the wide case made haiku look like a +free 9%, and a tier that is cheaper on breadth is not therefore cheaper. It asks +`scout` two questions whose code contains a confident wrong answer: + +- `withRetryIntake` loops `maxRetries` times, and `maxRetries` is 3. It also + breaks the moment `error?.retryable === false`, and every stage error is built + with `retryable: cause?.retryable ?? false`. So `validateOrder` is called + **once**, and the file has a 3 sitting in it for anyone who reads one line. +- `applyTaxPolicy` looks up `TAX_POLICY[batch.outcome]` while the table is keyed + `onWarn`, `onFatal`, `onTimeout`, `onUnknown`. An outcome of `"warn"` misses, + falls through to `onUnknown: "abort"`, and returns **null** - not the + `continue` that the `onWarn` row appears to promise. That mismatch is a real + defect in the fixture and the question asks whether it was intended. + +Three runs per tier, four deterministic graders, no judge model: + +| Tier | Graders passed | Cost per run | +| --- | ---: | ---: | +| opus (reference, 1 run) | 4/4 | $0.715 | +| sonnet - the shipped pin | **12/12** | $0.212 | +| haiku | 11/12 | $0.103 | + +haiku's miss was substantive, not formatting: it answered **3**, having read the +`maxRetries` line and not the loop. Once in three runs, on the question the +fixture was built to trap. + +One miss in three runs of one question cannot pin a failure rate - 1/3 against +0/3 is not a distinguishable difference at this sample size, and anyone quoting +"fails a third of the time" from this table is quoting noise. What decides the +pin is not the rate but the shape of the failure: it is asymmetric. A cheap +correct answer saves cents; a cheap wrong one sends the main session back to read +the files itself, which costs more than the tier ever saved, and it arrives +looking exactly like a right one. That asymmetry does not need a tight estimate +to act on. + +So the pin stays sonnet, and what the two cases together support is a split +rather than a re-pin: breadth to `surveyor` (haiku), judgement to `scout` +(sonnet). Overriding `scout` downward with `model=haiku` would have been the +smaller diff and is what the first draft did - the plugin's own floor rule +forbids exactly that, and prescribes a pinned agent instead. + +The case runs the tier directly with `agent: scout` in frontmatter, which makes +the session *be* `scout` and grades what comes back. That is the instrument this +question needed all along: no main session to argue with about whether it really +delegated, and no way for it to contaminate the result by reading. + +## Next + +The mirror of the small case: a genuinely trivial question, where dispatching at +all is the failure being measured. And `forced-dispatch-tier` still needs the fix +described in its NOTES.md before it can price delegation at the session tier - +the Superpowers-by-default question is still open. diff --git a/evals/delegates-codebase-question/graders/answer-is-correct.md b/evals/delegates-codebase-question/graders/answer-is-correct.md new file mode 100644 index 0000000..f6905da --- /dev/null +++ b/evals/delegates-codebase-question/graders/answer-is-correct.md @@ -0,0 +1,10 @@ +--- +type: regex +target: last_message +pattern: 'refreshSession' +match: contains +--- + +Routing to a subagent is worthless if the answer comes back wrong, so the arms +are also scored on getting there. `refreshSession` in `src/auth/session.js` is +the only place the fixture extends a session; naming it is the whole answer. diff --git a/evals/delegates-codebase-question/graders/delegated-to-a-subagent.md b/evals/delegates-codebase-question/graders/delegated-to-a-subagent.md new file mode 100644 index 0000000..e2a18a8 --- /dev/null +++ b/evals/delegates-codebase-question/graders/delegated-to-a-subagent.md @@ -0,0 +1,13 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Agent|Task)"' +match: contains +--- + +The behavioural claim under test: a codebase question should leave the main +session rather than be ground out inline. Both arms can pass this - the baseline +has the built-in Explore and general-purpose agents - so the score difference is +a real measurement, not an artefact of the plugin owning the only subagent. + +The tool has been named both `Agent` and `Task` across builds; match either. diff --git a/evals/delegates-codebase-question/graders/scout-was-the-agent.md b/evals/delegates-codebase-question/graders/scout-was-the-agent.md new file mode 100644 index 0000000..9f4acf4 --- /dev/null +++ b/evals/delegates-codebase-question/graders/scout-was-the-agent.md @@ -0,0 +1,11 @@ +--- +type: regex +target: trace +pattern: '"subagent_type"\s*:\s*"(?:model-routing:)?scout"' +match: contains +--- + +Indicator, not a fair scored comparison: the baseline arm has no `scout` to +dispatch, so it fails this by construction. Read it only as confirmation that +the with-plugin arm picked the agent the routing guidance names, rather than +falling back to a generic one. diff --git a/evals/delegates-codebase-question/prompt.md b/evals/delegates-codebase-question/prompt.md new file mode 100644 index 0000000..6f6e491 --- /dev/null +++ b/evals/delegates-codebase-question/prompt.md @@ -0,0 +1,10 @@ +--- +name: delegates-codebase-question +tags: [routing, scout] +plugins: ["../.."] +runs: 3 +max_turns: 10 +timeout_seconds: 300 +--- + +Where does this project extend a session's lifetime? Name the file and the function. diff --git a/evals/evidence/README.md b/evals/evidence/README.md new file mode 100644 index 0000000..b4a4293 --- /dev/null +++ b/evals/evidence/README.md @@ -0,0 +1,48 @@ +# Evidence + +Every number quoted in [`../README.md`](../README.md) and in the repository +README comes from one of these files. They are here because the traces they were +distilled from are gitignored, and a reader who cannot rerun a stochastic model +against the same historical sample would otherwise have to take the tables on +trust. + +| File | What it backs | +| --- | --- | +| `small-case-opus.json` | `delegates-codebase-question`: the 3/3 against 0/3 delegation split, and the first per-tier cost table | +| `wide-case-sonnet-pin-and-baseline.json` | `traces-a-flow-end-to-end`, no-plugin and shipped-pin arms - the $1.359 and $1.675 rows, and the 74k/451k/6k subagent token counts everything downstream is repriced from | +| `wide-case-haiku-pin.json` | the same case with `agents/scout.md` edited to `model: haiku` - the $1.239 row | +| `quality-case-sonnet.json` | `subagent-answer-quality` at sonnet: 12/12 | +| `quality-case-haiku.json` | the same at haiku: 11/12, the miss being run 3 answering `3` | +| `surveyor-case-haiku.json` | `surveyor-traces-the-chain` on the shipped haiku pin: 3/3 at $0.156 | +| `plugin-details.txt` | the always-on token cost, with the Claude Code version that produced it | + +Each aggregate carries per-run cost, per-model usage, and every grader verdict, +so the published means can be recomputed rather than believed. What they cannot +give you is the raw transcripts - those stay local, both for size and because +they contain absolute paths from the machine that ran them. + +## The one thing worth checking yourself + +The quality case rests on two claims about what the fixture does at runtime, and +the fixture is generated, so you can verify them in seconds rather than trusting +a reading of it: + +```sh +node evals/fixtures/order-service.gen.mjs +node --input-type=module -e " +const base='file://$PWD/evals/fixtures/order-service/src/'; +const {withRetryIntake}=await import(base+'intake/retry.js'); +const {errorsValidation}=await import(base+'validation/errors.js'); +let calls=0; +try { await withRetryIntake(async()=>{calls++; throw errorsValidation(new Error('x'),{id:'o'})},3) } catch {} +console.log('calls with a non-retryable error:', calls); +const tax=await import(base+'tax/index.js'); +console.log('applyTaxPolicy warn/strict:', tax.applyTaxPolicy({outcome:'warn'},{lenient:false})); +" +``` + +It prints `1` and `null`. Both were asserted from reading the code first, and an +earlier version of the generator emitted a fixture that could not link at all - +`import { retryIntake }` against `export withRetryIntake` - so the assertions +were true of code that never ran. Executing them is the only version of that +check worth having. diff --git a/evals/evidence/plugin-details.txt b/evals/evidence/plugin-details.txt new file mode 100644 index 0000000..3766009 --- /dev/null +++ b/evals/evidence/plugin-details.txt @@ -0,0 +1,29 @@ +model-routing 0.14.0 + Tiered model routing for token economy: the strongest model thinks, cheaper models grind. Ready-made subagents (sonnet scout, implementer and E2E runner, haiku test-runner and verifier, opus reviewer) plus a routing skill and dispatch stats. + Source: model-routing@model-routing + +Component inventory + Skills (2) model-routing, stats + Agents (7) e2e-runner, implementer, reviewer, scout, surveyor, test-runner, verifier + Hooks (2) SessionStart, PostToolUse (harness-only — no model context cost) + MCP servers (0) + LSP servers (0) + +Projected token cost + Always-on: ~700 tok added to every session + +Per-component (rounded) + component always-on on-invoke + model-routing ~100 ~9.3k + e2e-runner ~60 ~350 + implementer ~80 ~710 + reviewer ~60 ~420 + scout ~60 ~610 + surveyor ~110 ~760 + test-runner ~70 ~330 + verifier ~80 ~640 + stats ~80 ~410 + + On-invoke cost is paid each time a skill or agent fires. + Token counts are estimates and may differ from actual usage. +2.1.236 (Claude Code) diff --git a/evals/evidence/quality-case-haiku.json b/evals/evidence/quality-case-haiku.json new file mode 100644 index 0000000..6fb725d --- /dev/null +++ b/evals/evidence/quality-case-haiku.json @@ -0,0 +1,115 @@ +{ + "model": "haiku", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "subagent-answer-quality", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 0.12713370000000002, + "durationMs": 42717, + "turns": 14, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.12713370000000002, + "inputTokens": 5201, + "cacheReadInputTokens": 279557, + "cacheCreationInputTokens": 38196, + "outputTokens": 3517 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.09728060000000001, + "durationMs": 50436, + "turns": 22, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.09728060000000001, + "inputTokens": 100, + "cacheReadInputTokens": 381276, + "cacheCreationInputTokens": 16109, + "outputTokens": 5367 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.08500869999999999, + "durationMs": 59118, + "turns": 12, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.08500869999999999, + "inputTokens": 60, + "cacheReadInputTokens": 213647, + "cacheCreationInputTokens": 14292, + "outputTokens": 7000 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": false + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/quality-case-sonnet.json b/evals/evidence/quality-case-sonnet.json new file mode 100644 index 0000000..42b524a --- /dev/null +++ b/evals/evidence/quality-case-sonnet.json @@ -0,0 +1,115 @@ +{ + "model": "sonnet", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "subagent-answer-quality", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 0.2570082, + "durationMs": 47556, + "turns": 13, + "modelUsage": { + "claude-sonnet-5": { + "costUSD": 0.2570082, + "inputTokens": 24, + "cacheReadInputTokens": 449914, + "cacheCreationInputTokens": 12132, + "outputTokens": 3278 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.21288810000000002, + "durationMs": 80459, + "turns": 14, + "modelUsage": { + "claude-sonnet-5": { + "costUSD": 0.21288810000000002, + "inputTokens": 20, + "cacheReadInputTokens": 379817, + "cacheCreationInputTokens": 7233, + "outputTokens": 3699 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.1648107, + "durationMs": 72206, + "turns": 11, + "modelUsage": { + "claude-sonnet-5": { + "costUSD": 0.1648107, + "inputTokens": 14, + "cacheReadInputTokens": 259949, + "cacheCreationInputTokens": 6549, + "outputTokens": 3166 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/small-case-opus.json b/evals/evidence/small-case-opus.json new file mode 100644 index 0000000..c45f88d --- /dev/null +++ b/evals/evidence/small-case-opus.json @@ -0,0 +1,121 @@ +{ + "model": "opus", + "runs": 3, + "arms": [ + "with", + "without" + ], + "cases": [ + { + "name": "delegates-codebase-question", + "arms": { + "with": [ + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + }, + { + "grader": "scout-was-the-agent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + }, + { + "grader": "scout-was-the-agent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + }, + { + "grader": "scout-was-the-agent", + "pass": true + } + ] + } + ], + "without": [ + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + }, + { + "grader": "scout-was-the-agent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + }, + { + "grader": "scout-was-the-agent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + }, + { + "grader": "scout-was-the-agent", + "pass": false + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/surveyor-case-haiku.json b/evals/evidence/surveyor-case-haiku.json new file mode 100644 index 0000000..1ba2a77 --- /dev/null +++ b/evals/evidence/surveyor-case-haiku.json @@ -0,0 +1,79 @@ +{ + "model": "sonnet", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "surveyor-traces-the-chain", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 0.1793826, + "durationMs": 54029, + "turns": 20, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.1793826, + "inputTokens": 162, + "cacheReadInputTokens": 798076, + "cacheCreationInputTokens": 40454, + "outputTokens": 3701 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.14531750000000002, + "durationMs": 45688, + "turns": 17, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.14531750000000002, + "inputTokens": 285, + "cacheReadInputTokens": 614175, + "cacheCreationInputTokens": 33945, + "outputTokens": 3145 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.1434974, + "durationMs": 61965, + "turns": 26, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.1434974, + "inputTokens": 357, + "cacheReadInputTokens": 877604, + "cacheCreationInputTokens": 16650, + "outputTokens": 4416 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/wide-case-haiku-pin.json b/evals/evidence/wide-case-haiku-pin.json new file mode 100644 index 0000000..1ad9796 --- /dev/null +++ b/evals/evidence/wide-case-haiku-pin.json @@ -0,0 +1,112 @@ +{ + "model": "opus", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "traces-a-flow-end-to-end", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 1.1419868999999998, + "durationMs": 257314, + "turns": 19, + "modelUsage": { + "claude-opus-5": { + "costUSD": 0.9878954999999999, + "inputTokens": 28, + "cacheReadInputTokens": 605231, + "cacheCreationInputTokens": 35424, + "outputTokens": 13236 + }, + "claude-haiku-4-5-20251001": { + "costUSD": 0.1540914, + "inputTokens": 162, + "cacheReadInputTokens": 583094, + "cacheCreationInputTokens": 58540, + "outputTokens": 4489 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.2602386, + "durationMs": 291175, + "turns": 22, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.0320255, + "inputTokens": 24, + "cacheReadInputTokens": 546781, + "cacheCreationInputTokens": 42319, + "outputTokens": 13413 + }, + "claude-haiku-4-5-20251001": { + "costUSD": 0.22821309999999997, + "inputTokens": 234, + "cacheReadInputTokens": 1211291, + "cacheCreationInputTokens": 53708, + "outputTokens": 7943 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.31517195, + "durationMs": 332643, + "turns": 24, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.1048195, + "inputTokens": 24, + "cacheReadInputTokens": 538489, + "cacheCreationInputTokens": 40183, + "outputTokens": 17345 + }, + "claude-haiku-4-5-20251001": { + "costUSD": 0.21035245000000002, + "inputTokens": 218, + "cacheReadInputTokens": 1070882, + "cacheCreationInputTokens": 50537, + "outputTokens": 7975 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/wide-case-sonnet-pin-and-baseline.json b/evals/evidence/wide-case-sonnet-pin-and-baseline.json new file mode 100644 index 0000000..7f1319c --- /dev/null +++ b/evals/evidence/wide-case-sonnet-pin-and-baseline.json @@ -0,0 +1,211 @@ +{ + "model": "opus", + "runs": 3, + "arms": [ + "with", + "without" + ], + "cases": [ + { + "name": "traces-a-flow-end-to-end", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 1.5592256, + "durationMs": 190898, + "turns": 14, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.0736555, + "inputTokens": 24, + "cacheReadInputTokens": 469291, + "cacheCreationInputTokens": 58509, + "outputTokens": 10152 + }, + "claude-sonnet-5": { + "costUSD": 0.4855701, + "inputTokens": 36, + "cacheReadInputTokens": 460632, + "cacheCreationInputTokens": 65130, + "outputTokens": 6869 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.9984075499999998, + "durationMs": 300231, + "turns": 28, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.3613790000000001, + "inputTokens": 34, + "cacheReadInputTokens": 791958, + "cacheCreationInputTokens": 46058, + "outputTokens": 20186 + }, + "claude-sonnet-5": { + "costUSD": 0.63702855, + "inputTokens": 18, + "cacheReadInputTokens": 341761, + "cacheCreationInputTokens": 117811, + "outputTokens": 6177 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.4679628000000002, + "durationMs": 291462, + "turns": 24, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.0716640000000002, + "inputTokens": 30, + "cacheReadInputTokens": 661468, + "cacheCreationInputTokens": 38838, + "outputTokens": 14096 + }, + "claude-sonnet-5": { + "costUSD": 0.3962988, + "inputTokens": 38, + "cacheReadInputTokens": 550441, + "cacheCreationInputTokens": 38338, + "outputTokens": 5819 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + } + ], + "without": [ + { + "exitCode": 0, + "costUsd": 1.3390605, + "durationMs": 277983, + "turns": 29, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.000629, + "inputTokens": 549, + "cacheReadInputTokens": 0, + "cacheCreationInputTokens": 0, + "outputTokens": 16 + }, + "claude-opus-5": { + "costUSD": 1.3384315, + "inputTokens": 107, + "cacheReadInputTokens": 782453, + "cacheCreationInputTokens": 48142, + "outputTokens": 18610 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.3821754999999998, + "durationMs": 272155, + "turns": 29, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.000629, + "inputTokens": 549, + "cacheReadInputTokens": 0, + "cacheCreationInputTokens": 0, + "outputTokens": 16 + }, + "claude-opus-5": { + "costUSD": 1.3815464999999998, + "inputTokens": 263, + "cacheReadInputTokens": 897893, + "cacheCreationInputTokens": 48361, + "outputTokens": 17907 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.355927, + "durationMs": 246303, + "turns": 28, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.000629, + "inputTokens": 549, + "cacheReadInputTokens": 0, + "cacheCreationInputTokens": 0, + "outputTokens": 16 + }, + "claude-opus-5": { + "costUSD": 1.355298, + "inputTokens": 3831, + "cacheReadInputTokens": 863726, + "cacheCreationInputTokens": 46123, + "outputTokens": 17722 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/fixtures/mini-app/README.md b/evals/fixtures/mini-app/README.md new file mode 100644 index 0000000..b04898a --- /dev/null +++ b/evals/fixtures/mini-app/README.md @@ -0,0 +1,5 @@ +# mini-app + +Fixture project for the routing evals. Deliberately small, deliberately boring: +the point is that answering a "where is X" question about it requires reading a +few files, not that the code is interesting. diff --git a/evals/fixtures/mini-app/src/api/client.js b/evals/fixtures/mini-app/src/api/client.js new file mode 100644 index 0000000..bee2702 --- /dev/null +++ b/evals/fixtures/mini-app/src/api/client.js @@ -0,0 +1,9 @@ +import { refreshSession } from "../auth/session.js"; + +export async function request(session, path) { + const live = refreshSession(session); + const response = await fetch(`https://example.invalid${path}`, { + headers: { authorization: `Bearer ${live.token}` }, + }); + return { session: live, body: await response.json() }; +} diff --git a/evals/fixtures/mini-app/src/api/retry.js b/evals/fixtures/mini-app/src/api/retry.js new file mode 100644 index 0000000..a60c338 --- /dev/null +++ b/evals/fixtures/mini-app/src/api/retry.js @@ -0,0 +1,15 @@ +import { request } from "./client.js"; + +const MAX_ATTEMPTS = 3; + +export async function requestWithRetry(session, path) { + let last; + for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { + try { + return await request(session, path); + } catch (error) { + last = error; + } + } + throw last; +} diff --git a/evals/fixtures/mini-app/src/auth/session.js b/evals/fixtures/mini-app/src/auth/session.js new file mode 100644 index 0000000..abb6c4d --- /dev/null +++ b/evals/fixtures/mini-app/src/auth/session.js @@ -0,0 +1,13 @@ +import { now } from "../util/clock.js"; + +const SESSION_TTL_MS = 15 * 60 * 1000; + +export function isExpired(session) { + return now() - session.issuedAt > SESSION_TTL_MS; +} + +// The only place a session's lifetime is extended. +export function refreshSession(session) { + if (!isExpired(session)) return session; + return { ...session, issuedAt: now() }; +} diff --git a/evals/fixtures/mini-app/src/util/clock.js b/evals/fixtures/mini-app/src/util/clock.js new file mode 100644 index 0000000..05557f7 --- /dev/null +++ b/evals/fixtures/mini-app/src/util/clock.js @@ -0,0 +1,9 @@ +let offsetMs = 0; + +export function now() { + return Date.now() + offsetMs; +} + +export function setOffset(ms) { + offsetMs = ms; +} diff --git a/evals/fixtures/order-service.gen.mjs b/evals/fixtures/order-service.gen.mjs new file mode 100644 index 0000000..8562c91 --- /dev/null +++ b/evals/fixtures/order-service.gen.mjs @@ -0,0 +1,434 @@ +#!/usr/bin/env node +// Generates the `order-service` fixture: a pipeline whose shape can only be +// learned by reading it. Committed as a generator rather than as files because +// the fixture runs to a few hundred kilobytes and every user of this plugin +// would otherwise install it. +// +// Two properties matter, and both are deliberate. +// +// The question is broad - trace the flow end to end - because a narrow one is +// answered by a single grep, and a grep costs the same whether or not it happens +// in the expensive session. Delegation can only pay where the reading is wide. +// +// The stage modules are fat. Tracing a chain of small files is cheap no matter +// how many files there are, so each hop has to cost something to read: the +// handoff sits at the bottom of a few hundred lines of plausible logic, and the +// helpers name other stages so that grepping for a stage name is noisy. +// +// Deterministic: same output every run, no randomness, no network. + +import { mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const OUT = join(dirname(fileURLToPath(import.meta.url)), "order-service"); + +const STAGES = [ + ["intake", "acceptOrder", "validation", "the raw payload from the till"], + ["validation", "validateOrder", "pricing", "line items against the catalog"], + ["pricing", "priceOrder", "discount", "unit prices before discounts"], + ["discount", "applyDiscounts", "tax", "campaign and loyalty reductions"], + ["tax", "applyTax", "deposit", "VAT groups per line"], + ["deposit", "addDeposits", "payment", "returnable packaging charges"], + ["payment", "capturePayment", "tips", "the authorized amount"], + ["tips", "splitTips", "fiscal", "gratuity across the shift"], + ["fiscal", "signReceipt", "receipt", "the signature from the TSE"], + ["receipt", "renderReceipt", "export", "the printable document"], + ["export", "exportOrder", "archive", "the audit record"], + ["archive", "archiveOrder", null, "the closed order"], +]; + +const HELPERS = ["normalize", "guard", "metrics", "errors", "mapping", "retry"]; + +rmSync(OUT, { recursive: true, force: true }); + +for (const [stage, fn, next, subject] of STAGES) { + const dir = join(OUT, "src", stage); + mkdirSync(dir, { recursive: true }); + const nextStage = STAGES.find((s) => s[0] === next); + writeFileSync(join(dir, "index.js"), stageSource(stage, fn, subject, nextStage), "utf-8"); + for (const helper of HELPERS) writeFileSync(join(dir, `${helper}.js`), helperSource(helper, stage), "utf-8"); +} + +writeFileSync( + join(OUT, "README.md"), + `# order-service + +Fixture for the routing evals. A generated pipeline: every stage lives in its own +directory and hands off to exactly one other stage, and no file anywhere states +the order of the stages. Working it out means reading them. +`, + "utf-8", +); + +const files = STAGES.length * (HELPERS.length + 1) + 1; +console.log(`Generated ${files} modules in ${OUT}`); + +function cap(word) { + return word[0].toUpperCase() + word.slice(1); +} + +// Most helpers export ``; retry is the one that does not. +function helperExport(helper, S) { + return helper === "retry" ? `withRetry${S}` : `${helper}${S}`; +} + +// Every other stage name, so that grepping for one stage hits many files and +// the chain still has to be read rather than pattern-matched. +function otherStages(stage) { + return STAGES.map((s) => s[0]).filter((s) => s !== stage); +} + +function stageSource(stage, fn, subject, nextStage) { + const S = cap(stage); + const imports = [ + // The retry helper exports `withRetry`, not `retry` - importing + // it by the pattern name left every stage module unlinkable, which made the + // fixture unrunnable while still reading correctly. + ...HELPERS.map((h) => `import { ${helperExport(h, S)} } from "./${h}.js";`), + nextStage ? `import { ${nextStage[1]} } from "../${nextStage[0]}/index.js";` : "", + ] + .filter(Boolean) + .join("\n"); + + return `${imports} + +// This module owns ${subject}. Everything below it is stage-local: the handoff +// to the next stage is the last thing in the file. + +const ${stage.toUpperCase()}_LIMITS = { + maxLines: 500, + maxQuantityPerLine: 999, + maxTotalMinor: 100_000_00, + maxRetries: 3, + softTimeoutMs: 2_500, +}; + +const ${stage.toUpperCase()}_FEATURES = ${JSON.stringify( + Object.fromEntries(otherStages(stage).map((other, i) => [`${other}Aware`, i % 3 === 0])), + null, + 2, + )}; + +${block(stage, "rules")} + +${block(stage, "totals")} + +${block(stage, "audit")} + +${block(stage, "context")} + +${block(stage, "diff")} + +${block(stage, "batching")} + +${block(stage, "policy")} + +${block(stage, "serialize")} + +export async function ${fn}(order, context) { + const checked = guard${S}(order, context); + const shaped = normalize${S}(checked); + const violations = collect${S}Violations(shaped, context); + if (violations.length > 0) throw errors${S}(new Error(violations.join("; ")), shaped); + + const totals = compute${S}Totals(shaped); + const enriched = { ...shaped, totals, audit: build${S}Audit(shaped, context) }; + metrics${S}(enriched, context); + + const batches = split${S}Batches(enriched); + const settled = batches.map((batch) => apply${S}Policy(batch, context)); + const merged = merge${S}Diff(enriched, settled); + + try { +${ + nextStage + ? ` return await withRetry${S}(() => ${nextStage[1]}(merged, context), ${stage.toUpperCase()}_LIMITS.maxRetries);` + : " return serialize" + S + "(merged);" +} + } catch (cause) { + throw errors${S}(cause, merged); + } +} +`; +} + +// A bank of plausible stage-local logic. Each block is 30-60 lines so that a +// stage module costs something real to read. +function block(stage, kind) { + const S = cap(stage); + const U = stage.toUpperCase(); + const banks = { + rules: `const ${U}_RULES = [ + { code: "empty-lines", severity: "fatal", test: (order) => (order.lines ?? []).length === 0 }, + { code: "too-many-lines", severity: "fatal", test: (order) => (order.lines ?? []).length > ${U}_LIMITS.maxLines }, + { code: "negative-quantity", severity: "fatal", test: (order) => (order.lines ?? []).some((l) => l.quantity < 0) }, + { code: "quantity-ceiling", severity: "warn", test: (order) => (order.lines ?? []).some((l) => l.quantity > ${U}_LIMITS.maxQuantityPerLine) }, + { code: "missing-sku", severity: "fatal", test: (order) => (order.lines ?? []).some((l) => !l.sku) }, + { code: "duplicate-sku", severity: "warn", test: (order) => { + const seen = new Set(); + return (order.lines ?? []).some((l) => (seen.has(l.sku) ? true : (seen.add(l.sku), false))); + } }, + { code: "no-currency", severity: "fatal", test: (order) => !order.currency }, + { code: "stale-order", severity: "warn", test: (order) => Number(order.openedAt ?? 0) > 0 && Date.now() - order.openedAt > 86_400_000 }, +]; + +export function collect${S}Violations(order, context) { + const violations = []; + for (const rule of ${U}_RULES) { + let failed = false; + try { + failed = rule.test(order); + } catch { + failed = true; + } + if (!failed) continue; + if (rule.severity === "warn" && context?.lenient) continue; + violations.push(\`\${rule.code}\`); + } + return violations; +} + +export function describe${S}Rules() { + return ${U}_RULES.map((rule) => ({ code: rule.code, severity: rule.severity })); +}`, + + totals: `export function compute${S}Totals(order) { + const lines = order.lines ?? []; + let net = 0; + let gross = 0; + let discount = 0; + let quantity = 0; + + for (const line of lines) { + const lineNet = Math.round(line.unitPrice * line.quantity * 100) / 100; + const lineDiscount = Math.round((line.discountMinor ?? 0)) / 100; + net += lineNet; + discount += lineDiscount; + gross += lineNet - lineDiscount + (line.taxMinor ?? 0) / 100; + quantity += line.quantity; + } + + return { + lineCount: lines.length, + quantity, + net: round2(net), + discount: round2(discount), + gross: round2(gross), + average: lines.length === 0 ? 0 : round2(net / lines.length), + }; +} + +function round2(value) { + return Math.round(value * 100) / 100; +}`, + + audit: `export function build${S}Audit(order, context) { + return { + stage: "${stage}", + orderId: order.id, + tenantId: context?.tenantId ?? null, + operator: context?.operator?.id ?? null, + device: context?.device?.serial ?? null, + lineCount: (order.lines ?? []).length, + features: Object.entries(${U}_FEATURES) + .filter(([, on]) => on) + .map(([name]) => name), + }; +} + +export function summarize${S}Audit(entries) { + const byOperator = new Map(); + for (const entry of entries ?? []) { + const key = entry.operator ?? "unknown"; + byOperator.set(key, (byOperator.get(key) ?? 0) + 1); + } + return [...byOperator.entries()].map(([operator, count]) => ({ operator, count })); +}`, + + context: `export function derive${S}Context(context) { + return { + tenantId: context?.tenantId ?? null, + lenient: Boolean(context?.lenient), + locale: context?.locale ?? "de-DE", + timezone: context?.timezone ?? "Europe/Berlin", + softTimeoutMs: context?.softTimeoutMs ?? ${U}_LIMITS.softTimeoutMs, + }; +} + +export function assert${S}Context(context) { + const derived = derive${S}Context(context); + if (!derived.tenantId) throw new Error("${stage}: context carries no tenant"); + if (derived.softTimeoutMs <= 0) throw new Error("${stage}: soft timeout must be positive"); + return derived; +}`, + + diff: `export function merge${S}Diff(base, patches) { + let merged = { ...base }; + for (const patch of patches ?? []) { + if (!patch) continue; + merged = { + ...merged, + ...patch, + lines: mergeLines(merged.lines ?? [], patch.lines ?? []), + totals: { ...(merged.totals ?? {}), ...(patch.totals ?? {}) }, + }; + } + return merged; +} + +function mergeLines(left, right) { + if (right.length === 0) return left; + const bySku = new Map(left.map((line) => [line.sku, line])); + for (const line of right) { + const existing = bySku.get(line.sku); + bySku.set(line.sku, existing ? { ...existing, ...line } : line); + } + return [...bySku.values()]; +}`, + + batching: `export function split${S}Batches(order) { + const lines = order.lines ?? []; + if (lines.length <= 25) return [order]; + + const batches = []; + for (let i = 0; i < lines.length; i += 25) { + batches.push({ ...order, lines: lines.slice(i, i + 25), batchIndex: batches.length }); + } + return batches; +} + +export function joinBatches(batches) { + return (batches ?? []).reduce( + (acc, batch) => ({ ...batch, lines: [...(acc.lines ?? []), ...(batch.lines ?? [])] }), + {}, + ); +}`, + + policy: `const ${U}_POLICY = { + onWarn: "continue", + onFatal: "abort", + onTimeout: "retry", + onUnknown: "abort", +}; + +export function apply${S}Policy(batch, context) { + const decision = ${U}_POLICY[batch?.outcome ?? "onUnknown"] ?? ${U}_POLICY.onUnknown; + if (decision === "abort" && !context?.lenient) return null; + return { ...batch, decision }; +} + +export function ${stage}PolicyFor(outcome) { + return ${U}_POLICY[outcome] ?? ${U}_POLICY.onUnknown; +}`, + + serialize: `export function serialize${S}(order) { + return { + id: order.id, + stage: "${stage}", + currency: order.currency ?? "EUR", + totals: order.totals ?? null, + lines: (order.lines ?? []).map((line) => ({ + sku: line.sku, + quantity: line.quantity, + unitPrice: line.unitPrice, + taxMinor: line.taxMinor ?? 0, + discountMinor: line.discountMinor ?? 0, + })), + audit: order.audit ?? null, + }; +} + +export function deserialize${S}(payload) { + return { ...payload, lines: payload.lines ?? [], stage: "${stage}" }; +}`, + }; + return banks[kind]; +} + +function helperSource(helper, stage) { + const S = cap(stage); + const others = otherStages(stage); + const bodies = { + normalize: `// Shared shaping. The stages that consume this output are +// ${others.slice(0, 4).join(", ")}, so keep the field names stable. +export function normalize${S}(order) { + const lines = (order.lines ?? []).map((line) => ({ + ...line, + sku: String(line.sku ?? "").trim(), + quantity: Number(line.quantity ?? 1), + unitPrice: Math.round(Number(line.unitPrice ?? 0) * 100) / 100, + taxMinor: Math.round(Number(line.taxMinor ?? 0)), + discountMinor: Math.round(Number(line.discountMinor ?? 0)), + })); + return { ...order, lines, stage: "${stage}", currency: order.currency ?? "EUR" }; +}`, + guard: `const REQUIRED = ["id", "lines"]; + +// Mirrors the checks the ${others[0]} and ${others[1]} stages make on their own input. +export function guard${S}(order, context) { + for (const field of REQUIRED) { + if (order?.[field] == null) throw new Error(\`${stage}: missing \${field}\`); + } + if (context?.tenantId == null) throw new Error("${stage}: no tenant in context"); + if (!Array.isArray(order.lines)) throw new Error("${stage}: lines must be an array"); + return order; +}`, + metrics: `let counter = 0; + +// Counter names are compared against the ${others[2]} stage in dashboards. +export function metrics${S}(order, context) { + counter += 1; + context?.telemetry?.record?.({ stage: "${stage}", orderId: order.id, seq: counter }); + return counter; +} + +export function reset${S}Metrics() { + counter = 0; +}`, + errors: `export function errors${S}(cause, order) { + const error = new Error(\`${stage} failed for order \${order?.id ?? "unknown"}\`); + error.cause = cause; + error.stage = "${stage}"; + error.retryable = cause?.retryable ?? false; + return error; +} + +// The ${others[3]} stage unwraps these, so the shape is load-bearing. +export function unwrap${S}(error) { + return { stage: error?.stage ?? "${stage}", retryable: Boolean(error?.retryable), cause: error?.cause ?? null }; +}`, + mapping: `// Field names as they leave ${stage}. The ${others[4]} stage reads the same keys. +const FIELDS = { + id: "orderId", + currency: "currencyCode", + totals: "totalsBlock", + audit: "auditTrail", +}; + +export function mapping${S}(order) { + const out = {}; + for (const [from, to] of Object.entries(FIELDS)) out[to] = order?.[from] ?? null; + return out; +} + +export function unmapping${S}(payload) { + const out = {}; + for (const [from, to] of Object.entries(FIELDS)) out[from] = payload?.[to] ?? null; + return out; +}`, + retry: `export async function withRetry${S}(operation, attempts) { + let last; + for (let attempt = 0; attempt < attempts; attempt++) { + try { + return await operation(); + } catch (error) { + last = error; + if (error?.retryable === false) break; + } + } + throw last; +}`, + }; + return `${bodies[helper]}\n`; +} diff --git a/evals/forced-dispatch-tier/NOTES.md b/evals/forced-dispatch-tier/NOTES.md new file mode 100644 index 0000000..290e2f5 --- /dev/null +++ b/evals/forced-dispatch-tier/NOTES.md @@ -0,0 +1,73 @@ +# forced-dispatch-tier - unfinished, do not read numbers off it yet + +## What it is for + +The unforced cases cannot answer "is delegating at the session tier - what +Superpowers does by default - more expensive than delegating at a lower one", +because whether a session delegates at all is a coin it flips: observed 3/3, +3/3, 2/3, 2/3 and 0/3 across otherwise identical batches. This case takes the +decision away and holds everything else identical to +`traces-a-flow-end-to-end`, so the only variable left is the subagent's tier. + +Intended configurations, same prompt and follow-ups throughout: + +| Configuration | What it stands for | +| --- | --- | +| no plugin, unforced (already measured, $1.359) | ordinary work, no delegation | +| no plugin, forced | subagent inherits the session model - Superpowers | +| plugin, `scout` pinned sonnet | what ships today | +| plugin, `scout` pinned haiku | the change the wide case argues for | + +## Where it stands + +Isolation works and took three attempts. Every attempt is worth knowing about, +because each looked like it had worked: + +1. Asking in the system prompt: the session read the files anyway. +2. Denying `Read`, `Grep`, `Glob`: the session read the whole codebase through + Bash instead - `cat`, `grep -n`, `sed -n`. Three opus runs were spent before + the tool census showed it, and their cost figures were meaningless. +3. Denying the shells too: the session pulled a deferred `Read` back in through + `ToolSearch`. +4. An allowlist (`tools: [Task, Agent, TodoWrite]`) holds. Nothing to read with, + nothing to recover. + +The `main-session-read-nothing` grader now covers the shells and is the thing to +check before believing any cost number from this case. + +## The open problem, and the wrong diagnosis it was first given + +With the allowlist in place the answers come back **wrong**, on haiku and on opus +alike. This was first written up as "one dispatch and no way to check the result", +i.e. as evidence that forcing delegation costs quality. That was wrong, and the +`subagent-answer-quality` case is what exposed it: **`--tools` propagates to +subagents**. The allowlist did not merely stop the main session from reading - it +starved `scout` of file tools too, so the dispatches came back with nothing and +the main session correctly refused to invent an answer. One transcript says so in +as many words: "two scout runs returned zero real tool executions". + +So the isolation problem is worse than attempt 4 suggested. Every mechanism tried +so far either fails to stop the main session or disables the subagent as well: + +| Mechanism | Main session stopped | Subagent still able to read | +| --- | --- | --- | +| System-prompt request | no | yes | +| Deny Read/Grep/Glob | no - moves to Bash | untested | +| Deny the shells too | no - ToolSearch reloads Read | apparently not | +| `--tools` allowlist | yes | **no** | + +What is needed is a per-session tool restriction that does not reach the agents +that session dispatches, and nothing tried yet does that. Candidates, untested: +a settings-file `permissions.deny` passed with `--settings` (deny rules may scope +differently from `--tools`), or accepting main-session reads and instead pricing +the arms by which model did the reading, from the per-model split the runner +already prints. + +Until then this case cannot price delegation at the session tier, and the +Superpowers-by-default question stays open. + +## Also open + +Nothing about the `scout` pin: `subagent-answer-quality` settled that in this +same branch. It stays sonnet, and breadth work goes to `surveyor` (haiku) +instead. diff --git a/evals/forced-dispatch-tier/followup-1.md b/evals/forced-dispatch-tier/followup-1.md new file mode 100644 index 0000000..4f06479 --- /dev/null +++ b/evals/forced-dispatch-tier/followup-1.md @@ -0,0 +1 @@ +If the payment stage throws an error that is not retryable, what does the order end up as, and which stage decides that? Answer from the code, not from convention. diff --git a/evals/forced-dispatch-tier/followup-2.md b/evals/forced-dispatch-tier/followup-2.md new file mode 100644 index 0000000..2d4e7eb --- /dev/null +++ b/evals/forced-dispatch-tier/followup-2.md @@ -0,0 +1 @@ +I want to add a service charge that applies after discounts but before VAT. Which stage should own it, and what would it have to hand the next stage so nothing downstream breaks? diff --git a/evals/forced-dispatch-tier/followup-3.md b/evals/forced-dispatch-tier/followup-3.md new file mode 100644 index 0000000..fb70a67 --- /dev/null +++ b/evals/forced-dispatch-tier/followup-3.md @@ -0,0 +1 @@ +Which stages split their work into batches before handing off, and what is the batch size? List them. diff --git a/evals/forced-dispatch-tier/graders/answer-is-correct.md b/evals/forced-dispatch-tier/graders/answer-is-correct.md new file mode 100644 index 0000000..a22ae40 --- /dev/null +++ b/evals/forced-dispatch-tier/graders/answer-is-correct.md @@ -0,0 +1,12 @@ +--- +type: regex +target: last_message +pattern: 'intake[\s\S]*validation[\s\S]*pricing[\s\S]*discount[\s\S]*tax[\s\S]*deposit[\s\S]*payment[\s\S]*tips[\s\S]*fiscal[\s\S]*receipt[\s\S]*export[\s\S]*archive' +flags: 'i' +match: contains +--- + +All twelve stages in the right order, same as the unforced case. It matters more +here: the main session never reads a file, so this scores whether the subagent's +tier is capable of the work, not just whether it is cheap. A tier that saves +money and gets the order wrong has saved nothing. diff --git a/evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md b/evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md new file mode 100644 index 0000000..b4e2a3f --- /dev/null +++ b/evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md @@ -0,0 +1,11 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Agent|Task)"' +match: contains +--- + +Here this is a precondition rather than a finding: the system prompt orders the +delegation, so a run that fails this grader did not do the thing being priced and +its cost figure means nothing. Check it before reading any money number from +this case. diff --git a/evals/forced-dispatch-tier/graders/main-session-read-nothing.md b/evals/forced-dispatch-tier/graders/main-session-read-nothing.md new file mode 100644 index 0000000..e7cad92 --- /dev/null +++ b/evals/forced-dispatch-tier/graders/main-session-read-nothing.md @@ -0,0 +1,19 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Read|Grep|Glob|Bash|PowerShell)"' +match: not_contains +--- + +The main session must not read the project itself, or the run is a blend of both +strategies and its cost lands between the two things being compared - worse than +a missing data point, because it looks like one. + +Banning Read, Grep and Glob alone does not achieve this. The first attempt did +exactly that and the session simply read the whole codebase through Bash - +`cat`, `grep -n`, `sed -n` - and the cost figures from those runs were +meaningless. Shell counts as reading, so the shells are banned too and this +grader watches for them. + +Subagent tool calls do not appear in the parent transcript, so this only sees +what the main session did, which is exactly the scope it needs. diff --git a/evals/forced-dispatch-tier/prompt.md b/evals/forced-dispatch-tier/prompt.md new file mode 100644 index 0000000..10062c5 --- /dev/null +++ b/evals/forced-dispatch-tier/prompt.md @@ -0,0 +1,13 @@ +--- +name: forced-dispatch-tier +tags: [routing, tier, wide-reading, unfinished] +plugins: ["../.."] +fixture: order-service +runs: 3 +max_turns: 20 +timeout_seconds: 600 +append_system_prompt: 'Delegate all codebase exploration to a subagent. Do not read, grep, or glob project files in the main session - dispatch an agent to do it and work from what it reports back.' +tools: [Task, Agent, TodoWrite] +--- + +An order enters this service at intake and ends up archived. Trace the whole path: list every stage it passes through, in order, from intake to archive. diff --git a/evals/run-local.mjs b/evals/run-local.mjs new file mode 100644 index 0000000..0426d39 --- /dev/null +++ b/evals/run-local.mjs @@ -0,0 +1,451 @@ +#!/usr/bin/env node +// Local stand-in for `claude plugin eval`, which is early access and gated for +// this account. It reads the SAME case layout the real harness reads +// (evals//prompt.md + graders/*.md), runs each case with and without the +// plugin, and scores the regex graders. When the gate opens, delete this file +// and run `claude plugin eval model-routing --ablation with-without` instead. +// +// Only the subset of the format this suite actually uses is supported: regex +// graders over `trace` and `last_message`. Anything else is reported as skipped +// rather than silently scored. + +import { execFileSync, spawn } from "node:child_process"; +import { cpSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; + +const EVALS_DIR = dirname(fileURLToPath(import.meta.url)); +const PLUGIN_ROOT = resolve(EVALS_DIR, ".."); +const FIXTURES_DIR = join(EVALS_DIR, "fixtures"); + +// Frontmatter this runner reads: name, tags, fixture, agent, runs, +// timeout_seconds, tools, append_system_prompt. `max_turns` is part of the real +// harness's format and is carried in the case files, but the `claude` CLI has no +// flag for it - only `timeout_seconds` actually bounds a run here. +const args = parseArgs(process.argv.slice(2)); +// A case tagged `unfinished` has a reason not to be trusted written down beside +// it, so it does not get to burn real tokens just for sitting in the directory. +// Naming it with --case opts back in. +const cases = discoverCases().filter((c) => + args.case ? c.name.includes(args.case) : !toList(c.frontmatter.tags)?.includes("unfinished"), +); +if (cases.length === 0) { + console.error("No eval cases found."); + process.exit(1); +} + +const arms = args.arm === "both" ? ["with", "without"] : [args.arm]; +const stamp = new Date().toISOString().replace(/[:.]/g, "-"); +const resultsDir = join(EVALS_DIR, "results", stamp); +mkdirSync(resultsDir, { recursive: true }); + +const report = { model: args.model, runs: args.runs, arms, cases: [] }; + +for (const kase of cases) { + const runs = args.runs ?? kase.frontmatter.runs ?? 3; + const timeoutMs = (kase.frontmatter.timeout_seconds ?? 300) * 1000; + const armResults = {}; + + for (const arm of arms) { + // A case that runs AS a bundled agent has nothing to say without the plugin + // that defines the agent - the baseline arm would dispatch a name that does + // not exist and bill for the failure. + if (arm === "without" && kase.frontmatter.agent) { + console.log(`${kase.name}: skipping the without arm - agent "${kase.frontmatter.agent}" only exists with the plugin`); + continue; + } + armResults[arm] = []; + for (let i = 0; i < runs; i++) { + process.stderr.write(`${kase.name} [${arm}] run ${i + 1}/${runs} ... `); + const run = await runOnce(kase, arm, timeoutMs); + writeFileSync(join(resultsDir, `${kase.name}.${arm}.${i + 1}.jsonl`), run.trace, "utf-8"); + if (run.exitCode !== 0) { + writeFileSync(join(resultsDir, `${kase.name}.${arm}.${i + 1}.stderr.txt`), run.stderr, "utf-8"); + process.stderr.write(`exit ${run.exitCode}: ${run.stderr.trim().split("\n").pop() ?? ""}\n`); + } + // A run that died is not a run that scored zero - it is a run with no + // result. Grading its truncated trace would let an empty transcript pass + // every `not_contains` grader, and averaging its missing cost as absent + // rather than as failure would flatter the arm it belongs to. + const scores = + run.exitCode === 0 + ? kase.graders.map((g) => ({ grader: g.name, ...grade(g, run) })) + : kase.graders.map((g) => ({ grader: g.name, pass: false, notRun: true })); + armResults[arm].push({ + exitCode: run.exitCode, + costUsd: run.costUsd, + durationMs: run.durationMs, + turns: run.turns, + modelUsage: run.modelUsage, + scores, + }); + const passed = scores.filter((s) => s.pass).length; + process.stderr.write(`${passed}/${scores.length}\n`); + } + } + + if (Object.keys(armResults).length === 0) continue; + report.cases.push({ name: kase.name, arms: armResults }); + printCase(kase, armResults); +} + +writeFileSync(join(resultsDir, "aggregate.json"), JSON.stringify(report, null, 2), "utf-8"); +console.log(`\nTraces and aggregate.json: ${resultsDir}`); + +// --- running ------------------------------------------------------------- + +// The real harness gets its workspace from case.yaml `context.scaffold_script`, +// which needs a run to verify. Until then a case names its fixture directory +// under evals/fixtures; a fixture too large to commit ships as `.gen.mjs` +// next to it and is generated on first use. +function fixtureFor(kase) { + const name = args.fixture ?? kase.frontmatter.fixture ?? "mini-app"; + const dir = join(FIXTURES_DIR, name); + const generator = join(FIXTURES_DIR, `${name}.gen.mjs`); + + if (!isFile(generator)) { + if (isDir(dir)) return dir; + throw new Error(`No fixture "${name}" and no ${name}.gen.mjs to build one`); + } + + // Regenerate whenever the generator is newer than what it produced. A stale + // fixture kept because the directory happened to exist is how a fixed + // generator quietly fails to reach the runs that report on it. + if (!isDir(dir) || statSync(generator).mtimeMs > statSync(dir).mtimeMs) { + execFileSync(process.execPath, [generator], { stdio: "inherit" }); + } + if (!isDir(dir)) throw new Error(`${name}.gen.mjs ran but produced no ${dir}`); + return dir; +} + +// One turn measures the cost of delegating and none of the return: the context +// a subagent kept out of the main session only pays off when a later turn would +// have re-read it. A case with follow-ups runs them in the same session so that +// re-reading actually happens. +async function runOnce(kase, arm, timeoutMs) { + const workspace = mkdtempSync(join(tmpdir(), "model-routing-eval-")); + cpSync(fixtureFor(kase), workspace, { recursive: true }); + + const turns = []; + for (const [index, prompt] of [kase.prompt, ...kase.followups].entries()) { + turns.push( + await runTurn( + prompt, + arm, + workspace, + timeoutMs, + index > 0, + kase.frontmatter.append_system_prompt, + toList(kase.frontmatter.tools), + kase.frontmatter.agent, + ), + ); + } + + try { + rmSync(workspace, { recursive: true, force: true }); + } catch { + // A file left open by the child is not worth failing a run over. + } + + return { + trace: turns.map((t) => t.trace).join("\n"), + stderr: turns.map((t) => t.stderr).join("\n"), + // A killed child closes with code null, and `null ?? 0` would report a + // timed-out run as clean - its truncated trace graded, its null cost quietly + // dropped from the mean. Anything that is not a clean 0 counts as a failure. + exitCode: turns.find((t) => t.exitCode !== 0) ? (turns.find((t) => t.exitCode !== 0).exitCode ?? 1) : 0, + lastMessage: turns.map((t) => t.lastMessage).join("\n\n"), + // Graded per turn as well as joined: an ordered regex run over the + // concatenation can assemble a passing match out of pieces of several + // answers, which is a pass the run did not earn. + turnMessages: turns.map((t) => t.lastMessage), + costUsd: sum(turns.map((t) => t.costUsd)), + durationMs: sum(turns.map((t) => t.durationMs)), + turns: sum(turns.map((t) => t.turns)), + modelUsage: mergeModelUsage(turns.map((t) => t.modelUsage)), + }; +} + +function sessionModelFor(agent) { + if (args.modelGiven) return args.model; + return agent ? null : args.model; +} + +function toList(value) { + if (value == null) return null; + return Array.isArray(value) ? value : [value]; +} + +function sum(values) { + return values.reduce((a, b) => a + (b ?? 0), 0); +} + +function mergeModelUsage(all) { + const merged = {}; + for (const usage of all) { + for (const [model, u] of Object.entries(usage ?? {})) { + const acc = (merged[model] ??= { costUSD: 0, inputTokens: 0, cacheReadInputTokens: 0, cacheCreationInputTokens: 0, outputTokens: 0 }); + for (const key of Object.keys(acc)) acc[key] += u[key] ?? 0; + } + } + return merged; +} + +async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup, appendSystemPrompt, tools, agent) { + const argv = [ + "-p", + prompt, + ...(isFollowup ? ["--continue"] : []), + // Whether a session chooses to delegate is a coin the model flips - observed + // 3/3, 3/3, 2/3, 2/3 and 0/3 across otherwise identical batches. A case that + // means to measure the price of a tier cannot also be measuring that coin, + // so it takes the decision away here and leaves the user prompt untouched. + ...(appendSystemPrompt ? ["--append-system-prompt", appendSystemPrompt] : []), + // "Do not read the files yourself" cannot be enforced through the tool set. + // Denying Read/Grep/Glob moves the reading into Bash; denying the shells too + // leaves ToolSearch to reload a deferred Read; and an allowlist finally + // stops the main session by ALSO starving every subagent it dispatches, so + // the answers come back empty and the run measures nothing. Kept only for + // cases that want a deliberately tool-less session. + ...(tools ? ["--tools", ...tools] : []), + // Runs the session AS a bundled agent, on that agent's own pin. This is how + // a tier gets tested on its own merits: ask scout the question directly and + // grade what comes back, instead of asking a main session to delegate and + // then arguing about whether it really did. + ...(agent ? ["--agent", agent] : []), + "--output-format", + "stream-json", + "--verbose", + "--strict-mcp-config", + // Drop the user's own settings so the baseline arm is genuinely + // plugin-free: enabledPlugins lives in ~/.claude/settings.json, and + // model-routing is enabled there on this machine. + "--setting-sources", + "", + "--permission-mode", + "dontAsk", + // `--model` outranks an agent's frontmatter pin, so a case that runs AS an + // agent must not be handed the default or it silently measures the wrong + // tier. Pass one only when the caller asked for it by name. + ...(sessionModelFor(agent) ? ["--model", sessionModelFor(agent)] : []), + ]; + if (arm === "with") argv.push("--plugin-dir", PLUGIN_ROOT); + + // CLAUDE_CODE_SUBAGENT_MODEL overrides the tier of every subagent, which is + // the thing several of these cases exist to measure. Whatever the operator has + // set for their own work must not reach the experiment. + const env = { ...process.env }; + delete env.CLAUDE_CODE_SUBAGENT_MODEL; + + // No `shell: true` - it concatenates argv, which mangles the prompt. + const child = spawn(process.platform === "win32" ? "claude.exe" : "claude", argv, { cwd: workspace, env }); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (d) => (stdout += d)); + child.stderr.on("data", (d) => (stderr += d)); + + const timer = setTimeout(() => child.kill(), timeoutMs); + const exitCode = await new Promise((res) => child.on("close", res)); + clearTimeout(timer); + + const result = extractResult(stdout); + return { + trace: stdout, + stderr, + exitCode, + lastMessage: typeof result?.result === "string" ? result.result : "", + costUsd: result?.total_cost_usd ?? null, + durationMs: result?.duration_ms ?? null, + turns: result?.num_turns ?? null, + modelUsage: result?.modelUsage ?? {}, + }; +} + +function extractResult(stdout) { + for (const line of stdout.split("\n").reverse()) { + if (!line.trim()) continue; + try { + const parsed = JSON.parse(line); + if (parsed.type === "result") return parsed; + } catch { + // Non-JSON noise on stdout is not a result line. + } + } + return null; +} + +// --- grading ------------------------------------------------------------- + +function grade(grader, run) { + if (grader.type !== "regex") return { pass: false, skipped: `unsupported grader type: ${grader.type}` }; + const pattern = new RegExp(grader.pattern, grader.flags ?? ""); + const wanted = grader.match ?? "contains"; + + // `last_message` is graded turn by turn: one turn has to satisfy the pattern + // on its own. Matching the joined text instead would let a multi-part regex + // straddle two answers and pass on neither. + if (grader.target === "last_message") { + const messages = run.turnMessages ?? [run.lastMessage]; + return wanted === "not_contains" + ? { pass: messages.every((m) => !pattern.test(m)) } + : { pass: messages.some((m) => pattern.test(m)) }; + } + + const hit = pattern.test(run.trace); + return { pass: wanted === "not_contains" ? !hit : hit }; +} + +function printCase(kase, armResults) { + console.log(`\n${kase.name}`); + for (const grader of kase.graders) { + const cells = Object.entries(armResults).map(([arm, runs]) => { + const passed = runs.filter((r) => r.scores.find((s) => s.grader === grader.name)?.pass).length; + return `${arm} ${passed}/${runs.length}`; + }); + const rates = Object.values(armResults).map( + (runs) => runs.filter((r) => r.scores.find((s) => s.grader === grader.name)?.pass).length / runs.length, + ); + const delta = rates.length === 2 ? ` | delta ${(rates[0] - rates[1]).toFixed(2)}` : ""; + console.log(` ${grader.name}: ${cells.join(" | ")}${delta}`); + } + // A behavioural delta is only half the answer. Routing a question out of the + // main session buys context back and pays for it in dollars and latency, so + // print the price next to the score rather than letting the score stand alone. + const price = Object.entries(armResults).map(([arm, runs]) => { + const priced = runs.filter((r) => typeof r.costUsd === "number"); + const cost = mean(runs.map((r) => r.costUsd)); + const seconds = mean(runs.map((r) => r.durationMs)) / 1000; + // Say how many runs are behind the mean. A run that died has no cost to + // contribute, and a mean over two runs that looks like a mean over three is + // the kind of quiet thing that gets published. + const basis = priced.length === runs.length ? "" : ` (${priced.length}/${runs.length} runs)`; + return `${arm} $${cost.toFixed(3)} / ${seconds.toFixed(0)}s${basis}`; + }); + console.log(` cost per run: ${price.join(" | ")}`); + + // The total hides the thing the plugin is actually for. Whether work left the + // expensive tier only shows in the per-model split: a total that went up while + // the top-tier line held flat is the plugin working, not the plugin costing. + for (const [arm, runs] of Object.entries(armResults)) { + const perModel = new Map(); + for (const run of runs) { + for (const [model, usage] of Object.entries(run.modelUsage ?? {})) { + const acc = perModel.get(model) ?? { cost: 0, read: 0, n: 0 }; + acc.cost += usage.costUSD ?? 0; + acc.read += (usage.cacheReadInputTokens ?? 0) + (usage.inputTokens ?? 0); + acc.n = runs.length; + perModel.set(model, acc); + } + } + const parts = [...perModel.entries()] + .sort((a, b) => b[1].cost - a[1].cost) + .map(([model, a]) => `${model} $${(a.cost / a.n).toFixed(3)} / ${Math.round(a.read / a.n / 1000)}k in`); + console.log(` ${arm} by model: ${parts.join(" | ")}`); + } +} + +function mean(values) { + const usable = values.filter((v) => typeof v === "number"); + if (usable.length === 0) return NaN; + return usable.reduce((a, b) => a + b, 0) / usable.length; +} + +// --- case discovery ------------------------------------------------------ + +function discoverCases() { + return readdirSync(EVALS_DIR) + .filter((entry) => { + const path = join(EVALS_DIR, entry); + if (!isDir(path)) return false; + return isFile(join(path, "prompt.md")); + }) + .map((entry) => loadCase(join(EVALS_DIR, entry))); +} + +function loadCase(dir) { + const { frontmatter, body } = splitFrontmatter(readFileSync(join(dir, "prompt.md"), "utf-8")); + const gradersDir = join(dir, "graders"); + const graders = isDir(gradersDir) + ? readdirSync(gradersDir) + .filter((f) => f.endsWith(".md")) + .map((f) => ({ + name: f.replace(/\.md$/, ""), + ...splitFrontmatter(readFileSync(join(gradersDir, f), "utf-8")).frontmatter, + })) + : []; + // followup-1.md, followup-2.md ... are further turns of the same session. + const followups = readdirSync(dir) + .filter((f) => /^followup-\d+\.md$/.test(f)) + .sort() + .map((f) => splitFrontmatter(readFileSync(join(dir, f), "utf-8")).body.trim()); + + return { + name: frontmatter.name ?? dir.split(/[\\/]/).pop(), + frontmatter, + prompt: body.trim(), + followups, + graders, + }; +} + +// Enough YAML for this format: scalars, quoted scalars, and inline lists. A +// real parser would be a dependency for four keys. +function splitFrontmatter(text) { + const match = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?/.exec(text); + if (!match) return { frontmatter: {}, body: text }; + const frontmatter = {}; + for (const line of match[1].split(/\r?\n/)) { + const kv = /^([A-Za-z_][\w]*)\s*:\s*(.*)$/.exec(line); + if (!kv) continue; + frontmatter[kv[1]] = parseScalar(kv[2]); + } + return { frontmatter, body: text.slice(match[0].length) }; +} + +function parseScalar(raw) { + const value = raw.trim(); + if (value.startsWith("[") && value.endsWith("]")) { + return value + .slice(1, -1) + .split(",") + .map((v) => parseScalar(v)) + .filter((v) => v !== ""); + } + if (/^'.*'$/.test(value) || /^".*"$/.test(value)) return value.slice(1, -1); + if (/^-?\d+$/.test(value)) return Number(value); + return value; +} + +function parseArgs(argv) { + const out = { runs: null, model: "sonnet", modelGiven: false, arm: "both", case: null, fixture: null }; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === "--runs") out.runs = Number(argv[++i]); + else if (argv[i] === "--model") { + out.model = argv[++i]; + out.modelGiven = true; + } + else if (argv[i] === "--arm") out.arm = argv[++i]; + else if (argv[i] === "--case") out.case = argv[++i]; + else if (argv[i] === "--fixture") out.fixture = argv[++i]; + else throw new Error(`Unknown option: ${argv[i]}`); + } + return out; +} + +function isDir(path) { + try { + return statSync(path).isDirectory(); + } catch { + return false; + } +} + +function isFile(path) { + try { + return statSync(path).isFile(); + } catch { + return false; + } +} diff --git a/evals/subagent-answer-quality/followup-1.md b/evals/subagent-answer-quality/followup-1.md new file mode 100644 index 0000000..9431520 --- /dev/null +++ b/evals/subagent-answer-quality/followup-1.md @@ -0,0 +1,3 @@ +A batch reaches applyTaxPolicy with its outcome field set to the string "warn", and context.lenient is false. What does applyTaxPolicy return? + +Answer with the returned value alone on the first line, then say whether that is what the author of TAX_POLICY appears to have intended. diff --git a/evals/subagent-answer-quality/graders/policy-returns-null.md b/evals/subagent-answer-quality/graders/policy-returns-null.md new file mode 100644 index 0000000..cf67d76 --- /dev/null +++ b/evals/subagent-answer-quality/graders/policy-returns-null.md @@ -0,0 +1,16 @@ +--- +type: regex +target: last_message +pattern: '^\s*\**`?null`?\**\.?\s*$' +flags: 'im' +match: contains +--- + +`applyTaxPolicy` looks the batch outcome up as `TAX_POLICY[batch.outcome]`, and +the table is keyed `onWarn`, `onFatal`, `onTimeout`, `onUnknown`. An outcome of +"warn" therefore misses, falls through to `?? TAX_POLICY.onUnknown` which is +"abort", and with `lenient` false the function returns `null`. + +The plausible wrong answer is the batch with `decision: "continue"` - which is +what the table's `onWarn: "continue"` row looks like it promises, and what +reading the table without following the lookup produces. diff --git a/evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md b/evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md new file mode 100644 index 0000000..f813a47 --- /dev/null +++ b/evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md @@ -0,0 +1,11 @@ +--- +type: regex +target: last_message +pattern: 'onWarn' +match: contains +--- + +Naming `onWarn` is the difference between reporting the return value and +understanding it. The question asks whether the author intended this, and the +answer is only defensible by pointing at the gap between the table's `onWarn` key +and the raw `warn` the lookup uses - which is the actual defect in the code. diff --git a/evals/subagent-answer-quality/graders/retry-count-is-one.md b/evals/subagent-answer-quality/graders/retry-count-is-one.md new file mode 100644 index 0000000..5574709 --- /dev/null +++ b/evals/subagent-answer-quality/graders/retry-count-is-one.md @@ -0,0 +1,19 @@ +--- +type: regex +target: last_message +pattern: '^\s*\**`?1`?\**\.?\s*$' +flags: 'm' +match: contains +--- + +The answer is 1, and the confident wrong answer is 3. + +`withRetryIntake` loops `maxRetries` times but breaks the moment +`error?.retryable === false`, and `errorsValidation` builds every failure with +`retryable: cause?.retryable ?? false`. A rule violation carries no cause, so the +flag is false, so the loop breaks on the first catch. `maxRetries: 3` is sitting +right there in the file and never applies. + +The prompt asks for the number alone on its own line so this can be graded +without a judge model. Requiring the whole line to be the number is what keeps a +markdown list item ("1. Something") from passing. diff --git a/evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md b/evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md new file mode 100644 index 0000000..6754d6a --- /dev/null +++ b/evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md @@ -0,0 +1,11 @@ +--- +type: regex +target: last_message +pattern: 'retryable' +match: contains +--- + +Guards against a lucky number. There are only a few plausible answers to "how +many times", so one of them comes up by chance often enough to matter; an answer +that reaches 1 without ever mentioning `retryable` did not follow the chain from +`withRetryIntake` through `errorsValidation`, and got there some other way. diff --git a/evals/subagent-answer-quality/prompt.md b/evals/subagent-answer-quality/prompt.md new file mode 100644 index 0000000..607da6b --- /dev/null +++ b/evals/subagent-answer-quality/prompt.md @@ -0,0 +1,14 @@ +--- +name: subagent-answer-quality +tags: [routing, tier, quality] +plugins: ["../.."] +fixture: order-service +agent: scout +runs: 3 +max_turns: 30 +timeout_seconds: 900 +--- + +The intake stage hands off to validation through withRetryIntake with INTAKE_LIMITS.maxRetries set to 3. Suppose validateOrder fails on an order that violates one of validation's own rules. How many times does validateOrder actually get called? + +Answer with the number alone on the first line, then explain what in the code makes it that number. diff --git a/evals/surveyor-traces-the-chain/graders/answer-is-correct.md b/evals/surveyor-traces-the-chain/graders/answer-is-correct.md new file mode 100644 index 0000000..9d4f222 --- /dev/null +++ b/evals/surveyor-traces-the-chain/graders/answer-is-correct.md @@ -0,0 +1,15 @@ +--- +type: regex +target: last_message +pattern: 'intake[\s\S]*validation[\s\S]*pricing[\s\S]*discount[\s\S]*tax[\s\S]*deposit[\s\S]*payment[\s\S]*tips[\s\S]*fiscal[\s\S]*receipt[\s\S]*export[\s\S]*archive' +flags: 'i' +match: contains +--- + +The claim the routing anchor now makes out loud is that breadth work runs +correctly a tier down. This is the case that has to keep being true for that +sentence to stay in a file every session pays for. + +Twelve stages in the right order. The names are guessable from a directory +listing, so the ordering is the whole answer - and no file states it, since each +stage imports the next at the bottom of a few hundred lines. diff --git a/evals/surveyor-traces-the-chain/prompt.md b/evals/surveyor-traces-the-chain/prompt.md new file mode 100644 index 0000000..4f39362 --- /dev/null +++ b/evals/surveyor-traces-the-chain/prompt.md @@ -0,0 +1,12 @@ +--- +name: surveyor-traces-the-chain +tags: [routing, tier, breadth] +plugins: ["../.."] +fixture: order-service +agent: surveyor +runs: 3 +max_turns: 30 +timeout_seconds: 600 +--- + +An order enters this service at intake and ends up archived. Trace the whole path: list every stage it passes through, in order, from intake to archive. diff --git a/evals/traces-a-flow-end-to-end/followup-1.md b/evals/traces-a-flow-end-to-end/followup-1.md new file mode 100644 index 0000000..4f06479 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/followup-1.md @@ -0,0 +1 @@ +If the payment stage throws an error that is not retryable, what does the order end up as, and which stage decides that? Answer from the code, not from convention. diff --git a/evals/traces-a-flow-end-to-end/followup-2.md b/evals/traces-a-flow-end-to-end/followup-2.md new file mode 100644 index 0000000..2d4e7eb --- /dev/null +++ b/evals/traces-a-flow-end-to-end/followup-2.md @@ -0,0 +1 @@ +I want to add a service charge that applies after discounts but before VAT. Which stage should own it, and what would it have to hand the next stage so nothing downstream breaks? diff --git a/evals/traces-a-flow-end-to-end/followup-3.md b/evals/traces-a-flow-end-to-end/followup-3.md new file mode 100644 index 0000000..fb70a67 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/followup-3.md @@ -0,0 +1 @@ +Which stages split their work into batches before handing off, and what is the batch size? List them. diff --git a/evals/traces-a-flow-end-to-end/graders/answer-is-correct.md b/evals/traces-a-flow-end-to-end/graders/answer-is-correct.md new file mode 100644 index 0000000..e128d11 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/graders/answer-is-correct.md @@ -0,0 +1,14 @@ +--- +type: regex +target: last_message +pattern: 'intake[\s\S]*validation[\s\S]*pricing[\s\S]*discount[\s\S]*tax[\s\S]*deposit[\s\S]*payment[\s\S]*tips[\s\S]*fiscal[\s\S]*receipt[\s\S]*export[\s\S]*archive' +flags: 'i' +match: contains +--- + +All twelve stages, in the right order. The order is the answer: the stage names +are guessable from the directory listing alone, so an answer scored on the names +without the sequence would pass without anybody reading anything. + +No file states the sequence. Each stage imports the next at the bottom of a few +hundred lines, so the chain has to be walked. diff --git a/evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md b/evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md new file mode 100644 index 0000000..5ad7776 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md @@ -0,0 +1,12 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Agent|Task)"' +match: contains +--- + +Same behavioural claim as the small case, on a question where the reading is +actually wide: twelve stage modules at roughly 2k tokens each have to pass under +somebody's eyes, and which model's eyes those are is the entire point. + +Both arms can pass - the baseline has `Explore` and `general-purpose`. diff --git a/evals/traces-a-flow-end-to-end/prompt.md b/evals/traces-a-flow-end-to-end/prompt.md new file mode 100644 index 0000000..61928da --- /dev/null +++ b/evals/traces-a-flow-end-to-end/prompt.md @@ -0,0 +1,11 @@ +--- +name: traces-a-flow-end-to-end +tags: [routing, scout, wide-reading] +plugins: ["../.."] +fixture: order-service +runs: 3 +max_turns: 20 +timeout_seconds: 600 +--- + +An order enters this service at intake and ends up archived. Trace the whole path: list every stage it passes through, in order, from intake to archive. diff --git a/hooks/dispatch-counter.mjs b/hooks/dispatch-counter.mjs index 6a27322..d5c4cf1 100644 --- a/hooks/dispatch-counter.mjs +++ b/hooks/dispatch-counter.mjs @@ -81,6 +81,7 @@ function sessionFilterFromArgs(argv) { // Keep in sync with agents/*.md; one sync test guards both columns. const AGENT_PINS = { "model-routing:scout": { model: "sonnet", effort: "low" }, + "model-routing:surveyor": { model: "haiku", effort: "low" }, "model-routing:test-runner": { model: "haiku", effort: "low" }, "model-routing:e2e-runner": { model: "sonnet", effort: "medium" }, "model-routing:verifier": { model: "haiku", effort: "low" }, diff --git a/hooks/routing-anchor.md b/hooks/routing-anchor.md index 23e7073..987973f 100644 --- a/hooks/routing-anchor.md +++ b/hooks/routing-anchor.md @@ -2,7 +2,7 @@ MODEL ROUTING ACTIVE - expensive model thinks, cheap models grind. You cannot sw - Planning, specs, docs, architecture, small interactive edits: main session. - Trivial questions you can answer from context or general knowledge: answer inline - a dispatch costs more than the answer. -- Codebase exploration ("where is X", "how does Y work"): `scout`. Locate-only sweeps ("which files mention X"): the built-in Explore agent, when present, is cheaper. +- Codebase exploration: pick by what has to come back. A judgement about what code does ("how does Y work", "does this retry") is `scout`. A complete list or an ordering, verified by following the code ("every stage in order", "everything that imports X") is `surveyor` (haiku) - breadth holds a tier down, judgement does not. Candidate files to look at next, unverified ("which files mention X") is the built-in Explore agent, cheaper than either. - Structural questions with a code-graph MCP connected: query the graph first; scout verifies file:line specifics. - Implementing an approved plan/spec: `implementer` (sonnet). Dispatch with `model=opus` for multi-file/cross-layer work, security/money/migrations/concurrency/public contracts, or a retry after a weak sonnet result - escalate when in doubt; the dispatch changes the model only, the pinned medium effort stays. Ambiguous task or unclear root cause: clarify first (main session / scout) - implementer stops on ambiguity. Batch related tasks per agent. - Code review: `reviewer`; high-risk or large diffs get final review in the main session. diff --git a/skills/model-routing/SKILL.md b/skills/model-routing/SKILL.md index 9f69f66..b402aea 100644 --- a/skills/model-routing/SKILL.md +++ b/skills/model-routing/SKILL.md @@ -10,6 +10,18 @@ cannot be switched by Claude - routing works through subagent delegation (the `model` param of the Agent tool, or the agents bundled with this plugin). +**Routing makes a dispatch cheaper; it does not make dispatching cheap.** +A subagent starts empty, so everything it reads is a cache write, while +the main session pays cache read - 12.5x less - for context it already +holds. That penalty is paid whether or not the tier is routed down, so the +rules below only pay off on work that was going to be delegated anyway. +On one wide-reading session, three runs: doing it inline cost $1.36 and +delegating with the tier routed down $1.68, both measured; delegating at +the session tier comes to $2.01, which is those same measured subagent +tokens repriced at the higher tier rather than a run of its own. Route +every dispatch - roughly a sixth off work already leaving the session - +and do not manufacture dispatches to collect it. + ## Tiers Think in tiers, not model names - names rot, tiers do not: @@ -97,6 +109,7 @@ carries no effort param: |------|-------|---------------|--------| | Planning, brainstorming, specs, docs, architecture | main session | strongest (user's /model choice) | high | | Codebase exploration ("where is X", "how does Y work") | subagent | `scout` (sonnet) | low | +| Breadth sweeps: enumerate, list, trace a chain end to end | subagent | `surveyor` (haiku) | low | | Implementing an approved plan/spec (ordinary: single-file, clear shape) | subagent | `implementer` (sonnet) | medium | | Complex implementation: multi-file refactor, subtle concurrency/security | subagent | `implementer` with `model=opus` | medium (pinned) | | Trivial mechanical tasks: renames, boilerplate, mirrored constants | subagent | sonnet | low | @@ -210,10 +223,33 @@ actually ran with `/model-routing:stats`. `test-runner` and consume its compact report. - Route codebase exploration to `scout` - conclusions and file:line refs come back, file dumps stay in the subagent. -- For locate-only sweeps ("which files mention X") the harness's built-in - Explore agent, when present, is cheaper than `scout`. Use `scout` when - the answer needs verification - tracing real code paths and confirming - file:line - not just finding candidates. +- Three exploration routes, told apart by what has to come back rather + than by how the question is phrased. Unverified candidates to look at + next ("which files mention X"): the harness's built-in Explore agent, + cheaper than both bundled ones. A complete list or an ordering, verified + by following the code ("every stage in order", "everything that imports + X"): `surveyor`. A judgement about behaviour ("how does Y work", "does + this retry"): `scout`. "Which files import X" and "which files mention + X" look alike and are not: one is answered by grep and may be wrong at + the edges, the other has to be right. +- Split exploration by what the question demands, not by what it costs. + Enumerating and tracing - list these stages in order, which files import + X, where does this chain end - is breadth, and breadth runs correctly a + tier down: that is `surveyor` (haiku). Working out what code actually + does - does this loop retry, what does this function return for that + input - is not breadth, and the cheap tier fails it in a way that looks + confident: that is `scout` (sonnet). Both halves are measured in `evals/`, + on one generated fixture at three runs each: on a twelve-stage tracing + question haiku answered correctly every run at a third of sonnet's price; + on a question whose code contains an obvious wrong answer, haiku took the + bait once in three runs while sonnet took it in none. Three runs on one + fixture cannot pin a failure rate, and the reason to keep judgement on + sonnet is not the rate but the asymmetry: a cheap right answer saves + cents, while a cheap wrong one arrives looking identical and sends the + main session back to read the files itself. That recovery was not + measured; it does not need to be, to be worth avoiding. Reach for the + pinned agent rather than overriding `scout` downward - the floor rule + below is not a formality. - Batch related plan tasks per subagent. Each subagent re-reads files from scratch; one tiny task per agent costs more than it saves. - Subagents cannot see the conversation. Write self-contained task