From 8ad47db89f59708763a43fed4ddc5fdabd47f90e Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Mon, 17 Aug 2026 16:06:21 +0300 Subject: [PATCH 1/9] test: measure the routing guidance against a plugin-free control arm The unit tests cover the report and say nothing about the half of this plugin that is prose. This adds the missing half: an eval case that asks the same codebase question twice, once with the plugin loaded and once without, and scores both arms on graders either arm can pass. The cases are written for `claude plugin eval`, which is early access and gated per organization; where the gate is closed `evals/run-local.mjs` reads the same `prompt.md` + `graders/*.md` files and does the two-arm run through `claude -p`. It exists to be deleted once the gate opens. The baseline arm is made genuinely plugin-free by dropping the user's own settings, since `enabledPlugins` is where an installed copy would otherwise reach both arms - the `system/init` event in a baseline trace lists no `model-routing:*` agent, which is the check worth repeating before trusting any run. First result on an opus session, 3 runs per arm: the plugin arm delegated to `scout` in 3 of 3, the plugin-free arm delegated in 0 of 3 despite having `Explore` available, and both arms answered correctly. The same run also prices the trade - identical answers for 2.3x the money and 3x the wall clock on a four-file fixture, which is the plugin routing where its own anchor says a dispatch costs more than the answer. Both halves are recorded rather than only the flattering one, and the README no longer claims the behavioural half has no control group. --- .gitignore | 1 + README.md | 4 +- evals/README.md | 91 ++++++ .../graders/answer-is-correct.md | 10 + .../graders/delegated-to-a-subagent.md | 13 + .../graders/scout-was-the-agent.md | 11 + evals/delegates-codebase-question/prompt.md | 10 + evals/fixtures/mini-app/README.md | 5 + evals/fixtures/mini-app/src/api/client.js | 9 + evals/fixtures/mini-app/src/api/retry.js | 15 + evals/fixtures/mini-app/src/auth/session.js | 13 + evals/fixtures/mini-app/src/util/clock.js | 9 + evals/run-local.mjs | 260 ++++++++++++++++++ 13 files changed, 450 insertions(+), 1 deletion(-) create mode 100644 evals/README.md create mode 100644 evals/delegates-codebase-question/graders/answer-is-correct.md create mode 100644 evals/delegates-codebase-question/graders/delegated-to-a-subagent.md create mode 100644 evals/delegates-codebase-question/graders/scout-was-the-agent.md create mode 100644 evals/delegates-codebase-question/prompt.md create mode 100644 evals/fixtures/mini-app/README.md create mode 100644 evals/fixtures/mini-app/src/api/client.js create mode 100644 evals/fixtures/mini-app/src/api/retry.js create mode 100644 evals/fixtures/mini-app/src/auth/session.js create mode 100644 evals/fixtures/mini-app/src/util/clock.js create mode 100644 evals/run-local.mjs diff --git a/.gitignore b/.gitignore index 3ac9ac8..9fd7840 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ docs/superpowers/ .superpowers/ +evals/results/ nul diff --git a/README.md b/README.md index 6b2f27c..a9fd1e7 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,9 @@ Attribution over the same week, across all 266 dispatches, 199 of which ran belo | An explicit `model=` on the dispatch | 127 | Behavioural. This is the routing rules being applied, and nothing enforces them | | Claude Code's built-in `Explore` | 4 | Not this plugin at all | -The honest reading: the automatic half is fully attributable, the behavioural half depends on the session actually following the anchor, and there is no control group - this is the author's own workload measured with the author's own tool, with no before-install baseline. `CLAUDE_CODE_SUBAGENT_MODEL` was not set in this window, which the report would otherwise have annotated per row. +The honest reading: the automatic half is fully attributable, the behavioural half depends on the session actually following the anchor, and these numbers have no control group of their own - this is the author's own workload measured with the author's own tool, with no before-install baseline. `CLAUDE_CODE_SUBAGENT_MODEL` was not set in this window, which the report would otherwise have annotated per row. + +For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. On the first case the arms separate completely - the plugin arm delegated a codebase question to `scout` in 3 of 3 runs, the plugin-free arm delegated in 0 of 3 and grepped inline despite having `Explore` available - and the same run shows the other side of the trade: identical answers for 2.3x the money and 3x the wall clock on a four-file fixture. Guidance that changes behaviour is not automatically guidance that pays; the eval is there to keep both halves visible. Where that volume actually ran with routing active: diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..c009ce2 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,91 @@ +# Evals + +The unit tests in `hooks/` cover the report: given a transcript, does the maths +come out right. They say nothing about the half of this plugin that is prose - +whether the routing guidance actually changes what a session does. That is what +these cases are for, and until they existed the README's admission that there is +no control group was simply true. + +## The one thing that makes this a measurement + +Every case runs twice: once with the plugin loaded, once without. A number from +the with-arm alone means nothing, because a capable model routes sensibly on its +own a fair share of the time. The delta between the arms is the claim. + +`delegates-codebase-question` is scored on two graders both arms can genuinely +pass or fail - did the session hand the question to a subagent, and did the +answer come back right. A third grader checks that the with-arm picked `scout` +specifically; the baseline has no `scout` to pick, so read that one as an +indicator and never as a score. + +## Running them + +`claude plugin eval` is the harness these cases are written for: + +```sh +claude plugin eval model-routing --ablation with-without +``` + +It is early access and gated per organization; on a machine where the gate is +closed it prints `` `plugin eval` is currently in early access `` and exits 1. +Until then, `run-local.mjs` reads the same case files and does the same two-arm +run through `claude -p`: + +```sh +node evals/run-local.mjs --runs 3 --model opus +node evals/run-local.mjs --runs 1 --model haiku --arm without # smoke test +``` + +The baseline arm is made plugin-free by dropping the user's own settings +(`--setting-sources ""`), since `enabledPlugins` is where a globally installed +copy of this plugin would otherwise sneak into both arms. Verify that in a trace +before trusting a run: the `system/init` event lists the available agents, and +the baseline's list must not contain any `model-routing:*` entry. + +Runs are real sessions against the real API, so they cost real tokens. Traces +and an `aggregate.json` land in `evals/results//`, which is +gitignored. + +## The first result + +Opus main session, 3 runs per arm, 2026-08-17: + +| Grader | With | Without | Delta | +| --- | ---: | ---: | ---: | +| delegated to a subagent | 3/3 | 0/3 | +1.00 | +| answer is correct | 3/3 | 3/3 | 0.00 | +| scout was the agent (indicator) | 3/3 | 0/3 | - | +| cost / wall clock per run | $0.310 / 32s | $0.136 / 11s | +$0.174 / +21s | + +Two things are true at once and both matter. The guidance does change what the +session does, cleanly and every time - the arms separate perfectly, and the +baseline had `Explore` available and still chose to grep inline. That is the +control-group number the README did not have. + +And on this fixture the change is a net loss: same answer, 2.3x the money, 3x +the wall clock. The plugin's own anchor says a dispatch costs more than the +answer for anything trivial, and four files is trivial - so the honest reading +is that the routing fired where its own rule says it should not have. What the +fixture cannot show is the other side of the trade: delegation exists to keep +file contents out of the main context, and a four-file project has no context +worth protecting. + +So the next case wants a fixture big enough that reading it inline actually +floods the session - that is where the trade is supposed to pay - and the case +after that wants the mirror of this one: a genuinely trivial question, where +dispatching at all is the failure being measured. + +## Writing another case + +`evals//prompt.md` carries the prompt in its body and its settings in +frontmatter; `evals//graders/.md` carries one grader each, its type +and pattern in frontmatter and its reasoning in the body - the body is for the +next person, who needs to know why a grader is worth having and what it cannot +tell them. `run-local.mjs` supports `type: regex` over `target: trace` and +`target: last_message`; the real harness additionally has `tool_used`, +`tool_order`, `file_exists`, `llm`, and `baseline`. Prefer the deterministic +ones: a judge model scoring prose is one more thing that can be wrong. + +Cases share one fixture, `evals/fixtures/mini-app`. Keep it boring. A fixture +that is interesting to read is a fixture the model answers from memory of +something similar. diff --git a/evals/delegates-codebase-question/graders/answer-is-correct.md b/evals/delegates-codebase-question/graders/answer-is-correct.md new file mode 100644 index 0000000..f6905da --- /dev/null +++ b/evals/delegates-codebase-question/graders/answer-is-correct.md @@ -0,0 +1,10 @@ +--- +type: regex +target: last_message +pattern: 'refreshSession' +match: contains +--- + +Routing to a subagent is worthless if the answer comes back wrong, so the arms +are also scored on getting there. `refreshSession` in `src/auth/session.js` is +the only place the fixture extends a session; naming it is the whole answer. diff --git a/evals/delegates-codebase-question/graders/delegated-to-a-subagent.md b/evals/delegates-codebase-question/graders/delegated-to-a-subagent.md new file mode 100644 index 0000000..e2a18a8 --- /dev/null +++ b/evals/delegates-codebase-question/graders/delegated-to-a-subagent.md @@ -0,0 +1,13 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Agent|Task)"' +match: contains +--- + +The behavioural claim under test: a codebase question should leave the main +session rather than be ground out inline. Both arms can pass this - the baseline +has the built-in Explore and general-purpose agents - so the score difference is +a real measurement, not an artefact of the plugin owning the only subagent. + +The tool has been named both `Agent` and `Task` across builds; match either. diff --git a/evals/delegates-codebase-question/graders/scout-was-the-agent.md b/evals/delegates-codebase-question/graders/scout-was-the-agent.md new file mode 100644 index 0000000..9f4acf4 --- /dev/null +++ b/evals/delegates-codebase-question/graders/scout-was-the-agent.md @@ -0,0 +1,11 @@ +--- +type: regex +target: trace +pattern: '"subagent_type"\s*:\s*"(?:model-routing:)?scout"' +match: contains +--- + +Indicator, not a fair scored comparison: the baseline arm has no `scout` to +dispatch, so it fails this by construction. Read it only as confirmation that +the with-plugin arm picked the agent the routing guidance names, rather than +falling back to a generic one. diff --git a/evals/delegates-codebase-question/prompt.md b/evals/delegates-codebase-question/prompt.md new file mode 100644 index 0000000..6f6e491 --- /dev/null +++ b/evals/delegates-codebase-question/prompt.md @@ -0,0 +1,10 @@ +--- +name: delegates-codebase-question +tags: [routing, scout] +plugins: ["../.."] +runs: 3 +max_turns: 10 +timeout_seconds: 300 +--- + +Where does this project extend a session's lifetime? Name the file and the function. diff --git a/evals/fixtures/mini-app/README.md b/evals/fixtures/mini-app/README.md new file mode 100644 index 0000000..b04898a --- /dev/null +++ b/evals/fixtures/mini-app/README.md @@ -0,0 +1,5 @@ +# mini-app + +Fixture project for the routing evals. Deliberately small, deliberately boring: +the point is that answering a "where is X" question about it requires reading a +few files, not that the code is interesting. diff --git a/evals/fixtures/mini-app/src/api/client.js b/evals/fixtures/mini-app/src/api/client.js new file mode 100644 index 0000000..bee2702 --- /dev/null +++ b/evals/fixtures/mini-app/src/api/client.js @@ -0,0 +1,9 @@ +import { refreshSession } from "../auth/session.js"; + +export async function request(session, path) { + const live = refreshSession(session); + const response = await fetch(`https://example.invalid${path}`, { + headers: { authorization: `Bearer ${live.token}` }, + }); + return { session: live, body: await response.json() }; +} diff --git a/evals/fixtures/mini-app/src/api/retry.js b/evals/fixtures/mini-app/src/api/retry.js new file mode 100644 index 0000000..a60c338 --- /dev/null +++ b/evals/fixtures/mini-app/src/api/retry.js @@ -0,0 +1,15 @@ +import { request } from "./client.js"; + +const MAX_ATTEMPTS = 3; + +export async function requestWithRetry(session, path) { + let last; + for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { + try { + return await request(session, path); + } catch (error) { + last = error; + } + } + throw last; +} diff --git a/evals/fixtures/mini-app/src/auth/session.js b/evals/fixtures/mini-app/src/auth/session.js new file mode 100644 index 0000000..abb6c4d --- /dev/null +++ b/evals/fixtures/mini-app/src/auth/session.js @@ -0,0 +1,13 @@ +import { now } from "../util/clock.js"; + +const SESSION_TTL_MS = 15 * 60 * 1000; + +export function isExpired(session) { + return now() - session.issuedAt > SESSION_TTL_MS; +} + +// The only place a session's lifetime is extended. +export function refreshSession(session) { + if (!isExpired(session)) return session; + return { ...session, issuedAt: now() }; +} diff --git a/evals/fixtures/mini-app/src/util/clock.js b/evals/fixtures/mini-app/src/util/clock.js new file mode 100644 index 0000000..05557f7 --- /dev/null +++ b/evals/fixtures/mini-app/src/util/clock.js @@ -0,0 +1,9 @@ +let offsetMs = 0; + +export function now() { + return Date.now() + offsetMs; +} + +export function setOffset(ms) { + offsetMs = ms; +} diff --git a/evals/run-local.mjs b/evals/run-local.mjs new file mode 100644 index 0000000..d2ce007 --- /dev/null +++ b/evals/run-local.mjs @@ -0,0 +1,260 @@ +#!/usr/bin/env node +// Local stand-in for `claude plugin eval`, which is early access and gated for +// this account. It reads the SAME case layout the real harness reads +// (evals//prompt.md + graders/*.md), runs each case with and without the +// plugin, and scores the regex graders. When the gate opens, delete this file +// and run `claude plugin eval model-routing --ablation with-without` instead. +// +// Only the subset of the format this suite actually uses is supported: regex +// graders over `trace` and `last_message`. Anything else is reported as skipped +// rather than silently scored. + +import { spawn } from "node:child_process"; +import { cpSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; + +const EVALS_DIR = dirname(fileURLToPath(import.meta.url)); +const PLUGIN_ROOT = resolve(EVALS_DIR, ".."); +// The real harness gets its workspace from case.yaml `context.scaffold_script`. +// Until a run can verify that field, every case here shares one fixture. +const FIXTURE = join(EVALS_DIR, "fixtures", "mini-app"); + +const args = parseArgs(process.argv.slice(2)); +const cases = discoverCases().filter((c) => !args.case || c.name.includes(args.case)); +if (cases.length === 0) { + console.error("No eval cases found."); + process.exit(1); +} + +const arms = args.arm === "both" ? ["with", "without"] : [args.arm]; +const stamp = new Date().toISOString().replace(/[:.]/g, "-"); +const resultsDir = join(EVALS_DIR, "results", stamp); +mkdirSync(resultsDir, { recursive: true }); + +const report = { model: args.model, runs: args.runs, arms, cases: [] }; + +for (const kase of cases) { + const runs = args.runs ?? kase.frontmatter.runs ?? 3; + const timeoutMs = (kase.frontmatter.timeout_seconds ?? 300) * 1000; + const armResults = {}; + + for (const arm of arms) { + armResults[arm] = []; + for (let i = 0; i < runs; i++) { + process.stderr.write(`${kase.name} [${arm}] run ${i + 1}/${runs} ... `); + const run = await runOnce(kase, arm, timeoutMs); + writeFileSync(join(resultsDir, `${kase.name}.${arm}.${i + 1}.jsonl`), run.trace, "utf-8"); + if (run.exitCode !== 0) { + writeFileSync(join(resultsDir, `${kase.name}.${arm}.${i + 1}.stderr.txt`), run.stderr, "utf-8"); + process.stderr.write(`exit ${run.exitCode}: ${run.stderr.trim().split("\n").pop() ?? ""}\n`); + } + const scores = kase.graders.map((g) => ({ grader: g.name, ...grade(g, run) })); + armResults[arm].push({ + exitCode: run.exitCode, + costUsd: run.costUsd, + durationMs: run.durationMs, + turns: run.turns, + scores, + }); + const passed = scores.filter((s) => s.pass).length; + process.stderr.write(`${passed}/${scores.length}\n`); + } + } + + report.cases.push({ name: kase.name, arms: armResults }); + printCase(kase, armResults); +} + +writeFileSync(join(resultsDir, "aggregate.json"), JSON.stringify(report, null, 2), "utf-8"); +console.log(`\nTraces and aggregate.json: ${resultsDir}`); + +// --- running ------------------------------------------------------------- + +async function runOnce(kase, arm, timeoutMs) { + const workspace = mkdtempSync(join(tmpdir(), "model-routing-eval-")); + cpSync(FIXTURE, workspace, { recursive: true }); + + const argv = [ + "-p", + kase.prompt, + "--output-format", + "stream-json", + "--verbose", + "--strict-mcp-config", + // Drop the user's own settings so the baseline arm is genuinely + // plugin-free: enabledPlugins lives in ~/.claude/settings.json, and + // model-routing is enabled there on this machine. + "--setting-sources", + "", + "--permission-mode", + "dontAsk", + "--model", + args.model, + ]; + if (arm === "with") argv.push("--plugin-dir", PLUGIN_ROOT); + + // No `shell: true` - it concatenates argv, which mangles the prompt. + const child = spawn(process.platform === "win32" ? "claude.exe" : "claude", argv, { cwd: workspace }); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (d) => (stdout += d)); + child.stderr.on("data", (d) => (stderr += d)); + + const timer = setTimeout(() => child.kill(), timeoutMs); + const exitCode = await new Promise((res) => child.on("close", res)); + clearTimeout(timer); + + try { + rmSync(workspace, { recursive: true, force: true }); + } catch { + // A file left open by the child is not worth failing a run over. + } + + const result = extractResult(stdout); + return { + trace: stdout, + stderr, + exitCode, + lastMessage: typeof result?.result === "string" ? result.result : "", + costUsd: result?.total_cost_usd ?? null, + durationMs: result?.duration_ms ?? null, + turns: result?.num_turns ?? null, + }; +} + +function extractResult(stdout) { + for (const line of stdout.split("\n").reverse()) { + if (!line.trim()) continue; + try { + const parsed = JSON.parse(line); + if (parsed.type === "result") return parsed; + } catch { + // Non-JSON noise on stdout is not a result line. + } + } + return null; +} + +// --- grading ------------------------------------------------------------- + +function grade(grader, run) { + if (grader.type !== "regex") return { pass: false, skipped: `unsupported grader type: ${grader.type}` }; + const haystack = grader.target === "last_message" ? run.lastMessage : run.trace; + const hit = new RegExp(grader.pattern, grader.flags ?? "").test(haystack); + const wanted = grader.match ?? "contains"; + return { pass: wanted === "not_contains" ? !hit : hit }; +} + +function printCase(kase, armResults) { + console.log(`\n${kase.name}`); + for (const grader of kase.graders) { + const cells = Object.entries(armResults).map(([arm, runs]) => { + const passed = runs.filter((r) => r.scores.find((s) => s.grader === grader.name)?.pass).length; + return `${arm} ${passed}/${runs.length}`; + }); + const rates = Object.values(armResults).map( + (runs) => runs.filter((r) => r.scores.find((s) => s.grader === grader.name)?.pass).length / runs.length, + ); + const delta = rates.length === 2 ? ` | delta ${(rates[0] - rates[1]).toFixed(2)}` : ""; + console.log(` ${grader.name}: ${cells.join(" | ")}${delta}`); + } + // A behavioural delta is only half the answer. Routing a question out of the + // main session buys context back and pays for it in dollars and latency, so + // print the price next to the score rather than letting the score stand alone. + const price = Object.entries(armResults).map(([arm, runs]) => { + const cost = mean(runs.map((r) => r.costUsd)); + const seconds = mean(runs.map((r) => r.durationMs)) / 1000; + return `${arm} $${cost.toFixed(3)} / ${seconds.toFixed(0)}s`; + }); + console.log(` cost per run: ${price.join(" | ")}`); +} + +function mean(values) { + const usable = values.filter((v) => typeof v === "number"); + if (usable.length === 0) return NaN; + return usable.reduce((a, b) => a + b, 0) / usable.length; +} + +// --- case discovery ------------------------------------------------------ + +function discoverCases() { + return readdirSync(EVALS_DIR) + .filter((entry) => { + const path = join(EVALS_DIR, entry); + if (!isDir(path)) return false; + return isFile(join(path, "prompt.md")); + }) + .map((entry) => loadCase(join(EVALS_DIR, entry))); +} + +function loadCase(dir) { + const { frontmatter, body } = splitFrontmatter(readFileSync(join(dir, "prompt.md"), "utf-8")); + const gradersDir = join(dir, "graders"); + const graders = isDir(gradersDir) + ? readdirSync(gradersDir) + .filter((f) => f.endsWith(".md")) + .map((f) => ({ + name: f.replace(/\.md$/, ""), + ...splitFrontmatter(readFileSync(join(gradersDir, f), "utf-8")).frontmatter, + })) + : []; + return { name: frontmatter.name ?? dir.split(/[\\/]/).pop(), frontmatter, prompt: body.trim(), graders }; +} + +// Enough YAML for this format: scalars, quoted scalars, and inline lists. A +// real parser would be a dependency for four keys. +function splitFrontmatter(text) { + const match = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?/.exec(text); + if (!match) return { frontmatter: {}, body: text }; + const frontmatter = {}; + for (const line of match[1].split(/\r?\n/)) { + const kv = /^([A-Za-z_][\w]*)\s*:\s*(.*)$/.exec(line); + if (!kv) continue; + frontmatter[kv[1]] = parseScalar(kv[2]); + } + return { frontmatter, body: text.slice(match[0].length) }; +} + +function parseScalar(raw) { + const value = raw.trim(); + if (value.startsWith("[") && value.endsWith("]")) { + return value + .slice(1, -1) + .split(",") + .map((v) => parseScalar(v)) + .filter((v) => v !== ""); + } + if (/^'.*'$/.test(value) || /^".*"$/.test(value)) return value.slice(1, -1); + if (/^-?\d+$/.test(value)) return Number(value); + return value; +} + +function parseArgs(argv) { + const out = { runs: null, model: "sonnet", arm: "both", case: null }; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === "--runs") out.runs = Number(argv[++i]); + else if (argv[i] === "--model") out.model = argv[++i]; + else if (argv[i] === "--arm") out.arm = argv[++i]; + else if (argv[i] === "--case") out.case = argv[++i]; + else throw new Error(`Unknown option: ${argv[i]}`); + } + return out; +} + +function isDir(path) { + try { + return statSync(path).isDirectory(); + } catch { + return false; + } +} + +function isFile(path) { + try { + return statSync(path).isFile(); + } catch { + return false; + } +} From 3bd3a44158d33eaa29d0201324bc2544061b7066 Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Mon, 17 Aug 2026 16:15:09 +0300 Subject: [PATCH 2/9] test: price the ablation by tier instead of by total The first write-up read the total cost and concluded the plugin arm was a 2.3x net loss. The per-model split says something different: opus cost is flat at $0.140 against $0.135 while opus reads 34% fewer tokens, and the entire increase in the total is the sonnet subagent doing the file reading instead. That is the mechanism working, not the plugin costing - a total that goes up while the top tier holds flat is precisely the shape this plugin is trying to produce. Two corrections to the numbers that were published: the 2.3x came from averaging in the first run of the batch, which paid a cold prompt cache at $0.518 against $0.206 for the two warm runs that followed, and the totals were never split by tier, so the one number that matters was not visible at all. The runner now prints cost and input tokens per model per arm. The structural caveat is worth more than either number: a `-p` run is one shot, so the tokens kept out of the main context have no later turn to be re-read in and the saving has nowhere to accumulate. Every case written against this harness under-measures the plugin, and that belongs in the README rather than in someone's head. --- README.md | 2 +- evals/README.md | 53 +++++++++++++++++++++++++++++++-------------- evals/run-local.mjs | 22 +++++++++++++++++++ 3 files changed, 60 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index a9fd1e7..159b08e 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ Attribution over the same week, across all 266 dispatches, 199 of which ran belo The honest reading: the automatic half is fully attributable, the behavioural half depends on the session actually following the anchor, and these numbers have no control group of their own - this is the author's own workload measured with the author's own tool, with no before-install baseline. `CLAUDE_CODE_SUBAGENT_MODEL` was not set in this window, which the report would otherwise have annotated per row. -For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. On the first case the arms separate completely - the plugin arm delegated a codebase question to `scout` in 3 of 3 runs, the plugin-free arm delegated in 0 of 3 and grepped inline despite having `Explore` available - and the same run shows the other side of the trade: identical answers for 2.3x the money and 3x the wall clock on a four-file fixture. Guidance that changes behaviour is not automatically guidance that pays; the eval is there to keep both halves visible. +For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. On the first case the arms separate completely - the plugin arm delegated a codebase question to `scout` in 3 of 3 runs, the plugin-free arm delegated in 0 of 3 and grepped inline despite having `Explore` available. The price of that, per run: opus cost stayed flat ($0.140 against $0.135) while opus read 34% fewer tokens, and the whole increase in the total ($0.206 against $0.136) is the sonnet subagent doing the reading instead. The mechanism is visible even on a four-file fixture; the payoff is not, because a one-shot `-p` run has no later turn in which the spared context would have been re-read. The eval is there to keep both halves visible, including the half that says this case was too small to earn its dispatch. Where that volume actually ran with routing active: diff --git a/evals/README.md b/evals/README.md index c009ce2..21610c7 100644 --- a/evals/README.md +++ b/evals/README.md @@ -55,25 +55,46 @@ Opus main session, 3 runs per arm, 2026-08-17: | delegated to a subagent | 3/3 | 0/3 | +1.00 | | answer is correct | 3/3 | 3/3 | 0.00 | | scout was the agent (indicator) | 3/3 | 0/3 | - | -| cost / wall clock per run | $0.310 / 32s | $0.136 / 11s | +$0.174 / +21s | -Two things are true at once and both matter. The guidance does change what the -session does, cleanly and every time - the arms separate perfectly, and the -baseline had `Explore` available and still chose to grep inline. That is the -control-group number the README did not have. - -And on this fixture the change is a net loss: same answer, 2.3x the money, 3x -the wall clock. The plugin's own anchor says a dispatch costs more than the -answer for anything trivial, and four files is trivial - so the honest reading -is that the routing fired where its own rule says it should not have. What the -fixture cannot show is the other side of the trade: delegation exists to keep -file contents out of the main context, and a four-file project has no context -worth protecting. +The guidance does change what the session does, cleanly and every time - the +arms separate perfectly, and the baseline is not a straw man: it had `Explore` +available and still chose to grep inline. That is the control-group number the +README did not have. + +What it costs, per run, split by tier because the total hides the point: + +| | With | Without | +| --- | ---: | ---: | +| opus (the expensive tier) | $0.140, 59k in | $0.135, 90k in | +| the subagent | sonnet-5 $0.065, 94k in | - | +| total | $0.206 | $0.136 | +| wall clock | 31s | 11s | + +Opus cost is flat and opus read 34% fewer tokens: the file contents went to +sonnet instead, which is exactly the mechanism the plugin exists for, visible +even at this size. The whole increase in the total is the second model - you now +pay two models for a question one of them could answer alone. + +On a four-file fixture that is overhead and nothing else. The plugin's own anchor +says a dispatch costs more than the answer for anything trivial, and four files +is trivial, so the routing fired where its own rule says it should not have. + +The deeper limitation is structural, not a matter of fixture size: a `-p` run is +one shot. The 31k opus tokens that did not enter the main context have no later +turn to be re-read in, so the saving has nowhere to accumulate and the eval +measures the cost of delegation without the return. A real session re-reads its +context every turn, which is where the trade is supposed to flip. Any case +written here under-measures the plugin by construction, and a case that claims +otherwise is misreading its own numbers. + +(First run of a batch pays a cold prompt cache - it came in at $0.518 against +$0.206 for the two that followed. The figures above are the two warm runs; the +cold one is in the traces.) So the next case wants a fixture big enough that reading it inline actually -floods the session - that is where the trade is supposed to pay - and the case -after that wants the mirror of this one: a genuinely trivial question, where -dispatching at all is the failure being measured. +floods the session, and a multi-turn prompt so the saved context gets re-read at +least once. The case after that wants the mirror of this one: a genuinely +trivial question, where dispatching at all is the failure being measured. ## Writing another case diff --git a/evals/run-local.mjs b/evals/run-local.mjs index d2ce007..d3be9ea 100644 --- a/evals/run-local.mjs +++ b/evals/run-local.mjs @@ -56,6 +56,7 @@ for (const kase of cases) { costUsd: run.costUsd, durationMs: run.durationMs, turns: run.turns, + modelUsage: run.modelUsage, scores, }); const passed = scores.filter((s) => s.pass).length; @@ -121,6 +122,7 @@ async function runOnce(kase, arm, timeoutMs) { costUsd: result?.total_cost_usd ?? null, durationMs: result?.duration_ms ?? null, turns: result?.num_turns ?? null, + modelUsage: result?.modelUsage ?? {}, }; } @@ -169,6 +171,26 @@ function printCase(kase, armResults) { return `${arm} $${cost.toFixed(3)} / ${seconds.toFixed(0)}s`; }); console.log(` cost per run: ${price.join(" | ")}`); + + // The total hides the thing the plugin is actually for. Whether work left the + // expensive tier only shows in the per-model split: a total that went up while + // the top-tier line held flat is the plugin working, not the plugin costing. + for (const [arm, runs] of Object.entries(armResults)) { + const perModel = new Map(); + for (const run of runs) { + for (const [model, usage] of Object.entries(run.modelUsage ?? {})) { + const acc = perModel.get(model) ?? { cost: 0, read: 0, n: 0 }; + acc.cost += usage.costUSD ?? 0; + acc.read += (usage.cacheReadInputTokens ?? 0) + (usage.inputTokens ?? 0); + acc.n = runs.length; + perModel.set(model, acc); + } + } + const parts = [...perModel.entries()] + .sort((a, b) => b[1].cost - a[1].cost) + .map(([model, a]) => `${model} $${(a.cost / a.n).toFixed(3)} / ${Math.round(a.read / a.n / 1000)}k in`); + console.log(` ${arm} by model: ${parts.join(" | ")}`); + } } function mean(values) { From 1256b71255ddafa1abbde65d8af09415372b99ac Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Mon, 17 Aug 2026 17:34:42 +0300 Subject: [PATCH 3/9] test: add the wide-reading case that can answer whether routing saves money The small case could only ever show the cost of delegating, never the return. This adds the case that can show both, and the three things it took to get there - each of which was a measurement that came back negative first. The question has to be wide, because a needle question is answered by one grep and a grep costs the same in any session. The fixture is a twelve-stage pipeline where each stage hands off to exactly one other, no file states the order, and the stage modules are fat enough that walking the chain costs something. It ships as a generator rather than 85 committed modules. The session has to continue, because a single `-p` turn gives the spared context no later turn to be re-read in. Cases now take `followup-*.md` turns, run through `--continue` in the same session. And the subagent's tier has to be cheap enough, which is arithmetic rather than judgement: a fresh subagent pays cache write for everything it reads, a main session pays cache read for what it already has, and read is 12.5x cheaper than write at the same tier. Result over 4 turns, 3 runs per configuration, opus main session. The expensive tier drops either way - opus $1.169 with sonnet and $1.042 with haiku against $1.358 with no plugin - but the shipped `scout` pin ends up 23% more expensive in total, and haiku 9% cheaper. Every haiku run came in under the cheapest baseline run, every sonnet run over the dearest, and correctness held at 3/3 throughout. The plugin dispatched once per run and answered the follow-ups from the summary, which is what the guidance asks for. The `scout` pin is deliberately left as it ships. One workload that rewards breadth over judgement is not evidence that haiku answers a subtle question well enough, and a wrong answer that sends the main session back to re-read is the expensive failure mode this case does not measure. The numbers, the arithmetic, and what they do not license are in evals/README.md; the main README no longer implies the routing pays for itself unconditionally. --- .gitignore | 1 + README.md | 4 +- evals/README.md | 82 +++- evals/fixtures/order-service.gen.mjs | 426 ++++++++++++++++++ evals/run-local.mjs | 93 +++- evals/traces-a-flow-end-to-end/followup-1.md | 1 + evals/traces-a-flow-end-to-end/followup-2.md | 1 + evals/traces-a-flow-end-to-end/followup-3.md | 1 + .../graders/answer-is-correct.md | 14 + .../graders/delegated-to-a-subagent.md | 12 + evals/traces-a-flow-end-to-end/prompt.md | 11 + 11 files changed, 624 insertions(+), 22 deletions(-) create mode 100644 evals/fixtures/order-service.gen.mjs create mode 100644 evals/traces-a-flow-end-to-end/followup-1.md create mode 100644 evals/traces-a-flow-end-to-end/followup-2.md create mode 100644 evals/traces-a-flow-end-to-end/followup-3.md create mode 100644 evals/traces-a-flow-end-to-end/graders/answer-is-correct.md create mode 100644 evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md create mode 100644 evals/traces-a-flow-end-to-end/prompt.md diff --git a/.gitignore b/.gitignore index 9fd7840..5f00405 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ docs/superpowers/ .superpowers/ evals/results/ +evals/fixtures/order-service/ nul diff --git a/README.md b/README.md index 159b08e..c91f0a5 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,9 @@ Attribution over the same week, across all 266 dispatches, 199 of which ran belo The honest reading: the automatic half is fully attributable, the behavioural half depends on the session actually following the anchor, and these numbers have no control group of their own - this is the author's own workload measured with the author's own tool, with no before-install baseline. `CLAUDE_CODE_SUBAGENT_MODEL` was not set in this window, which the report would otherwise have annotated per row. -For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. On the first case the arms separate completely - the plugin arm delegated a codebase question to `scout` in 3 of 3 runs, the plugin-free arm delegated in 0 of 3 and grepped inline despite having `Explore` available. The price of that, per run: opus cost stayed flat ($0.140 against $0.135) while opus read 34% fewer tokens, and the whole increase in the total ($0.206 against $0.136) is the sonnet subagent doing the reading instead. The mechanism is visible even on a four-file fixture; the payoff is not, because a one-shot `-p` run has no later turn in which the spared context would have been re-read. The eval is there to keep both halves visible, including the half that says this case was too small to earn its dispatch. +For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. The behavioural claim holds - on a codebase question the plugin arm delegated to `scout` in 3 of 3 runs and the plugin-free arm in 0 of 3, grepping inline despite having `Explore` available. + +The money is a more interesting answer than "it saves". On a wide-reading question across a four-turn session, the expensive tier drops either way - opus down 14% - but the shipped configuration ends up **23% more expensive in total than not having the plugin at all**, because a fresh subagent pays cache *write* for everything it reads while a main session pays cache *read*, 12.5x cheaper, for what it already has. Pinning `scout` to haiku instead lands 9% below the baseline with correctness unchanged. Both distributions separate cleanly from the control in opposite directions. The pin has not been changed on the strength of one workload, and [`evals/README.md`](evals/README.md) carries the numbers, the arithmetic behind them, and what they do not license. Where that volume actually ran with routing active: diff --git a/evals/README.md b/evals/README.md index 21610c7..cabcad0 100644 --- a/evals/README.md +++ b/evals/README.md @@ -91,10 +91,64 @@ otherwise is misreading its own numbers. $0.206 for the two that followed. The figures above are the two warm runs; the cold one is in the traces.) -So the next case wants a fixture big enough that reading it inline actually -floods the session, and a multi-turn prompt so the saved context gets re-read at -least once. The case after that wants the mirror of this one: a genuinely -trivial question, where dispatching at all is the failure being measured. +## The second case, and what it took to make the numbers mean something + +`traces-a-flow-end-to-end` is the case that can answer "does this save money". +Three things had to be true before it could, and each one started out false: + +**The question has to be wide.** A needle question is answered by one grep, and a +grep costs the same in any session. The fixture is a twelve-stage pipeline where +each stage hands off to exactly one other and no file states the order, so the +chain has to be walked - and the stage modules are fat, so walking it costs +something. The first fixture was four small files and could never have shown +anything. + +**The session has to continue.** A single `-p` turn measures the cost of +delegating and none of the return. Follow-up turns (`followup-*.md`, run through +`--continue` in the same session) are where a main session pays again for the +context it carries: on one run the baseline re-wrote 80k of context to cache on +its third turn. + +**The subagent's tier has to be cheap enough.** This is the one that decided the +result, and it is arithmetic rather than judgement. A fresh subagent pays cache +*write* for everything it reads; a main session pays cache *read* for what it +already has, and read is 12.5x cheaper than write at the same tier. Delegation +converts cheap re-reads into expensive first-reads, so only a large tier discount +pays for the conversion. + +Opus main session, 4 turns, 3 runs per configuration. Per-run totals are shown +rather than only means, because with three runs a mean can hide an overlap: + +| Configuration | opus | subagent | total per run | +| --- | ---: | ---: | --- | +| no plugin | $1.358 | - | $1.339 $1.382 $1.356 | +| `scout` pinned sonnet (shipped) | $1.169 | sonnet-5 $0.506 | $1.559 $1.998 $1.468 | +| `scout` pinned haiku | $1.042 | haiku-4.5 $0.198 | $1.142 $1.260 $1.315 | + +Both configurations separate cleanly from the baseline, in opposite directions. +Every haiku run came in under the cheapest baseline run; every sonnet run came in +over the dearest. Correctness held at 3/3 throughout, and the plugin dispatched +exactly once per run - the follow-ups were answered from `scout`'s summary rather +than by dispatching again, which is what the guidance asks for. + +So the shipped configuration costs 23% more than not having the plugin on this +workload, and pinning `scout` to haiku costs 9% less. The expensive tier drops +either way - opus down 14% with sonnet, 23% with haiku - which is the mechanism +working in both cases. What differs is whether the subagent's bill comes to less +than the saving it produces. + +Two things this does not license. One case is no mandate to re-pin `scout`: this +question rewards breadth over judgement, haiku's answer quality on a subtle +question is untested, and a wrong answer that sends the main session back to +re-read is the expensive failure mode nothing here measures. And the largest +single line in every run is opus *output* tokens, $0.33 to $0.43, which no amount +of tier routing touches. + +What the eval still cannot see is context-window pressure. Four turns never +compact; a real session does, and a compaction costs a full re-read plus lost +fidelity. That is a cost the baseline dodges here and would not dodge in +practice, so read these numbers as the floor of the plugin's case rather than the +whole of it. ## Writing another case @@ -107,6 +161,20 @@ tell them. `run-local.mjs` supports `type: regex` over `target: trace` and `tool_order`, `file_exists`, `llm`, and `baseline`. Prefer the deterministic ones: a judge model scoring prose is one more thing that can be wrong. -Cases share one fixture, `evals/fixtures/mini-app`. Keep it boring. A fixture -that is interesting to read is a fixture the model answers from memory of -something similar. +A case names its fixture directory under `evals/fixtures` with a `fixture:` key, +defaulting to `mini-app`. A fixture too large to commit ships as +`.gen.mjs` beside it and is generated on first use - `order-service` is 85 +modules and 122k of source, which nobody installing this plugin should have to +download. Keep fixtures boring: one that is interesting to read is one the model +answers from memory of something similar. + +Add `followup-1.md`, `followup-2.md` and so on to turn a case into a session +rather than a single question. That is not decoration - it is the only way the +context a subagent kept out of the main session gets a chance to pay for itself. + +## Next + +A case that stresses answer *quality* rather than breadth, run against both +`scout` pins, so re-pinning has evidence behind it instead of one workload. Then +the mirror of the small case: a genuinely trivial question, where dispatching at +all is the failure being measured. diff --git a/evals/fixtures/order-service.gen.mjs b/evals/fixtures/order-service.gen.mjs new file mode 100644 index 0000000..c0a36dd --- /dev/null +++ b/evals/fixtures/order-service.gen.mjs @@ -0,0 +1,426 @@ +#!/usr/bin/env node +// Generates the `order-service` fixture: a pipeline whose shape can only be +// learned by reading it. Committed as a generator rather than as files because +// the fixture runs to a few hundred kilobytes and every user of this plugin +// would otherwise install it. +// +// Two properties matter, and both are deliberate. +// +// The question is broad - trace the flow end to end - because a narrow one is +// answered by a single grep, and a grep costs the same whether or not it happens +// in the expensive session. Delegation can only pay where the reading is wide. +// +// The stage modules are fat. Tracing a chain of small files is cheap no matter +// how many files there are, so each hop has to cost something to read: the +// handoff sits at the bottom of a few hundred lines of plausible logic, and the +// helpers name other stages so that grepping for a stage name is noisy. +// +// Deterministic: same output every run, no randomness, no network. + +import { mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const OUT = join(dirname(fileURLToPath(import.meta.url)), "order-service"); + +const STAGES = [ + ["intake", "acceptOrder", "validation", "the raw payload from the till"], + ["validation", "validateOrder", "pricing", "line items against the catalog"], + ["pricing", "priceOrder", "discount", "unit prices before discounts"], + ["discount", "applyDiscounts", "tax", "campaign and loyalty reductions"], + ["tax", "applyTax", "deposit", "VAT groups per line"], + ["deposit", "addDeposits", "payment", "returnable packaging charges"], + ["payment", "capturePayment", "tips", "the authorized amount"], + ["tips", "splitTips", "fiscal", "gratuity across the shift"], + ["fiscal", "signReceipt", "receipt", "the signature from the TSE"], + ["receipt", "renderReceipt", "export", "the printable document"], + ["export", "exportOrder", "archive", "the audit record"], + ["archive", "archiveOrder", null, "the closed order"], +]; + +const HELPERS = ["normalize", "guard", "metrics", "errors", "mapping", "retry"]; + +rmSync(OUT, { recursive: true, force: true }); + +for (const [stage, fn, next, subject] of STAGES) { + const dir = join(OUT, "src", stage); + mkdirSync(dir, { recursive: true }); + const nextStage = STAGES.find((s) => s[0] === next); + writeFileSync(join(dir, "index.js"), stageSource(stage, fn, subject, nextStage), "utf-8"); + for (const helper of HELPERS) writeFileSync(join(dir, `${helper}.js`), helperSource(helper, stage), "utf-8"); +} + +writeFileSync( + join(OUT, "README.md"), + `# order-service + +Fixture for the routing evals. A generated pipeline: every stage lives in its own +directory and hands off to exactly one other stage, and no file anywhere states +the order of the stages. Working it out means reading them. +`, + "utf-8", +); + +const files = STAGES.length * (HELPERS.length + 1) + 1; +console.log(`Generated ${files} modules in ${OUT}`); + +function cap(word) { + return word[0].toUpperCase() + word.slice(1); +} + +// Every other stage name, so that grepping for one stage hits many files and +// the chain still has to be read rather than pattern-matched. +function otherStages(stage) { + return STAGES.map((s) => s[0]).filter((s) => s !== stage); +} + +function stageSource(stage, fn, subject, nextStage) { + const S = cap(stage); + const imports = [ + ...HELPERS.map((h) => `import { ${h}${S} } from "./${h}.js";`), + nextStage ? `import { ${nextStage[1]} } from "../${nextStage[0]}/index.js";` : "", + ] + .filter(Boolean) + .join("\n"); + + return `${imports} + +// This module owns ${subject}. Everything below it is stage-local: the handoff +// to the next stage is the last thing in the file. + +const ${stage.toUpperCase()}_LIMITS = { + maxLines: 500, + maxQuantityPerLine: 999, + maxTotalMinor: 100_000_00, + maxRetries: 3, + softTimeoutMs: 2_500, +}; + +const ${stage.toUpperCase()}_FEATURES = ${JSON.stringify( + Object.fromEntries(otherStages(stage).map((other, i) => [`${other}Aware`, i % 3 === 0])), + null, + 2, + )}; + +${block(stage, "rules")} + +${block(stage, "totals")} + +${block(stage, "audit")} + +${block(stage, "context")} + +${block(stage, "diff")} + +${block(stage, "batching")} + +${block(stage, "policy")} + +${block(stage, "serialize")} + +export async function ${fn}(order, context) { + const checked = guard${S}(order, context); + const shaped = normalize${S}(checked); + const violations = collect${S}Violations(shaped, context); + if (violations.length > 0) throw errors${S}(new Error(violations.join("; ")), shaped); + + const totals = compute${S}Totals(shaped); + const enriched = { ...shaped, totals, audit: build${S}Audit(shaped, context) }; + metrics${S}(enriched, context); + + const batches = split${S}Batches(enriched); + const settled = batches.map((batch) => apply${S}Policy(batch, context)); + const merged = merge${S}Diff(enriched, settled); + + try { +${ + nextStage + ? ` return await withRetry${S}(() => ${nextStage[1]}(merged, context), ${stage.toUpperCase()}_LIMITS.maxRetries);` + : " return serialize" + S + "(merged);" +} + } catch (cause) { + throw errors${S}(cause, merged); + } +} +`; +} + +// A bank of plausible stage-local logic. Each block is 30-60 lines so that a +// stage module costs something real to read. +function block(stage, kind) { + const S = cap(stage); + const U = stage.toUpperCase(); + const banks = { + rules: `const ${U}_RULES = [ + { code: "empty-lines", severity: "fatal", test: (order) => (order.lines ?? []).length === 0 }, + { code: "too-many-lines", severity: "fatal", test: (order) => (order.lines ?? []).length > ${U}_LIMITS.maxLines }, + { code: "negative-quantity", severity: "fatal", test: (order) => (order.lines ?? []).some((l) => l.quantity < 0) }, + { code: "quantity-ceiling", severity: "warn", test: (order) => (order.lines ?? []).some((l) => l.quantity > ${U}_LIMITS.maxQuantityPerLine) }, + { code: "missing-sku", severity: "fatal", test: (order) => (order.lines ?? []).some((l) => !l.sku) }, + { code: "duplicate-sku", severity: "warn", test: (order) => { + const seen = new Set(); + return (order.lines ?? []).some((l) => (seen.has(l.sku) ? true : (seen.add(l.sku), false))); + } }, + { code: "no-currency", severity: "fatal", test: (order) => !order.currency }, + { code: "stale-order", severity: "warn", test: (order) => Number(order.openedAt ?? 0) > 0 && Date.now() - order.openedAt > 86_400_000 }, +]; + +export function collect${S}Violations(order, context) { + const violations = []; + for (const rule of ${U}_RULES) { + let failed = false; + try { + failed = rule.test(order); + } catch { + failed = true; + } + if (!failed) continue; + if (rule.severity === "warn" && context?.lenient) continue; + violations.push(\`\${rule.code}\`); + } + return violations; +} + +export function describe${S}Rules() { + return ${U}_RULES.map((rule) => ({ code: rule.code, severity: rule.severity })); +}`, + + totals: `export function compute${S}Totals(order) { + const lines = order.lines ?? []; + let net = 0; + let gross = 0; + let discount = 0; + let quantity = 0; + + for (const line of lines) { + const lineNet = Math.round(line.unitPrice * line.quantity * 100) / 100; + const lineDiscount = Math.round((line.discountMinor ?? 0)) / 100; + net += lineNet; + discount += lineDiscount; + gross += lineNet - lineDiscount + (line.taxMinor ?? 0) / 100; + quantity += line.quantity; + } + + return { + lineCount: lines.length, + quantity, + net: round2(net), + discount: round2(discount), + gross: round2(gross), + average: lines.length === 0 ? 0 : round2(net / lines.length), + }; +} + +function round2(value) { + return Math.round(value * 100) / 100; +}`, + + audit: `export function build${S}Audit(order, context) { + return { + stage: "${stage}", + orderId: order.id, + tenantId: context?.tenantId ?? null, + operator: context?.operator?.id ?? null, + device: context?.device?.serial ?? null, + lineCount: (order.lines ?? []).length, + features: Object.entries(${U}_FEATURES) + .filter(([, on]) => on) + .map(([name]) => name), + }; +} + +export function summarize${S}Audit(entries) { + const byOperator = new Map(); + for (const entry of entries ?? []) { + const key = entry.operator ?? "unknown"; + byOperator.set(key, (byOperator.get(key) ?? 0) + 1); + } + return [...byOperator.entries()].map(([operator, count]) => ({ operator, count })); +}`, + + context: `export function derive${S}Context(context) { + return { + tenantId: context?.tenantId ?? null, + lenient: Boolean(context?.lenient), + locale: context?.locale ?? "de-DE", + timezone: context?.timezone ?? "Europe/Berlin", + softTimeoutMs: context?.softTimeoutMs ?? ${U}_LIMITS.softTimeoutMs, + }; +} + +export function assert${S}Context(context) { + const derived = derive${S}Context(context); + if (!derived.tenantId) throw new Error("${stage}: context carries no tenant"); + if (derived.softTimeoutMs <= 0) throw new Error("${stage}: soft timeout must be positive"); + return derived; +}`, + + diff: `export function merge${S}Diff(base, patches) { + let merged = { ...base }; + for (const patch of patches ?? []) { + if (!patch) continue; + merged = { + ...merged, + ...patch, + lines: mergeLines(merged.lines ?? [], patch.lines ?? []), + totals: { ...(merged.totals ?? {}), ...(patch.totals ?? {}) }, + }; + } + return merged; +} + +function mergeLines(left, right) { + if (right.length === 0) return left; + const bySku = new Map(left.map((line) => [line.sku, line])); + for (const line of right) { + const existing = bySku.get(line.sku); + bySku.set(line.sku, existing ? { ...existing, ...line } : line); + } + return [...bySku.values()]; +}`, + + batching: `export function split${S}Batches(order) { + const lines = order.lines ?? []; + if (lines.length <= 25) return [order]; + + const batches = []; + for (let i = 0; i < lines.length; i += 25) { + batches.push({ ...order, lines: lines.slice(i, i + 25), batchIndex: batches.length }); + } + return batches; +} + +export function joinBatches(batches) { + return (batches ?? []).reduce( + (acc, batch) => ({ ...batch, lines: [...(acc.lines ?? []), ...(batch.lines ?? [])] }), + {}, + ); +}`, + + policy: `const ${U}_POLICY = { + onWarn: "continue", + onFatal: "abort", + onTimeout: "retry", + onUnknown: "abort", +}; + +export function apply${S}Policy(batch, context) { + const decision = ${U}_POLICY[batch?.outcome ?? "onUnknown"] ?? ${U}_POLICY.onUnknown; + if (decision === "abort" && !context?.lenient) return null; + return { ...batch, decision }; +} + +export function ${stage}PolicyFor(outcome) { + return ${U}_POLICY[outcome] ?? ${U}_POLICY.onUnknown; +}`, + + serialize: `export function serialize${S}(order) { + return { + id: order.id, + stage: "${stage}", + currency: order.currency ?? "EUR", + totals: order.totals ?? null, + lines: (order.lines ?? []).map((line) => ({ + sku: line.sku, + quantity: line.quantity, + unitPrice: line.unitPrice, + taxMinor: line.taxMinor ?? 0, + discountMinor: line.discountMinor ?? 0, + })), + audit: order.audit ?? null, + }; +} + +export function deserialize${S}(payload) { + return { ...payload, lines: payload.lines ?? [], stage: "${stage}" }; +}`, + }; + return banks[kind]; +} + +function helperSource(helper, stage) { + const S = cap(stage); + const others = otherStages(stage); + const bodies = { + normalize: `// Shared shaping. The stages that consume this output are +// ${others.slice(0, 4).join(", ")}, so keep the field names stable. +export function normalize${S}(order) { + const lines = (order.lines ?? []).map((line) => ({ + ...line, + sku: String(line.sku ?? "").trim(), + quantity: Number(line.quantity ?? 1), + unitPrice: Math.round(Number(line.unitPrice ?? 0) * 100) / 100, + taxMinor: Math.round(Number(line.taxMinor ?? 0)), + discountMinor: Math.round(Number(line.discountMinor ?? 0)), + })); + return { ...order, lines, stage: "${stage}", currency: order.currency ?? "EUR" }; +}`, + guard: `const REQUIRED = ["id", "lines"]; + +// Mirrors the checks the ${others[0]} and ${others[1]} stages make on their own input. +export function guard${S}(order, context) { + for (const field of REQUIRED) { + if (order?.[field] == null) throw new Error(\`${stage}: missing \${field}\`); + } + if (context?.tenantId == null) throw new Error("${stage}: no tenant in context"); + if (!Array.isArray(order.lines)) throw new Error("${stage}: lines must be an array"); + return order; +}`, + metrics: `let counter = 0; + +// Counter names are compared against the ${others[2]} stage in dashboards. +export function metrics${S}(order, context) { + counter += 1; + context?.telemetry?.record?.({ stage: "${stage}", orderId: order.id, seq: counter }); + return counter; +} + +export function reset${S}Metrics() { + counter = 0; +}`, + errors: `export function errors${S}(cause, order) { + const error = new Error(\`${stage} failed for order \${order?.id ?? "unknown"}\`); + error.cause = cause; + error.stage = "${stage}"; + error.retryable = cause?.retryable ?? false; + return error; +} + +// The ${others[3]} stage unwraps these, so the shape is load-bearing. +export function unwrap${S}(error) { + return { stage: error?.stage ?? "${stage}", retryable: Boolean(error?.retryable), cause: error?.cause ?? null }; +}`, + mapping: `// Field names as they leave ${stage}. The ${others[4]} stage reads the same keys. +const FIELDS = { + id: "orderId", + currency: "currencyCode", + totals: "totalsBlock", + audit: "auditTrail", +}; + +export function mapping${S}(order) { + const out = {}; + for (const [from, to] of Object.entries(FIELDS)) out[to] = order?.[from] ?? null; + return out; +} + +export function unmapping${S}(payload) { + const out = {}; + for (const [from, to] of Object.entries(FIELDS)) out[from] = payload?.[to] ?? null; + return out; +}`, + retry: `export async function withRetry${S}(operation, attempts) { + let last; + for (let attempt = 0; attempt < attempts; attempt++) { + try { + return await operation(); + } catch (error) { + last = error; + if (error?.retryable === false) break; + } + } + throw last; +}`, + }; + return `${bodies[helper]}\n`; +} diff --git a/evals/run-local.mjs b/evals/run-local.mjs index d3be9ea..f736521 100644 --- a/evals/run-local.mjs +++ b/evals/run-local.mjs @@ -9,7 +9,7 @@ // graders over `trace` and `last_message`. Anything else is reported as skipped // rather than silently scored. -import { spawn } from "node:child_process"; +import { execFileSync, spawn } from "node:child_process"; import { cpSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; import { dirname, join, resolve } from "node:path"; import { tmpdir } from "node:os"; @@ -17,9 +17,7 @@ import { fileURLToPath } from "node:url"; const EVALS_DIR = dirname(fileURLToPath(import.meta.url)); const PLUGIN_ROOT = resolve(EVALS_DIR, ".."); -// The real harness gets its workspace from case.yaml `context.scaffold_script`. -// Until a run can verify that field, every case here shares one fixture. -const FIXTURE = join(EVALS_DIR, "fixtures", "mini-app"); +const FIXTURES_DIR = join(EVALS_DIR, "fixtures"); const args = parseArgs(process.argv.slice(2)); const cases = discoverCases().filter((c) => !args.case || c.name.includes(args.case)); @@ -73,13 +71,73 @@ console.log(`\nTraces and aggregate.json: ${resultsDir}`); // --- running ------------------------------------------------------------- +// The real harness gets its workspace from case.yaml `context.scaffold_script`, +// which needs a run to verify. Until then a case names its fixture directory +// under evals/fixtures; a fixture too large to commit ships as `.gen.mjs` +// next to it and is generated on first use. +function fixtureFor(kase) { + const name = args.fixture ?? kase.frontmatter.fixture ?? "mini-app"; + const dir = join(FIXTURES_DIR, name); + if (isDir(dir)) return dir; + + const generator = join(FIXTURES_DIR, `${name}.gen.mjs`); + if (!isFile(generator)) throw new Error(`No fixture "${name}" and no ${name}.gen.mjs to build one`); + execFileSync(process.execPath, [generator], { stdio: "inherit" }); + if (!isDir(dir)) throw new Error(`${name}.gen.mjs ran but produced no ${dir}`); + return dir; +} + +// One turn measures the cost of delegating and none of the return: the context +// a subagent kept out of the main session only pays off when a later turn would +// have re-read it. A case with follow-ups runs them in the same session so that +// re-reading actually happens. async function runOnce(kase, arm, timeoutMs) { const workspace = mkdtempSync(join(tmpdir(), "model-routing-eval-")); - cpSync(FIXTURE, workspace, { recursive: true }); + cpSync(fixtureFor(kase), workspace, { recursive: true }); + + const turns = []; + for (const [index, prompt] of [kase.prompt, ...kase.followups].entries()) { + turns.push(await runTurn(prompt, arm, workspace, timeoutMs, index > 0)); + } + + try { + rmSync(workspace, { recursive: true, force: true }); + } catch { + // A file left open by the child is not worth failing a run over. + } + + return { + trace: turns.map((t) => t.trace).join("\n"), + stderr: turns.map((t) => t.stderr).join("\n"), + exitCode: turns.find((t) => t.exitCode !== 0)?.exitCode ?? 0, + lastMessage: turns.map((t) => t.lastMessage).join("\n\n"), + costUsd: sum(turns.map((t) => t.costUsd)), + durationMs: sum(turns.map((t) => t.durationMs)), + turns: sum(turns.map((t) => t.turns)), + modelUsage: mergeModelUsage(turns.map((t) => t.modelUsage)), + }; +} +function sum(values) { + return values.reduce((a, b) => a + (b ?? 0), 0); +} + +function mergeModelUsage(all) { + const merged = {}; + for (const usage of all) { + for (const [model, u] of Object.entries(usage ?? {})) { + const acc = (merged[model] ??= { costUSD: 0, inputTokens: 0, cacheReadInputTokens: 0, cacheCreationInputTokens: 0, outputTokens: 0 }); + for (const key of Object.keys(acc)) acc[key] += u[key] ?? 0; + } + } + return merged; +} + +async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup) { const argv = [ "-p", - kase.prompt, + prompt, + ...(isFollowup ? ["--continue"] : []), "--output-format", "stream-json", "--verbose", @@ -107,12 +165,6 @@ async function runOnce(kase, arm, timeoutMs) { const exitCode = await new Promise((res) => child.on("close", res)); clearTimeout(timer); - try { - rmSync(workspace, { recursive: true, force: true }); - } catch { - // A file left open by the child is not worth failing a run over. - } - const result = extractResult(stdout); return { trace: stdout, @@ -222,7 +274,19 @@ function loadCase(dir) { ...splitFrontmatter(readFileSync(join(gradersDir, f), "utf-8")).frontmatter, })) : []; - return { name: frontmatter.name ?? dir.split(/[\\/]/).pop(), frontmatter, prompt: body.trim(), graders }; + // followup-1.md, followup-2.md ... are further turns of the same session. + const followups = readdirSync(dir) + .filter((f) => /^followup-\d+\.md$/.test(f)) + .sort() + .map((f) => splitFrontmatter(readFileSync(join(dir, f), "utf-8")).body.trim()); + + return { + name: frontmatter.name ?? dir.split(/[\\/]/).pop(), + frontmatter, + prompt: body.trim(), + followups, + graders, + }; } // Enough YAML for this format: scalars, quoted scalars, and inline lists. A @@ -254,12 +318,13 @@ function parseScalar(raw) { } function parseArgs(argv) { - const out = { runs: null, model: "sonnet", arm: "both", case: null }; + const out = { runs: null, model: "sonnet", arm: "both", case: null, fixture: null }; for (let i = 0; i < argv.length; i++) { if (argv[i] === "--runs") out.runs = Number(argv[++i]); else if (argv[i] === "--model") out.model = argv[++i]; else if (argv[i] === "--arm") out.arm = argv[++i]; else if (argv[i] === "--case") out.case = argv[++i]; + else if (argv[i] === "--fixture") out.fixture = argv[++i]; else throw new Error(`Unknown option: ${argv[i]}`); } return out; diff --git a/evals/traces-a-flow-end-to-end/followup-1.md b/evals/traces-a-flow-end-to-end/followup-1.md new file mode 100644 index 0000000..4f06479 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/followup-1.md @@ -0,0 +1 @@ +If the payment stage throws an error that is not retryable, what does the order end up as, and which stage decides that? Answer from the code, not from convention. diff --git a/evals/traces-a-flow-end-to-end/followup-2.md b/evals/traces-a-flow-end-to-end/followup-2.md new file mode 100644 index 0000000..2d4e7eb --- /dev/null +++ b/evals/traces-a-flow-end-to-end/followup-2.md @@ -0,0 +1 @@ +I want to add a service charge that applies after discounts but before VAT. Which stage should own it, and what would it have to hand the next stage so nothing downstream breaks? diff --git a/evals/traces-a-flow-end-to-end/followup-3.md b/evals/traces-a-flow-end-to-end/followup-3.md new file mode 100644 index 0000000..fb70a67 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/followup-3.md @@ -0,0 +1 @@ +Which stages split their work into batches before handing off, and what is the batch size? List them. diff --git a/evals/traces-a-flow-end-to-end/graders/answer-is-correct.md b/evals/traces-a-flow-end-to-end/graders/answer-is-correct.md new file mode 100644 index 0000000..e128d11 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/graders/answer-is-correct.md @@ -0,0 +1,14 @@ +--- +type: regex +target: last_message +pattern: 'intake[\s\S]*validation[\s\S]*pricing[\s\S]*discount[\s\S]*tax[\s\S]*deposit[\s\S]*payment[\s\S]*tips[\s\S]*fiscal[\s\S]*receipt[\s\S]*export[\s\S]*archive' +flags: 'i' +match: contains +--- + +All twelve stages, in the right order. The order is the answer: the stage names +are guessable from the directory listing alone, so an answer scored on the names +without the sequence would pass without anybody reading anything. + +No file states the sequence. Each stage imports the next at the bottom of a few +hundred lines, so the chain has to be walked. diff --git a/evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md b/evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md new file mode 100644 index 0000000..5ad7776 --- /dev/null +++ b/evals/traces-a-flow-end-to-end/graders/delegated-to-a-subagent.md @@ -0,0 +1,12 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Agent|Task)"' +match: contains +--- + +Same behavioural claim as the small case, on a question where the reading is +actually wide: twelve stage modules at roughly 2k tokens each have to pass under +somebody's eyes, and which model's eyes those are is the entire point. + +Both arms can pass - the baseline has `Explore` and `general-purpose`. diff --git a/evals/traces-a-flow-end-to-end/prompt.md b/evals/traces-a-flow-end-to-end/prompt.md new file mode 100644 index 0000000..61928da --- /dev/null +++ b/evals/traces-a-flow-end-to-end/prompt.md @@ -0,0 +1,11 @@ +--- +name: traces-a-flow-end-to-end +tags: [routing, scout, wide-reading] +plugins: ["../.."] +fixture: order-service +runs: 3 +max_turns: 20 +timeout_seconds: 600 +--- + +An order enters this service at intake and ends up archived. Trace the whole path: list every stage it passes through, in order, from intake to archive. From 5674e9bb8d342f72694a4b18b3d26c954a0839e5 Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Tue, 18 Aug 2026 03:37:10 +0300 Subject: [PATCH 4/9] test: isolate the tier question from the coin flip, and stop before trusting it Whether a session delegates at all turns out to be a coin it flips - 3/3, 3/3, 2/3, 2/3 and then 0/3 across otherwise identical batches - so a case meaning to price a tier cannot also be measuring that. This adds the case that takes the decision away: identical prompt and follow-ups to the wide-reading case, delegation required, the subagent's tier the only variable left. Getting "the main session does not read the files itself" to actually hold took three attempts, and the first two produced numbers that looked fine. Asking in the system prompt was ignored. Denying Read, Grep and Glob moved the reading into Bash - three opus runs were spent before a tool census showed `cat`, `grep -n` and `sed -n` doing the work, and those figures were void. Denying the shells left ToolSearch, which loads a deferred Read back in. An allowlist holds, so the runner now takes `tools:` from frontmatter and the grader watches the shells too. The case is committed unfinished and tagged as such. With the allowlist in place the answers come back wrong on both haiku and opus - one dispatch, nothing left to check it with - and a configuration that is cheap and wrong has saved nothing, so the tier comparison is not run and no number from this case is reported. NOTES.md carries the intended configurations, the three failed isolation attempts, and the two candidate fixes. `scout` stays pinned sonnet, as it ships. --- evals/forced-dispatch-tier/NOTES.md | 60 +++++++++++++++++++ evals/forced-dispatch-tier/followup-1.md | 1 + evals/forced-dispatch-tier/followup-2.md | 1 + evals/forced-dispatch-tier/followup-3.md | 1 + .../graders/answer-is-correct.md | 12 ++++ .../graders/delegated-to-a-subagent.md | 11 ++++ .../graders/main-session-read-nothing.md | 19 ++++++ evals/forced-dispatch-tier/prompt.md | 13 ++++ evals/run-local.mjs | 30 +++++++++- 9 files changed, 146 insertions(+), 2 deletions(-) create mode 100644 evals/forced-dispatch-tier/NOTES.md create mode 100644 evals/forced-dispatch-tier/followup-1.md create mode 100644 evals/forced-dispatch-tier/followup-2.md create mode 100644 evals/forced-dispatch-tier/followup-3.md create mode 100644 evals/forced-dispatch-tier/graders/answer-is-correct.md create mode 100644 evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md create mode 100644 evals/forced-dispatch-tier/graders/main-session-read-nothing.md create mode 100644 evals/forced-dispatch-tier/prompt.md diff --git a/evals/forced-dispatch-tier/NOTES.md b/evals/forced-dispatch-tier/NOTES.md new file mode 100644 index 0000000..195e5b5 --- /dev/null +++ b/evals/forced-dispatch-tier/NOTES.md @@ -0,0 +1,60 @@ +# forced-dispatch-tier - unfinished, do not read numbers off it yet + +## What it is for + +The unforced cases cannot answer "is delegating at the session tier - what +Superpowers does by default - more expensive than delegating at a lower one", +because whether a session delegates at all is a coin it flips: observed 3/3, +3/3, 2/3, 2/3 and 0/3 across otherwise identical batches. This case takes the +decision away and holds everything else identical to +`traces-a-flow-end-to-end`, so the only variable left is the subagent's tier. + +Intended configurations, same prompt and follow-ups throughout: + +| Configuration | What it stands for | +| --- | --- | +| no plugin, unforced (already measured, $1.359) | ordinary work, no delegation | +| no plugin, forced | subagent inherits the session model - Superpowers | +| plugin, `scout` pinned sonnet | what ships today | +| plugin, `scout` pinned haiku | the change the wide case argues for | + +## Where it stands + +Isolation works and took three attempts. Every attempt is worth knowing about, +because each looked like it had worked: + +1. Asking in the system prompt: the session read the files anyway. +2. Denying `Read`, `Grep`, `Glob`: the session read the whole codebase through + Bash instead - `cat`, `grep -n`, `sed -n`. Three opus runs were spent before + the tool census showed it, and their cost figures were meaningless. +3. Denying the shells too: the session pulled a deferred `Read` back in through + `ToolSearch`. +4. An allowlist (`tools: [Task, Agent, TodoWrite]`) holds. Nothing to read with, + nothing to recover. + +The `main-session-read-nothing` grader now covers the shells and is the thing to +check before believing any cost number from this case. + +## The open problem + +With the allowlist in place the answer comes back **wrong** - on haiku and on +opus alike, one dispatch and no way to check the result. A configuration that is +cheap and wrong has saved nothing, so the tier comparison cannot be run until +the case can produce correct answers. + +Two candidate fixes, untested: + +- Let the main session dispatch repeatedly and verify through further dispatches + rather than by reading. That is closer to what a real delegating session does + and may just need a longer `max_turns` plus a system prompt that says to check + the subagent's answer. +- Put the verification burden on the grader instead: score the subagent's report + rather than the main session's final message. + +The first is the honest one. The second measures a different thing. + +## Also open + +`scout` is still pinned sonnet, as it ships. The wide case says haiku is 9% +cheaper on that workload with correctness held; that is one workload, and the +quality case that would justify re-pinning has not been written. diff --git a/evals/forced-dispatch-tier/followup-1.md b/evals/forced-dispatch-tier/followup-1.md new file mode 100644 index 0000000..4f06479 --- /dev/null +++ b/evals/forced-dispatch-tier/followup-1.md @@ -0,0 +1 @@ +If the payment stage throws an error that is not retryable, what does the order end up as, and which stage decides that? Answer from the code, not from convention. diff --git a/evals/forced-dispatch-tier/followup-2.md b/evals/forced-dispatch-tier/followup-2.md new file mode 100644 index 0000000..2d4e7eb --- /dev/null +++ b/evals/forced-dispatch-tier/followup-2.md @@ -0,0 +1 @@ +I want to add a service charge that applies after discounts but before VAT. Which stage should own it, and what would it have to hand the next stage so nothing downstream breaks? diff --git a/evals/forced-dispatch-tier/followup-3.md b/evals/forced-dispatch-tier/followup-3.md new file mode 100644 index 0000000..fb70a67 --- /dev/null +++ b/evals/forced-dispatch-tier/followup-3.md @@ -0,0 +1 @@ +Which stages split their work into batches before handing off, and what is the batch size? List them. diff --git a/evals/forced-dispatch-tier/graders/answer-is-correct.md b/evals/forced-dispatch-tier/graders/answer-is-correct.md new file mode 100644 index 0000000..a22ae40 --- /dev/null +++ b/evals/forced-dispatch-tier/graders/answer-is-correct.md @@ -0,0 +1,12 @@ +--- +type: regex +target: last_message +pattern: 'intake[\s\S]*validation[\s\S]*pricing[\s\S]*discount[\s\S]*tax[\s\S]*deposit[\s\S]*payment[\s\S]*tips[\s\S]*fiscal[\s\S]*receipt[\s\S]*export[\s\S]*archive' +flags: 'i' +match: contains +--- + +All twelve stages in the right order, same as the unforced case. It matters more +here: the main session never reads a file, so this scores whether the subagent's +tier is capable of the work, not just whether it is cheap. A tier that saves +money and gets the order wrong has saved nothing. diff --git a/evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md b/evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md new file mode 100644 index 0000000..b4e2a3f --- /dev/null +++ b/evals/forced-dispatch-tier/graders/delegated-to-a-subagent.md @@ -0,0 +1,11 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Agent|Task)"' +match: contains +--- + +Here this is a precondition rather than a finding: the system prompt orders the +delegation, so a run that fails this grader did not do the thing being priced and +its cost figure means nothing. Check it before reading any money number from +this case. diff --git a/evals/forced-dispatch-tier/graders/main-session-read-nothing.md b/evals/forced-dispatch-tier/graders/main-session-read-nothing.md new file mode 100644 index 0000000..e7cad92 --- /dev/null +++ b/evals/forced-dispatch-tier/graders/main-session-read-nothing.md @@ -0,0 +1,19 @@ +--- +type: regex +target: trace +pattern: '"name"\s*:\s*"(Read|Grep|Glob|Bash|PowerShell)"' +match: not_contains +--- + +The main session must not read the project itself, or the run is a blend of both +strategies and its cost lands between the two things being compared - worse than +a missing data point, because it looks like one. + +Banning Read, Grep and Glob alone does not achieve this. The first attempt did +exactly that and the session simply read the whole codebase through Bash - +`cat`, `grep -n`, `sed -n` - and the cost figures from those runs were +meaningless. Shell counts as reading, so the shells are banned too and this +grader watches for them. + +Subagent tool calls do not appear in the parent transcript, so this only sees +what the main session did, which is exactly the scope it needs. diff --git a/evals/forced-dispatch-tier/prompt.md b/evals/forced-dispatch-tier/prompt.md new file mode 100644 index 0000000..10062c5 --- /dev/null +++ b/evals/forced-dispatch-tier/prompt.md @@ -0,0 +1,13 @@ +--- +name: forced-dispatch-tier +tags: [routing, tier, wide-reading, unfinished] +plugins: ["../.."] +fixture: order-service +runs: 3 +max_turns: 20 +timeout_seconds: 600 +append_system_prompt: 'Delegate all codebase exploration to a subagent. Do not read, grep, or glob project files in the main session - dispatch an agent to do it and work from what it reports back.' +tools: [Task, Agent, TodoWrite] +--- + +An order enters this service at intake and ends up archived. Trace the whole path: list every stage it passes through, in order, from intake to archive. diff --git a/evals/run-local.mjs b/evals/run-local.mjs index f736521..d83d2a4 100644 --- a/evals/run-local.mjs +++ b/evals/run-local.mjs @@ -97,7 +97,17 @@ async function runOnce(kase, arm, timeoutMs) { const turns = []; for (const [index, prompt] of [kase.prompt, ...kase.followups].entries()) { - turns.push(await runTurn(prompt, arm, workspace, timeoutMs, index > 0)); + turns.push( + await runTurn( + prompt, + arm, + workspace, + timeoutMs, + index > 0, + kase.frontmatter.append_system_prompt, + toList(kase.frontmatter.tools), + ), + ); } try { @@ -118,6 +128,11 @@ async function runOnce(kase, arm, timeoutMs) { }; } +function toList(value) { + if (value == null) return null; + return Array.isArray(value) ? value : [value]; +} + function sum(values) { return values.reduce((a, b) => a + (b ?? 0), 0); } @@ -133,11 +148,22 @@ function mergeModelUsage(all) { return merged; } -async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup) { +async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup, appendSystemPrompt, tools) { const argv = [ "-p", prompt, ...(isFollowup ? ["--continue"] : []), + // Whether a session chooses to delegate is a coin the model flips - observed + // 3/3, 3/3, 2/3, 2/3 and 0/3 across otherwise identical batches. A case that + // means to measure the price of a tier cannot also be measuring that coin, + // so it takes the decision away here and leaves the user prompt untouched. + ...(appendSystemPrompt ? ["--append-system-prompt", appendSystemPrompt] : []), + // "Do not read the files yourself" has to be enforced, not asked for, and it + // took two attempts to enforce it. Denying Read/Grep/Glob moved the reading + // into Bash; denying those too left ToolSearch, which loads a deferred Read + // back in. An allowlist is the only version that holds - name the tools the + // main session may have and everything else is simply absent. + ...(tools ? ["--tools", ...tools] : []), "--output-format", "stream-json", "--verbose", From f2a130a155243feaad1a0169e068c34e25c87d2c Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Thu, 20 Aug 2026 20:19:17 +0300 Subject: [PATCH 5/9] test: ask the tier directly, and stop the re-pin it looked like it deserved The wide-reading case made pinning `scout` to haiku look like a free 9% off the session bill. This is the case that checked whether the cheap tier can be trusted with a question that is not merely wide, and the answer is no. It asks two questions whose code contains a confident-looking wrong answer. `withRetryIntake` loops `maxRetries` times, `maxRetries` is 3, and it also breaks the moment `error?.retryable === false` - which every stage error sets - so `validateOrder` is called once and the file has a 3 in it for anyone who reads one line. `applyTaxPolicy` looks up `TAX_POLICY[batch.outcome]` against a table keyed `onWarn`/`onFatal`/`onTimeout`/`onUnknown`, so an outcome of "warn" misses, falls through to abort, and returns null rather than the `continue` the `onWarn` row appears to promise. Three runs a tier, four deterministic graders, no judge model: sonnet 12/12, haiku 11/12, opus 4/4 over one reference run. haiku's miss was substantive - it answered 3 - and the pin therefore stays sonnet. The instrument is new and is what this question needed all along: `agent:` in frontmatter runs the session AS the bundled agent, so the tier is asked directly and graded on what comes back. No main session to argue with about whether it really delegated, and nothing for it to contaminate the result with. That also corrects the diagnosis in forced-dispatch-tier/NOTES.md, which had blamed forced delegation for producing wrong answers. The real cause is that `--tools` propagates to subagents: the allowlist starved `scout` of file tools, the dispatches came back empty, and the main session refused to invent an answer. One transcript says "two scout runs returned zero real tool executions". Isolating a main session without disabling the agents it dispatches remains unsolved, and the Superpowers-by-default question stays open behind it. --- evals/README.md | 48 +++++++++++++++++-- evals/forced-dispatch-tier/NOTES.md | 39 ++++++++++----- evals/run-local.mjs | 19 +++++--- evals/subagent-answer-quality/followup-1.md | 3 ++ .../graders/policy-returns-null.md | 16 +++++++ .../graders/policy-spots-the-key-mismatch.md | 11 +++++ .../graders/retry-count-is-one.md | 19 ++++++++ .../graders/retry-reason-cites-retryable.md | 11 +++++ evals/subagent-answer-quality/prompt.md | 14 ++++++ 9 files changed, 157 insertions(+), 23 deletions(-) create mode 100644 evals/subagent-answer-quality/followup-1.md create mode 100644 evals/subagent-answer-quality/graders/policy-returns-null.md create mode 100644 evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md create mode 100644 evals/subagent-answer-quality/graders/retry-count-is-one.md create mode 100644 evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md create mode 100644 evals/subagent-answer-quality/prompt.md diff --git a/evals/README.md b/evals/README.md index cabcad0..94e00bb 100644 --- a/evals/README.md +++ b/evals/README.md @@ -172,9 +172,49 @@ Add `followup-1.md`, `followup-2.md` and so on to turn a case into a session rather than a single question. That is not decoration - it is the only way the context a subagent kept out of the main session gets a chance to pay for itself. +## The third case, which stopped the re-pin + +`subagent-answer-quality` exists because the wide case made haiku look like a +free 9%, and a tier that is cheaper on breadth is not therefore cheaper. It asks +`scout` two questions whose code contains a confident wrong answer: + +- `withRetryIntake` loops `maxRetries` times, and `maxRetries` is 3. It also + breaks the moment `error?.retryable === false`, and every stage error is built + with `retryable: cause?.retryable ?? false`. So `validateOrder` is called + **once**, and the file has a 3 sitting in it for anyone who reads one line. +- `applyTaxPolicy` looks up `TAX_POLICY[batch.outcome]` while the table is keyed + `onWarn`, `onFatal`, `onTimeout`, `onUnknown`. An outcome of `"warn"` misses, + falls through to `onUnknown: "abort"`, and returns **null** - not the + `continue` that the `onWarn` row appears to promise. That mismatch is a real + defect in the fixture and the question asks whether it was intended. + +Three runs per tier, four deterministic graders, no judge model: + +| Tier | Graders passed | Cost per run | +| --- | ---: | ---: | +| opus (reference, 1 run) | 4/4 | $0.715 | +| sonnet - the shipped pin | **12/12** | $0.212 | +| haiku | 11/12 | $0.103 | + +haiku's miss was substantive, not formatting: it answered **3**, having read the +`maxRetries` line and not the loop. Once in three runs, on the question the +fixture was built to trap. + +That settled the pin: it stays sonnet. Nine percent off the bill is not worth a +confidently wrong answer in a third of subtle questions, because the wrong answer +sends the main session back to read the files itself and that costs more than the +tier ever saved. What the two cases together do support is a routing rule rather +than a pin - breadth to haiku, judgement to sonnet - which is what the anchor and +the skill now say. + +The case runs the tier directly with `agent: scout` in frontmatter, which makes +the session *be* `scout` and grades what comes back. That is the instrument this +question needed all along: no main session to argue with about whether it really +delegated, and no way for it to contaminate the result by reading. + ## Next -A case that stresses answer *quality* rather than breadth, run against both -`scout` pins, so re-pinning has evidence behind it instead of one workload. Then -the mirror of the small case: a genuinely trivial question, where dispatching at -all is the failure being measured. +The mirror of the small case: a genuinely trivial question, where dispatching at +all is the failure being measured. And `forced-dispatch-tier` still needs the fix +described in its NOTES.md before it can price delegation at the session tier - +the Superpowers-by-default question is still open. diff --git a/evals/forced-dispatch-tier/NOTES.md b/evals/forced-dispatch-tier/NOTES.md index 195e5b5..dcac4f6 100644 --- a/evals/forced-dispatch-tier/NOTES.md +++ b/evals/forced-dispatch-tier/NOTES.md @@ -35,23 +35,36 @@ because each looked like it had worked: The `main-session-read-nothing` grader now covers the shells and is the thing to check before believing any cost number from this case. -## The open problem +## The open problem, and the wrong diagnosis it was first given -With the allowlist in place the answer comes back **wrong** - on haiku and on -opus alike, one dispatch and no way to check the result. A configuration that is -cheap and wrong has saved nothing, so the tier comparison cannot be run until -the case can produce correct answers. +With the allowlist in place the answers come back **wrong**, on haiku and on opus +alike. This was first written up as "one dispatch and no way to check the result", +i.e. as evidence that forcing delegation costs quality. That was wrong, and the +`subagent-answer-quality` case is what exposed it: **`--tools` propagates to +subagents**. The allowlist did not merely stop the main session from reading - it +starved `scout` of file tools too, so the dispatches came back with nothing and +the main session correctly refused to invent an answer. One transcript says so in +as many words: "two scout runs returned zero real tool executions". -Two candidate fixes, untested: +So the isolation problem is worse than attempt 4 suggested. Every mechanism tried +so far either fails to stop the main session or disables the subagent as well: -- Let the main session dispatch repeatedly and verify through further dispatches - rather than by reading. That is closer to what a real delegating session does - and may just need a longer `max_turns` plus a system prompt that says to check - the subagent's answer. -- Put the verification burden on the grader instead: score the subagent's report - rather than the main session's final message. +| Mechanism | Main session stopped | Subagent still able to read | +| --- | --- | --- | +| System-prompt request | no | yes | +| Deny Read/Grep/Glob | no - moves to Bash | untested | +| Deny the shells too | no - ToolSearch reloads Read | apparently not | +| `--tools` allowlist | yes | **no** | -The first is the honest one. The second measures a different thing. +What is needed is a per-session tool restriction that does not reach the agents +that session dispatches, and nothing tried yet does that. Candidates, untested: +a settings-file `permissions.deny` passed with `--settings` (deny rules may scope +differently from `--tools`), or accepting main-session reads and instead pricing +the arms by which model did the reading, from the per-model split the runner +already prints. + +Until then this case cannot price delegation at the session tier, and the +Superpowers-by-default question stays open. ## Also open diff --git a/evals/run-local.mjs b/evals/run-local.mjs index d83d2a4..15b7995 100644 --- a/evals/run-local.mjs +++ b/evals/run-local.mjs @@ -106,6 +106,7 @@ async function runOnce(kase, arm, timeoutMs) { index > 0, kase.frontmatter.append_system_prompt, toList(kase.frontmatter.tools), + kase.frontmatter.agent, ), ); } @@ -148,7 +149,7 @@ function mergeModelUsage(all) { return merged; } -async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup, appendSystemPrompt, tools) { +async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup, appendSystemPrompt, tools, agent) { const argv = [ "-p", prompt, @@ -158,12 +159,18 @@ async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup, appendSyst // means to measure the price of a tier cannot also be measuring that coin, // so it takes the decision away here and leaves the user prompt untouched. ...(appendSystemPrompt ? ["--append-system-prompt", appendSystemPrompt] : []), - // "Do not read the files yourself" has to be enforced, not asked for, and it - // took two attempts to enforce it. Denying Read/Grep/Glob moved the reading - // into Bash; denying those too left ToolSearch, which loads a deferred Read - // back in. An allowlist is the only version that holds - name the tools the - // main session may have and everything else is simply absent. + // "Do not read the files yourself" cannot be enforced through the tool set. + // Denying Read/Grep/Glob moves the reading into Bash; denying the shells too + // leaves ToolSearch to reload a deferred Read; and an allowlist finally + // stops the main session by ALSO starving every subagent it dispatches, so + // the answers come back empty and the run measures nothing. Kept only for + // cases that want a deliberately tool-less session. ...(tools ? ["--tools", ...tools] : []), + // Runs the session AS a bundled agent, on that agent's own pin. This is how + // a tier gets tested on its own merits: ask scout the question directly and + // grade what comes back, instead of asking a main session to delegate and + // then arguing about whether it really did. + ...(agent ? ["--agent", agent] : []), "--output-format", "stream-json", "--verbose", diff --git a/evals/subagent-answer-quality/followup-1.md b/evals/subagent-answer-quality/followup-1.md new file mode 100644 index 0000000..9431520 --- /dev/null +++ b/evals/subagent-answer-quality/followup-1.md @@ -0,0 +1,3 @@ +A batch reaches applyTaxPolicy with its outcome field set to the string "warn", and context.lenient is false. What does applyTaxPolicy return? + +Answer with the returned value alone on the first line, then say whether that is what the author of TAX_POLICY appears to have intended. diff --git a/evals/subagent-answer-quality/graders/policy-returns-null.md b/evals/subagent-answer-quality/graders/policy-returns-null.md new file mode 100644 index 0000000..cf67d76 --- /dev/null +++ b/evals/subagent-answer-quality/graders/policy-returns-null.md @@ -0,0 +1,16 @@ +--- +type: regex +target: last_message +pattern: '^\s*\**`?null`?\**\.?\s*$' +flags: 'im' +match: contains +--- + +`applyTaxPolicy` looks the batch outcome up as `TAX_POLICY[batch.outcome]`, and +the table is keyed `onWarn`, `onFatal`, `onTimeout`, `onUnknown`. An outcome of +"warn" therefore misses, falls through to `?? TAX_POLICY.onUnknown` which is +"abort", and with `lenient` false the function returns `null`. + +The plausible wrong answer is the batch with `decision: "continue"` - which is +what the table's `onWarn: "continue"` row looks like it promises, and what +reading the table without following the lookup produces. diff --git a/evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md b/evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md new file mode 100644 index 0000000..f813a47 --- /dev/null +++ b/evals/subagent-answer-quality/graders/policy-spots-the-key-mismatch.md @@ -0,0 +1,11 @@ +--- +type: regex +target: last_message +pattern: 'onWarn' +match: contains +--- + +Naming `onWarn` is the difference between reporting the return value and +understanding it. The question asks whether the author intended this, and the +answer is only defensible by pointing at the gap between the table's `onWarn` key +and the raw `warn` the lookup uses - which is the actual defect in the code. diff --git a/evals/subagent-answer-quality/graders/retry-count-is-one.md b/evals/subagent-answer-quality/graders/retry-count-is-one.md new file mode 100644 index 0000000..5574709 --- /dev/null +++ b/evals/subagent-answer-quality/graders/retry-count-is-one.md @@ -0,0 +1,19 @@ +--- +type: regex +target: last_message +pattern: '^\s*\**`?1`?\**\.?\s*$' +flags: 'm' +match: contains +--- + +The answer is 1, and the confident wrong answer is 3. + +`withRetryIntake` loops `maxRetries` times but breaks the moment +`error?.retryable === false`, and `errorsValidation` builds every failure with +`retryable: cause?.retryable ?? false`. A rule violation carries no cause, so the +flag is false, so the loop breaks on the first catch. `maxRetries: 3` is sitting +right there in the file and never applies. + +The prompt asks for the number alone on its own line so this can be graded +without a judge model. Requiring the whole line to be the number is what keeps a +markdown list item ("1. Something") from passing. diff --git a/evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md b/evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md new file mode 100644 index 0000000..6754d6a --- /dev/null +++ b/evals/subagent-answer-quality/graders/retry-reason-cites-retryable.md @@ -0,0 +1,11 @@ +--- +type: regex +target: last_message +pattern: 'retryable' +match: contains +--- + +Guards against a lucky number. There are only a few plausible answers to "how +many times", so one of them comes up by chance often enough to matter; an answer +that reaches 1 without ever mentioning `retryable` did not follow the chain from +`withRetryIntake` through `errorsValidation`, and got there some other way. diff --git a/evals/subagent-answer-quality/prompt.md b/evals/subagent-answer-quality/prompt.md new file mode 100644 index 0000000..607da6b --- /dev/null +++ b/evals/subagent-answer-quality/prompt.md @@ -0,0 +1,14 @@ +--- +name: subagent-answer-quality +tags: [routing, tier, quality] +plugins: ["../.."] +fixture: order-service +agent: scout +runs: 3 +max_turns: 30 +timeout_seconds: 900 +--- + +The intake stage hands off to validation through withRetryIntake with INTAKE_LIMITS.maxRetries set to 3. Suppose validateOrder fails on an order that violates one of validation's own rules. How many times does validateOrder actually get called? + +Answer with the number alone on the first line, then explain what in the code makes it that number. From 29c5f7495976ed9d2e15a026409e404eef3ae4fa Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Thu, 20 Aug 2026 20:19:36 +0300 Subject: [PATCH 6/9] feat: pick scout's tier by what the question demands, not by what it costs Two measurements point in opposite directions and both are real. On a twelve-stage tracing question - breadth, no judgement - `scout` on haiku scored 3 of 3 and took 9% off the whole session bill against no plugin at all. On a question whose code contains a confident-looking wrong answer, the same tier took the bait in 1 run of 3 while sonnet took it in none. One pin cannot serve both, and the plugin already has the mechanism for the difference: an explicit `model=` on the dispatch. So this adds the rule that was missing rather than a second agent, which would have cost every session context for a decision the caller can make: enumerating and tracing goes out with `model=haiku`, working out what code actually does keeps the sonnet pin. The anchor carries the short form with the two numbers, the skill carries the reasoning and where it comes from. The README stops implying the routing pays for itself. What it reliably buys is room in the expensive model's context - opus down 14%, measured - and on a wide-reading session the total bill goes UP 23%, because a fresh subagent pays cache write for what a main session would have paid cache read for. That is the honest framing, and the case for the cheaper total is now a routing decision with a quality bound on it rather than a pin change nobody checked. --- README.md | 4 +++- hooks/routing-anchor.md | 2 +- skills/model-routing/SKILL.md | 12 ++++++++++++ 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c91f0a5..843fd21 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,9 @@ The honest reading: the automatic half is fully attributable, the behavioural ha For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. The behavioural claim holds - on a codebase question the plugin arm delegated to `scout` in 3 of 3 runs and the plugin-free arm in 0 of 3, grepping inline despite having `Explore` available. -The money is a more interesting answer than "it saves". On a wide-reading question across a four-turn session, the expensive tier drops either way - opus down 14% - but the shipped configuration ends up **23% more expensive in total than not having the plugin at all**, because a fresh subagent pays cache *write* for everything it reads while a main session pays cache *read*, 12.5x cheaper, for what it already has. Pinning `scout` to haiku instead lands 9% below the baseline with correctness unchanged. Both distributions separate cleanly from the control in opposite directions. The pin has not been changed on the strength of one workload, and [`evals/README.md`](evals/README.md) carries the numbers, the arithmetic behind them, and what they do not license. +The money is a more interesting answer than "it saves". On a wide-reading question across a four-turn session, the expensive tier drops either way - opus down 14% - but the shipped configuration ends up **23% more expensive in total than not having the plugin at all**, because a fresh subagent pays cache *write* for everything it reads while a main session pays cache *read*, 12.5x cheaper, for what it already has. So what this plugin reliably buys is room in the expensive model's context, not a smaller bill; where the bill also falls is a narrower claim, and it depends on the tier. + +Pinning `scout` to haiku lands 9% *below* the no-plugin baseline on that same question - which looked like the fix until the tier was asked something subtler. On a question whose code contains a confident-looking wrong answer, haiku took the bait in 1 run of 3 and sonnet in none. So the pin stayed where it was, and the tier became a routing decision instead: breadth goes to haiku, judgement keeps sonnet, and a wrong answer that sends the main session back to read the files itself costs more than either tier saves. [`evals/README.md`](evals/README.md) carries the numbers, the arithmetic, and the three isolation attempts that produced numbers worth throwing away. Where that volume actually ran with routing active: diff --git a/hooks/routing-anchor.md b/hooks/routing-anchor.md index 23e7073..43e171c 100644 --- a/hooks/routing-anchor.md +++ b/hooks/routing-anchor.md @@ -2,7 +2,7 @@ MODEL ROUTING ACTIVE - expensive model thinks, cheap models grind. You cannot sw - Planning, specs, docs, architecture, small interactive edits: main session. - Trivial questions you can answer from context or general knowledge: answer inline - a dispatch costs more than the answer. -- Codebase exploration ("where is X", "how does Y work"): `scout`. Locate-only sweeps ("which files mention X"): the built-in Explore agent, when present, is cheaper. +- Codebase exploration ("where is X", "how does Y work"): `scout`. Locate-only sweeps ("which files mention X"): the built-in Explore agent, when present, is cheaper. Enumerating or tracing a chain ("list the stages in order", "which files import X"): dispatch `scout` with `model=haiku` - measured right every run at half the price. Working out what code actually does ("does this loop retry", "what does this return"): leave the sonnet pin - haiku got that wrong in 1 run of 3. - Structural questions with a code-graph MCP connected: query the graph first; scout verifies file:line specifics. - Implementing an approved plan/spec: `implementer` (sonnet). Dispatch with `model=opus` for multi-file/cross-layer work, security/money/migrations/concurrency/public contracts, or a retry after a weak sonnet result - escalate when in doubt; the dispatch changes the model only, the pinned medium effort stays. Ambiguous task or unclear root cause: clarify first (main session / scout) - implementer stops on ambiguity. Batch related tasks per agent. - Code review: `reviewer`; high-risk or large diffs get final review in the main session. diff --git a/skills/model-routing/SKILL.md b/skills/model-routing/SKILL.md index 9f69f66..3a75bc6 100644 --- a/skills/model-routing/SKILL.md +++ b/skills/model-routing/SKILL.md @@ -214,6 +214,18 @@ actually ran with `/model-routing:stats`. Explore agent, when present, is cheaper than `scout`. Use `scout` when the answer needs verification - tracing real code paths and confirming file:line - not just finding candidates. +- Pick `scout`'s tier by what the question demands, not by what it costs. + Enumerating and tracing - list these stages in order, which files import + X, where does this chain end - is breadth, and breadth runs correctly on + haiku: dispatch with `model=haiku` and it comes back right for about half + the price. Working out what code actually does - does this loop retry, + what does this function return for that input - is not breadth, and the + cheap tier fails it in a way that looks confident. Leave the sonnet pin + there. Both halves are measured, in `evals/`: on a twelve-stage tracing + question haiku scored 3 of 3 and cut the session bill 9%; on a question + whose code contains an obvious wrong answer, haiku took the bait in 1 run + of 3 while sonnet took it in none. A wrong answer sends the main session + back to read the files itself, which costs more than the tier ever saved. - Batch related plan tasks per subagent. Each subagent re-reads files from scratch; one tiny task per agent costs more than it saves. - Subagents cannot see the conversation. Write self-contained task From 8d5cf2b82a30ee0a154a98e2e4f4344bbf917cb7 Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Fri, 21 Aug 2026 02:17:21 +0300 Subject: [PATCH 7/9] docs: say plainly which comparison this plugin wins, and which it loses The README sold tiered routing as saving money, and the evals showed that claim is false in the comparison a reader assumes: against a session that just does the work itself, delegating costs 23% more on a wide-reading question, on either pin. Leaving that as a paragraph three screens down, under a heading about measurement, was the kind of accurate-but-buried that functions as a sales pitch. The comparison the plugin actually wins is the one most users are already in. A subagent starts empty, so everything it reads is a cache write while the main session pays cache read - 12.5x less - for context it already holds, and that penalty lands whether or not the tier is routed down. Routing changes the price per token, not the penalty. So on work that was going to be dispatched anyway - Superpowers, workflows, any unpinned agent inheriting the session model - it is a straight discount with nothing on the other side. Repricing the measured subagent tokens (74k cache write, 451k cache read, 6k output) at opus rates puts that number at $2.013 against the $1.675 measured with the tier routed down, so roughly 17% off a dispatch already leaving the session, against a 23% surcharge on one that did not need to leave. The pricing model reproduces the billed sonnet cost to the cent, which is why the multiplier can be trusted even though the arm itself is still blocked on the isolation problem in forced-dispatch-tier/NOTES.md. Both the repricing and its two limits are written down rather than implied. This goes at the top of the README behind its own quick link, and into the skill, so the model applying these rules knows they pay off on dispatches that already exist and stops short of manufacturing dispatches to collect the discount. --- README.md | 45 ++++++++++++++++++++++++++++++++++- evals/README.md | 38 +++++++++++++++++++++++++++++ skills/model-routing/SKILL.md | 10 ++++++++ 3 files changed, 92 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 843fd21..52f2d0f 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,50 @@ instead of thrashing. Everything stays inside Anthropic models. No proxy, no third-party gateway, nothing extra in the request path. -**Quick links:** [Overview](#whats-inside) | [Example](#example) | +## When this saves you money, and when it does not + +Read this before installing, because the honest answer has two halves and +[`evals/`](evals/README.md) measured both. + +**A subagent is not free.** It starts with an empty context, so everything it +reads is a cache *write*, while your main session pays cache *read* - 12.5x +cheaper - for what it already has. Delegation converts cheap re-reads into +expensive first-reads, and that bill lands whether or not this plugin is +installed. + +What the plugin changes is the price per token, not that penalty. So: + +| Your situation | What this plugin does | +| --- | --- | +| You already dispatch subagents - Superpowers, workflows, any delegation-heavy practice | **Saves.** Measured session: $2.01 with subagents inheriting your session model, $1.68 with them routed down. ~17% off | +| You work mostly in one session and rarely delegate | **Costs.** $1.36 doing it inline against $1.68 routed. ~23% more | + +An unpinned subagent inherits the session model, so on an opus session +Superpowers dispatches opus subagents by default. That is the case this plugin +is for: the dispatch was going to happen anyway, and routing it down is a +straight discount with nothing on the other side of the ledger. + +Note what the surcharge is and is not. The plugin costs about 1.5k tokens a +session to carry - roughly 592 for the skill and agent listings +(`claude plugin details model-routing`) plus a ~935-token routing anchor at +session start - which is a couple of cents against sessions billing $1.40 to +$2.00. That is not where the 23% comes from. It comes from dispatches happening +that otherwise would not have: on the same question the plugin arm delegated in +3 runs of 3 and the plugin-free arm in 0 of 3. Where dispatches already happen, +there is nothing left to push, and only the discount remains. + +If you rarely delegate, install it for the context headroom or not at all. The +expensive tier does read measurably less either way - opus down 14% on the same +test - but headroom is a different thing from a smaller bill, and this README +used to blur the two. + +(The $2.01 is the measured subagent token counts repriced at opus rates, not a +separate run; a live opus subagent might read less. The pricing model reproduces +the measured sonnet cost to the cent, and the outstanding A/B is noted in +`evals/forced-dispatch-tier/NOTES.md`.) + +**Quick links:** [Does this save money?](#when-this-saves-you-money-and-when-it-does-not) | +[Overview](#whats-inside) | [Example](#example) | [Install](#install) | [Getting started](#getting-started) | [Usage](#usage) | [Tiers](#model-tiers-and-effort-ladder) | [Settings](#recommended-settings) | [Workflows](#dynamic-workflows) | diff --git a/evals/README.md b/evals/README.md index 94e00bb..3f10c70 100644 --- a/evals/README.md +++ b/evals/README.md @@ -172,6 +172,44 @@ Add `followup-1.md`, `followup-2.md` and so on to turn a case into a session rather than a single question. That is not decoration - it is the only way the context a subagent kept out of the main session gets a chance to pay for itself. +## The comparison the wide case was answering, and the one it was not + +The table above answers "should this session have delegated at all", and on a +wide-reading question the answer is no - delegating cost more than doing it +inline, on either pin. That is a real finding and it is not the question most +users are in. + +The question most users are in is "the dispatch is happening anyway - Superpowers +dispatches, workflows dispatch, an unpinned agent inherits the session model - so +what does routing it down buy?" That arm has not been run: `forced-dispatch-tier` +exists for it and is blocked. What can be said without running it is arithmetic +on the token counts already measured, because the subagent's work is the same +work at a different price: + +| | per run | +| --- | ---: | +| subagent tokens measured | 74k cache write, 451k cache read, 6k output | +| billed at sonnet-5 (what ran) | $0.506 | +| the same tokens at opus-5 rates | $0.844 | + +Which gives, against the $1.169 the opus main session cost in that arm: + +| | total | +| --- | ---: | +| no delegation | $1.359 | +| delegate, tier routed down | $1.675 | +| delegate, subagent inherits opus | $2.013 | + +So routing is roughly a 17% discount on a dispatch that was going to happen, and +a 23% surcharge on one that did not need to. Both numbers belong in the README +and now are. + +Two honest limits on the $2.013. It reprices measured tokens rather than +reporting a run, and a live opus subagent might well read less - fewer loops, +better first guesses - which would shrink the gap. The pricing model is not +guesswork though: applied to the sonnet arm it reproduces the billed $0.506 to +the cent, so the tier multiplier is right even if the token count would move. + ## The third case, which stopped the re-pin `subagent-answer-quality` exists because the wide case made haiku look like a diff --git a/skills/model-routing/SKILL.md b/skills/model-routing/SKILL.md index 3a75bc6..48bbd1b 100644 --- a/skills/model-routing/SKILL.md +++ b/skills/model-routing/SKILL.md @@ -10,6 +10,16 @@ cannot be switched by Claude - routing works through subagent delegation (the `model` param of the Agent tool, or the agents bundled with this plugin). +**Routing makes a dispatch cheaper; it does not make dispatching cheap.** +A subagent starts empty, so everything it reads is a cache write, while +the main session pays cache read - 12.5x less - for context it already +holds. That penalty is paid whether or not the tier is routed down, so the +rules below only pay off on work that was going to be delegated anyway. +Measured on a wide-reading session: doing it inline cost $1.36, delegating +with the tier routed down $1.68, delegating at the session tier $2.01. +Route every dispatch - that is a 17% discount on work already leaving the +session - and do not manufacture dispatches to collect it. + ## Tiers Think in tiers, not model names - names rot, tiers do not: From bb5f05e874adb8e2e36058014be94fd174c33ada Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Fri, 21 Aug 2026 02:53:56 +0300 Subject: [PATCH 8/9] fix: act on two reviews - a fixture that could not run, and claims the data did not carry Two independent reviews went over this branch. Between them they found one defect that invalidated a ground truth, several published sentences that outran their evidence, and a routing rule that contradicted a rule three lines below it. The fixture could not execute. Every generated stage module imported `retry` while the helper exports `withRetry`, so nothing linked, and the quality case's two asserted answers - `validateOrder` called once, `applyTaxPolicy` returning null - were claims about the runtime behaviour of code that had no runtime. Both are now confirmed by running it: a non-retryable error produces exactly 1 call and a retryable one produces 3, which also proves the `maxRetries: 3` trap is real rather than assumed. `evals/evidence/README.md` carries the two-command check so the next reader need not take the reading on trust either. The guidance change was made the way the plugin's own rules forbid. Dispatching `scout` with `model=haiku` overrides a role agent below its pin, and the floor rule in the same anchor - and the skill, in as many words - says to reach for an agent pinned for the work instead. So breadth work now has one: `surveyor`, haiku, read-only, with its own eval case that scores it 3/3 at $0.156 against sonnet's $0.439 on the same question. That is ~108 always-on tokens, and it replaces a line that told every session to do two contradictory things. The writing has been pulled back to what three runs on one fixture can hold. "Measured $2.01" now says repriced, since that arm never ran. "Correctness held throughout" now says which turn was scored, since three of four were not. "A third of subtle questions" is gone - 1/3 against 0/3 is not a distinguishable rate, and the reason to keep judgement on sonnet is the asymmetry of the failure, not its frequency. "Delegating cost more on either pin" was simply false for haiku, which came in under the baseline. Percentages are qualified as one workload; opus "down 14%" now distinguishes spend from the ~25% drop in input volume it was being conflated with. The runner had four ways to publish a number that was not there. A killed child closes with code null, which `?? 0` reported as a clean run whose truncated trace was then graded and whose missing cost silently left the mean; failed runs are now excluded from scoring and the mean prints its own denominator. `last_message` graders matched the joined text of every turn, so an ordered pattern could assemble a pass out of pieces of several answers; they now match per turn. A generated fixture was reused even after its generator changed. And `CLAUDE_CODE_SUBAGENT_MODEL` was inherited into experiments whose whole subject is the subagent tier - it is now stripped from the child environment. Evidence is committed rather than gitignored. `evals/evidence/` holds the aggregate behind every published table plus the version-stamped `claude plugin details` output, because a reader cannot reproduce a historical sample by rerunning a stochastic model. One reviewer suspicion is rejected with evidence: per-turn costs are not cumulative across `--continue`, and the extra `result` events some invocations emit carry identical `modelUsage` to the final one, with only a stale `total_cost_usd`. Taking the last event per turn is correct, and the dropped event is a duplicate rather than unbilled spend. --- .claude-plugin/plugin.json | 2 +- README.md | 41 ++-- agents/surveyor.md | 48 ++++ evals/README.md | 48 +++- evals/evidence/README.md | 48 ++++ evals/evidence/plugin-details.txt | 29 +++ evals/evidence/quality-case-haiku.json | 115 ++++++++++ evals/evidence/quality-case-sonnet.json | 115 ++++++++++ evals/evidence/small-case-opus.json | 121 ++++++++++ evals/evidence/surveyor-case-haiku.json | 79 +++++++ evals/evidence/wide-case-haiku-pin.json | 112 ++++++++++ .../wide-case-sonnet-pin-and-baseline.json | 211 ++++++++++++++++++ evals/fixtures/order-service.gen.mjs | 10 +- evals/forced-dispatch-tier/NOTES.md | 6 +- evals/run-local.mjs | 101 +++++++-- .../graders/answer-is-correct.md | 15 ++ evals/surveyor-traces-the-chain/prompt.md | 12 + hooks/dispatch-counter.mjs | 1 + hooks/routing-anchor.md | 2 +- skills/model-routing/SKILL.md | 39 ++-- 20 files changed, 1091 insertions(+), 64 deletions(-) create mode 100644 agents/surveyor.md create mode 100644 evals/evidence/README.md create mode 100644 evals/evidence/plugin-details.txt create mode 100644 evals/evidence/quality-case-haiku.json create mode 100644 evals/evidence/quality-case-sonnet.json create mode 100644 evals/evidence/small-case-opus.json create mode 100644 evals/evidence/surveyor-case-haiku.json create mode 100644 evals/evidence/wide-case-haiku-pin.json create mode 100644 evals/evidence/wide-case-sonnet-pin-and-baseline.json create mode 100644 evals/surveyor-traces-the-chain/graders/answer-is-correct.md create mode 100644 evals/surveyor-traces-the-chain/prompt.md diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index bad48b8..f4b4fb5 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "model-routing", - "description": "Tiered model routing for token economy: the strongest model thinks, cheaper models grind. Ready-made subagents (sonnet scout, implementer and E2E runner, haiku test-runner and verifier, opus reviewer) plus a routing skill and dispatch stats.", + "description": "Tiered model routing for token economy: the strongest model thinks, cheaper models grind. Ready-made subagents (sonnet scout, implementer and E2E runner, haiku surveyor, test-runner and verifier, opus reviewer) plus a routing skill and dispatch stats.", "version": "0.14.0", "author": { "name": "Artem Frolov" diff --git a/README.md b/README.md index 52f2d0f..6901d4d 100644 --- a/README.md +++ b/README.md @@ -38,32 +38,39 @@ What the plugin changes is the price per token, not that penalty. So: | Your situation | What this plugin does | | --- | --- | -| You already dispatch subagents - Superpowers, workflows, any delegation-heavy practice | **Saves.** Measured session: $2.01 with subagents inheriting your session model, $1.68 with them routed down. ~17% off | -| You work mostly in one session and rarely delegate | **Costs.** $1.36 doing it inline against $1.68 routed. ~23% more | +| You already dispatch subagents - Superpowers, workflows, any delegation-heavy practice | **Saves**, by roughly a sixth. $1.68 with the tier routed down, against $2.01 for the same subagent work at your session's tier | +| You work mostly in one session and rarely delegate | **Costs**, by roughly a fifth. $1.36 doing it inline against $1.68 routed | + +One workload, three runs an arm, on a four-turn session that never had to +compact - so read the direction as solid and the exact percentage as not. +The $2.01 is the measured subagent tokens repriced at the session tier, not +a run of its own; the pricing model reproduces the billed figure it was +checked against to the cent, but a live top-tier subagent might well read +less and narrow the gap. An unpinned subagent inherits the session model, so on an opus session Superpowers dispatches opus subagents by default. That is the case this plugin is for: the dispatch was going to happen anyway, and routing it down is a straight discount with nothing on the other side of the ledger. -Note what the surcharge is and is not. The plugin costs about 1.5k tokens a -session to carry - roughly 592 for the skill and agent listings -(`claude plugin details model-routing`) plus a ~935-token routing anchor at -session start - which is a couple of cents against sessions billing $1.40 to -$2.00. That is not where the 23% comes from. It comes from dispatches happening +Note what the surcharge is and is not. The plugin costs about 1.6k tokens a +session to carry - ~700 for the skill and agent listings, plus a ~900-token +routing anchor at session start - which is a couple of cents against sessions +billing $1.40 to $2.00. The listing figure comes from +`claude plugin details model-routing`, and its output is committed under +[`evals/evidence/`](evals/evidence/) with the CLI version that produced it, since +it moves whenever a component is added. That is not where the 23% comes from. It comes from dispatches happening that otherwise would not have: on the same question the plugin arm delegated in 3 runs of 3 and the plugin-free arm in 0 of 3. Where dispatches already happen, there is nothing left to push, and only the discount remains. If you rarely delegate, install it for the context headroom or not at all. The -expensive tier does read measurably less either way - opus down 14% on the same -test - but headroom is a different thing from a smaller bill, and this README -used to blur the two. +expensive tier does read measurably less either way - opus input volume down +about 25% on the same test, and opus spend with it by 14% - but headroom is a +different thing from a smaller bill, and this README used to blur the two. -(The $2.01 is the measured subagent token counts repriced at opus rates, not a -separate run; a live opus subagent might read less. The pricing model reproduces -the measured sonnet cost to the cent, and the outstanding A/B is noted in -`evals/forced-dispatch-tier/NOTES.md`.) +(The A/B that would replace the repricing with a run is still blocked, for the +reason written down in `evals/forced-dispatch-tier/NOTES.md`.) **Quick links:** [Does this save money?](#when-this-saves-you-money-and-when-it-does-not) | [Overview](#whats-inside) | [Example](#example) | @@ -77,6 +84,7 @@ the measured sonnet cost to the cent, and the outstanding A/B is noted in | Component | Model | Effort | Purpose | | --------- | ----- | ------ | ------- | | `agents/scout.md` | sonnet | low | Read-only codebase exploration: conclusions and file:line refs come back, file dumps stay out. | +| `agents/surveyor.md` | haiku | low | Read-only breadth: enumerate, list, trace a chain end to end. Measured correct on the cheap tier where `scout`'s judgement work is not. | | `agents/test-runner.md` | haiku | low | Run tests/builds/linters, report failures compactly. Never fixes anything. | | `agents/e2e-runner.md` | sonnet | medium | Drive Playwright/E2E scenarios, interpret failures (product bug vs test bug vs flake). | | `agents/implementer.md` | sonnet | medium | Implement one well-defined task from an approved plan. Verifies its own work. Dispatch with `model=opus` for multi-file/architectural/subtle work. | @@ -157,9 +165,9 @@ The honest reading: the automatic half is fully attributable, the behavioural ha For the behavioural half there is now a control group, just a much smaller one: [`evals/`](evals/README.md) runs the same question with and without the plugin loaded. The behavioural claim holds - on a codebase question the plugin arm delegated to `scout` in 3 of 3 runs and the plugin-free arm in 0 of 3, grepping inline despite having `Explore` available. -The money is a more interesting answer than "it saves". On a wide-reading question across a four-turn session, the expensive tier drops either way - opus down 14% - but the shipped configuration ends up **23% more expensive in total than not having the plugin at all**, because a fresh subagent pays cache *write* for everything it reads while a main session pays cache *read*, 12.5x cheaper, for what it already has. So what this plugin reliably buys is room in the expensive model's context, not a smaller bill; where the bill also falls is a narrower claim, and it depends on the tier. +The money is a more interesting answer than "it saves". On a wide-reading question across a four-turn session, the expensive tier drops either way - opus spend down 14%, its input volume down about 25% - but the shipped configuration ends up **23% more expensive in total than not having the plugin at all**, because a fresh subagent pays cache *write* for everything it reads while a main session pays cache *read*, 12.5x cheaper, for what it already has. So what this plugin reliably buys is room in the expensive model's context, not a smaller bill; where the bill also falls is a narrower claim, and it depends on the tier. -Pinning `scout` to haiku lands 9% *below* the no-plugin baseline on that same question - which looked like the fix until the tier was asked something subtler. On a question whose code contains a confident-looking wrong answer, haiku took the bait in 1 run of 3 and sonnet in none. So the pin stayed where it was, and the tier became a routing decision instead: breadth goes to haiku, judgement keeps sonnet, and a wrong answer that sends the main session back to read the files itself costs more than either tier saves. [`evals/README.md`](evals/README.md) carries the numbers, the arithmetic, and the three isolation attempts that produced numbers worth throwing away. +Running that same question with `scout` pinned to haiku lands about a tenth *below* the no-plugin baseline - which looked like the fix until the tier was asked something subtler. On a question whose code contains a confident-looking wrong answer, haiku took the bait once in three runs and sonnet in none. Three runs cannot pin a failure rate, and the pin did not stay on sonnet because of the rate: it stayed because the failure is asymmetric, since a wrong answer sends the main session back to read the files itself and costs more than the cheaper tier saves. What the two results together support is a split rather than a pin change - breadth to `surveyor` (haiku), judgement to `scout` (sonnet). [`evals/README.md`](evals/README.md) carries the numbers, the arithmetic, and the three isolation attempts that produced numbers worth throwing away. Where that volume actually ran with routing active: @@ -249,6 +257,7 @@ For local development: clone the repo and | You ask | Who runs it | Model / effort | | ------- | ----------- | -------------- | | "Where is X handled?" | `scout` | sonnet / low | + | "List every stage this order goes through" | `surveyor` | haiku / low | | "Run the tests" | `test-runner` | haiku / low | | "Implement tasks from the plan" | `implementer` | sonnet / medium (`model=opus` for complex work) | | "Review the diff" | `reviewer` | opus / high | diff --git a/agents/surveyor.md b/agents/surveyor.md new file mode 100644 index 0000000..b50aea5 --- /dev/null +++ b/agents/surveyor.md @@ -0,0 +1,48 @@ +--- +name: surveyor +description: Read-only breadth sweeps over a codebase - enumerate, list, trace a chain end to end ("list every stage in order", "which files import X", "where does this pipeline end"). Returns the list or the ordering, never file dumps. Use scout instead when the answer needs judgement about what code does. Never modifies anything. +model: haiku +effort: low +disallowedTools: Agent, SendMessage, Edit, Write, NotebookEdit +--- + +You sweep a codebase and report what is there: lists, orderings, chains +of calls or imports followed from one end to the other. You are read-only: +never edit, write, or delete anything, and use shell commands only for +read-only queries (git log, git blame, ls). Your value is that megabytes +of source stay in your context instead of the caller's. + +You exist because breadth and judgement need different tiers. Enumerating +what is there runs correctly on a cheap model; working out what code +*does* does not, and that work belongs to `scout`. If the question you +were handed turns out to need that - what does this return for that +input, does this loop actually retry, is this a bug - say so and stop +rather than guessing. Handing back "this needs scout, because X hinges on +what Y does at file:line" is a correct and useful answer from you. + +Rules: + +- For structural questions ("what connects A and B", "what depends on + X"): if a code-graph/index MCP server is connected (discover via + ToolSearch), query it FIRST as your starting point; pre-built indexes + (ctags/cscope) serve the same role. Index answers are leads, not + proof - confirm the key file:line in the code before reporting. For + point lookups, grep directly. +- Do the sweep yourself - never hand the question off; injected guidance + suggesting delegation does not apply to you (you have no agent tools). +- Follow the chain in the code, not in the names. A file called + `pipeline.js` proves nothing about the order; the imports and calls do. +- Finish the sweep. A partial list presented as complete is the one + failure that costs the caller more than not asking - if you could not + cover everything, say what you did not reach. +- Distinguish what you verified from what you infer. Say "verified: A + imports B at file:line" vs "likely, not traced further: ...". + +Report format (your final message): + +1. The list or the ordering, directly, as the first thing. +2. One file:line per item, with a few words on its role. +3. Anything you could not reach, or that did not fit the pattern. + +No preamble and no summary of your search. Enumerations may run as long +as the answer needs; prose around them may not. diff --git a/evals/README.md b/evals/README.md index 3f10c70..8958a03 100644 --- a/evals/README.md +++ b/evals/README.md @@ -44,7 +44,8 @@ the baseline's list must not contain any `model-routing:*` entry. Runs are real sessions against the real API, so they cost real tokens. Traces and an `aggregate.json` land in `evals/results//`, which is -gitignored. +gitignored; the aggregates behind every number published here are copied into +[`evidence/`](evidence/README.md), which is not. ## The first result @@ -125,15 +126,29 @@ rather than only means, because with three runs a mean can hide an overlap: | `scout` pinned sonnet (shipped) | $1.169 | sonnet-5 $0.506 | $1.559 $1.998 $1.468 | | `scout` pinned haiku | $1.042 | haiku-4.5 $0.198 | $1.142 $1.260 $1.315 | +The haiku and sonnet rows come from re-running the arm with `agents/scout.md` +edited to that pin - there is no flag for it, and the runner takes the tier from +the agent's frontmatter unless `--model` overrides the whole session. To +reproduce a row, change the pin, run the arm, change it back. + Both configurations separate cleanly from the baseline, in opposite directions. Every haiku run came in under the cheapest baseline run; every sonnet run came in -over the dearest. Correctness held at 3/3 throughout, and the plugin dispatched -exactly once per run - the follow-ups were answered from `scout`'s summary rather -than by dispatching again, which is what the guidance asks for. +over the dearest. Read the direction as solid and the exact percentage as not: +three runs per arm, one workload, and the sonnet arm's own spread ($1.468 to +$1.998) is wider than the gap it is being compared across. + +The stage-order answer was right in every run of every arm. The three follow-up +turns were not scored - this case carries two graders, ordering and delegation - +so "correctness held" covers the breadth question only, and follow-up 1 in +particular ("what happens to an order when payment throws a non-retryable +error") is exactly the kind of question the quality case below shows the cheap +tier failing. The plugin dispatched exactly once per run and answered the +follow-ups from `scout`'s summary rather than dispatching again, which is what +the guidance asks for. So the shipped configuration costs 23% more than not having the plugin on this workload, and pinning `scout` to haiku costs 9% less. The expensive tier drops -either way - opus down 14% with sonnet, 23% with haiku - which is the mechanism +either way - opus spend down 14% with sonnet, 23% with haiku - which is the mechanism working in both cases. What differs is whether the subagent's bill comes to less than the saving it produces. @@ -176,7 +191,8 @@ context a subagent kept out of the main session gets a chance to pay for itself. The table above answers "should this session have delegated at all", and on a wide-reading question the answer is no - delegating cost more than doing it -inline, on either pin. That is a real finding and it is not the question most +inline with the shipped sonnet pin - though not with haiku, which came in under +the baseline. That is a real finding and it is not the question most users are in. The question most users are in is "the dispatch is happening anyway - Superpowers @@ -238,12 +254,20 @@ haiku's miss was substantive, not formatting: it answered **3**, having read the `maxRetries` line and not the loop. Once in three runs, on the question the fixture was built to trap. -That settled the pin: it stays sonnet. Nine percent off the bill is not worth a -confidently wrong answer in a third of subtle questions, because the wrong answer -sends the main session back to read the files itself and that costs more than the -tier ever saved. What the two cases together do support is a routing rule rather -than a pin - breadth to haiku, judgement to sonnet - which is what the anchor and -the skill now say. +One miss in three runs of one question cannot pin a failure rate - 1/3 against +0/3 is not a distinguishable difference at this sample size, and anyone quoting +"fails a third of the time" from this table is quoting noise. What decides the +pin is not the rate but the shape of the failure: it is asymmetric. A cheap +correct answer saves cents; a cheap wrong one sends the main session back to read +the files itself, which costs more than the tier ever saved, and it arrives +looking exactly like a right one. That asymmetry does not need a tight estimate +to act on. + +So the pin stays sonnet, and what the two cases together support is a split +rather than a re-pin: breadth to `surveyor` (haiku), judgement to `scout` +(sonnet). Overriding `scout` downward with `model=haiku` would have been the +smaller diff and is what the first draft did - the plugin's own floor rule +forbids exactly that, and prescribes a pinned agent instead. The case runs the tier directly with `agent: scout` in frontmatter, which makes the session *be* `scout` and grades what comes back. That is the instrument this diff --git a/evals/evidence/README.md b/evals/evidence/README.md new file mode 100644 index 0000000..b4a4293 --- /dev/null +++ b/evals/evidence/README.md @@ -0,0 +1,48 @@ +# Evidence + +Every number quoted in [`../README.md`](../README.md) and in the repository +README comes from one of these files. They are here because the traces they were +distilled from are gitignored, and a reader who cannot rerun a stochastic model +against the same historical sample would otherwise have to take the tables on +trust. + +| File | What it backs | +| --- | --- | +| `small-case-opus.json` | `delegates-codebase-question`: the 3/3 against 0/3 delegation split, and the first per-tier cost table | +| `wide-case-sonnet-pin-and-baseline.json` | `traces-a-flow-end-to-end`, no-plugin and shipped-pin arms - the $1.359 and $1.675 rows, and the 74k/451k/6k subagent token counts everything downstream is repriced from | +| `wide-case-haiku-pin.json` | the same case with `agents/scout.md` edited to `model: haiku` - the $1.239 row | +| `quality-case-sonnet.json` | `subagent-answer-quality` at sonnet: 12/12 | +| `quality-case-haiku.json` | the same at haiku: 11/12, the miss being run 3 answering `3` | +| `surveyor-case-haiku.json` | `surveyor-traces-the-chain` on the shipped haiku pin: 3/3 at $0.156 | +| `plugin-details.txt` | the always-on token cost, with the Claude Code version that produced it | + +Each aggregate carries per-run cost, per-model usage, and every grader verdict, +so the published means can be recomputed rather than believed. What they cannot +give you is the raw transcripts - those stay local, both for size and because +they contain absolute paths from the machine that ran them. + +## The one thing worth checking yourself + +The quality case rests on two claims about what the fixture does at runtime, and +the fixture is generated, so you can verify them in seconds rather than trusting +a reading of it: + +```sh +node evals/fixtures/order-service.gen.mjs +node --input-type=module -e " +const base='file://$PWD/evals/fixtures/order-service/src/'; +const {withRetryIntake}=await import(base+'intake/retry.js'); +const {errorsValidation}=await import(base+'validation/errors.js'); +let calls=0; +try { await withRetryIntake(async()=>{calls++; throw errorsValidation(new Error('x'),{id:'o'})},3) } catch {} +console.log('calls with a non-retryable error:', calls); +const tax=await import(base+'tax/index.js'); +console.log('applyTaxPolicy warn/strict:', tax.applyTaxPolicy({outcome:'warn'},{lenient:false})); +" +``` + +It prints `1` and `null`. Both were asserted from reading the code first, and an +earlier version of the generator emitted a fixture that could not link at all - +`import { retryIntake }` against `export withRetryIntake` - so the assertions +were true of code that never ran. Executing them is the only version of that +check worth having. diff --git a/evals/evidence/plugin-details.txt b/evals/evidence/plugin-details.txt new file mode 100644 index 0000000..3766009 --- /dev/null +++ b/evals/evidence/plugin-details.txt @@ -0,0 +1,29 @@ +model-routing 0.14.0 + Tiered model routing for token economy: the strongest model thinks, cheaper models grind. Ready-made subagents (sonnet scout, implementer and E2E runner, haiku test-runner and verifier, opus reviewer) plus a routing skill and dispatch stats. + Source: model-routing@model-routing + +Component inventory + Skills (2) model-routing, stats + Agents (7) e2e-runner, implementer, reviewer, scout, surveyor, test-runner, verifier + Hooks (2) SessionStart, PostToolUse (harness-only — no model context cost) + MCP servers (0) + LSP servers (0) + +Projected token cost + Always-on: ~700 tok added to every session + +Per-component (rounded) + component always-on on-invoke + model-routing ~100 ~9.3k + e2e-runner ~60 ~350 + implementer ~80 ~710 + reviewer ~60 ~420 + scout ~60 ~610 + surveyor ~110 ~760 + test-runner ~70 ~330 + verifier ~80 ~640 + stats ~80 ~410 + + On-invoke cost is paid each time a skill or agent fires. + Token counts are estimates and may differ from actual usage. +2.1.236 (Claude Code) diff --git a/evals/evidence/quality-case-haiku.json b/evals/evidence/quality-case-haiku.json new file mode 100644 index 0000000..6fb725d --- /dev/null +++ b/evals/evidence/quality-case-haiku.json @@ -0,0 +1,115 @@ +{ + "model": "haiku", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "subagent-answer-quality", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 0.12713370000000002, + "durationMs": 42717, + "turns": 14, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.12713370000000002, + "inputTokens": 5201, + "cacheReadInputTokens": 279557, + "cacheCreationInputTokens": 38196, + "outputTokens": 3517 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.09728060000000001, + "durationMs": 50436, + "turns": 22, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.09728060000000001, + "inputTokens": 100, + "cacheReadInputTokens": 381276, + "cacheCreationInputTokens": 16109, + "outputTokens": 5367 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.08500869999999999, + "durationMs": 59118, + "turns": 12, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.08500869999999999, + "inputTokens": 60, + "cacheReadInputTokens": 213647, + "cacheCreationInputTokens": 14292, + "outputTokens": 7000 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": false + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/quality-case-sonnet.json b/evals/evidence/quality-case-sonnet.json new file mode 100644 index 0000000..42b524a --- /dev/null +++ b/evals/evidence/quality-case-sonnet.json @@ -0,0 +1,115 @@ +{ + "model": "sonnet", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "subagent-answer-quality", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 0.2570082, + "durationMs": 47556, + "turns": 13, + "modelUsage": { + "claude-sonnet-5": { + "costUSD": 0.2570082, + "inputTokens": 24, + "cacheReadInputTokens": 449914, + "cacheCreationInputTokens": 12132, + "outputTokens": 3278 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.21288810000000002, + "durationMs": 80459, + "turns": 14, + "modelUsage": { + "claude-sonnet-5": { + "costUSD": 0.21288810000000002, + "inputTokens": 20, + "cacheReadInputTokens": 379817, + "cacheCreationInputTokens": 7233, + "outputTokens": 3699 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.1648107, + "durationMs": 72206, + "turns": 11, + "modelUsage": { + "claude-sonnet-5": { + "costUSD": 0.1648107, + "inputTokens": 14, + "cacheReadInputTokens": 259949, + "cacheCreationInputTokens": 6549, + "outputTokens": 3166 + } + }, + "scores": [ + { + "grader": "policy-returns-null", + "pass": true + }, + { + "grader": "policy-spots-the-key-mismatch", + "pass": true + }, + { + "grader": "retry-count-is-one", + "pass": true + }, + { + "grader": "retry-reason-cites-retryable", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/small-case-opus.json b/evals/evidence/small-case-opus.json new file mode 100644 index 0000000..c45f88d --- /dev/null +++ b/evals/evidence/small-case-opus.json @@ -0,0 +1,121 @@ +{ + "model": "opus", + "runs": 3, + "arms": [ + "with", + "without" + ], + "cases": [ + { + "name": "delegates-codebase-question", + "arms": { + "with": [ + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + }, + { + "grader": "scout-was-the-agent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + }, + { + "grader": "scout-was-the-agent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + }, + { + "grader": "scout-was-the-agent", + "pass": true + } + ] + } + ], + "without": [ + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + }, + { + "grader": "scout-was-the-agent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + }, + { + "grader": "scout-was-the-agent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + }, + { + "grader": "scout-was-the-agent", + "pass": false + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/surveyor-case-haiku.json b/evals/evidence/surveyor-case-haiku.json new file mode 100644 index 0000000..1ba2a77 --- /dev/null +++ b/evals/evidence/surveyor-case-haiku.json @@ -0,0 +1,79 @@ +{ + "model": "sonnet", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "surveyor-traces-the-chain", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 0.1793826, + "durationMs": 54029, + "turns": 20, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.1793826, + "inputTokens": 162, + "cacheReadInputTokens": 798076, + "cacheCreationInputTokens": 40454, + "outputTokens": 3701 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.14531750000000002, + "durationMs": 45688, + "turns": 17, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.14531750000000002, + "inputTokens": 285, + "cacheReadInputTokens": 614175, + "cacheCreationInputTokens": 33945, + "outputTokens": 3145 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 0.1434974, + "durationMs": 61965, + "turns": 26, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.1434974, + "inputTokens": 357, + "cacheReadInputTokens": 877604, + "cacheCreationInputTokens": 16650, + "outputTokens": 4416 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/wide-case-haiku-pin.json b/evals/evidence/wide-case-haiku-pin.json new file mode 100644 index 0000000..1ad9796 --- /dev/null +++ b/evals/evidence/wide-case-haiku-pin.json @@ -0,0 +1,112 @@ +{ + "model": "opus", + "runs": 3, + "arms": [ + "with" + ], + "cases": [ + { + "name": "traces-a-flow-end-to-end", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 1.1419868999999998, + "durationMs": 257314, + "turns": 19, + "modelUsage": { + "claude-opus-5": { + "costUSD": 0.9878954999999999, + "inputTokens": 28, + "cacheReadInputTokens": 605231, + "cacheCreationInputTokens": 35424, + "outputTokens": 13236 + }, + "claude-haiku-4-5-20251001": { + "costUSD": 0.1540914, + "inputTokens": 162, + "cacheReadInputTokens": 583094, + "cacheCreationInputTokens": 58540, + "outputTokens": 4489 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.2602386, + "durationMs": 291175, + "turns": 22, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.0320255, + "inputTokens": 24, + "cacheReadInputTokens": 546781, + "cacheCreationInputTokens": 42319, + "outputTokens": 13413 + }, + "claude-haiku-4-5-20251001": { + "costUSD": 0.22821309999999997, + "inputTokens": 234, + "cacheReadInputTokens": 1211291, + "cacheCreationInputTokens": 53708, + "outputTokens": 7943 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.31517195, + "durationMs": 332643, + "turns": 24, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.1048195, + "inputTokens": 24, + "cacheReadInputTokens": 538489, + "cacheCreationInputTokens": 40183, + "outputTokens": 17345 + }, + "claude-haiku-4-5-20251001": { + "costUSD": 0.21035245000000002, + "inputTokens": 218, + "cacheReadInputTokens": 1070882, + "cacheCreationInputTokens": 50537, + "outputTokens": 7975 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/evidence/wide-case-sonnet-pin-and-baseline.json b/evals/evidence/wide-case-sonnet-pin-and-baseline.json new file mode 100644 index 0000000..7f1319c --- /dev/null +++ b/evals/evidence/wide-case-sonnet-pin-and-baseline.json @@ -0,0 +1,211 @@ +{ + "model": "opus", + "runs": 3, + "arms": [ + "with", + "without" + ], + "cases": [ + { + "name": "traces-a-flow-end-to-end", + "arms": { + "with": [ + { + "exitCode": 0, + "costUsd": 1.5592256, + "durationMs": 190898, + "turns": 14, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.0736555, + "inputTokens": 24, + "cacheReadInputTokens": 469291, + "cacheCreationInputTokens": 58509, + "outputTokens": 10152 + }, + "claude-sonnet-5": { + "costUSD": 0.4855701, + "inputTokens": 36, + "cacheReadInputTokens": 460632, + "cacheCreationInputTokens": 65130, + "outputTokens": 6869 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.9984075499999998, + "durationMs": 300231, + "turns": 28, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.3613790000000001, + "inputTokens": 34, + "cacheReadInputTokens": 791958, + "cacheCreationInputTokens": 46058, + "outputTokens": 20186 + }, + "claude-sonnet-5": { + "costUSD": 0.63702855, + "inputTokens": 18, + "cacheReadInputTokens": 341761, + "cacheCreationInputTokens": 117811, + "outputTokens": 6177 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.4679628000000002, + "durationMs": 291462, + "turns": 24, + "modelUsage": { + "claude-opus-5": { + "costUSD": 1.0716640000000002, + "inputTokens": 30, + "cacheReadInputTokens": 661468, + "cacheCreationInputTokens": 38838, + "outputTokens": 14096 + }, + "claude-sonnet-5": { + "costUSD": 0.3962988, + "inputTokens": 38, + "cacheReadInputTokens": 550441, + "cacheCreationInputTokens": 38338, + "outputTokens": 5819 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": true + } + ] + } + ], + "without": [ + { + "exitCode": 0, + "costUsd": 1.3390605, + "durationMs": 277983, + "turns": 29, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.000629, + "inputTokens": 549, + "cacheReadInputTokens": 0, + "cacheCreationInputTokens": 0, + "outputTokens": 16 + }, + "claude-opus-5": { + "costUSD": 1.3384315, + "inputTokens": 107, + "cacheReadInputTokens": 782453, + "cacheCreationInputTokens": 48142, + "outputTokens": 18610 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.3821754999999998, + "durationMs": 272155, + "turns": 29, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.000629, + "inputTokens": 549, + "cacheReadInputTokens": 0, + "cacheCreationInputTokens": 0, + "outputTokens": 16 + }, + "claude-opus-5": { + "costUSD": 1.3815464999999998, + "inputTokens": 263, + "cacheReadInputTokens": 897893, + "cacheCreationInputTokens": 48361, + "outputTokens": 17907 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + } + ] + }, + { + "exitCode": 0, + "costUsd": 1.355927, + "durationMs": 246303, + "turns": 28, + "modelUsage": { + "claude-haiku-4-5-20251001": { + "costUSD": 0.000629, + "inputTokens": 549, + "cacheReadInputTokens": 0, + "cacheCreationInputTokens": 0, + "outputTokens": 16 + }, + "claude-opus-5": { + "costUSD": 1.355298, + "inputTokens": 3831, + "cacheReadInputTokens": 863726, + "cacheCreationInputTokens": 46123, + "outputTokens": 17722 + } + }, + "scores": [ + { + "grader": "answer-is-correct", + "pass": true + }, + { + "grader": "delegated-to-a-subagent", + "pass": false + } + ] + } + ] + } + } + ] +} \ No newline at end of file diff --git a/evals/fixtures/order-service.gen.mjs b/evals/fixtures/order-service.gen.mjs index c0a36dd..8562c91 100644 --- a/evals/fixtures/order-service.gen.mjs +++ b/evals/fixtures/order-service.gen.mjs @@ -68,6 +68,11 @@ function cap(word) { return word[0].toUpperCase() + word.slice(1); } +// Most helpers export ``; retry is the one that does not. +function helperExport(helper, S) { + return helper === "retry" ? `withRetry${S}` : `${helper}${S}`; +} + // Every other stage name, so that grepping for one stage hits many files and // the chain still has to be read rather than pattern-matched. function otherStages(stage) { @@ -77,7 +82,10 @@ function otherStages(stage) { function stageSource(stage, fn, subject, nextStage) { const S = cap(stage); const imports = [ - ...HELPERS.map((h) => `import { ${h}${S} } from "./${h}.js";`), + // The retry helper exports `withRetry`, not `retry` - importing + // it by the pattern name left every stage module unlinkable, which made the + // fixture unrunnable while still reading correctly. + ...HELPERS.map((h) => `import { ${helperExport(h, S)} } from "./${h}.js";`), nextStage ? `import { ${nextStage[1]} } from "../${nextStage[0]}/index.js";` : "", ] .filter(Boolean) diff --git a/evals/forced-dispatch-tier/NOTES.md b/evals/forced-dispatch-tier/NOTES.md index dcac4f6..290e2f5 100644 --- a/evals/forced-dispatch-tier/NOTES.md +++ b/evals/forced-dispatch-tier/NOTES.md @@ -68,6 +68,6 @@ Superpowers-by-default question stays open. ## Also open -`scout` is still pinned sonnet, as it ships. The wide case says haiku is 9% -cheaper on that workload with correctness held; that is one workload, and the -quality case that would justify re-pinning has not been written. +Nothing about the `scout` pin: `subagent-answer-quality` settled that in this +same branch. It stays sonnet, and breadth work goes to `surveyor` (haiku) +instead. diff --git a/evals/run-local.mjs b/evals/run-local.mjs index 15b7995..0426d39 100644 --- a/evals/run-local.mjs +++ b/evals/run-local.mjs @@ -19,8 +19,17 @@ const EVALS_DIR = dirname(fileURLToPath(import.meta.url)); const PLUGIN_ROOT = resolve(EVALS_DIR, ".."); const FIXTURES_DIR = join(EVALS_DIR, "fixtures"); +// Frontmatter this runner reads: name, tags, fixture, agent, runs, +// timeout_seconds, tools, append_system_prompt. `max_turns` is part of the real +// harness's format and is carried in the case files, but the `claude` CLI has no +// flag for it - only `timeout_seconds` actually bounds a run here. const args = parseArgs(process.argv.slice(2)); -const cases = discoverCases().filter((c) => !args.case || c.name.includes(args.case)); +// A case tagged `unfinished` has a reason not to be trusted written down beside +// it, so it does not get to burn real tokens just for sitting in the directory. +// Naming it with --case opts back in. +const cases = discoverCases().filter((c) => + args.case ? c.name.includes(args.case) : !toList(c.frontmatter.tags)?.includes("unfinished"), +); if (cases.length === 0) { console.error("No eval cases found."); process.exit(1); @@ -39,6 +48,13 @@ for (const kase of cases) { const armResults = {}; for (const arm of arms) { + // A case that runs AS a bundled agent has nothing to say without the plugin + // that defines the agent - the baseline arm would dispatch a name that does + // not exist and bill for the failure. + if (arm === "without" && kase.frontmatter.agent) { + console.log(`${kase.name}: skipping the without arm - agent "${kase.frontmatter.agent}" only exists with the plugin`); + continue; + } armResults[arm] = []; for (let i = 0; i < runs; i++) { process.stderr.write(`${kase.name} [${arm}] run ${i + 1}/${runs} ... `); @@ -48,7 +64,14 @@ for (const kase of cases) { writeFileSync(join(resultsDir, `${kase.name}.${arm}.${i + 1}.stderr.txt`), run.stderr, "utf-8"); process.stderr.write(`exit ${run.exitCode}: ${run.stderr.trim().split("\n").pop() ?? ""}\n`); } - const scores = kase.graders.map((g) => ({ grader: g.name, ...grade(g, run) })); + // A run that died is not a run that scored zero - it is a run with no + // result. Grading its truncated trace would let an empty transcript pass + // every `not_contains` grader, and averaging its missing cost as absent + // rather than as failure would flatter the arm it belongs to. + const scores = + run.exitCode === 0 + ? kase.graders.map((g) => ({ grader: g.name, ...grade(g, run) })) + : kase.graders.map((g) => ({ grader: g.name, pass: false, notRun: true })); armResults[arm].push({ exitCode: run.exitCode, costUsd: run.costUsd, @@ -62,6 +85,7 @@ for (const kase of cases) { } } + if (Object.keys(armResults).length === 0) continue; report.cases.push({ name: kase.name, arms: armResults }); printCase(kase, armResults); } @@ -78,11 +102,19 @@ console.log(`\nTraces and aggregate.json: ${resultsDir}`); function fixtureFor(kase) { const name = args.fixture ?? kase.frontmatter.fixture ?? "mini-app"; const dir = join(FIXTURES_DIR, name); - if (isDir(dir)) return dir; - const generator = join(FIXTURES_DIR, `${name}.gen.mjs`); - if (!isFile(generator)) throw new Error(`No fixture "${name}" and no ${name}.gen.mjs to build one`); - execFileSync(process.execPath, [generator], { stdio: "inherit" }); + + if (!isFile(generator)) { + if (isDir(dir)) return dir; + throw new Error(`No fixture "${name}" and no ${name}.gen.mjs to build one`); + } + + // Regenerate whenever the generator is newer than what it produced. A stale + // fixture kept because the directory happened to exist is how a fixed + // generator quietly fails to reach the runs that report on it. + if (!isDir(dir) || statSync(generator).mtimeMs > statSync(dir).mtimeMs) { + execFileSync(process.execPath, [generator], { stdio: "inherit" }); + } if (!isDir(dir)) throw new Error(`${name}.gen.mjs ran but produced no ${dir}`); return dir; } @@ -120,8 +152,15 @@ async function runOnce(kase, arm, timeoutMs) { return { trace: turns.map((t) => t.trace).join("\n"), stderr: turns.map((t) => t.stderr).join("\n"), - exitCode: turns.find((t) => t.exitCode !== 0)?.exitCode ?? 0, + // A killed child closes with code null, and `null ?? 0` would report a + // timed-out run as clean - its truncated trace graded, its null cost quietly + // dropped from the mean. Anything that is not a clean 0 counts as a failure. + exitCode: turns.find((t) => t.exitCode !== 0) ? (turns.find((t) => t.exitCode !== 0).exitCode ?? 1) : 0, lastMessage: turns.map((t) => t.lastMessage).join("\n\n"), + // Graded per turn as well as joined: an ordered regex run over the + // concatenation can assemble a passing match out of pieces of several + // answers, which is a pass the run did not earn. + turnMessages: turns.map((t) => t.lastMessage), costUsd: sum(turns.map((t) => t.costUsd)), durationMs: sum(turns.map((t) => t.durationMs)), turns: sum(turns.map((t) => t.turns)), @@ -129,6 +168,11 @@ async function runOnce(kase, arm, timeoutMs) { }; } +function sessionModelFor(agent) { + if (args.modelGiven) return args.model; + return agent ? null : args.model; +} + function toList(value) { if (value == null) return null; return Array.isArray(value) ? value : [value]; @@ -182,13 +226,21 @@ async function runTurn(prompt, arm, workspace, timeoutMs, isFollowup, appendSyst "", "--permission-mode", "dontAsk", - "--model", - args.model, + // `--model` outranks an agent's frontmatter pin, so a case that runs AS an + // agent must not be handed the default or it silently measures the wrong + // tier. Pass one only when the caller asked for it by name. + ...(sessionModelFor(agent) ? ["--model", sessionModelFor(agent)] : []), ]; if (arm === "with") argv.push("--plugin-dir", PLUGIN_ROOT); + // CLAUDE_CODE_SUBAGENT_MODEL overrides the tier of every subagent, which is + // the thing several of these cases exist to measure. Whatever the operator has + // set for their own work must not reach the experiment. + const env = { ...process.env }; + delete env.CLAUDE_CODE_SUBAGENT_MODEL; + // No `shell: true` - it concatenates argv, which mangles the prompt. - const child = spawn(process.platform === "win32" ? "claude.exe" : "claude", argv, { cwd: workspace }); + const child = spawn(process.platform === "win32" ? "claude.exe" : "claude", argv, { cwd: workspace, env }); let stdout = ""; let stderr = ""; child.stdout.on("data", (d) => (stdout += d)); @@ -228,9 +280,20 @@ function extractResult(stdout) { function grade(grader, run) { if (grader.type !== "regex") return { pass: false, skipped: `unsupported grader type: ${grader.type}` }; - const haystack = grader.target === "last_message" ? run.lastMessage : run.trace; - const hit = new RegExp(grader.pattern, grader.flags ?? "").test(haystack); + const pattern = new RegExp(grader.pattern, grader.flags ?? ""); const wanted = grader.match ?? "contains"; + + // `last_message` is graded turn by turn: one turn has to satisfy the pattern + // on its own. Matching the joined text instead would let a multi-part regex + // straddle two answers and pass on neither. + if (grader.target === "last_message") { + const messages = run.turnMessages ?? [run.lastMessage]; + return wanted === "not_contains" + ? { pass: messages.every((m) => !pattern.test(m)) } + : { pass: messages.some((m) => pattern.test(m)) }; + } + + const hit = pattern.test(run.trace); return { pass: wanted === "not_contains" ? !hit : hit }; } @@ -251,9 +314,14 @@ function printCase(kase, armResults) { // main session buys context back and pays for it in dollars and latency, so // print the price next to the score rather than letting the score stand alone. const price = Object.entries(armResults).map(([arm, runs]) => { + const priced = runs.filter((r) => typeof r.costUsd === "number"); const cost = mean(runs.map((r) => r.costUsd)); const seconds = mean(runs.map((r) => r.durationMs)) / 1000; - return `${arm} $${cost.toFixed(3)} / ${seconds.toFixed(0)}s`; + // Say how many runs are behind the mean. A run that died has no cost to + // contribute, and a mean over two runs that looks like a mean over three is + // the kind of quiet thing that gets published. + const basis = priced.length === runs.length ? "" : ` (${priced.length}/${runs.length} runs)`; + return `${arm} $${cost.toFixed(3)} / ${seconds.toFixed(0)}s${basis}`; }); console.log(` cost per run: ${price.join(" | ")}`); @@ -351,10 +419,13 @@ function parseScalar(raw) { } function parseArgs(argv) { - const out = { runs: null, model: "sonnet", arm: "both", case: null, fixture: null }; + const out = { runs: null, model: "sonnet", modelGiven: false, arm: "both", case: null, fixture: null }; for (let i = 0; i < argv.length; i++) { if (argv[i] === "--runs") out.runs = Number(argv[++i]); - else if (argv[i] === "--model") out.model = argv[++i]; + else if (argv[i] === "--model") { + out.model = argv[++i]; + out.modelGiven = true; + } else if (argv[i] === "--arm") out.arm = argv[++i]; else if (argv[i] === "--case") out.case = argv[++i]; else if (argv[i] === "--fixture") out.fixture = argv[++i]; diff --git a/evals/surveyor-traces-the-chain/graders/answer-is-correct.md b/evals/surveyor-traces-the-chain/graders/answer-is-correct.md new file mode 100644 index 0000000..9d4f222 --- /dev/null +++ b/evals/surveyor-traces-the-chain/graders/answer-is-correct.md @@ -0,0 +1,15 @@ +--- +type: regex +target: last_message +pattern: 'intake[\s\S]*validation[\s\S]*pricing[\s\S]*discount[\s\S]*tax[\s\S]*deposit[\s\S]*payment[\s\S]*tips[\s\S]*fiscal[\s\S]*receipt[\s\S]*export[\s\S]*archive' +flags: 'i' +match: contains +--- + +The claim the routing anchor now makes out loud is that breadth work runs +correctly a tier down. This is the case that has to keep being true for that +sentence to stay in a file every session pays for. + +Twelve stages in the right order. The names are guessable from a directory +listing, so the ordering is the whole answer - and no file states it, since each +stage imports the next at the bottom of a few hundred lines. diff --git a/evals/surveyor-traces-the-chain/prompt.md b/evals/surveyor-traces-the-chain/prompt.md new file mode 100644 index 0000000..4f39362 --- /dev/null +++ b/evals/surveyor-traces-the-chain/prompt.md @@ -0,0 +1,12 @@ +--- +name: surveyor-traces-the-chain +tags: [routing, tier, breadth] +plugins: ["../.."] +fixture: order-service +agent: surveyor +runs: 3 +max_turns: 30 +timeout_seconds: 600 +--- + +An order enters this service at intake and ends up archived. Trace the whole path: list every stage it passes through, in order, from intake to archive. diff --git a/hooks/dispatch-counter.mjs b/hooks/dispatch-counter.mjs index 6a27322..d5c4cf1 100644 --- a/hooks/dispatch-counter.mjs +++ b/hooks/dispatch-counter.mjs @@ -81,6 +81,7 @@ function sessionFilterFromArgs(argv) { // Keep in sync with agents/*.md; one sync test guards both columns. const AGENT_PINS = { "model-routing:scout": { model: "sonnet", effort: "low" }, + "model-routing:surveyor": { model: "haiku", effort: "low" }, "model-routing:test-runner": { model: "haiku", effort: "low" }, "model-routing:e2e-runner": { model: "sonnet", effort: "medium" }, "model-routing:verifier": { model: "haiku", effort: "low" }, diff --git a/hooks/routing-anchor.md b/hooks/routing-anchor.md index 43e171c..f624e0d 100644 --- a/hooks/routing-anchor.md +++ b/hooks/routing-anchor.md @@ -2,7 +2,7 @@ MODEL ROUTING ACTIVE - expensive model thinks, cheap models grind. You cannot sw - Planning, specs, docs, architecture, small interactive edits: main session. - Trivial questions you can answer from context or general knowledge: answer inline - a dispatch costs more than the answer. -- Codebase exploration ("where is X", "how does Y work"): `scout`. Locate-only sweeps ("which files mention X"): the built-in Explore agent, when present, is cheaper. Enumerating or tracing a chain ("list the stages in order", "which files import X"): dispatch `scout` with `model=haiku` - measured right every run at half the price. Working out what code actually does ("does this loop retry", "what does this return"): leave the sonnet pin - haiku got that wrong in 1 run of 3. +- Codebase exploration ("where is X", "how does Y work"): `scout`. Enumerating or tracing a chain ("list the stages in order", "which files import X"): `surveyor` (haiku) - breadth runs correctly a tier down, judgement does not. Locate-only sweeps ("which files mention X"): the built-in Explore agent, when present, is cheaper than either. - Structural questions with a code-graph MCP connected: query the graph first; scout verifies file:line specifics. - Implementing an approved plan/spec: `implementer` (sonnet). Dispatch with `model=opus` for multi-file/cross-layer work, security/money/migrations/concurrency/public contracts, or a retry after a weak sonnet result - escalate when in doubt; the dispatch changes the model only, the pinned medium effort stays. Ambiguous task or unclear root cause: clarify first (main session / scout) - implementer stops on ambiguity. Batch related tasks per agent. - Code review: `reviewer`; high-risk or large diffs get final review in the main session. diff --git a/skills/model-routing/SKILL.md b/skills/model-routing/SKILL.md index 48bbd1b..4f69111 100644 --- a/skills/model-routing/SKILL.md +++ b/skills/model-routing/SKILL.md @@ -15,10 +15,12 @@ A subagent starts empty, so everything it reads is a cache write, while the main session pays cache read - 12.5x less - for context it already holds. That penalty is paid whether or not the tier is routed down, so the rules below only pay off on work that was going to be delegated anyway. -Measured on a wide-reading session: doing it inline cost $1.36, delegating -with the tier routed down $1.68, delegating at the session tier $2.01. -Route every dispatch - that is a 17% discount on work already leaving the -session - and do not manufacture dispatches to collect it. +On one wide-reading session, three runs: doing it inline cost $1.36 and +delegating with the tier routed down $1.68, both measured; delegating at +the session tier comes to $2.01, which is those same measured subagent +tokens repriced at the higher tier rather than a run of its own. Route +every dispatch - roughly a sixth off work already leaving the session - +and do not manufacture dispatches to collect it. ## Tiers @@ -107,6 +109,7 @@ carries no effort param: |------|-------|---------------|--------| | Planning, brainstorming, specs, docs, architecture | main session | strongest (user's /model choice) | high | | Codebase exploration ("where is X", "how does Y work") | subagent | `scout` (sonnet) | low | +| Breadth sweeps: enumerate, list, trace a chain end to end | subagent | `surveyor` (haiku) | low | | Implementing an approved plan/spec (ordinary: single-file, clear shape) | subagent | `implementer` (sonnet) | medium | | Complex implementation: multi-file refactor, subtle concurrency/security | subagent | `implementer` with `model=opus` | medium (pinned) | | Trivial mechanical tasks: renames, boilerplate, mirrored constants | subagent | sonnet | low | @@ -224,18 +227,24 @@ actually ran with `/model-routing:stats`. Explore agent, when present, is cheaper than `scout`. Use `scout` when the answer needs verification - tracing real code paths and confirming file:line - not just finding candidates. -- Pick `scout`'s tier by what the question demands, not by what it costs. +- Split exploration by what the question demands, not by what it costs. Enumerating and tracing - list these stages in order, which files import - X, where does this chain end - is breadth, and breadth runs correctly on - haiku: dispatch with `model=haiku` and it comes back right for about half - the price. Working out what code actually does - does this loop retry, - what does this function return for that input - is not breadth, and the - cheap tier fails it in a way that looks confident. Leave the sonnet pin - there. Both halves are measured, in `evals/`: on a twelve-stage tracing - question haiku scored 3 of 3 and cut the session bill 9%; on a question - whose code contains an obvious wrong answer, haiku took the bait in 1 run - of 3 while sonnet took it in none. A wrong answer sends the main session - back to read the files itself, which costs more than the tier ever saved. + X, where does this chain end - is breadth, and breadth runs correctly a + tier down: that is `surveyor` (haiku). Working out what code actually + does - does this loop retry, what does this function return for that + input - is not breadth, and the cheap tier fails it in a way that looks + confident: that is `scout` (sonnet). Both halves are measured in `evals/`, + on one generated fixture at three runs each: on a twelve-stage tracing + question haiku answered correctly every run at a third of sonnet's price; + on a question whose code contains an obvious wrong answer, haiku took the + bait once in three runs while sonnet took it in none. Three runs on one + fixture cannot pin a failure rate, and the reason to keep judgement on + sonnet is not the rate but the asymmetry: a cheap right answer saves + cents, while a cheap wrong one arrives looking identical and sends the + main session back to read the files itself. That recovery was not + measured; it does not need to be, to be worth avoiding. Reach for the + pinned agent rather than overriding `scout` downward - the floor rule + below is not a formality. - Batch related plan tasks per subagent. Each subagent re-reads files from scratch; one tiny task per agent costs more than it saves. - Subagents cannot see the conversation. Write self-contained task From 69ad03eddc338cc750eb89dcac1227228191b54a Mon Sep 17 00:00:00 2001 From: Artem Frolov Date: Fri, 21 Aug 2026 02:54:52 +0300 Subject: [PATCH 9/9] fix: tell the three exploration routes apart by what has to come back "Which files mention X" and "which files import X" sat one line apart in the anchor, routed to different agents, and read as the same question. A rule a session cannot apply is worse than no rule, and this one is paid for in every session. The three routes now split on the answer rather than the phrasing: unverified candidates to look at next go to the built-in Explore, a complete list or a verified ordering goes to `surveyor`, and a judgement about what code does goes to `scout`. The skill says outright that the two lookalike questions are not the same - one is answered by grep and may be wrong at the edges, the other has to be right. --- hooks/routing-anchor.md | 2 +- skills/model-routing/SKILL.md | 13 +++++++++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/hooks/routing-anchor.md b/hooks/routing-anchor.md index f624e0d..987973f 100644 --- a/hooks/routing-anchor.md +++ b/hooks/routing-anchor.md @@ -2,7 +2,7 @@ MODEL ROUTING ACTIVE - expensive model thinks, cheap models grind. You cannot sw - Planning, specs, docs, architecture, small interactive edits: main session. - Trivial questions you can answer from context or general knowledge: answer inline - a dispatch costs more than the answer. -- Codebase exploration ("where is X", "how does Y work"): `scout`. Enumerating or tracing a chain ("list the stages in order", "which files import X"): `surveyor` (haiku) - breadth runs correctly a tier down, judgement does not. Locate-only sweeps ("which files mention X"): the built-in Explore agent, when present, is cheaper than either. +- Codebase exploration: pick by what has to come back. A judgement about what code does ("how does Y work", "does this retry") is `scout`. A complete list or an ordering, verified by following the code ("every stage in order", "everything that imports X") is `surveyor` (haiku) - breadth holds a tier down, judgement does not. Candidate files to look at next, unverified ("which files mention X") is the built-in Explore agent, cheaper than either. - Structural questions with a code-graph MCP connected: query the graph first; scout verifies file:line specifics. - Implementing an approved plan/spec: `implementer` (sonnet). Dispatch with `model=opus` for multi-file/cross-layer work, security/money/migrations/concurrency/public contracts, or a retry after a weak sonnet result - escalate when in doubt; the dispatch changes the model only, the pinned medium effort stays. Ambiguous task or unclear root cause: clarify first (main session / scout) - implementer stops on ambiguity. Batch related tasks per agent. - Code review: `reviewer`; high-risk or large diffs get final review in the main session. diff --git a/skills/model-routing/SKILL.md b/skills/model-routing/SKILL.md index 4f69111..b402aea 100644 --- a/skills/model-routing/SKILL.md +++ b/skills/model-routing/SKILL.md @@ -223,10 +223,15 @@ actually ran with `/model-routing:stats`. `test-runner` and consume its compact report. - Route codebase exploration to `scout` - conclusions and file:line refs come back, file dumps stay in the subagent. -- For locate-only sweeps ("which files mention X") the harness's built-in - Explore agent, when present, is cheaper than `scout`. Use `scout` when - the answer needs verification - tracing real code paths and confirming - file:line - not just finding candidates. +- Three exploration routes, told apart by what has to come back rather + than by how the question is phrased. Unverified candidates to look at + next ("which files mention X"): the harness's built-in Explore agent, + cheaper than both bundled ones. A complete list or an ordering, verified + by following the code ("every stage in order", "everything that imports + X"): `surveyor`. A judgement about behaviour ("how does Y work", "does + this retry"): `scout`. "Which files import X" and "which files mention + X" look alike and are not: one is answered by grep and may be wrong at + the edges, the other has to be right. - Split exploration by what the question demands, not by what it costs. Enumerating and tracing - list these stages in order, which files import X, where does this chain end - is breadth, and breadth runs correctly a