From dd6d7b0659334f6e885643acf8ecd05f349d380c Mon Sep 17 00:00:00 2001 From: VickyXAI <115643921+VickyXAI@users.noreply.github.com> Date: Wed, 2 Sep 2026 12:46:56 -0500 Subject: [PATCH] docs: how we measure the context cost, and the SDK's dead $schema header MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The measurement method and the ecosystem finding behind 0.45.0 existed only in a session scratch directory, which is a temporary path. The wrapper is a technique other MCP authors can copy and the verification recipe is what keeps the claim checkable a year from now, so it belongs in the repo. Covers: what each profile costs and how to reproduce it against any stdio MCP server; why the SDK stamps a draft-07 $schema header onto every zod-defined tool and how to strip it through public API only; how far "it is dead weight" can actually be defended (no SDK code path reads it, its bundled ajv validator never receives inputSchema, and compiling with and without it gives identical verdicts on 100/100 pairs — closed-source clients were not instrumented, and the doc says so); and the ensure_ascii footgun that inflated the first published figures by 4.4% and forced a retraction. README's card now links here instead of to a section heading. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WUL3ExR4Nz7uebxKaKwjDi --- README.md | 2 +- docs/mcp-schema-overhead.md | 183 ++++++++++++++++++++++++++++++++++++ 2 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 docs/mcp-schema-overhead.md diff --git a/README.md b/README.md index 3020757..ffbf167 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ claude mcp add blockrun -s user -- npx -y @blockrun/mcp@latest -
Every MCP server costs you this. Almost none of them tell you. How we measure it →
+
Every MCP server costs you this. Almost none of them tell you. How we measure it, and how to measure anyone else →
--- diff --git a/docs/mcp-schema-overhead.md b/docs/mcp-schema-overhead.md new file mode 100644 index 0000000..fab5cec --- /dev/null +++ b/docs/mcp-schema-overhead.md @@ -0,0 +1,183 @@ +# What this server costs your context, and how we measure it + +Installing an MCP server spends the model's context on **every turn** — the client loads each +tool's schema into the prompt and re-sends it for the whole session, whether or not a tool is +ever called. This documents what BlockRun MCP costs, how the figure is produced, and one +ecosystem-wide finding that came out of measuring it. + +Harness: [`scripts/measure-tool-schema.mjs`](../scripts/measure-tool-schema.mjs) +(`npm run measure:schema`). Guard: [`test/schema-tokens.test.ts`](../test/schema-tokens.test.ts), +which fails the build when the README card disagrees with a live measurement. + +Written 2026-09-01, verified against `@modelcontextprotocol/sdk` 1.29.0. + +## Our number + +| Profile | Tools | Context | +|---------|-------|---------| +| `full` *(default)* | 20 | 12,900 | +| `trading` | 9 | 5,554 | +| `media` | 7 | 5,436 | +| `research` | 6 | 3,024 | +| `chat` | 3 | 1,924 | + +Descriptions are ~54% of it, input schemas ~41%. `--profile trading` costs 57% less than the +default for the same workflow. + +Measure it yourself, against us or anyone else: + +```bash +npm i gpt-tokenizer +npm run measure:schema # this server, every profile +node scripts/measure-tool-schema.mjs -- npx -y @some/other-mcp-server +``` + +Measure a *published* package from a directory outside this repo — `npx -y @blockrun/mcp@x.y.z` +run from inside the checkout resolves to the local build instead. + +## The dead `$schema` header — an ecosystem finding + +If you build an MCP server with the official TypeScript SDK and define your tools with zod — +which is the documented, idiomatic way — every tool's `inputSchema` goes out to the client +carrying this: + +```json +"$schema": "http://json-schema.org/draft-07/schema#" +``` + +It is a JSON Schema *dialect declaration*, and as far as can be verified it does nothing (see +**How dead is it, exactly** below — the answer is strong but not unlimited). It rides in the tool +block the model carries on every turn, and the server author never wrote it and cannot see it in +their source. + +Cost: **~15 tokens per tool.** For us, 20 tools → 300 tokens. On a 49-tool server of the kind +Uber cited, ~735 tokens. Nobody's tool budget is blown by this, but it is 100% waste, it is +invisible from the source code, and it is in essentially every SDK-built server in the ecosystem. + +## How dead is it, exactly + +Stronger than inference, weaker than "every client ignores it". Verified: + +- **No code path in the SDK reads `$schema`.** Grepped the whole `dist`, server and client: zero + reads. +- **The SDK's bundled validator never receives it.** ajv (`validation/ajv-provider.js`) is invoked + on `tool.outputSchema` and on elicitation `requestedSchema` — never on `inputSchema`. In the + reference implementation the header is not even handed to a validator. +- **It is inert for ajv anyway.** Compiling all 20 tool schemas under the SDK's exact ajv config, + with and without the header, against 5 input samples each: identical verdicts on **100/100** + pairs, 0 differences. + +Not verified: proprietary client behaviour. Claude Code, Cursor, VS Code and Claude Desktop were +not instrumented with and without the header, and their source is not readable. So the publishable +sentence is: + +> No code path in the reference SDK reads it, its bundled validator never receives it, and +> compiling the schemas with and without it produces identical validation verdicts. We did not +> instrument closed-source clients. + +One nuance that cuts *toward* removing it: ajv normally uses `$schema` to select a dialect, so such +a header is not inert by construction — a draft-2020-12 URI on a draft-07 ajv instance throws. It +is inert here because the SDK emits draft-07 and configures ajv with `validateSchema: false`. That +is "happens to be fine", not "cannot matter". + +## How it was verified + +Not from reading one code path. Both zod branches were executed directly: + +```ts +import { toJsonSchemaCompat } from "@modelcontextprotocol/sdk/server/zod-json-schema-compat.js"; +import * as z4 from "zod"; +import * as z3 from "zod/v3"; + +toJsonSchemaCompat(z4.object({ a: z4.string() }), { pipeStrategy: "input" }).$schema +// → "http://json-schema.org/draft-07/schema#" +toJsonSchemaCompat(z3.object({ a: z3.string() }), { pipeStrategy: "input" }).$schema +// → "http://json-schema.org/draft-07/schema#" +``` + +The v4 branch goes through `z4mini.toJSONSchema(schema, { target: "draft-7" })`; the v3 branch +through the vendored `zod-to-json-schema`. **Both** emit it, so the version of zod a server +happens to be on makes no difference. + +It was originally found the honest way — by capturing the actual `tools/list` payload off the +wire from a running server and tokenizing it, rather than by reading our own source. That is the +transferable lesson, and arguably the better headline: **the schema the model receives is +generated, not written. If you have only ever read your source, you have not seen what you ship.** + +## Why you cannot just delete it + +The SDK exposes no option to suppress it. `registerTool` converts your zod schema at +`tools/list` time, inside a handler the SDK installs itself. + +The fix that works without forking, patching `node_modules`, or touching SDK internals: wrap the +`tools/list` handler as it is being installed, using only the public `setRequestHandler`. Call it +before registering any tool — that is when the SDK lazily installs the handler. + +Shipped here as [`src/utils/strip-schema-dialect.ts`](../src/utils/strip-schema-dialect.ts), wired +in [`src/mcp-handler.ts`](../src/mcp-handler.ts) and guarded by +[`test/schema-dialect.test.ts`](../test/schema-dialect.test.ts). Copy freely: + +```ts +import { ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js"; + +export function stripJsonSchemaDialect(server: McpServer): void { + const lowLevel = server.server; + if (typeof lowLevel?.setRequestHandler !== "function") return; + const original = lowLevel.setRequestHandler.bind(lowLevel); + + lowLevel.setRequestHandler = ((requestSchema, handler) => { + if (requestSchema !== ListToolsRequestSchema) return original(requestSchema, handler); + return original(requestSchema, async (...args) => { + const result = await handler(...args); + for (const tool of result?.tools ?? []) delete tool.inputSchema?.$schema; + return result; + }); + }) as typeof lowLevel.setRequestHandler; +} +``` + +Two properties worth copying along with the code, because they are what make it safe to ship: + +1. **It is never load-bearing.** The wrapper is identity-matched against + `ListToolsRequestSchema`. If a future SDK stops routing through it, the wrapper stops matching + and the header comes back. Degraded output, never a crash. +2. **It no-ops when there is nothing to wrap.** Test suites commonly pass a minimal fake + `McpServer` with just `registerTool`. An earlier version assumed `server.server` exists and + broke 8 tests. An optimization that throws is worse than no optimization. + +And guard it with a test. An identity-matched wrapper can be silently undone by a dependency +bump, and a saving that reverts unobserved is worse than one never made. + +## Caveats worth stating whenever the number is quoted + +1. **It only affects servers that define tools with zod via `registerTool`.** A server passing + raw JSON Schema is unaffected unless it puts the header there itself. "Every SDK-built server" + means every one using the idiomatic path — most, not all. +2. **`$schema` is a 2% win, not the story.** For us it was 300 tokens out of 13,200. The real + weight is tool *descriptions* — ~54% of what remains, and in our case route catalogues and + per-model pricing tables that belong in on-demand skills. That work is not done. `$schema` is + worth citing as the illustration of the actual lesson: you cannot audit what you have not + measured on the wire. +3. **Tool schemas are prompt-cached.** They sit at the front of the prompt, so after the first + turn they re-send at cache-read rates (~10% of input price). The *context-window* cost is 100% + every turn; the *dollar* cost is roughly a tenth of the naive figure. Say so — the honest + version is damning enough, and the inflated one gets fact-checked. +4. **`o200k_base` is a proxy.** Claude's tokenizer is not public and runs a few percent higher on + JSON, so every figure here is a slight **under**-count, never an over-count. + +## The tokenizer footgun + +Count the projection with the same JSON encoding the wire uses. Python's `json.dumps` defaults to +`ensure_ascii=True`, which rewrites every em-dash, arrow and `·` as `\uXXXX` — six ASCII +characters where the model sees one glyph. That inflated the first published figures by 565 tokens +(4.4%), entirely in the tools whose descriptions use typographic punctuation, and the numbers had +to be retracted after a second implementation disagreed. + +Use `json.dumps(obj, ensure_ascii=False)`, or JS `JSON.stringify`, which never escapes non-ASCII. + +Diagnostic fingerprint: if two counts disagree while their *description* totals match to the +token, this is why — and the tools whose descriptions are pure ASCII will show a delta of exactly +zero. + +**No figure is final until a second harness reproduces it.** The numbers in this document have +been through four independent implementations.