docs: how we measure the context cost, and the SDK's dead $schema header - #133
Merged
Conversation
The measurement method and the ecosystem finding behind 0.45.0 existed only in a session scratch directory, which is a temporary path. The wrapper is a technique other MCP authors can copy and the verification recipe is what keeps the claim checkable a year from now, so it belongs in the repo. Covers: what each profile costs and how to reproduce it against any stdio MCP server; why the SDK stamps a draft-07 $schema header onto every zod-defined tool and how to strip it through public API only; how far "it is dead weight" can actually be defended (no SDK code path reads it, its bundled ajv validator never receives inputSchema, and compiling with and without it gives identical verdicts on 100/100 pairs — closed-source clients were not instrumented, and the doc says so); and the ensure_ascii footgun that inflated the first published figures by 4.4% and forced a retraction. README's card now links here instead of to a section heading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUL3ExR4Nz7uebxKaKwjDi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The measurement method and the ecosystem finding behind 0.45.0 lived only in a session scratch directory — a temporary path that disappears. The wrapper is a technique other MCP authors can copy, and the verification recipe is what keeps the claim checkable a year from now.
What the doc covers
npxresolves to the local build.$schemafinding. The SDK's zod→JSON-Schema conversion stamps"$schema": "http://json-schema.org/draft-07/schema#"onto every tool, on both the zod v3 and v4 branches, and exposes no way to suppress it. ~15 tokens per tool that the author never wrote and cannot see in their source.tools/listthrough the publicsetRequestHandleronly — no forking, nonode_modulespatching, no private-field access — plus the two properties that make it safe to ship: never load-bearing, and no-ops when there is nothing to wrap. Links to the shipped files.inputSchema; compiling all 20 schemas with and without it gives identical verdicts on 100/100 pairs. Closed-source clients were not instrumented, and the doc says so rather than overclaiming.ensure_asciifootgun. Python'sjson.dumpsescapes non-ASCII by default, which inflated the first published figures by 565 tokens (4.4%) and forced a retraction. Includes the diagnostic fingerprint: description totals agreeing to the token while grand totals diverge, and pure-ASCII tools showing a delta of exactly zero.Also
README's context-cost card now links to this doc rather than to a section heading, so the "how we measure it" promise leads somewhere that explains it.
Caveats are stated in the doc rather than omitted:
$schemawas a 2% win and descriptions are the remaining ~54%; schemas are prompt-cached so the dollar cost is roughly a tenth of the context cost;o200k_baseis a proxy that under-counts.Verification
431/431 tests pass;
sync-brand-numbers --checkclean. Docs-only otherwise — no version bump, so this does not trigger a release.🤖 Generated with Claude Code
https://claude.ai/code/session_01WUL3ExR4Nz7uebxKaKwjDi