Skip to content

feat(readme): publish the context-window cost as a badge, guarded by CI - #126

Closed
VickyXAI wants to merge 1 commit into
perf/strip-json-schema-dialectfrom
feat/context-token-badge
Closed

feat(readme): publish the context-window cost as a badge, guarded by CI#126
VickyXAI wants to merge 1 commit into
perf/strip-json-schema-dialectfrom
feat/context-token-badge

Conversation

@VickyXAI

@VickyXAI VickyXAI commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Stacked on #125. Base is perf/strip-json-schema-dialect, not main. The 12,900 figure assumes the $schema header is gone; without #125 every number here is 300 higher.

Why

A package manager has shown you install size for thirty years. Installing an MCP server spends your context window on every turn, whether or not you call the tools — and essentially no server tells you how much. There is no norm because nobody established one.

The precedent worth copying is bundlephobia: it did not win by adding a badge, it won by defining a metric and shipping the tool that measures it, after which bundle size became a competitive axis for everyone. Same move here.

What ships

A badge and a table. The hero gets 🧮 12.9K Context Tokens, and the profile section — where a reader is already choosing — gets what each profile costs:

Profile Tools Context
full (default) 20 12,900
trading 9 5,554
media 7 5,436
research 6 3,024
chat 3 1,924

--profile trading is 57% less context than the default for the same workflow. That was already true and already shipped; it was just never stated anywhere a user would see it.

The harness, committed. scripts/measure-tool-schema.mjs (npm run measure:schema) runs against this server or, with -- <cmd>, any stdio MCP server. The claim is checkable by anyone, against anyone — which is the part that makes it a standard rather than marketing.

A guard. test/schema-tokens.test.ts measures the live server in-process and fails when the README disagrees. It asserts the published form (12.9K) and the table's exact totals: pinning raw counts would fail CI on every wording tweak, while pinning the published figure fails precisely when the claim stops being true. Verified by deliberately corrupting the badge and a table row and confirming each fails.

Honesty, stated in the README rather than omitted

  • Counts the model-visible projection ({name, description, input_schema} + host prefix); excludes annotations/_meta/outputSchema, a further ~3.7% on the wire.
  • o200k_base is a proxy — Claude's tokenizer is not public and runs higher on JSON, so every figure is a slight under-count, never an over-count.
  • Schemas are prompt-cached: 100% context cost every turn, roughly a tenth of that in dollars.
  • 54% of our own cost is descriptions, not schemas. The badge does not flatter us; that work is not done.

Verification

  • 429/429 tests pass (425 before, +4).
  • Typecheck, build, and sync-brand-numbers --check all clean.
  • The number reproduces to the token across four independent implementations: this script, an earlier Python harness, a third-party rewrite, and the in-process test.

Follow-up, not in this PR

Promoting mcp.contextTokens into brand-numbers.json needs a matching change in blockrun's src/app/brand/numbers.json/route.ts first — adding the key here alone would be dropped on the next --refresh and break markers. Kept out deliberately.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WUL3ExR4Nz7uebxKaKwjDi

Package managers have shown install size for decades. Installing an MCP server
spends the user's context on every turn — whether or not the tools are ever
called — and almost nothing shows that number. This publishes ours.

- README hero badge: 12.9K context tokens, plus a per-profile table at the
  point where a reader is already choosing a profile. `--profile trading` is
  57% less context than the default for the same trading workflow, which was
  already true and already shipped, but nowhere stated.

- scripts/measure-tool-schema.mjs: the harness, committed rather than left in
  a scratch directory. Runs against this repo's server or, with `-- <cmd>`,
  any stdio MCP server, so the claim is checkable by anyone against anyone.
  `npm run measure:schema`.

- test/schema-tokens.test.ts: the guard. Measures the live server in-process
  and fails if the README disagrees. Asserts the PUBLISHED form ("12.9K") and
  the table's exact per-profile totals — pinning raw counts would fail CI on
  every wording tweak, while pinning the published figure fails exactly when
  the claim becomes wrong. Verified by deliberately corrupting both the badge
  and a table row and confirming each fails.

The projection counted is {name, description, input_schema} per tool with the
host prefix, matching what lands in the API tools array. Numbers agree to the
token across four independent implementations: this script, an earlier Python
harness, a third-party rewrite, and the in-process test.

Stacked on #125 — the 12,900 figure assumes the $schema header is gone. Merge
that first; without it every number here is 300 higher.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WUL3ExR4Nz7uebxKaKwjDi
@VickyXAI
VickyXAI deleted the branch perf/strip-json-schema-dialect September 2, 2026 03:01
@VickyXAI VickyXAI closed this Sep 2, 2026
@VickyXAI

VickyXAI commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #127 — same commit rebased onto main after #125 merged, plus the 0.45.0 bump. GitHub auto-closed this one when its base branch was deleted; it cannot be reopened with a deleted base.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant