Skip to content

test(mcp-core): pin the MCP wire surface with a per-combo snapshot baseline - #537

Open
aliasunder wants to merge 3 commits into
mainfrom
worktree-tool-surface-snapshot
Open

test(mcp-core): pin the MCP wire surface with a per-combo snapshot baseline#537
aliasunder wants to merge 3 commits into
mainfrom
worktree-tool-surface-snapshot

Conversation

@aliasunder

Copy link
Copy Markdown
Owner

What

Adds a committed baseline of the MCP wire surface — tool input schemas, descriptions, annotations, the prompt surface, and the server instructions — captured per config combination, with a vitest drift test that fails on any byte change and a one-command regen (npm run snapshot:update).

Why

Nothing in CI byte-gated the tool surface: a PR could rename a param, drop an annotation, or reword an Errors literal with every check staying green. The committed baseline turns any surface change into a deliberate, reviewable diff in the same PR, and the baseline at each release commit becomes the stability contract's regression reference.

Design

  • Capture — a real McpServer booted through the real registerTools/registerPrompts path, read over InMemoryTransport via client.listTools(), client.listPrompts(), and client.getInstructions(). This pins the SDK's own Zod→JSON-Schema serialization — exactly what clients see on the wire. A dependency bump that changes that serialization fails the drift test until the baseline is regenerated in the same PR; that failure is the gate working, not flake.
  • Combos — the cross-product of the four gating axes (READONLY_MODE × MEMORY_ENABLED × FILE_TOOLS_ENABLED × EMBEDDING_ENABLED) plus one DISABLED_TOOLS representative (disabling vault_patch_note, which several other tool descriptions cross-reference) = 17 snapshot files. The axis list is a declared array, so a new gating axis is a one-line addition.
  • Snapshots — byte-exact JSON written by toMatchFileSnapshot (the string is pre-serialized, so the files are plain JSON). The snapshot directory is prettier-ignored so lint-staged and prettier:check can't reformat it. A directory-listing assertion pins exactly one file per combo — an orphaned file after an axis change, or a locally auto-created one that skipped a deliberate regen, fails the test.
  • Review ergonomics.gitattributes marks the multi-flip combo files (*+*.json) linguist-generated, so GitHub collapses their PR diffs; the default, four single-flip, and disabled-tools files stay expanded, and every distinct rendered text has a visible diff home.
  • buildServerMetadata is now exported from mcp-router.ts so the capture builds instructions through the same code path the router uses.

Tests

18 tests: 17 combo baselines + the directory-listing assertion. Full suite green (3,576 passing; the one local failure is the pre-existing OAuth sliding-expiry test that fails on Eastern-Time machines across the November DST boundary and passes in CI's UTC — unrelated to this change).

Mutation checks — each mutation applied, the failing set observed, then reverted:

Mutation Expected failing combos Observed
One word in vault_read_note's description all 17 (tool served everywhere) 17 ✓
Annotation flip (additive-write idempotentHint) the 9 non-readonly combos 9 ✓
Zod .default(false).default(true) (prune_empty_folders) the 9 combos serving vault_delete_note 9 ✓
Embedding-off ternary branch only (vault_memory_recall description) the 4 embedding-off combos still serving memory tools exactly those 4 ✓
Read-only instructions clause in buildServerMetadata the 8 readonly combos exactly those 8 ✓

Regen flow exercised end-to-end: an intentional description edit → npm run snapshot:update (scoped to the one test file, ~0.5 s) → 17 one-line diffs showing exactly the edit → reverted, baseline byte-identical afterward. Lint (0 errors), knip, prettier:check, markdownlint, and the full build (server + cli + sst) are green.

Not exercised: GitHub's collapsed rendering of the linguist-generated files — verifiable on this PR's Files view. Runtime response shapes are intentionally out of scope: the snapshot sees schemas and rendered text; response-shape stability stays enforced by the integration suite's exact assertions.

🤖 Generated with Claude Code

aliasunder and others added 2 commits September 5, 2026 21:24
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@umm-actually umm-actually Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm-actually — review skipped

diff too large for context budget (483111 tokens, limit 150000 of 300000)


umm-actually

…clause

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@umm-actually umm-actually Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm-actually — review skipped

diff too large for context budget (483084 tokens, limit 150000 of 300000)


umm-actually

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant