test(mcp-core): pin the MCP wire surface with a per-combo snapshot baseline - #537
Open
aliasunder wants to merge 3 commits into
Open
test(mcp-core): pin the MCP wire surface with a per-combo snapshot baseline#537aliasunder wants to merge 3 commits into
aliasunder wants to merge 3 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…clause Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a committed baseline of the MCP wire surface — tool input schemas, descriptions, annotations, the prompt surface, and the server instructions — captured per config combination, with a vitest drift test that fails on any byte change and a one-command regen (
npm run snapshot:update).Why
Nothing in CI byte-gated the tool surface: a PR could rename a param, drop an annotation, or reword an Errors literal with every check staying green. The committed baseline turns any surface change into a deliberate, reviewable diff in the same PR, and the baseline at each release commit becomes the stability contract's regression reference.
Design
McpServerbooted through the realregisterTools/registerPromptspath, read overInMemoryTransportviaclient.listTools(),client.listPrompts(), andclient.getInstructions(). This pins the SDK's own Zod→JSON-Schema serialization — exactly what clients see on the wire. A dependency bump that changes that serialization fails the drift test until the baseline is regenerated in the same PR; that failure is the gate working, not flake.READONLY_MODE×MEMORY_ENABLED×FILE_TOOLS_ENABLED×EMBEDDING_ENABLED) plus oneDISABLED_TOOLSrepresentative (disablingvault_patch_note, which several other tool descriptions cross-reference) = 17 snapshot files. The axis list is a declared array, so a new gating axis is a one-line addition.toMatchFileSnapshot(the string is pre-serialized, so the files are plain JSON). The snapshot directory is prettier-ignored so lint-staged andprettier:checkcan't reformat it. A directory-listing assertion pins exactly one file per combo — an orphaned file after an axis change, or a locally auto-created one that skipped a deliberate regen, fails the test..gitattributesmarks the multi-flip combo files (*+*.json)linguist-generated, so GitHub collapses their PR diffs; the default, four single-flip, and disabled-tools files stay expanded, and every distinct rendered text has a visible diff home.buildServerMetadatais now exported frommcp-router.tsso the capture builds instructions through the same code path the router uses.Tests
18 tests: 17 combo baselines + the directory-listing assertion. Full suite green (3,576 passing; the one local failure is the pre-existing OAuth sliding-expiry test that fails on Eastern-Time machines across the November DST boundary and passes in CI's UTC — unrelated to this change).
Mutation checks — each mutation applied, the failing set observed, then reverted:
vault_read_note's descriptionidempotentHint).default(false)→.default(true)(prune_empty_folders)vault_delete_notevault_memory_recalldescription)buildServerMetadataRegen flow exercised end-to-end: an intentional description edit →
npm run snapshot:update(scoped to the one test file, ~0.5 s) → 17 one-line diffs showing exactly the edit → reverted, baseline byte-identical afterward. Lint (0 errors), knip, prettier:check, markdownlint, and the full build (server + cli + sst) are green.Not exercised: GitHub's collapsed rendering of the
linguist-generatedfiles — verifiable on this PR's Files view. Runtime response shapes are intentionally out of scope: the snapshot sees schemas and rendered text; response-shape stability stays enforced by the integration suite's exact assertions.🤖 Generated with Claude Code