Skip to content

Latest commit

 

History

History
308 lines (263 loc) · 24.4 KB

CONFORMANCE-RUN.md

File metadata and controls

308 lines (263 loc) · 24.4 KB

The conformance build and the oracle run

Issue 13. The core is accepted on the MCP project's own oracles: the server suite of modelcontextprotocol/conformance, run over loopback against the served core, and the 2026-07-28 schema.json, run over every message of the transcripts of that run. This file says what is built, how to reproduce the run, what the run says today, and what a planted defect looks like. The IDs are CF-01 to CF-07 in CONFORMANCE.md; a passing scenario here is evidence for what it checks, not a proof about the core.

What is built

Part Where
The declaration, a PrismPM ServerApplication whose core is UorMcp.ConformanceFixture.conformanceRun, the step serve over the fixture provider conformance/src/Application.lex.tex, generated from the vectors of the pinned scenarios: cargo run -p lifecycle-oracle --example conformance_application -- --write
The LexLean and PrismPM projects the declaration is built in conformance/lexlean.conformance.toml, conformance/lexlean.conformance.lock, conformance/prismpm.toml; conformance/ is the Prism project root, because PrismPM takes the consumer's Cargo workspace root to be the project root, and conformance/stage-shared.sh copies the sources it shares with the main project (src/, language/mcp.core) into conformance/shared/, which git ignores, because LexLean accepts no path that leaves its project; the main project (lexlean.toml) does not see any of it
The copies of PrismPM's Foundation.Bytes, Foundation.Server.V1.Model and the prism.arch lexicon conformance/src/Foundation, conformance/language/prism.arch; byte for byte the files at the revision of .github/prismpm.rev (authority A-PRISMPM-SERVER-APPLICATION)
The binding: the host effects tool.calls (the count of tools/call requests in the last 60 s), clock.now, task.id, task.table and request.state.key (the 32-byte key that seals and opens a requestState, held for the life of the process), which fill the request frame, and task.store, which the core requests to store the task table conformance/bindings/mcp-binding
The consumer workspace: the generated core and host packages beside the binding conformance/Cargo.toml; the three generated packages are not committed
The oracles conformance/oracle/: server-suite.mjs, recorder.mjs, verdict.mjs, schema-oracle.mjs, identity.mjs, fetch-suite.sh
What is not a pass, and which pull request it waits on; empty today conformance/waiting.json
The identity of the core conformance/core-identity.json

prismpm build (PrismPM at .github/prismpm.rev) certifies the core and generates three packages: mcp_conformance_core, mcp-conformance-host-api and mcp-conformance-host. conformance/lay-out.sh copies them into the consumer workspace, where cargo build --release builds mcp-conformance-host. mcp-conformance-host serve listens on 127.0.0.1 and prints listening 127.0.0.1:PORT.

The adapter enforces only what PrismPM's model states: loopback, the origins localhost, 127.0.0.1 and [::1], POST, the content type application/json, and the bounds. The Accept field is not enforced, so what the core answers is what the suite sees.

Reproduce

bash conformance/oracle/fetch-suite.sh "$SUITE"            # the pinned commit, then npm ci
( . .github/lexlean-ci/lib.sh && PRISMPM_ROOT="$PRISMPM" install_prismpm )   # PrismPM at .github/prismpm.rev, built with the LexLean of .github/lexlean.rev
bash conformance/stage-shared.sh                           # src and language/mcp.core into conformance/shared
(cd conformance && PATH="$PRISMPM/bin:$PATH" prismpm build)   # prints: build published: <id>
bash conformance/lay-out.sh conformance/.prism/build/<id> production-result.json
(cd conformance && cargo build --release --locked)
node conformance/oracle/server-suite.mjs --suite-root "$SUITE" --out out \
    "$PWD/conformance/target/release/mcp-conformance-host" serve
node conformance/oracle/schema-oracle.mjs --suite-root "$SUITE" \
    crates/lifecycle-oracle/suite/mcp-spec-80041ab0a139a8f4fb3adbc99f7ae9113ea59671/schema/2026-07-28/schema.json out/transcript.json
node conformance/oracle/identity.mjs --result production-result.json --lock conformance/lexlean.conformance.lock \
    --package conformance/mcp_conformance_core \
    --reference crates/lifecycle-oracle/generated/mcp_lifecycle/provenance.json --check conformance/core-identity.json

The conformance job of .github/workflows/lexlean.yml runs these steps, and MCP_CONFORMANCE_SUITE_ROOT="$SUITE" cargo test -p lifecycle-oracle --test conformance -- --ignored before the server suite.

The workspace gate does not run the node oracles. just vv runs cargo test --workspace in the locked SDK image, where the pinned suite is not installed (R6 forbids fetching it there). The tests behind CF-03 to CF-06 that run a node program, or read the installed suite, are therefore #[ignore]d with a reason that names the conformance job, and the pure-Rust checks of the same claims (the declaration, the vectors, the waiting list, the digests, the pins and the authorities) stay in the gate. The job runs the ignored tests with --ignored (run_oracle_tests) and is green only when as many passed as conformance.rs ignores; a run that selected none, or skipped some, fails. CF-08 refuses a test that needs node or the suite without the ignore, an ignore without that reason, an ignored test the job does not run, and a job that does not require the count.

The verdict

The suite's runner exits 1 for a failing scenario and for a harness error alike, and exits 0 when a scenario it was asked for produced nothing, so its exit status is not the verdict. server-suite.mjs reads checks.json of every scenario of the frozen requirements file (requirements/2026-07-28.yaml, SHA-256 ae2f4f6210fd729e2e318edd5bbfa31a43cee0bc608e48052fa26dbf1d939b57; the runner refuses any other file) and decides:

  • a scenario passes when it produced a check, no check failed, and at least one succeeded;
  • a scenario with no checks, or whose checks are all skipped or warnings, is a failure;
  • every failing check is reported as scenario: check id: message, the suite's own text;
  • the job passes (exit 0) only when the set of failing scenarios equals the set conformance/waiting.json lists as waiting, each entry naming its issue and pull request. A failing scenario that is not listed fails the job (FAILING and not named in waiting.json, printed first); a listed scenario that does not fail fails the job (named as waiting but not failing); an entry that names neither an issue nor a pull request fails the job. When there are failing scenarios, the last lines of the diagnostic group them by the pull request that is to deliver what they need.
  • a waiting scenario is still a failure: it is fail in the table and in report.json, which says valid: false, the step summary groups it under the pull request it waits on, and the scenarios passing are counted without it (37 of 37 required today). The list classifies; it never turns a failure into a pass, and it is shortened in the pull request that delivers a scenario.

One scenario cannot pass on any server at this pin: tasks-status-notifications sends no request and reports one SKIPPED check (src/scenarios/server/tasks/notifications.ts, pending a subscriptions/listen rewrite). It is named in conformance/waiting.json under suite_skipped, reported as skipped by the suite, not counted as passing, and held to the rule above as soon as the suite stops skipping it.

The schema oracle

schema-oracle.mjs validates every JSON-RPC message of transcript.json with the Ajv that the suite locks, against schema.json at the pinned specification commit. A request or notification is held to the definition its method names; a response to the definition of the request it answers; a typed error to the definition of its code. Two kinds of message are counted, not excused:

  • A probe the server refused. The suite sends requests that are invalid on purpose (no _meta, an unknown method, a mistyped field). The server's JSON-RPC error answer is validated like every other response; the probe itself is not asserted valid. A request the server answers with a result is never a refused probe and must validate.
  • A transport refusal. A 403, 406 or 415 whose body is not JSON-RPC is the HTTP refusal of a request before the core sees it (the DNS-rebinding probe). Any other body that is not JSON is a finding.

What passes today

Run of the core of this commit (core digest below), spec 2026-07-28, 434 JSON-RPC messages, 35 refused probes, 1 transport refusal, every message valid: the schema oracle held 194 of them to the tasks extension's schema, and counted one inputResponses entry, a key the suite made up for a task that was not waiting for it, as ignored (the extension tells a server to ignore it).

Required: 37 of 37 pass. Not scored: 12 of 13 pass, 1 is skipped by the suite, none fails. No scenario waits. The suite is at cf022cbb541f9d939068d7f84758b9728be7fc23 (MCP#44); the table is the notice annotation "the per-scenario table" of the conformance job of run 40534826655.

# kind scenario result checks waits on
1 required server-stateless pass 25 of 30 (5 skipped)
2 required completion-complete pass 2 of 2
3 required tools-list pass 4 of 4
4 required tools-call-simple-text pass 2 of 2
5 required tools-call-image pass 2 of 2
6 required tools-call-audio pass 2 of 2
7 required tools-call-embedded-resource pass 2 of 2
8 required tools-call-mixed-content pass 2 of 2
9 required tools-call-error pass 2 of 2
10 required tools-call-with-progress pass 2 of 2
11 required server-sse-multiple-streams pass 1 of 2 (1 info)
12 required resources-list pass 2 of 2
13 required resources-read-text pass 2 of 2
14 required resources-read-binary pass 2 of 2
15 required resources-templates-read pass 2 of 2
16 required sep-2164-resource-not-found pass 4 of 4
17 required prompts-list pass 2 of 2
18 required prompts-get-simple pass 2 of 2
19 required prompts-get-with-args pass 2 of 2
20 required prompts-get-embedded-resource pass 2 of 2
21 required prompts-get-with-image pass 2 of 2
22 required dns-rebinding-protection pass 2 of 2
23 required caching pass 8 of 8
24 required input-required-result-basic-elicitation pass 3 of 3
25 required input-required-result-basic-sampling pass 3 of 3
26 required input-required-result-basic-list-roots pass 3 of 3
27 required input-required-result-request-state pass 3 of 3
28 required input-required-result-multiple-input-requests pass 3 of 3
29 required input-required-result-multi-round pass 4 of 4
30 required input-required-result-missing-input-response pass 2 of 2
31 required input-required-result-non-tool-request pass 3 of 3
32 required input-required-result-result-type pass 2 of 2
33 required input-required-result-unsupported-methods pass 2 of 2
34 required input-required-result-tampered-state pass 2 of 2
35 required input-required-result-capability-check pass 2 of 2
36 required input-required-result-ignore-extra-params pass 2 of 2
37 required input-required-result-validate-input pass 3 of 3
38 not scored tasks-lifecycle pass 9 of 9
39 not scored tasks-capability-negotiation pass 5 of 5
40 not scored tasks-wire-fields pass 4 of 4
41 not scored tasks-request-state-removal pass 3 of 3
42 not scored tasks-mrtr-input pass 4 of 4
43 not scored tasks-request-headers pass 5 of 5
44 not scored tasks-dispatch-and-envelope pass 9 of 9
45 not scored tasks-status-notifications skipped by the suite 0 of 1 (1 skipped)
46 not scored tasks-required-task-error pass 3 of 3
47 not scored tasks-mrtr-composition pass 2 of 2
48 not scored json-schema-2020-12 pass 8 of 8
49 not scored http-header-validation pass 14 of 14
50 not scored http-custom-header-server-validation pass 10 of 10

Two passes need a caution. server-sse-multiple-streams passes with an info: the core answers JSON, and event streams are optional there. server-stateless passes with 5 of its 30 checks skipped by the suite: the checks that need a subscriptions/listen stream. The suite skips them for a server that advertises no listChanged or subscribe capability (stateless.ts, the comment above advertisesSubscriptions), and the specification schema makes those capabilities and the list-changed notifications optional (ServerCapabilities.tools.listChanged: "Whether this server supports notifications for changes to the tool list"; ToolListChangedNotification: "An optional notification"). The core advertises none and does not register subscriptions/listen, so the skips stand; they are listed in the report with their check ids.

The requestState key

Five input-required-result-* scenarios ask for a requestState: request-state, multiple-input-requests, multi-round, tampered-state and capability-check. The core writes a state only with the host's key (LIFECYCLE.md, "key"), and the declaration carried no key section, so the core refused them with an internal error (IR-16, as designed). The declaration now declares request.state.key as a host effect and as a frame section after the task table; its acceptance vectors carry a 32-byte key, the floor the core accepts, so the served frames are the modeled frames; and the binding holds one key for the life of the process, from the same operating-system source as the task identifiers. The five scenarios pass. The core did not change, so its program digest stands.

The pin of the suite (MCP#44)

The suite is pinned at cf022cbb541f9d939068d7f84758b9728be7fc23 (fix(validation): accept extension result envelopes (#468), committed 2026-09-06), the earliest commit that contains modelcontextprotocol/conformance#468. The pin before was 232a9014457eaf4c59559916e3d616d2f7f28f05. At that commit eight tasks-* scenarios failed one check each, the suite's own wire-schema-valid: it holds the result of every tools/call to the core schema's CallToolResult, which requires content, and a task-creating call is answered with the extension's CreateTaskResult (resultType: "task", Result & Task), which has none. #468 admits that resultType as an extension result. The core is not changed; the schema oracle, which holds the messages of the extension to the extension's schema, found every message valid before and after.

What the move changes, from the two checkouts:

  • The frozen requirements file is byte for byte the same (SHA-256 ae2f4f6210fd729e2e318edd5bbfa31a43cee0bc608e48052fa26dbf1d939b57): the same 37 required and 13 not-scored server scenarios, none added, removed or re-scored. The twelve tasks/*.ts files are the same too.
  • The pin is not #468 alone: 18 other commits come with it. Four of the 29 vendored files differ: tools.ts, input-required-result.ts, negative-mrtr.test.ts and everything-server.ts; the vectors cite the new lines, and the authority row records the new manifest checksum.
  • tools-list has a fourth check, tools-list-deterministic-order (the suite lists the tools twice more and compares the order; a warning when it differs, a success here), which is why the row reads 4 of 4. Vector TS-12 stands for the two requests and TL-01 states the order.
  • tools-name-format holds tool names to the specification's rule (1 to 128 characters of A-Za-z0-9_.-, no solidus) and reports a violation as a warning, not a failure. The core admits 1 to 64 characters of the same set, inside both; the vector follows the suite's new rule.
  • input-required-result-capability-check reports an input_required result that names no inputRequests as not testable, which fails the check. It failed before the key section and passes with it.
  • The runner bounds each scenario to 30 s (--timeout); the whole server run takes about 11 s.

Every scenario that passed at 232a9014 passes at cf022cbb, and so do the eight tasks-*: 37 of 37 required and 12 of 13 not scored pass; none fails.

Not met

  • No required scenario fails: 37 of 37 pass and the set of failing scenarios is empty. The one not-scored scenario the suite skips at this pin, tasks-status-notifications, sends no request and is not a pass (suite_skipped); its row stands as skipped, and the stream it needs is #11's.

server-stateless and the two diagnostic tools

The scenario calls test_streaming_elicitation (line 887) and test_logging_tool (line 947) and reports a call that is rejected, or a tool the server does not list, as not testable, which fails the check. The conformance fixture lists both (TL-19) and each answers one text result, as in the suite's reference server. Neither is an input-required tool: the first probe fails only on a frame with a method other than notifications/*, the second only on a notifications/message for a request that set no logLevel. The core writes progress only for a request that carries a progressToken, declares no logging capability and writes no log message, so both probes pass. The core digest changed with the fixture catalog.

The identity of the core

conformance/core-identity.json records the core by the SHA-256 of its compiled program, e95e5363acbb9e398639953dd1cbf172625d9e7e830d78f15bbe1153ab561bb4, with the package, its file digests, the source and semantic IDs and the lock it was exported from, and the compiler that exported it. That program digest is the one the committed export mcp_lifecycle of the same root records: the compiled program does not depend on the compiler that exported it, while the generated src/lib.rs does. CI rebuilds the core and the job fails unless the record is reproduced; lexlean-mcp checks the program digest.

Planted defects

Each was planted alone against the green state (every scenario passing but the one the suite skips, exit 0), the build and the run repeated. A reverted tree is the clean run above. "Green" below means the exit status of server-suite.mjs, which is the status of step 9a.

Defect Planted in Run Diagnostic
test_simple_text answers an empty content array instead of its text block src/Fixture.lex.tex, the callTool of the fixture provider; production build and host rebuilt, core digest 8f1335055a9149073644484f22f1aa59f6feb1190b647ebe34e0d21b17a1b01c exit 1; 36 of 37 required pass; check_core_identity fails as well: the program differs from the committed export mcp_lifecycle FAILING and not named in waiting.json (1): tools-call-simple-text, then tools-call-simple-text: tools-call-simple-text: content array is empty; No text content found
a failing scenario is not listed conformance/waiting.json as committed (empty), same build exit 1 FAILING and not named in waiting.json (1): tasks-lifecycle, then the suite's check messages, and the failing scenarios are not the ones .../waiting.json names
a listed scenario passes (stale) conformance/waiting.json with caching added, same build exit 1 named as waiting but not failing (1): caching
an entry names neither an issue nor a pull request conformance/waiting.json, tasks-lifecycle with only a reason exit 1 named as waiting without an issue or a pull request (1): tasks-lifecycle
the split is broken by dropping #[ignore] from the_schema_oracle_node_tests_pass_cf_04 crates/lifecycle-oracle/tests/conformance.rs; cargo test -p lifecycle-oracle --test conformance with MCP_CONFORMANCE_SUITE_ROOT unset, as in the SDK image exit 101; the_schema_oracle_node_tests_pass_cf_04 fails and so does CF-08 MCP_CONFORMANCE_SUITE_ROOT names the installed conformance suite (conformance/oracle/fetch-suite.sh DIR installs it): NotPresent (the failure that turned bootstrap red), and CF-08: the_schema_oracle_node_tests_pass_cf_04 needs node or the installed suite and is not ignored: the workspace gate runs without them
the conformance job selects fewer ignored tests than the source ignores run_oracle_tests with the filter cf_03 after -- --ignored exit 1 1 passed of the ignored tests, expected 4

The first defect passes lexlean check and the production build of the conformance project, whose entrypoints are the declaration and the fixture root, and so is found by the suite and by the core identity record, not by the kernel.

The pin move (MCP#44) was checked with a stale pin planted in one place at a time, in the tree that passes (cargo test -p lifecycle-oracle, fetch-suite.sh and server-suite.mjs), and each reverted with git checkout:

Defect Planted in Run Diagnostic
the requirements digest is stale: its last digit is 8 for 7 FROZEN_REQUIREMENTS_SHA256 in conformance/oracle/server-suite.mjs cargo test -p lifecycle-oracle --test conformance the_pins_of_the_oracle_run_agree_with_their_authority_records_cf_07: exit 101; server-suite.mjs --suite-root <the suite at cf022cbb> --out <dir> true: exit 2 CF-07: assertion failed: runner.contains(&format!("const FROZEN_REQUIREMENTS_SHA256 = '{REQUIREMENTS_SHA256}';")); the oracle: the suite's 2026-07-28 requirements differ from the frozen file: expected ...939b58, observed ...939b57, before the subject starts
the suite commit is stale: COMMIT is 232a9014 conformance/oracle/fetch-suite.sh the same CF-07 test: exit 101; fetch-suite.sh DIR: exit 1 CF-07: assertion failed: fetch.contains(&format!("COMMIT={PIN}\n")); the script: cd: .../crates/lifecycle-oracle/suite/conformance-232a9014...: No such file or directory
a vendored suite file is not the suite's: a newline appended crates/lifecycle-oracle/suite/conformance-cf022cbb.../src/scenarios/server/tools.ts cargo test -p lifecycle-oracle --test provenance the_vendored_suite_is_the_pinned_commit_and_matches_its_recorded_checksum_lc_12: exit 101; fetch-suite.sh DIR: exit 1 LC-12: the vendored files are not the files the authority records (left b0a6b091..., right 11ec4b0e...); the script: fetch-suite: vendored ./src/scenarios/server/tools.ts differs from the suite at cf022cbb541f9d939068d7f84758b9728be7fc23

The PrismPM pin, and what slice 3 still needs

.github/prismpm.rev names PrismPM main at 585ee8eb3123e624a189edbcba1001b6b7c57855 (version 0.4.0). Nothing in PrismPM changes. PrismPM vendors LexLean e0b74b81e (the v0.4.0 tag), which has no nested_induction, so for development install_prismpm builds prismpm from a copy of the PrismPM tree whose vendor/lexlean is the package cargo package makes of the LexLean tree at .github/lexlean.rev (5dccf9e5b, release/v0.4 head, with SM-37). One pin drives both tools. Cargo.lock is PrismPM's own (--locked). The compiler that certifies the core is then the compiler both locks were made with, and CI checks that: lexlean.lock and lexlean.conformance.lock record the same compiler_semantics, and so does the production result of prismpm build. The generated src/lib.rs of the core is byte for byte the src/lib.rs of the committed export mcp_lifecycle.

Release: MCP v0.1.0 pins a PrismPM SDK release, and that release must pin a LexLean release that contains SM-37 (tracked in MCP#16).

For PrismPM slice 3:

  • the vendored LexLean has to read MCP's sources, which needs SM-37 (nested_induction) or a release that contains it;
  • the build controller reads the LexLean build root as .lexlean, so conformance/lexlean.conformance.toml sets build_root = ".lexlean";
  • prismpm verify of a server application is still PP2014;
  • the release profile of prismpm (thin LTO, one codegen unit) is killed for memory on a 7 GB runner, so CI installs the debug profile;
  • the response frame must carry no notifications; the core writes the event-stream bytes into the response body, which satisfies it.