Repository navigation
fix(mcp-host): repair lone UTF-16 surrogates before LLM dispatch; report a provider 400 as a rejected request - #1434
Conversation
A lone UTF-16 surrogate in any request text made every provider reject the
whole request with HTTP 400: JSON.stringify emits it as an escape such as
\ud83d, which Anthropic reports as "unexpected end of hex escape" and the
Codex and Grok upstreams refuse. In production a third-party MCP tool
returned such text, and the agent failed with every provider while the user
saw "Connection Error" and assumed an exhausted subscription.
makeProvider is the only place that constructs a provider, so it now wraps
each one with withWellFormedRequestText. The request methods the instance
has are overridden in place, so instanceof, optional-method probes and
`this` are unchanged. Every string in messages (content, content parts,
tool-call ids and arguments, keys included, image source ids), tool
definitions and system prompt parts goes through toWellFormed(), which
replaces a lone surrogate with U+FFFD. Image bytes and MIME types are
skipped. Well-formed input passes through as the same references, so its
requests and the subscription request hashes are unchanged. The repair
runs before the subscription providers build and hash the request, so
control-api and the proxies re-derive the same hash.
The repair is visible: one warn line per repaired message (provider, role,
tool name, tool call id, unit count; never the text) and the counter
clerum_llm_request_text_repaired_total{provider,role}. A stored user
message holding a lone unit is repaired, logged and counted on every
replay; the store is not rewritten.
Tests go through the real makeProvider/createLLMProvider: the OpenAI and
Anthropic SDKs against a local HTTP server, and the Codex and Grok
providers with stubbed authorizer and proxy deps. They also cover the
golden subscription hashes, WorkflowService with its default factory,
FailoverSingleTurnProvider, LlmBridge with createLLMProvider, and a
two-turn conversation. With the wrap removed from makeProvider, 11 of the
14 tests fail; the remaining 3 assert that well-formed input is unchanged
and have their own mutations.
Refs #1433
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both subscription proxies mapped an upstream HTTP 400 to invalid_request, the same code they use when their own transport contract refuses a body. mcp-host has no arm for invalid_request, so it fell through to LLM_API_CALL_FAILED, which Desktop shows as "Connection Error". A provider rejecting the request read as a connectivity or quota problem (#1433). The proxies now report an upstream 400 like every other non-transient upstream 4xx: upstream_rejected with upstreamStatus 400 (HTTP 422, or the same code on an SSE error frame). invalid_request stays the code for the proxy's own refusals. The context-window checks run before this mapping and are unchanged. The proxy failure metric moves those events from the invalid_request label to upstream_rejected. mcp-host already classifies upstream_rejected as LLM_UPSTREAM_REJECTED ("Provider Rejected Request"), non-retryable and without failover. For upstream status 400 its message is now a fixed sentence that holds for every 400: "The provider rejected this request (HTTP 400). This is not a connection problem, and retrying the same request will not help." Other upstream 4xx, including the 402/403 entitlement refusals, keep their text. The proxy conformance and server tests expect upstream_rejected for an upstream 400 and add 400 to the G1-3a cases. Contract-refusal tests still expect invalid_request. With the old 400 arm restored, 5 tests fail in each proxy; with the message arm disabled, the two client tests fail. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0 as a rejected request The shared HTTP classifier mapped a provider's HTTP 400 to LLM_API_CALL_FAILED, which Desktop renders as "Connection Error". In the #1433 incident the Anthropic API-key path answered "The request body is not valid JSON: unexpected end of hex escape", and the user read it as a connectivity or quota problem. A 400 that no body code maps elsewhere (insufficient_quota, model_not_found and invalid_api_key keep their codes) is now LLM_UPSTREAM_REJECTED: still non-retryable, still without failover, now labelled "Provider Rejected Request". The message stays the provider's own text. This covers the OpenAI, Azure and OpenAI-compatible providers and Claude. Every 400 on those paths moves, including OpenAI context-window refusals (there is no context_length_exceeded body mapping in this classifier) and parameter errors. Vertex keeps its own 400 arms, and so does Bedrock's modeled ValidationException; only an unmodeled Bedrock HTTP 400 reaches this classifier. The Anthropic credit-balance quirk (insufficient credit reported as HTTP 400) keyed on the ApiCallFailed code. It now keys on the HTTP status, so a credit-balance 400 is still InsufficientQuota. Tests: the shared classifier, the plain and enveloped Anthropic 400s, the "not valid JSON" envelope, and the enveloped credit-balance case. With ApiCallFailed restored for 400, 4 tests fail; with the credit guard keyed on ApiCallFailed again, 2 fail. Open PR #1420 adds two tests that pin an OpenAI 400 to ApiCallFailed (responsesLoopErrors.test.ts T-E3, openaiFamily.contract.test.ts "too many tools 400"). Whichever PR merges second updates those expectations to UpstreamRejected. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ding keys Round 1 of the #1434 review found three defects in the lone-surrogate repair wrap. The walker recursed once per nesting level. A tool schema or tool-call arguments nested deeply enough overflowed the stack (20000 levels in the round-2 comparison table; the threshold depends on the process stack size), and the wrap threw a synchronous RangeError. That replaced the providers' own typed refusal (invalid_request, "nesting depth") with an error classified as retryable, which also triggered failover. The walker now uses an explicit stack, reads each property once, and keeps a container that is already on the current path (a cycle) as is instead of entering it. The overrides are async, so any failure reaches the caller as a rejected promise. The overrides were enumerable own properties, so Object.keys of a wrapped provider differed from a bare instance; open PR #1419 compares them. They are now non-enumerable, like class methods. Repairing two keys that differ only in their lone surrogates made them equal, and one value was silently dropped. Such a request is now refused before dispatch with RequestTextKeyCollisionError. The wrap also overrides classifyError, which reports it as the new LLM_REQUEST_TEXT_REJECTED: not retryable, not dispatched, no failover class. Desktop labels it "Invalid Request Text". Repairs are logged and counted only once the whole request has been repaired, so a refused request reports none. Tests (wellFormedRequestText.test.ts): a 150000-deep schema and arguments on Codex and Grok keep the provider's refusal without authorizing; a 150000-deep schema passes through by reference or is repaired at its bottom; cycles; collisions in arguments and schemas, a repaired key equal to an untouched one, distinct repaired keys; the refusal through LlmPortAdapter, FailoverSingleTurnProvider and LlmBridge; both Anthropic cache paths with tools and context; exact warn fields and unit counts; descriptors on all eight arms. The tests in this file that drive a real SDK run behind a guard that refuses any non-loopback connection. Texts: the failover classifier comment names a non-retryable ApiCallFailed without calling it a 400 and lists RequestTextRejected; the makeProvider docblock and the monitoring doc state that a refused request is not counted as a repair. Mutations, each run and reverted: the recursive walker fails T10-T12; no cycle check aborts the worker on heap exhaustion; sync overrides fail T13-T17; no collision count fails T13-T19; no classifyError override fails T13, T19a-c and T6/T14; enumerable overrides fail T6/T14; per-repair reporting fails T15b; the guard removed fails its witness tests. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests After this PR's API-key change, an HTTP 400 read "Provider Rejected Request" on the OpenAI-family, Anthropic and subscription paths but still "Connection Error" on Vertex and on Bedrock's modeled ValidationException. Vertex: a numeric 400, INVALID_ARGUMENT and FAILED_PRECONDITION are now LLM_UPSTREAM_REJECTED. Bedrock: ValidationException is now LLM_UPSTREAM_REJECTED; an unmodeled Bedrock 400 already reached the shared classifier. Quota, retired-model and auth codes keep their mappings. Texts made false by this PR are corrected: the UpstreamRejected doc comment now names the Vertex and Bedrock arms, and two test comments that called a non-retryable ApiCallFailed a 400 now describe a replayed subscription ticket. Tests: the driver classification rows for numeric and code-field 400s, FAILED_PRECONDITION, ValidationException and an unmodeled Bedrock 400. With ApiCallFailed restored on Vertex 2 tests fail, on Bedrock 1. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…provider arm Every provider makeProvider builds now has a test on its real transport: openai, azure, the OpenAI-compatible data arm (zai), claude, the Codex and Grok subscriptions, vertex and bedrock. Each arm runs its real SDK or proxy client; only the last hop is local. openai, claude and vertex use a loopback HTTP server through their base-URL variables; bedrock a loopback HTTP/2 server through AWS_ENDPOINT_URL_BEDROCK_RUNTIME, with the user's AWS files and instance metadata disabled; azure and zai the SDK client's fetch; the subscriptions the proxy clients' fetchFn. Vertex stubs only the token hop of the google-auth-library copy @google/genai loads. A guard refuses every non-loopback connection, with a witness test. wellFormedRequestText.providers.test.ts: one request per arm with a lone surrogate in a tool result and a tool description. The wire body is well-formed, both warns carry the exact fields, both counters rise, and on the subscriptions the authorized hash equals the server re-derivation and the proxy body. providerRejectedRequest.providers.test.ts: a provider 400 on each arm is LLM_UPSTREAM_REJECTED, not retryable, with no failover class and the provider's message kept; the Codex and Grok proxies' upstream_rejected carries the fixed message. Z.ai code 1211, the Anthropic credit-balance 400 and Bedrock ServiceQuotaExceededException keep their own codes. Mutations: the wrap removed from makeProvider fails all eight repair arms; the shared 400 arm back to ApiCallFailed fails openai, azure, zai and claude; the Vertex and Bedrock arms back to ApiCallFailed fail their rows. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e new error literal Round 2 of the #1434 review found changed behaviour of the text repair that no test would catch. No production change in this commit. - T12b: a sub-object referenced twice (as the internal tool schemas do) is repaired at both references and counted once per path. - T9: the completeSingleTurn override reports its repair (warn and counter), on the first turn and on the replay. - T15c: on all four request methods, colliding keys are refused as a rejected promise and no repair is reported, with the collision in the last part each method repairs. - T5b: an own "__proto__" key and a null-prototype object survive the repair of a sibling. - T6: each makeProvider arm is still an instance of its class. - errors.test: the literal value LLM_REQUEST_TEXT_REJECTED, which Desktop's labels key on. Each test fails under its mutation, run and reverted on this tree: onPath never cleared (T12b), no report in completeSingleTurn (T9, T15c), a synchronous override or a report before the last repair step (T15c), plain assignment or no null prototype (T5b), a detached prototype (T6), a renamed code literal (errors.test). Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Anthropic SDK keeps the response envelope in `error` and builds `message` as
`400 {json}`. The shared classifier returned that string, so the user read the raw JSON
envelope instead of the provider's text, which is the case #1433 started from.
The 400 arm of ClaudeProvider.classifyError now uses the inner message, as the
credit-balance arm already does. Code, retryable, status, provider code and failover class
are unchanged, and no other status changes.
Tests: the `claude 400` row of providerRejectedRequest.providers.test.ts and the "request
body is not valid JSON" test now assert the exact message; both fail with the arm removed.
Refs #1433
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ification Comments, test headers and one docs row only; no statement changes. Corrected claims: - When repairs are reported: after the whole request is repaired and before the provider method is called. A request refused for colliding keys reports none, but a repaired request the provider then refuses (over-deep or cyclic input) has already been reported. - Cyclic and shared input: a cycle's back-edge is kept unrepaired and every arm refuses or fails that request before dispatch; a sub-object reachable along several paths is walked once per path. Code-built tool schemas do reuse sub-objects; no producer of cyclic or mutually referencing input is known. "Each property is read once" is removed. - Vertex: a numeric 400 can also come from Google's token endpoint (`invalid_grant`) with no request reaching Vertex (#1437). - The members of the non-retryable ApiCallFailed failover row. - What the transport fixtures stub besides the endpoint (the subscription authorizer, attempt context and platform JWT; the Vertex token hop). - What the repair warn names (the role, and the tool for tool results and definitions). Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…epair Conflicts: mcp-host/package.json and package-lock.json (version only); dev's files kept, the pre-commit hook bumps the version above dev's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n on dev dev now has #1419 and #1420. Twenty tests failed on the merged tree; no production change is needed, only expectations and fixtures. - The `openai` profile sends both operations through the Responses API. The OpenAI fixture replies of wellFormedRequestText.test.ts and of the `openai` transport arm now use the Responses shape; Azure and the compatible profiles keep chat completions. T18 reads the assistant's `function_call` item from the Responses `input`. - #1420's T-E3 (responsesLoopErrors) and "too many tools 400" (openaiFamily.contract) expect LLM_UPSTREAM_REJECTED for a provider HTTP 400. The chat corpus is regenerated locally with UPDATE_CHAT_CORPUS=1: only `code` changes, on the 16 classification rows of an HTTP 400. - #1419's alias guard compares Reflect.ownKeys with the registry-built provider, which the factory wraps with withWellFormedRequestText (three own non-enumerable members). The alias is now compared after the same wrap, so any other hidden member is still caught. - claude.classifyError: an enveloped 401 keeps the SDK's message, pinning that the inner message is used for the 400 arm only (review round 3, R3-L2). Mutations, run and reverted: the FR-1 arm widened to every status fails the new 401 test; a hidden member in OpenAICompatibleProvider fails the alias guard; chat-shaped OpenAI replies fail 15 tests in wellFormedRequestText.test.ts and the `openai` transport arm. Full mcp-host suite on the merged tree: 601 files, 8703 passed, 1 skipped, 1 todo. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evenfire Reviewer — Round 1Reviewed head: Decision: No objection The repair runs at the single place every provider is built, and a provider HTTP 400 now reads as a rejected request on every arm instead of "Connection Error". I traced the request path from Must fix
Verified
Validation gaps
PR descriptionAccurate. It names the head, records the test counts, and states the changes that are easy to misread: the proxy metric label moving from Pre-existingPre-existing: none identified during the scoped review. Next actionNone required before merge. The follow-ups already filed cover the rest: a Vertex Full review archive: — Review performed by Evenfire Reviewer |
jozer-rami
left a comment
There was a problem hiding this comment.
Approving at 41c32d19b. I found no blocker and no major. The six items below are minor, so they can be fixed here or tracked as follow-ups. T2 on this head is still pending, as the description says.
Run at head (Node 24.22.1): the mcp-host suite passed 8703 of 8705 (1 skipped, 1 todo); tsc --noEmit exited 0; codex-llm-proxy passed 413/413 and grok-llm-proxy 410/410. 25 of the 27 mutations I applied made tests fail.
Test gaps (both reproduced with a mutation and a probe test)
- A lone surrogate that is a direct array element has no test (
wellFormedRequestText.ts:204). I made the array arm count the string but not write the repaired value, and all the repair tests still pass. Fields that hit this arm include a schemaenumorrequiredlist and string-array tool arguments. If it regressed, the provider would get the #1433 400 again, with no warn and no counter. Test idea:enum: ['x\uD83D']and arguments{ tags: ['a\uD83D'] }, asserting on the repaired text sent to the provider and on the warn and counter. - Key-collision order (
wellFormedRequestText.ts:120). The code at head is correct. A mutation that checks only the repaired keys still passes every test. With that mutation,{ 'k\uD83D': …, 'k�': … }(the repaired key first) silently drops one value. Test idea: the existing collision case with the key order reversed.
Minor
3. On the Responses path, the repair changes the continuation digest. The digest is computed over the parsed arguments at decode (openaiFamily/protocols/responses.ts:484). Once the wrap repairs those arguments, the check at :266 reports digest_mismatch, and reasoning replay is dropped with only a debug line. Before this PR that request already went out well-formed, because the arguments are sent through JSON.stringify (:186/:237). Possible fix: make the digest invariant to the repair, or repair at decode.
4. When a key collision is refused, no log line or metric tells an operator which tool caused it (wellFormedRequestText.ts:326, :352). The adapter logs only the code and the error name, and the logger reduces an Error to name/code/status (logger.ts:57-67), so the tool name never reaches mcp-host logs. Possible fix: emit one warn and increment a refused counter, with the same bounded fields as a repair.
5. The role label and log field take values from hook-rewritten messages. remoteLlmHook.ts:189 does not validate role, so each new role becomes a new Prometheus series. The same flow already bounds hook-sourced labels with boundedReason (hookedLlmPort.ts:64-72). Possible fix: map role to a closed set, with other for anything else, and cap toolName and toolCallId in the warn.
6. The FR-1 arm (claude.ts:135-139) keeps only Anthropic's inner message. That drops the rest of the body, including request_id when the body carries it, and err.request_id is not logged in providerErrorDiagnostics either. Possible fix: add the request id to the diagnostics.
… the 401 test text Review round 4 on 6cc62bd..41c32d1, two Low findings, test and text only. - L-1: the #1419 alias guard compared Reflect.ownKeys only after withWellFormedRequestText, which defines classifyError and both request methods as own members. An own member of OpenAICompatibleProvider with one of those names was masked. The guard now also requires the unwrapped alias to carry exactly the factory's own keys minus the wrap's three members, asserts the factory carries those three, and keeps the wrapped comparison. - L-3: the 401 test said the inner Anthropic message is used for the 400 arm only. The body-type arms (not_found, overloaded, billing, permission) already return it at any status. The title and comment now name the HTTP-status 400 arm and the shared-classifier path the 401 takes. No assertion changed. Mutations, run and reverted: an own non-enumerable classifyError, completeSingleTurnWithTools or extraHidden on the alias each fail the guard (the first two passed before this change); the FR-1 arm widened to every status fails the renamed 401 test. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t wall time sweepRetention.realWorker.test.ts (e38c9e4, #1111) compared the db worker's 1000 ms progress bound and a 900 ms sync timeout with wall time across 1,261 batch operations. On a loaded machine one acknowledgement can take longer than that without the batch doing more work (waiting for a core, the disk or the checkpoint, or the main thread reading the reply late), so the test failed with "db worker made no progress for 1000ms" on dev e611ebf and on unrelated branches, and failed T0. A 250-row batch costs at most about 38 ms of worker CPU (77 ms when CPU-starved); the pre-fix single DELETE costs about 4.7 s. The test keeps its fixture, the real worker thread and the 1000 ms bound, and compares the bound with the CPU time the worker thread spends on each operation (worker.cpuUsage(), sampled around each enqueueSync): - no operation takes 10% or more of the sweep's worker CPU (owner decision: a future batch that large fails this test even under 1000 ms); - the sweep needs more than 1000 ms of worker CPU, and no operation does; - every operation was sampled, and there was more than one. The queue's wall-clock limits become 300 s and the hook and test limits 600 s; they only stop a wedged worker from hanging the run. No production code changes. Red reasons now: pre-fix 3158019 (one operation) "expected 1 to be less than 0.1"; dispatcher ignoring the limit and an unbounded batch size, share 0.951 and 0.950; caller loop dropped, "expected +0 to be 15000". Green on this branch (11 s at load average 6); the plan's run with 16 busy processes on the test's cores was green, where the original failed. Refs #1433 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogate-repair * origin/dev: (27 commits) test(desktop-app): pin the order of a removal and a replace on one chat test(desktop-app): name catalog isolation tests by behaviour docs(desktop-app): state what a failed write of a kept turn leaves test(desktop-app): name rejected-send tests by behaviour test(desktop-app): fix rejected-send test titles and a witness docs(desktop-app): describe what each send failure reason and flag records docs(desktop-app): state what the removal rule and a withdrawal cover fix(desktop-app): release a send's activity entry on every exit fix(desktop-app): keep a held send that was posted and failed without a rejection refactor(desktop-app): share the durable write of a kept turn fix(desktop-app): show the failure sentence for a send kept behind Recover input test(desktop-app): cover the tombstone and sender checks of chat:removeMessages test(desktop-app): cover the unprotected guards of the rejected-send path test(desktop-app): cover the attachment guards of a rejected send refactor(desktop-app): simplify the rejected-send path docs(desktop-app): describe connector rows for an agent the catalog does not list test(desktop-app): state why the second catalog differs fix(desktop-app): return a rejected send to the composer feat(desktop-app): remove unanswered local turns from the chat store test(desktop-app): cover a held agent restored from a workspace snapshot ...
…ew-r2 Brings in #1434 (lone UTF-16 surrogate repair). The only conflict is the mcp-host version line: dev 0.1.635, branch 0.1.639, merged as 0.1.640. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om dev dev (#1434) now wraps every provider from makeProvider with withWellFormedRequestText and adds ES2024.String to the tsconfig lib. Three comments of the wire-name projection became inaccurate: - the annotation helper said toWellFormed is unavailable under ES2022; - the UTF-16LE hash note implied injectivity over lone surrogates on the production path, where names arrive already repaired to U+FFFD; - the Claude counter claimed to count exactly the sent names, which no longer holds for a name with lone surrogates (the counter is unwrapped). Comment-only; no behaviour change. Clean merge, no textual conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
Closes #1433.
makeProvidernow sends only well-formed text. A lone UTF-16 surrogate in the request is replaced with U+FFFD. Each repair emits awarnlog line and increments the counterclerum_llm_request_text_repaired_total{provider,role}.upstream_rejected(withupstreamStatus: 400), where it wasinvalid_requestbefore. mcp-host shows it as "Provider Rejected Request" instead of "Connection Error", with the text: "The provider rejected this request (HTTP 400). This is not a connection problem, and retrying the same request will not help."LLM_UPSTREAM_REJECTEDinstead ofLLM_API_CALL_FAILED. For an HTTP 400 the message is the provider's own error text for the OpenAI family, Anthropic and Bedrock. For Vertex it is the SDK's error string, which holds Vertex's JSON error body (mcp-host: give provider 400 subclasses their own error codes #1439).k\ud83dandk\udc00), the request is not sent. The Host reports the new codeLLM_REQUEST_TEXT_REJECTED, which Desktop labels "Invalid Request Text". It is not retryable and never fails over.Why
A third-party MCP tool returned text that contained half of an emoji (a lone high surrogate). mcp-host passed it to the model unchanged.
JSON.stringifyemitted the escape\ud83d, and every provider rejected the whole request with HTTP 400:The request body is not valid JSON: unexpected end of hex escape.The user saw "Connection Error · " on every provider they tried and concluded their subscriptions were exhausted. It was neither quota nor connectivity: every call that included that tool output failed, and every other call succeeded.
How
Commit 1 —
fix(mcp-host): repair lone UTF-16 surrogates before LLM dispatchmakeProvider(mcp-host/src/llm/registry.ts) is the only place that constructs a provider. It now returnswithWellFormedRequestText(constructProvider(...)). Only the signature line of the existing body changed, which keeps refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419's edits to the return arms mergeable.mcp-host/src/llm/wellFormedRequestText.ts).instanceof, the adapter's capability probes andthisbehave as before. (Commit 4 makes the overrides non-enumerable andasync.)reasoning_content, imagesourceids), tool definitions, and system prompt parts. Image bytes and MIME types are skipped.makeProviderand still match.warnper repaired message, tool definition or system prompt: provider, role, the tool name and tool call id where they apply, and the number of repaired units, never the text.mcp-host/tsconfig.jsonadds theES2024.Stringlib forisWellFormed/toWellFormed. The runtime is Node 24.22.1.docs/operations/monitoring.mdlists the new counter.Commit 2 —
fix(llm-proxy): report an upstream 400 as upstream_rejectedinvalid_requeststays the code for the proxy's own contract refusals.upstream_rejectedcases keep their current text, including the 402/403 entitlement refusals, where "not a quota problem" would be false.Commit 3 —
fix(mcp-host): classify an OpenAI-family or Anthropic API-key HTTP 400 as a rejected requesterrorClassification.ts: an HTTP 400 that no body code maps elsewhere becomesUpstreamRejected. It stays non-retryable and without failover.InsufficientQuota.Review round 1 fixes (commits 4 to 6)
The first review of
abebed35dfound two Medium and five Low findings. All are fixed.fix(mcp-host): repair request text without recursion and refuse colliding keys.RangeError, classified as retryable, so it also triggered failover and hid the subscription providers' owninvalid_request("nesting depth") refusal. The walker now uses an explicit stack and keeps a cycle's back-edge as it is, unrepaired, without entering it.async, so any failure reaches the caller as a rejected promise, and non-enumerable, soObject.keys(provider)equals a bare instance's (refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419 compares them).RequestTextKeyCollisionError). The wrap'sclassifyErrorreportsLLM_REQUEST_TEXT_REJECTEDwithproviderDispatched: falseand delegates every other error. Repairs are logged and counted only after the whole request is repaired, so a request refused for colliding keys reports none. A request that the provider itself then refuses or fails before dispatch (over-deep or cyclic input, below) has already logged and counted its repairs.makeProviderdocblock and the monitoring doc are updated.fix(mcp-host): classify Vertex and Bedrock HTTP 400s as rejected requests. Vertex numeric 400,INVALID_ARGUMENTandFAILED_PRECONDITION, and BedrockValidationException, move fromLLM_API_CALL_FAILEDtoLLM_UPSTREAM_REJECTED. Quota, retired-model and auth codes keep their mappings.test(mcp-host): cover the repair and the 400 classification on every provider arm. Two test files run all eightmakeProviderarms (openai, azure, zai, claude, the Codex and Grok subscriptions, vertex, bedrock) on their real SDKs or proxy clients. Each arm talks to a local endpoint: a loopback HTTP/1.1 or HTTP/2 server, the SDK client'sfetch, or the proxy client'sfetchFn. The subscription arms also stub the control-api authorizer (it echoes the request hash), the attempt context and the platform JWT; their proxy client and the server-side re-derivation of the hash are real. Vertex stubs the token hop. A guard refuses every non-loopback connection.Review round 2 fixes (commits 7 to 9)
The second review, of
96495523d(three blind lanes and a final review), found no High or Medium finding. Fixed here:test(mcp-host): cover shared sub-objects, every request method and the new error literal. Tests for a sub-object referenced twice (T12b), the repair report ofcompleteSingleTurn(T9), the rejected promise and the deferred report on all four request methods (T15c), an own__proto__key and a null-prototype object (T5b),instanceof(T6), and the literalLLM_REQUEST_TEXT_REJECTED.fix(mcp-host): report Anthropic's own message for a rejected HTTP 400. An Anthropic API-key HTTP 400 now carries Anthropic's own message. It was the SDK's400 {"type":"error",…}string.docs(mcp-host): correct comments on the text repair and the 400 classification. Comment and doc corrections; no behaviour change.Behaviour changes to note
LLM_API_CALL_FAILED("Connection Error") toLLM_UPSTREAM_REJECTED("Provider Rejected Request"). That includes OpenAI context-window refusals, which have no body-code mapping in this classifier, and parameter errors (mcp-host: give provider 400 subclasses their own error codes #1439). An Anthropic 400 shows Anthropic's own message, where it showed the SDK's400 {json}string (round 2).INVALID_ARGUMENTandFAILED_PRECONDITION, and Bedrock'sValidationException, move too (commit 5). An unmodeled Bedrock HTTP 400 already reached the shared classifier.LLM_REQUEST_TEXT_REJECTEDinstead of being sent with one value dropped. Scope: MCP tool definitions arrive live fromtools/list, and the SDK's schema does not reject lone surrogates in property names, so a third-party server whose schema has two such keys in one object (for example names truncated mid-emoji) makes every request that carries that tool be refused until the tool is fixed or removed. A collision needs two keys of one object that differ only by different lone units; none has been observed.invalid_requestbefore authorizing; the API-key SDKs serialize it and, aboveJSON.stringify's own limit, fail with a retryableRangeError, as they do ondev(pre-existing, out of scope). Every arm refuses or fails a cyclic request in the provider before dispatch, after its repairs were logged and counted, and the back-edge keeps the unrepaired original. A sub-object reachable along several paths is repaired, and counted, once per path; for objects that all reference each other the number of paths grows factorially (9 objects: about 40 to 50 ms of synchronous work, about 9 times more per extra object;abebed35dfailed in about 3 ms withRangeError). Code-built tool schemas reuse sub-objects (the internal tools' schemas: at most 1.4 visits per unique container); no producer of cycles or of mutually referencing objects was found, so this is documented and not guarded.invalid_requestlabel toupstream_rejected, and the proxy reply becomes HTTP 422 instead of 400.core/tokenizer/anthropicTokenCounter.tssends unrepaired text to AnthropiccountTokenswhenCLERUM_TOKENIZER_OFFLINE=false(the default istrue). A 400 there already falls back to the heuristic count. Not changed here.Merge of
devwith #1419 and #1420dev(e611ebfcb) now has #1419 and #1420. Merged into this branch as3fce597c5; the only conflicts were themcp-hostversion fields (dev's kept; the next commit's hook raises the version to0.1.633, above dev's0.1.632, and dev's service-version gate passes on this head). No test line of either parent is lost in the merge (check a.3).On the merged tree 20 tests failed, all expectations or fixtures;
41c32d19badapts them with no production change:LLM_UPSTREAM_REJECTEDfor a provider HTTP 400. The chat corpus is regenerated locally (UPDATE_CHAT_CORPUS=1): onlycodechanges, on the 16 classification rows of an HTTP 400.openaiprofile sends both operations through the Responses API, so its fixture replies use the Responses shape (Azure and the compatible profiles keep chat completions), and T18 reads thefunction_callitem from the Responsesinput.Reflect.ownKeyssince7733fe136) compares the alias with the registry-built provider, which the factory wraps withwithWellFormedRequestText(three own non-enumerable members:classifyError,completeSingleTurn,completeSingleTurnWithTools). Since0db9d6f61the unwrapped alias must carry exactly the factory's own keys minus those three, and the wrapped alias must match the factory exactly, so a hidden member is caught under any name, the wrap's three included.Each adapted assertion fails under its mutation (FR-1 widened to every status; a hidden member on the alias, including one named
classifyErrororcompleteSingleTurnWithTools; chat-shaped OpenAI replies).Review round 4 (
6cc62bdef..41c32d19b)One lane (test quality and contract). No High or Medium. Three Low:
0db9d6f61: the alias guard above, compared only after the wrap, missed an own member named like one of the wrap's three.0db9d6f61: the 401 test's title and comment said the inner message is used for the 400 arm only; they now name the HTTP-status 400 arm. No assertion changed.id,name,arguments) still matches. When the model's own tool-call arguments hold a lone surrogate, the repair changes them, the digest no longer matches (digest_mismatch, logged atdebug), and that turn is sent without its reasoning item. The turn still succeeds. A lone surrogate in a tool result, the case behind fix(mcp-host): repair lone UTF-16 surrogates before LLM dispatch; a provider 400 is shown as "Connection Error" #1433, is not covered by the digest.41c32d19bmerged with #1445's head98779cb2f(in memory, informational): conflicts only in themcp-hostversion fields; typecheckEXIT=0; full suite 8931 passed, 1 failed:sweepRetention.realWorker.test.ts(db worker made no progress for 1000ms), the load-dependent test fixed below.sweepRetention.realWorker.test.tsmade independent of machine load (c64284be0)The test (from #1111) compared the db worker's 1000 ms progress bound with wall time across 1,261 batch operations, so on a loaded machine one late acknowledgement failed it: on
deve611ebfcb, on this branch, and in T0, with no product defect (a 250-row batch costs at most about 38 ms of worker CPU; the pre-fix singleDELETEabout 4.7 s). It now compares the bound with the worker thread's CPU time per operation (worker.cpuUsage()): no operation takes 10% or more of the sweep's worker CPU (a named decision: a future batch that large fails the test even under 1000 ms), the sweep needs more than 1000 ms of worker CPU and no operation does, and every operation is sampled. The queue's wall-clock limits (300 s) and the hook and test limits (600 s) only stop a wedged worker. Test file only; no production change. Plan by Fable 5.2, validated by Opus 5.5 (APPROVE). Red on the pre-fix code (expected 1 to be less than 0.1), on a dispatcher ignoring the limit (share 0.951), on an unbounded batch (0.950), and with the caller loop dropped (expected +0 to be 15000); green on this branch, and green with 16 busy processes on the test's cores where the original failed.Tests
Head
d4df3e686(dev4fd7b2878, with #1425, merged in; no conflicts, no test line of either parent lost, mcp-host version0.1.635above dev's). Since41c32d19bthe branch changed three test files and the version only (0db9d6f61,c64284be0). Ond4df3e686: dev's service-version gate valid; T0T0_AFFECTED_CHECKS=PASS,T0_EXIT=0(codex-llm-proxy 413/413, desktop-app 5879/5879, grok-llm-proxy 410/410, mcp-host 601 files, 8703 passed, 1 skipped, 1 todo, each with build/typecheck). The remaining rows below are from41c32d19b.EXIT=0; full suite 601 files, 8703 passed, 1 skipped, 1 todo (8705). Before the merge,6cc62bdef: 590 files, 8448 of 8450;96495523d: 8442 of 8444;abebed35d: 8402 of 8404EXIT=0(no change sinceabebed35d)EXIT=0(no change sinceabebed35d)npm ciandnpm run verify:electronEXIT=0;ChatThread.renderSemantics.test.tsx13/13 (no change since96495523d)scripts/minikube/t0.sh, Node 24.22.1)T0_AFFECTED_CHECKS=PASS,T0_EXIT=0on41c32d19b(test and build of the 4 affected packages: codex-llm-proxy, desktop-app, grok-llm-proxy, mcp-host)Behaviour preserved between
abebed35dand this head (checks a to d): R's test files on this head's code fail only the 4 rows this round rewrites (T6 ownclassifyError, the Vertex numeric 400 andINVALID_ARGUMENTrows, BedrockValidationException). R's and this head's wrap and classifiers on one table differ only on deep or cyclic input (F1), colliding keys (F3),Object.keysand the ownclassifyError(F2, F3), the Vertex/Bedrock 400 rows (F6), and a well-formedNaNleaf:abebed35dcopied the message and logged arepairedUnits: 0warn for it (NaN !== NaN), and this head passes it through.Behaviour preserved between
96495523dand this head (checks a to d): the test files of96495523dpass on this head's code (590 files, 8442 passed,EXIT=0);ClaudeProvider.classifyError, bare and wrapped, on one 99-row table differs only in the message of an HTTP 400 that falls to the shared classifier (15 rows; FR-1); code, retryable, status, provider code and failover class are equal on every row; the comment-only commit changes no statement; no shell script changed.Behaviour preserved between
41c32d19band this head: this branch changed no production file (the Desktop production changes come fromdev's #1425 through the merge);41c32d19b's versions of the two files changed in0db9d6f61pass on its code (49/49), and their assertions are unchanged or added to;c64284be0replaces the wall-clock bound of one test with a CPU-time bound, with the recorded changes above; no shell changed; the text naming the changed behaviour is in those test files and this description.T2 on
d4df3e686, fresh branch-owned profile (make minikube-t2,transition=full-bootstrap):MINIKUBE_T2_PASS,T2_EXIT=0. T0T0_AFFECTED_CHECKS=PASS; full bootstrap andpre-gate-sync(host tests of every package); T1T1_STATUS=PASS, 1064/1064 tests, 132/132 files, 0 pending;NP08_HCC_AUTHORIZATION=PASS;PortForwards=PASS;Health=NOT_RUN(no health command for a full bootstrap). Earlier attempts on the same head stopped before a verdict: missing local T1 producer inputs;mcp-host/chatllmcreated by HCC after the first bootstrap's 180 s wait (Ready afterwards, no restart, no error); one load-dependent Desktop test inpre-gate-syncthat passes alone, filed as #1447.mcp-host/src/llm/__tests__/wellFormedRequestText.test.ts(36 tests after round 2; the tests of commits 4 and 7 are listed above) runs through the realmakeProvider/createLLMProvider.wellFormedRequestText.providers.test.ts(9) andproviderRejectedRequest.providers.test.ts(12) cover the eight arms. The original 14 cover:makeProvider;instanceoffor the four classes that exist on both sides of refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419;WorkflowServicewith its default factory,FailoverSingleTurnProviderover two real providers, andLlmBridgewithcreateLLMProvider;Every negative assertion has a liveness witness: the request count, the repaired text present on the wire, and the log line and counter.
Vacuity mutations (each run, logged, then reverted):
makeProviderrepairMessagesalways copiesApiCallFailedrestored for a 400ApiCallFailedagainRound 1 fixes, on the working tree later committed as commits 4 to 6 (before the commit hook's formatting and before T10 gained its
providerDispatched: falseassertion); named test files plus the drivers and the shared classifier; each run, logged, then reverted; the working tree was verified unchanged afterwards:FATAL ERROR: Reached heap limit)asyncclassifyErroroverride removedmakeProvidercontextnot repaired; tools unrepaired on the tools cache path; parts unrepaired or wrap removed on the other cache pathrepairedUnitsalways 1ApiCallFailedValidationExceptionback toApiCallFailedRound 2 fixes, on this branch's tree before commit 9 (the 5 changed test files; each run, then restored and checked by hash):
onPathentry never clearedcompleteSingleTurnoverridecompleteSingleTurn) failasync(each of the four)'claude 400'and the "request body is not valid JSON" test failmakeProviderOut of scope
core/spillover/storage.ts,core/tools/desktop/browserTools.tsandcore/orchestration/toolUseLoopPreviews.ts, which can split a surrogate pair. The repair here neutralizes the effect at dispatch; the cuts still need a follow-up.AuthenticationFailed(mcp-host: classify a Vertex token-hop credential failure as AuthenticationFailed #1437); identifiers in different objects that become equal after repair (mcp-host: identifiers that differ only by a lone surrogate merge after repair #1438); dedicated codes for provider 400 subclasses such as context length, content filter and invalid model, including parsing the Vertex error body (mcp-host: give provider 400 subclasses their own error codes #1439); unlisted provider 4xx that are retried and can fail over (mcp-host: unlisted provider 4xx are retried and can fail over #1440).🤖 Generated with Claude Code