Skip to content

fix(mcp-host): repair lone UTF-16 surrogates before LLM dispatch; report a provider 400 as a rejected request - #1434

Merged
alfredolopez80 merged 14 commits into
devfrom
fix/mcp-host-lone-surrogate-repair
Nov 12, 2026
Merged

alfredolopez80 merged 14 commits into
devfrom
fix/mcp-host-lone-surrogate-repair

Conversation

@alfredolopez80

@alfredolopez80 alfredolopez80 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

PR 1434 main fixes: lone UTF-16 surrogate repair before every LLM call, and provider HTTP 400 reported as a rejected request

What

Closes #1433.

  1. Repair at one point. Every provider returned by makeProvider now sends only well-formed text. A lone UTF-16 surrogate in the request is replaced with U+FFFD. Each repair emits a warn log line and increments the counter clerum_llm_request_text_repaired_total{provider,role}.
  2. Proxy classification. In both subscription proxies an upstream HTTP 400 becomes upstream_rejected (with upstreamStatus: 400), where it was invalid_request before. mcp-host shows it as "Provider Rejected Request" instead of "Connection Error", with the text: "The provider rejected this request (HTTP 400). This is not a connection problem, and retrying the same request will not help."
  3. API-key classification. On the API-key paths (OpenAI family, Anthropic, Vertex and Bedrock), a provider's HTTP 400 is LLM_UPSTREAM_REJECTED instead of LLM_API_CALL_FAILED. For an HTTP 400 the message is the provider's own error text for the OpenAI family, Anthropic and Bedrock. For Vertex it is the SDK's error string, which holds Vertex's JSON error body (mcp-host: give provider 400 subclasses their own error codes #1439).
  4. Refusal instead of data loss. When repairing two object keys would make them equal (for example k\ud83d and k\udc00), the request is not sent. The Host reports the new code LLM_REQUEST_TEXT_REJECTED, which Desktop labels "Invalid Request Text". It is not retryable and never fails over.

Why

A third-party MCP tool returned text that contained half of an emoji (a lone high surrogate). mcp-host passed it to the model unchanged. JSON.stringify emitted the escape \ud83d, and every provider rejected the whole request with HTTP 400:

  • Codex and Grok through the proxies;
  • Anthropic through an API key, with The request body is not valid JSON: unexpected end of hex escape.

The user saw "Connection Error · " on every provider they tried and concluded their subscriptions were exhausted. It was neither quota nor connectivity: every call that included that tool output failed, and every other call succeeded.

How

Commit 1 — fix(mcp-host): repair lone UTF-16 surrogates before LLM dispatch

  • Where the repair runs. makeProvider (mcp-host/src/llm/registry.ts) is the only place that constructs a provider. It now returns withWellFormedRequestText(constructProvider(...)). Only the signature line of the existing body changed, which keeps refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419's edits to the return arms mergeable.
  • How the provider is wrapped (mcp-host/src/llm/wellFormedRequestText.ts).
    • The request methods the instance has are overridden in place. Optional methods that are absent stay absent, so instanceof, the adapter's capability probes and this behave as before. (Commit 4 makes the overrides non-enumerable and async.)
    • Fields walked: messages (content, content parts, tool-call ids and arguments with keys, reasoning_content, image source ids), tool definitions, and system prompt parts. Image bytes and MIME types are skipped.
    • Well-formed input passes through as the same references, so the subscription request hashes do not change. The golden hashes are rebuilt through makeProvider and still match.
    • The repair runs before the subscription providers build and hash the request, so control-api and the proxies re-derive the same hash.
  • What gets logged. One warn per repaired message, tool definition or system prompt: provider, role, the tool name and tool call id where they apply, and the number of repaired units, never the text.
  • Type library. mcp-host/tsconfig.json adds the ES2024.String lib for isWellFormed/toWellFormed. The runtime is Node 24.22.1.
  • Docs. docs/operations/monitoring.md lists the new counter.

Commit 2 — fix(llm-proxy): report an upstream 400 as upstream_rejected

  • Both proxies drop their dedicated 400 arm. The generic non-transient 4xx arm now handles an upstream 400.
  • invalid_request stays the code for the proxy's own contract refusals.
  • The context-window checks run before this mapping and do not change.
  • The contract docs are updated.
  • mcp-host's proxy clients show the fixed sentence above only when the upstream status is 400. Other upstream_rejected cases keep their current text, including the 402/403 entitlement refusals, where "not a quota problem" would be false.

Commit 3 — fix(mcp-host): classify an OpenAI-family or Anthropic API-key HTTP 400 as a rejected request

  • errorClassification.ts: an HTTP 400 that no body code maps elsewhere becomes UpstreamRejected. It stays non-retryable and without failover.
  • The Anthropic credit-balance quirk now keys on the HTTP status, so a credit-balance 400 is still InsufficientQuota.

Review round 1 fixes (commits 4 to 6)

The first review of abebed35d found two Medium and five Low findings. All are fixed.

  • Commit 4 — fix(mcp-host): repair request text without recursion and refuse colliding keys.
    • The walker used one stack frame per nesting level. A deep enough tool schema or tool-call arguments threw a synchronous RangeError, classified as retryable, so it also triggered failover and hid the subscription providers' own invalid_request ("nesting depth") refusal. The walker now uses an explicit stack and keeps a cycle's back-edge as it is, unrepaired, without entering it.
    • The overrides are async, so any failure reaches the caller as a rejected promise, and non-enumerable, so Object.keys(provider) equals a bare instance's (refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419 compares them).
    • Two keys that become equal after repair are refused before dispatch (RequestTextKeyCollisionError). The wrap's classifyError reports LLM_REQUEST_TEXT_REJECTED with providerDispatched: false and delegates every other error. Repairs are logged and counted only after the whole request is repaired, so a request refused for colliding keys reports none. A request that the provider itself then refuses or fails before dispatch (over-deep or cyclic input, below) has already logged and counted its repairs.
    • Desktop label "Invalid Request Text"; the failover classifier comment, the makeProvider docblock and the monitoring doc are updated.
  • Commit 5 — fix(mcp-host): classify Vertex and Bedrock HTTP 400s as rejected requests. Vertex numeric 400, INVALID_ARGUMENT and FAILED_PRECONDITION, and Bedrock ValidationException, move from LLM_API_CALL_FAILED to LLM_UPSTREAM_REJECTED. Quota, retired-model and auth codes keep their mappings.
  • Commit 6 — test(mcp-host): cover the repair and the 400 classification on every provider arm. Two test files run all eight makeProvider arms (openai, azure, zai, claude, the Codex and Grok subscriptions, vertex, bedrock) on their real SDKs or proxy clients. Each arm talks to a local endpoint: a loopback HTTP/1.1 or HTTP/2 server, the SDK client's fetch, or the proxy client's fetchFn. The subscription arms also stub the control-api authorizer (it echoes the request hash), the attempt context and the platform JWT; their proxy client and the server-side re-derivation of the hash are real. Vertex stubs the token hop. A guard refuses every non-loopback connection.

Review round 2 fixes (commits 7 to 9)

The second review, of 96495523d (three blind lanes and a final review), found no High or Medium finding. Fixed here:

  • Commit 7 — test(mcp-host): cover shared sub-objects, every request method and the new error literal. Tests for a sub-object referenced twice (T12b), the repair report of completeSingleTurn (T9), the rejected promise and the deferred report on all four request methods (T15c), an own __proto__ key and a null-prototype object (T5b), instanceof (T6), and the literal LLM_REQUEST_TEXT_REJECTED.
  • Commit 8 — fix(mcp-host): report Anthropic's own message for a rejected HTTP 400. An Anthropic API-key HTTP 400 now carries Anthropic's own message. It was the SDK's 400 {"type":"error",…} string.
  • Commit 9 — docs(mcp-host): correct comments on the text repair and the 400 classification. Comment and doc corrections; no behaviour change.

Behaviour changes to note

  • API-key 400s. Every OpenAI-family and Anthropic API-key 400 moves from LLM_API_CALL_FAILED ("Connection Error") to LLM_UPSTREAM_REJECTED ("Provider Rejected Request"). That includes OpenAI context-window refusals, which have no body-code mapping in this classifier, and parameter errors (mcp-host: give provider 400 subclasses their own error codes #1439). An Anthropic 400 shows Anthropic's own message, where it showed the SDK's 400 {json} string (round 2).
    • Vertex's HTTP 400, INVALID_ARGUMENT and FAILED_PRECONDITION, and Bedrock's ValidationException, move too (commit 5). An unmodeled Bedrock HTTP 400 already reached the shared classifier.
  • Colliding keys. A request whose repair would merge two object keys is refused with LLM_REQUEST_TEXT_REJECTED instead of being sent with one value dropped. Scope: MCP tool definitions arrive live from tools/list, and the SDK's schema does not reject lone surrogates in property names, so a third-party server whose schema has two such keys in one object (for example names truncated mid-emoji) makes every request that carries that tool be refused until the tool is fixed or removed. A collision needs two keys of one object that differ only by different lone units; none has been observed.
  • Over-deep or cyclic input. The repair no longer throws. An over-deep request reaches each provider's own handling: Codex and Grok refuse it with invalid_request before authorizing; the API-key SDKs serialize it and, above JSON.stringify's own limit, fail with a retryable RangeError, as they do on dev (pre-existing, out of scope). Every arm refuses or fails a cyclic request in the provider before dispatch, after its repairs were logged and counted, and the back-edge keeps the unrepaired original. A sub-object reachable along several paths is repaired, and counted, once per path; for objects that all reference each other the number of paths grows factorially (9 objects: about 40 to 50 ms of synchronous work, about 9 times more per extra object; abebed35d failed in about 3 ms with RangeError). Code-built tool schemas reuse sub-objects (the internal tools' schemas: at most 1.4 visits per unique container); no producer of cycles or of mutually referencing objects was found, so this is documented and not guarded.
  • Proxy failure metric. For an upstream 400, the metric moves from the invalid_request label to upstream_rejected, and the proxy reply becomes HTTP 422 instead of 400.
  • Repeated warnings on replayed history. A stored user message that holds a lone unit is repaired, logged and counted again on every replay of the session history. The store is not rewritten. This was an accepted decision; the counter counts repaired units per call, not distinct inputs.
  • Token counting (known gap). core/tokenizer/anthropicTokenCounter.ts sends unrepaired text to Anthropic countTokens when CLERUM_TOKENIZER_OFFLINE=false (the default is true). A 400 there already falls back to the heuristic count. Not changed here.

Merge of dev with #1419 and #1420

dev (e611ebfcb) now has #1419 and #1420. Merged into this branch as 3fce597c5; the only conflicts were the mcp-host version fields (dev's kept; the next commit's hook raises the version to 0.1.633, above dev's 0.1.632, and dev's service-version gate passes on this head). No test line of either parent is lost in the merge (check a.3).

On the merged tree 20 tests failed, all expectations or fixtures; 41c32d19b adapts them with no production change:

  • feat(mcp-host): route official OpenAI through the Responses protocol #1420's T-E3 and "too many tools 400" now expect LLM_UPSTREAM_REJECTED for a provider HTTP 400. The chat corpus is regenerated locally (UPDATE_CHAT_CORPUS=1): only code changes, on the 16 classification rows of an HTTP 400.
  • 16 tests of this PR on the OpenAI arm: the openai profile sends both operations through the Responses API, so its fixture replies use the Responses shape (Azure and the compatible profiles keep chat completions), and T18 reads the function_call item from the Responses input.
  • refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419's alias guard (Reflect.ownKeys since 7733fe136) compares the alias with the registry-built provider, which the factory wraps with withWellFormedRequestText (three own non-enumerable members: classifyError, completeSingleTurn, completeSingleTurnWithTools). Since 0db9d6f61 the unwrapped alias must carry exactly the factory's own keys minus those three, and the wrapped alias must match the factory exactly, so a hidden member is caught under any name, the wrap's three included.
  • New: an enveloped Anthropic 401 that reaches the shared classifier keeps the SDK's message, pinning that FR-1 changes the HTTP-status 400 arm only (the body-type arms already used Anthropic's message at any status).

Each adapted assertion fails under its mutation (FR-1 widened to every status; a hidden member on the alias, including one named classifyError or completeSingleTurnWithTools; chat-shaped OpenAI replies).

Review round 4 (6cc62bdef..41c32d19b)

One lane (test quality and contract). No High or Medium. Three Low:

  • L-1, fixed in 0db9d6f61: the alias guard above, compared only after the wrap, missed an own member named like one of the wrap's three.
  • L-3, fixed in 0db9d6f61: the 401 test's title and comment said the inner message is used for the 400 arm only; they now name the HTTP-status 400 arm. No assertion changed.
  • L-2, accepted, follow-up issue: feat(mcp-host): route official OpenAI through the Responses protocol #1420 replays a Responses reasoning item only when a digest of the assistant turn's tool calls (id, name, arguments) still matches. When the model's own tool-call arguments hold a lone surrogate, the repair changes them, the digest no longer matches (digest_mismatch, logged at debug), and that turn is sent without its reasoning item. The turn still succeeds. A lone surrogate in a tool result, the case behind fix(mcp-host): repair lone UTF-16 surrogates before LLM dispatch; a provider 400 is shown as "Connection Error" #1433, is not covered by the digest.

41c32d19b merged with #1445's head 98779cb2f (in memory, informational): conflicts only in the mcp-host version fields; typecheck EXIT=0; full suite 8931 passed, 1 failed: sweepRetention.realWorker.test.ts (db worker made no progress for 1000ms), the load-dependent test fixed below.

sweepRetention.realWorker.test.ts made independent of machine load (c64284be0)

The test (from #1111) compared the db worker's 1000 ms progress bound with wall time across 1,261 batch operations, so on a loaded machine one late acknowledgement failed it: on dev e611ebfcb, on this branch, and in T0, with no product defect (a 250-row batch costs at most about 38 ms of worker CPU; the pre-fix single DELETE about 4.7 s). It now compares the bound with the worker thread's CPU time per operation (worker.cpuUsage()): no operation takes 10% or more of the sweep's worker CPU (a named decision: a future batch that large fails the test even under 1000 ms), the sweep needs more than 1000 ms of worker CPU and no operation does, and every operation is sampled. The queue's wall-clock limits (300 s) and the hook and test limits (600 s) only stop a wedged worker. Test file only; no production change. Plan by Fable 5.2, validated by Opus 5.5 (APPROVE). Red on the pre-fix code (expected 1 to be less than 0.1), on a dispatcher ignoring the limit (share 0.951), on an unbounded batch (0.950), and with the caller loop dropped (expected +0 to be 15000); green on this branch, and green with 16 busy processes on the test's cores where the original failed.

Tests

Head d4df3e686 (dev 4fd7b2878, with #1425, merged in; no conflicts, no test line of either parent lost, mcp-host version 0.1.635 above dev's). Since 41c32d19b the branch changed three test files and the version only (0db9d6f61, c64284be0). On d4df3e686: dev's service-version gate valid; T0 T0_AFFECTED_CHECKS=PASS, T0_EXIT=0 (codex-llm-proxy 413/413, desktop-app 5879/5879, grok-llm-proxy 410/410, mcp-host 601 files, 8703 passed, 1 skipped, 1 todo, each with build/typecheck). The remaining rows below are from 41c32d19b.

Package Result
mcp-host lint (0 errors), typecheck and build EXIT=0; full suite 601 files, 8703 passed, 1 skipped, 1 todo (8705). Before the merge, 6cc62bdef: 590 files, 8448 of 8450; 96495523d: 8442 of 8444; abebed35d: 8402 of 8404
codex-llm-proxy test (413/413) and build EXIT=0 (no change since abebed35d)
grok-llm-proxy test (410/410) and build EXIT=0 (no change since abebed35d)
desktop-app Node 24.22.1, npm ci and npm run verify:electron EXIT=0; ChatThread.renderSemantics.test.tsx 13/13 (no change since 96495523d)
T0 (scripts/minikube/t0.sh, Node 24.22.1) T0_AFFECTED_CHECKS=PASS, T0_EXIT=0 on 41c32d19b (test and build of the 4 affected packages: codex-llm-proxy, desktop-app, grok-llm-proxy, mcp-host)

Behaviour preserved between abebed35d and this head (checks a to d): R's test files on this head's code fail only the 4 rows this round rewrites (T6 own classifyError, the Vertex numeric 400 and INVALID_ARGUMENT rows, Bedrock ValidationException). R's and this head's wrap and classifiers on one table differ only on deep or cyclic input (F1), colliding keys (F3), Object.keys and the own classifyError (F2, F3), the Vertex/Bedrock 400 rows (F6), and a well-formed NaN leaf: abebed35d copied the message and logged a repairedUnits: 0 warn for it (NaN !== NaN), and this head passes it through.

Behaviour preserved between 96495523d and this head (checks a to d): the test files of 96495523d pass on this head's code (590 files, 8442 passed, EXIT=0); ClaudeProvider.classifyError, bare and wrapped, on one 99-row table differs only in the message of an HTTP 400 that falls to the shared classifier (15 rows; FR-1); code, retryable, status, provider code and failover class are equal on every row; the comment-only commit changes no statement; no shell script changed.

Behaviour preserved between 41c32d19b and this head: this branch changed no production file (the Desktop production changes come from dev's #1425 through the merge); 41c32d19b's versions of the two files changed in 0db9d6f61 pass on its code (49/49), and their assertions are unchanged or added to; c64284be0 replaces the wall-clock bound of one test with a CPU-time bound, with the recorded changes above; no shell changed; the text naming the changed behaviour is in those test files and this description.

T2 on d4df3e686, fresh branch-owned profile (make minikube-t2, transition=full-bootstrap): MINIKUBE_T2_PASS, T2_EXIT=0. T0 T0_AFFECTED_CHECKS=PASS; full bootstrap and pre-gate-sync (host tests of every package); T1 T1_STATUS=PASS, 1064/1064 tests, 132/132 files, 0 pending; NP08_HCC_AUTHORIZATION=PASS; PortForwards=PASS; Health=NOT_RUN (no health command for a full bootstrap). Earlier attempts on the same head stopped before a verdict: missing local T1 producer inputs; mcp-host/chatllm created by HCC after the first bootstrap's 180 s wait (Ready afterwards, no restart, no error); one load-dependent Desktop test in pre-gate-sync that passes alone, filed as #1447.

mcp-host/src/llm/__tests__/wellFormedRequestText.test.ts (36 tests after round 2; the tests of commits 4 and 7 are listed above) runs through the real makeProvider / createLLMProvider. wellFormedRequestText.providers.test.ts (9) and providerRejectedRequest.providers.test.ts (12) cover the eight arms. The original 14 cover:

  • the OpenAI and Anthropic SDKs against a local HTTP server;
  • Codex and Grok with stubbed authorizer and proxy dependencies;
  • golden subscription hashes rebuilt through makeProvider;
  • every walked field, image bytes kept, and inputs not mutated;
  • optional methods preserved across the provider arms, and instanceof for the four classes that exist on both sides of refactor(mcp-host): compose the OpenAI driver family from endpoint profiles #1419;
  • WorkflowService with its default factory, FailoverSingleTurnProvider over two real providers, and LlmBridge with createLLMProvider;
  • a two-turn conversation whose first user message holds a lone surrogate.

Every negative assertion has a liveness witness: the request count, the repaired text present on the wire, and the log line and counter.

Vacuity mutations (each run, logged, then reverted):

Mutation Result
Wrap removed from makeProvider 11 of 14 tests fail; the 3 that pass assert well-formed input is unchanged
repairMessages always copies the same-reference test fails
Optional cache method always defined the optional-method test fails
Proxy 400 arm restored 5 tests fail in each proxy
Fixed 400 message disabled both client tests fail
ApiCallFailed restored for a 400 4 classifier tests fail
Credit guard keyed on ApiCallFailed again 2 tests fail

Round 1 fixes, on the working tree later committed as commits 4 to 6 (before the commit hook's formatting and before T10 gained its providerDispatched: false assertion); named test files plus the drivers and the shared classifier; each run, logged, then reverted; the working tree was verified unchanged afterwards:

Mutation Result
Recursive walker restored T10 (Codex, Grok), T11 and T12 fail
Cycle check removed the worker aborts on heap exhaustion (FATAL ERROR: Reached heap limit)
Overrides not async T13/T15, T15b, T16 and T17 fail
Key collisions never counted T13/T15 to T19c fail (7 tests)
classifyError override removed T13/T15, T19a-c and T6/T14 fail
Overrides enumerable T6/T14 fails
Each repair reported as it happens T15b fails
Wrap removed from makeProvider 35 tests fail, all 8 repair arms and T11 among them
context not repaired; tools unrepaired on the tools cache path; parts unrepaired or wrap removed on the other cache path T2 / T20 fail
Tool-definition report dropped; repairedUnits always 1 11 and 4 tests fail (T21, T22, all 8 repair arms)
Shared 400 arm back to ApiCallFailed the openai, azure, zai and claude 400 rows fail, plus 2 classifier tests
Vertex 400 / Bedrock ValidationException back to ApiCallFailed the vertex row and 2 driver rows / the bedrock row and 1 driver row fail
Loopback guard not installed the 3 witness tests fail

Round 2 fixes, on this branch's tree before commit 9 (the 5 changed test files; each run, then restored and checked by hash):

Mutation Result
onPath entry never cleared T12b fails
No repair report in the completeSingleTurn override T9 and T15c (completeSingleTurn) fail
One override not async (each of the four) its T15c row fails (the tools method: 5 tests)
Report before the last repair step (six placements on the three methods that repair more than one part) the matching T15c row fails each time (on the tools method, 13 tests)
Code literal renamed the errors test fails
Plain assignment for keys; no null prototype T5b fails
Detached prototype T6/T14 fails
FR-1 arm removed 'claude 400' and the "request body is not valid JSON" test fail
Wrap removed from makeProvider 41 of 81 fail

Out of scope

🤖 Generated with Claude Code

alfredolopez80 and others added 6 commits November 11, 2026 17:28
A lone UTF-16 surrogate in any request text made every provider reject the
whole request with HTTP 400: JSON.stringify emits it as an escape such as
\ud83d, which Anthropic reports as "unexpected end of hex escape" and the
Codex and Grok upstreams refuse. In production a third-party MCP tool
returned such text, and the agent failed with every provider while the user
saw "Connection Error" and assumed an exhausted subscription.

makeProvider is the only place that constructs a provider, so it now wraps
each one with withWellFormedRequestText. The request methods the instance
has are overridden in place, so instanceof, optional-method probes and
`this` are unchanged. Every string in messages (content, content parts,
tool-call ids and arguments, keys included, image source ids), tool
definitions and system prompt parts goes through toWellFormed(), which
replaces a lone surrogate with U+FFFD. Image bytes and MIME types are
skipped. Well-formed input passes through as the same references, so its
requests and the subscription request hashes are unchanged. The repair
runs before the subscription providers build and hash the request, so
control-api and the proxies re-derive the same hash.

The repair is visible: one warn line per repaired message (provider, role,
tool name, tool call id, unit count; never the text) and the counter
clerum_llm_request_text_repaired_total{provider,role}. A stored user
message holding a lone unit is repaired, logged and counted on every
replay; the store is not rewritten.

Tests go through the real makeProvider/createLLMProvider: the OpenAI and
Anthropic SDKs against a local HTTP server, and the Codex and Grok
providers with stubbed authorizer and proxy deps. They also cover the
golden subscription hashes, WorkflowService with its default factory,
FailoverSingleTurnProvider, LlmBridge with createLLMProvider, and a
two-turn conversation. With the wrap removed from makeProvider, 11 of the
14 tests fail; the remaining 3 assert that well-formed input is unchanged
and have their own mutations.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both subscription proxies mapped an upstream HTTP 400 to invalid_request,
the same code they use when their own transport contract refuses a body.
mcp-host has no arm for invalid_request, so it fell through to
LLM_API_CALL_FAILED, which Desktop shows as "Connection Error". A provider
rejecting the request read as a connectivity or quota problem (#1433).

The proxies now report an upstream 400 like every other non-transient
upstream 4xx: upstream_rejected with upstreamStatus 400 (HTTP 422, or the
same code on an SSE error frame). invalid_request stays the code for the
proxy's own refusals. The context-window checks run before this mapping and
are unchanged. The proxy failure metric moves those events from the
invalid_request label to upstream_rejected.

mcp-host already classifies upstream_rejected as LLM_UPSTREAM_REJECTED
("Provider Rejected Request"), non-retryable and without failover. For
upstream status 400 its message is now a fixed sentence that holds for
every 400: "The provider rejected this request (HTTP 400). This is not a
connection problem, and retrying the same request will not help." Other
upstream 4xx, including the 402/403 entitlement refusals, keep their text.

The proxy conformance and server tests expect upstream_rejected for an
upstream 400 and add 400 to the G1-3a cases. Contract-refusal tests still
expect invalid_request. With the old 400 arm restored, 5 tests fail in each
proxy; with the message arm disabled, the two client tests fail.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0 as a rejected request

The shared HTTP classifier mapped a provider's HTTP 400 to
LLM_API_CALL_FAILED, which Desktop renders as "Connection Error". In the
#1433 incident the Anthropic API-key path answered "The request body is
not valid JSON: unexpected end of hex escape", and the user read it as a
connectivity or quota problem.

A 400 that no body code maps elsewhere (insufficient_quota,
model_not_found and invalid_api_key keep their codes) is now
LLM_UPSTREAM_REJECTED: still non-retryable, still without failover, now
labelled "Provider Rejected Request". The message stays the provider's own
text. This covers the OpenAI, Azure and OpenAI-compatible providers and
Claude. Every 400 on those paths moves, including OpenAI context-window
refusals (there is no context_length_exceeded body mapping in this
classifier) and parameter errors. Vertex keeps its own 400 arms, and so
does Bedrock's modeled ValidationException; only an unmodeled Bedrock
HTTP 400 reaches this classifier.

The Anthropic credit-balance quirk (insufficient credit reported as HTTP
400) keyed on the ApiCallFailed code. It now keys on the HTTP status, so a
credit-balance 400 is still InsufficientQuota.

Tests: the shared classifier, the plain and enveloped Anthropic 400s, the
"not valid JSON" envelope, and the enveloped credit-balance case. With
ApiCallFailed restored for 400, 4 tests fail; with the credit guard keyed
on ApiCallFailed again, 2 fail.

Open PR #1420 adds two tests that pin an OpenAI 400 to ApiCallFailed
(responsesLoopErrors.test.ts T-E3, openaiFamily.contract.test.ts "too many
tools 400"). Whichever PR merges second updates those expectations to
UpstreamRejected.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ding keys

Round 1 of the #1434 review found three defects in the lone-surrogate
repair wrap.

The walker recursed once per nesting level. A tool schema or tool-call
arguments nested deeply enough overflowed the stack (20000 levels in the
round-2 comparison table; the threshold depends on the process stack
size), and the wrap threw a synchronous RangeError. That replaced the
providers' own typed refusal (invalid_request, "nesting depth") with an
error classified as retryable, which also triggered failover. The walker
now uses an explicit stack, reads each property once, and keeps a
container that is already on the current path (a cycle) as is instead of
entering it. The overrides are async, so
any failure reaches the caller as a rejected promise.

The overrides were enumerable own properties, so Object.keys of a wrapped
provider differed from a bare instance; open PR #1419 compares them. They
are now non-enumerable, like class methods.

Repairing two keys that differ only in their lone surrogates made them
equal, and one value was silently dropped. Such a request is now refused
before dispatch with RequestTextKeyCollisionError. The wrap also overrides
classifyError, which reports it as the new LLM_REQUEST_TEXT_REJECTED: not
retryable, not dispatched, no failover class. Desktop labels it "Invalid
Request Text". Repairs are logged and counted only once the whole request
has been repaired, so a refused request reports none.

Tests (wellFormedRequestText.test.ts): a 150000-deep schema and arguments
on Codex and Grok keep the provider's refusal without authorizing; a
150000-deep schema passes through by reference or is repaired at its
bottom; cycles; collisions in arguments and schemas, a repaired key equal
to an untouched one, distinct repaired keys; the refusal through
LlmPortAdapter, FailoverSingleTurnProvider and LlmBridge; both Anthropic
cache paths with tools and context; exact warn fields and unit counts;
descriptors on all eight arms. The tests in this file that drive a real
SDK run behind a guard that refuses any non-loopback connection.

Texts: the failover classifier comment names a non-retryable
ApiCallFailed without calling it a 400 and lists RequestTextRejected;
the makeProvider docblock and the monitoring doc state that a refused
request is not counted as a repair.

Mutations, each run and reverted: the recursive walker fails T10-T12; no
cycle check aborts the worker on heap exhaustion; sync overrides fail
T13-T17; no collision count fails T13-T19; no classifyError override fails
T13, T19a-c and T6/T14; enumerable overrides fail T6/T14; per-repair reporting
fails T15b; the guard removed fails its witness tests.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests

After this PR's API-key change, an HTTP 400 read "Provider Rejected
Request" on the OpenAI-family, Anthropic and subscription paths but still
"Connection Error" on Vertex and on Bedrock's modeled ValidationException.

Vertex: a numeric 400, INVALID_ARGUMENT and FAILED_PRECONDITION are now
LLM_UPSTREAM_REJECTED. Bedrock: ValidationException is now
LLM_UPSTREAM_REJECTED; an unmodeled Bedrock 400 already reached the shared
classifier. Quota, retired-model and auth codes keep their mappings.

Texts made false by this PR are corrected: the UpstreamRejected doc
comment now names the Vertex and Bedrock arms, and two test comments that
called a non-retryable ApiCallFailed a 400 now describe a replayed
subscription ticket.

Tests: the driver classification rows for numeric and code-field 400s,
FAILED_PRECONDITION, ValidationException and an unmodeled Bedrock 400.
With ApiCallFailed restored on Vertex 2 tests fail, on Bedrock 1.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…provider arm

Every provider makeProvider builds now has a test on its real transport:
openai, azure, the OpenAI-compatible data arm (zai), claude, the Codex and
Grok subscriptions, vertex and bedrock. Each arm runs its real SDK or
proxy client; only the last hop is local. openai, claude and vertex use a
loopback HTTP server through their base-URL variables; bedrock a loopback
HTTP/2 server through AWS_ENDPOINT_URL_BEDROCK_RUNTIME, with the user's AWS
files and instance metadata disabled; azure and zai the SDK client's fetch;
the subscriptions the proxy clients' fetchFn. Vertex stubs only the token
hop of the google-auth-library copy @google/genai loads. A guard refuses
every non-loopback connection, with a witness test.

wellFormedRequestText.providers.test.ts: one request per arm with a lone
surrogate in a tool result and a tool description. The wire body is
well-formed, both warns carry the exact fields, both counters rise, and on
the subscriptions the authorized hash equals the server re-derivation and
the proxy body.

providerRejectedRequest.providers.test.ts: a provider 400 on each arm is
LLM_UPSTREAM_REJECTED, not retryable, with no failover class and the
provider's message kept; the Codex and Grok proxies' upstream_rejected
carries the fixed message. Z.ai code 1211, the Anthropic credit-balance
400 and Bedrock ServiceQuotaExceededException keep their own codes.

Mutations: the wrap removed from makeProvider fails all eight repair arms;
the shared 400 arm back to ApiCallFailed fails openai, azure, zai and
claude; the Vertex and Bedrock arms back to ApiCallFailed fail their rows.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alfredolopez80 and others added 5 commits November 11, 2026 22:01
…e new error literal

Round 2 of the #1434 review found changed behaviour of the text repair that no test would
catch. No production change in this commit.

- T12b: a sub-object referenced twice (as the internal tool schemas do) is repaired at both
  references and counted once per path.
- T9: the completeSingleTurn override reports its repair (warn and counter), on the first
  turn and on the replay.
- T15c: on all four request methods, colliding keys are refused as a rejected promise and no
  repair is reported, with the collision in the last part each method repairs.
- T5b: an own "__proto__" key and a null-prototype object survive the repair of a sibling.
- T6: each makeProvider arm is still an instance of its class.
- errors.test: the literal value LLM_REQUEST_TEXT_REJECTED, which Desktop's labels key on.

Each test fails under its mutation, run and reverted on this tree: onPath never cleared
(T12b), no report in completeSingleTurn (T9, T15c), a synchronous override or a report
before the last repair step (T15c), plain assignment or no null prototype (T5b), a detached
prototype (T6), a renamed code literal (errors.test).

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Anthropic SDK keeps the response envelope in `error` and builds `message` as
`400 {json}`. The shared classifier returned that string, so the user read the raw JSON
envelope instead of the provider's text, which is the case #1433 started from.

The 400 arm of ClaudeProvider.classifyError now uses the inner message, as the
credit-balance arm already does. Code, retryable, status, provider code and failover class
are unchanged, and no other status changes.

Tests: the `claude 400` row of providerRejectedRequest.providers.test.ts and the "request
body is not valid JSON" test now assert the exact message; both fail with the arm removed.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ification

Comments, test headers and one docs row only; no statement changes. Corrected claims:

- When repairs are reported: after the whole request is repaired and before the provider
  method is called. A request refused for colliding keys reports none, but a repaired
  request the provider then refuses (over-deep or cyclic input) has already been reported.
- Cyclic and shared input: a cycle's back-edge is kept unrepaired and every arm refuses or
  fails that request before dispatch; a sub-object reachable along several paths is walked
  once per path. Code-built tool schemas do reuse sub-objects; no producer of cyclic or
  mutually referencing input is known. "Each property is read once" is removed.
- Vertex: a numeric 400 can also come from Google's token endpoint (`invalid_grant`) with no
  request reaching Vertex (#1437).
- The members of the non-retryable ApiCallFailed failover row.
- What the transport fixtures stub besides the endpoint (the subscription authorizer,
  attempt context and platform JWT; the Vertex token hop).
- What the repair warn names (the role, and the tool for tool results and definitions).

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…epair

Conflicts: mcp-host/package.json and package-lock.json (version only); dev's files kept,
the pre-commit hook bumps the version above dev's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n on dev

dev now has #1419 and #1420. Twenty tests failed on the merged tree; no production change is
needed, only expectations and fixtures.

- The `openai` profile sends both operations through the Responses API. The OpenAI fixture
  replies of wellFormedRequestText.test.ts and of the `openai` transport arm now use the
  Responses shape; Azure and the compatible profiles keep chat completions. T18 reads the
  assistant's `function_call` item from the Responses `input`.
- #1420's T-E3 (responsesLoopErrors) and "too many tools 400" (openaiFamily.contract) expect
  LLM_UPSTREAM_REJECTED for a provider HTTP 400. The chat corpus is regenerated locally with
  UPDATE_CHAT_CORPUS=1: only `code` changes, on the 16 classification rows of an HTTP 400.
- #1419's alias guard compares Reflect.ownKeys with the registry-built provider, which the
  factory wraps with withWellFormedRequestText (three own non-enumerable members). The alias
  is now compared after the same wrap, so any other hidden member is still caught.
- claude.classifyError: an enveloped 401 keeps the SDK's message, pinning that the inner
  message is used for the 400 arm only (review round 3, R3-L2).

Mutations, run and reverted: the FR-1 arm widened to every status fails the new 401 test; a
hidden member in OpenAICompatibleProvider fails the alias guard; chat-shaped OpenAI replies
fail 15 tests in wellFormedRequestText.test.ts and the `openai` transport arm.

Full mcp-host suite on the merged tree: 601 files, 8703 passed, 1 skipped, 1 todo.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alfredolopez80
alfredolopez80 marked this pull request as ready for review November 11, 2026 22:32
@Mircala

Mircala commented Nov 11, 2026

Copy link
Copy Markdown

Evenfire Reviewer — Round 1

Reviewed head: 41c32d19b87060833c0f1f72c8f8f9d407443168

Decision: No objection

The repair runs at the single place every provider is built, and a provider HTTP 400 now reads as a rejected request on every arm instead of "Connection Error". I traced the request path from makeProvider through the subscription hashes, the failover classifier and the Desktop label, and I ran the walker on Node 24.22.1. Nothing I found changes the outcome of a request or drops a value.

Must fix

No must-fix items found.

Verified

  • makeProvider is the only provider constructor outside tests, and the wrap covers the four request methods plus classifyError, leaving optional methods absent when the class lacks them.
  • The walker returns the same reference when nothing changed, copies only the changed path, keeps a null prototype and an own __proto__ key, and refuses a repair that would make two keys of one object equal before anything is logged or sent. Reproduced on Node 24.22.1, including a shared sub-object, a cycle, and a 200,000-deep object.
  • An upstream 400 is upstream_rejected in both proxies, with the context-window check still running first, and LLM_UPSTREAM_REJECTED on the OpenAI-family, Anthropic, Vertex and Bedrock arms. The Anthropic credit-balance 400 and the Z.AI/Bailian native codes keep their own classification. Failover class is null for both new codes.
  • projectMessage re-derives content from the repaired text parts, so the Codex and Grok request hashes match what control-api and the proxies re-derive.
  • CI at this head: 63 checks, all success, including CodeQL and the build/test of mcp-host, both proxies and desktop-app. git diff --check is clean.

Validation gaps

  • Previous-head regression proof was not run in this environment; the checkout here is not at the PR head. The CI run on 41c32d19b is the evidence.
  • T2 is still pending, as the description says.

PR description

Accurate. It names the head, records the test counts, and states the changes that are easy to misread: the proxy metric label moving from invalid_request to upstream_rejected, the repeated warn on replayed history, and the token-counter gap. The deferred follow-ups match the code.

Pre-existing

Pre-existing: none identified during the scoped review.

Next action

None required before merge. The follow-ups already filed cover the rest: a Vertex invalid_grant reported as a rejected request rather than an authentication failure (#1437), identifiers in different objects that become equal after repair (#1438), dedicated codes for the 400 subclasses (#1439), and the unlisted provider 4xx (#1440).

Full review archive: pr-reviews/pr-1434/r1-26-11-11.md

— Review performed by Evenfire Reviewer

@jozer-rami jozer-rami left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 41c32d19b. I found no blocker and no major. The six items below are minor, so they can be fixed here or tracked as follow-ups. T2 on this head is still pending, as the description says.

Run at head (Node 24.22.1): the mcp-host suite passed 8703 of 8705 (1 skipped, 1 todo); tsc --noEmit exited 0; codex-llm-proxy passed 413/413 and grok-llm-proxy 410/410. 25 of the 27 mutations I applied made tests fail.

Test gaps (both reproduced with a mutation and a probe test)

  1. A lone surrogate that is a direct array element has no test (wellFormedRequestText.ts:204). I made the array arm count the string but not write the repaired value, and all the repair tests still pass. Fields that hit this arm include a schema enum or required list and string-array tool arguments. If it regressed, the provider would get the #1433 400 again, with no warn and no counter. Test idea: enum: ['x\uD83D'] and arguments { tags: ['a\uD83D'] }, asserting on the repaired text sent to the provider and on the warn and counter.
  2. Key-collision order (wellFormedRequestText.ts:120). The code at head is correct. A mutation that checks only the repaired keys still passes every test. With that mutation, { 'k\uD83D': …, 'k�': … } (the repaired key first) silently drops one value. Test idea: the existing collision case with the key order reversed.

Minor
3. On the Responses path, the repair changes the continuation digest. The digest is computed over the parsed arguments at decode (openaiFamily/protocols/responses.ts:484). Once the wrap repairs those arguments, the check at :266 reports digest_mismatch, and reasoning replay is dropped with only a debug line. Before this PR that request already went out well-formed, because the arguments are sent through JSON.stringify (:186/:237). Possible fix: make the digest invariant to the repair, or repair at decode.
4. When a key collision is refused, no log line or metric tells an operator which tool caused it (wellFormedRequestText.ts:326, :352). The adapter logs only the code and the error name, and the logger reduces an Error to name/code/status (logger.ts:57-67), so the tool name never reaches mcp-host logs. Possible fix: emit one warn and increment a refused counter, with the same bounded fields as a repair.
5. The role label and log field take values from hook-rewritten messages. remoteLlmHook.ts:189 does not validate role, so each new role becomes a new Prometheus series. The same flow already bounds hook-sourced labels with boundedReason (hookedLlmPort.ts:64-72). Possible fix: map role to a closed set, with other for anything else, and cap toolName and toolCallId in the warn.
6. The FR-1 arm (claude.ts:135-139) keeps only Anthropic's inner message. That drops the rest of the body, including request_id when the body carries it, and err.request_id is not logged in providerErrorDiagnostics either. Possible fix: add the request id to the diagnostics.

alfredolopez80 and others added 3 commits November 12, 2026 11:09
… the 401 test text

Review round 4 on 6cc62bd..41c32d1, two Low findings, test and text only.

- L-1: the #1419 alias guard compared Reflect.ownKeys only after
  withWellFormedRequestText, which defines classifyError and both request
  methods as own members. An own member of OpenAICompatibleProvider with one of
  those names was masked. The guard now also requires the unwrapped alias to
  carry exactly the factory's own keys minus the wrap's three members, asserts
  the factory carries those three, and keeps the wrapped comparison.
- L-3: the 401 test said the inner Anthropic message is used for the 400 arm
  only. The body-type arms (not_found, overloaded, billing, permission) already
  return it at any status. The title and comment now name the HTTP-status 400
  arm and the shared-classifier path the 401 takes. No assertion changed.

Mutations, run and reverted: an own non-enumerable classifyError,
completeSingleTurnWithTools or extraHidden on the alias each fail the guard
(the first two passed before this change); the FR-1 arm widened to every status
fails the renamed 401 test.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t wall time

sweepRetention.realWorker.test.ts (e38c9e4, #1111) compared the db
worker's 1000 ms progress bound and a 900 ms sync timeout with wall time
across 1,261 batch operations. On a loaded machine one acknowledgement can
take longer than that without the batch doing more work (waiting for a
core, the disk or the checkpoint, or the main thread reading the reply
late), so the test failed with "db worker made no progress for 1000ms" on
dev e611ebf and on unrelated branches, and failed T0. A 250-row batch
costs at most about 38 ms of worker CPU (77 ms when CPU-starved); the
pre-fix single DELETE costs about 4.7 s.

The test keeps its fixture, the real worker thread and the 1000 ms bound,
and compares the bound with the CPU time the worker thread spends on each
operation (worker.cpuUsage(), sampled around each enqueueSync):
- no operation takes 10% or more of the sweep's worker CPU (owner
  decision: a future batch that large fails this test even under 1000 ms);
- the sweep needs more than 1000 ms of worker CPU, and no operation does;
- every operation was sampled, and there was more than one.
The queue's wall-clock limits become 300 s and the hook and test limits
600 s; they only stop a wedged worker from hanging the run. No production
code changes.

Red reasons now: pre-fix 3158019 (one operation) "expected 1 to be less
than 0.1"; dispatcher ignoring the limit and an unbounded batch size, share
0.951 and 0.950; caller loop dropped, "expected +0 to be 15000". Green on
this branch (11 s at load average 6); the plan's run with 16 busy processes
on the test's cores was green, where the original failed.

Refs #1433

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogate-repair

* origin/dev: (27 commits)
  test(desktop-app): pin the order of a removal and a replace on one chat
  test(desktop-app): name catalog isolation tests by behaviour
  docs(desktop-app): state what a failed write of a kept turn leaves
  test(desktop-app): name rejected-send tests by behaviour
  test(desktop-app): fix rejected-send test titles and a witness
  docs(desktop-app): describe what each send failure reason and flag records
  docs(desktop-app): state what the removal rule and a withdrawal cover
  fix(desktop-app): release a send's activity entry on every exit
  fix(desktop-app): keep a held send that was posted and failed without a rejection
  refactor(desktop-app): share the durable write of a kept turn
  fix(desktop-app): show the failure sentence for a send kept behind Recover input
  test(desktop-app): cover the tombstone and sender checks of chat:removeMessages
  test(desktop-app): cover the unprotected guards of the rejected-send path
  test(desktop-app): cover the attachment guards of a rejected send
  refactor(desktop-app): simplify the rejected-send path
  docs(desktop-app): describe connector rows for an agent the catalog does not list
  test(desktop-app): state why the second catalog differs
  fix(desktop-app): return a rejected send to the composer
  feat(desktop-app): remove unanswered local turns from the chat store
  test(desktop-app): cover a held agent restored from a workspace snapshot
  ...
@alfredolopez80
alfredolopez80 merged commit 05419da into dev Nov 12, 2026
63 checks passed
@alfredolopez80
alfredolopez80 deleted the fix/mcp-host-lone-surrogate-repair branch November 12, 2026 14:46
jozer-rami added a commit that referenced this pull request Nov 12, 2026
…ew-r2

Brings in #1434 (lone UTF-16 surrogate repair). The only conflict is the
mcp-host version line: dev 0.1.635, branch 0.1.639, merged as 0.1.640.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zach88 added a commit that referenced this pull request Nov 12, 2026
Brings #1434 (lone UTF-16 surrogate repair at makeProvider, HTTP 400
classified as UpstreamRejected) and the repo-hygiene CI check (#1448).
Only textual conflict: the mcp-host version bump; kept 0.1.648.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zach88 added a commit that referenced this pull request Nov 12, 2026
…om dev

dev (#1434) now wraps every provider from makeProvider with
withWellFormedRequestText and adds ES2024.String to the tsconfig lib.
Three comments of the wire-name projection became inaccurate:

- the annotation helper said toWellFormed is unavailable under ES2022;
- the UTF-16LE hash note implied injectivity over lone surrogates on the
  production path, where names arrive already repaired to U+FFFD;
- the Claude counter claimed to count exactly the sent names, which no
  longer holds for a name with lone surrogates (the counter is unwrapped).

Comment-only; no behaviour change. Clean merge, no textual conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants