[Fix] DeepSeek/OpenAI requests fail permanently when task history contains a lone UTF-16 surrogate

[zoomote]

​Created by Roomote. Follow up by mentioning @roomote, in the web UI, or in Discord.

Related GitHub Issue

Fixes #461

Description

DeepSeek rejects the entire request body with 400 Failed to parse the request body as JSON: messages[N].content: lone leading surrogate in hex escape when any string in the payload contains a lone UTF-16 surrogate — which can happen when task history text is sliced through an astral-plane character (emoji, CJK extension, etc.). Once a task's history contains such a string, every retry of that task fails the same way while new tasks work fine.

This applies the same sanitization pattern that #1605 added for the VS Code LM provider to the shared OpenAI-chat request path, covering the DeepSeek and OpenAI providers (and every other provider built on the same shared transforms):

• The sanitizeSurrogates / sanitizeIdentifierSurrogates / sanitizeSurrogatesDeep helpers move from vscode-lm-format.ts into a shared module, src/api/transform/sanitize-surrogates.ts (vscode-lm-format.ts re-exports them, so existing consumers and tests are unchanged). Valid surrogate pairs are untouched; lone surrogates become U+FFFD. The Copilot-specific tool-name alphabet encoding from fix(vscode-lm): sanitize lone UTF-16 surrogates in text and tool input #1605 stays vscode-lm-only — DeepSeek/OpenAI have no such constraint.
• convertToOpenAiMessages (openai-format.ts) and convertToR1Format (r1-format.ts, used by DeepSeek) sanitize: message text (simple strings and content blocks), tool_result content (string and blocks), tool_use names, nested strings inside tool arguments (JSON.stringify(sanitizeSurrogatesDeep(input))), and reasoning_content pass-through.
• Tool call/result ids use the injective sanitizeIdentifierSurrogates (escape existing U+FFFD, then encode each lone surrogate as U+FFFD + hex), composed after any caller-provided normalizeToolCallId, so ids differing only in a lone surrogate never collapse and a tool_use stays paired with its tool_result.
• OpenAiHandler sanitizes the system prompt (including the o3-family developer message; DeepSeek's system prompt flows through convertToR1Format).
• BaseProvider.convertToolsForOpenAI sanitizes tool definition name, description, and parameter schemas.

Reviewer notes / deliberate scope limits (mirroring #1605): completePrompt is unchanged; reasoning_details pass-through (OpenRouter/Gemini-specific) is unchanged. sanitizeSurrogatesDeep keeps #1605's documented lossy-key limitation (keys differing only in a lone surrogate collapse, last value wins) — accepted there and here because a request that reaches the provider beats one rejected outright.

Test Procedure

• New src/api/transform/__tests__/sanitize-surrogates.spec.ts: lone high/low surrogate, valid pair untouched, trailing lone surrogate, reversed pair, nested deep sanitization, id injectivity and U+FFFD escape ordering.
• New #461 suites in openai-format.spec.ts and r1-format.spec.ts: every serialization point above, id injectivity + tool call/result pairing, composition with normalizeToolCallId, and a whole-body JSON.stringify assertion that no lone surrogate remains.
• base-provider.spec.ts: tool definition name/description/parameters sanitization.
• Handler-level regression tests in deepseek.spec.ts and openai.spec.ts: a history carrying lone surrogates in the system prompt, message text, tool_use id/name/input, and tool_result content produces a request body free of lone surrogates with tool call/result ids still paired.
• Verified: targeted vitest suites (7 files, 378 tests passed, 1 pre-existing skip), full pnpm test from the repo root (13/13 turbo tasks successful), tsc --noEmit, and eslint with --max-warnings=0 on all touched files. Reviewers can reproduce with cd src && npx vitest run api/transform/__tests__/sanitize-surrogates.spec.ts api/transform/__tests__/openai-format.spec.ts api/transform/__tests__/r1-format.spec.ts api/providers/__tests__/deepseek.spec.ts api/providers/__tests__/openai.spec.ts.

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: New and/or updated tests have been added to cover my changes (if applicable).
• Visual Snapshot (UI changes only): Not applicable — no UI change.
• Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Visual Snapshots

Not applicable — internal request-serialization fix with no rendered surface.

Videos (interaction / animation only)

Not applicable.

Documentation Updates

• No documentation updates are required.

Zoo-Code-Docs documents provider setup (API keys, model lists) only; nothing there documents request serialization or this error mode, so no paired docs PR is needed for this internal robustness fix.

Additional Notes

The same fix pattern automatically covers the other OpenAI-compatible providers that share these transforms (OpenRouter, xAI, Requesty, etc. via convertToOpenAiMessages / convertToolsForOpenAI), since the sanitization lives in the shared path rather than in the DeepSeek provider alone.

Get in Touch

Available via the linked Roomote task or the Discord thread on #461 for reviewer questions.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• coderabbit-review-active

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: Zoo-Code-Org/Zoo-Code/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0868962c-dc6d-4e6b-ad71-2241df292551

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 94.44444% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/api/transform/r1-format.ts	84.61%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Review status

This PR was opened by an automated account. A human maintainer must verify the change intent, provenance, and validation before merging.

Current step: Mark the PR ready. Required CI must pass before CodeRabbit starts.

Review-state labels are managed by this workflow; do not edit them manually.