Skip to content

DeepSeek thinking mode: residual "reasoning_content must be passed back" 400 on the first post-fold request (0.1.109 / acp-kernel 0.0.69) — the forwarded body is not persisted #762

Closed

Description

@joshwapohlmann

Symptom

One request in a session was rejected by the upstream with

{"error":{"message":"The `reasoning_content` in the thinking mode must be passed back to the API.","type":"invalid_request_error","param":null,"code":"invalid_request_error"}}

It happened 216 ms after a compress fold, on the first request that saw the new projection. The next request of the same turn (client retry, 29 → 30 messages, fold=new, cache hit 64 %) was accepted, and the session then ran for many more turns. So this is a single-shot failure, not a wedged session.

This is the #684 signature (closed as fixed by acp-kernel #245/#247 plus the proxy gate, learn-on-400 and exit sentinels from #690), reproduced on 0.1.109 / acp-kernel 0.0.69. Your own review of #690 states the sentinels should be unreachable with the kernel gate in place, so this looks like a residual path.

Measured evidence

From $TMPDIR/bili-proxy-57852.log (session 01a09d05-5b17-738d-996a-ac7f9aad1d85, client omp 18.1.20, upstream api.deepseek.com, model deepseek-v4-flash, thinking enabled):

23:39:47.528  usage input=116041
23:39:47.557  compress requested 3 range(s)
23:39:47.612  processTurn 29 msgs
23:39:47.612  warning reasoning-pair-violated: 2 assistant tool-call message(s) lack
              reasoning_content while 4 carry it — strict-echo upstreams (DeepSeek
              thinking mode) will reject the request (#684)
23:39:48.302  ← upstream 400  (reasoning_content … must be passed back)

The client's pre-proxy body for that request is dumped at ~/.omp/logs/http-400-requests/1789342788308-2r0bbrc3az1ov.json (prompt_cache_key identifies session 01a09d05…, 424 messages, 6 assistant tool-call messages with reasoning_content: \"\").

Two observations that matter:

  • The sentinel does not discriminate. It kept firing on every later request of the same session with the identical 2 lack count (carry count grew 4 → 20) and none of those requests failed. Its test is typeof msg.reasoning_content === \"string\" && msg.reasoning_content.length > 0, so it cannot tell a blank string (accepted by DeepSeek) from an absent field (rejected).
  • The warning is chronic on this host. reasoning-pair-violated fired 38× for the affected session while it ran healthily for hours, and 26×, 96×, 101× in three other concurrent sessions, plus two pfa-… sessions. As it stands the warning cannot serve as an alarm.
  • Learn-on-400 did not fire for this session: metadata in its persisted record holds only pluginAgent, outputBudgetHighWater, effectiveContextLimit, lastTurnWindow, effectiveAbsorb, systemPromptTokens — no strictReasoningEcho.

Request

1. Persist the forwarded body when the upstream returns 4xx. The failing projection is the only datum that can explain this, and it is not retained anywhere: the client's dumps are pre-proxy, the session envelope stores the fold state rather than the rendered wire body, and the proxy logs the request only in summary form. A bounded dump (env-gated, size-capped, holding the forwarded messages array) under $TMPDIR or the instance state dir would make this class diagnosable from real data instead of requiring a token-billed replay.

2. Consider normalizing instead of warning on strict-echo paths. For a strict-echo session, an assistant message that carries tool_calls but no reasoning_content field can be sent with reasoning_content: \"\" — DeepSeek accepts the blank form. That turns the class into something the projection repairs rather than something the sentinel reports. Precedent: hermes-agent PR #15527 (referenced from openclaw#71455) injects the blank field both when creating such messages and when replaying an already-poisoned history, with a provider-detection helper scoped to DeepSeek/Kimi thinking mode.

3. Optional: if the intent stays warn-only, the sentinel could count absent rather than blank (\"reasoning_content\" in msg instead of .length > 0), which is the actual condition the provider rejects and would remove the chronic noise.

Not claimed

I could not reproduce the failure on demand — it occurred once in ~180 requests, immediately after a fold, and a replay experiment would cost real tokens. This report deliberately does not claim a root cause inside the fold; it reports the correlation (post-fold request, 216 ms), the identical #684 signature, and the missing data needed to localize it.

Activity

  1. ranxianglei commented on Sep 14, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Triage update — working from the v0.1.109 / acp-kernel 0.0.69 sources now. Three verifications, one new finding, and the plan:

    Verified against source:

    1. Sentinel description is exact. warnReasoningPairs (src/server.ts:1787) tests typeof msg.reasoning_content === "string" && msg.reasoning_content.length > 0 — blank ≡ absent, so a thinking session whose tool-call turns echo "" fires on every turn. That's your 38×/26×/96×/101× chronic noise.
    2. No failure-triggered persistence of the forwarded body. The only request-side dump today is the opt-in ACP_DUMP_BODY=1 (src/server.ts:3407), which must be armed before the incident; the non-2xx branch (src/server.ts:3613+) buffers the response body only (overflow detection + 600-char snippet log). Confirmed: the rejected request body is retained nowhere.
    3. The missing-flag mystery is solved, and it's a real gap. Learn-on-400 (strictReasoningEcho) exists only on the compress-loop re-request path (src/loop/core.ts:627-635). Your 400 was a client-originated main-path request, so the learner never ran — hence no strictReasoningEcho in the persisted metadata despite the identical signature. Closing this gap.

    New finding (candidate mechanism, deliberately not claimed as root cause): the kernel's OpenAI codec drops blank echoes on every rebuild. openaiToCore creates a core reasoning message only when reasoning_content is non-empty (acp-kernel dist/wire/index.js:276), and coreToOpenai emits the field only when non-empty (:333/:339) — so "" becomes absent through any openai→core→openai round-trip. Note the tension: if every request round-trips, your pre-fold requests should also have shipped absent fields yet were accepted for hours. That contradiction is exactly why ask #1 matters — with the rejected body on disk, the next residual gets explained byte-for-byte instead of inferred.

    Plan (all three asks + the gap):

    1. Failure-triggered dump of the exact forwarded body on upstream 4xx — env-gated (BILI_DUMP_4XX=1, size-capped via BILI_DUMP_4XX_MAX_BYTES, written to the existing ACP_DUMP_DIR/state-dir location), at both the main request path and the loop re-request path. Off by default (conversation bodies on disk, 无 GC:dumps/、raw/、launcher 临时日志、bili.log.old 永久累积 #276 precedent).
    2. Strict-echo normalization: when the session is strict-echo (learned flag OR /deepseek/i static match — the existing [bug] compress 保护区过滤在完整性调整之后挖洞,导致 assistant tool-call 消息缺 reasoning_content(DeepSeek 思考模式 400) #684 gate), inject reasoning_content: "" into any assistant tool-call message lacking the field before forwarding. Precedent: hermes-agent PR #15527 as you cite. Scoped to the chat-completions wire (the reported class); Responses/Anthropic stay warn-only — those shapes can't safely fabricate the counterpart item/thinking-block.
    3. Sentinel precision: count absent (field not a string) instead of .length > 0, so blank echoes stop firing it. With fix: address all code review findings (condense, routing, session-id, dead code) #2 in place it becomes a pure regression canary.

    PR incoming shortly.

  2. ranxianglei commented on Sep 14, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Done — all three asks are implemented in PR #763 (#763), plus one root-cause fix your evidence pointed at. Pre-flight: typecheck clean · 1476 tests pass (0 fail, 2 skipped) · build green.

    What the PR does

    1. Persist the rejected body (ask 1). New src/error-dump.ts: on upstream 4xx the exact forwarded bytes go to <ACP_DUMP_DIR|stateDir/dumps>/err-<ts>-<sid>-<status>.json. Env-gated (BILI_DUMP_4XX=1), size-capped (BILI_DUMP_4XX_MAX_BYTES, default 2 MiB + truncation marker), off by default so conversation bodies don't leak to disk uninvited (无 GC:dumps/、raw/、launcher 临时日志、bili.log.old 永久累积 #276). Armed at both failure sites — the forward() main path and the compress-loop re-request (src/loop/core.ts). Your next incident gets a byte-faithful artifact automatically.
    2. Normalize instead of warn (ask 2). New normalizeStrictEchoReasoning() in src/server.ts: when strict-echo applies (static /deepseek/i origin or learned flag), every assistant message carrying tool_calls but no reasoning_content field gets "" injected before the request leaves — the blank form DeepSeek accepts (hermes-agent PR #15527 precedent). Runs on both compression modes; it touches neither tool calls nor summary carriers. Applied in prepareOpenai before the sentinel, so a repaired body never fires its own canary.
    3. Sentinel precision (ask 3). warnReasoningPairs now counts presence, not non-emptiness. Verified your description exactly: the old test typeof rc === "string" && rc.length > 0 made blank ≡ absent, which is why it fired 38× in a healthy session. Healthy thinking sessions are now silent; a genuinely absent field still warns.

    The missing-flag mystery — solved. Your observation that the session metadata lacked strictReasoningEcho was the key clue: the learn-on-400 learner existed only in the compress loop (src/loop/core.ts) and never saw client-originated 400s on the main request path. Your 400 was a main-path request, so nothing ever set the flag — #651's reasoning-drop stayed armed and every following turn could split again. The main path now learns the flag too (400 + reasoning_content in the error body), mirroring the loop.

    Candidate mechanism, stated honestly. While tracing how an absent field reaches the wire I found the kernel codec drops blank echoes on every rebuild: acp-kernel dist/wire/index.js:266/:276 (openaiToCore materializes a core reasoning message only when non-empty) and :333/:339 (coreToOpenai emits the field only when non-empty). So reasoning_content: "" becomes absent after any round-trip — which makes the DeepSeek rejection class reachable from any history containing assistant tool-call turns, independently of the fold. I'm deliberately not claiming this as the proven trigger of your specific incident: if every request rebuilds, the same drop existed pre-fold yet those requests were accepted for hours, and explaining that gap requires the actual forwarded bodies — which is precisely what ask 1 now captures. The e2e test proves the full arc (400 → learn → next request normalized → one sentinel fire → dump persisted), but the fold-specific question stays open until a dump exists.

    One judgment call, flagged for you: the kernel blank-drop is a real wire-fidelity gap, but the proxy-side normalization fully contains the user-visible impact, and "empty string = no reasoning content" is defensible as the kernel's design. So I did not file a separate acp-kernel issue unilaterally — say the word and I'll file it there (fix would need a coordinated kernel release + pin bump first).

  3. joshwapohlmann commented on Sep 14, 2026

    @joshwapohlmann
    ContributorAuthor

    Answering your judgment call: do not open a kernel issue for the blank-drop — it already exists, with a fix and tests. ranxianglei/acp-kernel#289 (issue) and #290 (PR: 3 files, +126/-6, 7 new cases, full suite 745/745, typecheck clean, open and mergeable). It cites the same two functions you traced.

    Two things from my side that bear on #763:

    1. The rule your normalization encodes is measured, not inferred. POST https://api.deepseek.com/chat/completions, thinking enabled, one assistant turn with tool_calls answered by its own tool result: reasoning_content: "" → 200, key omitted → 400 The reasoning_content in the thinking mode must be passed back to the API., non-empty → 200. Presence is what the backend validates, "" is the correct filler, and the hermes precedent agrees. Keep fix: address all code review findings (condense, routing, session-id, dead code) #2 even once bili claude 没有自动注入native插件导致用的纯服务模式 #290 lands: the codec fix stops the codec from destroying a blank, while a fold that dropped a reasoning run entirely still ships an absent field, and that is exactly the shape fix: address all code review findings (condense, routing, session-id, dead code) #2 repairs.
    2. The fold-specific question is no longer open. It is a rendering defect in the kernel's prune.ts: collectSummaryAnchors anchors a block summary at the earliest message the block covers, and when that index falls inside an assistant run the summary renders between the run's reasoning cores and its tool call, so the call reaches the wire without its reasoning run. Filed as ranxianglei/acp-kernel#286 (symptom) and fixed by #287 (the anchor half is commit dc12b3e). That is why your sentinel fires on the first post-fold request: nothing was lost in transit, the fold created the shape. Your dump (ask 1) will show it as reasoning_content absent on exactly the assistant message whose turn the summary split.

    My local install carried the equivalent of your #2 as ensureReasoningEcho before #763 existed; once #763 is released, that local patch is redundant with yours and I will build from the release instead.

  4. ranxianglei commented on Sep 14, 2026

    @ranxianglei
    Owner

    ework接单

  5. ranxianglei commented on Sep 14, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 收到 👍 PR #763 已就绪,等你 review/merge(合并由你操作)。四项改动 + 测试都在上面那条回复里;如果 review 中有想调整的地方(比如 dump 的默认值、normalization 的触发范围),直接说,我改完补 commit。

  6. ranxianglei commented on Sep 14, 2026

    @ranxianglei
    Owner

    看上面两条补充评论

  7. ranxianglei commented on Sep 14, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Both supplements read, and all four references verified live just now:

    Consequences for #763 — no code changes needed:

    1. Judgment call accepted. Not filing a codec issue — 话说最新版本bili omp 每次打开还是完全全新的页面,只有配置的自定义模型,跟之前的描述一样 #289/bili claude 没有自动注入native插件导致用的纯服务模式 #290 already own it.
    2. Normalization stays, by design. Your measured rule (presence validated, "" is the filler, omit → 400) is precisely what fix: close the #762 strict-echo residual 400 class #763 encodes, and I agree with keeping it post-bili claude 没有自动注入native插件导致用的纯服务模式 #290: the codec fix stops destroyed blanks, while a fold that drops a reasoning run entirely still ships an absent field — different origin, same rejection shape, one repair point. That's also why the e2e test drives the split shape directly rather than through a real fold: once fix: key sessions on the clients native session id, not rotating credentials #287 lands, folds stop producing it, and a fold-driven regression test would start failing after the pin bump. Deliberately did not add one.
    3. Fold-specific question: closed, thank you. Your dump prediction is the built-in acceptance check — with BILI_DUMP_4XX=1, the next residual (if any) should show reasoning_content absent on exactly the assistant message whose turn the summary split. If a dump ever shows a different shape, that's the canary doing its job.
    4. Pin bump deliberately NOT in fix: close the #762 strict-echo residual 400 class #763. Both kernel PRs are unmerged and nothing newer than 0.0.69 is on npm, so per the cross-repo flow the pin bump follows the first kernel release carrying fix: key sessions on the clients native session id, not rotating credentials #287 (+bili claude 没有自动注入native插件导致用的纯服务模式 #290) — separate change, standard release sequence. Until then, fix: close the #762 strict-echo residual 400 class #763's normalization is what keeps sessions alive.

    #763 remains as posted and ready; merge is yours to operate. And noted — your local ensureReasoningEcho goes away once the release ships.

    中文摘要:已核实 acp-kernel #286/#287/#289/#290(两个 PR 均未合并,npm 最新仍是 0.0.69);#763 无需改动——normalization 按你的实测规则保留、pin 升级按跨仓流程等内核发版后单独做,可以合并。

  8. added a commit that references this issue on Sep 14, 2026
  9. added a commit that references this issue on Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions