Skip to content

[Bug][dsh] Turning off pluginFullResendWatermark stops x-bili-pf-seq, so x-bili-pf-tail-ids grows unbounded until it overflows Node's 16 KB header limit and every request fails with "Connection error." (0.1.216) #7367

Closed

Description

@SAXEM1997

Description

With rollout.features.pluginFullResendWatermark set to false, the proxy stops emitting the x-bili-pf-seq response header, so the dsh native client's k2 never advances. pfRequestHeaders() then keeps emitting x-bili-pf-tail-ids = every id from the frozen stamp to the end of the conversation, with no cap. Once that header block passes Node's default 16,384-byte limit the proxy drops the connection at the HTTP parser, before any request handling — so the session fails with Connection error. on every retry, for every model, permanently. k2 can only advance on a successful response, and no response can now be produced: deadlock.

Turning off an experimental fastpath should not break long sessions.

Reproduction

  1. Run bili in dsh native/plugin mode with the watermark fastpath on for a while, in a session that stays under 2,048 messages (see "Why only some sessions" below), so the client has learned a k2.
  2. Set rollout.features.pluginFullResendWatermark: false (live; no restart needed).
  3. Keep using that session. It works until roughly k2 + 440 messages, then every request fails.

Observed evidence (0.1.216, commit 6a641f378)

Client-side error, model-independent (app.attempt recorded in dsh's session log):

turn 31 step 116  retry 1..5  failure: {"message":"Connection error.","code":"TRANSPORT"}
turn 32 step 1    retry 1..5  failure: {"message":"Connection error.","code":"TRANSPORT"}
turn 33 step 1    retry 1..5  failure: {"message":"Connection error.","code":"TRANSPORT"}  (last delay 8037 ms)

bili.log at each attempt — and, crucially, no forward POST / request: line for that session at all, so the request never reached the handler:

14:18:56.090 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
14:18:56.618 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
14:18:57.560 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
...
14:24:48.746 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
14:24:49.047 [debug] [conn#30306] tcp closed reason=server-end age=349ms reqs=1

reqs=0 on the parallel connections = a connection opened but no request completed.

Across 14 rotated logs covering 11-14 → 11-20 13:20 the string Header overflow appears 0 times; all 35 occurrences today fall inside this session's retry windows.

Direct confirmation that an oversized header produces exactly this signature:

$ curl -X POST http://127.0.0.1:18787/v1/chat/completions \
    -H "x-bili-pf-tail-ids: <1200 uuids>" -d '{"model":"probe","messages":[]}'
curl: (52) Empty reply from server
→ bili.log: [conn] clientError: Parse Error: Header overflow — draining then closing

Timing fits the arithmetic exactly

  • Last successful request before the flag flip: 1,397 messages (00:40:19).
  • Session grows by 2 messages per request.
  • 1,810 / 1,812 / 1,814 messages → tail = 414 / 416 / 418 ids ≈ 15.3 / 15.4 / 15.5 KB, plus ~0.9 KB of other request headers → still < 16,384 → OK.
  • 1,816 messages → tail = 420 ids ≈ 15.5 KB + 0.9 KB → > 16,384 → HPE_HEADER_OVERFLOW, connection dropped.

The transition is between 14:18:21 (1,814 msgs, OK) and 14:18:56 (1,816 msgs, fails).

Code path

prepareOpenai only evaluates the fastpath — and therefore only calls finalizeWatermarkTurn, the sole writer of the seq header — when the feature flag is on:

if (bodyText !== void 0 && pluginMode && !isTitleGen && featureEnabled("pluginFullResendWatermark")) {
  wpFullText = bodyText;
  wpOutcome = evaluateWatermarkFastpath(...);
  ...
}
if (wpFullText !== void 0 && wpOutcome !== null && wpTurnStrip !== void 0) {
  wpResponseHeaders = finalizeWatermarkTurn(session, wpOutcome, {...});
}

finalizeWatermarkTurn is the only place that does headers[PF_SEQ_HEADER] = String(T). With the flag off, wpOutcome stays null and the header is never sent.

Client side (dist/agent/dsh-native.js):

function pfLearnStamp(headersIn, respStatus, respHeaders){
  if(respStatus!==200) return;
  const raw = respHeaders.get(SEQ_HEADER);
  if(raw===null||!/^\d+$/.test(raw)) return;   // ← never reached when the flag is off
  takeSlot(sid).t = Number(raw);
}
...
function pfRequestHeaders(incoming){
  const k2 = slot.t;
  const out = {[COUNT_HEADER]: String(k2)};
  const ids = slot.ids;
  if(ids !== void 0 && ids.length >= k2){
    out[TAIL_IDS_HEADER] = ids.slice(k2-1).map(encodeURIComponent).join(",");  // ← uncapped
  }
  return out;
}

ids.slice(k2-1) is everything since the stamp, and k2 has no way to move forward once requests start failing.

Why only some sessions

pfCaptureIds() bails out above MAX_IDS = 2048:

if(!Array.isArray(ids)||ids.length===0||ids.length>MAX_IDS){delete slot.ids;return}

So sessions with more than 2,048 messages never emit the header at all and are immune (in my deployment a 9,127-message session and a 4,830-message session kept working throughout). Any session under 2,048 messages that had learned a k2 before the flag was flipped is a time bomb: it wedges ~440 messages later.

Suggested fixes

  1. Decouple the seq stamp from the fastpath. x-bili-pf-seq is the heartbeat for the pf tail list, not fastpath-only state — emit it whenever measuredT is known (or emit k2 alone when the fold scan is skipped), even with pluginFullResendWatermark off.
  2. Cap the tail list in pfRequestHeaders(), e.g. ids.slice(Math.max(k2-1, ids.length-160)), and/or skip the header entirely when ids.length - k2 exceeds a threshold. A tail that large would miss the fold anyway.
  3. Client-side frozen-stamp detection. If k2 has not moved for N consecutive requests, drop slot.ids (or reset k2) instead of growing the header forever. Right now the only escape is restarting the host process, because tracked lives in memory.

Option 2 alone would have prevented the outage; option 1 fixes the root coupling.

Environment

  • bili version: 0.1.216 (commit 6a641f378), plugin native mode (shim loaded inside the dsh process)
  • Client: dsh 0.2.2-rc.4 (DeepSeek Harness), Node v24.22.2
  • Upstream: self-hosted new-api → SGLang; models Deepseek-Flash and deepseek-v4.1-flash-1017
  • Config: rollout.features.pluginFullResendWatermark: false, compat.dropFields: ["max_tokens","max_completion_tokens"], compress.maxContextLimit: 60%
  • Local mitigation I applied: capped the tail at 160 ids in dsh-native.js (pending restart). Happy to test a proper fix.

Activity

  1. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷

    (ai: qwen/qwen3.8-27b)

    查重: 不重复 — 已检索 open+closed issues/PRs,无同机制条目(watermark 相关在途 PR #7358/#7352/#7180/#5427 均为不同面)。

    Fixed in PR #7381 — two-part fix, both halves shipped:

    1. Client safety net (src/agent/pf-channel.ts): x-bili-pf-tail-ids is now capped at the newest 160 ids. The capped tail still ends at the true window origin; a delta wider than 160 degrades to a conservative server-side miss (the width check fails) instead of a 16 KB header overflow. 160 ids ≈ 5.8 KB for UUIDs — well under the limit. This alone would have prevented the outage.

    2. Server root fix (src/server/fastpath.ts + src/server/prepare-openai.ts): the x-bili-pf-seq stamp is now emitted via a structural scan even when pluginFullResendWatermark is off. The stamp is the client's watermark advance, not fastpath-only state — decoupling it from the gate keeps k2 advancing, which is what bounds the tail list at its natural width.

    Behavior change (disclosed per owner policy 2026-09-22): with pluginFullResendWatermark OFF, the proxy now emits x-bili-pf-seq on plugin-lane non-title-gen openai responses (old: never emitted). The client-visible x-bili-pf-tail-ids is now capped at the newest 160 ids (old: unbounded from a frozen k2). Both changes are the fix.

    Regression tests: tests/pf-channel.test.ts (tail capped at 160 when stamp freezes; narrow deltas sent whole) + tests/fastpath.test.ts (flag-off stamp emission; unscannable bodies degrade to no stamp). All targeted suites green (36+96+130 tests), npm run typecheck clean.


    结构化八问(中文):

    1. 是什么问题:关掉 pluginFullResendWatermark 后,服务端不再发 x-bili-pf-seq 戳,客户端 k2 冻结,x-bili-pf-tail-ids 从冻结点起无限增长,超过 Node 16 KB 头限制后 HTTP 解析器直接断连,会话永久卡死。
    2. 什么原因导致的:x-bili-pf-seq 戳的唯一写入点 finalizeWatermarkTurn 被 fastpath 门控完全包住——flag off 时 wpOutcome 恒为 null,戳永不发出。根因是「客户端水位推进信号」与「fastpath 命中判定」被错误耦合在同一个门控后面。
    3. 影响范围与严重程度:所有在 flag 翻转前已学到 k2 且消息数 < 2048 的 plugin 车道会话(dsh/opencode/pi 等)都是定时炸弹,约 440 条消息后必炸。p1——会话完全不可用,且无法自愈(k2 只能在成功响应上推进,而成功响应已不可能产生)。
    4. 怎么修复的:双管齐下——①服务端把戳的发射从 fastpath 门控中解耦(flag off 时走结构扫描发戳,不建 seed、不存状态);②客户端 tail 列表封顶 160 条(即使戳再冻结也不会溢出)。
    5. 是否有回归问题:flag ON 路径完全未动(finalizeWatermarkTurn 原样);flag OFF 路径新增戳发射,戳值 = 结构扫描测出的 T(与 flag ON 时 finalizeWatermarkTurn 发的值一致);客户端封顶 160 条对 flag ON 路径无影响(正常 delta 远小于 160)。无回归。
    6. 是否破坏缓存:不涉及——pf 通道是请求级状态,不触碰前缀缓存/重试/增量游标。
    7. 是否改了设计方向:不涉及——戳的语义(客户端水位推进信号)未变,只是发射点从「fastpath 命中时」改为「每次成功响应时」。
    8. 是否引入了新的配置:不涉及——无新 env/config/默认值。

    收工判定:修复完整,双半都落地,回归测试钉住,typecheck 干净。PR #7381 待 CI 绿后合并。

  2. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷

    CI 已绿:PR #7381 全部 30 项 check 通过(0 failed, 0 pending),静态门禁 + 架构文档 + 配置面守卫全绿。mergeable_state: behind(分支落后 master,非阻塞)。

    标签已打齐:

    待 owner 合并。

  3. added a commit that references this issue on Nov 21, 2026
  4. added a commit that references this issue on Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions