Repository navigation
Description
Description
With rollout.features.pluginFullResendWatermark set to false, the proxy stops emitting the x-bili-pf-seq response header, so the dsh native client's k2 never advances. pfRequestHeaders() then keeps emitting x-bili-pf-tail-ids = every id from the frozen stamp to the end of the conversation, with no cap. Once that header block passes Node's default 16,384-byte limit the proxy drops the connection at the HTTP parser, before any request handling — so the session fails with Connection error. on every retry, for every model, permanently. k2 can only advance on a successful response, and no response can now be produced: deadlock.
Turning off an experimental fastpath should not break long sessions.
Reproduction
- Run bili in dsh native/plugin mode with the watermark fastpath on for a while, in a session that stays under 2,048 messages (see "Why only some sessions" below), so the client has learned a
k2. - Set
rollout.features.pluginFullResendWatermark: false(live; no restart needed). - Keep using that session. It works until roughly
k2 + 440messages, then every request fails.
Observed evidence (0.1.216, commit 6a641f378)
Client-side error, model-independent (app.attempt recorded in dsh's session log):
turn 31 step 116 retry 1..5 failure: {"message":"Connection error.","code":"TRANSPORT"}
turn 32 step 1 retry 1..5 failure: {"message":"Connection error.","code":"TRANSPORT"}
turn 33 step 1 retry 1..5 failure: {"message":"Connection error.","code":"TRANSPORT"} (last delay 8037 ms)
bili.log at each attempt — and, crucially, no forward POST / request: line for that session at all, so the request never reached the handler:
14:18:56.090 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
14:18:56.618 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
14:18:57.560 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
...
14:24:48.746 [warn] [conn] clientError: Parse Error: Header overflow — draining then closing
14:24:49.047 [debug] [conn#30306] tcp closed reason=server-end age=349ms reqs=1
reqs=0 on the parallel connections = a connection opened but no request completed.
Across 14 rotated logs covering 11-14 → 11-20 13:20 the string Header overflow appears 0 times; all 35 occurrences today fall inside this session's retry windows.
Direct confirmation that an oversized header produces exactly this signature:
$ curl -X POST http://127.0.0.1:18787/v1/chat/completions \
-H "x-bili-pf-tail-ids: <1200 uuids>" -d '{"model":"probe","messages":[]}'
curl: (52) Empty reply from server
→ bili.log: [conn] clientError: Parse Error: Header overflow — draining then closing
Timing fits the arithmetic exactly
- Last successful request before the flag flip: 1,397 messages (00:40:19).
- Session grows by 2 messages per request.
- 1,810 / 1,812 / 1,814 messages → tail = 414 / 416 / 418 ids ≈ 15.3 / 15.4 / 15.5 KB, plus ~0.9 KB of other request headers → still < 16,384 → OK.
- 1,816 messages → tail = 420 ids ≈ 15.5 KB + 0.9 KB → > 16,384 →
HPE_HEADER_OVERFLOW, connection dropped.
The transition is between 14:18:21 (1,814 msgs, OK) and 14:18:56 (1,816 msgs, fails).
Code path
prepareOpenai only evaluates the fastpath — and therefore only calls finalizeWatermarkTurn, the sole writer of the seq header — when the feature flag is on:
if (bodyText !== void 0 && pluginMode && !isTitleGen && featureEnabled("pluginFullResendWatermark")) {
wpFullText = bodyText;
wpOutcome = evaluateWatermarkFastpath(...);
...
}
if (wpFullText !== void 0 && wpOutcome !== null && wpTurnStrip !== void 0) {
wpResponseHeaders = finalizeWatermarkTurn(session, wpOutcome, {...});
}finalizeWatermarkTurn is the only place that does headers[PF_SEQ_HEADER] = String(T). With the flag off, wpOutcome stays null and the header is never sent.
Client side (dist/agent/dsh-native.js):
function pfLearnStamp(headersIn, respStatus, respHeaders){
if(respStatus!==200) return;
const raw = respHeaders.get(SEQ_HEADER);
if(raw===null||!/^\d+$/.test(raw)) return; // ← never reached when the flag is off
takeSlot(sid).t = Number(raw);
}
...
function pfRequestHeaders(incoming){
const k2 = slot.t;
const out = {[COUNT_HEADER]: String(k2)};
const ids = slot.ids;
if(ids !== void 0 && ids.length >= k2){
out[TAIL_IDS_HEADER] = ids.slice(k2-1).map(encodeURIComponent).join(","); // ← uncapped
}
return out;
}ids.slice(k2-1) is everything since the stamp, and k2 has no way to move forward once requests start failing.
Why only some sessions
pfCaptureIds() bails out above MAX_IDS = 2048:
if(!Array.isArray(ids)||ids.length===0||ids.length>MAX_IDS){delete slot.ids;return}So sessions with more than 2,048 messages never emit the header at all and are immune (in my deployment a 9,127-message session and a 4,830-message session kept working throughout). Any session under 2,048 messages that had learned a k2 before the flag was flipped is a time bomb: it wedges ~440 messages later.
Suggested fixes
- Decouple the seq stamp from the fastpath.
x-bili-pf-seqis the heartbeat for the pf tail list, not fastpath-only state — emit it whenevermeasuredTis known (or emitk2alone when the fold scan is skipped), even withpluginFullResendWatermarkoff. - Cap the tail list in
pfRequestHeaders(), e.g.ids.slice(Math.max(k2-1, ids.length-160)), and/or skip the header entirely whenids.length - k2exceeds a threshold. A tail that large would miss the fold anyway. - Client-side frozen-stamp detection. If
k2has not moved for N consecutive requests, dropslot.ids(or resetk2) instead of growing the header forever. Right now the only escape is restarting the host process, becausetrackedlives in memory.
Option 2 alone would have prevented the outage; option 1 fixes the root coupling.
Environment
- bili version: 0.1.216 (commit
6a641f378), plugin native mode (shim loaded inside the dsh process) - Client: dsh 0.2.2-rc.4 (DeepSeek Harness), Node v24.22.2
- Upstream: self-hosted new-api → SGLang; models
Deepseek-Flashanddeepseek-v4.1-flash-1017 - Config:
rollout.features.pluginFullResendWatermark: false,compat.dropFields: ["max_tokens","max_completion_tokens"],compress.maxContextLimit: 60% - Local mitigation I applied: capped the tail at 160 ids in
dsh-native.js(pending restart). Happy to test a proper fix.
Activity
ranxianglei commentedon Nov 20, 2026
[bot] 🏷
(ai: qwen/qwen3.8-27b)
查重: 不重复 — 已检索 open+closed issues/PRs,无同机制条目(watermark 相关在途 PR #7358/#7352/#7180/#5427 均为不同面)。
Fixed in PR #7381 — two-part fix, both halves shipped:
Client safety net (
src/agent/pf-channel.ts):x-bili-pf-tail-idsis now capped at the newest 160 ids. The capped tail still ends at the true window origin; a delta wider than 160 degrades to a conservative server-side miss (the width check fails) instead of a 16 KB header overflow. 160 ids ≈ 5.8 KB for UUIDs — well under the limit. This alone would have prevented the outage.Server root fix (
src/server/fastpath.ts+src/server/prepare-openai.ts): thex-bili-pf-seqstamp is now emitted via a structural scan even whenpluginFullResendWatermarkis off. The stamp is the client's watermark advance, not fastpath-only state — decoupling it from the gate keepsk2advancing, which is what bounds the tail list at its natural width.Behavior change (disclosed per owner policy 2026-09-22): with
pluginFullResendWatermarkOFF, the proxy now emitsx-bili-pf-seqon plugin-lane non-title-gen openai responses (old: never emitted). The client-visiblex-bili-pf-tail-idsis now capped at the newest 160 ids (old: unbounded from a frozenk2). Both changes are the fix.Regression tests:
tests/pf-channel.test.ts(tail capped at 160 when stamp freezes; narrow deltas sent whole) +tests/fastpath.test.ts(flag-off stamp emission; unscannable bodies degrade to no stamp). All targeted suites green (36+96+130 tests),npm run typecheckclean.结构化八问(中文):
pluginFullResendWatermark后,服务端不再发x-bili-pf-seq戳,客户端k2冻结,x-bili-pf-tail-ids从冻结点起无限增长,超过 Node 16 KB 头限制后 HTTP 解析器直接断连,会话永久卡死。x-bili-pf-seq戳的唯一写入点finalizeWatermarkTurn被 fastpath 门控完全包住——flag off 时wpOutcome恒为 null,戳永不发出。根因是「客户端水位推进信号」与「fastpath 命中判定」被错误耦合在同一个门控后面。k2且消息数 < 2048 的 plugin 车道会话(dsh/opencode/pi 等)都是定时炸弹,约 440 条消息后必炸。p1——会话完全不可用,且无法自愈(k2 只能在成功响应上推进,而成功响应已不可能产生)。finalizeWatermarkTurn原样);flag OFF 路径新增戳发射,戳值 = 结构扫描测出的 T(与 flag ON 时finalizeWatermarkTurn发的值一致);客户端封顶 160 条对 flag ON 路径无影响(正常 delta 远小于 160)。无回归。收工判定:修复完整,双半都落地,回归测试钉住,typecheck 干净。PR #7381 待 CI 绿后合并。
ranxianglei commentedon Nov 20, 2026
[bot] 🏷
CI 已绿:PR #7381 全部 30 项 check 通过(0 failed, 0 pending),静态门禁 + 架构文档 + 配置面守卫全绿。
mergeable_state: behind(分支落后 master,非阻塞)。标签已打齐:
bug+p1+scope/kernelbug+p1+scope/kernel待 owner 合并。
Merge pull request #7381 from ranxianglei/2026-11-20_pf-tail-ids-cap-…
merge: resolve conflicts with master (#7381 pf-tail-ids-cap-and-stamp…
fix: cap x-bili-pf-tail-ids at 160 ids and decouple the pf-seq stamp …
Merge master 98fa957 into 2026-10-30_drifted-name-leak
Merge remote-tracking branch 'origin/master' into ci-5855-fix