Skip to content

[PR] fix: cap x-bili-pf-tail-ids at 2 KiB so a lagging stamp can't overflow the header (#7212) - #7219

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-11-19_pf-tail-ids-size-cap
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-11-19_pf-tail-ids-size-cap

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

#7212: x-bili-pf-tail-ids was unbounded. Its width is (T - k + 1) ids, so when the learned stamp k lags far behind the current element count T (a side request with a smaller seq overwrote k), the encoded value grew past Node's 16 KiB maxHeaderSize. The parser dropped the connection, killing the whole session — and every retry re-sent the same oversized header, so it never recovered.

Fix

pfRequestHeaders (the single client-side chokepoint every lane funnels through) now byte-caps the encoded tail-ids value at 2048. Over the budget it drops the x-bili-pf-tail-ids header and sends count only. The proxy degrades to the legacy full-parse for that turn and re-stamps the true count, so the window self-heals next turn. Slower but correct — never dead.

This is the root-cause fix at the producer (the shim that builds the header), not a mask at the proxy: the Node parser kills the connection before any proxy code runs, so a proxy-side "warn and treat as miss" is unreachable, and raising maxHeaderSize only pushes the cliff (the issue's own verdict).

Behavior change (flagged per deliverables rule 4)

  • Pathological case (the bug): old = oversized header → parser drops connection → session permanently dead (retry re-sends the same header). new = header dropped → count-only → legacy full-parse → session alive (one degraded turn, then self-heals).
  • Large legitimate turn (≈55+ new elements in one turn): old = tail-ids sent (fast path). new = if the window exceeds 2 KiB, tail-ids dropped → legacy full-parse for that turn (slower, self-heals next turn). Rare — the normal window is a few ids (well under 2 KiB) and never trips the cap.

Tests

  • tests/pf-channel.test.ts: new regression test — a lagging stamp (k=2, T=450, 449-id window ≈ 16.4 KB) drops the tail-ids header but keeps the count; a tight window (k≈T) keeps the full list.
  • Targeted suites green: pf-channel, watermark-cache-eviction, omp-msgids, msgid-rule-parity, fastpath, issue5403-watermark-fastpath.
  • typecheck + build green.

Closes #7212

…w the header

The watermark channel's tail-ids header was unbounded: its width is
(T - k + 1) ids, so when the learned stamp k lags far behind the current
element count T (a side request with a smaller seq overwrites k), the
encoded value grew past Node's 16 KiB maxHeaderSize. The parser then
drops the connection, killing the whole session — and every retry
re-sends the same oversized header, so it never recovers.

pfRequestHeaders now byte-caps the encoded value at 2048; over the
budget it drops the tail-ids header (count only). The proxy degrades to
the legacy full-parse for that turn and re-stamps the true count, so the
window self-heals next turn. Slower but correct — never dead.

Closes #7212
@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-19_pf-tail-ids-size-cap (93cb8b1)

Latest build — install this exact version

npm install -g billion-context@0.0.10877-pr.7219.794b551

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
93cb8b1 billion-context@0.0.10877-pr.7219.794b551 build
77bfdd7 billion-context@0.0.10874-pr.7219.2dc5eb5 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7219.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: cap x-bili-pf-tail-ids at 2 KiB so a lagging stamp can't overflow the header (#7212) [PR] fix: cap x-bili-pf-tail-ids at 2 KiB so a lagging stamp can't overflow the header (#7212) Nov 18, 2026
… the real size

The fixture generates 13-char ids (not 36-char uuid-shaped), so the 449-id
window is ~6.5 KB, not 16.4 KB — it exceeds the 2 KiB cap this test pins but
does NOT trip Node's 16 KiB maxHeaderSize. The production overflow was the same
shape at a much wider lag (16+ KiB). Comment-only; no behavior change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: not a duplicate. #7219 is the sole fix PR for #7212. I searched open+closed issues/PRs for pf-tail-ids, maxHeaderSize, and "16 KiB header" — no other item targets this same defect.

Review

I pulled the PR branch (2026-11-19_pf-tail-ids-size-cap @ 77bfdd722a) and read the diff segment by segment, independently verifying the load-bearing claims rather than taking the description on faith:

Direction (①) — root-cause layer, correct. Node's HTTP parser drops the connection at the 16 KiB maxHeaderSize limit before any proxy code runs, so a proxy-side "warn + treat as miss" is unreachable, and raising maxHeaderSize only pushes the cliff (the issue's own verdict). The producer is the shim that builds the header; the fix belongs at pfRequestHeaders, the single client-side chokepoint. ✓

Regression (②) — existing behavior on the touched path:

  • Chokepoint is single: pfRequestHeaders is the one emit shared by all 3 send sites in native-intercept.ts (routed/self/rewrite); the server side (fastpath.ts:246) only reads, never builds. One change covers every lane. ✓
  • Normal path is byte-identical: when tail.length <= 2048 the header is emitted exactly as before; only the over-budget pathological case drops it. ✓
  • Self-heal verified end-to-end: dropped header ⇒ count mode ⇒ pre-v4 ladder ⇒ miss ⇒ legacy full-parse; and finalizeWatermarkTurn (fastpath.ts:474) re-stamps x-bili-pf-seq = T on both hit (line 494) and miss (line 511), so the client's pfLearnStamp learns the true T and the window converges next turn. ✓
  • No kernel change (host-side src/agent/pf-channel.ts, not kernel/); no config-surface change (2048 is a hardcoded safety cap, not a tunable knob).

Code (③) — correct. encodeURIComponent output is pure ASCII ⇒ .length == byte length, so the cap check is sound; 2048 sits far under 16 KiB, leaving room for the other headers in the block.

Tests (④) — the new regression test has teeth. k=2/T=450 pathological window asserts "tail-ids dropped, count kept"; a tight window (k≈T) asserts "full list kept"; it self-checks the fixture genuinely exceeds the cap. Targeted run locally: 16/16 green.

Drive-by fix (1, commit 93cb8b1543): the test comment claimed the fixture was "36-char ids / ≈16.4 KB / exactly the shape that tripped Node's 16 KiB" — but it actually generates 13-char ids, 449 of them ≈ 6.5 KB, which exceeds the 2 KiB cap but does NOT trip the 16 KiB parser limit. The original wording would mislead a reader into thinking the fixture reproduces the production connection-drop shape. Corrected to match the code (comment-only, no behavior change).

CI / state: the PR is 29 commits behind master (no conflicts, mergeable: True); I did not rebase the owner's branch. CI will re-run on my pushed 93cb8b1543.

结构化八问

  1. 是什么问题:x-bili-pf-tail-ids 头无上限,宽度 = (T−k+1) 个 id;当学到的戳 k 远落后于当前元素数 T(一次小 seq 的旁路请求覆盖了 k)时,编码值超过 Node 16 KiB maxHeaderSize,解析器断连,整段会话永久死掉(重试重发同样的大头,救不回)。
  2. 什么原因导致的:产生者(shim)构建头时未对编码后的值做字节上限;宽度随 k 落后程度无界增长,撞穿解析器硬限。
  3. 影响范围与严重程度:病态情形(戳严重滞后)下会话完全不可用且不可恢复 = p1 可用性缺陷;正常窗口(几个 id)不受影响。
  4. 怎么修复的:在唯一收口点 pfRequestHeaders 对编码值做 2 KiB 字节上限,超预算则丢 tail-ids 头只发 count,proxy 降级 legacy 全量解析并回盖真计数,下一轮自愈——修在产生者而非 proxy 侧 mask。
  5. 是否有回归问题:无。正常路径字节不变;病态路径从「会话死」变为「一轮降级 + 自愈」,严格是改进;高危模式(语义漂移 / fail-fast 变吞错 / 阈值·时序·输出格式变化)逐项过——无。
  6. 是否破坏缓存:不。丢头那轮走 legacy 全量解析,产出同样的 canonical cores,前缀缓存不受影响;下一轮恢复快路径。
  7. 是否改了设计方向:否。完全在既有 watermark 通道设计(KDD docs: env vars primary config method + cleanup #15)内,只是让产生者在病态情形选择既有的 count 模式形状。
  8. 是否引入了新的配置:否。2048 是硬编码安全常量,非配置旋钮。

Verdict: mergeable (pending CI green on 93cb8b1543). Root-cause fix at the producer, teeth in the test, docs en/zh in sync, behavior change flagged per deliverables rule 4.

@ranxianglei

Copy link
Copy Markdown
Owner Author

Closing as superseded by #7381.

Both fix the same root cause (unbounded x-bili-pf-tail-ids overflowing Node's 16 KB header limit — #7212 / #7367), and they edit the same pfRequestHeaders lines, so only one can land. #7381 is strictly more complete: in addition to bounding the tail it decouples the x-bili-pf-seq stamp from the fastpath gate (stampOnlyWatermarkTurn), which is what actually self-heals the client's frozen k2 on the #7367 flag-off path. This PR's byte-cap-and-drop approach prevents the crash but leaves the watermark frozen there.

The one idea worth carrying over is the byte-based budget — if #7381's 160-id cap is tightened to also respect a byte limit, that piece transfers. Otherwise redundant. Reopen if you'd rather carry this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant