Skip to content

[PR] fix: stop the single-threaded proxy from self-killing and full-parsing every turn - #5856

Closed
jeffcwj wants to merge 3 commits into
ranxianglei:masterfrom
jeffcwj:fix/event-loop-block-2026-11-07
Closed

jeffcwj wants to merge 3 commits into
ranxianglei:masterfrom
jeffcwj:fix/event-loop-block-2026-11-07

Conversation

@jeffcwj

@jeffcwj jeffcwj commented Nov 7, 2026

Copy link
Copy Markdown

What this fixes

A single dsh-lane proxy was dying every 12-28 minutes and, between deaths, stalling every
session for 3-15 minutes. All three symptoms trace to the same property: the proxy is one
single-threaded process that every session and every model share.

Measured on the reporting machine (bili 0.1.212, Windows/node 24, 6900-message sessions):

  • main event loop silent for 303s (hard threshold 300s) -> process.exit(1), 2-3 times a
    night. After the exit the respawn frequently failed (bootstrap failed ... code 1), traffic
    degraded to uncompressed direct, oversized bodies were refused locally, and the host had to
    be restarted by hand.
  • six sessions finishing inside the same two seconds with local=99162ms / 99467ms / 99528ms / 99611ms / 99709ms / 107474ms / 107691ms - queueing behind one worker, not seven slow turns.
  • one turn spending 207392ms in preflight to save ~3149 tokens, then local=236991ms.
  • every turn logging watermark fastpath miss (no-stamp) - legacy full-parse this turn, i.e.
    the incremental path was never taken on the plugin lane.

The three commits

1. fix: bound preflight wall-clock and make the watchdog hard threshold tunable

The 300 s watchdog timeout was a hard-coded constant with exactly one call site that did not
pass it. It is now BILI_WATCHDOG_HARD_TIMEOUT_MS, default unchanged (300 s).
BILI_PREFLIGHT_MAX_ROUNDS / BILI_PREFLIGHT_EMERGENCY_ROUND_CAP expose the round caps,
defaults unchanged.

Defaults stay identical; operators who legitimately run 700k-token sessions can now give the
loop enough rope instead of being killed mid-work.

2. fix(fastpath): self-held watermark digest so lanes without client stamp still hit

evaluateWatermarkFastpath trusts a client echo of the T it stamped last turn
(x-bili-pf-count). Hosts that never echo it - dsh's plugin lane among them - fell through to
legacy full-parse on every turn, which is the O(messages) scan that starves the loop.

This commit lets the proxy hold the stamp itself: pf:state now also stores a sha256 over the
message element bytes covered by t. On the next turn the proxy hashes the same prefix and,
when the digest matches, takes the fast path exactly as if the client had echoed t. The digest
is strictly stronger than the count echo it replaces (an equal-width in-place edit keeps the
count but breaks the digest), so no trust is widened; a mismatch costs one hash and falls back
to legacy, which re-stamps.

Two follow-on fixes were needed to make the lane actually reach the fast path:

  • readSaved rebuilds the state through a whitelist - it now forwards p, otherwise the digest
    is unreadable and every turn is no-stamp again.
  • multimodal detection inspected the whole history, so any session that ever contained an image
    was demoted to seeded-multimodal forever. It now inspects only the trailing user run
    (tailRunHasImage).

Result on the reporting machine: 53 fast-path hits / 5 misses (misses are the first turn after a
proxy restart, where the seed is in-memory only), and local for a 6898-message / 27.6 MiB turn
dropped from 5775ms to 1568 / 2264 / 2524ms.

3. fix(handle): write a rescued output budget through to the wire body

restoreOutputBudget (#546) only mutates the parsed object. Lanes that forward bodyBuffer
verbatim - the plugin lane's own normalized body, the #1117 passthrough, the #920 bypass - then
re-emit the client's starved value, so the log says restored 1 -> 384000 while the upstream
still receives max_tokens=1, and the turn ends at max-output with zero visible text (#2904).

The rescue now re-serializes bodyBuffer/bodyText only when the value actually changed, so
nothing is paid by turns that did not need a rescue.

Evidence: a 6921-message turn logged restored 1 -> 384000, outbound=1.0MiB,
[degenerate-turn] plugin-passthrough-openai: turn ended length (max-output) with zero visible text, and never reached the rebuilt prepare path ([diag-budget] absent). A synthetic request
with max_tokens=1 on the rebuilt path reached the wire with 32768; after this change the same
request logs rescue written through to the wire body (1 -> 32768).

Compatibility

  • tsc --noEmit clean.
  • Commits 1 defaults are byte-identical to today's values.
  • Commit 2 only engages when a stored digest exists (i.e. after one legacy turn); existing
    sessions upgrade on their next turn.
  • Commit 3 engages only when a rescue changed the budget.

Not included

A temporary [diag-budget] log line used to prove the wire value is deliberately left out of
these commits. Local operator tuning (round caps of 4/8, watchdog of 1800 s) is likewise a
deployment choice, not part of the patch - the env knobs above make it configurable instead.

…tunable

A single-threaded proxy turns any long synchronous stretch into global slowness
for every session, and the 300s event-loop watchdog turned our worst stretch
into a proxy death that could not respawn.

Evidence from a live profile (v0.1.212, Windows, dsh lane):

  payload ~750601 tok vs window 750000 (~1.0x)
    -> raising summarization budget 16 -> 17, rounds 16 -> 17
  [preflight] round 1/17 ...
  preflight compressed 1 range(s), ~3149.86 tokens saved in 207392ms
  request: 2449 msgs, inbound=2.6MiB, outbound=2.6MiB, local=236991ms

  FATAL: main event loop silent for 303s (hard threshold 300s) - exiting so the
  spawner can respawn onto the freed port (ranxianglei#4762)
  proxy respawn gave up - model traffic goes direct (uncompressed): send-failed

While that one session folded 3149 tokens, six unrelated sessions queued
(local=99162/99467/99528/99611/99709/107474/107691ms, all finishing inside the
same 2s window), and 3s later the watchdog killed the proxy. The respawn failed,
so every session lost compression until the host was restarted.

Changes (both default-off, so behavior is unchanged unless opted in):

- BILI_PREFLIGHT_MAX_ROUNDS / BILI_PREFLIGHT_EMERGENCY_ROUND_CAP bound how many
  full kernel passes one preflight invocation may run. The emergency coverage
  bound still derives from the historical 16-round constant, so the fold-regime
  threshold does not move with the knob.

- BILI_WATCHDOG_HARD_TIMEOUT_MS lets a deployment whose worst-case pass exceeds
  300s keep the proxy alive instead of losing it mid-turn. The threshold cannot
  tell "wedged" from "still working"; the default stays 300s.

Both edits are env-gated only: no config schema, no new module surface.
Copilot AI balanced review requested due to automatic review settings November 7, 2026 06:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

restoreOutputBudget (ranxianglei#546) only mutates the parsed object. Lanes that
forward bodyBuffer verbatim - the plugin lane's own normalized body, the
ranxianglei#1117 passthrough and the ranxianglei#920 bypass - then re-emit the client's starved
value, so the log claims "restored 1 -> 384000" while the upstream still
sees max_tokens=1 and the turn ends at max-output with zero visible text
(ranxianglei#2904 / ranxianglei#546).

Capture the incoming budget and, only when the rescue actually changed it,
re-serialize bodyBuffer and bodyText so every downstream lane carries the
rescued value. Turns that did not need a rescue pay nothing.

Evidence: 1031f08f (6921 msgs, 27.7MiB) logged
  "output budget restored 1 -> 384000"
  "request: 6921 msgs, inbound=27.7MiB, outbound=1.0MiB, local=1928ms"
  "[degenerate-turn] plugin-passthrough-openai: turn ended length (max-output)
   with zero visible text and zero tool calls"
while a synthetic request on the rebuilt path reached the wire with 32768
([diag-budget] wire max_tokens=32768). After this change the same synthetic
request logs "rescue written through to the wire body (1 -> 32768)".
@jeffcwj
jeffcwj force-pushed the fix/event-loop-block-2026-11-07 branch from f18f9b1 to eac62f0 Compare November 7, 2026 06:05
@jeffcwj

jeffcwj commented Nov 7, 2026

Copy link
Copy Markdown
Author

Opened by mistake - the change is meant to stay on the fork for now. Reopening later is easy if it turns out to be wanted upstream.

@jeffcwj jeffcwj closed this Nov 7, 2026
@ranxianglei ranxianglei changed the title fix: stop the single-threaded proxy from self-killing and full-parsing every turn [PR] fix: stop the single-threaded proxy from self-killing and full-parsing every turn Nov 7, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

查重:不重复(整条线程),但第 2 项与在途的 #5813 / #5815 同症状、同根因。

状态:PR #5856 已被作者 jeffcwj 在打开 90 秒后自行关闭(未合并),留言 "Opened by mistake - the change is meant to stay on the fork for now. Reopening later is easy if it turns out to be wanted upstream." 所以本线程没有可合并对象;我做的是把他声称的三个缺陷逐一到 master(bebc239)上核验。标签本想按惯例打 bug / dsh / scope/wiring / p1,但镜像 API 的 labels 端点在本环境 shim 里未实现(endpoint not implemented in ework shim),打不上,如实报告。

核验结论(对照 master 代码逐条过):

  1. watchdog 300s 硬编码 + preflight 轮次上限——事实成立:src/event-loop-watchdog.ts:45 WATCHDOG_HARD_TIMEOUT_MS = 5 * 60_000,唯一调用点 src/server.ts:676 不传 hardTimeoutMs;src/preflight.ts 的 16 轮 / 512 紧急上限也是常量。但模块头注释明文写着 "Thresholds are deliberately hardcoded — a new config/env surface for an internal safety timer is not worth the owner-gated cost"——commit 1 新增 3 个 env 变量与该已记录的设计决定正面冲突,且按仓库纪律新 env 必须在 CONFIGURATION.md en+zh 落文档,此 PR 一行文档没动。另外这个故障域已有在途条目:Event-loop-watchdog self-exit + split-brain under 30–45 MB sessions: proxy disappears mid-request (422 oversize refusal) and two instances commit the same session #5507(watchdog self-exit + split-brain)、[Bug] 0.1.208 dsh/Windows: event loop fully blocked ~19 min after cold-load/eviction storm on 300MB+ sessions; wedged child holds port so all respawns fail (twin-collision code 3) #5138(事件循环阻塞 ~19 min)、[Bug] 0.1.209 dsh/Windows: an orphaned proxy outlives its dead owner (33 min, ~51% CPU, write lock held) and is never reaped → the replacement drifts 18789→18790 while the caller stalls 826 s with 0 bytes #5449(孤儿代理),相关 PR compress merges disjoint ranges into earlier coverage but retains only the last summary #5830/[PR] fix(kernel): reject range folds that would merge blocks beyond the requested ref window #5835 也在做。把 kill 阈值调高只是把「代理自杀」换成「全体会话多停几分钟」。
  2. dsh 车道不回带 x-bili-pf-count → 每轮 no-stamp——症状真实(他测得 3,795 次求值 0 命中;我确认 v0.1.212 服务端确有该评估门与 miss("no-stamp") 路径,fastpath 默认 5% 灰度、开了才会出现这行日志),但根因已被定位且修复在途上游:[Bug][dsh] watermark fastpath never hits: the client never sends x-bili-pf-count because pfRequestHeaders is handed the pre-merge headers object #5813(open)——客户端 pfRequestHeaders() 拿到的是合并前的头对象,x-bili-plugin-conversation 只在合并后的对象里,count 头永远发不出去;[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815(open PR)正是修这个。他的 commit 2(服务端自持 sha256 摘要)是对这个客户端缺陷的绕路方案,而且落点恰在刚被 revert([PR] Revert "[PR] feat(fastpath): v6 composite-identity incremental fastpath lane (full rework, #5576 line voided)" #5854 回滚 [PR] feat(fastpath): v6 composite-identity incremental fastpath lane (full rework, #5576 line voided) #5652)、马上要重做([PR] Re-land v6 composite-identity fastpath lane — inverse of #5854 revert (base for snapshot-centric rework, #5643) #5855 Re-land v6,open)的 fastpath 区域——并行打补丁会和主线方向正面相撞。
  3. rescued output budget 没写进线体——这条有真价值且无现有 issue 跟踪:restoreOutputBudget([bug] bili dsh 长会话中 max_output_tokens 被压到 ≤200,主请求被误判为 side request 并绕过压缩 #546)只改 parsed 对象,而 fastpath HIT 路径把 max_tokens 等标量字段从原始 bodyText 的 raw slice 重新解码(src/server/prepare-openai.ts fastpath 分支),rescue 改过的值在命中轮被原样覆盖回去——这个洞我在代码层确认了。他的 commit 3(rescue 生效时在 chokepoint 重写 bodyBuffer/bodyText)机制上能堵上这个洞,用到的 outputBudgetField/readOutputBudget 在 master 已有导出(src/server/side-request.ts:424/:438)。但 PR 描述声称修的是 "「BUG」native-intercept 进程级 globalThis.fetch 补丁按 URL 形状接管所有模型流量,无关调用者(如 MemOS)被卷入宿主压缩流程 #1117 passthrough / opencode v1: keep opencode-acp legacy sessions working after switching to bili #920 bypass"——这两条路径在 parse 之前就 early-return,根本不走 rescue,说法不成立。

三个 commit 的共同缺陷:零测试(仓库要求回归测试 + 门的双向断言)。commit 2 另把多模态降级判定从「历史任意位置有图」收窄为「尾部 user run 有图」并丢掉 seed.multimodal 传播位——安全门语义变化(宁降级不偏离),必须人审。

下一条我会发 🔶 需要决策(选项 + 八问)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
🔶 需要决策
(ai: vllm-qwen/qwen3.8-27b)

背景:外部贡献者 PR #5856(作者已自行关闭、留 fork)报告单线程代理在超大会话下的三个缺陷;核验结论——第 2 项与在途 #5813/#5815 同根因,第 1 项与文档化设计决定及在途 #5507 家族冲突,第 3 项是无人跟踪的真实洞(fastpath 命中轮把 rescued 的 max_tokens 覆盖回客户端饿死值)。

待决问题:这三项上游怎么处理(采纳 / 适配 / 拒绝 / 让作者按意见重提)。

选项(各带代价):

推荐:B;第 2 项交给 #5815 收敛,第 1 项不加 knob(真问题是 preflight 对超大历史的 wall-clock 成本与 respawn 可靠性,归 #5507 家族处理)。不决定的默认行为:A(维持现状)。


八问:

  1. 是什么问题:单线程代理在超大会话(测得 700k token / 6900 消息)上 preflight 阻塞事件循环数百秒 → watchdog 300s 自杀 → respawn 失败 → 压缩丢失直到人工重启;阻塞期间其他会话全体排队(6 个会话同秒内 local≈100s);另:dsh 车道 watermark fastpath 恒 no-stamp 每轮全量解析,rescued output budget 没写回线体。
  2. 什么原因导致的:① preflight N 轮每轮都是全历史 kernel pass,CPU-bound 占满唯一线程(机制),watchdog 阈值硬编码不可调(放大器);② 客户端 pfRequestHeaders 拿到合并前的头对象,count 头永远发不出([Bug][dsh] watermark fastpath never hits: the client never sends x-bili-pf-count because pfRequestHeaders is handed the pre-merge headers object #5813 已定位);③ restoreOutputBudget 只改 parsed,fastpath 命中路径从原始 bodyText raw slice 解码标量字段,把 rescue 结果覆盖掉。
  3. 影响范围与严重程度(按选项对应):A/C 涉及第 1 项——仅超大会话部署,核心能力不可用类(p1 级),典型用户无感;第 2 项——开了 pluginFullResendWatermark(dsh plugin 车道,默认 5% 灰度)的会话性能退化,p2;第 3 项——触发条件窄(fastpath 开 + rescue 生效 + 命中轮),后果是该轮零可见文本,体验瑕疵偏 p2。选 B 则只消除第 3 项。
  4. 怎么修复的:第 3 项 = chokepoint 写回(rescue 改值时同步重写 wire 字节,所有下游 verbatim 路径一次覆盖);第 2 项正解 = 修客户端头合并([PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815 在途),不做服务端摘要绕路;第 1 项不加配置面,归 Event-loop-watchdog self-exit + split-brain under 30–45 MB sessions: proxy disappears mid-request (422 oversize refusal) and two instances commit the same session #5507 家族治本(preflight wall-clock 成本 / respawn 可靠性)。
  5. 回归风险(按选项):A = 无回归(维持现状,洞保留);B = 仅 rescue 轮 verbatim 车道字节变化(值等价、一次缓存过渡 miss),非 rescue 轮逐字节不变;C = B 之外再加:commit 2 收窄多模态降级门(整史有图→尾部 user run 有图)+ 丢 seed.multimodal 传播位 = 安全门语义变化,且新 p 字段进 pf:state 存储格式、新增 self-stamp verdict——必须人审 + 回归测试;commit 1 改变「硬编码」这一既有设计承诺。
  6. 是否破坏缓存:B 在 rescue 轮造成一次 provider 前缀缓存过渡 miss(此后稳定);C 的摘要方案每个被评估轮多一遍全量 sha256(单线程 CPU 成本,换掉的是更贵的全量解析,实测净赚但成本存在);[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815 客户端修复零额外成本。
  7. 是否改了设计方向:推荐方案不改方向(第 3 项是接线层写回;第 2 项修客户端握手不动服务端信任模型)。若采纳 C 的 commit 2,则 fastpath 协议面扩展(pf:state 加 p 字段 + self-stamp 判据),应并入 [PR] Re-land v6 composite-identity fastpath lane — inverse of #5854 revert (base for snapshot-centric rework, #5643) #5855 重做而非独立落地。
  8. 是否引入了新的配置:PR 提出 3 个新 env(BILI_WATCHDOG_HARD_TIMEOUT_MS / BILI_PREFLIGHT_MAX_ROUNDS / BILI_PREFLIGHT_EMERGENCY_ROUND_CAP,默认不变)——不建议采纳(配置面纪律 + 未落文档);真需要须补 en/zh 文档 + 你明确批准。推荐方案(B)不引入任何新配置。

收工判定:本线程无代码交付(PR 已被作者关闭,无可合并对象);等你对 A/B/C/D 拍板,默认 A。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants