Repository navigation
Conversation
…tunable
A single-threaded proxy turns any long synchronous stretch into global slowness
for every session, and the 300s event-loop watchdog turned our worst stretch
into a proxy death that could not respawn.
Evidence from a live profile (v0.1.212, Windows, dsh lane):
payload ~750601 tok vs window 750000 (~1.0x)
-> raising summarization budget 16 -> 17, rounds 16 -> 17
[preflight] round 1/17 ...
preflight compressed 1 range(s), ~3149.86 tokens saved in 207392ms
request: 2449 msgs, inbound=2.6MiB, outbound=2.6MiB, local=236991ms
FATAL: main event loop silent for 303s (hard threshold 300s) - exiting so the
spawner can respawn onto the freed port (ranxianglei#4762)
proxy respawn gave up - model traffic goes direct (uncompressed): send-failed
While that one session folded 3149 tokens, six unrelated sessions queued
(local=99162/99467/99528/99611/99709/107474/107691ms, all finishing inside the
same 2s window), and 3s later the watchdog killed the proxy. The respawn failed,
so every session lost compression until the host was restarted.
Changes (both default-off, so behavior is unchanged unless opted in):
- BILI_PREFLIGHT_MAX_ROUNDS / BILI_PREFLIGHT_EMERGENCY_ROUND_CAP bound how many
full kernel passes one preflight invocation may run. The emergency coverage
bound still derives from the historical 16-round constant, so the fold-regime
threshold does not move with the knob.
- BILI_WATCHDOG_HARD_TIMEOUT_MS lets a deployment whose worst-case pass exceeds
300s keep the proxy alive instead of losing it mid-turn. The threshold cannot
tell "wedged" from "still working"; the default stays 300s.
Both edits are env-gated only: no config schema, no new module surface.
…mp still hit the incremental path
restoreOutputBudget (ranxianglei#546) only mutates the parsed object. Lanes that forward bodyBuffer verbatim - the plugin lane's own normalized body, the ranxianglei#1117 passthrough and the ranxianglei#920 bypass - then re-emit the client's starved value, so the log claims "restored 1 -> 384000" while the upstream still sees max_tokens=1 and the turn ends at max-output with zero visible text (ranxianglei#2904 / ranxianglei#546). Capture the incoming budget and, only when the rescue actually changed it, re-serialize bodyBuffer and bodyText so every downstream lane carries the rescued value. Turns that did not need a rescue pay nothing. Evidence: 1031f08f (6921 msgs, 27.7MiB) logged "output budget restored 1 -> 384000" "request: 6921 msgs, inbound=27.7MiB, outbound=1.0MiB, local=1928ms" "[degenerate-turn] plugin-passthrough-openai: turn ended length (max-output) with zero visible text and zero tool calls" while a synthetic request on the rebuilt path reached the wire with 32768 ([diag-budget] wire max_tokens=32768). After this change the same synthetic request logs "rescue written through to the wire body (1 -> 32768)".
f18f9b1 to
eac62f0
Compare
|
Opened by mistake - the change is meant to stay on the fork for now. Reopening later is easy if it turns out to be wanted upstream. |
[bot] 🏷 查重:不重复(整条线程),但第 2 项与在途的 #5813 / #5815 同症状、同根因。 状态:PR #5856 已被作者 jeffcwj 在打开 90 秒后自行关闭(未合并),留言 "Opened by mistake - the change is meant to stay on the fork for now. Reopening later is easy if it turns out to be wanted upstream." 所以本线程没有可合并对象;我做的是把他声称的三个缺陷逐一到 master(bebc239)上核验。标签本想按惯例打 bug / dsh / scope/wiring / p1,但镜像 API 的 labels 端点在本环境 shim 里未实现(endpoint not implemented in ework shim),打不上,如实报告。 核验结论(对照 master 代码逐条过):
三个 commit 的共同缺陷:零测试(仓库要求回归测试 + 门的双向断言)。commit 2 另把多模态降级判定从「历史任意位置有图」收窄为「尾部 user run 有图」并丢掉 seed.multimodal 传播位——安全门语义变化(宁降级不偏离),必须人审。 下一条我会发 🔶 需要决策(选项 + 八问)。 |
[bot] 🏷 背景:外部贡献者 PR #5856(作者已自行关闭、留 fork)报告单线程代理在超大会话下的三个缺陷;核验结论——第 2 项与在途 #5813/#5815 同根因,第 1 项与文档化设计决定及在途 #5507 家族冲突,第 3 项是无人跟踪的真实洞(fastpath 命中轮把 rescued 的 max_tokens 覆盖回客户端饿死值)。 待决问题:这三项上游怎么处理(采纳 / 适配 / 拒绝 / 让作者按意见重提)。 选项(各带代价):
推荐:B;第 2 项交给 #5815 收敛,第 1 项不加 knob(真问题是 preflight 对超大历史的 wall-clock 成本与 respawn 可靠性,归 #5507 家族处理)。不决定的默认行为:A(维持现状)。 八问:
收工判定:本线程无代码交付(PR 已被作者关闭,无可合并对象);等你对 A/B/C/D 拍板,默认 A。 |
What this fixes
A single dsh-lane proxy was dying every 12-28 minutes and, between deaths, stalling every
session for 3-15 minutes. All three symptoms trace to the same property: the proxy is one
single-threaded process that every session and every model share.
Measured on the reporting machine (bili 0.1.212, Windows/node 24, 6900-message sessions):
main event loop silent for 303s (hard threshold 300s)->process.exit(1), 2-3 times anight. After the exit the respawn frequently failed (
bootstrap failed ... code 1), trafficdegraded to uncompressed direct, oversized bodies were refused locally, and the host had to
be restarted by hand.
local=99162ms / 99467ms / 99528ms / 99611ms / 99709ms / 107474ms / 107691ms- queueing behind one worker, not seven slow turns.207392msin preflight to save~3149tokens, thenlocal=236991ms.watermark fastpath miss (no-stamp) - legacy full-parse this turn, i.e.the incremental path was never taken on the plugin lane.
The three commits
1.
fix: bound preflight wall-clock and make the watchdog hard threshold tunableThe 300 s watchdog timeout was a hard-coded constant with exactly one call site that did not
pass it. It is now
BILI_WATCHDOG_HARD_TIMEOUT_MS, default unchanged (300 s).BILI_PREFLIGHT_MAX_ROUNDS/BILI_PREFLIGHT_EMERGENCY_ROUND_CAPexpose the round caps,defaults unchanged.
Defaults stay identical; operators who legitimately run 700k-token sessions can now give the
loop enough rope instead of being killed mid-work.
2.
fix(fastpath): self-held watermark digest so lanes without client stamp still hitevaluateWatermarkFastpathtrusts a client echo of theTit stamped last turn(
x-bili-pf-count). Hosts that never echo it - dsh's plugin lane among them - fell through tolegacy full-parseon every turn, which is the O(messages) scan that starves the loop.This commit lets the proxy hold the stamp itself:
pf:statenow also stores a sha256 over themessage element bytes covered by
t. On the next turn the proxy hashes the same prefix and,when the digest matches, takes the fast path exactly as if the client had echoed
t. The digestis strictly stronger than the count echo it replaces (an equal-width in-place edit keeps the
count but breaks the digest), so no trust is widened; a mismatch costs one hash and falls back
to legacy, which re-stamps.
Two follow-on fixes were needed to make the lane actually reach the fast path:
readSavedrebuilds the state through a whitelist - it now forwardsp, otherwise the digestis unreadable and every turn is
no-stampagain.was demoted to
seeded-multimodalforever. It now inspects only the trailing user run(
tailRunHasImage).Result on the reporting machine: 53 fast-path hits / 5 misses (misses are the first turn after a
proxy restart, where the seed is in-memory only), and
localfor a 6898-message / 27.6 MiB turndropped from
5775msto1568 / 2264 / 2524ms.3.
fix(handle): write a rescued output budget through to the wire bodyrestoreOutputBudget(#546) only mutates the parsed object. Lanes that forwardbodyBufferverbatim - the plugin lane's own normalized body, the #1117 passthrough, the #920 bypass - then
re-emit the client's starved value, so the log says
restored 1 -> 384000while the upstreamstill receives
max_tokens=1, and the turn ends at max-output with zero visible text (#2904).The rescue now re-serializes
bodyBuffer/bodyTextonly when the value actually changed, sonothing is paid by turns that did not need a rescue.
Evidence: a 6921-message turn logged
restored 1 -> 384000,outbound=1.0MiB,[degenerate-turn] plugin-passthrough-openai: turn ended length (max-output) with zero visible text, and never reached the rebuilt prepare path ([diag-budget]absent). A synthetic requestwith
max_tokens=1on the rebuilt path reached the wire with32768; after this change the samerequest logs
rescue written through to the wire body (1 -> 32768).Compatibility
tsc --noEmitclean.sessions upgrade on their next turn.
Not included
A temporary
[diag-budget]log line used to prove the wire value is deliberately left out ofthese commits. Local operator tuning (round caps of 4/8, watchdog of 1800 s) is likewise a
deployment choice, not part of the patch - the env knobs above make it configurable instead.