Repository navigation
[PR] fix: egress fuse and preflight doom gates price the prompt mass, not the whole body (#7013) - #7016
[PR] fix: egress fuse and preflight doom gates price the prompt mass, not the whole body (#7013)#7016ranxianglei wants to merge 2 commits into
Conversation
73e5682 to
19abd00
Compare
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10616-pr.7016.cd7f679This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr7016.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 无其他 PR 修同一计量缺陷。相关但不同: #6984(在 egress-fuse breakdown 里命名超大顶层字段)、#6877(命名 Responses input 桶)都是 breakdown 诊断/命名改进,本 PR 是判据计价(prompt 量 vs 整包量),机制不同;两者都触碰 审核结论: 代码正确,范围干净,可合(待 CI 全绿)验证过程(全部在 rebase 后的 head 上)
|
…the whole body (#7013) estimateRawBodyBreakdown now also reports the wire-format's prompt mass (messages/tools/system per protocol), and the window-fit gates (egress fuse, preflight doom tiers, side-request guard) use it instead of the whole-body total. A non-prompt upload field (e.g. DSH's dsh_session_log, telemetry that rides the wire but consumes no window) no longer dooms a fitting prompt. - preflight: estimateRawBodyBreakdown gains a promptKeys param and a `prompt` field on RawBodyBreakdown; PROMPT_BODY_KEYS maps each wire protocol to its prompt top-level keys. - server: measuredOutbound now requires protocol and returns the prompt mass; the two preflight doom tiers use outboundBreakdown.prompt. - relay: the egress fuse call passes prepared.protocol. - side-request: the guard estimates the prompt mass, not the whole body. - 413 messages reworded body→prompt, noting the other bucket is excluded. Consumers kept as whole-body total (not window-fit): - handle.ts rawInputTokens (#1082 GC rebuild-cost signal). - threads-lane synthetic prompt-only shapes. Tests: raw-body-walk-depth (prompt mass per protocol + a giant non-prompt field excluded), side-request-isolation (guard uses prompt mass), issue4079 egress fuse (both-ways: giant upload + small prompt forwards; over-window prompt still dooms), issue4185 (doom wording updated). Closes #7013
89e4b31 to
ff258d0
Compare
) The preflight doom messages now say 'the prompt ITSELF exceeds the window' (prompt mass, not whole body). The zero-fold calibration test asserted the old 'body ITSELF' wording; update it to the new wording. Closes #7013
(ai: vllm-qwen/qwen3.8-27b) CI fixed — final head What I did this run:
Notes: the mirror check cache was stale this run (frozen ~1h), so live state was verified with single GitHub API fetches (no polling). One-line conclusion also posted to #4599 per CI coordination policy. Verdict: no CI blockers remain on the final head — ready for human merge. 结构化八问
|
(ai: vllm-qwen/qwen3.8-27b) All green — no work needed. Re-verified live this run: branch head is |
fuse/doom 判据从「整包 body token 量」改为「prompt 字段 token 量」:
measuredOutbound返回breakdown.total(含非 prompt 字段)→ 大非 prompt 上传字段 + 小 prompt 的请求被 413 误杀([Bug] estimateRawBodyBreakdown counts non-prompt extension fields toward the fuse budget, causing legitimate requests to | [Bug] estimateRawBodyBreakdown 将非 prompt 扩展字段计入 fuse 预算,导致合法请求被 413 误杀 #7013 的 DSHdsh_session_log8 MiB 场景)。measuredOutbound返回breakdown.prompt(仅 wire-format 的 prompt 字段)→ 非 prompt 上传字段不再 doom 一个能装下的 prompt。问题
estimateRawBodyBreakdown()把整个 HTTP body 的 token 量都计入 egress window fuse 和 preflight doom 判据,包括非 prompt 的上传字段(如 DSH 的dsh_session_log会话日志遥测)。当该字段膨胀到 8 MiB(~1.53M tok)时,即使真实 prompt 只有 ~100K tok,请求也被 413 拒绝,重试永远无效。根因
estimateRawBodyBreakdown解析整个 body 时,把所有顶层键的字符串叶子都计入total。但只有 wire-format 的 prompt 字段(messages/tools/system等)才真正消耗上游窗口;非 prompt 的上传字段不参与 LLM 推理。把非 prompt 载荷计入 prompt 预算,导致 fuse 判据失真。修法
estimateRawBodyBreakdown新增promptKeys参数和prompt字段:单次遍历同时累计「wire-format prompt 字段的 token 量」。PROMPT_BODY_KEYS按协议映射 prompt 顶层键:{messages, tools}(system 在 messages 内){messages, tools, system}{contents, systemInstruction, tools}{input, instructions, tools}窗口判据(egress fuse、preflight 两道 doom、side-request guard)改用
prompt量;total不变(=estimateRawBodyTokens不变量保持)。不是硬编码排除dsh_session_log——是原则性地只计入上游 tokenizer 真正计数的字段,下个插件注册别的非 prompt 字段也不会复发。保持
total(非窗口判据)的消费方:handle.tsrawInputTokens([Feature] Default state file cleanup strategy #1082 GC 重建成本信号,统计整包 wire 量)。threads-lane合成 prompt-only 形状。测试
raw-body-walk-depth.test.ts:各协议 prompt 量 + 巨大非 prompt 字段不膨胀 prompt 量。side-request-isolation.test.ts:guard 用 prompt 量(非整包)。issue4079-egress-window-fuse.test.ts:双向——巨大上传 + 小 prompt 转发(复现 issue);真超窗 prompt 仍 413(不过度抑制)。issue4185-identified-fresh-forward.test.ts:doom 措辞更新。本地定向测试全绿(raw-body-walk-depth 50/50、side-request-isolation、issue4079 6/6、issue4185 9/9、preflight/overflow 67/67);typecheck 通过。全量回归交 CI。
Closes #7013