Skip to content

[PR] fix: egress fuse and preflight doom gates price the prompt mass, not the whole body (#7013) - #7016

Open
ranxianglei wants to merge 2 commits into
masterfrom
2026-11-17_7013-fuse-prompt-mass
Open

ranxianglei wants to merge 2 commits into
masterfrom
2026-11-17_7013-fuse-prompt-mass

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

⚠️ 行为变更(旧→新 + 原因)

fuse/doom 判据从「整包 body token 量」改为「prompt 字段 token 量」:

问题

estimateRawBodyBreakdown() 把整个 HTTP body 的 token 量都计入 egress window fuse 和 preflight doom 判据,包括非 prompt 的上传字段(如 DSH 的 dsh_session_log 会话日志遥测)。当该字段膨胀到 8 MiB(~1.53M tok)时,即使真实 prompt 只有 ~100K tok,请求也被 413 拒绝,重试永远无效。

根因

estimateRawBodyBreakdown 解析整个 body 时,把所有顶层键的字符串叶子都计入 total。但只有 wire-format 的 prompt 字段(messages/tools/system 等)才真正消耗上游窗口;非 prompt 的上传字段不参与 LLM 推理。把非 prompt 载荷计入 prompt 预算,导致 fuse 判据失真。

修法

estimateRawBodyBreakdown 新增 promptKeys 参数和 prompt 字段:单次遍历同时累计「wire-format prompt 字段的 token 量」。PROMPT_BODY_KEYS 按协议映射 prompt 顶层键:

  • openai: {messages, tools}(system 在 messages 内)
  • anthropic: {messages, tools, system}
  • google: {contents, systemInstruction, tools}
  • responses: {input, instructions, tools}

窗口判据(egress fuse、preflight 两道 doom、side-request guard)改用 prompt 量;total 不变(= estimateRawBodyTokens 不变量保持)。不是硬编码排除 dsh_session_log——是原则性地只计入上游 tokenizer 真正计数的字段,下个插件注册别的非 prompt 字段也不会复发。

保持 total(非窗口判据)的消费方:

测试

  • raw-body-walk-depth.test.ts:各协议 prompt 量 + 巨大非 prompt 字段不膨胀 prompt 量。
  • side-request-isolation.test.ts:guard 用 prompt 量(非整包)。
  • issue4079-egress-window-fuse.test.ts:双向——巨大上传 + 小 prompt 转发(复现 issue);真超窗 prompt 仍 413(不过度抑制)。
  • issue4185-identified-fresh-forward.test.ts:doom 措辞更新。

本地定向测试全绿(raw-body-walk-depth 50/50、side-request-isolation、issue4079 6/6、issue4185 9/9、preflight/overflow 67/67);typecheck 通过。全量回归交 CI。

Closes #7013

@ranxianglei ranxianglei added bug Something isn't working p2 labels Nov 17, 2026
@ranxianglei ranxianglei changed the title fix: egress fuse and preflight doom gates price the prompt mass, not the whole body (#7013) [PR] fix: egress fuse and preflight doom gates price the prompt mass, not the whole body (#7013) Nov 17, 2026
@ranxianglei
ranxianglei force-pushed the 2026-11-17_7013-fuse-prompt-mass branch from 73e5682 to 19abd00 Compare November 17, 2026 07:18
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-17_7013-fuse-prompt-mass (8cd3690)

Latest build — install this exact version

npm install -g billion-context@0.0.10616-pr.7016.cd7f679

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
8cd3690 billion-context@0.0.10616-pr.7016.cd7f679 build
ff258d0 billion-context@0.0.10595-pr.7016.6b8dbce build
89e4b31 billion-context@0.0.10587-pr.7016.c140035 build
19abd00 billion-context@0.0.10573-pr.7016.4e3fbcf build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7016.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 无其他 PR 修同一计量缺陷。相关但不同: #6984(在 egress-fuse breakdown 里命名超大顶层字段)、#6877(命名 Responses input 桶)都是 breakdown 诊断/命名改进,本 PR 是判据计价(prompt 量 vs 整包量),机制不同;两者都触碰 measuredBreakdownClause 周边,合入时需留意文本叠加。

审核结论: 代码正确,范围干净,可合(待 CI 全绿)

验证过程(全部在 rebase 后的 head 上)

⚠️ MODEL-FACING TEXT CHANGE(413 措辞)

413 消息从 "outbound body measures ~N TEXT tokens … the body ITSELF exceeds the window" 改为 "outbound prompt measures ~N tokens … the prompt ITSELF exceeds the window",并新增 "non-prompt fields in the other bucket are excluded from the window budget — #7013"。这是模型/客户端可见的 413 诊断文案,PR 已在「行为变更」节披露;消费链推理:文案里不再有整包量,模型不会再被「整包超窗」误导去删 prompt 内容——措辞方向正确。

结构化八问

  1. 是什么问题:egress fuse 和 preflight doom 把整包 body 的 token 量(含非 prompt 上传字段)当作「请求能否装进上游窗口」的判据;DSH 的 dsh_session_log 膨胀到 8 MiB(~1.53M tok)时,真实 prompt 只有 ~100K tok 也被 413,重试永远无效。
  2. 什么原因导致的:estimateRawBodyBreakdown 把所有顶层字符串叶子计入 total,而窗口判据直接用了 total——但只有 wire-format prompt 字段(messages/tools/system 等)消耗上游窗口;非 prompt 字段是网络/负载开销,不参与窗口预算。判据用错了计量对象。
  3. 影响范围与严重程度:触发条件 = 非 prompt 上传字段膨胀 + prompt 本身装得下;受影响请求完全不可用(413 不可重试),DSH 车道实锤([Bug] estimateRawBodyBreakdown counts non-prompt extension fields toward the fuse budget, causing legitimate requests to | [Bug] estimateRawBodyBreakdown 将非 prompt 扩展字段计入 fuse 预算,导致合法请求被 413 误杀 #7013)。属可用性缺陷,对受影响客户端是 p1 级;未受影响的请求(无非 prompt 大字段)行为不变。
  4. 怎么修复的:按协议 PROMPT_BODY_KEYS 映射只计上游 tokenizer 真正计数的 prompt 字段,三处窗口判据改用 prompt 量;total 不变量保留给非窗口消费方。原则性修复,非逐字段打补丁。
  5. 是否有回归问题:无。逐项过:双向测试钉住(真超窗 prompt 仍 413 / 巨大上传+小 prompt 转发);total 消费方不变;wire 形状未变;estimateRawBodyTokens 不变量保持;413 措辞是 model-facing 变更(已披露,方向正确)。master 上触碰同文件的两个 commit([Bug] Inconsistent system prompt separator in Responses internal re-request breaks cache prefix | [Bug] Responses 内部重请求的系统提示分隔符不一致,破坏缓存前缀 #6922 relay.ts round-2 system joiner、[Design][dsh] Stop guessing dsh's auxiliary requests — the in-process plugin can declare each request's purpose explicitly #6880 side-request.ts isSideRequest)都在不同函数,无语义交互。
  6. 是否破坏缓存:不涉及。逐请求计量,不碰缓存/重试/增量游标/去重键。
  7. 是否改了设计方向:不涉及。仅修正窗口判据的计量语义(计 prompt 而非整包),不改架构/边界/数据流。
  8. 是否引入了新的配置:不涉及。PROMPT_BODY_KEYS 是硬编码协议映射,无新 env/config/默认值。

CI 状态(实时核查)

判定: 代码审核通过,可合;等 89e4b31b41 的 CI 全绿即无阻塞。

…the whole body (#7013)

estimateRawBodyBreakdown now also reports the wire-format's prompt mass
(messages/tools/system per protocol), and the window-fit gates (egress
fuse, preflight doom tiers, side-request guard) use it instead of the
whole-body total. A non-prompt upload field (e.g. DSH's dsh_session_log,
telemetry that rides the wire but consumes no window) no longer dooms a
fitting prompt.

- preflight: estimateRawBodyBreakdown gains a promptKeys param and a
  `prompt` field on RawBodyBreakdown; PROMPT_BODY_KEYS maps each wire
  protocol to its prompt top-level keys.
- server: measuredOutbound now requires protocol and returns the prompt
  mass; the two preflight doom tiers use outboundBreakdown.prompt.
- relay: the egress fuse call passes prepared.protocol.
- side-request: the guard estimates the prompt mass, not the whole body.
- 413 messages reworded body→prompt, noting the other bucket is excluded.

Consumers kept as whole-body total (not window-fit):
- handle.ts rawInputTokens (#1082 GC rebuild-cost signal).
- threads-lane synthetic prompt-only shapes.

Tests: raw-body-walk-depth (prompt mass per protocol + a giant non-prompt
field excluded), side-request-isolation (guard uses prompt mass),
issue4079 egress fuse (both-ways: giant upload + small prompt forwards;
over-window prompt still dooms), issue4185 (doom wording updated).

Closes #7013
@ranxianglei
ranxianglei force-pushed the 2026-11-17_7013-fuse-prompt-mass branch from 89e4b31 to ff258d0 Compare November 17, 2026 08:45
)

The preflight doom messages now say 'the prompt ITSELF exceeds the
window' (prompt mass, not whole body). The zero-fold calibration test
asserted the old 'body ITSELF' wording; update it to the new wording.

Closes #7013
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI fixed — final head 8cd3690a98 is fully green.

What I did this run:

  1. Re-verified live state first. The wake reported 3 red test shards on head ff258d0070. Live check showed an earlier CI wake had already pushed the fix as the final head 8cd3690a98 ("test: align zero-fold doom wording with the prompt-mass rewording" — no [ework:ci-held] marker). I verified the fix is correct: tests/preflight-zero-fold-calibration.test.ts now asserts "prompt ITSELF exceeds the window", matching the new doom wording at src/server.ts:4547, src/server.ts:4651, src/server/relay.ts:1047.
  2. Found the real blocker: the final head's CI run was cancelled. Run 44052160999 (head 8cd3690a98): all 13 guard jobs succeeded, but the test matrix (12 shards) + static-gates + mysql-contract started at 10:04:48Z and were all cancelled at 10:06:12Z — ~84 seconds later, simultaneously, i.e. a manual run cancellation, not a code failure. No comment on the PR explains it.
  3. Re-triggered the cancelled jobs via rerun-failed-jobs on the same run (no new push, no force-push — this completes the branch's one full CI pass on the exact final head).
  4. Verified the result: run 44052160999 is now completed / success — 27 success, 1 skipped (red-signal), 0 failures. The three originally-red shards are all green: test (windows-latest, 22, 2), test (ubuntu-latest, 22, 3), test (ubuntu-latest, 24, 3).

Notes: the mirror check cache was stale this run (frozen ~1h), so live state was verified with single GitHub API fetches (no polling). One-line conclusion also posted to #4599 per CI coordination policy.

Verdict: no CI blockers remain on the final head — ready for human merge.


结构化八问

  1. 是什么问题:本 PR 的 3 个 test 分片 CI 红;修复 commit 推上去后,最终 head 的 CI run 又被整体取消,test 矩阵从未跑完,PR 一直无绿。
  2. 什么原因导致的:红 = PR 把 413 doom 措辞从 "body ITSELF" 改为 "prompt ITSELF",tests/preflight-zero-fold-calibration.test.ts 两处断言仍钉旧措辞(前一轮 CI 工程师已修,8cd3690a98)。取消 = 人工 cancel run(16 个 job 在 10:06:12Z 同一秒全灭,非代码失败),PR 内无说明。
  3. 影响范围与严重程度:仅本 PR 的 CI 状态;对用户行为零影响(措辞变更已在 PR 披露)。严重度:阻塞合并,非阻塞使用。
  4. 怎么修复的:代码修复已由前一轮完成(2 行测试措辞对齐);本轮只做机械动作——rerun-failed-jobs 重触发被取消的 job,让最终 head 完成唯一一次全量 CI,无新 push。
  5. 是否有回归问题:无。全量 CI 绿(13 guards + 12 test 分片 + static-gates×2 + mysql-contract);修复本身只动测试断言措辞,与源码新措辞逐字一致。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

All green — no work needed. Re-verified live this run: branch head is 8cd3690a98 (the wording-alignment fix, final head), and its check runs are 40 success / 16 skipped / 0 failed / 0 cancelled (the 3 shards named in this event — test (windows-latest, 22, 2), test (ubuntu-latest, 22, 3), test (ubuntu-latest, 24, 3) — are all green on the re-triggered run 44052160999). This event still references the superseded head ff258d0070; details in my previous comment. Ready for human merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 root-cause

Projects

None yet

1 participant