Repository navigation
[PR] fix: clamp the output budget against a safety-inflated input estimate (#5278) - #5291
Conversation
…5278) The clamp's safety margin (est * 5%) was sized from the very estimate whose error it must absorb. The heuristic estimator understates CJK-heavy payloads by ~5% — the same order as the margin — so on those sessions margin < actual error by construction and the clamped total landed at exactly window+1, which request-rebuilding upstreams reject with ContextWindowExceededError 502 (three field incidents, billed/estimated ratios 1.050011/1.050016/1.050020). Judge on a safety-inflated estimate (x1.12) before sizing both the subtraction and the margin: any true input up to 1.12x the estimate provably fits (cap path and #3071 floor path alike), and the compound reservation covers underestimates up to ~17.6%. When the inflated estimate drops below the capability floor (input alone nearly fills the window), fall back to the raw formula so the #2490/#3071 pinned regimes stay byte-identical instead of surrendering the clamp entirely. Behavior change: emitted caps tighten by up to ~12% of the input estimate (field-validated 120-18891 tokens across a 1088-sample sweep); previously- unclamped oversized requests may now clamp. No new config surface.
…oldens New tests/issue5278-clamp-safety-margin.test.ts: the three field incidents (clamped + real input <= window), the owner-validated patch arithmetic (142946/120575/175719), a systematic sweep over underestimate rates up to the safety factor, fitting-request no-churn, and byte-identity pins for the fallback regimes (#2490/#3071 shapes). Goldens updated for the intended tightening: issue453-backstop, issue3071-planning-floor-min-output (136144 -> 121024), issue3833-cross-model-output-ceiling (incident numbers no longer demonstrate ceiling binding under the inflation; rewritten with clear-margin numbers), issue2011-integer-output-budget, issue2096-clamp-headroom-window, deferred-tool-budget (derived from the documented formula instead of magic numbers).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.7320-pr.5291.d5fa3cfThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5291.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 本条是 PR #5291(分支 标签: 已核实(进行中):
拿到 CI 全绿结论后我会补最终判定回复。 |
(ai: vllm-qwen/qwen3.8-27b) 中间状态(本地验证全过,CI 在排队):
|
(ai: vllm-qwen/qwen3.8-27b) 审查结论 + 状态:代码侧 ready-to-merge(pending CI)。① diff 与 #5278 描述一致:clamp 判定改在 |
(ai: vllm-qwen/qwen3.8-27b) 更正上一条的处置计划(经实证核对后不改分支):原计划 merge origin/ci-fixes——实测三个红签名对应的测试文件: |
ranxianglei
left a comment
There was a problem hiding this comment.
审查完成(本地分支 @786c9d98,已核对完整补丁 + clamp 家族回归)。结论:代码通过,无阻塞项;CI 全绿即可人工合并。 这是对 issue #5278 修复方向的「方案 A」实现,issue 本身分析准确(算术还原、代码路径、自指涉论证均与 master 代码逐项对上),此 PR 按建议落地并补上了建议里特别提醒的回退路径。
逐项核验:
- 补丁形态最小且等价重构干净(src/server/budget.ts):旧函数体逐字抽进内部闭包
clampFor(est)(margin/#2011 floor-at-decision-point/#3833 ceilingBound/#3071 floorOut 全部保留原位注释),新逻辑只有两行语义:clampFor(ceil(est × 1.12))有值即返回;否则回退clampFor(est)。没有顺手改动任何其它行为。✓ - 安全路证明成立:cap 路与 #3071 floor 路的输出都以放大后的基准界定——floor 路
floorOut ≤ window − safeEst、cap 路cap = window − safeEst − margin(safeEst)⇒ 真实输入 ≤ 1.12×est 时总量恒 ≤ window;叠加 5% margin 后复合预订 1.12×1.05 覆盖至 ~17.6% 低估(实测 5% 的 3.5 倍余量)。我手工推演过两条绑定路径与小输入侧(MIN_MARGIN 2048 兜底段)均闭合。✓ - 回退语义正确且无新风险:仅当放大基准把 cap 压到
OUTPUT_CLAMP_FLOOR(1024) 以下、而原始公式仍会钳制的近墙区(#2490/#3071 pinned ledger 形态)才走原始公式——这些形态与 master 逐字节相同,不是新增溢出风险(master 今天在同区域就是这个行为);其余所有安全路 no-op 因单调性必然蕴含原始路同 no-op(更松的估计只抬高各项),注释把这句话说清楚了。✓ 净效果:本 PR 对任何请求的输出要么更安全、要么与现状逐字节一致,不存在变差的分支。 - 常数而非配置面:
OUTPUT_CLAMP_ESTIMATE_SAFETY = 1.12是模块常量——加固不变量的量不该做成调参旋钮,判断正确。✓ - 测试质量高(tests/issue5278-clamp-safety-margin.test.ts):三次现场事故对钉精确值(142946/120575/175719,来自 issue 里 bundle 抽真实函数的验证算术);系统性性质扫描 u∈{0, .03, .05, .1, .12} × est 1K–220K 全部 ≤ window;装得下的请求零扰动;回退区 #2490/#3071/真投降四个形状逐字节钉死。六个受影响 golden 的改法是从文档公式推导断言值(不再硬编码魔数),且 #453 backstop 的不变式升级为含低估的强形式(
capped + ceil(130000×1.05) <= window)——这是借修复机会把回归网织得更密,值得肯定。
非阻塞备注:
- 分支基于较旧的 master(落后 tip 较多),GitHub 判 MERGEABLE、变更面只限 clamp 域 8 个文件,语义冲突风险低,CI 会按合入态跑终验。
- 合并后效果可用 #5278 的影响面三条自查:CJK 长会话 + max_tokens 超窗口 + 无新鲜 usage 基线场景下,
output budget clamped日志的钳制值会比旧版收紧最多约 12%×估算值;上游 262145 型 502 应消失。若线上仍见同类 502,说明实际低估率 >17.6%,届时再议 estimator 本身的升级(issue 方案 C)。
本地验证(正式全量留给远程 CI):
- 定向批 219/219 绿:issue5278 新套件 + issue453-backstop / issue3071 / issue3833 / issue2011 / issue2096 / deferred-tool-budget / issue2490-ledger-floor-doomed-probe / issue2122-clamp-calibration / issue2558-clamp-image / issue3148-upstream-circuit / issue4185 / fix-488-multimodal / lane-decisions-b-class / lane-decisions-matrix / side-request-isolation
tsc --noEmitexit 0- 无 kernel/配置/env/持久化面变化 ✓
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.964-master.8acc980 # this exact build
npm install -g billion-context@master # always the newest merged build
|
What's broken
The #453 output-budget clamp (
clampOutputBudget,src/server/budget.ts) must guaranteeinput + output <= windowon request-rebuilding upstreams. Its safety margin ismax(2048, ceil(est x 0.05))— sized FROM the input estimate, while its job is to absorb the ESTIMATOR's error. Self-referential.The heuristic estimator (
defaultCountTokens: CJK 1 token/char, rest chars/4) systematically understates CJK-heavy payloads by ~5% — the same order as the 5% margin — so on those sessionsmargin < actual errorby construction. Three independent field incidents (#5278, bili 0.1.206/0.1.209, DSH desktop, window 262144, client-reported max_tokens 384000): billed/estimated ratios 1.050011 / 1.050016 / 1.050020, each clamped total = 262145 = window + 1 -> upstreamContextWindowExceededError502.Fix (option A from the issue)
OUTPUT_CLAMP_ESTIMATE_SAFETY = 1.12. The clamp judges onceil(est x 1.12)before sizing both the subtraction and the margin. Any true input up to 1.12x the estimate provably fits (cap path and [bug] 0.1.195 输出预算三路饿死仍复现:无 output 上限模型 restore 无下限(1→252、fork 原样转发 1)、#2490 planning floor 钳到 1479–2623 → 3 会话 9 次 finish=length 空轮不重试(#546/#1665/#1840/#2904/#2756 残留) #3071 floor path alike); the compound reservation1.12 x 1.05additionally covers underestimates up to ~17.6% (~3.5x the observed 5%).OUTPUT_CLAMP_FLOOR(input alone nearly fills the window), re-run on the raw estimate. This keeps the dsh压缩与bili压缩冲突 #2490/[bug] 0.1.195 输出预算三路饿死仍复现:无 output 上限模型 restore 无下限(1→252、fork 原样转发 1)、#2490 planning floor 钳到 1479–2623 → 3 会话 9 次 finish=length 空轮不重试(#546/#1665/#1840/#2904/#2756 残留) #3071 pinned ledger regimes byte-identical instead of surrendering the clamp entirely (which would forward the oversized request unchanged — strictly worse). Every other safe-path no-op implies the same no-op on the raw path, sosafe ?? clampFor(raw)is exact.Behavior change (old -> new)
Verification
tests/issue5278-clamp-safety-margin.test.ts: the three incident pairs (clamped + real input <= window), owner-validated patch arithmetic, systematic sweep over underestimate rates 0–12%, fitting-request no-churn, byte-identity pins for the fallback regimes.outputBudgetHighWaterrestored as-is into GLM request → upstream 400 invalidmax_tokensparam | [bug] DSH :跨模型的 outputBudgetHighWater 被原样还原进 GLM 请求 → 上游 400 max_tokens参数非法:限制数值范围[1,131072](restore 与 clamp 都不受"已知输出上限"约束) #3833/bili 0.1.181:clampOutputBudget 把小数写进 max_tokens,上游 400 max_tokens: Input should be a valid integer / 422 Endpoint is unavailable #2011/clampOutgoingOutput sizes max_tokens against the raw native window while preflight enforces the reserve-adjusted one — up to 25% of slack the clamp never sees #2096/Output clamp starves DSH compression despite ample actual context #2122/输出预算裁剪把图像token计了两遍,usage级基线已含图像,ctx.imageTokens又被加一次 #2558/[bug] 上游在会话中途不可用时,bili 把一次故障放大成四重代价:固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次(最狠 65536→1360)、紧急收缩与 nudge 停摆均不可见 #3148/413 Deadlock: When an authenticated session exceeds the reservation window with zero collapsibles, the #3146 forwarding | 413 死锁:带身份会话超预留窗口且零可折叠时,#3146 转发门因 anonymous 条件不触发,会话从首条消息起永久不可用 #4185/多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488/lane-decisions/side-request-isolation), typecheck clean, build clean. Full-suite verdict: CI on this head.Fixes #5278