Skip to content

[PR] fix: count masked mid-history sys/dev items in responses estimates (#3713) - #3717

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-18_responses-sysdev-estimate
Oct 20, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-18_responses-sysdev-estimate

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & Why — #3713

On the Responses wire, plugin/codex requests mask every non-leading role: system|developer item in place (__bili_inplace_sysdev, #1638) while projecting to core, so those items enter neither projection.msgs nor projection.systemParts. prepare-responses built three estimates from exactly those two projections:

Meanwhile estimateWireOverhead — the preflight base multiplied by k̂ — counts ALL sys/dev items in input[]. Every usage report therefore settled an inflated sample (billed ÷ under-counted denominator), and k̂ drifted toward the accumulation ratio as mid-history developer notifications grew (field: 2.85× on DSH native plugin / Codex wire). raw × k̂ then crossed the context limit and preflight rejected near-window turns with a non-retryable local 502.

Fix

Count the masked items back into all three estimates with the SAME extraction + chars/4 method estimateWireOverhead applies to the wire, extracted into a shared helper responsesSysDevItemText in src/server/budget.ts so both sides provably measure the same set (estimateWireOverhead's responses branch now maps through it — pure extraction refactor, byte-identical, pinned by tests/wire-overhead.test.ts).

Behavior change (old → new)

  • Responses plugin/codex lane only. lastLocalTextEstimate, localInputEstimate and systemPromptTokens now INCLUDE the mid-history sys/dev token mass. Old: omitted (incident turn: ~110k recorded vs ~347k actually billed). New: included. Why: the items are provider-billed and counted by the preflight base; omitting them from the denominator is what inflated k̂.
  • Downstream: k̂ samples drop to the true billing ratio (~0.94 in the incident instead of 2.6–2.85); preflight stops multiplying an inflated factor onto near-window payloads, so the false non-retryable 502 goes away. Panel surfaces (systemPromptTokens, context upper bound) become MORE accurate.
  • Proxy (non-codex) mode unaffected: its mid-history sys/dev is hoisted into the front block and already counted via responsesDevContent (negative pin in the [Bug] bili 启动 OMP 后缓存命中率 82.7%,连续请求多次骤降至 4992 cached tokens #1638 dev-position suite). Anthropic/OpenAI/Google lanes unchanged.

Tests

  • New tests/issue3713-responses-sysdev-estimate.test.ts — two pins driven through the real proxy in plugin mode:
    1. denominator delta: appending one large mid-history developer item raises BOTH lastLocalTextEstimate and systemPromptTokens by that item's token cost. Mutation-checked pre-fix: delta=36, expected ≈ 6559.
    2. calibration stability: upstream bills chars/4 of the received wire ⇒ k̂ stays near 1 (ring spread ≤ 1.35×) across 4 turns of accumulating mid-history notifications; fails pre-fix.
  • Targeted suites green locally (49 tests): responses-plugin-dev-position, wire-overhead, issue1933-preflight-calibration, issue2122-clamp-calibration, issue3211-calibration-policy, issue2407-google-khat-lane, preflight-compress-responses; typecheck green. Full-suite verdict = remote CI.

No config surface change. No kernel change. No version bump.

Fixes #3713

@ranxianglei ranxianglei changed the title fix: count masked mid-history sys/dev items in responses estimates (#3713) [PR] fix: count masked mid-history sys/dev items in responses estimates (#3713) Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-18_responses-sysdev-estimate (585bbb5)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4384-pr.3717.56c117c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3717.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 全仓 open+closed 检索(按失败签名: k̂ 校准 / responses sys/dev 估计):本 PR 是 #3713 唯一在途修复。同家族但机制不同:#2407/#2418(k̂ 分母漏 thinking 质量,已合)、#3645/#3650(校准后 deficit 与 foldable mass 尺度不一致,仍 open,不同机制)、#2398(Responses tools 预算延后,已关)。

标签: bug, codex, dsh, scope/kernel, p2 —— 缺陷修复;现场证据点名 DSH native plugin / Codex wire;范围属上下文管理/校准核心;能用但近窗口轮次被误杀,定 p2。


审核结论(独立审读,非复述 PR 描述)

① 方向(原 issue 关): 成立。我独立核对了代码路径:#1638 的掩码循环(src/server/prepare-responses.ts:175-200)在投影期间把非头部 sys/dev item 临时改型为 __bili_inplace_sysdev,于是它们既不进 projection.msgs 也不进 projection.systemParts;而三个估计量恰好只由这两个投影构建,预检基线 estimateWireOverhead 却数完整 wire 上的全部 sys/dev item——分子分母量的不是同一集合,分母系统性偏小,k̂ 样本(billed÷denominator)向累积比漂移,raw×k̂ 越过窗口 → 伪不可重试 502。修复把被掩码项用同一提取函数 + 同一计数器加回三个估计量,是在测量点修根因,不是打补丁遮症状。掩码是临时性的(finally 恢复),出站 wire 逐字保留全部 item——我核对过,转发语义未动。

② 回归审查(逐项清点触碰路径的既有行为):

  1. 位置保留转发语义([Bug] bili 启动 OMP 后缓存命中率 82.7%,连续请求多次骤降至 4992 cached tokens #1638/[Bug] Codex Responses proxy hoists mid-history developer items into prompt head #3227):掩码循环本体零改动;tests/responses-plugin-dev-position.test.ts T4/T5 + 哨兵泄漏断言钉住 → 保持。
  2. proxy 非-codex lane:该 lane 不掩码,inplaceSysDev 为空、新增项恒为 0,三个估计量字节级不变;负向 pin 即上述 T4(proxy 保持 hoist)→ 保持。
  3. 头部 sys/dev item:不进入 inplaceSysDev(head 跳过),仍只经 systemParts/responsesDevContent 计一次,无双计。测试 1 的上界断言(delta ≤ bigTok+640)专门防双计 → 保持。
  4. estimateWireOverhead 重构:我逐行比对了旧内联闭包与新 helper——纯提取重构,逻辑字节等价;wire-overhead 套件在 CI 已过 → 保持。
  5. k̂ 环纪律(DSH: 切换模型后,preflight 在窗口远未满时提前触发压缩 #1933/[Bug] 上下文展示估算 206K,usage 返回后为 98K:希望校准估算并区分安全上界 #2117/[Bug] 虚报线路把 k̂ 学成 1.86 后,决策数反超上游自身报数 47%、长期压在压力带:到 10W 线就折一次、每次只折 1W 多 #3211):ring 键控(origin+model)、样本准入带、发布一致性、clamp、route policy 全部未动;只有分母输入变准 → 有意变更(已披露)。高危模式过一遍:无 fail-fast→吞错、无常量/阈值漂移、无输出格式变化。
  6. effectiveTokenCount 回退([Bug] 上游从不报告 usage 时 nudge 永不触发——上下文无界增长直到超窗(ChatGPT 登录态后端) #728 永不报 usage 的上游):localInputEstimate 略升 → min(est, raw) 只会更早触发 nudge,正是该估计器文档化的 fail-EARLY 契约方向,且 masked 项本就真实计费 → 更准,非回归。
  7. 派生子会话 sizing(plugin.ts:1330):systemPromptTokens 变高 → 子会话初始 contextTokens 更准 → 有意变更(面板更准,已披露)。
  8. 持久化格式:字段早已持久化,schema 未动,只是值更正确 → 保持。

③ 代码: helper 放 budget.ts 合理(两侧共用正是去重);命名达意(responsesSysDevItemText/inplaceSysDevTokens);注释讲 why(#3713 因果链);计数在 finally 恢复之后对原始内容做,顺序正确;无死代码、无魔法值、风格一致。未发现需要修的问题。

④ 测试有效性: 两个 pin 都走真实 proxy 的 plugin mode(x-bili-plugin 头),不是 mock-only:

  • pin 1(分母增量):追加一个大 mid-history developer item 后,lastLocalTextEstimate 与 systemPromptTokens 的增量必须落在 [bigTok−160, bigTok+640]——下界抓漏计、上界抓双计,双向有牙;作者已做 mutation 验证(pre-fix delta=36 vs 期望 ≈6559)。
  • pin 2(校准稳定):fake 上游按收到的 wire 精确计 chars/4(真实比率=1.0),4 轮累积通知后断言 k̂≤1.6、环 spread≤1.35、末/首≤1.3——pre-fix 末样本会漂到 ~2.5+,必挂。唯一小依赖是 250ms settle 宽限,但它先断言 lastInputTokens>0,失败会响不会静默,hermetic loopback 下可接受。
  • 负向方向由 [Bug] bili 启动 OMP 后缓存命中率 82.7%,连续请求多次骤降至 4992 cached tokens #1638 dev-position 套件兜底(见②-2)。真回归时这两个测试都抓得住。

⑤ diff 整洁度: 3 个文件(+307/−16),每一行都与本 PR 目的相关;无空白重排、无 lockfile/生成物 churn;单线性 commit,base 为最新 master(0 behind)。干净。

⑤b 代码整洁硬标准: 全项过(命名/死代码/重复/错误处理/魔法值/注释/风格/依赖/职责/测试同标准)。

结构化八问

  1. 是什么问题:Responses wire(codex/dsh plugin lane)上,历史中段的 system/developer 条目在 bili 的本地估计里"隐身",但真实计费和预检都算它们——校准系数 k̂(实际计费÷本地估计)因此虚高到现场 ~2.85×,把接近窗口的正常轮次误判超限,返回不可重试的本地 502。
  2. 什么原因导致的:三个估计量(k̂ 分母 lastLocalTextEstimate、localInputEstimate、systemPromptTokens)由"投影后视图"构建,而 [Bug] bili 启动 OMP 后缓存命中率 82.7%,连续请求多次骤降至 4992 cached tokens #1638 为位置保留在投影时掩掉了非头部 sys/dev 条目;预检基线却按完整 wire 计数。两侧量的集合不一致,分母系统性偏小——分子(计费)真实、分母(估计)漏项,比值必然向累积比漂移。
  3. 影响范围与严重程度:仅 Responses wire 的 plugin/codex 请求,且需 mid-history developer 通知持续累积(如 DSH 自动评审通知)才恶化;p2——会话可用,但近窗口轮次会被误杀,通知越多越严重;Anthropic/OpenAI/Google lane 与 proxy 非-codex lane 不受影响。
  4. 怎么修复的:把 wire 侧 sys/dev 文本提取抽成共享 helper responsesSysDevItemText(estimateWireOverhead 的 responses 分支改为映射它,纯提取重构),再用同一提取 + 同一 tokenizer(defaultCountTokens)把被掩码项加回三个估计量——使分母与预检基线可证明地量同一集合。
  5. 是否有回归问题:无。既有行为逐项清点结论见上②(位置保留转发、proxy lane、头部单计、overhead 重构等价、k̂ 环纪律、持久化格式均保持)。唯一有意变更:三个估计量现在包含 mid-history sys/dev 质量(旧:漏计,事故轮 ~110k 记录 vs ~347k 实际计费;新:计入)——已在 PR 描述披露 old→new 及原因;k̂ 样本随之落回真实计费比(事故场景 ~0.94 vs 2.6–2.85),伪 502 消失;面板数字(systemPromptTokens、context 上限)变准。
  6. 是否破坏缓存:否。不动前缀缓存内容与压缩决策阈值;计量更准,nudge 回退方向为 fail-early(更早压而非更晚压),不引入任何缓存失效。
  7. 是否改了设计方向:否。仍是"投影管压缩、wire 管计费",本次只是让两侧的测量口径一致。
  8. 是否引入了新的配置:无。无新 env/config/默认值。

验证与判定

  • 分支与 master 同步(0 behind),单 commit,CI:除 test (ubuntu-latest, 24, 1) 仍在跑外全部绿(win22×2、ubuntu22×3 四个 test shard 已过——新测试文件在其覆盖范围内;static-gates、kernel-guard、config-surface-guard、docs-sync 等全部门禁通过)。
  • 本地定向跑未执行:本 clone 无 node_modules,且按 §7.2 全量/定向裁决以远程 CI 为准,不复跑。
  • 判定:建议可合(待最后一个 CI shard 转绿确认;合并动作留给 owner)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收工小结(完整审核见上一条回复):

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 状态更正(接替上一条的"慢速分片"说法): test (ubuntu-latest, 24, 1) 已跑 57+ 分钟,而同批其他 5 个 test shard 只用了 5–12 分钟——这不是"慢",是挂住或病态慢。核查结果:

  • 卡住的步骤是 node scripts/run-ci-tests.mjs(该 shard 的唯一测试步骤),job 日志此刻取不到(BlobNotFound,运行中日志未落盘);
  • 排除了真实失败:本 head 上所有已完成的 check 全部 success/skipped;另有一条 ci-gate-plugins 的 push 事件 run 显示 failure,但它是零 job 的幽灵 run(created 后无 job 生成、completed_at 为空),不构成门禁失败;
  • ci.yml 与 run-ci-tests.mjs 均未配置 timeout-minutes,所以这个 job 会一直跑到 GitHub 默认 job 上限(6 小时)——期间 PR 的 mergeable 状态保持 blocked;
  • 能否归因到本 PR 的新测试:无法确定(没有日志定位卡在哪个文件)。已知的新测试文件已在 ubuntu-22 ×3 和 windows-22 ×2 五个 shard 里全部通过,唯一没验证的组合是 ubuntu-24/node-24。若重跑后仍卡在同一处,才值得怀疑 node-24 特有的 flake。

给 owner 的选项(CI 控制动作我不擅自执行):

  1. 在 Actions UI 对该 run 点 Re-run jobs(或只重跑这一个 job)——最省事,一次重跑同时回答"infra flake 还是确定性挂起";
  2. Cancel 该 run 再重跑——效果类似,但会把已绿的 shard 也作废重跑;
  3. 不管它,等 6 小时默认上限自动终止。

审核判定不变:代码审核结论(建议可合)基于其余 5 个绿色 test shard + 全部门禁(static-gates/kernel-guard/config-surface-guard/docs-sync/lock-sync 等)+ 我对 diff 与测试的独立审读;这个挂起的 shard 属于 CI 基础设施噪音类(§7.2),不改变对代码的判断。

…3713)

On the Responses lane, plugin/codex requests mask every non-leading
role system/developer item in place (__bili_inplace_sysdev, #1638) while
projecting to core, so those items enter neither projection.msgs nor
projection.systemParts. prepare-responses built systemPromptTokens,
localInputEstimate and lastLocalTextEstimate from exactly those two
projections, so mid-history sys/dev mass was invisible to them while
estimateWireOverhead (the preflight base multiplied by k-hat) DID bill
it: every usage report settled an inflated sample and k-hat drifted
toward the accumulation ratio (2.85x in the field), until raw x k-hat
crossed the context limit and preflight rejected near-window turns with
a non-retryable 502.

Count the masked items back into all three estimates with the same
extraction + chars/4 method estimateWireOverhead applies to the wire
(shared helper responsesSysDevItemText), so the k-hat denominator and
the preflight base measure the same set of wire items.
@ranxianglei
ranxianglei force-pushed the 2026-10-18_responses-sysdev-estimate branch from a09abbd to 585bbb5 Compare October 18, 2026 14:35
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI is fully green on the rebased head (585bbb50b, on top of master cda8c837a): all six test shards plus every guard/gate/e2e lane passed (run 39136701634 — 19 success, 1 skipped red-signal).

Notes on the one transient red along the way:

Ready for human merge.

@ranxianglei
ranxianglei merged commit 0e9ea92 into master Oct 20, 2026
100 of 138 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.640-master.0e9ea92 (rolling master tag).

npm install -g billion-context@0.0.640-master.0e9ea92  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][responses] Mid-history system/developer items omitted from lastLocalTextEstimate inflate k-hat ~2.85x and trigger false-overflow 502 (0.1.200)

1 participant