Skip to content

[PR] fix: count mid-history sys/dev once in wire overhead instead of double-billing it (#6293) - #6474

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-11-13_responses-sysdev-overhead-double-count
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-11-13_responses-sysdev-overhead-double-count

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #6293

What

estimateWireOverhead (src/server/budget.ts) double-counted mid-history system/developer text on the Responses and OpenAI lanes. The kernel projections (responsesToCore / openaiToCore) hoist only the LEADING run of sys/dev items out of the counted core view — mid-history sys/dev items are core messages, already billed by estimateCoreMessages — but the overhead estimator walked EVERY sys/dev item in input[] / messages[] and billed the same bytes a second time. On cold-start requests with no usage baseline, core + overhead inflated the local estimate past the judging limit and tripped spurious preflight compression (reporter's 3 MB history: 1,170,334 estimated vs 705,090 true).

Fix

The overhead side now counts EXACTLY the projection's hoisted set, using the same codec the core view uses:

  • responses: responsesToCore(body).systemParts (instructions + leading sys/dev run)
  • openai: openaiToCore(body).systemText (leading contiguous prefix)

Projection failure keeps the legacy all-sys/dev walk (fail-closed overcount — the call sites degrade to an empty core on the same failure, so nothing goes unbilled). Tools / loaded-tools / other-protocol logic untouched. No threshold, window, timeout, or request-body changes. No kernel changes (the codec was already exported).

Behavior change (old → new)

  • responses bodies with mid-history sys/dev items: overhead previously billed that text a second time; now billed once (by the core side). Cold-start local estimates drop by the duplicated mass; requests that fit the window no longer trip preflight compression.
  • openai chat bodies with mid-history system messages: same fix, same shape.
  • openai leading system messages with structured (parts) content: previously silently dropped from overhead (string-only read); now counted via the projection's stringContent.
  • Bodies the codec refuses: unchanged (legacy walk).

Tests

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-13_responses-sysdev-overhead-double-count (2d3d56d)

Latest build — install this exact version

npm install -g billion-context@0.0.9560-pr.6474.a589c56

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
2d3d56d billion-context@0.0.9560-pr.6474.a589c56 build
3f38650 billion-context@0.0.9556-pr.6474.99cb3ed build
c89e75f billion-context@0.0.9401-pr.6474.fe68149 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6474.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: count mid-history sys/dev once in wire overhead instead of double-billing it (#6293) [PR] fix: count mid-history sys/dev once in wire overhead instead of double-billing it (#6293) Nov 12, 2026
@ranxianglei
ranxianglei force-pushed the 2026-11-13_responses-sysdev-overhead-double-count branch from c89e75f to 3f38650 Compare November 13, 2026 12:59
@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Nov 13, 2026
ci-check added 3 commits November 13, 2026 21:10
The fix commit referenced the internal mirror issue number, which points
at an unrelated open PR on GitHub (and the PR body's Closes line would
close that PR on merge). The bug report is GitHub issue #6291 — rename
the test file to the repo's issue<N>- convention with the right number
and repoint the src/server/budget.ts comments at it.
…s gate

The byte-equality assertion stripped the proxy-stamped ACP tag line with
a regex that REQUIRED the time="..." attribute. That attribute exists
only when messageTimestamps is on (#4313, default ON in production); the
CI suite pins it OFF via tests/preload-test-env.mjs, so the regex never
matched and the assertion failed on every CI shard while passing in
standalone local runs. Make the attribute group optional so the test is
deterministic in both regimes (AGENTS.md 7.2: no environmental luck).
@ranxianglei
ranxianglei force-pushed the 2026-11-13_responses-sysdev-overhead-double-count branch from 3f38650 to 2d3d56d Compare November 13, 2026 13:11
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — no other open/closed issue or PR fixes this double-count; the related prior work (#829 head counting, #3713 calibration denominator) is calibration, not this defect.

Review verdict: mergeable after ONE human edit

The code fix is correct, the CI red is root-caused and fixed by me, but the PR body's Closes #6293 points at the wrong GitHub thread and must be changed to Closes #6291 before merge. I cannot edit the PR body/title myself (policy), so that one line is yours.

1. CI red — root cause found, fixed, pushed (all 3 shards, one single test)

All three failing shards (ubuntu 22/24, windows 22) failed on exactly ONE test each — the PR's own new e2e #6291: fitting cold-start request forwards once. Root cause:

Fix: the time attribute group is now optional (commit 2d3d56d4b2). Verified 9/9 green in both regimes (CI preload set and standalone).

2. Wrong GitHub number (must fix before merge)

3. Code review — the fix itself is correct

Verified against the kernel codecs: responsesToCore hoists instructions (string-only) + the LEADING sys/dev run; mid-history sys/dev items stay core messages, already billed by estimateCoreMessages. openaiToCore hoists only the leading contiguous prefix. The estimator now bills EXACTLY the hoisted set via the same codec — no double-bill, and the fail-closed fallback is verified: on projection failure the call sites degrade to an empty core while the fallback walk bills all sys/dev, so nothing goes unbilled.

4. Minor findings (non-blocking)

  • Undisclosed behavior change: array-form responses instructions — the old estimator billed them; the new try-path silently ignores non-string instructions (the projection is string-only and does not throw, so no fallback). The rest of the pipeline also ignores array instructions (kernel codec + canonicalResponsesSystem are string-only), so this is a consistent alignment — but it's missing from the PR's behavior-change list.
  • Improvement (not in the list either): the openai fallback walk gained a m && null guard; the old walk would have crashed on a null element in messages[].
  • Perf note: the estimator now runs a full projection per request on the responses/openai lanes (the projection hashes every message text — ~10–30 ms CPU on a 3 MB body). Correctness trade-off, acceptable; flagging for awareness.

Housekeeping I did

  • Rebased onto current master (a007974683); typecheck clean; targeted suites green: 9/9 new file (both regimes) + 10/10 wire-overhead + 2/2 issue3713. Full suite is CI's job on head 2d3d56d4b2.
  • Labeled the PR from its own diff: bug, p2, scope/wiring.

八问(结构化审核)

  1. 是什么问题: wire overhead 估算器在 responses/openai 车道把 mid-history 的 system/developer 文本重复计费一次(core 侧已计),冷启动(无 usage 基线)时本地估算虚高,触发本不该发生的 preflight 压缩(报障者 3MB 历史:估算 1,170,334 vs 真实 705,090)。
  2. 什么原因导致的: 估算器与 kernel 投影对"谁拥有 mid-history sys/dev 字节"口径不一致——投影只把 leading run 提升到 system 侧(mid 留在 core 消息里,由 estimateCoreMessages 计费),旧估算器却遍历 input[]/messages[] 里所有 sys/dev 项再计一遍。
  3. 影响范围与严重程度: 带 mid-history sys/dev 的 responses/openai 请求,冷启动最明显(有 usage 基线时走真实计数,不受影响);p2——请求仍能发出,但本可放进窗口的会话被不必要地(有损)压缩;不丢数据、不影响可用性。
  4. 怎么修复的: overhead 侧改用与 core 视图同一个 codec 计算投影提升的那一组(responses: responsesToCore().systemParts;openai: openaiToCore().systemText),投影失败回退旧全量遍历(fail-closed:多计不漏计)。
  5. 是否有回归问题: 审过——[preflight] responses 协议 overhead 估算漏计 input[] 内的 developer/system 消息:codex 请求被系统性低估整个 system prompt #829 head 计数与 [Bug][responses] Mid-history system/developer items omitted from lastLocalTextEstimate inflate k-hat ~2.85x and trigger false-overflow 502 (0.1.200) #3713 校准分母保持绿(12/12);两处口径变化需知悉:① array 形式的 responses instructions 不再被 overhead 计费(旧代码计)——与管线其余部分一致,但 PR 行为清单未列;② openai 回退遍历新增 null 守卫(旧代码遇 null 元素会崩)——属改善。性能面:responses/openai 车道每请求多一次完整投影(3MB body 约 10–30ms CPU)——正确性代价,可接受。
  6. 是否破坏缓存: 不涉及——估算器纯本地,不改任何出站字节(e2e 测试钉住转发历史逐字不变),无缓存/重试/游标变化。
  7. 是否改了设计方向: 不涉及——估算与判定收敛到同一投影,与 KDD refactor: reapOrphanBlocks should live in acp-kernel, not the proxy #12(资格判定与执行共用单一投影)方向一致。
  8. 是否引入了新的配置: 不涉及。

判定: 可合 — 前提两条:① PR body Closes #6293 改为 Closes #6291(标题同理),否则合并会误关无关的 open PR #6293;② 新 head 2d3d56d4b2 的 CI 绿。代码本身无阻塞项。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant