Skip to content

fix: account for deferred Responses tools in context budgets (#2391) - #2398

Merged
ranxianglei merged 2 commits into
ranxianglei:masterfrom
zhou1736948757-cpu:2026-10-08_fix-responses-deferred-tool-budget
Oct 8, 2026
Merged

ranxianglei merged 2 commits into
ranxianglei:masterfrom
zhou1736948757-cpu:2026-10-08_fix-responses-deferred-tool-budget

Conversation

@zhou1736948757-cpu

Copy link
Copy Markdown
Contributor

Summary

Fixes #2391.

Count Responses tool-search catalogs according to what is loaded into model context. Previously the estimator charged the entire JSON catalog, including function schemas marked defer_loading: true. A large discoverable catalog could exceed the fixed input budget by itself and produce a preflight overflow with zero compressible ranges.

  • Use a budget-only projection when a top-level tool_search is present and a valid flat Responses function explicitly has defer_loading: true. Keep a small discovery stub, namespace descriptions, eager members, and unknown/malformed shapes.
  • Count loaded additional_tools, tool_search_call, tool_search_output, and mcp_list_tools records in full, including definitions that retain the deferred flag.
  • Apply the same accounting to wire overhead, Responses metadata/local estimates, and output headroom. Loaded estimates enter calibration before comparison with measured usage; actual usage is never reduced or charged twice.
  • Log numeric catalog/visible/loaded/instruction estimates without dumping definitions or request contents.

The forwarded catalog and input item order are preserved. No context-window increase, new configuration, dependency, version bump, or kernel change.

Regression coverage

Seven new tests cover deferred/eager catalogs, mixed namespaces without flag inheritance, conservative handling of unknown formats, loaded definitions, usage provenance/calibration, output headroom, and a real proxy forwarding to a local fake upstream.

The proxy fixture keeps a 272,000-token total window with a 68,000-token output allocation. Its 1.2-million-character deferred schema is forwarded intact; eager and loaded versions are rejected before reaching the upstream. Small loaded records retain their contents and positions after request rebuilding.

Verification

  • TypeScript typecheck: passed.
  • npm run build: passed; distribution import annotations passed.
  • Unused-export gate and git diff --check: passed.
  • Focused budget/panel/headroom regressions: 27 passed, including the seven new tests.
  • Real installed Codex CLI 0.160.0, isolated configuration and local fake upstream: overflow/compression and under-window control 2 passed; no real model calls.
  • Full suite: 4,928 passed, 23 skipped, 2 failed out of 4,953. This is not a fully green local run. Windows ambient proxy detection was isolated in test processes only; no system proxy settings were changed. The DNS failure test expects ENOTFOUND|EAI_AGAIN, but this host returns a connect timeout; the same failure reproduces on unmodified base f2b84b6. The other failure was an intermittent Windows MCP child-process exit 3221226505: observed during the full run and one isolated branch run, while the isolated baseline check and a subsequent branch check passed. Its cause remains unproven; canonical CI is still needed.

The reporter also confirmed that the deployed compatibility patch resolved the original old-conversation failure while retaining the 272k limit. The exact original request was not captured, so this does not assert that every byte of the earlier overhead estimate has been attributed.

Related work

#2393 adds an actionable fixed-overhead diagnostic. This PR corrects deferred/loaded tool accounting and complements that diagnostic. Both touch src/server/budget.ts, so merge order may require a small rebase.

Reference: https://developers.openai.com/api/docs/guides/tools-tool-search

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 开始审查本 PR(方向关 → 回归关 → 代码关 → 测试关 → diff 整洁度)。先核对原 issue #2391、与 #2393 的关系及 CI 状态,随后按结构化八问给出结论。

…w nit ranxianglei#2398)

The function returns a chars/4 TOKEN total of loaded tool records, not an
item count; the old name misleads readers of a brand-new exported API.
Rename only — no behavior change. All call sites are inside PR ranxianglei#2398's own
diff.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 审查完成(方向关 → 回归关 → 代码关 → 测试关 → diff 整洁度),按结构化八问给结论。

方向关(原 issue #2391):#2391 要求排查「陈旧/重复的指令快照或工具 schema 序列化是否膨胀固定开销」。本 PR 解决其中工具 schema 侧机制——OpenAI tool-search(defer_loading)下,延迟 schema 在被加载前并不进入模型上下文,而估算器却按完整 JSON 目录计费;大目录单独即可超预算,产生 preflight 假溢出 + 零可压缩范围(#2391 的症状族)。rollout 里观察到的指令侧膨胀(121 个 dev items、~97K tokens)是另一机制,由 #2393 的诊断与后续清理覆盖;PR 正文诚实声明「不主张逐字节归因」,边界切分得当。报告者确认部署补丁在保留 272K 上限下解决了原始故障(自报,与代码证据一致)。

八问:

  1. 是什么问题:Responses 线的上下文估算器把整个 JSON 工具目录计入固定开销,包括尚未加载进模型上下文的 defer_loading: true schema;大目录单独就能超输入预算 → preflight 假溢出(零可压缩范围)、输出配额被饿死。
  2. 什么原因导致的:estimateWireOverhead / countSystemAndToolsTokens / estimateInputTokens 以 JSON.stringify(全量 tools) 计固定开销,与 tool-search 计费语义(deferred = 加载前不计费)不符。分层:症状 = 「preflight 溢出 + 0 range」;机制 = 「固定开销估计 ≠ 实际计费」;根因 = 「估算器未实现 tool-search 可见性语义」。修在会计决策点,不是放宽阈值或过滤载荷。
  3. 影响范围与严重程度:仅影响带顶层 tool_search 且含显式 defer_loading: true 合法扁平 function 的 Responses 会话(典型为 Codex Desktop / 大 MCP 目录);此类会话 preflight、output clamp、面板、k̂ 校准全部高估输入,严重到会话死锁([Bug][Codex Responses] fixed instruction/tool overhead exceeds 272k budget; preflight has zero compressible ranges #2391 事件即属此类)。其余协议与非 tool-search 的 Responses 行为字节级不变(projection 退化为恒等映射)。
  4. 怎么修复的:budget-only projection modelVisibleTools(永不转发):deferred function 收缩为 {type,name,defer_loading} 存根;namespace 描述、eager 成员、tool_search 本身保留;未知/畸形形状保守全计(不继承 flag、字符串 flag、嵌套 chat-completions 形状均保守保留,测试钉住)。已加载记录(additional_tools/tool_search_call/tool_search_output/mcp_list_tools)由 countLoadedToolTokens 全量计一次——这些记录经 kernel/src/wire/responses.ts 的 OPAQUE/default 分支进入 kernel preamble、位于 fold space 之外,与 estimateCoreMessages 无交集,不可能双计。统一应用于 preflight overhead、面板 systemPromptTokens、localInputEstimate/lastLocalTextEstimate(k̂ 对)、output clamp。loaded 估计先入校准再与实测 usage 取 max——usage 保持权威,从不被扣减或双计。日志仅数值(不 dump 定义)。
  5. 是否有回归问题:逐项审过——anthropic / openai-chat / google 线:投影恒等(那些 wire 无 tool_search 类型,嵌套 function 形状不满足扁平谓词,有测试钉住);非 tool-search responses:不变;nudge/preflight 消息质量侧:不变;wire 字节:不变(真代理测试 deepEqual 断言转发保真 + loaded 记录位置保留);usage 路径语义:不变(精确值断言钉住 clamp=62_000)。方向性变化仅一处:受影响会话的 output clamp 变松(旧版因高估输入而饿死输出配额)——有意变更,见下方披露。本地定向 52/52 通过(含 7 个新测试)+ typecheck 干净;CI 25/26 绿(仅 windows-latest(22) 仍在跑,无红;作者自报的两个本地失败——DNS 错误码差异、Windows MCP 子进程退出——CI 迄今未复现)。无 fail-fast→吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:否。头部 system 字节不变,prefix cache 不受影响;去重键/游标/重试策略未动。
  7. 是否改了设计方向:无——既有估算架构内的纯会计修正。与 [PR] fix: fail fast with actionable breakdown when fixed overhead alone exceeds budget (#2391) #2393 的交互:没有本 PR,[PR] fix: fail fast with actionable breakdown when fixed overhead alone exceeds budget (#2391) #2393 的 fail-fast 诊断会把 deferred-catalog 会话误报为「不可恢复的固定开销地板」。建议先合 fix: account for deferred Responses tools in context budgets (#2391) #2398,再让 [PR] fix: fail fast with actionable breakdown when fixed overhead alone exceeds budget (#2391) #2393 rebase(共同文件仅 src/server/budget.ts,小 rebase);两者合并后分别覆盖「假开销(本 PR)+ 真地板诊断([PR] fix: fail fast with actionable breakdown when fixed overhead alone exceeds budget (#2391) #2393)」。
  8. 是否引入新配置:无(无 env/config/默认值)。新增一条 info 日志行,仅在 catalog≥50K tokens 或 visible≠catalog 或 loaded>0 时触发,纯数值。

行为变更披露(旧 → 新,均为有意,仅限 tool_search+defer_loading 会话):

  • preflight/输入估计:全量目录 JSON → 可见投影 + loaded 记录(显著下降,消除假溢出);
  • 面板 SysPrompt 行(metadata.systemPromptTokens):同步下降(面板准确性);
  • output clamp(max_output_tokens):允许输出变大(旧版被高估输入饿死配额);
  • k̂ 学习对(localInputEstimate/lastLocalTextEstimate):同步下降,k̂ 向真实计费比收敛。

审查发现(已处置):一个小命名问题——新导出的 countLoadedToolItems 返回的是 token 总量(chars/4)而非条目数。按「小问题直接修」已在你的分支推送独立标注 commit 3e7cbe090(更名 countLoadedToolTokens;纯更名零行为变化,本地 7/7 测试 + typecheck 通过)。CI 正在新 head 上重跑。

测试有效性:7 个新测试断言均有牙——精确值断言钉住「usage 权威 / 校准顺序」语义,畸形形状保守处理逐例枚举,真代理 e2e 钉住 wire 保真与转发前拦截;端口 listen(0) + 超时,确定性达标。一个小的覆盖缺口:metadata.systemPromptTokens/localInputEstimate 的 prepare 层接线无直接单测断言(经单元函数 + 代理行为间接覆盖),可接受。

diff 整洁度:3 文件、单提交(+ 更名提交),每一行都与目的相关,无无关改动、无空白噪音。

判定:可合并 —— 待最后一个 CI job(windows-latest 22)完成及更名 commit 的重跑转绿后,请人工合并。方向正确、修在根因,wire 保真与双计费核查均通过,diff 干净。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Closing status update for #2398

CI is now fully green on the current head 3e7cbe090 (the rename-nit commit): 26/26 checks passed, including the Windows test matrix (node 22/24) that was still running when I wrote my review. The original head 19a018542 is likewise fully green (26/26) — so neither of the two local failures you reported (DNS test expecting ENOTFOUND|EAI_AGAIN but getting a connect timeout, and the intermittent Windows MCP child-process exit 3221226505) reproduced in canonical CI; both look host-environment-specific, consistent with your own baseline reproduction of the DNS one.

Final state:

Nothing else outstanding from my side. Ready for human merge: #2398

@ranxianglei
ranxianglei merged commit a00b28f into ranxianglei:master Oct 8, 2026
26 checks passed
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.153-master.a00b28f (rolling master tag).

npm install -g billion-context@0.0.153-master.a00b28f  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Codex Responses] fixed instruction/tool overhead exceeds 272k budget; preflight has zero compressible ranges

2 participants