Skip to content

fix: multimodal token accounting + store:false in Responses preflight summary (#488) - #489

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-03_multimodal-image-tokens-store-false
Sep 3, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-03_multimodal-image-tokens-store-false

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Sep 3, 2026 •

Copy link
Copy Markdown
Owner

What

Issue #488: codex session with 7 screenshots → payload ~1.49M tokens vs 1.05M window → every request 502-stuck. Two root causes, both verified in code:

A. Image bytes were invisible to every payload-size decision. The wire codecs move images out of CoreMessage.text into sidecars (imageBase64/rawResponsesItem/…), so estimateCoreMessages() (text-only) undercounts image-bearing payloads up to 15× — yet the images ARE forwarded verbatim. Consequences: the preflight fit gate (result.payloadEstimate < limit) declared "fits" for an over-window multimodal payload and forwarded it anyway (upstream 400 → retry → identical → permanent loop); self-heal window learning recorded text-only sizes.

B. Preflight summary request omitted store:false for the Responses protocol. Codex relays reject it with HTTP 400 {"detail":"Store must be set to false"}, killing the rescue exactly when it is needed. Only the synthesized summary call was affected: the main forward path passes the client body through (codex sends its own store:false) and compress-loop re-requests spread {...requestBody}.

Changes

  • src/image-tokens.ts (new): host-side image share of a RAW request body. Known base64 payloads charged ceil(b64len/4) — the same chars/4 rule defaultCountTokens uses, matching how byte-counting relays actually bill (多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488 evidence: 7 screenshots ≈ 1.4M tokens rules out pixel-tile billing, which would cap at ~11K). Remote URLs we cannot size get a flat REMOTE_IMAGE_TOKENS = 4096. BILI_IMAGE_TOKEN_CAP env clamps the per-image cost for pixel-tile upstreams that bill by tiles (official Anthropic/OpenAI), where the default intentionally overestimates to trigger compression earlier rather than undercount.
  • src/preflight.ts: summaryPayload responses branch now sends store: false; new optional PreflightDeps.imageFloor added to every size decision inside the preflight loop (initial estimate, per-round floor-max, per-round result) so folds can never declare success while image bytes alone exceed the window.
  • src/server.ts: preflight trigger + fail-fast detail + self-heal window learning all add the image share of the actual forwarded body; when images alone exceed the window the fail-fast error says so explicitly ("Images alone account for ~N tokens … compression cannot remove them").
  • tests/fix-488-multimodal.test.ts: unit tests for the estimator (all three protocols, data URLs, remote URLs, multi-image sums, raw-body parse gate, cap clamp) + two e2e regressions through the real proxy: (B) preflight summary requests carry store:false and the rescued session forwards 200; (A) 7 × 15K-token images vs a 10K window → withheld 502 preflight_compress_failed with the actionable image note, nothing forwarded upstream.

Scope notes

  • Both compression modes covered per AGENTS.md §6: preflight runs unconditionally (plugin mode included); the store:false fix applies to the proxy-synthesized summary call in either mode; image accounting is mode-independent (it measures the forwarded body).
  • No acp-kernel change — counting happens host-side from the raw wire body, which also avoids the kernel's pre-existing lossiness for remote-URL images (its allImageParts only captures data: URLs).
  • Text-only sessions are byte-for-byte unaffected (estimator returns 0 without image markers; no JSON.parse even runs — string sniff gate).

Pre-flight (on rebased branch f7deccc)

  • npm run typecheck ✅
  • npm test 984/986 — the 2 fails (plugin-agent.test.ts) fail identically on pristine master in this sandbox; environment-dependent, unrelated to this change
  • npm run build ✅

Open question for @Rika-xie before merge: the default billing model (b64 length / 4) matches the byte-counting relay in the report; if your upstream bills by pixel tiles, set BILI_IMAGE_TOKEN_CAP accordingly.

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-03_multimodal-image-tokens-store-false (f7deccc)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-489

Each push to this PR publishes a new version under the pr-489 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr489.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Review of PR #488 in progress. Per AGENTS.md triage rules I'm doing: (1) verify both root causes actually exist on master with code evidence, (2) layer analysis (root cause vs symptom), (3) evaluate whether the fix hits the core, including both compression modes (§6). I'll run typecheck/test/build myself and post a full verdict here.

@ranxianglei
ranxianglei force-pushed the 2026-09-03_multimodal-image-tokens-store-false branch from acd9dce to f7deccc Compare September 3, 2026 08:35
@ranxianglei

Copy link
Copy Markdown
Owner Author

[bot] 🏷 Review complete — verdict: fix is correct; branch was stale, now rebased + extended.

What the review found

  1. Branch was stale. It was cut before fix: land #454 (omp native compaction disable) on master — it was merged into its already-merged base branch #483/fix: abort-path usage drop, upstream-timer leak, and bare AbortError logs (#411) #485/fix: plugin mode gets the fake-completion detect+retry when enabled (#473) #486/fix: restart restores savedAt as lastSeen instead of fabricating boot-time activity (#404) #487 landed on master. Rebased cleanly onto current master (zero conflicts); force-pushed as f7deccc.
  2. Same-class gap in newer code. Master's plugin mode: no hard backstop when the agent ignores the compress nudge — context grows past the window and upstream 400s #453 output-budget clamp (estimateInputTokens / clampOutgoingOutput, added after this branch was cut) had the exact same blind spot: image bytes invisible to the input estimate → cap too generous → input+output could still overflow on multimodal sessions. Extended the fix: clampOutgoingOutput now takes imageTokens (counted from the rebuilt outgoing body — the same source preflight uses) at both call sites (OpenAI + Responses). New e2e regression A2 pins it: text-only request keeps its 5k output budget; the same request plus one 4k-token screenshot gets clamped below 5k.
  3. Separate pre-existing gap noted (not fixed, out of scope): the Responses output clamp does not count the injected compress developer prompt (~1.9K tokens) — processedMessages is computed before the dev-message injection and ctx.systemText excludes it. Affects turn-1 estimates of all compress-injected Responses sessions (lastInputTokens dominates from turn 2 onward). Bounded impact; flagging for a follow-up decision rather than bundling into this fix.

Verification (on the rebased branch)

  • npm run typecheck ✅
  • npm test 984/986 — the 2 fails (plugin-agent.test.ts: "pi extension is inert without a proxy", "before_provider_headers stays silent when the manifest fetch keeps failing") fail identically on pristine master in this sandbox; environment-dependent, unrelated to this change
  • npm run build ✅
  • tests/fix-488-multimodal.test.ts 7/7 ✅ (was 6; +A2)

The open question from the description stands: default billing = b64 length/4 (matches the byte-counting relay in the report); pixel-tile upstreams should set BILI_IMAGE_TOKEN_CAP. Merge stays human-only per repo rules.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Independent review of PR #488 (branch 2026-09-03_multimodal-image-tokens-store-false, commit acd9dce) complete. I re-derived both root causes from master code myself, audited the fix design, ran the checks locally, and screened for duplicates. Verdict: both root causes verified, the fix hits the core, both compression modes covered. Two minor doc gaps to close before merge.

Root-cause verification (independent of the PR's own claims)

A. Image bytes invisible to size decisions — CONFIRMED, and the layering is right. On master, estimateCoreMessages() (src/preflight.ts) sums defaultCountTokens(m.text) only. I traced all three wire codecs in the bundled acp-kernel: decode moves images out of text into sidecars (anthropic → [image] placeholder + rawAnthropicBlock; openai → rawOpenaiContent/imageBase64/rawOpenaiContentParts; responses → rawResponsesItem + imageBase64 for data URLs only), and encode re-emits them verbatim (coreToAnthropic pushes the raw block; coreToOpenai pushes raw parts / reconstructs the data URL; coreToResponses pushes the raw item when text is unchanged). So the forwarded payload carries the full image bytes while every estimateCoreMessages-based decision (preflight trigger, fit gate, self-heal learning) sees ~text-only size. The PR's diagnosis is accurate: the failure mode is a false success ("fits") on an over-window multimodal payload, not merely a missed nudge.

B. Missing store:false — CONFIRMED, scope is exact. Master's summaryPayload() responses branch builds {model, max_output_tokens, instructions, input, stream:false} with no store field → codex relays reject the synthesized summary call with HTTP 400 {"detail":"Store must be set to false"}, killing the rescue precisely when needed. Main forward path (client body passthrough, codex sends its own store:false) and compress-loop re-requests ({...requestBody} spread) are correctly identified as unaffected.

Fix-design audit

  • Constant imageFloor is sound by construction: folds only ever remove images (summaries are text), so the original forwarded body's image share is an upper bound on any post-fold payload. It can never declare "fits" while image bytes alone exceed the window. The pessimistic edge case (old images folded away, floor still high → fail-fast when post-fold would actually fit) self-heals because preflight mutates session state every round even on failure — the next request sees the smaller payload. Correct trade against 多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488's permanent 400 loop.
  • Estimator placement on the rebuilt wire body (not kernel sidecars) is the right call: it measures exactly what gets forwarded, and sidesteps two pre-existing kernel codec gaps (below). Sniff gate means text-only bodies skip JSON.parse entirely — zero behavior change for them.
  • store:false is spec-valid for any Responses implementation (official OpenAI supports store), so low regression risk beyond codex relays.
  • Both compression modes (§6): preflight runs unconditionally (plugin mode included); the store:false change touches only the proxy-synthesized summary call, which exists in either mode; image accounting measures the forwarded body, mode-independent. ✓

Verification run (my sandbox, on the PR head)

  • npm run typecheck ✅
  • npm test ✅ 952/953 — single failure is launcher.test.ts: resolveClientCommand codex/claude. The diffstat touches no launcher file (src/image-tokens.ts, src/preflight.ts, src/server.ts, new test file only), so this is provably environment-dependent and unrelated.
  • npm run build ✅
  • New tests reviewed: 4 unit (all three protocols, data/remote URLs, multi-image sums, sniff gate incl. malformed JSON → 0, cap clamp) + 2 real-proxy e2e. Confirmed the asserted error code preflight_compress_failed is what src/server.ts:1304 emits, and e2e-A's "nothing forwarded upstream" assertion actually guards the fail-fast path. Good coverage of both root causes.
  • Duplicate screening: searched open+closed issues for multimodal/image-token/store-false topics — no prior or overlapping items.

Findings from the review

Minor (report-only, please close before merge):

  1. BILI_IMAGE_TOKEN_CAP is a user-facing knob but absent from the env tables in CONFIGURATION.md (~L340–365) and CONFIGURATION.zh-CN.md (~L345). Since the PR's open question hinges on it, the doc line should ship with the change.
  2. No CHANGELOG.md [Unreleased] entry — repo convention is that content PRs carry theirs (see recent fixes bc57a09, 5e35adb, 380e323).
  3. Known limitation, no action required now: the Anthropic scan covers top-level messages[].content[] parts only; images nested inside tool_result.content arrays are forwarded but not counted (rare shape; same conservative-direction argument applies since the fit gate also floors on the trigger-side count).

Major problem found during analysis → filed separately: the two acp-kernel codec gaps the PR works around are silent data-loss bugs in the kernel itself — openaiToCore drops a single non-data-URL image (no sidecar chosen → encode emits text only), and responsesToCore drops image-only user items (empty effText → no core message). Filed with repros and suggested fixes as ranxianglei/acp-kernel#187 (source-tagged to this work). This is not a defect of PR #488 — host-side raw-body counting is deliberately robust to both (overcounts conservatively vs. what the kernel forwards) — but worth tracking.

Open question (default billing model)

Recommendation stands: keep chars/4 as the default. The #488 evidence (7 screenshots ≈ 1.4M tokens; pixel-tile billing would cap at ~11K) proves byte-counting relays exist in the wild, and undercounting there reproduces exactly the permanent 400 loop this PR fixes. Overcounting on tile-billing upstreams (official Anthropic/OpenAI) only triggers compression earlier — the safe direction. The BILI_IMAGE_TOKEN_CAP escape hatch covers the rest; it just needs the doc line (finding 1).

Ready to merge once findings 1–2 land (or accepted as an immediate follow-up — your call). Per repo rules I won't merge it myself.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Re-review of f7deccc (rebased head) — verified independently. Verdict: the rebase is clean, the output-clamp extension is sound, and the code is ready to merge once the two minor doc items below land.

Rebase validated

My checks at f7deccc (this sandbox)

  • npm run typecheck ✅ · npm run build ✅
  • npm test: 985/986. The single failure is launcher.test.ts:22 resolveClientCommand codex/claude — /usr/bin/codex exists in my sandbox so the resolver returns the absolute path instead of the bare name; it fails identically on pristine master here and the diff touches no launcher code. (Your 984/986 with two plugin-agent.test.ts fails is the same class of environment-dependency on your machine; neither file is touched by the diff.)
  • tests/fix-488-multimodal.test.ts: 7/7 pass, including the new A2.

Clamp extension — soundness review

  • clampOutgoingOutput's ctx gains imageTokens, added to inputEstimate before clampOutputBudget (src/server.ts:1545-1577). Both call sites count it from the rebuilt outgoing body (imageTokensInParsedBody("openai", rebuilt) at :1681, "responses" at :1890) — the same "count what is actually forwarded" principle as the preflight trigger and self-heal learning. Consistent source of truth across all three size decisions.
  • Adding image tokens monotonically shrinks the cap (cap = nativeWindow − inputEstimate − margin, margin ≥ 2048 or 5%), so behavior is strictly more conservative than master; text-only sessions get imageTokens = 0 and are byte-for-byte unaffected. Mode-independent (measures the forwarded body) — covers both compression modes per §6.
  • One mechanism note (pre-existing, not introduced here): nativeWindow is captured at src/server.ts:1256 before the output-headroom reservation block, so the clamp operates against the full/native window while the kernel nudge/truncate bands see the reserved one. That's current master behavior and A2's control assertion (text-only keeps its 5k budget against the 10k window) documents exactly that. No action needed — just flagging so the interaction isn't rediscovered later.
  • A2 is well-designed: it captures max_output_tokens from the actual upstream request (not internal state), includes a text-only control, and asserts both directions of the clamp (< 5000 and ≥ 1024 floor).

Your out-of-scope flag (dev prompt ~1.9K) — verified accurate

Carried over from round 1 (still holds at f7deccc)

Still open before merge (minor)

  1. BILI_IMAGE_TOKEN_CAP is absent from CONFIGURATION.md and CONFIGURATION.zh-CN.md (grep-verified at f7deccc) — add the env-var lines alongside the other BILI_* entries.
  2. No [Unreleased] entry in CHANGELOG.md; repo convention (bc57a09, 5e35adb, 380e323) is that content PRs carry their own entry.
  3. Known limitation (non-blocking, noted for follow-up): the anthropic scan misses base64 images nested inside tool_result.content parts — the estimator undercounts that rarer shape. Small follow-up candidate, not a merge blocker.

One pre-merge recommendation: the PR touches server.ts/preflight.ts, so per AGENTS.md §3 the codex e2e regression should run before merge. Zero-token preflight (E2E_CHECK=1) passes in my sandbox, but the full ACP_TEST_E2E=1 run needs a live Responses-compatible upstream that this sandbox doesn't have — please run it locally or via the manual-dispatch ci-e2e.yml workflow before merging.

Merge stays human-only per repo rules. If items 1–2 land (or you explicitly defer them), I consider this reviewed and ready.

@ranxianglei

Copy link
Copy Markdown
Owner Author

第三方独立 review(在 f7deccc 上全部本机复跑):

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 收到第三方 review 汇总,逐点确认:

1. 全量 codex e2e 通过 — 补上了我这边缺的最后一道门槛。 这正是我在两轮 verdict 里标记的 pre-merge 建议(我的沙箱只跑通了 E2E_CHECK=1 零 token 预检,完整 run 需要 live upstream)。真实 codex 0.147.0 + 本地 sglang、ACP_TEST_E2E=1 452s 通过,AGENTS.md §3 对 server.ts/preflight.ts 改动的回归要求即满足。单测差异互证无碍:你的 984/986(plugin-agent.test.ts ×2)vs 我的 985/986(launcher.test.ts ×1),都是沙箱本地二进制存在性导致的环境依赖失败,两个文件均不在 diff 内。

2. #496 的定性成立,我修正第一轮的一个措辞。 我当时说"高估只是提前触发压缩,方向安全"——这个论断只在 fit 门表现为"提前压缩"时成立;我低估了极端情形:imageFloor 单项 ≥ 窗口时 fit 门是硬 fail-fast(502,绝不转发,src/server.ts preflight 的 imageTokens >= limit 分支),"高估"就从保守方向变成保证误杀。数字对照代码复核过:400KB JPEG → b64 ≈ 546K 字符 → 估算 ~136K tokens/张,2 张 ≈ 273K ≥ 200K 窗口 → 命中硬拦;而官方 tile 计费真实成本 ~3K,同一场景在 master 上正常转发。所以这是官方 API 多模态用户的回归向量,不是单纯的保守取舍;BILI_IMAGE_TOKEN_CAP opt-in + 错误信息引导用户删图/调窗口,进一步放大。单独立案(#496)、不阻塞本 PR——同意。

3. forward-once-then-learn 方案的两个实现观察(供 #496 实现参考,不阻塞):

  • 门条件"图片是唯一超窗成分"可落为 currentTokens − imageFloor < limit && currentTokens >= limit(文本折叠后已 fit),再叠加"无上游实证"(lastInputTokens < limit 且该 model 无 learnedContextLimits 记录)即 issue 所述。
  • 终止性依赖"拒绝被记录":字节计费 relay 首转发 400 → self-heal 写 learnedContextLimits(本 PR 已含 rejectedImageTokens,学窗路径齐备)→ 第二次请求"无实证"条件翻 false → 硬 fail-fast,一次额外转发后收敛,严格优于 master 的无限 400 循环。边界情形:若 relay 的 400 body 里没有可解析的窗口信息(inspectContextOverflow 学不到东西),"无实证"条件第二次仍为真,会反复转发反复被拒。建议 overflow 拒绝时无条件记一个 marker(哪怕没解析出窗口值),否则 "forward once" 会退化成 "forward every time"。
  • tile 计费上游成功转发后无需学习(估算保持偏高但无害,反正总是放行),行为零变化——与 issue 预期一致。

4. 合并前剩余项(与你的清单一致,补一条 rebase 注意):

结论不变:代码本身 ready;文档两项 + rebase 验证完成后即可由 owner 合并(按仓库惯例合并人工执行)。

@ranxianglei
ranxianglei merged commit 6185be7 into master Sep 3, 2026
6 checks passed
antclt pushed a commit to antclt/billion-context that referenced this pull request Sep 5, 2026
 & ranxianglei#496 changelog

Carrying PR ranxianglei#489's doc gaps noted in ranxianglei#496 (items 1 & 2): add BILI_IMAGE_TOKEN_CAP to the CONFIGURATION env tables (EN+ZH) and add [Unreleased] changelog entries for the ranxianglei#488 image-count fix and the ranxianglei#496 forward-once fix. Drop these hunks if already added to ranxianglei#489.
@ranxianglei
ranxianglei deleted the 2026-09-03_multimodal-image-tokens-store-false branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant