Skip to content

[Bug] Gemini tool-response screenshots have zero image cost and disappear from summary notes #6146

Closed

Description

@Afloat16

Reproduction

At e1e33a9, send a Gemini functionResponse with response:{output:"Captured two views"} and two image/png inlineData entries in functionResponse.parts. The actual codec retains these parts on the tool-result core, but parsed and raw image-token/count entry points return zero. imagePlaceholdersForSummary also returns no image notes.

Expected behavior

Every forwarded tool image must contribute to image budgets/counts. Folding a tool result should retain one descriptive note and recovery reference per image. Text-only results should incur no image cost.

Verified patch

Share Google media traversal across cost, count and placeholders. Four new unit regressions fail against unchanged source; 31 targeted tests pass. A real proxy/Gemini-upstream test confirms folding, both recovery notes in the summary request and removal of base64 from the forwarded request without sending it to the summarizer. Typecheck and build pass. #2022 already stores recovery for these images; #3412 concerns other opaque-media accounting.

Activity

  1. ranxianglei commented on Nov 10, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷
    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复。 Searched open+closed issues and PRs for the same mechanism (Gemini nested tool-media accounting). Closest neighbors: #2022 (restore channel — already indexes these images, unchanged here), #3412 (opaque custom-shape media — different part shapes), #2609/#2614 (fileData treated as media payload — closed, different surface). No prior issue/PR covers "functionResponse.parts media priced 0 + no fold notes". Note: PR #6147 already targets this exact issue (opened by the reporter), so per one-fix-PR-per-thread I verified and supplemented that PR instead of opening a parallel one.

    标签: bug · gemini · scope/wiring · p2 (window-gate misestimate + missing recovery notes + possible mis-413; usable-with-workaround class, not a cache-leak/data-loss class) — applied to this thread; PR #6147 labeled bug · p2 · gemini · image on its own diff (an automated labeler had also added scope/kernel despite zero kernel diff — removed).

    Triage (three steps, done before touching code)

    1. Verified the bug exists at master HEAD. Reproduced the report exactly: a functionResponse with response:{output:"Captured two views"} plus two image/png inlineData entries in functionResponse.parts yields cost 0 / count 0 from imageTokensInParsedBody/imageTokensInRawBody/countImagesIn*, and no notes from messageImages/imagePlaceholdersForSummary. The 4 new unit tests fail 4/4 against unfixed source and pass with the fix — genuine regression teeth, not author claims taken on faith.
    2. Layering. The kernel side was already nesting-aware: the codec (kernel/src/wire/google.ts googleToCore) carries the whole functionResponse part verbatim on the tool-result core (rawGoogleParts), the restore channel (src/image-restore.ts:169) indexes nested image/* media, protected.ts treats nested inlineData as archivable/foldable, and media-refs.ts recurses for re-access handles. Only the host-side accounting surfaces missed the nesting: the google branches of src/image-tokens.ts continue over any part carrying a functionResponse (nested media unpriced → window gates, ingest-cap exemption at src/server/handle.ts:460, learned-cost attribution all see 0), and src/image-note.ts messageImages has no rawGoogleParts branch (folded screenshots leave no note). Fix belongs at those three surfaces — the producer side of the accounting — not by filtering payloads or patching the summary prompt. That is exactly where the PR puts it.
    3. Plan assessment. The PR shares ONE traversal (googleMediaParts generator, top-level parts + every functionResponse.parts) across cost, count and notes, so the three stay in lockstep by construction (the Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 discipline). Notes are gated to image/* mimes — deliberate and correct: a note carries a decompress({imageRef}) recovery ref, and the restore channel can only deliver nested image/*; billing stays mime-agnostic because every forwarded byte reaches upstream. Direction is right; no better layer exists without a kernel change (which would need the human gate and isn't needed).

    Review of PR #6147 (independent pass)

    • Rebased-head verification: cherry-picked onto current master (52 commits ahead of the PR base; zero drift in any touched file), ran npm run typecheck clean + 10 targeted suites covering every consumer of the touched functions — 106/106 pass, including the new real-proxy/Gemini-upstream integration test (fold fires, both · mNNNNN recovery notes reach the summarization request, base64 gone from the forwarded model request, summary input carries no pixels) and the Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 opaque-media suite (no double-charging from the loop refactor).
    • Test validity: assertions have teeth (exact byte/pixel arithmetic, caps, mixed top-level+nested totals that break on any double or missed charge, text-only negative gates, bidirectional e2e assertions). One gap found and fixed: the image/* note gate had only its positive direction pinned (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 requires both) — pushed supplement commit 5f3d479dbf to the PR branch adding the nested application/pdf case (billed + counted, no note) plus a stray double-blank-line cleanup. The contributor's original commit is preserved untouched; CI on the new head is green (13 success / 17 conditional-skip / 0 failed).
    • Regression inventory (touched paths): top-level inlineData/fileData billing unchanged; text/functionCall/videoMetadata carriers still free; unknown-shape opaque charging unchanged; raw entry points funnel through the same walkers (probe already matched nested literals — unchanged); top-level image cores hit either the new branch or the old imageBase64 fallback with byte-identical output. One intentional behavior widening beyond the literal repro, disclosed in the PR body: top-level fileData-only cores previously billed+counted but note-less when folded now emit [image: <mime>] — same direction as the fix (removes billed-but-silent), not a regression.
    • Adjacent finding, parked here as a checklist item (no new issue per policy): src/image-rescale.ts rescaleGoogle walks top-level rawGoogleParts only, so nested tool screenshots are never downscaled — same non-recursive-traversal family, different symptom (payload transform vs accounting). Deliberately out of scope: rewriting nested parts touches verbatim-forwarded wire bytes (high-risk tier). Worth a follow-up if anyone wants nested-image rescaling.

    结构化八问

    1. 是什么问题:Gemini 工具返回里的截图(functionResponse.parts 中的 inlineData/fileData)会逐字节转发给上游并被上游计费,但 bili 自己的图像成本/计数通道把整个 functionResponse part 跳过,按 0 计费;折叠时摘要里也不留任何图像注记。
    2. 什么原因导致的:宿主侧三个计量面(src/image-tokens.ts 的 google 分支、src/image-note.ts 的 messageImages)只遍历顶层 contents[].parts,遇到带 functionResponse 的 part 直接 continue——嵌套媒体对它们不可见。内核侧(codec/恢复通道/protected/media-refs)早已递归感知嵌套,缺的只是宿主计量面。
    3. 影响范围与严重程度:影响 Gemini 车道中工具返回内联图片(截图等)的会话。后果:窗口门禁的图像预留从 0 起算(重度使用下可能放行超窗请求→上游报错)、折叠后截图在模型视野中静默消失(无恢复注记)、携带嵌套图片的大体积 Google 请求被 ingest-cap 误判 413。可用但有明显问题、有绕行空间 → p2(非缓存泄露、非数据丢失、会话可恢复)。
    4. 怎么修复的:cost/count/notes 三面共用同一条遍历(googleMediaParts 生成器:顶层 parts + 每个 functionResponse.parts),锁步由构造保证;注记面只对 image/* 生效——因为注记携带 decompress({imageRef}) 恢复引用,而恢复通道对嵌套 google 媒体只索引 image/*,非图像 mime 出注记就是承诺无法兑现的恢复。计费保持 mime 无关(转发出去的字节都要算)。无需改内核。
    5. 是否有回归问题:逐项清点过——顶层 inline/file 计费不变、Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 不透明件套件全绿(重构未引入双计费)、文本/函数载体仍免费、raw 入口走同一遍历、探测逻辑未动、顶层图像核新旧分支输出逐字一致。唯一有意扩大:顶层 fileData-only 核折叠后新增 [image: <mime>] 注记(此前计费但不留痕,同方向修正,已在 PR 描述披露旧→新)。副作用:含截图的 Gemini 会话折叠可能比之前更早触发——这正是预留修正的预期结果。
    6. 是否破坏缓存:不破坏。缓存键/前缀/重试/去重均无变化;注记只出现在摘要请求与折叠渲染中,主线路径前缀字节不变。
    7. 是否改了设计方向:没有。仅宿主侧计量与注记面;内核零改动(避开人工门禁);线上字节零改动(修的是计量与注记,不是载荷转换)。
    8. 是否引入了新的配置:没有。

    判定:可合。 PR #6147(head 5f3d479dbf,含我的补充 commit)本地 rebased-head 验证全绿、镜像 CI 缓存显示新 head 全绿,等待 owner 合并(agent 无权合并 PR)。

  2. ranxianglei commented on Nov 12, 2026

    @ranxianglei
    Owner

    Fixed in batch PR #6439 (merged).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions