You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] Gemini tool-response screenshots have zero image cost and disappear from summary notes #6146
At e1e33a9, send a Gemini functionResponse with response:{output:"Captured two views"} and two image/png inlineData entries in functionResponse.parts. The actual codec retains these parts on the tool-result core, but parsed and raw image-token/count entry points return zero. imagePlaceholdersForSummary also returns no image notes.
Expected behavior
Every forwarded tool image must contribute to image budgets/counts. Folding a tool result should retain one descriptive note and recovery reference per image. Text-only results should incur no image cost.
Verified patch
Share Google media traversal across cost, count and placeholders. Four new unit regressions fail against unchanged source; 31 targeted tests pass. A real proxy/Gemini-upstream test confirms folding, both recovery notes in the summary request and removal of base64 from the forwarded request without sending it to the summarizer. Typecheck and build pass. #2022 already stores recovery for these images; #3412 concerns other opaque-media accounting.
查重: 不重复。 Searched open+closed issues and PRs for the same mechanism (Gemini nested tool-media accounting). Closest neighbors: #2022 (restore channel — already indexes these images, unchanged here), #3412 (opaque custom-shape media — different part shapes), #2609/#2614 (fileData treated as media payload — closed, different surface). No prior issue/PR covers "functionResponse.parts media priced 0 + no fold notes". Note: PR #6147 already targets this exact issue (opened by the reporter), so per one-fix-PR-per-thread I verified and supplemented that PR instead of opening a parallel one.
标签: bug · gemini · scope/wiring · p2 (window-gate misestimate + missing recovery notes + possible mis-413; usable-with-workaround class, not a cache-leak/data-loss class) — applied to this thread; PR #6147 labeled bug · p2 · gemini · image on its own diff (an automated labeler had also added scope/kernel despite zero kernel diff — removed).
Triage (three steps, done before touching code)
Verified the bug exists at master HEAD. Reproduced the report exactly: a functionResponse with response:{output:"Captured two views"} plus two image/pnginlineData entries in functionResponse.parts yields cost 0 / count 0 from imageTokensInParsedBody/imageTokensInRawBody/countImagesIn*, and no notes from messageImages/imagePlaceholdersForSummary. The 4 new unit tests fail 4/4 against unfixed source and pass with the fix — genuine regression teeth, not author claims taken on faith.
Layering. The kernel side was already nesting-aware: the codec (kernel/src/wire/google.ts googleToCore) carries the whole functionResponse part verbatim on the tool-result core (rawGoogleParts), the restore channel (src/image-restore.ts:169) indexes nested image/* media, protected.ts treats nested inlineData as archivable/foldable, and media-refs.ts recurses for re-access handles. Only the host-side accounting surfaces missed the nesting: the google branches of src/image-tokens.tscontinue over any part carrying a functionResponse (nested media unpriced → window gates, ingest-cap exemption at src/server/handle.ts:460, learned-cost attribution all see 0), and src/image-note.ts messageImages has no rawGoogleParts branch (folded screenshots leave no note). Fix belongs at those three surfaces — the producer side of the accounting — not by filtering payloads or patching the summary prompt. That is exactly where the PR puts it.
Plan assessment. The PR shares ONE traversal (googleMediaParts generator, top-level parts + every functionResponse.parts) across cost, count and notes, so the three stay in lockstep by construction (the Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 discipline). Notes are gated to image/* mimes — deliberate and correct: a note carries a decompress({imageRef}) recovery ref, and the restore channel can only deliver nested image/*; billing stays mime-agnostic because every forwarded byte reaches upstream. Direction is right; no better layer exists without a kernel change (which would need the human gate and isn't needed).
Rebased-head verification: cherry-picked onto current master (52 commits ahead of the PR base; zero drift in any touched file), ran npm run typecheck clean + 10 targeted suites covering every consumer of the touched functions — 106/106 pass, including the new real-proxy/Gemini-upstream integration test (fold fires, both · mNNNNN recovery notes reach the summarization request, base64 gone from the forwarded model request, summary input carries no pixels) and the Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 opaque-media suite (no double-charging from the loop refactor).
Test validity: assertions have teeth (exact byte/pixel arithmetic, caps, mixed top-level+nested totals that break on any double or missed charge, text-only negative gates, bidirectional e2e assertions). One gap found and fixed: the image/* note gate had only its positive direction pinned (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 requires both) — pushed supplement commit 5f3d479dbf to the PR branch adding the nested application/pdf case (billed + counted, no note) plus a stray double-blank-line cleanup. The contributor's original commit is preserved untouched; CI on the new head is green (13 success / 17 conditional-skip / 0 failed).
Regression inventory (touched paths): top-level inlineData/fileData billing unchanged; text/functionCall/videoMetadata carriers still free; unknown-shape opaque charging unchanged; raw entry points funnel through the same walkers (probe already matched nested literals — unchanged); top-level image cores hit either the new branch or the old imageBase64 fallback with byte-identical output. One intentional behavior widening beyond the literal repro, disclosed in the PR body: top-level fileData-only cores previously billed+counted but note-less when folded now emit [image: <mime>] — same direction as the fix (removes billed-but-silent), not a regression.
Adjacent finding, parked here as a checklist item (no new issue per policy):src/image-rescale.ts rescaleGoogle walks top-level rawGoogleParts only, so nested tool screenshots are never downscaled — same non-recursive-traversal family, different symptom (payload transform vs accounting). Deliberately out of scope: rewriting nested parts touches verbatim-forwarded wire bytes (high-risk tier). Worth a follow-up if anyone wants nested-image rescaling.
结构化八问
是什么问题:Gemini 工具返回里的截图(functionResponse.parts 中的 inlineData/fileData)会逐字节转发给上游并被上游计费,但 bili 自己的图像成本/计数通道把整个 functionResponse part 跳过,按 0 计费;折叠时摘要里也不留任何图像注记。
什么原因导致的:宿主侧三个计量面(src/image-tokens.ts 的 google 分支、src/image-note.ts 的 messageImages)只遍历顶层 contents[].parts,遇到带 functionResponse 的 part 直接 continue——嵌套媒体对它们不可见。内核侧(codec/恢复通道/protected/media-refs)早已递归感知嵌套,缺的只是宿主计量面。
Activity
test(google): pin the image/* note gate negative direction for nested…
ranxianglei commentedon Nov 10, 2026
[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)
查重: 不重复。 Searched open+closed issues and PRs for the same mechanism (Gemini nested tool-media accounting). Closest neighbors: #2022 (restore channel — already indexes these images, unchanged here), #3412 (opaque custom-shape media — different part shapes), #2609/#2614 (fileData treated as media payload — closed, different surface). No prior issue/PR covers "functionResponse.parts media priced 0 + no fold notes". Note: PR #6147 already targets this exact issue (opened by the reporter), so per one-fix-PR-per-thread I verified and supplemented that PR instead of opening a parallel one.
标签: bug · gemini · scope/wiring · p2 (window-gate misestimate + missing recovery notes + possible mis-413; usable-with-workaround class, not a cache-leak/data-loss class) — applied to this thread; PR #6147 labeled bug · p2 · gemini · image on its own diff (an automated labeler had also added scope/kernel despite zero kernel diff — removed).
Triage (three steps, done before touching code)
functionResponsewithresponse:{output:"Captured two views"}plus twoimage/pnginlineDataentries infunctionResponse.partsyields cost 0 / count 0 fromimageTokensInParsedBody/imageTokensInRawBody/countImagesIn*, and no notes frommessageImages/imagePlaceholdersForSummary. The 4 new unit tests fail 4/4 against unfixed source and pass with the fix — genuine regression teeth, not author claims taken on faith.kernel/src/wire/google.ts googleToCore) carries the wholefunctionResponsepart verbatim on the tool-result core (rawGoogleParts), the restore channel (src/image-restore.ts:169) indexes nestedimage/*media,protected.tstreats nested inlineData as archivable/foldable, andmedia-refs.tsrecurses for re-access handles. Only the host-side accounting surfaces missed the nesting: the google branches ofsrc/image-tokens.tscontinueover any part carrying afunctionResponse(nested media unpriced → window gates, ingest-cap exemption atsrc/server/handle.ts:460, learned-cost attribution all see 0), andsrc/image-note.ts messageImageshas norawGooglePartsbranch (folded screenshots leave no note). Fix belongs at those three surfaces — the producer side of the accounting — not by filtering payloads or patching the summary prompt. That is exactly where the PR puts it.googleMediaPartsgenerator, top-level parts + everyfunctionResponse.parts) across cost, count and notes, so the three stay in lockstep by construction (the Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 discipline). Notes are gated toimage/*mimes — deliberate and correct: a note carries adecompress({imageRef})recovery ref, and the restore channel can only deliver nestedimage/*; billing stays mime-agnostic because every forwarded byte reaches upstream. Direction is right; no better layer exists without a kernel change (which would need the human gate and isn't needed).Review of PR #6147 (independent pass)
npm run typecheckclean + 10 targeted suites covering every consumer of the touched functions — 106/106 pass, including the new real-proxy/Gemini-upstream integration test (fold fires, both· mNNNNNrecovery notes reach the summarization request, base64 gone from the forwarded model request, summary input carries no pixels) and the Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 opaque-media suite (no double-charging from the loop refactor).image/*note gate had only its positive direction pinned (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 requires both) — pushed supplement commit5f3d479dbfto the PR branch adding the nestedapplication/pdfcase (billed + counted, no note) plus a stray double-blank-line cleanup. The contributor's original commit is preserved untouched; CI on the new head is green (13 success / 17 conditional-skip / 0 failed).imageBase64fallback with byte-identical output. One intentional behavior widening beyond the literal repro, disclosed in the PR body: top-level fileData-only cores previously billed+counted but note-less when folded now emit[image: <mime>]— same direction as the fix (removes billed-but-silent), not a regression.src/image-rescale.ts rescaleGooglewalks top-levelrawGooglePartsonly, so nested tool screenshots are never downscaled — same non-recursive-traversal family, different symptom (payload transform vs accounting). Deliberately out of scope: rewriting nested parts touches verbatim-forwarded wire bytes (high-risk tier). Worth a follow-up if anyone wants nested-image rescaling.结构化八问
[image: <mime>]注记(此前计费但不留痕,同方向修正,已在 PR 描述披露旧→新)。副作用:含截图的 Gemini 会话折叠可能比之前更早触发——这正是预留修正的预期结果。判定:可合。 PR #6147(head
5f3d479dbf,含我的补充 commit)本地 rebased-head 验证全绿、镜像 CI 缓存显示新 head 全绿,等待 owner 合并(agent 无权合并 PR)。ranxianglei commentedon Nov 12, 2026
Fixed in batch PR #6439 (merged).
test(google): pin the image/* note gate negative direction for nested…