Skip to content

[PR] fix: include Gemini tool-response images in budgets and summary notes - #6147

Closed
Afloat16 wants to merge 4 commits into
ranxianglei:masterfrom
Afloat16:2026-11-10_preserve-google-image-notes
Closed

Afloat16 wants to merge 4 commits into
ranxianglei:masterfrom
Afloat16:2026-11-10_preserve-google-image-notes

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Gemini screenshots in functionResponse.parts survive on the wire, but image cost/count walkers skip their enclosing result. They therefore contribute zero to fitting and per-image learning. Summary rendering also ignores rawGoogleParts, losing descriptions and recovery references when results are folded.

The recovery index additionally skipped an entire function response when its payload contained $ref, even though that reference can name an inlineData image carried in the same response.

Changes

  • Share a Google-parts iterator across cost, count and summary notes, including nested response media under the existing per-image billing rules and caps.
  • Emit one descriptive placeholder per image, including remote references, before falling back to the singular sidecar. Plain result text keeps its existing accounting.
  • Index nested image/* inline bytes even when the response references them with $ref. A reference without inline bytes still produces no indexed image or recovery note. Indexing leaves the wire response unchanged.
  • Preserve the supplemental test commit 5f3d479 by ci-check, covering non-image nested media: billed and counted, without a recovery note.

Validation

The branch includes master dc041dd1ab1a014efa352ba47a6f31fc47f55284 through merge commit c24efc51d0b28c7fbe17067a713cbc27750f92bc. All three existing commits and their original authors remain intact. The merge's source tree 07fcfc86014f6057d05c198cf1d38693c339f3c7 is identical to the locally validated integration tree.

  • 70 related tests passed, covering image accounting, opaque media, summary placeholders, actual proxy preflight and recovery of the exact original PNG bytes.
  • All 5 npm run test:e2e:image tests passed. Both test runs had 0 failures, skips or cancellations.
  • npm run typecheck, the full npm run build including dist checks, and git diff --check passed.
  • Real Gemini proxy/upstream integration verifies that both recovery notes reach summarization, no base64 reaches the summarizer, and folded pixels are absent from the model request.
  • Main CI run 42997294497 is green on this head: all 11 test shards and both static-gates jobs passed. The ci:full label is still needed to execute the image-billing CI lane.

Related work checked: #2022 and #3412. The recovery storage format is unchanged.

Closes #6146

Behavior changes (old → new)

  • Nested functionResponse media (inlineData/fileData): image cost 0 → real cost (bytes mode: base64 length/4; pixels mode: tile estimate or fallback); image count 0 → 1 per media part. Downstream: window-gate image reserve for Gemini sessions carrying tool screenshots rises from 0 (folds may trigger earlier in such sessions — intended), the ingest-cap exemption no longer mis-rejects large Google bodies that carry nested images, and learned per-image cost attribution now counts them.
  • Fold/summary rendering: Gemini tool-result cores now emit one descriptive note per image ([image: WxH · mNNNNN]) in folded views and summary input; previously no note at all — folded screenshots silently vanished from the model's view.
  • Top-level fileData-only cores: previously billed and counted but note-less when folded → now emit [image: ]. Same direction as above (billed-but-silent is the inconsistency this removes).
  • Inline screenshots referenced with $ref: actual nested image bytes remain indexed and recoverable; a dangling reference does not promise recovery.
  • No kernel change, no wire format change, no config surface change, no cache-key/retry/dedup change.

@ranxianglei ranxianglei changed the title fix: include Gemini tool-response images in budgets and summary notes [PR] fix: include Gemini tool-response images in budgets and summary notes Nov 10, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。镜像内按 "Gemini functionResponse image cost / summary notes / rawGoogleParts" 检索 open+closed issue 与 PR,无同症状同机制条目。作者已核对的 #2022(image recovery channel)与 #3412(opaque media accounting)均为 closed、相邻但不同机制——本 PR 修的是「Gemini tool-response parts 里的图片既不进成本/计数,折叠时也不进摘要占位」这一独立缺口。

开始按 PR Review 规范审核(方向 → 回归 → 代码 → 测试有效性),结论稍后在本线程给出。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review of #6147 — "fix: include Gemini tool-response images in budgets and summary notes" (Closes #6146)

Dup screen (from my first reply): 不重复 — no same-symptom/same-mechanism issue or PR exists; the cited #2022/#3412 are adjacent-but-different-mechanism (recovery-index lane and custom-shape counting lane, vs. this one's accounting/notes lane).

① Direction — correct layer, root cause. The kernel codec (kernel/src/wire/google.ts) already hands rawGoogleParts — including nested functionResponse.parts — to cores, and the recovery index (src/image-restore.ts:148-177) already walks those nested parts for spill-to-disk by mNNNNN. The defect was purely host-side: both accounting walkers in src/image-tokens.ts looped only top-level c.parts, and messageImages() in src/image-note.ts had no google branch at all. Sharing one parts iterator across cost/count/notes fixes the producer, not the symptom.

② Regression audit (existing behaviors on the touched paths):

  • No double-count: each wire part is yielded exactly once; the functionResponse carrier itself still hits the existing skip gate (typeof part.text === string || functionCall || functionResponse → continue), and its nested parts are billed individually.
  • Cost/count lockstep (Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 discipline) preserved: both walkers consume the same iterator with the same gates. Deeper nesting (fr.parts[i].functionResponse) is charged opaquePartCost in BOTH walkers — safe direction.
  • Raw-body variants delegate to the parsed path after JSON.parse → covered automatically. bodyHasImagesProbe's google probe (literal inlineData/fileData) still fires for nested-only bodies.
  • Top-level non-image inlineData falls through to the legacy imageBase64 branch — master behavior byte-preserved.
  • Wire bytes forwarded upstream are UNCHANGED (accounting + summary-text only) → no prefix-cache impact, no persistence format change, no config surface, no kernel change. KDD#7 reasoned in both modes: proxy-mode preflight notes + fit budget fixed; plugin-mode budget/nudge/side-request estimates also benefit; injection/carrier logic untouched.

③ Code quality — clean 4-file diff (+130/-5), generator idiom matches repo style, no dead code, comments accurate. One flagged asymmetry (non-blocking, needs your/owner's call):

messageImageBytes (src/image-restore.ts:164) carries a Gemini-3 guard: when functionResponse.response uses {"$ref": displayName}, nested parts are NOT indexed for recovery ("bytes aren't inline"). The new messageImages block has no such guard, so in a $ref shape that still carries image bytes in fr.parts, the summary would emit [image: … · mNNNNN] notes whose decompress({imageRef}) fails gracefully (toolFail lists available refs). Edge-within-edge; from the code alone I can't tell whether $ref shapes ever co-carry real bytes. If they do: mirror the guard or extend the recovery index. If they don't: current code is harmless. Please confirm against wire evidence.

Behavior changes (disclosed per policy) — old → new:

  1. Gemini functionResponse.parts images: fit-decision cost 0 → per-image billing under existing rules (pixels mode: tile est ≤2805 / 16384 fallback, caps apply; bytes mode: b64/4); count 0 → 1 each (per-image learning attribution restored).
  2. Folding such a tool result: no image notes → one [image: <mime> WxH · mNNNNN] placeholder per image in summarizer input (recovery ref included where indexed).
  3. Top-level fileData images: no notes → [image: <mime>] notes (remote URIs included — same convention as master's Anthropic source: url handling).

Consequence: screenshot-heavy sessions' payload estimates rise → folding may trigger slightly earlier than before. This is the intended correction of the up-to-~15× undercount vs actual screenshot cost (pixel-billing rationale per #488), not drift.

④ Tests — real teeth, verified live. I checked out the origin/master versions of src/image-note.ts + src/image-tokens.ts into a worktree of the PR head: all 4 new unit cases FAIL on unchanged source (not ok 1..4, 0 pass) and pass on the fix — genuine regressions, not tautologies. Adjacent suites on PR head: issue3398-opaque-media-reserve + fix-781-summary-images + image-restore + issue1843-learned-image-cost + issue4124-image-count-cap = 60/60 pass. The new e2e runs a real proxy against a fake Gemini SSE upstream and pins BOTH claimed outcomes: both mNNNNN notes reach summarization, AND no base64 pixels reach the model request after the fold. It correctly reuses the file's existing helpers (makeGeminiUpstream / longGeminiContents / startProxy), no copy-paste drift.

Rebase & CI status (live-checked). PR base e1e33a974c is 52 commits behind master; GitHub reports mergeable=true, no conflicts. Local verification rebase onto origin/master (58a8796224): clean, and NONE of the 4 PR files were touched by master in that window (no semantic-conflict risk). On the rebased head: PR's 7/7 tests pass, npm run typecheck clean, git diff --check clean. Cached CI view for the pushed head: 30 checks, 0 failed / 0 pending, but the test shards / static-gates / red-signal / mysql-contract / version-guard / lane-guard / plan lanes show cancelled (superseded-looking runs) — so I cannot claim green; the full-suite verdict is CI's job on a fresh push. A rebase onto current master is advisable before merge regardless.

Auto-merge gate (§7.4): strong candidate — pure fix:, one mechanism across the coupled accounting pair, reproducing regression tests, Closes #6146, clean diff, no load-bearing infra / config / wire-shape / kernel touch. Merge stays human-only per repo rules.

标签: bug · gemini · scope/kernel · p2(缺陷;Gemini 原生线客户端;上下文管理/折叠决策域;能用但有明显问题——无数据丢失无缓存泄露,不升 p1)


结构化八问(中文,供人工评审)

  1. 是什么问题:Gemini 原生协议里工具结果(functionResponse.parts)内嵌的截图,在折叠预算和图片计数里完全不算账(0 token、0 张),折叠后摘要里也不留任何图片占位——像素其实可恢复,但模型既看不到成本、也不知道能恢复。
  2. 什么原因导致的:根因在宿主侧——两个记账 walker(src/image-tokens.ts)只遍历顶层 c.parts,src/image-note.ts 的 messageImages 根本没有 google 分支。kernel codec 与恢复索引(src/image-restore.ts)本就覆盖嵌套 parts,修复层选对了:补的是记账+摘要层,不是症状层。
  3. 影响范围与严重程度:所有走 Gemini 原生线且工具结果带截图的会话。p2——能用但有明显问题(折叠决策少算最多约 15×、恢复提示丢失),无数据丢失、无缓存泄露,不达 p1。
  4. 怎么修复的:cost/count/notes 三处共用一个 Google-parts 迭代器(顶层 + functionResponse.parts 一层嵌套),沿用既有单图计费规则与上限不变;每图产出一条描述性占位符,原有兜底逻辑不动。
  5. 是否有回归问题:逐项审过——无重复计费(functionResponse 载体本身仍走既有 skip 门)、cost/count 保持 lockstep(Image errors accumulating in the conversation | 对话中图片错误堆积 #3398 纪律)、更深嵌套两 walker 一致按 opaquePartCost 计(安全方向)、顶层非图 inlineData 落回旧 imageBase64 分支(master 行为字节保留)、转发上游的线上字节完全不变。唯一 flag:$ref 守卫不对称(正文已述,低危边界,需作者/owner 拍板,不阻塞)。
  6. 是否破坏缓存:不涉及——出站请求字节不变,未触碰任何缓存键/游标/去重键,prefix-cache 锚点稳定。
  7. 是否改了设计方向:不涉及——纯记账修正,无架构/数据流转向。
  8. 是否引入了新的配置:不涉及——无新 env/config/默认值。

判定:可合(建议先 rebase 到当前 master;$ref 不对称为非阻塞项,可后续单独确认)。合并动作按仓库规则由人执行。

@ranxianglei ranxianglei added bug Something isn't working gemini p2 scope/kernel labels Nov 10, 2026
… tool media (ranxianglei#6146)

Review supplement to PR ranxianglei#6147: ranxianglei#3617 gate discipline requires both
directions of a gate. The image/* mime filter on the Gemini note path had
only its positive case pinned. Adds a nested application/pdf case asserting
it is billed and counted but earns no recovery note — the note carries a
decompress({imageRef}) recovery ref, and the restore channel indexes nested
google media for image/* mimes only (src/image-restore.ts), so a note there
would promise a restore it cannot deliver. Billing stays mime-agnostic.

Also drops a stray double blank line in tests/preflight-google-summary.test.ts.
@Afloat16

Copy link
Copy Markdown
Contributor Author

Confirmed the $ref shape in Google's official FunctionResponse documentation: response.image_ref = { "$ref": "capture.png" } can name an inlineData part with displayName: "capture.png" and actual image bytes in the same response. See https://ai.google.dev/api/generate-content#FunctionResponse .

Fixed this in 378ba55, on top of your 5f3d479 test additions: the recovery index now includes those inline images instead of discarding the entire function response when it contains $ref. The new regression failed before the fix (empty recovery index) and now checks that the summary note resolves through the real recovery path to the exact original PNG bytes. A $ref with no inline bytes still produces no note or indexed image, and indexing preserves the wire response.

Validation: 70 related tests, all 5 test:e2e:image tests, typecheck, and build pass. The live Codex four-phase gate remains outstanding because no configured live upstream is reachable in this environment.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Round 2 — reviewing your $ref fix (378ba55151) and the supplement commit (5f3d479dbf).

One attribution note first: 5f3d479dbf ("pin the image/* note gate negative direction") was pushed by an earlier automated review pass (commit author ci-check), not by me in this session — my prior reply was review-only, I pushed nothing. Reviewed it as part of the PR diff anyway: sound, on-topic, and it correctly completes the #3617 both-directions discipline (nested application/pdf is billed + counted but earns no recovery note, since the restore channel indexes nested google media for image/* only).

The $ref fix — verified, resolves my round-1 flag.

  • Direction is right. The old guard assumed $ref ⇒ "bytes are not inline"; your check against Google's FunctionResponse docs shows $ref names an inlineData part in the same response that carries the bytes — so the guard was suppressing indexing in exactly the case where restore works. Removing it makes the index side (messageImageBytes) symmetric with the note side (messageImages), which keys off "inline image/* bytes present in fr.parts" — one definition on both sides, no more divergence possible.
  • Bonus: the old guard was a naive JSON.stringify(response).includes('"$ref"') substring test — any tool result merely containing the literal text "$ref" would have suppressed indexing. Removal fixes that false-positive class too.
  • No residual guards: grepped src/ + kernel/src/ at the new head — the only $ref left is your explanatory comment. Nothing elsewhere interacts with the removed behavior.
  • Wire fidelity held: indexing stays a read-only walk; your new test pins JSON.stringify(body) byte-equal after buildIndex. Good — that's the invariant that matters most here.
  • Teeth verified live: I swapped origin/master's src/image-restore.ts into a worktree of your head — the new positive test fails (not ok 12, 12/13 pass), the negative test ("$ref without inline bytes adds no image") passes on both versions. Genuine regression pin, both directions.

Local verification at 378ba55151: 21/21 across image-restore + google-tool-image-accounting + preflight-google-summary; 49/49 adjacent suites (issue3398-opaque-media-reserve, fix-781-summary-images, issue1843-learned-image-cost, issue4124-image-count-cap). Master has moved on (now 59 commits ahead of your base, was 52) — zero of the six PR files touched in the whole window; local rebase onto current master (e3220adc0f) clean, 21/21 still pass, npm run typecheck exit 0, git diff --check clean. (Rebase done locally for verification only — your branch history is untouched.)

CI (mirror-cached view of your pushed head): 30 checks — 0 failed / 0 pending / 0 cancelled. All test shards (ubuntu 22/24 ×3, windows 22 ×5), static-gates (both OS/node), mysql-contract, test-import-graph, gates green; 14 skipped = the env-gated e2e lanes. One coverage gap to close before merge:

"real-image billing e2e" (ci-image.yml, the home of tests/e2e/e2e-image-billing.test.ts) shows SKIPPED — despite this PR modifying exactly src/image-*.ts, which is in that lane's own path filter. Cause: its gate job (.github/workflows/ci-image.yml:58-61) only runs for PRs carrying the ci:full label; plain pushes skip it. Your local 5/5 run covers it, but for an image-accounting PR I'd want the lane green upstream too: @ranxianglei, adding the ci:full label to this PR is the designed one-click trigger. (Side observation, master-side doc drift, not fixed here as it's outside this PR's diff: AGENTS.md still calls this lane "ungated and always runs in CI" — stale vs the gate job as written.)

Your note on the live Codex four-phase gate: fine — this PR touches none of the request-pipeline paths (server.ts/loop/**/adapters/preflight) that make that lane mandatory.


结构化八问(中文,供人工评审)

  1. 是什么问题:上一轮我标记的 $ref 不对称——恢复索引侧有「见到 $ref 就跳过整个 functionResponse」的守卫,而摘要占位符侧没有,$ref 形态下可能产出无法兑现的恢复引用。作者已按官方文档确认:$ref 指向同一响应内携带真实字节的 inlineData part,旧守卫的前提是错的。
  2. 什么原因导致的:旧守卫是从已删除的 kernel strip-images.ts 镜像来的启发式,对 Gemini $ref 语义的判断反了(以为 $ref ⇒ 字节不内联);且用子串匹配,工具结果文本里出现字面量 "$ref" 也会误伤。修复=删掉守卫,让索引侧与占位符侧共用同一判定「fr.parts 里有带字节的 image/* inlineData」。
  3. 影响范围与严重程度:Gemini 原生线、functionResponse 用 $ref 引用内联图片的会话(如截图工具)。此前这类图片被整个跳过索引——可恢复的图片没有恢复通道;属 p2 内的边界修正,无数据丢失、无缓存泄露。
  4. 怎么修复的:删除 src/image-restore.ts 中的 $ref 子串守卫,嵌套 image/* inlineData 一律索引;索引保持只读遍历不改线上字节(新测试钉住 body 字节相等)。双向测试:有字节→占位符经真实恢复路径还原出原始 PNG;无字节($ref 悬空)→无占位符、无索引项。
  5. 是否有回归问题:逐项审过——全仓(含 kernel/src)再无其他 $ref 守卫残留,无交互;无重复索引(每个 part 恰好遍历一次);线上转发字节不变;非 image/* 内联媒体仍不计恢复注记(ci-check 补测钉住负向)。本地实测:master 版 image-restore.ts 下新正向测试失败(12/13),证明测试有牙。
  6. 是否破坏缓存:不涉及——出站请求字节不变,未触碰缓存键/游标/去重键。
  7. 是否改了设计方向:不涉及——把两侧收敛到同一判定,方向不变。
  8. 是否引入了新的配置:不涉及。

判定:可合(较上轮新增两个前置项:① rebase 到当前 master——本地已验证干净,建议推分支时做;② 建议 owner 加 ci:full 标签让 image-billing e2e 车道在 CI 上跑绿)。合并动作按仓库规则由人执行。

@Afloat16

Copy link
Copy Markdown
Contributor Author

Updated the branch with master dc041dd1 in merge commit c24efc51, preserving all three existing commits and their original authors, including ci-check's test supplement.

The merged source tree is identical to the locally validated tree: 70 related tests and all 5 image-billing E2E tests passed, with no failures, skips or cancellations. Typecheck, the full build with dist checks, and git diff --check also passed. Main CI run 42997294497 is green on this head: all 11 test shards and both static-gates jobs passed.

@ranxianglei, could you add the ci:full label so the real-image billing lane runs upstream as requested in the round-2 review?

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6439 (merged) — closing as superseded.

Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Nov 12, 2026
… tool media (ranxianglei#6146)

Review supplement to PR ranxianglei#6147: ranxianglei#3617 gate discipline requires both
directions of a gate. The image/* mime filter on the Gemini note path had
only its positive case pinned. Adds a nested application/pdf case asserting
it is billed and counted but earns no recovery note — the note carries a
decompress({imageRef}) recovery ref, and the restore channel indexes nested
google media for image/* mimes only (src/image-restore.ts), so a note there
would promise a restore it cannot deliver. Billing stays mime-agnostic.

Also drops a stray double blank line in tests/preflight-google-summary.test.ts.

(cherry picked from commit 5f3d479)
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Close-out confirmed — verified live before saying so:

To @Afloat16: your merge-commit update (c24efc51 over master dc041dd1) did exactly what the repo's "fix directly, don't replace" rule asks — additive, no history rewrite, authorship intact (now confirmed on master, not just your branch). And your ci:full request is moot with the work landed via #6439 — no label needed; the merged tree rides master's normal CI. Thanks for the fast, evidence-backed turnaround on the $ref fix: doc check, guard removal instead of a second special case, wire-fidelity pin, and regression tests in both directions — that's the shape these fixes should take.

Review record complete (round 1: accounting/notes fix; round 2: $ref index fix). No open items on this thread. 判定:已收工——工作经 #6439 合入 master,本 PR 按 superseded 关闭,无遗留。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Re-dispatch check — nothing new to act on. Verified live just now:

  • PR [PR] fix: include Gemini tool-response images in budgets and summary notes #6147: still closed, merged=false. One honest discrepancy: closed_at now reads 2026-11-14T01:58:35Z vs 2026-11-12T14:08:59Z when I verified the close-out — i.e. the PR was re-closed after the owner's close comment, with no user comment in between (only platform plumbing floors 21–22 around that time). I can't attribute the re-close from here; if it wasn't intentional, it's a sync-pipeline artifact worth a look. It changes nothing substantive: the PR was never merged directly and stays superseded.
  • Work landed: batch PR [PR] chore(batch): land 50 open Afloat16 fixes in one history-preserving batch #6439 merge 873715d40a is on master, carrying all three commits (Afloat ×2, ci-check ×1) with authorship intact — re-verified present.
  • Thread: no new user comments since the owner's "closing as superseded" (floor 15); my close-out confirmation (floor 19) stands.

No action taken this round; no open items. 判定:线程已完结,本轮仅复核状态,无新增工作。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Gemini tool-response screenshots have zero image cost and disappear from summary notes

2 participants