Skip to content

[PR] fix: recover each Responses image from its own ref - #6854

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-15_responses-image-recovery
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-15_responses-image-recovery

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

Fixes #6851.

A multi-image Responses user item is split into separate core messages and refs. Each image core carries its own pixels but also retains the whole original wire item. The recovery index walks that whole item, so every image ref restores all its sibling images.

This produces both incorrect retrieval and multiplicative disk work. Two screenshots are restored together even when only one ref was requested; a remote-only image ref can incorrectly return a neighboring inline image. Twenty image parts create 400 recovery files.

Changes

When a Responses core has the parser's existing imagePart flag, recover its own imageBase64 sidecar only. A split URL-only occurrence has no carried bytes and remains unavailable for recovery. Unsplit carriers and tool-result groups still recover all their images.

Repeated identical images retain separate occurrence refs. The change is confined to the host recovery index; kernel identity, outbound rendering, image encoding, file naming, and expiry behavior are unchanged.

Validation

Tested against parent/master 2e3cfc973b8620b1f7bc8a415ddcbd31e7f4aaee (0.1.215).

Unchanged master: four new regressions fail and sixteen controls pass. The tests use the real Responses parser, ref assignment, recovery file writer, and decompress tool path with valid red/blue PNGs.

Scenario Master Candidate
Two different image refs Each restores both images Each restores its own pixels
Remote-only image beside an inline image Remote ref borrows the inline image Remote ref has no recoverable bytes
Two identical image occurrences Two images per ref One image per ref; distinct occurrence paths
Twenty image parts 400 indexed spill files 20 indexed spill files
Unsplit opaque carrier / multi-image tool output All associated images recovered Preserved

Candidate: 76 focused tests passed.

node --import ./tests/preload-test-env.mjs --import tsx --test --test-concurrency=1 tests/image-restore.test.ts tests/responses-tool-images.test.ts tests/responses-block-wire.test.ts tests/issue3400-image-rescale.test.ts tests/egress-shed-media.test.ts

The controls include image recovery across all four wires, incomplete-spill repair, original-image recovery, Responses partial-fold rendering, media shedding, and real local HTTP proxy/plugin paths. The fixtures and HTTP upstreams are hermetic.

Also passed:

  • node node_modules/typescript/bin/tsc --noEmit --project tsconfig.json
  • npm run build
  • node scripts/check-test-import-graph.mjs
  • git diff --check

The live Codex four-phase E2E has not been run. E2E_CHECK=1 confirms the distribution and CLI, but its configured http://127.0.0.1:8199/v1/models upstream is unreachable.

Related work checked

#4770 addresses outbound Responses reconstruction in responses-tool-output.ts; this patch fixes the separate recovery-index consumer. Open #5492 archives missing OpenAI assistant/tool image sidecars and does not change Responses split-image ownership. #6089 repairs truncated image files, while #6701 recognizes escaped JSON image markers.

Use split image sidecars instead of expanding the whole carrier under
every image ref; preserve unsplit and tool-result image groups.

Fixes ranxianglei#6851
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Dedup: not a duplicate. Searched open+closed issues/PRs for the same symptom/mechanism (split Responses image refs, recovery index, spill cross-product): #6854 is the only open PR against #6851. The related items the PR itself lists are distinct failure domains and are all closed or out of scope (#4770 outbound reconstruction, #5492 OpenAI assistant/tool sidecars, #6089 truncated files, #6701 escaped JSON markers).

Review verdict: mergeable (可合), once CI lands green. I did not take the PR's claims on faith — I reproduced both halves locally on the exact head f5aeb282 and base 2e3cfc97:

  • Bug is real, at the claimed layer. With the PR's tests present but the src/image-restore.ts fix reverted to base, exactly the four new regressions fail (split-own-pixels, remote-borrows-sibling, repeated-identical, 20-parts→400-files) and the unsplit-carrier control plus all 16 pre-existing controls pass. With the fix, 76/76 focused tests pass across the five named files. Typecheck clean.
  • Root cause correctly located. The kernel parser (kernel/src/wire/responses.ts:351-406) deliberately gives each split image core the WHOLE rawResponsesItem as a byte-fidelity sidecar, and imagePart: true is set only on split-path image cores (responses.ts:395 — the non-split merged core at :483-492 and tool-output cores never get it). messageImageBytes walked that whole carrier before consulting the singular sidecar, so every split ref indexed every sibling. The fix gates on imagePart === true inside the rawResponsesItem branch and returns only the core's own imageBase64 sidecar (or [] for URL-only). That is a decision-point fix, not a mask: the producer of the wrong binding (the index's walk order) is what changes, and the gate covers every split shape the parser can emit.
  • Regression sweep of the touched path. (a) Non-Responses wires never enter the new branch (rawResponsesItem absent) — the four-wire extraction test passes unchanged. (b) Non-split Responses cores and tool-output cores have no imagePart flag → carrier walk preserved, pinned by the new control test. (c) All three index consumers improve rather than break: resolveImageRestore (src/decompress-shared.ts:292) now returns exactly the ref's own file and its existing failure message already anticipates the URL-only case; executeReadOriginalImage (src/image-rescale.ts:618) previously served entries[0] = the FIRST SIBLING's pixels for a split ref — wrong pixels, now its own; refForItemBytes (src/image-rescale.ts:520) previously degraded to size-only notes for every multi-image split item (same bytes found under two refs → undefined) — now resolves the correct ref. No consumer depended on the cross-product. (d) File naming, 7d expiry, kernel identity, outbound rendering, encoding: untouched by the diff. (e) The index is a sidecar, not wire content — no prompt-cache impact.
  • Tests have teeth. Real responsesToCore parser, real ref assignment, real spill writer, real decompress path; assertions compare exact on-disk bytes, include negative directions (sibling path absent from receipt; remote ref fails; remote ref absent from index), quantitative (20 files, exact total bytes), and a preserved-behavior control. Fixtures verified as valid 2×2 PNGs (signature + decodeImageDims 2×2). This satisfies the two-direction gate discipline (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617).
  • Diff hygiene. 2 files, +70/-0, single commit, every line on-topic; no kernel/, no config surface, no version touch. Base is 4 commits behind master but those commits touch only scripts/gate/* — zero overlap, no rebase needed.
  • CI (live-checked via mirror cache): 30 checks, 0 failed, 6 still queued (docs/config-surface/mysql-contract/unused-exports/pi-subagents-guard); all fast guards (kernel-guard, version-guard, lane-guard, architecture-docs, test-import-graph, gate lanes) green.
  • Auto-merge gate (§7.4): all eight conditions met (single-module pure fix, reproducing regression tests, clean diff, Fixes #6851, no load-bearing infra). Merge itself stays human-only.

Recommended labels for this PR (not applied — source PR labels are owner/automation territory): bug, p2 (wrong-pixels retrieval + quadratic disk work on multi-image Responses sessions; workaround: single-image messages), scope/kernel-adjacent but host-side → scope/wiring (recovery index is the host's protocol-boundary consumer).


八问(中文结论)

  1. 是什么问题:Responses 协议里一条多图用户消息会被拆成多个 core(每个图一个 ref),但恢复索引按「整条原始 wire item」扫描,导致每个图 ref 都索引到所有兄弟图——取一张图会还回一堆图,纯 URL 图 ref 会错误借到邻居家内联图的像素,20 张图产生 400 个落盘文件。
  2. 什么原因导致的:kernel 解析器有意让每个拆分图 core 携带整条 rawResponsesItem 作为字节保真 sidecar;messageImageBytes 在查自己的 imageBase64 sidecar 之前先遍历整条 carrier,绑定关系在索引时刻就被写错。根因在恢复索引的消费顺序,不在解析器。
  3. 影响范围与严重程度:所有走 Responses 协议、单条消息含 ≥2 张内联图(或 1 张纯 URL 图 + 1 张内联图)的会话;decompress({imageRef}) 与 acp_read_original_image 会返回错误像素,图片重缩放标注退化为无 ref 的尺寸注记,磁盘写放大 N 倍。特定条件下有实质错误(错误像素),非全量不可用 → p2。
  4. 怎么修复的:在 messageImageBytes 的 Responses 分支内,对带解析器既有 imagePart: true 标志的拆分图 core 只恢复其自身 imageBase64 sidecar(纯 URL 拆分图无字节可恢复,返回空);未拆分 carrier 与工具输出保持整条扫描。改动 9 行,host 侧单函数,kernel 不动。
    4a. 本地实证:base 上 4 个新回归测试失败、16 个对照通过;candidate 上 76 个聚焦测试全过、typecheck 干净。
  5. 是否有回归问题:无。逐项核过:四条 wire 的非 Responses 路径不进入新分支;未拆分 core 与工具输出 core 无 imagePart 标志、行为由新对照测试钉住;三个索引消费方(decompress 恢复 / 原图读取 / 重缩放 ref 反查)全部改善或不变,无一依赖旧的交叉积行为;文件命名、7 天过期、kernel 身份、出站渲染、编码均未触碰。
  6. 是否破坏缓存:不涉及——恢复索引是 sidecar 元数据,不在 wire 上,prompt cache 前缀不受影响。
  7. 是否改了设计方向:不涉及——未拆分 carrier「一个 ref 收全部图」的既有设计保留,只修正拆分 core 的归属。
  8. 是否引入了新的配置:不涉及。

判定:可合(CI 剩余 6 个排队项转绿后由 owner 合并;合并是人工操作)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI follow-up (live-checked via mirror cache): all 30 checks now complete, 0 failed, 0 pending — the six queued guards (docs-sync, docs-coverage, config-surface-guard, unused-exports, mysql-contract, pi-subagents-guard) all landed green. The PR is fully green and ready for a human merge: #6854

@ranxianglei

Copy link
Copy Markdown
Owner

This fix is included verbatim (original commits and authorship preserved) in the merged batch PR #7074 — the changes are already on master. Closing to avoid a duplicate merge. 该修复已随批量 PR #7074 原样合入 master,为避免重复合并,关闭此 PR。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Responses split image refs restore sibling images and create quadratic spill files

2 participants