Skip to content

[PR] fix: repair incomplete image spills when the original bytes return - #6089

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-10_repair-truncated-image-spills
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-10_repair-truncated-image-spills

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

persistImageToDisk treats every existing path as a complete image. Zero-byte or partial spill files stay broken even after a later request provides the complete original image again, so restored references keep pointing at truncated bytes.

Changes

Skip writing only when the existing path is a regular file with the expected decoded byte count. Repair incomplete spills from original bytes. A third unchanged indexing pass preserves the completed file mtime; same-size corruption is outside this size-based repair.

Validation

  • Before: tests/image-restore.test.ts — 11 passed, 2 failed.
  • After: 13 passed, 0 failed, with real image blocks, byte comparisons and unchanged-file mtime controls.
  • npm run typecheck — passed.
  • npm run build — passed.
  • Published fork tree abee4c71de4643abbf1a395398029b9b411399b4 exactly matches tested local commit 0f6a7b6dd35e6324f9442e99ce7451641e966799.

The full remote-suite verdict belongs to CI. #2022 introduced image references and the existing-path fast path. Prior identity/collision and session cleanup work does not restore incomplete image bytes. All 3,221 historical PRs were screened; independent of #6046–#6055.

Closes #6087

@ranxianglei ranxianglei changed the title fix: repair incomplete image spills when the original bytes return [PR] fix: repair incomplete image spills when the original bytes return Nov 10, 2026

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE.

Direction (issue #6087). Verified the defect independently before reviewing the code: the fast path in writeRestoredImage (src/image-restore.ts:441) used existsSync(path) alone, so once a spill file existed — even zero-byte or partial after an interrupted writeFileSync — every later turn skipped the write forever, and decompress({ imageRef }) kept serving truncated bytes. The fix targets the PREDICATE (the producer of the wrong decision), not a downstream mask: skip only when the path is a regular file of exactly the expected decoded size; anything else falls through and is rewritten from the resent original. The module header already documents that clients re-send full raw history every turn, so the repair opportunity arrives naturally each turn. Root cause, not symptom.

Verification (evidence, not "should work").

  • Ran the PR's test file against the PRE-FIX source (parent commit 541cf260c4): exactly the two new repair tests fail — 11 pass / 2 fail, matching the author's before-report byte-for-byte. At head 6f2a54640: 13/13 pass.
  • npm run typecheck clean at head; no leftover references to the removed existsSync import.
  • CI on head 6f2a54640: green per lane status; mirror-cache snapshot shows 0 failed checks.
  • Master has moved 10 commits since the PR base but ZERO of them touch src/image-restore.ts or tests/image-restore.test.ts; PR reports mergeable: true. BEHIND is not a blocker.

Regression inventory (existing behaviors on the touched paths, each judged):

  1. Steady-state skip cost — PRESERVED: still exactly one stat per image per turn (existsSync → statSync, same order); the new mtime-control pass pins "complete file ⇒ no rewrite".
  2. Content-salt idempotency & rebase protection (#1995 F1) — PRESERVED: same bytes ⇒ same digest-named path ⇒ size matches ⇒ skip; the rebase-generation test still passes.
  3. Cross-session isolation (per-session dirs) — UNTOUCHED.
  4. Write-failure handling — UNCHANGED (warn + null, caller skips the entry). One degenerate-case delta, disclosed: if a DIRECTORY ever sat at the content-addressed file path, old code returned the broken path silently; new code hits isFile()==false, writeFileSync throws EISDIR into the existing catch → warn log + null (image simply not indexed that turn). Strictly more honest than silent broken-path return; degenerate state only.
  5. 7-day spill prune (pruneRetrieveImgExports) — UNTOUCHED.
  6. Wire / both compression modes / caches — UNTOUCHED: this module only writes local spill files; no wire bytes, no prompt-cache surface.

Behavior change disclosure (old → new). Skip-write condition: "path exists" → "path is a regular file with size == decoded byte count". Consequence: zero-byte/partial spills are now REPAIRED from the resent original on the next index pass instead of staying broken indefinitely — the intended fix. Author-documented boundary: same-size content corruption remains outside this size-based repair (digest salt makes same-path-different-content practically unreachable otherwise). Comparison uses body.length (the actual buffer written), correctly avoiding the padding-sensitive floor(b64len×3/4) approximation in img.bytes.

Test quality. Real wire conversion (openaiToCore) + real disk + byte-exact comparison; both gate directions pinned per #3617 discipline (repair fires for 0-byte AND 8-byte truncation; third pass proves the fast path still skips when complete — via mtime equality, granularity-safe on Windows too). Isolated XDG_STATE_HOME, unique session ids, deterministic, no ports.

Diff cleanliness. 2 files, +20/−3; every line in scope; dead import removed. No lockfile/generated churn.


八问(结构化审核结论)

  1. 是什么问题:图片溢出文件(spill)被截断成零字节/部分字节后,即使客户端后续轮次重新携带完整原图,恢复路径也永远指向截断的字节——decompress({ imageRef }) 还原出坏图。
  2. 什么原因导致的:快路径谓词只查 existsSync(path),不校验文件是否完整;写一半被打断(进程崩溃/磁盘故障)后路径永久存在,后续所有轮次都被「已存在」跳过。根因在谓词本身,修复打在谓词上而非下游掩盖。
  3. 影响范围与严重程度:仅影响发生过写中断的会话中的对应图片(触发条件窄:需要 writeFileSync 中途失败);一旦命中该图永久不可恢复,手动删除坏文件可绕行——特定条件下的可用性缺陷,p2 级,非会话级数据丢失。
  4. 怎么修复的:收紧跳过条件为「常规文件且大小 == 解码字节数」,其余情况一律用重传的原字节重写修复;比较用实际写入缓冲 body.length(避免 base64 padding 近似误差)。
  5. 是否有回归问题:逐项清点既有行为——稳态成本(每图每轮一次 stat,保持)、幂等与内容盐 rebase 保护(#1995 F1,保持)、跨会话隔离(未动)、错误处理(保持;唯一差异是「目录占据文件路径」退化态从静默返回坏路径变为 warn+null,更诚实)、7 天清理(未动)、双压缩模式线协议与缓存(未触)。无回归。
  6. 是否破坏缓存:不涉及——只写本地溢出文件,不触碰 prompt cache/前缀缓存/重试策略。
  7. 是否改了设计方向:否——同模块内单谓词收紧,架构、边界、数据流不变。
  8. 是否引入了新的配置:无新 env/config/默认值。

判定:APPROVE。 单模块纯 fix:、回归测试有牙(旧码上实测红、head 上绿)、diff 干净、引用 Closes #6087、CI 绿;满足 §7.4 auto-merge 条件(未触 kernel/、未触承重设施),合并判定交人工/auto-merge 管道。

Review lane done: PR #6089 reviewed end-to-end — APPROVE (red/green A/B on the new tests, typecheck clean, regression inventory 6/6 preserved).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — APPROVE posted on the PR (review #5837317071) at head 6f2a54640.

What was done & verified:

  • Read the PR body, linked issue Resending an image does not repair a truncated existing spill file #6087, and the full diff (2 files, +20/−3); CI state taken from the mirror cache (green at head, 0 failed checks).
  • Direction check: root cause is the existsSync-only fast-path predicate in writeRestoredImage (src/image-restore.ts:441) — once a spill path existed (even zero-byte/partial after an interrupted write), every later turn skipped the rewrite forever. The fix tightens that predicate to "regular file AND size == expected decoded byte count" and repairs anything else from the resent original. Producer-side fix, not a downstream mask.
  • Red/green A/B evidence: ran the PR's test file against pre-fix source (parent 541cf260c4) → exactly the 2 new repair tests fail (11/13 pass), matching the author's before-report; at head → 13/13 pass. npm run typecheck clean; no leftover existsSync references.
  • Master drift: 10 commits ahead of the PR base, zero of them touching the changed files → no semantic conflict; BEHIND is not a blocker.
  • Regression inventory (existing behaviors on the touched paths): steady-state cost (one stat per image per turn — preserved), idempotency + content-salt rebase protection (stripImages 滑窗剥离的图片无恢复通道,且滑窗每轮击穿 prompt cache(billion-context-pi#593 同族缺口) #1995 F1 — preserved), cross-session isolation (untouched), error handling (unchanged; one degenerate-case delta disclosed — directory-at-file-path now yields warn+null instead of silently returning a broken path, strictly more honest), 7-day spill prune (untouched), wire / both compression modes / caches (untouched).
  • Behavior change old→new (disclosed): skip-write condition changed from "path exists" to "path is a regular file of exactly the expected decoded size"; truncated/partial spills now repair on the next index pass instead of staying broken indefinitely. Same-size corruption explicitly out of scope (author-documented size-based boundary). Comparison uses body.length (the actual buffer written), correctly avoiding the padding-sensitive img.bytes approximation.
  • Test quality: real wire conversion + real disk + byte-exact comparison; both gate directions pinned per Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 discipline (repair fires for 0-byte AND 8-byte truncation; third pass proves the fast path still skips when complete, via granularity-safe mtime equality); isolated state dir, unique session ids, no ports.
  • Diff cleanliness: every line in scope, dead import removed, no lockfile/generated churn.

八问(结构化审核结论)

  1. 是什么问题:图片溢出文件被截断成零字节/部分字节后,即使客户端后续轮次重传完整原图,恢复路径也永远指向截断字节(decompress({imageRef}) 还原出坏图)。
  2. 什么原因导致的:快路径谓词只查 existsSync(path)、不校验完整性;写中断后路径永久存在,后续所有轮次都被「已存在」跳过。修复打在谓词本身(根因),非下游掩盖。
  3. 影响范围与严重程度:仅影响发生过写中断的会话中对应的图片(触发条件窄);一旦命中该图永久不可恢复,手动删除坏文件可绕行——特定条件下的可用性缺陷,p2 级,非会话级数据丢失。
  4. 怎么修复的:跳过条件收紧为「常规文件且大小 == 解码字节数」,其余情况一律用重传的原字节重写修复;比较用实际写入缓冲 body.length(避开 base64 padding 近似误差)。
  5. 是否有回归问题:六项既有行为逐项清点——稳态成本(保持)、幂等+内容盐 rebase 保护(stripImages 滑窗剥离的图片无恢复通道,且滑窗每轮击穿 prompt cache(billion-context-pi#593 同族缺口) #1995 F1,保持)、跨会话隔离(未动)、错误处理(保持,退化态更诚实)、清理 TTL(未动)、双压缩模式线协议与缓存(未触)。无回归。
  6. 是否破坏缓存:不涉及——只写本地溢出文件,不触碰 prompt cache/前缀缓存/重试策略。
  7. 是否改了设计方向:否——同模块内单谓词收紧,架构、边界、数据流不变。
  8. 是否引入了新的配置:无新 env/config/默认值。

判定:APPROVE。 单模块纯 fix:、回归测试有牙(旧码上实测红、head 上绿)、diff 干净、引用 Closes #6087、CI 绿;满足 §7.4 auto-merge 条件(未触 kernel/、未触承重设施),合并判定交人工/auto-merge 管道。

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6439 (merged) — closing as superseded.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Acknowledged — verified before closing the loop on my side: the batch PR (GitHub #6439, merged 2026-11-12T14:01:25Z) landed this fix on master as commit 1eb3843a27 "fix: repair incomplete image spills when originals are resent" (history preserved), and origin/master:src/image-restore.ts:434 now carries the size-gated statSync(path, { throwIfNoEntry: false }) predicate with no existsSync left in the file. The APPROVE review at head 6f2a54640 stands; nothing further to do on this thread.

@ranxianglei ranxianglei reopened this Nov 14, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Status: already complete — no further action needed.

This is a re-trigger of an already-finished thread. I verified the current state live before replying (not from memory):

The fix is in, the PR is closed as superseded, and the owner's floor-6 ruling ("Landed via batch PR #6439 (merged) — closing as superseded") is the final word. Nothing to re-review, re-label, or re-open — closing the loop on my side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resending an image does not repair a truncated existing spill file

2 participants