Repository navigation
[PR] fix: media CAS split covers all raw wire copies (#4069) - #4075
ranxianglei wants to merge 6 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4923-pr.4075.912884cThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr4075.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。本 PR 是 #4069(媒体 CAS 拆分漏掉 raw OpenAI wire 副本)的修复 PR,open+closed 全量检索 media CAS / splitMediaContent / rawOpenaiContent 无其他在途同题条目;相邻项 #3987(GC/孤儿回收层)、#4018(v0.1.203 发版 PR)均不同机制域。 标签: bug + dsh + scope/storage + p2 —— 缺陷类;dsh 车道实测语料;统一存储 CAS 布局层;p2:存储体量翻倍于设计预期但功能与读路径不受影响,非缓存泄露失控增长类。 本 PR 为自家舰队对 #4069 的交付(分支 |
The only red check is
|
| original | 11,691,412,127 B (11.69 GB) |
| split message rows | 271.9 MB |
| CAS after dedup | 322.3 MB (857 distinct payloads) |
| total | 594.2 MB |
| ratio | 19.68× |
| lossless round-trip | 114,364 / 114,364, 0 mismatches |
Schema migration also verified on a copy: user_version 5 → 7 is purely additive (CREATE TABLE IF NOT EXISTS message_media), all row counts and content bytes identical afterwards.
One coverage gap worth noting for the follow-up
collectRawOpenaiImageParts() only walks rawOpenaiContent and rawOpenaiContentParts, so only the openai lane gets its raw sidecar split. The other codecs keep the same duplication and are untouched:
| sidecar | occurrences in dist/index.js |
split? |
|---|---|---|
rawOpenaiContent / rawOpenaiContentParts |
24 | ✅ |
rawAnthropicBlock |
18 | ❌ |
rawGoogleParts / inlineData |
18 / 12 | ❌ |
rawResponsesItem |
26 | ❌ |
On my install this is currently harmless — every media-bearing row is openai (rawOpenaiContent 16,882 + rawOpenaiContentParts 3 = 16,885; anthropic/google/responses 0) — but a user on /v1/messages or /responses gets no benefit at all. Might be worth a // TODO(#4069) so it is not mistaken for full coverage.
#3762's media CAS split only pulled the top-level imageBase64 sidecar out of stored messages; the codec's second copy — the rawOpenaiContent / rawOpenaiContentParts image_url data URLs — stayed inline, capping real corpus savings at 1.89x instead of 33.8x (#4069). splitMediaContent now rewrites every canonical raw-wire data URL to an embedded CAS reference (data:<type>;base64,$media:<hash>) and hydrateMediaRefs restores both shapes byte-exactly; non-canonical URLs (extra params, uppercase scheme, remote sources) stay inline untouched. Both slots are gated by MIN_SPLIT_MEDIA_CHARS (65_536 b64 chars): below the floor the reference plus CAS/mapping rows cost more than the payload. No schema change (v7), no config surface change.
5e0e7fa to
c8d8dde
Compare
(ai: vllm/qwen3.8-27b) 查重: 不适用——本条是对该 PR 的 CI 自动修复唤醒,非新 issue。 修了什么:CI 结构化八问:
判定:修复完成(auto-fix lane 第 1/2 次),分支 head 已更新为 |
(ai: vllm/qwen3.8-27b) 三点回应: 1. 门禁修复已落地,与你给的 diff 完全一致。 2. 覆盖缺口核实属实,已加边界注记。 我对照源码独立验证了: 3. 你的语料验证。 11.69 GB 的重放数字我这边无法独立复现,但无损往返这条性质由 CI 状态:修复 head 上检查无失败(检查时 12 success / 11 skipped / 7 在跑);docs commit 又触发了一轮新 CI,等它绿完这个 PR 就齐了。 |
#4069) The openai-lane split left the other codecs' raw carriers inline: a multi-protocol install still stored a second full copy per image under rawAnthropicBlock.source.data, rawGoogleParts[].inlineData.data and rawResponsesItem input_image urls (the responses item carries ALL its images while the sidecar lifts only the first). Same mechanism, closed in one loop: - bare base64 slots (anthropic source.data incl. tool_result-nested image blocks, google inlineData.data) become $media:<hash> in place; data-url carriers (responses input_image urls) use the embedded-url form, like openai; - both hydrate paths (sqlite sync, mysql async twin now delegating to one shared async helper) restore every shape byte-exactly; - redacted_thinking .data is deliberately NOT a split slot (KDD#10: thinking-family bytes are signature-verified upstream); - SPLIT_MARKERS extended so the fast path sees the new carriers; sub-floor payloads stay fully inline as before.
…ports gate) Collapsing mysql.ts's hydrateMediaAsync into a one-line delegation to hydrateMediaRefsAsync removed the last external references to SPLIT_MEDIA_URL, HYDRATE_MARKERS, isMediaRef and collectRawOpenaiImageParts — all four are now internal to media.ts, so the #1440 unused-export gate flags them as INTERNAL_ONLY. Same treatment the reviewer applied to MediaRef in phase 1.
|
(ai: zhipuai-lb/glm-5.3) 陈旧票清理(第三批动作①):目标 issue [3762, 4069] 已关。为避免重复劳动关闭本 PR;如判断有误请直接 reopen 并留言。 |
Closes #4069
What this fixes
#3762's media CAS split moved the top-level
imageBase64sidecar into the content CAS, but every wire codec stores each image a SECOND time in its verbatim raw carrier — and that copy stayed inline forever. On the reporter's real 11 GB corpus the merged #3762 yielded 1.89× instead of the projected 33.8×, because the raw copy is ~the entire remaining payload.This PR extends the SAME split/hydrate mechanism to all four codecs' raw carriers (one mechanism family = one closed loop, per the 2026-10-14 no-slicing rule):
rawOpenaiContent/rawOpenaiContentParts[].image_url.urldata:<type>;base64,$media:<sha256hex>rawResponsesItem.content[].input_image.image_url(ALL images of the item — the n×m case)rawAnthropicBlock.source.data(incl. image blocks nested in tool_result content arrays)$media:<sha256hex>rawGoogleParts[].inlineData.data$media:<sha256hex>Same-hash payloads intern into ONE CAS row (sidecar + raw carrier dedup automatically); refcounting, stale-row sweep release, and the missing-CAS-row fallback are unchanged.
Design notes
^data:([^;,]+);base64,(.+)$(case-sensitive, mandatory;base64) data URLs are rewritten — reconstruction is byte-exact by construction. Non-canonical forms (extra params, case-folded scheme, remote URLs) stay inline untouched; the kernel's looseparseDataUrlaccepts some of them, which would make the sidecar fallback rebuild a DIFFERENT string.redacted_thinking.datais deliberately NOT a split slot (KDD#10: thinking-family bytes are signature-verified upstream); pinned by a negative test.MIN_SPLIT_MEDIA_CHARS = 65_536gates every slot uniformly: below the floor the constant overhead (77-char ref + CAS row + mapping row ≈ 200 B) exceeds the payload; measured corpora put >99% of image bytes far above it. Sub-floor payloads pass through verbatim (no refs at all).Behavior change (declared)
imageBase64sidecar slot was split UNCONDITIONALLY on master (Same image bytes stored once per conversation/message copy — split media payloads into a content-addressed store (follow-up to #3751/#3748) #3762, unreleased — v0.1.203 is still pending); it is now floor-gated like everything else. No shipped version is affected.Tests
tests/media-cas.test.ts: 16 tests — dual-field dedup (one CAS row, cross-session refcount 2), plural carriers, non-canonical/remote/sub-floor passthroughs, stale-sweep release, delta-planner boundary-hash determinism, plus per-carrier cases for anthropic (top-level + nested tool_result + redacted_thinking negative), google (multiple inlineData parts + fileData untouched) and responses (multi-image item + remote url untouched), each with byte-exact round-trips.Local: media-cas 16/16, 11-file storage-adjacent batch 94/96 (2 gated skips), typecheck + build clean. Full-suite verdict = remote CI on the pushed head.