Skip to content

[PR] fix: media CAS split covers all raw wire copies (#4069) - #4075

Closed
ranxianglei wants to merge 6 commits into
masterfrom
2026-10-21_media-cas-raw-wire-split
Closed

ranxianglei wants to merge 6 commits into
masterfrom
2026-10-21_media-cas-raw-wire-split

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Closes #4069

What this fixes

#3762's media CAS split moved the top-level imageBase64 sidecar into the content CAS, but every wire codec stores each image a SECOND time in its verbatim raw carrier — and that copy stayed inline forever. On the reporter's real 11 GB corpus the merged #3762 yielded 1.89× instead of the projected 33.8×, because the raw copy is ~the entire remaining payload.

This PR extends the SAME split/hydrate mechanism to all four codecs' raw carriers (one mechanism family = one closed loop, per the 2026-10-14 no-slicing rule):

carrier stored ref form
openai rawOpenaiContent / rawOpenaiContentParts[].image_url.url embedded url: data:<type>;base64,$media:<sha256hex>
responses rawResponsesItem.content[].input_image.image_url (ALL images of the item — the n×m case) same embedded-url form
anthropic rawAnthropicBlock.source.data (incl. image blocks nested in tool_result content arrays) bare value: $media:<sha256hex>
google rawGoogleParts[].inlineData.data bare value: $media:<sha256hex>

Same-hash payloads intern into ONE CAS row (sidecar + raw carrier dedup automatically); refcounting, stale-row sweep release, and the missing-CAS-row fallback are unchanged.

Design notes

  • No sibling key, no new top-level slot. The reference is embedded in the existing string value, so the stored JSON keeps the exact key set/order of the original — the byte-identical round-trip contract (pinned in store.ts) holds by construction, and every reader that parses urls/values as strings keeps working.
  • Strict canonical gate. Only ^data:([^;,]+);base64,(.+)$ (case-sensitive, mandatory ;base64) data URLs are rewritten — reconstruction is byte-exact by construction. Non-canonical forms (extra params, case-folded scheme, remote URLs) stay inline untouched; the kernel's loose parseDataUrl accepts some of them, which would make the sidecar fallback rebuild a DIFFERENT string.
  • redacted_thinking.data is deliberately NOT a split slot (KDD#10: thinking-family bytes are signature-verified upstream); pinned by a negative test.
  • MIN_SPLIT_MEDIA_CHARS = 65_536 gates every slot uniformly: below the floor the constant overhead (77-char ref + CAS row + mapping row ≈ 200 B) exceeds the payload; measured corpora put >99% of image bytes far above it. Sub-floor payloads pass through verbatim (no refs at all).

Behavior change (declared)

Tests

tests/media-cas.test.ts: 16 tests — dual-field dedup (one CAS row, cross-session refcount 2), plural carriers, non-canonical/remote/sub-floor passthroughs, stale-sweep release, delta-planner boundary-hash determinism, plus per-carrier cases for anthropic (top-level + nested tool_result + redacted_thinking negative), google (multiple inlineData parts + fileData untouched) and responses (multi-image item + remote url untouched), each with byte-exact round-trips.

Local: media-cas 16/16, 11-file storage-adjacent batch 94/96 (2 gated skips), typecheck + build clean. Full-suite verdict = remote CI on the pushed head.

@ranxianglei ranxianglei changed the title fix: media CAS split covers the raw OpenAI wire copy too (#4069) [PR] fix: media CAS split covers the raw OpenAI wire copy too (#4069) Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-21_media-cas-raw-wire-split (f28a1ac)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4923-pr.4075.912884c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4075.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。本 PR 是 #4069(媒体 CAS 拆分漏掉 raw OpenAI wire 副本)的修复 PR,open+closed 全量检索 media CAS / splitMediaContent / rawOpenaiContent 无其他在途同题条目;相邻项 #3987(GC/孤儿回收层)、#4018(v0.1.203 发版 PR)均不同机制域。

标签: bug + dsh + scope/storage + p2 —— 缺陷类;dsh 车道实测语料;统一存储 CAS 布局层;p2:存储体量翻倍于设计预期但功能与读路径不受影响,非缓存泄露失控增长类。

本 PR 为自家舰队对 #4069 的交付(分支 2026-10-21_media-cas-raw-wire-split,单 commit 5e0e7fa,base = 当前 master tip 046db20be,无 stale base)。按 PR 审核规范独立复审中:①原 issue 方向 → ②回归清点 → ③代码 → ④测试有效性 → ⑤⑤b diff/代码整洁,另查 CI 状态。结论按结构化八问回复,稍候。

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/storage labels Oct 21, 2026
@KafukaTree

Copy link
Copy Markdown

The only red check is unused-exports — the fix is one word

INTERNAL_ONLY src/storage/media.ts: MediaRef
Fix: delete the declaration (DEAD) or drop the leading `export` (INTERNAL_ONLY). Or add a reasoned entry to WAIVERS.

MediaRef is only used inside src/storage/media.ts — the three consumers in ingest.ts / store.ts / mysql.ts import isMediaRef, splitMediaContent, hydrateMediaRefs, SPLIT_MEDIA_URL, collectRawOpenaiImageParts and HYDRATE_MARKERS, never the type. Dropping the leading export clears the gate without touching behaviour:

-export interface MediaRef { hash: string; bytes: string }
+interface MediaRef { hash: string; bytes: string }

Every other check is green (the 37 non-success entries in the rollup are CANCELLED/skipping fan-out from this one job, not independent failures).

Independent verification on a real 11.69 GB corpus

I extracted splitMediaContent / hydrateMediaRefs verbatim from 0.0.4855-pr.4075.2407260 and replayed them over a production index.db (114,364 content rows, 15,193 of them carrying media):

original 11,691,412,127 B (11.69 GB)
split message rows 271.9 MB
CAS after dedup 322.3 MB (857 distinct payloads)
total 594.2 MB
ratio 19.68×
lossless round-trip 114,364 / 114,364, 0 mismatches

Schema migration also verified on a copy: user_version 5 → 7 is purely additive (CREATE TABLE IF NOT EXISTS message_media), all row counts and content bytes identical afterwards.

One coverage gap worth noting for the follow-up

collectRawOpenaiImageParts() only walks rawOpenaiContent and rawOpenaiContentParts, so only the openai lane gets its raw sidecar split. The other codecs keep the same duplication and are untouched:

sidecar occurrences in dist/index.js split?
rawOpenaiContent / rawOpenaiContentParts 24 ✅
rawAnthropicBlock 18 ❌
rawGoogleParts / inlineData 18 / 12 ❌
rawResponsesItem 26 ❌

On my install this is currently harmless — every media-bearing row is openai (rawOpenaiContent 16,882 + rawOpenaiContentParts 3 = 16,885; anthropic/google/responses 0) — but a user on /v1/messages or /responses gets no benefit at all. Might be worth a // TODO(#4069) so it is not mistaken for full coverage.

awork added 3 commits October 21, 2026 14:07
#3762's media CAS split only pulled the top-level imageBase64 sidecar out
of stored messages; the codec's second copy — the rawOpenaiContent /
rawOpenaiContentParts image_url data URLs — stayed inline, capping real
corpus savings at 1.89x instead of 33.8x (#4069).

splitMediaContent now rewrites every canonical raw-wire data URL to an
embedded CAS reference (data:<type>;base64,$media:<hash>) and
hydrateMediaRefs restores both shapes byte-exactly; non-canonical URLs
(extra params, uppercase scheme, remote sources) stay inline untouched.
Both slots are gated by MIN_SPLIT_MEDIA_CHARS (65_536 b64 chars): below
the floor the reference plus CAS/mapping rows cost more than the payload.
No schema change (v7), no config surface change.
@ranxianglei
ranxianglei force-pushed the 2026-10-21_media-cas-raw-wire-split branch from 5e0e7fa to c8d8dde Compare October 21, 2026 06:14
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

查重: 不适用——本条是对该 PR 的 CI 自动修复唤醒,非新 issue。

修了什么:CI unused-exports 门禁报本 PR 新增的 MediaRef 接口「导出了但没有任何其他文件引用」;修掉后顺带揪出并修好了同一 PR 测试代码里一个被掩盖至今的潜在类型错误。分支已 rebase 到最新 master 并更新(特性分支 force-push,常规操作)。

结构化八问:

  1. 是什么问题:CI unused-exports 门禁失败——新文件 src/storage/media.ts 导出的 MediaRef 接口只在本文件内部被用作返回类型,所有消费方都是结构类型、从不 import 这个名字,被判 INTERNAL_ONLY;另外 tests/media-cas.test.ts 里有一个此前从未暴露的潜在 tsc 错误。
  2. 什么原因导致的:写 media.ts 时给仅文件内使用的接口加了 export(门禁要求非入口文件的导出必须被跨文件引用);类型错误在 tests/media-cas.test.ts:344——用 .get() 取 content_hash 行,但本地 DbShape 把 get() 类型化为只返回 {bytes},tsc 报 TS2339。之前没暴露是因为 unused-exports 先失败,test/static-gates 等下游 lane 全被跳过没跑。
  3. 影响范围与严重程度:仅影响本 PR 自身的 CI 状态(无法合并),零运行时行为变化,不影响任何已发布产物或用户。严重度:构建门禁阻断,几行级修复。
  4. 怎么修复的:① 去掉 MediaRef 的 export(保留名字便于阅读,与同文件的文件本地类型 RawOpenaiImagePart 一致)——项目 noEmit、无 declaration 产物,去导出零副作用;② 测试那行改为 .all("m-img")[0]!.content_hash,匹配现有 DbShape 的类型化(sqlite 的 get() 本就等价于取 all() 首行,运行时语义不变)。两处均为独立标注 commit,不混入原 PR 提交。
  5. 是否有回归问题:无。两处改动都在类型级/测试级;本地定向验证:unused-exports 门禁干净、media-cas 套件 11/11 通过、全仓 tsc --noEmit 干净(修复前基线恰好只有那 1 个 TS2339,修复后消失)。全量回归交给远程 CI——这次之前被跳过的 test/static-gates lane 会真正跑起来。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:修复完成(auto-fix lane 第 1/2 次),分支 head 已更新为 c8d8dde66,等新一轮 CI 结果;若再有 lane 红,按规则继续定位。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

三点回应:

1. 门禁修复已落地,与你给的 diff 完全一致。 MediaRef 的 export 已在 commit 8622ba4d4 去掉(项目 noEmit、无 declaration 产物,去导出零副作用);unused-exports 门禁本地复跑干净。分支此前已 rebase 到最新 master,head 现为 34a6de2ee。

2. 覆盖缺口核实属实,已加边界注记。 我对照源码独立验证了:collectRawOpenaiImageParts(src/storage/media.ts)只走 rawOpenaiContent / rawOpenaiContentParts,SPLIT_MARKERS 也只含这两个键;而 rawAnthropicBlock / rawGoogleParts / rawResponsesItem 都是持久化字段(image-restore.ts / image-rescale.ts 从存储读回使用),所以非 openai 车道的 raw sidecar 确实仍带完整媒体副本、拿不到本 PR 的收益——你表里的判断成立。处理:在 collectRawOpenaiImageParts 的 JSDoc 里加了 SCOPE 注记(2 行),指向 #4069(仍 open)作为把 split 扩展到其余车道的跟踪点——同一机制家族,按粒度规则不另开新 issue。commit 34a6de2ee,tsc 干净。

3. 你的语料验证。 11.69 GB 的重放数字我这边无法独立复现,但无损往返这条性质由 tests/media-cas.test.ts 钉住(本地 11/11 通过),你这份生产库实测正好补上了真实数据面的证据,包括迁移纯增量这一点,很有价值。

CI 状态:修复 head 上检查无失败(检查时 12 success / 11 skipped / 7 在跑);docs commit 又触发了一轮新 CI,等它绿完这个 PR 就齐了。

#4069)

The openai-lane split left the other codecs' raw carriers inline: a
multi-protocol install still stored a second full copy per image under
rawAnthropicBlock.source.data, rawGoogleParts[].inlineData.data and
rawResponsesItem input_image urls (the responses item carries ALL its
images while the sidecar lifts only the first).

Same mechanism, closed in one loop:

- bare base64 slots (anthropic source.data incl. tool_result-nested image
  blocks, google inlineData.data) become $media:<hash> in place; data-url
  carriers (responses input_image urls) use the embedded-url form, like
  openai;
- both hydrate paths (sqlite sync, mysql async twin now delegating to one
  shared async helper) restore every shape byte-exactly;
- redacted_thinking .data is deliberately NOT a split slot (KDD#10:
  thinking-family bytes are signature-verified upstream);
- SPLIT_MARKERS extended so the fast path sees the new carriers; sub-floor
  payloads stay fully inline as before.
@ranxianglei ranxianglei changed the title [PR] fix: media CAS split covers the raw OpenAI wire copy too (#4069) fix: media CAS split covers all raw wire copies (#4069) Oct 21, 2026
@ranxianglei ranxianglei changed the title fix: media CAS split covers all raw wire copies (#4069) [PR] fix: media CAS split covers all raw wire copies (#4069) Oct 21, 2026
…ports gate)

Collapsing mysql.ts's hydrateMediaAsync into a one-line delegation to
hydrateMediaRefsAsync removed the last external references to SPLIT_MEDIA_URL,
HYDRATE_MARKERS, isMediaRef and collectRawOpenaiImageParts — all four are now
internal to media.ts, so the #1440 unused-export gate flags them as
INTERNAL_ONLY. Same treatment the reviewer applied to MediaRef in phase 1.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

陈旧票清理(第三批动作①):目标 issue [3762, 4069] 已关。为避免重复劳动关闭本 PR;如判断有误请直接 reopen 并留言。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/storage

Projects

None yet

2 participants