Skip to content

[PR] Image ids stop re-minting when an earlier image is folded away (#3751) - #3761

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-18_issue3751-media-identity
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-18_issue3751-media-identity

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

kernel-change: image core ids in kernel/src/wire/* now seed from a payload fingerprint (sha256 of media type + base64) instead of the shared "[image]" placeholder; bumps acp-kernel 0.0.134 → 0.0.135.

什么坏了(#3751)

同一会话里,只要更早的一张图片被压缩折叠(或客户端 rewind/删除),后面所有存活图片的内部 id 全部重新生成——内容一个字节没变,身份却换了。

原因:所有图片部件的身份种子是同一个常量("[image]"),全仓库的图片共享一个 base id,唯一区分符是"本次转换中的第 N 次出现"后缀。前面图片一退出,后面图片的后缀整体位移。每次重铸:该位置 wire 字节变化 → 每次折断一次 prefix-cache;存储层按整条消息 JSON 做内容寻址,id 在 JSON 里 → 重写一个全新 blob、留一个 rc=0 孤儿(这就是 #3748 里同一张图被记录 56 次、11.06× 冗余、约 17.8 GiB/天的上游机制)。

修法

身份种子改为载荷指纹:sha256(media type + base64 字节),四个协议转换器(anthropic/openai/responses/google)统一,包括 responses 的单部件折叠路径(与多部件拆分路径铸造同一个 id)。

  • 同像素 → 同 base id:成员变化(折叠/rewind)不再重铸任何存活图片;
  • 不同像素 → 天然不同 base id;
  • 残余面:同一张图在同一会话出现多次时,副本间仍靠 _N 后缀区分(窄得多,需要彻底解决得做媒体级内容寻址存储——已单独立 issue 跟踪);
  • 无载荷部件(url/file 源、仅 raw sidecar 的形状)退回旧常量种子,这些 id 升级后不动;
  • 消息的 text 字段仍是 "[image]",只有种子变了——存储 JSON 面除 id 外无任何变化。

测试

Closes #3751(该 issue 标注的跨会话/跨消息字节级去重已拆分为后续 issue)。

@ranxianglei ranxianglei changed the title Image ids stop re-minting when an earlier image is folded away (#3751) [PR] Image ids stop re-minting when an earlier image is folded away (#3751) Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-18_issue3751-media-identity (8c07fab)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4449-pr.3761.1a7fe45

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3761.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 相关但不同的条目已核对:

标签: bug + dsh + scope/kernel + p2(缺陷修复;证据来自 dsh 生产数据 #3748;改动全在 kernel/src/wire/*;严重度与源 issue #3751 的 p2 对齐)。

开始 review:kernel 变更属最高评审档,我会独立走原 issue 方向审核、diff 精读、既有行为回归清点、测试有效性、定向本地验证,随后按八问体例给结论。CI 状态与 rebase(master 已前进 7 个提交)一并核查。

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/kernel labels Oct 19, 2026
Every image part shared one base id (CANONICAL_IMAGE_TEXT in the seed)
and the only discriminator was the ClusterCounter suffix — the Nth
occurrence in THIS conversion pass. Fold/rewind/client-compaction of any
earlier image shifted every surviving suffix: identity degraded from a
function of content to a function of set membership. Each re-mint cost
one prefix-cache break on the wire and one fresh wrapper blob + rc=0
orphan in the CAS (#3748: same image recorded 56×, 11.06× redundancy,
~17.8 GiB/day).

Seed image cores by an exact payload fingerprint instead — sha256 over
the declared media type + base64 bytes — across all four converters
(anthropic/openai/responses/google), including the responses non-splittable
fold path, so single-part and multi-part shapes of the same image mint
the same id. Same pixels → same base id across membership changes;
different pixels → different bases. Residual churn is confined to true
duplicates of the SAME image inside one pass (cluster suffix still
orders them). Unparseable parts (url/file sources, raw-only sidecars)
keep the legacy constant seed so those ids never move. The core text
field stays "[image]" — only the seed changed.

Bump acp-kernel 0.0.134 -> 0.0.135 (kernel-guard rule 2).
@ranxianglei
ranxianglei force-pushed the 2026-10-18_issue3751-media-identity branch from 121a5ce to 8c07fab Compare October 19, 2026 02:31
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核进展(第一轮):

  1. 基线验证(红→绿):把 merge-base 的旧 wire 文件换回来跑新测试——5 例中恰好 2 例红(fold 稳定性、不同像素不同 base id),换回 PR 代码后 5/5 绿。回归测试确实复现了 Image message ids re-mint whenever an earlier image leaves the visible tail (membership-based ClusterCounter suffix) — prefix-cache breaks per fold, feeds #3748 storage growth #3751 的缺陷,不是恒真断言。
  2. 已 rebase 到当前 master(ff3536c,干净无冲突;master 侧漂移全是 host 文件,与本 PR 零交集),并补了一个小测试后 push 到 PR 分支(新 head 8c07fabd6):
    • 原测试第 5 例只覆盖了 responses 的多部件拆分路径;PR 描述声称「单部件折叠路径与多部件拆分路径铸造同一个 id」但没有测试钉住它。已读代码确认两条路径对同一张图使用完全相同的种子参数(deriveImageMessageId("user", mediaType, base64),折叠路径经 userDisplayText 收敛到 [image] 标记后走指纹种子),新增一例把「同一张图的两种 shapes 铸同一个 id」钉死。独立标注 commit:kernel: pin responses collapsed/splittable image-id equivalence (#3751)。
  3. 本地定向验证(rebase 后的 head 上):kernel wire/identity 相关 14 个测试文件 139/139 绿(含新文件 6/6、The intermediate message (core) is not protocol-agnostic: switching protocols/models causes identity churn; cross-protoc | 中间消息(core)没有做到协议无关:切协议/切模型制造身份 churn,补齐跨协议映射 #3642 cross-protocol-identity);host 侧图片/golden/wire 定向 14 个文件 93/93 绿——golden-storage-render 无需重生成,印证了 PR 描述的「语料图片是 raw-only 形状走 fallback」判断;typecheck + build(dist-import-annotations / dist-external-zod)全过。

等 CI 全量结果出来后给八问结论。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant