Skip to content

[PR] fix(storage): hash the split media form on both verification legs, so media-bearing tail slots can converge (Closes #6356) - #6357

Closed
dimas623 wants to merge 2 commits into
ranxianglei:masterfrom
dimas623:2026-11-12_media-split-digest
Closed

dimas623 wants to merge 2 commits into
ranxianglei:masterfrom
dimas623:2026-11-12_media-split-digest

Conversation

@dimas623

Copy link
Copy Markdown
Contributor

What

BatchedViewVerification compares the envelope's lastMessages against the same messages read back through UnifiedStore.getMessages(). Those two legs hold the same message in different media representations, so messageSignature hashes a placeholder against an inlined payload and marks drift on every sample. This fixes the digest by normalizing both legs to the split form, which is the form the rows are actually written in.

Closes #6356.

Why it matters

Rows are written media-split (planIngest -> splitMediaContent: an imageBase64 payload and every data URL of 2048+ bytes become {"$media":"<sha256>"} CAS references), but getMessages() returns them media-hydrated (hydrateMedia inlines the payload again). The envelope keeps the split form.

Measured on a live store: one tool result is 1991 bytes stored and 201293 bytes hydrated — a 101x difference for the same message id, same role, only text differing. Ten 390-byte user rows hydrate to 112 KB–672 KB.

Because the mismatch is structural rather than a content change, the sample never converges:

  1. the first drifted sample latches the session (driftedSessions);
  2. the latch forces the exact full-ingest path on every save (DSH raised a version storage layer issue during self-check. I'm unsure of the severity or urgency, so I'm sharing it her | DSH在自检时提出了版本存储层问题,我不确定严重性和急迫性,在此分享。 #5677);
  3. verification keeps failing, so the latch never releases.

Cost while latched, measured on the reporting machine: slow-ingest: … path=full at 1.2–1.6 s per save sustained on a ~3.5k-row session, and 8–9 s per save on a heavier one (24 consecutive full ingests, ~171 s of event-loop starvation) until a host-side history rewrite happened to change the tail.

How

const json = JSON.stringify(canonicalize(value)) ?? "null";
const normalized = splitMediaContent(json).content;
const digest = sha256(normalized);
// slot.length now reports normalized.length

splitMediaContent is idempotent: a {"$media": …} marker does not match its data-URL regex, so an already-split envelope passes through byte-identical. And because splitMediaContent(hydrate(x)) === x on every media-bearing row sampled, a genuine media edit still changes the digest — the comparison keeps its detection power instead of being skipped.

Not the flagged alternative

Gating on tail.tailFolded was tried first and rejected: planIngest writes tailFolded from session.lastMessagesFolded, which is only set by foldResidentTail, while the envelope sets messagesFolded from buildRecord whenever a bounded snapshot exists. On both live sessions tailFolded read false while the envelope was in fact a folded suffix, so the guard would never fire. Skipping the comparison would also stop detecting real divergence inside the window.

Tests

New tests/media-split-digest-parity.test.ts:

  • split envelope slot must verify clean against its hydrated row — builds a fixture with 4096-byte imageBase64 messages, asserts the stored CAS row really contains $media (reading content.bytes directly, since getMessages() is the code that hydrates), folds the envelope to the newest messages, and requires verifyDrift === 0 with no latch and no envelope freeze across the second sample (VERIFY_EVERY_INGESTS = 20).
  • a real media change inside the envelope window still reports drift — rewrites a row's content bytes to different media, then requires verifyDrift >= 1 and a latched session.

Negative control: replacing splitMediaContent(json).content with the raw json fails the first case with exactly the reported shape (expect … len=144 vs got … len=4166, fields: imageBase64) and leaves the second passing.

Run locally in the branch: # tests 43 / # pass 43 / # fail 0 across the 10-file storage suite (storage-batched-verification, issue5677-drift-field-diag, issue4323-drift-escalation, issue5408-frozen-recovery, issue5412-async-drift-latch-parity, issue4986-digest-drift-ref-order, storage-bounded-load, storage-canonical-digest, issue4392-canonical-key-order, media-split-digest-parity), plus tsc --noEmit clean, check-test-import-graph, and check-unused-exports clean.

Scope

Two files, one behavior change. src/storage/identity.ts (splitMediaContent) is already a top-level export of the storage layer and has no cycle with shadow.ts (identity.ts is imported by ingest.ts, store.ts, schema.ts, mysql.ts; none import shadow.ts).

This is independent of #6350 (suffix alignment): that fix corrected which slots get compared, this one corrects how a compared slot is hashed. Both were needed on the reporting machine — with only #6350 applied, drift continued on media-bearing slots.

@ranxianglei ranxianglei changed the title fix(storage): hash the split media form on both verification legs, so media-bearing tail slots can converge (Closes #6356) [PR] fix(storage): hash the split media form on both verification legs, so media-bearing tail slots can converge (Closes #6356) Nov 12, 2026
@dimas623

Copy link
Copy Markdown
Contributor Author

(ai: q9router/codebuddy-deepseek-v4-flash)

Live confirmation of the reported shape, measured on the machine that filed #6356 with the patched build running.

I took the session's current row tail (tail_json.tailSeqs), read each slot's stored bytes out of content.bytes, and applied the same substitutions hydrateMedia performs. For every slot whose hydration actually changes the bytes ($media markers AND the CAS row still present), I built the object the envelope holds for that slot (the split form) and ran both comparators against the hydrated row object:

session-3cc98742-71b5-4036-a1ba-79ae3530a9ef
  asym_media_slots_in_tail=1
  OLD comparator matches=0/1
  NEW comparator matches=1/1
  seq=3184 id=h_48f497f178ec6e73 role=tool stored=1991 hydrated=201293 envLen=1991 OLD_match=False NEW_match=True

So on a real slot with intact media the old comparator can never match — 1991 bytes against 201293, confirmed as exactly the got len=201293 in the drift line — and the patched one matches immediately.

Two honest limits on this evidence:

  1. Only 1 such slot survives in the current tail. The other 84 marker slots in that session's tail have lost their CAS rows (cas_missing=84), so hydrateMedia is a no-op for them and no asymmetry is possible. If the CAS payload is gone, the reported drift for that slot is not just a digest-shape problem — it cannot be replayed byte-faithfully at all. Worth calling out as a separate concern, not something this PR claims to fix.
  2. This does not prove the fix on a fresh store by itself, because the regression test is what pins the behavior (tests/media-split-digest-parity.test.ts builds real split bytes, asserts the CAS row contains $media, and requires a clean second sample; the second case rewrites a row's content bytes and requires drift is still reported). The negative control on that test reproduces the reported shape exactly — expect … len=144 vs got … len=4166, fields: imageBase64.

Also worth noting for reviewers: the fixture had to read content.bytes directly rather than going through getMessages(), since getMessages() is the code that hydrates. That is the reason this asymmetry stayed invisible — every other consumer of the row tail sees one representation only.

dimas623 and others added 2 commits November 13, 2026 07:44
The sampled view verifier compared the same message in two different media
representations and marked drift on every sample.

Rows are written media-SPLIT: `planIngest` runs `splitMediaContent`, which
replaces an `imageBase64` payload and every >= 2 KiB data URL with a
`{"$media": hash}` CAS reference. The verifier reads the tail back through
`UnifiedStore.getMessages()`, which returns those rows media-HYDRATED
(`hydrateMedia` inlines the payload again). The envelope's `lastMessages`
keeps the split form, so `messageSignature` hashed a placeholder against the
inlined payload for the SAME message: measured 1,991 vs 201,293 bytes on one
tool result (101x), and ten 390-byte user rows that hydrate to 112-672 KB.

That is a digest-shape mismatch (ranxianglei#4392 family) rather than a content change,
so the sample never converges: the session latches into the ranxianglei#5677 full-ingest
path and transforms its whole tail on every save (measured 8-9 s per save
while latched, starving the event loop).

Normalize both legs to the split form before hashing. `splitMediaContent` is
idempotent on already-split input, so the envelope leg passes through
unchanged, and `splitMediaContent(hydrate(x)) === x` was verified
byte-identical on every media-bearing row sampled, so a genuine edit still
changes the digest and still raises drift.

Tests: new `tests/media-split-digest-parity.test.ts` builds a fixture whose
stored bytes really are split (asserted against the CAS row), folds the
envelope to the newest messages, and requires a clean sample; a second case
rewrites a row's content bytes and requires that the drift is still reported.
Replacing the normalization with the raw JSON fails the first case and
leaves the second passing.
…the unrelated ranxianglei#6351

The two comments introduced by this PR cite ranxianglei#6351, which is an unrelated
open PR (fold before pi's cancelled auto-compaction). The bug they
describe is ranxianglei#6356, which this PR closes.
@ranxianglei
ranxianglei force-pushed the 2026-11-12_media-split-digest branch from 6919eef to 828aed7 Compare November 13, 2026 00:10
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
🔶 需要决策

背景(一句话): 本 PR #6357 修复 #6356(媒体槽位校验永久漂移 → 会话被 latch 后每次保存全量 re-ingest),代码已审核通过、原 head CI 全绿;但同一修复的 commit 也被 cherry-pick 进了在途批量 PR #6464(8 个碎片修复打包,含 #6350/#6357,刚核实仍 open、head da2fa39、3 个测试 lane 红),两个 PR 同时 open,存在双落地冲突。

待决问题: 这个修复走哪个载体落地?

选项:

不决定的默认行为: 两个 PR 都保持 open;谁先合,后合的那个在 shadow.ts 同一段 hunk 上冲突,批量 lane 的合并会被卡住。


查重

查重: 不重复 —— #6357 是 #6356 的原始独立修复 PR,没有其他独立修复 PR;但其 commit 同时出现在在途批量 PR #6464 中(cherry-pick,patch 逐字一致、仅行号偏移),载体二选一即上述待决问题。

Review of PR #6357

@dimas623 — I reviewed the branch, CI, and the mechanism against the storage code. Summary:

The fix is correct. messageSignature now normalizes both verification legs to the media-split form before hashing (splitMediaContent(json).content), which matches how rows are actually written (planIngest → splitMediaContent in src/storage/ingest.ts) while getMessages() returns hydrated rows. splitMediaContent is idempotent on already-split input (a {"$media": ...} object fails the typeof bytes === "string" check and cannot match the ≥2048-byte base64 URL regex), and splitMediaContent(hydrate(x)) === x byte-for-byte when the CAS row exists — so clean sessions verify clean and real media edits still diverge. Both the sync and async verification paths share this one function, so both legs are covered.

The tests are genuine, both directions asserted (gate discipline): test 1 pins "split envelope verifies clean against hydrated rows" (drift stays 0 across two sample points, no latch/freeze); test 2 pins "a real media change still reports drift" (one-shot row corruption via a mocked getSessionRow → verifyDrift >= 1). The Reflect.construct pattern matches the existing tests/storage-batched-verification.test.ts.

What I changed on the branch (additive commits, authorship preserved):

  1. Rebased onto current master — the PR was 85 commits behind; the rebase is clean (master touched no line of shadow.ts; its identity.ts change is a cosmetic let→const refactor of the same function).
  2. 828aed77bd — the code comments cited [PR] feat: fold before pi's cancelled auto-compaction (POST /__bili/plugin/compress) #6351, which is an unrelated open PR ("fold before pi's cancelled auto-compaction", Closes pi plugin: a host sees nothing while the proxy compresses a conversation #6199). The correct reference is [Bug][storage] Sampled verifier hashes split media against hydrated media, so any media-bearing tail slot drifts forever and the session full-ingests every save | 采样校验器把 split 媒体与 hydrated 媒体对比 → 含媒体槽位永久漂移、每次保存全量 ingest #6356 (the bug this fixes). Fixed in src/storage/shadow.ts and the test header.

Verification:

  • Original head 6919eefb: CI 48 check runs — 36 success, 12 skipped, 0 failed.
  • Rebased head 828aed77bd: local npm run typecheck clean; targeted suites media-split-digest-parity + storage-batched-verification + issue5677-drift-field-diag = 10/10 pass. CI on the new head is still running (PR state blocked); I won't claim green until it lands.
  • One nit I checked and left: test 2's corruption UPDATE sets byte_len=4096 while the real JSON is ~4206 bytes — byte_len is accounting-only (never compared in verification), so it's harmless.

Recommended labels (not applied — the mirror's label endpoint is not implemented in this environment, and I don't mutate the source PR's labels): bug, dsh, scope/storage, p1 (chronic permanent latch + event-loop starvation up to ~171s measured; any lane with media in the tail can hit it).

八问

  1. 是什么问题: 采样校验器把同一条消息按两种媒体形态哈希(envelope 腿 split、行腿 hydrated)→ 含媒体槽位永久漂移 → 会话被 latch 后每次保存全量 re-ingest(实测 1.2–1.6s/次,重会话 8–9s/次、~171s 事件循环饥饿)。
  2. 什么原因导致的: digest 形态失配 —— messageSignature 直接哈希 canonical JSON 未做媒体归一;行写入是 split(planIngest→splitMediaContent),读回是 hydrated(getMessages→hydrateMedia),结构性失配,永不收敛。
  3. 影响范围与严重程度: 任何尾部含媒体的 lane/会话(任意 imageBase64 或 ≥2048B data URL);永久 latch 直到宿主侧改写恰好改变尾部;无数据丢失,但慢性每次保存卡顿 + 事件循环饥饿,「能用但体验毁灭」档,定 p1。
  4. 怎么修复的: 两腿哈希前统一归一到 split 形态 —— split 是行实际写入的规范形态,splitMediaContent 对 split 输入幂等、split(hydrate(x))===x 逐字节成立;修的是 digest 输入的产生机制,不是掩码。
  5. 是否有回归问题: 审过:sync+async 两校验腿共用 messageSignature(一致归一);非媒体消息 splitMediaContent 是 no-op(digest 与旧值逐字节相同);真实媒体编辑仍发散(负向测试钉住);漂移诊断 slot.length 改报归一后长度(已披露:旧 = raw canonical JSON 长度,新 = split 后长度;纯媒体编辑时长度相等、sha8 承载信号);wire/config/persistence/version 均未动。无回归。
  6. 是否破坏缓存: 不涉及(存储内部校验 digest,prompt cache/重试/增量游标未触碰)。
  7. 是否改了设计方向: 不涉及(留在 BatchedViewVerification 内,无架构转向)。
  8. 是否引入了新的配置: 不涉及。

结论: 代码正确、测试双向钉住、原 head CI 全绿、rebase 后本地 typecheck + 定向测试 10/10;新 head CI 运行中。可合(待新 head CI 绿 + owner 拍板载体 A/B)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
CI 状态更新(新 head 828aed77bd):镜像缓存显示已记录的 30 个 check run 全部为 success/skipped(18 + 12),0 failed;但 PR 的 mergeable_state 仍为 blocked(完整矩阵在原 head 上是 48 个 run,尚未全部落进缓存),所以还不能宣布全绿——状态翻成 clean 后我会在本线程确认一句。除此之外我这边没有未完成项:代码审核、rebase、注释引用修正(#6351→#6356,commit 828aed77bd)均已完成并推到 PR 分支;本地验证 worktree 已清理。载体 A/B 的决策仍待 owner 拍板(见上一条)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
最终确认:新 head 828aed77bd 的 CI 已全绿——48 个 check run = 36 success + 12 skipped,0 failed(直接查的 check run 列表,镜像缓存当时还停在旧快照)。至此本 PR 我这边全部完成:审核通过、rebase 到当前 master、注释引用修正(828aed77bd)、CI 全绿。唯一剩余事项是 owner 拍板载体 A/B(合 #6357 单发并从 #6464 摘除对应 commit,还是修绿 #6464 后随批量落地)——见线程内 🔶 需要决策 一条。

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6464 (merged) — closing as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants