Repository navigation
[PR] fix(storage): hash the split media form on both verification legs, so media-bearing tail slots can converge (Closes #6356) - #6357
Conversation
|
(ai: q9router/codebuddy-deepseek-v4-flash) Live confirmation of the reported shape, measured on the machine that filed #6356 with the patched build running. I took the session's current row tail ( So on a real slot with intact media the old comparator can never match — Two honest limits on this evidence:
Also worth noting for reviewers: the fixture had to read |
The sampled view verifier compared the same message in two different media
representations and marked drift on every sample.
Rows are written media-SPLIT: `planIngest` runs `splitMediaContent`, which
replaces an `imageBase64` payload and every >= 2 KiB data URL with a
`{"$media": hash}` CAS reference. The verifier reads the tail back through
`UnifiedStore.getMessages()`, which returns those rows media-HYDRATED
(`hydrateMedia` inlines the payload again). The envelope's `lastMessages`
keeps the split form, so `messageSignature` hashed a placeholder against the
inlined payload for the SAME message: measured 1,991 vs 201,293 bytes on one
tool result (101x), and ten 390-byte user rows that hydrate to 112-672 KB.
That is a digest-shape mismatch (ranxianglei#4392 family) rather than a content change,
so the sample never converges: the session latches into the ranxianglei#5677 full-ingest
path and transforms its whole tail on every save (measured 8-9 s per save
while latched, starving the event loop).
Normalize both legs to the split form before hashing. `splitMediaContent` is
idempotent on already-split input, so the envelope leg passes through
unchanged, and `splitMediaContent(hydrate(x)) === x` was verified
byte-identical on every media-bearing row sampled, so a genuine edit still
changes the digest and still raises drift.
Tests: new `tests/media-split-digest-parity.test.ts` builds a fixture whose
stored bytes really are split (asserted against the CAS row), folds the
envelope to the newest messages, and requires a clean sample; a second case
rewrites a row's content bytes and requires that the drift is still reported.
Replacing the normalization with the raw JSON fails the first case and
leaves the second passing.
…the unrelated ranxianglei#6351 The two comments introduced by this PR cite ranxianglei#6351, which is an unrelated open PR (fold before pi's cancelled auto-compaction). The bug they describe is ranxianglei#6356, which this PR closes.
6919eef to
828aed7
Compare
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 背景(一句话): 本 PR #6357 修复 #6356(媒体槽位校验永久漂移 → 会话被 latch 后每次保存全量 re-ingest),代码已审核通过、原 head CI 全绿;但同一修复的 commit 也被 cherry-pick 进了在途批量 PR #6464(8 个碎片修复打包,含 #6350/#6357,刚核实仍 open、head da2fa39、3 个测试 lane 红),两个 PR 同时 open,存在双落地冲突。 待决问题: 这个修复走哪个载体落地? 选项:
不决定的默认行为: 两个 PR 都保持 open;谁先合,后合的那个在 查重查重: 不重复 —— #6357 是 #6356 的原始独立修复 PR,没有其他独立修复 PR;但其 commit 同时出现在在途批量 PR #6464 中(cherry-pick,patch 逐字一致、仅行号偏移),载体二选一即上述待决问题。 Review of PR #6357@dimas623 — I reviewed the branch, CI, and the mechanism against the storage code. Summary: The fix is correct. The tests are genuine, both directions asserted (gate discipline): test 1 pins "split envelope verifies clean against hydrated rows" (drift stays 0 across two sample points, no latch/freeze); test 2 pins "a real media change still reports drift" (one-shot row corruption via a mocked What I changed on the branch (additive commits, authorship preserved):
Verification:
Recommended labels (not applied — the mirror's label endpoint is not implemented in this environment, and I don't mutate the source PR's labels): 八问
结论: 代码正确、测试双向钉住、原 head CI 全绿、rebase 后本地 typecheck + 定向测试 10/10;新 head CI 运行中。可合(待新 head CI 绿 + owner 拍板载体 A/B)。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) |
|
Landed via batch PR #6464 (merged) — closing as superseded. |
What
BatchedViewVerificationcompares the envelope'slastMessagesagainst the same messages read back throughUnifiedStore.getMessages(). Those two legs hold the same message in different media representations, somessageSignaturehashes a placeholder against an inlined payload and marks drift on every sample. This fixes the digest by normalizing both legs to the split form, which is the form the rows are actually written in.Closes #6356.
Why it matters
Rows are written media-split (
planIngest->splitMediaContent: animageBase64payload and every data URL of 2048+ bytes become{"$media":"<sha256>"}CAS references), butgetMessages()returns them media-hydrated (hydrateMediainlines the payload again). The envelope keeps the split form.Measured on a live store: one tool result is
1991bytes stored and201293bytes hydrated — a 101x difference for the same message id, same role, onlytextdiffering. Ten 390-byte user rows hydrate to 112 KB–672 KB.Because the mismatch is structural rather than a content change, the sample never converges:
driftedSessions);Cost while latched, measured on the reporting machine:
slow-ingest: … path=fullat 1.2–1.6 s per save sustained on a ~3.5k-row session, and 8–9 s per save on a heavier one (24 consecutive full ingests, ~171 s of event-loop starvation) until a host-side history rewrite happened to change the tail.How
splitMediaContentis idempotent: a{"$media": …}marker does not match its data-URL regex, so an already-split envelope passes through byte-identical. And becausesplitMediaContent(hydrate(x)) === xon every media-bearing row sampled, a genuine media edit still changes the digest — the comparison keeps its detection power instead of being skipped.Not the flagged alternative
Gating on
tail.tailFoldedwas tried first and rejected:planIngestwritestailFoldedfromsession.lastMessagesFolded, which is only set byfoldResidentTail, while the envelope setsmessagesFoldedfrombuildRecordwhenever a bounded snapshot exists. On both live sessionstailFoldedreadfalsewhile the envelope was in fact a folded suffix, so the guard would never fire. Skipping the comparison would also stop detecting real divergence inside the window.Tests
New
tests/media-split-digest-parity.test.ts:imageBase64messages, asserts the stored CAS row really contains$media(readingcontent.bytesdirectly, sincegetMessages()is the code that hydrates), folds the envelope to the newest messages, and requiresverifyDrift === 0with no latch and no envelope freeze across the second sample (VERIFY_EVERY_INGESTS= 20).contentbytes to different media, then requiresverifyDrift >= 1and a latched session.Negative control: replacing
splitMediaContent(json).contentwith the rawjsonfails the first case with exactly the reported shape (expect … len=144vsgot … len=4166,fields: imageBase64) and leaves the second passing.Run locally in the branch:
# tests 43 / # pass 43 / # fail 0across the 10-file storage suite (storage-batched-verification,issue5677-drift-field-diag,issue4323-drift-escalation,issue5408-frozen-recovery,issue5412-async-drift-latch-parity,issue4986-digest-drift-ref-order,storage-bounded-load,storage-canonical-digest,issue4392-canonical-key-order,media-split-digest-parity), plustsc --noEmitclean,check-test-import-graph, andcheck-unused-exportsclean.Scope
Two files, one behavior change.
src/storage/identity.ts(splitMediaContent) is already a top-level export of the storage layer and has no cycle withshadow.ts(identity.tsis imported byingest.ts,store.ts,schema.ts,mysql.ts; none importshadow.ts).This is independent of #6350 (suffix alignment): that fix corrected which slots get compared, this one corrects how a compared slot is hashed. Both were needed on the reporting machine — with only #6350 applied, drift continued on media-bearing slots.