Repository navigation
[PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) - #4685
Conversation
…adow drift (ranxianglei#4392) shadowDigest hashes JSON.stringify(canonicalize(view)) and canonicalize walked Object.entries in insertion order, so two views holding equal values hashed apart whenever their key order differed. The verify legs really do differ: the legacy load leg carries state.messageRefs.byRef in kernel mint order while loadLegacyView rebuilds it from the refs ledger in ord order (ingest.ts:530). One process therefore logged [diverged: state] sample after sample and escalated into freezing its own JSON envelope writes. canonicalize now sorts object keys at every depth — arrays keep their order and Map/Set keep their __map__/__set__ form. The ESCALATED wording no longer asserts MULTIPLE instances: both legs are built in this process, so a digest shape mismatch is the first suspect and a second writer is one cause among others (ranxianglei#394, ranxianglei#4392). Pins move because canonicalize is what they hash. storage-render goldens were re-generated with scripts/update-storage-render-goldens.ts; the storage-corpus manifest digests were re-pinned from the untouched fixtures, because that script also rewrites every fixture with newer buildRecord fields — churn unrelated to the digest contract, left out of this PR. tests/issue4392-canonical-key-order.test.ts fails before the fix (2 of its 4 cases) and passes after. CHANGELOG records the single-instance shape of the drift.
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。全仓(含已关闭)只有本 PR 触碰 结论先行:方向对、根因定位准、修法最小且正确。本地已验证:新回归测试 4/4 绿;把 ① 是什么问题 ② 什么原因(根因) ③ 影响范围与严重程度 ④ 怎么修复的 ⑤ 是否有回归
⑥ 是否破坏缓存 ⑦ 是否改了设计方向 ⑧ 是否引入新配置 代码质量(外部提交从严):命名/注释达意且讲"为什么"; 测试有效性:强。新测试驱动真实 diff 整洁度(拦点): 合并判定:可合,但需先修上述 manifest 缩进。修完即为干净的单模块 fix(不过 auto-merge 门:动了金标 digest 定义 = 契约变更 + 外部提交需人审),由 owner 人工合并。 |
The regenerated manifest was rewritten with 4-space indentation while the generator scripts/update-storage-corpus.ts:467 writes JSON.stringify(x,null,2), so every regeneration re-churns all 158 lines and buries the real change in the diff. Restored the original key order + 2-space layout, keeping only the new digest values (deep-diff verified: exactly the 10 digest fields differ).
(ai: vllm-qwen/qwen3.8-27b) CI 已处理完毕,分支已推到 证据(本 PR head e83fc64 run 40187774233 vs master ac844fa 自己的最新 run 40189242801):
反向验证:master 上红的 storage-render golden ×4(chat/anthropic/responses/google)与「#1365 apply() attach mode」在本分支上全部 ok——本 PR 的排序键摘要改动恰好让 master 这一族转绿。本分支比 master 更绿,零新增红。 继承红的处置按 #4599 政策已留一行协调评论(master 哨兵已触顶停手,等 owner 决策,不重复排查)。 顺手修复(独立 commit d620095): 八问:
判定:可合——待新 CI(040f30e)确认;预期剩余结果 = master 现状(家族 A/B/C 若仍未修则仍会红,届时按 #4599 协调即可,与本 PR 无关)。 |
# Conflicts: # tests/golden/storage-render/anthropic.json # tests/golden/storage-render/chat.json # tests/golden/storage-render/google.json # tests/golden/storage-render/responses.json
… table (ranxianglei#4392) canonicalize hashed insertion order, so the shadow live leg and the replay leg (messageRefs.byRef rebuilt in ledger order by loadLegacyView) produced two digests for the same value on a SINGLE instance, escalated after three samples and froze envelope writes. - sort object keys at every depth; arrays keep order, Map/Set envelopes keep their shape - move the storageMode/tailGaps drops into one exported table DIGEST_SKIPPED_KEYS so the streamed canonicalDigest (ranxianglei#4532) imports the same skip set instead of hand-copying a second list - pin the table literally and pin depth-independent dropping
…alize (ranxianglei#4392) Only the 10 digest values change (9 cases + twinfile flatFallbackDigest); file lists and the generator 2-space format are untouched. storage-render goldens need no change — that drive already sorts keys harness-side.
|
Synced with master and reworked per the review and the #4392 ruling. New head 1. Master merged ( 2. storage-render goldens do not change (correction). The previous round re-pinned them; that was wrong. 3. Corpus manifest: digest values only. Regenerating the corpus rewrote every case (versions, 4. One skip set for every digest producer. The two inline export const DIGEST_SKIPPED_KEYS: Readonly<Record<string, true>> = { storageMode: true, tailGaps: true };The loop walks 5. Tests ( A/B on the same two views (equal content, different key order): master 6. Gates on the new head
The 11 local failures ( |
(ai: vllm-qwen/qwen3.8-27b) 对新 head 逐项核验结果
既有行为清点(回归关):shadowDigest 判定(顺序差不再算 drift=本修复目标;真实差异仍算=负控钉住)/ 升级 WARN 文案(有意变更,旧「usually means MULTIPLE bili instances」→ 新「digest-shape mismatch first, second writer 是原因之一」,CHANGELOG 已披露;面板消费的是结构化 八问
CI:新 head 正在跑,当前 0 红(static-gates ubuntu、mysql-contract 已绿,12 项排队中)。 判定:可合——待 CI 全绿即可,无遗留阻塞项。 |
(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: merged origin/master |
… parsing (ranxianglei#3203) waitForFile gated on existence only. The child fixture creates out-*.json before its content lands (a zero-byte window that widens under Windows CI load with four concurrent tsx spawns), so the parent could JSON.parse an empty file: SyntaxError 'Unexpected end of JSON input' (run 40555535547, shard windows-22-4). Gate on non-empty size stable across two polls; the slot-distinctness assertion itself is unchanged.
(ai: vllm-qwen/qwen3.8-27b) CI 修复完成(第 2/2 轮自动修复) — 本轮 head
判定:无需决策升级(几行级显而易见修复,按政策直接在 PR 分支修掉)。等 |
# Conflicts: # CHANGELOG.md
|
Merged master |
…est from ranxianglei#4997) ranxianglei#4997 opened for ranxianglei#4986 the same root cause ranxianglei#4685 fixes in canonicalize. Its regression test fails on master b08fbf3 (DIGEST DRIFT for corpus-folded) and passes here without the shadowDigest change, so it carries over unchanged.
|
Update: the head is now
Typecheck is clean, and the two digest test files pass 6/6. Details are in #4997. |
(ai: vllm-qwen/qwen3.8-27b) 新 head
判定:新 head |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.910-master.38965bc # this exact build
npm install -g billion-context@master # always the newest merged build
|
…xianglei#4532) ranxianglei#4685 landed on master (canonicalize sorts object keys at every depth + shared DIGEST_SKIPPED_KEYS table), but this PR's streaming canonicalDigest still iterated raw Object.keys order with its own literal skip list. The two producers then disagreed for any object whose keys arrive in a different order — the false DIGEST DRIFT of ranxianglei#4392/ranxianglei#4986 again, this time between the streaming digest and canonicalize; tests/issue4392-canonical-key-order.test.ts went red on 8 shards (e.g. run 40909901398). Align visit() with the legacy producer's effective byte order: - skip via Object.hasOwn(DIGEST_SKIPPED_KEYS, k), keeping the __proto__ guard; - iterate array-index keys first in ascending numeric order, then the remaining keys lexicographically. Plain .sort() alone is NOT byte-identical: spec enumeration (Object.keys / JSON.stringify) emits array-index keys ascending numeric regardless of insertion or lexicographic order, so an index-shaped pair like {"10","2"} must serialize as "2","10". The pinned byte-identity test (streaming digest preserves JSON omissions, sparse arrays and property order) catches exactly this. Byte-identity to sha256(JSON.stringify(canonicalize(v))) now holds for any key arrival order, in either merge order relative to ranxianglei#4685. Verified locally: issue4392/issue4986/storage-canonical-digest suites 13/13, storage parity integration (engine-proofs + ranxianglei#3407 + ranxianglei#3919 + corpus) 26/26, typecheck clean.
…tion alarm (Follows #4614) e83fc64 (#4685) rewrote this alarm line for the #4392 digest-shape-first wording and dropped the #4614 clause in the same edit, so every test lane failed tests/issue4323-drift-escalation.test.ts:193 ('alarm also names the single-writer repeating-rewrite producer') from the merge onward. Restore both producers in one line — digest-shape mismatch first (#4392), then the writer side: a second bili instance on one sessions dir (#394/#4177) or ONE host re-writing history slots below the ingest watermark (#4614).
What & why
shadowDigesthashesJSON.stringify(canonicalize(view))(
src/storage/shadow.ts:1166), andcanonicalizewalked objects withObject.entries, i.e. it hashed insertion order(
src/storage/canonical.ts:41). The two legs the shadow verifier compares buildthe same values in different key orders, so a single instance reported drift
against itself:
src/storage/shadow.ts:379) = the legacy load path,legacyLoad → buildSession → mergeState;src/storage/shadow.ts:380) =loadLegacyView, which rebuildsstate.messageRefsfrom the refs ledger —byRawkeys come out in ledgerordorder (src/storage/ingest.ts:338,:541),byRefin first-winsvisit order (
:546).Every 20th ingest therefore logged
[diverged: state], three consecutivesamples escalated to
frozenEnvelopeIds.add(...)(
src/storage/shadow.ts:404) plus a watermark clear (:426), and theescalation text asserted "This usually means MULTIPLE bili instances share one
sessions dir" (
src/storage/shadow.ts:413) — pointing a single-process user atthe #394 anti-pattern.
Proof that the two legs differ (single process, no second writer)
Loading three real envelopes with
SessionStore.loadSyncin one fresh processand comparing
toLegacyLike(session)againstloadLegacyView(planIngest(...)):messageRefs.byRefmessageRefs.byRef<session-a>m00514, m00515, m00510, m00527, m00528, m00531, …m00001, m00002, …<session-b>m00267, m02251, …m02882, m00052, …Identical key sets, identical values, different key order — and the repo's own
verifier fired
[storage] unified shadow DIGEST DRIFT … [diverged: state]onthose very envelopes.
tests/issue4392-canonical-key-order.test.tspins this: it builds both viewsfor one session, asserts the live leg mints
[m00002, m00001]while the replayleg rebuilds
[m00001, m00002], then asserts the shadow verifier samples onceand drifts zero times.
Change
canonicalizesorts object keys at every depth (src/storage/canonical.ts).Arrays keep their order (order is content there);
Map/Setkeep their__map__/__set__form; thestorageMode/tailGapsdrop,__VER__,__TS__and__UUID__masks are unchanged.legs are built in this process, so a persistent gap is a digest-shape
mismatch first (a field only one leg builds, or ordering) and a second
writer is one cause among others (无单实例守卫:多个 bili 代理进程并存共写 sessions 目录,伴随孤儿 ref、有注册表无 transcript、计数自相矛盾的污染簇 #394, [Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392).
Pins
canonicalizeis exactly what the pins hash, so they move:tests/golden/storage-render/*.json— regenerated with the documentednode --import tsx scripts/update-storage-render-goldens.ts; the diff isstateDigest+stateKeyDigestsonly (4 wires × 4 lines). Turn hashes,landmarks and the render shape are unchanged.
tests/golden/storage-corpus/manifest.json— the 9 case digests (and thetwinfileflat-fallback digest) were re-pinned from the untouchedfixtures.
scripts/update-storage-corpus.tswas not used wholesale because italso rewrites every fixture file with newer
buildRecordfields — churnunrelated to the digest contract; the fixtures stay byte-identical here.
Test evidence
node --import tsx --test tests/issue4392-canonical-key-order.test.tsbyRef reordered by the ref-ledger rebuild …and the canonical
key sets, not insertion ordercase), 2 pass (thearrays/Set controls);
Affected storage suites (all green after the fix):
tests/storage-corpus.test.ts,tests/storage-unified.test.ts,tests/storage-replicas.test.ts,tests/storage-ingest-watermark.test.ts,tests/issue3407-shadow-blockcontents-order.test.ts,tests/issue3919-shadow-drift-forensics.test.ts,tests/issue4323-drift-escalation.test.ts,tests/golden-storage-render.test.ts— 64 tests, 0 failures.tsc --noEmitclean.Pre-existing on
masterand unrelated (verified with and without this change):tests/storage-engine-proofs.test.ts"full-mode restart is invisible on thewire" fails on
outbound body #13 digest drift vs legacy restartforall four wires.
Risk
Digest-only change: no wire bytes, no persistence bytes, no load semantics move.
It makes verification less sensitive to key order, which is what #3113
already asks of these pins ("pin logical content, not serialization order"), and
it removes a false positive that freezes writes.
Fixes #4392
Code references (at
master,d1b46fc0)canonicalizewalkedObject.entries: https://github.com/ranxianglei/billion-context/blob/d1b46fc0/src/storage/canonical.ts#L41shadowDigest=sha256(JSON.stringify(canonicalize(view))): https://github.com/ranxianglei/billion-context/blob/d1b46fc0/src/storage/shadow.ts#L1166messageRefsrebuilt from the ledger: https://github.com/ranxianglei/billion-context/blob/d1b46fc0/src/storage/ingest.ts#L338 , https://github.com/ranxianglei/billion-context/blob/d1b46fc0/src/storage/ingest.ts#L541 , https://github.com/ranxianglei/billion-context/blob/d1b46fc0/src/storage/ingest.ts#L546