Skip to content

[PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) - #4685

Merged
ranxianglei merged 13 commits into
ranxianglei:masterfrom
Maziar123:2026-10-25_digest-key-order
Oct 29, 2026
Merged

ranxianglei merged 13 commits into
ranxianglei:masterfrom
Maziar123:2026-10-25_digest-key-order

Conversation

@Maziar123

Copy link
Copy Markdown
Contributor

What & why

shadowDigest hashes JSON.stringify(canonicalize(view))
(src/storage/shadow.ts:1166), and canonicalize walked objects with
Object.entries, i.e. it hashed insertion order
(src/storage/canonical.ts:41). The two legs the shadow verifier compares build
the same values in different key orders, so a single instance reported drift
against itself:

  • expect leg (src/storage/shadow.ts:379) = the legacy load path,
    legacyLoad → buildSession → mergeState;
  • replay leg (src/storage/shadow.ts:380) = loadLegacyView, which rebuilds
    state.messageRefs from the refs ledger — byRaw keys come out in ledger
    ord order (src/storage/ingest.ts:338, :541), byRef in first-wins
    visit order (:546).

Every 20th ingest therefore logged [diverged: state], three consecutive
samples escalated to frozenEnvelopeIds.add(...)
(src/storage/shadow.ts:404) plus a watermark clear (:426), and the
escalation text asserted "This usually means MULTIPLE bili instances share one
sessions dir" (src/storage/shadow.ts:413) — pointing a single-process user at
the #394 anti-pattern.

Proof that the two legs differ (single process, no second writer)

Loading three real envelopes with SessionStore.loadSync in one fresh process
and comparing toLegacyLike(session) against loadLegacyView(planIngest(...)):

session live messageRefs.byRef replay messageRefs.byRef
<session-a> m00514, m00515, m00510, m00527, m00528, m00531, … m00001, m00002, …
<session-b> m00267, m02251, … m02882, m00052, …

Identical key sets, identical values, different key order — and the repo's own
verifier fired [storage] unified shadow DIGEST DRIFT … [diverged: state] on
those very envelopes.

tests/issue4392-canonical-key-order.test.ts pins this: it builds both views
for one session, asserts the live leg mints [m00002, m00001] while the replay
leg rebuilds [m00001, m00002], then asserts the shadow verifier samples once
and drifts zero times.

Change

Pins

canonicalize is exactly what the pins hash, so they move:

  • tests/golden/storage-render/*.json — regenerated with the documented
    node --import tsx scripts/update-storage-render-goldens.ts; the diff is
    stateDigest + stateKeyDigests only (4 wires × 4 lines). Turn hashes,
    landmarks and the render shape are unchanged.
  • tests/golden/storage-corpus/manifest.json — the 9 case digests (and the
    twinfile flat-fallback digest) were re-pinned from the untouched
    fixtures. scripts/update-storage-corpus.ts was not used wholesale because it
    also rewrites every fixture file with newer buildRecord fields — churn
    unrelated to the digest contract; the fixtures stay byte-identical here.

Test evidence

node --import tsx --test tests/issue4392-canonical-key-order.test.ts

  • before the fix: 2 of 4 fail (byRef reordered by the ref-ledger rebuild …
    and the canonical key sets, not insertion order case), 2 pass (the
    arrays/Set controls);
  • after the fix: 4 pass.

Affected storage suites (all green after the fix):
tests/storage-corpus.test.ts, tests/storage-unified.test.ts,
tests/storage-replicas.test.ts, tests/storage-ingest-watermark.test.ts,
tests/issue3407-shadow-blockcontents-order.test.ts,
tests/issue3919-shadow-drift-forensics.test.ts,
tests/issue4323-drift-escalation.test.ts,
tests/golden-storage-render.test.ts — 64 tests, 0 failures.
tsc --noEmit clean.

Pre-existing on master and unrelated (verified with and without this change):
tests/storage-engine-proofs.test.ts "full-mode restart is invisible on the
wire" fails on outbound body #13 digest drift vs legacy restart for
all four wires.

Risk

Digest-only change: no wire bytes, no persistence bytes, no load semantics move.
It makes verification less sensitive to key order, which is what #3113
already asks of these pins ("pin logical content, not serialization order"), and
it removes a false positive that freezes writes.

Fixes #4392

Code references (at master, d1b46fc0)

…adow drift (ranxianglei#4392)

shadowDigest hashes JSON.stringify(canonicalize(view)) and canonicalize walked
Object.entries in insertion order, so two views holding equal values hashed
apart whenever their key order differed. The verify legs really do differ: the
legacy load leg carries state.messageRefs.byRef in kernel mint order while
loadLegacyView rebuilds it from the refs ledger in ord order (ingest.ts:530).
One process therefore logged [diverged: state] sample after sample and
escalated into freezing its own JSON envelope writes.

canonicalize now sorts object keys at every depth — arrays keep their order and
Map/Set keep their __map__/__set__ form. The ESCALATED wording no longer
asserts MULTIPLE instances: both legs are built in this process, so a digest
shape mismatch is the first suspect and a second writer is one cause among
others (ranxianglei#394, ranxianglei#4392).

Pins move because canonicalize is what they hash. storage-render goldens were
re-generated with scripts/update-storage-render-goldens.ts; the storage-corpus
manifest digests were re-pinned from the untouched fixtures, because that
script also rewrites every fixture with newer buildRecord fields — churn
unrelated to the digest contract, left out of this PR.

tests/issue4392-canonical-key-order.test.ts fails before the fix (2 of its 4
cases) and passes after.

CHANGELOG records the single-instance shape of the drift.
@ranxianglei ranxianglei changed the title fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) Oct 25, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。全仓(含已关闭)只有本 PR 触碰 canonicalize 键序修复;源 bug 仅 #4392。#4677/#4684 虽同属 storage 域,但机制不同(v8 identity backfill 流式化),非重复。分类:scope/storage + p2(影子漂移冻结 → 降级 unified-only,有绕行、非数据丢失)。

结论先行:方向对、根因定位准、修法最小且正确。本地已验证:新回归测试 4/4 绿;把 canonicalize 改动 revert 掉后 test1/test3 变红(证明测试有牙);受影响 8 个 storage 套件 59 绿 0 红;tsc --noEmit 干净。唯一拦点:manifest.json 被整文件从 2 空格重排成 4 空格,与其自身生成器冲突(scripts/update-storage-corpus.ts:463 用 JSON.stringify(manifest, null, 2)),158 行空白噪音盖住了真正只改的 10 个 digest hex。请恢复原 2 空格缩进、只替换 digest 值后再合。


① 是什么问题
单实例在 shadow 模式下自我报告 [diverged: state],连续 3 次采样即冻结本进程的信封写入并清入库水位——实际并没有第二个 bili 实例。旧升级文案把它归因为"多个实例共享 sessions 目录"(#394),误导排查方向。

② 什么原因(根因)
摘要定义把插入序当成了契约的一部分。shadowDigest = sha256(JSON.stringify(canonicalize(view))),canonicalize 按 Object.entries(插入序)展开对象。两条腿对同一份等值状态用不同顺序构造键:expect 腿走 kernel live 状态(byRef 按发号序),replay 腿走 loadLegacyView 按账本 ord 重建 messageRefs.byRef(src/storage/ingest.ts:338/:541/:546)。等值内容、不同键序 → 两个不同哈希 → 假漂移。这是摘要形状问题,不是多写者问题。

③ 影响范围与严重程度
凡触发 ref-ledger 重对齐(折叠/ref 重分配)后 replay 腿重建 byRef 的会话:单进程自我漂移 → 冻结信封写入 → 降级为 unified-only(文件 envelope 停写,unified store 仍收每次保存)。能用但有明显降级(p2),非数据丢失。触发条件:带折叠历史 + 定期 ingest 采样的会话。

④ 怎么修复的
canonicalize 对普通对象按键排序后再参与哈希(任意深度);数组保序(order 是内容)、Map/Set __map__/__set__ 形态不变、storageMode drop 与 __VER__/__TS__/__UUID__ mask 均不变。等值状态现在无论构造顺序都同哈希,消除假漂移;真实内容差异仍会漂移(test 4 钉住)。同时把升级文案从"断言多实例"改为"先查摘要形状、多写者只是可能原因之一",不再误导。

⑤ 是否有回归
审过 canonicalize 的全部既有行为:唯一消费者是 src/storage/shadow.ts(其余同名 canonicalize 是无关局部函数);两条腿都是每采样现算、无跨进程持久化 → 无升级期失配风险。逐项判定:

  • 排序对键集是单射,不会把两个真不同的状态判成相等 → 无新假阴性(test 4 证);
  • mask / storageMode drop / array / Map·Set 语义均不变(test 2 证);
  • 金标重钉仅动 digest 字段:storage-render 四个文件只改 stateDigest/metadata(不动 turn hash/landmark/render 形状),corpus manifest 重钉 9 case + flatFallbackDigest;storage-corpus / golden-storage-render 套件绿证渲染与入库语义未变。
    未发现回归。

⑥ 是否破坏缓存
不涉及。digest 是内存态影子对账量,不落盘、不参与任何前缀缓存/重试/增量游标键;排序只改变内部对账哈希值,不影响出站 wire 或客户端前缀缓存。

⑦ 是否改了设计方向
不改架构/边界/数据流。只收紧了 "canonical form" 的定义(插入序 → 排序键),使契约更诚实(顺序本就不是持久化契约的一部分)。属契约澄清,非转向。

⑧ 是否引入新配置
不涉及。无新 env/config/默认值。唯一可观测变化是金标 digest 的 hex 值随定义变化(旧→新已在 CHANGELOG 披露),纯内部、用户不可见。


代码质量(外部提交从严):命名/注释达意且讲"为什么";as Record<string, unknown> 比被禁的 as any 更窄、合理;无死代码/魔法数;风格一致。小瑕疵(不拦):文件头 doc 注释与循环内联注释几乎逐字重复,可留其一。

测试有效性:强。新测试驱动真实 loadLegacyView + digest 路径,双向钉住 —— 正向(byRef 重排不再漂移)+ 负向(真实内容差异仍漂移),非恒真/mock-only。我本地 A/B:fix 在 = 4/4 绿;revert 掉 canonicalize 改动 = test1(核心键集断言)、test3(byRef 重排)红,test2/test4 仍绿 —— 证明真抓得住这个 bug。

diff 整洁度(拦点):manifest.json 整文件 2→4 空格重排,与生成器 null, 2 冲突,158 行空白淹没 10 处真实 digest 变更,且下次跑 update-storage-corpus.ts 会再 churn 一次。请恢复原 2 空格格式、只替换 digest hex。(storage-render 四个文件没这个问题 —— 它们保留了原生缩进、只有 digest 行变动。)

合并判定:可合,但需先修上述 manifest 缩进。修完即为干净的单模块 fix(不过 auto-merge 门:动了金标 digest 定义 = 契约变更 + 外部提交需人审),由 owner 人工合并。

ci-check added 2 commits October 25, 2026 21:00
The regenerated manifest was rewritten with 4-space indentation while the
generator scripts/update-storage-corpus.ts:467 writes JSON.stringify(x,null,2),
so every regeneration re-churns all 158 lines and buries the real change in
the diff. Restored the original key order + 2-space layout, keeping only the
new digest values (deep-diff verified: exactly the 10 digest fields differ).
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已处理完毕,分支已推到 040f30e1a(等待新 CI)。结论先行:7 个红 shard 的失败没有一条来自本 PR——全部是 master 自带的继承性红;顺带把上一轮 review 指出的唯一拦路项(manifest 缩进重排)按生成器格式修掉了。

证据(本 PR head e83fc64 run 40187774233 vs master ac844fa 自己的最新 run 40189242801):

失败项 本 PR master
plugin-fork「HTTP fork revision and originals survive…」×3 u24s3 / u22s3 / w22s4 红 同三项红(已知家族 A,#4652 在途未合)
storage-engine-proofs full-mode restart ×4 wire + rollback contract + corrupt index.db u24s3 / u22s3 / w22s4 红 同 shard 全红(家族 B)
orphan-recovery「cold load recovers the pre-fix orphan freeze…」×3 u22s2 / u24s2 / w22s3 红 同三项红(家族 C)
web-blogfeed「default probing finds the repository tree…」×1 w22s2 红(bundled meet-bili post found,tests/web-blogfeed.test.ts:150) 同 shard w22s2 同条红(not ok 1414)

反向验证:master 上红的 storage-render golden ×4(chat/anthropic/responses/google)与「#1365 apply() attach mode」在本分支上全部 ok——本 PR 的排序键摘要改动恰好让 master 这一族转绿。本分支比 master 更绿,零新增红。 继承红的处置按 #4599 政策已留一行协调评论(master 哨兵已触顶停手,等 owner 决策,不重复排查)。

顺手修复(独立 commit d620095):tests/golden/storage-corpus/manifest.json 整文件从 2 空格被重排成 4 空格,与生成器 scripts/update-storage-corpus.ts:467(JSON.stringify(x, null, 2))冲突——每次重新生成都会把 158 行全部churn,淹没真实改动。已用脚本深比对重建:新旧两版仅 10 个 digest 值不同(9 个 case digest + twinfile flatFallbackDigest),其余字节级一致;恢复原键序 + 2 空格 + 尾换行后,该文件相对 base 的 diff 收敛为纯 20 行 digest 值变更。本地 typecheck 干净,定向套件 storage-corpus / issue4392-canonical-key-order / golden-storage-render 共 18/18 通过。

八问:

  1. 是什么问题:CI 七红看起来像 PR 引入回归,实际全是 master 继承性失败;另有 PR diff 里一处格式噪音(非功能)。
  2. 什么原因导致的:master 上多组既有测试 born-red(见上表,CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 有总账);manifest 噪音是作者用与仓库生成器不同的缩进设置重新导出了 golden。
  3. 影响范围与严重程度:继承红波及所有基于当前 master 的 PR(严重,但非本 PR 责任,已有 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 统一跟踪);manifest 重排只影响 diff 可读性,无行为影响。
  4. 怎么修复的:分支同步到当前 master(ac844fa;ci-fixes 已是其祖先,按政策无需另合)+ 按生成器契约重建 manifest 保留新 digest 值。
  5. 是否有回归问题:无。typecheck 干净 + 定向 18/18 过;且本分支让 master 红的 storage-render golden 族转绿,方向是减红不是加红。
  6. 是否破坏缓存:不涉及。shadowDigest 是状态一致性自检值,不参与缓存/重试/游标/去重键;本轮新增改动仅分支同步与测试夹具格式。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:可合——待新 CI(040f30e)确认;预期剩余结果 = master 现状(家族 A/B/C 若仍未修则仍会红,届时按 #4599 协调即可,与本 PR 无关)。

# Conflicts:
#	tests/golden/storage-render/anthropic.json
#	tests/golden/storage-render/chat.json
#	tests/golden/storage-render/google.json
#	tests/golden/storage-render/responses.json
… table (ranxianglei#4392)

canonicalize hashed insertion order, so the shadow live leg and the
replay leg (messageRefs.byRef rebuilt in ledger order by loadLegacyView)
produced two digests for the same value on a SINGLE instance, escalated
after three samples and froze envelope writes.

- sort object keys at every depth; arrays keep order, Map/Set envelopes
  keep their shape
- move the storageMode/tailGaps drops into one exported table
  DIGEST_SKIPPED_KEYS so the streamed canonicalDigest (ranxianglei#4532) imports the
  same skip set instead of hand-copying a second list
- pin the table literally and pin depth-independent dropping
…alize (ranxianglei#4392)

Only the 10 digest values change (9 cases + twinfile flatFallbackDigest);
file lists and the generator 2-space format are untouched. storage-render
goldens need no change — that drive already sorts keys harness-side.
@Maziar123

Copy link
Copy Markdown
Contributor Author

Synced with master and reworked per the review and the #4392 ruling. New head 0b1a26cb.

1. Master merged (9ef49ccc, master bcbd1a3d). The only conflicts were tests/golden/storage-render/{anthropic,chat,google,responses}.json; master's side taken. Nothing under kernel/ in the PR diff.

2. storage-render goldens do not change (correction). The previous round re-pinned them; that was wrong. tests/golden-storage-drive.ts:603 and :616 already call canonicalize(sortObjectKeys({...})), so sorting inside canonicalize is a no-op for those digests. Regenerated values equal master's byte for byte, and git diff origin/master -- tests/golden/storage-render/ is empty.

3. Corpus manifest: digest values only. Regenerating the corpus rewrote every case (versions, storageMode, stats, extra index.db*), so that is reverted. The 10 digests (9 cases + twinfile flatFallbackDigest) are recomputed from the committed fixtures with the test's own sessionDigest. The manifest diff is 20 lines; file lists, key order and the generator's JSON.stringify(manifest, null, 2) + "\n" format are unchanged.

4. One skip set for every digest producer. The two inline continues (storageMode, tailGaps) are now one exported table:

export const DIGEST_SKIPPED_KEYS: Readonly<Record<string, true>> = { storageMode: true, tailGaps: true };

The loop walks Object.keys(record).sort() and skips with Object.hasOwn(DIGEST_SKIPPED_KEYS, k) (hasOwn, so a __proto__ key is never read as skipped). canonicalDigest lives in #4532 and is not on master, so this PR does not copy it; #4532 can import the table. The hunk and a two-path consistency test are in my review on #4532.

5. Tests (tests/issue4392-canonical-key-order.test.ts, 5 cases): equal values in different key order give one digest, while a changed value or key set changes it; arrays keep order and Map/Set envelopes are unchanged; the skip table is pinned ({storageMode, tailGaps}, any depth, biliVersion masked not dropped); end to end, a byRef rebuilt in ledger order no longer drifts; a real messageRefs difference still drifts.

A/B on the same two views (equal content, different key order): master bcbd1a3d gives d13c991d841cc9e0 vs 7cbbeb231bb82aef; this branch gives 6105407047a23129 for both. The new test fails on master (does not provide an export named 'DIGEST_SKIPPED_KEYS').

6. Gates on the new head

Gate Result
npm run typecheck exit 0
node scripts/check-unused-exports.mjs clean (6 waived)
node scripts/check-test-import-graph.mjs clean
issue4392 + storage-corpus + golden-storage-render 19/19 pass
full npm test (local) 9414 tests, 11 fail

The 11 local failures (issue3304-incremental-direct-send, issue4725-loop-429-observe, issue4743-degenerate-refetch-429, opencode-native, opencode-v2, pi-native) fail identically on a clean origin/master worktree in the same shell; they depend on host detection inside an agent shell and are green in CI. The four families behind the 7 red shards on the old head (plugin-fork HTTP fork revision, storage-engine-proofs full-mode restart, orphan-recovery cold load, web-blogfeed repository probing) pass locally after the merge, and master's ci.yml at bcbd1a3d (run 40525463444) is green.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

对新 head 0b1a26cbb 做了独立核验(不是照抄你的清单),逐条对过代码与本地实测:

逐项核验结果

  1. diff 范围:vs base bcbd1a3df 恰好 5 个文件——CHANGELOG(+1)、canonical.ts、shadow.ts、corpus manifest(20 行)、issue4392 测试(+212)。无 kernel/,无锁文件。✓
  2. storage-render 金标不动:实测 git diff bcbd1a3df..HEAD -- tests/golden/storage-render/ 为空;并读了 tests/golden-storage-drive.ts:603/:616——harness 侧确实先 sortObjectKeys({...}) 再 canonicalize(注释还记着 会话详情面板:运行中的会话一律显示 legacy(仅 JSON)存储、代理版本 —,落盘后才恢复正常(实际存储正常) #3093 重排曾把四个金标打红),所以排键对那套摘要是 no-op。你撤回上轮误钉是对的,我上轮「本分支让 master 的 storage-render 族转绿」的说法基于旧 base,作废。✓
  3. manifest 纯度:20 行变更逐行 grep,非 digest 值行为 0;键序/格式/文件清单不变(与生成器 JSON.stringify(x,null,2)+"\n" 一致,我上轮修的缩进问题被你这条路线等价解决)。✓
  4. 单一跳过表:canonical.ts 里旧的两个内联 continue 已收敛为 DIGEST_SKIPPED_KEYS,Object.hasOwn 守卫正确(__proto__ 不会被误判为跳过项);全仓运行时 importer 只有 src/storage/shadow.ts:32 一处,不存在第三份手写跳过表。诊断用的 top-level key-diff helper(shadow.ts:1268)仍内联跳 tailGaps——它是键差报告器不是摘要生产者,且只在真实 drift 时运行,不构成漂移风险,不拦。(顺带:[PR] fix: stream storage verification digests to reduce peak memory #4532 那边的引用评论我看到了,跨 PR 协调在本仓内,合规。)
  5. 测试有效性(质量关重点看了):5 个 case 双向都有牙——等值乱序同摘要 / 改值与改键集必变 / 数组保序 / Map·Set 形态不变 / 跳过表字面钉死且 biliVersion 是 mask 非 drop / e2e 用真 UnifiedShadow+loadLegacyView 复现 [Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392 的字段形状(live {m00002,m00001} vs replay {m00001,m00002} → verifyDrift=0)/ 负控:真实 ref 映射差异仍 drift(heal 路径不被本次修复废掉)。✓
  6. 本地门禁(我在 0b1a26cbb 上重跑,非引用你的结果):typecheck exit 0;issue4392 + storage-corpus + golden-storage-render 19/19 过;check-unused-exports clean(6 waived);check-test-import-graph ok。你本地全量 11 红的说明符合本地测试纪律的全量判定归 CI 原则,不采信也不追究,以远端为准。

既有行为清点(回归关):shadowDigest 判定(顺序差不再算 drift=本修复目标;真实差异仍算=负控钉住)/ 升级 WARN 文案(有意变更,旧「usually means MULTIPLE bili instances」→ 新「digest-shape mismatch first, second writer 是原因之一」,CHANGELOG 已披露;面板消费的是结构化 {disabled,lastError} 字段不吃日志文本,无消费者影响)/ 金标两套(零改动)/ 跳过集成员与 __TS__/__VER__/__UUID__ mask(不变,有测试钉住)/ 无配置面、无 env、无持久化格式变化。

八问

  1. 是什么问题:单实例自我报告 [diverged: state]——摘要哈希吃进了对象插入序,而 live 腿与 replay 腿对等值状态天然给出不同键序。
  2. 什么原因导致的:canonicalize 按 Object.entries 插入序参与哈希;byRef 在 live 腿按 kernel 发号序、replay 腿按账本 ord 重建,键序不同内容相同 → 3 次采样后误升级成「多实例」并冻结信封写入。
  3. 影响范围与严重程度:所有走 shadow 采样校验的 legacy 会话在 ref 账本重对齐后可能误报 drift 并被冻结(特定条件触发,体验为会话写入降级+误导性告警);修复前是 p1 级缓存/存储信任面问题,现已消除。
  4. 怎么修复的:在决策点(canonicalize)改为任意深度按键排序(数组保序、Map/Set 形态不变),跳过键收敛为单一导出表供所有摘要生产者共用;升级文案如实降调。疏而非堵——修的是产生错误摘要的机制本身。
  5. 是否有回归问题:无。见上方既有行为清点;唯一有意变更(WARN 文案)已披露。
  6. 是否破坏缓存:不涉及。shadowDigest 是自检值,不参与 KV 前缀缓存/重试/游标/去重键;金标零改动。
  7. 是否改了设计方向:不涉及(摘要契约从「吃插入序」修正为「吃键集」,属 bug 修复范畴;[PR] fix: stream storage verification digests to reduce peak memory #4532 的流式路径将复用同一张表,方向一致)。
  8. 是否引入了新的配置:不涉及(新增的只是模块内导出常量,非配置面)。

CI:新 head 正在跑,当前 0 红(static-gates ubuntu、mysql-contract 已绿,12 项排队中)。

判定:可合——待 CI 全绿即可,无遗留阻塞项。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: merged origin/master 9b6ec2345 into the PR branch — zero conflicts (only #4980 version-sync and #4904 dsh window guard landed since your last sync, neither touches this PR's files); new head 7ae198bb2 pushed to Maziar123:billion-context:2026-10-25_digest-key-order; PR state is now blocked (was behind) with CI running on the new head — 0 red so far (6 gates green, shards in flight).

… parsing (ranxianglei#3203)

waitForFile gated on existence only. The child fixture creates out-*.json
before its content lands (a zero-byte window that widens under Windows CI
load with four concurrent tsx spawns), so the parent could JSON.parse an
empty file: SyntaxError 'Unexpected end of JSON input' (run 40555535547,
shard windows-22-4). Gate on non-empty size stable across two polls; the
slot-distinctness assertion itself is unchanged.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 修复完成(第 2/2 轮自动修复) — 本轮 head 7ae198bb2 唯一红项已定位并修掉,新 head 18a9d97e2,CI 重跑中。八问:

  1. 是什么问题:本次运行(run 40555535547)30 项 check 只红 1 项——test (windows-latest, 22, 4) 里的 not ok 120 - concurrent shared-mode launches claim distinct slots (#3203),报错 SyntaxError: Unexpected end of JSON input(tests/codex-concurrent-launch.test.ts:216)。其余 28 项绿/跳过,2 项在跑。
  2. 什么原因导致的:这是既有测试框架的竞态,不是本 PR 的代码回归。该测试起 4 个并发 tsx 子进程各写一个 out-*.json;子进程是「先建文件、再写内容」(test:141),而父进程的 waitForFile(test:117-123)只轮询文件存在——Windows CI 负载下存在性为真的瞬间文件还是 0 字节,父进程抢先读空文件去 JSON.parse 就炸了。同文件的兄弟用例(exhaustion/stale-lease/migration/dual-writer/quarantine)全绿,进一步证明与 PR 无关:本 PR diff 只碰 storage digest 路径(canonical/shadow/manifest/issue4392 测试),完全不经过 launcher 并发槽位逻辑;head 里合入的 master 增量([PR] fix(#4894): arm the direct-fallback oversize guard on dsh not-ready sends #4904 dsh-native)也不涉及此测试。
  3. 影响范围与严重程度:仅影响这一个测试在 Windows CI 高负载下的稳定性(偶发 flake);无任何运行时行为、无用户可见影响。此前未见该 flake 的历史 issue(检索过),本次是首录。
  4. 怎么修复的:加固 waitForFile——从「文件存在即返回」改为「大小非空且连续两次轮询(50ms 间隔)稳定才返回」,超时语义不变;该 helper 在此文件中仅此一处调用,断言(slot 互不相同)一字未动。6 行改动,独立标注 commit 18a9d97e2 加在当前 head 之上(该 head 已含 master 同步,merge-base = master tip,分支完全同步)。本地验证:该套件 9/9 通过(含原失败用例)、tsc 干净。
  5. 是否有回归问题:无——纯测试基建加固,不触碰任何生产代码;断言语义不变(只把「读到一半的文件」这类假失败挡掉,真实失败仍会照旧炸)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:无需决策升级(几行级显而易见修复,按政策直接在 PR 分支修掉)。等 18a9d97e2 的 CI 全绿后,第 2 轮的审核结论维持:可合。协调记录已按惯例留了一行到 #4599。

@Maziar123

Copy link
Copy Markdown
Contributor Author

Merged master b08fbf36 (new head 4b67ab01, on top of your 18a9d97e launch-test fix). The only conflict was CHANGELOG.md [Unreleased] → Fixed, where #4984 added its I3 line in the same spot; both entries are kept. Against master the PR touches the same 6 files, none under kernel/. Local checks on the new head: tsc --noEmit 0 errors, tests/issue4392-canonical-key-order.test.ts 5/5, tests/codex-concurrent-launch.test.ts 9/9.

…est from ranxianglei#4997)

ranxianglei#4997 opened for ranxianglei#4986 the same root cause ranxianglei#4685 fixes in canonicalize. Its regression test fails on master b08fbf3 (DIGEST DRIFT for corpus-folded) and passes here without the shadowDigest change, so it carries over unchanged.
@Maziar123

Copy link
Copy Markdown
Contributor Author

Update: the head is now 428c7cdb.

Typecheck is clean, and the two digest test files pass 6/6. Details are in #4997.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

新 head 428c7cdb 复审完成(独立验证,未照抄你的声明)。 本轮增量 = 移植 #4997 的回归测试 + 同步 master b08fbf36 + 同步 origin/ci-fixes,无运行时代码改动。八问:

  1. 是什么问题:不是新缺陷——是 PR head 更新后的复审轮。背景:Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986(ingest 回读校验层的假 DIGEST DRIFT)与 [Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392(shadow state digest 层)同症状同机制,[PR] fix: parity digest compares ref-map contents, not key order (#4986) #4997 车道与本 PR 车道的二选一决策请求已在 [PR] fix: parity digest compares ref-map contents, not key order (#4986) #4997 楼层 3 升级给 owner(推荐 B);你选择了 B 并完成了配套动作。
  2. 什么原因导致的(被修缺陷的根因,复核确认):parity digest(shadowDigest→canonicalize)把对象键插入顺序当内容比——expect 腿 byRef 是内核变更序(After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 restamp 后为 ledger-seq 序),got 腿按全量 ingest 时冻结的行 ord 重建;同键同值、不同插入序 → 哈希不等 → 每次保存假漂移 + 丢 watermark + 全量重灌。本 PR 在共享的 canonicalize 定义点排键,一处覆盖全部 digest 路径。
  3. 影响范围与严重程度:本轮增量面 = 新增测试文件(+91 行)+ CHANGELOG 一行 + 两个同步 merge(master 与 ci-fixes;ci-fixes 净差仅 scripts/live-files.mjs +5,即哨兵收敛点登记的 [PR] release v0.1.209 #4982 套件 LIVE_FILES 项)。零运行时/wire/持久化/配置变化。
  4. 怎么修复的:本轮无需修码,做的是独立核验:①移植的测试文件与 [PR] fix: parity digest compares ref-map contents, not key order (#4986) #4997 版本逐字节一致(git diff 为空);②在未打补丁的 master b08fbf36 上(worktree 隔离环境)跑该测试 → 红,真实打出 [storage] unified shadow DIGEST DRIFT for corpus-folded … [diverged: state] error 行、verifyDrift=1;③在本 PR head 上跑 → 绿。测试有牙且本 PR 的 canonicalize 修法确实覆盖 Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 形状,你的证据表成立。
  5. 是否有回归问题:逐项审过——①定向套件 29/29 绿(issue4392×5、issue4986×1、codex-concurrent-launch×9(含我上一轮的 flake 修复)、storage-corpus×9、golden-storage-render×5);真漂移检测未被废掉(issue4392「真实内容差异仍漂移」用例绿);②tsc --noEmit 干净(kernel 已按本次 master 同步里的 [Bug] Merging stale compressed blocks retracts existing coverage, causing repeated coverage-regression rejections | [Bug] 合并旧压缩块时撤回既有覆盖,导致 coverage-regression 反复拒绝 #4484 重新构建);③diff vs master 共 8 个文件、全部可指认:PR 本体 5(canonical/shadow/manifest/CHANGELOG/issue4392 测试)+ 我的 launch-test flake 修复 1 + 移植测试 1 + ci-fixes LIVE_FILES 登记 1;无 kernel/、无 version、无 lockfile;④CHANGELOG 冲突解决核对过:双方条目都在,PR 条目扩展为同时覆盖 [Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392/Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 并注明测试取自 [PR] fix: parity digest compares ref-map contents, not key order (#4986) #4997(作者自己的条目行,合理)。
  6. 是否破坏缓存:不涉及——反而消除了错误的 watermark 丢弃(watermark 被丢才会逼出全量重灌)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:无。

判定:新 head 428c7cdb 复审通过,机械层面已按 B 选项就绪,等 owner 拍板车道(决策请求在 #4997 楼层 3;你已明确支持 B 并完成全部 B 侧动作)。若拍 B:CI 全绿即可合入,#4997 标 superseded(关单由你执行);若拍 A:#4997 先合,本 PR 按你楼层 19 说的裁剪跟进。当前 CI 在新 head 上排队中,0 红。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

这条更新我已在上方楼层完整复审过(head 428c7cdb):移植测试与 #4997 版本逐字节一致、master 上红/head 上绿已独立复现、定向套件 29/29、tsc 干净、diff vs master 8 个文件全部可指认(无 kernel/)。结论不变:复审通过,等 owner 在 #4997 楼层 3 的 A/B 车道请求里拍板(你支持 B,机械层面 B 已就绪)。CI 在新 head 上运行中,无需新增动作。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.910-master.38965bc (rolling master tag).

npm install -g billion-context@0.0.910-master.38965bc  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit to z0d0dOr/billion-context that referenced this pull request Oct 30, 2026
…xianglei#4532)

ranxianglei#4685 landed on master (canonicalize sorts object keys at every depth +
shared DIGEST_SKIPPED_KEYS table), but this PR's streaming canonicalDigest
still iterated raw Object.keys order with its own literal skip list. The
two producers then disagreed for any object whose keys arrive in a
different order — the false DIGEST DRIFT of ranxianglei#4392/ranxianglei#4986 again, this time
between the streaming digest and canonicalize; tests/issue4392-canonical-key-order.test.ts
went red on 8 shards (e.g. run 40909901398).

Align visit() with the legacy producer's effective byte order:
- skip via Object.hasOwn(DIGEST_SKIPPED_KEYS, k), keeping the __proto__ guard;
- iterate array-index keys first in ascending numeric order, then the
  remaining keys lexicographically. Plain .sort() alone is NOT
  byte-identical: spec enumeration (Object.keys / JSON.stringify) emits
  array-index keys ascending numeric regardless of insertion or lexicographic
  order, so an index-shaped pair like {"10","2"} must serialize as "2","10".
  The pinned byte-identity test (streaming digest preserves JSON omissions,
  sparse arrays and property order) catches exactly this.

Byte-identity to sha256(JSON.stringify(canonicalize(v))) now holds for any
key arrival order, in either merge order relative to ranxianglei#4685.

Verified locally: issue4392/issue4986/storage-canonical-digest suites
13/13, storage parity integration (engine-proofs + ranxianglei#3407 + ranxianglei#3919 + corpus)
26/26, typecheck clean.
ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
…tion alarm (Follows #4614)

e83fc64 (#4685) rewrote this alarm line for the #4392 digest-shape-first
wording and dropped the #4614 clause in the same edit, so every test lane
failed tests/issue4323-drift-escalation.test.ts:193 ('alarm also names the
single-writer repeating-rewrite producer') from the merge onward. Restore
both producers in one line — digest-shape mismatch first (#4392), then the
writer side: a second bili instance on one sessions dir (#394/#4177) or ONE
host re-writing history slots below the ingest watermark (#4614).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919)

2 participants