Skip to content

fix: shadow parity digest compares blockContents by content, not Map insertion order - #3423

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-16_storage-shadow-digest-drift-3407
Oct 17, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-16_storage-shadow-digest-drift-3407

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

Long dsh persona-fork sessions repeatedly logged:

[storage] unified shadow DIGEST DRIFT for <id>: unified replay != legacy load view — migration parity broken, report this

Root cause: the unified-shadow parity digest renders session.blockContents as a Map in insertion order, but the two legs it compares legitimately populate that map in different orders:

  • expect leg keeps the live in-memory Map's insertion order, and completeBlockContents appends synthesized entries at the end whenever a block has no slot;
  • got leg (loadLegacyView) lays blocks out in state.blocks array order via the blocks table.

A long-lived session hits this every time orphan GC deactivates a non-terminal block (it stays in state.blocks but its content is deleted from blockContents, src/orphan-gc.ts) or fork-adoption sets contents out of array order. Each such save then false-alarms DIGEST DRIFT on the sampled verify (every VERIFY_EVERY_INGESTS=20 ingests). In shadow mode this is telemetry-only (read path stays legacy); in full mode the read follows the unified view, so the drift would mask a real read-path divergence.

Fix

Key-sort both legs' blockContents by blockId before hashing (orderedBlockContents), so the digest compares block contents, not insertion order. Sorting changes neither keys nor values, so a genuine fork (different blockIds or different text) still trips it — only pure ordering differences are neutralized.

Scope: src/storage/shadow.ts verify path only. No wire/message-shape, config-schema, or persistence-format change. Not under kernel/.

Behavior change (flagged)

  • Old: parity digest sensitive to blockContents Map insertion order → false DIGEST DRIFT on any partial/reordered blockContents (e.g. an orphan-GC'd non-terminal block).
  • New: parity digest key-sorts blockContents on both legs → order differences no longer trip it; real content forks still do.
  • Effect: removes the repeated false-positive drift logs on long dsh sessions. Detection of genuine data forks is unchanged (strictly better signal-to-noise on the drift counter/log).

Verification

  • New tests/issue3407-shadow-blockcontents-order.test.ts drives the real UnifiedShadow.record() sampled-verify for both production shapes (orphan-GC partial, reordered-complete): both fail pre-fix (verifyDrift=1) and pass post-fix (verifyDrift=0) — verified locally by reverting the fix, observing the failures, restoring it, and re-running.
  • Local typecheck clean; targeted test green. Full suite delegated to CI.

Config surface

None — no env/config/CLI/default added or re-semantics'd.

Closes #3407

…rder (#3407)

The unified shadow parity digest rendered blockContents as a Map in insertion
order. The two verify legs legitimately populate it differently — the expect
leg keeps the live Map's order (completeBlockContents appends synthesized
entries at the end) while the got leg lays blocks out in state.blocks array
order — so long dsh persona-fork sessions whose orphan-GC'd or reordered
blockContents diverge in order false-alarmed DIGEST DRIFT on every sampled save
(telemetry-only in shadow mode, but a real read-path divergence in full mode).

Key-sort both legs' blockContents before hashing so only genuine content forks
trip the digest. Regression test drives the real UnifiedShadow.record() path for
both the orphan-GC partial shape and the reordered shape; both fail pre-fix and
pass post-fix.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-16_storage-shadow-digest-drift-3407 (13a4b10)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4020-pr.3423.9807dfc

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3423.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复——#3423 是缺陷 #3407 的唯一修复 PR;检索 open+closed 无并列的同类「unified shadow digest 顺序漂移」PR/issue(#3407 自身已带 查重:无)。

PR #3423 独立审核结论

先排除一个易误判点:此 PR 真实改动只有 2 个文件(src/storage/shadow.ts +10/-2,新增 tests/issue3407-shadow-blockcontents-order.test.ts +167)。若用 git diff master..head(两点)去看,会看到它「删掉 src/server/threads-lane.ts(418 行)+ 其测试」——那是假象:分支从较旧 master 切出,master 之后才加入 threads-lane(#2926),两点 diff 把「master 新增、本分支尚无」误显示成「本分支删除」。三点 diff(merge-base→head)与 GitHub Files-changed(2 files,+177/-2)均确认只动这 2 个文件;rebase 后假象自然消失。

结构化八问

  1. 是什么问题:长 dsh persona-fork 会话反复打 unified shadow DIGEST DRIFT 假告警(shadow 模式仅遥测;full 模式读跟随 unified 视图,会被它掩盖真实读路径分叉)。
  2. 什么原因导致的:parity digest 经 canonicalize 渲染 blockContents——Map 按插入序进哈希(canonical.ts:35),而两条腿合法地以不同插入序填这张表:expect 腿保留 live Map 顺序且 completeBlockContents 把补全项追加到末尾,got 腿(loadLegacyView)按 state.blocks 数组序铺块。orphan-GC 停用非终态块(留在 state.blocks、内容从 blockContents 删)或 fork-adoption 乱序写内容时,两腿顺序分叉 → 纯顺序差被当 drift。我核对过其余顺序敏感结构:messageRefs.byRaw/byRef 经 ord 台账保序、deadRefs 是数组保序,round-trip 稳定——所以 blockContents 确为真正分叉字段,是根因而非表象。
  3. 影响范围与严重程度:仅 storage 层抽样校验的遥测面;触发条件 = 长会话发生 orphan-GC/fork-adoption 后的第 20 次 ingest 抽样。严重度 = 日志噪声/信噪比(shadow 模式无功能影响);full 模式会压低真实 drift 检出信号。非数据损坏。
  4. 怎么修复的:两腿哈希前各自按 blockId 键排序(orderedBlockContents,返回新 Map、不改 live 对象),让 digest 比块内容而非插入序。排序不动键也不动值 → 真分叉(不同 blockId 或不同文本)仍触发,只中和纯顺序差。
  5. 是否有回归问题:逐项过——① 唯一行为变化是「纯顺序差不再误报」,既有正确行为不破;真数据分叉检测不变(严格提升信噪比)。② 不掩盖真 bug:两视图内容完全一致、仅 Map 迭代序不同时本就 parity 成立,无需报。③ 副作用偏正:误报减少 → 减少 fix: per-session ingest watermark — storage saves cost O(new messages), not O(full log) (#3133) #3166 watermark 被误清引发的多余全量重灌。高危模式(语义漂移 / fail-fast 变吞错 / 阈值·时序·输出格式变化)均不涉及。
  6. 是否破坏缓存:不涉及——digest 仅在校验时计算、不落盘;persist 格式/版本、prefix-cache、delta/watermark 游标逻辑均未改(只是少被误触发)。
  7. 是否改了设计方向:不涉及——仍是「unified replay == legacy load view」同一 parity oracle,只修掉它的一个假阳性来源。
  8. 是否引入新配置:不涉及——无 env/config/CLI/默认值变更(config-surface-guard CI 绿)。

验证

  • 测试有牙:两个 case(orphan-GC partial / reordered-complete)走真实 UnifiedShadow.record() 抽样校验,非 mock。我逐条推演:pre-fix expect 序([b1,b3,b2] / [b2,b3,b1])≠ got 序([b1,b2,b3])必 verifyDrift=1;post-fix 两腿同序必 =0;断言 verifySamples=1 / verifyDrift=0 / verifyOk=1 能兜住回归。时间戳 pin 在 canonicalize 阈值之上,把变量隔离到纯顺序。
  • CI(head 13a4b10):static-gates / lock-sync / kernel-guard / config-surface-guard / pi-subagents-guard / docs-sync / build-artifact 全绿;test 分片 5/6 success,1 片(ubuntu·node24·shard1)仍 in_progress;real-client e2e 各 lane 按各自 env gate skipped(正常)。
  • 本地 typecheck/test:作者已自报绿;本环境无 node_modules,按本地测试纪律不重装依赖跑全量,以远程 CI 为最终裁决。

合并判定

可合——机械上满足 auto-merge 全部 8 条(单模块、回归测试复现+通过、无 config/persistence/wire/kernel 改动、纯 fix:、diff 干净 2 文件、引用 #3407、不碰 load-bearing infra)。给 owner 两点提示:① 分支 behind master,rebase 到当前 master 再合更稳(master 未动 shadow.ts,rebase 零冲突,threads-lane 假删除随之消失);② 等最后一片 test shard 转绿。合并由人执行——系统规则禁止 agent 合 PR,请自行合并:#3423。

@ranxianglei
ranxianglei merged commit 53ebe00 into master Oct 17, 2026
60 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.506-master.53ebe00 (rolling master tag).

npm install -g billion-context@0.0.506-master.53ebe00  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unified shadow storage digest drift on long dsh sessions (telemetry-only) | 统一存储影子模式摘要漂移

1 participant