Skip to content

[PR] fix: parity digest compares ref-map contents, not key order (#4986) - #4997

Closed
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-28_digest-drift-ref-order
Closed

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-28_digest-drift-ref-order

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4986

Problem

The ingest read-back verify reported DIGEST DRIFT on EVERY save of sessions whose in-memory messageRefs key order diverges from the stored refs-row order (the folded-corpus shape; also after #4316 ledger alignment). Cost per affected save: error-level log + dropped ingest watermark + full-tail re-ingest next save. No data loss, but write amplification + log noise on a hot path, and repeated streaks feed the #4323 escalation path.

Root cause

The parity digest (shadowDigest → canonicalize) treats plain-object/Map key insertion order as content:

Identical keys and values under different insertion orders → hash mismatch → false drift on every sampled save (the first ingest of a fresh sidecar is always sampled).

Fix (direction A from the issue)

Normalize key/entry order before hashing, inside shadowDigest's single private definition — both verify legs (sync sqlite AND async mysql) are covered at once. Same philosophy as #3407's orderedBlockContents and #3113's sortObjectKeys golden pins ("pin logical content, not serialization order"). Key order of these lookup maps carries no contract meaning (identity is per-key lookup, never position), so an order-only difference is ALWAYS a false positive; sorting removes the whole class while every real divergence stays visible (key sets and values still compare byte-for-byte).

Why not direction B (keep producer order aligned): row order is frozen at full-ingest time, so alignment would require rewriting persisted ord bytes on every save (more write amplification than the bug itself), changes persisted envelope bytes (persistence-format territory), and makes every future byRef mutation site a latent drift source. B does not remove the verifier's fragility; A removes the false-positive mechanism itself.

Behavior change disclosure (intended)

  • Old: any key-insertion-order-only difference between the two legs reported as DIGEST DRIFT (false positive → watermark drop → full-tail re-ingest + error log on every affected save).
  • New: order-only differences are ignored; the digest compares key sets + values.
  • Real content-drift detection is UNCHANGED (drift-inducing suites below still fire and escalate).

Audit (per the issue's "must audit other shadowDigest consumers")

  • shadowDigest is private to src/storage/shadow.ts; only callers are the two verify legs. No persisted digest, golden pin, or cross-version comparison depends on its output — both legs always hash under one build.
  • canonicalize untouched → corpus manifest pins (scripts/update-storage-corpus.ts) and all test-side digests unaffected.
  • describeViewDivergence untouched (runs only on detected real drift).

Verification (targeted local runs on rebased head; full-suite verdict = CI on the pushed head)

  • New regression tests/issue4986-digest-drift-ref-order.test.ts: RED on master (verifyDrift actual=1 vs expected=0, DIGEST DRIFT line fired), GREEN post-fix. Asserts the corpus precondition (byRef order ≠ row-projected order) so it cannot pass vacuously.
  • tests/storage-json-writer-off.test.ts (the issue repro): 3/3 green, DIGEST DRIFT line gone.
  • Real-drift suites still catch content drift: issue4323-drift-escalation 6/6, storage-ingest-watermark 16/16, issue3919-shadow-drift-forensics 3/3, issue3407-shadow-blockcontents-order 2/2, storage-save-path 3/3, storage-full-flip 10/10, storage-ccr 3/3.
  • Adjacent: storage-unified 14/14, storage-corpus 9/9, storage-engine-proofs 12/12, storage-replicas 4/4.
  • npm run typecheck clean.

The sampled ingest read-back verify false-alarmed DIGEST DRIFT on every
save of a session whose in-memory messageRefs key order diverges from
the stored refs-row order: the expect leg carries the kernel-mutated
in-memory order (#4316 ledger-aligned byRef is rebuilt in ledger-seq
order) while the got leg rebuilds from rows in ingest ord, which
finishPlan froze at the last full ingest. Identical content under
different key insertion orders then dropped the ingest watermark and
forced a full-tail re-ingest on the next save, with an error-level log
line each time (repro: tests/storage-json-writer-off.test.ts).

shadowDigest now normalizes plain-object/Map/Set key order before
hashing, at its single private definition so both verify legs (sync
sqlite and async mysql) are covered at once. Real divergences stay
visible: key sets and values still compare byte-for-byte. No persisted
digest or golden depends on shadowDigest output (private to shadow.ts,
both legs always hash under one build), so cross-version comparability
is unaffected; canonicalize itself is untouched. Same philosophy as
#3407's orderedBlockContents and #3113's sortObjectKeys golden pins.

Behavior change (intended): order-only differences between the two legs
are no longer reported as drift. Pre-fix this class fired on every
affected save; post-fix it never fires. Content-drift detection is
unchanged — issue4323-drift-escalation, storage-ingest-watermark and
issue3919-shadow-drift-forensics still catch real drift and escalate.

Regression test: tests/issue4986-digest-drift-ref-order.test.ts (red on
master — verifyDrift 1 vs expected 0 — green post-fix).
@ranxianglei ranxianglei changed the title fix: parity digest compares ref-map contents, not key order (#4986) [PR] fix: parity digest compares ref-map contents, not key order (#4986) Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-28_digest-drift-ref-order (a4e3e1a)

Latest build — install this exact version

npm install -g billion-context@0.0.6619-pr.4997.2935b1a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
a4e3e1a billion-context@0.0.6619-pr.4997.2935b1a build
67aa504 billion-context@0.0.6610-pr.4997.b8199d3 build
03f4f3d billion-context@0.0.6533-pr.4997.18cf27a build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4997.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重结论:本 PR 代码本身无重复实现,但与在途 PR #4685 重叠——同一机制的两个修法,需 owner 拍板走哪条车道。

标签:bug + scope/storage + p2(存储层采样校验假阳性:无数据丢失、能用,但受影响会话每次保存都有写放大 + error 日志噪音)。

独立核验(未照抄 PR 描述,逐条对过代码与本地实测)

机制查证:

  • planIngest 把行 ord 冻结为 byRaw 键序(src/storage/ingest.ts:338-339);
  • loadLegacyView 按行 ord 重建 byRef(src/storage/ingest.ts:526-534,注释自述 "digest-sensitive");
  • canonicalize 按对象插入序展开进哈希(src/storage/canonical.ts:31-60);
  • shadowDigest 私有的确只有两个调用点(sync sqlite leg shadow.ts:391-392 / async mysql leg shadow.ts:968-969),无持久化 digest、无 golden 依赖其输出 → 审计声明成立;
  • 全仓扫描:src/ 内 sha256(JSON.stringify(...)) 形态仅此一处,不存在同类顺序敏感点;corpus manifest 钉值脚本直接走 canonicalize(scripts/update-storage-corpus.ts:157)不经 shadowDigest → 不受影响,声明成立。

本地定向实测:

  • 新回归测试:未打补丁的 master 上红(真实打出 [storage] unified shadow DIGEST DRIFT … [diverged: state] error 行,verifyDrift=1),PR head 上绿(1/1)✓ 与 PR 声明一致;
  • 真漂移检测未被废掉:issue4323-drift-escalation + issue3919-shadow-drift-forensics + issue3407-shadow-blockcontents-order + storage-json-writer-off = 14/14 绿;
  • 水位/保存路径:storage-ingest-watermark + storage-save-path = 19/19 绿;
  • npm run typecheck 干净。

CI(镜像快照 @17:47Z,sha 03f4f3d):6 绿 + 11 skipped + 13 运行中/排队,0 红;最终判定等 CI 全量跑完。

结构化八问

  1. 是什么问题:统一存储的采样回读校验把「内存 ref map 键序 ≠ 存储行序」误判为 DIGEST DRIFT,每次都丢 ingest watermark → 下次保存全量重灌尾部 + 一行 error 日志。无数据丢失。
  2. 什么原因导致的:parity digest 把键插入顺序当内容比。expect 腿 byRef 是内核变更序(After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 restamp 按 ledger-seq 序改写),got 腿按最后一次全量 ingest 冻结的行 ord 重建;同一批键值、不同插入序 → 哈希不等 → 假漂移。
  3. 影响范围与严重程度:p2。受影响面 = sqlite/mysql 双后端下所有内存 ref 序与存储行序分叉的会话(folded corpus 形状即典型,新 sidecar 首次 ingest 必被采样必命中);后果 = 每次保存一次全量尾部重灌 + error 级日志噪音 + 连续假漂移喂给 After idling for one hour, the fold override is lost: the client resends byte-for-byte identical content, and the previo | 闲置一小时后折叠覆盖丢失:客户端重发逐字节不变,已折叠的 ~240K token 内容自己回到上下文 #4323 升级路径(最坏到信封写入冻结直到干净样本);读取始终正确。
  4. 怎么修复的:方向 A——在 shadowDigest 唯一私有定义里哈希前归一键序,一处覆盖两条 verify leg。这些查找表的键序不承载契约语义(身份是按键查表、从不按位置),纯顺序差异必为假阳性;排序后键集与值仍逐字节比较,真漂移一个不少。
  5. 是否有回归问题:逐项审过的既有行为——①真漂移检测:不变,全部诱导真漂移的套件照常触发并升级(实测 14/14 + 19/19 绿);②采样节律/阈值/时序/日志输出格式:零变化(diff 只动 digest 定义);③canonicalize、describeViewDivergence、持久化字节、wire:全部未动。有意行为变更(已披露):旧 = 两侧仅键插入顺序不同也报 DRIFT(假阳性);新 = 纯顺序差异忽略,digest 比键集+值——这正是修复目标本身。
  6. 是否破坏缓存:不涉及——反而消除了错误的 watermark 丢弃(watermark 被丢才会逼出全量重灌)。
  7. 是否改了设计方向:不涉及——仅把 parity digest 的比较语义从「序列化顺序敏感」收窄为「逻辑内容敏感」,与 master CI 常红:storage-render 四个 golden 测试漂移(#3093 版本戳进会话元数据后未重生 golden),拖红所有新 PR #3113/unified shadow storage digest drift on long dsh sessions (telemetry-only) | 统一存储影子模式摘要漂移 #3407 既有钉值哲学对齐。
  8. 是否引入了新的配置:无。

代码质量(从严标准):命名达意、注释讲为什么、无死代码/魔法数、风格一致;回归测试有牙(断言 corpus 前置条件防空转通过 + verifySamples/verifyDrift/verifyOk 三向断言)——合格。

结论:代码本身干净、可合(待 CI 全绿)。但因与 #4685 同机制重叠,建议 owner 先拍板车道再合,避免合完再拆冗余。

决策请求:#4997 与 #4685 走哪条车道?

A:合 #4997,#4685 裁剪跟进 B:合 #4685,#4997 作 superseded
修法位置 shadowDigest 内(私有,恰好 2 调用点) canonicalize 内(共享 canonical form 定义点)
波及面 零 golden/pin 影响,审计面最小 corpus manifest 10 个 digest 值重钉(PR 内已完成,前轮 review 验证过是纯 digest 值变更)
各选项回归风险 低;但 canonicalize 仍顺序敏感 → 后续新的摘要生产者(#4532 流式路径等)会再踩同一个坑,除非各自再排一遍键 低;golden 契约一次性变化,重钉在同一 commit 边界内,跨版本比较面封闭
保留的独有价值 最小改动面 升级文案修正(去掉误导性「多半是多实例」归因)+ 导出共享跳过表供后续摘要路径复用
现状 CI 在跑,0 红 同 base,CI 在跑,1 红 shard 处理中(自动修复触顶,等 owner)

推荐:B。 理由:在共享 canonical form 的定义点修更近根因(canonical form 本就不该含序列化顺序,正是 #3113「pin logical content, not serialization order」的哲学),且堵住后续摘要路径的同类坑;#4685 已经两轮 review 收敛,弃掉已通过审阅的外部贡献不符合仓库惯例。配套动作:#4685 合并后把 #4997 标 superseded 关闭,并将其回归测试 tests/issue4986-digest-drift-ref-order.test.ts 移植进 #4685 分支——它 e2e 钉住的是 #4986 特有的 folded-corpus 面,与 #4685 自己的测试互补,值得保留。若 owner 想要最小 golden churn,选 A 也可行:#4997 先合,#4685 裁成「文案修正 + 跳过表」的小 follow-up。

(另:#4392 与 #4986 同症状+同机制,按政策应并成一个管理,关闭与否由 owner 定。)

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收工状态确认(完整 review 见上方楼层,此处为收尾摘要):

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已修,push 完成,新一轮 CI 已在 a4e3e1a02 上排队运行。结论:不是本 PR 的代码问题,是继承自 master 红的失败,按 #4599 协调政策处理(merge ci-fixes),一行代码没动本 PR 的修复本身。

八问:

  1. 是什么问题:PR 的全部 11 个 test shard(windows×22 s1-5、ubuntu×22 s1-3、ubuntu×24 s1-3)在跑任何测试之前就整体挂掉。
  2. 什么原因导致的:master 当前头 b08fbf3([PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) #5016 合入)新增了 process-spawning 套件 tests/issue4982-draining-twin-ladder.test.ts,但没在 scripts/live-files.mjs 的 LIVE_FILES 里登记 → scripts/run-ci-tests.mjs 的 SPAWN_RE tripwire(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248 并发假设审查门)把所有 shard 拒之门外。已核对失败 job log,错误行与 master 红(run 40621087792)逐字一致。
  3. 影响范围与严重程度:所有基于当前 master 切出的 PR(含 master 自身)测试矩阵全灭;本 PR 自身改动(src/storage/shadow.ts + tests/issue4986-digest-drift-ref-order.test.ts,共 +123/-1)不受影响、未被测到是因为根本没跑到测试。
  4. 怎么修复的:不动本 PR 代码;把哨兵收敛点 origin/ci-fixes(commit 02ef0d1,登记该套件进 LIVE_FILES)merge 进 PR 分支,merge commit a4e3e1a02,净变更仅 scripts/live-files.mjs +5 行,已 push。ci-fixes 自身的验证 run(40625472882)已完成的全部 8 个 test shard 均绿,佐证该修法成立。
  5. 是否有回归问题:无。merge 只动 CI 测试编排的登记清单,不触碰运行时、wire、配置、持久化任何路径。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

备注两点:①镜像 git 端点当时返回 502,本次直接 push 到 GitHub(PR 分支的规范位置,CI 触发正常);②#4599 floor 394 有另一个 agent 报告该 twin-ladder 套件的 e2e case 2 断言对 master 文案可能"先天红"——但 ci-fixes 验证 run 的 8 个 shard 全绿说明它在 CI 矩阵下能过(e2e 段大概率被门控);若本轮重跑仍有单个 shard 红且签名指向那里,请在 #4599 跟进,不在本 PR 处理。

判定:修复动作已完成,等 a4e3e1a02 的 CI 结果即可。

Maziar123 added a commit to Maziar123/billion-context that referenced this pull request Oct 28, 2026
…est from ranxianglei#4997)

ranxianglei#4997 opened for ranxianglei#4986 the same root cause ranxianglei#4685 fixes in canonicalize. Its regression test fails on master b08fbf3 (DIGEST DRIFT for corpus-folded) and passes here without the shadowDigest change, so it carries over unchanged.
@Maziar123

Copy link
Copy Markdown
Contributor

I support option B from the review above. Here is the evidence.

I copied this PR's regression test, tests/issue4986-digest-drift-ref-order.test.ts, unchanged into #4685 (commit 3b0993c2):

Tree Result
master b08fbf36 fails 0/1: DIGEST DRIFT for corpus-folded
#4685, without this PR's shadowDigest change passes 1/1
#4685's own issue4392-canonical-key-order tests pass 5/5

So the canonicalize fix in #4685 also covers the #4986 shape, and the test now ships with #4685.

#4685 also has origin/ci-fixes merged in (head 428c7cdb), so the LIVE_FILES tripwire no longer blocks its CI. Typecheck is clean.

If you prefer option A, for the smallest golden change, that works for us too. Merge this PR, and I will trim #4685 down to the wording fix and the shared skip table.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

merge-guard 同步结果(本任务收尾):master b08fbf361 已并入本分支并推送(merge commit 67aa5041c);随后 ci-worker 又把 origin/ci-fixes(LIVE_FILES 登记 issue4982 套件,修继承自 master 的 CI 红)合入,当前 head a4e3e1a02 完整包含最新 master,不再 behind。CI 状态:两个 PR 的新一轮套件都在跑、目前 0 失败(#4997 @a4e3e1a02 17/24 pending,#4685 @428c7cdb 21/21 pending)。

对你 option B 证据的独立核验(拉了 #4685 实际树,没照抄结论):

车道决策(A/B)留给 owner 拍板——agent 无权合并或关闭任何一方。floor 3 的 review 推荐 B,你的证据进一步支持 B(在共享 canonical form 定义点修更近根因,且堵住后续摘要路径同款坑;外部贡献已过两轮 review)。两条路径都只合一个:

除上述同步 push 外未对任何 PR 做其他改动;等 owner 定车道。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

merge-guard 同步已完成:master b08fbf361 已并入本分支并推送(merge commit 67aa5041c);ci-worker 随后把 origin/ci-fixes(LIVE_FILES 登记 issue4982 套件,修继承自 master 的 CI 红)合入,当前 head a4e3e1a02 完整包含最新 master,不再 behind。第二次 behind 触发为陈旧状态,复核确认无需再动。CI:新一轮套件运行中、目前 0 失败(17/24 pending)。

对 #4685 侧 option B 证据的独立核验(拉了实际树):head 428c7cdb0 含 commit 3b0993c2,其中移植的 tests/issue4986-digest-drift-ref-order.test.ts 与本 PR 版本逐字节一致;#4685 在共享 canonicalize 内对所有深度对象键排序 + 导出 DIGEST_SKIPPED_KEYS,两条 verify leg 均经其哈希,结构上覆盖 #4986 形状。最终判据以 #4685 @428c7cdb 全量 CI 绿为准。

A/B 车道决策留给 owner(agent 无权合并/关闭):floor 3 review 推荐 B(在共享 canonical form 定义点修更近根因),#4685 作者证据进一步支持 B。B = 合 #4685 → 本 PR 标 superseded、#4392/#4986 并案(owner 执行关单);A = 合本 PR → #4685 裁成「文案修正 + 共享跳过表」follow-up。除同步 push 外未做任何改动,等 owner 拍板。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

这条评论已在上方楼层(#21、#23)完整回复过,此处不重复全文,只给结论指针:

  1. merge-guard 同步已完成:当前 head a4e3e1a02 完整包含最新 master b08fbf361,不再 behind;CI 运行中、0 失败。
  2. 你的 option B 证据已独立核验通过:[PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 @428c7cdb0 含 3b0993c2(移植的回归测试与本 PR 逐字节一致),canonicalize 键序归一在共享定义点消除顺序敏感,覆盖 Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 形状;最终判据 = [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 全量 CI 绿。
  3. A/B 拍板在 owner(agent 无权合并/关闭):现有证据链支持 B(合 [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 → 本 PR superseded、[Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392/Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 并案);选 A 也成立(合本 PR → [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 按你承诺裁剪)。等 owner 定车道,在此之前双方都不动。

@ranxianglei

Copy link
Copy Markdown
Owner Author

plan b合并

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

2 participants