Skip to content

[PR] fix: stream storage verification digests to reduce peak memory - #4532

Merged
ranxianglei merged 15 commits into
ranxianglei:masterfrom
z0d0dOr:2026-10-24_streaming-storage-digest
Oct 30, 2026
Merged

ranxianglei merged 15 commits into
ranxianglei:masterfrom
z0d0dOr:2026-10-24_streaming-storage-digest

Conversation

@z0d0dOr

@z0d0dOr z0d0dOr commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Storage shadow verification materializes a canonical object graph and a complete JSON string for each digest. Large retained histories therefore cause a substantial temporary memory spike during sampled verification.

This change walks the storage view directly and feeds canonical JSON incrementally to SHA-256. Long strings are escaped in 64 Ki-character chunks without splitting surrogate pairs. Scalar normalization is shared with the existing canonicalizer, and the shared shadow digest entry point covers both synchronous and asynchronous verification. Existing storage formats and digest values are preserved.

Fork diagnostics also read the session ID from dispatchScope, matching the upstream sidScope rename; the stale reference otherwise prevents current master from typechecking.

Fixes #4529. Related: #3133, #3140, #3437.

Validation

  • 27/27 targeted tests passed: storage-canonical-digest, issue3407-shadow-blockcontents-order, storage-corpus, issue4215-fork-gate-diagnostics, and dsh-llm-stream-attribution.
  • New comparisons use the previous sha256(JSON.stringify(canonicalize(value))) as the oracle, covering normalization, collection/property order, JSON omissions, long Unicode strings, and changed retained history.
  • Typecheck (including the prebuilds), npm run build, and git diff --check passed. Independent scoped review passed.
  • The docs-sync generation and artifact-diff check pass with the updated configuration seeds from master. No generated documentation changes are required.
  • Isolated Node 24.12.0 processes with the same 512 MiB heap limit hashed 512 block entries sharing 256 KiB text each (128 MiB logical history). Digests matched exactly; peak RSS was 329 MiB before / 76 MiB after. This synthetic comparison measures the digest path, not total server memory.

Full-suite validation is left to repository CI. The memory optimization is limited to digest allocation; it does not resolve the separate full-view reload or active-session admission costs.

@ranxianglei ranxianglei changed the title fix: stream storage verification digests to reduce peak memory [PR] fix: stream storage verification digests to reduce peak memory Oct 24, 2026
@z0d0dOr
z0d0dOr force-pushed the 2026-10-24_streaming-storage-digest branch from ab707eb to 418118e Compare October 24, 2026 12:46
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。#4532 是唯一针对 #4529 的 PR(open+closed 全量检索,无其他 PR 触碰 streaming canonical digest)。相关历史条目均已合并且机制不同:#3423(digest 按内容而非 Map 插入序比较)、#3948(漂移维度命名)、#4332(结构性漂移升级告警);#3437(活跃会话峰值内存 ~1.26GiB)是 #4529 正文自己声明的独立关注点,不同失败域。

标签:bug scope/storage p2(存储验证路径的内存尖峰缺陷;可用、有绕行,不属于缓存泄露/会话丢失类,不升 p1;客户端不涉及)。

结构化八问

  1. 是什么问题:统一存储的抽样完整性校验,每算一次摘要都要先把整个会话视图「规范化成完整对象图 → JSON.stringify 成完整字符串」再喂给 SHA-256。保留历史大的会话,每次抽样校验都会临时多分配一份同量级的副本,造成堆尖峰。
  2. 什么原因导致的:digest 计算路径写成了一次性序列化(materialize 全量再哈希)。抽样只降低了频率,没有降低单次分配的规模——根因在计算方式本身,不在调用频率。
  3. 影响范围与严重程度:跑统一存储(shadow/full)且有大保留历史的机器,在抽样校验命中时触发;贡献堆耗尽压力。特定条件下有问题(大历史 × 校验采样),有绕行(小历史/降频),无数据损坏、无功能失效 → p2。
  4. 怎么修复的:digest 从「整串序列化再哈希」改为流式遍历——直接走存储视图,把转义后的规范化 JSON 增量喂给 SHA-256;长字符串按 64Ki 字符块转义且不拆代理对(surrogate pair);标量归一化规则与原 canonicalize 共用一份实现(抽出 canonicalScalar,纯重构);唯一入口 shadowDigest 同时覆盖同步/异步两条校验腿(shadow.ts:381、shadow.ts:888)。
  5. 是否有回归问题:无。逐项清点既有行为:
    • digest 值逐字节不变(核心回归面):新测试以旧公式 sha256(JSON.stringify(canonicalize(v))) 为 oracle 断言等价,覆盖归一化(VER/TS/UUID)、Map/Set 集合序与空集、JSON 省略(undefined/function/symbol、稀疏数组)、__proto__ 污染键、长 Unicode 含孤立代理、转义 map 键;storage-corpus 的 golden pin 通过。我独立复核:PR 分支上 50 MiB 合成历史两条路径 digest 完全一致,峰值 RSS 179→71 MiB,方向与量级和 PR 声称的 329→76 MiB(128 MiB 历史)一致;
    • canonicalize 导出行为不变,其余消费方(prefix-affinity、corpus manifest、ser())未触碰;
    • 存储格式、wire、配置面、kernel/ 均无改动;失败模式不变(top-level undefined/function/symbol 两路径同样 throw,既有行为);
    • 有意行为变更(即修复目标本身):抽样校验期间峰值分配从「≈3–4× 逻辑历史大小」变为「有界常量缓冲(pending 上限约 450Ki 字符)」,digest 输出不变。
  6. 是否破坏缓存:否——digest 是校验比较键,值不变,既有 pin/golden 全部继续有效。
  7. 是否改了设计方向:否——仍是「digest 相等 = 一致」,只改计算的分配形态。
  8. 是否引入新配置:否——无 env/config/默认值变化。

测试与 CI

  • 测试有效性:oracle 等价断言有牙(任何 parity 回归立即红);负向也覆盖了(Map 插入序敏感 notEqual、保留历史变化后 digest 必变)。本地定向验证 15/15 通过(storage-canonical-digest + issue3407-shadow-blockcontents-order + storage-corpus);全量交给远程 CI。
  • CI 注册:新测试文件无需登记——CI 按 tests/*.test.ts glob 发现并确定性分片,LIVE_FILES 只管进程派生类套件,本文件不派生进程。
  • diff 整洁度:3 文件、每行都与目的相关;命名达意、注释讲为什么(代理对、__proto__ setter 语义)、无死代码/魔法数字。

判定:可合。 单模块闭环小修,digest 值逐字节不变,内存收益已独立复现确认。合并权在你:#4532

@ranxianglei ranxianglei added bug Something isn't working p2 scope/storage labels Oct 24, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 自动修复车道结论(docs-sync)

查重:与 #4520 同根因(master 侧缺 config-reference seed 条目)——#4520 已修复,本 PR 分支随后被 rebase 到含该修复的 master,我本地准备的同名修复判定为重复、已丢弃,未向分支推送任何提交。

八问

  1. 是什么问题:docs-sync 检查红——CONFIGURATION.md(中英两份)里有一行 rollout.features.pluginLaneServerHistory,但生成器的种子文件(website/config-reference/*.yaml)里没有对应条目,每次重新生成都把这一行删掉,diff 非空即红。
  2. 什么原因导致的:该行当初是手工写进文档的,没同步补种子条目,master 自身失步于自己的生成器——任何合到这个 master 窗口上的 PR,diff 都会包含「PR 头合并进 master」时带出来的这一行删除。与本 PR 的代码改动无关(它只动了存储摘要校验流的三个文件,不碰任何文档/种子)。
  3. 影响范围与严重程度:影响该 master 窗口上所有 PR 的 docs-sync(CI 红挡合并);运行时零影响,纯文档面问题。
  4. 怎么修复的:已在 master 侧由 docs-sync has been all red on master since the merge of #4505: the rollout.features.pluginLaneServerHistory seed entry i | docs-sync 自 #4505 合并起在 master 全红:rollout.features.pluginLaneServerHistory 种子条目丢失 #4520(commit 8daae67)修复——给 website/config-reference/rollout.yaml 补上缺失的种子条目并重新生成。我排查期间分支被强制 rebase 到含该修复的最新 master(新头 418118e),因此无需再推提交;若重推我本地那份会变成重复条目,故丢弃。
  5. 是否有回归问题:无。已在新头本地复跑 docs-sync 同款命令验证:node tools/gen-config-docs.mjs generate 后对 CONFIGURATION.md / CONFIGURATION.zh-CN.md / website/main.js / website/docs/index.html 做 git diff --exit-code 为空(再次 generate 字节级幂等),check-coverage 绿(185 env names + 58 top-level keys all documented)。无行为变更:键名/默认值/env 名与原行一致。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标逻辑)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及(仅为既有键补文档注册,无新增配置项)。

判定:无需追加提交;CI 在新头 418118e 重跑后 docs-sync 应转绿。若其余 job 另有红项,另行处理。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决,PR 恢复可合并。已 rebase 到最新 master(ba5ab0b),新 head ca791aaae(旧 head 71c4361 被 force-push 替换)。GitHub 状态:mergeable ✅(state 为 behind——master 还在前进,属正常,不阻塞合并)。本地验证:typecheck 通过,定向测试 15/15 通过(新 digest 测试 + shadow drift forensics + snapshot-shadow);全量回归留给远程 CI。

八问:

  1. 是什么问题:PR 分支与 master 冲突,冲突点只有一处——src/agent/dsh-native.ts 里的一行代码。
  2. 什么原因导致的:两边各自独立修好了同一行坏代码。该行原本引用旧变量名 sidScope,Plugin lanes: pin identity to client msgids, render outbound from the server-owned history | 插件车道服务端正史:消息身份钉在客户端 id 上,出站前缀由正史渲染 #4491 在 master 上把该变量改名为 dispatchScope 时漏掉了这一处,master 一度无法编译;随后 master 自己修了(de29698),而 PR 作者在开发本 PR 时也独立给同一行写了自己的修法——两个修法语义完全等价,文本上撞车。
  3. 影响范围与严重程度:零行为变化。这是两个等价修复的撞车,不是语义分歧;解完后该文件与 master 完全一致(diff = 0)。
  4. 怎么修复的:保留已合入 master 的版本(含其解释性注释),删掉 PR 侧的重复 hunk。rebase 后 PR 的最终 diff 只剩它原本的目的:src/storage/canonical.ts 流式 digest(+85 行)、src/storage/shadow.ts 调用点切换(2 行)、新增测试文件(+82 行)。
  5. 是否有回归问题:无。逐项审过的既有行为:① digest 值逐字节不变(PR 自带测试钉住:流式 digest === sha256(JSON.stringify(canonicalize(v))),含 surrogate pair 跨块边界、MB 级历史、深嵌套、宽扇出);② fork 盲区日志语义不变(两个版本算出同一个布尔值);③ typecheck 干净;④ 定向测试全绿。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/重试/增量游标/去重键逻辑。
  7. 是否改了设计方向:不涉及——纯冲突消解,无架构调整。
  8. 是否引入了新的配置:不涉及。

判定:冲突已解,等 CI 绿灯后可由 owner 合并。PR: #4532

@z0d0dOr
z0d0dOr force-pushed the 2026-10-24_streaming-storage-digest branch from ca791aa to b36fc4e Compare October 25, 2026 20:11
Maziar123 added a commit to Maziar123/billion-context that referenced this pull request Oct 27, 2026
… table (ranxianglei#4392)

canonicalize hashed insertion order, so the shadow live leg and the
replay leg (messageRefs.byRef rebuilt in ledger order by loadLegacyView)
produced two digests for the same value on a SINGLE instance, escalated
after three samples and froze envelope writes.

- sort object keys at every depth; arrays keep order, Map/Set envelopes
  keep their shape
- move the storageMode/tailGaps drops into one exported table
  DIGEST_SKIPPED_KEYS so the streamed canonicalDigest (ranxianglei#4532) imports the
  same skip set instead of hand-copying a second list
- pin the table literally and pin depth-independent dropping
@Maziar123

Copy link
Copy Markdown
Contributor

For #4392, both digest producers must agree on key order and skipped keys (see the reporter's decision in #4392). #4685 changes canonicalize to walk Object.keys(record).sort() and moves its skipped keys into one exported table, DIGEST_SKIPPED_KEYS in src/storage/canonical.ts. Once one of the two PRs lands, the other needs two small changes in canonicalDigest:

  1. Sort the keys. The object branch here walks Object.keys(current) in insertion order (diff line 79). If shadowDigest switches to canonicalDigest with that loop, the [Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392 single-instance drift returns on the streamed path: the live leg builds messageRefs.byRef in id order and the replay leg rebuilds it in ledger order, so equal values hash differently.
  2. Use the shared skip table instead of a second list (diff line 81). Keep __proto__ as its own check: in canonicalize it is an assignment artifact, not a skip-set entry.
for (const k of Object.keys(current).sort()) {
    if (Object.hasOwn(DIGEST_SKIPPED_KEYS, k)) continue;
    // Assigning __proto__ to canonicalize's plain output creates no JSON property.
    if (k === "__proto__") continue;
    // ...rest of the loop unchanged
}

Your matchesLegacy helper already pins the two paths together; one key-order case would cover #4392 on both:

test("#4392: key order is not content on either digest path", () => {
    const a = { state: { messageRefs: { byRef: { m00002: "b", m00001: "a" } } }, stats: { requests: 1 } };
    const b = { stats: { requests: 1 }, state: { messageRefs: { byRef: { m00001: "a", m00002: "b" } } } };
    assert.equal(canonicalDigest(a), canonicalDigest(b));
    matchesLegacy(a);
    matchesLegacy(b);
});

Whichever PR lands second takes this on. If #4532 lands first, I will add the import and the test to #4685.

z0d0dOr and others added 4 commits October 28, 2026 00:27
…nk seams (ranxianglei#4529)

Extend storage-canonical-digest.test.ts with deterministic cases the original
suite left uncovered: surrogate pairs landing exactly on the 64 KiB hash-chunk
seam (writeString boundary pull-back), an MB-scale retained history spanning many
chunks end-to-end, deep nesting (visit/JSON.stringify recursion parity) and wide
fan-out (separator + pending-buffer flush). Each asserts
canonicalDigest(v) === sha256(JSON.stringify(canonicalize(v))), so any chunking
or surrogate-split regression fails loudly rather than drifting silently.
Compare unified reads with an independent envelope reader plus the documented migration. Persist alignment before rollback/corrupt-index checks, select legacy explicitly, and assert the recovered raw ID under its ledger ref. Keep checkpoint writes deterministic so sampled ingest cadence cannot change the seed between reader legs. All 28 targeted storage tests pass; no product migration or digest assertion is disabled.
@ranxianglei
ranxianglei force-pushed the 2026-10-24_streaming-storage-digest branch from b36fc4e to f8d91aa Compare October 27, 2026 19:14
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决:分支已 rebase 到当前 master(9b6ec23),新头 f8d91aa75,已推回原分支。PR 现在 mergeable,状态 blocked = 等 CI 跑完。本轮冲突不需要任何设计决策,全部是「两边各自修了同一个问题」或「作者的方案已被 master 更新方案取代」的碰撞,按「保留 master 已验证设计」原则处理。

逐处冲突的处理(共 5 个提交碰撞,全部已在本地定向验证):

  1. Windows 启动器测试(launcher-opencode-inline):master 侧是更新的超集(大小写不敏感的 COMSPEC 探测 + verbatim 字段,CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 相关),保留 master。
  2. 博客 CRLF 归一化(src/web/blogfeed.ts):这里发现了一个真实的取舍——master 的归一化是有条件的(仅当文件以 frontmatter 开头才转换),而本 PR 带来的 CRLF 回归测试覆盖了没有 frontmatter 的帖子(如 paper/*.md)。只留 master 会让该测试红:\r 会漏进渲染出的 HTML。取并集:改为无条件归一化。行为变更披露:旧值=仅 frontmatter 开头的文件做 CRLF→LF;新值=所有文件无条件做。原因:非 frontmatter 正文里的 \r 会进入渲染结果,属缺陷修复而非漂移。
  3. fork 重启双模式测试(plugin-fork):取并集——作者的双存储模式循环 + master 侧 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 的竞态等待,两者独立有效。
  4. 渲染 golden:engineLane 元数据 master 已自带,直接取 master 版本,并用定向运行证明在本树全绿(核心修复不改任何持久化/渲染字节)。
  5. 账本引用一次性迁移测试(作者第 6 个提交):作者的「300 秒防抖 + 对齐 oracle」组合是为绕过他分支上的 CI 抖动设计的,而 master 已用另一套方案(After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 引用规范化:shaCiteNorm/stateDigestAligned)解决了同一问题。我起初保留了 300 秒防抖、删掉了 oracle,结果 4 个 wire-parity 测试在本地变红(调试确认:无 oracle 时全模式腿的最终状态残留一次性对齐痕迹,规范化函数抹不平)。已把驱动脚本整体还原为 master 版(1ms 防抖 + 显式 flushAllSessions 检查点,master 的既有 CI 绿色设计),保留该提交中真正有增量的部分:orphan-recovery 的 restampSuperseded 断言(+2 行)、proofs 里两处显式 writeNow(消除防抖竞态,纯增益)。

本地验证(按仓库纪律只跑定向套件,全量交给远程 CI):typecheck 绿;storage-canonical-digest、issue3919-shadow-drift-forensics、snapshot-shadow、storage-orphan-recovery、web-blogfeed、launcher-opencode-inline、plugin-fork 共 89/89 通过;storage-engine-proofs + golden-storage-render 17/17 通过(含最初变红的 4 个 wire-parity)。

关于 @Maziar123 的跨 PR 协调(#4392/#4685):同意「后合入的一方适配」。具体地:无论 #4532 还是 #4685 先合,后合的一方需要在流式 canonicalDigest 里做两处小改——对象分支按键排序、改用共享跳过表 DIGEST_SKIPPED_KEYS(同时保留 __proto__ 独立判断),并加上你给的那条 key-order 一致性测试。本分支现在不能预先应用:现有字节恒等钉 canonicalDigest(v) === sha256(JSON.stringify(canonicalize(v))) 在 master 未排序的 canonicalize 下会立刻变红,适配必须与 #4685 同批落地。若 #4685 先合,我在本分支补一个提交承接;若本 PR 先合,按你的提议由你在 #4685 侧加 import 和测试。

结构化八问

  1. 是什么问题:PR 分支与目标分支两次出现合并冲突(系统自动转发),需要 rebase 解冲。
  2. 什么原因导致的:master 期间合入了改动同一批文件的其他 PR;其中 dsh 日志一行是两边独立修了同一个坏点(Plugin lanes: pin identity to client msgids, render outbound from the server-owned history | 插件车道服务端正史:消息身份钉在客户端 id 上,出站前缀由正史渲染 #4491 改名遗漏),博客/启动器/迁移测试则是作者的 CI 修复与 master 后来的正式方案撞车。
  3. 影响范围与严重程度:冲突解决本身零用户可见变化;唯一有意的行为变更是博客 CRLF 归一化从「有条件」改「无条件」(见上,修 \r 泄漏)。严重程度:无(纯冲突,功能面不变)。
  4. 怎么修复的:rebase 到当前 master,等价/被取代的碰撞保留 master 已验证设计,双方各有价值的取并集,并本地定向跑绿受影响的全部套件。
  5. 是否有回归问题:审过的既有行为——摘要摘要值字节保持不变(有恒等测试钉死)、dsh fork 盲点日志语义不变、存储 wire-parity 四线全绿、golden 未改动、博客渲染新增无条件去 \r(有意,已披露)。未发现无意回归。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标逻辑)。
  7. 是否改了设计方向:不涉及——明确放弃作者分支上被 master 取代的替代方案,方向回到 master。
  8. 是否引入了新的配置:不涉及。

收工判定:冲突清零,PR 可合(等 CI 绿);无需 owner 决策,故不开决策 issue。全量回归以远程 CI 为准。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

merge-guard-r1 结果:冲突已清零——分支已 rebase 到当前 master(9b6ec23)、新头 f8d91aa75 由并行会话(conflict-worker-N-r1,见上条)推送;我这边同时进行的合并提交被拒(non-fast-forward),未 force-push,不与已有解决结果竞争。推送后状态:mergeable,blocked = 等 CI 跑完(最近一次缓存快照 30 项检查无失败项)。本会话对 f8d91aa75 做了独立本地验证:typecheck 干净 + 受影响套件全绿(storage-engine-proofs 12/12、golden-storage-render 5/5、storage-canonical-digest 7/7、storage-orphan-recovery 11/11、launcher-opencode-inline 2/2、web-blogfeed 11/11、plugin-fork 50/50),与上条的解冲结论一致。无需设计决策,除验证外未对分支做任何改动。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b) Synced origin/master into this branch (clean merge, no conflicts) and pushed head → e683df6. Merge state cleared from behind to blocked (now awaiting review approval, not behind); CI re-running on the new sha with 0 failures so far.

@Maziar123

Copy link
Copy Markdown
Contributor

The sync comments say this PR now only waits for review approval. One point from my review is still open at head 20069028.

canonicalDigest in src/storage/canonical.ts walks objects with for (const k of Object.keys(current)) (L128), in insertion order. It also has its own skip list: k === "storageMode" || k === "tailGaps" (L130).

#4685 changes canonicalize to sort object keys at every depth (Object.keys(record).sort()). It also exports the shared skip table DIGEST_SKIPPED_KEYS. Whichever of the two merges second, the two producers then disagree for any object whose keys arrive in a different order. That is the false DIGEST DRIFT of #4392 / #4986 again, this time between the streaming digest and canonicalize. Your byte-identity test in tests/storage-canonical-digest.test.ts should turn red on that rebase, which is good, but the fix belongs in this PR:

  • iterate Object.keys(current).sort() in visit;
  • skip keys with Object.hasOwn(DIGEST_SKIPPED_KEYS, k) instead of the literal list. Keep the __proto__ guard.

With those two changes the streaming digest stays byte-identical to the canonicalize digest in either merge order.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

方向确认成立,但核查发现一个排序前提:这两行修复只有在「#4685 已并入 master」的基础上才能编译并转绿。

核查依据(head 20069028、master b08fbf361、对照 #4685 当前 diff):

八问:

  1. 是什么问题:两个摘要生产者各自维护键序规则与跳过清单;[PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 让 canonicalize 排键后,后合入的一方会让两者对任何键到达顺序不同的对象给出不同摘要 → 假 DIGEST DRIFT([Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392/Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 同类),而本 PR 的 shadow parity 通道(shadowDigest 已改走 canonicalDigest)会误报、连续 3 次即冻结信封写入。
  2. 什么原因导致的:同一 canonicalization 契约的两份手写平行实现,没有单一事实源。
  3. 影响范围与严重程度:两个都未合入时无影响(当前两两一致);风险集中在两者合并之间的中间窗口——对任何键序不同的对象误报漂移(messageRefs.byRef 等正是 [Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392 的字段证据,属常见形态),shadow 校验冻结 = 可用性级。
  4. 怎么修复的:按你给的方案——visit 迭代 Object.keys(current).sort() + 跳过条件换 Object.hasOwn(DIGEST_SKIPPED_KEYS, k)(保留 __proto__ 守卫)。建议顺序:先合 [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685(它自洽:自带回归测试 + storage-corpus manifest 重钉),再 rebase 本分支落这两行——byte-identity 测试本来就是为钉住 streaming ≡ canonicalize 设计的,rebase 后立即转绿。我可以马上接手,改动即两行 + 定向验证。
  5. 是否有回归问题:建议顺序下零行为漂移(rebase 后两个生产者共享同一序规则与同一跳过表)。不建议在本 PR 预先定义该常量:会与 [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 在 canonical.ts 同一批行上制造文本冲突,收益只是两边 rebase 各小一点。
  6. 是否破坏缓存:不涉及(digest 语义不变;manifest 重钉在 [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 侧完成)。
  7. 是否改了设计方向:不变,是把两个生产者收敛到单一 canonicalization 契约。
  8. 是否引入了新的配置:无。

若你希望本 PR 先合,也请说明——那样 #4685 的 rebase 需要吸收 sorted visit()(且表只在一处定义),我按那个顺序执行。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

本轮处理小结(详细八问分析见本楼上方我的评论):

z0d0dOr and others added 5 commits October 28, 2026 03:47
Windows SIGTERM terminates the child instead of running its graceful
shutdown handler. Trigger the existing loopback shutdown route on Windows
so the draining-marker regression exercises the same lifecycle as POSIX.

Wait for child close before deleting its database, destroy sockets owned
by the silent TCP fixture, and stop health polling when the child exits.
Keep both regression scenarios and their behavioral assertions intact.

Validation: the isolated CI-preloaded serial test file passes 6/6 and exits
naturally in 11.58 seconds; npm run typecheck and npm run build pass.
Windows CI can reset an aborted probe connection instead of ending it.
The silent TCP fixture did not handle errors on accepted sockets, so
ECONNRESET became an uncaught test exception after the probe finished.

Destroy reset sockets and add a real resetAndDestroy regression against
the same fixture. Keep the existing port-owner and ladder assertions.

Validation: the new regression reproduces ECONNRESET before the fix;
the isolated CI-preloaded serial test file then passes 7/7 and exits
naturally. Pure tsc and git diff --check pass.
…com/z0d0dOr/billion-context into pr4532-sync2

# Conflicts:
#	tests/issue4982-draining-twin-ladder.test.ts
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)
Merged origin/master (tip 09a5b5b77) into this branch and reconciled the concurrent fork-side CI-timeout commits; resolved the one resulting conflict in tests/issue4982-draining-twin-ladder.test.ts (kept master's #5050 always-HTTP P1-shutdown + the fork's fixture hardening, and collapsed a duplicate socket error handler the textual auto-merge had left behind). Pushed head 9a31c6121 — merge state is now clean (mergeable=true, blocked only on pending CI checks, no longer dirty); no design decisions were needed.

…xianglei#4532)

ranxianglei#4685 landed on master (canonicalize sorts object keys at every depth +
shared DIGEST_SKIPPED_KEYS table), but this PR's streaming canonicalDigest
still iterated raw Object.keys order with its own literal skip list. The
two producers then disagreed for any object whose keys arrive in a
different order — the false DIGEST DRIFT of ranxianglei#4392/ranxianglei#4986 again, this time
between the streaming digest and canonicalize; tests/issue4392-canonical-key-order.test.ts
went red on 8 shards (e.g. run 40909901398).

Align visit() with the legacy producer's effective byte order:
- skip via Object.hasOwn(DIGEST_SKIPPED_KEYS, k), keeping the __proto__ guard;
- iterate array-index keys first in ascending numeric order, then the
  remaining keys lexicographically. Plain .sort() alone is NOT
  byte-identical: spec enumeration (Object.keys / JSON.stringify) emits
  array-index keys ascending numeric regardless of insertion or lexicographic
  order, so an index-shaped pair like {"10","2"} must serialize as "2","10".
  The pinned byte-identity test (streaming digest preserves JSON omissions,
  sparse arrays and property order) catches exactly this.

Byte-identity to sha256(JSON.stringify(canonicalize(v))) now holds for any
key arrival order, in either merge order relative to ranxianglei#4685.

Verified locally: issue4392/issue4986/storage-canonical-digest suites
13/13, storage parity integration (engine-proofs + ranxianglei#3407 + ranxianglei#3919 + corpus)
26/26, typecheck clean.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 红已修并推送(b54dafd9a)——上一轮讨论的排序前提已自然兑现:#4685 先合入了 master,本分支 merge master 后两个生产者失步,修复即按你给的方案落地(有一处必要修正,见第 4 问)。

八问:

  1. 是什么问题:8 个含 tests/issue4392-canonical-key-order.test.ts 的分片全红,失败断言「an order-only difference must not drift」(1 !== 0)。[PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 合入后 canonicalize 排键 + 共享跳过表,而本 PR 的流式 canonicalDigest 仍按 Object.keys 原始序迭代、自带字面跳过清单——两者对任何键到达顺序不同的对象给出不同摘要,[Bug][storage][shadow-digest-drift] Repeated DIGEST DRIFT in shadow mode on v0.1.204 (follow-up to #3919) #4392/Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 的假 DIGEST DRIFT 在「流式 vs canonicalize」之间复现。
  2. 什么原因导致的:合并顺序窗口内本分支未做对齐([PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 先落地);非 flake、确定性。
  3. 影响范围与严重程度:当前只表现为该测试分片必红;若带着失步合入,生产 shadow parity 对任何键序不同的对象误报漂移,连续 3 次即冻结信封写入(可用性级)。
  4. 怎么修复的:按你的方案 + 一处必要修正。(a) 跳过条件换 Object.hasOwn(DIGEST_SKIPPED_KEYS, k),保留 __proto__ 守卫;(b) 迭代顺序 = 数组索引形键先按数值升序,其余键字典序。没有直接用纯 .sort():JS 规范枚举(Object.keys 与 JSON.stringify 同规则)把整数字形键("2"、"10")恒按数值升序输出,纯字典序会写成 "10","2",与 JSON.stringify(canonicalize(v)) 的最终字节不符——本 PR 的 byte-identity 测试(fixture 含 "10":"ten","2":"two")在纯 .sort() 下本地立刻变红(修复前后各复现一次确认)。修正后的顺序使 byte-identity 对任意键到达顺序、任意合并顺序都成立。
  5. 是否有回归问题:逐项过过—canonicalize 本体零改动([PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 语义与 storage-corpus manifest 钉不受影响,corpus round-trip 本地绿);流式侧仅改变「≥2 个整数字形键对象」的输出字节(相对本 PR 上一 head,从未发布,已发布构建无影响);数组/Map/Set 顺序语义、__proto__ 与跳过清单语义均不变。本地定向:三个 digest 套件 13/13 + storage parity 集成(engine-proofs ×4 wire、unified shadow storage digest drift on long dsh sessions (telemetry-only) | 统一存储影子模式摘要漂移 #3407、[Bug][storage] Shadow digest drift clears ingest watermark and forces full-tail re-ingest (v0.1.201) #3919、corpus)26/26 + typecheck 干净;全矩阵以 CI 为准。
  6. 是否破坏缓存:不涉及(digest 每次现算,无缓存/重试/游标参与)。
  7. 是否改了设计方向:不变——两个生产者收敛到同一 canonicalization 契约(共享跳过表 + 同一有效键序)。
  8. 是否引入了新的配置:无。

@Maziar123 第 4 问的 index-key 顺序修正是对你原方案的偏离(纯 .sort() 会被 byte-identity 钉抓出),请过目;如认可,此 PR 待 CI 重跑转绿后即回到「仅待 review 批准」状态。

@ranxianglei
ranxianglei merged commit 44aca70 into ranxianglei:master Oct 30, 2026
42 of 49 checks passed
@Maziar123

Maziar123 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Confirmed: the index-key correction is right, and my plain .sort() suggestion was wrong on that point. JSON.stringify walks keys in OrdinaryOwnPropertyKeys order: array-index keys first in ascending numeric order, then string keys in insertion order. So JSON.stringify(canonicalize(v)) puts "2" before "10", whatever order canonicalize inserted them in. A plain lexical sort gives "10","2" and breaks byte identity; numeric-first-then-lexical matches what is actually serialized. The "10"/"2" fixture in the byte-identity test is the right pin for it. Nothing further from our side on #4532.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Storage shadow digests materialize full-session JSON and spike heap

3 participants