Repository navigation
[PR] fix: stream storage verification digests to reduce peak memory - #4532
ranxianglei merged 15 commits into
Conversation
ab707eb to
418118e
Compare
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。#4532 是唯一针对 #4529 的 PR(open+closed 全量检索,无其他 PR 触碰 streaming canonical digest)。相关历史条目均已合并且机制不同:#3423(digest 按内容而非 Map 插入序比较)、#3948(漂移维度命名)、#4332(结构性漂移升级告警);#3437(活跃会话峰值内存 ~1.26GiB)是 #4529 正文自己声明的独立关注点,不同失败域。 标签: 结构化八问
测试与 CI
判定:可合。 单模块闭环小修,digest 值逐字节不变,内存收益已独立复现确认。合并权在你:#4532 |
(ai: vllm-qwen/qwen3.8-27b) CI 自动修复车道结论(docs-sync) 查重:与 #4520 同根因(master 侧缺 config-reference seed 条目)——#4520 已修复,本 PR 分支随后被 rebase 到含该修复的 master,我本地准备的同名修复判定为重复、已丢弃,未向分支推送任何提交。 八问
判定:无需追加提交;CI 在新头 418118e 重跑后 docs-sync 应转绿。若其余 job 另有红项,另行处理。 |
71c4361 to
ca791aa
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决,PR 恢复可合并。已 rebase 到最新 master(ba5ab0b),新 head 八问:
判定:冲突已解,等 CI 绿灯后可由 owner 合并。PR: #4532 |
ca791aa to
b36fc4e
Compare
… table (ranxianglei#4392) canonicalize hashed insertion order, so the shadow live leg and the replay leg (messageRefs.byRef rebuilt in ledger order by loadLegacyView) produced two digests for the same value on a SINGLE instance, escalated after three samples and froze envelope writes. - sort object keys at every depth; arrays keep order, Map/Set envelopes keep their shape - move the storageMode/tailGaps drops into one exported table DIGEST_SKIPPED_KEYS so the streamed canonicalDigest (ranxianglei#4532) imports the same skip set instead of hand-copying a second list - pin the table literally and pin depth-independent dropping
|
For #4392, both digest producers must agree on key order and skipped keys (see the reporter's decision in #4392). #4685 changes
for (const k of Object.keys(current).sort()) {
if (Object.hasOwn(DIGEST_SKIPPED_KEYS, k)) continue;
// Assigning __proto__ to canonicalize's plain output creates no JSON property.
if (k === "__proto__") continue;
// ...rest of the loop unchanged
}Your test("#4392: key order is not content on either digest path", () => {
const a = { state: { messageRefs: { byRef: { m00002: "b", m00001: "a" } } }, stats: { requests: 1 } };
const b = { stats: { requests: 1 }, state: { messageRefs: { byRef: { m00001: "a", m00002: "b" } } } };
assert.equal(canonicalDigest(a), canonicalDigest(b));
matchesLegacy(a);
matchesLegacy(b);
});Whichever PR lands second takes this on. If #4532 lands first, I will add the import and the test to #4685. |
…nk seams (ranxianglei#4529) Extend storage-canonical-digest.test.ts with deterministic cases the original suite left uncovered: surrogate pairs landing exactly on the 64 KiB hash-chunk seam (writeString boundary pull-back), an MB-scale retained history spanning many chunks end-to-end, deep nesting (visit/JSON.stringify recursion parity) and wide fan-out (separator + pending-buffer flush). Each asserts canonicalDigest(v) === sha256(JSON.stringify(canonicalize(v))), so any chunking or surrogate-split regression fails loudly rather than drifting silently.
Compare unified reads with an independent envelope reader plus the documented migration. Persist alignment before rollback/corrupt-index checks, select legacy explicitly, and assert the recovered raw ID under its ledger ref. Keep checkpoint writes deterministic so sampled ingest cadence cannot change the seed between reader legs. All 28 targeted storage tests pass; no product migration or digest assertion is disabled.
b36fc4e to
f8d91aa
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决:分支已 rebase 到当前 master(9b6ec23),新头 逐处冲突的处理(共 5 个提交碰撞,全部已在本地定向验证):
本地验证(按仓库纪律只跑定向套件,全量交给远程 CI):typecheck 绿;storage-canonical-digest、issue3919-shadow-drift-forensics、snapshot-shadow、storage-orphan-recovery、web-blogfeed、launcher-opencode-inline、plugin-fork 共 89/89 通过;storage-engine-proofs + golden-storage-render 17/17 通过(含最初变红的 4 个 wire-parity)。 关于 @Maziar123 的跨 PR 协调(#4392/#4685):同意「后合入的一方适配」。具体地:无论 #4532 还是 #4685 先合,后合的一方需要在流式 结构化八问
收工判定:冲突清零,PR 可合(等 CI 绿);无需 owner 决策,故不开决策 issue。全量回归以远程 CI 为准。 |
(ai: vllm-qwen/qwen3.8-27b) merge-guard-r1 结果:冲突已清零——分支已 rebase 到当前 master(9b6ec23)、新头 |
|
The sync comments say this PR now only waits for review approval. One point from my review is still open at head
#4685 changes
With those two changes the streaming digest stays byte-identical to the |
(ai: vllm-qwen/qwen3.8-27b) 方向确认成立,但核查发现一个排序前提:这两行修复只有在「#4685 已并入 master」的基础上才能编译并转绿。 核查依据(head
八问:
若你希望本 PR 先合,也请说明——那样 #4685 的 rebase 需要吸收 sorted |
…28_pr4532-ci-timeout
[bot] 🏷 本轮处理小结(详细八问分析见本楼上方我的评论):
|
Windows SIGTERM terminates the child instead of running its graceful shutdown handler. Trigger the existing loopback shutdown route on Windows so the draining-marker regression exercises the same lifecycle as POSIX. Wait for child close before deleting its database, destroy sockets owned by the silent TCP fixture, and stop health polling when the child exits. Keep both regression scenarios and their behavioral assertions intact. Validation: the isolated CI-preloaded serial test file passes 6/6 and exits naturally in 11.58 seconds; npm run typecheck and npm run build pass.
Windows CI can reset an aborted probe connection instead of ending it. The silent TCP fixture did not handle errors on accepted sockets, so ECONNRESET became an uncaught test exception after the probe finished. Destroy reset sockets and add a real resetAndDestroy regression against the same fixture. Keep the existing port-owner and ladder assertions. Validation: the new regression reproduces ECONNRESET before the fix; the isolated CI-preloaded serial test file then passes 7/7 and exits naturally. Pure tsc and git diff --check pass.
…com/z0d0dOr/billion-context into pr4532-sync2 # Conflicts: # tests/issue4982-draining-twin-ladder.test.ts
(ai: vllm-qwen/qwen3.8-27b) |
…xianglei#4532) ranxianglei#4685 landed on master (canonicalize sorts object keys at every depth + shared DIGEST_SKIPPED_KEYS table), but this PR's streaming canonicalDigest still iterated raw Object.keys order with its own literal skip list. The two producers then disagreed for any object whose keys arrive in a different order — the false DIGEST DRIFT of ranxianglei#4392/ranxianglei#4986 again, this time between the streaming digest and canonicalize; tests/issue4392-canonical-key-order.test.ts went red on 8 shards (e.g. run 40909901398). Align visit() with the legacy producer's effective byte order: - skip via Object.hasOwn(DIGEST_SKIPPED_KEYS, k), keeping the __proto__ guard; - iterate array-index keys first in ascending numeric order, then the remaining keys lexicographically. Plain .sort() alone is NOT byte-identical: spec enumeration (Object.keys / JSON.stringify) emits array-index keys ascending numeric regardless of insertion or lexicographic order, so an index-shaped pair like {"10","2"} must serialize as "2","10". The pinned byte-identity test (streaming digest preserves JSON omissions, sparse arrays and property order) catches exactly this. Byte-identity to sha256(JSON.stringify(canonicalize(v))) now holds for any key arrival order, in either merge order relative to ranxianglei#4685. Verified locally: issue4392/issue4986/storage-canonical-digest suites 13/13, storage parity integration (engine-proofs + ranxianglei#3407 + ranxianglei#3919 + corpus) 26/26, typecheck clean.
(ai: vllm-qwen/qwen3.8-27b) CI 红已修并推送( 八问:
@Maziar123 第 4 问的 index-key 顺序修正是对你原方案的偏离(纯 |
|
Confirmed: the index-key correction is right, and my plain |
Storage shadow verification materializes a canonical object graph and a complete JSON string for each digest. Large retained histories therefore cause a substantial temporary memory spike during sampled verification.
This change walks the storage view directly and feeds canonical JSON incrementally to SHA-256. Long strings are escaped in 64 Ki-character chunks without splitting surrogate pairs. Scalar normalization is shared with the existing canonicalizer, and the shared shadow digest entry point covers both synchronous and asynchronous verification. Existing storage formats and digest values are preserved.
Fork diagnostics also read the session ID from
dispatchScope, matching the upstreamsidScoperename; the stale reference otherwise prevents current master from typechecking.Fixes #4529. Related: #3133, #3140, #3437.
Validation
storage-canonical-digest,issue3407-shadow-blockcontents-order,storage-corpus,issue4215-fork-gate-diagnostics, anddsh-llm-stream-attribution.sha256(JSON.stringify(canonicalize(value)))as the oracle, covering normalization, collection/property order, JSON omissions, long Unicode strings, and changed retained history.npm run build, andgit diff --checkpassed. Independent scoped review passed.docs-syncgeneration and artifact-diff check pass with the updated configuration seeds from master. No generated documentation changes are required.Full-suite validation is left to repository CI. The memory optimization is limited to digest allocation; it does not resolve the separate full-view reload or active-session admission costs.