Repository navigation
[PR] perf: verify stored sessions in bounded batches - #4540
ranxianglei merged 12 commits into
Conversation
85afa71 to
ab13c43
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突车道处理完毕:已把 master 合入本 PR 分支并推送,PR 恢复可合并。 过程:系统报告 merge_conflict(dirty)。接手时先核对 GitHub 实时状态——此时 API 已是
验证(合并树上,改动区域定向):typecheck 通过、build 通过(dist/index.js 等全部产物正常)、PR 自带定向测试 八问:
判定:冲突已解除,PR 处于可合并状态,等 CI 转绿即可由 owner 合并。 |
|
(ai: vllm-qwen/qwen3.8-27b) PR Review — perf: verify stored sessions in bounded batches(#独立审核,未复用 issue 侧结论) 审核对象:合并 master 后的 head 结构化八问
合成内存对比(issue 要求的验证项,PR 内未附,审核方代跑)300 块 × 300 KiB(≈90 MiB 历史),Node 22.23.2 linux,--expose-gc,gc 后测验证腿 RSS 增量:旧路径 553.9 MB vs 新路径 7.2 MB(约 77×)。与 #4529 的流式 digest 结论方向一致。 发现与建议(均不阻塞)
结论: 可合(N1–N3 非阻塞;等 CI 车道在合并 head 上转绿 + owner 合并)。 |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 本 PR 修 #4538(采样验证全量水合 + 重建第二份完整会话 → 内存尖峰),与 #4532 互补且无 diff 重叠;无其他 batched-verification 在途 PR。PR 内三处附带改动各自有源、均不构成重复:CI commit = #4548(专门 PR #4549 已关未合,本 PR 成为载体);relay.ts 改动 + 两处测试期望更新 = #4525 train 合并的残留损伤(#4554 家族,专门修复 PR #4556 已关未合,对照当前 master 代码确认该缺陷仍在)。 标签: pr, bug, scope/storage, p2(存储验证路径的性能/内存缺陷;触发条件窄——verify 采样命中大会话——故评 p2 非 p1)。 审查对象: 当前 head 结构化八问
行为变更披露(旧→新 + 原因):
请作者补一项(非代码阻塞): PR 正文目前只描述 #4538 主体,请加一小节列明三处附带件及其来源——① CI commit(Fixes #4548,吸收已关闭的 #4549);② relay.ts hunk(#4554 家族 #4241 碰撞修复,#4556 关而未合、master 仍带缺陷,本分支为载体);③ 两处测试期望更新(陈旧测试修复,反事实证据见第 5 问)。否则 reviewer 无法追溯这些 diff 的来历。 收工判定: 代码可合;待作者补正文披露 + 确认合入后红 shard 随 #4548 修复收敛(或归入 master 基线线处理)。合并权在 owner,我不代合。 |
…ot this PR's master 146b34f fails the unused-exports gate on this exact line (INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck): the type-only export #4545 introduced was missed by the 4d34abf de-export sweep. This PR touches none of it — every PR rebased past that master inherits the red (#4599 storm log). Same one-word fix as #4606 commit 1 and the in-branch precedents (#4607/#4595/#4540/#4618/#4622): drop the leading export on a symbol referenced only within its own file. Drop this commit when rebasing after #4606 merges.
The engineLane=server metadata added by f590da9 explains the entire drift: removing that one field reproduces the old metadata digest. Regeneration changes only metadata and the enclosing digest; all wire bodies, state, originals and statistics remain identical.
Compare unified reads with an independent envelope reader plus the documented migration. Persist alignment before rollback/corrupt-index checks, select legacy explicitly, and assert the recovered raw ID under its ledger ref. Keep checkpoint writes deterministic so sampled ingest cadence cannot change the seed between reader legs. All 28 targeted storage tests pass; no product migration or digest assertion is disabled.
e274b22 to
9397bc8
Compare
Reconcile shared storage proofs with upstream citation-normalized restart comparisons, retain legacy/full fork coverage and Windows handle cleanup, and preserve both storage preview and batched verification types.
Windows SIGTERM terminates the child instead of running its graceful shutdown handler. Trigger the existing loopback shutdown route on Windows so the draining-marker regression exercises the same lifecycle as POSIX. Wait for child close before deleting its database, destroy sockets owned by the silent TCP fixture, and stop health polling when the child exits. Keep both regression scenarios and their behavioral assertions intact. Validation: the isolated CI-preloaded serial test file passes 6/6 and exits naturally in 11.58 seconds; npm run typecheck and npm run build pass.
Windows CI can reset an aborted probe connection instead of ending it. The silent TCP fixture did not handle errors on accepted sockets, so ECONNRESET became an uncaught test exception after the probe finished. Destroy reset sockets and add a real resetAndDestroy regression against the same fixture. Keep the existing port-owner and ladder assertions. Validation: the new regression reproduces ECONNRESET before the fix; the isolated CI-preloaded serial test file then passes 7/7 and exits naturally. Pure tsc and git diff --check pass.
tests/issue4982-draining-twin-ladder.test.ts: both sides independently fixed ranxianglei#5050. Ours (858cb70+47b54b36): socket error->destroy + close() wrapper + platform-branched shutdown. Master (0a1f534+9bbf3165): /__bili/shutdown convergence (ranxianglei#3544) + peer-abort-safe squatter fixture. Take master's canonical version (shipped in v0.1.209); our duplicate fix drops, same precedent as the dropped ranxianglei#4606-duplicate unused-exports commit. # Conflicts: # tests/issue4982-draining-twin-ladder.test.ts
[bot] (ai: vllm-qwen/qwen3.8-27b) 上游冲突已解决并推送回 PR 分支: 八问:
验证状态(如实):本次合并带入的增量不含运行时代码,唯一人工触碰的文件与 master 已发布版本字节一致,故未本地跑定向/全量测试(§7.2:全量套件判定归远程 CI);CI 正在新 head 过程披露:处理期间沙箱工作区被重置过一次(旧 .git 被清),全部状态从远端重新拉取重建;出站网络间歇性断连,多次重试后完成。首次自动修复尝试内完成,冲突不涉及设计取舍,无需 decision request。 结论:冲突已清除,待 CI 绿 + owner 合并。 |
Fixes #4538. Complements #4532 without depending on it.
Sampled verification previously hydrated all stored messages and blocks, then reconstructed a second complete session while the live session remained resident. It now compares the header separately, reads blocks in batches of 32, and hydrates only the sequences referenced by tail/plugin snapshot pointers in batches of 32. Only message hashes and diagnostic fingerprints survive between batches. The synchronous and async paths share the comparison logic; SQLite and MySQL support the bounded reads.
The change preserves canonical normalization, block order, pointer order and duplicates, media hydration, orphan-original precedence, and existing drift recovery/diagnostics. Normal cold loads, persistence formats, configuration, and sampling cadence are unchanged. Session-row metadata/CCR/orphan JSON and individual oversized rows can still be large; this bounds the hydrated verification batches, not total process memory.
Rebased on master 0d9fdae. The branch carries #4549's verified-cache mtime fix with its original author retained, and makes the Windows redaction fixture pin/restore HOME and USERPROFILE. Two lifecycle fixtures now follow #4525's preserved-fold recovery contract. The relay avoids applying a raw-client tail offset to #4531's already-rendered judge baseline; the original dropped-prefix integration test remains unchanged and detects the defect again. The scope-attribution and lifecycle-ledger fixes are now supplied by master rather than duplicated here.
Validation on the refreshed branch:
A single synthetic memory comparison on Node v24.12.0 used 512 distinct 256 KiB block texts (128 MiB resident logical text) and 32 small tail messages in separate processes. The old full replay/digest flow peaked at 870.0 MiB RSS; the actual new verifier peaked at 346.3 MiB (60.2% lower). Both verified successfully. Rows were generated on demand to simulate database results, so this isolates verification allocation and excludes SQL/driver I/O; it is not a claim about total production memory.