Repository navigation
[PR] perf(storage): build bounded boot file indexes incrementally - #6663
Conversation
[bot] (ai: vllm-qwen/qwen3.8-27b) PR #6663 审核完成:代码可合;2 个红分片是继承自 master 的已知 flake(已在 master 修复),作者 rebase 到最新 master 后即全绿。 (说明:本会话曾因 32 分钟无输出被重启一次——前一个会话耗在了全新工作区的环境准备上(依赖安装 + kernel 构建);本次会话已完成全部审核与定向验证。) 查重: 不重复 — 按 bounded boot file index / quadratic / filesByRoot 检索 open+closed 无同题;本 PR 是 #6662 的唯一修复,与 #5219(decode cap)、#3199(background scan)机制不重叠,PR 描述中的 prior-work 核对与我的检索一致。 标签: p2 + scope/storage(存储启动路径性能问题;跨客户端,无特定客户端标签)。 ① 方向(原 issue #6662): ② 回归审查(触碰路径的既有行为,逐项):
③ 代码:4 行改动,每根目录一个数组、原地 push;无别名风险(每根目录各自新建数组,两个 map 条目互不共享);下游两处只读。干净。 ④ 测试有效性:未新增回归测试——合理,该改动可证明行为保持(同序同内容),4 个 focused 套件(27 测试)已钉住 bounded-boot 行为,在 rebase 后的 head 本地全绿(27/27)。benchmark 脚本的功能断言是真的(精确 indexed 计数含 companion 排除、zero-warm、双根首/尾原文恢复),不是恒真断言;CI 不加 timing 阈值是合理取舍(CI 上 timing 易 flake),PR 描述对测量边界(合成文件树、非生产吞吐)的诚实限定值得肯定。 ⑤/⑤b diff 与代码整洁:2 文件 +75/-1,每一行都与 PR 目的相关;benchmark 脚本 hermetic(隔离 XDG 目录、BILI_SESSIONS_DIR、unified=full、sqlite、zstd off)、argv 校验、try/finally 清理、命名达意,沿用既有 本地定向验证(rebase 到 origin/master 9495c76 后):
CI:35 绿 / 13 skip / 2 红。2 红均为继承自 master 的已知 flake,与本 PR 零交集:
八问:
判定:代码可合 — 待作者 rebase 最新 master(清掉 2 个继承红分片)→ CI 全绿 → owner 人工合并。 |
|
(ai: vllm-qwen/qwen3.8-27b) Hi @Afloat16 — thanks for this fix; the O(N^2)→O(N) index build in Heads-up: a parallel PR #6688 was opened later with the same fix plus a regression gate ( To make this PR merge-ready under the auto-merge gate (§7.4 gate 2 requires a regression test), could you add the test file from #6688 to this branch? I can post the full file content here if that is easier. Your benchmark script ( |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Verified the parallel PR before weighing in (didn't just take the comment at face value). What I confirmed:
Clarification on my earlier "代码可合": that was for a human merge (provably behavior-preserving; the 4 focused suites pin it). For auto-merge, §7.4 gate 2 (a regression test that reproduces the bug and now passes) is a hard requirement — #6663 as-is fails only that gate; every other gate (single-module, clean diff, no format/wire/kernel change, refs #6662, green-on-rebase) passes. Adding #6688's test closes the gap. Recommendation — keep #6663, port #6688's test in, close #6688:
Who does what:
八问:
判定: 建议保留 #6663 + 移植 #6688 回归测试(已验证红/绿)+ rebase,关 #6688;需 @Afloat16 补测试文件(我无法代推其 fork),或你拍板走 fallback(保留 #6688、我移植 benchmark 进去)。等你定。 |
Closes #6662
Problem
Bounded boot copies the complete root file list on every accepted path. For large session archives this adds quadratic reference copying before listen, including the default zero-warm-session path.
Change
Create one list for each hot/cold root and append accepted paths in place. The complete disk index and traversal order are preserved. No persistence format, warm-budget policy, decode window, configuration, dependency or kernel behavior changes.
Include a standalone real-store benchmark with exact indexed-file counts, excluded companion controls, zero-warm assertions and hot/cold original-content recovery checks. It uses isolated temporary data roots and cleans them up after the background scan finishes.
Validation
bounded-storage,issue5219-bounded-boot-decode-cap,issue3199-overview-cold-scanandstorage-full-flip, using the test-environment and storage-baseline preloads.src/persist.ts, benchmark syntax checking,npm run buildandgit diff --checkpass.On Node 24.22.1/Linux, three fresh-process runs per revision with 20,000 state files in each root (40,000 total) measured:
Each run indexed all 40,000 state files, excluded the two content-store companions, warmed zero sessions, and successfully recovered the first and last originals from both roots after the background scan. A 100-file-per-root control took 5.65 ms before and 5.51 ms after.
These are local synthetic file-tree measurements, run in two consecutive groups. They demonstrate this startup path's avoidable copying cost; they do not establish a general CLI startup or production throughput speedup. Filesystem walking, hot-file stat/sort and background decoding remain separate costs.
Reproduction:
The benchmark is on
Afloat16:2026-11-14_linear-boot-file-index. To reproduce the comparison, run that same script in a checkout of the base1ff89e878522f3e4d4147067973e2dc26e648923, then on the proposed branch. It creates temporary hot and cold session trees, configures an isolated SQLite sidecar, and times the realSessionStore.boot({ maxSessions: 0, maxBytes: 64 * 1024 * 1024 }). Fixture generation and module loading precede the timer; the background scan and lazy-load checks follow it. It leaves the caller's data roots untouched and removes its temporary data.The complete repository typecheck/full test suite and live-provider client E2E were not run locally for this branch; those remain CI/maintainer gates. The focused check and normal build completed successfully.
Scope and prior work
Base
1ff89e878522f3e4d4147067973e2dc26e648923. Published tree720bc34a4ec37b50d3fbef60536de99e43b5499dmatches the tested local tree.Checked all-state searches and Afloat16's history. This complements the decode cap in #5219 and background scan in #3199 by removing a separate pre-listen path-index copying cost; it does not repeat those changes.