Skip to content

[PR] perf(storage): build bounded boot file indexes incrementally - #6663

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-boot-file-index
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-boot-file-index

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6662

Problem

Bounded boot copies the complete root file list on every accepted path. For large session archives this adds quadratic reference copying before listen, including the default zero-warm-session path.

Change

Create one list for each hot/cold root and append accepted paths in place. The complete disk index and traversal order are preserved. No persistence format, warm-budget policy, decode window, configuration, dependency or kernel behavior changes.

Include a standalone real-store benchmark with exact indexed-file counts, excluded companion controls, zero-warm assertions and hot/cold original-content recovery checks. It uses isolated temporary data roots and cleans them up after the background scan finishes.

Validation

  • 27/27 focused tests pass across bounded-storage, issue5219-bounded-boot-decode-cap, issue3199-overview-cold-scan and storage-full-flip, using the test-environment and storage-baseline preloads.
  • Focused strict TypeScript checking of src/persist.ts, benchmark syntax checking, npm run build and git diff --check pass.
  • The unchanged source and proposed source both pass the benchmark's functional assertions. The measured improvement is performance; no timing threshold was added to CI.

On Node 24.22.1/Linux, three fresh-process runs per revision with 20,000 state files in each root (40,000 total) measured:

Revision Boot times (ms) Median (ms)
Base 3599.4, 3595.0, 3187.7 3595.0
Incremental index 264.5, 236.8, 253.5 253.5

Each run indexed all 40,000 state files, excluded the two content-store companions, warmed zero sessions, and successfully recovered the first and last originals from both roots after the background scan. A 100-file-per-root control took 5.65 ms before and 5.51 ms after.

These are local synthetic file-tree measurements, run in two consecutive groups. They demonstrate this startup path's avoidable copying cost; they do not establish a general CLI startup or production throughput speedup. Filesystem walking, hot-file stat/sort and background decoding remain separate costs.

Reproduction:

node --max-old-space-size=512 --expose-gc --import tsx scripts/bench-bounded-boot-index.mjs 20000

The benchmark is on Afloat16:2026-11-14_linear-boot-file-index. To reproduce the comparison, run that same script in a checkout of the base 1ff89e878522f3e4d4147067973e2dc26e648923, then on the proposed branch. It creates temporary hot and cold session trees, configures an isolated SQLite sidecar, and times the real SessionStore.boot({ maxSessions: 0, maxBytes: 64 * 1024 * 1024 }). Fixture generation and module loading precede the timer; the background scan and lazy-load checks follow it. It leaves the caller's data roots untouched and removes its temporary data.

The complete repository typecheck/full test suite and live-provider client E2E were not run locally for this branch; those remain CI/maintainer gates. The focused check and normal build completed successfully.

Scope and prior work

Base 1ff89e878522f3e4d4147067973e2dc26e648923. Published tree 720bc34a4ec37b50d3fbef60536de99e43b5499d matches the tested local tree.

Checked all-state searches and Afloat16's history. This complements the decode cap in #5219 and background scan in #3199 by removing a separate pre-listen path-index copying cost; it does not repeat those changes.

@ranxianglei ranxianglei changed the title perf(storage): build bounded boot file indexes incrementally [PR] perf(storage): build bounded boot file indexes incrementally Nov 14, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

PR #6663 审核完成:代码可合;2 个红分片是继承自 master 的已知 flake(已在 master 修复),作者 rebase 到最新 master 后即全绿。

(说明:本会话曾因 32 分钟无输出被重启一次——前一个会话耗在了全新工作区的环境准备上(依赖安装 + kernel 构建);本次会话已完成全部审核与定向验证。)

查重: 不重复 — 按 bounded boot file index / quadratic / filesByRoot 检索 open+closed 无同题;本 PR 是 #6662 的唯一修复,与 #5219(decode cap)、#3199(background scan)机制不重叠,PR 描述中的 prior-work 核对与我的检索一致。

标签: p2 + scope/storage(存储启动路径性能问题;跨客户端,无特定客户端标签)。

① 方向(原 issue #6662):bootBounded 对每个被接受的文件用 [...旧数组, file] 重建每根目录的文件索引——O(N²) 引用拷贝,发生在 listen 之前,且 maxSessions=0(零 warm)时同样发生。已对照 base 代码核实缺陷真实存在(src/persist.ts:1005)。修复打在根因(拷贝本身)上,不是治标。方向正确。

② 回归审查(触碰路径的既有行为,逐项):

  • diskFiles 播种顺序(hot 根在前、cold 在后;根内 walkJsonFiles 序):保持——同一循环同一 push;
  • filesByRoot 映射语义:旧代码仅对「有 ≥1 个被接受文件」的根建条目,新代码恒有条目(可能为空数组)——两个消费点(src/persist.ts:1021 unified hot window、:1061 legacy full parse)都用 ?? [],行为完全等价;
  • legacy 别名 tie-break(savedAt >= previous.savedAt,src/persist.ts:1070)依赖迭代顺序:顺序未变 → 同一胜出者;
  • warm 集合仲裁(savedAt-desc 稳定排序):输入顺序未变 → 输出相同;
  • content-store companion 排除、warm 预算、decode 窗口、持久化格式、配置、依赖、kernel:全部未动。
  • base 1ff89e8785 是当前 master 的祖先,且 src/persist.ts 在 base 与 origin/master 之间逐字节一致——cherry-pick 干净应用,无语义漂移。
    结论:无回归。

③ 代码:4 行改动,每根目录一个数组、原地 push;无别名风险(每根目录各自新建数组,两个 map 条目互不共享);下游两处只读。干净。

④ 测试有效性:未新增回归测试——合理,该改动可证明行为保持(同序同内容),4 个 focused 套件(27 测试)已钉住 bounded-boot 行为,在 rebase 后的 head 本地全绿(27/27)。benchmark 脚本的功能断言是真的(精确 indexed 计数含 companion 排除、zero-warm、双根首/尾原文恢复),不是恒真断言;CI 不加 timing 阈值是合理取舍(CI 上 timing 易 flake),PR 描述对测量边界(合成文件树、非生产吞吐)的诚实限定值得肯定。

⑤/⑤b diff 与代码整洁:2 文件 +75/-1,每一行都与 PR 目的相关;benchmark 脚本 hermetic(隔离 XDG 目录、BILI_SESSIONS_DIR、unified=full、sqlite、zstd off)、argv 校验、try/finally 清理、命名达意,沿用既有 scripts/bench-*.mjs 惯例(其余 bench-* 脚本同样不被 package.json 引用,一致)。通过。

本地定向验证(rebase 到 origin/master 9495c76 后):

  • 27/27:bounded-storage + issue5219-bounded-boot-decode-cap + issue3199-overview-cold-scan + storage-full-flip(带 test-environment + storage-baseline preload);
  • A/B benchmark(5000 文件/根):base boot 100.4ms → PR 53.6ms,差距随规模二次方扩大(作者 20k 数据:3595 → 253ms)——机制真实、修复有效;
  • benchmark 脚本端到端可跑(2000 文件/根:indexed=4000、warm=0、恢复断言全过)。

CI:35 绿 / 13 skip / 2 红。2 红均为继承自 master 的已知 flake,与本 PR 零交集:

八问:

  1. 是什么问题:磁盘会话文件多时,代理每次启动在 listen 前把文件索引二次方拷贝 N² 次,拖慢启动(20k 文件/根实测 ~3.6s,可降到 ~0.25s)。
  2. 什么原因导致的:filesByRoot.set(root, [...(filesByRoot.get(root) ?? []), file]) 每文件整数组拷贝——根因是拷贝本身,不是下游消费慢。
  3. 影响范围与严重程度:任何带大会话档案(数千文件)的实例每次 boot 都付出该代价,零 warm(maxSessions=0)也发生;p2——能用但启动明显延迟,无数据丢失。
  4. 怎么修复的:每根目录一次性分配数组、原地 push(O(N)),遍历顺序与过滤逻辑逐字节保持。
  5. 是否有回归问题:无——② 所列 5 项既有行为逐项等价;27/27 定向测试绿;A/B 实测唯一差异是速度。
  6. 是否破坏缓存:不涉及(未触碰缓存/游标/去重逻辑)。
  7. 是否改了设计方向:不涉及(同架构、同数据流)。
  8. 是否引入了新的配置:无(benchmark 脚本是开发工具,非配置面;无新 env/默认值)。

判定:代码可合 — 待作者 rebase 最新 master(清掉 2 个继承红分片)→ CI 全绿 → owner 人工合并。

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Hi @Afloat16 — thanks for this fix; the O(N^2)→O(N) index build in bootBounded() is verified correct (red/green regression check done on the parallel PR).

Heads-up: a parallel PR #6688 was opened later with the same fix plus a regression gate (tests/issue6662-bounded-boot-file-index.test.ts — fails on pre-fix master with bounded boot took 3426.4ms ... bound 1000ms, passes on the fix). Per the repo's one-PR-per-issue policy, this PR (opened first) is the one that should be kept, and the owner is deciding on that.

To make this PR merge-ready under the auto-merge gate (§7.4 gate 2 requires a regression test), could you add the test file from #6688 to this branch? I can post the full file content here if that is easier. Your benchmark script (scripts/bench-bounded-boot-index.mjs) is a good complement and stays either way.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Verified the parallel PR before weighing in (didn't just take the comment at face value).

What I confirmed:

Clarification on my earlier "代码可合": that was for a human merge (provably behavior-preserving; the 4 focused suites pin it). For auto-merge, §7.4 gate 2 (a regression test that reproduces the bug and now passes) is a hard requirement — #6663 as-is fails only that gate; every other gate (single-module, clean diff, no format/wire/kernel change, refs #6662, green-on-rebase) passes. Adding #6688's test closes the gap.

Recommendation — keep #6663, port #6688's test in, close #6688:

Who does what:

八问:

  1. 是什么问题: 同一修复([Performance] Bounded boot repeatedly copies the growing history-file index before listen #6662 O(N²)→O(N))有两个并行 PR——[PR] perf(storage): build bounded boot file indexes incrementally #6663(Afloat16,先开,含 benchmark)与 [PR] fix: build bounded-boot file index in O(N) instead of O(N^2) #6688(你,后开,含回归测试),需按 one-PR-per-issue 收敛成一个。
  2. 什么原因导致的: 并行开发;先开的 [PR] perf(storage): build bounded boot file indexes incrementally #6663 缺自动合并门禁要求的回归测试,后开的 [PR] fix: build bounded-boot file index in O(N) instead of O(N^2) #6688 补上了。
  3. 影响范围与严重程度: 纯流程/协调问题,非代码缺陷;不处理则两 PR 并存,且 [PR] perf(storage): build bounded boot file indexes incrementally #6663 无法走自动合并(缺 gate 2)。
  4. 怎么修复的: 保留 [PR] perf(storage): build bounded boot file indexes incrementally #6663,移植 [PR] fix: build bounded-boot file index in O(N) instead of O(N^2) #6688 的回归测试,关闭 [PR] fix: build bounded-boot file index in O(N) instead of O(N^2) #6688;benchmark 脚本保留。
  5. 是否有回归问题: 不涉及(本轮无代码改动;两 PR 的修复逐字节相同,已本地验证红/绿)。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及。
  8. 是否引入了新的配置: 无(回归测试是 CI 门禁,非配置面)。

判定: 建议保留 #6663 + 移植 #6688 回归测试(已验证红/绿)+ rebase,关 #6688;需 @Afloat16 补测试文件(我无法代推其 fork),或你拍板走 fallback(保留 #6688、我移植 benchmark 进去)。等你定。

@ranxianglei
ranxianglei merged commit 76640c7 into ranxianglei:master Nov 14, 2026
86 of 126 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] Bounded boot repeatedly copies the growing history-file index before listen

2 participants