Skip to content

[PR] spawn path: log the blocking integrity-scan phase before it starts; correct stale BILI_STORAGE_QUICK_CHECK docs - #4946

Closed
ranxianglei wants to merge 8 commits into
masterfrom
2026-10-27_spawn-budget-observability
Closed

ranxianglei wants to merge 8 commits into
masterfrom
2026-10-27_spawn-budget-observability

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Part of #4940 — proposals ② (phase observability) + ④ (documented cost). Proposal ① (configurable spawn budget) awaits the owner's A/B/C decision in the issue thread; if approved it extends this same branch (one issue → one PR).

Changes

1. Announce the synchronous integrity scan before startup blocks on it (src/storage/store.ts + regression test)

  • In UnifiedStore.open()'s sync branch (stores ≤512 MiB, verdict-cache miss), one info log line now fires BEFORE PRAGMA quick_check: [storage] integrity quick_check started for <path> (<MiB>, <N pages>) — synchronous pre-migration scan; startup stays blocked until it finishes (#4940).
  • Kills the zero-byte-child window from [PR] fix: refuse oversized degraded direct-sends using the lane known window (#4934) #4940: the launcher's failure report (..., N bytes — EMPTY) can no longer be misread as a hung child — the scan phase is visible, with the exact size/page count needed to estimate its duration.
  • Pure log addition: zero behavior change, no wire impact (both compression modes unaffected).

2. Correct stale BILI_STORAGE_QUICK_CHECK docs (website/config-reference/env-only.yaml → regenerated CONFIGURATION.md / CONFIGURATION.zh-CN.md, plus LIFECYCLE.md)

3. Pin the REAL 512 MiB sync/async dispatch boundary (tests/storage-integrity-cache.test.ts)

  • Regression from the [PR] fix: refuse oversized degraded direct-sends using the lane known window (#4934) #4940 thread (Windows / pi / ~893 MiB store report): pads a small valid store past 512 MiB with a sparse tail (statSync sees >512 MiB — the dispatch key in integrityModeFor — while the SQLite page count stays tiny), then asserts open() takes the BACKGROUND path ("background quick_check started … over the 512 MiB sync threshold"), runs NO synchronous rescan, and lands a cached ok verdict. Exercises the real threshold constant instead of only the test-override seam used by the existing large-store test.

Behavior-change disclosure

  • One NEW info log line at startup on the sync integrity-check path (≤512 MiB store, cache miss). No defaults/thresholds/timing changed; no config surface added; no wire/message-shape changes.

Verification

  • npm run typecheck clean on the rebased head (branch re-synced with master merges through c499b14; the merged store.ts changes are in fork-lineage paths, untouched by this PR).
  • Targeted local runs: tests/storage-integrity-cache.test.ts 8/8 (incl. the both-directions announce test and the real-boundary dispatch test), tests/storage-unified.test.ts + tests/store.test.ts 43 pass / 1 gated skip / 0 fail. Full suite = remote CI.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-27_spawn-budget-observability (81fc5f2)

Latest build — install this exact version

npm install -g billion-context@0.0.7121-pr.4946.28ad1a2

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
81fc5f2 billion-context@0.0.7121-pr.4946.28ad1a2 build
f94d2b5 billion-context@0.0.7084-pr.4946.ff7f1a4 build
a59b03a billion-context@0.0.6864-pr.4946.0dec864 build
8f332ee billion-context@0.0.6525-pr.4946.9516091 build
112104a billion-context@0.0.6459-pr.4946.2f80984 build
96a6672 billion-context@0.0.6455-pr.4946.454f152 build
7d00160 billion-context@0.0.6435-pr.4946.5db9f73 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4946.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title spawn path: log the blocking integrity-scan phase before it starts; correct stale BILI_STORAGE_QUICK_CHECK docs [PR] spawn path: log the blocking integrity-scan phase before it starts; correct stale BILI_STORAGE_QUICK_CHECK docs Oct 27, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 是 #4940 提案 ②(扫描阶段可观测)+ ④(成本文档勘误)的交付件(见 #4940 floor 4)。逐一核对在途 PR:#4852(已并入 master,#4609)修的是「>512 MiB 阻塞启动」根因本身,不同范围;其余在途 storage 类 PR(#4936/#4924/#4914/#4888/#4800/#4685/#4628)均不触碰 integrity-scan 日志或 BILI_STORAGE_QUICK_CHECK 文档条目。提案 ①(可配置 spawn 预算)正确地未包含在内——等 owner 对 A/B/C 拍板后扩展同一分支。

标签: scope/storage / p3 —— 改动落在统一存储开库自检路径(一行 info 日志)+ 配置文档勘误;本 PR 自身是观测性/文档级小改(父 issue #4940 保持 p2)。

这是 agent 交付的 [PR],我不重做实现,走审核流程:拉分支核对代码、验证 CI、按八问给结论。当前状态:build 作业已产出测试构建(0.0.6435-pr.4946.5db9f73),test 车道仍在跑;我本地做定向验证(typecheck + integrity/store 定向套件)后补结构化八问结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR #4946 审核结论 — 可合并(等你 merge)

独立核查(全部在 rebase head 7d00160 上重做,不引用 issue 侧结论):

  1. 代码核对:UnifiedStore.open() sync 分支(≤512 MiB + verdict-cache miss)在 PRAGMA quick_check 之前先跑两个只读 pragma(page_count/page_size,µs 级、无状态变更)拼出大小摘要,再打一条 info 日志;pragma 失败落入既有 try/catch → [Bug][storage] corrupt index.db bricks SessionStore construction in shadow/full mode (violates the sidecar never-breaks-legacy policy) #2771/windows-22-1 lane randomly red: during test cleanup, unlink index.db hit an EBUSY external lock, couldn't survive tmp-rm | windows-22-1 车道随机红:测试清理期 unlink index.db 遇 EBUSY 外部锁,熬不过 tmp-rm 的 9.6 秒重试梯子(#2799/#2908 族复发) #4347 隔离路径,catch 链未动。
  2. 文档逐条对码:512 MiB 阈值(INTEGRITY_SYNC_THRESHOLD_BYTES)、>512 MiB 用第二条只读连接后台检查、index.db.chk 缓存键 = schema + index.db/-wal 的 size/mtime、=0 跳过检查与全部缓存 IO、60 s spawn 预算(20 s + 40 s grace)——与当前代码全部一致;旧文本确实描述的是 pre-[Bug] 0.1.205+: open-time quick_check is O(index.db) — 14 GB takes 260 s, so the 60 s spawn budget always kills the child and the host stays on direct | 0.1.205+ 开库 quick_check 使启动变成 O(库大小):14GB 库 260 秒 → 必超 60 秒 spawn 预算 → 整生命周期直连 #4609 行为("每次 open 都在迁移前同步 quick_check")。
  3. 再生成验证:node tools/gen-config-docs.mjs generate 输出与已提交的 CONFIGURATION.md / CONFIGURATION.zh-CN.md 字节一致(跑完 git 树干净),EN/ZH 键集相等——md 是生成器产物而非手改。
  4. 本地定向验证:typecheck 干净;storage-integrity-cache 7/7(含新双向测试 feat: bili CLI + debug in config + README rewrite #7「sync check announces its phase…opt-out stays silent」)、storage-unified + store 合计 43 pass / 1 gated skip / 0 fail;config-surface-guard OK;lifecycle-ledger-guard OK。
  5. 远程 CI(push head):全绿 — CI、PR Build Artifact、e2e-native-dsh/pi/opencode/codex-persona/fake-codex/pi-codex-ws/local-registry、Windows codex overlay 均 success。

八问:

  1. 是什么问题:两件独立小事——①≤512 MiB store 启动时的同步完整性扫描会阻塞几十秒到几分钟却无任何日志,零字节子进程窗口([PR] fix: refuse oversized degraded direct-sends using the lane known window (#4934) #4940)里无法把「正在扫」和「子进程挂死」区分开;②BILI_STORAGE_QUICK_CHECK 文档停留在 pre-[Bug] 0.1.205+: open-time quick_check is O(index.db) — 14 GB takes 260 s, so the 60 s spawn budget always kills the child and the host stays on direct | 0.1.205+ 开库 quick_check 使启动变成 O(库大小):14GB 库 260 秒 → 必超 60 秒 spawn 预算 → 整生命周期直连 #4609 描述,误导用户判断启动会不会被阻塞。
  2. 什么原因导致的:[Bug] 0.1.205+: open-time quick_check is O(index.db) — 14 GB takes 260 s, so the 60 s spawn budget always kills the child and the host stays on direct | 0.1.205+ 开库 quick_check 使启动变成 O(库大小):14GB 库 260 秒 → 必超 60 秒 spawn 预算 → 整生命周期直连 #4609 把大 store 检查移到后台后,小 store 的同步路径没有留下阶段可见性(扫描本身从不宣告自己开始);文档没随 [Bug] 0.1.205+: open-time quick_check is O(index.db) — 14 GB takes 260 s, so the 60 s spawn budget always kills the child and the host stays on direct | 0.1.205+ 开库 quick_check 使启动变成 O(库大小):14GB 库 260 秒 → 必超 60 秒 spawn 预算 → 整生命周期直连 #4609 更新,残留旧行为描述。
  3. 影响范围与严重程度:观测缺口影响所有 ≤512 MiB store 的首次 open / 文件集变化后的 open(阻塞窗口内现在有一行带大小+页数的日志,可据此估算时长);文档错误影响阅读该 env 语义的用户。均为观测/文档缺陷,无功能损伤 — p3。
  4. 怎么修复的:在决策点源头宣告——sync 模式判定之后、扫描开始之前打一条 info 日志(路径 + MiB + 页数),而不是在症状处补日志;文档在 yaml seed 上修正、由生成器重新产出 md,LIFECYCLE.md 同步行更新。修法符合「疏不堵」:日志打在产生阻塞的那条路径上。
  5. 是否有回归问题:逐项审过——五模式分发(cached-ok/cached-fail/sync/skip/async)未动;Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 损坏隔离链(catch → recordIntegrityOutcome(false) → close → rethrow)未动;=0 opt-out 语义未动(既有测试 feat: persistent + namespaced sessions, multi-tenant session isolation, stream robustness #5 钉住,通过);稳态日志量可忽略(verdict-cache 命中不打该行,只在首开或文件集变化时出现);无 wire/消息形状变化(两种压缩模式都不受影响);kernel/、update.ts、持久化格式均未触碰。未发现回归。
  6. 是否破坏缓存:不涉及 — 未触碰任何缓存/重试/增量游标/去重策略;index.db.chk verdict 缓存逻辑本身零改动。
  7. 是否改了设计方向:不涉及 — 纯日志追加 + 文档勘误。提案 ①(可配置 spawn 预算)正确留在 [PR] fix: refuse oversized degraded direct-sends using the lane known window (#4934) #4940 线程等 owner 拍板,没有夹带进本 PR。
  8. 是否引入了新的配置:无。config-surface-guard 通过(174 个叶子字段,无新增)。

一个非阻断观察:新测试 #7 钉住了日志行存在性、格式(MiB+页数)与 opt-out 静默,但无法钉住「先于扫描开始」的顺序(open() 内没有其他日志锚点可比序);该顺序由源码语句顺序保证,肉眼可核,可接受。

判定:可合并。 满足 auto-merge 门禁各项(单模块 fix+docs、回归测试复现意图且通过、rebase head 绿、diff 干净每行相关、引用 #4940、未触碰 load-bearing infra);行为变更披露已在 PR 正文(新增一条 info 日志行,无默认值/阈值/时序/wire 变化)。合并权在你:#4946

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)
Synced origin/master into this branch (twice — master kept landing merges during the sync), zero conflicts, local typecheck clean. Head is now 112104a, no longer behind master; mergeable with no conflicts, checks re-running on the new head.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Merge-guard sync 完成:已将 origin/master 合入 PR 分支(期间 master 又落了新提交,共同步两次),零冲突,合并后本地 typecheck 干净。head 现为 112104a,GitHub API 确认 mergeable、不再 behind(mergeable_state 已从 behind 转为 blocked,CI 正在跑新 head)。结果评论已发到 GitHub PR:#4946 (comment)

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到的是我刚才发到 GitHub PR 的结果评论被镜像回来的回显(upstream-sync),没有新的用户输入。merge-guard 同步已全部完成并验证:分支已含最新 master(head 112104a)、零冲突、typecheck 干净、GitHub 侧 mergeable 且不再 behind。无需进一步操作,等 CI 跑完新 head 后即可由你合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Merge-guard re-sync: merged origin/master (~200 commits since last sync, incl. #5097/#5010) into this branch on top of the new test commit 8f332ee — zero conflicts, local typecheck clean. Head is now a59b03a, no longer behind master; mergeable with no conflicts, checks re-running on the new head.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 自动修复结论(attempt 2/2):继承性红,本分支零代码改动,已触发失败分片重跑,等结果。

验证过程(证据先行,不是照抄转发):

八问:

  1. 是什么问题:主 CI 三个 test 分片(ubu24-s3、ubu22-s3、win22-s4)红,挡住本 PR 转绿。
  2. 什么原因导致的:继承性失败——[PR] fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) #5144(10-29 落 master)在 tests/plugin-fork.test.ts 引入的用例存在单 cell flake,间歇性返回 409 而非 200;master 自身即同签名红,与本分支 diff 无关。
  3. 影响范围与严重程度:仅检查矩阵信号层面(挡合并按钮),用户侧零影响;与 [Bug] Drifted open-tag names (acpt/acct/accent…) and mid-text draft envelopes reach the client verbatim — two thinking-leak shapes #5175/[PR] fix(#5175): strip drifted open-tag names and mid-turn draft envelopes #5178/[PR] fix: bound ordinary session loads and narrow lazy field reads #4750/[PR] kernel: heal system positions on the final Anthropic wire array (#4570) #4715/[PR] fix: pi /tree recovery respects deliberately-off bili tools — pi-loadout selections survive navigation #4996/[PR] fix: stop the welcome 302 and favicon routes hijacking MITM-vhost requests (#5024) #5035 等同家族,已在 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 由哨兵统一跟踪。
  4. 怎么修复的:A/B 证据钉死继承判定后,本分支不加任何代码改动(政策禁止在自己分支重复修 master 级失败),直接重跑三个失败分片;根治归属 [PR] fix: recover non-retained parent fork snapshot from intact unified raw tail after cold restart (#5140) #5144 侧,收敛哨兵/ci-fixes。
  5. 是否有回归问题:无——本轮零 push、head 未变;本 PR 本身此前已按八问审过(floor 4,未发现回归)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:无。

另注:分支现落后 master 30 个提交——那是 merge-guard 车道的事(此前已派过),且同步 master 也去不掉这个 flake(master 本身就带着它),CI 车道不动它。

判定:等 attempt 2 重跑结果。转绿 → 待你合并(合并前 merge-guard 会再同步一次 master);若再现同一 #5140 签名 → 属 #5144 侧待处置的既有 flake(#4599 已立项跟踪),本自动修复车道无可修的决策点,按政策不为 CI 问题另开 issue。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Merge-guard re-sync: merged origin/master (~110 commits since last sync) into this branch — zero conflicts, local typecheck clean. Head is now f94d2b5, no longer behind master; mergeable with no conflicts, checks re-running on the new head.

@ranxianglei

Copy link
Copy Markdown
Owner Author

其中文档修正已随最终轮次的 CONFIGURATION/env seed 同步落入 master;日志提示部分被取代。冗余关闭。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant