Skip to content

[PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) - #4177

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-21_storage-durability
Oct 22, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-21_storage-durability

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Fixes #4175 — the unified store is the single source of truth but shipped none of the reliability trio: no corruption self-check, no online backup, no second-process detection (#3959/#3923 were exactly this class, with "delete index.db" as the only recovery).

1. Integrity self-check at open (PRAGMA quick_check)

  • Runs in UnifiedStore.open before migrateSchema — a corrupt file is never migrated against.
  • Failure throws a typed StorageIntegrityError; the shadow layer catches it, quarantines the file set (index.db/-wal/-shm → .corrupt-<ts>, same suffix discipline as session envelopes), and reopens fresh — JSON envelopes remain the recovery source (full mode re-imports organically).
  • Escape hatch BILI_STORAGE_QUICK_CHECK=0 for boot-time-sensitive huge deployments.
  • Outcome + duration recorded in runtime-status (integrity on the wire).

2. Online backup (VACUUM INTO)

  • UnifiedStore.backupInto(dst) — SQLite's native consistent-snapshot backup, safe while WAL writers are active; refuses an existing target (a backup path is produced exactly once).
  • Exposed: SessionStore.unifiedBackup(dst?) → POST /__bili/storage/backup {dst?} (default sessionsDir/backups/index-<timestamp>.db); backup recorded on the wire.
  • Automatic scheduling deliberately out of scope — manual capability first, policy later.

3. Second-process-on-the-same-db detection

Verification

  • Stake-first tests/storage-durability.test.ts 5/5: corrupt db → typed error; shadow open → quarantine + fresh reopen + post-quarantine ingest works; backup opens as a store with identical sessions + refuses overwrite; live foreign pid (real child process) reported / stale takeover / close removes lock; wire surfaces integrity/secondProcess/backup.
  • Families green: storage-unified 14, archive-gc 5, full-flip 8, golden-storage-render 5, acp-status 7, engine-bootstrap 44, web-sessions 26, reap-journal 2, maintenance-cli 7.
  • Gates clean: unused-exports, lifecycle-ledger (new BILI_STORAGE_QUICK_CHECK registered), gen-config-docs coverage (env-only seed + regenerated docs), tsc 0.
  • web banner added (page/client/i18n, zh+en).

kernel-change: none

@ranxianglei ranxianglei changed the title storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) [PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-21_storage-durability (1da8eca)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5126-pr.4177.4c06d38

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4177.tgz
npm install -g package

Install instructions are refreshed on each push.

ranxianglei pushed a commit that referenced this pull request Oct 21, 2026
The new durability suite spawns short-lived child processes (the lockfile
test drives a real sibling pid), which trips the SPAWN_RE tripwire in
scripts/run-ci-tests.mjs: every process-spawning suite must be listed in
LIVE_FILES or all test shards die at start with exit 1 (observed on head
7d6a6e4, #4177). Concurrency reviewed per #1248: at most two ~1s-lived
'node -e setTimeout' children per test, stdio ignore, mkdtemp isolation,
no ports — safe under --test-concurrency 4.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复,但强相关。#4175 本身(可靠性三件套)是独立立项,无重复项;实现面与在途 PR #3965(corrupt index.db 自动隔离,Closes #3959)重叠——两者都往 src/storage/shadow.ts 的同一个 try 块里加隔离逻辑,后合入者会冲突返工。但覆盖面互补:#4177 = 开库前主动 quick_check(open 成功但数据已坏);#3965 = 对 corruption 类 open 错误的被动兜底(含垃圾头/NOTADB)。处置建议见下文「协调请求」。字段案例 #3959/#3923 是用户报告,非重复。

标签:pr + scope/storage + p2(已打;理由:可靠性能力 PR 非缺陷报告不打 bug,核心存储域,严重程度随源 issue #4175)。

总体判断

方向对、实现扎实、测试有牙(stake-first、gate 双向钉住 #3617、lockfile 用真实子进程、NOTADB 降级路径按 #2771/#2781 原样钉死)。代码与回归我逐项过完,没有发现阻断性问题。CI 六条 test lane 全红——不是这个 PR 的代码问题,根因已钉死并本地复现,见下。

CI 红的根因(中间态 master 产物,本地字节级复现)

  • .github/workflows/ci.yml 用裸 actions/checkout@v4(不带 ref)→ pull_request 事件检出的是 refs/pull/N/merge,即 PR head 合并当时的 master,不是 PR head 本身。
  • 失败的那轮(~10-21 16:24 UTC)恰好落在一个 ~80 分钟的窗口里:master 刚合入 [PR] Reserve output headroom on every wire (drop the Anthropic exemption) #4184(「所有 wire 预留输出空间,取消 Anthropic 豁免」,15:19Z),它的 revert([PR] Revert "#3900: reserve output headroom on every wire (drop the Anthropic exemption)" (#4184) — unblock v0.1.204 #4198)还没合(16:37Z)。
  • [PR] Reserve output headroom on every wire (drop the Anthropic exemption) #4184 让 anthropic wire 的有效窗口也减去预留输出;失败 fixture 全是 max_tokens: 1024 → 每条断言差值恰好 −1024(270976 !== 272000、198976 !== 200000……);golden-storage-render 也只有 metadata 段漂移(该段含 effectiveContextLimit),state/blockContents/stats 全匹配。一个根因解释全部六条红 lane;日志里零 EBUSY/durability 相关失败。
  • 本地复现链(npm ci + kernel/pisub 构建):PR head 单独跑绿 → PR+当前 master(revert 后)绿 → PR+fe5ba5912(revert 前,即 CI 实际检出的合并树)失败,错误签名与 CI 逐字一致(270976≠272000;golden digest a2a1053e4f5e2c7b/metadata 15f9400ac13336ed)。
  • 处置:把分支更新到当前 master(已含 revert)重跑 CI 即可转绿,无需为这次红改任何代码。ci.yml 里给 mysql-contract lane 补 npm run pretest 那笔是合理的 parity 修复,保留。

本地验证(定向跑,全量交给 CI)

在「PR+当前 master」合并树上:tests/storage-durability.test.ts 7/7;storage 家族(storage-unified / archive-gc / full-flip / acp-status / engine-bootstrap / web-sessions / reap-journal / maintenance-cli)113 过 / 0 挂;golden-storage-render 5/5。

协调请求:与 #3965 的顺序(owner 拍板)

冲突面:同一 try 块 + 重复的文件组改名逻辑,应收敛成一个公共 helper。两个选项:

无论选哪个,垃圾头文件目前在本 PR 里走 legacy-only 降级(有意为之、测试已钉),只有并入 #3965 的被动分支后才进隔离路径——这点要在最终形态里确认一次。

小发现(非阻断,可顺手修或后续跟)

  1. lockfile 单槽位:第二进程打开同库会用自己 pid 覆写锁文件;随后第一进程正常退出时(foreignLockHolder 为 null)删锁,把仍在写库的第二进程的归属记录抹掉 → 第三者探测不到。建议「只报不写」语义:检测到存活外来持有者就不写自己的锁。同进程多 store 也有同盲点(A.close() 删掉 B 的锁)。
  2. 锁文件 JSON 损坏 → parse 抛错落外层 catch,连自己的锁也不写 → 探测被禁用且每次 open 刷一条 warn,直到手工删文件。建议把解析失败当 stale 直接接管。
  3. pid 复用会误报(kill-0 分不清复用的 pid;boot time 记了但跨进程比不了)。warning-only 可接受,注明即可。
  4. wire secondProcess 置位后永不清除 → 外来进程退出后横幅留到重启才消失。
  5. 新成本披露:每次 index.db 打开跑 quick_check 全文扫描(仅启动时、store 单例)——启动延迟随库大小线性增长;逃生口 BILI_STORAGE_QUICK_CHECK=0 已提供且文档齐全,[storage] 0.1.202 existing install: 12.6 GiB of refcount=0 orphans (87% of content bytes), still growing on a long session — interim reclaim before the GC/CLI release? #3987 那种巨型库场景尤其需要,合理。
  6. VACUUM INTO 目标是 journal_mode=delete 的全 checkpoint 副本(作备份工件没问题);dst 用 '' 加倍转义(SQLite 下足够);admin 端点接受任意 dst 路径,与既有 restore 端点姿态一致(loopback 管理面)。

结构化八问

  1. 是什么问题:统一存储(单一事实源,full 默认以来)没带可靠性三件套——损坏自检、在线备份、双进程写同库检测。字段案例 SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959/The same sessionId is served by two bili processes simultaneously: interleaved logs from global + pi-native dual instanc | 同一 sessionId 被两个 bili 进程同时服务:全局 + pi-native 双实例日志交织、重复 cold-load(split-brain) #3923 当时只能靠手工删 index.db 恢复。
  2. 什么原因导致的:不是 bug,是能力缺失;底层风险 = SQLite WAL 静默损坏 + 没有自动恢复路径,也没有第二个写者的可见性。
  3. 影响范围与严重程度:p2——今天能用(JSON 信封仍是恢复源、降级路径存在),但真发生损坏时要人工删文件;波及所有统一存储用户(默认开启)。
  4. 怎么修复的:三个独立机制——① 主动:迁移前 quick_check,类型化错误 → 影子层隔离文件组(.corrupt-<ts> 后缀纪律、取证保留)→ 重建新库(full 模式有机重导入信封);② VACUUM INTO 在线快照(手动端点,自动调度明确推迟);③ lockfile pid 探测,只报不拦(守住 Multi-instance posture is undocumented, and "/__bili/replicas" means session checkpoints — not pod replicas #3633 一目录一实例姿态——硬拦会把第二条 lane 砖掉)。
  5. 是否有回归问题:审过的既有行为——storage 家族 8 个套件 113/113 绿(rebased 树);golden-storage-render 5/5 绿(CI 那次红是中间态 master 产物,见上);NOTADB→legacy-only 降级路径不变且有测试钉住;非 corruption 类 open 失败(CANTOPEN/BUSY/权限)仍走原降级路径,diff 未触碰;wire 新增 integrity/secondProcess/backup 三个字段,客户端 null-safe 处理;新 env 已注册 lifecycle ledger + 文档(gates 绿)。未发现语义漂移。一处行为变更需点名:每次 index.db 打开新增 quick_check(启动成本 ∝ 库大小)——有意、已披露、带逃生口。
  6. 是否破坏缓存:不涉及——纯存储层,前缀缓存/wire/压缩机制均未触碰。
  7. 是否改了设计方向:否——在既有 UnifiedStore/UnifiedShadow/runtime-status 分层内扩展;admin 端点沿用 /__bili/storage/* 模式;web 横幅沿用既有 banner 模式。
  8. 是否引入新配置:BILI_STORAGE_QUICK_CHECK(env-only,默认 on,=0 跳过自检),已注册 env-only.yaml seed + CONFIGURATION.md 中英 + LIFECYCLE.md;另新增 POST /__bili/storage/backup {dst?} 管理端点(无配置键,缺省落 sessionsDir/backups/index-<时间戳>.db)。

判定

Hold,等两件事:(A) 分支更新到当前 master 重跑 CI 转绿(本次红与 PR 代码无关,rebase 即清);(B) owner 拍板与 #3965 的协调顺序(推荐 A:先合 #3965,本 PR rebase 复用其隔离 helper)。小发现 1–4 非阻断,可在 rebase 时顺手修或另行跟踪。

…econd-process lockfile (#4175)

The unified store is the single source of truth but shipped none of the
reliability trio. quick_check runs BEFORE migrations (a corrupt file is
never migrated against); on failure the shadow quarantines the file set
(.corrupt-<ts>) and reopens fresh. backupInto produces a consistent
online snapshot (VACUUM INTO — safe under active WAL writers) exposed
via unifiedBackup + POST /__bili/storage/backup. A <db>.lock lockfile
surfaces a live foreign pid (error log + wire + web banner, never a hard
block); stale locks are taken over; close() releases the lock.
@ranxianglei
ranxianglei force-pushed the 2026-10-21_storage-durability branch from 08b9795 to 1da8eca Compare October 22, 2026 15:09
@ranxianglei
ranxianglei merged commit aa198fe into master Oct 22, 2026
38 of 45 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.738-master.aa198fe (rolling master tag).

npm install -g billion-context@0.0.738-master.aa198fe  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 22, 2026
CI red found while verifying this PR: master merge aa198fe (#4177) added
tests/storage-durability.test.ts without registering it in the process-spawning
LIVE_FILES table, so the SPAWN_RE tripwire in scripts/run-ci-tests.mjs fails
every test shard on any PR whose checkout includes current master (and master
itself). Append at the end of the array: slot = index drives each file's zone
port band, so mid-list insertion would renumber every later band.
ranxianglei pushed a commit that referenced this pull request Oct 22, 2026
#4177 added a process-spawning test without registering it, so the run-ci-tests.mjs tripwire dies every shard of every PR whose merge tree includes it (red since aa198fe). Concurrency reviewed: unique mkdtempSync dirs, stdio:ignore child, no port binding. Found while reviewing #4325.
ranxianglei added a commit that referenced this pull request Oct 22, 2026
#4177 landed the storage durability trio but its new suite spawns
child processes without registering in scripts/live-files.mjs, so the
#1248 SPAWN_RE tripwire fails every CI run — master aa198fe is red
and every branch rebased onto it inherits the failure.

Concurrency review: the suite only spawns `node -e setTimeout` probes
(no port binding) and isolates data in mkdtempSync dirs; slot 31 /
zone band 19811 is free. Fixes #4333.

(cherry picked from commit 9759e3e)
ranxianglei added a commit that referenced this pull request Oct 22, 2026
Since #4177 the storage contract suites transitively import acp-kernel
at runtime (backend -> store -> runtime-status -> knobs -> config), and
tsx resolves it to kernel/dist — which this job never built. Mirror the
prerequisite the main test matrix (run-ci-tests pretest parity) and
ci-e2e-dsh.yml already carry. Fixes #4336.
ranxianglei added a commit that referenced this pull request Oct 22, 2026
Since #4177 the storage contract suites transitively import acp-kernel
at runtime (backend -> store -> runtime-status -> knobs -> config), and
tsx resolves it to kernel/dist — which this job never built. Mirror the
prerequisite the main test matrix (run-ci-tests pretest parity) and
ci-e2e-dsh.yml already carry. Fixes #4336.

(cherry picked from commit 0f21448)
ranxianglei pushed a commit that referenced this pull request Oct 23, 2026
The lane installs with --ignore-scripts and runs the storage-backend suites
directly, never building the in-repo components. Since the in-repo kernel
(#2092), node_modules/acp-kernel/dist only exists after
'npm --prefix kernel run build'. #4177 (aa198fe) added a store.ts ->
runtime-status.ts import that pulls the acp-kernel runtime chain into these
tests' import graph via src/config.ts, so the lane has failed with
ERR_MODULE_NOT_FOUND since its first run after that merge (master CI run
39803248052 at 2026-10-22T15:16Z; last green master run 399c8bc at
12:55Z). Reproduced locally: identical command passes on c1d7eab and fails
identically on aa198fe. The main test job never hit this because
scripts/run-ci-tests.mjs runs the pretest build first; mirror that build
step here. Pre-existing master breakage, unrelated to this PR's diff.
yanhuojiyusheng pushed a commit to yanhuojiyusheng/billion-context that referenced this pull request Oct 23, 2026
)

Since ranxianglei#4177 the storage suites transitively import acp-kernel at runtime
(store.ts -> runtime-status -> knobs -> config); tsx resolves it to the
built dist, so the mysql-contract job dies with ERR_MODULE_NOT_FOUND on a
fresh runner. Same prerequisite the main test matrix already has via
run-ci-tests' pretest step. Cherry-picked from ranxianglei#4334.
ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
…tion alarm (Follows #4614)

e83fc64 (#4685) rewrote this alarm line for the #4392 digest-shape-first
wording and dropped the #4614 clause in the same edit, so every test lane
failed tests/issue4323-drift-escalation.test.ts:193 ('alarm also names the
single-writer repeating-rewrite producer') from the merge onward. Restore
both producers in one line — digest-shape mismatch first (#4392), then the
writer side: a second bili instance on one sessions dir (#394/#4177) or ONE
host re-writing history slots below the ingest watermark (#4614).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant