Skip to content

fix: storage refactor pre-work — atomic content-store write, symmetric fork retention, prefix-scoped parentRevision (#2675) - #2681

Closed
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-10_storage-hotfixes
Closed

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-10_storage-hotfixes

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #2675

Three independent pre-storage-refactor hotfixes split out of #2671 (full evidence there). One commit per fix; every intermediate commit verified green in a throwaway worktree (bisect-safe). Rebased onto current master (v0.1.191 line); only conflict was the diagnostics config block shared with #2412's dump-GC knobs — resolved as union.

1. Atomic content-store companion write

saveContentStore wrote <...>.content-store.json with a bare writeFileSync while every session record went through the kernel StateStore's temp+rename discipline. A crash mid-write truncated the target and the loader silently degraded to a fresh store (malformed => createContentStore()), losing all retrievable CCR originals.
Now written via atomicWriteFileSync (src/persist.ts): fresh same-dir temp per attempt, rename over the target, transient-fs retry with backoff mirroring the kernel semantics. Temp names match STALE_ENC_TEMP_RE so boot sweeps orphans, and deliberately do not end in .json so no record walker ever sees them. Payload-first ordering and the dirty-flag restore on failure are unchanged.
New test file tests/persist-content-store-atomic.test.ts: plants the orphaned temp of a would-be next save (kill-between-writes) and boots a fresh store over the directory — orphan swept, companion still parses, restored session resolves the intact store.

2. Symmetric fork retention

A successful fork sets publicSnapshotRetained on the parent so its pluginSnapshot survives restart, but forkContentStore was embedded in the session record only when a publicForkReceipt existed (children). Trigger: resetSessionCompression deletes the companion while the retained flag + storedRefs survive — a restarted parent then failed closed forever with "CCR original index unavailable".
buildRecord now embeds the content store under the same condition as the snapshot. On load the embedded copy wins over the companion (the staleness profile children already accept).
Tests: the corruption matrix now corrupts the EMBEDDED record copy (companion-only loss is masked by design), plus a new regression asserting a retained parent embeds both refs and survives companion deletion across a cold restore.

3. Prefix-scoped parentRevision

forkSnapshot().parentRevision hashed the ENTIRE session including every CCR payload (O(store) sha256 per cache miss): any store append outside the snapshot changed the revision, invalidated clients' cached revisions, and stalled the session lock path on large stores.
The revision now derives from prefix-scoped inputs only: ordered message identities, the block set projected to snapshot-relevant fields, rule/nudge/counter state, and each snapshot ref's (ref, rawId, hash) binding. Payloads stay pinned by their content-addressed hash without re-hashing; excluded state surfaces carry documented rationale where they are provably harmless to the child clone. The pre-fix whole-store byte/orphan/alias scan moves behind an opt-in knob — the issue's requested debug/verify option.
Tests: revision stable across unrelated store growth; audit knob flips orphan/corruption detection on and off.

Config surface

New field: diagnostics.forkStoreAudit (boolean, default false; env BILI_FORK_STORE_AUDIT). The issue explicitly asked to keep the payload re-hash available as a debug/verify option; no existing knob expresses it (the three-level compress hierarchy is compression policy, not fork-path verification), so a single boolean under the existing diagnostics section is the minimal mapping. Documented in CONFIGURATION.md en+zh.

Behavior changes (flagged per policy)

  1. Companion writes are now atomic. Old: crash mid-write could truncate the target ⇒ silent fresh-store degrade (all originals lost). New: target holds old-or-new bytes only; an orphaned temp may remain and is boot-swept.
  2. Retained parents embed forkContentStore in the session record. Disk growth proportional to the live store for such parents (children already had this profile); on load the embedded copy wins over the companion.
  3. parentRevision values change for ALL sessions after upgrade. Clients see one refetch cycle (stale revision ⇒ 409 ⇒ refetch); wire shape unchanged (v1-compatible, 64-hex sha256). Invalidation semantics narrow: unrelated store growth no longer changes the revision.
  4. Fail-closed surface narrows. The whole-store byte/orphan/alias integrity scan is no longer run by default — it is opt-in via BILI_FORK_STORE_AUDIT / diagnostics.forkStoreAudit. Snapshot refs remain structurally validated by default (presence, rawId↔ref roundtrip, payload presence).

Verification

  • typecheck clean, build green, on the rebased head
  • tests/plugin-fork.test.ts 44/44; persist/store/fork/ccr/session batch 197/197 incl. adjacent dsh/pi/side-lane fork suites
  • intermediate commits verified green via throwaway worktrees (C2: 46/46)
  • full regression delegated to CI per local testing discipline

ework-agent added 3 commits October 10, 2026 21:49
saveContentStore wrote <...>.content-store.json with a direct writeFileSync while every session record went through the kernel StateStore's temp+rename discipline. A crash mid-write truncated the target and the loader silently degraded to a fresh store (malformed => createContentStore), losing all retrievable CCR originals.

Write the companion through the same discipline: fresh temp in the same directory per attempt, rename over the target, transient-fs retry with backoff. Temp names match STALE_ENC_TEMP_RE so boot sweeps orphans, and they deliberately do not end in .json so no record walker ever picks them up. Payload-first ordering and the dirty-flag restore on failure are unchanged.

Tests: tests/persist-content-store-atomic.test.ts drives a kill-between-writes by planting the orphaned temp of a would-be next save and booting a fresh store over the directory — orphan swept, companion still parses, restored session resolves the intact store.
On a successful fork the parent gets publicSnapshotRetained so its pluginSnapshot survives restart, but the content store was embedded in the session record only when a publicForkReceipt existed (children). A restarted parent kept raw refs whose originals had left the companion - resetSessionCompression deletes the companion while the retained flag and storedRefs survive - and forkSnapshot then failed closed forever with 'CCR original index unavailable'.

Align buildRecord's forkContentStore condition with the pluginSnapshot one: retained parents embed their live content store in the record. On load the embedded copy wins over the companion (buildSession), the same staleness profile children already accept.

Tests: corruption matrix now corrupts the embedded record copy (companion-only loss is masked by design); new regression asserts a retained parent embeds both refs and survives companion deletion across a cold restore.
forkSnapshot hashed the ENTIRE session including every CCR payload (O(store) sha256 per cache miss), so any store append outside the snapshot changed parentRevision, invalidated clients' cached revisions, and stalled the session lock path on large stores.

The revision now derives from prefix-scoped inputs only: ordered message identities, the block set projected to snapshot-relevant fields, rule/nudge/counter state, and each snapshot ref's (ref, rawId, hash) binding. Payloads stay pinned by their content-addressed hash without re-hashing; excluded state surfaces are documented where they are provably harmless to the child clone.

The pre-fix whole-store byte/orphan/alias scan moves behind an opt-in knob (env BILI_FORK_STORE_AUDIT > diagnostics.forkStoreAudit > false) as the issue's debug/verify option; snapshot refs remain structurally validated by default.

Config surface: new diagnostics.forkStoreAudit boolean (default false), documented in CONFIGURATION.md en+zh.

Behavior changes: parentRevision values change for all sessions (one client refetch cycle after upgrade; wire shape stays v1-compatible 64-hex sha256); invalidation no longer fires on unrelated store growth; whole-store audit is opt-in instead of default.
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-10_storage-hotfixes (e6a9f96)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2884-pr.2681.f1610c7

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2681.tgz
npm install -g package

Install instructions are refreshed on each push.

docs-sync/docs-coverage gates require every config key to be seeded in website/config-reference/*.yaml — the hand-written doc lines drifted from generator output. Seed diagnostics.forkStoreAudit/BILI_FORK_STORE_AUDIT and regenerate both docs from the seed (canonical wording replaces the hand lines; adjacent generated rows normalize whitespace).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

PR #2681 审查结论(storage refactor 前置三热修)

已按「审核先于实现」走完方向关 / 回归关 / 代码关 / 测试关,并在当前 PR head e6a9f96(v0.1.191 线)本地验证:typecheck 干净、定向测试 48/48 全绿(persist-content-store-atomic 4 + plugin-fork 44)。全程回归按本地测试纪律交给 CI。

过程说明(一处我最初误判、已被第 4 个 commit 解决):我在旧 head(3-commit 版 417e82ea)上发现 CONFIGURATION.md 手写行与生成器输出漂移。当前 head 的第 4 个 commit(作者 ework-agent)正是为此——把 forkStoreAudit 补进 website/config-reference/diagnostics.yaml 种子并重新生成两份文档,这是 docs-sync 门禁要求的正确路径,commit message 也透明披露「相邻生成行归一化空格」。且该前导空格是生成器既有格式(master 上已有 9 行同格式),非本 PR 引入的脏改动。故无需我再推任何修复。

分支状态:当前 head 落后 origin/master(fd9aea232,#2541 preflight 断连)3 个 commit。已核对:#2541 只动 preflight.ts/server.ts/server/relay.ts,与本 PR 的 persist/plugin/config/knobs/tests/docs/yaml 零文件重叠、无共享机制 → 合并前更新到最新 master 即可,预期零冲突。

结构化八问

1. 是什么问题 —— 存储层三个独立缺陷(源自 #2671 深度评审,拆成 #2675 独立执行):(a) CCR 内容库伴生文件用裸 writeFileSync 落盘,写一半崩溃会截断目标,loader 静默降级成空库,整会话可检索原文全部丢失;(b) 成功 fork 后父会话置 publicSnapshotRetained 保留原始快照,但 forkContentStore 只在子会话(有 publicForkReceipt)时才内嵌进记录——重启后的父会话带着悬空 refs 却拿不到原文,forkSnapshot 永久 fail-closed("CCR original index unavailable"),无恢复路径;(c) parentRevision 对整个 CCR store 做 O(store) sha256,任何快照外的 store 追加都改变 revision,击穿客户端缓存 + 大 store 卡住会话锁。

2. 什么原因导致的 —— (a) 持久化不对称:所有会话记录走内核 StateStore 的 temp+rename,唯独伴生文件走裸写;(b) buildRecord 里快照保留条件(publicSnapshotRetained)与内容库内嵌条件(publicForkReceipt)分叉,对「保留父」这一态两者不一致;(c) forkSnapshot 把 {messages, state, blockContents, contentStore} 整体哈希,使 revision 耦合到 store 体量而非真正影响 fork 的前缀。三者均为根因(不是表面现象)。

3. 影响范围与严重程度 —— (a) 数据丢失类:每次 scheduleSave/flush 的伴生写入窗口内崩溃都可能触发,严重度最高,但需恰好在写入瞬间崩溃;现已消除。(b) 中危、特定触发:凡发生过成功 fork 的父会话,若其伴生文件丢失(如 resetSessionCompression 删除),重启后永久不可再 fork。(c) 性能/陈旧类:不破坏正确性,但随 store 增大放大——无谓 409 + 重试环 + 锁停顿,劣化扩展性。

4. 怎么修复的(机制层,非逐行)—— (a) 伴生文件改走新增的 atomicWriteFileSync:每次尝试在同目录新建 temp、rename 覆盖目标、瞬态 fs 错误退避重试(对齐内核语义);temp 名匹配 STALE_ENC_TEMP_RE(boot 清扫孤儿)且刻意不以 .json 结尾(记录/GC/web walker 只认 *.json,永不窥见活动 temp);payload-first 顺序与失败恢复 dirty-flag 不变。(b) 对齐两个 buildRecord 条件——保留父按与快照相同的规则内嵌内容库;加载时内嵌副本优先于伴生(本就是子会话行为,现对父同样成立)。(c) parentRevision 改由前缀投影派生:有序消息身份(经 forkMessageIdentityHash)、投影到 fork 相关字段的块集、rule/nudge/计数状态、每个快照 ref 的 (ref,rawId,hash) 绑定;payload 靠内容寻址哈希钉住不再重哈希;旧的整库 byte/orphan/alias 扫描移到一个可选诊断开关后(即 issue 要求的 debug/verify 项)。

5. 是否有回归问题 —— 无意回归未发现。逐点核过既有行为:默认路径对快照 refs 的结构性校验(存在性 / rawId↔ref 回环 / payload 存在)保持不变——fail-closed 对真正相关的 refs 完好;收窄掉的只是冗余的整库 byte/orphan 扫描(内容寻址已钉住快照 ref 字节 + 修复(a)消除了撕裂写这一主向量,故安全)。blockContents 能排除出投影,依赖「每 blockId 只 set 一次、失活才 delete」不变量——已核实源码只有创建处 .set 与回收处 .delete,无原地改写。排除 runtime metadata(lastModel/effectiveConfig 等 7 键)是正确的:revision 追踪的是「继承到的压缩上下文」,宿主自管运行期旋钮,继承的是当前值而非陈旧 T0 值反而更合理。唯一接受的残留:某快照 ref 的 payload 若发生位腐(哈希键存在但字节不符)会通过默认路径(仅可选审计能抓)——原子写后概率极低,已在行为变更 #4 披露。

6. 是否破坏缓存 —— 升级后所有会话的 parentRevision 值变一次 → 客户端经历一轮 stale→409→refetch(已披露);wire 形状不变(仍 v1、64-hex sha256),失效面是收窄(无谓 refetch 减少),对前缀缓存净利好。持久化格式未变(forkContentStore 字段本就存在、可选),旧记录照常加载(无内嵌→走伴生路径),无需版本号变更。

7. 是否改了设计方向 —— 无架构转向。这正是 #2675 明确划定的「统一存储方案(#2671 结构性方向,本次不动)之前」的三个前置热修;可选审计开关让旧校验能力保留而不改默认设计。

8. 是否引入了新的配置 —— 是:diagnostics.forkStoreAudit(boolean,默认 false;env BILI_FORK_STORE_AUDIT,每次请求实时读取)。en+zh 均已文档化,并已补进 website/config-reference/diagnostics.yaml 种子过 docs-sync 门禁。注意这是默认行为的收窄(#2675 之前整库扫描默认开、现为默认关的可选项),已在 PR「Behavior changes」第 4 条披露;无需迁移,想要旧校验的用户显式打开即可。

判定

可合(pending:① 分支更新到最新 master fd9aea232,零冲突预期;② CI 全量回归绿)。三个修法均打在根因(疏非堵),测试带牙且精确编码了新行为——含把 byte/orphan 用例从常开矩阵正确地迁到可选审计测试、corruption matrix 改为破坏内嵌副本(因内嵌优先于伴生)。配置面合规且双语齐全。本 PR 跨 persist/plugin/config/knobs 四模块并触及 fork/持久化语义,按 §7.4 不属于可自动合并范畴,交由人工合并即可。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Storage refactor pre-work: three hotfixes (atomic content-store write, symmetric fork retention, prefix-scoped parentRevision)

1 participant