Repository navigation
fix: storage refactor pre-work — atomic content-store write, symmetric fork retention, prefix-scoped parentRevision (#2675) - #2681
ranxianglei wants to merge 4 commits into
Conversation
saveContentStore wrote <...>.content-store.json with a direct writeFileSync while every session record went through the kernel StateStore's temp+rename discipline. A crash mid-write truncated the target and the loader silently degraded to a fresh store (malformed => createContentStore), losing all retrievable CCR originals. Write the companion through the same discipline: fresh temp in the same directory per attempt, rename over the target, transient-fs retry with backoff. Temp names match STALE_ENC_TEMP_RE so boot sweeps orphans, and they deliberately do not end in .json so no record walker ever picks them up. Payload-first ordering and the dirty-flag restore on failure are unchanged. Tests: tests/persist-content-store-atomic.test.ts drives a kill-between-writes by planting the orphaned temp of a would-be next save and booting a fresh store over the directory — orphan swept, companion still parses, restored session resolves the intact store.
On a successful fork the parent gets publicSnapshotRetained so its pluginSnapshot survives restart, but the content store was embedded in the session record only when a publicForkReceipt existed (children). A restarted parent kept raw refs whose originals had left the companion - resetSessionCompression deletes the companion while the retained flag and storedRefs survive - and forkSnapshot then failed closed forever with 'CCR original index unavailable'. Align buildRecord's forkContentStore condition with the pluginSnapshot one: retained parents embed their live content store in the record. On load the embedded copy wins over the companion (buildSession), the same staleness profile children already accept. Tests: corruption matrix now corrupts the embedded record copy (companion-only loss is masked by design); new regression asserts a retained parent embeds both refs and survives companion deletion across a cold restore.
forkSnapshot hashed the ENTIRE session including every CCR payload (O(store) sha256 per cache miss), so any store append outside the snapshot changed parentRevision, invalidated clients' cached revisions, and stalled the session lock path on large stores. The revision now derives from prefix-scoped inputs only: ordered message identities, the block set projected to snapshot-relevant fields, rule/nudge/counter state, and each snapshot ref's (ref, rawId, hash) binding. Payloads stay pinned by their content-addressed hash without re-hashing; excluded state surfaces are documented where they are provably harmless to the child clone. The pre-fix whole-store byte/orphan/alias scan moves behind an opt-in knob (env BILI_FORK_STORE_AUDIT > diagnostics.forkStoreAudit > false) as the issue's debug/verify option; snapshot refs remain structurally validated by default. Config surface: new diagnostics.forkStoreAudit boolean (default false), documented in CONFIGURATION.md en+zh. Behavior changes: parentRevision values change for all sessions (one client refetch cycle after upgrade; wire shape stays v1-compatible 64-hex sha256); invalidation no longer fires on unrelated store growth; whole-store audit is opt-in instead of default.
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.2884-pr.2681.f1610c7This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr2681.tgz
npm install -g packageInstall instructions are refreshed on each push. |
docs-sync/docs-coverage gates require every config key to be seeded in website/config-reference/*.yaml — the hand-written doc lines drifted from generator output. Seed diagnostics.forkStoreAudit/BILI_FORK_STORE_AUDIT and regenerate both docs from the seed (canonical wording replaces the hand lines; adjacent generated rows normalize whitespace).
PR #2681 审查结论(storage refactor 前置三热修)已按「审核先于实现」走完方向关 / 回归关 / 代码关 / 测试关,并在当前 PR head 过程说明(一处我最初误判、已被第 4 个 commit 解决):我在旧 head(3-commit 版 分支状态:当前 head 落后 结构化八问1. 是什么问题 —— 存储层三个独立缺陷(源自 #2671 深度评审,拆成 #2675 独立执行):(a) CCR 内容库伴生文件用裸 2. 什么原因导致的 —— (a) 持久化不对称:所有会话记录走内核 StateStore 的 temp+rename,唯独伴生文件走裸写;(b) 3. 影响范围与严重程度 —— (a) 数据丢失类:每次 scheduleSave/flush 的伴生写入窗口内崩溃都可能触发,严重度最高,但需恰好在写入瞬间崩溃;现已消除。(b) 中危、特定触发:凡发生过成功 fork 的父会话,若其伴生文件丢失(如 4. 怎么修复的(机制层,非逐行)—— (a) 伴生文件改走新增的 5. 是否有回归问题 —— 无意回归未发现。逐点核过既有行为:默认路径对快照 refs 的结构性校验(存在性 / rawId↔ref 回环 / payload 存在)保持不变——fail-closed 对真正相关的 refs 完好;收窄掉的只是冗余的整库 byte/orphan 扫描(内容寻址已钉住快照 ref 字节 + 修复(a)消除了撕裂写这一主向量,故安全)。 6. 是否破坏缓存 —— 升级后所有会话的 7. 是否改了设计方向 —— 无架构转向。这正是 #2675 明确划定的「统一存储方案(#2671 结构性方向,本次不动)之前」的三个前置热修;可选审计开关让旧校验能力保留而不改默认设计。 8. 是否引入了新的配置 —— 是: 判定可合(pending:① 分支更新到最新 master |
Closes #2675
Three independent pre-storage-refactor hotfixes split out of #2671 (full evidence there). One commit per fix; every intermediate commit verified green in a throwaway worktree (bisect-safe). Rebased onto current master (v0.1.191 line); only conflict was the
diagnosticsconfig block shared with #2412's dump-GC knobs — resolved as union.1. Atomic content-store companion write
saveContentStorewrote<...>.content-store.jsonwith a barewriteFileSyncwhile every session record went through the kernel StateStore's temp+rename discipline. A crash mid-write truncated the target and the loader silently degraded to a fresh store (malformed => createContentStore()), losing all retrievable CCR originals.Now written via
atomicWriteFileSync(src/persist.ts): fresh same-dir temp per attempt, rename over the target, transient-fs retry with backoff mirroring the kernel semantics. Temp names matchSTALE_ENC_TEMP_REso boot sweeps orphans, and deliberately do not end in.jsonso no record walker ever sees them. Payload-first ordering and the dirty-flag restore on failure are unchanged.New test file
tests/persist-content-store-atomic.test.ts: plants the orphaned temp of a would-be next save (kill-between-writes) and boots a fresh store over the directory — orphan swept, companion still parses, restored session resolves the intact store.2. Symmetric fork retention
A successful fork sets
publicSnapshotRetainedon the parent so itspluginSnapshotsurvives restart, butforkContentStorewas embedded in the session record only when apublicForkReceiptexisted (children). Trigger:resetSessionCompressiondeletes the companion while the retained flag + storedRefs survive — a restarted parent then failed closed forever with "CCR original index unavailable".buildRecordnow embeds the content store under the same condition as the snapshot. On load the embedded copy wins over the companion (the staleness profile children already accept).Tests: the corruption matrix now corrupts the EMBEDDED record copy (companion-only loss is masked by design), plus a new regression asserting a retained parent embeds both refs and survives companion deletion across a cold restore.
3. Prefix-scoped parentRevision
forkSnapshot().parentRevisionhashed the ENTIRE session including every CCR payload (O(store) sha256 per cache miss): any store append outside the snapshot changed the revision, invalidated clients' cached revisions, and stalled the session lock path on large stores.The revision now derives from prefix-scoped inputs only: ordered message identities, the block set projected to snapshot-relevant fields, rule/nudge/counter state, and each snapshot ref's
(ref, rawId, hash)binding. Payloads stay pinned by their content-addressed hash without re-hashing; excluded state surfaces carry documented rationale where they are provably harmless to the child clone. The pre-fix whole-store byte/orphan/alias scan moves behind an opt-in knob — the issue's requested debug/verify option.Tests: revision stable across unrelated store growth; audit knob flips orphan/corruption detection on and off.
Config surface
New field:
diagnostics.forkStoreAudit(boolean, defaultfalse; envBILI_FORK_STORE_AUDIT). The issue explicitly asked to keep the payload re-hash available as a debug/verify option; no existing knob expresses it (the three-levelcompresshierarchy is compression policy, not fork-path verification), so a single boolean under the existingdiagnosticssection is the minimal mapping. Documented in CONFIGURATION.md en+zh.Behavior changes (flagged per policy)
forkContentStorein the session record. Disk growth proportional to the live store for such parents (children already had this profile); on load the embedded copy wins over the companion.parentRevisionvalues change for ALL sessions after upgrade. Clients see one refetch cycle (stale revision ⇒ 409 ⇒ refetch); wire shape unchanged (v1-compatible, 64-hex sha256). Invalidation semantics narrow: unrelated store growth no longer changes the revision.BILI_FORK_STORE_AUDIT/diagnostics.forkStoreAudit. Snapshot refs remain structurally validated by default (presence, rawId↔ref roundtrip, payload presence).Verification