Skip to content

[PR] storage: jsonWriter=off also stops the CCR content-store sidecar (#2960) - #5988

Merged
ranxianglei merged 1 commit into
masterfrom
2026-11-09_jsonwriter-off-ccr-sidecar
Nov 9, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-11-09_jsonwriter-off-ccr-sidecar

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

storage.jsonWriter: "off" (the Phase 3 envelope switch, #2960) now also stops writing the per-session CCR content-store sidecar (*.content-store.json), not just the session envelope. Before this PR, off mode promised "the SQLite index is the only persistent copy" but still wrote the CCR sidecar file every save — dead weight, since in full mode that file is never read (reads go unified-first) and the same save already ingests the store into the index.

Changes

  • src/persist.ts saveContentStore: when the off switch is active, the sidecar write is skipped (after the emptied-store delete branch). The durable copy is the index's ccr mirror (sessions.ccr_json), ingested by the same save's builder.
  • The gate clears contentStoreDirty (found while testing): the off save drops the in-memory store right after ingesting it (dropResidentContentStore). If the dirty flag were left true, a later save — once the writer resumes because the sidecar self-disabled — would see "dirty + empty in-memory store" and take the emptied-store branch, which deletes the sidecar file. Clearing the flag removes that trap; after a writer-resume the kernel re-adopts its in-memory store on the next turn and the file converges from there.
  • Fork/restore materialization stays unconditional: writeContentStoreSidecar (the one-shot sidecar write when restoring a replica) is intentionally unchanged — the index does not carry forked CCR stores, so skipping it there would strand forked payloads.
  • Docs: config docblock, process.yaml (EN/ZH), regenerated docs/configuration.md — the old wording "the content-store sidecar keeps being written either way" is replaced.
  • Tests: new test in tests/storage-json-writer-off.test.ts with three legs — (1) under off the sidecar bytes are untouched (a differently-grafted store would have changed them), (2) a fresh handle resolves the originals from the index mirror, (3) control: with the switch on, the same kind of save does write the file.

Rollback / safety

  • Unset the knob → both the envelope and the sidecar resume writing; index-only sessions keep loading via the boot union (unchanged from Phase 3).
  • The off lane must remain the sole writer of its sessions directory (already documented); this PR does not change that contract.
  • Crash-window note: in off mode durability moves from "file written payload-first" to "index ingested in the same save cycle" — the same shift the envelope already made in Phase 3.

Verification

  • npm run typecheck ✓
  • tests/storage-json-writer-off.test.ts 4/4 ✓ (clean XDG)
  • Regression set 113/113 ✓: storage-review-fixes, plugin-fork, storage-corpus, export-jsonl, issue5805-jsonwriter-ownership, issue5046-second-writer-degrade, storage-full-flip, memory-contentstore-lazy, issue4816-integrity-throw-quarantine, storage-save-path
  • check:unused-exports ✓ · check:test-import-graph ✓ · gen-config-docs check-coverage ✓ · check-architecture-docs ✓

Phase 3 said the off switch covers the session envelope; the per-session
CCR content-store sidecar kept being written either way. In effective-full
mode that file is never read (contentStoreOf is unified-first) and the
same save already ingests the store into the index (sessions.ccr_json),
so the sidecar write was dead weight contradicting the switch's contract:
the SQLite index is the ONLY persistent copy.

- saveContentStore: writerOffActive() gate after the emptied-store rm
  branch — off skips the file write; the builder's ccr ingest (same save
  cycle) is the durable copy.
- The gate clears contentStoreDirty: the off save drops the resident
  store right after the ingest (dropResidentContentStore), and a stale
  dirty=true with contentStore=null would make a later writer-resume
  save hit the emptied-store branch and DELETE the sidecar file.
  Convergence after a writer-resume comes from the kernel re-adopting
  its in-memory store on the next turn.
- Fork/restore one-shot materialization (writeContentStoreSidecar) stays
  unconditional: the index does not carry forked CCR stores, so skipping
  there would strand forked payloads.
- Tests: new suite proves the sidecar bytes are untouched under off, the
  index mirror resolves the originals, and an on-control leg still
  writes the file. Docs: config docblock + process.yaml (EN/ZH) +
  regenerated configuration.md.
@ranxianglei ranxianglei changed the title storage: jsonWriter=off also stops the CCR content-store sidecar (#2960) [PR] storage: jsonWriter=off also stops the CCR content-store sidecar (#2960) Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-09_jsonwriter-off-ccr-sidecar (3554fe3)

Latest build — install this exact version

npm install -g billion-context@0.0.8713-pr.5988.068d45a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
3554fe3 billion-context@0.0.8713-pr.5988.068d45a build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5988.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Dup check: not a duplicate. Searched open+closed issues/PRs for jsonWriter / content-store sidecar / CCR mirror topics. This PR (#5988) is the only open item on "off switch also stops the sidecar". Lineage: #2960 (identity pipeline, closed) and its Phase 3 envelope-off #3020 (closed). Related but distinct, all closed: #5805/#5808 (off-lane CPU cost — where the resident-store-keep-alive guard came from), #4733 (restamp never re-keys the CCR store), #3010 (CCR originals served from the unified index — the mirror this PR relies on). No parallel fix PR exists for this thread.

Labels applied to PR #5988 (was unlabeled): enhancement · storage · scope/storage · p3 — judged from the diff itself: it completes the opt-in storage.jsonWriter=off contract (behavior change for an opt-in lane only, no user-visible defect ⇒ not bug; low blast radius ⇒ p3).

Review in progress. Initial read of the diff and the surrounding save/read/fork/restore paths is positive; I am now verifying the PR's claims against the code (unified-first read path, same-save ccr ingest, fork lane under off mode, restore-replica path), running the targeted test file locally, and will post the full structured conclusion (八问) with a CI status re-check shortly.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review of PR #5988 — verdict: correct and safe to merge once CI is green; one trivial pre-merge rebase recommended. Labels applied to the PR: enhancement, p3, storage, scope/storage (judged from this diff: completes the opt-in off-lane contract, opt-in only ⇒ p3); mirror issue labeled p3 + scope/storage.

What I verified (head 3554fe3fdb, checked out locally; targeted suite + typecheck run here):

  1. Gate placement & semantics. The new off gate in saveContentStore (src/persist.ts:847-863) sits after the emptied-store delete branch and before the durable write. writerOffActive() = jsonWriterOff && shadow !== undefined (src/persist.ts:3862-3864) is static per process (jsonWriterOff assigned once at src/persist.ts:647; tryRecoverSidecar src/persist.ts:3652-3676 can only flip false→true mid-process, never back). All four save paths (scheduleSave :3127, writeNow :3378, flushSync :3681, persistForkSession :3747) hit the gate before any file write, and the same save's builder ingests ccr_json + ccr_entries from the live store (src/storage/ingest.ts:505) — so the index mirror lands in the same cycle the file write is skipped. The read path is unified-first (contentStoreOf() src/store.ts:118-128), so in effective-full mode the file is never consulted. Claim (a)/(b) of the PR body hold.
  2. Dirty-flag clear. Semantically correct: the growth already landed in the index, and clearing keeps the dirty⇒unsaved invariant clean (it is also what lets dropResidentContentStore's dirty guard pass had it run — see nit 1 on why it doesn't under off).
  3. Fork/restore lanes — verified empirically, not just by reading. Under full+off I ran a scratch flow: an existing sidecar stays byte-identical after an off-mode save (gate works); createReplica+restoreReplica still materializes the restored id's sidecar (unconditional by design, unchanged by this PR) and its CCR store lands in the index via shadow.record(record) (src/persist.ts:2713 → planIngest ccrJson) — a fresh handle's unifiedCcrFor(restoredId) returned the stored originals. Fork publication is equally covered (createForkAsync → cloneSessionRows → ingestSession, src/storage/store.ts:3001-3095). No stranded payloads under off.
  4. Regression inventory. Default-on users: zero change (the gate only engages with jsonWriter:"off"). Off lane: sidecar stops updating (intended); the emptied-store delete is preserved (a genuinely emptied store still removes a stale sidecar); boot/web/GC walkers already skip sidecars (src/persist.ts:1003/1631/1687, src/storage-maintenance.ts:71); delete-session paired removal unchanged (src/persist.ts:1561); rollback = unset the knob and both writers resume.
  5. Test validity. The new leg has teeth in all three directions: (1) grafts a different store onto the session and asserts the sidecar stays byte-identical after an off-mode save — any accidental write fails it; (2) a fresh handle must resolve the grafted originals from sessions.ccr_json; (3) control leg with the switch ON writes the file, proving the skip is exactly the off gate (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 both-directions discipline). Local run: 4/4 pass (hermetic XDG), npm run typecheck clean.
  6. Diff cleanliness. 6 files, +119/-13, every line on-topic (one gate hunk + comment, one test, docblock + regenerated docs + yaml wording). No whitespace churn, no lockfile, no version bump.

Behavior-change disclosure (required): under the OPT-IN storage.jsonWriter:"off" (default remains "on"): OLD — the per-session *.content-store.json sidecar kept being written/updated on every save even while the envelope was suppressed; NEW — it is no longer written or updated, and the durable copy is the index's sessions.ccr_json mirror ingested in the same save cycle. Durability shifts from "file written payload-first" to "index ingested in the same save" — the identical shift Phase 3 already made for the envelope. Default users are untouched.

Two wording nits (report-only — per [PR] wake rules I did not push anything to the branch):

  1. The gate comment says "this save drops the resident store right after the ingest (dropResidentContentStore)" — but dropResidentContentStore's first line early-returns under writerOffActive() (src/persist.ts:3593, pre-existing [perf][storage] jsonWriter=off still burns 12–20 s CPU per round: full bundle construction + 138 MB envelope readback + | [perf][storage] jsonWriter=off 仍每轮烧 12–20 s CPU:整包构建 + 138 MB envelope 回读 + drop→restore 死循环 #5805 guard), so under off mode the resident store is never dropped; the described stale-dirty → writer-resume → deletion trap is unreachable in-process (and the flag can't outlive the process — restarts reload with dirty=false). Clearing the flag is still right; the justification sentence should be rewritten to match current code.
  2. The PR body justifies keeping restore materialization unconditional because "the index does not carry forked CCR stores" — empirically false in effective-full mode: the index carries them via buildSession's forkContentStore mapping (src/persist.ts:4117), ingested at restore (src/persist.ts:2713) and at fork publication. The decision itself stays safe and conservative; only the stated reason is inaccurate.

Pre-merge housekeeping: the branch is 26 commits behind master (merge-base 33d495180a). Since base, master touched neither src/persist.ts nor the test file — only small config-doc churn overlapping this PR's files (108b1fde08 +4 lines docs/configuration.md; fa447f9ce5 1-line doc + 2-line process.yaml), so a rebase should be near-clean (regenerate config docs if the drift guard complains). GitHub mergeable-state is still computing; the mirror reports "behind", not conflicting.

CI (live-checked 02:07Z): 0 failed / 6 success (gates + docs-sync) / 12 queued / 12 skipped of 30 total; the 6 earlier "cancelled" legs belong to superseded prior runs. Full test lanes still in flight — if anything turns red I'll follow up with targeted diagnosis.

八问(中文结构化结论):

  1. 是什么问题:opt-in 的 storage.jsonWriter:"off"(Phase 3,Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960)此前只停写 JSON envelope,却仍在每次保存时写 CCR content-store sidecar(*.content-store.json)——该文件在 full 模式下从不被读取(读路径 unified-first),属死重。本 PR 让 off 开关同时停写 sidecar。
  2. 什么原因导致的:Phase 3 设计时把「索引即唯一持久副本」的承诺只兑现到 envelope 一层,sidecar 写入路径(saveContentStore)没有纳入同一开关的管辖范围。
  3. 影响范围与严重程度:仅影响显式配置 jsonWriter:"off" 的用户(opt-in 车道;默认 "on" 完全不受影响)。修复前无功能错误,只是多写一个永不读取的文件;修复后 off 会话的 CCR 原文持久化由 sidecar 文件改为同一保存周期内摄入的索引 ccr_json 镜像。严重程度 p3。
  4. 怎么修复的:在 saveContentStore 的空店删除分支之后、durableWriteFile 之前加 off 门控——跳过文件写入并清掉 contentStoreDirty(增长已落入索引,保持 dirty⇒未落盘 不变量干净);fork/restore 的一次性物化(writeContentStoreSidecar)刻意不动,保住可移植性出口。
  5. 是否有回归问题:逐项审过——默认 on 用户零变化;off 车道空店删除保留(真清空仍删陈旧 sidecar);boot/web/GC 遍历本就跳过 sidecar;删除会话成对删除不变;回滚=去掉开关恢复双写。已披露行为变更(old→new 见上):off 模式下 sidecar 停止更新,持久化语义从「payload 先写文件」变为「同周期摄入索引」——与 Phase 3 对 envelope 做的同一转移,非新增风险面。另发现两处措辞失实(门控注释称本保存会 drop resident store,实际 dropResidentContentStore 自 [perf][storage] jsonWriter=off still burns 12–20 s CPU per round: full bundle construction + 138 MB envelope readback + | [perf][storage] jsonWriter=off 仍每轮烧 12–20 s CPU:整包构建 + 138 MB envelope 回读 + drop→restore 死循环 #5805 起在 off 下首行即返回;PR 正文称索引不承载 forked CCR store,实测 full 模式下经 buildSession:4117→restore :2713/fork cloneSessionRows 均承载)——不影响正确性,建议合并前顺手改注释。
  6. 是否破坏缓存:不涉及。缓存/重试/增量游标/去重键均未触碰;CCR 镜像键(content hash/sha256)机制未变。
  7. 是否改了设计方向:不改方向,是把 Phase 3(Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960)既有方向补全到 sidecar 这一层;数据流、边界、双模式划分均不变。
  8. 是否引入了新的配置:不引入新配置。复用既有 storage.jsonWriter(env BILI_STORAGE_JSON_WRITER,默认 "on");文档(process.yaml 双语 + configuration.md 再生成 + config.ts docblock)同步更新措辞。

判定:可合并 —— 代码正确、测试三向有牙、diff 干净、CI 未见失败;前置条件为 CI 全绿 + 一次轻量 rebase(落后 master 26 提交,无 persist.ts/测试文件冲突面)。按规则我不执行合并、不代推分支;两处措辞建议留给作者或后续小修。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI is now green (live-checked via mirror cache at 02:21Z): all 30 checks resolved on head 3554fe3fdb — 14 success / 16 skipped / 0 failed. The earlier "cancelled" legs were superseded prior runs, not failures.

Updated verdict: mergeable. The only remaining pre-merge item is the trivial rebase onto current master (branch is 26 commits behind; no overlap on src/persist.ts or the test file since base — only small config-doc/yaml churn, regenerate docs if the drift guard complains), plus optionally the two wording fixes noted above (gate comment vs dropResidentContentStore's off-mode early return; PR body's "index does not carry forked CCR stores" — empirically false under full mode). Merging stays human-only per repo rules.

@maozhuoshushu

Copy link
Copy Markdown

本机可以给「off 之后还有哪些写入残留」补一份实测,正好是这条 PR 想收掉的那个 sidecar。

环境:0.1.213 / 133f3525c / storage.unified=full / sqlite / Windows 11。storage.jsonWriter: "off" 自 2026-11-08 23:26 起生效(POST /__bili/shutdown → 宿主自动重新 spawn,40 s 内回来)。

一、信封侧确实归零,和 #3275 的结论一致

~/.local/share/billion-context/sessions/ 下 289 个 *.json 信封合计 717.2 MB,最新 mtime 停在 2026-11-08 23:26:42,此后零写入。

*.content-store.json 在这台机器上是 0 个——所以本机无法复现 #2960 描述的 sidecar 残留写入,但这不能证明它不存在:本机 ccr_entries 表 0 行、content 表 715,995 行 / 3,728 MB,CCR 取回路径基本没被走过。要复现得先有用过 CCR 取回的会话。

二、但 off 之后仍有 190 GiB/天,且文件系统只能解释 1.4%

这是我觉得最值得写进 docs 的一条:光看文件增长会严重低估 off 模式下的物理写。

300 s 对账窗口(纯 SMART 前后差 vs 目录递归 + 文件重写 + WAL 追加)
SMART 实测                        684.6 MB
可归因                              9.8 MB
解释比例                            1.4 %

剩下的在 index.db:主库 5.83 GB + 恒定 8,388,608 B 的 -wal 循环。SQLite 的页级重写在文件系统层只表现为「mtime 变了、size 没变」——任何 Get-ChildItem / find -newer 都扫不到。

所以 #5992 那个 CI gate(off 模式下 sessions 目录只允许白名单 .json)很有价值,但它只能拦住「文件」这一层;如果验收口径想覆盖「off 到底省了多少物理写」,还得配一条盘级测量(下面第三节)。

三、可以照抄的最小测量法(只读,PowerShell)

$sm='C:\Program Files\smartmontools\bin\smartctl.exe'
$d0=(& $sm -A -d nvme /dev/sdb | Select-String 'Data Units Written:') -replace '\D',''
Start-Sleep 600
$d1=(& $sm -A -d nvme /dev/sdb | Select-String 'Data Units Written:') -replace '\D',''
"{0:N1} GiB/day" -f ((($d1-$d0)*512000/1GB)/(600/86400))

两个必须提醒的坑:

  1. 短窗口不可信。Data Units Written 会被写缓存 / SLC 状态推迟计入:我们写 512 MB 顺序 + 512 MB 重写 + 156 MB 4K 随机,SMART 只涨了 1.2 MB;反过来静止 600 s 又会补涨 1,326 MB。窗口取 ≥300 s,最好 600 s。
  2. 逐进程计数器不能用。Win32_PerfRawData_PerfProc_Process.IOWriteBytesPersec 会把 loopback 网络流量算进「写」——所有进程加起来能到 LogicalDisk 实测的 1160%,bili 自己显示 9.85 MB/s 写而数据目录同期增长 0.00 MB/s。要逐进程归因只能上 ETW(wpr -start FileIO,注意把 ETL 输出到另一块盘,否则污染基线)。

四、我们的实测序列(同一台机器,同一块盘)

阶段 GiB/天
问题期(10-09→11-09 墙钟均值) 636
jsonWriter=off 124–184
+ 三个快照接 zstd 102–125
+ 快照 debounce 500 ms→30 s 190(600 s 安静窗口,2,787 units,零次 ≥500u 跳变)

注意 off 本身占了大头(636 → 124–184),但离「信封停写就没事了」还差一个数量级。

完整方法、四道治理的收益与代价、复现与回滚:https://gist.github.com/maozhuoshushu/11331b6e9c3388d431365f2fffd1df92

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

谢谢实测数据——「光看文件增长会严重低估 off 模式下的物理写」是重要的现场证据。几点与代码的对照:

1. 「本机无法复现 sidecar 残留」的推断成立。 saveContentStore 只在会话的 content store 非空且 dirty 时才进写分支;你本机 ccr_entries 0 行 ⇒ CCR 取回路径没被走过 ⇒ store 恒空,永远到不了写分支。在跑过压缩、CCR 原文有增长的机器上,本 PR 合并前(即当前所有已发布版本)+ off 应能看到 *.content-store.json 的 mtime 随带 CCR 增长的保存推进(kernel 保留 resident store,增长即置 dirty)——这正是本 PR 要收掉的残留;合并后同一场景下 mtime 应冻结在开关生效时刻。可拿这条做复现验收。

2. 范围澄清:index.db 的物理写与本 PR 无关。 你测到的主库 + WAL 页级重写(190 GiB/day 量级)是 Phase 3 起就存在的存储通道,本 PR 既不引入也不改变它——只去掉 sidecar 文件写入(对你这台无 CCR 使用的机器几乎是零差量;对重度用 CCR 的机器,去掉的是每次保存一次的 sidecar 重写)。你测到的恒定 -wal 尺寸 8,388,608 B 与引擎固定 wal-cap 一致(我们本地运行日志里同样是 wal-cap: 8388608),所以「WAL 循环 + 页级重写对文件遍历不可见」的归因成立。#5992 那个 CI gate(文件层)与盘级测量(物理写层)互补,你的定位准确。

3. 一个小疑点:表里最后一行 190 高于上一行的 102–125 区间。按你自己指出的 SMART 延迟计入坑,这更像窗口方差而非 debounce 引起回退;若复测仍稳定高于上一行,才值得单独查。

测量方法是否写进 docs、off 是否加盘级验收口径,属于 owner 决策——我不会基于未经验证的外部数据单方面改文档;若决定采纳,你 gist 里的完整方法可直接作为输入。

ranxianglei pushed a commit that referenced this pull request Nov 9, 2026
… the sessions dir (#2960)

#5988 stopped the CCR content-store sidecar write under the off switch,
but the contract "the SQLite index is the ONLY persistent copy" had no
mechanical guard: any future save path that starts writing JSON there
would silently regress it (the sidecar itself is exactly that shape of
violation). This adds the gate.

tests/storage-json-writer-off-json-writes.test.ts:
- snapshot-diff design: every *.json file in the store dir is hashed
  before/after each phase; added/modified entries are checked against the
  whitelist. No fs interception — whatever API a write path uses shows up
  in the diff, so a new writer cannot hide behind an un-intercepted call.
- Phases: A first save (ZERO .json writes), B restart adopt + save
  (ZERO), C replica + restore (the restored id's own materialized files
  ONLY — envelope + CCR sidecar, named via relPathFor/
  contentStoreRelPathFor; session files are hash-named, so the concrete
  per-run set is derived from the restored id, not a regex), plus an
  on-control leg proving the snapshot mechanism actually sees writes.
- JSON_WRITE_WHITELIST documents the allowed category (restore
  materialization: the index is not a restore source and does not carry
  forked CCR stores); extending it requires the code path and the reason
  the index cannot carry the bytes.
- Mutation-verified: disabling the #5988 sidecar gate makes phase A fail
  with the offender file listed — the gate catches exactly the regression
  #5988 fixed.

Scope: the sessions directory (persistence layer). CLI-level state files
under stateDir() are handled separately by #5757 (kv_store migration).

Verification: typecheck ✓, gate suite 2/2 ✓, sibling off suites +
issue5805 9/9 ✓, storage regression battery 110/110 ✓,
check:unused-exports ✓, check:test-import-graph ✓,
gen-config-docs check-coverage ✓, check-architecture-docs ✓
@ranxianglei
ranxianglei merged commit 8efaa4c into master Nov 9, 2026
126 of 164 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner Author

我已经单独转载为issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants