Repository navigation
[PR] storage: jsonWriter=off also stops the CCR content-store sidecar (#2960) - #5988
Conversation
Phase 3 said the off switch covers the session envelope; the per-session CCR content-store sidecar kept being written either way. In effective-full mode that file is never read (contentStoreOf is unified-first) and the same save already ingests the store into the index (sessions.ccr_json), so the sidecar write was dead weight contradicting the switch's contract: the SQLite index is the ONLY persistent copy. - saveContentStore: writerOffActive() gate after the emptied-store rm branch — off skips the file write; the builder's ccr ingest (same save cycle) is the durable copy. - The gate clears contentStoreDirty: the off save drops the resident store right after the ingest (dropResidentContentStore), and a stale dirty=true with contentStore=null would make a later writer-resume save hit the emptied-store branch and DELETE the sidecar file. Convergence after a writer-resume comes from the kernel re-adopting its in-memory store on the next turn. - Fork/restore one-shot materialization (writeContentStoreSidecar) stays unconditional: the index does not carry forked CCR stores, so skipping there would strand forked payloads. - Tests: new suite proves the sidecar bytes are untouched under off, the index mirror resolves the originals, and an on-control leg still writes the file. Docs: config docblock + process.yaml (EN/ZH) + regenerated configuration.md.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8713-pr.5988.068d45aThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5988.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Dup check: not a duplicate. Searched open+closed issues/PRs for Labels applied to PR #5988 (was unlabeled): Review in progress. Initial read of the diff and the surrounding save/read/fork/restore paths is positive; I am now verifying the PR's claims against the code (unified-first read path, same-save ccr ingest, fork lane under off mode, restore-replica path), running the targeted test file locally, and will post the full structured conclusion (八问) with a CI status re-check shortly. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Review of PR #5988 — verdict: correct and safe to merge once CI is green; one trivial pre-merge rebase recommended. Labels applied to the PR: What I verified (head
Behavior-change disclosure (required): under the OPT-IN Two wording nits (report-only — per [PR] wake rules I did not push anything to the branch):
Pre-merge housekeeping: the branch is 26 commits behind master (merge-base CI (live-checked 02:07Z): 0 failed / 6 success (gates + docs-sync) / 12 queued / 12 skipped of 30 total; the 6 earlier "cancelled" legs belong to superseded prior runs. Full test lanes still in flight — if anything turns red I'll follow up with targeted diagnosis. 八问(中文结构化结论):
判定:可合并 —— 代码正确、测试三向有牙、diff 干净、CI 未见失败;前置条件为 CI 全绿 + 一次轻量 rebase(落后 master 26 提交,无 persist.ts/测试文件冲突面)。按规则我不执行合并、不代推分支;两处措辞建议留给作者或后续小修。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI is now green (live-checked via mirror cache at 02:21Z): all 30 checks resolved on head Updated verdict: mergeable. The only remaining pre-merge item is the trivial rebase onto current master (branch is 26 commits behind; no overlap on src/persist.ts or the test file since base — only small config-doc/yaml churn, regenerate docs if the drift guard complains), plus optionally the two wording fixes noted above (gate comment vs |
|
本机可以给「off 之后还有哪些写入残留」补一份实测,正好是这条 PR 想收掉的那个 sidecar。 环境:0.1.213 / 一、信封侧确实归零,和 #3275 的结论一致
二、但 off 之后仍有 190 GiB/天,且文件系统只能解释 1.4% 这是我觉得最值得写进 docs 的一条:光看文件增长会严重低估 off 模式下的物理写。 剩下的在 所以 三、可以照抄的最小测量法(只读,PowerShell) $sm='C:\Program Files\smartmontools\bin\smartctl.exe'
$d0=(& $sm -A -d nvme /dev/sdb | Select-String 'Data Units Written:') -replace '\D',''
Start-Sleep 600
$d1=(& $sm -A -d nvme /dev/sdb | Select-String 'Data Units Written:') -replace '\D',''
"{0:N1} GiB/day" -f ((($d1-$d0)*512000/1GB)/(600/86400))两个必须提醒的坑:
四、我们的实测序列(同一台机器,同一块盘)
注意 off 本身占了大头(636 → 124–184),但离「信封停写就没事了」还差一个数量级。 完整方法、四道治理的收益与代价、复现与回滚:https://gist.github.com/maozhuoshushu/11331b6e9c3388d431365f2fffd1df92 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 谢谢实测数据——「光看文件增长会严重低估 off 模式下的物理写」是重要的现场证据。几点与代码的对照: 1. 「本机无法复现 sidecar 残留」的推断成立。 2. 范围澄清:index.db 的物理写与本 PR 无关。 你测到的主库 + WAL 页级重写(190 GiB/day 量级)是 Phase 3 起就存在的存储通道,本 PR 既不引入也不改变它——只去掉 sidecar 文件写入(对你这台无 CCR 使用的机器几乎是零差量;对重度用 CCR 的机器,去掉的是每次保存一次的 sidecar 重写)。你测到的恒定 3. 一个小疑点:表里最后一行 190 高于上一行的 102–125 区间。按你自己指出的 SMART 延迟计入坑,这更像窗口方差而非 debounce 引起回退;若复测仍稳定高于上一行,才值得单独查。 测量方法是否写进 docs、off 是否加盘级验收口径,属于 owner 决策——我不会基于未经验证的外部数据单方面改文档;若决定采纳,你 gist 里的完整方法可直接作为输入。 |
… the sessions dir (#2960) #5988 stopped the CCR content-store sidecar write under the off switch, but the contract "the SQLite index is the ONLY persistent copy" had no mechanical guard: any future save path that starts writing JSON there would silently regress it (the sidecar itself is exactly that shape of violation). This adds the gate. tests/storage-json-writer-off-json-writes.test.ts: - snapshot-diff design: every *.json file in the store dir is hashed before/after each phase; added/modified entries are checked against the whitelist. No fs interception — whatever API a write path uses shows up in the diff, so a new writer cannot hide behind an un-intercepted call. - Phases: A first save (ZERO .json writes), B restart adopt + save (ZERO), C replica + restore (the restored id's own materialized files ONLY — envelope + CCR sidecar, named via relPathFor/ contentStoreRelPathFor; session files are hash-named, so the concrete per-run set is derived from the restored id, not a regex), plus an on-control leg proving the snapshot mechanism actually sees writes. - JSON_WRITE_WHITELIST documents the allowed category (restore materialization: the index is not a restore source and does not carry forked CCR stores); extending it requires the code path and the reason the index cannot carry the bytes. - Mutation-verified: disabling the #5988 sidecar gate makes phase A fail with the offender file listed — the gate catches exactly the regression #5988 fixed. Scope: the sessions directory (persistence layer). CLI-level state files under stateDir() are handled separately by #5757 (kv_store migration). Verification: typecheck ✓, gate suite 2/2 ✓, sibling off suites + issue5805 9/9 ✓, storage regression battery 110/110 ✓, check:unused-exports ✓, check:test-import-graph ✓, gen-config-docs check-coverage ✓, check-architecture-docs ✓
|
我已经单独转载为issue |
What
storage.jsonWriter: "off"(the Phase 3 envelope switch, #2960) now also stops writing the per-session CCR content-store sidecar (*.content-store.json), not just the session envelope. Before this PR, off mode promised "the SQLite index is the only persistent copy" but still wrote the CCR sidecar file every save — dead weight, since in full mode that file is never read (reads go unified-first) and the same save already ingests the store into the index.Changes
src/persist.tssaveContentStore: when the off switch is active, the sidecar write is skipped (after the emptied-store delete branch). The durable copy is the index's ccr mirror (sessions.ccr_json), ingested by the same save's builder.contentStoreDirty(found while testing): the off save drops the in-memory store right after ingesting it (dropResidentContentStore). If the dirty flag were lefttrue, a later save — once the writer resumes because the sidecar self-disabled — would see "dirty + empty in-memory store" and take the emptied-store branch, which deletes the sidecar file. Clearing the flag removes that trap; after a writer-resume the kernel re-adopts its in-memory store on the next turn and the file converges from there.writeContentStoreSidecar(the one-shot sidecar write when restoring a replica) is intentionally unchanged — the index does not carry forked CCR stores, so skipping it there would strand forked payloads.process.yaml(EN/ZH), regenerateddocs/configuration.md— the old wording "the content-store sidecar keeps being written either way" is replaced.tests/storage-json-writer-off.test.tswith three legs — (1) under off the sidecar bytes are untouched (a differently-grafted store would have changed them), (2) a fresh handle resolves the originals from the index mirror, (3) control: with the switch on, the same kind of save does write the file.Rollback / safety
Verification
npm run typecheck✓tests/storage-json-writer-off.test.ts4/4 ✓ (clean XDG)check:unused-exports✓ ·check:test-import-graph✓ ·gen-config-docs check-coverage✓ ·check-architecture-docs✓