Skip to content

[PR] storage: CI gate — jsonWriter=off may only write whitelisted .json in the sessions dir (#2960) - #5992

Merged
ranxianglei merged 2 commits into
masterfrom
2026-11-09_jsonwriter-off-json-write-gate
Nov 9, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-11-09_jsonwriter-off-json-write-gate

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

A CI gate for the storage.jsonWriter: "off" contract (#2960 Phase 3, hardened by #5988): while the switch is off, the sessions directory may gain a .json file only through the whitelisted materialization paths — anything else fails CI.

The off switch promises "the SQLite index is the ONLY persistent copy." Nothing enforced that: #5988's CCR sidecar write was exactly a silent violation of the promise, found by audit, not by CI. This PR adds the mechanical guard so the next such writer is caught in CI, not in a review.

How

tests/storage-json-writer-off-json-writes.test.ts — snapshot diff, not fs interception:

  • Every *.json file in the store dir is hashed before and after each phase; added/modified entries are checked against the whitelist. Whatever API a write path uses (writeFileSync, rename, fd) shows up in the diff — a new writer cannot hide behind an un-intercepted syscall.
  • Phase A first save of a fresh store → zero .json writes.
  • Phase B restart → unified-first adopt of the index-only session + save → zero.
  • Phase C replica + restore → only the restored id's own materialized files (portable envelope + CCR sidecar). Session files are named by a hash of the id (relPathFor), never by the id itself, so the allowed set is derived per-run from the restored id via relPathFor/contentStoreRelPathFor — a regex over names cannot anchor a restore.
  • Phase D control: switch ON, same first save → the snapshot MUST see writes (guards against a vacuous gate).
  • JSON_WRITE_WHITELIST documents the allowed category (restore materialization: the index is not a restore source, and it does not carry forked CCR stores — buildRecord drops forkContentStore on the next live save). Extending it requires naming the code path and why the index cannot carry the bytes — callout in the assertion message.
  • Mutation-verified: with [PR] storage: jsonWriter=off also stops the CCR content-store sidecar (#2960) #5988's sidecar gate disabled, phase A fails with the offender file listed. The gate catches exactly the regression [PR] storage: jsonWriter=off also stops the CCR content-store sidecar (#2960) #5988 fixed.

Scope

Sessions directory (persistence layer) only. CLI-level state files under stateDir() are handled separately by #5757 (kv_store migration).

Stacking note

This branch is stacked on #5988 (3554fe3fd is included) — the gate's phase A requires the sidecar fix to pass, by design. Merge #5988 first; this PR then reduces to the single gate commit.

Verification

  • npm run typecheck ✓
  • gate suite 2/2 ✓ · sibling off suites + issue5805-jsonwriter-ownership 9/9 ✓
  • storage regression battery 110/110… (review-fixes, plugin-fork, storage-corpus, export-jsonl, issue5046, storage-full-flip, memory-contentstore-lazy, issue4816, storage-save-path) ✓
  • check:unused-exports ✓ · check:test-import-graph ✓ · gen-config-docs check-coverage ✓ · check-architecture-docs ✓

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-09_jsonwriter-off-json-write-gate (2f89ca0)

Latest build — install this exact version

npm install -g billion-context@0.0.8833-pr.5992.4017916

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
2f89ca0 billion-context@0.0.8833-pr.5992.4017916 build
e45be45 billion-context@0.0.8722-pr.5992.f729432 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5992.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title storage: CI gate — jsonWriter=off may only write whitelisted .json in the sessions dir (#2960) [PR] storage: CI gate — jsonWriter=off may only write whitelisted .json in the sessions dir (#2960) Nov 9, 2026
ework-agent added 2 commits November 9, 2026 23:28
Phase 3 said the off switch covers the session envelope; the per-session
CCR content-store sidecar kept being written either way. In effective-full
mode that file is never read (contentStoreOf is unified-first) and the
same save already ingests the store into the index (sessions.ccr_json),
so the sidecar write was dead weight contradicting the switch's contract:
the SQLite index is the ONLY persistent copy.

- saveContentStore: writerOffActive() gate after the emptied-store rm
  branch — off skips the file write; the builder's ccr ingest (same save
  cycle) is the durable copy.
- The gate clears contentStoreDirty: the off save drops the resident
  store right after the ingest (dropResidentContentStore), and a stale
  dirty=true with contentStore=null would make a later writer-resume
  save hit the emptied-store branch and DELETE the sidecar file.
  Convergence after a writer-resume comes from the kernel re-adopting
  its in-memory store on the next turn.
- Fork/restore one-shot materialization (writeContentStoreSidecar) stays
  unconditional: the index does not carry forked CCR stores, so skipping
  there would strand forked payloads.
- Tests: new suite proves the sidecar bytes are untouched under off, the
  index mirror resolves the originals, and an on-control leg still
  writes the file. Docs: config docblock + process.yaml (EN/ZH) +
  regenerated configuration.md.
… the sessions dir (#2960)

#5988 stopped the CCR content-store sidecar write under the off switch,
but the contract "the SQLite index is the ONLY persistent copy" had no
mechanical guard: any future save path that starts writing JSON there
would silently regress it (the sidecar itself is exactly that shape of
violation). This adds the gate.

tests/storage-json-writer-off-json-writes.test.ts:
- snapshot-diff design: every *.json file in the store dir is hashed
  before/after each phase; added/modified entries are checked against the
  whitelist. No fs interception — whatever API a write path uses shows up
  in the diff, so a new writer cannot hide behind an un-intercepted call.
- Phases: A first save (ZERO .json writes), B restart adopt + save
  (ZERO), C replica + restore (the restored id's own materialized files
  ONLY — envelope + CCR sidecar, named via relPathFor/
  contentStoreRelPathFor; session files are hash-named, so the concrete
  per-run set is derived from the restored id, not a regex), plus an
  on-control leg proving the snapshot mechanism actually sees writes.
- JSON_WRITE_WHITELIST documents the allowed category (restore
  materialization: the index is not a restore source and does not carry
  forked CCR stores); extending it requires the code path and the reason
  the index cannot carry the bytes.
- Mutation-verified: disabling the #5988 sidecar gate makes phase A fail
  with the offender file listed — the gate catches exactly the regression
  #5988 fixed.

Scope: the sessions directory (persistence layer). CLI-level state files
under stateDir() are handled separately by #5757 (kv_store migration).

Verification: typecheck ✓, gate suite 2/2 ✓, sibling off suites +
issue5805 9/9 ✓, storage regression battery 110/110 ✓,
check:unused-exports ✓, check:test-import-graph ✓,
gen-config-docs check-coverage ✓, check-architecture-docs ✓
@ranxianglei
ranxianglei force-pushed the 2026-11-09_jsonwriter-off-json-write-gate branch from e45be45 to 2f89ca0 Compare November 9, 2026 15:29
@ranxianglei
ranxianglei merged commit 9bbc2a4 into master Nov 9, 2026
49 checks passed
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1130-master.9bbc2a4 (rolling master tag).

npm install -g billion-context@0.0.1130-master.9bbc2a4  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant