Skip to content

feat: zstd-compress session files by default, decoupled from encryption (#1080) - #1083

Merged
ranxianglei merged 4 commits into
masterfrom
2026-09-21_persist-zstd-default
Sep 22, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-09-21_persist-zstd-default

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1080

What changed

Session files are now zstd-compressed by default, and compression is fully decoupled from BILI_ENCRYPTION_KEY:

  • New format for unencrypted files: BILIZSTD1 — 9-byte magic + format version (0x01) + mode byte (0x00 raw / 0x01 zstd) + body. The body is zstd-compressed on Node ≥ 22.15 and stored raw on older runtimes; readers try native zstd first, then fall back to the already-bundled fzstd WASM decompressor, so files stay readable across runtimes/versions (no corruption in mixed-version trees).
  • BILIENC1 ([Feature] Can bili context encrypt the data files it outputs? #708) stays byte-compatible. Its mode byte now records whether the body is compressed, and BILI_PERSIST_ZSTD=0 applies inside it too — encryption and compression are independent knobs.
  • Opt-out: BILI_PERSIST_ZSTD=0 (fits the existing BILI_PERSIST* env family).
  • One-time migration reuses the [Feature] Can bili context encrypt the data files it outputs? #708 boot path: legacy plain-JSON files are re-encoded via temp+rename atomic replace; anything already carrying either magic is skipped — in particular BILIENC1 files are never touched when no key is set (they cannot be decoded).
  • createSessionCodec(key) → createStorageCodec({ key?, compress? }); no codec when neither is active (opt-out users behave exactly as before).
  • bili export, the web UI, and decompress search need no changes — every read goes through the store codec.

One deviation from the issue thread

The assessment comment suggested running migration "as a background pass after the listener starts". It stays at boot (awaited, exactly like #708): a post-listen pass does direct fs ops bypassing the kernel's per-session write chain, so an active session's save racing the rename could roll the file back to a stale parsed snapshot. Boot-time has no race window; the cost is a one-time bounded delay (walk+peek is O(files); only plain-JSON files get rewritten).

Tests

New tests/persist-zstd.test.ts (14 tests): header/determinism, config matrix, plaintext passthrough + cross-tree dispatch, #708 mode-byte regression, static-fixture fzstd decode, store-level default-on, shrinkage vs opt-out baseline, opt-out keeps plain JSON, pre-22.15 raw fallback, cross-runtime read (native write → WASM decode), once-only migration, mixed tree, keyed boot leaves BILIZSTD1 untouched, no-key boot reports BILIENC1 as unreadable with an actionable message, decoupled raw-body encryption. Four existing test files that inspect raw session files pin BILI_PERSIST_ZSTD=0.

Preflight (this commit)

  • npm run typecheck — clean
  • npm test — 2097 pass / 0 fail / 2 skip (same skips as master)
  • npm run build — success

Docs: CONFIGURATION.md + CONFIGURATION.zh-CN.md (new row + rewritten BILI_ENCRYPTION_KEY row) and CHANGELOG [Unreleased].

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-21_persist-zstd-default (55723e2)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1083

Each push to this PR publishes a new version under the pr-1083 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1083.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewed — verified on rebased head, ready for human merge.

I did not redo the work; I pulled the branch, audited the diff, and re-verified against live master.

Code audit (diff vs master)

  • src/encrypt.ts: createSessionCodec(key) → createStorageCodec({ key?, compress? }) returning undefined when neither is active (opt-out users keep plain JSON). Dispatch on magic in decode(): BILIENC1 → AES-256-GCM, BILIZSTD1 → zstd/raw, else plaintext passthrough. No-key boot reading a BILIENC1 file throws an actionable error naming BILI_ENCRYPTION_KEY instead of surfacing as corrupt garbage. inflateZstd() uses native node:zlib when present, bundled fzstd WASM otherwise — files written by any supported runtime read back on every supported one.
  • src/persist.ts: persistZstdEnabled() reads BILI_PERSIST_ZSTD (0/false opts out); migrateLegacyFiles() now skips both magics via startsWithMagic() so already-encoded files (incl. undecodable BILIENC1 without a key) are never touched. Boot-time migration kept over the thread's suggested post-listen background pass — the race argument holds (direct fs ops bypass the kernel's per-session write chain; an active save racing the rename would roll back to a stale snapshot).
  • fzstd@0.1.1 is a declared devDependency (tsup-inlined like acp-kernel), not a transitive leak. package.json untouched by this commit.
  • New tests/persist-zstd.test.ts (14 tests) covers header/determinism, config matrix, cross-tree dispatch, [Feature] Can bili context encrypt the data files it outputs? #708 mode-byte regression, static-fixture fzstd decode, default-on store behavior, shrinkage vs opt-out baseline, raw fallback pre-22.15, cross-runtime read, once-only migration, mixed tree, keyed-boot no-touch, no-key actionable error, decoupled raw-body encryption. Four existing raw-file-inspecting test files correctly pin BILI_PERSIST_ZSTD=0.
  • Docs: CONFIGURATION.md + zh-CN row updates, CHANGELOG [Unreleased].

Stale base — fixed
The PR was ~40 commits behind live origin/master (incl. acp-kernel pin → 0.0.82, release v0.1.133). Rebased cleanly onto 7cec6d2 (no overlap beyond package.json, which this commit doesn't touch) and force-pushed the branch: b67074f → 4d27208.

Preflight on rebased head 4d27208 (local, Node v22.23.2)

  • npm ci — ok
  • npm run typecheck — clean
  • npm test — 2185 pass / 0 fail / 2 skip (same 2 skips as master)
  • npm run build — success

CI: on the old head, build-artifact, version-guard, e2e zero-token preflight, and the codex-overlay job were green while the 4× test-matrix jobs were still running; the push above supersedes them and CI will re-run on 4d27208.

This is a feat introducing a new on-disk format, so per §7.4 it stays a human merge — nothing blocks it.

中文摘要:代码审查通过、无问题;PR 落后最新 master(含 acp-kernel 0.0.82 pin)已 rebase 并把分支更新为 4d27208,rebase 后本地全量验证全绿(typecheck 干净、测试 2185 过 0 失败、构建成功),可以合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI follow-up — all 8 checks green on rebased head 4d27208:

  • test (ubuntu-latest, 22) ✅ · test (ubuntu-latest, 24) ✅
  • test (windows-latest, 22) ✅ · test (windows-latest, 24) ✅
  • real codex vs deterministic fake upstream (zero tokens) ✅
  • codex accepts CODEX_HOME overlay + registers bili MCP ✅
  • build-artifact ✅ · version-guard ✅

Nothing outstanding — ready for you to merge whenever.

中文摘要:rebase 后的新提交 4d27208 上 CI 全部通过(含 Windows/Ubuntu × Node 22/24 测试矩阵),无遗留问题,可以合并。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查结论:NEEDS-HUMAN + fix-first(今天不能合)

实作质量整体扎实——magic 分发向后兼容(旧明文/BILIENC1 全部可读)、加密与压缩解耦有回归测试、kernel writeNow tmp+rename 原子性未动、fzstd 纯 JS 已在 master 的 tsup.config.ts noExternal 里(dist 保持自包含,全局安装安全)。但有一个 HIGH 必须先决策:

🔴 HIGH:降级回滚 = 静默销毁会话(默认未加密路径,无人承认此风险)

链路:boot 迁移把所有未加密文件改写为 BILIZSTD1(默认路径=大多数用户)→ 手动降回 0.1.134(标准调试/逃生动作)→ 旧版 src/encrypt.ts:87 对非 BILIENC1 一律 buf.toString("utf8") → JSON.parse 抛 → kernel [persist] skipping corrupt file(acp-kernel persist/index.js:358-362,报错不含格式标记提示)→ loadSync 返回 null → resume 该会话 id 铸出全新空会话 → 下一次 writeNow 覆盖 BILIZSTD1 文件 → 原历史被销毁,不只是读不了。

旧版本侧无法识别 magic(不可能给旧码加探测),所以缓解必须在新代码侧三选一:

  1. 明文保持默认,zstd 先 opt-in 过渡一段时间(最保守);
  2. 迁移加门控(如显式 env 才执行 boot 全量改写);
  3. owner 明确接受降级即丢数据的风险并写进 CONFIGURATION.md + PR body(现在三处都没提)。

持久化格式变更按仓库规范必须人审拍板(AGENTS §7.4),这一条就是为什么。

小项

  • [Low] encode 无尺寸守卫(src/encrypt.ts:125):实测 71B 会话 JSON → 91B(+28%,header+帧开销)。真实会话 ~45× 缩(60,290→1,322B),聚合影响可忽略,但 "压缩后 ≥ 原文则保原样" 一行守卫更整洁。
  • [Process] mergeable_state: behind(base 7cec6d2 < 9c01dbe,间隔只有 #1087 revert + release commit,无文件交叠)——合并前 rebase + 全量门禁重跑。注意:与 #1084 有跨 PR 语义冲突,那边 readRawFile 只认 BILIENC1,两 PR 谁后合谁要适配(详见 #1084 评论)。

ranxianglei pushed a commit that referenced this pull request Sep 21, 2026
- readRawFile is format-agnostic: plain JSON first, then codec decode —
  the sweep no longer silently no-ops on any future on-disk frame
  (zstd once #1083 lands); unknown frames decode-fail to null and are
  left in place, never deleted
- rawInputTokens includes image tokens (image bytes ride every re-send;
  an image-heavy idle session must not look cheap to the sweep)
ranxianglei pushed a commit that referenced this pull request Sep 21, 2026
…ression wins (#1083 review)

- drop migrateLegacyFiles: the boot-time mass re-encode made a downgrade
  (rollback during incident triage) read its own writes as corrupt files,
  resume into empty sessions and overwrite real history on next save.
  Legacy plaintext keeps loading via the codec's magic dispatch and
  converts organically on its next save; boot only sweeps orphaned
  .tmp-enc-* temps from crashed writes.
- encode() size guard: frame-and-compress only when zstd actually shrinks
  the payload; key-less raw bodies and tiny sessions stay bare JSON on
  disk (maximum backward compatibility — older bili reads them natively).
- tests: rewrite the three boot-migration tests to the new contract
  (never-rewrite + organic conversion + mixed-format tree loads), size
  guard coverage at codec level, no-native-zstd path asserts bare JSON.
- docs: CONFIGURATION(.zh-CN).md describe the never-rewrite contract and
  the residual downgrade window (BILIZSTD1 files need bili >= 0.1.135;
  set BILI_PERSIST_ZSTD=0 for downgrade headroom).
@ranxianglei

Copy link
Copy Markdown
Owner Author

Follow-up on the earlier review (#1083 (comment) level findings): fixed on this branch in 69b341f, gates green (typecheck / 2188 tests / build).

1. Boot-time mass migration removed (src/persist.ts)

migrateLegacyFiles() is deleted. Boot no longer rewrites ANY existing session file — it only sweeps orphaned .tmp-enc-* temps from crashed writes (sweepStaleTemps()). Legacy plaintext files load via the codec fallback (JSON parse → fall through) and convert organically on their next save.

Why this matters: the draft migration re-encoded every plaintext file into frames an older bili (≤ 0.1.134) reads as skipping corrupt. Downgrade → resume into an empty session → next save overwrites the real history. A silent mass rewrite at boot is a downgrade trap; the file format must only change when a session is actually being saved.

2. Frame only when compression wins (src/encrypt.ts)

createStorageCodec.encode now guards: compress with zstd, and only emit the BILIZSTD1/BILIENC1 frame when the compressed body is actually smaller; a key-less raw body is stored as bare JSON. Tiny sessions (where the header + zstd overhead exceeded the payload) no longer grow on disk.

3. Tests & docs

  • tests/persist-encryption.test.ts: rewritten to the new contract — "boot never rewrites legacy plaintext files (downgrade safety)" (byte-identity after boot + organic conversion on next save + mixed-format tree loads), foreign/corrupt files untouched, temp sweep without re-encoding.
  • tests/persist-zstd.test.ts: tiny/incompressible payloads stay bare JSON; keyless round-trip returns bare JSON.
  • CONFIGURATION.md / CONFIGURATION.zh-CN.md: "never rewritten at boot … convert organically on next save"; added the downgrade-headroom note.

Residual (documented, not fixed here): once a session is saved as BILIZSTD1, bili < 0.1.135 cannot read it. Anyone needing downgrade headroom should set BILI_PERSIST_ZSTD=0 before saving. If that trade-off is unacceptable the default can still be flipped off at release time — flagging for the owner.

ework-agent and others added 2 commits September 21, 2026 23:25
…ression wins (#1083 review)

- drop migrateLegacyFiles: the boot-time mass re-encode made a downgrade
  (rollback during incident triage) read its own writes as corrupt files,
  resume into empty sessions and overwrite real history on next save.
  Legacy plaintext keeps loading via the codec's magic dispatch and
  converts organically on its next save; boot only sweeps orphaned
  .tmp-enc-* temps from crashed writes.
- encode() size guard: frame-and-compress only when zstd actually shrinks
  the payload; key-less raw bodies and tiny sessions stay bare JSON on
  disk (maximum backward compatibility — older bili reads them natively).
- tests: rewrite the three boot-migration tests to the new contract
  (never-rewrite + organic conversion + mixed-format tree loads), size
  guard coverage at codec level, no-native-zstd path asserts bare JSON.
- docs: CONFIGURATION(.zh-CN).md describe the never-rewrite contract and
  the residual downgrade window (BILIZSTD1 files need bili >= 0.1.135;
  set BILI_PERSIST_ZSTD=0 for downgrade headroom).
@ranxianglei
ranxianglei force-pushed the 2026-09-21_persist-zstd-default branch from 69b341f to 5945e8d Compare September 21, 2026 15:27
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Verified 69b341f — the fixes are exactly as described, and I agree with the downgrade-trap reasoning.

Code audit of the fix commit

  • src/persist.ts: migrateLegacyFiles() deleted. boot() now calls only sweepStaleTemps(), which does pure deletion of orphaned .tmp-enc-* temps matching STALE_ENC_TEMP_RE — no content write path remains at boot. Legacy plaintext loads through the codec's magic-dispatch fallback (decode → JSON parse fallthrough) and converts organically on its next save via the normal codec write chain. Your analysis holds: a boot-time mass re-encode is a downgrade trap (older bili → "skipping corrupt" → empty-session resume → history overwritten on next save), and "the format must only change when a session is actually being saved" is the right invariant.
  • src/encrypt.ts encode guard: compresses first, frames only when z.length < plain.length (strict < — equal-size payloads correctly stay unframed); key-less + MODE_RAW returns the bare JSON buffer byte-identical; keyed raw bodies still get the BILIENC1 frame (correct — encrypted-at-rest can't be stored unframed). Side benefit I'd highlight: tiny sessions now stay backward-compatible indefinitely, not just until boot, since they're framed only once compression genuinely wins.
  • Tests match the new contract: boot leaves legacy files byte-identical with no "re-encoded" log, organic conversion asserted by magic-checking after the next save, mixed-format tree loads, foreign spill file untouched, temp sweep without re-encoding, and codec-level size-guard coverage (tiny/incompressible → bare JSON head 0x7b, compressible payload still frames).

Residual window: agreed it's acceptable as documented — it only opens when a session has grown enough that zstd wins, and BILI_PERSIST_ZSTD=0 is the documented escape hatch for teams that need downgrade headroom during incident triage. I would not flip the default off; default-on with a size guard is the better trade. Owner's call either way.

Rebase + preflight: master moved while this was up (release v0.1.134 merged, #1079 chain-sig reverted). Rebased both commits cleanly onto 9c01dbe; new head 5945e8d, force-pushed to the PR branch. Local preflight on the rebased head: npm ci ok, typecheck clean, npm test 2178 pass / 0 fail / 2 skip, build success. (Your 2188 count included tests/bili-chain-sig.test.ts, which the #1079 revert removed from master — hence the lower total on the rebased tree.)

CI is re-running on 5945e8d; I'll confirm here once the matrix finishes.

中文摘要:已逐行核实 69b341f —— 启动期批量迁移确实删除(boot 只清扫孤儿临时文件、旧文件靠下次保存有机转换)、encode 仅在压缩实际减小时打帧,代码与测试均符合新契约;同意保留默认开启(残余降级窗口已有文档化逃生门)。分支已再次 rebase 到最新 master(含 v0.1.134),rebase 后本地全量验证全绿,新提交 5945e8d 已推送,CI 跑完即可以合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

@ework — the head has moved to 5945e8d (rebased onto current master + the review fix cherry-picked; the fix comment is directly above). Your last review covered the pre-fix c5a5ab9. Please re-review the current head in full: the removed boot migration, the size-guard framing in src/encrypt.ts, and the rewritten persistence tests are the deltas that need fresh eyes.

)

The review fix removed migrateLegacyFiles but two prose references to the
old boot-time mass re-encode survived: a comment in encrypt.ts and the
CHANGELOG entry for #1080. Both now describe the actual contract: legacy
plain-JSON files are never rewritten at boot and convert organically on
their next save.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Full re-review of current head done — this pass read the final state of both core files and both test files end-to-end (not just the incremental fix diff), plus docs. Verdict: correct as shipped, with one small doc residue I fixed directly.

1. Removed boot migration (src/persist.ts) — verified
boot() is now exactly: sweepStaleTemps() → store.loadAll() → marker-gated #286 identity migration. sweepStaleTemps() does pure deletion of names matching STALE_ENC_TEMP_RE = /\.tmp-enc-\d+-\d+$/ — no content-write path remains at boot, and walkJsonFiles (persist.ts:685) explicitly includes those temp names so the sweep sees them. Legacy plaintext loads through the codec's magic-dispatch fallthrough and converts organically via the normal codec write chain. I also checked the two other boot-time writers that remain (#408 negative-token clamp in loadAll, #286 rekey) — both go through the kernel store's own writeNow/delete chain, never raw fs, and are pre-existing/marker-gated, so they don't reopen the downgrade trap.

2. Size-guard framing (src/encrypt.ts) — verified
encode(): compresses first, frames only on strict z.length < plain.length (equal-size stays unframed — correct); key-less + MODE_RAW returns the bare JSON buffer byte-identical; key-less + zstd-wins → BILIZSTD1; keyed always frames (BILIENC1, mode byte records raw vs zstd — necessary since ciphertext/raw can't be stored unframed). decode() dispatches on magic with plaintext passthrough; decryptEnvelope throws the actionable BILI_ENCRYPTION_KEY error before any auth attempt; inflateZstd() native→WASM fallback intact. One property worth calling out: tiny sessions stay backward-compatible indefinitely (framed only once zstd genuinely wins), which strictly widens the downgrade-safe surface beyond "until next save".

3. Rewritten persistence tests — verified, both files read in full

  • tests/persist-zstd.test.ts (16 tests): header/determinism, config matrix, cross-tree dispatch, [Feature] Can bili context encrypt the data files it outputs? #708 mode-byte regression, static-fixture fzstd decode, default-on store, shrinkage baseline, opt-out, no-native-zstd bare JSON, cross-runtime read, boot-never-rewrites (byte-check after boot + no "re-encoded" log + organic conversion asserted by magic after the next save + untouched sibling stays plaintext), mixed tree, keyed-boot no-touch, no-key actionable error, decoupled raw-body encryption, size guard.
  • tests/persist-encryption.test.ts (10 tests): includes the three-phase downgrade-safety test (plaintext tree → keyed boot leaves files byte-identical → s-1 converts on its next save while s-2 stays plaintext → second boot loads the mixed tree), foreign/corrupt files untouched, orphan-temp sweep without re-encoding, wrong-key skip without crash, fail-fast invalid key.

4. Docs — accurate, one residue fixed by me
CONFIGURATION.md / zh-CN rows match the final behavior exactly. But two prose references to the removed boot migration survived the fix commit: a comment in src/encrypt.ts ("boot migration can peek the magic before rewriting") and the CHANGELOG [Unreleased] entry ("the boot migration takes over legacy plain-JSON files once through the same temp+rename atomic path"). Both now describe the actual contract (never rewritten at boot; organic conversion on next save). Committed as 78d8dac (2-line docs-only change) and pushed — new PR head is 78d8dac. Verified: typecheck clean, tests/persist-zstd.test.ts + tests/persist-encryption.test.ts → 26/26 pass.

CI is re-running on 78d8dac; I'll confirm here when the matrix finishes.

中文摘要:对当前头做了完整复审(通读最终代码与两份测试文件,非仅看增量 diff)——移除启动期批量迁移、encode 尺寸守卫、重写的持久化测试三项全部核实无误,降级安全契约有端到端断言覆盖;顺手直接修掉了两处遗留的旧"启动迁移"文档表述(src/encrypt.ts 注释 + CHANGELOG,提交 78d8dac,仅 2 行),本地验证全绿,CI 跑完即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI follow-up — all 8 checks green on final head 78d8dac:

  • test (ubuntu-latest, 22) ✅ · test (ubuntu-latest, 24) ✅
  • test (windows-latest, 22) ✅ · test (windows-latest, 24) ✅
  • real codex vs deterministic fake upstream (zero tokens) ✅
  • codex accepts CODEX_HOME overlay + registers bili MCP ✅
  • build-artifact ✅ · version-guard ✅

Full re-review complete, doc residue fixed, everything verified — ready for you to merge whenever.

中文摘要:最终头 78d8dac 上 CI 全部通过(含 Windows/Ubuntu × Node 22/24 测试矩阵),完整复审与文档修正均已完成,无遗留问题,可以合并。

ranxianglei pushed a commit that referenced this pull request Sep 21, 2026
- readRawFile is format-agnostic: plain JSON first, then codec decode —
  the sweep no longer silently no-ops on any future on-disk frame
  (zstd once #1083 lands); unknown frames decode-fail to null and are
  left in place, never deleted
- rawInputTokens includes image tokens (image bytes ride every re-send;
  an image-heavy idle session must not look cheap to the sweep)
@ranxianglei

Copy link
Copy Markdown
Owner Author

@ework — owner decision on the default: flip zstd to opt-in. Default storage stays plain JSON (no compression, no encryption). Compression+encryption is for relay/shared deployments (中转站) that opt in explicitly.

Rationale:

  • Plain JSON maximizes recoverability: jq/grep-debuggable, and immune to the downgrade tail risk (old bili reading a BILIZSTD1 file as corrupt → empty resume → overwrite destroys history — the risk the review already shrank once; default-off removes it for everyone who never asked for compression).
  • The benefit (disk) is silent and unfelt for typical users; the complexity it drags in was the source of 3 of the 4 review findings (boot migration, GC no-op, tiny-file growth).
  • Consistent with BILI_ENCRYPTION_KEY being opt-in — same class of on-disk format change.

Exact change (keep everything else — size guard, organic conversion, BILIZSTD1 format, dual-mode readers):

  1. src/persist.ts persistZstdEnabled(): default false; enable only on BILI_PERSIST_ZSTD=1/true (treat 0/false/unset as off).
  2. src/encrypt.ts createStorageCodec default compress ?? true → follow the same opt-in default (only matters for callers that don't pass it — align to avoid a second silent default).
  3. Tests that currently assert compression with the env unset (e.g. tests/persist-zstd.test.ts deletes BILI_PERSIST_ZSTD and expects frames) → set BILI_PERSIST_ZSTD=1 explicitly; keep at least one test asserting the new default writes bare JSON.
  4. Docs flip: CONFIGURATION.md + CONFIGURATION.zh-CN.md BILI_PERSIST_ZSTD rows (default off, 1 to enable; downgrade caveat now applies only to opt-in users) + CHANGELOG.md entry wording.

This is a default-value/product decision per AGENTS.md §7.3 — owner-approved here, please implement on this branch and re-run gates.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,按 owner 决定执行:zstd 翻转为 opt-in(默认 plain JSON),BILI_PERSIST_ZSTD=1/true 才启用;size guard、有机转换、BILIZSTD1 格式、双模式读取全部保留。正在本分支实现四点改动(persistZstdEnabled 默认 false、codec 默认对齐、测试显式设 env + 保留默认裸 JSON 断言、文档/CHANGELOG 翻转),完成后重跑全部 gate 并汇报。

…JSON (#1083 owner decision)

Owner decision: default-on compression traded silent disk savings for
recoverability risk (jq/grep-debuggability lost, downgrade tail risk for
users who never asked for the format). Compression+encryption is for relay/
shared deployments that opt in explicitly — consistent with
BILI_ENCRYPTION_KEY being opt-in (same class of on-disk format change).

- persistZstdEnabled(): enable only on BILI_PERSIST_ZSTD=1/true; 0/false/
  unset all stay plain JSON.
- createStorageCodec(): compress ?? true -> ?? false, so no second silent
  default for callers that omit the option (no key + no opt-in -> undefined
  codec -> bare JSON, exactly the pre-#1080 behavior).
- Everything else kept per the decision: size guard, organic conversion,
  BILIZSTD1 format, dual-mode readers, never-rewrite-at-boot.
- Tests: store/codec cases that relied on the old default now set
  BILI_PERSIST_ZSTD=1 or pass compress explicitly; new test pins that the
  unset-env default writes bare JSON with no codec log; key-alone codec now
  asserts the raw-body mode byte.
- Docs flip: CONFIGURATION(.zh-CN).md BILI_PERSIST_ZSTD row (default off,
  1 enables, downgrade caveat scoped to opt-in users) + BILI_ENCRYPTION_KEY
  rows + CHANGELOG entry.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Opt-in flip implemented on this branch — commit 55723e2, pushed. All four points exactly as specified:

  1. src/persist.ts persistZstdEnabled() — now env === "1" || env === "true"; 0/false/unset all stay plain JSON. Doc comment records the owner decision (recoverability over silent disk savings).
  2. src/encrypt.ts createStorageCodec() — compress ?? true → ?? false, so there's no second silent default: no key + no opt-in → undefined codec → bare JSON, byte-for-byte the pre-[Feature] Use zstd compress by default #1080 behavior. (The only production caller, persist.ts, passes both options explicitly.)
  3. Tests — every store/codec case that relied on the old default now sets BILI_PERSIST_ZSTD=1 or passes compress: true explicitly; new test pins the new default: unset env → file on disk starts with {, loads back, no "compression enabled" log. The [Feature] Can bili context encrypt the data files it outputs? #708 mode-byte test now asserts both halves: key+explicit-compress → BILIENC1 zstd body (mode 0x01) and key-alone → raw body (mode 0x00). Kept per your spec: size guard, organic conversion, BILIZSTD1 format, dual-mode readers, never-rewrite-at-boot.
  4. Docs — CONFIGURATION.md + zh-CN BILI_PERSIST_ZSTD rows flipped (default off, 1/true enables, downgrade caveat scoped to opt-in users), BILI_ENCRYPTION_KEY rows in both languages corrected ("unset = plain JSON", "compressed only when opted in"), CHANGELOG entry rewritten to lead with the opt-in decision.

One semantic consequence worth naming explicitly (it follows directly from your spec, not something I added): with the codec default now off, a deployment that opts in, saves framed files, then unsets BILI_PERSIST_ZSTD again will read its own BILIZSTD1 files as corrupt until it re-enables or restores them — that's precisely the documented caveat, now confined to deployments that ever opted in. Nothing else changes for anyone who leaves the env unset: their files are identical to what bili ≤ 0.1.134 wrote.

Gates on 55723e2: typecheck clean · npm test 2179 pass / 0 fail / 2 skip · build success. CI is re-running on the new head — I'll confirm here when the matrix finishes.

中文摘要:已按 owner 决定把 zstd 翻转为 opt-in(默认纯 JSON,BILI_PERSIST_ZSTD=1 才启用),codec 默认值同步对齐、测试显式设 env 并新增"默认写裸 JSON"断言、中英文档与 CHANGELOG 全部翻转;size guard/有机转换/BILIZSTD1 格式/双模式读取全部保留。本地 gate 全绿(typecheck、2179 过 0 失败、构建成功),提交 55723e2 已推送,CI 跑完即汇报。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI follow-up — all 8 checks green on 55723e2 (opt-in flip):

  • test (ubuntu-latest, 22) ✅ · test (ubuntu-latest, 24) ✅
  • test (windows-latest, 22) ✅ · test (windows-latest, 24) ✅
  • real codex vs deterministic fake upstream (zero tokens) ✅
  • codex accepts CODEX_HOME overlay + registers bili MCP ✅
  • build-artifact ✅ · version-guard ✅

Owner decision fully implemented and verified end-to-end. Ready for you to merge whenever.

中文摘要:opt-in 翻转后的最终头 55723e2 上 CI 全部通过(含 Windows/Ubuntu × Node 22/24 测试矩阵),本地与远端验证一致,无遗留问题,可以合并。

@ranxianglei
ranxianglei merged commit baf6a53 into master Sep 22, 2026
8 checks passed
ranxianglei pushed a commit that referenced this pull request Sep 23, 2026
…le audit log (#1082)

Rebased onto master v0.1.139 (b2f28c6): adopt createStorageCodec({key})
in tests (codec rename from the #1083 era), union #1090 hasProcessedState
beside dropSessionForGc, keep master's updated BILI_ENCRYPTION_KEY doc
row (drop pre-#1083 wording). Squashed of the five prior commits.
@ranxianglei
ranxianglei deleted the 2026-09-21_persist-zstd-default branch October 15, 2026 05:32
cskwork pushed a commit to cskwork/billion-context that referenced this pull request Oct 21, 2026
- readRawFile is format-agnostic: plain JSON first, then codec decode —
  the sweep no longer silently no-ops on any future on-disk frame
  (zstd once ranxianglei#1083 lands); unknown frames decode-fail to null and are
  left in place, never deleted
- rawInputTokens includes image tokens (image bytes ride every re-send;
  an image-heavy idle session must not look cheap to the sweep)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Use zstd compress by default

1 participant