Repository navigation
feat: zstd-compress session files by default, decoupled from encryption (#1080) - #1083
Conversation
📦 Built Package ArtifactBranch: Option A — Install from npm PR tag (recommended)npm install -g billion-context@pr-1083Each push to this PR publishes a new version under the Option B — Download artifact
tar xzf billion-context-pr1083.tgz
npm install -g packageInstall instructions are refreshed on each push. |
b67074f to
4d27208
Compare
[bot] 🏷 Reviewed — verified on rebased head, ready for human merge. I did not redo the work; I pulled the branch, audited the diff, and re-verified against live master. Code audit (diff vs master)
Stale base — fixed Preflight on rebased head
CI: on the old head, build-artifact, version-guard, e2e zero-token preflight, and the codex-overlay job were green while the 4× test-matrix jobs were still running; the push above supersedes them and CI will re-run on This is a 中文摘要:代码审查通过、无问题;PR 落后最新 master(含 acp-kernel 0.0.82 pin)已 rebase 并把分支更新为 |
[bot] 🏷 CI follow-up — all 8 checks green on rebased head
Nothing outstanding — ready for you to merge whenever. 中文摘要:rebase 后的新提交 |
ranxianglei
left a comment
There was a problem hiding this comment.
审查结论:NEEDS-HUMAN + fix-first(今天不能合)
实作质量整体扎实——magic 分发向后兼容(旧明文/BILIENC1 全部可读)、加密与压缩解耦有回归测试、kernel writeNow tmp+rename 原子性未动、fzstd 纯 JS 已在 master 的 tsup.config.ts noExternal 里(dist 保持自包含,全局安装安全)。但有一个 HIGH 必须先决策:
🔴 HIGH:降级回滚 = 静默销毁会话(默认未加密路径,无人承认此风险)
链路:boot 迁移把所有未加密文件改写为 BILIZSTD1(默认路径=大多数用户)→ 手动降回 0.1.134(标准调试/逃生动作)→ 旧版 src/encrypt.ts:87 对非 BILIENC1 一律 buf.toString("utf8") → JSON.parse 抛 → kernel [persist] skipping corrupt file(acp-kernel persist/index.js:358-362,报错不含格式标记提示)→ loadSync 返回 null → resume 该会话 id 铸出全新空会话 → 下一次 writeNow 覆盖 BILIZSTD1 文件 → 原历史被销毁,不只是读不了。
旧版本侧无法识别 magic(不可能给旧码加探测),所以缓解必须在新代码侧三选一:
- 明文保持默认,zstd 先 opt-in 过渡一段时间(最保守);
- 迁移加门控(如显式 env 才执行 boot 全量改写);
- owner 明确接受降级即丢数据的风险并写进 CONFIGURATION.md + PR body(现在三处都没提)。
持久化格式变更按仓库规范必须人审拍板(AGENTS §7.4),这一条就是为什么。
小项
- [Low] encode 无尺寸守卫(
src/encrypt.ts:125):实测 71B 会话 JSON → 91B(+28%,header+帧开销)。真实会话 ~45× 缩(60,290→1,322B),聚合影响可忽略,但 "压缩后 ≥ 原文则保原样" 一行守卫更整洁。 - [Process]
mergeable_state: behind(base 7cec6d2 < 9c01dbe,间隔只有 #1087 revert + release commit,无文件交叠)——合并前 rebase + 全量门禁重跑。注意:与 #1084 有跨 PR 语义冲突,那边readRawFile只认BILIENC1,两 PR 谁后合谁要适配(详见 #1084 评论)。
- readRawFile is format-agnostic: plain JSON first, then codec decode — the sweep no longer silently no-ops on any future on-disk frame (zstd once #1083 lands); unknown frames decode-fail to null and are left in place, never deleted - rawInputTokens includes image tokens (image bytes ride every re-send; an image-heavy idle session must not look cheap to the sweep)
…ression wins (#1083 review) - drop migrateLegacyFiles: the boot-time mass re-encode made a downgrade (rollback during incident triage) read its own writes as corrupt files, resume into empty sessions and overwrite real history on next save. Legacy plaintext keeps loading via the codec's magic dispatch and converts organically on its next save; boot only sweeps orphaned .tmp-enc-* temps from crashed writes. - encode() size guard: frame-and-compress only when zstd actually shrinks the payload; key-less raw bodies and tiny sessions stay bare JSON on disk (maximum backward compatibility — older bili reads them natively). - tests: rewrite the three boot-migration tests to the new contract (never-rewrite + organic conversion + mixed-format tree loads), size guard coverage at codec level, no-native-zstd path asserts bare JSON. - docs: CONFIGURATION(.zh-CN).md describe the never-rewrite contract and the residual downgrade window (BILIZSTD1 files need bili >= 0.1.135; set BILI_PERSIST_ZSTD=0 for downgrade headroom).
|
Follow-up on the earlier review (#1083 (comment) level findings): fixed on this branch in 69b341f, gates green (typecheck / 2188 tests / build). 1. Boot-time mass migration removed (
Why this matters: the draft migration re-encoded every plaintext file into frames an older bili (≤ 0.1.134) reads as 2. Frame only when compression wins (
3. Tests & docs
Residual (documented, not fixed here): once a session is saved as |
…ression wins (#1083 review) - drop migrateLegacyFiles: the boot-time mass re-encode made a downgrade (rollback during incident triage) read its own writes as corrupt files, resume into empty sessions and overwrite real history on next save. Legacy plaintext keeps loading via the codec's magic dispatch and converts organically on its next save; boot only sweeps orphaned .tmp-enc-* temps from crashed writes. - encode() size guard: frame-and-compress only when zstd actually shrinks the payload; key-less raw bodies and tiny sessions stay bare JSON on disk (maximum backward compatibility — older bili reads them natively). - tests: rewrite the three boot-migration tests to the new contract (never-rewrite + organic conversion + mixed-format tree loads), size guard coverage at codec level, no-native-zstd path asserts bare JSON. - docs: CONFIGURATION(.zh-CN).md describe the never-rewrite contract and the residual downgrade window (BILIZSTD1 files need bili >= 0.1.135; set BILI_PERSIST_ZSTD=0 for downgrade headroom).
69b341f to
5945e8d
Compare
[bot] 🏷 Verified Code audit of the fix commit
Residual window: agreed it's acceptable as documented — it only opens when a session has grown enough that zstd wins, and Rebase + preflight: master moved while this was up (release v0.1.134 merged, #1079 chain-sig reverted). Rebased both commits cleanly onto CI is re-running on 中文摘要:已逐行核实 |
|
@ework — the head has moved to 5945e8d (rebased onto current master + the review fix cherry-picked; the fix comment is directly above). Your last review covered the pre-fix c5a5ab9. Please re-review the current head in full: the removed boot migration, the size-guard framing in |
) The review fix removed migrateLegacyFiles but two prose references to the old boot-time mass re-encode survived: a comment in encrypt.ts and the CHANGELOG entry for #1080. Both now describe the actual contract: legacy plain-JSON files are never rewritten at boot and convert organically on their next save.
[bot] 🏷 Full re-review of current head done — this pass read the final state of both core files and both test files end-to-end (not just the incremental fix diff), plus docs. Verdict: correct as shipped, with one small doc residue I fixed directly. 1. Removed boot migration ( 2. Size-guard framing ( 3. Rewritten persistence tests — verified, both files read in full
4. Docs — accurate, one residue fixed by me CI is re-running on 中文摘要:对当前头做了完整复审(通读最终代码与两份测试文件,非仅看增量 diff)——移除启动期批量迁移、encode 尺寸守卫、重写的持久化测试三项全部核实无误,降级安全契约有端到端断言覆盖;顺手直接修掉了两处遗留的旧"启动迁移"文档表述(src/encrypt.ts 注释 + CHANGELOG,提交 |
[bot] 🏷 CI follow-up — all 8 checks green on final head
Full re-review complete, doc residue fixed, everything verified — ready for you to merge whenever. 中文摘要:最终头 |
- readRawFile is format-agnostic: plain JSON first, then codec decode — the sweep no longer silently no-ops on any future on-disk frame (zstd once #1083 lands); unknown frames decode-fail to null and are left in place, never deleted - rawInputTokens includes image tokens (image bytes ride every re-send; an image-heavy idle session must not look cheap to the sweep)
|
@ework — owner decision on the default: flip zstd to opt-in. Default storage stays plain JSON (no compression, no encryption). Compression+encryption is for relay/shared deployments (中转站) that opt in explicitly. Rationale:
Exact change (keep everything else — size guard, organic conversion,
This is a default-value/product decision per AGENTS.md §7.3 — owner-approved here, please implement on this branch and re-run gates. |
[bot] 🏷 收到,按 owner 决定执行:zstd 翻转为 opt-in(默认 plain JSON), |
…JSON (#1083 owner decision) Owner decision: default-on compression traded silent disk savings for recoverability risk (jq/grep-debuggability lost, downgrade tail risk for users who never asked for the format). Compression+encryption is for relay/ shared deployments that opt in explicitly — consistent with BILI_ENCRYPTION_KEY being opt-in (same class of on-disk format change). - persistZstdEnabled(): enable only on BILI_PERSIST_ZSTD=1/true; 0/false/ unset all stay plain JSON. - createStorageCodec(): compress ?? true -> ?? false, so no second silent default for callers that omit the option (no key + no opt-in -> undefined codec -> bare JSON, exactly the pre-#1080 behavior). - Everything else kept per the decision: size guard, organic conversion, BILIZSTD1 format, dual-mode readers, never-rewrite-at-boot. - Tests: store/codec cases that relied on the old default now set BILI_PERSIST_ZSTD=1 or pass compress explicitly; new test pins that the unset-env default writes bare JSON with no codec log; key-alone codec now asserts the raw-body mode byte. - Docs flip: CONFIGURATION(.zh-CN).md BILI_PERSIST_ZSTD row (default off, 1 enables, downgrade caveat scoped to opt-in users) + BILI_ENCRYPTION_KEY rows + CHANGELOG entry.
[bot] 🏷 Opt-in flip implemented on this branch — commit
One semantic consequence worth naming explicitly (it follows directly from your spec, not something I added): with the codec default now off, a deployment that opts in, saves framed files, then unsets Gates on 中文摘要:已按 owner 决定把 zstd 翻转为 opt-in(默认纯 JSON, |
[bot] 🏷 CI follow-up — all 8 checks green on
Owner decision fully implemented and verified end-to-end. Ready for you to merge whenever. 中文摘要:opt-in 翻转后的最终头 |
…le audit log (#1082) Rebased onto master v0.1.139 (b2f28c6): adopt createStorageCodec({key}) in tests (codec rename from the #1083 era), union #1090 hasProcessedState beside dropSessionForGc, keep master's updated BILI_ENCRYPTION_KEY doc row (drop pre-#1083 wording). Squashed of the five prior commits.
- readRawFile is format-agnostic: plain JSON first, then codec decode — the sweep no longer silently no-ops on any future on-disk frame (zstd once ranxianglei#1083 lands); unknown frames decode-fail to null and are left in place, never deleted - rawInputTokens includes image tokens (image bytes ride every re-send; an image-heavy idle session must not look cheap to the sweep)
Closes #1080
What changed
Session files are now zstd-compressed by default, and compression is fully decoupled from
BILI_ENCRYPTION_KEY:BILIZSTD1— 9-byte magic + format version (0x01) + mode byte (0x00raw /0x01zstd) + body. The body is zstd-compressed on Node ≥ 22.15 and stored raw on older runtimes; readers try native zstd first, then fall back to the already-bundledfzstdWASM decompressor, so files stay readable across runtimes/versions (no corruption in mixed-version trees).BILIENC1([Feature] Can bili context encrypt the data files it outputs? #708) stays byte-compatible. Its mode byte now records whether the body is compressed, andBILI_PERSIST_ZSTD=0applies inside it too — encryption and compression are independent knobs.BILI_PERSIST_ZSTD=0(fits the existingBILI_PERSIST*env family).BILIENC1files are never touched when no key is set (they cannot be decoded).createSessionCodec(key)→createStorageCodec({ key?, compress? }); no codec when neither is active (opt-out users behave exactly as before).bili export, the web UI, and decompress search need no changes — every read goes through the store codec.One deviation from the issue thread
The assessment comment suggested running migration "as a background pass after the listener starts". It stays at boot (awaited, exactly like #708): a post-listen pass does direct fs ops bypassing the kernel's per-session write chain, so an active session's save racing the rename could roll the file back to a stale parsed snapshot. Boot-time has no race window; the cost is a one-time bounded delay (walk+peek is O(files); only plain-JSON files get rewritten).
Tests
New
tests/persist-zstd.test.ts(14 tests): header/determinism, config matrix, plaintext passthrough + cross-tree dispatch, #708 mode-byte regression, static-fixture fzstd decode, store-level default-on, shrinkage vs opt-out baseline, opt-out keeps plain JSON, pre-22.15 raw fallback, cross-runtime read (native write → WASM decode), once-only migration, mixed tree, keyed boot leavesBILIZSTD1untouched, no-key boot reportsBILIENC1as unreadable with an actionable message, decoupled raw-body encryption. Four existing test files that inspect raw session files pinBILI_PERSIST_ZSTD=0.Preflight (this commit)
npm run typecheck— cleannpm test— 2097 pass / 0 fail / 2 skip (same skips as master)npm run build— successDocs: CONFIGURATION.md + CONFIGURATION.zh-CN.md (new row + rewritten
BILI_ENCRYPTION_KEYrow) and CHANGELOG[Unreleased].