Skip to content

feat: encrypt session files at rest behind BILI_ENCRYPTION_KEY (#708) - #711

Merged
ranxianglei merged 8 commits into
masterfrom
2026-09-11_persist-encryption
Sep 13, 2026
Merged

ranxianglei merged 8 commits into
masterfrom
2026-09-11_persist-encryption

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Session-file encryption at rest (#708) for deployments on untrusted nodes:

  • BILI_ENCRYPTION_KEY (env-only; hex or base64, exactly 32 bytes) turns on encryption for every persisted session file. Unset = today's plaintext behavior, zero change.
  • Format: BILIENC1 magic + version + mode byte + random 12-byte nonce + AES-256-GCM ciphertext (auth tag appended). The body is zstd-compressed BEFORE encryption where Node >= 22.15 provides zlib.zstdCompressSync (raw mode byte on older runtimes) — compression also shrinks files ~5-10x, addressing the size concern raised in the thread.
  • Key handling: env-only — deliberately NO key-file support, because a key file would sit on the same untrusted filesystem as the data. Key material never touches disk or logs. An invalid key value aborts startup (fail fast instead of silently running unencrypted).
  • Legacy takeover (requested in-thread): when enabled, boot re-encodes every unencoded .json under the sessions dir ONCE — temp write + rename onto the SAME path, so the atomic replace IS the old-file deletion. A crash mid-migration self-heals on the next boot (the 8-byte magic peek decides per file); unreadable files are left in place with a warning. Spill .fb.json files are covered too.
  • Wrong key at boot -> GCM auth failure -> affected sessions skip as corrupt (logged, no crash).
  • Mechanism lives in the kernel: optional codec hook on StateStore (ranxianglei/acp-kernel#249); this repo stays policy-only (key parse, format, migration, docs).

Blocked on acp-kernel 0.0.64

The codec hook ships in acp-kernel 0.0.64 (PR ranxianglei/acp-kernel#249, awaiting merge). Until it is published this branch pins 0.0.63, so CI will be red at typecheck (StateStoreCodec not exported by 0.0.63). The follow-up commit bumping the exact pin + lockfile lands as soon as 0.0.64 is live on npm, with a full pre-flight re-run.

Pre-flight (local, against a local build of the #249 kernel)

  • npm run typecheck — clean
  • npm test — 1376 pass / 0 fail / 2 e2e-skipped (new tests/persist-encryption.test.ts: key parsing, roundtrip, per-write nonce, tamper + wrong-key rejection, legacy passthrough, encrypted writes, eager boot migration + idempotent second boot, spill coverage, corrupt-file tolerance, invalid-key fail-fast, no-key zero change)
  • npm run build — OK

Docs: CONFIGURATION.md + CONFIGURATION.zh-CN.md env-var tables.

Fixes #708

AES-256-GCM over zstd-compressed JSON (BILIENC1 format), wired through the
kernel StateStore codec hook (acp-kernel#249). Key comes only from the
environment variable; invalid values fail fast at startup. Boot takes over
legacy plaintext files once: each is re-encoded and atomically replaced in
place (rename IS the old-file deletion; crash mid-run self-heals). Off by
default — unset key = today's plaintext behavior.

Requires acp-kernel >= 0.0.64 (codec hook); pin bump + lockfile follow in a
separate commit once 0.0.64 is published.
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-11_persist-encryption (5dd8c7d)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-711

Each push to this PR publishes a new version under the pr-711 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr711.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Status check — still blocked on acp-kernel 0.0.64, nothing new to push yet, but the branch is re-verified healthy:

Verified just now at branch head 29d0403a:

  • acp-kernel on npm is still 0.0.63; acp-kernel PR ranxianglei/acp-kernel#249 is still open/unmerged (mergeable_state: clean; the kernel repo has no CI runs attached to the head commit, so nothing is failing — it's purely awaiting a human merge).
  • Re-ran the full local pre-flight of this branch against a fresh local build of the Fix Codex Responses session continuity #249 kernel (npm pack + swap into node_modules, tracked files untouched, confirmed via git status):
    • npm run typecheck — clean
    • npm test — 1376 pass / 0 fail / 2 e2e-skipped (matches the numbers in the description)
    • npm run build — OK
  • Kernel-side sanity on the Fix Codex Responses session continuity #249 branch: its own suite passes 651/0; StateStoreCodec is exported from acp-kernel/persist and the codec is applied around both canonical and spill writes (src/persist/store.ts:468 encode / :549 decode).
  • Branch is up to date with master (master is an ancestor of head).

So the branch is ready; the only remaining step is external: merge acp-kernel#249 → CI publishes 0.0.64. Once npm view acp-kernel version shows 0.0.64, I'll land the exact-pin bump + lockfile commit on this branch and re-run the full pre-flight.

One minor observation (non-blocking, reported only per policy): if the process dies between the temp write and the rename inside migrateLegacyFiles (src/persist.ts), the <file>.json.tmp-enc-<pid>-<ts> temp is left behind permanently — the migration walker only picks up *.json names, so later boots never reclaim it. Cosmetic garbage only (the original file is re-migrated correctly on the next boot); happy to add a sweep of stale .tmp-enc-* names during migration if you want it clean.

中文摘要:分支已用本地构建的 #249 kernel 全量复验通过(typecheck 干净、测试 1376/0、build OK),代码就绪;目前仍卡在 acp-kernel#249 未合并(npm 上还是 0.0.63),需人工合并并发布 0.0.64 后我再提交 pin 升级并重跑预检——当前尚不可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Follow-up: rather than wait for your call on the minor item above, I fixed it directly on the branch per review policy — commit 0f9c3f1 (fix: sweep orphaned .tmp-enc-* temps left by a crashed encryption migration (#708)), now the PR head:

  • src/persist.ts: the migration walker also collects <file>.tmp-enc-<pid>-<ts> names; any such file present at boot is stale by definition (the walk runs before this boot writes anything) and gets removed before the re-encode loop. This makes the crash-mid-migration path fully self-healing, matching the "self-heals on the next boot" claim in the description — previously the orphaned temp sat there forever since the walker only picked up *.json.
  • tests/persist-encryption.test.ts: new regression test simulating a process death between temp write and rename (stale temp + unencoded legacy file) and asserting the next boot re-encodes the session AND sweeps the orphan.

Re-ran full pre-flight at this head against the local #249 kernel build: typecheck clean, 1377 pass / 0 fail / 2 e2e-skipped, build OK. Still waiting on acp-kernel#249 merge → 0.0.64 publish for the pin bump.

中文摘要:直接在 PR 分支补了一个小修复(commit 0f9c3f1)——迁移时清扫崩溃残留的 .tmp-enc-* 临时文件,使"下次启动自愈"名副其实,并加了回归测试;预检全绿(typecheck 干净、1377/0、build OK)。当前仍被 acp-kernel#249 未合并卡住,待 0.0.64 发布后提交 pin 升级即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

ci失败

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Investigated the CI failure — it's the expected, documented red, not a new regression.

What failed: all 4 matrix jobs (ubuntu-latest × node 22/24, windows-latest × node 22/24) fail at the npm run typecheck step; test + build are skipped. Same 3 errors on every job:

src/encrypt.ts(3,15): error TS2724: '"acp-kernel/persist"' has no exported member named 'StateStoreCodec'
src/persist.ts(5,68): error TS2724: '"acp-kernel/persist"' has no exported member named 'StateStoreCodec'
src/persist.ts(221,13): error TS2353: Object literal may only specify known properties, and 'codec' does not exist in type 'StateStoreOptions<PersistedSession>'

Why: CI runs npm install against the pinned acp-kernel 0.0.63, which predates the codec hook. The StateStoreCodec / StateStoreOptions.codec surface ships in acp-kernel 0.0.64 (ranxianglei/acp-kernel#249). This is precisely the "Blocked on acp-kernel 0.0.64" state called out in the PR description — the branch intentionally pins 0.0.63 until 0.0.64 is published, so typecheck is red by design.

Verified the code itself is healthy: I ran the full pre-flight locally against a local build of the #249 kernel — typecheck clean, npm test 1377 pass / 0 fail / 2 e2e-skipped, build OK. So the source is correct; the red is purely the unpublished dependency.

Note: the newest commit on the branch (85c3121, docs-only threat-model clarification from the #708 discussion) does not affect this — it touches only CONFIGURATION docs.

To turn CI green (the only remaining step, and it's human-gated):

  1. Merge ranxianglei/acp-kernel#249 → its CI publishes acp-kernel 0.0.64 to npm.
  2. I confirm npm view acp-kernel version = 0.0.64.
  3. I land the follow-up commit bumping the exact pin ("acp-kernel": "0.0.64") + refreshed package-lock.json, re-run full pre-flight, push.

I can't merge the kernel PR myself (PR merges are human-only per repo rules), so step 1 is the blocker. Ping me as soon as #249 is merged and 0.0.64 is live and I'll do steps 2–3 immediately.

中文摘要:CI 失败是预期内的红色——4 个矩阵任务都卡在 typecheck,因为分支按设计仍锁定 acp-kernel 0.0.63(尚无 StateStoreCodec),而该 hook 要等 0.0.64(acp-kernel#249)发布;代码本身已用本地 #249 内核构建验证全绿(typecheck 干净、1377 测试通过、构建 OK)。唯一卡点是人工合并 acp-kernel#249 发布 0.0.64,之后我再补精确 pin + lockfile 提交并重跑预检即可转绿。

@ranxianglei

Copy link
Copy Markdown
Owner Author

解决ci失败问题

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 I pulled the actual published npm artifacts apart to find a way to turn CI green. Important correction to my previous message: bumping the pin to 0.0.64 will NOT fix it.

Verified by inspecting the real tarballs' .d.ts (which is exactly what tsc resolves):

acp-kernel version StateStoreCodec in persist/*.d.ts codec?: opt Hook present
0.0.64 (current latest) ❌ absent ❌ absent No
0.0.63 (current pin) ❌ absent ❌ absent No
0.0.63-pr.249.95 (per-PR prerelease) ✅ index.d.ts + store.d.ts ✅ store.d.ts:87 Yes

Cross-checked against source too: StateStoreCodec lives only on #249's branch (src/persist/store.ts:40,96) — not on master (GitHub code-search: 0 hits repo-wide). So 0.0.64 was cut from master before #249 landed; it simply doesn't carry the codec hook.

Consequence: no stable published acp-kernel satisfies this branch's typecheck. I can't reach a shippable green by bumping the pin alone — the root blocker is still the human-only merge of acp-kernel#249, which I'm not permitted to merge.

Two ways to get CI green — your call:

  • Path A (recommended — clean & shippable): You merge acp-kernel#249. Its CI cuts the next stable release (likely 0.0.65, since 0.0.64 is already consumed). The moment it's on npm I immediately land the exact-pin bump + refreshed package-lock.json, re-run full pre-flight (typecheck / test / build), and push → CI green, reproducible, ready to merge. Your only action is the merge.
  • Path B (immediate stopgap — not what I'd ship): I pin the branch to the per-PR prerelease 0.0.63-pr.249.95 right now → CI goes green in a single commit, no waiting. Caveat: it's a -pr. artifact (semver-sorts below the current 0.0.63) tied to a per-PR CI build — fine as a temporary green, but we must swap it to a stable release before this PR merges, or end users get a prerelease kernel bundled. Say the word and I'll push it.

My recommendation is Path A — it's the only route that ends in a clean, shippable pin, and it needs just that one merge from you. If you need the check green immediately for other reasons, I'll do Path B now.

中文摘要:我把 npm 上真实的 acp-kernel 包拆开核对了 .d.ts——刚发布的 0.0.64 其实不含 codec hook(persist 声明文件里没有 StateStoreCodec/codec?),它是在 #249 合并前从 master 切的,所以把 pin 升到 0.0.64 并不能让 CI 转绿;该 hook 目前只存在于未合并的 acp-kernel#249 分支及其 per-PR 预发布版 0.0.63-pr.249.95。因此没有任何“稳定版”能满足本分支 typecheck,根因仍是需人工合并 acp-kernel#249(我无权代合)。两条路:A(推荐)你合并 #249 → CI 切出下一个稳定版(约 0.0.65),我立刻补精确 pin + lockfile 并重跑全量预检转绿;B(临时)我现在就把分支锁到预发布版 0.0.63-pr.249.95 让 CI 立刻变绿,但本 PR 合并前必须换回稳定版。你拍板即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

ci失败

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI failure resolved — fix is pushed, verifying the new run now.

What changed since my last message: acp-kernel#249 was merged today 12:00 UTC, and two kernel releases followed — 0.0.65 (12:24 UTC) and 0.0.66 (14:02 UTC). I verified both by unpacking the npm tarballs: each carries StateStoreCodec + codec?: in dist/persist/store.d.ts.

Why the latest red run (77dbd06) still failed: that head had merged master only up to the point where master pinned kernel 0.0.64 (chore(#723)) — which, as established, was cut before #249 landed. Same StateStoreCodec typecheck error.

What I did (commit 5dd8c7d, pushed): merged current master into the branch. That pull-in matters twice over:

  • Master has since bumped its own pin to acp-kernel 0.0.66 (pin acp-kernel 0.0.66 (packs module live on npm)), because the new promptPack feature depends on 0.0.66's packs module. Pinning this branch to 0.0.65 instead would have downgraded master's kernel when the PR merges and broken promptPack — so 0.0.66 is the correct target.
  • The merge auto-resolved package.json + package-lock.json to exactly 0.0.66 — no separate pin-bump commit needed.

Full pre-flight locally, clean npm ci from the committed lockfile (real published 0.0.66, no local-build swaps):

  • installed kernel verified: 0.0.66, StateStoreCodec present in its .d.ts
  • npm run typecheck — clean
  • npm test — 1439 pass / 0 fail / 2 e2e-skipped
  • npm run build — OK

The fresh CI run on 5dd8c7d should be green across all four matrix jobs; I'll confirm as soon as it lands.

中文摘要:CI 已修复并推送(5dd8c7d)。原因:上一轮红是因为分支合入的 master 还锁着 acp-kernel 0.0.64(在 #249 合并前发布,不含 codec hook);现在 #249 已合并、内核发布了 0.0.65/0.0.66(均含 hook,已拆包核实)。我把最新 master 合进分支——master 自身已把 pin 升到 0.0.66(新的 promptPack 功能依赖 0.0.66 的 packs 模块,若只锁 0.0.65 会在合并时把 master 的内核降级弄坏 promptPack),合并后 package.json/lockfile 自动落到精确的 0.0.66,无需单独 bump 提交。本地用 lockfile 干净 npm ci 装真实发布的 0.0.66 全量预检:typecheck 干净、1439 通过/0 失败/2 e2e 跳过、构建 OK。新 CI 跑完我确认结果。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 ✅ Confirmed: CI is green on 5dd8c7d — the main CI workflow (all 4 matrix jobs: ubuntu/windows × node 22/24) completed success, plus e2e-fake-codex and PR Build Artifact both pass.

The branch is up to date with master, pinned to acp-kernel 0.0.66 (exact, lockfile-synced, contains the codec hook), and fully pre-flighted locally against the real published artifact. Nothing left on my side — ready for your review/merge whenever you are.

中文摘要:已确认新提交 5dd8c7d 上 CI 全绿(主 CI 四个矩阵任务全部通过,e2e-fake-codex 与 PR Build Artifact 也通过);分支已与 master 同步、精确锁定 acp-kernel 0.0.66(含 codec hook、lockfile 一致),本地已用真实发布版全量预检。我这边没有遗留事项,随时可以 review/合并。

@ranxianglei
ranxianglei merged commit 1952fc9 into master Sep 13, 2026
7 checks passed
@ranxianglei
ranxianglei deleted the 2026-09-11_persist-encryption branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Can bili context encrypt the data files it outputs?

1 participant