Skip to content

[PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) — master pick - #4708

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-25_v8-migration-oom-master
Oct 25, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-25_v8-migration-oom-master

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: zai/glm-5.3)

Closes #4677 (master side; the fix shipped to users in v0.1.208 via bugfix-release/v0.1.208-v8-migration, reviewed in #4684 — this PR lands the same two commits on master so the next master release carries them)

啥问题

v0.1.207 的 v8 迁移把全库 messages JOIN content 的字节一次性 .all() 进 Node 堆且整库单事务:几 GB 的老库首开即 OOM,失败回滚 user_version,下次启动重炸——老库永久打不开(#4677,p1)。

修复(与 #4684 同内容,cherry-pick -x 保留出处)

  • src/storage/schema.ts:user_version 挪出 DDL 事务(崩溃重开幂等,两个 ALTER 有 probe 守卫);identity 回填改为按 session 分事务、每事务 500 行窗口分页游标(seq > cursor,ORDER BY seq LIMIT 500);ABSENT_IDENTITY="absent:" 占位与 sha256 hex 不可能碰撞;同 (msg_id,identity) 保最低 seq 的去重语义与 v0.1.207 一致。
  • 新增 tests/issue4677-v8-migration-streaming.test.ts(199 行)。

验证

说明

ranxianglei and others added 2 commits October 25, 2026 21:02
…tion (#4677)

v0.1.207 loaded every message's content bytes into memory in a single
.all() inside the migration transaction, so multi-GB stores OOM'd on
first open — and because the user_version bump sat in the same
transaction, every subsequent open re-ran the whole migration and
re-OMM'd: permanently unopenable stores.

The v8 DDL (columns + index) still lands in the v1-v7 transaction, but
the identity backfill now runs after it, streamed: sessions are picked
by identity = '', rows flow through a 500-row seq-cursor page join, and
each session commits in its own transaction. A crash between sessions
leaves the done sessions committed with user_version still 7, so the
next open skips them and resumes. Memory holds one page window + one
session's keeper keys, never the store's bytes.

Rows whose content bytes are missing from the CAS get the printable
'absent:' marker — no sha256 hex output can equal it (colon + short
length), so their session terminates instead of rescanning forever.
node:sqlite truncates bound strings at an embedded NUL, which ruled out
the \u0000-shaped marker (it silently bound as '' and the row looked
unmigrated forever).

Dedup semantics unchanged from v0.1.207: same (msg_id, identity) after
the strip keeps the lowest-seq row; later duplicates hand their media
mappings, message rows and CAS refs back.

Fixes #4677

(cherry picked from commit 80f1bb1)
…4677)

Completes the issue's fix direction 4 (progress logging) for the
streamed identity backfill: a start line with the session count, a
line every 100 sessions and at completion, and an error line on
failure stating that committed sessions keep their progress and the
migration resumes automatically on the next open — so an operator
facing a slow or failing first open reads "migrating", not "hung",
and does not hand-edit user_version.

No behavior change: log lines only.

(cherry picked from commit 316efcc)
@ranxianglei ranxianglei changed the title fix(storage): stream the v8 identity backfill one session per transaction (#4677) — master pick [PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) — master pick Oct 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_v8-migration-oom-master (af0ffdd)

Latest build — install this exact version

npm install -g billion-context@0.0.5736-pr.4708.3d33a86

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
af0ffdd billion-context@0.0.5736-pr.4708.3d33a86 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4708.tgz
npm install -g package

Install instructions are refreshed on each push.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.838-master.99823b0 (rolling master tag).

npm install -g billion-context@0.0.838-master.99823b0  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit that referenced this pull request Oct 27, 2026
The 0.1.208 carve-out lane shipped off the v0.1.207 tag (storage v8
emergency, #4677) without landing a version bump on master, so master's
package.json (0.1.207) no longer equals npm latest (0.1.208) and the
one-click release drift guard aborts (run 40520077248, step 'Drift
guard'). Version-string sync ONLY: master already carries the 0.1.208
content under its own commits (#4708 for #4677, #4666). Plugin manifests
synced via scripts/sync-codex-plugin-version.mjs (#4449) so the publish
workflows' --check gate stays green. No code change; branch deliberately
does NOT match *_release-v* so merging publishes nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant