Repository navigation
[PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) — master pick - #4708
Merged
Conversation
…tion (#4677) v0.1.207 loaded every message's content bytes into memory in a single .all() inside the migration transaction, so multi-GB stores OOM'd on first open — and because the user_version bump sat in the same transaction, every subsequent open re-ran the whole migration and re-OMM'd: permanently unopenable stores. The v8 DDL (columns + index) still lands in the v1-v7 transaction, but the identity backfill now runs after it, streamed: sessions are picked by identity = '', rows flow through a 500-row seq-cursor page join, and each session commits in its own transaction. A crash between sessions leaves the done sessions committed with user_version still 7, so the next open skips them and resumes. Memory holds one page window + one session's keeper keys, never the store's bytes. Rows whose content bytes are missing from the CAS get the printable 'absent:' marker — no sha256 hex output can equal it (colon + short length), so their session terminates instead of rescanning forever. node:sqlite truncates bound strings at an embedded NUL, which ruled out the \u0000-shaped marker (it silently bound as '' and the row looked unmigrated forever). Dedup semantics unchanged from v0.1.207: same (msg_id, identity) after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back. Fixes #4677 (cherry picked from commit 80f1bb1)
…4677) Completes the issue's fix direction 4 (progress logging) for the streamed identity backfill: a start line with the session count, a line every 100 sessions and at completion, and an error line on failure stating that committed sessions keep their progress and the migration resumes automatically on the next open — so an operator facing a slow or failing first open reads "migrating", not "hung", and does not hand-edit user_version. No behavior change: log lines only. (cherry picked from commit 316efcc)
Contributor
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5736-pr.4708.3d33a86This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4708.tgz
npm install -g packageInstall instructions are refreshed on each push. |
Contributor
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.838-master.99823b0 # this exact build
npm install -g billion-context@master # always the newest merged build
|
This was referenced Oct 26, 2026
ranxianglei
added a commit
that referenced
this pull request
Oct 27, 2026
The 0.1.208 carve-out lane shipped off the v0.1.207 tag (storage v8 emergency, #4677) without landing a version bump on master, so master's package.json (0.1.207) no longer equals npm latest (0.1.208) and the one-click release drift guard aborts (run 40520077248, step 'Drift guard'). Version-string sync ONLY: master already carries the 0.1.208 content under its own commits (#4708 for #4677, #4666). Plugin manifests synced via scripts/sync-codex-plugin-version.mjs (#4449) so the publish workflows' --check gate stays green. No code change; branch deliberately does NOT match *_release-v* so merging publishes nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(ai: zai/glm-5.3)
Closes #4677 (master side; the fix shipped to users in v0.1.208 via bugfix-release/v0.1.208-v8-migration, reviewed in #4684 — this PR lands the same two commits on master so the next master release carries them)
啥问题
v0.1.207 的 v8 迁移把全库 messages JOIN content 的字节一次性 .all() 进 Node 堆且整库单事务:几 GB 的老库首开即 OOM,失败回滚 user_version,下次启动重炸——老库永久打不开(#4677,p1)。
修复(与 #4684 同内容,cherry-pick -x 保留出处)
src/storage/schema.ts:user_version 挪出 DDL 事务(崩溃重开幂等,两个 ALTER 有 probe 守卫);identity 回填改为按 session 分事务、每事务 500 行窗口分页游标(seq > cursor,ORDER BY seq LIMIT 500);ABSENT_IDENTITY="absent:" 占位与 sha256 hex 不可能碰撞;同 (msg_id,identity) 保最低 seq 的去重语义与 v0.1.207 一致。tests/issue4677-v8-migration-streaming.test.ts(199 行)。验证
说明