Repository navigation
[PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) - #4684
Merged
ranxianglei merged 2 commits intoOct 25, 2026
Merged
Conversation
…tion (#4677) v0.1.207 loaded every message's content bytes into memory in a single .all() inside the migration transaction, so multi-GB stores OOM'd on first open — and because the user_version bump sat in the same transaction, every subsequent open re-ran the whole migration and re-OMM'd: permanently unopenable stores. The v8 DDL (columns + index) still lands in the v1-v7 transaction, but the identity backfill now runs after it, streamed: sessions are picked by identity = '', rows flow through a 500-row seq-cursor page join, and each session commits in its own transaction. A crash between sessions leaves the done sessions committed with user_version still 7, so the next open skips them and resumes. Memory holds one page window + one session's keeper keys, never the store's bytes. Rows whose content bytes are missing from the CAS get the printable 'absent:' marker — no sha256 hex output can equal it (colon + short length), so their session terminates instead of rescanning forever. node:sqlite truncates bound strings at an embedded NUL, which ruled out the \u0000-shaped marker (it silently bound as '' and the row looked unmigrated forever). Dedup semantics unchanged from v0.1.207: same (msg_id, identity) after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back. Fixes #4677
This was referenced Oct 25, 2026
…4677) Completes the issue's fix direction 4 (progress logging) for the streamed identity backfill: a start line with the session count, a line every 100 sessions and at completion, and an error line on failure stating that committed sessions keep their progress and the migration resumes automatically on the next open — so an operator facing a slow or failing first open reads "migrating", not "hung", and does not hand-edit user_version. No behavior change: log lines only.
ranxianglei
commented
Oct 25, 2026
ranxianglei
left a comment
Owner
Author
There was a problem hiding this comment.
(ai: zhipuai-lb/glm-5.3)
Review 结论:改动本身可合(代码正确性已本地验证);CI 6 条红腿全部为车道基底继承,与本 PR 无关(双重验证见下)。 附两条运营级提醒。
代码审查(逐点核过)
- user_version 挪出 DDL 事务 + 崩溃重开幂等性 — 重开时
from=7会整段重跑 DDL 事务。查实 v8 的两个 ALTER 都有 probe 守卫(PRAGMA table_info(messages)探列名,v2 discipline,schema.ts:348-355)+CREATE INDEX IF NOT EXISTS→ 重跑幂等 ✓,续跑语义成立。 - 分页游标 —
WHERE ... AND m.seq > ? ORDER BY m.seq LIMIT 500,cursor = r.seq逐行推进,rows.length < 500终止:正确(顺序扫描下首达=最低 seq,简化掉旧代码的 keeper-swap 是对的;旧 swap 是为无序.all()设计的)。 - ABSENT 标记 —
identityOfStoredContent= sha256 hex(64 个 [0-9a-f],identity.ts:105-107),"absent:"(6 字符含冒号)不可能与之相等 ✓;LETF JOIN 缺字节的行从「永远重扫」(v0.1.207 INNER JOIN 直接漏掉)变为终止 ✓;注释里 node:sqlite 在 NUL 处截断绑定串的坑(\u0000 形状静默绑成 '')解释了为何用可打印前缀,好。 - 去重语义 = v0.1.207 结果 — 同 (msg_id, identity) 保最低 seq、后续 dup 还 media/行/引用(#4242 先删引用行纪律保留)✓;retired 墓碑行参与同序去重与旧行为一致,无回归。
- 内存边界 — 每次只持一个 500 行窗口(带 content 字节)+ 单 session 的 keeper Set + updates/drops 数组;session_id 清单只收字符串。OOM 路径确实消除。
- 一个非阻塞观察:
SELECT DISTINCT session_id FROM messages WHERE identity = ''命中不了idx_messages_identity(identity 是该索引第 3 列),pending 期间每次 open 一次全表扫 messages(不碰 content 字节,量级可接受);日后可用部分索引优化,不必现在。
验证
- 新测试
tests/issue4677-v8-migration-streaming.test.ts本地 5/5 绿(判别式与 body 声明一一对应:dedup 最低 seq+引用归还 / 崩溃续跑 stale-planted 不重算 / absent 二次 no-op / 跨 500 行页边界的去重组 / 空库)。 - 红腿继承性双重验证:① master 自己的 CI(ac844fa,11:31Z,不含本 PR)同家族失败(storage-render golden ×4、full-mode restart ×4、rollback、#2771 corrupt、HTTP fork revision、#1365 attach、orphan freeze);② 本地在车道基底 6c01ca7(不含本 PR 的 fix commit)单跑
tests/storage-engine-proofs.test.ts同样 6/12 红、同签名(serving golden-chat from the legacy envelope: last sampled digest drifted (#3135 ④)— #4392 shadow-digest 家族)。本 PR 的 fix commit 对该家族零增量。 - 本地首跑 storage-engine-proofs 曾因陈旧 kernel dist 在 import 处报
restampRefsToLedger缺失 — 重建 kernel 后消失,与 #4548 本地同型,非本 PR 问题。
运营提醒(不阻塞本 PR)
Owner
Author
|
报告审查结论先 |
1 similar comment
Owner
Author
|
报告审查结论先 |
This was referenced Oct 25, 2026
Merged
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(ai: zhipuai-lb/glm-5.3)
What broke (the bug, in one sentence): upgrading to v0.1.207 made large unified stores OOM on first open — the v8 migration loaded every message's content bytes into memory in a single query inside one transaction, and because the
user_versionbump sat in the same transaction, every retry re-ran the whole migration and re-OMM'd: the store became permanently unopenable (#4677, p1).The fix: the v8 DDL (columns + index) still lands atomically with v1-v7, but the identity backfill now runs AFTER it, streamed — sessions are picked by
identity = '', rows flow through a 500-rowseq-cursor page join, and each session commits in its own transaction:user_versionstill 7, so the next open skips them and resumes — no rollback-and-redo;absent:marker (no sha256 hex can equal it — it carries a:and is shorter than 64 hex chars), so their session terminates instead of being rescanned forever. A ``-shaped marker was tried first and is load-bearingly wrong: node:sqlite truncates bound strings at an embedded NUL, so it silently bound as''and the row looked unmigrated on every open;(msg_id, identity)after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back.Tests:
tests/issue4677-v8-migration-streaming.test.ts— 5 discriminators (backfill + wireIndex-split dedup keeps lowest seq + refcounts; crash resume never recomputes a done session; content-less rows terminate with the marker and a second migrateSchema is a no-op; dedup groups spanning the 500-row page boundary; empty store). Storage family + typecheck + build green on this head.Lane bookkeeping:
bugfix-v0.1.207— under the v3 gate-branch model merges into a lane NEVER publish (the lane was cut from thebugfixgate branch per the epoch-1 procedure, so it carries the v3 workflow, not the v2 wide trigger).bugfix-v0.1.207->bugfix, labelci:full) publishes v0.1.208.Fixes #4677
(supersedes #4680 — same head commit, branch renamed to the
backport/carry-*shape so version-guard's head-shape allowlist passes; the old branch name made that guard red on the PR.)