Skip to content

[PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) - #4684

Merged
ranxianglei merged 2 commits into
bugfix-v0.1.207from
backport/carry-v8-migration-streaming
Oct 25, 2026
Merged

ranxianglei merged 2 commits into
bugfix-v0.1.207from
backport/carry-v8-migration-streaming

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: zhipuai-lb/glm-5.3)

What broke (the bug, in one sentence): upgrading to v0.1.207 made large unified stores OOM on first open — the v8 migration loaded every message's content bytes into memory in a single query inside one transaction, and because the user_version bump sat in the same transaction, every retry re-ran the whole migration and re-OMM'd: the store became permanently unopenable (#4677, p1).

The fix: the v8 DDL (columns + index) still lands atomically with v1-v7, but the identity backfill now runs AFTER it, streamed — sessions are picked by identity = '', rows flow through a 500-row seq-cursor page join, and each session commits in its own transaction:

  • memory holds one page window + one session's keeper keys, never the store's bytes;
  • a crash between sessions leaves the done sessions committed with user_version still 7, so the next open skips them and resumes — no rollback-and-redo;
  • rows whose content bytes are missing from the CAS get the printable absent: marker (no sha256 hex can equal it — it carries a : and is shorter than 64 hex chars), so their session terminates instead of being rescanned forever. A ``-shaped marker was tried first and is load-bearingly wrong: node:sqlite truncates bound strings at an embedded NUL, so it silently bound as '' and the row looked unmigrated on every open;
  • dedup semantics unchanged from v0.1.207: same (msg_id, identity) after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back.

Tests: tests/issue4677-v8-migration-streaming.test.ts — 5 discriminators (backfill + wireIndex-split dedup keeps lowest seq + refcounts; crash resume never recomputes a done session; content-less rows terminate with the marker and a second migrateSchema is a no-op; dedup groups spanning the 500-row page boundary; empty store). Storage family + typecheck + build green on this head.

Lane bookkeeping:

  • carry PR into lane bugfix-v0.1.207 — under the v3 gate-branch model merges into a lane NEVER publish (the lane was cut from the bugfix gate branch per the epoch-1 procedure, so it carries the v3 workflow, not the v2 wide trigger).
  • after this merges, the ship PR (bugfix-v0.1.207 -> bugfix, label ci:full) publishes v0.1.208.
  • merging alignment PR [PR] align master with v0.1.207 (shipped from bugfix-v0.1.206) #4673 first is still safe: the P4 auto-cut no-ops on the existing branch.

Fixes #4677

(supersedes #4680 — same head commit, branch renamed to the backport/carry-* shape so version-guard's head-shape allowlist passes; the old branch name made that guard red on the PR.)

…tion (#4677)

v0.1.207 loaded every message's content bytes into memory in a single
.all() inside the migration transaction, so multi-GB stores OOM'd on
first open — and because the user_version bump sat in the same
transaction, every subsequent open re-ran the whole migration and
re-OMM'd: permanently unopenable stores.

The v8 DDL (columns + index) still lands in the v1-v7 transaction, but
the identity backfill now runs after it, streamed: sessions are picked
by identity = '', rows flow through a 500-row seq-cursor page join, and
each session commits in its own transaction. A crash between sessions
leaves the done sessions committed with user_version still 7, so the
next open skips them and resumes. Memory holds one page window + one
session's keeper keys, never the store's bytes.

Rows whose content bytes are missing from the CAS get the printable
'absent:' marker — no sha256 hex output can equal it (colon + short
length), so their session terminates instead of rescanning forever.
node:sqlite truncates bound strings at an embedded NUL, which ruled out
the \u0000-shaped marker (it silently bound as '' and the row looked
unmigrated forever).

Dedup semantics unchanged from v0.1.207: same (msg_id, identity) after
the strip keeps the lowest-seq row; later duplicates hand their media
mappings, message rows and CAS refs back.

Fixes #4677
…4677)

Completes the issue's fix direction 4 (progress logging) for the
streamed identity backfill: a start line with the session count, a
line every 100 sessions and at completion, and an error line on
failure stating that committed sessions keep their progress and the
migration resumes automatically on the next open — so an operator
facing a slow or failing first open reads "migrating", not "hung",
and does not hand-edit user_version.

No behavior change: log lines only.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: zhipuai-lb/glm-5.3)

Review 结论:改动本身可合(代码正确性已本地验证);CI 6 条红腿全部为车道基底继承,与本 PR 无关(双重验证见下)。 附两条运营级提醒。

代码审查(逐点核过)

  1. user_version 挪出 DDL 事务 + 崩溃重开幂等性 — 重开时 from=7 会整段重跑 DDL 事务。查实 v8 的两个 ALTER 都有 probe 守卫(PRAGMA table_info(messages) 探列名,v2 discipline,schema.ts:348-355)+ CREATE INDEX IF NOT EXISTS → 重跑幂等 ✓,续跑语义成立。
  2. 分页游标 — WHERE ... AND m.seq > ? ORDER BY m.seq LIMIT 500,cursor = r.seq 逐行推进,rows.length < 500 终止:正确(顺序扫描下首达=最低 seq,简化掉旧代码的 keeper-swap 是对的;旧 swap 是为无序 .all() 设计的)。
  3. ABSENT 标记 — identityOfStoredContent = sha256 hex(64 个 [0-9a-f],identity.ts:105-107),"absent:"(6 字符含冒号)不可能与之相等 ✓;LETF JOIN 缺字节的行从「永远重扫」(v0.1.207 INNER JOIN 直接漏掉)变为终止 ✓;注释里 node:sqlite 在 NUL 处截断绑定串的坑(\u0000 形状静默绑成 '')解释了为何用可打印前缀,好。
  4. 去重语义 = v0.1.207 结果 — 同 (msg_id, identity) 保最低 seq、后续 dup 还 media/行/引用(#4242 先删引用行纪律保留)✓;retired 墓碑行参与同序去重与旧行为一致,无回归。
  5. 内存边界 — 每次只持一个 500 行窗口(带 content 字节)+ 单 session 的 keeper Set + updates/drops 数组;session_id 清单只收字符串。OOM 路径确实消除。
  6. 一个非阻塞观察:SELECT DISTINCT session_id FROM messages WHERE identity = '' 命中不了 idx_messages_identity(identity 是该索引第 3 列),pending 期间每次 open 一次全表扫 messages(不碰 content 字节,量级可接受);日后可用部分索引优化,不必现在。

验证

  • 新测试 tests/issue4677-v8-migration-streaming.test.ts 本地 5/5 绿(判别式与 body 声明一一对应:dedup 最低 seq+引用归还 / 崩溃续跑 stale-planted 不重算 / absent 二次 no-op / 跨 500 行页边界的去重组 / 空库)。
  • 红腿继承性双重验证:① master 自己的 CI(ac844fa,11:31Z,不含本 PR)同家族失败(storage-render golden ×4、full-mode restart ×4、rollback、#2771 corrupt、HTTP fork revision、#1365 attach、orphan freeze);② 本地在车道基底 6c01ca7(不含本 PR 的 fix commit)单跑 tests/storage-engine-proofs.test.ts 同样 6/12 红、同签名(serving golden-chat from the legacy envelope: last sampled digest drifted (#3135 ④) — #4392 shadow-digest 家族)。本 PR 的 fix commit 对该家族零增量。
  • 本地首跑 storage-engine-proofs 曾因陈旧 kernel dist 在 import 处报 restampRefsToLedger 缺失 — 重建 kernel 后消失,与 #4548 本地同型,非本 PR 问题。

运营提醒(不阻塞本 PR)

  1. 发车前的红矩阵:digest-drift 家族(#4392)在 master 已有打开的修复 PR #4685(digest key 排序)。若它不合、车道也不带,ship PR(v0.1.208)会带着这族已知红出库 — 鉴于 #4677 是 p1(老库永久打不开),应急发版可接受,但建议把 #4685 也 carry 上车道或紧随其后。
  2. flake 单例(HTTP fork revision / #1365 attach / orphan freeze)在 master 自身 run 也出现,属既有 flake 面,与本 PR 无关。

@ranxianglei

Copy link
Copy Markdown
Owner Author

报告审查结论先

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner Author

报告审查结论先

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant