Skip to content

[PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) - #4680

Closed
ranxianglei wants to merge 1 commit into
bugfix-v0.1.207from
2026-10-25_v8-migration-streaming
Closed

ranxianglei wants to merge 1 commit into
bugfix-v0.1.207from
2026-10-25_v8-migration-streaming

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: zhipuai-lb/glm-5.3)

What broke (the bug, in one sentence): upgrading to v0.1.207 made large unified stores OOM on first open — the v8 migration loaded every message's content bytes into memory in a single query inside one transaction, and because the user_version bump sat in the same transaction, every retry re-ran the whole migration and re-OMM'd: the store became permanently unopenable (#4677, p1).

The fix: the v8 DDL (columns + index) still lands atomically with v1-v7, but the identity backfill now runs AFTER it, streamed — sessions are picked by identity = '', rows flow through a 500-row seq-cursor page join, and each session commits in its own transaction:

  • memory holds one page window + one session's keeper keys, never the store's bytes;
  • a crash between sessions leaves the done sessions committed with user_version still 7, so the next open skips them and resumes — no rollback-and-redo;
  • rows whose content bytes are missing from the CAS get the printable absent: marker (no sha256 hex can equal it — it carries a : and is shorter than 64 hex chars), so their session terminates instead of being rescanned forever. A ``-shaped marker was tried first and is load-bearingly wrong: node:sqlite truncates bound strings at an embedded NUL, so it silently bound as '' and the row looked unmigrated on every open;
  • dedup semantics unchanged from v0.1.207: same (msg_id, identity) after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back.

Tests: tests/issue4677-v8-migration-streaming.test.ts — 5 discriminators (backfill + wireIndex-split dedup keeps lowest seq + refcounts; crash resume never recomputes a done session; content-less rows terminate with the marker and a second migrateSchema is a no-op; dedup groups spanning the 500-row page boundary; empty store). Storage family + typecheck + build green on this head.

Lane bookkeeping:

  • carry PR into lane bugfix-v0.1.207 — under the v3 gate-branch model merges into a lane NEVER publish (the lane was cut from the bugfix gate branch per the epoch-1 procedure, so it carries the v3 workflow, not the v2 wide trigger).
  • after this merges, the ship PR (bugfix-v0.1.207 -> bugfix, label ci:full) publishes v0.1.208.
  • merging alignment PR [PR] align master with v0.1.207 (shipped from bugfix-v0.1.206) #4673 first is still safe: the P4 auto-cut no-ops on the existing branch.

Fixes #4677

…tion (#4677)

v0.1.207 loaded every message's content bytes into memory in a single
.all() inside the migration transaction, so multi-GB stores OOM'd on
first open — and because the user_version bump sat in the same
transaction, every subsequent open re-ran the whole migration and
re-OMM'd: permanently unopenable stores.

The v8 DDL (columns + index) still lands in the v1-v7 transaction, but
the identity backfill now runs after it, streamed: sessions are picked
by identity = '', rows flow through a 500-row seq-cursor page join, and
each session commits in its own transaction. A crash between sessions
leaves the done sessions committed with user_version still 7, so the
next open skips them and resumes. Memory holds one page window + one
session's keeper keys, never the store's bytes.

Rows whose content bytes are missing from the CAS get the printable
'absent:' marker — no sha256 hex output can equal it (colon + short
length), so their session terminates instead of rescanning forever.
node:sqlite truncates bound strings at an embedded NUL, which ruled out
the \u0000-shaped marker (it silently bound as '' and the row looked
unmigrated forever).

Dedup semantics unchanged from v0.1.207: same (msg_id, identity) after
the strip keeps the lowest-seq row; later duplicates hand their media
mappings, message rows and CAS refs back.

Fixes #4677
@ranxianglei ranxianglei changed the title fix(storage): stream the v8 identity backfill one session per transaction (#4677) [PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) Oct 25, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

Superseded by #4684 — identical head commit (d17c9c7), branch renamed to backport/carry-v8-migration-streaming so version-guard's head-shape allowlist (backport/carry-*) passes; this PR's branch name made that guard red on the PR.

@ranxianglei
ranxianglei deleted the 2026-10-25_v8-migration-streaming branch October 25, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant