Repository navigation
[PR] fix(storage): stream the v8 identity backfill one session per transaction (#4677) - #4680
Closed
ranxianglei wants to merge 1 commit into
Closed
ranxianglei wants to merge 1 commit into
ranxianglei wants to merge 1 commit into
Conversation
…tion (#4677) v0.1.207 loaded every message's content bytes into memory in a single .all() inside the migration transaction, so multi-GB stores OOM'd on first open — and because the user_version bump sat in the same transaction, every subsequent open re-ran the whole migration and re-OMM'd: permanently unopenable stores. The v8 DDL (columns + index) still lands in the v1-v7 transaction, but the identity backfill now runs after it, streamed: sessions are picked by identity = '', rows flow through a 500-row seq-cursor page join, and each session commits in its own transaction. A crash between sessions leaves the done sessions committed with user_version still 7, so the next open skips them and resumes. Memory holds one page window + one session's keeper keys, never the store's bytes. Rows whose content bytes are missing from the CAS get the printable 'absent:' marker — no sha256 hex output can equal it (colon + short length), so their session terminates instead of rescanning forever. node:sqlite truncates bound strings at an embedded NUL, which ruled out the \u0000-shaped marker (it silently bound as '' and the row looked unmigrated forever). Dedup semantics unchanged from v0.1.207: same (msg_id, identity) after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back. Fixes #4677
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(ai: zhipuai-lb/glm-5.3)
What broke (the bug, in one sentence): upgrading to v0.1.207 made large unified stores OOM on first open — the v8 migration loaded every message's content bytes into memory in a single query inside one transaction, and because the
user_versionbump sat in the same transaction, every retry re-ran the whole migration and re-OMM'd: the store became permanently unopenable (#4677, p1).The fix: the v8 DDL (columns + index) still lands atomically with v1-v7, but the identity backfill now runs AFTER it, streamed — sessions are picked by
identity = '', rows flow through a 500-rowseq-cursor page join, and each session commits in its own transaction:user_versionstill 7, so the next open skips them and resumes — no rollback-and-redo;absent:marker (no sha256 hex can equal it — it carries a:and is shorter than 64 hex chars), so their session terminates instead of being rescanned forever. A ``-shaped marker was tried first and is load-bearingly wrong: node:sqlite truncates bound strings at an embedded NUL, so it silently bound as''and the row looked unmigrated on every open;(msg_id, identity)after the strip keeps the lowest-seq row; later duplicates hand their media mappings, message rows and CAS refs back.Tests:
tests/issue4677-v8-migration-streaming.test.ts— 5 discriminators (backfill + wireIndex-split dedup keeps lowest seq + refcounts; crash resume never recomputes a done session; content-less rows terminate with the marker and a second migrateSchema is a no-op; dedup groups spanning the 500-row page boundary; empty store). Storage family + typecheck + build green on this head.Lane bookkeeping:
bugfix-v0.1.207— under the v3 gate-branch model merges into a lane NEVER publish (the lane was cut from thebugfixgate branch per the epoch-1 procedure, so it carries the v3 workflow, not the v2 wide trigger).bugfix-v0.1.207->bugfix, labelci:full) publishes v0.1.208.Fixes #4677