Repository navigation
[PR] fix(storage): keep CCR originals safe during bulk migration (#5599) - #6773
mariopasquali wants to merge 14 commits into
Conversation
…d fold-time alignment (ranxianglei#5608-D)
(ai: vllm-qwen/qwen3.8-27b) 查重: 同家族但机制不同 — 不重复, 保留两个 PR 待 owner 取舍。
CI 状态(核对过当前 head 6c624e4 的 check runs):
审查进展: 生产 diff(20 文件, +1590/−50)已全量精读完毕; 正在过 4 个新测试文件 + 行为变更点, 结论(八问 + 可合判定)随后在本线程给出。 |
(ai: vllm-qwen/qwen3.8-27b) Review complete. I read the full production diff (20 files, +1590/−50), reviewed all 9 test files, and ran the 9 PR-touched test files locally on head 6c624e4: 85 tests — 84 pass, 0 fail, 1 skip. CI on the same head: 49/50 green; the single red is the Verdict: mergeable. The design is conservative (refuse rather than corrupt), the commit guard is atomic (strict read + generation check + merge + ingest inside one SQLite transaction), the tests have teeth in both directions, and the diff is clean (7 labeled commits, no unrelated churn, no new config surface — CLI command only). Two notes for the owner:
结构化八问
收工判定:可合。 唯一红 CI lane 是未触碰文件的既有时序 flake(#4599 有记录),master 自身另一 lane 红为继承性问题,均非本 PR 引入。 |
|
Heads-up from the batch review — I'm holding this one out of the current wave pending a fix or an explicit ack on two 1. 2. Startup pre-warm silently skips every session that has originals. Minor notes: (3) If you (or the owner) consider "loud failure + restart" acceptable for the #2401 multi-instance case, finding 1 can be an explicit ack rather than a fix — but finding 2 should be fixed either way. Happy to review a follow-up PR. |
…re projection Two jsonWriter=off regressions flagged in the ranxianglei#6773 review: 1. converge-to-disk permanent wedge (liveness): the converge branch replaced state/stats/blockContents but not session.contentStore, so the guarded commit's mergeDurableOriginals saw a stale resident with no basis and threw 'unproven CCR generation' on every subsequent save until restart. Rebind contentStore from the just-read index mirror and seed the basis before the guarded commit — the state is being discarded anyway, so the stale in-memory originals should be too. 2. startup pre-warm silently skipped every CCR-bearing session: estimateDecodedBytes invoked the committing guardedBuild, whose CCR generation guard throws for a cold session whose contentStore is not yet materialized (the request path materializes it via contentStoreOf, but the warm loop invokes the builder directly). The catch returned MAX_SAFE_INTEGER and the session was skipped from pre-warming. Measure from the pure buildRecordForSave projection instead — the same record guardedBuild returns on the normal path, without the commit side effect.
[bot] 🏷 (ai: claude-opus-5-5) Both fixes are pushed to the PR branch ( Finding 1 — converge-to-disk permanent wedge. In the converge branch, before Finding 2 — pre-warm silently skips every CCR-bearing session. Cold first-save verification: I traced the main request path — every Minor note (3) nuance: "self-heals on the writer's next save" is half-true. A fresh process self-heals (no basis → disk refs ⊆ resident → union merge). But a long-running writer with an existing basis throws Pre-existing multi-instance test failures: CI is running on the pushed head. |
|
(ai: claude-opus-5-5) Added one commit: Without it, the owner check in this PR refused sessions whose originals were still filed under refs from before a renumbering. On a fresh copy of a production 0.1.200 legacy store, 10 of 8,309 sessions failed with The cause is that Result on the same copy, with the same limits as production (5 GiB RAM, 4 CPUs):
Tests: the test that expected a stale ref to fail now expects the re-key. A new test still requires an inconsistent ref map (a ref the map gives to a different message in each direction) to fail. The re-key test fails without the fix. |
[bot] 🏷 (ai: claude-opus-5-5) Reviewed the new commit What it does: In Verified:
No issues found. |
…dup spy The ranxianglei#6773 originals commit (requireCommit + settleCcr) wraps the actual ingest in a transaction and re-enters shadow.record() for the inner write, so one full build now yields two record() calls. The ranxianglei#6780 pre-forward dedup suite (landed on master after this branch forked) counts record() calls as its 'a full build ran' proxy and read 2 instead of 1. Count only the inner (settleCcr-less) ingest so one full build is one count, preserving the suite's pre-ranxianglei#6773 shape. No behavior change to the ranxianglei#6780 skip feature.
(ai: vllm-qwen/qwen3.8-27b) CI Fix Summary — PR #6773Original failure: 3 test jobs (ubuntu-24/3, ubuntu-22/3, windows-22/4) all failing on Root cause: The #6780 pre-forward-dedup test (new on master, landed after this PR forked) uses a spy on Fix applied (commit f231f29):
Local verification: #6780 test 4/4 pass, PR storage tests 45/45 pass, typecheck clean. New Windows failures (post-push): 2 new failures appeared on Windows (windows-22/1: Final state: All 88 check runs green (59 success, 29 skipped, 0 failure). PR is mergeable. 八问
判定:可合并。 全部 88 个 check run 已绿(59 success / 29 skipped / 0 failure)。 |
…originals-master-merge # Conflicts: # tests/storage-durability.test.ts
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI status — PR #6773 (conflict handling)Conflict: resolved — by the PR author, before this session started.
CI on the merged head What remains: the full-suite run on the merged head. Risk is low: the #6780 dedup-spy fix ( |
[ework] auto-fix stalled: test (windows-latest, 22, 5), test (ubuntu-latest, 22, 2), test (ubuntu-latest, 24, 2) — stopping for human review |
1 similar comment
[ework] auto-fix stalled: test (windows-latest, 22, 5), test (ubuntu-latest, 22, 2), test (ubuntu-latest, 24, 2) — stopping for human review |
…originals-master-merge
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
Verdict: APPROVE.
Read the full diff (21 files, +1650/−61) at head 3d478d6, the linked issue #5599, and the PR body. Independently verified the core logic:
Correctness. The write path (guardedBuild) is correct: the commit guard is atomic (strict read + generation check + merge + ingest inside one SQLite transaction via withOriginalsTransaction), the counter check distinguishes stale writers (behind the index → refuse) from newer requests (ahead → may reset), and the converge branch rebinds session.contentStore from the index mirror before the guarded commit (fixing the permanent-wedge issue from the previous review round). The read path (loadContentStore/contentStoreById) is correct: for writer-off/frozen sessions, the index mirror is consulted first (with strict sha256 validation via getCcr(id, true)), falling back to the file sidecar only when the mirror is empty. The re-stamp completeness check (maybeRestampToLedger) is correct: the one-time ledger re-stamp now requires every mapped id to own a row (live or retired), so an imported collapsed ledger (folded ghosts with no rows) keeps its legacy refs instead of re-targeting saved citations at newer bodies. The migration (migrateLegacyToUnified) is correct: it is read-only over the JSON files, validates the paired originals (corrupt/missing/conflicting → fail that session, no partial import), verifies the import via digest reconciliation (verifyImported), and detects drift (CCR originals differing from the migration source). The strict validation (getCcr(sessionId, requireValid)) is thorough: it checks the mirror's JSON shape, every original body's sha256 against its key, every reference's binding, and the ccr_entries table (count + content).
Tests. The 9 test files (~1300 lines of new tests) have teeth in both directions (positive and negative). The S16 test pins the re-stamp completeness gate; the persist-foreign-originals-control tests pin the commit guard (stale writer refused, newer reset allowed, atomicity, corrupt mirror refused); the storage-migrate/storage-migrate-originals tests pin the migration (import, verify, drift detection, re-key, frozen aliases).
Behavior drift (old→new, all intentional, disclosed in PR body + prior review round):
- writerOff/frozen CCR read source: file sidecar → index mirror first (strict validation), file fallback.
- Corrupt CCR mirror: degrade-to-file → refuse (throw) for writerOff/frozen.
shadow.recordnormal path (requireCommit=false, all existing callers): unchanged (still never throws). For requireCommit=true (writerOff/frozen): now throws on failure (was fire-and-forget) — fixes silent data loss.- Ledger re-stamp: gated on completeness (every mapped id owns a row). Native sessions unchanged; imported collapsed ledgers now skip the re-stamp.
decompositionLogclear: moved from beforerecord()to after — a rejected commit retains the log for retry.estimateDecodedBytes: measured frombuildRecordForSave(pure projection) instead ofguardedBuild(committing builder) — fixes the pre-warm skip.saveContentStore: writerOff/frozen skip the sidecar file write (the index mirror is the durable copy).
House conventions. No as any, no @ts-ignore, hex escapes for ACP tags, no new config surface (CLI command only), clean diff (21 files, all related to the PR's purpose). The merge with master (3d478d6) is clean (verified: no conflict markers, both sides' changes present — master brought #6947 ref backfill into store.ts/ingest.ts/mysql.ts, no interaction with the PR's storage changes).
CI. The mirror cache (fetched 2026-11-18T08:55Z) shows the test shards still in_progress on head 3d478d6 (13 in_progress, 5 success, 12 skipped, 0 failed). The previous head (0f11a30) was fully green (88 check runs, 0 failures). The current head is a clean merge of master with no PR-file changes from the master side that interact with the PR's storage changes.
No small findings to fix on the spot. The PR is clean.
结构化八问
- 是什么问题:
bili storage migrate批量导入命令(#5599/#5636 设计)缺少 originals(CCR 原文镜像)侧的安全防护——迁移约 8,300 个 legacy 会话时,存在多条路径会丢失、混用或把已存原文重指到错误的消息体上。 - 什么原因导致的:原 #5636 只迁会话体,不导入/校验配对的 content-store sidecar;writerOff/frozen 的 index-only 保存路径没有提交保证(shadow.record 原契约 "never throws"、无代际守卫),陈旧 writer 可覆盖另一 writer 的 originals;严格读取不校验 body 的 sha256;一次性账本重钉(ref restamp)无完整性守卫,对导入的 collapsed ledger 会把已存引用重指到更新的消息体。
- 影响范围与严重程度:使用
bili storage migrate迁移 legacy JSON 会话的用户(真实安装约 8,300 个会话,其中 7 个已有冲突)——不修则 originals 被静默丢失/混用,属数据损坏类(p1)。writerOff/frozen 读写路径的行为变更影响既有 pure-sqlite 用户(读源从文件 sidecar 变为 index 镜像;损坏镜像从降级读文件变为拒载),见第 5 问逐项披露。 - 怎么修复的:在决策点修,不在掩码层修——① 导入时校验配对 sidecar(损坏/缺失/冲突 → 该会话 fail,不做部分导入;重跑只校验不重写);② 提交时在单个 SQLite 事务内完成「严格读 + 代际检查 + 合并 + ingest」(原子边界:陈旧 writer 的提交被拒;更新请求可有意重置);③ 严格读取校验每个 body 的 sha256 + ccr_entries 绑定逐字节一致;④ 账本重钉仅在「每个 mapped id 都有行」时执行(导入的 collapsed ledger 保留 legacy refs,与 JSON 加载平价)。
- 是否有回归问题:逐项审过触碰路径的既有行为——full 模式双写路径不变;shadow.record 普通路径(requireCommit=false,全部既有调用方:增量车道 ingestRowsNow、网络后端、full 双写)不变(仍 never throws);账本重钉对 native 会话不变(已核实全部 DELETE 路径:字节相同的 retired 重复行保留存活行、整会话 drop/archive 删整会话——不存在「live 会话的 mapped id 缺行」,historian 是 retire 不是 delete);decompositionLog 清除从 record 前移到 record 后 = 被拒提交保留日志供重试(有意变更)。有意变更清单(旧→新):a) writerOff/frozen CCR 读源:文件 sidecar → index 镜像优先(严格校验)+ 文件回退;b) 损坏 CCR 镜像:降级读文件 → 拒载(throw);c) requireCommit=true 的 record:fire-and-forget → 失败 throw(修静默数据丢失);d) 账本重钉:seqByRawId 非空 → 每个 mapped id 都有行(native 不变,collapsed ledger 跳过);e) estimateDecodedBytes:guardedBuild → buildRecordForSave(修 pre-warm 跳过);f) saveContentStore:writerOff/frozen 跳过 sidecar 文件写(index 镜像是持久副本)。
- 是否破坏缓存:不涉及。本 PR 不改缓存/重试/增量游标/去重键;账本重钉的完整性守卫只影响一次性对齐(native 会话不变,collapsed ledger 保留 legacy refs),不影响前缀缓存。
- 是否改了设计方向:不涉及。本 PR 在既有存储架构内加固(commit guard、strict reads、re-stamp completeness),不改变架构/边界/数据流。
- 是否引入了新的配置:不涉及。本 PR 只加 CLI 命令(
bili storage migrate),不加新的 env/config/默认值。
合并判定:可合。 核心逻辑正确,测试有牙,行为变更已披露,diff 干净。
[bot] 🏷 Review complete: APPROVE. Posted an APPROVE review on PR #6773 (head 3d478d6) with full justification and the 八问 checklist.
Review: #6773 (review) |
# Conflicts: # README.md # src/cli.ts
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Conflict resolved — PR is now mergeable (CI-only gate)Live re-verification first: the PR head had moved since my last run — the author re-landed the branch at What I did this run:
八问(中文结论):
收工判定:冲突已解决,PR 现为 mergeable,只剩新 head 的 CI 全量跑绿即可由 owner 合并。 |
(ai: claude-opus-5-5)
This brings the
bili storage migratecommand from #5636 onto current master, together with the originals fixes we needed to migrate a real install of about 8,300 legacy sessions without losing or mixing up stored originals. #5636 is far behind master and conflicts, so this is a fresh port. If you prefer, the fixes can go onto #5636 instead.What it changes
Test changes on existing tests
Checked on this branch: typecheck and unused-export gate clean; full suite 12,279 tests, 12,205 pass, 0 fail, 74 skipped.
Not included: an offline repair for sessions whose stored originals already conflict (7 of about 8,300 sessions needed it). I will propose that separately.