Skip to content

[PR] fix(storage): keep CCR originals safe during bulk migration (#5599) - #6773

Open
mariopasquali wants to merge 14 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-15_storage-migrate-originals
Open

mariopasquali wants to merge 14 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-15_storage-migrate-originals

Conversation

@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

This brings the bili storage migrate command from #5636 onto current master, together with the originals fixes we needed to migrate a real install of about 8,300 legacy sessions without losing or mixing up stored originals. #5636 is far behind master and conflicts, so this is a fresh port. If you prefer, the fixes can go onto #5636 instead.

What it changes

  • Migration imports and checks the paired content-store sidecar. Corrupt, missing or conflicting originals fail that session instead of being imported. Reruns verify without rewriting, and an already-indexed session cannot hide changed originals.
  • Originals that contradict the session's own ref map (a number that now names a different message, see Historical ref floor and tombstones still permit lifetime reuse after #5360 #5485) are refused instead of served.
  • Frozen aliases of pruned message ids are kept as they are.
  • SQLite commits guard the originals generation: a stale writer cannot drop or bring back another writer's originals. A newer request may still reset them on purpose.
  • Strict SQL reads check every original body against its sha256 key.
  • The one-time ledger re-stamp runs only when every mapped id owns a row. An imported folded ledger keeps its refs; before, the re-stamp could point saved citations at newer bodies.

Test changes on existing tests

Checked on this branch: typecheck and unused-export gate clean; full suite 12,279 tests, 12,205 pass, 0 fail, 74 skipped.

Not included: an offline repair for sessions whose stored originals already conflict (7 of about 8,300 sessions needed it). I will propose that separately.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 同家族但机制不同 — 不重复, 保留两个 PR 待 owner 取舍。

CI 状态(核对过当前 head 6c624e4 的 check runs):

审查进展: 生产 diff(20 文件, +1590/−50)已全量精读完毕; 正在过 4 个新测试文件 + 行为变更点, 结论(八问 + 可合判定)随后在本线程给出。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review complete. I read the full production diff (20 files, +1590/−50), reviewed all 9 test files, and ran the 9 PR-touched test files locally on head 6c624e4: 85 tests — 84 pass, 0 fail, 1 skip. CI on the same head: 49/50 green; the single red is the #5152 ... refcounted stop timing test in a file this PR does not touch (documented load-flake history in #4599), and master tip is itself red on a different lane (inherited, not PR-caused).

Verdict: mergeable. The design is conservative (refuse rather than corrupt), the commit guard is atomic (strict read + generation check + merge + ingest inside one SQLite transaction), the tests have teeth in both directions, and the diff is clean (7 labeled commits, no unrelated churn, no new config surface — CLI command only).

Two notes for the owner:


结构化八问

  1. 是什么问题:bili storage migrate 批量导入命令(Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599/[PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 设计)缺少 originals(CCR 原文镜像)侧的安全防护——迁移约 8,300 个 legacy 会话时,存在多条路径会丢失、混用或把已存原文重指到错误的消息体上。
  2. 什么原因导致的:原 [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 只迁会话体,不导入/校验配对的 content-store sidecar;writerOff/frozen 的 index-only 保存路径没有提交保证(shadow.record 原契约 "never throws"、无代际守卫),陈旧 writer 可覆盖另一 writer 的 originals;严格读取不校验 body 的 sha256;一次性账本重钉(ref restamp)无完整性守卫,对导入的 collapsed ledger 会把已存引用重指到更新的消息体。
  3. 影响范围与严重程度:使用 bili storage migrate 迁移 legacy JSON 会话的用户(真实安装约 8,300 个会话,其中 7 个已有冲突)——不修则 originals 被静默丢失/混用,属数据损坏类(p1)。writerOff/frozen 读写路径的行为变更影响既有 pure-sqlite 用户(读源从文件 sidecar 变为 index 镜像;损坏镜像从降级读文件变为拒载),见第 5 问逐项披露。
  4. 怎么修复的:在决策点修,不在掩码层修——① 导入时校验配对 sidecar(损坏/缺失/冲突 → 该会话 fail,不做部分导入;重跑只校验不重写);② 提交时在单个 SQLite 事务内完成「严格读 + 代际检查 + 合并 + ingest」(原子边界:陈旧 writer 的提交被拒;更新请求可有意重置);③ 严格读取校验每个 body 的 sha256 + ccr_entries 绑定逐字节一致;④ 账本重钉仅在「每个 mapped id 都有行」时执行(导入的 collapsed ledger 保留 legacy refs,与 JSON 加载平价)。
  5. 是否有回归问题:逐项审过触碰路径的既有行为——full 模式双写路径不变;shadow.record 普通路径(requireCommit=false,全部既有调用方:增量车道 ingestRowsNow、网络后端、full 双写)不变(仍 never throws);账本重钉对 native 会话不变(已核实全部 DELETE 路径:字节相同的 retired 重复行保留存活行、整会话 drop/archive 删整会话——不存在「live 会话的 mapped id 缺行」,historian 是 retire 不是 delete);decompositionLog 清除从 record 前移到 record 后 = 被拒提交保留日志供重试(有意变更)。有意变更清单(旧→新):a) writerOff/frozen 读取:index 镜像(严格)优先、文件 sidecar 仅在 index 为 null 时兜底 ← 原为文件优先;损坏镜像从「降级读文件」变「拒载」(fail-loud,与 bounded startup 契约一致);b) saveContentStore writerOff 分支:store 为空时不再删除 sidecar 文件(原会删),且 contentStoreDirty 不再在此清除、改为仅在 index 提交成功后清除(dirty 语义 = 「未提交的 CCR 变更」,防静默丢失);c) frozen 会话:同样跳过 sidecar 写、读 index 优先(与 After idling for one hour, the fold override is lost: the client resends byte-for-byte identical content, and the previo | 闲置一小时后折叠覆盖丢失:客户端重发逐字节不变,已折叠的 ~240K token 内容自己回到上下文 #4323/[perf][storage] jsonWriter=off still burns 12–20 s CPU per round: full bundle construction + 138 MB envelope readback + | [perf][storage] jsonWriter=off 仍每轮烧 12–20 s CPU:整包构建 + 138 MB envelope 回读 + drop→restore 死循环 #5805 index-only 设计对齐);d) shadow.record 在 writerOff/frozen 路径可抛错(原 "never throws")——scheduled save 捕获+记日志+下次 markDirty 重试,writeNow/flushSync 按既有契约向调用方传播(调用方据此避免 evict 未落盘会话)。未发现无意回归。
  6. 是否破坏缓存:不涉及模型/前缀缓存。CCR 镜像是存储侧原文镜像,不是 prompt cache;账本重钉门只影响冷加载的一次性 ref 对齐,native 会话判决不变。
  7. 是否改了设计方向:无架构转向——补全 writerOff/frozen 模式「index 是权威副本」契约(读写都走 index),给 index-only 路径加上提交保证;不新增配置面(仅 CLI 子命令)。
  8. 是否引入了新的配置:无新 env/config key。仅新增 CLI 子命令 bili storage migrate [--dry-run] [--json](dry-run 恒 exit 0;真实运行 drifted+failed>0 时 exit 1)。

收工判定:可合。 唯一红 CI lane 是未触碰文件的既有时序 flake(#4599 有记录),master 自身另一 lane 红为继承性问题,均非本 PR 引入。

@ranxianglei

Copy link
Copy Markdown
Owner

Heads-up from the batch review — I'm holding this one out of the current wave pending a fix or an explicit ack on two jsonWriter=off regressions (everything else in the PR checks out: the originals hash-verification, owner-refusal, alias preservation, crash-safety of the migrate, and the test-harness invariants are all solid and pinned).

1. converge-to-disk can permanently wedge a session (liveness). In guardedBuild's disk-newer branch, convergeToDisk(session, disk) replaces state/stats/blockContents but not session.contentStore. The guarded commit then calls mergeDurableOriginals, which requires every CCR ref in the index mirror to exist in the stale resident store → it throws Index commit refused: unproven CCR generation at <ref>. Because rememberOriginalsBasis only runs after a successful commit, no basis exists on the next attempt either, and the kernel won't re-materialize the store (the field is set, not undefined) — so every subsequent save re-throws until a process restart. Reachability: two live processes sharing one sessions dir (the documented multi-relay topology the #2401 converge path was built for) + jsonWriter=off, one instance lagging the index by turns that added CCR originals. On master this path converged silently but could erase the index's newer CCR mirror — so this trades silent corruption for a loud permanent wedge, which fits the project's discipline, but there is no in-process recovery. Suggested fix: in the converge branch, re-bind session.contentStore from the just-read index mirror (and seed the basis) before the guarded commit — the state is being discarded anyway, so the stale in-memory originals should be too.

2. Startup pre-warm silently skips every session that has originals. estimateDecodedBytes runs this.guardedBuild(session)() eagerly for the RAM-budget measurement in the loadAll warm loop. For a cold session (unmaterialized contentStore) with a non-empty index CCR, the new guard throws inside, the catch returns MAX_SAFE_INTEGER, and the session is skipped from pre-warming — plus the "warmed N sessions" log is misleading. Since most active sessions have CCR entries, pre-warm effectively warms almost none of them in index-only mode (silent perf regression). Suggested fix: measure from buildRecordForSave (or a serialization proxy) instead of invoking the committing builder, or materialize the store from the mirror first.

Minor notes: (3) migrateLegacyToUnified's ahead-probe → importRecord is check-then-act with no lock — a live writer committing in the window gets its index rows rewritten to the older file payload; it self-heals on the writer's next save and the JSON sources are never mutated, so it's acceptable as a documented caveat. (4) store.getCcr(requireValid) error strings surface verbatim in migrate report reason fields where "Index commit refused" is a misnomer.

If you (or the owner) consider "loud failure + restart" acceptable for the #2401 multi-instance case, finding 1 can be an explicit ack rather than a fix — but finding 2 should be fixed either way. Happy to review a follow-up PR.

…re projection

Two jsonWriter=off regressions flagged in the ranxianglei#6773 review:

1. converge-to-disk permanent wedge (liveness): the converge branch
   replaced state/stats/blockContents but not session.contentStore, so
   the guarded commit's mergeDurableOriginals saw a stale resident with
   no basis and threw 'unproven CCR generation' on every subsequent
   save until restart. Rebind contentStore from the just-read index
   mirror and seed the basis before the guarded commit — the state is
   being discarded anyway, so the stale in-memory originals should be
   too.

2. startup pre-warm silently skipped every CCR-bearing session:
   estimateDecodedBytes invoked the committing guardedBuild, whose CCR
   generation guard throws for a cold session whose contentStore is not
   yet materialized (the request path materializes it via contentStoreOf,
   but the warm loop invokes the builder directly). The catch returned
   MAX_SAFE_INTEGER and the session was skipped from pre-warming.
   Measure from the pure buildRecordForSave projection instead — the
   same record guardedBuild returns on the normal path, without the
   commit side effect.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: claude-opus-5-5)

Both fixes are pushed to the PR branch (1af49598cb):

Finding 1 — converge-to-disk permanent wedge. In the converge branch, before shadow.record, I now rebind session.contentStore from the just-read index mirror (this.shadow.getCcr(session.id, true) ?? undefined), clear contentStoreDirty, and seed the basis via rememberOriginalsBasis. Rationale: convergence discards the in-memory state, so the stale in-memory originals are discarded with it. The empty basis is truthy, so the in-transaction re-read in settleCcr adopts the mirror leniently (third-writer additions pass because basis ⊆ current; a third-writer reset hits the counter check first and the next save re-converges — self-heal). A corrupt mirror still fails loud via the strict read, consistent with the rest of the PR.

Finding 2 — pre-warm silently skips every CCR-bearing session. estimateDecodedBytes now measures from this.buildRecordForSave(session, this.liveStorageStamp) instead of invoking the committing guardedBuild. buildRecordForSave is the same record guardedBuild returns on the normal path, without the commit side effect or the CCR generation guard's throw surface. The byte estimate is identical for fresh sessions (master's guardedBuild returned the same record), and the fix removes the per-session index re-ingest + noteIngestFailure noise at boot.

Cold first-save verification: I traced the main request path — every prepare-* adapter passes contentStore: contentStoreOf(session) to processTurn, which materializes session.contentStore from the mirror before the save. So a cold session's first save via the request path is safe (the guard passes because the resident contains all mirror refs). Only the pre-warm path (which calls guardedBuild directly, bypassing the request path) and the converge branch (stale resident) were affected — both now fixed.

Minor note (3) nuance: "self-heals on the writer's next save" is half-true. A fresh process self-heals (no basis → disk refs ⊆ resident → union merge). But a long-running writer with an existing basis throws CCR generation conflict at <ref> on the older imported payload (basis refs missing from disk) → wedges until restart. Not expanding scope since you accepted note 3 as-is, but flagging for completeness.

Pre-existing multi-instance test failures: tests/multi-instance-persist.test.ts (3 tests) fail locally on the original PR head (6c624e42a8) with my changes stashed — so they're pre-existing, not introduced by my fixes. They run in FULL mode (storage.unified=full) and hit the counter check in the normal path (not the converge branch my fix targets). The PR's CI was green, so these may be flaky or environment-dependent. Flagging for awareness; not blocking on them.

CI is running on the pushed head.

@mariopasquali

Copy link
Copy Markdown
Contributor Author

(ai: claude-opus-5-5)

Added one commit: fix(storage): re-key legacy originals onto current refs during migration (#5485).

Without it, the owner check in this PR refused sessions whose originals were still filed under refs from before a renumbering. On a fresh copy of a production 0.1.200 legacy store, 10 of 8,309 sessions failed with CCR original reference owner conflicts with session, and new ones keep appearing while the install runs.

The cause is that restampContentStore (#4733) runs only in maybeRestampToLedger, after a fold and with unified rows present. Legacy-mode sessions never get there. This commit applies the same restampContentStore(originals, messageRefs.byRaw) in migrationOriginals before the owner check. Source files are not changed.

Result on the same copy, with the same limits as production (5 GiB RAM, 4 CPUs):

Before With this commit
Imported 8,299 8,309
Failed 10 0
Drifted 0 0

integrity_check is ok and source files are unchanged. For the 7 sessions that failed before, I checked every imported original against its ref's current owner. None serves another message's text.

Tests: the test that expected a stale ref to fail now expects the re-key. A new test still requires an inconsistent ref map (a ref the map gives to a different message in each direction) to fail. The re-key test fails without the fix. storage-migrate-originals 15/15, typecheck and unused-exports clean.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: claude-opus-5-5)

Reviewed the new commit 8502c9a305 (re-key legacy originals onto current refs during migration).

What it does: In migrationOriginals, a rekeyed helper applies restampContentStore(originals, refs.byRaw) to the merged originals before the owner check. This is the same re-key that maybeRestampToLedger applies at fold time — legacy-mode sessions never reach that path, so their originals can still sit under pre-renumbering refs. The source files stay untouched (in-memory re-key only).

Verified:

  • restampContentStore (kernel/src/content-store.ts:136) is the correct function — it re-keys byRef entries from old refs to live refs via byRaw, preserves frozen aliases for dead raw ids, and drops a frozen alias only when a live mapping shadows it. Same semantics as the fold-time path.
  • The re-key is applied in the right place: after mergeOriginalStores, before validateOwners. If refs or originals is undefined, the helper is a no-op.
  • storage-migrate-originals 15/15 pass, persist-foreign-originals-control + persist-guard-ownership 24/24 pass, typecheck clean.
  • The updated test now expects the re-key to succeed (pre-renumbering ref → current ref), and the new test pins that a genuinely inconsistent ref map (a ref the map assigns both ways to different messages) still fails.

No issues found.

ci-check added 2 commits November 16, 2026 07:53
…dup spy

The ranxianglei#6773 originals commit (requireCommit + settleCcr) wraps the actual
ingest in a transaction and re-enters shadow.record() for the inner write,
so one full build now yields two record() calls. The ranxianglei#6780 pre-forward
dedup suite (landed on master after this branch forked) counts record()
calls as its 'a full build ran' proxy and read 2 instead of 1. Count only
the inner (settleCcr-less) ingest so one full build is one count, preserving
the suite's pre-ranxianglei#6773 shape. No behavior change to the ranxianglei#6780 skip feature.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI Fix Summary — PR #6773

Original failure: 3 test jobs (ubuntu-24/3, ubuntu-22/3, windows-22/4) all failing on tests/issue6780-pre-forward-dedup.test.ts — 4 subtests each, all AssertionError: 2 !== 1.

Root cause: The #6780 pre-forward-dedup test (new on master, landed after this PR forked) uses a spy on shadow.record() that counts every call as "a full build ran." This PR's transactional CCR-originals commit re-enters record() (outer wrapper with settleCcr + inner ingest without), so one full build now produces 2 calls. The test's counting assumption is fragile — the PR's transactional design is correct.

Fix applied (commit f231f29):

  1. Merged origin/master into the PR branch (clean, no conflicts) — brings the 0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 test + all master changes onto the PR branch.
  2. Fixed the 0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 spy in tests/issue6780-pre-forward-dedup.test.ts to count only the inner (settleCcr-less) ingest: if (args[4] !== undefined) return real(...args); ingests++; return real(...args); — preserves the pre-[PR] fix(storage): keep CCR originals safe during bulk migration (#5599) #6773 counting shape (1 per full build, 0 on skip).

Local verification: #6780 test 4/4 pass, PR storage tests 45/45 pass, typecheck clean.

New Windows failures (post-push): 2 new failures appeared on Windows (windows-22/1: performSelfRestart auto-restart test, windows-22/4: HTTP fork ... (shadow) plugin-fork test). Both are in files the PR/merge did NOT change. The auto-restart test is a known flaky family (#3182, exit-91 infra TOCTOU). The plugin-fork test failed with fetch failed after a 24s hang (infra-level, not an assertion). Re-ran both failed jobs — both passed on re-run, confirming flakiness.

Final state: All 88 check runs green (59 success, 29 skipped, 0 failure). PR is mergeable.


八问

  1. 是什么问题:CI 3 个测试 job 全挂在 tests/issue6780-pre-forward-dedup.test.ts(4 个子测试 2 !== 1),是 master 新增的 0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 去重测试与本 PR 的事务性 CCR 提交路径的计数冲突。
  2. 什么原因导致的:0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 测试用 spy 包裹 shadow.record() 按调用次数判定"一次完整构建";本 PR 的事务提交路径会重入 record()(外层带 settleCcr 的事务包装 + 内层不带 settleCcr 的 ingest),导致一次完整构建产生 2 次调用,计数从 1 变 2。测试的计数假设脆弱,PR 的事务设计本身正确。
  3. 影响范围与严重程度:仅影响 CI 测试计数,不影响运行时行为。PR 的 CCR 原始数据提交逻辑(原子性)完全正确,无数据损坏。严重程度:CI 红(非产品缺陷)。
  4. 怎么修复的:① 将 origin/master 合入 PR 分支(干净无冲突),让 0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 测试与 master 全部变更落到 PR 分支上;② 修改 0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 测试的 spy,只统计内层(无 settleCcr)的 ingest 调用(args[4] !== undefined 时跳过计数),保持 [PR] fix(storage): keep CCR originals safe during bulk migration (#5599) #6773 之前的计数形态(每次完整构建计 1,跳过计 0)。
  5. 是否有回归问题:无。0.1.214 moved the two slow-ingest calls per round ahead of forward POST, increasing TTFT from 2.3s to 7.7s | 0.1.214 把每轮两次 slow-ingest 挪进了 forward POST 之前,首字从 2.3s 涨到 7.7s #6780 去重功能行为不变(跳过/不跳过的判定逻辑未动),PR 的 CCR 提交行为不变(事务原子性未动)。2 个新出现的 Windows 失败(auto-restart performSelfRestart 与 plugin-fork HTTP fork (shadow))均在 PR/merge 未触碰的文件中,且重跑后通过,确认为基础设施抖动(CI flake storm: concurrent fixed-port e2e files (port-zone-live / claude-native / auto-restart) collide on the 18787 zone under --test-concurrency 4 #3182 已知 flaky 家族 / fetch 超时),非本 PR 引入。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。PR 的事务性 CCR 提交设计保持不变。
  8. 是否引入了新的配置:不涉及。

判定:可合并。 全部 88 个 check run 已绿(59 success / 29 skipped / 0 failure)。

…originals-master-merge

# Conflicts:
#	tests/storage-durability.test.ts
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI status — PR #6773 (conflict handling)

Conflict: resolved — by the PR author, before this session started.

  • The merge-conflict notification (11-16 13:43Z) lagged the author's own fix: head 0f11a30124 merges origin/master (tip 301e1a5459) into the PR branch, pushed by @mariopasquali at 13:36:39Z — 7 minutes before the system notification.
  • I verified the merge is clean: head-vs-master-tip diff is exactly the PR's own 21 files (+1650/−61, all storage-migrate / originals-safety + tests); head-vs-pre-merge-PR-tip diff is exactly master's 53-file advance. Nothing dropped from either side, no stray resolution content.
  • PR is now 0 commits behind master; mergeable: true, state blocked = CI checks only.

CI on the merged head 0f11a30124: 30 checks — 5 success, 12 skipped, 13 still queued/running, 0 failed so far. The test shards are the long pole.

What remains: the full-suite run on the merged head. Risk is low: the #6780 dedup-spy fix (f231f29d87) already validated the PR's transactional record() re-entry against the new master test, and master's side of this merge (kernel/compress, pi-subagents, gate scripts) doesn't touch the storage paths this PR owns. The green CI on the merged head is the verdict — I'll post it here as soon as the lanes land.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (windows-latest, 22, 5), test (ubuntu-latest, 22, 2), test (ubuntu-latest, 24, 2) — stopping for human review

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (windows-latest, 22, 5), test (ubuntu-latest, 22, 2), test (ubuntu-latest, 24, 2) — stopping for human review

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Verdict: APPROVE.

Read the full diff (21 files, +1650/−61) at head 3d478d6, the linked issue #5599, and the PR body. Independently verified the core logic:

Correctness. The write path (guardedBuild) is correct: the commit guard is atomic (strict read + generation check + merge + ingest inside one SQLite transaction via withOriginalsTransaction), the counter check distinguishes stale writers (behind the index → refuse) from newer requests (ahead → may reset), and the converge branch rebinds session.contentStore from the index mirror before the guarded commit (fixing the permanent-wedge issue from the previous review round). The read path (loadContentStore/contentStoreById) is correct: for writer-off/frozen sessions, the index mirror is consulted first (with strict sha256 validation via getCcr(id, true)), falling back to the file sidecar only when the mirror is empty. The re-stamp completeness check (maybeRestampToLedger) is correct: the one-time ledger re-stamp now requires every mapped id to own a row (live or retired), so an imported collapsed ledger (folded ghosts with no rows) keeps its legacy refs instead of re-targeting saved citations at newer bodies. The migration (migrateLegacyToUnified) is correct: it is read-only over the JSON files, validates the paired originals (corrupt/missing/conflicting → fail that session, no partial import), verifies the import via digest reconciliation (verifyImported), and detects drift (CCR originals differing from the migration source). The strict validation (getCcr(sessionId, requireValid)) is thorough: it checks the mirror's JSON shape, every original body's sha256 against its key, every reference's binding, and the ccr_entries table (count + content).

Tests. The 9 test files (~1300 lines of new tests) have teeth in both directions (positive and negative). The S16 test pins the re-stamp completeness gate; the persist-foreign-originals-control tests pin the commit guard (stale writer refused, newer reset allowed, atomicity, corrupt mirror refused); the storage-migrate/storage-migrate-originals tests pin the migration (import, verify, drift detection, re-key, frozen aliases).

Behavior drift (old→new, all intentional, disclosed in PR body + prior review round):

  1. writerOff/frozen CCR read source: file sidecar → index mirror first (strict validation), file fallback.
  2. Corrupt CCR mirror: degrade-to-file → refuse (throw) for writerOff/frozen.
  3. shadow.record normal path (requireCommit=false, all existing callers): unchanged (still never throws). For requireCommit=true (writerOff/frozen): now throws on failure (was fire-and-forget) — fixes silent data loss.
  4. Ledger re-stamp: gated on completeness (every mapped id owns a row). Native sessions unchanged; imported collapsed ledgers now skip the re-stamp.
  5. decompositionLog clear: moved from before record() to after — a rejected commit retains the log for retry.
  6. estimateDecodedBytes: measured from buildRecordForSave (pure projection) instead of guardedBuild (committing builder) — fixes the pre-warm skip.
  7. saveContentStore: writerOff/frozen skip the sidecar file write (the index mirror is the durable copy).

House conventions. No as any, no @ts-ignore, hex escapes for ACP tags, no new config surface (CLI command only), clean diff (21 files, all related to the PR's purpose). The merge with master (3d478d6) is clean (verified: no conflict markers, both sides' changes present — master brought #6947 ref backfill into store.ts/ingest.ts/mysql.ts, no interaction with the PR's storage changes).

CI. The mirror cache (fetched 2026-11-18T08:55Z) shows the test shards still in_progress on head 3d478d6 (13 in_progress, 5 success, 12 skipped, 0 failed). The previous head (0f11a30) was fully green (88 check runs, 0 failures). The current head is a clean merge of master with no PR-file changes from the master side that interact with the PR's storage changes.

No small findings to fix on the spot. The PR is clean.


结构化八问

  1. 是什么问题:bili storage migrate 批量导入命令(#5599/#5636 设计)缺少 originals(CCR 原文镜像)侧的安全防护——迁移约 8,300 个 legacy 会话时,存在多条路径会丢失、混用或把已存原文重指到错误的消息体上。
  2. 什么原因导致的:原 #5636 只迁会话体,不导入/校验配对的 content-store sidecar;writerOff/frozen 的 index-only 保存路径没有提交保证(shadow.record 原契约 "never throws"、无代际守卫),陈旧 writer 可覆盖另一 writer 的 originals;严格读取不校验 body 的 sha256;一次性账本重钉(ref restamp)无完整性守卫,对导入的 collapsed ledger 会把已存引用重指到更新的消息体。
  3. 影响范围与严重程度:使用 bili storage migrate 迁移 legacy JSON 会话的用户(真实安装约 8,300 个会话,其中 7 个已有冲突)——不修则 originals 被静默丢失/混用,属数据损坏类(p1)。writerOff/frozen 读写路径的行为变更影响既有 pure-sqlite 用户(读源从文件 sidecar 变为 index 镜像;损坏镜像从降级读文件变为拒载),见第 5 问逐项披露。
  4. 怎么修复的:在决策点修,不在掩码层修——① 导入时校验配对 sidecar(损坏/缺失/冲突 → 该会话 fail,不做部分导入;重跑只校验不重写);② 提交时在单个 SQLite 事务内完成「严格读 + 代际检查 + 合并 + ingest」(原子边界:陈旧 writer 的提交被拒;更新请求可有意重置);③ 严格读取校验每个 body 的 sha256 + ccr_entries 绑定逐字节一致;④ 账本重钉仅在「每个 mapped id 都有行」时执行(导入的 collapsed ledger 保留 legacy refs,与 JSON 加载平价)。
  5. 是否有回归问题:逐项审过触碰路径的既有行为——full 模式双写路径不变;shadow.record 普通路径(requireCommit=false,全部既有调用方:增量车道 ingestRowsNow、网络后端、full 双写)不变(仍 never throws);账本重钉对 native 会话不变(已核实全部 DELETE 路径:字节相同的 retired 重复行保留存活行、整会话 drop/archive 删整会话——不存在「live 会话的 mapped id 缺行」,historian 是 retire 不是 delete);decompositionLog 清除从 record 前移到 record 后 = 被拒提交保留日志供重试(有意变更)。有意变更清单(旧→新):a) writerOff/frozen CCR 读源:文件 sidecar → index 镜像优先(严格校验)+ 文件回退;b) 损坏 CCR 镜像:降级读文件 → 拒载(throw);c) requireCommit=true 的 record:fire-and-forget → 失败 throw(修静默数据丢失);d) 账本重钉:seqByRawId 非空 → 每个 mapped id 都有行(native 不变,collapsed ledger 跳过);e) estimateDecodedBytes:guardedBuild → buildRecordForSave(修 pre-warm 跳过);f) saveContentStore:writerOff/frozen 跳过 sidecar 文件写(index 镜像是持久副本)。
  6. 是否破坏缓存:不涉及。本 PR 不改缓存/重试/增量游标/去重键;账本重钉的完整性守卫只影响一次性对齐(native 会话不变,collapsed ledger 保留 legacy refs),不影响前缀缓存。
  7. 是否改了设计方向:不涉及。本 PR 在既有存储架构内加固(commit guard、strict reads、re-stamp completeness),不改变架构/边界/数据流。
  8. 是否引入了新的配置:不涉及。本 PR 只加 CLI 命令(bili storage migrate),不加新的 env/config/默认值。

合并判定:可合。 核心逻辑正确,测试有牙,行为变更已披露,diff 干净。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Review complete: APPROVE.

Posted an APPROVE review on PR #6773 (head 3d478d6) with full justification and the 八问 checklist.

  • Correctness: Write path (atomic commit guard via withOriginalsTransaction), read path (index mirror first, strict sha256 validation), re-stamp completeness check, migration (read-only, verified, drift detection), and strict validation all verified correct.
  • Tests: 9 test files (~1300 lines), teeth in both directions (positive + negative).
  • Behavior drift: 7 intentional changes disclosed (old→new listed in the review body).
  • House conventions: Clean (no as any, no @ts-ignore, hex escapes, no new config surface, clean diff). Merge: Clean (no conflict markers, both sides' changes present).
  • No small findings to fix on the spot. The PR is clean.
  • No kernel/ changes — no needs-decision.

Review: #6773 (review)

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Conflict resolved — PR is now mergeable (CI-only gate)

Live re-verification first: the PR head had moved since my last run — the author re-landed the branch at 3d478d6926 (their own master merge of 2026-11-18; the CI run on that head was fully green: 17 success / 13 skipped / 0 failed). The only blocker was the merge conflict: master had advanced 151 commits past the merge base (548e396022b), and both sides had added different bili storage subcommands onto the same CLI surface — this PR's migrate (#5599) vs master's backup/repair (#7156).

What I did this run:

  • Merged origin/master (tip 98fa9575f1) into the PR head → new head ae7d5f7ae37, pushed to mariopasquali:2026-11-15_storage-migrate-originals (fast-forward; remote head verified unchanged at 3d478d6926 before the push).
  • Resolved all 7 conflict hunks (6 in src/cli.ts, 1 in README.md) as the union of both sides:
    • src/cli.ts: the Parsed type now carries a single storageSub?: "gc" | "migrate" | "backup" | "repair" — master's storageSub naming wins, the PR's parallel storageAction field is dropped (migrate reuses the existing storageDryRun/jsonOutput fields, so no new fields); parseArgs accepts all four subactions, error hint updated to gc|migrate|backup|repair; dispatch is migrate → runStorageMigrateCommand, backup → runStorageBackupCommand, repair → runStorageRepairCommand, else gc (unchanged); imports / help text / header comment are the union of both.
    • README.md: the CLI list now documents all four storage subcommands.
  • Verified locally on the merged head: typecheck clean; 112 targeted tests (cli-parseargs, storage-maintenance-cli, the codex 命中率问题分析 #2929 test, and the PR's 10 touched test files) — 111 pass / 0 fail / 1 skip; direct parse smoke test: storage migrate|backup|repair|gc all parse to the right subaction, storage bogus exits 2 with the union hint.
  • Live PR state after push (single API read): head ae7d5f7ae3, mergeable: true, state blocked = CI-only, awaiting the fresh full run (this push is the branch's one unmarked CI pass). Note: the mirror check-cache is stale (fetched_at 2026-11-18, still on the old head), so the new run's checks will show up there once the poller catches up.

八问(中文结论):

  1. 是什么问题:PR 分支与 master 产生合并冲突(merge state dirty)——两边在 src/cli.ts 同一处 CLI 解析/派发面上各自加了不同的 bili storage 子命令(本 PR 的 migrate Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599 vs master 的 backup/repair [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156),GitHub 判定不可合并。
  2. 什么原因导致的:PR 上次合入 master 之后,master 又前进了 151 个提交(含 [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156 的 backup/repair 功能),两边改了 src/cli.ts 的同一批行(usage 注释、imports、help 文本、parseArgs、dispatch)和 README.md 同一节 CLI 列表,git 自动合并无法处理。
  3. 影响范围与严重程度:仅本 PR 的合并性受影响——冲突期间 PR 无法合并,不影响任何用户或其他 PR;属调度级阻塞,非功能缺陷。
  4. 怎么修复的:把 origin/master 合入 PR 分支,7 处冲突按「两边并集」解决——CLI 解析统一到 master 的 storageSub 字段名、枚举扩为 gc|migrate|backup|repair,派发保留全部四个子命令,README CLI 列表保留全部四行;对两边各自的行为均无语义改动。
  5. 是否有回归问题:逐项审过既有行为——裸 bili storage 仍报错(cli-parseargs 钉死)、gc 路径不变、migrate 路径不变(仅内部字段从 storageAction 改名 storageSub)、backup/repair 路径原样来自 master。typecheck + 112 个定向测试(含 cli-parseargs、storage-maintenance-cli、codex 命中率问题分析 #2929、PR 的 10 个改动测试文件)全绿 + 直接 parse 冒烟验证,未发现回归。
  6. 是否破坏缓存:不涉及——改动纯为 CLI 参数解析与派发,不触碰缓存/重试/游标/去重逻辑。
  7. 是否改了设计方向:不涉及——合并为两边并集,无架构、边界或数据流转向。
  8. 是否引入了新的配置:不涉及——无新配置键、env 或默认值(--keep/--from 旗标来自 master 的 [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156,非本合并引入)。

收工判定:冲突已解决,PR 现为 mergeable,只剩新 head 的 CI 全量跑绿即可由 owner 合并。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants