Repository navigation
[PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) - #5636
ranxianglei wants to merge 17 commits into
Conversation
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — 这是 #5599 设计评审线程里 owner 点名的交付物(「麻烦提交一个pr 专门用来迁移json到sqlite的」),open/closed 全量检索 storage migrate / json→sqlite import 无其他实现 PR 或同机制 issue。 标签: scope/storage(owner 请求的功能类 PR,非缺陷,不打 bug/严重度标)。 开始核对:拉分支审 diff、本地定向验证(typecheck + 新增测试套件)、确认 CI 状态,结论随后按八问回复。 |
cf3895a to
4092d1b
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突已解 + 顺带修掉了 CI 红(auto-fix lane 1/3)。分支 本轮做了什么① 冲突解决(rebase 到 origin/master,落后 143 个提交)
② 发现并修复了 CI 红(先于本次 rebase 就存在)
③ 验证
结构化结论(八问)
判定:冲突已清、已知红已修,交棒给 CI 全量把关;若 CI 再红,按 lane 规则继续(attempt 2/3)。 |
(ai: vllm-qwen/qwen3.8-27b) 两个红 check 已修(auto-fix lane,增量推送 逐项① unused-exports —
② docs-sync — 生成物漂移,定位到一行:
验证:两个门禁脚本本地复跑均干净(unused-export gate clean / generate 后四件生成物零漂移);tsc --noEmit 干净;定向套件 tests/storage-migrate.test.ts 6/6 绿。全量回归交 CI。 结构化结论(八问)
判定:已知红全部修完并已推送,等 CI 重跑;若再有红,按 lane 规则继续(attempt 2/3)。 |
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10080-pr.5636.d76284aThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5636.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 背景一句话: 待决问题:migrate 的代表文件选择要不要对齐 boot 的真实仲裁顺序?
结构化八问(本轮工作 = PR 审核 + 冲突解决 + CI 转绿):
CI 状态:新 head 33a37d0 已推送,全矩阵重跑中(上一红 head 上唯一失败就是 static-gates,其余门禁绿或运行中)。本地证据:build + dist-external 门禁 OK、typecheck 干净、定向套件(storage-migrate/cli-parseargs/storage-maintenance-cli/门禁)全绿;全量判定以远程 CI 为准。 标签说明:镜像侧标签 API 未实现(shim 返回 endpoint not implemented,本工作区也没有 add-label.sh),已在 GitHub PR #5636 上打好 判定:CI 转绿后即可合并(合并按惯例由人来)。唯一开放项是上面的 🔶 survivor-rule 对齐——不阻塞,默认 A。 |
(ai: vllm-qwen/qwen3.8-27b) 4 个红 check 处理完毕:3 个已修并推送( 逐项根因与修法① static-gates(ubuntu-24 / windows-22 两个 job,同一根因) — 本地完整复现:
② test(windows-22, shard 2) — PR 自己的测试的平台假设: ③ test(windows-22, shard 5) — 两个既有测试,均不在本 PR diff 内,判定与本 PR 无关:
验证(最终树 30f2c14)rebuild + 结构化结论(八问)
判定:已知可修的红全部修完;windows shard 5 的两个失败带着「非本 PR 引入」的证据交回 CI 重跑,若复现则按 lane 规则升级处理。 |
(ai: vllm-qwen/qwen3.8-27b)
🔶 survivor-rule 对齐的决策请求维持上一条不变(推荐 B,默认 A,不阻塞)。 |
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
APPROVE. Clean, correctly-scoped additive feature. Reviewed the live head — the lane's 33a37d07b was stale; I reviewed 30f2c14e4 and now 8616d28e1 (after the on-spot test below). This is an independent read of the diff plus the pre-existing machinery it reuses, not a restatement of #5599.
Structured review (eight questions)
-
What it is. A new
bili storage migrate [--dry-run] [--json]that bulk-imports every legacy JSON session under the sessions dir (incl.-cold) into the unified SQLite index and digest-reconciles each against its source file. It closes the gap in #5599 where today's JSON→SQLite migration is purely lazy (organic import on first open), so a dormant install upgrading past a future JSON-reader removal could lose never-opened old files. -
Root cause / mechanism. Capability add, not a bug. Reuses boot's survivor rule (newest
savedAt, lexicographic tie-break) for same-id routes, the existing organic-ingest path (shadow.importRecord), and a new side-effect-free verifier (UnifiedShadow.verifyImported) that runs the sameBatchedViewVerificationas the steady-state sample with latch/freeze stripped. -
Blast radius & severity. Additive CLI action word only. Sole delta to existing behavior: the unknown-action help string (
try "bili storage gc"→try "bili storage gc | migrate") — non-semantic. Zero new config keys/env/flags (--dry-run/--jsonare pre-existing globals). Wire protocol, persistence schema (v10), self-update, kernel: untouched. -
How it works — correctness checks I ran. Confirmed every reused helper exists with the expected signature (
fileNameMatchesId,parseBootPayload,readRawFileWithBytes,loadUnified,unifiedListSessionIds,recordToLegacyView,walkJsonFilesSync). Load-bearing invariant verified:persist.ts:603 const legacyLoad = recordToLegacyViewis exactly what's wired into the shadow, so ingest and post-import verification compare identical views — fresh imports match, anddriftedfires only on genuine file-vs-index divergence (no false positives).verifyImported(shadow.ts:780) is byte-for-byte the verify-sample loop minus side effects. Dispatch guard (cli.ts:765) and the read-only guarantee (onlyindex.dbwritten; JSON never deleted/renamed/quarantined) hold. Thecheck-dist-external.mjsNOT_A_SPECIFIERfix is safe — it rejects only captures containing characters impossible in a bare npm specifier — and is regression-pinned both directions. -
Regressions. None found. Walked each touched path against prior behavior; the only change is the help-text wording above. No fail-fast→swallow, threshold/timing/output-format drift. No
console.erroradded to server modules (house rule); new code reuses the samefail/indexDbSize/replicasAvailableplumbing as the sibling gc/archive commands. -
Cache invalidation. None — no cache/retry/cursor/dedup-key logic touched.
-
Design direction. No pivot. Deliberately read-only over the JSON corpus (corrupt files reported, never quarantined/deleted); cleanup stays an operator decision after a clean report. Consistent with the single-writer storage-lane contract.
-
New configuration. None — one new CLI action word, explicitly not a config knob.
Tests. The six shipped tests have teeth (index-only service proof by deleting the JSON corpus, idempotent re-run, dry-run safety, corrupt isolation + exit 1, route arbitration, read-only invariant via before/after deepEqual). One gap I closed: none exercised the drifted status — the feature's headline "digest reconciliation." Added a deterministic pin (bump a non-request stat in the source after import so it diverges without tripping the ahead-probe). Verified locally on node:sqlite — 7/7 green; the log shows the second run reports 0 imported, 1 drifted, i.e. drift detected and not re-imported.
已顺手修:补齐 drifted(摘要对账分歧)路径的回归测试,本地 7/7 通过 + commit 8616d28
Not auto-mergeable (§7.4: feat + build-gate change → human merge), but nothing blocks approval. At review time the CI test lanes were still running with 0 failures; static gates green.
Note: posted as a comment rather than a formal “Approve”, because GitHub forbids approving your own PR (the reviewer runs under the same account). The verdict above stands.
(ai: vllm-qwen/qwen3.8-27b) ✅ 专职 reviewer 收口(独立端到端审,非照抄前序结论)。审的是 live head 独立核过(自己读 diff + 追既有机制,非复述前序):
已顺手修(唯一实质缺口): 开放项(不阻塞,维持前序 🔶 不动):survivor-rule 代表文件选择是否对齐 boot 真实仲裁次序(floor 12/17,推荐 B、默认 A,自愈、只读、无数据丢失)——我的 APPROVE 不预判它,归 owner 拍板。CI 审时 0 失败、测试分片运行中、static gates 绿。 |
(ai: vllm-qwen/qwen3.8-27b) 本次运行:head c41a33f(run 41841727312)6 个 test 分片红、guards 全绿。已定位并修复其中 5 个确定性失败(同一根因的两族测试回归),推送 结构化八问
环境性失败说明:ubuntu24-s3(job …42813)日志在 协作说明:本轮失败根因来自同 PR 另一实例的并行提交( |
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
Re-review (round 2) — head 7c99bd8ce. Verdict: APPROVE. Round 1 approved the feature base; this round covers the 4 commits added on top of my 8616d28e1 fix: dbc4ad8d0 (code) + 3 test commits.
What the new code does (dbc4ad8d0: src/persist.ts + src/storage/shadow.ts + src/storage/store.ts). The one-time cold-load ledger re-stamp (restampRefsToLedger) now runs only when the ledger is COMPLETE — every non-BLOCKED mapped msg-id must own ≥1 unified row (live OR retired). Otherwise it fails safe to keeping legacy refs (JSON-load parity); a later cold load retries once rows exist. New UnifiedStore.backedMsgIds() = chunked (500) parameterized SELECT DISTINCT msg_id … IN (…). This closes a real corruption path: an imported collapsed ledger (folded ghosts own no rows, tail re-based at seq 0) would otherwise have its ghost bindings evicted onto fresh numbers and saved citations re-targeted at newer bodies.
Why it's safe / no regression:
- Native sessions always qualify — the historian sweep retires covered rows instead of deleting them (#4390), so every mapped id keeps a row and the guard passes → restamps exactly as before.
- Fail-safe direction throughout: store unavailable (
stats.disabled, defaults false) or query error → null → skip → legacy refs. Mirrors the ~19 existingif (!this.shadow || this.shadow.stats.disabled) return nullsibling guards inpersist.ts. - SQL is parameterized (no injection), bounded chunking.
Tests (have teeth):
- S16 (
tests/seq-ledger-pr3.test.ts): both directions — the incomplete ledger keeps legacy refs (refLedgerAligned !== true,restampSuperseded === undefined) AND pins citation integrity (m00001resolves to o1's original body, not the tail); the control half proves a complete ledger still aligns (the guard discriminates on row coverage, not on folds). tests/storage-unified.test.tsoracle mirrors the same guard so the "full serve = legacy serve + alignment" equivalence stays valid under the changed path.tests/issue4986-digest-drift-ref-order.test.tshonestly synthesizes its divergent shape in-test (cold-load no longer produces it post-fix) and keeps the anti-vacuity precondition.
Local evidence (targeted run, node v22.23.2 / node:sqlite): the 3 touched files → 23 pass / 0 fail / 1 skip, exit 0; S15 unchanged-green confirms no regression to the normal restamp path. Full-suite verdict = remote CI (0 failures at this head, test shards running; static gates green).
八问(本轮 delta + 累计)
- 是什么问题:migrate 导入的「折叠账本」在冷加载的一次性 re-stamp 时会被错误改写(幽灵绑定被挪到新号、已存引用重定向到新正文)。
- 什么原因导致的:re-stamp 原先只判
seqByRaw非空就执行,未校验账本完整性;导入的折叠账本里被折叠的消息没有行、尾部从 seq 0 重建,恰好落进这个缺口。 - 影响范围与严重程度:仅影响「经 migrate 导入且含折叠」的会话冷加载;不修则引用错绑(citation 指向错误正文)= 数据语义损坏。触发前提是先跑过 migrate,故非全量用户面,但一旦发生属 p1 级隐患。
- 怎么修复的:re-stamp 前加完整性闸——每个非 BLOCKED mapped id 必须有行(存活或已退役),否则保持 legacy refs(fail-safe 方向),后续冷加载在有行后自动重试。
- 是否有回归问题:无。原生会话恒满足(#4390 只退役不删行)→ 行为不变;store 不可用 / 查询失败 → 保持 legacy refs(S15 仍绿证明正常路径未动);高危模式(语义漂移 / fail-fast 变吞错 / 阈值时序输出格式变化)逐项不涉及。
- 是否破坏缓存:不涉及(re-stamp 是加载期引用对齐,不动缓存 / 重试 / 增量游标 / 去重键)。
- 是否改了设计方向:不涉及(把既有 re-stamp 的触发条件收窄为安全子集,不转向)。
- 是否引入新配置:零配置键 / 环境变量 / 默认值改动。
开放项(不阻塞,维持前序 🔶 不动):survivor-rule 代表文件选择是否对齐 boot 真实仲裁次序(floor 12/17,推荐 B、默认 A)——与本次 re-stamp 完整性闸是不同机制,我的 APPROVE 不预判它,归 owner 拍板。
Note: posted as a comment rather than a formal “Approve” because GitHub forbids approving your own PR (reviewer runs under the same account); the verdict above stands.
(ai: vllm-qwen/qwen3.8-27b) ✅ round-2 复审收口(独立审 delta,非照抄)。新 head 本轮唯一实质改动 = 测试有牙:S16 双向钉(不完整→保 legacy refs + 本轮无需顺手小修(delta 干净);无 kernel/ 改动 → 不触发授权卡点。开放项维持前序 🔶 survivor-rule(floor 12/17,推荐 B、默认 A、不阻塞)——与本次 re-stamp 闸是不同机制,我的 APPROVE 不预判它,归 owner 拍板。 |
84fa388 to
4b18642
Compare
(bot) 🏷 冲突已解决并推送(attempt 1/3)。本条按七问汇报。 说明:同一份摘要此前误发到了线程 #5636(该线程 floor 11,已就地标注);以本条为准。 核对说明:镜像线程标注的上游号 5599 实为该 PR 引用/关闭的 issue 编号;真正的上游 PR 是 GitHub #5636(head 做了什么:把分支 rebase 到当前 master( 七问
当前实况(推送后即时核实):GitHub |
[ework] CI fix verified green ✓ |
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
Review — ROUND 3 @ head 4b186429f (post-rebase)
Verdict: APPROVE ✅
Third independent pass. Since my last review (7c99bd8ce) the branch was rebased onto current master — merge-base == master, so the PR's own contribution is exactly 16 files, zero under kernel/. The large 7c99bd8ce..head range (~415 files incl. kernel/) is purely inherited master churn (fastpath v6 + its revert, docs→docs/ reorg, kernel 0.0.18x bumps, identity-harvest, msgid-lane, …), not this PR's work.
What actually changed since round 2: one new commit on top of the rebased base.
4b186429fd docs: note env-tier limits in child-process deployments (field finding from #5599)— pure docs, +4 lines with en/zh-CN parity: a correct "Child-process deployments" paragraph explaining that a spawned child inherits its env block at spawn time (vars set after host launch, e.g. Windowssetx, only reach new children after the host restarts) and that the config-file tier is the reliable override channel there. Accurate, additive, no code/behavior change. Cleared.
The other 14 commits are the same logical work already reviewed/approved in rounds 1–2 (rebased → new hashes), including my round-1 regression test b7bcfd56e0 (pin the drifted path) and the round-2 re-stamp completeness guard — both present at head.
Rebase regression check (the real risk this round). Hot files src/persist.ts / src/storage/shadow.ts / src/storage/store.ts were heavily modified by master (kernel bump, identity-harvest, msgid-lane). Verified each touched path's existing behavior survives intact at head:
- re-stamp completeness guard chain byte-intact:
src/persist.ts:2101-2102(mappedfilter +unifiedBackedMsgIds) →:2337→src/storage/shadow.ts:1078→src/storage/store.ts:2996; theif (!this.shadow || this.shadow.stats.disabled) return nullfail-safe idiom remains pervasive; - migrate command entry + help + dispatch (
src/cli.ts:242storageAction:"gc"|"migrate",runStorageMigrateCommandimport) intact; - digest-reconciliation statuses + exit-code rule (
src/storage-maintenance.ts:357-384:dryRun || (drifted+failed)===0 ? 0 : 1) intact; UnifiedShadow.verifyImported(src/storage/shadow.ts:800) side-effect-free parity verifier intact; read-only-over-JSON + idempotency + exit-1-on-drifted/failed intact.
No unintended semantic drift from the rebase. Full-suite green across all shards/OS/node versions corroborates the integration.
Check states (mirror cache, cached_sha == 4b186429f): total 30, failed 0, pending 0 — every test shard (ubuntu/windows × node 22/24), static-gates, mysql-contract, gate = success.
Config surface: zero new keys/env/flags (--dry-run/--json are pre-existing globals) — matches the PR's own disclosure. Behavior changes: none (purely additive CLI action); wire protocol, persistence schema (v10), self-update, kernel all untouched — matches disclosure. No kernel-change: declaration in the body; no kernel/ in the diff → the human-only authorization gate is NOT triggered.
Open ends from this reviewer: none. (The survivor-rule representative-file-selection question was raised by other lanes and already carries the needs-decision label — outside this review's scope, already converted-to-decision.)
八问(中文)
- 是什么问题:补齐 #5599 指出的缺口——JSON→SQLite 迁移目前纯惰性(首次打开才导入),休眠安装若越过未来的 JSON 读取器移除会丢失从未打开的旧文件;本 PR 新增一次性、可校验的批量导入命令。
- 什么原因导致的:现有迁移只在首次打开时惰性导入,缺少一次性批量路径(能力缺口,非缺陷);本 PR 复用既有引擎引导 + 导入 + shadow 校验机制补上,不另起炉灶。
- 影响范围与严重程度:纯增量 CLI 入口(
bili storage migrate [--dry-run] [--json]),对既有会话、线协议、持久化格式零改动;未迁移安装行为不变。非缺陷、无严重度风险。 - 怎么修复的:同 id 路由文件按 boot 幸存者规则取代表(最新
savedAt,字典序 tie-break),代表走既有有机导入通道导入、其余报告不动;逐会话跑完整 shadow 校验(verifyImported,与稳态样本同一校验器、无 latch/freeze 副作用),状态 imported/indexed/drifted/failed/planned;对 JSON 语料只读(损坏只报告、不隔离/删除);幂等,drifted/failed 且非 dry-run 时退出码 1。 - 是否有回归问题:本轮重点核查 rebase 到更新 master 后热文件(persist/shadow/store)的既有行为——re-stamp 完整性守卫链字节级完好,migrate 命令/派发/digest 对账/退出码规则/verifyImported/只读保证均保持;无意外语义漂移。全量 CI 30/30 绿佐证集成无回归。
- 是否破坏缓存:不涉及——不触碰前缀缓存/重试/增量游标/去重键等失效策略;仅新增一个离线批量导入命令。
- 是否改了设计方向:不改架构/边界/数据流;沿用统一存储 + shadow 校验既有设计,仅暴露一个 CLI 动作词。
- 是否引入了新的配置:无新 config key / env / flag(
--dry-run/--json为既有全局项),与 PR 披露一致。
结论:可合(APPROVE)。 唯一新增提交是良性文档补充;核心逻辑经 rebase 后字节级完好且全量 CI 绿。无 kernel/ 触碰、无 kernel-change 声明,不触发人工授权门禁。
R3 review of PR #5636 @ head |
|
需要基于最新的master 看是否有缺失 因为master也增加了很多结构 |
(ai: vllm-qwen/qwen3.8-27b) 🔶 需要决策:对照最新 master 审计 migrate —— 发现「CCR 内容边车未吸收」+ 一处会误导删库的文案已按你要求,对照**最新 master( master 会话目录现有结构 × migrate 各自处理
核心缺失:CCR 边车没被吸收,「clean 后可自行删除」是陷阱证据链(均在 head
后果:一个有折叠/CCR 内容、且其 ccr 尚未被索引完整捕获的遗留会话(正是 #5599 想批量导入的大/长会话人群),在 migrate 报 clean、操作者照提示删掉 JSON 兄弟文件后, 诚实边界(不夸大):migrate 对 JSON 只读、运行中不破坏任何数据;丢数据是条件触发——需同时满足 ① 会话有折叠/CCR 内容 ② 该会话此前未被索引的 ccr 镜像完整捕获 ③ 操作者在 clean 信号后删掉了 八问
处置选项
我的建议:B 合入前做掉(或至少先把 storage-maintenance.ts:375 那句改成保留提示),A 作为「真·自包含迁移」后续单独立项。 次要(非阻塞,供参考,不重复展开)
请就 A/B/C(及是否需要升 P1)拍板;选定后我在本 PR 分支落地并补回归测试。 |
|
(ai: vllm-qwen/qwen3.8-27b) 🔶 需要决策:对照最新 master 审计 —— migrate 未吸收 CCR 内容边车(误删陷阱)按 owner 要求对照最新 master(ff5bdc1b,比本 PR base 更新)清点 证据链(head 4b18642):
后果:带折叠/CCR 内容且未被索引 ccr 镜像完整捕获的遗留会话(#5599 目标人群),在 clean 信号后删掉 处置选项(完整审计见协调线程 #5639):
建议 B 合入前做掉,A 后续单独立项。属语义/设计层(跨文件),不擅自改,待 owner 拍板。已打 needs-decision。 (其余新增结构——协议子目录、.fb.json、热+cold、崩溃临时名、迁移标记、backups/*.db——均已正确覆盖,无缺失。) |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复——本线程即该缺陷的处置现场(floor 33 owner 要求对照 master 审计 → floor 36/37 发现边车缺失并列出 A/B/C),无平行条目;本次是把其中无需决策的部分(B)直接修掉。 标签:bug + p1 + scope/storage(保留已有 pr / needs-decision)。理由:migrate 收尾文案诱导删除仍承载 acp_retrieve 原文的 已做:B 护栏已实现并推入本 PR(commit
|
| 选项 | 内容 | 代价 | 建议 |
|---|---|---|---|
| A | 迁移时读入配对边车喂给导入,ccr_entries 落地 + 校验覆盖 ccr | 设计级(跨三文件),需定「部分捕获/多路由文件」边界 | 单独立项,后续做 |
| B | 护栏(本次) | 小改动,已落地 | ✅ 随本 PR 合入 |
| C | 仅文档 | 运行时误导文案仍在 | 放弃(B 已覆盖) |
另审计更正一处:floor 36 引用的 docs/configuration.md:210「边车恒写」一句在 master #5988 后已改写(jsonWriter=off 停写边车),该引用过期;但代码级结论不受影响,我已独立重验全部证据链(head 4b18642),其余结构(协议子目录/.fb/热+cold/崩溃临时名/迁移标记/backups)确认无缺失。本地环境 issue4612-gc-online-guard.test.ts 在 test 1 后挂起 >240s,已用 stash A/B 证明干净树同样挂起——非本 diff 引入,若 CI 该 lane 异常请以此为参照。
八问
- 是什么问题:migrate 静默跳过 CCR 边车却给出「可删 JSON」的 clean 信号——照做会不可恢复地丢掉 acp_retrieve 原文(Feature: content-addressed message store — ID-referenced tool results + acp_retrieve (built-in CCR) #1097)。
- 什么原因导致的:导入路径只吃信封(importRecord(rec.payload)),从不读盘上边车 → 迁移行 ccr_entries 为空;而运行期 CCR 读取 unified 优先、空则回落边车,边车保持权威;discovery 又静默跳过它(persist.ts:1968)、收尾文案无条件邀删(storage-maintenance.ts:375)——上报面与真实数据依赖脱节。
- 影响范围与严重程度:恰好命中 Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599 目标人群(带折叠/CCR 内容的遗留会话);条件触发(需有边车 + 操作者采信 clean 信号);后果 = 原文不可恢复、acp_retrieve 对该会话失效。工具本身只读、不自动丢数据;按数据丢失类定级 p1。
- 怎么修复的:B = 让报告说真话(检出 + 上报 + 撤回误导文案),不动导入语义;A = 让工具真正自包含(导入边车内容),留待拍板后立项。
- 是否有回归问题:审过的既有行为逐项——退出码语义(不变)、scannedFiles 语义(不变,仍只数信封)、JSON 只读保证(字节级测试钉死)、零边车输出(逐字一致,负向门)、--json 形状(纯增量两字段)、日志(增量扩展)。无 wire/持久化/kernel/配置面变更。已知本地环境 GC guard 套件挂起经 A/B 排除与本 diff 相关。
- 是否破坏缓存:不涉及——未触碰任何缓存/重试/游标/去重键;出站消息路径零改动,前缀缓存字节稳定性不受影响。
- 是否改了设计方向:不涉及——仍是「信封导入 + digest 对账」;CCR 内容导入刻意不并入本 PR(那是 A 的决策面)。
- 是否引入了新的配置:无——零新 key/env/flag。
判定:B 部分 CI 绿即可合(护栏完整、风险低、有回归测试钉住);A 等 owner 拍板后单独立项。needs-decision 标签保留至 A 有结论。
(ai: vllm-qwen/qwen3.8-27b) Conflict resolved and pushed. This run: What I did
八问
判定:冲突已解决,PR 恢复可合并状态(mergeable, behind 为 master 后续 6 个无交互提交所致,非阻塞),CI 在新 head 上运行中,等待全绿与人工 review。 |
[ework] CI fix verified green ✓ |
1 similar comment
[ework] CI fix verified green ✓ |
9a24eee to
f8b50ff
Compare
(ai: vllm-qwen/qwen3.8-27b) Conflict resolved and pushed. This run: What I did
八问
判定:冲突已解决,PR 恢复可合并状态(mergeable, blocked 等待 CI/review),CI 在新 head 上运行中,等待全绿与人工 review。 |
[ework] CI fix verified green ✓ |
1 similar comment
[ework] CI fix verified green ✓ |
…ions into the unified SQLite index (#5599) Adds the missing one-shot migration path between the lazy organic import (on first open) and JSON deprecation: bili storage migrate [--dry-run] [--json] walks every legacy state file under the sessions dir (incl. the -cold root), arbitrates same-id route files with boot's survivor rule (newest savedAt, lexicographic tie-break), imports the representative via the existing ingest path, then runs the full digest reconciliation against the legacy source view per session (new UnifiedShadow.verifyImported). Report statuses: imported / indexed / drifted / failed / planned(dry-run); corrupt or mismatched files are reported failed WITHOUT touching them — the command is read-only over the JSON corpus by design (no quarantine renames), so cleanup stays an operator decision after a clean report. No config keys, no env vars, no default-behavior change, no kernel touch.
Covers bulk import into a fresh index with index-only service proof (JSON corpus deleted after import), idempotent re-run, dry-run writes nothing, corrupt-file failure isolation with non-zero exit, same-id route arbitration, and the read-only guarantee over the whole sessions dir.
README Storage-maintenance-CLI section + CONFIGURATION archiveAfterDays row gain the migrate entry in both languages. In passing: the zh README block was missing the bili sessions foreign-blocks line present in the English block (#5312) — added for zh/en parity.
…g storage behavior (#5599) CI-lane fix on the rebased head. Both failures reproduced identically on the original PR head cf3895a (verified by running the suite there), i.e. the suite was red before this rebase; the conflict had kept CI from ever running. 1. 'already-indexed sessions report indexed without re-writing' assumed the first migrate run is what imports. The writing instance's own store open already ran the first-open unified auto-import (present since well before this PR's base), so the first run only verified: imported=0 != 2. Added dropIndex(dir) after writeSessions — same isolation every other test in the file already uses. 2. 'migrate never modifies any file under the sessions dir' snapshot excluded index.db/-wal/-shm but not the integrity-check verdict sidecar index.db.chk (src/storage/integrity-check.ts), which the engine rewrites when migrate opens/closes the store. Widened the exclusion to the whole index.db family (prefix match) per the existing comment's intent.
CI-lane fix: the type is referenced only inside src/persist.ts (the exported LegacyMigrateReport keeps naming it structurally; no file outside persist.ts imports it, tests mirror it locally). INTERNAL_ONLY per scripts/check-unused-exports.mjs.
…enerate (docs-sync gate, #5599) CI-lane fix: the original PR hand-edited the storage.archiveAfterDays row in CONFIGURATION.md/zh to document 'bili storage migrate' but never touched the website/config-reference/*.yaml seeds (the single source the docs-sync guard regenerates from), so the generator overwrote the note. Moved the clause into the process.yaml desc_en/desc_zh entries and re-ran tools/gen-config-docs.mjs: the EN artifact now regenerates byte-identical to what the PR committed; the zh artifact differs only by normalizing the clause's half-width punctuation (,还有 / (#5599: / ;) to full-width Chinese punctuation.
…import after minification (#5599) CI-lane fix (static-gates): the message ended in the bare word 'import', so after tsup minification it fused with the following statement into ...Error("...between scan and import");decodedBytes=...throw new Error("unified sidecar unavailable") which the dist gate's side-effect-import pattern latches onto, capturing a garbage 'specifier' spanning both strings. Reworded to end in 'import attempt'; no test pins the old wording.
…r (windows CI, #5599) CI-lane fix (windows shard 2): dup.file comes from path.relative and carries backslashes on Windows, so startsWith("openai/") failed there while passing on Linux. Assert against 'openai'+path.sep instead.
… whose source diverged, #5599)
An imported session materializes only its live tail into message rows (folded ghosts keep their ref bindings but own no rows; the tail is re-based at seq 0). The #4316 cold-load alignment then stamps the tail onto seq+1 = m00001.., evicting the historical ghost bindings onto fresh numbers and re-targeting saved summary citations at newer bodies — retrieve(m00001) returned the tail message instead of the original. Native sessions are unaffected by construction: the historian sweep retires covered rows instead of deleting them (#4390), so every mapped id keeps its row and survivor seqs never shift. The guard therefore discriminates exactly on the broken shape: every mapped id must own at least ONE row (live or retired) before the stamp may run; otherwise the load keeps legacy refs (JSON-load parity) and retries on a later cold load once rows exist. Query failure fails safe to skip. (cherry picked from commit fe8f27d142280f6c9e894f76cbf3778d45f2d89c)
…5599) Negative half: an imported collapsed ledger (ghost bindings without rows, tail at seq 0..2, summary citing m00001, CCR holding both originals) must cold-load with all legacy refs intact, alignment skipped, and the saved citation still retrieving the original body. Control half: the same ref map over a complete ledger still aligns (no-op here), proving the guard discriminates on row coverage, not on the presence of folds. S15 remains the positive-direction pin for drift alignment. (cherry picked from commit 54aa3f307c4e0e725c6076e6ff8022b923078e3b)
… guard (#5599) The alignedLegacyDigest oracle applied restampRefsToLedger unconditionally, modeling the pre-#5599 cold-load behavior. The one-time re-stamp now refuses incomplete ledgers (an imported collapsed ledger's folded ghosts own no rows), so full-mode loads of the folded corpus keep their legacy refs and the oracle's stamped digest no longer matches production. The oracle now takes the open UnifiedStore probe and applies the alignment only when the guard would pass (every mapped id owns at least one row) — same predicate as the SessionStore cold-load path.
…eletion (#5599) `bili storage migrate` skipped `.content-store.json` sidecars silently while its closing note invited deleting all JSON once the report was clean — deleting a sidecar loses the acp_retrieve originals (#1097) that migrate does not import into ccr_entries. - discovery now collects paired sidecars during the walk (never counted in scannedFiles, never imported, read-only as before) - per-session ccrSidecar flag + report.ccrSidecarSessions / unpairedCcrSidecars (orphan sidecars listed by rel path) - human output gains a KEEP section naming each sidecar; the delete invitation is withdrawn whenever a sidecar is present (zero-sidecar runs print the original sentence verbatim) - exit code semantics unchanged: sidecars alone never fail a run Master-drift audit finding on this PR (owner request 2026-11-10); implements guardrail option B from the decision thread — self-contained CCR import (option A) stays open.
f8b50ff to
9f3f386
Compare
(ai: vllm-qwen/qwen3.8-27b) Conflict resolved and pushed. This run: What I did
八问
判定:冲突已解决,PR 恢复可合并状态(mergeable, blocked 等待 CI/review),等待 CI 全绿与人工 review。 |
[ework] CI fix verified green ✓ |
The failing check Why it's a flake (evidence):
Action taken: re-ran only the failed job via the GitHub API ( Note: the mirror cache is stale ( 八问
判定:PR 代码正确,CI 红为瞬时 flake,重跑已全绿,可合并。 |
Already green — no work needed this run. Re-verified live against the authoritative GitHub API: run The mirror cache is stale: |
Refs #5599 — owner-requested deliverable from that design review thread (「麻烦提交一个pr 专门用来迁移json到sqlite的」); scope pinned in-thread before implementation.
What
New
bili storage migrate [--dry-run] [--json]— one-shot bulk import of every legacy JSON session under the sessions dir (incl. the-coldroot) into the unified SQLite index, with per-session digest reconciliation against the legacy source view.savedAt, lexicographic tie-break); the representative imports through the existing organic-import ingest path, alternates are reported (alternates) but left untouched.UnifiedShadow.verifyImported— same verifier as the steady-state sample, without latch/freeze side effects): statusesimported/indexed/drifted(divergent keys listed) /failed(unreadable, invalid record, filename mismatch) /planned(dry-run).23be109922):.content-store.jsonsidecars (Feature: content-addressed message store — ID-referenced tool results + acp_retrieve (built-in CCR) #1097, hold the acp_retrieve originals) are now DETECTED and REPORTED instead of skipped silently — per-entryccrSidecarflag, report fieldsccrSidecarSessions/unpairedCcrSidecars(orphans listed by rel path), and a KEEP section naming each sidecar in human output. They are still NOT imported intoccr_entries(that self-contained import stays an open decision, option A in the thread).Fills the gap flagged in #5599: today's JSON→SQLite migration is purely lazy (organic import on first open), so a dormant install upgrading past a future JSON-reader removal would lose never-opened old files. This gives operators a one-shot, verified bulk path.
Config surface
One new CLI action word (
storage migrate). Zero new config keys, zero env vars, zero new flags (--dry-run/--jsonare pre-existing globals). No defaults touched.Behavior changes
storage migrateskipped.content-store.jsonsidecars silently (neither counted nor reported), and its human-facing closing note ALWAYS read “archive or delete them yourself once this report is clean” — inviting deletion of files that hold acp_retrieve originals migrate does not import.scannedFiles, still read-only, still never imported) and reported; when at least one exists, the delete invitation is WITHDRAWN and replaced by an explicit do-NOT-delete warning plus a KEEP list of the affected files. Runs with zero sidecars print the original sentence verbatim. Exit-code semantics unchanged (sidecars alone never fail a run);--jsonoutput gains the two new report fields.Wire protocol, persistence schema (v10), self-update, kernel: untouched.
Verification
npx tsc --noEmit -p tsconfig.jsoncleantests/storage-migrate.test.ts(10 tests, green locally): bulk import + index-only service proof (JSON corpus deleted post-import), idempotent re-run, dry-run writes nothing, corrupt-file isolation + non-zero exit, same-id route arbitration, read-only guarantee over the whole sessions dir, drifted-without-reimport; NEW: paired-sidecar detection across dry-run/import/re-run + KEEP copy + byte-untouched proof, zero-sidecar negative gate (original closing note preserved verbatim), orphan-sidecar listing (uncounted, non-failing)