Skip to content

[PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) - #5636

Open
ranxianglei wants to merge 17 commits into
masterfrom
2026-11-04_storage-migrate-json-to-sqlite
Open

ranxianglei wants to merge 17 commits into
masterfrom
2026-11-04_storage-migrate-json-to-sqlite

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Refs #5599 — owner-requested deliverable from that design review thread (「麻烦提交一个pr 专门用来迁移json到sqlite的」); scope pinned in-thread before implementation.

What

New bili storage migrate [--dry-run] [--json] — one-shot bulk import of every legacy JSON session under the sessions dir (incl. the -cold root) into the unified SQLite index, with per-session digest reconciliation against the legacy source view.

  • Same-id route files arbitrate with boot's survivor rule (newest savedAt, lexicographic tie-break); the representative imports through the existing organic-import ingest path, alternates are reported (alternates) but left untouched.
  • Each session then runs the full shadow verification (new UnifiedShadow.verifyImported — same verifier as the steady-state sample, without latch/freeze side effects): statuses imported / indexed / drifted (divergent keys listed) / failed (unreadable, invalid record, filename mismatch) / planned (dry-run).
  • Read-only over the JSON corpus by design: corrupt files are reported, never quarantined/renamed/deleted — cleanup stays an operator decision after a clean report. Idempotent; exit 1 on any drifted/failed result (non-dry-run).
  • CCR sidecar guardrail (master-drift audit finding, commit 23be109922): .content-store.json sidecars (Feature: content-addressed message store — ID-referenced tool results + acp_retrieve (built-in CCR) #1097, hold the acp_retrieve originals) are now DETECTED and REPORTED instead of skipped silently — per-entry ccrSidecar flag, report fields ccrSidecarSessions / unpairedCcrSidecars (orphans listed by rel path), and a KEEP section naming each sidecar in human output. They are still NOT imported into ccr_entries (that self-contained import stays an open decision, option A in the thread).

Fills the gap flagged in #5599: today's JSON→SQLite migration is purely lazy (organic import on first open), so a dormant install upgrading past a future JSON-reader removal would lose never-opened old files. This gives operators a one-shot, verified bulk path.

Config surface

One new CLI action word (storage migrate). Zero new config keys, zero env vars, zero new flags (--dry-run/--json are pre-existing globals). No defaults touched.

Behavior changes

  • Old: storage migrate skipped .content-store.json sidecars silently (neither counted nor reported), and its human-facing closing note ALWAYS read “archive or delete them yourself once this report is clean” — inviting deletion of files that hold acp_retrieve originals migrate does not import.
  • New: sidecars are detected during the walk (still uncounted in scannedFiles, still read-only, still never imported) and reported; when at least one exists, the delete invitation is WITHDRAWN and replaced by an explicit do-NOT-delete warning plus a KEEP list of the affected files. Runs with zero sidecars print the original sentence verbatim. Exit-code semantics unchanged (sidecars alone never fail a run); --json output gains the two new report fields.
  • Why: master-audit of this branch found that post-migrate, session CCR reads fall back to the sidecar file (unified-first, sidecar fallback), so a “clean” report + the delete invitation = unrecoverable loss of folded/retrievable originals for exactly the legacy sessions this tool targets. Guardrail only — no import-semantics change.

Wire protocol, persistence schema (v10), self-update, kernel: untouched.

Verification

  • npx tsc --noEmit -p tsconfig.json clean
  • Targeted suite tests/storage-migrate.test.ts (10 tests, green locally): bulk import + index-only service proof (JSON corpus deleted post-import), idempotent re-run, dry-run writes nothing, corrupt-file isolation + non-zero exit, same-id route arbitration, read-only guarantee over the whole sessions dir, drifted-without-reimport; NEW: paired-sidecar detection across dry-run/import/re-run + KEEP copy + byte-untouched proof, zero-sidecar negative gate (original closing note preserved verbatim), orphan-sidecar listing (uncounted, non-failing)
  • Full-suite verdict: remote CI on this head

@ranxianglei ranxianglei changed the title feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) Nov 4, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 这是 #5599 设计评审线程里 owner 点名的交付物(「麻烦提交一个pr 专门用来迁移json到sqlite的」),open/closed 全量检索 storage migrate / json→sqlite import 无其他实现 PR 或同机制 issue。

标签: scope/storage(owner 请求的功能类 PR,非缺陷,不打 bug/严重度标)。

开始核对:拉分支审 diff、本地定向验证(typecheck + 新增测试套件)、确认 CI 状态,结论随后按八问回复。

@ranxianglei
ranxianglei force-pushed the 2026-11-04_storage-migrate-json-to-sqlite branch from cf3895a to 4092d1b Compare November 4, 2026 03:14
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解 + 顺带修掉了 CI 红(auto-fix lane 1/3)。分支 2026-11-04_storage-migrate-json-to-sqlite 已 force-push:cf3895a0c → 4092d1b86,与 master 的 merge 状态已验证为干净。

本轮做了什么

① 冲突解决(rebase 到 origin/master,落后 143 个提交)

② 发现并修复了 CI 红(先于本次 rebase 就存在)

  • 本地定向跑 tests/storage-migrate.test.ts:6 个用例挂 2 个。
  • 关键证据:我把 PR 原始头 cf3895a0c 单独检出实跑了一遍——同样这 2 个用例、同样的失败。即这个套件在原始分支上就是红的,只是合并冲突导致 CI 从未运行过,一直没被发现。与我的 rebase 无关。
  • 两处都是测试断言与存储层既有行为不符(两个机制在本 PR 分叉点之前就已存在且未变过):
    1. "already-indexed sessions report indexed without re-writing":漏了 dropIndex()——写会话时 store 自己的首次打开就触发了统一存储的首开自动导入,索引已被填好,第一次 migrate 自然只 verify 不 import(imported=0 ≠ 2)。补上 dropIndex(dir),与该文件其余所有用例相同的隔离方式。
    2. "migrate never modifies any file under the sessions dir":快照排除规则只排了 index.db/-wal/-shm,没排完整性校验的 sidecar index.db.chk(src/storage/integrity-check.ts,migrate 开关 store 时引擎会重写它)。把排除放宽到整个 index.db 家族(前缀匹配),与该行既有注释的本意一致。
  • 独立标注 commit:test: storage-migrate suite — align two stale assertions with existing storage behavior (#5599)(仅 +4/−1 行,测试意图不变)。

③ 验证

  • typecheck 干净(tsc --noEmit);定向套件 6/6 全绿;全仓回归交给 CI(本地不预跑全量)。

结构化结论(八问)

  1. 是什么问题:PR 分支落后 master 143 个提交产生合并冲突;同时该 PR 自带的新测试套件有 2 个用例基于错误假设写成,一直是红的(CI 因冲突从未运行而未暴露)。
  2. 什么原因导致的:冲突是常规漂移(cli.ts 热文件被 kernel authorization request: release v0.1.211 bundles kernel 0.0.176 (#5415 abort-absorb nudge debt) #5428 与本 PR 同时改到返回语句区);测试红是作者写用例时没考虑存储层已有的「首开自动导入」和 index.db.chk sidecar 这两个既有机制。
  3. 影响范围与严重程度:冲突面 = 整个 PR 无法合入;测试红面 = 该文件 2/6 用例,均为断言口径问题,不涉及 bili storage migrate 功能代码本身(功能路径由其余 4 个绿用例覆盖且行为未动)。无用户可见影响。
  4. 怎么修复的:冲突取两边意图并集(非取舍);测试按套件自身既有模式做最小对齐(dropIndex 隔离 + index.db 家族整体排除),不动任何产品代码。
  5. 是否有回归问题:无。rebase 后 diff 与原 PR 逐文件一致;两处测试修改只改变测试自身的测量方式,不改变被测行为的定义;高危模式(语义漂移/fail-fast 变吞错/阈值时序输出格式变化)逐项核对均不涉及。
  6. 是否破坏缓存:不涉及(纯 CLI 参数解析返回项 + 测试文件改动)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:冲突已清、已知红已修,交棒给 CI 全量把关;若 CI 再红,按 lane 规则继续(attempt 2/3)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

两个红 check 已修(auto-fix lane,增量推送 4092d1b86 → 2017098c5,两个独立标注 commit)。两处根因都在 PR 自身(原始头上就是红的,与 rebase 无关),均为机械修复,未动任何产品行为。

逐项

① unused-exports — INTERNAL_ONLY src/persist.ts: LegacyMigrateSessionResult

  • 该类型全仓只在 src/persist.ts 内部引用(导出接口 LegacyMigrateReport 结构化地用到它;测试文件是自己本地镜像了一份接口,并不 import 它)。按门禁脚本的建议去掉行首 export(不删声明、不加豁免)。
  • commit:fix: unexport LegacyMigrateSessionResult (unused-exports gate, #5599)

② docs-sync — 生成物漂移,定位到一行:storage.archiveAfterDays 描述里的 bili storage migrate 文档句

  • 根因:PR 当初手改了 CONFIGURATION.md/zh-CN.md 的生成区来记录新 CLI,但没动单一事实源 website/config-reference/*.yaml——docs-sync 守卫从种子重新生成时必然把这句冲掉。正确修法是把句子写进种子再重新生成:
    • EN 句子逐字取自 PR 原提交 → 重新生成后 CONFIGURATION.md 与 PR 提交字节一致,零 diff;
    • ZH 句子同样落进种子,顺手把该句的半角标点(,还有 / (#5599: / ;)规范为全角中文标点——内容一字未变,仅标点(commit message 里已标注)。
  • commit:docs: seed the storage-migrate CLI note into config-reference and regenerate (docs-sync gate, #5599)

验证:两个门禁脚本本地复跑均干净(unused-export gate clean / generate 后四件生成物零漂移);tsc --noEmit 干净;定向套件 tests/storage-migrate.test.ts 6/6 绿。全量回归交 CI。

结构化结论(八问)

  1. 是什么问题:CI 两个守卫红——persist.ts 有一个无人外部引用的导出;CONFIGURATION 文档手改了生成区却没更新 yaml 种子。
  2. 什么原因导致的:都是 PR 首次落地时的疏漏(导出本可不开口;文档应走种子→生成流程),冲突导致 CI 从未运行所以一直没暴露,本次 rebase 后 CI 首跑才现形。
  3. 影响范围与严重程度:仅 CI 门禁面;无用户可见影响,功能代码零改动(unexport 的是一个纯类型声明)。
  4. 怎么修复的:导出收窄为文件内可见;文档句子从手改产物迁移到单一事实源(yaml 种子)并重新生成,让「种子→产物」链路闭合。
  5. 是否有回归问题:无。unexport 不影响任何调用方(已全仓 grep 证实);EN 文档字节不变,ZH 仅标点规范化;高危模式逐项不涉及。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及(只是给既有键 storage.archiveAfterDays 的描述补上了 PR 本就打算写的 CLI 说明)。

判定:已知红全部修完并已推送,等 CI 重跑;若再有红,按 lane 规则继续(attempt 2/3)。

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-04_storage-migrate-json-to-sqlite (9f3f386)

Latest build — install this exact version

npm install -g billion-context@0.0.10080-pr.5636.d76284a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
9f3f386 billion-context@0.0.10080-pr.5636.d76284a build
f8b50ff billion-context@0.0.9630-pr.5636.2efa3d5 build
9a24eee billion-context@0.0.9509-pr.5636.059c5cb build
e38340b billion-context@0.0.9227-pr.5636.6d215ab build
31f96e8 billion-context@0.0.9173-pr.5636.e127ee6 build
23be109 billion-context@0.0.8872-pr.5636.4942451 build
4b18642 billion-context@0.0.8819-pr.5636.21bc5dd build
7c99bd8 billion-context@0.0.8155-pr.5636.b4abde3 build
c41a33f billion-context@0.0.8152-pr.5636.03774cf build
8616d28 billion-context@0.0.8089-pr.5636.aa3324c build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5636.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)
🔶 需要决策

背景一句话:bili storage migrate 对同一 session id 的多个 JSON 文件挑代表文件时用的是「savedAt 最新者胜」,而 boot 打开会话时的真实仲裁顺序并不完全一样(请求数降序 → nextBlockId 降序 → 精确路由文件优先 → savedAt 降序;flat 文件则是磁盘遍历顺序里第一个可读的)。

待决问题:migrate 的代表文件选择要不要对齐 boot 的真实仲裁顺序?

  • 选项 A — 维持现状(savedAt 优先):零改动。代价:同 id 多文件的边缘场景下,导入可能选中与 boot 实际加载不同的代表,报告对该健康会话给出假 drifted + exit 1;用户一打开该会话即自愈(ahead-probe 会重新导入更丰富的那份),全程只读、无数据丢失风险。
  • 选项 B — 对齐 boot 真实顺序:改选择逻辑跟随 boot 的仲裁次序 + 调整对应的一个测试钉。代价:小工作量(约 20 行 + 测试);收益:彻底消除这类假警报,且符合 Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599 钉住的意图(「按 boot 的 survivor rule 仲裁」)。
  • 推荐:B。不决定的默认行为:保持 A——PR 其余部分完全可合并,此项不阻塞。

结构化八问(本轮工作 = PR 审核 + 冲突解决 + CI 转绿):

  1. 是什么问题:不是缺陷,是 owner 点名的功能交付(Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599):新增 bili storage migrate [--dry-run],把全部 legacy JSON 会话一次性批量导入统一 SQLite 索引并逐会话摘要对账。我的工作是审核这份 PR、解决它与 master 的冲突、把 CI 修绿并报告结论。
  2. 什么原因导致的(审核中暴露的三个问题):①分支基于落后 143 个提交的 master,src/cli.ts 一处冲突(master 加了 -i/-s posture 块、PR 加了 storageAction,两者都是加法);②docs-sync 门禁红(config-reference 种子缺 migrate 一句);③static-gates 红——dist-external 扫描器把这个 PR 的错误消息字符串「…between scan and import」(以单词 import 结尾)在压缩成单行后误认成了外部 import。这是扫描器的缺陷,不是 bundle 真的引入了外部依赖(dist 本身自包含)。
  3. 影响范围与严重程度:功能面只新增一个 CLI 入口点,既有行为零变更;CI 阻塞已解除(本地复现转绿)。扫描器修复严格收窄检测面(只丢弃含「不可能出现在合法模块标识符里的字符」的捕获),不可能漏掉真外部 import。严重程度:无。
  4. 怎么修复的:①冲突按并集解(两边加法都保留);②分支上另一车道已提交的修复(rebase 到 master、两个陈旧测试断言对齐、未用类型取消导出、docs 种子)直接采纳,我本地重复的版本弃掉;③我的提交 33a37d0:给扫描器加模块标识符字符集过滤 + 双向回归测试(假阳性消失、真 side-effect import 仍被抓),门禁测试 10/10 过。
  5. 是否有回归问题:逐项审过既有行为——cli parseArgs(kernel authorization request: release v0.1.211 bundles kernel 0.0.176 (#5415 abort-absorb nudge debt) #5428 posture 行为保留,cli-parseargs 套件验证)、persist.ts 纯新增方法(loadSync/importRecord 路径未动)、shadow.ts verifyImported 为新增只读方法、storage-maintenance 纯新增;无阈值/时序/输出格式漂移。扫描器改动有 9 条旧测试 + 1 条新测试双向钉住。
  6. 是否破坏缓存:不涉及(缓存/重试/增量游标/去重键均未触碰)。
  7. 是否改了设计方向:不涉及(纯加法 CLI 入口,复用既有 bootstrap/ingest/verification 机制)。
  8. 是否引入了新的配置:零配置键、零环境变量;仅一个新 CLI 动作词 storage migrate(PR 正文已披露)。

CI 状态:新 head 33a37d0 已推送,全矩阵重跑中(上一红 head 上唯一失败就是 static-gates,其余门禁绿或运行中)。本地证据:build + dist-external 门禁 OK、typecheck 干净、定向套件(storage-migrate/cli-parseargs/storage-maintenance-cli/门禁)全绿;全量判定以远程 CI 为准。

标签说明:镜像侧标签 API 未实现(shim 返回 endpoint not implemented,本工作区也没有 add-label.sh),已在 GitHub PR #5636 上打好 needs-decision + scope/storage。

判定:CI 转绿后即可合并(合并按惯例由人来)。唯一开放项是上面的 🔶 survivor-rule 对齐——不阻塞,默认 A。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

4 个红 check 处理完毕:3 个已修并推送(33a37d07b → 30f2c14e4,fast-forward),第 4 个(windows shard 5 的两个失败)已证明与本 PR 无关,证据如下。另:期间有一个并行 CI-lane 实例推了与我同类的 scanner 修复(33a37d0,字符集更宽且带回归测试),我保留它的、丢弃了我重复的那个,在其上叠了另外两个不重叠的修复——分支无分叉、无 force-push。

逐项根因与修法

① static-gates(ubuntu-24 / windows-22 两个 job,同一根因) — 本地完整复现:

  • 机制:src/persist.ts 的迁移错误消息以单词 import 结尾;tsup 压缩后与下一条语句拼成一行 ...Error("representative file changed between scan and import");decodedBytes=parsed.bytes;...throw new Error("unified sidecar unavailable"),dist 门禁的 side-effect-import 文本扫描把 import" 当成导入、捕获到下一个引号为止的整段垃圾"specifier",三个 bundle 各报一条同款假阳性。该字符串是 PR 独有,master 不受影响。
  • 修复(双层):
    • scanner 侧(并行 lane 实例已推 33a37d0):collectRefs 过滤含不可能出现在合法 npm specifier 中的字符的捕获,附回归测试。我验证其字符集是超集后直接采纳;
    • 源头侧(我的 3c7df65):消息改写为 "...between scan and import attempt",消除压缩拼接地雷。行为变更披露:用户可见错误文案 old representative file changed between scan and import → new representative file changed between scan and import attempt,仅措辞、语义不变,无任何测试钉住旧文案。

② test(windows-22, shard 2) — PR 自己的测试的平台假设:tests/storage-migrate.test.ts:206 断言 dup.file.startsWith("openai/"),但 file 来自 path.relative,Windows 上是反斜杠(CI 报错原文:newer route wins: openai\b.example_...)。改为 startsWith("openai" + path.sep)(67448d876)。Linux 上原本就绿。

③ test(windows-22, shard 5) — 两个既有测试,均不在本 PR diff 内,判定与本 PR 无关:

  • tests/issue4804-ws-delta-e2e.test.ts「unknown checkpoint id」:failureType=cancelledByParent,错误 Promise resolution is still pending but the event loop has already resolved(约 9.6s)= 挂起被取消,典型 Windows 高负载时序 flake;
  • tests/seq-interval-lane.test.ts:断言 one block recorded: [] 0 !== 1(fold 未记块,kernel/loop 域)。
  • 证据:两个都在本地 Linux 实测通过(2/2 与 1/1);本 PR 对存储侧的唯一改动是 shadow.ts 里一个纯新增的只读方法 verifyImported()(不改任何既有方法),结构上影响不到这两条车道;master 自身也有红工作流(e2e-pi-codex-ws @ 23a826d failure)。
  • 处置:不修(修无关既有测试超出本 lane 机械修复范围,且本地无法复现),等本轮重跑;若下轮仍红即按继承红流程走(attempt 2/3,核 master windows shards 状态,必要时 merge ci-fixes)。

验证(最终树 30f2c14)

rebuild + check-dist-external OK、tsc 干净、tests/storage-migrate 6/6、checker 自带套件 13/13(含并行实例新增的回归测试)。全量回归交 CI。

结构化结论(八问)

  1. 是什么问题:4 个红 check——dist 文本扫描误报(×2 平台)、PR 自己的 Windows 路径断言、两条无关既有测试在 Windows 挂掉。
  2. 什么原因导致的:错误消息以 "import" 结尾在压缩后与后续语句融合成伪导入形态;测试硬编码正斜杠;shard 5 两条为既有测试的 Windows 时序不稳/疑似继承红(非本 PR 引入)。
  3. 影响范围与严重程度:static-gates 阻断合并(门禁级);shard 2 仅 Windows 上误杀本功能测试;shard 5 对本 PR 代码正确性零影响,属 CI 环境噪音。
  4. 怎么修复的:扫描器加"不可能 specifier"字符集守卫(带回归测试)+ 源头消息改写双保险;测试断言改用平台分隔符;无关失败不碰,留证据观察重跑。
  5. 是否有回归问题:产品行为零变化;唯一用户可见变化是一条错误消息措辞(已按旧值→新值披露);新过滤器不可能掩盖真实外部依赖(合法 npm specifier 不含那些字符)。高危模式逐项不涉及。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:已知可修的红全部修完;windows shard 5 的两个失败带着「非本 PR 引入」的证据交回 CI 重跑,若复现则按 lane 规则升级处理。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)
CI 进展收口(当前 head 30f2c14):

  • static-gates(ubuntu-24)已绿——就是上一轮红掉的那条门禁,我的扫描器修复(33a37d0)在 CI 上验证通过;windows 腿还在跑。
  • 期间另一车道在我提交之上补了两个提交:① 3c7df65 把触发误报的那句错误消息改写(不再以单词 import 结尾)——和我修的扫描器是同一问题的两层防护,互不冲突、都保留(消息改写消掉这个触发点,扫描器加固关掉这一整类假阳性);② 30f2c14 修一个 storage-migrate 测试的路径分隔符断言(windows CI)。
  • 全矩阵现状:0 失败,11 个测试分片 + windows static-gates 运行中(e2e 独立工作流已在该 head 上全绿)。按惯例不轮询了,转绿即可合并。

🔶 survivor-rule 对齐的决策请求维持上一条不变(推荐 B,默认 A,不阻塞)。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE. Clean, correctly-scoped additive feature. Reviewed the live head — the lane's 33a37d07b was stale; I reviewed 30f2c14e4 and now 8616d28e1 (after the on-spot test below). This is an independent read of the diff plus the pre-existing machinery it reuses, not a restatement of #5599.

Structured review (eight questions)

  1. What it is. A new bili storage migrate [--dry-run] [--json] that bulk-imports every legacy JSON session under the sessions dir (incl. -cold) into the unified SQLite index and digest-reconciles each against its source file. It closes the gap in #5599 where today's JSON→SQLite migration is purely lazy (organic import on first open), so a dormant install upgrading past a future JSON-reader removal could lose never-opened old files.

  2. Root cause / mechanism. Capability add, not a bug. Reuses boot's survivor rule (newest savedAt, lexicographic tie-break) for same-id routes, the existing organic-ingest path (shadow.importRecord), and a new side-effect-free verifier (UnifiedShadow.verifyImported) that runs the same BatchedViewVerification as the steady-state sample with latch/freeze stripped.

  3. Blast radius & severity. Additive CLI action word only. Sole delta to existing behavior: the unknown-action help string (try "bili storage gc" → try "bili storage gc | migrate") — non-semantic. Zero new config keys/env/flags (--dry-run/--json are pre-existing globals). Wire protocol, persistence schema (v10), self-update, kernel: untouched.

  4. How it works — correctness checks I ran. Confirmed every reused helper exists with the expected signature (fileNameMatchesId, parseBootPayload, readRawFileWithBytes, loadUnified, unifiedListSessionIds, recordToLegacyView, walkJsonFilesSync). Load-bearing invariant verified: persist.ts:603 const legacyLoad = recordToLegacyView is exactly what's wired into the shadow, so ingest and post-import verification compare identical views — fresh imports match, and drifted fires only on genuine file-vs-index divergence (no false positives). verifyImported (shadow.ts:780) is byte-for-byte the verify-sample loop minus side effects. Dispatch guard (cli.ts:765) and the read-only guarantee (only index.db written; JSON never deleted/renamed/quarantined) hold. The check-dist-external.mjs NOT_A_SPECIFIER fix is safe — it rejects only captures containing characters impossible in a bare npm specifier — and is regression-pinned both directions.

  5. Regressions. None found. Walked each touched path against prior behavior; the only change is the help-text wording above. No fail-fast→swallow, threshold/timing/output-format drift. No console.error added to server modules (house rule); new code reuses the same fail/indexDbSize/replicasAvailable plumbing as the sibling gc/archive commands.

  6. Cache invalidation. None — no cache/retry/cursor/dedup-key logic touched.

  7. Design direction. No pivot. Deliberately read-only over the JSON corpus (corrupt files reported, never quarantined/deleted); cleanup stays an operator decision after a clean report. Consistent with the single-writer storage-lane contract.

  8. New configuration. None — one new CLI action word, explicitly not a config knob.

Tests. The six shipped tests have teeth (index-only service proof by deleting the JSON corpus, idempotent re-run, dry-run safety, corrupt isolation + exit 1, route arbitration, read-only invariant via before/after deepEqual). One gap I closed: none exercised the drifted status — the feature's headline "digest reconciliation." Added a deterministic pin (bump a non-request stat in the source after import so it diverges without tripping the ahead-probe). Verified locally on node:sqlite — 7/7 green; the log shows the second run reports 0 imported, 1 drifted, i.e. drift detected and not re-imported.

已顺手修:补齐 drifted(摘要对账分歧)路径的回归测试,本地 7/7 通过 + commit 8616d28

Not auto-mergeable (§7.4: feat + build-gate change → human merge), but nothing blocks approval. At review time the CI test lanes were still running with 0 failures; static gates green.

Note: posted as a comment rather than a formal “Approve”, because GitHub forbids approving your own PR (the reviewer runs under the same account). The verdict above stands.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

✅ 专职 reviewer 收口(独立端到端审,非照抄前序结论)。审的是 live head 30f2c14e4(任务头里的 33a37d07b 已过期)。判定 APPROVE;已发到 GitHub PR #5636 —— 因同一账号不能对自己的 PR 点正式 Approve(GitHub 限制),故以 review 评论形式带出 verdict,八问齐全。

独立核过(自己读 diff + 追既有机制,非复述前序):

  • reused helpers 签名逐一核对无误(fileNameMatchesId/parseBootPayload/readRawFileWithBytes/loadUnified/unifiedListSessionIds/recordToLegacyView/walkJsonFilesSync);
  • 关键不变量:persist.ts:603 const legacyLoad = recordToLegacyView 正是喂进 shadow 的那个函数,所以「导入」与「事后校验」比的是同一视图 → 新导入必匹配,drifted 只在真分歧时触发,不会假报;
  • verifyImported(shadow.ts:780)确认是 verify-sample 循环去掉 latch/freeze 副作用的只读副本;
  • CLI dispatch guard(cli.ts:765)与「对 JSON 只读、只写 index.db」成立;check-dist-external.mjs 的 NOT_A_SPECIFIER 收窄安全(合法 npm specifier 不含那些字符)且双向回归钉住;无新增 console.error;en/zh 文档同步;零配置键 / 零 wire / schema(v10) / kernel 改动。

已顺手修(唯一实质缺口):drifted(摘要对账分歧)路径原本没有任何测试——而这正是本 PR 主打的「逐会话摘要对账」。补了确定性回归钉(导入后改一个非请求 stat → 摘要分歧但不触发 ahead-probe 重导),本地 node:sqlite 实测 7/7 绿,第二轮日志 0 imported, 1 drifted(即检出分歧且未重导)+ commit 8616d28e1。

开放项(不阻塞,维持前序 🔶 不动):survivor-rule 代表文件选择是否对齐 boot 真实仲裁次序(floor 12/17,推荐 B、默认 A,自愈、只读、无数据丢失)——我的 APPROVE 不预判它,归 owner 拍板。CI 审时 0 失败、测试分片运行中、static gates 绿。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

本次运行:head c41a33f(run 41841727312)6 个 test 分片红、guards 全绿。已定位并修复其中 5 个确定性失败(同一根因的两族测试回归),推送 7c99bd8ce(fast-forward,推前确认远端无并行新提交)。ubuntu24-s3 为环境性进程杀死,无代码动作。等 CI 在新 head 上重跑。

结构化八问

  1. 是什么问题:并行车道实例在生产提交 dbc4ad8d0(一次性 ledger re-stamp 拒绝不完整账本——导入的折叠会话里 ghost 消息没有行,盖戳会把历史引用重定向到新消息体,属有意修复且已有 S16 测试钉住)之后,三个既有测试仍编码着旧语义,在全部 OS 上确定性失败:issue4986-digest-drift-ref-order 1 个 + storage-unified 2 个子测试(跨 ubuntu24-s2 / ubuntu22-s2 / ubuntu22-s3 / win22-s3 / win22-s5 五个分片)。
  2. 什么原因导致的:(a) Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 测试的前置断言依赖冷加载时的 re-stamp 制造出 byRaw/byRef 键序分歧;守卫生效后折叠语料加载后保留 legacy refs,两序一致,前置断言(防空转检查)直接挂。(b)(c) storage-unified 的 parity oracle(alignedLegacyDigest)无条件调用 restampRefsToLedger,而生产冷加载路径现在会跳过盖戳 → 期望摘要(盖戳态 5d598edb…)与生产实际(legacy refs 态 3026f3d7…)不匹配。
  3. 影响范围与严重程度:仅测试层(CI 红),无生产行为变化——我的两个 commit 只动测试文件。守卫本身的行为变更(导入折叠会话冷加载保留 legacy ref 号,不再重钉到 ledger seq)是并行提交的有意修复,其 commit message 与 S16 测试已披露,此处补记备查。
  4. 怎么修复的:让测试对齐守卫语义而非回退守卫——① oracle 改为接收打开的 UnifiedStore probe,仅在守卫谓词通过(每个 mapped id 至少有一行,含 retired)时才应用对齐,与 SessionStore 冷加载路径同一判定;② Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 改为加载后在内存中把 byRaw 按反序重建(ref 值不动、只换键序),由 finishPlan 冻结成行 ord,精确复原被测形状。未动共享 golden 语料,未动任何生产代码。
  5. 是否有回归问题:审过两个被改文件的既有行为——oracle 仅两处调用点(同文件内,After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 等价性断言语义保持:完整账本仍走对齐路径);Unified-store ingest verification reports DIGEST DRIFT and re-ingests the full tail for sessions whose in-memory ref order diverges from stored row order #4986 改动封闭在本文件内,verify 不变量(顺序无关摘要)原样保留。定向验证:四个相关套件 31 测全过(1 个条件跳过)+ tsc 干净。高危模式逐项:无语义漂移、无吞错、无阈值/时序/输出格式变化。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标/去重路径)。
  7. 是否改了设计方向:不涉及——是把测试对齐到同一条车道已落地的设计决策。
  8. 是否引入了新的配置:不涉及。

环境性失败说明:ubuntu24-s3(job …42813)日志在 ok 1815 - isMitmHost… 处截断、无任何 TAP 失败行、进程中途被杀(OOM/runner 类),非代码问题,重跑即可见。

协作说明:本轮失败根因来自同 PR 另一实例的并行提交(8616d28e1/dbc4ad8d0/c41a33fc,均已核对内容与动机);推送前 fetch 确认远端无更新,fast-forward 干净落地。若下一轮 CI 仍有红,按车道规则继续(attempt 2/3)。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Re-review (round 2) — head 7c99bd8ce. Verdict: APPROVE. Round 1 approved the feature base; this round covers the 4 commits added on top of my 8616d28e1 fix: dbc4ad8d0 (code) + 3 test commits.

What the new code does (dbc4ad8d0: src/persist.ts + src/storage/shadow.ts + src/storage/store.ts). The one-time cold-load ledger re-stamp (restampRefsToLedger) now runs only when the ledger is COMPLETE — every non-BLOCKED mapped msg-id must own ≥1 unified row (live OR retired). Otherwise it fails safe to keeping legacy refs (JSON-load parity); a later cold load retries once rows exist. New UnifiedStore.backedMsgIds() = chunked (500) parameterized SELECT DISTINCT msg_id … IN (…). This closes a real corruption path: an imported collapsed ledger (folded ghosts own no rows, tail re-based at seq 0) would otherwise have its ghost bindings evicted onto fresh numbers and saved citations re-targeted at newer bodies.

Why it's safe / no regression:

  • Native sessions always qualify — the historian sweep retires covered rows instead of deleting them (#4390), so every mapped id keeps a row and the guard passes → restamps exactly as before.
  • Fail-safe direction throughout: store unavailable (stats.disabled, defaults false) or query error → null → skip → legacy refs. Mirrors the ~19 existing if (!this.shadow || this.shadow.stats.disabled) return null sibling guards in persist.ts.
  • SQL is parameterized (no injection), bounded chunking.

Tests (have teeth):

  • S16 (tests/seq-ledger-pr3.test.ts): both directions — the incomplete ledger keeps legacy refs (refLedgerAligned !== true, restampSuperseded === undefined) AND pins citation integrity (m00001 resolves to o1's original body, not the tail); the control half proves a complete ledger still aligns (the guard discriminates on row coverage, not on folds).
  • tests/storage-unified.test.ts oracle mirrors the same guard so the "full serve = legacy serve + alignment" equivalence stays valid under the changed path.
  • tests/issue4986-digest-drift-ref-order.test.ts honestly synthesizes its divergent shape in-test (cold-load no longer produces it post-fix) and keeps the anti-vacuity precondition.

Local evidence (targeted run, node v22.23.2 / node:sqlite): the 3 touched files → 23 pass / 0 fail / 1 skip, exit 0; S15 unchanged-green confirms no regression to the normal restamp path. Full-suite verdict = remote CI (0 failures at this head, test shards running; static gates green).

八问(本轮 delta + 累计)

  1. 是什么问题:migrate 导入的「折叠账本」在冷加载的一次性 re-stamp 时会被错误改写(幽灵绑定被挪到新号、已存引用重定向到新正文)。
  2. 什么原因导致的:re-stamp 原先只判 seqByRaw 非空就执行,未校验账本完整性;导入的折叠账本里被折叠的消息没有行、尾部从 seq 0 重建,恰好落进这个缺口。
  3. 影响范围与严重程度:仅影响「经 migrate 导入且含折叠」的会话冷加载;不修则引用错绑(citation 指向错误正文)= 数据语义损坏。触发前提是先跑过 migrate,故非全量用户面,但一旦发生属 p1 级隐患。
  4. 怎么修复的:re-stamp 前加完整性闸——每个非 BLOCKED mapped id 必须有行(存活或已退役),否则保持 legacy refs(fail-safe 方向),后续冷加载在有行后自动重试。
  5. 是否有回归问题:无。原生会话恒满足(#4390 只退役不删行)→ 行为不变;store 不可用 / 查询失败 → 保持 legacy refs(S15 仍绿证明正常路径未动);高危模式(语义漂移 / fail-fast 变吞错 / 阈值时序输出格式变化)逐项不涉及。
  6. 是否破坏缓存:不涉及(re-stamp 是加载期引用对齐,不动缓存 / 重试 / 增量游标 / 去重键)。
  7. 是否改了设计方向:不涉及(把既有 re-stamp 的触发条件收窄为安全子集,不转向)。
  8. 是否引入新配置:零配置键 / 环境变量 / 默认值改动。

开放项(不阻塞,维持前序 🔶 不动):survivor-rule 代表文件选择是否对齐 boot 真实仲裁次序(floor 12/17,推荐 B、默认 A)——与本次 re-stamp 完整性闸是不同机制,我的 APPROVE 不预判它,归 owner 拍板。

Note: posted as a comment rather than a formal “Approve” because GitHub forbids approving your own PR (reviewer runs under the same account); the verdict above stands.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

✅ round-2 复审收口(独立审 delta,非照抄)。新 head 7c99bd8ce 在我上轮 8616d28e1 之上多了 4 个提交(dbc4ad8d0 代码 + 3 测试)。判定 APPROVE,已发 GitHub PR #5636(review id 5770138907,评论形式——同账号不能对自己 PR 点正式 Approve)。

本轮唯一实质改动 = dbc4ad8d0:冷加载一次性 ledger re-stamp 前加完整性闸——每个非 BLOCKED mapped id 必须有行(存活或已退役),否则 fail-safe 回退保持 legacy refs。核过:原生会话恒满足(#4390 只退役不删行)→ 行为零变化;store 不可用/查询失败→保持 legacy refs(与 persist.ts 里 ~19 处既有 stats.disabled 守卫同型);SQL 参数化无注入、分块有界。修的是真损坏路径(migrate 导入的折叠账本被错误 re-stamp → 引用错绑)。

测试有牙:S16 双向钉(不完整→保 legacy refs + m00001 引用仍解析到原始正文而非尾部)+ 完整账本仍对齐的控制半;oracle 同步镜像该闸;#4986 诚实地在测试内合成其形状并保留反空转前置。本地定向跑 3 个改动文件 23 pass / 0 fail / 1 skip(exit 0),S15 不变绿证正常路径未动;CI 该 head 0 失败、分片运行中、static gates 绿。

本轮无需顺手小修(delta 干净);无 kernel/ 改动 → 不触发授权卡点。开放项维持前序 🔶 survivor-rule(floor 12/17,推荐 B、默认 A、不阻塞)——与本次 re-stamp 闸是不同机制,我的 APPROVE 不预判它,归 owner 拍板。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(bot) 🏷 冲突已解决并推送(attempt 1/3)。本条按七问汇报。

说明:同一份摘要此前误发到了线程 #5636(该线程 floor 11,已就地标注);以本条为准。

核对说明:镜像线程标注的上游号 5599 实为该 PR 引用/关闭的 issue 编号;真正的上游 PR 是 GitHub #5636(head 2026-11-04_storage-migrate-json-to-sqlite,已按标题精确匹配并在 GitHub 实况核实)。后续处理本线程请以该 PR 号为准。

做了什么:把分支 rebase 到当前 master(4be21b08ed),解决全部冲突,force-with-lease 推送:84fa38814e → 4b186429fd(推送前已 fetch 确认分支无他人并发提交)。本地验证:typecheck 干净、build 成功(dist-external OK)、定向跑本 PR 回归套件 tests/storage-migrate.test.ts 7/7 通过;全量套件留给 CI。

七问

  1. 是什么问题:PR 与 master 合并状态 dirty(有冲突),无法合并。
  2. 什么原因导致的:master 在分支基点(11-04)之后完成了文档布局迁移(根目录只留四个核心文件,CONFIGURATION.md/CONFIGURATION.zh-CN.md 移到 docs/configuration*.md,根路径留重定向 stub),而该 PR 的三个文档 commit 仍在编辑旧根路径文件——纯文档路径漂移,代码零冲突。
  3. 影响范围与严重程度:仅合并被阻塞,无任何行为影响(冲突全部落在文档文件);不修则 PR 永远合不进。
  4. 怎么修复的:rebase 后逐点保住两侧意图——① 根路径文件一律保留 master 的 stub;② migrate 说明文字改由 yaml 种子(website/config-reference/process.yaml)承载并重跑 tools/gen-config-docs.mjs 生成到新位置 docs/configuration*.md(已验证生成物相对 master 恰好只有 archiveAfterDays 一行变化,无既有漂移);③ 最后一个 commit 的「子进程部署」段落按原锚点重新落进 docs/configuration*.md。非文档文件与原 PR 意图逐字节一致(最终 diff vs master = 16 文件 +714/−31,与原 PR 足迹相同,仅两个根 CONFIGURATION 换成 docs/ 对应文件)。
  5. 是否有回归问题:无。代码、测试、脚本均未经过我的手工改动(冲突只在文档);文档内容 en/zh 双语完整保留(migrate 条款 + 子进程部署段落各就位);typecheck/build/定向套件本地绿。高危模式逐项过:无语义漂移、无阈值/时序/输出格式变化。
  6. 是否破坏缓存:不涉及(纯文档重定位 + rebase,未触碰任何缓存/存储逻辑)。
  7. 是否改了设计方向 / 引入新配置:不涉及;无 kernel/ diff,未触发任何授权门禁,无需决策项。

当前实况(推送后即时核实):GitHub mergeable=true,mergeable_state=blocked(冲突已清,blocked = 等检查/批准,属正常态);新 head CI 已触发:25 success / 12 skipped / 12 in_progress,暂无失败。镜像缓存此刻仍显示旧 sha(poller ~60s 刷新延迟),稍后自会跟上。CI 全量结论以远程为准,红了再说。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Review — ROUND 3 @ head 4b186429f (post-rebase)

Verdict: APPROVE ✅

Third independent pass. Since my last review (7c99bd8ce) the branch was rebased onto current master — merge-base == master, so the PR's own contribution is exactly 16 files, zero under kernel/. The large 7c99bd8ce..head range (~415 files incl. kernel/) is purely inherited master churn (fastpath v6 + its revert, docs→docs/ reorg, kernel 0.0.18x bumps, identity-harvest, msgid-lane, …), not this PR's work.

What actually changed since round 2: one new commit on top of the rebased base.

  • 4b186429fd docs: note env-tier limits in child-process deployments (field finding from #5599) — pure docs, +4 lines with en/zh-CN parity: a correct "Child-process deployments" paragraph explaining that a spawned child inherits its env block at spawn time (vars set after host launch, e.g. Windows setx, only reach new children after the host restarts) and that the config-file tier is the reliable override channel there. Accurate, additive, no code/behavior change. Cleared.

The other 14 commits are the same logical work already reviewed/approved in rounds 1–2 (rebased → new hashes), including my round-1 regression test b7bcfd56e0 (pin the drifted path) and the round-2 re-stamp completeness guard — both present at head.

Rebase regression check (the real risk this round). Hot files src/persist.ts / src/storage/shadow.ts / src/storage/store.ts were heavily modified by master (kernel bump, identity-harvest, msgid-lane). Verified each touched path's existing behavior survives intact at head:

  • re-stamp completeness guard chain byte-intact: src/persist.ts:2101-2102 (mapped filter + unifiedBackedMsgIds) → :2337 → src/storage/shadow.ts:1078 → src/storage/store.ts:2996; the if (!this.shadow || this.shadow.stats.disabled) return null fail-safe idiom remains pervasive;
  • migrate command entry + help + dispatch (src/cli.ts:242 storageAction:"gc"|"migrate", runStorageMigrateCommand import) intact;
  • digest-reconciliation statuses + exit-code rule (src/storage-maintenance.ts:357-384: dryRun || (drifted+failed)===0 ? 0 : 1) intact;
  • UnifiedShadow.verifyImported (src/storage/shadow.ts:800) side-effect-free parity verifier intact; read-only-over-JSON + idempotency + exit-1-on-drifted/failed intact.
    No unintended semantic drift from the rebase. Full-suite green across all shards/OS/node versions corroborates the integration.

Check states (mirror cache, cached_sha == 4b186429f): total 30, failed 0, pending 0 — every test shard (ubuntu/windows × node 22/24), static-gates, mysql-contract, gate = success.

Config surface: zero new keys/env/flags (--dry-run/--json are pre-existing globals) — matches the PR's own disclosure. Behavior changes: none (purely additive CLI action); wire protocol, persistence schema (v10), self-update, kernel all untouched — matches disclosure. No kernel-change: declaration in the body; no kernel/ in the diff → the human-only authorization gate is NOT triggered.

Open ends from this reviewer: none. (The survivor-rule representative-file-selection question was raised by other lanes and already carries the needs-decision label — outside this review's scope, already converted-to-decision.)


八问(中文)

  1. 是什么问题:补齐 #5599 指出的缺口——JSON→SQLite 迁移目前纯惰性(首次打开才导入),休眠安装若越过未来的 JSON 读取器移除会丢失从未打开的旧文件;本 PR 新增一次性、可校验的批量导入命令。
  2. 什么原因导致的:现有迁移只在首次打开时惰性导入,缺少一次性批量路径(能力缺口,非缺陷);本 PR 复用既有引擎引导 + 导入 + shadow 校验机制补上,不另起炉灶。
  3. 影响范围与严重程度:纯增量 CLI 入口(bili storage migrate [--dry-run] [--json]),对既有会话、线协议、持久化格式零改动;未迁移安装行为不变。非缺陷、无严重度风险。
  4. 怎么修复的:同 id 路由文件按 boot 幸存者规则取代表(最新 savedAt,字典序 tie-break),代表走既有有机导入通道导入、其余报告不动;逐会话跑完整 shadow 校验(verifyImported,与稳态样本同一校验器、无 latch/freeze 副作用),状态 imported/indexed/drifted/failed/planned;对 JSON 语料只读(损坏只报告、不隔离/删除);幂等,drifted/failed 且非 dry-run 时退出码 1。
  5. 是否有回归问题:本轮重点核查 rebase 到更新 master 后热文件(persist/shadow/store)的既有行为——re-stamp 完整性守卫链字节级完好,migrate 命令/派发/digest 对账/退出码规则/verifyImported/只读保证均保持;无意外语义漂移。全量 CI 30/30 绿佐证集成无回归。
  6. 是否破坏缓存:不涉及——不触碰前缀缓存/重试/增量游标/去重键等失效策略;仅新增一个离线批量导入命令。
  7. 是否改了设计方向:不改架构/边界/数据流;沿用统一存储 + shadow 校验既有设计,仅暴露一个 CLI 动作词。
  8. 是否引入了新的配置:无新 config key / env / flag(--dry-run/--json 为既有全局项),与 PR 披露一致。

结论:可合(APPROVE)。 唯一新增提交是良性文档补充;核心逻辑经 rebase 后字节级完好且全量 CI 绿。无 kernel/ 触碰、无 kernel-change 声明,不触发人工授权门禁。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

R3 review of PR #5636 @ head 4b186429f: APPROVE — branch rebased onto current master (PR's own diff = 16 files, zero under kernel/; the ~415-file/kernel range vs my stale base was inherited master churn); sole new commit since R2 is a benign docs note (4b186429fd, child-process env-tier limits, en/zh parity); re-stamp completeness guard + migrate command + my drift-test are byte-intact through the rebase and CI is 30/30 green on the exact head; no kernel-change declaration so no authorization gate; no open ends from this reviewer. Posted to GitHub as COMMENT id 5828149945 (self-approval blocked under the owner account).

@ranxianglei

Copy link
Copy Markdown
Owner Author

需要基于最新的master 看是否有缺失 因为master也增加了很多结构

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

🔶 需要决策:对照最新 master 审计 migrate —— 发现「CCR 内容边车未吸收」+ 一处会误导删库的文案

已按你要求,对照**最新 master(ff5bdc1b,比本 PR base 4be21b08 更新)**逐条清点 bili storage migrate 是否漏了 master 新增的会话目录结构。结论:一处实质缺失(CCR 内容边车 .content-store.json 未被吸收)+ 一处会误导删库的输出文案;其余新增结构均已正确覆盖。这属语义/设计层面(跨 persist/shadow/store),按规则不擅自改,转 🔶 需要你拍板处置方式。已给本线程打上 needs-decision。

master 会话目录现有结构 × migrate 各自处理

master 结构 migrate 处理 判定
主信封 <id>.json(可嵌套协议子目录 <proto>/…) 递归 walkJsonFilesSync + fileNameMatchesId 校验 basename↔id ✅ 覆盖
.fb.json 别名形式(↔ 规范 .json) fileNameMatchesId 显式归一到规范名(persist.ts:333-341),按同 id 归组仲裁 ✅ 覆盖
热目录 + -cold 目录 两个 root 都扫(persist.ts:1965) ✅ 覆盖
崩溃残留临时名 .tmp-* walker 识别 stale temp ✅ 覆盖
迁移标记 .bili-migration-286.done(非 json) 非 .json,天然忽略 ✅ 无噪音
backups/index-*.db(SQLite 备份) 非 .json,天然忽略 ✅ 无噪音
.content-store.json 边车(#1097,存被折叠 / 可 acp_retrieve 的原始正文) discovery 直接跳过且不计数(persist.ts:1968);导入只吃信封,从不读这个文件 ❌ 缺失

核心缺失:CCR 边车没被吸收,「clean 后可自行删除」是陷阱

证据链(均在 head 4b186429f):

  1. discovery 跳过边车:persist.ts:1968 if (basename.endsWith(".content-store.json")) continue;——且在 scannedFiles++ 之前,报告里完全看不到它。
  2. 导入只吃信封:persist.ts:2019 shadow.importRecord(rec.payload)(payload=解析出的信封,不含边车)。
  3. importRecord→recordToLegacyView→buildSession 从不读盘上边车:persist.ts:4249 contentStore: parsed.forkContentStore(正常会话为空);而 ingestSession 仅按视图里的 contentStore 写 ccr_entries(store.ts:1413-1419)→ 迁移后的行 ccr_entries 为空。
  4. 对比正常保存路径 record(...,liveCcr):shadow.ts:481 if (liveCcr) legacy={...legacy, contentStore: liveCcr} → 活跃会话的 ccr_entries 是被填的;迁移 / 首次打开的 importRecord 没有这一步。
  5. 服务期 CCR 读是「unified 优先、空则回落边车文件」:persist.ts:2247-2254 unifiedCcrFor;代码自证边车才是权威源——createReplica persist.ts:2786-2790「Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) #2690 … restore without the content-store envelope → CCR cannot fetch originals」、persist.ts:3200「degrades to retrieve misses, never crashes」。
  6. 文档亦确认边车独立于索引且与会话同生命周期:docs/configuration.md:210(jsonWriter=off 时 "The per-session content-store sidecar keeps being written either way")、:2120(GC 把 <hash>.content-store.json 与主文件一起删)、:1595(CCR "no cap and no eviction, ever … shares the session's lifecycle")。
  7. 而 migrate 收尾文案在邀请删除:storage-maintenance.ts:375 "archive or delete them yourself once this report is clean."

后果:一个有折叠/CCR 内容、且其 ccr 尚未被索引完整捕获的遗留会话(正是 #5599 想批量导入的大/长会话人群),在 migrate 报 clean、操作者照提示删掉 JSON 兄弟文件后,acp_retrieve 的可检索原文 / 折叠原文会静默丢失(retrieve miss,不报错)。即 migrate 并未让这类会话在 SQLite 里自包含,与它「一次性全量导入后即可清理 JSON」的定位相悖。

诚实边界(不夸大):migrate 对 JSON 只读、运行中不破坏任何数据;丢数据是条件触发——需同时满足 ① 会话有折叠/CCR 内容 ② 该会话此前未被索引的 ccr 镜像完整捕获 ③ 操作者在 clean 信号后删掉了 *.content-store.json。只要保留 JSON(至少保留 .content-store.json)就不会丢。真正的问题是工具既不上报也不吸收边车,却给出「可删除」的 false-clean 信号。(以上为逐行静态追踪结论;选定处置方案后我会补一个「带 CCR 边车的会话 → migrate → 删边车 → acp_retrieve」的复现回归测试钉死。)

八问

  1. 是什么问题:migrate 只导入会话信封,不吸收配套 .content-store.json(存可检索原始正文),其「clean 后可删」指引会让这类会话的可检索原文丢失。
  2. 什么原因导致:导入走 importRecord(envelope),视图构建只取信封内字段、从不读盘上边车;ingest 仅按视图 contentStore 写 ccr_entries,故迁移行 ccr 为空;服务期 CCR 空则回落边车文件,边车一删即 retrieve miss。(活跃保存路径有 liveCcr 合并这一步,迁移路径没有。)
  3. 影响范围与严重程度:影响「有折叠/CCR 内容且未被索引 ccr 镜像完整捕获」的遗留会话(Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599 目标人群)。非自动丢数据,条件触发且静默。我判 P2(可用,但工具给出会误导删库的 false-clean 信号);若你认为「按工具指引即可能永久丢原文」应升 P1,请你定。
  4. 怎么修(选项):见下 A/B/C,推荐 B 先行。
  5. 是否有回归:不改代码则零回归;任一选项都只增强迁移的报告/吸收面,不动既有保存/读路径语义。
  6. 是否破坏缓存:不涉及 prompt-cache/重试/游标;ccr_entries 属存储层而非缓存。
  7. 是否改设计方向:Option A 会把「迁移=自包含」立为新契约——注意当前 master 连 jsonWriter=off 都要继续写边车,说明索引并非自包含,这是方向性决定,需你拍板;B/C 不改方向。
  8. 是否引入新配置:无新增 env/config/flag(沿用 --dry-run/--json 既有全局开关)。

处置选项

  • A|彻底自包含:让 migrate 读取每个会话配对的 .content-store.json 并喂给导入(参照 record(...,liveCcr) 扩展 importRecord/视图接受 contentStore),使 ccr_entries 落地、会话在 SQLite 里自包含;并把 digest 校验扩展到 ccr_entries。跨 persist/shadow/store,触碰 ingest 语义 → 设计级,工作量/风险最高,但彻底。
  • B|安全护栏(推荐先做):导入逻辑不动,但 ① discovery 检出配对 .content-store.json ② 按会话上报「N 个会话带 CCR 边车」③ 把收尾文案从「可自行删除」改成「必须保留 .content-store.json(或让该会话重开/重存一次以把 CCR 吸收进索引)——它是 acp_retrieve 原文的权威来源」。小改动、低风险、立即堵住误删陷阱。仍属输出/行为变更,建议你也点头。
  • C|仅文档:只在 en/zh CLI 参考里写明局限——最弱,运行时那句 "delete them yourself" 仍是误导。

我的建议:B 合入前做掉(或至少先把 storage-maintenance.ts:375 那句改成保留提示),A 作为「真·自包含迁移」后续单独立项。

次要(非阻塞,供参考,不重复展开)

  • survivor-rule 仲裁仍是简化版(仅 savedAt + 字典序 tie-break,非 boot 的 requests→nextBlockId→route-first→savedAt 全序)——此条已是既有 🔶 待决项,我不重复决策,仍挂起等你。
  • digest 校验(BatchedViewVerification)覆盖 blocks/messages/refs/deadRefs,不含 ccr_entries——纯 ccr 漂移不会被判 drifted(与 A 的校验扩展相关)。
  • archive/ 子目录里的已归档会话 JSON 会被递归扫到并按 ahead 规则处理——仅提示,无数据风险。

请就 A/B/C(及是否需要升 P1)拍板;选定后我在本 PR 分支落地并补回归测试。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

🔶 需要决策:对照最新 master 审计 —— migrate 未吸收 CCR 内容边车(误删陷阱)

按 owner 要求对照最新 master(ff5bdc1b,比本 PR base 更新)清点 bili storage migrate 是否漏了新增会话目录结构。一处实质缺失:.content-store.json 边车(#1097,存被折叠/可 acp_retrieve 的原始正文)既不被导入、也不被上报,但工具收尾文案邀请“clean 后可自行删除 JSON”。

证据链(head 4b18642):

  • discovery 跳过边车且不计数:persist.ts:1968 if (basename.endsWith(".content-store.json")) continue;
  • 导入只吃信封:persist.ts:2019 shadow.importRecord(rec.payload);buildSession 从不读盘上边车:persist.ts:4249 contentStore: parsed.forkContentStore;ingestSession 仅按视图 contentStore 写 ccr_entries(store.ts:1413-1419)→ 迁移行 ccr_entries 为空。
  • 对比活跃保存 record(...,liveCcr):shadow.ts:481 会合并 liveCcr → 填 ccr_entries;迁移路径没有这步。
  • 服务期 CCR “unified 优先、空则回落边车文件”:persist.ts:2247-2254;代码自证边车为权威源 persist.ts:2786-2790 / :3200;文档 docs/configuration.md:210/:2120/:1595 亦确认边车与会话同生命周期。
  • 收尾文案邀请删除:storage-maintenance.ts:375 “archive or delete them yourself once this report is clean.”

后果:带折叠/CCR 内容且未被索引 ccr 镜像完整捕获的遗留会话(#5599 目标人群),在 clean 信号后删掉 *.content-store.json → acp_retrieve 原文静默丢失。migrate 对 JSON 只读、非自动丢数据(条件触发),但工具给出会误导删库的 false-clean 信号。

处置选项(完整审计见协调线程 #5639):

  • A 彻底自包含:迁移时读入配对 .content-store.json 喂给导入,使 ccr_entries 落地 + 校验扩展到 ccr(跨 persist/shadow/store,设计级)。
  • B 安全护栏(推荐先做):discovery 检出并上报配对边车 + 把收尾文案改成“必须保留 .content-store.json / 或重存一次吸收 CCR”。小改动、低风险、立即堵陷阱。
  • C 仅文档:最弱,运行时误导文案仍在。

建议 B 合入前做掉,A 后续单独立项。属语义/设计层(跨文件),不擅自改,待 owner 拍板。已打 needs-decision。

(其余新增结构——协议子目录、.fb.json、热+cold、崩溃临时名、迁移标记、backups/*.db——均已正确覆盖,无缺失。)

@ranxianglei ranxianglei added bug Something isn't working p1 labels Nov 10, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复——本线程即该缺陷的处置现场(floor 33 owner 要求对照 master 审计 → floor 36/37 发现边车缺失并列出 A/B/C),无平行条目;本次是把其中无需决策的部分(B)直接修掉。

标签:bug + p1 + scope/storage(保留已有 pr / needs-decision)。理由:migrate 收尾文案诱导删除仍承载 acp_retrieve 原文的 .content-store.json = 数据丢失类陷阱;按 2026-10-19 修订,数据丢失类一律 p1。

已做:B 护栏已实现并推入本 PR(commit 23be109922)

按「无决策依赖的部分先修」,B(安全护栏)不需要拍板,已直接做掉:

  • 检出:扫描阶段收集配对边车(ccrSidecarForRouteFile 镜像 candidateContentStorePaths 的 .fb.json 配对规则);不计入 scannedFiles,不导入,JSON 依旧严格只读;
  • 上报:每会话 ccrSidecar 标志 + 报告新字段 ccrSidecarSessions / unpairedCcrSidecars(孤儿边车按相对路径列出);日志行同步扩展;
  • 文案:人类输出新增 KEEP 段逐一指名边车文件;只要存在边车,「clean 后可自行删除」的邀请整句撤回,替换为明确的勿删警告;零边车的运行输出与原句逐字一致(负向门测试钉死)。

验证:typecheck 干净;定向套件 tests/storage-migrate.test.ts 7→10 全绿(新增:配对边车跨 dry-run/导入/复跑三态 + KEEP 文案 + 字节未动证明、零边车负向门、孤儿边车列出且不影响退出码)。全量判定以 CI 为准(已在新 head 起跑)。PR 描述已同步更新(行为变更披露 + 验证清单)。

行为变更披露(旧→新)

  • 旧:bili storage migrate 对 .content-store.json 静默跳过(不计数、不上报),收尾句恒定输出 “archive or delete them yourself once this report is clean”;
  • 新:边车被检出并上报;存在边车时删除邀请撤回、改出勿删警告 + 文件清单;--json 输出新增两个字段(纯增量);退出码语义不变(边车本身不会让运行失败);零边车场景人类输出逐字不变;
  • 原因:迁移后 CCR 读取是 unified 优先、空则回落边车文件(persist.ts:2247-2254),而迁移从不读盘上边车(buildSession 只吃信封内联字段,persist.ts:4249)→ 边车仍是事实权威源,原邀请构成误删陷阱。

待拍板:A —— CCR 内容导入(设计级,跨 persist/shadow/store)

一个补充事实影响 A 的收益边界:shadow 模式下边车永远是读取源(unified 镜像仅 full 模式参与读,persist.ts:2247 注释自证)——所以即使 A 落地,shadow 安装仍须永久保留边车;A 的真实收益是让 full 模式安装迁移后自包含、可安全删 JSON。选项不变,建议维持:

选项 内容 代价 建议
A 迁移时读入配对边车喂给导入,ccr_entries 落地 + 校验覆盖 ccr 设计级(跨三文件),需定「部分捕获/多路由文件」边界 单独立项,后续做
B 护栏(本次) 小改动,已落地 ✅ 随本 PR 合入
C 仅文档 运行时误导文案仍在 放弃(B 已覆盖)

另审计更正一处:floor 36 引用的 docs/configuration.md:210「边车恒写」一句在 master #5988 后已改写(jsonWriter=off 停写边车),该引用过期;但代码级结论不受影响,我已独立重验全部证据链(head 4b18642),其余结构(协议子目录/.fb/热+cold/崩溃临时名/迁移标记/backups)确认无缺失。本地环境 issue4612-gc-online-guard.test.ts 在 test 1 后挂起 >240s,已用 stash A/B 证明干净树同样挂起——非本 diff 引入,若 CI 该 lane 异常请以此为参照。

八问

  1. 是什么问题:migrate 静默跳过 CCR 边车却给出「可删 JSON」的 clean 信号——照做会不可恢复地丢掉 acp_retrieve 原文(Feature: content-addressed message store — ID-referenced tool results + acp_retrieve (built-in CCR) #1097)。
  2. 什么原因导致的:导入路径只吃信封(importRecord(rec.payload)),从不读盘上边车 → 迁移行 ccr_entries 为空;而运行期 CCR 读取 unified 优先、空则回落边车,边车保持权威;discovery 又静默跳过它(persist.ts:1968)、收尾文案无条件邀删(storage-maintenance.ts:375)——上报面与真实数据依赖脱节。
  3. 影响范围与严重程度:恰好命中 Migration design review: legacy JSON to SQLite-only, wire/cache contract, duplicate routes and durability fixes #5599 目标人群(带折叠/CCR 内容的遗留会话);条件触发(需有边车 + 操作者采信 clean 信号);后果 = 原文不可恢复、acp_retrieve 对该会话失效。工具本身只读、不自动丢数据;按数据丢失类定级 p1。
  4. 怎么修复的:B = 让报告说真话(检出 + 上报 + 撤回误导文案),不动导入语义;A = 让工具真正自包含(导入边车内容),留待拍板后立项。
  5. 是否有回归问题:审过的既有行为逐项——退出码语义(不变)、scannedFiles 语义(不变,仍只数信封)、JSON 只读保证(字节级测试钉死)、零边车输出(逐字一致,负向门)、--json 形状(纯增量两字段)、日志(增量扩展)。无 wire/持久化/kernel/配置面变更。已知本地环境 GC guard 套件挂起经 A/B 排除与本 diff 相关。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/重试/游标/去重键;出站消息路径零改动,前缀缓存字节稳定性不受影响。
  7. 是否改了设计方向:不涉及——仍是「信封导入 + digest 对账」;CCR 内容导入刻意不并入本 PR(那是 A 的决策面)。
  8. 是否引入了新的配置:无——零新 key/env/flag。

判定:B 部分 CI 绿即可合(护栏完整、风险低、有回归测试钉住);A 等 owner 拍板后单独立项。needs-decision 标签保留至 A 有结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved and pushed. This run:

What I did

  1. Verified live state first: master had moved 134 commits since the previous rebase base (c26c31ace4 → fcc2e84848), so the PR was genuinely dirty again — this blocker was real, not stale.
  2. Rebased the 17 PR commits onto fcc2e84848. One conflict, in website/config-reference/process.yaml (the storage.archiveAfterDays entry): master had added a desc_ja field while the PR had added the bili storage migrate note to desc_en/desc_zh. Resolved as the union — PR's en/zh (with the migrate note) + master's ja line.
  3. Ran node tools/gen-config-docs.mjs generate to regenerate the config docs from the yaml seed (the docs-sync gate), so docs/configuration.md / .zh-CN.md / .ja.md all match the seed; check-coverage passes.
  4. Verified the rebased head locally: npm run typecheck clean, npm run build clean (dist-import-annotations + dist-external OK), tests/storage-migrate.test.ts 10/10 pass. Full-suite verdict is left to remote CI.
  5. Re-fetched before pushing (remote head still e38340bc6e, no concurrent agent work) and completed the task-authorized conflict rebase with a lease-verified force-push: e38340bc6e → 9a24eeec01.
  6. Post-push live verification: PR [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 is mergeable: true, mergeable_state: behind — i.e. no conflict; master picked up 6 more commits (announcement/notices policy work, [notice] The dsh-live-compact plugin is not installed, yet a "dsh-live-compact conflicts with bili compression" notice s | [notice] 没装 dsh-live-compact 插件,却出现「dsh-live-compact 与 bili 压缩冲突」公告 #6508/[PR] fix(notices): gate announcement records on the live conflict scan via requiresScan (#6508) #6509/[PR] docs(governance): Announcement policy — no broadcast to all, owner confirmation required, specify scope of impact, plugi | docs(governance): 公告政策——禁止全员广播、owner 确认、写明影响范围、插件公告必须触发式 (#6508) #6514) after my rebase, and they touch none of this PR's code paths. CI on the new head: 26 success / 12 skipped / 6 in_progress / 5 queued, 0 failures so far. (The mirror cache was stale for this PR — last fetch ~35h old on the previous sha — so I used single non-polling GETs for the new head's state.)

八问

  1. 是什么问题:master 前进 134 个提交后,PR [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 再次与 base 冲突(dirty),无法合并。
  2. 什么原因导致的:PR 分支基于 c26c31ace4,期间 master 合入大量工作;process.yaml 的 archiveAfterDays 条目被双方同时修改(master 加 desc_ja 日文描述,PR 给 en/zh 加 bili storage migrate 说明)产生文本冲突。
  3. 影响范围与严重程度:仅阻塞本 PR 合并;无用户面影响。
  4. 怎么修复的:rebase 17 个提交到 fcc2e84848,唯一冲突按并集解决(en/zh 取 PR 的 migrate 说明 + ja 保留 master 版本),用 gen-config-docs 从 yaml 种子重新生成三语文档过 docs-sync 门禁,lease 校验后 force-push。
  5. 是否有回归问题:无。typecheck/build 干净,PR 定向测试 10/10,文档与种子同步(coverage ok);冲突解决逐字段核对过,双方内容均保留无覆盖;全量回归由 CI 在新 head 上把关(进行中,目前 0 失败)。
  6. 是否破坏缓存:不涉及——纯分支历史整理与文档同步,未触碰缓存/存储逻辑。
  7. 是否改了设计方向:不涉及,未改任何代码语义。
  8. 是否引入了新的配置:不涉及。

判定:冲突已解决,PR 恢复可合并状态(mergeable, behind 为 master 后续 6 个无交互提交所致,非阻塞),CI 在新 head 上运行中,等待全绿与人工 review。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei
ranxianglei force-pushed the 2026-11-04_storage-migrate-json-to-sqlite branch from 9a24eee to f8b50ff Compare November 13, 2026 17:14
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved and pushed. This run:

What I did

  1. Verified live state first: master had moved 14 commits since the previous rebase (99c22d283f → e1f31ad9d5), including the storage-default change [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 (pure-SQLite default — JSON envelopes stop being written unless dual-write is requested). The PR was genuinely dirty again (single GET confirmed mergeable: false / dirty) — this blocker was real.
  2. Rebased the 17 PR commits onto e1f31ad9d5. One conflict, in README.md + README.zh-CN.md (the storage-maintenance CLI paragraph): master's [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 had updated the BILI_STORAGE_UNIFIED value list to sqlite|shadow|full and added "the default is now pure sqlite", while the PR had added migrate to the command list. Resolved as the union — archive/restore/gc/migrate + sqlite|shadow|full + the pure-sqlite default note, in both languages. No code conflicts: the PR's src/persist.ts / src/storage/* changes applied cleanly on top of the new storage default, and the PR's tests pass under it (42/42, 1 platform-skip) — the new pure-SQLite default actually strengthens this PR's premise (legacy JSON files become the only thing left to import).
  3. Verified the rebased head locally: npm run typecheck clean, npm run build clean (dist-import-annotations + dist-external OK), PR's targeted suites (storage-migrate, storage-unified, seq-ledger-pr3, check-dist-external) all green, docs in sync with the yaml seed (docs-sync gate).
  4. Re-fetched before pushing (remote head still 9a24eeec01, no concurrent agent work) and completed the task-authorized conflict rebase with a lease-verified force-push: 9a24eeec01 → f8b50ffe13.
  5. Post-push live verification: PR [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 is mergeable: true, mergeable_state: blocked — no conflict, waiting on CI/review. CI just started on the new head (23 check runs queued). (The mirror cache entry for this PR is stuck on the previous sha — last fetch 11-13 10:36 UTC — so I used single non-polling GETs for the new head's state.)

八问

  1. 是什么问题:master 前进 14 个提交(含 [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 纯 SQLite 默认值变更)后,PR [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 再次与 base 冲突(dirty),无法合并。
  2. 什么原因导致的:[PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 改写了 README 存储 CLI 段落(BILI_STORAGE_UNIFIED 取值列表加 sqlite、注明缺省即纯 sqlite),与本 PR 给同一段落加 migrate 命令的修改重叠,产生文本冲突;代码层无冲突。
  3. 影响范围与严重程度:仅阻塞本 PR 合并;无用户面影响。
  4. 怎么修复的:rebase 17 个提交到 e1f31ad9d5,唯一冲突按并集解决(命令列表取 PR 的 migrate + 取值列表与缺省说明取 master 的 [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 版本),双语 README 同步处理;PR 代码在新存储默认值下原样适用,本地定向验证全绿后 lease 校验 force-push。
  5. 是否有回归问题:无。typecheck/build 干净,PR 定向测试 42/42 通过(1 个平台性 skip),文档与种子同步;冲突解决逐句核对,双方内容均保留;新默认值([PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480)与 PR 代码的交互由 CI 全量套件在新 head 上把关(刚启动)。
  6. 是否破坏缓存:不涉及——纯分支历史整理,未触碰缓存/存储逻辑。
  7. 是否改了设计方向:不涉及,未改任何代码语义;仅文档段落按并集合并。
  8. 是否引入了新的配置:不涉及。

判定:冲突已解决,PR 恢复可合并状态(mergeable, blocked 等待 CI/review),CI 在新 head 上运行中,等待全绿与人工 review。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

ci-check added 17 commits November 15, 2026 11:59
…ions into the unified SQLite index (#5599)

Adds the missing one-shot migration path between the lazy organic import
(on first open) and JSON deprecation: bili storage migrate [--dry-run]
[--json] walks every legacy state file under the sessions dir (incl. the
-cold root), arbitrates same-id route files with boot's survivor rule
(newest savedAt, lexicographic tie-break), imports the representative via
the existing ingest path, then runs the full digest reconciliation
against the legacy source view per session (new UnifiedShadow.verifyImported).

Report statuses: imported / indexed / drifted / failed / planned(dry-run);
corrupt or mismatched files are reported failed WITHOUT touching them —
the command is read-only over the JSON corpus by design (no quarantine
renames), so cleanup stays an operator decision after a clean report.
No config keys, no env vars, no default-behavior change, no kernel touch.
Covers bulk import into a fresh index with index-only service proof (JSON
corpus deleted after import), idempotent re-run, dry-run writes nothing,
corrupt-file failure isolation with non-zero exit, same-id route
arbitration, and the read-only guarantee over the whole sessions dir.
README Storage-maintenance-CLI section + CONFIGURATION archiveAfterDays
row gain the migrate entry in both languages. In passing: the zh README
block was missing the bili sessions foreign-blocks line present in the
English block (#5312) — added for zh/en parity.
…g storage behavior (#5599)

CI-lane fix on the rebased head. Both failures reproduced identically on the
original PR head cf3895a (verified by running the suite there), i.e. the
suite was red before this rebase; the conflict had kept CI from ever running.

1. 'already-indexed sessions report indexed without re-writing' assumed the
   first migrate run is what imports. The writing instance's own store open
   already ran the first-open unified auto-import (present since well before
   this PR's base), so the first run only verified: imported=0 != 2. Added
   dropIndex(dir) after writeSessions — same isolation every other test in
   the file already uses.
2. 'migrate never modifies any file under the sessions dir' snapshot excluded
   index.db/-wal/-shm but not the integrity-check verdict sidecar
   index.db.chk (src/storage/integrity-check.ts), which the engine rewrites
   when migrate opens/closes the store. Widened the exclusion to the whole
   index.db family (prefix match) per the existing comment's intent.
CI-lane fix: the type is referenced only inside src/persist.ts (the exported
LegacyMigrateReport keeps naming it structurally; no file outside persist.ts
imports it, tests mirror it locally). INTERNAL_ONLY per scripts/check-unused-exports.mjs.
…enerate (docs-sync gate, #5599)

CI-lane fix: the original PR hand-edited the storage.archiveAfterDays row in
CONFIGURATION.md/zh to document 'bili storage migrate' but never touched the
website/config-reference/*.yaml seeds (the single source the docs-sync guard
regenerates from), so the generator overwrote the note. Moved the clause into
the process.yaml desc_en/desc_zh entries and re-ran tools/gen-config-docs.mjs:
the EN artifact now regenerates byte-identical to what the PR committed; the
zh artifact differs only by normalizing the clause's half-width punctuation
(,还有 / (#5599: / ;) to full-width Chinese punctuation.
…import after minification (#5599)

CI-lane fix (static-gates): the message ended in the bare word 'import', so
after tsup minification it fused with the following statement into
...Error("...between scan and import");decodedBytes=...throw new Error("unified
sidecar unavailable") which the dist gate's side-effect-import pattern latches
onto, capturing a garbage 'specifier' spanning both strings. Reworded to end
in 'import attempt'; no test pins the old wording.
…r (windows CI, #5599)

CI-lane fix (windows shard 2): dup.file comes from path.relative and carries
backslashes on Windows, so startsWith("openai/") failed there while passing on
Linux. Assert against 'openai'+path.sep instead.
An imported session materializes only its live tail into message rows
(folded ghosts keep their ref bindings but own no rows; the tail is
re-based at seq 0). The #4316 cold-load alignment then stamps the tail
onto seq+1 = m00001.., evicting the historical ghost bindings onto fresh
numbers and re-targeting saved summary citations at newer bodies —
retrieve(m00001) returned the tail message instead of the original.

Native sessions are unaffected by construction: the historian sweep
retires covered rows instead of deleting them (#4390), so every mapped
id keeps its row and survivor seqs never shift. The guard therefore
discriminates exactly on the broken shape: every mapped id must own at
least ONE row (live or retired) before the stamp may run; otherwise the
load keeps legacy refs (JSON-load parity) and retries on a later cold
load once rows exist. Query failure fails safe to skip.

(cherry picked from commit fe8f27d142280f6c9e894f76cbf3778d45f2d89c)
…5599)

Negative half: an imported collapsed ledger (ghost bindings without rows,
tail at seq 0..2, summary citing m00001, CCR holding both originals) must
cold-load with all legacy refs intact, alignment skipped, and the saved
citation still retrieving the original body. Control half: the same ref
map over a complete ledger still aligns (no-op here), proving the guard
discriminates on row coverage, not on the presence of folds. S15 remains
the positive-direction pin for drift alignment.

(cherry picked from commit 54aa3f307c4e0e725c6076e6ff8022b923078e3b)
… guard (#5599)

The alignedLegacyDigest oracle applied restampRefsToLedger unconditionally,
modeling the pre-#5599 cold-load behavior. The one-time re-stamp now refuses
incomplete ledgers (an imported collapsed ledger's folded ghosts own no rows),
so full-mode loads of the folded corpus keep their legacy refs and the
oracle's stamped digest no longer matches production. The oracle now takes
the open UnifiedStore probe and applies the alignment only when the guard
would pass (every mapped id owns at least one row) — same predicate as the
SessionStore cold-load path.
…eletion (#5599)

`bili storage migrate` skipped `.content-store.json` sidecars silently while its closing note invited deleting all JSON once the report was clean — deleting a sidecar loses the acp_retrieve originals (#1097) that migrate does not import into ccr_entries.

- discovery now collects paired sidecars during the walk (never counted in scannedFiles, never imported, read-only as before)
- per-session ccrSidecar flag + report.ccrSidecarSessions / unpairedCcrSidecars (orphan sidecars listed by rel path)
- human output gains a KEEP section naming each sidecar; the delete invitation is withdrawn whenever a sidecar is present (zero-sidecar runs print the original sentence verbatim)
- exit code semantics unchanged: sidecars alone never fail a run

Master-drift audit finding on this PR (owner request 2026-11-10); implements guardrail option B from the decision thread — self-contained CCR import (option A) stays open.
@ranxianglei
ranxianglei force-pushed the 2026-11-04_storage-migrate-json-to-sqlite branch from f8b50ff to 9f3f386 Compare November 15, 2026 04:02
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved and pushed. This run:

What I did

  1. Verified live state first: master had moved 202 commits since the previous rebase (e1f31ad9d5 → faf0af61a6). The PR was genuinely dirty (single GET confirmed mergeable: false / dirty) — this blocker was real, not a stale re-trigger.
  2. Rebased the 17 PR commits onto faf0af61a6. One conflict, in src/cli.ts (main()'s destructuring line): master had added the -c/--config fail-fast block (checkNamedConfigRef — typo'd config names now exit 2 with the sibling list) inside main(), while the PR had added storageAction (the new migrate subcommand) to the same destructuring line. Resolved as the union — storageAction in the destructure AND master's config fail-fast block kept verbatim. The other overlapping files (src/persist.ts, src/storage-maintenance.ts, src/storage/shadow.ts, READMEs, docs) auto-merged cleanly; typecheck + tests confirm no semantic breakage from the 202-commit master move.
  3. Verified the rebased head locally: npm run typecheck clean, npm run build clean (dist-import-annotations + dist-external OK), PR's targeted suites (storage-migrate, storage-unified, seq-ledger-pr3, check-dist-external) 43/43 pass (1 platform-skip), docs in sync with the yaml seed (docs-sync gate).
  4. Re-fetched before pushing (remote head still f8b50ffe13, no concurrent agent work) and completed the task-authorized conflict rebase with a lease-verified force-push: f8b50ffe13 → 9f3f386b83.
  5. Post-push live verification: PR [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 is mergeable: true, mergeable_state: blocked — no conflict, waiting on CI/review. (The mirror cache entry for this PR is stuck on an old sha — last fetch 11-13 23:05 UTC — so I used single non-polling GETs for the live state.)

八问

  1. 是什么问题:master 前进 202 个提交后,PR [PR] feat: bili storage migrate — one-shot bulk import of legacy JSON sessions into the unified SQLite index (#5599) #5636 再次与 base 冲突(dirty),无法合并。
  2. 什么原因导致的:master 在 main() 里新增 -c/--config 缺省名 fail-fast 块,与本 PR 在同一解构行加 storageAction(migrate 子命令)的修改重叠,产生文本冲突;其余重叠文件自动合并干净。
  3. 影响范围与严重程度:仅阻塞本 PR 合并;无用户面影响。
  4. 怎么修复的:rebase 17 个提交到 faf0af61a6,唯一冲突按并集解决(解构行取 PR 的 storageAction + master 的 config fail-fast 块原样保留),本地定向验证全绿后 lease 校验 force-push。
  5. 是否有回归问题:无。typecheck/build 干净,PR 定向测试 43/43 通过(1 个平台性 skip),文档与种子同步;202 个 master 提交中存储子系统的相关改动(persist/storage-maintenance/shadow)自动合并后由 typecheck + 定向测试验证无语义破坏,CI 全量套件在新 head 上继续把关。
  6. 是否破坏缓存:不涉及——纯分支历史整理,未触碰缓存/存储逻辑。
  7. 是否改了设计方向:不涉及,未改任何代码语义;冲突解决为双方内容的并集。
  8. 是否引入了新的配置:不涉及(保留的 config fail-fast 是 master 已有行为)。

判定:冲突已解决,PR 恢复可合并状态(mergeable, blocked 等待 CI/review),等待 CI 全绿与人工 review。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

The failing check test (ubuntu-latest, 24, 3) was a transient CI-runner flake, not a code regression. I re-ran the failed job and it now passes — run 43674967401 is completed / success, all jobs green, head unchanged at 9f3f386b83. No code changes were needed.

Why it's a flake (evidence):

  • The failing test tests/issue5631-restart-survival.test.ts (subtest pi lane (anthropic shape), error proxy exited before becoming healthy (pi-main-0, code=1)) is not in this PR's diff.
  • The proxy startup is identical between the dsh lane (started 6+ times, all OK) and the pi lane (failed once) — startProxy does not take the lane; both use the same compressCfg(WALL_TOKENS). A deterministic code bug would have failed the dsh lane too.
  • This PR's changes (src/storage/store.ts, src/storage/shadow.ts, src/persist.ts) are all additive / fail-safe and do not touch the proxy startup path.
  • Ran the test locally: both lanes pass (3/3, 45.8s), including the pi lane that failed on CI.
  • The shard runs 4 test files concurrently (--test-concurrency 4); the late-publishing listener lines in the job log come from a concurrent test (launcher.test.ts via src/launcher.ts:6209), confirming high concurrency — the pi lane's first proxy hit a transient port-collision / resource-contention on the runner.

Action taken: re-ran only the failed job via the GitHub API (POST …/runs/43674967401/rerun-failed-jobs, HTTP 201). Re-run: test (ubuntu-latest, 24, 3) → success (04:59:00Z → 05:07:09Z).

Note: the mirror cache is stale (fetched_at 04:23:21Z, ~50 min old) and still shows the old failed state; the authoritative GitHub API confirms the run is now green. The cache should catch up on the next poller refresh.


八问

  1. 是什么问题:CI 分片 test (ubuntu-latest, 24, 3) 里 tests/issue5631-restart-survival.test.ts 的 pi 车道报 proxy exited before becoming healthy (code=1)——是 CI runner 上的瞬时 flake,不是代码回归。
  2. 什么原因导致的:该分片以 --test-concurrency 4 并发跑多个测试文件,pi 车道首个 proxy 启动时撞上端口竞争/资源争用(日志里的 late-publishing listener 来自并发跑的 launcher.test.ts,佐证高并发)。proxy 启动配置在 dsh/pi 两车道完全相同(dsh 车道成功启动 6+ 次),且本 PR 改动全是增量/失败安全、不触碰启动路径。
  3. 影响范围与严重程度:仅影响该次 CI 运行的一个分片;PR 代码本身无问题(本地两车道均通过)。严重程度:体验瑕疵(CI 偶发红),无外部影响。
  4. 怎么修复的:未改代码。通过 GitHub API 重跑失败 job(rerun-failed-jobs),重跑通过(run 43674967401 → completed / success,全绿)。
  5. 是否有回归问题:无。未改任何代码,PR 既有行为不变。
  6. 是否破坏缓存:不涉及(未改代码)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:PR 代码正确,CI 红为瞬时 flake,重跑已全绿,可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

Already green — no work needed this run.

Re-verified live against the authoritative GitHub API: run 43674967401 is completed / success (head 9f3f386b83, all 28 jobs green, 0 failing, 0 in-progress). That's the re-run I triggered in the previous turn — it passed.

The mirror cache is stale: fetched_at is still 2026-11-15T04:23:21Z (~54 min old) and it's still reporting the old failed test (ubuntu-latest, 24, 3) job. That staleness is what fired this wake; the actual CI is green. No code changes, no further work.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant