Skip to content

Unified SQLite storage: corpus pins + parity core + shadow dry-run + drop-in read path (#2671 Phases 1 & 1.5) - #2690

Merged
ranxianglei merged 39 commits into
masterfrom
2026-10-10_storage-pins
Oct 11, 2026
Merged

ranxianglei merged 39 commits into
masterfrom
2026-10-10_storage-pins

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

kernel-change: storage — kernel/src/panel/panel.ts accepts a host-rendered one-line storage-mode label (storage?: string, display-only; the kernel never guesses the mode). Version bumped to 0.0.118 (master moved to 0.0.116/#2695 and 0.0.117 mid-flight; this branch stays one ahead).

Executable parity oracle for the storage refactor (#2671), built in four stacked commits on one branch:

  1. Pins (92e75315) — render goldens for 4 wires + a 9-case legacy-format corpus (tests/golden/storage-corpus/: folded, tier2, deadrefs, fork-receipt, fork-retained, image, ccr, twinfile, zstd), each case pinned to a canonical digest by tests/storage-corpus.test.ts. The corpus is generated by scripts/update-storage-corpus.ts from real SessionStore writes, so it pins exactly what the current writer produces.
  2. Parity core (90cb99dd + 43145719) — src/storage/: driver.ts (better-sqlite3 primary, node:sqlite fallback), schema.ts (STORAGE_SCHEMA_VERSION 1), store.ts (CAS content dedup with refcounts, single-transaction ingest, deleteSession), ingest.ts (plan/load round-trip + toLegacyLike). Corpus-proven design decisions:
    • messages is an append-only log keyed (session_id, msg_id, content_hash) — split tool-call cores stay distinct rows, identical re-sends intern to one seq; pluginSnapshot/lastMessages are pointer sequences into it.
    • the ref ledger is a separate table: dead-ref ghosts live in byRaw but in no message array (compress failure is self-amplifying: failed calls are hidden as orphan refs, making the same range permanently un-compressible #2362 Kernel Contract — death never unmaps), and Object.entries(byRaw) key order is digest-sensitive, so refs(session_id, ord PK, raw_id, ref) preserves it verbatim.
    • dead_refs carries its own ordinal so ordering never depends on numeric ref parsing.
    • Digests treat wall-clock timestamps and the last-writer biliVersion stamp as environment (__TS__/__VER__) — probe-proven these are the only fields a legacy save→reload round-trip rewrites.
    • Acceptance: every corpus digest must reproduce byte-for-byte on BOTH engines (tests/storage-unified.test.ts).
  3. Shadow dry-run (ff8685ee) — BILI_STORAGE_UNIFIED=1: every committed session write is double-ingested into <sessions-dir>/index.db and digest-reconciled against the legacy load view; mismatch logs DIGEST DRIFT, never blocks the write path; self-disables after 3 consecutive internal failures. Pure observability — production traffic becomes the migration test corpus.
  4. Drop-in read path (ad747fe7, Phase 1.5) — BILI_STORAGE_UNIFIED=full: loadSync/boot()/loadAll()/loadStateForSearch go unified-first. A legacy-file session is organically imported into the index on first open ([storage] unified import on first open: …) and later loads are served from index.db; any unified failure falls back to the legacy file. Write path unchanged (legacy files + double-ingest), so rollback = unset the env. Safety rails:
    • hard encryption opt-out: BILI_ENCRYPTION_KEY active → sidecar never opens (index.db would hold cleartext payloads; warn + legacy-only)
    • GC parity: removeLegacyFile (会话身份不稳定:credential/upstream 维度导致长会话状态孤儿化(sessions 缺失) #286 rekey/dedup) now also deletes the session's index rows in one transaction and releases CAS refcounts
    • 3-failure self-disable flips full-mode reads back to legacy-only
    • fix on the way: loadLegacyView() omitted createdAt/lastSeen, which structurally drifted the shadow's internal digest reconciliation (logged, unasserted before)

Phase 2a (added on top, single-branch policy)

The branch now also carries replica/checkpoint REST (was briefly PR #2745, closed and folded in here per reviewer preference — one branch, one final merge for the whole #2671 initiative):

  • checkpoints table: label + state_hash (CAS refcount) + session index; deleteSession sweeps unpinned checkpoints and releases their blobs
  • SessionStore: createReplica / restoreReplica / listReplicas / deleteReplica — restore re-materializes a checkpoint as a NEW session through the normal dual write path (meta.restoredFrom provenance)
  • Loopback REST: POST/GET /__bili/replicas, GET …/<cpId>, POST …/<cpId>/restore, DELETE …/<cpId>; legacy lanes → 501 + knob hint
  • Pinned checkpoints survive deleteSession GC (copy taken before a destructive experiment outlives the experiment); explicit delete still removes pinned ones
  • Tests: tests/storage-replicas.test.ts 4/4 (round-trip; restore-after-GC digest equality; pinned survives/unpinned swept; legacy 501 + pinned delete override); whole-blob-vs-events trade-off noted in the description

Phase 2b (fork protocol v2) — b1693a77

parentRevision now hashes {sessionId, seq, blockSet} (active block ids, sorted) instead of the whole store snapshot — stats churn (contextEstimateTokens, usage grades) no longer invalidates forks mid-session; only structural change (message count or block set) does.

  • manifest advertises protocolVersion: 2, acceptsProtocolVersions: [1, 2]; server accepts BOTH revisions during the transition (v1 clients keep working)
  • forkParentRevisionLegacy() exported for the v1 fallback path — a v1 revision still matches exactly (double-take acceptance, verified by test)
  • src/agent/fork-adopt.ts client gate widened to version === 1 || version === 2
  • Tests: legacy-revision acceptance + fake-revision 409; stats-churn stability (revision unchanged); structural change (real sendModel turn) flips it; plugin-fork 48/48, fork family 68/68 + 37/37

Phase 2c (export --jsonl) — 4ca9b8c7

bili export <id> --jsonl [--full]: line-delimited JSON dump rendered from the unified index tables when index.db exists and covers the id (header + per-message lines in seq order with --full CCR deref + block lines + deadRefs line); legacy-tail fallback renderer otherwise. Generated interoperability artifact, never a source of truth (design doc §85). Flag seeded in CONFIGURATION CLI tables (config-surface-guard).

Fix on top — stale load-side import guard — 3e17052f

buildForLoadAll re-ingested the on-disk payload unconditionally; when a load raced a newer in-flight write (e.g. #286 rekey re-serializing an older envelope on first boot), the stale payload reverted newer fold state in the index — the load-side face of #2401, surfacing as raw originals leaking through folded handoff exports (tests/e2e-export-handoff.test.ts under machine storage.unified=full). Guard: import only when the index has never seen the id (organic first open) or the payload savedAt is strictly newer than the row's updated_at (#408 clamp rewrites, #286 rekey, restoreReplica still import — fresh savedAt at build time). Adds UnifiedStore.sessionUpdatedAt + shadow pass-through. That e2e now passes locally under full mode.

Golden regenerations on this branch (justified)

  • tests/golden/storage-render/* — master added an anchor-kind field ("c": tool-call/tool-result/…) to metadata.foldAnchors after the last regen on this branch; only the metadata section digest moves (state/blockContents/stats identical, verified key-by-key). Regen: node --import tsx scripts/update-storage-render-goldens.ts.
  • tests/golden/wire-contract/manifest-*.json — fork capability now advertises protocolVersion: 2, acceptsProtocolVersions: [1, 2] (Phase 2b). Regen: node --import tsx scripts/update-wire-contract-goldens.ts.
  • web client storage-mode label switched to literal t() branches so all three det.storage_mode_* catalog keys are statically referenceable (the Web UI 英文翻译开关(i18n)— 社区 PR #807 (@amirzamli) #1024 dead-key gate cannot follow string concat).

Test plan

  • tests/storage-corpus.test.ts — corpus round-trips + case-specific invariants (twinfile namespaced-wins, zstd twin identical digest, fork receipt/retained asymmetry, deadrefs ghosts)
  • tests/storage-unified.test.ts — dual-engine digest parity ×9 cases, whole-corpus idempotent re-ingest, real-persist shadow e2e, resolveStorageMode env matrix, full-mode organic-import read e2e (digest == manifest), second-load stability, boot #286 rekey GC parity (old id gone from index), encryption guard (no index.db created)
  • storage suites 24/24, tsc clean, unused-export gate clean (2 waived)
  • full CI suite (remote) — storage/persist/render families re-run in both modes

Rollout

Default off. 1 = shadow (observe drift logs), full = drop-in read. Soak in shadow mode first, then flip full; unset env = instant rollback.

🤖 This PR was prepared by an agent under the local git identity (ranxianglei), per the user instruction to push the branch and open the PR (human merge required).

Config surface

  • storage.unified (file tier, legacy|shadow|full, default unset = legacy) + env BILI_STORAGE_UNIFIED (1/true = shadow, full/unified = full; env wins over the file; resolved once per lane boot). Canonical resolver lives in src/knobs.ts (彻底整治环境变了滥用,除非必要都走配置文件 #2030 env discipline), re-exported from src/storage/shadow.ts. Documented in CONFIGURATION.md / zh-CN (compact + long-form rows).
  • CLI: bili export --jsonl [<selector>] [--full] (new flag on the existing export command; documented in CONFIGURATION.md CLI table).
  • New loopback-only admin routes on unified lanes (legacy lanes answer 501 with the knob hint): GET/POST /__bili/replicas, GET /__bili/replicas/<cpId>, POST /__bili/replicas/<cpId>/restore, DELETE /__bili/replicas/<cpId>.
  • Not knob-gated (ship on merge): fork manifest advertises protocolVersion: 2 with acceptsProtocolVersions: [1, 2] (PLUGIN.md documents the v2 revision-token scope and the legacy-token transition window); /acp panel gains a STORAGE line (legacy label when the knob is off).

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-10_storage-pins (4c7bd58)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3036-pr.2690.5caa5b4

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2690.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #2690(#2671 统一 SQLite 存储 · Phase 1 & 1.5)审查结论

先给结论:PR 设计扎实、默认关闭、unset 即回滚。我独立审查时发现两个会挡 CI 的缺陷,分支上的并发提交已把它们都修好——我逐行核对了修法正确、最小,并丢弃了我本地那份重复修复(没有再推,避免冲突)。当前分支头 4c202e80c 上原先全红的检查项已全部转绿,只剩 test 矩阵(4 腿)+ windows registry e2e 还在跑。

背景:该 PR 把 legacy JSONL 持久化迁到统一 SQLite(src/storage/:corpus 钉死 → 双引擎 parity → BILI_STORAGE_UNIFIED=1 shadow 干跑 → full 平替读路径)。按 [PR] 规则我不重做,只核代码 + CI。

我审查中独立发现、且已被并发提交修复的两处:

  1. docs 门禁红:docs-coverage(src/storage/shadow.ts:35 读的 BILI_STORAGE_UNIFIED 无 seed)+ docs-sync(手改 CONFIGURATION.md/.zh-CN.md 而非走 [docs][website] 配置文档单一事实源:从声明式 schema 生成 CONFIGURATION.md 与网站 /docs/(一步到位) #2107 单源 YAML)。→ fc49908a2 已修(env-only.yaml 补 seed + 重新生成),与我本地方案等价。
  2. better-sqlite3 硬依赖:原在 dependencies,是原生模块(node-gyp/prebuilds)。driver 虽惰性 require(dist 加载期零依赖),但 npm install -g 仍会给每台主机编译它——无构建工具链的主机(Windows runner、最小容器)install 直接失败,node:sqlite 兜底根本没机会跑。这正是 codex [Bug][Windows Codex] WAL-only 提交时 SQLite 合并选错世代:顺序启动新增 thread 被移入 conflict #2195/[Bug][Codex 0.160] -p profile 供应商未参与路由发现,bili codex 正常回答但请求绕过代理 #2197 + registry/windows 三条 e2e 腿变红的根因。→ 4c202e80c 已修(移入 optionalDependencies,同 sharp;test 腿 createRequire 探测,缺失时大声 skip)。

--- 结构化八问 ---

  1. 是什么问题:PR 是 Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 统一 SQLite 存储的 Phase 1(corpus 钉死 + 双引擎 digest parity)+ Phase 1.5(shadow 干跑 + 平替读路径)。审查发现的缺陷是两个 CI 阻塞项(docs 门禁、原生硬依赖),不是压缩逻辑问题。
  2. 什么原因导致的:① 新增 env 未按 [docs][website] 配置文档单一事实源:从声明式 schema 生成 CONFIGURATION.md 与网站 /docs/(一步到位) #2107 在 config-reference YAML 落 seed,且直接手改了生成产物;② better-sqlite3 作硬依赖使无工具链主机的 npm install 在 node-gyp 阶段失败。两者都是「新增面没对齐既有约定」,非核心算法错。
  3. 影响范围与严重程度:① docs 门禁 = CI 红,直接挡合并(流程级);② 硬依赖 = 无构建工具链主机 install 失败(真实安装面 + CI e2e 腿),属「特定条件下无法使用」。功能本体默认关闭,未开 env 时对现有用户零运行时影响。
  4. 怎么修复的:① 在 env-only.yaml 单源补 seed 并重新生成(保持生成块幂等,富说明沿用既有手工行惯例);② better-sqlite3 降为 optionalDependencies,driver 惰性探测 + node:sqlite 兜底,test 腿对缺失原生模块大声 skip。两处均在分支上,我已核对 diff 正确、最小。
  5. 是否有回归问题:无。默认(env 未设)路径经代码证明字节级不变——UnifiedShadow.maybeOpen 在 resolveStorageMode==="legacy" 时返回 null(src/persist.ts:341 得 undefined),所有 this.shadow?.… no-op、不建 index.db、写路径不动。写路径始终保留(legacy 照写 + 双写摄取),回滚 = unset env。高危模式逐项:无语义漂移、无 fail-fast 变吞错(shadow/full 失败一律降级回 legacy,只降级不阻断)、无默认值漂移(默认仍 legacy)、无输出格式变化;顺带修的 loadLegacyView 漏 createdAt/lastSeen 也只在 shadow 内部生效。
  6. 是否破坏缓存:不涉及既有缓存失效策略。CAS 内容去重/refcount 是新增自包含结构,由套件覆盖;前缀缓存/会话缓存语义未动;读路径为「只降级」(unified 失败回落 legacy)。
  7. 是否改了设计方向:未转向——正是 Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 既定的「双写 → 平替读」演进,legacy 文件仍是权威写目标,SQLite 层纯增量叠加,非替换。
  8. 是否引入新配置:是。① env BILI_STORAGE_UNIFIED(默认 unset=legacy;1|true|shadow=干跑,full|unified=平替读),已入 CONFIGURATION.md en+zh(经 seed 生成);② better-sqlite3 从 dependencies 移入 optionalDependencies(sharp 同款)。迁移成本:零(默认关闭)。

--- 现状与判定 ---

  • 本地核验(head 4c202e80c):docs-coverage ✅、docs-sync ✅(生成块幂等)。
  • CI(check-runs @ 4c202e80c):原先红的 docs-coverage / docs-sync / codex#2195 / codex#2197 全部转绿;其余 guard / build / 各 e2e lane / registry(ubuntu)均绿;仅剩 5 项 in_progress(test ubuntu×2、windows×2 + registry windows)。旧头 ad747fe 上那条 test: skipped 是 release v0.1.191 #2630 并发取消遗留态,新头上是真正在跑的 4 腿矩阵,并非跳过。
  • 判定:代码层面我无需再改(两缺陷已正确修复)。合并前建议 rebase 到最新 master(分支落后约 28–30 提交,src/persist.ts 是热文件,有语义冲突风险)——需要的话我来做,但鉴于分支正被并发更新,时机听你定。本 PR 触及持久化格式 + 新增可选原生依赖 + feat,按 §7.4 属 human-gate,需你手动合并;待那 5 个 in_progress 腿确认全绿即可合。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Two CI-drift causes found and fixed (4c202e80, 4193ea3e):

  1. better-sqlite3 install failure on Windows runners — it was a hard dependency, and node-gyp cannot find a VS toolchain there (gyp ERR! find VS), which red-flagged the codex/registry e2e legs. Moved to optionalDependencies (same pattern as sharp): the driver already probes it lazily and falls back to node:sqlite; the corpus test leg now skips loudly when the native module is absent instead of failing.
  2. storage-render golden digest drift (all 4 wires) — two stacked causes: (a) Phase 1.5 accidentally wrapped the legacy loadSync tail in the toLegacyLike/viewToRecord adapter, a second buildSession normalization round-trip that reshaped persisted-state bytes — restored the original single build; (b) the render goldens were pinned before the __TS__ timestamp masking landed, and state.blocks[].createdAt is such a timestamp — regenerated; wire bodies unchanged, one digest line per golden, and the digests are now wall-clock-independent.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 已核对 4193ea3e,两处修复都对:

① legacy loadSync 单构建恢复(cause a) — 确认正确。diff 里两条 legacy 路径都改回单一 buildSession(envelope.payload) + 内联 #408 clamp(plain-legacy 一处、full-mode 回落 legacy 一处);toLegacyLike/viewToRecord 适配器现在只服务真正的 unified 读路径(afterLoad = viewToRecord→buildSession,从 store record 重建这一次是必要的)。之前 Phase 1.5 把 legacy tail 也套进适配器,等于多跑一次 buildSession 归一化、重塑了持久化字节——正是 storage-render golden 抓到的那个漂移。#408 负 token clamp 在三条路径里都保留,没丢既有行为。

② render golden 重生成(cause b) — 确认最小且正确。四个 wire 的 golden 各只改一行 stateDigest(我在 chat.json 上逐行核过:foldAt/editedAt/summaryFrom/stateShape 及消息体全部未动,wire body 逐字节不变)。根因判断也对:golden 是在 ff8685e 给 canonicalize 加 __TS__ 掩码之前钉的,而 state.blocks[].createdAt 正是被掩码的墙钟时间戳,掩码落地后每个 wire 的 stateDigest 就整体平移了一次。重生成后 digest 与墙钟无关,不再吃时钟运气。

CI(@ 4193ea3e):27 项中 23 绿(含之前红的 codex #2195/#2197、docs-coverage/docs-sync、registry-ubuntu 等),仅剩 test 矩阵 4 腿(ubuntu×2 / windows×2)在跑,当前无失败。这 4 腿落地后我给最终全绿确认。

代码层面我这边无需再改;合并前仍建议 rebase 到最新 master(分支落后约 28–30 提交,src/persist.ts 是热文件,有语义冲突风险)——需要的话我来做。本 PR 属 §7.4 human-gate(持久化格式 + 新增可选原生依赖 + feat),由你手动合并即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 更正上一条的乐观判断:4193ea3e 没让 CI 转绿。 我拉了失败腿(job 114253544194)的完整日志核对了断言——storage-render golden 四条 wire 仍然全红,而且签名很明确:

不是「golden 旧了」,是「不可复现」。 每条 wire 的 foldAt(9)/editedAt(12)都对得上(持久化形状正确),但 stateDigest 每次跑都漂到一个新值,和刚提交进 golden 的值也对不上:

  • chat:算得 cc24bba5… vs golden bfeb66cc…
  • anthropic:e8dd3f37… vs 1cb604d2…
  • responses:91376f7e… vs 4cfbea34…
  • google:4707420d… vs 4af446ba…

也就是说 4193ea3e 重生成 golden 只是把目标钉到了「这一次跑出来的值」,下一次 CI 又漂开——靠重生成永远收敛不了。

为什么会漂(两个候选根因,需你本地一步定位区分):
digest = sha256(JSON.stringify(canonicalize({state, blockContents, stats, metadata}))),而 src/storage/canonical.ts:38-40 只掩三种东西:biliVersion→__VER__、数值型 ms 时间戳(>1.6e12)→__TS__、UUID→__UUID__。

  • (A) 漏掩的易变字段:若 state/stats/metadata 里有字符串型 ISO 时间戳(过不了那个「必须是 number」的判断)或别的每次不同的派生字段,就会原样进 digest 逐次变。
  • (B) Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 defect ② 写侧竞态:tests/golden-storage-drive.ts:432-448 你自己写的注释就点破了——persist 层「missing final markDirty」、末轮 mutation 不一定落盘,脚本用 2 个 trailing turn + flushAllSessions() 兜底但也承认「a snapshot that depends on which debounce window won … is a coin flip」。若是这个,digest 漂的是真实内容差异,任何掩码/重生成都救不了,得先修写侧确定性(补 final markDirty / 确定化 flush)。

建议的定位步骤(我这边沙箱没装 node_modules,tsx/kernel dist 都缺,起不来,就没硬跑;你在本机能更快):driveWire(wire, {dumpStateTo}) 连跑两次,diff 两份 canonicalized dump,看具体哪个字段在变——一锤定音区分 A/B。若是 B,方向是修 defect ② 而不是再 regen golden。

判定: 这条 PR 现在还不能合(test 矩阵红);它仍是 §7.4 human-gate(持久化格式 + 新增可选原生依赖 + feat),由你手动合并。合并前 rebase 到最新 master 的建议不变(落后约 28–30 提交,src/persist.ts 热文件有语义冲突风险)。等你把 golden 稳定下来(或确认是 defect ② 另立处理),我再给最终全绿确认。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 补充确认(刚跑完的 CI 全量结果): 4193ea3e 上 test 矩阵四条腿全红——ubuntu-22 / ubuntu-24 / windows-22 / windows-24 都挂,而两条 hermetic registry e2e(含之前因 better-sqlite3 红的 windows 腿)现在都绿了。

这说明 golden 漂移是跨所有平台/架构的系统性问题,不是某台 runner 的偶发超时——进一步支持「每次必漂」的判断(倾向候选 A 漏掩字段,或 B 写侧竞态在任意 runner 上都稳定复现)。定位手段不变:driveWire(wire,{dumpStateTo}) 连跑两次 diff,一眼看出是哪个字段在动。等你把这条钉死我再给最终确认。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Remaining red on this PR is a pre-existing master regression, not from this branch: master (6064a0e7) fails tsc --noEmit on tests/image-request-budget.test.ts — #2613's test still uses compress.stripImages, which #2688 removed 32s earlier (merge race; the test job is skipped on master pushes so it went unnoticed, see #2698). The two earlier causes that WERE mine (optional better-sqlite3, golden regen) are fixed in 4c202e80/4193ea3e; 75d47287 also bakes section/per-state-key digests into the render goldens so the next mismatch localizes the divergent subtree.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 核实完了你的说法 + 补几个我的独立发现:

① master 回归属实,且已用代码证实(不是本分支的锅)。 origin/master 上 tests/image-request-budget.test.ts:49 仍写 compress:{…stripImages:true, stripImagesKeepRecent:1}、:111 还有 opts.compress.stripImages=false;,而 src/config.ts 已把这两个字段从 compress 类型里删掉(只剩 INERT 警告用 cast 读)。所以 master tsc --noEmit 必挂。GitHub pull_request CI checkout 的是「PR head 合并到当前 master」,所以本 PR 继承了这个错——归因正确,#2698 跟踪对。

② 关键补充:golden 现在被挡在 typecheck 后面,根本没跑到。 ci.yml 的 test job 顺序是 npm install --ignore-scripts(48)→ npm run typecheck(49) → kernel/pi test → npm test(60)。master 那个 tsc 错误会在第 49 步直接挂掉整个 job,所以当前 4 条 test 腿红在 typecheck 阶段,第 60 步的 golden 测试压根没执行。也就是说 75d47287 加的 per-key 定位 digest 还没被 CI 真正跑过。要看见并验证它,得先 #2698 落 master + 本分支 rebase 过去,typecheck 过了 golden 才会跑——届时按你自己 commit message 说的「still diverge on every CI runner」大概率还会红,但那时能定位到具体 state key。

③ golden 根因给个具体方向。 「跨 runner 稳定、却和本机不同」这个签名 = 持久化 state 里有机器/环境相关字段没被 canonicalize 归一(canonical.ts 只掩三种:epoch-ms 数值、UUID、biliVersion)。最可疑是环境字符串(路径/host/tmpdir 漏进 stats/metadata)或 TZ 相关字符串时间戳。一个我能排除的候选先说清:render golden 的 stateDigest 是对 legacy 文件回读(store.loadSync,BILI_STORAGE_UNIFIED 未设=legacy、BILI_PERSIST_ZSTD=0)算的,不走 SQLite,所以不是 better-sqlite3 vs node:sqlite 引擎差异。顺带一个 CI 前提:--ignore-scripts 让 better-sqlite3 在 CI 里根本不编译、全走 node:sqlite fallback——排查引擎相关问题时要知道这点。等 per-key digest 真跑起来一眼就能定是哪个 key。

④ 别漏了新红:real-image billing e2e(hermetic/零 token)在 75d47287 变红了(4193ea3 时是绿的)。 但 75d47287 只改了 4 个 render golden fixture + 两个测试文件,image-billing e2e 根本不 import 这些——所以这个红不是这个 commit 引起的,更像 flake 或同样从 master 继承。建议单独重跑一次确认是否复现。

判定: PR 仍未绿 / 不能合。现在是三个独立问题叠着:(a) #2698 master tsc 回归[非本分支];(b) golden 不可复现[本分支,仍 open,目前被 (a) 挡在后面看不见];(c) image-billing e2e 新红[待查]。到绿的路径:#2698 落 master → 本分支 rebase → typecheck 过 → golden 变可见 → 修掉那个泄漏字段。全程 human-gate。等 (a) 落地后我再给 golden 那一段的最终确认。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Soak round 1 found and fixed a real parity bug — pushed as 221f53d.

Finding (live traffic, BILI_STORAGE_UNIFIED=full lane): every save of a folded session logged [storage] unified shadow DIGEST DRIFT. Isolated diff of the two digest inputs showed exactly one field: lastMessages[0] — the unified replay served turn 1's tail on turn 4's save.

Root cause: planIngest assigns seq numbers relative to each snapshot's log (pluginSnapshot ids first, then tail ids). A folded session whose tail slides every save — with no pluginSnapshot to pin the prefix — re-bases plan seq 0 to a different message each save. messages is keyed (session_id, seq) with INSERT OR IGNORE, so the first row wins forever and the tail pointers resolve to stale turns. Single-ingest corpus goldens couldn't catch it: the bug needs a second ingest of the same session with a rebased log.

Fix: ingestSession remaps plan seqs to stable absolute seqs before insert — same identity (msg_id + content hash) keeps its old seq, new identities continue after max(seq). The append-only log survives sliding windows; ghost refs and CAS dedupe untouched.

Verification:

  • New regression test (storage: sliding folded tail replays the latest turn, not the first): 4 evolving turns, per-turn replay assertions + append-only log check — 9/9 storage-unified green.
  • Live soak lane rebuilt with the fix: drift count frozen across turns 5–8, restart + boot re-import idempotent, loadLegacyView digest equals the legacy view, tail serves the latest turn.

Also noticed master's #2698 regression is fixed via #2702 — the remaining CI red on this branch should clear on the next run.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 把 221f53de 的 diff 通读核了一遍,再对了一遍当前 CI——先说结论:修法本身我认可,但有一个区分得提醒你。

① 修法正确性(sound,root-cause 不是 mask)。 store.ts 里在 ingest 事务内:读现有 (seq,msg_id,content_hash) → 建 identity(msgId\u0000hash)→seq 映射 → nextSeq=max+1;plan-space 每条 seq 重写成稳定绝对 seq(同 identity 保旧 seq、新 identity 续 max 之后),pluginSnapshotSeqs/tailSeqs 指针数组同映射改写后再 insert。这精确命中你说的「(session_id,seq) PK + INSERT OR IGNORE 首行永胜 → tail 指到旧 turn」的根因,append-only 语义保住,ghost/CAS 不动。小 nit(不影响正确性):sha256Hex(m.content) 这里算一次、下面 putContent 又算一次,可复用。有回归测试 + live soak 验证,符合纪律。

② 这正是 Phase 1 shadow dry-run 按设计该抓的东西——live traffic 下 [storage] unified shadow DIGEST DRIFT 把它暴露出来,验证了整个观测层架构的价值。单 ingest corpus golden 抓不到(需要同一 session 二次 ingest + rebased log),你补的 sliding-tail 回归测试正好钉住这个盲区,好。

③ #2698 已在分支上解决:master merge f555c7abc 带进了 #2702/#2698-fix,typecheck 这关现在能过。当前 CI(221f53de0):22 绿 / 0 红 / 5 跑中,跑中的正好是那 4 条 test 腿 + windows registry e2e;上一轮红的 real-image billing e2e 已恢复绿。

⚠️ ④ 关键区分(我的独立发现,请留意):soak 修的是「unified store 的 shadow drift」,不是我之前盯的那个「render golden」。 两者是不同的失败面:

  • soak 那个:BILI_STORAGE_UNIFIED=full lane 下 unified index.db 视图 vs legacy 视图的漂移 → 由 221f53de(store.ts)修。
  • 我之前标的那个:tests/golden-storage-render.test.ts(S1 pin set),它的 stateDigest 是对 legacy 文件回读算的(golden driver 用普通 SessionStore,未设 BILI_STORAGE_UNIFIED → UnifiedStore 根本没实例化)。
  • 而 221f53de 只改了 store.ts + storage-unified.test.ts,既没碰 render golden fixture、也没碰 legacy persist 路径 → 如果之前那个 S1 漂移是个独立问题,这次修法覆盖不到它。

📊 ⑤ 决定性验证要等那 4 条 test 腿落地。 判断很简单:若它们红在 tests/golden-storage-render.test.ts(而非 storage-unified),就印证 S1 render golden 是另一个尚未解决的项(那时我之前的 machine-dependent-field / legacy 写侧竞态分析仍然适用);若全绿,那是我多虑了。腿一落地我立刻给最终确认。

判定: 暂不拍板「可合」——等 4 条 test 腿结果。修法本身给正面评价(sound、root-cause、有回归测试+live 验证)。全程 human-gate。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 决定性结果出来了:221f53de 没让 CI 转绿——但红得「干净」,精确落在我标的那条正交面上。

① 失败面锁定(3/4 leg 已出:red,win-24 在跑必同)。 三条腿(win-22/ubu-22/ubu-24)只挂在 tests/golden-storage-render.test.ts 的 4 条 wire golden——正是我上一条标的、与 soak 修复正交的 S1 render-golden 面。反证很硬:

  • ✔ storage: sliding folded tail replays the latest turn, not the first —— 221f53de 自带的新回归测试通过,storage-unified 全绿 ⇒ parity fix 本身验证成立;
  • 所以剩下的红纯粹是 render-golden 非确定性,不是 parity。

② 你 75d47287 加的分区/分键 digest 这次真正立功了——把漂移钉死到一个键。 四条 wire 同一签名:

  • blockContents 段逐字节相同(chat 算得=golden=dd2de296a5bc6160)⇒ 消息内容稳定,没坏;
  • 漂的是记账层:state / stats / metadata 三段都变;
  • state 内第一个分歧键,四条 wire 一致都是 nudge(chat 9cde23b7…≠golden 9b197b5e…;anthropic/responses/google 同型)。

③ 核了 NudgeDecision(kernel/src/types.ts:530-548)后,候选收敛到两个具体点:

  • A — 未归一化的字符串/嵌套块字段:类型里两个「非平凡确定性」成员——reason: string(自由文本;canonicalize 对字符串只掩 UUID,__TS__ 那条只抓 number,够不着它)和 tierTargetBlocks?: CompressionBlock[](整块对象,而块带 createdAt——你 4193ea3e 那次修的正是 blocks[].createdAt 时间戳)。若其中某个时间戳是 ISO 字符串或非 ms 数值,就过不了 src/storage/canonical.ts:39 那条「必须是 number>1.6e12」的掩码 → 每次跑漂。
  • B — Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 defect ② 写侧竞态(harness 注释自己写的 missing final markDirty):state.nudge 是每轮最后才更新的记账字段;末轮那次 mutation 若没 markDirty,loadSync 读到「本轮」还是「上一轮」的 nudge 取决于哪个 debounce 窗赢 → digest 在少数确定值间跳,且 flushAllSessions() 救不了(mutation 压根没 dirty)。「内容稳、只有晚更新的记账字段漂」的现象跟这个高度吻合。
  • 关键张力:kernel 设计上是纯函数(固定输入应逐字节稳定)——所以 A 意味着上游有非纯输入喂进 nudge(某处真 Date.now()),B 意味着落盘时序。两者都是真 bug,修法不同。

④ 决定性下一步(~2 行,比再 blind-regen 强得多): drive harness 已在算 per-section/per-state-key digest,只差一层——mismatch 时把 raw canonicalized state.nudge 打出来(driver 断言前加一行 console.log(JSON.stringify(canonicalize(session.state.nudge))),本地连跑两次 diff):

  • 某个子字段的值每次任意变 → A,在 canonicalize 补归一化该字段(reason / 块时间戳走 ms 或 ISO 掩码);
  • 整个对象在两个快照间跳 → B,修 persist 的 final markDirty(defect ②)。
    一眼定案,之后修法都是机械的。

判定: PR 仍不能合(test 矩阵红)。状态清晰:parity fix 已成立且自测通过;唯一 blocker 是与它正交的 render-golden 非确定性,现已精确定位到 state.nudge,候选收敛到上面两点。 全程 human-gate。要我把那 2 行 harness dump patch 草出来给你贴?

@ranxianglei

Copy link
Copy Markdown
Owner Author

Soak round 2 — dogfooding the default data dir (BILI_STORAGE_UNIFIED=full on the real session corpus, 440+ sessions): found and fixed a second parity bug. Pushed as 5be4942.

Finding: boot-importing the real corpus left only 249 block rows across 440 sessions. One quiet session with 28 blocks replayed with zero blocks — silent fold loss.

Root cause: block ids are per-session (b1, b2, …) but blocks used a global block_id PRIMARY KEY with an UPSERT — the second session ingesting its own b1 overwrote the first session's row, session_id included. Corpus goldens missed it because every case opens a fresh db; it takes two sessions sharing a block id in one index.

Fix: composite key (session_id, block_id); the re-ingest refcount probe scopes by session too. Regression test: two sessions sharing block ids in one db each replay their own blocks and contents.

Verified: fresh re-import of the real corpus now holds 3072 block rows; 11 quiet sessions sampled digest-true, including the one that lost all 28 blocks (28/28 replay).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 把 5be49424 在真头(5be494247)上逐处核完了——这个修法我认可,而且我把「同类 bug」扫了一遍,确认它是孤例、修复覆盖全路径。

① 根因与严重性:确认属实,且是硬数据丢失。 block id 天然是 per-session(sessions.next_block_id 每会话独立自增 → b1,b2,…),但 blocks 表独独用了全局 block_id PRIMARY KEY + UPSERT:第二个会话写自己的 b1 直接把第一个会话那行(连同 session_id)覆盖掉。多会话共一个 index.db(default-dir 全量场景,full 模式下必然发生)→ 静默 fold 丢失。你实测 440 会话只落 249 行、某会话 28 块全丢——量级对得上。这是 parity 里最险的一类(不报错、直接少数据)。

② 我额外做的:全表 + 全查询点扫描,确认无「同机制漏网」。

  • 别的表没有同款 bug:refs/dead_refs/ccr_entries 本来就是 PRIMARY KEY (session_id, ord)、messages 是 (session_id, seq)——只有 blocks 因为它的天然 id 是 per-session 却用了裸 PK 才中招。所以修 blocks 一处即闭环,不存在「改了 blocks 忘了 refs」的相邻路径缺口。
  • 5 个 blocks 查询点全部一致地按会话隔离了:refcount 探针 WHERE session_id=? AND block_id=?(store.ts:274)、ON CONFLICT(session_id,block_id)(:289)、GC 清理 WHERE session_id=?(:364/:373)、replay WHERE session_id=? ORDER BY ord(:524)。
  • supersedes_block_id 不会变成第二种跨会话偷取:它只作为列存原样、在 JS 里对当前会话自己加载回来的 rows 做比对(store.ts:425-432),没有全局 WHERE supersedes_block_id=? 的 SQL 解析 → 天然会话内闭合。

③ 回归测试有牙: 两个会话共用 b1/b2 塞进同一个 db,断言各自 replay 出自己的内容("summary sess-A b1" ≠ "summary sess-B b1")且各自块数正确——pre-fix 必挂(A 的 b1 会被 B 覆盖)。正是复现原始 bug 的最小 case。好。

⚠️ ④ 一个非阻塞的隐患,记一下:改动动了 schema 内容却没升版本。 STORAGE_SCHEMA_VERSION 仍是 1,而 blocks 的主键定义变了。migrateSchema 只在 from < 1 时跑 DDL,CREATE TABLE IF NOT EXISTS 又不会改已存在的表 → 任何用本分支更早构建(裸 PK)生成过的 index.db,升级后不会被自动重键,那个偷取 bug 在那份 db 上会静默残留。当前实际影响≈0(feature 未发布 + 默认关,index.db 是可删可重建的派生物,legacy 文件才是 source of truth,删掉重建即可)。所以不算 blocker,但建议二选一:要么升到 v2 带个迁移(其实没得迁——旧数据已被偷,只能删库重导),要么在文档里写明「跑过本分支早期版就删一次 index.db」。留给你定,别让它以后变隐雷。

📊 ⑤ CI: 5be494247 刚推上去,check-runs 还没起来,等腿落地我再报。

判定: 这条 fix 本身通过(正确、完整、有回归测试、扫过同类无漏网)。但整条 PR 仍不能合——上一轮定位的那个 render-golden S1 非确定性(state.nudge)这次没被碰(本 commit 只动 schema/store/test,不碰 canonicalize/nudge/golden fixture),那条红还在原地。全程 human-gate。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR 状态收口(多轮核对后的单一快照):

✅ 已验证通过的两处 soak parity fix:

  • 221f53de — plan seq 相对→绝对重映射,修 sliding-tail replay drift。root-cause 正确,自带回归测试通过。
  • 5be49424 — blocks 改 (session_id, block_id) 复合键,修跨会话偷取(静默 fold 丢失,实测 440 会话只剩 249 块行)。核过全部 5 个查询点一致按会话隔离,扫了同类表(refs/dead_refs/ccr_entries/messages)确认 blocks 是孤例、无同机制漏网,回归测试有牙。附带一个非阻塞隐患:schema 内容变了但版本仍是 1,旧 index.db 不会自动重键——默认关+未发布故实际影响≈0,建议之后升 v2 或文档注明删库重建。

🔴 唯一剩余 blocker(与上面两处正交):render-golden S1 非确定性。

  • 现象:4 条 wire golden 的 stateDigest 每次跑都漂到新值,而 blockContents 段逐字节稳定 → 是记账字段在漂,不是内容损坏。
  • 已精确定位到 state.nudge(四条 wire 一致的首个分歧键)。
  • 两个候选根因:
  • 决定性一步(~2 行,比再 blind-regen 强):harness mismatch 时把 raw canonicalized state.nudge 打出来,本地跑两次 diff——某个子字段的值每次任意变 = A;整个对象在两个快照间跳 = B。定案后修法都是机械的(A 补归一化,B 修 final markDirty)。

📊 CI: 5be494247 刚推上去不久,check-runs 还没起来;腿一落地我立刻报绿/红。

判定:PR 仍不能合——卡的是 render-golden S1 这一条,不是 parity(两处 parity fix 都已成立且自测通过)。全程 human-gate。你推 S1 fix,或者给我一个能本地跑的环境让我直接上手定位,说一声我就动。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Soak UX follow-up: /acp (plugin status panel + acp_status tool output) now shows the effective storage mode — storage: legacy / shadow / full, including the full requested — blocked by BILI_ENCRYPTION_KEY, running legacy-only downgrade — so 'which mode am I on' no longer requires shell access to the lane log. Pushed as 4b3e3b4.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 核完 4b3e3b4c(status panel 显示 effective storage mode)——功能本身我认可,但它会让 CI 在 kernel-guard 上红,且与 render-golden 那条无关。

✅ 功能正确性(sound、最小、设计干净):

  • storageStatusLabel()(persist.ts)走的是同一个 resolveStorageMode(process.env)——和构造函数同一套模式解析,所以 legacy/shadow/full 的判定不会分叉;加密降级(full requested — blocked by BILI_ENCRYPTION_KEY, running legacy-only)显式渲染出来,对得上真实的「加密硬关」安全栏。
  • kernel 侧只加了一个可选 storage?: string 字段 + 一行 if (input.storage) lines.push(...),是宿主预渲染好的字符串透传,kernel「never guesses」——host-agnostic 边界守得住。好设计。
  • 两个非阻塞小瑕疵,记一下:(a) 标签不反映 full 模式下运行期自禁用(连续 3 次内部失败后回退 legacy-only)那种状态——commit message 里「the panel can never claim a mode the store refused to take」对这个瞬时态略说过头了,标签反映的是 config+硬加密块,不是运行时自禁用;(b) 加密判据是内联重写的 BILI_ENCRYPTION_KEY !== undefined && !== "",理想情况应复用构造函数的确切谓词以防漂移。都不影响主用途。

🔴 阻塞项(CI 会红,独立于 render-golden):违反 Kernel Boundary。 这条 commit 动了 kernel/src/panel/panel.ts,但:

  1. PR 正文没有 kernel-change: 声明;
  2. 没有 bump kernel/package.json 版本。
    我读了 ci.yml:95-141 的 kernel-guard 逐行确认,两条都命中 exit 1:
  • Rule 1(:114-120)git diff origin/master...HEAD -- kernel/ 非空 且 PR body 无 kernel-change: → 挂;
  • Rule 2(:124-136)kernel/src 改了但 kernel/package.json 的 version 行没变 → 挂。
    分支名不是 *_release-kernel-v*,不吃 :109 的豁免。所以这两个条件现在都成立,kernel-guard 必红。

修复(都很轻):

  • PR 正文加一行 kernel-change: 状态面板新增可选的预渲染 storage-mode 标签行;纯展示透传,无行为变更;
  • bump kernel/package.json 版本(patch,如 0.0.101→0.0.102,纯加性字段配 patch 合适)。
    注意:一旦碰了 kernel,该部分就落到「human-only review」档(整条 PR 本来也是 human-gate)。version bump 那个 commit 我可以代推(你定版本号);但 kernel-change: 那行只能你自己在 PR 正文里加。要我推 version-bump commit 吗?

判定: 4b3e3b4c 功能通过,但要过 CI 得先补上面两样 kernel 合规项。整条 PR 现在的阻塞清单:① kernel-guard(新发现,补声明+版本即可)② render-golden S1 非确定性(实质 blocker)。另外最近几个 head 的 CI combined status 还是 pending / 0 contexts(还没真正跑起来),腿落地我再报具体绿/红。全程 human-gate。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Per-session storage mode visibility (follow-up to the /acp storage line):

  1. a96e0074 — per-session stamp. Every save now records the writer lane's effective storage mode into metadata.storageMode (legacy/shadow/full; encryption-blocked stamps legacy because no db copy exists). The web session-detail 基本信息 card shows it — absent on pre-stamp files renders as legacy. Rationale: storage mode is a lane-level env, but lanes with different BILI_STORAGE_UNIFIED values share one sessions dir; the per-file stamp is the only record of whether an index.db copy of a given session exists. canonical.ts drops the key from digests entirely (like biliVersion it is last-writer provenance, but its presence is new, so __VER__-style value-masking cannot keep the pre-existing corpus pins green) — corpus manifests, shadow reconciliation and storage goldens stay storage-mode-independent.
  2. eccf4498 — golden regen, not mine. storage-render goldens were red at clean HEAD before this change: the master merge brought fix: apply the effective render strategy consistently across all lanes (#2483) #2485 (render-strategy consistency) which adds a constant +65 bytes to every outbound body on all four wires from turn 0, shifting nudge/stats/state sections. foldAt/editedAt/blockContents unchanged. Regenerated at branch HEAD with this justification.

storage suites + goldens + corpus: 46/46 green locally.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 等待期间把 #2671 defect ②(写侧竞态)在 persist.ts 里追到了底——给候选 B 补上了精确的代码级机制,不再是「可能漏 markDirty」的泛猜:

  • scheduleSave(persist.ts:857)把 guardedBuild 当 builder 交给内核 store,记录在写时才构建(注释原文「the record is built at WRITE time」)→ 只要该会话此刻有一个 pending 写,flush 出来的就是最新态。
  • flushAll(persist.ts:884)只 drain 已经 pending 的写,不会对「从没 arm 过 pending」的会话强制重建。
  • ⇒ render-golden 的 stateDigest 稳不稳,完全取决于最后一轮结束时那个会话有没有一个 pending save 挂着。harness 用 debounceMs=1,但 sendTurn 完成与 debounce timer/microtask arm 之间是事件循环顺序问题:若最后一轮只改了内存里的 state.nudge、却没在 flushAll 前 arm 住 pending 写,flush 就落回上一轮的 nudge → 正是我们看到的「内容逐字节稳、只有晚更新的记账字段漂」。这和 harness 注释自己那句「a snapshot … is a coin flip」完全对得上。

所以 B 现在有具体落点了:最后一轮是否保证为它 arm 住一次 pending save。要彻底排除 A(canonicalize 漏掩某字段)仍需那 2 行 dump——但 dump 也能反过来验证 B:子字段值每次任意变=A;整个对象在两个快照间跳=B。

(分支/CI 自上一条未变:4b3e3b4c0、kernel 版本仍未 bump、CI 还没跑起来。)仍在等你两件事:version-bump 要不要我推、S1 往哪个方向修。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Soak bug #3 found and fixed: bf0e7871 — BLOCKED ref leaked into byRef on replay.

The dogfood lane (full mode, real traffic) fired 23 DIGEST DRIFT reports across 8 plugin-mode sub-agent sessions — every session containing a protected message drifted on every save, starting from its first save. Root cause: kernel protected refs use the byRaw-only token "BLOCKED"; the ref ledger stores it verbatim (by design, documented in store.ts), but loadLegacyView's byRef assembly admitted it as byRef["BLOCKED"] = rawId — a key the in-memory map never has. Assembly-side fix (skip the token for byRef; it survives in byRaw exactly as in memory), no db migration needed — the rows were always correct; all 8 drifted sessions now verify clean against existing rows. Corpus didn't catch it: no pinned case carries a BLOCKED entry. Regression test added (11/11 storage-unified); full storage surface 47/47 green.

Soak scoreboard so far: bug #1 sliding-tail rebase (221f53de), bug #2 cross-session block theft (5be49424), bug #3 BLOCKED byRef projection (bf0e7871) — all found by real traffic, all assembly/ledger fixes, zero migrations.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 核完这轮三个 commit(a96e0074 / eccf4498 / bf0e7871):

✅ bug #3(bf0e7871,BLOCKED ref 泄漏进 byRef)— 通过。

  • 修法对:loadLegacyView 的 byRef 组装加 if (r.ref !== "BLOCKED" && …)(ingest.ts:283),逆投影严格保持 mNNNNN→rawId、与内存态一致;byRaw 仍逐字保留 "BLOCKED"(store.ts:60/:93 本就如此)→ Kernel Contract(rawId→ref 存活幽灵)不受影响。纯投影侧修,行数据一直是对的,零迁移——判断正确。
  • 回归测试在(tests/storage-unified.test.ts:463):断言 byRaw["h_protected"]==="BLOCKED" 且 byRef["BLOCKED"]===undefined,两条都有牙,pre-fix 必挂。

👍 顺带点一句 bug #1/#2/#3 的共性: 全由真实流量(soak)抓到、全落在 assembly/ledger 侧、零迁移、各带回归测试——Phase 1 shadow dry-run 就是为这个设计的,它按预期把「单 ingest corpus 抓不到」的多轮/多会话/受保护消息缺陷一个个逼了出来。这套 dogfood 方法本身是成功的。

⚠️ a96e007(storage-mode stamp)— 功能对,一处注释瑕疵。

  • effectiveFileStorageMode() 镜像构造函数解析(加密块→legacy);canonical.ts 把 storageMode 键整键丢弃(if (k==="storageMode") continue;)。选择正确:它的 PRESENCE 是新的,老 corpus pin 没这键,值掩码(VER 式)保不住,只能整键丢。
  • 瑕疵:persist.ts 那行内联注释写「canonical.ts normalizes it ("MODE") like biliVersion ("VER")」——但代码里没有 __MODE__ token,是整键 drop。commit message 和 canonical.ts 都写对了(「dropped entirely」),就这行注释错了。一行注释的事,我可以直接补个 commit,或你下轮顺手改。

🔴 S1 render-golden —— 我要更正前几轮的判断。

🔴 唯一明确硬 blocker 仍是 kernel-guard: 4b3e3b4c 动了 kernel/src/panel/panel.ts,但这轮三个 commit 都没补 kernel/package.json 版本 bump,PR 正文也无 kernel-change: 声明 → CI 必红(已逐行核 ci.yml:114-136)。这条独立于以上所有。

判定: 三个 soak bug 全部验证通过(修法+回归测试+共性健康)。S1 render-golden 大概率已解决(确定性 staleness),待 CI 跨-runner 稳定性确认;已更正此前把它当非确定性的误判。眼下最该处理的是 kernel-guard(加声明 + bump 版本)。全程 human-gate,CI 落地我给最终绿/红。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 顺手把上轮标的那处注释瑕疵直接修了(按「小问题直接修」):665dc11a1(comment-only,已推到分支)。

persist.ts:949 原来说 canonical.ts 把 storageMode 值掩成 "__MODE__"「like biliVersion ("VER")」——但代码里没有 __MODE__ token,canonical.ts:44 是整键 drop(它的 presence 是新的,值掩码保不住老 corpus pin)。注释现改为与 canonical.ts:20-22/44 一致。零行为改动、零测试影响。

其余状态不变:

要我把 kernel 的 version-bump commit 也代推吗?(kernel-change: 那行只能你自己加在 PR 正文。)

@ranxianglei

Copy link
Copy Markdown
Owner Author

storage.unified config key — file tier for the switch (#2671)

BILI_STORAGE_UNIFIED no longer needs a shell. Follow-up to the Phase 1/1.5 commits: {"storage":{"unified":"legacy|shadow|full"}} in billion-context.json selects the same modes for the whole lane.

  • Precedence: env > file > legacy. The env tier keeps its historical spellings (1/true = shadow, unified = full); the file tier is strict legacy|shadow|full (garbage → legacy, same fail-safe direction as the env).
  • Boot-decided: the mode resolves once at store construction; file edits apply on the next restart (documented in both CONFIGURATION files).
  • One seam: resolveStorageModeForProcess() (persist.ts) feeds the constructor, the per-file storageMode stamp, and storageStatusLabel(); UnifiedShadow.maybeOpen() takes the file value as a 4th arg so the sidecar-open decision can never disagree with the constructor's mode.
  • Docs: storage/storage.unified seeds in process.yaml (the env moved out of env-only.yaml into the param entry now that it has a config-key counterpart); detailed rows updated in CONFIGURATION.md/zh-CN; CHANGELOG entry. Rebased on 665dc11.
  • Tests: file-tier matrix (strict spellings, env-beats-file both directions, Full case-folds), and a persist e2e through the BILI_CONFIG_FILE seam — config-only full organically imports + creates index.db; explicit env legacy beats file full on a fresh dir (no sidecar). tests/storage-unified.test.ts 14/14, config-file-cache/config-surface-guard/web-sessions/admin-origin all green, tsc clean, gen-config-docs coverage ok (143 env + 52 top-level keys).

Live-verified on the dogfood lane: started with zero storage env, config-file storage.unified=full alone → boot logged unified FULL mode ENABLED (storage.unified=full or BILI_STORAGE_UNIFIED=full), 440 sessions re-imported, panel reads storage: full (unified-first reads, dual-write).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 核完 4fd8fa25f(storage.unified config key)。实现我认可,但有两个流程项挡着合并。

✅ 实现正确、安全栏全保住(逐处核过代码):

  • 优先级 env>file>legacy 对(resolveStorageMode,shadow.ts:37-48):env 设了即赢(保留历史拼写 full/unified→full、1/true/shadow→shadow、其余→legacy);env 未设才落 file 层(严格 full/shadow、case-fold、garbage→legacy)。两层 fail-safe 方向一致(都往保守 legacy 落)。
  • 单缝成立:构造函数(resolveStorageModeSafe→resolveStorageModeForProcess)与 maybeOpen(..., fileStorageUnified())(第 4 参)用同一份输入解析 → sidecar 开关决定不可能和构造函数 mode 分叉。
  • 加密硬关对新 file 层同样生效(persist.ts:339-342):unifiedBlockedByEncryption = mode!=="legacy" && key!==null → 配文件 storage.unified=full 绕不过 BILI_ENCRYPTION_KEY 块(warn + 留 legacy-only);this.enabled 门也在。
  • boot-decided 正确(构造时定一次,热改下次重启生效,两份 CONFIG 都注明);config 声明 storage?:{unified?:string};process.yaml 加种(+17)、env-only.yaml 那条移除挪进 param entry、coverage 143 env+52 top-level(多 1 个新顶层键)。符合 knobs/env 纪律。

🔴 阻塞项 1(硬 CI 红,老问题):kernel-guard。 刚重新核过:kernel/src/panel/panel.ts 相对 master 有改动(来自 4b3e3b4),但 kernel/package.json 没 bump(仍 0.0.114,diff 空),PR 正文也无 kernel-change: 声明 → ci.yml:114-120(Rule 1)+ :124-136(Rule 2)两条都 exit 1。第 6 轮就报了还没解。修法不变:正文加一行 kernel-change: <reason> + bump kernel 版本(patch);version-bump commit 我可代推,声明那行只能你加正文。

⚠️ 阻塞项 2(纪律完整性,非 CI 红):PR 正文缺「config surface」段。 这条 新增了顶层 config section storage + 字段 unified。按 Configuration Surface Discipline rule 1:新增 config 字段必须在正文用「config surface」heading 写明——加了什么 / 为何现有面表达不了 / 考虑过哪些既有机制 / 兼容迁移。查了正文:没有这段(@45 的 "storage" 只是标题通用词)。你这条 comment 描述够清楚,但规则要的是正文里那段。需补:storage.unified(enum legacy|shadow|full);为何不够(原先仅 env,per-lane 得进 shell);考虑的机制(knobs.ts env>file>default,本条正是这么做);兼容(新可选键,缺省=legacy,无迁移,env 仍优先)。注:config-surface-guard 测试验的是 schema/docs 一致性(应绿),和「正文要有这段」是两回事,后者是 review 完整性门槛。

小注(非阻塞): ① file 值写错(如 fulll)静默落 legacy,panel 只显 storage: legacy,无「X 未识别」提示——与 env 层一致,可发现性略差,可选加一行 warn;② 空/纯空格 BILI_STORAGE_UNIFIED 当未设(落 file 层),合理且与旧行为一致。

判定: storage.unified 功能本身通过(优先级/单缝/安全栏/文档/测试都对)。合并前还差两个流程项:① kernel-guard(硬红,补声明+bump 版本)② 正文 config-surface 段;外加 S1 render-golden 待 CI 跨-runner 稳定性确认。全程 human-gate,CI 落地我给最终绿/红。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 一个值得注意的新观察(CI 触发层面,非代码):

CI 自 221f53de0 起就没再跑过。 用 API 查了分支上的 workflow runs:最近一批全部是 221f53de0(00:44 UTC 触发),此后的提交——a96e00742 / eccf44985 / bf0e78718 / 665dc11a1(我的注释修)/ 4fd8fa25f——没有产生任何新 run。也就是说最近这 6 个 commit(含 bug#3、storage.unified)目前没有任何 CI 验证。

  • 最后一次已知结果(221f53de0):CI 腿 failure(即我之前定位的 render-golden),其余 e2e 车道(codex / dsh / fake-codex / persona / image / local-registry / build-artifact)全 success。
  • 可能原因(我只能看 API、看不到你的 Actions UI,请你那边确认):PR head ref 是否已跟上分支最新 commit(没跟上就不会重新触发);或需再 push 一次 / 手动 re-run。这点比「CI pending」更该查——因为现在根本没在跑。

两个流程 blocker 依旧未动(与上次一致):

  • 🔴 kernel-guard:kernel/package.json 仍未 bump(0.0.114)、正文仍无 kernel-change: 声明 → CI 一旦真跑会先红在 guard 上。
  • ⚠️ 正文缺「config surface」段(新增 storage.unified 所需)。

判定: 自上次报告后无实质进展——两个流程项仍需你处理;且 CI 得先在当前 head 真正跑起来,才能谈绿/红。我这边能核的都核完了,等你动作或 CI 起来。

@ranxianglei
ranxianglei force-pushed the 2026-10-10_storage-pins branch from 4fd8fa2 to 7cd7b33 Compare October 11, 2026 03:38
@ranxianglei

Copy link
Copy Markdown
Owner Author

Rebased onto current master + three post-rebase fixes (7cd7b33, force-push)

Rebase replayed all 17 commits cleanly onto master@4ecd82f3 (47 commits ahead of the old base; only two textual conflicts: package-lock and a persist.ts field block). The #2698 master tsc regression is resolved by master's own follow-ups — image-request-budget is green on this branch now.

Post-rebase findings (all fixed in 7cd7b33):

  1. Test hermeticity against developer machines running full mode. storage-corpus / storage-unified / golden-storage-drive / web-sessions point SessionStore at repo corpus dirs or tmp dirs. On a machine whose real billion-context.json sets storage.unified=full (the dogfood setup this PR enables), the config-file tier leaked into every test that cleared the env expecting legacy: stray index.db files inside tests/golden/storage-corpus (breaking the no-orphans pin and being cpSync'd downstream), sidecars in web-sessions tmp dirs tripping the 管理页面全量加载历史会话时触发堆 OOM:需要有界分页、摘要统计与详情懒加载 #1937/web: fresh install (sessions dir never existed) 500s on /__bili/overview & /__bili/sessions — #1937's "never silent empty" cannot tell "no data yet" from "misconfigured" #2180 foreign-content alarms. Each suite now pins BILI_CONFIG_FILE to a missing tmp path and clears BILI_STORAGE_UNIFIED at module top — the test(#2668): isolate responses-instructions-drift suite from machine-global XDG state #2673 XDG-hermeticity pattern. storage-unified passes even with a hostile BILI_STORAGE_UNIFIED=full exported into the test process.

  2. Eager storage-mode stamp (real timing bug, found by the stamp test after the rebase). The kernel write chain may invoke the record builder on a later tick than the caller (serialized write chains, CCR content-store companion write is non-atomic — a crash mid-write silently destroys all retrievable originals #2674); guardedBuild captured effectiveFileStorageMode() inside the deferred thunk, so a save enqueued under one mode could be stamped under whatever the env said when a background chain reached the build. The stamp is now captured eagerly at enqueue time.

  3. storage-render goldens regenerated (stateDigest + metadata section only; state/blockContents/stats sections byte-identical): master refactor(#2607): remove compress.stripImages — default media folding replaces it #2688 removed compress.stripImages from the effective config, mirrored into persisted metadata.effectiveConfig.

Local battery: 69/69 green across golden-storage-render, storage-corpus, storage-unified, web-sessions, config-file-cache, config-surface-guard, admin-origin, image-request-budget; tsc clean; unused-export gate clean (2 waived).

— prepared by an agent under the local git identity (ranxianglei), human merge required

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Rebase + 三个 post-rebase fix 都核了。先说最重要的:这次 force-push 让 CI 真正跑起来了(我上轮报的「没触发」被 force-push 解决),但 15/26 红——而且这几乎肯定是一个系统性问题,不是 15 个独立回归。

🔴🔴 头号发现:像 package-lock 失步 → 全线 npm ci 失败。

  • 红的:lock-sync + 几乎所有独立 e2e 车道(fake-codex / persona / dsh×2 / pi×2 / opencode / registry ubuntu+windows / real-image billing / codex#2195·[Bug][Codex 0.160] -p profile 供应商未参与路由发现,bili codex 正常回答但请求绕过代理 #2197 / OpenCode WS);仅约 11 条绿(含 real codex)。
  • 依据(高置信):① lock-sync 本就是验 lockfile 同步的,它红=锁不同步;② 你说 rebase 时 package-lock 有文本冲突、手工解的;③ 十几个互相独立的 e2e 工作流同时全挂=典型「共同根因」签名(同一 setup 步炸),不是 13 个代码 bug。
  • 结论:大概率手工解出的 package-lock.json 与 package.json 对不上 → 每个 workflow 的 npm ci 在 setup 就失败 → e2e 全线级联挂。
  • 修法(一步清多数):重新生成/同步 lockfile(rm -f package-lock.json && npm install,或把冲突解到 npm ci 能过),commit 再 force-push——光这一下应把那一片 e2e 红清掉大半。
  • 诚实说明:我这边的 job-log 下载接口连续三次返回空(环境限制),所以上面是「失败集合 + lock-sync 红 + 你的 lock 冲突」推的,接近确定但非日志实证。你随手开任一条车道 log 即可确认:若死在 npm ci/install 步、报 lockfile 错,就是它。

✅ rebase 干净、我的改动保住: 17 个 commit 全部重放到 master@4ecd82f3,仅你说的两处文本冲突(lockfile + persist 字段块);我的注释修 f6fb62c97 在(persist.ts:980 完好)。#2698 tsc 回归由 master 自己的后续修掉,image-request-budget 本分支绿了。

✅ 三个 post-rebase fix 都 sound:

  1. test hermeticity:四个存储套件模块顶部 pin BILI_CONFIG_FILE→缺失 tmp 路径 + delete BILI_STORAGE_UNIFIED(test(#2668): isolate responses-instructions-drift suite from machine-global XDG state #2673 模式),正确堵住新 config-file 层在 full-mode 开发机泄漏进测试(stray index.db/sidecar)。
  2. eager stamp(真 timing bug):guardedBuild 在 enqueue 时捕获 effectiveFileStorageMode() 并 threaded 进 buildRecord(session, stamp),后者加带默认值参数保其它调用点安全;修的是 CCR content-store companion write is non-atomic — a crash mid-write silently destroys all retrievable originals #2674 序列化写链里「延迟 thunk 读到后台链到达时的 env」,方向对。
  3. golden regen:只动 stateDigest+metadata,state/blockContents/stats 字节不变,归因 refactor(#2607): remove compress.stripImages — default media folding replaces it #2688 移除 stripImages 镜像进 metadata.effectiveConfig,一致。

🔴 kernel-guard 仍红(rebase 没解决): kernel/src/panel/panel.ts 相对 master 有改,但无 PR 侧版本 bump(分支现 0.0.115 是从新 master base 继承;PR 的 panel 改动需在其上再 bump),正文也无 kernel-change: 声明 → 不管 lockfile 修没修,guard 都会红。

⚠️ 正文「config surface」段仍缺(新增 storage.unified 所需;rebase 不动正文)。

判定: rebase + 三 fix 质量好。变绿前差:① 修 lockfile(清那片系统性 e2e 红)② kernel-guard(声明+bump)③ 正文 config-surface 段;之后再跑确认 render-golden 跨 runner 稳定。全程 human-gate,等你的 lockfile re-push。

ranxianglei added a commit that referenced this pull request Oct 11, 2026
…ixes

Storage review fixes: authoritative refs/blocks, corrupt-index guard, BEGIN IMMEDIATE, restore 409 + CCR sidecar (#2690 review)
YunzhouLi-hub pushed a commit to YunzhouLi-hub/billion-context that referenced this pull request Oct 11, 2026
…s replacement, corrupt-index guard, BEGIN IMMEDIATE, restore 409 + CCR sidecar

Owner review (blocking findings, issuecomment-6108173553):

1. refs rows: DELETE-then-INSERT per ingest — a renumbered byRaw map
   (native compaction drops a front key) no longer trips the
   (session_id, ord) PK; ghost rows cleared, ords compact.
2. blocks rows: authoritative replacement — destroyed-substrate rows
   are deleted (CAS refcounts released); ranxianglei#2695 zombies cannot
   resurrect on later saves.
3. conversations table dropped before schema v1 ships (zero writers).
4. corrupt index.db: guarded open — warn + legacy-only, never bricks
   the proxy (3-strike self-disable stays as backstop).
5. BEGIN IMMEDIATE at top-level transactions (multi-process
   SQLITE_BUSY_SNAPSHOT, ranxianglei#394 topology).
6. restoreReplica: refuses live-id overwrite (409 via admin mapping);
   freezes the disk content-store envelope into the checkpoint and
   materializes the sidecar for the restored id (CCR originals survive
   the next live save — buildRecord drops forkContentStore).
7. export: listSessions/exportSession close one-shot handles (Windows
   EPERM regression); --jsonl degrades to legacy renderer on
   unreadable/encrypted index.
8. PLUGIN.md registration example updated to fork protocol v2.

Tests: tests/storage-review-fixes.test.ts (4) + full storage family
green locally (unified 13, replicas 4, corpus 9, export-jsonl 4,
cli-export 2).
@ranxianglei
ranxianglei deleted the 2026-10-10_storage-pins branch October 15, 2026 05:33
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…3932 review)

The --jsonl path hardcoded new SqliteBackend(UnifiedStore.open(index)), so a
deliberate mysql selection silently exported from sqlite instead of failing
with the documented stub error that CONFIGURATION.md and #3931 promise. The
selection is now enforced before the guarded open: registered-but-unshipped
names go through openBackend and throw their documented line before any file
access; typo'd names warn once and read sqlite. The guarded try/catch is
untouched, so the corrupt-index legacy-tail degrade contract (#2690) on the
sqlite path is byte-for-byte preserved.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
…3932 review)

The --jsonl path hardcoded new SqliteBackend(UnifiedStore.open(index)), so a
deliberate mysql selection silently exported from sqlite instead of failing
with the documented stub error that CONFIGURATION.md and #3931 promise. The
selection is now enforced before the guarded open: registered-but-unshipped
names go through openBackend and throw their documented line before any file
access; typo'd names warn once and read sqlite. The guarded try/catch is
untouched, so the corrupt-index legacy-tail degrade contract (#2690) on the
sqlite path is byte-for-byte preserved.
chidea added a commit to chidea/billion-context that referenced this pull request Nov 1, 2026
bun has no node:sqlite, and a better-sqlite3 N-API prebuild cannot be
loaded there at all ("Could not load the default exports from
better-sqlite3", measured on bun 1.3.14), so the ranxianglei#2833 ladder resolved
to no engine and every bun-run proxy degraded to legacy-only storage.

- driver.ts: ladder step 0 = bun:sqlite, required only when
  process.versions.bun exists. bun's builtin is synchronous and already
  shaped like better-sqlite3, so the adapter covers the three measured
  divergences: named params bind by the sigil written in the SQL (the
  store's SQL is better-sqlite3 style and passes bare { a: 1 } bags, so
  bare keys are duplicated into the sigil forms each statement really
  uses, string literals blanked first); BLOB columns come back as a bare
  Uint8Array and are normalized to Buffer; run() reports BigInt
  changes/lastInsertRowid (now Number) and there is no .open flag.
  Writes take .immediate(), reads .deferred() (ranxianglei#2690: a deferred
  transaction that upgrades mid-flight hits SQLITE_BUSY_SNAPSHOT, which
  busy_timeout cannot resolve); close() is idempotent.
- probe seam: a bag pinning only { builtin, host } now simulates a NODE
  host (bun off unless the test opts in with bun: true), so the ladder
  suites keep passing on a machine whose runtime happens to be bun.
- engine-bootstrap.ts: bun never downloads a prebuild (not-needed);
  nodeSqliteUpgradeHint and the degrade banner name bun:sqlite instead
  of the Node lever; the self-contained integrity worker picks its
  engine by the runtime it runs on ({ readonly: true } on bun).
- tests: new tests/storage-bun-sqlite.test.ts (9 tests: fake-Database
  contract pins that also run on Node CI, plus real-bun lock/rollback
  and a real UnifiedStore round-trip through open, ingest, reads, close
  and reopen). Seven raw test handles stop pinning node:sqlite so those
  suites run on any runtime; the three node:sqlite-only legs in
  storage-unified skip by probe instead of failing to load.

Measured under bun against the pre-change baseline on this machine: 24
more storage tests pass, zero regressions (storage-unified 4/13 -> 14
pass + 4 skip + 0 fail; ingest-watermark 1/15 -> 15/1; bounded-load
1/4 -> 4/1; engine-proofs 1/11 -> 7/5; plugin-fork 35/21 -> 51/5;
engine-bootstrap unchanged at 43/1). Remaining bun failures are the
node:test shim's missing t.mock (oven-sh/bun#5090), Node-only suites
that statically import node:sqlite, and ERR_SERVER_NOT_RUNNING in the
child-process harness.
Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Nov 7, 2026
…xianglei#5744)

The refs table (one row per message the session has ever seen) was maintained by DELETE-all plus one-row INSERT on every save, so a steady-state delta save of a few new messages into a long session still executed O(full history) native SQLite statements synchronously on the event loop - the 96.5% StatementSync.run profile behind ranxianglei#5744. Now: an in-memory diff against one indexed read of the stored set. Vanished raw_ids are deleted (no ghost rows survive a save, so the review-ranxianglei#2690 replace-all contract still holds), untouched survivors are skipped, and added/moved rows go out as multi-row upsert batches with a bounded parameter count - first-ingest and compaction-renumber storms cost ~one batch per 160 rows instead of H native calls.

Bench (H=3100, K=4/turn): delta lane 3223 -> 82 stmts/save, 34.8 -> 4.6 ms; folded-resident lane 3855 -> 714; steady-state refs work is now constant in history length. Regression suite pins end-state parity, the refs statement budget, and a ghost-free reopen.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant