From ea0566cd283efcf0622c41f2f51d1e9555a446b6 Mon Sep 17 00:00:00 2001 From: ranxianglei Date: Sat, 21 Nov 2026 13:12:27 +0800 Subject: [PATCH] =?UTF-8?q?fix(storage):=20open=20=E4=B8=8D=E5=86=8D?= =?UTF-8?q?=E5=86=99=E5=BA=93=20=E2=80=94=20=E5=85=A8=E8=A1=A8=E6=B0=B4?= =?UTF-8?q?=E4=BD=8D=E7=BA=BF=E9=87=8D=E7=BD=AE=E6=94=B9=E9=80=90=E4=BC=9A?= =?UTF-8?q?=E8=AF=9D=E6=83=B0=E6=80=A7=20prime=20(#3285=20per-session=20re?= =?UTF-8?q?work,=20#7156=20=E4=BA=8B=E6=95=85=E9=9D=A2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit UPDATE sessions SET last_ingested_seq = NULL —— 它让 open 本身变成 一次写,且一次性触碰每个会话。opencode 对照研究(同 SQLite+WAL、 多客户端并发多年无损坏)确认了纪律边界:多进程 WAL 库的稳态是 「只排队写、绝不在 open/维护窗口制造协议外写入」;#7156 复现 (第二条连接 3 行 UPDATE 即损坏活库)证明 open 期写入正是高危 窗口。 信任边界其实是逐会话的:一个进程只需要不信任「它没设过的、它 即将保存的那个会话」的水位线。resetAllIngestWatermarks() 删除, 改为 UnifiedStore.primeIngestTrust(sessionId):本进程首次触碰某 会话时(读水位线前)只清该会话的外来水位线(下一次保存走精确 全量路径、用本进程自己的视图重建信任),同进程后续触碰幂等无 写;从未触碰的会话水位线原样保留。对 (进程,会话) 逐一点对, 信任判决与旧全表重置逐字节一致。 行为变化:open 零写 —— 大库 pre-verdict 损坏不再拦 boot(boot 成功,损坏在首次保存的大声失败 + 后台 quick_check 判决落定后由 状态面暴露);#7156 的 first-write 分类保留覆盖 open 路径裸写 (migration/DDL)。noteIngestFailure 新增损坏分类:ingest 写撞 proven corruption 时大声 error + 置完整性 latch(此前的唯一 暴露面是 open 期那一次写)。mysql 后端接口同步。 回归:tests/per-session-ingest-trust.test.ts(新增:open 不清 水位线 / prime 单会话 + 幂等 + 全量重设信任链)、 tests/issue7156-storage-corruption.test.ts FIX4 重写为新语义 (corrupt page 下 boot 成功 + 首次 ingest 大声失败 + latch)、 tests/issue7156-pure-sqlite-refusal.test.ts 大库异步用例改钉新 时序(pre-verdict 损坏 boot 成功 + 首次 ingest 大声失败 + latch; quick_check 同步判决的拒启路径与真值表不变)、 tests/storage-ingest-watermark.test.ts handoff 用例改钉「open 不清 + 首触 prime + 全量路径」(安全性质不变:新进程首个保存 仍全量、零 drift)、tests/storage-mysql-seam.test.ts 接口同步。 --- CHANGELOG.md | 3 +- src/storage/backend.ts | 8 +- src/storage/mysql.ts | 10 +- src/storage/shadow.ts | 36 ++++-- src/storage/store.ts | 38 +++--- tests/issue7156-pure-sqlite-refusal.test.ts | 38 +++--- tests/issue7156-storage-corruption.test.ts | 28 +++-- tests/per-session-ingest-trust.test.ts | 122 ++++++++++++++++++++ tests/storage-ingest-watermark.test.ts | 12 +- tests/storage-mysql-seam.test.ts | 2 +- 10 files changed, 245 insertions(+), 52 deletions(-) create mode 100644 tests/per-session-ingest-trust.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 978e98698a..0855e5faec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,8 +16,9 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag - **系统架构文档体系落地(owner 指令 2026-11-08):`docs/architecture/` 成为实时维护的架构枢纽**:新增 `docs/architecture.md` / `docs/architecture.zh-CN.md` 总枢纽(整体架构图中英双版转正 + 8 模块总表),首个样板模块页 `docs/architecture/storage.{md,zh-CN.md}`(统一存储:职责/原理/工作机制/硬不变量,配模块图中英双版)。文件级地图 `reference/architecture.md` 补齐 125 个缺失条目(新存储层 / marker / fastpath / lanes 等全部新模块)并升级为 CI 强制全量覆盖。新增 CI 车道 `architecture-docs`(`tools/check-architecture-docs.mjs`):地图覆盖全 `src/` 文件、总表↔页面成对、图 svg 优先 + png 备选齐备、SVG 基础良构(未转义 `&` 曾让英文图线上破图)。AGENTS.md「Documentation Layout」新增实时维护规则:改结构 ⇒ 同 PR 更新页 + 图 + 总表。其余 7 个模块页(pipeline / lanes / kernel / identity / fastpaths / webui / update)同一 PR 全部补齐——总表 8 行全 ✅(每页 = 中英双版文档 + 中英双版图)。 ### Fixed -- **纯 SQLite 姿态下证实损坏不再静默降级 JSON,改为响亮拒启(owner 策略 2026-11-21,#7156 生产事故)**:quick_check 证实 index.db 损坏(含「首个 post-open 写入」撞损坏的大库/缓存判决分支,#7156 FIX4)且进程有效姿态是**纯 SQLite**(有效 mode full + sqlite 后端 + 无加密 + jsonWriter off = 2026-11-13 缺省翻转后的缺省,或显式 off)时,旧契约(响亮告警 + 不挂索引照常启动、JSON 信封服务)在这里会服务一个近乎为空的非持久视图,其后每一轮都只落在非持久副本里——静默丢持久历史,比损坏本身更糟(实测现场:0.1.216 的 3.2GB index.db 损坏后自动「降级 JSON」,面板显示 requests:2/上下文 0,而真实 1563 请求/124k 上下文/$12.41 成本全在索引里)。现在该姿态 REFUSES TO START:文件原样不动(不搬不拷不重建),日志与诊断面只给可操作的手工修复(备份 index.db/-wal/-shm → 手工移开 → 再启动,从双写期信封重建);`BILI_STORAGE_ALLOW_LEGACY_DEGRADE=1` 可有意识地恢复旧降级路径(永不缺省、不被任何配置值隐含)。其余姿态(显式双写 full、shadow、legacy、网络后端、加密)信封仍是真恢复来源,旧契约逐字节保持。姿态判定是纯函数(与 persist 的 jsonWriterOff 同一公式,两开关永不分歧);拒启被记录进诊断面(`/__bili/status` 与 diagnostics 可见「REFUSING TO START (pure-sqlite stance)」前缀)且每次启动都重新生效(latch 只短路进程内重扫)。回归:tests/issue7156-pure-sqlite-refusal.test.ts(姿态真值表 + 两种纯 sqlite 拼法与 file 层的损坏拒启/逃生阀/重复拒启 + 字节不变断言 + 双写姿态保留降级契约)、tests/issue4816-integrity-throw-quarantine.test.ts、tests/acp-status-storage-degraded.test.ts、tests/web-storage-banner.test.ts、tests/storage-durability.test.ts 与 storage 全量套件绿。 +- **纯 SQLite 姿态下证实损坏不再静默降级 JSON,改为响亮拒启(owner 策略 2026-11-21,#7156 生产事故)**:quick_check 证实 index.db 损坏(含「首个 post-open 写入」撞损坏的大库/缓存判决分支,#7156 FIX4)且进程有效姿态是**纯 SQLite**(有效 mode full + sqlite 后端 + 无加密 + jsonWriter off = 2026-11-13 缺省翻转后的缺省,或显式 off)时,旧契约(响亮告警 + 不挂索引照常启动、JSON 信封服务)在这里会服务一个近乎为空的非持久视图,其后每一轮都只落在非持久副本里——静默丢持久历史,比损坏本身更糟(实测现场:0.1.216 的 3.2GB index.db 损坏后自动「降级 JSON」,面板显示 requests:2/上下文 0,而真实 1563 请求/124k 上下文/$12.41 成本全在索引里)。现在该姿态 REFUSES TO START(quick_check 同步判决的损坏路径不变;大库异步判决路径见下条 open 零写:pre-verdict 损坏 boot 成功,损坏在首次 ingest 写大声失败+完整性 latch 落定,该姿态仍永不写 JSON、无静默降级面):文件原样不动(不搬不拷不重建),日志与诊断面只给可操作的手工修复(备份 index.db/-wal/-shm → 手工移开 → 再启动,从双写期信封重建);`BILI_STORAGE_ALLOW_LEGACY_DEGRADE=1` 可有意识地恢复旧降级路径(永不缺省、不被任何配置值隐含)。其余姿态(显式双写 full、shadow、legacy、网络后端、加密)信封仍是真恢复来源,旧契约逐字节保持。姿态判定是纯函数(与 persist 的 jsonWriterOff 同一公式,两开关永不分歧);拒启被记录进诊断面(`/__bili/status` 与 diagnostics 可见「REFUSING TO START (pure-sqlite stance)」前缀)且每次启动都重新生效(latch 只短路进程内重扫)。回归:tests/issue7156-pure-sqlite-refusal.test.ts(姿态真值表 + 两种纯 sqlite 拼法与 file 层的损坏拒启/逃生阀/重复拒启 + 字节不变断言 + 双写姿态保留降级契约)、tests/issue4816-integrity-throw-quarantine.test.ts、tests/acp-status-storage-degraded.test.ts、tests/web-storage-banner.test.ts、tests/storage-durability.test.ts 与 storage 全量套件绿。 - **信封导入不再顶掉 index 已有的压缩块谱系与生命周期统计(#7156 事故第二现场,2026-11-21)**:#7156 的生产现场里,降级窗口(不挂索引、纯信封服务)内的一次会话保存把信封里的 32 个粗块结构写进了 `importRecord`——它把信封当成权威谱系:按 incoming 集合先删后插,index 里 103 个更细的压缩块(及其 summary/content 引用链)被整批删除,生命周期统计也从 requests:1563/124k 上下文/$12.41 成本回退成 requests:2/0。之后每一轮 preflight 都在 32 粗块的地基上折叠,上下文装不进模型窗口、会话不可用。现在 `ingestSession` 新增 `envelopeImport` 标记(仅 `UnifiedShadow.importRecord`/`importRecordAsync` 两个导入入口传):存储侧已有块谱系时**只采纳信封里 seq 更高的新消息**,块谱系与 stale 清理一律跳过(index 无块的真正首开导入不变,仍整体采纳信封结构);stats_json 同带单调守卫——envelopeImport 下已存 requests ≥ 传入 requests 时保留已存值(仅导入路径生效,organic 保存仍是权威且零额外开销)。回归:tests/envelope-import-block-lineage.test.ts(细块保留 + stats 只升不降 + 首开导入采纳 + organic 权威语义不变 + importRecord wiring 五用例)、tests/storage-ingest-watermark.test.ts。 +- **存储索引 open 不再写库:全表水位线重置改为逐会话惰性 prime(#3285 逐会话化,2026-11-21,#7156 事故面的最后一块)**:#3285 的跨进程信任交接原先在 writer init(UnifiedShadow.maybeOpen)里执行一条 `UPDATE sessions SET last_ingested_seq = NULL`(无 WHERE)——它让 **open 本身变成一次写**,且一次性触碰**每一个**会话。opencode 对照研究(同 SQLite+WAL、多客户端并发多年无损坏)确认了纪律边界:多进程 WAL 库的安全前提是「稳态只排队写、绝不在 open/维护窗口制造协议外写入」,而本机事故(#7156 复现:第二条连接 3 行 UPDATE 即损坏活库)证明 open 期写入正是高危窗口(旧构建双写者+434MiB WAL 风暴叠加时库被写坏,0.1.216 起每次启动还多一个「first post-open write failed」降级面)。信任边界本身其实是**逐会话**的:一个进程只需要不信任「它没设过的、它即将保存的那个会话」的水位线。现在 `resetAllIngestWatermarks()` 删除,改为 `UnifiedStore.primeIngestTrust(sessionId)`:本进程首次触碰某会话时(读水位线之前)只清该会话的外来水位线(→ 下一次保存走精确全量路径、用本进程自己的视图重建信任),同进程后续触碰幂等无写;从未触碰的会话水位线原样保留。对 (进程,会话) 逐一点对,信任判决与旧全表重置**逐字节一致**;行为变化:open 零写(大库 pre-verdict 损坏不再拦 boot——boot 成功、损坏在首次保存的大声失败+后台 quick_check 判决落定后由状态面暴露);#7156 的「first post-open write」分类保留覆盖 open 路径裸写(migration/DDL)。mysql 后端接口同步。回归:tests/per-session-ingest-trust.test.ts(新增:open 不清水位线/prime 单会话+幂等+全量重设信任链)、tests/issue7156-storage-corruption.test.ts FIX4 重写为新语义(corrupt page 下 boot 成功+首次 ingest 大声失败+latch)、tests/storage-mysql-seam.test.ts 接口同步。 - **dsh 运行时注册的 preset 继承压缩完全禁止(#7190,关 #7178 / #7062 / #7169)**:dsh 的 preset 注册表 `AgentPresetRegistry.register(definition)` 是**启动后才出生**的 preset 行唯一的出生点,它把嵌套行直接挂进内存 PresetTree(其 `commit()` 为空实现),从不经过 loader 的 `internal/config` 变换——bili 的 #4147 结构性 disable 重写(`bili-compaction-gate`,#3354/#3662)永远碰不到这些行。于是运行时注册 preset 的压缩引擎是全副武装的(每步压力触发 + 模型切换前置压缩 + 溢出恢复三条触发器都在 dsh 的 surface 计量上点火),而 bili 侧 wire guard 拒绝每一次压缩调用:#7062 的逐步 422 风暴、#7178 的模型切换死局(`MODEL_SWITCH_COMPACTION_FAILED` 永不满足前提)、#7169 的 landed-checkpoint 触发 rebase 抹掉 bili 自己的折叠块再 DIGEST DRIFT 冻结信封,全是这台引擎在开火。修法与 boot-time 行同一判决、同一机制:gate 入口(always-on,bili-native 关着也活着)现在包一层 `agentPresets.register`——定义里的 compaction-basic / command-compact 行在挂入前就地改写为 `disabled: true`。判决按每次注册调用检查(bili-native 没宣告持线 = 定义原样通过,dsh 原生压缩照常);无 compaction 行 / 畸形定义按身份原样通过(注册表自己的校验看到字节级相同的输入);Symbol-for 标记保证重 apply 不双重包裹;宿主没有该注册表 = 无运行时 preset 可注册,静默不包裹。遥测新增 `runtime` / `runtimeWrapped` 维度,#6770 覆盖一行据此报告「register() 已覆盖——尚无运行时注册」或「N 个运行时注册被改写」,wrap 未知时不报假缺口。**非模型可见文本改动**(只动 dsh 侧 web banner 的覆盖一行与日志)。dsh 源码依据(deepseek-ai/deepseek-harness @ 2e39e70,与现场 0.2.2-rc.3 逐点核验一致):`packages/preset/agent-preset-registry/src/index.ts:96`(register 立即激活、重复 id 抛错)、`mount.ts:22-23`(PresetTree 空 commit)、per-fiber `internal/config` waterfall(`vendor/cordis/src/fiber.ts:691-700`)——preset 作用域 fiber 在自己的上下文里解析配置,根上注册的 global hook 不触发。回归:tests/issue7190-dsh-runtime-preset-coverage.test.ts(新增 10 条行驱动正反用例:改写/身份保持/判决关/重包裹/`this` 转发/无注册表降级/get 回退/遥测去重 + 覆盖文案四形态)、tests/issue3354-dsh-compaction-disable.test.ts、tests/issue6770-compaction-coverage-text.test.ts、tests/issue4147-dsh-overlay-structural.test.ts 全绿。 - **⚠️ MODEL-FACING TEXT CHANGE** 压缩成功回执去掉 head/tail 标签(保留摘录)+ 打上时间戳 + kernel 锚点引文标注来源(owner 指令 2026-11-18,会话 01a1e89d 诊断续):成功回执是唯一一类**引用用户话语却在线路上不带任何 `` 时间戳**的系统行(user 消息都带 `time=`,tool 结果全是裸文本)——弱模型扫尾部找「用户刚说了什么」时只能看到回执里嵌的引文,会把它当成最新用户指令(ti_an 现场:qwen3.8-27b 编出「user's latest message (m02386/m02387) 要求一段 PR 总结」,ref 与内容全是幻觉)。四处文本改动:① 指纹行**去掉 head/tail 标签、保留摘录**(#2544 误读的源头是标签本身——`head "…" … tail "…"` 读起来像「只存了开头和结尾」,触发对刚建块的防御性 decompress;摘录的 #1294 自查价值与 #1615 surrogate 切口守卫原样保留)——`· b1 full summary 10600ch [2026-11-17 10:35:14 – …] · head "…" … tail "…"` → `· b1 full summary 10600ch [2026-11-17 10:35:14 – …] "… … …"`。② 成功回执头追加折叠时刻——`[Compressed m00001–m00192 → 1 block(s), ~51613 tokens saved.]` → 同行尾部 ` folded at 2026-11-18 10:05:12`(提交时墙钟;回执是全新尾消息,不进前缀缓存稳定区,无缓存后果)。③ #2544 指引句去掉与摘录绑定的前半句——`Fingerprints show head/tail only — every summary is stored in full. …decompress only if the excerpt doesn't answer it.` → `Every summary is stored in full — to retrieve a specific detail from a new block, call search_context({ query }) first; decompress only if that doesn't answer it.`(search_context 优先的检索路由保留)。④(kernel 0.0.198 → 0.0.199)保护带排除警告里的 Last-user anchor 引文标注来源——`Last-user anchor: m00003 "…"` → `Last-user anchor (historical quote, NOT a new user message): m00003 "…"`。回执行号结构不变(folded at 与原行内的标签移除都不改变行数);log 侧(#1718)本来就是 length-only,仅注释同步。回归:tests/compress-fingerprint-refold.test.ts(断言改无标签形状,#1615 四个 surrogate 切口/清洗用例保留并钉新格式)、tests/issue4313-fingerprint.test.ts、tests/issue4313-message-timestamp-toggle.test.ts、tests/text-safe.test.ts、tests/issue5173-last-user-anchor.test.ts、tests/issue6021-image-offload-anchor.test.ts、kernel/tests/issue5109-anchor-skip-injections.test.ts。 - **⚠️ MODEL-FACING TEXT CHANGE** 压缩失败回执里的 ref 快照行标记为「时点快照」:会话 01a1e89d 现场(2026-11-17,qwen3.8-27b)——旧文案 `[Current context: N visible message(s), refs m00001–mXXXX, …]` 回答的是「回执发出那一刻」的 ref 前沿,但读起来像当前状态;失败回执会持久进模型可见历史,长会话累积多份 max 不同的旧副本后,弱模型把它和自己的历史 compress header 对不上(「summaries claimed m02129–m02325, but the highest ref is m02255 — something's off」),把时间差误读成数据损坏,随后 23:23–23:29 连发 7 次指向不存在终点(m02383/m02489/m02384,全部高于真实前沿)的压缩调用(内核原子守卫每次正确拦截,零脏状态)。现在该行为 `[Context snapshot at the moment of THIS receipt — point-in-time only: refs are minted continuously, so every earlier copy of this line in the transcript is stale, and the newest copy is the current frontier. …]`:每份副本自证是快照、旧副本明示过期、最新一份才是前沿。数字(条数/lo/hi/块数)与来源零变化,不引入新锚点;仅压缩失败回执(loop-guard 武装时原本就不带此行)受影响。回归:tests/issue1029-acp-marker-stale-refs.test.ts ③ 钉新精确文案。 diff --git a/src/storage/backend.ts b/src/storage/backend.ts index 794e20c5ad..3bdf174f86 100644 --- a/src/storage/backend.ts +++ b/src/storage/backend.ts @@ -96,7 +96,9 @@ export interface StorageBackend { ingestWatermark(sessionId: string): Promise<{ seq: number; tailSeqs: number[] } | null>; tailSeqs(sessionId: string): Promise; clearIngestWatermark(sessionId: string): Promise; - resetAllIngestWatermarks(): Promise; + /** #3285 (2026-11-21 per-session rework): first-touch trust prime — drop + * any FOREIGN-set watermark for one session; idempotent per process. */ + primeIngestTrust(sessionId: string): Promise; probeIngestBoundary(sessionId: string, seq: number): Promise<{ msgId: string; contentHash: string; maxSeq: number } | null>; // --- reap journal (#3841) ---------------------------------------------- @@ -241,8 +243,8 @@ export class SqliteBackend implements StorageBackend { clearIngestWatermark(sessionId: string): Promise { return Promise.resolve(this.store.clearIngestWatermark(sessionId)); } - resetAllIngestWatermarks(): Promise { - return Promise.resolve(this.store.resetAllIngestWatermarks()); + primeIngestTrust(sessionId: string): Promise { + return Promise.resolve(this.store.primeIngestTrust(sessionId)); } probeIngestBoundary(sessionId: string, seq: number): Promise<{ msgId: string; contentHash: string; maxSeq: number } | null> { return Promise.resolve(this.store.probeIngestBoundary(sessionId, seq)); diff --git a/src/storage/mysql.ts b/src/storage/mysql.ts index 0fcb26775a..59e091c7d6 100644 --- a/src/storage/mysql.ts +++ b/src/storage/mysql.ts @@ -1262,8 +1262,14 @@ export class MysqlBackend implements StorageBackend { await this.pool.query("UPDATE sessions SET last_ingested_seq = NULL WHERE session_id = ?", [sessionId]); } - async resetAllIngestWatermarks(): Promise { - await this.pool.query("UPDATE sessions SET last_ingested_seq = NULL"); + // #3285 (2026-11-21 per-session rework): the open-time GLOBAL watermark + // reset is gone (it made open a write and touched every session); trust + // is primed per session on first touch instead. + private readonly ingestTouchedThisProcess = new Set(); + async primeIngestTrust(sessionId: string): Promise { + if (this.ingestTouchedThisProcess.has(sessionId)) return; + this.ingestTouchedThisProcess.add(sessionId); + await this.pool.query("UPDATE sessions SET last_ingested_seq = NULL WHERE session_id = ?", [sessionId]); } async reapBlocks( diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 8eaade1bbc..7c34907014 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -354,6 +354,18 @@ export class UnifiedShadow { this.lastWarnKey = key; loggerLog("warn", `[storage] unified sidecar ${this.dbPath ?? "backend"} failed to ingest (${this.consecutiveFailures} in a row; ${detail})${sqliteOpenFailureHint(String(err))} — the index write will be RETRIED on every subsequent save until it succeeds again (no self-disable; the JSON envelope keeps serving meanwhile)`); } + // #7156 (2026-11-21): open no longer writes — with the per-session + // watermark prime, the FIRST write of a process happens at its first + // ingest. If that write hits proven corruption before the integrity + // verdict has landed (large store: async quick_check in flight, or a + // cached .chk verdict skipped the rescan), say so LOUDLY and trip the + // latch so the status surface shows the proven-corrupt state; the + // save itself falls back to the envelope as usual (the warn above). + const corruption = sqliteCorruptionDetail(err); + if (corruption !== null) { + integrityQuickCheckFailed = true; + loggerLog("error", `[storage] index.db FAILED at an ingest write (${corruption}) — the integrity check has not run/landed yet (large store / cached verdict), so the corruption surfaced here instead of at quick_check. No file moved, no copy, no automatic rebuild (2026-10-30 owner policy): back up ${this.dbPath ?? "index.db"}{,-wal,-shm}, then MOVE those files aside by hand and start again (#7156)`); + } } /** #3931 step 3: true when this shadow rides a network backend (mysql). @@ -440,11 +452,11 @@ export class UnifiedShadow { ); return null; } - // #3285: watermark trust does not cross process boundaries — a - // restart-handoff overlap leaves the previous writer's rows AND - // watermark behind; drop them all so this process's first save - // per session re-establishes trust from its own view. - store.resetAllIngestWatermarks(); + // #3285 (2026-11-21 per-session rework): watermark trust no longer + // gets a global open-time reset — it would make OPEN a write and + // touch every session (the #7156 first-write hazard). Each + // session's foreign watermark is dropped lazily on this process's + // first touch of it, right before the watermark is read (below). } catch (err) { // The inner gate's refusal throw (or any StorageIntegrityError the // inner catch let through) re-enters THIS catch — it is already @@ -458,7 +470,7 @@ export class UnifiedShadow { // 512 MiB sync threshold checks in the background, or a poisoned // .chk verdict cache skipped the rescan (see the close-time gate // in store.ts) — the corruption surfaces at the FIRST post-open - // write (resetAllIngestWatermarks) instead of at quick_check. + // write (an open-path write: migration/DDL) instead of at quick_check. // Classify it exactly like the inner catch: proven-corrupt → // start WITHOUT the index, files left in place, loud alarm naming // the manual fix (2026-10-30 owner policy). @@ -479,8 +491,10 @@ export class UnifiedShadow { applyProvenCorruptionPolicy(dir, env, fileUnified, fileJsonWriter, fileBackend, `first post-open write failed: ${corruptionDetail}`, err); loggerLog( "error", - `[storage] index.db FAILED at the first post-open write (${corruptionDetail}) — the integrity check did not run synchronously (large store / cached verdict), so the corruption surfaced here instead of at quick_check. Starting WITHOUT the index (no file moved, no copy, no automatic rebuild; 2026-10-30 owner policy): back up ${path.join(dir, "index.db")}{,-wal,-shm}, then MOVE those files aside by hand and start again — the fresh index re-imports the JSON envelopes (#7156/#4175/#4626)`, + `[storage] index.db FAILED at an open-path write (${corruptionDetail}) — the integrity check did not run synchronously (large store / cached verdict), so the corruption surfaced here instead of at quick_check. Starting WITHOUT the index (no file moved, no copy, no automatic rebuild; 2026-10-30 owner policy): back up ${path.join(dir, "index.db")}{,-wal,-shm}, then MOVE those files aside by hand and start again — the fresh index re-imports the JSON envelopes (#7156/#4175/#4626)`, ); + recordSidecarOpenFailure(`open-path write failed: ${corruptionDetail}`); + integrityQuickCheckFailed = true; // proven-corrupt: operator hand-fix only — no in-process auto-retry return null; } // Deleting the index cannot fix a no-engine failure (the engine is @@ -805,6 +819,11 @@ export class UnifiedShadow { private ingestDelta(legacy: LegacySessionLike, hasRawTail: boolean): boolean { if (this.sync === null) return false; try { + // #3285 (2026-11-21 per-session): prime THIS session's trust + // before reading the watermark — a foreign-set watermark + // (previous process) must not vouch for its rows; own watermarks + // stay trusted (idempotent, no second write). + this.sync.primeIngestTrust(legacy.id); const mark = this.sync.ingestWatermark(legacy.id); if (!mark) return false; // The pre-transaction probe is an optimization (skip building a @@ -1360,6 +1379,9 @@ export class UnifiedShadow { // generation so the next sample can re-admit and release it. if (!this.driftedSessions.has(legacy.id)) { try { + // #3285 (2026-11-21 per-session): prime THIS session's + // trust before reading the watermark (idempotent). + await backend.primeIngestTrust(legacy.id); const mark = await backend.ingestWatermark(legacy.id); if (mark) { const boundary = await backend.probeIngestBoundary(legacy.id, mark.seq); diff --git a/src/storage/store.ts b/src/storage/store.ts index 0dcf1f4835..ddb452e7e3 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -1915,21 +1915,29 @@ export class UnifiedStore { this.db.prepare("UPDATE sessions SET last_ingested_seq = NULL WHERE session_id = ?").run(sessionId); } - /** #3285: drop EVERY session's ingest watermark at once. A watermark - * vouches "the writer that set it verified this prefix" but carries no - * writer identity, so the trust must not survive a process handoff: - * under the multi-process shared-directory contract (#2671) a - * restart-handoff overlap (old instance still draining while the new - * one boots, both sharing index.db) leaves a foreign process's rows - * AND its watermark behind, and inheriting that trust lets the new - * process take unverified empty deltas against rows it never wrote. - * Called once at writer init (UnifiedShadow.maybeOpen); pure readers - * (export) must not call it. After the reset each session's first save - * takes the exact full path — an authoritative replace, see - * ingestSession — re-establishing trust from this process's own view. - * Cost: one full ingest per saved session per boot. */ - resetAllIngestWatermarks(): void { - this.db.prepare("UPDATE sessions SET last_ingested_seq = NULL").run(); + /** #3285 (2026-11-21 per-session rework): watermark trust does not cross + * process boundaries — a watermark vouches "the writer that set it + * verified this prefix" but carries no writer identity, so under the + * multi-process shared-directory contract (#2671) a restart-handoff + * overlap leaves a foreign process's rows AND its watermark behind, + * and inheriting that trust lets the new process take unverified empty + * deltas against rows it never wrote. + * The 2026-10-31 implementation dropped EVERY session's watermark in one + * global write at open — it touched every other session and made open + * itself a write (the #7156 first-write hazard). The trust boundary is + * actually per-session: a process only needs to distrust watermarks it + * did not set, for the session it is about to save. So the invalidation + * moved here — first touch of a session in THIS process drops any + * foreign-set watermark for that ONE session (→ the next save takes the + * exact full path, re-establishing trust from this process's own view); + * later touches trust this process's own watermarks. Per (process, + * session) the trust decision is byte-identical to the global reset; + * untouched sessions keep their watermarks, and open performs no write. */ + private readonly ingestTouchedThisProcess = new Set(); + primeIngestTrust(sessionId: string): void { + if (this.ingestTouchedThisProcess.has(sessionId)) return; + this.ingestTouchedThisProcess.add(sessionId); + this.db.prepare("UPDATE sessions SET last_ingested_seq = NULL WHERE session_id = ?").run(sessionId); } /** #3248 lane-ownership probe for the message sweep (#3285): every ext diff --git a/tests/issue7156-pure-sqlite-refusal.test.ts b/tests/issue7156-pure-sqlite-refusal.test.ts index d45a541980..b95c4c455d 100644 --- a/tests/issue7156-pure-sqlite-refusal.test.ts +++ b/tests/issue7156-pure-sqlite-refusal.test.ts @@ -231,17 +231,23 @@ test("#7156 refusal is RECORDED for the diagnostics surface and repeats on every } }); -test("#7156 pure-sqlite: a first-post-open-write corruption REFUSES TO START too", { skip: SKIP_NO_ENGINE }, () => { +test("#7156 pure-sqlite (2026-11-21 per-session rework): open no longer writes — a pre-verdict corruption boots, then surfaces loudly at the first ingest write", { skip: SKIP_NO_ENGINE }, () => { const dir = tmpDir(); const prevUnified = process.env.BILI_STORAGE_UNIFIED; process.env.BILI_STORAGE_UNIFIED = "sqlite"; try { // The incident's large-store path: the integrity check does not run // synchronously (async mode forced), so a proven corruption surfaces - // at the FIRST post-open write (resetAllIngestWatermarks) instead of - // at quick_check — the branch the truncateHalf tests above cannot - // reach. Same recipe as the #7156 FIX 4 dual-write test, but the - // pure-sqlite stance must REFUSE, not degrade. + // at the process's FIRST INGEST WRITE instead of at quick_check — + // the branch the truncateHalf tests above cannot reach. Before + // 2026-11-21 the open-time GLOBAL watermark reset + // (resetAllIngestWatermarks) made OPEN a write and this corruption + // refused the boot. Now open performs NO write at all: the store + // BOOTS even with the verdict not yet landed, and the corruption + // surfaces — loudly (integrity latch + error) — at the process's + // first ingest write. The pure-sqlite stance never writes JSON + // envelopes, so the failed save leaves the in-memory view serving; + // there is no durable JSON copy to silently degrade to. const p = join(dir, "index.db"); const s0 = UnifiedStore.open(p); s0.close(); @@ -256,15 +262,21 @@ test("#7156 pure-sqlite: a first-post-open-write corruption REFUSES TO START too const errors: string[] = []; setLogCapture((level, msg) => { if (level === "error") errors.push(String(msg)); }); try { - assert.throws( - () => UnifiedShadow.maybeOpen(dir, { BILI_STORAGE_UNIFIED: "sqlite" }, () => null), - /malformed|not a database/i, - "a first-write corruption in the pure-sqlite stance refuses to start", - ); - assert.equal(hadIntegrityQuickCheckFailure(), true, "the integrity latch tripped"); + const legacy = { + id: "s-corrupt-" + "x".repeat(10), + createdAt: 1, lastSeen: 2, meta: {}, stats: {}, metadata: {}, + state: { blocks: [], messageRefs: { byRaw: {}, byRef: {} }, nextBlockId: 1, nextRunId: 1 }, + blockContents: new Map(), + lastMessages: [{ role: "user", content: "hello" }], + }; + const sh = UnifiedShadow.maybeOpen(dir, { BILI_STORAGE_UNIFIED: "sqlite" }, () => legacy as never); + assert.ok(sh, "open performs NO write anymore — a pre-verdict corrupt store BOOTS (2026-11-21 per-session watermark fix)"); + assert.equal(hadIntegrityQuickCheckFailure(), false, "no integrity latch before the first write"); + sh.record({}); + assert.equal(hadIntegrityQuickCheckFailure(), true, "the integrity latch tripped at the first ingest write"); assert.ok( - errors.some((m) => m.includes("first post-open write")), - `the error names the first-write corruption; got ${JSON.stringify(errors)}`, + errors.some((m) => m.includes("ingest write")), + `the error names the first-ingest-write corruption; got ${JSON.stringify(errors)}`, ); } finally { setLogCapture(null); diff --git a/tests/issue7156-storage-corruption.test.ts b/tests/issue7156-storage-corruption.test.ts index fe2635221f..3eb98f5f21 100644 --- a/tests/issue7156-storage-corruption.test.ts +++ b/tests/issue7156-storage-corruption.test.ts @@ -179,7 +179,7 @@ test("#7156 FIX 2: unifiedWebSummaries logs a warn when webSummaryRows throws (f } }); -test("#7156 FIX 4: a first-post-open-write corruption starts WITHOUT the index (not REFUSING TO START)", { skip: SKIP }, () => { +test("#7156 FIX 4 (2026-11-21 per-session rework): open no longer writes — a pre-verdict corruption boots, then surfaces loudly at the first ingest write", { skip: SKIP }, () => { const dir = tmpDir(); const prevUnified = process.env.BILI_STORAGE_UNIFIED; process.env.BILI_STORAGE_UNIFIED = "full"; @@ -196,19 +196,31 @@ test("#7156 FIX 4: a first-post-open-write corruption starts WITHOUT the index ( } ro.exec("PRAGMA wal_checkpoint(TRUNCATE)"); ro.close(); - // Corrupt page 2: open() survives (async mode — the sync quick_check never - // runs), but the first write (resetAllIngestWatermarks) throws CORRUPT. + // Corrupt page 2. Before 2026-11-21 the open-time GLOBAL watermark + // reset made OPEN a write and this corruption refused the boot. Now + // open performs no write at all: the store BOOTS even with the + // verdict not yet landed, and the corruption surfaces — loudly — at + // the process's first ingest write (the per-session watermark prime). zeroPage(p, 2); setIntegritySyncThresholdForTests(1); // force async mode for any size const errors: string[] = []; setLogCapture((level, msg) => { if (level === "error") errors.push(String(msg)); }); try { - const sh = UnifiedShadow.maybeOpen(dir, { BILI_STORAGE_UNIFIED: "full" }, () => null); - assert.equal(sh, null, "a first-write corruption yields 'no sidecar' (start without index) — never a REFUSING-TO-START throw"); - assert.equal(hadIntegrityQuickCheckFailure(), true, "the integrity latch tripped"); + const legacy = { + id: "s-corrupt-" + "x".repeat(10), + createdAt: 1, lastSeen: 2, meta: {}, stats: {}, metadata: {}, + state: { blocks: [], messageRefs: { byRaw: {}, byRef: {} }, nextBlockId: 1, nextRunId: 1 }, + blockContents: new Map(), + lastMessages: [{ role: "user", content: "hello" }], + }; + const sh = UnifiedShadow.maybeOpen(dir, { BILI_STORAGE_UNIFIED: "full" }, () => legacy as never); + assert.ok(sh, "open performs NO write anymore — a pre-verdict corrupt store BOOTS (2026-11-21 per-session watermark fix)"); + assert.equal(hadIntegrityQuickCheckFailure(), false, "no integrity latch before the first write"); + sh.record({}); + assert.equal(hadIntegrityQuickCheckFailure(), true, "the integrity latch tripped at the first ingest write"); assert.ok( - errors.some((m) => m.includes("first post-open write")), - `the error names the first-write corruption; got ${JSON.stringify(errors)}`, + errors.some((m) => m.includes("ingest write")), + `the error names the first-ingest-write corruption; got ${JSON.stringify(errors)}`, ); } finally { setLogCapture(null); diff --git a/tests/per-session-ingest-trust.test.ts b/tests/per-session-ingest-trust.test.ts new file mode 100644 index 0000000000..e0f521b561 --- /dev/null +++ b/tests/per-session-ingest-trust.test.ts @@ -0,0 +1,122 @@ +/** + * #3285 (2026-11-21 per-session rework): watermark trust no longer crosses + * process boundaries via a GLOBAL open-time reset. + * + * The pre-rework implementation dropped EVERY session's ingest watermark in + * one `UPDATE sessions SET last_ingested_seq = NULL` at writer init + * (UnifiedShadow.maybeOpen). That made OPEN a write and touched every other + * session — the exact first-write hazard #7156 demonstrated (a second live + * writer corrupting the db during that window). The trust boundary is in fact + * per-session: a process only needs to distrust watermarks it did not set, + * for the session it is about to save. So the invalidation moved to + * UnifiedStore.primeIngestTrust(sessionId) — first touch of a session in THIS + * process drops any foreign-set watermark for that ONE session (the next save + * then takes the exact full path, re-establishing trust from this process's + * own view); later touches trust this process's own watermarks. Per + * (process, session) the trust decision is byte-identical to the global + * reset; untouched sessions keep their watermarks, and open performs no write. + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import type { CoreMessage } from "acp-kernel"; + +process.env.BILI_PERSIST_ZSTD = "0"; +// Same hermeticity pin as the other storage suites. +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-psit-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +function tmpdir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function rmBestEffort(p: string): void { + try { + fs.rmSync(p, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + /* leaked temp dir is harmless */ + } +} + +function msg(i: number): CoreMessage { + return { id: `h_msg${i}`, role: i % 2 === 0 ? "user" : "assistant", contentType: "text", text: `hello ${i}` } as CoreMessage; +} + +function mkLegacy(id: string, lastMessages: CoreMessage[]): LegacySessionLike { + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "anthropic" }, + stats: { requests: 1 }, + metadata: {}, + state: { + blocks: [], + messageRefs: { byRaw: {}, byRef: {} }, + nextBlockId: 1, + nextRunId: 1, + }, + blockContents: new Map(), + lastMessages, + lastMessagesFolded: false, + }; +} + +function countMessages(dbPath: string, sessionId: string): number { + const ro = new DatabaseSync(dbPath, { readOnly: true }); + try { + return (ro.prepare("SELECT COUNT(*) AS n FROM messages WHERE session_id = ?").get(sessionId) as { n: number }).n; + } finally { + ro.close(); + } +} + +test("#3285 per-session trust: open no longer clears watermarks; prime is per-session, idempotent, and full-ingest re-establishes trust", () => { + const dir = tmpdir("bili-psit-"); + const p = path.join(dir, "index.db"); + try { + // --- "process A": save two sessions; each gets its own watermark. + const a = UnifiedStore.open(p); + a.ingestSession(planIngest(mkLegacy("sessX", [msg(1), msg(2)]))); + a.ingestSession(planIngest(mkLegacy("sessY", [msg(3), msg(4)]))); + assert.ok(a.ingestWatermark("sessX"), "process A's save of sessX set its watermark"); + assert.ok(a.ingestWatermark("sessY"), "process A's save of sessY set its watermark"); + a.close(); + + // --- "process B" opens the SAME db: open performs NO write and must + // NOT clear either watermark (the old global reset wiped both here). + const b = UnifiedStore.open(p); + assert.ok(b.ingestWatermark("sessX"), "process B's OPEN no longer clears sessX's foreign watermark"); + assert.ok(b.ingestWatermark("sessY"), "process B's OPEN no longer clears sessY's foreign watermark"); + + // Prime ONLY sessX: its foreign watermark is dropped; sessY — never + // touched by this process — keeps its watermark. + b.primeIngestTrust("sessX"); + assert.equal(b.ingestWatermark("sessX"), null, "primeIngestTrust dropped the foreign watermark for the ONE primed session"); + assert.ok(b.ingestWatermark("sessY"), "untouched sessions keep their watermarks"); + + // Idempotent per (process, session): the second touch trusts this + // process's own watermarks and writes nothing. + const wmAfterFirstPrime = b.ingestWatermark("sessX"); + b.primeIngestTrust("sessX"); + assert.equal(b.ingestWatermark("sessX"), wmAfterFirstPrime, "second prime in the same process is a no-op"); + + // The distrust gate did its job: the next save of sessX takes the + // exact FULL path (watermark null → authoritative replace), not a + // delta against rows this process never verified — and re-establishes + // trust from this process's own view. + b.ingestSession(planIngest(mkLegacy("sessX", [msg(1), msg(2), msg(5)]))); + assert.ok(b.ingestWatermark("sessX"), "the full re-ingest re-established this process's own watermark"); + assert.equal(countMessages(p, "sessX"), 3, "full path replaced sessX's rows (3 rows, not 2 stale + 1 appended)"); + b.close(); + } finally { + rmBestEffort(dir); + } +}); diff --git a/tests/storage-ingest-watermark.test.ts b/tests/storage-ingest-watermark.test.ts index 6ab6229b19..7b3cec9d1c 100644 --- a/tests/storage-ingest-watermark.test.ts +++ b/tests/storage-ingest-watermark.test.ts @@ -668,7 +668,7 @@ test("watermark: v4 database migrates to v11 additively; newer user_version no-o } }); -test("watermark: restart handoff — the previous writer's rows and watermark do not survive (#3285)", () => { +test("watermark: restart handoff — a foreign writer's trust is never inherited (#3285, per-session prime)", () => { const dir = tmpdir("bili-wm-handoff-"); try { // Live saves carry the raw tail (persist.ts guardedBuild) — that is @@ -694,7 +694,15 @@ test("watermark: restart handoff — the previous writer's rows and watermark do assert.ok(b, "second process must open"); const storeB = UnifiedStore.open(path.join(dir, "index.db")); try { - assert.equal(storeB.ingestWatermark("handoff"), null, "a foreign writer's watermark must not be inherited"); + // 2026-11-21 per-session rework: open no longer clears watermarks + // (the global reset made OPEN a write — the #7156 first-write + // hazard). The foreign watermark is still on disk right after + // open; the new process's FIRST touch of the session primes its + // trust (drops exactly that session's foreign watermark) before + // reading it — so the first save still takes the full path, which + // the assertions below pin (deltaIngests stays 0, A's rows are + // replaced, no drift). + assert.ok(storeB.ingestWatermark("handoff"), "the foreign watermark is still on disk after open (open performs no write)"); const view = [msg(0), msg(1), msg(2)]; b.record(...withRaw("handoff", view)); const s = stats(b); diff --git a/tests/storage-mysql-seam.test.ts b/tests/storage-mysql-seam.test.ts index ef7104b77e..48aa8e8a1c 100644 --- a/tests/storage-mysql-seam.test.ts +++ b/tests/storage-mysql-seam.test.ts @@ -176,7 +176,7 @@ class StubMysqlBackend implements StorageBackend { clearIngestWatermark(): Promise { return Promise.resolve(); } - resetAllIngestWatermarks(): Promise { + primeIngestTrust(): Promise { return Promise.resolve(); } probeIngestBoundary(): Promise {