diff --git a/docs/configuration.ja.md b/docs/configuration.ja.md index 215a6ffc3e..cd3ecc1232 100644 --- a/docs/configuration.ja.md +++ b/docs/configuration.ja.md @@ -212,6 +212,10 @@ | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | フェーズ 3 エンベロープスイッチ (#2960) — 今回のリリースでデフォルト反転: 未設定 (UNSET) はピュア SQLite 選択 (storage.unified 未設定または "sqlite") ではデフォルト OFF — SQLite インデックスが唯一の永続コピーとなる (読み取りはすでに統合優先; 起動時にインデックスの和集合を列挙するため、インデックスのみのセッションは正常にロードされる)。明示的な full は #3818 由来のダブルライトデフォルトを維持 (このキーが指定しない限りエンベロープは書き込まれ続ける)。有効な full モードではいずれの階層でも明示的な on/off が常に優先: shadow/legacy (または暗号化が有効な場合) で要求された off は、JSON ファイルが唯一の真実のソースであるため 1 回限りの警告付きで無視される。環境変数階層 0/false/off = off、1/true/on = on が厳密なファイル表記 ("off"/"on") より優先される。フェイルセーフ: サイドカーが実行中に自身を無効化した場合、次のセーブでエンベロープ書き込みが再開される。ダブルライトの復元: storage.jsonWriter=on (または BILI_STORAGE_JSON_WRITER=1) — インデックスのみのセッションは起動和集合経由で引き続きロードされ、レプリカ/リストアはポータブルな JSON エンベロープを再物質化する。バイナリのダウングレード (または SQLite エンジンの喪失) は、>=v0.1.197 バイナリが再び実行されるまで、インデックスのみのセッションを v0.1.196 以前のバージョンから不可視にする - データは index.db に無傷のまま残るが、古いコードによって読み取られないだけである。セッションごとの CCR コンテンツストアサイドカーも書き込みが停止する - full モードでは、すべての読み取りで最初にインデックスの ccr ミラーが参照されるため、ミラーが永続コピーとなる。jsonWriter=off レーンは、そのセッションディレクトリの唯一のライターでなければならない: off レーンは統合優先で読み取り、決してファイルにフォールバックしないため、同じディレクトリに対して legacy/shadow で実行されるピアインスタンスは静かに分岐する (その JSON コピーは更新を受け取らなくなり、off レーンはピアの書き込みを決して見ない)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal の WAL 回収上限 (#3250)、バイト単位: 完了したチェックポイントごとに WAL ファイルがこのサイズまで切り詰められ、長寿命プロセスにおけるディスク使用量とクラッシュ復旧リプレイを制限する。セッションごとのデータクォータではない - データは書き込まれた時点で index.db に着地し、WAL はチェックポイント間のフレームをステージングするだけなので、セッション数/サイズはこれをスケーリングしない; 大きなバーストは、次の完了チェックポイントが切り詰めるまで一時的に超過しうる。高い = 切り詰め IO が稀になるが、定常時のディスク使用量が増え、クラッシュリプレイのテールが長くなる。環境変数階層 (正の整数) がファイル階層より優先される; ゴミまたは非正の環境変数値は、起動を失敗させるのではなくファイル/デフォルト階層にフォールスルーする。レーン起動時にストアが開かれる時点で適用される - ホット編集には再起動が必要。有効な値は統合有効ログ行とストレージ診断ブロブに表示される。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | 日単位の GC 年齢スイープウィンドウ (#3463, #3901): これより長くアイドル状態のセッションはアーカイブされる - 単一のアーカイブログ (セッションディレクトリ配下の archive/archive.jsonl — セッションごとのファイルスプレーなし) に 1 行の JSON が追記され、トombストーン行がラベルとアイデンティティを保持し、コンテンツ/メディア参照が解放され、リストアは 1 つの POST のみで可能。0 は年齢スイープを無効にする; POST /__bili/storage/gc は常に実行され、既存の index.db を縮小するには ?vacuum=full を取る (新規書き込みには常に auto_vacuum=INCREMENTAL が有効)。同じ操作に CLI レイヤーがある (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run]、および bili sessions ls/rm/archive/restore (README の 'Storage maintenance CLI' セクション)。環境変数がファイル階層より優先される。 | +| `storage.snapBase.enabled` | boolean | false | — | スナップショットテーブル圧縮プレーンのマスターフラグ (#5874, schema v12): 4 つのプレーンテーブル (l1_world / snap_rows / snap_log / snap_ckpt)、単一事務コミット原語、読み出し側のチェックサム再構築を有効化する (不一致はそのセッションのプレーンを永続的に固着させる)。デフォルトオフ — 設定するまで挙動は不変; 設定ファイルのみ、環境変数階層なし (#2030 の規律)。sqlite パスにのみ配線 — mysql バックエンドはこのフラグを無視する。 | +| `storage.snapBase.shadowWrite` | boolean | false | — | プレーンのシャドウライト用のレーンフラグ、storage.snapBase.enabled にネスト (#5874): 両方が設定されると、各 ingest コミットが同一 tick 内で独立した事務所でプレーンを追加導出する (定常状態は delta 高速パス、それ以外は最終状態 diff; 導出可能なイベントのないラウンドは何も書かず、クラッシュで失われたラウンドは次の保存時に自己修復)。読み出し側は legacy 行との比較のみを行い、決して提供しない — 送信バイトは不変。レーン単独では無効; ロールバック = 解除 (または 4 テーブルの DROP + user_version の復元)。 | +| `storage.snapBase.egress` | boolean | false | — | #5874 P2 (mapEgress): スナップショットプレーンが直接の出站権威 — 出站圧縮コンテキストは S4 検証済みプレーンから提供され、legacy バイトとのラウンド毎の比較は行わない (tripwire なし、正規化 diff なし、sticky 除外なし)。プレーンはカーネル自身のタグ付き processTurn 出力を保存する (不変条件 v: post_json は書き換え後のコア形式であり、決してワイヤバイトではない); 書き込み側はカーネルがイベントレベルの増分として見たバイトを正確に永続化する (cast / update / delete / compact — 保存毎に 1 事務、legacy コミットと同一 tick)、そのため圧縮ラウンドの新規ブロックは同一ラウンドでプレーンから読み出せる。Egress はプレーン行を pos 順に読み (生存する active 行 + 各 active ブロックの保存済みアンカー要約をその被カバー run の pair-safe 位置へ移動; 被カバー行はプレーン自身の active フラグ / cover_ref で抑制し、id 一致では決して行わない)、その後レーン自身の後処理 (absorb → attach → strip → drop) を実行 — 出力は構造上 legacy egress とバイト単位で一致する (プレーン行が legacy 変換の入力そのものだから)。フラグオフ、cohort 除外、このセッションのプレーン dead/欠如、いずれかの行がパース不能、アセンブリ例外 — いずれの場合も legacy バイトへ降格 (1 ラウンド; 次ラウンドで再決定)、各理由は /__bili/status の snaplane.egress 下に表示。書き込み側は crash 等級: 不可解参照の view エントリ、または fold 時の view とプレーンの ID 不一致 (A/B 判定) はセッションを永続的に plane-dead に固定し、legacy が引き続き提供する。opt-in: 未設定時はデフォルト OFF (#5874 §9) — `egress: true` で切り替え; ロールバック = `egress: false` (バイト変化ゼロ、プレーンはシャドウライト継続)。既知のソフトエッジ: google countTokens と responses compact の二次 processTurn ポイントは legacy ソースのまま。 | +| `storage.snapBase.egressCohorts` | string[] | ["openai","anthropic","google","responses"] | — | egress ソースの切替を許可するプロトコル cohort (#5874 P2); openai/anthropic/google/responses の部分集合。プロトコルがリストにないセッションは無条件で legacy バイトを保持する (プレーンはシャドウライト継続)。空/欠如の配列は 4 つすべてを意味する。 | | `sessions` | object | {} (all defaults) | — | セッションテーブルサイズ上限とオプトインによるゴミ収集。 | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | メモリ内セッションテーブル上限 (アイドルセッションの LRU 退避)。 | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界な起動 (#2803/#3050): リスニング前に起動時に水合される最大最近保存セッション数; 古い状態はオンデマンドで再ロードされる。デフォルト 0 (急激なウォームセットなし — 最初のタッチが 1 回のオンデマンドロードを支払う; 急激な動作を復元するには 8 を設定)。sessions.max で上限が設定される。 | diff --git a/docs/configuration.md b/docs/configuration.md index 6eaf046056..a2e417c2db 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -215,6 +215,10 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960) — default flipped this release: UNSET now defaults to OFF for the pure-sqlite selection (storage.unified absent or "sqlite") — the SQLite index is the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Explicit full keeps the historical #3818 dual-write default (envelopes stay written unless this key says otherwise). An explicit on/off at either tier always wins in effective-full mode: under shadow/legacy (or with encryption active) a requested off is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings ("off"/"on"). Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. To restore dual-write: set storage.jsonWriter=on (or BILI_STORAGE_JSON_WRITER=1) — index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session CCR content-store sidecar stops being written as well - in full mode the index's ccr mirror is consulted first on every read, so the mirror is the durable copy. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier. | +| `storage.snapBase.enabled` | boolean | false | — | Master flag for the snapshot-table compression plane (#5874, schema v12): arms the four plane tables (l1_world / snap_rows / snap_log / snap_ckpt), the single-transaction commit primitive, and the read-side checksum rebuild (a mismatch permanently wedges that session's plane). Off by default — zero behavior change until set; config-file only, no env tier (#2030 discipline). Wired on the sqlite path only — the mysql backend ignores the flag. | +| `storage.snapBase.shadowWrite` | boolean | false | — | Lane flag for the plane's shadow write, nested under storage.snapBase.enabled (#5874): with both set, every ingest commit additionally derives the plane in its own transaction within the same tick (steady-state delta fast path, otherwise final-state diff; rounds yielding no derivable event write nothing, a crash-dropped round self-heals on the next save). The read side only compares against the legacy rows and never serves them — outbound bytes are unchanged. The lane alone has no effect; rollback = unset (or DROP the four tables + revert user_version). | +| `storage.snapBase.egress` | boolean | false | — | #5874 P2 (mapEgress): the snapshot plane is the DIRECT outbound authority — the outbound compressed context is served from the S4-verified plane, with NO per-round comparison against the legacy bytes (no tripwire, no normalized diff, no sticky exclusion). The plane stores the kernel's OWN tagged processTurn output (invariant v: post_json is the post-rewrite core form, never wire bytes); the write side persists exactly the bytes the kernel saw as event-level increments (cast / update / delete / compact — one transaction per save, same tick as the legacy commit), so a compression turn's brand-new block is plane-readable in the same round. Egress reads the plane rows in pos order (surviving active rows + each active block's stored anchor summary, moved to its covered run's pair-safe position; covered rows are suppressed by the plane's own active flag / cover_ref, never by id matching), then runs the lane's exact post-processing (absorb → attach → strip → drop) — the output is byte-identical to the legacy egress BY CONSTRUCTION, because the plane rows ARE the legacy transform input. Degrades to the legacy bytes (one round; the next round re-decides) on: flag off, cohort excluded, plane dead/missing for this session, any row unparsable, or an assembly exception — each reason surfaces on /__bili/status under snaplane.egress. The write side is crash-grade: an unaddressable view entry or a fold-time view-vs-plane identity mismatch (A/B oracle) latches the session plane-dead permanently, legacy keeps serving. Opt-in: default OFF when unset (#5874 §9) — set `egress: true` to cut over; rollback = `egress: false` (zero bytes change, the plane keeps shadow-writing). Known soft edges: google countTokens and responses compact secondary processTurn points still ride the legacy source. | +| `storage.snapBase.egressCohorts` | string[] | ["openai","anthropic","google","responses"] | — | Protocol cohorts allowed to cut the egress source over (#5874 P2); a subset of openai/anthropic/google/responses. Sessions whose protocol is absent keep legacy bytes unconditionally (the plane still shadow-writes for them). An empty/absent array means all four. | | `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. | diff --git a/docs/configuration.zh-CN.md b/docs/configuration.zh-CN.md index 6209c06540..877d8036c6 100644 --- a/docs/configuration.zh-CN.md +++ b/docs/configuration.zh-CN.md @@ -215,6 +215,10 @@ | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960)—— 本次发布翻转缺省:UNSET(未设置)现在对纯 SQLite 选择(storage.unified 未设置或 "sqlite")缺省为 OFF —— SQLite 索引成为唯一持久副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。显式 full 保留 #3818 的历史双写缺省(除非本键另说,信封照写)。有效 full 模式下任一层的显式 on/off 总是生效:shadow/legacy(或加密激活)下请求的 off 会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层("off"/"on")。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。恢复双写:设 storage.jsonWriter=on(或 BILI_STORAGE_JSON_WRITER=1)—— 纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 CCR content-store 边车也不再写 —— full 模式下读取时先查索引里的 ccr 镜像,镜像才是持久副本。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。 | +| `storage.snapBase.enabled` | boolean | false | — | 快照表压缩平面的主开关(#5874,schema v12):启用四张平面表(l1_world / snap_rows / snap_log / snap_ckpt)、单事务提交原语与读侧校验和重建(对不上会永久关死该会话的平面)。默认关闭 —— 不设置时零行为变化;仅配置文件层,无环境变量档(#2030 纪律)。目前只接 sqlite 路径 —— mysql 后端忽略该旗。 | +| `storage.snapBase.shadowWrite` | boolean | false | — | 平面影子写的车道开关,嵌套在 storage.snapBase.enabled 之下(#5874):两者都设置后,每次 ingest 提交在同一 tick 内用独立事务推导平面(稳态 delta 走快路,其余走最终态 diff;推导不出事件的轮次不写,crash 丢掉的轮次由下一轮保存自愈)。读侧只与 legacy 行比对、永不对外提供服务 —— 出站字节一个都不变。只开车道不生效;回滚 = 去掉开关(或 DROP 四张表 + user_version 退回)。 | +| `storage.snapBase.egress` | boolean | false | — | #5874 P2(mapEgress):快照平面就是直接的出站权威 —— 出站压缩上下文直接从 S4 校验过的平面读,不做任何与旧字节的逐轮比对(无 tripwire、无归一化 diff、无粘性排除)。平面存的是内核自己打标的 processTurn 输出(不变量 v:post_json 恒为改写后的核心形态,绝不含线路字节);写侧以事件级增量(cast / update / delete / compact —— 每次保存一个事务,与 legacy 提交同 tick)持久化内核看到的那份字节,所以压缩轮的新块同轮即可被平面读出。出站按 pos 顺序读平面行(存活活动行 + 每个活动块存的锚点摘要,移到其覆盖区段的安全位置;被覆盖行靠平面自身的 active 旗 / cover_ref 抑制,绝不靠 id 匹配),再跑本车道的同一后处理链(absorb → attach → strip → drop)—— 输出与旧出站逐字节一致是结构保证,因为平面行本身就是旧后处理的输入。降级回旧字节的情形(一轮,下一轮重新决定):旗关、cohort 不含、该会话平面死/缺、任一行不可解析、拼装异常 —— 每个原因都在 /__bili/status 的 snaplane.egress 下可见。写侧是 crash 级的:视图里出现不可寻址条目、或折叠时视图对平面身份对不上(A/B 裁判),都会把该会话平面永久锁死,legacy 继续兜底。显式开启(opt-in):不设置时默认关(#5874 §9)—— 设 `egress: true` 才切换;回滚 = `egress: false`(零字节变化,平面继续影子写)。已知软边界:google countTokens 与 responses compact 两个次级 processTurn 点仍走旧源。 | +| `storage.snapBase.egressCohorts` | string[] | ["openai","anthropic","google","responses"] | — | 允许切出站源的协议 cohort(#5874 P2),取 openai/anthropic/google/responses 的子集。协议不在名单内的会话无条件保留旧字节(平面照常影子写)。空/缺省 = 四个全开。 | | `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | | `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 | | `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 | diff --git a/docs/index.md b/docs/index.md index 102650852b..3f6d29e9b4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,6 +13,7 @@ | [clients](clients.md) / [中文](clients.zh-CN.md) / [日本語](clients.ja.md) | Per-client integration deep dives: how each mode (launcher / `/bili/` / native plugin) wires traffic in | | [how-it-works](how-it-works.en.svg) / [中文](how-it-works.zh-CN.svg) | One diagram: request flow + the injected context tools | | [README](README.md) / [中文](README.zh-CN.md) / [日本語](README.ja.md) | Directory landing page: section map + the rules that keep this directory live | +| [storage-relations](storage-relations.en.svg) / [中文](storage-relations.zh.svg) | One diagram: every storage table — what it stores, when it is written/read, which feature it serves (snapshot-table foundation #5901) | ## Identity & protocol @@ -35,3 +36,4 @@ | [lifecycle](lifecycle.md) | Transition-switch ledger / retirement inventory / hygiene record (CI-enforced `BILI_*` coverage) | | [auto-merge-guardrails](auto-merge-guardrails.md) | Evidence appendix behind §7 of `AGENTS.md` (AI auto-dev review history) | | [unified-loop-spec](unified-loop-spec.md) | Historical design: unified compress-loop spec (implemented; code is the source of truth) | +| [snapshot-table foundation](proposals/snapshot-compression-foundation/README.md) / [中文](proposals/snapshot-compression-foundation/README.zh-CN.md) | Design record: six-stage fixed chain, single storage truth (#5874). P0+P1+P2 landed in #5901; P2 = tripwire-guarded mapEgress cutover | diff --git a/docs/index.zh-CN.md b/docs/index.zh-CN.md index 248574b6d6..e3cd468daf 100644 --- a/docs/index.zh-CN.md +++ b/docs/index.zh-CN.md @@ -13,6 +13,7 @@ | [clients](clients.zh-CN.md) / [English](clients.md) / [日本語](clients.ja.md) | 各客户端接入深潜:launcher / `/bili/` / 原生插件三种模式如何接线 | | [how-it-works](how-it-works.zh-CN.svg) / [English](how-it-works.en.svg) | 一张图:请求流转 + 注入的上下文工具 | | [README](README.zh-CN.md) / [English](README.md) / [日本語](README.ja.md) | 目录落地页:分区地图 + 保持本目录不再变乱的规则 | +| [storage-relations](storage-relations.zh.svg) / [English](storage-relations.en.svg) | 一张图:存储关系——每张表存什么、何时写/读、对接哪个功能(快照表底座 #5901) | ## 身份与协议 @@ -35,3 +36,4 @@ | [lifecycle](lifecycle.md) | 过渡开关台账 / 退役模块盘点 / 卫生记录(CI 强制 `BILI_*` 覆盖) | | [auto-merge-guardrails](auto-merge-guardrails.md) | `AGENTS.md` §7 背后的证据附录(AI 自动开发评审史) | | [unified-loop-spec](unified-loop-spec.md) | 历史设计:统一压缩循环规格(已落地,以代码为准) | +| [快照表底座](proposals/snapshot-compression-foundation/README.zh-CN.md) / [EN](proposals/snapshot-compression-foundation/README.md) | 设计记录:六段固定链、单一存储真源(#5874)。P0+P1+P2 已在 #5901 落地;P2 = 带 tripwire 保护的 mapEgress 出站切换 | diff --git a/docs/proposals/snapshot-compression-foundation/README.md b/docs/proposals/snapshot-compression-foundation/README.md new file mode 100644 index 0000000000..15da17d53f --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/README.md @@ -0,0 +1,371 @@ +# Proposal: Snapshot-Table Compression Foundation (six-stage fixed chain, single storage truth) + +> **Status: P0 + P1 + P2 implemented in [#5901](https://github.com/ranxianglei/billion-context/pull/5901)** (schema v12; shadow write; P2 = mapEgress **direct-authority** egress — owner ruling 2026-11-11 removed the proposal's tripwire: the plane stores the kernel's OWN tagged processTurn output, so byte-identity with the legacy egress is a construction guarantee, and any plane gap degrades to the legacy bytes for one round). All flags ship **default OFF** (explicit opt-in; the 2026-11-08 soak ran them at 100% and was reverted before merge). P2-final lands the six-stage chain in the proposal's own order: the snapshot write (events → tables) runs **in-request**, after the kernel rewrite and before the egress stage — the plane is readable before the response is sent, and this round's own resend carries an empty tail. +> Written 2026-11-07 as a design proposal; this doc is the design record of the landed implementation. +> **Baseline:** origin/master (written 2026-11-07; storage schema v11; #5854 has fully reverted the v6 line from master). +> **Origin thread:** [#5643](https://github.com/ranxianglei/billion-context/issues/5643) (incl. the re-positioning comment). +> **Companion doc:** [中文版](./README.zh-CN.md) (fuller wording where the two diverge). + +--- + +## 0. TL;DR + +Replace today's parallel state sources (process memory + kstate JSON + table spread) with **one snapshot table set as the single storage truth**. Every model request walks the same fixed chain end to end: + +**raw bytes → protocol conversion (canonical cores) → rewrite/compression → apply to snapshot tables → egress mapping → outbound** + +The compression kernel (fold judgment, summaries, prompts, ref/msgid contracts) stays **byte-for-byte untouched**. Sessions whose host supplies both a session id and per-element msgids run entirely on the new chain; hosts without msgids degrade to order-then-hash identity and write into the same tables. Any mismatch anywhere sends that turn back to today's legacy full-resend path (degrade, never guess). Rollback = DROP the four new tables + flip the flag off — byte-identical return to today. + +Objective ordering is unchanged: **stability / extensibility / generality first, performance is a byproduct** — ask "is the flow correct", not "is it fast". + +--- + +## 1. Background & positioning + +### 1.1 History + +- **#5643** started as the implementation contract for "fast-forward watermark v6": a `(sess_raw, msgid)` composite-identity incremental fast lane for plugin hosts. +- The entire v6 line (fastpath + render-snapshot plane) was reverted out of master via **[#5854](https://github.com/ranxianglei/billion-context/pull/5854)** (clean revert, merged). +- The owner's re-positioning comment on #5643 (2026-11-07) fixed the new direction: this line is **not an incremental lane**, and not a sequel of v5 watermark — it is a **snapshot-table-based compression foundation**: storage-centric, kernel unchanged, with one fixed chain deciding what gets rewritten, stored, and rendered. +- [#5855](https://github.com/ranxianglei/billion-context/pull/5855) (transaction-skeleton re-land) is the **engineering substrate** of this line, not its end state. It was still open (conflicts unresolved) when this proposal was written — the first timing dependency. + +### 1.2 What this line is NOT (drift guard) + +| Easily conflated direction | Why it's not that | +|---|---| +| The incremental plugin-turn lane | That lane optimizes REQUEST SHAPE (delta-only transfer); here the server still receives full history each turn — it just stops doing indiscriminate parse/store over all of it | +| A sequel of the v5 watermark fastpath | v5 attacks PARSE COST (stamp handshake + tail-window parsing); this proposal fixes the STATE TRUTH problem; v5 keeps serving non-cohort traffic | +| Kernel swap / fold-algorithm change | Kernel zero-change is a hard constraint: fold judgment, summaries, cadence, id contracts stay as-is | +| A "faster" perf feature | Perf wins (stable prefix cache, lower memory) are byproducts of a correct flow, never the design criterion | + +### 1.3 Carried baseline rulings + +All standing rulings from the #5643 body remain in force without restatement here: objective ordering, the fixed chain, the two verification points (kernel-internal state + post-rewrite canonical form), append-only row keys, evidence-addressed diffing only (no fuzzy similarity), the four-piece protection (tripwire / S4 per-row checksums / witness requirement / audit sampling), pure-ignore of client-side compaction, invariants P5–P9. Conflicts, if any, must be registered explicitly in the ruling table (§14). + +--- + +## 2. Current architecture & three structural problems + +![Current architecture](./assets/arch-current.svg) (bitmap fallback: [arch-current.png](./assets/arch-current.png)) + +Today a model request passes through `prepare-*` adapters → `lane-decisions` branching (legacy full resend / incremental plugin-turn / v5 watermark 5% rollout / D-lane snapshot render ≈0% rollout) → kernel → `coreTo*` egress mapping. State lives in three places: process memory (lastMessages ASTs, resident core arrays), the unified SQLite v11 table family, and kstate JSON (hydrated through `mergeState()`). Three structural problems: + +1. **State lives in three places.** kstate JSON and the tables form a dual-write fork point; the direct producer of the #5403 OOM class is lastMessages pinned whole; restart depends on hydration assembly — assemble wrong and you either degrade or brick the session. The line we just reverted had exactly this parallel-truth disease at its root. +2. **Identity never lands on disk.** Host `(sess_raw, msgid)` pairs are stripped at ingest and live exactly one round in-flight. The `messages.msgId` column stores bili/kernel canonical ids; no table carries the host's raw identity. "Have I seen this element before?" cannot be answered across processes or restarts (P8 requires it always answerable), so every round cold-starts re-probing. +3. **Fastpaths patch independently.** v5 / incremental lane / D-lane each own a slice of traffic; cache stability is layered patches stacked on top of each other. One layer out of step = cache poisoning or full fallback. There is no single storage truth underneath them. + +--- + +## 3. Target architecture overview + +The full view is **pixel-aligned one-to-one with the §2 current-architecture diagram** — compare side by side to see exactly which part of the global picture changed and how big the blast radius is (grey = untouched · cyan = new stage/table · amber = role change · red dashed = fallback safety net): + +![Target architecture](./assets/arch-target.svg) (bitmap fallback: [arch-target.png](./assets/arch-target.png)) + +How the six-stage chain flows on the inside — **stage-by-stage expansion** (commit primitive / DDL chips / red-gate band R1–R4 / coexistence routing & rollout rollback): + +![Stage expansion](./assets/arch-stages.svg) (bitmap fallback: [arch-stages.png](./assets/arch-stages.png)) + +Where each piece of today's architecture lands under the new design — **component migration map** (bottom band maps the three structural pain points one-to-one to their fixes): + +![Component migration map](./assets/arch-migration.svg) (bitmap fallback: [arch-migration.png](./assets/arch-migration.png)) + +One-line duties per stage: + +| # | Stage | One line | Status | +|---|------|----------|--------| +| 0 | Ingress · identity registration | Dual-channel identity + L1 world match decides each row's provenance this turn | **NEW** | +| 1 | Protocol conversion → canonical cores | Reuse existing wire matrix; span-parse only the identity-diff NEW elements | Existing + one new regression pin | +| 2 | Rewrite / compression | Kernel untouched; emits structured events; landing timing per appendix-two | Kernel zero-change | +| 3 | Apply to snapshot tables | Single-transaction commit primitive writes the snap trio (rows/log/ckpt); ckpt meta carries kernel state + coverage map + S4 | **NEW CORE** | +| 4 | Egress mapping | Live rows ordered by pos ++ fresh-parse of tail window; dual-carrier branch | Read-side rebuild | +| 5 | Outbound | The verified plane IS the outbound authority; any gap degrades to the legacy bytes for one round (no per-round tripwire — owner ruling 2026-11-11) | Guard: degrade-only | + +Two cross-cutting bands: the **red-gate degradation band** (§7 — any doubt falls back to legacy) and **coexistence routing** (§8 — every traffic class has exactly one home). + +--- + +## 4. Stage-by-stage design + +### 4.0 Stage 0 — Ingress · identity registration (does not exist today) + +**Inputs:** the raw body plus dual-channel identity — `sess_raw` (host session id, original value) in the request header, per-element `msgids[]` in envelopes. **Collection reuses the carrier resolvers already merged in [#5842](https://github.com/ranxianglei/billion-context/pull/5842)** (shim / pi entry-walker / opencode transform hook / codex / cc / gemini candidate carriers) — no second collection system. + +**World-match decision** (sequential, first hit wins): + +1. **Echo direct-hit + spot check:** the id list the host echoes matches the server transcript positionally; adopt after sampled witness verification; +2. **Composite probe:** `(sess_raw, msgid)` hits the L1 table ⇒ continue the existing ledger identity; +3. **Mint new identity:** neither hits ⇒ mint a new ledger id, record it in L1, pay one arming round (that element enters storage but not cache decisions). + +**Cohort vs degradation boundary** (ruling I, §14): a cohort host (one that DECLARES dual-channel capability) missing ids in some round = failure → hard reject + zero side effects + feed the breaker (baseline §3.2 verbatim). Non-cohort hosts without msgids take the sticky order-then-hash degraded channel writing into the same tables (pos issued by ordinal). The mode is sticky per session; switching mid-session is one explicit resync — silent mixing forbidden. + +**L1 table** (the vehicle of ruling G): `l1_world(sess_raw PK, ledger_id, …)`, queryable globally and across restarts; original values never truncated (P9); sig/fingerprint go to warn diagnostics only, never into judgment (P8/P9). + +### 4.1 Stage 1 — Protocol conversion → canonical cores (reuse) + +- The wire matrix `{openai,anthropic,responses,google}ToCore ↔ coreToX` and cross-wire stable canonical ids (#3642) are already on master — zero rebuild. +- **Span parsing** covers only the NEW element set decided in stage 0 — these can sit at ANY body position (a client-compaction summary may appear at pos 0). Dirty implementation detail: the codec's ClusterCounter numbers `_N` per call, so parsing a scattered subset must reproduce the numbering context of a WHOLE-ARRAY call, or egress bytes drift = prefix cache breaks. **A byte-level regression pin for scattered-subset encoding is a P2 entry condition** (contiguous tail slices were proven feasible under v5; scattered subsets are the new case). +- Media cells (images etc.) reference the existing content CAS + `message_media`; base64 never enters rows. +- P5: exactly two legal entry points (fresh session / new-identity arming round); full re-parsing forbidden elsewhere. + +### 4.2 Stage 2 — Rewrite / compression (kernel zero-change) + +acp-kernel stays one line untouched: fold judgment (projectUnit shared projection), the four prompts, nudge flat-50K cadence, the id-never-reused contract. Its external output is defined here as a **structured event stream**: + +``` +cast first-time row minting (fresh session load) +append new rows (turn tail additions) +update in-row shape change (e.g. tool result backfill) +delete coverage/inactivation (active=0, never physically removed) +compact fold compaction (block absorbs covered span, meta updates coverage map) +``` + +Addressing rule follows baseline: events locate affected rows ONLY by kernel pipeline evidence (block coverage spans / ledgerSeq / frame boundaries) — no fuzzy similarity matching of any kind; if nothing lines up, the turn produces no events and runs legacy wholesale (§7 R2). + +**Landing timing (per #5643 appendix two):** first scene = the plugin endpoint `POST /__bili/plugin/tool` (src/server/admin.ts:797) completes events→table mutation **synchronously, in a single transaction, inside that request**; the next model turn only reconciles. Hosts bypassing the endpoint (dsh native compaction and kin) feed the SAME table-mutation routine via checkpoint-frame signature + rebase detection. Between endpoint and landed write the plane carries a `structural-pending` mark; losing the mark across a process restart = conservatively dead (fail-safe direction, §7 R4). + +### 4.3 Stage 3 — Apply to snapshot tables (the body of this proposal) + +**Commit primitive = one transaction** with three steps: + +``` +upsert snap_rows (rows addressed by evidence) + → append snap_log (seq strictly > max(log.max_seq, ckpt.seq)) + → (optional) compact snap_ckpt (absorb log payloads, update meta) +any step throws ⇒ abandon the whole turn, no half-state (invariant i) +``` + +**Invariant table** (draft v0.1 + review hardening, carried forward): + +| # | Invariant | +|---|-----------| +| i | Single writer per session: ALL table mutations go through this one primitive | +| ii | log.seq strictly monotonic: > max(log.max_seq, ckpt.seq); no numbering break across compactions | +| iii | S4: per-row checksum mismatch ⇒ that sid's plane is CLOSED FOREVER, whole thing falls back to the old path (red line unchanged) | +| iv | Rows are never physically deleted: folds flip active; pre/post forms preserved for decompress | +| v | post_json is always the "right-of-rewrite canonical form" — wire bytes may never be cached | + +**ckpt.meta_json triple** (where rulings F+C converge): (1) serialized kernel conversation state (CompressionState: blocks/refs/deadRefs/nudge/tokenSnapshot/…) + its own checksum — the kernel has NO deterministic "event stream → state" rebuild entry, so serialization-with-checkpoint is the only option that both keeps the kernel untouched and satisfies P7 (memory is just a cache); (2) the **coverage map** (summary pos → covered pos ranges) — delete/compact payloads get absorbed into the checkpoint, and after compaction this explicit map is decompress/GC's sole basis; (3) the S4 aggregate checksum. + +**Inactive-row lifecycle** (ruling B recommendation): inactive rows persist until their covering block itself is archived/deleted, then physical cleanup rides along; against both "never clean" (#3752 unbounded growth is a p1 red line) and global TTL (avoids extra alignment proofs between GC windows and dereference windows). + +### 4.4 Stage 4 — Egress mapping + +`mapEgress` = live rows ordered by pos, their cores, ++ freshly parsed cores of the tail window `[K..T)` this turn, then `coreToX` into the target protocol family. Prefix bytes stable across turns ⇒ prefix-cache friendly — this IS the objective function, structurally guaranteed rather than coincidental. + +- **Dual-carrier branch** (sticky per session; BOTH modes must be reasoned together — classic #377-class bug in this repo): pluginMode suppresses tool+nudge injection (the agent executes compression locally); proxy mode revoices acp_summary as a user message (systemToUser, keeping the head system message byte-stable). +- **Transparent changes** (NOT red conditions): switching model across protocol families, maxDim change, toolArmed change — they only vary mapping inputs, since snapshots store only the right-of-rewrite canonical form. +- Desync handling: join-chain broken / fails to align ⇒ the whole turn runs the legacy full path and re-stamps (client converges within one round). + +### 4.5 Stage 5 — Outbound + +**Final form after the owner's ruling (replaces ruling J):** the per-round tripwire was REMOVED — the plane is the direct outbound authority, and byte-identity with the legacy egress holds by construction because the plane rows ARE the legacy transform input (invariant v: post_json is the kernel's own tagged core form). The final runtime guard is the fold-time A/B identity oracle (view-vs-plane multiset): a mismatch latches that session plane-dead permanently and legacy keeps serving — it protects against "feeding stale prefixes out of a drifted state" (cache poisoning) exactly as the tripwire did, without a per-round comparison. + +**Protocol hygiene:** pf-* / envelope-style handshake headers are protocol headers, not hop headers — strip before forwarding upstream (UPSTREAM_HOP_HEADERS whitelist discipline); any NEW handshake header introduced by this proposal joins the same whitelist management. A/B byte pins stay in place throughout rollout. + +--- + +## 5. Storage schema (final DDL landed as schema v12 in #5901) + +Version-stacking rule: #5901 landed this DDL as schema **v12** (`SNAPBASE_DDL_V12` in `src/storage/schema.ts`) directly on the v11 master, because #5855 was still open at the time; if #5855 lands first, the later-landing side renumbers mechanically (same DDL, one version bump). Old-library upgrade is purely additive (`CREATE ... IF NOT EXISTS`), no data migration — and the application itself is probe-first (below), so the version number is a hygiene step, not a gate. Per-table read/write map (zh/en): [storage-relations.en.svg](../../storage-relations.en.svg) / [storage-relations.zh.svg](../../storage-relations.zh.svg). DDL (final types as landed): + +```sql +-- L1 identity thin table (vehicle of ruling G; restores baseline §4.1.1's two columns as a dedicated table) +-- v12's 2×2 stage-0 ADDS the per-element msgid envelope (envelope_json / +-- envelope_checksum) and the R3 latch (r3_latched_at) via probe-first +-- ALTERs — a fresh DB already has all of it. +CREATE TABLE IF NOT EXISTS l1_world ( + sess_raw TEXT PRIMARY KEY, -- host session id, ORIGINAL value, never truncated (P9) + ledger_id TEXT NOT NULL, -- bili canonical sid (existing key space, ruling D) + witness_probe TEXT NOT NULL, -- echo spot-check witness sample (diagnostics stream only) + issued_at INTEGER NOT NULL, + armed_at INTEGER, -- arming-round mark for freshly minted identities + envelope_json TEXT, -- 2×2: {id, pos} element canon (matchWorld / matchAppend) + envelope_checksum TEXT, -- 2×2: canonical-checksum over the envelope + r3_latched_at INTEGER -- R3 identity-breach persistent latch (M3) +); +CREATE INDEX IF NOT EXISTS idx_l1_ledger ON l1_world(ledger_id); + +-- Row table: the transcript proper +CREATE TABLE IF NOT EXISTS snap_rows ( + sid TEXT NOT NULL, + pos INTEGER NOT NULL, -- ledger absolute position, append-only, never reused + core_ids TEXT NOT NULL, -- canonical-core element id list (json text) + pre_json TEXT, -- pre-rewrite form reference (CAS digest); ruling A② defaults to reference, not inline + post_json TEXT, -- post-rewrite canonical form (the ONLY form allowed to be cached) + tokens_est INTEGER NOT NULL DEFAULT 0, -- estimated tokens (ruling L: growth data source for judgment + billing share) + active INTEGER NOT NULL DEFAULT 1, -- 0 = covered/inactive (kept for decompress; lifecycle per §4.3) + cover_ref TEXT, -- id of the summary block covering it (lifecycle anchor for physical cleanup) + updated_at INTEGER NOT NULL, + PRIMARY KEY (sid, pos) +); + +-- Event log (LSM log layer) +CREATE TABLE IF NOT EXISTS snap_log ( + sid TEXT NOT NULL, + seq INTEGER NOT NULL, -- strictly > max(log.max_seq, ckpt.seq) + kind TEXT NOT NULL, -- cast|append|update|delete|compact + payload TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (sid, seq) +); + +-- Checkpoint (LSM meta layer; absorbs log payloads AND explicitly retains the coverage map) +CREATE TABLE IF NOT EXISTS snap_ckpt ( + sid TEXT PRIMARY KEY, + ckpt_seq INTEGER NOT NULL, + meta_json TEXT NOT NULL DEFAULT '{}', -- serialized kernel state + coverage map + own checksum + checksum TEXT NOT NULL, -- S4 aggregate checksum + updated_at INTEGER NOT NULL +); +``` + +**Migration is probe-first, not version-gated** (landed behavior): the DDL is pure `CREATE ... IF NOT EXISTS`, so it runs at ANY from-version; the migrator probes for `snap_rows` itself. Real-world reason: #5855 consumed v12+v13 (`pf_*` planes) in a shared database while this was unmerged, and the old `if (from < 12)` gate silently skipped the plane on first save ("no such table: l1_world"). The final master version number stays a mechanical merge-time hygiene step. + +**Write-model assumption** (ruling H item): default is the single-process assumption (WAL serializes cross-process writes naturally). If multi-instance concurrent writes are wanted, an in-memory-view invalidation protocol must ship with it (who invalidates resident core arrays / metadata caches on commit) — P0 contract picks one, definitively. + +--- + +## 6. Read paths & consumer-switching ledger + +**Hydration dual path:** green path (new tables readable AND S4 passes) rebuilds the read view from snap_rows + meta; red path (flag off / no active block / tables unreadable / checksum fail) falls back to the kstate-old path verbatim. Switching is per-session and one-directional; any step out of alignment sends the whole turn back to legacy — same discipline as D-lane #3756. + +Consumers annotated individually (P0 lists the complete inventory; this is the mainline): + +| Consumer | Disposition | +|----------|-------------| +| decompress (restore/decompress tool path) | P3 switch to new-table pre/post dual forms + coverage map; missing pre bytes MUST surface through an explicit audit diagnostics exit — never silent empty references | +| export | P3 switch; honest-output discipline unchanged (#155: a 0-block session may never claim "original conversation") | +| billing | shares the tokens_est column; switching read source does not change billing semantics | +| Web UI (sessions panel / acp_status uncompressed view) | acp_status judgment face (KDD#12 shared projection) input source points at new-table rows; panel gains table-health metrics (§10) | +| kv_store namespaces / notice store / persona fork (`\|sub:`) | Untouched (independent mechanisms; the fork records its suffixed id into the same lane) | +| CCR (ccr_entries) | Not touched during P0–P2; P3 evaluates merge vs long-term coexistence separately | + +--- + +## 7. Red-gate enumeration (hit ANY ⇒ the turn/session falls back wholesale to the legacy full path and re-stamps) + +| # | Red condition | Consequence | +|---|---------------|-------------| +| R1 | S4 / checksum corrupted | that sid's plane CLOSED FOREVER, whole fallback to the old path (substrate corruption unrecoverable; red line unchanged) | +| R2 | Egress join chain broken (core_ids ↔ rows fail to align; chain walk-back fails) | no events produced this turn; legacy wholesale | +| R3 | Cohost identity gate fault (cohort host fails to deliver dual-channel ids) | hard reject + zero side effects + feed breaker (baseline §3.2 verbatim) | +| R4 | structural-pending unconsumed (incl. restart-lost mark) | conservatively dead (fail-safe direction) | + +**NOT red conditions:** model switch across protocol families, maxDim change, toolArmed change, client-side compaction traces (purely ignored — increments only). These boundaries become decision functions in the lane-decisions matrix, one positive+negative row pair per combination (#3617 discipline). + +--- + +## 8. Coexistence routing with existing lanes (every traffic class has exactly one home) + +| Traffic | Home | Notes | +|---------|------|-------| +| Cohort hosts (dual channel) full-resend sessions | **New foundation** | the primary served population | +| Non-cohort (no msgid) traffic | v5 watermark keeps serving | same tables, different lane; independent flag bits; the order-then-hash degraded writes use the same tables | +| Anthropic-wire divert turns | incremental plugin-turn lane keeps its job | dsh shim picks one per turn; no double processing | +| Endpoint-bypassing hosts (dsh native compaction et al.) | stage-2's SAME event routine (checkpoint-frame signature + rebase detection feed-in) | see §4.2 timing | +| #3756 D-lane | **Yields/retires** on lanes where the new foundation is live (flag mutual exclusion) | pending ruling K; retirement decision lands in P3 | + +lane-decisions matrix: one row per combination, positive+negative pairs ("triggers when it should; does NOT trigger when it should not"); added rows discoverable. + +--- + +## 9. Rollout & config surface (owner-gated; keys finalized in #5901) + +- **Two-level nested flags:** master flag ⊃ lane flags (a lane flag means anything only when master is on), config-file keys resolved through `src/knobs.ts` (env > file > default); **no new env knobs** (scarce-surface discipline #2030). Keys as landed, all **default OFF** (explicit opt-in): config-file `storage.snapBase.enabled` (master flag) / `storage.snapBase.shadowWrite` (lane flag, master ⊃ lane) / `storage.snapBase.egress` (P2 direct-authority egress, opt-in) / `storage.snapBase.egressCohorts` (protocol subset, default all four), see [configuration](../../configuration.md). +- **Cohort roster:** config-level declaration (which carriers/hosts claim dual-channel capability); initial value TBD (open item D4: verify the six candidate carriers one by one). +- **Web config page** tri-state quick switches shipped in lockstep; i18n en/zh pairs; registry-test dead-key guard rows; website/config-reference yaml seeds where a knob ships. +- Default OFF (landed state: all four keys ship off; the 2026-11-08 soak ran them at 100% and was reverted before merge), bucketed per session (an independent salted bucket), P2 starts as single-lane gray rollout. + +--- + +## 10. Observability surface + +- **Decision log:** unified vocabulary (hit / miss / red: / degrade / resync / arm …), every line grep-able and aggregatable; during rollout it IS the confidence evidence. +- **`/__bili/status`:** attribution field extended with foundation dimensions (which chain serves this session, plane health). +- **Panel health metrics:** snap table row counts / log depth / ckpt lag (seq distance) / GC waterline / egress degrade-reason counters (`snaplane.egress` on `/__bili/status`) — sessions panel or a dedicated card. +- **acp_status uncompressed view:** judgment input source repointed to new-table rows (KDD#12 shared-projection apply/recommend/status consistency re-pinned on the new tables). + +--- + +## 11. Test matrix + +Phase test skeleton (positive+negative pairing applies throughout): + +| # | Test | Phase | +|---|------|-------| +| T1 | Upgrade additivity pin: pre/post-upgrade existing query behavior byte-identical; DROP-four-new-tables rollback drill = byte-identical return to today | P0 | +| T2 | Event transaction unit: commit primitive atomicity; seq monotonic across compaction; throw leaves no half-state | P0 | +| T3 | B1 trap pin × 3 consecutive restarts oracle comparison (hydrated snapshot must carry the active world origin o; frontier movable only by arm/reset) | P1 | +| T4 | A/B byte-level: on N-turn real corpora, new-chain egress bytes == legacy full-path artifact (modulo time attr) | P1–P3, always on | +| T5 | Scattered-NEW encoding byte-equivalence pin (ClusterCounter reproduction) | P2 entry | +| T6 | deleteSession cascade incl. l1_world — no orphans | P0/P1 | +| T7 | Coverage map survives compaction: decompress/GC basis still queryable after compact | P1 | +| T8 | Post-GC decompress honest-failure exit (missing pre ≠ silent empty reference) | P3 | +| T9 | Long-session volume growth curve model (row count / log depth / ckpt size vs the #3752 red line), long-run acceptance | P1 | +| T10 | Red-gate matrix: R1–R4 each with a "triggers-and-falls-back" + "must-NOT-trigger" pair row | per phase, alongside modules | +| T11 | Real-client e2e (pi / dsh × sglang) full chain | P2 entry | +| T12 | Dual-mode (pluginMode/proxyMode) carrier branch double verification (§4.4) | P2 | + +CI: docker gate + solver-free; hermetic; ports via listen(0) (#360 discipline). + +--- + +## 12. Phased delivery (each phase = its own PR; human gates marked) + +| Phase | Scope | Behavior surface | Acceptance | +|-------|-------|------------------|------------| +| **P0** | DDL four-piece + commit primitive + upgrade pin + rollback drill + event transaction unit tests | **ZERO behavior change** (foundation laying only) | T1/T2/T6; write-model assumption (H) and key naming (config-surface section) finalized here — human gate ① | +| **P1** | Shadow write: new tables populated in parallel with the old path; read side completely untouched | Zero behavior surface | T3/T4(shadow compare)/T7/T9 long-run curves; volume model passes red-line check — human gate ② (critical trust-rebuild period) | +| **P2** | Single-lane hydrate-egress cutover gray rollout (first cohort member; suggest pi or dsh) | Gray sessions read egress from new tables | T5/T10/T11/T12 + A/B identity oracle at fold + direct-authority egress; one-click exit switch | +| **P3** | decompress/export/UI/billing switched one consumer at a time; CCR/persona disposition; #3756 retirement decision; cohort roster expansion | Per-consumer switches | T8 + full matrix rerun; each phase its own PR | + +**Ruling E recommendation stood down for P2:** the owner directed single-PR completion (#5901 now carries P0+P1+P2). P2 landed as `mapEgress` ([src/server/snap-egress.ts](../../../src/server/snap-egress.ts)): the snapshot plane is the **DIRECT outbound authority** — the outbound context is served from the S4-verified plane with NO per-round comparison against the legacy bytes (the proposal's tripwire — ruling J — was removed by the owner: the plane stores the kernel's OWN tagged output, so byte-identity with the legacy egress holds by construction and a per-round diff adds nothing but complexity). The write side persists exactly the bytes the kernel saw as event-level increments (cast / update / delete / compact — one transaction per save, same tick as the legacy commit), so a compression turn's brand-new block is plane-readable in the same round. Egress reads the plane rows in pos order (surviving active rows + each block's stored anchor summary at its covered run's pair-safe position; covered rows are suppressed by the plane's own active flag / cover_ref, never by id matching), then runs the lane's exact post-processing (absorb → attach → strip → drop). Degrades to the legacy bytes for one round (next round re-decides) on: flag off, cohort excluded, plane dead/missing, any row unparsable, or an assembly exception; an unaddressable view entry or a fold-time view-vs-plane identity mismatch latches the session plane-dead permanently. Cohorts = protocol lanes (`storage.snapBase.egressCohorts`, default all four), see [configuration](../../configuration.md). + +--- + +## 13. Prerequisites & sequencing + +1. **#5855 merges FIRST** (human merge): it is the engineering substrate of the transaction skeleton AND the schema-version layer beneath ours. While it sits open/dirty, P0 DDL cannot stack versions cleanly. (2026-11-08 reality: #5901 did not wait for it — the P0 DDL landed as v12 directly on the v11 master; if #5855 merges afterward, the later-landing side renumbers mechanically, DDL unchanged.) +2. #5842 (identity collection) ✓ already on master — stage 0 reuses it directly; nothing to wait for. +3. #5854 (revert) ✓ merged — confirms zero v6 symbol residue on master. +4. Rulings closed (§14): F/G/B decide the table-shape skeleton — decide those first; the rest ride with the P0 PR. + +--- + +## 14. Pending owner rulings + +Already discussed online (A–E, draft-v0.1 numbering) + gap-analysis additions (F–L) + carryovers (D3–D5): + +| # | Item | Recommendation | Blocks | +|---|------|----------------|--------| +| A | In-row form | ② store post only, pre references existing CAS; addendum: pre bytes referenced by a LIVE row must outlive that row | P0 DDL | +| **B** | Inactive-row retention | compaction cleanup bound to covering-block lifecycle; against both never-clean (#3752 red line) and global TTL | P0 DDL / GC design | +| C | Summary placement | ① occupy covered-span head pos in place + ckpt meta explicitly retains the coverage map | P0 DDL | +| D | Session key space | reuse existing sid | P0 DDL | +| E | Scope this cycle | hand off at P0+P1 for acceptance; P2 separate PR with rollout | Scheduling | +| **F** | Kernel state residence | ckpt.meta_json carries serialized CompressionState + checksum (kernel untouched, O(1) hydration) | P0/P1 core | +| **G** | L1 identity thin table | restore persistent (sess_raw, ledger_id) per baseline §4.1.1 (dedicated thin table + index), reusing #5842 collection | P0 DDL | +| **H** | Write model | single-process assumption (WAL serializes) OR multi-process + in-memory-view invalidation protocol — pick one, definitive, in P0 | P0 | +| **I** | Hard gate vs degradation boundary | cohort-claimed host: fault reject + zero side effects + breaker; non-cohort: order-then-hash same-table degraded writes, sticky per session, mode switch = resync | P0/P2 routing | +| **J** | Tripwire | runtime low-ratio sampling + immediate legacy drop + alert on mismatch | P2 | +| **K** | Coexistence with #3756 / v5 | #3756 yields/retires (flag mutual exclusion) on new-foundation lanes; v5 continues serving non-cohort | P2/P3 | +| **L** | In-row token accounting column | tokens_est in-table: growth data source for judgment + billing share | P0 DDL | +| D3 | Fault-response shape | HTTP code / error body / client retry contract (peer-side view of R3 hard reject) | P2 | +| D4 | Initial cohort roster | verify the six candidate carriers (shim/pi/opencode/codex/cc/gemini) one by one for carrier stability | P0 config initial value | +| D5 | Re-numbering semantics | same content, different id: count as new duplicate pair or pair-level dedup? | P0 semantics | + +--- + +## 15. Risks & honest boundaries + +- **"Element-count-preserving prefix byte rewrites are undetectable"** (carried cost clause from baseline): the price is a trust-model change that skips per-item re-verification — human review required, auto-merge forbidden (same gate tier as KDD §7.4 items). +- **Once a multimodal tail enters the cache, that history segment permanently degrades to legacy full turns** (carried KDD#15 honest boundary ②): degrade rather than deviate. +- **Marker-strip counts inherit with the seed ledger**; mid-history regions are never rescanned (boundary ③ above). +- The final arbiter of every "invariant" in this document is the runtime tripwire + S4 sampling, not paper argument — which is precisely why the P1 shadow phase exists: to accumulate that empirical evidence before any read-side cutover. + +--- + +## 16. Reference numbers + +#5643 (origin + re-positioning) · #5854 (full revert, merged) · #5855 (substrate re-land, open) · #5842 (identity wiring, merged) · #3756 (D-lane snapshot render; disposition pending ruling K) · #3752 (unbounded storage growth p1 red line) · #5403 (OOM: lastMessages pinning) · #2432 (client-compaction death spiral; source of the pure-ignore policy) · #3236 (shared projection judgment) · #3642 (cross-wire stable canonical ids) · #3617 (gate bidirectional assertions) · #1104/#1106/#1107 (fingerprint permitlist discipline) · #4491/#4834 (msgid identity hierarchy) · #5539 (schema v11 FK subindex) · #155 (honest output) · #360 (listen(0) port discipline) · MESSAGE-IDENTITY.md (identity doctrine) · #5643 re-positioning comment: diff --git a/docs/proposals/snapshot-compression-foundation/README.zh-CN.md b/docs/proposals/snapshot-compression-foundation/README.zh-CN.md new file mode 100644 index 0000000000..b8a4a7cbae --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/README.zh-CN.md @@ -0,0 +1,364 @@ +# 提案:快照表式压缩底座(六段固定链,单一存储真源) + +> **状态:P0 + P1 + P2 已在 [#5901](https://github.com/ranxianglei/billion-context/pull/5901) 实现**(schema v12;影子写;P2 = mapEgress 直接权威出站 —— 提案里的 tripwire(J 裁定)后被 owner 移除:平面存内核自己的打标输出,与旧出站逐字节一致是结构保证,逐轮比对只添复杂度)。全部开关**出厂默认关**(显式开启;2026-11-08 的 100% soak 已在合并前翻回)。P2-final 按提案原序落地六段链:快照写入(事件→表,cast/update/delete/compact 事件级增量)在**请求内**完成 —— 内核改写之后、出站改写之前,平面在响应发出前即可读,本轮自身重发的尾窗为空。 +> 2026-11-07 作为设计提案撰写;本文件是已落地实现的设计记录。 +> **基线:** origin/master(写于 2026-11-07;存储 schema v11,#5854 已把 v6 整线撤出 master)。 +> **来源讨论:** [#5643](https://github.com/ranxianglei/billion-context/issues/5643)(含定位纠正评论)。 +> **姊妹文档:** [English version](./README.md)。 + +--- + +## 0. 摘要(TL;DR) + +本提案把上下文压缩的状态底座从「进程内存 + kstate JSON + 多张表」的平行真源格局,换成**一套快照表作为唯一存储真源**。每轮请求固定走完同一条链: + +**原始字节 → 协议转换(通用核) → 改写(压缩) → 快照表应用 → 出站映射 → 出站** + +压缩内核(折判定、摘要、prompts、ref/msgid 契约)**零改动**。有宿主 session+msgid 双信道的会话全部走新链路;没有 msgid 的宿主按「顺序 > hash」降级身份写同一套表。任何环节失步,该轮整体回落今天的 legacy 全量路径(降级,不猜)。回滚 = DROP 四张新表 + 关旗,逐字节回到今天。 + +目标函数排序不变:**稳定性 / 扩展性 / 通用性优先,性能是副产品**——先问流转对不对,不问快不快。 + +--- + +## 1. 背景与定位 + +### 1.1 来龙去脉 + +- **#5643** 最初是「快进水位 v6」实现契约:用 `(sess_raw, msgid)` 复合身份给插件车道做增量快路。 +- 整条 v6 线(快路径 + render snapshot 平面)经 **[#5854](https://github.com/ranxianglei/billion-context/pull/5854)** 完整撤出 master(干净 revert)。 +- 撤回后的定位纠正(#5643 评论区,2026-11-07)把方向定死了:这条线**不是增量车道,也不是 v5 watermark 的续作**,而是**基于快照表的压缩底座**——以新存储为主,内核不动,让「改什么、存什么、渲染什么」全部由一条固定链决定。 +- [#5855](https://github.com/ranxianglei/billion-context/pull/5855)(事务骨架 re-land)是本线的**工程基底**,不是终态。截至本提案写作时它仍 open(冲突未解)——这是本提案的第一时序依赖。 + +### 1.2 这条线不是什么(防止再次漂移) + +| 容易误认的方向 | 为什么不是 | +|---|---| +| 增量车道(plugin-turn divert) | 那条是「只传新增 delta」的请求形状优化;本提案每轮仍收全量重发,只是服务端不再对全量做无差别解析/存储 | +| v5 watermark 快路径的续作 | v5 解决的是「解析成本」(戳握手 + 尾窗解析);本提案解决的是「状态真源」,v5 继续服务非 cohort 流量 | +| 换内核/改折叠算法 | 内核零改动是硬约束;折叠判定、摘要、节拍、id 契约原样保留 | +| 「快一点」的性能特性 | 性能收益(前缀缓存稳定、内存下降)是流转正确的副产品,不作为设计判据 | + +### 1.3 保留的基准裁定(继承自 #5643 正文,全文有效) + +目标函数排序、固定链、两处校验点(内核内部状态 + 改写后核形态)、行键 append-only、diff 只按内核管道证据寻址(禁模糊相似)、四件套保护(tripwire / S4 逐行校验和 / 见证要求 / 审计抽验)、客户端压缩纯忽略、P5–P9 五组不变量,全部沿用,不在本文重复展开;与本文冲突处以本节为准并在裁定表(§14)显式登记。 + +--- + +## 2. 现状架构与三个结构性问题 + +![现状架构](./assets/arch-current.svg)(位图备选:[arch-current.png](./assets/arch-current.png)) + +今天的一条模型请求从入站到出站要经过:`prepare-*` 适配器 → `lane-decisions` 分轨(legacy 全量 / 增量 plugin-turn / v5 watermark 5%灰度 / D-lane 快照渲染 ≈0%灰度)→ 内核 → `coreTo*` 出站映射。状态住在三个地方:进程内存(lastMessages AST、常驻核数组)、统一 SQLite v11 表族、kstate JSON(经 `mergeState()` 水合)。三个结构性问题: + +1. **状态三处住(痛点①)**——kstate JSON 与表是双通道分叉点;#5403 OOM 的直接生产者就是 lastMessages 整包驻留;重启靠水合拼装,拼错即降级或烧会话。刚被撤回的那条线的根病正是这种平行真源。 +2. **身份不落盘(痛点②)**——宿主的 `(sess_raw, msgid)` 对在 ingest 时剥掉,只活在 in-flight 信封里一轮。`messages.msgId` 列存的是 bili/内核自己的规范 id,宿主的原始身份没有任何表承载。「这个元素见过没?」跨进程、跨重启无法回答(P8 要求恒可答),于是每轮都冷启动重新探测。 +3. **快路径各自打补丁(痛点③)**——v5 / 增量车道 / D-lane 各管一段流量,缓存稳定性靠多层补丁堆叠;任何一层失步就是缓存投毒或整体回退。没有一个单一存储真源兜底。 + +--- + +## 3. 目标架构总览 + +全貌与 §2 现状架构图**逐位对齐**,直接对照即可看出改的是全局哪一块、影响范围多大(灰 = 零改动 · 青 = 新增段/表 · 琥珀 = 角色变更 · 红虚线 = 兜底安全网): + +![目标架构](./assets/arch-target.svg)(位图备选:[arch-target.png](./assets/arch-target.png)) + +六段固定链自身的内部流转 —— **分阶段展开**(提交原语 / DDL chip / 降级红门 R1–R4 / 共存路由与回滚灰度): + +![分阶段展开](./assets/arch-stages.svg)(位图备选:[arch-stages.png](./assets/arch-stages.png)) + +现状图里的每一块在新架构下变成什么样子 —— **组件迁移总览**(底部为三个结构性痛点的一一对应解决锚点): + +![组件迁移总览](./assets/arch-migration.svg)(位图备选:[arch-migration.png](./assets/arch-migration.png)) + +六个阶段的职责一句话版: + +| # | 阶段 | 一句话 | 状态 | +|---|------|--------|------| +| 0 | 入站·身份入册 | 双信道身份 + L1 世界匹配,决定这轮每行的出身 | **新增** | +| 1 | 协议转换→通用核 | wire 矩阵现成复用;只对身份差集 NEW 元素做 span 解析 | 现成 + 一个新回归钉 | +| 2 | 改写/压缩 | 内核一行不改,产出结构化事件;落表时序按附录二 | 内核零改动 | +| 3 | 快照表应用 | 单事务提交原语写入 snap 三件套(rows/log/ckpt);ckpt meta 携带内核态+覆盖映射+S4 | **新增核心** | +| 4 | 出站映射 | 存活行按 pos 排序 ++ 尾窗新解析;双 carrier 分支 | 读侧重建 | +| 5 | 出站 | S4 校验过的平面即直接出站权威;平面缺/死/异常时降级旧字节一轮(无逐轮 tripwire —— owner 裁定 2026-11-11) | 守卫:仅降级 | + +横切两条:**红门降级带**(§7,任何疑点回落 legacy)与**共存路由**(§8,每种流量只有一个家)。 + +--- + +## 4. 六段固定链逐段设计 + +### 4.0 第 0 段:入站 · 身份入册(现状中不存在的一段) + +**输入**:原始字节体 + 双信道身份——请求头里的 `sess_raw`(宿主会话 id 原值)+ 逐元素 `msgid[]` 信封。**采集直接消费 [#5842](https://github.com/ranxianglei/billion-context/pull/5842) 已合入的载体 resolver 输出**(shim / pi entry-walker / opencode transform 钩子 / codex / cc / gemini 六候选载体),不建第二套采集系统。 + +**世界匹配裁决**(顺序执行,第一个命中即止): + +1. **echo 直中 + 抽验**:宿主回带的 id 列表与服务端正史逐位吻合,抽样见证核对通过后直接采用; +2. **复合探测**:`(sess_raw, msgid)` 在 L1 表中探测命中 ⇒ 沿用既有账本身份; +3. **发放新身份**:都没命中 ⇒ 发放新账本 id 并记入 L1,付一轮武装(本轮该元素只进不入缓存决策)。 + +**cohort vs 降级的边界**(裁定 I,见 §14):声称具备双信道的 cohort 宿主某轮收不齐 id = 故障,硬拒绝 + 零副作用 + 喂 breaker(baseline §3.2 原样);天生没有 msgid 的非 cohort 宿主走「顺序 > hash」粘滞降级通道,写同一套表(pos 由序位发号);模式每会话粘滞,中途换模式 = 一次显式 resync,禁止静默混用。 + +**L1 表**(G 裁定的承载体):`l1_world(sess_raw PK, ledger_id, …)`,全局可跨重启查询;原值禁截断(P9),sig/fingerprint 只进 warn 诊断流永不入判决(P8/P9)。 + +### 4.1 第 1 段:协议转换 → 通用核(现成复用) + +- wire 矩阵 `{openai,anthropic,responses,google}ToCore ↔ coreToX` 与跨线稳定规范 id(#3642)均在 master,零重建。 +- **span 解析**:只对第 0 段判定的 NEW 元素集合做解析——它们可以散布在 body 任意体位(客户端压缩后摘要可能出现在 pos 0)。这是一个脏实现点:codec 的 ClusterCounter 按单次调用内次序编号 `_N`,解析散布子集必须复现「整数组调用」的编号上下文,否则出站字节漂移 = 前缀缓存碎掉。**散布子集编码的字节级回归钉是 P2 准入条件**(连续尾切片情形 v5 已证可行,散布子集是新情形)。 +- 媒体格(图片等)走既有 content CAS + `message_media` 引用,base64 不进行表。 +- P5:合法入口恰两个(全新会话 / 新身份发放轮),其余轮次禁止全量重解析。 + +### 4.2 第 2 段:改写 / 压缩(内核零改动) + +acp-kernel 保持一行不改:折叠判定(projectUnit 共享投影)、四条 prompts、nudge flat-50K 节拍、ref/msgid 永不复用契约。它对外的输出在本设计中定义为**结构化事件流**: + +``` +cast 首次铸入行(新会话首载) +append 新增行(新回合尾部) +update 行内形态变更(如 tool result 回填) +delete 覆盖/失效(active=0,不物理删) +compact 折叠压实(块吸收覆盖段,meta 更新覆盖映射) +``` + +事件寻址规则沿用 baseline:**只按内核管道证据**(block 覆盖区间 / ledgerSeq / 帧边界)定位受影响行,禁止任何模糊相似匹配;对不上 ⇒ 该轮不产事件、整轮走 legacy(§8 R2)。 + +**落表时序(以 #5643 附录二为准)**:第一现场 = 插件端点 `POST /__bili/plugin/tool`(src/server/admin.ts:797)在**当次请求内同步单事务**完成事件→表修改;下一轮模型请求只做核对。绕过端点的宿主(dsh native compaction 一类)由 checkpoint 帧签名 + rebase 检测喂**同一个**表修改例程兜底。端点到落表之间平面挂 `structural-pending` 标记;进程重启丢失标记 = 保守判死(fail-safe 方向,§8 R4)。 + +### 4.3 第 3 段:快照表应用(本提案主体) + +**提交原语 = 单事务**,内含三步: + +``` +upsert snap_rows(按证据寻址的行) + → 追加 snap_log(seq 严格 > max(log.max_seq, ckpt_seq)) + → (可选)折 snap_ckpt(吸收 log 载荷,更新 meta) +任一步抛错 ⇒ 整轮放弃,不留半态(不变量 i) +``` + +**不变量表**(草案 v0.1 起、review 补强后沿用): + +| # | 不变量 | +|---|--------| +| i | 单会话单写者:所有表修改只经这一个提交原语 | +| ii | log.seq 严格单调:> max(log.max_seq, ckpt_seq),跨压实不断号 | +| iii | S4:逐行校验和不符 ⇒ 该 sid 平面永久关死,整体回落旧路(红线不变) | +| iv | 行不物理删除:折叠只翻 active,pre/post 双形态保留供 decompress | +| v | post_json 恒为「改写层右侧的通用核形态」——缓存永远不允许 wire 字节 | + +**ckpt.meta_json 三件套**(F+C 裁定的合流):① 内核会话态(CompressionState:blocks/refs/deadRefs/nudge/tokenSnapshot/…)序列化 + 自身校验和——内核没有确定性「事件流→状态」重建入口,序列化随 checkpoint 走是唯一既零改内核又满足 P7(内存只是缓存)的方案;② **覆盖映射**(摘要 pos → 被覆盖 pos 区间)——delete/compact 载荷会被 ckpt 吸收,压实之后 decompress/GC 的唯一依据就是这份显式映射;③ S4 聚合校验和。 + +**失效行生命周期**(B 裁定建议):inactive 行保留至其覆盖块本身被归档/删除时一并物理清理;反对永不物理清(#3752 单调无界增长是 p1 红线)也反对全局 TTL(GC 窗口和解引用窗口不用额外对齐证明)。 + +### 4.4 第 4 段:出站映射 + +`mapEgress` = 存活行按 pos 排序取核 ++ 尾窗 `[K..T)` 本轮新解析的核,再过 `coreToX` 到目标协议族。前缀跨轮字节稳定 ⇒ 前缀缓存友好(这就是目标函数,不是巧合而是结构保证)。 + +- **双 carrier 分支**(会话粘滞绑定,两模式必须同时想清——本仓库惯犯病类 #377):pluginMode 抑制 tool+nudge 注入(agent 本地执行压缩);proxy 模式 acp_summary revoice 为 user 消息(systemToUser,保护头部 system 逐字稳定)。 +- **透明变化**(不是红条件):跨协议族切模型、maxDim 变、toolArmed 变——都只动映射输入。因为快照只存改写右侧的通用核形态,族切换天然透明。 +- 失步处理:join 链断 / 对不上 ⇒ 该轮整条 legacy 全量路径并重新盖戳(客户端一轮收敛)。 + +### 4.5 第 5 段:出站 + +**tripwire 运行期形态**(J 裁定建议,后被 owner 移除):原案低比例抽样把出站字节与 legacy 全量路径产物逐字比对(除 time attr),不一致摘回 legacy + 告警。**终局形态改为直接权威**:平面存内核自己打标的 processTurn 输出(不变量 v:post_json 恒为改写后核心形态),平面行本身就是旧后处理的输入 ⇒ 与旧出站逐字节一致是结构保证,无需逐轮比对;防「漂移状态下供出陈旧前缀 = 缓存投毒」的终局护栏改为折叠时 A/B 身份裁判(视图对平面活动行的身份多重集),对不上即该会话平面永久锁死、legacy 继续兜底。 + +**协议卫生**:pf-* / 信封类握手头是协议头不是 hop 头,转发上游前剥掉(UPSTREAM_HOP_HEADERS 白名单纪律);本提案引入的任何新握手头同样进白名单管理。灰度期全程 A/B 字节钉在场。 + +--- + +## 5. 存储 schema(最终 DDL 已在 #5901 落地为 schema v12) + +版本叠加规则:因 #5855 当时仍 open,#5901 在 v11 master 上直接把本 DDL 落为 schema **v12**(`src/storage/schema.ts` 的 `SNAPBASE_DDL_V12`);若 #5855 先落地,则由后落地者机械重编号(同一份 DDL,版本号 +1)。老库升级纯 `CREATE ... IF NOT EXISTS` 加性变更,无数据迁移。逐表读写地图(中/英):[storage-relations.zh.svg](../../storage-relations.zh.svg) / [storage-relations.en.svg](../../storage-relations.en.svg)。DDL(P0 落地的最终列型): + +```sql +-- L1 身份薄表(裁定 G 的承载体;baseline §4.1.1 的两列恢复为独立表) +CREATE TABLE IF NOT EXISTS l1_world( + sess_raw TEXT NOT NULL, -- 宿主会话 id 原值,禁截断(P9) + ledger_id TEXT NOT NULL, -- bili 规范 sid(沿用现有键空间,裁定 D) + witness_probe TEXT, -- echo 抽验见证样本(仅诊断流) + issued_at INTEGER NOT NULL, + armed_at INTEGER, -- 新身份发放的武装轮标记 + PRIMARY KEY (sess_raw) +); +CREATE INDEX IF NOT EXISTS idx_l1_ledger ON l1_world(ledger_id); + +-- 行表:正史本体 +CREATE TABLE IF NOT EXISTS snap_rows( + sid TEXT NOT NULL, + pos INTEGER NOT NULL, -- ledger 绝对位置,append-only 永不复用 + core_ids TEXT NOT NULL, -- 通用核元素 id 序列(json text) + pre_json TEXT, -- 改写前形态引用(CAS digest),裁定 A② 默认引用不内嵌 + post_json TEXT NOT NULL, -- 改写后通用核形态(唯一允许被缓存的形态) + tokens_est INTEGER, -- 估算 token(裁定 L:判决面增长数据源 + 计费共用) + active INTEGER NOT NULL DEFAULT 1, -- 0=被覆盖/失效(保留供 decompress,生命周期见 §4.3) + cover_ref TEXT, -- 覆盖它的摘要块 id(物理清理的生命周期锚) + updated_at INTEGER NOT NULL, + PRIMARY KEY (sid, pos) +); + +-- 事件日志(LSM log 层) +CREATE TABLE IF NOT EXISTS snap_log( + sid TEXT NOT NULL, + seq INTEGER NOT NULL, -- 严格 > max(log.max_seq, ckpt_seq) + kind TEXT NOT NULL, -- cast|append|update|delete|compact + payload TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (sid, seq) +); + +-- 检查点(LSM meta 层;吸收 log 载荷并显式保留覆盖映射) +CREATE TABLE IF NOT EXISTS snap_ckpt( + sid TEXT NOT NULL PRIMARY KEY, + ckpt_seq INTEGER NOT NULL, + meta_json TEXT NOT NULL, -- 内核态序列化 + 覆盖映射 + 自身校验和 + checksum TEXT NOT NULL, -- S4 聚合校验和 + updated_at INTEGER NOT NULL +); +``` + +**写模型假设**(H 裁定项):默认单进程假设(WAL 天然串行化跨进程写);若要多实例并发写,需附内存视图失效协议(常驻核数组/元数据缓存谁作废谁通知)——P0 契约里二选一钉死。 + +--- + +## 6. 读路径与消费方切换台账 + +**hydrate 双道**:绿道(新表可读且 S4 通过)从 snap_rows+meta 重建读视图;红道(开关关 / 无活跃块 / 表不可读 / 校验失败)原样走 kstate 旧路承接。切换按会话粒度、单向渐进,任何一道失步整轮回退 legacy——与 D-lane #3756 同纪律。 + +消费方逐个标注(P0 列全清单,此处为主干): + +| 消费方 | 处置 | +|--------|------| +| decompress(restore/decompress 工具路径) | P3 切到新表 pre/post 双形态 + 覆盖映射;pre 字节缺失时必须走审计显式诊断出口,不得静默空引用 | +| export | P3 切换;诚实输出纪律不变(#155:0-block 会话不许谎称「原始对话」) | +| 计费 | tokens_est 列共用;切读源不改变计费口径 | +| Web UI(sessions 面板 / acp_status uncompressed view) | acp_status 判决面(KDD#12 共享投影)输入源指到新表;面板新增表健康度指标(§11) | +| kv_store namespaces / notice store / persona fork(`\|sub:`) | 不触碰(独立机制,fork 记录自己后缀 id 进同一车道即可) | +| CCR(ccr_entries) | P0–P2 不碰;P3 单独评估是否并入或长期并存 | + +--- + +## 7. 降级红门枚举(命中任一 ⇒ 该轮/该会话整体回落 legacy 全量路径并重新盖戳) + +| # | 红条件 | 后果 | +|---|--------|------| +| R1 | S4 / 校验和坏 | 该 sid 平面**永久关死**,整体回落旧路(底材损坏不可恢复,红线不变) | +| R2 | 出站 join 链断(core_ids ↔ 行对不上,链回溯失败) | 该轮不产事件,整轮 legacy | +| R3 | cohort 宿主身份门故障(收不齐双信道 id) | 硬拒绝 + 零副作用 + 喂 breaker(baseline §3.2 原样) | +| R4 | structural-pending 未消费(含重启丢失标记) | 保守判死(fail-safe 方向) | + +**不是红条件的**:跨协议族切模型、maxDim 变化、toolArmed 变化、客户端压缩痕迹(纯忽略,只看增量)。把这组边界写成判定函数放进 lane-decisions 矩阵,每种组合正负成对断言(#3617 纪律)。 + +--- + +## 8. 与既有车道的共存路由(每种流量只有一个家) + +| 流量 | 归属 | 说明 | +|------|------|------| +| cohort 宿主(有双信道)全量重发会话 | **新底座** | 本提案服务的主对象 | +| 非 cohort(无 msgid)流量 | v5 watermark 快路径继续服务 | 与本底座同表不同车道,旗位独立;「顺序>hash」降级写入走同一套表 | +| anthropic wire divert 回合 | 增量 plugin-turn 车道维持原职 | dsh shim 逐回合二选一,互不双处理 | +| dsh native compaction 等绕过端点的宿主 | 第 2 段同一事件例程(checkpoint 帧签名 + rebase 检测喂入) | 见 §4.2 时序 | +| #3756 D-lane | 新底座就绪的车道**让位退役**(旗互斥) | 待裁定 K;退役判断放在 P3 | + +lane-decisions 矩阵:每个组合一行,正负成对(触发时触发、不该触发时不触发),新增行可追溯。 + +--- + +## 9. Rollout 与配置面(owner-gated,key 已在 #5901 定稿) + +- **两级嵌套开关**:master 旗 ⊃ lane 旗(lane 旗只在 master 开时有意义),config-file key 经 `src/knobs.ts`(env > file > default)解析;**不新增 env 旋钮**(env 稀缺面纪律 #2030)。落地 key:config-file `storage.snapBase.enabled`(master 旗)/ `storage.snapBase.shadowWrite`(lane 旗,master ⊃ lane),见[配置文档](../../configuration.zh-CN.md)。 +- **cohort 名单**:配置级声明(哪些 carrier/宿主声称具备双信道),初值待定(遗留 D4:shim/pi/opencode/codex/cc/gemini 六候选逐个核实)。 +- **Web「配置」页**三态快速开关同步出厂;i18n en/zh 对;registry-test 死键守护行;website/config-reference yaml 种子(若 ship 旋钮)。 +- 默认 OFF(落地状态:四个键全部出厂关;2026-11-08 的 100% soak 已在合并前翻回),按会话分桶(announcement-id 盐值桶之外的独立桶),P2 单车道灰度起步。 + +--- + +## 10. 观测面 + +- **决策日志**:统一词汇表(hit / miss / red: / degrade / resync / arm …),每行可 grep 可聚合;rollout 期间就是信心证据。 +- **`/__bili/status`**:attribution 字段扩展新底座维度(哪条链路在服务本会话、平面健康)。 +- **面板健康度**:snap 表行数 / log 深度 / ckpt lag(seq 落后量)/ GC 水位 / egress 降级原因计数(`/__bili/status` 的 `snaplane.egress`),进 sessions 面板或独立小卡。 +- **acp_status uncompressed view**:判决输入源指到新表行(KDD#12 共享投影的 apply/recommend/status 三方一致性在新表上重新钉测试)。 + +--- + +## 11. 测试矩阵 + +分期测试骨架(正负成对原则全程适用): + +| # | 测试 | 期 | +|---|------|----| +| T1 | 升级纯加性钉:老库升级前后既有查询行为逐字节不变;DROP 四张新表回滚演练 = 逐字节回到今天 | P0 | +| T2 | 事件事务单元:提交原语单事务原子性;seq 单调跨压实不断号;抛错无半态 | P0 | +| T3 | B1 陷阱钉 ×3 连续重启 oracle 比对(水合快照必须携带活跃世界原点 o;frontier 只能由 arm/reset 移动) | P1 | +| T4 | A/B 字节级:N 回合真实语料下新链路出站字节 == legacy 全量产物(除 time attr) | P1–P3 常开 | +| T5 | 散布 NEW 编码字节一致钉(ClusterCounter 复现) | P2 准入 | +| T6 | deleteSession 级联连删(含 l1_world)无孤儿 | P0/P1 | +| T7 | 覆盖映射压实时存活:compact 后 decompress/GC 依据仍可查 | P1 | +| T8 | GC 后 decompress 的诚实失败出口(pre 缺失 ≠ 静默空引用) | P3 | +| T9 | 长会话体量增长曲线模型(行数/log 深度/ckpt 尺寸,对照 #3752 红线)长跑验收 | P1 | +| T10 | 红门矩阵:R1–R4 各配「触发时回落」+「不该触发时不回落」成对行 | 各期随对应模块 | +| T11 | 真实客户端 e2e(pi / dsh × sglang)全链路 | P2 准入 | +| T12 | 双模式(pluginMode/proxyMode)carrier 分支双验证(§4.4) | P2 | + +CI:docker gate + 求解器无关;hermetic,端口 listen(0)(#360 纪律)。 + +--- + +## 12. 分期交付(每期为独立 PR,人审关卡标出) + +| 期 | 范围 | 行为面 | 验收 | +|----|------|--------|------| +| **P0** | DDL 四件套 + 提交原语 + 升级钉 + 回滚演练 + 事件事务单元测试 | **零行为变化**(纯铺底) | T1/T2/T6;写模型假设(H)与 key 命名(config-surface 章)在此期定稿——人审点① | +| **P1** | 影子写:新表与旧路并行落数据,读侧完全不切 | 零行为面 | T3/T4(影子比对)/T7/T9 长跑曲线;体量模型过红线检查——人审点②(信任重建关键期) | +| **P2** | 单车道 hydrate-egress 切源灰度(cohort 首个成员,建议 pi 或 dsh 之一) | 灰度会话读出站来自新表 | T5/T10/T11/T12 + A/B 身份裁判(折叠时)+ 直接权威出站;exit switch 一键回退 | +| **P3** | decompress/export/UI/计费逐一切换;CCR/persona 处置;#3756 退役判断;cohort 名单扩员 | 消费方逐个切换 | T8 + 全矩阵复跑;每期独立 PR | + +**E 裁定的 P2 部分被 owner 推翻**:owner 要求单 PR 完成(#5901 现承载 P0+P1+P2)。P2 落地为 `mapEgress`([src/server/snap-egress.ts](../../../src/server/snap-egress.ts)):快照平面就是**直接出站权威** —— 出站上下文直接从 S4 校验过的平面读,不做任何与旧字节的逐轮比对(tripwire 已被 owner 移除,理由同上:逐字节一致是结构保证)。写侧以事件级增量(cast/update/delete/compact,每次保存一个事务,与 legacy 提交同 tick)持久化内核看到的那份字节,压缩轮的新块同轮即可被平面读出;出站按 pos 顺序读平面行(存活活动行 + 每块存的锚点摘要,移到其覆盖区段的安全位置;被覆盖行靠平面自身 active 旗 / cover_ref 抑制,绝不靠 id 匹配),再跑本车道后处理链(absorb → attach → strip → drop)。降级回旧字节(一轮,下一轮重新决定):旗关、cohort 不含、平面死/缺、任一行不可解析、拼装异常;不可寻址视图条目或折叠时 A/B 身份对不上 ⇒ 该会话平面永久锁死。cohort = 协议车道(`storage.snapBase.egressCohorts`,默认四条全开),见[配置文档](../../configuration.zh-CN.md)。 + +--- + +## 13. 前置依赖与时序 + +1. **#5855 先合并**(人合):它是事务骨架的工程基底,也是 schema 版本的垫层。它 open/dirty 一天,P0 的 DDL 就没法干净地叠版本。(2026-11-08 现实:#5901 未等它——P0 DDL 在 v11 master 上直接落 v12;若 #5855 之后合并,由后落地者机械重编号,DDL 不动。) +2. #5842(身份采集)✓ 已在 master —— 第 0 段直接复用,无需等待。 +3. #5854(撤回)✓ 已 merged —— 确认 v6 符号零残留于 master。 +4. 裁定闭环(§14):F/G/B 三项决定表结构骨架,先行拍板;其余随 P0 PR 一并裁。 + +--- + +## 14. 待 owner 裁定汇总 + +已在线讨过的(A–E,沿用草案 v0.1 编号)+ 缺口分析新增(F–L)+ 遗留(D3–D5): + +| # | 事项 | 倾向 | 阻塞什么 | +|---|------|------|----------| +| A | 行内形态 | ② 只存 post,pre 引用既有 CAS;附加:被存活行引用的 pre 字节活得比该行久 | P0 DDL | +| **B** | 失效行保留策略 | 绑定覆盖块生命周期的压实清理;反对永不清(#3752 红线)与全局 TTL | P0 DDL/GC 设计 | +| C | 摘要落位 | ① 就地占覆盖段首 pos + ckpt meta 显式保留覆盖映射 | P0 DDL | +| D | 会话键空间 | 沿用现有 sid | P0 DDL | +| E | 本期范围 | P0+P1 收手交验收,P2 另开 PR 灰度 | 排期 | +| **F** | 内核状态居所 | ckpt.meta_json 携带 CompressionState 序列化 + 校验和(内核零改动、水合 O(1)) | P0/P1 核心 | +| **G** | L1 身份薄表 | 恢复 baseline §4.1.1 的 (sess_raw, ledger_id) 持久化(独立薄表 + 索引),复用 #5842 采集 | P0 DDL | +| **H** | 写模型 | 单进程假设(WAL 串行化)or 多进程 + 内存视图失效协议,P0 二选一定稿 | P0 | +| **I** | 硬门 vs 降级边界 | cohort 声称宿主故障拒绝 + 零副作用 + breaker;非 cohort 走顺序>hash 同表降级、会话粘滞、换模式=resync | P0/P2 路由 | +| **J** | tripwire | 运行期低比例抽样 + 不匹配即摘回 legacy + 告警 | P2(后被 owner 移除,改直接权威 + 折叠时 A/B 身份裁判,见 §4.5) | +| **K** | 与 #3756 / v5 共存 | 新底座车道上 #3756 让位退役(旗互斥);v5 继续服务非 cohort | P2/P3 | +| **L** | 行内 token 计账列 | tokens_est 入表:判决面增长数据源 + 计费共用 | P0 DDL | +| D3 | 故障响应形态 | HTTP 码 / 错误体 / 客户端重试约定(R3 硬拒绝的对端视角) | P2 | +| D4 | cohort 初始名单 | shim/pi/opencode/codex/cc/gemini 六候选逐个核实载体稳定性 | P0 配置初值 | +| D5 | 重编号语义 | 内容同 id 异:判新增双份 or 对级去重 | P0 语义 | + +--- + +## 15. 风险与诚实边界 + +- **「保元素数的前缀字节改写不可检测」**(继承 baseline 代价条款):代价就是跳过逐条复核的信任模型变化——需人工评审把关,不得自动合并(KDD §7.4 同类门槛)。 +- **多模态尾部进入缓存后该段历史永久降级 legacy 全量轮**(继承 KDD#15 诚实边界②):宁降级不偏离。 +- **marker-strip 计数随种子账本继承**,中部区从不重扫(同上③)。 +- 本提案所有「不变量」的最终裁判是运行期折叠时 A/B 身份裁判 + S4 抽验(tripwire 已按 owner 裁定移除,见 §4.5),不是纸面论证——所以 P1 影子期存在意义就是攒这份实证。 + +--- + +## 16. 参考号速查 + +#5643(本线起源+定位纠正)· #5854(整线撤回,merged)· #5855(工程基底 re-land,open)· #5842(身份采集 wiring,merged)· #3756(D-lane 快照渲染,将被 K 裁定处置)· #3752(存储无界增长 p1 红线)· #5403(OOM:lastMessages 驻留)· #2432(客户端压缩死螺旋,纯忽略策略的来源)· #3236(共享投影判据)· #3642(跨线稳定规范 id)· #3617(gate 双向断言)· #1104/#1106/#1107(fingerprint 白名单纪律)· #4491/#4834(msgid 身份层级)· #5539(schema v11 FK 子索引)· #155(诚实输出)· #360(listen(0) 端口纪律)· MESSAGE-IDENTITY.md(身份教义)· #5643 定位纠正评论: diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-current.png b/docs/proposals/snapshot-compression-foundation/assets/arch-current.png new file mode 100644 index 0000000000..f731b97a19 Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-current.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-current.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-current.svg new file mode 100644 index 0000000000..87cc009c8a --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-current.svg @@ -0,0 +1,174 @@ + + + + + + + + + 现状架构 · bili 上下文压缩代理(origin/master,存储 schema v11) + 四条车道各自为政 + kstate JSON 平行真源 —— 本提案要替换的格局(#5854 已把 v6 整线撤出 master) + + + + + 宿主/既有管道 + + 内核(不改) + + 统一存储 + + 结构性痛点 + + 默认主路径 + + 实验性灰度车道 + + + + + 宿主车道(今天已有身份采集) + dsh — shim / 增量信封 + pi — entry-walker 采集 + opencode — transform 钩子 + codex — responses wire + 其他宿主 — 裸代理客户端 + 请求头带 session id;插件车道 + 信封里逐元素带宿主 msgid + (#5842 采集器已就位) + + + + 入站适配器 prepare-* + anthropic | openai | responses | google + + + lane-decisions 分轨 + 按请求形状 × 会话粘滞绑定 + + + + legacy 全量重发路径(今天的默认主路径) + JSON.parse 全量 + openaiToCore 全量转换 + lastMessages AST 钉在内存跨轮驻留(#5403 OOM 生产者) + + + 增量 plugin-turn 车道 + anthropic wire divert · x-bili-plugin-turn 回合 + + + v5 watermark 快路径(5% 灰度) + fastpath-scan 结构扫描 · x-bili-pf-seq 戳握手 + + + D-lane 快照渲染 #3756(≈0% 灰度) + 服务端正史渲染出站前缀 · 失败整体回退 legacy + + + + acp-kernel 压缩内核 + (本提案零改动) + 折叠判定 projectUnit(共享投影) + 摘要生成 + 四条 prompts + nudge flat-50K 固定节拍 + ref/msgid 账本(id 永不复用) + CompressionState: + blocks / refs / deadRefs / ckpt + tokenSnapshot / nextBlockId … + 状态由谁水合、存哪 —— 见下 + + + + 出站映射 coreTo* + 四线协议重新序列化 + carrier 分支: + pluginMode:抑制 tool+nudge + proxy:acp_summary + revoice 为 user 消息 + pf-*/信封头永不出代理 + + 模型端点 + sglang / openai·responses + anthropic / google + + + + 统一 SQLite · schema v11(#5539 FK 子索引)—— 行级真源只有一半 + + + sessions + content + messages + blocks / dead_refs + refs / checkpoints + ccr_entries + kv_store + message_media / ext + + + conversations + session_ext + kstate JSON(hydration 字段清单) + + messages.msgId 列存的是 bili/内核规范 id —— 宿主 msgid 没有落点;sess_raw 无任何表承载。 + CompressionState 的存活字段由 src/state-hydration.ts mergeState() 单一定义,经 persist.ts 走 kstate JSON 存取。 + #5855(事务骨架 re-land)仍 open/冲突未解 —— 新四表 DDL 必须叠在它落地之后。 + + + + ⚠ 痛点① 状态三处住:进程内存 / 表 / kstate JSON + 双写分叉是刚撤回那条线的根病;#5403 OOM 来自 lastMessages 整包驻留。 + 重启靠 mergeState 水合拼装,拼错即降级或烧会话。 + + ⚠ 痛点② 身份不落盘:世界匹配永远冷启动 + 宿主 (sess_raw, msgid) 对 ingest 即剥掉,只在 in-flight 信封里活一轮。 + 「这元素见过没」跨进程/跨重启无法回答 ⇒ 每轮重探测、重武装。 + + ⚠ 痛点③ 快路径各自打补丁,无统一快照表 + v5 水位 / 增量车道 / D-lane 各管一段流量;缓存稳定性靠多层补丁堆叠, + 任何一层失步就是缓存投毒或整体回退。缺一个单一存储真源。 + + + + 一句话总结 → 本提案目标:原始字节 → 协议转换 → 改写(压缩) → 快照表应用 → 出站映射 → 出站,六段固定链每轮全走完,单一存储真源。 + 内核(折判定/摘要/prompts/ref 契约)零改动 · 有 session+msgid 的宿主全走此路 · 无 msgid 按 顺序>hash 降级写同一套表。 + 现状图对应物:左起入站 → 四车道(替代为本链第 0–1 段的身份化分轨)→ 内核(第 2 段,不动)→ 出站(第 4–5 段);新增的第 0 段(L1 身份)与第 3 段(快照表)在现状中不存在。 + + + + + + + + + + + + + + + + + + + + + + + + + 状态读写 = kstate JSON + 表 双通道(分叉点) + diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-migration.png b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.png new file mode 100644 index 0000000000..82914bb01d Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-migration.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.svg new file mode 100644 index 0000000000..69d51ef404 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.svg @@ -0,0 +1,35 @@ + + + + + + + + + + + 组件迁移总览 · 「现状」的十块组件在新架构下去向何处 + 同一条链路、同一个内核 —— 多数组件原样保留或收窄职责;变化集中在「状态住哪、身份记在哪」两处收敛到单一存储真源 + 保留 / 复用 / 扩展 · 零改动替换 / 收敛角色变更 / 域收窄让位 · 退役(待裁)红 = 痛点 绿 = 解决锚点 + + 今天 · origin/master(storage schema v11,四车道各自为政) + 底座落地后(本提案 · 六段固定链) + 宿主车道 msgid 采集#5842 四载体已就位(shim/pi/opencode/codex…)保留·复用第 0 段 身份入册输入(采集零改)双信道身份原样进链;新下游 l1_world 落盘 (sess_raw↔ledger_id),跨进程/跨重启恒可答(P8/P9)入站适配器 prepare-*{openai,anthropic,responses,google}ToCore 矩阵保留·复用第 1 段 协议转换复用wire 矩阵不动;span 解析只做身份差集 NEW 元素(任意体位,含 pos 0);散布编码字节等价钉 = P2 准入lane-decisions 分轨请求头 session id + 会话粘滞绑定保留·扩展路由矩阵 · cohort 判定成行每组合一条正负成对行(#3617 gate 双向断言);cohort 边界(故障硬拒绝 vs 顺序>hash 降级)全进判定函数legacy 全量重发路径今天的默认主路径(JSON.parse 全包 + 钉存 AST)角色变更兜底安全网 · 整轮回落通道非 cohort 仍是主路径;cohort 会话命中 R1–R4 任一红门 ⇒ 该轮整体走它并重新盖戳,客户端一轮收敛增量 plugin-turn 车道anthropic-wire divert · x-bili-plugin-turn 回合保留·原职原职维持 · 按请求形状分轨anthropic-wire divert 回合照旧;dsh shim 逐回合与快照底座二选一,服务端不打混合标v5 watermark 快路径5% 灰度 · fastpath-scan 结构扫描保留·域收窄域收窄 · 只服务非 cohort管解析成本、不管状态真源;与快照底座同表不同车道、旗位独立,互不双处理D-lane 快照渲染 #3756≈0% 灰度 · 正史渲染出站前缀让位·退役让位 · 旗位互斥(待裁定 K)在新底座车道上与 snap 三件套旗位互斥运行;退役与否由 P3 依影子期数据裁定⚠ 待裁定 Kacp-kernel 压缩内核projectUnit / prompts / nudge flat-50K / ref 账本零改动零改动 · I/O 契约扩展为五事件流cast / append / update / delete / compact;折叠判定、四条 prompts、flat-50K nudge、id 永不复用合同一行不动出站映射 coreTo*双 carrier 分支 · pf-/信封头永不出代理替换mapEgress + tripwire 抽样存活行按 pos 排序 ++ 本轮尾窗 [K..T) 新解析;carrier 分支逻辑照搬(#377 教训);抽样不一致 ⇒ 立即摘回 legacy + 告警状态存储:kstate JSON + 表族v11 · hydration 水合拼装(mergeState 单一事实源)收敛单一存储真源 · 三件套 + L1snap_rows/snap_log/snap_ckpt 单事务提交(不变量 i–v);kstate 降为红道回退源,绿道读三件套 + + 三个结构性痛点 → 一一对应的解决锚点 + 痛点① 状态三处住进程内存 + 表 + kstate 双写分叉;#5403 OOM 钉存来源解决:ckpt.meta_json 单一真源(F 裁定)内存降为缓存;绿道读三件套 / 红道随时承接 kstate 旧路,切换按会话单向、失步整轮回退痛点② 宿主身份不落盘(sess_raw,msgid) ingest 即剥,只在 in-flight 信封活一轮解决:L1 身份薄表持久化(G 裁定)复用 #5842 载体采集器;首见即记、永不截断删除(P8/P9);裁决三分支 echo 抽验 / 复合探测 / 发放新身份痛点③ 快路径各自打补丁v5/增量/D-lane 各管一段流量,缓存稳定性靠补丁叠加解决:单事务提交原语 + 四件套保护三步原子:upsert rows → append log(seq 严格单调) → compact 可选;tripwire / S4 校验和 / 见证要求 / 审计抽验 + \ No newline at end of file diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-stages.png b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.png new file mode 100644 index 0000000000..bcf9d58d0e Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-stages.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.svg new file mode 100644 index 0000000000..027c6d81e0 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.svg @@ -0,0 +1,271 @@ + + + + + + + + + 目标架构 · 快照表式压缩底座(六段固定链,单一存储真源) + 每轮全走完:原始字节 → 协议转换 → 改写(压缩) → 快照表应用 → 出站映射 → 出站 | 内核零改动 | 任何疑点该轮回落 legacy 路径(降级,不猜) + + + + 入站 · 宿主车道与分轨(零改动 —— #5842 msgid 采集已就位;分轨走既有 lane-decisions) + dsh shim + pi entry-walker + opencode transform 钩子 + codex responses 车道 + 其他裸代理客户端 + prepare-* 适配器(wire 矩阵,现成) + + + + + + 底座命中的请求 ⇒ 进入下方六段固定链 + cohort:双信道身份(sess_raw+msgid[]) · 非 cohort:顺序 > hash 粘滞降级,同表写入(见第 0 段) + + 增量 plugin-turn 形状回合 + anthropic wire divert · 维持原职(见下方共存带) + + 其余流量(v5 形状会话 / 底座灰度关) + watermark 快路径或 legacy 全量路径 · 现状机制不变 + 分轨判定函数不新增:提案只在 lane-decisions 加一行「底座命中」正负成对断言行(#3617);任何红门命中 ⇒ 该轮回落 legacy 全量并重新盖戳。 + + + + + + 0 + 入站 · 身份入册 + 新增段(现状不存在) + + 原始字节体(全量重发历史) + 双信道身份: + 请求头 sess_raw + + 逐元素 msgid[] 信封 + #5842 载体 resolver 采集复用 + + L1 世界匹配(新薄表) + l1_world: sess_raw ↔ ledger_id + 全局复合索引,原值禁截断(P9) + 裁决三分支: + echo 直中 + 抽验 + 复合探测 + 发放新身份(+一轮武装) + + 非 cohort 宿主: + 顺序 > hash 粘滞降级,同表写入 + (pos 由序位发号;换模式=resync) + P8 id 永生:首见即记、跨重启恒可答 + + + + + 1 + 协议转换 → 通用核 + 现成复用(master 已有) + + wire 矩阵 *ToCore + openai/anthropic/responses/google + + 跨线稳定规范 id(#3642) + + span 解析仅 NEW 元素 + 身份差集散布任意体位 + (客户端压缩后摘要可能在 pos 0) + + ClusterCounter 复现钉 + 散布子集解析必须复现整数组 + 调用的 _N 编号上下文 + ⇒ 字节级回归钉(P2 准入) + + 媒体格 → CAS 引用 + base64 不进行表(message_media) + + 合法入口恰两个: + 全新会话 / 新身份发放轮 + (P5 禁全量重解析) + + + + + 2 + 改写 / 压缩 + 内核零改动 + + acp-kernel(不动一行) + projectUnit 共享投影判定 + 摘要 prompts / nudge flat-50K + ref/msgid 账本 id 永不复用 + + 产出 = 结构化事件 + append / update / delete / compact + 按内核管道证据寻址(block 覆盖/ + ledgerSeq/帧边界),禁模糊相似 + + 时序(附录二) + 第一现场 POST /__bili/plugin/tool + 当次请求内同步单事务落表 + 下一轮模型请求只做核对 + 绕过宿主(dsh native compaction) + ⇒ checkpoint 帧签名+rebase 检测 + 喂同一例程兜底 + + 窗口期挂 structural-pending 红门 + + + + + 3 + 快照表应用 + 新增核心(本提案主体) + + 提交原语 = 单事务 + upsert snap_rows → 追加 snap_log + (seq 严格 > max(log.max_seq, + ckpt_seq)) → 可选折 ckpt + 抛错 = 整轮放弃,不落半态 + + snap_rows + snap_log + snap_ckpt + l1_world(L1) + + 行键 = ledger 绝对 pos + append-only 永不复用 + ckpt.meta_json 三件套 + 内核态序列化(F) + 覆盖映射(C) + + S4 校验和 + + 失效行 lifecycle(B) + inactive 保留双形态供 decompress; + 物理清绑覆盖块生命周期(#3752 红线) + + + + + 4 + 出站映射 + mapEgress(读侧重建) + + 存活行按 pos 排序 + ++ 尾窗 [K..T) 本轮新解析 + 前缀跨轮字节稳定 + ⇒ 前缀缓存友好(目标函数) + + 双 carrier 分支(粘滞) + pluginMode:抑制 tool+nudge 注入 + (agent 本地执行压缩) + proxy 模式:acp_summary + revoice 为 user 消息 + + 不是红条件的变化: + 跨协议族切模型 / maxDim 变 / + toolArmed 变 = 只动映射输入 + (快照只存改写右侧通用核形态) + + 失步处理 + 链断/join 不上 ⇒ 该轮整条走 + legacy 全量路径并重新盖戳 + + + + + 5 + 出站 + tripwire 守卫 + + tripwire 抽样比对 + 出站字节 vs legacy 全量产物 + (除 time attr),低比例运行期采样 + 一致 → 放行上游 + + 不一致(J): + 该会话立即摘回 legacy + 告警 + 防漂移状态供出陈旧前缀=缓存投毒 + + + 模型端点 + sglang / openai·responses + anthropic / google + + 协议卫生 + pf-*/信封握手头永不出到模型端点 + (UPSTREAM_HOP_HEADERS 白名单纪律) + + 灰度期全程 A/B 字节钉在场 + + + + + + + + + + + + + + 降级红门枚举(任一命中 ⇒ 该轮/该会话整体回落 legacy 全量路径并重新盖戳 —— 降级不猜) + + R1 S4 / 校验和坏 + 该 sid 平面永久关死,整体回落旧路 + (底材损坏不可恢复,红线不变) + + R2 出站 join 链断 + core_ids ↔ 行对不上(链回溯失败) + ⇒ 该轮不产事件,整轮 legacy + + R3 cohort 宿主身份门故障(I) + 声称有双信道的宿主某轮收不齐 id + ⇒ 硬拒绝 + 零副作用 + 喂 breaker + + R4 structural-pending 未消费 + 压缩事件到落表之间窗口未闭合;重启丢失 + 标记 = 保守判死(fail-safe 方向) + 注:跨协议族切模型 / maxDim / toolArmed 变化【不是】红条件 —— 它们是每轮映射输入。 + + + + 共存路由(每种流量只有一个家,互不双处理) + v5 watermark 快路径 → 继续服务非 cohort(无 msgid)流量;与本底座同表不同车道,旗位独立。 + 增量 plugin-turn 车道 → anthropic wire divert 回合维持原职(dsh shim 逐回合二选一)。 + 绕过宿主(dsh native compaction 等) → checkpoint 帧签名 + rebase 检测喂第 2 段同一事件例程。 + #3756 D-lane → 新底座就绪的车道让位退役(旗互斥,待裁定 K);lane-decisions 矩阵每组合配正负成对断言。 + + 回滚与灰度 + + 回滚 = DROP 四表 + 关旗 = 逐字节回到今天(无迁移负担,老库升级纯 CREATE IF NOT EXISTS) + rollout:master ⊃ lane 两级嵌套开关(config-file key,knobs env>file>default, + 不新增 env 旋钮);cohort carrier 名单配置级声明;Web「配置」页三态开关同步出厂。 + 默认 OFF,按会话分桶;P2 单车道灰度起 A/B 钉常开。 + + + + 新增段 + 现成复用 + 内核(零改动) + 存储(三表+L1) + 红门/降级 + 回滚保证 + + 基线不变量:P5 禁全量重解析(合法入口恰两个) · P6 持久不重建(重启只水合,无武装轮) · P8 (sess_raw,ledger_id) 首见即记、永不删改 · P9 身份层原值禁截断(sig/fingerprint 只进诊断流) · tripwire/S4/见证/审计抽验四件套保护。 + 目标函数排序:稳定性 / 扩展性 / 通用性优先,性能是副产品 —— 先问流转对不对,不问快不快。 + diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-target.png b/docs/proposals/snapshot-compression-foundation/assets/arch-target.png new file mode 100644 index 0000000000..68f7bf511a Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-target.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-target.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-target.svg new file mode 100644 index 0000000000..51a0c1d4b3 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-target.svg @@ -0,0 +1,189 @@ + + + + + + + + + 目标架构 · 快照表式压缩底座(全貌 —— 与现状图逐位对齐,可直接对照) + 灰 = 零改动 | 青 = 新增段/表 | 琥珀 = 角色变更 | 红虚线框 = 兜底安全网 | 内核零改动 | 任何疑点该轮回落 legacy(降级,不猜) + + + + + 宿主/既有管道 + + 零改动 + + 新增段/表 + + 角色变更 + + 兜底安全网 + + 统一存储 + + + + + 宿主车道(零改动) + dsh — shim / 增量信封 + pi — entry-walker 采集 + opencode — transform 钩子 + codex — responses wire + 其他宿主 — 裸代理客户端 + 请求头带 session id;插件车道 + 信封里逐元素带宿主 msgid + #5842 采集已就位 —— 原样复用 + + + + 入站适配器 prepare-* + anthropic | openai | responses | google(现成复用) + + + lane-decisions 分轨 + 按请求形状 × 会话粘滞绑定 + +一行「底座命中」正负成对断言(#3617) + + + + 快照表底座 · 六段固定链(新主路径) + 身份入册 → 协议转换 → 改写 → 快照表 → 出站映射 + cohort 双信道身份 · 非 cohort 顺序>hash 粘滞降级同表 + + + 增量 plugin-turn 车道(共存 · 维持原职) + anthropic wire divert · dsh shim 逐回合二选一 + + + v5 watermark 快路径(共存 · 旗位独立) + 继续服务非 cohort 流量的解析成本 + + + D-lane 快照渲染 #3756(让位 · 待裁定 K) + 新底座就绪后车道让位退役 · 旗互斥 + + + legacy 全量重发路径(降为兜底安全网) + 任何红门 / 失步命中 ⇒ 该轮整条回到这里并重新盖戳 + + + + acp-kernel 压缩内核 + (本提案零改动 —— 不动一行) + 折叠判定 projectUnit(共享投影) + 摘要生成 + 四条 prompts + nudge flat-50K 固定节拍 + ref/msgid 账本(id 永不复用) + CompressionState(字段集不变): + blocks / refs / deadRefs / ckpt … + 序列化入 snap_ckpt.meta_json 单一水合源 + 状态存哪、谁水合 —— 见下存储带 + + + + 出站映射 mapEgress + 存活行按 pos 排序 ++ 尾窗 [K..T) + 前缀跨轮字节稳定 ⇒ 缓存友好 + carrier 分支粘滞(pluginMode/proxy) + pf-*/信封头永不出代理 + + 模型端点 + sglang / openai·responses + anthropic / google + + + + 统一 SQLite · schema v11 底座 —— 既有 10 表写语义不变;新增四表纯加性(#5855 落地后取新版本) + + + sessions + content + messages + blocks / dead_refs + refs / checkpoints + ccr_entries + kv_store + message_media / ext + + + conversations + session_ext + kstate JSON(仅 legacy 路径保留) + + + + l1_world · sess_raw↔ledger_id + snap_rows · 绝对 pos append-only + snap_log · 事件流 seq + snap_ckpt · meta 三件套(F+C+S4) + + 新增四表 = CREATE IF NOT EXISTS 纯加性、零迁移;回滚 = DROP 四表 + 关旗 = 逐字节回到今天。 + messages/blocks/dead_refs 成为只读审计源(S4 校验和基线);persona fork / conversations / kv_store 不动。 + #5855(事务骨架 re-land)仍 open —— P0 DDL 必须叠在它之后;P1 影子写期行为零变化。 + + + + ✔ 零改动区 —— 内核 / 宿主 / 适配器 / 审计源 + acp-kernel 不动一行;#5842 采集、prepare-*、carrier 规则原样复用; + 既有 10 张统一存储表写语义不变(messages/blocks 只作审计源)。 + + + 新增区 —— 四薄表 + 两个新段 + L1 身份薄表(sess_raw→ledger_id,P8 id 永生) + snap 三件套(rows/log/ckpt); + 新段 = 入站身份入册 + 快照表应用(内部细节见分阶段展开图)。 + + ⇄ 角色变更区 —— 三处降级,其余维持原职 + legacy 全量 → 兜底安全网 · D-lane #3756 → 让位(待裁定 K) · kstate JSON → legacy 专属; + v5 / 增量 plugin-turn 维持原职不变,每种流量只有一个家、互不双处理。 + + + + 一句话影响范围 = lane-decisions +1 行 · 一条新链(六段底座) · 四张新表 · 三处角色变更 —— 其余组件与现状图同形同位。 + 降级红门 R1–R4:S4 坏=平面永久关死 · join 断=该轮不产事件 · cohort 身份门故障=硬拒绝+零副作用 · structural-pending 未消费=保守判死(细节见分阶段展开图)。 + rollout:master ⊃ lane 两级嵌套开关(config-file key,不新增 env 旋钮,默认 OFF 按会话分桶) · 灰度期 A/B 字节钉常开 · Web「配置」页三态开关同步出厂。 + + + + + + + + + + + + + + + + + + + + + + + + + + + 状态读写 = snap_ckpt.meta_json 单一真源(kstate 双通道分叉点消除) + diff --git a/docs/proposals/snapshot-compression-foundation/assets/gen-migration.mjs b/docs/proposals/snapshot-compression-foundation/assets/gen-migration.mjs new file mode 100644 index 0000000000..55d270cab0 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/gen-migration.mjs @@ -0,0 +1,88 @@ +import fs from 'node:fs'; +const W = 1560, H = 1050; +const rows = [ + { l:['宿主车道 msgid 采集','#5842 四载体已就位(shim/pi/opencode/codex…)'], f:'保留·复用', c:'g', r:['第 0 段 身份入册输入(采集零改)','双信道身份原样进链;新下游 l1_world 落盘 (sess_raw↔ledger_id),跨进程/跨重启恒可答(P8/P9)'] }, + { l:['入站适配器 prepare-*','{openai,anthropic,responses,google}ToCore 矩阵'], f:'保留·复用', c:'g', r:['第 1 段 协议转换复用','wire 矩阵不动;span 解析只做身份差集 NEW 元素(任意体位,含 pos 0);散布编码字节等价钉 = P2 准入'] }, + { l:['lane-decisions 分轨','请求头 session id + 会话粘滞绑定'], f:'保留·扩展', c:'g', r:['路由矩阵 · cohort 判定成行','每组合一条正负成对行(#3617 gate 双向断言);cohort 边界(故障硬拒绝 vs 顺序>hash 降级)全进判定函数'] }, + { l:['legacy 全量重发路径','今天的默认主路径(JSON.parse 全包 + 钉存 AST)'], f:'角色变更', c:'a', r:['兜底安全网 · 整轮回落通道','非 cohort 仍是主路径;cohort 会话命中 R1–R4 任一红门 ⇒ 该轮整体走它并重新盖戳,客户端一轮收敛'] }, + { l:['增量 plugin-turn 车道','anthropic-wire divert · x-bili-plugin-turn 回合'], f:'保留·原职', c:'g', r:['原职维持 · 按请求形状分轨','anthropic-wire divert 回合照旧;dsh shim 逐回合与快照底座二选一,服务端不打混合标'] }, + { l:['v5 watermark 快路径','5% 灰度 · fastpath-scan 结构扫描'], f:'保留·域收窄', c:'a', r:['域收窄 · 只服务非 cohort','管解析成本、不管状态真源;与快照底座同表不同车道、旗位独立,互不双处理'] }, + { l:['D-lane 快照渲染 #3756','≈0% 灰度 · 正史渲染出站前缀'], f:'让位·退役', c:'gr', tag:'待裁定 K', r:['让位 · 旗位互斥(待裁定 K)','在新底座车道上与 snap 三件套旗位互斥运行;退役与否由 P3 依影子期数据裁定'] }, + { l:['acp-kernel 压缩内核','projectUnit / prompts / nudge flat-50K / ref 账本'], f:'零改动', c:'k', r:['零改动 · I/O 契约扩展为五事件流','cast / append / update / delete / compact;折叠判定、四条 prompts、flat-50K nudge、id 永不复用合同一行不动'] }, + { l:['出站映射 coreTo*','双 carrier 分支 · pf-/信封头永不出代理'], f:'替换', c:'i', r:['mapEgress + tripwire 抽样','存活行按 pos 排序 ++ 本轮尾窗 [K..T) 新解析;carrier 分支逻辑照搬(#377 教训);抽样不一致 ⇒ 立即摘回 legacy + 告警'] }, + { l:['状态存储:kstate JSON + 表族','v11 · hydration 水合拼装(mergeState 单一事实源)'], f:'收敛', c:'i', r:['单一存储真源 · 三件套 + L1','snap_rows/snap_log/snap_ckpt 单事务提交(不变量 i–v);kstate 降为红道回退源,绿道读三件套'] }, +]; +const pains = [ + ['痛点① 状态三处住','进程内存 + 表 + kstate 双写分叉;#5403 OOM 钉存来源','解决:ckpt.meta_json 单一真源(F 裁定)','内存降为缓存;绿道读三件套 / 红道随时承接 kstate 旧路,切换按会话单向、失步整轮回退'], + ['痛点② 宿主身份不落盘','(sess_raw,msgid) ingest 即剥,只在 in-flight 信封活一轮','解决:L1 身份薄表持久化(G 裁定)','复用 #5842 载体采集器;首见即记、永不截断删除(P8/P9);裁决三分支 echo 抽验 / 复合探测 / 发放新身份'], + ['痛点③ 快路径各自打补丁','v5/增量/D-lane 各管一段流量,缓存稳定性靠补丁叠加','解决:单事务提交原语 + 四件套保护','三步原子:upsert rows → append log(seq 严格单调) → compact 可选;tripwire / S4 校验和 / 见证要求 / 审计抽验'], +]; +const col = { g:['#dcfce7','#16a34a'], i:['#e0e7ff','#4f46e5'], a:['#fef3c7','#b45309'], gr:['#f1f5f9','#64748b'], k:['#ffffff','#16a34a'], rd:['#fee2e2','#b91c1c'] }; +const mark = { g:'arrG', i:'arrI', a:'arrA', gr:'arrGr', k:'arrG' }; +const fcol = { g:'#15803d', i:'#4338ca', a:'#b45309', gr:'#475569', k:'#15803d' }; +let s = ''; +const rowH = 52, pitch = 64, y0 = 158; +rows.forEach((row, i) => { + const y = y0 + pitch * i; + const [fill, stroke] = col[row.c]; + const sw = row.c === 'k' ? ' stroke-width="2.5"' : ''; + s += ``; + s += `${row.l[0]}`; + s += `${row.l[1]}`; + s += ``; + s += `${row.f}`; + s += ``; + s += `${row.r[0]}`; + s += `${row.r[1]}`; + if (row.tag) s += `⚠ ${row.tag}`; +}); +let p = ''; +pains.forEach((pn, j) => { + const y = 846 + j * 56; + p += ``; + p += `${pn[0]}`; + p += `${pn[1]}`; + p += ``; + p += ``; + p += `${pn[2]}`; + p += `${pn[3]}`; +}); +const legend = [[30,'g','保留 / 复用 / 扩展 · 零改动'],[250,'i','替换 / 收敛'],[380,'a','角色变更 / 域收窄'],[530,'gr','让位 · 退役(待裁)'],[710,'rd','红 = 痛点 绿 = 解决锚点']] + .map(([x,k,txt])=>`${txt}`).join(''); +const svg = ` + + + + + + + + + + 组件迁移总览 · 「现状」的十块组件在新架构下去向何处 + 同一条链路、同一个内核 —— 多数组件原样保留或收窄职责;变化集中在「状态住哪、身份记在哪」两处收敛到单一存储真源 + ${legend} + + 今天 · origin/master(storage schema v11,四车道各自为政) + 底座落地后(本提案 · 六段固定链) + ${s} + + 三个结构性痛点 → 一一对应的解决锚点 + ${p} +`; +fs.writeFileSync('arch-migration.svg', svg); +console.log('written', svg.length, 'bytes; canvas', W + 'x' + H, '; last content y ≈ 1002'); diff --git a/docs/storage-relations.en.svg b/docs/storage-relations.en.svg new file mode 100644 index 0000000000..31ddf38a3c --- /dev/null +++ b/docs/storage-relations.en.svg @@ -0,0 +1,65 @@ + + + +Storage relationships — unified SQLite (schema v12 · PR #5901 snapshot-table foundation) +Per-table card: what it stores / written: when / read: by whom · red = write · blue = read · dashed = P2 (proposal end-state, not wired in this PR) + +Host +B1 full resend + +Inbound adapters +prepare-* · protocol conversion + +① ingestSession +legacy single-transaction commit + +② shadow-write snapShadowSync +same tick · own tx · flag-gated + +acp-kernel compression core +zero changes (hard constraint) + +⑤ egress mapping +today: legacy view → coreTo* + +Model endpoint +sglang / openai / anthropic + + + + + + +Archive layer · ingest bookkeeping (① writes · ②⑤ read)conversationssession_id ↔ sess_raw (host conversation id)written: once, first sight of sessionread: L1 identity registry, legacy viewsessionssession meta + kstate_json (kernel state) + last_ingested_seqwritten: upsert every ingestread: restart hydration, ckpt meta, statusmessages(session_id, seq): msg_id / identity / content_hash / retired_atwritten: every ingest (full resend / delta)read: plane derivation, legacy view, A/B checkcontent (CAS)hash → bytes + refcount (dedup · refcounting)written: on new bytes at ingestread: byte restore (derivation / A/B)message_mediamessage ↔ media ref mappingwritten: ingestread: media restore +Kernel ledger · acp-kernel (unchanged)blocksfolded blocks: summary + coverage range (start_seq, end_seq)written: compression kernelread: plane coverage mapping, legacy viewrefs / dead_refsrefcount ledger and dead refswritten: kernel / ingestread: kernel hydrationccr_entriescached compressed responseswritten: kernelread: kernelcheckpointslegacy checkpointswritten: kernelread: kernel +Plane · four new tables (#5901 · schema v12)l1_worldsess_raw → ledger_id + witness probe + issued/armwritten: issue (first sight) / arm (stable probe)read: per-round identity (issue/arm/echo/hit/wait)snap_rows(sid,pos): core_ids + pre/post dual form + active + cover_refwritten: cast/append/update/delete/compact per roundread: P2 egress, A/B check, GCsnap_log(sid,seq): event + post-apply S4 aggregate (strictly monotonic)written: one row per eventread: rebuild anchor, R1 verificationsnap_ckpt(sid): ckpt_seq + meta (kstate / coverage map / S4)written: on fold (log depth ≥128 or compact)read: rebuild anchor, seq lower bound, status +Others (not touched by this PR)session_extinc:* incremental lane / pf:state v5 watermarkwritten: each laneread: lane features; plane skips inc: sessionskv_storegeneric KVwritten: variousread: various + + +① write + + +② read: messages / content / sessions.kstate / conversations + blocks + +② write + + +⑤ legacy egress (default today) + + +⑥ plane egress mapEgress (P2) + +③ fold +③ fold (in-plane)C.snap_log → C.snap_ckpttrigger: log depth ≥128 or a compactS4 anchor moves into ckpt, log truncated +④ state hydration (restart)A.sessions.kstate_json → CompressionStatedefined singly by mergeState()today kstate is authoritative; the plane ckpt is a mirror +⑦ deleteSessioncascades over ALL tables of A + B + Cl1_world deleted by ledger_idno orphans left (pinned by T6) +⑧ GC (per-save, opportunistic)when a B.blocks covering block vanishes→ C.snap_rows physical row deleteivar release valve; rows never physically deleted in steady state (iv) +⑨ /__bili/status .snaplane (ops)reads C four-table counts + dead-session listsnapplaneStatusWire()zero behavior, observability only + + + + + + +This PR (P0+P1) = shadow flow: legacy tables as usual → the four new tables are write-only, never read for egress · proposal end-state (P2) = plane becomes the sole egress read source, absorbs kstate hydration, persists identity · rollback = DROP the four tables + unflag + \ No newline at end of file diff --git a/docs/storage-relations.zh.svg b/docs/storage-relations.zh.svg new file mode 100644 index 0000000000..90e4eee1a5 --- /dev/null +++ b/docs/storage-relations.zh.svg @@ -0,0 +1,65 @@ + + + +存储关系图 — 统一 SQLite(schema v12 · PR #5901 快照表底座) +每张表卡:存什么 / 写:何时写 / 读:谁读 · 红=写 · 蓝=读 · 虚线=P2(提案终态,本 PR 未接) + +宿主 +B1 全量重发 + +入站适配器 +prepare-* · 协议转换 + +① ingestSession +legacy 单事务提交 + +② 影子写 snapShadowSync +同 tick · 独立事务 · 旗门控 + +压缩内核 acp-kernel +零改动(硬约束) + +⑤ 出站映射 +今天:legacy 视图 → coreTo* + +模型端点 +sglang / openai / anthropic + + + + + + +归档层 · 入站记账(① 写 · ②⑤ 读)conversationssession_id ↔ sess_raw(宿主会话 id)写:会话首现记一行读:L1 身份入册、legacy 视图sessions会话 meta + kstate_json(内核态)+ last_ingested_seq写:每轮 ingest upsert读:重启水合、ckpt meta、statusmessages(session_id, seq):msg_id / identity / content_hash / retired_at写:每轮 ingest(全量重发 / delta)读:平面推导、legacy 视图、A/B 比对content(CAS)hash → 字节 + refcount(去重 · 引用计数)写:ingest 见新字节读:字节还原(推导 / A/B)message_media消息 ↔ 媒体引用映射写:ingest读:媒体还原 +内核账本 · acp-kernel(零改动)blocks折叠块:摘要 + 覆盖区间(start_seq, end_seq)写:压缩内核读:平面覆盖映射、legacy 视图refs / dead_refs引用计数账本与死引用写:内核 / ingest读:内核水合ccr_entries缓存压缩响应写:内核读:内核checkpointslegacy checkpoint写:内核读:内核 +平面 · 新四表(#5901 · schema v12)l1_worldsess_raw → ledger_id + witness 指纹 + issued/arm写:issue(首现)/ arm(指纹稳定)读:每轮身份入册(issue/arm/echo/hit/wait)snap_rows(sid,pos):core_ids + pre/post 双形态 + active + cover_ref写:每轮 cast/append/update/delete/compact读:P2 出站、A/B 比对、GCsnap_log(sid,seq):事件 + 应用后 S4 聚合(seq 严格单调)写:每事件一行读:重建锚点、R1 校验snap_ckpt(sid):ckpt_seq + meta(kstate / 覆盖映射 / S4)写:折叠时(log 深度 ≥128 或 compact)读:重建锚点、seq 下界、status +其他(本 PR 不涉及)session_extinc:* 增量车道 / pf:state v5 水位写:各车道读:车道功能;平面跳过 inc: 会话kv_store通用 KV写:各功能读:各功能 + + +① 写 + + +② 读:messages / content / sessions.kstate / conversations + blocks + +② 写 + + +⑤ legacy 出站(当前默认) + + +⑥ 平面出站 mapEgress(P2) + +③ 折叠 +③ 折叠(平面内)C.snap_log → C.snap_ckpt触发:log 深度 ≥128 或发生 compactS4 锚点移入 ckpt,log 截断 +④ 状态水合(重启)A.sessions.kstate_json → CompressionState经 mergeState() 单一定义今天 kstate 是权威,平面 ckpt 只是镜像 +⑦ deleteSession(删会话)级联删 A + B + C 全部表l1_world 按 ledger_id 删不留孤儿行(T6 钉死) +⑧ GC(每保存顺手)B.blocks 覆盖块消失时→ C.snap_rows 物理删该行ivar 释放阀;平时行永不物理删(iv) +⑨ /__bili/status .snaplane(运维)读 C 四表计数 + 死会话列表snapplaneStatusWire()零行为,纯可观测 + + + + + + +本 PR(P0+P1)= 影子流:旧表照常 → 新四表只写不读不出站 · 提案终态(P2)= 平面做出站唯一读源、收编 kstate 水合、身份落盘 · 回滚 = DROP 四表 + 关旗 + \ No newline at end of file diff --git a/reference/architecture.md b/reference/architecture.md index 484cd70bf1..3a89fb43a5 100644 --- a/reference/architecture.md +++ b/reference/architecture.md @@ -40,6 +40,7 @@ billion-context/ │ │ ├── lane-decisions.ts # Single resolver for per-lane gating decisions (#3417) │ │ ├── pi-subagent-channel.ts # Channel fallback for pi-subagent children missing ACP tools (#2268) │ │ ├── snapshot-render.ts # Snapshot render: outbound prefix from server transcript (#3756) +│ │ ├── snap-egress.ts # Outbound egress from the snapshot plane, direct authority (#5874 P2) │ │ ├── thread-tracking.ts # Anthropic Threads previous_message_id repair (#2701) │ │ ├── threads-lane.ts # Anthropic Threads lane: mirror, budget, reforge (#2926) │ │ ├── thinking-posture.ts # Signature-less thinking posture: 400-wording detector + wire strip + session memory (#5998) @@ -145,6 +146,14 @@ billion-context/ │ ├── session-gc.ts # Session-file GC, opt-in (sessions.gc.* / BILI_SESSION_GC*) (#1082) │ ├── persist.ts # On-disk session persistence (kernel StateStore) │ ├── persist-eperm.ts # EPERM/EACCES write-failure alert layer (#362) +│ ├── snapshot/ # 2×2 snapshot plane logic: stage-0 world match, assembly, delta egress (#6181) +│ │ ├── assemble.ts # Stage-0 kernel-input assembly: plane rows + new spans + anchors, envelope order (#6181) +│ │ ├── checksums.ts # Snapshot checksums (lowercase hex sha256) + canon digests (#5874) +│ │ ├── delta.ts # Outbound delta selection: seed/delta/no-op + full-resend detection (#6181) +│ │ ├── identity.ts # matchWorld verdicts (echo/probe/scatter/issue/diverge/resync) + canon parse (#5874) +│ │ ├── plane.ts # Stage-0 decision glue: R3 identity check, 400 fault breaker, lane decide (#6181) +│ │ ├── stage0.ts # Lane-agnostic stage-0 entry: pin + R3 + stage-0 decision + assembly, shared by the openai and responses lanes (#6181) +│ │ └── types.ts # Shared stage-0 types (type-only, kernel-public-types only) (#6181) │ ├── storage/ # Unified storage layer (schema, SQLite engine ladder, backends) (#2671) │ │ ├── backend.ts # Storage backend contract: sqlite + mysql operation seam (#3931) │ │ ├── canonical.ts # Shared canonicalization for storage digests (#2671) @@ -155,8 +164,9 @@ billion-context/ │ │ ├── integrity-check.ts # Cached + background quick_check off the boot critical path (#4609) │ │ ├── mysql.ts # MySQL 8.0 backend implementing the storage contract (#3388) │ │ ├── runtime-status.ts # Runtime storage degradation status for status surfaces (#3040) -│ │ ├── schema.ts # Unified storage schema DDL + migrations (v11) +│ │ ├── schema.ts # Unified storage schema DDL + migrations (v12) │ │ ├── shadow.ts # Unified-storage sidecar: shadow double-write / full read flip +│ │ ├── snap.ts # Snapshot-table compression foundation: four v12 tables, commit primitive, S4-verified rebuild (#5874) │ │ ├── sqlite-errors.ts # Corruption-shaped SQLite error classification (#5108) │ │ ├── store.ts # UnifiedStore: index.db (WAL) + content blobs, one txn per mutation │ ├── encrypt.ts # At-rest encoding: AES-256-GCM (#708) + zstd (#1080), independent diff --git a/src/config.ts b/src/config.ts index e8c42325c5..f7f2a92798 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2279,6 +2279,26 @@ type FileConfig = { * the file/default tier. Applied when the store opens (lane boot) — * hot edits apply on the next restart. */ walJournalSizeLimitBytes?: number; + /** #5874: snapshot-table compression foundation — two-level + * rollout flags (config-file only; no env tier by design, #2030). + * `enabled` = master (the plane may exist); `shadowWrite` = lane + * (the shadow write hook runs after each ingest, read side still + * never serves). The lane only counts with the master on. Both + * default OFF: the four v12 tables stay empty and the hook is a + * no-op — rollback is DROP tables + user_version rewind. */ + snapBase?: { + enabled?: boolean; + shadowWrite?: boolean; + /** #5874 P2 (mapEgress): serve the outbound compressed context + * from the verified snap plane — DIRECT authority (the plane + * stores the kernel's own tagged processTurn output; any plane + * gap/exception degrades to the legacy bytes for one round). + * Opt-in: default OFF when unset (#5874 §9). */ + egress?: boolean; + /** Protocol cohorts allowed to cut over (subset of + * openai|anthropic|google|responses). Default: all four. */ + egressCohorts?: string[]; + }; /** #3901/#3463: session retention window in days for the GC age * sweep (archive + tombstone + CAS release). 0 disables; env * BILI_STORAGE_ARCHIVE_AFTER_DAYS wins. Default 365. */ diff --git a/src/knobs.ts b/src/knobs.ts index 32e49ad6a0..5938e4877c 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -981,6 +981,24 @@ export function fileStorageWalJournalSizeLimit(): number | undefined { return typeof v === "number" && Number.isInteger(v) && v > 0 ? v : undefined; } +/** #5874: snapshot-table compression foundation flags — config-file only + * (no env tier: #2030 reserves env for documented-essential categories, + * and this is a rollout knob, not an environment fact). Two-level + * nesting: `storage.snapBase.enabled` is the MASTER flag (the plane is + * allowed to exist); `storage.snapBase.shadowWrite` is the LANE flag + * (the shadow write hook actually runs). The lane flag only counts with + * the master on — resolved here so a config with shadowWrite=true but + * enabled=false (or absent) can never turn the plane on by itself. + * Defaults: both false — the four tables exist (schema v12, additive) but + * stay untouched, and the write hook is a no-op. */ +export function fileSnapBase(): { enabled: boolean; shadowWrite: boolean } | undefined { + const v = loadConfigFile().storage?.snapBase; + if (v === undefined || v === null || typeof v !== "object") return undefined; + const enabled = v.enabled === true; + const shadowWrite = v.shadowWrite === true; + return { enabled, shadowWrite: enabled && shadowWrite }; +} + /** Tiered storage-migration switch (#2671): legacy | shadow | full | sqlite. * Lives here per the env-var discipline — env tier BILI_STORAGE_UNIFIED * (1/true = shadow, full/unified = full, sqlite = pure-sqlite, 0/false/legacy/ diff --git a/src/msgid-pin.ts b/src/msgid-pin.ts index 6f93e72a64..55993cfa86 100644 --- a/src/msgid-pin.ts +++ b/src/msgid-pin.ts @@ -82,6 +82,20 @@ export function incomingMsgIdsHeader(headerValue: string | string[] | undefined) type SeqValidation = { ok: true; seqs: number[] } | { ok: false; reason: string }; +// #5874 2×2 (brief 19544722, Part A): the SEQS half of the dual-channel +// identity (biliLedgerSeqs, #4316 layer 3 transport). Lenient read like +// incomingMsgIdsOf — absent/malformed → undefined → callers take exactly +// the pre-2×2 path (the strict pin stays with validateMsgSeqs). +export function incomingSeqsOf(body: unknown): number[] | undefined { + if (typeof body !== "object" || body === null) return undefined; + const raw = (body as Record)[MSG_SEQS_BODY_FIELD]; + if (!Array.isArray(raw) || raw.length === 0) return undefined; + for (const v of raw) { + if (typeof v !== "number" || !Number.isSafeInteger(v) || v < 1) return undefined; + } + return raw as number[]; +} + /** Strict validation for the seq stamp array: 1-based positive integers, * exactly one per message. Non-monotonic or non-contiguous values are * ACCEPTED on purpose — the branch pointers pick arbitrary row subsets diff --git a/src/persist.ts b/src/persist.ts index 4b16a3ebb9..85803bc5bb 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -20,6 +20,7 @@ import { sessionResidentHardMaxBytes, resolveArchiveAfterDays, fileStorageUnified, + fileSnapBase, fileStorageJsonWriter, fileStorageArchiveAfterDays, fileStorageWalJournalSizeLimit, @@ -716,7 +717,7 @@ export class SessionStore { // triggered a paced retry — stamping here kept that read failing // closed for a full cooldown window. this.syncSidecarOpen = { legacyLoad }; - this.shadow = UnifiedShadow.maybeOpen(this.sessionsDir, process.env, legacyLoad, fileStorageUnified(), fileStorageWalJournalSizeLimit(), fileStorageBackend()) ?? undefined; + this.shadow = UnifiedShadow.maybeOpen(this.sessionsDir, process.env, legacyLoad, fileStorageUnified(), fileStorageWalJournalSizeLimit(), fileStorageBackend(), fileSnapBase()) ?? undefined; } else { this.shadow = undefined; } @@ -2451,6 +2452,75 @@ export class SessionStore { } } + /** #5874: snapshot-plane status for /__bili/status. Null when the + * unified store is unavailable (legacy-only lanes) — the status wire + * then reports the flags with zeroed counters. */ + snapplaneStatus(): import("./storage/snap.js").SnapPlaneStatusWire | null { + if (!this.replicasAvailable()) return null; + try { + return this.liveShadow().snapplaneStatus(); + } catch { + return null; // sidecar degraded mid-call + } + } + + /** #5874 P2 (mapEgress): the S4-verified plane view for egress assembly. + * null when the unified store is unavailable or the sid has no plane + * yet — callers keep the legacy egress bytes. */ + snapEgressView(sessionId: string): import("./storage/snap.js").SnapView | null { + if (!this.replicasAvailable()) return null; + try { + return this.liveShadow().snapEgressView(sessionId); + } catch { + return null; // sidecar degraded mid-call — degrade, never fail the turn + } + } + + // #5874 2×2 stage 0 (brief 19544722): errors PROPAGATE (the lane maps the + // R3 reject to a hard 400 and any other fault to the legacy full path) — + // the egress-view swallow above must not be copied here. + + /** Stage 0 (element world match) for one turn. null when the unified + * store is unavailable — the lane keeps the legacy path. */ + snapStage0(sessionId: string, incoming: string[], opts?: { resync?: boolean, seqs?: number[] | null }): import("./snapshot/plane.js").Stage0Result | null { + if (!this.replicasAvailable()) return null; + return this.liveShadow().snapStage0(sessionId, incoming, opts); + } + + /** The plane's ACTIVE cores, pos-ordered (the "old cores from the + * table" half of the kernel-input assembly). */ + snapStage0Cores(sessionId: string): { rows: { pos: number; ids: string[]; postJson: string }[]; coverMax: Record } | null { + if (!this.replicasAvailable()) return null; + return this.liveShadow().snapStage0Cores(sessionId); + } + + /** #6181 round 2, Residual 2 hardening: the plane's max ACTIVE pos + * (-1 when empty) — see UnifiedStore.snapPlaneMaxPos. */ + snapPlaneMaxPos(sessionId: string): number { + if (!this.replicasAvailable()) return -1; + return this.liveShadow().snapPlaneMaxPos(sessionId); + } + + /** The stage-0 resync (R4 / table behind): reset the session plane so + * the next turn re-issues from the client's current array. */ + snapStage0Reset(sessionId: string): void { + if (!this.replicasAvailable()) return; + this.liveShadow().snapStage0Reset(sessionId); + } + + /** #6181 M3: whether this sid's R3 breaker latch persisted from a + * previous process (a restart must not re-arm a misbehaving host). */ + snapR3Latched(sessionId: string): boolean { + if (!this.replicasAvailable()) return false; + return this.liveShadow().snapR3Latched(sessionId); + } + + /** #6181 M3: persist the R3 breaker latch (best-effort). */ + snapR3Latch(sessionId: string): void { + if (!this.replicasAvailable()) return; + this.liveShadow().snapR3Latch(sessionId); + } + /** #3928: physical index.db shrink after deletions (`bili sessions rm`). * Throws on failure — the deletion itself is already committed, callers * report the shrink as partial. Throws in legacy-only lanes — check @@ -3056,7 +3126,11 @@ export class SessionStore { this.shadow?.record(payload !== undefined ? { ...record, decompositionLog: payload } : record, tail && tail.length > 0 ? { lastMessages: tail, lastMessagesFolded: session.lastMessagesFolded } - : undefined, session.contentStore ?? undefined); + : undefined, session.contentStore ?? undefined, + // #5874 mapEgress: the kernel's tagged turn view rides the + // save live (the envelope record never serializes it) so + // the snapshot plane persists the tagged forms. + session.lastPlaneMessages); this.noteOwnCommit(session.id, record); return record; } @@ -3544,6 +3618,61 @@ export class SessionStore { this.drainFlushWaiters(session.id, true); } + /** #5874 P2-final: the IN-REQUEST snapshot application — persist the + * session's current canonical state right now (single store transaction + * through writeNow: legacy tables AND the snapshot plane via + * snapShadowSync), so the egress stage of the SAME request reads a plane + * that already contains this turn (chain: raw → wire→core → kernel + * rewrite → snapshot write → egress rewrite → egress). Before this the + * plane write waited for the post-response debounce, so egress always + * saw the plane one tick behind and re-stitched the live tail from the + * client resend at the seam. + * + * Never throws — a failed flush leaves the round on the legacy egress + * path (the seam walk carries it) and the debounced retry is untouched. + * On success the armed debounce is settled: our live build already + * covers every mutation up to the epoch it captured, so the pending + * timer is dropped (its build would be a byte-identical no-op); a + * mutation that landed DURING the write's chain-run window re-arms via + * the caller's end-of-prepare markDirty (the returned epoch lets it + * distinguish "already saved" from "new state since"). + * @returns { ok: write landed, epoch: revisionEpoch captured BEFORE the + * write (compare against session.revisionEpoch afterwards). */ + async flushSessionNow(session: Session): Promise<{ ok: boolean; epoch: number | undefined }> { + const epoch = session.revisionEpoch; + try { + await this.writeNow(session); + } catch (err) { + this.log("warn", `[persist] ${session.id}: in-request snapshot flush failed (round stays on the legacy egress path): ${String(err).slice(0, 200)}`); + return { ok: false, epoch }; + } + if ((session.revisionEpoch ?? 0) !== (epoch ?? 0)) { + // Newer mutation landed during the chain-run window — keep the + // debounce alive for it (re-arm; the pending timer already covers + // it when the timer still exists). + this.scheduleSave(session); + } else { + const t = this.saveTimers.get(session.id); + if (t) { + clearTimeout(t); + this.saveTimers.delete(session.id); + } + this.saveBuilders.delete(session.id); + this.writerOffPending.delete(session.id); + this.runIdleSpillIfIdle(session.id); + } + return { ok: true, epoch }; + } + + /** #5874 P2-final: lane-facing wrapper — a missing or PARTIAL store (test + * stubs that answer only the accessors under test) degrades to "not + * flushed" instead of throwing into the lane's kernel-fail catch. */ + static async flushOrSkip(session: Session): Promise<{ ok: boolean; epoch: number | undefined }> { + const st = getStore(); + if (!st || typeof st.flushSessionNow !== "function") return { ok: false, epoch: session.revisionEpoch }; + return st.flushSessionNow(session); + } + /** #4739: where this id's record lives RIGHT NOW — the one build-scoped * source decision shared by both lazy-field restores after a save. * Mirrors loadSync's routing exactly: no readable unified index (never @@ -3788,7 +3917,7 @@ export class SessionStore { const cooldown = shadowReopenCooldownMsOverride ?? SHADOW_REOPEN_COOLDOWN_MS; this.sidecarReopenNextAttemptAt = now + cooldown; try { - this.shadow = UnifiedShadow.maybeOpen(this.sessionsDir, process.env, args.legacyLoad, fileStorageUnified(), fileStorageWalJournalSizeLimit(), fileStorageBackend()) ?? undefined; + this.shadow = UnifiedShadow.maybeOpen(this.sessionsDir, process.env, args.legacyLoad, fileStorageUnified(), fileStorageWalJournalSizeLimit(), fileStorageBackend(), fileSnapBase()) ?? undefined; } catch (err) { // Transient refusal class re-occurred mid-process (unreachable via // boot — that path exits first — kept defensive per #5188): diff --git a/src/server.ts b/src/server.ts index 03912500e6..6362456a07 100644 --- a/src/server.ts +++ b/src/server.ts @@ -165,6 +165,7 @@ import { artifactSeedHit, detectAcpArtifacts } from "./server/chain-artifacts.js import { droppedOpenaiParts } from "./wire-drop-warn.js"; import { BP_MAX, ROLLOUT_FEATURES, rolloutDecisions, rolloutInstallId, warnRolloutConfigProblems } from "./rollout.js"; import { SnapshotAnchorFallbackError, WS_ANCHORED_REQ_PROP } from "./server/snapshot-render.js"; +import { FoundationRejectError } from "./snapshot/plane.js"; export { forward } from "./server/relay.js"; // #1086/#1218: the per-session chain-verdict memory moved to plugin.ts @@ -780,6 +781,21 @@ export async function startServer(opts: ProxyOptions): Promise { // The rethrow rides ONLY the request the WS session stamped // (WS_ANCHORED_REQ_PROP); plain-HTTP lanes keep the 503 shape. if (err instanceof SnapshotAnchorFallbackError && (req as unknown as Record)[WS_ANCHORED_REQ_PROP] !== undefined) throw err; + // #6181 B1/B2: an R3 identity fault (the client declared a + // snapshot envelope that fails the identity gates) is a HARD 400 + // with the identity marker — NOT the generic 503, NOT a silent + // raw forward: the host must learn that the envelope is broken. + // Both lanes rethrow FoundationRejectError into this funnel; it + // throws before processTurn, so the session is untouched. + if (err instanceof FoundationRejectError) { + if (!res.headersSent) { + res.writeHead(400, { "content-type": "application/json", "x-bili-snapshot": "identity" }); + res.end(JSON.stringify({ error: { message: err.message, type: "snapshot_identity_fault" } })); + } else { + res.end(); + } + return; + } const msg = String(err); const e = err as { name?: string; message?: string }; // #411: a client cancel aborts the upstream fetch via @@ -2001,6 +2017,21 @@ export type Prepared = { responsesTextProtocol?: boolean; resetAfterSuccess?: boolean; responsesProjection?: ResponsesProjection; + /** #6181 B4: the delta frontier selected for THIS turn's outbound — + * COMMITTED to the session only AFTER the upstream ACCEPTS the request + * (relay.ts, first forward, status < 400). A failed/4xx forward drops + * it (null) so the next round RE-SEEDS (at-least-once: over-serving is + * safe, under-serving is the context loss). Absent (undefined) = no + * delta was selected this turn. */ + pendingResponsesDelta?: { state: { frontierPos: number; servedAnchors: string[] }; kind: "full" | "noop" | "partial" } | null; + /** #6181 round 2, Residual 2 hardening: the plane max pos (post + * in-request flush) the provider chain will hold once THIS forward is + * accepted — the relay pairs it with the response id it observes on + * the outbound stream and records session.lastMintedResponse (accepted + * forwards only; a refused round mints nothing). Set by the responses + * lane on every round whose accepted forward leaves the provider + * holding the full world. */ + mintFrontierPos?: number; anthropicSystem?: AnthropicRequestBody["system"]; /** #2189: the client's billing-attribution block captured from the INBOUND * anthropic system (pre-anchor). Preflight summary calls must carry it — diff --git a/src/server/admin.ts b/src/server/admin.ts index dd3684131b..b9ec98736e 100644 --- a/src/server/admin.ts +++ b/src/server/admin.ts @@ -23,6 +23,9 @@ import { getUpdateVisibility, updateBannerPayload } from "../update-notes.js"; import { installVersionAction, triggerManualCheck, updateActionsAvailable, updateStateSnapshot } from "./update-actions.js"; import { getUnrecognizedPathStats } from "./observability.js"; import { getStore } from "../persist.js"; +import { snapEgressStatusWire } from "./snap-egress.js"; +import { fileSnapBase } from "../knobs.js"; +import type { SnapPlaneStatusWire } from "../storage/snap.js"; import { BodyTooLargeError, headerValue, readBody } from "../server.js"; import { dismissHostPlugin, visibleHostPluginNotices } from "../host-plugin-notice.js"; import { dismissUpgradeNotice } from "../pck-notice.js"; @@ -1317,7 +1320,37 @@ async function sendStatus(res: http.ServerResponse, opts: ProxyOptions, laneHost loggerLog("warn", `split-session canary (#2170): conversation ${w.base} has live traffic under multiple session keys (design persona forks are excluded): ${w.sessions.map((s) => `${s.id} (requests=${s.requests})`).join("; ")}. For a non-persona host this is the #2165 failure shape (stolen anchor / never-compressing split) — investigate if unexpected.`); } } - res.end(JSON.stringify({ version: VERSION, commit: BUILD_COMMIT, origin: laneHost ?? null, diskVersion, stale, lastInstall, updateReady: updateReadyPayload(), autoRestartOnUpdate: opts.autoRestartOnUpdate, advisory: currentAdvisoryPayload(), resolverWedge: getResolverWedgeState(), updatePolicy: updatePolicyPayload(), pluginAdvisories: detectCostAdvisories(process.env), inFlight: totalInFlight(), splitSessions: splitWarnings, conflicts: summarizeConflicts(listSessions()), storage: storageStatusWire(), snapshotShadow: snapshotShadowStatusWire(), attribution: attributionStatusWire() }, null, 2)); + res.end(JSON.stringify({ version: VERSION, commit: BUILD_COMMIT, origin: laneHost ?? null, diskVersion, stale, lastInstall, updateReady: updateReadyPayload(), autoRestartOnUpdate: opts.autoRestartOnUpdate, advisory: currentAdvisoryPayload(), resolverWedge: getResolverWedgeState(), updatePolicy: updatePolicyPayload(), pluginAdvisories: detectCostAdvisories(process.env), inFlight: totalInFlight(), splitSessions: splitWarnings, conflicts: summarizeConflicts(listSessions()), storage: storageStatusWire(), snapshotShadow: snapshotShadowStatusWire(), snaplane: snapplaneStatusPayload(), egress: snapEgressStatusWire(), attribution: attributionStatusWire() }, null, 2)); +} + +// #5874: the snapshot-table plane's /__bili/status field — flag state +// (master/lane from the config file), counters (sessions/rows/log/folds/L1) +// and the per-sid dead latches. Store unavailable (legacy-only lane) → the +// flags with zeroed counters; a throw → all-off with zeroed counters +// (status is informational and must never take the endpoint down). +function snapplaneStatusPayload(): SnapPlaneStatusWire { + // Default off when the config file is silent (#5874 §9) — the 2026-11-08 + // soak ON default was reverted before the merge (2026-11-13). + const flags = fileSnapBase() ?? { enabled: false, shadowWrite: false }; + const zeroed: SnapPlaneStatusWire = { + enabled: flags.enabled, + shadowWrite: flags.shadowWrite, + deadSids: [] as string[], + sessions: 0, + rows: 0, + activeRows: 0, + logEntries: 0, + ckpts: 0, + l1Entries: 0, + l1Armed: 0, + }; + try { + const st = getStore().snapplaneStatus(); + if (st === null) return zeroed; + return st; + } catch { + return { ...zeroed, deadSids: [] }; + } } // #2090 plan A — read-only view backing the web UI's "Signed upstreams" card: diff --git a/src/server/handle.ts b/src/server/handle.ts index 8263a88cb1..3b343ebf59 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -3071,11 +3071,39 @@ async function handleRequest( // a real model (the loop's synthetic frames read it back). return await prepareGoogle(work as GoogleRequestBody, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, nativeWindow, googleModel ?? v1Model, googlePathKind(urlPath) === "stream-generate", visibilityMarkers, upstreamOrigin, req, decide); } + // #6181 B3: a PLUGIN-lane chained round whose input + // carries no ARRAY content (absent or empty — a + // bare-string input is admitted too: the provider + // concatenates it to its retained chain, the same + // outcome the kernel path would produce, with none of + // its side effects) is a pure chain continuation — the + // provider + // already has the full world via its retained chain. The + // kernel path for such a round is HARMFUL: it injects the + // standing dev message into an otherwise-empty input and + // (on the no-delta egress path) strips the chain id, so + // the provider answers from nothing and the host then + // chains from that BLIND response → permanent context + // loss. Forward the client's bytes verbatim instead (the + // same passthrough the /responses/compact endpoint + // already owns: no kernel, no dev injection, no plane + // side effects — the response-side ingest still updates + // the session from the answer). + const pureChainContinuation = + protocol === "responses" && + !responsesCompact && + !wsLaneEnvelope && + pluginMode && + typeof (work as ResponsesRequestBody).previous_response_id === "string" && + (work as ResponsesRequestBody).previous_response_id!.length > 0 && + (!Array.isArray((work as ResponsesRequestBody).input) || (work as ResponsesRequestBody).input!.length === 0); const preparedOut = protocol === "anthropic" ? await prepareAnthropic(work as AnthropicRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, reasoningCfg, visibilityMarkers, requestModel, decide) : protocol === "openai" ? await prepareOpenai(work as OpenAIRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg, visibilityMarkers, route?.rewrittenUrl, decide, bodyText) - : responsesCompact + : pureChainContinuation + ? prepareResponsesCompact(bodyBuffer, work as ResponsesRequestBody, session, req, core, reqConfig, log) + : responsesCompact // #618: when no bili compaction item is present, // prepareResponsesCompact falls back to the raw bodyBuffer — // forward it unchanged (byte-identical passthrough). @@ -3085,6 +3113,17 @@ async function handleRequest( // Anthropic-on-Vertex wire the body carries none, and the // relay's compress-loop rebuild reads it from here. preparedOut.model = requestModel; + // #6181 B3: the compact-passthrough base hard-codes + // resetAfterSuccess — which schedules a native-compaction + // REBASE for the next Responses turn (reconcileNative- + // CompactionBoundary → resetSessionCompression). A pure + // chain-continuation round compacts nothing: honoring the + // marker would wipe this session's compression state on + // the next turn with no compaction ever having happened. + // Drop it for the B3 route (the next turn's stage-0 + // resync already converges the plane when the host + // resends a genuinely-compact world). + if (pureChainContinuation) preparedOut.resetAfterSuccess = false; return preparedOut; }; // #1295: stamp the envelope meta onto every Prepared for this diff --git a/src/server/prepare-anthropic.ts b/src/server/prepare-anthropic.ts index 51a28bd890..734690b5aa 100644 --- a/src/server/prepare-anthropic.ts +++ b/src/server/prepare-anthropic.ts @@ -13,6 +13,7 @@ import { buildDecisionPrompt, buildDirectiveText, consumeFallback, ladderMode, r import { excludeDeadRefRanges } from "../acp-status.js"; import { reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; import { maybeSnapshotView } from "./snapshot-render.js"; +import { applySnapEgress } from "./snap-egress.js"; import { storedMessagesFor, storedRowLookupFor } from "../fold-decomposition-lookup.js"; import { nudgeSuppressed } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; @@ -36,6 +37,7 @@ import { reconcileSystemAnchor } from "../system-anchor.js"; import { isStrictReasoningEcho, modelIdOf } from "../strict-echo.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; import { imageRescaleMaxDim, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore } from "../persist.js"; import { applyImageRescalePass } from "../image-rescale.js"; import { isAutoModeClassifier } from "../auto-mode-forge.js"; import { countSystemAndToolsTokens, emergencyNudge, emergencyNudgeArmed, projectThinkingMass } from "./budget.js"; @@ -218,6 +220,9 @@ export async function prepareAnthropic( const markerCounts = stripRawStage(parsed, "anthropic"); logRawStripCounts(markerCounts, log, sessionId); + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; try { const converted = anthropicToCore(parsed); let msgs = converted.msgs; @@ -378,6 +383,27 @@ export async function prepareAnthropic( const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, tokenCountSource, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); + // #5874 mapEgress (owner ruling 2026-11-11): the plane persists the + // kernel's OWN tagged processTurn output (invariant v: post_json is + // the post-rewrite core form, never wire bytes). Deep-copy — the lane + // mutates turn.messages in place later (absorb view, subagent attach) + // and the egress transform re-applies exactly those passes, so the + // stored rows must stay the pre-mutation bytes. The anchor rows ride + // the copy too: they are processTurn output. + session.lastPlaneMessages = JSON.parse(JSON.stringify(turn.messages)); + // #5874 P2-final: snapshot application (in-request) — the canonical + // transcript up to THIS user message plus the kernel's post-fold state + // must be on the snapshot plane before the egress stage reads it + // (chain: raw → wire→core → kernel rewrite → snapshot write → egress + // rewrite → egress). snapshotMessages first: the persist build reads + // session.lastMessages — and it stays the RAW client resend (never + // the D-lane view: its synthetic acp_summary anchor is not a + // transcript message). The plane rows have a separate source: + // session.lastPlaneMessages above (the kernel's tagged output). + // Never fatal — a failed flush leaves the round on the legacy egress + // path. + snapshotMessages(session, originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -446,6 +472,15 @@ export async function prepareAnthropic( if (caLine) log("info", caLine); log("info", diagNudge(turn, sessionId, tokenCount, config.modelContextLimit, resolvedModel ?? parsed.model, willInjectNudge, decideTier1Arm && !decideFallbackOverride, surfaceOffDetail(laneDecisions, opts.compress.injectTool))); processedMessages = applyReasoningDrop(stripKernelSummaries(turn.messages, turn.state), reasoningPlan, log, sessionId); + // #5874 P2 (mapEgress): tripwire-guarded plane egress (see + // prepare-openai.ts for the chain rationale). + processedMessages = applySnapEgress(turn.state, processedMessages, { + sessionId, + protocol: "anthropic", + log, + isCompressTool: loopConfig.isCompressTool, + transform: (m) => applyReasoningDrop(stripKernelSummaries(attachSubagentSessions(applyAbsorbView(m, session.state, loopConfig, tokenCount), session), turn.state), reasoningPlan, log, sessionId), + }); // #1001: a silent client history rewrite takes the same archive+prune path // as an announced /compact boundary — syncBlocks above has already // deactivated the blocks whose sources left the context. Skipped when @@ -616,7 +651,6 @@ export async function prepareAnthropic( // SysPrompt row — the kernel breakdown classifies messages only, and on // this wire the system rides the top-level `system` field outside the fold. session.metadata.systemPromptTokens = countSystemAndToolsTokens(extractSystem(systemOut), toolsOut); - snapshotMessages(session, originalMessages); // #3397 durability-before-render: make this turn's ledger rows durable NOW // (synchronous SQLite ingest, no JSON envelope) so the refs the model is // about to see are on disk BEFORE the response is sent. Without this, a @@ -625,7 +659,10 @@ export async function prepareAnthropic( // gate rejects as "above the highest allocated ref". markDirty below keeps // the JSON envelope debounced; this only moves the small row ingest ahead. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); // [#5604] a preflight fold of the tail (soft protection relaxed to 0, or a // tail tool pair in neverPreserveRecentTools) can leave the outbound array diff --git a/src/server/prepare-google.ts b/src/server/prepare-google.ts index f85b4dc1c1..2b03d79e5d 100644 --- a/src/server/prepare-google.ts +++ b/src/server/prepare-google.ts @@ -12,6 +12,7 @@ import { buildDecisionPrompt, buildDirectiveText, consumeFallback, ladderMode, r import { excludeDeadRefRanges } from "../acp-status.js"; import { reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; import { maybeSnapshotView } from "./snapshot-render.js"; +import { applySnapEgress } from "./snap-egress.js"; import { storedMessagesFor, storedRowLookupFor } from "../fold-decomposition-lookup.js"; import { nudgeSuppressed } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; @@ -33,6 +34,7 @@ import { attachSubagentSessions } from "../subagent-sessions.js"; import { reconcileSystemAnchor } from "../system-anchor.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; import { imageRescaleMaxDim, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore } from "../persist.js"; import { applyImageRescalePass } from "../image-rescale.js"; import { clampOutgoingOutput, countSystemAndToolsTokens, dshLedgerFloorTokens, emergencyNudge, emergencyNudgeArmed } from "./budget.js"; import { estimateCoreMessages } from "../preflight.js"; @@ -122,6 +124,9 @@ export async function prepareGoogle( const markerCounts = stripRawStage(parsed, "google"); logRawStripCounts(markerCounts, log, sessionId); + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; try { const { msgs, systemText } = googleToCore(parsed); googleClientSystem = systemText; @@ -208,6 +213,27 @@ export async function prepareGoogle( const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, tokenCountSource, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); + // #5874 mapEgress (owner ruling 2026-11-11): the plane persists the + // kernel's OWN tagged processTurn output (invariant v: post_json is + // the post-rewrite core form, never wire bytes). Deep-copy — the lane + // mutates turn.messages in place later (absorb view, subagent attach) + // and the egress transform re-applies exactly those passes, so the + // stored rows must stay the pre-mutation bytes. The anchor rows ride + // the copy too: they are processTurn output. + session.lastPlaneMessages = JSON.parse(JSON.stringify(turn.messages)); + // #5874 P2-final: snapshot application (in-request) — the canonical + // transcript up to THIS user message plus the kernel's post-fold state + // must be on the snapshot plane before the egress stage reads it + // (chain: raw → wire→core → kernel rewrite → snapshot write → egress + // rewrite → egress). snapshotMessages first: the persist build reads + // session.lastMessages — and it stays the RAW client resend (never + // the D-lane view: its synthetic acp_summary anchor is not a + // transcript message). The plane rows have a separate source: + // session.lastPlaneMessages above (the kernel's tagged output). + // Never fatal — a failed flush leaves the round on the legacy egress + // path. + snapshotMessages(session, originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -273,6 +299,16 @@ export async function prepareGoogle( if (caLine) log("info", caLine); log("info", diagNudge(turn, sessionId, tokenCount, config.modelContextLimit, model, willInjectNudge, decideTier1Arm && !decideFallbackOverride, surfaceOffDetail(laneDecisions, opts.compress.injectTool))); processedMessages = stripKernelSummaries(turn.messages, turn.state); + // #5874 P2 (mapEgress): tripwire-guarded plane egress. Google's lane + // applies NO reasoning drop (#651: thoughtSignature rides verbatim), + // so the plane transform is strip-only — same chain, same guard. + processedMessages = applySnapEgress(turn.state, processedMessages, { + sessionId, + protocol: "google", + log, + isCompressTool: loopConfig.isCompressTool, + transform: (m) => stripKernelSummaries(attachSubagentSessions(applyAbsorbView(m, session.state, loopConfig, tokenCount), session), turn.state), + }); // #1001: a silent client history rewrite takes the same archive+prune path // as an announced /compact boundary — syncBlocks above has already // deactivated the blocks whose sources left the context. @@ -415,10 +451,12 @@ export async function prepareGoogle( + imageReserveFor(session, "google", rebuilt, opts, upstreamOrigin); if (upstreamOrigin) session.stats.lastLocalTextEstimateOrigin = upstreamOrigin; } - snapshotMessages(session, originalMessages); // #3397 durability-before-render (see prepare-anthropic.ts): rows durable NOW, before the model call. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); return { body: JSON.stringify(rebuilt), parsedBody: rebuilt, session, processedMessages, originalMessages, judgeBaseline, protocol: "google", stream, compressInjected: injectTools, pluginMode, mcpDialectNote: laneDecisions.mcpDialect, nudge, nudgeOutcome, prompts, surface, google: { system: googleClientSystem, model }, systemNotes: sysNotes, renderTags: renderStrategy } as Prepared; } diff --git a/src/server/prepare-openai.ts b/src/server/prepare-openai.ts index 84a453fcee..4950851cb7 100644 --- a/src/server/prepare-openai.ts +++ b/src/server/prepare-openai.ts @@ -7,7 +7,7 @@ import type { ProxyOptions } from "../config.js"; import { effectivePromptCacheRouting } from "../config.js"; import { currentCalibrationFactor, hardenOpenaiAssistantContent, systemToUser } from "../util.js"; import { conversationHeaderSource, resolveRelayAffinityPck } from "../session-id.js"; -import { MSG_IDS_BODY_FIELD, applyMsgIdPin, validateMsgIds } from "../msgid-pin.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, applyMsgSeqStamps, incomingSeqsOf, validateMsgSeqs } from "../msgid-pin.js"; import { pluginAgentHeader } from "../plugin.js"; import { applyReasoningDrop, diagNudge, diagTagSummary, deriveTitle, effectiveTokenCount, imageBillingFor, imageReserveFor, imageTokenCapFor, isAutoInjectedNotification, planReasoningDrop, reapOrphansLogged, resolveKnownOutputCeiling, runNudgeDecision, stripKernelSummaries, warnDroppedOpenaiParts, warnReasoningPairs, type Prepared } from "../server.js"; import { recordNudgeObservation } from "../context-analysis.js"; @@ -16,6 +16,7 @@ import { buildDecisionPrompt, buildDirectiveText, consumeFallback, ladderMode, r import { excludeDeadRefRanges } from "../acp-status.js"; import { reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; import { maybeSnapshotView } from "./snapshot-render.js"; +import { applySnapEgress } from "./snap-egress.js"; import { storedMessagesFor, storedRowLookupFor } from "../fold-decomposition-lookup.js"; import { nudgeSuppressed } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; @@ -40,6 +41,9 @@ import { isStrictReasoningEcho, modelIdOf, normalizeStrictEchoReasoning } from " import { reconcileSystemAnchor } from "../system-anchor.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage, type RawStripCounts } from "../marker/engine.js"; import { imageRescaleMaxDim, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore } from "../persist.js"; +import { FoundationRejectError } from "../snapshot/plane.js"; +import { runStage0 } from "../snapshot/stage0.js"; import { applyImageRescalePass } from "../image-rescale.js"; import { clampOutgoingOutput, countSystemAndToolsTokens, dshLedgerFloorTokens, emergencyNudge, emergencyNudgeArmed } from "./budget.js"; import { effectiveAbsorbBlock, effectiveBlockOpsEnabled } from "./prepare-responses.js"; @@ -76,6 +80,13 @@ export async function prepareOpenai( // host body stamps (pi incremental lane) or the #5825 handle-seam // re-stamp of stamped-header/harvested ids. const incomingMsgIds = (parsed as Record)[MSG_IDS_BODY_FIELD]; + // #5874 2×2: the seqs carrier (biliLedgerSeqs) — the M2 incremental + // join evidence. An optimization channel, not an identity channel: + // malformed/absent → the M2 fast path is disabled (the full-resend + // matchWorld path still works); it never 400s. (Stamped once msgs + // exists, below.) + const incomingSeqsRaw = incomingSeqsOf(parsed); + delete parsed[MSG_SEQS_BODY_FIELD]; delete (parsed as Record)[MSG_IDS_BODY_FIELD]; // Real wire element count; a watermark hit overrides it with the measured // total T (see the #5403/#5418 block below). @@ -160,6 +171,9 @@ export async function prepareOpenai( const markerCounts = wpHit !== undefined ? { ...wpHit.markerCounts, strippedMs: 0 } : stripRawStage(parsed, "openai"); logRawStripCounts(markerCounts, log, sessionId); + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; try { // Kernel 0.0.37 hoists the contiguous leading system/developer prefix // OUT of the fold space: system content is host runtime state and @@ -167,6 +181,14 @@ export async function prepareOpenai( // otherwise the proxy would forward payloads without any system. const converted = wpHit !== undefined ? { msgs: wpHit.view, systemText: wpHit.systemText } : openaiToCore(parsed); let msgs = converted.msgs; + // #5874 2×2: stamp the M2 seqs carrier onto the converted array + // (parallel to messages; malformed/absent → null → M2 fast path + // disabled, the full-resend path still works). + const incomingSeqs: number[] | null = + incomingSeqsRaw !== undefined && incomingSeqsRaw.length === msgs.length && validateMsgSeqs(incomingSeqsRaw, msgs.length).ok + ? incomingSeqsRaw + : null; + if (incomingSeqs !== null) applyMsgSeqStamps(msgs, incomingSeqs); let systemText = converted.systemText; openaiSystemText = systemText; wpSeedSystemText = systemText; @@ -180,33 +202,35 @@ export async function prepareOpenai( openaiSystemText = outcome.outbound; } originalMessages = msgs; - // #4491/#5825 parity: pin identity to client-supplied ids (plugin - // lanes only — pure proxy traffic has no msgid channel and stays on - // snapshot render D), the SAME gated contract as the anthropic twin. - // Skipped on watermark hits on purpose: fastpath turns reuse the - // seed stash's prefix view whose rows were already pinned when it was - // materialized, and the decoded tail window is SHORTER than the - // stamped list, so count-parity there would fail by design — pinning - // only adds noise. - if (wpHit === undefined && pluginMode && incomingMsgIds !== undefined) { - if (featureEnabled("pluginLaneServerHistory")) { - const v = validateMsgIds(incomingMsgIds, msgs.length); - if (v.ok) { - msgs = applyMsgIdPin(msgs, v.ids); - originalMessages = msgs; - } else { - log("warn", `[${sessionId}] [msgid] openai-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#5825)`); - } - } else { - log("info", `[${sessionId}] [msgid] openai-wire biliMsgIds present but pluginLaneServerHistory rollout is off for this install — field stripped, ids not pinned (#5825)`); - } + // #4491/#5825 + #5874 2×2 stage 0 (brief 19544722 / #6181): the + // SHARED pin + world-match + kernel-input assembly for cohort + // traffic (src/snapshot/stage0.ts — the responses lane runs the + // SAME helper, same red gate, same one-round-convergence + // contract). Runs BEFORE processTurn, so a thrown + // FoundationRejectError (R3) carries zero kernel side effects + // (the server funnel maps it to the hard 400); every other fault + // → the legacy full path (one-round convergence). + const s0 = runStage0({ + sessionId, + lane: "openai", + msgs, + incomingMsgIds, + incomingSeqs, + pluginMode, + wpHit, + log, + }); + if (s0.pinned !== null) { + msgs = s0.pinned; + originalMessages = msgs; } // #3756 snapshot render: the outbound prefix renders from the unified // transcript (server-authoritative, fold-time ids — immune to identity // re-mints #3752); only the live tail keeps the client's bytes. // No-op returns `msgs` when the lane is inactive/degraded. // reconcileFoldCoverage below still observes the CLIENT view. - const renderMsgs = maybeSnapshotView(session, msgs, { sessionId, pluginMode, log }); + const renderMsgs = s0.kernelInput ?? maybeSnapshotView(session, msgs, { sessionId, pluginMode, log }); + // #4241: the render lane replaced the view (reference-inequality is the // discriminator — maybeSnapshotView returns the SAME array when idle), // so the judged baseline is the rendered transcript, not the inbound. @@ -327,6 +351,27 @@ export async function prepareOpenai( const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, tokenCountSource, renderTags: renderStrategy, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); + // #5874 mapEgress (owner ruling 2026-11-11): the plane persists the + // kernel's OWN tagged processTurn output (invariant v: post_json is + // the post-rewrite core form, never wire bytes). Deep-copy — the lane + // mutates turn.messages in place later (absorb view, subagent attach) + // and the egress transform re-applies exactly those passes, so the + // stored rows must stay the pre-mutation bytes. The anchor rows ride + // the copy too: they are processTurn output. + session.lastPlaneMessages = JSON.parse(JSON.stringify(turn.messages)); + // #5874 P2-final: snapshot application (in-request) — the canonical + // transcript up to THIS user message plus the kernel's post-fold state + // must be on the snapshot plane before the egress stage reads it + // (chain: raw → wire→core → kernel rewrite → snapshot write → egress + // rewrite → egress). snapshotMessages first: the persist build reads + // session.lastMessages — and it stays the RAW client resend (never + // the D-lane view: its synthetic acp_summary anchor is not a + // transcript message). The plane rows have a separate source: + // session.lastPlaneMessages above (the kernel's tagged output). + // Never fatal — a failed flush leaves the round on the legacy egress + // path. The watermark window pin is preserved. + snapshotMessages(session, wpHit !== undefined ? wpHit.windowCores : originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -399,6 +444,16 @@ export async function prepareOpenai( if (caLine) log("info", caLine); log("info", diagNudge(turn, sessionId, tokenCount, config.modelContextLimit, parsed.model, willInjectNudge, decideTier1Arm && !decideFallbackOverride, surfaceOffDetail(laneDecisions, opts.compress.injectTool))); processedMessages = applyReasoningDrop(stripKernelSummaries(turn.messages, turn.state), reasoningPlan, log, sessionId); + // #5874 P2 (mapEgress): when the S4-verified snap plane can assemble + // the same view, switch the outbound source to it (tripwire-guarded; + // any normalized mismatch keeps these legacy bytes). + processedMessages = applySnapEgress(turn.state, processedMessages, { + sessionId, + protocol: "openai", + log, + isCompressTool: loopConfig.isCompressTool, + transform: (m) => applyReasoningDrop(stripKernelSummaries(attachSubagentSessions(applyAbsorbView(m, session.state, loopConfig, tokenCount), session), turn.state), reasoningPlan, log, sessionId), + }); // #1001: a silent client history rewrite takes the same archive+prune path // as an announced /compact boundary — syncBlocks above has already // deactivated the blocks whose sources left the context. Skipped when @@ -563,6 +618,11 @@ export async function prepareOpenai( attachedRetrievalNoteIds = retrNotes.map((n) => n.id); } } catch (err) { + // #6181 B1: the R3 identity fault is a HARD reject, not a degrade — + // rethrow it to the server funnel (400 + x-bili-snapshot:identity) + // BEFORE the generic 503; zero kernel side effects by construction + // (it throws before processTurn). + if (err instanceof FoundationRejectError) throw err; log("warn", `[${sessionId}] kernel transform failed, forwarding unchanged: ${String(err)}`); if (attachedRetrievals.length > 0) dropRetrievals(session, attachedRetrievals.map((i) => i.ref), "prepare failed; forwarded unprocessed"); processedMessages = []; @@ -638,10 +698,12 @@ export async function prepareOpenai( // #5403/#5418: on a watermark hit pin only this turn's window — the // stashed prefix lives OUTSIDE the Session by design; the fold machinery // applies to the pin exactly as it already does for folded sessions. - snapshotMessages(session, wpHit !== undefined ? wpHit.windowCores : originalMessages); // #3397 durability-before-render (see prepare-anthropic.ts): rows durable NOW, before the model call. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); let wpResponseHeaders: Record | undefined; if (wpFullText !== undefined && wpOutcome !== null && wpTurnStrip !== undefined) { wpResponseHeaders = finalizeWatermarkTurn(session, wpOutcome, { diff --git a/src/server/prepare-responses.ts b/src/server/prepare-responses.ts index bd9b491f2a..15f117fc8f 100644 --- a/src/server/prepare-responses.ts +++ b/src/server/prepare-responses.ts @@ -16,11 +16,15 @@ import { mergeAdjacentConfigurationUpdates, patchResponsesInputWithToolImages as import { dropWhitespaceResponsesMessages, normalizeResponsesMessageItems, sanitizeResponsesInputIds } from "../loop/adapter-responses.js"; import { METADATA_LAST_RESPONSES_MODE, noteCanonicalSystemFingerprint, reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; import { buildAnchoredView, maybeSnapshotView, noteAnchoredRender, SnapshotAnchorFallbackError, wsAnchoredPayload } from "./snapshot-render.js"; +import { applySnapEgress } from "./snap-egress.js"; +import type { DeltaState } from "../snapshot/delta.js"; +import { FoundationRejectError } from "../snapshot/plane.js"; +import { runStage0 } from "../snapshot/stage0.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, applyMsgSeqStamps, incomingSeqsOf, validateMsgSeqs } from "../msgid-pin.js"; import { storedMessagesFor, storedRowLookupFor } from "../fold-decomposition-lookup.js"; import { nudgeSuppressed, pluginLaneDegraded } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; import { absorbAbortedTurnGrowth, applyCompactionArchive, coveredRealHistoryIds, foldCoverage, markDirty, markNativeCompactionBoundary, reconcileNativeCompactionBoundary, REWRITE_MIN_INCOMING_TOTAL, snapshotMessages, type Session } from "../session.js"; -import { getStore } from "../persist.js"; import { carriesDshLocalCompactionSummary, DSH_LOCAL_COMPACTION_MIN_MISSING } from "./dsh-compaction-guard.js"; import { carriesBiliSnapshotCheckpoint } from "./dsh-authored-compaction.js"; import { recordConflict } from "../conflict-watch.js"; @@ -43,6 +47,7 @@ import { pckKeyAbated } from "../pck-probe.js"; import { reconcilePluginFoldCallIds } from "../plugin.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; import { imageRescaleMaxDim, keepResponseId as knobKeepResponseId, noCompressPrompt as knobNoCompressPrompt, noInjectTool as knobNoInjectTool, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore, getStore } from "../persist.js"; import { applyImageRescaleItems } from "../image-rescale.js"; import { clampOutgoingOutput, countLoadedToolTokens, countSystemAndToolsTokens, dshLedgerFloorTokens, emergencyNudge, emergencyNudgeArmed, modelVisibleTools } from "./budget.js"; import { FORCE_TEXT_PROTOCOL, injectResponsesTool, injectTool } from "./inject.js"; @@ -104,6 +109,14 @@ export async function prepareResponses( } let processedMessages: CoreMessage[] = []; + // #6181 2×2: the msgid + ledger-seq dual-channel the plugin host rides + // on the body (src/msgid-pin.ts) — read and STRIPPED before any + // projection so the real upstream never sees a field it does not know. + // (Same discipline as the openai twin.) + const incomingMsgIds = (parsed as unknown as Record)[MSG_IDS_BODY_FIELD]; + const incomingSeqsRaw = incomingSeqsOf(parsed); + delete (parsed as unknown as Record)[MSG_SEQS_BODY_FIELD]; + delete (parsed as unknown as Record)[MSG_IDS_BODY_FIELD]; // #564/#2627: the ordering separators THIS turn's repair actually inserted, // recorded by the producer so provenance never has to be inferred from item // text later. Empty on the overwhelming majority of turns. @@ -117,6 +130,39 @@ export async function prepareResponses( let toolsOut = parsed.tools; let transformOk = false; let responsesDevContent: string | undefined; + // 2×2 Part B: true when this round's outbound is a PLANE DELTA (the + // provider's retained chain carries the rest) — gates the developer + // message re-injection and the previous_response_id strip below. + let deltaServed = false; + // 2×2 Part B (review-round-2 Residual 1a): a chained round whose plane + // frontier is ALREADY at the top of the world — nothing new to send. + // The outbound is an EMPTY input with the chain id KEPT and no developer + // message: the provider's retained chain holds the whole world. Without + // this classification the no-op read as "no delta" → id stripped + dev + // injected + empty input = a BLIND response the host would chain from. + let noopServed = false; + // 2×2 Part B: a persisted delta frontier existed BEFORE this round's + // outbound selection (the provider's retained context equals exactly the + // elements sent up to it) — see the strip site below for the contract. + let hadFrontier = false; + // 2×2 Part B (B4): hoisted so the return literal can hand the selected + // frontier to the relay (committed only after the upstream accepts). + let deltaState: DeltaState | null | undefined; + let selectionFired = false; + let pendingDelta: { state: { frontierPos: number; servedAnchors: string[] }; kind: "full" | "noop" | "partial" } | null = null; + // #6181 B0: a chained turn that cannot be addressed against the plane + // (non-cohort — its kernel view is a delta, not the world) skips BOTH + // the plane write for this turn's flush and the plane egress — see the + // flag set site below the processTurn block. + let planeBypass = false; + // #6181 round 2, Residual 2 hardening: the plane max pos the provider + // chain will hold once THIS forward is accepted — the relay pairs it + // with the response id it observes on the outbound stream (see + // relay.ts). Set after the in-request flush: the plane then holds + // exactly this round's world (or, for a planeBypass round, the + // previous world — which is precisely the boundary the chain already + // holds, since the bypass skips the flush). + let mintFrontierPos: number | undefined; let sysNotes: string[] = []; // #3713: billed tokens of the #1638-masked mid-history sys/dev items — they // ride the rebuilt input verbatim yet enter neither projection.msgs nor @@ -173,6 +219,9 @@ export async function prepareResponses( resolveCompressProtocol(opts.routes, upstreamOrigin) === "marker"; const renderTags: "text-only" | "none" = knobRenderNone() || isCompactionTrigger ? "none" : "text-only"; + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; try { // [#1638] Position-preserve mid-history system/developer items for // plugin agents and native Codex. Both append developer @@ -242,8 +291,37 @@ export async function prepareResponses( sysNotes = outcome.notes; if (outcome.outbound !== fresh) projection.systemParts = outcome.outbound ? [outcome.outbound] : []; } - const { msgs } = projection; + let { msgs } = projection; originalMessages = msgs; + // #6181 2×2: stamp the M2 seqs carrier onto the converted array + // (parallel to input; malformed/absent → null → the M2 fast path is + // disabled, the full-resend world-match path still works). + const incomingSeqs: number[] | null = + incomingSeqsRaw !== undefined && incomingSeqsRaw.length === msgs.length && validateMsgSeqs(incomingSeqsRaw, msgs.length).ok + ? incomingSeqsRaw + : null; + if (incomingSeqs !== null) applyMsgSeqStamps(msgs, incomingSeqs); + // #6181 2×2 stage 0 (brief 19544722): the SHARED pin + world-match + // + kernel-input assembly (src/snapshot/stage0.ts — the SAME helper + // as the openai lane, same red gate). Before processTurn, so an R3 + // FoundationRejectError carries zero kernel side effects. + const s0 = runStage0({ + sessionId, + lane: "responses", + msgs, + incomingMsgIds, + incomingSeqs, + pluginMode, + log, + }); + if (s0.pinned !== null) { + msgs = s0.pinned; + originalMessages = msgs; + } + // #6181 M6: a stage-0 resync re-issues the pos space — a stale delta + // frontier against the re-issued pos is a dead loop (empty delta + // forever); clear it so the next chained round reseeds. + if (s0.resynced) session.responsesDelta = null; // #3756 snapshot render: outbound prefix from the unified transcript // (fold-time ids, immune to identity re-mints); live tail keeps the // client's bytes. No-op when the lane is inactive/degraded; the @@ -266,7 +344,10 @@ export async function prepareResponses( noteAnchoredRender(session, log); renderMsgs = anchored.view; } else { - renderMsgs = maybeSnapshotView(session, msgs, { sessionId, pluginMode, log }); + // #6181 2×2: the stage-0 assembly outranks the D-lane view for + // cohort turns (the plane IS the kernel-input authority); the + // D lane keeps serving everyone else unchanged. + renderMsgs = s0.kernelInput ?? maybeSnapshotView(session, msgs, { sessionId, pluginMode, log }); } // #4241: the render lane replaced the view (reference-inequality is the // discriminator — maybeSnapshotView returns the SAME array when idle), @@ -374,6 +455,66 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const turn = core.processTurn({ messages: renderMsgs, state: session.state, config: loopConfig, tokenCount, tokenCountSource, renderTags, contentStore: contentStoreOf(session) }); session.state = turn.state; adoptContentStore(session, turn.contentStore); + // #5874 mapEgress (owner ruling 2026-11-11): the plane persists the + // kernel's OWN tagged processTurn output (invariant v: post_json is + // the post-rewrite core form, never wire bytes). Deep-copy — the lane + // mutates turn.messages in place later (absorb view, subagent attach) + // and the egress transform re-applies exactly those passes, so the + // stored rows must stay the pre-mutation bytes. The anchor rows ride + // the copy too: they are processTurn output. + session.lastPlaneMessages = JSON.parse(JSON.stringify(turn.messages)); + // #6181 B0: set BEFORE the in-request flush below — a chained turn + // that was not assembled from the plane (non-cohort: its kernel view + // is a delta, not the world) must not flush its collapsed view into + // the plane (snapViewEvents is a final-state diff — it would DELETE + // every row the view does not contain). One-shot: cleared right + // after the flush; the plane egress below gets the same verdict as + // `bypass` (the plane is one turn stale — serving it would drop the + // new input). + const isChained = typeof parsed.previous_response_id === "string" && parsed.previous_response_id.length > 0; + planeBypass = isChained && !s0.cohortRan && !wsAnchored; + if (planeBypass) { + session.skipPlaneFlush = true; + // #5901 review fix (2026-11-13, review round 2): the view just + // captured is a DELTA, not the world — it must not outlive this + // round. The one-shot skipPlaneFlush above only protects the + // in-request flush; the relay success path re-arms the debounced + // save (relay markDirty) and that save would ride this delta as a + // TAGGED world (persist guardedBuild → shadow.record livePlane → + // snapShadowSync tagged path, B4 guard untagged-only) — + // snapViewEvents would DELETE every plane row the delta lacks. + // Untag: the later save derives from the DB final state instead + // (append-only rows for this exchange — correct). + session.lastPlaneMessages = undefined; + } + // #5874 P2-final: snapshot application (in-request) — the canonical + // transcript up to THIS user message plus the kernel's post-fold state + // must be on the snapshot plane before the egress stage reads it + // (chain: raw → wire→core → kernel rewrite → snapshot write → egress + // rewrite → egress). snapshotMessages first: the persist build reads + // session.lastMessages — and it stays the RAW client resend (never + // the D-lane view: its synthetic acp_summary anchor is not a + // transcript message). The plane rows have a separate source: + // session.lastPlaneMessages above (the kernel's tagged output). + // Never fatal — a failed flush leaves the round on the legacy egress + // path. + snapshotMessages(session, originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); + // #6181 B0: the in-request flush above either skipped the plane + // write (flag consumed) or ran it — either way the one-shot flag is + // done for this turn. + session.skipPlaneFlush = false; + // #6181 round 2, Residual 2 hardening: record the frontier this + // ACCEPTED round leaves the provider chain holding (post-flush + // plane max pos — see the declaration). Best-effort: a store error + // just leaves the mint unset (no verifiable chain → re-seeds, the + // status quo, never a loss). + try { + const s = getStore(); + if (s !== null && s !== undefined) mintFrontierPos = s.snapPlaneMaxPos(session.id); + } catch { + /* no mint record for this round */ + } // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -443,6 +584,139 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const ordered = repairResponsesAssistantOrdering(applyReasoningDrop(stripKernelSummaries(turn.messages, turn.state), reasoningPlan, log, sessionId), originalMessages); processedMessages = ordered.messages; separatorSlots = ordered.slots; + // #5874 P2 (mapEgress) direct-authority plane egress. 2×2 Part B + // (出站增量): the Responses protocol is the only cohort with + // provider-side retention (previous_response_id chaining), so this is + // the only lane with an incremental outbound mode. Delta mode requires + // the client to chain (previous_response_id present) AND the owner to + // opt in via the EXISTING keepResponseId knob (PR config-surface + // section: no new field — the knob's documented "diagnostic only" + // status is what this upgrades to: keep + delta input). + // state===null = no persisted frontier (first round / resync / lost + // state) → reseed (full send, frontier reset). + const chains = knobKeepResponseId() && typeof parsed.previous_response_id === "string" && parsed.previous_response_id.length > 0; + deltaState = chains ? (session.responsesDelta ?? null) : undefined; + // #6181 review round 2, Residual 2: the first CHAINED round of a + // keepResponseId chain carries a previous_response_id but NO committed + // frontier — round 1 carried no id, so no selection fired and B4 had + // nothing to commit. A plain re-seed would resend the WHOLE world that + // the provider's chain already holds from this proxy's own round-1 + // full view (2× the world's tokens, the conversation twice). Stage 0 + // can PROVE the safe case: the current world is a strict extension of + // the world the plane last held (probe, nothing gone, new elements). + // The chain was built from the client's own resends of exactly that + // world, so initialize the frontier at the stored-world boundary and + // serve the new elements only. Every other shape (scatter / issue / + // resync / degraded stage 0 / no chain) falls through to the plain + // re-seed — worst case there is the one-time duplication the re-seed + // always had, never a loss. Gated on !responsesChainBroken: a rejected + // selection (B4) means the provider never ran that round, so the plane + // holds more than the provider's chain — the stored-world boundary is + // NOT safe and the round must re-seed the full view. + if ( + deltaState === null && chains + && !session.responsesChainBroken + && s0.kernelInput !== undefined && s0.storedFrontierPos !== null + && session.lastMintedResponse !== null && session.lastMintedResponse !== undefined + && session.lastMintedResponse.id === parsed.previous_response_id + && session.lastMintedResponse.frontierPos === s0.storedFrontierPos + ) { + deltaState = { frontierPos: s0.storedFrontierPos, servedAnchors: [] }; + log("info", `[${sessionId}] [snaplane] frontier seeded at the stored-world boundary (pos ${s0.storedFrontierPos}) — the provider chain from this proxy's own prior round holds the stored world; serving the new elements only instead of re-seeding (Residual 2)`); + } + // True once THIS proxy has established a delta chain with the provider + // (a persisted frontier exists). Only then may a non-delta round strip + // the id: the provider's retained context equals exactly the elements + // up to the frontier, so a full replay would duplicate it. Before the + // first frontier the native #1954 KEEP=1 contract holds (id forwarded + // intact, whatever the outbound is). + hadFrontier = deltaState !== undefined && deltaState !== null; + // #6181 round 2, Residual 2 hardening: a persisted frontier is a + // claim about the PROVIDER's chain, not just the plane — and + // previous_response_id is OPAQUE. A host can fork/rewind to an older + // response id (or carry an id another proxy minted); the strict- + // extension proof bounds only the plane, so a delta built from a + // frontier the chain never provably holds loses the gap. Trust the + // frontier ONLY when the host continues from an id THIS proxy minted + // last for this session; any other chained round drops it (the egress + // then does a plain re-seed: full view, id kept — the status quo, + // at-least-once, never a loss) and the persisted state is cleared so + // the next round re-establishes verifiability from a fresh seed. + // (planeBypass rounds never reach the delta machinery — the egress + // serves the legacy bytes — so this is inert there.) + if ( + deltaState !== undefined && deltaState !== null + && !(session.lastMintedResponse !== null && session.lastMintedResponse !== undefined + && session.lastMintedResponse.id === parsed.previous_response_id) + ) { + log("info", `[${sessionId}] [snaplane] chained round carries a previous_response_id this proxy did not mint last (${session.lastMintedResponse === null || session.lastMintedResponse === undefined ? "no mint record" : "id mismatch / fork / rewind"}) — dropping the persisted frontier and re-seeding the full view (Residual 2 hardening)`); + deltaState = null; + session.responsesDelta = null; + // The re-seed is a "no frontier" round for the strip/dev gates + // too: id kept (native #1954), developer message injected (the + // full view is self-contained) — exactly the pre-seed status quo. + hadFrontier = false; + } + // #6181 B0 (decided at the flush site, above): a CHAINED turn that was + // NOT assembled from the plane (non-cohort — no msgids — or a cohort + // turn whose stage 0 did not run) cannot be addressed against the + // plane: its kernel view is a DELTA, not the world. It must touch + // neither the plane write (deriving events from the collapsed view + // would DELETE every row the view does not contain) nor the plane + // egress (the plane is one turn stale — serving it would drop the + // new input). Outbound = the legacy bytes verbatim, exactly what + // master serves for this turn; the plane stays stale-but-intact + // until the next full/cohort turn re-syncs it (one-round + // convergence). + // #5874 P2 (mapEgress): the plane view is the outbound source; + // the lane's exact chain (strip → drop → repair) is reapplied to it. + processedMessages = applySnapEgress(turn.state, processedMessages, { + sessionId, + protocol: "responses", + log, + isCompressTool: loopConfig.isCompressTool, + transform: (m) => repairResponsesAssistantOrdering(applyReasoningDrop(stripKernelSummaries(attachSubagentSessions(applyAbsorbView(m, session.state, loopConfig, tokenCount), session), turn.state), reasoningPlan, log, sessionId), originalMessages).messages, + ...(deltaState !== undefined ? { + deltaState, + onSelection: (sel, totalElements) => { + selectionFired = true; + // kind (review round 2, Residual 2 companion): the relay + // needs to know WHAT went out to maintain responsesChainBroken. + // full = reseed/full-resend (the provider now provably holds + // the whole world) · noop = nothing new requested (the chain + // is intact) · partial = the held world extended by the new + // suffix only. + pendingDelta = { + state: sel.nextState, + kind: sel.reseed || sel.delta.length >= totalElements ? "full" : sel.delta.length === 0 ? "noop" : "partial", + }; + // deltaServed = a PARTIAL delta went out: new content only. + // A reseed (state lost / first round) and a full-resend + // (delta grew to the whole view — isFullResend) both send + // the FULL view, which must read as "no delta" for the id + // handling below. + // noopServed = the frontier already covers the whole world + // (Residual 1a: same-body retry of an M1/M2 turn) — empty + // input goes out WITH the chain id kept and NO developer + // message: pure chain continuation. + noopServed = !sel.reseed && sel.delta.length === 0; + deltaServed = !sel.reseed && sel.delta.length > 0 && sel.delta.length < totalElements; + }, + } : {}), + ...(planeBypass ? { bypass: true } : {}), + }); + if (deltaState !== undefined && !selectionFired) { + // Delta was requested but the plane degraded to legacy this round + // (the legacy bytes are the FULL replay). With an established + // frontier the id is stripped below and the provider's chain no + // longer matches the stored frontier → forget it: the next chained + // round re-seeds. Without one (native KEEP=1 client, plane never + // seeded) nothing to forget — the native contract stays intact. + session.responsesDelta = null; + } + if (deltaServed) { + log("info", `[${sessionId}] [snaplane] outbound=delta frontier=${(pendingDelta as { state: { frontierPos: number } } | null)?.state.frontierPos} — input carries only the unsent suffix; the retained chain carries the rest (re-injected developer message / forged summaries skipped to avoid duplicating what the provider keeps)`); + } // #3477: consume the announced native-compaction boundary (pi session_compact // → /__bili/plugin/compact → markCompactionBoundary) exactly once, after the // post-compaction history was processed: archives the deactivated blocks, @@ -484,7 +758,12 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const devContent = devParts.join("\n\n---\n\n"); responsesDevContent = devContent; if (forgedSummaries.length > 0) log("debug", `[${sessionId}] [inject] ${forgedSummaries.length} captured summary block(s) re-injected into developer message`); - rebuiltInput = injectResponsesDeveloperMessage(rebuiltInput, devContent, projection.headRole ?? "developer"); + // 2×2 Part B: on a delta round the provider KEEPS the seed round's + // developer message — re-injecting would duplicate it. On a no-op + // round (Residual 1a) there is NO new content at all — the provider + // chain holds everything, so injecting dev + sending an empty input + // would be a BLIND response. + if (!deltaServed && !noopServed) rebuiltInput = injectResponsesDeveloperMessage(rebuiltInput, devContent, projection.headRole ?? "developer"); if (!knobNoInjectTool() && injectTools) { // #1712: decompress's startId/endId execute only on CCR-armed // sessions (#1179) — serve the no-range schema otherwise. @@ -498,7 +777,10 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const devContent = [...projection.systemParts, ...forgedSummaries, ...(tagsOnlyPrompt !== "" ? [tagsOnlyPrompt] : []), ...(genBoundaryActive && !isCompactionTrigger ? [GENERATION_BOUNDARY_NOTE] : [])].join("\n\n---\n\n"); responsesDevContent = devContent; if (forgedSummaries.length > 0) log("debug", `[${sessionId}] [inject] ${forgedSummaries.length} captured summary block(s) re-injected into developer message`); - rebuiltInput = injectResponsesDeveloperMessage(rebuiltInput, devContent, projection.headRole ?? "developer"); + // Residual 1a: a no-op round sends an EMPTY input — injecting the + // developer message there would make the provider answer from the + // dev prompt alone (a blind response the host would chain from). + if (!deltaServed && !noopServed) rebuiltInput = injectResponsesDeveloperMessage(rebuiltInput, devContent, projection.headRole ?? "developer"); } if (sysNotes.length > 0) { const inputItems: ResponseInputItem[] = typeof rebuiltInput === "string" @@ -599,6 +881,11 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr } transformOk = true; } catch (err) { + // #6181 B1: the R3 identity fault is a HARD reject, not a degrade — + // rethrow it to the server funnel (400 + x-bili-snapshot:identity) + // BEFORE the generic 503; zero kernel side effects by construction + // (it throws before processTurn). + if (err instanceof FoundationRejectError) throw err; log("warn", `[${sessionId}] kernel transform failed, forwarding unchanged: ${String(err)}`); processedMessages = []; } @@ -679,7 +966,15 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr // #1954: stripping is only lossless when input already holds the full // conversation. Warn when we strip a non-empty id so a native-chaining // (delta) continuation that loses its history is visible, not silent 200s. - if (!knobKeepResponseId()) { + // 2×2 Part B: the id is governed by the delta chain this proxy established + // (hadFrontier). NO prior frontier → native #1954 semantics: KEEP=1 + // forwards the id intact regardless of what the outbound is (seed / full + // view / legacy — the pinned contract). WITH a prior frontier the provider + // retains exactly the sent elements, so only a PARTIAL delta keeps the id; + // a reseed / full-resend / degraded-legacy round STRIPS it (the chain + // breaks, the client re-chains from this round's fresh response id and the + // frontier re-seeds — one-round convergence, acceptance #7). + if (!knobKeepResponseId() || (hadFrontier && !deltaServed && !noopServed)) { const chainWarn = strippedResponseIdWarning(rebuilt.previous_response_id); if (chainWarn) log("warn", `[${sessionId}] ${chainWarn}`); delete rebuilt.previous_response_id; @@ -735,10 +1030,12 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const responsesPairOrigin = billingUpstream ?? upstreamOrigin; if (responsesPairOrigin) session.stats.lastLocalTextEstimateOrigin = responsesPairOrigin; } - snapshotMessages(session, originalMessages); // #3397 durability-before-render (see prepare-anthropic.ts): rows durable NOW, before the model call. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); // #564/#2627: ONE serialization of the outbound body. The same string is // both what we send and what the separator manifest binds to, so the claims // can never be attributed to a body that differs by even one byte. @@ -757,6 +1054,13 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr judgeBaseline, ...(separatorManifest !== undefined ? { separatorManifest } : {}), responsesProjection, + // #6181 B4: commit the selected frontier only after the upstream + // accepts the request (relay.ts); a failed forward drops it. + ...(deltaState !== undefined && selectionFired ? { pendingResponsesDelta: pendingDelta } : {}), + // #6181 round 2, Residual 2 hardening: the relay pairs this with + // the response id observed on the outbound stream (accepted forwards + // only) and records session.lastMintedResponse. + ...(mintFrontierPos !== undefined ? { mintFrontierPos } : {}), systemNotes: sysNotes, protocol: "responses", stream, diff --git a/src/server/relay.ts b/src/server/relay.ts index a9123e7253..87a189ba71 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -327,6 +327,51 @@ export function tryAnswerEarlyHold(o: { return true; } +// #6181 round 2, Residual 2 hardening: scans the OUTBOUND SSE for the response +// id the client actually chains from (response.created / response.completed). +// The event type rides in the data JSON for real upstreams +// ({"type":"response.completed",...}); some clients/tests emit it on the +// `event:` line instead — accept both. Feeds are cheap (bounded tail, string +// ops only) so the scan can ride a tee'd twin of any streaming branch. +class MintIdTap { + private tail = ""; + created?: string; + completed?: string; + feed(text: string): void { + this.tail += text; + if (this.tail.length > 65536) this.tail = safeSuffix(this.tail, 32768); + if (!this.tail.includes("response.completed") && !this.tail.includes("response.created")) return; + let cut = this.tail.indexOf("\n\n"); + while (cut !== -1) { + const event = this.tail.slice(0, cut); + this.tail = this.tail.slice(cut + 2); + // The recompute must run on EVERY pass — the marker check below + // `continue`s, and skipping the recompute there pins `cut` on a + // shrinking tail: a non-marker event preceding a marker event + // (e.g. response.output_item.done before response.completed) + // spins the loop forever on an empty tail. + cut = this.tail.indexOf("\n\n"); + if (!event.includes("response.completed") && !event.includes("response.created")) continue; + const lines = event.split("\n"); + const dataLine = lines.find((l) => l.startsWith("data: ")); + const eventLine = lines.find((l) => l.startsWith("event: ")); + if (dataLine !== undefined) { + try { + const evt = JSON.parse(dataLine.slice(6)) as { type?: string; response?: { id?: unknown } }; + const id = evt.response?.id; + if (typeof id === "string" && id.length > 0) { + const type = evt.type ?? (eventLine !== undefined ? eventLine.slice(7).trim() : ""); + if (type === "response.completed") this.completed = id; + else if (type === "response.created") this.created = id; + } + } catch { + /* partial/corrupt event — ignore */ + } + } + } + } +} + export function forward(...args: Parameters): Promise { return withCacheCapture(args[4]?.session, () => forwardCaptured(...args)); } @@ -1281,6 +1326,36 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare try { upstreamResult = await fetchPromise; firstAttemptClear = upstreamResult.clearTimer; + // #6181 B4: commit the delta frontier ONLY after the upstream + // accepts the request (fetch resolves on 4xx too — check the + // status). A 4xx/failed forward DROPS the pending state: the + // next round re-seeds (at-least-once — over-serving the full + // view is safe, under-serving is the context loss). The pending + // field is consumed at this single point (the first forward of + // the turn; compress-loop re-requests carry full views). + if (prepared && prepared.pendingResponsesDelta !== undefined) { + const accepted = upstreamResult.response.status < 400; + const pending = prepared.pendingResponsesDelta; + prepared.session.responsesDelta = accepted ? (pending ? pending.state : null) : null; + // #6181 review round 2, Residual 2 companion (B4 hardening): + // a REFUSED selection means the provider never ran that round — + // its chain holds at most the previously-accepted world while + // the plane already holds more (the in-request flush wrote the + // refused turn's rows before the forward). While broken, the + // first-chained-round frontier seed is blocked and the round + // re-seeds the FULL view (at-least-once). A full-view or noop + // acceptance re-proves the chain holds the whole world and + // clears the flag; a partial acceptance leaves it as-is (it + // cannot be set — a partial needs a valid frontier, and the + // broken state forces a reseed instead). + if (accepted) { + if (pending && pending.kind !== "partial") prepared.session.responsesChainBroken = false; + } else { + prepared.session.responsesChainBroken = true; + } + markDirty(prepared.session); + prepared.pendingResponsesDelta = undefined; + } } finally { clearTimeout(firstByteTimer); } @@ -2285,6 +2360,27 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare pluginBody = a; dumpRaw = dumpStreamToFile(b, opts.dumpSse, `${Date.now()}-${safeSessionId(prepared.session.id)}-raw.sse`); } + // #6181 round 2, Residual 2: the mint scan (the record that lets + // the NEXT chained round prove the provider chain holds the + // stored world) must ride this branch — plugin turns return + // before the proxy-lane taps. The client-facing bytes here ARE + // the upstream bytes (the strip pipes are byte-identical for + // tag-free streams; a response id is never a render tag), so a + // raw tap sees exactly what the client chains from. + let mintTap: MintIdTap | undefined; + let mintDone: Promise | undefined; + if (prepared.stream && prepared.protocol === "responses" && prepared.mintFrontierPos !== undefined && upstreamResult.response.status < 400) { + const [mMain, mTwin] = pluginBody.tee(); + pluginBody = mMain; + mintTap = new MintIdTap(); + mintDone = (async () => { + try { + for await (const c of mTwin) mintTap?.feed(Buffer.from(c).toString("utf8")); + } catch { + /* upstream cut — the client-facing branch decides */ + } + })(); + } if (prepared.stream) { // #2328: what the pipes know about the upstream HTTP response — // first attempt's status/content-type (a resolveFakeCompletion @@ -2378,6 +2474,15 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare } else { await pipePluginJson(pluginBody, res, prepared.session, prepared.protocol, targetOrigin, absorbInstructed, wireBodyText, reply); } + if (mintTap !== undefined && mintDone !== undefined && upstreamResult.response.status < 400 && prepared.mintFrontierPos !== undefined) { + // Drain BEFORE the finally's session persist rides markDirty, + // so lastMintedResponse is recorded in the same save. + await mintDone; + const seen = mintTap.completed ?? mintTap.created; + if (seen !== undefined) { + prepared.session.lastMintedResponse = { id: seen, frontierPos: prepared.mintFrontierPos }; + } + } } finally { clearUpstreamTimer(); // #2347: finish the dump before the turn closes — its value is the @@ -2475,6 +2580,30 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare prepared !== null && prepared.compressInjected && prepared.processedMessages.length > 0; + // #6181 round 2, Residual 2 hardening: the mint scan above lives in the + // compress-loop (injected) branch — but a non-injected responses turn + // (small context, classifier bypass, ACP_NO_INJECT_TOOL) never enters it, + // so session.lastMintedResponse was never recorded and the first chained + // round re-seeded the whole world. Tee a twin of the raw stream for those + // turns: here the client-facing bytes ARE the upstream bytes (the strip + // pipes are byte-identical for tag-free streams, and a response id is + // never a render tag), so the raw tap sees exactly what the client sees. + let plainMintTap: MintIdTap | undefined; + let plainMintDone: Promise | undefined; + let plainMintFrontier: number | undefined; + if ( + !useRewriter && prepared !== null && prepared.stream + && prepared.protocol === "responses" && prepared.mintFrontierPos !== undefined + && upstreamResult.response.status < 400 + ) { + plainMintFrontier = prepared.mintFrontierPos; + const [mintMain, mintTap] = responseBody.tee(); + responseBody = mintMain; + plainMintTap = new MintIdTap(); + plainMintDone = (async () => { + for await (const c of mintTap) plainMintTap?.feed(Buffer.from(c).toString("utf8")); + })(); + } if (!useRewriter || prepared === null) { // #2328: pipes cite the first upstream response's HTTP identity. const upstreamMeta: UpstreamMeta = { @@ -2615,6 +2744,15 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare } // #2347: the raw-SSE dump is finished in the safety-net finally below — // an inline await here would be skipped when a pipe rejects. + // #6181: drain the mint twin BEFORE the finally's markDirty persists + // the session, so lastMintedResponse rides the same save. + if (plainMintTap !== undefined && plainMintDone !== undefined && prepared !== null && plainMintFrontier !== undefined) { + await plainMintDone; + const seen = plainMintTap.completed ?? plainMintTap.created; + if (seen !== undefined) { + prepared.session.lastMintedResponse = { id: seen, frontierPos: plainMintFrontier }; + } + } return; } const ctx: RewriteCtx = { @@ -2779,6 +2917,17 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare clientAbort.signal, ); let protocolFragmentWarned = false; + // #6181 round 2, Residual 2 hardening: capture the response id + // the CLIENT actually sees (response.created / response.completed + // in the outbound SSE — the upstream's id when present, the + // proxy's minted one otherwise, inline-minted included: the scan + // runs on the exact bytes written to the client). The relay pairs + // it with prepared.mintFrontierPos into session.lastMintedResponse. + const mintTap = new MintIdTap(); + const mintScan = (text: string): void => { + if (prepared.protocol !== "responses" || prepared.mintFrontierPos === undefined) return; + mintTap.feed(text); + }; for await (const chunk of loop) { if (res.destroyed || res.writableEnded) break; { @@ -2791,12 +2940,25 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare } } res.write(chunk); + mintScan(chunk.toString("utf8")); if (res.writableNeedDrain) { await awaitDrain(res); } if (res.destroyed || res.writableEnded) break; } res.end(); + // #6181 round 2, Residual 2 hardening: pair the observed id with + // the frontier the lane recorded — accepted forwards only (a + // refused round mints nothing the client chains from) and only + // when the lane proved this round leaves the provider holding the + // full world (mintFrontierPos set). The markDirty in the finally + // below persists it with the rest of the turn's state. + if (prepared.protocol === "responses" && upstreamResult.response.status < 400 && prepared.mintFrontierPos !== undefined) { + const seen = mintTap.completed ?? mintTap.created; + if (seen !== undefined) { + prepared.session.lastMintedResponse = { id: seen, frontierPos: prepared.mintFrontierPos }; + } + } } catch (e) { emitStreamError(res, prepared.commandcode ? "commandcode" : prepared.protocol, (e as Error)?.message ?? String(e), (m) => log("error", `[${prepared.session.id}] ${m}`), opts.streamErrorShape); } finally { @@ -2906,6 +3068,14 @@ const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepare await withSessionLock(prepared.session, () => rewriteJsonResponseAsync(json, ctx, clientAbort.signal)); } captureHandoffJson(reply, prepared.protocol, json); + // #6181 round 2, Residual 2 hardening: non-streaming twin of + // the streaming mint capture — json.id here is the final + // client-visible id (post rewriteResponsesJsonResponseAsync). + if (prepared.protocol === "responses" && upstreamResult.response.status < 400 && prepared.mintFrontierPos !== undefined) { + if (typeof json.id === "string" && json.id.length > 0) { + prepared.session.lastMintedResponse = { id: json.id, frontierPos: prepared.mintFrontierPos }; + } + } reply?.save(); res.end(JSON.stringify(json)); } catch { diff --git a/src/server/snap-egress.ts b/src/server/snap-egress.ts new file mode 100644 index 0000000000..a0962ed131 --- /dev/null +++ b/src/server/snap-egress.ts @@ -0,0 +1,502 @@ +/** + * #5874 mapEgress — direct-authority egress from the verified snapshot plane. + * + * Owner ruling (2026-11-11): the snapshot plane is the DIRECT outbound + * authority, not a "verified egress source" that gets compared against the + * legacy bytes every round. In the uncompressed steady state a new message, + * once converted, is appended to the snapshot and served from it — no + * per-round diff, no tripwire, no seam-walk re-alignment. + * + * The plane stores the kernel's OWN tagged processTurn output (invariant v: + * post_json is the post-rewrite core form — the kernel's render-refs output, + * never wire bytes). The write side persists exactly the bytes the kernel + * saw, so the egress stage mirrors the kernel's prune/rebuild step over the + * plane rows (surviving ledger rows, including covered ones — the pair-safe + * walk needs them — plus each active block's stored anchor row) and serves + * the result through the lane's transform (absorb → attach → strip → drop, + * identical to what the lane runs on the legacy view). + * + * The legacy processedMessages still exist for the ONE-round crash-grade + * fallback: if the plane is dead, uninitialized for this sid, or the assembly + * throws, this round serves the legacy bytes and the next round re-decides. + * There is no sticky exclusion and no normalized comparison — the plane bytes + * are the transform input, so the output is byte-identical to the legacy + * egress by construction. + */ +import { baseIdOf, isRenderedSummaryMessage, SUMMARY_HEADER, summaryMessageId, type CompressionState } from "acp-kernel"; +import type { BiliMessage } from "acp-kernel/wire"; +import { loadConfigFile } from "../config.js"; +import { getStore } from "../persist.js"; +import { isSnapPlaneDead, type SnapView } from "../storage/snap.js"; +import { selectOutbound, isFullResend, type DeltaState, type DeltaSelection, type EgressElement } from "../snapshot/delta.js"; + +export type SnapEgressProtocol = "openai" | "anthropic" | "google" | "responses"; + +interface SnapEgressDeps { + sessionId: string; + /** Wire-protocol family, gates the cohort check. */ + protocol: SnapEgressProtocol; + log: (level: string, msg: string) => void; + /** Lane-identical post-processing for the plane view (absorb → attach → + * strip → drop, exactly what the lane applied to the legacy view). The + * plane rows ARE the lane's transform input, so applying this transform + * reproduces the legacy bytes by construction. */ + transform: (msgs: BiliMessage[]) => BiliMessage[]; + /** #4311: which tool names are compress calls (kernel config + * isCompressTool). Gates the visible-compress-call anchor skip. */ + isCompressTool?: (name: string) => boolean; + /** 2×2 Part B (出站增量): the persisted delta state. Present only for + * lanes whose target protocol has provider-side retention (Responses + * chaining); undefined = snapshot outbound (send the whole active view, + * the existing #5901 behavior). See src/snapshot/delta.ts. */ + deltaState?: DeltaState | null; + /** Called once per successful plane egress with the selection outcome so + * the lane can persist the new frontier (only after the outbound + * succeeds — a failed round keeps the old state and re-sends the same + * delta: at-least-once). */ + onSelection?: (sel: DeltaSelection, totalElements: number) => void; + /** #6181 B0: this turn cannot be addressed against the plane (a + * non-cohort chained delta whose kernel view is a delta, not the + * world) — serve the LEGACY bytes verbatim for this turn: the plane + * is one turn stale (its write was skipped), and serving it would + * drop the new input element. The legacy bytes are exactly what + * master serves for this turn (the delta the provider chains from). */ + bypass?: boolean; +} + +// --------------------------------------------------------------------------- +// Gate (config-file only — #2030: no env knob; opt-in, #5874 §9). +// --------------------------------------------------------------------------- + +const DEFAULT_COHORTS: ReadonlySet = new Set(["openai", "anthropic", "google", "responses"]); + +function snapEgressFlags(): { egress: boolean; cohorts: ReadonlySet } { + const file = loadConfigFile(); + const v = file.storage?.snapBase; + // Opt-in (#5874 §9): the egress transform ships behind an explicit + // storage.snapBase.egress=true (the 2026-11-08 soak ON default was + // reverted before the merge, 2026-11-13). + const egress = v?.egress === true; + const cohorts = Array.isArray(v?.egressCohorts) && v.egressCohorts.length > 0 + ? new Set(v.egressCohorts as SnapEgressProtocol[]) + : DEFAULT_COHORTS; + return { egress, cohorts }; +} + +// --------------------------------------------------------------------------- +// Counters (process-lifetime; surfaced via /__bili/status → snaplane.egress). +// --------------------------------------------------------------------------- + +const degradeReasons = new Map(); +const egressStats = { plane: 0, degrade: 0, planeMiss: 0 }; + +interface SnapEgressStatusWire { + plane: number; + degrade: number; + planeMiss: number; + degradeReasons: Record; +} + +export function snapEgressStatusWire(): SnapEgressStatusWire { + return { + ...egressStats, + degradeReasons: Object.fromEntries(degradeReasons), + }; +} + +export function _resetSnapEgressForTests(): void { + degradeReasons.clear(); + egressStats.plane = 0; + egressStats.degrade = 0; + egressStats.planeMiss = 0; +} + +function noteDegrade(reason: string): void { + egressStats.degrade++; + degradeReasons.set(reason, (degradeReasons.get(reason) ?? 0) + 1); +} + +// --------------------------------------------------------------------------- +// View assembly (proposal stage 4). Mirrors the kernel's prune/rebuild step +// EXACTLY (kernel/src/prune.ts) over the S4-verified plane rows: the +// non-anchor rows (surviving ledger rows, covered ones included — the +// pair-safe walk and the coverage skips need them present, same as in the +// kernel input) are the walk list; each active block's stored anchor row is +// moved to the same position the kernel's renderSummary would take (earliest +// covered id → pair-safe over the walk list → pin/fallback clamp). Covered +// and stale-rendered-summary rows are skipped exactly as the kernel skips +// them (base-id coverage + #4316 layer-3 ledgerSeq intervals + known-summary +// drop), the pinned opening wire message is served even when covered +// (#2733/#3642), and the #4311 visible-compress-call skip applies. +// --------------------------------------------------------------------------- + +interface EgressRow { + pos: number; + ids: Set; + msg: BiliMessage; // parsed post_json + active: number; + isAnchor: boolean; // coreIds carry an acp_summary_ id (the stored render) + coverRef: string | null; // the block that compacted this row (plane-native coverage) +} + +/** prune.ts pairing parity: an anchor takes the EARLIEST covered message's + * position, then moves back to the assistant-run start / past the paired + * tool result so it never splits a wire-validated unit. Walks the FULL row + * list — covered rows are still present there (same reasoning as prune: + * covered cores between wire-relevant cores are dropped and the runs merge). */ +function pairSafeAnchorIndex(rows: EgressRow[], earliest: number): number { + const n = rows.length; + const at = (i: number) => rows[i].msg; + // runStart: start of the maximal consecutive assistant-role run containing i. + const runStart = new Int32Array(n).fill(-1); + for (let i = 0; i < n;) { + if (at(i)?.role !== "assistant") { i++; continue; } + let j = i; + while (j + 1 < n && at(j + 1)?.role === "assistant") j++; + for (let k = i; k <= j; k++) runStart[k] = i; + i = j + 1; + } + // prefixEnd[i]: farthest (first result index of an assistant tool-call + // before i) + 1. + const resultIndexByCallId = new Map(); + for (let i = 0; i < n; i++) { + const m = at(i); + if (m?.contentType === "tool-result" && typeof m.toolCallId === "string" && !resultIndexByCallId.has(m.toolCallId)) { + resultIndexByCallId.set(m.toolCallId, i); + } + } + const prefixEnd = new Int32Array(n + 1); + for (let i = 0; i < n; i++) { + const m = at(i); + let end = 0; + if (m?.role === "assistant" && m.contentType === "tool-call" && typeof m.toolCallId === "string") { + const r = resultIndexByCallId.get(m.toolCallId); + if (r !== undefined) end = r + 1; + } + prefixEnd[i + 1] = Math.max(prefixEnd[i]!, end); + } + let safe = earliest; + if (safe > 0 && safe < n) { + const start = runStart[safe]!; + if (start >= 0 && at(safe - 1)?.role === "assistant") safe = start; + } + if (safe < 0 || safe > n) return safe; + for (let guard = 0; guard < n; guard++) { + const next = Math.max(safe, prefixEnd[safe]!); + if (next === safe) break; + safe = next; + } + return safe; +} + +/** renderSummary parity (kernel/src/prune.ts): topic header line + body. + * Fallback only — the plane normally stores the kernel's own anchor row and + * it is served verbatim; this is used when a block's anchor row is somehow + * absent from the plane. */ +function renderAnchor(b: CompressionState["blocks"][number]): BiliMessage { + const body = b.summary.trim(); + const topicLine = b.topic ? `${SUMMARY_HEADER} — ${b.topic}` : SUMMARY_HEADER; + const text = body.length === 0 ? topicLine : `${topicLine}\n${body}`; + return { id: summaryMessageId(b.blockId), role: "system", contentType: "text", text } as BiliMessage; +} + +interface AnchorSeed { + insertAt: number; + unresolved: boolean; + msg: BiliMessage; + /** The stored anchor row's (negative) plane pos — 0 when synthesized. + * Carried to the side channel so the incremental-outbound selector can + * track which anchors were already sent. */ + pos: number; +} + +/** Per-element bookkeeping for the assembled egress view (parallel to + * `msgs`). The snapshot outbound ignores it; the incremental outbound + * (2×2 Part B, src/snapshot/delta.ts) uses it to pick the unsent suffix. */ +interface EgressElementSide { + /** The plane row pos (append-only). Anchors carry their negative pos. */ + pos: number; + /** true for rendered summary anchors. */ + isAnchor: boolean; + /** The anchor summary id (isAnchor only). */ + anchorId?: string; +} + +/** Assemble the outbound view from the S4-verified plane (kernel prune/ + * rebuild parity — see the section header). */ +export function buildSnapEgressView( + view: SnapView, + state: CompressionState, + isCompressTool: (name: string) => boolean, +): { ok: true; msgs: BiliMessage[]; n: number; side: EgressElementSide[] } | { ok: false; reason: string } { + if (!view.rows.length) return { ok: false, reason: "no-plane" }; + + // Parse EVERY row (active and covered): the covered rows stay in the walk + // list for pairing + coverage parity, exactly as in the kernel input. + const rows: EgressRow[] = []; + for (const r of view.rows) { + if (typeof r.postJson !== "string" || !r.postJson) { + return { ok: false, reason: "row-unparsable" }; + } + let msg: BiliMessage; + try { + msg = JSON.parse(r.postJson) as BiliMessage; + } catch { + return { ok: false, reason: "row-unparsable" }; + } + const ids = new Set(r.coreIds); + rows.push({ pos: r.pos, ids, msg, active: r.active, coverRef: r.coverRef, isAnchor: [...ids].some((id) => id.startsWith("acp_summary_")) }); + } + // The kernel's input list (rows-sourced D views suppress covered rows, + // client-sourced views carry them but the kernel prune drops them) + // reduces to the plane's ACTIVE non-anchor rows: the write side maps + // exactly the kernel output onto the plane and compacts everything the + // kernel covered. Covered/inactive rows therefore take NO part in + // pairing or serving — coverage suppression by active flag, not by + // alias matching (an alias match mis-fired on a FRESH reminted row + // whose core_ids carried a covered base id — the #4052 fresh-repeat + // message vanished from egress while the kernel served it). + const nonAnchor = rows.filter((r) => !r.isAnchor && r.active === 1); + const anchorMsgById = new Map(); + for (const r of rows) { + if (!r.isAnchor) continue; + for (const id of r.ids) if (id.startsWith("acp_summary_")) anchorMsgById.set(id, { msg: r.msg, pos: r.pos }); + } + + const n = nonAnchor.length; + const isCompress = isCompressTool; + // Kernel parity: prune() returns the input UNTOUCHED (no anchors, no + // coverage skips, no stale-summary drop) when no active block covers + // any message — the whole pipeline is a no-op then. + const coveredIds = new Set(); + for (const b of state.blocks) if (b.active) for (const id of b.effectiveMessageIds) coveredIds.add(id); + if (coveredIds.size === 0) { + return { + ok: true, + msgs: nonAnchor.map((r) => r.msg), + n, + side: nonAnchor.map((r) => ({ pos: r.pos, isAnchor: false })), + }; + } + // #4311: a compress tool-call visible in the input suppresses its block's + // separate summary render. + const visibleCompressCallIds = new Set(); + for (const r of nonAnchor) { + const m = r.msg; + if (m.contentType === "tool-call" && typeof m.toolCallId === "string" && typeof m.toolName === "string" && isCompress(m.toolName)) { + visibleCompressCallIds.add(m.toolCallId); + } + } + // Stale rendered-summary copies: the kernel drops a resented summary of a + // block it (re)renders itself — all blocks, active or consumed. + const knownSummaryIds = new Set(state.blocks.map((b) => summaryMessageId(b.blockId))); + const firstUserIndex = nonAnchor.findIndex((r) => r.msg.role === "user"); + let lastUserIndex = -1; + for (let i = n - 1; i >= 0; i--) { + if (nonAnchor[i]!.msg.role === "user") { lastUserIndex = i; break; } + } + // #3642: the pin run — the first user's wireIndex core run (the kernel + // serves it even when covered). + let firstUserPinEnd = firstUserIndex >= 0 ? firstUserIndex + 1 : -1; + if (firstUserIndex >= 0) { + // #3642: wireIndex is a runtime core field (codec-set, not on the + // CoreMessage type) — kernel parity via local cast. + const w = (nonAnchor[firstUserIndex]!.msg as { wireIndex?: number }).wireIndex; + if (typeof w === "number") { + let j = firstUserIndex + 1; + while (j < n && nonAnchor[j]!.msg.role === "user" && (nonAnchor[j]!.msg as { wireIndex?: number }).wireIndex === w) j++; + firstUserPinEnd = j; + } + } + // base id → EARLIEST index (kernel baseIndexById parity, plane-extended: + // every alias of a row seeds the index, so a reminted row — whose msg.id + // no longer matches the fold-time block effIds — still resolves through + // its old alias kept in core_ids). + const baseIndexById = new Map(); + for (let i = 0; i < n; i++) { + for (const id of nonAnchor[i]!.ids) { + const base = baseIdOf(id); + if (!baseIndexById.has(base)) baseIndexById.set(base, i); + } + } + // Rendered summary id → index (LAST occurrence wins — kernel parity). + const summaryIndexById = new Map(); + for (let i = 0; i < n; i++) { + if (isRenderedSummaryMessage(nonAnchor[i]!.msg)) summaryIndexById.set(nonAnchor[i]!.msg.id, i); + } + // Covered run → anchor position (kernel parity: "a rendered summary + // replaces the messages its block covers, so it takes the position of + // the EARLIEST covered message" — prune.ts collectSummaryAnchors). A + // folded block's rows are INACTIVE on the plane (the write side compacts + // them with cover_ref = block id), so they are absent from the pairing + // list above; the anchor takes the first pairing row strictly ABOVE the + // covered run, and the pin clamp below moves it past the pinned run. + // + // The covered run is read from the rows' OWN cover_ref — the plane-native + // ground truth the write side stamped at compact time. It is deliberately + // NOT resolved through the block's effectiveMessageIds: on an + // identity-seed upgrade the reconcile re-keys the block to the client's + // re-minted ids (#3752), and those ids first-seen on the plane are the + // re-appended covered DUPLICATES (fresh ledger seqs, above every live + // row) — resolving through them parks the anchor at the tail instead of + // after the original covered run. cover_ref is immune: the original + // covered rows keep it across remints, and MIN over the run is the + // earliest covered position (a fold covers one contiguous run, so min + // and max give the same "first row above" for the original run; the + // min is what disambiguates when remint duplicates exist). + const coverRefMinPos = new Map(); // block id → earliest covered pos + for (const r of rows) { + if (r.isAnchor || r.coverRef === null) continue; + const prev = coverRefMinPos.get(r.coverRef); + if (prev === undefined || r.pos < prev) coverRefMinPos.set(r.coverRef, r.pos); + } + // Anchor seeds (kernel collectSummaryAnchors parity: active blocks only; + // direct range preferred over merged effective, #2733). + const fallbackIndex = lastUserIndex >= 0 ? lastUserIndex : n; + const anchors: AnchorSeed[] = []; + for (const b of state.blocks) { + if (!b.active) continue; + // #5765 retire parity (kernel collectSummaryAnchors / D-lane render): a + // retired block stays active but its summary is intentionally NOT + // re-rendered — its covered rows stay covered (suppressed here by the + // active flag, exactly as the D-lane does). + if (b.retired === true) continue; + if (b.compressCallId !== undefined && visibleCompressCallIds.has(b.compressCallId)) continue; + const sid = summaryMessageId(b.blockId); + const existing = summaryIndexById.get(sid); + const stored = anchorMsgById.get(sid); + const msg = stored?.msg ?? renderAnchor(b); + const anchorPos = stored?.pos ?? 0; + if (existing !== undefined) { + anchors.push({ insertAt: existing, unresolved: false, msg, pos: anchorPos }); + continue; + } + const anchorIds = b.directMessageIds && b.directMessageIds.length > 0 ? b.directMessageIds : b.effectiveMessageIds; + const covLo = coverRefMinPos.get(b.blockId); + let insertAt: number; + let unresolved: boolean; + if (covLo !== undefined) { + // First pairing (active) row strictly above the covered run. + insertAt = 0; + while (insertAt < n && nonAnchor[insertAt]!.pos <= covLo) insertAt++; + unresolved = false; + } else { + let earliest: number | null = null; + for (const id of anchorIds) { + const idx = baseIndexById.get(baseIdOf(id)); + if (idx === undefined) continue; + if (earliest === null || idx < earliest) earliest = idx; + } + insertAt = earliest ?? fallbackIndex; + unresolved = earliest === null; + } + anchors.push({ insertAt, unresolved, msg, pos: anchorPos }); + } + // Pair-safe adjustment + pin/fallback clamp (kernel rebuildMessages + // parity: non-head-anchors clamp past the pin when resolved, to the first + // user when unresolved). + for (const a of anchors) { + a.insertAt = pairSafeAnchorIndex(nonAnchor, a.insertAt); + const clamp = a.unresolved ? firstUserIndex : firstUserPinEnd; + if (clamp >= 0) a.insertAt = Math.max(a.insertAt, clamp); + } + anchors.sort((a, b) => a.insertAt - b.insertAt); + + // The walk (kernel rebuildMessages parity): anchors at their index, the + // pinned run verbatim, covered / positional / stale-summary skips. + const merged: BiliMessage[] = []; + const side: EgressElementSide[] = []; + const pushAnchor = (a: AnchorSeed) => { + merged.push(a.msg); + side.push({ pos: a.pos, isAnchor: true, anchorId: a.msg.id }); + }; + const pushRow = (r: EgressRow) => { + merged.push(r.msg); + side.push({ pos: r.pos, isAnchor: false }); + }; + let ai = 0; + for (let i = 0; i < n; i++) { + while (ai < anchors.length && anchors[ai]!.insertAt === i) pushAnchor(anchors[ai++]!); + if (i === firstUserIndex && firstUserIndex >= 0) { + for (let k = i; k < firstUserPinEnd && k < n; k++) pushRow(nonAnchor[k]!); + i = firstUserPinEnd - 1; + continue; + } + const r = nonAnchor[i]!; + const m = r.msg; + // nonAnchor is the active set already — covered rows never reach + // the walk (see the filter above). + if (isRenderedSummaryMessage(m) && knownSummaryIds.has(m.id)) continue; + pushRow(r); + } + while (ai < anchors.length) pushAnchor(anchors[ai++]!); + return { ok: true, msgs: merged, n: merged.length, side }; +} + +// Entry point — called by each protocol lane right after the lane computed +// its legacy processedMessages (the one-round crash-grade fallback). The +// plane view, once assembled and transformed, is the outbound source +// directly — no per-round comparison against the legacy bytes. +// --------------------------------------------------------------------------- + +export function applySnapEgress( + state: CompressionState, + legacyProcessed: BiliMessage[], + deps: SnapEgressDeps, +): BiliMessage[] { + const flags = snapEgressFlags(); + if (!flags.egress || !flags.cohorts.has(deps.protocol)) return legacyProcessed; + if (isSnapPlaneDead(deps.sessionId)) return legacyProcessed; + if (deps.bypass) return legacyProcessed; // #6181 B0 — one turn; the plane stays stale-but-intact + + let view: SnapView | null; + try { + view = getStore()?.snapEgressView(deps.sessionId) ?? null; + } catch { + view = null; + } + if (!view) { + egressStats.planeMiss++; + return legacyProcessed; // plane not initialized for this sid yet — nothing to serve + } + + // Never-fail contract: the plane path must degrade, never throw. A throw + // escaping here reached the lane catch, which forwarded the UNPROCESSED + // client bytes — kernel render, watermark and egress all skipped — and + // corrupted the round (found in CI: a media-offload object fed to a hash). + try { + const built = buildSnapEgressView(view, state, deps.isCompressTool ?? ((name: string) => name === "compress")); + if (!built.ok) { + noteDegrade(built.reason); + return legacyProcessed; + } + // 2×2 Part B: incremental outbound. Both modes read the SAME plane + // view — the difference is only how many elements go over the wire. + let outMsgs = built.msgs; + let selection: DeltaSelection | null = null; + if (deps.deltaState !== undefined) { + const elements: EgressElement[] = built.msgs.map((m, i) => ({ + msg: m, + pos: built.side[i]!.pos, + isAnchor: built.side[i]!.isAnchor, + anchorId: built.side[i]!.anchorId, + })); + selection = selectOutbound(elements, deps.deltaState); + if (!selection.reseed && !isFullResend(selection, elements)) { + const keep = new Set(selection.delta.map((e) => e.msg)); + outMsgs = built.msgs.filter((m) => keep.has(m)); + } + deps.onSelection?.(selection, elements.length); + } + const planeProcessed = deps.transform(outMsgs); + // DIRECT AUTHORITY: the plane view is the outbound source. The legacy + // bytes exist only as the crash-grade fallback above. + egressStats.plane++; + deps.log("info", `[snaplane] sid=${deps.sessionId} egress=plane n=${planeProcessed.length}`); + return planeProcessed; + } catch (err) { + noteDegrade("egress-assembly-error"); + deps.log("warn", `[snaplane] sid=${deps.sessionId} decision=degrade:egress:assembly-error (${err instanceof Error ? err.message : String(err)}) — legacy egress kept`); + return legacyProcessed; + } +} diff --git a/src/server/snapshot-render.ts b/src/server/snapshot-render.ts index ecbc57ed68..ed24734130 100644 --- a/src/server/snapshot-render.ts +++ b/src/server/snapshot-render.ts @@ -68,18 +68,36 @@ const MIN_SEAM_RUN = 2; /** #3658 canonical identity signature minus the hash: the tuple the kernel * derives message ids from. Equal signature ⇒ same logical message even * when the id bytes were re-minted by an upgrade. Image cores share the - * "[image]" text, so the sidecar length rides the signature (same image ⇒ - * same bytes ⇒ same length; different images practically always differ). */ + * "[image]" text, so the image BYTES ride the signature (see #4390). */ export function identitySig(m: BiliMessage): string { const text = typeof m.text === "string" ? m.text : ""; // #4390: hash the image BYTES, never the length — same-size screenshots // used to collide and chain a false seam (silent truncation). The walk // only touches the tail region, so per-request hashing is bounded; a // 24-hex prefix is far beyond collision reach for seam evidence. - const img = m.imagePart === true ? `|img:${sha256Hex(m.imageBase64 ?? "").slice(0, 24)}` : ""; + const img = m.imagePart === true ? `|img:${imageSigPart(m.imageBase64)}` : ""; return `${m.role}|${m.contentType}|${m.toolName ?? ""}|${text}${img}`; } +/** The 24-hex image identity, robust to the media-OFFLOADED storage form: + * persisted rows carry {"$media": sha256Hex(bytes)} instead of the bytes + * (#3762), and that hash IS the identity — so both forms sign identically. + * Anything else degrades to a stable stringify. Never throws: a signature + * helper throwing takes the whole seam walk (and turn) down. */ +function imageSigPart(v: unknown): string { + if (typeof v === "string") return v.length > 0 ? sha256Hex(v).slice(0, 24) : "none"; + if (typeof v === "object" && v !== null) { + const h = (v as { $media?: unknown }).$media; + if (typeof h === "string" && h.length >= 24) return h.slice(0, 24); + try { + return `s${sha256Hex(JSON.stringify(v) ?? "").slice(0, 24)}`; + } catch { + return "unserializable"; + } + } + return "none"; +} + /** Test seam (#4390): the seam fingerprint under test — see identitySig. */ export function identitySigForTest(m: BiliMessage): string { return identitySig(m); diff --git a/src/session.ts b/src/session.ts index 33c9833c52..6cfeff65ca 100644 --- a/src/session.ts +++ b/src/session.ts @@ -685,6 +685,56 @@ export type Session = { * the state ranges). Cleared by snapshotMessages on the next live * request — the client re-sends full raw history, restoring the invariant. */ lastMessagesFolded?: boolean; + /** #5874 mapEgress direct authority: the RAW processTurn output for this + * turn (the kernel's tagged, pre-absorbView/attach view — exactly the + * bytes the egress stage re-derives no longer has to re-derive). Set as + * a deep copy right after processTurn (before the in-request flush), so + * the plane persists the kernel's OWN tagged output instead of the + * pre-rewrite transcript. The egress then reads the plane rows and runs + * only the light transform (absorb/attach/strip/drop) — no seam walk, + * no re-tagging, no id re-keying. Set per world-view round after + * processTurn; deliberately UNSET on #6181 B0 bypass rounds (the view + * there is a chained delta, not the world — a later debounced save + * riding it tagged would let snapViewEvents delete every plane row the + * delta lacks), so such saves fall back to the untagged DB final-state + * derivation. */ + lastPlaneMessages?: CoreMessage[]; + /** #6181 B0: one-shot "skip the snapshot-plane write on the next + * save" — set by the lane for a turn that CANNOT be addressed against + * the plane (a non-cohort chained delta whose kernel view is a delta, + * not the world): deriving events from it would delete every row the + * view does not contain. snapShadowSync clears it when it consumes it + * (one save, then the flag is gone). */ + skipPlaneFlush?: boolean; + /** 2×2 Part B (出站增量, responses lane only): the incremental-outbound + * frontier — the highest plane row pos + anchor summary ids the provider + * already has via its retained chain. Persisted with the session; lost + * state → next round reseeds (full send), one-round convergence. See + * src/snapshot/delta.ts. */ + responsesDelta?: { frontierPos: number; servedAnchors: string[] } | null; + /** #6181 review round 2, Residual 2 companion: the provider's chain may + * NOT be assumed to hold the stored world. Set when a delta-selection + * round was REFUSED upstream (the provider never ran it — its chain + * holds at most the previously-accepted world while the plane already + * holds more). While set, the first-chained-round frontier seed is + * blocked and the round re-seeds the FULL view (at-least-once — + * over-serving is safe, under-serving is the context loss). Cleared + * when a full-view round is accepted (the provider provably holds the + * whole world again). Transient within a turn? No — persisted with the + * session, like responsesDelta. */ + responsesChainBroken?: boolean; + /** #6181 round 2, Residual 2 hardening: the latest response id THIS + * proxy emitted for this session (responses lane), paired with the + * plane frontier the provider chain provably held after that response + * (committed only on an ACCEPTED forward — a refused round mints + * nothing the client chains from). The chain id itself is opaque: + * the first-chained-round frontier seed AND every persisted frontier + * are trusted only when previous_response_id equals this id (the host + * continues from our latest response — a fork/rewind to an older id, + * or an id another proxy minted, can carry a world that chain never + * held, so it re-seeds instead). Persisted with the session, like + * responsesDelta. */ + lastMintedResponse?: { id: string; frontierPos: number } | null; pluginSnapshot?: CoreMessage[]; /** #3301: original message decompositions supplied by an external state * converter (e.g. opencode-acp migration) at import time. Transient diff --git a/src/snapshot/assemble.ts b/src/snapshot/assemble.ts new file mode 100644 index 0000000000..2eb3edbb25 --- /dev/null +++ b/src/snapshot/assemble.ts @@ -0,0 +1,193 @@ +// 2×2 stage 0 — kernel-input assembly (brief 19544722, Part A core). +// +// The stage-0 verdict (identity.ts) tells us WHICH envelope elements are new +// and where every element sits (append-only pos). This module assembles the +// kernel input for the turn: +// +// kernelInput = the envelope in CLIENT ORDER, where each element resolves to +// (a) a FRESH core — the parsed NEW span, converted, in this same turn +// (the only parse of this turn: P5 — non-new rounds never re-parse), or +// (b) an OLD core — the plane row's stored kernel byte (post_json), +// hydrated, verbatim. The kernel re-renders its own tag idempotently +// (render-refs.ts:124-131) and the token count is a first-render +// snapshot, so a table-old core rides byte-stable (prefix cache). +// +// The kernel input order is the ENVELOPE (client) order, NOT the plane pos +// order: a scattered NEW span (a mid-history edit) lands at its conversation +// position, not at the tail. That is exactly the kernel input the full-parse +// path (M1) would have produced for the same client array — the byte +// identity M1 vs M2 (acceptance #2) falls out by construction. +// +// R4 (table gap): an envelope element that is neither new nor plane-resolvable +// is a stage-0 verdict we cannot honor — the table is behind the client. +// The lane catches the throw, resets the plane (snapStage0Reset) and serves +// the round from the legacy full path: one-round convergence, never a half +// conversation. + +import type { BiliMessage } from "acp-kernel/wire"; +import type { WorldDecision } from "./types.ts"; + +/** A plane row's kernel-input contribution: the stored tagged core, hydrated. */ +interface PlaneRowCore { + pos: number; + /** The row's bound element ids (core_ids, alias-chain included). */ + ids: string[]; + /** post_json hydrated (media CAS resolved) — the kernel byte. */ + msg: BiliMessage; +} + +interface AssembleInput { + /** The stage-0 verdict (identity.ts). */ + decision: WorldDecision; + /** The plane's ACTIVE rows, pos-ordered (the old cores). Inactive + * (fold-covered) rows are NOT part of the kernel input — the kernel + * folds them; their summary anchor is carried by the kernel state. */ + rows: readonly PlaneRowCore[]; + /** The NEW spans, parsed: the kernel-input form of each new element, in + * envelope order. Length must equal decision.newIdPos.length. */ + newCores: readonly BiliMessage[]; + /** Envelope ids that may legally have NO plane row and are NOT new: + * the fold-explained gone set (a fold absorbed the row — the kernel + * state carries the block, the anchor row carries the summary). These + * envelope entries are SKIPPED in the walk (the R4 table-gap throw is + * suppressed for them and only). */ + skipIds?: ReadonlySet; + /** The plane's ANCHOR rows (id = acp_summary_, active), in + * kernel-input form. Inserted after the walk at the covered-run + * boundary (coverMax). */ + anchors?: readonly EgressAnchor[]; + /** blockId → highest covered row pos (store.snapStage0Cores). The + * anchor renders before the first served element whose pos exceeds + * the covered run (the D-view placement parity). */ + coverMax?: Record; +} + +/** A plane anchor row in kernel-input form (the rendered summary message). + * `blockId` is the suffix of the anchor id (acp_summary_). */ +interface EgressAnchor { + msg: BiliMessage; + blockId: string; +} + +interface AssembleResult { + /** The assembled kernel input: envelope-ordered elements (old cores and + * fresh cores interleaved at their conversation positions) with the + * anchor rows inserted at their covered-run boundary. Feed this to + * processTurn. */ + kernelInput: BiliMessage[]; + /** The number of OLD element cores taken from the plane (anchor rows + * excluded — they are not ledger elements). */ + oldCount: number; +} + +/** + * Assemble the kernel input for a stage-0-armed turn. + * + * @throws when the plane cannot satisfy the verdict (R4-shaped): the caller + * resets the plane and serves the round from the legacy full path. + */ +export function assembleKernelInput(input: AssembleInput): AssembleResult { + const { decision, rows, newCores, skipIds, anchors, coverMax } = input; + if (newCores.length !== decision.newIdPos.length) { + throw new Error( + `[snapshot 2×2] assembleKernelInput: newCores length ${newCores.length} != decision.newIdPos length ${decision.newIdPos.length} (P5: parse exactly the NEW spans)`, + ); + } + // Anchor rows are plane rows in their own right — their ids must not be + // mistaken for element cores (an anchor id can never match an envelope + // element; excluding them keeps rowById element-only). + const anchorIds = new Set(); + for (const a of anchors ?? []) anchorIds.add(`acp_summary_${a.blockId}`); + const newRowById = new Map(); + decision.newIdPos.forEach(({ id }, i) => { + if (!newRowById.has(id)) newRowById.set(id, newCores[i]!); + }); + // Element id → active plane row core (first row in pos order whose core + // ids contain the element id — alias-safe: a row bound to several ids + // serves whichever the client echoes first). + const rowById = new Map(); + for (const row of rows) { + if (row.ids.some((id) => anchorIds.has(id))) continue; // anchor row + for (const id of row.ids) if (!rowById.has(id)) rowById.set(id, row.msg); + } + + // The walk in CLIENT (envelope) order, recording each element's plane + // pos so the anchors can be placed against the surviving sequence. + const kernelInput: BiliMessage[] = []; + const elementPos: number[] = []; // parallel to kernelInput + let oldCount = 0; + for (const e of decision.envelope) { + const fresh = newRowById.get(e.id); + if (fresh !== undefined) { + kernelInput.push(fresh); + elementPos.push(e.pos); + continue; + } + const old = rowById.get(e.id); + if (old !== undefined) { + kernelInput.push(old); + elementPos.push(e.pos); + oldCount++; + continue; + } + if (skipIds?.has(e.id)) { + // Fold-covered and fold-explained: the kernel state carries the + // block, the anchor row (inserted below) carries the summary. + continue; + } + // An envelope element that is neither new nor plane-resolvable: + // the stage-0 verdict says it is incoming, so this is a plane/table + // gap (R4-shaped) — refuse to assemble half a conversation. + throw new Error( + `[snapshot 2×2] assembleKernelInput: envelope element ${e.id} (pos ${e.pos}) is neither a new span nor a plane row (table behind the client — resync required)`, + ); + } + + // Anchor insertion (D-view placement parity): an anchor renders before + // the first served element whose pos exceeds its block's covered run + // (coverMax). No coverMax evidence (the covered rows were GC'd) → the + // view start (the kernel's "earliest covered message" fallback). The pin + // run (first user) is served verbatim by the kernel on top of this input, + // so the anchor never needs a pin clamp here. + const placed: Array<{ at: number, anchor: EgressAnchor }> = []; + for (const anchor of anchors ?? []) { + const coverHi = coverMax?.[anchor.blockId] ?? -1; + let at = 0; // default: view start + for (let i = 0; i < kernelInput.length; i++) { + if ((elementPos[i] ?? -1) > coverHi) { + at = i; + break; + } + } + placed.push({ at, anchor }); + } + if (placed.length > 0) { + placed.sort((a, b) => a.at - b.at); + const out: BiliMessage[] = []; + let ki = 0; + for (const p of placed) { + while (ki < p.at) out.push(kernelInput[ki++]!); + out.push(p.anchor.msg); + } + while (ki < kernelInput.length) out.push(kernelInput[ki++]!); + return { kernelInput: out, oldCount }; + } + return { kernelInput, oldCount }; +} + +/** + * The resync predicate (brief: the resync safety net). + * + * - `diverge` — the client's array structurally disagrees with the stored + * canon (head moved / order broken): the table cannot be trusted for + * THIS round → reset + re-issue from the client's current array. + * - `resync` — an explicit mode switch (the lane requested it). + * + * A plain `issue` with an EMPTY plane is the NORMAL seed (first turn of a + * fresh session) — NOT a resync: resetting there would wipe the just-issued + * canon and re-issue forever. R4 (table gap) surfaces through + * assembleKernelInput's throw instead — it is per-round, not structural. + */ +export function shouldResync(decision: WorldDecision, _planeRowCount: number): boolean { + return decision.mode === "diverge" || decision.mode === "resync"; +} diff --git a/src/snapshot/checksums.ts b/src/snapshot/checksums.ts new file mode 100644 index 0000000000..7d90b38c68 --- /dev/null +++ b/src/snapshot/checksums.ts @@ -0,0 +1,62 @@ +/** + * #5874 — snapshot-table compression foundation: checksums (S4 + canon). + * + * Pure, dependency-light. Every checksum in the foundation is a lowercase + * hex sha256 so the values are grep-able across the audit stream and the + * storage layer (they are stored verbatim in snap_rows / snap_ckpt / + * l1_world). The S4 per-row aggregate is the red-line invariant (iii): a + * mismatch means the base material is corrupt and the sid plane is PERMANENTLY + * dead (R1). + */ +import { createHash } from "node:crypto"; + +/** Lowercase-hex sha256 of a UTF-8 string. */ +function sha256hex(text: string): string { + return createHash("sha256").update(text, "utf8").digest("hex"); +} + +/** Canonical JSON for checksum inputs: sorted-object-key, no whitespace. This + * is the ONLY serialization the foundation checksums over, so a key-order + * shuffle (JSON.stringify non-determinism, client churn) cannot move a + * checksum. */ +export function canonicalJson(value: unknown): string { + return stableStringify(value); +} + +function stableStringify(value: unknown): string { + if (value === null || typeof value !== "object") { + if (typeof value === "string") return JSON.stringify(value); + return JSON.stringify(value) ?? "null"; + } + if (Array.isArray(value)) { + return "[" + value.map((v) => stableStringify(v)).join(",") + "]"; + } + const keys = Object.keys(value as Record).sort(); + const parts: string[] = []; + for (const k of keys) { + const v = (value as Record)[k]; + if (v === undefined) continue; + parts.push(JSON.stringify(k) + ":" + stableStringify(v)); + } + return "{" + parts.join(",") + "}"; +} + +/** sha256 of the canonical envelope JSON (the L1 world's fast-mismatch + * sentinel, §4.0.1). */ +export function envelopeChecksum(envelope: readonly string[]): string { + return sha256hex(canonicalJson(Array.from(envelope))); +} + +/** sha256 of one row's post form (S4 unit). The post form is the common-core + * JSON; checksumming the canonical form keeps it byte-churn-proof. */ +function rowPostChecksum(postJson: string): string { + let parsed: unknown; + try { + parsed = JSON.parse(postJson); + } catch { + // A non-JSON post form is itself a fault — checksum the raw bytes so + // the mismatch is still detected (and grep-able) rather than masked. + return sha256hex(postJson); + } + return sha256hex(canonicalJson(parsed)); +} diff --git a/src/snapshot/delta.ts b/src/snapshot/delta.ts new file mode 100644 index 0000000000..38e854a929 --- /dev/null +++ b/src/snapshot/delta.ts @@ -0,0 +1,137 @@ +/** + * #5874 / 2×2 — Part B: incremental outbound (seed / delta selection). + * + * The 出站协议改写 stage reads the SAME snapshot plane for every mode + * (brief 19544722): 出站 snapshot (M1/2) sends ALL active rows (the existing + * #5901 direct-authority egress — prefix bytes stable by construction); + * 出站增量 (M3/4) sends only what the provider has NOT seen yet: + * + * - the SEED round (the first outbound for the session, and the first + * after a resync / re-arm) must send the FULL view — the provider + * (Responses-style retention) has nothing to chain from. + * - a DELTA round sends only the new elements: active rows with + * pos > frontier (never sent) + anchor rows whose summary id was never + * sent. + * + * WHY THE PROVIDER'S CONTEXT GROWS MONOTONICALLY: a Responses-style + * provider retains every element it was ever sent (the chain is + * append-only — there is no delete API). Rows that later get fold-covered + * stay at the provider; the provider's effective conversation is therefore + * a SUPERSET of the current egress view. That is the documented trade-off + * of 出站增量 (the brief's acceptance #3: the CUMULATIVE rows sent — + * including the seed round — are byte-identical to the active-row set the + * snapshot mode sends). When the provider's window overflows, the next + * outbound RE-SEEDS (full view) and converges in one round. + * + * LCP-based delta selection (diff the assembled views) is REJECTED: a fold + * deletes covered rows from the view's prefix, so the new and old views no + * longer share a prefix and the "delta" would re-send the tail (duplicating + * it at the provider). The row-level frontier (append-only pos) is the + * only correct basis. + */ + +/** One element of the assembled egress view, with the bookkeeping the + * selector needs (buildSnapEgressView returns plain messages; the lane + * provides this side channel in the same order). */ +export interface EgressElement { + /** The wire message (anchor = the rendered summary message, data row = + * the transformed plane row). */ + msg: unknown; + /** The plane row pos (append-only). Anchor elements carry their + * NEGATIVE anchor pos. */ + pos: number; + /** true for rendered summary anchors (id = acp_summary_). */ + isAnchor: boolean; + /** The anchor's summary id (isAnchor only). */ + anchorId?: string; +} + +export interface DeltaState { + /** The highest row pos sent to the provider so far (-1 = nothing). */ + frontierPos: number; + /** The anchor summary ids already sent (acp_summary_*). */ + servedAnchors: string[]; +} + +export interface DeltaSelection { + /** true = the provider has nothing (or we re-armed) → send the FULL + * view (seed). false = send `delta`. */ + reseed: boolean; + /** The elements to send, in view order (anchors at their pair-safe + * positions, rows pos-ordered). Empty when reseed. */ + delta: readonly EgressElement[]; + /** The new state to persist after this outbound (no-op when reseed + * failed upstream — the lane keeps the old state on error). */ + nextState: DeltaState; +} + +const ANCHOR_ID_PREFIX = "acp_summary_"; + +/** The frontier state after the WHOLE view has been sent (seed / re-seed). */ +function stateFromElements(elements: readonly EgressElement[]): DeltaState { + let maxPos = -1; + const served: string[] = []; + for (const el of elements) { + if (el.isAnchor) { + const id = el.anchorId ?? anchorIdOf(el.msg); + if (id !== null && !served.includes(id)) served.push(id); + } else if (el.pos > maxPos) { + maxPos = el.pos; + } + } + return { frontierPos: maxPos, servedAnchors: served }; +} + +/** Select the outbound elements for one round. + * + * `state` is the persisted delta state (session.metadata); pass null for a + * fresh session → reseed. `elements` is the full egress view (snapshot + * mode's output), element side-channel included, in send order. + * + * The selector is PURE (no storage, no clock) — the lane persists + * `nextState` only AFTER the outbound succeeds (a failed round keeps the + * old frontier and re-sends the same delta next round: at-least-once, the + * provider dedupes by the chain). + */ +export function selectOutbound(elements: readonly EgressElement[], state: DeltaState | null): DeltaSelection { + if (state === null || elements.length === 0) { + // Reseed: the provider has nothing to chain from — send the FULL view + // and record everything as served (the frontier moves to the max pos; + // a later delta sends only what lands above it). + return { reseed: true, delta: [], nextState: stateFromElements(elements) }; + } + const served = new Set(state.servedAnchors); + const delta: EgressElement[] = []; + let maxPos = state.frontierPos; + const newAnchors: string[] = []; + for (const el of elements) { + if (el.isAnchor) { + const id = el.anchorId ?? anchorIdOf(el.msg); + if (id !== null && !served.has(id)) { + delta.push(el); + newAnchors.push(id); + } + continue; + } + if (el.pos > state.frontierPos) { + delta.push(el); + if (el.pos > maxPos) maxPos = el.pos; + } + } + return { + reseed: false, + delta, + nextState: { frontierPos: maxPos, servedAnchors: [...state.servedAnchors, ...newAnchors] }, + }; +} + +/** A delta that grew to the whole view (everything unsent) is a re-seed in + * disguise — the provider's chain is empty, so send it as a full seed. */ +export function isFullResend(sel: DeltaSelection, elements: readonly EgressElement[]): boolean { + return !sel.reseed && sel.delta.length >= elements.length; +} + +function anchorIdOf(msg: unknown): string | null { + const m = msg as { id?: unknown }; + return typeof m?.id === "string" && m.id.startsWith(ANCHOR_ID_PREFIX) ? m.id : null; +} diff --git a/src/snapshot/identity.ts b/src/snapshot/identity.ts new file mode 100644 index 0000000000..aae9be5ffc --- /dev/null +++ b/src/snapshot/identity.ts @@ -0,0 +1,285 @@ +/** + * #5874 — snapshot-table compression foundation: stage 0 (identity / world + * match). + * + * The host's `(sess_raw, msgid[])` pair is the identity channel (§4.0). The + * `l1_world` table is the server's standing canon: for a raw session id it + * records the canonical host-msgid ENVELOPE — an ordered list of + * `{id, pos}` where `pos` is the APPEND-ONLY ledger position the msgid was + * first issued at (never reused, the DDL invariant) — plus the ledger + * identity it was issued under. + * + * Every turn the incoming envelope (host msgids in host order) is matched + * against the stored canon: + * + * - echo — the id sequence matches positionally → all known. + * - probe — the stored id sequence is a strict PREFIX of the incoming → + * tail append (the common, v5-proven shape). + * - scatter — known ids keep their relative order but NEW ids are inserted + * at any position → the span-parse case (T5 byte pin). + * - issue — no stored world → issue a fresh ledger identity + arm. + * - diverge — the LIVE head changed, or a KNOWN id changed its relative + * order (history rewrite) → this turn legacy + re-stamp; the + * canon is NOT updated (the client converges in one turn). + * - resync — an explicit re-baseline (mode switch). + * + * Deletions (a stored id absent from the incoming) are NOT a divergence while + * the live head and the survivors' relative order are unchanged: they are + * reported via `goneIds` and the reconcile turns them into `delete` events. + * + * THE LIVE HEAD: after a fold the client's envelope starts at the first + * SURVIVING element while the canon still lists the covered head entries + * (their pos is never reused). A head change is a divergence only when the + * first stored id that is NOT fold-explained (`explainedGone`) differs from + * the incoming head — a legal post-fold shrink therefore keeps the plane + * green, while a host that drops its head WITHOUT fold evidence diverges. + * (Middle deletions without fold evidence stay legal — reconcile emits the + * `delete` events and the egress skips the row.) + * + * NEW ids are issued fresh append-only pos (maxPos+1.., in incoming order), so + * a middle insertion never shifts a stored row — the rows (and the prefix + * cache) stay put. + */ +import { envelopeChecksum } from "./checksums.ts"; +import type { WorldDecision, WorldEnvelopeEntry } from "./types.ts"; + +interface WorldMatchInput { + /** The stored L1 world for this sess_raw, or null when none exists. */ + stored: { + ledgerId: string; + /** The stored canon (ordered {id, pos}). */ + envelope: WorldEnvelopeEntry[]; + envelopeChecksum: string; + issuedAt: number; + armedAt?: number; + } | null; + /** The incoming host envelope (per-element msgids, in host order). For a + * non-cohort host this is the sequence-derived pseudo-envelope. */ + incoming: string[]; + /** Stored ids whose absence a fold already explains (their row is + * inactive with a coverRef). Only such ids may legally vanish from the + * HEAD of the envelope; any other gone id is a client-side drop + * (legal in the middle, divergent at the head). */ + explainedGone?: ReadonlySet; + /** The raw session key (host value, never truncated, P9). */ + sessRaw: string; + /** The bili canonical ledger identity to issue when no world exists + * (ruling D: the existing sid keyspace). */ + ledgerId: string; + now: number; + /** True when this turn is an explicit re-baseline (mode switch). */ + resync?: boolean; +} + +/** Match the incoming envelope against the stored canon. Pure — returns the + * verdict; the persistence (upsertL1World) is done by the caller so a + * `diverge` turn has ZERO side effects (the canon is not touched). */ +export function matchWorld(input: WorldMatchInput): WorldDecision { + const incoming = Array.from(input.incoming); + const incomingCk = envelopeChecksum(incoming); + + // Explicit re-baseline always re-issues the canon (the mode changed). + if (input.resync) { + const env: WorldEnvelopeEntry[] = incoming.map((id, i) => ({ id, pos: i + 1 })); + return { + mode: "resync", + envelope: env, + envelopeChecksum: incomingCk, + newIdPos: env.map((e) => ({ id: e.id, pos: e.pos })), + goneIds: [], + issuedLedgerId: input.ledgerId, + issuedAt: input.now, + armedAt: input.now, + }; + } + + if (input.stored === null) { + // No world yet: issue a fresh ledger identity, pay one arming turn. + const env: WorldEnvelopeEntry[] = incoming.map((id, i) => ({ id, pos: i + 1 })); + return { + mode: "issue", + envelope: env, + envelopeChecksum: incomingCk, + newIdPos: env.map((e) => ({ id: e.id, pos: e.pos })), + goneIds: [], + issuedLedgerId: input.ledgerId, + issuedAt: input.now, + armedAt: input.now, + }; + } + + const stored = input.stored; + const storedIds = stored.envelope.map((e) => e.id); + + // 1. echo: the id sequence matches positionally → all known. + if (stored.envelopeChecksum === incomingCk && stored.envelope.length === incoming.length) { + return { + mode: "echo", + envelope: stored.envelope.map((e) => ({ id: e.id, pos: e.pos })), + envelopeChecksum: incomingCk, + newIdPos: [], + goneIds: [], + issuedAt: stored.issuedAt, // B1 trap: origin never moves on echo + }; + } + + // Known-id bookkeeping. + const storedPos = new Map(); + for (const e of stored.envelope) storedPos.set(e.id, e.pos); + const incomingSet = new Set(incoming); + const goneIds: string[] = []; + for (const id of storedIds) if (!incomingSet.has(id)) goneIds.push(id); + + // A KNOWN id that changed its relative order (or the LIVE head changed) + // is a history rewrite → diverge (the canon is not updated; the turn + // runs legacy and re-stamps so the client converges in one turn). + // The live head = the first stored id that a fold did NOT absorb: + // after a post-fold shrink the incoming envelope legitimately starts at + // the first surviving element, not at the (covered) canon head. + const aliveStored = storedIds.filter((id) => !input.explainedGone?.has(id)); + const headMoved = aliveStored.length > 0 && incoming.length > 0 && incoming[0] !== aliveStored[0]; + let orderBroken = false; + if (!headMoved) { + // The survivor ids must appear in the same relative order as stored. + const survivorsStored = storedIds.filter((id) => incomingSet.has(id)); + const survivorsIncoming = incoming.filter((id) => storedPos.has(id)); + if (survivorsStored.length !== survivorsIncoming.length) { + orderBroken = true; + } else { + for (let i = 0; i < survivorsStored.length; i++) { + if (survivorsStored[i] !== survivorsIncoming[i]) { + orderBroken = true; + break; + } + } + } + } + if (headMoved || orderBroken) { + return { + mode: "diverge", + envelope: stored.envelope.map((e) => ({ id: e.id, pos: e.pos })), + envelopeChecksum: stored.envelopeChecksum, + newIdPos: [], + goneIds: [], + issuedAt: stored.issuedAt, + witness: `head ${storedIds[0] ?? "-"} -> ${incoming[0] ?? "-"}; orderBroken=${orderBroken}`, + }; + } + + // Head stable + survivor order preserved → a legal extension (probe or + // scatter). Issue fresh pos to every NEW id (maxPos+1.., incoming order). + let maxPos = 0; + for (const e of stored.envelope) maxPos = Math.max(maxPos, e.pos); + const newIdPos: Array<{ id: string; pos: number }> = []; + const posMap = new Map(); + for (const e of stored.envelope) posMap.set(e.id, e.pos); + for (const id of incoming) { + if (!posMap.has(id)) { + maxPos += 1; + posMap.set(id, maxPos); + newIdPos.push({ id, pos: maxPos }); + } + } + const envelope: WorldEnvelopeEntry[] = incoming.map((id) => ({ id, pos: posMap.get(id)! })); + + // probe = stored is a strict prefix of incoming (tail append only); + // scatter = new ids are interleaved. + let isPrefix = storedIds.length < incoming.length; + if (isPrefix) { + for (let i = 0; i < storedIds.length; i++) { + if (incoming[i] !== storedIds[i]) { + isPrefix = false; + break; + } + } + } + const mode = newIdPos.length > 0 + ? (isPrefix ? "probe" : "scatter") + : goneIds.length > 0 + ? "probe" // canon shrink (deletions only) + : "echo"; + + return { + mode, + envelope, + envelopeChecksum: incomingCk, + newIdPos, + goneIds, + issuedAt: stored.issuedAt, // B1 trap: origin never moves on extend + }; +} + +/** + * #5874 2×2 (brief 19544722, Part A) — the M2 INCREMENTAL envelope: + * the client sends ONLY the new elements (with msgids), so the incoming + * list shares NO ids with the stored canon (a pure addition). The caller + * has ALREADY located the sequence position (世界匹配定位序列位置) via the + * seqs carrier — the incoming ledger seqs are contiguous and start exactly + * after the plane's max stamped seq — so every incoming id gets a fresh + * append-only pos (maxPos+1.., in incoming order) and the envelope is the + * stored canon ++ the new tail. + * + * This is the "probe" shape of a client that never re-sends: the covered + * (fold-explained) canon entries stay in the canon (the caller prunes them + * from the kernel-input walk via explainedGone — the plane keeps serving + * them as covered rows until GC). + */ +export function matchAppend( + stored: { + ledgerId: string; + envelope: WorldEnvelopeEntry[]; + envelopeChecksum: string; + issuedAt: number; + armedAt?: number; + }, + incoming: string[], +): WorldDecision { + const incomingCk = envelopeChecksum(incoming); + let maxPos = 0; + const posMap = new Map(); + for (const e of stored.envelope) { + maxPos = Math.max(maxPos, e.pos); + posMap.set(e.id, e.pos); + } + const newIdPos: Array<{ id: string, pos: number }> = []; + for (const id of incoming) { + if (posMap.has(id)) continue; // defensive: the caller guarantees a pure addition + maxPos += 1; + posMap.set(id, maxPos); + newIdPos.push({ id, pos: maxPos }); + } + const envelope: WorldEnvelopeEntry[] = [ + ...stored.envelope.map((e) => ({ id: e.id, pos: e.pos })), + ...incoming.map((id) => ({ id, pos: posMap.get(id)! })), + ]; + return { + mode: "probe", + envelope, + // The stored canon's checksum covers the WHOLE envelope sequence + // (the next M1 full resend must echo against it), not just the new + // tail. + envelopeChecksum: envelopeChecksum(envelope.map((e) => e.id)), + newIdPos, + goneIds: [], + issuedAt: stored.issuedAt, // B1 trap: origin never moves on extend + }; +} + +/** Parse the persisted element envelope (the l1_world envelope_json column — + * `JSON.stringify(WorldEnvelopeEntry[])`, host order). Validated: a + * malformed canon is a storage-corruption signal and throws (the lane + * maps it to a resync — the client converges in one round) rather than + * being silently re-issued over a half-read canon. */ +export function parseEnvelope(envelopeJson: string | null): { envelope: WorldEnvelopeEntry[] } { + if (envelopeJson === null) return { envelope: [] }; + const parsed: unknown = JSON.parse(envelopeJson); + if (!Array.isArray(parsed)) throw new Error("parseEnvelope: the canon is not an array (corrupted l1_world row)"); + const envelope: WorldEnvelopeEntry[] = parsed.map((e, i) => { + const entry = e as { id?: unknown, pos?: unknown }; + if (typeof entry.id !== "string" || !Number.isInteger(entry.pos) || (entry.pos as number) < 0) { + throw new Error(`parseEnvelope: canon entry ${i} is not a {id, pos} pair (corrupted l1_world row)`); + } + return { id: entry.id, pos: entry.pos as number }; + }); + return { envelope }; +} diff --git a/src/snapshot/plane.ts b/src/snapshot/plane.ts new file mode 100644 index 0000000000..d9408934ad --- /dev/null +++ b/src/snapshot/plane.ts @@ -0,0 +1,285 @@ +/** + * #5874 / 2×2 — stage 0 (identity / world match) decision glue. + * + * Sits at the head of the six-stage chain (raw bytes → **stage 0** → protocol + * conversion → kernel → plane write → egress). It runs the pure `matchWorld` + * (identity.ts) on the l1_world element envelope and carries the R3 identity + * fault policy (brief 19544722 red gate R3): + * + * - a cohort host that DECLARED the dual-channel (per-element msgids) but + * did not collect them all → the request is HARD-REJECTED (400) BEFORE + * any table/kernel side effect, and the fault feeds a fastpath-style + * breaker (10-minute window, 3 strikes → a persistent per-sid latch that + * keeps rejecting until an operator clears it). + * + * The decision itself is pure (identity.ts); this module adds the storage + * glue (read the stored envelope, build the fold-explained-gone set from the + * full row set, persist the new canon) — and ONLY for non-diverge verdicts + * (a diverge turn has zero side effects by construction). + */ +import { matchAppend, matchWorld, parseEnvelope } from "./identity.ts"; +import { snapL1EnvelopeRead, snapL1EnvelopeUpsert, snapL1Lookup, snapRowsAll } from "../storage/snap.ts"; +import type { DbLike } from "../storage/snap.ts"; +import type { WorldDecision } from "./types.ts"; + +/** R3 identity fault — the lane rejects the request with HTTP 400 and takes + * NO side effect (no plane event, no kernel run, no session ext write other + * than the breaker counter). */ +export class FoundationRejectError extends Error { + readonly status = 400; + constructor(public readonly reason: string) { + super(`[snapshot] R3 identity fault: ${reason}`); + this.name = "FoundationRejectError"; + } +} + +/** R3 breaker: a per-process fault window (10 minutes) with a strike cap + * (3). On the third strike within the window the sid gets a PERSISTENT + * latch (the caller persists it, e.g. session_ext) and every subsequent + * R3 fault for that sid rejects immediately without re-counting. Mirrors + * the incremental-lane breaker discipline (the brief's R3 reference). */ +export class R3FaultTracker { + private readonly faults = new Map(); + private readonly latched = new Set(); + constructor( + private readonly windowMs: number = 10 * 60 * 1000, + private readonly strikeCap: number = 3, + ) {} + + /** Record an R3 fault. Returns the persistent-latch verdict. */ + recordFault(sid: string, now: number = Date.now()): { latched: boolean } { + if (this.latched.has(sid)) return { latched: true }; + let entry = this.faults.get(sid); + if (!entry || now - entry.firstAt > this.windowMs) { + entry = { count: 0, firstAt: now }; + this.faults.set(sid, entry); + } + entry.count += 1; + if (entry.count >= this.strikeCap) { + this.latched.add(sid); + return { latched: true }; + } + return { latched: false }; + } + + isLatched(sid: string): boolean { + return this.latched.has(sid); + } + + /** Operator escape hatch (the latch is persistent across the window but + * not across a process restart unless the caller also persists it). */ + clearLatched(sid: string): void { + this.latched.delete(sid); + this.faults.delete(sid); + } +} + +/** The process-wide R3 fault tracker (one per process; the lane consults it + * before every stage-0 run and records on every fault). Exported so tests + * can inspect/reset it. */ +export const r3FaultTracker = new R3FaultTracker(); + +/** The stage-0 input: the raw identity channel for one turn. + * @param db the storage db (snap.ts DbLike surface). + * @param sessRaw the raw host session id (P9: never truncated). + * @param ledgerId the bili canonical ledger identity (the sid). + * @param incoming the per-element msgids in host order. + * @param resync true for an explicit re-baseline (mode switch). + * @param now clock (injectable for tests). + */ +interface Stage0Input { + db: DbLike; + sessRaw: string; + ledgerId: string; + incoming: string[]; + /** The `biliLedgerSeqs` carrier (M2 position locating, #4316 layer-3 + * transport). When it is contiguous and starts exactly after the + * plane's max stamped seq, an ALL-UNKNOWN incoming is a pure M2 + * append (matchAppend); without it, an all-unknown incoming is + * treated as a full (possibly re-minted) envelope → the normal + * matchWorld verdict (usually diverge → resync, one-round + * convergence). */ + seqs?: number[] | null; + resync?: boolean; + now?: number; +} + +export interface Stage0Result { + /** The pure world-match verdict (identity.ts). */ + decision: WorldDecision; + /** True when the verdict is `diverge` — the caller runs this turn on the + * legacy path with ZERO plane side effects (the canon is untouched; the + * client converges one turn later). */ + diverge: boolean; + /** The fold-explained gone set (stored ids whose row is inactive with a + * coverRef) — exposed for the reconcile (delete events) and tests. */ + explainedGone: Set; +} + +/** Run stage 0: read the stored element envelope + the full row set, match + * the incoming ids, and (for every verdict except `diverge`) persist the + * new canon. A `diverge` turn writes NOTHING (zero side effects). + * + * The pos values the verdict issues are the LOGICAL append-only ledger + * positions (maxPos+1..); the physical snap_rows pos (ledger seq) is + * assigned by the ingest path in the same incoming order, so the two agree + * for tail-appends. For scatter (middle insertion) the reconcile re-homes + * the new rows to their envelope pos in the same transaction. */ +export function stage0Decide(input: Stage0Input): Stage0Result { + const now = input.now ?? Date.now(); + const l1 = snapL1Lookup(input.db, input.sessRaw); + const env = snapL1EnvelopeRead(input.db, input.sessRaw); + + // Fold-explained gone: a stored id may legally vanish from the envelope + // head when its row was deactivated by a fold (coverRef set). Read the + // FULL row set (active + inactive) — the egress-side view filters the + // inactive rows out and would under-explain. + const rows = snapRowsAll(input.db, input.ledgerId); + const explainedGone = new Set(); + for (const r of rows) { + if (r.active === 0 && r.coverRef !== null) { + for (const id of r.coreIds) explainedGone.add(id); + } + } + + const stored = + l1 && env && env.envelopeJson !== null && env.envelopeChecksum !== null + ? { + ledgerId: l1.ledgerId, + envelope: parseEnvelope(env.envelopeJson).envelope, + envelopeChecksum: env.envelopeChecksum, + issuedAt: l1.issuedAt, + armedAt: l1.armedAt ?? undefined, + } + : null; + + // M2 pure append (brief 19544722): the client sent ONLY new elements — + // every incoming id is unknown to the canon — and the seqs carrier says + // they land as a contiguous run starting exactly after the plane's max + // stamped seq. That is the 世界匹配定位序列位置 evidence: issue append- + // only pos (maxPos+1..) without any head/order risk. Anything less + // certain (missing seqs, a gap, a known id mixed in) falls through to + // matchWorld — an all-unknown full array there is usually a re-mint and + // diverges → resync, one-round convergence. Never guess. + let decision: WorldDecision; + const seqs = input.seqs ?? null; + const storedSet = stored !== null ? new Set(stored.envelope.map((e) => e.id)) : null; + // The plane's max stamped seq: rows are stamped ledgerSeq = pos+1 + // (the D-lane contract, snapshot-render.ts:154), so the frontier is + // derivable from the row pos alone — no post_json parse. + let planeMaxSeq = 0; + for (const r of rows) { + if (r.active === 1 && r.pos + 1 > planeMaxSeq) planeMaxSeq = r.pos + 1; + } + // Review-round-2 Residual 1b: the host re-sent a body this proxy already + // processed — every id is already in the stored canon and the carrier + // seqs (when present) sit at/below the plane frontier (the original + // turn's in-request flush stamped them before its response left). Zero + // side effects: the canon is kept as-is, the lane re-assembles the SAME + // world (an idempotent kernel round), and the egress no-op + // classification keeps the chain id and skips the developer message. + // Without this guard the M2 retry ([m5] vs head m1) would headMove → + // diverge → snapStage0Reset: a harmless retry would WIPE the plane. + if ( + stored !== null && + storedSet !== null && + !input.resync && + input.incoming.length > 0 && + input.incoming.every((id) => storedSet.has(id)) && + (seqs === null || seqs.every((s) => s <= planeMaxSeq)) + ) { + decision = { + mode: "echo", + replay: true, + envelope: stored.envelope, + envelopeChecksum: stored.envelopeChecksum, + newIdPos: [], + goneIds: [], + issuedAt: stored.issuedAt, + witness: `replay: all ${input.incoming.length} ids known (maxSeq ${seqs ? Math.max(...seqs) : "-"} <= frontier ${planeMaxSeq})`, + }; + } else { + // M2 pure append (brief 19544722): the client sent ONLY new elements — + // every incoming id is unknown to the canon — and the seqs carrier says + // they land as a contiguous run starting exactly after the plane's max + // stamped seq. That is the 世界匹配定位序列位置 evidence: issue append- + // only pos (maxPos+1..) without any head/order risk. Anything less + // certain (missing seqs, a gap, a known id mixed in) falls through to + // matchWorld — an all-unknown full array there is usually a re-mint and + // diverges → resync, one-round convergence. Never guess. + const m2Candidate = + stored !== null && + !input.resync && + input.incoming.length > 0 && + seqs !== null && + seqs.length === input.incoming.length; + if (m2Candidate) { + const allUnknown = input.incoming.every((id) => !storedSet!.has(id)); + let contig = seqs![0] === planeMaxSeq + 1; + if (contig) { + for (let i = 1; i < seqs!.length; i++) { + if (seqs![i] !== seqs![i - 1] + 1) { + contig = false; + break; + } + } + } + decision = allUnknown && contig + ? matchAppend(stored, input.incoming) + : matchWorld({ + stored, + incoming: input.incoming, + explainedGone, + sessRaw: input.sessRaw, + ledgerId: input.ledgerId, + now, + resync: input.resync, + }); + } else { + decision = matchWorld({ + stored, + incoming: input.incoming, + explainedGone, + sessRaw: input.sessRaw, + ledgerId: input.ledgerId, + now, + resync: input.resync, + }); + } + } + + // Persist the canon for every verdict EXCEPT diverge (zero side effects). + if (decision.mode !== "diverge") { + snapL1EnvelopeUpsert( + input.db, + input.sessRaw, + JSON.stringify(decision.envelope), + decision.envelopeChecksum, + ); + } + + return { decision, diverge: decision.mode === "diverge", explainedGone }; +} + +/** The R3 identity check for a cohort host that DECLARED the dual-channel: + * the per-element msgid list must cover EVERY element of the body. A short + * (or empty, when the body is non-empty) collection is the identity gate + * fault → the caller throws FoundationRejectError (400, zero side effects) + * and records the fault on the tracker. + * + * @param declared the host profile declares per-element msgids (cohort). + * @param elementCount the body element count (the msgs to be parsed). + * @param msgids the collected per-element msgids (null/undefined = + * the host sent none). + * @returns true when the turn may proceed (non-cohort, or full collection). + */ +export function checkR3Identity( + declared: boolean, + elementCount: number, + msgids: string[] | null | undefined, +): boolean { + if (!declared) return true; // non-cohort: sticky order>hash degrade, same tables + if (elementCount === 0) return true; + if (!msgids || msgids.length !== elementCount) return false; + return true; +} diff --git a/src/snapshot/stage0.ts b/src/snapshot/stage0.ts new file mode 100644 index 0000000000..a9cbb7633b --- /dev/null +++ b/src/snapshot/stage0.ts @@ -0,0 +1,233 @@ +// #5874 2×2 stage 0 (#6181): world match + kernel-input assembly for +// cohort (dual-channel DECLARED) plugin traffic. SHARED by the openai and +// responses lanes so both carry the same red gate and the same +// one-round-convergence contract — the HTTP responses lane previously had +// NO assembly layer at all, which is how a chained (delta) turn's collapsed +// kernel view reached the plane and destroyed it (#6181 B0: the plane +// invariant is "the resent array IS the world", and only a cohort turn +// proves that — a non-cohort chained turn cannot be addressed against the +// plane at all). +// +// Runs BEFORE processTurn in the lane, so a thrown FoundationRejectError +// (R3) carries ZERO kernel side effects. Every other fault → the legacy +// full path (degrade, never guess — one-round convergence). +// +// Contract (brief 19544722 + review 2026-11-12): +// - armed = pluginMode + dual-channel DECLARED (msgids present) + the +// pluginLaneServerHistory rollout (M1: the red gate must be unreachable +// wherever the pin is unreachable) + no R3 latch on the sid (in-memory +// window OR the persisted latch, M3). +// - R3: declared but incomplete → checkR3Identity is the single arbiter +// (M2: it is CALLED here, in both lanes); an unlatched fault throws +// FoundationRejectError → the server maps it to 400 (B1); 3 faults in a +// 10-minute window latch the sid to the legacy path and the latch +// PERSISTS across proxy restarts (snapR3Latch). +// - stage 0: snapStage0 → resync (diverge / explicit re-baseline) → plane +// reset + legacy this turn + the lane clears session.responsesDelta +// (M6: a stale frontier against re-issued pos is a dead loop); +// otherwise assemble the kernel input from the plane rows (old cores — +// the kernel's own tagged output, invariant v) + the NEW spans (P5) + +// anchor rows at their covered-run boundary. + +import type { BiliMessage } from "acp-kernel/wire"; +import { getStore } from "../persist.js"; +import { featureEnabled } from "../rollout.js"; +import { applyMsgIdPin, validateMsgIds } from "../msgid-pin.js"; +import { ANCHOR_ID_PREFIX } from "../storage/snap.js"; +import { assembleKernelInput, shouldResync } from "./assemble.js"; +import { checkR3Identity, FoundationRejectError, r3FaultTracker } from "./plane.js"; + +export interface Stage0Input { + sessionId: string; + /** Lane name for log prefixes only ("openai" | "responses"). */ + lane: string; + /** The converted array for this turn (seq-stamped by the lane). */ + msgs: BiliMessage[]; + /** The raw declared biliMsgIds field value (whatever shape it had). */ + incomingMsgIds: unknown; + /** Validated ledger seqs (null when absent/mismatched — the lane + * already applied them to msgs when non-null). */ + incomingSeqs: number[] | null; + pluginMode: boolean; + /** The openai fast-path watermark hit; the responses lane has none. */ + wpHit?: unknown; + log: (level: string, msg: string) => void; +} + +export interface Stage0Result { + /** Pinned msg array when the pin applied — the lane adopts it + * (`msgs = pinned; originalMessages = msgs`). null otherwise. */ + pinned: BiliMessage[] | null; + /** Assembled kernel input (stage 0 hit). undefined → the lane renders + * its usual view (anchored / D lane). */ + kernelInput: BiliMessage[] | undefined; + /** True when the cohort armed and RAN (assembled, resynced, or + * degraded) — the lane gates the chained-non-assembled plane bypass + * on its NEGATION: a non-cohort chained turn's kernel view is a + * delta, not the world, and must touch neither the plane write nor + * the plane egress for this turn (B0 defensive). */ + cohortRan: boolean; + /** True when stage 0 reset the plane this turn (diverge / explicit + * re-baseline) — the lane must clear session.responsesDelta (M6). */ + resynced: boolean; + /** Review-round-2 Residual 2: the highest PLANE row pos of the STORED + * world at decision time (the new rows are not flushed yet) — present + * only when the decision is a strict extension (mode probe, nothing + * gone, new elements only) AND the assembly succeeded. The responses + * lane uses it to INITIALIZE the delta frontier on the first chained + * round of a keepResponseId chain: the provider's chain — built from + * this proxy's own full-view round — already holds exactly the stored + * world, so the outbound is the new elements only, never a re-seed. + * null = the lane re-seeds as before (every other shape). */ + storedFrontierPos: number | null; +} + +export function runStage0(opts: Stage0Input): Stage0Result { + const { sessionId, lane, msgs, incomingMsgIds, incomingSeqs, pluginMode, wpHit, log } = opts; + const rollout = featureEnabled("pluginLaneServerHistory"); + const declared = Array.isArray(incomingMsgIds) && incomingMsgIds.length > 0; + // M1: the arm rides the SAME rollout as the pin — under the default + // (rollout off) the whole 2×2 path is unreachable, so R3 is not dead + // code waiting for a rollout that never arms it. + const armed0 = rollout && wpHit === undefined && pluginMode && declared; + const persistedLatch = (() => { + try { + return getStore()?.snapR3Latched(sessionId) ?? false; + } catch { + return false; // a store fault must not change the arming verdict + } + })(); + let armed = armed0 && !r3FaultTracker.isLatched(sessionId) && !persistedLatch; + if (armed0 && !armed) { + log("debug", `[${sessionId}] [2x2:${lane}] sid is R3-latched — legacy path this turn`); + } + let pinned: BiliMessage[] | null = null; + if (wpHit === undefined && pluginMode && incomingMsgIds !== undefined) { + if (rollout) { + const v = validateMsgIds(incomingMsgIds, msgs.length); + if (v.ok) { + pinned = applyMsgIdPin(msgs, v.ids); + } else if (armed) { + // #6181 M2: checkR3Identity is the single arbiter of the + // identity fault (declared + element count mismatched) — + // validateMsgIds' shape verdicts are its evidence, not a + // second gate. + if (!checkR3Identity(true, msgs.length, Array.isArray(incomingMsgIds) ? (incomingMsgIds as string[]) : null)) { + // R3: a cohort host that DECLARED the dual-channel but + // did not collect every element → hard 400 BEFORE any + // kernel side effect; the fault feeds the breaker + // (10-minute window, 3 strikes → the sid latches to the + // legacy path, PERSISTED across restarts (M3), instead + // of bricking the session on a misbehaving host). + const { latched } = r3FaultTracker.recordFault(sessionId); + if (latched) { + armed = false; + try { + getStore()?.snapR3Latch(sessionId); + } catch { /* the in-memory latch already fires; persistence is best-effort */ } + log("error", `[${sessionId}] [2x2:${lane}] R3 identity fault ×3 in window (${v.reason}) — sid latched to the legacy path (persisted)`); + } else { + throw new FoundationRejectError(`inbound msgid envelope failed validation (${v.reason})`); + } + } + } else { + log("warn", `[${sessionId}] [msgid] ${lane}-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#5825)`); + } + } else { + log("info", `[${sessionId}] [msgid] ${lane}-wire biliMsgIds present but pluginLaneServerHistory rollout is off for this install — field stripped, ids not pinned (#5825)`); + } + } + let kernelInput: BiliMessage[] | undefined; + let resynced = false; + let storedFrontierPos: number | null = null; + if (armed) { + try { + const store = getStore(); + const stage0 = store?.snapStage0(sessionId, msgs.map((m) => m.id), { seqs: incomingSeqs ?? undefined }); + const cores = stage0 ? (store?.snapStage0Cores(sessionId) ?? null) : null; + if (stage0 && cores) { + if (shouldResync(stage0.decision, cores.rows.length)) { + // Structural disagreement (diverge) or an explicit + // re-baseline: reset the plane; THIS turn runs the + // legacy full path and re-seeds it through the + // in-request flush — the NEXT turn re-issues the + // envelope from the client's current array. + store!.snapStage0Reset(sessionId); + resynced = true; + log("warn", `[${sessionId}] [2x2:${lane}] stage-0 ${stage0.decision.mode} — plane reset, this turn legacy (one-round convergence)`); + } else { + // newCores = the converted spans of the NEW ids, in + // envelope order. The front stage already converted the + // whole array (front-stage invariance), so the new + // spans carry their full-parse bytes — T5 holds by + // construction. + const newIdSet = new Set(stage0.decision.newIdPos.map((n) => n.id)); + const newCores: BiliMessage[] = []; + const taken = new Set(); + for (const e of stage0.decision.envelope) { + if (!newIdSet.has(e.id) || taken.has(e.id)) continue; + taken.add(e.id); + const m = msgs.find((x) => x.id === e.id); + if (m === undefined) throw new Error(`new span ${e.id} absent from the converted array`); + newCores.push(m); + } + // The plane's ANCHOR rows (active acp_summary_) + // ride the kernel input at their covered-run boundary — + // the D-view placement parity. snapStage0Cores returns + // both the rows and the coverMax map. + const anchors = cores.rows + .filter((r) => r.ids.some((id) => id.startsWith(ANCHOR_ID_PREFIX))) + .map((r) => { + const id = r.ids.find((x) => x.startsWith(ANCHOR_ID_PREFIX))!; + return { + msg: JSON.parse(r.postJson) as BiliMessage, + blockId: id.slice(ANCHOR_ID_PREFIX.length), + }; + }); + const asm = assembleKernelInput({ + decision: stage0.decision, + rows: cores.rows + .filter((r) => !r.ids.some((id) => id.startsWith(ANCHOR_ID_PREFIX))) + .map((r) => ({ pos: r.pos, ids: r.ids, msg: JSON.parse(r.postJson) as BiliMessage })), + newCores, + skipIds: stage0.explainedGone, + anchors, + coverMax: cores.coverMax, + }); + kernelInput = asm.kernelInput; + // Residual 2: at THIS moment every active plane row is a + // stored-world row (this round's new rows land only in the + // lane's in-request flush, which runs AFTER stage 0). The + // stored world's boundary is therefore the max active row + // pos — a strict-extension decision (probe, nothing gone, + // new elements) is what makes "the chain holds exactly + // these rows" provable; every other shape stays null and + // the lane re-seeds as before. + if ( + stage0.decision.mode === "probe" + && stage0.decision.goneIds.length === 0 + && stage0.decision.newIdPos.length > 0 + ) { + storedFrontierPos = cores.rows.reduce((mx, r) => (r.pos > mx ? r.pos : mx), -1); + } + log("info", `[${sessionId}] [2x2:${lane}] stage-0 ${stage0.decision.mode} — kernel input from plane (old=${asm.oldCount}, new=${newCores.length}, anchors=${anchors.length})`); + } + } + } catch (err) { + if (err instanceof FoundationRejectError) throw err; // R3 outranks the fallback + const why = err instanceof Error ? err.message : String(err); + if (why.startsWith("[snapshot 2×2]")) { + // The assembly could not address the envelope against the + // plane (table behind — R4-shaped): reset and take the + // legacy path this turn (one-round convergence). M6: the + // lane clears session.responsesDelta for a resynced sid. + try { + getStore()?.snapStage0Reset(sessionId); + } catch { /* the reset is best-effort — the legacy path is the floor */ } + resynced = true; + } + log("warn", `[${sessionId}] [2x2:${lane}] stage-0 failed (${why}) — legacy full path`); + } + } + return { pinned, kernelInput, cohortRan: armed, resynced, storedFrontierPos }; +} diff --git a/src/snapshot/types.ts b/src/snapshot/types.ts new file mode 100644 index 0000000000..160335aab0 --- /dev/null +++ b/src/snapshot/types.ts @@ -0,0 +1,74 @@ +/** + * #5874 / 2×2 — snapshot-foundation stage-0 shared types. + * + * TYPE-ONLY surface. Depends on nothing but the kernel's public types so it + * can be imported from any layer (storage, server, tests) without a cycle. + * + * Aligned to the reference implementation (#6155 `src/snapshot/`) — the stage-0 + * world-match / hydrate / lane decision vocabulary is shared verbatim. The + * PLANE itself stays on the #5901 message-level model (one `snap_rows` row per + * ledger element, `post_json` = the post-rewrite common-core BiliMessage JSON, + * `pos` = the append-only ledger position), so the reconcile/egress event + * types live in `src/storage/snap.ts` (unchanged) and are not re-declared here. + */ + +/** One entry in the standing canon envelope: a host msgid bound to the + * append-only ledger pos it was first issued at (pos is never reused — the + * DDL invariant). The canon is stored as an ordered [{id, pos}] list so the + * L1 table is self-contained (world = id + pos, §4.0 ruling G). */ +export interface WorldEnvelopeEntry { + id: string; + pos: number; +} + +/** World-match verdicts for stage 0 (§4.0). Executed in order; first hit + * wins. `pos` is append-only: a middle insertion does NOT shift stored rows. + * - `echo`: the incoming id sequence matches the stored one positionally. + * - `probe`: the stored id sequence is a strict PREFIX of the incoming one + * (tail append — the common, v5-proven shape) or a canon shrink. + * - `scatter`: known ids keep their relative order but NEW ids are inserted + * in the middle / at any position (the span-parse case, T5 byte pin). + * - `issue`: no stored world for this sess_raw → fresh ledger identity + arm. + * - `diverge`: the live head changed, or a KNOWN id changed its relative + * order (history rewrite) → this turn runs legacy and re-stamps; the canon + * is NOT updated (the client converges in one turn). + * - `resync`: an explicit re-baseline (mode switch / ledger re-issue). + * + * Deletions (a stored id absent from the incoming) are orthogonal to the mode + * and are reported via `goneIds` → the reconcile turns them into `delete` + * events (not a divergence while the head + survivor order are unchanged). */ +type WorldMode = "echo" | "probe" | "scatter" | "issue" | "diverge" | "resync"; + +export interface WorldDecision { + mode: WorldMode; + /** The canon envelope to persist after this turn (host ids in host order, + * each bound to its append-only pos). For `diverge` this is the UNCHANGED + * stored canon. */ + envelope: WorldEnvelopeEntry[]; + /** sha256 of the id sequence (host order) — the fast mismatch before the + * positional walk. */ + envelopeChecksum: string; + /** The NEW ids issued this turn (id → fresh pos), in incoming order. + * Empty for echo / diverge. */ + newIdPos: Array<{ id: string; pos: number }>; + /** The stored ids ABSENT from the incoming (client deletions) — the + * reconcile turns these into `delete` events. Empty for issue / echo. */ + goneIds: string[]; + /** For `issue`: the freshly issued ledger identity (the existing sid + * keyspace, ruling D). Absent otherwise. */ + issuedLedgerId?: string; + /** The world origin (issued_at) — carried so the caller can preserve it + * on a plain update (B1 trap: it never moves except on issue/resync). */ + issuedAt: number; + /** The arming-turn marker set on `issue`/`resync`. */ + armedAt?: number; + /** Diagnostic witness sample (echo spot-check) — warn stream only. */ + witness?: string; + /** Review-round-2 Residual 1b: the host re-sent a body this proxy ALREADY + * processed (every id known, carrier seqs at/below the frontier) — an + * upstream-200 retry. The lane must neither wipe the plane (the diverge + * this would otherwise cause) nor strip the chain id (the egress no-op + * classification does the continuation). Zero side effects. */ + replay?: boolean; +} + diff --git a/src/storage/ingest.ts b/src/storage/ingest.ts index 907fc103d2..095d4c0a6e 100644 --- a/src/storage/ingest.ts +++ b/src/storage/ingest.ts @@ -59,6 +59,16 @@ export interface LegacySessionLike { blockContents: Map; lastMessages?: CoreMessage[]; lastMessagesFolded?: boolean; + /** #5874 mapEgress direct authority: the RAW tagged processTurn output for + * this turn (deep-copied by the lane before the in-request flush). The + * snapshot plane persists THESE bytes (the kernel's own tagged output, + * not the pre-rewrite transcript) so the egress stage reads the rows and + * runs only the light transform. Carried into IngestSessionInput as + * planeMessages. */ + lastPlaneMessages?: CoreMessage[]; + /** #6181 B0: one-shot plane-write skip — see Session.skipPlaneFlush. + * Consumed (cleared) by snapShadowSync on the save that carries it. */ + skipPlaneFlush?: boolean; pluginSnapshot?: CoreMessage[]; /** #3301: external-converter import payload — original message * decompositions interned BEFORE the snapshot/tail so their rows get the @@ -125,6 +135,9 @@ export function toLegacyLike(s: { blockContents: LegacySessionLike["blockContents"]; lastMessages?: LegacySessionLike["lastMessages"]; lastMessagesFolded?: boolean; + lastPlaneMessages?: LegacySessionLike["lastPlaneMessages"]; + /** #6181 B0: one-shot plane-write skip (see Session.skipPlaneFlush). */ + skipPlaneFlush?: LegacySessionLike["skipPlaneFlush"]; pluginSnapshot?: LegacySessionLike["pluginSnapshot"]; decompositionLog?: LegacySessionLike["decompositionLog"]; contentStore?: LegacySessionLike["contentStore"]; @@ -140,6 +153,8 @@ export function toLegacyLike(s: { blockContents: s.blockContents, lastMessages: s.lastMessages, lastMessagesFolded: s.lastMessagesFolded, + lastPlaneMessages: s.lastPlaneMessages, + skipPlaneFlush: s.skipPlaneFlush, pluginSnapshot: s.pluginSnapshot, decompositionLog: s.decompositionLog, contentStore: s.contentStore, @@ -505,7 +520,7 @@ function finishPlan( ccrJson: session.contentStore ? JSON.stringify(session.contentStore) : null, }; - return { session: sessionRow, messages, blocks, deadRefs, refs, ccrEntries, pluginSnapshotSeqs, tailSeqs }; + return { session: sessionRow, messages, blocks, deadRefs, refs, ccrEntries, pluginSnapshotSeqs, tailSeqs, planeMessages: session.lastPlaneMessages ?? null, ...(session.skipPlaneFlush ? { skipPlaneFlush: true } : {}) }; } /** Rebuilds the legacy load view from unified rows alone. */ diff --git a/src/storage/schema.ts b/src/storage/schema.ts index 5efaae598c..a6e983b68a 100644 --- a/src/storage/schema.ts +++ b/src/storage/schema.ts @@ -83,7 +83,22 @@ // discipline as v9 (#4796); building them once costs about what // open-time quick_check pays on EVERY open (#4175). Additive-only: an // older build opening a v11 file ignores the extra indexes. -export const STORAGE_SCHEMA_VERSION = 11; +// - v12: snapshot-table compression foundation (#5874 P0) — four +// ADDITIVE tables + one index (l1_world / snap_rows / snap_log / +// snap_ckpt). No backfill, no rewrite of existing rows: the plane is +// dormant until storage.snapBase.enabled turns it on, and rollback is +// DROP the four tables + index and rewind user_version — T1 pins the +// byte-identical behavior in both directions. Additive-only: an +// older build opening a v12 file ignores the extra tables. +// Applied PROBE-FIRST (table-existence check), NOT version-gated: +// the unmerged fastpath re-land (#5855) numbers its two table groups +// v12/v13 and landed them into a SHARED database first (field, +// 2026-11-08: user_version 13 + pf_* present, snap tables absent → +// the plane died with "no such table: l1_world"). The probe makes +// both merge orderings coexist; whichever branch lands second in +// master still renumbers its constant mechanically (hygiene only — +// the DDL shapes are independent). +export const STORAGE_SCHEMA_VERSION = 12; /** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and * auto checkpoints move frames into the main db and reset the write offset @@ -311,6 +326,81 @@ CREATE INDEX IF NOT EXISTS idx_message_media_media_hash ON message_media(media_h CREATE INDEX IF NOT EXISTS idx_checkpoints_state_hash ON checkpoints(state_hash); `; +/** #5874 snapshot-table compression foundation (P0, v12) — four additive + * tables + one index, created here so fresh installs and in-place upgrades + * converge on the same shape. The plane is DORMANT unless the + * storage.snapBase config surface turns it on: with the flag off these + * tables stay empty forever, and rollback is DROP + user_version rewind + * (no data migration exists in either direction — pure additivity, T1). + * + * l1_world: stage-0 identity ledger — the raw host session id (sess_raw) + * maps to the proxy's ledger identity (ledger_id = sessions.session_id). + * witness_probe is the sampled id fingerprint the arm decision was based on + * (composite-probe evidence). issued_at precedes armed_at until one full + * echo round validates the mapping (the "one round of arming" the proposal + * charges for a fresh identity). + * + * snap_rows: the materialized snapshot projection — one row per ledger + * position (pos = messages.seq for the same session). core_ids is the JSON + * array of host message ids bound to the position; pre_json / post_json + * carry the canonical core shape (NEVER wire bytes — invariant v); an + * active=0 row is folded/tombstoned but NEVER physically deleted by events + * (invariant iv — physical cleanup waits for the covering block's archive, + * snapGcCoveredRows). cover_ref names the covering block for folded rows. + * + * snap_log: the append-only event stream. seq is strictly monotonic per + * session and always > max(log max, ckpt_seq) (invariant ii — enforced by + * the commit primitive, snap.ts). payload carries the applied event, the + * per-row S4 checksums of the touched rows, and the post-apply aggregate + * checksum over ALL rows (R1's verification anchor). Entries fold into + * snap_ckpt and are truncated at fold time — bounded depth (T9). + * + * snap_ckpt: the fold point. ckpt_seq = the last log seq absorbed into the + * rows; checksum = the S4 aggregate over ALL rows at fold time — the read + * side re-derives it and any mismatch kills the plane for the sid (R1); + * meta_json is the three-piece set (kernel state verbatim, coverage + * mapping, folded at fold time). */ +const SNAPBASE_DDL_V12 = `-- #5874 snapshot-table compression foundation (P0, v12). +CREATE TABLE IF NOT EXISTS l1_world ( + sess_raw TEXT PRIMARY KEY, + ledger_id TEXT NOT NULL, + witness_probe TEXT NOT NULL, + issued_at INTEGER NOT NULL, + armed_at INTEGER, + envelope_json TEXT, + envelope_checksum TEXT, + r3_latched_at INTEGER +); +CREATE INDEX IF NOT EXISTS idx_l1_ledger ON l1_world(ledger_id); +CREATE TABLE IF NOT EXISTS snap_rows ( + sid TEXT NOT NULL, + pos INTEGER NOT NULL, + core_ids TEXT NOT NULL, + pre_json TEXT, + post_json TEXT, + tokens_est INTEGER NOT NULL DEFAULT 0, + active INTEGER NOT NULL DEFAULT 1, + cover_ref TEXT, + updated_at INTEGER NOT NULL, + PRIMARY KEY (sid, pos) +); +CREATE TABLE IF NOT EXISTS snap_log ( + sid TEXT NOT NULL, + seq INTEGER NOT NULL, + kind TEXT NOT NULL, + payload TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (sid, seq) +); +CREATE TABLE IF NOT EXISTS snap_ckpt ( + sid TEXT PRIMARY KEY, + ckpt_seq INTEGER NOT NULL, + meta_json TEXT NOT NULL DEFAULT '{}', + checksum TEXT NOT NULL, + updated_at INTEGER NOT NULL +); +`; + /** Applied inside one transaction whenever user_version < STORAGE_SCHEMA_VERSION. */ export function migrateSchema(db: { exec(sql: string): void; @@ -319,6 +409,38 @@ export function migrateSchema(db: { }): { from: number; to: number } { const row = db.prepare("PRAGMA user_version").get() as { user_version: number } | undefined; const from = row?.user_version ?? 0; + // v12 (#5874) is PROBE-FIRST, not version-gated. A sibling branch may + // have consumed the version numbers after v11 for its own additive DDL + // while ours was still unmerged — real-world 2026-11-08: #5855 landed + // v12+v13 (the pf_* planes) into a SHARED database, and this branch's + // old `if (from < 12)` block was skipped by the early return below, so + // the plane died on first save with "no such table: l1_world". The DDL + // is pure CREATE ... IF NOT EXISTS (no backfill, no rewrite of existing + // rows), so it is safe to run at ANY from-version; probing for the + // table itself (the discipline v2/v5/v6/v8 already use) makes both + // merge orderings coexist without renumbering — the final master + // version number stays a mechanical merge-time hygiene step. + const hasSnapRows = db + .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'snap_rows'") + .get(); + if (!hasSnapRows) db.transaction(() => db.exec(SNAPBASE_DDL_V12)); + // 2×2 (#5874 brief, element-level world): the L1 world carries the + // per-element msgid ENVELOPE ({id, pos} canon, §4.0) in addition to the + // session-level witness_probe. Pure additive probe-first ALTER (a dev + // window that created l1_world before this column exists upgrades with + // one ALTER; a fresh DB already has it from the DDL above). The 2×2 + // stage-0 (src/snapshot/identity.ts matchWorld) reads/writes these; the + // session-level v1 path (#5901) leaves them NULL and is unaffected. + const l1Cols = (db.prepare("PRAGMA table_info(l1_world)").all() as unknown as { name?: unknown }[]) + .map((c) => c.name); + if (l1Cols.length > 0) { + if (!l1Cols.includes("envelope_json")) db.exec("ALTER TABLE l1_world ADD COLUMN envelope_json TEXT"); + if (!l1Cols.includes("envelope_checksum")) db.exec("ALTER TABLE l1_world ADD COLUMN envelope_checksum TEXT"); + // #6181 M3: the R3 breaker latch PERSISTS (a proxy restart must not + // re-arm a misbehaving host's sid and re-400 the same broken + // envelope); NULL/0 = not latched, else the epoch-ms of the latch. + if (!l1Cols.includes("r3_latched_at")) db.exec("ALTER TABLE l1_world ADD COLUMN r3_latched_at INTEGER"); + } if (from >= STORAGE_SCHEMA_VERSION) return { from, to: from }; let needV8Backfill = false; db.transaction(() => { @@ -442,6 +564,13 @@ export function migrateSchema(db: { db.exec("CREATE INDEX IF NOT EXISTS idx_message_media_media_hash ON message_media(media_hash)"); db.exec("CREATE INDEX IF NOT EXISTS idx_checkpoints_state_hash ON checkpoints(state_hash)"); } + // v12 (#5874 P0): snapshot-table compression foundation — ensured + // PROBE-FIRST above (coexists with sibling-branch DDL that consumed + // the v12/v13 numbers, see the note before the early return). + // Pure additivity: no backfill, no rewrite of existing rows; the + // plane stays dormant until storage.snapBase.enabled turns it on. + // Rollback = DROP the four tables + index and set user_version back + // (T1 pins the byte-identical behavior in both directions). }); if (needV8Backfill) migrateV8Identities(db); db.exec(`PRAGMA user_version = ${STORAGE_SCHEMA_VERSION}`); diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 8c3062e01d..7785b168dd 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -280,7 +280,9 @@ export class UnifiedShadow { : { name: "mysql", source: "network" }; } - /** Opens the sidecar when a knob asks for one; null (no cost) otherwise. */ + /** Opens the sidecar when a knob asks for one; null (no cost) otherwise. + * fileSnapBase (#5874): the two-level plane flags from the config file + * (knobs.fileSnapBase — already master⊃lane resolved). */ static maybeOpen( dir: string, env: NodeJS.ProcessEnv = process.env, @@ -288,6 +290,7 @@ export class UnifiedShadow { fileUnified?: string, fileWalLimit?: number, fileBackend?: string, + fileSnapBase?: { enabled: boolean; shadowWrite: boolean }, ): UnifiedShadow | null { const mode = resolveStorageMode(env, fileUnified); if (mode === "legacy") return null; @@ -307,7 +310,10 @@ export class UnifiedShadow { let store: UnifiedStore; try { try { - store = UnifiedStore.open(`${dir}/index.db`, undefined, { walJournalSizeLimitBytes: walLimit }); + store = UnifiedStore.open(`${dir}/index.db`, undefined, { + walJournalSizeLimitBytes: walLimit, + snapBase: fileSnapBase, + }); } catch (err) { // #4175: anything else throwing StorageIntegrityError means quick_check // proved the file set corrupt — handled below WITHOUT moving any file @@ -394,9 +400,11 @@ export class UnifiedShadow { fileWalLimit?: number, fileBackend?: string, fileMysqlDsn?: string, + fileSnapBase?: { enabled: boolean; shadowWrite: boolean }, ): Promise { const backendName = resolveBackendName(env, fileBackend); - if (backendName !== "mysql") return UnifiedShadow.maybeOpen(dir, env, legacyLoad, fileUnified, fileWalLimit, fileBackend); + if (backendName !== "mysql") + return UnifiedShadow.maybeOpen(dir, env, legacyLoad, fileUnified, fileWalLimit, fileBackend, fileSnapBase); const mode = resolveStorageMode(env, fileUnified); if (mode === "legacy") return null; const dsn = (env.BILI_MYSQL_DSN ?? fileMysqlDsn ?? "").trim(); @@ -446,11 +454,11 @@ export class UnifiedShadow { * message log authoritatively, so the ingest uses the raw tail whenever * available; both digest legs go through the same override, so parity * accounting stays honest. */ - record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore): void { + record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore, livePlane?: CoreMessage[]): void { // #3931 step 3: network backend — the ingest runs on the serial // queue (fire-and-forget; failures are logged once per distinct message — never disabled). if (this.sync === null) { - this.enqueueAsyncRecord(record, rawTail, liveCcr); + this.enqueueAsyncRecord(record, rawTail, liveCcr, livePlane); return; } // #5627: which stage is running — the failure forensics must name @@ -480,6 +488,12 @@ export class UnifiedShadow { // stay empty for every plain compressed session. Both digest legs // flow through the same override. if (liveCcr) legacy = { ...legacy, contentStore: liveCcr }; + // #5874 mapEgress: the live kernel turn view (tagged processTurn + // output) rides the save the same way — the envelope record never + // serializes it, so without this override the plane would never + // see the tagged forms and every save would fall back to the + // legacy CAS bytes. + if (livePlane && livePlane.length > 0) legacy = { ...legacy, lastPlaneMessages: livePlane }; // #3133 watermark: pay the full O(log) ingest only when the // trusted-prefix probe cannot vouch for the stored prefix — // steady-state saves stringify+hash+INSERT just the NEW tail. @@ -965,6 +979,64 @@ export class UnifiedShadow { return this.sync.sessionStorageStats(sessionId); } + /** #5874: snapshot-plane status (flag state + counters + dead latches). + * null when there is no sync store (async backend). */ + snapplaneStatus(): import("./snap.js").SnapPlaneStatusWire | null { + if (this.sync === null) return null; + return this.sync.snapplaneStatus(); + } + + /** #5874 P2 (mapEgress): S4-verified plane view for egress assembly. + * null when there is no sync store (async backend) — callers keep the + * legacy egress bytes. */ + snapEgressView(sessionId: string): import("./snap.js").SnapView | null { + if (this.sync === null) return null; + return this.sync.snapEgressView(sessionId); + } + + // #5874 2×2 stage 0 (brief 19544722): the element world-match accessors. + // Errors propagate (the lane maps the R3 reject to a hard 400 and every + // other fault to the legacy full path) — unlike snapEgressView these may + // NOT swallow. + + /** Stage 0 for one turn (see UnifiedStore.snapStage0). null when there + * is no sync store. */ + snapStage0(sessionId: string, incoming: string[], opts?: { resync?: boolean, seqs?: number[] | null }): import("../snapshot/plane.js").Stage0Result | null { + if (this.sync === null) return null; + return this.sync.snapStage0(sessionId, incoming, opts); + } + + /** The plane's ACTIVE cores, pos-ordered (old cores from the table). */ + snapStage0Cores(sessionId: string): { rows: { pos: number; ids: string[]; postJson: string }[]; coverMax: Record } | null { + if (this.sync === null) return null; + return this.sync.snapStage0Cores(sessionId); + } + + /** #6181 round 2, Residual 2 hardening: the plane's max ACTIVE pos + * (-1 when empty) — see UnifiedStore.snapPlaneMaxPos. */ + snapPlaneMaxPos(sessionId: string): number { + if (this.sync === null) return -1; + return this.sync.snapPlaneMaxPos(sessionId); + } + + /** The stage-0 resync (R4 / table-behind): reset the session plane. */ + snapStage0Reset(sessionId: string): void { + if (this.sync === null) return; + this.sync.snapStage0Reset(sessionId); + } + + /** #6181 M3: persisted R3 breaker latch (see UnifiedStore.snapR3Latched). */ + snapR3Latched(sessionId: string): boolean { + if (this.sync === null) return false; + return this.sync.snapR3Latched(sessionId); + } + + /** #6181 M3: persist the R3 breaker latch (best-effort). */ + snapR3Latch(sessionId: string): void { + if (this.sync === null) return; + this.sync.snapR3Latch(sessionId); + } + vacuum(mode: "full" | "incremental"): void { if (this.sync === null) throw new Error("[storage] sync vacuum unavailable on a network backend"); return this.sync.vacuum(mode); @@ -1174,7 +1246,7 @@ export class UnifiedShadow { * above (orphan guard + #3841 reap journal, #3133 watermark delta, * full ingest, #3133 sampled verify + drift latch). Serialized by the * shadow queue; failures feed the same per-message warn — nothing disables (#2432-era breaker removed 2026-10-30). */ - private enqueueAsyncRecord(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore): void { + private enqueueAsyncRecord(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore, livePlane?: CoreMessage[]): void { void this.enqueue(async () => { // #5627: same failing-stage forensics as the sync record() above. let phase = "legacy-load"; @@ -1191,7 +1263,7 @@ export class UnifiedShadow { legacy = { ...legacy, lastMessages: rawTail.lastMessages, lastMessagesFolded: rawTail.lastMessagesFolded === true }; } if (liveCcr) legacy = { ...legacy, contentStore: liveCcr }; - // #3818/#3841: same orphan-guard + reap-journal semantics. + if (livePlane && livePlane.length > 0) legacy = { ...legacy, lastPlaneMessages: livePlane }; // #4216/#4242: orphan shape rides the plan journal — normal // ingest (the async twin of the sync fix); no freeze. // #3133 watermark delta. diff --git a/src/storage/snap.ts b/src/storage/snap.ts new file mode 100644 index 0000000000..7ad937187e --- /dev/null +++ b/src/storage/snap.ts @@ -0,0 +1,842 @@ +/** + * snap.ts — #5874 snapshot-table compression foundation (P0: plane core, P1: + * shadow write). + * + * Design anchor: discussions/5874 "snapshot-table compression foundation + * (six-stage fixed chain, single storage truth source)". This file is the + * storage half of the chain — the four tables (l1_world / snap_rows / + * snap_log / snap_ckpt, schema v12), the commit primitive, the read-side + * rebuild with S4 verification, and the L1 identity ledger. It does NOT + * touch the compression kernel (zero-change hard constraint) and does NOT + * change any outbound bytes in P0/P1: the plane is a SHADOW — it writes + * alongside the legacy path and the read side only compares, never serves. + * + * Invariants enforced here (proposal §5): + * i single writer per session (one process — H; WAL serializes); + * ii seq strictly monotonic, always > max(log max, ckpt_seq) — the commit + * primitive derives seq; callers never choose it; + * iii S4 checksum mismatch ⇒ the plane is permanently dead for that sid + * (R1) — the dead flag is sticky (see snapPlaneDead); + * iv rows are never physically deleted by events — delete/compact flip + * active=0 and keep pre/post forms (both forms retained); physical + * cleanup happens only when the covering block's row is gone + * (snapGcCoveredRows); + * v post_json / pre_json carry the canonical core shape (unified-store + * message rows), NEVER wire bytes. + * + * Commit primitive = one transaction (proposal §5): upsert snap_rows → + * append snap_log (seq per invariant ii) → optional fold into snap_ckpt + * (fold truncates log entries ≤ ckpt_seq and records the three-piece meta: + * kernel state verbatim + coverage mapping + S4 aggregate). Any statement + * throwing rolls the WHOLE commit back — no half-state (T2). + * + * The red conditions this file participates in: R1 (S4 mismatch → plane + * dead for the sid, sticky) and R2 (join-chain break → the round produces + * no events and the legacy path carries it — expressed as: the plane never + * throws at the write hook; derivation failures are logged decisions, not + * errors, and the round is simply absent from the plane, self-healing on + * the next save because the plane is derived from final DB state). + */ + +import { sha256Hex } from "./store.ts"; + +/** A single structured plane event (proposal §4 event kinds). Events carry + * NO address of their own beyond the ledger position — addressing comes + * from the kernel pipeline's own evidence (block coverage intervals / + * ledger seqs / frame boundaries), never fuzzy similarity (R2: an event + * that cannot be addressed does not exist for this round). */ +export type SnapEvent = + | { ev: "cast"; pos: number; coreIds: string[]; preJson: string | null; postJson: string; tokensEst: number } + | { ev: "append"; pos: number; coreIds: string[]; preJson: string | null; postJson: string; tokensEst: number } + | { ev: "update"; pos: number; postJson: string; coreIds?: string[]; tokensEst?: number } + | { ev: "delete"; pos: number } + | { ev: "compact"; blockId: string; posRange: [number, number] }; + +/** Read-side view of one session's plane (proposal stage 4, mapEgress's + * input in P2 — in P1 it is the A/B oracle). `rows` is pos-ordered with + * the S4 checksums recomputed; `tailSeq`/`checksum` anchor verification. */ +export interface SnapView { + sid: string; + rows: { + pos: number; + coreIds: string[]; + preJson: string | null; + postJson: string | null; + tokensEst: number; + active: 0 | 1; + coverRef: string | null; + /** S4 row checksum, recomputed at read time (never stored). */ + rowSum: string; + }[]; + /** Last applied log seq (0 when no log entries), always ≥ ckptSeq. */ + tailSeq: number; + ckptSeq: number; + /** S4 aggregate over all rows in pos order, recomputed at read time. */ + checksum: string; + /** Stored verification anchor: last log sumAfter (or ckpt checksum). */ + expectedChecksum: string; + /** true when the recomputed aggregate matches the stored anchor. */ + verified: boolean; + ckptMeta: SnapCkptMeta | null; +} + +/** snap_ckpt.meta_json — the three-piece set (proposal §5): kernel state + * verbatim (the sessions.kstate_json blob, copied byte-for-byte), the + * coverage mapping (summary blocks → covered pos intervals, in pos order), + * and the S4 aggregate (mirrored in the checksum column). */ +interface SnapCkptMeta { + kstate: string; + kstateHash: string; + coverage: [string, number, number][]; + s4: string; + foldedAt: number; +} + +/** R1: S4 checksum mismatch — the plane is permanently dead for this sid. + * Carries the sid so the caller can latch the sticky dead flag. */ +export class SnapPlaneDeadError extends Error { + constructor(readonly sid: string, detail: string) { + super(`[snaplane] sid=${sid} dead (R1): ${detail}`); + this.name = "SnapPlaneDeadError"; + } +} + +/** Sticky R1 dead flag (proposal: "S4 checksum mismatch ⇒ permanently close + * that sid's plane" — per-sid, permanent for the process lifetime). The + * flag is process-local: a restart re-derives from storage and re-verifies + * (a fresh view that verifies cleanly stays open; one that does not dies + * again — same outcome, no cross-restart latch needed because verification + * is the latch). */ +const deadSids = new Set(); +export function isSnapPlaneDead(sid: string): boolean { + return deadSids.has(sid); +} +export function markSnapPlaneDead(sid: string): void { + deadSids.add(sid); +} +/** 2×2 stage-0 resync: clear ONE sid's sticky dead latch (the plane is + * being re-baselined from the client's current array — a stale R1 verdict + * about the OLD rows must not shadow the new plane). */ +export function clearSnapPlaneDead(sid: string): void { + deadSids.delete(sid); +} +/** Test-only: clear sticky flags between hermetic fixtures. */ +export function _resetSnapPlaneDeadForTests(): void { + deadSids.clear(); +} + +/** S4 row checksum over the stored form (invariant v: canonical core + * shape only). NULL pre/post contribute an empty field (a cast's pre_json + * may be NULL for the first-ever form of a position). */ +function snapRowChecksum( + sid: string, + pos: number, + coreIds: string, + preJson: string | null, + postJson: string | null, + tokensEst: number, + active: number, + coverRef: string | null, +): string { + return sha256Hex( + `${sid}\u0000${pos}\u0000${coreIds}\u0000${preJson ?? ""}\u0000${postJson ?? ""}\u0000${tokensEst}\u0000${active}\u0000${coverRef ?? ""}`, + ); +} + +/** S4 aggregate over all rows of a session, pos-ordered. The read side + * recomputes this and compares against the stored anchor; the write side + * stores it as each log entry's `sumAfter` (and the fold's checksum). */ +function snapAggregate(rows: { + pos: number; + core_ids: string; + pre_json: string | null; + post_json: string | null; + tokens_est: number; + active: number; + cover_ref: string | null; +}[]): string { + const parts = rows.map((r) => + sha256Hex( + `${r.pos}\u0000${r.core_ids}\u0000${r.pre_json ?? ""}\u0000${r.post_json ?? ""}\u0000${r.tokens_est}\u0000${r.active}\u0000${r.cover_ref ?? ""}`, + ), + ); + return sha256Hex(parts.join("\u0000")); +} + +/** #5874 mapEgress anchor-row position domain: a PLANE pos is either a + * ledger position (>= 0 — the messages-table seq space, the compact / + * delete / coverage domain) or a negative ANCHOR position (the stored + * render of an active block's summary). Anchor rows carry no ledger + * position — the egress re-inserts each render at its pair-safe coverage + * index — so they are parked in the negative space, which can never + * collide with a ledger seq (seqs are >= 0) and never enters a compact + * range (compacts stay >= 0). Bounds are generous (a session's block + * count, not its message count, allocates anchor positions). */ +const ANCHOR_POS_MIN = -100000; + +/** #5874 2×2: the stored anchor-row id prefix (acp_summary_). The + * write side, the D view and the 2×2 stage-0 assembly all key on it — + * one constant, one source of truth. */ +export const ANCHOR_ID_PREFIX = "acp_summary_"; + +function isAnchorPosDomain(pos: number, coreIds: string[] | undefined): boolean { + return pos >= ANCHOR_POS_MIN && pos < 0 && coreIds !== undefined && coreIds.some((id) => id.startsWith(ANCHOR_ID_PREFIX)); +} + +/** Validate event payloads (invariant v + addressing sanity). Throws on + * any violation — inside the commit transaction that rolls the whole + * commit back (T2: no half-state), and at the derivation hook where the + * caller catches and degrades the round to legacy (R2). */ +function validateEvents(sid: string, events: SnapEvent[]): void { + if (!events.length) return; + if (!sid) throw new Error(`[snaplane] sid=${sid} cast/append: empty sid`); + const seen = new Set(); + for (const e of events) { + switch (e.ev) { + case "cast": + case "append": { + if (!Number.isInteger(e.pos) || (e.pos < 0 && !isAnchorPosDomain(e.pos, e.coreIds))) { + throw new Error(`[snaplane] sid=${sid} ${e.ev}: bad pos ${e.pos}`); + } + if (seen.has(e.pos)) throw new Error(`[snaplane] sid=${sid} ${e.ev}: duplicate pos ${e.pos}`); + seen.add(e.pos); + if (!Array.isArray(e.coreIds) || !e.coreIds.length || !e.coreIds.every((c) => typeof c === "string" && c.length > 0)) { + throw new Error(`[snaplane] sid=${sid} ${e.ev} pos=${e.pos}: empty core_ids`); + } + requireCoreShape(sid, e.ev, e.pos, e.postJson); + if (e.preJson !== null) requireCoreShape(sid, e.ev, e.pos, e.preJson); + if (!Number.isInteger(e.tokensEst) || e.tokensEst < 0) { + throw new Error(`[snaplane] sid=${sid} ${e.ev} pos=${e.pos}: bad tokens_est ${e.tokensEst}`); + } + break; + } + case "update": { + // update carries no coreIds of its own when omitted — a + // negative pos here addresses the existing anchor row. + if (!Number.isInteger(e.pos) || (e.pos < 0 && !isAnchorPosDomain(e.pos, e.coreIds ?? []))) { + throw new Error(`[snaplane] sid=${sid} update: bad pos ${e.pos}`); + } + if (seen.has(e.pos)) throw new Error(`[snaplane] sid=${sid} update: duplicate pos ${e.pos}`); + seen.add(e.pos); + requireCoreShape(sid, "update", e.pos, e.postJson); + break; + } + case "delete": { + // delete is position-only (no coreIds): a negative pos may + // retire an anchor row (block consumed / render superseded). + if (!Number.isInteger(e.pos) || e.pos < ANCHOR_POS_MIN) { + throw new Error(`[snaplane] sid=${sid} delete: bad pos ${e.pos}`); + } + break; + } + case "compact": { + const [lo, hi] = e.posRange; + if (!Number.isInteger(lo) || !Number.isInteger(hi) || lo < 0 || hi < lo || !e.blockId) { + throw new Error(`[snaplane] sid=${sid} compact: bad range [${lo},${hi}] block=${e.blockId}`); + } + break; + } + default: { + const unknown = e as { ev: string }; + throw new Error(`[snaplane] sid=${sid}: unknown event kind ${unknown.ev}`); + } + } + } +} + +/** Invariant v: pre/post payloads must be canonical core shape — JSON + * documents (unified message rows), never wire bytes. A payload that is + * not parseable JSON fails the round at the write side (R2 degrade) and + * the commit (T2 rollback). */ +function requireCoreShape(sid: string, ev: string, pos: number, json: string): void { + try { + JSON.parse(json); + } catch { + throw new Error(`[snaplane] sid=${sid} ${ev} pos=${pos}: payload is not canonical core shape (not JSON)`); + } +} + +/** The minimal db surface the plane functions need (exported so the stage-0 + * decision module (src/snapshot/plane.ts) can be type-coupled to it without + * importing a driver). */ +export interface DbLike { + exec(sql: string, ...params: unknown[]): unknown; + prepare(sql: string): { + run(...params: unknown[]): unknown; + get(...params: unknown[]): unknown; + all(...params: unknown[]): unknown[]; + }; + /** Same contract as the storage driver's transaction: runs fn inside + * BEGIN/COMMIT and returns fn's own result (nested calls savepoint). */ + transaction(fn: () => T): T; +} + +interface RowRaw { + pos: number; + core_ids: string; + pre_json: string | null; + post_json: string | null; + tokens_est: number; + active: number; + cover_ref: string | null; +} + +/** Read the raw row form for one session, pos-ordered. Exported for the + * write side's post-apply aggregate (it must hash EXACTLY what was + * stored — reading back through prepared statements, not echoing inputs). */ +function readRawRows(db: DbLike, sid: string): RowRaw[] { + const rows = db + .prepare("SELECT pos, core_ids, pre_json, post_json, tokens_est, active, cover_ref FROM snap_rows WHERE sid = ? ORDER BY pos") + .all(sid) as RowRaw[]; + return rows; +} + +function applyEvent(db: DbLike, sid: string, e: SnapEvent, now: number): void { + switch (e.ev) { + case "cast": + case "append": { + // cast and append differ only in expectation (cast = position + // the plane had not materialized at its last fold; append = + // next position after the tail). The table treats both as + // "materialize this position NOW" — pre_json carries the prior + // form (NULL when none), post_json the new one (invariant v). + db.prepare( + `INSERT INTO snap_rows (sid, pos, core_ids, pre_json, post_json, tokens_est, active, cover_ref, updated_at) + VALUES (?, ?, ?, ?, ?, ?, 1, NULL, ?) + ON CONFLICT (sid, pos) DO UPDATE SET + core_ids = excluded.core_ids, + pre_json = snap_rows.post_json, + post_json = excluded.post_json, + tokens_est = excluded.tokens_est, + active = 1, + cover_ref = NULL, + updated_at = excluded.updated_at`, + ).run( + sid, + e.pos, + JSON.stringify(e.coreIds), + e.preJson ?? null, + e.postJson, + e.tokensEst ?? 0, + now, + ); + break; + } + case "update": { + // In-place re-form: pre_json becomes the OLD post (the two-form + // retention, invariant iv), post_json the new form. + const row = db + .prepare("SELECT core_ids, post_json, tokens_est, active, cover_ref FROM snap_rows WHERE sid = ? AND pos = ?") + .get(sid, e.pos) as + | { core_ids: string; post_json: string | null; tokens_est: number; active: number; cover_ref: string | null } + | undefined; + if (!row) { + // A plane without the row cannot apply an in-place update — + // that would be fabricating a position (R2: unaddressable). + throw new Error(`[snaplane] sid=${sid} update pos=${e.pos}: no row to update (R2)`); + } + db.prepare( + `UPDATE snap_rows SET + core_ids = ?, pre_json = post_json, post_json = ?, tokens_est = ?, active = 1, cover_ref = NULL, updated_at = ? + WHERE sid = ? AND pos = ?`, + ).run( + e.coreIds ? JSON.stringify(e.coreIds) : row.core_ids, + e.postJson, + e.tokensEst ?? row.tokens_est, + now, + sid, + e.pos, + ); + break; + } + case "delete": { + // Invariant iv: flip, never delete. Both forms stay; cover_ref + // marks the reason (NULL = host-side removal, no covering block). + db.prepare( + "UPDATE snap_rows SET active = 0, cover_ref = COALESCE(cover_ref, 'deleted'), updated_at = ? WHERE sid = ? AND pos = ?", + ).run(now, sid, e.pos); + break; + } + case "compact": { + // The summary block covers [lo, hi]: flip the whole interval, + // keep every pre/post form (iv), stamp the covering block so + // GC knows these rows die when the block's row goes. + db.prepare( + "UPDATE snap_rows SET active = 0, cover_ref = ?, updated_at = ? WHERE sid = ? AND pos >= ? AND pos <= ?", + ).run(e.blockId, now, sid, e.posRange[0], e.posRange[1]); + break; + } + } +} + +/** The commit primitive (proposal §5, one transaction): apply events in + * order, append one log entry per event (kind = ev, payload = the event + + * post-apply aggregate), then optionally fold into snap_ckpt. seq is + * derived: max(log max, ckpt_seq) + 1 strictly increasing (invariant ii). + * Throws ⇒ the whole transaction rolls back — the plane is exactly as it + * was before the call (T2). The P1 hook composes this with the L1 record + * in ONE store transaction, so the non-wrapped form is also exported. */ +export function applySnapCommit( + db: DbLike, + sid: string, + events: SnapEvent[], + opts: { fold?: boolean; now?: number } = {}, +): { tailSeq: number; folded: boolean; checksum: string } { + validateEvents(sid, events); + return db.transaction(() => applySnapInTransaction(db, sid, events, opts)); +} + +/** Same as applySnapCommit but assumes an open store transaction (node: + * sqlite throws on nested db.transaction calls — the P1 hook needs the + * L1 write and the plane commit in the SAME transaction). */ +export function applySnapInTransaction( + db: DbLike, + sid: string, + events: SnapEvent[], + opts: { fold?: boolean; now?: number } = {}, +): { tailSeq: number; folded: boolean; checksum: string } { + validateEvents(sid, events); + const now = opts.now ?? Date.now(); + let seq = db.prepare("SELECT COALESCE(MAX(seq), 0) AS m FROM snap_log WHERE sid = ?").get(sid) as { m: number }; + let ckptSeq = (db.prepare("SELECT ckpt_seq FROM snap_ckpt WHERE sid = ?").get(sid) as { ckpt_seq: number } | undefined)?.ckpt_seq ?? 0; + let next = Math.max(seq.m, ckptSeq); + if (!events.length) { + // No events: nothing to log; folding a clean plane still + // re-anchors (used to compact a deep log after a no-op round). + const rows = readRawRows(db, sid); + if (opts.fold) return fold(db, sid, next, rows, now); + return { tailSeq: seq.m, folded: false, checksum: snapAggregate(rows) }; + } + for (const e of events) { + applyEvent(db, sid, e, now); + // Post-apply aggregate: re-read the stored form (what R1 + // verifies against later is exactly what was stored). + const rows = readRawRows(db, sid); + const sumAfter = snapAggregate(rows); + next += 1; + const payload = JSON.stringify({ + ev: e, + sumAfter, + }); + db.prepare("INSERT INTO snap_log (sid, seq, kind, payload, created_at) VALUES (?, ?, ?, ?, ?)").run( + sid, + next, + e.ev, + payload, + now, + ); + } + if (opts.fold) { + const rows = readRawRows(db, sid); + const out = fold(db, sid, next, rows, now); + return out; + } + return { tailSeq: next, folded: false, checksum: snapAggregate(readRawRows(db, sid)) }; +} + +/** Fold: checkpoint at the current log tail. meta_json carries the + * three-piece set; log entries ≤ ckpt_seq are truncated (the rows table + * is the truth — folding is pure re-anchoring, T7/T9). ckptSeq may stay + * at 0 when folding an empty log (first anchor before any event). */ +function fold( + db: DbLike, + sid: string, + ckptSeq: number, + rows: RowRaw[], + now: number, +): { tailSeq: number; folded: boolean; checksum: string } { + const kstate = (db.prepare("SELECT kstate_json FROM sessions WHERE session_id = ?").get(sid) as { kstate_json: string } | undefined) + ?.kstate_json ?? "{}"; + const coverage = db + .prepare( + `SELECT block_id, start_seq, end_seq FROM blocks + WHERE session_id = ? AND active = 1 AND start_seq >= 0 AND end_seq >= start_seq + ORDER BY start_seq`, + ) + .all(sid) as { block_id: string; start_seq: number; end_seq: number }[]; + const checksum = snapAggregate(rows); + const meta: SnapCkptMeta = { + kstate, + kstateHash: sha256Hex(kstate), + coverage: coverage.map((c) => [c.block_id, c.start_seq, c.end_seq]), + s4: checksum, + foldedAt: now, + }; + db.prepare( + `INSERT INTO snap_ckpt (sid, ckpt_seq, meta_json, checksum, updated_at) VALUES (?, ?, ?, ?, ?) + ON CONFLICT (sid) DO UPDATE SET ckpt_seq = excluded.ckpt_seq, meta_json = excluded.meta_json, + checksum = excluded.checksum, updated_at = excluded.updated_at`, + ).run(sid, ckptSeq, JSON.stringify(meta), checksum, now); + db.prepare("DELETE FROM snap_log WHERE sid = ? AND seq <= ?").run(sid, ckptSeq); + return { tailSeq: ckptSeq, folded: true, checksum }; +} + +/** Read-side rebuild (proposal stage 4, P1 use: the A/B oracle). Returns + * the full pos-ordered row set with recomputed S4, the stored anchor, and + * the verification verdict. Throws SnapPlaneDeadError (R1) when the + * aggregate does not match the anchor — AND latches the sticky dead flag + * (invariant iii: the plane stays closed for this sid for the process + * lifetime; further calls fail fast without touching the tables). + * + * A session with no rows and no log and no ckpt has no plane yet: + * returns null (NOT dead — nothing to verify). */ +export function readSnapView(db: DbLike, sid: string): SnapView | null { + if (isSnapPlaneDead(sid)) { + throw new SnapPlaneDeadError(sid, "plane dead (R1, sticky)"); + } + const rows = readRawRows(db, sid); + const tail = db.prepare("SELECT COALESCE(MAX(seq), 0) AS m FROM snap_log WHERE sid = ?").get(sid) as { m: number }; + const ckptRow = db.prepare("SELECT ckpt_seq, meta_json, checksum FROM snap_ckpt WHERE sid = ?").get(sid) as + | { ckpt_seq: number; meta_json: string; checksum: string } + | undefined; + const tailSeq = tail.m; + const ckptSeq = ckptRow?.ckpt_seq ?? 0; + if (!rows.length && !tailSeq && !ckptRow) return null; + + // Stored anchor: the last log entry's sumAfter (it reflects EVERY row, + // not just the touched ones — that is what makes a single hash verify + // the whole plane), else the ckpt checksum, else (rows exist but no + // anchor — only reachable via manual tampering or a hand-edited + // database) unverified. + let expectedChecksum: string; + if (tailSeq > ckptSeq) { + const lastLog = db + .prepare("SELECT payload FROM snap_log WHERE sid = ? AND seq = ?") + .get(sid, tailSeq) as { payload: string }; + expectedChecksum = (JSON.parse(lastLog.payload) as { sumAfter: string }).sumAfter; + } else { + expectedChecksum = ckptRow?.checksum ?? snapAggregate(rows); + } + + // Tail-seq monotonicity (invariant ii, read-side check): every log seq + // > ckpt_seq must be strictly increasing by 1 from ckpt_seq (the + // commit primitive guarantees it; a violation is tampering ⇒ R1). + if (tailSeq > ckptSeq) { + const logRows = db + .prepare("SELECT seq FROM snap_log WHERE sid = ? AND seq > ? ORDER BY seq") + .all(sid, ckptSeq) as { seq: number }[]; + for (let i = 0; i < logRows.length; i++) { + if (logRows[i].seq !== ckptSeq + i + 1) { + markSnapPlaneDead(sid); + throw new SnapPlaneDeadError(sid, `snap_log seq gap at ${logRows[i].seq} (expected ${ckptSeq + i + 1})`); + } + } + } + + const checksum = snapAggregate(rows); + if (checksum !== expectedChecksum) { + markSnapPlaneDead(sid); + throw new SnapPlaneDeadError(sid, `S4 aggregate mismatch (stored ${expectedChecksum.slice(0, 12)}…, recomputed ${checksum.slice(0, 12)}…)`); + } + + let ckptMeta: SnapCkptMeta | null = null; + if (ckptRow) { + try { + ckptMeta = JSON.parse(ckptRow.meta_json) as SnapCkptMeta; + } catch { + ckptMeta = null; // meta corruption is non-fatal for P1 (the rows are the truth) + } + } + + return { + sid, + rows: rows.map((r) => ({ + pos: r.pos, + coreIds: JSON.parse(r.core_ids) as string[], + preJson: r.pre_json, + postJson: r.post_json, + tokensEst: r.tokens_est, + active: (r.active ? 1 : 0) as 0 | 1, + coverRef: r.cover_ref, + rowSum: snapRowChecksum(sid, r.pos, r.core_ids, r.pre_json, r.post_json, r.tokens_est, r.active, r.cover_ref), + })), + tailSeq, + ckptSeq, + checksum, + expectedChecksum, + verified: true, + ckptMeta, + }; +} + +// --------------------------------------------------------------------------- +// L1 world — stage-0 identity ledger (proposal §0). +// +// sess_raw = the raw host session id as it arrived on the wire +// (conversations.conversation_id in v1); ledger_id = the proxy's own +// identity (sessions.session_id). The mapping is issued on first sight +// (issued_at) and ARMED only after one full round validates it +// (armed_at): the validation evidence is that a later round for the same +// ledger re-presents the same sess_raw with matching witness content — +// the "one round of arming" the proposal charges. witness_probe is a +// sampled fingerprint of the ids seen at issue time, so the arm check is +// cheap and does not store payload. +// --------------------------------------------------------------------------- + +interface L1Entry { + sessRaw: string; + ledgerId: string; + witnessProbe: string; + issuedAt: number; + armedAt: number | null; +} + +/** Record (or refresh) the sess_raw → ledger_id mapping. Returns the + * decision: + * - "hit": mapping existed and is armed — the round is identity- + * validated (the normal path for a live cohort session); + * - "echo": mapping existed and is armed, witness matched — same as + * hit, distinguished for the decision log (the echo + * spot-check the proposal mentions); + * - "arm": mapping existed, not yet armed, and this round's witness + * validates it — armed_at set now (the one-round charge + * completes); + * - "wait": mapping existed, not yet armed, witness did not validate + * — issued_at unchanged, no arm (still one round out); + * - "issue": new mapping issued (armed_at = NULL — the round that + * issued it cannot validate itself). + * A sess_raw that was issued under a DIFFERENT ledger_id is a hard + * identity conflict (R3 territory): returns "red" and writes NOTHING — + * the caller keeps the round on the legacy path and the breaker counts + * it. */ +export function snapL1Record( + db: DbLike, + sessRaw: string, + ledgerId: string, + witnessProbe: string, + now: number, +): "hit" | "echo" | "arm" | "wait" | "issue" | "red" { + const existing = db + .prepare("SELECT ledger_id, witness_probe, issued_at, armed_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { ledger_id: string; witness_probe: string; issued_at: number; armed_at: number | null } | undefined; + if (!existing) { + db.prepare( + "INSERT INTO l1_world (sess_raw, ledger_id, witness_probe, issued_at, armed_at) VALUES (?, ?, ?, ?, NULL)", + ).run(sessRaw, ledgerId, witnessProbe, now); + return "issue"; + } + if (existing.ledger_id !== ledgerId) { + // Hard conflict: the same raw session id now belongs to another + // ledger. Never overwrite — the mapping is the identity anchor; + // corrupting it would poison every later round for this host. + return "red"; + } + if (existing.armed_at !== null) { + return existing.witness_probe === witnessProbe ? "echo" : "hit"; + } + // Not armed yet: arm when this round re-validates the probe (the + // witness is content-derived, so an identical probe means the host's + // id space survived the round unchanged — the echo round). + if (witnessProbe === existing.witness_probe) { + db.prepare("UPDATE l1_world SET armed_at = ? WHERE sess_raw = ?").run(now, sessRaw); + return "arm"; + } + return "wait"; +} + +/** Read the L1 entry for a raw session id (status/diagnostics). */ +export function snapL1Lookup(db: DbLike, sessRaw: string): L1Entry | null { + const r = db + .prepare("SELECT sess_raw, ledger_id, witness_probe, issued_at, armed_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { sess_raw: string; ledger_id: string; witness_probe: string; issued_at: number; armed_at: number | null } | undefined; + if (!r) return null; + return { sessRaw: r.sess_raw, ledgerId: r.ledger_id, witnessProbe: r.witness_probe, issuedAt: r.issued_at, armedAt: r.armed_at }; +} + +// --------------------------------------------------------------------------- +// GC — physical cleanup (invariant iv's release valve). Rows flipped by +// delete/compact stay until their cover vanishes: a compacted row's +// cover_ref is the covering block, and when that block's row is gone +// (deleted or itself archived) the row has no consumer left and is +// physically removed. Rows covered by a LIVE block stay (the fold view +// still renders their summary). "deleted" cover_ref (host removal, no +// block) releases immediately — the position is gone from the host +// forever and no summary references it. Physical removal changes the row +// set the S4 anchor aggregates over, so any removal re-anchors in the +// same transaction (a pure-reanchoring fold at the current log tail): +// without it the last log sumAfter would cover rows that no longer exist +// and the next readSnapView would trip R1 on a healthy plane (sticky dead +// latch after an ordinary host-side rewind). Returns the number of rows +// physically removed. +// --------------------------------------------------------------------------- +export function snapGcCoveredRows(db: DbLike, sid: string): number { + return db.transaction(() => { + const rows = db + .prepare("SELECT pos, cover_ref FROM snap_rows WHERE sid = ? AND active = 0 AND cover_ref IS NOT NULL") + .all(sid) as { pos: number; cover_ref: string }[]; + if (!rows.length) return 0; + const coverSet = new Set(rows.map((r) => r.cover_ref)); + // Which covers still exist as block rows? + const placeholders = [...coverSet].map(() => "?").join(","); + const liveCovers = db + .prepare(`SELECT DISTINCT block_id FROM blocks WHERE session_id = ? AND block_id IN (${placeholders})`) + .all(sid, ...[...coverSet]) as { block_id: string }[]; + const liveCoverSet = new Set(liveCovers.map((c) => c.block_id)); + let removed = 0; + const del = db.prepare("DELETE FROM snap_rows WHERE sid = ? AND pos = ?"); + for (const r of rows) { + const coverGone = r.cover_ref === "deleted" || !liveCoverSet.has(r.cover_ref); + if (coverGone) { + del.run(sid, r.pos); + removed += 1; + } + } + if (removed > 0) { + const tail = db + .prepare("SELECT COALESCE(MAX(seq), 0) AS m FROM snap_log WHERE sid = ?") + .get(sid) as { m: number }; + fold(db, sid, tail.m, readRawRows(db, sid), Date.now()); + } + return removed; + }); +} + +// --------------------------------------------------------------------------- +// 2×2 (proposal §4.0, brief 19544722) — the element-level WORLD ENVELOPE. +// +// The session-level v1 path above (snapL1Record/snapL1Lookup) stays the +// identity ANCHOR (sess_raw → ledger_id + one-round arming). The 2×2 stage 0 +// adds, on the same row, the per-element msgid envelope: an ordered +// [{id, pos}] canon binding every host msgid to its append-only ledger pos +// (ruling G — the table is self-contained). The stage-0 decision function +// (src/snapshot/identity.ts matchWorld) runs on this envelope; the plane rows +// (snap_rows) keep the #5901 message-level model (pos = ledger seq). +// --------------------------------------------------------------------------- + +interface L1Envelope { + /** The canon envelope JSON: an array of {id, pos} in host order. + * Null when the session is still on the session-level v1 path (the + * 2×2 element world has not been issued for it yet). */ + envelopeJson: string | null; + /** sha256 of the canonical envelope (fast mismatch before the walk). */ + envelopeChecksum: string | null; +} + +/** Read the element-level envelope for a raw session id (null when the l1 + * row is absent; envelopeJson null when present but not yet 2×2-issued). */ +export function snapL1EnvelopeRead(db: DbLike, sessRaw: string): L1Envelope | null { + const r = db + .prepare("SELECT envelope_json, envelope_checksum FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { envelope_json: string | null; envelope_checksum: string | null } | undefined; + if (!r) return null; + return { envelopeJson: r.envelope_json, envelopeChecksum: r.envelope_checksum }; +} + +/** Persist the element-level envelope for a raw session id (UPDATE only — + * the l1 row itself is created by snapL1Record; a missing row is a + * programming error and is surfaced, not silently created). */ +export function snapL1EnvelopeUpsert(db: DbLike, sessRaw: string, envelopeJson: string, envelopeChecksum: string): void { + const res = db + .prepare("UPDATE l1_world SET envelope_json = ?, envelope_checksum = ? WHERE sess_raw = ?") + .run(envelopeJson, envelopeChecksum, sessRaw) as { changes: number }; + if (res.changes === 0) throw new Error(`snapL1EnvelopeUpsert: no l1_world row for sess_raw ${sessRaw.slice(0, 12)}… (record it first)`); +} + +/** #6181 M3: read the persisted R3 breaker latch (true when the sid's + * breaker latched in a PREVIOUS process — a restart must not re-arm a + * misbehaving host). Missing row or NULL → false. */ +export function snapR3LatchedRead(db: DbLike, sessRaw: string): boolean { + const r = db + .prepare("SELECT r3_latched_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { r3_latched_at: number | null } | undefined; + return !!r && !!r.r3_latched_at; +} + +/** #6181 M3: persist the R3 breaker latch. UPSERT (not UPDATE-only) — the + * first fault of a fresh session happens BEFORE its stage-0 snapL1Record + * call (the R3 check runs at pin validation), so an UPDATE-only latch + * would silently no-op for sessions with no l1 row yet. */ +export function snapR3LatchWrite(db: DbLike, sessRaw: string, nowMs: number): void { + // Upsert — the first fault of a fresh session happens BEFORE its stage-0 + // snapL1Record call (the R3 check runs at pin validation), so an + // UPDATE-only latch would silently no-op for sessions with no l1 row yet. + // ledger_id takes the placeholder sessRaw and witness_probe/issued_at the + // sentinel values: the latch row's envelope stays NULL until a later + // (repaired) stage-0 snapL1Record upsert carries the real ledger id and + // witness. A latched session never re-arms (runStage0 checks the persisted + // latch before any store call), so the placeholder can never hit the + // "red" ledger-conflict path of snapL1Record. + db.prepare("INSERT INTO l1_world (sess_raw, ledger_id, witness_probe, issued_at, r3_latched_at) VALUES (?, ?, '', ?, ?) ON CONFLICT(sess_raw) DO UPDATE SET r3_latched_at = excluded.r3_latched_at, ledger_id = COALESCE(l1_world.ledger_id, excluded.ledger_id), witness_probe = COALESCE(NULLIF(l1_world.witness_probe, ''), excluded.witness_probe)").run(sessRaw, sessRaw, nowMs, nowMs); +} + +// --------------------------------------------------------------------------- +// 2×2 read side — the FULL row set (active AND inactive), for the stage-0 +// hydrate: S4 drift checks, explainedGone (fold-covered ids may legally +// vanish from the head) and the coverage evidence all need the deactivated +// rows, which snapRowsView/readSnapView filter out of the egress path. +// --------------------------------------------------------------------------- + +interface SnapRowAll { + pos: number; + coreIds: string[]; + postJson: string | null; + active: 0 | 1; + coverRef: string | null; +} + +/** Every snap_rows entry for a sid, pos-ordered (active + inactive). Unlike + * readSnapView this does NOT verify S4 and does NOT latch R1 — it is a raw + * evidence read for the stage-0 decision; the verification path (readSnapView) + * is unchanged and remains the only S4 authority. */ +export function snapRowsAll(db: DbLike, sid: string): SnapRowAll[] { + const rows = readRawRows(db, sid); + return rows.map((r) => ({ + pos: r.pos, + coreIds: JSON.parse(r.core_ids) as string[], + postJson: r.post_json, + active: (r.active ? 1 : 0) as 0 | 1, + coverRef: r.cover_ref, + })); +} + +// --------------------------------------------------------------------------- +// Status (wire-safe, for the /__bili/status attribution surface). +// --------------------------------------------------------------------------- +interface SnapPlaneStatus { + /** master flag (storage.snapBase.enabled) — false ⇒ plane untouched. */ + enabled: boolean; + /** lane flag (storage.snapBase.shadowWrite) — false ⇒ shadow off. */ + shadowWrite: boolean; + /** per-sid dead latches currently held (R1). */ + deadSids: string[]; +} + +/** Wire shape of the /__bili/status `snaplane` field. */ +export type SnapPlaneStatusWire = SnapPlaneStatus & { + sessions: number; + rows: number; + activeRows: number; + logEntries: number; + ckpts: number; + l1Entries: number; + l1Armed: number; +}; + +/** Build the status payload from the store's db. `flags` come from the + * config layer (knobs) — the storage layer itself is flag-agnostic. */ +export function snapplaneStatusWire( + db: DbLike, + flags: { enabled: boolean; shadowWrite: boolean }, +): SnapPlaneStatusWire { + const one = (q: string): number => { + try { + return (db.prepare(q).get() as { n: number }).n; + } catch { + // Tables absent (v<12 database, or a test fixture built from + // the raw v1 DDL) — the plane is simply not here. + return 0; + } + }; + return { + enabled: flags.enabled, + shadowWrite: flags.shadowWrite, + deadSids: [...deadSids], + sessions: one("SELECT COUNT(DISTINCT sid) AS n FROM snap_rows"), + rows: one("SELECT COUNT(*) AS n FROM snap_rows"), + activeRows: one("SELECT COUNT(*) AS n FROM snap_rows WHERE active = 1"), + logEntries: one("SELECT COUNT(*) AS n FROM snap_log"), + ckpts: one("SELECT COUNT(*) AS n FROM snap_ckpt"), + l1Entries: one("SELECT COUNT(*) AS n FROM l1_world"), + l1Armed: one("SELECT COUNT(*) AS n FROM l1_world WHERE armed_at IS NOT NULL"), + }; +} diff --git a/src/storage/store.ts b/src/storage/store.ts index 1e71740687..93c7404167 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -22,12 +22,32 @@ import { dirname, join } from "node:path"; import { log as loggerLog } from "../logger.js"; import { openSqliteDatabase, type SqliteDatabase, type SqliteEngineName, type SqliteEngineSource } from "./driver.js"; import { migrateSchema, storagePragmas, WAL_JOURNAL_SIZE_LIMIT } from "./schema.js"; -import { identityOfStoredContent } from "./identity.js"; +import { identityOfStoredContent, splitMediaContent } from "./identity.js"; +import { + applySnapInTransaction, + clearSnapPlaneDead, + isSnapPlaneDead, + markSnapPlaneDead, + readSnapView, + snapGcCoveredRows, + snapL1EnvelopeRead, + snapL1Record, + snapR3LatchedRead, + snapR3LatchWrite, + snapplaneStatusWire, + snapRowsAll, + SnapPlaneDeadError, + type SnapEvent, + type SnapView, +} from "./snap.js"; +import { parseEnvelope } from "../snapshot/identity.js"; +import { FoundationRejectError, stage0Decide, type Stage0Result } from "../snapshot/plane.js"; import { recordIntegrityOutcome, recordSecondProcessOwner, recordBackupOutcome } from "./runtime-status.js"; import { integrityModeFor, writeIntegrityCache, scheduleBackgroundIntegrityCheck, backgroundIntegrityInflightForTests, noteIntegrityCheckRun } from "./integrity-check.js"; import { sqliteCorruptionDetail } from "./sqlite-errors.js"; import { listInstances, type RegistryEntry } from "../instance.js"; -import type { MessageContentStore } from "acp-kernel"; +import { baseIdOf } from "acp-kernel"; +import type { CoreMessage, MessageContentStore } from "acp-kernel"; import type { BlockView } from "../session.js"; /** #3250: run a PASSIVE wal_checkpoint every N successful ingests. The @@ -302,6 +322,13 @@ export interface SessionStorageStats { sessionExt: number; checkpoints: number; pinnedCheckpoints: number; + /** #5874: snapshot-plane rows for this session (folded + active), + * its log entries past the fold, its fold points, and the L1 + * identity entry it owns. Zero for a plane-never-written session. */ + snapRows: number; + snapLogEntries: number; + snapCkpts: number; + l1Entries: number; }; /** Bytes held in inline JSON columns of this session's rows (sessions * record + blocks payload_json + session_ext value_json) — everything @@ -344,6 +371,20 @@ export interface IngestSessionInput { * absolute seqs before insert — see remapPlanSeqs. */ pluginSnapshotSeqs?: number[] | null; tailSeqs?: number[] | null; + /** #5874 mapEgress direct authority: the lane's RAW tagged processTurn + * output for this turn (deep-copied before the in-request flush). The + * snapshot plane persists THESE bytes — the kernel's own tagged output, + * not the legacy transcript (invariant v: post_json is the post-rewrite + * core form). When present, snapShadowSync derives the plane events + * from this view; null falls back to the defensive full-transcript diff + * (saves that don't carry a turn view). */ + planeMessages?: CoreMessage[] | null; + /** #6181 B0: one-shot plane-write skip for THIS save — set by the lane + * for a turn that cannot be addressed against the plane (a non-cohort + * chained delta). snapShadowSync returns before any plane work; the + * flag is consumed at the persist build (the live session flag is + * cleared by the lane after the in-request flush). */ + skipPlaneFlush?: boolean; /** Serialized orphan map (Record) — blockContents * entries whose block is absent from state.blocks (#4216/#4242: the reap * shape is REPRESENTABLE; rides the same transaction so the session row, @@ -388,6 +429,20 @@ function likeEscape(prefix: string): string { return prefix.replace(/[\\%_]/g, (ch) => "\\" + ch); } +/** #5874 A/B oracle: effectiveMessageIds from a blocks row's payload_json + * (the verbatim CompressionBlock the ingest stored). '' on any parse + * failure — a malformed payload just misses the base-set, it never kills + * the oracle (the throw path latches dead like any A/B failure). */ +function effectiveIdsOfPayload(payload: string | null): string[] { + if (!payload) return []; + try { + const p = JSON.parse(payload) as { effectiveMessageIds?: unknown }; + return Array.isArray(p.effectiveMessageIds) ? (p.effectiveMessageIds as string[]) : []; + } catch { + return []; + } +} + /** * #4175: raised when PRAGMA quick_check fails at open. Migrations must never * run against a corrupt file, so this fires BEFORE migrateSchema (and from the @@ -475,6 +530,16 @@ export class UnifiedStore { private readonly db: SqliteDatabase; /** #3250: successful-ingest counter driving the WAL_CHECKPOINT_EVERY_INGESTS cadence. */ private ingestCount = 0; + /** #5874: snapshot-plane flags (storage.snapBase: master ⊃ shadowWrite + * lane). Off ⇒ the plane tables are never touched by the write hook. + * Set from UnifiedStore.open's opts — the constructor stays untouched + * (readSnapshotAsync's reader stores inherit the flag harmlessly: the + * reader connection is query-only and never ingests). + * DEFAULT OFF (discussion #5874 §9): when the config file is silent + * the plane is fully off; opt in with storage.snapBase. + * (2026-11-08 soak ran this at ON per owner directive; reverted to + * off on 2026-11-13 before the merge — review of #5901 round 2.) */ + snapPlane: { enabled: boolean; shadowWrite: boolean } = { enabled: false, shadowWrite: false }; /** #5744: last periodic checkpoint result (plus its wall time) for the * slow-ingest warn line — lets an operator attribute a stall spike to a * checkpoint burst instead of guessing. Null until the first tick. */ @@ -508,7 +573,7 @@ export class UnifiedStore { static open( path: string, engine?: SqliteEngineName, - opts?: { walJournalSizeLimitBytes?: number }, + opts?: { walJournalSizeLimitBytes?: number; snapBase?: { enabled?: boolean; shadowWrite?: boolean } }, ): UnifiedStore { mkdirSync(dirname(path), { recursive: true }); const { db, engine: used, source } = openSqliteDatabase(path, engine); @@ -704,6 +769,19 @@ export class UnifiedStore { loggerLog("warn", `[storage] WAL checkpoint on open failed for ${path}: ${String(err)}`); } const store = new UnifiedStore(db, used, source, lockPath, foreignLockHolder, path); + { + // #5874: the config layer already enforces master⊃lane (the + // lane flag only means something with the master flag); the + // && here is the storage layer's own belt (a store opened + // directly, bypassing knobs, can never shadow without the master). + // Default off when the config is absent (#5874 §9; the 2026-11-08 + // soak ON default was reverted before merge, 2026-11-13). + const f = opts?.snapBase ?? { enabled: false, shadowWrite: false }; + store.snapPlane = { + enabled: f.enabled === true, + shadowWrite: f.enabled === true && f.shadowWrite === true, + }; + } if (integrityOk) store.noteIntegrityOk(); // #4609: a large store's cache-miss check runs post-open on a second // read-only connection — open() returns immediately; the verdict @@ -1515,9 +1593,1070 @@ export class UnifiedStore { return true; }); if (ok) this.maybeCheckpointWal(); + if (ok) this.snapShadowSync(input, delta); return ok; } + // ----------------------------------------------------------------------- + // #5874 snapshot-table compression foundation — P1 shadow write. + // + // Runs AFTER the legacy ingest committed, in the same tick, as its own + // store transaction. The plane is derived from FINAL DB state (messages + // / blocks / plane rows) — never from fuzzy matching — so a round that + // cannot be addressed produces no events and the legacy path carries it + // (R2). Any throw here is caught: the plane NEVER fails the save, and a + // failed plane round self-heals on the next save (the derivation is + // idempotent final-state diffing). Lane sessions are excluded (their + // transcripts are not authoritative — the main session is). + // ----------------------------------------------------------------------- + private snapShadowSync( + input: IngestSessionInput, + delta: { trustedSeq: number; boundaryMsgId: string; boundaryHash: string; trustedTailSeqs: number[] } | undefined, + ): void { + const fl = this.snapPlane; + if (!fl.enabled || !fl.shadowWrite) return; + const sid = input.session.sessionId; + if (this.hasLaneExt(sid)) return; // skip:lane — lane transcripts are not the cohort truth + if (isSnapPlaneDead(sid)) return; // dead (R1) — sticky, silent; status surfaces it + // #6181 B0: a turn that CANNOT be addressed against the plane (a + // non-cohort chained delta — its kernel view is a delta, not the + // world) skips the plane write for THIS save: deriving events from + // the collapsed view would DELETE every row the view does not + // contain (snapViewEvents is a final-state diff). The flag is + // one-shot — consumed by the persist build, not here. + if (input.skipPlaneFlush) return; + const now = Date.now(); + // Stage-0 identity: raw host session id → ledger id (v1 mapping: the + // conversation binding, session_id when there is none). + const sessRaw = + (this.db.prepare("SELECT conversation_id FROM conversations WHERE session_id = ?").get(sid) as { conversation_id: string } | undefined) + ?.conversation_id ?? sid; + const witness = this.snapWitnessProbe(sid); + // #5874 mapEgress: when the save carries the lane's tagged turn view + // (planeMessages), the plane is the DIRECT outbound authority — the + // events are derived from the view itself (kernel evidence + // addressing, no CAS I/O), never from the legacy diff (the legacy + // bytes are the pre-tag transcript and must not overwrite the + // plane's tagged forms). + const view = input.planeMessages; + const tagged = Array.isArray(view) && view.length > 0; + // #6181 B4 (stack on the #5901 head with #6326 durability-before-render): + // an UNTAGGED save against a sid that already holds active plane rows + // can only update the plane when its ledger-id view EXTENDS the + // stage-0 world: stage0Decide writes the l1 envelope and declares the + // new positions, but the ROWS for those positions land at the NEXT + // save — and for a pinned cohort that next save is often an untagged + // one (#6326's ingestRowsNow persists before render). So: + // • view ids ⊇ envelope ids (prefix-identical) → ALLOW: the legacy + // diff materializes the declared rows (ids agree, idempotent); + // • anything else is STALE w.r.t. the pinned world — re-derived + // kernel ids (h_…), a pre-turn transcript, a compaction — and the + // legacy diff could only delete/re-home the pinned rows (the R2 + // doctrine: the tagged plane is addressable from tagged views + // only) → SKIP the plane; the disk record still lands (the + // #6326 durability win survives) and the plane is updated by the + // next TAGGED save or a stage-0 resync (one-round convergence). + // MUST run before the transaction — after it the destructive legacy + // diff is already committed. + if (!tagged && (input.planeMessages ?? []).length === 0) { + const anyRow = (snapRowsAll(this.db, sid) ?? []).some((r) => r.active === 1); + if (anyRow) { + const env = snapL1EnvelopeRead(this.db, sessRaw); + const envIds = + env && env.envelopeJson !== null + ? parseEnvelope(env.envelopeJson).envelope.map((e) => e.id) + : []; + const viewIds = input.messages.map((m) => m.msgId); + const extendsEnvelope = + viewIds.length >= envIds.length && envIds.every((id, i) => viewIds[i] === id); + if (!extendsEnvelope) return; + } + } + try { + const out = this.db.transaction(() => { + const l1 = snapL1Record(this.db, sessRaw, sid, witness, now); + if (l1 === "red") { + // Identity conflict (R3 territory): the raw id maps to a + // different ledger. Hard-refuse this round — zero side + // effects on the plane, legacy carries the round. + throw new Error(`l1 red: sess_raw=${sessRaw} maps to another ledger (R3)`); + } + // Steady-state delta appends take the fast path (no full + // scan); everything else is a final-state diff (seed, edit, + // rewind, compaction, reap — the bounded passes). Tagged + // saves skip the fast path (the view is the source of truth). + const fast = !tagged && delta ? this.snapFastAppendEvents(sid, input) : null; + const events = tagged ? this.snapViewEvents(sid, view as CoreMessage[]) : (fast ?? this.snapDeriveEvents(sid)); + const depth = this.snapLogDepth(sid); + const fold = events.some((e) => e.ev === "compact") || depth >= UnifiedStore.SNAP_FOLD_DEPTH; + const commit = applySnapInTransaction(this.db, sid, events, { fold, now }); + snapGcCoveredRows(this.db, sid); + return { l1, nEvents: events.length, commit }; + }); + // A/B oracle on fold (amortized O(n), off the per-save fast + // path): the folded plane must agree with the legacy view — + // positions and forms. A mismatch kills the plane for this sid + // (R2 as a red door on the read side); the save itself is done + // and untouched. + if (out.commit.folded) this.snapAbCheck(sid, tagged ? (view as CoreMessage[]) : null); + const kind = + out.nEvents === 0 + ? "noop" + : out.commit.folded + ? `sync,fold(n=${out.nEvents})` + : `sync(n=${out.nEvents})`; + loggerLog("info", `[snaplane] sid=${sid} l1=${out.l1} ${kind} tail=${out.commit.tailSeq}`); + } catch (err) { + if (!(err instanceof Error) || err.name !== "SnapPlaneDeadError") markSnapPlaneDead(sid); + // R2/R3 degrade: the round stays on the legacy path. Log the + // decision with the unified vocabulary and stop retrying this + // sid for the process lifetime (a persistent derivation error + // must not cost a transaction per save; a restart re-verifies + // from storage, which is the honest re-arm). + loggerLog("warn", `[snaplane] sid=${sid} decision=red:R2 ${String(err).slice(0, 200)}`); + } + } + + /** Snapshot fold cadence: absorb the log into the ckpt once it holds + * this many entries past the ckpt, or whenever a compact lands (the + * coverage mapping must then be current for the fold view). Keeps the + * log depth bounded (T9). */ + private static readonly SNAP_FOLD_DEPTH = 128; + + /** L1 witness probe: a sampled fingerprint of the FIRST live ids (the + * head of the id space is what survives rounds — the tail churns). */ + private snapWitnessProbe(sid: string): string { + const rows = this.db + .prepare("SELECT msg_id FROM messages WHERE session_id = ? AND retired_at IS NULL ORDER BY seq LIMIT 8") + .all(sid) as { msg_id: string }[]; + return sha256Hex(rows.length ? rows.map((r) => r.msg_id).join("\u0000") : "empty"); + } + + /** Log entries past the last ckpt for one session (fold heuristic). */ + private snapLogDepth(sid: string): number { + const tail = (this.db.prepare("SELECT COALESCE(MAX(seq), 0) AS m FROM snap_log WHERE sid = ?").get(sid) as { m: number }).m; + const ckpt = + (this.db.prepare("SELECT ckpt_seq FROM snap_ckpt WHERE sid = ?").get(sid) as { ckpt_seq: number } | undefined)?.ckpt_seq ?? 0; + return Math.max(0, tail - ckpt); + } + + /** Fast path for steady-state delta appends (no full scan): the delta + * guard already proved every incoming identity is NEW, so the plane + * work is exactly "append n rows at tail+1.." — valid when the plane + * tail equals the pre-append legacy max. Anything else (plane never + * seeded, plane behind the legacy tail, empty delta) returns null and + * the caller takes the final-state diff. */ + private snapFastAppendEvents(sid: string, input: IngestSessionInput): SnapEvent[] | null { + const n = input.messages.length; + const maxLive = (this.db.prepare("SELECT COALESCE(MAX(seq), -1) AS m FROM messages WHERE session_id = ? AND retired_at IS NULL").get(sid) as { m: number }).m; + const planeTail = (this.db.prepare("SELECT COALESCE(MAX(pos), -1) AS m FROM snap_rows WHERE sid = ?").get(sid) as { m: number }).m; + if (planeTail < 0) return null; // plane not seeded — the diff seeds it + if (n === 0) return maxLive === planeTail ? [] : null; + if (maxLive - n + 1 !== planeTail + 1) return null; + // The incoming rows are exactly planeTail+1 .. maxLive; read their + // hashes back from the committed messages table (the canonical form + // the plane stores — invariant v). + const seqs: number[] = []; + for (let i = 1; i <= n; i++) seqs.push(planeTail + i); + const rows = this.db + .prepare( + `SELECT m.seq, m.msg_id, m.content_hash, c.byte_len AS clen FROM messages m JOIN content c ON c.hash = m.content_hash WHERE m.session_id = ? AND m.seq IN (${seqs.map(() => "?").join(",")}) ORDER BY m.seq`, + ) + .all(sid, ...seqs) as { seq: number; msg_id: string; content_hash: string; clen: number }[]; + if (rows.length !== n) return null; // defensive — the guard proved them new + const content = this.snapContentByHash(rows.map((r) => r.content_hash)); + const events: SnapEvent[] = []; + for (const r of rows) { + const body = content.get(r.content_hash); + if (body === undefined) return null; // defensive — hash mismatch is a red door + events.push({ + ev: "append", + pos: r.seq, + coreIds: [r.msg_id], + preJson: null, + postJson: body, + tokensEst: Math.ceil(r.clen / 4), + }); + } + return events; + } + + /** Final-state diff: derive the plane events that make the plane agree + * with the committed legacy state. Bounded by the session's live rows + * and blocks (one indexed read each) — this is the reconciliation pass + * the fast path exists to avoid in steady state. */ + private snapDeriveEvents(sid: string): SnapEvent[] { + const live = this.db + .prepare( + "SELECT m.seq, m.msg_id, m.content_hash, c.byte_len AS clen FROM messages m JOIN content c ON c.hash = m.content_hash WHERE m.session_id = ? AND m.retired_at IS NULL ORDER BY m.seq", + ) + .all(sid) as { seq: number; msg_id: string; content_hash: string; clen: number }[]; + const plane = this.db + .prepare("SELECT pos, core_ids, post_json, active, cover_ref FROM snap_rows WHERE sid = ? ORDER BY pos") + .all(sid) as { pos: number; core_ids: string; post_json: string | null; active: number; cover_ref: string | null }[]; + const covers = this.db + .prepare( + "SELECT block_id, start_seq, end_seq FROM blocks WHERE session_id = ? AND active = 1 AND start_seq >= 0 AND end_seq >= start_seq ORDER BY start_seq", + ) + .all(sid) as { block_id: string; start_seq: number; end_seq: number }[]; + const planeByPos = new Map(plane.map((r) => [r.pos, r])); + const livePos = new Set(live.map((r) => r.seq)); + const content = this.snapContentByHash(live.map((r) => r.content_hash)); + const inCover = (pos: number) => covers.some((c) => pos >= c.start_seq && pos <= c.end_seq); + const events: SnapEvent[] = []; + // 1) casts / updates / revivals, in pos order (deterministic seqs). + for (const m of live) { + const body = content.get(m.content_hash); + if (body === undefined) throw new Error(`[snaplane] sid=${sid} pos=${m.seq}: content hash ${m.content_hash.slice(0, 12)}… missing from CAS`); + const coreIds = [m.msg_id]; + const p = planeByPos.get(m.seq); + const tokensEst = Math.ceil(m.clen / 4); + if (!p) { + events.push({ ev: "cast", pos: m.seq, coreIds, preJson: null, postJson: body, tokensEst }); + continue; + } + if (p.active === 0) { + // Revival (retired row back in the transcript, or a + // compacted row the host re-presents): re-activate in place, + // pre_json becomes the old post (two-form retention). A + // still-covered position is re-flipped by the compact + // emitted below — the batch lands in the covered state. + events.push({ ev: "update", pos: m.seq, postJson: body, coreIds, tokensEst }); + continue; + } + if (p.post_json !== body || p.core_ids !== JSON.stringify(coreIds)) { + events.push({ ev: "update", pos: m.seq, postJson: body, coreIds, tokensEst }); + } + } + // 2) deletions: ACTIVE rows whose position the host transcript no + // longer presents and no block covers (host-side removal has no + // covering block; covered rows go to the compact, which retains + // both forms for forensics). + for (const r of plane) { + if (r.active === 1 && !livePos.has(r.pos) && !inCover(r.pos)) events.push({ ev: "delete", pos: r.pos }); + } + // 3) compactions LAST (they flip whatever 1) just cast): an active + // block covers a ledger interval — every row inside becomes + // covered (active=0, cover_ref set, both forms kept). Emitted + // only when the batch changes state (an active row inside, or a + // cast landing inside), so steady-state saves do not re-log a + // no-op compact every round. + for (const c of covers) { + let flip = false; + for (const r of plane) if (r.active === 1 && r.pos >= c.start_seq && r.pos <= c.end_seq) { flip = true; break; } + if (!flip) for (const m of live) if (m.seq >= c.start_seq && m.seq <= c.end_seq && !planeByPos.has(m.seq)) { flip = true; break; } + if (flip) events.push({ ev: "compact", blockId: c.block_id, posRange: [c.start_seq, c.end_seq] }); + } + return events; + } + + // ----------------------------------------------------------------------- + // #5874 mapEgress write side (direct authority): plane events from the + // lane's tagged processTurn output. The plane persists the kernel's OWN + // tagged bytes (invariant v), so the egress that reads them back is + // byte-identical to the legacy transform BY CONSTRUCTION — the legacy + // diff above stays for saves that carry no turn view (defensive + // fallback, and the lane-less seed path). + // + // Addressing (proposal §4.2 — kernel evidence only, no fuzzy matching): + // a view entry resolves to its ledger row by MESSAGE ID (steady state: + // the view id IS the client raw id — the kernel only remints a + // content-hash id on the rare "new byte-identical instance after a + // fold" edge, kernel instance-reid), with a position-independent + // IDENTITY fallback for that edge (stored bytes minus tags / id / + // wireIndex / ledgerSeq — the same identity the legacy intern computed + // from the RAW bytes, so the two forms agree). An entry that resolves + // to nothing throws (R2): the caller latches the plane dead for this + // sid and the legacy path carries the round. + // + // Event order: casts/updates in view order, deletes, compacts LAST (a + // compact flips whatever this batch just cast — a fold covers this + // turn's messages the moment it lands). The pinned opening wire message + // (#2733/#3642) is carved out of compact ranges: the kernel serves it + // even when covered, so its row stays active. + // ----------------------------------------------------------------------- + /** Canonical stored bytes for a view entry: same serialize as the legacy + * intern (#4390/#4313 — ts out of the payload) but over the TAGGED text + * (the tag is part of the plane's post form; the egress serves it as-is). + * Local copy of ingest.ts's serializeForStore (circular import). */ + private planeSerializeForStore(m: CoreMessage): string { + if ("ts" in m) { + const copy: Record = { ...m }; + delete copy["ts"]; + return JSON.stringify(copy); + } + return JSON.stringify(m); + } + + /** Tag-stripped identity of a tagged view entry — the kernel prepends + * `mNNNNN\n` to the text, which is NOT part of the + * message content identity (the legacy rows' identity column was + * computed from the untagged transcript). Identity also strips + * id/wireIndex/ledgerSeq, so a reminted id with identical content + * still matches its ledger row. */ + private planeViewIdentity(m: CoreMessage): string { + try { + const stored = splitMediaContent(this.planeSerializeForStore(m)).content; + const parsed = JSON.parse(stored) as { text?: unknown }; + if (typeof parsed.text === "string") parsed.text = parsed.text.replace(/^\n]*>m\d+<\/acp>\n/, ""); + return identityOfStoredContent(JSON.stringify(parsed)); + } catch { + return ""; // unmatchable — the caller treats it as an id-path miss + } + } + + private snapViewEvents(sid: string, view: CoreMessage[]): SnapEvent[] { + const live = this.db + .prepare( + "SELECT m.seq, m.msg_id, m.identity, m.content_hash, c.byte_len AS clen FROM messages m JOIN content c ON c.hash = m.content_hash WHERE m.session_id = ? AND m.retired_at IS NULL ORDER BY m.seq", + ) + .all(sid) as { seq: number; msg_id: string; identity: string; content_hash: string; clen: number }[]; + const plane = this.db + .prepare("SELECT pos, core_ids, post_json, active, cover_ref FROM snap_rows WHERE sid = ? ORDER BY pos") + .all(sid) as { pos: number; core_ids: string; post_json: string | null; active: number; cover_ref: string | null }[]; + const blocks = this.db + .prepare("SELECT block_id, active, start_seq, end_seq, payload_json FROM blocks WHERE session_id = ? ORDER BY ord") + .all(sid) as { block_id: string; active: number; start_seq: number; end_seq: number; payload_json: string | null }[]; + const blockIds = new Set(blocks.map((b) => b.block_id)); + const activeBlocks = blocks.filter((b) => b.active === 1); + const parseIds = (json: string): string[] => { + try { + const a = JSON.parse(json); + return Array.isArray(a) ? (a as string[]) : []; + } catch { + return []; + } + }; + const idsByPos = new Map(); + for (const r of plane) idsByPos.set(r.pos, parseIds(r.core_ids)); + const planeByPos = new Map(plane.map((r) => [r.pos, r])); + let minAnchorPos = 0; + for (const r of plane) if (r.pos < minAnchorPos) minAnchorPos = r.pos; + // First unconsumed candidate wins: view order == transcript order == + // live seq order (the kernel never reorders regular rows), so + // sequential consumption keeps reminted / duplicate-content entries + // mapped to the right instance. + const liveById = new Map(); + const liveByIdentity = new Map(); + live.forEach((r, i) => { + for (const map of [liveById, liveByIdentity] as const) { + const key = map === liveById ? r.msg_id : r.identity; + if (!key) continue; // pre-v8 rows carry no identity — id path only + const a = map.get(key); + if (a) a.push(i); + else map.set(key, [i]); + } + }); + const consumed = new Array(live.length).fill(false); + const pickUnconsumed = (list: number[] | undefined): number | undefined => (list ? (list.find((i) => !consumed[i]) ?? undefined) : undefined); + // Plane-first addressing (remint stability, #3752): the plane row is + // the STABLE address. A reminted client round re-mints only the + // LEGACY ids and re-appends the legacy table under new seqs (the old + // rows are sweep-retired) — the view still rides the plane's ids (the + // D lane builds the kernel input from the transcript rows), so the + // right mapping target is the EXISTING plane row (alias id, then + // tag-stripped identity), never the fresh legacy rows. Mapping onto + // the fresh rows re-casts the whole prefix at new seqs every upgrade + // and orphans the stable rows (host-removed + GC → pin lost). + // Coverage evidence computed BEFORE the plane indexes: a fold stamps + // coverage in THIS save's pass 3, so at index-construction time the + // covered rows are still ACTIVE — the indexes must exclude them + // explicitly (an active-only filter alone cannot see the fold that + // happens a few lines later in the same save). + // Id → pos (first occurrence in pos order) for coverage addressing: + // every alias in every plane row, then live rows without one. + const posById = new Map(); + for (const r of plane) for (const id of idsByPos.get(r.pos) ?? []) if (!posById.has(id)) posById.set(id, r.pos); + for (const l of live) if (!posById.has(l.msg_id)) posById.set(l.msg_id, l.seq); + // Covered PLANE pos (mirrors the kernel's coverage evidence): effIds + // → pos (aliases survive remint — the plane row keeps every id the + // content ever had) ∪ the block's durable table interval, TRANSLATED + // through the fold-time rows: the interval is LEGACY-seq space (fold + // time), and after a remint round the legacy table re-appends under + // new seqs while the plane keeps its own stable pos — so the raw + // interval must never be read as plane pos directly. + const coveredPos = new Set(); + const coveredLegacySeqs = new Set(); // legacy-seq space — materialization + const blockEffIds = (b: { payload_json: string | null }): string[] => { + if (!b.payload_json) return []; + try { + const p = JSON.parse(b.payload_json) as { effectiveMessageIds?: unknown }; + return Array.isArray(p.effectiveMessageIds) ? (p.effectiveMessageIds as string[]) : []; + } catch { + return []; + } + }; + { + const intervalSeqs = new Set(); + for (const b of activeBlocks) if (b.start_seq >= 0 && b.end_seq >= b.start_seq) for (let s = b.start_seq; s <= b.end_seq; s++) intervalSeqs.add(s); + if (intervalSeqs.size > 0) { + // Fold-time rows — RETIRED ones included: the sweep retires the + // reminted originals but they stay queryable. One bounded + // query per interval. + for (const b of activeBlocks) { + if (b.start_seq < 0 || b.end_seq < b.start_seq) continue; + const rows = this.db + .prepare("SELECT seq, msg_id FROM messages WHERE session_id = ? AND seq BETWEEN ? AND ?") + .all(sid, b.start_seq, b.end_seq) as { seq: number; msg_id: string }[]; + for (const row of rows) { + const p = posById.get(row.msg_id); + if (p !== undefined && p >= 0) coveredPos.add(p); + } + } + for (const s of intervalSeqs) coveredLegacySeqs.add(s); + } + } + for (const b of activeBlocks) { + for (const id of blockEffIds(b)) { + const p = posById.get(id); + if (p !== undefined && p >= 0) coveredPos.add(p); + for (const li of liveById.get(id) ?? []) coveredLegacySeqs.add(live[li]!.seq); + } + } + + const planeIdx = plane.map((r, i) => (r.pos >= 0 ? i : -1)).filter((i) => i >= 0); + const planeById = new Map(); + const planeByIdentity = new Map(); + const rowIdentity = new Map(); // plane idx → tag-stripped identity + let maxUpdateViewIdx = -1; + const reentryCandidates: number[] = []; + for (const i of planeIdx) { + const r = plane[i]!; + // Same doctrine as the identity index below: the view never + // presents a covered row, so an id equal to a covered row's id is + // a REMINT collision (the kernel re-mints a fresh instance onto a + // folded row's id after the fold) — it must be cast as a fresh + // row at its own ledger seq, never update the covered row in + // place (that re-activates the folded slot and parks the new + // content at the original's position: the golden's repeated + // "running a check" row was re-homed onto pos 2 that way, and the + // egress served it as a stray mid-history message). + if (r.active !== 1 || coveredPos.has(r.pos)) continue; + for (const id of idsByPos.get(r.pos) ?? []) { + const a = planeById.get(id); + if (a) a.push(i); + else planeById.set(id, [i]); + } + const pj = r.post_json; + if (typeof pj !== "string" || !pj) continue; + let id2 = ""; + try { + const o = JSON.parse(pj) as { id?: unknown; role?: unknown; text?: unknown; wireIndex?: unknown; ledgerSeq?: unknown }; + if (o && typeof o === "object" && typeof o.role === "string" && typeof o.id === "string") { + const t = o.text; + if (typeof t === "string") o.text = t.replace(/^\n]*>m\d+<\/acp>\n/, ""); + delete (o as Record).ts; + id2 = identityOfStoredContent(JSON.stringify(o)); + } + } catch { + id2 = ""; + } + if (!id2) continue; + rowIdentity.set(i, id2); + // Identity is a WEAK address: it collides across duplicate content + // (e.g. a model that returns the identical reply N times — every + // instance shares one content-hash base, the kernel re-mints each + // with a _k suffix). The view NEVER presents a covered message + // (it is folded into its block's summary), so a covered row can + // only be an identity match for a NEW instance of that content — + // which must be cast as a fresh row, never update the covered one + // in place (that stole the covered row's pos and re-ordered the + // egress: the new reply parked at a folded row's ledger slot). + // Only ACTIVE rows are identity-match candidates; the remint case + // (identity-seed upgrade) re-mints live messages, which are active. + if (r.active !== 1 || coveredPos.has(r.pos)) continue; + const a = planeByIdentity.get(id2); + if (a) a.push(i); + else planeByIdentity.set(id2, [i]); + } + const consumedPlane = new Array(plane.length).fill(false); + const pickUnconsumedPlane = (list: number[] | undefined): number | undefined => (list ? (list.find((i) => !consumedPlane[i]) ?? undefined) : undefined); + const viewIdentity = (m: CoreMessage): string => this.planeViewIdentity(m); + const events: SnapEvent[] = []; + const mappedPos = new Set(); // ledger pos claimed by a view entry this round + const materializedPos = new Set(); // covered rows cast this batch (pre-compact) — the flip check must count them + const seqAt = new Array(view.length).fill(undefined); // view idx → mapped ledger pos (undefined for anchor entries) + const anchorKept = new Set(); // active block ids with a render in THIS view + // Pass 1: view entries → ledger rows (casts/updates) + anchor rows. + for (let vi = 0; vi < view.length; vi++) { + const m = view[vi]!; + const stored = splitMediaContent(this.planeSerializeForStore(m)).content; + const tokensEst = Math.ceil(Buffer.byteLength(stored, "utf8") / 4); + if (typeof m.id === "string" && m.id.startsWith("acp_summary_") && blockIds.has(m.id.slice("acp_summary_".length))) { + // A rendered summary whose block is KNOWN to the plane: the + // carrier is the ANCHOR row (negative pos — no ledger + // position; the egress re-inserts it at its pair-safe index), + // never the ledger row (a host-resent copy of it is a stale + // rendered-summary copy that the kernel DROPS, so pass 2 + // retires it). Consumed blocks render no anchor in the view + // — their stale copies are retired by the generic pass-2 + // rule (no mapping, not covered, not kept). + const bid = m.id.slice("acp_summary_".length); + if (!activeBlocks.some((b) => b.block_id === bid)) continue; + anchorKept.add(bid); + let anchorPos: number | null = null; + for (const r of plane) if (r.pos < 0 && idsByPos.get(r.pos)!.includes(m.id)) { anchorPos = r.pos; break; } + if (anchorPos === null) { + const pos = minAnchorPos - 1; // newest anchor = most negative + minAnchorPos = pos; + events.push({ ev: "cast", pos, coreIds: [m.id], preJson: null, postJson: stored, tokensEst }); + } else { + const r = planeByPos.get(anchorPos)!; + if (r.active === 0 || r.post_json !== stored) { + events.push({ ev: "update", pos: anchorPos, postJson: stored, coreIds: [m.id], tokensEst }); + } + } + continue; + } + // Regular entry → PLANE row first (stable address), then a fresh + // legacy row for genuinely new content. + const vid = viewIdentity(m); + let pi = typeof m.id === "string" ? pickUnconsumedPlane(planeById.get(m.id)) : undefined; + if (pi === undefined) pi = pickUnconsumedPlane(planeByIdentity.get(vid)); + if (pi !== undefined) { + consumedPlane[pi] = true; + const rp = plane[pi]!; + const pos = rp.pos; + mappedPos.add(pos); + seqAt[vi] = pos; + const ids = idsByPos.get(pos) ?? []; + const merged = typeof m.id === "string" ? [m.id, ...ids.filter((id) => id !== m.id)] : [...ids]; + // Bridge the reminted legacy id (same content, fresh row): + // keep the alias chain whole so the egress's alias coverage + // and the next round's id addressing both resolve. + for (const li of liveByIdentity.get(vid) ?? []) { + const l = live[li]!; + if (!consumed[li] && !merged.includes(l.msg_id)) merged.push(l.msg_id); + break; + } + if (rp.active === 0 || rp.post_json !== stored || rp.core_ids !== JSON.stringify(merged)) { + events.push({ ev: "update", pos, postJson: stored, coreIds: merged, tokensEst }); + } + continue; + } + // New content → fresh row from the legacy ledger: by id, then by + // tag-stripped identity (pre-v8 rows carry no identity — id only). + let li = pickUnconsumed(liveById.get(m.id)); + if (li === undefined) li = pickUnconsumed(liveByIdentity.get(vid)); + if (li === undefined) { + throw new Error(`[snaplane] sid=${sid} view[${vi}] id=${String(m.id).slice(0, 40)}: unaddressable — no plane row or live ledger row (R2)`); + } + consumed[li] = true; + const l = live[li]!; + mappedPos.add(l.seq); + seqAt[vi] = l.seq; + const coreIds = new Set([m.id]); + if (l.msg_id !== m.id) coreIds.add(l.msg_id); + const p = planeByPos.get(l.seq); + if (p) { + const merged = [...coreIds, ...idsByPos.get(l.seq)!.filter((id) => !coreIds.has(id))]; + if (p.active === 0 || p.post_json !== stored || p.core_ids !== JSON.stringify(merged)) { + events.push({ ev: "update", pos: l.seq, postJson: stored, coreIds: merged, tokensEst }); + } + maxUpdateViewIdx = Math.max(maxUpdateViewIdx, vi); + } else { + // New content → fresh row from the legacy ledger. Normally a + // TAIL append (pos = fresh legacy seq). If the kernel instead + // placed it BEFORE existing content (mid-history re-entry — + // e.g. the user edited an old row after compression and the + // re-sent transcript carries the edit in mid position), the + // egress — which serves plane rows in pos order — would park + // it at the end and drift from the kernel's order (golden + // drift on every body after the edit). Re-home it in a + // post-pass below (mappedPos must be final first). + const castAt = { ev: "cast" as const, pos: l.seq, coreIds: [...coreIds], preJson: null, postJson: stored, tokensEst }; + events.push(castAt); + maxUpdateViewIdx = Math.max(maxUpdateViewIdx, vi); + reentryCandidates.push(vi); + } + } + // Post-pass: re-home mid-history re-entries (see above). The old row + // this entry replaced is the single non-anchor plane row strictly + // between the mapped neighbors that no view entry claims (covered or + // active — the edit case finds it covered). Cast there (the upsert + // re-activates it, clears cover_ref, keeps its ids as aliases) so the + // plane's pos order reproduces the kernel's output order exactly; + // mappedPos then protects the pos from pass 2's deletes and pass 3's + // compacts. If no such row exists, fall back to the first FREE slot + // between the neighbors (the old row was physically GC'd), else keep + // the fresh tail pos. A cast whose content identity matches a COVERED + // row is the duplicate-reply shape (#3812): a NEW instance of content + // a block folded — the kernel serves the fresh copy as the newest + // message, it must stay a tail cast, never occupy a folded row's + // slot. (Pass 1 runs before pass 2/3 — the pre-flush plane row is + // ground truth, the same invariant pass 3 uses.) + if (reentryCandidates.length > 0) { + // Identity of EVERY non-anchor plane row (active and covered). + // A cast entry whose identity is already on the plane matched an + // existing row — or is a new instance of folded content (the + // #3812 duplicate-reply shape, which may collide with an ACTIVE + // row too: the first instance is still live when the second + // arrives). Either way the entry belongs at the live tail — the + // kernel serves it there — and must never be re-homed into an + // old row's slot (that parks a duplicate at the original's + // position and serves the content twice). + const planeSigs = new Set(); + for (const i of planeIdx) { + const pr = plane[i]!; + if (pr.pos < 0) continue; + const rid = rowIdentity.get(i); + if (rid) planeSigs.add(rid); + } + for (const vi of reentryCandidates) { + const m = view[vi]!; + const vid = viewIdentity(m); + if (vid && planeSigs.has(vid)) continue; // duplicate-instance / stale-content shape + let prevVi = -1; + for (let k = vi - 1; k >= 0; k--) { + if (seqAt[k] !== undefined) { + prevVi = k; + break; + } + } + if (prevVi < 0) continue; // no earlier mapped entry → true tail append + const lo = (seqAt[prevVi] ?? -1) + 1; + const laterPos: number[] = []; + for (let k = vi + 1; k < view.length; k++) { + const s = seqAt[k]; + if (s !== undefined) laterPos.push(s); + } + const hi = laterPos.length ? Math.min(...laterPos) : Infinity; + // locate this entry's cast event: the cast whose coreIds lead + // with the view id (re-homed casts keep the new id first — + // old ids are appended, never prepended). + const castEv = events.find((e) => e.ev === "cast" && (e as { coreIds?: string[] }).coreIds?.[0] === (typeof m.id === "string" ? m.id : "")); + if (castEv === undefined || castEv.ev !== "cast") continue; + // Only an INVERSION is a re-entry: the fresh tail pos sorts + // after a row the view places LATER. A true tail append + // (no later-mapped row, or all later rows at higher pos) + // keeps its fresh pos untouched. + if (laterPos.length === 0 || Math.min(...laterPos) > castEv.pos) continue; + if (lo >= hi) continue; + // The REPLACED row is identified by its retired legacy row: + // the client's re-sent transcript still carries every + // unchanged (covered) row, so their legacy rows are alive; + // only the edited row's old content is absent from the + // re-send and was swept (retired). An unmapped plane row in + // the window whose legacy row is gone is the unique edit + // target. Unmapped rows with a LIVE legacy row are ordinary + // covered siblings — leave them alone. + const liveSeqs = new Set(live.map((r) => r.seq)); + const candidates = plane.filter((pr) => pr.pos >= lo && pr.pos < hi && !mappedPos.has(pr.pos) && !liveSeqs.has(pr.pos)); + let target: number | undefined; + if (candidates.length === 1) { + target = candidates[0]!.pos; + const oldIds = idsByPos.get(target) ?? []; + castEv.coreIds = [castEv.coreIds[0]!, ...oldIds.filter((id) => !castEv.coreIds.includes(id))]; + mappedPos.add(target); + } else if (candidates.length === 0) { + const taken = new Set([...mappedPos, ...(events.filter((e) => e.ev === "cast").map((e) => e.pos)), ...plane.map((r) => r.pos), ...live.map((r) => r.seq)]); + for (let p2 = lo; p2 < hi; p2++) { + if (!taken.has(p2)) { + target = p2; + break; + } + } + } + if (target !== undefined) castEv.pos = target; + } + } + // Pass 2: deletions — an active row the view no longer presents and + // no block covers (host-side removal, or a stale rendered-summary + // copy). Covered rows stay: both forms retained (forensics + the + // egress's pair-safe walk needs the covered rows present). + for (const r of plane) { + if (r.active !== 1) continue; + if (r.pos < 0) { + const ids = idsByPos.get(r.pos) ?? []; + if (ids.some((id) => id.startsWith("acp_summary_")) && ids.every((id) => anchorKept.has(id.slice("acp_summary_".length)))) continue; + } + if (mappedPos.has(r.pos) || coveredPos.has(r.pos)) continue; + events.push({ ev: "delete", pos: r.pos }); + } + // Materialize covered rows that the plane has never recorded + // (migration seed / first tagged save after a block already exists): + // the egress's pair-safe walk needs the covered rows PRESENT to place + // the anchor at the right position, so the block's covered range is + // recorded once (cast from the legacy CAS bytes — they are covered, + // never served, so their untagged form is fine), then flipped covered + // by the compact below. Bounded to the block's range, one-time per + // row. Steady-state folds already have the rows (cast while active), + // so this is a no-op there. + { + const liveBySeq = new Map(live.map((r) => [r.seq, r])); + const missing: { seq: number; msg_id: string; content_hash: string; clen: number }[] = []; + // Legacy-seq space: materialization is a migration-seed concern + // (the plane did not exist when the block was folded — no remint + // can have happened yet, so plane pos == legacy seq there). In + // steady state the covered rows were cast while active and are + // already in the plane, so this is a no-op. + for (const s of coveredLegacySeqs) { + const l = liveBySeq.get(s); + if (!l || planeByPos.has(s)) continue; + if (mappedPos.has(s)) continue; + let hasAlias = false; + for (const r of plane) { + if ((idsByPos.get(r.pos) ?? []).includes(l.msg_id)) { hasAlias = true; break; } + } + if (hasAlias) continue; // the row exists under another pos + missing.push(l); + } + if (missing.length > 0) { + const content = this.snapContentByHash(missing.map((r) => r.content_hash)); + for (const r of missing) { + const body = content.get(r.content_hash); + if (body === undefined) { + throw new Error(`[snaplane] sid=${sid} pos=${r.seq}: content hash ${r.content_hash.slice(0, 12)}… missing from CAS (covered materialization)`); + } + events.push({ ev: "cast", pos: r.seq, coreIds: [r.msg_id], preJson: null, postJson: body, tokensEst: Math.ceil(r.clen / 4) }); + materializedPos.add(r.seq); + } + } + } + // Pass 3: compactions LAST (they flip whatever pass 1 just cast), + // with the protected spans carved out of the flip range: the view + // pin (a), the transcript pin (b), and every row the view mapped + // (mapped rows stay active — the egress serves them and the A/B + // expects them active; coverage suppression at egress is alias-based, + // so the flip only carries bookkeeping for rows the view dropped). + for (const b of activeBlocks) { + const effIds = blockEffIds(b); + const perPos: number[] = []; + for (const id of effIds) { + const p = posById.get(id); + if (p !== undefined && p >= 0) perPos.push(p); + } + // The table interval (legacy-seq space) translated to plane pos + // through the fold-time rows — same translation as coveredPos. + if (b.start_seq >= 0 && b.end_seq >= b.start_seq) { + const rows = this.db + .prepare("SELECT msg_id FROM messages WHERE session_id = ? AND seq BETWEEN ? AND ?") + .all(sid, b.start_seq, b.end_seq) as { msg_id: string }[]; + for (const row of rows) { + const p = posById.get(row.msg_id); + if (p !== undefined && p >= 0) perPos.push(p); + } + } + if (perPos.length === 0) continue; // no coverage evidence resolvable to positions + const lo = Math.min(...perPos); + const hi = Math.max(...perPos); + // Protected points: both pin spans + every mapped row (a mapped + // row inside the min/max span but outside the actual coverage — + // a span gap — must not be flipped). + // Only MAPPED positions are protected. The pin spans are NOT: + // the kernel pins the first user only when that row is in the + // kernel's INPUT (rows-sourced views suppress covered rows, so a + // covered pin row never reaches the kernel and is absent from the + // output). A pin row in the kernel output is a MAPPED view row — + // already protected here; a covered pin row not in the output + // must compact like any other covered row. Keeping pin rows + // active made the plane carry rows the kernel never serves (out2 + // pin divergence). + const prot = new Set(); + for (const p of mappedPos) prot.add(p); + const ranges: [number, number][] = []; + let cur = lo; + for (let p = lo; p <= hi; p++) { + if (prot.has(p)) { + if (cur <= p - 1) ranges.push([cur, p - 1]); + cur = p + 1; + } + } + if (cur <= hi) ranges.push([cur, hi]); + for (const [rl, rh] of ranges) { + let flip = false; + for (const r of plane) if (r.active === 1 && r.pos >= rl && r.pos <= rh) { flip = true; break; } + if (!flip) for (const p of mappedPos) if (p >= rl && p <= rh && !planeByPos.has(p)) { flip = true; break; } + if (!flip) for (const p of materializedPos) if (p >= rl && p <= rh) { flip = true; break; } + if (flip) events.push({ ev: "compact", blockId: b.block_id, posRange: [rl, rh] }); + } + } + return events; + } + + /** Batch CAS read: hash → stored bytes (string). The plane stores the + * canonical core shape — exactly what the messages table stores. */ + private snapContentByHash(hashes: string[]): Map { + const out = new Map(); + const uniq = [...new Set(hashes)]; + const per = SEQ_SPAN_QUERY_BATCH; + for (let i = 0; i < uniq.length; i += per) { + const chunk = uniq.slice(i, i + per); + const rows = this.db + .prepare(`SELECT hash, bytes FROM content WHERE hash IN (${chunk.map(() => "?").join(",")})`) + .all(...chunk) as { hash: string; bytes: Buffer | Uint8Array }[]; + for (const r of rows) out.set(r.hash, Buffer.from(r.bytes).toString("utf8")); + } + return out; + } + + /** A/B oracle (P1): the folded plane must agree with the legacy view. + * Compared: every live ledger position NOT covered by an active block + * ↔ an ACTIVE plane row, and the plane row's post form ↔ the stored + * message content. Covered positions are legitimately inactive in the + * plane (the block summary is their carrier) and are excluded here — + * they are accounted for in the ckpt coverage map (T7). Any mismatch ⇒ + * the plane is dead for this sid (R2 as a red door on the read side). + * Called only at fold time — amortized, off the per-save fast path. + * + * #5874 tagged planes (view present): the post form is the TAGGED + * bytes, so the byte A/B against the legacy CAS form cannot hold by + * construction — the check becomes STRUCTURAL: active non-anchor plane + * pos set == live pos − covered, and one active anchor row per active + * block that rendered this turn (bytes equality is guaranteed by the + * write side copying the view itself). */ + private snapAbCheck(sid: string, view: CoreMessage[] | null): void { + if (view !== null) { + this.snapAbCheckTagged(sid, view); + return; + } + try { + const view = readSnapView(this.db, sid); + const covers = this.db + .prepare("SELECT start_seq, end_seq FROM blocks WHERE session_id = ? AND active = 1 AND start_seq >= 0 AND end_seq >= start_seq") + .all(sid) as { start_seq: number; end_seq: number }[]; + const covered = (pos: number) => covers.some((c) => pos >= c.start_seq && pos <= c.end_seq); + const live = this.db + .prepare("SELECT seq, content_hash FROM messages WHERE session_id = ? AND retired_at IS NULL ORDER BY seq") + .all(sid) as { seq: number; content_hash: string }[]; + const uncovered = live.filter((r) => !covered(r.seq)); + const content = this.snapContentByHash(uncovered.map((r) => r.content_hash)); + const activeByPos = new Map((view?.rows ?? []).filter((r) => r.active === 1).map((r) => [r.pos, r])); + if (uncovered.length !== activeByPos.size) { + markSnapPlaneDead(sid); + loggerLog("warn", `[snaplane] sid=${sid} decision=red:R2 A/B count mismatch (legacy uncovered ${uncovered.length} vs plane active ${activeByPos.size})`); + return; + } + for (const m of uncovered) { + const row = activeByPos.get(m.seq); + if (!row) { + markSnapPlaneDead(sid); + loggerLog("warn", `[snaplane] sid=${sid} decision=red:R2 A/B: live pos ${m.seq} has no active plane row`); + return; + } + const body = content.get(m.content_hash); + if (row.postJson !== body) { + markSnapPlaneDead(sid); + loggerLog("warn", `[snaplane] sid=${sid} decision=red:R2 A/B: pos ${m.seq} form mismatch`); + return; + } + } + } catch (err) { + if (err instanceof Error && err.name === "SnapPlaneDeadError") { + // R1 already latched (sticky) — nothing more to do. + return; + } + markSnapPlaneDead(sid); + loggerLog("warn", `[snaplane] sid=${sid} decision=red:R2 A/B check failed: ${String(err).slice(0, 200)}`); + } + } + + /** A/B for tagged planes (#5874): the plane must agree with the + * KERNEL VIEW, not with the legacy table. Invariant v makes the view + * (the kernel's tagged processTurn output) the egress source, so the + * correct oracle is "active non-anchor plane contents == the view's + * regular entries' contents" (identity multisets, remint-proof — + * id/wireIndex/ledgerSeq/tag are stripped from both sides). The + * table-derived expectation was the wrong oracle: the table and the + * view legitimately drift (reply-lag rows the client re-sent but the + * rows-sourced D view gap-consumed; reminted re-appends), and a + * red on that drift latched the plane dead, sending egress to legacy. + * Runs fold-time only; a mismatch ⇒ plane dead for this sid (R2 as a + * red door on the read side). */ + private snapAbCheckTagged(sid: string, view: CoreMessage[]): void { + try { + const plane = this.db + .prepare("SELECT pos, core_ids, post_json, active FROM snap_rows WHERE sid = ? ORDER BY pos") + .all(sid) as { pos: number; core_ids: string; post_json: string | null; active: number }[]; + const idsOf = (json: string): string[] => { + try { + const a = JSON.parse(json); + return Array.isArray(a) ? (a as string[]) : []; + } catch { + return []; + } + }; + const rowIdentity = (pj: string | null): string => { + if (typeof pj !== "string" || !pj) return ""; + try { + const o = JSON.parse(pj) as { id?: unknown; text?: unknown }; + if (!o || typeof o !== "object" || typeof o.id !== "string") return ""; + const t = o.text; + if (typeof t === "string") o.text = t.replace(/^\n]*>m\d+<\/acp>\n/, ""); + return identityOfStoredContent(JSON.stringify(o)); + } catch { + return ""; + } + }; + const expected = new Map(); + const actual = new Map(); + const add = (mm: Map, id: string) => { + if (id) mm.set(id, (mm.get(id) ?? 0) + 1); + }; + for (const m of view) { + if (typeof m.id === "string" && m.id.startsWith("acp_summary_")) continue; // anchor — its own row + add(expected, this.planeViewIdentity(m)); + } + for (const r of plane) { + if (r.active !== 1 || r.pos < 0) continue; + if (idsOf(r.core_ids).some((id) => id.startsWith("acp_summary_"))) continue; + add(actual, rowIdentity(r.post_json)); + } + const eq = expected.size === actual.size && [...expected.entries()].every(([k, v]) => actual.get(k) === v); + if (!eq) { + markSnapPlaneDead(sid); + loggerLog( + "warn", + `[snaplane] sid=${sid} decision=red:R2 A/B(tagged) view-vs-plane identity multiset mismatch (view ${[...expected.values()].reduce((a, b) => a + b, 0)} rows vs plane active ${[...actual.values()].reduce((a, b) => a + b, 0)})`, + ); + } + } catch (err) { + if (err instanceof Error && err.name === "SnapPlaneDeadError") return; + markSnapPlaneDead(sid); + loggerLog("warn", `[snaplane] sid=${sid} decision=red:R2 A/B(tagged) check failed: ${String(err).slice(0, 200)}`); + } + } + + + /** #5874: the status payload for the /__bili/status wire (flag state + + * plane counters + dead latches). Wire-safe: no Buffer, no functions. */ + snapplaneStatus() { + return snapplaneStatusWire(this.db as unknown as Parameters[0], this.snapPlane); + } + + /** #5874 P2 (mapEgress): the S4-verified plane view for egress assembly. + * null when the plane has no rows for this sid yet (nothing initialized + * — not a degrade). A fresh R1 verdict (checksum mismatch) latches the + * sticky dead flag and returns null — the caller keeps legacy bytes. */ + snapEgressView(sessionId: string): SnapView | null { + try { + const view = readSnapView(this.db as unknown as Parameters[0], sessionId); + if (!view) return null; + // #3762: rows persist the media-OFFLOADED form (imageBase64 = + // {"$media": hash}); every other row accessor hydrates on read — + // the egress view must too, or the seam walk and tripwire hash an + // offload object instead of the image bytes (identitySig TypeError + // took a whole turn down: the plane path must never fail a turn). + return { + ...view, + rows: view.rows.map((r) => r.postJson ? { ...r, postJson: this.hydrateMedia(r.postJson) } : r), + }; + } catch (err) { + if (err instanceof SnapPlaneDeadError) { + loggerLog("warn", `[snaplane] sid=${sessionId} egress read hit a dead plane (R1) — legacy egress kept`); + return null; + } + throw err; + } + } + + // #5874 2×2 stage 0 (brief 19544722): the element world match lives at the + // head of the chain, BEFORE protocol conversion. These accessors are the + // storage side of that stage — the pure decision (snapshot/plane.ts) and + // the kernel-input assembly (snapshot/assemble.ts) are lane-side. + + private snapSessRawOf(sid: string): string { + // Raw host session id (P9: never truncated) → the l1_world key; the + // conversation binding when the host supplied one, the sid otherwise. + // (Same resolution as the save path below — stage 0 and the plane + // commit must address the SAME l1 row.) + return ( + (this.db.prepare("SELECT conversation_id FROM conversations WHERE session_id = ?").get(sid) as { conversation_id: string } | undefined) + ?.conversation_id ?? sid + ); + } + + /** Stage 0 for one turn: ensure the l1 row exists (idempotent session- + * level v1 record), run the element world match on the stored envelope, + * and persist the new canon (every verdict except `diverge` — that one + * has zero side effects by construction). A session-level identity + * conflict ("red") throws — the lane maps it to the R3 hard 400 BEFORE + * any kernel side effect. + */ + snapStage0(sid: string, incoming: string[], opts?: { resync?: boolean, seqs?: number[] | null }): Stage0Result { + const sessRaw = this.snapSessRawOf(sid); + const now = Date.now(); + const l1 = snapL1Record(this.db, sessRaw, sid, this.snapWitnessProbe(sid), now); + if (l1 === "red") { + // Session-level identity conflict (R3): the raw host id maps to a + // DIFFERENT ledger. Hard-reject BEFORE any kernel side effect — + // snapL1Record wrote nothing on this path. + throw new FoundationRejectError(`l1 ledger conflict: sess_raw=${sessRaw} maps to another ledger (R3)`); + } + return stage0Decide({ db: this.db, sessRaw, ledgerId: sid, incoming, seqs: opts?.seqs ?? null, resync: opts?.resync, now }); + } + + /** The plane's ACTIVE rows, pos-ordered — the "old cores from the table" + * half of the kernel-input assembly. post_json is the tagged kernel + * byte (invariant v) in its STORED (media-offloaded) form; the caller + * hydrates before feeding the kernel. `coverMax` maps each active + * block id to the HIGHEST row pos it covers (over the FULL row set — + * the covered rows themselves) — the kernel-input anchor placement + * input (an anchor renders before the first surviving element above + * its covered run). */ + snapStage0Cores(sid: string): { rows: { pos: number; ids: string[]; postJson: string }[]; coverMax: Record } { + const all = snapRowsAll(this.db, sid); + const coverMax: Record = {}; + for (const r of all) { + if (r.coverRef !== null && r.pos > (coverMax[r.coverRef] ?? -1)) { + coverMax[r.coverRef] = r.pos; + } + } + const rows = all + .filter((r) => r.active === 1 && r.postJson !== null) + .map((r) => ({ pos: r.pos, ids: r.coreIds, postJson: this.hydrateMedia(r.postJson as string) })); + return { rows, coverMax }; + } + + /** #6181 round 2, Residual 2 hardening: the plane's max ACTIVE pos for + * this sid (-1 when the plane has no active row) — the frontier a + * full-world round leaves the provider chain holding. */ + snapPlaneMaxPos(sid: string): number { + const all = snapRowsAll(this.db, sid); + let max = -1; + for (const r of all) { + if (r.active === 1 && r.pos > max) max = r.pos; + } + return max; + } + + /** The stage-0 resync (brief red gate R4 / the resync ruling): reset the + * session's plane (rows + log + ckpt + the l1 element envelope + the + * sticky R1 latch) in one txn. This turn then runs the legacy full path + * and re-seeds the plane through the normal in-request flush; the NEXT + * turn re-issues the envelope from the client's current array (one-round + * convergence). */ + snapStage0Reset(sid: string): void { + const sessRaw = this.snapSessRawOf(sid); + this.db.transaction(() => { + this.db.prepare("DELETE FROM snap_rows WHERE sid = ?").run(sid); + this.db.prepare("DELETE FROM snap_log WHERE sid = ?").run(sid); + this.db.prepare("DELETE FROM snap_ckpt WHERE sid = ?").run(sid); + this.db.prepare("UPDATE l1_world SET envelope_json = NULL, envelope_checksum = NULL WHERE sess_raw = ?").run(sessRaw); + }); + clearSnapPlaneDead(sid); + } + + /** #6181 M3: whether this sid's R3 breaker latch PERSISTED from a + * previous process (a restart must not re-arm a misbehaving host and + * re-400 the same broken envelope). */ + snapR3Latched(sid: string): boolean { + try { + return snapR3LatchedRead(this.db, this.snapSessRawOf(sid)); + } catch { + return false; // schema/column edge: treat as unlatched, the in-memory breaker still fires + } + } + + /** #6181 M3: persist the R3 breaker latch (best-effort — a failed + * write degrades to the in-memory latch, which still fires this + * process). */ + snapR3Latch(sid: string): void { + try { + snapR3LatchWrite(this.db, this.snapSessRawOf(sid), Date.now()); + } catch { /* best-effort — see above */ } + } + /** #3250: cadence-tick WAL reclamation (see WAL_CHECKPOINT_EVERY_INGESTS). * Never throws into the ingest path — a failed checkpoint just leaves the * WAL longer than ideal until the next tick. */ @@ -1971,6 +3110,14 @@ export class UnifiedStore { this.db.prepare("DELETE FROM conversations WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM session_ext WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM sessions WHERE session_id = ?").run(sessionId); + // #5874: the snapshot plane rides the session — rows / log / + // ckpt are keyed by sid, the L1 entry by ledger_id. The plane + // stores canonical forms INLINE (not CAS), so no release here; + // no orphan may survive (T6). + this.db.prepare("DELETE FROM snap_rows WHERE sid = ?").run(sessionId); + this.db.prepare("DELETE FROM snap_log WHERE sid = ?").run(sessionId); + this.db.prepare("DELETE FROM snap_ckpt WHERE sid = ?").run(sessionId); + this.db.prepare("DELETE FROM l1_world WHERE ledger_id = ?").run(sessionId); for (const hash of refs) this.releaseContent(hash); // #4796 follow-up: scoped collection over this session's own // released hashes (see collectZeroRefIn) — never the whole table. @@ -1996,12 +3143,17 @@ export class UnifiedStore { "(SELECT COUNT(*) FROM ccr_entries WHERE session_id = ?) AS ccrEntries, " + "(SELECT COUNT(*) FROM session_ext WHERE session_id = ?) AS ext, " + "(SELECT COUNT(*) FROM checkpoints WHERE session_id = ?) AS cps, " + - "(SELECT COUNT(*) FROM checkpoints WHERE session_id = ? AND pinned = 1) AS pinnedCps", + "(SELECT COUNT(*) FROM checkpoints WHERE session_id = ? AND pinned = 1) AS pinnedCps, " + + "(SELECT COUNT(*) FROM snap_rows WHERE sid = ?) AS snapRows, " + + "(SELECT COUNT(*) FROM snap_log WHERE sid = ?) AS snapLogEntries, " + + "(SELECT COUNT(*) FROM snap_ckpt WHERE sid = ?) AS snapCkpts, " + + "(SELECT COUNT(*) FROM l1_world WHERE ledger_id = ?) AS l1Entries", ) - .get(sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId) as { + .get(sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId) as { sessions: number; conversations: number; messages: number; media: number; blocks: number; deadRefs: number; refs: number; ccrEntries: number; ext: number; cps: number; pinnedCps: number; + snapRows: number; snapLogEntries: number; snapCkpts: number; l1Entries: number; }; const inlineBytes = ((this.db @@ -2018,6 +3170,15 @@ export class UnifiedStore { .get(sessionId) as { b: number }).b ?? 0) + ((this.db .prepare("SELECT COALESCE(SUM(LENGTH(CAST(value_json AS BLOB))), 0) AS b FROM session_ext WHERE session_id = ?") + .get(sessionId) as { b: number }).b ?? 0) + + // #5874: the plane stores canonical forms INLINE (not CAS) — the + // folded rows' pre/post payloads are part of this session's + // inline footprint. + ((this.db + .prepare( + "SELECT COALESCE(SUM(LENGTH(core_ids) + COALESCE(LENGTH(pre_json), 0) + COALESCE(LENGTH(post_json), 0) + COALESCE(LENGTH(cover_ref), 0)), 0) AS b " + + "FROM snap_rows WHERE sid = ?", + ) .get(sessionId) as { b: number }).b ?? 0); const cas = this.db .prepare( @@ -2048,6 +3209,10 @@ export class UnifiedStore { sessionExt: counts.ext, checkpoints: counts.cps, pinnedCheckpoints: counts.pinnedCps, + snapRows: counts.snapRows, + snapLogEntries: counts.snapLogEntries, + snapCkpts: counts.snapCkpts, + l1Entries: counts.l1Entries, }, inlineBytes, casExclusiveBytes: cas.exclusiveBytes, diff --git a/tests/e2e/e2e-advisory-rollback.test.ts b/tests/e2e/e2e-advisory-rollback.test.ts index f7082e2053..09a46eeae3 100644 --- a/tests/e2e/e2e-advisory-rollback.test.ts +++ b/tests/e2e/e2e-advisory-rollback.test.ts @@ -74,8 +74,14 @@ function freePort(): Promise { } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } // The updater's uninstall→reinstall half-deletes the install dir (npm drops diff --git a/tests/e2e/e2e-delegate-notify.test.ts b/tests/e2e/e2e-delegate-notify.test.ts index 0a5103b4b4..58ae1b0bc4 100644 --- a/tests/e2e/e2e-delegate-notify.test.ts +++ b/tests/e2e/e2e-delegate-notify.test.ts @@ -265,6 +265,12 @@ async function runScenario(name: string, prompt: string, totalDirectives: number XDG_CACHE_HOME: ctx.xdg.cache, XDG_STATE_HOME: ctx.xdg.state, XDG_DATA_HOME: ctx.xdg.data, + // The scripted filler issues 9 identical fast `sleep 2` calls; the + // repetition guard's rapidity ladder (warn 3 / refuse 5 / abort 6) then + // trips on fast runners (no round reaches the 5s long-poll reset) and + // aborts the turn mid-scenario — a runner-speed coin flip. The guard is + // covered by its own unit suite and is orthogonal to #2320/#2546 here. + BILI_GUARDRAIL_REPETITION: "0", ...(cliEntry ? { PI_CLI_PATH: cliEntry } : {}), }); pi = new RpcClient(ctx.piCwd, env); diff --git a/tests/e2e/e2e-native-dsh-persona.test.ts b/tests/e2e/e2e-native-dsh-persona.test.ts index 06d0d4ab65..71206e723d 100644 --- a/tests/e2e/e2e-native-dsh-persona.test.ts +++ b/tests/e2e/e2e-native-dsh-persona.test.ts @@ -77,6 +77,10 @@ function cleanEnv(): NodeJS.ProcessEnv { if (key.startsWith("BILI") || key.startsWith("BILLION_CONTEXT") || key.startsWith("ACP_") || key.startsWith("DSH_") || key.startsWith("ZCODE")) delete env[key]; } for (const key of ["NODE_EXTRA_CA_CERTS", "NODE_OPTIONS", "NODE_TEST_CONTEXT", "HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "NO_PROXY", "no_proxy"]) delete env[key]; + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface (raw + // conversation keys persisting sessions), so pin the dual-write writer. + env.BILI_STORAGE_JSON_WRITER = "on"; return env; } diff --git a/tests/e2e/e2e-native-dsh.test.ts b/tests/e2e/e2e-native-dsh.test.ts index 93d2521c7e..b1a01c50ba 100644 --- a/tests/e2e/e2e-native-dsh.test.ts +++ b/tests/e2e/e2e-native-dsh.test.ts @@ -218,6 +218,10 @@ function cleanEnv(): NodeJS.ProcessEnv { ) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT: set by the node:test runner and inherited by the // spawned dsh — dsh-native.ts deliberately stands down inside node:test // (unit-test guard), which would silently disable the very bootstrap this diff --git a/tests/e2e/e2e-native-opencode.test.ts b/tests/e2e/e2e-native-opencode.test.ts index cf2200f2d2..b9355971e6 100644 --- a/tests/e2e/e2e-native-opencode.test.ts +++ b/tests/e2e/e2e-native-opencode.test.ts @@ -183,6 +183,10 @@ function cleanEnv(): NodeJS.ProcessEnv { ) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT: set by the node:test runner and inherited by the spawned // client — the native entry deliberately stands down inside node:test (unit- // test guard), which would silently disable the bootstrap this suite diff --git a/tests/e2e/e2e-native-pi.test.ts b/tests/e2e/e2e-native-pi.test.ts index 997d14734f..b180ed7cff 100644 --- a/tests/e2e/e2e-native-pi.test.ts +++ b/tests/e2e/e2e-native-pi.test.ts @@ -205,6 +205,10 @@ function cleanEnv(): NodeJS.ProcessEnv { ) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT: set by the node:test runner and inherited by the spawned // pi — pi-native.ts deliberately stands down inside node:test (unit-test // guard), which would silently disable the very bootstrap this suite diff --git a/tests/e2e/e2e-pi-codex-ws.test.ts b/tests/e2e/e2e-pi-codex-ws.test.ts index bab066d3f4..9c57ff1a1f 100644 --- a/tests/e2e/e2e-pi-codex-ws.test.ts +++ b/tests/e2e/e2e-pi-codex-ws.test.ts @@ -257,6 +257,10 @@ function cleanEnv(extra: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { for (const key of Object.keys(env)) { if (key.startsWith("BILI") || key.startsWith("BILLION_CONTEXT") || key.startsWith("ACP_")) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT stands down pi-native inside node:test; the spawned pi // is a real client, not a test context (same rationale as e2e-native-pi). for (const key of ["NODE_EXTRA_CA_CERTS", "PI_SESSION_FILE", "PI_PROVIDER", "PI_MODEL", "NODE_OPTIONS", "NODE_TEST_CONTEXT"]) delete env[key]; diff --git a/tests/e2e/e2e-registry.test.ts b/tests/e2e/e2e-registry.test.ts index 459737a209..0eba927299 100644 --- a/tests/e2e/e2e-registry.test.ts +++ b/tests/e2e/e2e-registry.test.ts @@ -48,8 +48,14 @@ function runBili(installDir: string, args: string[], env: Record return { code: res.status ?? -1, stdout: res.stdout ?? "", stderr: res.stderr ?? "" }; } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } // Stage a publishable tarball of THIS package at a synthetic version: same diff --git a/tests/e2e/e2e-release-canary.test.ts b/tests/e2e/e2e-release-canary.test.ts index d6931b7bca..7ac7ee59fa 100644 --- a/tests/e2e/e2e-release-canary.test.ts +++ b/tests/e2e/e2e-release-canary.test.ts @@ -125,8 +125,14 @@ async function extractInstall(work: string, tgz: string): Promise { return installDir; } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } function freePort(): Promise { diff --git a/tests/e2e/e2e-update-state.test.ts b/tests/e2e/e2e-update-state.test.ts index e31baf7659..158ff10081 100644 --- a/tests/e2e/e2e-update-state.test.ts +++ b/tests/e2e/e2e-update-state.test.ts @@ -83,8 +83,14 @@ async function extractInstall(work: string, tgz: string): Promise { return installDir; } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } function freePort(): Promise { diff --git a/tests/fork-ccr-adoption.test.ts b/tests/fork-ccr-adoption.test.ts index 5b5bedde45..22e77d7b9f 100644 --- a/tests/fork-ccr-adoption.test.ts +++ b/tests/fork-ccr-adoption.test.ts @@ -38,6 +38,10 @@ import { maybeAdoptForkBlocks } from "../src/fork-adoption.ts"; import { rmrf } from "./tmp-rm.ts"; process.env.BILI_PERSIST = "0"; +// #6480 flipped the absent-default to pure-sqlite (JSON envelope writer +// off) — this suite reads the companion session ENVELOPE off disk, so pin +// the dual-write writer explicitly. +process.env.BILI_STORAGE_JSON_WRITER = "on"; const LIMIT = 200_000; const META = { protocol: "openai" as const, upstreamOrigin: "http://upstream.example/v1" }; diff --git a/tests/issue4666-migration-dedup.test.ts b/tests/issue4666-migration-dedup.test.ts index 4d0f0e51aa..55b00cdd66 100644 --- a/tests/issue4666-migration-dedup.test.ts +++ b/tests/issue4666-migration-dedup.test.ts @@ -155,7 +155,7 @@ test("#4666 sqlite: v8 migration collapses wireIndex-split duplicates and refund check = openSqliteDatabase(path.join(dir, "index.db")).db; const ver = check.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "stamped to v11 (v10 ts column + v11 FK child indexes, #5539)"); + assert.equal(ver.user_version, 12, "stamped to v12 (v10 ts column + v11 FK child indexes, #5539 + v12 snapshot-table foundation, #5874)"); // Exactly the lowest-seq arrival of each (msg_id, identity) group. const survivors = check diff --git a/tests/issue4677-v8-migration-streaming.test.ts b/tests/issue4677-v8-migration-streaming.test.ts index b9fa507ba7..5290291720 100644 --- a/tests/issue4677-v8-migration-streaming.test.ts +++ b/tests/issue4677-v8-migration-streaming.test.ts @@ -99,7 +99,7 @@ test("#4677 backfill sets identities and drops wireIndex-split duplicates (lowes const { from, to } = migrateSchema(db); assert.equal(from, 7); - assert.equal(to, 11); + assert.equal(to, 12); const identity = identityOfStoredContent(storedMessage("stripped", "hello")); assert.equal(identityOf(db, "s1", 1), identity); // the later duplicate row is gone and its content CAS ref handed back @@ -109,7 +109,7 @@ test("#4677 backfill sets identities and drops wireIndex-split duplicates (lowes ); assert.equal(refcountOf(db, a2), 0); assert.equal(refcountOf(db, a1), 1); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 11); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); db.close(); }); @@ -129,7 +129,7 @@ test("#4677 crash resume: sessions that already carry identities are skipped, no assert.equal(identityOf(db, "s-done", 1), "stale-planted"); assert.equal(identityOf(db, "s-pending", 1), identityOfStoredContent(storedMessage("pending", "still fresh"))); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 11); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); db.close(); }); @@ -150,7 +150,7 @@ test("#4677 rows without content bytes terminate with the ABSENT marker instead assert.equal(identity, "absent:"); // second run is a no-op — the marker is not identity = '', so the // session is not selected again - assert.equal(migrateSchema(db).from, 11); + assert.equal(migrateSchema(db).from, 12); assert.equal(identityOf(db, "s-ghost", 1), "absent:"); db.close(); }); @@ -193,7 +193,7 @@ test("#4677 empty store migrates to the schema head untouched", () => { downgradeToV7(db); const { from, to } = migrateSchema(db); assert.equal(from, 7); - assert.equal(to, 11); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 11); + assert.equal(to, 12); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); db.close(); }); diff --git a/tests/issue4796-gc-batched-sweep.test.ts b/tests/issue4796-gc-batched-sweep.test.ts index ad6aa53e8e..cf8e0ee7fe 100644 --- a/tests/issue4796-gc-batched-sweep.test.ts +++ b/tests/issue4796-gc-batched-sweep.test.ts @@ -84,7 +84,7 @@ test("#4796 v9: fresh install carries idx_content_refcount and candidate selecti seedContentRows(rawDb(store), 20_000, 1, "live"); seedContentRows(rawDb(store), 50, 0, "orphan"); const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_content_refcount'") .get(); @@ -118,7 +118,7 @@ test("#4796 v9: reopening a v8-era db gains the index", () => { const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_content_refcount'") .get(); @@ -243,7 +243,7 @@ test("#4796 e2e: v8 identity backfill releases wireIndex twins -> gc collects th try { const db2 = rawDb(reopened); const ver = db2.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); const twinLeft = db2 .prepare("SELECT COUNT(*) AS n FROM messages WHERE session_id = ? AND seq = ?") .get(sessId, twinSeq) as { n: number }; diff --git a/tests/issue5539-fk-content-indexes.test.ts b/tests/issue5539-fk-content-indexes.test.ts index b098db9008..73bfc5f005 100644 --- a/tests/issue5539-fk-content-indexes.test.ts +++ b/tests/issue5539-fk-content-indexes.test.ts @@ -89,7 +89,7 @@ test("#5539 v11: fresh install carries all six FK-child indexes and the DELETE p const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); for (const name of FK_CHILD_INDEXES) { const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?") @@ -129,7 +129,7 @@ test("#5539 v11: a v10-era library (no FK-child indexes) gains them on reopen", const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "v10 -> v11 migration ran"); + assert.equal(ver.user_version, 12, "v10 -> v12 migration ran"); for (const name of FK_CHILD_INDEXES) { const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?") diff --git a/tests/issue5874-2x2-p3.test.ts b/tests/issue5874-2x2-p3.test.ts new file mode 100644 index 0000000000..b7a162f2ac --- /dev/null +++ b/tests/issue5874-2x2-p3.test.ts @@ -0,0 +1,1141 @@ +/** + * #5874 2×2 — stage-0 identity enrollment (M1/M2 inbound) + seed/delta + * egress (M1–M4 outbound), acceptance items from the brief (comment + * 19544722, tracking #6181). + * + * M1 full-inbound + snapshot egress (the #5901 behavior, extended) + * M2 incremental-inbound + snapshot (this PR: matchAppend + seqs carrier) + * M3 incremental-inbound + delta (this PR: selectOutbound, responses) + * M4 full-inbound + delta (this PR: selectOutbound, responses) + * + * Unit scope (the lane wiring is e2e-verified by CI; these are the brief's + * acceptance pins 1–8 + the red-gate matrix): + * W* the world verdicts — echo/probe/scatter/issue/diverge/resync ± pairs + * (acceptance 5), the B1 origin-immutability trap, matchAppend. + * A* assembly — #2 M1-vs-M2 kernel-input byte identity, T5 scattered NEW + * (acceptance 4), #7 resync one-round convergence (acceptance 7), + * R4 table-gap throw, anchor placement. + * D* delta egress — #8 seed/delta/no-op pins (acceptance 8). + * R* red gates — R3 identity fault (400 + zero side effects + breaker), + * R2 diverge zero side effects. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import type { BiliMessage } from "acp-kernel/wire"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { + matchWorld, + matchAppend, +} from "../src/snapshot/identity.ts"; +import type { WorldEnvelopeEntry, WorldDecision } from "../src/snapshot/types.ts"; +import { envelopeChecksum } from "../src/snapshot/checksums.ts"; +import { + assembleKernelInput, + shouldResync, +} from "../src/snapshot/assemble.ts"; +import { + selectOutbound, + isFullResend, + type EgressElement, + type DeltaState, +} from "../src/snapshot/delta.ts"; +import { + FoundationRejectError, + R3FaultTracker, + checkR3Identity, + stage0Decide, +} from "../src/snapshot/plane.ts"; +import { snapL1Lookup, snapL1EnvelopeRead, snapL1Record } from "../src/storage/snap.ts"; +import { runStage0 } from "../src/snapshot/stage0.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-5874-p3-")); +const configPath = path.join(hermetic, "config.json"); +fs.writeFileSync(configPath, "{}"); + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +type StoredCanon = { + ledgerId: string; + envelope: WorldEnvelopeEntry[]; + envelopeChecksum: string; + issuedAt: number; + armedAt?: number; +}; + +interface WorldMatchInputLocal { + stored: StoredCanon | null; + incoming: string[]; + explainedGone?: Set; + sessRaw: string; + ledgerId: string; + now: number; + resync?: boolean; +} + +const now0 = 1_700_000_000_000; + +/** A stored element envelope (canon) for ids 0..n-1 (1-based ids m1.. + * 1-based pos, the issue/resync shape). */ +function canon(n: number): StoredCanon { + const ids = Array.from({ length: n }, (_, i) => `m${i + 1}`); + return { + ledgerId: "sid-t", + envelope: ids.map((id, i) => ({ id, pos: i + 1 })), + envelopeChecksum: envelopeChecksum(ids), + issuedAt: now0, + }; +} + +function matchInput(stored: StoredCanon | null, incoming: string[], extra: Partial = {}): WorldMatchInputLocal { + return { + stored, + incoming, + sessRaw: "sess-t", + ledgerId: "sid-t", + now: now0 + 1, + ...extra, + }; +} + +function openPlane(p: string): UnifiedStore { + return UnifiedStore.open(p, undefined, { snapBase: { enabled: true, shadowWrite: true } }); +} + +/** A plain (untagged) core message — the lane's converted form. */ +function core(i: number, id?: string): BiliMessage { + return { + id: id ?? `m${i}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + contentType: "text", + text: `message ${i}`, + } as BiliMessage; +} + +function plainSession(id: string, n: number): LegacySessionLike { + const msgs = Array.from({ length: n }, (_, i) => ({ + id: `m${i + 1}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + contentType: "text" as const, + text: `message ${i + 1}`, + })); + const byRaw: Record = {}; + const byRef: Record = {}; + msgs.forEach((m, i) => { + byRaw[m.id] = `m${String(i + 1).padStart(5, "0")}`; + byRef[`m${String(i + 1).padStart(5, "0")}`] = m.id; + }); + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw, byRef } }, + blockContents: new Map(), + lastMessages: msgs, + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: undefined, + }; +} + +// --------------------------------------------------------------------------- +// W — the world verdicts (acceptance 5: each verdict with paired +/- tests) +// --------------------------------------------------------------------------- + +test("W1 echo: the same envelope checksums → echo, canon untouched, B1 origin frozen", () => { + const stored = canon(4); + const ids = stored.envelope.map((e) => e.id); + const d = matchWorld(matchInput(stored, ids)); + assert.equal(d.mode, "echo"); + assert.deepEqual(d.newIdPos, []); + assert.deepEqual(d.goneIds, []); + assert.equal(d.issuedAt, now0, "B1: the origin never moves on echo"); + assert.equal(d.envelopeChecksum, stored.envelopeChecksum); +}); + +test("W2 echo +/- : one id changed → NOT echo (the checksum is content-binding)", () => { + const stored = canon(4); + const d = matchWorld(matchInput(stored, ["m1", "m2", "m3", "mX"])); + assert.notEqual(d.mode, "echo"); + // mX is a new interleaved span (the head is stable, survivor order holds) + // → scatter; the replaced m4 is reported gone (reconcile: delete). + assert.equal(d.mode, "scatter"); + assert.deepEqual(d.newIdPos, [{ id: "mX", pos: 5 }]); + assert.deepEqual(d.goneIds, ["m4"]); +}); + +test("W3 probe: a tail extension → probe, fresh pos at maxPos+1", () => { + const d = matchWorld(matchInput(canon(4), ["m1", "m2", "m3", "m4", "m5"])); + assert.equal(d.mode, "probe"); + assert.deepEqual(d.newIdPos, [{ id: "m5", pos: 5 }]); + assert.deepEqual(d.goneIds, []); +}); + +test("W4 probe +/- : the extension with a GAP in the middle → scatter", () => { + const d = matchWorld(matchInput(canon(4), ["m1", "m2", "NEW", "m3", "m4"])); + assert.equal(d.mode, "scatter", "an interleaved new id is a scatter, not a probe"); + assert.deepEqual(d.newIdPos, [{ id: "NEW", pos: 5 }]); +}); + +test("W5 issue: no stored canon → issue, fresh pos 0.., armed now", () => { + const d = matchWorld(matchInput(null, ["a", "b"])); + assert.equal(d.mode, "issue"); + assert.deepEqual(d.newIdPos, [{ id: "a", pos: 1 }, { id: "b", pos: 2 }]); + assert.equal(d.armedAt, now0 + 1, "the issue arms the lane now"); +}); + +test("W6 issue +/- : resync with a stored canon → re-issue pos 1.. from the client array", () => { + const d = matchWorld(matchInput(canon(3), ["x", "y", "z"], { resync: true })); + assert.equal(d.mode, "resync"); + assert.deepEqual(d.newIdPos, [{ id: "x", pos: 1 }, { id: "y", pos: 2 }, { id: "z", pos: 3 }]); +}); + +test("W7 diverge: head moved → diverge, canon untouched", () => { + const d = matchWorld(matchInput(canon(4), ["OTHER", "m1", "m2", "m3"])); + assert.equal(d.mode, "diverge"); + assert.equal(d.witness, "head m1 -> OTHER; orderBroken=false"); +}); + +test("W8 diverge: survivor order broken → diverge", () => { + const d = matchWorld(matchInput(canon(4), ["m1", "m3", "m2", "m4"])); + assert.equal(d.mode, "diverge"); + assert.equal(d.witness, "head m1 -> m1; orderBroken=true"); +}); + +test("W9 diverge +/- : a fold-explained gone at the head is NOT a head move", () => { + const d = matchWorld(matchInput(canon(4), ["m2", "m3", "m4"], { explainedGone: new Set(["m1"]) })); + assert.notEqual(d.mode, "diverge"); + assert.equal(d.mode, "probe", "the shrink with a fold-explained gone is a deletions-only probe"); + assert.deepEqual(d.goneIds, ["m1"]); +}); + +test("W10 matchAppend: pure M2 addition — canon order preserved, whole-sequence checksum", () => { + const stored = canon(4); + const d = matchAppend(stored, ["m5", "m6"]); + assert.equal(d.mode, "probe"); + assert.deepEqual( + d.envelope.map((e) => [e.id, e.pos]), + [["m1", 1], ["m2", 2], ["m3", 3], ["m4", 4], ["m5", 5], ["m6", 6]], + ); + assert.deepEqual(d.newIdPos, [{ id: "m5", pos: 5 }, { id: "m6", pos: 6 }]); + // The next M1 full resend (all six ids) must ECHO against this canon. + assert.equal(d.envelopeChecksum, envelopeChecksum(["m1", "m2", "m3", "m4", "m5", "m6"])); + assert.equal(d.issuedAt, now0, "B1: the origin never moves on extend"); +}); + +test("W11 matchAppend +/- : the next M1 full resend echoes (the round-trip)", () => { + const stored = canon(4); + const app = matchAppend(stored, ["m5"]); + const d = matchWorld(matchInput( + { ...stored, envelopeChecksum: app.envelopeChecksum, envelope: app.envelope }, + ["m1", "m2", "m3", "m4", "m5"], + )); + assert.equal(d.mode, "echo"); +}); + +// --------------------------------------------------------------------------- +// A — assembly (acceptance 2, 4, 7 + R4 + anchors) +// --------------------------------------------------------------------------- + +function storeWithRows(n: number, sid: string): UnifiedStore { + const store = openPlane(path.join(hermetic, `a-${sid}-${n}`, "index.db")); + store.ingestSession(planIngest(plainSession(sid, n))); + return store; +} + +test("A1 #2 M1 vs M2: same turn, full-inbound vs incremental-inbound kernel input byte-identical", () => { + const sid = "a1"; + const store = storeWithRows(10, sid); + const cores = store.snapStage0Cores(sid); + assert.equal(cores.rows.length, 10); + + const newCore = core(11); + const rowCores = cores.rows.map((r) => ({ pos: r.pos, ids: r.ids, msg: JSON.parse(r.postJson) as BiliMessage })); + const ten = Array.from({ length: 10 }, (_, i) => `m${i + 1}`); + + // Seed the world (first turn: the client's full array of 10). + const seed = store.snapStage0(sid, ten); + assert.equal(seed.decision.mode, "issue"); + + // M1: the client full-resends all 11 (the plane holds 10 → the 11th is + // the only new span). seqs present but the array is NOT all-unknown → + // the matchWorld path (probe/scatter), new id appended. + const m1 = store.snapStage0(sid, [...ten, newCore.id], { seqs: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] }); + assert.equal(m1.decision.mode, "probe"); + assert.deepEqual(m1.decision.newIdPos, [{ id: "m11", pos: 11 }]); + const asm1 = assembleKernelInput({ decision: m1.decision, rows: rowCores, newCores: [newCore] }); + + // A FRESH store (a different DB — the two legs are ALTERNATIVE client + // behaviors on the same turn; sharing the file would let leg M1's canon + // contaminate leg M2's stage-0 verdict) with the same plane state: + // rows m1..m10 only. + const store2 = openPlane(path.join(hermetic, `a-${sid}-m2-10`, "index.db")); + store2.ingestSession(planIngest(plainSession(sid, 10))); + // Enroll the canon first (the M2 branch requires a stored canon): the + // seed turn was the M1 full resend of 10. + store2.snapStage0(sid, ten); + const m2 = store2.snapStage0(sid, [newCore.id], { seqs: [11] }); + assert.equal(m2.decision.mode, "probe", "a contiguous all-unknown seqs run is the M2 append"); + assert.deepEqual(m2.decision.newIdPos, [{ id: "m11", pos: 11 }]); + const cores2 = store2.snapStage0Cores(sid); + const asm2 = assembleKernelInput({ + decision: m2.decision, + rows: cores2.rows.map((r) => ({ pos: r.pos, ids: r.ids, msg: JSON.parse(r.postJson) as BiliMessage })), + newCores: [newCore], + }); + + assert.equal(asm1.oldCount, 10); + assert.equal(asm2.oldCount, 10); + assert.equal( + JSON.stringify(asm1.kernelInput), + JSON.stringify(asm2.kernelInput), + "#2: the assembled kernel input cores are byte-identical (M1 vs M2)", + ); +}); + +test("A1b #2 +/- : the M2 branch REQUIRES the seqs carrier (no seqs → matchWorld path)", () => { + const sid = "a1b"; + const store = storeWithRows(10, sid); + store.snapStage0(sid, Array.from({ length: 10 }, (_, i) => `m${i + 1}`)); + // All-unknown WITHOUT seqs: a full (re-minted) envelope → matchWorld → + // the old canon ids are gone → head moved → diverge (never guess). + const d = store.snapStage0(sid, ["m11"]); + assert.equal(d.decision.mode, "diverge"); +}); + +test("A1c #2 +/- : a non-contiguous seqs run falls back to matchWorld", () => { + const sid = "a1c"; + const store = storeWithRows(10, sid); + store.snapStage0(sid, Array.from({ length: 10 }, (_, i) => `m${i + 1}`)); + const d = store.snapStage0(sid, ["m11"], { seqs: [15] }); + assert.notEqual(d.decision.mode, "probe", "seqs[0]=15 != frontier+1=11 → not the M2 append"); +}); + +test("A2 T5: a scattered NEW span lands at its CONVERSATION position (envelope order, not pos order)", () => { + const sid = "a2"; + const store = storeWithRows(10, sid); + const ten = Array.from({ length: 10 }, (_, i) => `m${i + 1}`); + store.snapStage0(sid, ten); // seed (issue) + const d = store.snapStage0(sid, ["m1", "m2", "m3", "NEW", "m4", "m5", "m6", "m7", "m8", "m9", "m10"]); + assert.equal(d.decision.mode, "scatter"); + assert.deepEqual(d.decision.newIdPos, [{ id: "NEW", pos: 11 }]); + const cores = store.snapStage0Cores(sid); + const newCore = { ...core(1), id: "NEW", text: "edited message" } as BiliMessage; + const asm = assembleKernelInput({ + decision: d.decision, + rows: cores.rows.map((r) => ({ pos: r.pos, ids: r.ids, msg: JSON.parse(r.postJson) as BiliMessage })), + newCores: [newCore], + }); + const ids = asm.kernelInput.map((m) => m.id); + assert.deepEqual(ids, ["m1", "m2", "m3", "NEW", "m4", "m5", "m6", "m7", "m8", "m9", "m10"]); + assert.equal(asm.kernelInput[3]!.text, "edited message"); + assert.equal(asm.oldCount, 10); +}); + +test("A3 R4: an envelope element that is neither new nor a plane row → throw (table gap)", () => { + const stored = canon(4); + const d = matchWorld(matchInput(stored, ["m1", "m2", "m3", "m4", "m5"])); + // Only rows 1-3 exist (the table is behind the client). + assert.throws( + () => + assembleKernelInput({ + decision: d, + rows: [ + { pos: 0, ids: ["m1"], msg: core(1) }, + { pos: 1, ids: ["m2"], msg: core(2) }, + { pos: 2, ids: ["m3"], msg: core(3) }, + ], + newCores: [core(5, "m5")], + }), + /table behind the client/, + ); +}); + +test("A3b R4 +/- : a fold-explained gone in the envelope is SKIPPED (skipIds), no throw", () => { + const stored = canon(4); + // m1 was fold-covered (explained gone); the client still lists it. + const d = matchWorld(matchInput(stored, ["m1", "m2", "m3", "m4"], { explainedGone: new Set(["m1"]) })); + assert.equal(d.mode, "echo", "checksum match is positional — the gone is reconciled out of band"); + const asm = assembleKernelInput({ + decision: d, + rows: [ + { pos: 2, ids: ["m2"], msg: core(2) }, + { pos: 3, ids: ["m3"], msg: core(3) }, + { pos: 4, ids: ["m4"], msg: core(4) }, + ], + newCores: [], + skipIds: new Set(["m1"]), + }); + assert.deepEqual(asm.kernelInput.map((m) => m.id), ["m2", "m3", "m4"]); +}); + +test("A4 #7 resync: diverge → zero side effects (the canon is untouched), shouldResync true", () => { + const sid = "a4"; + const store = storeWithRows(10, sid); + const ten = Array.from({ length: 10 }, (_, i) => `m${i + 1}`); + const seed = store.snapStage0(sid, ten); + assert.equal(seed.decision.mode, "issue"); + const db = (store as unknown as { db: import("../src/storage/snap.ts").DbLike }).db; + const sessRaw = sid; // plainSession id == conversation id + const before = snapL1EnvelopeRead(db, sessRaw); + assert.ok(before && before.envelopeChecksum); + + const d = store.snapStage0(sid, ["OTHER", "m1", "m2"]); + assert.equal(d.decision.mode, "diverge"); + assert.equal(d.diverge, true); + assert.ok(shouldResync(d.decision, 10)); + + // Zero side effects: the diverge turn persisted nothing. + const after = snapL1EnvelopeRead(db, sessRaw); + assert.equal(after!.envelopeChecksum, before!.envelopeChecksum, "the diverge wrote nothing"); + + // The resync: reset + re-issue from the client's CURRENT array → + // one-round convergence. + store.snapStage0Reset(sid); + const re = store.snapStage0(sid, ["OTHER", "m1", "m2"], { resync: true }); + assert.equal(re.decision.mode, "resync"); + assert.deepEqual( + re.decision.envelope.map((e) => e.id), + ["OTHER", "m1", "m2"], + ); + assert.equal(shouldResync(re.decision, 0), true, "the resync turn itself resets (the lane re-seeds from it)"); +}); + +test("A4b #7 +/- : a plain issue on an EMPTY plane is the NORMAL seed (not a resync)", () => { + const sid = "a4b"; + const store = openPlane(path.join(hermetic, "a4b", "index.db")); + const d = store.snapStage0(sid, ["a", "b", "c"]); + assert.equal(d.decision.mode, "issue"); + assert.equal(shouldResync(d.decision, 0), false, "seed must not loop the reset"); +}); + +test("A5 anchor placement: the anchor renders before the first element past the covered run", () => { + const stored = canon(6); + const d = matchWorld(matchInput(stored, ["m1", "m2", "m3", "m4", "m5", "m6"])); + const rows = Array.from({ length: 6 }, (_, i) => ({ pos: i, ids: [`m${i + 1}`], msg: core(i + 1) })); + const anchor = { id: "acp_summary_b1", role: "system", contentType: "text", text: "[Compressed conversation section]" } as BiliMessage; + // The block covered rows through pos 3 (1-based: m1..m3); m1 (the pin) + // stays active, m2/m3 flipped. The served elements: pos 1..6 → the + // anchor sits before the first served element with pos > 3. + const asm = assembleKernelInput({ + decision: d, + rows, + newCores: [], + anchors: [{ msg: anchor, blockId: "b1" }], + coverMax: { b1: 3 }, + }); + const ids = asm.kernelInput.map((m) => m.id); + assert.equal(ids.indexOf("acp_summary_b1"), 3, "the anchor precedes m4 (pos 4, the first past the run)"); + assert.deepEqual(ids, ["m1", "m2", "m3", "acp_summary_b1", "m4", "m5", "m6"]); +}); + +test("A5b anchor +/- : no coverMax evidence (covered rows GC'd) → the view start", () => { + const stored = canon(3); + const d = matchWorld(matchInput(stored, ["m1", "m2", "m3"])); + const anchor = { id: "acp_summary_b1", role: "system", contentType: "text", text: "[Compressed conversation section]" } as BiliMessage; + const asm = assembleKernelInput({ + decision: d, + rows: [ + { pos: 1, ids: ["m1"], msg: core(1) }, + { pos: 2, ids: ["m2"], msg: core(2) }, + { pos: 3, ids: ["m3"], msg: core(3) }, + ], + newCores: [], + anchors: [{ msg: anchor, blockId: "b1" }], + }); + assert.deepEqual(asm.kernelInput.map((m) => m.id), ["acp_summary_b1", "m1", "m2", "m3"]); +}); + +test("A6 P5: the new-core count must match the verdict (a parse of a non-new span throws)", () => { + const stored = canon(2); + const d = matchWorld(matchInput(stored, ["m1", "m2"])); + assert.throws( + () => + assembleKernelInput({ + decision: d, + rows: [{ pos: 0, ids: ["m1"], msg: core(1) }, { pos: 1, ids: ["m2"], msg: core(2) }], + newCores: [core(9)], + }), + /newCores length/, + ); +}); + +// --------------------------------------------------------------------------- +// D — delta egress (acceptance 8) +// --------------------------------------------------------------------------- + +function elementsOf(n: number): EgressElement[] { + const els: EgressElement[] = Array.from({ length: n }, (_, i) => ({ + msg: core(i + 1), + pos: i, + isAnchor: false, + })); + els.splice(2, 0, { + msg: { id: "acp_summary_b1", role: "system", contentType: "text", text: "s" } as BiliMessage, + pos: -1, + isAnchor: true, + anchorId: "b1", + }); + return els; +} + +test("D1 #8 seed: the first round (null state) is a FULL resend of the whole view", () => { + const els = elementsOf(5); + const sel = selectOutbound(els, null); + assert.equal(sel.reseed, true); + // A reseed IS the full view (the lane sends the whole view; `delta` is + // empty and isFullResend only flags the delta-grown-to-full case). + assert.equal(isFullResend(sel, els), false); + assert.equal(sel.nextState.frontierPos, 4, "the seed frontier covers every sent data row"); + assert.deepEqual(sel.nextState.servedAnchors, ["b1"]); +}); + +test("D2 #8 delta: the next round sends ONLY rows past the frontier (plus unsent anchors)", () => { + const els = elementsOf(5); + const seed = selectOutbound(els, null); + // One new tail row (pos 5) + the anchor re-sent? NO — the anchor was + // already served: the delta is exactly the one new row. + const els2 = [...els, { msg: core(6), pos: 5, isAnchor: false }]; + const sel = selectOutbound(els2, seed.nextState); + assert.equal(sel.reseed, false); + assert.deepEqual(sel.delta.map((e) => e.pos), [5]); + assert.equal(sel.nextState.frontierPos, 5); + assert.deepEqual(sel.nextState.servedAnchors, ["b1"]); +}); + +test("D3 #8 no-op: a steady round with nothing new sends an empty delta (the plane round rides the provider cache)", () => { + const els = elementsOf(5); + const seed = selectOutbound(els, null); + const sel = selectOutbound(els, seed.nextState); + assert.equal(sel.reseed, false); + assert.deepEqual(sel.delta, []); +}); + +test("D4 #8 +/- : an unsent anchor (a NEW fold) rides the next delta", () => { + const els = elementsOf(5); + const seed = selectOutbound(els, null); + const els2 = [ + ...els, + { msg: { id: "acp_summary_b2", role: "system", contentType: "text", text: "s2" } as BiliMessage, pos: -2, isAnchor: true, anchorId: "b2" }, + { msg: core(6), pos: 5, isAnchor: false }, + ]; + const sel = selectOutbound(els2, seed.nextState); + assert.deepEqual(sel.delta.map((e) => e.anchorId ?? e.pos), ["b2", 5]); + assert.deepEqual(sel.nextState.servedAnchors.sort(), ["b1", "b2"]); +}); + +test("D5 isFullResend: the delta grown to the whole view is a re-seed in disguise", () => { + // A view with NO anchors: 5 data rows, state frontier -1 (nothing sent) + // → the delta covers every element → full resend. + const els: EgressElement[] = Array.from({ length: 5 }, (_, i) => ({ + msg: core(i + 1), + pos: i, + isAnchor: false, + })); + const state: DeltaState = { frontierPos: -1, servedAnchors: [] }; + const sel = selectOutbound(els, state); + assert.equal(isFullResend(sel, els), true, "5 of 5 unsent → the delta IS the full view"); + // +/- : the frontier already past the data, an UNSENT anchor rides alone + // → the delta is 1 element, NOT a full resend. + const els2 = [...els, { + msg: { id: "acp_summary_b9", role: "system", contentType: "text", text: "s" } as BiliMessage, + pos: -1, + isAnchor: true, + anchorId: "b9", + }]; + const sel2 = selectOutbound(els2, { frontierPos: 4, servedAnchors: [] }); + assert.deepEqual(sel2.delta.map((e) => e.anchorId), ["b9"]); + assert.equal(isFullResend(sel2, els2), false, "one anchor alone is not a full resend"); +}); + +// --------------------------------------------------------------------------- +// R — red gates +// --------------------------------------------------------------------------- + +test("R1 R3: a declared host that fails to collect msgids → the fault (400 source)", () => { + assert.equal(checkR3Identity(true, 5, undefined), false, "declared, no msgids"); + assert.equal(checkR3Identity(true, 5, ["a", "b", "c"]), false, "declared, wrong length"); + assert.equal(checkR3Identity(true, 5, ["a", "b", "c", "d", "e"]), true, "declared, complete"); + assert.equal(checkR3Identity(false, 5, undefined), true, "non-cohort: never a fault"); + assert.equal(checkR3Identity(true, 0, undefined), true, "no elements: nothing to collect"); +}); + +test("R2 R3: FoundationRejectError is the 400 control channel (zero side effects are the caller's, the error is the signal)", () => { + const err = new FoundationRejectError("envelope fault"); + assert.ok(err instanceof Error); + assert.equal(err.status, 400); + assert.match(err.message, /R3 identity fault/); +}); + +test("R3 R3 breaker: 3 faults in the 10-minute window latch; the latch rejects further faults and outlives the window", () => { + const t = new R3FaultTracker(60_000, 3); + const t0 = 1_000_000; + assert.equal(t.recordFault("s", t0).latched, false); + assert.equal(t.recordFault("s", t0 + 1_000).latched, false); + assert.equal(t.recordFault("s", t0 + 2_000).latched, true, "the third strike latches"); + assert.equal(t.isLatched("s"), true); + // Latched sids reject immediately — even after the window would expire. + assert.equal(t.recordFault("s", t0 + 999_999).latched, true); + // A different sid is unaffected. + assert.equal(t.isLatched("other"), false); + t.clearLatched("s"); + assert.equal(t.isLatched("s"), false); +}); + +test("R4 R3 +/- : the window expires (a new window restarts the strike count)", () => { + const t = new R3FaultTracker(1000, 3); + t.recordFault("s", 0); + t.recordFault("s", 100); + t.recordFault("s", 200); // latched? strikeCap 3 → yes + assert.equal(t.isLatched("s"), true); + t.clearLatched("s"); + // Fresh window: two faults do NOT re-latch. + t.recordFault("s", 5_000); + t.recordFault("s", 6_000); + assert.equal(t.isLatched("s"), false); +}); + +test("R5 R2: diverge = zero side effects (stage0Decide writes nothing on a structural disagreement)", () => { + const dir = path.join(hermetic, "r5"); + fs.mkdirSync(dir, { recursive: true }); + const db = (openPlane(path.join(dir, "index.db")) as unknown as { db: import("../src/storage/snap.ts").DbLike }).db; + // The l1 row is recorded by the store facade before stage0 (the lane + // path) — do it here so the pure function has its world to match. + snapL1Record(db, "s5", "s5", "witness", now0); + // Seed the world: issue a 3-id envelope. + const seed = stage0Decide({ db, sessRaw: "s5", ledgerId: "s5", incoming: ["a", "b", "c"], now: now0 }); + assert.equal(seed.decision.mode, "issue"); + // A structural disagreement (head moved): the verdict is diverge and the + // stored envelope is UNCHANGED (zero side effects — the lane runs the + // legacy round and the client converges next turn). + const d = stage0Decide({ db, sessRaw: "s5", ledgerId: "s5", incoming: ["OTHER", "a", "b"], now: now0 + 1 }); + assert.equal(d.diverge, true); + const l1 = snapL1Lookup(db, "s5"); + assert.ok(l1, "the seed l1 row exists"); + const env = snapL1EnvelopeRead(db, "s5"); + assert.equal(env!.envelopeChecksum, seed.decision.envelopeChecksum, "the diverge persisted nothing"); +}); + +// --------------------------------------------------------------------------- +// S — the stage-0 store seam (M1 seed → M2 append round trip through SQLite) +// --------------------------------------------------------------------------- + +test("S1 the full M1→M2 round trip: seed (issue) → M1 probe → M2 append → M1 echo", () => { + const dir = path.join(hermetic, "s1"); + fs.mkdirSync(dir, { recursive: true }); + const store = openPlane(path.join(dir, "index.db")); + const sid = "s1"; + store.ingestSession(planIngest(plainSession(sid, 3))); + const rows = () => store.snapStage0Cores(sid).rows.map((r) => r.ids[0]!); + + // Seed: the client's first full array (3 ids). + const seed = store.snapStage0(sid, ["m1", "m2", "m3"]); + assert.equal(seed.decision.mode, "issue"); + + // M1 tail extension: full resend + one new id (1-based pos: seed is 1..3 + // → m4 lands at pos 4). + const m1 = store.snapStage0(sid, ["m1", "m2", "m3", "m4"]); + assert.equal(m1.decision.mode, "probe"); + assert.deepEqual(m1.decision.newIdPos, [{ id: "m4", pos: 4 }]); + + // M2: ONLY the new id + the seqs carrier (4 rows pos 0..3 → frontier 4, + // the next seq is 5). + store.ingestSession(planIngest(plainSession(sid, 4))); + const m2 = store.snapStage0(sid, ["m5"], { seqs: [5] }); + assert.equal(m2.decision.mode, "probe", "the M2 append"); + assert.deepEqual(m2.decision.newIdPos, [{ id: "m5", pos: 5 }]); + assert.deepEqual(m2.decision.envelope.map((e) => e.id), ["m1", "m2", "m3", "m4", "m5"]); + + // The next M1 full resend must ECHO the matchAppend canon. + const echo = store.snapStage0(sid, ["m1", "m2", "m3", "m4", "m5"]); + assert.equal(echo.decision.mode, "echo", "M1 full resend echoes the M2-persisted canon"); + assert.deepEqual(echo.decision.newIdPos, []); + void rows; +}); + +// --------------------------------------------------------------------------- +// Fix-round tests: M4 codec-level pin, B0 plane bypass, M3 latch persistence, +// and the HTTP e2e pins (B1 400 funnel, B3 pure continuation, B4 frontier +// commit on upstream accept). +// --------------------------------------------------------------------------- + +import http from "node:http"; +import { once } from "node:events"; +import { defaultConfig } from "acp-kernel"; +import { openaiToCore } from "acp-kernel/wire"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _resetSnapEgressForTests, applySnapEgress } from "../src/server/snap-egress.ts"; + +test("A1d M4: the codec-level pin — parsing the new element ALONE (M2 wire) yields the same core as parsing it inside the full array (M1 wire), wireIndex aside", () => { + const contents = Array.from({ length: 11 }, (_, i) => `codec content ${i + 1}`); + const fullBody = { model: "gpt-m4", messages: contents.map((c) => ({ role: "user", content: c })) }; + const full = openaiToCore(fullBody as never); + assert.equal(full.msgs.length, 11); + // M2 wire: the client sends ONLY the 11th element — a fresh parse with a + // fresh ClusterCounter. The canonical id derives from (role, contentType, + // text), so the counter state cannot change the id for an element whose + // content appears once. + const inc = openaiToCore({ model: "gpt-m4", messages: [fullBody.messages[10]!] } as never); + assert.equal(inc.msgs.length, 1); + const strip = (m: BiliMessage): Record => { + const c = { ...(m as unknown as Record) }; + delete c.wireIndex; // position on the wire — legitimately differs (10 vs 0) + return c; + }; + assert.equal( + JSON.stringify(strip(full.msgs[10]!)), + JSON.stringify(strip(inc.msgs[0]!)), + "the new element's core is byte-identical between the full-array parse (M1) and the standalone parse (M2), so the assembled kernel cores are codec-level identical, not just assembly-level identical", + ); + + // And through the assembly: plane rows m1..m10 + the new core from each + // leg's own parse — the kernel input bytes must match. + const sid = "a1d"; + const store = storeWithRows(10, sid); + store.snapStage0(sid, Array.from({ length: 10 }, (_, i) => `m${i + 1}`)); + const rows10 = store.snapStage0Cores(sid).rows.map((r) => ({ pos: r.pos, ids: r.ids, msg: JSON.parse(r.postJson) as BiliMessage })); + const ten = Array.from({ length: 10 }, (_, i) => `m${i + 1}`); + const m1 = store.snapStage0(sid, [...ten, "codec-11"], { seqs: ten.map((_, i) => i + 1).concat(11) }); + const asm1 = assembleKernelInput({ decision: m1.decision, rows: rows10, newCores: [full.msgs[10]!] }); + // Isolated DB for the M2 leg (see A1: the legs are alternative client + // behaviors on the same turn and must not share stage-0 canon state). + const store2 = openPlane(path.join(hermetic, `a-${sid}-m2-10`, "index.db")); + store2.ingestSession(planIngest(plainSession(sid, 10))); + store2.snapStage0(sid, ten); + const m2 = store2.snapStage0(sid, ["codec-11"], { seqs: [11] }); + const rows10b = store2.snapStage0Cores(sid).rows.map((r) => ({ pos: r.pos, ids: r.ids, msg: JSON.parse(r.postJson) as BiliMessage })); + const asm2 = assembleKernelInput({ decision: m2.decision, rows: rows10b, newCores: [inc.msgs[0]!] }); + // wireIndex is the parse-time position artifact (10 in the full parse, 0 + // in the standalone parse). It does not change the re-emitted wire: the + // plane rows carry no wireIndex, so the new element groups as its own + // wire message in BOTH legs. The semantic kernel cores must match. + const stripAll = (arr: BiliMessage[]) => JSON.stringify(arr.map(strip)); + assert.equal(stripAll(asm1.kernelInput), stripAll(asm2.kernelInput), "assembled kernel input byte-identical with codec-produced new cores (M1 full parse vs M2 standalone parse), wireIndex aside"); +}); + +test("B0a: a skipPlaneFlush save leaves the plane rows untouched — a collapsed view cannot delete history", () => { + const sid = "b0a"; + const store = storeWithRows(10, sid); + store.snapStage0(sid, Array.from({ length: 10 }, (_, i) => `m${i + 1}`)); + assert.equal(store.snapStage0Cores(sid).rows.length, 10); + + // A chained delta turn: the kernel view holds only the 4 messages it was + // given (a collapse). Without the flag, deriving events from this view + // would DELETE every row it does not contain. With the flag, the save + // runs the legacy ingest but skips ALL plane work. + const collapsed = [core(1, "m1"), core(2, "m2"), core(3, "m3"), core(4, "m4")]; + const plan = planIngest(plainSession(sid, 4)); + plan.planeMessages = collapsed as never; + plan.skipPlaneFlush = true; + store.ingestSession(plan); + const rows = store.snapStage0Cores(sid).rows; + assert.equal(rows.length, 10, "the plane still holds all 10 rows after the flagged save"); + assert.deepEqual(rows.map((r) => r.ids).flat().sort(), Array.from({ length: 10 }, (_, i) => `m${i + 1}`).sort(), "and the ids are exactly the original ten (nothing deleted, nothing added)"); + + // The flag is one-shot: the next save (full view, no flag) runs the plane + // path and stays at 10 rows (idempotent final-state diff). + const full = Array.from({ length: 10 }, (_, i) => core(i + 1, `m${i + 1}`)); + const plan2 = planIngest(plainSession(sid, 10)); + plan2.planeMessages = full as never; + store.ingestSession(plan2); + assert.equal(store.snapStage0Cores(sid).rows.length, 10); +}); + +test("B0b: egress bypass serves the legacy bytes verbatim — a stale plane never leaks into a delta turn", () => { + _resetSnapEgressForTests(); + const sid = "b0b"; + const store = storeWithRows(3, sid); + store.snapStage0(sid, ["m1", "m2", "m3"]); + // Register the store under test through the global seam (egress reads + // getStore() — the P2 G-test pattern). + const shadow = { + stats: { disabled: false }, + snapEgressView: (s: string) => store.snapEgressView(s), + } as never; + _setStoreForTest(new SessionStore({ enabled: true, shadow, log: () => undefined })); + + const state = { blocks: [] } as never; + const legacy = [core(1, "m1"), core(2, "m2"), core(3, "m3")]; + const deps = { sessionId: sid, protocol: "responses" as never, isCompressTool: (name: string) => name === "compress", log: () => undefined, transform: (msgs: BiliMessage[]) => msgs }; + const bypassed = applySnapEgress(state, legacy, { ...deps, bypass: true }); + assert.equal(bypassed, legacy, "bypass returns the legacy array BY REFERENCE (zero plane involvement for the turn)"); +}); + +test("M3: the R3 latch persists across store instances — a latched session degrades forever, never 400s again", () => { + const p = path.join(hermetic, "m3", "index.db"); + const a = openPlane(p); + assert.equal(a.snapR3Latched("sid-m3"), false); + a.snapR3Latch("sid-m3"); + // A FRESH instance over the same file — the process restart case: the + // in-memory breaker is cold, the persisted latch must carry the state. + const b = openPlane(p); + assert.equal(b.snapR3Latched("sid-m3"), true, "the latch survives a new store instance (process restart)"); + assert.equal(b.snapR3Latched("other-sid"), false, "other sessions are unaffected"); +}); + +// ── HTTP e2e (real proxy + fake upstream) ─────────────────────────────────── + +function listenT(server: http.Server): Promise { + if (server.listening) return Promise.resolve(); + return once(server, "listening").then(() => undefined); +} +function closeT(server: http.Server): Promise { + return new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); +} +function sseT(type: string, data: unknown): string { + return `event: ${type}\ndata: ${JSON.stringify(data)}\n\n`; +} +function sseDone(id = "resp_done"): string { + return [ + sseT("response.output_item.done", { item: { type: "message", id: "msg_1", role: "assistant", content: [{ type: "output_text", text: "ok" }] }, output_index: 0 }), + sseT("response.completed", { response: { id, status: "completed", output: [], usage: { input_tokens: 100, output_tokens: 5, total_tokens: 105 } } }), + ].join(""); +} + +type UpstreamBehavior = (n: number, body: string) => { status: number; payload: string }; + +async function withProxyT( + behavior: UpstreamBehavior, + fn: (url: string, bodies: string[]) => Promise, + opts: { storeDir?: string; env?: Record } = {}, +): Promise { + const bodies: string[] = []; + let reqN = 0; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + const body = Buffer.concat(chunks).toString("utf8"); + bodies.push(body); + const { status, payload } = behavior(++reqN, body); + const streaming = payload.startsWith("event: "); + res.writeHead(status, streaming ? { "content-type": "text/event-stream", "cache-control": "no-cache" } : { "content-type": "application/json" }); + res.write(payload); + res.end(); + }); + }); + upstream.listen(0, "127.0.0.1"); + await listenT(upstream); + const upstreamPort = (upstream.address() as { port: number }).port; + + const savedEnv: Record = {}; + for (const [k, v] of Object.entries(opts.env ?? {})) { savedEnv[k] = process.env[k]; process.env[k] = v; } + let store: SessionStore | undefined; + try { + if (opts.storeDir) { + process.env.BILI_STORAGE_UNIFIED = "shadow"; + // Plane + egress are OPT-IN since the 2026-11-13 soak revert + // (all four snapBase keys default off) — these e2e tests exercise + // the plane, so arm all three keys in a dedicated config file. + fs.mkdirSync(opts.storeDir, { recursive: true }); + const e2eConfig = path.join(opts.storeDir, "config.json"); + fs.writeFileSync(e2eConfig, JSON.stringify({ storage: { snapBase: { enabled: true, shadowWrite: true, egress: true } } })); + savedEnv.BILI_CONFIG_FILE = process.env.BILI_CONFIG_FILE; + process.env.BILI_CONFIG_FILE = e2eConfig; + store = new SessionStore({ enabled: true, dir: opts.storeDir, debounceMs: 0 }); + _setStoreForTest(store); + } else { + _setStoreForTest(new SessionStore({ enabled: false })); + } + const proxyOpts: ProxyOptions = { + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [`http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-e2e": { context: 400_000 } } } }, + modelContextLimit: 400_000, + kernelConfig: defaultConfig(400_000), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: true, + logFile: "off", + debug: false, + passthrough: false, + autoUpdate: false, + compat: { roles: {} }, + streamErrorShape: "protocol", + passthroughSource: null, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: false, + releaseNotesCheck: false, + mitm: { enabled: false, domains: [] }, + }; + const proxy = await startServer(proxyOpts); + await listenT(proxy); + const proxyPort = (proxy.address() as { port: number }).port; + try { + await fn(`http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`, bodies); + } finally { + await closeT(proxy); + } + } finally { + await closeT(upstream); + for (const [k, v] of Object.entries(savedEnv)) { if (v === undefined) delete process.env[k]; else process.env[k] = v; } + delete process.env.BILI_STORAGE_UNIFIED; + _setStoreForTest(new SessionStore({ enabled: false })); + void store; + } +} + +const PLUGIN_HEADERS = { "content-type": "application/json", "x-bili-plugin": "e2e-agent", "x-acp-session": "e2e-sess" } as Record; + +test("B1: a declared host that fails to collect msgids (count mismatch) → 400 + x-bili-snapshot: identity, no upstream forward", async () => { + await withProxyT( + () => ({ status: 200, payload: sseDone() }), + async (url, bodies) => { + const res = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ + model: "gpt-e2e", + stream: true, + input: [ + { type: "message", role: "user", content: "first" }, + { type: "message", role: "user", content: "second" }, + ], + biliMsgIds: ["b1m1"], // 1 declared vs 2 elements — the R3 fault + }), + }); + assert.equal(res.status, 400, "the identity fault is a 400 (the B1 control channel — not the generic 503)"); + assert.equal(res.headers.get("x-bili-snapshot"), "identity"); + const err = JSON.parse(await res.text()) as { error: { type: string; message: string } }; + assert.equal(err.error.type, "snapshot_identity_fault"); + assert.ok(err.error.message.includes("R3"), "the error names the fault class"); + assert.equal(bodies.length, 0, "zero side effects: the upstream was never contacted"); + }, + { env: { BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY: "1" } }, + ); +}); + +test("B3: a plugin chained round with EMPTY input is a pure continuation — the raw body passes through verbatim (no dev injection, id kept)", async () => { + await withProxyT( + () => ({ status: 200, payload: sseDone() }), + async (url, bodies) => { + const reqBody = { model: "gpt-e2e", stream: true, previous_response_id: "resp_b3", input: [] }; + const res = await fetch(url, { method: "POST", headers: PLUGIN_HEADERS, body: JSON.stringify(reqBody) }); + assert.equal(res.status, 200); + await res.text(); + assert.equal(bodies.length, 1); + const fwd = JSON.parse(bodies[0]!) as Record; + assert.equal(fwd.previous_response_id, "resp_b3", "the chain id is forwarded intact"); + assert.deepEqual(fwd.input, [], "the empty input passes through (NOT rebuilt, NOT dropped)"); + assert.ok(!JSON.stringify(fwd).includes("You are bili"), "no developer-message injection on the pure continuation (the B3 blind-response fix)"); + }, + {}, + ); +}); + +test("B4: the delta frontier commits only when the upstream ACCEPTS — a 400 turn leaves the frontier at its previous value (the next round re-seeds)", async () => { + await withProxyT( + (n) => (n === 2 ? { status: 400, payload: JSON.stringify({ error: { message: "upstream refused" } }) } : { status: 200, payload: sseDone() }), + async (url, bodies) => { + // Turn 1: seed (no chain ref yet). + const r1 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ model: "gpt-e2e", stream: true, input: [{ type: "message", role: "user", content: "one" }], biliMsgIds: ["b4m1"], biliLedgerSeqs: [1] }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + // Turn 2: CHAINED cohort turn (M2 append). Its outbound is the + // SEED round of the delta chain (no frontier yet) — and the + // upstream REFUSES it. The selected frontier must NOT commit. + const r2 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ model: "gpt-e2e", stream: true, previous_response_id: "resp_t1", input: [{ type: "message", role: "user", content: "two" }], biliMsgIds: ["b4m2"], biliLedgerSeqs: [2] }), + }); + assert.equal(r2.status, 400, "the upstream refusal passes through"); + await r2.text(); + + // Turn 3: another chained cohort turn. With the frontier + // uncommitted, the outbound must be a RE-SEED: the FULL view + // (all 3 world elements over the wire). If the 400 turn had + // advanced the frontier, this round would send a 1-element delta. + const r3 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ model: "gpt-e2e", stream: true, previous_response_id: "resp_t2", input: [{ type: "message", role: "user", content: "three" }], biliMsgIds: ["b4m3"], biliLedgerSeqs: [3] }), + }); + assert.equal(r3.status, 200); + await r3.text(); + const fwd3 = JSON.parse(bodies[2]!) as { input: { content?: unknown }[] }; + const texts = fwd3.input.flatMap((x) => (Array.isArray(x.content) ? x.content.map((p) => (p as { text?: string }).text ?? "") : [String(x.content ?? "")])); + for (const c of ["one", "two", "three"]) assert.ok(texts.some((t) => t.includes(c)), `turn 3's reseed view carries world element \"${c}\"`); + assert.equal(fwd3.input.length, 4, "the 3 world elements plus the re-injected developer message (a reseed round sends the full self-contained view + dev; a delta would have sent 1 world element and no dev)"); + }, + { storeDir: path.join(hermetic, "b4-store"), env: { BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY: "1", ACP_KEEP_RESPONSE_ID: "1" } }, + ); +}); + +test("Residual 2 (unit): storedFrontierPos is set only for a strict extension — probe + nothing gone + new elements; every other shape is null", () => { + const dir = path.join(hermetic, "r2u"); + fs.mkdirSync(dir, { recursive: true }); + const plane = openPlane(path.join(dir, "index.db")); + const sid = "r2-sid"; + const saved = process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY; + const savedUnified = process.env.BILI_STORAGE_UNIFIED; + process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY = "1"; + process.env.BILI_STORAGE_UNIFIED = "shadow"; // the SessionStore facade's snap* delegates need a live shadow + // The facade's replicasAvailable() reads shadow.stats.disabled — the raw + // UnifiedStore has no such surface, so shim it (the P2 G-test pattern). + const shadow = new Proxy(plane, { + get: (t: object, p: string | symbol) => (p === "stats" ? { disabled: false } : (t as Record)[p]), + }); + _setStoreForTest(new SessionStore({ enabled: true, shadow: shadow as never, log: () => undefined })); + try { + // Seed the plane with a 3-element world (rows pos 0..2). + plane.ingestSession(planIngest(plainSession(sid, 3))); + const run = (ids: string[]) => runStage0({ + sessionId: sid, + lane: "responses", + msgs: ids.map((id) => core(Number(id.slice(1)), id)), + incomingMsgIds: ids, + incomingSeqs: null, + pluginMode: true, + log: () => undefined, + }); + + // First stage-0 call: issue (the l1 envelope is recorded here — the + // plane rows already exist from the ingest above). + const seed = run(["m1", "m2", "m3"]); + assert.equal(seed.storedFrontierPos, null, "issue (no stored world yet) → null"); + + // Strict extension (+m4) → the stored world's boundary = max active + // row pos (the new rows are not flushed at stage-0 time). + const ext = run(["m1", "m2", "m3", "m4"]); + assert.equal(ext.storedFrontierPos, 2, "probe + nothing gone + new → the stored-world boundary (max active row pos)"); + + // A full resend with nothing new → echo → null (a re-seed is the + // honest answer; there is no provable chain boundary). + const echo = run(["m1", "m2", "m3", "m4"]); + assert.equal(echo.storedFrontierPos, null, "echo → null"); + + // An extension WITH a deletion (m2 gone) is not provable — the + // chain may not hold the hole → null, the lane re-seeds. + const del = run(["m1", "m3", "m4", "m5"]); + assert.equal(del.storedFrontierPos, null, "an extension with a deletion is not a strict prefix → null"); + } finally { + if (saved === undefined) delete process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY; + else process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY = saved; + if (savedUnified === undefined) delete process.env.BILI_STORAGE_UNIFIED; + else process.env.BILI_STORAGE_UNIFIED = savedUnified; + _setStoreForTest(new SessionStore({ enabled: false })); + } +}); + +test("Residual 1a+2 (e2e): the first CHAINED round of a keepResponseId chain serves the new elements ONLY, and a same-body RETRY of it is a no-op (id kept, empty input, no dev)", async () => { + await withProxyT( + (n) => ({ status: 200, payload: sseDone(`resp_done_${n}`) }), + async (url, bodies) => { + const msg = (role: string, content: string) => ({ type: "message", role, content }); + // Round 1: the full world, NO chain ref. The provider's chain + // starts here; no selection fires (no id in) → no frontier. + const r1 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ + model: "gpt-e2e", + stream: true, + input: [msg("user", "one"), msg("assistant", "ack one"), msg("user", "two")], + biliMsgIds: ["r2m1", "r2m2", "r2m3"], + }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + // Round 2: CHAINED (KEEP=1 forwards the id) + one new element. + // WITHOUT Residual 2 this would re-seed the full 4-element view; + // WITH it the outbound is exactly the new element, id kept, no + // developer re-injection. The chain id must be the one round 1's + // response actually carried (resp_done_1) — the frontier seed is + // trusted only for an id THIS proxy minted (Residual 2 hardening). + const r2 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ + model: "gpt-e2e", + stream: true, + previous_response_id: "resp_done_1", + input: [msg("user", "one"), msg("assistant", "ack one"), msg("user", "two"), msg("assistant", "ack two")], + biliMsgIds: ["r2m1", "r2m2", "r2m3", "r2m4"], + }), + }); + assert.equal(r2.status, 200); + await r2.text(); + const fwd2 = JSON.parse(bodies[1]!) as Record; + assert.equal(fwd2.previous_response_id, "resp_done_1", "the chain id is kept on the delta round"); + const texts2 = ((fwd2.input as unknown as { content?: unknown }[])).flatMap((x) => (Array.isArray(x.content) ? x.content.map((p) => (p as { text?: string }).text ?? "") : [String(x.content ?? "")])); + assert.ok(texts2.some((t) => t.includes("ack two")), "the new element is sent"); + assert.ok(!texts2.some((t) => t.includes("one")), "the stored world is NOT re-sent (the provider chain holds it)"); + assert.equal((fwd2.input as unknown[]).length, 1, "exactly one element on the wire (a re-seed would carry 4)"); + assert.ok(!JSON.stringify(fwd2).includes("You are bili"), "no developer re-injection on the delta round"); + + // Round 3: the host retries the SAME body as round 2 (upstream + // 200 lost in transit — the B4 retry class). Residual 1a: the + // delta comes out EMPTY — the no-op classification keeps the id, + // sends an EMPTY input, and injects NO dev message (the blind- + // response path). The plane must stay intact. + const r3 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ + model: "gpt-e2e", + stream: true, + previous_response_id: "resp_done_2", + input: [msg("user", "one"), msg("assistant", "ack one"), msg("user", "two"), msg("assistant", "ack two")], + biliMsgIds: ["r2m1", "r2m2", "r2m3", "r2m4"], + }), + }); + assert.equal(r3.status, 200); + await r3.text(); + const fwd3 = JSON.parse(bodies[2]!) as Record; + assert.equal(fwd3.previous_response_id, "resp_done_2", "the no-op round KEEPS the chain id (stripping it orphans the provider's context)"); + assert.deepEqual(fwd3.input, [], "the no-op round sends an EMPTY input (pure chain continuation)"); + assert.ok(!JSON.stringify(fwd3).includes("You are bili"), "no developer message on the no-op round (it would make a blind response)"); + + // Round 4: FORK — the host rewinds to round 1's response id while + // keeping the full current world plus one new element. The strict- + // extension proof would still pass (the plane is a strict prefix), + // but the chain from resp_done_1 may not hold the round-2/3 world + // if it was ever re-seeded — so the id mismatch drops the + // frontier and the round re-seeds the FULL view (id kept, never a + // loss). A 1-element delta here would be the loss edge. + const r4 = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ + model: "gpt-e2e", + stream: true, + previous_response_id: "resp_done_1", + input: [msg("user", "one"), msg("assistant", "ack one"), msg("user", "two"), msg("assistant", "ack two"), msg("assistant", "ack three")], + biliMsgIds: ["r2m1", "r2m2", "r2m3", "r2m4", "r2m5"], + }), + }); + assert.equal(r4.status, 200); + await r4.text(); + const fwd4 = JSON.parse(bodies[3]!) as Record; + assert.equal(fwd4.previous_response_id, "resp_done_1", "the forked id is forwarded (kept, not stripped)"); + const texts4 = ((fwd4.input as unknown as { content?: unknown }[])).flatMap((x) => (Array.isArray(x.content) ? x.content.map((p) => (p as { text?: string }).text ?? "") : [String(x.content ?? "")])); + assert.ok(texts4.some((t) => t.includes("one")), "the fork round RE-SEEDS the full view (the old chain is not provably holding the stored world)"); + assert.ok(texts4.some((t) => t.includes("ack three")), "the new element is in the re-seed view"); + const worldItems4 = (fwd4.input as { role?: string }[]).filter((x) => x.role !== "developer"); + assert.equal(worldItems4.length, 5, "all 5 world elements on the wire (a delta would have sent 1)"); + }, + { storeDir: path.join(hermetic, "r2-store"), env: { BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY: "1", ACP_KEEP_RESPONSE_ID: "1" } }, + ); +}); diff --git a/tests/issue5874-snapbase-p0.test.ts b/tests/issue5874-snapbase-p0.test.ts new file mode 100644 index 0000000000..d9d190cf73 --- /dev/null +++ b/tests/issue5874-snapbase-p0.test.ts @@ -0,0 +1,540 @@ +/** + * #5874 (P0): snapshot-table compression foundation — schema + commit + * primitive + red doors, zero behavior change. + * + * Pins the P0 acceptance tests from the proposal: + * T1 — the v12 upgrade is PURELY ADDITIVE: a v11 database opened by the + * new code gets the four tables (l1_world/snap_rows/snap_log/ + * snap_ckpt) and user_version 12, with every pre-existing table + * byte-identical; rollback (DROP the four + user_version 11) leaves + * the legacy dump byte-identical to the pre-upgrade one. + * T2 — the commit primitive is atomic: a batch with one invalid event + * leaves NO half-state (rows + log exactly as before); seq is + * strictly monotonic across folds (invariant ii); S4 mismatch on + * read kills the plane for the sid (R1, sticky); delete/compact + * never physically delete (invariant iv); post_json must be + * canonical core shape (invariant v). + * T6 — deleteSession cascades the four plane tables (no orphans). + * Plus: the two-level flag nesting (master ⊃ lane) as resolved by the + * store and by the config-file knob. + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { STORAGE_SCHEMA_VERSION, migrateSchema } from "../src/storage/schema.ts"; +import { UnifiedStore, sha256Hex } from "../src/storage/store.ts"; +import { + applySnapCommit, + readSnapView, + snapL1Record, + _resetSnapPlaneDeadForTests, + SnapPlaneDeadError, + type SnapEvent, +} from "../src/storage/snap.ts"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +// Hermeticity pin (same discipline as the other storage suites): a +// developer machine with storage settings in its real config must not +// leak into these tests. +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-5874-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); + +function tmpDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function rawDb(path: string) { + return openSqliteDatabase(path).db; +} + +/** Legacy dump of every PRE-#5874 table, byte-stable for one content: + * every row of every table, ordered, one sha256 over the whole thing. + * The upgrade and the rollback must both preserve it exactly. */ +const LEGACY_TABLES = [ + "sessions", + "content", + "messages", + "blocks", + "dead_refs", + "refs", + "checkpoints", + "ccr_entries", + "conversations", + "session_ext", + "kv_store", // #5901 review (2026-11-13): was missing — the byte-identity + // oracle skipped one of the twelve legacy tables (schema.ts kv_store) + "message_media", +] as const; + +function legacyDumpHash(db: ReturnType["db"]): string { + const parts: string[] = []; + for (const t of LEGACY_TABLES) { + const rows = db.prepare(`SELECT * FROM ${t}`).all() as unknown[]; + rows.sort((a, b) => JSON.stringify(a) < JSON.stringify(b) ? -1 : 1); + parts.push(`${t}:${sha256Hex(JSON.stringify(rows))}`); + } + return sha256Hex(parts.join("\u0000")); +} + +function tableCount(db: ReturnType["db"], table: string): number { + return (db.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get() as { n: number }).n; +} + +function hasTable(db: ReturnType["db"], table: string): boolean { + const row = db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) as + | { name: string } + | undefined; + return !!row; +} + +/** A minimal single-message legacy session (enough for T1/T6 to carry + * real rows through the ingest path). */ +function oneMsgSession(id: string, text: string): LegacySessionLike { + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(), + lastMessages: [{ id: "m1", role: "user", contentType: "text", text }], + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: undefined, + }; +} + +// --------------------------------------------------------------------------- +// T1 — upgrade is additive; rollback is byte-reversible. +// --------------------------------------------------------------------------- + +test("#5874 T1: v11 → v12 upgrade is purely additive (legacy tables byte-identical)", () => { + const dir = tmpDir("bili-5874-t1-"); + try { + const p = path.join(dir, "index.db"); + // 1. Build a database the NEW code considers v11: create it, seed + // real legacy rows, then drop the four plane tables and rewind + // the stamp to 11 (exactly the pre-upgrade on-disk state). + { + const s1 = UnifiedStore.open(p); + s1.ingestSession(planIngest(oneMsgSession("sess-t1", "hello t1"))); + s1.close(); + } + { + const db = rawDb(p); + db.exec("DROP TABLE IF EXISTS l1_world"); + db.exec("DROP TABLE IF EXISTS snap_rows"); + db.exec("DROP TABLE IF EXISTS snap_log"); + db.exec("DROP TABLE IF EXISTS snap_ckpt"); + db.exec("DROP INDEX IF EXISTS idx_l1_ledger"); + db.exec("PRAGMA user_version = 11"); + db.close(); + } + // 2. Reopen with the new code: the migration must bring the four + // tables and the stamp — and change NOTHING else. + const before = (() => { + const db = rawDb(p); + const h = legacyDumpHash(db); + db.close(); + return h; + })(); + const store = UnifiedStore.open(p); + { + const st = rawDb(p); + assert.equal((st.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); + for (const t of ["l1_world", "snap_rows", "snap_log", "snap_ckpt"]) assert.ok(hasTable(st, t), `missing table ${t}`); + st.close(); + } + const after = (() => { + const db = rawDb(p); + const h = legacyDumpHash(db); + db.close(); + return h; + })(); + assert.equal(after, before, "legacy tables must be byte-identical across the upgrade"); + { + const db = rawDb(p); + for (const t of ["snap_rows", "snap_log", "snap_ckpt"]) { + assert.equal(tableCount(db, t), 0, `plane table ${t} must start empty`); + } + db.close(); + } + store.close(); + // 3. Rollback: drop the four tables + rewind the stamp. The legacy + // dump must come back byte-identical. + { + const db = rawDb(p); + db.exec("DROP TABLE IF EXISTS l1_world"); + db.exec("DROP TABLE IF EXISTS snap_rows"); + db.exec("DROP TABLE IF EXISTS snap_log"); + db.exec("DROP TABLE IF EXISTS snap_ckpt"); + db.exec("DROP INDEX IF EXISTS idx_l1_ledger"); + db.exec("PRAGMA user_version = 11"); + db.close(); + } + const rolled = (() => { + const db = rawDb(p); + const h = legacyDumpHash(db); + db.close(); + return h; + })(); + assert.equal(rolled, before, "rollback must restore the pre-upgrade legacy dump exactly"); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test("#5874 T1b: a fresh install lands at v12 with the four tables", () => { + const dir = tmpDir("bili-5874-t1b-"); + try { + const store = UnifiedStore.open(path.join(dir, "index.db")); + try { + const db = rawDb(path.join(dir, "index.db")); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, STORAGE_SCHEMA_VERSION); + assert.equal(STORAGE_SCHEMA_VERSION, 12); + for (const t of ["l1_world", "snap_rows", "snap_log", "snap_ckpt"]) assert.ok(hasTable(db, t)); + db.close(); + } finally { + store.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +// --------------------------------------------------------------------------- +// T2 — commit primitive: atomicity, seq monotonicity, red doors. +// --------------------------------------------------------------------------- + +test("#5874 T1c: a sibling-branch database (v13, pf_* planes, no snap tables) coexists — probe-first DDL (field incident 2026-11-08)", () => { + const dir = tmpDir("bili-5874-t1c-"); + try { + const p = path.join(dir, "index.db"); + // 1. A real database with content. + { + const s1 = UnifiedStore.open(p); + s1.ingestSession(planIngest(oneMsgSession("sess-t1c", "hello coexistence"))); + s1.close(); + } + const before = (() => { + const db = rawDb(p); + const h = legacyDumpHash(db); + db.close(); + return h; + })(); + // 2. Simulate the field incident EXACTLY: the sibling branch (#5855) + // consumed v12+v13 for its own additive pf_* planes into this + // shared database and knows nothing about ours — its migration + // ran, ours never did (version already "ahead"). + { + const db = rawDb(p); + db.exec("DROP TABLE IF EXISTS l1_world"); + db.exec("DROP TABLE IF EXISTS snap_rows"); + db.exec("DROP TABLE IF EXISTS snap_log"); + db.exec("DROP TABLE IF EXISTS snap_ckpt"); + db.exec("DROP INDEX IF EXISTS idx_l1_ledger"); + db.exec("CREATE TABLE IF NOT EXISTS pf_rows (sid TEXT PRIMARY KEY, pos INTEGER NOT NULL, payload TEXT NOT NULL)"); + db.exec("CREATE TABLE IF NOT EXISTS pf_log (sid TEXT NOT NULL, seq INTEGER NOT NULL, payload TEXT NOT NULL, PRIMARY KEY (sid, seq))"); + db.exec("INSERT INTO pf_rows (sid, pos, payload) VALUES ('sess-t1c', 0, 'sibling data')"); + db.exec("PRAGMA user_version = 13"); + db.close(); + } + // 3. Reopen with our code: the probe must create the four snap + // tables WITHOUT touching the sibling's tables and WITHOUT + // downgrading the user_version (13 stays 13). + const store = UnifiedStore.open(p, undefined, { snapBase: { enabled: true, shadowWrite: true } }); + try { + const db = rawDb(p); + for (const t of ["l1_world", "snap_rows", "snap_log", "snap_ckpt"]) assert.ok(hasTable(db, t), `missing table ${t}`); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 13, "user_version must not be downgraded"); + assert.equal((db.prepare("SELECT payload FROM pf_rows WHERE sid = ?").get("sess-t1c") as { payload: string }).payload, "sibling data", "sibling DDL/rows must be untouched"); + assert.equal(legacyDumpHash(db), before, "legacy tables must be byte-identical across the coexistence upgrade"); + // 4. The plane is fully functional after the coexistence + // upgrade (the incident's fatal error was exactly this). + applySnapCommit(db, "sess-t1c", [ + { ev: "cast", pos: 0, coreIds: ["m1"], preJson: null, postJson: postJson("hello coexistence"), tokensEst: 1 }, + ]); + assert.equal(tableCount(db, "snap_rows"), 1); + assert.equal(tableCount(db, "snap_log"), 1); + db.close(); + } finally { + store.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +function postJson(text: string): string { + return JSON.stringify({ role: "user", content: text }); +} + +test("#5874 T2: the commit primitive is atomic (invalid event ⇒ no half-state)", () => { + const dir = tmpDir("bili-5874-t2-"); + try { + const p = path.join(dir, "index.db"); + const db = rawDb(p); + db.exec("PRAGMA foreign_keys = ON"); + migrateSchema(db); + // A valid first commit: two positions. + const ev1: SnapEvent[] = [ + { ev: "append", pos: 1, coreIds: ["m1"], preJson: null, postJson: postJson("a"), tokensEst: 3 }, + { ev: "append", pos: 2, coreIds: ["m2"], preJson: null, postJson: postJson("b"), tokensEst: 3 }, + ]; + const out1 = applySnapCommit(db, "s1", ev1); + assert.equal(tableCount(db, "snap_rows"), 2); + assert.equal(tableCount(db, "snap_log"), 2); + assert.equal(out1.tailSeq, 2); + // A batch with a good event followed by an invalid one (invariant v: + // post_json must be canonical core shape — this is wire bytes). + const bad: SnapEvent[] = [ + { ev: "append", pos: 3, coreIds: ["m3"], preJson: null, postJson: postJson("c"), tokensEst: 3 }, + { ev: "append", pos: 4, coreIds: ["m4"], preJson: null, postJson: "", tokensEst: 3 }, + ]; + assert.throws(() => applySnapCommit(db, "s1", bad), /not canonical core shape/); + assert.equal(tableCount(db, "snap_rows"), 2, "the good event in a failed batch must not land"); + assert.equal(tableCount(db, "snap_log"), 2, "the log must have no partial entry"); + // IN-TRANSACTION rollback (not just pre-txn validation): a VALID + // event followed by one that can only fail inside the txn — the R2 + // "no row to update" check lives in applyEvent (snap.ts), after the + // earlier append has already written its row + log entry. + const mixed: SnapEvent[] = [ + { ev: "append", pos: 3, coreIds: ["m3"], preJson: null, postJson: postJson("c"), tokensEst: 3 }, + { ev: "update", pos: 99, postJson: postJson("z") }, + ]; + assert.throws(() => applySnapCommit(db, "s1", mixed), /no row to update \(R2\)/); + assert.equal(tableCount(db, "snap_rows"), 2, "the earlier valid event of a failed batch must roll back"); + assert.equal(tableCount(db, "snap_log"), 2, "no partial log entry survives the in-txn rollback"); + // A structurally bad batch (duplicate pos) rolls back too. + const dup: SnapEvent[] = [ + { ev: "append", pos: 1, coreIds: ["x"], preJson: null, postJson: postJson("x"), tokensEst: 1 }, + { ev: "append", pos: 1, coreIds: ["y"], preJson: null, postJson: postJson("y"), tokensEst: 1 }, + ]; + assert.throws(() => applySnapCommit(db, "s1", dup), /duplicate pos/); + assert.equal(tableCount(db, "snap_rows"), 2); + assert.equal(tableCount(db, "snap_log"), 2); + // An update that cannot be addressed is a throw too (R2). + assert.throws( + () => applySnapCommit(db, "s1", [{ ev: "update", pos: 99, postJson: postJson("z") }]), + /no row to update \(R2\)/, + ); + db.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test("#5874 T2b: seq is strictly monotonic across folds (invariant ii)", () => { + const dir = tmpDir("bili-5874-t2b-"); + try { + const p = path.join(dir, "index.db"); + const db = rawDb(p); + migrateSchema(db); + const c1 = applySnapCommit(db, "s1", [ + { ev: "append", pos: 1, coreIds: ["m1"], preJson: null, postJson: postJson("a"), tokensEst: 1 }, + ], { fold: true }); + assert.equal(c1.folded, true); + assert.equal(c1.tailSeq, 1); + // After the fold the log is truncated; the next seq must still be + // strictly greater than the ckpt_seq. + assert.equal(tableCount(db, "snap_log"), 0, "the fold truncates the log up to the ckpt"); + const c2 = applySnapCommit(db, "s1", [ + { ev: "append", pos: 2, coreIds: ["m2"], preJson: null, postJson: postJson("b"), tokensEst: 1 }, + { ev: "append", pos: 3, coreIds: ["m3"], preJson: null, postJson: postJson("c"), tokensEst: 1 }, + ]); + assert.equal(c2.tailSeq, 3, "seq continues past the ckpt_seq"); + const view = readSnapView(db, "s1"); + assert.ok(view); + assert.equal(view.ckptSeq, 1); + assert.equal(view.tailSeq, 3); + assert.ok(view.verified, "the folded view verifies against the log tail anchor"); + db.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test("#5874 T2c: S4 mismatch kills the plane for the sid (R1, sticky)", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-t2c-"); + try { + const p = path.join(dir, "index.db"); + const db = rawDb(p); + migrateSchema(db); + applySnapCommit(db, "s1", [ + { ev: "append", pos: 1, coreIds: ["m1"], preJson: null, postJson: postJson("a"), tokensEst: 1 }, + ]); + // Tamper with a stored row: the recomputed aggregate can no longer + // match the log anchor. + db.prepare("UPDATE snap_rows SET post_json = ? WHERE sid = ? AND pos = 1").run( + JSON.stringify({ role: "user", content: "TAMPERED" }), + "s1", + ); + assert.throws(() => readSnapView(db, "s1"), (e: unknown) => e instanceof SnapPlaneDeadError); + // Sticky: a subsequent read fails fast even after the row is + // "repaired" — the plane stays closed for the process lifetime. + db.prepare("UPDATE snap_rows SET post_json = ? WHERE sid = ? AND pos = 1").run(postJson("a"), "s1"); + assert.throws(() => readSnapView(db, "s1"), (e: unknown) => e instanceof SnapPlaneDeadError); + // A different sid is unaffected. + applySnapCommit(db, "s2", [ + { ev: "append", pos: 1, coreIds: ["m1"], preJson: null, postJson: postJson("a"), tokensEst: 1 }, + ]); + assert.ok(readSnapView(db, "s2"), "R1 is per-sid, never global"); + db.close(); + } finally { + _resetSnapPlaneDeadForTests(); + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test("#5874 T2d: delete/compact flip active, never physically delete (invariant iv)", () => { + const dir = tmpDir("bili-5874-t2d-"); + try { + const p = path.join(dir, "index.db"); + const db = rawDb(p); + migrateSchema(db); + applySnapCommit(db, "s1", [ + { ev: "append", pos: 1, coreIds: ["m1"], preJson: null, postJson: postJson("a"), tokensEst: 1 }, + { ev: "append", pos: 2, coreIds: ["m2"], preJson: null, postJson: postJson("b"), tokensEst: 1 }, + { ev: "append", pos: 3, coreIds: ["m3"], preJson: null, postJson: postJson("c"), tokensEst: 1 }, + ]); + applySnapCommit(db, "s1", [ + { ev: "delete", pos: 1 }, + { ev: "compact", blockId: "blk-1", posRange: [2, 3] }, + ]); + assert.equal(tableCount(db, "snap_rows"), 3, "all three rows survive (both forms retained)"); + const rows = db.prepare("SELECT pos, active, cover_ref FROM snap_rows WHERE sid = ? ORDER BY pos").all("s1") as { + pos: number; + active: number; + cover_ref: string | null; + }[]; + assert.deepEqual( + rows.map((r) => [r.pos, r.active, r.cover_ref]), + [ + [1, 0, "deleted"], + [2, 0, "blk-1"], + [3, 0, "blk-1"], + ], + ); + // pre/post both survive the flip: + const r1 = db.prepare("SELECT pre_json, post_json FROM snap_rows WHERE sid = ? AND pos = 1").get("s1") as { + pre_json: string | null; + post_json: string | null; + }; + assert.equal(r1.pre_json, null); + assert.equal(r1.post_json, postJson("a")); + db.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +// --------------------------------------------------------------------------- +// T6 — deleteSession cascades the four plane tables (no orphans). +// --------------------------------------------------------------------------- + +test("#5874 T6: deleteSession cascades the plane tables", () => { + const dir = tmpDir("bili-5874-t6-"); + try { + const p = path.join(dir, "index.db"); + const store = UnifiedStore.open(p); + try { + store.ingestSession(planIngest(oneMsgSession("sess-t6", "hello t6"))); + const db = rawDb(p); + // Populate the plane for this sid the same way the shadow + // write hook does (outside the store's own transaction). + applySnapCommit(db, "sess-t6", [ + { ev: "append", pos: 1, coreIds: ["m1"], preJson: null, postJson: postJson("hello t6"), tokensEst: 2 }, + ]); + // L1 entry under the session's ledger id. + snapL1Record(db, "sess-t6", "sess-t6", "probe", Date.now()); + // Plane is default-off (no snapBase in the open above) — the + // manual applySnapCommit above is the only plane evidence; the + // cascade assertions below are the point of this test. + assert.ok(tableCount(db, "snap_rows") >= 1); + assert.ok(tableCount(db, "snap_log") >= 1); + assert.ok(tableCount(db, "l1_world") >= 1); + db.close(); + store.deleteSession("sess-t6"); + const db2 = rawDb(p); + assert.equal(tableCount(db2, "snap_rows"), 0); + assert.equal(tableCount(db2, "snap_log"), 0); + assert.equal(tableCount(db2, "snap_ckpt"), 0); + assert.equal(tableCount(db2, "l1_world"), 0); + db2.close(); + } finally { + store.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +// --------------------------------------------------------------------------- +// Flags — the two-level nesting (master ⊃ lane). +// --------------------------------------------------------------------------- + +test("#5874 flags: master ⊃ lane, resolved at the store", () => { + const dir = tmpDir("bili-5874-flags-"); + const stores: UnifiedStore[] = []; + try { + const mk = (p: string, snapBase: { enabled?: boolean; shadowWrite?: boolean } | undefined) => { + const s = UnifiedStore.open(p, undefined, { snapBase }); + stores.push(s); + return s["snapPlane"] as { enabled: boolean; shadowWrite: boolean }; + }; + assert.deepEqual(mk(path.join(dir, "a.db"), { enabled: true, shadowWrite: true }), { + enabled: true, + shadowWrite: true, + }); + assert.deepEqual(mk(path.join(dir, "b.db"), { enabled: true, shadowWrite: false }), { + enabled: true, + shadowWrite: false, + }); + // Lane without master: the plane must NOT turn on by itself. + assert.deepEqual(mk(path.join(dir, "c.db"), { enabled: false, shadowWrite: true }), { + enabled: false, + shadowWrite: false, + }); + assert.deepEqual(mk(path.join(dir, "d.db"), { enabled: true }), { enabled: true, shadowWrite: false }); + // Default OFF (#5874 §9): absent config ⇒ the plane stays fully off. + // (The 2026-11-08 soak pinned this expectation to ON; reverted with + // the store/admin defaults before the merge, 2026-11-13.) + assert.deepEqual(mk(path.join(dir, "e.db"), undefined), { enabled: false, shadowWrite: false }); + assert.deepEqual( + mk(path.join(dir, "f.db"), { enabled: false, shadowWrite: false }), + { enabled: false, shadowWrite: false }, + "an explicit config can still pin the plane off during the soak", + ); + } finally { + // Close before rmSync: Windows refuses to unlink open sqlite files (EBUSY). + for (const s of stores) s.close(); + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test("#5874 flags: the config-file knob (fileSnapBase) enforces the same nesting", async () => { + const { fileSnapBase } = await import("../src/knobs.ts"); + const cfg = path.join(hermetic, "snapbase-config.json"); + const prev = process.env.BILI_CONFIG_FILE; + try { + fs.writeFileSync(cfg, JSON.stringify({ storage: { snapBase: { enabled: true, shadowWrite: true } } })); + process.env.BILI_CONFIG_FILE = cfg; + assert.deepEqual(fileSnapBase(), { enabled: true, shadowWrite: true }); + fs.writeFileSync(cfg, JSON.stringify({ storage: { snapBase: { enabled: false, shadowWrite: true } } })); + assert.deepEqual(fileSnapBase(), { enabled: false, shadowWrite: false }, "lane alone must not arm the plane"); + fs.writeFileSync(cfg, JSON.stringify({ storage: { snapBase: { enabled: true } } })); + assert.deepEqual(fileSnapBase(), { enabled: true, shadowWrite: false }); + fs.writeFileSync(cfg, JSON.stringify({})); + assert.equal(fileSnapBase(), undefined, "absent section ⇒ undefined (defaults apply)"); + } finally { + process.env.BILI_CONFIG_FILE = prev; + } +}); diff --git a/tests/issue5874-snaplane-fastpath.test.ts b/tests/issue5874-snaplane-fastpath.test.ts new file mode 100644 index 0000000000..85949755e1 --- /dev/null +++ b/tests/issue5874-snaplane-fastpath.test.ts @@ -0,0 +1,171 @@ +/** + * #5874 P1 supplement — the steady-state fast path nail. Every other P1 + * test saves through planIngest(full) only, so snapFastAppendEvents never + * ran in CI. These drive the REAL watermark/delta machinery + * (UnifiedShadow.record, #3133): a steady-state append must ride the fast + * path (snap_log kind 'append', exactly n events), and any doubt must fall + * back to the final-state diff and still converge byte-for-byte. The + * rewind case also pins a GC/S4 regression: host-side row retirement + * releases the flipped rows physically, and that removal must re-anchor + * the plane in the same transaction — otherwise the next readSnapView + * trips R1 on a healthy plane and the sticky dead latch kills it forever. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { type LegacySessionLike } from "../src/storage/ingest.ts"; +import { readSnapView, _resetSnapPlaneDeadForTests } from "../src/storage/snap.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { UnifiedShadow } from "../src/storage/shadow.ts"; +import type { CoreMessage } from "acp-kernel"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-5874-fp-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); + +function tmpDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function coreOf(msgs: Array<{ id: string; text: string }>): CoreMessage[] { + return msgs.map((m, i) => ({ + id: m.id, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + contentType: "text" as const, + text: m.text, + })) as CoreMessage[]; +} + +/** One full-message turn of a legacy session (same shape as the P1 suite). */ +function session(id: string, msgs: Array<{ id: string; text: string }>): LegacySessionLike { + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(), + lastMessages: coreOf(msgs), + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: undefined, + }; +} + +/** Shadow with the plane armed on its OWN store (the production wiring: + * persist.ts passes fileSnapBase into maybeOpen); plus a plain assertion + * connection onto the same index.db for reading what the shadow wrote. */ +function openArmed(dir: string): { shadow: UnifiedShadow; store: UnifiedStore } { + const shadow = UnifiedShadow.maybeOpen(dir, { BILI_STORAGE_UNIFIED: "shadow" }, (r) => r as LegacySessionLike, undefined, undefined, undefined, { enabled: true, shadowWrite: true }); + assert.ok(shadow, "plane-armed shadow must open"); + const store = UnifiedStore.open(path.join(dir, "index.db")); + return { shadow, store }; +} + +function stats(shadow: UnifiedShadow): { deltaIngests: number } { + return (shadow as unknown as { stats: { deltaIngests: number } }).stats; +} + +function logKinds(store: UnifiedStore, sid: string): string[] { + const db = (store as unknown as { db: Parameters[0] }).db; + return (db.prepare("SELECT kind FROM snap_log WHERE sid = ? ORDER BY seq").all(sid) as Array<{ kind: string }>).map((r) => r.kind); +} + +function assertAbEqual(store: UnifiedStore, sid: string): void { + const db = (store as unknown as { db: Parameters[0] }).db; + const view = readSnapView(db, sid); + assert.ok(view, "plane must be initialized"); + assert.ok(view.verified, "plane checksum must verify"); + const live = (db.prepare("SELECT m.seq, c.bytes AS bytes FROM messages m JOIN content c ON c.hash = m.content_hash WHERE m.session_id = ? AND m.retired_at IS NULL ORDER BY m.seq").all(sid)) as Array<{ seq: number; bytes: Buffer | Uint8Array }>; + const active = view.rows.filter((r) => r.active === 1); + assert.equal(active.length, live.length, `active plane rows (${active.length}) must equal live legacy rows (${live.length})`); + for (const l of live) { + const row = active.find((r) => r.pos === l.seq); + assert.ok(row, `plane row for pos ${l.seq} missing`); + assert.equal(row.postJson, Buffer.from(l.bytes).toString("utf8"), `pos ${l.seq}: plane post_json must be byte-equal to the legacy content`); + } +} + +test("#5874 fast path: a steady-state delta append rides snapFastAppendEvents, not the diff", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-fp-steady-"); + try { + const { shadow, store } = openArmed(dir); + try { + // Save 1: five messages — no watermark yet, so the FULL path + // seeds the plane through the final-state diff ('cast'). + shadow.record(session("sess-fp", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }])); + assert.equal(stats(shadow).deltaIngests, 0, "first save has no watermark — full ingest"); + assert.deepEqual(logKinds(store, "sess-fp"), ["cast", "cast", "cast", "cast", "cast"], "seed round must derive casts, one per position"); + assertAbEqual(store, "sess-fp"); + + // Save 2: the classic B1 re-send + two new messages. The + // watermark names the old tail, so this rides the DELTA path — + // and with the plane aligned, the fast path must take it: + // exactly two 'append' events, nothing re-derived for the + // unchanged head (a silent diff fallback would log 'cast' here). + shadow.record(session("sess-fp", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }, { id: "m6", text: "six" }, { id: "m7", text: "seven" }])); + assert.equal(stats(shadow).deltaIngests, 1, "second save must take the delta path"); + assert.deepEqual(logKinds(store, "sess-fp"), ["cast", "cast", "cast", "cast", "cast", "append", "append"], "the append round must ride the fast path — 'append', not a diff re-cast"); + assertAbEqual(store, "sess-fp"); + + // Save 3: one more append — the fast path is the STEADY state, + // so it must keep holding turn after turn. + shadow.record(session("sess-fp", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }, { id: "m6", text: "six" }, { id: "m7", text: "seven" }, { id: "m8", text: "eight" }])); + assert.equal(stats(shadow).deltaIngests, 2, "third save stays on the delta path"); + assert.deepEqual(logKinds(store, "sess-fp"), ["cast", "cast", "cast", "cast", "cast", "append", "append", "append"], "steady state keeps riding the fast path"); + assertAbEqual(store, "sess-fp"); + } finally { + shadow.close(); + store.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +test("#5874 fast path fallback: a rewound tail voids the delta, the diff re-derives deletes", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-fp-rewind-"); + try { + const { shadow, store } = openArmed(dir); + try { + shadow.record(session("sess-fw", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }, { id: "m6", text: "six" }])); + assertAbEqual(store, "sess-fw"); + + // The host rewound (a shorter tail than the trusted watermark). + // The raw tail makes the plan the COMPLETE log (#3285), so the + // historian sweep tombstones the two dropped rows (#4390); the + // delta guard rejects the shorter tail and the plane's + // final-state diff must flip them inactive — the fast path's + // predicate cannot hold here, so a wrong 'append' would mean it + // ignored the retirement. + const rewound = [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }]; + shadow.record(session("sess-fw", rewound), { lastMessages: coreOf(rewound), lastMessagesFolded: false }); + assert.equal(stats(shadow).deltaIngests, 0, "a shorter tail cannot ride the delta"); + // The plane must be ALIVE and byte-converged on the rewound set: + // the diff flips the two retired positions inactive, GC releases + // them physically (invariant iv's valve), and the removal must + // re-anchor S4 — before that re-anchor existed, exactly this + // read threw SnapPlaneDeadError (R1) on a healthy plane. + assertAbEqual(store, "sess-fw"); + const db = (store as unknown as { db: Parameters[0] }).db; + const rowCount = (db.prepare("SELECT COUNT(*) AS n FROM snap_rows WHERE sid = ?").get("sess-fw") as { n: number }).n; + assert.equal(rowCount, 4, "the two released positions are physically gone; the four live ones remain"); + // One more steady turn after the rewind: the plane keeps serving + // (no sticky dead latch from the release round). + shadow.record(session("sess-fw", [...rewound, { id: "m9", text: "nine" }])); + assertAbEqual(store, "sess-fw"); + } finally { + shadow.close(); + store.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/tests/issue5874-snaplane-p1.test.ts b/tests/issue5874-snaplane-p1.test.ts new file mode 100644 index 0000000000..2343ac13bd --- /dev/null +++ b/tests/issue5874-snaplane-p1.test.ts @@ -0,0 +1,277 @@ +/** + * #5874 P1 — shadow-write plane: B1 trap nail, A/B shadow comparison, + * coverage mapping, bounded growth. The plane is shadow-only in this + * phase: it must agree with the legacy view on every save, and a + * mismatch or crash gap must degrade (or self-heal) without ever + * touching what a reader sees. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { readSnapView, snapL1Lookup, _resetSnapPlaneDeadForTests } from "../src/storage/snap.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-5874-p1-")); +const configPath = path.join(hermetic, "config.json"); +fs.writeFileSync(configPath, "{}"); +process.env.BILI_CONFIG_FILE = configPath; + +function tmpDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function openPlane(p: string): UnifiedStore { + return UnifiedStore.open(p, undefined, { snapBase: { enabled: true, shadowWrite: true } }); +} + +/** One full-message turn of a legacy session. */ +function session(id: string, msgs: Array<{ id: string; text: string; role?: "user" | "assistant" }>): LegacySessionLike { + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(), + lastMessages: msgs.map((m, i) => ({ + id: m.id, + role: (m.role ?? (i % 2 === 0 ? "user" : "assistant")) as "user" | "assistant", + contentType: "text" as const, + text: m.text, + })), + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: undefined, + }; +} + +/** The store's raw db handle (the plane reads share the store's own + * connection for a consistent snapshot). */ +function raw(store: UnifiedStore): Parameters[0] { + return (store as unknown as { db: Parameters[0] }).db; +} + +// --------------------------------------------------------------------------- +// T3 — the B1 trap nail: l1 identity is issued once, arms on the first echo, +// and survives restarts (the wire re-sends the same identity each turn). +// --------------------------------------------------------------------------- + +test("#5874 T3: B1 trap nail — l1 record is stable across restarts", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-t3-"); + const p = path.join(dir, "index.db"); + try { + for (let boot = 0; boot < 3; boot++) { + const store = openPlane(p); + // Every "request" re-sends the same full message history (B1): + // byte-identical wire, so the id space — and the l1 witness — + // is unchanged across boots. That is exactly the trap the + // record must survive without re-issuing. + store.ingestSession(planIngest(session("sess-t3", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }]))); + const db = raw(store); + const rec = snapL1Lookup(db, "sess-t3"); + assert.ok(rec, "l1 record must exist after the first save"); + assert.equal(rec.ledgerId, "sess-t3"); + if (boot === 0) { + assert.equal(rec.armedAt, null, "the first save ISSUES the identity; arming costs one round"); + } else { + assert.ok(rec.armedAt !== null, `l1 must be armed by save ${boot} (the wire re-sent the same identity)`); + } + const view = readSnapView(db, "sess-t3"); + assert.ok(view?.verified, "the plane must verify after every restart"); + assert.equal(view?.rows.filter((r) => r.active === 1).length, 3, "three live rows"); + store.close(); + } + // Exactly ONE l1 row — issued once, never re-issued. + const db = openSqliteDatabase(p).db; + const count = (db.prepare("SELECT COUNT(*) AS n FROM l1_world").get() as { n: number }).n; + assert.equal(count, 1, "l1 identity must not multiply across restarts"); + db.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +// --------------------------------------------------------------------------- +// T4 — A/B shadow comparison: after every save the plane's active rows must +// equal the legacy live set (pos-by-pos, byte-for-byte), including the +// crash-gap case (a save the plane never saw must self-heal on the next +// save, because the derivation is a final-state diff). +// --------------------------------------------------------------------------- + +test("#5874 T4: the shadow plane tracks the legacy view on every save (incl. crash gap)", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-t4-"); + const p = path.join(dir, "index.db"); + const store = openPlane(p); + try { + // Save 1: three messages. + store.ingestSession(planIngest(session("sess-t4", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }]))); + assertAbEqual(store, "sess-t4"); + // Save 2: the classic B1 re-send + one new message. + store.ingestSession(planIngest(session("sess-t4", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }]))); + assertAbEqual(store, "sess-t4"); + // CRASH GAP: the legacy write of save 3 lands, but the plane never + // saw it (simulate a crash between the two writes). Until the + // plane's next save, the view still agrees with what it last + // committed (4 rows) — and the next save (the wire re-sends the + // same history, B1) must heal the gap: the derivation is a + // final-state diff. + { + const raw = openSqliteDatabase(p).db; + const plan = planIngest(session("sess-t4", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }])); + const s2 = UnifiedStore.open(p, undefined, { snapBase: { enabled: false, shadowWrite: false } }); // plane explicitly OFF: the crash-gap store must not self-heal + s2.ingestSession(plan); + s2.close(); + raw.close(); + } + { const db = raw(store); + const stale = readSnapView(db, "sess-t4")!.rows.filter((r) => r.active === 1).length; + assert.equal(stale, 4, "before the next save the plane still shows its last committed state"); } + store.ingestSession(planIngest(session("sess-t4", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }]))); + assertAbEqual(store, "sess-t4"); // healed by the re-send + // Save 4: same, still equal. + store.ingestSession(planIngest(session("sess-t4", [{ id: "m1", text: "one" }, { id: "m2", text: "two" }, { id: "m3", text: "three" }, { id: "m4", text: "four" }, { id: "m5", text: "five" }, { id: "m6", text: "six" }]))); + assertAbEqual(store, "sess-t4"); + } finally { + store.close(); + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +function assertAbEqual(store: UnifiedStore, sid: string): void { + const db = raw(store); + const view = readSnapView(db, sid); + assert.ok(view, "plane must be initialized after the first save"); + assert.ok(view.verified, "plane checksum must verify"); + const live = (db.prepare("SELECT m.seq, c.bytes AS bytes FROM messages m JOIN content c ON c.hash = m.content_hash WHERE m.session_id = ? AND m.retired_at IS NULL ORDER BY m.seq").all(sid)) as Array<{ seq: number; bytes: Buffer | Uint8Array }>; + const active = view.rows.filter((r) => r.active === 1); + assert.equal(active.length, live.length, `active plane rows (${active.length}) must equal live legacy rows (${live.length})`); + for (const l of live) { + const row = active.find((r) => r.pos === l.seq); + assert.ok(row, `plane row for pos ${l.seq} missing`); + assert.equal(row.postJson, Buffer.from(l.bytes).toString("utf8"), `pos ${l.seq}: plane post_json must be byte-equal to the legacy content`); + } +} + +// --------------------------------------------------------------------------- +// T7 — the fold coverage mapping: the compacted-away region must be named +// in the checkpoint's meta (summary → covered pos range), and the rows +// stay on disk (double form) until GC. +// --------------------------------------------------------------------------- + +test("#5874 T7: a block's covered range lands in the checkpoint coverage map", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-t7-"); + const p = path.join(dir, "index.db"); + const store = openPlane(p); + try { + const msgs = Array.from({ length: 12 }, (_, i) => ({ id: `m${i + 1}`, text: `message ${i + 1}` })); + // A compression block (through the real ingest path, like a host + // that re-sends covered history — B1) covers positions 1..8. + const byRaw: Record = {}; + const byRef: Record = {}; + msgs.forEach((m, i) => { + const ref = `m${String(i + 1).padStart(5, "0")}`; + byRaw[m.id] = ref; + byRef[ref] = m.id; + }); + const blk = { + blockId: "blk-c1", + runId: 1, + createdAt: 1_000, + tier: 1, + active: true, + summary: "folded 1-8", + kind: "fold", + reason: "test", + effectiveMessageIds: msgs.slice(0, 8).map((m) => m.id), + directMessageIds: [] as string[], + }; + const sess: LegacySessionLike = { + id: "sess-t7", + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [blk as never], nextBlockId: 2, nextRunId: 2, messageRefs: { byRaw, byRef } }, + blockContents: new Map(), + lastMessages: msgs.map((m, i) => ({ id: m.id, role: i % 2 === 0 ? "user" : "assistant", contentType: "text", text: m.text })), + lastMessagesFolded: false, + pluginSnapshot: undefined, + contentStore: undefined, + }; + // One save: the derivation flips the covered rows compact and the + // compact-triggered fold must name the range in the ckpt meta. + store.ingestSession(planIngest(sess)); + const db = raw(store); + const ckpt = db.prepare("SELECT meta_json FROM snap_ckpt WHERE sid = 'sess-t7'").get() as { meta_json: string }; + assert.ok(ckpt, "a checkpoint must exist after the compact fold"); + const meta = JSON.parse(ckpt.meta_json) as { coverage: Array<[string, number, number]>; s4: string; kstate: string | null }; + // seqs are 0-based (planIngest's first-seen order), so m1..m8 = pos 0..7. + assert.deepEqual(meta.coverage, [["blk-c1", 0, 7]], "coverage map must name the block's exact range"); + assert.ok(meta.s4, "the S4 aggregate lives in the ckpt"); + // The covered rows are STILL on disk (invariant iv) but inactive. + const rows = db.prepare("SELECT pos, active, cover_ref FROM snap_rows WHERE sid = 'sess-t7' ORDER BY pos").all() as Array<{ pos: number; active: number; cover_ref: string | null }>; + for (const r of rows.slice(0, 8)) { + assert.equal(r.active, 0, `pos ${r.pos} must be compacted away`); + assert.equal(r.cover_ref, "blk-c1"); + } + for (const r of rows.slice(8)) { + assert.equal(r.active, 1, `pos ${r.pos} must survive`); + } + // The read-side view verifies even though 8 rows are covered. + const view = readSnapView(db, "sess-t7"); + assert.ok(view?.verified); + } finally { + store.close(); + fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +// --------------------------------------------------------------------------- +// T9 — bounded growth: N full saves (B1 re-sends) must not grow the plane +// unboundedly — log depth folds at the threshold, rows stay O(pos). +// --------------------------------------------------------------------------- + +test("#5874 T9: growth is bounded under continuous B1 re-sends", () => { + _resetSnapPlaneDeadForTests(); + const dir = tmpDir("bili-5874-t9-"); + const p = path.join(dir, "index.db"); + const store = openPlane(p); + // N MUST cross SNAP_FOLD_DEPTH (128) — at 40 saves the fold path was + // structurally unreachable (max depth ~51) and the ≤256 bound was + // vacuous; 160 forces several folds. + const N = 160; + try { + const base = Array.from({ length: 12 }, (_, i) => ({ id: `m${i + 1}`, text: `message ${i + 1}` })); + for (let i = 0; i < N; i++) { + const msgs = i === 0 ? base : [...base, { id: `x${i}`, text: `extra ${i}` }]; + store.ingestSession(planIngest(session("sess-t9", msgs))); + const db = raw(store); + const depth = (db.prepare("SELECT COUNT(*) AS n FROM snap_log WHERE sid = 'sess-t9'").get() as { n: number }).n; + assert.ok(depth <= 256, `log depth ${depth} must stay bounded (got > 256)`); + } + const db = raw(store); + // The fold actually ran: a ckpt exists past seq 0 and the log is + // shorter than the total number of events that passed through. + const ckpt = (db.prepare("SELECT ckpt_seq FROM snap_ckpt WHERE sid = 'sess-t9'").get() as { ckpt_seq: number } | undefined)?.ckpt_seq ?? 0; + assert.ok(ckpt >= 1, `a fold must have checkpointed by save ${N} (ckpt_seq ${ckpt})`); + const rows = (db.prepare("SELECT COUNT(*) AS n FROM snap_rows WHERE sid = 'sess-t9'").get() as { n: number }).n; + assert.ok(rows <= 12 + N, `rows (${rows}) must stay O(pos), not O(saves)`); + const view = readSnapView(db, "sess-t9"); + assert.ok(view?.verified, "the plane must still verify after all folds"); + } finally { + store.close(); + fs.rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/tests/issue5874-snaplane-p2.test.ts b/tests/issue5874-snaplane-p2.test.ts new file mode 100644 index 0000000000..dc5f8d3a40 --- /dev/null +++ b/tests/issue5874-snaplane-p2.test.ts @@ -0,0 +1,576 @@ +/** + * #5874 P2 — mapEgress: the snap plane is the DIRECT outbound egress + * authority (owner ruling 2026-11-11 — no per-round diff against the legacy + * view; the legacy bytes remain only the one-round crash-grade fallback). + * + * Model under test: + * WRITE (incremental, event-level): the save carries the lane's tagged + * processTurn output (planeMessages = the kernel's OWN tagged view). The + * plane persists exactly those bytes (invariant v) — surviving rows keep + * their tagged form, the pinned first user stays active, covered ranges + * are flipped covered (rows retained for the pair-safe walk, materialized + * from the legacy CAS only when the plane never recorded them), each + * active block's rendered anchor is stored at a negative pos, and rows the + * host no longer presents are deleted. No diff, no fuzzy matching: a view + * entry is addressed to its ledger row by message id (steady state), with + * a tag-stripped identity fallback for the rare kernel remint edge. + * EGRESS (direct): the outbound view is assembled from the plane rows + * alone — the non-anchor rows in pos order (covered ones included, for + * pairing parity) walked with EXACT kernel prune/rebuild semantics + * (earliest-covered anchor → pair-safe → pin clamp; alias-coverage / + * stale-summary skips; pinned first user served), + * each active block's stored anchor row re-inserted at its pair-safe + * index — then run through the lane's transform (identical to what the + * lane ran on the legacy view). Byte-identical to the legacy egress by + * construction, because the plane rows ARE the transform input. + * + * Unit scope: + * W* write side — seed+fold, steady-state no-op, tail append, host + * removal, unaddressable-entry death. + * E* egress assembly — kernel-view parity, no-fold passthrough, + * no-plane / row-unparsable degrade, #4311 visible-compress-call + * skip, remint coverage via row alias (core_ids), renderAnchor fallback. + * G* the gate — config off, cohort, plane miss, plane dead, and the + * plane being served directly regardless of the legacy bytes. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { SUMMARY_HEADER, type CompressionState } from "acp-kernel"; +import type { BiliMessage } from "acp-kernel/wire"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { _resetSnapPlaneDeadForTests, isSnapPlaneDead, type SnapView } from "../src/storage/snap.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { _setStoreForTest, SessionStore } from "../src/persist.ts"; +import type { UnifiedShadow } from "../src/storage/shadow.ts"; +import { + applySnapEgress, + buildSnapEgressView, + snapEgressStatusWire, + _resetSnapEgressForTests, + type SnapEgressProtocol, +} from "../src/server/snap-egress.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-5874-p2-")); +const configPath = path.join(hermetic, "config.json"); +fs.writeFileSync(configPath, "{}"); +process.env.BILI_CONFIG_FILE = configPath; + +function setConfig(v: unknown): void { + fs.writeFileSync(configPath, JSON.stringify(v)); +} + +function tmpDir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function openPlane(p: string): UnifiedStore { + return UnifiedStore.open(p, undefined, { snapBase: { enabled: true, shadowWrite: true } }); +} + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +const refOf = (i: number): string => `m${String(i + 1).padStart(5, "0")}`; +const tagOf = (i: number): string => `${refOf(i)}\n`; +const roleOf = (i: number): "user" | "assistant" => (i % 2 === 0 ? "user" : "assistant"); + +/** Legacy (client-resent, untagged) core message i (1-based id). */ +function plainMsg(i: number): { id: string; role: "user" | "assistant"; contentType: "text"; text: string } { + return { id: `m${i}`, role: roleOf(i - 1), contentType: "text", text: `message ${i}` }; +} + +/** Tagged view entry for message i (what the kernel's processTurn output + * carries — render-refs prepends the ref tag to the text). */ +function taggedMsg(i: number): BiliMessage { + const m = plainMsg(i); + return { ...m, text: tagOf(i - 1) + m.text } as BiliMessage; +} + +const ANCHOR_ID = "acp_summary_blk-c1"; +const ANCHOR_TEXT = `${SUMMARY_HEADER} — fold topic\nfolded 1-8`; +function anchorMsg(): BiliMessage { + return { id: ANCHOR_ID, role: "system", contentType: "text", text: ANCHOR_TEXT } as BiliMessage; +} + +function session(id: string, msgs: Array<{ id: string; text: string; role?: "user" | "assistant" }>): LegacySessionLike { + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: new Map(), + lastMessages: msgs.map((m, i) => ({ + id: m.id, + role: (m.role ?? (i % 2 === 0 ? "user" : "assistant")) as "user" | "assistant", + contentType: "text" as const, + text: m.text, + })), + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: undefined, + }; +} + +/** A folded session: `count` messages, block covers msgs[from..to). */ +function foldedSession( + sid: string, + opts: { from?: number; to?: number; count?: number } = {}, +): { sess: LegacySessionLike; blk: Record; msgs: Array<{ id: string; text: string }> } { + const { from = 0, to = 8, count = 12 } = opts; + const msgs = Array.from({ length: count }, (_, i) => ({ id: `m${i + 1}`, text: `message ${i + 1}` })); + const byRaw: Record = {}; + const byRef: Record = {}; + msgs.forEach((m, i) => { + byRaw[m.id] = refOf(i); + byRef[refOf(i)] = m.id; + }); + const blk = { + blockId: "blk-c1", + runId: 1, + createdAt: 1_000, + tier: 1, + active: true, + summary: "folded 1-8", + topic: "fold topic", + kind: "fold", + reason: "test", + effectiveMessageIds: msgs.slice(from, to).map((m) => m.id), + directMessageIds: [] as string[], + }; + const sess = session(sid, msgs); + sess.state = { blocks: [blk as never], nextBlockId: 2, nextRunId: 2, messageRefs: { byRaw, byRef } }; + return { sess, blk, msgs }; +} + +/** The kernel's tagged processTurn output for the folded session: pinned + * first user, the rendered anchor, then the surviving tail. */ +function foldedView(count = 12): BiliMessage[] { + return [taggedMsg(1), anchorMsg(), ...Array.from({ length: count - 8 }, (_, i) => taggedMsg(9 + i))]; +} + +function stateFor(blk: Record): CompressionState { + return { blocks: [blk as never], messageRefs: { byRaw: {}, byRef: {} } } as unknown as CompressionState; +} + +const NO_COMPRESS = (name: string): boolean => name === "compress"; + +function egressRows(store: UnifiedStore, sid: string): SnapView["rows"] { + const view = store.snapEgressView(sid); + assert.ok(view, "the plane must have rows after ingest"); + return view.rows; +} + +/** G-tests: the egress reads the plane through getStore() (a SessionStore), + * so register one whose sidecar wraps the store under test. SessionStore's + * replicasAvailable() reads shadow.stats.disabled and the read path only + * calls snapEgressView, so a minimal UnifiedShadow-shaped wrapper over the + * raw UnifiedStore is sufficient (private ctor — no public seam exists). */ +function registerStore(store: UnifiedStore): void { + const shadow = { + stats: { disabled: false }, + snapEgressView: (sid: string) => store.snapEgressView(sid), + } as unknown as UnifiedShadow; + _setStoreForTest(new SessionStore({ enabled: true, shadow, log: () => {} })); +} + +test.beforeEach(() => { + _resetSnapEgressForTests(); + _resetSnapPlaneDeadForTests(); + setConfig({}); +}); + +// --------------------------------------------------------------------------- +// W* — write side: incremental event-level persistence of the tagged view +// --------------------------------------------------------------------------- + +test("W1 seed+fold: surviving rows tagged, covered flipped, anchor at negative pos, A/B green", () => { + const store = openPlane(path.join(tmpDir("bili-5874-w1-"), "index.db")); + const sid = "w1"; + const { sess, msgs } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + // The fold is committed and the structural A/B check did NOT kill the + // plane (pin row stays active inside the covered interval). + assert.equal(isSnapPlaneDead(sid), false, "fold must pass the structural A/B check"); + + const rows = egressRows(store, sid); + const byPos = new Map(rows.map((r) => [r.pos, r])); + assert.equal(rows.length, 13, "12 ledger rows + 1 anchor row"); + // Anchor row at the most-negative pos, active, kernel's own bytes. + assert.equal(byPos.get(-1)!.active, 1); + assert.equal(byPos.get(-1)!.postJson, JSON.stringify(anchorMsg())); + assert.ok(byPos.get(-1)!.coreIds.includes(ANCHOR_ID)); + // Pinned first user: active, tagged bytes. + assert.equal(byPos.get(0)!.active, 1); + assert.equal(byPos.get(0)!.postJson, JSON.stringify(taggedMsg(1))); + // Covered rows: retained (pair-safe walk needs them), flipped covered, + // legacy bytes (they were never in the tagged view). + for (let i = 2; i <= 8; i++) { + const r = byPos.get(i - 1)!; + assert.equal(r.active, 0, `row ${i - 1} covered`); + assert.equal(r.coverRef, "blk-c1"); + assert.equal(r.postJson, JSON.stringify(plainMsg(i))); + } + // Surviving tail: active, tagged bytes. + for (let i = 9; i <= 12; i++) { + const r = byPos.get(i - 1)!; + assert.equal(r.active, 1); + assert.equal(r.postJson, JSON.stringify(taggedMsg(i))); + } +}); + +test("W2 steady state: an identical tagged save is a no-op (no rewrites, no new fold point)", () => { + const store = openPlane(path.join(tmpDir("bili-5874-w2-"), "index.db")); + const sid = "w2"; + const { sess } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + const snap1 = JSON.stringify(egressRows(store, sid)); + const stats1 = store.sessionStorageStats(sid).rows; + assert.equal(stats1.snapRows, 13, "seed must materialize the plane"); + assert.ok(stats1.snapCkpts >= 1, "the seed fold must checkpoint"); + + const sess2 = foldedSession(sid).sess; + sess2.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess2)); + const stats2 = store.sessionStorageStats(sid).rows; + assert.equal(JSON.stringify(egressRows(store, sid)), snap1, "steady-state save must not rewrite rows"); + assert.equal(stats2.snapCkpts, stats1.snapCkpts, "a no-op round must not add a fold point"); +}); + +test("W3 tail append: new tail rows are cast tagged, nothing else changes", () => { + const store = openPlane(path.join(tmpDir("bili-5874-w3-"), "index.db")); + const sid = "w3"; + const { sess } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + // Next turn: two more messages, no new fold. + const { sess: sess2 } = foldedSession(sid, { count: 14 }); + sess2.lastPlaneMessages = foldedView(14); + store.ingestSession(planIngest(sess2)); + + const rows = egressRows(store, sid); + const byPos = new Map(rows.map((r) => [r.pos, r])); + assert.equal(byPos.get(12)!.active, 1); + assert.equal(byPos.get(12)!.postJson, JSON.stringify(taggedMsg(13))); + assert.equal(byPos.get(13)!.active, 1); + assert.equal(byPos.get(13)!.postJson, JSON.stringify(taggedMsg(14))); + // The earlier rows are untouched. + assert.equal(byPos.get(0)!.postJson, JSON.stringify(taggedMsg(1))); + assert.equal(byPos.get(-1)!.postJson, JSON.stringify(anchorMsg())); +}); + +test("W4 host removal: an active row the view no longer presents is deleted", () => { + const store = openPlane(path.join(tmpDir("bili-5874-w4-"), "index.db")); + const sid = "w4"; + const { sess } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + // Host edit: m11 drops out of the transcript (and the tagged view). + const { sess: sess2 } = foldedSession(sid); + sess2.lastMessages = sess2.lastMessages!.filter((m) => m.id !== "m11"); + sess2.lastPlaneMessages = foldedView().filter((m) => m.id !== "m11"); + store.ingestSession(planIngest(sess2)); + + const byPos = new Map(egressRows(store, sid).map((r) => [r.pos, r])); + // Host-removed rows are physically GC'd (cover_ref='deleted' is the GC + // marker; only BLOCK-covered rows are retained for the pair-safe walk). + assert.equal(byPos.get(10), undefined, "m11 row GC'd"); + // Everything else intact. + assert.equal(byPos.get(0)!.active, 1); + assert.equal(byPos.get(9)!.active, 1, "m10 still active"); + assert.equal(byPos.get(11)!.active, 1, "m12 still active"); +}); + +test("W5 unaddressable view entry: the save latches the sid dead, legacy keeps the round", () => { + const store = openPlane(path.join(tmpDir("bili-5874-w5-"), "index.db")); + const sid = "w5"; + const { sess } = foldedSession(sid); + // A view entry with no live ledger row: not by id, not by identity. + sess.lastPlaneMessages = [ + ...foldedView(), + { id: "ghost-msg", role: "assistant", contentType: "text", text: "a content no legacy row carries" } as BiliMessage, + ]; + store.ingestSession(planIngest(sess)); + + assert.equal(isSnapPlaneDead(sid), true, "unaddressable entry must latch the sid dead"); +}); + +// --------------------------------------------------------------------------- +// E* — egress assembly: kernel prune/rebuild parity over the plane rows +// --------------------------------------------------------------------------- + +test("E1 folded plane: egress output is byte-identical to the kernel's tagged view", () => { + const store = openPlane(path.join(tmpDir("bili-5874-e1-"), "index.db")); + const sid = "e1"; + const { sess, blk } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + const out = buildSnapEgressView(store.snapEgressView(sid)!, stateFor(blk), NO_COMPRESS); + assert.equal(out.ok, true); + if (!out.ok) return; + assert.deepEqual( + out.msgs.map((m) => JSON.stringify(m)), + foldedView().map((m) => JSON.stringify(m)), + "the egress view must equal the kernel's tagged view, byte for byte", + ); +}); + +test("E2 no fold: the plane rows are served verbatim in pos order", () => { + const store = openPlane(path.join(tmpDir("bili-5874-e2-"), "index.db")); + const sid = "e2"; + const sess = session(sid, Array.from({ length: 12 }, (_, i) => ({ id: `m${i + 1}`, text: `message ${i + 1}`, role: roleOf(i) }))); + const view = Array.from({ length: 12 }, (_, i) => taggedMsg(i + 1)); + sess.lastPlaneMessages = view; + store.ingestSession(planIngest(sess)); + + const out = buildSnapEgressView(store.snapEgressView(sid)!, { blocks: [] } as unknown as CompressionState, NO_COMPRESS); + assert.equal(out.ok, true); + if (!out.ok) return; + assert.deepEqual( + out.msgs.map((m) => JSON.stringify(m)), + view.map((m) => JSON.stringify(m)), + ); +}); + +test("E3 empty plane: no-plane degrade", () => { + const out = buildSnapEgressView({ rows: [] } as unknown as SnapView, { blocks: [] } as unknown as CompressionState, NO_COMPRESS); + assert.deepEqual(out, { ok: false, reason: "no-plane" }); +}); + +test("E4 corrupt row: row-unparsable degrade", () => { + const view = { + rows: [{ pos: 0, coreIds: ["m1"], preJson: null, postJson: "{not-json", tokensEst: 1, active: 1, coverRef: null, rowSum: 1 }], + } as unknown as SnapView; + const out = buildSnapEgressView(view, { blocks: [] } as unknown as CompressionState, NO_COMPRESS); + assert.deepEqual(out, { ok: false, reason: "row-unparsable" }); +}); + +test("E5 #4311: a visible compress tool-call suppresses its block's summary render", () => { + // The kernel's own tagged output for this fold: the block covers m1..m3, + // the first-user pin m1 survives, the visible compress call m4 (the wire + // evidence of the fold) and the live tail m5 remain — and #4311 + // suppressed the separate summary render, so NO anchor entry. The plane + // persists exactly that (invariant v): m1/m4/m5 active, m2/m3 compacted. + const compressCall: BiliMessage = { + id: "m4", + role: "assistant", + contentType: "tool-call", + text: "", + toolCallId: "tc-1", + toolName: "compress", + } as BiliMessage; + const view = { + rows: [ + { pos: 0, coreIds: ["m1"], preJson: null, postJson: JSON.stringify(taggedMsg(1)), tokensEst: 1, active: 1, coverRef: null, rowSum: 1 }, + { pos: 1, coreIds: ["m2"], preJson: null, postJson: JSON.stringify(plainMsg(2)), tokensEst: 1, active: 0, coverRef: "blk-c1", rowSum: 1 }, + { pos: 2, coreIds: ["m3"], preJson: null, postJson: JSON.stringify(plainMsg(3)), tokensEst: 1, active: 0, coverRef: "blk-c1", rowSum: 1 }, + { pos: 3, coreIds: ["m4"], preJson: null, postJson: JSON.stringify(compressCall), tokensEst: 1, active: 1, coverRef: null, rowSum: 1 }, + { pos: 4, coreIds: ["m5"], preJson: null, postJson: JSON.stringify(taggedMsg(5)), tokensEst: 1, active: 1, coverRef: null, rowSum: 1 }, + ], + } as unknown as SnapView; + const blk = { + blockId: "blk-c1", + active: true, + summary: "folded", + effectiveMessageIds: ["m1", "m2", "m3"], + directMessageIds: [], + compressCallId: "tc-1", + }; + const out = buildSnapEgressView(view, stateFor(blk), NO_COMPRESS); + assert.equal(out.ok, true); + if (!out.ok) return; + // The visible call suppresses the block's separate summary render — + // the plane serves pin + call + tail verbatim, no anchor. + assert.deepEqual( + out.msgs.map((m) => m.id), + ["m1", "m4", "m5"], + ); +}); + +test("E6 remint: a reminted id is covered through its ALIAS (core_ids keeps the old id)", () => { + const m1 = { id: "m1", role: "user" as const, contentType: "text" as const, text: "message 1" }; + // Post-remint plane: the block's effIds still carry the OLD ids (m2..m4 + // — or the reconcile re-keyed them to the client's re-minted forms), but + // the covered run is read from the rows' OWN cover_ref, so no id + // resolution is involved. m2 was re-appended under its reminted id m2_k2 + // (core_ids keeps the old alias); all three covered rows are inactive. + const m2k = { id: "m2_k2", role: "assistant" as const, contentType: "text" as const, text: "message 2" }; + const m3 = { id: "m3", role: "assistant" as const, contentType: "text" as const, text: "message 3" }; + const m4 = { id: "m4", role: "assistant" as const, contentType: "text" as const, text: "message 4" }; + const view = { + rows: [ + { pos: 0, coreIds: ["m1"], preJson: null, postJson: JSON.stringify(m1), tokensEst: 1, active: 1, coverRef: null, rowSum: 1 }, + { pos: 1, coreIds: ["m2_k2", "m2"], preJson: null, postJson: JSON.stringify(m2k), tokensEst: 1, active: 0, coverRef: "blk-c1", rowSum: 1 }, + { pos: 2, coreIds: ["m3"], preJson: null, postJson: JSON.stringify(m3), tokensEst: 1, active: 0, coverRef: "blk-c1", rowSum: 1 }, + { pos: 3, coreIds: ["m4"], preJson: null, postJson: JSON.stringify(m4), tokensEst: 1, active: 0, coverRef: "blk-c1", rowSum: 1 }, + ], + } as unknown as SnapView; + // The block's ids do NOT resolve against any active row — coverage must + // come from cover_ref alone, and the anchor takes the first row above + // the covered run (after the pinned first user). + const blk = { + blockId: "blk-c1", + active: true, + summary: "folded", + effectiveMessageIds: ["m2", "m3", "m4"], + directMessageIds: [], + startSeq: -1, + endSeq: -1, + }; + const out = buildSnapEgressView(view, stateFor(blk), NO_COMPRESS); + assert.equal(out.ok, true); + if (!out.ok) return; + assert.deepEqual( + out.msgs.map((m) => m.id), + ["m1", ANCHOR_ID], + "pin first, anchor after the covered run, covered rows skipped", + ); +}); + +test("E7 anchor row missing from the plane: renderAnchor fallback is byte-parity", () => { + const store = openPlane(path.join(tmpDir("bili-5874-e7-"), "index.db")); + const sid = "e7"; + const { sess, blk } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + // Drop the anchor row from the egress view (degenerate plane). + const full = store.snapEgressView(sid)!; + const degenerate = { ...full, rows: full.rows.filter((r) => r.pos >= 0) }; + const out = buildSnapEgressView(degenerate, stateFor(blk), NO_COMPRESS); + assert.equal(out.ok, true); + if (!out.ok) return; + const expected = foldedView().map((m) => JSON.stringify(m)); + assert.deepEqual(out.msgs.map((m) => JSON.stringify(m)), expected, "the fallback render must match the stored anchor bytes"); +}); + +// --------------------------------------------------------------------------- +// G* — the gate: direct authority with the one-round crash-grade fallback +// --------------------------------------------------------------------------- + +test("G1 config off: the legacy bytes are served", () => { + const dir = tmpDir("bili-5874-g1-"); + const store = openPlane(path.join(dir, "index.db")); + registerStore(store); + const sid = "g1"; + const { sess, blk } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + setConfig({ storage: { snapBase: { egress: false } } }); + const legacy = [taggedMsg(99)]; + const out = applySnapEgress(stateFor(blk), legacy, { + sessionId: sid, + protocol: "openai", + log: (_l: string, _m: string) => {}, + transform: (m: BiliMessage[]) => m, + }); + assert.equal(out, legacy, "gate off must return the legacy array verbatim"); + assert.equal(snapEgressStatusWire().plane, 0); +}); + +test("G2 cohort: a protocol outside egressCohorts serves the legacy bytes", () => { + const dir = tmpDir("bili-5874-g2-"); + const store = openPlane(path.join(dir, "index.db")); + registerStore(store); + const sid = "g2"; + const { sess, blk } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + + // egress: true is explicit now (opt-in since the 2026-11-13 soak revert); + // this test exercises COHORT exclusion, not the master opt-in gate. + setConfig({ storage: { snapBase: { egress: true, egressCohorts: ["anthropic"] } } }); + const legacy = [taggedMsg(99)]; + const out = applySnapEgress(stateFor(blk), legacy, { + sessionId: sid, + protocol: "openai", + log: (_l: string, _m: string) => {}, + transform: (m: BiliMessage[]) => m, + }); + assert.equal(out, legacy); + assert.equal(snapEgressStatusWire().plane, 0); +}); + +test("G3 plane miss: an sid the plane never recorded serves the legacy bytes", () => { + const dir = tmpDir("bili-5874-g3-"); + const store = openPlane(path.join(dir, "index.db")); + registerStore(store); + setConfig({ storage: { snapBase: { egress: true } } }); + const legacy = [taggedMsg(99)]; + const out = applySnapEgress({ blocks: [] } as unknown as CompressionState, legacy, { + sessionId: "never-ingested", + protocol: "openai", + log: (_l: string, _m: string) => {}, + transform: (m: BiliMessage[]) => m, + }); + assert.equal(out, legacy); + assert.equal(snapEgressStatusWire().planeMiss, 1); +}); + +test("G4 plane dead: a latched sid serves the legacy bytes", () => { + const dir = tmpDir("bili-5874-g4-"); + const store = openPlane(path.join(dir, "index.db")); + registerStore(store); + const sid = "g4"; + const { sess } = foldedSession(sid); + sess.lastPlaneMessages = [ + ...foldedView(), + { id: "ghost-msg", role: "assistant", contentType: "text", text: "no legacy row" } as BiliMessage, + ]; + store.ingestSession(planIngest(sess)); // latches dead + setConfig({ storage: { snapBase: { egress: true } } }); + + const legacy = [taggedMsg(99)]; + const out = applySnapEgress({ blocks: [] } as unknown as CompressionState, legacy, { + sessionId: sid, + protocol: "openai", + log: (_l: string, _m: string) => {}, + transform: (m: BiliMessage[]) => m, + }); + assert.equal(out, legacy); + assert.equal(snapEgressStatusWire().plane, 0); + assert.equal(snapEgressStatusWire().planeMiss, 0, "dead is not a miss"); +}); + +test("G5 direct authority: the plane is served regardless of the legacy bytes", () => { + const dir = tmpDir("bili-5874-g5-"); + const store = openPlane(path.join(dir, "index.db")); + registerStore(store); + const sid = "g5"; + const { sess, blk } = foldedSession(sid); + sess.lastPlaneMessages = foldedView(); + store.ingestSession(planIngest(sess)); + setConfig({ storage: { snapBase: { egress: true } } }); + + const legacy = [{ id: "sentinel", role: "user", contentType: "text", text: "not served" } as BiliMessage]; + const out = applySnapEgress(stateFor(blk), legacy, { + sessionId: sid, + protocol: "openai", + log: (_l: string, _m: string) => {}, + transform: (m: BiliMessage[]) => m.map((x) => ({ ...x, text: `X${x.text}` })), + }); + const wire = snapEgressStatusWire(); + assert.equal(wire.plane, 1); + assert.equal(wire.degrade, 0); + // The plane view, through the lane transform — never the legacy bytes. + assert.deepEqual( + out.map((m) => JSON.stringify(m)), + foldedView().map((m) => JSON.stringify({ ...m, text: `X${m.text}` })), + ); + assert.ok(!out.some((m) => m.id === "sentinel")); +}); diff --git a/tests/persist-metadata-fold-gate.test.ts b/tests/persist-metadata-fold-gate.test.ts index 788d4cedaf..a557179279 100644 --- a/tests/persist-metadata-fold-gate.test.ts +++ b/tests/persist-metadata-fold-gate.test.ts @@ -50,8 +50,9 @@ const foldGate = (store: SessionStore): { dropResidentMetadata(session: Session) test("metadata fold is refused while live metadata diverges from the last committed bytes (#5506)", async () => { const dir = mkdtempSync(join(tmpdir(), "bili-fpgate-neg-")); + let store: SessionStore | undefined; try { - const store = new SessionStore({ dir, debounceMs: 60000, enabled: true }); + store = new SessionStore({ dir, debounceMs: 60000, enabled: true }); const s = makeSession("fpgate-negative"); s.metadata.anonymousPrefixAffinity = { depth: 22, tailHash: "test-tail", via: "prefix" }; await store.writeNow(s); @@ -64,14 +65,19 @@ test("metadata fold is refused while live metadata diverges from the last commit assert.equal((s.metadata as Record).lateStamp, "unsaved", "the unsaved mutation must stay resident"); store.closeQuietly(); // dual-write pin opens index.db — close before teardown (Windows cannot unlink an open SQLite file; see persist-zstd.test.ts for the family) } finally { + // Close before rm: under unified=full the store holds an index.db + // handle Windows cannot unlink (#6480 pinned full in this file). + store?.closeQuietly(); rmrf(dir); } }); test("metadata fold proceeds once the divergent mutation has been committed (#5506 positive direction)", async () => { const dir = mkdtempSync(join(tmpdir(), "bili-fpgate-pos-")); + let store: SessionStore | undefined; + let reloaded: SessionStore | undefined; try { - const store = new SessionStore({ dir, debounceMs: 60000, enabled: true }); + store = new SessionStore({ dir, debounceMs: 60000, enabled: true }); const s = makeSession("fpgate-positive"); s.metadata.anonymousPrefixAffinity = { depth: 22, tailHash: "test-tail", via: "prefix" }; await store.writeNow(s); @@ -81,7 +87,7 @@ test("metadata fold proceeds once the divergent mutation has been committed (#55 // Commit the mutation; the post-save fold must now see matching fingerprints. await store.writeNow(s); assert.equal(isDroppedMetadata(s.metadata), true, "fold must proceed once live == committed"); - const reloaded = new SessionStore({ dir, debounceMs: 60000, enabled: true }); + reloaded = new SessionStore({ dir, debounceMs: 60000, enabled: true }); const back = reloaded.loadSync("fpgate-positive"); assert.ok(back, "session reloaded"); assert.equal((back?.metadata as Record).lateStamp, "unsaved", "committed mutation survives the round trip"); @@ -89,14 +95,19 @@ test("metadata fold proceeds once the divergent mutation has been committed (#55 store.closeQuietly(); reloaded.closeQuietly(); } finally { + // Close before rm: under unified=full the stores hold index.db + // handles Windows cannot unlink (#6480 pinned full in this file). + store?.closeQuietly(); + reloaded?.closeQuietly(); rmrf(dir); } }); test("metadata fold stays closed for a session with no committed save yet (#5506 conservative default)", async () => { const dir = mkdtempSync(join(tmpdir(), "bili-fpgate-cold-")); + let store: SessionStore | undefined; try { - const store = new SessionStore({ dir, debounceMs: 60000, enabled: true }); + store = new SessionStore({ dir, debounceMs: 60000, enabled: true }); const s = makeSession("fpgate-cold"); (s.metadata as Record).onlyInMemory = true; foldGate(store).dropResidentMetadata(s); @@ -104,6 +115,9 @@ test("metadata fold stays closed for a session with no committed save yet (#5506 assert.equal((s.metadata as Record).onlyInMemory, true); store.closeQuietly(); } finally { + // Close before rm: under unified=full the store holds an index.db + // handle Windows cannot unlink (#6480 pinned full in this file). + store?.closeQuietly(); rmrf(dir); } }); diff --git a/tests/plugin-fork.test.ts b/tests/plugin-fork.test.ts index 8e555df402..5de5be3472 100644 --- a/tests/plugin-fork.test.ts +++ b/tests/plugin-fork.test.ts @@ -25,6 +25,10 @@ process.env.NODE_ENV = "test"; const testRoot = mkdtempSync(join(tmpdir(), "bili-fork-")); test.after(() => rmSync(testRoot, { recursive: true, force: true })); for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = testRoot; +// #6480 flipped the absent-default to pure-sqlite (JSON envelope writer +// off) — this suite's cold-disk-restore tests read the legacy JSON +// content-store files off disk, so pin the dual-write writer explicitly. +process.env.BILI_STORAGE_JSON_WRITER = "on"; const hash = (v: unknown) => createHash("sha256").update(JSON.stringify(v)).digest("hex"); interface SnapshotResponse { diff --git a/tests/storage-ext.test.ts b/tests/storage-ext.test.ts index 05f6dee5cc..3cf6c552de 100644 --- a/tests/storage-ext.test.ts +++ b/tests/storage-ext.test.ts @@ -174,7 +174,7 @@ test("v3 databases migrate to v11 in place, idempotently", async () => { const migrated = UnifiedStore.open(dbPath); try { const ver = (migrated as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "open() runs migrateSchema: v3 → v11"); + assert.equal(ver.user_version, 12, "open() runs migrateSchema: v3 → v12"); migrated.setKv("probe", "alive", true); assert.deepEqual(migrated.getKv("probe", "alive"), true); assert.deepEqual(migrated.listSessionIds(), [], "existing v3 content untouched"); @@ -184,7 +184,7 @@ test("v3 databases migrate to v11 in place, idempotently", async () => { // Reopening a v11 database is a no-op (idempotent migration). const again = UnifiedStore.open(dbPath); const ver2 = (again as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 11); + assert.equal(ver2.user_version, 12); again.close(); } finally { fs.rmSync(tmp, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); diff --git a/tests/storage-ingest-watermark.test.ts b/tests/storage-ingest-watermark.test.ts index 6ab6229b19..b6fb399b78 100644 --- a/tests/storage-ingest-watermark.test.ts +++ b/tests/storage-ingest-watermark.test.ts @@ -591,7 +591,7 @@ test("watermark: sampled verification still catches drift through the delta path } }); -test("watermark: v4 database migrates to v11 additively; newer user_version no-ops", async () => { +test("watermark: v4 database migrates to v12 additively; newer user_version no-ops", async () => { const dir = tmpdir("bili-wm-migrate-"); try { const dbPath = path.join(dir, "index.db"); @@ -620,7 +620,7 @@ test("watermark: v4 database migrates to v11 additively; newer user_version no-o const store = UnifiedStore.open(dbPath); try { const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "open() migrates v4 → v11"); + assert.equal(ver.user_version, 12, "open() migrates v4 → v12"); const row = store.getSessionRow("v4-sess"); assert.ok(row, "legacy session row survives the migration"); assert.equal(row!.lastIngestedSeq, null, "legacy rows read back with a NULL watermark (≡ full ingest)"); @@ -658,7 +658,7 @@ test("watermark: v4 database migrates to v11 additively; newer user_version no-o const older = UnifiedStore.open(dbPath); try { const ver2 = (older as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 11, "newer user_version is left untouched"); + assert.equal(ver2.user_version, 12, "newer user_version is left untouched"); assert.ok(older.getSessionRow("v4-sess"), "rows stay readable"); } finally { older.close(); diff --git a/tests/storage-null-identity.test.ts b/tests/storage-null-identity.test.ts index 5f2ad9f1a5..fe6ad2d5e7 100644 --- a/tests/storage-null-identity.test.ts +++ b/tests/storage-null-identity.test.ts @@ -45,14 +45,14 @@ test("v8 identity migration can finish with a JSON-null content row", () => { const migrated = migrateSchema(db); assert.equal(migrated.from, 7); - assert.equal(migrated.to, 11); + assert.equal(migrated.to, 12); const row = db.prepare("SELECT identity, content_hash FROM messages WHERE session_id = ?").get("null-session") as { identity: string; content_hash: string }; assert.equal(row.identity, hash); assert.equal(row.content_hash, hash); const content = db.prepare("SELECT bytes, refcount FROM content WHERE hash = ?").get(hash) as { bytes: Uint8Array; refcount: number }; assert.equal(Buffer.from(content.bytes).toString(), raw); assert.equal(content.refcount, 1); - assert.equal(migrateSchema(db).from, 11); + assert.equal(migrateSchema(db).from, 12); } finally { db?.close(); if (previousZstd === undefined) delete process.env.BILI_PERSIST_ZSTD; diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index 8e4908ba87..494f7ffba0 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -815,7 +815,7 @@ test("schema v2: dev-window v1 database upgrades in place (identity columns adde assert.equal(row!.namespace, "proxy", "namespace backfills with its constant default"); assert.equal(row!.origin, null, "origin backfills NULL"); const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "schema version stamped to 11 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 orphans journal + v7 media map + v8 identity/tombstones + v9 zero-ref collection index + v10 immutable message ts column + v11 FK child indexes)"); + assert.equal(ver.user_version, 12, "schema version stamped to 12 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 orphans journal + v7 media map + v8 identity/tombstones + v9 zero-ref collection index + v10 immutable message ts column + v11 FK child indexes + v12 snapshot-table foundation)"); // v3 backfill: every pre-existing sessions row claims its // (namespace, conversation_id) identity binding. const binding = (store as unknown as { lookupConversation(ns: string, id: string): string | null }).lookupConversation("proxy", "dev-window-1"); @@ -877,7 +877,7 @@ test("schema v10: v9→v10 migration adds messages.ts without backfill; idempote try { const ver = (reopened as unknown as { db: { prepare(q: string): { get(): unknown } } }).db .prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "v9 library migrates to v11"); + assert.equal(ver.user_version, 12, "v9 library migrates to v12"); const rows = reopened.getMessages("v10-mig-1") as { seq: number; ts?: number | null }[]; assert.equal(rows.length, 1, "row survives the migration"); assert.ok(rows[0].ts === null || rows[0].ts === undefined, "pre-existing row keeps NULL (never backfilled)"); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 6a0632a4e6..d5bf31cb93 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -447,6 +447,42 @@ entries: desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier." desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。" desc_ja: "日単位の GC 年齢スイープウィンドウ (#3463, #3901): これより長くアイドル状態のセッションはアーカイブされる - 単一のアーカイブログ (セッションディレクトリ配下の archive/archive.jsonl — セッションごとのファイルスプレーなし) に 1 行の JSON が追記され、トombストーン行がラベルとアイデンティティを保持し、コンテンツ/メディア参照が解放され、リストアは 1 つの POST のみで可能。0 は年齢スイープを無効にする; POST /__bili/storage/gc は常に実行され、既存の index.db を縮小するには ?vacuum=full を取る (新規書き込みには常に auto_vacuum=INCREMENTAL が有効)。同じ操作に CLI レイヤーがある (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]、および bili sessions ls/rm/archive/restore (README の 'Storage maintenance CLI' セクション)。環境変数がファイル階層より優先される。" + - key: storage.snapBase.enabled + type: boolean + default: "false" + kind: file-only + audience: user + group: process + desc_en: "Master flag for the snapshot-table compression plane (#5874, schema v12): arms the four plane tables (l1_world / snap_rows / snap_log / snap_ckpt), the single-transaction commit primitive, and the read-side checksum rebuild (a mismatch permanently wedges that session's plane). Off by default — zero behavior change until set; config-file only, no env tier (#2030 discipline). Wired on the sqlite path only — the mysql backend ignores the flag." + desc_zh: "快照表压缩平面的主开关(#5874,schema v12):启用四张平面表(l1_world / snap_rows / snap_log / snap_ckpt)、单事务提交原语与读侧校验和重建(对不上会永久关死该会话的平面)。默认关闭 —— 不设置时零行为变化;仅配置文件层,无环境变量档(#2030 纪律)。目前只接 sqlite 路径 —— mysql 后端忽略该旗。" + desc_ja: "スナップショットテーブル圧縮プレーンのマスターフラグ (#5874, schema v12): 4 つのプレーンテーブル (l1_world / snap_rows / snap_log / snap_ckpt)、単一事務コミット原語、読み出し側のチェックサム再構築を有効化する (不一致はそのセッションのプレーンを永続的に固着させる)。デフォルトオフ — 設定するまで挙動は不変; 設定ファイルのみ、環境変数階層なし (#2030 の規律)。sqlite パスにのみ配線 — mysql バックエンドはこのフラグを無視する。" + - key: storage.snapBase.shadowWrite + type: boolean + default: "false" + kind: file-only + audience: user + group: process + desc_en: "Lane flag for the plane's shadow write, nested under storage.snapBase.enabled (#5874): with both set, every ingest commit additionally derives the plane in its own transaction within the same tick (steady-state delta fast path, otherwise final-state diff; rounds yielding no derivable event write nothing, a crash-dropped round self-heals on the next save). The read side only compares against the legacy rows and never serves them — outbound bytes are unchanged. The lane alone has no effect; rollback = unset (or DROP the four tables + revert user_version)." + desc_zh: "平面影子写的车道开关,嵌套在 storage.snapBase.enabled 之下(#5874):两者都设置后,每次 ingest 提交在同一 tick 内用独立事务推导平面(稳态 delta 走快路,其余走最终态 diff;推导不出事件的轮次不写,crash 丢掉的轮次由下一轮保存自愈)。读侧只与 legacy 行比对、永不对外提供服务 —— 出站字节一个都不变。只开车道不生效;回滚 = 去掉开关(或 DROP 四张表 + user_version 退回)。" + desc_ja: "プレーンのシャドウライト用のレーンフラグ、storage.snapBase.enabled にネスト (#5874): 両方が設定されると、各 ingest コミットが同一 tick 内で独立した事務所でプレーンを追加導出する (定常状態は delta 高速パス、それ以外は最終状態 diff; 導出可能なイベントのないラウンドは何も書かず、クラッシュで失われたラウンドは次の保存時に自己修復)。読み出し側は legacy 行との比較のみを行い、決して提供しない — 送信バイトは不変。レーン単独では無効; ロールバック = 解除 (または 4 テーブルの DROP + user_version の復元)。" + - key: storage.snapBase.egress + type: boolean + default: "false" + kind: file-only + audience: user + group: process + desc_en: "#5874 P2 (mapEgress): the snapshot plane is the DIRECT outbound authority — the outbound compressed context is served from the S4-verified plane, with NO per-round comparison against the legacy bytes (no tripwire, no normalized diff, no sticky exclusion). The plane stores the kernel's OWN tagged processTurn output (invariant v: post_json is the post-rewrite core form, never wire bytes); the write side persists exactly the bytes the kernel saw as event-level increments (cast / update / delete / compact — one transaction per save, same tick as the legacy commit), so a compression turn's brand-new block is plane-readable in the same round. Egress reads the plane rows in pos order (surviving active rows + each active block's stored anchor summary, moved to its covered run's pair-safe position; covered rows are suppressed by the plane's own active flag / cover_ref, never by id matching), then runs the lane's exact post-processing (absorb → attach → strip → drop) — the output is byte-identical to the legacy egress BY CONSTRUCTION, because the plane rows ARE the legacy transform input. Degrades to the legacy bytes (one round; the next round re-decides) on: flag off, cohort excluded, plane dead/missing for this session, any row unparsable, or an assembly exception — each reason surfaces on /__bili/status under snaplane.egress. The write side is crash-grade: an unaddressable view entry or a fold-time view-vs-plane identity mismatch (A/B oracle) latches the session plane-dead permanently, legacy keeps serving. Opt-in: default OFF when unset (#5874 §9) — set `egress: true` to cut over; rollback = `egress: false` (zero bytes change, the plane keeps shadow-writing). Known soft edges: google countTokens and responses compact secondary processTurn points still ride the legacy source." + desc_zh: "#5874 P2(mapEgress):快照平面就是直接的出站权威 —— 出站压缩上下文直接从 S4 校验过的平面读,不做任何与旧字节的逐轮比对(无 tripwire、无归一化 diff、无粘性排除)。平面存的是内核自己打标的 processTurn 输出(不变量 v:post_json 恒为改写后的核心形态,绝不含线路字节);写侧以事件级增量(cast / update / delete / compact —— 每次保存一个事务,与 legacy 提交同 tick)持久化内核看到的那份字节,所以压缩轮的新块同轮即可被平面读出。出站按 pos 顺序读平面行(存活活动行 + 每个活动块存的锚点摘要,移到其覆盖区段的安全位置;被覆盖行靠平面自身的 active 旗 / cover_ref 抑制,绝不靠 id 匹配),再跑本车道的同一后处理链(absorb → attach → strip → drop)—— 输出与旧出站逐字节一致是结构保证,因为平面行本身就是旧后处理的输入。降级回旧字节的情形(一轮,下一轮重新决定):旗关、cohort 不含、该会话平面死/缺、任一行不可解析、拼装异常 —— 每个原因都在 /__bili/status 的 snaplane.egress 下可见。写侧是 crash 级的:视图里出现不可寻址条目、或折叠时视图对平面身份对不上(A/B 裁判),都会把该会话平面永久锁死,legacy 继续兜底。显式开启(opt-in):不设置时默认关(#5874 §9)—— 设 `egress: true` 才切换;回滚 = `egress: false`(零字节变化,平面继续影子写)。已知软边界:google countTokens 与 responses compact 两个次级 processTurn 点仍走旧源。" + desc_ja: "#5874 P2 (mapEgress): スナップショットプレーンが直接の出站権威 — 出站圧縮コンテキストは S4 検証済みプレーンから提供され、legacy バイトとのラウンド毎の比較は行わない (tripwire なし、正規化 diff なし、sticky 除外なし)。プレーンはカーネル自身のタグ付き processTurn 出力を保存する (不変条件 v: post_json は書き換え後のコア形式であり、決してワイヤバイトではない); 書き込み側はカーネルがイベントレベルの増分として見たバイトを正確に永続化する (cast / update / delete / compact — 保存毎に 1 事務、legacy コミットと同一 tick)、そのため圧縮ラウンドの新規ブロックは同一ラウンドでプレーンから読み出せる。Egress はプレーン行を pos 順に読み (生存する active 行 + 各 active ブロックの保存済みアンカー要約をその被カバー run の pair-safe 位置へ移動; 被カバー行はプレーン自身の active フラグ / cover_ref で抑制し、id 一致では決して行わない)、その後レーン自身の後処理 (absorb → attach → strip → drop) を実行 — 出力は構造上 legacy egress とバイト単位で一致する (プレーン行が legacy 変換の入力そのものだから)。フラグオフ、cohort 除外、このセッションのプレーン dead/欠如、いずれかの行がパース不能、アセンブリ例外 — いずれの場合も legacy バイトへ降格 (1 ラウンド; 次ラウンドで再決定)、各理由は /__bili/status の snaplane.egress 下に表示。書き込み側は crash 等級: 不可解参照の view エントリ、または fold 時の view とプレーンの ID 不一致 (A/B 判定) はセッションを永続的に plane-dead に固定し、legacy が引き続き提供する。opt-in: 未設定時はデフォルト OFF (#5874 §9) — `egress: true` で切り替え; ロールバック = `egress: false` (バイト変化ゼロ、プレーンはシャドウライト継続)。既知のソフトエッジ: google countTokens と responses compact の二次 processTurn ポイントは legacy ソースのまま。" + - key: storage.snapBase.egressCohorts + type: string[] + default: "[\"openai\",\"anthropic\",\"google\",\"responses\"]" + kind: file-only + audience: user + group: process + desc_en: "Protocol cohorts allowed to cut the egress source over (#5874 P2); a subset of openai/anthropic/google/responses. Sessions whose protocol is absent keep legacy bytes unconditionally (the plane still shadow-writes for them). An empty/absent array means all four." + desc_zh: "允许切出站源的协议 cohort(#5874 P2),取 openai/anthropic/google/responses 的子集。协议不在名单内的会话无条件保留旧字节(平面照常影子写)。空/缺省 = 四个全开。" + desc_ja: "egress ソースの切替を許可するプロトコル cohort (#5874 P2); openai/anthropic/google/responses の部分集合。プロトコルがリストにないセッションは無条件で legacy バイトを保持する (プレーンはシャドウライト継続)。空/欠如の配列は 4 つすべてを意味する。" - key: sessions type: object default: "{} (all defaults)"