diff --git a/AGENTS.md b/AGENTS.md index 0a67222405..98cbfc028a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,6 +61,14 @@ Tier A entries are every-session rules and stay in full below. Tier B entries ca 身份信号来源只有三类:**宿主写入**(msgid body 字段 / 信封 `msgIds` / pf 头)、**宿主盖戳头**(opencode V2 `x-bili-msg-ids`)、**自包含采集器**(`src/identity-harvest.ts` 注册表 + `src/identity-.ts` 适配器;双门:env `BILI_IDENTITY_HARVEST` 或灰度键 `rollout.features.identityChannelHarvest`,均默认关 × 仅回环来源)。**消费者(#5719 归属档 / pin-行链路 / membership / 水位)只从统一缝取数**:handle 入口每请求按「body 字段 > 盖戳头 > 采集」取一次快照——绝不新增「逐调用点单独取数」。新车道 = 新适配器文件 + barrel 一行登记 + 文档一行;适配器自包含、整轮退化(任一环节不判定 ⇒ 该轮整体回退接线前路径,绝不做部分映射)。车道状态表与信任边界:docs/identity-channels.md;身份判定逻辑:docs/session-identity-resolution.md。Full text: design-decisions.md #18. +### #20 — id 存储去重与 msgid 生命周期梳理(#7064, owner 裁定 2026-11-17) (Tier B) + +msgid 真身只有 `messages.msg_id` 一处,`l1_world`/`fold_journal`/`egress_snapshot` 里的 id 清单 JSON 全是投影,已全部降级为「区间引用 + 防伪指纹」:l1_world = 尾 512/头 32 id 窗 + count + max_pos(身份闸门→记账游标,中段不可见);fold_journal = [start_seq,end_seq] + input_hash(原文行防伪指纹,按区间重扫重推);egress_snapshot = 水位线以下存活行聚合指纹。**D1 永不判分叉**(判决最强动作是 reanchor 一条 UPDATE,永不 reject/400/reset);**D2 conversation-bound 跳过 id 判定**(宿主身份行的 sid 游标纯记账,R3 故障永不触发,短/缺 msgid 只降级 + 日志);**D3 尾锚常活**(水位命中轮同样 pin + 刷新游标;stage-0 推进 bump revisionEpoch,snapFlush epoch 证明看见推进——fork 拿不到过期父 revision)。改 l1/fold_journal/egress 任何一处前先读 discussion #7064 与本条目——三表回答的是同一个问题(「我缓存的和账本还对得上吗」),不需要 id 清单本身。Full text: design-decisions.md #20. + +### #21 — msgid pin 通道增量化:稳态只发尾部,快照轮才发全量(#7064 §8, owner 裁定 2026-11-18) (Tier B) + +全量 msgid 清单不再每轮发送:稳态靠既有有界通道 `x-bili-pf-tail-ids`(锚点+新增,≤2048,`?` 占位)携带增量,服务端从前缀**从账本 live 行重盖**(ingest 是 INSERT OR IGNORE on (session_id,seq)——位置上的 msg_id 只写一次,不从账本重盖 kernel ref 空间会漂)。接受规则(2026-11-19 定型:尾部对齐重叠段逐行对账):清单声明覆盖连续尾段 [T-L, T-1],与账本的重叠段 [T-L, C-1] 逐行相等(任一行漂移⇒降级,永不 reject);新增核(重叠之后)逐槽筛查;稳态形态(锚点+新增,L=N+1)= 单行重叠特例 = 原锚点检查,行为不变;**超长会话(>2048)截断尾清单同样可 pin**——重叠段更长、证明更强,「退回全量」退化为「发你有的」,全量不再是必需。任何失配降级 "full" 信号、下轮重锚(D1 家族)。全量形式不删、**全量优先**:只有快照轮(启动账本空 / 压缩/fork 后重锚)发全量。客户端经响应头 `x-bili-msgid-mode: incremental|full` 学 mode(与 x-bili-pf-seq 同头发出、4xx 即清、pf 状态机单学习点 pfLearnStamp);抑制单点 pfSuppressFullMsgIds(wire 短于最近实测 T ⇒ 压缩 ⇒ 清 mode + 发全量)。灰度 `pluginMsgIdIncremental` 默认 0(off = 字节级同改前);opencode 暂留全量(phase 2:core-aligned 列表跳过 leading system 而 /responses T 计 wire,tail off-by-one)。改 pin/信号/tail-ids 任何一处前先读 discussion #7064 §8 与本条目。Full text: design-decisions.md #21. **phase 2(owner 裁定 2026-11-18:全客户端覆盖 + 默认开)**:两形态规则上移到申报通道本身——数组长度 = 视图长度 ⇒ 全量(既有路径),短数组 ⇒ 合法尾部形态(漂移类:只降级发「full」,永不 400;R3 故障只来自全量数组的坏内容;**空账本上的短数组 = 采集故障,沿用旧 R3 400 路径**);body 短数组优先于 pf-tail-ids header。responses/anthropic 车道都接受尾部形态并发 `x-bili-msgid-mode`;opencode V1/V2 已接线(`pfIdTailForm` prevIds 前缀延伸校验,shrink/rewrite 回落全量;core-aligned 列表对 hoist 后 core 计数天然对齐,**无偏移**);omp 无全量申报面(delta 车道天然增量)、claude/codex/google 无申报 = 覆盖完整;`pluginMsgIdIncremental` **默认 100**(回滚 env `BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL=0`,失配一轮自愈)。 + ### Install-Lane & Update-Ownership Contract (#1196) Every bili presence on a machine follows ONE contract — decide changes against it, not ad hoc: diff --git a/docs/architecture/storage.ja.md b/docs/architecture/storage.ja.md index e7e7785843..7f13ceb991 100644 --- a/docs/architecture/storage.ja.md +++ b/docs/architecture/storage.ja.md @@ -20,7 +20,7 @@ **書き込み側。** インジェストパスは統合ビューをバッチ書き込みする(安定した絶対 seq; `identityOfStoredContent` によるコンテンツハッシュ ID; バウンダリフレーム `boundaryMsgId` はバッチとともに格納される)。`persist.ts` はセッションファイルの保存/読み込みと EPERM アラート層を処理する。カーネルは、カーネルが提供する StateStore インターフェースを通じて圧縮状態(ブロック / 参照 / dead_refs / チェックポイント)を書き込む — ホストは実装を差し替える。 -**kstate JSON は並列の第二の真実の源である(ペインポイント 1)。** 残存する `CompressionState` フィールドリストは `mergeState()` によってシリアライズされる(フィールドリストの単一の源 — `src/state-hydration.ts`)であり、テーブルとともに読み書きされる — 二重チャネルである。#5643 の進化により、カーネル状態のシリアライズは `snap_ckpt.meta_json` 内に移動され、テーブルが唯一の源となる。 +**kstate JSON は並列の第二の真実の源である(ペインポイント 1)。** 残存する `CompressionState` フィールドリストは `mergeState()` によってシリアライズされる(フィールドリストの単一の源 — `src/state-hydration.ts`)であり、テーブルとともに読み書きされる — 二重チャネルである。カーネル状態のテーブル統合はフォローアップ作業(#6858 が #6785 のスコープ外と明記);#5874 v2 の基盤(content CAS、messages、refs、fold_journal、egress_snapshot)がその収束の着地点である。 **読み込み側。** ハイドレーションは、残存フィールドの単一の源として `mergeState()` を使用する(テーブル + kstate が組み立てられる)。エグレスはフォールディングカバレッジチェックとサーバーサイドの正規レンダリングを実行する(D レーン、#3756 — どのデシンクもレガシーパスにフォールバックし、デグレードしても推測しない)。課金とキャッシュ台帳(`cache-ledger`、`store.ts` 内の CCR 取得)および表示/エクスポートサーフェス(`sessions-data`、`acp_status`、`export` — 誠実な出力、#155)はすべて同じ行を読み取る。メンテナンスは `state-gc` / `storage-maintenance` / オルファン GC(`sessions.gc_*` でのストレージ GC、オプトイン、dead-ref の回収)である。 diff --git a/docs/architecture/storage.md b/docs/architecture/storage.md index 64b6f6c8db..985337bcd7 100644 --- a/docs/architecture/storage.md +++ b/docs/architecture/storage.md @@ -20,7 +20,7 @@ Everything that must survive a process restart lands in ONE SQLite file: session **Write side.** The ingest path batch-writes the unified view (stable absolute seq; content-hash ids via `identityOfStoredContent`; boundary frame `boundaryMsgId` lands with the batch). `persist.ts` handles session file save/load and the EPERM alert layer. The kernel writes compression state (blocks / refs / dead_refs / checkpoints) through the StateStore interface the kernel ships — the host swaps the implementation. -**The kstate JSON is the parallel second source of truth (pain point 1).** The surviving `CompressionState` field list is serialized via `mergeState()` (single source of the field list — `src/state-hydration.ts`) and read/written alongside the tables — a dual channel. The #5643 evolution would move kernel state serialization into `snap_ckpt.meta_json` so the tables become the only source. +**The kstate JSON is the parallel second source of truth (pain point 1).** The surviving `CompressionState` field list is serialized via `mergeState()` (single source of the field list — `src/state-hydration.ts`) and read/written alongside the tables — a dual channel. Consolidating the kernel state into the tables is tracked as follow-up work (#6858 lists it as out of scope for #6785); the #5874 v2 substrate (content CAS, messages, refs, fold_journal, egress_snapshot) is what that consolidation lands on. **Read side.** Hydration uses `mergeState()` as the single source for surviving fields (tables + kstate assembled). Egress runs the fold-coverage check and the server-side canonical render (D-lane, #3756 — any desync falls back to the legacy path, degrade never guess). Billing & the cache ledger (`cache-ledger`, CCR retrieval in `store.ts`) and the display/export surfaces (`sessions-data`, `acp_status`, `export` — honest output, #155) all read the same rows. Maintenance is `state-gc` / `storage-maintenance` / orphan GC (storage GC on `sessions.gc_*`, opt-in, dead-ref reclaim). diff --git a/docs/architecture/storage.zh-CN.md b/docs/architecture/storage.zh-CN.md index f489a759fe..955645f219 100644 --- a/docs/architecture/storage.zh-CN.md +++ b/docs/architecture/storage.zh-CN.md @@ -20,7 +20,7 @@ **写入侧。** ingest 路径批量写统一视图(稳定绝对 seq;内容哈希 id 走 `identityOfStoredContent`;边界帧 `boundaryMsgId` 随批落地)。`persist.ts` 管会话文件存盘/装载与 EPERM 告警层。内核经 StateStore 接口写压缩状态(blocks / refs / dead_refs / checkpoints)——接口由内核自带,宿主换实现。 -**kstate JSON 是平行的第二个真源(痛点 1)。** 存活 `CompressionState` 字段清单经 `mergeState()` 序列化(字段清单的单一事实源在 `src/state-hydration.ts`),与表并行读写——双通道。#5643 演进会把内核态序列化挪进 `snap_ckpt.meta_json`,让表成为唯一真源。 +**kstate JSON 是平行的第二个真源(痛点 1)。** 存活 `CompressionState` 字段清单经 `mergeState()` 序列化(字段清单的单一事实源在 `src/state-hydration.ts`),与表并行读写——双通道。把内核态收敛进表是后续工作(#6858 将其列为 #6785 范围外);#5874 v2 基座(内容 CAS、messages、refs、fold_journal、egress_snapshot)就是那次收敛落地的目标。 **读取侧。** 水合用 `mergeState()` 作为存活字段的单一事实源(表 + kstate 拼装)。出站跑折叠覆盖检查与服务端正史渲染(D-lane,#3756——任何失步回退 legacy 路径,**降级不猜**)。计费与缓存账本(`cache-ledger`、CCR 取回在 `store.ts`)、展示/导出面(`sessions-data`、`acp_status`、`export`——诚实输出 #155)读的都是同一批行。维护面是 `state-gc` / `storage-maintenance` / 孤儿 GC(`sessions.gc_*` 上的 storage GC,opt-in,dead-ref 回收)。 diff --git a/docs/configuration.ja.md b/docs/configuration.ja.md index 079cfb04a5..f529cb6cb7 100644 --- a/docs/configuration.ja.md +++ b/docs/configuration.ja.md @@ -103,7 +103,10 @@ | `rollout.features.salt` | string | the feature name | — | バケット分割用のハッシュソルト。変更するとコホート全体が再ランダム化される(シャッフルする唯一の方法;パーセンテージを上げるだけではシャッフルされない)。 | | `rollout.features.base` | string | unset (roll out over everyone) | — | ネストされたロールアウト:指定された機能の有効化コホート内でのみこの機能を評価し、パーセンテージがその母集団に対する相対値となる(Aのユーザーに対してBをロールアウト;チェーンは許可、サイクルはクローズドに失敗する)。 | | `rollout.features.identityChannelHarvest` | boolean \| number | 0 (off) | BILI_ROLLOUT_IDENTITY_CHANNEL_HARVEST | IDチャネル収穫コホートゲート (#5825):インストールごとの環境スイッチなしで、ループバック専用のトランスクリプト/sqlite収穫(`src/identity-harvest.ts`: claude-code / codebuddy / mcode / goose / qwen レーン)を許可する。インストールごとのスイッチ(`BILI_IDENTITY_HARVEST`)もこのキーも設定されていない場合、クライアントストレージは一切読み取られない。オーナー裁定 2026-11-07:このコホートキーは `rollout.features.pluginLaneServerHistory`(承認済みIDの行固定を管理)とは意図的に分離されている — いずれかのイネーブラーが収穫を有効化できる。デフォルト 0%;小規模コホートでの degrade warn-once レートの観察後に引き上げる。`identity-channels.md` を参照。 | -| `rollout.features.pluginLaneServerHistory` | boolean \| number | 0 (off) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | プラグインレーンのサーバー所有履歴 (#4491):有効化されたインストールにおいて、(a) プラグインレーンのリクエストは `messages` と並行して内部 `biliMsgIds` 配列を持ち、メッセージIDをクライアント提供のIDに固定できる(上流転送前に常に除去;不正な配列は警告付きで無視、ID汚染形状は増分レーンでハードフェイル);(b) ループバック専用の `POST /__bili/plugin/render` エンドポイントは、統合トランスクリプトからレンダリングされた送信プレフィックスを提供する(折り畳み領域はサマリアンカーに縮小;デシンクは非2xxにデグレードし、シムはローカル履歴にフォールバック);(c) 増分レーンのプローブは `serverHistory` をアドバタイズする。デフォルト 0% — ロールアウトされるまで挙動は変わらない。 | +| `rollout.features.pluginLaneServerHistory` | boolean \| number | 100 (on, GA) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | プラグインレーンのサーバー所有履歴 (#4491):有効化されたインストールにおいて、(a) プラグインレーンのリクエストは `messages` と並行して内部 `biliMsgIds` 配列を持ち、メッセージIDをクライアント提供のIDに固定できる(上流転送前に常に除去;不正な配列は警告付きで無視、ID汚染形状は増分レーンでハードフェイル);(b) ループバック専用の `POST /__bili/plugin/render` エンドポイントは、統合トランスクリプトからレンダリングされた送信プレフィックスを提供する(折り畳み領域はサマリアンカーに縮小;デシンクは非2xxにデグレードし、シムはローカル履歴にフォールバック);(c) 増分レーンのプローブは `serverHistory` をアドバタイズする。デフォルト 100%(GA、オーナー裁定 2026-11-13);0 に設定するとインストールは GA 前の状態に戻る。 | +| `rollout.features.pluginMsgIdIncremental` | boolean \| number | 100 (on, GA) | BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL | 増分 msgid pin チャネル(#7064 §8、オーナー裁定 2026-11-18/19):定常プラグインターンはメッセージ ID を**テール形態**で申告する — 有界な `x-bili-pf-tail-ids` ヘッダー(アンカー+新規 ID、≤ 2048 個、`?` プレースホルダ)または短い body ID 配列 — 全リストを置き換える。サーバーは自前の台帳 live 行からプレフィックス ID を再スタンプする(ingest は INSERT OR IGNORE on (session_id,seq) — 位置の msg_id は一度だけ書き込まれ、決して上書きされない)。受理規則(2026-11-19 定石):申告されたテール [T-L, T-1] と台帳の重複区間が逐行一致すること(1行でもずれれば `full` にデグレード、400 はしない);超長セッション(> 2048)も切り詰められたテールで同じく pin できる — 重複区間が長いほど証拠は強い。スナップショットラウンド(台帳空/折り畳み後の再アンカー)のみ全形態を送る;両方到着時は全形態が優先。レスポンスは `x-bili-msgid-mode: incremental\|full` を載せる(4xx でクリア、x-bili-pf-seq と同じ学習ポイント)。不一致は 1 ラウンドで自己修復。デフォルト 100%;0 で全リスト pin にロールバック(機能前とバイト一致)。pluginLaneServerHistory が必要。session-identity-resolution.md 参照。 | +| `rollout.features.egressPlane` | boolean \| number | 0 (off) | BILI_ROLLOUT_EGRESS_PLANE | egress プレーン — 新しい経路全体の**単一スイッチ**(#7398/#7439;退役した egressOpLog/egressServeLog の二キーを置き換え)。ON:全 wire レーン(OpenAI/Anthropic/Responses/Google)が毎ターン送信会話ビューをスパース op(I/D/U)として egress_state/egress_flow デュアルテーブルに記録 — 行は**自己完結**(メッセージバイトが同行する;コンテンツ CAS を経由しない、裁定 7439-1)— 送信コアメッセージ部分を op ログから復元し再構築、構造によりバイト一致(行内バイトはパイプライン自身の最終 core JSON);ピン留め済みプラグインレーンは加えてサーバー権威を取得 — カーネル入力をフローからリプレイ(裁定 7439-3/5)。有界の遡行ラダーで最新の健全ログ点に回復;再構築不能行(バイト欠落/空、未知 role)、ログ破損、プローブ失敗のいずれかで**そのターン全体**をレガシーチェーンにデグレード — 部分的な swap は絶対にしない、クライアントを権威にしない。system ブロックは両経路でライブ構築のまま(プレフィックスキャッシュアンカー規律)。SQLite レーンのみ;タイトル生成/圧縮のサイドリクエストは除外。デフォルト 0 = オフ:レガシー経路、0 行、機能前とバイト一致。 | +| `rollout.features.pluginDeltaResend` | boolean \| number | 0 (off) | BILI_ROLLOUT_PLUGIN_DELTA_RESEND | Delta 再送 — フルリセンドウォーターマークレーンのテールスライス形態:定常プラグインレーンのターンは `x-bili-pf-delta: 1` + body.messages = 新要素のみ(L = T − stampT、プレフィックスなし)を送る;サーバーはスタッシュ済みプレフィックスコアとパース済みテールからビューを再構築する(フルレーンと同一のヒット由来)。満たせない主張(count/body/tail-ids 幅の不一致、既知リスト後のアンカーチェーン破断、ロールアウトオフ)は 400 `pf_delta_reanchor` + `x-bili-msgid-mode: full` を返し、クライアントはそのターンをフルフォームで再送する — 1ラウンドで収束、静かな誤パースは絶対にしない。デフォルトオフ:delta body はこのアームを持たないサーバーでは処理できないため、クライアント送信側が実装されるまでキーは 0 のまま。`technical-notes.md` の「Delta resend」を参照。 | | `rollout.features.pluginFullResendWatermark` | boolean \| number | 5 (experimental grey) | BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK | フルリセンドウォーターマークファストパス (#5403/#5418):前回のレスポンスに `x-bili-pf-seq` ス tamp を持っていたプラグインレーンのチャットコンプリートターンでは、エコーされたカウントにより再送信プレフィックスをアサートする(構造スキャン、フルパースなし);ヒット時にパイプラインはスタッシュされた正規プレフィックスコアと新しいテールウィンドウのみを消費する(バイト同一の送信プロベナンス;疑義がある場合はターンごとに未変更のフルパースパスにフォールバックし再スタンプ、5回のクレジットミスでレーンを1時間保持)。クライアントがスタンプを `x-bili-pf-count` としてエコーバックすることを要求(dsh/pi/opencode/omp は対応;未適応レーンはヒットしない)。実験的グレースケールロールアウト — インストールの約5%でデフォルトON(オーナー裁定 2026-11-02:GA前に小規模コホート;Web UIで全インストールに一度限りの導入通知、クイック設定に3状態スイッチ)。`true` で強制、`false`(またはパーセンテージ)/ 環境変数 = 0 でオフ。`technical-notes.md` の「Full-resend watermark fastpath」を参照。 | | `maskHosts` | boolean | true | BILI_LOG_MASK_HOSTS | ログ内の非公開ターゲットホストを としてマスクする。認証情報ヘッダーは常にマスクされる。 | | `security.adminRelay` | boolean | false | — | エンベディングホスト(現在は dsh web)が、リモートブラウザからパネルを開けるように(#3373)、その自身のエントリーオリジンを通じて /__bili/ 管理サーフェス全体をリレーすることを許可する。フェイルクローズ、ファイルのみ。リレーされたホップはローカル非ブラウザクライアントとしてプロキシに到達する(ループバックソース + ループバック Host、Origin なし)ため、管理ゲートは変更されない。これを有効にすると、ホスト Web エントリーに到達できる者に設定編集と認証情報の書き込みが露出する — まずそのエントリー(認証 / リバースプロキシ)を保護すること。 | @@ -212,19 +215,6 @@ | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | フェーズ 3 エンベロープスイッチ (#2960) — 今回のリリースでデフォルト反転: 未設定 (UNSET) はピュア SQLite 選択 (storage.unified 未設定または "sqlite") ではデフォルト OFF — SQLite インデックスが唯一の永続コピーとなる (読み取りはすでに統合優先; 起動時にインデックスの和集合を列挙するため、インデックスのみのセッションは正常にロードされる)。明示的な full は #3818 由来のダブルライトデフォルトを維持 (このキーが指定しない限りエンベロープは書き込まれ続ける)。有効な full モードではいずれの階層でも明示的な on/off が常に優先: shadow/legacy (または暗号化が有効な場合) で要求された off は、JSON ファイルが唯一の真実のソースであるため 1 回限りの警告付きで無視される。環境変数階層 0/false/off = off、1/true/on = on が厳密なファイル表記 ("off"/"on") より優先される。フェイルセーフ: サイドカーが実行中に自身を無効化した場合、次のセーブでエンベロープ書き込みが再開される。ダブルライトの復元: storage.jsonWriter=on (または BILI_STORAGE_JSON_WRITER=1) — インデックスのみのセッションは起動和集合経由で引き続きロードされ、レプリカ/リストアはポータブルな JSON エンベロープを再物質化する。バイナリのダウングレード (または SQLite エンジンの喪失) は、>=v0.1.197 バイナリが再び実行されるまで、インデックスのみのセッションを v0.1.196 以前のバージョンから不可視にする - データは index.db に無傷のまま残るが、古いコードによって読み取られないだけである。セッションごとの CCR コンテンツストアサイドカーも書き込みが停止する - full モードでは、すべての読み取りで最初にインデックスの ccr ミラーが参照されるため、ミラーが永続コピーとなる。jsonWriter=off レーンは、そのセッションディレクトリの唯一のライターでなければならない: off レーンは統合優先で読み取り、決してファイルにフォールバックしないため、同じディレクトリに対して legacy/shadow で実行されるピアインスタンスは静かに分岐する (その JSON コピーは更新を受け取らなくなり、off レーンはピアの書き込みを決して見ない)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal の WAL 回収上限 (#3250)、バイト単位: 完了したチェックポイントごとに WAL ファイルがこのサイズまで切り詰められ、長寿命プロセスにおけるディスク使用量とクラッシュ復旧リプレイを制限する。セッションごとのデータクォータではない - データは書き込まれた時点で index.db に着地し、WAL はチェックポイント間のフレームをステージングするだけなので、セッション数/サイズはこれをスケーリングしない; 大きなバーストは、次の完了チェックポイントが切り詰めるまで一時的に超過しうる。高い = 切り詰め IO が稀になるが、定常時のディスク使用量が増え、クラッシュリプレイのテールが長くなる。環境変数階層 (正の整数) がファイル階層より優先される; ゴミまたは非正の環境変数値は、起動を失敗させるのではなくファイル/デフォルト階層にフォールスルーする。レーン起動時にストアが開かれる時点で適用される - ホット編集には再起動が必要。有効な値は統合有効ログ行とストレージ診断ブロブに表示される。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | 日単位の GC 年齢スイープウィンドウ (#3463, #3901): これより長くアイドル状態のセッションはアーカイブされる - 単一のアーカイブログ (セッションディレクトリ配下の archive/archive.jsonl — セッションごとのファイルスプレーなし) に 1 行の JSON が追記され、トombストーン行がラベルとアイデンティティを保持し、コンテンツ/メディア参照が解放され、リストアは 1 つの POST のみで可能。0 は年齢スイープを無効にする; POST /__bili/storage/gc は常に実行され、既存の index.db を縮小するには ?vacuum=full を取る (新規書き込みには常に auto_vacuum=INCREMENTAL が有効)。同じ操作に CLI レイヤーがある (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run]、および bili sessions ls/rm/archive/restore (README の 'Storage maintenance CLI' セクション)。環境変数がファイル階層より優先される。 | -| `sessions` | object | {} (all defaults) | — | セッションテーブルサイズ上限とオプトインによるゴミ収集。 | -| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | メモリ内セッションテーブル上限 (アイドルセッションの LRU 退避)。 | -| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界な起動 (#2803/#3050): リスニング前に起動時に水合される最大最近保存セッション数; 古い状態はオンデマンドで再ロードされる。デフォルト 0 (急激なウォームセットなし — 最初のタッチが 1 回のオンデマンドロードを支払う; 急激な動作を復元するには 8 を設定)。sessions.max で上限が設定される。 | -| `sessions.warmMaxBytes` | number | 67108864 | BILI_SESSION_WARM_MAX_BYTES | 起動ウォームセットのデコード済みバイト予算 (#2803)。ディスク上の圧縮バイトではなくデコード済みサイズをカウントするため、zstd アーカイブが RAM 境界をすり抜けることはない。予算を超える単一のファイルは決してウォームされない。 | -| `sessions.residentMaxBytes` | number | 4294967296 | BILI_SESSION_RESIDENT_MAX_BYTES | 推定リザident セッションバイトのソフトターゲット (#2803; #5779 で 512 MiB から 4 GiB に引き上げ — 単一のセッションは通常 512 MiB を超え、より小さい予算に対して永遠にピンポンしていた)。アクティブ/進行中の状態はそれより上にピン留めされる; ターゲット超過時、アイドル状態はリクエスト完了時にディスクへ排出される。メモリが少ないマシンではこれを下げる。 | -| `sessions.hardMaxBytes` | number | (same as `sessions.residentMaxBytes`) | BILI_SESSION_RESIDENT_HARD_MAX_BYTES | セッションごとの推定リザident バイトのハード上限 (#4788, 決定 #4792 D1-A)。プールソフト上限とは異なり — それは他のコールドセッションのみを退避でき、単独アクティブセッションには決して触れない — これは 1 つのセッションを制限する: リクエスト終了境界で上限超過時、そのドロップ可能なフィールド (フォールドブロックのオリジナル、メタデータ台帳、有界ビューを超える生テール) はプール免除セッションであっても強制的にディスクへ戻され、次のターンは狭いディスク読み取りを通じて必要なものを透明に再読み取りする。最後の保存が確実に着地したことが証明される限り、安全なドロップはすべてのリクエスト境界で実行される; 上限はセッションごとのコールアウト (警告、60 秒にスロットリング) を追加する。`0` は上限チェックを無効にする。 | -| `sessions.contentStorePin` | boolean | false | BILI_SESSION_CONTENT_STORE_PIN | #4788 フェーズ2の常駐ドロップのエスケープハッチ: 各セッションのコンテンツストアをメモリに固定する(折りたたみは行われない)。デフォルトはオフ: 検証済みの保存が完了するたびに、アイドル状態のセッションのコンテンツストアはリクエスト間でドロップされ、統合インデックスミラーまたはサイドカーファイルからオンデマンドで再読み込みされる。検証済みのコピーがないストアを持つセッション(例: 新しいフォーク子プロセス)は自動的に常駐する。RAMを犠牲にしてゼロの再読み込みレイテンシが必要な場合にこれを設定する。 | -| `sessions.gc.enabled` | boolean | false | BILI_SESSION_GC | 古い永続化セッションのオプトインによるガベージコレクション(このフラグなしでユーザーデータを静かに削除することはない)。 | -| `sessions.gc.maxAgeDays` | number | 7 | BILI_SESSION_GC_MAX_AGE_DAYS | GCの年齢しきい値(日)。 | -| `sessions.gc.maxTokens` | number | 1000000 | BILI_SESSION_GC_MAX_TOKENS | セッションレコードごとのGCトークンサイズしきい値。 | -| `sessions.gc.intervalMs` | number | 3600000 | BILI_SESSION_GC_INTERVAL_MS | GCスイープ間隔。 | -| `plugin.snapshotCapBytes` | number | 104857600 (0 disables snapshots) | BILI_PUBLIC_SNAPSHOT_CAP_BYTES | ネイティブプラグインに配信されるフォークAPIパブリックスナップショットのサイズ上限。 | -| `snapshot.shadow` | boolean | true (on) | — | スナップショットレンダリングのシャドウ観測可能性(#3756のフォローアップ): レンダリングレーンがまだロールアウトゲートされている間、すべてのアームされたターンがスナップショットビューを計算し、レガシーアウトバウンドとの差分を分類し、セッションごとのカウンターを記録する(Webパネル行 + /__bili/status の snapshotShadow 集計 + snapshot-shadow 診断トピック)。デフォルトはオン - これはロールアウト切り替えをゲートするエラー率を持つフリートワイドのドライランである。環境変数 BILI_SNAPSHOT_SHADOW が優先される。追加のターンごとの計算を望まない場合は false/0/off でオフにする。 | **更新 & アドバイズリー** @@ -1986,7 +1976,6 @@ ACP ネイティブエージェント(現在は `pi` 拡張機能)は、プ | `BILI_LOG_MASK_HOSTS` | `maskHosts` | true | | `BILI_LOG_RETENTION_HOURS` | `diagnostics.logRetentionHours` | 48 (floor 1) | | `BILI_MAX_MESSAGES` | `network.maxMessages` | 0 | -| `BILI_MAX_SESSIONS` | `sessions.max` | 256 | | `BILI_MAX_SHRINK_PER_COMPRESS` | `network.maxShrinkPerCompress` | unset | | `BILI_MCP_PROXY` | `native.attachUrl` | unset (discover per lane) | | `BILI_MESSAGE_TIMESTAMPS` | `compress.messageTimestamps` | true | @@ -2013,7 +2002,6 @@ ACP ネイティブエージェント(現在は `pi` 拡張機能)は、プ | `BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS` | `network.preflightDeadEndCooldownMs` | 300000 | | `BILI_PREFLIGHT_HOLD_MS` | `network.preflightHoldMs` | 30000 | | `BILI_PROXY_KEEPALIVE_MAX_MS` | `network.proxyKeepAliveMaxMs` | 55000 (0 = one-shot connections) | -| `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | `plugin.snapshotCapBytes` | 104857600 (0 disables snapshots) | | `BILI_RECEIVE_BODY_BYTES` | `network.receiveBodyBytes` | 5368709120 | | `BILI_RELEASE_NOTES_CHECK` | `releaseNotesCheck` | true | | `BILI_RELEASE_NOTES_URL` | `releaseNotesUrl` | unset (built-in feed) | @@ -2025,18 +2013,12 @@ ACP ネイティブエージェント(現在は `pi` 拡張機能)は、プ | `BILI_RESIGN_PASSTHROUGH` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_RESUME_INHERITANCE` | `resumeInheritance` | true | | `BILI_ROLLOUT_` | `rollout.features.enabled` | unset (bucket math decides) | +| `BILI_ROLLOUT_EGRESS_PLANE` | `rollout.features.egressPlane` | 0 (off) | | `BILI_ROLLOUT_IDENTITY_CHANNEL_HARVEST` | `rollout.features.identityChannelHarvest` | 0 (off) | +| `BILI_ROLLOUT_PLUGIN_DELTA_RESEND` | `rollout.features.pluginDeltaResend` | 0 (off) | | `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK` | `rollout.features.pluginFullResendWatermark` | 5 (experimental grey) | -| `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 0 (off) | -| `BILI_SESSION_CONTENT_STORE_PIN` | `sessions.contentStorePin` | false | -| `BILI_SESSION_GC` | `sessions.gc.enabled` | false | -| `BILI_SESSION_GC_INTERVAL_MS` | `sessions.gc.intervalMs` | 3600000 | -| `BILI_SESSION_GC_MAX_AGE_DAYS` | `sessions.gc.maxAgeDays` | 7 | -| `BILI_SESSION_GC_MAX_TOKENS` | `sessions.gc.maxTokens` | 1000000 | -| `BILI_SESSION_RESIDENT_HARD_MAX_BYTES` | `sessions.hardMaxBytes` | (same as `sessions.residentMaxBytes`) | -| `BILI_SESSION_RESIDENT_MAX_BYTES` | `sessions.residentMaxBytes` | 4294967296 | -| `BILI_SESSION_WARM_MAX` | `sessions.warmMax` | 0 | -| `BILI_SESSION_WARM_MAX_BYTES` | `sessions.warmMaxBytes` | 67108864 | +| `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 100 (on, GA) | +| `BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL` | `rollout.features.pluginMsgIdIncremental` | 100 (on, GA) | | `BILI_SQLITE_AUTO_DOWNLOAD` | `storage.sqliteAutoDownload` | true | | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | `storage.archiveAfterDays` | 365 | diff --git a/docs/configuration.md b/docs/configuration.md index a2ba64df70..1bc01de4cc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -106,7 +106,10 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `rollout.features.salt` | string | the feature name | — | Hash salt for bucketing. Changing it re-randomizes the whole cohort (the only way to reshuffle; raising percent alone never reshuffles). | | `rollout.features.base` | string | unset (roll out over everyone) | — | Nested rollout: evaluate this feature only inside the named feature's enabled cohort, so the percent is relative to that population (roll B out to A's users; chains allowed, cycles fail closed). | | `rollout.features.identityChannelHarvest` | boolean \| number | 0 (off) | BILI_ROLLOUT_IDENTITY_CHANNEL_HARVEST | Identity-channel harvest cohort gate (#5825): allows the loopback-only transcript/sqlite harvest (src/identity-harvest.ts: claude-code / codebuddy / mcode / goose / qwen lanes) without any per-install env switch, when neither the per-install switch (BILI_IDENTITY_HARVEST) nor this key is set nothing ever reads client storage. Owner ruling 2026-11-07: this cohort key is deliberately separate from rollout.features.pluginLaneServerHistory (which governs row PINNING of admitted ids) — either enabler may activate harvesting. Default 0%; raise after small-cohort observation of the degrade warn-once rate. See identity-channels.md. | -| `rollout.features.pluginLaneServerHistory` | boolean \| number | 0 (off) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | Plugin-lane server-owned history (#4491): for enabled installs, (a) plugin-lane requests may carry an internal `biliMsgIds` array parallel to `messages` that pins message identity to client-supplied ids (always stripped before upstream forwarding; malformed arrays are ignored with a warning, id-poisoning shapes hard-fail on the incremental lane); (b) the loopback-only `POST /__bili/plugin/render` endpoint serves the outbound prefix rendered from the unified transcript (folded regions collapse to summary anchors; any desync degrades to a non-2xx so the shim falls back to its local history); (c) the incremental-lane probe advertises `serverHistory`. Default 0% — no behavior change until rolled out. | +| `rollout.features.pluginLaneServerHistory` | boolean \| number | 100 (on, GA) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | Plugin-lane server-owned history (#4491): for enabled installs, (a) plugin-lane requests may carry an internal `biliMsgIds` array parallel to `messages` that pins message identity to client-supplied ids (always stripped before upstream forwarding; malformed arrays are ignored with a warning, id-poisoning shapes hard-fail on the incremental lane); (b) the loopback-only `POST /__bili/plugin/render` endpoint serves the outbound prefix rendered from the unified transcript (folded regions collapse to summary anchors; any desync degrades to a non-2xx so the shim falls back to its local history); (c) the incremental-lane probe advertises `serverHistory`. Default 100% (GA, owner ruling 2026-11-13); set 0 to roll an install back to the pre-GA state. | +| `rollout.features.pluginMsgIdIncremental` | boolean \| number | 100 (on, GA) | BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL | Incremental msgid pin channel (#7064 §8, owner rulings 2026-11-18/19): steady plugin-lane turns declare their message ids in TAIL FORM — the bounded `x-bili-pf-tail-ids` header (anchor + new ids, ≤ 2048 ids, `?` placeholder slots) or a short body id array — instead of the FULL list; the server re-stamps the prefix ids from its own ledger live rows (ingest is INSERT OR IGNORE on (session_id,seq) — a position's msg_id is written once, never overwritten). Acceptance (2026-11-19 form): the declared tail [T-L, T-1] must match the ledger overlap segment row-by-row (any drift degrades to `full`, never 400); ultra-long sessions (> 2048 ids) pin a truncated tail the same way — the longer overlap proves MORE. Only snapshot rounds (empty ledger / post-fold re-anchor) send the full form; full form always wins when both arrive. The response carries `x-bili-msgid-mode: incremental\|full` (cleared on any 4xx; one learning point next to x-bili-pf-seq). Any mismatch self-heals within one round. Default 100%; set 0 to roll back to full-list pinning (byte-identical to pre-feature). Requires pluginLaneServerHistory. See docs/session-identity-resolution.md. | +| `rollout.features.egressPlane` | boolean \| number | 0 (off) | BILI_ROLLOUT_EGRESS_PLANE | The egress plane — ONE switch for the whole new path (#7398/#7439; replaces the retired egressOpLog/egressServeLog pair). ON: every wire lane (OpenAI/Anthropic/Responses/Google) records its outbound conversation view as sparse ops (I/D/U) into the egress_state/egress_flow dual tables — rows are SELF-CONTAINED (message bytes ride the row; no content CAS, ruling 7439-1) — and serves the outbound core-message portion back from the op log, byte-identical by construction (the row bytes are the pipeline's own final core JSON); pinned plugin lanes additionally take server authority — the kernel input is replayed from the flow (rulings 7439-3/5). A bounded backward ladder recovers to the newest healthy log point; ANY unrebuildable row (missing/empty bytes, unknown role), corrupt log, or probe failure degrades that turn to the legacy chain — never a partial swap, never client-as-authority. The system block stays live-built in both paths (prefix-cache anchor discipline). SQLite lane only; title-gen/compaction side requests excluded. Default 0 = off: legacy chain, zero rows, byte-identical to pre-feature. | +| `rollout.features.pluginDeltaResend` | boolean \| number | 0 (off) | BILI_ROLLOUT_PLUGIN_DELTA_RESEND | Delta resend — the tail-slice form of the full-resend watermark lane: steady plugin-lane turns send x-bili-pf-delta: 1 + body.messages = ONLY the new elements (L = T − stampT, no prefix); the server rebuilds the view from the stashed prefix cores plus the parsed tail (same hit provenance as the full lane). ANY unsatisfiable claim (count/body/tail-ids width mismatch, anchor-chain break after a known list, rollout off) returns 400 pf_delta_reanchor + x-bili-msgid-mode: full and the client resends that turn in full form — one-round convergence, never a silent misparse. OFF by default: delta bodies are unprocessable by servers that do not ship this arm, so the key stays 0 until client send-sides exist. See technical-notes.md “Delta resend”. | | `rollout.features.pluginFullResendWatermark` | boolean \| number | 5 (experimental grey) | BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK | Full-resend watermark fastpath (#5403/#5418): plugin-lane chat-completions turns whose previous response carried an x-bili-pf-seq stamp assert the resent prefix via the echoed count (structural scan, no full parse); on a hit the pipeline consumes stashed canonical prefix cores plus only the new tail window (byte-identical outbound provenance; ANY doubt falls back to the untouched full-parse path per turn and re-stamps, five credited misses hold the lane for one hour). Requires the client to echo the stamp back as x-bili-pf-count (dsh/pi/opencode/omp do; unadapted lanes never hit). EXPERIMENTAL grey rollout — ON by default for roughly 5% of installs (owner ruling 2026-11-02: small cohort before GA; one-time intro notice in the web UI for every install, tri-state switch in quick config). Force it with `true`, turn off with `false` (or a percent) / env = 0. See technical-notes.md “Full-resend watermark fastpath”. | | `maskHosts` | boolean | true | BILI_LOG_MASK_HOSTS | Mask non-public target hosts in logs as ; credential headers are always masked regardless. | | `security.adminRelay` | boolean | false | — | Let an embedding host (dsh web today) relay the ENTIRE /__bili/ admin surface through its own entry origin so a remote browser can open the panel (#3373). Fail-closed, file-only. The relayed hop reaches the proxy as a local non-browser client (loopback source + loopback Host, no Origin), so the admin gate stays unchanged; enabling it exposes config editing and credential writes to whoever can reach the host web entry — protect that entry (auth / reverse proxy) first. | @@ -215,19 +218,6 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 envelope switch (#2960) — default flipped this release: UNSET now defaults to OFF for the pure-sqlite selection (storage.unified absent or "sqlite") — the SQLite index is the only persistent copy (reads are already unified-first; boot enumerates the index union, so index-only sessions load normally). Explicit full keeps the historical #3818 dual-write default (envelopes stay written unless this key says otherwise). An explicit on/off at either tier always wins in effective-full mode: under shadow/legacy (or with encryption active) a requested off is ignored with a one-time warning because the JSON file remains the sole source of truth. Env tier 0/false/off = off and 1/true/on = on wins over the strict file spellings ("off"/"on"). Fail-safe: if the sidecar disables itself mid-run, envelope writes resume on the next save. To restore dual-write: set storage.jsonWriter=on (or BILI_STORAGE_JSON_WRITER=1) — index-only sessions keep loading via the boot union, and replicas/restore re-materialize portable JSON envelopes. Downgrading the binary (or losing the SQLite engine) makes index-only sessions invisible to v0.1.196 and older until a >=v0.1.197 binary runs again - the data stays intact in index.db, it is just not read by the older code. The per-session CCR content-store sidecar stops being written as well - in full mode the index's ccr mirror is consulted first on every read, so the mirror is the durable copy. A jsonWriter=off lane must be the SOLE writer of its sessions directory: an off lane reads unified-first and never falls back to the file, so any peer instance running legacy/shadow against the same directory diverges silently (its JSON copy stops receiving updates, and the off lane never sees the peer's writes). | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | WAL reclamation cap for index.db-wal (#3250), in bytes: every completed checkpoint truncates the WAL file back to this size, bounding disk use and crash-recovery replay in long-lived processes. NOT a per-session data quota - data lands in index.db as it is written and the WAL only stages frames between checkpoints, so session count/size does not scale this; a large burst can overshoot temporarily until the next completed checkpoint truncates back. Higher = rarer truncate IO but more steady disk use and a longer crash-replay tail. Env tier (positive integer) wins over the file tier; a garbage or non-positive env value falls through to the file/default tier instead of failing boot. Applied when the store opens at lane boot - hot edits need a restart. The effective value appears on the unified-enable log line and in the storage diagnostics blob. | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier. | -| `sessions` | object | {} (all defaults) | — | Session-table size cap and opt-in garbage collection. | -| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | In-memory session table cap (LRU eviction of idle sessions). | -| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max. | -| `sessions.warmMaxBytes` | number | 67108864 | BILI_SESSION_WARM_MAX_BYTES | Decoded-byte budget for the boot warm set (#2803). Counts decoded size, not compressed on-disk bytes, so zstd archives cannot sneak past the RAM bound. A single file over the budget is never warmed. | -| `sessions.residentMaxBytes` | number | 4294967296 | BILI_SESSION_RESIDENT_MAX_BYTES | Soft target for estimated resident session bytes (#2803; raised from 512 MiB to 4 GiB in #5779 — single sessions routinely exceed half a GiB and ping-ponged against the smaller budget forever). Active/in-flight state stays pinned above it; when over target, idle state discharges back to disk as requests complete. Lower it if your machine has little RAM. | -| `sessions.hardMaxBytes` | number | (same as `sessions.residentMaxBytes`) | BILI_SESSION_RESIDENT_HARD_MAX_BYTES | Per-SESSION hard cap on estimated resident bytes (#4788, decision #4792 D1-A). Unlike the pool soft cap — which can only evict OTHER cold sessions and never touches a sole-active session — this bounds ONE session: over the cap at a request-end boundary, its droppable fields (folded-block originals, metadata ledger, the raw tail beyond the bounded view) are forced back to disk even for a pool-exempt session, and the next turn transparently re-reads what it needs via narrow disk reads. Safe drops run at every request boundary as long as the last save has provably landed; the cap adds the per-session call-out (warn, throttled to 60 s). `0` disables the cap check. | -| `sessions.contentStorePin` | boolean | false | BILI_SESSION_CONTENT_STORE_PIN | Escape hatch for the #4788 phase-2 resident drop: keep each session's content store pinned in memory (nothing ever folded away). Default off: after every proven-landed save, idle sessions' content stores are dropped between requests and re-read on demand from the unified index mirror or the sidecar file; sessions whose store has no provably-landed copy (e.g. fresh fork children) stay resident automatically. Set this if you need zero re-read latency at the cost of RAM. | -| `sessions.gc.enabled` | boolean | false | BILI_SESSION_GC | Opt-in garbage collection of stale persisted sessions (user data is never pruned silently without this flag). | -| `sessions.gc.maxAgeDays` | number | 7 | BILI_SESSION_GC_MAX_AGE_DAYS | GC age threshold in days. | -| `sessions.gc.maxTokens` | number | 1000000 | BILI_SESSION_GC_MAX_TOKENS | GC token-size threshold per session record. | -| `sessions.gc.intervalMs` | number | 3600000 | BILI_SESSION_GC_INTERVAL_MS | GC sweep interval. | -| `plugin.snapshotCapBytes` | number | 104857600 (0 disables snapshots) | BILI_PUBLIC_SNAPSHOT_CAP_BYTES | Size cap for fork-API public snapshots served to native plugins. | -| `snapshot.shadow` | boolean | true (on) | — | Snapshot-render shadow observability (#3756 follow-up): with the render lane still rollout-gated, every armed turn computes the would-be snapshot view, classifies the diff against the legacy outbound, and records per-session counters (web panel rows + /__bili/status snapshotShadow aggregate + the snapshot-shadow diagnostics topic). Default ON - it is the fleet-wide dry run whose error rate gates the rollout flip. Env BILI_SNAPSHOT_SHADOW wins. Turn off with false/0/off if you do not want the extra per-turn computation. | **Updates & advisories** @@ -2000,7 +1990,6 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_LOG_MASK_HOSTS` | `maskHosts` | true | | `BILI_LOG_RETENTION_HOURS` | `diagnostics.logRetentionHours` | 48 (floor 1) | | `BILI_MAX_MESSAGES` | `network.maxMessages` | 0 | -| `BILI_MAX_SESSIONS` | `sessions.max` | 256 | | `BILI_MAX_SHRINK_PER_COMPRESS` | `network.maxShrinkPerCompress` | unset | | `BILI_MCP_PROXY` | `native.attachUrl` | unset (discover per lane) | | `BILI_MESSAGE_TIMESTAMPS` | `compress.messageTimestamps` | true | @@ -2027,7 +2016,6 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS` | `network.preflightDeadEndCooldownMs` | 300000 | | `BILI_PREFLIGHT_HOLD_MS` | `network.preflightHoldMs` | 30000 | | `BILI_PROXY_KEEPALIVE_MAX_MS` | `network.proxyKeepAliveMaxMs` | 55000 (0 = one-shot connections) | -| `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | `plugin.snapshotCapBytes` | 104857600 (0 disables snapshots) | | `BILI_RECEIVE_BODY_BYTES` | `network.receiveBodyBytes` | 5368709120 | | `BILI_RELEASE_NOTES_CHECK` | `releaseNotesCheck` | true | | `BILI_RELEASE_NOTES_URL` | `releaseNotesUrl` | unset (built-in feed) | @@ -2039,18 +2027,12 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_RESIGN_PASSTHROUGH` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_RESUME_INHERITANCE` | `resumeInheritance` | true | | `BILI_ROLLOUT_` | `rollout.features.enabled` | unset (bucket math decides) | +| `BILI_ROLLOUT_EGRESS_PLANE` | `rollout.features.egressPlane` | 0 (off) | | `BILI_ROLLOUT_IDENTITY_CHANNEL_HARVEST` | `rollout.features.identityChannelHarvest` | 0 (off) | +| `BILI_ROLLOUT_PLUGIN_DELTA_RESEND` | `rollout.features.pluginDeltaResend` | 0 (off) | | `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK` | `rollout.features.pluginFullResendWatermark` | 5 (experimental grey) | -| `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 0 (off) | -| `BILI_SESSION_CONTENT_STORE_PIN` | `sessions.contentStorePin` | false | -| `BILI_SESSION_GC` | `sessions.gc.enabled` | false | -| `BILI_SESSION_GC_INTERVAL_MS` | `sessions.gc.intervalMs` | 3600000 | -| `BILI_SESSION_GC_MAX_AGE_DAYS` | `sessions.gc.maxAgeDays` | 7 | -| `BILI_SESSION_GC_MAX_TOKENS` | `sessions.gc.maxTokens` | 1000000 | -| `BILI_SESSION_RESIDENT_HARD_MAX_BYTES` | `sessions.hardMaxBytes` | (same as `sessions.residentMaxBytes`) | -| `BILI_SESSION_RESIDENT_MAX_BYTES` | `sessions.residentMaxBytes` | 4294967296 | -| `BILI_SESSION_WARM_MAX` | `sessions.warmMax` | 0 | -| `BILI_SESSION_WARM_MAX_BYTES` | `sessions.warmMaxBytes` | 67108864 | +| `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 100 (on, GA) | +| `BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL` | `rollout.features.pluginMsgIdIncremental` | 100 (on, GA) | | `BILI_SQLITE_AUTO_DOWNLOAD` | `storage.sqliteAutoDownload` | true | | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | `storage.archiveAfterDays` | 365 | diff --git a/docs/configuration.zh-CN.md b/docs/configuration.zh-CN.md index 94d38a80f2..99305b78cf 100644 --- a/docs/configuration.zh-CN.md +++ b/docs/configuration.zh-CN.md @@ -106,7 +106,10 @@ | `rollout.features.salt` | string | the feature name | — | 分桶哈希盐。修改它会重新随机整个分组(唯一的洗牌方式;仅上调 percent 永不洗牌)。 | | `rollout.features.base` | string | unset (roll out over everyone) | — | 嵌套灰度:仅在所指特性的已入组人群内评估本特性,percent 相对该人群(B 只放量给 A 的用户;允许链式,环路判为关)。 | | `rollout.features.identityChannelHarvest` | boolean \| number | 0(关) | BILI_ROLLOUT_IDENTITY_CHANNEL_HARVEST | 身份信道采集的灰度门(#5825):当安装级开关(BILI_IDENTITY_HARVEST)未设置时,允许仅由本灰度键激活回环专属的转录/SQLite 采集(claude-code / codebuddy / mcode / goose / qwen 车道);两者都不设则永不读客户端存储。owner 裁定 2026-11-07:本灰度键与 rollout.features.pluginLaneServerHistory(它管已接纳 id 的行钉住)刻意分离——任一使能方可激活采集。默认 0%,小流量观察退化率后再抬。见 identity-channels.zh-CN.md。 | -| `rollout.features.pluginLaneServerHistory` | boolean \| number | 0(关) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | 插件车道服务端正史(#4491):对已放量的安装,(a) 插件车道请求可携带与 `messages` 平行的内部字段 `biliMsgIds`,把消息身份钉在客户端提供的 id 上(转发上游前一律剥掉;格式错误的数组告警后忽略,身份投毒形态在增量车道直接硬失败);(b) 仅回环可达的 `POST /__bili/plugin/render` 端点从统一存储渲染出站前缀(折叠区收敛为摘要锚点;任何失步都返回非 2xx,shim 退回本地历史);(c) 增量车道探测响应通告 `serverHistory`。默认 0%,放量前无行为变化。 | +| `rollout.features.pluginLaneServerHistory` | boolean \| number | 100(开,已 GA) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | 插件车道服务端正史(#4491):对已放量的安装,(a) 插件车道请求可携带与 `messages` 平行的内部字段 `biliMsgIds`,把消息身份钉在客户端提供的 id 上(转发上游前一律剥掉;格式错误的数组告警后忽略,身份投毒形态在增量车道直接硬失败);(b) 仅回环可达的 `POST /__bili/plugin/render` 端点从统一存储渲染出站前缀(折叠区收敛为摘要锚点;任何失步都返回非 2xx,shim 退回本地历史);(c) 增量车道探测响应通告 `serverHistory`。默认 100%(GA,owner 裁定 2026-11-13);设 0 可把安装回滚到 GA 前状态。 | +| `rollout.features.pluginMsgIdIncremental` | boolean \| number | 100(开,已 GA) | BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL | 增量 msgid pin 通道(#7064 §8,owner 裁定 2026-11-18/19):稳态插件轮用**尾形态**申报消息 id——有界的 `x-bili-pf-tail-ids` 头(锚点+新增,≤ 2048 个,`?` 占位槽)或短 body id 数组——替代全量清单;服务端从自己账本的 live 行重盖前缀 id(ingest 是 INSERT OR IGNORE on (session_id,seq)——位置上的 msg_id 只写一次,永不改写)。接受规则(2026-11-19 定型):声明的尾段 [T-L, T-1] 与账本重叠段逐行相等(任一行漂移⇒降级 full,永不 400);超长会话(>2048)截断的尾清单同样可 pin——重叠段更长、证明更强。只有快照轮(账本空/折叠后重锚)发全量形态;全量优先于尾形态同时到达。响应携带 `x-bili-msgid-mode: incremental\|full`(4xx 即清,与 x-bili-pf-seq 同一学习点)。任何失配一轮内自愈。默认 100%;设 0 回滚到全量清单 pin(与特性前字节一致)。需先开 pluginLaneServerHistory。见 session-identity-resolution.zh-CN.md。 | +| `rollout.features.egressPlane` | boolean \| number | 0(关) | BILI_ROLLOUT_EGRESS_PLANE | egress 平面——整条新链路的**唯一开关**(#7398/#7439;取代已退役的 egressOpLog/egressServeLog 双键)。开:四条 wire 车道(OpenAI/Anthropic/Responses/Google)每轮把出站会话视图记为稀疏 op(I/D/U)写入 egress_state/egress_flow 双表——行**自包含**(消息字节随行走,不经内容 CAS,裁定 7439-1)——并从 op 日志回放重建出站核心消息段,按构造字节一致(行内字节就是管线自己的最终 core JSON);pinned 插件车道另获服务端权威——内核输入从流水回放(裁定 7439-3/5)。有界回溯阶梯恢复到最近健康日志点;任何不可重建行(缺/空字节、未知 role)、日志损坏或探针失败,整轮降级 legacy 链路——绝不部分替换、绝不客户端为准。system 块两条路径都保持 live 构建(前缀缓存锚点纪律)。仅 SQLite 车道;标题生成/压缩侧路请求排除。默认 0 = 关:legacy 链路,零行,与特性前字节一致。 | +| `rollout.features.pluginDeltaResend` | boolean \| number | 0(关) | BILI_ROLLOUT_PLUGIN_DELTA_RESEND | Delta 重发——全量重发水位快路径的尾切片形态:稳态轮发 x-bili-pf-delta: 1 + body.messages = **只有**新元素(L = T − stampT,无前置);服务端用钉存前缀核 + 解析出的尾部重建视图(与全量车道同源的命中可证)。任何不成立的声明(count/body/tail-ids 宽度不符、已知清单后锚链断裂、灰度关)返回 400 pf_delta_reanchor + x-bili-msgid-mode: full,客户端当轮改发全量形态——一轮收敛,绝不静默错解。默认**关**:delta body 对未上车的服务端不可处理,故保持 0 直到客户端发送端就位。见 technical-notes.zh-CN.md「Delta 重发」。 | | `rollout.features.pluginFullResendWatermark` | boolean \| number | 5(实验性灰度) | BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK | 全量重发水位快路径(#5403/#5418):插件车道 chat-completions 回合若上一轮响应盖有 x-bili-pf-seq,则用回声计数声明重发前缀(结构扫描,不做全量解析);命中时管线只消费钉存的前缀核 + 新增尾巴窗口(出站字节同源可证;任何疑点当轮回退未动过的全量解析路径并重新盖戳,连续五次计名失配将车道按住一小时)。需客户端把戳回声为 x-bili-pf-count(适配的车道回带;其余车道不会命中)。**实验性灰度**——默认仅约 5% 的安装开启(owner 裁定 2026-11-02:先小流量再 GA;web 面板对全部安装显示一次性公告、快速配置提供三态开关);`true` 强制开、`false`(或百分比)/ env = 0 关。见 technical-notes.zh-CN.md「全量重发水位快路径」。 | | `maskHosts` | boolean | true | BILI_LOG_MASK_HOSTS | 日志中把非公开目标主机遮成 ;凭据头无论此开关如何总是遮盖。 | | `security.adminRelay` | boolean | false | — | 允许宿主(当前为 dsh web)把它自己入口域名下的整个 /__bili/ 管理面中继到本机代理,使远程浏览器能打开面板(#3373)。默认关闭、仅配置文件。中继跳到达代理时是本机非浏览器客户端(回环源地址 + 回环 Host、无 Origin 头),管理端门槛不变;开启后,凡能访问宿主 web 入口的人都能改配置、写凭据——请先给入口加鉴权/反代。 | @@ -215,19 +218,6 @@ | `storage.jsonWriter` | on\|off | unset (off for the pure-sqlite selection — absent/unified=sqlite — and on for explicit full) | BILI_STORAGE_JSON_WRITER | Phase 3 信封开关(#2960)—— 本次发布翻转缺省:UNSET(未设置)现在对纯 SQLite 选择(storage.unified 未设置或 "sqlite")缺省为 OFF —— SQLite 索引成为唯一持久副本(读路径本就 unified-first;boot 枚举索引并集,纯索引会话可正常加载)。显式 full 保留 #3818 的历史双写缺省(除非本键另说,信封照写)。有效 full 模式下任一层的显式 on/off 总是生效:shadow/legacy(或加密激活)下请求的 off 会被忽略并警告一次,因为 JSON 文件仍是唯一事实源。环境变量层 0/false/off = off、1/true/on = on,优先于严格拼写的文件层("off"/"on")。失败安全:若边车运行中自禁用,下次保存即恢复信封写入。恢复双写:设 storage.jsonWriter=on(或 BILI_STORAGE_JSON_WRITER=1)—— 纯索引会话经 boot 并集继续可加载,replicas/restore 可重新物化可移植的 JSON 信封。降级二进制(或丢失 SQLite 引擎)会让纯索引会话在 v0.1.196 及更早版本上不可见,直到重新运行 >=v0.1.197 —— 数据在 index.db 中完好无损,只是旧代码不读它。每会话的 CCR content-store 边车也不再写 —— full 模式下读取时先查索引里的 ccr 镜像,镜像才是持久副本。jsonWriter=off 的 lane 必须是其会话目录的唯一写者:off lane 读路径 unified-first、永不回退到文件,因此同一目录上跑 legacy/shadow 的对等实例会静默分叉(它的 JSON 副本收不到更新,off lane 也看不到对等实例的写入)。 | | `storage.walJournalSizeLimitBytes` | number | 8388608 | BILI_STORAGE_WAL_JOURNAL_SIZE_LIMIT_BYTES | index.db-wal 的 WAL 回收上限(#3250),单位字节:每次完成的 checkpoint 把 WAL 文件截回该大小,约束长驻进程的磁盘占用与崩溃恢复重放。不是每会话数据配额 —— 数据写入时就已落进 index.db,WAL 只是两次 checkpoint 之间的帧暂存区,会话数量/大小不会放大它;大 burst 可能暂时超出,直到下一次完成的 checkpoint 截回。调大 = truncate IO 更稀疏,但稳态磁盘占用更多、崩溃恢复重放尾更长。环境变量层(正整数)优先于配置文件层;env 值非法或非正数时静默落到文件/默认层,boot 不崩。lane 启动、store open 时生效 —— 热改动需重启。实际取值会出现在 unified-enable 日志行与 storage 诊断块中。 | | `storage.archiveAfterDays` | integer | 365 | BILI_STORAGE_ARCHIVE_AFTER_DAYS | GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full\|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。 | -| `sessions` | object | {} (all defaults) | — | 会话表容量上限与可选垃圾回收。 | -| `sessions.max` | number | 256 | BILI_MAX_SESSIONS | 内存会话表上限(空闲会话 LRU 逐出)。 | -| `sessions.warmMax` | number | 0 | BILI_SESSION_WARM_MAX | 有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。 | -| `sessions.warmMaxBytes` | number | 67108864 | BILI_SESSION_WARM_MAX_BYTES | 启动预热集的解码字节预算(#2803)。按解码后大小计而非磁盘压缩字节,zstd 归档无法绕过内存上限;单文件超预算则永不预热。 | -| `sessions.residentMaxBytes` | number | 4294967296 | BILI_SESSION_RESIDENT_MAX_BYTES | 驻留会话估算字节的软目标(#2803;#5779 从 512 MiB 提高到 4 GiB——单个会话经常就超过半 GiB,旧预算下会被反复逐出、永远驻留不下来)。活跃/在途状态始终固定保留;超过目标时,空闲状态随请求完成回写磁盘释放。内存小的机器可调低。 | -| `sessions.hardMaxBytes` | number | (same as `sessions.residentMaxBytes`) | BILI_SESSION_RESIDENT_HARD_MAX_BYTES | 单会话驻留字节的硬上限(#4788,决策 #4792 D1-A)。不同于池软上限——它只能逐出其他冷会话、从不触碰唯一活跃会话——它约束单个会话:在请求结束边界超过上限时,其可丢弃字段(已折叠块的原文、元数据账本、超出有界视图的原始尾巴)会被强制回写磁盘,即使该会话被池逐出豁免;下一轮通过窄盘读透明地重新读取所需内容。只要上一次保存已被证明落盘,安全丢弃就在每个请求边界运行;该上限额外加入单会话告警(warn,限流至 60 秒)。`0` 关闭该检查。 | -| `sessions.contentStorePin` | boolean | false | BILI_SESSION_CONTENT_STORE_PIN | #4788 二期驻留丢弃的逃生门:把会话内容存储钉在内存(永不折叠)。默认关闭:每次已证明落盘的保存之后,空闲会话的内容存储在请求间隙被释放,需要时从统一索引镜像或伴生文件按需重读;没有可证明落盘副本的会话(如新建 fork 子会话)自动保持驻留。需要零重读延迟且愿意用内存换时可设此项。 | -| `sessions.gc.enabled` | boolean | false | BILI_SESSION_GC | 过期持久化会话的可选垃圾回收(未开启时用户数据绝不被悄悄清理)。 | -| `sessions.gc.maxAgeDays` | number | 7 | BILI_SESSION_GC_MAX_AGE_DAYS | GC 年龄阈值(天)。 | -| `sessions.gc.maxTokens` | number | 1000000 | BILI_SESSION_GC_MAX_TOKENS | GC 单会话记录的 token 大小阈值。 | -| `sessions.gc.intervalMs` | number | 3600000 | BILI_SESSION_GC_INTERVAL_MS | GC 清扫间隔。 | -| `plugin.snapshotCapBytes` | number | 104857600 (0 disables snapshots) | BILI_PUBLIC_SNAPSHOT_CAP_BYTES | 提供给原生插件的 fork API 公共快照大小上限。 | -| `snapshot.shadow` | boolean | true (on) | — | 快照渲染影子对账(#3756 后续):渲染车道仍处灰度关闭时,每轮已折叠会话都现算一次快照视图,与 legacy 实际出站比对分类,逐会话记账(web 面板行 + /__bili/status snapshotShadow 汇总 + snapshot-shadow 诊断主题)。默认开 —— 这是全舰队干跑,其错误率决定灰度翻转。环境变量 BILI_SNAPSHOT_SHADOW 优先。不想要每轮额外计算可用 false/0/off 关闭。 | **更新与公告** @@ -1977,7 +1967,6 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_LOG_MASK_HOSTS` | `maskHosts` | true | | `BILI_LOG_RETENTION_HOURS` | `diagnostics.logRetentionHours` | 48 (floor 1) | | `BILI_MAX_MESSAGES` | `network.maxMessages` | 0 | -| `BILI_MAX_SESSIONS` | `sessions.max` | 256 | | `BILI_MAX_SHRINK_PER_COMPRESS` | `network.maxShrinkPerCompress` | unset | | `BILI_MCP_PROXY` | `native.attachUrl` | unset (discover per lane) | | `BILI_MESSAGE_TIMESTAMPS` | `compress.messageTimestamps` | true | @@ -2004,7 +1993,6 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS` | `network.preflightDeadEndCooldownMs` | 300000 | | `BILI_PREFLIGHT_HOLD_MS` | `network.preflightHoldMs` | 30000 | | `BILI_PROXY_KEEPALIVE_MAX_MS` | `network.proxyKeepAliveMaxMs` | 55000 (0 = one-shot connections) | -| `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | `plugin.snapshotCapBytes` | 104857600 (0 disables snapshots) | | `BILI_RECEIVE_BODY_BYTES` | `network.receiveBodyBytes` | 5368709120 | | `BILI_RELEASE_NOTES_CHECK` | `releaseNotesCheck` | true | | `BILI_RELEASE_NOTES_URL` | `releaseNotesUrl` | unset (built-in feed) | @@ -2016,18 +2004,12 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_RESIGN_PASSTHROUGH` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_RESUME_INHERITANCE` | `resumeInheritance` | true | | `BILI_ROLLOUT_` | `rollout.features.enabled` | unset (bucket math decides) | +| `BILI_ROLLOUT_EGRESS_PLANE` | `rollout.features.egressPlane` | 0 (off) | | `BILI_ROLLOUT_IDENTITY_CHANNEL_HARVEST` | `rollout.features.identityChannelHarvest` | 0 (off) | +| `BILI_ROLLOUT_PLUGIN_DELTA_RESEND` | `rollout.features.pluginDeltaResend` | 0 (off) | | `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK` | `rollout.features.pluginFullResendWatermark` | 5 (experimental grey) | -| `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 0 (off) | -| `BILI_SESSION_CONTENT_STORE_PIN` | `sessions.contentStorePin` | false | -| `BILI_SESSION_GC` | `sessions.gc.enabled` | false | -| `BILI_SESSION_GC_INTERVAL_MS` | `sessions.gc.intervalMs` | 3600000 | -| `BILI_SESSION_GC_MAX_AGE_DAYS` | `sessions.gc.maxAgeDays` | 7 | -| `BILI_SESSION_GC_MAX_TOKENS` | `sessions.gc.maxTokens` | 1000000 | -| `BILI_SESSION_RESIDENT_HARD_MAX_BYTES` | `sessions.hardMaxBytes` | (same as `sessions.residentMaxBytes`) | -| `BILI_SESSION_RESIDENT_MAX_BYTES` | `sessions.residentMaxBytes` | 4294967296 | -| `BILI_SESSION_WARM_MAX` | `sessions.warmMax` | 0 | -| `BILI_SESSION_WARM_MAX_BYTES` | `sessions.warmMaxBytes` | 67108864 | +| `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 100 (on, GA) | +| `BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL` | `rollout.features.pluginMsgIdIncremental` | 100 (on, GA) | | `BILI_SQLITE_AUTO_DOWNLOAD` | `storage.sqliteAutoDownload` | true | | `BILI_STABLE_SYSTEM_ANCHOR` | `stableSystemAnchor` | false | | `BILI_STORAGE_ARCHIVE_AFTER_DAYS` | `storage.archiveAfterDays` | 365 | diff --git a/docs/index.md b/docs/index.md index 102650852b..8b379f94f3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -35,3 +35,4 @@ | [lifecycle](lifecycle.md) | Transition-switch ledger / retirement inventory / hygiene record (CI-enforced `BILI_*` coverage) | | [auto-merge-guardrails](auto-merge-guardrails.md) | Evidence appendix behind §7 of `AGENTS.md` (AI auto-dev review history) | | [unified-loop-spec](unified-loop-spec.md) | Historical design: unified compress-loop spec (implemented; code is the source of truth) | +| [PR #6785 in depth](pr-6785-unified-storage-and-flows.md) / [中文](pr-6785-unified-storage-and-flows.zh-CN.md) | Review-grade walkthrough of PR #6785: unified-storage tables (schema v12) / implementation logic / per-turn data flow / full switch ledger + measured validation (#5874/#7064/#7398/#4491) | diff --git a/docs/index.zh-CN.md b/docs/index.zh-CN.md index 248574b6d6..63f7e44408 100644 --- a/docs/index.zh-CN.md +++ b/docs/index.zh-CN.md @@ -35,3 +35,4 @@ | [lifecycle](lifecycle.md) | 过渡开关台账 / 退役模块盘点 / 卫生记录(CI 强制 `BILI_*` 覆盖) | | [auto-merge-guardrails](auto-merge-guardrails.md) | `AGENTS.md` §7 背后的证据附录(AI 自动开发评审史) | | [unified-loop-spec](unified-loop-spec.md) | 历史设计:统一压缩循环规格(已落地,以代码为准) | +| [PR #6785 详尽梳理](pr-6785-unified-storage-and-flows.zh-CN.md) / [EN](pr-6785-unified-storage-and-flows.md) | PR #6785 审查级梳理:统一存储表结构(schema v12)/ 实现逻辑 / 逐轮数据流转 / 开关总账 + 实测验证(#5874/#7064/#7398/#4491) | diff --git a/docs/msgid-infrastructure.md b/docs/msgid-infrastructure.md index deb4f2fbf7..2f360418ba 100644 --- a/docs/msgid-infrastructure.md +++ b/docs/msgid-infrastructure.md @@ -1,6 +1,6 @@ # Message-ID lane infrastructure: the sessionid+msgid production/consumption map -> Status: companion doc for the gap fix mandated by the #5826 release review (owner ruling 2026-11-07). Identity-ladder doctrine lives in message-identity.md (kernel) and session-identity.md (session side); this file does ONE job — explain where sessionid+msgid, the cross-lane public substrate, is produced, how it travels, who validates it (six copies), where it is pinned, its five consumers, and its declared trust boundaries. Read this before touching any of it. +> Status: companion doc for the gap fix mandated by the #5826 release review (owner ruling 2026-11-07). Identity-ladder doctrine lives in message-identity.md (kernel) and session-identity.md (session side); this file does ONE job — explain where sessionid+msgid, the cross-lane public substrate, is produced, how it travels, who validates it (six copies), where it is pinned, its six consumers, and its declared trust boundaries. Read this before touching any of it. ## 1. What it is @@ -21,7 +21,7 @@ Discipline: collectors only "obtain and pre-screen"; they hold NO identity-adjud Four carriers, **ALL stripped before forwarding to the model upstream** — the model wire still carries no message-level ids (#2821 doctrine): -1. **Envelope field** `biliMsgIds`: parallel array on full-resend model bodies, length-locked to messages, same lifecycle (stripped at ingress, src/server/prepare-anthropic.ts et al.). +1. **Envelope field** `biliMsgIds`: parallel array on full-resend model bodies, length-locked to the **post-hoist core count** (not the wire element count) — clients prove core-alignment via the kernel's own #3642 `wireIndex` back-pointer before stamping (src/agent/pi-msgids.ts `coreAlignEnvelopeIds`; every non-leading wire element must own exactly one core; any unprovable shape bails the whole request to content-derived ids — never guessed, never partial, request ships byte-identical). Wired: dsh full-resend (src/agent/dsh-native.ts `resolveEnvelopeMsgIds`, openai + anthropic wire, #6525) and pi + omp full-resend (src/agent/pi.ts `stampMsgIdEnvelope`, reuse the same already-validated walker output), the latter gated by `destinationRoutedThroughProxy` — a blind tunnel never gets stamped (#1403). Same lifecycle (stripped at ingress, src/server/prepare-anthropic.ts et al.). 2. **Incremental turn payload** `msgIds`: envelope on `POST /__bili/plugin/turn` (src/agent/incremental-client.ts); server re-stamps `biliMsgIds` (src/incremental-lane.ts). 3. **Watermark headers** `x-bili-pf-tail-ids` (v2-v4, #5403/#5418): window segment only = last anchor element + all elements added since K, percent-encoded per segment, comma-separated (commas inside ids travel as `%2C`, separators stay literal); sibling `x-bili-pf-count` count stamp. pf-* headers are **protocol headers, not hop headers**: stripped before upstream via `UPSTREAM_HOP_HEADERS`. 4. **Ledger-seq sibling** `biliLedgerSeqs` / `ledgerSeq` stamps (#4316 L3 PR2/3): 1-based positive-int parallel arrays, validated by `validateMsgSeqs`, same strict-validate-then-degrade lifecycle. @@ -48,7 +48,7 @@ One-off closure from the review: `parseTailIds` alone lacked the reserved-prefix The unified-store `messages.msg_id` column is the ONLY id truth: ingestion pins validated ids into rows (T5 ruling: no parallel ledger; skeleton rows carry no ids). Ledger primary key `(session_id, msg_id, content_hash)`; canonical rows key on **MSGID, not content hash** (#4491); fork clones preserve numbers (#4316 L3 PR3/3 `cloneSessionRows`). -## 6. Consumers (five; what question each one asks of the ids) +## 6. Consumers (six; what question each one asks of the ids) | Consumer | Location | Question answered | Id-usage boundary | |---|---|---|---| diff --git a/docs/plugin-protocol.md b/docs/plugin-protocol.md index ff4dae5743..6bbba4603c 100644 --- a/docs/plugin-protocol.md +++ b/docs/plugin-protocol.md @@ -193,15 +193,19 @@ Discover `capabilities.fork` from the manifest; do not infer support from the pa "parentRevision": "", "branchPoint": { "messageCount": 1, "orderHash": "" }, "orderedMessages": [{ "rawId": "h_example", "ref": "m00001", "identityHash": "" }], - "idempotencyKey": "stable-operation-id" + "idempotencyKey": "stable-operation-id", + "prefixMsgIds": ["host-uuid-1", "host-uuid-2"], + "driftFrom": 1 } ``` +`prefixMsgIds` and `driftFrom` are optional (#6262). On a parent whose kernel ids are the host's own stable message ids (a pinned session, where snapshot `rawId` IS the host id), a client may additionally present the host id of every matched prefix message, plus the hash-run length as `driftFrom` (`0 ≤ driftFrom < prefixMsgIds.length`). The server accepts a byte-drifted prefix only when every presented id equals the parent's `rawId` — ids prove prefix identity where bytes no longer match — and marks the receipt `status: "drifted"` (or `"expanded+drifted"` when a block also crosses the branch) with the verified `driftFrom` echoed. A gate failure never fails the fork: the request falls back to the strict byte-verified fork (`"exact"`/`"expanded"`) and the extra fields are ignored. Malformed shapes (wrong length, bad identifier, `driftFrom` out of range or without `prefixMsgIds`) return `400 INVALID_REQUEST`. Parents without host-pinned ids make the track a no-op, byte path unchanged. + `protocolVersion` is `2` (current) or `1` (legacy) in both directions; the proxy accepts either (`acceptsProtocolVersions` in the manifest advertises this) and rejects anything newer with `400 INVALID_REQUEST`. Under protocol 2 the `parentRevision` token covers exactly the session id, the ordered-prefix length (`messageCount`), and the active block set — so routine stats churn (context estimates, usage bookkeeping) no longer changes the revision, while any structural change (messages appended or rewritten, a block folded or unfolded) still does. Protocol-1 clients that present the legacy whole-store revision token are still matched during the transition window; treat `parentRevision` as opaque in both cases: on `409 PARENT_REVISION_CONFLICT` take a fresh snapshot and re-match, regardless of which protocol version you speak. The branch is an ordered prefix, including an empty prefix. Compute its existing order hash as lowercase SHA-256 of UTF-8 `JSON.stringify(orderedMessages)` with each identity object's keys in order `rawId`, `ref`, `identityHash`; for a full prefix reuse snapshot `orderHash`. Do not hash descriptors or folded text. Extra child messages, including a final model reply that has not left the proxy yet, may be appended on the child's first model request after copying the matched prefix. -New success is `201`; exact payload replay is `200` with `replayed: true`, even after restart or later parent changes. Both return `status`, `parentRevision`, `childRevision`, `sessionId`, `branchPoint`, `inheritedBlocks`, and `expandedBlocks`. Reusing the child id with another parent, key or payload returns `409 CHILD_CONFLICT`. Invalid shapes/version/identifiers return `400 INVALID_REQUEST`; absent parent returns `404 PARENT_NOT_FOUND`; stale parent returns `409 PARENT_REVISION_CONFLICT`; altered order/raw/ref/identity returns `409 BRANCH_POINT_CONFLICT`. Missing nested originals return `409 PARENT_STATE_INCOMPLETE` with `status: unavailable`. Publication/persistence failure returns `503 FORK_FAILED` with `status: unavailable` and exposes no child; the same request may be retried. +New success is `201`; exact payload replay is `200` with `replayed: true`, even after restart or later parent changes. Both return `status`, `parentRevision`, `childRevision`, `sessionId`, `branchPoint`, `inheritedBlocks`, and `expandedBlocks` (plus `driftFrom` when the msgid track accepted a drift, above). Reusing the child id with another parent, key or payload returns `409 CHILD_CONFLICT`. Invalid shapes/version/identifiers return `400 INVALID_REQUEST`; absent parent returns `404 PARENT_NOT_FOUND`; stale parent returns `409 PARENT_REVISION_CONFLICT`; altered order/raw/ref/identity returns `409 BRANCH_POINT_CONFLICT`. Missing nested originals return `409 PARENT_STATE_INCOMPLETE` with `status: unavailable`. Publication/persistence failure returns `503 FORK_FAILED` with `status: unavailable` and exposes no child; the same request may be retried. Boundary handling is `exact` when complete blocks fit inside the prefix, `expanded` when a block crosses it (crossing summaries are omitted, affected nested descendants are expanded, matched originals are retained), and `unavailable` when safe reconstruction is impossible. Blocks, nested caches, raw messages, issued ref namespace, token snapshots, rules and prefix CCR payloads are independent copies. Note that the child inherits the parent's ENTIRE issued-ref namespace (dead refs included): refs are never reused, so forks keep allocating fresh refs without colliding with ids a sibling may still hold. The child never reads its parent's later state as a fallback. Registration and first-request resume heuristics do not replace its copied state. With existing persistence enabled, raw snapshots, receipts, nested originals and the child's CCR payloads survive restart; with persistence disabled, durability is not promised. Atomic child publication/idempotency covers concurrent requests to one proxy process; concurrent writers sharing one state directory across processes are not supported by this protocol. @@ -213,7 +217,7 @@ Marker divert (#3248 phase 2 — native extensions): instead of switching the pr 中文增量要点(#3248):`POST /__bili/plugin/turn`,首次/恢复用 `sinceSeq:0` 全量(种子隐含 `resync:true`,永不 409);此后每轮只发新增(含上一轮 assistant 回复,客户端自持)。响应即上游原文,`x-bili-applied-seq` 为应用后的长度,下一轮作 `sinceSeq` 回传。409 必须按 serverSeq 对齐或全量重同步;503 表示未开 storage.unified。会话身份沿用客户端自己的 conversation id——增量与全量重发同一会话可任意切换,折叠/工具/状态栏不变。阶段 2(标记分流):普通模型请求上盖 `x-bili-plugin-turn: 1` 头 + 增量形状 body(顶层 conversationId/sinceSeq/messages、无顶层 model)即可,增量形状 turn 属 lane 流量,与直连 turn 端点同信任级(#3343):非回环/隧道/不可信源一律 403(其探针本就 403,正常宿主不会远程发增量);管理前缀(`/__bili/`、`/__acp/`)永不走分流。lane 按原始 URL 转发;探针 `GET /__bili/plugin/turn`;开关 rollout `pluginIncrementalTurns`(默认 0%=零变化,`BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS=1` 或配置强制开)。客户端状态机在 src/agent/incremental-client.ts,新宿主复用勿重写。已接宿主:pi/pi-native(同一扩展工厂,后者零代码继承)、dsh-native(fetch 补丁 beforeSend/headersFor/onResponse)、opencode(http.request 钩子 incremental 适配回调换 body+盖章)、omp(无头事件通道,读 body 里 prompt_cache_key 当会话 id,直接在 init 上盖章);OpenAI 工具形状(tool_calls/role:tool)自动回退全量重发;每 URL 单飞行+30s 过期。 -中文集成要点:fork 协议现为 2(服务端同时接受 1);先读取 manifest 能力及 snapshot,按原文、role、工具身份确认匹配前缀,仅提交该前缀的 orderedMessages,不猜测多模态分支点。被 `compress.protectedTools` 硬排除的消息在两个数组中均为 `"ref": null`(参与顺序与身份、无地址 ref),与其他前缀条目一样逐字提交;ref 合法值仅 `mNNNNN` 字符串或 `null`。父 revision 变化必须重新取快照并重新匹配。子会话使用独立稳定 id,首次请求可追加尚未出站的模型回复,不能重置继承的 refs/blocks。手动摘要使用 snapshot 的 refs 和 `expectedRevision: snapshot.parentRevision` 调用公开 tool;摘要期间历史发生变化时返回 409,不执行旧摘要。状态栏读取 `contextTokens`、`contextTokensSource`、`contextTokensAt`、`model`、`contextLimit`、`contextGeneration`、`sessionId` 与 `sessionRevision`,不得用累计账单 input/cache 冒充有效上下文。手动 compress/decompress 后为独立时间及 generation 的 estimate,不改累计账单或 last-usage 基线;下一次正数 usage 才恢复 usage,零值或缺失 usage 不得复活旧观测。未知来源返回 unavailable,tokens/time/generation 为 null。 +中文集成要点:fork 协议现为 2(服务端同时接受 1);先读取 manifest 能力及 snapshot,按原文、role、工具身份确认匹配前缀,仅提交该前缀的 orderedMessages,不猜测多模态分支点。父会话若已 pin 宿主 msgid(snapshot 的 rawId 即宿主 id),可另附 `prefixMsgIds`(每个前缀消息的宿主 id)与 `driftFrom`(哈希前缀长度):宿主 id 全等时服务端接受字节已漂移的前缀,回执 `status: "drifted"`(跨块时 `"expanded+drifted"`)并回显已验证的 `driftFrom`;gate 失败不影响 fork 成功(回退严格字节 fork,附加字段被忽略);字段形状非法返回 400,未 pin 的父会话该轨自动无效。被 `compress.protectedTools` 硬排除的消息在两个数组中均为 `"ref": null`(参与顺序与身份、无地址 ref),与其他前缀条目一样逐字提交;ref 合法值仅 `mNNNNN` 字符串或 `null`。父 revision 变化必须重新取快照并重新匹配。子会话使用独立稳定 id,首次请求可追加尚未出站的模型回复,不能重置继承的 refs/blocks。手动摘要使用 snapshot 的 refs 和 `expectedRevision: snapshot.parentRevision` 调用公开 tool;摘要期间历史发生变化时返回 409,不执行旧摘要。状态栏读取 `contextTokens`、`contextTokensSource`、`contextTokensAt`、`model`、`contextLimit`、`contextGeneration`、`sessionId` 与 `sessionRevision`,不得用累计账单 input/cache 冒充有效上下文。手动 compress/decompress 后为独立时间及 generation 的 estimate,不改累计账单或 last-usage 基线;下一次正数 usage 才恢复 usage,零值或缺失 usage 不得复活旧观测。未知来源返回 unavailable,tokens/time/generation 为 null。 ### Incremental cache-mark updates diff --git a/docs/pr-6785-unified-storage-and-flows.md b/docs/pr-6785-unified-storage-and-flows.md new file mode 100644 index 0000000000..a2f079a548 --- /dev/null +++ b/docs/pr-6785-unified-storage-and-flows.md @@ -0,0 +1,340 @@ +# PR #6785 in depth: unified-storage tables / implementation logic / data flow / switches + +> Scope: `2026-11-15_msgid-snapshot-unified` @ `39941c816` (base `origin/master` `b10c1a0d4`). +> 134 files, +13740/−367. This is the **review-grade** full walkthrough of this PR: four parts (table structure / implementation logic / data flow / switches) plus measured validation and this round's findings and fixes. All line numbers refer to source at head. + +--- + +## 0. What this PR actually is (one sentence + five blocks) + +It turns the two O(full) paths — "recompute the outbound view from scratch every turn" and "re-dedupe the full inbound body every turn" — into **msgid-anchored sparse incremental structures**: an op-log materialized table on the outbound side (discussion #7398), watermark delta ingest on the inbound side (pre-existing #3133, this PR adds the parse layer), an incremental pin channel for identity (#7064 §8), and a full-resend watermark + delta tail slice on the fast path (#5403/#5418 + this PR). Five blocks: + +| Block | Proposal / origin | Switch (default) | +|---|---|---| +| ① Snapshot base v2 (stage-0 identity ledger + fold journal + outbound prefix cache) | #5874 v2 / #7064 slimming | `storage.snapBase.enabled` (off by default, config-only) | +| ② Outbound op-log (egress_state/egress_flow + rebuild-serve + server authority) | Discussion #7398/#7439 | `egressPlane` (0) — ONE switch for the whole new path | +| ③ Full-resend watermark + delta tail slice | #5403/#5418 + this PR | `pluginFullResendWatermark` (5) / `pluginDeltaResend` (0) | +| ④ Incremental msgid pin (tail-ids tail form) | #7064 §8 | `pluginMsgIdIncremental` (**100**, on by default from this PR) | +| ⑤ Server-side canonical history GA | #4491 | `pluginLaneServerHistory` (**100**, flipped 0→100 by this PR) | + +--- + +## 1. Table structure (unified storage, SQLite, schema v12) + +### 1.1 Migration mechanism (schema.ts) + +- `STORAGE_SCHEMA_VERSION = 12` (schema.ts:105; this PR bumps it from 11). +- v12 is **probe-first** (checks for table existence instead of gating on the version number): sibling branch #5855 had already landed the pf-plane tables into shared databases as "v12/v13" (field case 2026-11-08: user_version 13, pf tables present, l1_world absent), and a version gate would skip those databases. +- All new tables are **additive**: an old build opening a new file ignores the extra tables; a new build opening an old file probes and adds what is missing. +- Per-database pragma at open (auto_vacuum=INCREMENTAL must be first): `journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, foreign_keys=ON, journal_size_limit=8MiB`. + +### 1.2 The 8 tables added by this PR (all DDL in schema.ts) + +**l1_world** (schema.ts:377-391, `SNAPBASE_DDL_V12`) — the stage-0 identity ledger: host raw session id → proxy ledger identity + one-round arming + element-level world cursor + R3 breaker. + +``` +sess_raw TEXT PRIMARY KEY, -- raw session id as written by the host +ledger_id TEXT NOT NULL, -- resolved ledger session_id +witness_probe TEXT NOT NULL, -- witness probe at first sight (identity evidence) +issued_at INTEGER NOT NULL, armed_at INTEGER, +tail_json TEXT, -- LAST 512 host ids of the previous round's array (JSON string[]) +head_json TEXT, -- first 32 +count INTEGER, -- full array length +max_pos INTEGER, -- append-only position frontier +r3_latched_at INTEGER -- #6181 M3 persistent R3 breaker latch (NULL = not latched) ++ idx_l1_ledger ON l1_world(ledger_id) -- serves deleteSession by ledger_id +``` + +(Shape after the #7064 slimming: the original whole `envelope_json` blob was DROPPED, demoted to "tail-512 / head-32 window + count + max_pos" — identity gate demoted to bookkeeping cursor; the middle of the array is no longer visible.) + +**fold_journal** (schema.ts:413-427) — append-only fold-decision journal: + +``` +session_id TEXT NOT NULL, jseq INTEGER NOT NULL, +kind TEXT NOT NULL DEFAULT 'fold' CHECK (kind IN ('fold')), +block_id TEXT NOT NULL, start_seq INTEGER NOT NULL, end_seq INTEGER NOT NULL, +input_hash TEXT NOT NULL, -- sha256 over messages[ start_seq..end_seq ] (ALL generations, incl. retired) +created_at INTEGER NOT NULL, +PRIMARY KEY (session_id, jseq), +UNIQUE (session_id, block_id) -- idempotent re-ingest; a replaced block keeps its first fold row +``` + +UNPIN is **not stored** — it is derived at read time from three conditions (block no longer active / a live row exists in the range / an id inside the range becomes live again past end_seq). The `covered_ids` column existed during development and was DROPPED by migration in the #7064 slimming (schema.ts:573-576). + +**egress_snapshot** (schema.ts:441-453) — the cut-materialized outbound prefix **cache** (latest row wins, older rows trimmed in the same transaction): + +``` +session_id TEXT NOT NULL, snap_seq INTEGER NOT NULL, +watermark INTEGER NOT NULL, -- max live seq of the pin set at cut time = hash scope +input_hash TEXT NOT NULL, -- sha256 over LIVE rows seq <= watermark +prefix_ref TEXT, -- CAS ref of the derived prefix JSON (NULL = hash pin only) +last_stable_id TEXT NOT NULL, -- msgid at the stitch point +created_at INTEGER NOT NULL, +scope_rev INTEGER, -- egress_scope_rev value at cut time (NULL = pre-upgrade row) +PRIMARY KEY (session_id, snap_seq) +``` + +(`ids_json` was DROPPED by migration; `scope_rev` was ADDed for the upgrade.) + +**egress_scope_rev** (schema.ts:459-464) — per-session message-change revision for O(1) reader trust: + +``` +session_id TEXT PRIMARY KEY, rev INTEGER NOT NULL +``` + +Any write that can change the input_hash (retire/unretire, content_hash edit, hard delete, archive restore, backfilling an old seq) bypasses and bumps it; pure tail appends do not (seqs are monotonic ⇒ always outside every watermark). + +**orphan_blocks** (schema.ts:475-483, #6785 step 7) — reaped block content moved to a table (out of the `sessions.orphans_json` blob): + +``` +session_id TEXT NOT NULL, block_id TEXT NOT NULL, +one_json TEXT, full_json TEXT NOT NULL, +PRIMARY KEY (session_id, block_id) +``` + +**state_fields** (schema.ts:491-498, #6785 step 8) — the unbounded kstate fields moved to rows: + +``` +session_id TEXT NOT NULL, field TEXT NOT NULL, value_json TEXT NOT NULL, +PRIMARY KEY (session_id, field) +``` + +The field set is **frozen** at slim.ts `ROW_BACKED_KSTATE_FIELDS` (9 fields: tokenSnapshot, tsStamps, lastPassIds, hiddenOrphanRefs, sweptOrphanCallIds, renderDeferredRefs, viewLatches, absorbed, rules); `tests/issue5874-state-fields` pins it. + +**egress_state / egress_flow** (schema.ts:508-530, discussion #7398) — the outbound conversation op-log, two structurally identical tables: + +``` +session_id TEXT NOT NULL, log_seq INTEGER NOT NULL, +row_type TEXT NOT NULL CHECK (row_type IN ('B','I','D','U')), +msgid TEXT, -- NULL for B/D; required for I/U +payload TEXT NOT NULL, +created_at INTEGER NOT NULL, +PRIMARY KEY (session_id, log_seq) ++ idx_egress_state_msgid(session_id, msgid) ++ idx_egress_flow_msgid (session_id, msgid) +``` + +- `egress_state` = serving reads + compaction physically trims below the watermark; +- `egress_flow` = append-only, **never deleted** (any historical log_seq can be rebuilt; rollback is based on this table, not on client resends). + +Row payload contracts (egress-state.ts): +- `I`: `{"after": msgid|null, "ref", "tokens", "type", "content_hash", "role"?, "cas_ref"?}` +- `D`: `{"start": msgid, "end": msgid}` (closed interval; anchored to MSGIDS, not positions — "positions drift with other ops, msgids do not", KDD#11) +- `U`: `{"content_hash", "role"?, "cas_ref"?}` (the msgid column IS the target) +- `B`: `{"entries": EgressViewRow[], "watermark": seq}` (entries use CAMEL keys `contentHash/casRef`, unlike the snake keys of I/U) + +`EgressViewRow` = `{msgid, ref, tokens, type, contentHash, role?, casRef?}`; `casRef` = the key in the unified content CAS of that message's **FINAL core JSON** (after reasoning-drop / image rescale, before outbound conversion). + +### 1.3 Key columns on pre-existing tables (as they relate to this report) + +- `messages` (schema.ts:181-195): PK (session_id, seq), UNIQUE (session_id, msg_id, content_hash), v8 added `identity` (cross-save join key) + `retired_at` (tombstone; no hard deletes in steady state), v10 added `ts`. +- `content` CAS (schema.ts:173-179): `hash PK, byte_len, refcount, algo='sha256', bytes BLOB` — one store for all bytes (messages, block summaries, media, CCR, egress prefixes, egress core JSONs), refcounted. +- `ccr_entries` (schema.ts:254-264): **this PR adds `orig_hash`** (+ index) — the byHash half moves into the CAS. +- `conversations` (schema.ts:273-281): **this PR adds `idx_conversations_session(session_id)`** — stage-0 reverse-looks-up the binding by session_id every turn; the PK cannot serve that. +- `blocks` (197-214), `dead_refs` (216-223), `refs` (229-237), `checkpoints` (239-252), `session_ext` (291-298), `kv_store` (303-310), `message_media` (315-320), `sessions` (148-171, the semantics of `kstate_json/ccr_json` changed via slim, DDL untouched). + +### 1.4 MySQL parity (mysql.ts +274) + +Ported: `orphan_blocks`, `state_fields`, `ccr_entries.orig_hash` (+ index, probe migration, archive/restore/cascade). +**Not ported**: `l1_world / fold_journal / egress_snapshot / egress_scope_rev / egress_state / egress_flow` — the snapshot/egress planes are **SQLite-lane only** (shadow.ts `egressOpLogSync()` returns null on the network backend; the recorder skips the whole turn). + +--- + +## 2. Implementation logic (per module) + +### 2.1 slim.ts (new, 185 lines) — CCR slimming + kstate row split + +- `sessions.ccr_json` holds only the slim envelope `{version:1, ccrSlim:1, byRef:{key:{hash}}, orphanByHash?}` (slim.ts:20-48); `ccr_entries` rows hold the two CAS halves: `content_hash` = the entry JSON, `orig_hash` = the original text behind the entry. +- **null-orig contract**: a row with `orig_hash=NULL` ⇒ the whole store falls back to the file lane (hydrate returns null, never a partial byHash); `""` (empty-string hash) = a legitimately empty original (fork receipt stub); a missing byHash entry must land as null, never `""`. +- kstate: `splitKstate` moves the 9 frozen fields out of `kstate_json` into `state_fields` (slim.ts:155-185); readers apply rows over columns, writers upsert on change and delete the row when the value goes undefined. +- The steady-ingest skip check is upgraded to require all of `ord && content_hash && orig_hash` to match. + +### 2.2 snap.ts (new, 493 lines) — the stage-0 identity ledger (decides identity only; no compression, no byte serving) + +- `snapL1Record` → `hit|echo|arm|wait|issue|red` (snap.ts:80-114): new row issues; a different `ledger_id` = **red hard conflict, nothing is written** (R3 domain; the caller stays legacy); armed + probe match = echo; armed + mismatch = hit; not-armed + match → arm (one-round arming); otherwise wait. +- World cursor: `snapL1WindowRead/Upert` (snap.ts:138-165) read/write l1_world's tail/head/count/max_pos (window widths `WORLD_TAIL_K=512` / `WORLD_HEAD_K=32`, identity.ts:29/32). +- R3 latch: `snapR3LatchWrite` (snap.ts:179-191) UPSERTs a placeholder row (a first fault may precede any snapL1Record); latched sessions never re-arm. +- `foldInputHash` (snap.ts:252-268): sha256 over `[start_seq,end_seq]` in **ALL generations** (no retired filter — the fold's own ingest retires the covered rows in the same transaction, so a live-only aggregation at fold time would be empty), line format `${seq}\u0000${msg_id}\u0000${content_hash}\n`. +- `foldUnpins` (snap.ts:326-370): the single UNPIN predicate (the three conditions, see §1.2). +- Egress cut reads: `egressInputHash` (407-424, live-only, scope = watermark), `egressScopeRev`/`bumpEgressScopeRev` (430-450), `egressLatestRow` (477-486). + +### 2.3 egress-state.ts (new, 365 lines) — the op-log engine (pure functions, testable) + +- `diffEgressView` (153-204): the spine = the longest subsequence of `next`'s msgids that appears **in ascending order** in `prev` (two-pointer scan); a spine row whose contentHash changed ⇒ `U`; a prev row off the spine ⇒ emit the maximal contiguous `D` span in prev order; a next row off the spine ⇒ emit `I`s in next order (anchor = previous row); **a move = D+I, there is no M op**; op order D→U→I; invariant `replay(prev, diff(prev,next)) === next`. +- `nextLogSeq` is taken from **flow** (state is physically trimmed by compaction; flow is the superset). +- `egressStateAt(db, sid, x)` (264-274): replay flow up to `log_seq <= x` = rebuild at any historical point (rollback). +- `egressStateCompact` (278-292): the replay result is written as one `B` row (both tables) + physical delete of state rows below the watermark (flow untouched). +- Any anchor/endpoint/duplicate/malformed violation ⇒ throws `EgressStateCorruptError` (36-42) ⇒ the caller **degrades the whole session to legacy**, never a fuzzy fallback. + +### 2.4 egress-record.ts (new, 102 lines) — per-turn recording (seg 3) + +- `buildEgressViewRows`: the first row is system (`msgid="system:"+hash32`, `casRef=CAS(system text)` — **byte-stability IS the cache anchor**); one row per message, `contentHash=rowHash([id,role,contentType,text])`, `casRef=CAS(final core JSON)` (since head commit `39941c816` the tool-row sidecars (toolCallId/toolName) ride the body into the CAS, so serve no longer excludes tool rows). Ephemeral rows (nudge, sysNotes) are excluded by design (KDD#7). +- `recordEgressTurn`: prev = session cache or one-time hydrate; diff; append only if there are ops; **compact only when the batch contains a D (fold shape)**. +- Fix this round: unchanged rows **reuse** the prev `casRef` instead of putContent-ing again (previously every row's refcount grew +1 per turn with no matching release; the only release point was session cleanup — see §5). + +### 2.5 egress-serve.ts (new, 65 lines) — rebuild-from-log + real swap (seg 4b) + +`rebuildOutboundFromLog(view, store)`: skips the system row (the system block is ALWAYS live-built — the prefix-cache anchor discipline); a missing `casRef` (pre-seg4 row) or missing CAS body ⇒ returns null (**whole turn** stays legacy); the body must parse to an object with id/role and a legal role; then it runs the **exact same** chain as legacy, `hardenOpenaiAssistantContent(coreToOpenai(systemToUser(msgs)))` ⇒ byte-identical by construction. + +### 2.6 Inbound: watermark delta ingest (pre-existing #3133; this PR did not change its core) + +- Decision point `shadow.ts:679-703` (`ingestDelta`): `ingestWatermark` (sessions.last_ingested_seq + persisted tailSeqs, store.ts:2329) → boundary probe (:2403) → #3919 head id probe (:2418) → `planIngestDelta` → `ingestSession(delta)`; any null/false hop ⇒ the exact full path. +- `planIngestDelta` (ingest.ts:311-405) returns null when: pluginSnapshot non-empty / decompositionLog (#3301 import) / no watermark / `tail.length===trustedLen && !hasRawTail` (an equal-length tail is only a legal empty delta on the raw lineage — #2960 ⑤ fold re-renders may reorder) / **`tail.length < trustedLen` (the wire shrank = a fold happened ⇒ full)** / boundary or head probe mismatch. +- The new-messages loop serializes + hashes ONLY the new ones (`O(N)`), placeholder seqs `watermark+1+i`. +- Store-side guards `store.ts:1229-1281` (inside BEGIN IMMEDIATE): boundary **re-verified inside the transaction** (1231-1232); each new message is probed by `(session_id,msg_id,identity)` — any existing row ⇒ full path (1239-1247; #4390 identity is position-independent and retired counts as existing); renumbering `probe.maxSeq+1+i` (1253); tailSeqs prefix consistency (1256-1268). **A stale/external watermark costs at most one redundant full write; it never writes wrong rows.** + +### 2.7 Parse layer: parse-defer (new in this PR) + +- `fastpath-scan.ts`: `scanMessagesSpans` scans the raw body to extract messages-element spans (no parse); `synthesizeTopLevel` rebuilds the top-level object with `messages: null` as the "deferred" marker (null, not `[]` — `Array.isArray([])=true` would break the MISS re-parse path, and the relay guard passes null through). +- The `handle.ts` gate: openai-shaped + pf protocol stamp + body ≥ 1MiB ⇒ deferred; HIT turns skip the whole-body parse; only MISS re-parses once from the text in prepare-openai. + +### 2.8 Fast path: watermark + delta arm (fastpath.ts / handle.ts) + +- **Full-resend watermark** (#5418, rollout 5%): the response stamps `x-bili-pf-seq` (the messages element count, measured by a zero-allocation structural scan); the client echoes `x-bili-pf-count` + `x-bili-pf-tail-ids` (width `T − stampT + 1`, first slot = anchor, `?` placeholders); a HIT = previous round's canonical cores + only the tail window newly parsed (skips whole-body parse/conversion/pin-stash — the #5403 OOM producer). +- **Delta arm** (this PR, `pluginDeltaResend` 0): the client's body.messages carries ONLY the L new elements; the server rebuilds the view from the pinned prefix cores + the small slice. Any failing claim (count/body/tail-width mismatch, broken anchor chain after known ids, rollout off) ⇒ 400 `pf_delta_reanchor` + `x-bili-msgid-mode: full`; the client sends the full form that same turn — **one-round convergence, never a silent misread**; a delta miss NEVER falls through to legacy parse (treating the tail as the whole history = session destruction). +- Mismatch semantics (fastpath.ts:296-304): `stamp > T` = "ahead" (the wire shrank = a compression happened) ⇒ clear mode, full re-anchor next turn; `stamp === T` = "no-progress". +- **Bug A fix** (b4bbaaef): the HIT head check excludes the id field — after pinning, ledger rows carry host UUIDs while the fresh side carries content-hash ids, so per-slot comparison was guaranteed to false-negative. +- **Bug B fix** (2c3bdd10): a HIT turn's snapshot render produced ~2 rows ⇒ short rawTail ⇒ completeLog ⇒ sweep retired all prefix rows ⇒ the ledger drained ⇒ pin degraded every turn. Fix: on HIT set `session.lastMessagesFolded=true`. +- **Bug C fix** (9efbf8b7): new ledger rows on HIT turns carried h_ ids instead of host UUIDs (pinned rows only cover the window) ⇒ tail width perpetually off by one ⇒ full/incremental alternation forever. Fix: on HIT turns restamp `ledgerWindow` from the `s0.pinned` tail before snapshotMessages. + +### 2.9 Incremental msgid pin (#7064 §8, default 100) + +Steady turns declare ids via the bounded `x-bili-pf-tail-ids` (anchor + new, ≤2048 ids, `?` placeholders) or a short body id array; the server **re-stamps the prefix from ledger live rows** (ingest INSERT OR IGNORE on (session_id,seq) — a position's msg_id is written once). Acceptance rule (finalized 2026-11-19): the declared tail span [T-L,T-1] must be **row-by-row equal** to the ledger's overlap span; any drifted row ⇒ degrade to full, never 400; overlong sessions (>2048) pin fine with truncated tail lists (longer overlap = stronger proof). Snapshot rounds (empty ledger / post-fold re-anchor) send the full form; full form takes priority. The response stamps `x-bili-msgid-mode: incremental|full` (cleared on any 4xx; same learning point as pf-seq, pfLearnStamp). Mismatches self-heal in one round. + +### 2.10 Admin surface + +New loopback routes `GET /__bili/snapshot/r3-latches` / `POST /__bili/snapshot/r3-unlatch` (admin.ts +56). + +--- + +## 3. Data flow (the complete path of one request) + +### 3.1 Steady turn (+N new messages, pf HIT) — both incrementals active + +``` +Client POST /bili/http:///chat/completions + body: full messages (watermark full arm) OR only the L tail elements (delta arm) + + biliMsgIds (full form) OR x-bili-pf-tail-ids tail form + + x-bili-pf-count / x-bili-pf-seq (once learned) + → [handle] readBody (155MB-class) → pf protocol scan (fastpath-scan, zero parse) + → [fastpath] HIT (stamp advanced + anchor chain intact): skip whole-body parse/convert/pin-stash + → [prepare-openai] rebuild view = pinned prefix cores + tail-window parse (delta arm) + ├─ egressPlane ON: recordEgressTurn (sparse I ops, a few KB) + │ → rebuildOutboundFromLog (op log + CAS) → swap into outbound bytes + └─ OFF: legacy kernel view → coreToOpenai chain + → [ingest] shadow.ingestDelta: watermark + boundary + head probe → planIngestDelta + serializes/hashes only the N new rows → rows appended at watermark+1.. (watermark written in the same tx) + → outbound: render-refs tags + nudge (per-turn ephemeral, never persisted) → upstream + → response: x-bili-pf-seq (new T) + x-bili-msgid-mode: incremental +``` + +**New storage per turn = the new messages' CAS bytes + one I op (a few KB).** Outbound rebuild is O(tail) in steady state. + +### 3.2 Fold turn (client wire shrank / server-side fold landed) + +``` +Fold tx: blocks insert/update + covered rows retired (same tx) + one fold_journal row + (input_hash = all-generation aggregate over the range; egress_scope_rev bumped by the retire) +→ Next turn inbound: wire shrank ⇒ planIngestDelta null ⇒ the exact full path + (full parse + INSERT OR IGNORE dedupe + retire ledger + watermark reset) +→ Outbound: the egress view shows -D (range) + +I (summary row) ⇒ batch contains D ⇒ compact B row + (B = the full view row list; design-declared ≈8MB @ 100k rows; CAS bytes not copied, pointers only) +→ pf protocol: stamp > T "ahead" ⇒ clear mode ⇒ full form re-anchor this turn ⇒ incremental again next +``` + +**The cost is concentrated on the fold turn** (measured ~0.5–0.8s); steady turns return to O(tail). This is the designed cadence (fold = re-anchor turn), not a gap. + +### 3.3 Anomaly / degradation paths (all "whole turn legacy, never guess") + +| Trigger | Action | +|---|---| +| pf stamp missing/regressed/broken anchor chain | whole turn full parse + full path, restamp | +| delta claim fails | 400 pf_delta_reanchor + msgid-mode: full (one-round convergence) | +| any row in the tail-form ids vs ledger overlap drifts | degrade to full signal, re-anchor next turn (D1 family, never reject) | +| l1 red (ledger_id conflict) | write nothing, stay legacy, R3 domain handles it | +| EgressStateCorruptError (15 violation classes) | stop recording/serving for the session, whole turn legacy | +| serve rebuild missing casRef/CAS body/unknown role | whole turn legacy (4a shadow comparison keeps measuring) | +| stale watermark/external write | in-tx probe fails ⇒ exact full path, one redundant write | + +### 3.4 Lifecycle (delete/archive cascade) + +`deleteSession` (store.ts:~2586): messages/media/blocks/**fold_journal/egress_snapshot/egress_scope_rev/orphan_blocks/state_fields** + dead_refs/refs/ccr_entries/conversations/session_ext/sessions + l1_world (by ledger_id) + CAS release and scoped collection. +`archiveSession`: same (minus l1_world); the JSONL export is appended **before** COMMIT, state flips to archived. +**Fixed this round**: both now also sweep `egress_state/egress_flow` (previously leaked ⇒ op-log rows were permanently orphaned, the "storage only grows" p1 class; their CAS-referenced halves are released with them, while message bytes stay held by the messages rows). + +--- + +## 4. Switch ledger + +### 4.1 rollout.features (src/rollout.ts; env `BILI_ROLLOUT_`) + +| Key | Default | Change in this PR | Effect | Rollback | +|---|---|---|---|---| +| `pluginLaneServerHistory` | **100** | 0→100 (GA, owner ruling 2026-11-13) | ⑤ server-side canonical: biliMsgIds row pinning + /__bili/plugin/render + incremental-lane serverHistory announcement | env=0 | +| `pluginMsgIdIncremental` | **100** | new, on | ④ tail-form id declaration, ledger re-stamps prefix | env=0 | +| `pluginFullResendWatermark` | 5 | (existing 5% rollout) | ③ watermark fast path | env=0 | +| `pluginDeltaResend` | 0 | new, off | ③ delta tail-slice arm; 0 = 400 on delta requests, client converges to full in one round | born in rollback state | +| `egressPlane` | 0 | new, off | ② the WHOLE new path under one switch: sparse-op recording (all four lanes) + serve-from-log (byte-identical swap) + server authority for pinned plugin lanes (rulings #7439 3/5); off = legacy chain, zero rows, byte-identical to pre-feature. Replaces the retired `egressOpLog`/`egressServeLog` pair | born in rollback state | + +**No new static env vars, no new CLI flags** (rollout env names are derived dynamically from key names; the docs-coverage CI exempts dynamic names in both directions — which is also why `pluginMsgIdIncremental` went undocumented until now; the yaml entry was added this round). + +### 4.2 Config objects + +| Key | Default | Effect | +|---|---|---| +| `storage.snapBase.enabled` | off | ① snapshot-base plane (snapBase gate, knobs.ts:1001 fail-closed) | +| `storage.snapBase.cohorts` | — | protocol allowlist | +| `storage.snapBase.cutEveryKUserTurns` | 8 | egress_snapshot re-cut cadence (a re-cut = full prefix re-hash + re-serialize + CAS write, hence rate-limited; invalidation-class events always re-cut immediately) | +| `network.ingestBodyBytes` | 32MiB | inbound body cap (the measured 155MB scenario needs 1GiB; env `BILI_INGEST_BODY_BYTES` takes priority) | + +### 4.3 Response/request headers (protocol surface) + +`x-bili-pf-seq` (bili→client, element-count stamp), `x-bili-msgid-mode: incremental|full` (cleared on any 4xx), `x-bili-pf-delta: 1` (client→bili, hop header, stripped before upstream forward, headers.ts:40), `x-bili-pf-tail-ids` (client→bili, ≤8KB budget; over budget the header is omitted — the body full form is NOT suppressed). + +--- + +## 5. Measured validation (ideal-model incremental bench) and this round's fixes + +### 5.1 Bench method (perf/, 2026-11-21) + +Construction = the user scenario's plugin shape: client history grows from 1 message (pure append-only); the fake ideal model injects a compress tool call every ~300 refs of growth (≈50k tokens), folding the oldest span and keeping 1800 refs (≈300k-token live tail) — **the client applies folds host-side** (plugin semantics: the model initiates, the client executes), so the wire stays at the live view (1800–2100 messages, 0.7–0.8MB request body, outbound 200–253k tokens) while `cum` counts cumulative created messages (to 100k). Arms: master (`b10c1a0d4`, no pf) / PR full arm / PR delta arm. All arms share fake calibration and driver (node:http `connection: close` — bili closes the connection after each turn by design, #1982). **Note: every run in this batch was with the egress plane OFF** (the #7398 arm is untested, see §5.4); the 155MB raw-wire shape (a host that never folds) is not covered by this batch. + +### 5.2 Measured (RUN1 batch=500 all three arms 210 turns; RUN2 batch=100 only the PR full arm reached g943≈94k — the master arm was aborted by a mis-killed proxy, the delta arm never launched) + +**RUN2, PR full arm, by cumulative scale** (wire constant ~720–840KB, outbound 200–253k tokens, a fold every ~3 turns): + +| Cumulative created | 10k | 30k | 50k | 70k | 90k | 94k (non-fold turn) | +|---|---|---|---|---|---|---| +| PR full | 526ms | 501ms | 1170ms | 1346ms | 1650–1737ms | 1821ms | + +**RUN1 @ 100k cumulative, steady 10-turn segment** (wire 720KB, view 1801 rows, folding stopped): + +| Arm | Steady range | Notes | +|---|---|---| +| master full | 1312–1624ms (p50 ~1338) | pf=null (no watermark) | +| PR full | 1613–2322ms (last 1613) | pf=1804, mode=incremental | +| PR delta (invalid arm) | 1891–2849ms | config lacked `pluginDeltaResend` ⇒ x-bili-pf-delta ignored = full + dead header | + +**Honest conclusions**: +1. At view-shaped wire (~750KB), per-turn cost rises linearly with the **cumulative ledger size** (10k→0.5s, 94k→1.8s), independent of wire size — the cost sits in the inbound ingest/scan domain of 94k messages rows + 309 fold_journal rows (the slow-ingest log names #5744; WAL 33MB/ingest), not in this PR's structure. +2. The "<1s per turn" target is met up to ~30k scale (501ms) and crossed at 50k (1170ms). +3. Static replay (comp1200k: 781 rows / 1.2MB view, no ledger accumulation) measured the delta arm at 32–62ms vs ~91ms full — the structural ceiling (pure wire-bound) is far below the growing runs; the gap = accumulated-ledger cost, the next profiling target (see §5.4). +4. The delta arm has no valid data in the growing scenario (needs a rerun against a proxy with `pluginDeltaResend:true`). + +### 5.3 Findings and fixes this round (minor issues fixed directly, shipped with this report) + +1. **[p1 storage-only-grows] deleteSession/archiveSession missed egress_state/egress_flow** ⇒ op-log rows + their CAS references permanently orphaned. Fix: store.ts `deleteEgressOps` + both cascade call sites. +2. **[p1 storage-only-grows] buildEgressViewRows putContent-ed every unchanged row every turn** (CAS refcount +1 per row per turn, no matching release) ⇒ the content store of any egress-keyed session grew without bound. Fix: unchanged rows reuse the prev casRef (missing ones backfilled, changed rows re-put). +3. **[docs] `docs/proposals/snapshot-compression-foundation/`** described the RETIRED v1 four-table plane (snap_rows/snap_log/snap_ckpt were dropped before merge; the title still claimed "DDL landed in #5901") — contradicts the current v12 shape ⇒ **removed entirely**. +4. **[docs] `docs/storage-relations.{en,zh}.svg`** was missing 5 new tables, the fold_journal card still listed the dropped covered_ids, the egress_snapshot card still listed ids_json, and the ⑦ cascade card was inaccurate ⇒ **removed**, superseded by this report. +5. **[docs] rollout.yaml three stale entries**: pluginLaneServerHistory default "0 (off)" (code is 100), egressServeLog described as "seg 4a LOG ONLY never swaps" (4b really swaps), missing pluginMsgIdIncremental entry ⇒ all corrected/completed, docs regenerated. +6. **[comment] rollout.ts egressServeLog comment** — "(AND egressOpLog is on)" and "tool rows in v1" both stale ⇒ corrected. +7. **[switch] the egress keys collapsed to ONE** (owner ruling 2026-11-21): the `egressOpLog` + `egressServeLog` pair was retired and replaced by a single `egressPlane` rollout key — on = the complete new path (record + serve + server authority, all four lanes), off = legacy chain byte-identical to pre-feature. Both old keys never shipped (default 0, born in rollback state), so there is no migration. + +### 5.4 Left for follow-up (out of scope for this PR) + +- **Minimal validation with the egress keys ON** (pending): 155MB raw wire (host never folds) + egressPlane real serve ⇒ the full scenario with 300k-token outbound, checking the "few-KB new storage per turn" red line per turn (WAL diff) — this is the #7398 acceptance line; this batch did not cover it. +- Profiling the accumulated-ledger cost of 1.1–1.8s per turn at 50k+ cumulative scale (the slow-ingest #5744 domain; includes the fold-turn retire+re-anchor, the fold-crossing gate over 309 fold_journal rows, kernel scans). +- A valid delta-arm rerun in the growing scenario (with `pluginDeltaResend:true`). +- The compact B row ≈8MB @ 100k rows vs the "few-KB per turn" red line: a design-declared exception (#7398 §5); to hold the red line strictly, the B row could become a "range reference + anti-forgery fingerprint" (the #7064 l1_world demotion idea), at the cost of losing the state table's self-contained replay closure — needs an owner ruling. +- After an archive restore the egress op-log restarts empty (the next turn re-appends the full +I view — correct, but one full batch). diff --git a/docs/pr-6785-unified-storage-and-flows.zh-CN.md b/docs/pr-6785-unified-storage-and-flows.zh-CN.md new file mode 100644 index 0000000000..48f109b734 --- /dev/null +++ b/docs/pr-6785-unified-storage-and-flows.zh-CN.md @@ -0,0 +1,340 @@ +# PR #6785 详尽梳理:统一存储的表结构 / 实现逻辑 / 数据流转 / 开关 + +> 范围:`2026-11-15_msgid-snapshot-unified` @ `39941c816`(base `origin/master` `b10c1a0d4`)。 +> 134 文件 +13740/−367。本文是本 PR 的**审查级**完整梳理:四部分(表结构 / 实现逻辑 / 数据流转 / 开关)+ 实测验证 + 本轮发现与修复。行号均指 head 处源码。 + +--- + +## 0. 这个 PR 到底是什么(一句话 + 五块) + +把"出站视图每轮全量重算、入站每轮全量重去重"的两条 O(全量) 路径,改造成**以 msgid 为锚的稀疏增量结构**:出站建 op-log 物化表(7398 提案)、入站走水位 delta ingest(#3133 既有,本 PR 补 parse 层)、身份走增量 pin 通道(#7064 §8)、快路径走全量重发水位 + delta 尾切片(#5403/#5418 + 本 PR)。五块: + +| 块 | 提案/来源 | 开关(默认) | +|---|---|---| +| ① 快照底座 v2(stage-0 身份账本 + fold 日志 + 出站前缀缓存) | #5874 v2 / #7064 瘦身 | `storage.snapBase.enabled`(默认关,纯配置) | +| ② 出站 op-log(egress_state/egress_flow + 重建 serve + 服务端权威) | 讨论区 7398/7439 | `egressPlane`(0)——整条新链路的**唯一开关** | +| ③ 全量重发水位 + delta 尾切片 | #5403/#5418 + 本 PR | `pluginFullResendWatermark`(5)/ `pluginDeltaResend`(0) | +| ④ 增量 msgid pin(tail-ids 尾形态) | #7064 §8 | `pluginMsgIdIncremental`(**100**,本 PR 起默认开) | +| ⑤ 服务端正史 GA | #4491 | `pluginLaneServerHistory`(**100**,本 PR 从 0 翻到 100) | + +--- + +## 1. 表结构(统一存储 SQLite,schema v12) + +### 1.1 迁移机制(schema.ts) + +- `STORAGE_SCHEMA_VERSION = 12`(schema.ts:105,本 PR 从 11 提到 12)。 +- v12 **probe-first**(查表存在与否,不靠版本号门):因为兄弟分支 #5855 已把 pf 平面表当 "v12/v13" 落进过共享库(2026-11-08 现场:user_version 13 + 有 pf 表无 l1_world),版本门会让那些库跳过。 +- 所有新表都是**追加式**:旧构建打开新文件忽略多余表;新构建打开旧文件 probe 后补齐。 +- 每库打开时 pragma(auto_vacuum=INCREMENTAL 必须第一个):`journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, foreign_keys=ON, journal_size_limit=8MiB`。 + +### 1.2 本 PR 新增的 8 张表(DDL 均在 schema.ts) + +**l1_world**(schema.ts:377-391,`SNAPBASE_DDL_V12`)—— stage-0 身份账本:宿主原始 session id → 代理账本身份 + 一轮武装 + 元素级世界游标 + R3 断路器。 + +``` +sess_raw TEXT PRIMARY KEY, -- 宿主写入的原始会话 id +ledger_id TEXT NOT NULL, -- 解析出的账本 session_id +witness_probe TEXT NOT NULL, -- 首见时的见证探针(身份证据) +issued_at INTEGER NOT NULL, armed_at INTEGER, +tail_json TEXT, -- 上一轮宿主 id 数组的**尾 512**(JSON string[]) +head_json TEXT, -- 头 32 +count INTEGER, -- 数组全长 +max_pos INTEGER, -- append-only 位置前沿 +r3_latched_at INTEGER -- #6181 M3 持久 R3 断路器闩(NULL=未闩) ++ idx_l1_ledger ON l1_world(ledger_id) -- 服务 deleteSession 按 ledger_id 删 +``` + +(#7064 瘦身后的形态:原 `envelope_json` 整包已 DROP,降级为"尾 512/头 32 窗 + count + max_pos"——身份闸门→记账游标,中段不可见。) + +**fold_journal**(schema.ts:413-427)—— append-only 折叠决策日志: + +``` +session_id TEXT NOT NULL, jseq INTEGER NOT NULL, +kind TEXT NOT NULL DEFAULT 'fold' CHECK (kind IN ('fold')), +block_id TEXT NOT NULL, start_seq INTEGER NOT NULL, end_seq INTEGER NOT NULL, +input_hash TEXT NOT NULL, -- sha256 over messages[ start_seq..end_seq ](全代际,含 retired) +created_at INTEGER NOT NULL, +PRIMARY KEY (session_id, jseq), +UNIQUE (session_id, block_id) -- 幂等重摄入;被替换的块保留第一条 fold 行 +``` + +UNPIN **不存储**,读取时按三条件推导(块已非 active / 区间内有 live 行 / 区间内某 id 在 end_seq 之后重新 live)。`covered_ids` 列在 dev 期存在过,#7064 瘦身时由迁移 DROP(schema.ts:573-576)。 + +**egress_snapshot**(schema.ts:441-453)—— 切点物化的出站前缀**缓存**(最新行生效,旧行同事务裁剪): + +``` +session_id TEXT NOT NULL, snap_seq INTEGER NOT NULL, +watermark INTEGER NOT NULL, -- 切割时 pin 集合的最大 live seq = hash 作用域 +input_hash TEXT NOT NULL, -- sha256 over LIVE 行 seq<=watermark +prefix_ref TEXT, -- 派生前缀 JSON 的 CAS ref(NULL=仅 hash pin) +last_stable_id TEXT NOT NULL, -- 缝合点尾部 msgid +created_at INTEGER NOT NULL, +scope_rev INTEGER, -- 切割时 egress_scope_rev 的值(NULL=升级前行) +PRIMARY KEY (session_id, snap_seq) +``` + +(`ids_json` 已由迁移 DROP;`scope_rev` 为升级时 ADD。) + +**egress_scope_rev**(schema.ts:459-464)—— 每会话消息变更修订号,O(1) 读端信任: + +``` +session_id TEXT PRIMARY KEY, rev INTEGER NOT NULL +``` + +凡能改 input_hash 的写(retire/unretire、content_hash 编辑、硬删、归档恢复、旧 seq 填洞)旁路 bump;纯尾部追加不 bump(seq 单调 ⇒ 在所有 watermark 之外)。 + +**orphan_blocks**(schema.ts:475-483,#6785 step 7)—— 收割块内容落表(原 `sessions.orphans_json` blob 拆出): + +``` +session_id TEXT NOT NULL, block_id TEXT NOT NULL, +one_json TEXT, full_json TEXT NOT NULL, +PRIMARY KEY (session_id, block_id) +``` + +**state_fields**(schema.ts:491-498,#6785 step 8)—— kstate 无限增长字段拆行: + +``` +session_id TEXT NOT NULL, field TEXT NOT NULL, value_json TEXT NOT NULL, +PRIMARY KEY (session_id, field) +``` + +字段集**冻结**在 slim.ts `ROW_BACKED_KSTATE_FIELDS`(9 个:tokenSnapshot, tsStamps, lastPassIds, hiddenOrphanRefs, sweptOrphanCallIds, renderDeferredRefs, viewLatches, absorbed, rules),`tests/issue5874-state-fields` 钉死。 + +**egress_state / egress_flow**(schema.ts:508-530,7398 提案)—— 出站会话 op-log,双表同构: + +``` +session_id TEXT NOT NULL, log_seq INTEGER NOT NULL, +row_type TEXT NOT NULL CHECK (row_type IN ('B','I','D','U')), +msgid TEXT, -- B/D 为 NULL;I/U 必填 +payload TEXT NOT NULL, +created_at INTEGER NOT NULL, +PRIMARY KEY (session_id, log_seq) ++ idx_egress_state_msgid(session_id, msgid) ++ idx_egress_flow_msgid (session_id, msgid) +``` + +- `egress_state` = serve 读 + 压实物理裁剪(水位以下删); +- `egress_flow` = append-only **永不删**(任意 log_seq 可重建,回滚以此表为据,不靠客户端重发)。 + +行 payload 契约(egress-state.ts): +- `I`: `{"after": msgid|null, "ref", "tokens", "type", "content_hash", "role"?, "cas_ref"?}` +- `D`: `{"start": msgid, "end": msgid}`(闭区间,锚定 msgid 不是位置——"位置随其他 op 漂移,msgid 不会",KDD#11) +- `U`: `{"content_hash", "role"?, "cas_ref"?}`(msgid 列即目标) +- `B`: `{"entries": EgressViewRow[], "watermark": seq}`(条目 CAMEL 键 `contentHash/casRef`,与 I/U 的 snake 键不同) + +`EgressViewRow` = `{msgid, ref, tokens, type, contentHash, role?, casRef?}`;`casRef` = 该消息**最终 core JSON**(reasoning 剥离/图片缩放后、出站转换前)在统一内容 CAS 的键。 + +### 1.3 既有表的关键列(与本报相关的) + +- `messages` (schema.ts:181-195):PK (session_id, seq),UNIQUE (session_id, msg_id, content_hash),v8 加 `identity`(跨 save join 键)+ `retired_at`(墓碑,稳态不硬删),v10 加 `ts`。 +- `content` CAS (schema.ts:173-179):`hash PK, byte_len, refcount, algo='sha256', bytes BLOB`——所有字节(消息、块摘要、媒体、CCR、egress 前缀、egress core JSON)共一个库,引用计数。 +- `ccr_entries` (schema.ts:254-264):**本 PR 加 `orig_hash`**(+ 索引)—— byHash 半进 CAS。 +- `conversations` (schema.ts:273-281):**本 PR 加 `idx_conversations_session(session_id)`**——stage-0 每轮按 session_id 反查绑定,PK 服务不了。 +- `blocks`(197-214)、`dead_refs`(216-223)、`refs`(229-237)、`checkpoints`(239-252)、`session_ext`(291-298)、`kv_store`(303-310)、`message_media`(315-320)、`sessions`(148-171,`kstate_json/ccr_json` 语义被 slim 改变,DDL 未动)。 + +### 1.4 MySQL 对等(mysql.ts +274) + +移植了:`orphan_blocks`、`state_fields`、`ccr_entries.orig_hash`(+ 索引、probe 迁移、归档/恢复/级联)。 +**未移植**:`l1_world / fold_journal / egress_snapshot / egress_scope_rev / egress_state / egress_flow`——快照/egress 平面是 **SQLite 车道专属**(shadow.ts `egressOpLogSync()` 网络后端返回 null,录制器整轮跳过)。 + +--- + +## 2. 实现逻辑(按模块) + +### 2.1 slim.ts(新,185 行)—— CCR 瘦身 + kstate 拆行 + +- `sessions.ccr_json` 只存瘦信封 `{version:1, ccrSlim:1, byRef:{key:{hash}}, orphanByHash?}`(slim.ts:20-48);`ccr_entries` 行存两半 CAS:`content_hash`=条目 JSON,`orig_hash`=条目背后原文。 +- **null-orig 契约**:某行 `orig_hash=NULL` ⇒ 整个 store 回退文件车道(hydrate 返回 null,永不半份 byHash);`""`(空串 hash)=合法空原文(fork 回执 stub);byHash 缺条目必须落 null 不能落 `""`。 +- kstate:`splitKstate` 把 9 个冻结字段从 `kstate_json` 拆到 `state_fields`(slim.ts:155-185);读端行覆盖列,写端变更 upsert、变 undefined 删行。 +- 稳态 ingest 跳过判断升级为 `ord && content_hash && orig_hash` 三者全等。 + +### 2.2 snap.ts(新,493 行)—— stage-0 身份账本(只判身份,不压缩不供字节) + +- `snapL1Record` → `hit|echo|arm|wait|issue|red`(snap.ts:80-114):新行 issue;`ledger_id` 不同 = **red 硬冲突,什么都不写**(R3 域,调用方留 legacy);armed+探针匹配 echo;armed+不匹配 hit;未 armed 匹配 → arm(一轮武装);其余 wait。 +- 世界游标:`snapL1WindowRead/Upert`(snap.ts:138-165)读写 l1_world 的 tail/head/count/max_pos(窗宽 `WORLD_TAIL_K=512` / `WORLD_HEAD_K=32`,identity.ts:29/32)。 +- R3 闩:`snapR3LatchWrite`(snap.ts:179-191)UPSERT 占位行(首次故障可先于任何 snapL1Record);闩住的会话永不重新武装。 +- `foldInputHash`(snap.ts:252-268):对 `[start_seq,end_seq]` **全代际**(无 retired 过滤——折叠自己的 ingest 同事务 retire 覆盖行,live-only 聚合在折叠时刻是空的)逐行 `${seq}\u0000${msg_id}\u0000${content_hash}\n` sha256。 +- `foldUnpins`(snap.ts:326-370):唯一的 UNPIN 谓词(三条件,见 §1.2)。 +- egress 切读:`egressInputHash`(407-424,live-only,作用域=watermark)、`egressScopeRev`/`bumpEgressScopeRev`(430-450)、`egressLatestRow`(477-486)。 + +### 2.3 egress-state.ts(新,365 行)—— op-log 引擎(纯函数,可测) + +- `diffEgressView`(153-204):骨架 = `next` 中 msgid 在 `prev` 里**升序出现**的最长子序列(双指针扫描);骨架行 contentHash 变 ⇒ `U`;prev 非骨架 ⇒ 按 prev 顺序发最大连续 `D` 段;next 非骨架 ⇒ 按 next 顺序发 `I`(锚 = 前行);**移动 = D+I,无 M op**;op 顺序 D→U→I;不变式 `replay(prev, diff(prev,next)) === next`。 +- `nextLogSeq` 从 **flow** 取 MAX(state 被压实裁剪过,flow 是超集)。 +- `egressStateAt(db, sid, x)`(264-274):flow 重放到 `log_seq <= x` = 任意历史点重建(回滚)。 +- `egressStateCompact`(278-292):重放结果写成一条 `B` 行(两表)+ 物理删 state 水位以下(flow 不动)。 +- 任何锚点/端点/重复/畸形违例 ⇒ 抛 `EgressStateCorruptError`(36-42)⇒ 调用方**整会话降级 legacy**,永不模糊兜底。 + +### 2.4 egress-record.ts(新,102 行)—— 每轮录制(seg 3) + +- `buildEgressViewRows`:首行 system(`msgid="system:"+hash32`,`casRef=CAS(系统文本)`,**字节稳定就是缓存锚点**);每消息一行,`contentHash=rowHash([id,role,contentType,text])`,`casRef=CAS(最终 core JSON)`(head commit `39941c816` 起工具行 sidecar(toolCallId/toolName)随 body 进 CAS,serve 不再排除工具行)。临时行(nudge、sysNotes)按设计排除(KDD#7)。 +- `recordEgressTurn`:prev = 会话缓存或一次 hydrate;diff;有 op 才 append;**批内含 D(折叠形状)才压实**。 +- 本轮修复:unchanged 行**复用** prev 的 `casRef` 不再重复 putContent(原来每轮每行 refcount +1,只增不减;唯一释放点在会话清扫——见 §5)。 + +### 2.5 egress-serve.ts(新,65 行)—— 从日志重建 + 真 swap(seg 4b) + +`rebuildOutboundFromLog(view, store)`:跳过 system 行(系统块恒 live 构建,前缀缓存锚点纪律);缺 `casRef`(seg4 前行)或 CAS body 缺失 ⇒ 返回 null(**整轮**留 legacy);body 必须解析为含 id/role 的对象且 role 合法;然后走与 legacy **完全相同的**链 `hardenOpenaiAssistantContent(coreToOpenai(systemToUser(msgs)))` ⇒ 按构造字节一致。 + +### 2.6 入站:水位 delta ingest(#3133 既有,本 PR 未改内核) + +- 决策点 `shadow.ts:679-703`(`ingestDelta`):`ingestWatermark`(sessions.last_ingested_seq + 持久 tailSeqs,store.ts:2329)→ 边界探针(:2403)→ #3919 头部 id 探针(:2418)→ `planIngestDelta` → `ingestSession(delta)`;任何一环 null/false ⇒ 精确全量路径。 +- `planIngestDelta`(ingest.ts:311-405)返回 null 的形状:pluginSnapshot 非空 / decompositionLog(#3301 导入) / 无水位 / `tail.length===trustedLen && !hasRawTail`(等长尾只在 raw 谱系是合法空 delta,#2960 ⑤ 折叠重渲染可能乱序)/ **`tail.length < trustedLen`(线变短 = 折叠发生 ⇒ 全量)** / 尾部或头部探针不符。 +- 新消息循环只序列化+hash **新**的(`O(N)`),占位 seq `watermark+1+i`。 +- 店端守卫 `store.ts:1229-1281`(BEGIN IMMEDIATE 内):边界**事务内复验**(1231-1232);每条新消息按 `(session_id,msg_id,identity)` 探针,任一已有行 ⇒ 全量(1239-1247,#4390 identity 位置无关,retired 也算存在);重编号 `probe.maxSeq+1+i`(1253);tailSeqs 前缀一致性(1256-1268)。**陈旧/外部水位最多浪费一次冗余全量写,永不写错行。** + +### 2.7 parse 层 parse-defer(本 PR 新增) + +- `fastpath-scan.ts`:`scanMessagesSpans` 扫原始 body 提取 messages 元素 span(不 parse);`synthesizeTopLevel` 重建顶层对象,`messages: null` 作"延迟"标记(用 null 不是 `[]`——`Array.isArray([])=true` 会破 MISS 重 parse 路径,且 relay 守卫对 null 放行)。 +- `handle.ts` 门:openai 形 + pf 协议戳 + body ≥ 1MiB ⇒ 走延迟;HIT 轮省掉整包 parse;MISS 才在 prepare-openai 一次性从文本重 parse。 + +### 2.8 快路径:水位 + delta 臂(fastpath.ts / handle.ts) + +- **全量重发水位**(#5418,rollout 5%):响应盖 `x-bili-pf-seq`(messages 元素数,零分配结构扫描测出);客户端回带 `x-bili-pf-count` + `x-bili-pf-tail-ids`(宽度 `T − stampT + 1`,首格锚点,`?` 占位);命中 = 上一轮 canonical cores + 仅尾窗新解析(跳整包 parse/转换/钉存——#5403 OOM 生产者)。 +- **delta 臂**(本 PR,`pluginDeltaResend` 0):客户端 body.messages **只发**新元素 L 个;服务端用钉存前缀核 + 小切片重建视图。任何不成立声明(count/body/尾宽不符、已知清单后锚链断、灰度关)⇒ 400 `pf_delta_reanchor` + `x-bili-msgid-mode: full`,客户端当轮改发全量形态——**一轮收敛,永不静默错解**;delta miss 绝不 fall through 到 legacy parse(把尾当全史 = 毁会话)。 +- 失配语义(fastpath.ts:296-304):`stamp > T` = "ahead"(线缩了=发生过压缩)⇒ 清 mode 下轮全量重锚;`stamp === T` = "no-progress"。 +- **bug A 修复**(b4bbaaef):HIT 头检查排除 id 字段——pin 之后账本行 msg_id 是宿主 UUID,而 fresh 侧是内容 hash id,逐格比对必假阴性。 +- **bug B 修复**(2c3bdd10):HIT 轮快照渲染只出 ~2 行 ⇒ rawTail 短 ⇒ completeLog ⇒ sweep retire 全部前缀行 ⇒ 账本排空 ⇒ 每轮 pin 降级。修:命中即置 `session.lastMessagesFolded=true`。 +- **bug C 修复**(9efbf8b7):HIT 轮账本新行带 h_ id 不带宿主 UUID(pin 行只覆盖窗口内)⇒ 尾宽永远差 1 ⇒ full/incremental 永久交替。修:命中轮用 `s0.pinned` 尾窗重盖 ledgerWindow 再 snapshotMessages。 + +### 2.9 增量 msgid pin(#7064 §8,默认 100) + +稳态轮用有界 `x-bili-pf-tail-ids`(锚点+新增,≤2048 个,`?` 占位)或短 body id 数组申报;服务端**从账本 live 行重盖前缀**(ingest INSERT OR IGNORE on (session_id,seq)——位置上的 msg_id 只写一次)。接受规则(2026-11-19 定型):声明尾段 [T-L,T-1] 与账本重叠段**逐行相等**,任一行漂移 ⇒ 降级 full,永不 400;超长会话(>2048)截断尾清单同样可 pin(重叠段更长=证明更强)。快照轮(账本空/折叠后重锚)发全量;全量优先。响应盖 `x-bili-msgid-mode: incremental|full`(4xx 即清,与 pf-seq 同一学习点 pfLearnStamp)。失配一轮自愈。 + +### 2.10 admin 面 + +新增回环路由 `GET /__bili/snapshot/r3-latches` / `POST /__bili/snapshot/r3-unlatch`(admin.ts +56)。 + +--- + +## 3. 数据流转逻辑(一轮请求的完整路径) + +### 3.1 稳态轮(+N 条新消息,pf 命中)——两条增量都生效 + +``` +客户端 POST /bili/http:///chat/completions + body: 全量 messages(水位全量臂)或 仅尾 L 条(delta 臂) + + biliMsgIds(全量形态)或 x-bili-pf-tail-ids 尾形态 + + x-bili-pf-count / x-bili-pf-seq(若已学) + → [handle] readBody(155MB 级)→ pf 协议扫描(fastpath-scan,零 parse) + → [fastpath] HIT(戳前进+锚链对):跳整包 parse/转换/钉存 + → [prepare-openai] 重建视图 = 钉存前缀核 + 尾窗解析(delta 臂) + ├─ egressPlane 开:recordEgressTurn(稀疏 I op,几 KB) + │ → rebuildOutboundFromLog(op log + CAS)→ swap 进出站字节 + └─ 关:legacy 内核视图 → coreToOpenai 链 + → [ingest] shadow.ingestDelta:watermark+边界+头探针 → planIngestDelta + 只序列化/哈希新 N 条 → 行 append 到 watermark+1..(同事务写 watermark) + → 出站:render-refs 标签 + nudge(每轮临时,不入表)→ 上游 + → 响应:x-bili-pf-seq(新 T)+ x-bili-msgid-mode: incremental +``` + +**每轮新增存储 = 新消息 CAS 字节 + 一条 I op(几 KB)。** 出站重建稳态 O(尾部)。 + +### 3.2 折叠轮(客户端线变短 / 服务端 fold 落账) + +``` +fold tx:blocks 插/更 + 覆盖行 retire(同事务)+ fold_journal 一行 + (input_hash=区间全代际聚合;egress_scope_rev 因 retire 而 bump) +→ 下一轮入站:wire 变短 ⇒ planIngestDelta null ⇒ 精确全量路径 + (解析全量 + INSERT OR IGNORE 去重 + retire 落账 + watermark 重设) +→ 出站:egress 视图出现 -D(区间)+ +I(摘要行)⇒ 批内含 D ⇒ 压实 B 行 + (B = 全量视图行列表,设计自认 ≈8MB @10w 行;CAS 字节不复制,只指针) +→ pf 协议:stamp>T "ahead" ⇒ 清 mode ⇒ 该轮发全量形态重锚 ⇒ 次轮回增量 +``` + +**代价集中在折叠那一轮**(实测 ~0.5–0.8s),稳态轮回到 O(尾部)。这是设计内的节奏(折叠=重锚轮),不是 gap。 + +### 3.3 异常/降级路径(全部"整轮 legacy,不猜") + +| 触发 | 动作 | +|---|---| +| pf 戳缺失/回退/锚链断 | 整轮全量解析+全量路径,重新盖戳 | +| delta 声明不成立 | 400 pf_delta_reanchor + msgid-mode: full(一轮收敛) | +| 尾形态 id 与账本重叠段任一行漂移 | 降级 full 信号,下轮重锚(D1 家族,永不 reject) | +| l1 red(ledger_id 冲突) | 不写任何东西,留 legacy,R3 域处置 | +| EgressStateCorruptError(15 类违例) | 该会话停止录制/serve,整轮 legacy | +| serve 重建缺 casRef/CAS body/未知 role | 整轮 legacy(4a 影子对拍继续度量) | +| 水位陈旧/外部写 | 事务内探针失败 ⇒ 精确全量路径,一次冗余写 | + +### 3.4 生命周期(删除/归档级联) + +`deleteSession`(store.ts:~2586):messages/media/blocks/**fold_journal/egress_snapshot/egress_scope_rev/orphan_blocks/state_fields** + dead_refs/refs/ccr_entries/conversations/session_ext/sessions + l1_world(按 ledger_id)+ CAS 释放与 scoped collect。 +`archiveSession`:同上(除 l1_world),JSONL 导出**先于** COMMIT 追加,状态翻 archived。 +**本轮修复**:两处都补了 `egress_state/egress_flow` 清扫(原漏删 ⇒ op-log 行永久孤儿,属"存储只增不减"p1 类;CAS 引用的 egress 半也一并释放,消息字节仍由 messages 行持有)。 + +--- + +## 4. 开关总账 + +### 4.1 rollout.features(src/rollout.ts;env `BILI_ROLLOUT_`) + +| 键 | 默认 | 本 PR 变化 | 作用 | 回滚 | +|---|---|---|---|---| +| `pluginLaneServerHistory` | **100** | 0→100(GA,owner 裁定 2026-11-13) | ⑤ 服务端正史:biliMsgIds pin 行身份 + /__bili/plugin/render + 增量车道通告 serverHistory | env=0 | +| `pluginMsgIdIncremental` | **100** | 新增即开 | ④ 尾形态 id 申报,账本重盖前缀 | env=0 | +| `pluginFullResendWatermark` | 5 | (既有 5% 灰度) | ③ 水位快路径 | env=0 | +| `pluginDeltaResend` | 0 | 新增,关 | ③ delta 尾切片臂;0 = 对 delta 请求 400,客户端一轮收敛回全量 | 天生回滚态 | +| `egressPlane` | 0 | 新增,关 | ② 整条新链路一个开关:稀疏 op 记账(四车道)+ 日志回放 serve(字节一致 swap)+ pinned 插件车道服务端权威(裁定 7439-3/5);关 = legacy 链路零行、与特性前字节一致。取代已退役的 `egressOpLog`/`egressServeLog` 双键 | 天生回滚态 | + +**无新增静态 env 变量、无新增 CLI flag**(rollout env 由键名动态派生,docs-coverage CI 对动态名双向豁免——这也是 `pluginMsgIdIncremental` 此前漏文档的根因,本轮已补 yaml 条目)。 + +### 4.2 配置对象 + +| 键 | 默认 | 作用 | +|---|---|---| +| `storage.snapBase.enabled` | 关 | ① 快照底座平面(snapBase 门,knobs.ts:1001 fail-closed) | +| `storage.snapBase.cohorts` | — | 协议白名单 | +| `storage.snapBase.cutEveryKUserTurns` | 8 | egress_snapshot 重切节奏(重切=整前缀重哈希+重序列化+写 CAS,故限频;失效类永远立即重切) | +| `network.ingestBodyBytes` | 32MiB | 入站 body 上限(实测 155MB 场景需调 1GiB;env `BILI_INGEST_BODY_BYTES` 优先) | + +### 4.3 响应头(协议面) + +`x-bili-pf-seq`(bili→客户端,元素数戳)、`x-bili-msgid-mode: incremental|full`(4xx 即清)、`x-bili-pf-delta: 1`(客户端→bili,hop 头,转发上游前剥,headers.ts:40)、`x-bili-pf-tail-ids`(客户端→bili,≤8KB 预算,超预算省略头不抑制 body 全量形态)。 + +--- + +## 5. 实测验证(理想模型增量压测)与本轮修复 + +### 5.1 压测方法(perf/,2026-11-21) + +构造 = 用户场景的 plugin 形态:客户端历史从 1 条增长(纯 append-only),fake 理想模型每 ~300 refs(≈5w token 增长)注入一次 compress 工具调用(折最旧段、保 1800 refs≈30w token 存活尾)——**客户端在宿主侧应用折叠**(plugin 语义:模型发起、客户端执行),因此线上 wire 恒定 = 存活视图(1800–2100 条,0.7–0.8MB 请求体,出站 200–253k token),`cum` 计累计产生量(到 10w)。臂:master(`b10c1a0d4`,无 pf)/ PR 全量臂 / PR delta 臂。三臂同 fake 校准、同 driver(node:http `connection: close`——bili 按 #1982 设计每轮后关连)。**注意:本批所有跑都是 egress 平面关闭态**(7398 臂未测,见 §5.4);155MB 原始 wire 形态(宿主不折叠的客户端)不在本批覆盖内。 + +### 5.2 实测(RUN1 batch=500 三臂跑满 210 轮;RUN2 batch=100 仅 PR 全量臂跑到 g943≈9.4w,master 臂因代理误杀中止,delta 臂未启动) + +**RUN2,PR 全量臂,按累计规模**(wire 恒 ~720–840KB,出站 200–253k token,折叠每 ~3 轮): + +| 累计产生 | 1w | 3w | 5w | 7w | 9w | 9.4w(非折叠轮) | +|---|---|---|---|---|---|---| +| PR 全量 | 526ms | 501ms | 1170ms | 1346ms | 1650–1737ms | 1821ms | + +**RUN1 @ 10w 累计稳态段 10 轮**(wire 720KB,视图 1801 条,折叠已停): + +| 臂 | 稳态范围 | 备注 | +|---|---|---| +| master 全量 | 1312–1624ms(p50 ~1338) | pf=null(无水位) | +| PR 全量 | 1613–2322ms(末轮 1613) | pf=1804,mode=incremental | +| PR delta(无效臂) | 1891–2849ms | 配置缺 `pluginDeltaResend` ⇒ x-bili-pf-delta 被忽略 = 全量+死头 | + +**诚实结论**: +1. 视图形态 wire(~750KB)下,单轮耗时随**累计账本规模**线性上升(1w→0.5s,9.4w→1.8s),与 wire 大小无关——成本在 9.4w 条 messages 行 + 309 条 fold_journal 的入站 ingest/扫描域(slow-ingest 日志点名 #5744,WAL 33MB/次),不是本 PR 结构。 +2. "整轮 <1s" 在 ≤3w 规模达成(501ms),5w 起越过(1170ms)。 +3. 静态重放(comp1200k:781 条/1.2MB 视图,无账本累积)测得 delta 臂 32–62ms、全量 ~91ms——结构天花板(纯 wire 受限)远低于增长跑,差值 = 账本累积成本,是下一步 profiling 对象(见 §5.4)。 +4. delta 臂在增长场景无有效数据(需带 `pluginDeltaResend:true` 的代理重跑)。 + +### 5.3 本轮发现与修复(小问题直接修,已随本报告提交) + +1. **[p1 存储只增不减] deleteSession/archiveSession 漏删 egress_state/egress_flow** ⇒ op-log 行 + 其 CAS 引用永久孤儿。修:store.ts `deleteEgressOps` + 两处级联。 +2. **[p1 存储只增不减] buildEgressViewRows 每轮对每条未变行重复 putContent**(CAS refcount 每轮每行 +1,无对应释放)⇒ 开了 egress 键的会话内容库无限膨胀。修:未变行复用 prev casRef(缺的补上,变的重新 put)。 +3. **[文档] `docs/proposals/snapshot-compression-foundation/`** 描述已退役的 v1 四表平面(snap_rows/snap_log/snap_ckpt 合入前删除;标题还说 "DDL landed in #5901"),与现 v12 形态矛盾 ⇒ **整目录移除**。 +4. **[文档] `docs/storage-relations.{en,zh}.svg`** 缺 5 张新表、fold_journal 卡片还写已删的 covered_ids、egress_snapshot 卡片还写 ids_json、⑦级联卡片不准确 ⇒ **移除**,由本报告接替。 +5. **[文档] rollout.yaml 三处过时**:pluginLaneServerHistory 默认值 "0 (off)"(代码 100)、egressServeLog 描述 "seg 4a LOG ONLY never swaps"(4b 已真 swap)、缺 pluginMsgIdIncremental 条目 ⇒ 全部更正/补齐,重新生成 docs。 +6. **[注释] rollout.ts egressServeLog 注释** "(AND egressOpLog is on)"+"tool rows in v1" 两处过时 ⇒ 更正。 +7. **[开关] egress 双键合并为一个**(owner 裁定 2026-11-21):`egressOpLog` + `egressServeLog` 退役,由单一 `egressPlane` 灰度键取代——开 = 完整新链路(记账 + serve + 服务端权威,四车道),关 = legacy 链路与特性前字节一致。两个旧键从未发布(默认 0、天生回滚态),无需迁移。 + +### 5.4 留给后续(不在本 PR 范围) + +- **egress 键开启态的最小验证**(待定):155MB 原始 wire(宿主不折叠)+ egressServeLog 真 serve ⇒ 出站 30w token 的完整场景,逐轮对"每轮新增存储几 K"红线(WAL diff)——这是 7398 验收线,本批未覆盖。 +- 累计 5w+ 规模单轮 1.1–1.8s 的账本累积成本 profiling(slow-ingest #5744 域;含折叠轮 retire+重锚、fold_journal 309 行的折叠交叉门、kernel 扫描)。 +- delta 臂在增长场景的有效重跑(带 `pluginDeltaResend:true`)。 +- 压实 B 行 ≈8MB @10w 行 vs "每轮新增存储几 K" 红线:设计自认的例外(7398 §5);若要彻底守住红线,可把 B 行改成"区间引用+防伪指纹"(#7064 对 l1_world 的降级思路),代价是失去"状态表独立可重放"的封闭性——需 owner 裁定。 +- 归档恢复后 egress op-log 从空重启(下一轮全量 +I 重建,正确但有一次全批)。 diff --git a/docs/technical-notes.md b/docs/technical-notes.md index c3ce68259d..a82a94dbef 100644 --- a/docs/technical-notes.md +++ b/docs/technical-notes.md @@ -603,3 +603,19 @@ Declared trust boundary (§7.4 — human review required, no auto-merge): a clie Honest limits: (1) once a multimodal tail (content array containing `image_url`) lands in the stash, the lane degrades to full legacy turns for the remainder of that history — the kernel renders user content-array images differently when the message is the LAST element versus mid-history, so a frozen stash cannot reproduce legacy bytes position-independently; text workloads (the #5403 reporter's shape) are unaffected. (2) The head fragment is re-parsed (8 elements) each hit solely for the system-run check. (3) Both `x-bili-pf-*` headers are stripped upstream (`UPSTREAM_HOP_HEADERS`) — protocol headers never leak into model traffic. Measured (provenance: `tests/issue5403-watermark-fastpath.test.ts`, hermetic dual-proxy A/B on the same corpus): hit turns skip the full parse/conversion/pin entirely; resident `lastMessages` 8684 B → 1210 B (7.2×) on an 8-turn corpus including a multipart image landing turn; outbound bytes byte-identical to legacy modulo kernel `time="…"` attributes across all turns; `x-bili-pf-*` absent from every captured upstream request, both phases. Unit coverage: `tests/fastpath.test.ts` (full verdict ladder incl. breaker TTL re-arm), `tests/fastpath-scanner.test.ts` (adversarial JSON table, depth cap, surrogate pairs, 50k-element scale), `tests/lane-decisions-matrix.test.ts` (gate row matrix, both directions). + +## Delta resend — the tail-slice form of the watermark lane + +The watermark fastpath above skips the server's re-processing of the re-sent prefix, but the client still UPLOADS the whole history every turn. Delta resend removes the upload: on a steady turn the client sends ONLY the new elements. + +Wire form (client → proxy, all three headers, one extra): `x-bili-pf-delta: 1` + `x-bili-pf-count: T` (new total element count) + `x-bili-pf-tail-ids` = anchor+new host msgids, width exactly L+1 (L = T − stampT new elements; `tail-ids[0]` anchors the previous turn's last element), and `body.messages` = exactly the L NEW elements (`[T-L .. T-1]`, no prefix, no anchor). The anchor element itself is NOT in the body — it rides the header only. + +Server side: `evaluateDeltaResend` (src/server/fastpath.ts) runs BEFORE the full-body lane when the header is present. Rebuild: `view = stashed prefix cores + rekeyWindowCores(parsed tail)` → exactly the same `FastpathHit` the full lane produces → everything downstream (stage-0 pin, kernel processTurn, ingest, snapshot, ledger window) is the untouched full-lane code path. + +Re-anchor ladder — any unsatisfiable claim returns 400 `pf_delta_reanchor` with `x-bili-msgid-mode: full` and NO `x-bili-pf-seq` stamp; the client resends the SAME turn in full form and converges within one round-trip (KDD#15 one-round convergence). Gate order: header value / rollout / plugin-mode / seed presence / model match / no multimodal seed / count present / L in [1..2048] / body length == L / tail-ids width == L+1 / **anchor continuity** (strict after a known previous list — `tail-ids[0]` must equal the previous turn's last declared id; a tail-less seed lets the first delta turn open unverified, then every list is chained) / body scan total == L / window conversion shape. No anchor byte re-parse: the anchor's identity is verified on the ledger/pin side (the 2×2 channel), never by re-parsing the previous turn's bytes. + +Editing or deleting the LAST message of the previous turn breaks the anchor chain → re-anchor 400 → full form. Deleting/rewriting EARLIER history does NOT: the ledger/pin overlap check catches it one round later via the existing KDD#21 "truncated tail" degradation (full signal, re-anchor), never a 400. + +Rollout: `rollout.features.pluginDeltaResend` (default 0 — OFF). Delta bodies are unprocessable by servers that do not ship this arm, so the key stays at 0 until client send-sides exist; env override `BILI_ROLLOUT_PLUGIN_DELTA_RESEND`. A `x-bili-pf-delta` request hitting a rollout-off server gets the same 400 re-anchor (rollout-off gate) — never a silent misparse. All pf-* headers remain upstream-hop (stripped before model traffic). + +Measured (100k-message raw session, loopback): full resend 15.9s/turn → 12.7s/turn (inbound 155MB → KB-scale; the remaining cost is the egress rebuild of the full rendered view — see the compressed-shape numbers for the realistic case). Compressed steady state (context ≈ 100k tokens ≈ 400KB body): 36ms/turn full → 21ms/turn delta; ≈ 300k tokens ≈ 1.2MB body: 128ms → 53ms. 10k-message raw session: 880ms → 550ms. diff --git a/docs/technical-notes.zh-CN.md b/docs/technical-notes.zh-CN.md index f051d78c66..305078b0e6 100644 --- a/docs/technical-notes.zh-CN.md +++ b/docs/technical-notes.zh-CN.md @@ -180,3 +180,19 @@ chat wire 上 bili 自己 vs 客户端自带:插件车道上,请求体里等于 诚实边界:(1) 一旦多模态尾部(含 `image_url` 的 content array)进入缓存,该段历史此后整段降级回 legacy 全量轮——kernel 按位置区分 user 图片消息在尾部与在历史中部的渲染,冻结缓存无法做到位置无关地复现 legacy 字节;纯文本工作负载(#5403 报告者的形态)不受影响。(2) 每次命中会现解析头部 8 个元素,只为做 system-run 校验。(3) 两个 `x-bili-pf-*` 头都在转发上游前剥掉(`UPSTREAM_HOP_HEADERS`)——协议头绝不漏进模型流量。 实测(出处:`tests/issue5403-watermark-fastpath.test.ts`,同语料 hermetic 双代理 A/B):命中轮完全跳过全量解析/转换/钉存;含一个 multipart 图片落位回合的 8 回合语料上,常驻 `lastMessages` 8684 B → 1210 B(7.2×);全程出站字节与 legacy 逐字一致(仅 kernel `time="…"` 属性归一);两个阶段捕获到的上游请求里均无 `x-bili-pf-*`。单元覆盖:`tests/fastpath.test.ts`(verdict 阶梯全表,含熔断 TTL 解除后重播种)、`tests/fastpath-scanner.test.ts`(对抗 JSON 表、深度上限、surrogate 对、5 万元素规模)、`tests/lane-decisions-matrix.test.ts`(门矩阵,双向)。 + +## Delta 重发 —— 水位快路径的尾切片形态 + +上面的水位快路径只省了服务端对重发前缀的重复处理,客户端每轮仍然把**整个**历史上传。Delta 重发把上传也去掉:稳态轮客户端只发**新增**元素。 + +线上形态(客户端 → 代理,共四个头,body 只剩尾部):`x-bili-pf-delta: 1` + `x-bili-pf-count: T`(新的元素总数)+ `x-bili-pf-tail-ids` = 锚点+新增的宿主 msgid,宽度恰好 L+1(L = T − stampT 个新元素,`tail-ids[0]` 锚定上一轮最后一个元素),`body.messages` = **恰好**这 L 个新元素(`[T-L .. T-1]`,无前置、无锚点)。锚点元素本身不在 body 里,只走头。 + +服务端:`x-bili-pf-delta` 存在时 `evaluateDeltaResend`(src/server/fastpath.ts)在全量车道**之前**跑。重建:`view = 存好的前缀 cores + rekeyWindowCores(解析出的尾部)` → 产出与全量车道完全一致的 `FastpathHit` → 下游(stage-0 pin、kernel processTurn、ingest、快照、账本窗口)走的都是未改动的全量车道代码。 + +重锚阶梯——任何不成立的声明返回 400 `pf_delta_reanchor` + `x-bili-msgid-mode: full`,**且不打** `x-bili-pf-seq`;客户端本轮改发全量形态,一个来回内收敛(KDD#15 一轮收敛)。门序:头值 / 灰度 / plugin 模式 / 有无种子 / model 一致 / 非多模态种子 / count 在场 / L ∈ [1..2048] / body 长度 == L / tail-ids 宽度 == L+1 / **锚点连续性**(有已知上一清单后强制:`tail-ids[0]` 必须等于上一轮最后一个申报 id;无尾清单的种子允许第一轮 delta 不验证开局,之后逐链)/ body 扫描总数 == L / 窗口转换形状。不做锚点字节重解析:锚点身份由账本/pin 侧(2×2 通道)校验,绝不用上一轮字节复核。 + +编辑/删除上一轮**最后一条**消息会断锚链 → 400 重锚 → 全量形态。改写**更早**的历史不会:账本/pin 重叠检查在下一轮经 KDD#21「截断尾」降级(全量信号、重锚)接住,永不 400。 + +灰度:`rollout.features.pluginDeltaResend`(默认 0 —— **关**)。delta body 对未上车的服务端不可处理,故保持 0 直到客户端发送端就位;env 覆盖 `BILI_ROLLOUT_PLUGIN_DELTA_RESEND`。带 `x-bili-pf-delta` 的请求打到灰度关的服务端得到同样的 400 重锚(rollout-off 门)——绝不静默错解。所有 pf-* 头保持上游 hop(转发模型前剥掉)。 + +实测(10w 条原始会话,loopback):全量重发 15.9s/轮 → 12.7s/轮(入站 155MB → KB 量级;残余成本是整份渲染视图的出站重建——真实压缩场景见下)。压缩稳态(上下文 ≈ 10w token ≈ 400KB body):36ms/轮 全量 → 21ms/轮 delta;≈ 30w token ≈ 1.2MB body:128ms → 53ms。1w 条原始会话:880ms → 550ms。 diff --git a/reference/architecture.md b/reference/architecture.md index 728248834d..2ba6dd2b25 100644 --- a/reference/architecture.md +++ b/reference/architecture.md @@ -39,7 +39,7 @@ billion-context/ │ │ ├── fastpath-scan.ts # Zero-decode structural span scan of chat-completions bodies (#5403) │ │ ├── lane-decisions.ts # Single resolver for per-lane gating decisions (#3417) │ │ ├── pi-subagent-channel.ts # Channel fallback for pi-subagent children missing ACP tools (#2268) -│ │ ├── snapshot-render.ts # Snapshot render: outbound prefix from server transcript (#3756) +│ │ ├── snapshot-render.ts # Snapshot render: outbound prefix from server transcript (#3756) + egress stable-prefix cache splice/cut (#5874 v2) │ │ ├── thread-tracking.ts # Anthropic Threads previous_message_id repair (#2701) │ │ ├── threads-lane.ts # Anthropic Threads lane: mirror, budget, reforge (#2926) │ │ ├── thinking-posture.ts # Signature-less thinking posture: 400-wording detector + wire strip + session memory (#5998) @@ -48,6 +48,7 @@ billion-context/ │ ├── knobs.ts # Single knob resolver: env > config file > default (#2030) │ ├── config.ts # Config file (billion-context.json) + providers table + loadOptions │ ├── model-endpoints.ts # Declarable custom-wire model endpoints: pattern parse + longest-prefix match, one source feeds proxy classifier + client claim (#1295) +│ ├── responses-system-sections.ts # Single source of truth for the Responses-wire developer-section join: steady prepare + proxy-loop round-2 rebuild share one separator (#6922/#6944) │ ├── instance.ts # Live-instance registry record in the proxy-origin file (#394/#403/#417) │ ├── port-owner.ts # Post-bind port-ownership verification + pointer-clobber guard (#2941) + EADDRINUSE holder diagnostics (#4762) │ ├── event-loop-watchdog.ts # Worker-thread event-loop wedge detector: self-exit after 5 min of continuous main-loop silence (#4762) @@ -145,18 +146,28 @@ billion-context/ │ ├── session-gc.ts # Session-file GC, opt-in (sessions.gc.* / BILI_SESSION_GC*) (#1082) │ ├── persist.ts # On-disk session persistence (kernel StateStore) │ ├── persist-eperm.ts # EPERM/EACCES write-failure alert layer (#362) +│ ├── snapshot/ # Stage-0 snapshot identity plane: world match, envelope codec, decision glue (#6181/#5874) +│ │ ├── identity.ts # matchWorld verdicts (echo/probe/scatter/issue/diverge/resync) + canon parse (#5874) +│ │ ├── plane.ts # Stage-0 decision glue: R3 identity check, 400 fault breaker, lane decide (#6181) +│ │ ├── stage0.ts # Lane-agnostic stage-0 entry: pin + R3 + stage-0 decision + assembly, shared by the openai and responses lanes (#6181) +│ │ └── types.ts # Shared stage-0 types (type-only, kernel-public-types only) (#6181) │ ├── storage/ # Unified storage layer (schema, SQLite engine ladder, backends) (#2671) │ │ ├── backend.ts # Storage backend contract: sqlite + mysql operation seam (#3931) │ │ ├── canonical.ts # Shared canonicalization for storage digests (#2671) │ │ ├── driver.ts # SQLite driver ladder: bun/node builtin, host, self-downloaded (#2833) │ │ ├── engine-bootstrap.ts # Self-downloads pinned better-sqlite3 prebuilt when no engine exists │ │ ├── identity.ts # messages-table identity hash: strips position metadata (#4390) +│ │ ├── egress-state.ts # Egress op-log outbound view: dual-table state+flow, replay, compaction, any-seq rollback (#7398) +│ │ ├── egress-record.ts # Egress op-log RECORDING: build the outbound view rows, diff vs prev, append sparse ops, fold-triggered compaction (#7398, rollout egressPlane) +│ │ ├── egress-serve.ts # Egress op-log SERVE: rebuild the outbound elements from the self-contained op-log rows (byte-identity chain, ladder recovery, whole-turn degrade) + shadow byte comparison (#7398, rollout egressPlane) │ │ ├── ingest.ts # Legacy → unified row mapping + legacy view rebuild (#2671) │ │ ├── integrity-check.ts # Cached + background quick_check off the boot critical path (#4609) │ │ ├── mysql.ts # MySQL 8.0 backend implementing the storage contract (#3388) │ │ ├── runtime-status.ts # Runtime storage degradation status for status surfaces (#3040) -│ │ ├── schema.ts # Unified storage schema DDL + migrations (v11) +│ │ ├── schema.ts # Unified storage schema DDL + migrations (v12) │ │ ├── shadow.ts # Unified-storage sidecar: shadow double-write / full read flip +│ │ ├── slim.ts # Slim CCR envelope (envelope-only column, CAS-served originals) + row-backed kstate split (#6785 steps 6/8) +│ │ ├── snap.ts # Snapshot-plane storage primitives: l1_world stage-0 ledger, fold_journal rows, egress_snapshot cache (#5874 v2) │ │ ├── sqlite-errors.ts # Corruption-shaped SQLite error classification (#5108) │ │ ├── store.ts # UnifiedStore: index.db (WAL) + content blobs, one txn per mutation │ ├── encrypt.ts # At-rest encoding: AES-256-GCM (#708) + zstd (#1080), independent @@ -228,14 +239,14 @@ billion-context/ │ │ ├── conversation-recall.ts # Fallback identity when host attribution context is missing (#4082) │ │ ├── dsh-compaction-disable.ts # Load-time overlay standing dsh basic compaction down (#3354) │ │ ├── dsh-compaction-gate.ts # Always-on compaction gate entry, independent of bili-native (#3662) -│ │ ├── fork-adopt.ts # Client half of fork adoption: prefix match + receipt (#2399) +│ │ ├── fork-adopt.ts # Client half of fork adoption: prefix match + receipt (#2399, msgid track #6262) │ │ ├── host-log.ts # Host-safe log channel: durable record, zero terminal traffic (#2827) │ │ ├── incremental-client.ts # Host-process incremental-turn client: diff, ack, resync (#3248) │ │ ├── opencode-msgids.ts # Opencode msgid harvest + wire alignment (#4491) │ │ ├── opencode-v2-msgids.ts # Opencode V2 per-turn msgid extraction + header stamp (#5825) │ │ ├── pf-channel.ts # Watermark channel client state machine: learn, capture, emit (#5403) │ │ ├── pi-inline.ts # Pi inline lane: serves INLINE_ORIGIN traffic in-process (#3278) -│ │ ├── pi-msgids.ts # Pi wire-branch lockstep walker resolving host msgids (#4491) +│ │ ├── pi-msgids.ts # Pi wire-branch lockstep walker resolving host msgids (#4491) + core-alignment proof for the full-resend envelope (#6525) │ │ ├── pi-subagent-registry.ts # Self-registration into pi-subagents required-child extensions (#2185) │ │ ├── tool-guardrails.ts # Tool-output guardrails migrated from billion-context-pi (#3066) │ │ └── dsh-native-client-react-shim.d.ts # Ambient "react" types for the dsh bundle (#1590) @@ -304,7 +315,6 @@ billion-context/ │ ├── remote-notices.ts # Remote announcement feed watcher: fail-open polling, last-known-good snapshot (#5548) │ ├── request-membership.ts # Settle-side membership probe: is this usage report ours (#4805) │ ├── resolver-wedge.ts # Persistent local DNS resolver wedge detector + alarm (#5221) -│ ├── responses-system-sections.ts # Single source of truth for the Responses-wire developer-section join: steady prepare + round-2 rebuild share one separator (#6922) │ ├── responses-tool-output.ts # Responses tool-output image parts: projection + patch helpers │ ├── rollout.ts # Feature rollout: per-install uuid, basis-point buckets, nested cohorts (#3062) │ ├── runaway-guard.ts # Intrinsic stream terminator for runaway tag/codepoint enumeration (#2346) @@ -351,7 +361,7 @@ billion-context/ │ ├── src/ # 14 modules: delegate tool/lifecycle, fleet inspector/widget, config, events │ ├── tests/ # Component unit suite (`npm --prefix pi-subagents test`, runs on TS source) │ └── package.json # Own name/version (npm `billion-context-pi-subagents`), own build chain (tsup → dist + d.ts) - ├── tests/ # 614 test files (+ hermetic e2e lanes under tests/e2e/) + ├── tests/ # 615 test files (+ hermetic e2e lanes under tests/e2e/) ├── tsup.config.ts # Build config (inlines acp-kernel + billion-context-pi-subagents; zod external) └── package.json # npm manifest (consumes kernel via `"acp-kernel": "file:./kernel"`, pi-subagents via `"billion-context-pi-subagents": "file:./pi-subagents"`; version bumped ONLY on release branches) diff --git a/reference/design-decisions.md b/reference/design-decisions.md index ecd0227890..6559c7807f 100644 --- a/reference/design-decisions.md +++ b/reference/design-decisions.md @@ -67,3 +67,21 @@ The acp-kernel source is vendored under `kernel/` and consumed via `"acp-kernel" ### KDD #19 — 公告(remote feed)治理:禁止全员广播、owner 确认、插件公告必须触发式(#6508/#6509,owner 裁定 2026-11-13) — remote announcement feed([announcements/remote.json](announcements/remote.json),#5548 三源 notice 体系的 A 源)合入 master 后 **~3 分钟内触达所有在跑实例**:它是与 merge/release 同级的安全面,不是文档文件。owner 裁定四条(2026-11-13):(1) **严格禁止对所有用户公告**——无任何受众限制、对每台机器无条件显示的记录不许出厂,每条记录必须用 `versions` / `requiresScan` / 时间窗等门限定到明确受众;(2) **任何公告(新增/修改/撤下)需 owner 明确确认**后才开 PR,Agent 只能提议内容与范围、不得自行开 PR(与发版同级);(3) **公告 PR 必须说明影响范围**:受众(哪个客户端/版本/安装态)、版本覆盖(哪些旧版本不认识新门字段而会无条件显示)、挂的时长(何时何人撤下,或 `until` 窗口);(4) **原则上不允许针对某个插件的公告**——插件冲突的暴露机制是只读冲突扫描(#1206,天然逐机触发);任何针对特定插件的提示必须**触发式而非公告式**:走扫描面(CLI WARNING / 会话冲突台账 / `/__bili/stats` / 拦截失败 web 横幅),确需持久 web 横幅时给记录带 `requiresScan` 门(#6509 新增,client+knownId 精确匹配,fails closed),使投递与文案受众一致。先例:#5984 的 dsh-live-compact 记录曾以无条件广播出厂(当时 feed 通道没有机器维度门),2026-11-13 撤下(#6508,PR #6509);`requiresScan` 机制保留在代码里,待带门版本铺开后该公告可直接加回。规则全文:AGENTS.md §4「Remote Announcements」。 + +### KDD #20 — id 存储去重与 msgid 生命周期梳理(discussion #7064,owner 裁定 2026-11-17) + +msgid 真身只有一处(`messages.msg_id`,账本主键三元组之一),其余「id 清单 JSON」全是投影,全部降级为「区间引用 + 防伪指纹」:(1) `l1_world` 身份闸门→**记账游标**——只存尾部 512 / 头部 32 有序 id 窗(`tail_json`/`head_json`,下标即 pos 不另存)+ `count` + `max_pos`,中段不可见,删 `envelope_json` 与 `envelope_checksum`;(2) `fold_journal` = `[start_seq, end_seq]` + `input_hash`(被折叠**原文行**的防伪指纹,校验时按区间重扫重推,对不上→该轮弃折叠走全量重推),删 `covered_ids`;(3) `egress_snapshot` = `seq ≤ watermark` 存活行聚合指纹,删 `ids_json`。三条 owner 裁定:**D1 永不判分叉**——l1 判定永不 reject/400/重置,判决只有 echo/extend/shrink/reanchor/issue/resync/anchor,最重动作是 reanchor(拿新数组重发游标,一条 UPDATE),结构失配当轮重锚、下一轮同数组 echo(一轮收敛从车道 reset 变成游标性质);**D2 conversation-bound 跳过 id 判定**——`conversations` 表有宿主身份行的 sid,身份由宿主会话保证,游标纯记账:stage-0 直接 mode "anchor",R3 身份故障**永不触发**(短/缺 msgid 数组降级为内容 id + 日志,不 400 不 latch);**D3 尾锚常活**——水位命中轮同样 pin + 刷新游标(`wpHit === undefined` 门删除,c3623bf1),stage-0 游标推进触发 session `revisionEpoch` bump,使每轮 snapFlush 的 epoch 证明看见推进(fork 在游标写入与 flush 之间读取否则拿到过期父 revision,子会话落到父车道)。账本内部的 msgid 用途(唯一约束去重/增量探测/编辑墓碑)、跨会话用途(session-resolve #5719、fork #6262)、钉存(pin)、水位尾锚全部不动。Full text: discussion #7064 + 本 PR(#6785)。 + +### KDD #21 — msgid pin 通道增量化:稳态只发尾部,快照轮才发全量(discussion #7064 §8,owner 裁定 2026-11-18) + +背景(owner 提问 → 裁定):水位通道(x-bili-pf-*)早已增量化(pref-tail),但 msgid **pin 通道**每轮发全量清单——opencode 的 header 16KiB/4096 预算在 ~1000 条以上的长会话里让 pin 永久失效(有用户实际报告「实际发的全是全量,最后超长度发不了」),pi/dsh 的 body 数组同样无界增长。裁定:「不管哪个客户端、以什么方式,都不应该发全量的 msgid,而是只能增量的」。 + +设计:(1) **线形**:稳态轮 = 不发 `biliMsgIds` body 字段、不发 `x-bili-msg-ids` header,只靠既有的有界通道 `x-bili-pf-tail-ids`(锚点 + 新增 cores,三态,`?` 占位)携带增量--**零新增发送代码**(同一采集缝、同一 chokepoint,b83 分析:openai wire 的 core-aligned id 列表与 wire 元素位置索引一致,tail-ids 切片恰好就是增量载荷);快照轮(启动时账本为空、压缩/fork/截断后重锚)= 发全量清单。全量形式**不删除**且**全量优先**(全量在场时增量分支不跑,bootstrap/reanchor 永远可达)。(2) **服务端 pin**(src/snapshot/stage0.ts `tryIncrementalPin`,经 prepare-openai 传 decoded tail-ids;responses 车道不接线):接受规则 = **尾部对齐重叠段逐行对账**(2026-11-19 定型,取代原「锚点+宽度」三条)——清单声明覆盖连续尾段 [T-L, T-1](T=视图长度,L=清单长度,C=账本 live 行数);与账本的重叠段 [T-L, C-1](tail 槽 [0 .. L-N-1],N=T-C)**逐行相等**,任一行漂移⇒降级;稳态形态(锚点+新增,L=N+1)= 单行重叠特例,判定与改前字节级一致;**>2048 超长会话的截断尾清单(保留窗 2048+基线偏移)同样可 pin**——重叠段最多 2048 行、证明更强,「退回全量」退化为「发你有的」,全量不再是必需(ed71f127 死循环修复:截断清单的头在中部而非前沿,旧锚点规则永远失配⇒mode 永久卡 full);L < N+1(够不到账本、无重叠可证)或 L > T ⇒ 降级;`?` 槽位保持内容派生 id;新增核(槽 [L-N .. L-1])逐槽筛查。前缀 cores **无条件从账本 live 行重新盖戳**(`snapMsgIdLiveIds`,store.ts):ingest 是 INSERT OR IGNORE on (session_id, seq),每个位置的 msg_id 只写一次、永不覆盖--稳态发 h_ 内容 id 时账本行不变但 kernel 的 byRaw ref 空间会重复铸 ref,只有每轮从账本重盖才能让 ref 空间稳定。任何一条失配 = 降级(不 pin + "full" 信号 + 日志),**永不 400/latch/reject**(漂移类故障,D1/D2 精神一致,与 R3 身份故障严格区分)--宿主压缩 ⇒ 宽度失配 ⇒ 下轮发全量 ⇒ 全量 pin 重锚 ⇒ "incremental" 信号恢复:**两轮收敛**;折叠边缘瞬态(live 集与视图短暂错位)同样被此路径天然覆盖(ledger live 集由 #3285/#4390 sweep 对齐到宿主权威集,宽度检查是两者保持同步的闸门)。(3) **信号**:响应头 `x-bili-msgid-mode: incremental|full`(与 x-bili-pf-seq 同一响应头对象发出--客户端同一轮同时学到 T 与 mode,无额外往返;语义 = 下一轮可以发什么:本轮 pin 成功 ⇒ incremental,其余(降级/bootstrap 空账本/flag off 但 tail-ids 在场)⇒ full--flag off 时发 full 是**回滚自愈**,否则回滚后会话无 pin 直到 4xx 重学)。(4) **客户端**(src/agent/pf-channel.ts,单一状态机):Tracked slot + `mode?`;`pfLearnStamp` 单一学习点(>=4xx 立即清 mode);`pfSuppressFullMsgIds(sid, wireCount)` 单一抑制点(mode==="incremental" 且 wireCount >= 最近实测 T;wire 短于 T ⇒ 宿主压缩/截断 ⇒ 立即清 mode + 本轮发全量);pi(`stampMsgIdEnvelope`)与 dsh(`maybeStampEnvelopeMsgIds`)各一行接线;**opencode phase 2 已完成**(见下:off-by-one 判断被纠正--core-aligned 列表对 hoist 后的 core 计数天然对齐,无需偏移)。(5) **灰度**:rollout flag `pluginMsgIdIncremental` 默认 0--off = 行为与线形字节级同改前(请求无新字段、响应无新头);放量 0→5→100 跟信号走。 + +兼容性矩阵:旧客户端 × 新服务端 = 全量照发(行为不变);新客户端 × 旧服务端 = 收不到信号 ⇒ 永不抑制(等同旧客户端);回滚 = full 信号自愈。测试:增量 pin 成功 / 稳态单行重叠特例 / 超长截断清单 pin(重叠段逐行)/ 重叠一行漂移降级 / 过短无重叠 / 超长于视图 / 空账本 / 全量优先 / flag off / pf 学习+抑制+缩水清除+4xx 清除。 + +明确不做:已死超长会话不回填(pin 是轮级优化,重锚后自然回到稳态);pf 容错语义不动(`?` 占位、count-only 降级保留);fork hash fallback 不动(#6262);不加新 body 字段(tail-ids 即载体);google wire 继续内容派生(#6525 v1)。 + +Full text: discussion #7064 §8;实现:本 PR(#6785)。 + +**phase 2 追加(owner 裁定 2026-11-18:「opencode 需要做、所有客户端都需要做、默认开--这是整个新存储结构重要的部分」)**:(1) **两形态规则上移到申报通道本身**(src/snapshot/stage0.ts):声明的 id 数组长度 = 视图长度 ⇒ 全量形态(既有校验 + R3 路径一字不动);长度 < 视图长度且 > 0 ⇒ **尾部形态**,走 `tryIncrementalPin`(尾部对齐重叠段逐行对账,2026-11-19 定型——取代「锚点 = 账本前沿、宽度 C+N = 视图长度」;稳态单行重叠特例不变,超长截断清单可 pin,见上)--body 短数组与 pf-tail-ids header 双在场时 **body 优先**(宿主申报优先于通道推导)。**R3 语义迁移**(KDD #20 D1 精神):申报数组内容坏只在全量形态下是身份故障(400/latch);短数组是**合法尾部形态**,坏内容 = 漂移类,只降级发「full」信号,永不 400--坏宿主在它重新收敛出的全量清单上被抓住。**空账本例外**(B1/D2 语义保留):短数组打在**空账本**上没有锚点可言 = 采集故障而非漂移,仍走旧 R3 路径(非 bound sid 400、bound sid 降级告警);账本落盘跨重启,该 400 只在真·新会话出现,客户端 4xx 清 mode 后下轮全量自愈。(2) **全部申报车道覆盖**:responses(prepare-responses 经 runStage0)与 anthropic(prepare-anthropic 内联 pin 改两形态 + `tryIncrementalPin` 复用)都接受尾部形态并随响应发 `x-bili-msgid-mode`(relay 合并 responseExtraHeaders);**opencode V1(anthropic 车道 header)+ V2(responses 车道 header)已接线**--`pfIdTailForm(sid, list)`(pf-channel 单一导出:prevIds 前缀延伸校验,shrink/rewrite/bootstrap 全部回落全量并滚动 prevIds,4xx 清 prevIds+mode)取尾部,`stampedMsgIdsHeaderFromList`(预算/join 从 messages 版抽出)盖章。**偏移不是问题**(纠正 phase 1 注释的 off-by-one 判断):opencode 的 id 列表 core-aligned(跳 leading system run),服务端宽度校验对的是 hoist 后的 core 计数,两者天然对齐;off-by-one 只存在于 T 计数的 tail-ids 通道,而 opencode 不用它。**omp 无全量申报面**(#4491 delta 车道天然增量,full-resend 轮走内容派生);claude/codex/google 无客户端申报(#6525 v1)--覆盖完整。(3) **新增核筛查**:`tryIncrementalPin` 对重叠之后的新增核槽([L-N .. L-1])逐槽筛查(空 / >512 / acp_summary_ / 控制或空白 / 重复 ⇒ 整体降级 null;「?」豁免);重叠段以账本行为准逐行对账,前缀豁免(就是账本自己的行)。(4) **默认开**:`pluginMsgIdIncremental` percent 0 → **100**(仍要求 pluginLaneServerHistory);回滚 `BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL=0`;任何失配都一轮自愈(降级 → 全量信号 → 下轮全量重锚),故 owner 拍板直接放量。 \ No newline at end of file diff --git a/scripts/gate/plugin-lanes.mjs b/scripts/gate/plugin-lanes.mjs index b62208410a..164d6ce1f8 100644 --- a/scripts/gate/plugin-lanes.mjs +++ b/scripts/gate/plugin-lanes.mjs @@ -94,7 +94,18 @@ export function gateStorageCfg() { export function gateRolloutCfg() { const v = String(process.env.GATE_SNAPSHOT_RENDER || ""); if (!v || v === "0" || v.toLowerCase() === "off") return {}; - return { rollout: { features: { snapshotRender: true } } }; + // #6785 all-switches-on verification (LOCAL-ONLY, do not commit): also force + // the msgid-plane rollouts that ship with this candidate. Config tier only — + // the env tier (BILI_ROLLOUT_*) never reaches the lane proxies (cleanEnv). + return { + rollout: { + features: { + snapshotRender: true, + pluginMsgIdIncremental: true, + identityChannelHarvest: true, + }, + }, + }; } const MODEL_MAIN = MODEL_IDS[0]; const AGENTS = (process.env.GATE_AGENTS || "dsh,pi,codex,opencode,claude") diff --git a/scripts/release-gate-local.mjs b/scripts/release-gate-local.mjs index c273fbecf9..2b0d723d2a 100644 --- a/scripts/release-gate-local.mjs +++ b/scripts/release-gate-local.mjs @@ -26,11 +26,14 @@ // Usage: // node scripts/release-gate-local.mjs [--tarball ] [--keep] // [--skip a|b|u[,…]] [--with-update] [--base-dir ] [--reuse-base-dir] +// [--snapbase-full] // // --tarball built package tarball (default: `npm pack` on the current // tree — run `npm run build` first, as the release // pre-flight already does). // --with-update also run Lane U (needs verdaccio + GATE_VERDACCIO_NPMRC). +// --snapbase-full run the #5874 snapshot plane (P0+P1+P2) fully ON for this +// invocation (all four wire cohorts); default = plane off. // --keep keep the scratch dir (default: remove on success, keep on // failure for post-mortem). // --reuse-base-dir accept a NON-EMPTY --base-dir (default: refuse — a @@ -180,6 +183,12 @@ const G = { keep: !!arg("keep"), skip: String(arg("skip", "")).split(",").filter(Boolean), withUpdate: !!arg("with-update"), + // #5874 snapshot-plane validation: turn the plane's four config-file flags + // ON for this run only (master + shadow-write lane + direct egress + // authority, all four wire cohorts). The gate's storage default is + // pure-sqlite — the only path the plane operates on. Without the flag the + // gate runs the plane fully OFF (its shipped default). + snapbaseFull: process.argv.includes("--snapbase-full"), }; const results = []; @@ -372,6 +381,20 @@ async function main() { }, // Lane U reads `latest` from here (a dsh-managed proxy must NOT self-update in place). ...(verdaccioOk ? { update: { registry: G.verdaccio } } : {}), + // --snapbase-full: run the #5874 v2 snapshot-base plane fully ON + // for this gate invocation — egress stable-prefix cache for all + // four wire cohorts. Absent flag = plane off (the shipped + // default), so a normal gate run is byte-identical to pre-plane. + ...(G.snapbaseFull + ? { + storage: { + snapBase: { + enabled: true, + cohorts: ["openai", "anthropic", "google", "responses"], + }, + }, + } + : {}), }, null, 2, diff --git a/src/agent/dsh-native.ts b/src/agent/dsh-native.ts index 8654a7b2b7..65ab3f0d36 100644 --- a/src/agent/dsh-native.ts +++ b/src/agent/dsh-native.ts @@ -69,6 +69,8 @@ import { runOutsideFetchChain } from "./fetch-chain/index.js"; import { installNativeWebSocketIntercept } from "./native-ws-intercept.js"; import { asciiHeaderValue, fetchManifest, fetchProxyVersion, fetchStatus, fetchStatusLatest, forwardTool, reportAttributionHealth, reportRuntimeInfo, waitForProxyVersion, type ManifestTool } from "./shared.js"; import { createForkAdopter, sideShapedBody } from "./fork-adopt.js"; +import { detectWireFormat } from "acp-kernel/wire"; +import { coreAlignEnvelopeIds, laneMsgIdsForOpenAiWire, laneMsgIdsForResponsesWire, laneMsgIdsForWire } from "./pi-msgids.js"; import { clearIncrementalProbe, createPrepareDirectSend, INCREMENTAL_TURN_MARKER_HEADER, INCREMENTAL_TURN_ID_HEADER, newIncrementalTurnId, noteIncrementalRejection, noteIncrementalAck, planIncrementalTurn, probeIncrementalTurns, rememberOriginalBody, sweepStale, wireMessagesLookAnthropic, type IncrementalWireMessage } from "./incremental-client.js"; import { compactionCoverageText, declareDshCompactionActive, detectCompactionOverlayGap } from "./dsh-compaction-disable.js"; import { maskUrlForDiagnostics } from "../log-mask.js"; @@ -1246,6 +1248,13 @@ type GateAttribution = interface LlmDispatchScope { sid: string; msgIds: string[] | undefined; + // #6525: the raw host message list (pi-ai shape: {id, role, content, + // toolCallId}) — the full-resend envelope stamps biliMsgIds from these + // by lockstep fingerprint walk, not from the id array alone (pairing + // must be PROVEN, never assumed positional — pi-msgids.ts discipline). + messages: unknown; + // #6880: the dispatch's declared purpose (dsh auxiliary calls; absent + // for conversation turns). purpose?: string; } const dispatchScope = new AsyncLocalStorage(); @@ -1311,6 +1320,88 @@ function runLlmDispatch(scope: LlmDispatchScope, operation: () => R): R { return runOutsideFetchChain(() => dispatchScope.run(scope, operation)); } +/** dsh's flat host message shape differs from the pi-ai form the shared + * walkers' host-side fingerprint (entryFingerprint / assistantParts in + * pi-msgids.ts) understands in exactly two places, and the mismatch made + * every tool-bearing turn bail the all-or-nothing envelope — the host ids + * then churned back to content digests on the next full-resend (#6525 B7): + * 1. assistant tool-call blocks are `type:"tool-call"` (hyphen); the + * fingerprint only recognizes `type:"toolCall"` (camel) and reads `.id`. + * 2. tool results carry role `"tool"`; the fingerprint takes the `toolres` + * branch only for role `"toolResult"` (reading `.toolCallId`), so an + * un-normalized `tool` fell into the text branch and mismatched. + * Normalize ONLY for fingerprinting: the envelope ids come from `mm.id` and + * the wire-side pairing is unchanged, so this cannot cause a mis-pin. */ +function dshFpRole(role: unknown): unknown { + return role === "tool" ? "toolResult" : role; +} +function dshFpContent(content: unknown): unknown { + if (!Array.isArray(content)) return content; + return content.map((b) => { + if (b === null || typeof b !== "object") return b; + const o = b as Record; + if (o.type === "tool-call") return { type: "toolCall", id: o.id, name: o.name }; + return b; + }); +} + +/** #6525: resolve the envelope (carrier ①) id array for a full-resend model + * body — CORE-aligned (== the post-hoist core count the server validates + * against, prepare-openai/anthropic) or undefined (degrade to + * content-derived ids, never guess). Pure — unit-testable without a plugin + * ctx. Never re-implements the host's wire transforms: the pi-ai lockstep + * walkers (the SAME functions the pi/omp lanes pin with) prove + * wire-element ⇔ host-message pairing by order-preserving fingerprints and + * bail all-or-nothing; the kernel's own converters then prove the pin is + * core-alignable via the #3642 wireIndex back-pointer — every non-leading + * wire element must own exactly ONE core (parallel tool calls / + * reasoning+text / text+image parts split one host message into N cores, + * and N cores cannot share one host id: validateMsgIds rejects duplicates). + */ +function resolveEnvelopeMsgIds(parsed: Record, hostMessages: unknown): string[] | undefined { + const fmt = detectWireFormat(parsed); + if (fmt !== "openai" && fmt !== "anthropic" && fmt !== "responses") return undefined; // google stays content-derived (#6525 v1) + const wire = fmt === "responses" ? parsed.input : parsed.messages; + if (!Array.isArray(wire)) return undefined; + if (!Array.isArray(hostMessages) || hostMessages.length === 0) return undefined; + // pi-shaped entries: dsh's flat host messages share the pi-ai message + // shape ({role, content blocks} + toolCallId on results) and entry.id IS + // the host's stable message id (#4491) — the walkers consume them through + // the same makeEntryTake contract as pi's branch entries. System-role + // entries are EXCLUDED: they ride the hoisted/folded leading system (the + // bili-ps-N slots on the openai wire; the separate `system` field on + // anthropic), which never aligns 1:1 to a non-leading wire element. + const entries: Array> = []; + for (const m of hostMessages) { + if (m === null || typeof m !== "object") return; + const mm = m as Record; + if (mm.role === "system" || mm.role === "developer") continue; + entries.push({ type: "message", id: mm.id, message: { role: dshFpRole(mm.role), content: dshFpContent(mm.content), toolCallId: mm.toolCallId } }); + } + const wireAligned = fmt === "anthropic" + ? laneMsgIdsForWire(wire, entries) + : fmt === "responses" + ? laneMsgIdsForResponsesWire(wire, entries) + : laneMsgIdsForOpenAiWire(wire, entries); + if (wireAligned === undefined) return undefined; // pairing not provable — degrade + // Shared core-alignment proof (pi-msgids.ts) — the SAME function the pi/omp + // lanes stamp with (review M1: the inline copy was a cherry-pick artifact; + // any future fix to the proof must land once, not per-host). + return coreAlignEnvelopeIds(fmt, wireAligned, wire, parsed); +} + +/** #6525: the full-resend envelope's body field name, DUPLICATED from + * src/msgid-pin.ts MSG_IDS_BODY_FIELD — host code is injected into a foreign + * runtime and must not import server modules (same structural mirroring as + * hostMsgIdsOf's reject class). Kept at module level so + * tests/msgid-rule-parity.test.ts can mechanically assert the two never + * drift (review n2). */ +export const ENVELOPE_MSG_IDS_FIELD = "biliMsgIds"; + +/** Test-only accessor (#6525 review): pins the resolver against the kernel. + * The bail matrix lives in tests/issue6525-dsh-envelope.test.ts. */ +export const _envelopeMsgIdsForTest = resolveEnvelopeMsgIds; + /** Test-only accessors (#5826 review): pins tests/msgid-rule-parity.test.ts * drives these collectors directly against the shared reject-class set. */ export const _hostMsgIdsOfForTest = hostMsgIdsOf; @@ -1674,7 +1765,14 @@ async function maybeForkAdoptBeforeSend(ctx: PluginContext, init: RequestInit | `bili-native-dsh: fork adoption for ${child.sid} skipped at the proxy-base gate — the proxy origin is not known yet, so the fork protocol cannot be reached (#4215)`); return; } - await dshForkAdopter.maybeAdopt({ base, parent: child.parent, child: child.sid, body }); + // #6262 principle 1: host msgids for the msgid identity track — CORE-aligned + // via the SAME envelope resolver the full-resend stamp uses (review m1: the + // old per-wire-message hostMsgIdsOf could never pass the adopter's + // msgIds.length === core.length lock on the openai wire — the hoisted lead + // system shifts the count — or on a multi-block anthropic turn, so the + // track silently never engaged). undefined → byte track only, exactly as + // before; the mismatch can never mis-fire (the gate is identity-strict). + await dshForkAdopter.maybeAdopt({ base, parent: child.parent, child: child.sid, body, msgIds: resolveEnvelopeMsgIds(body as Record, dispatchScope.getStore()?.messages) }); } // #1677: session id of the command's invoking agent (host-passed invocation); malformed @@ -2104,7 +2202,7 @@ export async function apply(ctx: PluginContext): Promise { // (dsh sets it on every auxiliary call; conversation leaves it // unset). Absent/odd values simply stay off the turn. const purpose = typeof options?.purpose === "string" && options.purpose.length > 0 && options.purpose.length <= 64 && /^[\x21-\x7e]+$/.test(options.purpose) ? options.purpose : undefined; - const scope: LlmDispatchScope = { sid, msgIds: hostMsgIdsOf(options?.messages), purpose }; + const scope: LlmDispatchScope = { sid, msgIds: hostMsgIdsOf(options?.messages), messages: options?.messages, purpose }; // #5403/#5418 v3: feed the per-slot id sequence into the // watermark channel (the fetch chokepoint emits count + // tail-ids; core = pf-channel.ts) — "?" slots ride id-less @@ -2680,6 +2778,7 @@ export async function apply(ctx: PluginContext): Promise { await maybeRecallBeforeSend(ctx, url, init); await maybeForkAdoptBeforeSend(ctx, init); await maybeIncrementalNative(url, init); + await maybeStampEnvelopeMsgIds(ctx, url, init); emitHostParseDiagnostic(init); }; // #3304: the dsh swap stamps the turn id atomically with the body, so the @@ -2805,6 +2904,73 @@ export async function apply(ctx: PluginContext): Promise { } }; + // #6525 envelope wiring (carrier ①'s last mile): full-resend requests + // stamp the model body's own parallel id array — ENVELOPE_MSG_IDS_FIELD + // (module level, == MSG_IDS_BODY_FIELD, src/msgid-pin.ts; the host code + // must not import server modules — see the constant's doc). + // A pinned prefix then exists on EVERY turn, not only on the lane's + // anthropic-wire turns. Resolution never re-implements the host's wire + // transforms: the pi-ai lockstep walkers (laneMsgIdsForWire / + // laneMsgIdsForOpenAiWire, the SAME functions the pi/omp lanes pin with) + // prove wire-element ⇔ host-message pairing by order-preserving + // fingerprints and bail all-or-nothing; the kernel's own converters then + // prove the pin is core-alignable — every non-leading wire element must + // own exactly ONE core via the #3642 wireIndex back-pointer (parallel + // tool calls / reasoning+text / text+image parts split one host message + // into N cores, and N cores cannot share one host id: validateMsgIds + // rejects duplicates). Any bail leaves the body byte-identical → + // content-derived ids, exactly today's behavior. maybeIncrementalNative + // runs FIRST and swaps the body — the delta payload pins msgIds itself, + // so once it fired the envelope must not touch the body. The server + // strips the field before forwarding and gates the pin on pluginMode + + // the pluginLaneServerHistory rollout (prepare-openai/anthropic). + let envelopeStampedFor: string | undefined; // one log line per session + async function maybeStampEnvelopeMsgIds(ctx: PluginContext, _url: string, init: RequestInit | undefined): Promise { + if (!register.toolsReady || init === undefined) return; + const attr = attributionOf(ctx); + if (attr.state !== "ok") return; // non-plugin traffic carries no msgid channel + // Explicit guard (review m2): once the incremental lane diverted this + // request it owns the body — the delta form pins msgIds itself and the + // envelope must not re-stamp it. The marker header is set atomically + // with the swap, so its presence IS the "lane diverted" flag (the + // walker's delta-shape bail was the only guard before; keep both). + if (new Headers(init.headers).has(INCREMENTAL_TURN_MARKER_HEADER)) return; + const body = init.body; + if (typeof body !== "string" || body.length === 0) return; // stream/Request bodies ride content-derived + const parsed = (await parsedBodyOnce(init)) as Record | undefined; + if (parsed === undefined) return; + const coreAligned = resolveEnvelopeMsgIds(parsed, dispatchScope.getStore()?.messages); + if (coreAligned === undefined) return; + // #7064 §8: steady-state suppression — the pf tail-ids channel + // (captured at the pfCaptureIds site above, same chokepoint) carries + // anchor+new and the server pins the prefix from the ledger; the + // full list rides only at bootstrap / reanchor (pfSuppressFullMsgIds). + // #6785 review fix (owner): the tail-ids CARRIER is parsed by the + // openai lane ONLY — suppressing the envelope list on the responses + // /anthropic wire would leave that turn with NO ids at all (body + // list gone, header unread) ⇒ a pin-less stretch until the session + // is back on the openai wire. Those wires parse the body list + // themselves (two-form rule: the full form is always accepted), so + // it rides there — suppression is a per-WIRE decision, not a + // per-session one. + const fmt = detectWireFormat(parsed) as string | undefined; + // #6785 review fix (owner): compare in the SAME space T was measured + // in — the server stamps x-bili-pf-seq from the WIRE element total + // (scanMessagesSpans over the messages array, leading bili-ps- system + // run included), while coreAligned is core-space (system run + // excluded). Comparing core count vs wire T misfired the shrink + // check by L every turn once L > 1 (permanent re-anchor loop, §8 + // dead for that session). The wire array length IS the space. + const wireLen = Array.isArray(parsed.messages) ? (parsed.messages as unknown[]).length : coreAligned.length; + if (fmt === "openai" && pfChannel.pfSuppressFullMsgIds(attr.sid, wireLen)) return; + const stamped = { ...parsed, [ENVELOPE_MSG_IDS_FIELD]: coreAligned }; + init.body = JSON.stringify(stamped); + if (envelopeStampedFor !== attr.sid) { + envelopeStampedFor = attr.sid; + persistClientEvent(`bili-native-dsh: msgid envelope stamped on full-resend for ${attr.sid} (${coreAligned.length} ids, ${fmt} wire) — pinned prefix live from turn one (#6525)`); + } + } + // #1268: arm the interceptor's toolsReady gate — the first model request // holds until this resolves, so it stamps into plugin mode instead of // losing the boot race and silently riding wire mode. Resolves when the diff --git a/src/agent/fork-adopt.ts b/src/agent/fork-adopt.ts index 1396f8f99d..6aa10d386c 100644 --- a/src/agent/fork-adopt.ts +++ b/src/agent/fork-adopt.ts @@ -408,6 +408,10 @@ export async function tryForkAdoption(opts: { parentConversationId: string; childConversationId: string; body: unknown; + // #6262 principle 1: host-stable message ids aligned to the wire, as the lane's + // parity-pinned collector yields them. Length-mismatched or absent → byte track + // only (exactly today's behavior). + msgIds?: string[]; fetchImpl?: typeof fetch; log?: (line: string) => void; }): Promise { @@ -422,6 +426,11 @@ export async function tryForkAdoption(opts: { } const core = chatBodyToCore(opts.body); if (core === null || core.length === 0) return { outcome: "degraded", reason: "body unmappable" }; + // #6262 principle 1: the only check this module performs on msgIds is the + // core-alignment length lock — host-pinned parents expose the host ids as + // snapshot rawIds, so a misaligned array can never match and the byte track + // carries the fork (same outcome as absent). + const msgIds = opts.msgIds !== undefined && opts.msgIds.length === core.length ? opts.msgIds : undefined; const snapshotUrl = `${opts.base}/__bili/plugin/snapshot?conversationId=${encodeURIComponent(opts.parentConversationId)}`; for (let attempt = 0; attempt < 2; attempt += 1) { let snapshot: Awaited>; @@ -439,11 +448,32 @@ export async function tryForkAdoption(opts: { return { outcome: "degraded", reason: `snapshot error ${err instanceof Error ? err.message : String(err)}` }; } if (snapshot === null) return { outcome: "degraded", reason: "snapshot malformed" }; - const branchPoint = matchForkPrefix(core, snapshot.orderedMessages); - if (branchPoint === 0) return { outcome: "degraded", reason: "no prefix match" }; + const nHash = matchForkPrefix(core, snapshot.orderedMessages); + if (nHash === 0) return { outcome: "degraded", reason: "no prefix match" }; + // #6262 principle 1 (msgid identity track, discussion #6262): when the lane + // hands us host-stable message ids, identity outranks bytes. A parent whose + // messages were host-pinned carries the host ids AS its snapshot rawIds, so + // an id match at a slot whose bytes drifted (the host re-rendered that + // message after the parent saw it) does not truncate the prefix — the fork + // claim extends to the id-stable length, driftFrom marking where the byte + // stop was. The claimed identities stay the PARENT's (copied verbatim); the + // server re-gates every claimed id against its own snapshot before trusting + // the drift region. No new id validation here: the lane's collector is the + // parity-pinned one, and a length mismatch (wire/core misalignment — part + // splits, hoisted system) simply falls back to the byte track. + let branchPoint = nHash; + let driftFrom: number | undefined; + let nId = 0; + if (msgIds !== undefined) { + while (nId < msgIds.length && nId < snapshot.orderedMessages.length && msgIds[nId] === snapshot.orderedMessages[nId]!.rawId) nId += 1; + if (nId > nHash) { + branchPoint = nId; + driftFrom = nHash; + } + } const prefix = snapshot.orderedMessages.slice(0, branchPoint); const orderHash = branchPoint === snapshot.orderedMessages.length ? snapshot.orderHash : forkOrderHashOf(prefix); - const payload = { + const payload: Record = { protocolVersion: 1, parentConversationId: opts.parentConversationId, childConversationId: opts.childConversationId, @@ -452,6 +482,14 @@ export async function tryForkAdoption(opts: { orderedMessages: prefix, idempotencyKey: `fork:${opts.childConversationId}`, }; + // #6262: send the host ids for the claimed prefix only when EVERY claimed + // cell is id-verified client-side (nId >= branchPoint) — a shorter id run + // would put unverified ids in the gate's path for no gain. driftFrom rides + // only with an extended prefix. + if (msgIds !== undefined && nId >= branchPoint) { + payload.prefixMsgIds = msgIds.slice(0, branchPoint); + if (driftFrom !== undefined) payload.driftFrom = driftFrom; + } let res: Response; try { res = await doFetch(`${opts.base}/__bili/plugin/fork`, { @@ -513,7 +551,15 @@ export async function manifestForkCapable(base: string, fetchImpl?: typeof fetch } } -export type ForkAdoptInput = { base: string; parent: string; child: string; body: unknown }; +export type ForkAdoptInput = { + base: string; + parent: string; + child: string; + body: unknown; + // #6262 principle 1: lane-collected host msgids for the msgid identity track; + // absent → byte track (today's behavior). See tryForkAdoption. + msgIds?: string[]; +}; /** Per-host fork-adoption coordinator (#2399 stage 2): absorbs the bookkeeping * every host needs (done/attempts/single-flight keyed by child id, the @@ -569,6 +615,7 @@ export function createForkAdopter(log: (line: string) => void, opts?: { maxAttem parentConversationId: input.parent, childConversationId: input.child, body: input.body, + msgIds: input.msgIds, fetchImpl: opts?.fetchImpl, log, }); diff --git a/src/agent/opencode-native.ts b/src/agent/opencode-native.ts index 6aae0006a3..79a30197f3 100644 --- a/src/agent/opencode-native.ts +++ b/src/agent/opencode-native.ts @@ -285,6 +285,63 @@ export function _resetOpencodeDeclaredWindowForTest(): void { opencodeDeclaredWindow = undefined; } +// #6525/#6539 (发) opencode V1: full-resend envelope. V1's host msgids only +// arrive via the experimental.chat.messages.transform hook (harvested into +// opencode-msgids' per-session store by harvestMsgIdsTransform); the wire +// body only exists at fetch time. So the alignment runs HERE, at the shared +// fetch chokepoint — proving the harvested turn POSITIONALLY against the +// ACTUAL wire (alignedMsgIdsForWire), never a verbatim copy of the stored +// list: two concurrent same-session LLM requests can interleave +// (A-transform, B-transform, A-fetch), and only the fetch-time re-proof +// converts that race from a mis-pin into a degrade. Anthropic-wire only +// (opencode's LLM egress); a turn pins only when every message lowers 1:1 to +// one core (pure-text user/assistant) — tool/reasoning parts or any +// count/role/text drift drop the whole list to content-derived, exactly like +// the incremental lane. The server re-proves count parity + canonical shape +// and strips the field before any upstream forward (#2821). +const V1_ENVELOPE_MSG_IDS_HEADER = "x-bili-msg-ids"; // == the server's header channel (handle.ts) +const V1_ENVELOPE_MAX_IDS = 4096; // == server parser budget (msgid-pin.ts) +const V1_ENVELOPE_MAX_BYTES = 16 * 1024; // == stampedMsgIdsHeaderFromMessages budget +export async function opencodeV1EnvelopeBeforeSend(_url: string, init: RequestInit | undefined): Promise { + try { + if (init === undefined || typeof init.body !== "string" || init.body.length === 0) return; + // The incremental lane owns converted turns (delta body + its own + // msgIds channel) — never re-stamp a diverted request (same guard as + // dsh's envelope, review m2). Never clobber an already-present header + // either (a V2 request stamps its own). + const headers = new Headers(init.headers); + if (headers.has(INCREMENTAL_TURN_MARKER_HEADER) || headers.has(V1_ENVELOPE_MSG_IDS_HEADER)) return; + const parsed: unknown = JSON.parse(init.body); + if (parsed === null || typeof parsed !== "object") return; + const b = parsed as { model?: unknown; messages?: unknown }; + if (typeof b.model !== "string" || !Array.isArray(b.messages)) return; + if (!wireMessagesLookAnthropic(b.messages)) return; // anthropic-wire only + const sid = headers.get("x-bili-plugin-conversation"); + if (typeof sid !== "string" || sid.trim().length === 0) return; + const ids = alignedMsgIdsForWire(sid.trim(), b.messages); + // #5403/#5418: feed the shared watermark channel with the SAME proven + // list — undefined (any misalignment) CLEARS the slot; the chokepoint + // emitter degrades to count-only rather than riding a stale anchor. + // (This replaces the dead host-shaped pfCaptureIds call that sat in + // harvestMsgIdsTransform and could only ever clear the slot.) + pfChannel.pfCaptureIds(sid.trim(), ids); + // #7064 §8 phase 2 (owner 2026-11-18): steady-state turns ride the + // TAIL form on this same header (anchor + new cores from the + // previous full list) — the server's two-form rule accepts it and + // pins the prefix from the ledger (anthropic lane); the mode + // signal on the response flips this switch on. Full form stays the + // bootstrap / reanchor / degraded shape. + const sendIds = pfChannel.pfIdTailForm(sid.trim(), ids) ?? ids; + if (sendIds === undefined || sendIds.length === 0 || sendIds.length > V1_ENVELOPE_MAX_IDS) return; + const joined = sendIds.join(","); + if (Buffer.byteLength(joined, "utf8") > V1_ENVELOPE_MAX_BYTES) return; + headers.set(V1_ENVELOPE_MSG_IDS_HEADER, joined); + init.headers = headers; + } catch { + // never break the host's request on adapter failure + } +} + const state: NativeInterceptState = { origin: undefined, ready: Promise.resolve(undefined) }; // #3304: the opencode swap stamps the turn id atomically with the body, so the // strict shared guard applies — mutated + unrestorable ⇒ refuse locally. @@ -292,6 +349,9 @@ state.prepareDirectSend = createPrepareDirectSend(); // #5032: arm the guard's second window source — the V1 header stamp never runs // on the no-proxy path, so without this arm the reader was always empty there. state.declaredWindowForDirect = opencodeDeclaredWindowForDirect; +// #6525/#6539: full-resend envelope for the V1 lane — armed at the shared +// chokepoint (fires for every routed V1 model request; see the function). +state.beforeSend = opencodeV1EnvelopeBeforeSend; async function bootstrap(): Promise { try { @@ -715,13 +775,12 @@ function lastSessionFromMessages(messages: unknown): string | undefined { * "context" seam registered in createOpencodeV2Setup). Never mutates the * host's messages. */ const harvestMsgIdsTransform = async (input: unknown, output: { messages?: unknown }): Promise => { + // #4491: record the latest messages.transform fire per session (the ONLY + // opencode seam carrying host ids). The wire-shape alignment + delivery + // happens at fetch time in opencodeV1EnvelopeBeforeSend — the host shape + // ({info, parts}) cannot be aligned here (that was a dead call: it fed + // host objects to a wire-shape proof that then always cleared the slot). harvestOpencodeMsgIds(lastSessionFromMessages(output?.messages), output?.messages); - // #5403/#5418 unified watermark channel: feed the ONE chokepoint emitter - // with the SAME validated 1:1 list the harvest just aligned — undefined - // (any misalignment) CLEARS the previous list; the emitter degrades to - // count-only rather than sending a stale anchor. - const sid = (input === null || typeof input !== "object" ? undefined : (input as { sessionID?: unknown }).sessionID); - if (typeof sid === "string" && sid.length > 0) pfChannel.pfCaptureIds(sid, alignedMsgIdsForWire(sid, output?.messages)); }; // #1135: dynamic origin getter instead of a captured constant — after a diff --git a/src/agent/opencode-v2-msgids.ts b/src/agent/opencode-v2-msgids.ts index db6d34ce3a..1b3c1f4c03 100644 --- a/src/agent/opencode-v2-msgids.ts +++ b/src/agent/opencode-v2-msgids.ts @@ -14,8 +14,10 @@ // `id` OPTIONAL (ai/src/schema/messages.ts:240-270), so one missing or // foreign-shaped id poisons the whole turn → no header → the session rides // exactly as before. The server re-proves everything anyway (exact count -// parity vs the wire body + canonical validation + role spot-checks); -// client-side we just keep the obvious poison classes out. +// parity vs the POST-HOIST core count + canonical validation); client-side we +// core-align — skip the leading system/developer run the kernel hoists out of +// the fold space (see opencodeV2MsgIdList) — and keep the obvious poison +// classes out. /** Hard ceiling on ids per turn — real sessions are hundreds of messages, not * thousands of KB of header; beyond this the turn degrades. */ @@ -33,17 +35,50 @@ const HOST_MSG_ID = /^msg_[A-Za-z0-9]+$/; type OpencodeV2MessageLike = { id?: unknown; role?: unknown } | null | undefined; -/** Extract the full ordered host-id list from a V2 turn's message array, or - * undefined when ANY entry cannot be proven usable (whole-turn degrade). */ +/** Extract the CORE-ALIGNED ordered host-id list from a V2 turn's message + * array, or undefined when ANY entry cannot be proven usable (whole-turn + * degrade). + * + * #6525/#6539 core alignment: the V2 lane stamps turns on the `/responses` + * and `/messages` endpoints — the server pins against the POST-hoist core + * count of the wire the turn actually rides. Both hoist rules drop exactly + * the contiguous LEADING system/developer run: + * • responsesToCore: `sysDevHead` stays true until the first non-system/ + * developer item (kernel/src/wire/responses.ts) — mid-history system/ + * developer items are position-preserved sidecar cores (#4166), and the + * head-neutral opaque items (`additional_tools` / `mcp_list_tools`) do + * not clear the head flag. If opencode ever interleaves a head-neutral + * item inside the head run, this host-side skip cannot express that + * nuance and the counts diverge — SAFELY: the opaque item carries no + * msg_ id, so the whole turn degrades (no header) and the server never + * sees a miscounted list. + * • anthropicToCore: system is a separate top-level field (never inside + * messages), so the skip is a no-op there. + * If the header carried the leading system ids, its count would exceed the + * core count and the pin would be rejected (a safe content-derived degrade, + * but the session would never actually pin). A mid-conversation system + * message is NOT part of the leading run and keeps its id. */ export function opencodeV2MsgIdList(messages: readonly OpencodeV2MessageLike[] | null | undefined): string[] | undefined { if (!Array.isArray(messages) || messages.length === 0) return undefined; if (messages.length > MAX_IDS_PER_TURN) return undefined; + // Skip the contiguous leading system/developer run (matches the kernel's + // responsesToCore sysDevHead / anthropic separate-field rules) so the id + // list aligns to the post-hoist core count. + let start = 0; + while (start < messages.length) { + const m = messages[start]; + const role = m !== null && typeof m === "object" ? (m as { role?: unknown }).role : undefined; + if (role === "system" || role === "developer") start += 1; + else break; + } const ids: string[] = []; - for (const m of messages) { + for (let i = start; i < messages.length; i++) { + const m = messages[i]; const id = m !== null && typeof m === "object" ? (m as { id?: unknown }).id : undefined; if (typeof id !== "string" || !HOST_MSG_ID.test(id)) return undefined; // poison → no stamp ids.push(id); } + if (ids.length === 0) return undefined; // only a leading system run → nothing to pin return ids; } @@ -53,7 +88,15 @@ export function stampedMsgIdsHeaderFromMessages( messages: readonly OpencodeV2MessageLike[] | null | undefined, ): string | undefined { const ids = opencodeV2MsgIdList(messages); - if (ids === undefined) return undefined; + return stampedMsgIdsHeaderFromList(ids); +} + +/** #7064 §8 phase 2: the budget/join check for an ALREADY-extracted list + * (steady-state turns send the TAIL derived from the previous full list — + * same header, same budgets). */ +export function stampedMsgIdsHeaderFromList(ids: string[] | undefined): string | undefined { + if (ids === undefined || ids.length === 0) return undefined; + if (ids.length > MAX_IDS_PER_TURN) return undefined; const joined = ids.join(","); if (Buffer.byteLength(joined, "utf8") > STAMPED_HEADER_MAX_BYTES) return undefined; return joined; diff --git a/src/agent/opencode-v2.ts b/src/agent/opencode-v2.ts index 8cb898c601..5c20399d03 100644 --- a/src/agent/opencode-v2.ts +++ b/src/agent/opencode-v2.ts @@ -50,7 +50,8 @@ import { randomUUID } from "node:crypto"; import { ACP_TOOLS_OPENAI, ABSORB_TOOL_OPENAI } from "../compress-tool.js"; -import { stampedMsgIdsHeaderFromMessages } from "./opencode-v2-msgids.js"; +import { opencodeV2MsgIdList, stampedMsgIdsHeaderFromList } from "./opencode-v2-msgids.js"; +import * as pfChannel from "./pf-channel.js"; import { asciiHeaderValue, fetchManifest, fetchProxyVersion, fetchStatus, fitNoticeDescription, forwardTool, postIdentityRegister, proxyBaseFromEnv, proxyBaseFromUrl, reportCompactionBoundary, reportRuntimeInfoOnChange, V2_SYNTHETIC_TEXT } from "./shared.js"; import { createForkAdopter } from "./fork-adopt.js"; import { harvestOpencodeMsgIds } from "./opencode-msgids.js"; @@ -341,7 +342,19 @@ export function createOpencodeV2Setup(options: OpencodeV2SetupOptions = {}): (ct // ids ride only when every one carries a usable host id, else no // header); the server re-proves exact count parity vs the wire // before trusting them, so drift degrades instead of misaligning. - const msgIdsHeader = stampedMsgIdsHeaderFromMessages(e.messages); + // #7064 §8 phase 2 (owner 2026-11-18): steady-state turns send + // only the TAIL (anchor + new cores, derived from this session's + // previous full list — pfIdTailForm). The server's two-form rule + // accepts the short list on this same header channel and pins + // the prefix from the ledger; the mode signal (x-bili-msgid-mode + // on the response, learned at the shared fetch chokepoint) is + // what flips this switch on. This releases the 16KiB header + // ceiling past ~1000 messages that made the full form die + // permanently on long sessions. + const msgIdList = opencodeV2MsgIdList(e.messages); + pfChannel.pfCaptureIds(sid, msgIdList); + const pfTail = msgIdList !== undefined && msgIdList.length > 0 ? pfChannel.pfIdTailForm(sid, msgIdList) : undefined; + const msgIdsHeader = stampedMsgIdsHeaderFromList(pfTail ?? msgIdList); if (msgIdsHeader !== undefined) headers.set("x-bili-msg-ids", msgIdsHeader); // #1699: forward the host's per-request persona so the proxy can classify // side requests by intent — opencode v2 title-gen carries no max_tokens, so @@ -406,7 +419,12 @@ export function createOpencodeV2Setup(options: OpencodeV2SetupOptions = {}): (ct } catch { forkBody = undefined; } - await forkAdopter.maybeAdopt({ base: state.proxyBase, parent: forkParent, child: forkChild, body: forkBody }); + // #6785 owner ruling (2026-11-17): the fork branch point is the + // host-msgid prefix — feed the SAME ids the turn stamp sends + // (x-bili-msg-ids above), so the child compares against exactly + // the ids the parent ledger was pinned with. Unresolvable → + // undefined → tryForkAdoption degrades (hash track deleted). + await forkAdopter.maybeAdopt({ base: state.proxyBase, parent: forkParent, child: forkChild, body: forkBody, msgIds: opencodeV2MsgIdList(e.messages) }); } // #3248 phase 2: incremental-turn divert (native adapters only — // the body swap must land after routing/stamping so the marker diff --git a/src/agent/pf-channel.ts b/src/agent/pf-channel.ts index a1f3e5b287..0bd3971977 100644 --- a/src/agent/pf-channel.ts +++ b/src/agent/pf-channel.ts @@ -33,6 +33,10 @@ const COUNT_HEADER = "x-bili-pf-count"; const TAIL_IDS_HEADER = "x-bili-pf-tail-ids"; const CONVERSATION_HEADER = "x-bili-plugin-conversation"; const SEQ_HEADER = "x-bili-pf-seq"; +// #7064 §8: the pin-mode signal (server stamps it WITH the pf-seq header; +// the literal duplicates src/msgid-pin.ts MSGID_MODE_HEADER — the agent +// bundles are import-free leaves, same rule as PF_SLOT_UNKNOWN). +const MSGID_MODE_HEADER = "x-bili-msgid-mode"; export const PF_SLOT_UNKNOWN = "?"; const MAX_ID_LEN = 512; @@ -41,19 +45,41 @@ const RESERVED_PREFIX = "acp_summary_"; // the incremental-lane adapters; keep them in sync (a bad id degrades the WHOLE // list to absent, never a partial list onto the wire). const BAD_ID_CHARS = /[\s\p{C}]/u; -const MAX_IDS = 2048; // long-conversation ceiling; the sliced head stays small +const MAX_IDS = 2048; // long-conversation ceiling; a longer captured list keeps + // ONLY its last MAX_IDS entries (baseOffset below) — the + // steady-state tail slice lives in that window, so ultra-long + // sessions still send tail-only instead of the full body const MAX_TRACKED = 1024; // sid table cap — at capacity exactly ONE oldest session is evicted (#6044) -// #7367: the tail list is [id(K-1)] + new ids since the last stamp. When the -// stamp freezes (the flag flips off, a restart, a scan miss) K stops moving and -// this slice grows with every turn — unbounded it crosses Node's 16 KB header -// limit and the proxy drops the connection before any handler runs, wedging -// the session on every retry. A tail this large would miss the fold anyway -// (the server's width check requires len === T - stampT + 1), so keep only the -// newest MAX_TAIL_IDS ids: the window origin the fastpath needs is the LAST -// element, and a delta wider than this degrades to a (correct) miss. -const MAX_TAIL_IDS = 160; +// #6785 review fix (2026-11-19): byte budget for the tail-ids HEADER value. +// Node's default maxHeaderSize covers the WHOLE header block (16 KB), and +// post-§8 this header is the sole id carrier on the openai lane — an +// oversized value (hundreds of new cores in one turn) would 431 the entire +// model request. Over budget ⇒ the tail is not emitted and the full form is +// NOT suppressed either (the pre-§8 body send pins fine; one turn of extra +// bytes beats a dead request). +const TAIL_MAX_BYTES = 8 * 1024; -type Tracked = { t?: number; ids?: string[] }; +type Tracked = { + t?: number; + ids?: string[]; + /** #6785 review fix (2026-11-19, owner): when the captured list exceeded + * MAX_IDS, ids holds only its LAST MAX_IDS entries — baseOffset is where + * that window starts in FULL-list coordinates (undefined ⇒ full list + * kept). A requested slice reaching below the window degrades to + * count-only + full body (the full form pins fine) — never a pin-less + * loop. */ + baseOffset?: number; + /** #7064 §8: what the server told us we may send next turn (learned + * from x-bili-msgid-mode; cleared on any >=4xx response — the server + * re-converges one turn later). "incremental" ⇒ steady-state turns + * suppress the full msgid list and ride only the tail-ids header. */ + mode?: "incremental" | "full"; + /** #7064 §8 phase 2 (non-watermark lanes): this session's LAST FULL id + * list as captured this turn — the tail form is derived from it + * client-side (pfIdTailForm) because these lanes get no x-bili-pf-seq + * (the watermark plane is openai-lane only). Cleared alongside mode. */ + prevIds?: string[]; +}; const tracked = new Map(); @@ -75,11 +101,26 @@ function takeSlot(sid: string): Tracked { /** Central learn point (native-intercept success sites): record THIS * request's measured element total T against its conversation id. Non-2xx * responses and malformed stamps are ignored — a broken echo costs nothing - * (the server restamps every evaluated response). */ + * (the server restamps every evaluated response). #7064 §8 also learns the + * pin mode here: a >=4xx clears it immediately (mismatch ⇒ the server told + * us to go full anyway; the next 2xx restamps it). */ export function pfLearnStamp(headersIn: Record, respStatus: number, respHeaders: Headers | null | undefined): void { - if (respStatus !== 200) return; const sid = headersIn[CONVERSATION_HEADER]; - if (sid === undefined || sid.length === 0 || respHeaders === null || respHeaders === undefined) return; + if (respStatus >= 400) { + if (sid !== undefined && sid.length > 0) { + const slot = tracked.get(sid); + if (slot !== undefined) { + if (slot.mode !== undefined) slot.mode = undefined; + if (slot.prevIds !== undefined) slot.prevIds = undefined; + } + } + return; + } + if (respStatus !== 200 || sid === undefined || sid.length === 0 || respHeaders === null || respHeaders === undefined) return; + const modeRaw = respHeaders.get(MSGID_MODE_HEADER); + if (modeRaw === "incremental" || modeRaw === "full") { + takeSlot(sid).mode = modeRaw; + } const raw = respHeaders.get(SEQ_HEADER); if (raw === null || !/^\d+$/.test(raw)) return; const t = Number(raw); @@ -100,14 +141,25 @@ export function pfLearnStamp(headersIn: Record, respStatus: numb export function pfCaptureIds(sid: string, ids: unknown): void { if (sid.length === 0) return; const slot = takeSlot(sid); - if (!Array.isArray(ids) || ids.length === 0 || ids.length > MAX_IDS) { + if (!Array.isArray(ids) || ids.length === 0) { delete slot.ids; + delete slot.baseOffset; return; } + // #6785 review fix (owner): over the ceiling, keep the TAIL, not nothing — + // the steady-state slice (t-1, t ≈ previous total) sits one core short of + // the list end and is always inside this window; only a slice reaching + // below it (stale/small stamp after a big growth) degrades to full body. + let span: unknown[] = ids; + let baseOffset: number | undefined; + if (ids.length > MAX_IDS) { + span = ids.slice(ids.length - MAX_IDS); + baseOffset = ids.length - MAX_IDS; + } const seen = new Set(); let known = 0; const out: string[] = []; - for (const v of ids) { + for (const v of span) { if ( typeof v !== "string" || v.length === 0 || v.length > MAX_ID_LEN || v.startsWith(RESERVED_PREFIX) || BAD_ID_CHARS.test(v) || seen.has(v) @@ -121,6 +173,8 @@ export function pfCaptureIds(sid: string, ids: unknown): void { } // Nothing identifiable in the whole span ⇒ count-only beats noise. slot.ids = known > 0 ? out : undefined; + if (baseOffset !== undefined) slot.baseOffset = baseOffset; + else delete slot.baseOffset; } /** Chokepoint emit (native-intercept routed-send site): the handshake headers @@ -137,14 +191,94 @@ export function pfRequestHeaders(incoming: Record): Record = { [COUNT_HEADER]: String(k) }; + const tail = tailPayloadFor(slot); + if (tail !== undefined) out[TAIL_IDS_HEADER] = tail; + return out; +} + +/** #6785 review fix: the encoded tail-ids payload this turn, or undefined + * when the carrier must stay off: not enough captured ids, a slice reaching + * BELOW the kept tail window (a >2048 capture whose stamp predates the + * window — the full form rides instead), or the joined value exceeding + * TAIL_MAX_BYTES. The SAME verdict feeds the header emit + * (pfRequestHeaders) and the full-form suppression (pfSuppressFullMsgIds): + * suppressing the body while the header carrier is absent strands the + * session pin-less FOREVER — the server sees neither form, sends no mode + * signal, and the client keeps suppressing. */ +function tailPayloadFor(slot: Tracked): string | undefined { + const t = slot.t; const ids = slot.ids; - if (ids !== undefined && ids.length >= k) { - // #7367: cap the tail to the newest MAX_TAIL_IDS ids. When K is frozen - // the full [K-1..] slice would grow past the 16 KB header limit; a - // capped tail still ends at the true window origin (ids.length-1) and - // degrades a wide delta to a conservative server-side miss. - const start = Math.max(k - 1, ids.length - MAX_TAIL_IDS); - out[TAIL_IDS_HEADER] = ids.slice(start).map(encodeURIComponent).join(","); + if (t === undefined || t < 1 || ids === undefined) return undefined; + const base = slot.baseOffset ?? 0; + if (t - 1 < base) return undefined; // slice reaches below the kept window + if (ids.length < t - base) return undefined; + const payload = ids.slice(t - 1 - base).map(encodeURIComponent).join(","); + return payload.length <= TAIL_MAX_BYTES ? payload : undefined; +} + +/** #7064 §8: whether the full msgid list may be SUPPRESSED on THIS outbound + * request (steady-state incremental — the ledger pins the prefix; only the + * bounded tail-ids header above carries the new ids). Full form stays + * authoritative at bootstrap (nothing learned) and at reanchor: a wire + * SHORTER than the last measured T means the host compacted/truncated — + * the learned mode is stale and this turn MUST send the full list (the + * server re-anchors and restamps the mode one round-trip later). + * wireCount = the id list this turn would have carried. */ +export function pfSuppressFullMsgIds(sid: string, wireCount: number): boolean { + if (sid.length === 0) return false; + const slot = tracked.get(sid); + if (slot === undefined || slot.mode !== "incremental" || slot.t === undefined) return false; + if (wireCount < slot.t) { + slot.mode = undefined; // shrink ⇒ reanchor next turn — never suppress on the shrinking turn + return false; } - return out; + // #6785 review fix: suppression is coupled to the CARRIER actually going + // out this turn (see tailPayloadFor). When the carrier cannot go out + // (stale stamp below the kept tail window, or over-byte-budget tail), + // the full form rides instead: the pre-§8 behavior, which pins fine, + // rather than a pin-less loop. + return tailPayloadFor(slot) !== undefined; +} + +/** #7064 §8 phase 2 (owner ruling 2026-11-18): the TAIL form for lanes + * WITHOUT a watermark plane (anthropic / responses egress — their servers + * never stamp x-bili-pf-seq, so pfRequestHeaders can't run): derive the + * bounded "anchor + new cores" list from this turn's FULL list and this + * session's previous full list. Returns undefined ⇒ send the full list. + * + * Rules (one-round convergence, same contract as the server's + * tryIncrementalPin): + * - mode !== "incremental" → full (nothing learned yet), + * - no previous full list → full (bootstrap), store this one, + * - list SHORTER than the previous → full (host compacted/rewrote — + * reanchor), store this one, + * - prefix MISMATCH vs the previous → full (history rewrite — reanchor), + * store this one, + * - otherwise → [previous.last, ...new cores]; + * store this turn's list. + * The server re-proves anchor + width + entries and signals "full" back + * when anything drifted — a wrong tail costs one degraded turn, never a + * mis-pin. fullIds = this turn's whole core-aligned id list (the very + * list that would have been stamped full). */ +export function pfIdTailForm(sid: string, fullIds: string[] | undefined): string[] | undefined { + if (sid.length === 0 || fullIds === undefined || fullIds.length === 0) return undefined; + const slot = tracked.get(sid); + if (slot === undefined || slot.mode !== "incremental") return undefined; + const prev = slot.prevIds; + if (prev === undefined) { + slot.prevIds = [...fullIds]; // bootstrap: this turn establishes the baseline + return undefined; + } + if (fullIds.length < prev.length) { + slot.prevIds = [...fullIds]; // shrink ⇒ full reanchor next round trip + return undefined; + } + for (let i = 0; i < prev.length; i++) { + if (fullIds[i] !== prev[i]) { + slot.prevIds = [...fullIds]; // rewrite ⇒ full reanchor next round trip + return undefined; + } + } + slot.prevIds = [...fullIds]; + return [prev[prev.length - 1], ...fullIds.slice(prev.length)]; } diff --git a/src/agent/pi-msgids.ts b/src/agent/pi-msgids.ts index a25f636cac..d2299256a4 100644 --- a/src/agent/pi-msgids.ts +++ b/src/agent/pi-msgids.ts @@ -44,6 +44,11 @@ // stripped with pi's own regex; cache_control decorations are ignored (the // wire may add them; they never change identity). +// #6525: the core-alignment proof below runs the KERNEL's own wire→core +// projection (the same one the proxy runs before pinning), so a stamp that +// passes it is guaranteed to be the exact array the server will validate. +import { anthropicToCore, openaiToCore, responsesToCore } from "acp-kernel/wire"; + type FpPart = { t: "text" | "img" | "tooluse" | "toolres" | "think" | "redact"; v: string }; /** Mirror of pi's sanitizeSurrogates (utils/sanitize-unicode.js): strips @@ -465,3 +470,165 @@ export function laneMsgIdsForOpenAiWire(wire: unknown, branch: unknown): string[ const leftover = take(); return leftover === "exhausted" ? ids : undefined; } + +/** Fingerprint of a responses-wire USER content (string | input_text/ + * output_text / input_image parts). undefined = unprovable shape; [] = the + * conversion provably drops the message. */ +function respUserParts(content: unknown): FpPart[] | undefined { + if (content === null || content === undefined) return []; + if (typeof content === "string") { + const p = textPart(content); + return p === undefined ? [] : [p]; + } + if (!Array.isArray(content)) return undefined; + const parts: FpPart[] = []; + for (const raw of content) { + if (raw === null || typeof raw !== "object") return undefined; + const part = raw as Record; + if (part.type === "input_text" || part.type === "output_text") { + const p = textPart(part.text); + if (p !== undefined) parts.push(p); + } else if (part.type === "input_image") { + if (typeof part.image_url !== "string") return undefined; + parts.push({ t: "img", v: part.image_url }); + } else { + return undefined; // unknown part type — refuse + } + } + return parts; +} + +/** Fingerprint of one OUTGOING responses-wire input item. undefined = + * unprovable (opaque / reasoning / unknown item type — the caller bails). + * system|developer message items are handled by the caller (leading run → + * hoist slots, mid-history → bail) and return undefined here. */ +function respWireFingerprint(item: unknown): FpPart[] | undefined { + if (item === null || typeof item !== "object" || Array.isArray(item)) return undefined; + const it = item as Record; + switch (it.type) { + case "message": { + const msg = it as { role?: string; content?: unknown }; + if (msg.role === "user") return respUserParts(msg.content); + if (msg.role === "assistant") { + // Text only — tool calls ride as SEPARATE function_call items, + // so an assistant message item that would split (text + a call) + // never appears here; anything but plain text is unprovable. + if (typeof msg.content === "string") { + const p = textPart(msg.content); + return p === undefined ? undefined : [p]; + } + if (!Array.isArray(msg.content)) return undefined; + const parts: FpPart[] = []; + for (const raw of msg.content) { + if (raw === null || typeof raw !== "object") return undefined; + const part = raw as Record; + if (part.type === "input_text" || part.type === "output_text") { + const p = textPart(part.text); + if (p !== undefined) parts.push(p); + } else { + return undefined; + } + } + return parts.length > 0 ? parts : undefined; + } + return undefined; // system / developer — caller's job + } + case "function_call": + if (typeof it.call_id !== "string" || it.call_id.length === 0) return undefined; + return [{ t: "tooluse", v: it.call_id }]; + case "function_call_output": + if (typeof it.call_id !== "string" || it.call_id.length === 0) return undefined; + return [{ t: "toolres", v: it.call_id }]; + default: + return undefined; // reasoning / additional_tools / item_reference / … — bail + } +} + +/** Resolve the branch-entry id for EVERY item of a responses-wire outgoing + * request — 1:1 with the INPUT-ITEM COUNT — or undefined when the alignment + * cannot be proven end-to-end (all-or-nothing). The LEADING run of + * system|developer message items takes position-pinned bili-ps-N slots (== + * the kernel's hoisted head, responsesToCore sysDevHead). ANY non-leading + * system|developer item, reasoning item, or opaque item bails: those + * placements are not statically provable here, and an absent list costs one + * count-mode turn. Every other item walks the SHARED entry iterator with + * the same order-preserving proof (never similarity, never position alone). + * All-or-nothing: undefined ⇒ the caller sends nothing. */ +export function laneMsgIdsForResponsesWire(input: unknown, branch: unknown): string[] | undefined { + if (!Array.isArray(input) || !Array.isArray(branch)) return undefined; + const items = input as Array | null>; + const isSysDev = (it: unknown): boolean => { + if (it === null || typeof it !== "object") return false; + const o = it as Record; + return o.type === "message" && (o.role === "system" || o.role === "developer"); + }; + let lead = 0; + while (lead < items.length && isSysDev(items[lead])) lead += 1; + const take = makeEntryTake(branch); + const ids: string[] = []; + for (let i = 0; i < lead; i++) ids.push(`bili-ps-${i}`); + for (let i = lead; i < items.length; i++) { + if (isSysDev(items[i])) return undefined; // mid-history system/dev — ambiguous placement + const wParts = respWireFingerprint(items[i]); + if (wParts === undefined || wParts.length === 0) return undefined; + const n = take(); + if (typeof n === "string") return undefined; + if (!partsLowerThrough(n.parts, wParts)) return undefined; + ids.push(n.id); + } + const leftover = take(); + return leftover === "exhausted" ? ids : undefined; +} + +// #6525 (send): CORE-alignment proof shared by every client's full-resend +// msgid envelope stamp (dsh #6532 inlines the same logic; pi/omp call this). +// Input = the WALKER's wire-aligned output (openai/responses wires keep their +// leading bili-ps-N hoist slots) + the wire elements + the full payload. +// Returns the core-aligned id array — == the POST-hoist core count the server +// validates — or undefined when a 1:1 host pin is not provable; the caller +// must then send NO envelope (byte-identical to before, never a misaligned +// array). +// The kernel's wireIndex is the alignment authority: a wire element that +// splits into N cores (or vanishes) has no 1:1 host id, so the whole stamp +// bails. Duplicates are refused here too (validateMsgIds would reject them +// server-side anyway — refusing locally skips the one-time warn path). +export function coreAlignEnvelopeIds( + fmt: "openai" | "anthropic" | "responses", + wireAligned: string[], + wire: unknown, + parsed: Record, +): string[] | undefined { + if (!Array.isArray(wire)) return undefined; + // The walk's contract is 1:1 with the element count — a mismatched pair + // here would silently misalign the envelope, so refuse it outright. + if (wireAligned.length !== wire.length) return undefined; + // OpenAI/Responses wire only: the leading system/developer run becomes synthetic + // bili-ps-N slots (== the kernel's hoisted prefix) — drop them; the + // envelope must be CORE-aligned (the server validates against the + // post-hoist core count). Anthropic has no hoist (system is a separate + // field), so lead stays 0 there. A bili-ps- slot anywhere PAST the lead + // run means a mid-body system the walker could not place → bail. + let lead = 0; + while (lead < wireAligned.length && (wireAligned[lead] as string).startsWith("bili-ps-")) lead += 1; + for (let i = lead; i < wireAligned.length; i++) { + if ((wireAligned[i] as string).startsWith("bili-ps-")) return undefined; + } + const coreAligned = wireAligned.slice(lead); + if (coreAligned.length === 0) return undefined; + const flat = fmt === "anthropic" + ? anthropicToCore(parsed as Parameters[0]) + : fmt === "responses" + ? responsesToCore(parsed as Parameters[0]) + : openaiToCore(parsed as Parameters[0]); + const perEl = new Map(); + for (const c of flat.msgs) { + const w = c.wireIndex; + if (w === undefined) continue; + perEl.set(w, (perEl.get(w) ?? 0) + 1); + } + for (let i = lead; i < wire.length; i++) { + if ((perEl.get(i) ?? 0) !== 1) return undefined; // element split into N cores (or vanished) — no 1:1 host pin exists + } + if (new Set(coreAligned).size !== coreAligned.length) return undefined; + return coreAligned; +} diff --git a/src/agent/pi.ts b/src/agent/pi.ts index 1c6763d6e6..25c3153013 100644 --- a/src/agent/pi.ts +++ b/src/agent/pi.ts @@ -18,7 +18,8 @@ import { isModelApiUrl, nativeInterceptInstalled } from "./native-intercept.js"; import { wirePiSubagents, piCommandsEnabled } from "./pi-subagents.js"; import { wireToolGuardrails } from "./tool-guardrails.js"; import { clearIncrementalProbe, createPrepareDirectSend, noteHostMsgIdHint, INCREMENTAL_TURN_MARKER_HEADER, incrementalEnvelopeConversationId, incrementalWireHash, noteIncrementalRejection, noteIncrementalAck, planIncrementalTurn, probeIncrementalTurns, pullIncrementalReceipt, sweepStale, wireMessagesLookAnthropic, type HostMsgIdHintEntry, type IncrementalWireMessage } from "./incremental-client.js"; -import { laneMsgIdsForOpenAiWire, laneMsgIdsForWire, lanePinnableContent } from "./pi-msgids.js"; +import { coreAlignEnvelopeIds, laneMsgIdsForOpenAiWire, laneMsgIdsForResponsesWire, laneMsgIdsForWire, lanePinnableContent } from "./pi-msgids.js"; +import { detectWireFormat } from "acp-kernel/wire"; import * as pfChannel from "./pf-channel.js"; import { validateMsgIds } from "../msgid-pin.js"; import type { DirectSendPrep } from "./native-intercept.js"; @@ -516,35 +517,11 @@ export function stampPromptCacheKey(event: unknown, ctx: Ctx, agent: string): Re noteOmpCoworkBypass(ctx.model?.id ?? (ctx.model?.baseUrl ?? "")); return undefined; } - if (!destinationRoutedThroughProxy(ctx.model?.baseUrl)) { - // #1579: native omp mode routes at the FETCH layer — the model - // baseUrl stays the real upstream, so the launcher-shaped checks - // (a /bili/-wrapped URL, the BILLION_CONTEXT_PROXY origin, the MITM - // whitelist) can never match and #1403's gate silently killed the - // identity stamp for EVERY native session. Mirror the interceptor's - // own predicate instead: when the native intercept is installed it - // WILL rewrite any isModelApiUrl request URL to /bili/. - // The interceptor judges the FULL request URL while ctx carries only - // the baseUrl prefix (".../v1" — not suffix-matched by itself), so - // probe both the bare baseUrl and its natural chat-completions - // expansion; the proxy consumes (and strips) the stamped pck exactly - // as on the launcher lanes. Anything the interceptor would NOT - // rewrite keeps #1403's guarantee — never stamp where the proxy - // cannot see it. The install flag alone is not routing evidence — it - // outlives routing (failed bootstrap, or proxy death + respawn - // give-up: onGiveUp clears BILLION_CONTEXT_PROXY but not the flag), - // and an un-routed stamped request rides verbatim into strict-schema - // upstreams (#1403's 400). Live claim = same env-keyed signal - // ownsCompaction / registerTools / runtime-info use; bootstrap - // rewrites it on (re-)establishment so degrades/recovery track the - // traffic. Round 1 inside the pre-bootstrap window misses the stamp - // and self-heals on round 2. - if (!nativeInterceptInstalled()) return undefined; - if (detectProxyBase(ctx.model?.baseUrl) === undefined) return undefined; - const base = ctx.model?.baseUrl ?? ""; - const expanded = /\/v\d+\/?$/.test(base) ? `${base.replace(/\/+$/, "")}/chat/completions` : base; - if (!isModelApiUrl(base) && !isModelApiUrl(expanded)) return undefined; - } + // #6525: the routing/visibility gate now lives in bodyProxyVisible — + // shared with the full-resend msgid envelope stamp so the two stamp + // paths (pck #268, envelope #6525) can never drift. The #1579/#1403 + // native-fetch rationale moved with it (see bodyProxyVisible). + if (!bodyProxyVisible(ctx, agent)) return undefined; const payload = (event as { payload?: unknown } | undefined)?.payload; if (payload === null || typeof payload !== "object" || Array.isArray(payload)) return undefined; const p = payload as Record; @@ -557,6 +534,116 @@ export function stampPromptCacheKey(event: unknown, ctx: Ctx, agent: string): Re return { ...p, prompt_cache_key: sid }; } +// #6525: single source of truth for "the proxy will see this request's +// BODY". Body fields the proxy consumes and strips (prompt_cache_key #268, +// the biliMsgIds envelope #6525) may be stamped ONLY when true — a +// blind-tunneled destination never sees the field, and stamping there rides +// it verbatim into strict-schema upstreams (#1403's 400). Mirrors +// stampPromptCacheKey's gate exactly so the pck and envelope stamp paths can +// never drift. Rationale (from the pck gate it replaces): native omp mode +// routes at the FETCH layer — the model baseUrl stays the real upstream, so +// the launcher-shaped checks (a /bili/-wrapped URL, the BILLION_CONTEXT_PROXY +// origin, the MITM whitelist) can never match and #1403's gate silently +// killed the identity stamp for EVERY native session. Mirror the +// interceptor's own predicate instead: when the native intercept is +// installed it WILL rewrite any isModelApiUrl request URL to +// /bili/. The interceptor judges the FULL request URL while ctx +// carries only the baseUrl prefix (".../v1" — not suffix-matched by +// itself), so probe both the bare baseUrl and its natural chat-completions +// expansion; the proxy consumes (and strips) the stamped field exactly as on +// the launcher lanes. Anything the interceptor would NOT rewrite keeps +// #1403's guarantee — never stamp where the proxy cannot see it. The install +// flag alone is not routing evidence — it outlives routing (failed +// bootstrap, or proxy death + respawn give-up: onGiveUp clears +// BILLION_CONTEXT_PROXY but not the flag), and an un-routed stamped request +// rides verbatim into strict-schema upstreams. Round 1 inside the +// pre-bootstrap window misses the stamp and self-heals on round 2. +function bodyProxyVisibleForTest(ctx: Ctx, agent: string): boolean { + return bodyProxyVisible(ctx, agent); +} +export { bodyProxyVisibleForTest as _bodyProxyVisibleForTest }; +function bodyProxyVisible(ctx: Ctx, agent: string): boolean { + const profile = hostProfileFor(agent); + // Cowork transport bypasses the fetch intercept (#2583) — same reason it + // withholds the pck stamp: the field would ride verbatim into Anthropic's + // strict schema (#1403). The one-time diagnostic note stays at the pck + // site; this is the pure predicate. + if (profile.coworkTransportPossible && ompCoworkTransportBypassesFetch(ctx.model, process.env)) return false; + if (destinationRoutedThroughProxy(ctx.model?.baseUrl)) return true; + if (!nativeInterceptInstalled()) return false; + if (detectProxyBase(ctx.model?.baseUrl) === undefined) return false; + const base = ctx.model?.baseUrl ?? ""; + const expanded = /\/v\d+\/?$/.test(base) ? `${base.replace(/\/+$/, "")}/chat/completions` : base; + return isModelApiUrl(base) || isModelApiUrl(expanded); +} + +// #6525 (send): full-resend turns ride the msgid envelope too — the +// LANE-INDEPENDENT send path. The incremental lane stamps its own msgIds on +// the delta it diverts (and early-returns before this line); this covers the +// turns the lane shape-gates away (#3433: OpenAI-family full resends) plus +// every other full-resend for both pi and omp. Core-aligned via the kernel +// wireIndex proof (coreAlignEnvelopeIds); ANY mismatch bails to no field — +// byte-identical to today, never a misaligned array. The pairing source is +// the live branch (ctx.sessionManager.getBranch) — the same source the +// incremental lane and the watermark channel use, so no new collector. +function resolveEnvelopeIdsForFullResend(parsed: Record, ctx: Ctx): string[] | undefined { + try { + const fmt = detectWireFormat(parsed); + if (fmt !== "openai" && fmt !== "anthropic" && fmt !== "responses") return undefined; // google stays content-derived (#6525 v1) + const wire = fmt === "responses" ? parsed.input : parsed.messages; + if (!Array.isArray(wire) || wire.length === 0) return undefined; + let branch: unknown; + try { + branch = ctx.sessionManager?.getBranch?.(); + } catch { + branch = undefined; + } + if (branch === null || branch === undefined) return undefined; + const wireAligned = fmt === "anthropic" + ? laneMsgIdsForWire(wire, branch) + : fmt === "responses" + ? laneMsgIdsForResponsesWire(wire, branch) + : laneMsgIdsForOpenAiWire(wire, branch); + if (wireAligned === undefined) return undefined; // pairing not provable — degrade + return coreAlignEnvelopeIds(fmt, wireAligned, wire, parsed); + } catch { + return undefined; // kernel converter threw on a malformed payload — never stamp + } +} + +// #6525 (send): stamp the core-aligned host ids onto the outgoing full-resend +// body. Returns the payload to send (base + biliMsgIds), or undefined to +// leave the request exactly as-is. base = the pck-stamped payload when the +// pck stamp fired (omp), else the original event payload (pi / omp-not- +// visible). Gated by bodyProxyVisible so a blind-tunneled request is never +// stamped (#1403). +export { resolveEnvelopeIdsForFullResend as _resolveEnvelopeIdsForFullResendForTest, stampMsgIdEnvelope as _stampMsgIdEnvelopeForTest }; +function stampMsgIdEnvelope( + event: unknown, + ctx: Ctx, + agent: string, + pck: Record | undefined, +): Record | undefined { + if (!bodyProxyVisible(ctx, agent)) return undefined; + const baseRaw = pck !== null && typeof pck === "object" && !Array.isArray(pck) ? pck : (event as { payload?: unknown } | undefined)?.payload; + if (baseRaw === null || typeof baseRaw !== "object" || Array.isArray(baseRaw)) return undefined; + const base = baseRaw as Record; + if (base.biliMsgIds !== undefined) return undefined; // already stamped — never double-stamp + const ids = resolveEnvelopeIdsForFullResend(base, ctx); + if (ids === undefined) return undefined; + // #7064 §8: steady-state turns SUPPRESS the full list — the server pins + // the prefix from the ledger and the pf tail-ids channel (same + // chokepoint, same captured ids) carries only anchor+new. Unlearned / 4xx + // relearned / shrinking ⇒ the full list rides as before. + // #6785 review fix (owner): pass the WIRE element count (same space the + // server measures x-bili-pf-seq in — see the dsh call site) instead of + // the core-aligned length; core space is L short of wire space once the + // hoisted system run is non-trivial and the shrink check misfired. + const pfSid = sessionIdOf(ctx); + if (pfSid !== undefined && pfChannel.pfSuppressFullMsgIds(pfSid, Array.isArray(base.messages) ? (base.messages as unknown[]).length : ids.length)) return undefined; + return { ...base, biliMsgIds: ids }; +} + // Values this process stamped as prompt_cache_key. The proxy strips the field // before it forwards; a request that degrades to a direct send never reaches // the proxy, so the omp direct-send guard drops the stamp itself — only a @@ -1699,7 +1786,19 @@ export function createBiliPlugin(agentOverride?: string, opts?: { retryIntervalM let proxyBase = proxyBaseForCtx(ctx); if (proxyBase === undefined) proxyBase = await awaitNativeProxyOrigin(); if (proxyBase === undefined) return; - await forkAdopter.maybeAdopt({ base: proxyBase, parent, child: sid, body: (event as { payload?: unknown } | undefined)?.payload }); + // #6262 → #6785 owner ruling (2026-11-17): the fork branch point is + // the host-msgid prefix, so the ids MUST be the same ones the + // ledger was pinned with — reuse the full-resend turn-stamp + // collector (openai/anthropic/responses; core-aligned via the + // kernel wireIndex proof). It bails to undefined when pairing is + // not provable → tryForkAdoption degrades to the hash verdict + // (#6785 review fix of a wrong comment: the hash track is KEPT + // as the floor — #6262; the id track only EXTENDS the + // hash-matched branch point, so an unresolvable lane degrades to + // the hash track, and a lane with no hash match does not fork). + const payload = (event as { payload?: unknown } | undefined)?.payload; + const forkMsgIds = payload !== null && typeof payload === "object" ? resolveEnvelopeIdsForFullResend(payload as Record, ctx) : undefined; + await forkAdopter.maybeAdopt({ base: proxyBase, parent, child: sid, body: payload, msgIds: forkMsgIds }); } pi.on("before_provider_request", async (event, ctx) => { // #5403/#5418 unified watermark channel: capture THIS request's @@ -1789,7 +1888,13 @@ export function createBiliPlugin(agentOverride?: string, opts?: { retryIntervalM logHostEvent("error", "[native-client]", `bili-plugin(${agent}): incremental turn skipped (${err instanceof Error ? err.message : String(err)}) — full resend this turn`); } } - return stampPromptCacheKey(event, ctx, agent); + // #6525 (send): full-resend turns also carry the msgid envelope + // (the lane-independent send path). stampMsgIdEnvelope folds in + // the pck stamp when it fired (base = pck-stamped payload for + // omp, original payload for pi) and returns undefined to leave + // the request byte-identical when the envelope can't be stamped. + const pck = stampPromptCacheKey(event, ctx, agent); + return stampMsgIdEnvelope(event, ctx, agent, pck) ?? pck; }); // #3248 incremental turns: the ack channel. after_provider_response // fires with status + headers BEFORE the stream is consumed, so the diff --git a/src/config.ts b/src/config.ts index 520b9c8e66..86864aaf0f 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2268,6 +2268,22 @@ type FileConfig = { * edits apply on the next restart, not mid-session. */ storage?: { unified?: string; + /** #5874 v2 snapshot-base plane (pure-incremental): two-key switch, + * config-file only, no env tier (#2030). `enabled` (default false) + * arms the egress stable-prefix cache — a D-lane render materializes + * its server-rendered stable prefix into a CAS-referenced snapshot + * row and re-serves it verbatim on warm renders (three independent + * reject gates; any miss re-derives live for that round and the cut + * self-heals on the next render — never a sticky failure). Dormant + * by default: no cache row is cut or served, zero behavior change. + * `cohorts` (missing/empty = all) restricts the plane to listed + * protocols (openai/anthropic/google/responses); other sessions + * render live unconditionally. Rollback = unset (the fold-journal + * rows keep accumulating — inert until re-armed). + * `cutEveryKUserTurns` (default 8; #6785 review) bounds the re-cut + * cadence: a VALID cached row re-materializes only once K user turns + * have landed past it (between turns the short tail derives live, as + * it always has); any INVALIDATION still re-cuts immediately. */ /** SQLite engine self-download (#2833): when unified storage is * requested but neither builtin node:sqlite nor a host-installed * better-sqlite3 exists, bili downloads a pinned N-API prebuilt at diff --git a/src/incremental-lane.ts b/src/incremental-lane.ts index 27c7da2589..e88b0181c1 100644 --- a/src/incremental-lane.ts +++ b/src/incremental-lane.ts @@ -939,7 +939,20 @@ export async function handleIncrementalTurn( // stamp the canonical ids so prepare pins exactly what the lane owns // (content-derived when the client sent none; client-pinned when it // did). The pipeline strips the field before forwarding upstream. - if (featureEnabled("pluginLaneServerHistory")) { + // + // #6680 (rolling cache marks): a body that carries client cache_control + // marks must re-enter EXACTLY as it does with the rollout off — no + // stamps. The id/seq stamps make the kernel's first-sight pass + // re-render the marked content into mNNNNN markers, which + // drops the wire-level cache_control breakpoints (and drifts the ACP + // refs) — tests/opencode-rolling-cache-marks.test.ts. coreToAnthropic + // splices entry.marks into the assembled items, so a non-empty marks + // map is the precise signal that this body carries marks. Skipping the + // stamps on mark turns is a safe degrade to the pre-GA behavior; the + // lane still owns the canonical ids (entry.msgs) and non-mark turns + // keep pinning. + const markChannelActive = entry.marks.size > 0; + if (featureEnabled("pluginLaneServerHistory") && !markChannelActive) { body[MSG_IDS_BODY_FIELD] = entry.msgs.map((m) => m.id); // #4316 PR2/3: seq stamps ride the same internal transport. The // canonical prefix carries its stamps from the unified rows diff --git a/src/knobs.ts b/src/knobs.ts index 54869f278a..acdd3f703f 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -996,6 +996,7 @@ export function fileStorageWalJournalSizeLimit(): number | undefined { return typeof v === "number" && Number.isInteger(v) && v > 0 ? v : undefined; } + /** Tiered storage-migration switch (#2671): legacy | shadow | full | sqlite. * Lives here per the env-var discipline — env tier BILI_STORAGE_UNIFIED * (1/true = shadow, full/unified = full, sqlite = pure-sqlite, 0/false/legacy/ diff --git a/src/launcher.ts b/src/launcher.ts index 0fec707d61..16ac403365 100644 --- a/src/launcher.ts +++ b/src/launcher.ts @@ -5144,7 +5144,8 @@ function runCommandCapture(command: string, args: readonly string[], timeoutMs: * reused pid would inherit the dead occupant's observation history and get * force-taken-over on first sight. Sources (all best-effort, zero new deps): * linux: /proc//stat field 22 (starttime, ticks since boot) - * win32: powershell Get-Process StartTime (~1-3s cold start, rare path only) + * win32: powershell Get-Process StartTime (1-3s warm; 30s budget — a loaded + * CI runner's cold start can exceed 10s, #6785) * other POSIX: `ps -o lstart=` * Failure → undefined → the key degrades to origin#pid (previous behavior); a * stamp that appears or disappears across observations restarts the window @@ -5163,7 +5164,7 @@ export async function processBirthStamp(pid: number): Promise)[MSG_SEQS_BODY_FIELD]; + if (!Array.isArray(raw) || raw.length === 0) return undefined; + for (const v of raw) { + if (typeof v !== "number" || !Number.isSafeInteger(v) || v < 1) return undefined; + } + return raw as number[]; +} + +/** #7064 §8: the pin-mode signal the server stamps on the response — tells + * the client what it may send NEXT turn: "incremental" (the ledger pins the + * prefix; ride only x-bili-pf-tail-ids, suppress the full list) or "full" + * (send the full id list — bootstrap / reanchor / this turn degraded). + * Rides with the existing watermark response headers (same chokepoint). */ +export const MSGID_MODE_HEADER = "x-bili-msgid-mode"; + /** Strict validation for the seq stamp array: 1-based positive integers, * exactly one per message. Non-monotonic or non-contiguous values are * ACCEPTED on purpose — the branch pointers pick arbitrary row subsets diff --git a/src/persist.ts b/src/persist.ts index e31208952c..427bcdda0f 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -2478,6 +2478,81 @@ export class SessionStore { } } + // #5874 2×2 stage 0 (brief 19544722): errors PROPAGATE (the lane maps the + // R3 reject to a hard 400 and any other fault to the legacy full path). + + /** Stage 0 (element world match) for one turn. null when the unified + * store is unavailable — the lane keeps the legacy path. */ + snapStage0(sessionId: string, incoming: string[], opts?: { resync?: boolean, seqs?: number[] | null }): import("./snapshot/plane.js").Stage0Result | null { + if (!this.replicasAvailable()) return null; + return this.liveShadow().snapStage0(sessionId, incoming, opts); + } + + /** The stage-0 resync (diverge / re-baseline): clear the l1 envelope so + * the next turn re-issues from the client's current array. */ + snapStage0Reset(sessionId: string): void { + if (!this.replicasAvailable()) return; + this.liveShadow().snapStage0Reset(sessionId); + } + + /** #7064 D2: whether this sid is CONVERSATION-BOUND — the host already + * declared a conversation identity (a conversations row for this sid), + * so the per-turn R3 identity-fault policy never fires for it: the + * session IS the identity, the stage-0 cursor is bookkeeping. */ + snapConvBound(sessionId: string): boolean { + if (!this.replicasAvailable()) return false; + return this.liveShadow().snapConvBound(sessionId); + } + + /** #7064 §8: live stamped msg_ids in seq order (see UnifiedStore.snapMsgIdLiveIds). + * [] when the unified store is unavailable — the caller treats that as an + * empty ledger (C=0) and degrades to the full-list signal. */ + snapMsgIdLiveIds(sessionId: string): string[] { + if (!this.replicasAvailable()) return []; + return this.liveShadow().snapMsgIdLiveIds(sessionId); + } + + /** #7398 seg 3: egress op-log surface (dual tables egress_state/ + * egress_flow). Null when the sqlite lane is unavailable (legacy mode, + * network backends) — callers skip recording for the turn (shadow lane, + * zero impact). */ + egressOpLogStore(): import("./storage/store.js").UnifiedStore | null { + return this.replicasAvailable() ? this.liveShadow().egressOpLogSync() : null; + } + + /** #6181 M3: whether this sid's R3 breaker latch persisted from a + * previous process (a restart must not re-arm a misbehaving host). */ + snapR3Latched(sessionId: string): boolean { + if (!this.replicasAvailable()) return false; + return this.liveShadow().snapR3Latched(sessionId); + } + + /** #6181 M3: persist the R3 breaker latch (best-effort). */ + snapR3Latch(sessionId: string): void { + if (!this.replicasAvailable()) return; + this.liveShadow().snapR3Latch(sessionId); + } + + /** #6785 review: operator escape hatch — clear the persisted R3 latch + * (see UnifiedStore.snapR3LatchClear; /__bili/snapshot/r3-unlatch). */ + snapR3LatchClear(sessRaw: string): string | null { + if (!this.replicasAvailable()) return null; + return this.liveShadow().snapR3LatchClear(sessRaw); + } + + /** #6785 review: the persisted-latch roster (see UnifiedStore.snapR3LatchedList). */ + snapR3LatchedList(): Array<{ sessRaw: string; ledgerId: string; latchedAt: number }> { + if (!this.replicasAvailable()) return []; + return this.liveShadow().snapR3LatchedList(); + } + + /** #6785 review: conversation-binding sid of a raw id (see UnifiedStore.snapSidOfRaw). */ + snapSidOfRaw(sessRaw: string): string | null { + if (!this.replicasAvailable()) return null; + return this.liveShadow().snapSidOfRaw(sessRaw); + } + + /** #3928: physical index.db shrink after deletions (`bili sessions rm`). * Throws on failure — the deletion itself is already committed, callers * report the shrink as partial. Throws in legacy-only lanes — check @@ -3644,6 +3719,61 @@ export class SessionStore { this.drainFlushWaiters(session.id, true); } + /** #5874 P2-final: the IN-REQUEST snapshot application — persist the + * session's current canonical state right now (single store transaction + * through writeNow: legacy tables AND the snapshot plane via + * snapShadowSync), so the egress stage of the SAME request reads a plane + * that already contains this turn (chain: raw → wire→core → kernel + * rewrite → snapshot write → egress rewrite → egress). Before this the + * plane write waited for the post-response debounce, so egress always + * saw the plane one tick behind and re-stitched the live tail from the + * client resend at the seam. + * + * Never throws — a failed flush leaves the round on the legacy egress + * path (the seam walk carries it) and the debounced retry is untouched. + * On success the armed debounce is settled: our live build already + * covers every mutation up to the epoch it captured, so the pending + * timer is dropped (its build would be a byte-identical no-op); a + * mutation that landed DURING the write's chain-run window re-arms via + * the caller's end-of-prepare markDirty (the returned epoch lets it + * distinguish "already saved" from "new state since"). + * @returns { ok: write landed, epoch: revisionEpoch captured BEFORE the + * write (compare against session.revisionEpoch afterwards). */ + async flushSessionNow(session: Session): Promise<{ ok: boolean; epoch: number | undefined }> { + const epoch = session.revisionEpoch; + try { + await this.writeNow(session); + } catch (err) { + this.log("warn", `[persist] ${session.id}: in-request snapshot flush failed (round stays on the legacy egress path): ${String(err).slice(0, 200)}`); + return { ok: false, epoch }; + } + if ((session.revisionEpoch ?? 0) !== (epoch ?? 0)) { + // Newer mutation landed during the chain-run window — keep the + // debounce alive for it (re-arm; the pending timer already covers + // it when the timer still exists). + this.scheduleSave(session); + } else { + const t = this.saveTimers.get(session.id); + if (t) { + clearTimeout(t); + this.saveTimers.delete(session.id); + } + this.saveBuilders.delete(session.id); + this.writerOffPending.delete(session.id); + this.runIdleSpillIfIdle(session.id); + } + return { ok: true, epoch }; + } + + /** #5874 P2-final: lane-facing wrapper — a missing or PARTIAL store (test + * stubs that answer only the accessors under test) degrades to "not + * flushed" instead of throwing into the lane's kernel-fail catch. */ + static async flushOrSkip(session: Session): Promise<{ ok: boolean; epoch: number | undefined }> { + const st = getStore(); + if (!st || typeof st.flushSessionNow !== "function") return { ok: false, epoch: session.revisionEpoch }; + return st.flushSessionNow(session); + } + /** #4739: where this id's record lives RIGHT NOW — the one build-scoped * source decision shared by both lazy-field restores after a save. * Mirrors loadSync's routing exactly: no readable unified index (never diff --git a/src/plugin.ts b/src/plugin.ts index 867038aef8..eb575f4d32 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -1274,6 +1274,15 @@ type ForkRequest = { branchPoint: { messageCount: number; orderHash: string }; orderedMessages: ForkIdentity[]; idempotencyKey: string; + // #6262 principle 1 (msgid identity track, discussion #6262): the child's + // host-stable ids for the claimed prefix. A host-pinned parent exposes the + // host ids AS its snapshot rawIds, so the gate in handlePluginFork is rawId + // equality — no new validation copy (identifier() below is the SAME check + // every rawId already passes). + prefixMsgIds?: string[]; + // Where the byte-verified prefix stopped and the identity track extended it. + // Only rides WITH prefixMsgIds; >= messageCount normalizes to "no drift". + driftFrom?: number; }; function stableJson(value: unknown): string { @@ -1737,7 +1746,19 @@ function parseForkRequest(payload: string): ForkRequest { if (!identifier(item.rawId) || !(item.ref === null || (typeof item.ref === "string" && /^m\d{5,}$/.test(item.ref))) || !hash(item.identityHash)) throw new Error("invalid raw/ref identity"); return { rawId: item.rawId, ref: item.ref, identityHash: item.identityHash }; }); - return { protocolVersion: b.protocolVersion, parentConversationId: b.parentConversationId, childConversationId: b.childConversationId, parentRevision: b.parentRevision, branchPoint: { messageCount: point.messageCount as number, orderHash: point.orderHash }, orderedMessages, idempotencyKey: b.idempotencyKey }; + // #6262 principle 1: optional msgid track. identifier() is REUSED, not copied — + // the same shape check every claimed rawId already passes above. + let prefixMsgIds: string[] | undefined; + if (b.prefixMsgIds !== undefined) { + if (!Array.isArray(b.prefixMsgIds) || b.prefixMsgIds.length !== (point.messageCount as number) || b.prefixMsgIds.some((v: unknown) => !identifier(v))) throw new Error("invalid prefixMsgIds"); + prefixMsgIds = b.prefixMsgIds as string[]; + } + let driftFrom: number | undefined; + if (b.driftFrom !== undefined) { + if (prefixMsgIds === undefined || !Number.isSafeInteger(b.driftFrom) || (b.driftFrom as number) < 0 || (b.driftFrom as number) > (point.messageCount as number)) throw new Error("invalid driftFrom"); + driftFrom = (b.driftFrom as number) < (point.messageCount as number) ? (b.driftFrom as number) : undefined; + } + return { protocolVersion: b.protocolVersion, parentConversationId: b.parentConversationId, childConversationId: b.childConversationId, parentRevision: b.parentRevision, branchPoint: { messageCount: point.messageCount as number, orderHash: point.orderHash }, orderedMessages, idempotencyKey: b.idempotencyKey, ...(prefixMsgIds !== undefined ? { prefixMsgIds } : {}), ...(driftFrom !== undefined ? { driftFrom } : {}) }; } /** [#2399 stage 2] omp/pi identity-register their session ids at session_start @@ -1797,6 +1818,23 @@ export async function handlePluginFork(payload: string, res: ServerResponse): Pr } const prefix = snapshot.orderedMessages.slice(0, request.branchPoint.messageCount); if (prefix.length !== request.branchPoint.messageCount || forkOrderHash(request.orderedMessages) !== request.branchPoint.orderHash || stableJson(prefix) !== stableJson(request.orderedMessages)) return forkReply(res, 409, { ok: false, code: "BRANCH_POINT_CONFLICT", error: "ordered prefix/hash does not match parent" }); + // #6262 principle 1 (msgid identity track): the byte check above is + // final for the prefix AS CLAIMED (the client copies the parent's + // identities verbatim, including the drift region). When the client + // additionally claims host-stable ids for the prefix and every one + // equals the parent's rawId at its slot (host-pinned parents: the + // rawId IS the host id), the region [driftFrom, messageCount) is + // identity-stable byte drift — the host re-rendered those messages + // after the parent saw them but kept their ids. Blocks and refs are + // keyed by rawId, and the child's pinned ids match the parent's, so + // everything copied below stays valid in the child: accept and mark + // the receipt. Gate failure (parent not host-pinned — its rawIds are + // content digests, never host ids — or any id/length mismatch) → the + // byte-verified fork proceeds EXACTLY as before: strict, no 409. + let forkDriftFrom: number | undefined; + if (request.prefixMsgIds !== undefined && request.prefixMsgIds.length === prefix.length && request.prefixMsgIds.every((id, k) => id === prefix[k]!.rawId)) { + forkDriftFrom = request.driftFrom; + } const rawIds = new Set(prefix.map((m) => m.rawId)); const child = createSession(request.childConversationId, { ...parent.meta, label: request.childConversationId }); const crossing = parent.state.blocks.filter((b) => b.effectiveMessageIds.some((id) => rawIds.has(id)) && !b.effectiveMessageIds.every((id) => rawIds.has(id))); @@ -1918,7 +1956,7 @@ export async function handlePluginFork(payload: string, res: ServerResponse): Pr (child.transient ??= {}).latestKnownMsgId = parent.transient.latestKnownMsgId; child.transient.needlePayloadUpper = parent.transient.needlePayloadUpper; } - const response = { ok: true, status: crossing.length > 0 ? "expanded" : "exact", protocolVersion: 2, parentConversationId: request.parentConversationId, childConversationId: request.childConversationId, sessionId: child.id, parentRevision: request.parentRevision, childRevision: forkSnapshot(child).parentRevision, branchPoint: request.branchPoint, expandedBlocks: crossing.map((b) => b.blockId), inheritedBlocks: child.state.blocks.map((b) => ({ id: b.blockId, tier: b.tier, active: b.active, expanded: b.expanded === true })), replayed: false }; + const response = { ok: true, status: forkDriftFrom !== undefined ? (crossing.length > 0 ? "expanded+drifted" : "drifted") : crossing.length > 0 ? "expanded" : "exact", protocolVersion: 2, parentConversationId: request.parentConversationId, childConversationId: request.childConversationId, sessionId: child.id, parentRevision: request.parentRevision, childRevision: forkSnapshot(child).parentRevision, branchPoint: request.branchPoint, ...(forkDriftFrom !== undefined ? { driftFrom: forkDriftFrom } : {}), expandedBlocks: crossing.map((b) => b.blockId), inheritedBlocks: child.state.blocks.map((b) => ({ id: b.blockId, tier: b.tier, active: b.active, expanded: b.expanded === true })), replayed: false }; child.metadata.publicForkReceipt = { requestHash, response }; await publishForkSession(child, parent.id); recordPluginSession(request.childConversationId, child.id); diff --git a/src/rollout.ts b/src/rollout.ts index 4ff2ace44b..684b69060b 100644 --- a/src/rollout.ts +++ b/src/rollout.ts @@ -96,8 +96,54 @@ export const ROLLOUT_FEATURES: Record = { // serve the outbound prefix from the server-owned transcript // (POST /__bili/plugin/render). The id-set algebra classifies vanishing // cohorts (compressed vs deleted) and detects generation switches. - // Default 0% — force on with `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY=1`. - pluginLaneServerHistory: { percent: 0 }, + // GA (#6525/#6539, owner ruling 2026-11-13): pinned by default. The pin is + // the intended end-state of the msgid identity infrastructure — stamped + // ids become the recorded `messages.msg_id` so the resolve/settlement/fork + // consumers read real host identity, not content digests. The render + // endpoint it co-gates is inert until a client opts in (none calls it yet). + // Roll back with `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY=0`. + pluginLaneServerHistory: { percent: 100 }, + // #7064 §8 (owner rulings 2026-11-18): steady-state turns ride the + // bounded tail (anchor + new cores) instead of re-sending the FULL msgid + // list every turn (the opencode 16KiB header budget died past ~1000 + // messages); the full form survives only at bootstrap + reanchor. ALL + // stamping clients are covered (pi / dsh openai wire via + // x-bili-pf-tail-ids; opencode V1 anthropic + V2 responses via the + // x-bili-msg-ids channel carrying the tail — server two-form rule). The + // owner set it ON by default: it is a core part of the msgid storage + // structure, and every mismatch self-heals one round trip (degrade → + // full signal → reanchor). Roll back with + // `BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL=0`. Requires + // pluginLaneServerHistory. + pluginMsgIdIncremental: { percent: 100 }, + // #5403 family, delta arm: the client sends body.messages as the TAIL + // SLICE [T-L .. T-1] (anchor included) plus x-bili-pf-count = T and + // x-bili-pf-tail-ids (width L) — the server rebuilds the view from the + // stashed prefix cores + the small re-parsed slice. Inbound bytes drop + // from O(history) to O(delta). Every unsatisfiable claim is a 400 + // re-anchor the client answers with a full-form resend (one-round + // convergence); flipping the key back to 0 400s every delta turn, which + // is the rollback path (clients converge to the full form within one + // turn). Default OFF until the shim send-sides ship — a delta body is + // unprocessable by older servers, so both ends must roll together. + // Force with `rollout.features.pluginDeltaResend: true` or + // `BILI_ROLLOUT_PLUGIN_DELTA_RESEND=1`. + pluginDeltaResend: { percent: 0 }, + // #7398/#7439: the egress plane — ONE switch for the whole new path. + // ON: every lane (OpenAI/Anthropic/Responses/Google) records its + // outbound conversation view as sparse ops (I/D/U) into the + // egress_state/egress_flow dual tables (rows SELF-CONTAINED — message + // bytes ride the row, no content CAS, ruling 7439-1), serves the + // outbound core portion back from the op log (byte-identical by + // construction), and pinned plugin lanes take server authority — the + // kernel input is replayed from the flow (rulings 7439-3/5); a bounded + // backward ladder recovers to the newest healthy log point, and ANY + // unrebuildable/corrupt state degrades the turn to legacy (never a + // partial swap, never client-as-authority). OFF = legacy chain, zero + // rows, byte-identical to pre-feature. + // Force with `rollout.features.egressPlane: true` or + // `BILI_ROLLOUT_EGRESS_PLANE=1`. + egressPlane: { percent: 0 }, // #5825 identity-channel harvest seam (registry in identity-harvest.ts): // loopback clients without an explicit body channel get stable session+ // message identity aligned from their OWN local transcript/sqlite diff --git a/src/server.ts b/src/server.ts index 3fb954b5ff..7bbd9f7ad2 100644 --- a/src/server.ts +++ b/src/server.ts @@ -53,6 +53,7 @@ import { prepareOpenai } from "./server/prepare-openai.js"; import { prepareGoogle, prepareGoogleCountTokens } from "./server/prepare-google.js"; import { prepareResponses, prepareResponsesCompact } from "./server/prepare-responses.js"; import { FORCE_TEXT_PROTOCOL } from "./server/inject.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD } from "./msgid-pin.js"; export { isChatGptCodexUpstream, isCodexResponsesLite, resolvePromptCacheKey, shouldInjectPromptCacheKey } from "./server/prepare-responses.js"; import { biliToolsDeclaredOnWire, countBiliToolUses, evaluateSelfHealRound, nudgeSuppressed, pluginLaneDegraded, pluginLaneRestore } from "./session-self-heal.js"; import { getSession, hasProcessedState, listSessions, peekSession, type PendingRetrieval, type Session, type NudgeStamps, initSessions, markDirty, flushAllSessions, acquireInFlight, releaseInFlight, totalInFlight, snapshotMessages, ensureCanonicalId, storeEffectiveConfig, foldCoverage, postRebuildAnchorTokens, setPostRebuildAnchor, tickPostRebuildAnchor, splitSessionWarnings, usageAnchorCycleStale, staleSmallAnchorFallback } from "./session.js"; @@ -165,6 +166,7 @@ import { artifactSeedHit, detectAcpArtifacts } from "./server/chain-artifacts.js import { droppedOpenaiParts } from "./wire-drop-warn.js"; import { BP_MAX, ROLLOUT_FEATURES, rolloutDecisions, rolloutInstallId, warnRolloutConfigProblems } from "./rollout.js"; import { SnapshotAnchorFallbackError, WS_ANCHORED_REQ_PROP } from "./server/snapshot-render.js"; +import { FoundationRejectError } from "./snapshot/plane.js"; export { forward } from "./server/relay.js"; // #1086/#1218: the per-session chain-verdict memory moved to plugin.ts @@ -785,6 +787,21 @@ export async function startServer(opts: ProxyOptions): Promise { // The rethrow rides ONLY the request the WS session stamped // (WS_ANCHORED_REQ_PROP); plain-HTTP lanes keep the 503 shape. if (err instanceof SnapshotAnchorFallbackError && (req as unknown as Record)[WS_ANCHORED_REQ_PROP] !== undefined) throw err; + // #6181 B1/B2: an R3 identity fault (the client declared a + // snapshot envelope that fails the identity gates) is a HARD 400 + // with the identity marker — NOT the generic 503, NOT a silent + // raw forward: the host must learn that the envelope is broken. + // Both lanes rethrow FoundationRejectError into this funnel; it + // throws before processTurn, so the session is untouched. + if (err instanceof FoundationRejectError) { + if (!res.headersSent) { + res.writeHead(400, { "content-type": "application/json", "x-bili-snapshot": "identity" }); + res.end(JSON.stringify({ error: { message: err.message, type: "snapshot_identity_fault" } })); + } else { + res.end(); + } + return; + } const msg = String(err); const e = err as { name?: string; message?: string }; // #411: a client cancel aborts the upstream fetch via @@ -3045,6 +3062,35 @@ export function scrubAnthropicPck(protocol: WireProtocol | null, bodyBuffer: Buf return Buffer.from(JSON.stringify(p), "utf8"); } +// #6539: protocol-neutral egress scrub for the bili-owned msgid transport +// fields (biliMsgIds / biliLedgerSeqs, src/msgid-pin.ts). Same contract as +// prompt_cache_key (#1403): the four fully-processed prepares strip them +// unconditionally, but every VERBATIM forward branch (side-request passthrough, +// hop-marker chain, bypass/passthrough marks, route/global passthrough #661, +// decode-fail fallback, count_tokens) bypasses the prepares — a stamped +// full-resend body would otherwise ship biliMsgIds to a real model endpoint +// (strict-schema upstreams 400 'Extra inputs are not permitted'; #661 body +// fingerprinting breaks). A body WITHOUT the fields passes back +// byte-identical, so #661's contract is untouched in the normal case. +export function scrubMsgIdFields(bodyBuffer: Buffer, log: (level: string, msg: string) => void): Buffer { + if (bodyBuffer.length === 0) return bodyBuffer; + let parsed: unknown; + try { + parsed = JSON.parse(bodyBuffer.toString("utf8")); + } catch { + return bodyBuffer; + } + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) return bodyBuffer; + const p = parsed as Record; + const hasIds = Object.prototype.hasOwnProperty.call(p, MSG_IDS_BODY_FIELD); + const hasSeqs = Object.prototype.hasOwnProperty.call(p, MSG_SEQS_BODY_FIELD); + if (!hasIds && !hasSeqs) return bodyBuffer; + if (hasIds) delete p[MSG_IDS_BODY_FIELD]; + if (hasSeqs) delete p[MSG_SEQS_BODY_FIELD]; + log("debug", "stripped biliMsgIds/biliLedgerSeqs from verbatim forward (#6539)"); + return Buffer.from(JSON.stringify(p), "utf8"); +} + // #1757: protocol-neutral companion to scrubAnthropicPck for opt-in // compat.dropFields — client-fixed fields some strict-schema gateways reject // (SenseNova Responses 400 'json: unknown field "summary"' on pi-ai's fixed @@ -3093,6 +3139,8 @@ export function prepareCountTokens( log("info", `[${sessionId}] count_tokens pruned: ${msgs.length} → ${stripped.length} msgs`); const rebuilt: AnthropicRequestBody = { ...parsed, messages: rebuiltMessages }; delete (rebuilt as Record).prompt_cache_key; + delete (rebuilt as Record)[MSG_IDS_BODY_FIELD]; // #6539: egress hygiene + delete (rebuilt as Record)[MSG_SEQS_BODY_FIELD]; return { body: JSON.stringify(rebuilt), session, @@ -3106,6 +3154,8 @@ export function prepareCountTokens( log("warn", `[${sessionId}] count_tokens prune failed, forwarding unchanged: ${String(err)}`); const fallback: AnthropicRequestBody = { ...parsed }; delete (fallback as Record).prompt_cache_key; + delete (fallback as Record)[MSG_IDS_BODY_FIELD]; // #6539: egress hygiene + delete (fallback as Record)[MSG_SEQS_BODY_FIELD]; return { body: JSON.stringify(fallback), session, diff --git a/src/server/admin.ts b/src/server/admin.ts index dd3684131b..1944b386bc 100644 --- a/src/server/admin.ts +++ b/src/server/admin.ts @@ -23,6 +23,7 @@ import { getUpdateVisibility, updateBannerPayload } from "../update-notes.js"; import { installVersionAction, triggerManualCheck, updateActionsAvailable, updateStateSnapshot } from "./update-actions.js"; import { getUnrecognizedPathStats } from "./observability.js"; import { getStore } from "../persist.js"; +import { r3FaultTracker } from "../snapshot/plane.js"; import { BodyTooLargeError, headerValue, readBody } from "../server.js"; import { dismissHostPlugin, visibleHostPluginNotices } from "../host-plugin-notice.js"; import { dismissUpgradeNotice } from "../pck-notice.js"; @@ -695,6 +696,61 @@ export async function handleAdminRoute(req: http.IncomingMessage, res: http.Serv return; } } + // #6785 review: R3 breaker latch visibility + operator escape hatch. + // A latched raw session id degrades to the legacy path PERSISTENTLY + // (it survives restarts) and nothing else clears it — the operator + // must be able to see which sids are stuck (and since when) and unstick + // one that latched in error. Loopback-gated like every /__bili/ route; + // 501 in legacy mode (same discipline as /__bili/replicas). + if (req.method === "GET" && (req.url === "/__bili/snapshot/r3-latches" || req.url?.startsWith("/__bili/snapshot/r3-latches?"))) { + const store = getStore(); + if (!store.replicasAvailable()) { + res.writeHead(501, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "requires the unified store — start this instance with BILI_STORAGE_UNIFIED=shadow|full (or storage.unified in the config file)" })); + return; + } + const latched = store.snapR3LatchedList(); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: true, count: latched.length, latched })); + return; + } + if (req.method === "POST" && req.url === "/__bili/snapshot/r3-unlatch") { + try { + const parsed = JSON.parse((await readBody(req)).toString("utf8")) as { sessRaw?: unknown }; + if (typeof parsed.sessRaw !== "string" || parsed.sessRaw.length === 0) { + res.writeHead(400, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "expected { sessRaw: string } — take the raw id from GET /__bili/snapshot/r3-latches" })); + return; + } + const store = getStore(); + if (!store.replicasAvailable()) { + res.writeHead(501, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: "requires the unified store — start this instance with BILI_STORAGE_UNIFIED=shadow|full" })); + return; + } + // Clear BOTH latch planes: the persisted row (by raw id) and the + // in-memory breaker — for every sid the row can resolve to. The + // latch row's ledger_id can still be the sessRaw placeholder + // (first fault before any stage-0 run) while the breaker is + // keyed by the real sid, so clear both resolvable keys (plus the + // raw id itself, which is the sid in the common no-conversation + // case). clearLatched on a non-latched key is a no-op. + const ledgerId = store.snapR3LatchClear(parsed.sessRaw); + const keys = new Set([parsed.sessRaw]); + if (ledgerId !== null) keys.add(ledgerId); + const convSid = store.snapSidOfRaw(parsed.sessRaw); + if (convSid !== null) keys.add(convSid); + for (const k of keys) r3FaultTracker.clearLatched(k); + log("info", `[admin] POST /__bili/snapshot/r3-unlatch sessRaw=${parsed.sessRaw.slice(0, 12)}… ledgerId=${ledgerId ?? "no-row"} in-memory-keys=[${[...keys].join(", ")}]`); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: true, persistedRowCleared: ledgerId !== null, ledgerId: ledgerId ?? undefined })); + return; + } catch (err) { + res.writeHead(err instanceof BodyTooLargeError ? 413 : 400, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: false, error: String(err) })); + return; + } + } // #2671 Phase 2: replica/checkpoint management over the unified store. // Loopback-gated like every /__bili/ route; 501 in legacy mode so the // operator knows the knob, not the URL, is what is missing. diff --git a/src/server/egress-lane-io.ts b/src/server/egress-lane-io.ts new file mode 100644 index 0000000000..f5bf072795 --- /dev/null +++ b/src/server/egress-lane-io.ts @@ -0,0 +1,44 @@ +// #7398 all-lanes (ruling 7439-4): the egress plane's record/serve seam for +// the non-OpenAI lanes. The OpenAI lane carries the full mechanism inline in +// prepare-openai.ts (authority seam + serve + record); these helpers give the +// other wires the same lane-neutral core round-trip: +// serve — rebuild the session's recorded core view from the flow tables +// (recoverEgressBase: current healthy point, or a bounded walk-back +// + truncate past a structurally corrupt point) and hand the core +// to the lane's own conversion chain; +// record — append this turn's outbound core view as sparse ops (a corrupt +// log point skips this turn; the next turn re-derives). +// The system row lives INSIDE the recorded view (type "system") but is +// excluded from the served core — each lane builds its own system block live +// (prefix-cache anchor), exactly like the OpenAI lane. +// Rollout gates (egressPlane, default 0) are owned by the +// caller. Any corrupt/unreadable log point degrades that one turn to the +// lane's legacy core — never a crash, never partial, never client-as-authority. + +import { recoverEgressBase, recordEgressTurn, type EgressOpLogStore } from "../storage/egress-record.js"; +import { rebuildCoreFromLog } from "../storage/egress-serve.js"; +import type { BiliMessage } from "acp-kernel/wire"; + +type LaneLog = (level: string, msg: string) => void; + +export function serveLaneCore(st: EgressOpLogStore, sessionId: string, log: LaneLog): BiliMessage[] | null { + try { + const base = recoverEgressBase(st, sessionId); + if (base === null || base.view.length === 0) return null; + const core = rebuildCoreFromLog(base.view); + if (core === null || core.length === 0) return null; + log("info", `[${sessionId}] egress-serve: served ${base.view.length} rows from op-log${base.degraded ? " (degraded healthy point)" : ""}`); + return core; + } catch (err) { + log("error", `[${sessionId}] egress-serve: log point corrupt or unrebuildable — legacy this turn (session-recovery marker): ${String(err)}`); + return null; + } +} + +export function recordLaneTurn(st: EgressOpLogStore, sessionId: string, conversation: BiliMessage[], systemText: string, log: LaneLog): void { + try { + recordEgressTurn({ store: st, sessionId, prev: undefined, conversation, systemText, nowMs: Date.now() }); + } catch (err) { + log("warn", `[${sessionId}] egress op-log recording skipped this turn (corrupt log point — self-heals next turn): ${String(err)}`); + } +} diff --git a/src/server/fastpath-scan.ts b/src/server/fastpath-scan.ts index ef97a84be7..506328236b 100644 --- a/src/server/fastpath-scan.ts +++ b/src/server/fastpath-scan.ts @@ -19,6 +19,31 @@ export interface RawSpan { readonly start: number; readonly end: number } +/** Rebuild the top-level request object from the span scan WITHOUT decoding + * the "messages" array (the multi-hundred-MiB part of a plugin-lane resend). + * Every other top-level field is parsed from its own tiny span. Returns null + * when the body is outside the scannable shape or any field span fails to + * parse — the caller keeps the legacy full JSON.parse (fail-soft, never + * guess — same contract as the scan itself). */ +export function synthesizeTopLevel(bodyText: string): Record | null { + const scan = scanMessagesSpans(bodyText, "all"); + if (!scan) return null; + const out: Record = {}; + for (const [key, span] of scan.fieldSpans) { + if (key === "messages") continue; + try { + out[key] = JSON.parse(bodyText.slice(span.start, span.end)); + } catch { + return null; + } + } + // Marker for the deliberately-unparsed conversation array. No legal + // chat-completions body carries a non-array "messages", so downstream + // Array.isArray checks degrade exactly like a parse failure. + out["messages"] = null; + return out; +} + export interface FastpathScanResult { /** Element count T of the top-level "messages" array. */ readonly total: number; diff --git a/src/server/fastpath.ts b/src/server/fastpath.ts index 0773e16517..66171b0e13 100644 --- a/src/server/fastpath.ts +++ b/src/server/fastpath.ts @@ -46,6 +46,36 @@ import { scanMessagesSpans, type RawSpan } from "./fastpath-scan.js"; export const PF_COUNT_HEADER = "x-bili-pf-count"; export const PF_SEQ_HEADER = "x-bili-pf-seq"; export const PF_TAIL_IDS_HEADER = "x-bili-pf-tail-ids"; +// #5403 family, delta arm: the client claims body.messages is the TAIL SLICE +// [T-L .. T-1] (anchor = element T-1 of the PREVIOUS turn included), with the +// full logical count in x-bili-pf-count and per-element ids in x-bili-pf-tail-ids +// (width exactly L). The server rebuilds the view from the seed prefix cores + +// the re-parsed slice — the same FastpathHit the full-body lane produces, so +// every downstream stage (stage0 pin, kernel, ingest, snapshot) is untouched. +// Any unsatisfiable claim is a 400 re-anchor (DeltaReanchorError): the body +// CANNOT be processed by the legacy path (the prefix bytes are absent), so the +// only honest answer is "resend this turn in full form"; the client converges +// within one round (the same one-round contract as every pf miss). +export const PF_DELTA_HEADER = "x-bili-pf-delta"; + +/** Thrown from prepare-openai when a delta request cannot be satisfied — + * handle.ts maps it to the 400 re-anchor response (no pf-seq stamp, mode full). */ +export class DeltaReanchorError extends Error { + readonly reason: string; + constructor(reason: string) { + super(`pf delta re-anchor (${reason})`); + this.name = "DeltaReanchorError"; + this.reason = reason; + } +} + +/** A satisfiable delta claim: the verified hit plus the tail-id list it rode on. */ +interface DeltaVerdict { + readonly hit: FastpathHit; + readonly receivedTailIds?: string[]; +} + +const DELTA_WINDOW_MAX = 2048; // #5418 v3: UNIDENTIFIED slot marker — an element that carries no host id // (native-compaction summary, persona sidecar, foreign shape) travels as "?". // Wire literal duplicated per #3482 (agent bundles stay import-free leaves). @@ -158,6 +188,9 @@ function parseTailIds(raw: string): string[] | null { /** Test-only accessor (#5826 review): pins tests/msgid-rule-parity.test.ts. * Production callers use parseTailIds directly. */ +/** #7064 §8: the INCREMENTAL pin (stage0) parses the same header it pins + * from — one parser, one rule, parity-pinned via _parseTailIdsForTest. */ +export function parseTailIdsForPin(raw: string): string[] | null { return parseTailIds(raw); } export function _parseTailIdsForTest(raw: string): string[] | null { return parseTailIds(raw); } function readSaved(sessionId: string): PFSavedState | null { @@ -322,6 +355,18 @@ export function evaluateWatermarkFastpath( // histories the 8-element head fragment legitimately reaches INTO the // fresh tail region, whose content differs from nothing stashed yet — // positions >= coverage are verified by the window conversion itself. + // #7064 2×2 (2026-11-20 perf, 100k-session field data): the id field + // is EXCLUDED from the comparison. The stash re-seeds from the + // post-pin cores (prepare-openai rebinds originalMessages to + // s0.pinned before finalizeWatermarkTurn), whose ids live in the + // host-msgid/ledger namespace — structurally different from the + // fresh content-hash ids openaiToCore mints, on EVERY turn of a + // pinned session. Comparing ids therefore produced a permanent + // head-prefix-drift miss: the watermark fastpath never hit exactly + // for the 2×2 traffic it was designed for, and every turn paid the + // full legacy parse + re-anchor. Content drift is still caught: the + // same-body comparison below covers role/content, shape/length via + // head-shape, and remint-with-identical-bytes is not a rewrite. const headElements = Array.isArray((headFragment as { messages?: unknown }).messages) ? ((headFragment as { messages: unknown[] }).messages.filter((m): m is { role?: unknown } => typeof m === "object" && m !== null && (m as { role?: unknown }).role !== "system")).length : -1; @@ -332,7 +377,7 @@ export function evaluateWatermarkFastpath( for (let hi = 0; hi < seedHead.length; hi++) { const a = headCores.msgs[hi]; const b = seedHead[hi]; - if (a.id !== b.id || JSON.stringify(a) !== JSON.stringify(b)) return miss("head-prefix-drift", false, tailIds); + if (JSON.stringify({ ...a, id: undefined }) !== JSON.stringify({ ...b, id: undefined })) return miss("head-prefix-drift", false, tailIds); } } const windowRaw = openaiToCore(windowFragment as OpenAIRequestBody); @@ -512,6 +557,110 @@ export function finalizeWatermarkTurn( return headers; } +/** + * #5403 family, delta arm — evaluate ONE chat-completions request whose client + * declared `x-bili-pf-delta: 1` (body.messages = tail slice, full count in + * x-bili-pf-count). Returns null when the request does not claim delta (the + * full-body lane owns it), {hit} when every gate passes, or {reanchor} when + * the claim is unsatisfiable (the caller must 400 — the legacy path cannot + * process a prefix-less body). + * + * Gates (all must hold — any doubt is a re-anchor, never a partial trust): + * rollout pluginDeltaResend on; plugin mode, not title-gen; a seed exists + * for THIS model and is not multimodal-tainted; x-bili-pf-count is an + * integer T; L = T - seed.stampT + 1 satisfies 1 ≤ L ≤ DELTA_WINDOW_MAX; + * body.messages has EXACTLY L elements; x-bili-pf-tail-ids is shape-valid + * with width exactly L and the anchor id continuous with the seed's last + * tail-id list ("?" exempt); the structural scan of the (small) body agrees + * on L; the anchor element re-parses byte-identical (id-excluded, #7064 2×2 + * namespace rule) to the stashed core at position stampT-1; no system/developer + * element rides the slice (system lives in the seed's verified head). + * + * On a hit the window is re-parsed + rekeyed exactly like the full-body hit + * (K = stampT-1: the slice INCLUDES the anchor), the anchor's strip counts + * stay with the seed ledger (it was stripped on the previous turn — same + * convention as the full path's head fragment), and view = stashed prefix + * [0..stampT-2] + rekeyed slice = the canonical T-element array. + */ +export function evaluateDeltaResend( + parsed: OpenAIRequestBody, + bodyText: string, + req: http.IncomingMessage, + session: Session, + input: { pluginMode: boolean; titleGen: boolean; model: string }, + log: (level: string, msg: string) => void, +): DeltaVerdict | { readonly reanchor: string } | null { + const rawDelta = req.headers[PF_DELTA_HEADER]; + if (rawDelta === undefined) return null; + const reanchor = (reason: string): { reanchor: string } => { + log("info", `[${session.id}] watermark delta re-anchor (${reason}) — 400, the client resends this turn in full form`); + return { reanchor: reason }; + }; + if (String(rawDelta) !== "1") return reanchor("delta-bad-value"); + if (!featureEnabled("pluginDeltaResend")) return reanchor("rollout-off"); + if (!input.pluginMode || input.titleGen) return reanchor("gate"); + const tail: unknown[] | null = Array.isArray(parsed.messages) ? parsed.messages : null; + const seed = seeds.get(session); + if (seed === undefined) return reanchor("unseeded"); + if (seed.model !== input.model) return reanchor("model"); + if (seed.multimodal) return reanchor("seeded-multimodal"); + // Stashed cores are CORE-space: openaiToCore hoists the leading + // system/developer run OUT of them, so cores.length = stampT − S (S = + // hoisted systems), never positionally indexable by wire element. + if (seed.cores.length === 0) return reanchor("seed-empty"); + const rawStamp = req.headers[PF_COUNT_HEADER]; + const T = typeof rawStamp === "string" && /^\d+$/.test(rawStamp.trim()) ? parseInt(rawStamp.trim(), 10) : NaN; + if (!Number.isInteger(T) || T < 1) return reanchor("no-count"); + // The body carries the NEW elements [stampT .. T-1] ONLY (KDD#21 wire: + // tail-ids carry anchor+new, the body carries what the client is appending). + // L = T - stampT; the anchor is element stampT-1, already in the ledger. + const L = T - seed.stampT; + if (L < 1) return reanchor("behind"); + if (L > DELTA_WINDOW_MAX) return reanchor("window-too-large"); + if (tail === null || tail.length !== L) return reanchor("body-length"); + const rawTailIds = req.headers[PF_TAIL_IDS_HEADER]; + const tailIds = rawTailIds === undefined ? null : typeof rawTailIds === "string" ? parseTailIds(rawTailIds) : null; + // tail-ids = anchor + the L new ids (the real client's steady-state form). + if (tailIds === null || tailIds.length !== L + 1) return reanchor("tail-ids-shape"); + // Anchor continuity: once the previous turn stamped a tail list, the + // list MUST open with the id it ended on — the same identity the full + // lane's overlap row verifies against the ledger. A tail-less seed (the + // first request of the session carries no pf headers by design) has no + // stamped prev: the FIRST delta turn opens unverified, and from then on + // every list is chained. An edited/deleted last message by construction + // forces the full form too (the client's declared anchor id would not + // match the ledger on the pin side). + const prevKnown = seed.lastTailIds !== undefined && seed.lastTailIds.length > 0 ? seed.lastTailIds[seed.lastTailIds.length - 1] : undefined; + if (prevKnown !== undefined && prevKnown !== PF_SLOT_UNKNOWN && tailIds[0] !== prevKnown) return reanchor("tail-ids-prev"); + const scan = scanMessagesSpans(bodyText, "all"); + if (scan === null || scan.total !== L) return reanchor("scan-broken"); + // Strip + convert the new elements exactly as the full-body lane decodes + // its window (K = stampT); the anchor's strip counts stay with the seed + // ledger (they were counted on the previous turn). Rekey against the FULL + // stashed prefix so cluster ordinals and wireIndex match the legacy + // single full-body parse. + const windowFragment: { messages: unknown } = { messages: tail }; + const windowCounts = stripRawStage(windowFragment, "openai"); + const windowRaw = openaiToCore(windowFragment as OpenAIRequestBody); + if (windowRaw.systemText.trim() !== "") return reanchor("window-system-head"); + if (windowRaw.msgs.length !== L) return reanchor("window-shape"); + const rekeyedWindow = rekeyWindowCores(windowRaw.msgs, seed.stampT, seed.cores); + const view = [...seed.cores, ...rekeyedWindow] as CoreMessage[]; + return { + hit: { + view, + windowCores: rekeyedWindow as CoreMessage[], + systemText: seed.systemText, + totalT: T, + fieldSpans: scan.fieldSpans, + windowMessages: windowFragment.messages as unknown[], + seedMultimodal: JSON.stringify(tail).includes('"image_url"'), + markerCounts: addStripCounts(seed.stripCounts, windowCounts), + }, + receivedTailIds: tailIds, + }; +} + /** Stamp-only turn (flag off, #7367): the x-bili-pf-seq heartbeat is the * client's watermark advance, NOT fastpath-only state. When * pluginFullResendWatermark is gated off the fastpath never evaluates, so diff --git a/src/server/handle.ts b/src/server/handle.ts index e2e4afb07a..c063377a47 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -77,7 +77,9 @@ import { artifactSeedHit, detectAcpArtifacts } from "./chain-artifacts.js"; import { piSubagentChannelFallback } from "./pi-subagent-channel.js"; import { FORCE_TEXT_PROTOCOL } from "./inject.js"; import { prepareAnthropic } from "./prepare-anthropic.js"; -import { MSG_IDS_BODY_FIELD, incomingMsgIdsOf, incomingMsgIdsHeader } from "../msgid-pin.js"; +import { MSG_IDS_BODY_FIELD, MSGID_MODE_HEADER, incomingMsgIdsOf, incomingMsgIdsHeader } from "../msgid-pin.js"; +import { DeltaReanchorError } from "./fastpath.js"; +import { synthesizeTopLevel } from "./fastpath-scan.js"; import { harvestIdentity } from "../identity-harvest.js"; import "../identity-sources.js"; // #5825: register every harvested lane (side effect) import { prepareOpenai } from "./prepare-openai.js"; @@ -92,7 +94,7 @@ import { handlePluginRender, PLUGIN_RENDER_PATH } from "../plugin-render.js"; import { ccrPluginWireOk, storeEffectiveCcr, storeEffectiveImageRescale, type CcrSettings } from "../store.js"; import { DEFAULT_DECIDE_MAX_TOKENS } from "../nudge-decide.js"; import { featureEnabled } from "../rollout.js"; -import { BodyTooLargeError, buildForwardTarget, declPinConflictWarned, forward, forwardUpstreamUrl, googleModelFromPath, googlePathKind, headroomEffectiveLogged, headroomFallbackLogged, headerValue, imageBillingFor, imageReserveFor, imageTokenCapFor, isCountTokensRequest, isPreflightFailFast, NON_CONVERSATION_RELAY_WARN_CAP, nonConversationRelayWarned, prepareCountTokens, preflightCompressIfNeeded, readBody, resolveKnownOutputCeiling, resolveUpstream, resignSettingsFor, selfAddressedLocalPath, scrubAnthropicPck, scrubCompatDrop, vertexAnthropicModelFromPath, warnModelNameMissIfNew, warnRouteMissIfNew, type Prepared } from "../server.js"; +import { BodyTooLargeError, buildForwardTarget, declPinConflictWarned, forward, forwardUpstreamUrl, googleModelFromPath, googlePathKind, headroomEffectiveLogged, headroomFallbackLogged, headerValue, imageBillingFor, imageReserveFor, imageTokenCapFor, isCountTokensRequest, isPreflightFailFast, NON_CONVERSATION_RELAY_WARN_CAP, nonConversationRelayWarned, prepareCountTokens, preflightCompressIfNeeded, readBody, resolveKnownOutputCeiling, resolveUpstream, resignSettingsFor, selfAddressedLocalPath, scrubAnthropicPck, scrubCompatDrop, scrubMsgIdFields, vertexAnthropicModelFromPath, warnModelNameMissIfNew, warnRouteMissIfNew, type Prepared } from "../server.js"; import { logRequestCost } from "./request-cost.js"; import { normalizeVertexModelId } from "../vertex.js"; import { CREDENTIAL_HEADER_NAMES } from "../cache-ledger.js"; @@ -615,7 +617,7 @@ async function handleRequest( return; } log("debug", `passthrough: ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")} — unattributed in-process caller (#1117), relaying verbatim`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); + await forward(req, res, opts, scrubMsgIdFields(scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); return; } // #5795: pre-ingest 429-hold fast path — while the egress governor holds @@ -685,7 +687,7 @@ async function handleRequest( if (bypass) { const reason = bypassOverride === "on" ? "forced by BILI_PLUGIN_BYPASS=on" : "legacy in-process compression"; log("debug", `bypass: ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")} — raw passthrough (${reason})`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); + await forward(req, res, opts, scrubMsgIdFields(scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); return; } const countTokens = isCountTokensRequest(req.method ?? "GET", urlPath, bodyBuffer.length > 0); @@ -696,12 +698,29 @@ async function handleRequest( // Parse body once and reuse everywhere (fixes duplicate JSON.parse). let parsed: unknown = null; let bodyText: string | undefined; + let bodyParseDeferred = false; if (protocol && bodyBuffer.length > 0) { bodyText = bodyBuffer.toString("utf8"); - try { - parsed = JSON.parse(bodyText); - } catch { - parsed = null; + // #7398 perf: a multi-MiB OpenAI body on the pf handshake is a + // plugin-lane resend whose conversation array the fastpath/delta lane + // never needs whole (byte span scan + window-only parse). The full + // JSON.parse costs 150-600ms here for nothing on a hit, so synthesize + // `parsed` from the span scan (top-level fields only; messages stays + // the null marker). prepareOpenai re-parses the full body exactly once + // if — and only if — the lanes miss. No pf header / small body / + // unscannable shape / any field-span parse failure = legacy parse, + // byte-for-byte the old behavior. + const pfProto = req.headers["x-bili-pf-count"] !== undefined || req.headers["x-bili-pf-delta"] !== undefined || req.headers["x-bili-pf-tail-ids"] !== undefined; + if (protocol === "openai" && pfProto && bodyBuffer.length >= 1048576) { + const synth = synthesizeTopLevel(bodyText); + if (synth !== null) { parsed = synth; bodyParseDeferred = true; } + } + if (!bodyParseDeferred) { + try { + parsed = JSON.parse(bodyText); + } catch { + parsed = null; + } } } // Capture the raw-wire verdict before unwrapping or preparing can mutate @@ -734,7 +753,7 @@ async function handleRequest( v1ConvId = extractConversationId(unwrapped.inner) ?? undefined; if (isV1BackgroundSideRequest(unwrapped.meta, unwrapped.inner)) { log("info", `v1internal background utility call (requestType=${typeof unwrapped.meta.requestType === "string" ? unwrapped.meta.requestType : "micro"}) — verbatim passthrough, ACP state machine bypassed (#3129)`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, v1Model ?? googleModel ?? vertexModel); + await forward(req, res, opts, scrubMsgIdFields(scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), log), null, core, config, log, route, instanceId, undefined, undefined, undefined, v1Model ?? googleModel ?? vertexModel); return; } log("debug", `v1internal envelope unwrapped (model=${v1Model ?? "?"} convId=${v1ConvId ?? "none"})`); @@ -761,7 +780,7 @@ async function handleRequest( } log("warn", `[commandcode] declared endpoint body is not a convertible streaming CLI conversation — relaying verbatim to ${maskUrlsInText(upstreamOrigin)} (#1295) — ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")}`); } - await forward(req, res, opts, bodyBuffer, null, core, config, log, route, instanceId, undefined); + await forward(req, res, opts, scrubMsgIdFields(bodyBuffer, log), null, core, config, log, route, instanceId, undefined); return; } parsed = unwrapped.body; @@ -852,7 +871,7 @@ async function handleRequest( const throttleModel = v1Model ?? googleModel ?? vertexModel ?? bodyModel; if ((protocol === "anthropic" || protocol === "openai") && parsed !== null) { const p = typeof parsed === "object" && !Array.isArray(parsed) ? parsed as Record : undefined; - if (!p || !Array.isArray(p.messages)) { + if (!p || (p.messages !== null && !Array.isArray(p.messages))) { const relayKey = `${upstreamOrigin}${urlPath}`; if (!nonConversationRelayWarned.has(relayKey)) { nonConversationRelayWarned.add(relayKey); @@ -861,7 +880,7 @@ async function handleRequest( } log("warn", `[${protocol}] body has no "messages" array — not a model conversation; relaying verbatim to ${maskUrlsInText(upstreamOrigin)} instead of rejecting (#1284) — ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")}`); } - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, throttleModel); + await forward(req, res, opts, scrubMsgIdFields(scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), log), null, core, config, log, route, instanceId, undefined, undefined, undefined, throttleModel); return; } } @@ -1461,15 +1480,14 @@ async function handleRequest( } } } - // The pin-consumers are prepare-anthropic AND prepare-openai (#5825 - // parity): both read + strip MSG_IDS_BODY_FIELD unconditionally before - // it can reach the real upstream, then pin behind the rollout gate. - // So a non-body source (header / harvest) stamps the parsed body on - // BOTH wires now — nothing extra is ever forwarded either way; - // response-ish protocols keep feeding the resolve rung below via this - // local alone. - if (stampedInboundMsgIds === undefined && inboundMsgIds !== undefined && - (protocol === "openai" || protocol === "anthropic")) { + // The pin-consumers are prepare-openai, prepare-anthropic AND + // prepare-google/prepare-responses (#6525): all four read + strip + // MSG_IDS_BODY_FIELD unconditionally before it can reach the real + // upstream, then pin behind the rollout gate. So a non-body source + // (header / harvest) stamps the parsed body on EVERY wire now — + // nothing extra is ever forwarded either way, and a length/shape + // mismatch is rejected (bail) at the pin, never leaked or mis-pinned. + if (stampedInboundMsgIds === undefined && inboundMsgIds !== undefined) { (parsed as Record)[MSG_IDS_BODY_FIELD] = [...inboundMsgIds]; } // #5719: the RECORDED side of the msgid identity rung lives in the @@ -1761,7 +1779,7 @@ async function handleRequest( // the search backend keeps working instead of hard-400ing. if (protocol === "responses" && isResponsesServerToolSingleShot(parsed)) { log("info", `single-shot server-tool Responses call — no conversation identity possible; forwarding verbatim (#6548): ${maskUrlForLog(req.url ?? "")}`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); + await forward(req, res, opts, scrubMsgIdFields(scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); return; } log("warn", `400: no stable conversation identity on ${protocol} request → ${maskUrlsInText(upstreamOrigin)}; refusing to create a content-fingerprint session (#286)`); @@ -2740,7 +2758,7 @@ async function handleRequest( autoModeForge: { variant, body: forgeBody, contentType }, }; logRequestCost(log, session.id, inboundMsgs, inboundBytes, reqT0, bodyBuffer); - await forward(req, res, opts, bodyBuffer, amPrepared, core, reqConfig, log, route, instanceId, affinity); + await forward(req, res, opts, scrubMsgIdFields(bodyBuffer, log), amPrepared, core, reqConfig, log, route, instanceId, affinity); return; } } @@ -2844,6 +2862,7 @@ async function handleRequest( } } sideBody = scrubCompatDrop(sideBody, compatDropPaths, log); + sideBody = scrubMsgIdFields(sideBody, log); // Gemini and Vertex encode stream mode in the path rather than the body. const requestStream = parsed !== null && typeof parsed === "object" && !Array.isArray(parsed) && "stream" in parsed && parsed.stream === true; @@ -3234,11 +3253,39 @@ async function handleRequest( // a real model (the loop's synthetic frames read it back). return await prepareGoogle(work as GoogleRequestBody, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, nativeWindow, googleModel ?? v1Model, googlePathKind(urlPath) === "stream-generate", visibilityMarkers, upstreamOrigin, req, decide); } + // #6181 B3: a PLUGIN-lane chained round whose input + // carries no ARRAY content (absent or empty — a + // bare-string input is admitted too: the provider + // concatenates it to its retained chain, the same + // outcome the kernel path would produce, with none of + // its side effects) is a pure chain continuation — the + // provider + // already has the full world via its retained chain. The + // kernel path for such a round is HARMFUL: it injects the + // standing dev message into an otherwise-empty input and + // (on the no-delta egress path) strips the chain id, so + // the provider answers from nothing and the host then + // chains from that BLIND response → permanent context + // loss. Forward the client's bytes verbatim instead (the + // same passthrough the /responses/compact endpoint + // already owns: no kernel, no dev injection, no plane + // side effects — the response-side ingest still updates + // the session from the answer). + const pureChainContinuation = + protocol === "responses" && + !responsesCompact && + !wsLaneEnvelope && + pluginMode && + typeof (work as ResponsesRequestBody).previous_response_id === "string" && + (work as ResponsesRequestBody).previous_response_id!.length > 0 && + (!Array.isArray((work as ResponsesRequestBody).input) || (work as ResponsesRequestBody).input!.length === 0); const preparedOut = protocol === "anthropic" ? await prepareAnthropic(work as AnthropicRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, reasoningCfg, visibilityMarkers, requestModel, decide) : protocol === "openai" - ? await prepareOpenai(work as OpenAIRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg, visibilityMarkers, route?.rewrittenUrl, decide, bodyText) - : responsesCompact + ? await prepareOpenai(work as OpenAIRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg, visibilityMarkers, route?.rewrittenUrl, decide, bodyText, bodyParseDeferred) + : pureChainContinuation + ? prepareResponsesCompact(bodyBuffer, work as ResponsesRequestBody, session, req, core, reqConfig, log) + : responsesCompact // #618: when no bili compaction item is present, // prepareResponsesCompact falls back to the raw bodyBuffer — // forward it unchanged (byte-identical passthrough). @@ -3248,6 +3295,17 @@ async function handleRequest( // Anthropic-on-Vertex wire the body carries none, and the // relay's compress-loop rebuild reads it from here. preparedOut.model = requestModel; + // #6181 B3: the compact-passthrough base hard-codes + // resetAfterSuccess — which schedules a native-compaction + // REBASE for the next Responses turn (reconcileNative- + // CompactionBoundary → resetSessionCompression). A pure + // chain-continuation round compacts nothing: honoring the + // marker would wipe this session's compression state on + // the next turn with no compaction ever having happened. + // Drop it for the B3 route (the next turn's stage-0 + // resync already converges the plane when the host + // resends a genuinely-compact world). + if (pureChainContinuation) preparedOut.resetAfterSuccess = false; return preparedOut; }; // #1295: stamp the envelope meta onto every Prepared for this @@ -3561,6 +3619,13 @@ async function handleRequest( clearSignedRefusal(guardScheme); log("warn", `[signed-passthrough] built-in-scheme request without a working re-sign arm forwarded byte-untouched, no compression (#1884 escape hatch via resign["sdk-hmac-sha256"].passthrough / BILI_RESIGN_PASSTHROUGH — the user has acknowledged this link runs uncompressed)`); forwarded = true; + // #6539: no msgid-field scrub here on purpose — this is the #1884 + // signed-passthrough escape hatch with a body-covering signature + // (sdk-hmac-sha256); mutating the body would break the signature + // the client already shipped. The byte-untouched contract wins; + // a stamped body on this lane rides as-is (bounded: the lane is an + // explicit user opt-in, and the kernel is bypassed on this lane, + // so no bili pin ever consumes the field). await forward(req, res, opts, bodyBuffer, null, core, reqConfig, log, route, instanceId, undefined, undefined, undefined, throttleModel); return; } @@ -3575,7 +3640,23 @@ async function handleRequest( if (guardScheme !== undefined && !resignArmable && !guardResign.enabled) { clearSignedRefusal(guardScheme); } - const pendingForward = await withSessionLock(session, () => runPreparedPipeline!(true)); + // #5403 family delta arm: a prefix-less tail-slice body cannot be + // processed by the legacy path — the re-anchor is a 400 the client + // answers by resending THIS turn in full form (one-round + // convergence, the same contract as every pf miss). No pf-seq + // stamp; mode relearned as "full" via the header. + let pendingForward; + try { + pendingForward = await withSessionLock(session, () => runPreparedPipeline!(true)); + } catch (err) { + if (err instanceof DeltaReanchorError && !res.destroyed && !res.writableEnded) { + log("warn", `[${session.id}] watermark delta re-anchor 400 (${err.reason}) — client resends in full form`); + res.writeHead(400, { "content-type": "application/json", [MSGID_MODE_HEADER]: "full" }); + res.end(JSON.stringify({ error: { type: "pf_delta_reanchor", message: err.reason } })); + return; + } + throw err; + } if (pendingForward) { forwarded = true; // #1195: wire clients (omp/pi on the plain proxy path) treat an @@ -3649,6 +3730,6 @@ async function handleRequest( logUnrecognizedPath(log, req.url ?? ""); } } - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, reqConfig, log, route, instanceId, undefined, undefined, undefined, throttleModel); + await forward(req, res, opts, scrubMsgIdFields(scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), log), null, core, reqConfig, log, route, instanceId, undefined, undefined, undefined, throttleModel); } } diff --git a/src/server/headers.ts b/src/server/headers.ts index dc896a81c0..c10477f892 100644 --- a/src/server/headers.ts +++ b/src/server/headers.ts @@ -37,6 +37,7 @@ export const UPSTREAM_HOP_HEADERS = new Set([ "x-bili-pf-count", "x-bili-pf-seq", "x-bili-pf-tail-ids", + "x-bili-pf-delta", // #5658: incremental-lane turn correlation id — the native plugin stamps it // on a diverted model request alongside x-bili-plugin-turn to correlate the // applied-seq ack. Pure host↔bili bookkeeping; like the pf-* handshake diff --git a/src/server/prepare-anthropic.ts b/src/server/prepare-anthropic.ts index e840165aa6..dd44b65854 100644 --- a/src/server/prepare-anthropic.ts +++ b/src/server/prepare-anthropic.ts @@ -38,6 +38,7 @@ import { reconcileSystemAnchor } from "../system-anchor.js"; import { isStrictReasoningEcho, modelIdOf } from "../strict-echo.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; import { imageRescaleMaxDim, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore } from "../persist.js"; import { applyImageRescalePass } from "../image-rescale.js"; import { isAutoModeClassifier } from "../auto-mode-forge.js"; import { countSystemAndToolsTokens, emergencyNudge, emergencyNudgeArmed, projectThinkingMass } from "./budget.js"; @@ -46,10 +47,12 @@ import { effectiveBlockOpsEnabled } from "./prepare-responses.js"; import { injectSystem, injectTool } from "./inject.js"; import { nudgeOn, resolveLaneDecisions, surfaceOffDetail } from "./lane-decisions.js"; import { featureEnabled } from "../rollout.js"; -import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, applyMsgIdPin, applyMsgSeqStamps, validateMsgIds, validateMsgSeqs } from "../msgid-pin.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, MSGID_MODE_HEADER, applyMsgIdPin, applyMsgSeqStamps, validateMsgIds, validateMsgSeqs } from "../msgid-pin.js"; +import { tryIncrementalPin } from "../snapshot/stage0.js"; import { collectRequestIds, probeRequestMembership } from "../request-membership.js"; import { alignIdSet, looksLikeSummaryCarrier } from "../msgid-algebra.js"; import { yieldToEventLoop } from "../yield-loop.js"; +import { recordLaneTurn, serveLaneCore } from "./egress-lane-io.js"; export async function prepareAnthropic( parsed: AnthropicRequestBody, @@ -221,6 +224,12 @@ export async function prepareAnthropic( const markerCounts = stripRawStage(parsed, "anthropic"); logRawStripCounts(markerCounts, log, sessionId); + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; + // #7064 §8 phase 2: pin-mode signal (set in the pin block below). + let msgidPinMode: "full" | "incremental" | null = null; + let msgidModeHeaders: Record | undefined; try { const converted = anthropicToCore(parsed); let msgs = converted.msgs; @@ -232,13 +241,34 @@ export async function prepareAnthropic( // fold coverage, and the unified-table rows the rawTail ingest writes. if (pluginMode && (incomingMsgIds !== undefined || incomingMsgSeqs !== undefined)) { if (featureEnabled("pluginLaneServerHistory")) { - const v = incomingMsgIds === undefined ? undefined : validateMsgIds(incomingMsgIds, msgs.length); + // #7064 §8 phase 2: the same TWO-FORM rule as stage 0 (the + // openai / responses lanes) — this lane pins inline (no + // runStage0), so the split is local: len === view = FULL + // list (existing path), len !== view = TAIL form (the + // shared drift-class helper: anchor + width + entry + // screen; any mismatch degrades to the full signal, never + // rejects — opencode V1's anthropic-wire header is the + // steady-state consumer). + const idsFullForm = Array.isArray(incomingMsgIds) && incomingMsgIds.length === msgs.length ? (incomingMsgIds as string[]) : undefined; + const v = idsFullForm === undefined ? undefined : validateMsgIds(idsFullForm, msgs.length); if (v !== undefined && v.ok) { msgs = applyMsgIdPin(msgs, v.ids); originalMessages = msgs; noteMsgIdObservation(session, v.ids, msgs, log, sessionId); + msgidPinMode = "incremental"; } else if (v !== undefined) { log("warn", `[${sessionId}] [msgid] pin rejected (${v.reason}) — falling back to content-derived ids (#4491)`); + msgidPinMode = "full"; + } else if (Array.isArray(incomingMsgIds) && incomingMsgIds.length > 0 && incomingMsgIds.length !== msgs.length) { + const ip = tryIncrementalPin(sessionId, "anthropic", msgs, incomingMsgIds as string[], log); + if (ip !== null) { + msgs = ip; + originalMessages = ip; + noteMsgIdObservation(session, ip.map((m) => m.id), msgs, log, sessionId); + msgidPinMode = "incremental"; + } else { + msgidPinMode = "full"; + } } // #4316 PR2/3: re-attach the ledger seq stamps the lane sent — // the same parallel-array lifecycle as the ids (strip → @@ -255,6 +285,13 @@ export async function prepareAnthropic( log("info", `[${sessionId}] [msgid] biliMsgIds present but pluginLaneServerHistory rollout is off for this install — field stripped, ids not pinned (#4491)`); } } + // #6785 review fix: "incremental" is gated on the flag (the upgrade + // never travels while it is off); the "full" DOWNGRADE travels even + // when the flag is off — the rollback self-heal (an already upgraded + // client must be told to send the full form again). + if (msgidPinMode === "full" || (featureEnabled("pluginMsgIdIncremental") && msgidPinMode !== null)) { + msgidModeHeaders = { [MSGID_MODE_HEADER]: msgidPinMode }; + } // #3756 snapshot render: outbound prefix from the unified transcript // (fold-time ids, immune to identity re-mints); live tail keeps the // client's bytes. No-op when the lane is inactive/degraded; the @@ -384,6 +421,14 @@ export async function prepareAnthropic( const nudgeHeldAfterReject = session.transient?.nudgeHeldAfterReject === true; if (nudgeHeldAfterReject) restoreNudgeStamps(session, nudgeStampsBefore); adoptContentStore(session, turn.contentStore); + // In-request ledger flush: the transcript up to THIS user message is + // durable before the model call (#3397 durability-before-render) and + // the stage-0 envelope / M2 evidence read a current ledger. + // snapshotMessages first: the persist build reads session.lastMessages + // — and it stays the RAW client resend (never the D-lane view: its + // synthetic acp_summary anchor is not a transcript message). + snapshotMessages(session, originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -516,8 +561,21 @@ export async function prepareAnthropic( } // #3400: egress image rescale — outbound copies only; the originals // stay in session state/persistence untouched. - const egressMessages = await applyImageRescalePass(processedMessages as BiliMessage[], { session, sessionId, log, maxDim: imageRescaleMaxDim(), toolArmed: imageRescaleToolArmed(session) }); + let egressMessages = await applyImageRescalePass(processedMessages as BiliMessage[], { session, sessionId, log, maxDim: imageRescaleMaxDim(), toolArmed: imageRescaleToolArmed(session) }); + // #7398 all-lanes (ruling 7439-4): serve this turn's outbound core from + // the egress op-log (no title-gen side requests ride this lane, so no + // extra gate). The lane's own conversion chain turns it into wire below. + if (featureEnabled("egressPlane")) { + const st = getStore().egressOpLogStore(); + const served = st ? serveLaneCore(st, sessionId, log) : null; + if (served !== null) egressMessages = served; + } rebuiltMessages = coreToAnthropic(egressMessages, anthropicCacheMarks ?? clientCacheControls); + // #7398 all-lanes: record this turn's outbound core view (rollout-gated). + if (featureEnabled("egressPlane")) { + const st = getStore().egressOpLogStore(); + if (st) recordLaneTurn(st, sessionId, egressMessages, extractSystem(systemOut) ?? "", log); + } if (sysNotes.length > 0) { rebuiltMessages = insertBeforeTrailingSystemRun(rebuiltMessages, sysNotes.map((text) => ({ role: "user" as const, content: text }))); } @@ -634,7 +692,6 @@ export async function prepareAnthropic( // SysPrompt row — the kernel breakdown classifies messages only, and on // this wire the system rides the top-level `system` field outside the fold. session.metadata.systemPromptTokens = countSystemAndToolsTokens(extractSystem(systemOut), toolsOut); - snapshotMessages(session, originalMessages); // #3397 durability-before-render: make this turn's ledger rows durable NOW // (synchronous SQLite ingest, no JSON envelope) so the refs the model is // about to see are on disk BEFORE the response is sent. Without this, a @@ -643,7 +700,10 @@ export async function prepareAnthropic( // gate rejects as "above the highest allocated ref". markDirty below keeps // the JSON envelope debounced; this only moves the small row ingest ahead. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); // [#5604] a preflight fold of the tail (soft protection relaxed to 0, or a // tail tool pair in neverPreserveRecentTools) can leave the outbound array @@ -706,7 +766,7 @@ export async function prepareAnthropic( probeRequestMembership(session, collectRequestIds(incomingMsgIds, originalMessages), session.stats.localInputEstimate ?? 0); if (upstreamOrigin) session.stats.lastLocalTextEstimateOrigin = upstreamOrigin; if (nudgeOutcome === "injected") recordNudgeInject(session, nudge); - return { body: JSON.stringify(rebuilt), parsedBody: rebuilt, session, attachedRetrievals, attachedRetrievalNoteIds, processedMessages, originalMessages, judgeBaseline, anthropicSystem: parsed.system, anthropicBillingBlock, anthropicCacheMarks, anthropicClientCacheControls: clientCacheControls, systemNotes: sysNotes, protocol: "anthropic", stream, compressInjected: injectTools, pluginMode, mcpDialectNote: laneDecisions.mcpDialect, nudge, nudgeOutcome, nudgeStampsBefore: nudgeOutcome === "injected" ? nudgeStampsBefore : undefined, prompts, surface, renderTags: renderStrategy, dropReasoning: (msgs: BiliMessage[]) => applyReasoningDrop(msgs, reasoningPlan, log, sessionId) } as Prepared; + return { body: JSON.stringify(rebuilt), parsedBody: rebuilt, session, attachedRetrievals, attachedRetrievalNoteIds, processedMessages, originalMessages, judgeBaseline, anthropicSystem: parsed.system, anthropicBillingBlock, anthropicCacheMarks, anthropicClientCacheControls: clientCacheControls, systemNotes: sysNotes, protocol: "anthropic", responseExtraHeaders: msgidModeHeaders, stream, compressInjected: injectTools, pluginMode, mcpDialectNote: laneDecisions.mcpDialect, nudge, nudgeOutcome, nudgeStampsBefore: nudgeOutcome === "injected" ? nudgeStampsBefore : undefined, prompts, surface, renderTags: renderStrategy, dropReasoning: (msgs: BiliMessage[]) => applyReasoningDrop(msgs, reasoningPlan, log, sessionId) } as Prepared; } export const TRAILING_ASSISTANT_NUDGE = "[bili] Earlier context was compressed above. Continue the task."; diff --git a/src/server/prepare-google.ts b/src/server/prepare-google.ts index d12f08249d..73f594bce7 100644 --- a/src/server/prepare-google.ts +++ b/src/server/prepare-google.ts @@ -4,6 +4,8 @@ import { renderNudgeText, viableRanges } from "acp-kernel"; import { recordNudgeInject } from "../cache-ledger.js"; import { coreToGoogle, googleToCore, normalizeGoogleUserPartOrder, type BiliMessage, type GoogleContent, type GoogleRequestBody, type GoogleSystemInstruction, type GoogleTool } from "acp-kernel/wire"; import type { ProxyOptions } from "../config.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, applyMsgIdPin, validateMsgIds } from "../msgid-pin.js"; +import { featureEnabled } from "../rollout.js"; import { pluginAgentHeader } from "../plugin.js"; import { currentCalibrationFactor } from "../util.js"; import { diagNudge, diagTagSummary, deriveTitle, effectiveTokenCount, imageBillingFor, imageReserveFor, imageTokenCapFor, isAutoInjectedNotification, reapOrphansLogged, resolveKnownOutputCeiling, runNudgeDecision, stripKernelSummaries, usageGradeInputBaseline, type Prepared } from "../server.js"; @@ -34,12 +36,14 @@ import { attachSubagentSessions } from "../subagent-sessions.js"; import { reconcileSystemAnchor } from "../system-anchor.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; import { imageRescaleMaxDim, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore } from "../persist.js"; import { applyImageRescalePass } from "../image-rescale.js"; import { clampOutgoingOutput, countSystemAndToolsTokens, dshLedgerFloorTokens, emergencyNudge, emergencyNudgeArmed } from "./budget.js"; import { estimateCoreMessages } from "../preflight.js"; import { effectiveAbsorbBlock, effectiveBlockOpsEnabled } from "./prepare-responses.js"; import { injectGoogleTool, injectTool } from "./inject.js"; import { nudgeOn, resolveLaneDecisions, surfaceOffDetail } from "./lane-decisions.js"; +import { recordLaneTurn, serveLaneCore } from "./egress-lane-io.js"; function appendGoogleNudge(contents: GoogleContent[], text: string, model?: string): GoogleContent[] { const last = contents[contents.length - 1]; @@ -89,6 +93,16 @@ export async function prepareGoogle( ): Promise { const sessionId = session.id; ++session.stats.requests; + // #6525: the client may stamp the envelope (`biliMsgIds`) on plugin-wire + // full-resend turns. Read + strip it before anything can reach the real + // upstream, then pin behind the rollout gate (same contract as the openai/ + // anthropic twins). Nothing extra is ever forwarded either way. + const incomingMsgIds = (parsed as Record)[MSG_IDS_BODY_FIELD]; + delete (parsed as Record)[MSG_IDS_BODY_FIELD]; + // Same unconditional strip for the seqs carrier (biliLedgerSeqs) — parity + // with the openai/anthropic/responses prepare lanes: the real upstream + // must never see a field it does not know. + delete (parsed as Record)[MSG_SEQS_BODY_FIELD]; const inboundMessageCount = Array.isArray(parsed.contents) ? parsed.contents.length : 1; let googleClientSystem = ""; let sysNotes: string[] = []; @@ -124,8 +138,13 @@ export async function prepareGoogle( const markerCounts = stripRawStage(parsed, "google"); logRawStripCounts(markerCounts, log, sessionId); + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; try { - const { msgs, systemText } = googleToCore(parsed); + const coreOut = googleToCore(parsed); + let msgs = coreOut.msgs; + const systemText = coreOut.systemText; googleClientSystem = systemText; // Title-gen side-requests carry their own tiny system — reconciling // them would pollute the conversation's anchor state. @@ -135,6 +154,23 @@ export async function prepareGoogle( googleClientSystem = outcome.outbound; } originalMessages = msgs; + // #6525/#4491 parity: pin identity to client-supplied ids (plugin lanes + // only — pure proxy traffic has no msgid channel), the SAME gated + // contract as the openai/anthropic twins. No watermark fast-path on + // this wire, so there is no `wpHit` skip. + if (pluginMode && incomingMsgIds !== undefined) { + if (featureEnabled("pluginLaneServerHistory")) { + const v = validateMsgIds(incomingMsgIds, msgs.length); + if (v.ok) { + msgs = applyMsgIdPin(msgs, v.ids); + originalMessages = msgs; + } else { + log("warn", `[${sessionId}] [msgid] google-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#6525)`); + } + } else { + log("info", `[${sessionId}] [msgid] google-wire biliMsgIds present but pluginLaneServerHistory rollout is off for this install — field stripped, ids not pinned (#6525)`); + } + } // #3756 snapshot render: the outbound prefix renders from the unified // transcript (server-authoritative, fold-time ids — immune to identity // re-mints #3752); only the live tail keeps the client's bytes. @@ -216,6 +252,14 @@ export async function prepareGoogle( const nudgeHeldAfterReject = session.transient?.nudgeHeldAfterReject === true; if (nudgeHeldAfterReject) restoreNudgeStamps(session, nudgeStampsBefore); adoptContentStore(session, turn.contentStore); + // In-request ledger flush: the transcript up to THIS user message is + // durable before the model call (#3397 durability-before-render) and + // the stage-0 envelope / M2 evidence read a current ledger. + // snapshotMessages first: the persist build reads session.lastMessages + // — and it stays the RAW client resend (never the D-lane view: its + // synthetic acp_summary anchor is not a transcript message). + snapshotMessages(session, originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -301,7 +345,15 @@ export async function prepareGoogle( reapOrphansLogged(session, msgs, log, sessionId); // #3400: egress image rescale — outbound copies only; the originals // stay in session state/persistence untouched. - const egressMessages = await applyImageRescalePass(processedMessages as BiliMessage[], { session, sessionId, log, maxDim: imageRescaleMaxDim(), toolArmed: imageRescaleToolArmed(session) }); + let egressMessages = await applyImageRescalePass(processedMessages as BiliMessage[], { session, sessionId, log, maxDim: imageRescaleMaxDim(), toolArmed: imageRescaleToolArmed(session) }); + // #7398 all-lanes (ruling 7439-4): serve this turn's outbound core from + // the egress op-log — the recorded bytes are this session's server + // truth; the lane's own conversion chain turns it into wire below. + if (!isTitleGen && featureEnabled("egressPlane")) { + const st = getStore().egressOpLogStore(); + const served = st ? serveLaneCore(st, sessionId, log) : null; + if (served !== null) egressMessages = served; + } rebuiltContents = coreToGoogle(egressMessages, { model }); // ONLY the static compress prompt joins the client's system text — the @@ -333,6 +385,11 @@ export async function prepareGoogle( // instead of being re-serialized into a new shape. const extraSystemParts = sysParts.slice(googleClientSystem ? 1 : 0); systemInstruction = extraSystemParts.length > 0 ? { parts: sysParts.map((text) => ({ text })) } : parsed.systemInstruction; + // #7398 all-lanes: record this turn's outbound core view (rollout-gated). + if (!isTitleGen && featureEnabled("egressPlane")) { + const st = getStore().egressOpLogStore(); + if (st) recordLaneTurn(st, sessionId, egressMessages, googleOutboundSystem ?? googleClientSystem, log); + } if (injectTools) { toolsOut = injectGoogleTool(parsed.tools, [...(absorbActive ? [absorbTools.google] : []), ...(rulesActive ? [RULE_TOOL_GOOGLE] : []), ...(ccrEnabled(session) ? [retrieveToolsFor(retrieveToolName(session)).google] : []), ...(imageRescaleToolArmed(session) ? [READ_ORIGINAL_IMAGE_TOOL_GOOGLE] : []), ...(blockOpsActive ? blockOps.google : [])], surface?.toolPrompts, ccrEnabled(session), externalSummaryEnabled(config)); } @@ -417,10 +474,12 @@ export async function prepareGoogle( + imageReserveFor(session, "google", rebuilt, opts, upstreamOrigin); if (upstreamOrigin) session.stats.lastLocalTextEstimateOrigin = upstreamOrigin; } - snapshotMessages(session, originalMessages); // #3397 durability-before-render (see prepare-anthropic.ts): rows durable NOW, before the model call. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); if (nudgeOutcome === "injected") recordNudgeInject(session, nudge); return { body: JSON.stringify(rebuilt), parsedBody: rebuilt, session, processedMessages, originalMessages, judgeBaseline, protocol: "google", stream, compressInjected: injectTools, pluginMode, mcpDialectNote: laneDecisions.mcpDialect, nudge, nudgeOutcome, nudgeStampsBefore: nudgeOutcome === "injected" ? nudgeStampsBefore : undefined, prompts, surface, google: { system: googleClientSystem, model }, systemNotes: sysNotes, renderTags: renderStrategy } as Prepared; } diff --git a/src/server/prepare-openai.ts b/src/server/prepare-openai.ts index 96446f8a1f..afc3d969be 100644 --- a/src/server/prepare-openai.ts +++ b/src/server/prepare-openai.ts @@ -8,7 +8,7 @@ import type { ProxyOptions } from "../config.js"; import { effectivePromptCacheRouting } from "../config.js"; import { currentCalibrationFactor, hardenOpenaiAssistantContent, systemToUser } from "../util.js"; import { conversationHeaderSource, resolveRelayAffinityPck } from "../session-id.js"; -import { MSG_IDS_BODY_FIELD, applyMsgIdPin, validateMsgIds } from "../msgid-pin.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, MSGID_MODE_HEADER, applyMsgSeqStamps, incomingSeqsOf, validateMsgSeqs } from "../msgid-pin.js"; import { pluginAgentHeader } from "../plugin.js"; import { applyReasoningDrop, diagNudge, diagTagSummary, deriveTitle, effectiveTokenCount, imageBillingFor, imageReserveFor, imageTokenCapFor, isAutoInjectedNotification, planReasoningDrop, reapOrphansLogged, resolveKnownOutputCeiling, runNudgeDecision, stripKernelSummaries, warnDroppedOpenaiParts, warnReasoningPairs, type Prepared } from "../server.js"; import { recordNudgeObservation } from "../context-analysis.js"; @@ -42,15 +42,74 @@ import { isStrictReasoningEcho, modelIdOf, normalizeStrictEchoReasoning } from " import { reconcileSystemAnchor } from "../system-anchor.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage, type RawStripCounts } from "../marker/engine.js"; import { imageRescaleMaxDim, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore } from "../persist.js"; +import { FoundationRejectError } from "../snapshot/plane.js"; +import { runStage0 } from "../snapshot/stage0.js"; import { applyImageRescalePass } from "../image-rescale.js"; import { clampOutgoingOutput, countSystemAndToolsTokens, dshLedgerFloorTokens, emergencyNudge, emergencyNudgeArmed, outputBudgetField, readOutputBudget, writeOutputBudget } from "./budget.js"; import { effectiveAbsorbBlock, effectiveBlockOpsEnabled } from "./prepare-responses.js"; import { injectOpenaiTool, injectTool } from "./inject.js"; import { nudgeOn, resolveLaneDecisions, surfaceOffDetail } from "./lane-decisions.js"; import { featureEnabled } from "../rollout.js"; -import { evaluateWatermarkFastpath, finalizeWatermarkTurn, stampOnlyWatermarkTurn, type FastpathHit, type FastpathOutcome } from "./fastpath.js"; +import { recoverEgressBase, recordEgressTurn } from "../storage/egress-record.js"; +import type { EgressViewRow } from "../storage/egress-state.js"; +// Structural mirror of egress-record.ts's EgressSourceMsg (kept private +// there by the unused-export gate): the recording seam only reads this shape. +interface EgressRecordSourceMsg { + id: string; + role: string; + contentType?: unknown; + text?: unknown; +} +import { rebuildCoreFromLog, rebuildOutboundFromLog } from "../storage/egress-serve.js"; +import { DeltaReanchorError, PF_TAIL_IDS_HEADER, evaluateDeltaResend, evaluateWatermarkFastpath, finalizeWatermarkTurn, parseTailIdsForPin, type FastpathHit, type FastpathOutcome } from "./fastpath.js"; import { collectRequestIds, probeRequestMembership } from "../request-membership.js"; +/** Ruling 7439-5 input probe (discussions/7439) — light per-turn identity + * + byte SAMPLING on sampled rows only (never a full byte comparison). + * Part 1 ORDER: every client id that exists in the server view must appear + * in server-view order (append-only + client-side folding contract; a + * reorder means the client is not echoing the server's conversation → the + * turn degrades to legacy). Part 2 DIVERGENCE: up to 8 sampled rows compare + * contentType + text length + (short, ≤32KB) exact text. A mismatch does NOT + * fail the probe — the seam keeps server bytes regardless of what the client + * sent; sampling only LOGS the divergence event (ruling: 记日志 + 警告, + * 内容跟服务端, 出站自然收敛). */ +export function egressAuthorityProbe( + base: readonly BiliMessage[], + client: readonly BiliMessage[], + sessionId: string, + log: (level: string, msg: string) => void, +): boolean { + const baseIndex = new Map(); + base.forEach((m, i) => { + if (!baseIndex.has(m.id)) baseIndex.set(m.id, i); + }); + let cursor = -1; + for (const c of client) { + const i = baseIndex.get(c.id); + if (i === undefined) continue; + if (i <= cursor) return false; + cursor = i; + } + let diverged = 0; + const step = Math.max(1, Math.floor(client.length / 8)); + for (let i = 0; i < client.length; i += step) { + const c = client[i]; + const bi = baseIndex.get(c.id); + if (bi === undefined) continue; + const b = base[bi]; + const ct = typeof c.text === "string" ? c.text : ""; + const bt = typeof b.text === "string" ? b.text : ""; + if (c.contentType !== b.contentType || ct.length !== bt.length || (ct.length <= 32768 && ct !== bt)) diverged++; + if (diverged >= 4) break; + } + if (diverged > 0) { + log("warn", `[${sessionId}] egress-authority divergence: ${diverged} sampled row(s) differ client-vs-server — server bytes win (client edits do not stick until the egress update API exists; discussions/7439 ruling 5)`); + } + return true; +} + export async function prepareOpenai( parsed: OpenAIRequestBody, req: http.IncomingMessage, @@ -69,6 +128,7 @@ export async function prepareOpenai( billingUpstream?: string, decide?: DecideConfig, bodyText?: string, + bodyParseDeferred = false, ): Promise { const sessionId = session.id; const stream = parsed.stream === true; @@ -78,7 +138,30 @@ export async function prepareOpenai( // host body stamps (pi incremental lane) or the #5825 handle-seam // re-stamp of stamped-header/harvested ids. const incomingMsgIds = (parsed as Record)[MSG_IDS_BODY_FIELD]; + // #5874 2×2: the seqs carrier (biliLedgerSeqs) — the M2 incremental + // join evidence. An optimization channel, not an identity channel: + // malformed/absent → the M2 fast path is disabled (the full-resend + // matchWorld path still works); it never 400s. (Stamped once msgs + // exists, below.) + const incomingSeqsRaw = incomingSeqsOf(parsed); + delete parsed[MSG_SEQS_BODY_FIELD]; delete (parsed as Record)[MSG_IDS_BODY_FIELD]; + // #7064 §8: the INCREMENTAL pin reuses the watermark's bounded tail-ids + // header — the same bytes that ride the wire for the hit verdict pin the + // prefix when the full list is absent. Three-state handoff to stage 0 + // (undefined = header absent, null = malformed → degrade, array = valid, + // "?" slots kept). The fastpath re-parses the header itself for its own + // verdict — one parser, one rule (parseTailIdsForPin). + // #6785 review note (owner): this parse is DELIBERATELY ungated by + // pluginMsgIdIncremental — a flag-OFF server must still READ a client's + // tail-ids declaration, because the rollback self-heal signal ("full") + // is derived from it (stage 0's tail-ids branch). Gating the parse + // behind the flag would strand upgraded clients after a rollback. + const rawTailIdsHeader = req.headers[PF_TAIL_IDS_HEADER]; + const incomingTailIds: string[] | null | undefined = + rawTailIdsHeader === undefined ? undefined + : typeof rawTailIdsHeader === "string" ? parseTailIdsForPin(rawTailIdsHeader) + : null; // Real wire element count; a watermark hit overrides it with the measured // total T (see the #5403/#5418 block below). let inboundMessageCount = Array.isArray(parsed.messages) ? parsed.messages.length : 1; @@ -100,6 +183,18 @@ export async function prepareOpenai( let nudgeStampsBefore: Prepared["nudgeStampsBefore"]; let rebuiltMessages = parsed.messages; let toolsOut = parsed.tools; + // #7398 seg 4: the legacy core-portion elements (captured at the + // :coreToOpenai chain) — the shadow comparison target for serve-from-log. + let legacyCoreElements: unknown = undefined; + let egressBaseRows: EgressViewRow[] | null = null; + let egressSwapApplied = false; + let egressDegraded = false; + let egressArmed = false; + let egressServedFromLog = false; + // The EXACT array fed to the outbound chain (post reasoning-drop / ccr / + // image-rescale) — segment 3 records this, not processedMessages, so the + // log view is position-for-position what outbound saw. + let egressRecordSource: readonly EgressRecordSourceMsg[] | undefined; // #2483: single source for the effective render strategy — processTurn, the // diag log and Prepared must all report what was actually rendered (the // log label used to hardcode "text-only" and lie under renderNone). @@ -137,8 +232,28 @@ export async function prepareOpenai( const wpModel = typeof parsed.model === "string" ? parsed.model : ""; if (bodyText !== undefined && pluginMode && !isTitleGen && featureEnabled("pluginFullResendWatermark")) { wpFullText = bodyText; - wpOutcome = evaluateWatermarkFastpath(bodyText, req, session, { pluginMode, titleGen: isTitleGen, model: wpModel }, log); - log("debug", `[${session.id}] fastpath ${wpOutcome.hit ? "HIT" : wpOutcome.decision}${wpOutcome.creditedMiss ? "(credited)" : ""}${wpOutcome.measuredT !== undefined ? ` T=${wpOutcome.measuredT}` : ""}`); + // Delta arm first: a delta claim is EXCLUSIVE — the body carries only + // the tail slice, so the full-body lane cannot evaluate it at all. A + // satisfiable claim yields the same FastpathHit the full lane would; + // an unsatisfiable one throws the 400 re-anchor (handle.ts). + const wpDelta = evaluateDeltaResend(parsed, bodyText, req, session, { pluginMode, titleGen: isTitleGen, model: wpModel }, log); + if (wpDelta !== null) { + if ("reanchor" in wpDelta) throw new DeltaReanchorError(wpDelta.reanchor); + wpOutcome = { hit: wpDelta.hit, measuredT: wpDelta.hit.totalT, decision: "delta-hit", creditedMiss: false, receivedTailIds: wpDelta.receivedTailIds }; + log("debug", `[${session.id}] fastpath ${wpOutcome.decision} T=${wpDelta.hit.totalT}`); + } else { + wpOutcome = evaluateWatermarkFastpath(bodyText, req, session, { pluginMode, titleGen: isTitleGen, model: wpModel }, log); + log("debug", `[${session.id}] fastpath ${wpOutcome.hit ? "HIT" : wpOutcome.decision}${wpOutcome.creditedMiss ? "(credited)" : ""}${wpOutcome.measuredT !== undefined ? ` T=${wpOutcome.measuredT}` : ""}`); + } + // #7398 perf: the deferral gate skipped the full JSON.parse in handle.ts + // (150-600ms on a 155MiB resend). The lanes above only consumed the span + // scan + window fragments; a miss — or the gates staying closed — means + // the legacy path needs the whole conversation array, so parse it exactly + // once, here. A throw is equivalent to the legacy parse failure (same + // catch in handle.ts). + if (bodyParseDeferred && !wpOutcome?.hit && !Array.isArray(parsed.messages) && bodyText !== undefined) { + parsed = JSON.parse(bodyText) as OpenAIRequestBody; + } if (wpOutcome.hit) { wpHit = wpOutcome.hit; // #6752: handle.ts already ran restoreOutputBudget on `parsed` — @@ -176,6 +291,12 @@ export async function prepareOpenai( const markerCounts = wpHit !== undefined ? { ...wpHit.markerCounts, strippedMs: 0 } : stripRawStage(parsed, "openai"); logRawStripCounts(markerCounts, log, sessionId); + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; + // #7064 §8: hoisted so the end-of-prepare response-header block can stamp + // the pin-mode signal (stage 0 runs inside the try below). + let s0PinMode: "full" | "incremental" | null = null; try { // Kernel 0.0.37 hoists the contiguous leading system/developer prefix // OUT of the fold space: system content is host runtime state and @@ -183,6 +304,14 @@ export async function prepareOpenai( // otherwise the proxy would forward payloads without any system. const converted = wpHit !== undefined ? { msgs: wpHit.view, systemText: wpHit.systemText } : openaiToCore(parsed); let msgs = converted.msgs; + // #5874 2×2: stamp the M2 seqs carrier onto the converted array + // (parallel to messages; malformed/absent → null → M2 fast path + // disabled, the full-resend path still works). + const incomingSeqs: number[] | null = + incomingSeqsRaw !== undefined && incomingSeqsRaw.length === msgs.length && validateMsgSeqs(incomingSeqsRaw, msgs.length).ok + ? incomingSeqsRaw + : null; + if (incomingSeqs !== null) applyMsgSeqStamps(msgs, incomingSeqs); let systemText = converted.systemText; openaiSystemText = systemText; wpSeedSystemText = systemText; @@ -196,25 +325,81 @@ export async function prepareOpenai( openaiSystemText = outcome.outbound; } originalMessages = msgs; - // #4491/#5825 parity: pin identity to client-supplied ids (plugin - // lanes only — pure proxy traffic has no msgid channel and stays on - // snapshot render D), the SAME gated contract as the anthropic twin. - // Skipped on watermark hits on purpose: fastpath turns reuse the - // seed stash's prefix view whose rows were already pinned when it was - // materialized, and the decoded tail window is SHORTER than the - // stamped list, so count-parity there would fail by design — pinning - // only adds noise. - if (wpHit === undefined && pluginMode && incomingMsgIds !== undefined) { - if (featureEnabled("pluginLaneServerHistory")) { - const v = validateMsgIds(incomingMsgIds, msgs.length); - if (v.ok) { - msgs = applyMsgIdPin(msgs, v.ids); - originalMessages = msgs; + // #4491/#5825 + #5874 2×2 stage 0 (brief 19544722 / #6181): the + // SHARED pin + world-match + kernel-input assembly for cohort + // traffic (src/snapshot/stage0.ts — the responses lane runs the + // SAME helper, same red gate, same one-round-convergence + // contract). Runs BEFORE processTurn, so a thrown + // FoundationRejectError (R3) carries zero kernel side effects + // (the server funnel maps it to the hard 400); every other fault + // → the legacy full path (one-round convergence). + const s0 = runStage0({ + sessionId, + lane: "openai", + msgs, + incomingMsgIds, + incomingSeqs, + incomingTailIds, + pluginMode, + wpHit, + log, + }); + s0PinMode = s0.pinMode; + if (s0.pinned !== null) { + msgs = s0.pinned; + originalMessages = msgs; + } + // #7064 D3 (尾锚常活): the stage-0 cursor may have advanced BEFORE + // processTurn — bump the session revision so the per-turn snapshot + // flush below (snapFlush epoch proof) observes the advance. A fork + // taken between the cursor write and the flush otherwise reads a + // STALE parent revision and lands the child on the wrong lane. + if (s0.anchorAdvanced) markDirty(session); + // Ruling 7439-5 (owner strategy decision, discussions/7439 §1) — the + // server egress flow is the SOLE authority over conversation content. + // This is the ONE prepare-layer seam where the client's resend is + // demoted to what it is after turn 1: the identity/order channel (the + // probe below) and the byte source for NEW messages only. On a pinned + // plugin-lane turn with a live op-log flow, every EXISTING row below + // carries the bytes the SERVER recorded last turn; a client byte edit + // under a stable msgid is a divergence event (logged + warned), the + // server bytes win, and — because the outbound is served from the same + // log below — the client converges on its next echo. Legitimate client + // edits need the egress update API (follow-up, not in this PR). This + // OVERRIDES the pre-PR plugin-lane stance (snapshot-render.ts: "plugin + // lane: client-authoritative by design"). Scope is deliberately the + // PINNED plugin lane (both modes reasoned): bare proxy clients are + // tamper-evident by construction — their ids ARE content hashes, so a + // byte edit changes the id and already lands in the rewrite/drift + // gates — and proxy-lane kernel input must stay the RAW history (the + // kernel folds internally; a post-fold replay view would double-fold + // and trip #1001/#2432/#4521). The proxy lane gets ruling 5 on the + // EGRESS side (outbound served from the log); the pinned plugin lane + // gets it end-to-end (kernel input + outbound). + // Degrade (never guess, never client-as-authority): store missing, + // empty/unreadable flow, unrebuildable view, or a failed probe → this + // turn runs the legacy client-byte path and re-attempts next turn. + egressArmed = featureEnabled("egressPlane") && pluginMode && !isTitleGen && s0.pinned !== null; + if (egressArmed) { + const st = getStore().egressOpLogStore(); + if (st) { + const base0 = recoverEgressBase(st, sessionId); + if (base0 === null) { + log("error", `[${sessionId}] egress-authority: no healthy flow point — session-level recovery required (ruling 7439-5); legacy bytes this turn`); } else { - log("warn", `[${sessionId}] [msgid] openai-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#5825)`); + egressBaseRows = base0.view; + egressDegraded = base0.degraded; + const base = (base0.view.length > 0 ? rebuildCoreFromLog(base0.view) : null) ?? []; + if (base.length > 0 && egressAuthorityProbe(base, msgs, sessionId, log)) { + const baseById = new Map(base.map((m) => [m.id, m] as const)); + msgs = msgs.map((c) => baseById.get(c.id) ?? c); + originalMessages = msgs; + egressSwapApplied = true; + log("info", `[${sessionId}] egress-authority: ${base.length} existing row(s) served from the op-log, ${Math.max(0, msgs.length - base.length)} fresh from the client${base0.degraded ? ` (recovered to log_seq=${base0.seq}, truncated ${base0.truncated} row(s))` : ""}`); + } else if (base.length > 0) { + log("warn", `[${sessionId}] egress-authority: probe failed — flow NOT advanced this turn; legacy bytes served (re-attempts next turn)`); + } } - } else { - log("info", `[${sessionId}] [msgid] openai-wire biliMsgIds present but pluginLaneServerHistory rollout is off for this install — field stripped, ids not pinned (#5825)`); } } // #3756 snapshot render: the outbound prefix renders from the unified @@ -223,6 +408,7 @@ export async function prepareOpenai( // No-op returns `msgs` when the lane is inactive/degraded. // reconcileFoldCoverage below still observes the CLIENT view. const renderMsgs = maybeSnapshotView(session, msgs, { sessionId, pluginMode, log }); + // #4241: the render lane replaced the view (reference-inequality is the // discriminator — maybeSnapshotView returns the SAME array when idle), // so the judged baseline is the rendered transcript, not the inbound. @@ -349,6 +535,47 @@ export async function prepareOpenai( const nudgeHeldAfterReject = session.transient?.nudgeHeldAfterReject === true; if (nudgeHeldAfterReject) restoreNudgeStamps(session, nudgeStampsBefore); adoptContentStore(session, turn.contentStore); + // In-request ledger flush: the transcript up to THIS user message is + // durable before the model call (#3397 durability-before-render) and + // the stage-0 envelope / M2 evidence read a current ledger. + // snapshotMessages first: the persist build reads session.lastMessages + // — and it stays the RAW client resend (never the D-lane view: its + // synthetic acp_summary anchor is not a transcript message). The + // watermark window pin is preserved. + // #7366: on a pinned HIT turn the ledger window must carry the + // DECLARED ids, not the window cores' fresh content-hash ids. + // wpHit.windowCores is the tail fragment re-converted by openaiToCore + // (fresh h_… mints), while s0.pinned is the SAME view with the id + // field re-stamped to the client-declared host msgids (stage 0). + // Ingesting the raw window cores wrote h_… into the ledger for every + // post-seed row (msg_id is write-once — INSERT OR IGNORE on + // (session_id, seq)), so the next turn's KDD#21 tail-overlap check + // compared declared host ids against h_… rows → guaranteed drift → + // full form every turn. pinned is element-wise a map of the same + // view (ids only), so its tail slice aligns position-for-position + // with windowCores; the length guard falls back to the legacy cores + // on any shape anomaly, and unpinned turns are byte-identical. + const ledgerWindow = + wpHit !== undefined && s0.pinned !== null && s0.pinned.length === wpHit.totalT + ? s0.pinned.slice(s0.pinned.length - wpHit.windowCores.length) + : wpHit?.windowCores; + snapshotMessages(session, ledgerWindow ?? originalMessages); + // #7064 ledger drain: on a watermark-HIT turn that snapshot is the + // ~2-row tail WINDOW — a verified suffix of the ledger (head check + + // pf-seq vouch for the prefix), NOT the complete log. Claiming rawness + // (lastMessagesFolded=false) makes the save present it as the raw tail, + // and the #3285 authoritative-set sweep then retires every prefix row + // the window omits — draining the msgid pin ledger (snapMsgIdLiveIds) + // to the window, so tryIncrementalPin sees ledger≈2 and the KDD#21 + // tail form degrades to the full form every turn (full-list body + + // 100k id validation + path=full ingest, alternating with window turns). + // Mark the tail non-raw: persist presents the envelope view, the sweep + // stays off, and the ledger keeps the watermark-protected prefix live + // (KDD#20 水位线以下存活行). Every consumer of the flag already treats + // "pruned view, not raw log" as the skip-sweep / skip-append contract + // (persist tail selection, ingest tail_json, handoff stash, export). + if (wpHit !== undefined) session.lastMessagesFolded = true; + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -466,14 +693,10 @@ export async function prepareOpenai( // Client-origin mid system/developer messages (DSH tool_addition and // friends) are conversation data — they re-emit verbatim through the // kernel's rawOpenaiMessage sidecar and must never change voice (#4159). - rebuiltMessages = hardenOpenaiAssistantContent(coreToOpenai(systemToUser(egressMessages, (m) => isRenderedSummaryMessage(m)))); - - // ONLY the static compress prompt goes into the system message — the - // system prompt is the prefix-cache anchor and must be byte-stable - // across turns. The nudge (which changes every turn: token count, - // growth %, dynamic example) is appended as a trailing user message - // instead, mirroring pai-acp's design. Putting the nudge in system - // would invalidate the cache every turn. + // #7398 seg 4b: egress serve-from-log decision. The sysParts block that + // used to follow this line is computed FIRST (moved up) because the + // op-log recording stores the outbound system text and the rebuild + // replaces the conversation elements below. const sysParts: string[] = []; if (openaiSystemText) sysParts.push(openaiSystemText); let acpStandingPushed = false; @@ -488,15 +711,50 @@ export async function prepareOpenai( if (acpStandingPushed && detectGenerationBoundary(session, processedMessages, log)) sysParts.push(GENERATION_BOUNDARY_NOTE); if (absorbActive) sysParts.push(buildAbsorbSystemPrompt(absorbToolName(loopConfig))); if (blockOpsActive) sysParts.push(buildBlockOpsSystemPrompt("function")); + openaiOutboundSystem = sysParts.join("\n\n"); + if (!isTitleGen && featureEnabled("egressPlane") && !(egressArmed && !egressSwapApplied)) { + const st = getStore().egressOpLogStore(); + if (st) { + try { + const view = recordEgressTurn({ + store: st, + sessionId, + prev: egressBaseRows ?? undefined, + conversation: egressMessages, + systemText: openaiOutboundSystem, + nowMs: Date.now(), + }); + const reb = rebuildOutboundFromLog(view); + if (reb !== null) { + rebuiltMessages = reb.elements; + egressServedFromLog = true; + log("info", `[${sessionId}] egress-serve: served ${view.length} rows from op-log`); + } + } catch (err) { + egressServedFromLog = false; + log("warn", `[${sessionId}] egress-serve rebuild failed — legacy kept: ${String(err)}`); + } + } + } + if (!egressServedFromLog) { + rebuiltMessages = hardenOpenaiAssistantContent(coreToOpenai(systemToUser(egressMessages, (m) => isRenderedSummaryMessage(m)))); + } + legacyCoreElements = rebuiltMessages; + egressRecordSource = egressMessages; + + // ONLY the static compress prompt goes into the system message — the + // system prompt is the prefix-cache anchor and must be byte-stable + // across turns. The nudge (which changes every turn: token count, + // growth %, dynamic example) is appended as a trailing user message + // instead, mirroring pai-acp's design. Putting the nudge in system + // would invalidate the cache every turn. + // sysParts was computed above (seg 4b serve decision needs it). rebuiltMessages = injectOpenaiSystem(rebuiltMessages, sysParts); if (sysNotes.length > 0) { rebuiltMessages = [...rebuiltMessages, ...sysNotes.map((text) => ({ role: "user" as const, content: text }))]; } - // #532: capture what bili injects outside the fold space (client system - // + compress prompt). A head system message already in the rebuilt view - // is classified by the kernel breakdown — counting only these parts - // avoids double-counting it. - openaiOutboundSystem = sysParts.join("\n\n"); + // #532: openaiOutboundSystem was captured above (seg 4b moved it up so + // the op-log recording stores the outbound system text). if (injectTools) { toolsOut = injectOpenaiTool(parsed.tools, [...(absorbActive ? [absorbTools.openai] : []), ...(rulesActive ? [RULE_TOOL_OPENAI] : []), ...(ccrEnabled(session) ? [retrieveToolsFor(retrieveToolName(session)).openai] : []), ...(imageRescaleToolArmed(session) ? [READ_ORIGINAL_IMAGE_TOOL_OPENAI] : []), ...(blockOpsActive ? blockOps.openai : [])], surface?.toolPrompts, ccrEnabled(session), externalSummaryEnabled(config)); } @@ -589,6 +847,11 @@ export async function prepareOpenai( attachedRetrievalNoteIds = retrNotes.map((n) => n.id); } } catch (err) { + // #6181 B1: the R3 identity fault is a HARD reject, not a degrade — + // rethrow it to the server funnel (400 + x-bili-snapshot:identity) + // BEFORE the generic 503; zero kernel side effects by construction + // (it throws before processTurn). + if (err instanceof FoundationRejectError) throw err; log("warn", `[${sessionId}] kernel transform failed, forwarding unchanged: ${String(err)}`); if (attachedRetrievals.length > 0) dropRetrievals(session, attachedRetrievals.map((i) => i.ref), "prepare failed; forwarded unprocessed"); processedMessages = []; @@ -664,18 +927,12 @@ export async function prepareOpenai( // #5403/#5418: on a watermark hit pin only this turn's window — the // stashed prefix lives OUTSIDE the Session by design; the fold machinery // applies to the pin exactly as it already does for folded sessions. - snapshotMessages(session, wpHit !== undefined ? wpHit.windowCores : originalMessages); - // #7346: on a HIT the pin is this turn's window (the new tail), NOT the - // complete raw log. snapshotMessages stamps it raw (folded=false), so the - // save presents the window as the full log and the #3285 authoritative-set - // sweep retires every verified prefix row the window omits — a 100k-session - // ledger drained to the ~2-row window each turn. Mark the pin non-raw so - // the save ingests the window through the bounded envelope view and does - // NOT sweep the prefix. - if (wpHit !== undefined) session.lastMessagesFolded = true; // #3397 durability-before-render (see prepare-anthropic.ts): rows durable NOW, before the model call. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); let wpResponseHeaders: Record | undefined; if (wpFullText !== undefined && wpOutcome !== null && wpTurnStrip !== undefined) { wpResponseHeaders = finalizeWatermarkTurn(session, wpOutcome, { @@ -684,16 +941,65 @@ export async function prepareOpenai( rawSystemText: wpHit !== undefined ? wpHit.systemText : wpSeedSystemText, turnStripCounts: wpTurnStrip, }); - if (Object.keys(wpResponseHeaders).length === 0) wpResponseHeaders = undefined; - } else if (bodyText !== undefined && pluginMode && !isTitleGen && !featureEnabled("pluginFullResendWatermark")) { - // #7367: flag off — the fastpath never evaluates, but the x-bili-pf-seq - // heartbeat still must flow or the client's watermark freezes and its - // tail-ids grow past the 16 KB header limit. Stamp-only: no seed, no - // saved state (the fastpath re-establishes both when it next runs). - wpResponseHeaders = stampOnlyWatermarkTurn(bodyText); + // #7064 §8: the pin-mode signal rides WITH the watermark headers — + // the client relearns the mode exactly when it relearns T (same + // chokepoint, one round-trip). "incremental" is gated on the flag + // (off ⇒ the upgrade never travels); the "full" DOWNGRADE signal + // travels even when the flag is off — that is the rollback + // self-heal (#6785 review fix): an already-upgraded client must be + // told to go back to the full form when the flag is rolled back, + // or it keeps suppressing the full list while the server pins + // nothing (silent pin loss until compaction/restart). + if (s0PinMode === "full" || (featureEnabled("pluginMsgIdIncremental") && s0PinMode !== null)) { + wpResponseHeaders[MSGID_MODE_HEADER] = s0PinMode; + } if (Object.keys(wpResponseHeaders).length === 0) wpResponseHeaders = undefined; } if (nudgeOutcome === "injected") recordNudgeInject(session, nudge); + // #7398 seg 3: record this turn's outbound conversation view as sparse + // ops (rollout-gated, default off — zero rows, zero cost). Skipped when + // the serve path already ran or when the authority seam ran but did not + // swap (ruling 7439-5: never advance the flow with untrusted client + // bytes); a corrupt log skips this turn and self-heals next. + if (!isTitleGen && featureEnabled("egressPlane") && !egressServedFromLog && !(egressArmed && !egressSwapApplied)) { + try { + const st = getStore().egressOpLogStore(); + if (st) { + recordEgressTurn({ + store: st, + sessionId, + prev: egressBaseRows ?? undefined, + conversation: egressRecordSource ?? (processedMessages.length > 0 ? processedMessages : originalMessages), + systemText: openaiOutboundSystem ?? openaiSystemText, + nowMs: Date.now(), + }); + } + } catch (err) { + log("warn", `[${sessionId}] egress op-log recording skipped this turn (corrupt log point — self-heals next turn): ${String(err)}`); + } + } + // #7398 seg 4a: SHADOW comparison — rebuild the outbound core portion + // from the log and byte-compare against the legacy elements. NO + // swap: the pipeline always sends the legacy bytes; this measures the + // identity rate the swap will rely on (rollout egressPlane, default 0; + // reads the hydrated flow view). + if (!isTitleGen && featureEnabled("egressPlane") && !egressServedFromLog) { + try { + const st = getStore().egressOpLogStore(); + const view = egressBaseRows ?? (st ? st.egressStateView(sessionId) : undefined); + if (st && view && legacyCoreElements !== undefined) { + const reb = rebuildOutboundFromLog(view); + if (reb === null) { + log("info", `[${sessionId}] egress-serve shadow: unrebuildable (log gap or out-of-scope row) — legacy kept`); + } else { + const same = JSON.stringify(reb.elements) === JSON.stringify(legacyCoreElements); + log("info", `[${sessionId}] egress-serve shadow: ${same ? "IDENTICAL" : "DIVERGENT"} (${view.length} rows)`); + } + } + } catch (err) { + log("warn", `[${sessionId}] egress-serve shadow comparison failed: ${String(err)}`); + } + } return { body: JSON.stringify(rebuilt), parsedBody: rebuilt, session, attachedRetrievals, attachedRetrievalNoteIds, processedMessages, originalMessages, judgeBaseline, protocol: "openai", stream, compressInjected: injectTools, pluginMode, mcpDialectNote: laneDecisions.mcpDialect, nudge, nudgeOutcome, nudgeStampsBefore: nudgeOutcome === "injected" ? nudgeStampsBefore : undefined, prompts, surface, openaiSystemText, systemNotes: sysNotes, renderTags: renderStrategy, responseExtraHeaders: wpResponseHeaders, dropReasoning: (msgs: BiliMessage[]) => applyReasoningDrop(msgs, reasoningPlan, log, sessionId) } as Prepared; } diff --git a/src/server/prepare-responses.ts b/src/server/prepare-responses.ts index 94e16bfb2f..d80bc0dbce 100644 --- a/src/server/prepare-responses.ts +++ b/src/server/prepare-responses.ts @@ -17,11 +17,14 @@ import { mergeAdjacentConfigurationUpdates, patchResponsesInputWithToolImages as import { dropWhitespaceResponsesMessages, normalizeResponsesMessageItems, sanitizeResponsesInputIds } from "../loop/adapter-responses.js"; import { METADATA_LAST_RESPONSES_MODE, noteCanonicalSystemFingerprint, reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "../fold-reconcile.js"; import { buildAnchoredView, maybeSnapshotView, noteAnchoredRender, SnapshotAnchorFallbackError, wsAnchoredPayload } from "./snapshot-render.js"; +import { FoundationRejectError } from "../snapshot/plane.js"; +import { runStage0 } from "../snapshot/stage0.js"; +import { featureEnabled } from "../rollout.js"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, MSGID_MODE_HEADER, applyMsgSeqStamps, incomingSeqsOf, validateMsgSeqs } from "../msgid-pin.js"; import { storedMessagesFor, storedRowLookupFor } from "../fold-decomposition-lookup.js"; import { nudgeSuppressed, pluginLaneDegraded } from "../session-self-heal.js"; import { compressBreakerArmed } from "../stream.js"; -import { absorbAbortedTurnGrowth, nudgeStampsOf, restoreNudgeStamps, applyCompactionArchive, coveredRealHistoryIds, foldCoverage, markDirty, markNativeCompactionBoundary, reconcileNativeCompactionBoundary, REWRITE_MIN_INCOMING_TOTAL, snapshotMessages, type Session } from "../session.js"; -import { getStore } from "../persist.js"; +import { absorbAbortedTurnGrowth, applyCompactionArchive, coveredRealHistoryIds, foldCoverage, markDirty, markNativeCompactionBoundary, nudgeStampsOf, reconcileNativeCompactionBoundary, REWRITE_MIN_INCOMING_TOTAL, restoreNudgeStamps, snapshotMessages, type Session } from "../session.js"; import { carriesDshLocalCompactionSummary, DSH_LOCAL_COMPACTION_MIN_MISSING } from "./dsh-compaction-guard.js"; import { carriesBiliSnapshotCheckpoint } from "./dsh-authored-compaction.js"; import { recordConflict } from "../conflict-watch.js"; @@ -45,11 +48,13 @@ import { pckKeyAbated } from "../pck-probe.js"; import { reconcilePluginFoldCallIds } from "../plugin.js"; import { logRawStripCounts, markerArtifactCount, stripRawStage } from "../marker/engine.js"; import { imageRescaleMaxDim, keepResponseId as knobKeepResponseId, noCompressPrompt as knobNoCompressPrompt, noInjectTool as knobNoInjectTool, renderNone as knobRenderNone } from "../knobs.js"; +import { SessionStore, getStore } from "../persist.js"; import { applyImageRescaleItems } from "../image-rescale.js"; import { clampOutgoingOutput, countLoadedToolTokens, countSystemAndToolsTokens, dshLedgerFloorTokens, emergencyNudge, emergencyNudgeArmed, modelVisibleTools } from "./budget.js"; import { FORCE_TEXT_PROTOCOL, injectResponsesTool, injectTool } from "./inject.js"; import { nudgeOn, resolveLaneDecisions, surfaceOffDetail } from "./lane-decisions.js"; import { pluginAgentHeader } from "../plugin.js"; +import { recordLaneTurn, serveLaneCore } from "./egress-lane-io.js"; export async function prepareResponses( parsed: ResponsesRequestBody, @@ -106,6 +111,14 @@ export async function prepareResponses( } let processedMessages: CoreMessage[] = []; + // #6181 2×2: the msgid + ledger-seq dual-channel the plugin host rides + // on the body (src/msgid-pin.ts) — read and STRIPPED before any + // projection so the real upstream never sees a field it does not know. + // (Same discipline as the openai twin.) + const incomingMsgIds = (parsed as unknown as Record)[MSG_IDS_BODY_FIELD]; + const incomingSeqsRaw = incomingSeqsOf(parsed); + delete (parsed as unknown as Record)[MSG_SEQS_BODY_FIELD]; + delete (parsed as unknown as Record)[MSG_IDS_BODY_FIELD]; // #564/#2627: the ordering separators THIS turn's repair actually inserted, // recorded by the producer so provenance never has to be inferred from item // text later. Empty on the overwhelming majority of turns. @@ -176,6 +189,11 @@ export async function prepareResponses( resolveCompressProtocol(opts.routes, upstreamOrigin) === "marker"; const renderTags: "text-only" | "none" = knobRenderNone() || isCompactionTrigger ? "none" : "text-only"; + // #5874 P2-final: in-request snapshot flush verdict (set after processTurn, + // read at end-of-prepare to decide whether the debounce still needs a shot). + let snapFlush: { ok: boolean; epoch: number | undefined } | undefined; + // #7064 §8 phase 2: x-bili-msgid-mode signal (set after stage 0 below). + let msgidModeHeaders: Record | undefined; try { // [#1638] Position-preserve mid-history system/developer items for // plugin agents and native Codex. Both append developer @@ -245,8 +263,54 @@ export async function prepareResponses( sysNotes = outcome.notes; if (outcome.outbound !== fresh) projection.systemParts = outcome.outbound ? [outcome.outbound] : []; } - const { msgs } = projection; + let { msgs } = projection; originalMessages = msgs; + // #6181 2×2: stamp the M2 seqs carrier onto the converted array + // (parallel to input; malformed/absent → null → the M2 fast path is + // disabled, the full-resend world-match path still works). + const incomingSeqs: number[] | null = + incomingSeqsRaw !== undefined && incomingSeqsRaw.length === msgs.length && validateMsgSeqs(incomingSeqsRaw, msgs.length).ok + ? incomingSeqsRaw + : null; + if (incomingSeqs !== null) applyMsgSeqStamps(msgs, incomingSeqs); + // #6181 2×2 stage 0 (brief 19544722): the SHARED pin + world-match + // + kernel-input assembly (src/snapshot/stage0.ts — the SAME helper + // as the openai lane, same red gate). Before processTurn, so an R3 + // FoundationRejectError carries zero kernel side effects. + // #6564 note: the responses-wire pin that #6564 added inline here is + // SUBSUMED by runStage0 (same validateMsgIds+applyMsgIdPin under the + // pluginLaneServerHistory gate); the pinned `msgs` flows through the + // render below to the kernel/store. No double-pin. + const s0 = runStage0({ + sessionId, + lane: "responses", + msgs, + incomingMsgIds, + incomingSeqs, + pluginMode, + log, + }); + if (s0.pinned !== null) { + msgs = s0.pinned; + originalMessages = msgs; + } + // #7064 D3 (尾锚常活): the stage-0 cursor may have advanced BEFORE + // processTurn — bump the session revision so the per-turn snapshot + // flush below (snapFlush epoch proof) observes the advance. A fork + // taken between the cursor write and the flush otherwise reads a + // STALE parent revision and lands the child on the wrong lane. + if (s0.anchorAdvanced) markDirty(session); + // #7064 §8: the pin-mode signal (what the client may send NEXT + // turn) rides this lane's response headers — the responses lane + // has no watermark plane, so it is stamped independently of + // x-bili-pf-seq (the client's pfIdTailForm reads mode only). + // #6785 review fix: "incremental" is gated on the flag (the upgrade + // never travels while it is off); the "full" DOWNGRADE travels even + // when the flag is off — the rollback self-heal (an already + // upgraded client must be told to send the full form again). + if (s0.pinMode === "full" || (featureEnabled("pluginMsgIdIncremental") && s0.pinMode !== null)) { + msgidModeHeaders = { [MSGID_MODE_HEADER]: s0.pinMode }; + } // #3756 snapshot render: outbound prefix from the unified transcript // (fold-time ids, immune to identity re-mints); live tail keeps the // client's bytes. No-op when the lane is inactive/degraded; the @@ -383,6 +447,14 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const nudgeHeldAfterReject = session.transient?.nudgeHeldAfterReject === true; if (nudgeHeldAfterReject) restoreNudgeStamps(session, nudgeStampsBefore); adoptContentStore(session, turn.contentStore); + // In-request ledger flush: the transcript up to THIS user message is + // durable before the model call (#3397 durability-before-render) and + // the stage-0 envelope / M2 evidence read a current ledger. + // snapshotMessages first: the persist build reads session.lastMessages + // — and it stays the RAW client resend (never the D-lane view: its + // synthetic acp_summary anchor is not a transcript message). + snapshotMessages(session, originalMessages); + snapFlush = await SessionStore.flushOrSkip(session); // #4528: split the kernel pass from its post-processing scan passes so // each is a bounded synchronous stretch rather than one long block. await yieldToEventLoop(); @@ -462,7 +534,18 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr applyCompactionArchive(session, activeBefore, new Set(msgs.map((m) => m.id)), log); } reapOrphansLogged(session, msgs, log, sessionId); - rebuiltInput = patchResponsesInput(projection, processedMessages); + // #7398 all-lanes (ruling 7439-4): serve this turn's outbound core + // from the egress op-log before the item rebuild — the recorded bytes + // are this session's server truth. Compaction-generation turns + // (trigger/prompt form) are transient, like title-gen, and are not + // recorded nor served. + let laneCore = processedMessages; + if (!isCompactionTrigger && featureEnabled("egressPlane")) { + const st = getStore().egressOpLogStore(); + const served = st ? serveLaneCore(st, sessionId, log) : null; + if (served !== null) laneCore = served; + } + rebuiltInput = patchResponsesInput(projection, laneCore); if (Array.isArray(rebuiltInput)) rebuiltInput = mergeAdjacentConfigurationUpdates(hoistTrappedToolItems(rebuiltInput)); // #3400: the Responses steady path rebuilds items from the RAW parsed // input (patchOriginalItem), so BiliMessage-level modifications never @@ -509,6 +592,11 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr if (forgedSummaries.length > 0) log("debug", `[${sessionId}] [inject] ${forgedSummaries.length} captured summary block(s) re-injected into developer message`); rebuiltInput = injectResponsesDeveloperMessage(rebuiltInput, devContent, projection.headRole ?? "developer"); } + // #7398 all-lanes: record this turn's outbound core view (rollout-gated). + if (!isCompactionTrigger && featureEnabled("egressPlane")) { + const st = getStore().egressOpLogStore(); + if (st) recordLaneTurn(st, sessionId, laneCore, responsesDevContent ?? projection.systemParts.join("\n\n"), log); + } if (sysNotes.length > 0) { const inputItems: ResponseInputItem[] = typeof rebuiltInput === "string" ? [{ type: "message", role: "user", content: rebuiltInput }] @@ -602,6 +690,11 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr } transformOk = true; } catch (err) { + // #6181 B1: the R3 identity fault is a HARD reject, not a degrade — + // rethrow it to the server funnel (400 + x-bili-snapshot:identity) + // BEFORE the generic 503; zero kernel side effects by construction + // (it throws before processTurn). + if (err instanceof FoundationRejectError) throw err; log("warn", `[${sessionId}] kernel transform failed, forwarding unchanged: ${String(err)}`); processedMessages = []; } @@ -738,10 +831,12 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr const responsesPairOrigin = billingUpstream ?? upstreamOrigin; if (responsesPairOrigin) session.stats.lastLocalTextEstimateOrigin = responsesPairOrigin; } - snapshotMessages(session, originalMessages); // #3397 durability-before-render (see prepare-anthropic.ts): rows durable NOW, before the model call. getStore().ingestRowsNow(session); - markDirty(session); + // #5874 P2-final: the in-request snapshot flush above already persisted + // this revision; re-arm the debounce only if the session mutated since + // (stats probes land at the next flush) or the flush never landed. + if (!snapFlush || !snapFlush.ok || (session.revisionEpoch ?? 0) !== snapFlush.epoch) markDirty(session); // #564/#2627: ONE serialization of the outbound body. The same string is // both what we send and what the separator manifest binds to, so the claims // can never be attributed to a body that differs by even one byte. @@ -761,6 +856,7 @@ reconcileFoldCoverage(session, msgs, { mode: foldMode, sessionId, log, markerStr judgeBaseline, ...(separatorManifest !== undefined ? { separatorManifest } : {}), responsesProjection, + responseExtraHeaders: msgidModeHeaders, systemNotes: sysNotes, protocol: "responses", stream, @@ -791,11 +887,22 @@ export function prepareResponsesCompact( log: (level: string, msg: string) => void, ): Prepared { ++session.stats.requests; + // #6539 KDD #17: the msgid carriers ride the body on the PLUGIN-lane + // chained rounds that take THIS verbatim path (the main /responses + // pipeline strips them in prepareResponses; this function bypasses it, + // so the strip must live here too). Mutate `parsed` (same as the main + // pipeline) and force a rebuilt body whenever a field was present — + // the raw-buffer fallback would otherwise forward the carrier verbatim. + const hadMsgIdFields = + Object.prototype.hasOwnProperty.call(parsed, MSG_IDS_BODY_FIELD) || + Object.prototype.hasOwnProperty.call(parsed, MSG_SEQS_BODY_FIELD); + delete (parsed as unknown as Record)[MSG_IDS_BODY_FIELD]; + delete (parsed as unknown as Record)[MSG_SEQS_BODY_FIELD]; // A bili-forged compaction item is never for the upstream (it carries our // sentinel blob) — strip it on every forwarding path, same as the normal // /responses pipeline does. const cleaned = Array.isArray(parsed.input) ? stripBiliCompactionItems(parsed.input) : parsed.input; - const stripped = Array.isArray(parsed.input) && cleaned.length !== parsed.input.length; + const stripped = hadMsgIdFields || (Array.isArray(parsed.input) && cleaned.length !== parsed.input.length); const forgeBody: ResponsesRequestBody = { ...parsed, input: cleaned }; const base: Prepared = { body: stripped ? Buffer.from(JSON.stringify(forgeBody)) : body, diff --git a/src/server/snapshot-render.ts b/src/server/snapshot-render.ts index ecbc57ed68..6965512e45 100644 --- a/src/server/snapshot-render.ts +++ b/src/server/snapshot-render.ts @@ -68,18 +68,36 @@ const MIN_SEAM_RUN = 2; /** #3658 canonical identity signature minus the hash: the tuple the kernel * derives message ids from. Equal signature ⇒ same logical message even * when the id bytes were re-minted by an upgrade. Image cores share the - * "[image]" text, so the sidecar length rides the signature (same image ⇒ - * same bytes ⇒ same length; different images practically always differ). */ + * "[image]" text, so the image BYTES ride the signature (see #4390). */ export function identitySig(m: BiliMessage): string { const text = typeof m.text === "string" ? m.text : ""; // #4390: hash the image BYTES, never the length — same-size screenshots // used to collide and chain a false seam (silent truncation). The walk // only touches the tail region, so per-request hashing is bounded; a // 24-hex prefix is far beyond collision reach for seam evidence. - const img = m.imagePart === true ? `|img:${sha256Hex(m.imageBase64 ?? "").slice(0, 24)}` : ""; + const img = m.imagePart === true ? `|img:${imageSigPart(m.imageBase64)}` : ""; return `${m.role}|${m.contentType}|${m.toolName ?? ""}|${text}${img}`; } +/** The 24-hex image identity, robust to the media-OFFLOADED storage form: + * persisted rows carry {"$media": sha256Hex(bytes)} instead of the bytes + * (#3762), and that hash IS the identity — so both forms sign identically. + * Anything else degrades to a stable stringify. Never throws: a signature + * helper throwing takes the whole seam walk (and turn) down. */ +function imageSigPart(v: unknown): string { + if (typeof v === "string") return v.length > 0 ? sha256Hex(v).slice(0, 24) : "none"; + if (typeof v === "object" && v !== null) { + const h = (v as { $media?: unknown }).$media; + if (typeof h === "string" && h.length >= 24) return h.slice(0, 24); + try { + return `s${sha256Hex(JSON.stringify(v) ?? "").slice(0, 24)}`; + } catch { + return "unserializable"; + } + } + return "none"; +} + /** Test seam (#4390): the seam fingerprint under test — see identitySig. */ export function identitySigForTest(m: BiliMessage): string { return identitySig(m); @@ -174,11 +192,12 @@ function loadTranscriptBase(session: Session): TranscriptBase | { reason: string // bytes never ride the outbound and are never read. A covered row // with corrupt bytes therefore cannot degrade the render. let rows: ViewRow[] = []; + let bySeq = new Map(); try { const light = store.unifiedMessageIdRows(session.id); const seqs = store.unifiedTailSeqs(session.id); if (!light || light.length === 0 || !seqs || seqs.length === 0) return { reason: "transcript-empty" }; - const bySeq = new Map(); + bySeq = new Map(); for (const row of light) bySeq.set(row.seq, { seq: row.seq, msgId: row.msgId }); for (const seq of seqs) { const v = bySeq.get(seq); @@ -190,6 +209,7 @@ function loadTranscriptBase(session: Session): TranscriptBase | { reason: string } if (rows.length === 0) return { reason: "transcript-empty" }; + // #4052: the covered-region map must exist BEFORE the chain — the signature // fallback must not offer covered rows as bind targets (see the bySig filter // below). Per-block covered seqs seed from effectiveMessageIds (exact diff --git a/src/snapshot/identity.ts b/src/snapshot/identity.ts new file mode 100644 index 0000000000..b185d9ebac --- /dev/null +++ b/src/snapshot/identity.ts @@ -0,0 +1,252 @@ +/** + * #5874 — snapshot-table compression foundation: stage 0 (world cursor). + * + * #7064 msgid-storage slimming (owner ruling 2026-11-17) reshaped this file: + * the standing state is a BOOKKEEPING CURSOR — the head + tail id windows of + * the last seen host array plus its count and the append-only pos frontier — + * not a full canon. The middle is deliberately invisible: + * + * - D1 (永不判分叉): an id verdict NEVER diverges. Client compression + * removes ids freely; a head drop is a `shrink`, a middle rewrite or a + * head swap is a `reanchor` (re-issue the cursor). The worst outcome is + * one cheap re-issue — never a reject. Middle-TAMPER detection is the + * ledger's job ((msg_id, identity) probes), not the cursor's. + * - D2: sessions with a host conversation identity skip the id verdict + * entirely (`anchor` mode — the cursor follows the incoming + * unconditionally). + * - D3: the cursor refreshes on watermark-hit turns too (the hit view is + * the same canonical id space — the envelope is space-stable). + * + * pos stays APPEND-ONLY (never reused): a re-anchored id is the SAME id at + * the same pos (it survives the re-anchor by identity), only genuinely new + * ids draw fresh pos from maxPos+1. + */ + +import type { WorldDecision, WorldMode, WorldWindow } from "./types.ts"; + +/** Tail window width (the anchor chain). ~32 KiB JSON at ~60-char ids — the + * long-session cost this slimming exists to bound. */ +export const WORLD_TAIL_K = 512; +/** Head window width (the origin anchor — fork/resume-parent confusion is + * only detectable at the head; a small number suffices). */ +export const WORLD_HEAD_K = 32; + +interface WorldMatchInput { + /** The stored world anchor for this sess_raw, or null when none exists. */ + stored: WorldWindow | null; + /** The stored world origin (issued_at); 0 when no stored world. */ + issuedAt: number; + /** The incoming host ids (host order) — full-resend lanes send the whole + * array; M2 incremental sends only the new tail (matchAppendWindow). */ + incoming: string[]; + /** The raw session key (host value, never truncated, P9). */ + sessRaw: string; + /** The bili canonical ledger identity to issue when no world exists + * (ruling D: the existing sid keyspace). */ + ledgerId: string; + now: number; + /** True when this turn is an explicit re-baseline (mode switch). */ + resync?: boolean; +} + +function sameArr(a: readonly string[], b: readonly string[]): boolean { + if (a.length !== b.length) return false; + for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) return false; + return true; +} + +/** Cut the (tail, head) windows of an array. head = the first min(HEAD_K, n) + * ids; tail = the last min(TAIL_K, n) ids. */ +export function windowOf(ids: readonly string[], maxPos: number): WorldWindow { + return { + tailIds: ids.slice(-WORLD_TAIL_K), + headIds: ids.slice(0, WORLD_HEAD_K), + count: ids.length, + maxPos, + }; +} + +/** Count the longest run of stored-tail elements that sit at the START of + * the incoming array — the chain-containment measure. Pure extension + * (incoming continues the stored array) → the whole stored tail counts; + * equality (echo) → also the whole tail; a head swap or a middle rewrite + * caps the run at the first mismatch. (A tail-vs-tail overlap would count + * the same arrays and miss every pure extension — the verdict the W3 + * round trip needs.) */ +function prefixContainLen(storedTail: readonly string[], incoming: readonly string[]): number { + const cap = Math.min(storedTail.length, incoming.length); + let k = 0; + for (let i = 0; i < cap; i++) { + if (storedTail[i] !== incoming[i]) break; + k++; + } + return k; +} + +/** Match the incoming host-id array against the stored cursor. Pure — returns + * the verdict; persistence is the caller's. D1: NO verdict rejects; the + * strongest move is `reanchor` (re-issue from the incoming). */ +export function matchWindow(input: WorldMatchInput): WorldDecision { + const incoming = Array.from(input.incoming); + const fresh = (mode: WorldMode, witness?: string): WorldDecision => { + const n = incoming.length; + return { + mode, + window: windowOf(incoming, n), + newIdPos: incoming.map((id, i) => ({ id, pos: i + 1 })), + goneIds: [], + issuedLedgerId: input.ledgerId, + issuedAt: input.now, + witness, + }; + }; + + // Explicit re-baseline always re-issues (the mode changed). + if (input.resync) return fresh("resync"); + // No world yet: issue (one arming turn is charged at the session level). + if (input.stored === null) return fresh("issue"); + + const s = input.stored; + const incomingSet = new Set(incoming); + // Windowed gone-set: stored TAIL ids absent from the incoming (only what + // the window can see — the middle is invisible by design). + const goneIds = s.tailIds.filter((id) => !incomingSet.has(id)); + + // No overlap at all → total re-mint → re-issue (D1: bookkeeping). + if (s.tailIds.length === 0 || incoming.length === 0) { + return { + mode: "reanchor", + window: windowOf(incoming, s.maxPos + incoming.length), + newIdPos: incoming.map((id, i) => ({ id, pos: s.maxPos + i + 1 })), + goneIds, + issuedAt: input.issuedAt, + witness: `no overlap (stored ${s.count}, incoming ${incoming.length})`, + }; + } + + const k = prefixContainLen(s.tailIds, incoming); + // The stored anchor fully contained in the incoming tail: the chain + // extends (or holds). + if (k >= s.tailIds.length) { + // head + count + tail all agree → nothing moved (the middle is + // invisible by design — head digest + count is the strongest claim + // the windows can make). + if (incoming.length === s.count && sameArr(s.headIds, incoming.slice(0, WORLD_HEAD_K))) { + return { + mode: "echo", + window: s, + newIdPos: [], + goneIds: [], + issuedAt: input.issuedAt, + }; + } + // Extension: ids beyond the stored array length are new. The count + // cannot shrink across a full chain containment, so pos accounting + // is exact. + const newIdPos = incoming.slice(s.count).map((id, i) => ({ id, pos: s.count + i + 1 })); + return { + mode: "extend", + // The head must track the INCOMING array — the next M1 full + // resend re-compares against it (the W11 round trip): preserving + // the stored head here would never echo again. + window: { ...windowOf(incoming, s.maxPos), maxPos: s.maxPos + newIdPos.length }, + newIdPos, + goneIds, + issuedAt: input.issuedAt, + }; + } + // The whole incoming is a SUFFIX of the stored tail → the client dropped + // head elements (a fold). D1: legal shrink, never divergent. + if (incoming.length < s.count && incoming.every((id, i) => s.tailIds[s.tailIds.length - incoming.length + i] === id)) { + return { + mode: "shrink", + window: windowOf(incoming, s.maxPos), + newIdPos: [], + goneIds, + issuedAt: input.issuedAt, + witness: `head drop: ${s.count} -> ${incoming.length}`, + }; + } + // No anchor relation (partial overlap / head swap / middle rewrite / + // re-mint): re-issue the cursor. Ids that SURVIVE keep their pos via the + // stored tail; everything else is counted as new (pos is bookkeeping — + // M2 contiguity anchors on ledger seqs, not on this counter). + // pos reconstruction: the tail ids hold maxPos-tailLen+1 .. maxPos in + // order — the only positions the window knows exactly. + const storedPos = new Map(); + const base = s.maxPos - s.tailIds.length + 1; + for (let i = 0; i < s.tailIds.length; i++) storedPos.set(s.tailIds[i], base + i); + let pos = s.maxPos; + const newIdPos: Array<{ id: string; pos: number }> = []; + for (const id of incoming) { + const p = storedPos.get(id); + if (p === undefined) { + pos += 1; + newIdPos.push({ id, pos }); + } + } + return { + mode: "reanchor", + window: { ...windowOf(incoming, pos), maxPos: pos }, + newIdPos, + goneIds, + issuedAt: input.issuedAt, + witness: `chain broke (overlap ${k}/${s.tailIds.length}; stored ${s.count}, incoming ${incoming.length})`, + }; +} + +/** + * #5874 2×2 M2 INCREMENTAL window: the client sends ONLY the new elements + * (with msgids) — the caller has ALREADY located them at the ledger frontier + * (contiguous seqs starting exactly after maxSeq, all ids unknown). The new + * cursor = stored cursor ++ the new tail; the head is unchanged. + */ +export function matchAppendWindow(stored: WorldWindow, incoming: string[]): WorldDecision { + const ids = Array.from(incoming); + const all = [...stored.tailIds, ...ids]; + const prior = new Set(stored.tailIds); + const newIdPos = ids + .map((id, i) => ({ id, pos: stored.maxPos + i + 1, fresh: !prior.has(id) })) + .filter((e) => e.fresh) + .map(({ id, pos }) => ({ id, pos })); + return { + mode: "extend", + window: { + // tail tracks the COMBINED array — the next M1 full resend + // re-compares against it (the W11 round trip). The head: while + // the array is <= WORLD_HEAD_K the head IS the whole array, so it + // extends with the appends; past K the true head is not in + // stored.tailIds and re-slicing `all` would store a + // middle-of-array slice as head (PR#6785 review fix — that broke + // the echo gate at identity.ts:134 for every >WORLD_TAIL_K + // session after any M2 turn: perpetual re-anchor + window + // upsert instead of the steady-state no-write). + tailIds: all.slice(-WORLD_TAIL_K), + headIds: stored.count <= WORLD_HEAD_K ? all.slice(0, WORLD_HEAD_K) : stored.headIds, + count: stored.count + ids.length, + maxPos: stored.maxPos + ids.length, // M2 appends sequentially — every appended id takes the next pos, known or not + }, + newIdPos, + goneIds: [], + issuedAt: 0, // the caller preserves the stored origin + }; +} + +/** Parse a LEGACY persisted envelope (`JSON.stringify({id,pos}[])` — the + * pre-slimming l1_world.envelope_json) into the new window shape. Used by + * the schema migration ONLY (fresh rows store the window columns directly). + * Throws on malformed data (the migration degrades that row to a blank + * cursor instead of crashing the open). */ +export function parseLegacyEnvelopeToWindow(envelopeJson: string): WorldWindow { + const parsed: unknown = JSON.parse(envelopeJson); + if (!Array.isArray(parsed)) throw new Error("legacy envelope is not an array"); + const ids: string[] = []; + let maxPos = 0; + for (const e of parsed) { + const entry = e as { id?: unknown, pos?: unknown }; + if (typeof entry.id !== "string") throw new Error("legacy envelope entry is not {id, pos}"); + ids.push(entry.id); + if (typeof entry.pos === "number" && entry.pos > maxPos) maxPos = entry.pos; + } + return windowOf(ids, maxPos || ids.length); +} diff --git a/src/snapshot/plane.ts b/src/snapshot/plane.ts new file mode 100644 index 0000000000..0d80a54fb5 --- /dev/null +++ b/src/snapshot/plane.ts @@ -0,0 +1,305 @@ +/** + * #5874 / 2×2 — stage 0 (identity / world match) decision glue. + * + * Sits at the head of the chain (raw bytes → **stage 0** → protocol + * conversion → kernel → egress). It runs the pure `matchWindow` + * (identity.ts) on the l1_world world cursor and carries the R3 identity + * fault policy (red gate R3): + * + * - a cohort host that DECLARED the dual-channel (per-element msgids) but + * did not collect them all → the request is HARD-REJECTED (400) BEFORE + * any table/kernel side effect, and the fault feeds a fastpath-style + * breaker (10-minute window, 3 strikes → a persistent per-sid latch that + * keeps rejecting until an operator clears it). #7064 D1/D2: the fault + * path NEVER fires for CONVERSATION-BOUND sessions (the host conversation + * identity IS the identity) — the cursor is bookkeeping there. + * + * The decision itself is pure (identity.ts); this module adds the storage + * glue (read the stored cursor, persist the new window) — and only for + * verdicts that moved the cursor (echo/replay persist nothing: no write per + * steady-state turn). The gone-set and the ledger frontier are COMPUTED by + * the store (messages × blocks coverage, max live seq) and passed in: the + * pure core never touches a table (pure-incremental redesign: there is no + * per-row plane to read). + */ +import { matchAppendWindow, matchWindow, windowOf } from "./identity.ts"; +import { snapL1Lookup, snapL1WindowRead, snapL1WindowUpsert } from "../storage/snap.ts"; +import type { DbLike } from "../storage/snap.ts"; +import type { WorldDecision } from "./types.ts"; + +/** R3 identity fault — the lane rejects the request with HTTP 400 and takes + * NO side effect (no ledger write beyond the R1 mapping, no kernel run). */ +export class FoundationRejectError extends Error { + readonly status = 400; + constructor(public readonly reason: string) { + super(`[snapshot] R3 identity fault: ${reason}`); + this.name = "FoundationRejectError"; + } +} + +/** R3 breaker: a per-process fault window (10 minutes) with a strike cap + * (3). On the third strike within the window the sid gets a PERSISTENT + * latch (the caller persists it) and every subsequent R3 fault for that + * sid rejects immediately without re-counting. Mirrors the + * incremental-lane breaker discipline (the R3 reference). */ +export class R3FaultTracker { + private readonly faults = new Map(); + private readonly latched = new Set(); + constructor( + private readonly windowMs: number = 10 * 60 * 1000, + // readonly-public: the pre-latch strike log (stage0.ts) cites the + // cap in its message. + public readonly strikeCap: number = 3, + ) {} + + /** Record an R3 fault. Returns the persistent-latch verdict. */ + recordFault(sid: string, now: number = Date.now()): { latched: boolean } { + if (this.latched.has(sid)) return { latched: true }; + let entry = this.faults.get(sid); + if (!entry || now - entry.firstAt > this.windowMs) { + entry = { count: 0, firstAt: now }; + this.faults.set(sid, entry); + } + entry.count += 1; + if (entry.count >= this.strikeCap) { + this.latched.add(sid); + return { latched: true }; + } + return { latched: false }; + } + + /** Strikes recorded in the CURRENT window (a cleared or expired window + * reads 0) — the pre-latch 400s log their progress toward the latch so + * an operator sees a miscounting host BEFORE the sid latches for good. */ + strikes(sid: string, now: number = Date.now()): number { + const entry = this.faults.get(sid); + if (!entry || now - entry.firstAt > this.windowMs) return 0; + return entry.count; + } + + isLatched(sid: string): boolean { + return this.latched.has(sid); + } + + /** Operator escape hatch (paired with the persisted-latch clear at the + * `/__bili/snapshot/r3-unlatch` admin route — that route calls this for + * every sid the latched row can resolve to, since the in-memory latch + * alone would survive a same-process "clear" that misses its key). */ + clearLatched(sid: string): void { + this.latched.delete(sid); + this.faults.delete(sid); + } +} + +/** The process-wide R3 fault tracker (one per process; the lane consults it + * before every stage-0 run and records on every fault). Exported so tests + * can inspect/reset it. */ +export const r3FaultTracker = new R3FaultTracker(); + +/** The stage-0 input: the raw identity channel for one turn. + * @param db the storage db (snap.ts DbLike surface). + * @param sessRaw the raw host session id (P9: never truncated). + * @param ledgerId the bili canonical ledger identity (the sid). + * @param incoming the per-element msgids in host order. + * @param resync true for an explicit re-baseline (mode switch). + * @param maxSeq the ledger frontier: the max live stamped seq + * (store-computed; 0 for an empty ledger). The M2 + * contiguity evidence anchors on it. + * @param explainedGone the ids that legally vanished from the envelope + * head — stored messages covered by a live fold + * block (store-computed from messages × blocks). + * @param now clock (injectable for tests). + */ +interface Stage0Input { + db: DbLike; + sessRaw: string; + ledgerId: string; + incoming: string[]; + /** The `biliLedgerSeqs` carrier (M2 position locating, #4316 layer-3 + * transport). When it is contiguous and starts exactly after the + * ledger frontier, an ALL-UNKNOWN incoming is a pure M2 append + * (matchAppend); without it, an all-unknown incoming is + * treated as a full (possibly re-minted) envelope → the normal + * matchWorld verdict (usually diverge → resync, one-round + * convergence). */ + seqs?: number[] | null; + resync?: boolean; + maxSeq: number; + explainedGone: Set; + /** D2 (#7064): the session carries a host conversation identity — the + * per-turn id verdict is skipped entirely (the cursor follows the + * incoming unconditionally; no replay guard, no R3 fault). */ + convBound?: boolean; + now?: number; +} + +export interface Stage0Result { + /** The pure world-match verdict (identity.ts). */ + decision: WorldDecision; + /** The fold-explained gone set — exposed for the reconcile and tests. */ + explainedGone: Set; +} + +/** Run stage 0: read the stored world cursor, match the incoming ids + * against it (with the store-computed gone set + frontier), and persist + * the new window whenever it moved (echo/replay write nothing). #7064 + * D1: NO verdict rejects or resets — the strongest move is `reanchor` + * (re-issue the cursor from the incoming, one cheap UPDATE). + * + * The pos values the verdict issues are the LOGICAL append-only cursor + * positions (maxPos+1..); the physical ledger seq is assigned by the + * ingest in the same incoming order, so the two agree for tail-appends. + * For scatter (middle insertion) the ingest re-homes in the same + * transaction. */ +export function stage0Decide(input: Stage0Input): Stage0Result { + const now = input.now ?? Date.now(); + const l1 = snapL1Lookup(input.db, input.sessRaw); + const read = snapL1WindowRead(input.db, input.sessRaw); + const explainedGone = input.explainedGone; + const planeMaxSeq = input.maxSeq; + + const storedWindow = l1 && read !== null && read.window !== null ? read.window : null; + const storedIssuedAt = l1 ? (read !== null ? read.issuedAt : l1.issuedAt) : 0; + + let decision: WorldDecision; + const seqs = input.seqs ?? null; + const storedSet = storedWindow !== null + ? new Set([...storedWindow.headIds, ...storedWindow.tailIds]) + : null; + + // D2 (#7064): conversation-bound sessions skip the id verdict — the + // host conversation identity IS the identity; the cursor follows the + // incoming unconditionally (bookkeeping, never a reject). + if (input.convBound) { + const known = storedWindow !== null + ? new Map(storedWindow.tailIds.map((id, i) => [id, storedWindow.maxPos - storedWindow.tailIds.length + 1 + i])) + : null; + const incomingSet = new Set(input.incoming); + let pos = storedWindow?.maxPos ?? 0; + const newIdPos: Array<{ id: string; pos: number }> = []; + for (const id of input.incoming) { + if (known === null || known.get(id) === undefined) { + pos += 1; + newIdPos.push({ id, pos }); + } + } + // #6785 review (owner): an idempotent re-render (every incoming id + // already windowed) moves the cursor NOWHERE — issue the same + // zero-side-effect echo the non-bound replay path uses instead of + // re-anchoring + upserting the unchanged row on every such turn. + if (newIdPos.length === 0 && storedWindow !== null && input.incoming.length > 0 && !input.resync) { + decision = { + mode: "echo", + replay: true, + window: storedWindow, + newIdPos: [], + goneIds: [], + issuedAt: storedIssuedAt, + witness: "echo: conversation-bound, all ids known (cursor unchanged)", + }; + } else { + decision = { + mode: "anchor", + window: { ...windowOf(input.incoming, 0), maxPos: pos }, + newIdPos, + goneIds: storedWindow !== null ? storedWindow.tailIds.filter((id) => !incomingSet.has(id)) : [], + issuedAt: storedIssuedAt || now, + witness: "anchor: conversation-bound, id verdict skipped", + }; + } + } else { + // Review-round-2 Residual 1b: the host re-sent a body this proxy + // already processed — every id is already in the stored windows and + // the carrier seqs (when present) sit at/below the ledger frontier + // (the original turn's in-request flush stamped them before its + // response left). Zero side effects: the cursor is kept as-is, the + // lane re-renders the SAME world (an idempotent kernel round). + const replay = + storedWindow !== null && + storedSet !== null && + !input.resync && + input.incoming.length > 0 && + input.incoming.every((id) => storedSet.has(id)) && + (seqs === null || seqs.every((s) => s <= planeMaxSeq)); + if (replay) { + decision = { + mode: "echo", + replay: true, + window: storedWindow, + newIdPos: [], + goneIds: [], + issuedAt: storedIssuedAt, + witness: `replay: all ${input.incoming.length} ids known (maxSeq ${seqs ? Math.max(...seqs) : "-"} <= frontier ${planeMaxSeq})`, + }; + } else { + // M2 pure append: the client sent ONLY new elements — every + // incoming id is unknown to the windows — and the seqs carrier + // says they land as a contiguous run starting exactly after the + // ledger frontier. That is the 世界匹配定位序列位置 evidence: + // issue append-only pos (maxPos+1..) without any head/order + // risk. Anything less certain (missing seqs, a gap, a known id + // mixed in) falls through to matchWindow — D1: the worst + // outcome there is a reanchor, one cheap re-issue. + const m2Candidate = + storedWindow !== null && + !input.resync && + input.incoming.length > 0 && + seqs !== null && + seqs.length === input.incoming.length; + let contig = m2Candidate && seqs![0] === planeMaxSeq + 1; + if (contig) { + for (let i = 1; i < seqs!.length; i++) { + if (seqs![i] !== seqs![i - 1] + 1) { + contig = false; + break; + } + } + } + if (m2Candidate && contig && input.incoming.every((id) => !storedSet!.has(id))) { + const appended = matchAppendWindow(storedWindow, input.incoming); + decision = { ...appended, issuedAt: storedIssuedAt }; + } else { + decision = matchWindow({ + stored: storedWindow, + issuedAt: storedIssuedAt || now, + incoming: input.incoming, + sessRaw: input.sessRaw, + ledgerId: input.ledgerId, + now, + resync: input.resync, + }); + } + } + } + + // Persist the window whenever it MOVED (echo/replay: no write per + // steady-state turn — the cursor is unchanged by construction). + if (decision.mode !== "echo") { + snapL1WindowUpsert(input.db, input.sessRaw, decision.window); + } + + return { decision, explainedGone }; +} + +/** The R3 identity check for a cohort host that DECLARED the dual-channel: + * the per-element msgid list must cover EVERY element of the body. A short + * (or empty, when the body is non-empty) collection is the identity gate + * fault → the caller throws FoundationRejectError (400, zero side effects) + * and records the fault on the tracker. + * + * @param declared the host profile declares per-element msgids (cohort). + * @param elementCount the body element count (the msgs to be parsed). + * @param msgids the collected per-element msgids (null/undefined = + * the host sent none). + * @returns true when the turn may proceed (non-cohort, or full collection). + */ +export function checkR3Identity( + declared: boolean, + elementCount: number, + msgids: string[] | null | undefined, +): boolean { + if (!declared) return true; // non-cohort: sticky order>hash degrade + if (elementCount === 0) return true; + if (!msgids || msgids.length !== elementCount) return false; + return true; +} diff --git a/src/snapshot/stage0.ts b/src/snapshot/stage0.ts new file mode 100644 index 0000000000..a672e0c5d2 --- /dev/null +++ b/src/snapshot/stage0.ts @@ -0,0 +1,415 @@ +// #5874 2×2 stage 0 (#6181): identity gate + world cursor for cohort +// (dual-channel DECLARED) plugin traffic. SHARED by the openai and +// responses lanes so both carry the same red gate and the same +// one-round-convergence contract. +// +// Runs BEFORE processTurn in the lane, so a thrown FoundationRejectError +// (R3) carries ZERO kernel side effects. Every other fault → the legacy +// full path (degrade, never guess — one-round convergence). +// +// Contract (#7064 slimming, owner ruling 2026-11-17): +// - armed = pluginMode + dual-channel DECLARED (msgids present) + the +// pluginLaneServerHistory rollout (M1: the red gate must be unreachable +// wherever the pin is unreachable). D3: the arm no longer excludes +// watermark-HIT turns — the cursor refreshes on hit turns too (the hit +// view is the same canonical id space; the tail anchor stays live). +// - R3: a FULL-LENGTH declared array that fails entry validation +// (bad-entry / reserved / control / duplicate) → checkR3Identity is the single arbiter +// (M2: it is CALLED here, in both lanes); an unlatched fault throws +// FoundationRejectError → the server maps it to 400 (B1); 3 faults in a +// 10-minute window latch the sid to the legacy path and the latch +// PERSISTS across proxy restarts (snapR3Latch). D2: the fault path +// NEVER fires for conversation-bound sids — the host conversation +// identity IS the identity, the cursor is bookkeeping there. +// #7064 §8 phase 2 (owner ruling 2026-11-18): a SHORT declared array +// is now a LEGAL tail form (steady-state incremental — the channel +// carries anchor + new cores), so "declared but short" is a drift class +// that degrades to the full signal and NEVER 400s; a genuinely broken +// host re-converges (full form) and is caught there if its full list +// is also wrong (D1: never reject drift). +// - D1 永不判分叉: the stage-0 verdict NEVER rejects or resets — the +// strongest move is `reanchor` (re-issue the world cursor from the +// incoming array, one cheap UPDATE). The pre-slimming diverge/envelope +// reset path is gone: a structural disagreement re-anchors in the same +// turn and the NEXT same-array turn echoes (one-round convergence, now +// a property of the cursor, not a lane reset). + +import type { BiliMessage } from "acp-kernel/wire"; +import { getStore } from "../persist.js"; +import { featureEnabled } from "../rollout.js"; +import { applyMsgIdPin, validateMsgIds } from "../msgid-pin.js"; +import { checkR3Identity, FoundationRejectError, r3FaultTracker } from "./plane.js"; + +export interface Stage0Input { + sessionId: string; + /** Lane name for log prefixes only ("openai" | "responses"). */ + lane: string; + /** The converted array for this turn (seq-stamped by the lane). */ + msgs: BiliMessage[]; + /** The raw declared biliMsgIds field value (whatever shape it had). */ + incomingMsgIds: unknown; + /** Validated ledger seqs (null when absent/mismatched — the lane + * already applied them to msgs when non-null). */ + incomingSeqs: number[] | null; + /** #7064 §8: the decoded x-bili-pf-tail-ids value — three-state, exactly + * like the fastpath's parsedTailIds (undefined = header absent, null = + * malformed, array = valid, possibly containing "?" slots). The lane + * parses it; this path only pins from it when the FULL form is absent. + * Positioning contract: element 0 ⇔ the turn's LAST prefix core (the + * anchor), the rest are the NEW cores — the same shape the client's + * tail-ids arithmetic produces (core-aligned id list sliced at the + * last measured T, openai wire). */ + incomingTailIds?: string[] | null; + pluginMode: boolean; + /** The openai fast-path watermark hit; the responses lane has none. + * (D3: a hit turn still refreshes the cursor — this field is now + * informational for logs only.) */ + wpHit?: unknown; + log: (level: string, msg: string) => void; +} + +export interface Stage0Result { + /** Pinned msg array when the pin applied — the lane adopts it + * (`msgs = pinned; originalMessages = msgs`). null otherwise. */ + pinned: BiliMessage[] | null; + /** True when the cohort armed and RAN (gate passed) — the lane keeps + * its per-turn guards keyed on it. */ + cohortRan: boolean; + /** True when the world cursor MOVED this turn (extend/reanchor/ + * shrink/issue/resync/anchor — anything but a plain echo). The lane + * bumps the session's revisionEpoch so the per-turn snapshot flush + * (snapFlush epoch proof) observes the stage-0 advance — a fork + * taken between the cursor write and the flush otherwise reads a + * STALE parent revision (plugin-fork lane regression, #6785 WIP). */ + anchorAdvanced: boolean; + /** #7064 §8: what the client may send NEXT turn (stamped on the + * response as x-bili-msgid-mode, only when the flag is on): "full" + * — send the full id list (bootstrap / reanchor / this turn degraded + * / the flag is off and only the full pin runs); "incremental" — the + * ledger pins the prefix, ride only x-bili-pf-tail-ids. null = neither + * form was attempted (no field, no valid tail-ids) — no signal. */ + pinMode: "full" | "incremental" | null; +} + +export function runStage0(opts: Stage0Input): Stage0Result { + const { sessionId, lane, msgs, incomingMsgIds, incomingSeqs, incomingTailIds, pluginMode, wpHit, log } = opts; + void wpHit; // D3: the arm no longer keys on the hit state (logs only) + const rollout = featureEnabled("pluginLaneServerHistory"); + const declared = Array.isArray(incomingMsgIds) && incomingMsgIds.length > 0; + // #7064 §8 phase 2 (owner 2026-11-18): TWO-FORM rule on the declared + // channel — the SAME field carries either shape and the length + // discriminates: len === view = FULL list (bootstrap / reanchor), + // len !== view = TAIL form (anchor + new cores, steady state). No + // per-lane splitting: the handle.ts funnel puts body, header and + // harvest sources into this one field on every wire, so responses / + // anthropic clients go incremental with NO new server channel — they + // just stop sending the full list in steady state (the client derives + // the tail from its own previous full list, pfIdTailForm). + const declaredIds = Array.isArray(incomingMsgIds) ? (incomingMsgIds as string[]) : null; + const fullForm = declaredIds !== null && declaredIds.length === msgs.length; + const bodyTail = declaredIds !== null && !fullForm && declaredIds.length > 0 ? declaredIds : null; + // #7064 §8: the incremental pin is its OWN rollout on top of the lane's + // (the full-list pin below is untouched). Off = byte-identical to + // pre-§8 behavior. + const incrementalOn = rollout && pluginMode && featureEnabled("pluginMsgIdIncremental"); + // M1: the arm rides the SAME rollout as the pin — under the default + // (rollout off) the whole 2×2 path is unreachable, so R3 is not dead + // code waiting for a rollout that never arms it. #7064 D3: the + // wpHit === undefined exclusion is GONE — the cursor refreshes on + // hit turns too (尾锚常活: the hit view is the same canonical id + // space, so the tail anchor stays live across hit runs). + const armed0 = rollout && pluginMode && declared; + const persistedLatch = (() => { + try { + return getStore()?.snapR3Latched(sessionId) ?? false; + } catch { + return false; // a store fault must not change the arming verdict + } + })(); + let armed = armed0 && !r3FaultTracker.isLatched(sessionId) && !persistedLatch; + if (armed0 && !armed) { + log("debug", `[${sessionId}] [2x2:${lane}] sid is R3-latched — legacy path this turn`); + } + let pinned: BiliMessage[] | null = null; + // #7064 §8: the pin-mode signal (see Stage0Result.pinMode). Set wherever + // EITHER declared form arrives; the full form ALWAYS wins when present + // (bootstrap / reanchor must be able to run regardless of mode). + let pinMode: Stage0Result["pinMode"] = null; + // #6785 owner ruling (2026-11-17): the pin runs on watermark-hit turns TOO. + // The fork lane matches by host msgid against the ledger's rawIds — a + // hit turn that skipped the pin wrote h_ rows the child's host ids can + // never equal, and such parents are un-forkable by identity. The stamped + // array is captured pre-watermark (prepare-openai:83) and aligns to the + // FULL view length on hit exactly as on miss, so validation and the pin + // are unchanged; only the R3 consequences stay behind the `armed` gate. + if (pluginMode && declaredIds !== null) { + if (rollout) { + if (fullForm) { + const v = validateMsgIds(declaredIds, msgs.length); + if (v.ok) { + pinned = applyMsgIdPin(msgs, v.ids); + } else if (armed) { + // #7064 D2: conversation-bound sids never take the R3 fault + // path — the host conversation identity IS the identity and + // the cursor is bookkeeping; a short/missing msgid array + // degrades to content-derived ids (the log below) instead of + // 400/latch. + const convBound = (() => { + try { + return getStore()?.snapConvBound(sessionId) ?? false; + } catch { + return false; // a store fault must not OPEN the fault path + } + })(); + if (convBound) { + log("warn", `[${sessionId}] [msgid] ${lane}-wire pin rejected (${v.reason}) — conversation-bound sid skips the R3 fault, content-derived ids this turn`); + } else if (!checkR3Identity(true, msgs.length, Array.isArray(incomingMsgIds) ? (incomingMsgIds as string[]) : null)) { + // R3: a cohort host that DECLARED the dual-channel but + // did not collect every element → hard 400 BEFORE any + // kernel side effect; the fault feeds the breaker + // (10-minute window, 3 strikes → the sid latches to the + // legacy path, PERSISTED across restarts (M3), instead + // of bricking the session on a misbehaving host). + const { latched } = r3FaultTracker.recordFault(sessionId); + if (latched) { + armed = false; + try { + getStore()?.snapR3Latch(sessionId); + } catch { /* the in-memory latch already fires; persistence is best-effort */ } + log("error", `[${sessionId}] [2x2:${lane}] R3 identity fault ×3 in window (${v.reason}) — sid latched to the legacy path (persisted)`); + } else { + // The pre-latch strikes are otherwise SILENT — the + // 400 funnel (server.ts) does not log — so surface + // the progress toward the latch with the identity + // verdict: an operator needs to see a miscounting + // host BEFORE its sid latches to the legacy path + // for good (persisted across restarts). + const strike = r3FaultTracker.strikes(sessionId); + log("warn", `[${sessionId}] [2x2:${lane}] R3 identity fault — strike ${strike}/${r3FaultTracker.strikeCap} in window (${v.reason}) — 400 sent; the sid latches to the legacy path after ${r3FaultTracker.strikeCap - strike} more`); + throw new FoundationRejectError(`inbound msgid envelope failed validation (${v.reason})`); + } + } else { + log("warn", `[${sessionId}] [msgid] ${lane}-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#5825)`); + } + } else { + log("warn", `[${sessionId}] [msgid] ${lane}-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#5825)`); + } + } else { + // TAIL form on the declared channel (responses / anthropic + // steady state, or the openai body field carrying the tail): + // anchor + new cores, the ledger pins the prefix. Same + // drift-class contract as the pf-tail-ids branch — ANY + // mismatch degrades to the full signal, never rejects. + // Read the ledger ONCE here: it decides the gate below AND + // feeds the pin (single source of truth for C); a store + // fault reads empty (conservative: takes the FAULT side). + let liveIds: string[] = []; + try { + liveIds = getStore()?.snapMsgIdLiveIds(sessionId) ?? []; + } catch { + liveIds = []; + } + // EXCEPTION: a short array on an EMPTY ledger is not drift — + // there is no frontier to anchor against, so it is a failed + // collection on an unknown session: the legacy R3 fault + // applies (B1/D2 semantics: 400 for a non-bound sid). + if (liveIds.length === 0 && bodyTail !== null) { + const v = validateMsgIds(bodyTail, msgs.length); + if (!v.ok && armed) { + const convBound = (() => { + try { + return getStore()?.snapConvBound(sessionId) ?? false; + } catch { + return false; + } + })(); + if (convBound) { + log("warn", `[${sessionId}] [msgid] ${lane}-wire pin rejected (${v.reason}) — conversation-bound sid skips the R3 fault, content-derived ids this turn`); + } else if (!checkR3Identity(true, msgs.length, bodyTail)) { + const { latched } = r3FaultTracker.recordFault(sessionId); + if (latched) { + armed = false; + try { + getStore()?.snapR3Latch(sessionId); + } catch { /* best-effort */ } + log("error", `[${sessionId}] [2x2:${lane}] R3 identity fault ×3 in window (${v.reason}) — sid latched to the legacy path (persisted)`); + } else { + const strike = r3FaultTracker.strikes(sessionId); + log("warn", `[${sessionId}] [2x2:${lane}] R3 identity fault — strike ${strike}/${r3FaultTracker.strikeCap} in window (${v.reason}) — 400 sent; the sid latches to the legacy path after ${r3FaultTracker.strikeCap - strike} more`); + throw new FoundationRejectError(`inbound msgid envelope failed validation (${v.reason})`); + } + } else { + log("warn", `[${sessionId}] [msgid] ${lane}-wire pin rejected (${v.reason}) — falling back to content-derived ids (#4491/#5825)`); + } + } + } else { + const inc = incrementalOn && bodyTail !== null ? tryIncrementalPin(sessionId, lane, msgs, bodyTail, log, liveIds) : null; + if (inc !== null) pinned = inc; + } + } + } else { + log("info", `[${sessionId}] [msgid] ${lane}-wire biliMsgIds present but pluginLaneServerHistory rollout is off for this install — field stripped, ids not pinned (#5825)`); + } + // #7064 §8: the signal is what the client may send NEXT turn — a + // SUCCESSFUL pin (bootstrap / reanchor included) means the ledger + // holds the prefix, so the next turn may go incremental. This is + // what upgrades the client out of full-list mode: a hard "full" + // here would deadlock the ladder (the full form always wins while + // it is sent, so the incremental branch would never be reached). + pinMode = pinned !== null ? "incremental" : "full"; + } else if (pluginMode && Array.isArray(incomingTailIds) && incomingTailIds.length > 0) { + // #7064 §8 steady state: no full list, only the bounded tail (anchor + // + new cores). The prefix is pinned from the LEDGER's live rows, + // the new cores from the tail. ANY mismatch degrades to the full + // signal — D1: this path NEVER rejects (it is a drift class, not an + // R3 identity fault) — and the next turn re-converges with a full + // list (one-round convergence by construction). + if (incrementalOn) { + const inc = tryIncrementalPin(sessionId, lane, msgs, incomingTailIds, log); + if (inc !== null) { + pinned = inc; + pinMode = "incremental"; + } else { + pinMode = "full"; + } + } else { + // Flag off (or the lane rollout off): only the full-list pin + // runs server-side — tell the client to go back to the full + // form so a rollback self-heals instead of leaving the session + // pin-less until a 4xx relearns the mode. + pinMode = "full"; + } + } + let anchorAdvanced = false; + // D3: the cursor (world anchor) refreshes whenever the host declared + + // the pin APPLIED — hit or miss turns alike. A rejected pin leaves + // content-derived ids on the array and must NOT touch the host-id + // cursor. The verdict is PURE bookkeeping (D1): it can only re-issue, + // never reject. + if (pinned !== null) { + try { + const store = getStore(); + const stage0 = store?.snapStage0(sessionId, pinned.map((m) => m.id), { seqs: incomingSeqs ?? undefined }); + if (stage0) { + anchorAdvanced = stage0.decision.mode !== "echo"; + log(anchorAdvanced ? "info" : "debug", `[${sessionId}] [2x2:${lane}] stage-0 ${stage0.decision.mode} (new=${stage0.decision.newIdPos.length}, gone=${stage0.decision.goneIds.length})${stage0.decision.witness ? ` — ${stage0.decision.witness}` : ""}`); + } + } catch (err) { + if (err instanceof FoundationRejectError) { + // An R3-class fault (l1 ledger conflict — a real mapping + // under a different ledger sid) rejects the request on + // EVERY wire: the conflict is a durable ledger state, not a + // per-request glitch (the M3.2b contract). + throw err; + } else { + const why = err instanceof Error ? err.message : String(err); + // A stage-0 storage fault (corrupt cursor row / schema edge — + // R4-shaped): clear the cursor best-effort and take the + // legacy path this turn (one-round convergence: the next + // turn re-issues the cursor from the client's array). + if (why.includes("[snapshot") || why.includes("SQLITE") || why.includes("no such")) { + try { + getStore()?.snapStage0Reset(sessionId); + } catch { /* the reset is best-effort — the legacy path is the floor */ } + } + log("warn", `[${sessionId}] [2x2:${lane}] stage-0 failed (${why}) — legacy full path`); + } + } + } + return { pinned, cohortRan: armed, anchorAdvanced, pinMode }; +} + +// #7064 §8 (2026-11-19 owner ruling, the ed71f127 deadlock fix): the +// INCREMENTAL pin — the list is a CONTIGUOUS TAIL of the current view, +// [T-L .. T-1]; the overlap with the ledger's live rows ([T-L .. C-1]) +// must equal the ledger ROW BY ROW, the cores past the overlap are pinned +// from the list (a "?" slot keeps its content-derived id). Returns null — +// the full signal — on ANY structural disagreement: empty ledger +// (bootstrap), a list too SHORT to reach into the ledger (nothing to +// verify), a non-tail-aligned list (longer than the view), or any overlap +// row drifting from the ledger. The steady-state form (anchor + new cores, +// L = N+1) is the 1-row-overlap special case of the same rule. Over-ceiling +// sessions (> the client's 2048 retention window) send their TRUNCATED +// body list on carrier-off turns: the overlap then spans hundreds–2047 +// rows and the pin PROVES MORE, not less — so a degraded "full" turn can +// still pin and re-upgrade the client; the full list is never a +// requirement any client must be able to produce. Exported: the anthropic +// lane's inline pin reuses it (two-form rule). +export function tryIncrementalPin( + sessionId: string, + lane: string, + msgs: BiliMessage[], + tailIds: string[], + log: (level: string, msg: string) => void, + preReadLiveIds?: string[], +): BiliMessage[] | null { + let liveIds: string[]; + if (preReadLiveIds !== undefined) { + liveIds = preReadLiveIds; + } else { + try { + liveIds = getStore()?.snapMsgIdLiveIds(sessionId) ?? []; + } catch { + return null; // a store fault degrades — never a 500 on the pin path + } + } + const C = liveIds.length; + const T = msgs.length; + const L = tailIds.length; + if (C === 0) { + log("debug", `[${sessionId}] [2x2:${lane}] incremental form on an empty ledger — full signal (bootstrap)`); + return null; + } + const n = T - C; // new cores since the ledger caught up (0 on a pure resend) + if (L < n + 1) { + // A list that does not reach into the ledger has NO overlap to + // verify — it cannot prove the prefix, so it cannot pin. (A host + // that shrank/rewrote sends the FULL form instead — that path + // re-anchors by design.) + log("info", `[${sessionId}] [2x2:${lane}] incremental list too short to overlap the ledger (list=${L}, new cores=${n}, view=${T}, ledger=${C}) — full signal (reanchor next turn)`); + return null; + } + if (L > T) { + log("info", `[${sessionId}] [2x2:${lane}] incremental list longer than the view (list=${L}, view=${T}) — full signal (reanchor next turn)`); + return null; + } + // TAIL ALIGNMENT: the list covers [T-L .. T-1]; the overlap with the + // ledger is [T-L .. C-1] = tailIds[0 .. L-n-1]. The steady-state form + // (anchor + new cores) is exactly the 1-row overlap (tailIds[0] must be + // the ledger frontier); an over-ceiling truncated list (ed71f127: + // >2048 sessions can only ever send their retained tail) carries up to + // the client's window of overlap rows and proves MORE. ANY drift row + // degrades to the full signal — D1: never a reject. + const overlapEnd = L - n; // first slot that is a NEW core (>= C) + for (let j = 0; j < overlapEnd; j++) { + if (tailIds[j] !== liveIds[T - L + j]) { + log("info", `[${sessionId}] [2x2:${lane}] incremental tail drift at overlap row ${T - L + j} (got=${tailIds[j]}, ledger=${liveIds[T - L + j]}) — full signal (reanchor next turn)`); + return null; + } + } + // Entry screening on the NEW cores (same rules as msgid-pin + // validateMsgIds — a bad id must never reach the ledger's unique index). + // The seen-set is SEEDED WITH THE LEDGER PREFIX: a new core reusing an + // existing id (same id, different content) is the identity-anomaly + // validateMsgIds rejects on the full form — a legit host mints fresh ids + // for fresh positions, it never reuses one. + const seen = new Set(liveIds); + for (let j = overlapEnd; j < L; j++) { + const id = tailIds[j]; + if (id === "?") continue; + if (typeof id !== "string" || id.length === 0 || id.length > 512 || id.startsWith("acp_summary_") || /[\s\p{C}]/u.test(id) || seen.has(id)) { + log("info", `[${sessionId}] [2x2:${lane}] incremental tail carried an invalid id at slot ${j} — full signal (reanchor next turn)`); + return null; + } + seen.add(id); + } + // Overlap proven: the ledger's live row order IS the view order (ingest + // sweeps the ledger to the authoritative set, #3285/#4390 — the overlap + // check above is what keeps the two in step; a fold-edge transient + // degrades one turn and re-converges with the full list). + return msgs.map((m, i) => { + if (i < C) return { ...m, id: liveIds[i] }; + const id = tailIds[i - C + L - n]; // slot of view position i in the list (n = T - C) + return id === "?" ? m : { ...m, id }; + }); +} diff --git a/src/snapshot/types.ts b/src/snapshot/types.ts new file mode 100644 index 0000000000..5cddc62928 --- /dev/null +++ b/src/snapshot/types.ts @@ -0,0 +1,70 @@ +/** + * #5874 / 2×2 — snapshot-foundation stage-0 shared types. + * + * TYPE-ONLY surface. Depends on nothing but the kernel's public types so it + * can be imported from any layer (storage, server, tests) without a cycle. + * + * #7064 msgid-storage slimming (owner ruling 2026-11-17): the stage-0 world + * state is a BOOKKEEPING CURSOR, not an identity gate — the server keeps only + * the head and tail of the last seen host-id array plus the count. The + * middle is deliberately invisible (client compression removes ids freely — + * D1: an id verdict NEVER diverges; a middle rewrite surfaces via the ledger + * probes, not here). Every verdict is bookkeeping: the worst outcome is + * `reanchor` (re-issue the cursor), never a reject. + */ + +/** The stored world anchor: the head + tail id windows of the last seen host + * array (host order) plus its length and the append-only pos frontier. */ +export interface WorldWindow { + /** The last min(TAIL, count) ids, host order. */ + tailIds: string[]; + /** The first min(HEAD, count) ids, host order. */ + headIds: string[]; + /** The full array length this window was cut from. */ + count: number; + /** The largest append-only pos issued so far (never reused). */ + maxPos: number; +} + +/** Stage-0 verdicts (post-slimming). Executed in order; first hit wins. + * - `echo`: head + count + tail all agree — nothing moved. + * - `extend`: the stored anchor is contained in the incoming tail — new + * ids appended (the common shape). + * - `shrink`: the incoming is a suffix of the stored tail — the client + * dropped head elements (a fold; D1: legal, never divergent). + * - `reanchor`: no anchor relation (head swap / middle rewrite / re-mint) — + * the cursor is RE-ISSUED (D1: bookkeeping, never a reject). + * - `issue`: no stored world for this sess_raw → fresh ledger identity + + * arm. + * - `resync`: an explicit re-baseline (mode switch). + * - `anchor`: D2 — the session carries a host conversation identity, the + * id verdict is skipped entirely; the cursor follows the + * incoming unconditionally. + * + * Deletions of windowed ids are reported via `goneIds` (the reconcile turns + * them into `delete` events) — orthogonal to the mode. */ +export type WorldMode = "echo" | "extend" | "shrink" | "reanchor" | "issue" | "resync" | "anchor"; + +export interface WorldDecision { + mode: WorldMode; + /** The world anchor to persist after this turn. */ + window: WorldWindow; + /** The NEW ids issued this turn (id → fresh append-only pos), in + * incoming order. Empty for echo / reanchor-class verdicts. */ + newIdPos: Array<{ id: string; pos: number }>; + /** The STORED windowed ids ABSENT from the incoming (client deletions) — + * the reconcile turns these into `delete` events. Empty for issue/echo. */ + goneIds: string[]; + /** For `issue`: the freshly issued ledger identity (the existing sid + * keyspace, ruling D). Absent otherwise. */ + issuedLedgerId?: string; + /** The world origin (issued_at) — carried so the caller can preserve it + * on a plain update (B1 trap: it never moves except on issue/resync). */ + issuedAt: number; + /** Diagnostic witness sample — warn stream only. */ + witness?: string; + /** Review-round-2 Residual 1b: the host re-sent a body this proxy ALREADY + * processed (every id known, carrier seqs at/below the frontier) — an + * upstream-200 retry. Zero side effects: the cursor is kept as-is. */ + replay?: boolean; +} diff --git a/src/storage/backend.ts b/src/storage/backend.ts index 794e20c5ad..f1cc641f9b 100644 --- a/src/storage/backend.ts +++ b/src/storage/backend.ts @@ -91,6 +91,11 @@ export interface StorageBackend { readSnapshot(read: (reader: StorageReadView) => Promise): Promise; getCcr(sessionId: string): Promise; getCcrEntries(sessionId: string): Promise; + /** #6785 step 7: the merged reaped-block journal (table rows ∪ legacy + * column). null = none. */ + getOrphanViews(sessionId: string): Promise | null>; + /** #6785 step 8: row-backed CompressionState fields (state_fields table). */ + getStateFields(sessionId: string): Promise | null>; // --- ingest watermark / pointers (#3253, #3798) ------------------------ ingestWatermark(sessionId: string): Promise<{ seq: number; tailSeqs: number[] } | null>; @@ -151,7 +156,7 @@ export interface StorageBackend { init?(): Promise; } -export type StorageReadView = Pick; +export type StorageReadView = Pick; /** The sqlite backend: the existing UnifiedStore behind the async contract. * Synchronous under the hood; `Promise.resolve` wrapping only. */ @@ -224,6 +229,9 @@ export class SqliteBackend implements StorageBackend { getBlocks: async (id, opts) => store.getBlocks(id, opts), getDeadRefs: async (id) => store.getDeadRefs(id), getRefs: async (id) => store.getRefs(id), + getCcrEntries: async (id) => store.getCcrEntries(id), + getOrphanViews: async (id) => store.getOrphanViews(id), + getStateFields: async (id) => store.getStateFields(id), })); } getCcr(sessionId: string): Promise { @@ -232,6 +240,13 @@ export class SqliteBackend implements StorageBackend { getCcrEntries(sessionId: string): Promise { return Promise.resolve(this.store.getCcrEntries(sessionId)); } + getOrphanViews(sessionId: string): Promise | null> { + return Promise.resolve(this.store.getOrphanViews(sessionId)); + } + + getStateFields(sessionId: string): Promise | null> { + return Promise.resolve(this.store.getStateFields(sessionId)); + } ingestWatermark(sessionId: string): Promise<{ seq: number; tailSeqs: number[] } | null> { return Promise.resolve(this.store.ingestWatermark(sessionId)); } diff --git a/src/storage/egress-record.ts b/src/storage/egress-record.ts new file mode 100644 index 0000000000..cb185b02a1 --- /dev/null +++ b/src/storage/egress-record.ts @@ -0,0 +1,159 @@ +/** + * Egress op-log recording — segment 3 of the op-log outbound view (proposal + * discussions/7398, revised self-contained-bytes design discussions/7439 + * ruling 1, #6785 perf work). Each turn diffs the outbound conversation view + * against the previous turn's and appends the sparse op batch to the dual + * tables. Outbound bytes are served from the log only behind the serve key + * (segment 4b). Gated by rollout key `egressPlane` (default 0 = no recording + * at all). + * + * SELF-CONTAINED BYTES (ruling 7439-1): each view row carries the message's + * FINAL core JSON string directly (the system row carries the system text). + * No hash references, no fingerprint fields, no content table — the content + * identity of a row is the byte string itself (the diff keys on msgid and + * compares bytes directly), so the op-log replays and serves with nothing + * but the two egress tables. + * + * Identity contract (KDD#11/KDD#17): a view row's msgid is the message's + * kernel-space id — host msgid in pinned 2×2 lanes (the pin restamps ids + * position-for-position), content-hash mint otherwise (kernel mints are + * never re-issued). Ephemeral rows (nudge, sysNotes) are EXCLUDED: they are + * by-design transient (KDD#7 prefix-cache-anchor) and would churn +I/-D + * every nudge cycle with no serving value; the leading system block IS a + * row under the constant msgid "egress-system" (its byte stability is what + * the cache anchor relies on, and a system change shows up as a +U on that + * row — content identity is bytes, position/identity is the msgid). + */ + +import { diffEgressView, type EgressOp, type EgressViewRow } from "./egress-state.js"; + +/** Constant msgid of the system block row (no hash — identity is the + * role, not the content; a system change is a +U carrying the new bytes). */ +export const EGRESS_SYSTEM_MSGID = "egress-system"; + +/** Minimal structural shape — accepts both BiliMessage (wire sidecar) and + * bare kernel CoreMessage (the degraded-path originalMessages). Exported: + * prepare-openai names the source array it hands to recordEgressTurn. */ +export interface EgressSourceMsg { + id: string; + role: string; + contentType?: unknown; + text?: unknown; +} + +/** Build this turn's view rows: [system?] + one row per conversation + * message. Deterministic over (ids, roles, texts): same inputs → same rows + * (byte-equal), which is what makes the next diff sparse. `tokens` is a + * chars/4 estimate (it only needs to be stable while the content is + * unchanged); `bytes` is the FINAL core JSON of the message — the same + * bytes the serve path rebuilds from. */ +export function buildEgressViewRows( + conversation: readonly EgressSourceMsg[], + systemText: string | undefined, +): EgressViewRow[] { + const rows: EgressViewRow[] = []; + if (systemText !== undefined && systemText.length > 0) { + rows.push({ + msgid: EGRESS_SYSTEM_MSGID, + ref: "", + tokens: Math.max(1, Math.ceil(systemText.length / 4)), + type: "system", + role: "system", + bytes: systemText, + }); + } + for (const m of conversation) { + const text = typeof m.text === "string" ? m.text : ""; + rows.push({ + msgid: m.id, + ref: /^m\d{4,}$/.test(m.id) ? m.id : "", + tokens: Math.max(1, Math.ceil(text.length / 4)), + type: m.id.startsWith("acp_summary") ? "acp_summary" : m.role === "tool" ? "tool" : "text", + role: m.role, + bytes: JSON.stringify(m), + }); + } + return rows; +} + +/** The storage seam (UnifiedStore today; keeps the recorder testable). */ +export interface EgressOpLogStore { + egressStateAppendOps(sessionId: string, ops: EgressOp[], nowMs: number): number[]; + egressStateView(sessionId: string): EgressViewRow[]; + egressStateCompactView(sessionId: string, view: EgressViewRow[], nowMs: number): number; + /** Rollback reconstruction at any log_seq (flow table is total). */ + egressStateViewAt(sessionId: string, logSeq: number): EgressViewRow[]; + /** Highest log_seq in the flow (0 = no rows). */ + egressStateMaxSeq(sessionId: string): number; + /** Recovery (ruling 7439-5): delete rows past `logSeq` in BOTH tables — + * rows beyond a corrupt point are unusable by definition (no replay can + * ever include them), so recovery = flow back to the last healthy point. + * Append-only covers HEALTHY history only. Returns rows removed. */ + egressStateTruncateAfter(sessionId: string, logSeq: number): number; +} + +/** Record this turn's outbound conversation view as sparse ops. `prev` is + * this turn's starting point — the previous turn's view when present, + * otherwise a one-shot hydrate from the tables (owner ruling 7439-3: the + * tables are the only state; there is no long-lived in-memory view cache). + * Compacts when the batch carries a -D (the fold shape: cancellation is the + * trigger, per the proposal). Any EgressStateCorruptError propagates — the + * caller handles it per the degrade ladder (no fuzzy fallback, KDD + * discipline). */ +export function recordEgressTurn(args: { + store: EgressOpLogStore; + sessionId: string; + prev: EgressViewRow[] | undefined; + conversation: readonly EgressSourceMsg[]; + systemText: string | undefined; + nowMs: number; +}): EgressViewRow[] { + const { store, sessionId, conversation, systemText, nowMs } = args; + const prev = args.prev ?? store.egressStateView(sessionId); + const next = buildEgressViewRows(conversation, systemText); + const ops = diffEgressView(prev, next); + if (ops.length > 0) { + store.egressStateAppendOps(sessionId, ops, nowMs); + if (ops.some((o) => o.type === "D")) { + store.egressStateCompactView(sessionId, next, nowMs); + } + } + return next; +} + + +/** Bounded walk-back distance when the flow's current point is corrupt. */ +const EGRESS_HEALTHY_WALK_MAX = 16; + +/** Ruling 7439-5 degrade ladder — hydrate the flow's CURRENT point; when it + * fails replay (a corrupt row), walk back to the NEWEST healthy log_seq + * (bounded walk) and TRUNCATE past it: rows beyond a corrupt point are + * unusable by definition (no replay can ever include them), so recovery + * means flowing back to the last healthy point in the tables themselves — + * never to the client's bytes. Returns null = no healthy point at all + * (session-level recovery: the caller logs the marker and serves legacy for + * this turn only). The truncated count is observability for the warn log. */ +export function recoverEgressBase( + store: EgressOpLogStore, + sessionId: string, +): { view: EgressViewRow[]; seq: number; degraded: boolean; truncated: number } | null { + const maxSeq = store.egressStateMaxSeq(sessionId); + try { + const view = store.egressStateView(sessionId); + return { view, seq: maxSeq, degraded: false, truncated: 0 }; + } catch { + const floor = Math.max(0, maxSeq - EGRESS_HEALTHY_WALK_MAX); + for (let x = maxSeq - 1; x >= floor; x--) { + try { + const view = store.egressStateViewAt(sessionId, x); + if (view.length > 0 || x === 0) { + const truncated = store.egressStateTruncateAfter(sessionId, x); + return { view, seq: x, degraded: true, truncated }; + } + } catch { + // keep walking + } + } + return null; + } +} diff --git a/src/storage/egress-serve.ts b/src/storage/egress-serve.ts new file mode 100644 index 0000000000..b0519aa5a9 --- /dev/null +++ b/src/storage/egress-serve.ts @@ -0,0 +1,75 @@ +/** + * Egress serve-from-log — segment 4b of the op-log outbound view (proposal + * discussions/7398, revised self-contained-bytes design discussions/7439 + * ruling 1, #6785 perf work). Rebuilds the outbound core-message portion + * from the replayed log and runs it through the SAME legacy chain the + * pipeline uses, so the output is byte-identical by construction: the + * recorded bytes ARE the final core JSON (segment 3's recording guarantees + * it), no content table anywhere. + * + * rebuildCoreFromLog is LANE-NEUTRAL (owner ruling 7439-4): it returns the + * core BiliMessage[] and each lane feeds its own conversion chain + * (prepare-openai uses the openai wrapper below; anthropic/responses/google + * join in the same way). prepare-openai (rollout `egressPlane`, default + * 0) SERVES these elements as the outbound conversation when the rebuild + * succeeds, and falls back to the legacy path — with the 4a shadow + * byte-comparison still running on fallback turns — when it cannot (degrade, + * never guess). + * + * Scope (honest, pinned in tests): + * - text, tool, and acp_summary rows rebuild (the entire 10w-row + * uncompressed workload, coding sessions included); a row is the FULL + * core message JSON, so the toolCallId/toolName sidecars are present + * and the same legacy chain reproduces the bytes; + * - a row whose bytes do not parse / fail the role whitelist → null + * (caller stays on the legacy path — degrade, never guess). + */ + +import { isRenderedSummaryMessage } from "acp-kernel"; +import { coreToOpenai, type BiliMessage } from "acp-kernel/wire"; +import { hardenOpenaiAssistantContent, systemToUser } from "../util.js"; +import type { EgressViewRow } from "./egress-state.js"; + +interface EgressRebuildResult { + /** The rebuilt outbound elements for the conversation rows (system + * block excluded — sysParts are built live and stay live). */ + elements: ReturnType; +} + +const CORE_ROLES = ["system", "developer", "user", "assistant", "tool"] as const; + +/** Lane-neutral core rebuild: replayed view rows → core BiliMessage[] + * (system rows excluded — the system block is built live by every lane, + * never stored in the log). Returns null when ANY row cannot be rebuilt + * byte-exactly (the caller must stay on the legacy path for the whole turn + * — a partial rebuild would splice two code paths into one outbound body). */ +export function rebuildCoreFromLog(view: readonly EgressViewRow[]): BiliMessage[] | null { + const msgs: BiliMessage[] = []; + for (const r of view) { + if (r.type === "system") continue; + if (typeof r.bytes !== "string" || r.bytes.length === 0) return null; + let m: unknown; + try { + m = JSON.parse(r.bytes); + } catch { + return null; + } + if (typeof m !== "object" || m === null) return null; + const c = m as { id?: unknown; role?: unknown; [k: string]: unknown }; + if (typeof c.id !== "string" || typeof c.role !== "string" || !(CORE_ROLES as readonly string[]).includes(c.role)) return null; + msgs.push(m as BiliMessage); + } + return msgs; +} + +/** Rebuild the outbound core-message portion from a replayed log view. + * Returns null when ANY row cannot be rebuilt byte-exactly (see + * rebuildCoreFromLog). */ +export function rebuildOutboundFromLog(view: readonly EgressViewRow[]): EgressRebuildResult | null { + const msgs = rebuildCoreFromLog(view); + if (msgs === null) return null; + // Same chain, same predicates as prepare-openai :572 — identity is the + // claim, so the chain must be identical, not equivalent. + const elements = hardenOpenaiAssistantContent(coreToOpenai(systemToUser(msgs, (m: BiliMessage) => isRenderedSummaryMessage(m)))); + return { elements }; +} diff --git a/src/storage/egress-state.ts b/src/storage/egress-state.ts new file mode 100644 index 0000000000..a29a404b4e --- /dev/null +++ b/src/storage/egress-state.ts @@ -0,0 +1,359 @@ +/** + * Egress state log — the op-log view of the outbound conversation + * (proposal: discussions/7398 + the revised design in discussions/7439, + * owner rulings 2026-11-20/21; #6785 perf work). + * + * The outbound view changes sparsely: tail appends, occasional content edits + * (same msgid, new content), fold ranges replaced by one summary row. Instead + * of recomputing the whole view every turn (render/convert/serialize over all + * messages), the view is a TOTAL-ORDERED op log per session; replaying the + * log yields exactly one deterministic view. + * + * SELF-CONTAINED BYTES (owner ruling, discussion 7439 裁定 1): every I/U row + * carries the message's FINAL core JSON in its own `bytes` column — no hash + * references, no fingerprint fields, no global content table. Content + * identity is judged by DIRECT BYTE EQUALITY; replay needs nothing but this + * pair of tables. The two tables are not shipped in any release yet, so the + * row/payload format is free (no migration burden). + * + * Two tables (dual-table, owner ruling): + * egress_state — what serving reads; compaction PHYSICALLY deletes rows + * below the watermark, so it stays bounded; + * egress_flow — append-only, NEVER deleted; reconstructs ANY historical + * log_seq (rollback is based on this table, not on client + * resends). + * + * Row types (payload shapes are the contract — see the discussion post): + * B base msgid NULL {"entries": viewRow[], watermark} + * (entries carry the bytes inline) + * I insert msgid required {"after": msgid|null, ref, tokens, + * type, role?} + bytes column + * D delete-range msgid NULL {"start": msgid, "end": msgid} inclusive + * U update-content msgid required {"role"?} + bytes column (the new + * bytes; detected by byte inequality) + * + * Anchors are MSGIDS, never display positions — positions shift under other + * ops, msgids do not (KDD#11: content may change, msgid does not). Any + * anchor/endpoint violation, missing update target, or duplicate msgid is a + * HARD ERROR: the session is corrupt and the caller degrades per the + * degrade ladder (no fuzzy guessing — same discipline as every other lane's + * "any doubt, fall back whole-turn" rule). + */ + +import type { SqliteDatabase } from "./driver.js"; + +export class EgressStateCorruptError extends Error { + constructor(readonly sessionId: string, readonly reason: string) { + super(`egress-state corrupt (session=${sessionId}): ${reason}`); + this.name = "EgressStateCorruptError"; + } +} + +/** One row of the current view. The ref (mNNNNN) is identity, never hash + * and never re-issued (kernel contract). `bytes` is the message's FINAL core + * JSON (post reasoning-drop / image-rescale, pre outbound convert); for the + * system row it is the system text itself. The row is self-contained: + * replay + serve need nothing beyond this pair of tables. */ +export interface EgressViewRow { + msgid: string; + ref: string; + tokens: number; + type: string; + /** Final core JSON of the message (system row: the system text). */ + bytes: string; + /** The kernel-space role (rebuild needs it; the diff keys on msgid so a + * role-only drift is impossible — kernel ids are one-shot). */ + role?: string; +} + +type EgressOpType = "B" | "I" | "D" | "U"; + +interface EgressStateRow { + log_seq: number; + row_type: EgressOpType; + msgid: string | null; + payload: string; + bytes: string | null; +} + +const INSERT_STATE = `INSERT INTO egress_state (session_id, log_seq, row_type, msgid, payload, bytes, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)`; +const INSERT_FLOW = `INSERT INTO egress_flow (session_id, log_seq, row_type, msgid, payload, bytes, created_at) VALUES (?, ?, ?, ?, ?, ?, ?)`; + +function nextLogSeq(db: SqliteDatabase, sessionId: string): number { + // egress_flow is append-only and superset of egress_state after every + // compaction, so the flow max IS the session's sequence high-water. + const row = db + .prepare("SELECT MAX(log_seq) AS m FROM egress_flow WHERE session_id = ?") + .get(sessionId) as { m: number | null } | undefined; + return (row?.m ?? 0) + 1; +} + +interface InsertOp { + type: "I"; + msgid: string; + after: string | null; + ref: string; + tokens: number; + typeTag: string; + bytes: string; + role?: string; +} +interface DeleteOp { + type: "D"; + start: string; + end: string; +} +interface UpdateOp { + type: "U"; + msgid: string; + bytes: string; + role?: string; +} +export type EgressOp = InsertOp | DeleteOp | UpdateOp; + +/** Pure replay: apply ops (in log_seq order) on top of an existing view. + * Shared by the db-backed replays and the diff engine's own verification. */ +export function egressReplay(prev: EgressViewRow[], ops: EgressOp[]): EgressViewRow[] { + const view: EgressViewRow[] = prev.map((r) => ({ ...r })); + const idxOf = (msgid: string): number => view.findIndex((r) => r.msgid === msgid); + for (const op of ops) { + if (op.type === "I") { + if (idxOf(op.msgid) >= 0) throw new EgressStateCorruptError("diff", `duplicate msgid ${op.msgid}`); + const at = op.after === null ? -1 : idxOf(op.after); + if (at < -1) throw new EgressStateCorruptError("diff", `insert anchor ${op.after} missing`); + const entry: EgressViewRow = { msgid: op.msgid, ref: op.ref, tokens: op.tokens, type: op.typeTag, bytes: op.bytes }; + if (op.role !== undefined) entry.role = op.role; + view.splice(at + 1, 0, entry); + } else if (op.type === "D") { + const s = idxOf(op.start); + const e = idxOf(op.end); + if (s < 0) throw new EgressStateCorruptError("diff", `delete start ${op.start} missing`); + if (e < 0) throw new EgressStateCorruptError("diff", `delete end ${op.end} missing`); + if (s > e) throw new EgressStateCorruptError("diff", "delete range inverted"); + view.splice(s, e - s + 1); + } else { + const i = idxOf(op.msgid); + if (i < 0) throw new EgressStateCorruptError("diff", `update target ${op.msgid} missing`); + view[i] = { ...view[i], bytes: op.bytes, ...(op.role !== undefined ? { role: op.role } : {}) }; + } + } + return view; +} + +/** Deterministic diff between two consecutive views (owner design, 4 upper + * cases: tail append / nudge appear+vanish / fold replace / content edit). + * + * Spine = the longest subsequence of `next` whose msgids appear in `prev` in + * INCREASING prev order (pointer scan, O(n+m); exact for the sparse-change + * regime this lane exists for). Spine rows with CHANGED BYTES (direct + * inequality — no hash) emit +U; everything else emits -D (maximal + * contiguous runs in PREV order) plus +I in NEXT order with the preceding + * row as anchor. egressReplay(prev, diff) always reproduces `next` exactly. + * + * Moves (same msgid, new position) are deliberately expressed as D+I — there + * is no M op; the msgid stays the identity through both ops. */ +export function diffEgressView(prev: EgressViewRow[], next: EgressViewRow[]): EgressOp[] { + const prevIdx = new Map(); + prev.forEach((r, i) => { if (!prevIdx.has(r.msgid)) prevIdx.set(r.msgid, i); }); + const prevByMsg = new Map(); + prev.forEach((r) => { if (!prevByMsg.has(r.msgid)) prevByMsg.set(r.msgid, r); }); + const spine = new Set(); + let p = -1; + for (const r of next) { + const i = prevIdx.get(r.msgid); + if (i !== undefined && i > p) { + spine.add(r.msgid); + p = i; + } + } + const ops: EgressOp[] = []; + let runStart: string | null = null; + let runEnd: string | null = null; + for (const r of prev) { + if (!spine.has(r.msgid)) { + if (runStart === null) runStart = r.msgid; + runEnd = r.msgid; + } else if (runStart !== null) { + ops.push({ type: "D", start: runStart, end: runEnd! }); + runStart = runEnd = null; + } + } + if (runStart !== null) ops.push({ type: "D", start: runStart, end: runEnd! }); + for (const r of next) { + if (spine.has(r.msgid)) { + const old = prevByMsg.get(r.msgid)!; + if (old.bytes !== r.bytes) { + const u: UpdateOp = { type: "U", msgid: r.msgid, bytes: r.bytes }; + if (r.role !== undefined) u.role = r.role; + ops.push(u); + } + } + } + let last: string | null = null; + for (const r of next) { + if (!spine.has(r.msgid)) { + const ins: InsertOp = { type: "I", msgid: r.msgid, after: last, ref: r.ref, tokens: r.tokens, typeTag: r.type, bytes: r.bytes }; + if (r.role !== undefined) ins.role = r.role; + ops.push(ins); + } + last = r.msgid; + } + return ops; +} + +/** Append a batch of ops to BOTH tables (state + flow) in ONE transaction. + * Returns the assigned log_seqs in input order. */ +export function egressStateAppend( + db: SqliteDatabase, + sessionId: string, + ops: EgressOp[], + nowMs: number, +): number[] { + if (ops.length === 0) return []; + const first = nextLogSeq(db, sessionId); + db.transaction(() => { + for (let i = 0; i < ops.length; i++) { + const op = ops[i]; + const seq = first + i; + let rowType: EgressOpType; + let msgid: string | null; + let payload: string; + let bytes: string | null = null; + if (op.type === "I") { + rowType = "I"; + msgid = op.msgid; + payload = JSON.stringify({ after: op.after, ref: op.ref, tokens: op.tokens, type: op.typeTag, role: op.role }); + bytes = op.bytes; + } else if (op.type === "D") { + rowType = "D"; + msgid = null; + payload = JSON.stringify({ start: op.start, end: op.end }); + } else { + rowType = "U"; + msgid = op.msgid; + payload = JSON.stringify({ role: op.role }); + bytes = op.bytes; + } + db.prepare(INSERT_STATE).run(sessionId, seq, rowType, msgid, payload, bytes, nowMs); + db.prepare(INSERT_FLOW).run(sessionId, seq, rowType, msgid, payload, bytes, nowMs); + } + }); + return Array.from({ length: ops.length }, (_, i) => first + i); +} + +/** Replay the STATE table into the current view. Throws + * EgressStateCorruptError on any anchor/endpoint/id violation — the caller + * degrades per the degrade ladder. */ +export function egressStateView(db: SqliteDatabase, sessionId: string): EgressViewRow[] { + const rows = db + .prepare( + "SELECT log_seq, row_type, msgid, payload, bytes FROM egress_state WHERE session_id = ? ORDER BY log_seq ASC", + ) + .all(sessionId) as unknown as EgressStateRow[]; + return applyOps(sessionId, rows); +} + +/** Replay the FLOW table up to (and including) log_seq x — the rollback + * reconstruction (degrade ladder step 2). The latest B row with + * log_seq <= x is the starting point; ops after it up to x are applied. + * Throws like egressStateView. */ +export function egressStateAt(db: SqliteDatabase, sessionId: string, x: number): EgressViewRow[] { + const rows = db + .prepare( + "SELECT log_seq, row_type, msgid, payload, bytes FROM egress_flow WHERE session_id = ? AND log_seq <= ? ORDER BY log_seq ASC", + ) + .all(sessionId, x) as unknown as EgressStateRow[]; + return applyOps(sessionId, rows); +} + +/** Compaction: persist the replayed view as a new base row, then physically + * delete every STATE row below it (flow untouched — rollback stays total). + * The base carries the entries themselves (bytes inline in the payload); + * base + later ops must reproduce the current view WITHOUT any deleted row + * (that is the only constraint the base form has). One transaction. */ +export function egressStateCompact( + db: SqliteDatabase, + sessionId: string, + view: EgressViewRow[], + nowMs: number, +): number { + const seq = nextLogSeq(db, sessionId); + const payload = JSON.stringify({ entries: view, watermark: seq }); + db.transaction(() => { + db.prepare(INSERT_STATE).run(sessionId, seq, "B", null, payload, null, nowMs); + db.prepare(INSERT_FLOW).run(sessionId, seq, "B", null, payload, null, nowMs); + db.prepare("DELETE FROM egress_state WHERE session_id = ? AND log_seq < ?").run(sessionId, seq); + }); + return seq; +} + +function applyOps(sessionId: string, rows: EgressStateRow[]): EgressViewRow[] { + let view: EgressViewRow[] = []; + for (const row of rows) { + let parsed: Record; + try { + parsed = JSON.parse(row.payload) as Record; + } catch { + throw new EgressStateCorruptError(sessionId, `payload at log_seq=${row.log_seq} is not an object`); + } + const p: Record = parsed; + if (row.row_type === "B") { + const entries = Array.isArray(p.entries) ? (p.entries as EgressViewRow[]) : null; + if (!entries) throw new EgressStateCorruptError(sessionId, `base at log_seq=${row.log_seq} has no entries`); + view = entries.map((e) => ({ ...e })); + if (view.some((e) => typeof e.msgid !== "string" || typeof e.ref !== "string" || typeof e.bytes !== "string")) { + throw new EgressStateCorruptError(sessionId, `base at log_seq=${row.log_seq} has malformed entries`); + } + continue; + } + const idxOf = (msgid: string): number => view.findIndex((e) => e.msgid === msgid); + if (row.row_type === "I") { + if (typeof row.msgid !== "string") throw new EgressStateCorruptError(sessionId, `insert at log_seq=${row.log_seq} has no msgid`); + if (idxOf(row.msgid) >= 0) throw new EgressStateCorruptError(sessionId, `duplicate msgid ${row.msgid} at log_seq=${row.log_seq}`); + if (typeof p.ref !== "string" || typeof p.tokens !== "number" || typeof p.type !== "string" || typeof row.bytes !== "string") { + throw new EgressStateCorruptError(sessionId, `insert at log_seq=${row.log_seq} has a malformed payload or missing bytes`); + } + const entry: EgressViewRow = { + msgid: row.msgid, + ref: p.ref as string, + tokens: p.tokens as number, + type: p.type as string, + bytes: row.bytes, + }; + if (p.role !== undefined) entry.role = p.role as string; + if (p.after == null) { + view.push(entry); + } else if (typeof p.after !== "string") { + throw new EgressStateCorruptError(sessionId, `insert at log_seq=${row.log_seq} has a malformed anchor`); + } else { + const at = idxOf(p.after); + if (at < 0) throw new EgressStateCorruptError(sessionId, `insert anchor ${p.after} missing at log_seq=${row.log_seq}`); + view.splice(at + 1, 0, entry); + } + } else if (row.row_type === "D") { + if (typeof p.start !== "string" || typeof p.end !== "string") { + throw new EgressStateCorruptError(sessionId, `delete at log_seq=${row.log_seq} has a malformed payload`); + } + const s = idxOf(p.start); + const e = idxOf(p.end); + if (s < 0) throw new EgressStateCorruptError(sessionId, `delete start ${p.start} missing at log_seq=${row.log_seq}`); + if (e < 0) throw new EgressStateCorruptError(sessionId, `delete end ${p.end} missing at log_seq=${row.log_seq}`); + if (s > e) throw new EgressStateCorruptError(sessionId, `delete range inverted (start after end) at log_seq=${row.log_seq}`); + view.splice(s, e - s + 1); + } else if (row.row_type === "U") { + if (typeof row.msgid !== "string" || typeof row.bytes !== "string") { + throw new EgressStateCorruptError(sessionId, `update at log_seq=${row.log_seq} has a malformed payload or missing bytes`); + } + const i = idxOf(row.msgid); + if (i < 0) throw new EgressStateCorruptError(sessionId, `update target ${row.msgid} missing at log_seq=${row.log_seq}`); + view[i] = { + ...view[i], + bytes: row.bytes, + ...(p.role !== undefined ? { role: p.role as string } : {}), + }; + } else { + throw new EgressStateCorruptError(sessionId, `unknown row_type ${String(row.row_type)} at log_seq=${row.log_seq}`); + } + } + return view; +} diff --git a/src/storage/ingest.ts b/src/storage/ingest.ts index d1ba085308..95e6ecdea0 100644 --- a/src/storage/ingest.ts +++ b/src/storage/ingest.ts @@ -29,6 +29,7 @@ import type { CoreMessage, CompressionBlock } from "../../kernel/src/types.js"; import type { BlockView } from "../session.js"; import { refToIndex } from "../../kernel/src/refs.js"; import { sha256Hex, type BlockRow, type CcrEntryRow, type IngestSessionInput, type MessageRow, type RefRow, type SessionRow } from "./store.js"; +import { CcrLedgerRow, hydrateSlimCcrStore, slimCcrEnvelope, splitKstate } from "./slim.js"; import { splitMediaContent, identityOfStoredContent } from "./identity.js"; /** @@ -501,18 +502,42 @@ function finishPlan( })); const ccrEntries: CcrEntryRow[] = session.contentStore - ? Object.entries(session.contentStore.byRef).map(([key, value], ord) => { + ? Object.entries(session.contentStore.byRef).map(([key, value], ord): CcrEntryRow => { const content = JSON.stringify(value); - return { ord, key, content, contentHash: sha256Hex(content) }; + // #6785 step 6: the orig half — the kernel invariant + // (content-store.ts storeOriginal) pins byHash[entry.hash] + // as the original text behind each entry; the slim column + // keeps no bytes, so the CAS row must carry them. + const entry = value as { hash?: unknown } | null; + const origKey = entry !== null && typeof entry === "object" && typeof entry.hash === "string" ? entry.hash : null; + if (origKey === null) { + // Hashless/null entry (fork-receipt stubs): no original + // exists, so the row is complete as-is — "" keeps it + // slim-hydratable (null would mark the WHOLE store + // file-lane-authoritative and drop the orphan byHash half). + return { ord, key, content, contentHash: sha256Hex(content), origContent: "", origHash: sha256Hex("") }; + } + const stored = session.contentStore!.byHash[origKey]; + // A missing byHash entry (bytes absent from memory/sidecar) + // must land as null, not "": "" would look like a legitimate + // empty original and the slim store would reconstruct an + // empty original instead of degrading to the file lane + // (getCcrEntries null contract, store.ts). + const orig = stored === undefined ? null : stored; + return { ord, key, content, contentHash: sha256Hex(content), origContent: orig, origHash: orig === null ? null : sha256Hex(orig) }; }) : []; - const kstate: Record = {}; + const kstateRaw: Record = {}; for (const field of MERGE_STATE_FIELDS) { if (TABLE_OWNED_FIELDS.has(field)) continue; const v = (session.state as Record)[field]; - if (v !== undefined) kstate[field] = v; + if (v !== undefined) kstateRaw[field] = v; } + // #6785 step 8: the kstate slim split — bounded fields stay in the + // column (order preserved), unbounded record/array fields ride one row + // each in state_fields (see splitKstate). + const { column: kstateColumn, rows: stateFields } = splitKstate(MERGE_STATE_FIELDS, kstateRaw); const sessionRow: SessionRow = { sessionId: session.id, @@ -543,12 +568,15 @@ function finishPlan( metaJson: JSON.stringify(session.meta), statsJson: JSON.stringify(session.stats), metadataJson: JSON.stringify(session.metadata), - kstateJson: JSON.stringify(kstate), + kstateJson: JSON.stringify(kstateColumn), tailJson: JSON.stringify({ pluginSnapshotSeqs, tailSeqs, tailFolded: session.lastMessagesFolded === true }), - ccrJson: session.contentStore ? JSON.stringify(session.contentStore) : null, + // #6785 step 6: slim envelope — the column keeps key→hash only; the + // original bytes live in the CAS behind ccr_entries and are re-fetched + // on read (byte-identical by construction). + ccrJson: session.contentStore ? JSON.stringify(slimCcrEnvelope(session.contentStore)) : null, }; - return { session: sessionRow, messages, blocks, deadRefs, refs, ccrEntries, pluginSnapshotSeqs, tailSeqs }; + return { session: sessionRow, messages, blocks, deadRefs, refs, ccrEntries, stateFields, pluginSnapshotSeqs, tailSeqs }; } /** Rebuilds the legacy load view from unified rows alone. */ @@ -647,6 +675,9 @@ export class LegacyViewBuilder { private readonly blocks: unknown[] = []; private readonly blockContents: BlockContents = new Map(); private readonly messagesBySeq = new Map(); + private ccrRows: readonly CcrLedgerRow[] = []; + private orphanViews?: Record; + private stateFieldsRows: Record | null = null; constructor( private readonly session: LegacyViewRows["session"], @@ -690,12 +721,34 @@ export class LegacyViewBuilder { state.nextRunId = session.nextRunId; break; default: + // #6785 step 8: row-backed fields ride state_fields and + // are overlaid at finish() (addStateFields); the legacy + // column (pre-upgrade rows) still carries them, so the + // column value is the base either way. if (kstate[field] !== undefined) state[field] = kstate[field]; break; } } } + /** #6785 step 8: the unbounded CompressionState field rows (state_fields + * table). Call AFTER constructing, BEFORE finish(); rows overlay the + * column values (legacy sessions have no rows → column serves). */ + addStateFields(fields: Record): void { + this.stateFieldsRows = fields; + } + + /** #6785 step 8: apply the row-backed state fields OVER the column + * values (rows win; corrupt row → column value, if any, stays). */ + private applyStateFieldsRows(): void { + if (this.stateFieldsRows === null) return; + for (const [field, valueJson] of Object.entries(this.stateFieldsRows)) { + try { + this.state[field] = JSON.parse(valueJson); + } catch { /* corrupt row → column value (if any) stays */ } + } + } + addBlocks(rows: LegacyViewRows["blocks"]): void { for (const b of rows) this.blocks.push(JSON.parse(b.payloadJson)); appendBlockContents(this.blockContents, rows); @@ -705,10 +758,32 @@ export class LegacyViewBuilder { for (const m of rows) this.messagesBySeq.set(m.seq, JSON.parse(m.content) as CoreMessage); } + /** #6785 step 6: the ccr_entries ledger (CAS content) that backs a slim + * ccr_json envelope — finish() hydrates the store from these rows. */ + addCcrEntries(rows: readonly CcrLedgerRow[]): void { + this.ccrRows = rows; + } + + /** #6785 step 7: the merged orphan journal (orphan_blocks table ∪ + * legacy column, pre-merged by the caller). finish() merges these into + * blockContents INSTEAD of sessions.orphans_json once provided. + * undefined = not provided (callers without table access fall back to + * the column). */ + addOrphanViews(views: Record): void { + this.orphanViews = views; + } + finish(): LegacyView { const s = this.session; const tail = this.tail; - mergeOrphanBlockContents(this.blockContents, s.orphansJson); + this.applyStateFieldsRows(); + // #6785 step 7: the table-backed journal (addOrphanViews) wins over + // the legacy orphans_json column; no views fed ⇒ column fallback. + if (this.orphanViews !== undefined) { + if (Object.keys(this.orphanViews).length > 0) mergeOrphanBlockContents(this.blockContents, JSON.stringify(this.orphanViews)); + } else { + mergeOrphanBlockContents(this.blockContents, s.orphansJson); + } // #4390: a tail pointer whose row is gone used to be silently filtered // out — the view shrank with no trace. Count the gaps so callers can // warn (persist logs them); the surviving members still load. @@ -739,15 +814,43 @@ export class LegacyViewBuilder { lastMessagesFolded: tail.tailFolded, }; if (pluginSnapshot) out.pluginSnapshot = pluginSnapshot; - if (s.ccrJson !== null) out.contentStore = JSON.parse(s.ccrJson); + out.contentStore = resolveCcrContentStore(s.ccrJson, this.ccrRows); return out; } } +/** + * #6785 step 6: the ONE definition of "what does a sessions.ccr_json column + * mean" — shared by the builder (finish), the verifier header and + * loadLegacyView, so the slim/full/legacy degradation rule cannot drift + * between legs. Slim envelope → hydrate from the CAS-backed ledger; a slim + * row whose ledger is empty or still carries legacy NULL-orig rows is an + * inconsistency: no store (the file lane still has it), never a partial one. + * Full (pre-step-6) envelopes parse verbatim. Corrupt JSON → undefined. + */ +export function resolveCcrContentStore( + ccrJson: string | null, + ccrRows: readonly CcrLedgerRow[], +): LegacyView["contentStore"] { + if (ccrJson === null) return undefined; + try { + const parsed = JSON.parse(ccrJson) as { ccrSlim?: number; orphanByHash?: unknown }; + if (parsed.ccrSlim === 1) { + return hydrateSlimCcrStore(parsed, ccrRows) ?? undefined; + } + return JSON.parse(ccrJson) as LegacyView["contentStore"]; + } catch { + return undefined; + } +} + /** Compatibility wrapper for callers that already hold all rows. */ -export function loadLegacyView(rows: LegacyViewRows): LegacyView { +export function loadLegacyView(rows: LegacyViewRows, ccrRows?: readonly CcrLedgerRow[], stateFieldsRows?: Record | null, orphanViews?: Record): LegacyView { const builder = new LegacyViewBuilder(rows.session, rows.refs, rows.deadRefs); builder.addBlocks(rows.blocks); builder.addMessages(rows.messages); + if (ccrRows) builder.addCcrEntries(ccrRows); + if (stateFieldsRows) builder.addStateFields(stateFieldsRows); + if (orphanViews !== undefined) builder.addOrphanViews(orphanViews); return builder.finish(); } diff --git a/src/storage/integrity-check.ts b/src/storage/integrity-check.ts index 3bea3c966b..4fe6ba949e 100644 --- a/src/storage/integrity-check.ts +++ b/src/storage/integrity-check.ts @@ -175,8 +175,9 @@ const isBunRuntime = typeof process !== "undefined" && process.versions != null const openReadOnly = isBunRuntime ? (p) => new (require(BUN_SQLITE_SPEC).Database)(p, { readonly: true }) : (p) => new (require(SQLITE_SPEC).DatabaseSync)(p, { readOnly: true }); +let db; try { - const db = openReadOnly(workerData.path); + db = openReadOnly(workerData.path); const rows = db.prepare("PRAGMA quick_check").all(); db.close(); const verdict = rows.map((row) => String(row.quick_check ?? "")).join(" ").trim(); @@ -191,6 +192,13 @@ try { const c = String((err && err.code) || "").toUpperCase(); const corrupt = c === "SQLITE_CORRUPT" || c === "SQLITE_NOTADB" || m.includes("database disk image is malformed") || m.includes("file is not a database"); parentPort.postMessage(corrupt ? { ok: false, corrupt: true, error: m } : { ok: false, error: m }); + // #6785 CI: the read-only handle must be released on the error path too — + // a throwing quick_check left it open until the worker thread tore down, and + // a concurrent rmSync of the test's temp dir raced that teardown (EPERM on + // windows-latest). The verdict is already posted above, so close is best-effort. + if (db) { + try { db.close(); } catch { /* verdict already posted; close is best-effort */ } + } } `; diff --git a/src/storage/mysql.ts b/src/storage/mysql.ts index 0fcb26775a..6c2c8d6a57 100644 --- a/src/storage/mysql.ts +++ b/src/storage/mysql.ts @@ -233,8 +233,28 @@ CREATE TABLE IF NOT EXISTS ccr_entries ( ord BIGINT NOT NULL, ccr_key VARCHAR(255) NOT NULL, content_hash VARCHAR(64) NOT NULL, + -- #6785 step 6: the original text behind entry.hash (byHash half) — + -- NULL only on pre-step-6 rows (their full ccr_json envelope is the + -- read authority). See src/storage/slim.ts. + orig_hash VARCHAR(64) NULL, PRIMARY KEY (session_id, ord), - KEY idx_ccr_entries_content_hash (content_hash) + KEY idx_ccr_entries_content_hash (content_hash), + KEY idx_ccr_entries_orig_hash (orig_hash) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS orphan_blocks ( + session_id VARCHAR(255) NOT NULL, + block_id VARCHAR(255) NOT NULL, + one_json LONGTEXT NULL, + full_json LONGTEXT NOT NULL, + PRIMARY KEY (session_id, block_id) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS state_fields ( + session_id VARCHAR(255) NOT NULL, + field VARCHAR(64) NOT NULL, + value_json LONGTEXT NOT NULL, + PRIMARY KEY (session_id, field) ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; CREATE TABLE IF NOT EXISTS conversations ( @@ -378,6 +398,17 @@ export class MysqlBackend implements StorageBackend { if (!haveIdx.has("ccr_entries.idx_ccr_entries_content_hash")) { await conn.query("ALTER TABLE ccr_entries ADD INDEX idx_ccr_entries_content_hash (content_hash)"); } + // #6785 step 6: ccr_entries.orig_hash — probe-first like the v8/v10 + // columns; every pre-existing table must pick it up, fresh + // installs get it from the DDL. The key rides the same ALTER. + const [ccrCols] = (await conn.query( + "SELECT COLUMN_NAME AS col FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'ccr_entries'", + )) as unknown as Row[][]; + const haveCcr = new Set((ccrCols as unknown as { col?: unknown }[]).map((r) => String(r.col ?? ""))); + if (!haveCcr.has("orig_hash")) { + await conn.query("ALTER TABLE ccr_entries ADD COLUMN orig_hash VARCHAR(64) NULL"); + await conn.query("ALTER TABLE ccr_entries ADD INDEX idx_ccr_entries_orig_hash (orig_hash)"); + } if (!haveIdx.has("message_media.idx_message_media_media_hash")) { await conn.query("ALTER TABLE message_media ADD INDEX idx_message_media_media_hash (media_hash)"); } @@ -772,6 +803,11 @@ export class MysqlBackend implements StorageBackend { "INSERT IGNORE INTO conversations (namespace, conversation_id, session_id, claimed_at) VALUES (?, ?, ?, ?)", [s.namespace ?? "proxy", s.sessionId, s.sessionId, s.updatedAt], ); + // #6785 step 7: orphans ride the orphan_blocks table (the column + // merges + migrates once, then retires) — same as the sqlite leg. + await this.applyOrphanBlocksOn(c, s.sessionId, s.orphansJson ?? null); + // #6785 step 8: row-backed kstate fields (state_fields table). + await this.applyStateFieldsOn(c, s.sessionId, input.stateFields ?? {}); const batchTs = Date.now(); // #4313: one sample per save batch for (const m of messages) { const hash = await this.putContent(c, m.content, m.contentHash); @@ -1015,7 +1051,7 @@ export class MysqlBackend implements StorageBackend { const incomingCcr = new Map(); for (const e of input.ccrEntries) incomingCcr.set(e.key, e); const storedCcrRows = (await c.query( - "SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ?", + "SELECT ord, ccr_key, content_hash, orig_hash FROM ccr_entries WHERE session_id = ?", [s.sessionId], )) as unknown as Row[][]; const changedCcrKeys: string[] = []; @@ -1023,11 +1059,21 @@ export class MysqlBackend implements StorageBackend { const releasedOldCcrHashes: string[] = []; for (const row of storedCcrRows[0]) { const key = str(row.ccr_key); + const storedOrig = row.orig_hash === null || row.orig_hash === undefined ? null : str(row.orig_hash); const inc = incomingCcr.get(key); - if (!inc) { changedCcrKeys.push(key); releasedOldCcrHashes.push(str(row.content_hash)); continue; } - if (num(row.ord) !== inc.ord || str(row.content_hash) !== inc.contentHash) { + if (!inc) { changedCcrKeys.push(key); releasedOldCcrHashes.push(str(row.content_hash)); + if (storedOrig !== null) releasedOldCcrHashes.push(storedOrig); + continue; + } + // #6785 step 6 (mirror of store.ts): orig joins the unchanged + // test — pre-step-6 rows re-put their original and self-heal. + const incOrig = inc.origContent === null ? null : inc.origHash; + if (num(row.ord) !== inc.ord || str(row.content_hash) !== inc.contentHash || storedOrig !== incOrig) { + changedCcrKeys.push(key); + releasedOldCcrHashes.push(str(row.content_hash)); + if (storedOrig !== null) releasedOldCcrHashes.push(storedOrig); } else { unchangedCcrKeys.add(key); } @@ -1045,9 +1091,10 @@ export class MysqlBackend implements StorageBackend { for (const e of input.ccrEntries) { if (unchangedCcrKeys.has(e.key)) continue; const hash = await this.putContent(c, e.content, e.contentHash); + const orig = e.origContent === null ? null : await this.putContent(c, e.origContent, e.origHash ?? undefined); await c.query( - "INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)", - [s.sessionId, e.ord, e.key, hash], + "INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash, orig_hash) VALUES (?, ?, ?, ?, ?)", + [s.sessionId, e.ord, e.key, hash, orig], ); } return true; @@ -1283,24 +1330,121 @@ export class MysqlBackend implements StorageBackend { if (rows[0].one_hash !== null && rows[0].one_hash !== undefined) await this.releaseContent(c, str(rows[0].one_hash)); await c.query("DELETE FROM blocks WHERE session_id = ? AND block_id = ?", [sessionId, blockId]); } - if (orphanContents.size > 0) { - const [sess] = (await c.query( - "SELECT orphans_json FROM sessions WHERE session_id = ?", - [sessionId], - )) as unknown as Row[][]; - if (sess.length > 0) { - let merged: Record = {}; - try { - const parsed = JSON.parse(str(sess[0].orphans_json ?? "{}")) as unknown; - if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) merged = parsed as Record; - } catch { - /* corrupt journal → start fresh */ + // #6785 step 7: the reaped-block journal lives in orphan_blocks + // now (same merge semantics as the column did — incoming wins on + // key collision, trimmed entries survive). + await this.applyOrphanBlocksOn(c, sessionId, orphanContents.size > 0 ? JSON.stringify(Object.fromEntries(orphanContents)) : null); + }); + } + + /** #6785 step 7 (MySQL twin of UnifiedStore.applyOrphanBlocks): merge + * legacy column ∪ incoming onto the orphan_blocks table, upsert each + * entry, then retire the column. */ + private async applyOrphanBlocksOn(c: PoolConnection | Pool, sessionId: string, incomingJson: string | null): Promise { + const [col] = (await c.query("SELECT orphans_json FROM sessions WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + const colRaw = col.length > 0 ? col[0].orphans_json : null; + const hadColumn = colRaw !== null && colRaw !== undefined; + const all: Record = {}; + if (hadColumn) { + try { + const parsed = JSON.parse(str(colRaw)) as unknown; + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) Object.assign(all, parsed as Record); + } catch { + /* corrupt journal → start fresh */ + } + } + if (incomingJson !== null) { + try { + const parsed = JSON.parse(incomingJson) as unknown; + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) Object.assign(all, parsed as Record); + } catch { + /* corrupt incoming → keep existing only */ + } + } + if (Object.keys(all).length === 0 && !hadColumn && incomingJson === null) return; // steady state: nothing to merge, nothing to migrate (PR#6785 review) + for (const [blockId, view] of Object.entries(all)) { + const v = view as { one?: unknown; full?: unknown }; + await c.query( + "INSERT INTO orphan_blocks (session_id, block_id, one_json, full_json) VALUES (?, ?, ?, ?) " + + "ON DUPLICATE KEY UPDATE one_json = VALUES(one_json), full_json = VALUES(full_json)", + [sessionId, blockId, v.one === undefined ? null : JSON.stringify(v.one), JSON.stringify(v.full)], + ); + } + if (hadColumn) { + await c.query("UPDATE sessions SET orphans_json = NULL WHERE session_id = ?", [sessionId]); + } + } + + /** #6785 step 7 (MySQL twin of UnifiedStore.getOrphanViews). */ + async getOrphanViews(sessionId: string): Promise | null> { + const conn = await this.pool.getConnection(); + try { + return await this.getOrphanViewsOn(conn, sessionId); + } finally { + conn.release(); + } + } + + private async getOrphanViewsOn(conn: Pool | PoolConnection, sessionId: string): Promise | null> { + const [rows] = (await conn.query("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + let out: Record | null = null; + for (const r of rows as Row[]) { + if (out === null) out = {}; + const oneRaw = r.one_json; + out[str(r.block_id)] = { + one: oneRaw === null || oneRaw === undefined ? null : JSON.parse(str(oneRaw)), + full: JSON.parse(str(r.full_json)), + }; + } + const [col] = (await conn.query("SELECT orphans_json FROM sessions WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + if (col.length > 0 && col[0].orphans_json !== null && col[0].orphans_json !== undefined) { + try { + const parsed = JSON.parse(str(col[0].orphans_json)) as Record; + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) { + for (const [id, view] of Object.entries(parsed)) { + if (out === null) out = {}; + if (out[id] === undefined) out[id] = view; } - for (const [blockId, view] of orphanContents) merged[blockId] = view; - await c.query("UPDATE sessions SET orphans_json = ? WHERE session_id = ?", [JSON.stringify(merged), sessionId]); } + } catch { + /* corrupt legacy column → the table rows remain */ } - }); + } + return out; + } + + /** #6785 step 8 (MySQL twin of UnifiedStore.applyStateFields): row-backed + * CompressionState fields — upsert on change, drop a row for a field + * that becomes undefined (bounded ≤ 9). */ + private async applyStateFieldsOn(c: PoolConnection | Pool, sessionId: string, fields: Record): Promise { + const [existing] = (await c.query("SELECT field, value_json FROM state_fields WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + for (const r of existing as Row[]) { + if (fields[str(r.field)] === undefined) { + await c.query("DELETE FROM state_fields WHERE session_id = ? AND field = ?", [sessionId, str(r.field)]); + } + } + const have = new Map((existing as Row[]).map((r) => [str(r.field), str(r.value_json)])); + for (const [field, valueJson] of Object.entries(fields)) { + if (have.get(field) === valueJson) continue; + await c.query( + "INSERT INTO state_fields (session_id, field, value_json) VALUES (?, ?, ?) " + + "ON DUPLICATE KEY UPDATE value_json = VALUES(value_json)", + [sessionId, field, valueJson], + ); + } + } + + /** #6785 step 8 (MySQL twin of UnifiedStore.getStateFields). */ + private async getStateFieldsOn(c: PoolConnection | Pool, sessionId: string): Promise | null> { + const [rows] = (await c.query("SELECT field, value_json FROM state_fields WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + if (rows.length === 0) return null; + const out: Record = {}; + for (const r of rows as Row[]) out[str(r.field)] = str(r.value_json); + return out; + } + + async getStateFields(sessionId: string): Promise | null> { + return this.getStateFieldsOn(this.pool, sessionId); } async deleteSession(sessionId: string): Promise { @@ -1315,8 +1459,11 @@ export class MysqlBackend implements StorageBackend { refs.push(str(b.summary_hash)); if (b.one_hash !== null && b.one_hash !== undefined) refs.push(str(b.one_hash)); } - const ccrRows = (await c.query("SELECT content_hash FROM ccr_entries WHERE session_id = ?", [sessionId])) as unknown as Row[][]; - for (const e of ccrRows[0]) refs.push(str(e.content_hash)); + const ccrRows = (await c.query("SELECT content_hash, orig_hash FROM ccr_entries WHERE session_id = ?", [sessionId])) as unknown as Row[][]; + for (const e of ccrRows[0]) { + refs.push(str(e.content_hash)); + if (e.orig_hash !== null && e.orig_hash !== undefined) refs.push(str(e.orig_hash)); + } const cpRows = (await c.query("SELECT cp_id, state_hash FROM checkpoints WHERE session_id = ? AND pinned = 0", [sessionId])) as unknown as Row[][]; for (const cp of cpRows[0]) { await c.query("DELETE FROM checkpoints WHERE cp_id = ?", [cp.cp_id]); @@ -1328,6 +1475,11 @@ export class MysqlBackend implements StorageBackend { await c.query("DELETE FROM dead_refs WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM refs WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM ccr_entries WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM orphan_blocks WHERE session_id = ?", [sessionId]); + // #6785 step 8: the row-backed kstate rows cascade with the + // session (sqlite twin, store.ts deleteSession) — without this + // every MySQL-mode delete leaks them forever. + await c.query("DELETE FROM state_fields WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM conversations WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM session_ext WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM sessions WHERE session_id = ?", [sessionId]); @@ -1520,6 +1672,9 @@ export class MysqlBackend implements StorageBackend { getBlocks: (id, opts) => this.getBlocksOn(conn, id, opts), getDeadRefs: (id) => this.getDeadRefsOn(conn, id), getRefs: (id) => this.getRefsOn(conn, id), + getCcrEntries: (id) => this.getCcrEntriesOn(conn, id), + getOrphanViews: (id) => this.getOrphanViewsOn(conn, id), + getStateFields: (id) => this.getStateFieldsOn(conn, id), }); } finally { try { @@ -1781,13 +1936,27 @@ export class MysqlBackend implements StorageBackend { } async getCcrEntries(sessionId: string): Promise { - const [rows] = (await this.pool.query( - "SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord", + return this.getCcrEntriesOn(this.pool, sessionId); + } + + private async getCcrEntriesOn(conn: Pool | PoolConnection, sessionId: string): Promise { + const [rows] = (await conn.query( + "SELECT ord, ccr_key, content_hash, orig_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord", [sessionId], )) as unknown as Row[][]; const out: CcrEntryRow[] = []; for (const r of rows) { - out.push({ ord: num(r.ord), key: str(r.ccr_key), content: (await this.getContent(this.pool, str(r.content_hash))) ?? "", contentHash: str(r.content_hash) }); + // #6785 step 6: null marks pre-step-6 rows (the full ccr_json + // envelope is the read authority for those stores). + const origHash = r.orig_hash === null || r.orig_hash === undefined ? null : str(r.orig_hash); + out.push({ + ord: num(r.ord), + key: str(r.ccr_key), + content: (await this.getContent(conn, str(r.content_hash))) ?? "", + contentHash: str(r.content_hash), + origContent: origHash === null ? null : ((await this.getContent(conn, origHash)) ?? null), + origHash, + }); } return out; } @@ -1810,9 +1979,15 @@ export class MysqlBackend implements StorageBackend { const blocks = await q("SELECT block_id, ord, start_seq, end_seq, start_ref, end_ref, tier, summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json FROM blocks WHERE session_id = ? ORDER BY ord"); const refsRows = await q("SELECT ord, raw_id, ref, ref_num FROM refs WHERE session_id = ? ORDER BY ord"); const deadRefs = await q("SELECT ord, ref, ref_num, recorded_at FROM dead_refs WHERE session_id = ? ORDER BY ord"); - const ccr = await q("SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord"); + const ccr = await q("SELECT ord, ccr_key, content_hash, orig_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord"); const ext = await q("SELECT ext_key AS `key`, value_json, updated_at FROM session_ext WHERE session_id = ?"); const checkpoints = await q("SELECT cp_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash FROM checkpoints WHERE session_id = ?"); + // #6785 step 7: the reaped-block journal rides the archive too. + const orphanBlocks = await q("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?"); + // #6785 step 8: the row-backed kstate fields ride the archive too + // (sqlite twin) — without them a portable restore onto a rebuilt DB + // silently drops all nine row-backed fields. + const stateFields = await q("SELECT field, value_json FROM state_fields WHERE session_id = ?"); const wanted = new Set(); for (const m of messages) wanted.add(str(m.content_hash)); for (const mm of mediaMaps) wanted.add(str(mm.media_hash)); @@ -1820,7 +1995,10 @@ export class MysqlBackend implements StorageBackend { wanted.add(str(b.summary_hash)); if (b.one_hash !== null && b.one_hash !== undefined) wanted.add(str(b.one_hash)); } - for (const e of ccr) wanted.add(str(e.content_hash)); + for (const e of ccr) { + wanted.add(str(e.content_hash)); + if (e.orig_hash !== null && e.orig_hash !== undefined) wanted.add(str(e.orig_hash)); + } for (const cp of checkpoints) if (num(cp.pinned) === 0) wanted.add(str(cp.state_hash)); const content: Record = {}; for (const hash of wanted) { @@ -1831,7 +2009,7 @@ export class MysqlBackend implements StorageBackend { // sqlite twin: appended as the last step inside the sweep // transaction, so a failed sweep leaves no line for a session that // is still active. - const archiveLine = JSON.stringify({ version: 1, sessionId, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints }) + "\n"; + const archiveLine = JSON.stringify({ version: 1, sessionId, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints, orphanBlocks, stateFields }) + "\n"; let contentRowsDeleted = 0; await this.tx(async (c) => { const rel: string[] = []; @@ -1841,7 +2019,10 @@ export class MysqlBackend implements StorageBackend { rel.push(str(b.summary_hash)); if (b.one_hash !== null && b.one_hash !== undefined) rel.push(str(b.one_hash)); } - for (const e of ccr) rel.push(str(e.content_hash)); + for (const e of ccr) { + rel.push(str(e.content_hash)); + if (e.orig_hash !== null && e.orig_hash !== undefined) rel.push(str(e.orig_hash)); + } for (const cp of checkpoints) { if (num(cp.pinned) === 0) { rel.push(str(cp.state_hash)); @@ -1854,6 +2035,10 @@ export class MysqlBackend implements StorageBackend { await c.query("DELETE FROM dead_refs WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM refs WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM ccr_entries WHERE session_id = ?", [sessionId]); + await c.query("DELETE FROM orphan_blocks WHERE session_id = ?", [sessionId]); + // #6785 step 8: the row-backed kstate rows sweep with the archive + // (sqlite twin, store.ts archiveSession). + await c.query("DELETE FROM state_fields WHERE session_id = ?", [sessionId]); await c.query("DELETE FROM session_ext WHERE session_id = ?", [sessionId]); await c.query("UPDATE sessions SET status = 'archived' WHERE session_id = ?", [sessionId]); for (const hash of rel) await this.releaseContent(c, hash); @@ -1879,6 +2064,8 @@ export class MysqlBackend implements StorageBackend { ccrEntries: Record[]; sessionExt: Record[]; checkpoints: Record[]; + orphanBlocks?: Record[]; + stateFields?: Record[]; }; // sqlite twin: newest matching line wins. let lineNo = -1; @@ -1929,7 +2116,11 @@ export class MysqlBackend implements StorageBackend { addRef(b.summary_hash); addRef(b.one_hash); } - for (const e of dump.ccrEntries) addRef(e.content_hash); + for (const e of dump.ccrEntries) { + addRef(e.content_hash); + const eh = (e as { orig_hash?: unknown }).orig_hash; + if (typeof eh === "string") addRef(eh); + } for (const cp of dump.checkpoints) { const [twins] = await c.query("SELECT 1 FROM checkpoints WHERE cp_id = ?", [cp.cp_id]); if ((twins as Row[]).length === 0) addRef(cp.state_hash); @@ -1969,7 +2160,10 @@ export class MysqlBackend implements StorageBackend { await c.query("INSERT INTO dead_refs (session_id, ord, ref, ref_num, recorded_at) VALUES (?, ?, ?, ?, ?)", [sessionId, d.ord, d.ref, d.ref_num, d.recorded_at]); } for (const e of dump.ccrEntries) { - await c.query("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)", [sessionId, e.ord, e.ccr_key, e.content_hash]); + // #6785 step 6: pre-fix dumps lack orig_hash — NULL keeps the + // full ccr_json envelope as the read authority (sqlite twin). + const eh = (e as { orig_hash?: unknown }).orig_hash; + await c.query("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash, orig_hash) VALUES (?, ?, ?, ?, ?)", [sessionId, e.ord, e.ccr_key, e.content_hash, typeof eh === "string" ? eh : null]); } for (const x of dump.sessionExt) { await c.query( @@ -1984,6 +2178,22 @@ export class MysqlBackend implements StorageBackend { [cp.cp_id, sessionId, cp.seq_at, cp.block_set_hash, cp.kind, cp.created_at, cp.pinned, cp.label ?? null, cp.state_hash], ); } + // #6785 step 7: the journal round-trips through the archive. + for (const o of dump.orphanBlocks ?? []) { + await c.query( + "INSERT IGNORE INTO orphan_blocks (session_id, block_id, one_json, full_json) VALUES (?, ?, ?, ?)", + [sessionId, o.block_id, o.one_json ?? null, o.full_json], + ); + } + // #6785 step 8: the row-backed kstate fields round-trip too + // (sqlite twin: INSERT ... ON CONFLICT DO NOTHING — an existing + // row wins, so mirror that instead of overwriting). + for (const f of dump.stateFields ?? []) { + const [existing] = await c.query("SELECT 1 FROM state_fields WHERE session_id = ? AND field = ?", [sessionId, f.field]); + if ((existing as Row[]).length === 0) { + await c.query("INSERT INTO state_fields (session_id, field, value_json) VALUES (?, ?, ?)", [sessionId, f.field, f.value_json]); + } + } await c.query( `UPDATE sessions SET parent_session_id = ?, fork_point_seq = ?, protocol = ?, label = ?, title = ?, host_title = ?, namespace = ?, origin = ?, created_at = ?, updated_at = ?, next_block_id = ?, next_run_id = ?, meta_json = ?, stats_json = ?, metadata_json = ?, kstate_json = ?, tail_json = ?, ccr_json = ?, last_ingested_seq = ?, orphans_json = ?, status = 'active' WHERE session_id = ?`, diff --git a/src/storage/schema.ts b/src/storage/schema.ts index 5efaae598c..7adb139e4f 100644 --- a/src/storage/schema.ts +++ b/src/storage/schema.ts @@ -1,3 +1,5 @@ +import { parseLegacyEnvelopeToWindow } from "../snapshot/identity.js"; + /** * Unified storage schema v1 (#2671, 6-OVERALL-DESIGN.md §schema). * @@ -83,7 +85,23 @@ // discipline as v9 (#4796); building them once costs about what // open-time quick_check pays on EVERY open (#4175). Additive-only: an // older build opening a v11 file ignores the extra indexes. -export const STORAGE_SCHEMA_VERSION = 11; +// - v12: snapshot foundation v2 (#5874 P0, pure-incremental redesign +// after the owner ruling) — ONE additive table (l1_world, the +// stage-0 identity ledger; the original four-table plane was retired +// before merge — superseded by docs/pr-6785-unified-storage-and-flows.md). +// No backfill. Additive-only: an older build opening a v12 file +// ignores the extra tables. (The fold-journal / egress-snapshot / +// scope-rev tables that once lived here were retired before merge — +// discussions/7439; the egress op-log below is self-contained.) +// Applied PROBE-FIRST (table-existence check), NOT version-gated: +// the unmerged fastpath re-land (#5855) numbers its two table groups +// v12/v13 and landed them into a SHARED database first (field, +// 2026-11-08: user_version 13 + pf_* present, l1_world absent → +// stage-0 died with "no such table: l1_world"). The probe makes +// both merge orderings coexist; whichever branch lands second in +// master still renumbers its constant mechanically (hygiene only — +// the DDL shapes are independent). +export const STORAGE_SCHEMA_VERSION = 12; /** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and * auto checkpoints move frames into the main db and reset the write offset @@ -237,6 +255,10 @@ CREATE TABLE IF NOT EXISTS ccr_entries ( ord INTEGER NOT NULL, ccr_key TEXT NOT NULL, content_hash TEXT NOT NULL REFERENCES content(hash), + -- #6785 step 6: the ORIGINAL text behind entry.hash (byHash half of the + -- content store). NULL only on pre-step-6 rows — their full ccr_json + -- envelope still carries the text. See src/storage/slim.ts. + orig_hash TEXT REFERENCES content(hash), PRIMARY KEY (session_id, ord) ); @@ -255,6 +277,10 @@ CREATE TABLE IF NOT EXISTS conversations ( PRIMARY KEY (namespace, conversation_id) ) WITHOUT ROWID; +-- #6785 review: the stage-0 plane resolves the conversation binding per turn +-- (store.ts snapSessRawOf: WHERE session_id = ?) — the PK cannot serve it. +CREATE INDEX IF NOT EXISTS idx_conversations_session ON conversations(session_id); + -- Per-session arbitrary state (#3069 v4): the escape hatch for host / -- plugin / UI state that must survive restarts but is NOT kernel session -- state. Keys are namespaced by convention ("r3:...", "web:...", @@ -307,10 +333,132 @@ CREATE INDEX IF NOT EXISTS idx_messages_content_hash ON messages(content_hash); CREATE INDEX IF NOT EXISTS idx_blocks_one_hash ON blocks(one_hash); CREATE INDEX IF NOT EXISTS idx_blocks_summary_hash ON blocks(summary_hash); CREATE INDEX IF NOT EXISTS idx_ccr_entries_content_hash ON ccr_entries(content_hash); +-- #6785 step 6: orig_hash is the SECOND FK child column of content(hash) — +-- #5539's discipline applies to it too, or every parent delete full-scans +-- ccr_entries (MySQL twin: idx_ccr_entries_orig_hash). +CREATE INDEX IF NOT EXISTS idx_ccr_entries_orig_hash ON ccr_entries(orig_hash); CREATE INDEX IF NOT EXISTS idx_message_media_media_hash ON message_media(media_hash); CREATE INDEX IF NOT EXISTS idx_checkpoints_state_hash ON checkpoints(state_hash); `; +/** #5874 snapshot foundation v2 (pure-incremental, 2026-11-15 owner ruling) — + * one additive table for the stage-0 identity ledger. The original four-table + * plane (snap_rows/snap_log/snap_ckpt, #5874 v1) was RETIRED before merge in + * favor of the pure-incremental skeleton: the per-position materialized state + * is re-derived from the append-only substrate (messages × blocks × refs × + * fold journal) instead of being stored as its own updatable table. This DDL + * stays ADDITIVE + probe-first exactly as the plane's did (T1): fresh + * installs and in-place upgrades converge on the same shape, rollback is + * DROP + user_version rewind, and a sibling branch that consumed a version + * number after v11 (real-world 2026-11-08: #5855 landed the pf_* planes) + * coexists without renumbering. + * + * l1_world: stage-0 identity ledger — the raw host session id (sess_raw) + * maps to the proxy's ledger identity (ledger_id = sessions.session_id). + * witness_probe is the sampled id fingerprint the arm decision was based on + * (composite-probe evidence). issued_at precedes armed_at until one full + * echo round validates the mapping (the "one round of arming" the proposal + * charges for a fresh identity). envelope_json/envelope_checksum carry the + * per-element msgid ENVELOPE ({id, pos} canon, #6181 §4.0) — the committed + * element-level world M2 delta inbound (src/snapshot/identity.ts matchWorld) + * reads/writes; the R3 breaker latch (r3_latched_at, #6181 M3) persists so a + * proxy restart does not re-arm a misbehaving host's envelope. + * + * #7064 msgid-storage slimming (owner ruling 2026-11-17): the per-element + * world is a BOOKKEEPING CURSOR, not a full canon — tail_json/head_json + * hold the last/first id windows of the previous turn's host array + * (WORLD_TAIL_K/WORLD_HEAD_K, src/snapshot/identity.ts), count its length, + * max_pos the append-only pos frontier. The full {id,pos} envelope + * (envelope_json/envelope_checksum) is GONE: D1 (an id verdict never + * diverges) and D2 (conversation-bound sessions skip the verdict) made the + * full canon unnecessary. Pre-slimming rows migrate on first open + * (probe below). */ +const SNAPBASE_DDL_V12 = `-- #5874 snapshot foundation v2 (P0 schema, l1_world identity ledger). +CREATE TABLE IF NOT EXISTS l1_world ( + sess_raw TEXT PRIMARY KEY, + ledger_id TEXT NOT NULL, + witness_probe TEXT NOT NULL, + issued_at INTEGER NOT NULL, + armed_at INTEGER, + tail_json TEXT, + head_json TEXT, + count INTEGER, + max_pos INTEGER, + r3_latched_at INTEGER +); +CREATE INDEX IF NOT EXISTS idx_l1_ledger ON l1_world(ledger_id); +`; + +/** #6785 step 7: the ORPHAN BLOCKS table. The orphan journal (#4216/#4242: + * blockContents entries whose block was reaped out of state.blocks) used to + * accumulate in sessions.orphans_json — an ever-growing JSON blob re-merged + * and re-written on every save of the session. It is now ONE ROW PER BLOCK + * (immutable content; a refreshed view for the same id is an UPDATE), the + * same table-ization as #6325's kstate/ccr columns. The column becomes a + * one-time migration source (legacy rows move onto the table on first + * ingest, then the column is cleared). Probed separately (probe-first, + * no version bump). */ +const ORPHAN_BLOCKS_DDL = `-- #6785 step 7: one row per reaped block's retained contents. +CREATE TABLE IF NOT EXISTS orphan_blocks ( + session_id TEXT NOT NULL, + block_id TEXT NOT NULL, + one_json TEXT, + full_json TEXT NOT NULL, + PRIMARY KEY (session_id, block_id) +); +`; + +/** #6785 step 8: one row per UNBOUNDED CompressionState field (the kstate + * slim split). The sessions.kstate_json column keeps only the bounded + * fields; these rows carry the rest (tokenSnapshot, tsStamps, viewLatches, + * absorbed, rules, …). The field set is FROZEN in src/storage/slim.ts + * (ROW_BACKED_KSTATE_FIELDS) and pinned by tests/issue5874-state-fields. + * Probe-first, no version bump. */ +const STATE_FIELDS_DDL = `-- #6785 step 8: unbounded CompressionState fields off the sessions row. +CREATE TABLE IF NOT EXISTS state_fields ( + session_id TEXT NOT NULL, + field TEXT NOT NULL, + value_json TEXT NOT NULL, + PRIMARY KEY (session_id, field) +); +`; + +/** #6785 perf (proposal discussions/7398 + revised design discussions/7439, + * owner design 2026-11-20/21): the EGRESS STATE LOG — the op-log view of the + * outbound conversation (B base / I insert / D delete-range / U + * update-content; anchors are msgids, never positions). DUAL TABLES: + * egress_state is what serving reads and compaction physically trims below + * the watermark; egress_flow is append-only and never deleted (any log_seq + * stays reconstructible — rollback is based on this table, not on client + * resends). SELF-CONTAINED BYTES (ruling 7439-1): the `bytes` column holds + * the message's final core JSON for I/U rows — no hash refs, no content + * table; replay needs only this table pair. The secondary index serves +U + * targeting and -D endpoint resolution. Probe-first, no version bump. */ +const EGRESS_STATE_DDL = `-- #6785 perf (proposals discussions/7398 + 7439): op-log outbound view (state + flow), self-contained bytes. +CREATE TABLE IF NOT EXISTS egress_state ( + session_id TEXT NOT NULL, + log_seq INTEGER NOT NULL, + row_type TEXT NOT NULL CHECK (row_type IN ('B','I','D','U')), + msgid TEXT, + payload TEXT NOT NULL, + bytes TEXT, + created_at INTEGER NOT NULL, + PRIMARY KEY (session_id, log_seq) +); +CREATE INDEX IF NOT EXISTS idx_egress_state_msgid ON egress_state (session_id, msgid); +CREATE TABLE IF NOT EXISTS egress_flow ( + session_id TEXT NOT NULL, + log_seq INTEGER NOT NULL, + row_type TEXT NOT NULL CHECK (row_type IN ('B','I','D','U')), + msgid TEXT, + payload TEXT NOT NULL, + bytes TEXT, + created_at INTEGER NOT NULL, + PRIMARY KEY (session_id, log_seq) +); +CREATE INDEX IF NOT EXISTS idx_egress_flow_msgid ON egress_flow (session_id, msgid); +`; + /** Applied inside one transaction whenever user_version < STORAGE_SCHEMA_VERSION. */ export function migrateSchema(db: { exec(sql: string): void; @@ -319,6 +467,133 @@ export function migrateSchema(db: { }): { from: number; to: number } { const row = db.prepare("PRAGMA user_version").get() as { user_version: number } | undefined; const from = row?.user_version ?? 0; + // #6785 review fix (owner): DROP COLUMN needs SQLite ≥ 3.35. The host + // better-sqlite3 v7 in the engine ladder can carry SQLite 3.32 — there + // the redundant columns simply STAY (every reader probes table_info and + // every INSERT/UPDATE names its columns explicitly, so a kept column is + // inert). Never throw from a migration because the host engine is old. + let dropColumnOk = false; + try { + const v = (db.prepare("PRAGMA sqlite_version").get() as { sqlite_version: string } | undefined)?.sqlite_version ?? ""; + const m = v.match(/^(\d+)\.(\d+)/); + if (m) dropColumnOk = Number(m[1]) > 3 || (Number(m[1]) === 3 && Number(m[2]) >= 35); + } catch { /* leave false — keep the columns */ } + // v12 (#5874) is PROBE-FIRST, not version-gated. A sibling branch may + // have consumed the version numbers after v11 for its own additive DDL + // while ours was still unmerged — real-world 2026-11-08: #5855 landed + // v12+v13 (the pf_* planes) into a SHARED database, and this branch's + // old `if (from < 12)` block was skipped by the early return below, so + // the plane died on first save with "no such table: l1_world". The DDL + // is pure CREATE ... IF NOT EXISTS (no backfill, no rewrite of existing + // rows), so it is safe to run at ANY from-version; probing for the + // table itself (the discipline v2/v5/v6/v8 already use) makes both + // merge orderings coexist without renumbering — the final master + // version number stays a mechanical merge-time hygiene step. + const hasL1World = db + .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'l1_world'") + .get(); + if (!hasL1World) db.transaction(() => db.exec(SNAPBASE_DDL_V12)); + // #6785 step 7: orphan blocks — own probe (a db written between the + // egress_snapshot and orphan_blocks landings picks it up here; fresh dbs + // pass through — same discipline, no version bump). + const hasOrphanBlocks = db + .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'orphan_blocks'") + .get(); + if (!hasOrphanBlocks) db.exec(ORPHAN_BLOCKS_DDL); + // #6785 step 8: kstate rows — own probe (same discipline, no bump). + const hasStateFields = db + .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'state_fields'") + .get(); + if (!hasStateFields) db.exec(STATE_FIELDS_DDL); + // #6785 perf (proposal discussions/7398): the egress state log — own + // probe (same discipline, no version bump). + const hasEgressState = db + .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'egress_state'") + .get(); + if (!hasEgressState) db.exec(EGRESS_STATE_DDL); + // 7439 ruling 1: a dev DB created before the self-contained-bytes column + // upgrades with one probe-first ALTER (additive; the tables never + // shipped, so there is no real-migration burden). + const egressCols = (db.prepare("PRAGMA table_info(egress_state)").all() as unknown as { name?: unknown }[]) + .map((c) => c.name); + if (egressCols.length > 0 && !egressCols.includes("bytes")) { + db.exec("ALTER TABLE egress_state ADD COLUMN bytes TEXT"); + db.exec("ALTER TABLE egress_flow ADD COLUMN bytes TEXT"); + } + // 2×2 (#5874 brief, element-level world): the L1 world carries the + // per-element msgid ENVELOPE ({id, pos} canon, §4.0) in addition to the + // session-level witness_probe. Pure additive probe-first ALTER (a dev + // window that created l1_world before this column exists upgrades with + // one ALTER; a fresh DB already has it from the DDL above). The 2×2 + // stage-0 (src/snapshot/identity.ts matchWorld) reads/writes these; the + // session-level v1 path (#5901) leaves them NULL and is unaffected. + const l1Cols = (db.prepare("PRAGMA table_info(l1_world)").all() as unknown as { name?: unknown }[]) + .map((c) => c.name); + if (l1Cols.length > 0) { + if (!l1Cols.includes("tail_json")) db.exec("ALTER TABLE l1_world ADD COLUMN tail_json TEXT"); + if (!l1Cols.includes("head_json")) db.exec("ALTER TABLE l1_world ADD COLUMN head_json TEXT"); + if (!l1Cols.includes("count")) db.exec("ALTER TABLE l1_world ADD COLUMN count INTEGER"); + if (!l1Cols.includes("max_pos")) db.exec("ALTER TABLE l1_world ADD COLUMN max_pos INTEGER"); + // #7064 slimming migration: a pre-slimming row's full envelope + // (envelope_json) reshapes onto the window columns once — the + // checksum column is dropped with it (D1: no verdict reads it). + // Rows whose JSON does not parse degrade to a BLANK cursor (the + // next turn re-issues the world; D1: bookkeeping, never a reject). + if (l1Cols.includes("envelope_json")) { + const legacy = (db.prepare( + "SELECT sess_raw, envelope_json FROM l1_world WHERE envelope_json IS NOT NULL AND (tail_json IS NULL OR count IS NULL)" + ).all() as unknown as { sess_raw: string, envelope_json: string | null }[]); + for (const row of legacy) { + let tail = "[]"; + let head = "[]"; + let count = 0; + let maxPos = 0; + try { + const win = parseLegacyEnvelopeToWindow(row.envelope_json ?? "[]"); + tail = JSON.stringify(win.tailIds); + head = JSON.stringify(win.headIds); + count = win.count; + maxPos = win.maxPos; + } catch { + // blank cursor — world re-issues on the next turn + } + db.prepare( + "UPDATE l1_world SET tail_json = ?, head_json = ?, count = ?, max_pos = ? WHERE sess_raw = ?" + ).run(tail, head, count, maxPos, row.sess_raw); + } + // #6785 review fix (owner): on a < 3.35 host the reshaped data + // stays in the NEW columns and the legacy columns are kept + // (inert — readers probe table_info; the legacy SELECT above is + // re-run only while rows still lack tail_json, which it now has). + if (dropColumnOk) { + db.exec("ALTER TABLE l1_world DROP COLUMN envelope_json"); + if (l1Cols.includes("envelope_checksum")) db.exec("ALTER TABLE l1_world DROP COLUMN envelope_checksum"); + } + } + // #6181 M3: the R3 breaker latch PERSISTS (a proxy restart must not + // re-arm a misbehaving host's sid and re-400 the same broken + // envelope); NULL/0 = not latched, else the epoch-ms of the latch. + if (!l1Cols.includes("r3_latched_at")) db.exec("ALTER TABLE l1_world ADD COLUMN r3_latched_at INTEGER"); + } + // #6785 step 6: ccr_entries.orig_hash — PROBE-FIRST, not version-gated: + // a db whose user_version is already at the current version (the #5855 + // field build wrote v12/v13 into shared databases) takes the early + // return below and would never pick the column up if the probe sat + // inside the transaction. Every pre-existing db must get it, so probe + // unconditionally; a fresh db's table does not exist yet at this point + // (PRAGMA table_info → empty) and the base DDL above already carries + // the column. Pre-step-6 rows keep NULL; their full ccr_json envelope + // remains the read authority (slim.ts). The index rides the same probe + // (#5539 FK-child discipline; a lib that already carried the column but + // not the key converges on the idempotent CREATE). + const ccrCols = (db.prepare("PRAGMA table_info(ccr_entries)").all() as unknown as { name?: unknown }[]) + .map((c) => (typeof c?.name === "string" ? c.name : "")); + if (ccrCols.length > 0) { + if (!ccrCols.includes("orig_hash")) { + db.exec("ALTER TABLE ccr_entries ADD COLUMN orig_hash TEXT REFERENCES content(hash)"); + } + db.exec("CREATE INDEX IF NOT EXISTS idx_ccr_entries_orig_hash ON ccr_entries(orig_hash)"); + } if (from >= STORAGE_SCHEMA_VERSION) return { from, to: from }; let needV8Backfill = false; db.transaction(() => { @@ -430,6 +705,10 @@ export function migrateSchema(db: { db.exec("ALTER TABLE messages ADD COLUMN ts INTEGER"); } } + // #6785 step 6: ccr_entries.orig_hash — probe-first, moved ABOVE the + // version early return (a #5855-era db already at user_version 12 + // must pick the column up; the in-transaction probe it was at never + // ran there). See the probe-first note before the early return. if (from < 11) { // v11 (#5539): FK child indexes on content.hash — see the header // note and the DDL comment above for the mechanism and field @@ -442,6 +721,12 @@ export function migrateSchema(db: { db.exec("CREATE INDEX IF NOT EXISTS idx_message_media_media_hash ON message_media(media_hash)"); db.exec("CREATE INDEX IF NOT EXISTS idx_checkpoints_state_hash ON checkpoints(state_hash)"); } + // v12 (#5874 P0): snapshot-table compression foundation — ensured + // PROBE-FIRST above (coexists with sibling-branch DDL that consumed + // the v12/v13 numbers, see the note before the early return). + // Pure additivity: no backfill, no rewrite of existing rows. + // Rollback = DROP the tables and set user_version back + // (T1 pins the byte-identical behavior in both directions). }); if (needV8Backfill) migrateV8Identities(db); db.exec(`PRAGMA user_version = ${STORAGE_SCHEMA_VERSION}`); diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 0e306de770..4dc7a0a6c0 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -26,6 +26,7 @@ import { log as loggerLog } from "../logger.js"; import { planIngest, planIngestDelta, loadLegacyView, LegacyViewBuilder, appendBlockContents, mergeOrphanBlockContents, completeBlockContents, type LegacySessionLike } from "./ingest.ts"; +import type { CcrLedgerRow } from "./slim.ts"; import { splitMediaContent } from "./identity.js"; import type { BlockView } from "../session.js"; import { resolveStorageMode, resolveStorageWalJournalSizeLimit, effectiveJsonWriterOff, type StorageMode } from "../knobs.ts"; @@ -397,7 +398,9 @@ export class UnifiedShadow { let store: UnifiedStore; try { try { - store = UnifiedStore.open(`${dir}/index.db`, undefined, { walJournalSizeLimitBytes: walLimit }); + store = UnifiedStore.open(`${dir}/index.db`, undefined, { + walJournalSizeLimitBytes: walLimit, + }); } catch (err) { // #4175: anything else throwing StorageIntegrityError means quick_check // proved the file set corrupt — handled below WITHOUT moving any file @@ -678,7 +681,7 @@ export class UnifiedShadow { this.stats.verifySamples++; const sessionRow = this.sync.getSessionRow(legacy.id); if (!sessionRow) throw new Error(`session row missing after ingest: ${legacy.id}`); - const verifier = new BatchedViewVerification(legacy, sessionRow, this.sync.getRefs(legacy.id), this.sync.getDeadRefs(legacy.id)); + const verifier = new BatchedViewVerification(legacy, sessionRow, this.sync.getRefs(legacy.id), this.sync.getDeadRefs(legacy.id), this.sync.getCcrEntries(legacy.id), this.sync.getStateFields(legacy.id), this.sync.getOrphanViews(legacy.id) ?? {}); for (let offset = 0; ; offset += VERIFY_BATCH_SIZE) { const rows = this.sync.getBlocks(legacy.id, { limit: VERIFY_BATCH_SIZE, offset }); verifier.addBlocks(rows); @@ -835,6 +838,13 @@ export class UnifiedShadow { for (let offset = 0; offset < builder.messageSeqs.length; offset += VIEW_READ_BATCH_SIZE) { builder.addMessages(store.getMessages(id, { seqs: builder.messageSeqs.slice(offset, offset + VIEW_READ_BATCH_SIZE) })); } + // #6785 step 6: slim ccr envelopes hydrate from the CAS-backed + // ledger; full legacy envelopes ignore these rows. + builder.addCcrEntries(store.getCcrEntries(id)); + // #6785 step 7: the orphan journal now lives in orphan_blocks; + // getOrphanViews also unions any legacy column entries. + builder.addOrphanViews(store.getOrphanViews(id) ?? {}); + builder.addStateFields(store.getStateFields(id) ?? {}); // The builder preserves every MERGE_STATE_FIELDS key; its // generic state declaration only needs widening here. return builder.finish() as LegacySessionLike; @@ -870,7 +880,7 @@ export class UnifiedShadow { appendBlockContents(contents, blocks); if (blocks.length < VIEW_READ_BATCH_SIZE) break; } - mergeOrphanBlockContents(contents, store.getOrphansJson(id)); + mergeOrphanBlockContents(contents, JSON.stringify(store.getOrphanViews(id) ?? {})); result.blockContents = contents; } return result; @@ -1100,6 +1110,75 @@ export class UnifiedShadow { return this.sync.sessionStorageStats(sessionId); } + // #5874 2×2 stage 0 (brief 19544722): the element world-match accessors. + // Errors propagate (the lane maps the R3 reject to a hard 400 and every + // other fault to the legacy full path). + + /** Stage 0 for one turn (see UnifiedStore.snapStage0). null when there + * is no sync store. */ + snapStage0(sessionId: string, incoming: string[], opts?: { resync?: boolean, seqs?: number[] | null }): import("../snapshot/plane.js").Stage0Result | null { + if (this.sync === null) return null; + return this.sync.snapStage0(sessionId, incoming, opts); + } + + /** The stage-0 resync (diverge / re-baseline): clear the l1 envelope so + * the next turn re-issues from the client's current array. */ + snapStage0Reset(sessionId: string): void { + if (this.sync === null) return; + this.sync.snapStage0Reset(sessionId); + } + + /** #7064 D2: conversation-bound check (see UnifiedStore.snapConvBound). */ + snapConvBound(sessionId: string): boolean { + if (this.sync === null) return false; + return this.sync.snapConvBound(sessionId); + } + + /** #7064 §8: live stamped msg_ids in seq order (see UnifiedStore.snapMsgIdLiveIds). */ + snapMsgIdLiveIds(sessionId: string): string[] { + if (this.sync === null) return []; + return this.sync.snapMsgIdLiveIds(sessionId); + } + + /** #7398 seg 3: the sqlite UnifiedStore behind this shadow — the egress + * op-log recorder's narrow seam. Null on network backends (mysql): the + * shadow tables are sqlite-lane only, and the recorder skips the turn + * exactly as it does when no sidecar opened. */ + egressOpLogSync(): UnifiedStore | null { + return this.sync; + } + + /** #6181 M3: persisted R3 breaker latch (see UnifiedStore.snapR3Latched). */ + snapR3Latched(sessionId: string): boolean { + if (this.sync === null) return false; + return this.sync.snapR3Latched(sessionId); + } + + /** #6181 M3: persist the R3 breaker latch (best-effort). */ + snapR3Latch(sessionId: string): void { + if (this.sync === null) return; + this.sync.snapR3Latch(sessionId); + } + + /** #6785 review: operator escape hatch (see UnifiedStore.snapR3LatchClear). */ + snapR3LatchClear(sessRaw: string): string | null { + if (this.sync === null) return null; + return this.sync.snapR3LatchClear(sessRaw); + } + + /** #6785 review: the persisted-latch roster (see UnifiedStore.snapR3LatchedList). */ + snapR3LatchedList(): Array<{ sessRaw: string; ledgerId: string; latchedAt: number }> { + if (this.sync === null) return []; + return this.sync.snapR3LatchedList(); + } + + /** #6785 review: conversation-binding sid of a raw id (see UnifiedStore.snapSidOfRaw). */ + snapSidOfRaw(sessRaw: string): string | null { + if (this.sync === null) return null; + return this.sync.snapSidOfRaw(sessRaw); + } + + vacuum(mode: "full" | "incremental"): void { if (this.sync === null) throw new Error("[storage] sync vacuum unavailable on a network backend"); return this.sync.vacuum(mode); @@ -1326,7 +1405,6 @@ export class UnifiedShadow { legacy = { ...legacy, lastMessages: rawTail.lastMessages, lastMessagesFolded: rawTail.lastMessagesFolded === true }; } if (liveCcr) legacy = { ...legacy, contentStore: liveCcr }; - // #3818/#3841: same orphan-guard + reap-journal semantics. // #4216/#4242: orphan shape rides the plan journal — normal // ingest (the async twin of the sync fix); no freeze. // #3133 watermark delta. @@ -1376,7 +1454,7 @@ export class UnifiedShadow { this.stats.verifySamples++; const sessionRow = await backend.getSessionRow(legacy.id); if (!sessionRow) throw new Error(`session row missing after ingest: ${legacy.id}`); - const verifier = new BatchedViewVerification(legacy, sessionRow, await backend.getRefs(legacy.id), await backend.getDeadRefs(legacy.id)); + const verifier = new BatchedViewVerification(legacy, sessionRow, await backend.getRefs(legacy.id), await backend.getDeadRefs(legacy.id), await backend.getCcrEntries(legacy.id), await backend.getStateFields(legacy.id), (await backend.getOrphanViews(legacy.id)) ?? {}); for (let offset = 0; ; offset += VERIFY_BATCH_SIZE) { const rows = await backend.getBlocks(legacy.id, { limit: VERIFY_BATCH_SIZE, offset }); verifier.addBlocks(rows); @@ -1425,6 +1503,12 @@ export class UnifiedShadow { for (let offset = 0; offset < builder.messageSeqs.length; offset += VIEW_READ_BATCH_SIZE) { builder.addMessages(await reader.getMessages(id, { seqs: builder.messageSeqs.slice(offset, offset + VIEW_READ_BATCH_SIZE) })); } + // #6785 step 6: same slim-envelope hydration on the async + // (MySQL seam) leg — one source, both legs. + builder.addCcrEntries(await reader.getCcrEntries(id)); + // #6785 step 7: same orphan-journal union on the async leg. + builder.addOrphanViews(await reader.getOrphanViews(id) ?? {}); + builder.addStateFields(await reader.getStateFields(id) ?? {}); return builder.finish() as LegacySessionLike; }); if (!loaded) return null; @@ -1692,8 +1776,8 @@ class BatchedViewVerification { private tailGaps = 0; private tailSlot: DriftDiagnostics["tailSlot"]; - constructor(private readonly legacy: LegacySessionLike, session: SessionRow, refs: RefRow[], deadRefs: { ord: number; ref: string }[]) { - const header = loadLegacyView({ session, refs, deadRefs, messages: [], blocks: [] }); + constructor(private readonly legacy: LegacySessionLike, session: SessionRow, refs: RefRow[], deadRefs: { ord: number; ref: string }[], ccrRows: readonly CcrLedgerRow[] = [], stateFieldsRows: Record | null = null, orphanViews: Record | null = null) { + const header = loadLegacyView({ session, refs, deadRefs, messages: [], blocks: [] }, ccrRows, stateFieldsRows, orphanViews ?? undefined); this.orphans = header.blockContents; this.remainingContents = new Map(completeBlockContents(legacy.blockContents, legacy.state.blocks)); this.tail = JSON.parse(session.tailJson); diff --git a/src/storage/slim.ts b/src/storage/slim.ts new file mode 100644 index 0000000000..38a4b1e673 --- /dev/null +++ b/src/storage/slim.ts @@ -0,0 +1,185 @@ +import { sha256Hex } from "./store.js"; +import type { MessageContentStore, StoredEntry } from "acp-kernel"; + +/** + * #6785 step 6 — the slim ccr envelope. + * + * The sessions.ccr_json column used to carry the FULL content store + * (byHash: hash → original text, byRef: ref → entry), so every + * steady-state flush re-wrote the whole CCR body through the sessions + * row even when nothing changed. The content now lives in the + * content-addressed store behind ccr_entries — one row per entry with + * BOTH halves: the entry metadata (content_hash) and the original text + * it points at (orig_hash, #6785 step 6). The column therefore only + * needs the ENVELOPE: which refs exist and which content-hash backs + * each entry. Bytes come back through the CAS on read. + * + * Kernel invariant this leans on (content-store.ts storeOriginal): + * byHash[entry.hash] === original text, entry.hash = hashContent(text). + */ +interface SlimCcrEnvelope { + version: 1; + ccrSlim: 1; + byRef: Record; + /** byHash entries no byRef entry points at (host-pruned refs — the + * "retained CCR" case). Rare and small; they ride the envelope so the + * slim round-trip stays lossless in both halves. */ + orphanByHash?: Record; +} + +export function slimCcrEnvelope(contentStore: { byRef: Record; byHash?: Record }): SlimCcrEnvelope { + const byRef: Record = {}; + for (const [key, value] of Object.entries(contentStore.byRef)) { + byRef[key] = { hash: sha256Hex(JSON.stringify(value)) }; + } + // "referenced" = the original hashes some entry points at (entry.hash, + // kernel invariant) — NOT the envelope's content-hash values. + const referenced = new Set( + Object.values(contentStore.byRef) + .map((v) => (v !== null && typeof v === "object" && typeof (v as { hash?: unknown }).hash === "string" ? (v as { hash: string }).hash : null)), + ); + const orphanByHash: Record = {}; + for (const [hash, text] of Object.entries(contentStore.byHash ?? {})) { + if (!referenced.has(hash)) orphanByHash[hash] = text; + } + return Object.keys(orphanByHash).length > 0 + ? { version: 1, ccrSlim: 1, byRef, orphanByHash } + : { version: 1, ccrSlim: 1, byRef }; +} + +export interface CcrLedgerRow { + key: string; + /** JSON.stringify(byRef[key]) as planIngest wrote it. */ + content: string; + /** The original text behind entry.hash (CAS-served); null when the + * row predates the orig_hash column (legacy rows — the full column + * envelope still carries those stores). */ + origContent: string | null; +} + +/** + * Rebuilds the content store from ccr_entries rows (CAS-served content) in + * ledger order. byRef[key] is the parsed entry — the row content IS + * JSON.stringify(value) as planIngest wrote it, so the round-trip is exact. + * byHash is rebuilt from the SAME rows through each entry's own hash key, + * matching the live store's canonical form (the kernel invariant above). + * A legacy row (origContent null) contributes its byRef entry only — the + * full column envelope remains the authority for pre-step-6 stores, so no + * partial byHash is ever mixed with a slim one. Returns null when there + * are no rows at all — callers degrade to the file lane, never a partial + * store. + */ +function hydrateCcrContentStore( + rows: readonly CcrLedgerRow[], +): MessageContentStore | null { + if (rows.length === 0) return null; + const byHash: Record = {}; + const byRef: Record = {}; + for (const r of rows) { + let entry: unknown; + try { + entry = JSON.parse(r.content); + } catch { + continue; + } + byRef[r.key] = entry as StoredEntry; + if ( + r.origContent !== null + && entry !== null + && typeof entry === "object" + && typeof (entry as { hash?: unknown }).hash === "string" + ) { + byHash[(entry as { hash: string }).hash] = r.origContent; + } + } + return { version: 1, byHash, byRef }; +} + +/** + * The ONE definition of "rebuild a slim column's store" — used by BOTH the + * view leg (resolveCcrContentStore) and the direct getCcr path. Legacy + * NULL-orig rows, an EMPTY ledger (unless the orphans half alone forms a + * real store), a MISSING content body, or an envelope/ledger KEY-SET MISMATCH + * ⇒ null (the file lane stays authority; never a partial store). Orphan + * byHash entries ride the envelope and are merged over the ledger's byHash. + */ +export function hydrateSlimCcrStore( + parsed: { ccrSlim?: number; orphanByHash?: unknown; byRef?: Record }, + rows: readonly CcrLedgerRow[], +): MessageContentStore | null { + if (parsed.ccrSlim !== 1) return null; + const orphans: Record = + parsed.orphanByHash !== undefined && parsed.orphanByHash !== null && typeof parsed.orphanByHash === "object" + ? (parsed.orphanByHash as Record) + : {}; + // A ledger row whose CAS body was lost (getCcrEntries maps a missing + // body to "") would otherwise be silently DROPPED by the JSON.parse + // catch below — serving a partial store. Degrade instead. + if (rows.some((r) => r.content === "")) return null; + // Envelope/ledger key-set coverage: every byRef key the column + // declares must have a ledger row, or the hydrated byRef is a strict + // subset of the column's — a partial store the file lane would serve + // whole. (Column and ledger are written in one tx; this is the + // read-side safety net for the invariant, not a fast path.) + if (parsed.byRef !== undefined && parsed.byRef !== null) { + const rowKeys = new Set(rows.map((r) => r.key)); + for (const key of Object.keys(parsed.byRef)) { + if (!rowKeys.has(key)) return null; + } + } + if (rows.length === 0) { + // Empty envelope + empty ledger is the ALWAYS-empty store: server.ts + // materializes createContentStore() ({byHash:{},byRef:{}}) for every + // session (contentStoreOf ?? fallback), so the column carries a slim + // envelope with zero refs for every plain session. The envelope's own + // key-set coverage check above already proved nothing is declared, + // so this is a COMPLETE empty store, not a partial one. + return { version: 1, byHash: orphans, byRef: {} }; + } + if (!rows.every((r) => r.origContent !== null)) return null; + const store = hydrateCcrContentStore(rows); + if (!store) return null; + store.byHash = { ...store.byHash, ...orphans }; + return store; +} + +/** + * #6785 step 8: the kstate slim split. The sessions.kstate_json column keeps + * only the BOUNDED fields (nudge, stats, latches, counters, digests); the + * UNBOUNDED record/array fields (each can grow to MB across a long session) + * ride one row each in state_fields. The set is frozen: adding a field here + * is a read-path decision (the builder must know it is row-backed), so it is + * pinned by tests/issue5874-state-fields.test.ts and reviewed with the + * kernel-type pin (kernel/src/types.ts field shapes). + */ +export const ROW_BACKED_KSTATE_FIELDS: readonly string[] = Object.freeze([ + "tokenSnapshot", + "tsStamps", + "lastPassIds", + "hiddenOrphanRefs", + "sweptOrphanCallIds", + "renderDeferredRefs", + "viewLatches", + "absorbed", + "rules", +]); + +const ROW_BACKED_SET = new Set(ROW_BACKED_KSTATE_FIELDS); + +/** Split an ordered kstate object (MERGE_STATE_FIELDS order) into the + * column part (order-preserving, row-backed fields removed) and the row + * part (field → serialized value, only for DEFINED row-backed fields). */ +export function splitKstate( + fields: readonly string[], + state: Record, +): { column: Record; rows: Record } { + const column: Record = {}; + const rows: Record = {}; + for (const field of fields) { + const v = state[field]; + if (v === undefined) continue; + if (ROW_BACKED_SET.has(field)) rows[field] = JSON.stringify(v); + else column[field] = v; + } + return { column, rows }; +} diff --git a/src/storage/snap.ts b/src/storage/snap.ts new file mode 100644 index 0000000000..d313cdf839 --- /dev/null +++ b/src/storage/snap.ts @@ -0,0 +1,215 @@ +/** + * snap.ts — #5874 snapshot foundation v2 (pure-incremental, 2026-11-15 + * owner ruling): the stage-0 IDENTITY ledger only. + * + * The original four-table plane (l1_world / snap_rows / snap_log / + * snap_ckpt) was retired before merge: the per-position materialized state + * (snap_rows), the message-level event stream (snap_log) and the fold + * checkpoint (snap_ckpt) are replaced by the append-only substrate + * (messages × blocks × refs). The only plane piece that survives verbatim + * is the L1 world table below: the stage-0 identity + * gate (echo/probe/issue, the witness probe, the R3 breaker latch, the + * element-level msgid envelope for M2 delta inbound) is independent of how. + * (fold_journal / egress_snapshot / egress_scope_rev once shipped below + * were retired before merge, discussions/7439 — the egress op-log is now + * self-contained and the flow tables are the only egress state.) + * the compressed state is stored, and it is what makes M2 parse only new + * spans in the first place. + * + * Zero kernel changes remain a hard constraint; this file does no + * compression work and never serves bytes — it decides identity only. + */ + +import { createHash } from "node:crypto"; +import type { WorldWindow } from "../snapshot/types.js"; + +/** Minimal db surface so the identity ledger stays type-coupled to the + * storage driver without importing it (same contract as the driver's + * prepared-statement API). */ +export interface DbLike { + exec(sql: string, ...params: unknown[]): unknown; + prepare(sql: string): { + run(...params: unknown[]): unknown; + get(...params: unknown[]): unknown; + all(...params: unknown[]): unknown[]; + }; + /** Same contract as the storage driver's transaction: runs fn inside + * BEGIN/COMMIT and returns fn's own result (nested calls savepoint). */ + transaction(fn: () => T): T; +} + +// --------------------------------------------------------------------------- +// L1 world — stage-0 identity ledger (proposal §0). +// +// sess_raw = the raw host session id as it arrived on the wire +// (conversations.conversation_id in v1); ledger_id = the proxy's own +// identity (sessions.session_id). The mapping is issued on first sight +// (issued_at) and ARMED only after one full round validates it +// (armed_at): the validation evidence is that a later round for the same +// ledger re-presents the same sess_raw with matching witness content — +// the "one round of arming" the proposal charges. witness_probe is a +// sampled fingerprint of the ids seen at issue time, so the arm check is +// cheap and does not store payload. +// --------------------------------------------------------------------------- + +interface L1Entry { + sessRaw: string; + ledgerId: string; + witnessProbe: string; + issuedAt: number; + armedAt: number | null; +} + +/** Record (or refresh) the sess_raw → ledger_id mapping. Returns the + * decision: + * - "hit": mapping existed and is armed — the round is identity- + * validated (the normal path for a live cohort session); + * - "echo": mapping existed and is armed, witness matched — same as + * hit, distinguished for the decision log (the echo + * spot-check the proposal mentions); + * - "arm": mapping existed, not yet armed, and this round's witness + * validates it — armed_at set now (the one-round charge + * completes); + * - "wait": mapping existed, not yet armed, witness did not validate + * — issued_at unchanged, no arm (still one round out); + * - "issue": new mapping issued (armed_at = NULL — the round that + * issued it cannot validate itself). + * A sess_raw that was issued under a DIFFERENT ledger_id is a hard + * identity conflict (R3 territory): returns "red" and writes NOTHING — + * the caller keeps the round on the legacy path and the breaker counts + * it. */ +export function snapL1Record( + db: DbLike, + sessRaw: string, + ledgerId: string, + witnessProbe: string, + now: number, +): "hit" | "echo" | "arm" | "wait" | "issue" | "red" { + const existing = db + .prepare("SELECT ledger_id, witness_probe, issued_at, armed_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { ledger_id: string; witness_probe: string; issued_at: number; armed_at: number | null } | undefined; + if (!existing) { + db.prepare( + "INSERT INTO l1_world (sess_raw, ledger_id, witness_probe, issued_at, armed_at) VALUES (?, ?, ?, ?, NULL)", + ).run(sessRaw, ledgerId, witnessProbe, now); + return "issue"; + } + if (existing.ledger_id !== ledgerId) { + // Hard conflict: the same raw session id now belongs to another + // ledger. Never overwrite — the mapping is the identity anchor; + // corrupting it would poison every later round for this host. + return "red"; + } + if (existing.armed_at !== null) { + return existing.witness_probe === witnessProbe ? "echo" : "hit"; + } + // Not armed yet: arm when this round re-validates the probe (the + // witness is content-derived, so an identical probe means the host's + // id space survived the round unchanged — the echo round). + if (witnessProbe === existing.witness_probe) { + db.prepare("UPDATE l1_world SET armed_at = ? WHERE sess_raw = ?").run(now, sessRaw); + return "arm"; + } + return "wait"; +} + +/** Read the L1 entry for a raw session id (status/diagnostics). */ +export function snapL1Lookup(db: DbLike, sessRaw: string): L1Entry | null { + const r = db + .prepare("SELECT sess_raw, ledger_id, witness_probe, issued_at, armed_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { sess_raw: string; ledger_id: string; witness_probe: string; issued_at: number; armed_at: number | null } | undefined; + if (!r) return null; + return { sessRaw: r.sess_raw, ledgerId: r.ledger_id, witnessProbe: r.witness_probe, issuedAt: r.issued_at, armedAt: r.armed_at }; +} + +// --------------------------------------------------------------------------- +// 2×2 (proposal §4.0) — the element-level WORLD CURSOR on the same row. +// +// The session-level path above (snapL1Record/snapL1Lookup) is the identity +// ANCHOR (sess_raw → ledger_id + one-round arming). The 2×2 stage 0 adds, +// on the same row, the per-element world cursor: the head + tail id windows +// of the previous turn's host array plus its length and the append-only pos +// frontier (#7064 slimming — the full {id,pos} envelope is gone). The +// stage-0 decision function (src/snapshot/identity.ts matchWindow) runs on +// this cursor; M2 delta inbound appends onto it. +// --------------------------------------------------------------------------- + +/** Read the element-level world cursor for a raw session id (null when the + * l1 row is absent; window null when present but not yet issued). */ +export function snapL1WindowRead(db: DbLike, sessRaw: string): { window: WorldWindow | null, issuedAt: number } | null { + const r = db + .prepare("SELECT tail_json, head_json, count, max_pos, issued_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { tail_json: string | null; head_json: string | null; count: number | null; max_pos: number | null; issued_at: number } | undefined; + if (!r) return null; + if (r.tail_json === null || r.count === null) return { window: null, issuedAt: r.issued_at }; + return { + window: { + tailIds: JSON.parse(r.tail_json) as string[], + headIds: JSON.parse(r.head_json ?? "[]") as string[], + count: r.count, + maxPos: r.max_pos ?? r.count, + }, + issuedAt: r.issued_at, + }; +} + +/** Persist the element-level world cursor for a raw session id (UPDATE only + * — the l1 row itself is created by snapL1Record; a missing row is a + * programming error and is surfaced, not silently created). */ +export function snapL1WindowUpsert(db: DbLike, sessRaw: string, window: WorldWindow): void { + const res = db + .prepare("UPDATE l1_world SET tail_json = ?, head_json = ?, count = ?, max_pos = ? WHERE sess_raw = ?") + .run(JSON.stringify(window.tailIds), JSON.stringify(window.headIds), window.count, window.maxPos, sessRaw) as { changes: number }; + if (res.changes === 0) throw new Error(`snapL1WindowUpsert: no l1_world row for sess_raw ${sessRaw.slice(0, 12)}… (record it first)`); +} + +/** #6181 M3: read the persisted R3 breaker latch (true when the sid's + * breaker latched in a PREVIOUS process — a restart must not re-arm a + * misbehaving host). Missing row or NULL → false. */ +export function snapR3LatchedRead(db: DbLike, sessRaw: string): boolean { + const r = db + .prepare("SELECT r3_latched_at FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { r3_latched_at: number | null } | undefined; + return !!r && !!r.r3_latched_at; +} + +/** #6181 M3: persist the R3 breaker latch. UPSERT (not UPDATE-only) — the + * first fault of a fresh session happens BEFORE its stage-0 snapL1Record + * call (the R3 check runs at pin validation), so an UPDATE-only latch + * would silently no-op for sessions with no l1 row yet. */ +export function snapR3LatchWrite(db: DbLike, sessRaw: string, nowMs: number): void { + // Upsert — the first fault of a fresh session happens BEFORE its stage-0 + // snapL1Record call (the R3 check runs at pin validation), so an + // UPDATE-only latch would silently no-op for sessions with no l1 row yet. + // ledger_id takes the placeholder sessRaw and witness_probe/issued_at the + // sentinel values: the latch row's envelope stays NULL until a later + // (repaired) stage-0 snapL1Record upsert carries the real ledger id and + // witness. A latched session never re-arms (runStage0 checks the persisted + // latch before any store call), so the placeholder can never hit the + // "red" ledger-conflict path of snapL1Record. + db.prepare("INSERT INTO l1_world (sess_raw, ledger_id, witness_probe, issued_at, r3_latched_at) VALUES (?, ?, '', ?, ?) ON CONFLICT(sess_raw) DO UPDATE SET r3_latched_at = excluded.r3_latched_at, ledger_id = COALESCE(l1_world.ledger_id, excluded.ledger_id), witness_probe = COALESCE(NULLIF(l1_world.witness_probe, ''), excluded.witness_probe)").run(sessRaw, sessRaw, nowMs, nowMs); +} + +/** #6785 review: operator escape hatch — clear the persisted R3 latch. + * Returns the ledger id the row binds (the placeholder `sessRaw` until a + * later stage-0 snapL1Record upsert carries the real id) so the caller can + * also clear the in-memory breaker for every resolvable sid; null when no + * l1 row exists for the raw id. */ +export function snapR3LatchClear(db: DbLike, sessRaw: string): string | null { + const r = db.prepare("SELECT ledger_id FROM l1_world WHERE sess_raw = ?").get(sessRaw) as { ledger_id: string } | undefined; + if (!r) return null; + db.prepare("UPDATE l1_world SET r3_latched_at = NULL WHERE sess_raw = ?").run(sessRaw); + return r.ledger_id; +} + +/** #6785 review: the persisted-latch roster — operators must be able to SEE + * which raw session ids are stuck (and since when) before they can unstick + * one. Empty (not null) when none are latched. */ +export function snapR3LatchedList(db: DbLike): Array<{ sessRaw: string; ledgerId: string; latchedAt: number }> { + return (db + .prepare("SELECT sess_raw, ledger_id, r3_latched_at FROM l1_world WHERE r3_latched_at IS NOT NULL ORDER BY r3_latched_at DESC") + .all() as Array<{ sess_raw: string; ledger_id: string; r3_latched_at: number }>) + .map((r) => ({ sessRaw: r.sess_raw, ledgerId: r.ledger_id, latchedAt: r.r3_latched_at })); +} + +// --------------------------------------------------------------------------- diff --git a/src/storage/store.ts b/src/storage/store.ts index 0dcf1f4835..1a940d9cd5 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -22,12 +22,31 @@ import { dirname, join } from "node:path"; import { log as loggerLog } from "../logger.js"; import { openSqliteDatabase, type SqliteDatabase, type SqliteEngineName, type SqliteEngineSource } from "./driver.js"; import { migrateSchema, storagePragmas, WAL_JOURNAL_SIZE_LIMIT } from "./schema.js"; -import { identityOfStoredContent } from "./identity.js"; +import { identityOfStoredContent, splitMediaContent } from "./identity.js"; +import { hydrateSlimCcrStore } from "./slim.js"; +import { + egressStateAppend, + egressStateAt, + egressStateCompact, + egressStateView, + EgressStateCorruptError, + type EgressOp, + type EgressViewRow, +} from "./egress-state.js"; +import { + snapL1Record, + snapR3LatchedRead, + snapR3LatchClear, + snapR3LatchWrite, + snapR3LatchedList, +} from "./snap.js"; +import { FoundationRejectError, stage0Decide, type Stage0Result } from "../snapshot/plane.js"; import { recordIntegrityOutcome, recordSecondProcessOwner, recordBackupOutcome } from "./runtime-status.js"; import { integrityModeFor, writeIntegrityCache, scheduleBackgroundIntegrityCheck, backgroundIntegrityInflightForTests, noteIntegrityCheckRun } from "./integrity-check.js"; import { sqliteCorruptionDetail } from "./sqlite-errors.js"; import { listInstances, type RegistryEntry } from "../instance.js"; -import type { MessageContentStore } from "acp-kernel"; +import { baseIdOf } from "acp-kernel"; +import type { CoreMessage, MessageContentStore } from "acp-kernel"; import type { BlockView } from "../session.js"; /** #3250: run a PASSIVE wal_checkpoint every N successful ingests. The @@ -226,6 +245,14 @@ export interface CcrEntryRow { /** #5744: sha256(content) from the planner — unchanged entries skip the * CAS bump/release round-trips entirely on steady-state ingests. */ contentHash: string; + /** #6785 step 6: the original text behind entry.hash (the byHash half + * of the content store) — the slim ccr_json envelope has no bytes of + * its own, the CAS behind these rows is the content home. */ + origContent: string | null; + /** null with origContent null — the entry HAS a hash but its byHash + * bytes are missing (the file lane stays authority); "" = hashless + * entry (no original exists) or a legitimate empty original. */ + origHash: string | null; } /** One checkpoint/replica row (Phase 2, #2671). `state_hash` never leaves @@ -299,9 +326,16 @@ export interface SessionStorageStats { deadRefs: number; refs: number; ccrEntries: number; + /** #6785 step 7: the reaped-block journal rows (orphan_blocks). */ + orphanBlocks: number; sessionExt: number; checkpoints: number; pinnedCheckpoints: number; + /** #5874: the L1 identity entry this session owns (l1_world, keyed + * by the raw host session id). Zero for a never-observed session. */ + l1Entries: number; + /** #6785 step 8: the row-backed CompressionState fields (state_fields). */ + stateFields: number; }; /** Bytes held in inline JSON columns of this session's rows (sessions * record + blocks payload_json + session_ext value_json) — everything @@ -351,6 +385,11 @@ export interface IngestSessionInput { * journal entries survive, because LazyBlockContents means a live * session's in-memory map may not carry entries trimmed earlier. */ orphansJson?: string | null; + /** #6785 step 8: the unbounded CompressionState fields as one row each + * (field → JSON value) — the kstate slim split. The kstateJson column + * keeps only the bounded fields; rows are upserted on change and a row + * for a field that becomes undefined is dropped (bounded ≤ 9). */ + stateFields?: Record; } /** Extension-namespace guards (#3069): keys/namespaces are non-empty @@ -388,6 +427,20 @@ function likeEscape(prefix: string): string { return prefix.replace(/[\\%_]/g, (ch) => "\\" + ch); } +/** #5874 A/B oracle: effectiveMessageIds from a blocks row's payload_json + * (the verbatim CompressionBlock the ingest stored). '' on any parse + * failure — a malformed payload just misses the base-set, it never kills + * the oracle (the throw path latches dead like any A/B failure). */ +function effectiveIdsOfPayload(payload: string | null): string[] { + if (!payload) return []; + try { + const p = JSON.parse(payload) as { effectiveMessageIds?: unknown }; + return Array.isArray(p.effectiveMessageIds) ? (p.effectiveMessageIds as string[]) : []; + } catch { + return []; + } +} + /** * #4175: raised when PRAGMA quick_check fails at open. Migrations must never * run against a corrupt file, so this fires BEFORE migrateSchema (and from the @@ -473,7 +526,7 @@ export class UnifiedStore { /** Where the engine came from (#2833): builtin / host / self-downloaded — logged at enable so field diagnosis can tell which acquisition path a lane ran on. */ readonly engineSource: SqliteEngineSource; private readonly db: SqliteDatabase; - /** #3250: successful-ingest counter driving the WAL_CHECKPOINT_EVERY_INGESTS cadence. */ + /** #3250: ingest counter driving the cadence-tick WAL checkpoint. */ private ingestCount = 0; /** #5744: last periodic checkpoint result (plus its wall time) for the * slow-ingest warn line — lets an operator attribute a stall spike to a @@ -978,38 +1031,121 @@ export class UnifiedStore { } /** refcount-aware release; the CAS body itself never leaves. */ - /** #4242: merge the incoming orphan map over the journal's current - * contents. Incoming entries win on key collision (same id, refreshed - * view); existing entries the live session no longer carries (trimmed - * from memory by LazyBlockContents) survive. */ - private mergeOrphansJson(sessionId: string, incomingJson: string | null): string | null { - if (incomingJson === null) { - // No orphans in this save — keep whatever the journal already - // holds (a previous reap's trimmed-away contents must survive). - const row = this.db - .prepare("SELECT orphans_json FROM sessions WHERE session_id = ?") - .get(sessionId) as { orphans_json: string | null } | undefined; - return row?.orphans_json ?? null; + /** #6785 step 7: the orphan journal moved OFF the sessions row — one + * row per block in orphan_blocks (immutable content; a refreshed view + * for the same id is an UPDATE). Keeps the #4242 merge semantics exactly: + * incoming entries win on key collision, entries the live session no + * longer carries (trimmed by LazyBlockContents) survive — the table is + * only ever INSERT/UPDATE, never DELETE within a session. The legacy + * orphans_json column migrates onto the table on first ingest and is + * then cleared, so the column stops growing and stops being re-written + * on every save. Returns null: the column is deprecated. */ + private applyOrphanBlocks(sessionId: string, incomingJson: string | null): null { + const existing = this.db + .prepare("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?") + .all(sessionId) as { block_id: string; one_json: string | null; full_json: string }[]; + const rowMap = new Map(existing.map((r) => [r.block_id, r])); + const all: Record = {}; + const col = this.getOrphansJson(sessionId); + if (col !== null) { + try { + const parsed = JSON.parse(col) as Record; + if (parsed && typeof parsed === "object") Object.assign(all, parsed); + } catch { /* corrupt journal → start fresh (the read path skips bad entries too) */ } } - let merged: Record = {}; - try { - const existing = this.db - .prepare("SELECT orphans_json FROM sessions WHERE session_id = ?") - .get(sessionId) as { orphans_json: string | null } | undefined; - if (existing?.orphans_json) { - const parsed = JSON.parse(existing.orphans_json) as Record; - if (parsed && typeof parsed === "object") merged = parsed; + if (incomingJson !== null) { + try { + const parsed = JSON.parse(incomingJson) as Record; + if (parsed && typeof parsed === "object") Object.assign(all, parsed); + } catch { /* corrupt incoming → keep existing only */ } + } + const upsert = this.db.prepare( + "INSERT INTO orphan_blocks (session_id, block_id, one_json, full_json) VALUES (?, ?, ?, ?) " + + "ON CONFLICT(session_id, block_id) DO UPDATE SET one_json = excluded.one_json, full_json = excluded.full_json", + ); + for (const [blockId, view] of Object.entries(all)) { + const one = (view as { one?: unknown }).one; + const full = (view as { full?: unknown }).full; + const oneJson = one === undefined ? null : JSON.stringify(one); + const fullJson = JSON.stringify(full); + const row = rowMap.get(blockId); + if (row && row.one_json === oneJson && row.full_json === fullJson) continue; + upsert.run(sessionId, blockId, oneJson, fullJson); + } + if (col !== null) { + // Only the legacy migration step writes; steady-state columns are + // already NULL (PR#6785 review: the unconditional UPDATE made + // every ingest a sessions-row writer, contradicting the + // steady-state zero-write design). + this.db.prepare("UPDATE sessions SET orphans_json = NULL WHERE session_id = ?").run(sessionId); + } + return null; + } + + /** #6785 step 8: row-backed CompressionState fields — one row per field, + * upserted only on change (a steady-state flush writes zero rows) and + * a row for a row-backed field that becomes undefined is DROPPED (the + * column-merge drop-undefined semantics, bounded ≤ 9). Legacy sessions + * with no incoming rows keep their column copy (the read path falls + * back to it); rows always win over the column on read. */ + private applyStateFields(sessionId: string, fields: Record): void { + const existing = this.db + .prepare("SELECT field, value_json FROM state_fields WHERE session_id = ?") + .all(sessionId) as { field: string; value_json: string }[]; + const upsert = this.db.prepare( + "INSERT INTO state_fields (session_id, field, value_json) VALUES (?, ?, ?) " + + "ON CONFLICT(session_id, field) DO UPDATE SET value_json = excluded.value_json", + ); + for (const r of existing) { + if (fields[r.field] === undefined) { + this.db.prepare("DELETE FROM state_fields WHERE session_id = ? AND field = ?").run(sessionId, r.field); } - } catch { - // corrupt journal → start fresh (loadLegacyView skips bad entries too) } - try { - const incoming = JSON.parse(incomingJson) as Record; - if (incoming && typeof incoming === "object") merged = { ...merged, ...incoming }; - } catch { - // corrupt incoming → keep existing only + for (const [field, valueJson] of Object.entries(fields)) { + const row = existing.find((r) => r.field === field); + if (row && row.value_json === valueJson) continue; + upsert.run(sessionId, field, valueJson); } - return Object.keys(merged).length > 0 ? JSON.stringify(merged) : null; + } + + /** #6785 step 8: the unbounded CompressionState field rows (null when + * the session has none — legacy sessions read from the column). */ + getStateFields(sessionId: string): Record | null { + const rows = this.db + .prepare("SELECT field, value_json FROM state_fields WHERE session_id = ?") + .all(sessionId) as { field: string; value_json: string }[]; + if (rows.length === 0) return null; + const out: Record = {}; + for (const r of rows) out[r.field] = r.value_json; + return out; + } + + /** #6785 step 7: the merged orphan journal — orphan_blocks table rows + * plus legacy column entries the migration has not reached yet (defence + * in depth: after any ingest the column is NULL). null = no orphans + * anywhere (callers keep their "null ⇒ skip merge" semantics). */ + getOrphanViews(sessionId: string): Record | null { + const rows = this.db + .prepare("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?") + .all(sessionId) as { block_id: string; one_json: string | null; full_json: string }[]; + let out: Record | null = null; + for (const r of rows) { + if (out === null) out = {}; + out[r.block_id] = { one: r.one_json === null ? null : JSON.parse(r.one_json), full: JSON.parse(r.full_json) }; + } + const col = this.getOrphansJson(sessionId); + if (col !== null) { + try { + const parsed = JSON.parse(col) as Record; + if (parsed && typeof parsed === "object") { + for (const [id, view] of Object.entries(parsed)) { + if (out === null) out = {}; + if (out[id] === undefined) out[id] = view; + } + } + } catch { /* corrupt legacy column → the table rows remain */ } + } + return out; } private releaseContent(hash: string | null): void { @@ -1289,9 +1425,13 @@ export class UnifiedStore { // #4216/#4242: orphan journal rides the normal upsert with // MERGE semantics (existing entries survive — LazyBlockContents // means the live map may not carry trimmed entries). - orphansJson: this.mergeOrphansJson(s.sessionId, s.orphansJson ?? null), + orphansJson: this.applyOrphanBlocks(s.sessionId, s.orphansJson ?? null), }); + // #6785 step 8: row-backed kstate fields ride the same tx (the + // column keeps only the bounded fields — see splitKstate). + this.applyStateFields(s.sessionId, input.stateFields ?? {}); + // Identity index (#2960 R2.1): claim (namespace, conversation_id) // → session_id. ON CONFLICT DO NOTHING — first claimant wins; while // the v1 mapping is the identity a re-ingest is the same triple. @@ -1537,6 +1677,7 @@ export class UnifiedStore { } } + // dead_refs: tombstones can be LIFTED (backing message // reappears) — the array is authoritative, so rows at ords the // incoming set no longer covers are stale and must go. @@ -1637,17 +1778,26 @@ export class UnifiedStore { const incomingCcr = new Map(); for (const e of input.ccrEntries) incomingCcr.set(e.key, e); const storedCcr = this.db - .prepare("SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ?") - .all(s.sessionId) as Array<{ ord: number; ccr_key: string; content_hash: string }>; + .prepare("SELECT ord, ccr_key, content_hash, orig_hash FROM ccr_entries WHERE session_id = ?") + .all(s.sessionId) as Array<{ ord: number; ccr_key: string; content_hash: string; orig_hash: string | null }>; const changedCcrKeys: string[] = []; const unchangedCcrKeys = new Set(); const releasedOldCcrHashes: string[] = []; for (const row of storedCcr) { const inc = incomingCcr.get(row.ccr_key); - if (!inc) { changedCcrKeys.push(row.ccr_key); releasedOldCcrHashes.push(row.content_hash); continue; } - if (row.ord !== inc.ord || row.content_hash !== inc.contentHash) { + if (!inc) { + changedCcrKeys.push(row.ccr_key); + releasedOldCcrHashes.push(row.content_hash); + if (row.orig_hash !== null) releasedOldCcrHashes.push(row.orig_hash); + continue; + } + // #6785 step 6: the orig half joins the unchanged test — a + // pre-step-6 row (orig NULL) or an archived-restore row + // re-puts its original on the next save and self-heals. + if (row.ord !== inc.ord || row.content_hash !== inc.contentHash || row.orig_hash !== inc.origHash) { changedCcrKeys.push(row.ccr_key); releasedOldCcrHashes.push(row.content_hash); + if (row.orig_hash !== null) releasedOldCcrHashes.push(row.orig_hash); } else { unchangedCcrKeys.add(row.ccr_key); } @@ -1664,9 +1814,10 @@ export class UnifiedStore { for (const e of input.ccrEntries) { if (unchangedCcrKeys.has(e.key)) continue; const hash = this.putContent(e.content, e.contentHash); + const orig = e.origContent === null ? null : this.putContent(e.origContent, e.origHash ?? undefined); this.db - .prepare("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)") - .run(s.sessionId, e.ord, e.key, hash); + .prepare("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash, orig_hash) VALUES (?, ?, ?, ?, ?)") + .run(s.sessionId, e.ord, e.key, hash, orig); } return true; }); @@ -1674,6 +1825,250 @@ export class UnifiedStore { return ok; } + /** L1 witness probe: a sampled fingerprint of the FIRST live ids (the + * head of the id space is what survives rounds — the tail churns). */ + private snapWitnessProbe(sid: string): string { + const rows = this.db + .prepare("SELECT msg_id FROM messages WHERE session_id = ? AND retired_at IS NULL ORDER BY seq LIMIT 8") + .all(sid) as { msg_id: string }[]; + return sha256Hex(rows.length ? rows.map((r) => r.msg_id).join("\u0000") : "empty"); + } + + + // #5874 2×2 stage 0 (brief 19544722): the element world match lives at the + // head of the chain, BEFORE protocol conversion. These accessors are the + // storage side of that stage — the pure decision (snapshot/plane.ts) and + // the kernel-input assembly (snapshot/assemble.ts) are lane-side. + + private snapSessRawOf(sid: string): string { + // Raw host session id (P9: never truncated) → the l1_world key; the + // conversation binding when the host supplied one, the sid otherwise. + // (Same resolution as the save path below — stage 0 and the plane + // commit must address the SAME l1 row.) + return ( + (this.db.prepare("SELECT conversation_id FROM conversations WHERE session_id = ?").get(sid) as { conversation_id: string } | undefined) + ?.conversation_id ?? sid + ); + } + + /** Stage 0 for one turn: ensure the l1 row exists (idempotent session- + * level v1 record), run the element world match on the stored envelope, + * and persist the new canon (every verdict except `diverge` — that one + * has zero side effects by construction). A session-level identity + * conflict ("red") throws — the lane maps it to the R3 hard 400 BEFORE + * any kernel side effect. + */ + snapStage0(sid: string, incoming: string[], opts?: { resync?: boolean, seqs?: number[] | null }): Stage0Result { + const sessRaw = this.snapSessRawOf(sid); + const now = Date.now(); + const l1 = snapL1Record(this.db, sessRaw, sid, this.snapWitnessProbe(sid), now); + if (l1 === "red") { + // PR#6785 review: an R3 LATCH PLACEHOLDER row is not a real + // ledger — snapR3LatchWrite created it BEFORE the first stage-0 + // run with ledger_id = sessRaw (the real sid unknown; legitimate + // rows never carry witness_probe "" because snapWitnessProbe + // always hashes). After the operator unlatches + // (POST /__bili/snapshot/r3-unlatch), the placeholder would keep + // mapping sess_raw → a DIFFERENT ledger forever and 400 every + // turn for a conversation-bound session (snapSessRawOf(sid) ≠ + // sid). Rebind it to the real sid instead (the mapping is the + // identity anchor; a placeholder is no anchor yet). + const row = this.db + .prepare("SELECT ledger_id, witness_probe FROM l1_world WHERE sess_raw = ?") + .get(sessRaw) as { ledger_id: string; witness_probe: string } | undefined; + if (row && row.ledger_id === sessRaw && row.witness_probe === "") { + this.db + .prepare("UPDATE l1_world SET ledger_id = ?, witness_probe = ? WHERE sess_raw = ?") + .run(sid, this.snapWitnessProbe(sid), sessRaw); + } else { + // Session-level identity conflict (R3): the raw host id maps + // to a DIFFERENT ledger. Hard-reject BEFORE any kernel side + // effect — snapL1Record wrote nothing on this path. + throw new FoundationRejectError(`l1 ledger conflict: sess_raw=${sessRaw} maps to another ledger (R3)`); + } + } + // The gone-set: stored messages whose ids legally vanish from the + // envelope head because a LIVE fold block covers them (the covered + // messages stay in the ledger — compression deactivates the coverage, + // not the rows). Pure ledger math, no per-row state table. + const goneRows = this.db + .prepare("SELECT m.msg_id FROM messages m JOIN blocks b ON b.session_id = m.session_id AND m.seq >= b.start_seq AND m.seq <= b.end_seq WHERE m.session_id = ? AND m.retired_at IS NULL AND b.active = 1") + .all(sid) as { msg_id: string }[]; + const explainedGone = new Set(goneRows.map((r) => r.msg_id)); + // The ledger frontier: the max live stamped seq (0 for an empty + // ledger) — the M2 contiguity evidence anchors on it. + const maxSeq = (this.db.prepare("SELECT COALESCE(MAX(seq), 0) AS m FROM messages WHERE session_id = ? AND retired_at IS NULL").get(sid) as { m: number }).m; + return stage0Decide({ db: this.db, sessRaw, ledgerId: sid, incoming, seqs: opts?.seqs ?? null, resync: opts?.resync, maxSeq, explainedGone, now, convBound: this.snapConvBound(sid) }); + } + + /** The stage-0 resync (brief red gate R4 / the resync ruling): clear the + * l1 element envelope so the NEXT turn re-issues it from the client's + * current array (one-round convergence). This turn runs the legacy full + * path and the normal in-request flush re-baselines the ledger. The l1 + * identity row itself (the v1 mapping + witness + R3 latch) survives: + * a resync re-baselines the ENVELOPE, never the session identity. + */ + snapStage0Reset(sid: string): void { + const sessRaw = this.snapSessRawOf(sid); + this.db + .prepare("UPDATE l1_world SET tail_json = NULL, head_json = NULL, count = NULL, max_pos = NULL WHERE sess_raw = ?") + .run(sessRaw); + } + + /** D2 (#7064): whether this sid is CONVERSATION-BOUND — the host already + * declared a conversation identity (a conversations row for this sid), + * so the per-turn id verdict is skipped entirely and the R3 fault path + * (400 / latch) never fires for id mismatch: the session IS the + * identity. */ + snapConvBound(sid: string): boolean { + // The claim INSERT at ingest time writes a `proxy`-namespace row for + // EVERY session (conversation_id = session_id — a self-claim, not a + // host identity). D2 applies only to HOST-namespace claims (the + // agent registered a real conversation): namespace <> 'proxy'. + const row = this.db + .prepare("SELECT 1 FROM conversations WHERE session_id = ? AND namespace <> 'proxy' LIMIT 1") + .get(sid); + return row !== undefined; + } + + /** #7064 §8: the ledger's LIVE stamped ids in seq order (0-based index ⇔ + * ledger position). The incremental pin re-stamps a steady-state turn's + * prefix cores from this list — ingest is INSERT OR IGNORE on (session_id, + * seq), so a position's msg_id is written once and the kernel's ref space + * only stays stable when every presented id matches the ledger. Empty + * array = empty ledger (bootstrap: the full-list path must run). */ + snapMsgIdLiveIds(sid: string): string[] { + const rows = this.db + .prepare("SELECT msg_id FROM messages WHERE session_id = ? AND retired_at IS NULL ORDER BY seq") + .all(sid) as { msg_id: string }[]; + return rows.map((r) => r.msg_id); + } + + /** #6181 M3: whether this sid's R3 breaker latch PERSISTED from a + * previous process (a restart must not re-arm a misbehaving host and + * re-400 the same broken envelope). */ + snapR3Latched(sid: string): boolean { + try { + return snapR3LatchedRead(this.db, this.snapSessRawOf(sid)); + } catch { + return false; // schema/column edge: treat as unlatched, the in-memory breaker still fires + } + } + + /** #6181 M3: persist the R3 breaker latch (best-effort — a failed + * write degrades to the in-memory latch, which still fires this + * process). */ + snapR3Latch(sid: string): void { + try { + snapR3LatchWrite(this.db, this.snapSessRawOf(sid), Date.now()); + } catch { /* best-effort — see above */ } + } + + /** #6785 review: operator escape hatch — clear the persisted R3 latch + * for one RAW host session id. Returns the ledger id the l1 row binds + * (the caller uses it to clear the in-memory breaker too) or null when + * no row exists / a fault. Keyed by sess_raw (the roster key), not by + * sid — the admin surface has the raw id, and the latch row predates + * the sid→raw resolution in its placeholder phase. */ + snapR3LatchClear(sessRaw: string): string | null { + try { + return snapR3LatchClear(this.db, sessRaw); + } catch { + return null; + } + } + + /** #6785 review: the persisted-latch roster (an operator must be able + * to SEE which raw ids are stuck before clearing them). Empty on a + * fault — the roster is diagnostic, never load-bearing. */ + snapR3LatchedList(): Array<{ sessRaw: string; ledgerId: string; latchedAt: number }> { + try { + return snapR3LatchedList(this.db); + } catch { + return []; + } + } + + /** #6785 review: the ledger sid a raw host session id conversational- + * binds to (null when the host supplied no conversation binding — the + * common case, where sid === raw). The unlatch route clears the + * in-memory breaker for BOTH ids: the latch row's ledger_id can still + * be the sessRaw placeholder (first fault before any stage-0 run), + * while the breaker itself is keyed by the real sid. */ + snapSidOfRaw(sessRaw: string): string | null { + if (sessRaw.length === 0) return null; + try { + return (this.db.prepare("SELECT session_id FROM conversations WHERE conversation_id = ? LIMIT 1").get(sessRaw) as { session_id: string } | undefined) + ?.session_id ?? null; + } catch { + return null; + } + } + + + /** #6785 perf (proposal discussions/7398): EGRESS STATE LOG — op-log + * outbound view. The four methods below are the store surface over + * src/storage/egress-state.ts (dual table: bounded state + total flow). + * Corruption throws EgressStateCorruptError — callers degrade the whole + * session to the legacy full path (never a partial view). */ + egressStateAppendOps(sessionId: string, ops: EgressOp[], nowMs: number): number[] { + return egressStateAppend(this.db, sessionId, ops, nowMs); + } + + egressStateView(sessionId: string): EgressViewRow[] { + return egressStateView(this.db, sessionId); + } + + /** Rollback reconstruction at any log_seq (flow table is total). */ + egressStateViewAt(sessionId: string, logSeq: number): EgressViewRow[] { + return egressStateAt(this.db, sessionId, logSeq); + } + + egressStateCompactView(sessionId: string, view: EgressViewRow[], nowMs: number): number { + return egressStateCompact(this.db, sessionId, view, nowMs); + } + + egressStateMaxSeq(sessionId: string): number { + const row = this.db.prepare(`SELECT COALESCE(MAX(log_seq), 0) AS mx FROM egress_flow WHERE session_id = ?`).get(sessionId) as { mx: number }; + return Number(row.mx) || 0; + } + + egressStateTruncateAfter(sessionId: string, logSeq: number): number { + const a = this.db.prepare(`DELETE FROM egress_state WHERE session_id = ? AND log_seq > ?`).run(sessionId, logSeq); + const b = this.db.prepare(`DELETE FROM egress_flow WHERE session_id = ? AND log_seq > ?`).run(sessionId, logSeq); + return Number(a.changes) + Number(b.changes); + } + + /** #7398: delete the session's egress op-log rows (state AND flow) and + * return the unified-content-CAS refs the rows carried (cas_ref on I/U + * ops, casRef on B-row entries). The recorder's puts are the sole owner + * of those refs (message bodies are additionally pinned by their own + * messages rows); callers release + collect exactly as for other CAS. + * Runs inside the caller's sweep transaction. */ + deleteEgressOps(sessionId: string): string[] { + const refs: string[] = []; + const take = (payload: string): void => { + let parsed: { cas_ref?: unknown; entries?: unknown }; + try { + parsed = JSON.parse(payload) as { cas_ref?: unknown; entries?: unknown }; + } catch { + return; + } + if (typeof parsed.cas_ref === "string") refs.push(parsed.cas_ref); + if (Array.isArray(parsed.entries)) { + for (const e of parsed.entries as Array<{ casRef?: unknown }>) { + if (typeof e.casRef === "string") refs.push(e.casRef); + } + } + }; + for (const r of this.db.prepare("SELECT payload FROM egress_state WHERE session_id = ?").all(sessionId) as Array<{ payload: string }>) take(r.payload); + for (const r of this.db.prepare("SELECT payload FROM egress_flow WHERE session_id = ?").all(sessionId) as Array<{ payload: string }>) take(r.payload); + this.db.prepare("DELETE FROM egress_state WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM egress_flow WHERE session_id = ?").run(sessionId); + return refs; + } + + /** #3250: cadence-tick WAL reclamation (see WAL_CHECKPOINT_EVERY_INGESTS). * Never throws into the ingest path — a failed checkpoint just leaves the * WAL longer than ideal until the next tick. */ @@ -2090,22 +2485,10 @@ export class UnifiedStore { this.releaseContent(row.summary_hash); if (row.one_hash !== null) this.releaseContent(row.one_hash); } - if (orphanContents.size > 0) { - const sess = this.db - .prepare("SELECT orphans_json FROM sessions WHERE session_id = ?") - .get(sessionId) as { orphans_json: string | null } | undefined; - if (sess) { - let merged: Record = {}; - try { - const parsed = JSON.parse(sess.orphans_json ?? "{}") as unknown; - if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) merged = parsed as Record; - } catch { - // corrupt journal → start fresh rather than losing the reap - } - for (const [blockId, view] of orphanContents) merged[blockId] = view; - this.db.prepare("UPDATE sessions SET orphans_json = ? WHERE session_id = ?").run(JSON.stringify(merged), sessionId); - } - } + // #6785 step 7: the reaped-block journal lives in orphan_blocks + // now (same merge semantics the column had — incoming entries win + // on key collision, trimmed entries survive). + this.applyOrphanBlocks(sessionId, orphanContents.size > 0 ? JSON.stringify(Object.fromEntries(orphanContents)) : null); }); } @@ -2127,8 +2510,11 @@ export class UnifiedStore { if (b.one_hash !== null) refs.push(b.one_hash); } for (const e of this.db - .prepare("SELECT content_hash FROM ccr_entries WHERE session_id = ?") - .all(sessionId) as { content_hash: string }[]) refs.push(e.content_hash); + .prepare("SELECT content_hash, orig_hash FROM ccr_entries WHERE session_id = ?") + .all(sessionId) as { content_hash: string; orig_hash: string | null }[]) { + refs.push(e.content_hash); + if (e.orig_hash !== null) refs.push(e.orig_hash); + } // Checkpoints: pinned replicas keep their rows + CAS refcounts; the // rest die with the session. for (const cp of this.db @@ -2140,12 +2526,18 @@ export class UnifiedStore { this.db.prepare("DELETE FROM messages WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM message_media WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM blocks WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM orphan_blocks WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM state_fields WHERE session_id = ?").run(sessionId); + refs.push(...this.deleteEgressOps(sessionId)); this.db.prepare("DELETE FROM dead_refs WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM refs WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM ccr_entries WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM conversations WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM session_ext WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM sessions WHERE session_id = ?").run(sessionId); + // #5874: the L1 identity entry rides the session (keyed by + // ledger_id); no orphan may survive (T6). + this.db.prepare("DELETE FROM l1_world WHERE ledger_id = ?").run(sessionId); for (const hash of refs) this.releaseContent(hash); // #4796 follow-up: scoped collection over this session's own // released hashes (see collectZeroRefIn) — never the whole table. @@ -2169,14 +2561,18 @@ export class UnifiedStore { "(SELECT COUNT(*) FROM dead_refs WHERE session_id = ?) AS deadRefs, " + "(SELECT COUNT(*) FROM refs WHERE session_id = ?) AS refs, " + "(SELECT COUNT(*) FROM ccr_entries WHERE session_id = ?) AS ccrEntries, " + + "(SELECT COUNT(*) FROM orphan_blocks WHERE session_id = ?) AS orphanBlocks, " + "(SELECT COUNT(*) FROM session_ext WHERE session_id = ?) AS ext, " + "(SELECT COUNT(*) FROM checkpoints WHERE session_id = ?) AS cps, " + - "(SELECT COUNT(*) FROM checkpoints WHERE session_id = ? AND pinned = 1) AS pinnedCps", + "(SELECT COUNT(*) FROM checkpoints WHERE session_id = ? AND pinned = 1) AS pinnedCps, " + + "(SELECT COUNT(*) FROM l1_world WHERE ledger_id = ?) AS l1Entries, " + + "(SELECT COUNT(*) FROM state_fields WHERE session_id = ?) AS stateFields", ) - .get(sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId) as { + .get(sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId) as { sessions: number; conversations: number; messages: number; media: number; blocks: number; deadRefs: number; - refs: number; ccrEntries: number; ext: number; cps: number; pinnedCps: number; + refs: number; ccrEntries: number; orphanBlocks: number; ext: number; cps: number; pinnedCps: number; + l1Entries: number; stateFields: number; }; const inlineBytes = ((this.db @@ -2202,6 +2598,7 @@ export class UnifiedStore { UNION ALL SELECT summary_hash, 1 FROM blocks WHERE session_id = ? UNION ALL SELECT one_hash, 1 FROM blocks WHERE session_id = ? AND one_hash IS NOT NULL UNION ALL SELECT content_hash, 1 FROM ccr_entries WHERE session_id = ? + UNION ALL SELECT orig_hash, 1 FROM ccr_entries WHERE session_id = ? AND orig_hash IS NOT NULL UNION ALL SELECT state_hash, CASE WHEN pinned = 0 THEN 1 ELSE 0 END FROM checkpoints WHERE session_id = ? ), releases AS (SELECT hash, SUM(released) AS n FROM owned_refs GROUP BY hash) SELECT COALESCE(SUM(CASE WHEN r.n > 0 AND c.refcount <= r.n THEN c.byte_len ELSE 0 END), 0) AS exclusiveBytes, @@ -2209,7 +2606,7 @@ export class UnifiedStore { COALESCE(SUM(CASE WHEN c.refcount > r.n THEN 1 ELSE 0 END), 0) AS sharedBlobs FROM releases r JOIN content c ON c.hash = r.hash`, ) - .get(sessionId, sessionId, sessionId, sessionId, sessionId, sessionId) as { exclusiveBytes: number; exclusiveBlobs: number; sharedBlobs: number }; + .get(sessionId, sessionId, sessionId, sessionId, sessionId, sessionId, sessionId) as { exclusiveBytes: number; exclusiveBlobs: number; sharedBlobs: number }; return { rows: { sessions: counts.sessions, @@ -2220,9 +2617,12 @@ export class UnifiedStore { deadRefs: counts.deadRefs, refs: counts.refs, ccrEntries: counts.ccrEntries, + orphanBlocks: counts.orphanBlocks, sessionExt: counts.ext, checkpoints: counts.cps, pinnedCheckpoints: counts.pinnedCps, + l1Entries: counts.l1Entries, + stateFields: counts.stateFields, }, inlineBytes, casExclusiveBytes: cas.exclusiveBytes, @@ -2336,15 +2736,18 @@ export class UnifiedStore { const blocks = (this.db.prepare("SELECT block_id, ord, start_seq, end_seq, start_ref, end_ref, tier, summary_hash, one_hash, supersedes_block_id, active, created_at, covered_hash, payload_json FROM blocks WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); const refsRows = (this.db.prepare("SELECT ord, raw_id, ref, ref_num FROM refs WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); const deadRefs = (this.db.prepare("SELECT ord, ref, ref_num, recorded_at FROM dead_refs WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); - const ccr = (this.db.prepare("SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); + const ccr = (this.db.prepare("SELECT ord, ccr_key, content_hash, orig_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord").all(sessionId) as Record[]); const ext = (this.db.prepare("SELECT key, value_json, updated_at FROM session_ext WHERE session_id = ?").all(sessionId) as Record[]); const checkpoints = (this.db.prepare("SELECT cp_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash FROM checkpoints WHERE session_id = ?").all(sessionId) as Record[]); + // #6785 step 7: the reaped-block journal rides the archive too. + const orphans = (this.db.prepare("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?").all(sessionId) as Record[]); + const stateFields = (this.db.prepare("SELECT field, value_json FROM state_fields WHERE session_id = ?").all(sessionId) as Record[]); // --- collect the distinct CAS bodies referenced by those rows --- const wanted = new Set(); for (const m of messages) wanted.add(m.content_hash as string); for (const mm of mediaMaps) wanted.add(mm.media_hash as string); for (const b of blocks) { wanted.add(b.summary_hash as string); if (b.one_hash !== null && b.one_hash !== undefined) wanted.add(b.one_hash as string); } - for (const e of ccr) wanted.add(e.content_hash as string); + for (const e of ccr) { wanted.add(e.content_hash as string); if (e.orig_hash !== null && e.orig_hash !== undefined) wanted.add(e.orig_hash as string); } for (const cp of checkpoints) if (cp.pinned === 0) wanted.add(cp.state_hash as string); const content: Record = {}; for (const hash of wanted) { @@ -2363,7 +2766,7 @@ export class UnifiedStore { // and every failed attempt would add another full copy. The line // still reaches the log before COMMIT, so a committed sweep always // has its export. - const archiveLine = JSON.stringify({ version: 1, sessionId, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints }) + "\n"; + const archiveLine = JSON.stringify({ version: 1, sessionId, archivedAt: Date.now(), session: row, content, messages, mediaMaps, blocks, refs: refsRows, deadRefs, ccrEntries: ccr, sessionExt: ext, checkpoints, orphanBlocks: orphans, stateFields }) + "\n"; // --- sweep the row bodies (refcount discipline; pinned checkpoints survive) --- let contentRowsDeleted = 0; this.db.transaction(() => { @@ -2371,11 +2774,14 @@ export class UnifiedStore { for (const m of messages) rel.push(m.content_hash as string); for (const mm of mediaMaps) rel.push(mm.media_hash as string); for (const b of blocks) { rel.push(b.summary_hash as string); if (b.one_hash !== null && b.one_hash !== undefined) rel.push(b.one_hash as string); } - for (const e of ccr) rel.push(e.content_hash as string); + for (const e of ccr) { rel.push(e.content_hash as string); if (e.orig_hash !== null && e.orig_hash !== undefined) rel.push(e.orig_hash as string); } for (const cp of checkpoints) if (cp.pinned === 0) { rel.push(cp.state_hash as string); this.db.prepare("DELETE FROM checkpoints WHERE cp_id = ?").run(cp.cp_id); } this.db.prepare("DELETE FROM messages WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM message_media WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM blocks WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM orphan_blocks WHERE session_id = ?").run(sessionId); + this.db.prepare("DELETE FROM state_fields WHERE session_id = ?").run(sessionId); + rel.push(...this.deleteEgressOps(sessionId)); this.db.prepare("DELETE FROM dead_refs WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM refs WHERE session_id = ?").run(sessionId); this.db.prepare("DELETE FROM ccr_entries WHERE session_id = ?").run(sessionId); @@ -2409,6 +2815,8 @@ export class UnifiedStore { ccrEntries: Record[]; sessionExt: Record[]; checkpoints: Record[]; + orphanBlocks?: Record[]; + stateFields?: Record[]; }; // Scan the JSONL for the session's NEWEST line (#6317: a pre-fix // failed archive can leave stale lines behind; a later successful @@ -2462,7 +2870,11 @@ export class UnifiedStore { addRef(b.summary_hash); addRef(b.one_hash); } - for (const e of dump.ccrEntries) addRef(e.content_hash); + for (const e of dump.ccrEntries) { + addRef(e.content_hash); + const eh = (e as { orig_hash?: unknown }).orig_hash; + if (typeof eh === "string") addRef(eh); + } for (const cp of dump.checkpoints) { if (!this.getCheckpoint(cp.cp_id as string)) addRef(cp.state_hash); } @@ -2491,14 +2903,19 @@ export class UnifiedStore { for (const r of dump.refs) insRef.run(sessionId, r.ord, r.raw_id, r.ref, r.ref_num ?? null); const insDead = this.db.prepare("INSERT INTO dead_refs (session_id, ord, ref, ref_num, recorded_at) VALUES (?, ?, ?, ?, ?)"); for (const d of dump.deadRefs) insDead.run(sessionId, d.ord, d.ref, d.ref_num, d.recorded_at); - const insCcr = this.db.prepare("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash) VALUES (?, ?, ?, ?)"); - for (const e of dump.ccrEntries) insCcr.run(sessionId, e.ord, e.ccr_key, e.content_hash); + const insCcr = this.db.prepare("INSERT INTO ccr_entries (session_id, ord, ccr_key, content_hash, orig_hash) VALUES (?, ?, ?, ?, ?)"); + for (const e of dump.ccrEntries) insCcr.run(sessionId, e.ord, e.ccr_key, e.content_hash, typeof (e as { orig_hash?: unknown }).orig_hash === "string" ? (e as { orig_hash: string }).orig_hash : null); const insExt = this.db.prepare("INSERT INTO session_ext (session_id, key, value_json, updated_at) VALUES (?, ?, ?, ?)"); for (const x of dump.sessionExt) { insExt.run(sessionId, x.key, x.value_json, typeof x.updated_at === "number" && Number.isFinite(x.updated_at) ? x.updated_at : Date.now()); } const insCp = this.db.prepare("INSERT INTO checkpoints (cp_id, session_id, seq_at, block_set_hash, kind, created_at, pinned, label, state_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(cp_id) DO NOTHING"); for (const cp of dump.checkpoints) insCp.run(cp.cp_id, sessionId, cp.seq_at, cp.block_set_hash, cp.kind, cp.created_at, cp.pinned, cp.label ?? null, cp.state_hash); + // #6785 step 7: the journal round-trips through the archive. + const insOrphan = this.db.prepare("INSERT INTO orphan_blocks (session_id, block_id, one_json, full_json) VALUES (?, ?, ?, ?) ON CONFLICT(session_id, block_id) DO NOTHING"); + for (const o of dump.orphanBlocks ?? []) insOrphan.run(sessionId, o.block_id, o.one_json ?? null, o.full_json); + const insStateField = this.db.prepare("INSERT INTO state_fields (session_id, field, value_json) VALUES (?, ?, ?) ON CONFLICT(session_id, field) DO NOTHING"); + for (const f of dump.stateFields ?? []) insStateField.run(sessionId, f.field, f.value_json); this.db .prepare(`UPDATE sessions SET parent_session_id = ?, fork_point_seq = ?, protocol = ?, label = ?, title = ?, host_title = ?, namespace = ?, origin = ?, created_at = ?, updated_at = ?, next_block_id = ?, next_run_id = ?, meta_json = ?, stats_json = ?, metadata_json = ?, kstate_json = ?, tail_json = ?, ccr_json = ?, last_ingested_seq = ?, orphans_json = ?, status = 'active' WHERE session_id = ?`) @@ -3259,7 +3676,8 @@ export class UnifiedStore { * (storage-bounded-load / issue4739-storage-read-snapshot pin this shape); * - `getBlockContents(id)`: the full narrow replay-leg contents map for * lazy-field restore (#4739) — block-row summary/one CAS bytes plus the - * orphans_json journal precedence, WITHOUT resurrecting state.blocks + * orphan-journal precedence (orphan_blocks table ∪ legacy column, + * #6785 step 7), WITHOUT resurrecting state.blocks * (issue4739-bounded-session-load pins this). */ getBlockContents(sessionId: string, opts: { limit: number; offset?: number }): Pick[]; getBlockContents(sessionId: string): BlockContentsMap | null; @@ -3278,10 +3696,8 @@ export class UnifiedStore { one: row.one_hash && row.one_bytes !== null ? Buffer.from(row.one_bytes).toString("utf8") : null, })); } - const row = this.db - .prepare("SELECT orphans_json FROM sessions WHERE session_id = ?") - .get(sessionId) as { orphans_json: string | null } | undefined; - if (!row) return null; + const exists = this.db.prepare("SELECT 1 FROM sessions WHERE session_id = ?").get(sessionId); + if (!exists) return null; const out: BlockContentsMap = new Map(); for (const b of this.getBlocks(sessionId)) { out.set(b.blockId, { @@ -3289,14 +3705,13 @@ export class UnifiedStore { full: JSON.parse(b.summary) as BlockView, }); } - if (row.orphans_json !== null && row.orphans_json !== undefined && row.orphans_json !== "") { - try { - const orphans = JSON.parse(row.orphans_json) as Record; - for (const [blockId, view] of Object.entries(orphans)) { - if (!out.has(blockId)) out.set(blockId, view); - } - } catch { - // corrupt journal → skip (same boundary as loadLegacyView) + // #6785 step 7: the journal moved to the orphan_blocks table (the + // sessions.orphans_json column is retired on first ingest) — the + // merged reader is getOrphanViews(); null = no orphans anywhere. + const orphans = this.getOrphanViews(sessionId); + if (orphans !== null) { + for (const [blockId, view] of Object.entries(orphans)) { + if (!out.has(blockId)) out.set(blockId, view as { one: BlockView | null; full: BlockView }); } } return out; @@ -3775,7 +4190,15 @@ export class UnifiedStore { .get(sessionId) as { ccr_json: string | null } | undefined; if (!row || row.ccr_json === null) return null; try { - const parsed = JSON.parse(row.ccr_json) as MessageContentStore; + const parsed = JSON.parse(row.ccr_json) as MessageContentStore & { ccrSlim?: number }; + // #6785 step 6: slim envelope — the column keeps key→hash only; + // the bytes come back through the CAS via ccr_entries (same + // reconstruction as the view leg, one definition in slim.ts). + // A slim row with an empty ledger degrades to the file path. + if (parsed.ccrSlim === 1) { + const entries = this.getCcrEntries(sessionId); + return hydrateSlimCcrStore(parsed, entries); + } if (parsed && typeof parsed === "object" && parsed.version === 1 && parsed.byHash && typeof parsed.byHash === "object" && parsed.byRef && typeof parsed.byRef === "object") { @@ -3789,13 +4212,18 @@ export class UnifiedStore { getCcrEntries(sessionId: string): CcrEntryRow[] { const rows = this.db - .prepare("SELECT ord, ccr_key, content_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord") + .prepare("SELECT ord, ccr_key, content_hash, orig_hash FROM ccr_entries WHERE session_id = ? ORDER BY ord") .all(sessionId) as Record[]; return rows.map((r) => ({ ord: r.ord as number, key: r.ccr_key as string, content: this.getContent(r.content_hash as string) ?? "", contentHash: r.content_hash as string, + // #6785 step 6: null (not "") marks pre-step-6 rows — callers + // degrade to the full envelope / file lane instead of + // reconstructing an empty original. + origContent: r.orig_hash === null || r.orig_hash === undefined ? null : this.getContent(r.orig_hash as string) ?? null, + origHash: r.orig_hash === null || r.orig_hash === undefined ? null : (r.orig_hash as string), })); } } diff --git a/tests/dist-fixture-lock.ts b/tests/dist-fixture-lock.ts new file mode 100644 index 0000000000..33839bf228 --- /dev/null +++ b/tests/dist-fixture-lock.ts @@ -0,0 +1,82 @@ +import { closeSync, openSync, readFileSync, statSync, unlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +// Cross-process lock for tests that mutate a shared dist/ fixture file. +// +// tests/launcher.test.ts and tests/launcher-pi-inline.test.ts both juggle the +// same path /dist/agent/pi.js: the "-e injected" tests create an +// empty stub when the checkout has no build, and the "refuses" tests rename +// the file away (or rm it) to force absence. scripts/run-ci-tests.mjs runs +// files with --test-concurrency 4 and the CI test job never builds dist/, so +// the two files routinely overlap: one test's rename/rm lands inside a +// concurrent test's runLaunch window, its fs.existsSync(piExt) check misses, +// the -e injection is skipped, and the assertion fails (flaky red on PR +// #6785, test ubuntu-22-1: "injected only when not installed" saw []). +// +// Presence and absence are mutually exclusive states of one path, so the +// fixture sites must run one at a time. node --test runs each file in its +// own process, so the lock must be cross-process (O_EXCL create). A crashed +// holder is handled by the stale mtime cutoff; release only removes a lock +// whose recorded pid is still ours, so a stolen lock is never deleted from +// under its new holder. +// +// LOCK_DIR is read at module load — before any test body mutates +// HOME/USERPROFILE — so the lock path is stable on every platform. +const LOCK_DIR = tmpdir(); +const STALE_MS = 5 * 60 * 1000; +const WAIT_TIMEOUT_MS = 120 * 1000; +const RETRY_MS = 50; + +// "wx" fails with EEXIST on POSIX and EPERM on Windows when the file already +// exists — both mean "someone holds the lock". +const LOCK_HELD_CODES = new Set(["EEXIST", "EPERM", "EACCES"]); + +function lockPathFor(fixturePath: string): string { + return path.join(LOCK_DIR, `bili-test-dist-fixture-${path.basename(fixturePath)}.lock`); +} + +/** + * Serialize access to a shared dist/ fixture file across concurrently + * running test files. Call it BEFORE the first fixture read (existsSync / + * stub create / rename away) and invoke the returned release in the test's + * finally, AFTER the last fixture write (rmSync / rename back). Bounded + * wait: a test that cannot acquire within WAIT_TIMEOUT_MS fails loudly + * instead of hanging the shard. + */ +export async function acquireDistFixtureLock(fixturePath: string): Promise<() => void> { + const lockPath = lockPathFor(fixturePath); + const deadline = Date.now() + WAIT_TIMEOUT_MS; + for (;;) { + try { + const fd = openSync(lockPath, "wx"); + writeFileSync(fd, `${process.pid}\n`); + closeSync(fd); + break; + } catch (err) { + if (!LOCK_HELD_CODES.has((err as NodeJS.ErrnoException).code ?? "")) throw err; + try { + if (Date.now() - statSync(lockPath).mtimeMs > STALE_MS) { + unlinkSync(lockPath); + continue; + } + } catch { + // lock vanished between the failed open and the stat — retry + } + if (Date.now() > deadline) { + throw new Error( + `timed out after ${WAIT_TIMEOUT_MS}ms waiting for the dist fixture lock on ${fixturePath} (stale holder? lock: ${lockPath})`, + ); + } + await new Promise((resolve) => setTimeout(resolve, RETRY_MS)); + } + } + return () => { + try { + const content = readFileSync(lockPath, "utf8"); + if (content.startsWith(`${process.pid}\n`)) unlinkSync(lockPath); + } catch { + // lock already gone (stale-steal or manual cleanup) — nothing to release + } + }; +} diff --git a/tests/e2e-msgid-envelope.test.ts b/tests/e2e-msgid-envelope.test.ts new file mode 100644 index 0000000000..b04690881e --- /dev/null +++ b/tests/e2e-msgid-envelope.test.ts @@ -0,0 +1,178 @@ +// #6539 hermetic e2e: a REAL bili server (startServer) + fake upstream. +// Proves the "record" half of the proposal end-to-end over the HTTP stack: +// A. normal plugin lane — a full-resend body carrying the `biliMsgIds` +// envelope gets its host ids PINNED into the store: the plugin snapshot's +// rawIds come back as the host ids (not content digests). The field never +// reaches the upstream. +// B. verbatim (bypass) lane — the SAME stamped body is scrubbed before the +// forward, so biliMsgIds never reaches the upstream (the F1 review fix). +// No real agent is involved: the request is shaped exactly the way the dsh / +// pi / omp / opencode envelopes stamp it. +import assert from "node:assert"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; + +process.env.NODE_ENV = "test"; + +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; + +interface Captured { body: string; headers: Record; url: string } + +function anthropicSse(event: string, data: unknown): string { + return `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`; +} + +function textScript(): string[] { + return [ + anthropicSse("message_start", { type: "message_start", message: { id: "msg_probe_1", role: "assistant", usage: { input_tokens: 10 } } }), + anthropicSse("content_block_start", { type: "content_block_start", index: 0, content_block: { type: "text", text: "" } }), + anthropicSse("content_block_delta", { type: "content_block_delta", index: 0, delta: { type: "text_delta", text: "ok" } }), + anthropicSse("content_block_stop", { type: "content_block_stop", index: 0 }), + anthropicSse("message_delta", { type: "message_delta", delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 2 } }), + anthropicSse("message_stop", { type: "message_stop" }), + ]; +} + +interface Harness { + proxyPort: number; + upstreamPort: number; + captured: Captured[]; + close(): Promise; +} + +async function startHarness(): Promise { + const captured: Captured[] = []; + const upstream = http.createServer((req, res) => { + let body = ""; + req.on("data", (c) => body += c); + req.on("end", () => { + captured.push({ body, headers: req.headers, url: req.url ?? "" }); + res.writeHead(200, { "content-type": "text/event-stream" }); + res.write(textScript().join("")); + res.end(); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + const proxy = await startServer({ + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [`http://127.0.0.1:${upstreamPort}`]: { models: { "claude-test": { context: 400_000 } } } }, + modelContextLimit: 400_000, + kernelConfig: defaultConfig(400_000), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + autoUpdate: false, + compat: { roles: {} }, + streamErrorShape: "protocol", + passthroughSource: null, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: false, + releaseNotesCheck: false, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + return { + proxyPort, upstreamPort, captured, + close: async () => { + proxy.close(); + proxy.closeIdleConnections?.(); + await once(proxy, "close"); + upstream.close(); + upstream.closeIdleConnections?.(); + await once(upstream, "close"); + }, + }; +} + +const HOST_IDS = ["msg_host_a", "msg_host_b", "msg_host_c"]; + +async function sendStamped(h: Harness, conv: string, opts: { bypass?: boolean }): Promise { + const headers: Record = { + "content-type": "application/json", + "x-bili-plugin": "e2e-agent", + "x-bili-plugin-conversation": conv, + "x-acp-session": conv, + }; + if (opts.bypass) headers["x-bili-plugin-bypass"] = "1"; + const resp = await fetch(`http://127.0.0.1:${h.proxyPort}/bili/http://127.0.0.1:${h.upstreamPort}/v1/messages`, { + method: "POST", + headers, + body: JSON.stringify({ + model: "claude-test", + max_tokens: 1024, + stream: true, + system: "You are a test assistant.", + messages: [ + { role: "user", content: "hello" }, + { role: "assistant", content: "hi there" }, + { role: "user", content: "how are you" }, + ], + // #6539 envelope: host-stable ids, CORE-aligned (anthropic has no + // hoist — 3 messages = 3 cores). + biliMsgIds: HOST_IDS, + }), + }); + // drain the SSE stream + if (resp.body) { for await (const _ of resp.body) { /* drain */ } } + return resp.status; +} + +test("e2e msgid envelope A: normal plugin lane pins host ids + strips the field from the upstream", async () => { + const h = await startHarness(); + try { + const status = await sendStamped(h, "conv-pin-a", {}); + assert.equal(status, 200); + + // (1) the stamped field never reaches the upstream (prepare stripped it) + assert.equal(h.captured.length, 1); + const upstreamBody = h.captured[0]!.body; + assert.ok(!upstreamBody.includes("biliMsgIds"), "upstream must not see biliMsgIds"); + assert.ok(!upstreamBody.includes("msg_host_a"), "upstream must not see the host ids"); + + // (2) the store is PINNED: the plugin snapshot's rawIds are the host ids + const snap = await fetch(`http://127.0.0.1:${h.proxyPort}/__bili/plugin/snapshot?conversationId=${encodeURIComponent("conv-pin-a")}`); + assert.equal(snap.status, 200); + const data = (await snap.json()) as Record; + assert.equal(data.ok, true); + const msgs = (data.messages as Array>).map((m) => m.rawId); + // The pin made EVERY host id stick (a content digest would be h_*, never + // msg_host_*). A missing id means churn (the row was re-ingested as a + // digest) — the exact bug this PR fixes. + for (const id of HOST_IDS) { + assert.ok(msgs.includes(id), `expected pinned host rawId "${id}", got: ${JSON.stringify(msgs)}`); + } + } finally { + await h.close(); + } +}); + +test("e2e msgid envelope B: verbatim (bypass) lane scrubs the stamped field (F1)", async () => { + const h = await startHarness(); + try { + const status = await sendStamped(h, "conv-scrub-b", { bypass: true }); + assert.equal(status, 200); + assert.equal(h.captured.length, 1); + const upstreamBody = h.captured[0]!.body; + assert.ok(!upstreamBody.includes("biliMsgIds"), "bypass lane must scrub biliMsgIds (F1)"); + assert.ok(!upstreamBody.includes("msg_host_a"), "bypass lane must not leak host ids"); + } finally { + await h.close(); + } +}); diff --git a/tests/e2e/e2e-advisory-rollback.test.ts b/tests/e2e/e2e-advisory-rollback.test.ts index f7082e2053..09a46eeae3 100644 --- a/tests/e2e/e2e-advisory-rollback.test.ts +++ b/tests/e2e/e2e-advisory-rollback.test.ts @@ -74,8 +74,14 @@ function freePort(): Promise { } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } // The updater's uninstall→reinstall half-deletes the install dir (npm drops diff --git a/tests/e2e/e2e-delegate-notify.test.ts b/tests/e2e/e2e-delegate-notify.test.ts index 0a5103b4b4..58ae1b0bc4 100644 --- a/tests/e2e/e2e-delegate-notify.test.ts +++ b/tests/e2e/e2e-delegate-notify.test.ts @@ -265,6 +265,12 @@ async function runScenario(name: string, prompt: string, totalDirectives: number XDG_CACHE_HOME: ctx.xdg.cache, XDG_STATE_HOME: ctx.xdg.state, XDG_DATA_HOME: ctx.xdg.data, + // The scripted filler issues 9 identical fast `sleep 2` calls; the + // repetition guard's rapidity ladder (warn 3 / refuse 5 / abort 6) then + // trips on fast runners (no round reaches the 5s long-poll reset) and + // aborts the turn mid-scenario — a runner-speed coin flip. The guard is + // covered by its own unit suite and is orthogonal to #2320/#2546 here. + BILI_GUARDRAIL_REPETITION: "0", ...(cliEntry ? { PI_CLI_PATH: cliEntry } : {}), }); pi = new RpcClient(ctx.piCwd, env); diff --git a/tests/e2e/e2e-native-dsh-persona.test.ts b/tests/e2e/e2e-native-dsh-persona.test.ts index 06d0d4ab65..71206e723d 100644 --- a/tests/e2e/e2e-native-dsh-persona.test.ts +++ b/tests/e2e/e2e-native-dsh-persona.test.ts @@ -77,6 +77,10 @@ function cleanEnv(): NodeJS.ProcessEnv { if (key.startsWith("BILI") || key.startsWith("BILLION_CONTEXT") || key.startsWith("ACP_") || key.startsWith("DSH_") || key.startsWith("ZCODE")) delete env[key]; } for (const key of ["NODE_EXTRA_CA_CERTS", "NODE_OPTIONS", "NODE_TEST_CONTEXT", "HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "NO_PROXY", "no_proxy"]) delete env[key]; + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface (raw + // conversation keys persisting sessions), so pin the dual-write writer. + env.BILI_STORAGE_JSON_WRITER = "on"; return env; } diff --git a/tests/e2e/e2e-native-dsh.test.ts b/tests/e2e/e2e-native-dsh.test.ts index 93d2521c7e..b1a01c50ba 100644 --- a/tests/e2e/e2e-native-dsh.test.ts +++ b/tests/e2e/e2e-native-dsh.test.ts @@ -218,6 +218,10 @@ function cleanEnv(): NodeJS.ProcessEnv { ) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT: set by the node:test runner and inherited by the // spawned dsh — dsh-native.ts deliberately stands down inside node:test // (unit-test guard), which would silently disable the very bootstrap this diff --git a/tests/e2e/e2e-native-opencode.test.ts b/tests/e2e/e2e-native-opencode.test.ts index cf2200f2d2..b9355971e6 100644 --- a/tests/e2e/e2e-native-opencode.test.ts +++ b/tests/e2e/e2e-native-opencode.test.ts @@ -183,6 +183,10 @@ function cleanEnv(): NodeJS.ProcessEnv { ) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT: set by the node:test runner and inherited by the spawned // client — the native entry deliberately stands down inside node:test (unit- // test guard), which would silently disable the bootstrap this suite diff --git a/tests/e2e/e2e-native-pi.test.ts b/tests/e2e/e2e-native-pi.test.ts index 997d14734f..b180ed7cff 100644 --- a/tests/e2e/e2e-native-pi.test.ts +++ b/tests/e2e/e2e-native-pi.test.ts @@ -205,6 +205,10 @@ function cleanEnv(): NodeJS.ProcessEnv { ) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT: set by the node:test runner and inherited by the spawned // pi — pi-native.ts deliberately stands down inside node:test (unit-test // guard), which would silently disable the very bootstrap this suite diff --git a/tests/e2e/e2e-pi-codex-ws.test.ts b/tests/e2e/e2e-pi-codex-ws.test.ts index bab066d3f4..9c57ff1a1f 100644 --- a/tests/e2e/e2e-pi-codex-ws.test.ts +++ b/tests/e2e/e2e-pi-codex-ws.test.ts @@ -257,6 +257,10 @@ function cleanEnv(extra: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { for (const key of Object.keys(env)) { if (key.startsWith("BILI") || key.startsWith("BILLION_CONTEXT") || key.startsWith("ACP_")) delete env[key]; } + // #6480 flipped the absent-default to pure-sqlite (JSON envelope writer + // off) — this suite asserts the legacy JSON persistence surface, so pin + // the dual-write writer explicitly. + env.BILI_STORAGE_JSON_WRITER = "on"; // NODE_TEST_CONTEXT stands down pi-native inside node:test; the spawned pi // is a real client, not a test context (same rationale as e2e-native-pi). for (const key of ["NODE_EXTRA_CA_CERTS", "PI_SESSION_FILE", "PI_PROVIDER", "PI_MODEL", "NODE_OPTIONS", "NODE_TEST_CONTEXT"]) delete env[key]; diff --git a/tests/e2e/e2e-registry.test.ts b/tests/e2e/e2e-registry.test.ts index 459737a209..0eba927299 100644 --- a/tests/e2e/e2e-registry.test.ts +++ b/tests/e2e/e2e-registry.test.ts @@ -48,8 +48,14 @@ function runBili(installDir: string, args: string[], env: Record return { code: res.status ?? -1, stdout: res.stdout ?? "", stderr: res.stderr ?? "" }; } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } // Stage a publishable tarball of THIS package at a synthetic version: same diff --git a/tests/e2e/e2e-release-canary.test.ts b/tests/e2e/e2e-release-canary.test.ts index d6931b7bca..7ac7ee59fa 100644 --- a/tests/e2e/e2e-release-canary.test.ts +++ b/tests/e2e/e2e-release-canary.test.ts @@ -125,8 +125,14 @@ async function extractInstall(work: string, tgz: string): Promise { return installDir; } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } function freePort(): Promise { diff --git a/tests/e2e/e2e-update-state.test.ts b/tests/e2e/e2e-update-state.test.ts index e31baf7659..158ff10081 100644 --- a/tests/e2e/e2e-update-state.test.ts +++ b/tests/e2e/e2e-update-state.test.ts @@ -83,8 +83,14 @@ async function extractInstall(work: string, tgz: string): Promise { return installDir; } -async function readPkgVersion(dir: string): Promise { - return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; +async function readPkgVersion(dir: string): Promise { + try { + return (JSON.parse(await fs.promises.readFile(path.join(dir, "package.json"), "utf8")) as { version: string }).version; + } catch (err) { + // A poll landing mid-reinstall sees no package.json yet — "not yet", not an error. + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + throw err; + } } function freePort(): Promise { diff --git a/tests/egress-authority-ladder.test.ts b/tests/egress-authority-ladder.test.ts new file mode 100644 index 0000000000..a162cdb1c1 --- /dev/null +++ b/tests/egress-authority-ladder.test.ts @@ -0,0 +1,142 @@ +import * as assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { test } from "node:test"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { migrateSchema } from "../src/storage/schema.ts"; +import { egressStateAppend, egressStateAt, egressStateView } from "../src/storage/egress-state.ts"; +import { recordEgressTurn, recoverEgressBase, type EgressOpLogStore } from "../src/storage/egress-record.ts"; +import { egressAuthorityProbe } from "../src/server/prepare-openai.ts"; +import type { BiliMessage } from "acp-kernel/wire"; + +const S = "sess-authority"; +const M = (id: string, text: string, role: BiliMessage["role"] = "user"): BiliMessage => ({ id, role, contentType: "text", text }); + +function open() { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-authority-")); + const { db } = openSqliteDatabase(path.join(dir, "index.db")); + migrateSchema(db); + const store: EgressOpLogStore = { + egressStateAppendOps: (sid, ops, nowMs) => egressStateAppend(db, sid, ops, nowMs), + egressStateView: (sid) => egressStateView(db, sid), + egressStateCompactView: (sid, view, nowMs) => db.prepare(`SELECT 1`).get() ? 0 : 0, + egressStateViewAt: (sid, seq) => egressStateAt(db, sid, seq), + egressStateMaxSeq: (sid) => (db.prepare(`SELECT COALESCE(MAX(log_seq), 0) AS mx FROM egress_flow WHERE session_id = ?`).get(sid) as { mx: number }).mx, + egressStateTruncateAfter: (sid, seq) => { + const a = db.prepare(`DELETE FROM egress_state WHERE session_id = ? AND log_seq > ?`).run(sid, seq); + const b = db.prepare(`DELETE FROM egress_flow WHERE session_id = ? AND log_seq > ?`).run(sid, seq); + return Number(a.changes) + Number(b.changes); + }, + }; + return { db, store, dir }; +} + +test("recoverEgressBase: healthy flow returns the current point (no degrade, no truncation)", () => { + const H = open(); + try { + let view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: [M("m1", "alpha")], systemText: "sys", nowMs: 1 }); + view = recordEgressTurn({ store: H.store, sessionId: S, prev: view, conversation: [M("m1", "alpha"), M("m2", "beta")], systemText: "sys", nowMs: 2 }); + const got = recoverEgressBase(H.store, S); + assert.ok(got !== null, "a healthy flow always recovers"); + assert.equal(got.degraded, false); + assert.equal(got.truncated, 0); + assert.equal(got.seq, 3, "system I + m1 I (turn 1) + m2 I (turn 2)"); + assert.equal(got.view.length, 3, "system + m1 + m2"); + } finally { + H.db.close(); + fs.rmSync(H.dir, { recursive: true, force: true }); + } +}); + +test("recoverEgressBase: corrupt tail walks back to the newest healthy point and truncates past it", () => { + const H = open(); + try { + let view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: [M("m1", "alpha")], systemText: "sys", nowMs: 1 }); + view = recordEgressTurn({ store: H.store, sessionId: S, prev: view, conversation: [M("m1", "alpha"), M("m2", "beta")], systemText: "sys", nowMs: 2 }); + // structurally corrupt the newest row (log_seq 3 = m2's I op) in BOTH tables + H.db.prepare(`UPDATE egress_flow SET payload = ? WHERE session_id = ? AND log_seq = 3`).run('{"x":1}', S); + H.db.prepare(`UPDATE egress_state SET payload = ? WHERE session_id = ? AND log_seq = 3`).run('{"x":1}', S); + const got = recoverEgressBase(H.store, S); + assert.ok(got !== null); + assert.equal(got.degraded, true); + assert.equal(got.seq, 2, "newest healthy point = before the corrupt row"); + assert.equal(got.truncated, 2, "corrupt row removed from BOTH tables"); + assert.equal(H.store.egressStateMaxSeq(S), 2, "flow max collapsed to the healthy point"); + // healthy replay still works and records can continue after recovery + view = recordEgressTurn({ store: H.store, sessionId: S, prev: got.view, conversation: [M("m1", "alpha"), M("m3", "gamma")], systemText: "sys", nowMs: 3 }); + assert.equal(view.length, 3, "system + m1 + m3 after recovery"); + } finally { + H.db.close(); + fs.rmSync(H.dir, { recursive: true, force: true }); + } +}); + +test("recoverEgressBase: no healthy point at all returns null (session-level recovery)", () => { + const H = open(); + try { + let view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: [M("m1", "alpha")], systemText: "sys", nowMs: 1 }); + view = recordEgressTurn({ store: H.store, sessionId: S, prev: view, conversation: [M("m1", "alpha"), M("m2", "beta")], systemText: "sys", nowMs: 2 }); + // structurally corrupt EVERY op row in both tables => the only healthy + // point is the empty flow (seq 0) = the bootstrap point + H.db.prepare(`UPDATE egress_flow SET payload = ? WHERE session_id = ? AND row_type != 'D'`).run('{"x":1}', S); + H.db.prepare(`UPDATE egress_state SET payload = ? WHERE session_id = ? AND row_type != 'D'`).run('{"x":1}', S); + const got = recoverEgressBase(H.store, S); + assert.ok(got !== null, "the empty flow is always a valid recovery point"); + assert.equal(got.degraded, true); + assert.equal(got.seq, 0, "recovered to the bootstrap point"); + assert.equal(got.view.length, 0); + assert.equal(H.store.egressStateMaxSeq(S), 0, "flow collapsed to empty"); + // the next turn re-records the full batch (self-heal) + view = recordEgressTurn({ store: H.store, sessionId: S, prev: got.view, conversation: [M("m1", "alpha")], systemText: "sys", nowMs: 3 }); + assert.equal(view.length, 2, "bootstrap re-record: system + m1"); + } finally { + H.db.close(); + fs.rmSync(H.dir, { recursive: true, force: true }); + } +}); + +const quiet = () => {}; + +test("egressAuthorityProbe: identity + order + sampled-byte rows (#3617 both directions)", () => { + const base = [M("m1", "alpha"), M("m2", "beta"), M("m3", "gamma")]; + // unchanged client resend => pass + assert.equal(egressAuthorityProbe(base, [base[0], base[1], base[2], M("m4", "delta")], S, quiet), true, "identical rows + fresh tail pass"); + // reordered client row => fail (legacy this turn) + assert.equal(egressAuthorityProbe(base, [base[1], base[0], base[2]], S, quiet), false, "reorder fails the probe"); + // byte divergence under a stable msgid => STILL passes (server bytes win; divergence is logged, not fatal) + assert.equal(egressAuthorityProbe(base, [base[0], M("m2", "tampered beta"), base[2]], S, quiet), true, "divergence is non-fatal (ruling 7439-5: server wins)"); + // pure fresh tail (empty base ids in client) => pass + assert.equal(egressAuthorityProbe(base, [M("m9", "fresh")], S, quiet), true, "all-fresh client rows pass"); +}); + +import { serveLaneCore, recordLaneTurn } from "../src/server/egress-lane-io.ts"; + +test("egress-lane-io: serve/record round-trip + corrupt degrade (all-lanes helper, ruling 7439-4)", () => { + const H = open(); + try { + // empty flow => nothing to serve (bootstrap turn records instead) + assert.equal(serveLaneCore(H.store, S, quiet), null, "empty flow serves nothing"); + recordLaneTurn(H.store, S, [M("m1", "alpha"), M("m2", "beta")], "sys", quiet); + // second turn: the recorded core comes back lane-neutral (no system row, no wire shape) + const served = serveLaneCore(H.store, S, quiet); + assert.ok(served !== null, "a recorded flow serves"); + assert.deepEqual(served.map((m) => m.id), ["m1", "m2"], "core rows only — the system row stays out"); + assert.equal(served[0].text, "alpha"); + // a single corrupt tail row does NOT degrade — the ladder walks back + // and truncates, then serves the newest healthy point + H.db.prepare(`UPDATE egress_flow SET payload = ? WHERE session_id = ? AND log_seq = 3`).run('{"x":1}', S); + H.db.prepare(`UPDATE egress_state SET payload = ? WHERE session_id = ? AND log_seq = 3`).run('{"x":1}', S); + const partial = serveLaneCore(H.store, S, quiet); + assert.ok(partial !== null, "ladder recovery still serves the healthy point"); + assert.deepEqual(partial.map((m) => m.id), ["m1"], "served core = the healthy point (m2 row truncated away)"); + // total corruption (no healthy point except the empty flow) => serve + // degrades to null and the lane keeps its legacy core + H.db.prepare(`UPDATE egress_flow SET payload = ? WHERE session_id = ? AND row_type != 'D'`).run('{"x":1}', S); + H.db.prepare(`UPDATE egress_state SET payload = ? WHERE session_id = ? AND row_type != 'D'`).run('{"x":1}', S); + assert.equal(serveLaneCore(H.store, S, quiet), null, "no healthy point degrades to legacy (never throws)"); + } finally { + H.db.close(); + fs.rmSync(H.dir, { recursive: true, force: true }); + } +}); diff --git a/tests/egress-oplog-lifecycle.test.ts b/tests/egress-oplog-lifecycle.test.ts new file mode 100644 index 0000000000..8530233cb0 --- /dev/null +++ b/tests/egress-oplog-lifecycle.test.ts @@ -0,0 +1,101 @@ +/** + * Egress op-log self-containment (revised design discussions/7439 ruling 1, + * #6785): the op-log must be byte-self-contained — recording and replaying + * NEVER touch the global content table (the old casRef design put every row + * into it with a refcount bump per turn; that seam is gone): + * 1. three recorded turns (append / fold / edit) leave the content table + * EMPTY and every view row carries its bytes inline; replayed views + * stay byte-equal across turns; + * 2. deleteSession cascades egress_state/egress_flow — the tables are now + * plain rows (no CAS refs to release), and the session disappears + * atomically. + * Both directions asserted (#3617): the content table IS untouched on the + * happy path AND the cascade DOES delete (a kept row would be a leak). + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { openSqliteDatabase } from "../src/storage/driver.js"; +import { migrateSchema } from "../src/storage/schema.js"; +import { egressStateAppend, egressStateView } from "../src/storage/egress-state.js"; +import { buildEgressViewRows, recordEgressTurn, type EgressOpLogStore } from "../src/storage/egress-record.js"; +import { rebuildCoreFromLog } from "../src/storage/egress-serve.js"; +import { UnifiedStore } from "../src/storage/store.js"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = mkdtempSync(join(tmpdir(), "bili-egress-lifecycle-hermetic-")); +process.env.BILI_CONFIG_FILE = join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +const M = (id: string, text: string, role: string = "user") => ({ id, role, contentType: "text", text }); + +test("recording is content-table-free: rows carry bytes, content stays empty, replay is byte-equal", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-egress-lifecycle-")); + try { + const store = UnifiedStore.open(join(dir, "index.db")); + const db = (store as unknown as { db: ReturnType["db"] }).db; + migrateSchema(db); + const sid = "sess-egress-selfcontained"; + + const ops: EgressOpLogStore = { + egressStateAppendOps: (s, o, nowMs) => store.egressStateAppendOps(s, o, nowMs), + egressStateView: (s) => store.egressStateView(s), + egressStateCompactView: (s, v, nowMs) => store.egressStateCompactView(s, v, nowMs), + egressStateViewAt: (s, x) => store.egressStateViewAt(s, x), + egressStateMaxSeq: (s) => store.egressStateMaxSeq(s), + egressStateTruncateAfter: (s, x) => store.egressStateTruncateAfter(s, x), + }; + // turn 1: two messages; turn 2: tail append + fold shape; turn 3: an edit + let view = recordEgressTurn({ store: ops, sessionId: sid, prev: undefined, conversation: [M("m1", "alpha"), M("m2", "beta")], systemText: "sys", nowMs: 1000 }); + view = recordEgressTurn({ store: ops, sessionId: sid, prev: view, conversation: [M("m1", "alpha"), { id: "acp_summary_1", role: "system", contentType: "text", text: "beta" }], systemText: "sys", nowMs: 2000 }); + view = recordEgressTurn({ store: ops, sessionId: sid, prev: view, conversation: [M("m1", "alpha!"), { id: "acp_summary_1", role: "system", contentType: "text", text: "beta" }], systemText: "sys", nowMs: 3000 }); + assert.ok(view.every((r) => typeof r.bytes === "string" && r.bytes.length > 0), "every row carries its bytes inline"); + const rebuilt = rebuildCoreFromLog(view); + assert.ok(rebuilt !== null, "the in-turn view rebuilds"); + assert.equal(JSON.stringify(rebuilt), JSON.stringify([M("m1", "alpha!"), { id: "acp_summary_1", role: "system", contentType: "text", text: "beta" }]), "rebuilt core equals the recorded conversation"); + + // the replayed (table) view is byte-equal to the in-turn view + const fromTable = egressStateView(db, sid); + assert.deepEqual(fromTable.map((r) => [r.msgid, r.bytes]), view.map((r) => [r.msgid, r.bytes]), "table replay is byte-equal"); + + // THE self-containment claim: three turns of recording + fold + + // edit appended NOTHING to the content table. + const contentRows = (db.prepare("SELECT count(*) AS n FROM content").get() as { n: number }).n; + assert.equal(contentRows, 0, "the content table stayed untouched — the op-log is self-contained"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("deleteSession cascades egress_state/egress_flow (plain rows, no CAS to release)", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-egress-lifecycle-")); + try { + const store = UnifiedStore.open(join(dir, "index.db")); + const db = (store as unknown as { db: ReturnType["db"] }).db; + migrateSchema(db); + const sid = "sess-egress-cascade"; + + egressStateAppend(db, sid, [ + { type: "I", msgid: "m1", after: null, ref: "", tokens: 5, typeTag: "text", bytes: '{"id":"m1","role":"user","text":"alpha"}', role: "user" }, + { type: "I", msgid: "m2", after: "m1", ref: "", tokens: 5, typeTag: "text", bytes: '{"id":"m2","role":"user","text":"beta"}', role: "user" }, + { type: "D", start: "m1", end: "m1" }, + { type: "U", msgid: "m2", bytes: '{"id":"m2","role":"user","text":"beta!"}', role: "user" }, + ], Date.now()); + const count = (t: string) => (db.prepare(`SELECT count(*) AS n FROM ${t}`).get() as { n: number }).n; + assert.equal(count("egress_state"), 4); + assert.equal(count("egress_flow"), 4); + const v = egressStateView(db, sid); + assert.deepEqual(v.map((r) => r.msgid), ["m2"]); + assert.equal(v[0].bytes, '{"id":"m2","role":"user","text":"beta!"}', "the +U replaced the bytes in place"); + + store.deleteSession(sid); + + assert.equal(count("egress_state"), 0, "egress_state rows cascade-deleted"); + assert.equal(count("egress_flow"), 0, "egress_flow rows cascade-deleted"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/tests/egress-record.test.ts b/tests/egress-record.test.ts new file mode 100644 index 0000000000..4ebf5c411a --- /dev/null +++ b/tests/egress-record.test.ts @@ -0,0 +1,177 @@ +/** + * Egress op-log RECORDING (seg 3, proposal discussions/7398, #6785 perf + * work): shadow recording of the outbound conversation view as sparse ops. + * Pinned contract: + * - buildEgressViewRows determinism (same inputs → same rows; content + * edit → bytes change only; system block row present); + * - turn 1 = one +I per row; turn 2 tail append = exactly ONE +I op; + * - the four upper-layer shapes produce the expected op batches + * (append / nudge-excluded / fold = -D + +I summary / edit = +U); + * - a -D batch triggers compaction (state table collapses to base + tail); + * - #3617 both directions: with the rollout key off the recorder is not + * called (zero rows) — asserted at the gate level (featureEnabled + * default) since the wiring sits behind it in prepare-openai. + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { migrateSchema } from "../src/storage/schema.ts"; +import { + egressStateAt, + egressStateAppend, + egressStateCompact, + egressStateView, +} from "../src/storage/egress-state.ts"; +import { EGRESS_SYSTEM_MSGID, buildEgressViewRows, recordEgressTurn, type EgressOpLogStore } from "../src/storage/egress-record.ts"; +import { featureEnabled } from "../src/rollout.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-record-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +interface Harness { + db: ReturnType["db"]; + store: EgressOpLogStore; + dir: string; +} + +function open(): Harness { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-record-")); + const { db } = openSqliteDatabase(path.join(dir, "index.db")); + migrateSchema(db); + const store: EgressOpLogStore = { + egressStateAppendOps: (sid, ops, nowMs) => egressStateAppend(db, sid, ops, nowMs), + egressStateView: (sid) => egressStateView(db, sid), + egressStateCompactView: (sid, view, nowMs) => egressStateCompact(db, sid, view, nowMs), + egressStateViewAt: (sid, seq) => egressStateAt(db, sid, seq), + egressStateMaxSeq: (sid) => (db.prepare(`SELECT COALESCE(MAX(log_seq), 0) AS mx FROM egress_flow WHERE session_id = ?`).get(sid) as { mx: number }).mx, + egressStateTruncateAfter: (sid, seq) => { + const a = db.prepare(`DELETE FROM egress_state WHERE session_id = ? AND log_seq > ?`).run(sid, seq); + const b = db.prepare(`DELETE FROM egress_flow WHERE session_id = ? AND log_seq > ?`).run(sid, seq); + return Number(a.changes) + Number(b.changes); + }, + }; + return { db, store, dir }; +} + +function close(H: Harness): void { + H.db.close(); + fs.rmSync(H.dir, { recursive: true, force: true }); +} + +const M = (id: string, text: string, role = "user") => ({ id, role, contentType: "text", text }); + +test("buildEgressViewRows: deterministic, system row first, content-edit visible", () => { + const conv = [M("a", "hello"), M("b", "world", "assistant")]; + const r1 = buildEgressViewRows(conv, "SYS"); + const r2 = buildEgressViewRows([M("a", "hello"), M("b", "world", "assistant")], "SYS"); + assert.deepEqual(r1, r2, "same inputs → identical rows"); + assert.equal(r1[0].msgid, EGRESS_SYSTEM_MSGID, "system row carries the constant msgid (no hash)"); + assert.equal(r1.length, 3); + assert.equal(r1[1].msgid, "a"); + assert.equal(r1[1].type, "text"); + const edited = buildEgressViewRows([M("a", "hello"), M("b", "EDITED", "assistant")], "SYS"); + assert.equal(edited[2].bytes !== r1[2].bytes, true, "content edit changes bytes"); + assert.equal(edited[2].msgid, "b", "msgid stable across the edit"); + assert.deepEqual(buildEgressViewRows([], undefined), []); + // summary rows are typed acp_summary + const s = buildEgressViewRows([M("acp_summary_1", "…", "assistant")], undefined); + assert.equal(s[0].type, "acp_summary"); +}); + +test("recording: turn1 = full +I batch; turn2 tail append = exactly one +I", () => { + const H = open(); + try { + const S = "sess-r1"; + const t1 = [M("a", "one"), M("b", "two", "assistant")]; + let view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: t1, systemText: "SYS", nowMs: 1000 }); + assert.equal(view.length, 3); + let rows = H.db.prepare("SELECT COUNT(*) AS c FROM egress_flow WHERE session_id = ?").get(S) as { c: number }; + assert.equal(rows.c, 3, "flow has one +I row per view row"); + assert.equal(view.every((r) => r.ref === "" || /^m\d{4,}$/.test(r.ref)), true); + + const t2 = [t1[0], t1[1], M("c", "three")]; + view = recordEgressTurn({ store: H.store, sessionId: S, prev: view, conversation: t2, systemText: "SYS", nowMs: 2000 }); + const added = H.db.prepare("SELECT COUNT(*) AS c FROM egress_flow WHERE session_id = ? AND log_seq > 3").get(S) as { c: number }; + assert.equal(added.c, 1, "tail append records exactly ONE new op row"); + const last = H.db.prepare("SELECT row_type, msgid FROM egress_flow WHERE session_id = ? ORDER BY log_seq DESC LIMIT 1").get(S) as { row_type: string; msgid: string }; + assert.deepEqual({ row_type: last.row_type, msgid: last.msgid }, { row_type: "I", msgid: "c" }); + assert.deepEqual(egressStateView(H.db, S).map((r) => r.msgid), [EGRESS_SYSTEM_MSGID, "a", "b", "c"]); + } finally { + close(H); + } +}); + +test("recording: fold shape = -D range + +I summary, and the -D triggers compaction", () => { + const H = open(); + try { + const S = "sess-r2"; + const store = H.store; + let view = recordEgressTurn({ store, sessionId: S, prev: undefined, conversation: [M("a", "1"), M("b", "2"), M("c", "3"), M("d", "4"), M("e", "5")], systemText: "SYS", nowMs: 1000 }); + // fold: b..d replaced by one summary row + const folded = [M("a", "1"), M("acp_summary_9", "…", "assistant"), M("e", "5")]; + view = recordEgressTurn({ store, sessionId: S, prev: view, conversation: folded, systemText: "SYS", nowMs: 2000 }); + const ops = H.db.prepare("SELECT row_type, msgid FROM egress_flow WHERE session_id = ? AND log_seq > 5 ORDER BY log_seq").all(S) as { row_type: string; msgid: string | null }[]; + const types = ops.map((o) => o.row_type); + assert.ok(types.includes("D"), "fold emits a -D"); + assert.ok(types.includes("I"), "fold emits a +I for the summary"); + assert.equal(view.length, 4); + // compaction fired: state table = base row + nothing else (the I was + // replayed INTO the base by the same compact call) + const stateRows = H.db.prepare("SELECT row_type FROM egress_state WHERE session_id = ? ORDER BY log_seq").all(S) as { row_type: string }[]; + assert.ok(stateRows.some((r) => r.row_type === "B"), "compaction wrote a base row"); + assert.equal(egressStateView(H.db, S).length, 4, "view has the system row + a + summary + e"); + assert.deepEqual(egressStateView(H.db, S).map((r) => r.msgid), view.map((r) => r.msgid), "view survives compaction byte-identical"); + } finally { + close(H); + } +}); + +test("recording: content edit of one row = exactly one +U, no D/I churn", () => { + const H = open(); + try { + const S = "sess-r3"; + const store = H.store; + let view = recordEgressTurn({ store, sessionId: S, prev: undefined, conversation: [M("a", "1"), M("b", "2", "assistant"), M("c", "3")], systemText: "SYS", nowMs: 1000 }); + view = recordEgressTurn({ store, sessionId: S, prev: view, conversation: [M("a", "1"), M("b", "2-edited", "assistant"), M("c", "3")], systemText: "SYS", nowMs: 2000 }); + const rows = H.db.prepare("SELECT row_type, msgid FROM egress_flow WHERE session_id = ? AND log_seq > 4").all(S) as { row_type: string; msgid: string | null }[]; + assert.equal(rows.length, 1, "exactly one op"); + assert.equal(rows[0].row_type, "U", "it is a +U"); + assert.equal(rows[0].msgid, "b", "targeting the edited row"); + const expected = buildEgressViewRows([M("a", "1"), M("b", "2-edited", "assistant"), M("c", "3")], "SYS"); + const got = egressStateView(H.db, S); + assert.deepEqual(got.map((r) => [r.msgid, r.bytes]), expected.map((r) => [r.msgid, r.bytes]), "replayed view carries the edited bytes"); + } finally { + close(H); + } +}); + +test("recording: nudge-shaped churn is EXCLUDED (ephemeral rows never recorded)", () => { + const H = open(); + try { + const S = "sess-r4"; + const store = H.store; + let view = recordEgressTurn({ store, sessionId: S, prev: undefined, conversation: [M("a", "1"), M("b", "2", "assistant")], systemText: "SYS", nowMs: 1000 }); + // the prepare layer passes processedMessages (kernel view) — the + // nudge is appended AFTER this seam by design, so it is structurally + // absent; pin that the recorder's contract is the kernel view shape: + view = recordEgressTurn({ store, sessionId: S, prev: view, conversation: [M("a", "1"), M("b", "2", "assistant")], systemText: "SYS", nowMs: 2000 }); + const added = H.db.prepare("SELECT COUNT(*) AS c FROM egress_flow WHERE session_id = ? AND log_seq > 3").get(S) as { c: number }; + assert.equal(added.c, 0, "identical view → zero ops"); + } finally { + close(H); + } +}); + +test("#3617 negative direction: rollout key OFF by default → gate never opens", () => { + delete process.env.BILI_ROLLOUT_EGRESS_PLANE; + assert.equal(featureEnabled("egressPlane"), false, "egressPlane defaults to 0 — the prepare-openai gate (featureEnabled) sees nothing and records zero rows"); + process.env.BILI_ROLLOUT_EGRESS_PLANE = "1"; + assert.equal(featureEnabled("egressPlane"), true); + delete process.env.BILI_ROLLOUT_EGRESS_PLANE; +}); diff --git a/tests/egress-serve.test.ts b/tests/egress-serve.test.ts new file mode 100644 index 0000000000..0082337103 --- /dev/null +++ b/tests/egress-serve.test.ts @@ -0,0 +1,163 @@ +/** + * Egress serve-from-log byte identity (seg 4a, proposal discussions/7398, + * #6785 perf work): rebuildOutboundFromLog must produce EXACTLY the bytes + * the legacy chain (systemToUser → coreToOpenai → hardenOpenaiAssistantContent) + * produces for the same messages — that is the claim the shadow comparison + * in prepare-openai measures, and the precondition for any future swap. + * Also pins the degrade directions (#3617): corrupted/empty row bytes → + * null (caller keeps the legacy path), never a partial rebuild. + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { isRenderedSummaryMessage } from "acp-kernel"; +import { coreToOpenai, type BiliMessage } from "acp-kernel/wire"; +import { hardenOpenaiAssistantContent, systemToUser } from "../src/util.ts"; +import { egressStateAt, egressStateAppend, egressStateCompact, egressStateView } from "../src/storage/egress-state.ts"; +import { recordEgressTurn, type EgressOpLogStore } from "../src/storage/egress-record.ts"; +import { rebuildCoreFromLog, rebuildOutboundFromLog } from "../src/storage/egress-serve.ts"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { migrateSchema } from "../src/storage/schema.ts"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-serve-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +interface Harness { + db: ReturnType["db"]; + store: EgressOpLogStore; + dir: string; +} + +function open(): Harness { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-serve-")); + const { db } = openSqliteDatabase(path.join(dir, "index.db")); + migrateSchema(db); + const store = { + egressStateAppendOps: (sid: string, ops: Parameters[2], nowMs: number) => egressStateAppend(db, sid, ops, nowMs), + egressStateView: (sid: string) => egressStateView(db, sid), + egressStateCompactView: (sid: string, view: Parameters[2], nowMs: number) => egressStateCompact(db, sid, view, nowMs), + + egressStateViewAt: (sid: string, seq: number) => egressStateAt(db, sid, seq), + egressStateMaxSeq: (sid: string) => (db.prepare(`SELECT COALESCE(MAX(log_seq), 0) AS mx FROM egress_flow WHERE session_id = ?`).get(sid) as { mx: number }).mx, + egressStateTruncateAfter: (sid: string, seq: number) => { + const a = db.prepare(`DELETE FROM egress_state WHERE session_id = ? AND log_seq > ?`).run(sid, seq); + const b = db.prepare(`DELETE FROM egress_flow WHERE session_id = ? AND log_seq > ?`).run(sid, seq); + return Number(a.changes) + Number(b.changes); + }, + }; + return { db, store, dir }; +} + +function close(H: Harness): void { + H.db.close(); + fs.rmSync(H.dir, { recursive: true, force: true }); +} + +const M = (id: string, text: string, role: "user" | "assistant" = "user"): BiliMessage => + ({ id, role, contentType: "text", text } as BiliMessage); + +function legacyChain(msgs: BiliMessage[]): unknown { + return hardenOpenaiAssistantContent(coreToOpenai(systemToUser(msgs, (m) => isRenderedSummaryMessage(m)))); +} + +test("byte identity: plain text conversation rebuilds EXACTLY the legacy elements", () => { + const H = open(); + try { + const S = "sess-s1"; + const msgs: BiliMessage[] = [M("a", "hello"), M("b", "world", "assistant"), M("c", "third")]; + let view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: msgs, systemText: "SYS", nowMs: 1000 }); + const reb = rebuildOutboundFromLog(view); + assert.ok(reb !== null); + assert.equal(JSON.stringify(reb!.elements), JSON.stringify(legacyChain(msgs)), "rebuilt elements are byte-identical to the legacy chain"); + // steady turn: tail append keeps identity on the replayed (table) view + view = recordEgressTurn({ store: H.store, sessionId: S, prev: view, conversation: [...msgs, M("d", "fourth", "assistant")], systemText: "SYS", nowMs: 2000 }); + const fromTable = egressStateView(H.db, S); + const reb2 = rebuildOutboundFromLog(fromTable); + assert.ok(reb2 !== null); + assert.equal(JSON.stringify(reb2!.elements), JSON.stringify(legacyChain([...msgs, M("d", "fourth", "assistant")])), "table-replayed view stays byte-identical"); + } finally { + close(H); + } +}); + +test("byte identity: acp_summary rows re-voice to user exactly like the legacy chain", () => { + const H = open(); + try { + const S = "sess-s2"; + const msgs: BiliMessage[] = [M("a", "hello"), { id: "acp_summary_1", role: "system", contentType: "text", text: "..." } as BiliMessage, M("c", "third")]; + const view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: msgs, systemText: undefined, nowMs: 1000 }); + const reb = rebuildOutboundFromLog(view); + assert.ok(reb !== null); + assert.equal(JSON.stringify(reb!.elements), JSON.stringify(legacyChain(msgs)), "summary re-voicing is byte-identical"); + } finally { + close(H); + } +}); + +test("degrade #1: corrupted row bytes → null (whole turn stays legacy)", () => { + const H = open(); + try { + const S = "sess-s3"; + const msgs: BiliMessage[] = [M("a", "hello"), M("b", "world", "assistant")]; + const view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: msgs, systemText: undefined, nowMs: 1000 }); + const corrupted = view.map((r) => r.msgid === "a" ? { ...r, bytes: "not-json{{" } : r); + assert.equal(rebuildOutboundFromLog(corrupted), null, "any corrupt row nulls the whole rebuild"); + } finally { + close(H); + } +}); + +test("degrade #2: empty row bytes / bad role → null", () => { + const H = open(); + try { + const S = "sess-s4"; + const msgs: BiliMessage[] = [M("a", "hello")]; + const view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: msgs, systemText: undefined, nowMs: 1000 }); + const empty = view.map((r) => r.msgid === "a" ? { ...r, bytes: "" } : r); + assert.equal(rebuildOutboundFromLog(empty), null, "an empty-bytes row is unrebuildable"); + const badRole = view.map((r) => r.msgid === "a" ? { ...r, bytes: JSON.stringify({ id: "a", role: "weird", contentType: "text", text: "x" }) } : r); + assert.equal(rebuildCoreFromLog(badRole), null, "a role outside the core whitelist is unrebuildable"); + void S; + } finally { + close(H); + } +}); + +test("byte identity: tool rows rebuild (the CAS body carries the sidecars)", () => { + const H = open(); + try { + const S = "sess-s5"; + const callMsg = { id: "t0", role: "assistant", contentType: "tool-call", text: "\"{\\\"q\\\":1}\"", toolCallId: "call_1", toolName: "bash" } as BiliMessage; + const toolMsg = { id: "t1", role: "tool", contentType: "text", text: "result", toolCallId: "call_1", toolName: "bash" } as BiliMessage; + const msgs: BiliMessage[] = [M("a", "hello"), callMsg, toolMsg, M("b", "after", "assistant")]; + const view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: msgs, systemText: undefined, nowMs: 1000 }); + const reb = rebuildOutboundFromLog(view); + assert.ok(reb !== null, "tool rows rebuild — the recorded CAS bytes are the full core message (toolCallId/toolName included)"); + assert.equal(JSON.stringify(reb!.elements), JSON.stringify(legacyChain(msgs)), "tool-heavy conversation is byte-identical to the legacy chain"); + } finally { + close(H); + } +}); + +test("byte identity survives compaction: fold shape → base row → still identical", () => { + const H = open(); + try { + const S = "sess-s6"; + const five: BiliMessage[] = [M("a", "1"), M("b", "2", "assistant"), M("c", "3"), M("d", "4", "assistant"), M("e", "5")]; + let view = recordEgressTurn({ store: H.store, sessionId: S, prev: undefined, conversation: five, systemText: undefined, nowMs: 1000 }); + const folded: BiliMessage[] = [M("a", "1"), { id: "acp_summary_9", role: "system", contentType: "text", text: "2..4" } as BiliMessage, M("e", "5")]; + view = recordEgressTurn({ store: H.store, sessionId: S, prev: view, conversation: folded, systemText: undefined, nowMs: 2000 }); + const fromTable = egressStateView(H.db, S); + assert.equal(fromTable.length, 3, "view is the folded shape (a + summary + e)"); + const reb = rebuildOutboundFromLog(fromTable); + assert.ok(reb !== null); + assert.equal(JSON.stringify(reb!.elements), JSON.stringify(legacyChain(folded)), "post-compaction rebuild is byte-identical"); + } finally { + close(H); + } +}); diff --git a/tests/egress-state.test.ts b/tests/egress-state.test.ts new file mode 100644 index 0000000000..430868c4ff --- /dev/null +++ b/tests/egress-state.test.ts @@ -0,0 +1,334 @@ +/** + * Egress state log (proposal discussions/7398, #6785 perf work): the op-log + * outbound view. Tests pin the CONTRACT: + * - replay determinism: same op sequence → same view, always; + * - the four row types (+I tail/anchored, -D inclusive range, +U + * content-only, B base) and their payload shapes; + * - the three hard-error paths (missing anchor / missing or inverted + * delete endpoints / update target missing) throw, never degrade fuzzy; + * - compaction equivalence: view before == view after, row count bounded; + * - dual-table rollback: ANY log_seq stays reconstructible from egress_flow + * after the state table is compacted below it. + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { migrateSchema } from "../src/storage/schema.ts"; +import { + egressStateAppend, + egressStateCompact, + egressStateView, + egressStateAt, + EgressStateCorruptError, + diffEgressView, + egressReplay, + type EgressOp, + type EgressViewRow, +} from "../src/storage/egress-state.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-state-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +interface Harness { + db: ReturnType["db"]; + dir: string; +} + +function open(): Harness { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-egress-state-")); + const { db } = openSqliteDatabase(path.join(dir, "index.db")); + migrateSchema(db); + return { db, dir }; +} + +function close(h: Harness): void { + h.db.close(); + try { + fs.rmSync(h.dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { /* leaked temp dir on a Windows runner is harmless */ } +} + +const h = (m: string) => `h_${m}`; + +test("replay determinism: the five-op example from the proposal", () => { + const H = open(); + try { + const S = "sess-1"; + egressStateAppend(H.db, S, [ + { type: "I", msgid: "a", after: null, ref: "m00001", tokens: 3, typeTag: "text", bytes: "h_a1" }, + { type: "I", msgid: "b", after: null, ref: "m00002", tokens: 4, typeTag: "text", bytes: "h_b1" }, + { type: "I", msgid: "c", after: "b", ref: "m00003", tokens: 9, typeTag: "summary", bytes: "h_c1" }, + { type: "D", start: "a", end: "b" }, + { type: "U", msgid: "c", bytes: "h_c2" }, + ], 1_000); + const view = egressStateView(H.db, S); + assert.deepEqual(view.map((r) => r.msgid), ["c"]); + assert.equal(view[0].bytes, h("c2")); + // ref/tokens/type survive the content update untouched + assert.equal(view[0].ref, "m00003"); + assert.equal(view[0].tokens, 9); + assert.equal(view[0].type, "summary"); + // replay twice → byte-identical + assert.deepEqual(egressStateView(H.db, S), view); + } finally { + close(H); + } +}); + +test("+I anchored insert lands mid-history; -D endpoints are msgids, not positions", () => { + const H = open(); + try { + const S = "sess-2"; + egressStateAppend(H.db, S, [ + { type: "I", msgid: "a", after: null, ref: "m00001", tokens: 1, typeTag: "text", bytes: "h_a" }, + { type: "I", msgid: "d", after: null, ref: "m00002", tokens: 1, typeTag: "text", bytes: "h_d" }, + { type: "I", msgid: "e", after: null, ref: "m00003", tokens: 1, typeTag: "text", bytes: "h_e" }, + // mid-history insert anchored at a (positions of d/e shift) + { type: "I", msgid: "b", after: "a", ref: "m00004", tokens: 1, typeTag: "text", bytes: "h_b" }, + { type: "I", msgid: "c", after: "b", ref: "m00005", tokens: 1, typeTag: "text", bytes: "h_c" }, + ], 1_000); + assert.deepEqual(egressStateView(H.db, S).map((r) => r.msgid), ["a", "b", "c", "d", "e"]); + // a fold deletes b..d — a and e survive in place + egressStateAppend(H.db, S, [{ type: "D", start: "b", end: "d" }], 2_000); + assert.deepEqual(egressStateView(H.db, S).map((r) => r.msgid), ["a", "e"]); + // single-row delete (start == end) + egressStateAppend(H.db, S, [{ type: "D", start: "a", end: "a" }], 3_000); + assert.deepEqual(egressStateView(H.db, S).map((r) => r.msgid), ["e"]); + } finally { + close(H); + } +}); + +test("hard errors: missing anchor / missing or inverted endpoints / update target missing / duplicate msgid (#3617 negative direction)", () => { + const H = open(); + try { + // each scenario gets a FRESH session: replay throws at the FIRST + // violating op in seq order, so stacking bad ops in one session would + // only ever surface the earliest one. + const seed = (S: string) => { + egressStateAppend(H.db, S, [ + { type: "I", msgid: "a", after: null, ref: "m00001", tokens: 1, typeTag: "text", bytes: "h_a" }, + { type: "I", msgid: "b", after: null, ref: "m00002", tokens: 1, typeTag: "text", bytes: "h_b" }, + ], 1_000); + }; + seed("anchor-missing"); + egressStateAppend(H.db, "anchor-missing", [ + { type: "I", msgid: "x", after: "zz", ref: "m00003", tokens: 1, typeTag: "text", bytes: "h_x" }, + ], 2_000); + assert.throws(() => egressStateView(H.db, "anchor-missing"), (e: unknown) => e instanceof EgressStateCorruptError && /anchor zz missing/.test(e.message)); + seed("endpoint-missing"); + egressStateAppend(H.db, "endpoint-missing", [{ type: "D", start: "a", end: "nope" }], 2_000); + assert.throws(() => egressStateView(H.db, "endpoint-missing"), (e: unknown) => e instanceof EgressStateCorruptError && /delete end nope missing/.test(e.message)); + seed("start-missing"); + egressStateAppend(H.db, "start-missing", [{ type: "D", start: "nope", end: "b" }], 2_000); + assert.throws(() => egressStateView(H.db, "start-missing"), (e: unknown) => e instanceof EgressStateCorruptError && /delete start nope missing/.test(e.message)); + seed("inverted"); + egressStateAppend(H.db, "inverted", [{ type: "D", start: "b", end: "a" }], 2_000); + assert.throws(() => egressStateView(H.db, "inverted"), (e: unknown) => e instanceof EgressStateCorruptError && /inverted/.test(e.message)); + seed("update-missing"); + egressStateAppend(H.db, "update-missing", [{ type: "U", msgid: "ghost", bytes: "h_g" }], 2_000); + assert.throws(() => egressStateView(H.db, "update-missing"), (e: unknown) => e instanceof EgressStateCorruptError && /update target ghost missing/.test(e.message)); + seed("dup"); + egressStateAppend(H.db, "dup", [ + { type: "I", msgid: "a", after: null, ref: "m00009", tokens: 1, typeTag: "text", bytes: "h_a2" }, + ], 2_000); + assert.throws(() => egressStateView(H.db, "dup"), (e: unknown) => e instanceof EgressStateCorruptError && /duplicate msgid a/.test(e.message)); + // and the healthy sessions in the same db still replay cleanly (no + // cross-session leakage from the corrupt rows) + seed("healthy"); + assert.deepEqual(egressStateView(H.db, "healthy").map((r) => r.msgid), ["a", "b"]); + // a GOOD op after a BAD op must not be masked: drop the bad row, replay OK + H.db.prepare("DELETE FROM egress_state WHERE session_id = 'anchor-missing' AND log_seq = 3").run(); + H.db.prepare("DELETE FROM egress_flow WHERE session_id = 'anchor-missing' AND log_seq = 3").run(); + assert.deepEqual(egressStateView(H.db, "anchor-missing").map((r) => r.msgid), ["a", "b"]); + } finally { + close(H); + } +},); + +test("compaction equivalence: view before == view after; state bounded, flow total", () => { + const H = open(); + try { + const S = "sess-4"; + // 10 tail appends, then a fold collapses 1..6 into one summary + const ops: EgressOp[] = []; + for (let i = 1; i <= 10; i++) { + ops.push({ type: "I", msgid: `m${i}`, after: null, ref: `m0000${i}`, tokens: i, typeTag: "text", bytes: `h_m${i}` }); + } + ops.push({ type: "I", msgid: "fold", after: "m6", ref: "m00011", tokens: 77, typeTag: "summary", bytes: "h_fold" }); + ops.push({ type: "D", start: "m1", end: "m6" }); + egressStateAppend(H.db, S, ops, 1_000); + const before = egressStateView(H.db, S); + assert.deepEqual(before.map((r) => r.msgid), ["fold", "m7", "m8", "m9", "m10"]); + + const stateRowsBefore = (H.db.prepare("SELECT COUNT(*) AS n FROM egress_state WHERE session_id = ?").get(S) as { n: number }).n; + const flowRowsBefore = (H.db.prepare("SELECT COUNT(*) AS n FROM egress_flow WHERE session_id = ?").get(S) as { n: number }).n; + assert.equal(stateRowsBefore, 12); + assert.equal(flowRowsBefore, 12); + + egressStateCompact(H.db, S, before, 2_000); + const after = egressStateView(H.db, S); + assert.deepEqual(after, before); + // state: base + nothing below (the 12 ops are physically gone) + const stateRows = (H.db.prepare("SELECT COUNT(*) AS n FROM egress_state WHERE session_id = ?").get(S) as { n: number }).n; + assert.equal(stateRows, 1); + // flow: untouched — total history + const flowRows = (H.db.prepare("SELECT COUNT(*) AS n FROM egress_flow WHERE session_id = ?").get(S) as { n: number }).n; + assert.equal(flowRows, 13); + // ops after compaction still replay on top of the base + egressStateAppend(H.db, S, [ + { type: "I", msgid: "m11", after: null, ref: "m00012", tokens: 11, typeTag: "text", bytes: "h_m11" }, + ], 3_000); + const grown = egressStateView(H.db, S); + assert.deepEqual(grown.map((r) => r.msgid), ["fold", "m7", "m8", "m9", "m10", "m11"]); + } finally { + close(H); + } +}); + +test("any-log_seq rollback: compacted-away history stays reconstructible from egress_flow", () => { + const H = open(); + try { + const S = "sess-5"; + // snapshot the view at every seq 1..5, then compact at 5 + const snapshots = new Map(); + for (let i = 1; i <= 5; i++) { + egressStateAppend(H.db, S, [ + { type: "I", msgid: `m${i}`, after: null, ref: `m0000${i}`, tokens: i, typeTag: "text", bytes: `h_m${i}` }, + ], 1_000 + i); + snapshots.set(i, egressStateView(H.db, S)); + } + egressStateAppend(H.db, S, [{ type: "U", msgid: "m2", bytes: "h_m2-edit" }], 6_000); + snapshots.set(6, egressStateView(H.db, S)); + egressStateCompact(H.db, S, egressStateView(H.db, S), 10_000); + // the state table is now just the base — every pre-compaction seq must + // still come back byte-identical from the flow table + for (const [seq, expected] of snapshots) { + assert.deepEqual(egressStateAt(H.db, S, seq), expected, `rollback at seq ${seq}`); + } + // seq 0 = empty view + assert.deepEqual(egressStateAt(H.db, S, 0), []); + } finally { + close(H); + } +}); + +// ---- diff engine (segment 2) ------------------------------------------ + +const V = (msgid: string, bytes = "h_" + msgid, ref = "m" + msgid, tokens = 3, type = "text"): EgressViewRow => + ({ msgid, ref, tokens, type, bytes }); + +test("diff: case 1 — tail append emits exactly one +I anchored to the last row", () => { + const prev = [V("a"), V("b")]; + const next = [V("a"), V("b"), V("c")]; + const ops = diffEgressView(prev, next); + assert.deepEqual(ops, [{ type: "I", msgid: "c", after: "b", ref: "mc", tokens: 3, typeTag: "text", bytes: "h_c" }]); + assert.deepEqual(egressReplay(prev, ops), next); +}); + +test("diff: cases 2+3 — nudge appears (+I) and vanishes (-D) without touching the prefix", () => { + const base = [V("a"), V("b")]; + const nudged = [V("a"), V("b"), V("nudge", "h_n", "mn", 1, "nudge")]; + assert.deepEqual(diffEgressView(base, nudged), [ + { type: "I", msgid: "nudge", after: "b", ref: "mn", tokens: 1, typeTag: "nudge", bytes: "h_n" }, + ]); + assert.deepEqual(egressReplay(base, diffEgressView(base, nudged)), nudged); + assert.deepEqual(diffEgressView(nudged, base), [{ type: "D", start: "nudge", end: "nudge" }]); + assert.deepEqual(egressReplay(nudged, diffEgressView(nudged, base)), base); +}); + +test("diff: case 4 — fold replaces a range with one summary row after the surviving anchor", () => { + const prev = [V("a"), V("b"), V("c"), V("d"), V("e"), V("f")]; + const folded = [V("a"), V("sum", "h_sum", "ms", 42, "acp_summary"), V("f")]; + assert.deepEqual(diffEgressView(prev, folded), [ + { type: "D", start: "b", end: "e" }, + { type: "I", msgid: "sum", after: "a", ref: "ms", tokens: 42, typeTag: "acp_summary", bytes: "h_sum" }, + ]); + assert.deepEqual(egressReplay(prev, diffEgressView(prev, folded)), folded); +}); + +test("diff: case U — content edit of an existing row is a +U, position and triple untouched", () => { + const prev = [V("a", "h_a1"), V("b")]; + const next = [V("a", "h_a2"), V("b")]; + assert.deepEqual(diffEgressView(prev, next), [{ type: "U", msgid: "a", bytes: "h_a2" }]); + const replayed = egressReplay(prev, diffEgressView(prev, next)); + assert.deepEqual(replayed, next); + assert.equal(replayed[0].ref, "ma", "+U must not touch the ref"); + assert.equal(replayed[0].tokens, 3, "+U must not touch tokens"); +}); + +test("diff: move = D+I (no M op), msgid survives both ops", () => { + const prev = [V("a"), V("b")]; + const next = [V("b"), V("a")]; + assert.deepEqual(diffEgressView(prev, next), [ + { type: "D", start: "a", end: "a" }, + { type: "I", msgid: "a", after: "b", ref: "ma", tokens: 3, typeTag: "text", bytes: "h_a" }, + ]); + assert.deepEqual(egressReplay(prev, diffEgressView(prev, next)), next); +}); + +test("diff: empty views and empty diff (both directions)", () => { + assert.deepEqual(diffEgressView([], []), []); + assert.deepEqual(diffEgressView([V("a"), V("b")], [V("a"), V("b")]), []); + const fromEmpty = diffEgressView([], [V("a"), V("b")]); + assert.deepEqual(fromEmpty, [ + { type: "I", msgid: "a", after: null, ref: "ma", tokens: 3, typeTag: "text", bytes: "h_a" }, + { type: "I", msgid: "b", after: "a", ref: "mb", tokens: 3, typeTag: "text", bytes: "h_b" }, + ]); + const toEmpty = diffEgressView([V("a"), V("b")], []); + assert.deepEqual(toEmpty, [{ type: "D", start: "a", end: "b" }]); +}); + +test("diff property: 300 random sparse mutations — replay(prev, diff) ≡ next, ops apply to the DB state too", () => { + const H = open(); + try { + let view: EgressViewRow[] = []; + const rand = (n: number) => Math.floor(Math.random() * n); + for (let iter = 0; iter < 300; iter++) { + const op = rand(5); + const prev = view; + const next = [...view]; + if (op === 0 && next.length < 40) { + next.push(V(`m${Math.random().toString(36).slice(2, 8)}`)); + } else if (op === 1 && next.length > 1) { + next.splice(next.length - 1 - rand(2), 1); + } else if (op === 2 && next.length > 2) { + // fold: replace a run of 2..4 with one summary + const at = rand(next.length - 2); + const len = 2 + rand(3); + next.splice(at, len, V(`sum${iter}`, `h_sum${iter}`, `ms${iter}`, 40, "acp_summary")); + } else if (op === 3 && next.length > 0) { + const i = rand(next.length); + next[i] = { ...next[i], bytes: "edited_" + i + "_" + iter }; + } else if (next.length > 1) { + // move a row to a new position + const i = rand(next.length); + const j = rand(next.length); + next.splice(j, 0, ...next.splice(i, 1)); + } + const ops = diffEgressView(prev, next); + assert.deepEqual(egressReplay(prev, ops), next, `replay identity at iter ${iter}`); + view = next; + } + // the final view also round-trips through the DB: rebuild from ops + const S = "prop-db"; + const rebuild: EgressOp[] = []; + let cur: EgressViewRow[] = []; + for (let iter = 0; iter < 300; iter++) { + // re-derive is expensive; instead verify the FINAL view through + // a fresh log built from the final diff from empty + void iter; + } + const fromEmpty = diffEgressView([], view); + egressStateAppend(H.db, S, fromEmpty, 1_000); + assert.deepEqual(egressStateView(H.db, S), view, "DB log built from diff(empty, final) replays to the final view"); + } finally { + close(H); + } +}); diff --git a/tests/fastpath.test.ts b/tests/fastpath.test.ts index 1ff6ca5348..1627820387 100644 --- a/tests/fastpath.test.ts +++ b/tests/fastpath.test.ts @@ -7,7 +7,7 @@ import assert from "node:assert/strict"; import test from "node:test"; import http from "node:http"; import { openaiToCore, coreToOpenai } from "acp-kernel/wire"; -import { evaluateWatermarkFastpath, finalizeWatermarkTurn, stampOnlyWatermarkTurn, clearWatermarkSeed, watermarkDebugState, PF_COUNT_HEADER, PF_SEQ_HEADER, PF_TAIL_IDS_HEADER, PF_SLOT_UNKNOWN } from "../src/server/fastpath.ts"; +import { evaluateWatermarkFastpath, evaluateDeltaResend, finalizeWatermarkTurn, stampOnlyWatermarkTurn, clearWatermarkSeed, watermarkDebugState, PF_COUNT_HEADER, PF_SEQ_HEADER, PF_TAIL_IDS_HEADER, PF_DELTA_HEADER, PF_SLOT_UNKNOWN } from "../src/server/fastpath.ts"; import { stripRawStage } from "../src/marker/engine.ts"; import { INCREMENTAL_TURN_MARKER_HEADER } from "../src/incremental-lane.ts"; import { _setStoreForTest, _resetStoreForTest, type SessionStore } from "../src/persist.ts"; @@ -478,3 +478,201 @@ test("clearWatermarkSeed forces the next turn back through verification", () => const { out } = runTurn(session, { [PF_COUNT_HEADER]: "3" }, b2); assert.equal(out.decision, "unseeded"); }); + +// #7064 2×2 perf regression (2026-11-20, 100k-message sessions): in a +// pinned session prepare-openai rebinds originalMessages to s0.pinned +// BEFORE finalizeWatermarkTurn, so the prefix stash carries HOST-MSGID ids +// (host/ledger namespace), while a fresh turn's openaiToCore mints +// content-hash ids. The head check must not treat the id field as drift +// evidence or the fastpath misses (head-prefix-drift) on EVERY steady +// turn — the exact traffic the watermark was built for. runTurnPinned +// simulates that reseed: a full host-id list pins, and the reseed takes +// the pinned (host-id) cores. +const runTurnPinned = ( + session: Session, + reqHeaders: Record, + bodyText: string, + hostIds?: string[], +): { out: ReturnType; headers: Record } => { + const model = JSON.parse(bodyText).model as string; + const out = evaluateWatermarkFastpath(bodyText, headersOf(reqHeaders), session, { pluginMode: true, titleGen: false, model }, log); + const oracle = openaiToCore(JSON.parse(bodyText)); + const stripCounts = stripRawStage(JSON.parse(bodyText), "openai"); + const base = out.hit !== null ? out.hit.view : oracle.msgs; + const legacyCores = hostIds !== undefined && hostIds.length === base.length + ? base.map((m, i) => ({ ...m, id: hostIds[i]! })) + : base; + const headers = finalizeWatermarkTurn(session, out, { + model, + legacyCores, + rawSystemText: out.hit !== null ? out.hit.systemText : oracle.systemText, + turnStripCounts: stripCounts, + }); + return { out, headers }; +}; + +test("pinned (2x2) steady state still HITS — host-id stash must not break the head check", () => { + _resetStoreForTest(); + makeStubStore(); + const session = freshSession("pf-pinned"); + const ids = (n: number): string[] => Array.from({ length: n }, (_, i) => `host-id-${i}`); // n = NON-system core count + // t1: SEED turn already carries the full host-id list → stage0 pins → + // prepare-openai reseeds the stash from the PINNED cores (host-id + // namespace). This is the field sequence (first 100k turn sends the + // full biliMsgIds list); pre-fix the stash carried host ids from the + // very first stamp and every later turn missed head-prefix-drift. + const b1 = bodyOf([SYS("s"), USR("a"), USR("b"), ASST("c")]); + const t1 = runTurnPinned(session, {}, b1, ids(3)); + assert.equal(t1.out.decision, "no-stamp", LOGS.join("\n")); + // t2: steady state — tail-ids only (anchor + new), no body list. + // Pre-fix: head-prefix-drift (fresh content-hash ids vs stashed host ids). + const b2 = bodyOf([SYS("s"), USR("a"), USR("b"), ASST("c"), USR("d")]); + const t2 = runTurnPinned(session, { [PF_COUNT_HEADER]: "4", ...TAIL([ids(4)[2]!, ids(4)[3]!]) }, b2); + assert.equal(t2.out.decision, "hit", LOGS.join("\n")); + assert.ok(t2.out.hit, "hit view expected"); + // t3: one more steady growth — the bug was PERMANENT drift, so a second + // consecutive hit proves the stash stays consistent after a hit-turn + // reseed (hit paths stash outcome.hit.view, a mixed namespace). + const b3 = bodyOf([SYS("s"), USR("a"), USR("b"), ASST("c"), USR("d"), ASST("e")]); + const t3 = runTurnPinned(session, { [PF_COUNT_HEADER]: "5", ...TAIL([ids(5)[3]!, ids(5)[4]!]) }, b3); + assert.equal(t3.out.decision, "hit", LOGS.join("\n")); + assert.ok(t3.out.hit, "second consecutive hit expected"); + assert.deepEqual( + t3.out.hit.view.map((m) => String((m as { text?: unknown }).text ?? "")), + ["a", "b", "c", "d", "e"], + "view = stashed prefix + rekeyed window, byte-identical texts", + ); + // id-excluded head check must STILL catch a real in-place content edit + // in a pinned session (both directions, #3617). + const b4 = bodyOf([SYS("s"), USR("a"), USR("b-EDITED"), ASST("c"), USR("d"), ASST("e"), USR("f")]); + const t4 = runTurnPinned(session, { [PF_COUNT_HEADER]: "6", ...TAIL([ids(6)[4]!, ids(6)[5]!]) }, b4); + assert.equal(t4.out.decision, "head-prefix-drift", LOGS.join("\n")); + assert.equal(t4.out.creditedMiss, false); + _resetStoreForTest(); +}); + +// #5403 family, delta arm (x-bili-pf-delta): the body carries ONLY the tail +// slice [T-L .. T-1] (anchor = the previous turn's last element), the full +// logical count rides x-bili-pf-count, per-element ids ride x-bili-pf-tail-ids +// (width exactly L). A satisfiable claim yields the SAME FastpathHit the +// full-body lane would — every assertion is against the legacy full-parse +// oracle over the WHOLE logical body. An unsatisfiable claim is a re-anchor +// (the 400 the client answers with a full-form resend). + +const dTail = (n: number): string[] => Array.from({ length: n }, (_, i) => `dhost-${i}`); + +const runDeltaTurn = ( + session: Session, + reqHeaders: Record, + sliceMessages: unknown[], + model = "m1", +): { verdict: { hit: import("../src/server/fastpath.ts").FastpathHit; receivedTailIds?: string[] }; headers: Record } => { + const bodyText = bodyOf(sliceMessages, model); + const verdict = evaluateDeltaResend(JSON.parse(bodyText) as Parameters[0], bodyText, headersOf(reqHeaders), session, { pluginMode: true, titleGen: false, model }, log); + assert.ok(verdict !== null && "hit" in verdict, `expected a delta hit, got ${JSON.stringify(verdict)}\n${LOGS.slice(-6).join("\n")}`); + const hit = verdict; + const headers = finalizeWatermarkTurn(session, { hit: hit.hit, measuredT: hit.hit.totalT, decision: "delta-hit", creditedMiss: false, receivedTailIds: hit.receivedTailIds }, { + model, + legacyCores: hit.hit.view, + rawSystemText: hit.hit.systemText, + turnStripCounts: hit.hit.markerCounts, + }); + return { verdict: hit, headers }; +}; + +test("delta arm: hit reconstructs EXACTLY the legacy full-parse oracle across consecutive slices", () => { + makeStubStore(); + process.env.BILI_ROLLOUT_PLUGIN_DELTA_RESEND = "1"; + const session = freshSession("pf-delta-hit"); + const full: unknown[] = [SYS("s"), USR("a"), USR("b")]; + // Seed carries NO pf headers (field: the first request of a session + // never stamps) — the first delta turn opens unverified, from then on + // every list is chained on the previous turn's stamped last id. + const t1 = runTurn(session, {}, bodyOf(full)); + assert.equal(t1.out.decision, "no-stamp"); // seed stampT=3 + // t2: logical T=5, L=5-3=2 -> body = NEW elements only [USR c, ASST d], + // tail-ids = anchor(b)+new = [dhost-2, dhost-3, dhost-4] + full.push(USR("c"), ASST("d")); + const r2 = runDeltaTurn(session, { [PF_COUNT_HEADER]: "5", [PF_DELTA_HEADER]: "1", ...TAIL(dTail(5).slice(2)) }, full.slice(3)); + assert.equal(r2.headers[PF_SEQ_HEADER], "5"); + let oracle = openaiToCore(JSON.parse(bodyOf(full))); + assert.deepEqual(r2.verdict.hit!.view.map((m) => String((m as { text?: unknown }).text ?? "")), oracle.msgs.map((m) => m.text), "texts"); + for (let i = 0; i < oracle.msgs.length; i++) { + assert.equal(r2.verdict.hit!.view[i]!.id, oracle.msgs[i]!.id, `id @${i}`); + assert.equal((r2.verdict.hit!.view[i]! as { wireIndex?: number }).wireIndex, (oracle.msgs[i]! as { wireIndex?: number }).wireIndex, `wireIndex @${i}`); + } + assert.equal(r2.verdict.hit!.systemText, oracle.systemText); + assert.equal(r2.verdict.hit!.totalT, 5); + // t3: logical T=6, L=1 -> body = [USR e], tail-ids = anchor(d)+new = [dhost-4, dhost-5] + full.push(USR("e")); + const r3 = runDeltaTurn(session, { [PF_COUNT_HEADER]: "6", [PF_DELTA_HEADER]: "1", ...TAIL(dTail(6).slice(4)) }, full.slice(5)); + assert.equal(r3.headers[PF_SEQ_HEADER], "6"); + oracle = openaiToCore(JSON.parse(bodyOf(full))); + assert.deepEqual(r3.verdict.hit!.view.map((m) => String((m as { text?: unknown }).text ?? "")), oracle.msgs.map((m) => m.text), "texts after second slice"); + for (let i = 0; i < oracle.msgs.length; i++) { + assert.equal(r3.verdict.hit!.view[i]!.id, oracle.msgs[i]!.id, `id @${i} (t3)`); + assert.equal((r3.verdict.hit!.view[i]! as { wireIndex?: number }).wireIndex, (oracle.msgs[i]! as { wireIndex?: number }).wireIndex, `wireIndex @${i} (t3)`); + } + // t4: a delta turn must NOT disturb the full lane — a full-form turn right + // after a delta hit still hits against the delta-advanced seed. + full.push(ASST("f")); + const t4 = runTurn(session, { [PF_COUNT_HEADER]: "6", ...TAIL(dTail(7).slice(5)) }, bodyOf(full)); + assert.equal(t4.out.decision, "hit", LOGS.slice(-6).join("\n")); + assert.ok(t4.out.hit); + delete process.env.BILI_ROLLOUT_PLUGIN_DELTA_RESEND; + _resetStoreForTest(); +}); + +test("delta arm: every unsatisfiable claim re-anchors (both directions, #3617)", () => { + makeStubStore(); + process.env.BILI_ROLLOUT_PLUGIN_DELTA_RESEND = "1"; + const session = freshSession("pf-delta-reanchor"); + const full: unknown[] = [SYS("s"), USR("a"), USR("b")]; + runTurn(session, {}, bodyOf(full)); // seed stampT=3 (no pf headers — field) + full.push(USR("c"), ASST("d")); // logical T=5, L=2 -> body = [USR c, ASST d] + const deltaBody = JSON.stringify({ model: "m1", stream: false, messages: full.slice(3) }); + const delta = { [PF_DELTA_HEADER]: "1", [PF_COUNT_HEADER]: "5", ...TAIL(dTail(5).slice(2)) } as Record; + const ev = (h: Record, bodyText = deltaBody) => + evaluateDeltaResend(JSON.parse(bodyText) as Parameters[0], bodyText, headersOf(h), session, { pluginMode: true, titleGen: false, model: JSON.parse(bodyText).model as string }, log); + + // negative: no header = not a delta request (full lane owns it) + assert.equal(ev({}), null); + // bad header value + assert.deepEqual(ev({ [PF_DELTA_HEADER]: "0", [PF_COUNT_HEADER]: "5" }), { reanchor: "delta-bad-value" }); + // wrong model for the seed + assert.deepEqual(ev(delta, JSON.stringify({ model: "OTHER", stream: false, messages: full.slice(3) })), { reanchor: "model" }); + // missing count + assert.deepEqual(ev({ [PF_DELTA_HEADER]: "1" }), { reanchor: "no-count" }); + // behind the stamp (client desynced backwards: L = T - stampT <= 0) + assert.deepEqual(ev({ ...delta, [PF_COUNT_HEADER]: "3" }), { reanchor: "behind" }); + // window above the bound + assert.deepEqual(ev({ ...delta, [PF_COUNT_HEADER]: String(3 + 2048 + 1) }), { reanchor: "window-too-large" }); + // body length does not match L + assert.deepEqual(ev(delta, JSON.stringify({ model: "m1", stream: false, messages: [USR("c")] })), { reanchor: "body-length" }); + // messages not an array + assert.deepEqual(ev(delta, JSON.stringify({ model: "m1", stream: false, messages: "oops" })), { reanchor: "body-length" }); + // tail-ids absent / wrong width / garbage + assert.deepEqual(ev({ [PF_DELTA_HEADER]: "1", [PF_COUNT_HEADER]: "5" }), { reanchor: "tail-ids-shape" }); + assert.deepEqual(ev({ ...delta, ...TAIL(["x", "y"]) }), { reanchor: "tail-ids-shape" }); + assert.deepEqual(ev({ ...delta, [PF_TAIL_IDS_HEADER]: "a,b%2Cc%%" }), { reanchor: "tail-ids-shape" }); + // a system element riding the new slice (pre-chain: stamp still 3) + assert.deepEqual(ev(delta, JSON.stringify({ model: "m1", stream: false, messages: [SYS("intruder"), ASST("d")] })), { reanchor: "window-system-head" }); + // anchor continuity: after a delta hit stamps a list, the NEXT list must + // open with its last id — wrong head re-anchors (both directions #3617) + const rChain = runDeltaTurn(session, { ...delta }, full.slice(3)); + void rChain; // stamps lastTailIds = dhost-2..4, next anchor must be dhost-4 + full.push(USR("e")); // T=6, L=1 + const delta2 = { [PF_DELTA_HEADER]: "1", [PF_COUNT_HEADER]: "6", ...TAIL(dTail(6).slice(4)) } as Record; + const ev2 = (h: Record) => + evaluateDeltaResend({ model: "m1", stream: false, messages: [USR("e")] } as Parameters[0], JSON.stringify({ model: "m1", stream: false, messages: [USR("e")] }), headersOf(h), session, { pluginMode: true, titleGen: false, model: "m1" }, log); + assert.deepEqual(ev2({ ...delta2, ...TAIL(["WRONG", "dhost-5"]) }), { reanchor: "tail-ids-prev" }); + assert.ok("hit" in (ev2(delta2) as object), "chained head dhost-4 must pass"); + // unseeded session: a fresh id can never delta + const cold = freshSession("pf-delta-cold"); + assert.deepEqual(evaluateDeltaResend(JSON.parse(deltaBody) as Parameters[0], deltaBody, headersOf(delta), cold, { pluginMode: true, titleGen: false, model: "m1" }, log), { reanchor: "unseeded" }); + // rollout off = the rollback kill switch (LAST: it mutes every gate below it) + process.env.BILI_ROLLOUT_PLUGIN_DELTA_RESEND = "0"; + assert.deepEqual(ev(delta), { reanchor: "rollout-off" }); + delete process.env.BILI_ROLLOUT_PLUGIN_DELTA_RESEND; + _resetStoreForTest(); +}); diff --git a/tests/fork-ccr-adoption.test.ts b/tests/fork-ccr-adoption.test.ts index 5b5bedde45..22e77d7b9f 100644 --- a/tests/fork-ccr-adoption.test.ts +++ b/tests/fork-ccr-adoption.test.ts @@ -38,6 +38,10 @@ import { maybeAdoptForkBlocks } from "../src/fork-adoption.ts"; import { rmrf } from "./tmp-rm.ts"; process.env.BILI_PERSIST = "0"; +// #6480 flipped the absent-default to pure-sqlite (JSON envelope writer +// off) — this suite reads the companion session ENVELOPE off disk, so pin +// the dual-write writer explicitly. +process.env.BILI_STORAGE_JSON_WRITER = "on"; const LIMIT = 200_000; const META = { protocol: "openai" as const, upstreamOrigin: "http://upstream.example/v1" }; diff --git a/tests/issue4666-migration-dedup.test.ts b/tests/issue4666-migration-dedup.test.ts index 4d0f0e51aa..55b00cdd66 100644 --- a/tests/issue4666-migration-dedup.test.ts +++ b/tests/issue4666-migration-dedup.test.ts @@ -155,7 +155,7 @@ test("#4666 sqlite: v8 migration collapses wireIndex-split duplicates and refund check = openSqliteDatabase(path.join(dir, "index.db")).db; const ver = check.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "stamped to v11 (v10 ts column + v11 FK child indexes, #5539)"); + assert.equal(ver.user_version, 12, "stamped to v12 (v10 ts column + v11 FK child indexes, #5539 + v12 snapshot-table foundation, #5874)"); // Exactly the lowest-seq arrival of each (msg_id, identity) group. const survivors = check diff --git a/tests/issue4677-v8-migration-streaming.test.ts b/tests/issue4677-v8-migration-streaming.test.ts index b9fa507ba7..5290291720 100644 --- a/tests/issue4677-v8-migration-streaming.test.ts +++ b/tests/issue4677-v8-migration-streaming.test.ts @@ -99,7 +99,7 @@ test("#4677 backfill sets identities and drops wireIndex-split duplicates (lowes const { from, to } = migrateSchema(db); assert.equal(from, 7); - assert.equal(to, 11); + assert.equal(to, 12); const identity = identityOfStoredContent(storedMessage("stripped", "hello")); assert.equal(identityOf(db, "s1", 1), identity); // the later duplicate row is gone and its content CAS ref handed back @@ -109,7 +109,7 @@ test("#4677 backfill sets identities and drops wireIndex-split duplicates (lowes ); assert.equal(refcountOf(db, a2), 0); assert.equal(refcountOf(db, a1), 1); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 11); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); db.close(); }); @@ -129,7 +129,7 @@ test("#4677 crash resume: sessions that already carry identities are skipped, no assert.equal(identityOf(db, "s-done", 1), "stale-planted"); assert.equal(identityOf(db, "s-pending", 1), identityOfStoredContent(storedMessage("pending", "still fresh"))); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 11); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); db.close(); }); @@ -150,7 +150,7 @@ test("#4677 rows without content bytes terminate with the ABSENT marker instead assert.equal(identity, "absent:"); // second run is a no-op — the marker is not identity = '', so the // session is not selected again - assert.equal(migrateSchema(db).from, 11); + assert.equal(migrateSchema(db).from, 12); assert.equal(identityOf(db, "s-ghost", 1), "absent:"); db.close(); }); @@ -193,7 +193,7 @@ test("#4677 empty store migrates to the schema head untouched", () => { downgradeToV7(db); const { from, to } = migrateSchema(db); assert.equal(from, 7); - assert.equal(to, 11); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 11); + assert.equal(to, 12); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 12); db.close(); }); diff --git a/tests/issue4739-storage-read-snapshot.test.ts b/tests/issue4739-storage-read-snapshot.test.ts index 25edb0149a..dc090392a5 100644 --- a/tests/issue4739-storage-read-snapshot.test.ts +++ b/tests/issue4739-storage-read-snapshot.test.ts @@ -68,7 +68,9 @@ for (const engine of ["node:sqlite", "better-sqlite3"] as const) { db.prepare("UPDATE sessions SET stats_json = ? WHERE session_id = ?").run('{"requests":"2"}', "snapshot"); assert.equal(store.getLoadHeader("snapshot")!.requests, 2, "old numeric-string counter matches full-load arbitration"); assert.equal(store.getMetadataJson("snapshot"), '{"requests":1}'); - assert.deepEqual(JSON.parse(store.getOrphansJson("snapshot")!), JSON.parse(fixture().orphansJson!)); + // #6785 step 7: orphans_json migrates into orphan_blocks on ingest; the + // lazy read path is the union view (table rows + any legacy column keys). + assert.deepEqual(store.getOrphanViews("snapshot"), JSON.parse(fixture().orphansJson!)); const contents = store.getBlockContents("snapshot", { limit: 2, offset: 1 }); assert.deepEqual(contents.map((row) => row.blockId), ["b1", "b2"]); assert.deepEqual(JSON.parse(contents[0]!.summary), { text: "full1", count: 1 }); diff --git a/tests/issue4796-gc-batched-sweep.test.ts b/tests/issue4796-gc-batched-sweep.test.ts index ad6aa53e8e..cf8e0ee7fe 100644 --- a/tests/issue4796-gc-batched-sweep.test.ts +++ b/tests/issue4796-gc-batched-sweep.test.ts @@ -84,7 +84,7 @@ test("#4796 v9: fresh install carries idx_content_refcount and candidate selecti seedContentRows(rawDb(store), 20_000, 1, "live"); seedContentRows(rawDb(store), 50, 0, "orphan"); const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_content_refcount'") .get(); @@ -118,7 +118,7 @@ test("#4796 v9: reopening a v8-era db gains the index", () => { const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_content_refcount'") .get(); @@ -243,7 +243,7 @@ test("#4796 e2e: v8 identity backfill releases wireIndex twins -> gc collects th try { const db2 = rawDb(reopened); const ver = db2.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); const twinLeft = db2 .prepare("SELECT COUNT(*) AS n FROM messages WHERE session_id = ? AND seq = ?") .get(sessId, twinSeq) as { n: number }; diff --git a/tests/issue5539-fk-content-indexes.test.ts b/tests/issue5539-fk-content-indexes.test.ts index b098db9008..73bfc5f005 100644 --- a/tests/issue5539-fk-content-indexes.test.ts +++ b/tests/issue5539-fk-content-indexes.test.ts @@ -89,7 +89,7 @@ test("#5539 v11: fresh install carries all six FK-child indexes and the DELETE p const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11); + assert.equal(ver.user_version, 12); for (const name of FK_CHILD_INDEXES) { const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?") @@ -129,7 +129,7 @@ test("#5539 v11: a v10-era library (no FK-child indexes) gains them on reopen", const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "v10 -> v11 migration ran"); + assert.equal(ver.user_version, 12, "v10 -> v12 migration ran"); for (const name of FK_CHILD_INDEXES) { const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?") diff --git a/tests/issue5874-2x2-p3.test.ts b/tests/issue5874-2x2-p3.test.ts new file mode 100644 index 0000000000..cdf6aad3bf --- /dev/null +++ b/tests/issue5874-2x2-p3.test.ts @@ -0,0 +1,1146 @@ +/** + * #5874 2×2 — stage-0 identity enrollment (M1/M2 inbound) acceptance items + * (brief comment 19544722, tracking #6181), pure-incremental v2 shape: + * + * M1 full-inbound + snapshot egress (the #5901 behavior) + * M2 incremental-inbound + snapshot (matchAppend + seqs carrier) + * + * The OUTBOUND axis (snapshot/delta egress) is a separate follow-up — the + * stored plane it built on is retired in the v2 redesign; what remains + * here is the INBOUND identity engine (l1_world + stage-0 six verdicts + + * the R3 gate) with its paired ± pins. + * + * Unit scope: + * W* the world verdicts — echo/extend/shrink/reanchor/issue/resync ± pairs + * (acceptance 5, #7064 D1: NO verdict diverges), matchAppendWindow. + * A* store seam — M2 seqs-carrier gating (±), #7 reanchor one-round + * convergence (acceptance 7), M4 codec-level pin. + * R* red gates — R3 identity fault (400 + zero side effects + breaker), + * reanchor one cheap UPDATE (no reject). + * S* the stage-0 store seam — M1 seed → M2 append → M1 echo round trip + * through SQLite. + * B* HTTP e2e — R3 400 funnel, pure-continuation passthrough. + */ +import test from "node:test"; +import assert from "node:assert/strict"; +import fs, { mkdtempSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import type { BiliMessage } from "acp-kernel/wire"; +import { openaiToCore } from "acp-kernel/wire"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { + matchAppendWindow, + matchWindow, + windowOf, + WORLD_HEAD_K, + WORLD_TAIL_K, +} from "../src/snapshot/identity.ts"; +import type { WorldWindow } from "../src/snapshot/types.ts"; +import { + FoundationRejectError, + R3FaultTracker, + checkR3Identity, + r3FaultTracker, + stage0Decide, +} from "../src/snapshot/plane.ts"; +import { runStage0 } from "../src/snapshot/stage0.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { snapL1Lookup, snapL1WindowRead, snapL1Record, type DbLike } from "../src/storage/snap.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-5874-p3-")); +const configPath = path.join(hermetic, "config.json"); +fs.writeFileSync(configPath, "{}"); + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +type StoredWorld = { + ledgerId: string; + window: WorldWindow; + issuedAt: number; + armedAt?: number; +}; + +interface WorldMatchInputLocal { + stored: WorldWindow | null; + issuedAt: number; + incoming: string[]; + sessRaw: string; + ledgerId: string; + now: number; + resync?: boolean; +} + +const now0 = 1_700_000_000_000; + +/** A stored world cursor (windows) for ids 0..n-1 (1-based ids m1.., 1-based + * pos, the issue/resync shape). */ +function world(n: number): StoredWorld { + const ids = Array.from({ length: n }, (_, i) => `m${i + 1}`); + return { + ledgerId: "sid-t", + window: windowOf(ids, n), + issuedAt: now0, + }; +} + +function matchInput(stored: StoredWorld | null, incoming: string[], extra: Partial = {}): WorldMatchInputLocal { + return { + stored: stored === null ? null : stored.window, + issuedAt: stored === null ? 0 : stored.issuedAt, + incoming, + sessRaw: "sess-t", + ledgerId: "sid-t", + now: now0 + 1, + ...extra, + }; +} + +function openPlane(p: string): UnifiedStore { + return UnifiedStore.open(p); +} + +/** A plain (untagged) core message — the lane's converted form. */ +function core(i: number, id?: string): BiliMessage { + return { + id: id ?? `m${i}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + contentType: "text", + text: `message ${i}`, + } as BiliMessage; +} + +function plainSession(id: string, n: number): LegacySessionLike { + const msgs = Array.from({ length: n }, (_, i) => ({ + id: `m${i + 1}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + contentType: "text" as const, + text: `message ${i + 1}`, + })); + const byRaw: Record = {}; + const byRef: Record = {}; + msgs.forEach((m, i) => { + byRaw[m.id] = `m${String(i + 1).padStart(5, "0")}`; + byRef[`m${String(i + 1).padStart(5, "0")}`] = m.id; + }); + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw, byRef } }, + blockContents: new Map(), + lastMessages: msgs, + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: undefined, + }; +} + +// --------------------------------------------------------------------------- +// W — the world verdicts (acceptance 5: each verdict with paired +/- tests) +// --------------------------------------------------------------------------- + +test("W1 echo: the same window state → echo, cursor untouched, B1 origin frozen", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["m1", "m2", "m3", "m4"])); + assert.equal(d.mode, "echo"); + assert.equal(d.window, stored.window, "the stored cursor is returned VERBATIM (B1: origin ids never re-minted)"); + assert.deepEqual(d.newIdPos, []); +}); + +test("W2 echo +/- : one id changed → NOT echo (a middle rewrite is a reanchor, D1: never a reject)", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["m1", "m2", "X", "m4"])); + assert.notEqual(d.mode, "echo"); + assert.equal(d.mode, "reanchor"); +}); + +test("W3 extend: a tail extension → extend, fresh pos at maxPos+1", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["m1", "m2", "m3", "m4", "m5"])); + assert.equal(d.mode, "extend"); + assert.deepEqual(d.newIdPos, [{ id: "m5", pos: 5 }]); +}); + +test("W4 extend +/- : the extension with a GAP in the middle → reanchor (D1: no scatter reject)", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["m1", "m3", "m5"])); + assert.equal(d.mode, "reanchor"); +}); + +test("W5 issue: no stored cursor → issue, fresh pos 0.., issued now", () => { + const d = matchWindow(matchInput(null, ["a", "b"])); + assert.equal(d.mode, "issue"); + assert.deepEqual(d.newIdPos, [{ id: "a", pos: 1 }, { id: "b", pos: 2 }]); + assert.equal(d.issuedAt, now0 + 1); +}); + +test("W6 issue +/- : resync with a stored cursor → re-issue pos 1.. from the client array", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["X", "Y", "Z"], { resync: true })); + assert.equal(d.mode, "resync"); + assert.deepEqual(d.newIdPos, [{ id: "X", pos: 1 }, { id: "Y", pos: 2 }, { id: "Z", pos: 3 }]); +}); + +test("W7 head swap → reanchor (D1: NEVER diverge — the cursor re-issues, nothing is rejected)", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["OTHER", "m1", "m2", "m3"])); + assert.equal(d.mode, "reanchor"); +}); + +test("W8 survivor order broken → reanchor (D1: never diverge)", () => { + const stored = world(4); + const d = matchWindow(matchInput(stored, ["m1", "m3", "m2", "m4"])); + assert.equal(d.mode, "reanchor"); +}); + +test("W9 fold-explained head drop → shrink (legal, no new ids)", () => { + const stored = world(4); + // m1 vanished (covered by a live fold); the rest are intact — the whole + // incoming is a suffix of the stored tail. + const d = matchWindow(matchInput(stored, ["m2", "m3", "m4"])); + assert.equal(d.mode, "shrink"); + assert.deepEqual(d.newIdPos, []); + assert.deepEqual(d.goneIds, ["m1"], "the windowed gone-set sees the dropped head anchor"); +}); + +test("W10 matchAppendWindow: pure M2 addition — head unchanged, tail extended", () => { + const stored = world(4); + const d = matchAppendWindow(stored.window, ["m5", "m6"]); + assert.equal(d.mode, "extend"); + assert.deepEqual(d.newIdPos, [{ id: "m5", pos: 5 }, { id: "m6", pos: 6 }]); + assert.deepEqual( + d.window.tailIds, + ["m1", "m2", "m3", "m4", "m5", "m6"], + "the stored order is preserved, new ids appended", + ); + assert.equal(d.window.count, 6); +}); + +test("W11 matchAppendWindow +/- : the next M1 full resend echoes (the round-trip)", () => { + const stored = world(4); + const app = matchAppendWindow(stored.window, ["m5"]); + const re = matchWindow({ ...matchInput(stored, ["m1", "m2", "m3", "m4", "m5"]), stored: app.window }); + assert.equal(re.mode, "echo"); + assert.deepEqual(re.newIdPos, []); +}); + +test("W10b matchAppendWindow: >WORLD_TAIL_K stored — the head stays the TRUE head (PR#6785 review)", () => { + // count 600 > WORLD_TAIL_K(512): stored.tailIds holds only m89..m600, so + // `[...tailIds, ...new].slice(0, WORLD_HEAD_K)` is a MIDDLE slice. The + // pre-fix code stored it as headIds, breaking the echo gate + // (matchWindow's sameArr(s.headIds, incoming[0..K-1])) for every later + // full-resend turn of a >512-element session: perpetual re-anchor + + // window upsert (the steady-state no-write contract). + const n = WORLD_TAIL_K + 88; // 600 + const stored = world(n); + assert.equal(stored.window.tailIds.length, WORLD_TAIL_K); + assert.equal(stored.window.tailIds[0], `m${n - WORLD_TAIL_K + 1}`); // m89 + const d = matchAppendWindow(stored.window, [`m${n + 1}`, `m${n + 2}`]); + assert.equal(d.mode, "extend"); + assert.equal(d.window.count, n + 2); + assert.equal(d.window.headIds.length, WORLD_HEAD_K); + assert.equal(d.window.headIds[0], "m1", "head is UNCHANGED by an append"); + assert.deepEqual(d.window.headIds, stored.window.headIds); + // and the head check the echo gate performs really passes now: + const full = Array.from({ length: n + 2 }, (_, i) => `m${i + 1}`); + assert.deepEqual(d.window.headIds, full.slice(0, WORLD_HEAD_K)); +}); + +// --------------------------------------------------------------------------- +// A — the stage-0 store seam (M2 gating, resync convergence, codec pin) +// --------------------------------------------------------------------------- + +function storeWithRows(n: number, sid: string): UnifiedStore { + const store = openPlane(path.join(hermetic, `a-${sid}-${n}`, "index.db")); + store.ingestSession(planIngest(plainSession(sid, n))); + return store; +} + +test("A1b #2 +/- : the M2 branch REQUIRES the seqs carrier (no seqs → matchWindow path)", () => { + const sid = "a1b"; + const store = storeWithRows(10, sid); + store.snapStage0(sid, Array.from({ length: 10 }, (_, i) => `m${i + 1}`)); + // All-unknown WITHOUT seqs: a full (re-minted) array → matchWindow → + // no anchor overlap → reanchor (D1: the cursor re-issues, never rejects). + const d = store.snapStage0(sid, ["m11"]); + assert.equal(d.decision.mode, "reanchor"); +}); + +test("A1c #2 +/- : a non-contiguous seqs run falls back to matchWindow", () => { + const sid = "a1c"; + const store = storeWithRows(10, sid); + store.snapStage0(sid, Array.from({ length: 10 }, (_, i) => `m${i + 1}`)); + const d = store.snapStage0(sid, ["m11"], { seqs: [15] }); + assert.notEqual(d.decision.mode, "extend", "seqs[0]=15 != frontier+1=11 → not the M2 append"); +}); + +test("A4 #7 reanchor: a structural change re-issues the cursor, one-round convergence (D1: no reject)", () => { + const sid = "a4"; + const store = storeWithRows(10, sid); + const ten = Array.from({ length: 10 }, (_, i) => `m${i + 1}`); + const seed = store.snapStage0(sid, ten); + assert.equal(seed.decision.mode, "issue"); + const db = (store as unknown as { db: DbLike }).db; + const sessRaw = sid; // plainSession id == conversation id + const before = snapL1WindowRead(db, sessRaw); + assert.ok(before && before.window, "the seed cursor is persisted"); + + const d = store.snapStage0(sid, ["OTHER", "m1", "m2"]); + assert.equal(d.decision.mode, "reanchor", "D1: a head swap re-anchors — it never rejects"); + assert.deepEqual(d.decision.window.tailIds, ["OTHER", "m1", "m2"], "the new cursor follows the incoming"); + + // The next turn with the same array converges to echo (one-round). + const re = store.snapStage0(sid, ["OTHER", "m1", "m2"]); + assert.equal(re.decision.mode, "echo"); + + // An explicit re-baseline still re-issues from the client array. + const rs = store.snapStage0(sid, ["OTHER", "m1", "m2"], { resync: true }); + assert.equal(rs.decision.mode, "resync"); + assert.deepEqual(rs.decision.window.tailIds, ["OTHER", "m1", "m2"]); +}); + +test("A4b #7 +/- : a plain issue is the NORMAL seed (not a resync)", () => { + const sid = "a4b"; + const store = openPlane(path.join(hermetic, "a4b", "index.db")); + const d = store.snapStage0(sid, ["a", "b", "c"]); + assert.equal(d.decision.mode, "issue"); + assert.notEqual(d.decision.mode, "resync", "seed must not loop the reset"); +}); + +test("A1d M4: the codec-level pin — parsing the new element ALONE (M2 wire) yields the same core as parsing it inside the full array (M1 wire), wireIndex aside", () => { + const contents = Array.from({ length: 11 }, (_, i) => `codec content ${i + 1}`); + const fullBody = { model: "gpt-m4", messages: contents.map((c) => ({ role: "user", content: c })) }; + const full = openaiToCore(fullBody as never); + assert.equal(full.msgs.length, 11); + // M2 wire: the client sends ONLY the 11th element — a fresh parse with a + // fresh ClusterCounter. The canonical id derives from (role, contentType, + // text), so the counter state cannot change the id for an element whose + // content appears once. + const inc = openaiToCore({ model: "gpt-m4", messages: [fullBody.messages[10]!] } as never); + assert.equal(inc.msgs.length, 1); + const strip = (m: BiliMessage): Record => { + const c = { ...(m as unknown as Record) }; + delete c.wireIndex; // position on the wire — legitimately differs (10 vs 0) + return c; + }; + assert.equal( + JSON.stringify(strip(full.msgs[10]!)), + JSON.stringify(strip(inc.msgs[0]!)), + "the new element's core is byte-identical between the full-array parse (M1) and the standalone parse (M2) — the M2 parse of ONLY the new span is assembly-equivalent by construction, not by luck", + ); +}); + +// --------------------------------------------------------------------------- +// R — red gates +// --------------------------------------------------------------------------- + +test("R1 R3: a declared host that fails to collect msgids → the fault (400 source)", () => { + assert.equal(checkR3Identity(true, 5, undefined), false, "declared, no msgids"); + assert.equal(checkR3Identity(true, 5, ["a", "b", "c"]), false, "declared, wrong length"); + assert.equal(checkR3Identity(true, 5, ["a", "b", "c", "d", "e"]), true, "declared, complete"); + assert.equal(checkR3Identity(false, 5, undefined), true, "non-cohort: never a fault"); + assert.equal(checkR3Identity(true, 0, undefined), true, "no elements: nothing to collect"); +}); + +test("R2 R3: FoundationRejectError is the 400 control channel (zero side effects are the caller's, the error is the signal)", () => { + const err = new FoundationRejectError("envelope fault"); + assert.ok(err instanceof Error); + assert.equal(err.status, 400); + assert.match(err.message, /R3 identity fault/); +}); + +test("R3 R3 breaker: 3 faults in the 10-minute window latch; the latch rejects further faults and outlives the window", () => { + const t = new R3FaultTracker(60_000, 3); + const t0 = 1_000_000; + assert.equal(t.recordFault("s", t0).latched, false); + assert.equal(t.recordFault("s", t0 + 1_000).latched, false); + assert.equal(t.recordFault("s", t0 + 2_000).latched, true, "the third strike latches"); + assert.equal(t.isLatched("s"), true); + // Latched sids reject immediately — even after the window would expire. + assert.equal(t.recordFault("s", t0 + 999_999).latched, true); + // A different sid is unaffected. + assert.equal(t.isLatched("other"), false); + t.clearLatched("s"); + assert.equal(t.isLatched("s"), false); +}); + +test("R4 R3 +/- : the window expires (a new window restarts the strike count)", () => { + const t = new R3FaultTracker(1000, 3); + t.recordFault("s", 0); + t.recordFault("s", 100); + t.recordFault("s", 200); // latched? strikeCap 3 → yes + assert.equal(t.isLatched("s"), true); + t.clearLatched("s"); + // Fresh window: two faults do NOT re-latch. + t.recordFault("s", 5_000); + t.recordFault("s", 6_000); + assert.equal(t.isLatched("s"), false); +}); + +test("R5 R2: reanchor = one cheap cursor re-issue (stage0Decide NEVER rejects a structural disagreement)", () => { + const dir = path.join(hermetic, "r5"); + fs.mkdirSync(dir, { recursive: true }); + const db = (openPlane(path.join(dir, "index.db")) as unknown as { db: DbLike }).db; + // The l1 row is recorded by the store facade before stage0 (the lane + // path) — do it here so the pure function has its world to match. + snapL1Record(db, "s5", "s5", "witness", now0); + // Seed the world: issue a 3-id cursor. + const seed = stage0Decide({ db, sessRaw: "s5", ledgerId: "s5", incoming: ["a", "b", "c"], maxSeq: 0, explainedGone: new Set(), now: now0 }); + assert.equal(seed.decision.mode, "issue"); + // A structural disagreement (head swap): the verdict is reanchor — the + // cursor re-issues (D1: no reject, no lane reset; middle-tamper is the + // ledger probes' job, not the cursor's). + const d = stage0Decide({ db, sessRaw: "s5", ledgerId: "s5", incoming: ["OTHER", "a", "b"], maxSeq: 0, explainedGone: new Set(), now: now0 + 1 }); + assert.equal(d.decision.mode, "reanchor"); + const l1 = snapL1Lookup(db, "s5"); + assert.ok(l1, "the seed l1 row exists"); + const env = snapL1WindowRead(db, "s5"); + assert.ok(env && env.window, "the reanchor persisted the new cursor"); + assert.deepEqual(env!.window!.tailIds, ["OTHER", "a", "b"], "the re-issued cursor follows the incoming"); +}); + +// --------------------------------------------------------------------------- +// S — the stage-0 store seam (M1 seed → M2 append round trip through SQLite) +// --------------------------------------------------------------------------- + +test("S1 the full M1→M2 round trip: seed (issue) → M1 probe → M2 append → M1 echo", () => { + const dir = path.join(hermetic, "s1"); + fs.mkdirSync(dir, { recursive: true }); + const store = openPlane(path.join(dir, "index.db")); + const sid = "s1"; + store.ingestSession(planIngest(plainSession(sid, 3))); + + // Seed: the client's first full array (3 ids). + const seed = store.snapStage0(sid, ["m1", "m2", "m3"]); + assert.equal(seed.decision.mode, "issue"); + + // M1 tail extension: full resend + one new id (1-based pos: seed is 1..3 + // → m4 lands at pos 4). + const m1 = store.snapStage0(sid, ["m1", "m2", "m3", "m4"]); + assert.equal(m1.decision.mode, "extend"); + assert.deepEqual(m1.decision.newIdPos, [{ id: "m4", pos: 4 }]); + + // M2: ONLY the new id + the seqs carrier (3 live rows seq 1..3 → + // frontier 3, the next seq is 4). + store.ingestSession(planIngest(plainSession(sid, 4))); + const m2 = store.snapStage0(sid, ["m5"], { seqs: [4] }); + assert.equal(m2.decision.mode, "extend", "the M2 append"); + assert.deepEqual(m2.decision.newIdPos, [{ id: "m5", pos: 5 }]); + assert.deepEqual(m2.decision.window.tailIds, ["m1", "m2", "m3", "m4", "m5"]); + + // The next M1 full resend must ECHO the matchAppend canon. + const echo = store.snapStage0(sid, ["m1", "m2", "m3", "m4", "m5"]); + assert.equal(echo.decision.mode, "echo", "M1 full resend echoes the M2-persisted canon"); + assert.deepEqual(echo.decision.newIdPos, []); +}); + +// --------------------------------------------------------------------------- +// M3 — the R3 latch persistence seam +// --------------------------------------------------------------------------- + +test("M3: the R3 latch persists across store instances — a latched session degrades forever, never 400s again", () => { + const p = path.join(hermetic, "m3", "index.db"); + const a = openPlane(p); + assert.equal(a.snapR3Latched("sid-m3"), false); + a.snapR3Latch("sid-m3"); + // A FRESH instance over the same file — the process restart case: the + // in-memory breaker is cold, the persisted latch must carry the state. + const b = openPlane(p); + assert.equal(b.snapR3Latched("sid-m3"), true, "the latch survives a new store instance (process restart)"); + assert.equal(b.snapR3Latched("other-sid"), false, "other sessions are unaffected"); +}); + +// #6785 review: the operator escape hatch — the latch must be VISIBLE +// (roster) and CLEARABLE (persisted), or a mislatched session is bricked to +// the legacy path with no remedy. +test("M3.1: the latch roster lists the stuck sids and the clear PERSISTS (survives a restart)", () => { + const p = path.join(hermetic, "m3-unlatch", "index.db"); + const a = openPlane(p); + a.snapR3Latch("sid-u"); + a.snapR3Latch("sid-v"); + const roster = a.snapR3LatchedList(); + assert.deepEqual(roster.map((r) => r.sessRaw).sort(), ["sid-u", "sid-v"], "both latched sids visible in the roster"); + for (const r of roster) { + assert.equal(r.ledgerId, r.sessRaw, "no conversation binding → the bound ledger id is the raw id itself"); + assert.ok(r.latchedAt > 0, "the latch carries its timestamp (the operator sees how old the stick is)"); + } + const ledger = a.snapR3LatchClear("sid-u"); + assert.equal(ledger, "sid-u", "the clear returns the bound ledger id (the in-memory breaker key)"); + assert.equal(a.snapR3Latched("sid-u"), false, "cleared in the same instance"); + assert.equal(a.snapR3Latched("sid-v"), true, "the other latch is untouched"); + // The process-restart case: the clear must have persisted. + const b = openPlane(p); + assert.equal(b.snapR3Latched("sid-u"), false, "the clear survives a new store instance"); + assert.deepEqual(b.snapR3LatchedList().map((r) => r.sessRaw), ["sid-v"]); + assert.equal(b.snapR3LatchClear("no-such-sid"), null, "no row → null (the route reports persistedRowCleared=false)"); +}); + +// #6785 review (fix): the unlatch PLACEHOLDER trap. snapR3LatchWrite can run +// BEFORE the first stage-0 round — the placeholder row carries +// ledger_id = sessRaw and witness_probe "" (legitimate rows always carry a +// hashed probe). For a conversation-bound session the raw id ≠ the ledger +// sid, so without a self-heal the placeholder would map sessRaw → a +// DIFFERENT ledger forever and 400 every turn after the operator unlatched. +test("M3.2: unlatching an R3 LATCH PLACEHOLDER self-heals — the next stage-0 rebinds to the real ledger (no permanent 400)", () => { + const p = path.join(hermetic, "m3.2", "index.db"); + const a = openPlane(p); + // Conversation binding: ledger sid "sid-cb" → raw conversation id "conv-cb". + const db = (a as unknown as { db: DbLike }).db; + db.exec("INSERT INTO conversations (namespace, conversation_id, session_id, claimed_at) VALUES ('default', 'conv-cb', 'sid-cb', 0)"); + // The latch fires on the real sid — the placeholder row is keyed by the + // resolved raw id and carries ledger_id = sessRaw (the real sid unknown). + a.snapR3Latch("sid-cb"); + let row = db.prepare("SELECT ledger_id, witness_probe FROM l1_world WHERE sess_raw = ?").get("conv-cb") as { ledger_id: string; witness_probe: string }; + assert.equal(row.ledger_id, "conv-cb", "placeholder: the ledger id IS the raw id (the real sid was unknown at latch time)"); + assert.equal(row.witness_probe, "", "placeholder: no witness probe yet (legitimate rows always carry a hash)"); + // The operator unlatches (POST /__bili/snapshot/r3-unlatch {sessRaw}). + assert.equal(a.snapR3LatchClear("conv-cb"), "conv-cb"); + // The next turn: BEFORE the fix this threw FoundationRejectError (red) + // forever. Now the placeholder is rebindable — stage-0 succeeds and the + // row anchors the REAL ledger. + const d = a.snapStage0("sid-cb", ["m1"]); + assert.ok(d, "stage-0 completed — no R3 400 after unlatch"); + row = db.prepare("SELECT ledger_id, witness_probe FROM l1_world WHERE sess_raw = ?").get("conv-cb") as { ledger_id: string; witness_probe: string }; + assert.equal(row.ledger_id, "sid-cb", "the placeholder was rebound to the real ledger sid"); + assert.notEqual(row.witness_probe, "", "the rebased row carries a real witness probe"); + // And the FOLLOWING round is a clean echo (the mapping is stable now). + assert.doesNotThrow(() => a.snapStage0("sid-cb", ["m1"])); +}); + +// #6785 review (regression guard for the self-heal): a REAL ledger row +// (hashed witness probe) under a different sid must STILL hard-400 — the +// placeholder marker (probe "") is the ONLY rebindable shape. +test("M3.2b: a REAL ledger mapping still red 400s after unlatch (the self-heal never masks a genuine R3 conflict)", () => { + const p = path.join(hermetic, "m3.2b", "index.db"); + const a = openPlane(p); + const db = (a as unknown as { db: DbLike }).db; + // Real ledger "s1" (no conversation binding → raw = ledger) issues its + // l1 row with a hashed witness probe. + assert.equal(a.snapStage0("s1", ["a", "b"]).decision.mode, "issue"); + // A conversation-bound session whose raw id collides with that ledger. + db.exec("INSERT INTO conversations (namespace, conversation_id, session_id, claimed_at) VALUES ('default', 's1', 's2', 0)"); + a.snapR3Latch("s2"); // latch write is a no-op on the ledger fields (COALESCE keeps "s1") + a.snapR3LatchClear("s1"); + assert.throws( + () => a.snapStage0("s2", ["a", "b"]), + (e: unknown) => e instanceof FoundationRejectError && /l1 ledger conflict/.test(String(e)), + "genuine conflict (real probe, different ledger) still hard-rejects", + ); +}); + +// ── HTTP e2e (real proxy + fake upstream) ─────────────────────────────────── + +import http from "node:http"; +import { once } from "node:events"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; + +function listenT(server: http.Server): Promise { + if (server.listening) return Promise.resolve(); + return once(server, "listening").then(() => undefined); +} +function closeT(server: http.Server): Promise { + return new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); +} +function sseDone(id = "resp_done"): string { + const sse = (type: string, data: unknown): string => `event: ${type}\ndata: ${JSON.stringify(data)}\n\n`; + return [ + sse("response.output_item.done", { item: { type: "message", id: "msg_1", role: "assistant", content: [{ type: "output_text", text: "ok" }] }, output_index: 0 }), + sse("response.completed", { response: { id, status: "completed", output: [], usage: { input_tokens: 100, output_tokens: 5, total_tokens: 105 } } }), + ].join(""); +} + +type UpstreamBehavior = (n: number, body: string) => { status: number; payload: string }; + +async function withProxyT( + behavior: UpstreamBehavior, + fn: (url: string, bodies: string[]) => Promise, + opts: { env?: Record } = {}, +): Promise { + const bodies: string[] = []; + let reqN = 0; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + const body = Buffer.concat(chunks).toString("utf8"); + bodies.push(body); + const { status, payload } = behavior(++reqN, body); + const streaming = payload.startsWith("event: "); + res.writeHead(status, streaming ? { "content-type": "text/event-stream", "cache-control": "no-cache" } : { "content-type": "application/json" }); + res.write(payload); + res.end(); + }); + }); + upstream.listen(0, "127.0.0.1"); + await listenT(upstream); + const upstreamPort = (upstream.address() as { port: number }).port; + + const savedEnv: Record = {}; + for (const [k, v] of Object.entries(opts.env ?? {})) { savedEnv[k] = process.env[k]; process.env[k] = v; } + try { + _setStoreForTest(new SessionStore({ enabled: false })); + const proxyOpts: ProxyOptions = { + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [`http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-e2e": { context: 400_000 } } } }, + modelContextLimit: 400_000, + kernelConfig: defaultConfig(400_000), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: true, + logFile: "off", + debug: false, + passthrough: false, + autoUpdate: false, + compat: { roles: {} }, + streamErrorShape: "protocol", + passthroughSource: null, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: false, + releaseNotesCheck: false, + mitm: { enabled: false, domains: [] }, + }; + const proxy = await startServer(proxyOpts); + await listenT(proxy); + const proxyPort = (proxy.address() as { port: number }).port; + try { + await fn(`http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`, bodies); + } finally { + await closeT(proxy); + } + } finally { + await closeT(upstream); + for (const [k, v] of Object.entries(savedEnv)) { if (v === undefined) delete process.env[k]; else process.env[k] = v; } + delete process.env.BILI_STORAGE_UNIFIED; + _setStoreForTest(new SessionStore({ enabled: false })); + } +} + +const PLUGIN_HEADERS = { "content-type": "application/json", "x-bili-plugin": "e2e-agent", "x-acp-session": "e2e-sess" } as Record; + +test("B1: a declared host that fails to collect msgids (count mismatch) → 400 + x-bili-snapshot: identity, no upstream forward", async () => { + await withProxyT( + () => ({ status: 200, payload: sseDone() }), + async (url, bodies) => { + const res = await fetch(url, { + method: "POST", + headers: PLUGIN_HEADERS, + body: JSON.stringify({ + model: "gpt-e2e", + stream: true, + input: [ + { type: "message", role: "user", content: "first" }, + { type: "message", role: "user", content: "second" }, + ], + biliMsgIds: ["b1m1"], // 1 declared vs 2 elements — the R3 fault + }), + }); + assert.equal(res.status, 400, "the identity fault is a 400 (the B1 control channel — not the generic 503)"); + assert.equal(res.headers.get("x-bili-snapshot"), "identity"); + const err = JSON.parse(await res.text()) as { error: { type: string; message: string } }; + assert.equal(err.error.type, "snapshot_identity_fault"); + assert.ok(err.error.message.includes("R3"), "the error names the fault class"); + assert.equal(bodies.length, 0, "zero side effects: the upstream was never contacted"); + }, + { env: { BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY: "1" } }, + ); +}); + +test("M3.2: the admin unlatch routes exist and are gated (501 in legacy mode, 400 on a malformed body)", async () => { + await withProxyT( + () => ({ status: 200, payload: sseDone() }), + async (url) => { + const base = url.slice(0, url.indexOf("/bili/")); + const listRes = await fetch(`${base}/__bili/snapshot/r3-latches`); + assert.equal(listRes.status, 501, "legacy mode → 501 (the knob, not the URL, is missing)"); + const badRes = await fetch(`${base}/__bili/snapshot/r3-unlatch`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({}) }); + assert.equal(badRes.status, 400, "a body without sessRaw is rejected before touching the store"); + const unlatchRes = await fetch(`${base}/__bili/snapshot/r3-unlatch`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ sessRaw: "e2e-sess" }) }); + assert.equal(unlatchRes.status, 501, "a well-formed body still 501s in legacy mode (unified store required)"); + }, + {}, + ); +}); + +test("B3: a plugin chained round with EMPTY input is a pure continuation — the raw body passes through verbatim (no dev injection, id kept)", async () => { + await withProxyT( + () => ({ status: 200, payload: sseDone() }), + async (url, bodies) => { + const reqBody = { model: "gpt-e2e", stream: true, previous_response_id: "resp_b3", input: [] }; + const res = await fetch(url, { method: "POST", headers: PLUGIN_HEADERS, body: JSON.stringify(reqBody) }); + assert.equal(res.status, 200); + await res.text(); + assert.equal(bodies.length, 1); + const fwd = JSON.parse(bodies[0]!) as Record; + assert.equal(fwd.previous_response_id, "resp_b3", "the chain id is forwarded intact"); + assert.deepEqual(fwd.input, [], "the empty input passes through (NOT rebuilt, NOT dropped)"); + assert.ok(!JSON.stringify(fwd).includes("You are bili"), "no developer-message injection on the pure continuation (the B3 blind-response fix)"); + }, + {}, + ); +}); + +// --------------------------------------------------------------------------- +// D2/D3 — the #7064 lane-level contracts (runStage0, both-wire shared) +// --------------------------------------------------------------------------- +// D2: a conversation-bound sid (host identity registered) NEVER takes the +// R3 fault path — a short/missing declared array degrades to +// content-derived ids instead of 400/latch, on miss AND hit turns. +// D3: the world cursor (tail anchor) stays LIVE across watermark-hit runs — +// a hit turn with a valid declared array refreshes the cursor. + +type DLog = { level: string, msg: string }; + +function d23Harness(t: test.TestContext) { + const dir = mkdtempSync(path.join(os.tmpdir(), "bili-5874-d23-")); + const prevEnv: Record = {}; + for (const k of ["BILI_STORAGE_UNIFIED", "BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY", "BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL", "XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) { + prevEnv[k] = process.env[k]; + process.env[k] = k.startsWith("XDG") ? dir : (k === "BILI_STORAGE_UNIFIED" ? "full" : "1"); + } + const store = UnifiedStore.open(path.join(dir, "index.db")); + // runStage0 talks to the store through the SessionStore seam — give it + // the snap delegates, delegating to the real UnifiedStore. + _setStoreForTest({ + setSessionCommitListener() { /* the seam only needs the snap delegates */ }, + snapStage0: (sid: string, ids: string[], opts?: { seqs?: number[] | null }) => store.snapStage0(sid, ids, opts), + snapConvBound: (sid: string) => store.snapConvBound(sid), + snapMsgIdLiveIds: (sid: string) => store.snapMsgIdLiveIds(sid), + snapR3Latched: (sid: string) => store.snapR3Latched(sid), + } as unknown as Parameters[0]); + t.after(() => { + for (const [k, v] of Object.entries(prevEnv)) { + if (v === undefined) delete process.env[k]; + else process.env[k] = v; + } + }); + return { dir, store }; +} + +function dMsgs(n: number): BiliMessage[] { + const arr: { role: string, content: string }[] = [{ role: "system", content: "sys" }]; + for (let i = 1; i <= n; i++) arr.push({ role: i % 2 ? "user" : "assistant", content: `c${i}` }); + return (openaiToCore({ messages: arr } as never)).msgs; +} + +test("#7064 D2: a conversation-bound sid never takes the R3 fault path", async (t) => { + const { store } = d23Harness(t); + const sid = "d2-sid"; + const db = store as unknown as { db: DbLike }; + db.db.prepare(`INSERT INTO conversations (namespace, conversation_id, session_id, claimed_at) VALUES ('default', 'd2-conv', ?, 0)`).run(sid); + const logs: DLog[] = []; + const log = (level: string, msg: string) => logs.push({ level, msg }); + try { + assert.ok(store.snapConvBound(sid), "the conversations row makes the sid conversation-bound"); + // a non-bound sid with the SAME fault 400s (control): + assert.throws(() => runStage0({ + sessionId: "d2-sid-nobound", + lane: "openai", + msgs: dMsgs(5), + incomingMsgIds: ["a", "b"], // 2 ids for 6 messages + incomingSeqs: null, + pluginMode: true, + log, + }), FoundationRejectError, "non-bound sid: R3 fault 400s"); + const r = runStage0({ + sessionId: sid, + lane: "openai", + msgs: dMsgs(5), + incomingMsgIds: ["a", "b"], // the same fault — bound: no throw + incomingSeqs: null, + pluginMode: true, + log, + }); + assert.equal(r.pinned, null, "the pin is rejected — content-derived ids"); + assert.ok(logs.some((l) => l.level === "warn" && l.msg.includes("conversation-bound sid skips the R3 fault")), `the degrade is logged: ${logs.map((l) => l.msg).join(" | ")}`); + // and a watermark-HIT turn: same degrade, no throw, no latch (D3 arm). + r3FaultTracker.clearLatched(sid); + const r2 = runStage0({ + sessionId: sid, + lane: "openai", + msgs: dMsgs(5), + incomingMsgIds: ["a"], // worse: 1 id for 6 + incomingSeqs: null, + pluginMode: true, + wpHit: { view: [] as BiliMessage[], lastSeq: 5 }, + log, + }); + assert.equal(r2.pinned, null); + assert.ok(!r3FaultTracker.isLatched(sid), "no fault was recorded for the bound sid"); + // store seam: a bound sid whose pin DID apply skips the id verdict + // entirely — mode "anchor", no reject possible (D1 floor). + const msgs = dMsgs(2); + const res = store.snapStage0(sid, msgs.map((m) => m.id), {}); + assert.ok(res !== null, "stage0 ran"); + assert.equal(res.decision.mode, "anchor"); + assert.equal(res.decision.witness, "anchor: conversation-bound, id verdict skipped"); + assert.equal(snapL1WindowRead(db.db, sid)?.window ?? null, null, "anchor persists nothing — the ledger rows are the identity"); + } finally { + r3FaultTracker.clearLatched(sid); + r3FaultTracker.clearLatched("d2-sid-nobound"); + } +}); + +test("#6785-review D2: an idempotent re-render of a conv-bound sid is an echo (no cursor write)", async (t) => { + const { store } = d23Harness(t); + const sid = "d2-echo-sid"; + const db = store as unknown as { db: DbLike }; + db.db.prepare(`INSERT INTO conversations (namespace, conversation_id, session_id, claimed_at) VALUES ('default', 'd2-echo-conv', ?, 0)`).run(sid); + try { + const ids = dMsgs(4).map((m) => m.id); + // turn 1: the cursor issues (anchor — no stored window yet). + const r1 = store.snapStage0(sid, ids, {}); + assert.equal(r1.decision.mode, "anchor", "first turn anchors"); + assert.equal(r1.decision.newIdPos.length, ids.length, "every id issued"); + const w1 = snapL1WindowRead(db.db, "d2-echo-conv")?.window; + assert.ok(w1 !== null && w1 !== undefined, "the cursor persisted under the conversation key"); + // turn 2: the SAME array (an idempotent re-render) moves the cursor + // nowhere — it must take the zero-side-effect echo instead of a + // per-turn re-anchor + upsert. + const r2 = store.snapStage0(sid, ids, {}); + assert.equal(r2.decision.mode, "echo", "re-render ⇒ echo"); + assert.equal(r2.decision.replay, true); + assert.equal(r2.decision.newIdPos.length, 0); + const w2 = snapL1WindowRead(db.db, "d2-echo-conv")?.window; + assert.ok(w2 !== null && w2 !== undefined); + assert.equal(w2.maxPos, w1.maxPos, "the cursor is unchanged"); + assert.deepEqual(w2.tailIds, w1.tailIds, "the window is unchanged"); + // turn 3: a REAL append still advances (one fresh pos). + const r3 = store.snapStage0(sid, [...ids, "m-fresh"], {}); + assert.equal(r3.decision.mode, "anchor", "a real append anchors"); + assert.deepEqual(r3.decision.newIdPos.map((x) => x.id), ["m-fresh"], "only the new id issued"); + } finally { + r3FaultTracker.clearLatched(sid); + } +}); + + +test("#7064 D3: the tail anchor stays live across watermark-hit runs", async (t) => { + const { store } = d23Harness(t); + const sid = "d3-sid"; + const db = store as unknown as { db: DbLike }; + const logs: DLog[] = []; + const log = (level: string, msg: string) => logs.push({ level, msg }); + try { + const msgs1 = dMsgs(4); + const ids1 = msgs1.map((m) => m.id); + const r1 = runStage0({ + sessionId: sid, + lane: "openai", + msgs: msgs1, + incomingMsgIds: ids1, + incomingSeqs: null, + pluginMode: true, + log, + }); + assert.notEqual(r1.pinned, null, "turn 1 pins the host ids"); + assert.equal(r1.anchorAdvanced, true, "the cursor issued"); + assert.equal(snapL1WindowRead(db.db, sid)?.window?.maxPos ?? -1, ids1.length, "cursor issued at the full array length"); + // turn 2 is a watermark HIT (wpHit stubbed) and the host appended one + // message — pre-slimming this turn would have skipped stage-0 and the + // cursor would have gone stale. + const msgs2 = dMsgs(5); + const ids2 = msgs2.map((m) => m.id); + const r2 = runStage0({ + sessionId: sid, + lane: "openai", + msgs: msgs2, + incomingMsgIds: ids2, + incomingSeqs: null, + pluginMode: true, + wpHit: { view: msgs1, lastSeq: ids1.length }, + log, + }); + assert.notEqual(r2.pinned, null, "the hit turn pins too (c3623bf1)"); + assert.equal(r2.anchorAdvanced, true, "the hit turn refreshes the cursor (D3 尾锚常活)"); + const w = snapL1WindowRead(db.db, sid)?.window; + assert.ok(w !== null && w !== undefined, "the window was refreshed"); + assert.equal(w.maxPos, ids2.length, "the cursor now spans the hit turn's array"); + assert.deepEqual(w.tailIds.slice(-1), [ids2[ids2.length - 1]], "the new tail id is in the window"); + // a plain echo still reports no advance (epoch bump stays quiet). + const r3 = runStage0({ + sessionId: sid, + lane: "openai", + msgs: msgs2, + incomingMsgIds: ids2, + incomingSeqs: null, + pluginMode: true, + wpHit: { view: msgs2, lastSeq: ids2.length }, + log, + }); + assert.equal(r3.anchorAdvanced, false, "an echo turn does not bump the epoch"); + } finally { + r3FaultTracker.clearLatched(sid); + } +}); + +// --------------------------------------------------------------------------- +// I* — #7064 §8: the INCREMENTAL pin (steady state rides x-bili-pf-tail-ids, +// the prefix is re-stamped from the ledger, the full form is a snapshot turn) +// --------------------------------------------------------------------------- + +/** Seed the ledger's messages table (what ingestRowsNow does in production, + * after the pin) so snapMsgIdLiveIds has rows to pin from. */ +function seedLedger(db: DbLike, sid: string, ids: string[]): void { + ids.forEach((id, seq) => { + const hash = `h-${sid}-${seq}`; + db.prepare("INSERT OR IGNORE INTO content (hash, byte_len, refcount, algo, bytes) VALUES (?, 1, 0, 'sha256', x'00')").run(hash); + db.prepare("INSERT OR IGNORE INTO messages (session_id, seq, msg_id, ref, ref_num, role, kind, content_hash, identity, retired_at) VALUES (?, ?, ?, NULL, ?, 'user', 'message', ?, '', NULL)").run(sid, seq, id, seq + 1, hash); + }); +} + +test("#7064 §8 I1: bootstrap full pin signals 'incremental'; steady turn pins prefix from the ledger + tail from the header", (t: test.TestContext) => { + const { store } = d23Harness(t); + const sid = "i1-sid"; + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + // bootstrap: the full list arrives on an EMPTY ledger. + const r0 = runStage0({ + sessionId: sid, lane: "openai", msgs: dMsgs(5), + incomingMsgIds: ["a1", "a2", "a3", "a4", "a5"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.notEqual(r0.pinned, null, "bootstrap pins the full list"); + assert.equal(r0.pinMode, "incremental", "a successful pin ⇒ the NEXT turn may go incremental (this is what upgrades the client out of full-list mode)"); + // ingest lands the rows (production order: pin in-request, flush after). + seedLedger(db, sid, ["a1", "a2", "a3", "a4", "a5"]); + // steady state: no full list, only the tail (anchor a5 + new core a6). + const r1 = runStage0({ + sessionId: sid, lane: "openai", msgs: dMsgs(6), + incomingMsgIds: undefined, incomingSeqs: null, + incomingTailIds: ["a5", "a6"], pluginMode: true, log, + }); + assert.notEqual(r1.pinned, null, "the incremental pin applied"); + assert.equal(r1.pinMode, "incremental"); + assert.deepEqual(r1.pinned!.map((m) => m.id).slice(0, 5), ["a1", "a2", "a3", "a4", "a5"], "the prefix is re-stamped from the LEDGER, not the client"); + assert.equal(r1.pinned![5].id, "a6", "the new core is pinned from the tail"); + assert.equal(r1.anchorAdvanced, true, "the cursor extended"); +}); + +test("#7064 §8 I2: any structural mismatch degrades to the 'full' signal — never a reject", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + const sid = "i2-sid"; + seedLedger(db, sid, ["b1", "b2", "b3"]); + const cases: Array<[string, string[]]> = [ + // 2026-11-19: a whole-history tail is now LEGAL (tail-aligned + // overlap covers rows 0..2 and pins the one new core) — the old + // "anchor mismatch" shape is replaced by a drift row INSIDE the + // overlap, which is what the new rule actually rejects. + ["overlap row drift (b1 vs b9, list otherwise tail-aligned)", ["b9", "b2", "b3", "x4"]], + ["list too short to reach into the ledger (no overlap to verify)", ["?"]], + ["list longer than the view (not a tail of THIS turn)", ["b3", "x4", "x5"]], + ]; + for (const [name, tail] of cases) { + const r = runStage0({ + sessionId: sid, lane: "openai", msgs: dMsgs(4), + incomingMsgIds: undefined, incomingSeqs: null, + incomingTailIds: tail, pluginMode: true, log, + }); + assert.equal(r.pinned, null, `${name}: no pin`); + assert.equal(r.pinMode, "full", `${name}: the full signal re-converges next turn`); + } + // a malformed header (null) is simply not the incremental form — no + // pin, no signal, byte-identical to pre-§8. + const r2 = runStage0({ + sessionId: sid, lane: "openai", msgs: dMsgs(4), + incomingMsgIds: undefined, incomingSeqs: null, + incomingTailIds: null, pluginMode: true, log, + }); + assert.equal(r2.pinned, null); + assert.equal(r2.pinMode, null, "malformed tail ⇒ the incremental form was not attempted"); +}); + +test("#7064 §8 I3: empty ledger (true bootstrap) takes the full signal; full form wins when both arrive; a '?' NEW slot keeps its content-derived id", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + // (a) empty ledger + tail only ⇒ 'full' (the list must bootstrap). + const ra = runStage0({ + sessionId: "i3a-sid", lane: "openai", msgs: dMsgs(2), + incomingMsgIds: undefined, incomingSeqs: null, + incomingTailIds: ["c1", "c2"], pluginMode: true, log, + }); + assert.equal(ra.pinned, null, "no ledger ⇒ nothing to pin a prefix from"); + assert.equal(ra.pinMode, "full"); + // (b) BOTH forms present ⇒ the full form runs (bootstrap/reanchor must + // stay reachable regardless of the mode the client is in). + seedLedger(db, "i3b-sid", ["d1", "d2", "d3"]); + const rb = runStage0({ + sessionId: "i3b-sid", lane: "openai", msgs: dMsgs(4), + incomingMsgIds: ["d1", "d2", "d3", "d4"], incomingSeqs: null, + incomingTailIds: ["d3", "d4"], pluginMode: true, log, + }); + assert.notEqual(rb.pinned, null, "the full pin applied"); + assert.equal(rb.pinMode, "incremental", "successful full pin ⇒ incremental signal"); + // (c) a '?' NEW slot (id-less element) keeps its content-derived id — + // only the ledger prefix is re-stamped, the tail is trusted per-slot. + seedLedger(db, "i3c-sid", ["e1", "e2"]); + const rc = runStage0({ + sessionId: "i3c-sid", lane: "openai", msgs: dMsgs(3), + incomingMsgIds: undefined, incomingSeqs: null, + incomingTailIds: ["e2", "?"], pluginMode: true, log, + }); + assert.notEqual(rc.pinned, null); + assert.notEqual(rc.pinned![2].id, "?", "the '?' slot was NOT pinned verbatim"); + assert.equal(rc.pinned![2].id, dMsgs(3)[2].id, "the '?' slot keeps the core's pre-existing content-derived id"); + assert.equal(rc.pinMode, "incremental"); +}); + +test("#7064 §8 phase 2 I4: the TAIL form rides the declared channel itself (responses/anthropic steady state); the body short array wins over the pf header", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + // bootstrap on the ledger with the FULL list (any carrier — here the + // declared channel, exactly as the responses/anthropic lanes see it). + seedLedger(db, "i4-sid", ["f1", "f2", "f3"]); + const r0 = runStage0({ + sessionId: "i4-sid", lane: "responses", msgs: dMsgs(3), + incomingMsgIds: ["f1", "f2", "f3"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.notEqual(r0.pinned, null, "the full form pinned at bootstrap"); + // steady state: the SAME field now carries the short tail (anchor + + // new cores) — len !== view ⇒ the two-form rule takes the tail path + // (no pf header at all on these lanes). + const r1 = runStage0({ + sessionId: "i4-sid", lane: "responses", msgs: dMsgs(4), + incomingMsgIds: ["f3", "f4"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.notEqual(r1.pinned, null, "the short declared array pinned as the tail form"); + assert.deepEqual(r1.pinned!.map((m) => m.id), ["f1", "f2", "f3", "f4"], "prefix re-stamped from the ledger, tail from the array"); + assert.equal(r1.pinMode, "incremental"); + // both carriers present (openai lane shape): the body short array is + // the host-declared channel and WINS over the pf-tail-ids header. + const r2 = runStage0({ + sessionId: "i4-sid", lane: "openai", msgs: dMsgs(4), + incomingMsgIds: ["f3", "f4"], incomingSeqs: null, + incomingTailIds: ["f3", "f9"], pluginMode: true, log, + }); + assert.notEqual(r2.pinned, null); + assert.deepEqual(r2.pinned!.map((m) => m.id), ["f1", "f2", "f3", "f4"], "the body tail won (f9 from the header was NOT used)"); +}); + +test("#7064 §8 phase 2 I5: an invalid entry in the TAIL form degrades to 'full' (drift class — NEVER an R3 throw), on every lane", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + seedLedger(db, "i5-sid", ["g1", "g2"]); + const mk = (badId: string) => runStage0({ + sessionId: "i5-sid", lane: "anthropic", msgs: dMsgs(3), + incomingMsgIds: ["g2", badId], incomingSeqs: null, + pluginMode: true, log, + }); + for (const badId of ["g2", "acp_summary_x", "bad id", "x".repeat(513)]) { + const r = mk(badId); + assert.equal(r.pinned, null, `bad tail entry ${JSON.stringify(badId)} ⇒ no pin`); + assert.equal(r.pinMode, "full", `bad tail entry ${JSON.stringify(badId)} ⇒ the client re-sends the full list next turn`); + } + // the anchor slot is screened too (it is a NEW-core slot: i >= C). + const rA = runStage0({ + sessionId: "i5-sid", lane: "anthropic", msgs: dMsgs(3), + incomingMsgIds: ["g3", "g4"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.equal(rA.pinned, null, "anchor ≠ ledger frontier ⇒ drift, not a fault"); + assert.equal(rA.pinMode, "full"); +}); + +test("#7064 §8 phase 2 I6: the flag OFF leaves a short array pinned-less and signals 'full' (rollback self-heals the client back to the full form)", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + seedLedger(db, "i6-sid", ["h1", "h2"]); + const prev = process.env["BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL"]; + process.env["BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL"] = "0"; + try { + const r = runStage0({ + sessionId: "i6-sid", lane: "responses", msgs: dMsgs(3), + incomingMsgIds: ["h2", "h3"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.equal(r.pinned, null, "flag off ⇒ only the full pin runs server-side"); + assert.equal(r.pinMode, "full", "…and the client is told to go back to the full form (no 4xx, no pin until it does)"); + } finally { + if (prev === undefined) delete process.env["BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL"]; + else process.env["BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL"] = prev; + } +}); + +test("#7064 §8 2026-11-19 I7: an over-ceiling session's TRUNCATED tail pins (the ed71f127 deadlock fix) — the overlap row-by-row proves more than the anchor did", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + const sid = "i7-sid"; + // a 3000-turn session: the ledger holds all rows; the client's carrier + // is over its 8KiB budget this turn (a big growth turn), so the BODY + // carries its TRUNCATED retained list — the last 2048 ids of the view, + // NOT the full form (the client no longer HAS a full list). + const C = 3000; + const ledgerIds = Array.from({ length: C }, (_, i) => `k${i}`); + seedLedger(db, sid, ledgerIds); + const T = C + 2; // two new cores this turn + const retained = 2048; // the client's over-ceiling retention window + const list = Array.from({ length: retained }, (_, i) => ledgerIds[C - retained + i]) + .concat(["new1", "new2"]); + const r = runStage0({ + sessionId: sid, lane: "openai", msgs: dMsgs(T), + incomingMsgIds: list, incomingSeqs: null, + pluginMode: true, log, + }); + assert.notEqual(r.pinned, null, "the truncated tail PINS — the 2047-row overlap matched the ledger"); + assert.equal(r.pinMode, "incremental", "…and re-upgrades the client to the carrier (no dead 'full' loop)"); + assert.deepEqual(r.pinned!.map((m) => m.id).slice(0, C), ledgerIds, "the full prefix re-stamped from the LEDGER"); + assert.equal(r.pinned![C].id, "new1"); + assert.equal(r.pinned![C + 1].id, "new2"); + // the SAME turn with one drift row inside the overlap ⇒ no pin, 'full'. + const drifted = list.slice(); + drifted[0] = "k0-rewritten"; + const rd = runStage0({ + sessionId: sid, lane: "openai", msgs: dMsgs(T), + incomingMsgIds: drifted, incomingSeqs: null, + pluginMode: true, log, + }); + assert.equal(rd.pinned, null, "one drift row in the overlap ⇒ degrade"); + assert.equal(rd.pinMode, "full", "…drift class, never a reject"); +}); + +test("#7064 §8 2026-11-19 I8: the steady-state form is the 1-row-overlap special case — behavior unchanged", (t: test.TestContext) => { + const { store } = d23Harness(t); + const db = (store as unknown as { db: DbLike }).db; + const log = (level: string, msg: string): void => {}; + seedLedger(db, "i8-sid", ["j1", "j2"]); + // anchor + new (L = N+1): overlap = exactly 1 row = the old anchor check. + const ok = runStage0({ + sessionId: "i8-sid", lane: "openai", msgs: dMsgs(3), + incomingMsgIds: ["j2", "j3"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.notEqual(ok.pinned, null); + assert.deepEqual(ok.pinned!.map((m) => m.id), ["j1", "j2", "j3"]); + // anchor ≠ frontier ⇒ the 1-row overlap drifts ⇒ degrade (same verdict + // as the old anchor rule, same shape). + const bad = runStage0({ + sessionId: "i8-sid", lane: "openai", msgs: dMsgs(3), + incomingMsgIds: ["j9", "j3"], incomingSeqs: null, + pluginMode: true, log, + }); + assert.equal(bad.pinned, null); + assert.equal(bad.pinMode, "full"); +}); diff --git a/tests/issue5874-ccr-slim.test.ts b/tests/issue5874-ccr-slim.test.ts new file mode 100644 index 0000000000..ac460d97ca --- /dev/null +++ b/tests/issue5874-ccr-slim.test.ts @@ -0,0 +1,255 @@ +// #6785 step 6 — ccr mirror narrowing (± gate discipline, #3617). +// +// The sessions.ccr_json column used to carry the FULL content store +// (byHash hash→original-text + byRef ref→entry), so EVERY steady-state +// flush re-wrote the whole CCR body (file-read outputs, MB-scale each) +// through the sessions row even when nothing changed. Now: +// +// - the column keeps the SLIM envelope: {version:1, ccrSlim:1, +// byRef: {key: {hash: }}} — KB-scale; +// - ccr_entries carries BOTH halves: content_hash (the entry metadata +// json) AND orig_hash (the original text behind entry.hash — the +// kernel invariant content-store.ts storeOriginal pins +// byHash[entry.hash] === original); +// - steady-state re-saves are a no-op for CCR: the #5744 skip compares +// ord + content_hash + orig_hash, so unchanged entries write ZERO +// bytes; +// - legacy rows keep working: a full column envelope parses byte-exact +// (pre-step-6 stores), and a slim column over NULL-orig ledger rows +// degrades to null (the file lane serves) — never a partial store. +// +// Digest safety: the shadow digest canonicalizes the (byHash pairs, byRef +// values) — the rebuild from the ledger must produce the SAME pair set. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createHash } from "node:crypto"; +import { + planIngest, + type LegacySessionLike, +} from "../src/storage/ingest.js"; +import { openSqliteDatabase, type SqliteDatabase } from "../src/storage/driver.js"; +import { UnifiedStore } from "../src/storage/store.js"; +import { canonicalize } from "../src/storage/canonical.js"; + +const sha = (s: string) => createHash("sha256").update(s).digest("hex"); + +function open(dir: string): UnifiedStore { + return UnifiedStore.open(join(dir, "index.db")); +} +function rm(dir: string): void { + try { rmSync(dir, { recursive: true, force: true }); } catch { /* best effort */ } +} + +const BIG_BODY = "B".repeat(64 * 1024); +const BIG_H = sha(BIG_BODY); +const TOOL_JSON = JSON.stringify({ tool: "bash", output: "ok" }); +const TOOL_H = sha(TOOL_JSON); + +// Kernel invariant (content-store.ts storeOriginal): entry.hash = +// sha256(original text); byHash[entry.hash] = original text. +function makeCcr(): { version: 1, byHash: Record, byRef: Record } { + return { + version: 1, + byHash: { [BIG_H]: BIG_BODY, [TOOL_H]: TOOL_JSON }, + byRef: { + m00010: { hash: BIG_H, rawId: "raw-10", kind: "file read", tokens: 5, chars: BIG_BODY.length, head: "BBBBBBBBBB" }, + m00011: { hash: TOOL_H, rawId: "raw-11", kind: "command", tokens: 2, chars: TOOL_JSON.length, head: "{...}" }, + }, + }; +} + +function session(ccr?: { version: 1, byHash: Record, byRef: Record }): LegacySessionLike { + return { + id: "sess-ccr", + createdAt: 1000, + lastSeen: 2000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: { blocks: [], nextBlockId: 1, nextRunId: 1, messageRefs: { byRaw: { m1: "1001" }, byRef: { "1001": "m1" } } }, + blockContents: new Map(), + lastMessages: [{ id: "m1", role: "user", contentType: "text", text: "hi" }], + lastMessagesFolded: true, + pluginSnapshot: undefined, + contentStore: ccr, + } as LegacySessionLike; +} + +test("ccr slim: the column keeps the envelope only; bytes round-trip through the CAS (#6785 step 6)", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-ccr-slim-")); + const store = open(dir); + const raw = openSqliteDatabase(join(dir, "index.db")); + try { + const ccr = makeCcr(); + store.ingestSession(planIngest(session(ccr)), undefined, { completeLog: true }); + + const col = raw.db + .prepare("SELECT ccr_json FROM sessions WHERE session_id = ?") + .get("sess-ccr") as { ccr_json: string } | undefined; + assert.ok(col, "session row missing"); + const parsed = JSON.parse(col!.ccr_json) as Record; + assert.equal(parsed.version, 1); + assert.equal(parsed.ccrSlim, 1, "column must hold the slim envelope"); + const e10 = makeCcr().byRef.m00010; + assert.deepEqual( + (parsed.byRef as Record)["m00010"], + { hash: sha(JSON.stringify(e10)) }, + "slim byRef entry keeps the content-hash only"); + assert.ok(!col!.ccr_json.includes(BIG_BODY), "original bytes leaked into the ccr_json column"); + assert.ok(!col!.ccr_json.includes(BIG_H.slice(0, 32)), "the ORIGINAL hash key must not ride the column either"); + + // The bytes live in the CAS behind the ledger row. + const entries = store.getCcrEntries("sess-ccr"); + assert.equal(entries.length, 2); + const en10 = entries.find((e) => e.key === "m00010"); + assert.ok(en10, "ledger lost m00010"); + assert.equal(en10!.content, JSON.stringify(e10), "entry metadata json must round-trip"); + assert.equal(en10!.origContent, BIG_BODY, "orig CAS half missing"); + + // getCcr rebuilds the store byte-identically from slim column + CAS. + const served = store.getCcr("sess-ccr"); + assert.ok(served, "slim envelope did not hydrate a store"); + assert.equal(served!.byHash[BIG_H], BIG_BODY, "CAS bytes did not round-trip"); + assert.equal(served!.byHash[TOOL_H], TOOL_JSON); + assert.deepEqual(served!.byRef, makeCcr().byRef, "byRef values must round-trip exactly"); + assert.deepEqual(Object.keys(served!.byHash).sort(), [BIG_H, TOOL_H].sort()); + } finally { + raw.db.close(); + store.close(); + rm(dir); + } +}); + +test("ccr slim: pre-step-6 full columns still serve byte-exact; slim over NULL-orig rows degrades (never partial)", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-ccr-slim-legacy-")); + const store = open(dir); + const raw = openSqliteDatabase(join(dir, "index.db")); + try { + store.ingestSession(planIngest(session(makeCcr())), undefined, { completeLog: true }); + + // A pre-step-6 row in an upgraded db: the full parse path must win, + // entries ignored, bytes verbatim. + const full = { + version: 1, + byHash: { "sha256-ccr-legacy": "LEGACY-BODY" }, + byRef: { m9: { hash: "sha256-ccr-legacy", rawId: "r9", kind: "file read", tokens: 1, chars: 11, head: "L" } }, + }; + raw.db.prepare("UPDATE sessions SET ccr_json = ? WHERE session_id = ?").run(JSON.stringify(full), "sess-ccr"); + const served = store.getCcr("sess-ccr"); + assert.ok(served, "legacy full column did not parse"); + assert.equal(served!.byHash["sha256-ccr-legacy"], "LEGACY-BODY", "legacy full column must serve its own bytes"); + assert.equal((served!.byRef["m9"] as { rawId: string }).rawId, "r9"); + + // Slim column + ledger rows with NULL orig (pre-step-6 rows): the + // bytes are unreachable from the ledger → null (the file lane still + // holds the truth), never a partial store. + raw.db.prepare("UPDATE sessions SET ccr_json = ? WHERE session_id = ?") + .run(JSON.stringify({ version: 1, ccrSlim: 1, byRef: { m1: { hash: "deadbeef" } } }), "sess-ccr"); + raw.db.prepare("UPDATE ccr_entries SET orig_hash = NULL WHERE session_id = ?").run("sess-ccr"); + assert.equal(store.getCcr("sess-ccr"), null, "slim + NULL-orig ledger must degrade to null, not a partial store"); + + // And an empty ledger under a slim column: same degrade. + raw.db.prepare("DELETE FROM ccr_entries WHERE session_id = ?").run("sess-ccr"); + assert.equal(store.getCcr("sess-ccr"), null); + } finally { + raw.db.close(); + store.close(); + rm(dir); + } +}); + +test("ccr slim: steady-state re-ingest writes zero ccr bytes (the point of slimming)", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-ccr-slim-noop-")); + const store = open(dir); + const raw = openSqliteDatabase(join(dir, "index.db")); + try { + const plan = planIngest(session(makeCcr())); + store.ingestSession(plan, undefined, { completeLog: true }); + const casBefore = (raw.db.prepare("SELECT COUNT(*) AS n FROM content").get() as { n: number }).n; + const entriesBefore = JSON.stringify(store.getCcrEntries("sess-ccr").map((e) => [e.key, e.contentHash, e.origHash])); + const en11Before = store.getCcrEntries("sess-ccr").find((e) => e.key === "m00011")!.contentHash; + + // Same CCR again (steady-state save): rows and CAS untouched. + store.ingestSession(planIngest(session(makeCcr())), undefined, { completeLog: true }); + const casAfter = (raw.db.prepare("SELECT COUNT(*) AS n FROM content").get() as { n: number }).n; + assert.equal(casAfter, casBefore, "unchanged CCR must not re-put its bytes into the CAS"); + const entriesAfter = JSON.stringify(store.getCcrEntries("sess-ccr").map((e) => [e.key, e.contentHash, e.origHash])); + assert.equal(entriesAfter, entriesBefore, "ledger rows must be stable across unchanged saves"); + + // Changed CCR content re-puts (positive direction of the skip): the + // ledger row points at a NEW content hash (old row released). + const changed = makeCcr(); + changed.byRef.m00011 = { hash: TOOL_H, rawId: "raw-11b", kind: "command", tokens: 2, chars: TOOL_JSON.length, head: "{...}" }; + store.ingestSession(planIngest(session(changed)), undefined, { completeLog: true }); + const after = store.getCcrEntries("sess-ccr").find((e) => e.key === "m00011"); + assert.equal(after!.content, JSON.stringify(changed.byRef.m00011), "a changed entry must re-put"); + assert.notEqual(after!.contentHash, en11Before, "changed entry must land on a new content hash"); + } finally { + raw.db.close(); + store.close(); + rm(dir); + } +}); + +// Digest-relevant sanity: the canonical digest is a pure function of the +// (byHash pairs, byRef values) — order cannot leak. Pin the pair-set +// equality of the slim rebuild against the live store. +test("ccr slim: canonical digest of a slim-restored store equals the live store's", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-ccr-slim-digest-")); + const store = open(dir); + try { + store.ingestSession(planIngest(session(makeCcr())), undefined, { completeLog: true }); + const live = canonicalize(makeCcr()); + const served = canonicalize(store.getCcr("sess-ccr")); + assert.ok(served, "slim store did not hydrate"); + assert.equal( + sha(JSON.stringify(live)), + sha(JSON.stringify(served)), + "slim round-trip changed the canonical form of the content store"); + } finally { + store.close(); + rm(dir); + } +}); + +// #6785-review regression: the ALWAYS-empty store. Every session materializes +// createContentStore() ({byHash:{},byRef:{}}) via contentStoreOf's ?? fallback, +// so plain (never-compressed) sessions persist a slim envelope with ZERO +// declared refs and ZERO ledger rows. The empty-envelope case is COMPLETE, +// not partial: hydrating it to null desynced the unified replay (undefined) +// from the live view (the empty store object) — a permanent structural +// "diverged: contentStore" digest drift on every sampled verify, forcing a +// full re-ingest per save (O(n²) at large sessions) and walking the +// #4323 escalation → freeze chain. The key-set coverage guard above already +// rejects envelopes that DECLARE refs the ledger lacks (the true partial case). +test("ccr slim: empty envelope + empty ledger hydrates the empty store, digest-parity with the live view", () => { + const dir = mkdtempSync(join(tmpdir(), "bili-ccr-slim-empty-")); + const store = open(dir); + const raw = openSqliteDatabase(join(dir, "index.db")); + try { + const empty = { version: 1 as const, byHash: {} as Record, byRef: {} as Record }; + store.ingestSession(planIngest(session(empty)), undefined, { completeLog: true }); + + // The column carries the slim empty envelope; the ledger stays empty. + const col = raw.db.prepare("SELECT ccr_json FROM sessions WHERE session_id = ?").get("sess-ccr") as { ccr_json: string }; + assert.equal(col.ccr_json, JSON.stringify({ version: 1, ccrSlim: 1, byRef: {} })); + assert.equal(store.getCcrEntries("sess-ccr").length, 0); + + // The hydration must yield the exact empty store — null is the regression. + const served = store.getCcr("sess-ccr"); + assert.ok(served, "empty slim envelope must hydrate (null = the digest-drift regression)"); + assert.deepEqual((served as { byRef: unknown }).byRef, {}); + assert.deepEqual((served as { byHash: unknown }).byHash, {}); + assert.equal( + sha(JSON.stringify(canonicalize(served))), + sha(JSON.stringify(canonicalize(empty))), + "unified replay digest must equal the live view digest for the empty store"); + } finally { + raw.db.close(); + store.close(); + rm(dir); + } +}); diff --git a/tests/issue5874-orphan-blocks.test.ts b/tests/issue5874-orphan-blocks.test.ts new file mode 100644 index 0000000000..2c718aab9a --- /dev/null +++ b/tests/issue5874-orphan-blocks.test.ts @@ -0,0 +1,186 @@ +/** + * #6785 step 7 — the reaped-block journal leaves the sessions row. + * + * One row per reaped block in orphan_blocks (immutable content; refreshed + * views UPDATE); the legacy sessions.orphans_json column migrates onto the + * table on first ingest and then retires (NULL). #4242 merge semantics stay: + * incoming entries win on key collision; entries the live session no longer + * carries survive (table is INSERT/UPDATE, never DELETE within a session). + * Every fallback is pinned in BOTH directions. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { planIngest, LegacyViewBuilder, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { UnifiedStore } from "../src/storage/store.ts"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import type { CompressionBlock } from "acp-kernel"; + +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_STORAGE_UNIFIED = "full"; +process.env.BILI_CONFIG_FILE = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), "bili-orphan-blocks-")), + "no-config.json", +); + +type RawDb = ReturnType["db"]; + +interface Env { + dir: string; + store: UnifiedStore; + raw: () => RawDb; +} + +function env(): Env { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bili-orphan-test-")); + const dbPath = path.join(dir, "index.db"); + const store = UnifiedStore.open(dbPath); + return { dir, store, raw: () => openSqliteDatabase(dbPath).db }; +} + +const block = (i: number, summary: string): CompressionBlock => ({ + blockId: `b${i}`, + runId: `r${i}`, + tier: 1, + summary, + directMessageIds: [], + effectiveMessageIds: [`m${i}`], + directBlockIds: [], + compressedTokens: 10, + createdAt: 1_000 + i, + survivedCount: 0, + generation: "young", + active: true, +}); + +/** A session whose blockContents carry an ORPHAN (b-gone) that state.blocks + * no longer lists — the #2695 reap shape. */ +const mkSession = (id: string, orphan?: { one: unknown; full: unknown }): LegacySessionLike => { + const blocks = [block(1, `${id} b1`)]; + const contents = new Map([ + ["b1", { one: null, full: { text: `${id} b1 ORIGINAL TEXT`, count: 1 } }], + ]); + if (orphan) contents.set("b-gone", orphan); + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "openai" }, + stats: {}, + metadata: {}, + state: { blocks, nextBlockId: 2, nextRunId: 1, messageRefs: { byRaw: {}, byRef: {} } }, + blockContents: contents, + lastMessages: undefined, + lastMessagesFolded: undefined, + pluginSnapshot: undefined, + contentStore: undefined, + } as unknown as LegacySessionLike; +}; + +const ingest = (store: UnifiedStore, legacy: LegacySessionLike) => { + store.ingestSession(planIngest(legacy), undefined, { completeLog: true }); +}; + +const loadView = (store: UnifiedStore, id: string) => { + const rows = store.readSnapshot(() => store.sessionViewRows(id)); + if (rows === null) throw new Error(`no rows for ${id}`); + const builder = new LegacyViewBuilder(rows.session, rows.refs, rows.deadRefs); + builder.addBlocks(rows.blocks); + builder.addMessages(rows.messages); + builder.addCcrEntries(store.getCcrEntries(id)); + builder.addOrphanViews(store.getOrphanViews(id) ?? {}); + return builder.finish(); +}; + +test("orphan_blocks T1: write shape — one row per reaped block, column retires", () => { + const e = env(); + ingest(e.store, mkSession("sess-obs", { one: null, full: { text: "REAPED FULL TEXT", count: 9 } })); + const db = e.raw(); + const rows = db.prepare("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?").all("sess-obs") as { block_id: string; one_json: string | null; full_json: string }[]; + assert.equal(rows.length, 1); + assert.equal(rows[0].block_id, "b-gone"); + assert.equal(rows[0].one_json === null || rows[0].one_json === "null", true); + assert.equal(JSON.parse(rows[0].full_json).text, "REAPED FULL TEXT"); + const col = db.prepare("SELECT orphans_json FROM sessions WHERE session_id = ?").get("sess-obs") as { orphans_json: string | null }; + assert.equal(col.orphans_json, null); + const view = loadView(e.store, "sess-obs"); + assert.equal((view.blockContents.get("b-gone")?.full as { text: string }).text, "REAPED FULL TEXT"); + e.store.close(); +}); + +test("orphan_blocks T2: steady-state re-ingest is a no-op; refresh updates in place", () => { + const e = env(); + ingest(e.store, mkSession("sess-noop", { one: null, full: { text: "STABLE TEXT", count: 1 } })); + const pick = () => JSON.stringify(e.raw().prepare("SELECT block_id, one_json, full_json FROM orphan_blocks WHERE session_id = ?").all("sess-noop")); + const before = pick(); + ingest(e.store, mkSession("sess-noop", { one: null, full: { text: "STABLE TEXT", count: 1 } })); + assert.equal(pick(), before); + ingest(e.store, mkSession("sess-noop", { one: null, full: { text: "REFRESHED TEXT", count: 2 } })); + const rows = e.raw().prepare("SELECT block_id, full_json FROM orphan_blocks WHERE session_id = ?").all("sess-noop") as { block_id: string; full_json: string }[]; + assert.equal(rows.length, 1); + assert.equal(rows[0].block_id, "b-gone"); + assert.equal(JSON.parse(rows[0].full_json).text, "REFRESHED TEXT"); + e.store.close(); +}); + +test("orphan_blocks T3: legacy column migrates onto the table once, column retires", () => { + const e = env(); + e.store.ingestSession(planIngest(mkSession("sess-mig")), undefined, { completeLog: true }); + const db = e.raw(); + db.prepare("UPDATE sessions SET orphans_json = ? WHERE session_id = ?").run(JSON.stringify({ "b-old": { one: null, full: { text: "LEGACY ORPHAN", count: 3 } } }), "sess-mig"); + db.prepare("DELETE FROM orphan_blocks WHERE session_id = ?").run("sess-mig"); + ingest(e.store, mkSession("sess-mig", { one: null, full: { text: "NEW ORPHAN", count: 4 } })); + const rows = db.prepare("SELECT block_id, full_json FROM orphan_blocks WHERE session_id = ? ORDER BY block_id").all("sess-mig") as { block_id: string; full_json: string }[]; + assert.deepEqual(rows.map((r) => r.block_id), ["b-gone", "b-old"]); + assert.equal(JSON.parse(rows[0].full_json).text, "NEW ORPHAN"); + assert.equal(JSON.parse(rows[1].full_json).text, "LEGACY ORPHAN"); + const col = db.prepare("SELECT orphans_json FROM sessions WHERE session_id = ?").get("sess-mig") as { orphans_json: string | null }; + assert.equal(col.orphans_json, null); + const merged = e.store.getOrphanViews("sess-mig"); + assert.ok(merged !== null && Object.keys(merged).length === 2); + assert.equal(JSON.parse(JSON.stringify(merged!["b-old"].full)).text, "LEGACY ORPHAN"); + e.store.close(); +}); + +test("orphan_blocks T4: read union — table and legacy column give the same view; corrupt column keeps table rows", () => { + const e = env(); + ingest(e.store, mkSession("sess-union", { one: null, full: { text: "UNION TEXT", count: 5 } })); + const viewA = loadView(e.store, "sess-union"); + const a = JSON.stringify([...viewA.blockContents.entries()].map(([k, v]) => [k, JSON.stringify(v)]).sort()); + // Legacy column only (table emptied to simulate a pre-upgrade read). + e.raw().prepare("UPDATE sessions SET orphans_json = ? WHERE session_id = ?").run(JSON.stringify({ "b-gone": { one: null, full: { text: "UNION TEXT", count: 5 } } }), "sess-union"); + e.raw().prepare("DELETE FROM orphan_blocks WHERE session_id = ?").run("sess-union"); + const viewB = loadView(e.store, "sess-union"); + const b = JSON.stringify([...viewB.blockContents.entries()].map(([k, v]) => [k, JSON.stringify(v)]).sort()); + assert.equal(b, a); + assert.ok(a.includes("UNION TEXT"), "the orphan text must ride the view"); + // Corrupt column with NO table rows: nothing serves (never a fake store). + e.raw().prepare("UPDATE sessions SET orphans_json = 'not json' WHERE session_id = ?").run("sess-union"); + assert.equal(e.store.getOrphanViews("sess-union"), null); + // Corrupt column WITH table rows: the table stays authoritative. + ingest(e.store, mkSession("sess-union", { one: null, full: { text: "UNION TEXT", count: 5 } })); + e.raw().prepare("UPDATE sessions SET orphans_json = 'not json' WHERE session_id = ?").run("sess-union"); + const merged = e.store.getOrphanViews("sess-union"); + assert.ok(merged !== null && merged["b-gone"] !== undefined, "corrupt column must not drop table rows"); + e.store.close(); +}); + +test("orphan_blocks T5: cascade — deleteSession and archiveSession drop the rows", () => { + const e = env(); + ingest(e.store, mkSession("sess-casc", { one: null, full: { text: "CASC", count: 1 } })); + e.store.deleteSession("sess-casc"); + assert.equal((e.raw().prepare("SELECT COUNT(*) AS n FROM orphan_blocks WHERE session_id = ?").get("sess-casc") as unknown as { n: number }).n, 0); + ingest(e.store, mkSession("sess-arch", { one: null, full: { text: "ARCH", count: 1 } })); + e.store.archiveSession("sess-arch", { archiveDir: e.dir }); + assert.equal((e.raw().prepare("SELECT COUNT(*) AS n FROM orphan_blocks WHERE session_id = ?").get("sess-arch") as unknown as { n: number }).n, 0); + e.store.close(); +}); + +test("orphan_blocks T6: stats count the journal rows", () => { + const e = env(); + ingest(e.store, mkSession("sess-stats", { one: null, full: { text: "STATS", count: 1 } })); + assert.equal(e.store.sessionStorageStats("sess-stats").rows.orphanBlocks, 1); + e.store.close(); +}); diff --git a/tests/issue5874-state-fields.test.ts b/tests/issue5874-state-fields.test.ts new file mode 100644 index 0000000000..2af0e2a0cb --- /dev/null +++ b/tests/issue5874-state-fields.test.ts @@ -0,0 +1,155 @@ +/** + * #6785 step 8 — the frozen state_fields field set, pinned (± #3617 gate + * discipline). src/storage/slim.ts ROW_BACKED_KSTATE_FIELDS is a READ-PATH + * decision: the LegacyView builder must know which kstate fields are + * row-backed, so adding/removing one here silently changes what the column + * carries on old rows. This test pins: + * T1 the exact 9-field set (list equality — a drift is a red build, not a + * quiet column/row repartition); + * T2 all nine fields round-trip through planIngest → state_fields rows → + * loadLegacyView with their values byte-intact; + * T3 steady state: an unchanged re-ingest rewrites zero state_fields rows + * (applyStateFields compares value_json per row, like the ccr slim skip). + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createInitialState } from "acp-kernel"; +import { COMPRESSION_STATE_FIELDS, mergeState } from "../src/state-hydration.js"; +import { planIngest, loadLegacyView, type LegacySessionLike } from "../src/storage/ingest.js"; +import { ROW_BACKED_KSTATE_FIELDS, splitKstate } from "../src/storage/slim.js"; +import { openSqliteDatabase } from "../src/storage/driver.js"; +import { UnifiedStore } from "../src/storage/store.js"; + +process.env.BILI_PERSIST_ZSTD = "0"; +process.env.BILI_STORAGE_UNIFIED = "legacy"; +process.env.BILI_CONFIG_FILE = join(mkdtempSync(join(tmpdir(), "sf-pin-")), "missing.json"); + +/** Every row-backed field DEFINED with a non-empty representative value. */ +function fullState(): Record { + return { + ...createInitialState(), + tokenSnapshot: { "m00003": 1234 }, + tsStamps: { "m00004": 1731000000000 }, + lastPassIds: ["m00005", "m00006"], + hiddenOrphanRefs: ["m00007"], + sweptOrphanCallIds: ["tc_1"], + renderDeferredRefs: ["m00008"], + viewLatches: { m00009: { text: "latched", hidden: true } }, + absorbed: [ + { toolCallId: "tc_2", callMessageId: "m00010", resultMessageId: "m00011", summary: "did the thing", tokensReclaimed: 42, createdAt: 1731000001000 }, + ], + rules: [{ id: "r1", text: "always run the tests" }], + }; +} + +function session(state: Record): LegacySessionLike { + return { + id: "sf-pin", + createdAt: 1000, + lastSeen: 2000, + meta: { protocol: "openai" }, + stats: { requests: 1 }, + metadata: {}, + state: state as never, + blockContents: new Map(), + lastMessages: [{ id: "m1", role: "user", contentType: "text", text: "hi" }], + lastMessagesFolded: true, + } as LegacySessionLike; +} + +test("T1: ROW_BACKED_KSTATE_FIELDS is exactly the nine frozen fields, and every one is in the derived merge list", () => { + // The frozen set — a change here is a read-path decision, so it must be + // a visible, reviewed diff against this literal (not a quiet repartition). + assert.deepEqual( + [...ROW_BACKED_KSTATE_FIELDS].sort(), + [ + "absorbed", + "hiddenOrphanRefs", + "lastPassIds", + "renderDeferredRefs", + "rules", + "sweptOrphanCallIds", + "tokenSnapshot", + "tsStamps", + "viewLatches", + ], + ); + // Every row-backed field must be one the derived merge list can EMIT, + // or it can never reach the rows (and the split silently ignores it). + const derived = [...COMPRESSION_STATE_FIELDS] as string[]; + for (const f of ROW_BACKED_KSTATE_FIELDS) { + assert.ok(derived.includes(f), `row-backed field "${f}" missing from COMPRESSION_STATE_FIELDS`); + } + // And the split partitions exactly: row part = the defined row-backed + // fields, column part = everything else, nothing dropped either way. + const { column, rows } = splitKstate(COMPRESSION_STATE_FIELDS as readonly string[], fullState()); + assert.deepEqual(Object.keys(rows).sort(), [...ROW_BACKED_KSTATE_FIELDS].sort()); + for (const f of Object.keys(rows)) assert.ok(!(f in column), `"${f}" landed in BOTH the column and the rows`); + const all = new Set([...Object.keys(column), ...Object.keys(rows)]); + for (const f of COMPRESSION_STATE_FIELDS) { + if ((fullState()[f as keyof typeof fullState] ?? undefined) !== undefined) assert.ok(all.has(f), `"${f}" lost by the split`); + } +}); + +test("T2: all nine row-backed fields round-trip through planIngest → state_fields → loadLegacyView", () => { + const dir = mkdtempSync(join(tmpdir(), "sf-pin-rt-")); + const store = UnifiedStore.open(join(dir, "index.db")); + try { + store.ingestSession(planIngest(session(fullState()))); + const fields = store.getStateFields("sf-pin"); + assert.ok(fields, "no state_fields rows written"); + assert.deepEqual(Object.keys(fields).sort(), [...ROW_BACKED_KSTATE_FIELDS].sort(), "exactly the nine row-backed fields, no more"); + for (const f of ROW_BACKED_KSTATE_FIELDS) { + assert.equal(fields![f], JSON.stringify(fullState()[f]), `field "${f}" not byte-intact through the round-trip`); + } + const rows = { + session: store.getSessionRow("sf-pin")!, + messages: store.getMessages("sf-pin"), + blocks: store.getBlocks("sf-pin"), + deadRefs: store.getDeadRefs("sf-pin"), + refs: store.getRefs("sf-pin"), + }; + const view = loadLegacyView(rows, undefined, store.getStateFields("sf-pin")); + const st = view.state as Record; + for (const f of ROW_BACKED_KSTATE_FIELDS) { + assert.deepEqual(st[f], fullState()[f], `view.state["${f}"] lost in hydration`); + } + } finally { + store.close(); + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("T3: steady-state re-ingest rewrites zero state_fields rows", () => { + const dir = mkdtempSync(join(tmpdir(), "sf-pin-noop-")); + const store = UnifiedStore.open(join(dir, "index.db")); + const raw = openSqliteDatabase(join(dir, "index.db")); + try { + store.ingestSession(planIngest(session(fullState()))); + const snap = (): string => JSON.stringify( + (raw.db.prepare("SELECT field, value_json, rowid FROM state_fields WHERE session_id = ? ORDER BY field").all("sf-pin") as unknown[]), + ); + const before = snap(); + // Change ONE field only: the other eight rows must stay untouched. + const next = fullState(); + next.rules = [{ id: "r1", text: "always run the tests" }, { id: "r2", text: "no silent failures" }]; + store.ingestSession(planIngest(session(next))); + const after = raw.db.prepare("SELECT field, value_json, rowid FROM state_fields WHERE session_id = ? ORDER BY field").all("sf-pin") as { field: string; value_json: string; rowid: number }[]; + const beforeRows = JSON.parse(before) as { field: string; value_json: string; rowid: number }[]; + for (const b of beforeRows) { + if (b.field === "rules") continue; + const a = after.find((r) => r.field === b.field); + assert.ok(a, `"${b.field}" row vanished on an unrelated change`); + assert.equal(a!.value_json, b.value_json, `"${b.field}" rewritten by an unrelated change`); + } + const rules = after.find((r) => r.field === "rules"); + assert.equal(rules!.value_json, JSON.stringify(next.rules), "the changed field must land"); + } finally { + raw.db.close(); + store.close(); + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/tests/issue6262-fork-msgid-track.test.ts b/tests/issue6262-fork-msgid-track.test.ts new file mode 100644 index 0000000000..4d3711e44b --- /dev/null +++ b/tests/issue6262-fork-msgid-track.test.ts @@ -0,0 +1,338 @@ +// #6262 principle 1 (msgid identity track, discussion #6262): the fork protocol +// gains an OPTIONAL identity track. When both sides expose host-stable message +// ids (a host-pinned parent carries the host ids AS its snapshot rawIds), a +// cell whose id matches but whose bytes drifted no longer truncates the fork +// prefix — the claim extends to the id-stable length, `driftFrom` marking the +// byte stop. Everything degrades to today's byte-only track: absent ids on +// either side, a misaligned array, or an unpinned parent (content-digest +// rawIds never equal host ids) → the byte path, byte for byte. +// +// Server observability: the fork response gains `driftFrom` and status +// "drifted" / "expanded+drifted" when the gate passes; a gate failure is +// SILENT (strict byte fork, no 409 — old clients and unpinned parents see +// exactly today's behavior). +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _resetPluginStateForTest } from "../src/plugin.ts"; +import { _resetSessionsForTest } from "../src/session.ts"; +import { _setForTest } from "../src/registry.ts"; +import { resetToolRingForTest } from "../src/tool-ring.ts"; +import { forkIdentityHashOf, forkOrderHashOf, openaiBodyToCore, tryForkAdoption, type ForkIdentity } from "../src/agent/fork-adopt.ts"; + +process.env.NODE_ENV = "test"; +const testRoot = mkdtempSync(join(tmpdir(), "bili-6262-fork-")); +test.after(() => rmSync(testRoot, { recursive: true, force: true })); +for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = testRoot; + +// --------------------------------------------------------------------------- +// Client track (tryForkAdoption) — stubbed snapshot/fork transport. +// --------------------------------------------------------------------------- + +const PARENT_MESSAGES = [ + { role: "user", content: "u1" }, + { role: "assistant", content: "a1" }, + { role: "user", content: "u2" }, + { role: "assistant", content: "a2" }, + { role: "user", content: "u3" }, +] as const; +const PARENT_BODY = { model: "m", messages: [...PARENT_MESSAGES] }; +const RAW_IDS = ["r-0", "r-1", "r-2", "r-3", "r-4"]; +const PARENT_CORE = openaiBodyToCore(PARENT_BODY); +assert(PARENT_CORE !== null && PARENT_CORE.length === 5, "harness: parent body must project to 5 core messages"); + +// Snapshot identities derived from the parent core: identityHash = the same +// hash the client will recompute, rawId = the (pinned) host id. +const IDENTITIES: ForkIdentity[] = PARENT_CORE.map((m, k) => ({ rawId: RAW_IDS[k]!, ref: null, identityHash: forkIdentityHashOf(m) })); +const SNAPSHOT = { + ok: true, + protocolVersion: 2, + status: "exact", + sessionId: "parent-sess", + parentRevision: "a".repeat(64), + orderHash: forkOrderHashOf(IDENTITIES), + orderedMessages: IDENTITIES, +}; + +type Call = { url: string; body?: string }; +function stubTransport(forkResponse?: { status: number; body?: Record }): { calls: Call[]; fetchImpl: typeof fetch } { + const calls: Call[] = []; + const fetchImpl = (async (url: string | URL | Request, init?: RequestInit) => { + const u = String(url); + calls.push({ url: u, body: typeof init?.body === "string" ? init.body : undefined }); + if (u.includes("/__bili/plugin/snapshot")) { + return new Response(JSON.stringify(SNAPSHOT), { status: 200, headers: { "content-type": "application/json" } }); + } + return new Response(JSON.stringify(forkResponse?.body ?? { ok: true, replayed: false }), { status: forkResponse?.status ?? 201, headers: { "content-type": "application/json" } }); + }) as unknown as typeof fetch; + return { calls, fetchImpl }; +} + +// Child = parent with the 4th cell (assistant "a2") re-rendered by the host — +// same stable id, different bytes. Byte stop = 3, id run = 5. +const DRIFT_BODY = { + model: "m", + messages: [ + { role: "user", content: "u1" }, + { role: "assistant", content: "a1" }, + { role: "user", content: "u2" }, + { role: "assistant", content: "a2-DRIFT" }, + { role: "user", content: "u3" }, + ], +}; + +test("#6262: msgid track extends the prefix across an identity-stable drift cell", async () => { + const { calls, fetchImpl } = stubTransport(); + const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "p", childConversationId: "c", body: DRIFT_BODY, msgIds: [...RAW_IDS], fetchImpl }); + assert.deepEqual(result, { outcome: "adopted", branchPoint: 5, replayed: false }); + const fork = calls.find((c) => c.url.includes("/__bili/plugin/fork")); + assert(fork !== undefined && fork.body !== undefined); + const payload = JSON.parse(fork!.body!) as { + branchPoint: { messageCount: number; orderHash: string }; + orderedMessages: ForkIdentity[]; + prefixMsgIds?: string[]; + driftFrom?: number; + idempotencyKey: string; + }; + assert.equal(payload.branchPoint.messageCount, 5); + assert.equal(payload.driftFrom, 3, "driftFrom = the byte stop"); + assert.deepEqual(payload.prefixMsgIds, RAW_IDS, "host ids for the whole claimed prefix"); + assert.deepEqual(payload.orderedMessages, IDENTITIES, "claimed identities stay the PARENT's, verbatim"); + assert.equal((payload.branchPoint as { orderHash: string }).orderHash, SNAPSHOT.orderHash, "full-length prefix reuses the snapshot orderHash"); + assert.equal(payload.idempotencyKey, "fork:c"); +}); + +test("#6262: absent msgIds → byte track, byte for byte (legacy client)", async () => { + const { calls, fetchImpl } = stubTransport(); + const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "p", childConversationId: "c", body: DRIFT_BODY, fetchImpl }); + assert.deepEqual(result, { outcome: "adopted", branchPoint: 3, replayed: false }); + const payload = JSON.parse(calls.find((c) => c.url.includes("/__bili/plugin/fork"))!.body!) as Record; + assert.equal(payload.branchPoint && (payload.branchPoint as { messageCount: number }).messageCount, 3); + assert.equal("prefixMsgIds" in payload, false, "no id track fields on the byte path"); + assert.equal("driftFrom" in payload, false); + assert.equal((payload.orderedMessages as unknown[]).length, 3); +}); + +test("#6262: unpinned parent (ids never align) → byte track, fields omitted", async () => { + const { calls, fetchImpl } = stubTransport(); + const hostUuids = ["uuid-0", "uuid-1", "uuid-2", "uuid-3", "uuid-4"]; + const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "p", childConversationId: "c", body: DRIFT_BODY, msgIds: hostUuids, fetchImpl }); + assert.deepEqual(result, { outcome: "adopted", branchPoint: 3, replayed: false }); + const payload = JSON.parse(calls.find((c) => c.url.includes("/__bili/plugin/fork"))!.body!) as Record; + assert.equal("prefixMsgIds" in payload, false, "nId=0 < branchPoint → unverified ids are never sent"); + assert.equal("driftFrom" in payload, false); +}); + +test("#6262: id run shorter than the byte run → byte track wins, fields omitted", async () => { + const { calls, fetchImpl } = stubTransport(); + const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "p", childConversationId: "c", body: DRIFT_BODY, msgIds: ["r-0", "r-1", "different"], fetchImpl }); + assert.deepEqual(result, { outcome: "adopted", branchPoint: 3, replayed: false }); + const payload = JSON.parse(calls.find((c) => c.url.includes("/__bili/plugin/fork"))!.body!) as Record; + assert.equal("prefixMsgIds" in payload, false, "nId=2 < branchPoint=3 → omitted"); + assert.equal("driftFrom" in payload, false); +}); + +test("#6262: pinned parent, no drift → ids ride the full byte prefix, no driftFrom", async () => { + const { calls, fetchImpl } = stubTransport(); + const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "p", childConversationId: "c", body: { ...PARENT_BODY, messages: [...PARENT_MESSAGES] }, msgIds: [...RAW_IDS], fetchImpl }); + assert.deepEqual(result, { outcome: "adopted", branchPoint: 5, replayed: false }); + const payload = JSON.parse(calls.find((c) => c.url.includes("/__bili/plugin/fork"))!.body!) as Record; + assert.deepEqual(payload.prefixMsgIds, RAW_IDS); + assert.equal("driftFrom" in payload, false, "no extension → no driftFrom"); +}); + +test("#6262: misaligned msgIds (length ≠ core) → byte track, fields omitted", async () => { + const { calls, fetchImpl } = stubTransport(); + const result = await tryForkAdoption({ base: "http://proxy", parentConversationId: "p", childConversationId: "c", body: DRIFT_BODY, msgIds: RAW_IDS.slice(0, 4), fetchImpl }); + assert.deepEqual(result, { outcome: "adopted", branchPoint: 3, replayed: false }); + const payload = JSON.parse(calls.find((c) => c.url.includes("/__bili/plugin/fork"))!.body!) as Record; + assert.equal("prefixMsgIds" in payload, false); + assert.equal("driftFrom" in payload, false); +}); + +// --------------------------------------------------------------------------- +// Server gate (full chain) — real proxy, real snapshot/fork endpoints. +// --------------------------------------------------------------------------- + +async function harness() { + const dir = mkdtempSync(join(testRoot, "run-")); + for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; + _resetSessionsForTest(); + _resetPluginStateForTest(); + resetToolRingForTest(); + const store = new SessionStore({ dir: dir + "/sessions", enabled: false, debounceMs: 60000 }); + _setStoreForTest(store); + _setForTest({}); + const upstream = http.createServer((req, res) => { + req.on("data", () => {}); + req.on("end", () => { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ id: "chat_test", object: "chat.completion", choices: [{ index: 0, message: { role: "assistant", content: "answer" }, finish_reason: "stop" }], usage: { prompt_tokens: 10000, completion_tokens: 10, total_tokens: 10010 } })); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const addr = upstream.address(); + assert(addr && typeof addr === "object"); + const upstreamUrl = `http://127.0.0.1:${addr.port}`; + const proxy = await startServer({ port: 0, host: "127.0.0.1", upstream: upstreamUrl, routes: { [upstreamUrl]: { models: { "claude-test": { context: 400000 } } } }, modelContextLimit: 400000, kernelConfig: defaultConfig(400000), compress: { injectTool: true, injectNudge: true, preserveRecentMessages: 1, preserveRecentTokens: 0, minCompressRangeChars: 100 }, promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", log: false, debug: false, passthrough: false, autoUpdate: false, mitm: { enabled: false, domains: [] } } as ProxyOptions); + await once(proxy, "listening"); + const paddr = proxy.address(); + assert(paddr && typeof paddr === "object"); + return { upstreamUrl, origin: `http://127.0.0.1:${paddr.port}`, close: async () => { store.cancelAll(); proxy.close(); proxy.closeIdleConnections?.(); upstream.close(); upstream.closeIdleConnections?.(); await Promise.all([once(proxy, "close"), once(upstream, "close")]); } }; +} + +// Parent core = 4 cells (the system message hoists out of the fold space): +// [u, a, u, a]. The drift cell is the LAST parent cell (assistant "a4"). +const PARENT4 = [ + { role: "system", content: "system prompt" }, + { role: "user", content: "first original ".repeat(250) }, + { role: "assistant", content: "second original ".repeat(250) }, + { role: "user", content: "third original ".repeat(250) }, + { role: "assistant", content: "fourth original" }, +]; + +test("#6262: full chain — drifted fork adopts past the byte stop and the child replay rides it", async () => { + const h = await harness(); + try { + const seed = await fetch(`${h.origin}/bili/${h.upstreamUrl}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-bili-plugin": "dsh", "x-bili-plugin-conversation": "parent-sid" }, + body: JSON.stringify({ model: "claude-test", max_tokens: 1024, stream: false, messages: PARENT4 }), + }); + assert.equal(seed.status, 200, await seed.text()); + + const parentSnap = (await (await fetch(`${h.origin}/__bili/plugin/snapshot?conversationId=parent-sid`)).json()) as { + parentRevision: string; + orderHash: string; + orderedMessages: Array<{ rawId: string; ref: string | null; identityHash: string }>; + }; + assert.equal(parentSnap.orderedMessages.length, 4, "system hoists out of the fold space"); + const rawIds = parentSnap.orderedMessages.map((m) => m.rawId); + + // Child: parent with the last cell re-rendered (same host id, new bytes) + a new tail. + const childMessages = [ + PARENT4[0]!, + PARENT4[1]!, + PARENT4[2]!, + PARENT4[3]!, + { role: "assistant", content: "fourth original RE-RENDERED" }, + { role: "user", content: "new fork tail" }, + ]; + const adoption = await tryForkAdoption({ + base: h.origin, + parentConversationId: "parent-sid", + childConversationId: "child-sid", + body: { model: "claude-test", max_tokens: 1024, stream: false, messages: childMessages }, + msgIds: [...rawIds, "tail-host-id"], + }); + assert.deepEqual(adoption, { outcome: "adopted", branchPoint: 4, replayed: false }, "the id track extends the byte stop (3) to 4"); + + const childSnap = (await (await fetch(`${h.origin}/__bili/plugin/snapshot?conversationId=child-sid`)).json()) as { orderedMessages: unknown[] }; + assert.equal(childSnap.orderedMessages.length, 4, "the child inherits the full parent prefix, drift cell included"); + + // The child's first replay (drifted text) must be accepted — the ledger + // reserved the drift cell's id/ref, so the kernel rides it instead of + // 409ing the child conversation. + const replay = await fetch(`${h.origin}/bili/${h.upstreamUrl}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-bili-plugin": "dsh", "x-bili-plugin-conversation": "child-sid" }, + body: JSON.stringify({ model: "claude-test", max_tokens: 1024, stream: false, messages: childMessages }), + }); + assert.equal(replay.status, 200, await replay.text()); + } finally { + await h.close(); + } +}); + +test("#6262: server marks the receipt — status drifted + driftFrom echo; gate failure stays strict (no 409)", async () => { + const h = await harness(); + try { + const seed = await fetch(`${h.origin}/bili/${h.upstreamUrl}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-bili-plugin": "dsh", "x-bili-plugin-conversation": "p2" }, + body: JSON.stringify({ model: "claude-test", max_tokens: 1024, stream: false, messages: PARENT4 }), + }); + assert.equal(seed.status, 200, await seed.text()); + const parentSnap = (await (await fetch(`${h.origin}/__bili/plugin/snapshot?conversationId=p2`)).json()) as { + parentRevision: string; + orderedMessages: Array<{ rawId: string; ref: string | null; identityHash: string }>; + }; + const rawIds = parentSnap.orderedMessages.map((m) => m.rawId); + const prefix = parentSnap.orderedMessages; + + // (1) gate passes → status "drifted" + driftFrom echo. + const driftedRes = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-drift", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-drift", prefixMsgIds: rawIds, driftFrom: 3 }), + }); + const driftedText = await driftedRes.text(); + assert.equal(driftedRes.status, 201, driftedText); + const driftedBody = JSON.parse(driftedText) as { status: string; driftFrom?: number; inheritedBlocks: unknown[] }; + assert.equal(driftedBody.status, "drifted"); + assert.equal(driftedBody.driftFrom, 3); + + // (2) gate failure (ids ≠ parent rawIds) → the byte-verified fork still + // succeeds, STRICT: status "exact", no driftFrom, no 409. + const strictRes = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-strict", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-strict", prefixMsgIds: ["wrong-0", "wrong-1", "wrong-2", "wrong-3"] }), + }); + const strictText = await strictRes.text(); + assert.equal(strictRes.status, 201, strictText); + const strictBody = JSON.parse(strictText) as { status: string; driftFrom?: number }; + assert.equal(strictBody.status, "exact"); + assert.equal("driftFrom" in strictBody, false); + + // (3) prefixMsgIds present but driftFrom === messageCount → normalized + // to no drift: exact. + const normRes = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-norm", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-norm", prefixMsgIds: rawIds, driftFrom: 4 }), + }); + const normText = await normRes.text(); + assert.equal(normRes.status, 201, normText); + const normBody = JSON.parse(normText) as { status: string; driftFrom?: number }; + assert.equal(normBody.status, "exact"); + assert.equal("driftFrom" in normBody, false); + + // (4) parse rejections — 400, never a 409, never a silent ignore. + const badLength = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-bad1", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-bad1", prefixMsgIds: rawIds.slice(0, 3) }), + }); + assert.equal(badLength.status, 400); + const badShape = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-bad2", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-bad2", prefixMsgIds: ["ok", "", "x", "y"] }), + }); + assert.equal(badShape.status, 400); + const driftOnly = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-bad3", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-bad3", driftFrom: 3 }), + }); + assert.equal(driftOnly.status, 400); + const driftOob = await fetch(`${h.origin}/__bili/plugin/fork`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ protocolVersion: 1, parentConversationId: "p2", childConversationId: "c-bad4", parentRevision: parentSnap.parentRevision, branchPoint: { messageCount: 4, orderHash: forkOrderHashOf(prefix) }, orderedMessages: prefix, idempotencyKey: "fork:c-bad4", prefixMsgIds: rawIds, driftFrom: 9 }), + }); + assert.equal(driftOob.status, 400); + } finally { + await h.close(); + } +}); diff --git a/tests/issue6525-dsh-envelope.test.ts b/tests/issue6525-dsh-envelope.test.ts new file mode 100644 index 0000000000..2b1b42026f --- /dev/null +++ b/tests/issue6525-dsh-envelope.test.ts @@ -0,0 +1,143 @@ +/** + * #6525: dsh full-resend envelope (carrier ①) — resolver tests. + * + * The envelope stamps the model body's own parallel id array + * (`biliMsgIds`, core-aligned) so host-pinned prefixes exist on EVERY + * full-resend turn, not only the anthropic-wire incremental lane. + * Resolution never re-implements the host's wire transforms: the pi-ai + * lockstep walkers prove wire-element ⇔ host-message pairing, and the + * kernel's own converters prove core-alignment via the #3642 wireIndex + * back-pointer. These tests pin the resulting accept/bail matrix against + * REAL kernel + REAL walker behavior (no re-implementations). + */ +import assert from "node:assert"; +import { test } from "node:test"; +import { _envelopeMsgIdsForTest as resolve } from "../src/agent/dsh-native.js"; + +// dsh host messages = pi-ai Message shape (id added by the host): +// user: string | (text|image)[]; assistant: (text|thinking|toolCall)[]; +// toolResult: toolCallId + (text|image)[]; system: hoisted/folded prefix. +const sys = (content: string) => ({ id: "s0", role: "system", content, timestamp: 1 }); +const user = (id: string, content: string | unknown[]) => ({ id, role: "user", content, timestamp: 1 }); +const asst = (id: string, content: unknown[]) => ({ id, role: "assistant", content, api: "openai-completions", provider: "fake", model: "fake-model", usage: { input: 1, output: 1, cacheRead: 0, cacheWrite: 0, totalTokens: 2, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, stopReason: "stop", timestamp: 1 }); +const toolResult = (id: string, toolCallId: string, text: string) => ({ id, role: "toolResult", toolCallId, toolName: "bash", content: [{ type: "text", text }], isError: false, timestamp: 1 }); +const text = (t: string) => ({ type: "text", text: t }); +const oai = (messages: unknown[]) => ({ model: "fake-model", messages }); + +test("openai: leading system + plain conversation → core-aligned ids, lead dropped", () => { + const wire = oai([ + { role: "system", content: "S" }, + { role: "user", content: "hello" }, + { role: "assistant", content: "hi" }, + { role: "tool", tool_call_id: "c1", content: "r1" }, + ]); + const host = [sys("S"), user("u1", "hello"), asst("a1", [text("hi")]), toolResult("t1", "c1", "r1")]; + assert.deepStrictEqual(resolve(wire, host), ["u1", "a1", "t1"]); +}); + +test("openai: no leading system → every id", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "user", content: "b" }]); + const host = [user("u1", "a"), user("u2", "b")]; + assert.deepStrictEqual(resolve(wire, host), ["u1", "u2"]); +}); + +test("openai: user [text, image_url] → bail (kernel splits one element into N cores)", () => { + const wire = oai([{ role: "user", content: [{ type: "text", text: "see" }, { type: "image_url", image_url: { url: "x" } }] }]); + const host = [user("u1", [text("see"), { type: "image", data: "x", mimeType: "image/png" }])]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: assistant with parallel tool_calls → bail (N tool_call cores, one host id)", () => { + const wire = oai([{ role: "assistant", content: null, tool_calls: [{ id: "c1", type: "function", function: { name: "f", arguments: "{}" } }, { id: "c2", type: "function", function: { name: "g", arguments: "{}" } }] }]); + const host = [asst("a1", [{ type: "toolCall", id: "c1", name: "f", arguments: {} }, { type: "toolCall", id: "c2", name: "g", arguments: {} }])]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: reasoning_content anywhere → bail (walker refuse, conversion not statically provable)", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "assistant", content: "hi", reasoning_content: "R" }]); + const host = [user("u1", "a"), asst("a1", [text("hi")])]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: non-leading system element → bail (ambiguous placement)", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "system", content: "mid" }, { role: "user", content: "b" }]); + const host = [user("u1", "a"), sys("mid"), user("u2", "b")]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: host list shorter than the wire (dropped message) → bail (leftover proof fails)", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "user", content: "b" }]); + const host = [user("u1", "a")]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: host entry with a different byte stream → bail (fingerprint mismatch, never positional)", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "user", content: "b" }]); + const host = [user("u1", "a"), user("u2", "c")]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("duplicate host ids → bail (reject-class parity with hostMsgIdsOf / validateMsgIds)", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "user", content: "b" }]); + const host = [user("dup", "a"), user("dup", "b")]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: host entry missing an id → bail (all-or-nothing)", () => { + const wire = oai([{ role: "user", content: "a" }, { role: "user", content: "b" }]); + const host = [user("u1", "a"), { role: "user", content: "b", timestamp: 1 }]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("openai: dsh flat host shape (role tool + block tool-call) → core-aligned (churn fix #6525 B7)", () => { + // dsh's flat host messages differ from the pi-ai form the shared walker's + // host fingerprint assumes: tool results carry role "tool" (not + // "toolResult") and assistant tool-call blocks are type "tool-call" + // (hyphen, not camel "toolCall"). Without the resolver's fingerprint-only + // normalization, EVERY tool-bearing turn bailed the all-or-nothing + // envelope and the host ids churned back to content digests on the next + // full-resend (the B7 root cause found via a real-dsh e2e dump). + const wire = oai([ + { role: "system", content: "S" }, + { role: "user", content: "hello" }, + { role: "assistant", content: null, tool_calls: [{ id: "call_1", type: "function", function: { name: "acp_status", arguments: "{}" } }] }, + { role: "tool", tool_call_id: "call_1", content: "r1" }, + { role: "assistant", content: "done" }, + ]); + const host = [ + sys("S"), + user("u1", "hello"), + { id: "a1", role: "assistant", content: [{ type: "tool-call", id: "call_1", name: "acp_status", arguments: "{}" }], timestamp: 1 }, + { id: "t1", role: "tool", toolCallId: "call_1", content: [{ type: "text", text: "r1" }], timestamp: 1 }, + { id: "a2", role: "assistant", content: [{ type: "text", text: "done" }], timestamp: 1 }, + ]; + assert.deepStrictEqual(resolve(wire, host), ["u1", "a1", "t1", "a2"]); +}); + +test("openai: dsh parallel tool_calls still bail (one host id cannot cover N cores)", () => { + const wire = oai([{ role: "assistant", content: null, tool_calls: [{ id: "c1", type: "function", function: { name: "f", arguments: "{}" } }, { id: "c2", type: "function", function: { name: "g", arguments: "{}" } }] }]); + const host = [{ id: "a1", role: "assistant", content: [{ type: "tool-call", id: "c1", name: "f", arguments: "{}" }, { type: "tool-call", id: "c2", name: "g", arguments: "{}" }], timestamp: 1 }]; + assert.strictEqual(resolve(wire, host), undefined); +}); + +test("anthropic: system field separate, full conversation → 1:1 ids", () => { + const wire = { model: "fake-model", max_tokens: 100, system: "S", messages: [ + { role: "user", content: "hi" }, + { role: "assistant", content: [{ type: "text", text: "yo" }] }, + { role: "user", content: [{ type: "tool_result", tool_use_id: "c1", content: "r1" }] }, + ] }; + const host = [sys("S"), user("u1", "hi"), asst("a1", [text("yo")]), toolResult("t1", "c1", "r1")]; + assert.deepStrictEqual(resolve(wire, host), ["u1", "a1", "t1"]); +}); + +test("non-openai/anthropic payload → undefined (google/responses stay content-derived, #6525 v1)", () => { + const wire = { model: "fake-model", input: [{ role: "user", content: "hi" }] }; + assert.strictEqual(resolve(wire, [user("u1", "hi")]), undefined); +}); + +test("degenerate inputs → undefined (never throws)", () => { + assert.strictEqual(resolve({ model: "m" }, []), undefined); + assert.strictEqual(resolve({ model: "m", messages: [] }, [user("u1", "a")]), undefined); + assert.strictEqual(resolve({ model: "m", messages: [{ role: "user", content: "a" }] }, null), undefined); + assert.strictEqual(resolve({ model: "m", messages: [{ role: "user", content: "a" }] }, [null]), undefined); +}); diff --git a/tests/issue6525-pi-omp-envelope.test.ts b/tests/issue6525-pi-omp-envelope.test.ts new file mode 100644 index 0000000000..4584841823 --- /dev/null +++ b/tests/issue6525-pi-omp-envelope.test.ts @@ -0,0 +1,335 @@ +/** #6525 (发): pi/omp full-resend msgid envelope — the LANE-INDEPENDENT send + * path. The incremental lane already stamps `msgIds` on the deltas it diverts + * (anthropic wire only); this is the envelope (`biliMsgIds`) on EVERY + * full-resend body — core-aligned via the kernel wireIndex proof, any + * mismatch bails to no field (byte-identical to today). Tests: + * A. coreAlignEnvelopeIds — the core-alignment proof itself + * B. _bodyProxyVisibleForTest — the #1403 gate (blind tunnel: never stamp) + * C. _resolveEnvelopeIdsForFullResendForTest — resolver (walker + proof) + * D. _stampMsgIdEnvelopeForTest — the stamp (base + biliMsgIds, guards) + * No new msgid VALIDATOR is introduced here (the parity-matrix discipline): + * pairing reuses the lane walkers; the server re-validates at pin time. */ +import assert from "node:assert/strict"; +import test from "node:test"; +import { coreAlignEnvelopeIds, laneMsgIdsForResponsesWire } from "../src/agent/pi-msgids.ts"; +import { + _bodyProxyVisibleForTest, + _resolveEnvelopeIdsForFullResendForTest, + _stampMsgIdEnvelopeForTest, +} from "../src/agent/pi.ts"; + +// --------------------------------------------------------------------------- +// A. coreAlignEnvelopeIds +// --------------------------------------------------------------------------- + +/** The 1:1 openai conversation: lead system hoist + a plain tool round. + * Kernel projection (verified against acp-kernel/wire): user→1 core, + * assistant(content null, one tool_call)→1 core, tool→1 core, + * assistant text→1 core. Every element lands on exactly one core. */ +const OAI_WIRE_1TO1 = [ + { role: "system", content: "you are a test" }, + { role: "user", content: "q" }, + { role: "assistant", content: null, tool_calls: [{ id: "call_1", type: "function", function: { name: "t", arguments: "{}" } }] }, + { role: "tool", tool_call_id: "call_1", content: "result" }, + { role: "assistant", content: "done" }, +]; +const OAI_ALIGNED_1TO1 = ["bili-ps-0", "u1", "a2", "t3", "a4"]; + +test("coreAlignEnvelopeIds: openai 1:1 with lead system → core ids (lead dropped)", () => { + const parsed = { messages: OAI_WIRE_1TO1, model: "m" }; + assert.deepEqual( + coreAlignEnvelopeIds("openai", OAI_ALIGNED_1TO1, OAI_WIRE_1TO1, parsed), + ["u1", "a2", "t3", "a4"], + ); +}); + +test("coreAlignEnvelopeIds: openai assistant text+tool_call splits one element into two cores → bail", () => { + const wire = [ + { role: "user", content: "q" }, + { role: "assistant", content: "talking", tool_calls: [{ id: "call_1", type: "function", function: { name: "t", arguments: "{}" } }] }, + ]; + assert.equal(coreAlignEnvelopeIds("openai", ["u0", "a1"], wire, { messages: wire, model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: openai user [text, image] splits into two cores → bail", () => { + const wire = [ + { role: "user", content: [{ type: "text", text: "see" }, { type: "image_url", image_url: { url: "http://x/i.png" } }] }, + { role: "user", content: "b" }, + ]; + assert.equal(coreAlignEnvelopeIds("openai", ["u0", "u1"], wire, { messages: wire, model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: openai two tool_calls on one assistant → bail", () => { + const wire = [ + { role: "user", content: "q" }, + { + role: "assistant", + content: null, + tool_calls: [ + { id: "call_1", type: "function", function: { name: "t", arguments: "{}" } }, + { id: "call_2", type: "function", function: { name: "t", arguments: "{}" } }, + ], + }, + ]; + assert.equal(coreAlignEnvelopeIds("openai", ["u0", "a1"], wire, { messages: wire, model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: a bili-ps- slot past the lead run (mid-body system) → bail", () => { + const wire = [ + { role: "system", content: "sys" }, + { role: "user", content: "q" }, + { role: "system", content: "mid" }, + ]; + assert.equal(coreAlignEnvelopeIds("openai", ["bili-ps-0", "u1", "bili-ps-2"], wire, { messages: wire, model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: lead-only (everything hoisted) → nothing to align → bail", () => { + assert.equal(coreAlignEnvelopeIds("openai", ["bili-ps-0"], [{ role: "system", content: "s" }], { messages: [{ role: "system", content: "s" }], model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: duplicate id in the aligned tail → bail (no mis-pin)", () => { + const wire = [ + { role: "user", content: "q" }, + { role: "user", content: "r" }, + ]; + assert.equal(coreAlignEnvelopeIds("openai", ["u1", "u1"], wire, { messages: wire, model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: aligned array not 1:1 with the wire element count → bail", () => { + const wire = [ + { role: "user", content: "q" }, + { role: "user", content: "r" }, + ]; + assert.equal(coreAlignEnvelopeIds("openai", ["u1"], wire, { messages: wire, model: "m" }), undefined); +}); + +test("coreAlignEnvelopeIds: anthropic 1:1 (system is a separate field — no lead drop)", () => { + const wire = [ + { role: "user", content: "q" }, + { role: "assistant", content: [{ type: "text", text: "a" }] }, + { role: "user", content: [{ type: "tool_result", tool_use_id: "tu_1", content: "r" }] }, + ]; + const parsed = { system: "sys", messages: wire, model: "claude" }; + assert.deepEqual( + coreAlignEnvelopeIds("anthropic", ["u0", "a1", "t2"], wire, parsed), + ["u0", "a1", "t2"], + ); +}); + +test("coreAlignEnvelopeIds: non-array wire → bail without throwing", () => { + assert.equal(coreAlignEnvelopeIds("openai", ["u0"], "garbage", { messages: "garbage", model: "m" }), undefined); +}); + +// --------------------------------------------------------------------------- +// B. bodyProxyVisible — the #1403 gate +// --------------------------------------------------------------------------- + +test("bodyProxyVisible: a /bili/-wrapped baseUrl is proxy-visible", () => { + const visible = _bodyProxyVisibleForTest({ model: { baseUrl: "http://127.0.0.1:9999/bili/https://upstream.example.com/v1" } }, "pi"); + assert.equal(visible, true); +}); + +test("bodyProxyVisible: a plain upstream baseUrl (blind tunnel) is NOT proxy-visible", () => { + const saved = { proxy: process.env.BILLION_CONTEXT_PROXY, mitm: process.env.BILI_MITM_HOSTS }; + delete process.env.BILLION_CONTEXT_PROXY; + delete process.env.BILI_MITM_HOSTS; + try { + const visible = _bodyProxyVisibleForTest({ model: { baseUrl: "https://api.upstream.example.com/v1" } }, "pi"); + assert.equal(visible, false); + } finally { + if (saved.proxy === undefined) delete process.env.BILLION_CONTEXT_PROXY; + else process.env.BILLION_CONTEXT_PROXY = saved.proxy; + if (saved.mitm === undefined) delete process.env.BILI_MITM_HOSTS; + else process.env.BILI_MITM_HOSTS = saved.mitm; + } +}); + +// --------------------------------------------------------------------------- +// C. resolveEnvelopeIdsForFullResend +// --------------------------------------------------------------------------- + +/** A pi branch that pairs 1:1 onto OAI_WIRE_1TO1 (the walker's lockstep + * contract): entry id is the host msgid the envelope must carry. */ +function pairBranch() { + return [ + { type: "message", id: "u1", message: { role: "user", content: "q" } }, + { type: "message", id: "a2", message: { role: "assistant", content: [{ type: "toolCall", id: "call_1", name: "t", arguments: {} }] } }, + { type: "message", id: "t3", message: { role: "toolResult", toolCallId: "call_1", content: [{ type: "text", text: "result" }] } }, + { type: "message", id: "a4", message: { role: "assistant", content: [{ type: "text", text: "done" }] } }, + ]; +} + +const ctxWithBranch = (branch: unknown) => ({ + model: { baseUrl: "http://127.0.0.1:9999/bili/https://upstream.example.com/v1" }, + sessionManager: { getBranch: () => branch }, +}); + +test("resolveEnvelopeIds: openai full-resend pairs the branch → core-aligned host ids", () => { + const parsed = { messages: OAI_WIRE_1TO1, model: "m" }; + assert.deepEqual(_resolveEnvelopeIdsForFullResendForTest(parsed, ctxWithBranch(pairBranch())), ["u1", "a2", "t3", "a4"]); +}); + +test("resolveEnvelopeIds: a google wire (contents array) is out of scope → undefined", () => { + const parsed = { contents: [{ role: "user", parts: [{ text: "hi" }] }], model: "gemini" }; + assert.equal(_resolveEnvelopeIdsForFullResendForTest(parsed, ctxWithBranch(pairBranch())), undefined); +}); + +test("resolveEnvelopeIds: no branch available → undefined (never guess)", () => { + const parsed = { messages: OAI_WIRE_1TO1, model: "m" }; + assert.equal(_resolveEnvelopeIdsForFullResendForTest(parsed, { model: { baseUrl: "http://127.0.0.1:9999/bili/x" } }), undefined); +}); + +test("resolveEnvelopeIds: branch shorter than the wire (dropped message) → undefined", () => { + const parsed = { messages: OAI_WIRE_1TO1, model: "m" }; + const short = pairBranch().slice(0, 3); + assert.equal(_resolveEnvelopeIdsForFullResendForTest(parsed, ctxWithBranch(short)), undefined); +}); + +test("resolveEnvelopeIds: getBranch throwing is a bail, not a throw", () => { + const parsed = { messages: OAI_WIRE_1TO1, model: "m" }; + const ctx = { + model: { baseUrl: "http://127.0.0.1:9999/bili/x" }, + sessionManager: { getBranch: () => { throw new Error("manager gone"); } }, + }; + assert.equal(_resolveEnvelopeIdsForFullResendForTest(parsed, ctx), undefined); +}); + +// --------------------------------------------------------------------------- +// D. stampMsgIdEnvelope +// --------------------------------------------------------------------------- + +test("stamp: pi visible + pairable → payload gets biliMsgIds, everything else intact", () => { + const payload: Record = { messages: OAI_WIRE_1TO1, model: "m", stream: true }; + const out = _stampMsgIdEnvelopeForTest({ payload }, ctxWithBranch(pairBranch()), "pi", undefined); + assert.deepEqual(out, { messages: OAI_WIRE_1TO1, model: "m", stream: true, biliMsgIds: ["u1", "a2", "t3", "a4"] }); + assert.notEqual(out, payload); // spread copy — the original payload is not mutated + assert.equal(payload.biliMsgIds, undefined); +}); + +test("stamp: blind tunnel (not proxy-visible) → undefined → byte-identical request", () => { + const saved = { proxy: process.env.BILLION_CONTEXT_PROXY, mitm: process.env.BILI_MITM_HOSTS }; + delete process.env.BILLION_CONTEXT_PROXY; + delete process.env.BILI_MITM_HOSTS; + try { + const payload = { messages: OAI_WIRE_1TO1, model: "m" }; + const ctx = { model: { baseUrl: "https://api.upstream.example.com/v1" }, sessionManager: { getBranch: () => pairBranch() } }; + assert.equal(_stampMsgIdEnvelopeForTest({ payload }, ctx, "pi", undefined), undefined); + } finally { + if (saved.proxy === undefined) delete process.env.BILLION_CONTEXT_PROXY; + else process.env.BILLION_CONTEXT_PROXY = saved.proxy; + if (saved.mitm === undefined) delete process.env.BILI_MITM_HOSTS; + else process.env.BILI_MITM_HOSTS = saved.mitm; + } +}); + +test("stamp: omp pck object is the base — the envelope rides beside prompt_cache_key", () => { + const pck = { messages: OAI_WIRE_1TO1, model: "m", prompt_cache_key: "session-x" }; + const out = _stampMsgIdEnvelopeForTest({ payload: { messages: OAI_WIRE_1TO1, model: "m" } }, ctxWithBranch(pairBranch()), "omp", pck); + assert.deepEqual(out, { messages: OAI_WIRE_1TO1, model: "m", prompt_cache_key: "session-x", biliMsgIds: ["u1", "a2", "t3", "a4"] }); +}); + +test("stamp: already-stamped payload is never re-stamped", () => { + const payload = { messages: OAI_WIRE_1TO1, model: "m", biliMsgIds: ["existing"] }; + assert.equal(_stampMsgIdEnvelopeForTest({ payload }, ctxWithBranch(pairBranch()), "pi", undefined), undefined); +}); + +test("stamp: unpairable resolver result → undefined (the request ships as-is)", () => { + const payload = { messages: OAI_WIRE_1TO1, model: "m" }; + assert.equal(_stampMsgIdEnvelopeForTest({ payload }, ctxWithBranch(pairBranch().slice(0, 2)), "pi", undefined), undefined); +}); + +test("stamp: non-object payload → undefined without throwing", () => { + assert.equal(_stampMsgIdEnvelopeForTest({ payload: "garbage" }, ctxWithBranch(pairBranch()), "pi", undefined), undefined); + assert.equal(_stampMsgIdEnvelopeForTest({}, ctxWithBranch(pairBranch()), "pi", undefined), undefined); +}); + +// --------------------------------------------------------------------------- +// E. responses wire — the OpenAI /responses dialect (#6539: all protocols) +// --------------------------------------------------------------------------- + +/** A pi branch that pairs 1:1 onto RESP_WIRE_1TO1 (the responses-wire + * walker's lockstep contract): entry id is the host msgid the envelope must + * carry. Tool call = assistant toolCall entry; tool result = toolResult. */ +function respBranch() { + return [ + { type: "message", id: "u1", message: { role: "user", content: "q" } }, + { type: "message", id: "a2", message: { role: "assistant", content: [{ type: "toolCall", id: "call_1", name: "t", arguments: {} }] } }, + { type: "message", id: "t3", message: { role: "toolResult", toolCallId: "call_1", content: [{ type: "text", text: "result" }] } }, + { type: "message", id: "a4", message: { role: "assistant", content: [{ type: "text", text: "done" }] } }, + ]; +} + +const RESP_WIRE_1TO1 = [ + { type: "message", role: "user", content: "q" }, + { type: "function_call", call_id: "call_1", name: "t", arguments: "{}" }, + { type: "function_call_output", call_id: "call_1", output: "result" }, + { type: "message", role: "assistant", content: [{ type: "output_text", text: "done" }] }, +]; + +test("laneMsgIdsForResponsesWire: 1:1 tool round → entry ids", () => { + assert.deepEqual(laneMsgIdsForResponsesWire(RESP_WIRE_1TO1, respBranch()), ["u1", "a2", "t3", "a4"]); +}); + +test("laneMsgIdsForResponsesWire: leading system run → position-pinned bili-ps slots", () => { + const wire = [{ type: "message", role: "system", content: "you are a test" }, { type: "message", role: "user", content: "q" }, { type: "message", role: "assistant", content: "a" }]; + // explicit minimal branch: user + final assistant (system rides the hoist) + const minimal = [ + { type: "message", id: "u1", message: { role: "user", content: "q" } }, + { type: "message", id: "a2", message: { role: "assistant", content: [{ type: "text", text: "a" }] } }, + ]; + assert.deepEqual(laneMsgIdsForResponsesWire(wire, minimal), ["bili-ps-0", "u1", "a2"]); +}); + +test("laneMsgIdsForResponsesWire: multi-part user (kernel splits it) → bail", () => { + const wire = [{ type: "message", role: "user", content: [{ type: "input_text", text: "a" }, { type: "input_text", text: "b" }] }]; + const branch = [{ type: "message", id: "u1", message: { role: "user", content: "a b" } }]; + assert.equal(laneMsgIdsForResponsesWire(wire, branch), undefined); +}); + +test("laneMsgIdsForResponsesWire: mid-history system item → bail (ambiguous sidecar placement)", () => { + const wire = [{ type: "message", role: "user", content: "q" }, { type: "message", role: "system", content: "mid" }]; + const branch = [{ type: "message", id: "u1", message: { role: "user", content: "q" } }]; + assert.equal(laneMsgIdsForResponsesWire(wire, branch), undefined); +}); + +test("laneMsgIdsForResponsesWire: opaque item (additional_tools) → bail", () => { + const wire = [{ type: "additional_tools", tools: [] }, { type: "message", role: "user", content: "q" }]; + const branch = [{ type: "message", id: "u1", message: { role: "user", content: "q" } }]; + assert.equal(laneMsgIdsForResponsesWire(wire, branch), undefined); +}); + +test("laneMsgIdsForResponsesWire: reasoning item → bail (conservative)", () => { + const wire = [{ type: "reasoning", summary: [{ type: "summary_text", text: "hmm" }] }]; + const branch = [{ type: "message", id: "a1", message: { role: "assistant", content: [{ type: "text", text: "hmm" }] } }]; + assert.equal(laneMsgIdsForResponsesWire(wire, branch), undefined); +}); + +test("coreAlignEnvelopeIds: responses 1:1 with lead system → core ids (lead dropped)", () => { + const wire = [{ type: "message", role: "system", content: "sys" }, { type: "message", role: "user", content: "q" }, { type: "message", role: "assistant", content: "a" }]; + const parsed = { input: wire, model: "m" }; + assert.deepEqual(coreAlignEnvelopeIds("responses", ["bili-ps-0", "u1", "a1"], wire, parsed), ["u1", "a1"]); +}); + +test("coreAlignEnvelopeIds: responses multi-part user splits into N cores → bail", () => { + const wire = [{ type: "message", role: "user", content: [{ type: "input_text", text: "a" }, { type: "input_text", text: "b" }] }]; + const parsed = { input: wire, model: "m" }; + assert.equal(coreAlignEnvelopeIds("responses", ["u0"], wire, parsed), undefined); +}); + +test("coreAlignEnvelopeIds: responses duplicate id in the tail → bail (no mis-pin)", () => { + const wire = [{ type: "message", role: "user", content: "q" }, { type: "message", role: "user", content: "r" }]; + const parsed = { input: wire, model: "m" }; + assert.equal(coreAlignEnvelopeIds("responses", ["u1", "u1"], wire, parsed), undefined); +}); + +test("resolveEnvelopeIds: responses full-resend pairs the branch → core-aligned host ids", () => { + const wire = RESP_WIRE_1TO1; + const parsed = { input: wire, model: "m" }; + assert.deepEqual(_resolveEnvelopeIdsForFullResendForTest(parsed, ctxWithBranch(respBranch())), ["u1", "a2", "t3", "a4"]); +}); + +test("resolveEnvelopeIds: responses + lead system → hoisted slot dropped, host ids kept", () => { + const wire = [{ type: "message", role: "system", content: "sys" }, ...RESP_WIRE_1TO1]; + const parsed = { input: wire, model: "m" }; + assert.deepEqual(_resolveEnvelopeIdsForFullResendForTest(parsed, ctxWithBranch(respBranch())), ["u1", "a2", "t3", "a4"]); +}); diff --git a/tests/issue6539-verbatim-scrub.test.ts b/tests/issue6539-verbatim-scrub.test.ts new file mode 100644 index 0000000000..f051d6fc64 --- /dev/null +++ b/tests/issue6539-verbatim-scrub.test.ts @@ -0,0 +1,110 @@ +/** #6539 (双-review F1/F2): biliMsgIds / biliLedgerSeqs must NEVER reach a + * real model endpoint on a verbatim-forward lane. The four fully-processed + * prepares already strip them; this pins the verbatim scrub + * (`scrubMsgIdFields`) + the count_tokens egress hygiene, so a stamped + * full-resend body can't ride a passthrough/bypass/hop-marker/decode-fail + * lane to a strict-schema upstream (#1403 doctrine) or break #661's body + * fingerprinting. A body WITHOUT the fields must pass back byte-identical + * (same Buffer object), so the normal case is untouched. */ +import test from "node:test"; +import assert from "node:assert/strict"; +import type http from "node:http"; +import { scrubMsgIdFields } from "../src/server.ts"; +import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD } from "../src/msgid-pin.ts"; +import { prepareResponsesCompact } from "../src/server/prepare-responses.ts"; +import type { CompressionCore, Config } from "acp-kernel"; +import type { ResponsesRequestBody } from "acp-kernel/wire"; +import type { Session } from "../src/session.ts"; + +const log = (_level: string, _msg: string) => {}; + +test("scrubMsgIdFields: strips both fields when present", () => { + const body = { + model: "m", + messages: [{ role: "user", content: "hi" }], + [MSG_IDS_BODY_FIELD]: ["a", "b"], + [MSG_SEQS_BODY_FIELD]: [1, 2], + }; + const out = scrubMsgIdFields(Buffer.from(JSON.stringify(body), "utf8"), log); + const parsed = JSON.parse(out.toString("utf8")) as Record; + assert.ok(!Object.prototype.hasOwnProperty.call(parsed, MSG_IDS_BODY_FIELD)); + assert.ok(!Object.prototype.hasOwnProperty.call(parsed, MSG_SEQS_BODY_FIELD)); + // the rest of the body survives + assert.deepEqual(parsed.messages, [{ role: "user", content: "hi" }]); + assert.equal(parsed.model, "m"); +}); + +test("scrubMsgIdFields: strips a lone field", () => { + const body = { messages: [{ role: "user", content: "hi" }], [MSG_SEQS_BODY_FIELD]: [7] }; + const out = scrubMsgIdFields(Buffer.from(JSON.stringify(body), "utf8"), log); + const parsed = JSON.parse(out.toString("utf8")) as Record; + assert.ok(!Object.prototype.hasOwnProperty.call(parsed, MSG_SEQS_BODY_FIELD)); + assert.ok(!Object.prototype.hasOwnProperty.call(parsed, MSG_IDS_BODY_FIELD)); +}); + +test("scrubMsgIdFields: body without the fields passes back the SAME buffer (byte-identical, #661)", () => { + const body = { model: "m", messages: [{ role: "user", content: "hi" }] }; + const buf = Buffer.from(JSON.stringify(body), "utf8"); + const out = scrubMsgIdFields(buf, log); + assert.strictEqual(out, buf, "must be the same Buffer object — zero re-serialization in the normal case"); +}); + +test("scrubMsgIdFields: non-JSON body passes through untouched", () => { + const buf = Buffer.from("not json at all", "utf8"); + assert.strictEqual(scrubMsgIdFields(buf, log), buf); +}); + +test("scrubMsgIdFields: JSON array body passes through untouched (top-level must be an object)", () => { + const buf = Buffer.from(JSON.stringify([{ [MSG_IDS_BODY_FIELD]: ["x"] }]), "utf8"); + assert.strictEqual(scrubMsgIdFields(buf, log), buf); +}); + +test("scrubMsgIdFields: null body passes through untouched", () => { + const buf = Buffer.from("null", "utf8"); + assert.strictEqual(scrubMsgIdFields(buf, log), buf); +}); + +test("scrubMsgIdFields: empty buffer passes through untouched", () => { + const buf = Buffer.alloc(0); + assert.strictEqual(scrubMsgIdFields(buf, log), buf); +}); + +// #6181 B3 (pure-chain continuation): a plugin-lane chained Responses round +// with a previous_response_id + empty input bypasses prepareResponses and +// goes through prepareResponsesCompact VERBATIM. The main pipeline strips the +// msgid carriers; the compact path must do the same, or the carriers leak to +// the real upstream (KDD #17: all four carriers stripped before forwarding). +test("prepareResponsesCompact (B3 passthrough) never forwards the msgid carriers", () => { + const session = { id: "s1", stats: { requests: 0 } } as unknown as Session; + const req = { headers: {} } as unknown as http.IncomingMessage; + const core = {} as unknown as CompressionCore; + const config = { modelContextLimit: 100_000 } as unknown as Config; + const parsed = { + model: "m", + stream: false, + previous_response_id: "resp_1", + input: [], + [MSG_IDS_BODY_FIELD]: ["a", "b"], + [MSG_SEQS_BODY_FIELD]: [1, 2], + } as unknown as ResponsesRequestBody; + const buf = Buffer.from(JSON.stringify(parsed), "utf8"); + const prepared = prepareResponsesCompact(buf, parsed, session, req, core, config, log); + const outBody = JSON.parse(prepared.body.toString("utf8")) as Record; + assert.ok(!Object.prototype.hasOwnProperty.call(outBody, MSG_IDS_BODY_FIELD), "biliMsgIds must not reach the upstream"); + assert.ok(!Object.prototype.hasOwnProperty.call(outBody, MSG_SEQS_BODY_FIELD), "biliLedgerSeqs must not reach the upstream"); + // the rest of the body survives the rebuild + assert.equal(outBody.model, "m"); + assert.deepEqual(outBody.input, []); + assert.equal(outBody.previous_response_id, "resp_1"); +}); + +test("prepareResponsesCompact (B3 passthrough): body without the fields stays byte-identical (same buffer)", () => { + const session = { id: "s1", stats: { requests: 0 } } as unknown as Session; + const req = { headers: {} } as unknown as http.IncomingMessage; + const core = {} as unknown as CompressionCore; + const config = { modelContextLimit: 100_000 } as unknown as Config; + const parsed = { model: "m", stream: false, previous_response_id: "resp_1", input: [] } as unknown as ResponsesRequestBody; + const buf = Buffer.from(JSON.stringify(parsed), "utf8"); + const prepared = prepareResponsesCompact(buf, parsed, session, req, core, config, log); + assert.strictEqual(prepared.body, buf, "no carriers → raw buffer passthrough unchanged"); +}); diff --git a/tests/issue7064-pf-window-ledger.test.ts b/tests/issue7064-pf-window-ledger.test.ts new file mode 100644 index 0000000000..f12550eab9 --- /dev/null +++ b/tests/issue7064-pf-window-ledger.test.ts @@ -0,0 +1,156 @@ +// #7064 ledger drain (PR #6785 family): on a watermark-HIT turn the persist +// snapshot is the ~2-row tail WINDOW — a verified suffix of the ledger, NOT +// the complete log. If that save claims rawness, the #3285 +// authoritative-set sweep retires every prefix row the window omits, +// draining the msgid pin ledger (snapMsgIdLiveIds) to the window: the KDD#21 +// tail form then degrades to the full form every turn (full id list in the +// body, full validation, path=full ingest) and the "唯一 id 事实源" +// oscillates between 100k live rows and the 2-row window. +// +// Both directions are asserted (#3617): the window save must NOT sweep, and a +// genuine complete raw-tail save (history truncation) must STILL sweep. + +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import { mkdtempSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import test from "node:test"; +import { DatabaseSync } from "node:sqlite"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { setLogCapture } from "../src/logger.ts"; + +const MODEL = "gpt-test"; + +function listen(server: http.Server): Promise { + if (server.listening) return Promise.resolve(); + return once(server, "listening").then(() => undefined); +} +function close(server: http.Server): Promise { + return new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); +} + +const msg = (role: "user" | "assistant", i: number) => + ({ role, content: `${role}-message-${i} ${"x".repeat(64)}-${i}` }); + +test("watermark-HIT window snapshot must not retire the ledger prefix (#7064)", async () => { + const dir = mkdtempSync(join(tmpdir(), "pf-window-ledger-")); + test.after(() => rmSync(dir, { recursive: true, force: true })); + for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; + process.env.BILI_STORAGE_UNIFIED = "full"; + const savedRollout = process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK; + process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK = "1"; + const logs: string[] = []; + setLogCapture((_lvl, m) => { logs.push(m); }); + try { + setRegistryForTest({}); + const store = new SessionStore({ dir: join(dir, "sessions"), enabled: true, debounceMs: 15 }); + _setStoreForTest(store); + const upstream = http.createServer((req, res) => { + req.resume(); + req.on("end", () => { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "chatcmpl-test", object: "chat.completion", model: MODEL, + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 }, + })); + }); + }); + upstream.listen(0, "127.0.0.1"); + await listen(upstream); + const llmUrl = `http://127.0.0.1:${(upstream.address() as { port: number }).port}`; + const opts: ProxyOptions = { + port: 0, host: "127.0.0.1", upstream: llmUrl, + routes: { [llmUrl]: { models: { [MODEL]: { context: 400_000 } } } }, + modelContextLimit: 400_000, kernelConfig: defaultConfig(400_000), + compress: { injectTool: true, injectNudge: false }, + promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", + log: true, debug: true, passthrough: false, autoUpdate: false, + compat: { roles: {} }, streamErrorShape: "protocol", passthroughSource: null, + autoRestartOnUpdate: false, updateTag: "latest", advisoryCheck: false, releaseNotesCheck: false, + mitm: { enabled: false, domains: [] }, + }; + const srv = await startServer(opts); + await listen(srv); + try { + const proxyPort = (srv.address() as { port: number }).port; + const sessionId = "conv-pf-window-ledger"; + const post = async (messages: unknown[], headers: Record = {}): Promise> => { + const resp = await fetch(`http://127.0.0.1:${proxyPort}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-bili-plugin": "opencode", "x-acp-session": sessionId, ...headers }, + body: JSON.stringify({ model: MODEL, stream: false, messages }), + }); + assert.equal(resp.status, 200, `expected 200, headers=${JSON.stringify(headers)}`); + await resp.text(); + return new Map([...resp.headers].map(([k, v]) => [k.toLowerCase(), v])); + }; + const settle = async () => { + const ok = await store.afterNextFlush(sessionId); + assert.ok(ok, "first debounced flush landed"); + assert.ok(await store.afterNextFlush(sessionId), "second flush settled"); + }; + const rows = () => { + const db = new DatabaseSync(join(dir, "sessions", "index.db"), { readOnly: true }); + try { + const stmt = db.prepare("SELECT seq, retired_at FROM messages WHERE session_id = ? ORDER BY seq"); + return (stmt.all(sessionId) as unknown as { seq: number; retired_at: number | null }[]); + } finally { + db.close(); + } + }; + + // Turn 1: seed — 6 messages, no pf headers. The success response + // stamps x-bili-pf-seq so the client can send the tail form next. + const seed = [msg("user", 0), msg("assistant", 1), msg("user", 2), msg("assistant", 3), msg("user", 4), msg("assistant", 5)]; + const h1 = await post(seed); + assert.equal(h1.get("x-bili-pf-seq"), "6", "seed turn stamps the watermark seq"); + await settle(); + assert.equal(rows().length, 6, "seed rows ingested"); + + // Turn 2: +1 message, tail form (count=6, tail = anchor + new). + // This must be a fastpath HIT — the window snapshot then presents + // only the tail to the unified ingest. + logs.length = 0; + const h2 = await post([...seed, msg("user", 6)], { + "x-bili-pf-count": "6", + "x-bili-pf-tail-ids": "seed-anchor-5,seed-new-6", + }); + assert.equal(h2.get("x-bili-pf-seq"), "7", "hit turn re-stamps the watermark"); + assert.ok( + logs.some((l) => l.includes("fastpath HIT") && l.includes(sessionId)), + `turn 2 must be a fastpath HIT (log: ${logs.filter((l) => l.includes("fastpath")).slice(0, 4).join(" | ")})`, + ); + await settle(); + const r2 = rows(); + assert.equal(r2.length, 7, "all 7 rows present after the window save"); + assert.ok( + r2.every((r) => r.retired_at === null), + `the window save must NOT retire the verified prefix — retired: ${JSON.stringify(r2.filter((r) => r.retired_at !== null))}`, + ); + + // Turn 3 (negative direction): a real history truncation is a + // legacy full turn whose RAW tail is genuinely complete — the + // #3285 sweep must still retire the dropped prefix. + await post([msg("user", 5), msg("assistant", 6)], { "x-bili-pf-count": "7", "x-bili-pf-tail-ids": "seed-new-6,dropped" }); + await settle(); + const r3 = rows(); + assert.equal(r3.filter((r) => r.retired_at === null).length, 2, "truncated-away prefix rows are retired by the genuine raw-tail sweep"); + } finally { + srv.closeAllConnections?.(); + await close(srv); + store.closeQuietly(); + await close(upstream); + } + } finally { + setLogCapture(null); + if (savedRollout === undefined) delete process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK; + else process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK = savedRollout; + } +}); diff --git a/tests/issue7156-storage-corruption.test.ts b/tests/issue7156-storage-corruption.test.ts index fe2635221f..deacff8f15 100644 --- a/tests/issue7156-storage-corruption.test.ts +++ b/tests/issue7156-storage-corruption.test.ts @@ -74,15 +74,16 @@ function seedSessionWithBlock(dir: string, payload: string): string { return p; } -/** Zero one 4096-byte page. Page 2 (the schema-overflow page for the real 12-table - * DDL) is the deterministic recipe: open() survives (the pragmas + migration only - * touch the header + page 1) but the first write that resolves the schema throws - * "database disk image is malformed" — verified against the real store, stable - * across runs and row counts. */ +/** Zero one 4096-byte page (1-indexed SQLite page number). The deterministic recipe + * is to zero the sessions table's ROOT page (looked up from sqlite_master, so it is + * stable across schema sizes): open() survives (async mode — the sync quick_check + * never runs, and the migrations only touch the schema on page 1) but the first + * write that traverses the sessions b-tree (resetAllIngestWatermarks) throws + * "database disk image is malformed". */ function zeroPage(p: string, page: number): void { const fd = openSync(p, "r+"); try { - writeSync(fd, Buffer.alloc(4096, 0), 0, 4096, page * 4096); + writeSync(fd, Buffer.alloc(4096, 0), 0, 4096, (page - 1) * 4096); } finally { closeSync(fd); } @@ -195,10 +196,12 @@ test("#7156 FIX 4: a first-post-open-write corruption starts WITHOUT the index ( ro.prepare("INSERT INTO sessions (session_id, created_at, updated_at) VALUES (?, 1, 1)").run("s" + i + "-" + "x".repeat(300)); } ro.exec("PRAGMA wal_checkpoint(TRUNCATE)"); + const rootPage = (ro.prepare("SELECT rootpage FROM sqlite_master WHERE name = 'sessions'").get() as { rootpage: number }).rootpage; ro.close(); - // Corrupt page 2: open() survives (async mode — the sync quick_check never - // runs), but the first write (resetAllIngestWatermarks) throws CORRUPT. - zeroPage(p, 2); + // Corrupt the sessions root page: open() survives (async mode — the sync + // quick_check never runs), but the first write (resetAllIngestWatermarks) + // traverses the sessions b-tree and throws CORRUPT. + zeroPage(p, rootPage); setIntegritySyncThresholdForTests(1); // force async mode for any size const errors: string[] = []; setLogCapture((level, msg) => { if (level === "error") errors.push(String(msg)); }); diff --git a/tests/issue7366-pf-hit-ledger-ids.test.ts b/tests/issue7366-pf-hit-ledger-ids.test.ts new file mode 100644 index 0000000000..f49b386591 --- /dev/null +++ b/tests/issue7366-pf-hit-ledger-ids.test.ts @@ -0,0 +1,211 @@ +// #7366 (PR #6785 family): on a watermark-HIT turn in a pinned (2x2) +// session, the rows written for the NEW tail messages must carry the +// CLIENT-DECLARED host msgids — not the content-hash ids minted when the +// tail fragment is re-converted by openaiToCore. Pre-fix, snapshotMessages +// took wpHit.windowCores (fresh h_… ids) and ingest (INSERT OR IGNORE on +// (session_id, seq) — msg_id write-once) wrote h_… into the ledger for +// every post-seed row. The next turn's KDD#21 tail-overlap check then +// compared declared host ids against h_… rows → guaranteed drift → the +// session was locked into permanent full/incremental alternation and the +// tail-only steady state was unreachable. + +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import { mkdtempSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import test from "node:test"; +import { DatabaseSync } from "node:sqlite"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { setLogCapture } from "../src/logger.ts"; + +const MODEL = "gpt-test"; + +function listen(server: http.Server): Promise { + if (server.listening) return Promise.resolve(); + return once(server, "listening").then(() => undefined); +} +function close(server: http.Server): Promise { + return new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); +} + +const msg = (role: "user" | "assistant", i: number) => + ({ role, content: `${role}-message-${i} ${"x".repeat(64)}-${i}` }); + +test("pinned watermark-HIT turns ledger the DECLARED tail ids (#7366)", async () => { + const dir = mkdtempSync(join(tmpdir(), "pf-hit-ledger-ids-")); + test.after(() => rmSync(dir, { recursive: true, force: true })); + for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; + process.env.BILI_STORAGE_UNIFIED = "full"; + const savedWm = process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK; + const savedInc = process.env.BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL; + process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK = "1"; + process.env.BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL = "1"; + const logs: string[] = []; + setLogCapture((_lvl, m) => { logs.push(m); }); + try { + setRegistryForTest({}); + const store = new SessionStore({ dir: join(dir, "sessions"), enabled: true, debounceMs: 15 }); + _setStoreForTest(store); + const upstream = http.createServer((req, res) => { + req.resume(); + req.on("end", () => { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "chatcmpl-test", object: "chat.completion", model: MODEL, + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 }, + })); + }); + }); + upstream.listen(0, "127.0.0.1"); + await listen(upstream); + const llmUrl = `http://127.0.0.1:${(upstream.address() as { port: number }).port}`; + const opts: ProxyOptions = { + port: 0, host: "127.0.0.1", upstream: llmUrl, + routes: { [llmUrl]: { models: { [MODEL]: { context: 400_000 } } } }, + modelContextLimit: 400_000, kernelConfig: defaultConfig(400_000), + compress: { injectTool: true, injectNudge: false }, + promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", + log: true, debug: true, passthrough: false, autoUpdate: false, + compat: { roles: {} }, streamErrorShape: "protocol", passthroughSource: null, + autoRestartOnUpdate: false, updateTag: "latest", advisoryCheck: false, releaseNotesCheck: false, + mitm: { enabled: false, domains: [] }, + }; + const srv = await startServer(opts); + await listen(srv); + try { + const proxyPort = (srv.address() as { port: number }).port; + const sessionId = "conv-pf-hit-ledger-ids"; + const post = async ( + messages: unknown[], + headers: Record = {}, + extraBody: Record = {}, + ): Promise> => { + const resp = await fetch(`http://127.0.0.1:${proxyPort}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-bili-plugin": "opencode", "x-acp-session": sessionId, ...headers }, + body: JSON.stringify({ model: MODEL, stream: false, messages, ...extraBody }), + }); + assert.equal(resp.status, 200, `expected 200, headers=${JSON.stringify(headers)}`); + await resp.text(); + return new Map([...resp.headers].map(([k, v]) => [k.toLowerCase(), v])); + }; + const settle = async () => { + const ok = await store.afterNextFlush(sessionId); + assert.ok(ok, "first debounced flush landed"); + assert.ok(await store.afterNextFlush(sessionId), "second flush settled"); + }; + const idRows = () => { + const db = new DatabaseSync(join(dir, "sessions", "index.db"), { readOnly: true }); + try { + const stmt = db.prepare("SELECT seq, msg_id, retired_at FROM messages WHERE session_id = ? ORDER BY seq"); + return (stmt.all(sessionId) as unknown as { seq: number; msg_id: string; retired_at: number | null }[]); + } finally { + db.close(); + } + }; + + const hostId = (i: number) => `host-id-${i}`; + const seed = [msg("user", 0), msg("assistant", 1), msg("user", 2), msg("assistant", 3), msg("user", 4), msg("assistant", 5)]; + + // Turn 1: seed — full msgid list in the body (bootstrap form). + const h1 = await post(seed, {}, { biliMsgIds: [0, 1, 2, 3, 4, 5].map(hostId) }); + assert.equal(h1.get("x-bili-pf-seq"), "6", "seed turn stamps the watermark seq"); + await settle(); + const r1 = idRows(); + assert.equal(r1.length, 6, "seed rows ingested"); + assert.deepEqual( + r1.map((r) => r.msg_id), + [0, 1, 2, 3, 4, 5].map(hostId), + "seed rows carry the declared host ids", + ); + + // Turn 2: +1 message, KDD#21 tail form (count + anchor/new ids). + // Must be a fastpath HIT — and the NEW row must land with the + // declared id, not a content-hash mint. + logs.length = 0; + const h2 = await post([...seed, msg("user", 6)], { + "x-bili-pf-count": "6", + "x-bili-pf-tail-ids": `${hostId(5)},${hostId(6)}`, + }); + assert.equal(h2.get("x-bili-pf-seq"), "7", "hit turn re-stamps the watermark"); + assert.ok( + logs.some((l) => l.includes("fastpath HIT") && l.includes(sessionId)), + `turn 2 must be a fastpath HIT (log: ${logs.filter((l) => l.includes("fastpath")).slice(0, 4).join(" | ")})`, + ); + await settle(); + const r2 = idRows(); + assert.equal(r2.length, 7, "all 7 rows present after the window save"); + assert.ok(r2.every((r) => r.retired_at === null), "the window save retires nothing"); + assert.equal(r2[6].msg_id, hostId(6), `the HIT-turn new row must carry the DECLARED id (got ${r2[6].msg_id})`); + assert.deepEqual( + r2.slice(0, 6).map((r) => r.msg_id), + [0, 1, 2, 3, 4, 5].map(hostId), + "prefix rows keep their declared ids", + ); + + // Turn 3: the STEADY state — the overlap check compares the + // declared anchor (host-id-6) against the ledger. Pre-fix the + // ledger held h_… here and this turn degraded to the full form. + logs.length = 0; + const h3 = await post([...seed, msg("user", 6), msg("assistant", 7)], { + "x-bili-pf-count": "7", + "x-bili-pf-tail-ids": `${hostId(6)},${hostId(7)}`, + }); + assert.equal(h3.get("x-bili-pf-seq"), "8", "steady turn re-stamps the watermark"); + assert.ok( + logs.some((l) => l.includes("fastpath HIT") && l.includes(sessionId)), + `turn 3 must be a fastpath HIT (log: ${logs.filter((l) => l.includes("fastpath")).slice(0, 4).join(" | ")})`, + ); + assert.ok( + !logs.some((l) => l.includes(sessionId) && l.includes("tail drift")), + `turn 3 must not log a tail drift (log: ${logs.filter((l) => l.includes(sessionId)).slice(-6).join(" | ")})`, + ); + await settle(); + const r3 = idRows(); + assert.equal(r3[7].msg_id, hostId(7), `steady-turn new row carries the declared id (got ${r3[7].msg_id})`); + + // Negative direction (#3617): an UNPINNED session keeps the + // legacy behavior — the HIT-turn window cores land with their + // content-hash ids and no pin mode is advertised. + const logs2 = logs; + const unseed = [msg("user", 0), msg("assistant", 1), msg("user", 2)]; + const h4 = await post(unseed, { "x-acp-session": "conv-unpinned" } as Record); + void h4; + logs2.length = 0; + const h5 = await post([...unseed, msg("assistant", 3)], { "x-acp-session": "conv-unpinned" } as Record, {}); + void h5; + const unrows = (() => { + const db = new DatabaseSync(join(dir, "sessions", "index.db"), { readOnly: true }); + try { + const stmt = db.prepare("SELECT seq, msg_id FROM messages WHERE session_id = ? ORDER BY seq"); + return (stmt.all("conv-unpinned") as unknown as { seq: number; msg_id: string }[]); + } finally { + db.close(); + } + })(); + assert.equal(unrows.length, 4, "unpinned rows ingested"); + assert.ok( + unrows[3].msg_id.startsWith("h_"), + `unpinned sessions keep the legacy content-hash ids (got ${unrows[3].msg_id})`, + ); + } finally { + srv.closeAllConnections?.(); + await close(srv); + store.closeQuietly(); + await close(upstream); + } + } finally { + setLogCapture(null); + if (savedWm === undefined) delete process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK; + else process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK = savedWm; + if (savedInc === undefined) delete process.env.BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL; + else process.env.BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL = savedInc; + } +}); diff --git a/tests/issue7398-egress-serve-e2e.test.ts b/tests/issue7398-egress-serve-e2e.test.ts new file mode 100644 index 0000000000..34a99a73ee --- /dev/null +++ b/tests/issue7398-egress-serve-e2e.test.ts @@ -0,0 +1,173 @@ +// #7398 seg 4b: egress serve-from-log — when rollout egressPlane is on the +// outbound conversation elements come from the op-log (rebuildOutboundFromLog, +// self-contained bytes per #7439 ruling 1) instead of the legacy re-conversion, +// and are byte-identical to the legacy bytes by construction. Direction asserts +// (#3617): the served sessions (text AND tool — tool rows ride the row bytes) +// log "egress-serve: served"; the rollout-off controls never do. +// +// Method: one server, two sessions with IDENTICAL message sequences. Session A +// runs with the rollouts off (control = legacy). Session B runs with +// egressPlane on (served). The fake upstream captures every +// request body; A[n] and B[n] must be byte-identical, and B's log must show +// "egress-serve: served" lines scoped to the served sessions (both scenarios +// serve now — tool rows ride the row bytes); the -A controls never do. + +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import { mkdtempSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import test from "node:test"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { setLogCapture } from "../src/logger.ts"; + +const MODEL = "gpt-test"; + +function listen(server: http.Server): Promise { + if (server.listening) return Promise.resolve(); + return once(server, "listening").then(() => undefined); +} + +const msg = (role: "user" | "assistant", i: number) => + ({ role, content: `${role}-message-${i} ${"x".repeat(64)}-${i}` }); + +function buildBody(history: unknown[]): string { + return JSON.stringify({ model: MODEL, stream: false, messages: history }); +} + +async function runSession( + base: string, + sessionName: string, + script: Array<{ messages: unknown[] }>, + captured: string[], +): Promise { + const ctrl = new AbortController(); + try { + for (const turn of script) { + const res = await fetch(`${base}/v1/chat/completions`, { + method: "POST", + // The server only honors opts.sessionHeader ("x-acp-session"); + // send the session NAME as its value so session.id (and every + // egress-serve log line) carries it for the direction asserts. + headers: { "content-type": "application/json", authorization: "Bearer k", "x-acp-session": sessionName }, + body: buildBody(turn.messages), + signal: ctrl.signal, + }); + const t = await res.text(); + assert.equal(res.status, 200, `turn failed: ${res.status} ${t.slice(0, 200)}`); + captured.push(t); + } + } finally { + ctrl.abort(); + } +} + +test("egress serve-from-log: served bytes == legacy bytes, text and tool alike (#7398 seg 4b)", async () => { + const dir = mkdtempSync(join(tmpdir(), "egress-serve-e2e-")); + test.after(() => rmSync(dir, { recursive: true, force: true })); + for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; + process.env.BILI_STORAGE_UNIFIED = "full"; + process.env.BILI_ROLLOUT_EGRESS_PLANE = "1"; + const savedPlane = process.env.BILI_ROLLOUT_EGRESS_PLANE; + delete process.env.BILI_ROLLOUT_EGRESS_PLANE; + const logs: string[] = []; + setLogCapture((_lvl, m) => { logs.push(m); }); + try { + setRegistryForTest({}); + const store = new SessionStore({ dir: join(dir, "sessions"), enabled: true, debounceMs: 15 }); + _setStoreForTest(store); + const seenBodies: string[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + seenBodies.push(Buffer.concat(chunks).toString("utf8")); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "chatcmpl-test", object: "chat.completion", model: MODEL, + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 }, + })); + }); + }); + upstream.listen(0, "127.0.0.1"); + await listen(upstream); + const llmUrl = `http://127.0.0.1:${(upstream.address() as { port: number }).port}`; + const opts: ProxyOptions = { + port: 0, host: "127.0.0.1", upstream: llmUrl, + routes: { [llmUrl]: { models: { [MODEL]: { context: 400_000 } } } }, + modelContextLimit: 400_000, kernelConfig: defaultConfig(400_000), + compress: { injectTool: false, injectNudge: false }, + promptCache: { routing: "auto" }, sessionHeader: "x-acp-session", + log: true, debug: true, passthrough: false, autoUpdate: false, + compat: { roles: {} }, streamErrorShape: "protocol", passthroughSource: null, + autoRestartOnUpdate: false, updateTag: "latest", advisoryCheck: false, releaseNotesCheck: false, + mitm: { enabled: false, domains: [] }, + }; + const srv = await startServer(opts); + await listen(srv); + const base = `http://127.0.0.1:${(srv.address() as { port: number }).port}`; + + const textScript = [ + { messages: [msg("user", 1), msg("assistant", 2)] }, + { messages: [msg("user", 1), msg("assistant", 2), msg("user", 3), msg("assistant", 4)] }, + { messages: [msg("user", 1), msg("assistant", 2), msg("user", 3), msg("assistant", 4), msg("user", 5)] }, + ]; + const toolScript = [ + { messages: [msg("user", 1)] }, + { + messages: [ + msg("user", 1), + { role: "assistant", content: null, tool_calls: [{ id: "call_t1", type: "function", function: { name: "lookup", arguments: "{\"q\":1}" } }] }, + { role: "tool", tool_call_id: "call_t1", content: "result-1" }, + ], + }, + ]; + + // Session A: rollouts off — pure legacy control. + const ctrlA: string[] = []; + await runSession(base, "serve-e2e-text-A", textScript, ctrlA); + const ctrlAT: string[] = []; + await runSession(base, "serve-e2e-tool-A", toolScript, ctrlAT); + + // Session B: serve on — same sequences must produce identical upstream bytes. + process.env.BILI_ROLLOUT_EGRESS_PLANE = "1"; + const serveB: string[] = []; + await runSession(base, "serve-e2e-text-B", textScript, serveB); + const servedT: string[] = []; + await runSession(base, "serve-e2e-tool-B", toolScript, servedT); + + assert.equal(serveB.length, 3); + for (let i = 0; i < textScript.length; i++) { + const a = JSON.parse(ctrlA[i]); + const b = JSON.parse(serveB[i]); + assert.deepEqual(b.messages, a.messages, `text turn ${i + 1}: served bytes must equal legacy bytes`); + } + const textBLogs = logs.filter((l) => l.includes("egress-serve")); + assert.ok(textBLogs.some((l) => l.includes("egress-serve: served") && l.includes("serve-e2e-text-B")), "text session B must be served from the op-log at least on a steady turn"); + assert.ok(!logs.some((l) => l.includes("egress-serve: served") && (l.includes("serve-e2e-text-A") || l.includes("serve-e2e-tool-A"))), "control sessions (rollouts off) must never be served"); + for (let i = 0; i < toolScript.length; i++) { + const a = JSON.parse(ctrlAT[i]); + const b = JSON.parse(servedT[i]); + assert.deepEqual(b.messages, a.messages, `tool turn ${i + 1}: served bytes must equal legacy bytes`); + } + // Both directions (#3617): the tool session IS served now (the CAS + // body carries the toolCallId/toolName sidecars) and byte-identical. + assert.ok(logs.some((l) => l.includes("egress-serve: served") && l.includes("serve-e2e-tool-B")), "tool session B must be served from the op-log too"); + + srv.closeAllConnections?.(); + await new Promise((r) => srv.close(() => r(undefined))); + upstream.closeAllConnections?.(); + await new Promise((r) => upstream.close(() => r(undefined))); + store.closeQuietly(); + } finally { + if (savedPlane === undefined) delete process.env.BILI_ROLLOUT_EGRESS_PLANE; + else process.env.BILI_ROLLOUT_EGRESS_PLANE = savedPlane; + setLogCapture(null); + } +}); diff --git a/tests/launcher-pi-inline.test.ts b/tests/launcher-pi-inline.test.ts index 23470a3598..4852e9ad39 100644 --- a/tests/launcher-pi-inline.test.ts +++ b/tests/launcher-pi-inline.test.ts @@ -4,6 +4,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import { rmrf } from "./tmp-rm.ts"; +import { acquireDistFixtureLock } from "./dist-fixture-lock.ts"; import { runLaunch } from "../src/launcher.js"; import type { SpawnChild, SpawnFn } from "../src/launcher.js"; import { selfPackageRoot } from "../src/plugin-install.js"; @@ -63,6 +64,7 @@ test("runLaunch pi-inline: no proxy spawned, BILI_PI_INLINE=1, attach/proxy env // tests/launcher.test.ts). const root = selfPackageRoot(); const distAgent = path.join(root, "dist", "agent", "pi.js"); + const releaseFixture = await acquireDistFixtureLock(distAgent); const stubbed = !fs.existsSync(distAgent); if (stubbed) { fs.mkdirSync(path.dirname(distAgent), { recursive: true }); @@ -143,6 +145,7 @@ test("runLaunch pi-inline: no proxy spawned, BILI_PI_INLINE=1, attach/proxy env if (prevMitmHosts === undefined) delete process.env.BILI_MITM_HOSTS; else process.env.BILI_MITM_HOSTS = prevMitmHosts; if (stubbed) fs.rmSync(distAgent, { force: true }); + releaseFixture(); rmrf(home); } }); @@ -168,6 +171,7 @@ test("runLaunch pi-inline: refuses (exit 2) when the bili extension cannot load" const root = selfPackageRoot(); const distAgent = path.join(root, "dist", "agent", "pi.js"); + const releaseFixture = await acquireDistFixtureLock(distAgent); const distBackup = `${distAgent}.bak-test`; const distExisted = fs.existsSync(distAgent); if (distExisted) fs.renameSync(distAgent, distBackup); @@ -207,6 +211,7 @@ test("runLaunch pi-inline: refuses (exit 2) when the bili extension cannot load" if (prevPiDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = prevPiDir; if (distExisted) fs.renameSync(distBackup, distAgent); + releaseFixture(); rmrf(home); } }); diff --git a/tests/launcher.test.ts b/tests/launcher.test.ts index f9bee58dcc..4cec78d73f 100644 --- a/tests/launcher.test.ts +++ b/tests/launcher.test.ts @@ -4,6 +4,7 @@ import { createHash } from "node:crypto"; import fs from "node:fs"; import type { PathLike } from "node:fs"; import { rmrf } from "./tmp-rm.ts"; +import { acquireDistFixtureLock } from "./dist-fixture-lock.ts"; import { supportsFileSymlink } from "./platform-capabilities.ts"; type SymlinkKind = "dir" | "file" | "junction"; import net from "node:net"; @@ -469,6 +470,7 @@ test("runLaunch pi: native -e plugin injected only when not installed", async () // when running tests from a checkout without a prior build. const root = selfPackageRoot(); const distAgent = path.join(root, "dist", "agent", "pi.js"); + const releaseFixture = await acquireDistFixtureLock(distAgent); const stubbed = !fs.existsSync(distAgent); if (stubbed) { fs.mkdirSync(path.dirname(distAgent), { recursive: true }); @@ -553,6 +555,7 @@ test("runLaunch pi: native -e plugin injected only when not installed", async () if (prevPiDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = prevPiDir; if (stubbed) fs.rmSync(distAgent, { force: true }); + releaseFixture(); rmrf(home); } }); @@ -585,6 +588,7 @@ test("runLaunch pi #5428: -s posture override reaches the child env over a hosti // when running tests from a checkout without a prior build. const root = selfPackageRoot(); const distAgent = path.join(root, "dist", "agent", "pi.js"); + const releaseFixture = await acquireDistFixtureLock(distAgent); const stubbed = !fs.existsSync(distAgent); if (stubbed) { fs.mkdirSync(path.dirname(distAgent), { recursive: true }); @@ -646,6 +650,7 @@ test("runLaunch pi #5428: -s posture override reaches the child env over a hosti if (prevInline === undefined) delete process.env.BILI_PI_INLINE; else process.env.BILI_PI_INLINE = prevInline; if (stubbed) fs.rmSync(distAgent, { force: true }); + releaseFixture(); rmrf(home); } }); @@ -664,6 +669,7 @@ test("runLaunch pi #535: refuses launch when http rewrites needed and extension const root = selfPackageRoot(); const distAgent = path.join(root, "dist", "agent", "pi.js"); + const releaseFixture = await acquireDistFixtureLock(distAgent); const distBackup = `${distAgent}.bak-test`; const distExisted = fs.existsSync(distAgent); if (distExisted) fs.renameSync(distAgent, distBackup); @@ -724,6 +730,7 @@ test("runLaunch pi #535: refuses launch when http rewrites needed and extension if (prevPiDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = prevPiDir; if (distExisted) fs.renameSync(distBackup, distAgent); + releaseFixture(); rmrf(home); } }); @@ -910,6 +917,7 @@ test("runLaunch pi #535: refuses launch when ONLY https (hand-wrapped) rewrites const root = selfPackageRoot(); const distAgent = path.join(root, "dist", "agent", "pi.js"); + const releaseFixture = await acquireDistFixtureLock(distAgent); const distBackup = `${distAgent}.bak-test`; const distExisted = fs.existsSync(distAgent); if (distExisted) fs.renameSync(distAgent, distBackup); @@ -927,6 +935,7 @@ test("runLaunch pi #535: refuses launch when ONLY https (hand-wrapped) rewrites if (prevPiDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = prevPiDir; if (distExisted) fs.renameSync(distBackup, distAgent); + releaseFixture(); rmrf(home); } }); diff --git a/tests/msgid-rule-parity.test.ts b/tests/msgid-rule-parity.test.ts index 96af7cd901..557ab45b36 100644 --- a/tests/msgid-rule-parity.test.ts +++ b/tests/msgid-rule-parity.test.ts @@ -16,7 +16,7 @@ import assert from "node:assert/strict"; import { MSG_IDS_BODY_FIELD, incomingMsgIdsOf, validateMsgIds } from "../src/msgid-pin.js"; import { collectRequestIds } from "../src/request-membership.js"; import { PF_SLOT_UNKNOWN, pfCaptureIds, pfLearnStamp, pfRequestHeaders } from "../src/agent/pf-channel.js"; -import { _hostMsgIdsOfForTest, _pfSlotIdsOfForTest } from "../src/agent/dsh-native.js"; +import { ENVELOPE_MSG_IDS_FIELD, _hostMsgIdsOfForTest, _pfSlotIdsOfForTest } from "../src/agent/dsh-native.js"; import { _screenMsgIdsForTest } from "../src/agent/incremental-client.js"; import { _parseTailIdsForTest } from "../src/server/fastpath.js"; @@ -143,10 +143,18 @@ test("pf-channel end-to-end screening via the public handshake API (#5826 parity assert.deepEqual(pfRoundTrip(3, [GOOD, GOOD2, "\u200bcf"]), { "x-bili-pf-count": "3", "x-bili-pf-tail-ids": "%3F" }, "\\p{C} slot screens to ? (emitted percent-encoded)"); assert.deepEqual(pfRoundTrip(3, ["a b", "c d"]), { "x-bili-pf-count": "3" }, "nothing identifiable -> count-only"); assert.deepEqual(pfRoundTrip(2, [GOOD, GOOD]), { "x-bili-pf-count": "2", "x-bili-pf-tail-ids": "%3F" }, "duplicate second slot screens to ?"); - assert.deepEqual(pfRoundTrip(3, LONG_LIST), { "x-bili-pf-count": "3" }, "MAX_IDS (2048) cap degrades to count-only"); + assert.deepEqual(pfRoundTrip(3, LONG_LIST), { "x-bili-pf-count": "3" }, "over-ceiling capture keeps a tail window, but the stamp (3) is far below its base ⇒ slice would be a 2048-id header ⇒ byte budget keeps it count-only"); assert.deepEqual(pfRoundTrip(3, "not-an-array"), { "x-bili-pf-count": "3" }, "failed resolution clears stored list (honest absence beats stale)"); }); +test("dsh envelope body field == server MSG_IDS_BODY_FIELD (structural mirror, review n2)", () => { + // The host bundle mirrors the server's field name (it cannot import + // server modules); if either side renames biliMsgIds the dsh full-resend + // envelope would silently stop being consumed. This pin makes the drift + // a CI failure instead of a one-time "pin rejected" warn. + assert.equal(ENVELOPE_MSG_IDS_FIELD, MSG_IDS_BODY_FIELD); +}); + test("watermark wire encode/decode contract: %2C inside ids, literal commas separate (#5826 parity)", () => { const withComma = "g,B"; // Literal separator commas stay unencoded; commas INSIDE ids travel %2C — diff --git a/tests/openai-msgid-pin.test.ts b/tests/openai-msgid-pin.test.ts index 1f2f085b3c..f6125ded77 100644 --- a/tests/openai-msgid-pin.test.ts +++ b/tests/openai-msgid-pin.test.ts @@ -126,7 +126,8 @@ test("openai wire: biliMsgIds never reach the upstream; rows carry client ids on assert.deepEqual(idsA.sort(), ["open-a-1", "open-u-1", "open-u-2"].sort(), "rows carry the CLIENT ids (pin)"); // Turn B: rollout OFF — same channel, stripped, NOT pinned. - delete process.env[ROLLOUT_ENV]; + // Force off explicitly (the default is now GA-on, #6525). + process.env[ROLLOUT_ENV] = "0"; await post("conv-openai-nopin"); const bodyB = JSON.parse(forwardedOf("question-two conv-openai-nopin")) as Record; assert.equal(bodyB.biliMsgIds, undefined, "strip is unconditional even when rollout is off"); diff --git a/tests/opencode-v1-envelope.test.ts b/tests/opencode-v1-envelope.test.ts new file mode 100644 index 0000000000..7216edafbf --- /dev/null +++ b/tests/opencode-v1-envelope.test.ts @@ -0,0 +1,122 @@ +// #6525/#6539 (发) opencode V1 full-resend envelope — the fetch-time glue in +// opencode-native.ts that positionally proves the transform-harvested host ids +// against the ACTUAL wire and stamps the x-bili-msg-ids header (the server's +// header channel, handle.ts) + feeds the shared watermark slot. The core +// alignment proof (alignedMsgIdsForWire) is covered by opencode-msgids.test.ts; +// these tests cover the glue: wire gating, sid lookup, degrade classes, and +// the never-overwrite / never-restamp-diverted guards. +import assert from "node:assert/strict"; +import { describe, it, beforeEach } from "node:test"; + +import { opencodeV1EnvelopeBeforeSend } from "../src/agent/opencode-native.js"; +import { harvestOpencodeMsgIds, __resetOpencodeMsgIdsForTests } from "../src/agent/opencode-msgids.js"; +import { INCREMENTAL_TURN_MARKER_HEADER } from "../src/agent/incremental-client.js"; + +const SID = "ses_v1_test"; + +function hostMsg(id: string, role: "user" | "assistant", texts: string[]): unknown { + return { info: { id, role, sessionID: SID }, parts: texts.map((text) => ({ type: "text", text })) }; +} +function wireMsg(role: "user" | "assistant", content: unknown): unknown { + return { role, content }; +} +function initFor(body: unknown, headers: Record = {}): RequestInit { + return { method: "POST", headers, body: JSON.stringify(body) }; +} + +describe("opencodeV1EnvelopeBeforeSend (opencode V1 full-resend envelope, #6525/#6539)", () => { + beforeEach(() => { + __resetOpencodeMsgIdsForTests(); + }); + + it("stamps x-bili-msg-ids for a pure-text aligned turn", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_a000000000001", "user", ["hello"]), hostMsg("msg_a000000000002", "assistant", ["hi there"])]); + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", "hello"), wireMsg("assistant", "hi there")] }, + { "x-bili-plugin-conversation": SID }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), "msg_a000000000001,msg_a000000000002"); + }); + + it("accepts block-array content that lowers to the same text sequence", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_b000000000001", "user", ["line1", "line2"])]); + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", [{ type: "text", text: "line1" }, { type: "text", text: "line2" }])] }, + { "x-bili-plugin-conversation": SID }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), "msg_b000000000001"); + }); + + it("degrades (no header) when a harvested message carries a non-text part", async () => { + // tool part → shape:null → poisons alignment for the whole turn. + harvestOpencodeMsgIds(SID, [ + hostMsg("msg_c000000000001", "user", ["call the tool"]), + { info: { id: "msg_c000000000002", role: "assistant", sessionID: SID }, parts: [{ type: "text", text: "" }, { type: "tool", tool: "bash" }] }, + ]); + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", "call the tool"), wireMsg("assistant", "done")] }, + { "x-bili-plugin-conversation": SID }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), null); + }); + + it("degrades when the wire count diverges from the harvest (race / drift)", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_d000000000001", "user", ["one"])]); + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", "one"), wireMsg("assistant", "two")] }, // 2 wire, 1 harvested + { "x-bili-plugin-conversation": SID }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), null); + }); + + it("degrades when no harvest exists for the conversation", async () => { + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", "hello")] }, + { "x-bili-plugin-conversation": "ses_never_harvested" }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), null); + }); + + it("degrades when the conversation header is absent", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_e000000000001", "user", ["hello"])]); + const init = initFor({ model: "claude-x", messages: [wireMsg("user", "hello")] }); // no conversation header + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), null); + }); + + it("never restamps an incremental-lane (diverted) turn", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_f000000000001", "user", ["hello"])]); + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", "hello")] }, + { "x-bili-plugin-conversation": SID, [INCREMENTAL_TURN_MARKER_HEADER]: "turn-1" }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), null); + }); + + it("never overwrites an already-present x-bili-msg-ids (a V2 request stamps its own)", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_g000000000001", "user", ["hello"])]); + const init = initFor( + { model: "claude-x", messages: [wireMsg("user", "hello")] }, + { "x-bili-plugin-conversation": SID, "x-bili-msg-ids": "msg_preexisting0000" }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/messages", init); + assert.equal(new Headers(init.headers).get("x-bili-msg-ids"), "msg_preexisting0000"); + }); + + it("leaves non-anthropic-wire / non-object bodies untouched", async () => { + harvestOpencodeMsgIds(SID, [hostMsg("msg_h000000000001", "user", ["hello"])]); + // openai-shape (role tool present) fails wireMessagesLookAnthropic + const openai = initFor( + { model: "gpt-x", messages: [{ role: "tool", tool_call_id: "call_1", content: "x" }] }, + { "x-bili-plugin-conversation": SID }, + ); + await opencodeV1EnvelopeBeforeSend("https://up/v1/chat/completions", openai); + assert.equal(new Headers(openai.headers).get("x-bili-msg-ids"), null); + }); +}); diff --git a/tests/opencode-v2-msgids.test.ts b/tests/opencode-v2-msgids.test.ts index ecd6bf8f04..178b80cc63 100644 --- a/tests/opencode-v2-msgids.test.ts +++ b/tests/opencode-v2-msgids.test.ts @@ -34,6 +34,52 @@ test("v2 extraction: non-message-array inputs degrade", () => { assert.equal(opencodeV2MsgIdList(["not-an-object" as unknown as { id?: string; role?: string }]), undefined); }); +// #6525/#6539 core alignment: the header must align to the POST-HOIST core +// count the server pins against, so the contiguous leading system/developer +// run (which openaiToCore hoists out of the fold space) is excluded. +const lead = (roles: string[], id = (i: number) => `msg_lead${i}`) => + roles.map((role, i) => ({ id: id(i), role })); + +test("v2 core-align: leading system run is skipped (openai wire case)", () => { + // [system, user, assistant] → only [user, assistant] ids ride. + assert.deepEqual( + opencodeV2MsgIdList(lead(["system", "user", "assistant"])), + ["msg_lead1", "msg_lead2"], + ); + // multiple leading systems + developer all hoist out. + assert.deepEqual( + opencodeV2MsgIdList(lead(["system", "system", "developer", "user", "assistant"])), + ["msg_lead3", "msg_lead4"], + ); +}); + +test("v2 core-align: no leading system is a no-op (anthropic wire case)", () => { + // anthropic keeps system as a separate field, never inside messages → all ride. + assert.deepEqual( + opencodeV2MsgIdList(lead(["user", "assistant", "user"])), + ["msg_lead0", "msg_lead1", "msg_lead2"], + ); +}); + +test("v2 core-align: mid-conversation system keeps its id", () => { + // A system message that is NOT part of the leading run is a real core. + assert.deepEqual( + opencodeV2MsgIdList(lead(["user", "system", "assistant"])), + ["msg_lead0", "msg_lead1", "msg_lead2"], + ); +}); + +test("v2 core-align: only a leading system run → nothing to pin", () => { + assert.equal(opencodeV2MsgIdList(lead(["system"])), undefined); + assert.equal(opencodeV2MsgIdList(lead(["system", "developer"])), undefined); + assert.equal(stampedMsgIdsHeaderFromMessages(lead(["system"])), undefined); +}); + +test("v2 core-align: a poison entry after the leading run still kills the stamp", () => { + // system is skipped, but the missing id on the user message poisons. + assert.equal(opencodeV2MsgIdList([{ id: "msg_s", role: "system" }, { role: "user" }]), undefined); +}); + test("v2 stamp budget: >4096 ids or >16KiB joined degrades without riding", () => { const big = Array.from({ length: 4097 }, (_, i) => ({ id: `msg_x${i}`, role: "user" })); assert.equal(opencodeV2MsgIdList(big), undefined, "per-turn id cap"); @@ -42,3 +88,69 @@ test("v2 stamp budget: >4096 ids or >16KiB joined degrades without riding", () = assert.notEqual(opencodeV2MsgIdList(many), undefined, "list itself is valid at exactly the cap"); assert.equal(stampedMsgIdsHeaderFromMessages(many), undefined, "join budget degrades the STAMP only"); }); + +// --------------------------------------------------------------------------- +// #6785 review / KDD#17: PARITY PIN — the leading-run skip above is only +// correct while the kernel's responsesToCore hoists EXACTLY the same run the +// plugin skips. These tests run the SAME message shapes through both sides +// and assert count parity (stamped ids === post-hoist core message count, +// which is what validateMsgIds checks server-side). If the kernel's hoist +// rule changes — the run boundary moves, a new head-neutral item type lands, +// or mid-history system items stop being position-preserved — this build +// breaks instead of every opencode v2 turn silently degrading to +// content-derived ids. +// --------------------------------------------------------------------------- +import { responsesToCore } from "../kernel/src/wire/responses.ts"; +import type { ResponsesRequestBody } from "../kernel/src/wire/responses.ts"; + +const wireBody = (roles: string[]): ResponsesRequestBody => + ({ model: "test-model", input: roles.map((role) => ({ type: "message", role, content: `t-${role}` })) } as ResponsesRequestBody); + +const hostTurn = (roles: string[]) => roles.map((role, i) => ({ id: `msg_parity${i}`, role })); + +for (const roles of [ + ["system", "developer", "user", "assistant"], + ["developer", "system", "system", "user"], + ["system", "user", "system", "user"], // mid-history system is a positioned sidecar core + ["user", "assistant", "user"], + ["developer", "user"], + ["user"], +] as const) { + test(`v2 parity pin: ${JSON.stringify(roles)} → stamped ids align to the post-hoist core count`, () => { + const core = responsesToCore(wireBody([...roles])); + const ids = opencodeV2MsgIdList(hostTurn([...roles])); + assert.notEqual(ids, undefined, `the turn must stamp for ${JSON.stringify(roles)}`); + assert.equal(ids!.length, core.msgs.length, `stamped id count (${ids!.length}) must equal the kernel's post-hoist core count (${core.msgs.length}) — the server pins on exactly this`); + }); +} + +test("v2 parity pin: an all-leading-run turn hoists to ZERO core messages and the plugin stamps nothing", () => { + const core = responsesToCore(wireBody(["system", "system"])); + assert.equal(core.msgs.length, 0, "kernel: the whole run hoists into systemParts"); + assert.equal(opencodeV2MsgIdList(hostTurn(["system", "system"])), undefined, "plugin: nothing left to pin → no stamp (aligned: there is no core count to match)"); +}); + +test("v2 parity pin: a head-neutral tooling item keeps the kernel's head run OPEN — the plugin side CANNOT express that and must degrade", () => { + // Kernel side (pinned): additional_tools / mcp_list_tools inside the + // leading run do not break sysDevHead — the system after them still + // hoists. The plugin's contiguous-run skip cannot express this nuance, + // and that asymmetry is SAFE only because the head-neutral item carries + // no msg_ id: the whole turn degrades (no header) and the server never + // sees a miscounted list. Pin both halves of that contract. + const core = responsesToCore({ + model: "test-model", + input: [ + { type: "additional_tools", tools: [] }, + { type: "message", role: "system", content: "s" }, + { type: "message", role: "user", content: "u" }, + ], + } as ResponsesRequestBody); + assert.equal(core.systemParts.length, 1, "kernel: the system AFTER the tooling item still hoists (head stayed open)"); + // Plugin side: the same turn arrives as host messages; the tooling item + // has no msg_ id → poison → the whole turn degrades. + assert.equal( + opencodeV2MsgIdList([{ id: undefined, role: "additional_tools" }, { id: "msg_s1", role: "system" }, { id: "msg_u1", role: "user" }]), + undefined, + "plugin: the unexpressible nuance degrades the whole turn (no miscounted list ever rides)", + ); +}); diff --git a/tests/persist-metadata-fold-gate.test.ts b/tests/persist-metadata-fold-gate.test.ts index 3389d7f470..b9f228f1e9 100644 --- a/tests/persist-metadata-fold-gate.test.ts +++ b/tests/persist-metadata-fold-gate.test.ts @@ -72,6 +72,8 @@ test("metadata fold is refused while live metadata diverges from the last commit assert.equal((s.metadata as Record).lateStamp, "unsaved", "the unsaved mutation must stay resident"); store.closeQuietly(); // dual-write pin opens index.db — close before teardown (Windows cannot unlink an open SQLite file; see persist-zstd.test.ts for the family) } finally { + // Close before rm: under unified=full the stores hold index.db + // handles Windows cannot unlink (#6480 pinned full in this file; #6547). closeAll(stores); rmrf(dir); } @@ -101,6 +103,8 @@ test("metadata fold proceeds once the divergent mutation has been committed (#55 store.closeQuietly(); reloaded.closeQuietly(); } finally { + // Close before rm: under unified=full the stores hold index.db + // handles Windows cannot unlink (#6480 pinned full in this file; #6547). closeAll(stores); rmrf(dir); } @@ -119,6 +123,8 @@ test("metadata fold stays closed for a session with no committed save yet (#5506 assert.equal((s.metadata as Record).onlyInMemory, true); store.closeQuietly(); } finally { + // Close before rm: under unified=full the stores hold index.db + // handles Windows cannot unlink (#6480 pinned full in this file; #6547). closeAll(stores); rmrf(dir); } diff --git a/tests/pf-channel.test.ts b/tests/pf-channel.test.ts index 36a1c5f858..4d7f25b7e3 100644 --- a/tests/pf-channel.test.ts +++ b/tests/pf-channel.test.ts @@ -12,12 +12,18 @@ import { makeRecordingFetch, type RecordedCall } from "./helpers/hot-replace.ts" const CONVO = "x-bili-plugin-conversation"; -function learn(headersIn: Record, seq: string | null | undefined, status = 200): void { +function learn(headersIn: Record, seq: string | null | undefined, status = 200, mode?: "incremental" | "full"): void { const respHeaders = new Headers(); if (seq !== null && seq !== undefined) respHeaders.set("x-bili-pf-seq", seq); + if (mode !== undefined) respHeaders.set("x-bili-msgid-mode", mode); pfChannel.pfLearnStamp(headersIn, status, respHeaders); } +// #6785 review fix tests: n distinct short ids (the steady-state shape) +function gen(n: number, prefix = "m"): string[] { + return Array.from({ length: n }, (_, i) => `${prefix}${i + 1}`); +} + test("pf-channel: no stamp yet ⇒ no headers", () => { assert.deepEqual(pfChannel.pfRequestHeaders({ [CONVO]: "sid-fresh-1" }), {}); assert.deepEqual(pfChannel.pfRequestHeaders({}), {}); @@ -43,20 +49,20 @@ test("pf-channel: count mode; tail-ids ride only a validated list; encoding", () assert.deepEqual(pfChannel.pfRequestHeaders({ [CONVO]: sidShort }), { "x-bili-pf-count": "2" }); }); -test("pf-channel: tail-ids capped at 160 when the stamp freezes (#7367)", () => { +test("pf-channel: tail-ids full slice when the stamp freezes and under the byte budget (#7367)", () => { const sid = "sid-cap-1"; const inH = { [CONVO]: sid }; // k2 frozen at 2 (the flag flipped off, so the stamp never advances again) learn(inH, "2"); - // a long session: 200 ids captured; the full [K-1..] slice would be 199 ids + // a long session: 200 ids captured; the full [K-1..] slice is 199 ids pfChannel.pfCaptureIds(sid, Array.from({ length: 200 }, (_, i) => `id${i}`)); const tail = pfChannel.pfRequestHeaders(inH)["x-bili-pf-tail-ids"]; assert.ok(tail !== undefined, "tail-ids present"); const segs = tail!.split(","); - // capped to the NEWEST 160 ids (the window origin ids.length-1 is kept) - assert.equal(segs.length, 160, "tail capped at 160 ids"); - assert.equal(segs[0], "id40", "cap keeps the newest ids, drops the oldest"); - assert.equal(segs[159], "id199", "cap ends at the true window origin"); + // under the byte budget ⇒ the full slice goes out (no count-cap) + assert.equal(segs.length, 199, "full slice under the byte budget"); + assert.equal(segs[0], "id1", "full slice starts at the window origin"); + assert.equal(segs[198], "id199", "full slice ends at the newest id"); // a delta NARROWER than the cap is sent whole (no truncation) const sidNarrow = "sid-cap-narrow"; learn({ [CONVO]: sidNarrow }, "2"); @@ -354,3 +360,162 @@ test("laneMsgIdsForOpenAiWire: bails honestly on anything unprovable", () => { undefined, ); }); + +// --------------------------------------------------------------------------- +// #7064 §8: the pin-mode signal (x-bili-msgid-mode) — the client sends the +// FULL msgid list only until the server proves steady-state incremental pins +// work for this conversation; a 'full' signal / 4xx / shrink re-converges. +// --------------------------------------------------------------------------- + +test("pf-channel: 'incremental' signal ⇒ the full msgid list stops riding the wire", () => { + const sid = "sid-mode-inc-1"; + const inH = { [CONVO]: sid }; + // not learned yet ⇒ never suppress + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), false); + // the signal rides WITH the seq stamp (the server stamps both) + learn(inH, "10", 200, "incremental"); + // #6785 review fix: suppression requires the CARRIER to be capturable — + // production always captures this turn's ids before the stamp decision + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), true, "wireCount == learned total ⇒ steady state"); + pfChannel.pfCaptureIds(sid, gen(12)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 12), true, "growing wire ⇒ still steady state"); + // a mode without a usable seq total is inert (structural coupling to the + // watermark cohort — the server only stamps the signal there) + const sidNoSeq = "sid-mode-inc-noseq"; + learn({ [CONVO]: sidNoSeq }, null, 200, "incremental"); + assert.equal(pfChannel.pfSuppressFullMsgIds(sidNoSeq, 10), false, "no seq total ⇒ nothing to anchor a count against"); +}); + +test("pf-channel: 'full' signal ⇒ the full list keeps riding (no suppression)", () => { + const sid = "sid-mode-full-1"; + const inH = { [CONVO]: sid }; + learn(inH, "10", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), true); + // a degraded turn tells the client to go full again + learn(inH, "10", 200, "full"); + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), false, "'full' overwrites 'incremental'"); + // the client can grow back into steady state once pins succeed again + learn(inH, "11", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(11)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 11), true); +}); + +test("pf-channel: shrink (wire shorter than the learned total) ⇒ clear + send full this turn", () => { + const sid = "sid-mode-shrink-1"; + const inH = { [CONVO]: sid }; + learn(inH, "10", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), true); + // compaction/rewind: the wire shrank — the stale anchor cannot be pinned + pfChannel.pfCaptureIds(sid, gen(7)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 7), false, "shrinking turn sends the full list"); + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), false, "the mode is CLEARED, not just skipped for one turn"); + // a fresh incremental signal re-arms it + learn(inH, "7", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(7)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 7), true); +}); + +test("pf-channel: a >=4xx response clears the learned mode immediately", () => { + const sid = "sid-mode-4xx-1"; + const inH = { [CONVO]: sid }; + learn(inH, "10", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), true); + learn(inH, null, 400); + pfChannel.pfCaptureIds(sid, gen(10)); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 10), false, "4xx ⇒ back to full until a 2xx restamps"); + // 4xx on an unknown sid is a no-op + learn({ [CONVO]: "sid-mode-4xx-unknown" }, null, 429); +}); + +// #6785 review fix (2026-11-19, owner): ultra-long sessions still send +// TAIL-ONLY — the capture keeps the last MAX_IDS entries (baseOffset) and +// the steady-state slice (t-1, one core short of the list end) always lands +// in that window. Only a slice reaching BELOW the kept window (a stale +// small stamp after a big growth) or an over-byte-budget tail degrades to +// the full form riding instead — the server then re-proves and restamps. +test("pf-channel: list over the capture ceiling ⇒ tail window kept, tail-only slice goes out, body suppressed", () => { + const sid = "sid-mode-oversize-1"; + const inH = { [CONVO]: sid }; + // steady state: previous total 2999, this turn adds one core (m3000) + learn(inH, "2999", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(3000)); // > MAX_IDS ⇒ last 2048 kept, baseOffset 952 + const h = pfChannel.pfRequestHeaders(inH); + assert.equal(h["x-bili-pf-count"], "2999"); + assert.equal(h["x-bili-pf-tail-ids"], "m2999,m3000", "anchor + new core only — the body's full list stays suppressed"); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 3000), true, "carrier out ⇒ suppression on (no full-body bytes for a 3000-id session)"); +}); + +test("pf-channel: stamp below the kept tail window ⇒ count-only AND full form rides (safe degrade)", () => { + const sid = "sid-mode-oversize-2"; + const inH = { [CONVO]: sid }; + // stale tiny stamp (e.g. first stamp after the slot was evicted) while + // the host already holds a 3000-element list: the slice would start at + // index 99 — below the kept window (baseOffset 952) and would be a + // 2901-id header — so the carrier stays off and the full form rides + learn(inH, "100", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(3000)); + const h = pfChannel.pfRequestHeaders(inH); + assert.equal(h["x-bili-pf-count"], "100"); + assert.equal(h["x-bili-pf-tail-ids"], undefined, "slice reaches below the window ⇒ count-only"); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 3000), false, "no carrier ⇒ the full form rides (pins fine; server restamps next turn)"); + // one 2xx later the stamp tracks the real size and tail-only resumes + learn(inH, "3000", 200, "incremental"); + pfChannel.pfCaptureIds(sid, gen(3001)); + assert.equal(pfChannel.pfRequestHeaders(inH)["x-bili-pf-tail-ids"], "m3000,m3001", "steady state resumes tail-only"); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 3001), true); +}); + +test("pf-channel: tail payload over the byte budget ⇒ no tail header AND no suppression", () => { + const sid = "sid-mode-budget-1"; + const inH = { [CONVO]: sid }; + learn(inH, "1", 200, "incremental"); + // one turn adds 500 cores × ~29 chars ⇒ encoded tail ≈ 15 KB > the 8 KB + // header budget (a 431 would kill the WHOLE model request) + pfChannel.pfCaptureIds(sid, Array.from({ length: 500 }, (_, i) => `${"a".repeat(28)}${i}`)); + const h = pfChannel.pfRequestHeaders(inH); + assert.equal(h["x-bili-pf-count"], "1"); + assert.equal(h["x-bili-pf-tail-ids"], undefined, "over budget ⇒ the whole request must not 431"); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 500), false, "no carrier ⇒ full form rides"); + // within budget again ⇒ steady state resumes + pfChannel.pfCaptureIds(sid, gen(500)); + assert.equal(pfChannel.pfRequestHeaders(inH)["x-bili-pf-tail-ids"], gen(500).join(","), "short ids ⇒ the tail goes out"); + assert.equal(pfChannel.pfSuppressFullMsgIds(sid, 500), true); +}); + +// #7064 §8 phase 2: pfIdTailForm — the client-side tail derivation for +// lanes WITHOUT a watermark plane (anthropic / responses egress). +{ + const learnMode = (sid: string, mode: "incremental" | "full"): void => + learn({ [CONVO]: sid }, null, 200, mode); + test("pfIdTailForm: bootstrap stores the baseline and sends full", () => { + learnMode("tail-a", "incremental"); + assert.equal(pfChannel.pfIdTailForm("tail-a", ["p1", "p2", "p3"]), undefined, "no previous list ⇒ full (bootstrap)"); + assert.deepEqual(pfChannel.pfIdTailForm("tail-a", ["p1", "p2", "p3", "p4"]), ["p3", "p4"], "next turn: anchor + new core"); + assert.deepEqual(pfChannel.pfIdTailForm("tail-a", ["p1", "p2", "p3", "p4", "p5"]), ["p4", "p5"], "rolling forward"); + }); + test("pfIdTailForm: shrink and rewrite both reanchor with the full list", () => { + learnMode("tail-b", "incremental"); + pfChannel.pfIdTailForm("tail-b", ["q1", "q2", "q3"]); + assert.equal(pfChannel.pfIdTailForm("tail-b", ["q1"]), undefined, "shrink (host compacted) ⇒ full reanchor"); + assert.deepEqual(pfChannel.pfIdTailForm("tail-b", ["q1", "q9"]), ["q1", "q9"], "…and the shortened list becomes the new baseline"); + pfChannel.pfIdTailForm("tail-b", ["q1", "q9", "q5"]); + assert.equal(pfChannel.pfIdTailForm("tail-b", ["r1", "q9", "q5"]), undefined, "prefix rewrite ⇒ full reanchor"); + }); + test("pfIdTailForm: mode 'full' or absent ⇒ always full; a 4xx clears the learned state", () => { + learnMode("tail-c", "full"); + pfChannel.pfIdTailForm("tail-c", ["s1"]); + assert.equal(pfChannel.pfIdTailForm("tail-c", ["s1", "s2"]), undefined, "mode 'full' ⇒ never tail"); + learnMode("tail-d", "incremental"); + pfChannel.pfIdTailForm("tail-d", ["u1", "u2"]); + assert.deepEqual(pfChannel.pfIdTailForm("tail-d", ["u1", "u2", "u3"]), ["u2", "u3"]); + learn({ [CONVO]: "tail-d" }, null, 400); + pfChannel.pfIdTailForm("tail-d", ["u1", "u2"]); + assert.equal(pfChannel.pfIdTailForm("tail-d", ["u1", "u2", "u3"]), undefined, "after a 4xx the baseline + mode are cleared ⇒ full again"); + }); +} diff --git a/tests/plugin-fork.test.ts b/tests/plugin-fork.test.ts index 8e555df402..5de5be3472 100644 --- a/tests/plugin-fork.test.ts +++ b/tests/plugin-fork.test.ts @@ -25,6 +25,10 @@ process.env.NODE_ENV = "test"; const testRoot = mkdtempSync(join(tmpdir(), "bili-fork-")); test.after(() => rmSync(testRoot, { recursive: true, force: true })); for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = testRoot; +// #6480 flipped the absent-default to pure-sqlite (JSON envelope writer +// off) — this suite's cold-disk-restore tests read the legacy JSON +// content-store files off disk, so pin the dual-write writer explicitly. +process.env.BILI_STORAGE_JSON_WRITER = "on"; const hash = (v: unknown) => createHash("sha256").update(JSON.stringify(v)).digest("hex"); interface SnapshotResponse { diff --git a/tests/plugin-render-endpoint.test.ts b/tests/plugin-render-endpoint.test.ts index 345803d0f7..41193891d8 100644 --- a/tests/plugin-render-endpoint.test.ts +++ b/tests/plugin-render-endpoint.test.ts @@ -43,7 +43,7 @@ async function harness(gateOn: boolean): Promise { for (const key of ["XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[key] = dir; process.env.BILI_STORAGE_UNIFIED = "full"; if (gateOn) process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY = "1"; - else delete process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY; + else process.env.BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY = "0"; _resetSessionsForTest(); _resetPluginStateForTest(); resetToolRingForTest(); @@ -123,7 +123,7 @@ test("probe advertises serverHistory with the gate state (capability discovery)" try { const p = await off.probe(); assert.equal(p.status, 200); - assert.equal(p.body.serverHistory, false, "default rollout 0% = capability hidden"); + assert.equal(p.body.serverHistory, false, "gate forced off = capability hidden"); } finally { await off.close(); } diff --git a/tests/resume-miss-timestamp-seed.test.ts b/tests/resume-miss-timestamp-seed.test.ts index c73731bd74..15627830b7 100644 --- a/tests/resume-miss-timestamp-seed.test.ts +++ b/tests/resume-miss-timestamp-seed.test.ts @@ -273,8 +273,13 @@ test("resume-inheritance MISS seeds the timestamp ledger from the closest tracke } finally { setLogCapture(null); store2.closeQuietly(); - // shared singleton hygiene: leave no chain behind for later tests in this file + // shared singleton hygiene: leave no chain behind for later tests in this file. + // "old" is re-imported by the phase-2 hydrate-on-listening from the file, so it + // must be forgotten here too — the next e2e test in this file reuses id "old" + // and a leftover "old:8" (this test's miss-seed content) poisons its tracker + // (#5608 v3 gate test, Windows-CI flake run 44864811218). prefixAffinity.forget("new"); + prefixAffinity.forget("old"); await closeServers([proxy2, upstream], "miss-seed phase 2"); if (prevStampEnv === undefined) delete process.env.BILI_MESSAGE_TIMESTAMPS; else process.env.BILI_MESSAGE_TIMESTAMPS = prevStampEnv; @@ -430,9 +435,12 @@ test("weak-adoption gate: equal-depth replays and truncated subviews start FRESH } finally { setLogCapture(null); store2.closeQuietly(); - // shared singleton hygiene: leave no chain behind for later tests + // shared singleton hygiene: leave no chain behind for later tests. + // "old" was forgotten in phase 1 but re-imported by the phase-2 + // hydrate-on-listening, so forget it again here. prefixAffinity.forget("eq-replay"); prefixAffinity.forget("trunc-resend"); + prefixAffinity.forget("old"); await closeServers([proxy2, upstream], "miss-gate phase 2"); if (prevStampEnv === undefined) delete process.env.BILI_MESSAGE_TIMESTAMPS; else process.env.BILI_MESSAGE_TIMESTAMPS = prevStampEnv; diff --git a/tests/storage-bounded-load.test.ts b/tests/storage-bounded-load.test.ts index f0097dcecb..2d207a6e20 100644 --- a/tests/storage-bounded-load.test.ts +++ b/tests/storage-bounded-load.test.ts @@ -82,7 +82,7 @@ test("ordinary sync cold load reads only unique tail/snapshot seqs and bounded b const store = (shadow as unknown as { sync: UnifiedStore }).sync; t.after(() => { shadow.close(); rmSync(dir, { recursive: true, force: true }); }); seed(store); - const expected = loadLegacyView(allRows(store)); + const expected = loadLegacyView(allRows(store), undefined, store.getStateFields("bounded"), store.getOrphanViews("bounded") ?? {}); const getMessages = store.getMessages.bind(store); const getBlocks = store.getBlocks.bind(store); const requested: number[] = []; @@ -115,7 +115,7 @@ test("ordinary async cold load uses the snapshot reader for each bounded page", const backend = SqliteBackend.open(join(dir, "index.db")); t.after(async () => { await backend.close(); rmSync(dir, { recursive: true, force: true }); }); seed(backend.store); - const expected = loadLegacyView(allRows(backend.store)); + const expected = loadLegacyView(allRows(backend.store), undefined, backend.store.getStateFields("bounded"), backend.store.getOrphanViews("bounded") ?? {}); const readSnapshot = backend.readSnapshot.bind(backend); const offsets: number[] = []; const requested: number[] = []; @@ -172,7 +172,7 @@ test("resident field reads avoid unrelated JSON, messages, refs, and block paylo const store = (shadow as unknown as { sync: UnifiedStore }).sync; t.after(() => { shadow.close(); rmSync(dir, { recursive: true, force: true }); }); seed(store); - const expected = loadLegacyView(allRows(store)); + const expected = loadLegacyView(allRows(store), undefined, store.getStateFields("bounded"), store.getOrphanViews("bounded") ?? {}); for (const key of ["getSessionRow", "getMessages", "getBlocks", "getRefs", "getDeadRefs"] as const) { store[key] = () => { throw new Error(`resident field read called ${key}`); }; } diff --git a/tests/storage-ccr.test.ts b/tests/storage-ccr.test.ts index bb0fabe29c..3c62f861c3 100644 --- a/tests/storage-ccr.test.ts +++ b/tests/storage-ccr.test.ts @@ -141,7 +141,7 @@ test("ccr: shadow parity — live store ingest keeps both digest legs equal (#29 blocks: unified.getBlocks("ccr-shadow-1"), deadRefs: unified.getDeadRefs("ccr-shadow-1"), refs: unified.getRefs("ccr-shadow-1"), - }); + }, unified.getCcrEntries("ccr-shadow-1"), unified.getStateFields("ccr-shadow-1")); assert.equal((view.contentStore?.byRef["m00002"] as { hash?: string } | undefined)?.hash, "sha256-ccr-original", "replayed view lost the CCR ref entry"); assert.equal(view.contentStore?.byHash["sha256-ccr-original"], "ORIGINAL-BODY-XYZ", "replayed view lost the CCR original"); } finally { diff --git a/tests/storage-cleanup-accounting.test.ts b/tests/storage-cleanup-accounting.test.ts index aa81251037..9918bb7b95 100644 --- a/tests/storage-cleanup-accounting.test.ts +++ b/tests/storage-cleanup-accounting.test.ts @@ -33,7 +33,7 @@ function input(id = "target", rich = false): IngestSessionInput { }] : [], refs: rich ? [0, 1].map((ord) => ({ ord, rawId: `m${ord}`, ref: `m0000${ord + 1}`, refNum: ord + 1 })) : [], deadRefs: rich ? [{ ord: 0, ref: "m00003", refNum: 3, recordedAt: 1 }] : [], - ccrEntries: rich ? [0, 1].map((ord) => ({ ord, key: `m0000${ord + 1}`, content, contentHash: sha256Hex(content) })) : [], + ccrEntries: rich ? [0, 1].map((ord) => ({ ord, key: `m0000${ord + 1}`, content, contentHash: sha256Hex(content), origContent: content, origHash: sha256Hex(content) })) : [], }; } diff --git a/tests/storage-engine-proofs.test.ts b/tests/storage-engine-proofs.test.ts index 65bbe2aaec..5a88510f88 100644 --- a/tests/storage-engine-proofs.test.ts +++ b/tests/storage-engine-proofs.test.ts @@ -156,7 +156,7 @@ function unifiedRows(store: UnifiedStore, id: string): ReturnType { const migrated = UnifiedStore.open(dbPath); try { const ver = (migrated as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "open() runs migrateSchema: v3 → v11"); + assert.equal(ver.user_version, 12, "open() runs migrateSchema: v3 → v12"); migrated.setKv("probe", "alive", true); assert.deepEqual(migrated.getKv("probe", "alive"), true); assert.deepEqual(migrated.listSessionIds(), [], "existing v3 content untouched"); @@ -184,7 +184,7 @@ test("v3 databases migrate to v11 in place, idempotently", async () => { // Reopening a v11 database is a no-op (idempotent migration). const again = UnifiedStore.open(dbPath); const ver2 = (again as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 11); + assert.equal(ver2.user_version, 12); again.close(); } finally { fs.rmSync(tmp, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); diff --git a/tests/storage-ingest-watermark.test.ts b/tests/storage-ingest-watermark.test.ts index 6ab6229b19..b6fb399b78 100644 --- a/tests/storage-ingest-watermark.test.ts +++ b/tests/storage-ingest-watermark.test.ts @@ -591,7 +591,7 @@ test("watermark: sampled verification still catches drift through the delta path } }); -test("watermark: v4 database migrates to v11 additively; newer user_version no-ops", async () => { +test("watermark: v4 database migrates to v12 additively; newer user_version no-ops", async () => { const dir = tmpdir("bili-wm-migrate-"); try { const dbPath = path.join(dir, "index.db"); @@ -620,7 +620,7 @@ test("watermark: v4 database migrates to v11 additively; newer user_version no-o const store = UnifiedStore.open(dbPath); try { const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "open() migrates v4 → v11"); + assert.equal(ver.user_version, 12, "open() migrates v4 → v12"); const row = store.getSessionRow("v4-sess"); assert.ok(row, "legacy session row survives the migration"); assert.equal(row!.lastIngestedSeq, null, "legacy rows read back with a NULL watermark (≡ full ingest)"); @@ -658,7 +658,7 @@ test("watermark: v4 database migrates to v11 additively; newer user_version no-o const older = UnifiedStore.open(dbPath); try { const ver2 = (older as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 11, "newer user_version is left untouched"); + assert.equal(ver2.user_version, 12, "newer user_version is left untouched"); assert.ok(older.getSessionRow("v4-sess"), "rows stay readable"); } finally { older.close(); diff --git a/tests/storage-mysql-seam.test.ts b/tests/storage-mysql-seam.test.ts index ef7104b77e..003305df5b 100644 --- a/tests/storage-mysql-seam.test.ts +++ b/tests/storage-mysql-seam.test.ts @@ -123,6 +123,9 @@ class StubMysqlBackend implements StorageBackend { getBlocks: async (id, _opts) => (row(id)?.blocks ?? []).map((b) => ({ ...b, summaryHash: sha256(b.summary), oneHash: b.one === null ? null : sha256(b.one) })), getDeadRefs: async (id) => row(id)?.deadRefs ?? [], getRefs: async (id) => row(id)?.refs ?? [], + getCcrEntries: async () => [], + getOrphanViews: async () => null, + getStateFields: async () => null, }; return read(view); } @@ -165,6 +168,12 @@ class StubMysqlBackend implements StorageBackend { getCcrEntries(sessionId: string): Promise { return Promise.resolve(this.rows.get(sessionId)?.ccrEntries ?? []); } + getOrphanViews(sessionId: string): Promise | null> { + return Promise.resolve(null); + } + getStateFields(sessionId: string): Promise | null> { + return Promise.resolve(null); + } // No watermark on the stub → every record takes the full-ingest path // (deterministic seeding; the delta probe would need boundary state). ingestWatermark(): Promise { diff --git a/tests/storage-null-identity.test.ts b/tests/storage-null-identity.test.ts index 5f2ad9f1a5..fe6ad2d5e7 100644 --- a/tests/storage-null-identity.test.ts +++ b/tests/storage-null-identity.test.ts @@ -45,14 +45,14 @@ test("v8 identity migration can finish with a JSON-null content row", () => { const migrated = migrateSchema(db); assert.equal(migrated.from, 7); - assert.equal(migrated.to, 11); + assert.equal(migrated.to, 12); const row = db.prepare("SELECT identity, content_hash FROM messages WHERE session_id = ?").get("null-session") as { identity: string; content_hash: string }; assert.equal(row.identity, hash); assert.equal(row.content_hash, hash); const content = db.prepare("SELECT bytes, refcount FROM content WHERE hash = ?").get(hash) as { bytes: Uint8Array; refcount: number }; assert.equal(Buffer.from(content.bytes).toString(), raw); assert.equal(content.refcount, 1); - assert.equal(migrateSchema(db).from, 11); + assert.equal(migrateSchema(db).from, 12); } finally { db?.close(); if (previousZstd === undefined) delete process.env.BILI_PERSIST_ZSTD; diff --git a/tests/storage-orphan-representable.test.ts b/tests/storage-orphan-representable.test.ts index 4d2cc9a443..7c37e288f6 100644 --- a/tests/storage-orphan-representable.test.ts +++ b/tests/storage-orphan-representable.test.ts @@ -84,7 +84,7 @@ test("#4216: post-reap saves keep advancing (no freeze) and cold loads are fresh blocks: store.getBlocks("sess"), deadRefs: store.getDeadRefs("sess"), refs: store.getRefs("sess"), - }); + }, undefined, store.getStateFields("sess"), store.getOrphanViews("sess") ?? {}); const viewState = view.state as unknown as { blocks: { blockId: string }[] }; assert.equal(viewState.blocks.length, 2, "reaped block does NOT resurrect"); assert.ok(view.blockContents.has("b3"), "orphan original survives for decompress"); @@ -100,7 +100,7 @@ test("#4216: post-reap saves keep advancing (no freeze) and cold loads are fresh blocks: store.getBlocks("sess"), deadRefs: store.getDeadRefs("sess"), refs: store.getRefs("sess"), - }); + }, undefined, store.getStateFields("sess"), store.getOrphanViews("sess") ?? {}); assert.ok(view2.blockContents.has("b3"), "trimmed-away journal entry survives the merge"); store.close(); diff --git a/tests/storage-reap-journal.test.ts b/tests/storage-reap-journal.test.ts index 2f965f6e30..f669949df4 100644 --- a/tests/storage-reap-journal.test.ts +++ b/tests/storage-reap-journal.test.ts @@ -85,7 +85,7 @@ test("reap journal: reaped block does not resurrect after restart; orphan conten deadRefs: reopened.getDeadRefs("sess-reap"), refs: reopened.getRefs("sess-reap"), }; - const view = loadLegacyView(rows) as LegacySessionLike; + const view = loadLegacyView(rows, undefined, undefined, reopened.getOrphanViews("sess-reap") ?? {}) as LegacySessionLike; const state = view.state as { blocks: CompressionBlock[] }; assert.equal(state.blocks.length, 1, "reaped block must NOT resurrect into state.blocks"); assert.equal(state.blocks[0].blockId, "b1"); @@ -114,7 +114,7 @@ test("reap journal: idempotent re-reap and no-op for unknown blocks", () => { blocks: store.getBlocks("sess-idem"), deadRefs: store.getDeadRefs("sess-idem"), refs: store.getRefs("sess-idem"), - }) as LegacySessionLike; + }, undefined, undefined, store.getOrphanViews("sess-idem") ?? {}) as LegacySessionLike; assert.equal((view.state as { blocks: CompressionBlock[] }).blocks.length, 1); assert.ok(view.blockContents.has("b1")); store.close(); diff --git a/tests/storage-state-fields.test.ts b/tests/storage-state-fields.test.ts index 501daeab1a..3484b14e39 100644 --- a/tests/storage-state-fields.test.ts +++ b/tests/storage-state-fields.test.ts @@ -47,7 +47,7 @@ test("#4192: sweptOrphanCallIds / pendingOrphanSweepMinRef survive the unified r deadRefs: store.getDeadRefs("sf-1"), refs: store.getRefs("sf-1"), }; - const view = loadLegacyView(rows); + const view = loadLegacyView(rows, undefined, store.getStateFields("sf-1")); // The two fields the hand list dropped (#4192) must round-trip intact. assert.deepEqual((view.state as Record).sweptOrphanCallIds, ["m00042", "m00043"]); assert.equal((view.state as Record).pendingOrphanSweepMinRef, 42); diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index 8e4908ba87..dd1360280e 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -175,7 +175,7 @@ function runCase(engine: SqliteEngineName, name: string): void { deadRefs: store.getDeadRefs(loaded.id), refs: store.getRefs(loaded.id), }; - const view = loadLegacyView(rows); + const view = loadLegacyView(rows, store.getCcrEntries(loaded.id), store.getStateFields(loaded.id)); assert.equal(sessionDigest(view), manifest[name]!.digest, `${engine}/${name}: unified round-trip digest drift`); } finally { store.close(); @@ -239,7 +239,7 @@ test("storage unified: re-ingest is idempotent across the whole corpus (digest s deadRefs: store.getDeadRefs(loaded.id), refs: store.getRefs(loaded.id), }; - assert.equal(sessionDigest(loadLegacyView(rows)), manifest[name]!.digest, `${name}: digest drift after re-ingest`); + assert.equal(sessionDigest(loadLegacyView(rows, store.getCcrEntries(loaded.id), store.getStateFields(loaded.id))), manifest[name]!.digest, `${name}: digest drift after re-ingest`); } finally { store.close(); } @@ -306,7 +306,7 @@ test("storage shadow (BILI_STORAGE_UNIFIED=1): real persist path double-writes a // what the override received, whichever that is. const overridePresent = !!liveTail && liveTail.length > 0 && liveFolded !== true; assert.equal( - sessionDigest(loadLegacyView(rows)), + sessionDigest(loadLegacyView(rows, unified.getCcrEntries(loaded.id), unified.getStateFields(loaded.id))), overridePresent ? recordDigest({ ...written, messages: liveTail, messagesFolded: liveFolded }) : recordDigest(written), @@ -815,7 +815,7 @@ test("schema v2: dev-window v1 database upgrades in place (identity columns adde assert.equal(row!.namespace, "proxy", "namespace backfills with its constant default"); assert.equal(row!.origin, null, "origin backfills NULL"); const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "schema version stamped to 11 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 orphans journal + v7 media map + v8 identity/tombstones + v9 zero-ref collection index + v10 immutable message ts column + v11 FK child indexes)"); + assert.equal(ver.user_version, 12, "schema version stamped to 12 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 orphans journal + v7 media map + v8 identity/tombstones + v9 zero-ref collection index + v10 immutable message ts column + v11 FK child indexes + v12 snapshot-table foundation)"); // v3 backfill: every pre-existing sessions row claims its // (namespace, conversation_id) identity binding. const binding = (store as unknown as { lookupConversation(ns: string, id: string): string | null }).lookupConversation("proxy", "dev-window-1"); @@ -877,7 +877,7 @@ test("schema v10: v9→v10 migration adds messages.ts without backfill; idempote try { const ver = (reopened as unknown as { db: { prepare(q: string): { get(): unknown } } }).db .prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 11, "v9 library migrates to v11"); + assert.equal(ver.user_version, 12, "v9 library migrates to v12"); const rows = reopened.getMessages("v10-mig-1") as { seq: number; ts?: number | null }[]; assert.equal(rows.length, 1, "row survives the migration"); assert.ok(rows[0].ts === null || rows[0].ts === undefined, "pre-existing row keeps NULL (never backfilled)"); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 6d152d5023..167be53db5 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -446,116 +446,4 @@ entries: group: process desc_en: "GC age sweep window in days (#3463, #3901): sessions idle longer than this are archived - one JSON line appended to a single archive log (archive/archive.jsonl under the sessions directory — no per-session file spray), a tombstone row keeps the label and identity, content/media refs are released, and restore is one POST away. 0 disables the age sweep; POST /__bili/storage/gc always runs and takes ?vacuum=full to shrink an existing index.db (auto_vacuum=INCREMENTAL is always on for new writes). The same operations have a CLI layer (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run], plus bili sessions ls/rm/archive/restore (README 'Storage maintenance CLI' section). Env wins over the file tier." desc_zh: "GC 年龄清扫窗口(天,#3463/#3901):闲置超过该时长的会话被归档 —— 以单行 JSON 追加到单一归档日志(会话目录下的 archive/archive.jsonl——不撒每会话小文件),墓碑行保留标签与身份,释放内容/媒体引用,一条 POST 即可恢复。0 关闭年龄清扫;POST /__bili/storage/gc 始终可用并支持 ?vacuum=full 收缩存量 index.db(新写入始终 auto_vacuum=INCREMENTAL)。同一套操作另有 CLI 层(#3928):bili storage gc [--days N] [--vacuum full|incremental] [--dry-run],以及 bili sessions ls/rm/archive/restore (README「存储维护 CLI」一节)。环境变量优先于配置文件层。" - desc_ja: "日単位の GC 年齢スイープウィンドウ (#3463, #3901): これより長くアイドル状態のセッションはアーカイブされる - 単一のアーカイブログ (セッションディレクトリ配下の archive/archive.jsonl — セッションごとのファイルスプレーなし) に 1 行の JSON が追記され、トombストーン行がラベルとアイデンティティを保持し、コンテンツ/メディア参照が解放され、リストアは 1 つの POST のみで可能。0 は年齢スイープを無効にする; POST /__bili/storage/gc は常に実行され、既存の index.db を縮小するには ?vacuum=full を取る (新規書き込みには常に auto_vacuum=INCREMENTAL が有効)。同じ操作に CLI レイヤーがある (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]、および bili sessions ls/rm/archive/restore (README の 'Storage maintenance CLI' セクション)。環境変数がファイル階層より優先される。" - - key: sessions - type: object - default: "{} (all defaults)" - group: process - id: tt.sessions - site_group: proc - desc_en: "Session-table size cap and opt-in garbage collection." - desc_zh: "会话表容量上限与可选垃圾回收。" - desc_ja: "セッションテーブルサイズ上限とオプトインによるゴミ収集。" - site_en: "sessions — max 256 (LRU) · gc { enabled false, maxAgeDays 7, maxTokens 1000000, intervalMs 3600000 } — GC is opt-in; user data is never pruned silently." - site_zh: "sessions —— max 256 (LRU) · gc { enabled false, maxAgeDays 7, maxTokens 1000000, intervalMs 3600000 } —— GC 为可选:用户数据不会被悄悄清理。" - - key: sessions.max - type: number - default: "256" - env: BILI_MAX_SESSIONS - group: process - desc_en: "In-memory session table cap (LRU eviction of idle sessions)." - desc_zh: "内存会话表上限(空闲会话 LRU 逐出)。" - desc_ja: "メモリ内セッションテーブル上限 (アイドルセッションの LRU 退避)。" - - key: sessions.warmMax - type: number - default: "0" - env: BILI_SESSION_WARM_MAX - group: process - desc_en: "Bounded startup (#2803/#3050): max most-recently-saved sessions hydrated at boot before listening; older state reloads on demand. Default 0 (no eager warm set — first touch pays one on-demand load; set 8 to restore the eager behavior). Capped by sessions.max." - desc_zh: "有界启动(#2803/#3050):启动监听前预热加载的最近保存会话数上限,更早的状态按需重新加载。默认 0(不预热——首次访问按需加载一次;设 8 恢复预热行为)。受 sessions.max 封顶。" - desc_ja: "有界な起動 (#2803/#3050): リスニング前に起動時に水合される最大最近保存セッション数; 古い状態はオンデマンドで再ロードされる。デフォルト 0 (急激なウォームセットなし — 最初のタッチが 1 回のオンデマンドロードを支払う; 急激な動作を復元するには 8 を設定)。sessions.max で上限が設定される。" - - key: sessions.warmMaxBytes - type: number - default: "67108864" - env: BILI_SESSION_WARM_MAX_BYTES - group: process - desc_en: "Decoded-byte budget for the boot warm set (#2803). Counts decoded size, not compressed on-disk bytes, so zstd archives cannot sneak past the RAM bound. A single file over the budget is never warmed." - desc_zh: "启动预热集的解码字节预算(#2803)。按解码后大小计而非磁盘压缩字节,zstd 归档无法绕过内存上限;单文件超预算则永不预热。" - desc_ja: "起動ウォームセットのデコード済みバイト予算 (#2803)。ディスク上の圧縮バイトではなくデコード済みサイズをカウントするため、zstd アーカイブが RAM 境界をすり抜けることはない。予算を超える単一のファイルは決してウォームされない。" - - key: sessions.residentMaxBytes - type: number - default: "4294967296" - env: BILI_SESSION_RESIDENT_MAX_BYTES - group: process - desc_en: "Soft target for estimated resident session bytes (#2803; raised from 512 MiB to 4 GiB in #5779 — single sessions routinely exceed half a GiB and ping-ponged against the smaller budget forever). Active/in-flight state stays pinned above it; when over target, idle state discharges back to disk as requests complete. Lower it if your machine has little RAM." - desc_zh: "驻留会话估算字节的软目标(#2803;#5779 从 512 MiB 提高到 4 GiB——单个会话经常就超过半 GiB,旧预算下会被反复逐出、永远驻留不下来)。活跃/在途状态始终固定保留;超过目标时,空闲状态随请求完成回写磁盘释放。内存小的机器可调低。" - desc_ja: "推定リザident セッションバイトのソフトターゲット (#2803; #5779 で 512 MiB から 4 GiB に引き上げ — 単一のセッションは通常 512 MiB を超え、より小さい予算に対して永遠にピンポンしていた)。アクティブ/進行中の状態はそれより上にピン留めされる; ターゲット超過時、アイドル状態はリクエスト完了時にディスクへ排出される。メモリが少ないマシンではこれを下げる。" - - key: sessions.hardMaxBytes - type: number - default: "(same as `sessions.residentMaxBytes`)" - env: BILI_SESSION_RESIDENT_HARD_MAX_BYTES - group: process - desc_en: "Per-SESSION hard cap on estimated resident bytes (#4788, decision #4792 D1-A). Unlike the pool soft cap — which can only evict OTHER cold sessions and never touches a sole-active session — this bounds ONE session: over the cap at a request-end boundary, its droppable fields (folded-block originals, metadata ledger, the raw tail beyond the bounded view) are forced back to disk even for a pool-exempt session, and the next turn transparently re-reads what it needs via narrow disk reads. Safe drops run at every request boundary as long as the last save has provably landed; the cap adds the per-session call-out (warn, throttled to 60 s). `0` disables the cap check." - desc_zh: "单会话驻留字节的硬上限(#4788,决策 #4792 D1-A)。不同于池软上限——它只能逐出其他冷会话、从不触碰唯一活跃会话——它约束单个会话:在请求结束边界超过上限时,其可丢弃字段(已折叠块的原文、元数据账本、超出有界视图的原始尾巴)会被强制回写磁盘,即使该会话被池逐出豁免;下一轮通过窄盘读透明地重新读取所需内容。只要上一次保存已被证明落盘,安全丢弃就在每个请求边界运行;该上限额外加入单会话告警(warn,限流至 60 秒)。`0` 关闭该检查。" - desc_ja: "セッションごとの推定リザident バイトのハード上限 (#4788, 決定 #4792 D1-A)。プールソフト上限とは異なり — それは他のコールドセッションのみを退避でき、単独アクティブセッションには決して触れない — これは 1 つのセッションを制限する: リクエスト終了境界で上限超過時、そのドロップ可能なフィールド (フォールドブロックのオリジナル、メタデータ台帳、有界ビューを超える生テール) はプール免除セッションであっても強制的にディスクへ戻され、次のターンは狭いディスク読み取りを通じて必要なものを透明に再読み取りする。最後の保存が確実に着地したことが証明される限り、安全なドロップはすべてのリクエスト境界で実行される; 上限はセッションごとのコールアウト (警告、60 秒にスロットリング) を追加する。`0` は上限チェックを無効にする。" - - key: sessions.contentStorePin - type: boolean - default: "false" - env: BILI_SESSION_CONTENT_STORE_PIN - group: process - desc_en: "Escape hatch for the #4788 phase-2 resident drop: keep each session's content store pinned in memory (nothing ever folded away). Default off: after every proven-landed save, idle sessions' content stores are dropped between requests and re-read on demand from the unified index mirror or the sidecar file; sessions whose store has no provably-landed copy (e.g. fresh fork children) stay resident automatically. Set this if you need zero re-read latency at the cost of RAM." - desc_zh: "#4788 二期驻留丢弃的逃生门:把会话内容存储钉在内存(永不折叠)。默认关闭:每次已证明落盘的保存之后,空闲会话的内容存储在请求间隙被释放,需要时从统一索引镜像或伴生文件按需重读;没有可证明落盘副本的会话(如新建 fork 子会话)自动保持驻留。需要零重读延迟且愿意用内存换时可设此项。" - desc_ja: "#4788 フェーズ2の常駐ドロップのエスケープハッチ: 各セッションのコンテンツストアをメモリに固定する(折りたたみは行われない)。デフォルトはオフ: 検証済みの保存が完了するたびに、アイドル状態のセッションのコンテンツストアはリクエスト間でドロップされ、統合インデックスミラーまたはサイドカーファイルからオンデマンドで再読み込みされる。検証済みのコピーがないストアを持つセッション(例: 新しいフォーク子プロセス)は自動的に常駐する。RAMを犠牲にしてゼロの再読み込みレイテンシが必要な場合にこれを設定する。" - - key: sessions.gc.enabled - type: boolean - default: "false" - env: BILI_SESSION_GC - group: process - desc_en: "Opt-in garbage collection of stale persisted sessions (user data is never pruned silently without this flag)." - desc_zh: "过期持久化会话的可选垃圾回收(未开启时用户数据绝不被悄悄清理)。" - desc_ja: "古い永続化セッションのオプトインによるガベージコレクション(このフラグなしでユーザーデータを静かに削除することはない)。" - - key: sessions.gc.maxAgeDays - type: number - default: "7" - env: BILI_SESSION_GC_MAX_AGE_DAYS - group: process - desc_en: "GC age threshold in days." - desc_zh: "GC 年龄阈值(天)。" - desc_ja: "GCの年齢しきい値(日)。" - - key: sessions.gc.maxTokens - type: number - default: "1000000" - env: BILI_SESSION_GC_MAX_TOKENS - group: process - desc_en: "GC token-size threshold per session record." - desc_zh: "GC 单会话记录的 token 大小阈值。" - desc_ja: "セッションレコードごとのGCトークンサイズしきい値。" - - key: sessions.gc.intervalMs - type: number - default: "3600000" - env: BILI_SESSION_GC_INTERVAL_MS - group: process - desc_en: "GC sweep interval." - desc_zh: "GC 清扫间隔。" - desc_ja: "GCスイープ間隔。" - - key: plugin.snapshotCapBytes - type: number - default: "104857600 (0 disables snapshots)" - env: BILI_PUBLIC_SNAPSHOT_CAP_BYTES - group: process - id: tt.snap - site_group: proc - desc_en: "Size cap for fork-API public snapshots served to native plugins." - desc_zh: "提供给原生插件的 fork API 公共快照大小上限。" - desc_ja: "ネイティブプラグインに配信されるフォークAPIパブリックスナップショットのサイズ上限。" - site_en: "plugin.snapshotCapBytes — fork API snapshot cap (default 100 MiB; 0 disables snapshots)." - site_zh: "plugin.snapshotCapBytes —— fork API 快照上限(默认 100 MiB;0 禁用快照)。" - - key: snapshot.shadow - type: boolean - default: "true (on)" - kind: file-only - audience: user - group: process - desc_en: "Snapshot-render shadow observability (#3756 follow-up): with the render lane still rollout-gated, every armed turn computes the would-be snapshot view, classifies the diff against the legacy outbound, and records per-session counters (web panel rows + /__bili/status snapshotShadow aggregate + the snapshot-shadow diagnostics topic). Default ON - it is the fleet-wide dry run whose error rate gates the rollout flip. Env BILI_SNAPSHOT_SHADOW wins. Turn off with false/0/off if you do not want the extra per-turn computation." - desc_zh: "快照渲染影子对账(#3756 后续):渲染车道仍处灰度关闭时,每轮已折叠会话都现算一次快照视图,与 legacy 实际出站比对分类,逐会话记账(web 面板行 + /__bili/status snapshotShadow 汇总 + snapshot-shadow 诊断主题)。默认开 —— 这是全舰队干跑,其错误率决定灰度翻转。环境变量 BILI_SNAPSHOT_SHADOW 优先。不想要每轮额外计算可用 false/0/off 关闭。" - desc_ja: "スナップショットレンダリングのシャドウ観測可能性(#3756のフォローアップ): レンダリングレーンがまだロールアウトゲートされている間、すべてのアームされたターンがスナップショットビューを計算し、レガシーアウトバウンドとの差分を分類し、セッションごとのカウンターを記録する(Webパネル行 + /__bili/status の snapshotShadow 集計 + snapshot-shadow 診断トピック)。デフォルトはオン - これはロールアウト切り替えをゲートするエラー率を持つフリートワイドのドライランである。環境変数 BILI_SNAPSHOT_SHADOW が優先される。追加のターンごとの計算を望まない場合は false/0/off でオフにする。" + desc_ja: "日単位の GC 年齢スイープウィンドウ (#3463, #3901): これより長くアイドル状態のセッションはアーカイブされる - 単一のアーカイブログ (セッションディレクトリ配下の archive/archive.jsonl — セッションごとのファイルスプレーなし) に 1 行の JSON が追記され、トombストーン行がラベルとアイデンティティを保持し、コンテンツ/メディア参照が解放され、リストアは 1 つの POST のみで可能。0 は年齢スイープを無効にする; POST /__bili/storage/gc は常に実行され、既存の index.db を縮小するには ?vacuum=full を取る (新規書き込みには常に auto_vacuum=INCREMENTAL が有効)。同じ操作に CLI レイヤーがある (#3928): bili storage gc [--days N] [--vacuum full|incremental] [--dry-run]、および bili sessions ls/rm/archive/restore (README の 'Storage maintenance CLI' セクション)。環境変数がファイル階層より優先される。" \ No newline at end of file diff --git a/website/config-reference/rollout.yaml b/website/config-reference/rollout.yaml index 70c94a109f..6588722885 100644 --- a/website/config-reference/rollout.yaml +++ b/website/config-reference/rollout.yaml @@ -66,14 +66,41 @@ entries: desc_zh: "身份信道采集的灰度门(#5825):当安装级开关(BILI_IDENTITY_HARVEST)未设置时,允许仅由本灰度键激活回环专属的转录/SQLite 采集(claude-code / codebuddy / mcode / goose / qwen 车道);两者都不设则永不读客户端存储。owner 裁定 2026-11-07:本灰度键与 rollout.features.pluginLaneServerHistory(它管已接纳 id 的行钉住)刻意分离——任一使能方可激活采集。默认 0%,小流量观察退化率后再抬。见 identity-channels.zh-CN.md。" desc_ja: "IDチャネル収穫コホートゲート (#5825):インストールごとの環境スイッチなしで、ループバック専用のトランスクリプト/sqlite収穫(`src/identity-harvest.ts`: claude-code / codebuddy / mcode / goose / qwen レーン)を許可する。インストールごとのスイッチ(`BILI_IDENTITY_HARVEST`)もこのキーも設定されていない場合、クライアントストレージは一切読み取られない。オーナー裁定 2026-11-07:このコホートキーは `rollout.features.pluginLaneServerHistory`(承認済みIDの行固定を管理)とは意図的に分離されている — いずれかのイネーブラーが収穫を有効化できる。デフォルト 0%;小規模コホートでの degrade warn-once レートの観察後に引き上げる。`identity-channels.md` を参照。" - key: rollout.features.pluginLaneServerHistory + type: "boolean | number" + default: "100 (on, GA)" + default_zh: "100(开,已 GA)" + env: BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY + group: behavior + desc_en: "Plugin-lane server-owned history (#4491): for enabled installs, (a) plugin-lane requests may carry an internal `biliMsgIds` array parallel to `messages` that pins message identity to client-supplied ids (always stripped before upstream forwarding; malformed arrays are ignored with a warning, id-poisoning shapes hard-fail on the incremental lane); (b) the loopback-only `POST /__bili/plugin/render` endpoint serves the outbound prefix rendered from the unified transcript (folded regions collapse to summary anchors; any desync degrades to a non-2xx so the shim falls back to its local history); (c) the incremental-lane probe advertises `serverHistory`. Default 100% (GA, owner ruling 2026-11-13); set 0 to roll an install back to the pre-GA state." + desc_zh: "插件车道服务端正史(#4491):对已放量的安装,(a) 插件车道请求可携带与 `messages` 平行的内部字段 `biliMsgIds`,把消息身份钉在客户端提供的 id 上(转发上游前一律剥掉;格式错误的数组告警后忽略,身份投毒形态在增量车道直接硬失败);(b) 仅回环可达的 `POST /__bili/plugin/render` 端点从统一存储渲染出站前缀(折叠区收敛为摘要锚点;任何失步都返回非 2xx,shim 退回本地历史);(c) 增量车道探测响应通告 `serverHistory`。默认 100%(GA,owner 裁定 2026-11-13);设 0 可把安装回滚到 GA 前状态。" + desc_ja: "プラグインレーンのサーバー所有履歴 (#4491):有効化されたインストールにおいて、(a) プラグインレーンのリクエストは `messages` と並行して内部 `biliMsgIds` 配列を持ち、メッセージIDをクライアント提供のIDに固定できる(上流転送前に常に除去;不正な配列は警告付きで無視、ID汚染形状は増分レーンでハードフェイル);(b) ループバック専用の `POST /__bili/plugin/render` エンドポイントは、統合トランスクリプトからレンダリングされた送信プレフィックスを提供する(折り畳み領域はサマリアンカーに縮小;デシンクは非2xxにデグレードし、シムはローカル履歴にフォールバック);(c) 増分レーンのプローブは `serverHistory` をアドバタイズする。デフォルト 100%(GA、オーナー裁定 2026-11-13);0 に設定するとインストールは GA 前の状態に戻る。" + - key: rollout.features.pluginMsgIdIncremental + type: "boolean | number" + default: "100 (on, GA)" + default_zh: "100(开,已 GA)" + env: BILI_ROLLOUT_PLUGIN_MSG_ID_INCREMENTAL + group: behavior + desc_en: "Incremental msgid pin channel (#7064 §8, owner rulings 2026-11-18/19): steady plugin-lane turns declare their message ids in TAIL FORM — the bounded `x-bili-pf-tail-ids` header (anchor + new ids, ≤ 2048 ids, `?` placeholder slots) or a short body id array — instead of the FULL list; the server re-stamps the prefix ids from its own ledger live rows (ingest is INSERT OR IGNORE on (session_id,seq) — a position's msg_id is written once, never overwritten). Acceptance (2026-11-19 form): the declared tail [T-L, T-1] must match the ledger overlap segment row-by-row (any drift degrades to `full`, never 400); ultra-long sessions (> 2048 ids) pin a truncated tail the same way — the longer overlap proves MORE. Only snapshot rounds (empty ledger / post-fold re-anchor) send the full form; full form always wins when both arrive. The response carries `x-bili-msgid-mode: incremental|full` (cleared on any 4xx; one learning point next to x-bili-pf-seq). Any mismatch self-heals within one round. Default 100%; set 0 to roll back to full-list pinning (byte-identical to pre-feature). Requires pluginLaneServerHistory. See docs/session-identity-resolution.md." + desc_zh: "增量 msgid pin 通道(#7064 §8,owner 裁定 2026-11-18/19):稳态插件轮用**尾形态**申报消息 id——有界的 `x-bili-pf-tail-ids` 头(锚点+新增,≤ 2048 个,`?` 占位槽)或短 body id 数组——替代全量清单;服务端从自己账本的 live 行重盖前缀 id(ingest 是 INSERT OR IGNORE on (session_id,seq)——位置上的 msg_id 只写一次,永不改写)。接受规则(2026-11-19 定型):声明的尾段 [T-L, T-1] 与账本重叠段逐行相等(任一行漂移⇒降级 full,永不 400);超长会话(>2048)截断的尾清单同样可 pin——重叠段更长、证明更强。只有快照轮(账本空/折叠后重锚)发全量形态;全量优先于尾形态同时到达。响应携带 `x-bili-msgid-mode: incremental|full`(4xx 即清,与 x-bili-pf-seq 同一学习点)。任何失配一轮内自愈。默认 100%;设 0 回滚到全量清单 pin(与特性前字节一致)。需先开 pluginLaneServerHistory。见 session-identity-resolution.zh-CN.md。" + desc_ja: "増分 msgid pin チャネル(#7064 §8、オーナー裁定 2026-11-18/19):定常プラグインターンはメッセージ ID を**テール形態**で申告する — 有界な `x-bili-pf-tail-ids` ヘッダー(アンカー+新規 ID、≤ 2048 個、`?` プレースホルダ)または短い body ID 配列 — 全リストを置き換える。サーバーは自前の台帳 live 行からプレフィックス ID を再スタンプする(ingest は INSERT OR IGNORE on (session_id,seq) — 位置の msg_id は一度だけ書き込まれ、決して上書きされない)。受理規則(2026-11-19 定石):申告されたテール [T-L, T-1] と台帳の重複区間が逐行一致すること(1行でもずれれば `full` にデグレード、400 はしない);超長セッション(> 2048)も切り詰められたテールで同じく pin できる — 重複区間が長いほど証拠は強い。スナップショットラウンド(台帳空/折り畳み後の再アンカー)のみ全形態を送る;両方到着時は全形態が優先。レスポンスは `x-bili-msgid-mode: incremental|full` を載せる(4xx でクリア、x-bili-pf-seq と同じ学習ポイント)。不一致は 1 ラウンドで自己修復。デフォルト 100%;0 で全リスト pin にロールバック(機能前とバイト一致)。pluginLaneServerHistory が必要。session-identity-resolution.md 参照。" + - key: rollout.features.egressPlane type: "boolean | number" default: "0 (off)" default_zh: "0(关)" - env: BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY + env: BILI_ROLLOUT_EGRESS_PLANE + group: behavior + desc_en: "The egress plane — ONE switch for the whole new path (#7398/#7439; replaces the retired egressOpLog/egressServeLog pair). ON: every wire lane (OpenAI/Anthropic/Responses/Google) records its outbound conversation view as sparse ops (I/D/U) into the egress_state/egress_flow dual tables — rows are SELF-CONTAINED (message bytes ride the row; no content CAS, ruling 7439-1) — and serves the outbound core-message portion back from the op log, byte-identical by construction (the row bytes are the pipeline's own final core JSON); pinned plugin lanes additionally take server authority — the kernel input is replayed from the flow (rulings 7439-3/5). A bounded backward ladder recovers to the newest healthy log point; ANY unrebuildable row (missing/empty bytes, unknown role), corrupt log, or probe failure degrades that turn to the legacy chain — never a partial swap, never client-as-authority. The system block stays live-built in both paths (prefix-cache anchor discipline). SQLite lane only; title-gen/compaction side requests excluded. Default 0 = off: legacy chain, zero rows, byte-identical to pre-feature." + desc_zh: "egress 平面——整条新链路的**唯一开关**(#7398/#7439;取代已退役的 egressOpLog/egressServeLog 双键)。开:四条 wire 车道(OpenAI/Anthropic/Responses/Google)每轮把出站会话视图记为稀疏 op(I/D/U)写入 egress_state/egress_flow 双表——行**自包含**(消息字节随行走,不经内容 CAS,裁定 7439-1)——并从 op 日志回放重建出站核心消息段,按构造字节一致(行内字节就是管线自己的最终 core JSON);pinned 插件车道另获服务端权威——内核输入从流水回放(裁定 7439-3/5)。有界回溯阶梯恢复到最近健康日志点;任何不可重建行(缺/空字节、未知 role)、日志损坏或探针失败,整轮降级 legacy 链路——绝不部分替换、绝不客户端为准。system 块两条路径都保持 live 构建(前缀缓存锚点纪律)。仅 SQLite 车道;标题生成/压缩侧路请求排除。默认 0 = 关:legacy 链路,零行,与特性前字节一致。" + desc_ja: "egress プレーン — 新しい経路全体の**単一スイッチ**(#7398/#7439;退役した egressOpLog/egressServeLog の二キーを置き換え)。ON:全 wire レーン(OpenAI/Anthropic/Responses/Google)が毎ターン送信会話ビューをスパース op(I/D/U)として egress_state/egress_flow デュアルテーブルに記録 — 行は**自己完結**(メッセージバイトが同行する;コンテンツ CAS を経由しない、裁定 7439-1)— 送信コアメッセージ部分を op ログから復元し再構築、構造によりバイト一致(行内バイトはパイプライン自身の最終 core JSON);ピン留め済みプラグインレーンは加えてサーバー権威を取得 — カーネル入力をフローからリプレイ(裁定 7439-3/5)。有界の遡行ラダーで最新の健全ログ点に回復;再構築不能行(バイト欠落/空、未知 role)、ログ破損、プローブ失敗のいずれかで**そのターン全体**をレガシーチェーンにデグレード — 部分的な swap は絶対にしない、クライアントを権威にしない。system ブロックは両経路でライブ構築のまま(プレフィックスキャッシュアンカー規律)。SQLite レーンのみ;タイトル生成/圧縮のサイドリクエストは除外。デフォルト 0 = オフ:レガシー経路、0 行、機能前とバイト一致。" + - key: rollout.features.pluginDeltaResend + type: "boolean | number" + default: "0 (off)" + default_zh: "0(关)" + env: BILI_ROLLOUT_PLUGIN_DELTA_RESEND group: behavior - desc_en: "Plugin-lane server-owned history (#4491): for enabled installs, (a) plugin-lane requests may carry an internal `biliMsgIds` array parallel to `messages` that pins message identity to client-supplied ids (always stripped before upstream forwarding; malformed arrays are ignored with a warning, id-poisoning shapes hard-fail on the incremental lane); (b) the loopback-only `POST /__bili/plugin/render` endpoint serves the outbound prefix rendered from the unified transcript (folded regions collapse to summary anchors; any desync degrades to a non-2xx so the shim falls back to its local history); (c) the incremental-lane probe advertises `serverHistory`. Default 0% — no behavior change until rolled out." - desc_zh: "插件车道服务端正史(#4491):对已放量的安装,(a) 插件车道请求可携带与 `messages` 平行的内部字段 `biliMsgIds`,把消息身份钉在客户端提供的 id 上(转发上游前一律剥掉;格式错误的数组告警后忽略,身份投毒形态在增量车道直接硬失败);(b) 仅回环可达的 `POST /__bili/plugin/render` 端点从统一存储渲染出站前缀(折叠区收敛为摘要锚点;任何失步都返回非 2xx,shim 退回本地历史);(c) 增量车道探测响应通告 `serverHistory`。默认 0%,放量前无行为变化。" - desc_ja: "プラグインレーンのサーバー所有履歴 (#4491):有効化されたインストールにおいて、(a) プラグインレーンのリクエストは `messages` と並行して内部 `biliMsgIds` 配列を持ち、メッセージIDをクライアント提供のIDに固定できる(上流転送前に常に除去;不正な配列は警告付きで無視、ID汚染形状は増分レーンでハードフェイル);(b) ループバック専用の `POST /__bili/plugin/render` エンドポイントは、統合トランスクリプトからレンダリングされた送信プレフィックスを提供する(折り畳み領域はサマリアンカーに縮小;デシンクは非2xxにデグレードし、シムはローカル履歴にフォールバック);(c) 増分レーンのプローブは `serverHistory` をアドバタイズする。デフォルト 0% — ロールアウトされるまで挙動は変わらない。" + desc_en: "Delta resend — the tail-slice form of the full-resend watermark lane: steady plugin-lane turns send x-bili-pf-delta: 1 + body.messages = ONLY the new elements (L = T − stampT, no prefix); the server rebuilds the view from the stashed prefix cores plus the parsed tail (same hit provenance as the full lane). ANY unsatisfiable claim (count/body/tail-ids width mismatch, anchor-chain break after a known list, rollout off) returns 400 pf_delta_reanchor + x-bili-msgid-mode: full and the client resends that turn in full form — one-round convergence, never a silent misparse. OFF by default: delta bodies are unprocessable by servers that do not ship this arm, so the key stays 0 until client send-sides exist. See technical-notes.md “Delta resend”." + desc_zh: "Delta 重发——全量重发水位快路径的尾切片形态:稳态轮发 x-bili-pf-delta: 1 + body.messages = **只有**新元素(L = T − stampT,无前置);服务端用钉存前缀核 + 解析出的尾部重建视图(与全量车道同源的命中可证)。任何不成立的声明(count/body/tail-ids 宽度不符、已知清单后锚链断裂、灰度关)返回 400 pf_delta_reanchor + x-bili-msgid-mode: full,客户端当轮改发全量形态——一轮收敛,绝不静默错解。默认**关**:delta body 对未上车的服务端不可处理,故保持 0 直到客户端发送端就位。见 technical-notes.zh-CN.md「Delta 重发」。" + desc_ja: "Delta 再送 — フルリセンドウォーターマークレーンのテールスライス形態:定常プラグインレーンのターンは `x-bili-pf-delta: 1` + body.messages = 新要素のみ(L = T − stampT、プレフィックスなし)を送る;サーバーはスタッシュ済みプレフィックスコアとパース済みテールからビューを再構築する(フルレーンと同一のヒット由来)。満たせない主張(count/body/tail-ids 幅の不一致、既知リスト後のアンカーチェーン破断、ロールアウトオフ)は 400 `pf_delta_reanchor` + `x-bili-msgid-mode: full` を返し、クライアントはそのターンをフルフォームで再送する — 1ラウンドで収束、静かな誤パースは絶対にしない。デフォルトオフ:delta body はこのアームを持たないサーバーでは処理できないため、クライアント送信側が実装されるまでキーは 0 のまま。`technical-notes.md` の「Delta resend」を参照。" - key: rollout.features.pluginFullResendWatermark type: "boolean | number" default: "5 (experimental grey)" diff --git a/website/docs/index.html b/website/docs/index.html index a86bfe9fad..2a1369e734 100644 --- a/website/docs/index.html +++ b/website/docs/index.html @@ -302,8 +302,6 @@

Other top-level fields

  • guardrails — pi/omp plugin tool-output + stream text-run guardrails: bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200.
  • persist — session persistence to disk: enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0.
  • storage — unified storage sidecar: unified legacy — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.
  • -
  • sessions — max 256 (LRU) · gc { enabled false, maxAgeDays 7, maxTokens 1000000, intervalMs 3600000 } — GC is opt-in; user data is never pruned silently.
  • -
  • plugin.snapshotCapBytes — fork API snapshot cap (default 100 MiB; 0 disables snapshots).
  • update — self-update channel: registry "npmjs" · checkIntervalMs 180000.
  • Diagnostics & updates
  • diagnostics — local troubleshooting knobs: dumpBody false · dumpReq true · rawDumpDir · dump4xx false (+MaxBytes 2 MiB) · renderNone/noInjectTool/noCompressPrompt false · countTokensPassthrough false · compressProtocol "tools".
  • diff --git a/website/main.js b/website/main.js index f56f126d83..f1ee6a45f8 100644 --- a/website/main.js +++ b/website/main.js @@ -137,8 +137,6 @@ "tt.guardrails": "guardrails — pi/omp plugin tool-output + stream text-run guardrails: bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200.", "tt.persist": "persist — session persistence to disk: enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0.", "tt.storage": "storage — unified storage sidecar: unified legacy — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.", - "tt.sessions": "sessions — max 256 (LRU) · gc { enabled false, maxAgeDays 7, maxTokens 1000000, intervalMs 3600000 } — GC is opt-in; user data is never pruned silently.", - "tt.snap": "plugin.snapshotCapBytes — fork API snapshot cap (default 100 MiB; 0 disables snapshots).", "tt.rollout": "rollout — canary gating: features enable for a percent of installs, bucketed on a stable per-machine UUID; nested base rolls B out only inside A's cohort.", "tt.port": "port — listen port (default 8787).", "tt.host": "host — bind address (default 127.0.0.1; \"::\" is dual-stack, 0.0.0.0 exposes outward).", @@ -396,8 +394,6 @@ "tt.guardrails": "guardrails —— pi/omp 插件工具输出与流式 text-run 守卫:bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200。", "tt.persist": "persist —— 会话落盘持久化:enabled true · zstd false · debounceMs 500 · maxDelayMs 5000 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0。", "tt.storage": "storage —— 统一存储边车:unified legacy —— 文件层选定全 lane 存储模式;设置了 BILI_STORAGE_UNIFIED 时环境变量优先。", - "tt.sessions": "sessions —— max 256 (LRU) · gc { enabled false, maxAgeDays 7, maxTokens 1000000, intervalMs 3600000 } —— GC 为可选:用户数据不会被悄悄清理。", - "tt.snap": "plugin.snapshotCapBytes —— fork API 快照上限(默认 100 MiB;0 禁用快照)。", "tt.rollout": "rollout —— 灰度门控:特性按安装百分比启用,以每台机器稳定的 UUID 分桶;嵌套 base 使 B 只在 A 的入组人群内放量。", "tt.port": "port —— 监听端口(默认 8787)。", "tt.host": "host —— 绑定地址(默认 127.0.0.1;\"::\" 双栈,0.0.0.0 对外暴露)。",