diff --git a/CONFIGURATION.md b/CONFIGURATION.md index fe422b9461..cbe472e301 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -103,6 +103,9 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `rollout.features.base` | string | unset (roll out over everyone) | — | Nested rollout: evaluate this feature only inside the named feature's enabled cohort, so the percent is relative to that population (roll B out to A's users; chains allowed, cycles fail closed). | | `rollout.features.pluginLaneServerHistory` | boolean \| number | 0 (off) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | Plugin-lane server-owned history (#4491): for enabled installs, (a) plugin-lane requests may carry an internal `biliMsgIds` array parallel to `messages` that pins message identity to client-supplied ids (always stripped before upstream forwarding; malformed arrays are ignored with a warning, id-poisoning shapes hard-fail on the incremental lane); (b) the loopback-only `POST /__bili/plugin/render` endpoint serves the outbound prefix rendered from the unified transcript (folded regions collapse to summary anchors; any desync degrades to a non-2xx so the shim falls back to its local history); (c) the incremental-lane probe advertises `serverHistory`. Default 0% — no behavior change until rolled out. | | `rollout.features.pluginFullResendWatermark` | boolean \| number | 5 (experimental grey) | BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK | Full-resend watermark fastpath (#5403/#5418): plugin-lane chat-completions turns whose previous response carried an x-bili-pf-seq stamp assert the resent prefix via the echoed count (structural scan, no full parse); on a hit the pipeline consumes stashed canonical prefix cores plus only the new tail window (byte-identical outbound provenance; ANY doubt falls back to the untouched full-parse path per turn and re-stamps, five credited misses hold the lane for one hour). Requires the client to echo the stamp back as x-bili-pf-count (dsh/pi/opencode/omp do; unadapted lanes never hit). EXPERIMENTAL grey rollout — ON by default for roughly 5% of installs (owner ruling 2026-11-02: small cohort before GA; one-time intro notice in the web UI for every install, tri-state switch in quick config). Force it with `true`, turn off with `false` (or a percent) / env = 0. See TECHNICAL-NOTES.md “Full-resend watermark fastpath”. | +| `rollout.features.watermarkV6MsgId` | boolean \| number | 0 (off) | BILI_ROLLOUT_WATERMARK_V6_MSG_ID | Watermark v6 msgid lane (#5576/#5600): nested grey inside pluginFullResendWatermark, dispatched BEFORE the canonical lane (which then runs only as its else-if). Identity is server-minted: each session world gets a pf_sid (pf_+12hex) issued via the x-bili-pf-session RESPONSE header (the response IS the issuance channel); resolution = echo (must name a live partition) -> probe real tail-ids against the global ledger index (budget 8) -> mint (v6m-arm cold start, P5's one legal full). Hit authority = ledger-width arithmetic against OUR frontier (K = resident element count, never a client echo) + a window-origin pin on element K-1 of our own pf_rows partition (hard pin; weak pin on id-silent frontiers); content bytes are never judged - in-place edits ride frozen cores. Hits append window rows + geom snapshots to the durable pf_* partition store (LSM meta plane: WAL + checksummed checkpoints folding every 32 entries; post-boot hydration from L1+L0; rows deleted ONLY by deleteSession - archive-and-reset doctrine). Shrink vs frontier = compaction class (uncredited passthrough by default, credited under BILI_PF_V6_REWRITE_STRICT=1); system-at-window-head is always credited; sess-change stays uncredited; model switches are non-events (no model gate); all credits feed the shared five-miss/60-min breaker above every lane. Soft plane: store absent => pre-existing lanes run byte-for-byte. Zero retention: hits pin no seed stash. Default 0% - measure in a small cohort first. See TECHNICAL-NOTES.md “Full-resend watermark fastpath” (§v6 msgid lane). | +| `rollout.features.watermarkCanonicalV6` | boolean \| number | 0 (off) | BILI_ROLLOUT_WATERMARK_CANONICAL_V6 | Watermark v6 canonical lane (#5576): nested grey inside pluginFullResendWatermark. When enabled, the watermark's hit verdict stops trusting client-asserted bytes: surviving pre-filters are pure server-side arithmetic over the client-declared ledger list (ahead / width-shape / no-progress / sess-change), then the body converts once with the REAL codec and anchors against stored unified-transcript rows via the canonical chain walk (kernel id fast path → content-sig fallback, min seam run 2). On a hit the outbound prefix = stored rows up to the seam (folded regions behind synthetic summary anchors) + this turn's fresh tail cores; system text re-decodes from the CURRENT head every turn (passthrough by construction — the v5 head-drift gate is retired); model switches are non-events (one logical ledger shared across models); host-side content rewrites break the chain and self-heal through UNcredited legacy turns unless BILI_PF_V6_REWRITE_STRICT=1 credits them toward the breaker. Zero-retention doctrine: canonical outcomes pin NO seed stash. Default 0% — measure in a small cohort before any promotion. See TECHNICAL-NOTES.md “Full-resend watermark fastpath → v6 canonical lane”. | +| `rollout.features.nativeIdentityObservation` | boolean \| number | 0 (off) | BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION | Native-identity observation (#5573 M1): pure-observation collection of client-native session/message identity evidence via a per-client profile table (WorkBuddy x-conversation-id family + X-Compaction headers, codex installation-id / item-id families / previous_response_id rotation + compaction prefaces, Claude Code session header + metadata dual-channel cross-check + continuation preface, opencode ses_ affinity + fork marker + wire-shape assertions, Gemini state-snapshot preface) into the session ext store under nidv:obs (sha1-8 digests only — id values never persist or log in plaintext) plus one debug log line per observed request; compaction hits escalate the line to WARN. Feeds NO decision: request handling, folding, parsing and outbound bytes are untouched, and off-or-unclassified traffic produces zero diff against the golden suites. Default 0% — enable to collect the field data the L2 consumers need (silent fold re-baseline for #5488, fastpath identity binding for #5403/#5418). | | `maskHosts` | boolean | true | BILI_LOG_MASK_HOSTS | Mask non-public target hosts in logs as ; credential headers are always masked regardless. | | `security.adminRelay` | boolean | false | — | Let an embedding host (dsh web today) relay the ENTIRE /__bili/ admin surface through its own entry origin so a remote browser can open the panel (#3373). Fail-closed, file-only. The relayed hop reaches the proxy as a local non-browser client (loopback source + loopback Host, no Origin), so the admin gate stays unchanged; enabling it exposes config editing and credential writes to whoever can reach the host web entry — protect that entry (auth / reverse proxy) first. | | `subagentSplit` | boolean | true | BILI_SUBAGENT_SPLIT | Claude Code subagents get their own session namespace (\|sub:) instead of queueing behind the main session. Anthropic-wire only — on the Responses wire this switch is a no-op (see the `subagentSplit` / BOTH-mechanisms scope note). | @@ -433,6 +436,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `BILI_MITM_HOSTS` | string[] (csv) | unset | — | Launcher channel: MITM whitelist hosts handed to the spawned proxy (distinct from the config-file twin BILI_MITM_DOMAINS). | | `BILI_PLUGIN_AGENT` | boolean-ish | off | — | Spawn channel: marks a plugin-agent process. | | `BILI_IMAGE_RESCALE_MAX_DIM` | integer | 1600 | — | Longest-edge trigger threshold (px) for the egress image downscale pass (#3400); non-numeric or below 128 ignored (default applies). Env-only by design. | +| `BILI_NATIVE_IDENTITY_PROFILES_JSON` | string | unset (built-in table) | — | Native-identity observation (#5573 M1): JSON object of per-client profile rows merged over the built-in evidence table (src/server/native-identity-profiles.ts) — entries ADD new rows or REPLACE a built-in row by name, never remove one (the built-ins are the floor). Same shape as a built-in row (name, matchHeaders, sessionHeaders, msgidHeaders, compactionEdgeHeader, countPatterns, compactionPrefixes, ...). Malformed JSON fails open silently (built-ins only). Only meaningful while rollout.features.nativeIdentityObservation is enabled; feeds no decision. | +| `BILI_PF_V6_REWRITE_STRICT` | boolean | unset (off) | — | Set `1` to CREDIT watermark-v6 misses toward the five-miss breaker instead of the default SELF-HEALING uncredited legacy turn, per lane: in the MSGID lane (`watermarkV6MsgId`) compaction-class shrinks against the stored frontier (`v6m-compact`); in the CANONICAL lane (`watermarkCanonicalV6`) host-side content rewrites (chain breaks: chain-not-started / seam-too-short) — for exotic hosts whose thrash would otherwise ride free legacy turns forever. (The msgid lane’s `v6m-window-system` verdict is ALWAYS credited, independent of this switch.) Default unset/off; pinned by tests/fastpath-pf-msgid.test.ts and tests/fastpath-v6-canonical.test.ts. | @@ -1931,8 +1936,11 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_RESIGN_PASSTHROUGH` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_RESUME_INHERITANCE` | `resumeInheritance` | true | | `BILI_ROLLOUT_` | `rollout.features.enabled` | unset (bucket math decides) | +| `BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION` | `rollout.features.nativeIdentityObservation` | 0 (off) | | `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK` | `rollout.features.pluginFullResendWatermark` | 5 (experimental grey) | | `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 0 (off) | +| `BILI_ROLLOUT_WATERMARK_CANONICAL_V6` | `rollout.features.watermarkCanonicalV6` | 0 (off) | +| `BILI_ROLLOUT_WATERMARK_V6_MSG_ID` | `rollout.features.watermarkV6MsgId` | 0 (off) | | `BILI_SESSION_CONTENT_STORE_PIN` | `sessions.contentStorePin` | false | | `BILI_SESSION_GC` | `sessions.gc.enabled` | false | | `BILI_SESSION_GC_INTERVAL_MS` | `sessions.gc.intervalMs` | 3600000 | @@ -2077,6 +2085,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_SESSION_WARM_MAX_BYTES` | Decoded-byte budget for the boot warm set (#2803; default `67108864` = 64 MiB). Counts DECODED size, not compressed on-disk bytes, so zstd archives cannot sneak past the RAM bound. A single file over the budget is never warmed. File twin: `sessions.warmMaxBytes`. | | `BILI_SESSION_RESIDENT_MAX_BYTES` | Soft target for estimated resident session bytes (#2803; default `536870912` = 512 MiB). Active/in-flight state stays pinned above it (the byte budget limits the idle cache, not active workload); when over target, idle state discharges back to disk as requests complete. File twin: `sessions.residentMaxBytes`. | | `BILI_SESSION_RESIDENT_HARD_MAX_BYTES` | Per-session hard cap on estimated resident memory (#4788, decision #4792 D1-A). Why it exists: the pool soft cap above can only pull OTHER idle sessions out of memory to stay under budget — with a single long-lived session there is nothing left to evict, so one conversation grew a ~2 GiB working set off a 58 MiB inbound stream (#4788 baseline). This cap bounds ONE session by itself: whenever a request has finished and the turn's save has provably landed on disk, the session's own resident estimate (the same number `GET /__bili/memory` reports) is compared against the cap; over it, the droppable fields — folded-block original contents, metadata bookkeeping, the unbounded part of the raw message tail — are forced back to disk even though pool eviction would exempt the session, and a warning is logged once per minute per session. Nothing is lost: each dropped field re-materializes on first use from a NARROW disk read (only that field, not the whole record), so the next turn pays one small read instead of carrying megabytes resident. Drops gated by a landed save already run at every request boundary regardless of the cap; the cap decides when a single session is called out for exceeding its own budget. Default: equal to the pool soft cap (`536870912` = 512 MiB); set lower (e.g. `53687091` = 64 MiB) to hold one busy long-running session under a personal ceiling. `0` disables the cap check. Compression state (`state`) and the media content store are non-droppable by design and are untouched by this path. File twin: `sessions.hardMaxBytes`. | +| `BILI_PF_V6_REWRITE_STRICT` | Set `1` to CREDIT watermark-v6 misses toward the five-miss breaker instead of the default SELF-HEALING uncredited legacy turn, per lane: in the MSGID lane (`watermarkV6MsgId`) compaction-class shrinks against the stored frontier (`v6m-compact`); in the CANONICAL lane (`watermarkCanonicalV6`) host-side content rewrites (chain breaks: chain-not-started / seam-too-short) — for exotic hosts whose thrash would otherwise ride free legacy turns forever. (The msgid lane’s `v6m-window-system` verdict is ALWAYS credited, independent of this switch.) Default unset/off; pinned by tests/fastpath-pf-msgid.test.ts and tests/fastpath-v6-canonical.test.ts. | | `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | Retention cap (bytes) for the raw wire-history snapshot kept per plugin session to serve the public fork API (#2017). A plugin session whose serialized snapshot exceeds the cap stops being forkable — `GET /__bili/plugin/snapshot` and `POST /__bili/plugin/fork` fail closed with `409` and the capped reason — instead of retaining an unbounded raw copy of the history forever. The cap is re-evaluated on every plugin model request: a session that shrinks back under the cap (after a fork trimmed it, or the host shrank the history) resumes being forkable. Default `104857600` (100 MiB); `0` disables retention entirely (no session is forkable; existing snapshots are dropped on the next request). File twin: `plugin.snapshotCapBytes`. | | `BILI_SESSIONS_DIR` | Directory for persisted session state (default XDG data dir). | | `BILI_SESSION_GC` | Cleanup of stale session files (#1082) is **opt-in**: set to `1`/`true`/`on` to enable — off by default, because session files are user data (exportable, resumable) and there is no silent deletion policy. When enabled, the sweep (boot + hourly) deletes a file only when BOTH conditions hold: older than `BILI_SESSION_GC_MAX_AGE_DAYS`, AND small in the lossless sense — the session was **never compressed** (zero folded blocks) and its newest request body ≤ the token ceiling below, so resuming it costs one cold rebuild from the client's own history and nothing else. Safety rails: compressed sessions are NEVER deleted (their summaries cannot be rebuilt losslessly); a session still held in memory is skipped unless idle since its last disk write; unreadable/corrupt files are left in place; every deletion is audit-logged individually (path, size, age) plus one summary line per non-empty sweep; only the sessions dir is ever touched; emptied protocol subdirectories are removed. Note the resident guard is per-process: another proxy instance sharing `BILI_SESSIONS_DIR` that does not persist (e.g. `BILI_PERSIST=0`) never refreshes file mtimes, so its still-live session files can age out and be swept — the cost is the same bounded cold rebuild, backstopped by the age gate. CCR content stores (#1097) share the session's lifecycle (#1180): a `.content-store.json` companion is deleted together with its session file, an orphaned companion (session file already gone) is swept once past the age gate, and an unreadable companion keeps its session file too (never guessed at). | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 116ca6fe51..6370177c2c 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -103,6 +103,9 @@ | `rollout.features.base` | string | unset (roll out over everyone) | — | 嵌套灰度:仅在所指特性的已入组人群内评估本特性,percent 相对该人群(B 只放量给 A 的用户;允许链式,环路判为关)。 | | `rollout.features.pluginLaneServerHistory` | boolean \| number | 0(关) | BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY | 插件车道服务端正史(#4491):对已放量的安装,(a) 插件车道请求可携带与 `messages` 平行的内部字段 `biliMsgIds`,把消息身份钉在客户端提供的 id 上(转发上游前一律剥掉;格式错误的数组告警后忽略,身份投毒形态在增量车道直接硬失败);(b) 仅回环可达的 `POST /__bili/plugin/render` 端点从统一存储渲染出站前缀(折叠区收敛为摘要锚点;任何失步都返回非 2xx,shim 退回本地历史);(c) 增量车道探测响应通告 `serverHistory`。默认 0%,放量前无行为变化。 | | `rollout.features.pluginFullResendWatermark` | boolean \| number | 5(实验性灰度) | BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK | 全量重发水位快路径(#5403/#5418):插件车道 chat-completions 回合若上一轮响应盖有 x-bili-pf-seq,则用回声计数声明重发前缀(结构扫描,不做全量解析);命中时管线只消费钉存的前缀核 + 新增尾巴窗口(出站字节同源可证;任何疑点当轮回退未动过的全量解析路径并重新盖戳,连续五次计名失配将车道按住一小时)。需客户端把戳回声为 x-bili-pf-count(适配的车道回带;其余车道不会命中)。**实验性灰度**——默认仅约 5% 的安装开启(owner 裁定 2026-11-02:先小流量再 GA;web 面板对全部安装显示一次性公告、快速配置提供三态开关);`true` 强制开、`false`(或百分比)/ env = 0 关。见 TECHNICAL-NOTES.zh-CN.md「全量重发水位快路径」。 | +| `rollout.features.watermarkV6MsgId` | boolean \| number | 0(关) | BILI_ROLLOUT_WATERMARK_V6_MSG_ID | 水位快路径 v6 msgid 车道(#5576/#5600):嵌套在 pluginFullResendWatermark 内的灰度,调度先于规范车道(后者只作为它的 else-if 运行)。身份由服务端铸造:每个会话世界获得一个 pf_sid(pf_+12 hex),经响应头 x-bili-pf-session 签发(响应本身即签发通道);解析顺序=回声(必须指向存活分区)→用真实尾 id 探测全局账本索引(预算 8)→铸造(v6m-arm 冷启动,P5 的唯一一次合法全量)。命中判据=对着我方前沿的账本宽度算术(K=驻留元素数,绝不取客户端回声)+我方 pf_rows 分区元素 K-1 上的窗口起点钉(硬钉;无 id 前沿弱钉);全程不判内容字节——就地改写乘冻结核。命中把窗口行+geom 快照追加入持久 pf_* 分区存储(LSM meta 面:WAL+带校验和检查点每 32 条折叠;重启后从 L1+L0 水合;行只被 deleteSession 删除——归档-重置教径)。对前沿收缩=压缩类(默认不计名透传,BILI_PF_V6_REWRITE_STRICT=1 时计名);窗口头部 system 恒计名;sess-change 不计名;换模型非事件(无模型门);一切计名汇入压在全部车道之上的五次失配/60 分钟共享熔断器。软平面:存储缺失 ⇒ 既有车道逐字节照跑。零留存:命中不钉种子堆。默认 0%——先小流量测数据。见 TECHNICAL-NOTES.zh-CN.md「全量重发水位快路径」(v6 msgid 节)。 | +| `rollout.features.watermarkCanonicalV6` | boolean \| number | 0(关) | BILI_ROLLOUT_WATERMARK_CANONICAL_V6 | 水位快路径 v6 规范车道(#5576):嵌套在 pluginFullResendWatermark 内的灰度。开启后,水位命中的判定不再信任客户端声明的字节:幸存的预过滤是纯服务端算术(回声戳超前/列表宽度/零增量/会话身份换绑),随后 body 用真编解码器转换一次并沿规范行链与已存统一转录稿行对齐(内核 id 快路→内容签名回退,最小接缝跑长 2)。命中时出站前缀 = 缝点前的存储行(折叠区以合成摘要锚点渲染)+ 本轮新尾核;system 文本每轮从当前头部重新解码(构造即透传,v5 head-drift 门退役);换模型是非事件(跨模型共享一个逻辑账本);宿主侧内容改写断链后走不计名 legacy 回合自愈,除非 BILI_PF_V6_REWRITE_STRICT=1 将其计名进入熔断器。零留存教条:规范车道结果绝不钉存种子堆。默认 0%——先小流量测数据再谈晋升。见 TECHNICAL-NOTES.zh-CN.md「全量重发水位快路径→v6 规范车道」。 | +| `rollout.features.nativeIdentityObservation` | boolean \| number | 0(关) | BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION | 原生身份观测(#5573 M1):纯观测采集客户端原生的会话/消息身份证据——按客户端画像表(WorkBuddy x-conversation-id 族 + X-Compaction 头、codex installation-id/item id 族/previous_response_id 轮换 + 压缩前缀文案、Claude Code 会话头 + metadata 双通道交叉核对 + 续接前缀文案、opencode ses_ 亲和键 + fork 标记 + 线上形状断言、Gemini state-snapshot 前缀)——写入会话扩展存储 nidv:obs(只存 sha1-8 摘要,id 值绝不以明文落盘或入日志)外加每请求一行 debug 日志;压缩事件把该行升级为 WARN。不喂任何决策:请求处理、折叠、解析与出站字节一律不动,关闭或未识别流量相对 golden 套件零 diff。默认 0%——开启用于收集 L2 消费方所需的现场数据(折叠静默重定基线 #5488、快路径身份绑定 #5403/#5418)。 | | `maskHosts` | boolean | true | BILI_LOG_MASK_HOSTS | 日志中把非公开目标主机遮成 ;凭据头无论此开关如何总是遮盖。 | | `security.adminRelay` | boolean | false | — | 允许宿主(当前为 dsh web)把它自己入口域名下的整个 /__bili/ 管理面中继到本机代理,使远程浏览器能打开面板(#3373)。默认关闭、仅配置文件。中继跳到达代理时是本机非浏览器客户端(回环源地址 + 回环 Host、无 Origin 头),管理端门槛不变;开启后,凡能访问宿主 web 入口的人都能改配置、写凭据——请先给入口加鉴权/反代。 | | `subagentSplit` | boolean | true | BILI_SUBAGENT_SPLIT | Claude 子代理拥有独立会话命名空间(\|sub:),不再排队在主会话锁后。仅 anthropic 线路 —— Responses 线路上本开关无效(见 `subagentSplit` / BOTH-mechanisms 作用域说明)。 | @@ -433,6 +436,8 @@ | `BILI_MITM_HOSTS` | string[] (csv) | unset | — | 启动器通道:传给被派生代理的 MITM 白名单主机(区别于配置文件对应项 BILI_MITM_DOMAINS)。 | | `BILI_PLUGIN_AGENT` | boolean-ish | off | — | 派生通道:标记插件 agent 进程。 | | `BILI_IMAGE_RESCALE_MAX_DIM` | integer | 1600 | — | 出站图片降采样 pass 的最长边触发阈值(像素,#3400);非数字或低于 128 的取值按默认处理。设计上仅环境变量。 | +| `BILI_NATIVE_IDENTITY_PROFILES_JSON` | string | unset (built-in table) | — | 原生身份观测(#5573 M1):JSON 对象,按内置证据表(src/server/native-identity-profiles.ts)同形提供客户端画像行——按名**新增**或**整体替换**内置行,绝不删除(内置为底)。字段形如内置行(name、matchHeaders、sessionHeaders、msgidHeaders、compactionEdgeHeader、countPatterns、compactionPrefixes……)。畸形 JSON 静默失效(只用内置表)。仅在 rollout.features.nativeIdentityObservation 开启时有意义;不喂任何决策。 | +| `BILI_PF_V6_REWRITE_STRICT` | boolean | unset (off) | — | 设 `1` 把水位 v6 失配**计名**进五次失配熔断器(替代默认的自愈型不计名 legacy 回合),按车道区分:msgid 车道(`watermarkV6MsgId`)计名对存储前沿的压缩类收缩(`v6m-compact`);规范车道(`watermarkCanonicalV6`)计名宿主侧内容改写(链断裂:chain-not-started / seam-too-short)——专为疯狂抖动的怪异宿主准备(否则它们将永远免费搭乘自愈回合)。msgid 车道的 `v6m-window-system`(窗口头部 system)恒计名,不受此开关影响。默认未设置/关闭;由 tests/fastpath-pf-msgid.test.ts 与 tests/fastpath-v6-canonical.test.ts 钉死。 | @@ -1908,8 +1913,11 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_RESIGN_PASSTHROUGH` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_RESUME_INHERITANCE` | `resumeInheritance` | true | | `BILI_ROLLOUT_` | `rollout.features.enabled` | unset (bucket math decides) | +| `BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION` | `rollout.features.nativeIdentityObservation` | 0 (off) | | `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK` | `rollout.features.pluginFullResendWatermark` | 5 (experimental grey) | | `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY` | `rollout.features.pluginLaneServerHistory` | 0 (off) | +| `BILI_ROLLOUT_WATERMARK_CANONICAL_V6` | `rollout.features.watermarkCanonicalV6` | 0 (off) | +| `BILI_ROLLOUT_WATERMARK_V6_MSG_ID` | `rollout.features.watermarkV6MsgId` | 0 (off) | | `BILI_SESSION_CONTENT_STORE_PIN` | `sessions.contentStorePin` | false | | `BILI_SESSION_GC` | `sessions.gc.enabled` | false | | `BILI_SESSION_GC_INTERVAL_MS` | `sessions.gc.intervalMs` | 3600000 | @@ -2046,6 +2054,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_SESSION_WARM_MAX` | 有界启动(#2803):监听前启动时最多预加载的最近保存会话数(默认 `0` = 不预热,受 `sessions.max` 封顶);更早的状态按需重载。设为如 `8` 可恢复对最近集的热加载。文件侧对应:`sessions.warmMax`。 | | `BILI_SESSION_WARM_MAX_BYTES` | 启动预热集的解码字节预算(#2803;默认 `67108864` = 64 MiB)。计的是**解码后**尺寸而非磁盘压缩字节,zstd 归档无法绕过内存上限。单文件超预算则永不预热。文件侧对应:`sessions.warmMaxBytes`。 | | `BILI_SESSION_RESIDENT_MAX_BYTES` | 常驻会话估算字节的软目标(#2803;默认 `536870912` = 512 MiB)。活跃/in-flight 状态不受此限制(字节预算约束的是空闲缓存而非活跃负载);超出目标时空闲状态随请求完成回落磁盘。文件侧对应:`sessions.residentMaxBytes`。 | +| `BILI_PF_V6_REWRITE_STRICT` | 设 `1` 把水位 v6 失配**计名**进五次失配熔断器(替代默认的自愈型不计名 legacy 回合),按车道区分:msgid 车道(`watermarkV6MsgId`)计名对存储前沿的压缩类收缩(`v6m-compact`);规范车道(`watermarkCanonicalV6`)计名宿主侧内容改写(链断裂:chain-not-started / seam-too-short)——专为疯狂抖动的怪异宿主准备(否则它们将永远免费搭乘自愈回合)。msgid 车道的 `v6m-window-system`(窗口头部 system)恒计名,不受此开关影响。默认未设置/关闭;由 tests/fastpath-pf-msgid.test.ts 与 tests/fastpath-v6-canonical.test.ts 钉死。 | | `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | 为公开 fork API 按插件会话保留的原始 wire 历史快照的大小上限(字节,#2017)。序列化快照超限的插件会话不再可 fork —— `GET /__bili/plugin/snapshot` 与 `POST /__bili/plugin/fork` 以 `409` fail-closed 并注明原因 —— 而不是永久保留无上限的原始历史副本。上限在每次插件模型请求时重新评估:会话缩回上限内(fork 裁剪后或宿主缩短历史)即恢复可 fork。默认 `104857600`(100 MiB);`0` 完全停用留存(所有会话不可 fork,已有快照在下一次请求时丢弃)。文件孪生键:`plugin.snapshotCapBytes`。 | | `BILI_SESSIONS_DIR` | 会话持久化目录(默认 XDG data 目录)。 | | `BILI_SESSION_GC` | 过期会话文件清理(#1082)为**可选开启**:设 `1`/`true`/`on` 启用 —— 默认关闭,因为会话文件是用户数据(可导出、可续聊),不应有静默删除策略。启用后,扫描(启动 + 每小时)只在**两个条件同时满足**时删除一个文件:年龄超过 `BILI_SESSION_GC_MAX_AGE_DAYS`,并且"小"到无损 —— 该会话**从未被压缩过**(零折叠块)且最近一次请求体 ≤ 下述 token 上限,这样继续对话只损失一次冷重建(用客户端自己的历史重建),别无其他。安全边界:被压缩过的会话永不删除(其摘要无法无损重建);内存中仍持有的会话会被跳过,除非该会话自上次落盘后一直空闲;不可读/损坏的文件原地保留;每次删除逐条写审计日志(路径、大小、年龄),另有一次非空扫描的汇总日志;只触碰会话目录;清空后的协议子目录一并删除。注意 resident 守卫是进程内的:共享 `BILI_SESSIONS_DIR` 但不落盘的另一实例(如 `BILI_PERSIST=0`)不会刷新文件 mtime,其仍活跃的会话文件可能老化被扫 —— 代价同样是有限的一次冷重建,且有年龄门兜底。CCR 内容存储(#1097)与会话共享生命周期(#1180):`.content-store.json` 伴随文件随其会话文件一起删除;孤儿伴随文件(会话文件已不存在)超过年龄门后被清扫;不可读的伴随文件会连同其会话文件一起保留(绝不猜测)。 | diff --git a/LIFECYCLE.md b/LIFECYCLE.md index cec830c84a..99a80b6e55 100644 --- a/LIFECYCLE.md +++ b/LIFECYCLE.md @@ -56,6 +56,7 @@ | 测试钩子:`BILI_DSH_RECOVERY_INTERVAL_MS` / `BILI_DSH_RETRY_INTERVAL_MS` / `BILI_MODEL_INFO_RETRY_MS` | #2082 等 | 缩短测试/运维节奏(dsh-native.ts;NODE_TEST_CONTEXT 感知,避免生产误缩) | 永久测试设施,随宿主代码走 | 测试钩子 | | `BILI_COMPRESS_AT_FOLD_POINT` | #5470 | compress-at-fold-point 实验性 opt-in(**默认关**;2026-11-01 真实模型验证通过,owner 决定保持 opt-in):把重放视图中可见的 compress 调用对挪到它自己的折叠点(布局 [已折叠区]–[调用对]–[保护消息]),弱注意力模型不再把历史尾部的工具交换误读成新命令;纯视图变换,关时布局字节级不变;文件键 `compress.compressAtFoldPoint`;含 env-only opt-in 可达性修复(默认短路保持不变) | 更多真实模型积累后转正为默认开(走评审),或按废弃流程删除 | opt-in(默认关) | | `BILI_COMPRESS_AT_FOLD_POINT_CLEAR_THINKING` | #5470 | 依附上一开关:挪位瞬间清紧邻的未签名思考段(默认 off,A 关时无效);strict-echo(#684) 或存在签名思考块(#1658)时整个机制不出手 | 同上 | opt-in(默认 off) | +| `BILI_NATIVE_IDENTITY_PROFILES_JSON` | #5573 M1 | 原生身份观测画像表的运维扩展钩子:JSON 客户端画像行按名追加/替换内置行(内置为底;畸形值静默失效);仅在 rollout.features.nativeIdentityObservation 开启时有意义,不喂任何决策 | 观测层转正(现场数据退出判据达成)后移 §2 常设面 | opt-in(缺省 = 内置表) | ## 2. 常设面全量登记表(src/ 实际读取的其余 BILI_* 环境变量) @@ -211,6 +212,7 @@ | `BILI_AUTO_MODE_FORGE` | autoModeForge.enabled,默认 on——auto-mode 分类判定本地 forge,零出账/计费(`=0` 恢复纯透传,留给付真实判定费的用户,#2740 owner 决定) | | `BILI_DSH_CONTEXT_TAB` | dsh.contextTab 的 env 层(#4237):`=0` 隐藏 DSH 顶栏「上下文」标签页(压缩及其余入口全部不受影响);其他值强制开并覆盖文件层;缺省按文件层(默认 true);下次页面加载 / dsh 重启生效 | | `BILI_DSH_MAX_TESTED_VERSION` | dsh native lane「高于已测最高版」启动提示的上限(#4545):从注入包上方最近的 node_modules/@deepseek-ai/dsh/package.json 读运行中的构建,**严格高于**上限才弹一条启动提示,建议先回滚;默认 0.2.0-rc.2(CI e2e 锚点),新版本验证通过后抬高;空串 = 关闭检查;只提示不拦截 | +| `BILI_PF_V6_REWRITE_STRICT` | fastpath watermark v6 msgid 车道(#5600)strict 开关:msgid 车道上把「压缩类前缀收缩」从可自愈未计名 miss 升级为计名(喂熔断);canonical 车道上同开关表示链断裂。v6m-window-system 恒计名、不受它影响;默认 unset/off | ## 3. 非环境变量的 BILI_* 标识符(勿与上表混淆) diff --git a/TECHNICAL-NOTES.md b/TECHNICAL-NOTES.md index 29b4d98feb..0025a5a703 100644 --- a/TECHNICAL-NOTES.md +++ b/TECHNICAL-NOTES.md @@ -587,3 +587,30 @@ Declared trust boundary (§7.4 — human review required, no auto-merge): a clie Honest limits: (1) once a multimodal tail (content array containing `image_url`) lands in the stash, the lane degrades to full legacy turns for the remainder of that history — the kernel renders user content-array images differently when the message is the LAST element versus mid-history, so a frozen stash cannot reproduce legacy bytes position-independently; text workloads (the #5403 reporter's shape) are unaffected. (2) The head fragment is re-parsed (8 elements) each hit solely for the system-run check. (3) Both `x-bili-pf-*` headers are stripped upstream (`UPSTREAM_HOP_HEADERS`) — protocol headers never leak into model traffic. Measured (provenance: `tests/issue5403-watermark-fastpath.test.ts`, hermetic dual-proxy A/B on the same corpus): hit turns skip the full parse/conversion/pin entirely; resident `lastMessages` 8684 B → 1210 B (7.2×) on an 8-turn corpus including a multipart image landing turn; outbound bytes byte-identical to legacy modulo kernel `time="…"` attributes across all turns; `x-bili-pf-*` absent from every captured upstream request, both phases. Unit coverage: `tests/fastpath.test.ts` (full verdict ladder incl. breaker TTL re-arm), `tests/fastpath-scanner.test.ts` (adversarial JSON table, depth cap, surrogate pairs, 50k-element scale), `tests/lane-decisions-matrix.test.ts` (gate row matrix, both directions). + +### v6 canonical lane (`watermarkCanonicalV6`, #5576) + +The v4/v5 lanes above trust a CLIENT-ASSERTED prefix (count echo, head fragment bytes, stashed cores). The owner contract for the next step binds the bookkeeping to the STANDARD STRUCTURE rather than the wire protocol, so the v6 lane changes what “hit” means: the sole decision authority is row-chain alignment of the current request against the stored canonical transcript (`src/server/snapshot-render.ts::buildView`, exported for this consumer), and every byte-compare machinery from v4/v5 is demoted to pure negative filters or diagnostic. It runs only when `watermarkV6MsgId` is off (dispatch tries the msgid lane FIRST — see the §below); its own switch off ⇒ the lane above runs byte-for-byte untouched; on ⇒ the pipeline below (default `0%`, nested grey inside `pluginFullResendWatermark` — measure in a small cohort before any promotion): + +1. **Negative filters (cheap, server-side arithmetic only)** — echoed stamp above our own measurement (`ahead`, credited); ledger-list width ≠ T−K+1 (`tail-ids-shape` — this is where compaction shrinks die, BEFORE anything expensive, credited neither way); count-only zero delta (`no-progress`); swapped client-native session identity (`sess-change`, R7 — unchanged from v5). Id VALUES are still never consulted (M1 observation data decides later consumers; slots `?` ride freely). +2. **Authority stage** — the body converts ONCE with the REAL codec (`openaiToCore` over exactly the bytes the legacy path would parse, whole-body when affordable so intra-call cluster numbering can only AGREE with stored rows, windowed to the walk budget beyond `MAX_CHAIN_WALK`) and the cores chain-walk the stored rows backwards: kernel id fast path, content-sig fallback, min seam run 2, one tolerated gap. Any infrastructure doubt degrades uncredited (`v6-store-unavailable` / `v6-pre-fold` / `v6-block-unanchored` / `v6-transcript-empty` / `v6-row-unparsable` / …). +3. **Hit shape** — view = stored rows up to the seam (covered regions render behind their synthetic `acp_summary_*` anchors — folded history rides as ingested) + this turn's fresh tail cores; `windowMessages` slices element-space from the seam origin (`wireIndex` sidecar, persisted-anchor fallback); system text RE-DECODES from the CURRENT head every turn — passthrough by construction, which retires the v5 head-drift gate entirely (there is no stashed copy to drift from). +4. **R6 logical ledger** — the transcript ledger is model-independent (one ledger shared across models); model switches are NON-EVENTS (a cross-model projection is derived rendering only, and inter-model byte sharing was physically out of scope to begin with); the per-model seed partitioning of v4/v5 stops gating. +5. **Rewrite class** — a chain break on otherwise-validated traffic (`chain-not-started` / `seam-too-short`) means the host rewrote content (client-side compaction passes through AS NEW CONTENT by default ruling, epoch rewrites included): default is a SELF-HEALING UNcredited legacy turn (host compaction is normal traffic, and the legacy rewrite-detection machinery then marks the boundary so subsequent turns align against the rebased view); `BILI_PF_V6_REWRITE_STRICT=1` credits those breaks toward the breaker for exotic thrash-rewriting hosts. +6. **Zero-retention doctrine** — canonical outcomes pin NO seed stash (no multi-MiB prefix cores resident between turns; memory stays flat across model switches). The v4/v5 stash remains ONLY on the v6-off lane. +7. **Known residual (guarded, tested)** — a FRESH trailing element whose content collides with stored history can false-match an old row through the sig fallback and produce a full-length join whose served view would silently DROP the new tail; the seam-origin guard turns that shape into an uncredited `v6-no-seam-origin` degrade instead (never a silent drop). + +Honest trade (stated for the rollout decision): a v6 hit costs roughly the legacy conversion's CPU every turn (the authority requires the real decode); it spends that to buy structural correctness (row-backed prefix, passthrough system, model-independent identity) and to remove the multi-MiB prefix retention the v5 stash carried. Default OFF keeps v4/v5 semantics for everyone until the cohort numbers say otherwise. Unit coverage: `tests/fastpath-v6-canonical.test.ts`. + +### v6 msgid lane (`watermarkV6MsgId`, #5576/#5600) + +The second v6 lane — and the one that owns dispatch when both switches are live (`evaluateWatermarkFastpath` tries it FIRST; the canonical lane above runs only as its `else if`). Where the canonical lane re-derives identity by chain-walking stored transcript rows, this lane mints its OWN identity: every session world gets a server-issued `pf_sid` (`pf_` + 12 hex digits, `randomBytes(6)`), issued over the response header `x-bili-pf-session` — the response itself is the issuance channel, and the client just stores what it receives. Default `0%`, nested grey inside `pluginFullResendWatermark`: + +1. **Identity resolution (echo → probe → mint)** — a well-formed echoed sid that names a LIVE partition (rows or meta present) is adopted; failing that, REAL ids from the tail-id list probe the GLOBAL ledger index (budget `PF_PROBE_BUDGET=8`); failing that, the turn MINTS a fresh sid and arms (`v6m-arm`) — P5's one legal full materialization per world. Fabricated echoes are never adopted (warn + fall through to probe/mint). A registry ext (`pf:sids`) records every mint under the owning bili session so deleteSession can cascade-sweep partitions; it is advisory only (worst case under-deletes an orphan, never another session's data). +2. **Persistent residency** — a hit appends THIS TURN'S window rows to the partition store (`pf_rows`, append-only L0 keyed on absolute body-element ordinals; split cores of one multimodal element share the ledger id AND the ordinal) plus a `geom` snapshot (T, model, row frontier, imgFlag, cumulative strip markers) into the LSM meta plane (`pf_log` WAL + checksummed `pf_ckpt` snapshots folding every 32 entries). Post-boot / lane-toggle / resident-drop rebuilds via `hydratePf` from the L1 checkpoint + in-order L0 tail replay; `reset`/`arm` entries move the active frontier (`fromPos`) while the archived world stays durable — **rows are deleted only by deleteSession**, never by resets (archive-and-reset doctrine). Any unrecoverable shape degrades uncredited (`v6m-hydrate`). +3. **Hit authority = ledger-width arithmetic against OUR frontier** — K is the resident's element count (never a client echo); the window decodes ONLY spans[K..T) with the real codec; the single proof required beyond width is the WINDOW ORIGIN: the claimed id must be recorded on element K−1 of OUR partition (hard pin — exact regardless of how many rows that element split into), or the frontier row was ingested id-silent (weak pin — any evidence accepted; later turns re-bind through newly stored ids); anything else is a `v6m-prev` self-heal (resident untouched, retry next turn). Shape mismatch / frontier-row misalignment ⇒ `v6m-shape`; zero delta ⇒ `v6m-no-progress` (an empty window pins nothing). Content BYTES are never consulted at any stage — an in-place edit under stable geometry+ids rides the frozen core (server snapshot absolute authority; the edit is invisible BY DESIGN). +4. **Event matrix** — shrink against our frontier (`T < K`) = client compaction class: archive-and-reset on the SAME sid, UNcredited passthrough-as-new-message by default (host compaction is normal traffic), `BILI_PF_V6_REWRITE_STRICT=1` credits it toward the breaker for exotic thrashing hosts; a system/developer element at the window head poisons the view and is CREDITED (`v6m-window-system`, independent of that switch); swapped client-native session carrier under one bili session remains the UNcredited `sess-change` identity event (R7 parity, asserted only when both sides claim); model switches are NON-events — the lane carries no model gate at all (the geom entry records the model for observability only). All credited verdicts feed the SAME five-miss / 60-min breaker shared with the v4/v5/canonical lanes, which stands ABOVE every lane (held turns keep measuring + restamping but never resolve a sid). +5. **Soft plane (§4.1.6)** — store absent, unshaped or misbehaving ⇒ the lane returns null and the pre-existing lanes run byte-for-byte; the plane NEVER throws into the request path (every pf call is wrapped; degradation is silent). +6. **Zero retention on hits** — like the canonical lane, msgid outcomes pin NO seed stash; the resident lives in a module-level `WeakMap` plus the partition on disk, so memory stays flat across model switches and a dropped resident costs one bounded hydration. + +Honest trade: a hit still pays the real decode of the WINDOW fragment (never the full body) plus one ledger lookup per origin pin; that price buys identity that resolves INSIDE our partition (a client cannot forge continuity — unknown sids mint fresh worlds, foreign origin ids miss) and removes the multi-MiB prefix stash. Unit coverage: `tests/fastpath-pf-msgid.test.ts` (soft plane, cold-start mint/arm, hard-pin hits, shape/wrong-origin self-heal, fabricated echo, compaction archive-and-reset, post-boot hydration across an L1 fold, sess-change event, model-switch non-event, shared-breaker naming incl. TTL recovery, id-stable edits riding frozen cores). \ No newline at end of file diff --git a/TECHNICAL-NOTES.zh-CN.md b/TECHNICAL-NOTES.zh-CN.md index d5ea5b5d97..0545a84c38 100644 --- a/TECHNICAL-NOTES.zh-CN.md +++ b/TECHNICAL-NOTES.zh-CN.md @@ -164,3 +164,30 @@ chat wire 上 bili 自己 vs 客户端自带:插件车道上,请求体里等于 诚实边界:(1) 一旦多模态尾部(含 `image_url` 的 content array)进入缓存,该段历史此后整段降级回 legacy 全量轮——kernel 按位置区分 user 图片消息在尾部与在历史中部的渲染,冻结缓存无法做到位置无关地复现 legacy 字节;纯文本工作负载(#5403 报告者的形态)不受影响。(2) 每次命中会现解析头部 8 个元素,只为做 system-run 校验。(3) 两个 `x-bili-pf-*` 头都在转发上游前剥掉(`UPSTREAM_HOP_HEADERS`)——协议头绝不漏进模型流量。 实测(出处:`tests/issue5403-watermark-fastpath.test.ts`,同语料 hermetic 双代理 A/B):命中轮完全跳过全量解析/转换/钉存;含一个 multipart 图片落位回合的 8 回合语料上,常驻 `lastMessages` 8684 B → 1210 B(7.2×);全程出站字节与 legacy 逐字一致(仅 kernel `time="…"` 属性归一);两个阶段捕获到的上游请求里均无 `x-bili-pf-*`。单元覆盖:`tests/fastpath.test.ts`(verdict 阶梯全表,含熔断 TTL 解除后重播种)、`tests/fastpath-scanner.test.ts`(对抗 JSON 表、深度上限、surrogate 对、5 万元素规模)、`tests/lane-decisions-matrix.test.ts`(门矩阵,双向)。 + +### v6 规范车道(`watermarkCanonicalV6`,#5576) + +上面的 v4/v5 车道信任的是**客户端声明的前缀**(计数回声、头部片段字节、钉存核)。owner 契约要求把账本绑到**标准结构**而非线上协议,v6 车道因此改变了“命中”的定义:唯一决策权威 = 当前请求与已存规范转录稿的行链对齐(`src/server/snapshot-render.ts::buildView`,为此导出),v4/v5 全部字节比对机制降级为纯负向过滤或诊断。它只在 `watermarkV6MsgId` 关闭时运行(调度先试 msgid 车道——见下面节);它自己的开关关 ⇒ 上面那条车道逐字节不动;开 ⇒(默认 `0%`,嵌套在 `pluginFullResendWatermark` 内灰度——先小流量测数据再谈晋升): + +1. **负向过滤(廉价,纯服务端算术)**——回声戳高于本端测量(`ahead`,计名);账本列表宽度 ≠ T−K+1(`tail-ids-shape`——压缩收缩在这里死掉,**在任何昂贵动作之前**,两种模式都不计名);纯计数零增量(`no-progress`);客户端原生会话身份换绑(`sess-change`,R7——与 v5 完全一致)。id 的**值**依旧从不被查询(M1 观测数据决定后续消费者;占位槽 `?` 无惩罚通行)。 +2. **权威阶段**——body 用**真编解码器**转换一次(`openaiToCore` 作用在 legacy 路径将解析的同一批字节上;长度可承受时整段转换,使调用内簇编号只能与已存行**一致**而不可能漂移;超过 `MAX_CHAIN_WALK` 时截取步长预算窗口),核心沿存储行链回溯:内核 id 快路 → 内容签名回退,最小接缝跑长 2,容忍一个空档。任何基础设施疑点不计名降级(`v6-store-unavailable` / `v6-pre-fold` / `v6-block-unanchored` / `v6-transcript-empty` / …)。 +3. **命中形态**——视图 = 缝点之前的存储行(折叠区以合成 `acp_summary_*` 锚点渲染——折叠历史按摄取原样乘车)+ 本轮新尾核;`windowMessages` 从缝点元素原点切片(`wireIndex` 侧车,持久锚回退);system 文本每轮从**当前**头部重新解码——构造即透传,v5 head-drift 门整体退役(没有钉存的副本可以漂移)。 +4. **R6 逻辑账本**——转录稿账本与模型无关(跨模型共享一个账本);换模型是**非事件**(跨模型投影只是派生渲染;模型间字节共享本就物理不在范围内);v4/v5 的按模型种子分片不再参与决策。 +5. **改写类**——其余校验全过但链断裂(`chain-not-started` / `seam-too-short`)= 宿主改写了内容(客户端压缩按默认裁定**作为新内容透传**,纪元级重写同理):默认走**自愈型不计名** legacy 回合(宿主压缩是正常流量,legacy 重写检测机制随后标记边界,后续回合对着重定基线的视图对齐);`BILI_PF_V6_REWRITE_STRICT=1` 时这些断裂计名进入熔断器,针对疯狂重写的怪异宿主。 +6. **零留存教条**——规范车道结果**绝不钉存**种子堆(多 MiB 前缀核不再驻留内存;跨模型切换内存持平)。v4/v5 钉存仅存在于 v6 关闭的车道。 +7. **已知残余(有守卫,有测试)**——一条与历史内容雷同的**新**尾部消息可能经签名回退误配旧行,产生“全长连接”,其服务视图会**静默丢掉新尾巴**;缝点守卫把该形态转为不计名的 `v6-no-seam-origin` 降级(宁可降级,不可静默丢尾)。 + +诚实的权衡(供放量决策参考):v6 命中每个回合要花大约 legacy 转换的 CPU(权威判定要求真解码),换来的是结构正确性(行支撑的前缀、透传 system、模型无关的身份)以及移除 v5 钉存携带的多 MiB 前缀驻留。默认关闭,保留 v4/v5 语义,直到灰度数据说话。单测:`tests/fastpath-v6-canonical.test.ts`。 + +### v6 msgid 车道(`watermarkV6MsgId`,#5576/#5600) + +第二条 v6 车道——两个开关同时开启时由它独占调度(`evaluateWatermarkFastpath` 先试它;上面的规范车道只作为它的 `else if` 运行)。规范车道靠链式回溯已存转录稿行重新推导身份,这条车道则**铸造自己的身份**:每个会话世界获得一个服务端签发的 `pf_sid`(`pf_` + 12 位 hex,`randomBytes(6)`),经响应头 `x-bili-pf-session` 发出——响应本身就是签发通道,客户端只管存下收到的值。默认 `0%`,嵌套灰度于 `pluginFullResendWatermark` 之内: + +1. **身份解析(回声→探针→铸造)**——形状合格的回声 sid 且对应一个存活分区(有行或有 meta)即被采纳;否则用尾 id 列表里的真实 id 探测全局账本索引(预算 `PF_PROBE_BUDGET=8`);再失败则本轮铸造新 sid 并武装(`v6m-arm`)——P5 允许的世界级一次合法全量解析。伪造回声绝不被采纳(warn 后落入探针/铸造)。注册表 ext(`pf:sids`)在所属 bili 会话名下记录每一次铸造,供 deleteSession 级联清扫分区;它只是参考性的(最坏少删一个孤儿,绝不会误删别的会话的数据)。 +2. **持久驻留**——命中时把本轮窗口行追加分区存储(`pf_rows`,append-only L0,键为绝对体元素序号;一个多模态元素拆出的核共享账本 id 与序号),并向 LSM meta 面写一条 `geom` 快照(T、模型、行前沿、imgFlag、累计剥离标记)(`pf_log` WAL + 带校验和的 `pf_ckpt` 快照,每 32 条折叠一次)。重启后/切车道/驻留丢失时,`hydratePf` 从 L1 ckpt + 按序 L0 尾部重放重建;`reset`/`arm` 条目移动活动前沿(`fromPos`),归档世界保持持久——**行只会被 deleteSession 删除,重置永不删行**(归档-重置教条)。任何不可恢复的形状不计名降级(`v6m-hydrate`)。 +3. **命中判据 = 对着我方前沿的账本宽度算术**——K 取驻留的元素数(绝不取客户端回声);窗口只用真编解码器解码 spans[K..T);宽度之外唯一需要的证明是窗口起点:声称的 id 必须记录在我方分区的元素 K−1 上(硬钉——无论该元素拆成几行都精确),或前沿行是无 id 摄入的(弱钉——接受任何证据,后续回合经新存 id 重新绑定);其余一律 `v6m-prev` 自愈(驻留不动,下回合重试)。形状不符/前沿行错位 ⇒ `v6m-shape`;零增量 ⇒ `v6m-no-progress`(空窗口钉不了任何东西)。全程**从不查看内容字节**——几何+id 稳定下的就地改写乘坐冻结核(服务端快照绝对权威;改写按设计不可见)。 +4. **事件矩阵**——对我方前沿收缩(`T < K`)= 客户端压缩类:同一 sid 上归档-重置,默认**不计名按新消息透传**(宿主压缩是正常流量),`BILI_PF_V6_REWRITE_STRICT=1` 时计名进熔断器以对付疯狂抖动的宿主;窗口头部出现 system/developer 元素会污染视图,恒计名(`v6m-window-system`,不依赖该开关);同一 bili 会话下换绑客户端原生会话载体仍是 R7 同款的**不计名** `sess-change` 身份事件(仅当两侧都声明时才断言);换模型是非事件——车道根本不带模型门(geom 条目记录模型仅供观测)。所有计名判定汇入与 v4/v5/规范车道**共享**的五次失配/60 分钟熔断器,它压在一切车道之上(持有期回合继续测量+重盖戳,但永不解析 sid)。 +5. **软平面(§4.1.6)**——存储缺失、未成形或行为异常 ⇒ 车道返回 null,既有车道逐字节照跑;平面永不向请求路径抛错(每次 pf 调用都有包装,降级静默)。 +6. **命中零留存**——与规范车道相同,msgid 结果不钉任何种子堆;驻留活在模块级 `WeakMap` + 磁盘分区里,内存跨模型切换保持平坦,丢驻留的代价只是一次有界水合。 + +诚实代价:命中仍要付窗口片段的真解码(永远不是全量 body)+ 每次起点钉一根账本查询;换来的是身份在我方分区内部解析(客户端无法伪造连续性——未知 sid 铸新世界,外来起点 id 直接失配),并去掉多 MiB 前缀堆。单测覆盖:`tests/fastpath-pf-msgid.test.ts`(软平面、冷启动铸造/武装、硬钉命中、形状/错源自愈、伪造回声、压缩归档-重置、跨 L1 折叠的重启后水合、sess-change 事件、换模型非事件、含 TTL 恢复的共享熔断计名、id 稳定改写乘冻结核)。 \ No newline at end of file diff --git a/src/incremental-lane.ts b/src/incremental-lane.ts index e770bc4996..5fe230b8de 100644 --- a/src/incremental-lane.ts +++ b/src/incremental-lane.ts @@ -66,6 +66,7 @@ import { peekSession } from "./session.js"; import { pluginAgentHeader, pluginReportedModel } from "./plugin.js"; import { headerValue, readBody } from "./server.js"; import { featureEnabled } from "./rollout.js"; +import { observeNativeIdentity } from "./server/native-identity.js"; import { MSG_IDS_BODY_FIELD, MSG_SEQS_BODY_FIELD, applyMsgIdPin, validateMsgIds } from "./msgid-pin.js"; export const INCREMENTAL_TURN_PATH = "/__bili/plugin/turn"; @@ -788,6 +789,24 @@ export async function handleIncrementalTurn( entry.systemSet = true; } touchCache(conversationId, entry); + // #5573 M1 (L1): native-identity observation — the lane delta is the + // cheap scan surface (small by construction); the msgIds envelope + // carries host-native per-element ids (#4491). Pure observation, + // rollout-gated off by default, feeds no decision. + if (featureEnabled("nativeIdentityObservation")) { + observeNativeIdentity( + { + key: conversationId, + headers: req.headers, + body: rawBody !== undefined ? rawBody : JSON.stringify(deltaWire), + protocol: "anthropic", + agent, + msgIds: deltaIds, + }, + getStore(), + log, + ); + } // Record the epoch BEFORE the forward lands the new branch's rows. if (resync) await recordResyncBaseSeq(sessionId, log); persistLaneState(sessionId, entry.system, entry.systemSet, entry.marks, log); diff --git a/src/persist.ts b/src/persist.ts index fa2d338934..8244dd6f00 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -2393,6 +2393,50 @@ export class SessionStore { return this.liveShadow().deleteSessionExt(sessionId, key); } + // ------------------------------------------------------------------ + // #5600 fastpath snapshot store (pfsid plane). SOFT availability: null ⇒ + // the lane degrades to status-quo behavior (§4.1.6 待裁④ 默认值). Knob-off + // traffic never reaches here — zero reads, zero writes. + // ------------------------------------------------------------------ + + pfStore(): { + appendTurn( + sid: string, + rows: { ledgerId: string | null; elem: number; coreId: string; coreJson: string }[], + entries: { kind: string; payload: unknown; bornPosHi?: number }[], + ): { maxPos: number; lsSeq: number }; + lookupLedger(ledgerId: string): { sid: string; pos: number } | null; + verifyOriginPin(sid: string, elem: number, ledgerId: string): boolean; + readMeta(sid: string): { + ckpt: { ckptLsSeq: number; metaJson: string; checksum: string } | null; + tailLog: { lsSeq: number; kind: string; payloadJson: string; bornPosHi: number | null }[]; + }; + writeCkpt(sid: string, ckptLsSeq: number, metaJson: string, checksum: string, truncateBeforeInclusive?: boolean): void; + readRows(sid: string, intervals: ReadonlyArray): { pos: number; elem: number; ledgerId: string | null; coreId: string; coreJson: string; ts: number }[]; + maxPos(sid: string): number; + } | null { + if (!this.replicasAvailable()) return null; + try { + const s = this.liveShadow(); + // #5576 review S5: a shadow whose sync plane is absent + // (replica/MySQL views) exposes no-op pf closures — handing them + // to the lane loops a per-turn fabrication warn plus unbounded + // sid minting. Report null instead (soft plane: null == absent). + if (!s.pfPlaneUsable()) return null; + return { + appendTurn: (sid, rows, entries) => s.pfAppendTurn(sid, rows, entries), + lookupLedger: (ledgerId) => s.pfLookupLedger(ledgerId), + readMeta: (sid) => s.pfReadMeta(sid), + writeCkpt: (sid, n, json, sum, t) => s.pfWriteCkpt(sid, n, json, sum, t), + verifyOriginPin: (sid, e, id) => s.pfVerifyOriginPin(sid, e, id), + readRows: (sid, iv) => s.pfReadRows(sid, iv), + maxPos: (sid) => s.pfMaxPos(sid), + }; + } catch { + return null; + } + } + setKv(namespace: string, key: string, value: unknown): void { this.liveShadow().setKv(namespace, key, value); } diff --git a/src/rollout.ts b/src/rollout.ts index adb1a85157..0ec8c8b862 100644 --- a/src/rollout.ts +++ b/src/rollout.ts @@ -64,6 +64,41 @@ export const ROLLOUT_FEATURES: Record = { // `rollout.features.pluginFullResendWatermark: true|false` (or a percent) // / `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK=1|0`. pluginFullResendWatermark: { percent: 5 }, + // #5576 v6 (owner ruling 2026-11-03): canonical-watermark lane — the hit + // verdict comes SOLELY from unified-transcript row-chain alignment + // (the #3482 chain discipline: fold-time id fast path, identity-sig + // fallback, MIN_SEAM_RUN), replacing the v4/v5 byte-compare + width- + // arithmetic + head-drift gate stack as decision authority (they remain + // negative filters where they are pure server-side arithmetic). The view + // is served from stored rows (covered regions suppressed, synthetic + // anchors spliced) plus this turn's converted tail — NOTHING prefix- + // shaped is pinned in memory on a hit. Requires storage.unified=full + // (reads) and one prior fold (an active block anchors the rows); every + // verification miss degrades to the untouched legacy full-parse path + // byte-identically. Default 0% — nested INSIDE the pluginFullResendWatermark + // cohort by design (v6 evaluates only where the base watermark lane is + // already live): force on with `rollout.features.watermarkCanonicalV6: true` + // (or a percent) / `BILI_ROLLOUT_WATERMARK_CANONICAL_V6=1`. Companion + // env `BILI_PF_V6_REWRITE_STRICT=1` credits content-rewrite misses toward + // the breaker instead of letting them self-heal silently. + watermarkCanonicalV6: { percent: 0, base: "pluginFullResendWatermark" }, + // #5600 v6 msgid lane (owner ruling 2026-11-03): identity rides + // server-minted pf_sids (x-bili-pf-session echo + ledger-id probe + mint), + // NEVER client-native ids; frozen prefix cores come from the persistent + // pf_* snapshot partition store (hydrated at boot, append-only, cleanup + // rides deleteSession only) instead of an in-memory seed stash — the P1 + // retention pair is removed even across restarts. Hit authority: ledger + // width arithmetic against our own persisted frontier + window-origin pin; + // the stamped x-bili-pf-count demotes to diagnostic. Every doubt degrades + // to the untouched legacy full-parse; compaction resets the world by + // archive-and-rebase (§6.1). SOFT plane: store unavailable ⇒ this lane + // stands aside byte-for-byte. Default 0% — nested INSIDE the + // pluginFullResendWatermark cohort; force on with + // `rollout.features.watermarkV6MsgId: true` / + // `BILI_ROLLOUT_WATERMARK_V6_MSG_ID=1`. Companion env + // `BILI_PF_V6_REWRITE_STRICT=1` credits compaction-class misses toward + // the breaker (shared with the canonical lane). + watermarkV6MsgId: { percent: 0, base: "pluginFullResendWatermark" }, // #4804 WS delta continuation: codex/opencode WebSocket clients that keep a // previous_response_id checkpoint stop reconstructing full history on every // turn — the session augments the delta with the server-committed previous @@ -97,6 +132,15 @@ export const ROLLOUT_FEATURES: Record = { // cohorts (compressed vs deleted) and detects generation switches. // Default 0% — force on with `BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY=1`. pluginLaneServerHistory: { percent: 0 }, + // #5573 M1 (L1) native-identity observation: pure-observation collection of + // client-native session/msgid identity evidence (header families, bounded + // body signals, compaction prefaces) into the session ext store + // ("nidv:obs") plus a per-request debug line. Feeds NO decision — request + // handling, folding, parsing and outbound bytes are untouched; off-or- + // unknown traffic produces zero diff against the golden suites. Default + // 0% — force on with `rollout.features.nativeIdentityObservation: true` or + // `BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION=1`. + nativeIdentityObservation: { percent: 0 }, }; export const BP_MAX = 10_000; diff --git a/src/server/fastpath.ts b/src/server/fastpath.ts index 161d05d7c4..6921f351f7 100644 --- a/src/server/fastpath.ts +++ b/src/server/fastpath.ts @@ -31,10 +31,23 @@ // change the count and are caught by the k-behind/k-ahead verdicts before // any stale core could feed the kernel; under v4 (count relaxation below) // the catch moves from the client echo to the ledger width arithmetic itself. +// +// v6 canonical lane (#5576, owner ruling 2026-11-03, rollout-gated +// `watermarkCanonicalV6`, default OFF): the hit verdict comes SOLELY from +// unified-transcript row-chain alignment (alignClientAgainstTranscript — +// fold-time id fast path, identity-sig fallback, MIN_SEAM_RUN discipline). +// The byte-compare / width-arithmetic machinery survives ONLY as negative +// filters built from server-side arithmetic over client-declared ledger ids; +// nothing compares prefix BYTES anymore, the rows ARE the prefix proof. +// On a hit the view is served from stored rows (covered regions suppressed, +// synthetic anchors spliced) plus this turn's converted tail; NO prefix- +// shaped state is pinned in memory (the seed stash is bypassed entirely on +// canonical outcomes). Every doubt degrades to the untouched legacy path. import type { CoreMessage } from "acp-kernel"; -import type { OpenAIRequestBody } from "acp-kernel/wire"; +import type { BiliMessage, OpenAIRequestBody } from "acp-kernel/wire"; import { openaiToCore } from "acp-kernel/wire"; import type http from "node:http"; +import { createHash, randomBytes } from "node:crypto"; import { featureEnabled } from "../rollout.js"; import { getStore } from "../persist.js"; import type { Session } from "../session.js"; @@ -42,16 +55,45 @@ import { type RawStripCounts, stripRawStage } from "../marker/engine.js"; import { INCREMENTAL_TURN_MARKER_HEADER } from "../incremental-lane.js"; import { watermarkGateAllowed } from "./lane-decisions.js"; import { scanMessagesSpans, type RawSpan } from "./fastpath-scan.js"; +import { alignClientAgainstTranscript, MAX_CHAIN_WALK } from "./snapshot-render.js"; export const PF_COUNT_HEADER = "x-bili-pf-count"; export const PF_SEQ_HEADER = "x-bili-pf-seq"; export const PF_TAIL_IDS_HEADER = "x-bili-pf-tail-ids"; +// #5600 §3.1: the msgid lane's identity channel. Echoing carries identity +// ONLY (content claims ride x-bili-pf-tail-ids); responses always carry the +// server's own sid for the request's partition, including the mint turn. +export const PF_SID_HEADER = "x-bili-pf-session"; + +// #5571 v5 (owner ruling 2026-11-03): identity-only verification takes the +// client's NATIVE session id as an identity input alongside host msgids. +// Canonical table lives in #5571's native-identity-profiles; these three +// spellings are byte-duplicated here (#3482 precedent) so this module stays +// dependency-free of the observation layer. +const CLIENT_SESS_HEADERS = ["x-conversation-id", "x-claude-code-session-id", "x-session-affinity"] as const; + +/** h8 digest of whichever known client session header carries a value + * (first hit wins); undefined when none does — absence claims nothing. */ +function clientSessDigest(req: http.IncomingMessage): string | undefined { + for (const h of CLIENT_SESS_HEADERS) { + const v = req.headers[h]; + if (typeof v === "string" && v.length > 0) return createHash("sha1").update(v).digest("hex").slice(0, 8); + } + return undefined; +} // #5418 v3: UNIDENTIFIED slot marker — an element that carries no host id // (native-compaction summary, persona sidecar, foreign shape) travels as "?". // Wire literal duplicated per #3482 (agent bundles stay import-free leaves). export const PF_SLOT_UNKNOWN = "?"; const HEAD_CHECK_ELEMENTS = 8; const PF_BREAKER_MISSES = 5; +// #5576 review V1 (bounded ride lifetime): an in-place client edit with stable +// host ids rides stale stashed bytes on EVERY subsequent turn — the ride +// re-stashes its own view, so nothing downstream ever notices. After this many +// CONSECUTIVE drift turns one uncredited legacy turn reseeds from real bytes +// (any non-canonical miss reseeds in finalize), then rides may resume. The cap +// also bounds the diagnostic warn stream below to one line per cap-many turns. +const PF_V5_DRIFT_RIDE_CAP = 5; const PF_DISABLED_TTL_MS = 60 * 60 * 1000; const PF_EXT_KEY = "pf:state"; @@ -63,6 +105,13 @@ interface PFSavedState { /** Last received tail-id list (#5418 v2) — survives restarts so the * prevId continuity check is live on the first turn back. */ l?: string[]; + /** #5571 v5: client-native session id digest recorded with this state + * (h8, never raw — privacy parity with #5571); sess-change gate input. */ + cs?: string; + /** #5576 review V1: consecutive id-mode drift RIDE turns (stale prefix + * served under stable host ids). A clean head or a reseeding miss resets + * it; exceeding PF_V5_DRIFT_RIDE_CAP forces one uncredited reseed turn. */ + d?: number; } interface PFSeed { @@ -123,6 +172,22 @@ export interface FastpathOutcome { * into the (re)seeded state either way so continuity holds across both. * Absent (undefined) for count-only turns or shape-invalid lists. */ readonly receivedTailIds?: string[]; + /** #5571 v5: client-native session id digest of THIS request + * (undefined when no known header carried a value). Persisted by + * finalize as state `cs`; feeds the sess-change identity gate. */ + readonly clientSess?: string; + /** #5576 review V1: the drift-ride count THIS hit carries (undefined on + * every non-drift hit — a clean head resets the episode in finalize). */ + readonly driftRides?: number; + /** #5576 v6: produced on the canonical lane (hit AND miss). Finalize + * must NOT pin seed-stash cores for these outcomes — retaining prefix- + * shaped state in memory is exactly what the canonical lane removes; a + * lane toggle back to v5 self-heals with one legacy verification turn. */ + readonly canonical?: boolean; + /** #5600 msgid lane: the server-minted pf_sid this request resolved to. + * Present on every lane outcome (minted on the arm turn); finalize + * performs the partition append/residency update and echoes the header. */ + readonly sid?: string; } /** #5418 v2: decode the shim's x-bili-pf-tail-ids value. @@ -173,6 +238,8 @@ function readSaved(sessionId: string): PFSavedState | null { l: Array.isArray(o.l) && o.l.length > 0 && o.l.length <= 2048 && (o.l as unknown[]).every((x): x is string => typeof x === "string" && x.length > 0 && x.length <= 512) ? (o.l as string[]) : undefined, + cs: typeof o.cs === "string" && o.cs.length === 8 ? o.cs : undefined, + d: typeof o.d === "number" && Number.isInteger(o.d) && o.d >= 0 ? o.d : undefined, }; } catch { return null; @@ -232,16 +299,33 @@ export function evaluateWatermarkFastpath( const T = scan.total; const rawStamp = req.headers[PF_COUNT_HEADER]; const stamped = typeof rawStamp === "string" && /^\d+$/.test(rawStamp.trim()) ? parseInt(rawStamp.trim(), 10) : NaN; + // #5571 v5: captured ONCE per evaluated turn — feeds the sess-change gate + // below and rides the outcome so finalize persists it as state `cs`. + const nowCs = clientSessDigest(req); const miss = (decision: string, credited: boolean, receivedTailIds?: string[]): FastpathOutcome => { if (credited) noteCreditedMiss(session.id, decision, log); log("info", `[${session.id}] watermark fastpath miss (${decision}) — legacy full-parse this turn`); - return { hit: null, measuredT: T, decision, creditedMiss: credited, receivedTailIds }; + return { hit: null, measuredT: T, decision, creditedMiss: credited, receivedTailIds, clientSess: nowCs }; }; const seed = seeds.get(session); const rawTailIds = req.headers[PF_TAIL_IDS_HEADER]; // three states: undefined = header absent (pure count mode); null = // present but malformed; array = shape-valid. const parsedTailIds = rawTailIds === undefined ? undefined : typeof rawTailIds === "string" ? parseTailIds(rawTailIds) : null; + // Lane dispatch (#5600/#5576): the msgid lane (default OFF) runs first + // when live — it owns identity via server-minted pf_sids over the + // snapshot partition store and returns null whenever the pf plane is + // unavailable, falling straight through to the lanes below untouched. + if (featureEnabled("watermarkV6MsgId")) { + const mid = evaluateMsgIdLane({ session, req, bodyText, scan, saved, parsedTailIds, nowCs, log }); + if (mid !== null) return mid; + } else if (featureEnabled("watermarkCanonicalV6")) { + // #5576 v6 (owner ruling 2026-11-03): when the canonical lane is live + // the verdict comes from unified-transcript row-chain alignment + // instead of the v4/v5 machinery below — UNTOUCHED for everyone else + // (default OFF keeps v5 byte-behavior the default). + return evaluateCanonicalV6({ session, bodyText, scan, saved, stamped, parsedTailIds, nowCs, miss }); + } // #5403/#5418 v4 count relaxation (owner ruling 2026-11-02): once a // session carries a ledger-proven tail-id list whose WIDTH matches its own // bookkeeping (len = T - seed.stampT + 1), msgid continuity proves the @@ -289,6 +373,17 @@ export function evaluateWatermarkFastpath( if (prevKnown !== undefined && prevKnown !== PF_SLOT_UNKNOWN && tailIds[0] !== prevKnown) return miss("tail-ids-prev", false); } } + // #5571 v5 (owner ruling 2026-11-03): session continuity joins the + // identity proof. State carrying a client-native session id digest + // means an earlier turn claimed an identity; this request must keep + // claiming the SAME one — a swapped conversation riding one bili + // session is an identity event, not a byte question. Neutral when + // either side claims nothing (header-less clients unchanged); + // asserted only when both sides do. Id mode only (no valid list + // reached here without one). + if (tailIds !== undefined && saved?.cs !== undefined) { + if (nowCs !== undefined && nowCs !== saved.cs) return miss("sess-change", false, tailIds); + } if (seed.multimodal) return miss("seeded-multimodal", false, tailIds); const headEnd = Math.min(HEAD_CHECK_ELEMENTS, T); const headFragment: { messages: unknown } = { messages: [] }; @@ -300,23 +395,36 @@ export function evaluateWatermarkFastpath( const windowCounts = stripRawStage(windowFragment, "openai"); const headCores = openaiToCore(headFragment as OpenAIRequestBody); if (headCores.systemText !== seed.systemText) return miss("head-drift", true, tailIds); - // In-place head EDIT detection. The system-text check alone would miss - // an in-place rewrite of any non-system head element (dsh message-edit - // flows): counts stay equal, the stamp stays honest, and the stale - // seed byte would ride the outbound again — the costliest divergence - // class. Compare every stashed prefix core against its freshly parsed - // counterpart: openaiToCore is deterministic per element bytes, so - // structural inequality (cluster id / role / content) marks a rewrite. - // Uncredited: a genuine client edit is not a lying stamp, so the - // breaker must not arm on it; finalizeWatermarkTurn reseeds from the - // legacy full parse of THIS turn, so the very next turn hits against - // the new baseline (self-healing). Shape mismatch covers small - // histories where the head fragment reaches past seed coverage — - // those turns legitimately go legacy. - // Compare ONLY the part of the head still under seed coverage: on small - // histories the 8-element head fragment legitimately reaches INTO the - // fresh tail region, whose content differs from nothing stashed yet — - // positions >= coverage are verified by the window conversion itself. + // In-place head EDIT detection (pre-v5 history): the system-text + // check alone misses an in-place rewrite of any non-system head + // element (dsh message-edit flows) — counts stay equal, stamp stays + // honest, stale seed byte rides outbound again. Pre-v5 that + // comparison GATED: openaiToCore is deterministic per element bytes, + // structural inequality marked a rewrite, uncredited self-healing + // legacy turn (a genuine client edit is not a lying stamp). + // v5 (owner ruling 2026-11-03, identity-only verification): with a + // shape-valid HOST msgid list the identity chain IS the proof — + // width arithmetic re-derives the count SERVER-side (never from a + // client echo), the prevId chain pins the window origin, sess-change + // rules out conversation swaps, model equality rules out cross-model + // reuse. Bili INTERNAL core ids are content-derived + // (deriveCanonicalMessageId(role,text)) — they carry no identity + // beyond the byte check itself. So under id mode BOTH the id- + // equality and byte-equality loops demote to a diagnostic warn: host + // rewrites ride the stashed prefix cores; the warn stream feeds M3 + // smart re-seeding should the rides ever prove harmful. Pure-COUNT + // clients have no ids at all — their byte check remains the sole + // truth and behaves exactly as pre-v5. System text has no msgid + // (flagged deviation in the ruling thread): head-drift above STAYS + // gating in both modes — serving a stale prompt beats saving one + // parse. Compare ONLY the part of the head still under seed coverage: + // on small histories the 8-element head fragment legitimately reaches + // INTO the fresh tail region, whose content differs from nothing + // stashed yet — positions >= coverage are verified by the window + // conversion itself. + let contentDrifts = 0; + let driftTotal = 0; + let driftRides: number | undefined; const headElements = Array.isArray((headFragment as { messages?: unknown }).messages) ? ((headFragment as { messages: unknown[] }).messages.filter((m): m is { role?: unknown } => typeof m === "object" && m !== null && (m as { role?: unknown }).role !== "system")).length : -1; @@ -324,12 +432,26 @@ export function evaluateWatermarkFastpath( const compareUpTo = Math.min(headElements, headCores.msgs.length, seed.cores.length); const seedHead = seed.cores.slice(0, compareUpTo); if (headCores.msgs.length < seedHead.length) return miss("head-shape", false, tailIds); + driftTotal = seedHead.length; for (let hi = 0; hi < seedHead.length; hi++) { const a = headCores.msgs[hi]; const b = seedHead[hi]; - if (a.id !== b.id || JSON.stringify(a) !== JSON.stringify(b)) return miss("head-prefix-drift", false, tailIds); + if (a.id !== b.id || JSON.stringify(a) !== JSON.stringify(b)) { + if (tailIds === undefined) return miss("head-prefix-drift", false, tailIds); + contentDrifts += 1; + } } } + if (contentDrifts > 0) { + // #5576 review V1: cap the episode — after PF_V5_DRIFT_RIDE_CAP + // consecutive drift turns one UNcredited legacy turn reseeds from + // real bytes (any non-canonical miss reseeds in finalize), then + // rides may resume. Self-heal by design: never breaker fuel. + const rides = (saved?.d ?? 0) + 1; + if (rides > PF_V5_DRIFT_RIDE_CAP) return miss("v5-drift-cap", false, tailIds); + log("warn", `[${session.id}] watermark v5: ${contentDrifts}/${driftTotal} head element(s) differ from the seed under stable host ids — riding the stashed prefix per owner ruling 2026-11-03 (diagnostic only; feeds M3 re-seeding signals; ${rides}/${PF_V5_DRIFT_RIDE_CAP} consecutive ride turns, next forces a reseed)`); + driftRides = rides; + } const windowRaw = openaiToCore(windowFragment as OpenAIRequestBody); if (windowRaw.systemText.trim() !== "") return miss("window-system-head", true, tailIds); const rekeyedWindow = rekeyWindowCores(windowRaw.msgs, K, seed.cores); @@ -352,11 +474,470 @@ export function evaluateWatermarkFastpath( decision: hitDecision, creditedMiss: false, receivedTailIds: tailIds, + clientSess: nowCs, + driftRides, }; } return miss(seed === undefined ? "unseeded" : "seed-stale", false); } +// #5576 v6 companion switch (env-only in v1): host-side rewrites that survive +// every filter and break the row-chain are content-rewrite class. Default: +// self-healing UNcredited legacy turns — host compaction/rewrite is normal +// traffic, not misbehavior (the legacy rewrite-detection machinery then +// marks the boundary and subsequent turns align against the rebased view). +// Strict mode credits them toward the breaker for exotic hosts whose +// thrash-rewriting would otherwise ride one free legacy turn forever. +function rewriteStrict(): boolean { + return process.env.BILI_PF_V6_REWRITE_STRICT === "1"; +} + +/** + * #5576 v6 canonical lane. Decision authority = transcript row-chain + * alignment ONLY; the surviving pre-filters are pure server-side arithmetic + * over client-declared ledger ids (negative filters, no byte comparison): + * ahead echoed stamp above our own measurement (lying upstream); + * tail-ids-shape ledger list width ≠ T - K + 1 (shrink/gap/misframe); + * no-progress count-only zero delta (empty window would pin nothing); + * sess-change swapped client-native session identity (R7, unchanged). + * The expensive stage converts the body once with the REAL codec (openaiToCore + * over the same bytes the legacy path would parse — zero codec mirroring, + * so cluster numbering can only agree with stored rows, never drift from it) + * and hands the cores to alignClientAgainstTranscript. On ok: view = stored + * rows up to the seam (covered regions suppressed behind synthetic anchors) + * + this turn's tail cores; system text is re-decoded from the CURRENT head + * every turn (passthrough by construction — there is no stashed system copy + * to drift from, which retires the v5 head-drift gate entirely). On any miss + * reason the caller runs the untouched legacy full-parse path; chain-class + * reasons (chain-not-started / seam-too-short) are content-rewrite class and + * credit per BILI_PF_V6_REWRITE_STRICT. + */ +function evaluateCanonicalV6(args: { + session: Session; + bodyText: string; + scan: { total: number; messageSpans: readonly RawSpan[]; fieldSpans: ReadonlyMap }; + saved: PFSavedState | null; + stamped: number; + parsedTailIds: string[] | null | undefined; + nowCs: string | undefined; + miss: (decision: string, credited: boolean, receivedTailIds?: string[]) => FastpathOutcome; +}): FastpathOutcome { + const { session, bodyText, scan, saved, stamped, parsedTailIds, nowCs, miss } = args; + const mm = (decision: string, credited: boolean, receivedTailIds?: string[]): FastpathOutcome => + ({ ...miss(decision, credited, receivedTailIds), canonical: true }); + const T = scan.total; + if (!Number.isNaN(stamped) && stamped > T) return mm("ahead", true); + // Width arithmetic on the client-declared ledger list (server-side check; + // count-only clients carry no list and skip to pure alignment — v6 is + // ALSO their first watermark access, the stamp echo is not required). + let tailIds: string[] | undefined; + const K = saved?.t; + if (parsedTailIds !== undefined) { + if (parsedTailIds === null || (K !== undefined && parsedTailIds.length !== T - K + 1)) return mm("tail-ids-shape", false); + tailIds = parsedTailIds; + } else if (K !== undefined && T <= K) { + return mm("no-progress", false); + } + // R7 (unchanged from v5): a swapped client-native session riding one bili + // session is an identity event, asserted only when both sides claim one. + if (saved?.cs !== undefined && nowCs !== undefined && nowCs !== saved.cs) return mm("sess-change", false, tailIds); + // Cheap structural pre-gates before paying for a conversion: + const store = getStore(); + if (!store || !store.replicasAvailable()) return mm("v6-store-unavailable", false, tailIds); + // One prior fold anchors the rows (unified-lane doctrine, #3482 client- + // authoritative pre-fold); pre-fold sessions run the untouched path. + if (!session.state.blocks.some((b) => b.active)) return mm("v6-pre-fold", false, tailIds); + // Conversion range: the WHOLE body when affordable — the intra-call + // cluster counter must see exactly the same occurrence stream the legacy + // single-parse minted (windowed conversion would renumber duplicate + // contents and desync the id fast path); the walk budget caps how far + // alignment ever looks, so beyond MAX_CHAIN_WALK elements we convert the + // walk-reachable suffix only (documented residual: dup-content bodies + // past the cap lean on the sig fallback there). + const spans = scan.messageSpans; + const lo = T > MAX_CHAIN_WALK ? T - MAX_CHAIN_WALK : 0; + const midBody: { messages: unknown } = { messages: [] }; + midBody.messages = parseElementSlices(bodyText, spans.slice(lo)); + if (!Array.isArray(midBody.messages)) return mm("fragment-parse", false, tailIds); + const midCounts = stripRawStage(midBody, "openai"); + const cores = openaiToCore(midBody as OpenAIRequestBody); + const res = alignClientAgainstTranscript(session, cores.msgs as BiliMessage[]); + if (!res.ok) { + const rewriteClass = res.reason === "chain-not-started" || res.reason === "seam-too-short"; + return mm(`v6-${res.reason}`, rewriteClass && rewriteStrict(), tailIds); + } + const clientJoin = res.clientJoin; + // A full-length join (every converted core "anchored", no unmatched + // trailing tail) is the signature of the documented sig-fallback + // residual: a FRESH trailing element whose content collides with stored + // history false-matches an old row through the sig path and swallows the + // new tail (view would silently drop it). Degrade to legacy instead — + // never serve a hit that ends before the request does. + if (typeof clientJoin !== "number" || clientJoin >= cores.msgs.length) return mm("v6-no-seam-origin", false, tailIds); + // Element-space window origin: codecs stamp each core with wireIndex = + // its source element ordinal in the converted array (kernel contract, + // bili-message sidecar doc). Windowed conversions number relative to lo. + const seamWi = (cores.msgs[clientJoin] as { wireIndex?: unknown }).wireIndex; + let eStar: number; + if (typeof seamWi === "number") eStar = lo + seamWi; + else if (K !== undefined) eStar = K; // codec gap: fall back to the persisted anchor + else return mm("v6-no-seam-origin", false, tailIds); + if (!(eStar >= lo && eStar < T)) return mm("v6-no-seam-origin", false, tailIds); + // System passthrough (R3): decoded fresh from the CURRENT head every + // turn — rows exclude hoisted system by design, so this is runtime state + // flow, not prefix identity. The ≤8-element fragment also carries the + // strip counts for the pre-window region on capped conversions. + const headEnd = Math.min(HEAD_CHECK_ELEMENTS, T); + const headFragment: { messages: unknown } = { messages: [] }; + headFragment.messages = parseElementSlices(bodyText, spans.slice(0, headEnd)); + if (!Array.isArray(headFragment.messages)) return mm("fragment-parse", false, tailIds); + const headCounts = stripRawStage(headFragment, "openai"); + const systemText = openaiToCore(headFragment as OpenAIRequestBody).systemText ?? ""; + const markerCounts = lo > 0 ? addStripCounts(headCounts, midCounts) : midCounts; + return { + hit: { + view: res.view as CoreMessage[], + windowCores: cores.msgs.slice(clientJoin) as CoreMessage[], + systemText, + totalT: T, + fieldSpans: scan.fieldSpans, + windowMessages: midBody.messages.slice(eStar - lo), + // Canonical lanes serve ACTUALLY-ingested cores: sidecars + // (imagePart/rawOpenaiContent) survive the fold pipeline intact + // and the egress positional rule computes over the real array — + // the v5 multimodal degradation exists solely because the seed + // STASH could not reproduce legacy bytes, which v6 has none of. + seedMultimodal: false, + markerCounts, + }, + measuredT: T, + decision: "hit-v6", + creditedMiss: false, + receivedTailIds: tailIds, + clientSess: nowCs, + canonical: true, + }; +} + +// ============================================================================ +// #5600 v6 msgid lane — owner rulings 2026-11-03, issue §3.1/§5/§6/§8. +// Identity: server-MINTED pf_sid (§3.1) — echo header x-bili-pf-session, +// ledger-id probe over x-bili-pf-tail-ids, mint as the last resort; client- +// native ids play NO identity role. Data: the pf_* partition store +// (append-only pf_rows + zero-state LSM metadata, cleanup rides +// deleteSession only). Hit authority: ledger-width arithmetic against OUR +// OWN persisted frontier (never a client echo — the stamp demotes fully) + +// the window-origin pin; local edits/invisible reorders ride by doctrine +// (§6.2 corner rulings). Frozen cores come from the RESIDENT array — +// hydrated from the partition after boot, bytes never re-derived (this is +// the P1 retention pair the feature exists to remove). Fresh cores decode +// from the CURRENT window fragment ONLY and are rekeyed onto the prefix +// numbering (#3642, same contract as v5); system text re-decodes from the +// head each turn (passthrough, no stashed copy to drift from). Semantic +// misses (arm / compaction-reset) re-base the partition and resident on the +// freshly parsed full view in finalize (§6.1: compaction = archive + reset, +// transcript stays durable). Every doubt degrades to the untouched legacy +// full-parse. SOFT plane: the store unavailable ⇒ null ⇒ the pre-existing +// lanes below run byte-for-byte as before. +const PF_SID_RE = /^pf_[0-9a-f]{12}$/u; +/** Session-ext registry naming every pf_sid minted under one bili session — + * deleteSession sweeps it (store cascade, no independent retention period). */ +const PF_EXT_SIDS_KEY = "pf:sids"; +/** Bounded probe budget for identity resolution from a tail-id list. */ +const PF_PROBE_BUDGET = 8; + +interface PFResident { + sid: string; + /** Frozen ingested cores of the ACTIVE world — the full view at the last + * ingest: reset-world cores plus every subsequent window, in order. */ + cores: CoreMessage[]; + /** Element count T of the last ingest (the ledger frontier in ELEMENTS — + * the window always decodes from spans[K..T), never from a client echo). */ + elems: number; + /** Exclusive high pf_row pos consumed so far (position cursor). */ + rowHi: number; + /** Origin (inclusive pf_row pos) of the ACTIVE world — the latest + * arm/reset fromPos (cold start: 0). Hydration reads exactly + * [worldOrigin .. rowHi): WINDOW starts (`f`) are never treated as the + * frontier — they mark where each turn's window began, one per window, + * not where the current world began (#5576 review B1). */ + worldOrigin: number; + /** Sticky: some resident core carries an image part. */ + imgFlag: boolean; + /** Cumulative strip ledger of ALL ingested bytes (carried forward — the + * middle is never re-scanned; each geom entry snapshots it so folds + * reconstruct it without a re-scan). */ + markers: RawStripCounts; +} +const residents = new WeakMap(); + +function mintPfSid(): string { + return `pf_${randomBytes(6).toString("hex")}`; +} + +/** Soft-plane accessor (§4.1.6): store absent, unshaped or misbehaving ⇒ + * null ⇒ this lane stands aside and the pre-existing lanes run byte-for- + * byte. NEVER throws into the request path. */ +function pfAvailable(): NonNullable>["pfStore"]>> | null { + try { + const s = getStore(); + return s?.pfStore ? s.pfStore() ?? null : null; + } catch { + return null; + } +} + +/** Register a minted sid under the owning bili session (advisory — the sweep + * only ever under-deletes an orphan at worst, never other sessions' data). */ +function registerPfSid(session: Session, sid: string): void { + try { + const store = getStore(); + const cur = store.getSessionExt(session.id, PF_EXT_SIDS_KEY) as unknown; + const list = Array.isArray(cur) + ? cur.filter((s): s is string => typeof s === "string" && s !== sid) + : []; + list.push(sid); + store.setSessionExt(session.id, PF_EXT_SIDS_KEY, list); + } catch { /* registry is advisory */ } +} + +/** Rebuild the resident array from the partition (post-boot / lane-toggle / + * resident-dropped). Fold: L1 checkpoint meta (last geom's cumulative state) + * + in-order L0 tail replay; 'reset'/'arm' entries move the active frontier + * to their recorded fromPos (archive stays durable — rows are only deleted + * by deleteSession). Any unrecoverable shape degrades to null. */ +function hydratePf( + pf: NonNullable>["pfStore"]>>, + session: Session, + sid: string, + log: (level: string, msg: string) => void, +): PFResident | null { + const maxP = pf.maxPos(sid); + if (maxP < 0) return null; + const meta = pf.readMeta(sid); + // Frontier reconstruction (#5576 review B1): the world frontier may be + // moved ONLY by arm/reset entries (their fromPos) — geom entries record + // their OWN window's pre-append start (`f`), which must never be read + // back as the frontier (replaying them walked the frontier backwards + // window-by-window, hydrating a truncated core set while `t` still + // claimed full width). Folding deletes the arm/reset entry, so the + // checkpoint carries the origin as `o`; pre-fix checkpoints lack `o` + // (and use the old checksum shape) — fail closed to legacy until the + // world re-arms. + let worldOrigin: number | null = null; + let elems = 0; + let imgFlag = false; + let markers: RawStripCounts = { panels: 0, statusBodies: 0, statusBodyLines: 0, markerLines: 0, nudges: 0, carriers: 0 }; + const applyGeom = (p: Record): void => { + if (typeof p.t === "number") elems = p.t; + if (typeof p.imgFlag === "boolean") imgFlag = p.imgFlag; + if (p.markers !== null && typeof p.markers === "object") { + const m = p.markers as Record; + const pick = (k: keyof RawStripCounts): number => (typeof m[k] === "number" && Number.isInteger(m[k] as number) && (m[k] as number) >= 0 ? (m[k] as number) : 0); + markers = { panels: pick("panels"), statusBodies: pick("statusBodies"), statusBodyLines: pick("statusBodyLines"), markerLines: pick("markerLines"), nudges: pick("nudges"), carriers: pick("carriers") }; + } + }; + if (meta.ckpt !== null) { + // Review S4: verify the checksum NOW — it covers the meta_json bytes + // themselves (pre-fix it hashed a {t} subset and was decorative). An + // UNTRUSTED checkpoint is skipped rather than fatal: the live tail + // may still hold the arm/reset naming the frontier, and only a fully + // unresolvable shape degrades (below) to legacy. + let ckptTrusted = true; + if (createHash("sha256").update(meta.ckpt.metaJson).digest("hex") !== meta.ckpt.checksum) { + log("warn", `[${session.id}] watermark v6-msgid: checkpoint checksum mismatch — ignoring checkpoint state`); + ckptTrusted = false; + } + if (ckptTrusted) { + try { + const m = JSON.parse(meta.ckpt.metaJson) as { s?: Record; o?: number }; + if (typeof m.s === "object" && m.s !== null) applyGeom(m.s); + worldOrigin = typeof m.o === "number" && Number.isInteger(m.o) && m.o >= 0 ? m.o : null; + } catch { + log("warn", `[${session.id}] watermark v6-msgid: undecodable checkpoint payload — ignoring checkpoint state`); + worldOrigin = null; + } + } + } + for (const e of meta.tailLog) { + let p: Record; + try { p = JSON.parse(e.payloadJson) as Record; } catch { return null; } + if (e.kind === "geom") applyGeom(p); + else if (e.kind === "reset" || e.kind === "arm") { + if (typeof p.t === "number") elems = p.t; + worldOrigin = typeof p.fromPos === "number" ? p.fromPos : maxP; + } + } + if (elems === 0) return null; // no completed ingest yet — arm flow owns this + if (worldOrigin === null) return null; // no trustworthy frontier (unfolded state missing arm/reset, or pre-fix ckpt) — legacy until re-arm + const rows = pf.readRows(sid, [[worldOrigin, maxP + 1]]); + const cores: CoreMessage[] = []; + for (const r of rows) { + try { cores.push(JSON.parse(r.coreJson) as CoreMessage); } + catch { log("warn", `[${session.id}] watermark v6-msgid: undecodable core row at ${r.pos} — resident rebuild degraded`); return null; } + } + if (cores.length === 0) return null; + const res: PFResident = { sid, cores, elems, rowHi: maxP + 1, worldOrigin, imgFlag, markers }; + residents.set(session, res); + return res; +} + +/** The lane's verdict for one turn. null = pf plane unavailable (caller + * falls through to the pre-existing lanes); anything else is authoritative + * for this turn (hit or miss — a miss still routes the LEGACY full-parse + * exactly like every other lane, and echoes the sid back). Read-side faults + * (SQLITE_BUSY under multi-instance sharing, disk full, ...) degrade to + * null the same way — never a 5xx on a live request (#5576 review S3). */ +function evaluateMsgIdLane(args: MsgIdLaneArgs): FastpathOutcome | null { + try { + return evaluateMsgIdLaneCore(args); + } catch (err) { + args.log( + "warn", + `[${args.session.id}] watermark v6-msgid: read-side fault (${err instanceof Error ? err.message : String(err)}) — legacy full-parse this turn`, + ); + return null; + } +} +type MsgIdLaneArgs = { + session: Session; + req: http.IncomingMessage; + bodyText: string; + scan: { total: number; messageSpans: readonly RawSpan[]; fieldSpans: ReadonlyMap }; + saved: PFSavedState | null; + parsedTailIds: string[] | null | undefined; + nowCs: string | undefined; + log: (level: string, msg: string) => void; +}; +function evaluateMsgIdLaneCore(args: MsgIdLaneArgs): FastpathOutcome | null { + const { session, req, bodyText, scan, saved, parsedTailIds, nowCs, log } = args; + const T = scan.total; + const pf = pfAvailable(); + if (pf === null) return null; // soft availability — §4.1.6 待裁④ default + const tailValid = parsedTailIds !== undefined && parsedTailIds !== null ? parsedTailIds : undefined; + const out = (decision: string, credited: boolean, extra?: Partial): FastpathOutcome => { + if (credited) noteCreditedMiss(session.id, decision, log); + log("info", `[${session.id}] watermark v6-msgid miss (${decision}) — legacy full-parse this turn`); + return { hit: null, measuredT: T, decision, creditedMiss: credited, receivedTailIds: tailValid, clientSess: nowCs, canonical: true, ...extra }; + }; + // ---- ① identity resolution (§3.1, priority: echo → probe → mint) ------ + const rawHdr = req.headers[PF_SID_HEADER]; + const echoed = (Array.isArray(rawHdr) ? String(rawHdr[0] ?? "") : String(rawHdr ?? "")).trim(); + let sid: string | undefined; + if (PF_SID_RE.test(echoed)) { + const meta = pf.readMeta(echoed); + if (pf.maxPos(echoed) >= 0 || meta.ckpt !== null || meta.tailLog.length > 0) sid = echoed; + else log("warn", `[${session.id}] watermark v6-msgid: echo of unknown pf_sid treated as fabrication — falling back to probe`); + } + if (sid === undefined && tailValid !== undefined) { + let budget = PF_PROBE_BUDGET; + for (const id of tailValid) { + if (id === PF_SLOT_UNKNOWN) continue; + const found = pf.lookupLedger(id); + if (found !== null) { sid = found.sid; break; } + if (--budget <= 0) break; + } + } + if (sid === undefined) { + // §8 case 7 cold start: the FIRST full parse IS the arm — P5 accepts + // one full materialization here and never again for this world. + const minted = mintPfSid(); + registerPfSid(session, minted); + return out("v6m-arm", false, { sid: minted }); + } + // ---- ② resident (in-process) or hydration (post-boot) ----------------- + let res: PFResident | null = residents.get(session) ?? null; + if (res === null || res.sid !== sid) res = hydratePf(pf, session, sid, log); + if (res === null) return out("v6m-hydrate", false, { sid }); + // R7 parity (unchanged from v5): swapped client-native session riding one + // bili session is an identity event, asserted only when both sides claim. + if (tailValid !== undefined && saved?.cs !== undefined) { + if (nowCs !== undefined && nowCs !== saved.cs) return out("sess-change", false, { sid }); + } + // ---- ③ world-shape arithmetic against OUR frontier --------------------- + const K = res.elems; + if (T < K) { + // §6.1 client compaction (announce class): shrink-with-evidence of a + // world change. Default = passthrough-as-new-message (uncredited — + // host compaction is normal traffic); BILI_PF_V6_REWRITE_STRICT + // credits it toward the breaker for thrashing exotic hosts. + return out("v6m-compact", rewriteStrict(), { sid }); + } + if (T === K) return out("v6m-no-progress", false, { sid }); // empty window pins nothing (v5 parity) + const W = T - K; + // ---- ④ window origin (only real proof needed: WHERE does new begin) ---- + let tailIds: string[] | undefined; + if (parsedTailIds !== undefined) { + if (parsedTailIds === null || parsedTailIds.length !== W + 1) return out("v6m-shape", false, { sid }); + tailIds = parsedTailIds; + const lastRow = res.rowHi > 0 ? pf.readRows(sid, [[res.rowHi - 1, res.rowHi]])[0] : undefined; + // Frontier coherence gate: the last stored row must BE element K-1 + // (absolute body ordinal, system hoisted out by design). A row that + // does not line up means the resident lost sync with the partition — + // self-heal via a legacy turn rather than trust the arithmetic. + if (lastRow === undefined || lastRow.elem !== K - 1) return out("v6m-shape", false, { sid }); + if (tailIds[0] !== PF_SLOT_UNKNOWN) { + if (pf.verifyOriginPin(sid, K - 1, tailIds[0])) { + // Hard pin: the claimed id is recorded on element K-1 of OUR + // partition — exact regardless of how many rows that element + // split into (multimodal pairs share one id). + } else if ((lastRow.ledgerId ?? null) === null) { + // Weak pin (null-row transition): the frontier row was ingested + // without a ledger id (client id-silent on the arm/reset turn). + // No recorded id can be drifted from, so any evidence for the + // slot — including an unbound real id — is accepted; the width + // arithmetic + window decode still guard the content, and every + // later turn re-binds through newly stored ids. A client that + // says "?" against a RECORDED id still fails the hard branch. + } else { + // Unknown / foreign / misplaced ⇒ world drift ⇒ self-healing + // legacy turn, resident untouched (retry next turn). Strictly + // stronger than v5's string compare: identity resolves inside + // OUR partition, never against a client echo. + return out("v6m-prev", false, { sid }); + } + } + } + // ---- ⑤ decode: head (system passthrough) + window ONLY ----------------- + const spans = scan.messageSpans; + const headEnd = Math.min(HEAD_CHECK_ELEMENTS, T); + const headFragment: { messages: unknown } = { messages: [] }; + const windowFragment: { messages: unknown } = { messages: [] }; + headFragment.messages = parseElementSlices(bodyText, spans.slice(0, headEnd)); + windowFragment.messages = parseElementSlices(bodyText, spans.slice(K)); + if (!Array.isArray(headFragment.messages) || !Array.isArray(windowFragment.messages)) return out("fragment-parse", false, { sid, receivedTailIds: tailIds }); + stripRawStage(headFragment, "openai"); // normalize before convert; head bytes were already counted when ingested + const windowCounts = stripRawStage(windowFragment, "openai"); + const windowRaw = openaiToCore(windowFragment as OpenAIRequestBody); + // Hoist hazard (parity with v5/canonical): a system/developer element at + // the window head poisons the view — credited, since it signals real drift. + if (windowRaw.systemText.trim() !== "") return out("v6m-window-system", true, { sid, receivedTailIds: tailIds }); + const rekeyedWindow = rekeyWindowCores(windowRaw.msgs, K, res.cores); + const headCores = openaiToCore(headFragment as OpenAIRequestBody); + return { + hit: { + view: [...res.cores, ...rekeyedWindow] as CoreMessage[], + windowCores: rekeyedWindow as CoreMessage[], + systemText: headCores.systemText ?? "", + totalT: T, + fieldSpans: scan.fieldSpans, + windowMessages: windowFragment.messages, + // ACTUAL ingested cores are served (canonical-lane doctrine): the + // egress positional rule computes over the real array, so the v5 + // stash-multimodal degradation has no referent here. + seedMultimodal: false, + markerCounts: addStripCounts(res.markers, windowCounts), + }, + measuredT: T, + decision: "hit(v6-mid)", + creditedMiss: false, + receivedTailIds: tailIds, + clientSess: nowCs, + canonical: true, + sid, + }; +} + /** #3642 identity continuity. openaiToCore mints cluster ids (base, or base_N * by INTRA-CALL occurrence order) and stamps wireIndex from the array position * it was given. The window fragment was parsed alone, so both are @@ -439,14 +1020,14 @@ function addStripCounts(a: RawStripCounts, b: RawStripCounts): RawStripCounts { } function noteCreditedMiss(sessionId: string, decision: string, log: (level: string, msg: string) => void): void { - const cur = readSaved(sessionId) ?? { t: 0, m: "", misses: 0 }; + const cur = readSaved(sessionId) ?? ({ t: 0, m: "", misses: 0 } as PFSavedState); // an EXPIRED hold has already paid its penalty — count fresh from zero, // or the very first verified miss after the TTL would instantly re-arm // (the persisted counter sits at PF_BREAKER_MISSES from the original trip). const staleHold = cur.disabledAt !== undefined && Date.now() - cur.disabledAt >= PF_DISABLED_TTL_MS; const misses = (staleHold ? 0 : cur.misses) + 1; if (misses >= PF_BREAKER_MISSES) { - writeSaved(sessionId, { t: cur.t, m: cur.m, misses, disabledAt: Date.now() }); + writeSaved(sessionId, { t: cur.t, m: cur.m, misses, disabledAt: Date.now(), cs: cur.cs, d: cur.d }); log("warn", `[${sessionId}] watermark fastpath disabled for ${Math.round(PF_DISABLED_TTL_MS / 60000)}m after ${misses} consecutive verified misses (last: ${decision}) — legacy full-parse continues`); } else { writeSaved(sessionId, { ...cur, misses, disabledAt: staleHold ? undefined : cur.disabledAt }); @@ -473,21 +1054,122 @@ export function finalizeWatermarkTurn( if (outcome.measuredT === undefined) return headers; const T = outcome.measuredT; const saved = readSaved(session.id); + // #5600 msgid plane: persistent residency replaces seed pinning. Hits + // append this turn's window rows + a geom snapshot; semantic misses + // (arm / compaction-reset) re-base the partition on the freshly parsed + // FULL view (§6.1 archive-and-reset — rows are never deleted here). All + // other miss classes write NOTHING (retry next turn against the same + // frontier). The shared writeSaved block below still runs for every + // outcome: ext state keeps t/m/l/cs live for debug + fallback lanes. + if (outcome.sid !== undefined) { + try { + const pf = pfAvailable(); + if (pf !== null) { + const res = residents.get(session); + if (outcome.hit !== null) { + if (res !== undefined && res.sid === outcome.sid) { + // Element→ledger-id map from THIS turn's list (offset 1: + // slot 0 is the origin pin); "?" slots ride id-less. + const idByElem = new Map(); + outcome.receivedTailIds?.slice(1).forEach((id, i) => { if (id !== PF_SLOT_UNKNOWN) idByElem.set(res.elems + i, id); }); + const rows = outcome.hit!.windowCores.map((c) => { + const wi = (c as { wireIndex?: unknown }).wireIndex; + return { + ledgerId: typeof wi === "number" ? idByElem.get(wi) ?? null : null, + // Absolute body-element ordinal (kernel stamps + // wireIndex from the source array); split cores + // of one multimodal element share it. + elem: typeof wi === "number" ? wi : -1, + coreId: c.id, + coreJson: JSON.stringify(c), + }; + }); + const geom = { kind: "geom", payload: { t: T, model: input.model, f: res.rowHi, imgFlag: res.imgFlag, markers: outcome.hit!.markerCounts }, bornPosHi: res.rowHi + rows.length } as const; + const marks = pf.appendTurn(outcome.sid, rows, [geom]); + residents.set(session, { + sid: outcome.sid, + cores: outcome.hit!.view, + elems: T, + rowHi: marks.maxPos + 1, + worldOrigin: res.worldOrigin, + imgFlag: res.imgFlag || coresHaveImagePart(outcome.hit!.windowCores), + markers: outcome.hit!.markerCounts, + }); + // L1 fold upkeep: collapse the L0 tail every 32 log + // entries so post-boot hydration stays bounded. + if (marks.lsSeq > 0 && marks.lsSeq % 32 === 0) { + const r2 = residents.get(session)!; + // B1/S4: the checkpoint carries the active world + // ORIGIN (`o`) — its tail fold may have deleted the + // arm/reset entry that recorded it — and its + // checksum covers the meta_json bytes. + const ckptMeta = JSON.stringify({ s: { t: T, imgFlag: r2.imgFlag, markers: r2.markers }, o: r2.worldOrigin }); + pf.writeCkpt(outcome.sid, marks.lsSeq, ckptMeta, createHash("sha256").update(ckptMeta).digest("hex")); + } + } + } else if ((outcome.decision === "v6m-arm" || outcome.decision === "v6m-compact" || outcome.decision === "v6m-hydrate") && input.legacyCores.length > 0) { + // §8 case-7 arm / §6.1 reset / failed-hydration self-heal: + // ingest the freshly parsed FULL view at CONTINUING + // positions (the old world stays archived in the + // partition — rollback reference). The v6m-hydrate leg + // re-bases a partition whose frontier cannot be + // reconstructed (untrusted checkpoint + empty tail), + // bounding the degradation to ONE legacy turn: the + // appended reset entry carries fromPos, so the NEXT turn + // hydrates cleanly off it. + const fromPos = res !== undefined && res.sid === outcome.sid ? res.rowHi : pf.maxPos(outcome.sid) + 1; + const idByElem = new Map(); + outcome.receivedTailIds?.forEach((id, i) => { if (id !== PF_SLOT_UNKNOWN) idByElem.set(i, id); }); + const rows = input.legacyCores.map((c) => { + const wi = (c as { wireIndex?: unknown }).wireIndex; + return { + ledgerId: typeof wi === "number" ? idByElem.get(wi) ?? null : null, + elem: typeof wi === "number" ? wi : -1, + coreId: c.id, + coreJson: JSON.stringify(c), + }; + }); + const entryKind = outcome.decision === "v6m-arm" ? "arm" : "reset"; + const entry = { kind: entryKind, payload: { t: T, fromPos }, bornPosHi: fromPos + rows.length } as const; + const marks = pf.appendTurn(outcome.sid, rows, [entry]); + residents.set(session, { + sid: outcome.sid, + cores: [...input.legacyCores], + elems: T, + rowHi: marks.maxPos + 1, + worldOrigin: fromPos, + imgFlag: coresHaveImagePart(input.legacyCores), + markers: input.turnStripCounts, + }); + } + } + } catch { /* pf plane is soft — degrade, never break the turn */ } + // Identity echo channel (§3.1): responses always carry the sid, which + // ALSO issues it on the mint turn (the client stores what it gets). + headers[PF_SID_HEADER] = outcome.sid; + } if (outcome.hit) { - seeds.set(session, { - cores: outcome.hit.view, - systemText: outcome.hit.systemText, - stampT: T, - model: input.model, - stripCounts: outcome.hit.markerCounts, - lastTailIds: outcome.receivedTailIds, - multimodal: outcome.hit.seedMultimodal, - }); - writeSaved(session.id, { t: T, m: input.model, misses: 0, l: outcome.receivedTailIds }); + // #5576 v6: canonical outcomes pin NOTHING prefix-shaped in memory — + // retaining the view here would recreate the very OOM pair (#5403) + // the lane removes. A lane toggle back to v5 self-heals in one turn. + if (!outcome.canonical) { + seeds.set(session, { + cores: outcome.hit.view, + systemText: outcome.hit.systemText, + stampT: T, + model: input.model, + stripCounts: outcome.hit.markerCounts, + lastTailIds: outcome.receivedTailIds, + multimodal: outcome.hit.seedMultimodal, + }); + } + // d: the ride count THIS hit carries (0 on every non-drift hit — a + // clean head resets the episode). + writeSaved(session.id, { t: T, m: input.model, misses: 0, l: outcome.receivedTailIds, cs: outcome.clientSess ?? saved?.cs, d: outcome.driftRides ?? 0 }); headers[PF_SEQ_HEADER] = String(T); return headers; } - if (input.legacyCores.length > 0) { + if (input.legacyCores.length > 0 && !outcome.canonical) { seeds.set(session, { cores: input.legacyCores, systemText: input.rawSystemText, @@ -500,16 +1182,23 @@ export function finalizeWatermarkTurn( } // Held (breaker-armed) turns flow through here too and DO restamp: the // stamp is how a client converges on truth while the lane stays hit-less. - writeSaved(session.id, { t: T, m: input.model, misses: saved?.misses ?? 0, disabledAt: saved?.disabledAt, l: outcome.receivedTailIds }); + // d follows the SEED: a miss that reseeds from real bytes starts a fresh + // ride episode; misses that keep the old seed carry the counter (canonical + // lane outcomes never touch the v5 seed). + const reseeded = input.legacyCores.length > 0 && !outcome.canonical; + writeSaved(session.id, { t: T, m: input.model, misses: saved?.misses ?? 0, disabledAt: saved?.disabledAt, l: outcome.receivedTailIds, cs: outcome.clientSess ?? saved?.cs, d: reseeded ? 0 : (saved?.d ?? 0) }); headers[PF_SEQ_HEADER] = String(T); return headers; } /** Drop the in-process seed (e.g. an announced compaction was detected while * the fastpath was silent on later turns). Cheap and idempotent; the next - * turn simply re-verifies via the legacy path. */ + * turn simply re-verifies via the legacy path. Also drops the #5600 msgid- + * lane resident — the partition on disk is the truth and re-hydrates on + * demand, so dropping memory never loses anything. */ export function clearWatermarkSeed(session: Session): void { seeds.delete(session); + residents.delete(session); } /** Introspection for panel/tests: whether a live prefix stash exists. */ diff --git a/src/server/handle.ts b/src/server/handle.ts index f944ca4cac..11e8e64089 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -34,6 +34,8 @@ import { codexCompactGatePre, codexCompactMode, hasCompactionTrigger, isCodexHos import { emitPreflightError } from "../stream-error.js"; import { agentProviderRecipes } from "../agent-providers.js"; import { affinityToken, anthropicFirstUserFrag, claudeAnthropicPersonaOrderApplies, conversationHeaderSource, claudeSubagentAgentId, claudeSubagentSplit, claudeSubagentLaneTokenOf, clientConversationHeader, codexTurnIdentity, dshPersonaFingerprintApplies, instructionsFingerprintApplies, openaiSystemTextForPersona, preferPromptCacheKeyIdentity, zcodeMemoryConversation } from "../session-id.js"; +import { observeNativeIdentity } from "./native-identity.js"; +import { featureEnabled } from "../rollout.js"; import { preferKiroConversationIdentity } from "../kiro.js"; import { personaConversationOrder } from "../session-resolve.js"; import { prefixAffinity, describeResumeMiss, type AnonymousAffinity } from "../prefix-affinity.js"; @@ -429,6 +431,18 @@ async function handleRequest( res.end(JSON.stringify({ error: { type: "invalid_request", message: String(err) } })); return; } + // #5573 M1 (L1): native-identity observation — pure logging + ext record, + // feeds no decision (zero-behavior-change contract), rollout-gated with + // default off. Sits ahead of the passthrough/bypass branches below so + // those lanes are observed too; the body is only touched when a matched + // profile declares body-level signals. + if (req.method === "POST" && bodyBuffer.length > 0 && featureEnabled("nativeIdentityObservation")) { + observeNativeIdentity( + { headers: req.headers, body: bodyBuffer, protocol, path: urlPath }, + getStore(), + log, + ); + } // #1757: resolve compat.dropFields once, ahead of the verbatim branches // that forward before reaching the final boundary (they can't use // forward()'s own resolution). Same destination derivation as diff --git a/src/server/native-identity-profiles.ts b/src/server/native-identity-profiles.ts new file mode 100644 index 0000000000..9ab0e5fe0e --- /dev/null +++ b/src/server/native-identity-profiles.ts @@ -0,0 +1,218 @@ +/** + * #5573 M1 (L1) — native-identity profile table (pure observation data). + * + * One row per known client family. Each profile declares: how to RECOGNIZE it + * on the wire (match headers / protocol), which header carries each identity + * granularity (session / turn / message), its explicit compaction signals, + * bounded text patterns counted over the (already decoded) body, and the + * compaction-preface texts its own compressor emits. Rows are DATA, not + * scattered comparisons: adding a client or correcting a drifted preface + * (each preface comes from independent binary forensics and CAN drift with + * client versions) is a table edit + a fixture test, not an architecture + * change. The env override (BILI_NATIVE_IDENTITY_PROFILES_JSON) may ADD or + * REPLACE rows by name but never remove built-ins — the built-in table is the + * floor (#5573 M1 design). + * + * All matching is lowercase-header-name based (Node lowercases inbound + * headers); body patterns are literal substrings located with indexOf — no + * regex, no JSON.parse (the chokepoint holds decoded BYTES, and parsing them + * is exactly the work #5403 removed from the hot path). + */ + +interface NativeIdPattern { + /** Literal substring to locate/count in the decoded body text. */ + pattern: string; + /** Label for the count in the observation line (short, grep-able). */ + label: string; + /** When true, a NON-zero count is itself an anomaly (protocol drift) and + * raises a shape_assert_fail event; such counts are always reported. */ + assertZero?: boolean; +} + +export interface NativeIdProfileDef { + name: string; + /** Any of these headers present (non-empty) makes the profile a match. */ + matchHeaders: readonly string[]; + /** Secondary match: this header must be present AND the wire protocol matches. */ + secondaryMatch?: { header: string; protocol: string }; + /** Protocol-only match (no header evidence), e.g. the Google wire. */ + matchProtocol?: string; + /** Session-level id candidates, first hit wins. */ + sessionHeaders: readonly string[]; + /** The session header's value must start with this prefix (shape gate). */ + sessionValuePrefix?: string; + /** Turn/thread-level marker candidates. */ + turnHeaders: readonly string[]; + /** Message-level id candidates (retry-stable where the host guarantees it). */ + msgidHeaders: readonly string[]; + /** Explicit compaction EVENT header (host attached it because IT just compacted). */ + compactionEdgeHeader?: string; + /** Compaction STATE header (this context was compacted — corroboration, never a trigger). */ + compactionLevelHeader?: string; + /** Values logged/stored as sha1-8 digest only (PII class, e.g. installation ids). */ + piHeaders: readonly string[]; + /** Informational host-fingerprint headers (name=value in the line tail). */ + fingerprintHeaders: readonly string[]; + /** Bounded substring counts over the body text (item-id families, anchor counts). */ + countPatterns: readonly NativeIdPattern[]; + /** Compaction-preface texts: occurrence inside a USER element = compaction event. */ + compactionPrefixes: readonly string[]; + /** Path suffix that is itself a compaction event (e.g. /responses/compact). */ + compactionPathSuffixes?: readonly string[]; + /** Dual-channel cross-check: body value that must equal the session header + * (mismatch = incident event). Located by a bounded windowed text walk. */ + dualSessionBodyAnchor?: { anchor: string; field: string; window: number }; + /** Top-level body field tracked for rotation (e.g. previous_response_id). */ + rotationBodyField?: string; +} + +const BUILTIN_PROFILES: readonly NativeIdProfileDef[] = [ + { + name: "wb", + matchHeaders: ["x-conversation-id"], + sessionHeaders: ["x-conversation-id"], + turnHeaders: ["x-conversation-request-id"], + msgidHeaders: ["x-conversation-message-id", "x-request-id"], + compactionEdgeHeader: "x-compaction", + compactionLevelHeader: "x-context-compacted", + piHeaders: [], + fingerprintHeaders: ["x-codebuddy-request", "x-ide-name", "x-ide-type", "x-ide-version"], + countPatterns: [], + compactionPrefixes: [], + }, + { + name: "codex", + matchHeaders: ["x-codex-installation-id"], + secondaryMatch: { header: "x-client-request-id", protocol: "responses" }, + sessionHeaders: ["session-id", "thread-id", "x-codex-window-id"], + turnHeaders: ["x-codex-turn-metadata", "turn-id"], + msgidHeaders: ["x-client-request-id", "x-request-id"], + piHeaders: ["x-codex-installation-id"], + fingerprintHeaders: [], + countPatterns: [ + { pattern: "\"msg_", label: "msg_" }, + { pattern: "\"ctc_", label: "ctc_" }, + { pattern: "\"ctco_", label: "ctco_" }, + { pattern: "\"rs_", label: "rs_" }, + { pattern: "\"internal_chat_message_metadata_passthrough\"", label: "turnmeta" }, + ], + compactionPrefixes: ["Another language model started to solve this problem"], + compactionPathSuffixes: ["/responses/compact"], + rotationBodyField: "previous_response_id", + }, + { + name: "cc", + matchHeaders: ["x-claude-code-session-id"], + sessionHeaders: ["x-claude-code-session-id"], + turnHeaders: [], + msgidHeaders: [], + piHeaders: [], + fingerprintHeaders: [], + countPatterns: [{ pattern: "toolu_", label: "toolu" }], + compactionPrefixes: ["This session is being continued from a previous conversation"], + dualSessionBodyAnchor: { anchor: "\"user_id\"", field: "\"session_id\"", window: 512 }, + }, + { + name: "opencode", + matchHeaders: ["x-session-affinity", "x-opencode-session-id"], + sessionHeaders: ["x-session-affinity", "x-opencode-session-id"], + sessionValuePrefix: "ses_", + turnHeaders: ["x-parent-session-id"], + msgidHeaders: [], + piHeaders: [], + fingerprintHeaders: [], + // Assertive counts: these shapes are NOT expected on opencode's outbound + // model wire; a non-zero count means upstream protocol drift (event). + countPatterns: [ + { pattern: "ses_", label: "ses_", assertZero: true }, + { pattern: "msg_", label: "msg_", assertZero: true }, + { pattern: "prt_", label: "prt_", assertZero: true }, + ], + compactionPrefixes: ["What did we do so far?", ""], + }, + { + name: "gemini", + matchHeaders: [], + matchProtocol: "google", + sessionHeaders: [], + turnHeaders: [], + msgidHeaders: [], + piHeaders: [], + fingerprintHeaders: [], + countPatterns: [], + compactionPrefixes: [""], + }, +]; + +/** Resolve the active profile table: built-ins are the floor; the env JSON may + * add/replace rows by name, never remove built-ins. Malformed env JSON is + * ignored (fail-open). */ +function nativeIdProfiles(): readonly NativeIdProfileDef[] { + const raw = process.env["BILI_NATIVE_IDENTITY_PROFILES_JSON"]; + if (raw === undefined || raw.trim().length === 0) return BUILTIN_PROFILES; + try { + const parsed: unknown = JSON.parse(raw); + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) throw new Error("not an object"); + const overrides = parsed as Record>; + const merged: NativeIdProfileDef[] = BUILTIN_PROFILES.map((p) => ({ ...p })); + const byName = new Map(merged.map((p) => [p.name, p])); + for (const [name, patch] of Object.entries(overrides)) { + if (patch === null || typeof patch !== "object") continue; + const existing = byName.get(name); + if (existing !== undefined) { + Object.assign(existing, patch, { name }); + } else { + merged.push({ + name, + matchHeaders: [], + sessionHeaders: [], + turnHeaders: [], + msgidHeaders: [], + piHeaders: [], + fingerprintHeaders: [], + countPatterns: [], + compactionPrefixes: [], + ...patch, + } as NativeIdProfileDef); + } + } + return merged; + } catch { + // Malformed override: silently keep the built-ins (fail-open, + // observation layer only). No logger on purpose — this module must + // stay import-light. + return BUILTIN_PROFILES; + } +} + +/** First matching profile wins: built-in rows in table order (wb → codex → cc + * → opencode → gemini), then env-added rows, then unknown. Plugin-lane + * traffic (agent-stamped) resolves to the AGENT name upstream of this table — + * those identities are bili-trusted, not forensics. */ +export function resolveNativeIdProfile( + headers: Record, + protocol: string | null | undefined, +): NativeIdProfileDef & { name: string } { + const present = (name: string): boolean => { + const v = headers[name]; + return typeof v === "string" && v.trim().length > 0; + }; + for (const p of nativeIdProfiles()) { + if (p.matchHeaders.some(present)) return p; + if (p.secondaryMatch !== undefined && present(p.secondaryMatch.header) && protocol === p.secondaryMatch.protocol) return p; + if (p.matchProtocol !== undefined && protocol === p.matchProtocol) return p; + } + return UNKNOWN_PROFILE; +} + +const UNKNOWN_PROFILE: NativeIdProfileDef & { name: string } = { + name: "unknown", + matchHeaders: [], + sessionHeaders: [], + turnHeaders: [], + msgidHeaders: [], + piHeaders: [], + fingerprintHeaders: [], + countPatterns: [], + compactionPrefixes: [], +}; diff --git a/src/server/native-identity.ts b/src/server/native-identity.ts new file mode 100644 index 0000000000..caf7955588 --- /dev/null +++ b/src/server/native-identity.ts @@ -0,0 +1,425 @@ +/** + * #5573 M1 (L1) — native-identity observation core. Pure OBSERVATION: + * records what identity evidence the host actually sent (headers, bounded + * body signals, compaction prefaces) into the session ext store plus a log + * line. It feeds NO decision — request handling, folding, parsing and + * outbound bytes are untouched by construction (red lines: zero upstream + * mutation, zero decisions, zero agent-side changes; off-or-unknown traffic + * produces zero diff against the existing golden suites). + * + * Privacy red line (#5573 M1 design): id VALUES never persist or log in + * plaintext — ext stores sha1-8 digests only, log lines print the 8-char + * digest + ellipsis. Full values live in memory for the duration of the + * request lifecycle at most (used for the dual-channel cross-check). + * + * Cost discipline: header extraction is constant-time; body signals use + * bounded indexOf walks over the ALREADY-decoded bytes (no JSON.parse — the + * chokepoint holds bytes, and parsing is exactly the work #5403 removed). + * Profiles without body signals never touch the body at all. The whole call + * is fail-open: any internal error drops the observation, never the request. + */ + +import { createHash } from "node:crypto"; +import type { SessionStore } from "../persist.js"; +import { resolveNativeIdProfile } from "./native-identity-profiles.js"; +import type { NativeIdProfileDef } from "./native-identity-profiles.js"; + +export const NIDV_EXT_KEY = "nidv:obs"; + +const EVENT_CAP = 32; +const RING_CAP = 64; +const EXT_BUDGET = 4096; +/** Bounded backward distance for enclosing-role attribution of a prefix hit. */ +const ROLE_WINDOW = 262_144; +/** Top-level body fields live near the head; bounded walk window. */ +const TOP_FIELD_WINDOW = 8192; +/** Incident-class events escalate the line to WARN (owner spec: compaction + * hits are the headline signal M2/L2a consumes). */ +const WARN_EVENTS = new Set(["compaction", "dual_mismatch", "shape_assert_fail"]); + +interface NidvEvent { + t: number; + k: string; + e?: string; +} + +interface NidvMsgStats { + n: number; + /** Lower bound: unique digests inside the bounded recent ring. */ + distinct: number; + /** yes = a digest repeated across turns (host reuses ids across retries); + * no = ≥8 turns observed, all distinct; unknown = not yet decidable. */ + stable: "unknown" | "yes" | "no"; + ring: string[]; +} + +interface NidvObs { + v: 1; + client: string; + firstSeen: number; + lastSeen: number; + reqs: number; + ids: { + sess?: { h: string; stable: "unknown" | "yes" | "no" }; + msg?: NidvMsgStats; + prevRespH?: string; + }; + events: NidvEvent[]; +} + +export interface NativeIdInput { + /** Explicit record BUCKET (the plugin-lane site passes its conversationId — + * the host-native conversation id). Stored DIGESTED as `n:` — + * the module redline is that id VALUES never persist or log in plaintext + * (#5576 review M1: the raw value must not reach the ext store). When + * omitted (relay site) the key is DERIVED FROM THE PROFILE'S OWN SESSION + * EVIDENCE (`n:`), not bili's identity chain: pre-#5526 the chain + * would key WorkBuddy traffic by the account-level x-session-id, merging + * every conversation of one account into one record and corrupting the + * stability stats. No session evidence ⇒ the shared "anon" bucket (coarse + * by design, documented). Deliberately NOT bili's internal session id + * either way — forks share the parent's client-visible key, which is what + * we are observing. */ + key?: string; + headers: Record; + /** Decoded body bytes/text. May be omitted (then body signals are skipped). */ + body?: Buffer | string; + protocol?: string | null; + path?: string; + /** Plugin-lane agent stamp (x-bili-plugin) — trusted identity, wins labeling. */ + agent?: string; + /** Host msgid envelope parallel to the lane delta (#4491). */ + msgIds?: string[]; +} + +interface NativeIdCount { + label: string; + n: number; + assertZero?: boolean; +} + +interface NativeIdExtraction { + client: string; + profileName: string; + sessH?: string; + msgidH?: string; + turnH?: string; + edge?: string; + level?: boolean; + fork?: boolean; + counts: NativeIdCount[]; + /** First compaction-preface hit with its enclosing-element attribution. */ + prefix?: { elemIdx: number; role: string; offset: number }; + dual?: "ok" | "mismatch" | "absent"; + prevRespH?: string; + /** Message elements observed this turn (envelope length, else role-marker count). */ + delta: number; + noteParts: string[]; +} + +export function digest8(value: string): string { + return createHash("sha1").update(value, "utf8").digest("hex").slice(0, 8); +} + +function hdrValue(headers: Record, name: string): string | undefined { + const v = headers[name]; + if (typeof v !== "string") return undefined; + const t = v.trim(); + return t.length > 0 ? t : undefined; +} + +function countSub(s: string, sub: string): number { + let n = 0; + let i = s.indexOf(sub); + while (i !== -1) { + n += 1; + i = s.indexOf(sub, i + sub.length); + } + return n; +} + +/** Nearest preceding `"role":""` marker before offset o, plus the element + * index (count of role markers seen). Approximate by design: wire elements + * carry role first, nested role keys are not expected on model wires. A hit + * farther than ROLE_WINDOW behind o degrades to role "unknown" (honest). */ +function roleAtOffset(s: string, o: number): { role: string; elemIdx: number } { + const mark = '"role":'; + let idx = -1; + let elemIdx = 0; + let lastRole = "unknown"; + let lastAt = -1; + while ((idx = s.indexOf(mark, idx + 1)) !== -1 && idx < o) { + elemIdx += 1; + let p = idx + mark.length; + while (p < s.length && s[p] === " ") p += 1; + if (s[p] === '"') { + const e = s.indexOf('"', p + 1); + if (e !== -1) lastRole = s.slice(p + 1, e); + } + lastAt = idx; + } + if (lastAt === -1 || o - lastAt > ROLE_WINDOW) return { role: "unknown", elemIdx }; + return { role: lastRole, elemIdx }; +} + +/** Quoted value following `key` (including its quotes), bounded to `window` + * chars past the key. Malformed shapes degrade to undefined (swallowed). */ +function quotedValueAfter(s: string, key: string, window: number): string | undefined { + const a = s.indexOf(key); + if (a === -1) return undefined; + const end = Math.min(s.length, a + window); + let p = a + key.length; + while (p < end && (s[p] === " " || s[p] === ":")) p += 1; + if (p >= end || s[p] !== '"') return undefined; + const e = s.indexOf('"', p + 1); + if (e === -1 || e > end) return undefined; + return s.slice(p + 1, e); +} + +export function extractNativeIdentity(input: NativeIdInput): NativeIdExtraction | null { + const profile: NativeIdProfileDef = resolveNativeIdProfile(input.headers, input.protocol); + const client = input.agent ?? profile.name; + if (client === "unknown") return null; + + const out: NativeIdExtraction = { + client, + profileName: profile.name, + counts: [], + delta: 0, + noteParts: [], + }; + + const sessRaw = pickFirst(profile.sessionHeaders, input.headers, profile.sessionValuePrefix); + if (sessRaw !== undefined) { + out.sessH = digest8(sessRaw); + out.dual = "absent"; + } + const msgidRaw = pickFirst(profile.msgidHeaders, input.headers); + if (msgidRaw !== undefined) out.msgidH = digest8(msgidRaw); + const turnRaw = pickFirst(profile.turnHeaders, input.headers); + if (turnRaw !== undefined) out.turnH = digest8(turnRaw); + + if (profile.compactionEdgeHeader !== undefined) { + const v = hdrValue(input.headers, profile.compactionEdgeHeader); + if (v !== undefined) out.edge = v.slice(0, 32); + } + if (profile.compactionLevelHeader !== undefined) { + out.level = hdrValue(input.headers, profile.compactionLevelHeader) !== undefined; + } + if (input.agent === undefined && profile.turnHeaders.includes("x-parent-session-id")) { + out.fork = hdrValue(input.headers, "x-parent-session-id") !== undefined; + } + for (const name of profile.fingerprintHeaders) { + const v = hdrValue(input.headers, name); + if (v !== undefined) out.noteParts.push(`fp:${name}=${v.slice(0, 24)}`); + } + + if (input.msgIds !== undefined) { + out.delta = input.msgIds.length; + } + + const needsBody = + profile.countPatterns.length > 0 || + profile.compactionPrefixes.length > 0 || + profile.dualSessionBodyAnchor !== undefined || + profile.rotationBodyField !== undefined; + if (needsBody && input.body !== undefined) { + const s = typeof input.body === "string" ? input.body : input.body.toString("utf8"); + if (out.delta === 0) out.delta = countSub(s, '"role":'); + + for (const pat of profile.countPatterns) { + const n = countSub(s, pat.pattern); + out.counts.push({ label: pat.label, n, assertZero: pat.assertZero }); + if (pat.assertZero === true || n > 0) out.noteParts.push(`${pat.label}=${String(n)}`); + if (pat.assertZero === true && n > 0) out.noteParts.push("assert-fail"); + } + + let best: { offset: number } | undefined; + for (const pre of profile.compactionPrefixes) { + const o = s.indexOf(pre); + if (o !== -1 && (best === undefined || o < best.offset)) best = { offset: o }; + } + if (best !== undefined) { + const attr = roleAtOffset(s, best.offset); + out.prefix = { elemIdx: attr.elemIdx, role: attr.role, offset: best.offset }; + if (attr.role !== "user") out.noteParts.push(`prefix@elem${String(attr.elemIdx)}/${attr.role}`); + } + + if (profile.dualSessionBodyAnchor !== undefined && sessRaw !== undefined) { + const a = profile.dualSessionBodyAnchor; + const anchorPos = s.indexOf(a.anchor); + let bodyVal: string | undefined; + if (anchorPos !== -1) { + const end = Math.min(s.length, anchorPos + a.window); + const f = s.indexOf(a.field, anchorPos + a.anchor.length); + if (f !== -1 && f < end) bodyVal = quotedValueAfter(s, a.field, end - f); + } + if (bodyVal === undefined) out.dual = "absent"; + else if (bodyVal === sessRaw) out.dual = "ok"; + else out.dual = "mismatch"; + out.noteParts.push(`dual=${out.dual}`); + } + + if (profile.rotationBodyField !== undefined) { + const v = quotedValueAfter(s, `"${profile.rotationBodyField}"`, TOP_FIELD_WINDOW); + if (v !== undefined) out.prevRespH = digest8(v); + } + } + + if (out.edge !== undefined) out.noteParts.unshift(`compact=${out.edge}`); + if (out.level === true) out.noteParts.push("ctx=yes"); + if (out.fork === true) out.noteParts.push("fork=yes"); + return out; +} + +function pickFirst( + names: readonly string[], + headers: Record, + valuePrefix?: string, +): string | undefined { + for (const name of names) { + const v = hdrValue(headers, name); + if (v === undefined) continue; + if (valuePrefix !== undefined && !v.startsWith(valuePrefix)) continue; + return v; + } + return undefined; +} + +export function observeNativeIdentity( + input: NativeIdInput, + store: SessionStore | null, + log: (level: string, msg: string) => void, +): void { + try { + const ex = extractNativeIdentity(input); + if (ex === null) return; + // #5576 review M1: explicit buckets are stored DIGESTED like every + // other key in this module — the raw host-native id value never + // reaches the ext store (privacy redline, module header). + const recordKey = input.key !== undefined ? `n:${digest8(input.key)}` : (ex.sessH !== undefined ? `n:${ex.sessH}` : "anon"); + if (recordKey.length === 0) return; + + const now = Date.now(); + let prev: NidvObs | null = null; + // Same house pattern as fastpath's pf:state (#5403): ext namespaces only + // exist on unified storage (shadow|full); legacy-only instances degrade + // to log-line-only observation instead of throwing into the pipeline. + const extUsable = store !== null && store.replicasAvailable(); + if (extUsable) { + let raw: unknown; + try { + raw = store.getSessionExt(recordKey, NIDV_EXT_KEY); + } catch { + raw = undefined; + } + if (typeof raw === "string" && raw.length > 0) { + try { + const p = JSON.parse(raw) as NidvObs; + if (p !== null && typeof p === "object" && p.v === 1) prev = p; + } catch { + prev = null; + } + } + } + + const events: NidvEvent[] = []; + const next: NidvObs = { + v: 1, + client: ex.client, + firstSeen: prev?.firstSeen ?? now, + lastSeen: now, + reqs: (prev?.reqs ?? 0) + 1, + ids: {}, + events: [...(prev?.events ?? [])], + }; + if (prev !== null && prev.client !== ex.client) { + events.push({ t: now, k: "profile_flip", e: `${prev.client}->${ex.client}` }); + } + + if (ex.sessH !== undefined) { + const ps = prev?.ids.sess; + if (ps === undefined) next.ids.sess = { h: ex.sessH, stable: "unknown" }; + else if (ps.h === ex.sessH) next.ids.sess = { h: ps.h, stable: ps.stable === "no" ? "no" : "yes" }; + else { + next.ids.sess = { h: ex.sessH, stable: "no" }; + events.push({ t: now, k: "id_change", e: "sess" }); + } + } else { + next.ids.sess = prev?.ids.sess; + } + + const curDigests = (input.msgIds ?? []).map((m) => digest8(m)); + if (curDigests.length > 0) { + const ringPrev = prev?.ids.msg?.ring ?? []; + const repeat = curDigests.some((h) => ringPrev.includes(h)); + const merged = [...curDigests, ...ringPrev.filter((h) => !curDigests.includes(h))].slice(0, RING_CAP); + const n = (prev?.ids.msg?.n ?? 0) + curDigests.length; + // Per-turn verdict, deliberately not sticky: in the incremental lane consecutive + // deltas do NOT overlap by design, so a resend is the only "yes" evidence. + const stable: NidvMsgStats["stable"] = repeat ? "yes" : n >= 8 ? "no" : "unknown"; + next.ids.msg = { n, distinct: new Set(merged).size, stable, ring: merged }; + } else { + next.ids.msg = prev?.ids.msg; + } + + if (ex.prevRespH !== undefined) { + if (prev?.ids.prevRespH !== undefined && prev.ids.prevRespH !== ex.prevRespH) { + events.push({ t: now, k: "id_change", e: "prevresp" }); + } + next.ids.prevRespH = ex.prevRespH; + } else { + next.ids.prevRespH = prev?.ids.prevRespH; + } + + if (ex.prefix !== undefined && ex.prefix.role === "user") { + events.push({ t: now, k: "compaction", e: `preface@elem${String(ex.prefix.elemIdx)}` }); + } + if (ex.edge !== undefined) events.push({ t: now, k: "compaction", e: `hdr=${ex.edge}` }); + if (ex.dual === "mismatch") events.push({ t: now, k: "dual_mismatch", e: `${next.ids.sess?.h ?? "?"}` }); + for (const c of ex.counts) { + if (c.assertZero === true && c.n > 0) events.push({ t: now, k: "shape_assert_fail", e: `${c.label}=${String(c.n)}` }); + } + if (ex.fork === true) events.push({ t: now, k: "fork", e: "parent-sess" }); + + for (const ev of events) { + const last = next.events[next.events.length - 1]; + if (last !== undefined && last.k === ev.k && last.e === ev.e) { + last.t = ev.t; + } else { + next.events.push(ev); + } + } + if (next.events.length > EVENT_CAP) next.events.splice(0, next.events.length - EVENT_CAP); + + const line = renderLine(ex, events); + log(events.some((e) => WARN_EVENTS.has(e.k)) ? "warn" : "debug", line); + if (extUsable) { + try { + store.setSessionExt(recordKey, NIDV_EXT_KEY, fitBudget(next)); + } catch { /* observation is advisory — losing it degrades to log-line-only */ } + } + } catch { + // fail-open: an observation defect must never disturb the request path. + } +} + +function renderLine(ex: NativeIdExtraction, fresh: NidvEvent[]): string { + const evt = fresh.length > 0 ? fresh.map((e) => (e.e !== undefined ? `${e.k}(${e.e})` : e.k)).join(",") : "-"; + const note = ex.noteParts.length > 0 ? ` ${ex.noteParts.join(" ").slice(0, 160)}` : ""; + const sess = ex.sessH !== undefined ? `${ex.sessH}\u2026` : "-"; + return `nidv: client=${ex.client} sess=${sess} msgΔ=+${String(ex.delta)} evt=${evt}${note}`; +} + +function fitBudget(obs: NidvObs): string { + let json = JSON.stringify(obs); + if (json.length <= EXT_BUDGET) return json; + obs.events = obs.events.slice(-8); + json = JSON.stringify(obs); + if (json.length <= EXT_BUDGET) return json; + if (obs.ids.msg !== undefined) obs.ids.msg.ring = obs.ids.msg.ring.slice(0, 16); + json = JSON.stringify(obs); + if (json.length <= EXT_BUDGET) return json; + obs.events = obs.events.slice(-4); + return JSON.stringify(obs); +} diff --git a/src/server/snapshot-render.ts b/src/server/snapshot-render.ts index 6cedbcf677..401f50d26f 100644 --- a/src/server/snapshot-render.ts +++ b/src/server/snapshot-render.ts @@ -50,7 +50,7 @@ import { createHash } from "node:crypto"; /** Client-tail walk budget. Full-resend histories are bounded by the wire; * beyond this the lane degrades rather than scanning unbounded tails. */ -const MAX_CHAIN_WALK = 8192; +export const MAX_CHAIN_WALK = 8192; /** #3812: a seam is only accepted after at least this many CONSECUTIVE chain * matches. The identity signature has no id component, so a single sig hit @@ -153,7 +153,10 @@ function parseRow(content: string, seq: number, ts?: number | null): BiliMessage // everything in clientMsgs at indices >= tailStart rides the outbound verbatim, // everything below is server-rendered territory. Exposed so the egress judge can // exempt the rendered prefix and validate only the tail it promises byte-intact. -type ViewResult = { ok: true; view: BiliMessage[]; tailStart: number } | { ok: false; reason: string }; +type ViewResult = { ok: true; view: BiliMessage[]; tailStart: number; /** #5576 v6: first CLIENT index of the tail (core space) — + * consumed by the canonical watermark lane to recover the element-space + * window origin via wireIndex. Absent from the WS-lane result shape is NOT + * a concern: every ok-return carries it, consumers that predate it ignore it. */ clientJoin?: number } | { ok: false; reason: string }; interface TranscriptBase { rows: ViewRow[]; @@ -403,6 +406,12 @@ function buildView(session: Session, clientMsgs: BiliMessage[]): ViewResult { // accepting it renders only the ancient prefix + the one tail message // and silently drops everything between. Degrade instead: never guess. if (run < MIN_SEAM_RUN) return { ok: false, reason: "seam-too-short" }; + // KNOWN RESIDUAL (#5632, found via #5576 review): run counts matches + // of EITHER class, so two consecutive SIGNATURE-ONLY hits on adjacent + // uncovered deep rows still pass this gate and truncate everything + // newer than the join (silent history loss). Narrow path — needs a + // byte-identical duplicate pair with fresh ids at the tail boundary. + // Direction: require byId corroboration for non-local sig-only runs. } const d = joinRow + 1; // prefix = rows[0..d) — everything up to the join; rows beyond (rewound-away tails) never render const merged = assembleView(rows, blockSeeds, coveredSeqUnion, d, clientMsgs.slice(clientJoin)); @@ -416,9 +425,17 @@ function buildView(session: Session, clientMsgs: BiliMessage[]): ViewResult { // the judge would validate nothing (vacuous clean — the regression that // blinded the #4241 teeth on the T1-consolidation train). const tailCount = clientMsgs.length - clientJoin; - return { ok: true, view: merged, tailStart: merged.length - tailCount }; + // #5576 v6: surface the client-domain seam (element-recovery input for + // the canonical watermark lane); tailStart above stays VIEW-domain (#4241). + return { ok: true, view: merged, tailStart: merged.length - tailCount, clientJoin }; } +/** #5576 v6 canonical watermark lane: same probe under a purpose-named + * export so the fastpath module never reaches into snapshot-render's + * private surface. Zero behavior change for the snapshotRender feature + * itself — maybeSnapshotView consumes .view/.tailStart exactly as before. */ +export const alignClientAgainstTranscript = buildView; + // --------------------------------------------------------------------------- // #4804 anchored render — WS delta continuation lane (codex/opencode clients // that keep a previous_response_id checkpoint and send small input tails) diff --git a/src/storage/mysql.ts b/src/storage/mysql.ts index 91e0928476..a54659062c 100644 --- a/src/storage/mysql.ts +++ b/src/storage/mysql.ts @@ -65,10 +65,10 @@ import type { * Same discipline as the jimp lazy load (src/image-rescale.ts, #3516). */ export type Mysql2Module = typeof import("mysql2/promise"); -/** v10 (#4313): immutable per-message wall-clock stamp. Existing stamped-v9 - * libraries re-enter the probe-first DDL block below exactly once on first - * open; pre-feature rows read NULL (no backfill). */ -const MYSQL_SCHEMA_VERSION = 10; +/** v11 (#5600): fastpath snapshot store (pf_rows/pf_log/pf_ckpt) rides the + * always-run fresh-install DDL below — no data step; the stamp keeps the two + * backends' shape bookkeeping in lockstep with STORAGE_SCHEMA_VERSION. */ +const MYSQL_SCHEMA_VERSION = 11; /** Parity constants with the sqlite v8 streaming backfill * (src/storage/schema.ts, #4677): same page size, same printable absent @@ -257,6 +257,41 @@ CREATE TABLE IF NOT EXISTS bili_meta ( meta_key VARCHAR(64) NOT NULL PRIMARY KEY, meta_value VARCHAR(256) NOT NULL ) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +-- #5600 fastpath snapshot store (sqlite twin: schema.ts v11). Sid = server- +-- minted pf_sid; ledger_id holds host msgids (<=512ch by the parse cap), so +-- its single-column utf8mb4 index stays under InnoDB's 3072B limit. +CREATE TABLE IF NOT EXISTS pf_rows ( + sid VARCHAR(255) NOT NULL, + pos BIGINT NOT NULL, + elem BIGINT NOT NULL, + ledger_id VARCHAR(512), + core_id VARCHAR(255) NOT NULL, + core_json LONGTEXT NOT NULL, + ts BIGINT NOT NULL, + PRIMARY KEY (sid, pos), + KEY idx_pf_rows_ledger (ledger_id), + KEY idx_pf_rows_elem (sid, elem), + KEY idx_pf_rows_core (sid, core_id) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS pf_log ( + sid VARCHAR(255) NOT NULL, + ls_seq BIGINT NOT NULL, + kind VARCHAR(64) NOT NULL, + payload_json LONGTEXT NOT NULL, + born_pos_hi BIGINT, + ts BIGINT NOT NULL, + PRIMARY KEY (sid, ls_seq) +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS pf_ckpt ( + sid VARCHAR(255) NOT NULL PRIMARY KEY, + ckpt_ls_seq BIGINT NOT NULL, + meta_json LONGTEXT NOT NULL, + checksum VARCHAR(64) NOT NULL, + updated_at BIGINT NOT NULL +) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4; `; type Row = Record; @@ -290,7 +325,13 @@ export class MysqlBackend implements StorageBackend { async init(): Promise { const conn = await this.pool.getConnection(); try { - const multi = MYSQL_DDL.trim(); + // #5578: strip `--` line comments BEFORE splitting — a comment may + // contain `;`, and its orphaned tail would execute as SQL + // (ER_PARSE_ERROR on every MySQL boot). The blob DOES carry + // literals ('{}', 'sha256', 'active'); splitting on ';' stays + // safe ONLY because none of them contains ';' or '--' — keep + // new defaults literal-free or move the DDL out of the blob. + const multi = MYSQL_DDL.replace(/--[^\n]*/g, "").trim(); for (const stmt of multi.split(";")) { const s = stmt.trim(); if (s.length === 0) continue; diff --git a/src/storage/schema.ts b/src/storage/schema.ts index 7bdd0b747a..1cf7b82393 100644 --- a/src/storage/schema.ts +++ b/src/storage/schema.ts @@ -68,7 +68,14 @@ // never updated). No backfill: pre-feature rows stay NULL and keep // rendering without the tag's time attribute. Same both-directions // compatibility as v5 (older build ignores, newer build ALTERs). -export const STORAGE_SCHEMA_VERSION = 10; +/** v11 (#5600 §4.1/§3.1): fastpath snapshot store — pf_sid-partitioned + * append-only data plane (pf_rows) + LSM metadata plane (pf_log/pf_ckpt). + * Fresh databases create the shape directly; older installs upgrade with + * three idempotent CREATE TABLE IF NOT EXISTS statements + two indexes + * (upgrade cost: empty tables, byte-zero impact on existing data). + * Tables are ZERO-STATE (no status/frontier columns); cleanup rides + * deleteSession only (owner ruling: no independent retention period). */ +export const STORAGE_SCHEMA_VERSION = 11; /** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and * auto checkpoints move frames into the main db and reset the write offset @@ -398,6 +405,21 @@ export function migrateSchema(db: { db.exec("ALTER TABLE messages ADD COLUMN ts INTEGER"); } } + if (from < 11) { + // v11 (#5600): fastpath snapshot store. Sid = server-minted + // pf_sid (§3.1) — client session ids NEVER appear in these + // tables. Idempotent CREATEs (dev-window discipline). + db.exec("CREATE TABLE IF NOT EXISTS pf_rows (\n sid TEXT NOT NULL,\n pos INTEGER NOT NULL,\n elem INTEGER NOT NULL,\n ledger_id TEXT,\n core_id TEXT NOT NULL,\n core_json TEXT NOT NULL,\n ts INTEGER NOT NULL,\n PRIMARY KEY (sid, pos)\n ) WITHOUT ROWID"); + // Global probe index — msgid observed globally unique so far; + // backs the §3.1 rule-② identity probe (any sid resolves). + db.exec("CREATE INDEX IF NOT EXISTS idx_pf_rows_ledger ON pf_rows (ledger_id)") + // Origin-pin verification (elem-scoped id EXISTS check). + db.exec("CREATE INDEX IF NOT EXISTS idx_pf_rows_elem ON pf_rows (sid, elem)"); + // block↔pos binding lookups. + db.exec("CREATE INDEX IF NOT EXISTS idx_pf_rows_core ON pf_rows (sid, core_id)"); + db.exec("CREATE TABLE IF NOT EXISTS pf_log (\n sid TEXT NOT NULL,\n ls_seq INTEGER NOT NULL,\n kind TEXT NOT NULL,\n payload_json TEXT NOT NULL,\n born_pos_hi INTEGER,\n ts INTEGER NOT NULL,\n PRIMARY KEY (sid, ls_seq)\n ) WITHOUT ROWID"); + db.exec("CREATE TABLE IF NOT EXISTS pf_ckpt (\n sid TEXT PRIMARY KEY,\n ckpt_ls_seq INTEGER NOT NULL,\n meta_json TEXT NOT NULL,\n checksum TEXT NOT NULL,\n updated_at INTEGER NOT NULL\n ) WITHOUT ROWID"); + } }); if (needV8Backfill) migrateV8Identities(db); db.exec(`PRAGMA user_version = ${STORAGE_SCHEMA_VERSION}`); diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index 6b77bf7ffc..f913188ee6 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -1089,6 +1089,63 @@ export class UnifiedShadow { return this.sync.deleteSessionExt(sessionId, key); } + // ------------------------------------------------------------------ + // #5600 fastpath snapshot store (pfsid plane) passthroughs — side-table + // substrate like checkpoints/ext. Availability gating happens at the + // persist level (pfStore() soft accessor); null-sync degrades to empty. + // ------------------------------------------------------------------ + + pfAppendTurn( + sid: string, + rows: { ledgerId: string | null; elem: number; coreId: string; coreJson: string }[], + entries: { kind: string; payload: unknown; bornPosHi?: number }[], + ): { maxPos: number; lsSeq: number } { + if (this.sync === null) return { maxPos: -1, lsSeq: -1 }; + return this.sync.pfAppendTurn(sid, rows, entries); + } + + /** #5576 review S5: true only when a local sync plane actually persists + * pf partitions. Replica/MySQL views run with sync===null and would + * otherwise expose no-op closures to the lane (per-turn fabrication + * warn + unbounded mint/registerPfSid growth). */ + pfPlaneUsable(): boolean { + return this.sync !== null; + } + + pfLookupLedger(ledgerId: string): { sid: string; pos: number } | null { + if (this.sync === null) return null; + return this.sync.pfLookupLedger(ledgerId); + } + + /** #5600 origin-pin check (see Store.pfVerifyOriginPin). */ + pfVerifyOriginPin(sid: string, elem: number, ledgerId: string): boolean { + if (this.sync === null) return false; + return this.sync.pfVerifyOriginPin(sid, elem, ledgerId); + } + + pfReadMeta(sid: string): { + ckpt: { ckptLsSeq: number; metaJson: string; checksum: string } | null; + tailLog: { lsSeq: number; kind: string; payloadJson: string; bornPosHi: number | null }[]; + } { + if (this.sync === null) return { ckpt: null, tailLog: [] }; + return this.sync.pfReadMeta(sid); + } + + pfWriteCkpt(sid: string, ckptLsSeq: number, metaJson: string, checksum: string, truncateBeforeInclusive = true): void { + if (this.sync === null) return; + this.sync.pfWriteCkpt(sid, ckptLsSeq, metaJson, checksum, truncateBeforeInclusive); + } + + pfReadRows(sid: string, intervals: ReadonlyArray): { pos: number; elem: number; ledgerId: string | null; coreId: string; coreJson: string; ts: number }[] { + if (this.sync === null) return []; + return this.sync.pfReadRows(sid, intervals); + } + + pfMaxPos(sid: string): number { + if (this.sync === null) return -1; + return this.sync.pfMaxPos(sid); + } + setKv(namespace: string, key: string, value: unknown): void { if (this.sync === null) return; this.sync.setKv(namespace, key, value); diff --git a/src/storage/store.ts b/src/storage/store.ts index ba6aed922f..98b2b2b1fb 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -1665,6 +1665,20 @@ export class UnifiedStore { deleteSession(sessionId: string): void { this.db.transaction(() => { + // #5600 §4.1.1: pfsid partitions ride session deletion (owner + // ruling: no independent retention period) — the session's + // ext registry (`pf:sids`) names every snapshot identity minted + // under it; all three pf_* tables sweep in THIS transaction. + try { + const sidsRow = this.db.prepare("SELECT value_json FROM session_ext WHERE session_id = ? AND key = 'pf:sids'").get(sessionId) as { value_json: string } | undefined; + if (sidsRow) { + const parsed: unknown = JSON.parse(sidsRow.value_json); + if (Array.isArray(parsed)) { + const sids = parsed.filter((s): s is string => typeof s === "string" && s.length > 0); + if (sids.length > 0) this.pfDeleteSids(sids, true); + } + } + } catch { /* malformed registry — nothing to sweep */ } const refs: string[] = []; for (const m of this.db .prepare("SELECT content_hash FROM messages WHERE session_id = ?") @@ -1903,6 +1917,20 @@ export class UnifiedStore { // --- sweep the row bodies (refcount discipline; pinned checkpoints survive) --- let contentRowsDeleted = 0; this.db.transaction(() => { + // #5576 review S2: the pf:sids registry dies with session_ext + // further down — sweep the pfsid partitions IN THIS transaction + // (deleteSession's pattern) so archiving without restore cannot + // leave a permanent full-transcript copy in the live DB. + try { + const sidsRow = this.db.prepare("SELECT value_json FROM session_ext WHERE session_id = ? AND key = 'pf:sids'").get(sessionId) as { value_json: string } | undefined; + if (sidsRow) { + const parsed: unknown = JSON.parse(sidsRow.value_json); + if (Array.isArray(parsed)) { + const sids = parsed.filter((s): s is string => typeof s === "string" && s.length > 0); + if (sids.length > 0) this.pfDeleteSids(sids, true); + } + } + } catch { /* malformed registry — nothing to sweep */ } const rel: string[] = []; for (const m of messages) rel.push(m.content_hash as string); for (const mm of mediaMaps) rel.push(mm.media_hash as string); @@ -2297,6 +2325,140 @@ export class UnifiedStore { : (this.db.prepare(`SELECT ${cols} FROM checkpoints ORDER BY created_at, cp_id`).all() as CheckpointRow[]); } + // ------------------------------------------------------------------ + // Fastpath snapshot store (#5600 §4.1/§3.1). Sid = server-minted pf_sid; + // append-only data plane + LSM metadata plane; ZERO-STATE tables (no + // status/frontier columns — current state derives from log folds). Every + // op is one transaction. Cleanup rides deleteSession only (no + // independent retention period — owner ruling). + // ------------------------------------------------------------------ + + /** Append one turn's data rows + log entries atomically (single tx). + * Returns the new high-water marks after this turn. */ + pfAppendTurn( + sid: string, + rows: { ledgerId: string | null; elem: number; coreId: string; coreJson: string }[], + entries: { kind: string; payload: unknown; bornPosHi?: number }[], + ): { maxPos: number; lsSeq: number } { + return this.db.transaction(() => { + const now = Date.now(); + const r = this.db.prepare("SELECT COALESCE(MAX(pos), -1) AS m FROM pf_rows WHERE sid = ?").get(sid) as { m: number }; + let pos = r.m; + if (rows.length > 0) { + const ins = this.db.prepare("INSERT INTO pf_rows (sid, pos, elem, ledger_id, core_id, core_json, ts) VALUES (?, ?, ?, ?, ?, ?, ?)"); + for (const row of rows) { + pos += 1; + ins.run(sid, pos, row.elem, row.ledgerId, row.coreId, row.coreJson, now); + } + } + const l = this.db.prepare("SELECT COALESCE(MAX(ls_seq), -1) AS m FROM pf_log WHERE sid = ?").get(sid) as { m: number }; + let seq = l.m; + if (entries.length > 0) { + const ins = this.db.prepare("INSERT INTO pf_log (sid, ls_seq, kind, payload_json, born_pos_hi, ts) VALUES (?, ?, ?, ?, ?, ?)"); + for (const e of entries) { + seq += 1; + ins.run(sid, seq, e.kind, JSON.stringify(e.payload), e.bornPosHi ?? null, now); + } + } + return { maxPos: pos, lsSeq: seq }; + }); + } + + /** §3.1 origin-pin verification: does element `elem` of THIS partition + * carry `ledgerId` on ANY of its rows (a multimodal element splits into + * several rows sharing one id)? Exactly the check the msgid lane needs + * instead of first-hit-by-pos equality. */ + pfVerifyOriginPin(sid: string, elem: number, ledgerId: string): boolean { + const r = this.db + .prepare("SELECT 1 AS x FROM pf_rows WHERE sid = ? AND elem = ? AND ledger_id = ? LIMIT 1") + .get(sid, elem, ledgerId); + return r !== undefined; + } + + /** §3.1 rule-② identity probe: does any partition carry this msgid? + * Msgid is observed globally unique so far, so one hit resolves the + * owner sid unambiguously; ties break to the lowest pos (diagnostic + * noise would show up there first). */ + pfLookupLedger(ledgerId: string): { sid: string; pos: number } | null { + const row = this.db + .prepare("SELECT sid, pos FROM pf_rows WHERE ledger_id = ? ORDER BY pos, sid LIMIT 1") + .get(ledgerId) as { sid: string; pos: number } | undefined; + return row ?? null; + } + + /** L1 checkpoint + bounded tail of the L0 hot log since it. + * Caller replays tailLog in order over the ckpt meta (§4.1.4 hydration). */ + pfReadMeta(sid: string): { + ckpt: { ckptLsSeq: number; metaJson: string; checksum: string } | null; + tailLog: { lsSeq: number; kind: string; payloadJson: string; bornPosHi: number | null }[]; + } { + const c = this.db + .prepare("SELECT ckpt_ls_seq, meta_json, checksum FROM pf_ckpt WHERE sid = ?") + .get(sid) as { ckpt_ls_seq: number; meta_json: string; checksum: string } | undefined; + const fromSeq = c ? c.ckpt_ls_seq : -1; + const tailLog = this.db + .prepare("SELECT ls_seq, kind, payload_json, born_pos_hi FROM pf_log WHERE sid = ? AND ls_seq > ? ORDER BY ls_seq") + .all(sid, fromSeq) as { ls_seq: number; kind: string; payload_json: string; born_pos_hi: number | null }[]; + return { + ckpt: c ? { ckptLsSeq: c.ckpt_ls_seq, metaJson: c.meta_json, checksum: c.checksum } : null, + tailLog: tailLog.map((r) => ({ lsSeq: r.ls_seq, kind: r.kind, payloadJson: r.payload_json, bornPosHi: r.born_pos_hi })), + }; + } + + /** Write the L1 checkpoint and (default recommendation, vetoable §4.1.6-⑤) + * physically drop the folded L0 prefix in the SAME transaction. */ + pfWriteCkpt(sid: string, ckptLsSeq: number, metaJson: string, checksum: string, truncateBeforeInclusive = true): void { + this.db.transaction(() => { + this.db + .prepare( + "INSERT INTO pf_ckpt (sid, ckpt_ls_seq, meta_json, checksum, updated_at) VALUES (?, ?, ?, ?, ?) " + + "ON CONFLICT (sid) DO UPDATE SET ckpt_ls_seq = excluded.ckpt_ls_seq, meta_json = excluded.meta_json, checksum = excluded.checksum, updated_at = excluded.updated_at", + ) + .run(sid, ckptLsSeq, metaJson, checksum, Date.now()); + if (truncateBeforeInclusive) { + this.db.prepare("DELETE FROM pf_log WHERE sid = ? AND ls_seq <= ?").run(sid, ckptLsSeq); + } + }); + } + + /** Windowed read of data-plane rows for visible intervals + * [lo, hi) — caller keeps intervals sorted & disjoint ascending so the + * result comes back globally pos-ordered. Deep-fold hydration can pass + * only the live windows (§4.1.4: hydration gets lighter as folds deepen). */ + pfReadRows(sid: string, intervals: ReadonlyArray): { pos: number; elem: number; ledgerId: string | null; coreId: string; coreJson: string; ts: number }[] { + const q = this.db.prepare( + "SELECT pos, elem, ledger_id, core_id, core_json, ts FROM pf_rows WHERE sid = ? AND pos >= ? AND pos < ? ORDER BY pos", + ); + const out: { pos: number; elem: number; ledgerId: string | null; coreId: string; coreJson: string; ts: number }[] = []; + for (const [lo, hi] of intervals) { + const rows = q.all(sid, lo, hi) as { pos: number; elem: number; ledger_id: string | null; core_id: string; core_json: string; ts: number }[]; + for (const r of rows) out.push({ pos: r.pos, elem: r.elem, ledgerId: r.ledger_id, coreId: r.core_id, coreJson: r.core_json, ts: r.ts }); + } + return out; + } + + pfMaxPos(sid: string): number { + const r = this.db.prepare("SELECT COALESCE(MAX(pos), -1) AS m FROM pf_rows WHERE sid = ?").get(sid) as { m: number }; + return r.m; + } + + /** Cascade cleanup for a set of pfsids — called from deleteSession via + * the session's ext registry (`pf:sids`), honoring the no-retention + * ruling. `insideTx=true` skips the wrapper when the caller already + * holds a transaction (better-sqlite3 has no nested BEGIN); the three + * deletes then ride the caller's transaction atomically. */ + pfDeleteSids(sids: string[], insideTx = false): void { + if (sids.length === 0) return; + const run = (): void => { + const ph = sids.map(() => "?").join(", "); + this.db.prepare(`DELETE FROM pf_rows WHERE sid IN (${ph})`).run(...sids); + this.db.prepare(`DELETE FROM pf_log WHERE sid IN (${ph})`).run(...sids); + this.db.prepare(`DELETE FROM pf_ckpt WHERE sid IN (${ph})`).run(...sids); + }; + if (insideTx) run(); + else this.db.transaction(run); + } + // ------------------------------------------------------------------ // Extension namespaces (#3069 v4). Host/plugin/UI substrate — never // kernel session state (that lives in kstate_json). Guards: key and diff --git a/tests/fastpath-pf-msgid.test.ts b/tests/fastpath-pf-msgid.test.ts new file mode 100644 index 0000000000..00d7e5b6ab --- /dev/null +++ b/tests/fastpath-pf-msgid.test.ts @@ -0,0 +1,590 @@ +// #5600 v6 msgid lane — identity rides server-minted pf_sids +// (x-bili-pf-session echo / ledger-id probe / mint), frozen cores come from +// the persistent pf_* partition store; hit authority is ledger-width +// arithmetic against OUR OWN frontier + window-origin pin. These tests pin: +// soft-plane degradation (store absent ⇒ legacy lanes byte-for-byte), +// cold-start mint/arm ingestion, geometry-only hits, hard-pin hits with +// ledger-id binding, shape/wrong-origin self-healing misses, compaction +// archive-and-reset (no row deletion, same sid echoed through the reset), +// post-boot hydration, L1 checkpoint fold round-trip, sess-change as an +// identity event, model-switch as a non-event, shared-breaker naming via +// credited misses, id-stable in-place edits riding frozen cores, and the +// untouched default (knob OFF) behavior as the regression anchor. +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import test from "node:test"; +import http from "node:http"; +import { openaiToCore } from "acp-kernel/wire"; +import { evaluateWatermarkFastpath, finalizeWatermarkTurn, watermarkDebugState, PF_SID_HEADER, PF_TAIL_IDS_HEADER } from "../src/server/fastpath.ts"; +import { stripRawStage } from "../src/marker/engine.ts"; +import { rolloutEnvName } from "../src/rollout.ts"; +import { _setStoreForTest, _resetStoreForTest, type SessionStore } from "../src/persist.ts"; +import type { Session } from "../src/session.ts"; + +const MSGID_ENV = rolloutEnvName("watermarkV6MsgId"); + +process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK = "1"; +process.env.BILI_ROLLOUT_WATERMARK_CANONICAL_V6 = "0"; + +const LOGS: string[] = []; +const log = (level: string, msg: string): void => { LOGS.push(`${level} ${msg}`); }; + +const USR = (t: string): { role: string; content: string } => ({ role: "user", content: t }); +const ASST = (t: string): { role: string; content: string } => ({ role: "assistant", content: t }); + +const headersOf = (h: Record): http.IncomingMessage => + ({ headers: h }) as unknown as http.IncomingMessage; + +const setMsgIdLane = (on: boolean): void => { + if (on) process.env[MSGID_ENV] = "1"; + else delete process.env[MSGID_ENV]; +}; + +// --------------------------------------------------------------------------- +// In-memory pf_* partition store (mirrors src/storage/store.ts pf* semantics: +// contiguous pos per sid, ls_seq continuing per sid, ckpt truncates tail +// inclusive, global ledger index first-hit by pos). +interface MemRow { pos: number; elem: number; ledgerId: string | null; coreId: string; coreJson: string; ts: number } +interface MemEntry { lsSeq: number; kind: string; payloadJson: string; bornPosHi: number | null } +class MemPF { + rows = new Map(); + logs = new Map(); + ckpts = new Map(); + ledger = new Map(); + appendTurn( + sid: string, + rws: { ledgerId: string | null; elem: number; coreId: string; coreJson: string }[], + entries: { kind: string; payload: unknown; bornPosHi?: number }[], + ): { maxPos: number; lsSeq: number } { + const rs = this.rows.get(sid) ?? []; + let pos = rs.length > 0 ? rs[rs.length - 1].pos : -1; + const now = Date.now(); + for (const r of rws) { + pos += 1; + rs.push({ pos, elem: r.elem, ledgerId: r.ledgerId, coreId: r.coreId, coreJson: r.coreJson, ts: now }); + if (r.ledgerId !== null && !this.ledger.has(r.ledgerId)) this.ledger.set(r.ledgerId, { sid, pos }); + } + this.rows.set(sid, rs); + const lg = this.logs.get(sid) ?? []; + let seq = lg.length > 0 ? lg[lg.length - 1].lsSeq : -1; + for (const e of entries) { + seq += 1; + lg.push({ lsSeq: seq, kind: e.kind, payloadJson: JSON.stringify(e.payload), bornPosHi: e.bornPosHi ?? null }); + } + this.logs.set(sid, lg); + return { maxPos: pos, lsSeq: seq }; + } + lookupLedger(id: string): { sid: string; pos: number } | null { return this.ledger.get(id) ?? null; } + verifyOriginPin(sid: string, elem: number, id: string): boolean { + return (this.rows.get(sid) ?? []).some((r) => r.elem === elem && r.ledgerId === id); + } + readMeta(sid: string): { ckpt: { ckptLsSeq: number; metaJson: string; checksum: string } | null; tailLog: MemEntry[] } { + return { ckpt: this.ckpts.get(sid) ?? null, tailLog: [...(this.logs.get(sid) ?? [])] }; + } + writeCkpt(sid: string, ckptLsSeq: number, metaJson: string, checksum: string, truncateBeforeInclusive = true): void { + this.ckpts.set(sid, { ckptLsSeq, metaJson, checksum }); + if (truncateBeforeInclusive) this.logs.set(sid, (this.logs.get(sid) ?? []).filter((e) => e.lsSeq > ckptLsSeq)); + } + readRows(sid: string, intervals: ReadonlyArray): MemRow[] { + const rs = this.rows.get(sid) ?? []; + return rs.filter((r) => intervals.some(([a, b]) => r.pos >= a && r.pos < b)).sort((a, b) => a.pos - b.pos); + } + maxPos(sid: string): number { + const rs = this.rows.get(sid) ?? []; + return rs.length > 0 ? rs[rs.length - 1].pos : -1; + } +} + +const makeStub = (pf: MemPF | "absent"): { ext: Map; pf: MemPF } => { + const ext = new Map(); + const mem = pf === "absent" ? undefined : pf; + _setStoreForTest({ + closeQuietly: (): void => {}, + replicasAvailable: (): boolean => true, + getSessionExt: (sid: string, k: string): unknown => ext.get(`${sid}\u0000${k}`), + setSessionExt: (sid: string, k: string, v: unknown): void => { void ext.set(`${sid}\u0000${k}`, v); }, + ...(mem === undefined ? {} : { pfStore: (): typeof mem => mem! }), + } as unknown as SessionStore); + return { ext, pf: mem as unknown as MemPF }; +}; + +const mkSession = (id: string): Session => ({ + id, + state: { blocks: [] }, + metadata: {}, +}) as unknown as Session; + +const runTurn = ( + session: Session, + reqHeaders: Record, + messages: unknown[], + model = "m1", +): { out: ReturnType; headers: Record } => { + const bodyText = JSON.stringify({ model, stream: false, messages }); + const out = evaluateWatermarkFastpath(bodyText, headersOf(reqHeaders), session, { pluginMode: true, titleGen: false, model }, log); + const oracle = openaiToCore(JSON.parse(bodyText) as never); + const stripCounts = stripRawStage(JSON.parse(bodyText), "openai"); + const headers = finalizeWatermarkTurn(session, out, { + model, + legacyCores: out.hit !== null ? out.hit.view : oracle.msgs, + rawSystemText: out.hit !== null ? out.hit.systemText : oracle.systemText, + turnStripCounts: stripCounts, + }); + return { out, headers }; +}; + +const SYSP = { role: "system", content: "pf-test-system-prompt-bytes" }; + +test("msgid lane off by default — legacy lanes untouched", () => { + try { + setMsgIdLane(false); + makeStub(new MemPF()); + const s = mkSession("s-off"); + const { out, headers } = runTurn(s, {}, [SYSP, USR("first user bytes off-lane")]); + assert.equal(out.sid, undefined, "default must not resolve a pf_sid"); + assert.equal(headers[PF_SID_HEADER], undefined, "no sid header when the lane stands aside"); + assert.ok(!String(out.decision).includes("v6m"), `unexpected msgid decision: ${out.decision}`); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("soft plane: lane live but pf store absent ⇒ legacy lanes byte-for-byte", () => { + try { + setMsgIdLane(true); + makeStub("absent"); + const s = mkSession("s-soft"); + const { out, headers } = runTurn(s, {}, [SYSP, USR("soft plane user bytes"), ASST("soft plane reply")]); + assert.equal(out.sid, undefined, "store unavailable must NOT mint (plane stands aside)"); + assert.equal(headers[PF_SID_HEADER], undefined); + assert.ok(!String(out.decision).includes("v6m"), `unexpected msgid decision: ${out.decision}`); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +const bootWorld = (pf: MemPF, session: Session): { sid: string; msgs: unknown[] } => { + // Cold start: sys + u1 + a1 → mint + arm ingest. + let msgs = [SYSP, USR("world u1 bytes"), ASST("world a1 reply")]; + let res = runTurn(session, {}, msgs); + assert.match(String(res.out.sid), /^pf_[0-9a-f]{12}$/, `arm must mint: ${res.out.decision}`); + assert.equal(res.out.decision, "v6m-arm"); + assert.equal(res.headers[PF_SID_HEADER], res.out.sid); + // Growth turn WITH ids: list [origin=?, new] binds the new element id. + msgs = [...msgs, USR("world u2 bytes")]; + res = runTurn(session, { [PF_SID_HEADER]: res.out.sid!, [PF_TAIL_IDS_HEADER]: "?,cidU2" }, msgs); + assert.equal(res.out.decision, "hit(v6-mid)", `geometry+weak-pin hit expected, got ${res.out.decision}`); + const sid = res.out.sid!; + // Frontier now carries cidU2 at its last pos. + assert.deepEqual(pf.lookupLedger("cidU2"), { sid, pos: pf.maxPos(sid) }); + return { sid, msgs }; +}; + +test("cold-start mints, arms, then hard-pin hit serves resident cores + rekeyed window", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + const stub = makeStub(pf); + const s = mkSession("s-hit"); + const { sid, msgs } = bootWorld(pf, s); + // Next turn: origin pin = recorded frontier id, one fresh message. + const next = [...msgs, ASST("world a2 reply")]; + const { out, headers } = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,cidA2" }, next); + assert.equal(out.decision, "hit(v6-mid)", `hard-pin hit expected, got ${out.decision}`); + assert.equal(out.canonical, true); + assert.ok(out.hit, "hit view present"); + // Resident cores (3 non-system) + window core (1) = 4 served cores. + assert.equal(out.hit!.view.length, 4); + assert.equal(headers[PF_SID_HEADER], sid, "echo continues the same partition"); + // Window row bound to cidA2; geom snapshot carries elements T=5 (sys,u1,a1,u2,a2). + const rows = pf.readRows(sid, [[0, 1_000_000]]); + assert.equal(rows.at(-1)?.ledgerId, "cidA2"); + const logTail = pf.readMeta(sid).tailLog.at(-1)!; + assert.equal(logTail.kind, "geom"); + assert.equal((JSON.parse(logTail.payloadJson) as { t: number }).t, 5); + void stub; + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("wrong origin / wrong length miss self-heals without touching the resident", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-heal"); + const { sid, msgs } = bootWorld(pf, s); + const hi = pf.maxPos(sid); + // Shape mismatch: two ids but W=1 ⇒ length must be 2… send 3. + let bad = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,x1,x2" }, [...msgs, USR("heal u3")]); + assert.equal(bad.out.decision, "v6m-shape"); + assert.equal(bad.out.creditedMiss, false); + // Unknown origin against a RECORDED frontier id ⇒ strict prev miss. + bad = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "bogus,cidU3" }, [...msgs, USR("heal u3")]); + assert.equal(bad.out.decision, "v6m-prev"); + assert.equal(bad.out.creditedMiss, false); + // No progress: same body as the stored world pins nothing new. + bad = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "?" }, msgs); + assert.equal(bad.out.decision, "v6m-no-progress"); + // Nothing was ingested by the misses: frontier unchanged. + assert.equal(pf.maxPos(sid), hi); + // Retry with the right origin heals to a hit. + const good = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,cidU3" }, [...msgs, USR("heal u3")]); + assert.equal(good.out.decision, "hit(v6-mid)", `self-heal expected, got ${good.out.decision}`); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("fabricated echo falls back to probe/mint and issues a fresh sid", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + const stub = makeStub(pf); + const s = mkSession("s-forged"); + const { out, headers } = runTurn(s, { [PF_SID_HEADER]: "pf_deadbeefcafe" }, [SYSP, USR("forged echo bytes")]); + assert.notEqual(out.sid, "pf_deadbeefcafe", "unknown partition must not be adopted"); + assert.match(String(out.sid), /^pf_[0-9a-f]{12}$/); + assert.equal(out.decision, "v6m-arm"); + assert.equal(headers[PF_SID_HEADER], out.sid, "the response IS the issuance channel"); + const reg = stub.ext.get("s-forged\u0000pf:sids") as unknown; + assert.deepEqual(reg, [out.sid], "registry records the mint under the bili session"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("client compaction: shrink is archive-and-reset, same sid, rows never deleted", () => { + try { + setMsgIdLane(true); + process.env.BILI_PF_V6_REWRITE_STRICT = "1"; + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-cmp"); + const { sid, msgs } = bootWorld(pf, s); + const posBefore = pf.maxPos(sid); + // Host compacted: the new view is SHORTER than the stored frontier. + const shrunken = [{ role: "system", content: "compact summary: earlier talk about X" }, USR("resume question after compaction")]; + const res = runTurn(s, { [PF_SID_HEADER]: sid }, shrunken); + assert.equal(res.out.decision, "v6m-compact"); + assert.equal(res.out.creditedMiss, true, "strict mode credits compaction-class misses"); + assert.equal(res.headers[PF_SID_HEADER], sid, "identity survives the reset"); + // Archive-and-reset: old rows remain, new world appended after them. + const rows = pf.readRows(sid, [[0, 1_000_000]]); + assert.ok(rows.length > posBefore + 1, "reset appends beyond the archived world"); + const entry = pf.readMeta(sid).tailLog.filter((e) => e.kind === "reset").at(-1)!; + const ep = JSON.parse(entry.payloadJson) as { t: number; fromPos: number }; + assert.equal(ep.fromPos, posBefore + 1, "new world starts after the archived one"); + assert.equal(ep.t, 2, "reset records the NEW world width in elements"); + delete process.env.BILI_PF_V6_REWRITE_STRICT; + // Next normal growth turn rides the NEW frontier (T'=3 > 2). + const grew = [...shrunken, USR("grew u")]; + // Two slots: origin pin "?" (weak-pin legal — the reset row was + // ingested id-silent) + one fresh "?" slot ⇒ geometry-only hit. + const g = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: ["?", "?"].join(",") }, grew); + assert.equal(g.out.decision, "hit(v6-mid)", `post-reset frontier ride expected, got ${g.out.decision}`); + } finally { setMsgIdLane(false); delete process.env.BILI_PF_V6_REWRITE_STRICT; _resetStoreForTest(); } +}); + +test("post-boot hydration rebuilds the resident from L1+L0 (and across an L1 fold)", async (t) => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + makeStub(pf); + const A = mkSession("s-hyd"); + const { sid, msgs: m0 } = bootWorld(pf, A); + // Grow until an L1 checkpoint fires (lsSeq % 32 === 0) — every even + // turn carries a USER multimodal element (openaiToCore splits it into + // text+IMG cores sharing the element wireIndex) so imgFlag sticks + // into the checkpoint state and the per-element id binding rides the + // split pair. + let msgs: unknown[] = [...m0]; + let lastT = 0; + for (let i = 5; i <= 40; i++) { + const body: unknown[] = i % 2 === 0 + ? [...msgs, { role: "user", content: [{ type: "text", text: `h u${i} look` }, { type: "image_url", image_url: { url: `data:image/png;base64,IMC${i}` } }] }] + : [...msgs, USR(`h u${i} plain`)]; + const last = pf.readRows(sid, [[pf.maxPos(sid), pf.maxPos(sid) + 1]]).at(-1)?.ledgerId; + const pin = last ?? "?"; + const r = runTurn(A, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: `${pin},hid${i}` }, body); + assert.equal(r.out.decision, "hit(v6-mid)", `turn ${i}: got ${r.out.decision}`); + msgs = body; + lastT = r.out.measuredT! + } + const ckpt = pf.readMeta(sid).ckpt; + assert.ok(ckpt, "an L1 fold must have fired by now"); + const sState = JSON.parse(ckpt.metaJson).s as { t: number; imgFlag: boolean }; + assert.equal(sState.imgFlag, true, "imgFlag persists into the checkpoint state"); + const elemsBefore = sState.t; + // Simulated restart: brand-new Session object, same store — the + // resident must rebuild itself from L1 ckpt + L0 tail, then serve a + // hard-pin hit at exactly the stored element frontier. + const B = mkSession("s-hyd-reborn"); + const frontierId = pf.readRows(sid, [[pf.maxPos(sid), pf.maxPos(sid) + 1]]).at(-1)?.ledgerId; + assert.ok(frontierId, "frontier row should carry a bound id"); + const next = [...msgs, USR("after reboot question")]; + const h = runTurn(B, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: `${frontierId},hidX` }, next); + assert.equal(h.out.decision, "hit(v6-mid)", `hydrated hit expected, got ${h.out.decision}`); + assert.equal(h.headers[PF_SID_HEADER], sid); + // Frontier is LAST ingest + 1 new element — later L0 turns after the + // fold advanced elemsBeyond the checkpoint state (elemsBefore ≤ lastT). + assert.ok(elemsBefore <= lastT, "fold state precedes the final ingest"); + assert.equal(h.out.measuredT, lastT + 1, "element frontier restored exactly across L1+L0"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("sess-change: swapped client-native session identity is an event (sid survives, resident untouched)", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-sess"); + const { sid, msgs } = bootWorld(pf, s); + // Teach the saved carrier baseline (cs) with a normal hit turn. + const a = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,cidQA", "x-conversation-id": "conv-A" }, [...msgs, USR("sess qA")]); + assert.equal(a.out.decision, "hit(v6-mid)", `baseline hit expected, got ${a.out.decision}`); + const hi = pf.maxPos(sid); + // Swap the carrier under the same bili session + sid ⇒ identity event. + // (The event fires BEFORE world-shape arithmetic — T===K here would + // otherwise read as no-progress, which pins the gate ordering too.) + const b = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,cidQB", "x-conversation-id": "conv-B" }, [...msgs, USR("sess qB")]); + assert.equal(b.out.decision, "sess-change", `carrier swap must be an event, got ${b.out.decision}`); + assert.equal(b.out.creditedMiss, false, "identity events are uncredited legal fulls"); + assert.equal(b.headers[PF_SID_HEADER], sid, "the partition identity survives the event"); + assert.equal(pf.maxPos(sid), hi, "the resident partition is untouched by the event"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("model switch within the wire family is not an event (the lane carries no model gate)", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-model"); + const { sid, msgs } = bootWorld(pf, s); + // Identical geometry + valid origin pin, different model string. + const r = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,cidA2" }, [...msgs, ASST("model a2 reply")], "m2"); + assert.equal(r.out.decision, "hit(v6-mid)", `model switch must not be an event, got ${r.out.decision}`); + assert.equal(r.out.canonical, true); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("breaker naming: msgid-lane credited misses arm the SHARED breaker (5 strikes / 60-min TTL)", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + const stub = makeStub(pf); + const s = mkSession("s-break"); + const { sid, msgs } = bootWorld(pf, s); + const hi = pf.maxPos(sid); + // A system element at the window head is the lane's always-credited + // drift verdict, and it writes NOTHING to the pf plane — repeatable + // against the same frontier, so it names the shared breaker cleanly. + for (let i = 0; i < 5; i++) { + const body = [...msgs, { role: "system", content: `intruder ${i}` }, USR(`break q${i}`)]; + const r = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "?,?,?" }, body); + assert.equal(r.out.decision, "v6m-window-system", `strike ${i + 1}: got ${r.out.decision}`); + assert.equal(r.out.creditedMiss, true); + } + const stateKey = `${s.id}\u0000pf:state`; + const st = JSON.parse(String(stub.ext.get(stateKey))) as Record; + assert.equal(typeof st.disabledAt, "number", "five consecutive credited misses arm the hold"); + assert.equal(pf.maxPos(sid), hi, "credited misses never touch the resident partition"); + // Held: the gate stands above every lane — no hit, but T keeps measuring. + const held = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "?,?,?" }, [...msgs, { role: "system", content: "intruder h" }, USR("break qh")]); + assert.equal(held.out.decision, "breaker-armed"); + assert.equal(held.headers[PF_SID_HEADER], undefined, "a held turn never resolves a sid"); + // TTL expiry pays off: the next verified miss counts fresh from zero. + stub.ext.set(stateKey, JSON.stringify({ ...st, disabledAt: Date.now() - 61 * 60 * 1000 })); + const rec = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "?,?,?" }, [...msgs, { role: "system", content: "intruder r" }, USR("break qr")]); + assert.equal(rec.out.decision, "v6m-window-system", `post-TTL miss expected, got ${rec.out.decision}`); + const st2 = JSON.parse(String(stub.ext.get(stateKey))) as Record; + assert.equal(st2.disabledAt, undefined, "an expired hold resets the strike counter"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("id-stable in-place edit rides the frozen core (geometry+ids judged, bytes never)", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-edit"); + const { sid, msgs } = bootWorld(pf, s); + const hi = pf.maxPos(sid); + // Client rewrites u1 IN PLACE, keeping geometry and ids; one new element. + const edited = [SYSP, USR("world u1 bytes EDITED"), ASST("world a1 reply"), USR("world u2 bytes"), USR("edit u3")]; + const r = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: "cidU2,cidU3" }, edited); + assert.equal(r.out.decision, "hit(v6-mid)", `origin pin must ride on the recorded id, got ${r.out.decision}`); + assert.equal(r.out.hit!.view[0]!.text, "world u1 bytes", "server snapshot stays absolute authority over the edited slot"); + // Only the window was ingested: exactly one new row, bound to cidU3. + const rows = pf.readRows(sid, [[hi + 1, 1_000_000]]); + assert.equal(rows.length, 1); + assert.equal(rows[0].ledgerId, "cidU3"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +// --------------------------------------------------------------------------- +// #5576 review round-1 (2026-11-04) regressions — B1 BLOCKER: the hydrated +// resident must serve the FULL active transcript, never just the last +// window. The pre-fix hydration consumed each geom entry's `f` (that +// WINDOW's pre-append start) as the row-read frontier, so after N windows +// a restart silently served "last window + new tail" while `t` still claimed +// full width. These tests assert the EXACT delivered texts across repeated +// simulated boots (fresh Session object, same persistent store), crossing +// L1 folds so the checkpoint's world-origin field carries the frontier. +// --------------------------------------------------------------------------- + +test("B1: repeated boots hydrate the FULL transcript, across L1 folds", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + makeStub(pf); + let session = mkSession("b1-s0"); + const { sid, msgs: m0 } = bootWorld(pf, session); + const served: string[] = ["world u1 bytes", "world a1 reply", "world u2 bytes"]; + let msgs: unknown[] = [...m0]; + const frontierPin = (): string => pf.readRows(sid, [[pf.maxPos(sid), pf.maxPos(sid) + 1]]).at(-1)?.ledgerId ?? "?"; + for (let c = 1; c <= 3; c++) { + // Grow 45 in-process turns before each boot — crosses the + // 32-entry L1 fold, so the checkpoint (not the raw arm entry) + // must carry the world origin through the restart. + for (let i = 1; i <= 45; i++) { + const t = `b1 c${c} u${i} bytes`; + served.push(t); + msgs = [...msgs, USR(t)]; + const r = runTurn(session, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: `${frontierPin()},bc${c}_${i}` }, msgs); + assert.equal(r.out.decision, "hit(v6-mid)", `cycle ${c} turn ${i}: got ${r.out.decision}`); + } + // Simulated restart: brand-new Session object (fresh residents + // map) over the SAME persistent store, then one more turn. + session = mkSession(`b1-s${c}`); + const t = `b1 c${c} reboot question`; + served.push(t); + msgs = [...msgs, USR(t)]; + const h = runTurn(session, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: `${frontierPin()},br${c}` }, msgs); + assert.equal(h.out.decision, "hit(v6-mid)", `boot ${c}: got ${h.out.decision}`); + // THE regression assertion: the view IS the complete transcript + // (pre-fix code served only the last window + this turn here). + assert.deepEqual( + h.out.hit!.view.map((v) => v.text), + served, + `boot ${c}: hydrated view must equal the full transcript`, + ); + } + // Writer side: every L1 checkpoint carries the active world ORIGIN + // (`o`) — cold-started here, never reset ⇒ always 0. + const ck = pf.readMeta(sid).ckpt; + assert.ok(ck, "an L1 fold must have fired"); + assert.equal(JSON.parse(ck.metaJson).o, 0, "checkpoint records the cold-start world origin"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +const ZERO_MARKERS = { panels: 0, statusBodies: 0, statusBodyLines: 0, markerLines: 0, nudges: 0, carriers: 0 }; +const sha256hex = (s: string): string => createHash("sha256").update(s).digest("hex"); + +test("B1/S4: untrusted checkpoints degrade exactly ONE turn, then self-heal (pre-fix shapes)", () => { + try { + setMsgIdLane(true); + const runBadCkptCase = (label: string, metaJson: string, checksum: string): void => { + const pf = new MemPF(); + makeStub(pf); + const s = mkSession(`bad-${label}`); + const { sid, msgs: m0 } = bootWorld(pf, s); + // Plant an UNTRUSTED checkpoint over the folded prefix: it + // truncates the tail exactly like a real fold did. + const tailSeq = pf.readMeta(sid).tailLog.at(-1)!.lsSeq; + pf.writeCkpt(sid, tailSeq, metaJson, checksum); + const bodyA = [...m0, USR(`bad-${label} qa`)]; + // Fresh Session objects for BOTH degraded turns — the in-process + // resident must NOT mask the broken partition (that is exactly + // the post-boot path under review). + const r1 = runTurn(mkSession(`bad-${label}-b1`), { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: ["?", "?"].join(",") }, bodyA); + assert.equal(r1.out.decision, "v6m-hydrate", `${label}: got ${r1.out.decision}`); + assert.equal(r1.out.hit, null, `${label}: degraded turn must not serve a fast view`); + // …and the failed turn itself repairs the partition by re-ingesting + // the FULL legacy view (append-only archive-and-reset). + const bodyB = [...bodyA, USR(`bad-${label} qb`)]; + const r2 = runTurn(mkSession(`bad-${label}-b2`), { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: ["?", "?"].join(",") }, bodyB); + assert.equal(r2.out.decision, "hit(v6-mid)", `${label} heal: got ${r2.out.decision}`); + assert.deepEqual( + r2.out.hit!.view.map((v) => v.text), + ["world u1 bytes", "world a1 reply", "world u2 bytes", `bad-${label} qa`, `bad-${label} qb`], + `${label}: healed view serves the full transcript`, + ); + }; + // (a) PRE-FIX shape: self-consistent hash but the old field set — + // `f` instead of `o`, i.e. no world origin recorded anywhere. + const oldMeta = JSON.stringify({ s: { t: 4, imgFlag: false, markers: ZERO_MARKERS }, f: 2 }); + runBadCkptCase("noo", oldMeta, sha256hex(oldMeta)); + // (b) CURRENT shape with a CORRUPTED checksum — state must be + // ignored the same way even though `o` parses fine inside it. + const curMeta = JSON.stringify({ s: { t: 4, imgFlag: false, markers: ZERO_MARKERS }, o: 0 }); + runBadCkptCase("sum", curMeta, "deadbeef"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +// #5576 review round-1 pins (V4/V5/V6, 2026-11-04). + +test("V4: compaction-class misses stay UNcredited when strict is off, and the frontier still self-heals", () => { + try { + setMsgIdLane(true); // BILI_PF_V6_REWRITE_STRICT deliberately UNset + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-cmp-loose"); + const { sid, msgs } = bootWorld(pf, s); + const shrunken = [{ role: "system", content: "compact summary round one" }, USR("resume q1")]; + const r1 = runTurn(s, { [PF_SID_HEADER]: sid }, shrunken); + assert.equal(r1.out.decision, "v6m-compact"); + assert.equal(r1.out.creditedMiss, false, "strict OFF: rewrite-class misses never feed the breaker"); + const grew = [...shrunken, USR("grew g")]; + const r2 = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: ["?", "?"].join(",") }, grew); + assert.equal(r2.out.decision, "hit(v6-mid)", `post-reset ride must heal, got ${r2.out.decision}`); + // second host compaction: still uncredited + const shrunk2 = [{ role: "system", content: "compact summary round two" }, USR("resume q2")]; + const r3 = runTurn(s, { [PF_SID_HEADER]: sid }, shrunk2); + assert.equal(r3.out.decision, "v6m-compact"); + assert.equal(watermarkDebugState(s).missCount ?? 0, 0, "no breaker credit accumulated across loose-mode compactions"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); + +test("S3/V5: a mid-flight read-side store fault degrades to the legacy lanes uncredited — no throw, no breaker credit (even under strict)", () => { + try { + setMsgIdLane(true); + process.env.BILI_PF_V6_REWRITE_STRICT = "1"; + const pf = new MemPF(); + makeStub(pf); + const s = mkSession("s-infra-fault"); + const { sid, msgs } = bootWorld(pf, s); + const before = watermarkDebugState(s).missCount ?? 0; + // Poison every READ surface the lane touches during evaluation. + const origMeta = pf.readMeta.bind(pf), origRows = pf.readRows.bind(pf), origLedger = pf.lookupLedger.bind(pf); + let fail = true; + const boom = (): never => { throw new Error("SQLITE_BUSY: database is locked"); }; + pf.readMeta = ((..._a: unknown[]) => { void _a; if (fail) return boom(); return origMeta.apply(pf as never, [] as never); }) as typeof pf.readMeta; + pf.readRows = ((...a: unknown[]) => fail ? boom() : origRows(...(a as Parameters))) as typeof pf.readRows; + pf.lookupLedger = ((id: string) => fail ? boom() : origLedger(id)) as typeof pf.lookupLedger; + try { + const r = runTurn(s, { [PF_SID_HEADER]: sid, [PF_TAIL_IDS_HEADER]: ["?", "?"].join(",") }, [...msgs, USR("during outage")]); + assert.ok(!String(r.out.decision).startsWith("v6m-"), `lane must have degraded away, got ${r.out.decision}`); + assert.equal(r.out.creditedMiss, false, "infra faults are NOT misses of this partition"); + } finally { + fail = false; + pf.readMeta = origMeta as typeof pf.readMeta; + pf.readRows = origRows as typeof pf.readRows; + pf.lookupLedger = origLedger as typeof pf.lookupLedger; + } + assert.equal(watermarkDebugState(s).missCount ?? before, before, "breaker untouched by the faulted turn"); + } finally { setMsgIdLane(false); delete process.env.BILI_PF_V6_REWRITE_STRICT; _resetStoreForTest(); } +}); + +test("V6 documented contract: forged-sid echoes mint one fresh sid per turn (registry grows linearly; cap policy pending owner ruling)", () => { + try { + setMsgIdLane(true); + const pf = new MemPF(); + const stub = makeStub(pf); + const s = mkSession("s-forge-burst"); + const mints: string[] = []; + for (let i = 0; i < 3; i++) { + const r = runTurn(s, { [PF_SID_HEADER]: `pf_forged${i}00` }, [SYSP, USR(`forge turn ${i}`)]); + assert.equal(r.out.decision, "v6m-arm"); + assert.equal(r.out.creditedMiss, false, "fabricated echoes are uncredited by design"); + assert.notEqual(r.headers[PF_SID_HEADER] === undefined, true); + mints.push(String(r.headers[PF_SID_HEADER])); + } + assert.ok(mints.every((m) => /^pf_[0-9a-f]{12}$/.test(m))); + assert.equal(new Set(mints).size, 3, "each forged echo probes, fails, and mints fresh (today: unbounded)"); + const reg = stub.ext.get("s-forge-burst\u0000pf:sids") as unknown; + assert.deepEqual(reg, mints, "registry grows linearly with the forged loop — pinned so any future cap lands deliberately"); + } finally { setMsgIdLane(false); _resetStoreForTest(); } +}); diff --git a/tests/fastpath-v6-canonical.test.ts b/tests/fastpath-v6-canonical.test.ts new file mode 100644 index 0000000000..e6bc3d4103 --- /dev/null +++ b/tests/fastpath-v6-canonical.test.ts @@ -0,0 +1,427 @@ +// #5576 v6 canonical-watermark lane — the hit verdict comes SOLELY from +// unified-transcript row-chain alignment (alignClientAgainstTranscript); +// the v4/v5 byte-compare machinery survives only as negative filters. +// These tests pin: anchored hits served from ROWS (covered regions behind +// synthetic anchors), fresh-head system passthrough, the R6 logical ledger +// (model switches are non-events), the R7 sess-change identity gate, +// rewrite-class misses (+ BILI_PF_V6_REWRITE_STRICT crediting), the ledger +// width/shrink filters, pre-fold/store degrade, the zero-retention doctrine +// (canonical outcomes pin NO seed stash), and — with the knob OFF — the +// untouched v5 behavior as the regression anchor. +import assert from "node:assert/strict"; +import test from "node:test"; +import http from "node:http"; +import { openaiToCore } from "acp-kernel/wire"; +import { evaluateWatermarkFastpath, finalizeWatermarkTurn, watermarkDebugState, PF_SEQ_HEADER, PF_TAIL_IDS_HEADER } from "../src/server/fastpath.ts"; +import { stripRawStage } from "../src/marker/engine.ts"; +import { _setStoreForTest, _resetStoreForTest, type SessionStore } from "../src/persist.ts"; +import type { Session } from "../src/session.ts"; + +process.env.BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK = "1"; + +const LOGS: string[] = []; +const log = (level: string, msg: string): void => { LOGS.push(`${level} ${msg}`); }; + +const SYS = (t: string): { role: string; content: string } => ({ role: "system", content: t }); +const USR = (t: string): { role: string; content: string } => ({ role: "user", content: t }); +const ASST = (t: string): { role: string; content: string } => ({ role: "assistant", content: t }); + +const bodyOf = (messages: unknown[], model = "m1"): string => + JSON.stringify({ model, stream: false, messages }); + +const headersOf = (h: Record): http.IncomingMessage => + ({ headers: h }) as unknown as http.IncomingMessage; + +type RowSeed = { id: string; role: string; contentType: string; text: string; wireIndex?: number }; +interface Row { seq: number; msgId: string; ref: null; refNum: null; role: string; kind: string; content: string; contentHash: string; tokens: number } + +/** Unified rows for `wirePrefix`, carrying EXACTLY the ids/sigs the legacy + * codec mints for those bytes (the alignment contract under test). */ +const rowsFrom = (wirePrefix: unknown[], offset = 1): { rows: Row[]; seeds: RowSeed[] } => { + const cores = openaiToCore({ model: "m1", messages: wirePrefix } as never) as { msgs: RowSeed[] }; + const rows: Row[] = cores.msgs.map((m, i) => ({ + seq: offset + i, + msgId: m.id, + ref: null, + refNum: null, + role: m.role, + kind: m.contentType, + content: JSON.stringify(m), + contentHash: `h${offset + i}`, + tokens: 3, + })); + return { rows, seeds: cores.msgs }; +}; + +interface StubStore { ext: Map; rows: Row[] | null } +const makeStubStore = (): StubStore => { + const ext = new Map(); + const s: StubStore = { ext, rows: null }; + _setStoreForTest({ + closeQuietly: (): void => {}, + replicasAvailable: (): boolean => true, + getSessionExt: (sid: string, k: string): unknown => ext.get(`${sid}\u0000${k}`), + setSessionExt: (sid: string, k: string, v: unknown): void => { void ext.set(`${sid}\u0000${k}`, v); }, + unifiedTailSeqs: (): number[] | null => (s.rows ? s.rows.map((_, i) => i + 1) : null), + unifiedMessageIdRows: () => (s.rows ? s.rows.map(({ seq, msgId }) => ({ seq, msgId })) : null), + unifiedTailViewRows: () => s.rows, + unifiedUncoveredRows: (_id: string, intervals: readonly [number, number][]) => + (s.rows ? s.rows.filter((r) => !intervals.some(([a, b]) => r.seq >= a && r.seq <= b)) : null), + } as unknown as SessionStore); + return s; +}; +const savedKey = (id: string): string => `${id}\u0000pf:state`; +const readSaved = (stub: StubStore, id: string): Record | null => { + const raw = stub.ext.get(savedKey(id)); + return raw === undefined ? null : (JSON.parse(String(raw)) as Record); +}; + +const mkSession = (id: string, coverRowIds: readonly string[]): Session => ({ + id, + state: { + blocks: [{ + blockId: "b1", runId: "r", tier: 1, summary: "S", + directMessageIds: [...coverRowIds], effectiveMessageIds: [...coverRowIds], + directBlockIds: [], compressedTokens: 9, createdAt: 0, survivedCount: 0, generation: 1, active: true, + }], + }, + metadata: {}, +}) as unknown as Session; + +const setV6 = (on: boolean): void => { process.env.BILI_ROLLOUT_WATERMARK_CANONICAL_V6 = on ? "1" : "0"; }; + +/** Drive one evaluated turn exactly like prepareOpenai does. */ +const runTurn = ( + session: Session, + reqHeaders: Record, + bodyText: string, +): { out: ReturnType; headers: Record } => { + const model = JSON.parse(bodyText).model as string; + const out = evaluateWatermarkFastpath(bodyText, headersOf(reqHeaders), session, { pluginMode: true, titleGen: false, model }, log); + const parsed = JSON.parse(bodyText) as { messages: unknown[] }; + const oracle = openaiToCore(parsed as never); + const stripCounts = stripRawStage(JSON.parse(bodyText), "openai"); + const headers = finalizeWatermarkTurn(session, out, { + model, + legacyCores: out.hit !== null ? out.hit.view : oracle.msgs, + rawSystemText: out.hit !== null ? out.hit.systemText : oracle.systemText, + turnStripCounts: stripCounts, + }); + return { out, headers }; +}; + +// Fixture: history U1/A1/U2 already folded-backed (rows), block b1 anchored +// on A1's row (covered → rendered behind its synthetic anchor). Every test +// below starts from exactly this state unless noted. +const FIX = { + u1: USR("unique-user-one-history-bytes"), + a1: ASST("unique-assistant-one-reply-bytes"), + u2: USR("unique-user-two-history-bytes"), + u3: USR("unique-user-three-fresh-question"), +}; + +const setupFix = (stub: StubStore): void => { + const { rows, seeds } = rowsFrom([FIX.u1, FIX.a1, FIX.u2]); + stub.rows = rows; + void seeds; +}; + +test("v6: count-only client hits via canonical alignment; view = rows + fresh tail; zero retention", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-hit", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + try { + // Arm the ledger the way turn A's (pre-fold) legacy turn left it: + // t = element count of the previous request (5 incl. system). + // Ledger arm = the previous turn's (pre-fold legacy) T: 4 elements + // ([sys,u1,a1,u2]); this turn appends u3 → T=5 > K=4. + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "m1", misses: 0 })); + const body = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2, FIX.u3]); + const r = runTurn(session, {}, body); + assert.equal(r.out.decision, "hit-v6"); + assert.equal(r.out.measuredT, 5); + assert.ok(r.out.hit); + const ids = r.out.hit.view.map((m) => m.id); + const seeded = rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds; + // Covered A1 renders BEHIND its synthetic anchor — exact order pinned. + assert.deepEqual(ids, [seeded[0].id, "acp_summary_b1", seeded[2].id, (openaiToCore({ model: "m1", messages: [FIX.u3] } as never) as { msgs: RowSeed[] }).msgs[0].id]); + assert.deepEqual(r.out.hit.windowMessages, [FIX.u3]); + assert.equal(r.out.hit.systemText, "sys-a"); + assert.equal(r.headers[PF_SEQ_HEADER], "5"); + const saved = readSaved(stub, session.id)!; + assert.equal(saved.t, 5); + assert.equal(saved.misses, 0); + // Zero-retention doctrine: a canonical hit pins NO seed stash. + assert.equal(watermarkDebugState(session).seeded, false); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: changed system text is passthrough, not drift (head-drift gate retired)", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-sys", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "m1", misses: 0 })); + const body = bodyOf([SYS("totally-different-system-now"), FIX.u1, FIX.a1, FIX.u2, FIX.u3]); + const r = runTurn(session, {}, body); + // v5 would answer head-drift + breaker credit here; v6 rides it. + assert.equal(r.out.decision, "hit-v6"); + assert.equal(r.out.creditedMiss, false); + assert.equal(r.out.hit?.systemText, "totally-different-system-now"); + assert.equal(readSaved(stub, session.id)?.misses, 0); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6 R6: model switch is a non-event on the shared logical ledger", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-model", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + try { + // State seeded by a DIFFERENT model (v5 answers seed-stale here). + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "other-model", misses: 0 })); + const body = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2, FIX.u3], "m2"); + const r = runTurn(session, {}, body); + assert.equal(r.out.decision, "hit-v6"); + assert.equal(readSaved(stub, session.id)?.m, "m2"); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6 R7: swapped client-native session is an identity event even on identical bytes", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-cs", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "m1", misses: 0, cs: "aa".repeat(4).slice(0, 8) })); + const body = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2, FIX.u3]); + const r = runTurn(session, { "x-conversation-id": "another-host-session" }, body); + assert.equal(r.out.decision, "sess-change"); + assert.equal(r.out.creditedMiss, false); + assert.equal(watermarkDebugState(session).seeded, false); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: client-side rewrite misses stay uncredited by default, credited under BILI_PF_V6_REWRITE_STRICT", () => { + for (const strict of ["unset", "on"]) { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession(`v6-rw-${strict}`, [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + if (strict === "on") process.env.BILI_PF_V6_REWRITE_STRICT = "1"; + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "m1", misses: 0 })); + // Two ADJACENT rewritten middle elements: one tolerated gap, the + // second consecutive miss ends the chain; the surviving run + // (u2, one match) is below MIN_SEAM_RUN → rewrite class. + const body = bodyOf([SYS("sys-a"), USR("edited-one"), ASST("edited-two"), FIX.u2, FIX.u3]); + const r = runTurn(session, {}, body); + assert.equal(r.out.decision, "v6-seam-too-short"); + const saved = readSaved(stub, session.id)!; + assert.equal(saved.misses, strict === "on" ? 1 : 0); + assert.equal(saved.disabledAt, undefined); + } finally { + delete process.env.BILI_PF_V6_REWRITE_STRICT; + setV6(false); + _resetStoreForTest(); + } + } +}); + +test("v6: epoch-style total rewrite → chain-not-started (rewrite class)", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-epoch", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "m1", misses: 0 })); + // Entirely novel content, ledger still growing (T=5 > K=4 so the + // width filters pass): nothing in the body anchors to a stored row. + const body = bodyOf([SYS("sys-a"), USR("brand-new-x"), ASST("brand-new-y"), USR("brand-new-z"), ASST("brand-new-w")]); + const r = runTurn(session, {}, body); + assert.equal(r.out.decision, "v6-chain-not-started"); + assert.equal(r.out.creditedMiss, false); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: ledger-width arithmetic catches a compaction shrink BEFORE the expensive stage", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-shrink", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + process.env.BILI_PF_V6_REWRITE_STRICT = "1"; // shrink stays uncredited even in strict + try { + // Previous turn stamped T=9; the host compacted down to 4 elements. + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 9, m: "m1", misses: 0 })); + const body = bodyOf([SYS("compaction-summary-passes-through"), ASST("kept-one"), USR("kept-two"), ASST("kept-three")]); + const list = ["?", "h_kepta", "h_keptb"].map(encodeURIComponent).join(","); + const r = runTurn(session, { [PF_TAIL_IDS_HEADER]: list }, body); + assert.equal(r.out.decision, "tail-ids-shape"); + assert.equal(r.out.creditedMiss, false); + assert.equal(readSaved(stub, session.id)?.misses, 0); + } finally { + delete process.env.BILI_PF_V6_REWRITE_STRICT; + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: no prior fold → cheap v6-pre-fold degrade; no store → v6-store-unavailable", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + setV6(true); + const body = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2, FIX.u3]); + // The no-blocks degrade must not depend on any stub being registered + // (store presence is irrelevant before the blocks pre-gate). + const noBlocks = { id: "v6-nofold", state: { blocks: [] as unknown[] }, metadata: {} }; + try { + const r1 = runTurn(noBlocks as never, {}, body); + assert.equal(r1.out.decision, "v6-pre-fold"); + assert.equal(r1.out.canonical, true); + } finally { + setV6(false); + _resetStoreForTest(); + } + // No STORE CAPABILITY at all: a registered store whose replicas report + // unavailable (getStore() auto-instantiates a real one when unset, so + // "null" cannot be simulated from here — capability-off is the honest + // injection). + _resetStoreForTest(); + setV6(true); + _setStoreForTest({ + closeQuietly: (): void => {}, + replicasAvailable: (): boolean => false, + getSessionExt: (): unknown => undefined, + setSessionExt: (): void => {}, + } as unknown as SessionStore); + try { + const r2 = runTurn(mkSession("v6-nostore", ["x"]), {}, body); + assert.equal(r2.out.decision, "v6-store-unavailable"); + assert.equal(r2.out.canonical, true); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: '?' slots ride the ledger list without penalty (identity-unverified, not a fault)", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v6-slots", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(true); + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 4, m: "m1", misses: 0 })); + const body = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2, FIX.u3]); + // Valid width (T-K+1 = 2) with a slot in it — v6 never consults the + // VALUES, only the width; slots must not disturb the hit. + const list = ["?"].concat(["h_some"]).map(encodeURIComponent).join(","); + const r = runTurn(session, { [PF_TAIL_IDS_HEADER]: list }, body); + assert.equal(r.out.decision, "hit-v6"); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: duplicate-content histories agree with the legacy full-parse numbering", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + // History CONTAINS a repeated user text — cluster numbering (_N suffixes) + // only agrees if the v6 conversion sees the same occurrence stream the + // legacy single-parse minted (full-body conversion). + const hist = [USR("ok"), ASST("reply-alpha"), USR("ok"), ASST("reply-beta")]; + const { rows, seeds } = rowsFrom(hist); + stub.rows = rows; + const session = mkSession("v6-dup", [seeds[1].id]); + setV6(true); + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 5, m: "m1", misses: 0 })); + const fresh = USR("unique-fresh-question"); + const body = bodyOf([SYS("sys-a"), ...hist, fresh]); + const r = runTurn(session, {}, body); + assert.equal(r.out.decision, "hit-v6", `expected canonical hit, got ${r.out.decision}`); + const last = r.out.hit!.view[r.out.hit!.view.length - 1]; + assert.equal(last.text, "unique-fresh-question"); + assert.deepEqual(r.out.hit!.windowMessages, [fresh]); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("v6: trailing element duplicating old content degrades safely (sig-fallback residual, never a silent tail drop)", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + const hist = [USR("ok"), ASST("reply-alpha"), USR("ok"), ASST("reply-beta")]; + const { rows, seeds } = rowsFrom(hist); + stub.rows = rows; + const session = mkSession("v6-dup-tail", [seeds[1].id]); + setV6(true); + try { + stub.ext.set(savedKey(session.id), JSON.stringify({ t: 5, m: "m1", misses: 0 })); + // The NEW trailing message repeats older text verbatim: its sig + // collides with a stored row and the full-length join would serve a + // view that silently drops the tail → guarded miss, legacy serves. + const body = bodyOf([SYS("sys-a"), ...hist, USR("ok")]); + const r = runTurn(session, {}, body); + assert.equal(r.out.decision, "v6-no-seam-origin"); + assert.equal(r.out.creditedMiss, false); + assert.equal(watermarkDebugState(session).seeded, false); + } finally { + setV6(false); + _resetStoreForTest(); + } +}); + +test("regression anchor: with v6 OFF the v5 machinery behaves byte-for-byte as before", () => { + _resetStoreForTest(); + const stub = makeStubStore(); + setupFix(stub); + const session = mkSession("v5-anchor", [rowsFrom([FIX.u1, FIX.a1, FIX.u2]).seeds[1].id]); + setV6(false); + try { + // Turn 1: FIRST request carries no stamp at all → v5 no-stamp miss, + // and finalize pins the seed stash (canonical skip must NOT fire on + // the v5 lane — retention semantics unchanged). + const b1 = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2]); + const r1 = runTurn(session, {}, b1); + assert.equal(r1.out.decision, "no-stamp"); + assert.notEqual(r1.out.canonical, true); + assert.equal(watermarkDebugState(session).seeded, true); + // Turn 2: echoes turn 1's measured T as stamp and appends u3 → the + // v5 seed path hits (NOT hit-v6). + const b2 = bodyOf([SYS("sys-a"), FIX.u1, FIX.a1, FIX.u2, FIX.u3]); + const r2 = runTurn(session, { "x-bili-pf-count": "4" }, b2); + assert.match(r2.out.decision, /^hit\b/); + assert.notEqual(r2.out.decision, "hit-v6"); + assert.ok(r2.out.hit && r2.out.hit.windowMessages.length === 1); + } finally { + _resetStoreForTest(); + } +}); diff --git a/tests/fastpath.test.ts b/tests/fastpath.test.ts index 8965a7b9f6..13a5632804 100644 --- a/tests/fastpath.test.ts +++ b/tests/fastpath.test.ts @@ -464,3 +464,114 @@ test("clearWatermarkSeed forces the next turn back through verification", () => const { out } = runTurn(session, { [PF_COUNT_HEADER]: "3" }, b2); assert.equal(out.decision, "unseeded"); }); + +// #5576 review round-1 (V1/V3, 2026-11-04): the v5 lane is the only LIVE +// identity lane today, so its #3617 bidirectionality matrix and the R7 +// sess-change gate get their positive-direction lines HERE — they previously +// existed only inside the canonical/msgid lanes. + +test("v5 #3617 matrix: id-mode stable-id edit RIDES with a warn (stale byte outbound); count-only same edit is GATED", () => { + // A) id mode: identity chain intact ⇒ hit; stashed cores ride OUTBOUND — + // the pre-edit byte is served, the edit is diagnostic-only (owner + // ruling 2026-11-03; feeds M3 re-seeding signals). + makeStubStore(); + const sA = freshSession("pf-v5-ride"); + assert.equal(runTurn(sA, {}, bodyOf([SYS("s"), USR("a"), USR("b")])).out.decision, "no-stamp"); + const bA2 = bodyOf([SYS("s"), USR("a"), USR("b"), USR("c")]); + assert.equal(runTurn(sA, { [PF_COUNT_HEADER]: "3", ...TAIL(["h-b", "h-c"]) }, bA2).out.decision, "hit"); + const LOGS_AT = LOGS.length; + // client rewrites element `b` IN PLACE while its host id stays `h-b` + const bA3 = bodyOf([SYS("s"), USR("a"), USR("b-EDITED"), USR("c"), ASST("d")]); + const rA = runTurn(sA, { [PF_COUNT_HEADER]: "4", ...TAIL(["h-c", "h-d"]) }, bA3); + assert.equal(rA.out.decision, "hit", LOGS.join("\n")); + assert.ok(LOGS.slice(LOGS_AT).some((l) => l.includes("riding the stashed prefix")), + "id-mode head drift must log the diagnostic ride line"); + const textsA = rA.out.hit!.view.map((m) => String((m as { text?: unknown }).text ?? "")); + assert.ok(textsA.includes("b"), "stale core byte rides outbound under id mode"); + assert.ok(!textsA.includes("b-EDITED"), "the edit must NOT be served from this lane"); + // B) count-only parallel: SAME bytes without an id list stay byte-gated — + // rewrite detected, uncredited miss, self-heal reseed (paired direction). + makeStubStore(); + const sB = freshSession("pf-v5-gate"); + assert.equal(runTurn(sB, {}, bodyOf([SYS("s"), USR("a"), USR("b")])).out.decision, "no-stamp"); + assert.equal(runTurn(sB, { [PF_COUNT_HEADER]: "3" }, bA2).out.decision, "hit"); + const mB = runTurn(sB, { [PF_COUNT_HEADER]: "4" }, bA3); + assert.equal(mB.out.decision, "head-prefix-drift", LOGS.join("\n")); + assert.equal(mB.out.creditedMiss, false); +}); + +test("v5 R7 port: sess-change on the DEFAULT lane — swap is an event, same carrier silent, absent carrier never one", () => { + const stub = makeStubStore(); + const s1 = freshSession("pf-r7-swap"); + assert.equal(runTurn(s1, {}, bodyOf([SYS("s"), USR("a"), USR("b")])).out.decision, "no-stamp"); + const h = { "x-conversation-id": "conv-A" }; + assert.equal(runTurn(s1, { [PF_COUNT_HEADER]: "3", ...TAIL(["h-b", "h-c"]), ...h }, + bodyOf([SYS("s"), USR("a"), USR("b"), USR("c")])).out.decision, "hit"); + assert.equal(typeof readSaved(stub, s1.id).cs, "string", "first claimed carrier persists as cs digest"); + // same carrier on the next turn = non-event + assert.equal(runTurn(s1, { [PF_COUNT_HEADER]: "4", ...TAIL(["h-c", "h-d"]), ...h }, + bodyOf([SYS("s"), USR("a"), USR("b"), USR("c"), ASST("d")])).out.decision, "hit"); + // swapped carrier = identity EVENT (uncredited, legacy + reseed) + const swapped = runTurn(s1, { [PF_COUNT_HEADER]: "5", ...TAIL(["h-d", "h-e"]), "x-conversation-id": "conv-B" }, + bodyOf([SYS("s"), USR("a"), USR("b"), USR("c"), ASST("d"), USR("e")])); + assert.equal(swapped.out.decision, "sess-change", LOGS.join("\n")); + assert.equal(swapped.out.creditedMiss, false); + // identity follows traffic: growth under the NEW carrier rides again + assert.equal(runTurn(s1, { [PF_COUNT_HEADER]: "6", ...TAIL(["h-e", "h-f"]), "x-conversation-id": "conv-B" }, + bodyOf([SYS("s"), USR("a"), USR("b"), USR("c"), ASST("d"), USR("e"), USR("f")])).out.decision, "hit"); + // neutral: state that never claimed a carrier never fires, even late + makeStubStore(); + const s2 = freshSession("pf-r7-neutral"); + assert.equal(runTurn(s2, {}, bodyOf([SYS("s"), USR("a"), USR("b")])).out.decision, "no-stamp"); + assert.equal(runTurn(s2, { [PF_COUNT_HEADER]: "3", ...TAIL(["h-b", "h-c"]) }, + bodyOf([SYS("s"), USR("a"), USR("b"), USR("c")])).out.decision, "hit"); + assert.equal(runTurn(s2, { [PF_COUNT_HEADER]: "4", ...TAIL(["h-c", "h-d"]), "x-conversation-id": "late-joiner" }, + bodyOf([SYS("s"), USR("a"), USR("b"), USR("c"), ASST("d")])).out.decision, "hit", + "carrier appearing where none was stored must be neutral (first claim, not a change)"); +}); + +test("v5 drift-ride lifetime cap: after 5 consecutive ride turns one UNcredited reseed forces the edit into the seed (review V1)", () => { + makeStubStore(); + const session = freshSession("pf-v1-cap"); + const b1 = bodyOf([SYS("s"), USR("a"), USR("b"), USR("c")]); + assert.equal(runTurn(session, {}, b1).out.decision, "no-stamp"); + // Rewrite element #2 IN PLACE with stable host ids + an honest count: + // id mode demotes the byte check to a diagnostic — the STALE byte rides. + let msgs = [SYS("s"), USR("a"), USR("b-EDITED"), USR("c"), ASST("d")]; + let r = runTurn(session, { [PF_COUNT_HEADER]: "4", ...TAIL(["h-c", "h-d"]) }, bodyOf(msgs)); + assert.equal(r.out.decision, "hit", LOGS.join("\n")); + assert.equal(r.out.creditedMiss, false); + assert.equal(r.out.driftRides, 1, "the first drift turn rides and counts itself"); + assert.equal(String((r.out.hit!.view[1] as { text?: unknown }).text ?? ""), "b", "stale stashed byte serves during the ride"); + // The ride re-stashes its own view, so a persistent edit keeps riding — + // the counter climbs one per consecutive drift turn... + msgs = [...msgs, USR("x0")]; + r = runTurn(session, { [PF_COUNT_HEADER]: "5", ...TAIL(["h-d", "h-x0"]) }, bodyOf(msgs)); + assert.equal(r.out.decision, "hit", LOGS.join("\n")); + assert.equal(r.out.driftRides, 2); + msgs = [...msgs, USR("x1")]; + r = runTurn(session, { [PF_COUNT_HEADER]: "6", ...TAIL(["h-x0", "h-x1"]) }, bodyOf(msgs)); + assert.equal(r.out.driftRides, 3); + msgs = [...msgs, USR("x2")]; + r = runTurn(session, { [PF_COUNT_HEADER]: "7", ...TAIL(["h-x1", "h-x2"]) }, bodyOf(msgs)); + assert.equal(r.out.driftRides, 4); + msgs = [...msgs, USR("x3")]; + r = runTurn(session, { [PF_COUNT_HEADER]: "8", ...TAIL(["h-x2", "h-x3"]) }, bodyOf(msgs)); + assert.equal(r.out.decision, "hit", LOGS.join("\n")); + assert.equal(r.out.driftRides, 5); + // ...until the sixth: the cap forces ONE uncredited legacy reseed — + // self-heal, never breaker fuel. + msgs = [...msgs, USR("x4")]; + r = runTurn(session, { [PF_COUNT_HEADER]: "9", ...TAIL(["h-x3", "h-x4"]) }, bodyOf(msgs)); + assert.equal(r.out.decision, "v5-drift-cap", LOGS.join("\n")); + assert.equal(r.out.creditedMiss, false); + assert.equal(watermarkDebugState(session).missCount ?? 0, 0, "the cap miss is a self-heal, not breaker fuel"); + // Reseeded from real bytes: the EDIT now rides and the head compares clean. + msgs = [...msgs, USR("x5")]; + r = runTurn(session, { [PF_COUNT_HEADER]: "10", ...TAIL(["h-x4", "h-x5"]) }, bodyOf(msgs)); + assert.equal(r.out.decision, "hit", LOGS.join("\n")); + assert.equal(r.out.driftRides, undefined, "fresh seed: the ride episode is over"); + const texts = r.out.hit!.view.map((m) => String((m as { text?: unknown }).text ?? "")); + assert.ok(texts.includes("b-EDITED"), "the reseed carried the edited bytes"); + assert.ok(!texts.some((t) => t === "b"), "the stale byte is gone from the view"); +}); diff --git a/tests/issue4666-migration-dedup.test.ts b/tests/issue4666-migration-dedup.test.ts index 465960d836..6a18f56c98 100644 --- a/tests/issue4666-migration-dedup.test.ts +++ b/tests/issue4666-migration-dedup.test.ts @@ -44,7 +44,7 @@ import * as os from "node:os"; import * as path from "node:path"; import { UnifiedStore, sha256Hex } from "../src/storage/store.ts"; import { openSqliteDatabase } from "../src/storage/driver.ts"; -import { STORAGE_DDL_V1_FOR_TESTS } from "../src/storage/schema.ts"; +import { STORAGE_DDL_V1_FOR_TESTS, STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; import { identityOfStoredContent } from "../src/storage/identity.ts"; import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; import { openBackend, type StorageBackend } from "../src/storage/backend.ts"; @@ -155,7 +155,7 @@ test("#4666 sqlite: v8 migration collapses wireIndex-split duplicates and refund check = openSqliteDatabase(path.join(dir, "index.db")).db; const ver = check.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10, "stamped to v10 (#4313 adds the timestamps column)"); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION, "stamped to the current schema head (#4313 timestamps column + #5600 pf plane)"); // Exactly the lowest-seq arrival of each (msg_id, identity) group. const survivors = check @@ -368,10 +368,10 @@ test("#4666 mysql: v8 upgrade collapses duplicates, crash-safe stamp gate, retir const H = await createMysqlHarness(t); let be: StorageBackend | undefined; try { - // Fresh install lands at v10. + // Fresh install lands at the schema head. be = await openBackend("mysql", H.dsn); await be.init?.(); - assert.equal(await H.stamp(), "10", "fresh install stamped v10"); + assert.equal(await H.stamp(), String(STORAGE_SCHEMA_VERSION), "fresh install stamped at the schema head"); // Downgrade to the legacy v7 shape and plant the dirty fixture. await H.rewindToV7(); @@ -381,7 +381,7 @@ test("#4666 mysql: v8 upgrade collapses duplicates, crash-safe stamp gate, retir await be.close(); be = await openBackend("mysql", H.dsn); await be.init?.(); - assert.equal(await H.stamp(), "10", "upgrade stamped v10"); + assert.equal(await H.stamp(), String(STORAGE_SCHEMA_VERSION), "upgrade stamped at the schema head"); const survivors = await H.q("SELECT seq, msg_id, content_hash, identity FROM messages WHERE session_id = 's1' ORDER BY seq"); assert.deepEqual( @@ -430,7 +430,7 @@ test("#4666 mysql: v8 upgrade collapses duplicates, crash-safe stamp gate, retir await be.close(); be = await openBackend("mysql", H.dsn); await be.init?.(); - assert.equal(await H.stamp(), "10", "re-run completed and stamped v10"); + assert.equal(await H.stamp(), String(STORAGE_SCHEMA_VERSION), "re-run completed and stamped at the schema head"); const eRows = await H.q("SELECT seq, identity FROM messages WHERE session_id = 's2' ORDER BY seq"); assert.deepEqual(eRows.map((r) => Number(r.seq)), [5, 13], "unbackfilled dup pair collapsed; the CAS-missing row survives (not a duplicate)"); assert.notEqual(String(eRows[0].identity), "", "identity backfilled on the re-run"); @@ -537,7 +537,7 @@ test("#4666 mysql: concurrent boots serialize on the migration lock", assert.equal(Number(released[0].ok), 1, "lock released exactly once"); locker.release(); await migrating; - assert.equal(await H.stamp(), "10", "migration completed after the lock freed"); + assert.equal(await H.stamp(), String(STORAGE_SCHEMA_VERSION), "migration completed after the lock freed"); const s1 = await H.q("SELECT seq FROM messages WHERE session_id = 's1' ORDER BY seq"); assert.deepEqual(s1.map((r) => Number(r.seq)), [1, 2, 3, 5], "fixture collapsed exactly once — no double-delete"); const sLock = await H.q("SELECT seq, identity FROM messages WHERE session_id = 's_lock' ORDER BY seq"); diff --git a/tests/issue4677-v8-migration-streaming.test.ts b/tests/issue4677-v8-migration-streaming.test.ts index a14cb6888c..71b6cc2147 100644 --- a/tests/issue4677-v8-migration-streaming.test.ts +++ b/tests/issue4677-v8-migration-streaming.test.ts @@ -26,7 +26,7 @@ import * as fs from "node:fs"; import * as os from "node:os"; import * as path from "node:path"; import { openSqliteDatabase } from "../src/storage/driver.ts"; -import { migrateSchema } from "../src/storage/schema.ts"; +import { migrateSchema, STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; import { identityOfStoredContent } from "../src/storage/identity.ts"; process.env.BILI_PERSIST_ZSTD = "0"; @@ -99,7 +99,7 @@ test("#4677 backfill sets identities and drops wireIndex-split duplicates (lowes const { from, to } = migrateSchema(db); assert.equal(from, 7); - assert.equal(to, 10); + assert.equal(to, STORAGE_SCHEMA_VERSION); const identity = identityOfStoredContent(storedMessage("stripped", "hello")); assert.equal(identityOf(db, "s1", 1), identity); // the later duplicate row is gone and its content CAS ref handed back @@ -109,7 +109,7 @@ test("#4677 backfill sets identities and drops wireIndex-split duplicates (lowes ); assert.equal(refcountOf(db, a2), 0); assert.equal(refcountOf(db, a1), 1); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 10); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, STORAGE_SCHEMA_VERSION); db.close(); }); @@ -129,7 +129,7 @@ test("#4677 crash resume: sessions that already carry identities are skipped, no assert.equal(identityOf(db, "s-done", 1), "stale-planted"); assert.equal(identityOf(db, "s-pending", 1), identityOfStoredContent(storedMessage("pending", "still fresh"))); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 10); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, STORAGE_SCHEMA_VERSION); db.close(); }); @@ -150,7 +150,7 @@ test("#4677 rows without content bytes terminate with the ABSENT marker instead assert.equal(identity, "absent:"); // second run is a no-op — the marker is not identity = '', so the // session is not selected again - assert.equal(migrateSchema(db).from, 10); + assert.equal(migrateSchema(db).from, STORAGE_SCHEMA_VERSION); assert.equal(identityOf(db, "s-ghost", 1), "absent:"); db.close(); }); @@ -193,7 +193,7 @@ test("#4677 empty store migrates to the schema head untouched", () => { downgradeToV7(db); const { from, to } = migrateSchema(db); assert.equal(from, 7); - assert.equal(to, 10); - assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, 10); + assert.equal(to, STORAGE_SCHEMA_VERSION); + assert.equal((db.prepare("PRAGMA user_version").get() as { user_version: number }).user_version, STORAGE_SCHEMA_VERSION); db.close(); }); diff --git a/tests/issue4796-gc-batched-sweep.test.ts b/tests/issue4796-gc-batched-sweep.test.ts index 5e9f3a23cd..f011ca8836 100644 --- a/tests/issue4796-gc-batched-sweep.test.ts +++ b/tests/issue4796-gc-batched-sweep.test.ts @@ -7,6 +7,7 @@ import { createHash } from "node:crypto"; import { DatabaseSync } from "node:sqlite"; import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; import { UnifiedStore } from "../src/storage/store.ts"; +import { STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; /** * #4796 — zero-ref CAS collection: schema v9 index + batched gc sweep. @@ -84,7 +85,7 @@ test("#4796 v9: fresh install carries idx_content_refcount and candidate selecti seedContentRows(rawDb(store), 20_000, 1, "live"); seedContentRows(rawDb(store), 50, 0, "orphan"); const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION); const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_content_refcount'") .get(); @@ -118,7 +119,7 @@ test("#4796 v9: reopening a v8-era db gains the index", () => { const store = UnifiedStore.open(dbPath); try { const ver = rawDb(store).prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION); const idx = rawDb(store) .prepare("SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_content_refcount'") .get(); @@ -243,7 +244,7 @@ test("#4796 e2e: v8 identity backfill releases wireIndex twins -> gc collects th try { const db2 = rawDb(reopened); const ver = db2.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION); const twinLeft = db2 .prepare("SELECT COUNT(*) AS n FROM messages WHERE session_id = ? AND seq = ?") .get(sessId, twinSeq) as { n: number }; diff --git a/tests/native-identity-observe.test.ts b/tests/native-identity-observe.test.ts new file mode 100644 index 0000000000..ac98d11ae3 --- /dev/null +++ b/tests/native-identity-observe.test.ts @@ -0,0 +1,446 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import http from "node:http"; +import { once } from "node:events"; +import os from "node:os"; +import path from "node:path"; +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest, _resetStoreForTest } from "../src/persist.ts"; +import { _resetPluginStateForTest } from "../src/plugin.ts"; +import { _resetSessionsForTest } from "../src/session.ts"; +import { setLogCapture } from "../src/logger.ts"; +import { closeServers } from "./server-close-guard.ts"; +import { NIDV_EXT_KEY, digest8, extractNativeIdentity, observeNativeIdentity, type NativeIdInput } from "../src/server/native-identity.ts"; + +// #5573 M1 (L1): native-identity observation layer. Unit pins: profile +// fixtures (positive + negative per client), the stability matrix, the +// compaction-prefix adversarial cases, the PII red line (digests only), the +// event FIFO/budget caps, and restart continuity. Hermetic pins: the rollout +// gate (off ⇒ nothing), byte-stable A/B (on ⇒ ZERO diff at the upstream even +// for known clients), the WARN escalation on compaction hits, and the +// decision-independence pin (x-conversation-id alone still resolves via +// prefix affinity until #5526 lands). + +// Ext namespaces only exist on the unified store (#2671); the fastpath pf:state +// pattern this module follows reads/writes them there. Pin "full" for the whole +// file so persistence assertions are deterministic (bounded-storage.test.ts pattern). +const prevUnified = process.env.BILI_STORAGE_UNIFIED; +process.env.BILI_STORAGE_UNIFIED = "full"; +test.after(() => { + if (prevUnified === undefined) delete process.env.BILI_STORAGE_UNIFIED; + else process.env.BILI_STORAGE_UNIFIED = prevUnified; +}); + +const WB_HEADERS: Record = { + "x-conversation-id": "conv-wb-1", + "x-session-id": "acct-1", + "x-conversation-request-id": "req-7", + "x-conversation-message-id": "msguuid", + "x-compaction": "auto", + "x-context-compacted": "1", + "x-codebuddy-request": "1", + "x-ide-name": "WorkBuddy", + "x-ide-type": "WorkBuddy", + "x-ide-version": "5.9.3", +}; + +function tmpStore(): SessionStore { + const dir = mkdtempSync(path.join(os.tmpdir(), "nidv-unit-")); + // enabled (default) is required: the unified sidecar — and with it the ext + // namespaces this module persists into — only opens for live instances. + return new SessionStore({ dir: path.join(dir, "sessions") }); +} + +function runObs(store: SessionStore | null, input: NativeIdInput): { lines: string[]; warns: string[] } { + const out: { lines: string[]; warns: string[] } = { lines: [], warns: [] }; + observeNativeIdentity(input, store, (l, m) => { + out.lines.push(m); + if (l === "warn") out.warns.push(m); + }); + return out; +} + +test("profile wb: full family extracts every field", () => { + const ex = extractNativeIdentity({ headers: WB_HEADERS }); + assert.equal(ex?.client, "wb"); + assert.equal(ex?.sessH, digest8("conv-wb-1")); + assert.equal(ex?.turnH, digest8("req-7")); + assert.equal(ex?.msgidH, digest8("msguuid")); + assert.equal(ex?.edge, "auto"); + assert.equal(ex?.level, true); + assert.ok(ex?.noteParts.some((p) => p === "fp:x-ide-version=5.9.3")); +}); + +test("no identity evidence → null (unclassified traffic is not recorded)", () => { + assert.equal(extractNativeIdentity({ headers: {} }), null); + assert.equal(extractNativeIdentity({ headers: { "content-type": "application/json", "user-agent": "curl/8" } }), null); +}); + +test("array-valued and empty headers count as absent", () => { + assert.equal( + extractNativeIdentity({ headers: { "x-conversation-id": ["a", "b"], "x-request-id": " " } }), + null, + ); +}); + +test("profile codex: installation-id match, body signals, rotation, preface in USER element", () => { + const body = JSON.stringify({ + model: "gpt-x", + previous_response_id: "rs_abc123", + input: [ + { type: "message", role: "user", content: [{ type: "input_text", text: "Another language model started to solve this problem. Its work follows." }] }, + { id: "msg_1", type: "message", role: "assistant", content: [{ type: "output_text", text: "ok" }] }, + { id: "ctc_1", type: "function_call", name: "f", call_id: "call_1", arguments: "{}" }, + ], + }); + const ex = extractNativeIdentity({ + headers: { "x-codex-installation-id": "inst-42", "session-id": "sess-cx" }, + body, + protocol: "responses", + path: "/v1/responses", + }); + assert.equal(ex?.client, "codex"); + assert.equal(ex?.sessH, digest8("sess-cx")); + assert.equal(ex?.prevRespH, digest8("rs_abc123")); + const counts = Object.fromEntries((ex?.counts ?? []).map((c) => [c.label, c.n])); + assert.equal(counts["msg_"], 1); + assert.equal(counts["ctc_"], 1); + assert.equal(ex?.prefix?.role, "user"); + assert.equal(ex?.prefix?.elemIdx, 1); + const r = runObs(tmpStore(), { headers: { "x-codex-installation-id": "inst-42", "session-id": "sess-cx" }, body, protocol: "responses" }); + assert.ok(r.warns.length >= 1, "preface in a user element escalates to WARN"); + assert.ok(r.lines[0].includes("evt=compaction(preface@elem1)")); +}); + +test("profile codex adversarial: preface inside an ASSISTANT element is NOT a compaction event", () => { + const body = JSON.stringify({ + input: [ + { type: "message", role: "user", content: [{ type: "input_text", text: "question" }] }, + { type: "message", role: "assistant", content: [{ type: "output_text", text: "Quoting the transcript: Another language model started to solve this problem ..." }] }, + ], + }); + const ex = extractNativeIdentity({ headers: { "x-codex-installation-id": "i" }, body, protocol: "responses" }); + assert.equal(ex?.prefix?.role, "assistant"); + const r = runObs(null, { headers: { "x-codex-installation-id": "i" }, body, protocol: "responses" }); + assert.ok(r.warns.length === 0, "mid-history assistant occurrence must not fire the incident class"); + assert.ok(r.lines[0].includes("prefix@elem2/assistant"), "recorded as an honest observation instead"); +}); + +test("profile codex secondary match: request-id ∧ responses protocol; wrong protocol stays unclassified", () => { + const ex = extractNativeIdentity({ headers: { "x-client-request-id": "r-9" }, protocol: "responses" }); + assert.equal(ex?.client, "codex"); + assert.equal(extractNativeIdentity({ headers: { "x-client-request-id": "r-9" }, protocol: "openai" }), null); +}); + +test("profile cc: dual-channel cross-check ok / mismatch / malformed-swallowed", () => { + const hdrs = { "x-claude-code-session-id": "cc-sess-1" }; + const ok = extractNativeIdentity({ headers: hdrs, body: '{"metadata":{"user_id":{"session_id":"cc-sess-1"}},"messages":[{"role":"user","content":"hi"}]}' }); + assert.equal(ok?.dual, "ok"); + const bad = extractNativeIdentity({ headers: hdrs, body: '{"metadata":{"user_id":{"session_id":"cc-OTHER"}},"messages":[]}' }); + assert.equal(bad?.dual, "mismatch"); + const rBad = runObs(null, { headers: hdrs, body: '{"metadata":{"user_id":{"session_id":"cc-OTHER"}},"messages":[]}' }); + assert.ok(rBad.warns.some((w) => w.includes("dual_mismatch"))); + // Malformed metadata shapes degrade to "absent" — swallowed, never a crash, never a false mismatch. + for (const mal of ['{"metadata":"garbage","messages":[]}', '{"metadata":{"user_id":123},"messages":[]}', '{"messages":[]}']) { + const m = extractNativeIdentity({ headers: hdrs, body: mal }); + assert.equal(m?.dual, "absent", mal); + assert.equal(runObs(null, { headers: hdrs, body: mal }).warns.length, 0, mal); + } +}); + +test("profile opencode: ses_ shape gate, assertive zero-counts, drift event, fork marker", () => { + const clean = extractNativeIdentity({ headers: { "x-session-affinity": "ses_abc" }, body: '{"messages":[{"role":"user","content":"hi"}]}' }); + assert.equal(clean?.client, "opencode"); + assert.equal(clean?.sessH, digest8("ses_abc")); + assert.deepEqual(clean?.counts.map((c) => c.n), [0, 0, 0]); + assert.equal(runObs(null, { headers: { "x-session-affinity": "ses_abc" }, body: '{"messages":[{"role":"user","content":"hi"}]}' }).warns.length, 0); + // Shape gate: a non-ses_ affinity value matches the profile but yields NO session id. + const shaped = extractNativeIdentity({ headers: { "x-session-affinity": "not-ses" } }); + assert.equal(shaped?.client, "opencode"); + assert.equal(shaped?.sessH, undefined); + // Wire-shape drift: any prt_/ses_/msg_ token on the outbound wire is an anomaly. + const drifted = runObs(null, { headers: { "x-session-affinity": "ses_abc" }, body: '{"messages":[{"role":"user","content":"see prt_123"}]}' }); + assert.ok(drifted.warns.some((w) => w.includes("shape_assert_fail(prt_=1)"))); + // Fork marker. + const forked = runObs(null, { headers: { "x-session-affinity": "ses_child", "x-parent-session-id": "ses_parent" } }); + assert.ok(forked.lines[0].includes("fork=yes")); + assert.ok(forked.lines[0].includes("evt=fork(parent-sess)")); +}); + +test("profile gemini: protocol-only match + state-snapshot preface", () => { + const body = '{"contents":[{"role":"user","parts":[{"text":"\\nwhat we know so far"}]}]}'; + const ex = extractNativeIdentity({ headers: {}, body, protocol: "google" }); + assert.equal(ex?.client, "gemini"); + assert.equal(ex?.prefix?.role, "user"); + assert.equal(runObs(null, { headers: {}, body, protocol: "google" }).warns.length, 1); +}); + +test("plugin lane: agent stamp labels the client, msgIds envelope drives the stats", () => { + const store = tmpStore(); + const first = runObs(store, { key: "lane-conv-1", headers: {}, agent: "dsh", msgIds: ["m1", "m2", "m1"] }); + assert.ok(first.lines[0].startsWith("nidv: client=dsh ")); + assert.ok(first.lines[0].includes("msgΔ=+3")); + const second = runObs(store, { key: "lane-conv-1", headers: {}, agent: "dsh", msgIds: ["m2", "m3"] }); + assert.ok(second.lines[0].includes("msgΔ=+2")); + const rec = JSON.parse(String(store.getSessionExt(`n:${digest8("lane-conv-1")}`, NIDV_EXT_KEY))) as { ids: { msg?: { n: number; distinct: number; stable: string } } }; + assert.equal(rec.ids.msg?.n, 5); + assert.equal(rec.ids.msg?.distinct, 3); + assert.equal(rec.ids.msg?.stable, "yes", "m2 repeated across turns = host reuses ids (retry-stable evidence)"); +}); + +test("stability matrix: session repeat→yes, change→no + id_change, restart continuity", () => { + const store = tmpStore(); + const base = { key: "fixed", headers: { "x-conversation-id": "conv-A" } }; + runObs(store, base); + let rec = JSON.parse(String(store.getSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY))) as { ids: { sess?: { h: string; stable: string } } }; + assert.equal(rec.ids.sess?.stable, "unknown", "first sighting is undecided"); + runObs(store, base); + rec = JSON.parse(String(store.getSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY))) as typeof rec; + assert.equal(rec.ids.sess?.stable, "yes"); + const r = runObs(store, { key: "fixed", headers: { "x-conversation-id": "conv-B" } }); + assert.ok(r.lines[0].includes("id_change(sess)")); + rec = JSON.parse(String(store.getSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY))) as typeof rec; + assert.equal(rec.ids.sess?.stable, "no"); + assert.equal(rec.ids.sess?.h, digest8("conv-B")); + // Restart continuity: a preloaded prior record continues seamlessly. + const store2 = tmpStore(); + const prior = { + v: 1, client: "wb", firstSeen: 1000, lastSeen: 2000, reqs: 5, + ids: { sess: { h: digest8("conv-A"), stable: "unknown" as const } }, events: [], + }; + store2.setSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY, JSON.stringify(prior)); + runObs(store2, base); + const cont = JSON.parse(String(store2.getSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY))) as { firstSeen: number; reqs: number; ids: { sess?: { stable: string } } }; + assert.equal(cont.firstSeen, 1000, "firstSeen survives the restart"); + assert.equal(cont.reqs, 6); + assert.equal(cont.ids.sess?.stable, "yes"); +}); + +test("msg stability: ≥8 all-distinct turns → no", () => { + const store = tmpStore(); + for (let i = 0; i < 9; i += 1) { + runObs(store, { key: "fixed", headers: {}, agent: "dsh", msgIds: [`id-${String(i)}`] }); + } + const rec = JSON.parse(String(store.getSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY))) as { ids: { msg?: { n: number; stable: string } } }; + assert.equal(rec.ids.msg?.n, 9); + assert.equal(rec.ids.msg?.stable, "no"); +}); + +test("profile_flip event + FIFO cap + 4KB budget", () => { + const store = tmpStore(); + for (let i = 0; i < 40; i += 1) { + runObs(store, { key: "fixed", headers: {}, agent: i % 2 === 0 ? "dsh" : "omp", msgIds: [`flip-${String(i)}`] }); + } + const raw = String(store.getSessionExt(`n:${digest8("fixed")}`, NIDV_EXT_KEY)); + assert.ok(raw.length <= 4096, `ext record fits the budget (${raw.length})`); + const rec = JSON.parse(raw) as { events: { k: string }[]; client: string }; + assert.ok(rec.events.length <= 32); + assert.ok(rec.events.some((e) => e.k === "profile_flip")); + assert.equal(rec.client, "omp"); +}); + +test("PII red line: raw id values never persist or log — sha1-8 digests only", () => { + const store = tmpStore(); + const r = runObs(store, { headers: WB_HEADERS, body: '{"messages":[{"role":"user","content":"hello"}]}' }); + const raw = String(store.getSessionExt(`n:${digest8("conv-wb-1")}`, NIDV_EXT_KEY)); + for (const secret of ["conv-wb-1", "msguuid", "req-7", "acct-1"]) { + assert.ok(!raw.includes(secret), `ext must not contain ${secret}`); + assert.ok(!r.lines[0].includes(secret), `log line must not contain ${secret}`); + } + assert.ok(r.lines[0].includes(`${digest8("conv-wb-1")}\u2026`), "line carries the digest + ellipsis form"); + assert.ok(raw.includes(digest8("conv-wb-1")), "ext carries the digest"); +}); + +test("explicit legacy storage opt-out degrades to log-line-only (no crash, no record)", () => { + // Absent env resolves to "full" (#3818), so the legacy path needs the + // explicit spelling. + const saved = process.env.BILI_STORAGE_UNIFIED; + process.env.BILI_STORAGE_UNIFIED = "legacy"; + try { + const dir = mkdtempSync(path.join(os.tmpdir(), "nidv-legacy-")); + const legacyStore = new SessionStore({ dir: path.join(dir, "sessions") }); + assert.equal(legacyStore.replicasAvailable(), false); + const r = runObs(legacyStore, { headers: WB_HEADERS }); + assert.equal(r.lines.length, 1, "observation still logs"); + assert.ok(r.lines[0].startsWith("nidv: client=wb "), r.lines[0]); + let threw = false; + try { + legacyStore.getSessionExt(`n:${digest8("conv-wb-1")}`, NIDV_EXT_KEY); + } catch { + threw = true; + } + assert.equal(threw, true, "legacy-only ext access still fails fast — the module must never have reached it"); + } finally { + if (saved === undefined) delete process.env.BILI_STORAGE_UNIFIED; + else process.env.BILI_STORAGE_UNIFIED = saved; + } +}); + +interface Capture { raw: string; headerKeys: string[] } + +function makeUpstream(sink: Capture[]) { + return http.createServer((req, res) => { + let data = ""; + req.on("data", (c) => (data += c)); + req.on("end", () => { + sink.push({ raw: data, headerKeys: Object.keys(req.headers).map((k) => k.toLowerCase()) }); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "chatcmpl-test", object: "chat.completion", created: 1, model: "model-test", + choices: [{ index: 0, message: { role: "assistant", content: "answer" }, finish_reason: "stop" }], + usage: { prompt_tokens: 100, completion_tokens: 4, total_tokens: 104 }, + })); + }); + }); +} + +async function makeHarness(tag: string) { + _resetStoreForTest(); + _resetPluginStateForTest(); + _resetSessionsForTest(); + const root = mkdtempSync(path.join(os.tmpdir(), `natid-${tag}-`)); + for (const k of ["HOME", "APPHOME", "HERMES_HOME", "XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME", "XDG_CONFIG_HOME"]) process.env[k] = root; + const store = new SessionStore({ dir: path.join(root, "sessions") }); + _setStoreForTest(store); + const captured: Capture[] = []; + const upstream = makeUpstream(captured); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const uaddr = upstream.address(); + assert(uaddr && typeof uaddr === "object"); + const upstreamUrl = `http://127.0.0.1:${uaddr.port}`; + const proxy = await startServer({ + port: 0, host: "127.0.0.1", upstream: upstreamUrl, + routes: { [upstreamUrl]: { models: { "model-test": { context: 64_000 } } } }, + modelContextLimit: 200_000, + kernelConfig: defaultConfig(200_000), + compress: { injectTool: false, injectNudge: false }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: true, debug: true, passthrough: false, autoUpdate: false, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions); + await once(proxy, "listening"); + const paddr = proxy.address(); + assert(paddr && typeof paddr === "object"); + return { root, store, captured, upstream, proxy, origin: `http://127.0.0.1:${paddr.port}` }; +} + +async function postChat(origin: string, headers: Record): Promise { + const body = JSON.stringify({ + model: "model-test", max_tokens: 64, stream: false, + messages: [ + { role: "system", content: "You are TestBuddy." }, + { role: "user", content: "hello" }, + ], + }); + const res = await fetch(origin + "/v1/chat/completions", { + method: "POST", + headers: { "content-type": "application/json", ...headers }, + body, + }); + assert.equal(res.status, 200); + await res.text(); +} + +test("rollout gate OFF (default): zero nidv output, zero wire diff", async () => { + const h = await makeHarness("off"); + const lines: string[] = []; + setLogCapture((_l, m) => lines.push(m)); + try { + await postChat(h.origin, { "x-acp-session": "s-bare" }); + await postChat(h.origin, { "x-acp-session": "s-wb", ...WB_HEADERS }); + assert.equal(h.captured.length, 2); + assert.equal(h.captured[1].raw, h.captured[0].raw, "wire bytes identical"); + assert.equal(lines.filter((l) => l.startsWith("nidv:")).length, 0, "feature off ⇒ no observation output"); + assert.equal(h.store.getSessionExt(`n:${digest8("conv-wb-1")}`, NIDV_EXT_KEY), undefined); + } finally { + setLogCapture(null); + await closeServers([h.proxy, h.upstream], "natid-off"); + } +}); + +test("rollout gate ON: byte-stable for known clients, WARN on compaction hit, ext record written", async () => { + const ENV = "BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION"; + const saved = process.env[ENV]; + process.env[ENV] = "1"; + const h = await makeHarness("on"); + const lines: string[] = []; + const warns: string[] = []; + setLogCapture((l, m) => { + lines.push(m); + if (l === "warn") warns.push(m); + }); + try { + await postChat(h.origin, { "x-acp-session": "s-bare" }); + await postChat(h.origin, { "x-acp-session": "s-wb", ...WB_HEADERS }); + assert.equal(h.captured.length, 2); + assert.equal(h.captured[1].raw, h.captured[0].raw, "ZERO diff at the upstream even for a known client"); + const nidv = lines.filter((l) => l.startsWith("nidv:")); + assert.equal(nidv.length, 1, "only the WorkBuddy request is classified; the bare one stays silent"); + assert.ok(nidv[0].includes("client=wb"), nidv[0]); + assert.ok(nidv[0].includes(`${digest8("conv-wb-1")}\u2026`), nidv[0]); + assert.ok(nidv[0].includes("evt=compaction(hdr=auto)"), nidv[0]); + assert.ok(warns.includes(nidv[0]), "compaction hit escalates the line to WARN"); + const rec = JSON.parse(String(h.store.getSessionExt(`n:${digest8("conv-wb-1")}`, NIDV_EXT_KEY))) as { v: number; client: string; reqs: number }; + assert.equal(rec.v, 1); + assert.equal(rec.client, "wb"); + assert.equal(rec.reqs, 1); + } finally { + setLogCapture(null); + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + await closeServers([h.proxy, h.upstream], "natid-on"); + } +}); + +test("decision independence: x-conversation-id alone is observed but NOT an identity source pre-#5526", async () => { + const ENV = "BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION"; + const saved = process.env[ENV]; + process.env[ENV] = "1"; + const h = await makeHarness("decis"); + const lines: string[] = []; + setLogCapture((_l, m) => lines.push(m)); + try { + await postChat(h.origin, { "x-conversation-id": "conv-only-1" }); + assert.ok(lines.some((l) => l.startsWith("nidv:") && l.includes(`${digest8("conv-only-1")}\u2026`)), "family was observed"); + assert.ok(lines.some((l) => /\[prefix-affinity\] new session pfa-/.test(l)), "identity resolved via prefix affinity, not via the observed header"); + } finally { + setLogCapture(null); + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + await closeServers([h.proxy, h.upstream], "natid-decis"); + } +}); + +test("latency smoke: 1MB codex-shaped body, extraction p99 stays cheap (CI-portable bound)", async () => { + const item = { id: "msg_lat", type: "message", role: "user", content: [{ type: "input_text", text: "x".repeat(200) }] }; + const itemCount = Math.ceil(1_000_000 / Buffer.byteLength(JSON.stringify(item))); + const bigBody = JSON.stringify({ + model: "gpt-x", + previous_response_id: "rs_big", + input: Array.from({ length: itemCount }, () => item), + }); + assert.ok(Buffer.byteLength(bigBody) >= 1_000_000); + const input: NativeIdInput = { + headers: { "x-codex-installation-id": "inst", "session-id": "sess-lat" }, + body: bigBody, + protocol: "responses", + }; + const samples: number[] = []; + for (let i = 0; i < 100; i += 1) { + const t0 = process.hrtime.bigint(); + extractNativeIdentity(input); + samples.push(Number(process.hrtime.bigint() - t0) / 1e6); + } + samples.sort((a, b) => a - b); + const p99 = samples[Math.min(samples.length - 1, Math.ceil(0.99 * samples.length) - 1)]; + console.log(`[nidv latency smoke] 1MB body p99=${p99.toFixed(3)}ms median=${samples[samples.length >> 1].toFixed(3)}ms`); + // Bound 20ms flaked on a Windows node22 runner (observed p99 27.3ms vs a + // typical-CI 3–4ms, one-off spike); 50ms absorbs that variance while staying + // far below any parse-class regression cost (full 1MB JSON.parse is ≫10× this). + assert.ok(p99 < 50, `p99 ${p99.toFixed(3)}ms exceeds the CI-portable bound (guards against accidental parse-class regressions; local dog-machine numbers land in the PR description)`); +}); diff --git a/tests/rollout.test.ts b/tests/rollout.test.ts index b96c96d187..78f5898622 100644 --- a/tests/rollout.test.ts +++ b/tests/rollout.test.ts @@ -207,7 +207,12 @@ test("rolloutDecisions: sorted, one per registered feature, real install id", () assert.equal(zeta.enabled, false); // percent-0 still flows through the bucketing tier (bucket < 0bp is // never true) — "off" is reserved for features with no percent at all. - for (const d of ds) assert.equal(d.source, "bucket", d.feature); + // Nested (`base:`) features resolve inside their base's cohort: base + // off ⇒ source "off" by design (NESTING in src/rollout.ts), not a bucket miss. + for (const d of ds) { + if (rollout.ROLLOUT_FEATURES[d.feature]?.base !== undefined) continue; + assert.equal(d.source, "bucket", d.feature); + } }); }); diff --git a/tests/storage-ext.test.ts b/tests/storage-ext.test.ts index bf044c3b7a..04b95f26e4 100644 --- a/tests/storage-ext.test.ts +++ b/tests/storage-ext.test.ts @@ -5,6 +5,7 @@ import * as os from "node:os"; import * as path from "node:path"; import { SessionStore } from "../src/persist.ts"; import { UnifiedStore } from "../src/storage/store.ts"; +import { STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; import type { Session } from "../src/session.ts"; /** @@ -174,7 +175,7 @@ test("v3 databases migrate to v10 in place, idempotently", async () => { const migrated = UnifiedStore.open(dbPath); try { const ver = (migrated as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10, "open() runs migrateSchema: v3 → v10"); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION, "open() runs migrateSchema to the current version"); migrated.setKv("probe", "alive", true); assert.deepEqual(migrated.getKv("probe", "alive"), true); assert.deepEqual(migrated.listSessionIds(), [], "existing v3 content untouched"); @@ -184,7 +185,7 @@ test("v3 databases migrate to v10 in place, idempotently", async () => { // Reopening a v10 database is a no-op (idempotent migration). const again = UnifiedStore.open(dbPath); const ver2 = (again as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 10); + assert.equal(ver2.user_version, STORAGE_SCHEMA_VERSION); again.close(); } finally { fs.rmSync(tmp, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); diff --git a/tests/storage-ingest-watermark.test.ts b/tests/storage-ingest-watermark.test.ts index 3e72b1f465..fb1acb6c68 100644 --- a/tests/storage-ingest-watermark.test.ts +++ b/tests/storage-ingest-watermark.test.ts @@ -25,6 +25,7 @@ import { UnifiedStore } from "../src/storage/store.ts"; import { UnifiedShadow } from "../src/storage/shadow.ts"; import { openSqliteDatabase } from "../src/storage/driver.ts"; import { planIngest, planIngestDelta, loadLegacyView, ingestSerializations, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; import type { CoreMessage } from "acp-kernel"; process.env.BILI_PERSIST_ZSTD = "0"; @@ -591,7 +592,7 @@ test("watermark: sampled verification still catches drift through the delta path } }); -test("watermark: v4 database migrates to v10 additively; newer user_version no-ops", async () => { +test("watermark: v4 database migrates to the schema head additively; newer user_version no-ops", async () => { const dir = tmpdir("bili-wm-migrate-"); try { const dbPath = path.join(dir, "index.db"); @@ -620,7 +621,7 @@ test("watermark: v4 database migrates to v10 additively; newer user_version no-o const store = UnifiedStore.open(dbPath); try { const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10, "open() migrates v4 → v10"); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION, "open() migrates v4 → schema head"); const row = store.getSessionRow("v4-sess"); assert.ok(row, "legacy session row survives the migration"); assert.equal(row!.lastIngestedSeq, null, "legacy rows read back with a NULL watermark (≡ full ingest)"); @@ -653,12 +654,12 @@ test("watermark: v4 database migrates to v10 additively; newer user_version no-o // probe version sits AT the current schema head so the no-op branch // still exercises after future bumps. const fwd = new DatabaseSync(dbPath); - fwd.exec("PRAGMA user_version = 10"); + fwd.exec(`PRAGMA user_version = ${STORAGE_SCHEMA_VERSION}`); fwd.close(); const older = UnifiedStore.open(dbPath); try { const ver2 = (older as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 10, "newer user_version is left untouched"); + assert.equal(ver2.user_version, STORAGE_SCHEMA_VERSION, "newer user_version is left untouched"); assert.ok(older.getSessionRow("v4-sess"), "rows stay readable"); } finally { older.close(); diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index 73f841cc8c..4b3af6c472 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -11,6 +11,7 @@ import { canonicalize, sha256 } from "./golden-canonical.ts"; import { planIngest, loadLegacyView, toLegacyLike, type LegacySessionLike } from "../src/storage/ingest.ts"; import { restampRefsToLedger } from "acp-kernel"; import { UnifiedStore } from "../src/storage/store.ts"; +import { STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; import { resolveStorageMode } from "../src/storage/shadow.ts"; import { resolveStorageWalJournalSizeLimit } from "../src/knobs.ts"; import { probeBuiltinSqlite, setPreferredSqliteEngineForTests, type SqliteEngineName } from "../src/storage/driver.ts"; @@ -801,7 +802,7 @@ test("schema v2: dev-window v1 database upgrades in place (identity columns adde assert.equal(row!.namespace, "proxy", "namespace backfills with its constant default"); assert.equal(row!.origin, null, "origin backfills NULL"); const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10, "schema version stamped to 10 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 orphans journal + v7 media map + v8 identity/tombstones + v9 zero-ref collection index + v10 immutable message ts column)"); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION, "schema stamped to the current version (v2 columns … v11 pf snapshot plane)"); // v3 backfill: every pre-existing sessions row claims its // (namespace, conversation_id) identity binding. const binding = (store as unknown as { lookupConversation(ns: string, id: string): string | null }).lookupConversation("proxy", "dev-window-1"); @@ -863,7 +864,7 @@ test("schema v10: v9→v10 migration adds messages.ts without backfill; idempote try { const ver = (reopened as unknown as { db: { prepare(q: string): { get(): unknown } } }).db .prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 10, "v9 library migrates to v10"); + assert.equal(ver.user_version, STORAGE_SCHEMA_VERSION, "v9 library migrates to the current version"); const rows = reopened.getMessages("v10-mig-1") as { seq: number; ts?: number | null }[]; assert.equal(rows.length, 1, "row survives the migration"); assert.ok(rows[0].ts === null || rows[0].ts === undefined, "pre-existing row keeps NULL (never backfilled)"); diff --git a/tests/text-safe.test.ts b/tests/text-safe.test.ts index 7d729446c9..56abff9464 100644 --- a/tests/text-safe.test.ts +++ b/tests/text-safe.test.ts @@ -176,6 +176,7 @@ test("family gate: no raw negative string slices outside the allowlist", () => { ["src/conflict-watch.ts", [/events\.slice\(-10\)/]], // array receiver ["src/web/client.ts", [/conflicts\.slice\(-10\)/]], // array receiver (#2102 session-detail card) ["src/web/report.ts", [/lines\.slice\(-12\)/, /folds\.slice\(-15\)/]], // array receiver (#3099 session-report tables) + ["src/server/native-identity.ts", [/events\.slice\(-8\)/, /events\.slice\(-4\)/]], // array receiver (#5576 M1 observation ring buffer) ["src/text-safe.ts", [/\.slice\(/]], // the clamped implementation itself ]); const srcDir = path.join(import.meta.dirname, "..", "src"); diff --git a/website/config-reference/env-only.yaml b/website/config-reference/env-only.yaml index 635c399999..dcc13cca53 100644 --- a/website/config-reference/env-only.yaml +++ b/website/config-reference/env-only.yaml @@ -530,3 +530,19 @@ entries: group: envonly desc_en: "Longest-edge trigger threshold (px) for the egress image downscale pass (#3400); non-numeric or below 128 ignored (default applies). Env-only by design." desc_zh: "出站图片降采样 pass 的最长边触发阈值(像素,#3400);非数字或低于 128 的取值按默认处理。设计上仅环境变量。" + - key: BILI_NATIVE_IDENTITY_PROFILES_JSON + type: string + default: "unset (built-in table)" + kind: env-only + audience: user + group: envonly + desc_en: "Native-identity observation (#5573 M1): JSON object of per-client profile rows merged over the built-in evidence table (src/server/native-identity-profiles.ts) — entries ADD new rows or REPLACE a built-in row by name, never remove one (the built-ins are the floor). Same shape as a built-in row (name, matchHeaders, sessionHeaders, msgidHeaders, compactionEdgeHeader, countPatterns, compactionPrefixes, ...). Malformed JSON fails open silently (built-ins only). Only meaningful while rollout.features.nativeIdentityObservation is enabled; feeds no decision." + desc_zh: "原生身份观测(#5573 M1):JSON 对象,按内置证据表(src/server/native-identity-profiles.ts)同形提供客户端画像行——按名**新增**或**整体替换**内置行,绝不删除(内置为底)。字段形如内置行(name、matchHeaders、sessionHeaders、msgidHeaders、compactionEdgeHeader、countPatterns、compactionPrefixes……)。畸形 JSON 静默失效(只用内置表)。仅在 rollout.features.nativeIdentityObservation 开启时有意义;不喂任何决策。" + - key: BILI_PF_V6_REWRITE_STRICT + type: boolean + default: "unset (off)" + kind: env-only + audience: user + group: envonly + desc_en: "Set `1` to CREDIT watermark-v6 misses toward the five-miss breaker instead of the default SELF-HEALING uncredited legacy turn, per lane: in the MSGID lane (`watermarkV6MsgId`) compaction-class shrinks against the stored frontier (`v6m-compact`); in the CANONICAL lane (`watermarkCanonicalV6`) host-side content rewrites (chain breaks: chain-not-started / seam-too-short) \u2014 for exotic hosts whose thrash would otherwise ride free legacy turns forever. (The msgid lane\u2019s `v6m-window-system` verdict is ALWAYS credited, independent of this switch.) Default unset/off; pinned by tests/fastpath-pf-msgid.test.ts and tests/fastpath-v6-canonical.test.ts." + desc_zh: "设 `1` 把水位 v6 失配**计名**进五次失配熔断器(替代默认的自愈型不计名 legacy 回合),按车道区分:msgid 车道(`watermarkV6MsgId`)计名对存储前沿的压缩类收缩(`v6m-compact`);规范车道(`watermarkCanonicalV6`)计名宿主侧内容改写(链断裂:chain-not-started / seam-too-short)——专为疯狂抖动的怪异宿主准备(否则它们将永远免费搭乘自愈回合)。msgid 车道的 `v6m-window-system`(窗口头部 system)恒计名,不受此开关影响。默认未设置/关闭;由 tests/fastpath-pf-msgid.test.ts 与 tests/fastpath-v6-canonical.test.ts 钉死。" diff --git a/website/config-reference/rollout.yaml b/website/config-reference/rollout.yaml index b01da282e7..bca518b832 100644 --- a/website/config-reference/rollout.yaml +++ b/website/config-reference/rollout.yaml @@ -65,3 +65,27 @@ entries: group: behavior desc_en: "Full-resend watermark fastpath (#5403/#5418): plugin-lane chat-completions turns whose previous response carried an x-bili-pf-seq stamp assert the resent prefix via the echoed count (structural scan, no full parse); on a hit the pipeline consumes stashed canonical prefix cores plus only the new tail window (byte-identical outbound provenance; ANY doubt falls back to the untouched full-parse path per turn and re-stamps, five credited misses hold the lane for one hour). Requires the client to echo the stamp back as x-bili-pf-count (dsh/pi/opencode/omp do; unadapted lanes never hit). EXPERIMENTAL grey rollout — ON by default for roughly 5% of installs (owner ruling 2026-11-02: small cohort before GA; one-time intro notice in the web UI for every install, tri-state switch in quick config). Force it with `true`, turn off with `false` (or a percent) / env = 0. See TECHNICAL-NOTES.md “Full-resend watermark fastpath”." desc_zh: "全量重发水位快路径(#5403/#5418):插件车道 chat-completions 回合若上一轮响应盖有 x-bili-pf-seq,则用回声计数声明重发前缀(结构扫描,不做全量解析);命中时管线只消费钉存的前缀核 + 新增尾巴窗口(出站字节同源可证;任何疑点当轮回退未动过的全量解析路径并重新盖戳,连续五次计名失配将车道按住一小时)。需客户端把戳回声为 x-bili-pf-count(适配的车道回带;其余车道不会命中)。**实验性灰度**——默认仅约 5% 的安装开启(owner 裁定 2026-11-02:先小流量再 GA;web 面板对全部安装显示一次性公告、快速配置提供三态开关);`true` 强制开、`false`(或百分比)/ env = 0 关。见 TECHNICAL-NOTES.zh-CN.md「全量重发水位快路径」。" + - key: rollout.features.watermarkV6MsgId + type: "boolean | number" + default: "0 (off)" + default_zh: "0(关)" + env: BILI_ROLLOUT_WATERMARK_V6_MSG_ID + group: behavior + desc_en: "Watermark v6 msgid lane (#5576/#5600): nested grey inside pluginFullResendWatermark, dispatched BEFORE the canonical lane (which then runs only as its else-if). Identity is server-minted: each session world gets a pf_sid (pf_+12hex) issued via the x-bili-pf-session RESPONSE header (the response IS the issuance channel); resolution = echo (must name a live partition) -> probe real tail-ids against the global ledger index (budget 8) -> mint (v6m-arm cold start, P5's one legal full). Hit authority = ledger-width arithmetic against OUR frontier (K = resident element count, never a client echo) + a window-origin pin on element K-1 of our own pf_rows partition (hard pin; weak pin on id-silent frontiers); content bytes are never judged - in-place edits ride frozen cores. Hits append window rows + geom snapshots to the durable pf_* partition store (LSM meta plane: WAL + checksummed checkpoints folding every 32 entries; post-boot hydration from L1+L0; rows deleted ONLY by deleteSession - archive-and-reset doctrine). Shrink vs frontier = compaction class (uncredited passthrough by default, credited under BILI_PF_V6_REWRITE_STRICT=1); system-at-window-head is always credited; sess-change stays uncredited; model switches are non-events (no model gate); all credits feed the shared five-miss/60-min breaker above every lane. Soft plane: store absent => pre-existing lanes run byte-for-byte. Zero retention: hits pin no seed stash. Default 0% - measure in a small cohort first. See TECHNICAL-NOTES.md \u201cFull-resend watermark fastpath\u201d (§v6 msgid lane)." + desc_zh: "水位快路径 v6 msgid 车道(#5576/#5600):嵌套在 pluginFullResendWatermark 内的灰度,调度先于规范车道(后者只作为它的 else-if 运行)。身份由服务端铸造:每个会话世界获得一个 pf_sid(pf_+12 hex),经响应头 x-bili-pf-session 签发(响应本身即签发通道);解析顺序=回声(必须指向存活分区)→用真实尾 id 探测全局账本索引(预算 8)→铸造(v6m-arm 冷启动,P5 的唯一一次合法全量)。命中判据=对着我方前沿的账本宽度算术(K=驻留元素数,绝不取客户端回声)+我方 pf_rows 分区元素 K-1 上的窗口起点钉(硬钉;无 id 前沿弱钉);全程不判内容字节——就地改写乘冻结核。命中把窗口行+geom 快照追加入持久 pf_* 分区存储(LSM meta 面:WAL+带校验和检查点每 32 条折叠;重启后从 L1+L0 水合;行只被 deleteSession 删除——归档-重置教径)。对前沿收缩=压缩类(默认不计名透传,BILI_PF_V6_REWRITE_STRICT=1 时计名);窗口头部 system 恒计名;sess-change 不计名;换模型非事件(无模型门);一切计名汇入压在全部车道之上的五次失配/60 分钟共享熔断器。软平面:存储缺失 ⇒ 既有车道逐字节照跑。零留存:命中不钉种子堆。默认 0%——先小流量测数据。见 TECHNICAL-NOTES.zh-CN.md「全量重发水位快路径」(v6 msgid 节)。" + - key: rollout.features.watermarkCanonicalV6 + type: "boolean | number" + default: "0 (off)" + default_zh: "0(关)" + env: BILI_ROLLOUT_WATERMARK_CANONICAL_V6 + group: behavior + desc_en: "Watermark v6 canonical lane (#5576): nested grey inside pluginFullResendWatermark. When enabled, the watermark's hit verdict stops trusting client-asserted bytes: surviving pre-filters are pure server-side arithmetic over the client-declared ledger list (ahead / width-shape / no-progress / sess-change), then the body converts once with the REAL codec and anchors against stored unified-transcript rows via the canonical chain walk (kernel id fast path → content-sig fallback, min seam run 2). On a hit the outbound prefix = stored rows up to the seam (folded regions behind synthetic summary anchors) + this turn's fresh tail cores; system text re-decodes from the CURRENT head every turn (passthrough by construction — the v5 head-drift gate is retired); model switches are non-events (one logical ledger shared across models); host-side content rewrites break the chain and self-heal through UNcredited legacy turns unless BILI_PF_V6_REWRITE_STRICT=1 credits them toward the breaker. Zero-retention doctrine: canonical outcomes pin NO seed stash. Default 0% — measure in a small cohort before any promotion. See TECHNICAL-NOTES.md “Full-resend watermark fastpath → v6 canonical lane”." + desc_zh: "水位快路径 v6 规范车道(#5576):嵌套在 pluginFullResendWatermark 内的灰度。开启后,水位命中的判定不再信任客户端声明的字节:幸存的预过滤是纯服务端算术(回声戳超前/列表宽度/零增量/会话身份换绑),随后 body 用真编解码器转换一次并沿规范行链与已存统一转录稿行对齐(内核 id 快路→内容签名回退,最小接缝跑长 2)。命中时出站前缀 = 缝点前的存储行(折叠区以合成摘要锚点渲染)+ 本轮新尾核;system 文本每轮从当前头部重新解码(构造即透传,v5 head-drift 门退役);换模型是非事件(跨模型共享一个逻辑账本);宿主侧内容改写断链后走不计名 legacy 回合自愈,除非 BILI_PF_V6_REWRITE_STRICT=1 将其计名进入熔断器。零留存教条:规范车道结果绝不钉存种子堆。默认 0%——先小流量测数据再谈晋升。见 TECHNICAL-NOTES.zh-CN.md「全量重发水位快路径→v6 规范车道」。" + - key: rollout.features.nativeIdentityObservation + type: "boolean | number" + default: "0 (off)" + default_zh: "0(关)" + env: BILI_ROLLOUT_NATIVE_IDENTITY_OBSERVATION + group: behavior + desc_en: "Native-identity observation (#5573 M1): pure-observation collection of client-native session/message identity evidence via a per-client profile table (WorkBuddy x-conversation-id family + X-Compaction headers, codex installation-id / item-id families / previous_response_id rotation + compaction prefaces, Claude Code session header + metadata dual-channel cross-check + continuation preface, opencode ses_ affinity + fork marker + wire-shape assertions, Gemini state-snapshot preface) into the session ext store under nidv:obs (sha1-8 digests only — id values never persist or log in plaintext) plus one debug log line per observed request; compaction hits escalate the line to WARN. Feeds NO decision: request handling, folding, parsing and outbound bytes are untouched, and off-or-unclassified traffic produces zero diff against the golden suites. Default 0% — enable to collect the field data the L2 consumers need (silent fold re-baseline for #5488, fastpath identity binding for #5403/#5418)." + desc_zh: "原生身份观测(#5573 M1):纯观测采集客户端原生的会话/消息身份证据——按客户端画像表(WorkBuddy x-conversation-id 族 + X-Compaction 头、codex installation-id/item id 族/previous_response_id 轮换 + 压缩前缀文案、Claude Code 会话头 + metadata 双通道交叉核对 + 续接前缀文案、opencode ses_ 亲和键 + fork 标记 + 线上形状断言、Gemini state-snapshot 前缀)——写入会话扩展存储 nidv:obs(只存 sha1-8 摘要,id 值绝不以明文落盘或入日志)外加每请求一行 debug 日志;压缩事件把该行升级为 WARN。不喂任何决策:请求处理、折叠、解析与出站字节一律不动,关闭或未识别流量相对 golden 套件零 diff。默认 0%——开启用于收集 L2 消费方所需的现场数据(折叠静默重定基线 #5488、快路径身份绑定 #5403/#5418)。"