From aa9cb2e68ea6a82746d43b9d8995235aab23ccde Mon Sep 17 00:00:00 2001 From: awork Date: Sat, 17 Oct 2026 11:24:15 +0800 Subject: [PATCH 1/3] =?UTF-8?q?feat:=20ingest=20host=20message=20ids=20int?= =?UTF-8?q?o=20the=20canonical=20store=20=E2=80=94=20reconcile=20becomes?= =?UTF-8?q?=20exact=20set=20algebra=20(#3464=20R1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plugin lanes (dsh/opencode) passively read each wire message's own host id off the outgoing body and stamp it in the reserved top-level field _bili_message_ids when the proxy advertises capabilities.hostMessageIds. The proxy extracts+strips the field at the body-parse chokepoint, zips the wire-aligned list onto cores via wireIndex sidecars, and persists it: - schema v6: messages gains host_id/host_slot + partial unique index (session_id, host_id, host_slot); migration additive/probe-guarded - planIngest dedups stamped legs by host key — one row per host message across drifted resends (fresher bytes win, first-seen position kept); unstamped lanes keep today's byte-keyed rows bit-for-bit - store reuses the host-keyed seq across byte drift and refreshes the row in place (documented append-only exception for #2960(5) digest parity) - fold-reconcile Pass -1 claims missing covered ids by exact host-id equality (role-gated, k-th within multi-core groups) ahead of every byte-based pass; anchor.h seeds/backfills across passes; unstamped lanes are bit-identical - fork snapshots expose an additive hostIds array mapping each ordered message back to its bili ref (outside every hash) No new config/env surface. No kernel changes. --- MESSAGE-IDENTITY.md | 54 ++- MESSAGE-IDENTITY.zh-CN.md | 16 +- PLUGIN.md | 6 +- TECHNICAL-NOTES.md | 35 ++ TECHNICAL-NOTES.zh-CN.md | 10 + src/agent/dsh-native.ts | 39 ++- src/agent/fork-adopt.ts | 4 + src/agent/opencode-native.ts | 30 +- src/agent/shared.ts | 21 ++ src/fold-reconcile.ts | 116 ++++++- src/host-id-channel.ts | 150 ++++++++ src/persist.ts | 36 +- src/plugin.ts | 50 ++- src/server.ts | 4 + src/server/handle.ts | 30 +- src/server/prepare-anthropic.ts | 13 +- src/session.ts | 15 +- src/storage/ingest.ts | 72 +++- src/storage/schema.ts | 35 +- src/storage/shadow.ts | 11 +- src/storage/store.ts | 78 ++++- tests/fold-reconcile.test.ts | 2 +- tests/issue3464-host-ids.test.ts | 456 +++++++++++++++++++++++++ tests/storage-ext.test.ts | 6 +- tests/storage-ingest-watermark.test.ts | 8 +- tests/storage-unified.test.ts | 2 +- 26 files changed, 1241 insertions(+), 58 deletions(-) create mode 100644 src/host-id-channel.ts create mode 100644 tests/issue3464-host-ids.test.ts diff --git a/MESSAGE-IDENTITY.md b/MESSAGE-IDENTITY.md index 6807dcdf2a..19aca804d5 100644 --- a/MESSAGE-IDENTITY.md +++ b/MESSAGE-IDENTITY.md @@ -48,8 +48,12 @@ host: | pi (chat, MITM/plugin) | yes (`convertToLlm()` rebuilds arrays; message model has no id field) | no channel | | omp | yes (pi-family plugin) | no channel | | hermes plugin | yes (own store, `pre_api_request` hook) | no channel | +| dsh | yes | **host-model id via side-channel** — `MessageBase.id` is stable across representation boundaries; the plugin reads it off the outgoing body and stamps `_bili_message_ids` when the host serializer includes it (#3464) | +| opencode | yes | **host-model id via side-channel** — `msg_…` ids, same mechanism (#3464) | -Coverage must be measured at the **re-join point**, not at ingress. +Coverage must be measured at the **re-join point**, not at ingress. The two +side-channel lanes above are the R1 adoption: the host's own storage id is +the join key, and bili never mints or rewrites it. ## Lane survey: where could an id even live? @@ -243,15 +247,49 @@ byte-identical. Identity bookkeeping is never involved. | Marker-as-identity (tags persisted by hosts) | The tag-echo incidents show the channel corrupts what it carries; current design already strips it at both ends. | | Implicit aliasing of host-rewritten tool-call ids | Pairing a missing covered id with an inbound twin that has identical content under a different toolCallId requires host knowledge; without it the claim is a guess and misattributes on decompress (#2396: Prime rewrote composite ids across a provider switch). Detection-only shipped; the fix belongs at the host boundary (PLUGIN.md obligation 5). #2480 update: Pass 0 now supplies that host knowledge (position + stored canonical fingerprint), so this class is claimed rather than only counted; PLUGIN.md obligation 5 stands for hosts that can canonicalize earlier. | -## Future direction - -If a future host or lane offers a genuinely stable client-side message id, -it may be adopted **only as an additional join hint** — never as a replacement -for the content-hash base. Until then: bytes are the anchor, hashes are the -join, mNNNNN is the ledger, tags are the view. +## Host-stable ids: the R1 channel (#3464, supersedes the old "future direction") + +The old stance here — a host-stable id could at best be an *additional join +hint*, never a replacement for the content-hash base — is superseded by the +owner rulings on #2938 (hash demoted to second-class citizen; order-centric, +never fuzzy) and #3464 (R1 of the #2960 identity pipeline). Lanes whose host +model carries a stable per-message id (dsh `MessageBase.id`, opencode +`msg_…`) now ride it into bili through a **side-channel top-level body field** +(`_bili_message_ids`, index-aligned with `messages`; the wire protocols +themselves still offer no id field — the survey above stands). On stamped +lanes identity becomes exact set algebra instead of byte-hash guessing: + +- **Storage** (`src/storage/schema.ts` v6): `messages.host_id`/`host_slot` + + partial unique index `(session_id, host_id, host_slot)`. A row's key is the + host id; the byte hash becomes the *content* column. Resending the same + host message under drifted bytes refreshes the row in place (documented + append-only exception, required so the DB view equals the in-memory tail + byte-for-byte — #2960⑤ digest parity) instead of minting a twin row. Seq + assignment prefers host-key continuity, so byte drift no longer churns the + watermark. +- **Reconcile** (`src/fold-reconcile.ts`): Pass -1, before every heuristic + pass — a covered id whose stored anchor carries a host id claims the unique + inbound core with the same host id (k-th pairing within multi-core groups, + role-gated). Heuristic passes (position/tool-call-id/twin/simhash/order + bracket) remain the fallback for unstamped lanes and for cells whose anchor + predates stamping. Unstamped lanes are bit-identical to before. +- **Fork** (`src/plugin.ts` fork snapshot): the response gains an additive, + index-aligned `hostIds` array — the host-id → bili-ref map — outside every + hash by construction, so branch-point comparison stays byte-based for old + and new clients alike. + +Adoption is per-lane and self-degrading: the plugin reads whatever ids the +host serializer actually puts in the outgoing body (passive read, no host-API +mutation), stamps only when the proxy advertises +`capabilities.hostMessageIds`, and degrades to the byte-id path whenever the +field is absent or malformed. Lanes whose host model has no message-level id +(pi, omp — see survey) stay on bytes until their hosts add one. Bytes remain +the universal fallback; host ids are the primary key where they exist. Related: #1496 (this document's prompt), `SESSION-IDENTITY.md` (session granularity), #1476 + kernel #459/#463 (echo discrimination), #242/#1475 (Responses id constraints), #206/#673 (tag echo), #1479/#1482 (strict-echo repair), #1039 (tool-call byte-exactness invariant), #2396 (host-rewritten tool-call ids across a provider switch), #2480 (positional fold coverage; -`tests/identity-proof.test.ts` is its proof matrix). +`tests/identity-proof.test.ts` is its proof matrix), #2938 (host message-id +side-channel proposal; owner ruling: order-centric, hash second-class), +#3464 (R1 implementation of the host-id channel). diff --git a/MESSAGE-IDENTITY.zh-CN.md b/MESSAGE-IDENTITY.zh-CN.md index 2d0ca0bfaf..cb429c4a5d 100644 --- a/MESSAGE-IDENTITY.zh-CN.md +++ b/MESSAGE-IDENTITY.zh-CN.md @@ -26,8 +26,10 @@ sha256;`assignRefs` 再 first-wins 挂 ref(`if (map.byRaw[message.id]) continue` | pi(chat,MITM/plugin) | 是(`convertToLlm()` 每次重建数组;消息模型无 id 字段) | 无通道 | | omp | 是(pi 系插件) | 无通道 | | hermes 插件 | 是(自有存储 + `pre_api_request` 钩子) | 无通道 | +| dsh | 是 | **宿主模型 id 走旁路字段**——`MessageBase.id` 跨表示边界稳定;插件从出站 body 读 id,宿主序列化器带上它就盖 `_bili_message_ids`(#3464) | +| opencode | 是 | **宿主模型 id 走旁路字段**——`msg_…` id,同一机制(#3464) | -覆盖率必须在 **re-join 时点**衡量,而不是 ingress 时点。 +覆盖率必须在 **re-join 时点**衡量,而不是 ingress 时点。上表两条旁路 lane 即 R1 的采用:宿主自己存储里的 id 就是 join key,bili 永不铸造或改写它。 ## Lane 普查:id 能放在哪? @@ -101,8 +103,14 @@ DeepSeek 类网关要求客户端传回网关自己发出的 reasoning("the reas | 标记当身份(标签由宿主持久化) | 标签回声事故证明该通道会污染所载内容;现行设计已在两端剥离。 | | 隐式别名化宿主改写的工具调用 id | 把一个缺失 covered id 与"同内容、不同 toolCallId"的入站孪生配对需要宿主知识;没有它,claim 就是猜测,decompress 时认错(#2396:Prime 跨 provider 切换改写了复合 id)。只交付检测;修复属于宿主边界(PLUGIN.md 义务 5)。#2480 更新:Pass 0 现在提供了那份宿主知识(位置 + 存储的规范指纹),这一类从"只计数"变为"可认领";能在更早边界规范化的宿主,PLUGIN.md 义务 5 依然成立。 | -## 未来方向 +## 宿主稳定 id:R1 通道(#3464,取代旧的「未来方向」) -若未来宿主或 lane 提供真正稳定的客户端消息 id,只允许作为**附加 join hint** 采纳——永不替代 content-hash 基座。在那之前:字节是锚,hash 是 join,mNNNNN 是账本,标签是视图。 +本文档旧立场——宿主稳定 id 至多算**附加 join hint**、永不替代 content-hash 基座——已被 #2938 的 owner 裁定(hash 降为二等公民;顺序为中心、绝不模糊)与 #3464(#2960 身份管线 R1)取代。宿主模型带稳定消息 id 的 lane(dsh `MessageBase.id`、opencode `msg_…`)现在通过**顶层 body 旁路字段**(`_bili_message_ids`,与 `messages` 下标对齐;wire 协议本身仍无 id 字段——上表普查结论不变)把 id 送进 bili。盖了章的 lane 上,身份从「猜内容像不像」变成精确集合代数: -相关:#1496(本文档的起因)、`SESSION-IDENTITY.zh-CN.md`(会话粒度)、#1476 + kernel #459/#463(回声判别)、#242/#1475(Responses id 约束)、#206/#673(标签回声)、#1479/#1482(strict-echo 修复)、#1039(工具调用字节不变量)、#2396(跨 provider 切换时宿主改写的工具调用 id)、#2480(位置折叠覆盖;`tests/identity-proof.test.ts` 是其证明矩阵)、#2636(切模型形状 churn:assistant-turn key + 图片占位变体 pass)。 +- **存储**(`src/storage/schema.ts` v6):`messages.host_id`/`host_slot` + 部分唯一索引 `(session_id, host_id, host_slot)`。行的键是宿主 id,字节 hash 降为内容列。同一条宿主消息以漂移后的字节重发时,**原地刷新该行**(有记录的 append-only 例外,保证 DB 视图与内存尾部逐字节相等——#2960⑤ digest parity),而不是再落一行孪生行。seq 分配优先沿用宿主键连续性,字节漂移不再搅动水位。 +- **对账**(`src/fold-reconcile.ts`):Pass -1,排在一切启发式 pass 之前——锚点带宿主 id 的 covered id 直接认领同宿主 id 的唯一入站 core(多 core 组内 k-th 配对,role 门)。启发式 pass(位置/toolCallId/孪生/simhash/顺序括号)保留给未盖章 lane 与盖章前留下的锚点。未盖章 lane 与此前逐位一致。 +- **fork**(`src/plugin.ts` fork 快照):响应新增加性的 `hostIds` 数组(与 orderedMessages 下标对齐)——宿主 id → bili ref 映射,构造上不进任何 hash,新旧客户端的分支点比较都保持基于字节。 + +采用按 lane 渐进、自动降级:插件被动读取宿主序列化器实际写进出站 body 的 id(不改宿主 API),仅在 proxy 广播 `capabilities.hostMessageIds` 时盖章,字段缺失或畸形一律退回字节 id 路径。宿主模型没有消息级 id 的 lane(pi、omp——见普查)在其宿主补齐前保持字节身份。字节仍是通用兜底;宿主 id 在有处即为主键。 + +相关:#1496(本文档的起因)、`SESSION-IDENTITY.zh-CN.md`(会话粒度)、#1476 + kernel #459/#463(回声判别)、#242/#1475(Responses id 约束)、#206/#673(标签回声)、#1479/#1482(strict-echo 修复)、#1039(工具调用字节不变量)、#2396(跨 provider 切换时宿主改写的工具调用 id)、#2480(位置折叠覆盖;`tests/identity-proof.test.ts` 是其证明矩阵)、#2636(切模型形状 churn:assistant-turn key + 图片占位变体 pass)、#2938(宿主消息 id 旁路通道提案;owner 裁定:顺序为中心、hash 二等公民)、#3464(R1 宿主 id 通道实现)。 diff --git a/PLUGIN.md b/PLUGIN.md index 2fadda12d1..c9e3440089 100644 --- a/PLUGIN.md +++ b/PLUGIN.md @@ -48,11 +48,14 @@ Fetch once at plugin startup. "toolEndpoint": "/__bili/plugin/tool", "statusEndpoint": "/__bili/plugin/status", "capabilities": { - "fork": { "protocolVersion": 2, "acceptsProtocolVersions": [1, 2], "endpoint": "/__bili/plugin/fork", "snapshotEndpoint": "/__bili/plugin/snapshot" } + "fork": { "protocolVersion": 2, "acceptsProtocolVersions": [1, 2], "endpoint": "/__bili/plugin/fork", "snapshotEndpoint": "/__bili/plugin/snapshot" }, + "hostMessageIds": { "bodyField": "_bili_message_ids", "maxEntries": 10000 } } } ``` +`capabilities.hostMessageIds` (#3464): the proxy accepts a **side-channel top-level body field** carrying your conversation's stable per-message ids. On model requests whose body is the Anthropic chat shape (`messages` + `model`), you may add `"_bili_message_ids": [, …]`, index-aligned with `messages` (use `null` for messages that have no id). The proxy strips the field before forwarding — it never reaches the upstream provider — and uses it as the PRIMARY message identity for that session: storage rows key on the host id (a resend under drifted bytes refreshes the row instead of minting a twin), fold reconciliation claims by exact host-id equality before any heuristic pass, and the fork snapshot answers with the same array so you can map host ids back to bili refs. Omit the field whenever your host has no message-level id (or its serializer does not put one in the body) — identity then falls back to content bytes exactly as before; a malformed array is stripped and ignored for that request only. Never stamp this field on requests carrying bili-internal marks (passthrough / plugin-bypass / compaction) or on incremental-turn marker'd bodies — those paths own their bytes. + Register the tools natively with your agent, in whichever wire format your agent speaks — register exactly what `toolNames` lists. `acp_rule` is advertised only when the proxy's config enables it (`compress.rules: true`, opt-in). Calling a tool that is known but disabled answers `200` with an explanatory `result` instead of an error (#1192). If `protocolVersion` is higher than you know, still register the tools — extra fields in schemas are ignored by agents. ### 2. Request headers @@ -220,6 +223,7 @@ Marker divert (#3248 phase 2 — native extensions): instead of switching the pr 3. Forward tool executions verbatim; return `result` as the tool result. 4. Self-disable when not running behind bili (e.g. the agent's baseURL does not point at the proxy) — same convention as billion-context-pi / opencode-acp extensions. 5. Keep tool-call ids byte-stable within one conversation, including when projecting it onto another provider. Tool-message identity keys on `toolCallId` as the protocol-stable pairing factor; a projection that rewrites those ids (e.g. sanitizing stored composite ids for a foreign Responses provider) silently loses fold coverage — compressed tool outputs re-enter the wire unfolded (#2396). Canonicalize ids at the host boundary before the payload reaches the proxy; the proxy detects the shape and names it in the fold-drift log, but never repairs onto guessed pairings (see MESSAGE-IDENTITY.md). +6. If your host model carries a stable per-message id AND its request serializer writes it into the outgoing body, stamp `capabilities.hostMessageIds.bodyField` (index-aligned with `messages`, `null` where absent) on every full-resend Anthropic-shaped model request — read passively off the body you already parse; do not mutate host state to force ids into existence. Gate on the manifest capability (older proxies have no such capability entry and must never see the field), skip incremental-turn marker'd bodies, and keep each id stable for the life of the message (the proxy treats one host id as exactly one message forever, mirroring the never-reused ref contract). Hosts without message-level ids (pi, omp today) simply omit the field (#3464). ## Reference diff --git a/TECHNICAL-NOTES.md b/TECHNICAL-NOTES.md index 7bace5a754..30c94852eb 100644 --- a/TECHNICAL-NOTES.md +++ b/TECHNICAL-NOTES.md @@ -266,6 +266,41 @@ its mode (`pm`) and prepare-responses logs one transition line with the reason are recorded — never the prompt body. No wire behavior changed: both projections stay exactly as before; the switch is now *recorded* instead of misattributed. +## Host message ids: exact set algebra on stamped lanes (#3464, R1 of #2960) + +Message identity was derived from serialized bytes in every layer; any byte +drift (protocol/model switch, provider rewrite, fork, compaction, marker strip) +collapsed session/message/fold-anchor identity together — the #2915/#2917/#2838 +family. R1 gives lanes whose host model has a stable per-message id (dsh +`MessageBase.id`, opencode `msg_…`) an exact identity channel: + +- **Transport**: top-level body field `_bili_message_ids` (index-aligned with + `messages`, `null` padding), stripped at the single body-parse chokepoint in + `src/server/handle.ts` before ANY forward path — reserved namespace never + reaches upstream. Plugins read ids passively off the outgoing body (no host + API mutation), gate on manifest `capabilities.hostMessageIds`, and stamp only + Anthropic-shaped full-resend requests (`src/agent/dsh-native.ts`, + `src/agent/opencode-native.ts`). The wire protocols themselves have no id + field (MESSAGE-IDENTITY.md lane survey) — this is a side channel, not a + protocol change. +- **Ingress**: `prepareAnthropic` threads the list to reconcile and the + snapshot; storage keys rows on `(session_id, host_id, host_slot)` (schema + v6, partial unique index). A resend under drifted bytes refreshes the row + IN PLACE (documented append-only exception — the DB view must equal the + in-memory tail byte-for-byte for #2960⑤ digest parity) and reuses the seq + (host-key continuity: drift no longer mints new seqs or churns the ingest + watermark). No host id → today's `(msg_id, content_hash)` behavior, + bit-identical. +- **Reconcile**: Pass -1 in `planReconciliation` claims by exact host-id + equality (k-th within multi-core groups, role-gated) before every heuristic + pass; anchors persist `h` across passes. Fork snapshots answer with the + index-aligned `hostIds` array (outside every hash) so host ids map back to + bili refs. + +Lanes without message-level ids (pi, omp, claude-code, codex, kimi, zcode) +stay on byte identity unchanged — adoption is per-lane and self-degrading; +malformed or absent fields never make a request worse than unstamped. + ## Responses native chaining is not supported — history must be replayed (#1954) The Responses-wire adapter (`prepareResponses` in `src/server.ts`, `buildRequest` diff --git a/TECHNICAL-NOTES.zh-CN.md b/TECHNICAL-NOTES.zh-CN.md index bf55388326..6eb4a117dd 100644 --- a/TECHNICAL-NOTES.zh-CN.md +++ b/TECHNICAL-NOTES.zh-CN.md @@ -118,3 +118,13 @@ bili 永不拥有用户数据:每个被启动的客户端都跑在**真实 home* 4. **用户发送的消息永不被改写。** 用户自己片段的逐字回声在武装清洗下也保留(#463 豁免,按请求文本匹配,不按形状)。 出口审计网(#2248)保持仅记日志:裸出口(未识别帧、解析失败)原样转发并打 `[tag-echo] raw exit` 日志,新泄漏形状可见而不改线上行为。由 `tests/unified-acp-invariants.test.ts` 钉住(8 个测试,两条 lane)。 + +## 宿主消息 id:盖章 lane 上的精确集合代数(#3464,#2960 R1) + +消息身份原先在每一层都从序列化字节推导;任何字节漂移(换协议/换模型、provider 改写、fork、压缩、marker 剥离)会让会话/消息/折叠锚点三层身份一起塌——#2915/#2917/#2838 那一族。R1 给宿主模型带稳定消息 id 的 lane(dsh `MessageBase.id`、opencode `msg_…`)一条精确身份通道: + +- **传输**:顶层 body 字段 `_bili_message_ids`(与 `messages` 下标对齐,`null` 占位),在 `src/server/handle.ts` 唯一的 body 解析咽喉处剥离——保留命名空间永不出站。插件被动读取出站 body 里的 id(不改宿主 API),按 manifest `capabilities.hostMessageIds` 门控,只盖 Anthropic 形状的全量重发请求(`src/agent/dsh-native.ts`、`src/agent/opencode-native.ts`)。wire 协议本身没有 id 字段(MESSAGE-IDENTITY.md 普查)——这是旁路通道,不是协议变更。 +- **入口**:`prepareAnthropic` 把列表传给对账与快照;存储按 `(session_id, host_id, host_slot)` 键控行(schema v6,部分唯一索引)。同一条宿主消息以漂移后的字节重发时**原地刷新该行**(有记录的 append-only 例外——DB 视图必须与内存尾部逐字节相等,#2960⑤ digest parity),并沿用原 seq(宿主键连续性:漂移不再铸造新 seq、不再搅动水位)。无宿主 id → 与今天 `(msg_id, content_hash)` 行为逐位一致。 +- **对账**:`planReconciliation` 的 Pass -1 在一切启发式 pass 之前按宿主 id 精确认领(多 core 组内 k-th,role 门);锚点跨 pass 持久化 `h`。fork 快照响应带同序 `hostIds` 数组(构造上不进任何 hash),宿主 id 可映射回 bili ref。 + +没有消息级 id 的 lane(pi、omp、claude-code、codex、kimi、zcode)保持字节身份不变——采用按 lane 渐进、自动降级;字段缺失或畸形绝不比未盖章更差。 diff --git a/src/agent/dsh-native.ts b/src/agent/dsh-native.ts index b90ec72d19..b646eeeba9 100644 --- a/src/agent/dsh-native.ts +++ b/src/agent/dsh-native.ts @@ -51,7 +51,8 @@ import { APIG_RESIGN_SCHEME, unresolvedRefusals } from "../apig-resign.js"; import { resolveResignSettings } from "../config.js"; import { markNativeHost, nativeAttachOrigin, nativeBootstrapGate, nativeProxyScriptPath, proxyEnvOrigin, singleFlight } from "./native-bootstrap.js"; import { installNativeFetchIntercept, noteRoutedOrigin, observeRoutedOrigin, type GateVerdict, type NativeInterceptState } from "./native-intercept.js"; -import { asciiHeaderValue, fetchManifest, fetchProxyVersion, fetchStatus, fetchStatusLatest, forwardTool, reportRuntimeInfo, waitForProxyVersion, type ManifestTool } from "./shared.js"; +import { asciiHeaderValue, fetchManifest, fetchProxyVersion, fetchStatus, fetchStatusLatest, forwardTool, hostIdChannelForProxy, reportRuntimeInfo, waitForProxyVersion, type ManifestTool } from "./shared.js"; +import { collectWireHostIds } from "../host-id-channel.js"; import { createForkAdopter, sideShapedBody } from "./fork-adopt.js"; import { clearIncrementalProbe, INCREMENTAL_TURN_MARKER_HEADER, INCREMENTAL_TURN_ID_HEADER, newIncrementalTurnId, noteIncrementalRejection, noteIncrementalAck, planIncrementalTurn, probeIncrementalTurns, sweepStale, type IncrementalWireMessage } from "./incremental-client.js"; import { registerCompactionDisable } from "./dsh-compaction-disable.js"; @@ -1437,6 +1438,9 @@ export function apply(ctx: PluginContext): void { state.beforeSend = async (url, init) => { await maybeForkAdoptBeforeSend(ctx, init); await maybeIncrementalNative(url, init); + // #3464 (R1): LAST — an incremental takeover already replaced the body + // (marker set), and this stamps only full-resend anthropic bodies. + await maybeStampHostIds(init); }; // #3248 incremental turns (#3370 hardened): reference wiring for the @@ -1510,6 +1514,39 @@ export function apply(ctx: PluginContext): void { init.headers = headers; incPendingAcks.set(turnId, { sid, url, messages: parsed.messages as IncrementalWireMessage[], system: "system" in parsed ? parsed.system : undefined, base, at: Date.now() }); } + // #3464 (R1): passive host-id side channel. Hosts that serialize their + // per-message stable id into the outgoing anthropic body (dsh + // MessageBase.id) get it echoed back to bili in the reserved top-level + // field — identity-by-id instead of byte-hash guessing on the proxy side. + // Reads the id off each message of THIS request's own body (no cross- + // request state, no races); stamps only when the proxy advertised the + // channel (old proxies never see the field) and at least one message + // carries an id. Never touches marker'd/delta bodies or non-anthropic + // shapes; a malformed id list ships nothing (honest degradation). + function maybeStampHostIds(init: RequestInit | undefined): void { + if (!register.toolsReady || init === undefined) return; + const headers = init.headers instanceof Headers ? init.headers : new Headers(init.headers); + if (headers.get(INCREMENTAL_TURN_MARKER_HEADER) === "1") return; // delta lane owns that body + const attr = attributionOf(ctx); + if (attr.state !== "ok") return; + const body = init.body; + if (typeof body !== "string" || body.length === 0) return; + let parsed: Record; + try { + parsed = JSON.parse(body) as Record; + } catch { + return; + } + if (!Array.isArray(parsed.messages) || typeof parsed.model !== "string") return; // anthropic wire only + const base = register.base ?? state.origin; + if (base === undefined) return; + const channel = hostIdChannelForProxy(base); + if (channel === undefined) return; // proxy predates the channel — never stamp + const ids = collectWireHostIds(parsed.messages, channel.maxEntries); + if (ids === null) return; + parsed[channel.bodyField] = ids; + init.body = JSON.stringify(parsed); + } state.onResponse = (_url, response, headers) => { const turnId = headers?.[INCREMENTAL_TURN_ID_HEADER]; if (turnId === undefined) return; // not an incremental dispatch — nothing to attribute (#3329) diff --git a/src/agent/fork-adopt.ts b/src/agent/fork-adopt.ts index ee82a7c595..78ba5e03db 100644 --- a/src/agent/fork-adopt.ts +++ b/src/agent/fork-adopt.ts @@ -52,6 +52,10 @@ type SnapshotResponse = { parentRevision?: string; orderHash?: string; orderedMessages?: ForkIdentity[]; + /** #3464 (R1): index-aligned host message ids for stamped lanes — the + * host-id → bili-ref map. Outside every hash by construction; absent on + * older proxies or unstamped lanes. */ + hostIds?: (string | null)[]; }; type ForkAdoptionResult = diff --git a/src/agent/opencode-native.ts b/src/agent/opencode-native.ts index 62818d15ee..882c46b1e6 100644 --- a/src/agent/opencode-native.ts +++ b/src/agent/opencode-native.ts @@ -79,7 +79,8 @@ import { markNativeHost, nativeAttachOrigin, nativeBootstrapGate, nativeProxyScr import { createLiveOriginResolver, installNativeFetchIntercept, isModelApiUrl, noteRoutedOrigin, observeRoutedOrigin, readyOrigin, replaceRequestPayload, replaceRequestTarget, routedBiliModelUrl, type LiveOriginResolverDeps, type NativeInterceptState } from "./native-intercept.js"; import { clearIncrementalProbe, INCREMENTAL_TURN_MARKER_HEADER, INCREMENTAL_TURN_ID_HEADER, newIncrementalTurnId, noteIncrementalAck, noteIncrementalRejection, planIncrementalTurn, probeIncrementalTurns, sweepStale, wireMessagesLookAnthropic, type IncrementalWireMessage } from "./incremental-client.js"; import { createOpencodeV2Setup, type V2HttpRequestEvent, type V2State } from "./opencode-v2.js"; -import { asciiHeaderValue, fetchProxyVersion, postIdentityRegister, reportRuntimeInfoOnChange, waitForProxyVersion, type ForwardedToolResult } from "./shared.js"; +import { asciiHeaderValue, fetchProxyVersion, hostIdChannelForProxy, postIdentityRegister, reportRuntimeInfoOnChange, waitForProxyVersion, type ForwardedToolResult } from "./shared.js"; +import { collectWireHostIds } from "../host-id-channel.js"; import { callLegacyAcpConfig, isLegacyAcpSession, loadLegacyAcp, type LegacyAcpModule } from "./opencode-legacy.js"; import { logHostEvent } from "./host-log.js"; @@ -826,7 +827,32 @@ const server = async (ctx: V1PluginContext): Promise => { return createV1ServerHooks(() => state.origin, ctx, { z, legacy }); }; -export default { id: "billion-context-opencode-native", setup: createOpencodeV2Setup({ route: createNativeRoute(state), incremental: maybeIncrementalOpencode }), server }; +// #3464 (R1): the divert runs first; when it took the request, the marker +// header is set on the replacement and the stamp skips it (delta bodies are +// not stamped). Otherwise the outgoing anthropic body is stamped with the +// host message ids it carries — same passive recipe as the dsh lane. +async function maybeStampHostIdsOpencode(e: V2HttpRequestEvent, s: V2State): Promise { + const base = s.proxyBase; + if (!base) return; + const channel = hostIdChannelForProxy(base); + if (channel === undefined) return; // proxy predates the channel — never stamp + const req = e.request as Request | undefined; + if (req === undefined || typeof req.clone !== "function") return; + try { + if (req.headers.get(INCREMENTAL_TURN_MARKER_HEADER) === "1") return; // delta lane owns that body + const body: unknown = await req.clone().json().catch(() => null); + if (body === null || typeof body !== "object") return; + const b = body as { messages?: unknown; model?: unknown }; + if (typeof b.model !== "string" || !wireMessagesLookAnthropic(b.messages)) return; // anthropic-wire only + const ids = collectWireHostIds(b.messages, channel.maxEntries); + if (ids === null) return; + replaceRequestPayload(e, JSON.stringify({ ...(b as Record), [channel.bodyField]: ids })); + } catch { + // never break the host's request on adapter failure + } +} + +export default { id: "billion-context-opencode-native", setup: createOpencodeV2Setup({ route: createNativeRoute(state), incremental: async (e, s) => { await maybeIncrementalOpencode(e, s); await maybeStampHostIdsOpencode(e, s); } }), server }; /** #3248 phase 2: incremental-turn divert for opencode. Runs LAST in the * http.request hook (routing + header stamping already happened, so the diff --git a/src/agent/shared.ts b/src/agent/shared.ts index 408dea25fd..3a396400df 100644 --- a/src/agent/shared.ts +++ b/src/agent/shared.ts @@ -175,6 +175,26 @@ function recordManifestToolDuration(proxyBase: string, json: unknown): void { if (typeof value === "number" && Number.isFinite(value) && value > 0) manifestToolDurations.set(proxyBase, value); } +// #3464 (R1): per-proxy host-message-id side-channel, learned from the manifest +// at bootstrap. Plugins stamp the body field ONLY when their proxy advertises +// it — an older build would forward the unknown top-level field upstream, so +// capability gating is what makes the channel safe to ship unconditionally on +// the server side. +const manifestHostIdChannels = new Map(); + +function recordManifestHostIdChannel(proxyBase: string, json: unknown): void { + const caps = (json as { capabilities?: { hostMessageIds?: { bodyField?: unknown; maxEntries?: unknown } } })?.capabilities?.hostMessageIds; + if (typeof caps?.bodyField !== "string" || caps.bodyField.length === 0) return; + const maxEntries = typeof caps.maxEntries === "number" && Number.isSafeInteger(caps.maxEntries) && caps.maxEntries > 0 ? caps.maxEntries : 10000; + manifestHostIdChannels.set(proxyBase, { bodyField: caps.bodyField, maxEntries }); +} + +/** The body field this proxy accepts for wire-aligned host message ids, or + * undefined when the proxy predates #3464 (do NOT stamp). */ +export function hostIdChannelForProxy(proxyBase: string): { bodyField: string; maxEntries: number } | undefined { + return manifestHostIdChannels.get(proxyBase); +} + /** Effective HTTP wait (ms) for a forwarded tool call. Only compress invokes the * external summary executor (bounded by totalTimeoutMs), so only it grows past the * base floor — and only when the proxy advertised a budget. Everything else, and any @@ -190,6 +210,7 @@ export async function fetchManifest(proxyBase: string, format: "anthropic" | "op const { ok, status, json } = await fetchJson(`${proxyBase}/__bili/plugin/manifest`, undefined, MANIFEST_TIMEOUT_MS); if (!ok || !json || typeof json !== "object") throw new Error(`manifest fetch failed: ${status}`); recordManifestToolDuration(proxyBase, json); + recordManifestHostIdChannel(proxyBase, json); if (format === "openai") { // OpenAI function style: {name, description, parameters} (plain JSON Schema). const data = json as { tools?: { openai?: { name?: string; description?: string; parameters?: unknown }[] } }; diff --git a/src/fold-reconcile.ts b/src/fold-reconcile.ts index a715f7af54..0e511986f9 100644 --- a/src/fold-reconcile.ts +++ b/src/fold-reconcile.ts @@ -136,6 +136,7 @@ import { simhashHex, hammingHex } from "./affinity-simhash.js"; import { reapDestroyedSubstrate, type Session } from "./session.js"; import { coveredRealHistoryIds, isResponsesTurnSeparatorId } from "./session.js"; import { dshNativeCompactionWitness, recordConflict } from "./conflict-watch.js"; +import { coreHostRefs } from "./host-id-channel.js"; type FoldReconcileMode = "off" | "warn" | "repair"; @@ -291,6 +292,13 @@ export interface FoldAnchor { * 8-bit bar); the word bag can. Absent alongside `sh` with the same * graceful degradation. */ wh?: number; + /** #3464 (R1): host-stamped id of the WIRE message this core belonged to + * at anchor time — exact-set-algebra identity on stamped lanes (the + * strongest claim signal, Pass -1). Absent on anchors minted before R1 + * and on lanes without a host id channel (graceful degradation to the + * byte-based passes below). Bare host id: cores of one wire message + * share it and pair k-th-to-k-th in conversation order. */ + h?: string; } interface ReconciliationPlan { @@ -298,6 +306,9 @@ interface ReconciliationPlan { * merged core stands in for a reasoning+text pair (model switch, #2636). * Many→one maps both members to the same single target. */ claims: Map; + /** #3464 (R1): claims made on host-id equality alone (Pass -1) — exact + * set algebra on stamped lanes, before any byte-based pass runs. */ + byHost: number; byPos: number; byTool: number; byTwin: number; @@ -337,6 +348,7 @@ interface FoldReconcileResult { kind: "off" | "noop" | "resend" | "reanchored" | "unmatched"; missing: number; claims: number; + byHost: number; byPos: number; byTool: number; byTwin: number; @@ -656,8 +668,12 @@ export function planReconciliation( msgs: CoreMessage[], covered: Set, positions?: { ids: string[]; canon: string[] }, + /** #3464 (R1): wire-aligned host message ids for THIS request (null + * entries = padding). Zipped onto cores via their wireIndex sidecars + * (coreHostRefs); absent on lanes without a host id channel. */ + hostIds?: (string | null)[], ): ReconciliationPlan { - const plan: ReconciliationPlan = { claims: new Map(), byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, unmatched: [], idRewriteSuspects: 0, newOrder: [], nextCanon: undefined, resyncs: 0 }; + const plan: ReconciliationPlan = { claims: new Map(), byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, unmatched: [], idRewriteSuspects: 0, newOrder: [], nextCanon: undefined, resyncs: 0 }; // #2480: the positional copy is built on EVERY pass, so the first pass // after an upgrade establishes it and every later drift is // position-claimable. The fast path is reuse-by-ID (#2487): same id = @@ -734,6 +750,63 @@ export function planReconciliation( if (missingSet.has(id)) missingMiddle.push(id); } + // #3464 (R1): zip this request's wire-aligned host ids onto cores via + // their wireIndex sidecars. Empty on unstamped lanes → Pass -1 no-ops and + // every byte-based pass behaves exactly as pre-R1 (gradual adoption). + const hostById = new Map(); + if (hostIds !== undefined) { + const refs = coreHostRefs(msgs as ReadonlyArray<{ id: string; wireIndex?: number }>, hostIds); + for (let i = 0; i < msgs.length; i++) { + const m = msgs[i]; + const ref = refs[i]; + if (m.id === undefined || ref === null) continue; + hostById.set(m.id, ref.hostId); + } + } + + // Pass -1 — host-id exact set algebra (#3464 R1). On stamped lanes the + // host's own stable message ids outrank every byte-based signal: a + // covered id that vanished from the wire reappears under the SAME host + // id when its bytes drifted (marker strip, protocol re-serialization), + // and it is genuinely gone only when its host id is gone too (compaction + // replaces the batch with one summary id). Claim a missing covered id + // onto the unclaimed inbound core carrying the same host id — k-th-to-k- + // th in conversation order within the shared id (one wire message can + // project N cores, e.g. text + tool_use). The anchor's recorded role + // cross-checks the claim. Uniqueness: each inbound core is claimed at + // most once (claimedCandidates), so a host-id collision across two + // missing anchors pairs in order rather than double-claiming. + if (hostById.size > 0) { + const candsByHost = new Map(); + for (const cand of candidates) { + if (claimedCandidates.has(cand.id)) continue; + const hid = hostById.get(cand.id); + if (hid === undefined) continue; + const list = candsByHost.get(hid); + if (list === undefined) candsByHost.set(hid, [cand.id]); + else list.push(cand.id); + } + const cursorByHost = new Map(); + for (const oldId of missingMiddle) { + if (plan.claims.has(oldId)) continue; + const anchor = anchors[oldId]; + const hid = anchor?.h; + if (hid === undefined) continue; + const list = candsByHost.get(hid); + if (list === undefined) continue; + const k = cursorByHost.get(hid) ?? 0; + cursorByHost.set(hid, k + 1); + const candId = list[k]; + if (candId === undefined) continue; + const message = byId.get(candId); + if (message === undefined) continue; + if (anchor.r !== undefined && message.role !== anchor.r) continue; + plan.claims.set(oldId, [candId]); + claimedCandidates.add(candId); + plan.byHost++; + } + } + // Pass 0 — positional identity (#2480), order-centric since #2915. A // codec switch mints new ids for unchanged content, but the array COUNT // and ORDER survive; bili's stored canonical copy is the coordinate @@ -1262,6 +1335,10 @@ export interface ReconcileOptions { * cleans the resend — benign churn that must not feed the #2193 destroy * streak. #2815/#2789. */ markerStripped?: number; + /** #3464 (R1): this request's wire-aligned host message ids (null + * entries = padding; absent = lane not stamped). Enables Pass -1 exact + * claims and stamps anchor.h on seeded/reused anchors. */ + hostIds?: (string | null)[]; } /** Entry point wired into the four wire sites, immediately BEFORE the @@ -1271,13 +1348,13 @@ export function reconcileFoldCoverage(session: Session, msgs: CoreMessage[], opt const mode = opts.mode ?? resolveFoldReconcileMode(process.env); if (mode === "off") { resetFoldDriftState(session); - return { kind: "off", missing: 0, claims: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; + return { kind: "off", missing: 0, claims: 0, byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; } const blocks = (session.state?.blocks ?? []) as BlockLike[]; const covered = coveredIdsOf(blocks); if (covered.size === 0) { resetFoldDriftState(session); - return { kind: "noop", missing: 0, claims: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; + return { kind: "noop", missing: 0, claims: 0, byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; } // #2202: auxiliary side-requests (title-gen, WebSearch refinement — #1075) // share the conversation id but do not carry the conversation. Reconciling @@ -1290,9 +1367,9 @@ export function reconcileFoldCoverage(session: Session, msgs: CoreMessage[], opt // the episode state is left exactly as found (the resets above stay // reserved for true episode boundaries: reconcile off / no folds at all). if (msgs.length < SIDE_REQUEST_MAX_MSGS) { - return { kind: "noop", missing: 0, claims: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; + return { kind: "noop", missing: 0, claims: 0, byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; } - if (!session.metadata) return { kind: "noop", missing: 0, claims: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; + if (!session.metadata) return { kind: "noop", missing: 0, claims: 0, byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }; const anchors: Record = (session.metadata[METADATA_ANCHORS] as Record | undefined) ?? {}; @@ -1310,11 +1387,24 @@ export function reconcileFoldCoverage(session: Session, msgs: CoreMessage[], opt positions = { ids: storedRaw.ids as string[], canon: storedRaw.canon as string[] }; } - const plan = planReconciliation(oldOrder, anchors, msgs, covered, positions); + const plan = planReconciliation(oldOrder, anchors, msgs, covered, positions, opts.hostIds); + // #3464 (R1): seed-side zip of this request's host ids onto cores — the + // same pairing rule as Pass -1 (bare host id per core via wireIndex), + // used only to stamp anchor.h below. + const hostById = new Map(); + if (opts.hostIds !== undefined) { + const refs = coreHostRefs(msgs as ReadonlyArray<{ id: string; wireIndex?: number }>, opts.hostIds); + for (let i = 0; i < msgs.length; i++) { + const m = msgs[i]; + const ref = refs[i]; + if (m.id === undefined || ref === null) continue; + hostById.set(m.id, ref.hostId); + } + } if (process.env.FOLD_RECONCILE_DEBUG === "1") { const active = blocks.filter((b) => b.active).length; const sample = msgs.slice(0, 6).map((m) => `${m.role}/${m.contentType ?? "-"}/${m.id ?? "?"}`); - process.stderr.write(`[fold-reconcile-dbg] blocks=${blocks.length} active=${active} covered=${covered.size} oldOrder=${oldOrder.length} positions=${positions?.canon.length ?? -1} missing=${plan.unmatched.length + plan.claims.size} claims=${plan.claims.size} byPos=${plan.byPos} resyncs=${plan.resyncs} byTool=${plan.byTool} byTwin=${plan.byTwin} bySim=${plan.bySim} byOrder=${plan.byOrder} byNorm=${plan.byNorm} byTurn=${plan.byTurn} byImage=${plan.byImage} unmatched=${plan.unmatched.length} | msgs[0..5]=${sample.join(" | ")}\n`); + process.stderr.write(`[fold-reconcile-dbg] blocks=${blocks.length} active=${active} covered=${covered.size} oldOrder=${oldOrder.length} positions=${positions?.canon.length ?? -1} missing=${plan.unmatched.length + plan.claims.size} claims=${plan.claims.size} byHost=${plan.byHost} byPos=${plan.byPos} resyncs=${plan.resyncs} byTool=${plan.byTool} byTwin=${plan.byTwin} bySim=${plan.bySim} byOrder=${plan.byOrder} byNorm=${plan.byNorm} byTurn=${plan.byTurn} byImage=${plan.byImage} unmatched=${plan.unmatched.length} | msgs[0..5]=${sample.join(" | ")}\n`); } // Seed/refresh anchors for covered ids present in this pass (including @@ -1408,6 +1498,15 @@ export function reconcileFoldCoverage(session: Session, msgs: CoreMessage[], opt anchorCount++; } } + // #3464 (R1): stamp anchor.h on every anchor this pass kept — claimed, + // freshly seeded, or reused. Reused anchors may be references into the + // persisted record: mutating them IS the backfill (host ids persist with + // the anchor, so the next churn claims exactly). + for (const [id, a] of Object.entries(nextAnchors)) { + if (a.h !== undefined) continue; + const hid = hostById.get(id); + if (hid !== undefined) a.h = hid; + } const nextOrder = plan.newOrder.slice(-MAX_ORDER); // #2480: roll the positional copy forward in the same window as the order // backbone (same slice, same pass), stored self-contained as {ids, canon} @@ -1616,7 +1715,7 @@ export function reconcileFoldCoverage(session: Session, msgs: CoreMessage[], opt } if (plan.unmatched.length === 0 && plan.claims.size === 0) { - return { kind: "resend", missing: 0, claims: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: reapedCount, unmatched: 0 }; + return { kind: "resend", missing: 0, claims: 0, byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: reapedCount, unmatched: 0 }; } if (opts.log !== undefined) { if (plan.claims.size > 0 && mode === "repair") { @@ -1654,6 +1753,7 @@ export function reconcileFoldCoverage(session: Session, msgs: CoreMessage[], opt kind: plan.claims.size > 0 ? (mode === "repair" ? "reanchored" : "unmatched") : "unmatched", missing: plan.claims.size + plan.unmatched.length, claims: plan.claims.size, + byHost: plan.byHost, byPos: plan.byPos, byTool: plan.byTool, byTwin: plan.byTwin, diff --git a/src/host-id-channel.ts b/src/host-id-channel.ts new file mode 100644 index 0000000000..03afffb4c7 --- /dev/null +++ b/src/host-id-channel.ts @@ -0,0 +1,150 @@ +/** + * #3464 (R1 of the #2960 identity pipeline): host message-id side channel. + * + * Native plugin hosts (dsh, opencode) assign every message a stable internal + * id, but the wire never carries it. The wire body is all bili sees, and all + * three identity layers were historically derived from content bytes + * (MESSAGE-IDENTITY.md) — any byte drift (marker stripping, re-encoding, + * provider rewrite, dsh edit) collapsed message identity and forced fold- + * reconcile into after-the-fact guessing ("are these two really the same + * message?"). + * + * This channel fixes it at the source: the plugin stamps the ordered list of + * host ids for the wire messages as a top-level body field; the proxy + * extracts it at the single parse chokepoint and NEVER forwards it upstream + * (the field is deleted from the parsed object before any path can serialize + * it again). Where the channel is active, reconciliation degrades from + * byte-hash inference to exact set algebra (host-id equality claims) and the + * canonical store keys ONE row per (host_id, slot) instead of per content + * hash — a drifted resend refreshes the row instead of minting a twin. + * + * Adoption is gradual by lane capability: only hosts that expose message ids + * stamp (dsh MessageBase.id, opencode msg_…). Lanes without ids keep byte-id + * behavior bit-identical. A malformed stamp is stripped and ignored for that + * request — honest degradation, never worse than status quo. + * + * Wire shape (anthropic v1; the field is protocol-neutral JSON): + * { ..., "messages": [...], "_bili_message_ids": ["", null, ...] } + * aligned index-for-index with `messages`. One wire message may expand into + * several kernel cores (one per content block); the intra-group ordinal + * ("slot") disambiguates them — see coreHostRefs. + */ + +/** Reserved top-level request-body field. Never forwarded upstream. */ +export const HOST_ID_BODY_FIELD = "_bili_message_ids"; +/** Host ids are opaque strings; printable ASCII keeps the storage columns + * cheap and the composite-key separator (\u0000) injection-proof. */ +const HOST_ID_PATTERN = /^[\x21-\x7e]+$/; +export const HOST_ID_MAX_ENTRY_LEN = 1024; +export const HOST_ID_MAX_ENTRIES = 10_000; + +export interface HostIdExtract { + /** Validated wire-aligned host ids (null = that wire message carries no + * id), or null when the field was absent or malformed. */ + ids: (string | null)[] | null; + /** True when the reserved field was present and removed from `parsed`. */ + stripped: boolean; +} + +/** Extracts AND strips the side channel from a parsed request body. The + * delete happens for EVERY presence — including malformed ones — so the + * reserved namespace can never reach an upstream through a later + * re-serialization. `wireMessages` is the body's own message array + * (protocol-specific: anthropic/openai `messages`, responses `input`, + * google `contents`); length mismatch or non-array ⇒ malformed. */ +export function extractHostIds( + parsed: Record, + wireMessages: readonly unknown[] | undefined, +): HostIdExtract { + if (!(HOST_ID_BODY_FIELD in parsed)) return { ids: null, stripped: false }; + const raw = parsed[HOST_ID_BODY_FIELD]; + delete parsed[HOST_ID_BODY_FIELD]; + if (!Array.isArray(wireMessages)) return { ids: null, stripped: true }; + if (!Array.isArray(raw) || raw.length !== wireMessages.length || raw.length > HOST_ID_MAX_ENTRIES) { + return { ids: null, stripped: true }; + } + const ids: (string | null)[] = []; + for (const v of raw) { + if (v === null) { + ids.push(null); + continue; + } + if (typeof v !== "string" || v.length === 0 || v.length > HOST_ID_MAX_ENTRY_LEN || !HOST_ID_PATTERN.test(v)) { + return { ids: null, stripped: true }; + } + ids.push(v); + } + return { ids, stripped: true }; +} + +/** Plugin-side collector: reads each wire message's own `id` field (hosts + * that serialize it into the request body — dsh MessageBase.id, opencode + * msg_…) into a stamp-ready list. Returns null when nothing should be + * stamped: no id present at all, more messages than the channel allows, or + * ANY entry malformed (non-string / empty / oversized / non-printable-ASCII) + * — mirroring extractHostIds' server-side validation so a half-valid stamp + * can never ship. Null PAD entries are fine (that message carries no id). */ +export function collectWireHostIds( + wireMessages: readonly unknown[], + maxEntries: number, +): (string | null)[] | null { + if (wireMessages.length > maxEntries) return null; + let present = 0; + const out: (string | null)[] = new Array(wireMessages.length); + for (let i = 0; i < wireMessages.length; i++) { + const m = wireMessages[i]; + const id = m !== null && typeof m === "object" ? (m as { id?: unknown }).id : undefined; + if (id === undefined || id === null) { + out[i] = null; + continue; + } + if (typeof id !== "string" || id.length === 0 || id.length > HOST_ID_MAX_ENTRY_LEN || !HOST_ID_PATTERN.test(id)) { + return null; + } + out[i] = id; + present++; + } + return present > 0 ? out : null; +} + +/** One core's position inside its host message: the host id plus the ordinal + * among the cores that the SAME wire message expanded into (kernel adapters + * emit one core per content block and tag each with `wireIndex`). */ +export interface CoreHostRef { + hostId: string; + slot: number; +} + +/** Composite identity used by dedup maps, anchors and the partial unique + * index — the separator cannot occur inside a validated host id. */ +export function hostKeyOf(ref: CoreHostRef): string { + return `${ref.hostId}\u0000${ref.slot}`; +} + +/** Zips a wire-aligned host-id list onto the converted cores. Returns null + * per core wherever the pairing is unavailable (lane not stamped, entry + * null, or the core lacks a usable wireIndex tag — the kernel tags it only + * on the anthropic wire today; other adapters degrade honestly). A core + * without a wireIndex gets a per-core-unique slot so it can never collide + * with another core's slot by accident (it also cannot dedup across legs — + * conservative, by design). */ +export function coreHostRefs( + msgs: ReadonlyArray<{ id: string; wireIndex?: number }>, + ids: readonly (string | null)[] | null | undefined, +): (CoreHostRef | null)[] { + if (ids === null || ids === undefined || ids.length !== msgs.length) return msgs.map(() => null); + const slotByWire = new Map(); + return msgs.map((m, i) => { + const h = ids[i]; + if (h === null || h === undefined) return null; + const wi = m.wireIndex; + let slot: number; + if (typeof wi === "number" && Number.isSafeInteger(wi) && wi >= 0) { + slot = slotByWire.get(wi) ?? 0; + slotByWire.set(wi, slot + 1); + } else { + slot = i; + } + return { hostId: h, slot }; + }); +} diff --git a/src/persist.ts b/src/persist.ts index 2ce5187ade..16fbdcf3c7 100644 --- a/src/persist.ts +++ b/src/persist.ts @@ -200,6 +200,12 @@ interface PersistedSession { * history and must still be pruned at export time. */ messagesFolded?: boolean; pluginSnapshot?: CoreMessage[]; + /** #3464 (R1): host message ids aligned with `pluginSnapshot` (one entry + * per wire message the snapshot cores expanded from; null = that wire + * message carried no id). Persisted only when the snapshot itself is + * persisted (fork-receipt gating) — the ingest dedup needs it across a + * restart exactly when the raw snapshot leg re-enters the union. */ + pluginSnapshotHostIds?: (string | null)[]; forkContentStore?: MessageContentStore; } @@ -1645,7 +1651,14 @@ export class SessionStore { // which the delta path already routes to the full ingest. const tail = session.lastMessagesFolded !== true ? session.lastMessages : undefined; this.shadow?.record(record, tail && tail.length > 0 - ? { lastMessages: tail, lastMessagesFolded: session.lastMessagesFolded } + ? { + lastMessages: tail, + lastMessagesFolded: session.lastMessagesFolded, + // #3464 (R1): the host-id list rides with the RAW tail + // only — a folded re-render has no stable per-core + // pairing with its stamped source. + ...(session.lastMessagesHostIds ? { lastMessagesHostIds: session.lastMessagesHostIds } : {}), + } : undefined, session.contentStore ?? undefined); this.noteOwnCommit(session.id, record); return record; @@ -1710,7 +1723,13 @@ export class SessionStore { // never older than what the disk copy carries. session.lastMessages = fresh.lastMessages ?? session.lastMessages; session.lastMessagesFolded = fresh.lastMessages !== undefined ? fresh.lastMessagesFolded : session.lastMessagesFolded; + // #3464 (R1): the host-id list is runtime-only (never in the + // envelope) — a disk tail that replaced the in-memory one has no + // pairing for it, so the channel is inert until the next stamped + // request re-stamps (the watermark protects the stored prefix). + if (fresh.lastMessages !== undefined) session.lastMessagesHostIds = undefined; session.pluginSnapshot = fresh.pluginSnapshot; + session.pluginSnapshotHostIds = fresh.pluginSnapshotHostIds; session.lastSeen = fresh.lastSeen; if (!session.meta.protocol && fresh.meta.protocol) session.meta.protocol = fresh.meta.protocol; if (!session.meta.upstreamOrigin && fresh.meta.upstreamOrigin) session.meta.upstreamOrigin = fresh.meta.upstreamOrigin; @@ -1983,6 +2002,9 @@ function buildRecord(session: Session, storageMode: "legacy" | "shadow" | "full" // sessions pay no disk cost; the in-memory copy self-heals on the next // model request because plugin agents resend their full history. pluginSnapshot: session.metadata.publicForkReceipt !== undefined || session.metadata.publicSnapshotRetained === true ? session.pluginSnapshot : undefined, + // #3464 (R1): rides the same fork-receipt gate as the snapshot it + // aligns with — absent snapshot, absent pairing. + pluginSnapshotHostIds: session.metadata.publicForkReceipt !== undefined || session.metadata.publicSnapshotRetained === true ? session.pluginSnapshotHostIds : undefined, forkContentStore: session.metadata.publicForkReceipt ? session.contentStore : undefined, // Per-session provenance: record the bili build that wrote this file so the // web UI can show which version last touched the session; pre-stamp files @@ -2129,6 +2151,12 @@ function buildSession(parsed: PersistedSession): Session { lastMessages: Array.isArray(parsed.messages) ? parsed.messages : undefined, lastMessagesFolded: parsed.messagesFolded === true, pluginSnapshot: Array.isArray(parsed.pluginSnapshot) ? parsed.pluginSnapshot : undefined, + // #3464 (R1): reject a malformed or misaligned list outright — a + // half-valid pairing would dedup the WRONG cores. + pluginSnapshotHostIds: Array.isArray(parsed.pluginSnapshot) && isHostIdList(parsed.pluginSnapshotHostIds) + && parsed.pluginSnapshotHostIds.length === parsed.pluginSnapshot.length + ? parsed.pluginSnapshotHostIds + : undefined, contentStore: parsed.forkContentStore, inFlight: 0, persisted: true, @@ -2142,6 +2170,12 @@ export function isValidRecord(parsed: unknown): parsed is PersistedSession { return typeof r.id === "string" && typeof r.state === "object" && r.state !== null && Array.isArray(r.state.blocks); } +/** #3464 (R1): shape check for a persisted host-id list — every entry is a + * non-empty string or an explicit null (the stamper's "no id here"). */ +function isHostIdList(v: unknown): v is (string | null)[] { + return Array.isArray(v) && v.every((e) => e === null || (typeof e === "string" && e.length > 0)); +} + /** #408: true when a persisted record (grouped v2+ or flat v1) carries a * negative lastInputTokens/contextTokens — only possible in files written by * pre-clamp versions. buildSession clamps on read; this lets the loaders diff --git a/src/plugin.ts b/src/plugin.ts index 7b46a9fca8..6d8ad5345a 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -1,5 +1,6 @@ import { BLOCKED_REF, COMPRESS_TOOL_NAME, type CompressionCore, type Config, type CoreMessage, type NudgeDecision, type RetrievalExport, applyRetrieve, countMessageTokens, parseStoredPlaceholder, prune } from "acp-kernel"; import { publicSnapshotCapBytes } from "./knobs.js"; +import { coreHostRefs, HOST_ID_BODY_FIELD, HOST_ID_MAX_ENTRIES } from "./host-id-channel.js"; import { buildStatusPanel } from "acp-kernel/panel"; import { fileURLToPath } from "node:url"; import type { ServerResponse } from "node:http"; @@ -351,7 +352,7 @@ export function recordPluginSession(conversationId: string, sessionId: string): /** Keep the last prepare()'s view for a plugin session so tool-API execution * sees the exact refs the model was shown (mirrors the wire-mode loop, which * runs executeProxyTool against prepared.processedMessages). */ -export function rememberPluginMessages(sessionId: string, processed: CoreMessage[], original: CoreMessage[], nudge?: NudgeDecision, rawWire?: Buffer): void { +export function rememberPluginMessages(sessionId: string, processed: CoreMessage[], original: CoreMessage[], nudge?: NudgeDecision, rawWire?: Buffer, hostIds?: (string | null)[]): void { // #1307: auxiliary requests (auto-review / classifier prompts) bound to the // same session key can carry a normal output budget and any message count, // so both the ≤200 heuristic and size-based guards are proxies that a new @@ -392,6 +393,9 @@ export function rememberPluginMessages(sessionId: string, processed: CoreMessage const cap = publicSnapshotCapBytes(); const dropOverCap = () => { session.pluginSnapshot = undefined; + // #3464 (R1): the pairing list dies with its snapshot — a stale + // pairing would mis-attribute identity on the next ingest. + session.pluginSnapshotHostIds = undefined; session.metadata.publicSnapshotCapped = true; delete session.metadata.publicSnapshotBytes; }; @@ -405,6 +409,10 @@ export function rememberPluginMessages(sessionId: string, processed: CoreMessage if (cap > 0 && bytes > cap) dropOverCap(); else { session.pluginSnapshot = structuredClone(original); + // #3464 (R1): wire-aligned stamp of THIS request — the snapshot + // now reflects exactly its wire messages, so the full list is + // the correct pairing for every core inside it. + session.pluginSnapshotHostIds = hostIds === undefined ? undefined : [...hostIds]; session.metadata.publicSnapshotBytes = bytes; delete session.metadata.publicSnapshotCapped; } @@ -438,6 +446,11 @@ export function rememberPluginMessages(sessionId: string, processed: CoreMessage if (cap > 0 && bytes > cap) dropOverCap(); else { for (let i = k; i < n; i++) prev.push(structuredClone(original[i])); + // #3464 (R1): extension appends cores whose wire messages + // continue this request's numbering, so the full wire list + // of THIS request pairs every core of the extended + // snapshot (wire indices never restart mid-conversation). + session.pluginSnapshotHostIds = hostIds === undefined ? undefined : [...hostIds]; session.metadata.publicSnapshotBytes = bytes; delete session.metadata.publicSnapshotCapped; } @@ -449,6 +462,9 @@ export function rememberPluginMessages(sessionId: string, processed: CoreMessage if (cap > 0 && bytes > cap) dropOverCap(); else { session.pluginSnapshot = structuredClone(original); + // #3464 (R1): same full-list pairing as first write — the + // rebuilt snapshot reflects exactly this request's wire. + session.pluginSnapshotHostIds = hostIds === undefined ? undefined : [...hostIds]; session.metadata.publicSnapshotBytes = bytes; delete session.metadata.publicSnapshotCapped; } @@ -858,7 +874,7 @@ export function handlePluginManifest(res: import("node:http").ServerResponse, co // #2652: maxToolDurationMs = the budget's totalTimeoutMs ceiling, so agent-side // bridges size their compress-call HTTP wait to the real server bound instead of a // fixed constant. Absent when the chain is off (classic summaries commit locally). - capabilities: { ...(externalSummary ? { externalSummary: { enabled: true, summaryOptional: true, submittedSummary: "hint", maxToolDurationMs: externalSummaryMaxToolDurationMs(config) } } : {}), fork: { protocolVersion: 2, acceptsProtocolVersions: [1, 2], endpoint: "/__bili/plugin/fork", snapshotEndpoint: "/__bili/plugin/snapshot" } }, + capabilities: { ...(externalSummary ? { externalSummary: { enabled: true, summaryOptional: true, submittedSummary: "hint", maxToolDurationMs: externalSummaryMaxToolDurationMs(config) } } : {}), fork: { protocolVersion: 2, acceptsProtocolVersions: [1, 2], endpoint: "/__bili/plugin/fork", snapshotEndpoint: "/__bili/plugin/snapshot" }, hostMessageIds: { bodyField: HOST_ID_BODY_FIELD, maxEntries: HOST_ID_MAX_ENTRIES } }, })); } @@ -1040,7 +1056,13 @@ function forkSnapshot(session: Session) { // the revision is only the cheap early-exit token. const activeBlockSet = session.state.blocks.filter((b) => b.active).map((b) => b.blockId).sort(); const parentRevision = forkHash({ sessionId: session.id, seq: orderedMessages.length, blockSet: activeBlockSet }); - return { protocolVersion: 2, status: "exact", sessionId: session.id, parentRevision, orderHash: forkOrderHash(orderedMessages), orderedMessages, messages: messages.map((m, i) => ({ rawId: m.id, ref: orderedMessages[i]!.ref, role: m.role, text: m.text, toolName: m.toolName, toolCallId: m.toolCallId, contentType: m.contentType, toolIsError: forkToolIsError(m) })) }; + // #3464 (R1): host-id → bili-ref map for stamped lanes. Deliberately OUTSIDE + // orderedMessages: that array feeds orderHash/stableJson branch-point + // comparison, so mixing host ids in would break old-client/new-server + // pairs. A parallel index-aligned array is additive — old clients ignore + // it, and it must never influence any hash. + const snapshotHostRefs = coreHostRefs(messages as ReadonlyArray<{ id: string; wireIndex?: number }>, session.pluginSnapshotHostIds); + return { protocolVersion: 2, status: "exact", sessionId: session.id, parentRevision, orderHash: forkOrderHash(orderedMessages), orderedMessages, hostIds: snapshotHostRefs.map((r) => r?.hostId ?? null), messages: messages.map((m, i) => ({ rawId: m.id, ref: orderedMessages[i]!.ref, role: m.role, text: m.text, toolName: m.toolName, toolCallId: m.toolCallId, contentType: m.contentType, toolIsError: forkToolIsError(m) })) }; } /** #2671 Phase 2b transition digest: fork protocol v1 hashed the ENTIRE @@ -1247,7 +1269,24 @@ export async function handlePluginFork(payload: string, res: ServerResponse): Pr child.state.stats.tokensCompressed = child.state.blocks.filter((b) => b.active).reduce((sum, b) => sum + b.compressedTokens, 0); child.state.stats.compressionCount = blocks.length; if (prefix.length === snapshot.orderedMessages.length) child.state.nudge = structuredClone(parent.state.nudge); - child.pluginSnapshot = structuredClone(parent.pluginSnapshot!.slice(0, request.branchPoint.messageCount)); + const forkedPrefix = parent.pluginSnapshot!.slice(0, request.branchPoint.messageCount); + // #3464 (R1): the pairing list is WIRE-aligned while the fork + // boundary is counted in CORES — derive the covered wire span from + // the prefix's wireIndex sidecars (non-decreasing down the array, + // every wire message yields ≥1 core ⇒ span is exactly 0..max). + // No wireIndex anywhere (lane without host ids) → no pairing, + // byte-keyed ingest as before. + let forkedHostIds: (string | null)[] | undefined; + if (parent.pluginSnapshotHostIds !== undefined) { + let maxWire = -1; + for (const m of forkedPrefix) { + const wi = (m as { wireIndex?: unknown }).wireIndex; + if (typeof wi === "number" && Number.isSafeInteger(wi) && wi > maxWire) maxWire = wi; + } + if (maxWire >= 0) forkedHostIds = parent.pluginSnapshotHostIds.slice(0, maxWire + 1); + } + child.pluginSnapshot = structuredClone(forkedPrefix); + child.pluginSnapshotHostIds = forkedHostIds; const forkView = prune(child.pluginSnapshot, child.state); // Hiding the original pair is safe only if its summary survives the fork. child.state.absorbed = structuredClone(parent.state.absorbed?.filter((a) => @@ -1256,6 +1295,9 @@ export async function handlePluginFork(payload: string, res: ServerResponse): Pr child.state.blocks.some((b) => b.active && !b.expanded && a.summary.length > 0 && b.summary.includes(a.summary)) )) ?? []); child.lastMessages = structuredClone(child.pluginSnapshot); + // #3464 (R1): lastMessages here IS the raw forked prefix, so it + // carries the same wire-aligned pairing as pluginSnapshot. + child.lastMessagesHostIds = child.pluginSnapshotHostIds; child.lastMessagesFolded = false; const parentStore = contentStoreOf(parent); for (const { ref, rawId } of prefix) { diff --git a/src/server.ts b/src/server.ts index a2d4fc103d..b90686dc73 100644 --- a/src/server.ts +++ b/src/server.ts @@ -1413,6 +1413,10 @@ export type Prepared = { * text (collectBlockContent reads message text by id); processedMessages * has compressed messages replaced with placeholders → empty content. */ originalMessages: CoreMessage[]; + /** #3464 (R1): wire-aligned host message ids for this request's inbound + * messages (null entries = padding). Present on stamped plugin lanes + * only; consumers zip per-core via BiliMessage.wireIndex. */ + hostIds?: (string | null)[]; protocol: WireProtocol; stream: boolean; compressInjected: boolean; diff --git a/src/server/handle.ts b/src/server/handle.ts index 372f447277..4431eb5fc3 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -37,6 +37,7 @@ import { personaConversationOrder } from "../session-resolve.js"; import { prefixAffinity, type AnonymousAffinity } from "../prefix-affinity.js"; import { buildAnonymousProvenance, type AnonymousProvenance } from "../anon-provenance.js"; import { maybeAdoptForkBlocks, maybeAdoptResume, maybeAdoptDerivedFork } from "../fork-adoption.js"; +import { HOST_ID_BODY_FIELD, extractHostIds } from "../host-id-channel.js"; import { executeThreadsReforge } from "./threads-lane.js"; import { resetThreadTailTracking } from "./thread-tracking.js"; import { resolveClaudeTranscriptLineage } from "../claude-transcript-lineage.js"; @@ -473,6 +474,27 @@ async function handleRequest( parsed = null; } } + // #3464 (R1): host message-id side channel — extracted at the single + // parse chokepoint so EVERY downstream path (processed rebuild, chain + // skip, verbatim relays) sees the field gone. extractHostIds deletes it + // for any presence, well-formed or not; the re-serialization below only + // fires when a stamp was actually carried, so unstamped traffic keeps its + // exact bytes. Malformed stamps degrade to "channel off for this + // request" — never worse than status quo. + let stampedWireIds: (string | null)[] | null = null; + { + const p = parsed !== null && typeof parsed === "object" && !Array.isArray(parsed) ? parsed as Record : undefined; + if (p && HOST_ID_BODY_FIELD in p) { + const wireMessages = Array.isArray(p.messages) ? p.messages : Array.isArray(p.input) ? p.input : Array.isArray(p.contents) ? p.contents : undefined; + const ex = extractHostIds(p, wireMessages); + stampedWireIds = ex.ids; + bodyText = JSON.stringify(p); + bodyBuffer = Buffer.from(bodyText, "utf8"); + log(ex.ids === null ? "debug" : "info", ex.ids === null + ? "[hostids] malformed _bili_message_ids field stripped upstream; identity channel ignored for this request (#3464)" + : `[hostids] host message-id channel active (${ex.ids.filter((e) => e !== null).length}/${wireMessages?.length ?? "?"} ids) (#3464)`); + } + } // #3129: CloudCode/Antigravity v1internal dialect — unwrap the outer envelope // so the pipeline sees a standard GoogleRequest, and decide identity. The // inner request.sessionId is a constant FNV-1a hash, so identity comes from @@ -2253,7 +2275,11 @@ async function handleRequest( return await prepareGoogle(work as GoogleRequestBody, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, nativeWindow, googleModel ?? v1Model, googlePathKind(urlPath) === "stream-generate", visibilityMarkers, upstreamOrigin, req, decide); } const preparedOut = protocol === "anthropic" - ? await prepareAnthropic(work as AnthropicRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, reasoningCfg, visibilityMarkers, requestModel, decide) + // #3464 (R1): wire-aligned host ids from the parse- + // chokepoint extraction; only the anthropic lane zips + // them onto cores today (the kernel tags wireIndex + // there), other lanes degrade to byte-id behavior. + ? await prepareAnthropic(work as AnthropicRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, reasoningCfg, visibilityMarkers, requestModel, decide, stampedWireIds) : protocol === "openai" ? await prepareOpenai(work as OpenAIRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg, visibilityMarkers, route?.rewrittenUrl, decide) : responsesCompact @@ -2543,7 +2569,7 @@ async function handleRequest( // plugin tool call sees a consistent window. if (pendingForward.prepared) { const preparedToRemember = pendingForward.prepared; - await withSessionLock(session, () => rememberPluginMessages(sessionId, preparedToRemember.processedMessages, preparedToRemember.originalMessages, preparedToRemember.nudge, bodyBuffer)); + await withSessionLock(session, () => rememberPluginMessages(sessionId, preparedToRemember.processedMessages, preparedToRemember.originalMessages, preparedToRemember.nudge, bodyBuffer, preparedToRemember.hostIds)); } } } finally { diff --git a/src/server/prepare-anthropic.ts b/src/server/prepare-anthropic.ts index 451a94bdc2..f2abd1fa06 100644 --- a/src/server/prepare-anthropic.ts +++ b/src/server/prepare-anthropic.ts @@ -50,6 +50,9 @@ export async function prepareAnthropic( * `model`, so the diag line below would otherwise print `undefined`. */ resolvedModel?: string, decide?: DecideConfig, + /** #3464 (R1): wire-aligned host message ids extracted at the parse + * chokepoint (null = lane not stamped or stamp malformed). */ + stampedWireIds?: (string | null)[] | null, ): Promise { const sessionId = session.id; const stream = parsed.stream === true; @@ -261,7 +264,9 @@ export async function prepareAnthropic( // [#1921] re-anchor fold coverage onto churned-but-same messages // before the #1195 snapshot, so covered ids surviving a client // re-serialization stay covered (src/fold-reconcile.ts). - reconcileFoldCoverage(session, msgs, { mode: resolveFoldReconcileMode(process.env, opts.compress.reconcile), sessionId, log, markerStripped: markerArtifactCount(markerCounts) }); + // #3464 (R1): wire-aligned host ids → Pass -1 exact claims + anchor.h + // stamps; absent on unstamped lanes (byte-based passes unchanged). + reconcileFoldCoverage(session, msgs, { mode: resolveFoldReconcileMode(process.env, opts.compress.reconcile), sessionId, log, markerStripped: markerArtifactCount(markerCounts), hostIds: stampedWireIds ?? undefined }); noteSystemPromptFingerprint(session, clientSystem, { sessionId, log }); // #1195: pre-turn snapshot of the fold's covered ids — syncBlocks inside // processTurn may deactivate fully-drifted blocks, erasing them. @@ -457,7 +462,9 @@ export async function prepareAnthropic( // SysPrompt row — the kernel breakdown classifies messages only, and on // this wire the system rides the top-level `system` field outside the fold. session.metadata.systemPromptTokens = countSystemAndToolsTokens(extractSystem(systemOut), toolsOut); - snapshotMessages(session, originalMessages); + // #3464 (R1): carry the wire-aligned host ids with the snapshot — + // ingest keys stamped-lane rows by (hostId, slot) instead of bytes. + snapshotMessages(session, originalMessages, stampedWireIds ?? undefined); markDirty(session); const rebuilt: AnthropicRequestBody = { ...parsed, messages: rebuiltMessages, system: systemOut, tools: toolsOut }; @@ -483,7 +490,7 @@ export async function prepareAnthropic( + countSystemAndToolsTokens(extractSystem(systemOut), toolsOut) + imageReserveFor(session, "anthropic", rebuilt, opts, upstreamOrigin); if (upstreamOrigin) session.stats.lastLocalTextEstimateOrigin = upstreamOrigin; - return { body: JSON.stringify(rebuilt), session, attachedRetrievals, attachedRetrievalNoteIds, processedMessages, originalMessages, anthropicSystem: parsed.system, anthropicBillingBlock, anthropicCacheMarks, anthropicClientCacheControls: clientCacheControls, systemNotes: sysNotes, protocol: "anthropic", stream, compressInjected: injectTools, pluginMode, nudge, nudgeOutcome, prompts, surface, renderTags: renderStrategy, dropReasoning: (msgs: BiliMessage[]) => applyReasoningDrop(msgs, reasoningPlan, log, sessionId) } as Prepared; + return { body: JSON.stringify(rebuilt), session, attachedRetrievals, attachedRetrievalNoteIds, processedMessages, originalMessages, anthropicSystem: parsed.system, anthropicBillingBlock, anthropicCacheMarks, anthropicClientCacheControls: clientCacheControls, systemNotes: sysNotes, protocol: "anthropic", stream, compressInjected: injectTools, pluginMode, nudge, nudgeOutcome, prompts, surface, renderTags: renderStrategy, dropReasoning: (msgs: BiliMessage[]) => applyReasoningDrop(msgs, reasoningPlan, log, sessionId), hostIds: stampedWireIds ?? undefined } as Prepared; } // #2648: true when any system block pins cache_control.scope:"global". bili's diff --git a/src/session.ts b/src/session.ts index cfdfb0bea8..18d0bd0fd5 100644 --- a/src/session.ts +++ b/src/session.ts @@ -514,6 +514,15 @@ export type Session = { * request — the client re-sends full raw history, restoring the invariant. */ lastMessagesFolded?: boolean; pluginSnapshot?: CoreMessage[]; + /** #3464 (R1) wire-aligned host message ids, one entry per WIRE message of + * the request that produced lastMessages (null entries = padding). + * In-memory only: cleared wherever lastMessages is re-rendered or replaced + * from disk — a stale pairing would mis-attribute identity. */ + lastMessagesHostIds?: (string | null)[]; + /** #3464 (R1): wire-aligned host ids for pluginSnapshot (same pairing + * rule as lastMessagesHostIds). Persisted beside pluginSnapshot under the + * same fork-receipt gate — the fork child replays it into ingest. */ + pluginSnapshotHostIds?: (string | null)[]; /** In-memory only (NOT persisted — buildRecord omits it): monotone counter * bumped by markDirty after every mutation. Keys the fork-revision cache * below so status polling skips re-hashing the whole history (#2017). */ @@ -1246,10 +1255,14 @@ export function findSessionByCanonicalId(canonicalId: string): Session | undefin * raw request messages (originalMessages from prepare*). One array per * session, replaced every request — bounded, always the newest state. Empty * arrays (parse failures) never clobber a good snapshot. */ -export function snapshotMessages(session: Session, messages: CoreMessage[]): void { +export function snapshotMessages(session: Session, messages: CoreMessage[], lastMessagesHostIds?: (string | null)[] | null): void { if (messages.length > 0) { session.lastMessages = messages; session.lastMessagesFolded = false; + // #3464 (R1): exact replacement, never merge — a shorter/absent stamp + // for this request means the lane is unstamped and any leftover pairing + // would mis-attribute identity on the next ingest. + session.lastMessagesHostIds = lastMessagesHostIds ?? undefined; } } diff --git a/src/storage/ingest.ts b/src/storage/ingest.ts index 429a8a8f57..57a1077ecd 100644 --- a/src/storage/ingest.ts +++ b/src/storage/ingest.ts @@ -11,6 +11,14 @@ * content parts → distinct rows. Byte-identical resends are the same row * (kernel identity is id-based; #1476 renumbers new instances, which get * new ids anyway). seq = first-seen arrival order. + * - #3464 (R1): stamped lanes additionally key by the host message id — + * one row per (host_id, host_slot), where slot is the core's ordinal + * inside its wire message (host-id channel). A byte-drifted resend of a + * host-identified message REFRESHES that row's bytes/ref instead of + * minting a twin (the documented append-only exception: the stored view + * must stay equal to the in-memory tail or the verify digest false- + * alarms); the row keeps its first-seen seq position. Lanes without + * host ids keep pure byte-key behavior. * - pluginSnapshot / lastMessages are POINTER SEQUENCES into that log * (tail_json): they are views (subsequences), never re-stored as content. * pluginSnapshot is the frozen raw prefix; lastMessages is the folded @@ -28,6 +36,7 @@ import type { CoreMessage, CompressionBlock } from "../../kernel/src/types.js"; import type { BlockView } from "../session.js"; import { refToIndex } from "../../kernel/src/refs.js"; +import { coreHostRefs, hostKeyOf, type CoreHostRef } from "../host-id-channel.js"; import { sha256Hex, type BlockRow, type CcrEntryRow, type IngestSessionInput, type MessageRow, type RefRow, type SessionRow } from "./store.js"; /** Field order of mergeState() output (src/persist.ts) — the loader must @@ -71,6 +80,10 @@ export interface LegacySessionLike { lastMessages?: CoreMessage[]; lastMessagesFolded?: boolean; pluginSnapshot?: CoreMessage[]; + /** #3464 (R1): wire-aligned host message ids, index-for-index with the + * matching leg (null entry = that wire message carries no id). */ + lastMessagesHostIds?: (string | null)[] | null; + pluginSnapshotHostIds?: (string | null)[] | null; contentStore?: { version: 1; byHash: Record; byRef: Record }; } @@ -98,6 +111,8 @@ interface LogEntry { role: string; kind: string; ref: string | null; + hostId: string | null; + hostSlot: number | null; } /** Adapts any Session-shaped object (persist loadSync output) to the @@ -115,6 +130,8 @@ export function toLegacyLike(s: { lastMessages?: LegacySessionLike["lastMessages"]; lastMessagesFolded?: boolean; pluginSnapshot?: LegacySessionLike["pluginSnapshot"]; + lastMessagesHostIds?: LegacySessionLike["lastMessagesHostIds"]; + pluginSnapshotHostIds?: LegacySessionLike["pluginSnapshotHostIds"]; contentStore?: LegacySessionLike["contentStore"]; }): LegacySessionLike { return { @@ -129,6 +146,8 @@ export function toLegacyLike(s: { lastMessages: s.lastMessages, lastMessagesFolded: s.lastMessagesFolded, pluginSnapshot: s.pluginSnapshot, + lastMessagesHostIds: s.lastMessagesHostIds, + pluginSnapshotHostIds: s.pluginSnapshotHostIds, contentStore: s.contentStore, }; } @@ -146,7 +165,14 @@ export function planIngest(session: LegacySessionLike): IngestPlan { // lives in planIngestDelta + the sessions.last_ingested_seq watermark. const log: LogEntry[] = []; const seqByIdentity = new Map(); - const intern = (m: CoreMessage): number => { + // #3464 (R1): on stamped lanes the host key (host_id + slot inside its + // wire message) is the primary identity — byte drift across legs must + // land in ONE row. First-seen wins the seq position; the LATER leg's + // bytes win the row's unit (the snapshot leg interns first, so the + // fresh tail refreshes drifted rows — the stored view must equal the + // in-memory tail or the verify digest false-alarms). + const seqByHostKey = new Map(); + const intern = (m: CoreMessage, hostRef: CoreHostRef | null | undefined): number => { const content = JSON.stringify(m); ingestSerializations.count++; const parsed = JSON.parse(content) as { role: string; contentType?: string }; @@ -154,13 +180,40 @@ export function planIngest(session: LegacySessionLike): IngestPlan { const key = `${m.id}\u0000${entry.hash}`; const existing = seqByIdentity.get(key); if (existing !== undefined) return existing; + if (hostRef !== null && hostRef !== undefined) { + const hk = hostKeyOf(hostRef); + const survivorIdx = seqByHostKey.get(hk); + if (survivorIdx !== undefined) { + const survivor = log[survivorIdx]; + if (survivor.msgId !== entry.msgId || survivor.hash !== entry.hash || survivor.role !== entry.role || survivor.kind !== entry.kind) { + for (const [bk, idx] of seqByIdentity) { + if (idx === survivorIdx) seqByIdentity.delete(bk); + } + survivor.msgId = entry.msgId; + survivor.content = entry.content; + survivor.hash = entry.hash; + survivor.role = entry.role; + survivor.kind = entry.kind; + survivor.ref = byRawRef[entry.msgId] ?? null; + } + seqByIdentity.set(key, survivorIdx); + return survivorIdx; + } + seqByHostKey.set(hk, log.length); + const seq = log.length; + seqByIdentity.set(key, seq); + log.push({ seq, msgId: entry.msgId, content: entry.content, hash: entry.hash, role: entry.role, kind: entry.kind, ref: byRawRef[entry.msgId] ?? null, hostId: hostRef.hostId, hostSlot: hostRef.slot }); + return seq; + } const seq = log.length; seqByIdentity.set(key, seq); - log.push({ seq, msgId: m.id, content: entry.content, hash: entry.hash, role: entry.role, kind: entry.kind, ref: byRawRef[m.id] ?? null }); + log.push({ seq, msgId: m.id, content: entry.content, hash: entry.hash, role: entry.role, kind: entry.kind, ref: byRawRef[m.id] ?? null, hostId: null, hostSlot: null }); return seq; }; - const pluginSnapshotSeqs = session.pluginSnapshot ? session.pluginSnapshot.map(intern) : null; - const tailSeqs = session.lastMessages ? session.lastMessages.map(intern) : null; + const snapshotRefs = session.pluginSnapshot ? coreHostRefs(session.pluginSnapshot, session.pluginSnapshotHostIds) : null; + const tailRefs = session.lastMessages ? coreHostRefs(session.lastMessages, session.lastMessagesHostIds) : null; + const pluginSnapshotSeqs = session.pluginSnapshot ? session.pluginSnapshot.map((m, i) => intern(m, snapshotRefs?.[i] ?? null)) : null; + const tailSeqs = session.lastMessages ? session.lastMessages.map((m, i) => intern(m, tailRefs?.[i] ?? null)) : null; return finishPlan(session, byRawRef, log, pluginSnapshotSeqs, tailSeqs); } @@ -225,11 +278,18 @@ export function planIngestDelta( const byRawRef = session.state.messageRefs.byRaw; const log: LogEntry[] = []; const appended: number[] = []; + // #3464 (R1): stamped entries carry their host key through to the row. + // No dedup against the trusted prefix here — a collision with an + // existing (host_id, host_slot) row is an integrity violation the + // store's strict-append guard aborts this delta for (full ingest then + // reconciles it). Slots are unique within one leg, so no local map. + const tailRefs = coreHostRefs(tail, session.lastMessagesHostIds); for (let i = trustedLen; i < tail.length; i++) { const m = tail[i]; const content = JSON.stringify(m); ingestSerializations.count++; const parsed = JSON.parse(content) as { role: string; contentType?: string }; + const hostRef = tailRefs[i]; // Placeholder seqs under the aligned assumption (no rows above the // watermark); ingestSession renumbers from max(seq)+1 inside the // transaction when a fold left rows above the trusted tip. @@ -243,6 +303,8 @@ export function planIngestDelta( role: parsed.role, kind: parsed.contentType ?? "message", ref: byRawRef[m.id] ?? null, + hostId: hostRef?.hostId ?? null, + hostSlot: hostRef?.slot ?? null, }); } // planIngest's snapshot ternary replicated on the ORIGINAL value: an @@ -284,6 +346,8 @@ function finishPlan( kind: e.kind, content: e.content, contentHash: e.hash, + hostId: e.hostId, + hostSlot: e.hostSlot, })); const blocks: BlockRow[] = session.state.blocks.map((b, ord) => { diff --git a/src/storage/schema.ts b/src/storage/schema.ts index 3267fd460f..e8a773f532 100644 --- a/src/storage/schema.ts +++ b/src/storage/schema.ts @@ -63,7 +63,18 @@ // index. Both directions compatible: an older build opening a v5 // file no-ops at the version gate (extra column ignored); a newer // build upgrades a v4 file with one idempotent ALTER TABLE. -const STORAGE_SCHEMA_VERSION = 5; +// - v6: host message ids (#3464 R1) — messages gains `host_id` / +// `host_slot` (the plugin-stamped host identity of the WIRE message +// that produced the row; NULL on lanes without a host id channel) +// plus partial UNIQUE index idx_messages_host(session_id, host_id, +// host_slot) WHERE host_id IS NOT NULL — under a stamped lane the +// host key is the row's identity, so a byte drift (marker strip, +// protocol re-serialization) refreshes ONE row instead of minting a +// twin. Both directions compatible: an older build opening a v6 +// file no-ops at the version gate (extra columns/index ignored); a +// newer build upgrades a v5 file with two probe-guarded ALTERs and +// one CREATE UNIQUE INDEX IF NOT EXISTS. +export const STORAGE_SCHEMA_VERSION = 6; /** #3250: cap for `journal_size_limit` (see STORAGE_PRAGMAS). Passive and * auto checkpoints move frames into the main db and reset the write offset @@ -133,6 +144,8 @@ CREATE TABLE IF NOT EXISTS messages ( role TEXT NOT NULL, kind TEXT NOT NULL DEFAULT 'message', content_hash TEXT NOT NULL REFERENCES content(hash), + host_id TEXT, + host_slot INTEGER, PRIMARY KEY (session_id, seq), UNIQUE (session_id, msg_id, content_hash) ); @@ -245,6 +258,10 @@ CREATE TABLE IF NOT EXISTS kv_store ( CREATE INDEX IF NOT EXISTS idx_messages_session ON messages(session_id, seq); CREATE INDEX IF NOT EXISTS idx_messages_msg_id ON messages(session_id, msg_id); +-- #3464 (R1): on stamped lanes the host key IS the row identity — one wire +-- message may refresh its bytes across resends (marker strip, re-serialization) +-- but must never mint a second row. Partial: NULL host_id rows are exempt. +CREATE UNIQUE INDEX IF NOT EXISTS idx_messages_host ON messages(session_id, host_id, host_slot) WHERE host_id IS NOT NULL; CREATE INDEX IF NOT EXISTS idx_blocks_session ON blocks(session_id, ord); CREATE INDEX IF NOT EXISTS idx_dead_refs_session ON dead_refs(session_id, ord); CREATE INDEX IF NOT EXISTS idx_refs_session ON refs(session_id, ord); @@ -300,6 +317,22 @@ export function migrateSchema(db: { db.exec("ALTER TABLE sessions ADD COLUMN last_ingested_seq INTEGER"); } } + if (from < 6) { + // v6 host message ids (#3464 R1): two additive NULLable columns + // on messages + the partial host-key uniqueness index. Same + // probe-guarded ALTER discipline as v2/v5; the index create is + // idempotent, so fresh-install DDL that already carries both + // re-lands cleanly. + const mcols = (db.prepare("PRAGMA table_info(messages)").all() as unknown as { name?: unknown }[]) + .map((c) => (typeof c?.name === "string" ? c.name : "")); + if (!mcols.includes("host_id")) { + db.exec("ALTER TABLE messages ADD COLUMN host_id TEXT"); + } + if (!mcols.includes("host_slot")) { + db.exec("ALTER TABLE messages ADD COLUMN host_slot INTEGER"); + } + db.exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_messages_host ON messages(session_id, host_id, host_slot) WHERE host_id IS NOT NULL"); + } db.exec(`PRAGMA user_version = ${STORAGE_SCHEMA_VERSION}`); }); return { from, to: STORAGE_SCHEMA_VERSION }; diff --git a/src/storage/shadow.ts b/src/storage/shadow.ts index fbd57e19de..85663e938d 100644 --- a/src/storage/shadow.ts +++ b/src/storage/shadow.ts @@ -136,7 +136,7 @@ export class UnifiedShadow { * message log authoritatively, so the ingest uses the raw tail whenever * available; both digest legs go through the same override, so parity * accounting stays honest. */ - record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean }, liveCcr?: MessageContentStore): void { + record(record: unknown, rawTail?: { lastMessages: CoreMessage[]; lastMessagesFolded?: boolean; lastMessagesHostIds?: (string | null)[] }, liveCcr?: MessageContentStore): void { if (this.stats.disabled) return; try { const loaded = this.legacyLoad(record); @@ -145,7 +145,14 @@ export class UnifiedShadow { let hasRawTail = false; if (rawTail && rawTail.lastMessages.length > 0) { hasRawTail = true; - legacy = { ...legacy, lastMessages: rawTail.lastMessages, lastMessagesFolded: rawTail.lastMessagesFolded === true }; + // #3464 (R1): the host-id pairing rides with the raw tail — + // absent here means the lane was not stamped for this save. + legacy = { + ...legacy, + lastMessages: rawTail.lastMessages, + lastMessagesFolded: rawTail.lastMessagesFolded === true, + ...(rawTail.lastMessagesHostIds ? { lastMessagesHostIds: rawTail.lastMessagesHostIds } : {}), + }; } // #2960 ③: the live kernel content store when the caller has one // materialized — the envelope only serializes it for fork-receipt diff --git a/src/storage/store.ts b/src/storage/store.ts index 65e660fd36..323cfebc6c 100644 --- a/src/storage/store.ts +++ b/src/storage/store.ts @@ -22,6 +22,7 @@ import { dirname } from "node:path"; import { log as loggerLog } from "../logger.js"; import { openSqliteDatabase, type SqliteDatabase, type SqliteEngineName, type SqliteEngineSource } from "./driver.js"; import { migrateSchema, storagePragmas, WAL_JOURNAL_SIZE_LIMIT } from "./schema.js"; +import { hostKeyOf } from "../host-id-channel.js"; import type { MessageContentStore } from "acp-kernel"; /** #3250: run a PASSIVE wal_checkpoint every N successful ingests. The @@ -90,6 +91,14 @@ export interface MessageRow { * lets ingestSession skip re-hashing every message on every save. * Identical value to sha256Hex(content); optional for direct callers. */ contentHash?: string; + /** #3464 (R1): host-stamped id of the WIRE message that produced this row + * (null on lanes without a host id channel). With hostSlot, the partial + * UNIQUE index makes (session_id, host_id, host_slot) the row identity + * under a stamped lane — byte drift refreshes instead of minting twins. */ + hostId: string | null; + /** Ordinal of this core message within its wire message's block group + * (0-based; one wire message can project N core messages). */ + hostSlot: number | null; } export interface BlockRow { @@ -383,9 +392,17 @@ export class UnifiedStore { const existingIdentity = this.db.prepare( "SELECT 1 FROM messages WHERE session_id = ? AND msg_id = ? AND content_hash = ?", ); + const existingHostKey = this.db.prepare( + "SELECT 1 FROM messages WHERE session_id = ? AND host_id = ? AND host_slot = ?", + ); for (const m of input.messages) { if (!m.contentHash) return false; if (existingIdentity.get(s.sessionId, m.msgId, m.contentHash)) return false; + // #3464 (R1): a delta row whose host key already exists is + // a rewound/re-derived tail too — the partial unique index + // would reject it and its tailSeqs slot would point at + // nothing. Same discipline: bail to full ingest. + if (m.hostId !== null && existingHostKey.get(s.sessionId, m.hostId, m.hostSlot)) return false; } // Placeholder→absolute renumber: the plan emitted // watermark+1.. under the aligned assumption; anchoring at @@ -438,18 +455,45 @@ export class UnifiedStore { // the boot watermark reset still re-establishes delta trust). if (opts?.completeLog === true && !this.hasLaneExt(s.sessionId)) { const incomingIds = new Set(); + // #3464 (R1): stamped-lane rows are identified by their + // host key, NOT by bytes — a drifted row keeps its host + // key with fresher bytes than the plan's byte identity. + // Sweeping it would delete a live row; the refresh step + // below updates it in place instead. + const incomingHostKeys = new Set(); for (const m of messages) { incomingIds.add(`${m.msgId}\u0000${m.contentHash ?? sha256Hex(m.content)}`); + if (m.hostId !== null) incomingHostKeys.add(hostKeyOf({ hostId: m.hostId, slot: m.hostSlot ?? 0 })); } const staleRows = this.db - .prepare("SELECT seq, msg_id, content_hash FROM messages WHERE session_id = ?") - .all(s.sessionId) as { seq: number; msg_id: string; content_hash: string }[]; + .prepare("SELECT seq, msg_id, content_hash, host_id, host_slot FROM messages WHERE session_id = ?") + .all(s.sessionId) as { seq: number; msg_id: string; content_hash: string; host_id: string | null; host_slot: number | null }[]; for (const row of staleRows) { if (incomingIds.has(`${row.msg_id}\u0000${row.content_hash}`)) continue; + if (row.host_id !== null && incomingHostKeys.has(hostKeyOf({ hostId: row.host_id, slot: row.host_slot ?? 0 }))) continue; this.releaseContent(row.content_hash); this.db.prepare("DELETE FROM messages WHERE session_id = ? AND seq = ?").run(s.sessionId, row.seq); } } + // #3464 (R1): stamped-lane refresh — one documented exception + // to the append-only log: a plan row that reuses an existing + // seq via its host key may carry FRESHER bytes than the + // stored row (marker strip / re-serialization drift). Update + // the SAME logical row in place so the raw-tail view stays + // digest-equal to the in-memory tail (#2960 ⑤). Identical + // bytes no-op; unstamped lanes never have hostId rows. + for (const m of messages) { + if (m.hostId === null || !m.contentHash) continue; + const cur = this.db + .prepare("SELECT msg_id, content_hash FROM messages WHERE session_id = ? AND seq = ?") + .get(s.sessionId, m.seq) as { msg_id: string; content_hash: string } | undefined; + if (!cur || cur.content_hash === m.contentHash) continue; + this.putContent(m.content, m.contentHash); + this.db.prepare( + "UPDATE messages SET msg_id = ?, ref = ?, ref_num = ?, role = ?, kind = ?, content_hash = ? WHERE session_id = ? AND seq = ?", + ).run(m.msgId, m.ref, m.refNum, m.role, m.kind, m.contentHash, s.sessionId, m.seq); + this.releaseContent(cur.content_hash); + } } this.db .prepare( @@ -518,9 +562,9 @@ export class UnifiedStore { const hash = this.putContent(m.content, m.contentHash); const res = this.db .prepare( - "INSERT OR IGNORE INTO messages (session_id, seq, msg_id, ref, ref_num, role, kind, content_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + "INSERT OR IGNORE INTO messages (session_id, seq, msg_id, ref, ref_num, role, kind, content_hash, host_id, host_slot) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", ) - .run(s.sessionId, m.seq, m.msgId, m.ref, m.refNum, m.role, m.kind, hash); + .run(s.sessionId, m.seq, m.msgId, m.ref, m.refNum, m.role, m.kind, hash, m.hostId, m.hostSlot); if (res.changes === 0) { // Idempotent re-receipt: the CAS row is still referenced // by the existing message row — undo our refcount bump. @@ -677,12 +721,17 @@ export class UnifiedStore { // stable absolute seqs: same identity (msgId + content hash) // keeps its old seq; new identities continue after max(seq). const existing = this.db - .prepare("SELECT seq, msg_id, content_hash FROM messages WHERE session_id = ?") - .all(s.sessionId) as { seq: number; msg_id: string; content_hash: string }[]; + .prepare("SELECT seq, msg_id, content_hash, host_id, host_slot FROM messages WHERE session_id = ?") + .all(s.sessionId) as { seq: number; msg_id: string; content_hash: string; host_id: string | null; host_slot: number | null }[]; const seqByIdentity = new Map(); + // #3464 (R1): on stamped lanes the host key IS the row's identity — + // byte drift must not mint a new seq (that would break tailSeqs / + // watermark continuity on every marker strip or re-serialization). + const seqByHostKey = new Map(); let nextSeq = 0; for (const row of existing) { seqByIdentity.set(`${row.msg_id}\u0000${row.content_hash}`, row.seq); + if (row.host_id !== null) seqByHostKey.set(hostKeyOf({ hostId: row.host_id, slot: row.host_slot ?? 0 }), row.seq); if (row.seq + 1 > nextSeq) nextSeq = row.seq + 1; } // #3133 watermark gate (full path): the watermark names the abs seq @@ -697,11 +746,20 @@ export class UnifiedStore { // callers without a planner pay the sha256 here. const hash = m.contentHash ?? sha256Hex(m.content); const key = `${m.msgId}\u0000${hash}`; - let abs = seqByIdentity.get(key); + let abs: number | undefined; + // #3464 (R1): host key outranks byte identity — a drifted row + // keeps its original seq and the refresh step rewrites bytes. + if (m.hostId !== null) { + abs = seqByHostKey.get(hostKeyOf({ hostId: m.hostId, slot: m.hostSlot ?? 0 })); + } + if (abs === undefined) { + abs = seqByIdentity.get(key); + } if (abs === undefined) { abs = nextSeq++; - seqByIdentity.set(key, abs); + if (m.hostId !== null) seqByHostKey.set(hostKeyOf({ hostId: m.hostId, slot: m.hostSlot ?? 0 }), abs); } + seqByIdentity.set(key, abs); planToAbs.set(m.seq, abs); return { ...m, seq: abs }; }); @@ -1272,7 +1330,7 @@ export class UnifiedStore { getMessages(sessionId: string): (MessageRow & { contentHash: string })[] { const rows = this.db - .prepare("SELECT seq, msg_id, ref, ref_num, role, kind, content_hash FROM messages WHERE session_id = ? ORDER BY seq") + .prepare("SELECT seq, msg_id, ref, ref_num, role, kind, content_hash, host_id, host_slot FROM messages WHERE session_id = ? ORDER BY seq") .all(sessionId) as Record[]; return rows.map((r) => ({ seq: r.seq as number, @@ -1282,6 +1340,8 @@ export class UnifiedStore { role: r.role as string, kind: r.kind as string, contentHash: r.content_hash as string, + hostId: (r.host_id as string | null) ?? null, + hostSlot: (r.host_slot as number | null) ?? null, content: this.getContent(r.content_hash as string) ?? "", })); } diff --git a/tests/fold-reconcile.test.ts b/tests/fold-reconcile.test.ts index e535336f30..e7c3ce26a7 100644 --- a/tests/fold-reconcile.test.ts +++ b/tests/fold-reconcile.test.ts @@ -822,7 +822,7 @@ describe("reconcileFoldCoverage anchor cap boundary (#2334)", () => { const snapshot = JSON.stringify(session.metadata); resetNormalizedIdentityWork(); const second = reconcileFoldCoverage(session, msgs, opts); - assert.deepEqual(second, { kind: "resend", missing: 0, claims: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }); + assert.deepEqual(second, { kind: "resend", missing: 0, claims: 0, byHost: 0, byPos: 0, byTool: 0, byTwin: 0, bySim: 0, byOrder: 0, byNorm: 0, byTurn: 0, byImage: 0, reaped: 0, unmatched: 0 }); assert.equal(normalizedIdentityWorkCount(), 0, "the overflow tail must not be normalized+hashed and dropped AGAIN every pass (#2334)"); assert.equal(JSON.stringify(session.metadata), snapshot, "steady-state resend leaves metadata byte-stable"); diff --git a/tests/issue3464-host-ids.test.ts b/tests/issue3464-host-ids.test.ts new file mode 100644 index 0000000000..117599933f --- /dev/null +++ b/tests/issue3464-host-ids.test.ts @@ -0,0 +1,456 @@ +/** + * #3464 (R1 of the #2960 identity pipeline): host message ids ingested into + * the canonical store — reconcile becomes exact set algebra instead of + * byte-hash guessing. + * + * Acceptance coverage: + * - stamped lane: one row per host message across drifted resends (no twins); + * - unstamped lane: behavior bit-identical to pre-R1; + * - fork snapshots can map a host id back to its bili ref (messages rows + * carry host_id + ref/ref_num); + * - schema v5→v6 migration is additive and probe-guarded; + * - Pass -1 claims by exact host-id equality before every heuristic pass, + * role-gated, k-th within multi-core groups, and no-ops on unstamped lanes. + */ +import { test, describe } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { STORAGE_DDL_V1_FOR_TESTS, STORAGE_SCHEMA_VERSION } from "../src/storage/schema.ts"; +import { UnifiedStore, sha256Hex } from "../src/storage/store.ts"; +import { openSqliteDatabase } from "../src/storage/driver.ts"; +import { planIngest, type LegacySessionLike } from "../src/storage/ingest.ts"; +import { + HOST_ID_BODY_FIELD, + HOST_ID_MAX_ENTRIES, + HOST_ID_MAX_ENTRY_LEN, + extractHostIds, + collectWireHostIds, + coreHostRefs, + hostKeyOf, +} from "../src/host-id-channel.ts"; +import { + planReconciliation, + reconcileFoldCoverage, + normalizedIdentity, + normalizedIdentityNoToolCallId, + type FoldAnchor, + type ReconcileOptions, +} from "../src/fold-reconcile.ts"; +import type { CoreMessage } from "acp-kernel"; +import type { Session } from "../src/session.ts"; + +process.env.BILI_PERSIST_ZSTD = "0"; +const hermetic = fs.mkdtempSync(path.join(os.tmpdir(), "bili-issue3464-hermetic-")); +process.env.BILI_CONFIG_FILE = path.join(hermetic, "no-config.json"); +process.env.BILI_STORAGE_UNIFIED = "legacy"; + +function tmpdir(prefix: string): string { + return fs.mkdtempSync(path.join(os.tmpdir(), prefix)); +} + +function rmBestEffort(p: string): void { + try { + fs.rmSync(p, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { /* leaked temp dir on a Windows runner is harmless */ } +} + +function msg(id: string, role: string, text: string, extra?: Record): CoreMessage { + return { id, role, contentType: "text", text, ...extra } as CoreMessage; +} + +/** LegacySessionLike fixture — state key order matters (digest-sensitive). */ +function mkLegacy( + id: string, + lastMessages: LegacySessionLike["lastMessages"], + opts?: { + lastMessagesHostIds?: (string | null)[]; + pluginSnapshot?: LegacySessionLike["pluginSnapshot"]; + pluginSnapshotHostIds?: (string | null)[]; + byRaw?: Record; + }, +): LegacySessionLike { + const byRaw = opts?.byRaw ?? {}; + return { + id, + createdAt: 1_000, + lastSeen: 2_000, + meta: { protocol: "anthropic" }, + stats: { requests: 1 }, + metadata: {}, + state: { + blocks: [], + messageRefs: { byRaw, byRef: Object.fromEntries(Object.entries(byRaw).map(([k, v]) => [v, k])) }, + nextBlockId: 1, + nextRunId: 1, + }, + blockContents: new Map(), + lastMessages, + lastMessagesFolded: false, + pluginSnapshot: opts?.pluginSnapshot, + lastMessagesHostIds: opts?.lastMessagesHostIds, + pluginSnapshotHostIds: opts?.pluginSnapshotHostIds, + }; +} + +describe("extractHostIds (#3464)", () => { + const field = HOST_ID_BODY_FIELD; + + test("valid stamp passes through and is stripped", () => { + const parsed: Record = { messages: [{}, {}], [field]: ["H1", null] }; + const ex = extractHostIds(parsed, parsed.messages as readonly unknown[] | undefined); + assert.deepEqual(ex.ids, ["H1", null]); + assert.equal(ex.stripped, true); + assert.ok(!(field in parsed), "reserved field must never survive extraction"); + }); + + test("absent field is a no-op", () => { + const parsed: Record = { messages: [{}] }; + const ex = extractHostIds(parsed, parsed.messages as readonly unknown[] | undefined); + assert.equal(ex.ids, null); + assert.equal(ex.stripped, false); + }); + + test("length mismatch ⇒ malformed, stripped, ignored", () => { + const parsed: Record = { messages: [{}, {}], [field]: ["only-one"] }; + const ex = extractHostIds(parsed, parsed.messages as readonly unknown[] | undefined); + assert.equal(ex.ids, null); + assert.equal(ex.stripped, true); + assert.ok(!(field in parsed)); + }); + + test("non-array field ⇒ malformed, stripped", () => { + const parsed: Record = { messages: [{}], [field]: "H1" }; + const ex = extractHostIds(parsed, parsed.messages as readonly unknown[] | undefined); + assert.equal(ex.ids, null); + assert.equal(ex.stripped, true); + assert.ok(!(field in parsed)); + }); + + test("non-string / empty / oversized / control-char entries ⇒ malformed", () => { + const bad: unknown[][] = [[42], [""], [["x".repeat(HOST_ID_MAX_ENTRY_LEN + 1)]], [["a\nb"]], [["a b"]]]; + for (const raw of bad) { + const parsed: Record = { messages: [{}], [field]: raw }; + const ex = extractHostIds(parsed, parsed.messages as readonly unknown[] | undefined); + assert.equal(ex.ids, null, JSON.stringify(raw)); + assert.equal(ex.stripped, true); + assert.ok(!(field in parsed)); + } + }); + + test("more entries than the channel allows ⇒ malformed", () => { + const n = HOST_ID_MAX_ENTRIES + 1; + const parsed: Record = { messages: new Array(n).fill({}), [field]: new Array(n).fill("H") }; + const ex = extractHostIds(parsed, parsed.messages as readonly unknown[] | undefined); + assert.equal(ex.ids, null); + assert.equal(ex.stripped, true); + }); + + test("missing message array ⇒ malformed (cannot verify alignment)", () => { + const parsed: Record = { [field]: ["H1"] }; + const ex = extractHostIds(parsed, undefined); + assert.equal(ex.ids, null); + assert.equal(ex.stripped, true); + assert.ok(!(field in parsed)); + }); +}); + +describe("collectWireHostIds (#3464 plugin side)", () => { + test("reads each wire message's own id, pads with null", () => { + const out = collectWireHostIds([{ id: "a" }, {}, { id: "c" }], 100); + assert.deepEqual(out, ["a", null, "c"]); + }); + + test("nothing to stamp ⇒ null", () => { + assert.equal(collectWireHostIds([{}, { content: 1 }], 100), null); + }); + + test("ANY malformed entry kills the whole stamp", () => { + assert.equal(collectWireHostIds([{ id: 7 }, { id: "ok" }], 100), null); + assert.equal(collectWireHostIds([{ id: "" }], 100), null); + assert.equal(collectWireHostIds([{ id: "x".repeat(HOST_ID_MAX_ENTRY_LEN + 1) }], 100), null); + assert.equal(collectWireHostIds([{ id: "bad\nid" }], 100), null); + }); + + test("over-cap message count ⇒ null", () => { + const msgs = new Array(HOST_ID_MAX_ENTRIES + 1).fill({ id: "x" }); + assert.equal(collectWireHostIds(msgs, HOST_ID_MAX_ENTRIES), null); + }); +}); + +describe("coreHostRefs (#3464 slot semantics)", () => { + test("multi-core wire message gets per-core slots", () => { + const refs = coreHostRefs( + [{ id: "a", wireIndex: 0 }, { id: "b", wireIndex: 0 }, { id: "c", wireIndex: 1 }], + ["H1", "H1", "H2"], + ); + assert.deepEqual(refs, [{ hostId: "H1", slot: 0 }, { hostId: "H1", slot: 1 }, { hostId: "H2", slot: 0 }]); + assert.notEqual(hostKeyOf(refs[0]!), hostKeyOf(refs[1]!)); + }); + + test("cores without wireIndex get per-core-unique slots", () => { + const refs = coreHostRefs([{ id: "a" }, { id: "b" }, { id: "c" }], ["H1", null, "H2"]); + assert.deepEqual(refs, [{ hostId: "H1", slot: 0 }, null, { hostId: "H2", slot: 2 }]); + }); + + test("unpairable list ⇒ all-null refs", () => { + const refs = coreHostRefs([{ id: "a" }, { id: "b" }], ["only"]); + assert.deepEqual(refs, [null, null]); + assert.deepEqual(coreHostRefs([{ id: "a" }], undefined), [null]); + }); + + test("composite key separator cannot occur inside a validated id", () => { + assert.equal(hostKeyOf({ hostId: "abc", slot: 2 }), "abc\u00002"); + }); +}); + +describe("schema v6 host-id columns (#3464)", () => { + test("fresh DB lands at v6 with columns and partial index", () => { + const dir = tmpdir("bili-3464-fresh-"); + try { + const dbPath = path.join(dir, "index.db"); + UnifiedStore.open(dbPath).close(); + const db = openSqliteDatabase(dbPath).db; + const version = (db.prepare("PRAGMA user_version").get() as unknown as Record)["user_version"]; + assert.equal(version, STORAGE_SCHEMA_VERSION); + const cols = (db.prepare("PRAGMA table_info(messages)").all() as unknown as { name?: unknown }[]) + .map((c) => (typeof c?.name === "string" ? c.name : "")); + assert.ok(cols.includes("host_id"), cols.join(",")); + assert.ok(cols.includes("host_slot"), cols.join(",")); + const indexes = (db.prepare("PRAGMA index_list(messages)").all() as unknown as { name?: unknown }[]) + .map((i) => (typeof i?.name === "string" ? i.name : "")); + assert.ok(indexes.includes("idx_messages_host"), indexes.join(",")); + db.close(); + } finally { + rmBestEffort(dir); + } + }); + + test("v5 database migrates additively to v6", () => { + const v5Ddl = STORAGE_DDL_V1_FOR_TESTS + .replace(/^\s*host_id\s+TEXT,$/m, "") + .replace(/^\s*host_slot\s+INTEGER,$/m, "") + .replace(/^-- #3464 \(R1\)[\s\S]*?CREATE UNIQUE INDEX IF NOT EXISTS idx_messages_host[^\n]*\n/m, ""); + assert.ok(!v5Ddl.includes("host_id"), "synthesized v5 DDL must not carry the v6 columns"); + const dir = tmpdir("bili-3464-migrate-"); + try { + const dbPath = path.join(dir, "index.db"); + const raw = openSqliteDatabase(dbPath).db; + raw.exec(v5Ddl); + raw.exec("PRAGMA user_version = 5"); + raw.close(); + UnifiedStore.open(dbPath).close(); + const db = openSqliteDatabase(dbPath).db; + assert.equal( + (db.prepare("PRAGMA user_version").get() as unknown as Record)["user_version"], + STORAGE_SCHEMA_VERSION, + ); + const cols = (db.prepare("PRAGMA table_info(messages)").all() as unknown as { name?: unknown }[]) + .map((c) => (typeof c?.name === "string" ? c.name : "")); + assert.ok(cols.includes("host_id")); + assert.ok(cols.includes("host_slot")); + const indexes = (db.prepare("PRAGMA index_list(messages)").all() as unknown as { name?: unknown }[]) + .map((i) => (typeof i?.name === "string" ? i.name : "")); + assert.ok(indexes.includes("idx_messages_host")); + // existing rows keep working: a NULL-host row inserts fine + db.exec("INSERT INTO content (hash, byte_len, refcount, algo, bytes) VALUES ('h', 1, 1, 'sha256', x'00')"); + db.exec("INSERT INTO sessions (session_id, namespace, created_at, updated_at) VALUES ('s', 'proxy', 1, 1)"); + db.exec("INSERT INTO messages (session_id, seq, msg_id, role, kind, content_hash) VALUES ('s', 0, 'm', 'user', 'message', 'h')"); + db.close(); + } finally { + rmBestEffort(dir); + } + }); +}); + +describe("planIngest host-keyed dedup (#3464 acceptance 1 & 2)", () => { + test("drifted resend under the same host id yields ONE row, fresher bytes win, position kept", () => { + const snapCore = msg("cs", "user", "original bytes", { wireIndex: 0 }); + const tailCoreA = msg("ctA", "user", "DRIFTED bytes after marker strip", { wireIndex: 0 }); + const tailCoreB = msg("ctB", "assistant", "next turn", { wireIndex: 1 }); + const legacy = mkLegacy("twins", [tailCoreA, tailCoreB], { + lastMessagesHostIds: ["H1", "H2"], + pluginSnapshot: [snapCore], + pluginSnapshotHostIds: ["H1"], + }); + const plan = planIngest(legacy); + const h1Rows = plan.messages.filter((r) => r.hostId === "H1"); + assert.equal(h1Rows.length, 1, "same host id across legs must not mint a twin row"); + assert.equal(h1Rows[0]!.content, JSON.stringify(tailCoreA), "later leg carries fresher bytes"); + assert.equal(h1Rows[0]!.seq, 0, "first-seen position is kept"); + assert.equal(plan.messages.length, 2); + assert.deepEqual(plan.messages.map((r) => r.hostSlot), [0, 0]); + }); + + test("unstamped lane keeps today's byte-keyed rows bit-for-bit", () => { + const snapCore = msg("cs", "user", "original bytes", { wireIndex: 0 }); + const tailCoreA = msg("ctA", "user", "DRIFTED bytes after marker strip", { wireIndex: 0 }); + const legacy = mkLegacy("notwins", [tailCoreA], { + pluginSnapshot: [snapCore], + }); + const plan = planIngest(legacy); + assert.equal(plan.messages.length, 2, "without host ids the twin stays (pre-R1 behavior)"); + assert.ok(plan.messages.every((r) => r.hostId === null && r.hostSlot === null)); + }); + + test("one wire message projecting two cores keeps both rows via slots", () => { + const c1 = msg("c1", "assistant", "text part", { wireIndex: 0 }); + const c2 = msg("c2", "assistant", "{\"rows\":1}", { wireIndex: 0, contentType: "tool_call", toolCallId: "t1", toolName: "q" }); + const legacy = mkLegacy("slots", [c1, c2], { lastMessagesHostIds: ["HP", "HP"] }); + const plan = planIngest(legacy); + assert.equal(plan.messages.length, 2); + assert.deepEqual(plan.messages.map((r) => r.hostSlot).sort(), [0, 1]); + assert.ok(plan.messages.every((r) => r.hostId === "HP")); + }); + + test("fork mapping: a stamped row carries its bili ref (acceptance 3)", () => { + const c1 = msg("c1", "user", "hello", { wireIndex: 0 }); + const c2 = msg("c2", "assistant", "world", { wireIndex: 1 }); + const legacy = mkLegacy("refs", [c1, c2], { + lastMessagesHostIds: ["HA", "HB"], + byRaw: { c1: "m00001", c2: "m00002" }, + }); + const plan = planIngest(legacy); + const ra = plan.messages.find((r) => r.hostId === "HA")!; + const rb = plan.messages.find((r) => r.hostId === "HB")!; + assert.equal(ra.ref, "m00001"); + assert.equal(rb.ref, "m00002"); + }); +}); + +describe("store: host-key seq continuity and in-place refresh (#3464)", () => { + function openStore(): { dir: string; store: UnifiedStore } { + const dir = tmpdir("bili-3464-store-"); + return { dir, store: UnifiedStore.open(path.join(dir, "index.db")) }; + } + + test("byte drift re-ingest reuses the host-keyed seq and refreshes content", () => { + const { dir, store } = openStore(); + try { + const m0 = msg("m0", "user", "first", { wireIndex: 0 }); + const m1v1 = msg("m1v1", "assistant", "second original", { wireIndex: 1 }); + store.ingestSession(planIngest(mkLegacy("cont", [m0, m1v1], { + lastMessagesHostIds: ["H0", "H1"], + }))); + const first = store.getMessages("cont"); + assert.equal(first.length, 2); + const h1Seq = first.find((r) => r.hostId === "H1")!.seq; + + // same turn resends with drifted bytes (marker strip), same host id + const m1v2 = msg("m1v2", "assistant", "second AFTER drift", { wireIndex: 1 }); + store.ingestSession(planIngest(mkLegacy("cont", [m0, m1v2], { + lastMessagesHostIds: ["H0", "H1"], + })), undefined, { completeLog: true }); + const second = store.getMessages("cont"); + assert.equal(second.length, 2, "drift must not mint a third row"); + const h1 = second.find((r) => r.hostId === "H1")!; + assert.equal(h1.seq, h1Seq, "host-key continuity: seq survives byte drift"); + assert.equal(h1.contentHash, sha256Hex(JSON.stringify(m1v2))); + const h0 = second.find((r) => r.hostId === "H0")!; + assert.equal(h0.msgId, "m0"); + } finally { + rmBestEffort(dir); + } + }); +}); + +describe("reconcile Pass -1: exact host-id set algebra (#3464)", () => { + function anchorOf(m: CoreMessage, h?: string): FoldAnchor { + const a: FoldAnchor = { n: normalizedIdentity(m), r: m.role, b: m.text?.length ?? 0, ...(h !== undefined ? { h } : {}) }; + if (m.toolCallId) { + a.t = m.toolCallId; + a.m = normalizedIdentityNoToolCallId(m); + } + return a; + } + + test("vanished covered id reappears under the same host id with alien bytes ⇒ claimed by host id", () => { + const u1 = msg("u1", "user", "first turn"); + const u2 = msg("u2", "user", "please analyze module 2"); + const u3 = msg("u3", "user", "final turn"); + const anchors = { u1: anchorOf(u1, "H1"), u2: anchorOf(u2, "H2"), u3: anchorOf(u3, "H3") }; + // churned: new id AND unrelated words — every byte-based pass misses it + const churned = msg("u2-new", "user", "completely unrelated wording here"); + const inbound = [u1, churned, u3]; + const plan = planReconciliation(["u1", "u2", "u3"], anchors, inbound, new Set(["u1", "u2", "u3"]), undefined, ["H1", "H2", "H3"]); + assert.deepEqual(plan.claims.get("u2"), ["u2-new"]); + assert.equal(plan.byHost, 1); + assert.equal(plan.unmatched.length, 0); + }); + + test("role guard rejects a cross-role claim on the same host id", () => { + const u2 = msg("u2", "user", "please analyze module 2"); + const anchors = { u2: anchorOf(u2, "H2") }; + const wrongRole = msg("u2-new", "assistant", "completely unrelated wording"); + const plan = planReconciliation(["u2"], anchors, [wrongRole], new Set(["u2"]), undefined, ["H2"]); + assert.equal(plan.byHost, 0); + assert.equal(plan.claims.size, 0); + assert.deepEqual(plan.unmatched, ["u2"]); + }); + + test("multi-core group pairs k-th to k-th within the shared host id", () => { + const x0 = msg("x0", "user", "stable head"); + const o1 = msg("o1", "user", "alpha words"); + const o2 = msg("o2", "user", "beta words"); + const x9 = msg("x9", "user", "stable tail"); + const n1 = msg("n1", "user", "alien alpha"); + const n2 = msg("n2", "user", "alien beta"); + const inbound = [x0, n1, n2, x9]; + const covered = new Set(["o1", "o2"]); + const plan = planReconciliation( + ["x0", "o1", "o2", "x9"], + { x0: anchorOf(x0), o1: anchorOf(o1, "HP"), o2: anchorOf(o2, "HP"), x9: anchorOf(x9) }, + inbound, + covered, + undefined, + [null, "HP", "HP", null], + ); + assert.deepEqual(plan.claims.get("o1"), ["n1"]); + assert.deepEqual(plan.claims.get("o2"), ["n2"]); + assert.equal(plan.byHost, 2); + }); + + test("unstamped lane: no host ids ⇒ Pass -1 inert, byte passes unchanged", () => { + const u1 = msg("u1", "user", "first turn"); + const u2 = msg("u2", "user", "please analyze module 2"); + const u3 = msg("u3", "user", "final turn"); + const anchors = { u1: anchorOf(u1), u2: anchorOf(u2), u3: anchorOf(u3) }; + const churnedWs = msg("u2-new", "user", "please analyze module 2\r\n"); + const inbound = [u1, churnedWs, u3]; + const without = planReconciliation(["u1", "u2", "u3"], anchors, inbound, new Set(["u1", "u2", "u3"])); + const withNulls = planReconciliation(["u1", "u2", "u3"], anchors, inbound, new Set(["u1", "u2", "u3"]), undefined, ["H1", "H2", "H3"]); + assert.equal(without.byHost, 0); + assert.deepEqual(without.claims.get("u2"), withNulls.claims.get("u2"), "stamped-lane claim must not change what the byte passes already claim"); + assert.equal(withNulls.byHost, 0, "whitespace churn is claimed positionally, not by host id — counters stay honest"); + }); + + test("anchor.h persists across passes and survives a later unstamped request", () => { + const originals = Array.from({ length: 10 }, (_, i) => msg(`a${i}`, "user", `stable context words ${i}`)); + const allIds = originals.map((m) => m.id!); + const hostIds = originals.map((_, i) => `H${i}`); + const session: Session = { + state: { blocks: [{ active: true, effectiveMessageIds: allIds, directMessageIds: allIds }] }, + metadata: {}, + } as unknown as Session; + const opts = (extra?: Partial): ReconcileOptions & { mode?: string } => + ({ mode: "repair", sessionId: "s1", log: () => {}, ...extra }); + + reconcileFoldCoverage(session, originals, opts({ hostIds })); + const anchorMeta = session.metadata["foldAnchors"] as Record; + assert.equal(anchorMeta["a4"].h, "H4", "first stamped pass seeds anchor.h"); + + const churned = originals.map((m, i) => (i === 4 ? msg("a4-new", "user", "totally different words four") : m)); + const result = reconcileFoldCoverage(session, churned, opts({ hostIds })); + assert.equal(result.kind, "reanchored"); + assert.equal(result.byHost, 1, "Pass -1 claims the drifted message by host id"); + const rewritten = allIds.map((id, i) => (i === 4 ? "a4-new" : id)); + assert.deepEqual(session.state.blocks[0].effectiveMessageIds, rewritten); + + // plugin stops stamping mid-conversation: degrade to heuristic passes, never worse + const wsChurn = originals.map((m, i) => (i === 5 ? msg("a5-new", "user", "stable context words 5\r\n") : m)); + const degraded = reconcileFoldCoverage(session, wsChurn, opts()); + assert.equal(degraded.byHost, 0); + assert.equal(degraded.kind, "reanchored", "positional layer still heals unstamped whitespace churn"); + }); +}); diff --git a/tests/storage-ext.test.ts b/tests/storage-ext.test.ts index 972434bc35..d8bfbaa363 100644 --- a/tests/storage-ext.test.ts +++ b/tests/storage-ext.test.ts @@ -174,17 +174,17 @@ test("v3 databases migrate to v4 in place, idempotently", async () => { const migrated = UnifiedStore.open(dbPath); try { const ver = (migrated as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 5, "open() runs migrateSchema: v3 → v5"); + assert.equal(ver.user_version, 6, "open() runs migrateSchema: v3 → v6"); migrated.setKv("probe", "alive", true); assert.deepEqual(migrated.getKv("probe", "alive"), true); assert.deepEqual(migrated.listSessionIds(), [], "existing v3 content untouched"); } finally { migrated.close(); } - // Reopening a v5 database is a no-op (idempotent migration). + // Reopening a v6 database is a no-op (idempotent migration). const again = UnifiedStore.open(dbPath); const ver2 = (again as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver2.user_version, 5); + assert.equal(ver2.user_version, 6); again.close(); } finally { fs.rmSync(tmp, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); diff --git a/tests/storage-ingest-watermark.test.ts b/tests/storage-ingest-watermark.test.ts index 7f717187fe..0753bcc9c9 100644 --- a/tests/storage-ingest-watermark.test.ts +++ b/tests/storage-ingest-watermark.test.ts @@ -543,7 +543,11 @@ test("watermark: v4 database migrates to v5 additively; newer user_version no-op // a mismatch would show up as sampled-verify drift, not a migration // problem. const { STORAGE_DDL_V1_FOR_TESTS } = await import("../src/storage/schema.ts"); - const v4Ddl = STORAGE_DDL_V1_FOR_TESTS.replace(/^\s*last_ingested_seq INTEGER,$/m, ""); + const v4Ddl = STORAGE_DDL_V1_FOR_TESTS + .replace(/^\s*last_ingested_seq INTEGER,$/m, "") + .replace(/^\s*host_id\s+TEXT,$/m, "") + .replace(/^\s*host_slot\s+INTEGER,$/m, "") + .replace(/^-- #3464 \(R1\)[\s\S]*?CREATE UNIQUE INDEX IF NOT EXISTS idx_messages_host[^\n]*\n/m, ""); const { DatabaseSync } = await import("node:sqlite"); const raw = new DatabaseSync(dbPath); raw.exec(v4Ddl); @@ -558,7 +562,7 @@ test("watermark: v4 database migrates to v5 additively; newer user_version no-op const store = UnifiedStore.open(dbPath); try { const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 5, "open() migrates v4 → v5"); + assert.equal(ver.user_version, 6, "open() migrates v4 → v6"); const row = store.getSessionRow("v4-sess"); assert.ok(row, "legacy session row survives the migration"); assert.equal(row!.lastIngestedSeq, null, "legacy rows read back with a NULL watermark (≡ full ingest)"); diff --git a/tests/storage-unified.test.ts b/tests/storage-unified.test.ts index 6595b846fb..a315ec5469 100644 --- a/tests/storage-unified.test.ts +++ b/tests/storage-unified.test.ts @@ -738,7 +738,7 @@ test("schema v2: dev-window v1 database upgrades in place (identity columns adde assert.equal(row!.namespace, "proxy", "namespace backfills with its constant default"); assert.equal(row!.origin, null, "origin backfills NULL"); const ver = (store as unknown as { db: { prepare(q: string): { get(): unknown } } }).db.prepare("PRAGMA user_version").get() as { user_version: number }; - assert.equal(ver.user_version, 5, "schema version stamped to 5 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column)"); + assert.equal(ver.user_version, 6, "schema version stamped to 6 (v2 columns + v3 identity index + v4 extension namespaces + v5 watermark column + v6 host-id columns)"); // v3 backfill: every pre-existing sessions row claims its // (namespace, conversation_id) identity binding. const binding = (store as unknown as { lookupConversation(ns: string, id: string): string | null }).lookupConversation("proxy", "dev-window-1"); From 7acb7c9984f8548aa4c9c909d74973585e594e5a Mon Sep 17 00:00:00 2001 From: awork Date: Sat, 17 Oct 2026 11:41:14 +0800 Subject: [PATCH 2/3] fix: un-export HostIdExtract flagged by the unused-exports gate Review fix for #3473 (separate from the feature commit): the interface is only used as extractHostIds' return type inside this file, so the leading export trips scripts/check-unused-exports.mjs (INTERNAL_ONLY). Same class as master's 8e110a696. Drops the export; no behavior change. --- src/host-id-channel.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/host-id-channel.ts b/src/host-id-channel.ts index 03afffb4c7..0f88b2c204 100644 --- a/src/host-id-channel.ts +++ b/src/host-id-channel.ts @@ -38,7 +38,7 @@ const HOST_ID_PATTERN = /^[\x21-\x7e]+$/; export const HOST_ID_MAX_ENTRY_LEN = 1024; export const HOST_ID_MAX_ENTRIES = 10_000; -export interface HostIdExtract { +interface HostIdExtract { /** Validated wire-aligned host ids (null = that wire message carries no * id), or null when the field was absent or malformed. */ ids: (string | null)[] | null; From a94d37105044a60723d072c18a31e8ad514cd7a4 Mon Sep 17 00:00:00 2001 From: awork Date: Sat, 17 Oct 2026 11:56:21 +0800 Subject: [PATCH 3/3] fix: regenerate wire-contract manifest goldens for the hostMessageIds capability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review fix for #3473 (separate from the feature commit): the feature adds capabilities.hostMessageIds to the plugin manifest but did not regenerate the golden snapshots, so the wire-contract-golden gate fails on CI (#1304: ledger only grows; goldens change only via explicit regeneration). Justification for the schema change: additive capability advertisement for the #3464 R1 host message-id side channel — old plugins ignore the unknown capability and never stamp; the field name and cap are the channel contract plugins learn at bootstrap. Regenerated with node --import tsx scripts/update-wire-contract-goldens.ts; the only delta is the hostMessageIds block in both manifest snapshots. --- tests/golden/wire-contract/manifest-default.json | 4 ++++ tests/golden/wire-contract/manifest-optin.json | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/tests/golden/wire-contract/manifest-default.json b/tests/golden/wire-contract/manifest-default.json index dfb64e1c5f..5b82108932 100644 --- a/tests/golden/wire-contract/manifest-default.json +++ b/tests/golden/wire-contract/manifest-default.json @@ -632,6 +632,10 @@ ], "endpoint": "/__bili/plugin/fork", "snapshotEndpoint": "/__bili/plugin/snapshot" + }, + "hostMessageIds": { + "bodyField": "_bili_message_ids", + "maxEntries": 10000 } } } diff --git a/tests/golden/wire-contract/manifest-optin.json b/tests/golden/wire-contract/manifest-optin.json index e10877f31b..0eabcef328 100644 --- a/tests/golden/wire-contract/manifest-optin.json +++ b/tests/golden/wire-contract/manifest-optin.json @@ -752,6 +752,10 @@ ], "endpoint": "/__bili/plugin/fork", "snapshotEndpoint": "/__bili/plugin/snapshot" + }, + "hostMessageIds": { + "bodyField": "_bili_message_ids", + "maxEntries": 10000 } } }