Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 46 additions & 8 deletions MESSAGE-IDENTITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,12 @@ host:
| pi (chat, MITM/plugin) | yes (`convertToLlm()` rebuilds arrays; message model has no id field) | no channel |
| omp | yes (pi-family plugin) | no channel |
| hermes plugin | yes (own store, `pre_api_request` hook) | no channel |
| dsh | yes | **host-model id via side-channel** — `MessageBase.id` is stable across representation boundaries; the plugin reads it off the outgoing body and stamps `_bili_message_ids` when the host serializer includes it (#3464) |
| opencode | yes | **host-model id via side-channel** — `msg_…` ids, same mechanism (#3464) |

Coverage must be measured at the **re-join point**, not at ingress.
Coverage must be measured at the **re-join point**, not at ingress. The two
side-channel lanes above are the R1 adoption: the host's own storage id is
the join key, and bili never mints or rewrites it.

## Lane survey: where could an id even live?

Expand Down Expand Up @@ -243,15 +247,49 @@ byte-identical. Identity bookkeeping is never involved.
| Marker-as-identity (tags persisted by hosts) | The tag-echo incidents show the channel corrupts what it carries; current design already strips it at both ends. |
| Implicit aliasing of host-rewritten tool-call ids | Pairing a missing covered id with an inbound twin that has identical content under a different toolCallId requires host knowledge; without it the claim is a guess and misattributes on decompress (#2396: Prime rewrote composite ids across a provider switch). Detection-only shipped; the fix belongs at the host boundary (PLUGIN.md obligation 5). #2480 update: Pass 0 now supplies that host knowledge (position + stored canonical fingerprint), so this class is claimed rather than only counted; PLUGIN.md obligation 5 stands for hosts that can canonicalize earlier. |

## Future direction

If a future host or lane offers a genuinely stable client-side message id,
it may be adopted **only as an additional join hint** — never as a replacement
for the content-hash base. Until then: bytes are the anchor, hashes are the
join, mNNNNN is the ledger, tags are the view.
## Host-stable ids: the R1 channel (#3464, supersedes the old "future direction")

The old stance here — a host-stable id could at best be an *additional join
hint*, never a replacement for the content-hash base — is superseded by the
owner rulings on #2938 (hash demoted to second-class citizen; order-centric,
never fuzzy) and #3464 (R1 of the #2960 identity pipeline). Lanes whose host
model carries a stable per-message id (dsh `MessageBase.id`, opencode
`msg_…`) now ride it into bili through a **side-channel top-level body field**
(`_bili_message_ids`, index-aligned with `messages`; the wire protocols
themselves still offer no id field — the survey above stands). On stamped
lanes identity becomes exact set algebra instead of byte-hash guessing:

- **Storage** (`src/storage/schema.ts` v6): `messages.host_id`/`host_slot` +
partial unique index `(session_id, host_id, host_slot)`. A row's key is the
host id; the byte hash becomes the *content* column. Resending the same
host message under drifted bytes refreshes the row in place (documented
append-only exception, required so the DB view equals the in-memory tail
byte-for-byte — #2960⑤ digest parity) instead of minting a twin row. Seq
assignment prefers host-key continuity, so byte drift no longer churns the
watermark.
- **Reconcile** (`src/fold-reconcile.ts`): Pass -1, before every heuristic
pass — a covered id whose stored anchor carries a host id claims the unique
inbound core with the same host id (k-th pairing within multi-core groups,
role-gated). Heuristic passes (position/tool-call-id/twin/simhash/order
bracket) remain the fallback for unstamped lanes and for cells whose anchor
predates stamping. Unstamped lanes are bit-identical to before.
- **Fork** (`src/plugin.ts` fork snapshot): the response gains an additive,
index-aligned `hostIds` array — the host-id → bili-ref map — outside every
hash by construction, so branch-point comparison stays byte-based for old
and new clients alike.

Adoption is per-lane and self-degrading: the plugin reads whatever ids the
host serializer actually puts in the outgoing body (passive read, no host-API
mutation), stamps only when the proxy advertises
`capabilities.hostMessageIds`, and degrades to the byte-id path whenever the
field is absent or malformed. Lanes whose host model has no message-level id
(pi, omp — see survey) stay on bytes until their hosts add one. Bytes remain
the universal fallback; host ids are the primary key where they exist.

Related: #1496 (this document's prompt), `SESSION-IDENTITY.md` (session
granularity), #1476 + kernel #459/#463 (echo discrimination), #242/#1475
(Responses id constraints), #206/#673 (tag echo), #1479/#1482 (strict-echo
repair), #1039 (tool-call byte-exactness invariant), #2396 (host-rewritten tool-call ids across a provider switch), #2480 (positional fold coverage;
`tests/identity-proof.test.ts` is its proof matrix).
`tests/identity-proof.test.ts` is its proof matrix), #2938 (host message-id
side-channel proposal; owner ruling: order-centric, hash second-class),
#3464 (R1 implementation of the host-id channel).
16 changes: 12 additions & 4 deletions MESSAGE-IDENTITY.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,10 @@ sha256;`assignRefs` 再 first-wins 挂 ref(`if (map.byRaw[message.id]) continue`
| pi(chat,MITM/plugin) | 是(`convertToLlm()` 每次重建数组;消息模型无 id 字段) | 无通道 |
| omp | 是(pi 系插件) | 无通道 |
| hermes 插件 | 是(自有存储 + `pre_api_request` 钩子) | 无通道 |
| dsh | 是 | **宿主模型 id 走旁路字段**——`MessageBase.id` 跨表示边界稳定;插件从出站 body 读 id,宿主序列化器带上它就盖 `_bili_message_ids`(#3464) |
| opencode | 是 | **宿主模型 id 走旁路字段**——`msg_…` id,同一机制(#3464) |

覆盖率必须在 **re-join 时点**衡量,而不是 ingress 时点。
覆盖率必须在 **re-join 时点**衡量,而不是 ingress 时点。上表两条旁路 lane 即 R1 的采用:宿主自己存储里的 id 就是 join key,bili 永不铸造或改写它。

## Lane 普查:id 能放在哪?

Expand Down Expand Up @@ -101,8 +103,14 @@ DeepSeek 类网关要求客户端传回网关自己发出的 reasoning("the reas
| 标记当身份(标签由宿主持久化) | 标签回声事故证明该通道会污染所载内容;现行设计已在两端剥离。 |
| 隐式别名化宿主改写的工具调用 id | 把一个缺失 covered id 与"同内容、不同 toolCallId"的入站孪生配对需要宿主知识;没有它,claim 就是猜测,decompress 时认错(#2396:Prime 跨 provider 切换改写了复合 id)。只交付检测;修复属于宿主边界(PLUGIN.md 义务 5)。#2480 更新:Pass 0 现在提供了那份宿主知识(位置 + 存储的规范指纹),这一类从"只计数"变为"可认领";能在更早边界规范化的宿主,PLUGIN.md 义务 5 依然成立。 |

## 未来方向
## 宿主稳定 id:R1 通道(#3464,取代旧的「未来方向」)

若未来宿主或 lane 提供真正稳定的客户端消息 id,只允许作为**附加 join hint** 采纳——永不替代 content-hash 基座。在那之前:字节是锚,hash 是 join,mNNNNN 是账本,标签是视图。
本文档旧立场——宿主稳定 id 至多算**附加 join hint**、永不替代 content-hash 基座——已被 #2938 的 owner 裁定(hash 降为二等公民;顺序为中心、绝不模糊)与 #3464(#2960 身份管线 R1)取代。宿主模型带稳定消息 id 的 lane(dsh `MessageBase.id`、opencode `msg_…`)现在通过**顶层 body 旁路字段**(`_bili_message_ids`,与 `messages` 下标对齐;wire 协议本身仍无 id 字段——上表普查结论不变)把 id 送进 bili。盖了章的 lane 上,身份从「猜内容像不像」变成精确集合代数:

相关:#1496(本文档的起因)、`SESSION-IDENTITY.zh-CN.md`(会话粒度)、#1476 + kernel #459/#463(回声判别)、#242/#1475(Responses id 约束)、#206/#673(标签回声)、#1479/#1482(strict-echo 修复)、#1039(工具调用字节不变量)、#2396(跨 provider 切换时宿主改写的工具调用 id)、#2480(位置折叠覆盖;`tests/identity-proof.test.ts` 是其证明矩阵)、#2636(切模型形状 churn:assistant-turn key + 图片占位变体 pass)。
- **存储**(`src/storage/schema.ts` v6):`messages.host_id`/`host_slot` + 部分唯一索引 `(session_id, host_id, host_slot)`。行的键是宿主 id,字节 hash 降为内容列。同一条宿主消息以漂移后的字节重发时,**原地刷新该行**(有记录的 append-only 例外,保证 DB 视图与内存尾部逐字节相等——#2960⑤ digest parity),而不是再落一行孪生行。seq 分配优先沿用宿主键连续性,字节漂移不再搅动水位。
- **对账**(`src/fold-reconcile.ts`):Pass -1,排在一切启发式 pass 之前——锚点带宿主 id 的 covered id 直接认领同宿主 id 的唯一入站 core(多 core 组内 k-th 配对,role 门)。启发式 pass(位置/toolCallId/孪生/simhash/顺序括号)保留给未盖章 lane 与盖章前留下的锚点。未盖章 lane 与此前逐位一致。
- **fork**(`src/plugin.ts` fork 快照):响应新增加性的 `hostIds` 数组(与 orderedMessages 下标对齐)——宿主 id → bili ref 映射,构造上不进任何 hash,新旧客户端的分支点比较都保持基于字节。

采用按 lane 渐进、自动降级:插件被动读取宿主序列化器实际写进出站 body 的 id(不改宿主 API),仅在 proxy 广播 `capabilities.hostMessageIds` 时盖章,字段缺失或畸形一律退回字节 id 路径。宿主模型没有消息级 id 的 lane(pi、omp——见普查)在其宿主补齐前保持字节身份。字节仍是通用兜底;宿主 id 在有处即为主键。

相关:#1496(本文档的起因)、`SESSION-IDENTITY.zh-CN.md`(会话粒度)、#1476 + kernel #459/#463(回声判别)、#242/#1475(Responses id 约束)、#206/#673(标签回声)、#1479/#1482(strict-echo 修复)、#1039(工具调用字节不变量)、#2396(跨 provider 切换时宿主改写的工具调用 id)、#2480(位置折叠覆盖;`tests/identity-proof.test.ts` 是其证明矩阵)、#2636(切模型形状 churn:assistant-turn key + 图片占位变体 pass)、#2938(宿主消息 id 旁路通道提案;owner 裁定:顺序为中心、hash 二等公民)、#3464(R1 宿主 id 通道实现)。
6 changes: 5 additions & 1 deletion PLUGIN.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,14 @@ Fetch once at plugin startup.
"toolEndpoint": "/__bili/plugin/tool",
"statusEndpoint": "/__bili/plugin/status",
"capabilities": {
"fork": { "protocolVersion": 2, "acceptsProtocolVersions": [1, 2], "endpoint": "/__bili/plugin/fork", "snapshotEndpoint": "/__bili/plugin/snapshot" }
"fork": { "protocolVersion": 2, "acceptsProtocolVersions": [1, 2], "endpoint": "/__bili/plugin/fork", "snapshotEndpoint": "/__bili/plugin/snapshot" },
"hostMessageIds": { "bodyField": "_bili_message_ids", "maxEntries": 10000 }
}
}
```

`capabilities.hostMessageIds` (#3464): the proxy accepts a **side-channel top-level body field** carrying your conversation's stable per-message ids. On model requests whose body is the Anthropic chat shape (`messages` + `model`), you may add `"_bili_message_ids": [<id-or-null>, …]`, index-aligned with `messages` (use `null` for messages that have no id). The proxy strips the field before forwarding — it never reaches the upstream provider — and uses it as the PRIMARY message identity for that session: storage rows key on the host id (a resend under drifted bytes refreshes the row instead of minting a twin), fold reconciliation claims by exact host-id equality before any heuristic pass, and the fork snapshot answers with the same array so you can map host ids back to bili refs. Omit the field whenever your host has no message-level id (or its serializer does not put one in the body) — identity then falls back to content bytes exactly as before; a malformed array is stripped and ignored for that request only. Never stamp this field on requests carrying bili-internal marks (passthrough / plugin-bypass / compaction) or on incremental-turn marker'd bodies — those paths own their bytes.

Register the tools natively with your agent, in whichever wire format your agent speaks — register exactly what `toolNames` lists. `acp_rule` is advertised only when the proxy's config enables it (`compress.rules: true`, opt-in). Calling a tool that is known but disabled answers `200` with an explanatory `result` instead of an error (#1192). If `protocolVersion` is higher than you know, still register the tools — extra fields in schemas are ignored by agents.

### 2. Request headers
Expand Down Expand Up @@ -220,6 +223,7 @@ Marker divert (#3248 phase 2 — native extensions): instead of switching the pr
3. Forward tool executions verbatim; return `result` as the tool result.
4. Self-disable when not running behind bili (e.g. the agent's baseURL does not point at the proxy) — same convention as billion-context-pi / opencode-acp extensions.
5. Keep tool-call ids byte-stable within one conversation, including when projecting it onto another provider. Tool-message identity keys on `toolCallId` as the protocol-stable pairing factor; a projection that rewrites those ids (e.g. sanitizing stored composite ids for a foreign Responses provider) silently loses fold coverage — compressed tool outputs re-enter the wire unfolded (#2396). Canonicalize ids at the host boundary before the payload reaches the proxy; the proxy detects the shape and names it in the fold-drift log, but never repairs onto guessed pairings (see MESSAGE-IDENTITY.md).
6. If your host model carries a stable per-message id AND its request serializer writes it into the outgoing body, stamp `capabilities.hostMessageIds.bodyField` (index-aligned with `messages`, `null` where absent) on every full-resend Anthropic-shaped model request — read passively off the body you already parse; do not mutate host state to force ids into existence. Gate on the manifest capability (older proxies have no such capability entry and must never see the field), skip incremental-turn marker'd bodies, and keep each id stable for the life of the message (the proxy treats one host id as exactly one message forever, mirroring the never-reused ref contract). Hosts without message-level ids (pi, omp today) simply omit the field (#3464).

## Reference

Expand Down
35 changes: 35 additions & 0 deletions TECHNICAL-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,41 @@ its mode (`pm`) and prepare-responses logs one transition line with the reason
are recorded — never the prompt body. No wire behavior changed: both projections
stay exactly as before; the switch is now *recorded* instead of misattributed.

## Host message ids: exact set algebra on stamped lanes (#3464, R1 of #2960)

Message identity was derived from serialized bytes in every layer; any byte
drift (protocol/model switch, provider rewrite, fork, compaction, marker strip)
collapsed session/message/fold-anchor identity together — the #2915/#2917/#2838
family. R1 gives lanes whose host model has a stable per-message id (dsh
`MessageBase.id`, opencode `msg_…`) an exact identity channel:

- **Transport**: top-level body field `_bili_message_ids` (index-aligned with
`messages`, `null` padding), stripped at the single body-parse chokepoint in
`src/server/handle.ts` before ANY forward path — reserved namespace never
reaches upstream. Plugins read ids passively off the outgoing body (no host
API mutation), gate on manifest `capabilities.hostMessageIds`, and stamp only
Anthropic-shaped full-resend requests (`src/agent/dsh-native.ts`,
`src/agent/opencode-native.ts`). The wire protocols themselves have no id
field (MESSAGE-IDENTITY.md lane survey) — this is a side channel, not a
protocol change.
- **Ingress**: `prepareAnthropic` threads the list to reconcile and the
snapshot; storage keys rows on `(session_id, host_id, host_slot)` (schema
v6, partial unique index). A resend under drifted bytes refreshes the row
IN PLACE (documented append-only exception — the DB view must equal the
in-memory tail byte-for-byte for #2960⑤ digest parity) and reuses the seq
(host-key continuity: drift no longer mints new seqs or churns the ingest
watermark). No host id → today's `(msg_id, content_hash)` behavior,
bit-identical.
- **Reconcile**: Pass -1 in `planReconciliation` claims by exact host-id
equality (k-th within multi-core groups, role-gated) before every heuristic
pass; anchors persist `h` across passes. Fork snapshots answer with the
index-aligned `hostIds` array (outside every hash) so host ids map back to
bili refs.

Lanes without message-level ids (pi, omp, claude-code, codex, kimi, zcode)
stay on byte identity unchanged — adoption is per-lane and self-degrading;
malformed or absent fields never make a request worse than unstamped.

## Responses native chaining is not supported — history must be replayed (#1954)

The Responses-wire adapter (`prepareResponses` in `src/server.ts`, `buildRequest`
Expand Down
10 changes: 10 additions & 0 deletions TECHNICAL-NOTES.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,13 @@ bili 永不拥有用户数据:每个被启动的客户端都跑在**真实 home*
4. **用户发送的消息永不被改写。** 用户自己片段的逐字回声在武装清洗下也保留(#463 豁免,按请求文本匹配,不按形状)。

出口审计网(#2248)保持仅记日志:裸出口(未识别帧、解析失败)原样转发并打 `[tag-echo] raw exit` 日志,新泄漏形状可见而不改线上行为。由 `tests/unified-acp-invariants.test.ts` 钉住(8 个测试,两条 lane)。

## 宿主消息 id:盖章 lane 上的精确集合代数(#3464,#2960 R1)

消息身份原先在每一层都从序列化字节推导;任何字节漂移(换协议/换模型、provider 改写、fork、压缩、marker 剥离)会让会话/消息/折叠锚点三层身份一起塌——#2915/#2917/#2838 那一族。R1 给宿主模型带稳定消息 id 的 lane(dsh `MessageBase.id`、opencode `msg_…`)一条精确身份通道:

- **传输**:顶层 body 字段 `_bili_message_ids`(与 `messages` 下标对齐,`null` 占位),在 `src/server/handle.ts` 唯一的 body 解析咽喉处剥离——保留命名空间永不出站。插件被动读取出站 body 里的 id(不改宿主 API),按 manifest `capabilities.hostMessageIds` 门控,只盖 Anthropic 形状的全量重发请求(`src/agent/dsh-native.ts`、`src/agent/opencode-native.ts`)。wire 协议本身没有 id 字段(MESSAGE-IDENTITY.md 普查)——这是旁路通道,不是协议变更。
- **入口**:`prepareAnthropic` 把列表传给对账与快照;存储按 `(session_id, host_id, host_slot)` 键控行(schema v6,部分唯一索引)。同一条宿主消息以漂移后的字节重发时**原地刷新该行**(有记录的 append-only 例外——DB 视图必须与内存尾部逐字节相等,#2960⑤ digest parity),并沿用原 seq(宿主键连续性:漂移不再铸造新 seq、不再搅动水位)。无宿主 id → 与今天 `(msg_id, content_hash)` 行为逐位一致。
- **对账**:`planReconciliation` 的 Pass -1 在一切启发式 pass 之前按宿主 id 精确认领(多 core 组内 k-th,role 门);锚点跨 pass 持久化 `h`。fork 快照响应带同序 `hostIds` 数组(构造上不进任何 hash),宿主 id 可映射回 bili ref。

没有消息级 id 的 lane(pi、omp、claude-code、codex、kimi、zcode)保持字节身份不变——采用按 lane 渐进、自动降级;字段缺失或畸形绝不比未盖章更差。
Loading
Loading